Truncating/vacuuming relations on full tablespaces
Hi,
Currently, when attempting to vacuum a table on a tablespace with no space
left, we get an error:
postgres=# vacuum test;
ERROR: could not extend file
"pg_tblspc/19605/PG_9.6_201508111/12382/19616_vm": No space left on device
HINT: Check free disk space.
This is because it first tries to grow the visibility map file.
We also get a similar problem when attempting to truncate with restart
identity:
postgres=# truncate table test restart identity;
ERROR: canceling autovacuum task
CONTEXT: automatic vacuum of table "postgres.public.test"
ERROR: could not extend file "base/12382/16391": No space left on device
HINT: Check free disk space.
STATEMENT: truncate table test restart identity;
I guess a workaround for the 2nd case is to truncate without restarting the
identify, then truncate again with restart identify, or just resetting the
sequence. In any case, someone will likely be running this command to free
up space, and they can't due to lack of space.
But shouldn't we not be creating FSM or VM files when truncating?
ISTM that the vacuum case is one we'd really want to avoid, though, as it's
trickier to work around the problem.
Thom
On 9/4/15 7:04 AM, Thom Brown wrote:
But shouldn't we not be creating FSM or VM files when truncating?
Maybe, but even then you still need to create a bunch of new files (at
least one for the table and one for each index), and AFAIK the first
page in each file will be properly initialized, which means each file
will be at least BLKSZ.
ISTM that the vacuum case is one we'd really want to avoid, though, as
it's trickier to work around the problem.
What might make sense is a special 'free up space NOW' mode that focuses
only on attempting to truncate the relation, because if you can't
actually shrink the heap you're not going to make any progress.
But since none of this will help at all in the default case where WAL is
on the same filesystem as the data, I don't know that it's worth it.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Fri, Sep 4, 2015 at 8:04 AM, Thom Brown <thom@linux.com> wrote:
Currently, when attempting to vacuum a table on a tablespace with no space
left, we get an error:postgres=# vacuum test;
ERROR: could not extend file
"pg_tblspc/19605/PG_9.6_201508111/12382/19616_vm": No space left on device
HINT: Check free disk space.This is because it first tries to grow the visibility map file.
We also get a similar problem when attempting to truncate with restart
identity:postgres=# truncate table test restart identity;
ERROR: canceling autovacuum task
CONTEXT: automatic vacuum of table "postgres.public.test"
ERROR: could not extend file "base/12382/16391": No space left on device
HINT: Check free disk space.
STATEMENT: truncate table test restart identity;I guess a workaround for the 2nd case is to truncate without restarting the
identify, then truncate again with restart identify, or just resetting the
sequence. In any case, someone will likely be running this command to free
up space, and they can't due to lack of space.But shouldn't we not be creating FSM or VM files when truncating?
That seems like it might possibly be a good thing to avoid, but we're
not doing it in either of those examples. So, I am confused. In the
first example, the error is happening during VACUUM, not TRUNCATE, and
it's unclear what else we could do besides error out. I mean, we
could make it so that VACUUM fails softly rather than emitting a hard
error if unable to grow the visibility map, but that sounds like an
anti-feature. In the second case, the error is happening during
TRUNCATE, but it's happening on the main fork of the sequence
relation, not any auxiliary fork. So you've got two examples of
things failing here but neither one matches the problem statement.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 15 January 2016 at 15:21, Robert Haas <robertmhaas@gmail.com> wrote:
On Fri, Sep 4, 2015 at 8:04 AM, Thom Brown <thom@linux.com> wrote:
Currently, when attempting to vacuum a table on a tablespace with no space
left, we get an error:postgres=# vacuum test;
ERROR: could not extend file
"pg_tblspc/19605/PG_9.6_201508111/12382/19616_vm": No space left on device
HINT: Check free disk space.This is because it first tries to grow the visibility map file.
We also get a similar problem when attempting to truncate with restart
identity:postgres=# truncate table test restart identity;
ERROR: canceling autovacuum task
CONTEXT: automatic vacuum of table "postgres.public.test"
ERROR: could not extend file "base/12382/16391": No space left on device
HINT: Check free disk space.
STATEMENT: truncate table test restart identity;I guess a workaround for the 2nd case is to truncate without restarting the
identify, then truncate again with restart identify, or just resetting the
sequence. In any case, someone will likely be running this command to free
up space, and they can't due to lack of space.But shouldn't we not be creating FSM or VM files when truncating?
That seems like it might possibly be a good thing to avoid, but we're
not doing it in either of those examples. So, I am confused.
So am I, reading it back I'm not sure why I said that now.
The problem is with attempting to extend some file on a full
tablespace during a vacuum or a truncate. I guess they are different
but related problems.
Thom
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Fri, Jan 15, 2016 at 11:05 AM, Thom Brown <thom@linux.com> wrote:
On 15 January 2016 at 15:21, Robert Haas <robertmhaas@gmail.com> wrote:
On Fri, Sep 4, 2015 at 8:04 AM, Thom Brown <thom@linux.com> wrote:
Currently, when attempting to vacuum a table on a tablespace with no space
left, we get an error:postgres=# vacuum test;
ERROR: could not extend file
"pg_tblspc/19605/PG_9.6_201508111/12382/19616_vm": No space left on device
HINT: Check free disk space.This is because it first tries to grow the visibility map file.
We also get a similar problem when attempting to truncate with restart
identity:postgres=# truncate table test restart identity;
ERROR: canceling autovacuum task
CONTEXT: automatic vacuum of table "postgres.public.test"
ERROR: could not extend file "base/12382/16391": No space left on device
HINT: Check free disk space.
STATEMENT: truncate table test restart identity;I guess a workaround for the 2nd case is to truncate without restarting the
identify, then truncate again with restart identify, or just resetting the
sequence. In any case, someone will likely be running this command to free
up space, and they can't due to lack of space.But shouldn't we not be creating FSM or VM files when truncating?
That seems like it might possibly be a good thing to avoid, but we're
not doing it in either of those examples. So, I am confused.So am I, reading it back I'm not sure why I said that now.
The problem is with attempting to extend some file on a full
tablespace during a vacuum or a truncate. I guess they are different
but related problems.
Well, I think that trying to extend a file on a full tablespace during
truncate would be a problem. However, I can't see any evidence that
we do that, except with RESTART IDENTITY, where it's unavoidable
because you need to recreate the sequence. On the other hand,
extending a file on a full tablespace during VACUUM does not seem to
me to be a bug. It is true that it is remotely possible that you
could have a table with empty space at the end which VACUUM would
truncate but for inability to create the FSM or VM, and that would
suck. On the other hand, suppose you have a table which just happens
to fill the tablespace until it's almost but (you think) not quite
full. Then you VACUUM the table. If it just silently failed to build
the visibility map and then all your subsequent vacuums were really
slow but without any user-visible notice that there's a problem, that
would be awful. So all in all I think the system seems to be behaving
as we would wish, unless there's some other test case that shows us
creating the VM or FSM when it's needless to do so.
Now, I do think it's a somewhat fair complaint that if you have a
tablespace which is totally, 100% full, recovery is difficult. You
can probably DROP the table, but TRUNCATE might fail, and so might
VACUUM. You could argue that DROP is usually a good substitute for
TRUNCATE, although there could be dependencies, but DROP is certainly
not a good substitute for VACUUM. We could address the VACUUM case by
having an optional argument to VACUUM which tells it to skip VM and
FSM maintenance, presumably to be used only in case of emergency. I'm
not sure if it's worth having for what is presumably a pretty rare
case, but it seems like it could be done.
We could try to address the TRUNCATE case by adding a flag to
optionally perform a non-transactional TRUNCATE, like we did prior to
7.4, but I wonder how that's ever really safe. Suppose PostgreSQL
tries to truncate either the table or one of its indexes but then,
when trying to truncate the other, we get an error from the operating
system. We cannot recover by aborting the transaction, nor can we
complete the operation by going forward. That mighta been the sort of
thing we didn't worry about much in the early 7 series, but I don't
think it has much chance of passing muster today.
Anybody else want to weigh in with thoughts on any of this?
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Fri, Jan 15, 2016 at 11:41 AM, Robert Haas <robertmhaas@gmail.com> wrote:
Anybody else want to weigh in with thoughts on any of this?
Leaving aside VACUUM issues for a moment, what problems to you see
with an empty table that has no visibility map or free space map
fork? In other words, for the TRUNCATE case we could either skip
these if there is an error, or not even try to build them at all.
Either seems better than the status quo.
--
Kevin Grittner
EDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Fri, Jan 15, 2016 at 1:24 PM, Kevin Grittner <kgrittn@gmail.com> wrote:
On Fri, Jan 15, 2016 at 11:41 AM, Robert Haas <robertmhaas@gmail.com> wrote:
Anybody else want to weigh in with thoughts on any of this?
Leaving aside VACUUM issues for a moment, what problems to you see
with an empty table that has no visibility map or free space map
fork? In other words, for the TRUNCATE case we could either skip
these if there is an error, or not even try to build them at all.
Either seems better than the status quo.
The status quo *is* that we don't try to build them at all.
rhaas=# create table foo (a int, b int);
CREATE TABLE
rhaas=# insert into foo values (1,1);
INSERT 0 1
rhaas=# vacuum analyze foo;
VACUUM
rhaas=# select relfilenode from pg_class where relname = 'foo';
relfilenode
-------------
16385
(1 row)
In another window:
-rw------- 1 rhaas staff 8192 Jan 15 13:45 16385
-rw------- 1 rhaas staff 24576 Jan 15 13:45 16385_fsm
-rw------- 1 rhaas staff 8192 Jan 15 13:45 16385_vm
Back to the first window:
rhaas=# truncate foo;
TRUNCATE TABLE
rhaas=# select relfilenode from pg_class where relname = 'foo';
relfilenode
-------------
16388
(1 row)
Back to the second window:
[rhaas 16384]$ ls -l 16388*
-rw------- 1 rhaas staff 0 Jan 15 13:46 16388
There's no full disk involved here or anything. Just a plain old TRUNCATE.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Robert Haas <robertmhaas@gmail.com> writes:
Now, I do think it's a somewhat fair complaint that if you have a
tablespace which is totally, 100% full, recovery is difficult. You
can probably DROP the table, but TRUNCATE might fail, and so might
VACUUM. You could argue that DROP is usually a good substitute for
TRUNCATE, although there could be dependencies, but DROP is certainly
not a good substitute for VACUUM. We could address the VACUUM case by
having an optional argument to VACUUM which tells it to skip VM and
FSM maintenance, presumably to be used only in case of emergency. I'm
not sure if it's worth having for what is presumably a pretty rare
case, but it seems like it could be done.
Presumably the hope would be that VACUUM would truncate off some of the
heap, else there's not much good that's going to happen. That leaves
me wondering exactly what the invariant is for the maps, and if it's
okay to not touch them during a heap truncation.
I believe that there would be ramifications for some of the index AMs
too. For example, if left to its own devices GIN would consider VACUUM
to include flushing its pending-list pages, which more than likely will
increase not reduce the total index size. I'm not sure that it has
any ability to omit that step; can it remove dead entries directly off
the pending pages, or only from the main index?
On the whole this sounds like a lot of work, and a lot of hard-to-test
seldom-exercised code, for a very very narrow corner case.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Fri, Jan 15, 2016 at 11:16 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
I believe that there would be ramifications for some of the index AMs
too. For example, if left to its own devices GIN would consider VACUUM
to include flushing its pending-list pages, which more than likely will
increase not reduce the total index size. I'm not sure that it has
any ability to omit that step; can it remove dead entries directly off
the pending pages, or only from the main index?
It cannot vacuum the pending list directly. That is why it is a bug
for the vacuum to short-cut out of the pending list cleanup step when
it finds someone else already cleaning it. For correctness it has to
either clean it itself, or wait until the other process is done (or at
least, done up to the point where the tail was at the time the vacuum
started).
Cheers,
Jeff
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Fri, Jan 15, 2016 at 2:16 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Robert Haas <robertmhaas@gmail.com> writes:
Now, I do think it's a somewhat fair complaint that if you have a
tablespace which is totally, 100% full, recovery is difficult. You
can probably DROP the table, but TRUNCATE might fail, and so might
VACUUM. You could argue that DROP is usually a good substitute for
TRUNCATE, although there could be dependencies, but DROP is certainly
not a good substitute for VACUUM. We could address the VACUUM case by
having an optional argument to VACUUM which tells it to skip VM and
FSM maintenance, presumably to be used only in case of emergency. I'm
not sure if it's worth having for what is presumably a pretty rare
case, but it seems like it could be done.Presumably the hope would be that VACUUM would truncate off some of the
heap, else there's not much good that's going to happen. That leaves
me wondering exactly what the invariant is for the maps, and if it's
okay to not touch them during a heap truncation.
No, you're missing the point, or at least I think you are. Suppose
somebody creates a big table and then deletes all the tuples in the
second half, but VACUUM never runs. When at last VACUUM does run on
that table, it will try to build the VM and FSM forks as it scans the
table, and will only truncate AFTER that has been done. If building
the VM and FSM forks fails because there is no freespace, we will
never reach the part of the operation that could create some.
The key point is that both the VM and the FSM are optional. If
there's no VM, vacuum will have to visit every page in the table and
index-only scans will never be index-only, but everything still works.
If there's no FSM, we'll assume the table has no internal freespace,
so inserts will extend the table. Those consequences are bad, of
course, so we really want vacuum to succeed in creating the VM and
FSM. However, when a failure creating the FSM or VM causes us not to
reach the truncation step, then there's no way to shrink the table.
That's not good.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Robert Haas <robertmhaas@gmail.com> writes:
On Fri, Jan 15, 2016 at 2:16 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Presumably the hope would be that VACUUM would truncate off some of the
heap, else there's not much good that's going to happen. That leaves
me wondering exactly what the invariant is for the maps, and if it's
okay to not touch them during a heap truncation.
No, you're missing the point, or at least I think you are. Suppose
somebody creates a big table and then deletes all the tuples in the
second half, but VACUUM never runs. When at last VACUUM does run on
that table, it will try to build the VM and FSM forks as it scans the
table, and will only truncate AFTER that has been done. If building
the VM and FSM forks fails because there is no freespace, we will
never reach the part of the operation that could create some.
No, I follow that perfectly. I think you missed *my* point, which is:
suppose that we do have a full-length VM and/or FSM fork for a relation,
and VACUUM decides to truncate the relation. Is it okay to not truncate
the VM/FSM? If it isn't, we're going to have to have very tricky
semantics for any "don't touch the map forks" option, because it will
have to suppress only some of VACUUM's map updates.
If the map invariants are such that leaving garbage in them is
unconditionally safe, then this isn't a problem; but I'm unsure of that.
The key point is that both the VM and the FSM are optional.
No, the key point is whether it's okay if they *are* there and contain
lies, or self-inconsistent data.
An alternative approach that might avoid such worries is to have a mode
wherein VACUUM always truncates the map forks to nothing, rather than
attempting to update them.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Mon, Jan 18, 2016 at 2:26 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Robert Haas <robertmhaas@gmail.com> writes:
On Fri, Jan 15, 2016 at 2:16 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Presumably the hope would be that VACUUM would truncate off some of the
heap, else there's not much good that's going to happen. That leaves
me wondering exactly what the invariant is for the maps, and if it's
okay to not touch them during a heap truncation.No, you're missing the point, or at least I think you are. Suppose
somebody creates a big table and then deletes all the tuples in the
second half, but VACUUM never runs. When at last VACUUM does run on
that table, it will try to build the VM and FSM forks as it scans the
table, and will only truncate AFTER that has been done. If building
the VM and FSM forks fails because there is no freespace, we will
never reach the part of the operation that could create some.No, I follow that perfectly. I think you missed *my* point, which is:
suppose that we do have a full-length VM and/or FSM fork for a relation,
and VACUUM decides to truncate the relation. Is it okay to not truncate
the VM/FSM? If it isn't, we're going to have to have very tricky
semantics for any "don't touch the map forks" option, because it will
have to suppress only some of VACUUM's map updates.
Oh, I see. I think it's probably not a good idea to skip truncating
those maps, but perhaps the option could be defined as making no new
entries, rather than ignoring them altogether, so that they never
grow. It seems that both of those are defined in such a way that if
page Y follows page X in the heap, the entries for page Y in the
relation fork will follow the one for page X. So truncating them
should be OK; it's just growing them that we need to avoid.
An alternative approach that might avoid such worries is to have a mode
wherein VACUUM always truncates the map forks to nothing, rather than
attempting to update them.
That could work, too, but might be stronger medicine than needed.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Tue, Jan 19, 2016 at 2:04 AM, Robert Haas <robertmhaas@gmail.com> wrote:
On Mon, Jan 18, 2016 at 2:26 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Robert Haas <robertmhaas@gmail.com> writes:
On Fri, Jan 15, 2016 at 2:16 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Presumably the hope would be that VACUUM would truncate off some of the
heap, else there's not much good that's going to happen. That leaves
me wondering exactly what the invariant is for the maps, and if it's
okay to not touch them during a heap truncation.No, you're missing the point, or at least I think you are. Suppose
somebody creates a big table and then deletes all the tuples in the
second half, but VACUUM never runs. When at last VACUUM does run on
that table, it will try to build the VM and FSM forks as it scans the
table, and will only truncate AFTER that has been done. If building
the VM and FSM forks fails because there is no freespace, we will
never reach the part of the operation that could create some.No, I follow that perfectly. I think you missed *my* point, which is:
suppose that we do have a full-length VM and/or FSM fork for a relation,
and VACUUM decides to truncate the relation. Is it okay to not truncate
the VM/FSM? If it isn't, we're going to have to have very tricky
semantics for any "don't touch the map forks" option, because it will
have to suppress only some of VACUUM's map updates.Oh, I see. I think it's probably not a good idea to skip truncating
those maps, but perhaps the option could be defined as making no new
entries, rather than ignoring them altogether, so that they never
grow. It seems that both of those are defined in such a way that if
page Y follows page X in the heap, the entries for page Y in the
relation fork will follow the one for page X. So truncating them
should be OK; it's just growing them that we need to avoid.
Thank you Robert. PFA basic patch, it introduces EMERGENCY option to VACUUM
that forces to avoid extend any entries in the VM or FSM. It seems working
fine in simple test scenarios e.g.
postgres=# create table test1 as (select generate_series(1,100000));
SELECT 100000
postgres=# vacuum EMERGENCY test1;
VACUUM
postgres=# select pg_relation_filepath('test1');
pg_relation_filepath
----------------------
base/13250/16384
(1 row)
[asif@centos66 inst_96]$ find . | grep base/13250/16384
./data/base/13250/16384
postgres=# vacuum test1;
VACUUM
[asif@centos66 inst_96]$ find . | grep base/13250/16384
./data/base/13250/16384
./data/base/13250/16384_fsm
./data/base/13250/16384_vm
Please do let me know if I missed something or more information is
required. Thanks.
Regards,
Muhammad Asif Naeem
Show quoted text
An alternative approach that might avoid such worries is to have a mode
wherein VACUUM always truncates the map forks to nothing, rather than
attempting to update them.That could work, too, but might be stronger medicine than needed.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
VACUUM_EMERGENCY_Option_v1.patchapplication/octet-stream; name=VACUUM_EMERGENCY_Option_v1.patchDownload
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index c740952..69bb0d4 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -944,7 +944,7 @@ terminate_brin_buildstate(BrinBuildState *state)
page = BufferGetPage(state->bs_currentInsertBuf);
RecordPageWithFreeSpace(state->bs_irel,
BufferGetBlockNumber(state->bs_currentInsertBuf),
- PageGetFreeSpace(page));
+ PageGetFreeSpace(page), true);
ReleaseBuffer(state->bs_currentInsertBuf);
}
diff --git a/src/backend/access/brin/brin_pageops.c b/src/backend/access/brin/brin_pageops.c
index d0ca485..65564ab 100644
--- a/src/backend/access/brin/brin_pageops.c
+++ b/src/backend/access/brin/brin_pageops.c
@@ -310,7 +310,7 @@ brin_doupdate(Relation idxrel, BlockNumber pagesPerRange,
if (extended)
{
Assert(BlockNumberIsValid(newblk));
- RecordPageWithFreeSpace(idxrel, newblk, freespace);
+ RecordPageWithFreeSpace(idxrel, newblk, freespace, true);
FreeSpaceMapVacuum(idxrel);
}
@@ -635,7 +635,8 @@ brin_page_cleanup(Relation idxrel, Buffer buf)
freespace = br_page_get_freespace(page);
if (freespace > GetRecordedFreeSpace(idxrel, BufferGetBlockNumber(buf)))
{
- RecordPageWithFreeSpace(idxrel, BufferGetBlockNumber(buf), freespace);
+ RecordPageWithFreeSpace(idxrel, BufferGetBlockNumber(buf), freespace,
+ true);
return true;
}
@@ -794,7 +795,7 @@ brin_getinsertbuffer(Relation irel, Buffer oldbuf, Size itemsz,
*/
if (*extended)
RecordPageWithFreeSpace(irel, BufferGetBlockNumber(buf),
- freespace);
+ freespace, true);
/*
* Lock the old buffer if not locked already. Note that in this
@@ -874,7 +875,7 @@ brin_initialize_empty_new_buffer(Relation idxrel, Buffer buffer)
* pages whose FSM records were forgotten in a crash.
*/
RecordPageWithFreeSpace(idxrel, BufferGetBlockNumber(buffer),
- br_page_get_freespace(page));
+ br_page_get_freespace(page), true);
}
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 34ba385..e281ea9 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -3028,7 +3028,7 @@ heap_delete(Relation relation, ItemPointer tid,
* the lock.
*/
if (PageIsAllVisible(page))
- visibilitymap_pin(relation, block, &vmbuffer);
+ visibilitymap_pin(relation, block, &vmbuffer, true);
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
@@ -3041,7 +3041,7 @@ heap_delete(Relation relation, ItemPointer tid,
if (vmbuffer == InvalidBuffer && PageIsAllVisible(page))
{
LockBuffer(buffer, BUFFER_LOCK_UNLOCK);
- visibilitymap_pin(relation, block, &vmbuffer);
+ visibilitymap_pin(relation, block, &vmbuffer, true);
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
}
@@ -3518,7 +3518,7 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup,
* the lock.
*/
if (PageIsAllVisible(page))
- visibilitymap_pin(relation, block, &vmbuffer);
+ visibilitymap_pin(relation, block, &vmbuffer, true);
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
@@ -3819,7 +3819,7 @@ l2:
if (vmbuffer == InvalidBuffer && PageIsAllVisible(page))
{
LockBuffer(buffer, BUFFER_LOCK_UNLOCK);
- visibilitymap_pin(relation, block, &vmbuffer);
+ visibilitymap_pin(relation, block, &vmbuffer, true);
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
goto l2;
}
@@ -7893,7 +7893,7 @@ heap_xlog_visible(XLogReaderState *record)
LockBuffer(vmbuffer, BUFFER_LOCK_UNLOCK);
reln = CreateFakeRelcacheEntry(rnode);
- visibilitymap_pin(reln, blkno, &vmbuffer);
+ visibilitymap_pin(reln, blkno, &vmbuffer, true);
/*
* Don't set the bit if replay has already passed this point.
@@ -8025,7 +8025,7 @@ heap_xlog_delete(XLogReaderState *record)
Relation reln = CreateFakeRelcacheEntry(target_node);
Buffer vmbuffer = InvalidBuffer;
- visibilitymap_pin(reln, blkno, &vmbuffer);
+ visibilitymap_pin(reln, blkno, &vmbuffer, true);
visibilitymap_clear(reln, blkno, vmbuffer);
ReleaseBuffer(vmbuffer);
FreeFakeRelcacheEntry(reln);
@@ -8103,7 +8103,7 @@ heap_xlog_insert(XLogReaderState *record)
Relation reln = CreateFakeRelcacheEntry(target_node);
Buffer vmbuffer = InvalidBuffer;
- visibilitymap_pin(reln, blkno, &vmbuffer);
+ visibilitymap_pin(reln, blkno, &vmbuffer, true);
visibilitymap_clear(reln, blkno, vmbuffer);
ReleaseBuffer(vmbuffer);
FreeFakeRelcacheEntry(reln);
@@ -8223,7 +8223,7 @@ heap_xlog_multi_insert(XLogReaderState *record)
Relation reln = CreateFakeRelcacheEntry(rnode);
Buffer vmbuffer = InvalidBuffer;
- visibilitymap_pin(reln, blkno, &vmbuffer);
+ visibilitymap_pin(reln, blkno, &vmbuffer, true);
visibilitymap_clear(reln, blkno, vmbuffer);
ReleaseBuffer(vmbuffer);
FreeFakeRelcacheEntry(reln);
@@ -8378,7 +8378,7 @@ heap_xlog_update(XLogReaderState *record, bool hot_update)
Relation reln = CreateFakeRelcacheEntry(rnode);
Buffer vmbuffer = InvalidBuffer;
- visibilitymap_pin(reln, oldblk, &vmbuffer);
+ visibilitymap_pin(reln, oldblk, &vmbuffer, true);
visibilitymap_clear(reln, oldblk, vmbuffer);
ReleaseBuffer(vmbuffer);
FreeFakeRelcacheEntry(reln);
@@ -8462,7 +8462,7 @@ heap_xlog_update(XLogReaderState *record, bool hot_update)
Relation reln = CreateFakeRelcacheEntry(rnode);
Buffer vmbuffer = InvalidBuffer;
- visibilitymap_pin(reln, newblk, &vmbuffer);
+ visibilitymap_pin(reln, newblk, &vmbuffer, true);
visibilitymap_clear(reln, newblk, vmbuffer);
ReleaseBuffer(vmbuffer);
FreeFakeRelcacheEntry(reln);
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index 8140418..ed86143 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -147,9 +147,9 @@ GetVisibilityMapPins(Relation relation, Buffer buffer1, Buffer buffer2,
/* Get pins. */
if (need_to_pin_buffer1)
- visibilitymap_pin(relation, block1, vmbuffer1);
+ visibilitymap_pin(relation, block1, vmbuffer1, true);
if (need_to_pin_buffer2)
- visibilitymap_pin(relation, block2, vmbuffer2);
+ visibilitymap_pin(relation, block2, vmbuffer2, true);
/* Relock buffers. */
LockBuffer(buffer1, BUFFER_LOCK_EXCLUSIVE);
@@ -328,7 +328,7 @@ RelationGetBufferForTuple(Relation relation, Size len,
/* easy case */
buffer = ReadBufferBI(relation, targetBlock, bistate);
if (PageIsAllVisible(BufferGetPage(buffer)))
- visibilitymap_pin(relation, targetBlock, vmbuffer);
+ visibilitymap_pin(relation, targetBlock, vmbuffer, true);
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
}
else if (otherBlock == targetBlock)
@@ -336,7 +336,7 @@ RelationGetBufferForTuple(Relation relation, Size len,
/* also easy case */
buffer = otherBuffer;
if (PageIsAllVisible(BufferGetPage(buffer)))
- visibilitymap_pin(relation, targetBlock, vmbuffer);
+ visibilitymap_pin(relation, targetBlock, vmbuffer, true);
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
}
else if (otherBlock < targetBlock)
@@ -344,7 +344,7 @@ RelationGetBufferForTuple(Relation relation, Size len,
/* lock other buffer first */
buffer = ReadBuffer(relation, targetBlock);
if (PageIsAllVisible(BufferGetPage(buffer)))
- visibilitymap_pin(relation, targetBlock, vmbuffer);
+ visibilitymap_pin(relation, targetBlock, vmbuffer, true);
LockBuffer(otherBuffer, BUFFER_LOCK_EXCLUSIVE);
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
}
@@ -353,7 +353,7 @@ RelationGetBufferForTuple(Relation relation, Size len,
/* lock target buffer first */
buffer = ReadBuffer(relation, targetBlock);
if (PageIsAllVisible(BufferGetPage(buffer)))
- visibilitymap_pin(relation, targetBlock, vmbuffer);
+ visibilitymap_pin(relation, targetBlock, vmbuffer, true);
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
LockBuffer(otherBuffer, BUFFER_LOCK_EXCLUSIVE);
}
diff --git a/src/backend/access/heap/visibilitymap.c b/src/backend/access/heap/visibilitymap.c
index eaab4be..6c98fa0 100644
--- a/src/backend/access/heap/visibilitymap.c
+++ b/src/backend/access/heap/visibilitymap.c
@@ -208,7 +208,7 @@ visibilitymap_clear(Relation rel, BlockNumber heapBlk, Buffer buf)
* If the page doesn't exist in the map file yet, it is extended.
*/
void
-visibilitymap_pin(Relation rel, BlockNumber heapBlk, Buffer *buf)
+visibilitymap_pin(Relation rel, BlockNumber heapBlk, Buffer *buf, bool extend)
{
BlockNumber mapBlock = HEAPBLK_TO_MAPBLOCK(heapBlk);
@@ -220,7 +220,7 @@ visibilitymap_pin(Relation rel, BlockNumber heapBlk, Buffer *buf)
ReleaseBuffer(*buf);
}
- *buf = vm_readbuf(rel, mapBlock, true);
+ *buf = vm_readbuf(rel, mapBlock, extend);
}
/*
@@ -283,6 +283,10 @@ visibilitymap_set(Relation rel, BlockNumber heapBlk, Buffer heapBuf,
if (BufferIsValid(heapBuf) && BufferGetBlockNumber(heapBuf) != heapBlk)
elog(ERROR, "wrong heap buffer passed to visibilitymap_set");
+ /* In case of invalid buffer just return */
+ if(vmBuf == InvalidBuffer)
+ return;
+
/* Check that we have the right VM page pinned */
if (!BufferIsValid(vmBuf) || BufferGetBlockNumber(vmBuf) != mapBlock)
elog(ERROR, "wrong VM buffer passed to visibilitymap_set");
diff --git a/src/backend/commands/vacuumlazy.c b/src/backend/commands/vacuumlazy.c
index 52e19b3..5dbd619 100644
--- a/src/backend/commands/vacuumlazy.c
+++ b/src/backend/commands/vacuumlazy.c
@@ -134,7 +134,7 @@ static TransactionId FreezeLimit;
static MultiXactId MultiXactCutoff;
static BufferAccessStrategy vac_strategy;
-
+static bool Extend_VM_FSM = true;
/* non-export function prototypes */
static void lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats,
@@ -209,6 +209,8 @@ lazy_vacuum_rel(Relation onerel, int options, VacuumParams *params,
else
elevel = DEBUG2;
+ Extend_VM_FSM = (options & VACOPT_EMERGENCY) ? false : true;
+
pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM,
RelationGetRelid(onerel));
@@ -737,7 +739,7 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats,
* cycle of index vacuuming.
*
*/
- visibilitymap_pin(onerel, blkno, &vmbuffer);
+ visibilitymap_pin(onerel, blkno, &vmbuffer, Extend_VM_FSM);
buf = ReadBufferExtended(onerel, MAIN_FORKNUM, blkno,
RBM_NORMAL, vac_strategy);
@@ -843,7 +845,7 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats,
MarkBufferDirty(buf);
UnlockReleaseBuffer(buf);
- RecordPageWithFreeSpace(onerel, blkno, freespace);
+ RecordPageWithFreeSpace(onerel, blkno, freespace, Extend_VM_FSM);
continue;
}
@@ -882,7 +884,7 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats,
}
UnlockReleaseBuffer(buf);
- RecordPageWithFreeSpace(onerel, blkno, freespace);
+ RecordPageWithFreeSpace(onerel, blkno, freespace, Extend_VM_FSM);
continue;
}
@@ -1223,7 +1225,7 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats,
* taken if there are no indexes.)
*/
if (vacrelstats->num_dead_tuples == prev_dead_count)
- RecordPageWithFreeSpace(onerel, blkno, freespace);
+ RecordPageWithFreeSpace(onerel, blkno, freespace, Extend_VM_FSM);
}
/* report that everything is scanned and vacuumed */
@@ -1382,7 +1384,7 @@ lazy_vacuum_heap(Relation onerel, LVRelStats *vacrelstats)
freespace = PageGetHeapFreeSpace(page);
UnlockReleaseBuffer(buf);
- RecordPageWithFreeSpace(onerel, tblk, freespace);
+ RecordPageWithFreeSpace(onerel, tblk, freespace, Extend_VM_FSM);
npages++;
}
@@ -1657,6 +1659,10 @@ should_attempt_truncation(LVRelStats *vacrelstats)
{
BlockNumber possibly_freeable;
+ /* In case of EMERGENCY option always attempt truncate */
+ if(!Extend_VM_FSM)
+ return true;
+
possibly_freeable = vacrelstats->rel_pages - vacrelstats->nonempty_pages;
if (possibly_freeable > 0 &&
(possibly_freeable >= REL_TRUNCATE_MINIMUM ||
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index b9aeb31..e01077e 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -406,7 +406,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <node> overlay_placing substr_from substr_for
%type <boolean> opt_instead
-%type <boolean> opt_unique opt_concurrently opt_verbose opt_full
+%type <boolean> opt_unique opt_concurrently opt_verbose opt_full opt_emergency
%type <boolean> opt_freeze opt_default opt_recheck
%type <defelt> opt_binary opt_oids copy_delimiter
@@ -580,8 +580,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
DEFERRABLE DEFERRED DEFINER DELETE_P DELIMITER DELIMITERS DESC
DICTIONARY DISABLE_P DISCARD DISTINCT DO DOCUMENT_P DOMAIN_P DOUBLE_P DROP
- EACH ELSE ENABLE_P ENCODING ENCRYPTED END_P ENUM_P ESCAPE EVENT EXCEPT
- EXCLUDE EXCLUDING EXCLUSIVE EXECUTE EXISTS EXPLAIN
+ EACH ELSE EMERGENCY ENABLE_P ENCODING ENCRYPTED END_P ENUM_P ESCAPE EVENT
+ EXCEPT EXCLUDE EXCLUDING EXCLUSIVE EXECUTE EXISTS EXPLAIN
EXTENSION EXTERNAL EXTRACT
FALSE_P FAMILY FETCH FILTER FIRST_P FLOAT_P FOLLOWING FOR
@@ -9228,7 +9228,7 @@ cluster_index_specification:
*
*****************************************************************************/
-VacuumStmt: VACUUM opt_full opt_freeze opt_verbose
+VacuumStmt: VACUUM opt_full opt_freeze opt_verbose opt_emergency
{
VacuumStmt *n = makeNode(VacuumStmt);
n->options = VACOPT_VACUUM;
@@ -9238,11 +9238,13 @@ VacuumStmt: VACUUM opt_full opt_freeze opt_verbose
n->options |= VACOPT_FREEZE;
if ($4)
n->options |= VACOPT_VERBOSE;
+ if ($5)
+ n->options |= VACOPT_EMERGENCY;
n->relation = NULL;
n->va_cols = NIL;
$$ = (Node *)n;
}
- | VACUUM opt_full opt_freeze opt_verbose qualified_name
+ | VACUUM opt_full opt_freeze opt_verbose opt_emergency qualified_name
{
VacuumStmt *n = makeNode(VacuumStmt);
n->options = VACOPT_VACUUM;
@@ -9252,13 +9254,15 @@ VacuumStmt: VACUUM opt_full opt_freeze opt_verbose
n->options |= VACOPT_FREEZE;
if ($4)
n->options |= VACOPT_VERBOSE;
- n->relation = $5;
+ if ($5)
+ n->options |= VACOPT_EMERGENCY;
+ n->relation = $6;
n->va_cols = NIL;
$$ = (Node *)n;
}
- | VACUUM opt_full opt_freeze opt_verbose AnalyzeStmt
+ | VACUUM opt_full opt_freeze opt_verbose opt_emergency AnalyzeStmt
{
- VacuumStmt *n = (VacuumStmt *) $5;
+ VacuumStmt *n = (VacuumStmt *) $6;
n->options |= VACOPT_VACUUM;
if ($2)
n->options |= VACOPT_FULL;
@@ -9266,6 +9270,8 @@ VacuumStmt: VACUUM opt_full opt_freeze opt_verbose
n->options |= VACOPT_FREEZE;
if ($4)
n->options |= VACOPT_VERBOSE;
+ if ($5)
+ n->options |= VACOPT_EMERGENCY;
$$ = (Node *)n;
}
| VACUUM '(' vacuum_option_list ')'
@@ -9298,6 +9304,7 @@ vacuum_option_elem:
| VERBOSE { $$ = VACOPT_VERBOSE; }
| FREEZE { $$ = VACOPT_FREEZE; }
| FULL { $$ = VACOPT_FULL; }
+ | EMERGENCY { $$ = VACOPT_EMERGENCY; }
;
AnalyzeStmt:
@@ -9341,6 +9348,11 @@ opt_freeze: FREEZE { $$ = TRUE; }
| /*EMPTY*/ { $$ = FALSE; }
;
+opt_emergency:
+ EMERGENCY { $$ = TRUE; }
+ | /*EMPTY*/ { $$ = FALSE; }
+ ;
+
opt_name_list:
'(' name_list ')' { $$ = $2; }
| /*EMPTY*/ { $$ = NIL; }
@@ -13996,6 +14008,7 @@ type_func_name_keyword:
| CONCURRENTLY
| CROSS
| CURRENT_SCHEMA
+ | EMERGENCY
| FREEZE
| FULL
| ILIKE
diff --git a/src/backend/storage/freespace/freespace.c b/src/backend/storage/freespace/freespace.c
index 2631080..cd6af1e 100644
--- a/src/backend/storage/freespace/freespace.c
+++ b/src/backend/storage/freespace/freespace.c
@@ -106,7 +106,7 @@ static Size fsm_space_cat_to_avail(uint8 cat);
/* workhorse functions for various operations */
static int fsm_set_and_search(Relation rel, FSMAddress addr, uint16 slot,
- uint8 newValue, uint8 minValue);
+ uint8 newValue, uint8 minValue, bool extend);
static BlockNumber fsm_search(Relation rel, uint8 min_cat);
static uint8 fsm_vacuum_page(Relation rel, FSMAddress addr, bool *eof);
@@ -156,7 +156,8 @@ RecordAndGetPageWithFreeSpace(Relation rel, BlockNumber oldPage,
/* Get the location of the FSM byte representing the heap block */
addr = fsm_get_location(oldPage, &slot);
- search_slot = fsm_set_and_search(rel, addr, slot, old_cat, search_cat);
+ search_slot = fsm_set_and_search(rel, addr, slot, old_cat, search_cat,
+ true);
/*
* If fsm_set_and_search found a suitable new block, return that.
@@ -176,7 +177,8 @@ RecordAndGetPageWithFreeSpace(Relation rel, BlockNumber oldPage,
* FreeSpaceMapVacuum call, which updates the upper level pages.
*/
void
-RecordPageWithFreeSpace(Relation rel, BlockNumber heapBlk, Size spaceAvail)
+RecordPageWithFreeSpace(Relation rel, BlockNumber heapBlk, Size spaceAvail,
+ bool extend)
{
int new_cat = fsm_space_avail_to_cat(spaceAvail);
FSMAddress addr;
@@ -185,7 +187,7 @@ RecordPageWithFreeSpace(Relation rel, BlockNumber heapBlk, Size spaceAvail)
/* Get the location of the FSM byte representing the heap block */
addr = fsm_get_location(heapBlk, &slot);
- fsm_set_and_search(rel, addr, slot, new_cat, 0);
+ fsm_set_and_search(rel, addr, slot, new_cat, 0, extend);
}
/*
@@ -606,13 +608,16 @@ fsm_extend(Relation rel, BlockNumber fsm_nblocks)
*/
static int
fsm_set_and_search(Relation rel, FSMAddress addr, uint16 slot,
- uint8 newValue, uint8 minValue)
+ uint8 newValue, uint8 minValue, bool extend)
{
Buffer buf;
Page page;
int newslot = -1;
- buf = fsm_readbuf(rel, addr, true);
+ buf = fsm_readbuf(rel, addr, extend);
+ if(buf == InvalidBuffer)
+ return -1;
+
LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
page = BufferGetPage(buf);
@@ -702,7 +707,7 @@ fsm_search(Relation rel, uint8 min_cat)
* rarely, and will be fixed by the next vacuum.
*/
parent = fsm_get_parent(addr, &parentslot);
- fsm_set_and_search(rel, parent, parentslot, max_avail, 0);
+ fsm_set_and_search(rel, parent, parentslot, max_avail, 0, true);
/*
* If the upper pages are badly out of date, we might need to loop
diff --git a/src/backend/storage/freespace/indexfsm.c b/src/backend/storage/freespace/indexfsm.c
index e060a40..e25e0e1 100644
--- a/src/backend/storage/freespace/indexfsm.c
+++ b/src/backend/storage/freespace/indexfsm.c
@@ -51,7 +51,7 @@ GetFreeIndexPage(Relation rel)
void
RecordFreeIndexPage(Relation rel, BlockNumber freeBlock)
{
- RecordPageWithFreeSpace(rel, freeBlock, BLCKSZ - 1);
+ RecordPageWithFreeSpace(rel, freeBlock, BLCKSZ - 1, true);
}
@@ -61,7 +61,7 @@ RecordFreeIndexPage(Relation rel, BlockNumber freeBlock)
void
RecordUsedIndexPage(Relation rel, BlockNumber usedBlock)
{
- RecordPageWithFreeSpace(rel, usedBlock, 0);
+ RecordPageWithFreeSpace(rel, usedBlock, 0, true);
}
/*
diff --git a/src/include/access/visibilitymap.h b/src/include/access/visibilitymap.h
index b8dc54c..5cabc7c 100644
--- a/src/include/access/visibilitymap.h
+++ b/src/include/access/visibilitymap.h
@@ -36,7 +36,7 @@
extern void visibilitymap_clear(Relation rel, BlockNumber heapBlk,
Buffer vmbuf);
extern void visibilitymap_pin(Relation rel, BlockNumber heapBlk,
- Buffer *vmbuf);
+ Buffer *vmbuf, bool extend);
extern bool visibilitymap_pin_ok(BlockNumber heapBlk, Buffer vmbuf);
extern void visibilitymap_set(Relation rel, BlockNumber heapBlk, Buffer heapBuf,
XLogRecPtr recptr, Buffer vmBuf, TransactionId cutoff_xid,
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 2fd0629..5f0e52f 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -2795,7 +2795,8 @@ typedef enum VacuumOption
VACOPT_FREEZE = 1 << 3, /* FREEZE option */
VACOPT_FULL = 1 << 4, /* FULL (non-concurrent) vacuum */
VACOPT_NOWAIT = 1 << 5, /* don't wait to get lock (autovacuum only) */
- VACOPT_SKIPTOAST = 1 << 6 /* don't process the TOAST table, if any */
+ VACOPT_SKIPTOAST = 1 << 6, /* don't process the TOAST table, if any */
+ VACOPT_EMERGENCY = 1 << 7 /* EMERGENCY option */
} VacuumOption;
typedef struct VacuumStmt
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 6e1e820..0773028 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -137,6 +137,7 @@ PG_KEYWORD("double", DOUBLE_P, UNRESERVED_KEYWORD)
PG_KEYWORD("drop", DROP, UNRESERVED_KEYWORD)
PG_KEYWORD("each", EACH, UNRESERVED_KEYWORD)
PG_KEYWORD("else", ELSE, RESERVED_KEYWORD)
+PG_KEYWORD("emergency", EMERGENCY, TYPE_FUNC_NAME_KEYWORD)
PG_KEYWORD("enable", ENABLE_P, UNRESERVED_KEYWORD)
PG_KEYWORD("encoding", ENCODING, UNRESERVED_KEYWORD)
PG_KEYWORD("encrypted", ENCRYPTED, UNRESERVED_KEYWORD)
diff --git a/src/include/storage/freespace.h b/src/include/storage/freespace.h
index 19dcb8d..54d11f7 100644
--- a/src/include/storage/freespace.h
+++ b/src/include/storage/freespace.h
@@ -26,7 +26,7 @@ extern BlockNumber RecordAndGetPageWithFreeSpace(Relation rel,
Size oldSpaceAvail,
Size spaceNeeded);
extern void RecordPageWithFreeSpace(Relation rel, BlockNumber heapBlk,
- Size spaceAvail);
+ Size spaceAvail, bool extend);
extern void XLogRecordPageWithFreeSpace(RelFileNode rnode, BlockNumber heapBlk,
Size spaceAvail);
On Wed, Apr 6, 2016 at 3:32 AM, Asif Naeem <anaeem.it@gmail.com> wrote:
Oh, I see. I think it's probably not a good idea to skip truncating
those maps, but perhaps the option could be defined as making no new
entries, rather than ignoring them altogether, so that they never
grow. It seems that both of those are defined in such a way that if
page Y follows page X in the heap, the entries for page Y in the
relation fork will follow the one for page X. So truncating them
should be OK; it's just growing them that we need to avoid.Thank you Robert. PFA basic patch, it introduces EMERGENCY option to VACUUM
that forces to avoid extend any entries in the VM or FSM. It seems working
fine in simple test scenarios e.g.postgres=# create table test1 as (select generate_series(1,100000));
SELECT 100000
postgres=# vacuum EMERGENCY test1;
VACUUM
postgres=# select pg_relation_filepath('test1');
pg_relation_filepath
----------------------
base/13250/16384
(1 row)
[asif@centos66 inst_96]$ find . | grep base/13250/16384
./data/base/13250/16384
postgres=# vacuum test1;
VACUUM
[asif@centos66 inst_96]$ find . | grep base/13250/16384
./data/base/13250/16384
./data/base/13250/16384_fsm
./data/base/13250/16384_vmPlease do let me know if I missed something or more information is required.
Thanks.
This is too late for 9.6 at this point and certainly requires
discussion anyway, so please add it to the next CommitFest. But in
the meantime, here are a few quick comments:
You should only support EMERGENCY using the new-style, parenthesized
options syntax. That way, you won't need to make EMERGENCY a keyword.
We don't want to do that, and we especially don't want it to be
partially reserved, as you have done here.
Passing the EMERGENCY flag around in a global variable is probably not
a good idea; use parameter lists. That's what they are for. Also,
calling the variable that decides whether EMERGENCY was set
Extend_VM_FSM is confusing.
I see why you changed the calling convention for visibilitymap_pin()
and RecordPageWithFreeSpace(), but that's awfully invasive. I wonder
if there's a better way to do that, like maybe having vacuumlazy.c ask
the VM and FSM for their length in pages and then not trying to use
those functions for block numbers that are too large.
Don't forget to update the documentation.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 4/6/16 11:06 AM, Robert Haas wrote:
This is too late for 9.6 at this point and certainly requires
discussion anyway, so please add it to the next CommitFest.
If the goal here is to free up space via truncation when there's a real
emergency, perhaps there's some other steps that should be taken as
well. What immediately comes to mind is scanning the heap backwards and
stopping on the first page we can't truncate.
There might be some other non-critical things we could skip in emergency
mode, with an eye towards making it as fast as possible.
BTW, if someone really wanted to put some effort into this, it would be
possible to better handle filling up a single filesystem that has both
data and WAL by slowly shrinking the VM/FSM to make room in the WAL for
vacuum records. ISTM that's a much more common problem people run into
than filling up a separate tablespace.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Thank you Robert. Sure, I will add the updated patch on the next CommitFest
with all the suggested changes.
On Wed, Apr 6, 2016 at 9:06 PM, Robert Haas <robertmhaas@gmail.com> wrote:
Show quoted text
On Wed, Apr 6, 2016 at 3:32 AM, Asif Naeem <anaeem.it@gmail.com> wrote:
Oh, I see. I think it's probably not a good idea to skip truncating
those maps, but perhaps the option could be defined as making no new
entries, rather than ignoring them altogether, so that they never
grow. It seems that both of those are defined in such a way that if
page Y follows page X in the heap, the entries for page Y in the
relation fork will follow the one for page X. So truncating them
should be OK; it's just growing them that we need to avoid.Thank you Robert. PFA basic patch, it introduces EMERGENCY option to
VACUUM
that forces to avoid extend any entries in the VM or FSM. It seems
working
fine in simple test scenarios e.g.
postgres=# create table test1 as (select generate_series(1,100000));
SELECT 100000
postgres=# vacuum EMERGENCY test1;
VACUUM
postgres=# select pg_relation_filepath('test1');
pg_relation_filepath
----------------------
base/13250/16384
(1 row)
[asif@centos66 inst_96]$ find . | grep base/13250/16384
./data/base/13250/16384
postgres=# vacuum test1;
VACUUM
[asif@centos66 inst_96]$ find . | grep base/13250/16384
./data/base/13250/16384
./data/base/13250/16384_fsm
./data/base/13250/16384_vmPlease do let me know if I missed something or more information is
required.
Thanks.
This is too late for 9.6 at this point and certainly requires
discussion anyway, so please add it to the next CommitFest. But in
the meantime, here are a few quick comments:You should only support EMERGENCY using the new-style, parenthesized
options syntax. That way, you won't need to make EMERGENCY a keyword.
We don't want to do that, and we especially don't want it to be
partially reserved, as you have done here.Passing the EMERGENCY flag around in a global variable is probably not
a good idea; use parameter lists. That's what they are for. Also,
calling the variable that decides whether EMERGENCY was set
Extend_VM_FSM is confusing.I see why you changed the calling convention for visibilitymap_pin()
and RecordPageWithFreeSpace(), but that's awfully invasive. I wonder
if there's a better way to do that, like maybe having vacuumlazy.c ask
the VM and FSM for their length in pages and then not trying to use
those functions for block numbers that are too large.Don't forget to update the documentation.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Thu, Apr 7, 2016 at 2:15 AM, Jim Nasby <Jim.Nasby@bluetreble.com> wrote:
On 4/6/16 11:06 AM, Robert Haas wrote:
This is too late for 9.6 at this point and certainly requires
discussion anyway, so please add it to the next CommitFest.If the goal here is to free up space via truncation when there's a real
emergency, perhaps there's some other steps that should be taken as well.
What immediately comes to mind is scanning the heap backwards and stopping
on the first page we can't truncate.There might be some other non-critical things we could skip in emergency
mode, with an eye towards making it as fast as possible.BTW, if someone really wanted to put some effort into this, it would be
possible to better handle filling up a single filesystem that has both data
and WAL by slowly shrinking the VM/FSM to make room in the WAL for vacuum
records. ISTM that's a much more common problem people run into than
filling up a separate tablespace.
Thank you Jim. I will look into it and share my findings about it.
Show quoted text
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
On Wed, Apr 6, 2016 at 5:15 PM, Jim Nasby <Jim.Nasby@bluetreble.com> wrote:
On 4/6/16 11:06 AM, Robert Haas wrote:
This is too late for 9.6 at this point and certainly requires
discussion anyway, so please add it to the next CommitFest.If the goal here is to free up space via truncation when there's a real
emergency, perhaps there's some other steps that should be taken as well.
What immediately comes to mind is scanning the heap backwards and stopping
on the first page we can't truncate.There might be some other non-critical things we could skip in emergency
mode, with an eye towards making it as fast as possible.
I think this comes down to what you think the remit of a VACUUM
(EMERGENCY) option ought to be. I think it ought to do just enough to
make VACUUM succeed instead of failing, but you could argue it ought
to focus more specifically on freeing up space and skip everything
else it might normally do.
BTW, if someone really wanted to put some effort into this, it would be
possible to better handle filling up a single filesystem that has both data
and WAL by slowly shrinking the VM/FSM to make room in the WAL for vacuum
records. ISTM that's a much more common problem people run into than filling
up a separate tablespace.
Really? The VM and FSM are extremely tiny; that doesn't seem likely
to work out.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Wed, Apr 6, 2016 at 9:06 PM, Robert Haas <robertmhaas@gmail.com> wrote:
On Wed, Apr 6, 2016 at 3:32 AM, Asif Naeem <anaeem.it@gmail.com> wrote:
Oh, I see. I think it's probably not a good idea to skip truncating
those maps, but perhaps the option could be defined as making no new
entries, rather than ignoring them altogether, so that they never
grow. It seems that both of those are defined in such a way that if
page Y follows page X in the heap, the entries for page Y in the
relation fork will follow the one for page X. So truncating them
should be OK; it's just growing them that we need to avoid.Thank you Robert. PFA basic patch, it introduces EMERGENCY option to
VACUUM
that forces to avoid extend any entries in the VM or FSM. It seems
working
fine in simple test scenarios e.g.
postgres=# create table test1 as (select generate_series(1,100000));
SELECT 100000
postgres=# vacuum EMERGENCY test1;
VACUUM
postgres=# select pg_relation_filepath('test1');
pg_relation_filepath
----------------------
base/13250/16384
(1 row)
[asif@centos66 inst_96]$ find . | grep base/13250/16384
./data/base/13250/16384
postgres=# vacuum test1;
VACUUM
[asif@centos66 inst_96]$ find . | grep base/13250/16384
./data/base/13250/16384
./data/base/13250/16384_fsm
./data/base/13250/16384_vmPlease do let me know if I missed something or more information is
required.
Thanks.
This is too late for 9.6 at this point and certainly requires
discussion anyway, so please add it to the next CommitFest. But in
the meantime, here are a few quick comments:
Sure. Sorry for delay caused.
You should only support EMERGENCY using the new-style, parenthesized
options syntax. That way, you won't need to make EMERGENCY a keyword.
We don't want to do that, and we especially don't want it to be
partially reserved, as you have done here.
Sure. I have removed EMERGENCY keyword, it made code lot small now i.e.
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.yindex b9aeb31..89c4ee0 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -9298,6 +9298,20 @@ vacuum_option_elem: | VERBOSE { $$ = VACOPT_VERBOSE; } | FREEZE { $$ = VACOPT_FREEZE; } | FULL { $$ = VACOPT_FULL; } + | IDENT + { + /* + * We handle identifiers that aren't parser keywords with + * the following special-case codes. + */ + if (strcmp($1, "emergency") == 0) + $$ = VACOPT_EMERGENCY; + else + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("unrecognized vacuum option \"%s\"", $1), + parser_errposition(@1))); + } ;AnalyzeStmt:
Passing the EMERGENCY flag around in a global variable is probably not
a good idea; use parameter lists. That's what they are for. Also,
calling the variable that decides whether EMERGENCY was set
Extend_VM_FSM is confusing.
Sure. I adopted this approach by find other similar cases in the same
source code file i.e.
src/backend/commands/vacuumlazy.c
/* A few variables that don't seem worth passing around as parameters */
static int elevel = -1;
static TransactionId OldestXmin;
static TransactionId FreezeLimit;
static MultiXactId MultiXactCutoff;
static BufferAccessStrategy vac_strategy;
Should I modify code to use parameter lists for these variables too ?
I see why you changed the calling convention for visibilitymap_pin()
Show quoted text
and RecordPageWithFreeSpace(), but that's awfully invasive. I wonder
if there's a better way to do that, like maybe having vacuumlazy.c ask
the VM and FSM for their length in pages and then not trying to use
those functions for block numbers that are too large.Don't forget to update the documentation.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Thank you for useful suggestions. PFA patch, I have tried to cover all the
points mentioned.
Regards,
Muhammad Asif Naeem
On Wed, Apr 6, 2016 at 9:06 PM, Robert Haas <robertmhaas@gmail.com> wrote:
Show quoted text
On Wed, Apr 6, 2016 at 3:32 AM, Asif Naeem <anaeem.it@gmail.com> wrote:
Oh, I see. I think it's probably not a good idea to skip truncating
those maps, but perhaps the option could be defined as making no new
entries, rather than ignoring them altogether, so that they never
grow. It seems that both of those are defined in such a way that if
page Y follows page X in the heap, the entries for page Y in the
relation fork will follow the one for page X. So truncating them
should be OK; it's just growing them that we need to avoid.Thank you Robert. PFA basic patch, it introduces EMERGENCY option to
VACUUM
that forces to avoid extend any entries in the VM or FSM. It seems
working
fine in simple test scenarios e.g.
postgres=# create table test1 as (select generate_series(1,100000));
SELECT 100000
postgres=# vacuum EMERGENCY test1;
VACUUM
postgres=# select pg_relation_filepath('test1');
pg_relation_filepath
----------------------
base/13250/16384
(1 row)
[asif@centos66 inst_96]$ find . | grep base/13250/16384
./data/base/13250/16384
postgres=# vacuum test1;
VACUUM
[asif@centos66 inst_96]$ find . | grep base/13250/16384
./data/base/13250/16384
./data/base/13250/16384_fsm
./data/base/13250/16384_vmPlease do let me know if I missed something or more information is
required.
Thanks.
This is too late for 9.6 at this point and certainly requires
discussion anyway, so please add it to the next CommitFest. But in
the meantime, here are a few quick comments:You should only support EMERGENCY using the new-style, parenthesized
options syntax. That way, you won't need to make EMERGENCY a keyword.
We don't want to do that, and we especially don't want it to be
partially reserved, as you have done here.Passing the EMERGENCY flag around in a global variable is probably not
a good idea; use parameter lists. That's what they are for. Also,
calling the variable that decides whether EMERGENCY was set
Extend_VM_FSM is confusing.I see why you changed the calling convention for visibilitymap_pin()
and RecordPageWithFreeSpace(), but that's awfully invasive. I wonder
if there's a better way to do that, like maybe having vacuumlazy.c ask
the VM and FSM for their length in pages and then not trying to use
those functions for block numbers that are too large.Don't forget to update the documentation.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
VACUUM_EMERGENCY_Option_v2.patchapplication/octet-stream; name=VACUUM_EMERGENCY_Option_v2.patchDownload
diff --git a/doc/src/sgml/ref/vacuum.sgml b/doc/src/sgml/ref/vacuum.sgml
index dee1c5b..6b5f96a 100644
--- a/doc/src/sgml/ref/vacuum.sgml
+++ b/doc/src/sgml/ref/vacuum.sgml
@@ -21,7 +21,7 @@ PostgreSQL documentation
<refsynopsisdiv>
<synopsis>
-VACUUM [ ( { FULL | FREEZE | VERBOSE | ANALYZE | DISABLE_PAGE_SKIPPING } [, ...] ) ] [ <replaceable class="PARAMETER">table_name</replaceable> [ (<replaceable class="PARAMETER">column_name</replaceable> [, ...] ) ] ]
+VACUUM [ ( { FULL | FREEZE | VERBOSE | ANALYZE | DISABLE_PAGE_SKIPPING | EMERGENCY } [, ...] ) ] [ <replaceable class="PARAMETER">table_name</replaceable> [ (<replaceable class="PARAMETER">column_name</replaceable> [, ...] ) ] ]
VACUUM [ FULL ] [ FREEZE ] [ VERBOSE ] [ <replaceable class="PARAMETER">table_name</replaceable> ]
VACUUM [ FULL ] [ FREEZE ] [ VERBOSE ] ANALYZE [ <replaceable class="PARAMETER">table_name</replaceable> [ (<replaceable class="PARAMETER">column_name</replaceable> [, ...] ) ] ]
</synopsis>
@@ -149,6 +149,16 @@ VACUUM [ FULL ] [ FREEZE ] [ VERBOSE ] ANALYZE [ <replaceable class="PARAMETER">
</varlistentry>
<varlistentry>
+ <term><literal>EMERGENCY</literal></term>
+ <listitem>
+ <para>
+ Don't extend the visibility map or free space map, it can be helpful in
+ disk space full cases.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><replaceable class="PARAMETER">table_name</replaceable></term>
<listitem>
<para>
diff --git a/src/backend/access/heap/visibilitymap.c b/src/backend/access/heap/visibilitymap.c
index b472d31..06bf2fa 100644
--- a/src/backend/access/heap/visibilitymap.c
+++ b/src/backend/access/heap/visibilitymap.c
@@ -213,6 +213,19 @@ visibilitymap_clear(Relation rel, BlockNumber heapBlk, Buffer buf)
void
visibilitymap_pin(Relation rel, BlockNumber heapBlk, Buffer *buf)
{
+ visibilitymap_pin_ex(rel, heapBlk, buf, true);
+}
+
+/*
+ * visibilitymap_pin_ex - pin a map page for setting a bit
+ *
+ * Same as visibilitymap_pin, additionally extend parameter can be provided
+ * to allow extend page or not.
+ */
+void
+visibilitymap_pin_ex(Relation rel, BlockNumber heapBlk, Buffer *buf,
+ bool extend)
+{
BlockNumber mapBlock = HEAPBLK_TO_MAPBLOCK(heapBlk);
/* Reuse the old pinned buffer if possible */
@@ -223,7 +236,7 @@ visibilitymap_pin(Relation rel, BlockNumber heapBlk, Buffer *buf)
ReleaseBuffer(*buf);
}
- *buf = vm_readbuf(rel, mapBlock, true);
+ *buf = vm_readbuf(rel, mapBlock, extend);
}
/*
@@ -286,6 +299,10 @@ visibilitymap_set(Relation rel, BlockNumber heapBlk, Buffer heapBuf,
if (BufferIsValid(heapBuf) && BufferGetBlockNumber(heapBuf) != heapBlk)
elog(ERROR, "wrong heap buffer passed to visibilitymap_set");
+ /* In case of invalid buffer just return */
+ if(vmBuf == InvalidBuffer)
+ return;
+
/* Check that we have the right VM page pinned */
if (!BufferIsValid(vmBuf) || BufferGetBlockNumber(vmBuf) != mapBlock)
elog(ERROR, "wrong VM buffer passed to visibilitymap_set");
diff --git a/src/backend/commands/vacuumlazy.c b/src/backend/commands/vacuumlazy.c
index 32b6fdd..e86626f 100644
--- a/src/backend/commands/vacuumlazy.c
+++ b/src/backend/commands/vacuumlazy.c
@@ -140,7 +140,8 @@ static BufferAccessStrategy vac_strategy;
static void lazy_scan_heap(Relation onerel, int options,
LVRelStats *vacrelstats, Relation *Irel, int nindexes,
bool aggressive);
-static void lazy_vacuum_heap(Relation onerel, LVRelStats *vacrelstats);
+static void lazy_vacuum_heap(Relation onerel, LVRelStats *vacrelstats,
+ int options);
static bool lazy_check_needs_freeze(Buffer buf, bool *hastup);
static void lazy_vacuum_index(Relation indrel,
IndexBulkDeleteResult **stats,
@@ -150,7 +151,8 @@ static void lazy_cleanup_index(Relation indrel,
LVRelStats *vacrelstats);
static int lazy_vacuum_page(Relation onerel, BlockNumber blkno, Buffer buffer,
int tupindex, LVRelStats *vacrelstats, Buffer *vmbuffer);
-static bool should_attempt_truncation(LVRelStats *vacrelstats);
+static bool should_attempt_truncation(LVRelStats *vacrelstats,
+ int options);
static void lazy_truncate_heap(Relation onerel, LVRelStats *vacrelstats);
static BlockNumber count_nondeletable_pages(Relation onerel,
LVRelStats *vacrelstats);
@@ -273,7 +275,7 @@ lazy_vacuum_rel(Relation onerel, int options, VacuumParams *params,
/*
* Optionally truncate the relation.
*/
- if (should_attempt_truncation(vacrelstats))
+ if (should_attempt_truncation(vacrelstats, options))
lazy_truncate_heap(onerel, vacrelstats);
/* Report that we are now doing final cleanup */
@@ -593,7 +595,7 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
/* see note above about forcing scanning of last page */
#define FORCE_CHECK_PAGE() \
- (blkno == nblocks - 1 && should_attempt_truncation(vacrelstats))
+ (blkno == nblocks - 1 && should_attempt_truncation(vacrelstats, options))
pgstat_progress_update_param(PROGRESS_VACUUM_HEAP_BLKS_SCANNED, blkno);
@@ -722,7 +724,7 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
pgstat_progress_update_multi_param(2, hvp_index, hvp_val);
/* Remove tuples from heap */
- lazy_vacuum_heap(onerel, vacrelstats);
+ lazy_vacuum_heap(onerel, vacrelstats, options);
/*
* Forget the now-vacuumed tuples, and press on, but be careful
@@ -746,7 +748,8 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
* cycle of index vacuuming.
*
*/
- visibilitymap_pin(onerel, blkno, &vmbuffer);
+ visibilitymap_pin_ex(onerel, blkno, &vmbuffer,
+ !(options & VACOPT_EMERGENCY));
buf = ReadBufferExtended(onerel, MAIN_FORKNUM, blkno,
RBM_NORMAL, vac_strategy);
@@ -852,7 +855,8 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
MarkBufferDirty(buf);
UnlockReleaseBuffer(buf);
- RecordPageWithFreeSpace(onerel, blkno, freespace);
+ RecordPageWithFreeSpace_ex(onerel, blkno, freespace,
+ !(options & VACOPT_EMERGENCY));
continue;
}
@@ -891,7 +895,8 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
}
UnlockReleaseBuffer(buf);
- RecordPageWithFreeSpace(onerel, blkno, freespace);
+ RecordPageWithFreeSpace_ex(onerel, blkno, freespace,
+ !(options & VACOPT_EMERGENCY));
continue;
}
@@ -1236,7 +1241,8 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
* taken if there are no indexes.)
*/
if (vacrelstats->num_dead_tuples == prev_dead_count)
- RecordPageWithFreeSpace(onerel, blkno, freespace);
+ RecordPageWithFreeSpace_ex(onerel, blkno, freespace,
+ !(options & VACOPT_EMERGENCY));
}
/* report that everything is scanned and vacuumed */
@@ -1295,7 +1301,7 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
/* Remove tuples from heap */
pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_PHASE_VACUUM_HEAP);
- lazy_vacuum_heap(onerel, vacrelstats);
+ lazy_vacuum_heap(onerel, vacrelstats, options);
vacrelstats->num_index_scans++;
}
@@ -1358,7 +1364,7 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
* process index entry removal in batches as large as possible.
*/
static void
-lazy_vacuum_heap(Relation onerel, LVRelStats *vacrelstats)
+lazy_vacuum_heap(Relation onerel, LVRelStats *vacrelstats, int options)
{
int tupindex;
int npages;
@@ -1395,7 +1401,8 @@ lazy_vacuum_heap(Relation onerel, LVRelStats *vacrelstats)
freespace = PageGetHeapFreeSpace(page);
UnlockReleaseBuffer(buf);
- RecordPageWithFreeSpace(onerel, tblk, freespace);
+ RecordPageWithFreeSpace_ex(onerel, tblk, freespace,
+ !(options & VACOPT_EMERGENCY));
npages++;
}
@@ -1666,10 +1673,14 @@ lazy_cleanup_index(Relation indrel,
* careful to depend only on fields that lazy_scan_heap updates on-the-fly.
*/
static bool
-should_attempt_truncation(LVRelStats *vacrelstats)
+should_attempt_truncation(LVRelStats *vacrelstats, int options)
{
BlockNumber possibly_freeable;
+ /* In case of EMERGENCY option always attempt truncate */
+ if(options & VACOPT_EMERGENCY)
+ return true;
+
possibly_freeable = vacrelstats->rel_pages - vacrelstats->nonempty_pages;
if (possibly_freeable > 0 &&
(possibly_freeable >= REL_TRUNCATE_MINIMUM ||
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index edf4516..2f4343e 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -9374,6 +9374,8 @@ vacuum_option_elem:
{
if (strcmp($1, "disable_page_skipping") == 0)
$$ = VACOPT_DISABLE_PAGE_SKIPPING;
+ else if (strcmp($1, "emergency") == 0)
+ $$ = VACOPT_EMERGENCY;
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
diff --git a/src/backend/storage/freespace/freespace.c b/src/backend/storage/freespace/freespace.c
index bbd90c9..0f8a4e9 100644
--- a/src/backend/storage/freespace/freespace.c
+++ b/src/backend/storage/freespace/freespace.c
@@ -107,6 +107,8 @@ static Size fsm_space_cat_to_avail(uint8 cat);
/* workhorse functions for various operations */
static int fsm_set_and_search(Relation rel, FSMAddress addr, uint16 slot,
uint8 newValue, uint8 minValue);
+static int fsm_set_and_search_ex(Relation rel, FSMAddress addr, uint16 slot,
+ uint8 newValue, uint8 minValue, bool extend);
static BlockNumber fsm_search(Relation rel, uint8 min_cat);
static uint8 fsm_vacuum_page(Relation rel, FSMAddress addr, bool *eof);
static BlockNumber fsm_get_lastblckno(Relation rel, FSMAddress addr);
@@ -180,6 +182,19 @@ RecordAndGetPageWithFreeSpace(Relation rel, BlockNumber oldPage,
void
RecordPageWithFreeSpace(Relation rel, BlockNumber heapBlk, Size spaceAvail)
{
+ RecordPageWithFreeSpace_ex(rel, heapBlk, spaceAvail, true);
+}
+
+/*
+ * RecordPageWithFreeSpace_ex - update info about a page.
+ *
+ * Same as RecordPageWithFreeSpace, additionally extend parameter can be
+ * provided to allow extend page or not.
+ */
+void
+RecordPageWithFreeSpace_ex(Relation rel, BlockNumber heapBlk, Size spaceAvail,
+ bool extend)
+{
int new_cat = fsm_space_avail_to_cat(spaceAvail);
FSMAddress addr;
uint16 slot;
@@ -650,11 +665,27 @@ static int
fsm_set_and_search(Relation rel, FSMAddress addr, uint16 slot,
uint8 newValue, uint8 minValue)
{
+ return fsm_set_and_search_ex(rel, addr, slot, newValue, minValue, true);
+}
+
+/*
+ * Set value in given FSM page and slot.
+ *
+ * Same as fsm_set_and_search, additionally extend parameter can be provided
+ * to allow extend page or not.
+ */
+static int
+fsm_set_and_search_ex(Relation rel, FSMAddress addr, uint16 slot,
+ uint8 newValue, uint8 minValue, bool extend)
+{
Buffer buf;
Page page;
int newslot = -1;
- buf = fsm_readbuf(rel, addr, true);
+ buf = fsm_readbuf(rel, addr, extend);
+ if(buf == InvalidBuffer)
+ return -1;
+
LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
page = BufferGetPage(buf);
diff --git a/src/include/access/visibilitymap.h b/src/include/access/visibilitymap.h
index fca99ca..c1e5aa0 100644
--- a/src/include/access/visibilitymap.h
+++ b/src/include/access/visibilitymap.h
@@ -38,6 +38,8 @@ extern void visibilitymap_clear(Relation rel, BlockNumber heapBlk,
Buffer vmbuf);
extern void visibilitymap_pin(Relation rel, BlockNumber heapBlk,
Buffer *vmbuf);
+extern void visibilitymap_pin_ex(Relation rel, BlockNumber heapBlk,
+ Buffer *vmbuf, bool extend);
extern bool visibilitymap_pin_ok(BlockNumber heapBlk, Buffer vmbuf);
extern void visibilitymap_set(Relation rel, BlockNumber heapBlk, Buffer heapBuf,
XLogRecPtr recptr, Buffer vmBuf, TransactionId cutoff_xid,
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index d36d9c6..d48bbba 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -2823,7 +2823,8 @@ typedef enum VacuumOption
VACOPT_FULL = 1 << 4, /* FULL (non-concurrent) vacuum */
VACOPT_NOWAIT = 1 << 5, /* don't wait to get lock (autovacuum only) */
VACOPT_SKIPTOAST = 1 << 6, /* don't process the TOAST table, if any */
- VACOPT_DISABLE_PAGE_SKIPPING = 1 << 7 /* don't skip any pages */
+ VACOPT_DISABLE_PAGE_SKIPPING = 1 << 7, /* don't skip any pages */
+ VACOPT_EMERGENCY = 1 << 8 /* EMERGENCY option */
} VacuumOption;
typedef struct VacuumStmt
diff --git a/src/include/storage/freespace.h b/src/include/storage/freespace.h
index 77b3bc3..8bf88fa 100644
--- a/src/include/storage/freespace.h
+++ b/src/include/storage/freespace.h
@@ -27,6 +27,8 @@ extern BlockNumber RecordAndGetPageWithFreeSpace(Relation rel,
Size spaceNeeded);
extern void RecordPageWithFreeSpace(Relation rel, BlockNumber heapBlk,
Size spaceAvail);
+extern void RecordPageWithFreeSpace_ex(Relation rel, BlockNumber heapBlk,
+ Size spaceAvail, bool extend);
extern void XLogRecordPageWithFreeSpace(RelFileNode rnode, BlockNumber heapBlk,
Size spaceAvail);
diff --git a/src/test/regress/expected/vacuum.out b/src/test/regress/expected/vacuum.out
index 9b604be..c006117 100644
--- a/src/test/regress/expected/vacuum.out
+++ b/src/test/regress/expected/vacuum.out
@@ -80,5 +80,6 @@ CONTEXT: SQL function "do_analyze" statement 1
SQL function "wrap_do_analyze" statement 1
VACUUM FULL vactst;
VACUUM (DISABLE_PAGE_SKIPPING) vaccluster;
+VACUUM (EMERGENCY) vaccluster;
DROP TABLE vaccluster;
DROP TABLE vactst;
diff --git a/src/test/regress/sql/vacuum.sql b/src/test/regress/sql/vacuum.sql
index 7b819f6..a50dcab 100644
--- a/src/test/regress/sql/vacuum.sql
+++ b/src/test/regress/sql/vacuum.sql
@@ -61,6 +61,7 @@ VACUUM FULL vaccluster;
VACUUM FULL vactst;
VACUUM (DISABLE_PAGE_SKIPPING) vaccluster;
+VACUUM (EMERGENCY) vaccluster;
DROP TABLE vaccluster;
DROP TABLE vactst;
On Mon, Jun 20, 2016 at 7:28 AM, Asif Naeem <anaeem.it@gmail.com> wrote:
Thank you for useful suggestions. PFA patch, I have tried to cover all the
points mentioned.
Thanks for the new patch. I think that you have failed to address
this point from my previous review:
# I see why you changed the calling convention for visibilitymap_pin()
# and RecordPageWithFreeSpace(), but that's awfully invasive. I wonder
# if there's a better way to do that, like maybe having vacuumlazy.c ask
# the VM and FSM for their length in pages and then not trying to use
# those functions for block numbers that are too large.
The patch has gotten a lot smaller, and that's clearly good, but
introducing extended versions of those functions still seems like a
thing we should try to avoid. In particular, there's no way this hunk
is going to be acceptable:
@@ -286,6 +299,10 @@ visibilitymap_set(Relation rel, BlockNumber
heapBlk, Buffer heapBuf,
if (BufferIsValid(heapBuf) && BufferGetBlockNumber(heapBuf) != heapBlk)
elog(ERROR, "wrong heap buffer passed to visibilitymap_set");
+ /* In case of invalid buffer just return */
+ if(vmBuf == InvalidBuffer)
+ return;
+
/* Check that we have the right VM page pinned */
if (!BufferIsValid(vmBuf) || BufferGetBlockNumber(vmBuf) != mapBlock)
elog(ERROR, "wrong VM buffer passed to visibilitymap_set");You're going to have to find a different approach there.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Mon, Jun 20, 2016 at 9:28 PM, Asif Naeem <anaeem.it@gmail.com> wrote:
Thank you for useful suggestions. PFA patch, I have tried to cover all the
points mentioned.
Patch needs rebase, it is failing to apply on latest master.
And also there are some pending comment fix from Robert.
Regards,
Hari Babu
Fujitsu Australia
On Thu, Sep 8, 2016 at 2:46 AM, Haribabu Kommi <kommi.haribabu@gmail.com> wrote:
Patch needs rebase, it is failing to apply on latest master.
And also there are some pending comment fix from Robert.
It's been almost three weeks and this hasn't been updated, so I think
it's pretty clear that it should be marked "Returned with Feedback" at
this point. I'll go do that. Asif, if you update the patch, you can
resubmit for the next CommitFest. Please make sure that all review
comments already given are addressed in your next revision so that
reviewers don't waste time giving you the same comments again.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers