CLUSTER vs. VACUUM FULL
PG 14.11 on RHEL8
Why is VACUUM FULL recommended for compressing a table, when CLUSTER does
the same thing (similarly doubling disk space), and apparently runs just as
fast?
My tests:
Table: CDSLBXW.public.log
Time 1 Time 2 Time 3
secs secs secs
VACUUM FULL 44.2 39.3 42.3
CLUSTER 41.7 38.9 41.3
CDSLBXW=# \d public.log
Table "public.log"
Column | Type | Collation | Nullable |
Default
-----------------+-----------------------------+-----------+----------+-------------------------------------
log_id | bigint | | not null |
nextval('log_log_id_seq'::regclass)
level | numeric(10,0) | | |
source | character varying(255) | | |
username | character varying(255) | | |
user_login_id | character varying(255) | | |
user_ip_address | character varying(255) | | |
computer | character varying(255) | | |
search_tag | character varying(4000) | | |
log_group_id | integer | | |
created_on | timestamp without time zone | | not null |
created_by | integer | | |
xml_detail | bytea | | |
Indexes:
"pk_log" PRIMARY KEY, btree (log_id)
"idx_log_attr_source" btree (source)
"idx_log_level" btree (level)
"idx_log_search_tag" btree (search_tag)
CDSLBXW=# SELECT COUNT(*) FROM public.log;
count
---------
6774664
(1 row)
CDSLBXW=# \timing
CDSLBXW=# vacuum (full) public.log;
VACUUM
Time: 44190.799 ms (00:44.191)
CDSLBXW=#
CDSLBXW=# vacuum (full) public.log;
VACUUM
Time: 39316.824 ms (00:39.317)
CDSLBXW=# vacuum (full) public.log;
VACUUM
Time: 42336.268 ms (00:42.336)
CDSLBXW=# CLUSTER public.log USING pk_log;
CLUSTER
Time: 41722.335 ms (00:41.722)
CDSLBXW=#
CDSLBXW=# CLUSTER public.log USING pk_log;
CLUSTER
Time: 38915.128 ms (00:38.915)
CDSLBXW=#
CDSLBXW=# CLUSTER public.log USING pk_log;
CLUSTER
Time: 41342.651 ms (00:41.343)
Ron Johnson <ronljohnsonjr@gmail.com> writes:
Why is VACUUM FULL recommended for compressing a table, when CLUSTER does
the same thing (similarly doubling disk space), and apparently runs just as
fast?
CLUSTER makes the additional effort to sort the data per the ordering
of the specified index. I'm surprised that's not noticeable in your
test case.
regards, tom lane
On Sun, Apr 21, 2024 at 6:45 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Ron Johnson <ronljohnsonjr@gmail.com> writes:
Why is VACUUM FULL recommended for compressing a table, when CLUSTER does
the same thing (similarly doubling disk space), and apparently runs justas
fast?
CLUSTER makes the additional effort to sort the data per the ordering
of the specified index. I'm surprised that's not noticeable in your
test case.
It's in a freshly restored database. However, regular deletions of old
records, and normal vacuums would have led me to expect that the "fresh"
public.log would have been in relatively random order.
On Sun, Apr 21, 2024 at 6:45 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Ron Johnson <ronljohnsonjr@gmail.com> writes:
Why is VACUUM FULL recommended for compressing a table, when CLUSTER does
the same thing (similarly doubling disk space), and apparently runs justas
fast?
CLUSTER makes the additional effort to sort the data per the ordering
of the specified index. I'm surprised that's not noticeable in your
test case.
Clustering on a completely different index was also 44 seconds.
On Mon, 22 Apr 2024 at 12:16, Ron Johnson <ronljohnsonjr@gmail.com> wrote:
On Sun, Apr 21, 2024 at 6:45 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Ron Johnson <ronljohnsonjr@gmail.com> writes:
Why is VACUUM FULL recommended for compressing a table, when CLUSTER does
the same thing (similarly doubling disk space), and apparently runs just as
fast?CLUSTER makes the additional effort to sort the data per the ordering
of the specified index. I'm surprised that's not noticeable in your
test case.Clustering on a completely different index was also 44 seconds.
Both VACUUM FULL and CLUSTER go through a very similar code path. Both
use cluster_rel(). VACUUM FULL just won't make use of an existing
index to provide presorted input or perform a sort, whereas CLUSTER
will attempt to choose the cheapest out of these two to get sorted
results.
If the timing for each is similar, it just means that using an index
scan or sorting isn't very expensive compared to the other work that's
being done. Both CLUSTER and VACUUM FULL require reading every heap
page and writing out new pages into a new heap and maintaining all
indexes on the new heap. That's quite an effort.
To satisfy your curiosity, you could always run some EXPLAIN ANALYZE
SELECT queries to measure how much time was spent sorting the entire
table. You'd have to set work_mem to the value of
maintenance_work_mem.
David
On Sun, Apr 21, 2024 at 9:35 PM David Rowley <dgrowleyml@gmail.com> wrote:
On Mon, 22 Apr 2024 at 12:16, Ron Johnson <ronljohnsonjr@gmail.com> wrote:
On Sun, Apr 21, 2024 at 6:45 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Ron Johnson <ronljohnsonjr@gmail.com> writes:
Why is VACUUM FULL recommended for compressing a table, when CLUSTER
does
the same thing (similarly doubling disk space), and apparently runs
just as
fast?
CLUSTER makes the additional effort to sort the data per the ordering
of the specified index. I'm surprised that's not noticeable in your
test case.Clustering on a completely different index was also 44 seconds.
Both VACUUM FULL and CLUSTER go through a very similar code path. Both
use cluster_rel(). VACUUM FULL just won't make use of an existing
index to provide presorted input or perform a sort, whereas CLUSTER
will attempt to choose the cheapest out of these two to get sorted
results.If the timing for each is similar, it just means that using an index
scan or sorting isn't very expensive compared to the other work that's
being done. Both CLUSTER and VACUUM FULL require reading every heap
page and writing out new pages into a new heap and maintaining all
indexes on the new heap. That's quite an effort.
My original CLUSTER command didn't have to change the order of the data
very much, thus, the sort didn't have to do much work.
CLUSTER on a different index was indeed much slower than VACUUM FULL.
Em dom., 21 de abr. de 2024 às 22:35, David Rowley <dgrowleyml@gmail.com>
escreveu:
Both VACUUM FULL and CLUSTER go through a very similar code path. Both
use cluster_rel(). VACUUM FULL just won't make use of an existing
index to provide presorted input or perform a sort, whereas CLUSTER
will attempt to choose the cheapest out of these two to get sorted
results.
But wouldn't it be good that VACUUM FULL uses that index defined by
Cluster, if it exists ? Maybe an additional option for VACUUM FULL ?
Because if I periodically reorganize all tables I have to run CLUSTER once,
which will run on clustered tables, and VACUUM FULL on every table that is
not clustered, because if I run VACUUM FULL for entire database it'll just
ignore cluster index defined for every table. So, INDISCLUSTERED is used
when running CLUSTER but is ignored when running VACUUM FULL.
regards
Marcos
Marcos Pegoraro <marcos@f10.com.br> writes:
But wouldn't it be good that VACUUM FULL uses that index defined by
Cluster, if it exists ?
No ... what would be the difference then?
regards, tom lane
On Mon, Apr 22, 2024 at 10:25 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Marcos Pegoraro <marcos@f10.com.br> writes:
But wouldn't it be good that VACUUM FULL uses that index defined by
Cluster, if it exists ?No ... what would be the difference then?
What the VACUUM docs "should" do, it seems, is suggest CLUSTER on the PK,
if the PK is a sequence (whether that be an actual sequence, or a timestamp
or something else that grows monotonically).
That's because the data is already roughly in PK order.
On 4/22/24 08:37, Ron Johnson wrote:
On Mon, Apr 22, 2024 at 10:25 AM Tom Lane <tgl@sss.pgh.pa.us
<mailto:tgl@sss.pgh.pa.us>> wrote:Marcos Pegoraro <marcos@f10.com.br <mailto:marcos@f10.com.br>> writes:
But wouldn't it be good that VACUUM FULL uses that index defined by
Cluster, if it exists ?No ... what would be the difference then?
What the VACUUM docs "should" do, it seems, is suggest CLUSTER on the
PK, if the PK is a sequence (whether that be an actual sequence, or a
timestamp or something else that grows monotonically).
Why?
That would, per David Rowley's comments, impose a sort cost on top of
the cost of hitting every heap page and rewriting it. You end up with
sorted table granted, until such time as you start making changes to it.
If you are to the point of running VACUUM FULL that indicates to me the
table has seen a heavy load of changes that you want to clean out. Given
the temporary nature of the effects of a CLUSTER under a change load I
don't see why it would be the way to go to clean up a changing table.
That's because the data is already roughly in PK order.
--
Adrian Klaver
adrian.klaver@aklaver.com
On Mon, Apr 22, 2024, 08:37 Ron Johnson <ronljohnsonjr@gmail.com> wrote:
On Mon, Apr 22, 2024 at 10:25 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Marcos Pegoraro <marcos@f10.com.br> writes:
But wouldn't it be good that VACUUM FULL uses that index defined by
Cluster, if it exists ?No ... what would be the difference then?
What the VACUUM docs "should" do, it seems, is suggest CLUSTER on the PK,
if the PK is a sequence (whether that be an actual sequence, or a timestamp
or something else that grows monotonically).That's because the data is already roughly in PK order.
If things are bad enough to require a vacuum full that doesn't seem like a
good assumption. Any insert-only table or one with a reduced fill-factor
maybe.
David J
Em seg., 22 de abr. de 2024 às 11:25, Tom Lane <tgl@sss.pgh.pa.us> escreveu:
No ... what would be the difference then
Well, I think if a cluster index was defined sometime in a table, it should
be respected for next commands, including VACUUM FULL.
If I want to come back to PK or any other index I would use CLUSTER ...
USING PK_INDEX.
regards
Marcos
On Mon, Apr 22, 2024 at 12:29 PM David G. Johnston <
david.g.johnston@gmail.com> wrote:
On Mon, Apr 22, 2024, 08:37 Ron Johnson <ronljohnsonjr@gmail.com> wrote:
On Mon, Apr 22, 2024 at 10:25 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Marcos Pegoraro <marcos@f10.com.br> writes:
But wouldn't it be good that VACUUM FULL uses that index defined by
Cluster, if it exists ?No ... what would be the difference then?
What the VACUUM docs "should" do, it seems, is suggest CLUSTER on the PK,
if the PK is a sequence (whether that be an actual sequence, or a timestamp
or something else that grows monotonically).That's because the data is already roughly in PK order.
If things are bad enough to require a vacuum full that doesn't seem like a
good assumption.
Sure it does.
For example, I just deleted the oldest half of the records in 30 tables.
Tables who's CREATED_ON timestamp value strongly correlates to the
synthetic PK sequence values.
Thus, the remaining records were still mostly in PK order. CLUSTERs on the
PK values would have taken just about as much time as the VACUUM FULL
statements which I *did* run.
On 4/22/24 11:45 AM, Ron Johnson wrote:
On Mon, Apr 22, 2024 at 12:29 PM David G. Johnston
<david.g.johnston@gmail.com <mailto:david.g.johnston@gmail.com>> wrote:On Mon, Apr 22, 2024, 08:37 Ron Johnson <ronljohnsonjr@gmail.com
<mailto:ronljohnsonjr@gmail.com>> wrote:On Mon, Apr 22, 2024 at 10:25 AM Tom Lane <tgl@sss.pgh.pa.us
<mailto:tgl@sss.pgh.pa.us>> wrote:Marcos Pegoraro <marcos@f10.com.br
<mailto:marcos@f10.com.br>> writes:But wouldn't it be good that VACUUM FULL uses that index
defined by
Cluster, if it exists ?
No ... what would be the difference then?
What the VACUUM docs "should" do, it seems, is suggest CLUSTER
on the PK, if the PK is a sequence (whether that be an actual
sequence, or a timestamp or something else that grows
monotonically).That's because the data is already roughly in PK order.
If things are bad enough to require a vacuum full that doesn't seem
like a good assumption.Sure it does.
For example, I just deleted the oldest half of the records in 30
tables. Tables who's CREATED_ON timestamp value strongly correlates to
the synthetic PK sequence values.Thus, the remaining records were still mostly in PK order. CLUSTERs on
the PK values would have taken just about as much time as the VACUUM
FULL statements which I /did/ run.
1) If they are already in enough of a PK order that the CLUSTER time vs
VACUUM FULL time would not be material as there is not much or any
sorting to do then what does the CLUSTER gain you? Unless this table
then became read only whatever small gain arose from the CLUSTER would
fade away as UPDATEs and DELETEs where done.
2) What evidence is there that the records where still in PK order just
because you deleted based on CREATED_ON? I understand the correlation
between CREATED_ON and the PK just not sure why that would necessarily
translate to an on disk order by PK?
--
Adrian Klaver
adrian.klaver@aklaver.com
On Mon, Apr 22, 2024 at 3:14 PM Adrian Klaver <adrian.klaver@aklaver.com>
wrote:
On 4/22/24 11:45 AM, Ron Johnson wrote:
On Mon, Apr 22, 2024 at 12:29 PM David G. Johnston
<david.g.johnston@gmail.com <mailto:david.g.johnston@gmail.com>> wrote:On Mon, Apr 22, 2024, 08:37 Ron Johnson <ronljohnsonjr@gmail.com
<mailto:ronljohnsonjr@gmail.com>> wrote:On Mon, Apr 22, 2024 at 10:25 AM Tom Lane <tgl@sss.pgh.pa.us
<mailto:tgl@sss.pgh.pa.us>> wrote:Marcos Pegoraro <marcos@f10.com.br
<mailto:marcos@f10.com.br>> writes:But wouldn't it be good that VACUUM FULL uses that index
defined by
Cluster, if it exists ?
No ... what would be the difference then?
What the VACUUM docs "should" do, it seems, is suggest CLUSTER
on the PK, if the PK is a sequence (whether that be an actual
sequence, or a timestamp or something else that grows
monotonically).That's because the data is already roughly in PK order.
If things are bad enough to require a vacuum full that doesn't seem
like a good assumption.Sure it does.
For example, I just deleted the oldest half of the records in 30
tables. Tables who's CREATED_ON timestamp value strongly correlates to
the synthetic PK sequence values.Thus, the remaining records were still mostly in PK order. CLUSTERs on
the PK values would have taken just about as much time as the VACUUM
FULL statements which I /did/ run.1) If they are already in enough of a PK order that the CLUSTER time vs
VACUUM FULL time would not be material as there is not much or any
sorting to do then what does the CLUSTER gain you?
Not much. Now they're just "slightly more ordered" instead of "slightly
less ordered" for little if any extra effort.
2) What evidence is there that the records where still in PK order just
because you deleted based on CREATED_ON? I understand the correlation
between CREATED_ON and the PK just not sure why that would necessarily
translate to an on disk order by PK?
1. Records are appended to tables in INSERT order, and INSERT order is
highly correlated to synthetic PK, by the nature of sequences.
2. My original email showed that CLUSTER took just as long as VACUUM FULL.
That means not many records had to be sorted, because... the on-disk order
was strongly correlated to PK and CREATED_ON.
Will that happen *every time* in *every circumstance* in *every database*?
No, and I never said it would. But it does in *my *database in *this *
application.
On 4/22/24 12:51, Ron Johnson wrote:
On Mon, Apr 22, 2024 at 3:14 PM Adrian Klaver <adrian.klaver@aklaver.com
<mailto:adrian.klaver@aklaver.com>> wrote:
1) If they are already in enough of a PK order that the CLUSTER time vs
VACUUM FULL time would not be material as there is not much or any
sorting to do then what does the CLUSTER gain you?Not much. Now they're just "slightly more ordered" instead of "slightly
less ordered" for little if any extra effort.2) What evidence is there that the records where still in PK order just
because you deleted based on CREATED_ON? I understand the correlation
between CREATED_ON and the PK just not sure why that would necessarily
translate to an on disk order by PK?1. Records are appended to tables in INSERT order, and INSERT order is
highly correlated to synthetic PK, by the nature of sequences.
Not something I would count on, see:
https://www.postgresql.org/docs/current/sql-createsequence.html
Notes
for how that may not always be the case.
Also any UPDATE or DELETE is going to change that. There is no guarantee
of order for the data in the table. If there where you would not need to
run CLUSTER.
2. My original email showed that CLUSTER took just as long as VACUUM
FULL. That means not many records had to be sorted, because... the
on-disk order was strongly correlated to PK and CREATED_ON. >
Will that happen *every time* in *every circumstance* in *every
database*? No, and I never said it would. But it does in *my *database
in *this *application.
Which gets us back to your comment upstream:
"What the VACUUM docs "should" do, it seems, is suggest CLUSTER on the
PK, if the PK is a sequence (whether that be an actual sequence, or a
timestamp or something else that grows monotonically)."
This is a case specific to you and this particular circumstance, not a
general rule for VACUUM FULL. If for no other reason then it might make
more sense for the application that the CLUSTER be done on some other
index then the PK.
--
Adrian Klaver
adrian.klaver@aklaver.com
On Mon, Apr 22, 2024 at 4:21 PM Adrian Klaver <adrian.klaver@aklaver.com>
wrote:
[snip]
Which gets us back to your comment upstream:
"What the VACUUM docs "should" do, it seems, is suggest CLUSTER on the
PK, if the PK is a sequence (whether that be an actual sequence, or a
timestamp or something else that grows monotonically)."This is a case specific to you and this particular circumstance, not a
general rule for VACUUM FULL. If for no other reason then it might make
more sense for the application that the CLUSTER be done on some other
index then the PK.
On Stack Exchange, I've got a question on how to determine when to run
CLUSTER. It ties in strongly with this thread..
On 4/22/24 13:59, Ron Johnson wrote:
On Mon, Apr 22, 2024 at 4:21 PM Adrian Klaver <adrian.klaver@aklaver.com
<mailto:adrian.klaver@aklaver.com>> wrote:
[snip]Which gets us back to your comment upstream:
"What the VACUUM docs "should" do, it seems, is suggest CLUSTER on the
PK, if the PK is a sequence (whether that be an actual sequence, or a
timestamp or something else that grows monotonically)."This is a case specific to you and this particular circumstance, not a
general rule for VACUUM FULL. If for no other reason then it might make
more sense for the application that the CLUSTER be done on some other
index then the PK.On Stack Exchange, I've got a question on how to determine when to run
CLUSTER. It ties in strongly with this thread..
And the link is?
--
Adrian Klaver
adrian.klaver@aklaver.com
On Mon, Apr 22, 2024 at 5:03 PM Adrian Klaver <adrian.klaver@aklaver.com>
wrote:
On 4/22/24 13:59, Ron Johnson wrote:
On Mon, Apr 22, 2024 at 4:21 PM Adrian Klaver <adrian.klaver@aklaver.com
<mailto:adrian.klaver@aklaver.com>> wrote:
[snip]Which gets us back to your comment upstream:
"What the VACUUM docs "should" do, it seems, is suggest CLUSTER on
the
PK, if the PK is a sequence (whether that be an actual sequence, or a
timestamp or something else that grows monotonically)."This is a case specific to you and this particular circumstance, not
a
general rule for VACUUM FULL. If for no other reason then it might
make
more sense for the application that the CLUSTER be done on some other
index then the PK.On Stack Exchange, I've got a question on how to determine when to run
CLUSTER. It ties in strongly with this thread..And the link is?
Sorry. Got distracted by the answer.
https://dba.stackexchange.com/questions/338870/when-to-rerun-cluster
On 4/22/24 14:35, Ron Johnson wrote:
On Stack Exchange, I've got a question on how to determine when
to run
CLUSTER. It ties in strongly with this thread..
And the link is?
It should have been the initial question of this thread and it explains
what you are really after.
Sorry. Got distracted by the answer.
https://dba.stackexchange.com/questions/338870/when-to-rerun-cluster
<https://dba.stackexchange.com/questions/338870/when-to-rerun-cluster>
--
Adrian Klaver
adrian.klaver@aklaver.com