CLUSTER progress: wrong index_rebuild_count for tables with TOAST
Hackorum builds and tests every patch posted to the lists, not only commitfest submissions. This is Hackorum's own CI rather than the PostgreSQL project's, and it is still under testing - please report anything that looks wrong.
This thread has been committed, so CI has stopped here. Anything below is the last result it produced.
You can run a PostgreSQL built from this patch straight from Docker, with no checkout and no build:
docker run --rm -p 5432:5432 ghcr.io/hackorum-dev/postgres-patch:t248644psql -h localhost -U postgresBuilt from patchset v4 (message #4), September 15, 2026 at 12:34 PM.
Every patchset is also pushed to a branch of our PostgreSQL fork, so you can check out the same tree CI built. Without a PostgreSQL checkout:
git clone --branch t248644_4 https://github.com/hackorum-dev/postgres.gitIn a checkout you already have, add the fork once:
git remote add hackorum https://github.com/hackorum-dev/postgres.gitthen, for this patchset and every later one:
git fetch hackorum t248644_4 && git checkout t248644_4Patchset v4 (message #4) is on t248644_4
Hi,
When I run CLUSTER (or VACUUM FULL / REPACK) on a table that has a TOAST
table, pg_stat_progress_cluster shows a wrong index_rebuild_count. During
the heap scan the count is already equal to the number of indexes, but it
should be 0 until the indexes are rebuilt at the end.
A simple way to reproduce it. Run CLUSTER in one session, and query the
view from another session at the same time:
```
CREATE TABLE t (i int, x text);
INSERT INTO t SELECT g, repeat(md5(g::text), 1000)
FROM generate_series(1, 5) g;
CREATE INDEX ON t (i);
CLUSTER t USING t_i_idx;
-- phase "seq scanning heap", index_rebuild_count = 2 (should be 0)
```
The reason is that make_new_heap() creates the new TOAST table, and its
TOAST index, before the heap is scanned. Building that index reports
CREATE INDEX progress. CREATE INDEX and the cluster command use the same
progress field for different things, so the cluster command then shows a
CREATE INDEX value as its index_rebuild_count.
The fix is to not report progress for the TOAST index build, because it
is an internal index, not a user CREATE INDEX. The patch also adds an
isolation test that pauses CLUSTER at the start of the heap scan and
checks that index_rebuild_count is 0.
--
Adam
Adam Lee <adam8157@gmail.com> wrote:
The reason is that make_new_heap() creates the new TOAST table, and its
TOAST index, before the heap is scanned. Building that index reports
CREATE INDEX progress. CREATE INDEX and the cluster command use the same
progress field for different things, so the cluster command then shows a
CREATE INDEX value as its index_rebuild_count.
I'd prefer enhanced progress monitoring that I proposed earlier [1]/messages/by-id/30939.1777888333@localhost. The patch
needs more work, I'll try to submit it for the next development cycle.
[1]: /messages/by-id/30939.1777888333@localhost
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
On Fri, Jun 26, 2026 at 02:45:57PM +0800, Adam Lee wrote:
When I run CLUSTER (or VACUUM FULL / REPACK) on a table that has a TOAST
table, pg_stat_progress_cluster shows a wrong index_rebuild_count. During
the heap scan the count is already equal to the number of indexes, but it
should be 0 until the indexes are rebuilt at the end.
Hm. Isn't this a regression in v19 caused by commit 28d534e?
--
nathan
On Fri, Jun 26, 2026 at 3:46 PM Adam Lee <adam8157@gmail.com> wrote:
The fix is to not report progress for the TOAST index build, because it
is an internal index, not a user CREATE INDEX. The patch also adds an
isolation test that pauses CLUSTER at the start of the heap scan and
checks that index_rebuild_count is 0.
Thanks for the patch!
I think the approach in the patch, i.e., passing INDEX_CREATE_SUPPRESS_PROGRESS
to suppress progress reporting when creating TOAST indexes, looks good.
But, adding that test seems overkill to me. I'd prefer to simplify the patch
as in the attached 0001 patch. Thoughts?
While working on this patch, I also found a related but separate issue:
during REPACK (CONCURRENTLY), index_rebuild_count in
pg_stat_progress_repack and pg_stat_progress_cluster did not advance
as indexes were rebuilt. The attached 0002 patch fixes this issue.
Regards,
--
Fujii Masao
Attachments:
t248644_4v2-0001-Suppress-progress-reporting-when-creating-TOAST-i.patchapplication/octet-stream; name=v2-0001-Suppress-progress-reporting-when-creating-TOAST-i.patchDownload+6-2
v2-0002-Fix-index-rebuild-progress-reporting-for-REPACK-C.patchapplication/octet-stream; name=v2-0002-Fix-index-rebuild-progress-reporting-for-REPACK-C.patchDownload+2-1
On 2026-Sep-09, Fujii Masao wrote:
I think the approach in the patch, i.e., passing INDEX_CREATE_SUPPRESS_PROGRESS
to suppress progress reporting when creating TOAST indexes, looks good.
But, adding that test seems overkill to me. I'd prefer to simplify the patch
as in the attached 0001 patch. Thoughts?
I agree that the test is overkill -- after all, we don't test any of
progress reporting, and I'm not sure it's really a great approach to do
that by adding bespoke injection points.
Your 0001 looks good to me.
Maybe in a future release we can discuss a framework for making progress
updates visible in debug builds, so that they can be observed from a
new test framework.
While working on this patch, I also found a related but separate issue:
during REPACK (CONCURRENTLY), index_rebuild_count in
pg_stat_progress_repack and pg_stat_progress_cluster did not advance
as indexes were rebuilt. The attached 0002 patch fixes this issue.
Hmm, yeah, this patch looks good also.
I admit that the flow is a bit confusingly different in the concurrent
vs. non-concurrent cases: in the former, finish_heap_swap() is called
with reindex=false, so reindex_relation() is not called from there, and
instead we get these counter updates (with your patch) from
build_new_indexes(); in the concurrent patch, the counter updates come
from inside finish_heap_swap() instead.
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
"There's no problem so awful that you can't add some
guilt to it and make it even worse" (Calvin [& Hobbes])
On Sun, Sep 13, 2026 at 9:52 PM Álvaro Herrera <alvherre@kurilemu.de> wrote:
I agree that the test is overkill -- after all, we don't test any of
progress reporting, and I'm not sure it's really a great approach to do
that by adding bespoke injection points.Your 0001 looks good to me.
Thanks for the review! I've pushed the patch.
Maybe in a future release we can discuss a framework for making progress
updates visible in debug builds, so that they can be observed from a
new test framework.
+1
While working on this patch, I also found a related but separate issue:
during REPACK (CONCURRENTLY), index_rebuild_count in
pg_stat_progress_repack and pg_stat_progress_cluster did not advance
as indexes were rebuilt. The attached 0002 patch fixes this issue.Hmm, yeah, this patch looks good also.
I've pushed this patch as well. Thanks!
Regards,
--
Fujii Masao