Fix stale relcache fields for TOAST indexes during REINDEX CONCURRENTLY

Started by Bharath Rupireddy26 days ago6 messageshackers
Jump to latest
#1Bharath Rupireddy
bharath.rupireddyforpostgres@gmail.com

Hi,

I received an off-list report of an "ERROR: missing chunk number x for
toast value xxxx in pg_toast_xxxx" during REINDEX CONCURRENTLY.

After the concurrent index rebuild marked the new index ready for
inserts, the backend that kept the TOAST indexes (both new and old)
open for inserts doesn't reload the new TOAST index entry as part of
the invalidation message processing. This leaves the new index not
ready for inserts in that backend's relcache, making the chunks miss
in the new index. This later shows up as the [1]ERROR: missing chunk number 0 for toast value 20810 in pg_toast_20801 when the value is
read.

Please find the attached patch that allows the TOAST indexes also to
re-read the pg_index row and reload the relcache fields in
RelationReloadIndexInfo(). Note that this change also reloads the
relcache fields for the TOAST indexes of the logical replication
conflict log tables, which seems fine IMO.

I think this needs to be back-patched to all the supported branches.
The check in RelationReloadIndexInfo() was introduced in 8ec943856a4,
and the issue was uncovered by 5dc92b844e68, which added REINDEX
CONCURRENTLY.

Thoughts?

[1]: ERROR: missing chunk number 0 for toast value 20810 in pg_toast_20801

--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com

Attachments:

t254413_1
v1-0001-Fix-stale-relcache-fields-for-TOAST-indexes-durin.patchapplication/octet-stream; name=v1-0001-Fix-stale-relcache-fields-for-TOAST-indexes-durin.patchDownload+175-3
#2Chao Li
li.evan.chao@gmail.com
In reply to: Bharath Rupireddy (#1)
Re: Fix stale relcache fields for TOAST indexes during REINDEX CONCURRENTLY

On Oct 28, 2026, at 02:44, Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> wrote:

Hi,

I received an off-list report of an "ERROR: missing chunk number x for
toast value xxxx in pg_toast_xxxx" during REINDEX CONCURRENTLY.

After the concurrent index rebuild marked the new index ready for
inserts, the backend that kept the TOAST indexes (both new and old)
open for inserts doesn't reload the new TOAST index entry as part of
the invalidation message processing. This leaves the new index not
ready for inserts in that backend's relcache, making the chunks miss
in the new index. This later shows up as the [1] when the value is
read.

Please find the attached patch that allows the TOAST indexes also to
re-read the pg_index row and reload the relcache fields in
RelationReloadIndexInfo(). Note that this change also reloads the
relcache fields for the TOAST indexes of the logical replication
conflict log tables, which seems fine IMO.

I think this needs to be back-patched to all the supported branches.
The check in RelationReloadIndexInfo() was introduced in 8ec943856a4,
and the issue was uncovered by 5dc92b844e68, which added REINDEX
CONCURRENTLY.

Thoughts?

[1] ERROR: missing chunk number 0 for toast value 20810 in pg_toast_20801

--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com
<v1-0001-Fix-stale-relcache-fields-for-TOAST-indexes-durin.patch>

The fix looks good to me. IsSystemRelation() returns true for system catalogs and all TOAST tables and indexes, including those belonging to user tables, while IsCatalogRelation() returns true only for system catalogs and their associated relations, excluding user-table TOAST indexes. So, with this fix, the TOAST index's indisready field gets refreshed.

I see one issue with the test. The test evaluates:
```
SELECT id, length(val) FROM reind_toast ORDER BY id;
```

text_lengh() has an optimization:
```
static int32
text_length(Datum str)
{
/* fastpath when max encoding length is one */
if (pg_database_encoding_max_length() == 1)
return (toast_raw_datum_size(str) - VARHDRSZ);
```

For single-byte encodings, length(val) won't actually fetch the TOAST values, so the test won't detect missing TOAST chunks. I confirmed this by running:
```
% make -C src/test/modules/injection_points check ISOLATION=reindex_concurrently_toast LANG=C ENCODING=LATIN1
```

The test still passed after I reverted the fix.

To force fetching the TOAST data, instead of evaluating the length of val, I think we can evaluate its first char. I made the following change locally:
```
diff --git a/src/test/modules/injection_points/expected/reindex_concurrently_toast.out b/src/test/modules/injection_points/expected/reindex_concurrently_toast.out
index 4ce089806b0..3892b8df413 100644
--- a/src/test/modules/injection_points/expected/reindex_concurrently_toast.out
+++ b/src/test/modules/injection_points/expected/reindex_concurrently_toast.out
@@ -74,11 +74,11 @@ injection_points_wakeup
 step reindex: <... completed>
 step reindexed:
-step read: SELECT id, length(val) FROM reind_toast ORDER BY id;
-id|length
---+------
- 1| 10000
- 2| 10000
- 3| 10000
+step read: SELECT id, left(val, 1) FROM reind_toast ORDER BY id;
+id|left
+--+----
+ 1|x
+ 2|y
+ 3|z
 (3 rows)
diff --git a/src/test/modules/injection_points/specs/reindex_concurrently_toast.spec b/src/test/modules/injection_points/specs/reindex_concurrently_toast.spec
index ba34a81adef..0ce146be848 100644
--- a/src/test/modules/injection_points/specs/reindex_concurrently_toast.spec
+++ b/src/test/modules/injection_points/specs/reindex_concurrently_toast.spec
@@ -52,7 +52,7 @@ step ins1     { INSERT INTO reind_toast VALUES (2, repeat('y', 10000)); }
 step inserted  { }
 # Written after the validation, so only a maintained index has it.
 step ins2      { INSERT INTO reind_toast VALUES (3, repeat('z', 10000)); }
-step read      { SELECT id, length(val) FROM reind_toast ORDER BY id; }
+step read      { SELECT id, left(val, 1) FROM reind_toast ORDER BY id; }

# Holds the rebuild back until the writer has the TOAST indexes open.
session s3
```

With this change, the test fails without the fix and passes with it.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

#3Michael Paquier
michael@paquier.xyz
In reply to: Chao Li (#2)
Re: Fix stale relcache fields for TOAST indexes during REINDEX CONCURRENTLY

On Wed, Oct 28, 2026 at 08:14:17AM +0800, Chao Li wrote:

On Oct 28, 2026, at 02:44, Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> wrote:

After the concurrent index rebuild marked the new index ready for
inserts, the backend that kept the TOAST indexes (both new and old)
open for inserts doesn't reload the new TOAST index entry as part of
the invalidation message processing. This leaves the new index not
ready for inserts in that backend's relcache, making the chunks miss
in the new index. This later shows up as the [1] when the value is
read.

This strikes me as a good find. The TOAST values are in the table,
and we miss them in the index. A rebuild of the TOAST index would
resolve the issue, but requiring again a REINDEX after a CONCURRENTLY
one defeats the purpose of the latter if we require the former because
of some silent corruption.

Please find the attached patch that allows the TOAST indexes also to
re-read the pg_index row and reload the relcache fields in
RelationReloadIndexInfo(). Note that this change also reloads the
relcache fields for the TOAST indexes of the logical replication
conflict log tables, which seems fine IMO.

I think this needs to be back-patched to all the supported branches.
The check in RelationReloadIndexInfo() was introduced in 8ec943856a4,
and the issue was uncovered by 5dc92b844e68, which added REINDEX
CONCURRENTLY.

Back when I worked on RIC, I've been heavily relying on the assumption
that we can safely rely on the facilities of CIC for it, so the blame
is on 5dc92b844e68, not really 8ec943856a4. Plop, my fault by
default.

It's good that you have been able to dig out a test, re-using some of
the existing injpoints in the tree.

For single-byte encodings, length(val) won't actually fetch the
TOAST values, so the test won't detect missing TOAST chunks.

Ah. Good point here. Better to tweak that and remove the dependency
on the length. Will look at all that head down..
--
Michael

#4Michael Paquier
michael@paquier.xyz
In reply to: Michael Paquier (#3)
Re: Fix stale relcache fields for TOAST indexes during REINDEX CONCURRENTLY

On Wed, Oct 28, 2026 at 10:43:26AM +0900, Michael Paquier wrote:

Ah. Good point here. Better to tweak that and remove the dependency
on the length. Will look at all that head down..

+    CREATE FUNCTION wait_for_point(text) RETURNS void LANGUAGE plpgsql AS $$
+    BEGIN
+        LOOP
+            PERFORM 1 FROM pg_stat_activity WHERE wait_event = $1;
+            EXIT WHEN FOUND;
+            PERFORM pg_sleep(.1);
+        END LOOP;
+    END $$;

Hardcoded sleeps are my enemies because they can lead to longer tests
on fast machines in some cases. This also leads to a couple of
complications related to the two extra injection points you have been
using for the builds.

I have tested an extra approach where I make the REINDEX wait for
older snapshots after validating the new index, pointing at the fact
that these two extra injpoint wakeups should not be required, as much
as wait_for_point(). I have also confirmed with the same tweaks back
when I have debugged basic.spec, checking that this new test case was
safe. For example, using a hardcoded sleep inside injection_wait()
after the wait finishes to delay the shutdown of the session running
the wait, the test is longer, still stable..

So I am feeling pretty confident that I got that right. The buildfarm
will judge, so we'll see. The CI was stable.

Anyway, applied the patch down to v14, with the test down to v17.
--
Michael

#5Bharath Rupireddy
bharath.rupireddyforpostgres@gmail.com
In reply to: Michael Paquier (#4)
Re: Fix stale relcache fields for TOAST indexes during REINDEX CONCURRENTLY

Hi,

On Tue, Oct 27, 2026 at 10:57 PM Michael Paquier <michael@paquier.xyz> wrote:

On Wed, Oct 28, 2026 at 10:43:26AM +0900, Michael Paquier wrote:

Ah. Good point here. Better to tweak that and remove the dependency
on the length. Will look at all that head down..

Thanks both for pointing me to the text_length() optimization.

I have tested an extra approach where I make the REINDEX wait for
older snapshots after validating the new index, pointing at the fact
that these two extra injpoint wakeups should not be required, as much
as wait_for_point().

Nice, this is better. I think this approach makes the
define-index-before-set-valid injection point redundant. I haven't dug
into it, but maybe reindex-conc-index-built and
reindex-relation-concurrently-before-{swap, set-dead, drop} too, since
they all seem to be before a wait for lockers or older snapshots.

So I am feeling pretty confident that I got that right. The buildfarm
will judge, so we'll see. The CI was stable.

BF looks fine so far.

Anyway, applied the patch down to v14, with the test down to v17.

Thanks for taking care of this.

--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com

#6Michael Paquier
michael@paquier.xyz
In reply to: Bharath Rupireddy (#5)
Re: Fix stale relcache fields for TOAST indexes during REINDEX CONCURRENTLY

On Wed, Oct 28, 2026 at 09:04:00AM -0700, Bharath Rupireddy wrote:

Nice, this is better. I think this approach makes the
define-index-before-set-valid injection point redundant. I haven't dug
into it, but maybe reindex-conc-index-built and
reindex-relation-concurrently-before-{swap, set-dead, drop} too, since
they all seem to be before a wait for lockers or older snapshots.

If you think that we could remove some of the INJECTION_POINT() macros
without impacting the coverage of the tests, we could do so.

I'm not planning to bother about a couple of lines of code, FWIW, and
the cost of these is low. And there is always the back-branch
consistency argument. As a whole I'm fine with the state of HEAD and
the stable branches.
--
Michael