REPACK (CONCURRENTLY) doesn't check the table AM

Started by Nathan Bossart3 months ago12 messageshackers
Jump to latest
#1Nathan Bossart
nathandbossart@gmail.com

REPACK (CONCURRENTLY) replays changes by decoding them from WAL, so it
needs the table AM to be logically decodable, but
check_concurrent_repack_requirements() doesn't check that. Should it be
restricted to heap for v19?

--
nathan

#2Bharath Rupireddy
bharath.rupireddyforpostgres@gmail.com
In reply to: Nathan Bossart (#1)
Re: REPACK (CONCURRENTLY) doesn't check the table AM

Hi,

On Thu, Aug 27, 2026 at 9:23 AM Nathan Bossart <nathandbossart@gmail.com> wrote:

REPACK (CONCURRENTLY) replays changes by decoding them from WAL, so it
needs the table AM to be logically decodable, but
check_concurrent_repack_requirements() doesn't check that. Should it be
restricted to heap for v19?

Logical decoding doesn't check the table AM type, but goes ahead and
decodes the WAL records (for that matter, none of CLUSTER, VACUUM
FULL, and REPACK have checks on table AM. They hand off at some point
to the table AM layer). I'm not sure if gating it just for concurrent
repack is the right choice. Is it that we want to have it just for
concurrent repack since it's new code with a new logical decoding
plugin and we want some field reports of needing it for other table
AMs?

--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com

#3Nathan Bossart
nathandbossart@gmail.com
In reply to: Bharath Rupireddy (#2)
Re: REPACK (CONCURRENTLY) doesn't check the table AM

On Thu, Aug 27, 2026 at 11:59:10AM -0700, Bharath Rupireddy wrote:

On Thu, Aug 27, 2026 at 9:23 AM Nathan Bossart <nathandbossart@gmail.com> wrote:

REPACK (CONCURRENTLY) replays changes by decoding them from WAL, so it
needs the table AM to be logically decodable, but
check_concurrent_repack_requirements() doesn't check that. Should it be
restricted to heap for v19?

Logical decoding doesn't check the table AM type, but goes ahead and
decodes the WAL records (for that matter, none of CLUSTER, VACUUM
FULL, and REPACK have checks on table AM. They hand off at some point
to the table AM layer). I'm not sure if gating it just for concurrent
repack is the right choice. Is it that we want to have it just for
concurrent repack since it's new code with a new logical decoding
plugin and we want some field reports of needing it for other table
AMs?

If the other commands are similarly broken, then perhaps this isn't a valid
open item for v19. But IIUC concurrent repack is the only one of the
commands you listed that depends on logical decoding, and if the table AM
doesn't support that, then it won't work properly.

--
nathan

#4Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Nathan Bossart (#3)
Re: REPACK (CONCURRENTLY) doesn't check the table AM

On Thu, Aug 27, 2026 at 12:16 PM Nathan Bossart
<nathandbossart@gmail.com> wrote:

On Thu, Aug 27, 2026 at 11:59:10AM -0700, Bharath Rupireddy wrote:

On Thu, Aug 27, 2026 at 9:23 AM Nathan Bossart <nathandbossart@gmail.com> wrote:

REPACK (CONCURRENTLY) replays changes by decoding them from WAL, so it
needs the table AM to be logically decodable, but
check_concurrent_repack_requirements() doesn't check that. Should it be
restricted to heap for v19?

Logical decoding doesn't check the table AM type, but goes ahead and
decodes the WAL records (for that matter, none of CLUSTER, VACUUM
FULL, and REPACK have checks on table AM. They hand off at some point
to the table AM layer). I'm not sure if gating it just for concurrent
repack is the right choice. Is it that we want to have it just for
concurrent repack since it's new code with a new logical decoding
plugin and we want some field reports of needing it for other table
AMs?

If the other commands are similarly broken, then perhaps this isn't a valid
open item for v19. But IIUC concurrent repack is the only one of the
commands you listed that depends on logical decoding, and if the table AM
doesn't support that, then it won't work properly.

If the table AM doesn't support logical decoding, concurrent repack
would silently lose some table data as it misses the changes
happened during the rewrites. I think we should have the check for it.
One idea would be to have a new table AM callback returning true if
the table AM supports concurrent repack (i.e., its rmgr provides
rm_decode callback), but I think it's not the right time to introduce
a new table AM callback. So restricting it to heap for v19 seems fine
to me.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

#5Bharath Rupireddy
bharath.rupireddyforpostgres@gmail.com
In reply to: Masahiko Sawada (#4)
Re: REPACK (CONCURRENTLY) doesn't check the table AM

Hi,

On Thu, Aug 27, 2026 at 12:49 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

On Thu, Aug 27, 2026 at 12:16 PM Nathan Bossart
<nathandbossart@gmail.com> wrote:

On Thu, Aug 27, 2026 at 11:59:10AM -0700, Bharath Rupireddy wrote:

On Thu, Aug 27, 2026 at 9:23 AM Nathan Bossart <nathandbossart@gmail.com> wrote:

REPACK (CONCURRENTLY) replays changes by decoding them from WAL, so it
needs the table AM to be logically decodable, but
check_concurrent_repack_requirements() doesn't check that. Should it be
restricted to heap for v19?

Logical decoding doesn't check the table AM type, but goes ahead and
decodes the WAL records (for that matter, none of CLUSTER, VACUUM
FULL, and REPACK have checks on table AM. They hand off at some point
to the table AM layer). I'm not sure if gating it just for concurrent
repack is the right choice. Is it that we want to have it just for
concurrent repack since it's new code with a new logical decoding
plugin and we want some field reports of needing it for other table
AMs?

If the other commands are similarly broken, then perhaps this isn't a valid
open item for v19. But IIUC concurrent repack is the only one of the
commands you listed that depends on logical decoding, and if the table AM
doesn't support that, then it won't work properly.

If the table AM doesn't support logical decoding, concurrent repack
would silently lose some table data as it misses the changes
happened during the rewrites.

Yes, that's correct. I came to the same conclusion.

I think we should have the check for it.
One idea would be to have a new table AM callback returning true if
the table AM supports concurrent repack (i.e., its rmgr provides
rm_decode callback), but I think it's not the right time to introduce
a new table AM callback.

Agreed. Adding a new table AM callback needs consensus and is time
consuming. Perhaps we can try for HEAD based on field reports that
require the concurrent repack feature for non-heap table AMs.

So restricting it to heap for v19 seems fine
to me.

+1. I will send a patch soon.

--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com

#6Bharath Rupireddy
bharath.rupireddyforpostgres@gmail.com
In reply to: Bharath Rupireddy (#5)
Re: REPACK (CONCURRENTLY) doesn't check the table AM

Hi,

On Thu, Aug 27, 2026 at 12:57 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:

If the table AM doesn't support logical decoding, concurrent repack
would silently lose some table data as it misses the changes
happened during the rewrites.

Yes, that's correct. I came to the same conclusion.

I think we should have the check for it.
One idea would be to have a new table AM callback returning true if
the table AM supports concurrent repack (i.e., its rmgr provides
rm_decode callback), but I think it's not the right time to introduce
a new table AM callback.

Agreed. Adding a new table AM callback needs consensus and is time
consuming. Perhaps we can try for HEAD based on field reports that
require the concurrent repack feature for non-heap table AMs.

So restricting it to heap for v19 seems fine
to me.

+1. I will send a patch soon.

Please find the attached patch. I didn't add a test, I don't think we need one.

--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com

Attachments:

t253576_6
v1-0001-Restrict-concurrent-repack-to-the-heap-access-met.patchapplication/x-patch; name=v1-0001-Restrict-concurrent-repack-to-the-heap-access-met.patchDownload+14-1
#7Bharath Rupireddy
bharath.rupireddyforpostgres@gmail.com
In reply to: Bharath Rupireddy (#6)
Re: REPACK (CONCURRENTLY) doesn't check the table AM

Hi,

On Thu, Aug 27, 2026 at 1:45 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:

Hi,

On Thu, Aug 27, 2026 at 12:57 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:

If the table AM doesn't support logical decoding, concurrent repack
would silently lose some table data as it misses the changes
happened during the rewrites.

Yes, that's correct. I came to the same conclusion.

I think we should have the check for it.
One idea would be to have a new table AM callback returning true if
the table AM supports concurrent repack (i.e., its rmgr provides
rm_decode callback), but I think it's not the right time to introduce
a new table AM callback.

Agreed. Adding a new table AM callback needs consensus and is time
consuming. Perhaps we can try for HEAD based on field reports that
require the concurrent repack feature for non-heap table AMs.

So restricting it to heap for v19 seems fine
to me.

+1. I will send a patch soon.

Please find the attached patch. I didn't add a test, I don't think we need one.

Added the limitation to the repack docs. Please find the attached v2
patch. Sorry for the noise.

--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com

Attachments:

t253576_7
v2-0001-Restrict-concurrent-repack-to-the-heap-access-met.patchapplication/octet-stream; name=v2-0001-Restrict-concurrent-repack-to-the-heap-access-met.patchDownload+20-1
#8Bharath Rupireddy
bharath.rupireddyforpostgres@gmail.com
In reply to: Bharath Rupireddy (#7)
Re: REPACK (CONCURRENTLY) doesn't check the table AM

Hi,

On Thu, Aug 27, 2026 at 3:51 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:

On Thu, Aug 27, 2026 at 12:57 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:

If the table AM doesn't support logical decoding, concurrent repack
would silently lose some table data as it misses the changes
happened during the rewrites.

Yes, that's correct. I came to the same conclusion.

Please find the attached v3 patch with a test added similar to other
restricted error cases. Sorry for the noise.

--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com

Attachments:

t253576_8
v3-0001-Restrict-concurrent-repack-to-the-heap-access-met.patchapplication/octet-stream; name=v3-0001-Restrict-concurrent-repack-to-the-heap-access-met.patchDownload+35-1
#9Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Bharath Rupireddy (#8)
Re: REPACK (CONCURRENTLY) doesn't check the table AM

On Mon, Aug 31, 2026 at 8:46 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:

Hi,

On Thu, Aug 27, 2026 at 3:51 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:

On Thu, Aug 27, 2026 at 12:57 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:

If the table AM doesn't support logical decoding, concurrent repack
would silently lose some table data as it misses the changes
happened during the rewrites.

Yes, that's correct. I came to the same conclusion.

Please find the attached v3 patch with a test added similar to other
restricted error cases. Sorry for the noise.

I've looked at the patch and have one comment:

+ if (rel->rd_rel->relam != HEAP_TABLE_AM_OID)

Probably it's better to use GetHeapamTableAmRoutine() instead of using
heap AM oid. It would allow the following use case the patch adds in
the regression tests:

+CREATE ACCESS METHOD heap2 TYPE TABLE HANDLER heap_tableam_handler;
+CREATE TABLE repack_conc_heap2 (i int) USING heap2;
+REPACK (CONCURRENTLY) repack_conc_heap2;

REPACK'ing on such a table technically works fine. It's unlikely to
drop the check so I think we can add the check without regression
tests.

I attached the updated patch. Please review it.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Attachments:

t253576_9
v4-0001-Restrict-REPACK-CONCURRENTLY-to-the-heap-access-m.patchtext/x-patch; charset=US-ASCII; name=v4-0001-Restrict-REPACK-CONCURRENTLY-to-the-heap-access-m.patchDownload+21-1
#10Bharath Rupireddy
bharath.rupireddyforpostgres@gmail.com
In reply to: Masahiko Sawada (#9)
Re: REPACK (CONCURRENTLY) doesn't check the table AM

Hi,

On Fri, Sep 4, 2026 at 2:05 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

I've looked at the patch and have one comment:

Thanks for reviewing it.

+ if (rel->rd_rel->relam != HEAP_TABLE_AM_OID)

Probably it's better to use GetHeapamTableAmRoutine() instead of using
heap AM oid. It would allow the following use case the patch adds in
the regression tests:
+CREATE ACCESS METHOD heap2 TYPE TABLE HANDLER heap_tableam_handler;
+CREATE TABLE repack_conc_heap2 (i int) USING heap2;
+REPACK (CONCURRENTLY) repack_conc_heap2;

REPACK'ing on such a table technically works fine. It's unlikely to
drop the check so I think we can add the check without regression
tests.

That works for me. If a table AM that just copies the heap AM handler
as-is works with repack, that is fine. If a table AM that copies the
heap AM handler but overwrites one or more functions errors out with
repack, that still serves the purpose. Also, I am fine without a test
for this because we don't have another table AM in the core and all
tests rely on reusing the same heap AM handler.

I attached the updated patch. Please review it.

The v4 patch looks good to me. pgindent and tests are happy.

--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com

#11Bharath Rupireddy
bharath.rupireddyforpostgres@gmail.com
In reply to: Bharath Rupireddy (#10)
Re: REPACK (CONCURRENTLY) doesn't check the table AM

Hi,

On Fri, Sep 4, 2026 at 4:13 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:

I attached the updated patch. Please review it.

The v4 patch looks good to me. pgindent and tests are happy.

Please find the attached v5 patch containing detail instead of hint
and also consistent wording as discussed here:
/messages/by-id/CALj2ACXyTPZEcGeKgHQjtSCScOOLpHNhAdH3Cn2T48aXg2LZfA@mail.gmail.com

--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com

Attachments:

t253576_11
v5-0001-Restrict-REPACK-CONCURRENTLY-to-the-heap-access-m.patchapplication/octet-stream; name=v5-0001-Restrict-REPACK-CONCURRENTLY-to-the-heap-access-m.patchDownload+20-1
#12Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Bharath Rupireddy (#11)
Re: REPACK (CONCURRENTLY) doesn't check the table AM

On Tue, Sep 8, 2026 at 10:24 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:

Hi,

On Fri, Sep 4, 2026 at 4:13 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:

I attached the updated patch. Please review it.

The v4 patch looks good to me. pgindent and tests are happy.

Please find the attached v5 patch containing detail instead of hint
and also consistent wording as discussed here:
/messages/by-id/CALj2ACXyTPZEcGeKgHQjtSCScOOLpHNhAdH3Cn2T48aXg2LZfA@mail.gmail.com

Thank you for updating the patch. It looks good to me so I've pushed.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com