BUG #19664: nbtree: Assertion failure when a custom index AM reuses bthandler

The following bug has been logged on the website:

Bug reference: 19664
Logged by: Jingtang Zhang
Email address: mrdrivingduck@gmail.com
PostgreSQL version: 19beta3
Operating system: Linux
Description:

## Repro

Build with assertions enabled, for example:

./configure --enable-debug --enable-cassert
make

---

Run SQL:

CREATE ACCESS METHOD hx_orph_ix TYPE INDEX HANDLER bthandler;

CREATE TABLE hx_orph_tab(a int, b text);
INSERT INTO hx_orph_tab
SELECT g, 'x' || g FROM generate_series(1, 100) AS g;

CREATE OPERATOR CLASS hx_orph_ix_int4_ops
DEFAULT FOR TYPE integer USING hx_orph_ix AS
OPERATOR 1 <,
OPERATOR 2 <=,
OPERATOR 3 =,
OPERATOR 4 >=,
OPERATOR 5 >,
FUNCTION 1 btint4cmp(integer, integer);

CREATE INDEX hx_orph_idx ON hx_orph_tab USING hx_orph_ix(a);

---

Get:

TRAP: failed Assert("wstate->index->rd_rel->relkind == RELKIND_INDEX &&
wstate->index->rd_rel->relam == BTREE_AM_OID")
File: "nbtsort.c"

---

## Thoughts

hx_orph_ix has a newly allocated AM OID, while its handler is bthandler. The
btree build path is therefore used, but BTGetFillFactor() and
BTGetDeduplicateItems() require the AM OID to be the built-in BTREE_AM_OID.

The OID check is not the relevant safety condition. These macros interpret
rd_options as BTOptions, so they should verify that the relation options
were parsed by btoptions. bthandler returns an IndexAmRoutine with
.amoptions = btoptions, making the layout compatible. Reusing built-in index
handlers is also an established pattern, e.g. the existing gist2 regression
test reuses gisthandler.

Proposed fix:

#define BTGetFillFactor(relation) \
(AssertMacro(relation->rd_rel->relkind == RELKIND_INDEX && \
relation->rd_indam->amoptions == btoptions), \
...)

#define BTGetDeduplicateItems(relation) \
(AssertMacro(relation->rd_rel->relkind == RELKIND_INDEX && \
relation->rd_indam->amoptions == btoptions), \
...)