Catcache rehash hits MaxAllocSize (1GB) and errors with "invalid memory alloc request size" on very large numbers of relations

Hi,

While running ANALYZE across a database with 500,000+ tables (~350

columns each) in a single client session, we hit with an error:

  ERROR: invalid memory alloc request size 1073741824

This isn't just "the catcache got big and used a lot of RAM" - it's a

specific, deterministic ceiling in RehashCatCache()/RehashCatCacheLists()

that any sufficiently large catcache will hit, regardless of how much

memory is available on the machine.

What We Observed

-----------------

After ANALYZE had processed ~505,180 tables (each contains 350 cols

approx) in one session, a memory context dump showed:

  === Memory Contexts after vacuuming 505180 tables ===

  CacheMemoryContext                       26 GB      26 GB      1

  Attopt cache                             120 MB     118 MB     1

  PgStat Shared Ref                        35 MB      35 MB      1

  Relcache by OID                          24 MB      19 MB      1

  PgStat Shared Ref Hash                   24 MB      24 MB      1

  ...

Eventually a subsequent allocation fails with:

  ERROR: invalid memory alloc request size 1073741824

Root cause

----------

catcache.c's CatalogCacheCreateEntry() triggers a rehash whenever the

load factor exceeds 2:

    if (cache->cc_ntup > cache->cc_nbuckets * 2)

        RehashCatCache(cache);

RehashCatCache() doubles the bucket count and allocates the new bucket

array with the *regular* allocator:

    newnbuckets = cp->cc_nbuckets * 2;

    newbucket = (dlist_head *) MemoryContextAllocZero(CacheMemoryContext,

                                                       newnbuckets * sizeof(dlist_head));

MemoryContextAllocZero() enforces MaxAllocSize (0x3fffffff =

1,073,741,823 bytes - "1 gigabyte minus 1", see memutils.h). sizeof

(dlist_head) is 16 bytes on 64-bit platforms. Buckets are powers of two,

so the bucket array size doubles:..., 512MB, then exactly 1GB. That

final doubling requests:

    67,108,864 buckets * 16 bytes = 1,073,741,824 bytes

which is exactly 1 byte over MaxAllocSize, so the allocation is refused

and the backend hard-errors with "invalid memory alloc request size

1073741824" - the exact message and exact number we observed.

Given the load factor of 2, this doubling occurs once a catcache

accumulates roughly 67.1 million live tuples. For ATTNUM specifically

(examine_attribute() in analyze.c calls SearchSysCache2(ATTNUM, ...)

once per column of every relation ANALYZE processes), that means any

session whose ANALYZE/autovacuum touches enough distinct relations to

push pg_attribute entries past ~67M will hit this.

Why this matters beyond "reduce your table count"

--------------------------------------------------

- It fails outright rather than degrading gracefully or scaling with

  available RAM. A machine with terabytes of free memory will still hit

  this at exactly the same tuple count, because the ceiling is

  MaxAllocSize, not physical memory.

- Felt patching a single call site as a partial fix. We tested

  replacing SearchSysCache2() with a direct systable_beginscan() in

  examine_attribute(), which will avoids populating the ATTNUM catcache

  from that one code path. That slows down how fast the cache fills, but

  any other code path that still uses the catcache heavily (in this backend

  or others) will eventually hit the same ceiling.

Open Questions / Possible Fixes

--------------------------------

1. Have RehashCatCache()/RehashCatCacheLists() allocate the bucket array

   with MemoryContextAllocHuge() instead of MemoryContextAllocZero().

   Is there a known reason this wasn't already done, or would a patch

   along these lines be welcome?

2. If there's a reason the 1GB ceiling must stay in place for this

   allocation, then is splitting ANALYZE into multiple sessions/backends,

   each covering a limited number of tables (so no single backend's catcache

   grows past the threshold), the only real workaround available today? Or

   is there some other supported way to Invalidate/reset a backend's catcache

   growth (short of restarting the connection) that we're missing?

The reproduction scripts we used are attached below.

Thanks & Regards,

Reshmithaa B

Member Technical Staff

ZOHO Corporation