[PATCH v1] Generic Datum Encoding and Compression Contexts (pg_encoding_type, pg_encoding_context)
Hackorum builds and tests every patch posted to the lists, not only commitfest submissions. This is Hackorum's own CI rather than the PostgreSQL project's, and it is still under testing - please report anything that looks wrong.
You can run a PostgreSQL built from this patch straight from Docker, with no checkout and no build:
docker run --rm -p 5432:5432 ghcr.io/hackorum-dev/postgres-patch:t254263psql -h localhost -U postgresBuilt from patchset v1 (message #1), October 26, 2026 at 07:01 PM.
Every patchset is also pushed to a branch of our PostgreSQL fork, so you can check out the same tree CI built. Without a PostgreSQL checkout:
git clone --branch t254263_1 https://github.com/hackorum-dev/postgres.gitIn a checkout you already have, add the fork once:
git remote add hackorum https://github.com/hackorum-dev/postgres.gitthen, for this patchset and every later one:
git fetch hackorum t254263_1 && git checkout t254263_1Patchset v1 (message #1) is on t254263_1
Hi hackers,
Attached is a patch introducing a generic, extensible Datum Encoding and
Compression Context subsystem for PostgreSQL (`pg_encoding_type` and
`pg_encoding_context`).
1. Motivation
=============
Currently, column-level compression in PostgreSQL is controlled by the single
`char` column `pg_attribute.attcompression` ('p' for pglz, 'l' for lz4) and
operates solely as a stateless, parameterless byte-stream transformation.
In practice, many high-value storage and analytical workloads benefit from
stateful, pre-trained, or schema-aware encoding metadata stored in system
catalogs and cached in backend memory:
- Dictionary-based compression (e.g., Zstd pre-trained dictionaries for JSONB
or log payloads, FSST symbol tables for short strings) where per-datum
compression ratios improve dramatically when a shared dictionary is
factored out of individual tuples.
- Vector quantization and entropy coding (e.g., scalar quantization min/max
calibration bounds, tANS / asymmetric numeral systems tables, or custom
codebooks) where a column or type modifier references a trained context.
- Schema-aware binary serialization (e.g., Avro or Protobuf schemas) and
multi-stage pipelines (e.g., schema encoding followed by dictionary
compression).
- Online dictionary / schema evolution without a full table rewrite: when a
better dictionary or new schema version is trained, new writes should use
the new context while historical tuples remain self-describing and readable
using the context with which they were written.
- Multi-variant fallback within a single column ("Context Groups"): a column
may prefer a specialized encoding variant (e.g., a restricted hex/UUID
alphabet or a strict schema) that falls back automatically to a more
general variant if a particular input datum cannot be represented by the
stricter variant, using only a 1-byte (`uint8`) variant selector in the
encoded payload.
2. Overview of the Design
=========================
2.1 System Catalogs
-------------------
* `pg_encoding_type` (OID 9440):
Registers pluggable encoding engines (built-in or extension-provided):
- `oid` (`oid`)
- `enctypname` (`name`), `enctypnamespace` (`oid`), `enctypowner` (`oid`)
- `enctypkind` (`char`):
'd' = dictionary, 'c' = compression, 'q' = quantization, 'e' = custom
- `enctyptype` (`text`): engine identifier (e.g., "pglz",
"zstd", "xor_lut")
- `enctypencfunc` (`regproc`): `enc_fn(internal, anyelement) -> bytea`
- `enctypdecfunc` (`regproc`): `dec_fn(internal, bytea) -> anyelement`
- `enctypexpfunc` (`regproc`, optional): `exp_fn(oid, text) -> json`
- `enctypoptions` (`text[]`): default key=value options
* `pg_encoding_context` (OID 9431):
Stores trained or configured encoding context instances:
- `oid` (`oid`)
- `ectxname` (`name`), `ectxnamespace` (`oid`), `ectxowner` (`oid`)
- `ectxtypid` (`oid` -> `pg_encoding_type.oid`)
- `ectxtargettypid` (`oid` -> `pg_type.oid`, or 0 for polymorphic/any)
- `ectxwriteok` (`bool`, default `true`): whether this context may be
used for encoding new data; setting `ectxwriteok = false` retires a
context for new writes while keeping it active for decoding existing
tuples.
- `ectxgroup` (`oid` -> `pg_encoding_context.oid`): context group OID
stored in `pg_attribute.attencoding_context`. Always equals `oid` for
standalone contexts and group leaders, and equals the group leader's
`oid` for additional variants in the group.
- `ectxvariant` (`uint8`, `0..255`, default `0`): 1-byte variant selector
within `ectxgroup`. `0` designates a standalone context
(`ectxgroup = oid`,
variants not supported). Non-zero values (`1..255`) identify and order
variants within a group.
- `ectxlabel` (`text`): human-readable version/schema label
- `ectxenctab` (`bytea`), `ectxdectab` (`bytea`): raw encoding and
decoding dictionaries/tables (TOAST-able)
- `ectxoptions` (`text[]`): per-context key=value options
* `pg_attribute.attencoding_context` (`oid`, default `0`):
Binds a column to an encoding context (or context group leader) for new
writes, tracked via `pg_depend` (`DEPENDENCY_NORMAL`).
2.2 Context Groups (`ectxgroup` / `ectxvariant`) and Lazy Fallback
------------------------------------------------------------------
When a column can be encoded by multiple related contexts (for example, a
high-density restricted-alphabet encoder as primary variant `1` and a general
compressor as fallback variant `2`), storing a 4-byte OID per inline value is
unnecessary overhead when a 1-byte (`uint8`) variant indicator suffices.
- `pg_attribute.attencoding_context` stores the 4-byte `ectxgroup` OID.
- Each member of a group has a unique `ectxvariant` in `1..255`.
- When encoding with a group context, `EncodingContextEncodeWithEffectiveOid`
starts at the lowest `ectxvariant` in the group. If a variant has
`ectxwriteok = false` or its encode handler declines/fails to encode the
input datum, execution automatically falls back to the context with the
same `ectxgroup` and the next higher `ectxvariant`, returning both the
encoded `bytea` and the effective context OID (`*used_context_id`) that
succeeded.
- In memory, `EncodingContextData.fallback` is initialized to
`ECTX_UNRESOLVED` (`(EncodingContextData *)(uintptr_t) 1`) when
`variant > 0` (and `NULL` when `variant == 0`), and is lazily resolved on
first fallback via `EncodingContextGetFallback()` by scanning the group for
the smallest `ectxvariant > current->variant`.
2.3 Backend Cache & Invalidation
--------------------------------
`LookupEncodingContext(Oid)` and
`LookupEncodingContextByGroupVariant(Oid, uint8)`
materialize `EncodingContextData` entries into a session-local `dynahash` table
allocated inside `EncodingContextMemoryContext` (a child of
`CacheMemoryContext`).
- Each cached entry has its own child `MemoryContext` so individual entries
can be freed cleanly without leaking or resetting sibling contexts.
- `CacheRegisterSyscacheCallback` hooks on `ENCODINGCONTEXTOID` and
`ENCODINGTYPEOID` invalidate cached entries on `ALTER` or `DROP`, and reset
resolved `fallback` pointers back to `ECTX_UNRESOLVED` so group membership
changes take effect immediately across backends.
- Active encode/decode calls pin entries via `ectx->refcount`; if an entry is
invalidated while pinned on the call stack, destruction is deferred until
the outermost caller releases its pin.
2.4 Common Name / Numeric OID Resolution
----------------------------------------
To provide uniform context resolution across core SQL functions, DDL, and
extension `typmodin` handlers (such as `sqvector(dim, 'ctx_name')` or
`sqvector(dim, 16385)`):
- C API: `Oid ResolveEncodingContextOid(const char *name_or_oid,
bool missing_ok)`
- SQL function: `pg_resolve_encoding_context_oid(name_or_oid text,
missing_ok bool DEFAULT false) RETURNS oid`
If the input string consists of decimal digits corresponding to an existing
`pg_encoding_context.oid`, that OID is returned directly; otherwise the string
is parsed as an identifier (supporting schema-qualified `schema.name` or
`search_path` resolution).
2.5 SQL Functions & Built-in Reference Handlers
-----------------------------------------------
This patch provides SQL-callable functions for managing, testing, and
inspecting encoding types and contexts:
- `pg_create_encoding_type`, `pg_drop_encoding_type`
- `pg_create_encoding_context`, `pg_alter_encoding_context`,
`pg_drop_encoding_context`
- `pg_set_column_encoding_context`
- `pg_resolve_encoding_context_oid`
- `pg_explain_encoding_context(context_id oid, direction text
DEFAULT 'encoding') -> json`
- `pg_encode_datum(context_id oid, val anyelement) -> bytea`
- `pg_decode_datum(context_id oid, payload bytea, target_type
anyelement) -> anyelement`
- `pg_column_raw_encoding(val anyelement) -> bytea`
And two built-in reference encoding engines:
- `pg_bytea_dict_encode` / `pg_bytea_dict_decode` / `pg_bytea_dict_explain`
(supporting both XOR dictionary transformations and restricted-alphabet
lookup tables for testing multi-variant group fallback)
- `pg_pglz_context_encode` / `pg_pglz_context_decode` /
`pg_pglz_context_explain`
3. Patch Roadmap
================
To keep review manageable, this work is structured as a 3-part series:
- Patch 0001 (attached): Core `pg_encoding_type` and `pg_encoding_context`
catalogs, `pg_attribute.attencoding_context`, backend cache, context group
lazy fallback, `ResolveEncodingContextOid`, `pg_depend`/`ObjectAddress`
integration, SQL management/inspection functions, built-in handlers, and
regression tests (`src/test/regress/sql/encoding_context.sql`).
- Patch 0002 (follow-up): Native SQL DDL grammar (`CREATE/ALTER/DROP ENCODING
TYPE`, `CREATE/ALTER/DROP ENCODING CONTEXT`, `ALTER TABLE ... ALTER COLUMN
... SET/DROP ENCODING CONTEXT`) and `COMMENT ON` / `SECURITY LABEL` /
`ALTER ... RENAME` / `SET SCHEMA` / `OWNER TO` support.
- Patch 0003 (follow-up, building on Direct TOAST): Per-chunk and whole-datum
transparent TOAST encoding/decoding via `chunk_ectx_id`, O(log N) chunk
slice decompression, and zero-rewrite online context switching.
Feedback on the catalog design, Context Group (`ectxgroup` / `ectxvariant`)
fallback semantics, and C/SQL APIs is very welcome.
Best regards,
Hannu Krosing
Hi
I'm writing this email to express my interest in reviewing your patch, sure
I'll review it once I'm free from my chores.
With Regards
Osama Abdul Qader
On Sun, 18 Oct 2026 at 22:19, Hannu Krosing <hannuk@google.com> wrote:
Show quoted text
Hi hackers,
Attached is a patch introducing a generic, extensible Datum Encoding and
Compression Context subsystem for PostgreSQL (`pg_encoding_type` and
`pg_encoding_context`).1. Motivation
=============Currently, column-level compression in PostgreSQL is controlled by the
single
`char` column `pg_attribute.attcompression` ('p' for pglz, 'l' for lz4) and
operates solely as a stateless, parameterless byte-stream transformation.In practice, many high-value storage and analytical workloads benefit from
stateful, pre-trained, or schema-aware encoding metadata stored in system
catalogs and cached in backend memory:- Dictionary-based compression (e.g., Zstd pre-trained dictionaries for
JSONB
or log payloads, FSST symbol tables for short strings) where per-datum
compression ratios improve dramatically when a shared dictionary is
factored out of individual tuples.
- Vector quantization and entropy coding (e.g., scalar quantization
min/max
calibration bounds, tANS / asymmetric numeral systems tables, or custom
codebooks) where a column or type modifier references a trained
context.
- Schema-aware binary serialization (e.g., Avro or Protobuf schemas) and
multi-stage pipelines (e.g., schema encoding followed by dictionary
compression).
- Online dictionary / schema evolution without a full table rewrite:
when a
better dictionary or new schema version is trained, new writes should
use
the new context while historical tuples remain self-describing and
readable
using the context with which they were written.
- Multi-variant fallback within a single column ("Context Groups"): a
column
may prefer a specialized encoding variant (e.g., a restricted hex/UUID
alphabet or a strict schema) that falls back automatically to a more
general variant if a particular input datum cannot be represented by
the
stricter variant, using only a 1-byte (`uint8`) variant selector in the
encoded payload.2. Overview of the Design
=========================2.1 System Catalogs
-------------------* `pg_encoding_type` (OID 9440):
Registers pluggable encoding engines (built-in or extension-provided):
- `oid` (`oid`)
- `enctypname` (`name`), `enctypnamespace` (`oid`), `enctypowner`
(`oid`)
- `enctypkind` (`char`):
'd' = dictionary, 'c' = compression, 'q' = quantization, 'e' =
custom
- `enctyptype` (`text`): engine identifier (e.g., "pglz",
"zstd", "xor_lut")
- `enctypencfunc` (`regproc`): `enc_fn(internal, anyelement) ->
bytea`
- `enctypdecfunc` (`regproc`): `dec_fn(internal, bytea) ->
anyelement`
- `enctypexpfunc` (`regproc`, optional): `exp_fn(oid, text) -> json`
- `enctypoptions` (`text[]`): default key=value options* `pg_encoding_context` (OID 9431):
Stores trained or configured encoding context instances:
- `oid` (`oid`)
- `ectxname` (`name`), `ectxnamespace` (`oid`), `ectxowner` (`oid`)
- `ectxtypid` (`oid` -> `pg_encoding_type.oid`)
- `ectxtargettypid` (`oid` -> `pg_type.oid`, or 0 for
polymorphic/any)
- `ectxwriteok` (`bool`, default `true`): whether this context may be
used for encoding new data; setting `ectxwriteok = false` retires a
context for new writes while keeping it active for decoding
existing
tuples.
- `ectxgroup` (`oid` -> `pg_encoding_context.oid`): context group OID
stored in `pg_attribute.attencoding_context`. Always equals `oid`
for
standalone contexts and group leaders, and equals the group
leader's
`oid` for additional variants in the group.
- `ectxvariant` (`uint8`, `0..255`, default `0`): 1-byte variant
selector
within `ectxgroup`. `0` designates a standalone context
(`ectxgroup = oid`,
variants not supported). Non-zero values (`1..255`) identify and
order
variants within a group.
- `ectxlabel` (`text`): human-readable version/schema label
- `ectxenctab` (`bytea`), `ectxdectab` (`bytea`): raw encoding and
decoding dictionaries/tables (TOAST-able)
- `ectxoptions` (`text[]`): per-context key=value options* `pg_attribute.attencoding_context` (`oid`, default `0`):
Binds a column to an encoding context (or context group leader) for new
writes, tracked via `pg_depend` (`DEPENDENCY_NORMAL`).2.2 Context Groups (`ectxgroup` / `ectxvariant`) and Lazy Fallback
------------------------------------------------------------------When a column can be encoded by multiple related contexts (for example, a
high-density restricted-alphabet encoder as primary variant `1` and a
general
compressor as fallback variant `2`), storing a 4-byte OID per inline value
is
unnecessary overhead when a 1-byte (`uint8`) variant indicator suffices.- `pg_attribute.attencoding_context` stores the 4-byte `ectxgroup` OID.
- Each member of a group has a unique `ectxvariant` in `1..255`.
- When encoding with a group context,
`EncodingContextEncodeWithEffectiveOid`
starts at the lowest `ectxvariant` in the group. If a variant has
`ectxwriteok = false` or its encode handler declines/fails to encode
the
input datum, execution automatically falls back to the context with the
same `ectxgroup` and the next higher `ectxvariant`, returning both the
encoded `bytea` and the effective context OID (`*used_context_id`) that
succeeded.
- In memory, `EncodingContextData.fallback` is initialized to
`ECTX_UNRESOLVED` (`(EncodingContextData *)(uintptr_t) 1`) when
`variant > 0` (and `NULL` when `variant == 0`), and is lazily resolved
on
first fallback via `EncodingContextGetFallback()` by scanning the
group for
the smallest `ectxvariant > current->variant`.2.3 Backend Cache & Invalidation
--------------------------------`LookupEncodingContext(Oid)` and
`LookupEncodingContextByGroupVariant(Oid, uint8)`
materialize `EncodingContextData` entries into a session-local `dynahash`
table
allocated inside `EncodingContextMemoryContext` (a child of
`CacheMemoryContext`).- Each cached entry has its own child `MemoryContext` so individual
entries
can be freed cleanly without leaking or resetting sibling contexts.
- `CacheRegisterSyscacheCallback` hooks on `ENCODINGCONTEXTOID` and
`ENCODINGTYPEOID` invalidate cached entries on `ALTER` or `DROP`, and
reset
resolved `fallback` pointers back to `ECTX_UNRESOLVED` so group
membership
changes take effect immediately across backends.
- Active encode/decode calls pin entries via `ectx->refcount`; if an
entry is
invalidated while pinned on the call stack, destruction is deferred
until
the outermost caller releases its pin.2.4 Common Name / Numeric OID Resolution
----------------------------------------To provide uniform context resolution across core SQL functions, DDL, and
extension `typmodin` handlers (such as `sqvector(dim, 'ctx_name')` or
`sqvector(dim, 16385)`):- C API: `Oid ResolveEncodingContextOid(const char *name_or_oid,
bool missing_ok)`
- SQL function: `pg_resolve_encoding_context_oid(name_or_oid text,
missing_ok bool DEFAULT false) RETURNS oid`If the input string consists of decimal digits corresponding to an existing
`pg_encoding_context.oid`, that OID is returned directly; otherwise the
string
is parsed as an identifier (supporting schema-qualified `schema.name` or
`search_path` resolution).2.5 SQL Functions & Built-in Reference Handlers
-----------------------------------------------This patch provides SQL-callable functions for managing, testing, and
inspecting encoding types and contexts:- `pg_create_encoding_type`, `pg_drop_encoding_type`
- `pg_create_encoding_context`, `pg_alter_encoding_context`,
`pg_drop_encoding_context`
- `pg_set_column_encoding_context`
- `pg_resolve_encoding_context_oid`
- `pg_explain_encoding_context(context_id oid, direction text
DEFAULT 'encoding') -> json`
- `pg_encode_datum(context_id oid, val anyelement) -> bytea`
- `pg_decode_datum(context_id oid, payload bytea, target_type
anyelement) -> anyelement`
- `pg_column_raw_encoding(val anyelement) -> bytea`And two built-in reference encoding engines:
- `pg_bytea_dict_encode` / `pg_bytea_dict_decode` /
`pg_bytea_dict_explain`
(supporting both XOR dictionary transformations and restricted-alphabet
lookup tables for testing multi-variant group fallback)
- `pg_pglz_context_encode` / `pg_pglz_context_decode` /
`pg_pglz_context_explain`3. Patch Roadmap
================To keep review manageable, this work is structured as a 3-part series:
- Patch 0001 (attached): Core `pg_encoding_type` and
`pg_encoding_context`
catalogs, `pg_attribute.attencoding_context`, backend cache, context
group
lazy fallback, `ResolveEncodingContextOid`, `pg_depend`/`ObjectAddress`
integration, SQL management/inspection functions, built-in handlers,
and
regression tests (`src/test/regress/sql/encoding_context.sql`).
- Patch 0002 (follow-up): Native SQL DDL grammar (`CREATE/ALTER/DROP
ENCODING
TYPE`, `CREATE/ALTER/DROP ENCODING CONTEXT`, `ALTER TABLE ... ALTER
COLUMN
... SET/DROP ENCODING CONTEXT`) and `COMMENT ON` / `SECURITY LABEL` /
`ALTER ... RENAME` / `SET SCHEMA` / `OWNER TO` support.
- Patch 0003 (follow-up, building on Direct TOAST): Per-chunk and
whole-datum
transparent TOAST encoding/decoding via `chunk_ectx_id`, O(log N) chunk
slice decompression, and zero-rewrite online context switching.Feedback on the catalog design, Context Group (`ectxgroup` / `ectxvariant`)
fallback semantics, and C/SQL APIs is very welcome.Best regards,
Hannu Krosing