Possible to go without page headers?
I'm writing an index access method with its own unique file format. It
involves storing large blobs that break across pages.
The file format itself doesn't need or use page headers. There's no need
for a checksum or to manage free space within the page.
Can I treat pages as just a flat, open 8k buffer and fill them with
arbitrary data?
The reason I ask is that I see some reference to an LSN, used to determine
when to dump a dirty buffer to disk, and don't know whether that is
actually required. I plan to write a large number of pages all at once and
I'm not yet quite sure how WAL logging will work. I also see some
suggestion that the vacuum process uses page headers, but I haven't quite
figured that out either.
Chris Cleveland <ccleve+github@dieselpoint.com> writes:
Can I treat pages as just a flat, open 8k buffer and fill them with
arbitrary data?
No, at least not unless you plan to reimplement much of the WAL
mechanism. You do need at least an LSN in the right place.
I kinda doubt that you can get away with ignoring checksumming,
either. On the whole, I think you'd be best off to use a standard
page header; the amount you're saving by avoiding that will be
minuscule, and the amount of work you cause for yourself probably
not so much.
BTW, there are also tools such as pg_filedump that expect that index
pages can be identified by some sort of magic number kept in the
"special space" at the page tail. You're not absolutely bound to make
that work, but you'll be cutting yourself off from some potentially
handy support.
regards, tom lane
On 2/14/22 16:19, Tom Lane wrote:
Chris Cleveland <ccleve+github@dieselpoint.com> writes:
Can I treat pages as just a flat, open 8k buffer and fill them with
arbitrary data?No, at least not unless you plan to reimplement much of the WAL
mechanism. You do need at least an LSN in the right place.
I kinda doubt that you can get away with ignoring checksumming,
either. On the whole, I think you'd be best off to use a standard
page header; the amount you're saving by avoiding that will be
minuscule, and the amount of work you cause for yourself probably
not so much.BTW, there are also tools such as pg_filedump that expect that index
pages can be identified by some sort of magic number kept in the
"special space" at the page tail. You're not absolutely bound to make
that work, but you'll be cutting yourself off from some potentially
handy support.
You'll also get errors from external tools (like pgBackRest) that
validate checksums and headers.
Regards,
-David
On Mon, Feb 14, 2022 at 2:19 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
No, at least not unless you plan to reimplement much of the WAL
mechanism. You do need at least an LSN in the right place.
I kinda doubt that you can get away with ignoring checksumming,
either. On the whole, I think you'd be best off to use a standard
page header; the amount you're saving by avoiding that will be
minuscule, and the amount of work you cause for yourself probably
not so much.
It isn't actually necessary for an index AM to use the standard
slotted page format to get the benefits that you mention, of course --
whether or not an index AM that uses standard page headers *also* uses
slotted pages with standard line pointers is a separate question. For
example, GIN posting tree pages don't use standard line pointers, but
still have a standard page header (and a generic GIN special area in
the opaque space).
I agree that it's hard to imagine that opting out of using the
standard page header format could ever make much sense. Principally
because the restrictions imposed on an index AM that uses the standard
page header format are very minimal, while the benefits are
substantial.
--
Peter Geoghegan
Peter Geoghegan <pg@bowt.ie> writes:
It isn't actually necessary for an index AM to use the standard
slotted page format to get the benefits that you mention, of course --
whether or not an index AM that uses standard page headers *also* uses
slotted pages with standard line pointers is a separate question.
Right, you don't need to use a line pointer array if you don't want
to. (IIRC, hash also opts out of that in some pages.) I took the
question to be just about the page header proper.
regards, tom lane