WAL Record Header Size Reduction
Current WAL Header uses 32 bytes on a 64-bit CPU. It seems possible to
reduce this to 24 bytes, without reducing resilience, when
full_page_writes = off. This will reduce overall WAL volumes by around
5-15%, depending upon the application with performance gains in various
ways.
If full_page_writes = off then it this is true
xl_tot_len == xl_len + SizeOfXLogRecord
since there are no backup blocks. As a result, there is no loss in
resilience by removing this field.
xlog.h shows this definition currently:
typedef struct XLogRecord
{
pg_crc32 xl_crc; /* CRC for this record */
XLogRecPtr xl_prev; /* ptr to previous record in log */
TransactionId xl_xid; /* xact id */
uint32 xl_tot_len; /* total len of entire record */
uint32 xl_len; /* total len of rmgr data */
uint8 xl_info; /* flag bits, see below */
RmgrId xl_rmid; /* resource manager for this record */
/* Depending on MAXALIGN, there are either 2 or 6 wasted bytes here */
I propose to rearrange the XLogRecord structure to this:
pg_crc32 xl_crc; /* CRC for this record */
uint8 xl_info; /* flag bits, see below */
RmgrId xl_rmid; /* resource manager for this record */
XLogRecPtr xl_prev; /* ptr to previous record in log */
TransactionId xl_xid; /* xact id */
uint32 xl_tot_len; /* total len of entire record */
which will occupy 24 bytes, saving 4 bytes on 32-bit and 8 bytes on
64-bit architectures, once alignment is considered.
The xl_len field would be included only if backup blocks are included
with the record. This is already marked by flags in the xl_info field,
so no new flags are required.
These changes can be mostly isolated to xlog.c, since only XLogInsert(),
ReadRecord() and pg_resetxlog need to know about the changes. The xlog
record would dynamically adjust according to whether backup blocks are
present, so it can still work as full_page_writes is switched on/off by
user or during the period between start/stop backup.
The saving is only really relevant when full_page_writes = off, so I'm
not worried about changing the xlrec header in that case anyway.
--
Simon Riggs
EnterpriseDB http://www.enterprisedb.com
Simon Riggs wrote:
Current WAL Header uses 32 bytes on a 64-bit CPU. It seems possible to
reduce this to 24 bytes, without reducing resilience, when
full_page_writes = off. This will reduce overall WAL volumes by around
5-15%, depending upon the application with performance gains in various
ways.
Actually, it would help even when full_page_writes=on, because even then
most xlog records don't have backup blocks attached to them.
xlog.h shows this definition currently:
typedef struct XLogRecord
{
pg_crc32 xl_crc; /* CRC for this record */
XLogRecPtr xl_prev; /* ptr to previous record in log */
TransactionId xl_xid; /* xact id */
uint32 xl_tot_len; /* total len of entire record */
uint32 xl_len; /* total len of rmgr data */
uint8 xl_info; /* flag bits, see below */
RmgrId xl_rmid; /* resource manager for this record *//* Depending on MAXALIGN, there are either 2 or 6 wasted bytes here */
I propose to rearrange the XLogRecord structure to this:
I think you got your alignment wrong:
pg_crc32 xl_crc; /* CRC for this record */
uint8 xl_info; /* flag bits, see below */
RmgrId xl_rmid; /* resource manager for this record */
Because of xl_prev below which is two uint32 fields, there will be 2
bytes of wasted space in here.
XLogRecPtr xl_prev; /* ptr to previous record in log */
TransactionId xl_xid; /* xact id */
uint32 xl_tot_len; /* total len of entire record */
ISTM that we would get the effect your looking for by just moving the
xl_tot_len field to the end, and only storing it for records with backup
blocks:
pg_crc32 xl_crc; /* CRC for this record */
XLogRecPtr xl_prev; /* ptr to previous record in log */
TransactionId xl_xid; /* xact id */
uint32 xl_len; /* total len of rmgr data */
uint8 xl_info; /* flag bits, see below */
RmgrId xl_rmid; /* resource manager for this record */uint32 xl_tot_len; /* total len of entire record, if backup blocks
indicated in xl_info*/
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
On Thu, 2007-01-25 at 14:04 +0000, Heikki Linnakangas wrote:
Simon Riggs wrote:
Current WAL Header uses 32 bytes on a 64-bit CPU. It seems possible to
reduce this to 24 bytes, without reducing resilience, when
full_page_writes = off. This will reduce overall WAL volumes by around
5-15%, depending upon the application with performance gains in various
ways.Actually, it would help even when full_page_writes=on, because even then
most xlog records don't have backup blocks attached to them.
Sure it would be active, but I meant it probably would not reduce the
overall WAL volume by very much. We'll see, I guess.
xlog.h shows this definition currently:
typedef struct XLogRecord
{
pg_crc32 xl_crc; /* CRC for this record */
XLogRecPtr xl_prev; /* ptr to previous record in log */
TransactionId xl_xid; /* xact id */
uint32 xl_tot_len; /* total len of entire record */
uint32 xl_len; /* total len of rmgr data */
uint8 xl_info; /* flag bits, see below */
RmgrId xl_rmid; /* resource manager for this record *//* Depending on MAXALIGN, there are either 2 or 6 wasted bytes here */
I propose to rearrange the XLogRecord structure to this:
I think you got your alignment wrong:
pg_crc32 xl_crc; /* CRC for this record */
uint8 xl_info; /* flag bits, see below */
RmgrId xl_rmid; /* resource manager for this record */Because of xl_prev below which is two uint32 fields, there will be 2
bytes of wasted space in here.
No escaping that though.
ISTM that we would get the effect your looking for by just moving the
xl_tot_len field to the end, and only storing it for records with backup
blocks:pg_crc32 xl_crc; /* CRC for this record */
XLogRecPtr xl_prev; /* ptr to previous record in log */
TransactionId xl_xid; /* xact id */
uint32 xl_len; /* total len of rmgr data */
uint8 xl_info; /* flag bits, see below */
RmgrId xl_rmid; /* resource manager for this record */uint32 xl_tot_len; /* total len of entire record, if backup blocks
indicated in xl_info*/
OK. I was assuming XLogRecPtr was 8-byte aligned, but its a struct with
2 4-byte aligned elements, so I thought I had to move it to get the
benefit. Seems not...
--
Simon Riggs
EnterpriseDB http://www.enterprisedb.com
Heikki Linnakangas <heikki@enterprisedb.com> writes:
ISTM that we would get the effect your looking for by just moving the
xl_tot_len field to the end, and only storing it for records with backup
blocks:
I like that formulation better --- seems like less
change-for-the-sake-of-change.
regards, tom lane
On Thu, 2007-01-25 at 11:03 -0500, Tom Lane wrote:
Heikki Linnakangas <heikki@enterprisedb.com> writes:
ISTM that we would get the effect your looking for by just moving the
xl_tot_len field to the end, and only storing it for records with backup
blocks:I like that formulation better --- seems like less
change-for-the-sake-of-change.
Will do.
--
Simon Riggs
EnterpriseDB http://www.enterprisedb.com