[HACKERS] Max size of data types and tuples. (fwd)
Here's a blast from the past. Shows how I keep those open issues in my
mailbox.
Forwarded message:
From scrappy@postgreSQL.org Wed Jan 29 14:49:26 1997
X-Authentication-Warning: hub.org: pgsql set sender to owner-pg95-dev@postgreSQL.org using -f
Date: Wed, 29 Jan 1997 13:38:10 -0500
From: aixssd!darrenk@abs.net (Darren King)
Message-Id: <9701291838.AA22296@ceodev>
To: hackers@postgreSQL.org
Subject: [HACKERS] Max size of data types and tuples.
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Content-Md5: 9bBKVeeTrq97EhvkS6qT3A==
Sender: owner-pg95-dev@hub.org
Reply-To: hackers@hub.org, aixssd!darrenk@abs.net (Darren King)Does anyone know of a valid reason for the 4096 byte limit
on the textual fields (char, varchar, text, etc...)?Could it be bumped to (MAXTUPLEN - sizeof(int)) in the parser
and in utils/adt/varchar.c? MAXTUPLEN would have to be
calculated more accurately though as noted in the comment
around its definition.The following are just some of my observations and comments
on things that I think should be cleaned up a little in
the source (and I'll volunteer to do them).1. Clean up the #defines for the max block size. Currently,
there are at least four references to 8192...include/config.h
include/storage/bufmgr.h
include/optimizer/internal.h
backend/rewrite/rewriteDefine.cThe _PAGE_SIZE_ in internal.h is _not_ used anywhere in the
optimizer. This define should be linked to BLCKSZ, but would
be better to remove it and just use BLCKSZ.optimizer/costsize.c includes storage/bufpage.h for BLCKSZ,
but in fact this is defined in config.h!. Also included inexecutor/nodeHash.c
executor/nodeHashjoin.c
utils/sort/psort.c__These includes of storage/bufpage.h can be removed.__
There should be #define MAX_BLOCK_SIZE 8192
#define CURRENT_BLOCK_SIZE 8192The MAX_BLOCK_SIZE is a hard and fast limit since only 13 bits
can be used in offsets and the like. I believe that in the
future, PostgreSql should support block sizes other than 8k,
like 2k and 4k. Long-term goal, but the code should be done
to allow this at a future time. The comments in storage/bufpage.h
suggest to me that the original designers of Postgres had this
in mind.2. Once the block size issue is taken care of, calculate the
maximum tuple size more accurately.3. When #1 & #2 are resolved, let the textual fields have a max
of (MAX_TUPLE_SIZE - sizeof(int)).4. Since only 13 bits are needed for storing the size of these
textual fields in a tuple, could PostgreSql use a 16-bit int to
store it? Currently, the size is padded to four bytes in the
tuple and this eats space if you have many textual fields.
Without further digging, I'm assuming that the size is double-word
aligned so that the actual text starts on a double-word boundary.Thanks for reading this far. :)
Comments, suggestions most welcome...
Darren darrenk@insightdist.com
--
Bruce Momjian
maillist@candle.pha.pa.us
Here's a blast from the past. Shows how I keep those open issues in my
mailbox.Forwarded message:
Date: Wed, 29 Jan 1997 13:38:10 -0500
From: aixssd!darrenk@abs.net (Darren King)
To: hackers@postgreSQL.org
Subject: [HACKERS] Max size of data types and tuples.
Still buried in my 'received' box here too. Can't imagine all the bugs
and/or issues you have kept in yours.
1. Clean up the #defines for the max block size. Currently,
there are at least four references to 8192...
Think I found and fixed all of these up.
__These includes of storage/bufpage.h can be removed.__
Still _quite_ a few #includes that can be removed throughout. First,
"utils/elog.h" and "util/palloc.h" are include in "postgres.h", so are
unnecessary to include by themselves since "postgres.h" is include in
_every_ .c file, correct?
Also numerous #includes of "storage/bufpage.h" and "storage/fd.h" that are
unnecessary since the things they were included for (BLCKSZ and SEEK_*) are
now either in "config.h" or found in a system include file.
2. Once the block size issue is taken care of, calculate the
maximum tuple size more accurately.
...
3. When #1 & #2 are resolved, let the textual fields have a max
of (MAX_TUPLE_SIZE - sizeof(int)).
This could be done as soon as I come up with a way of defining the packet
size for the interfaces since this is the newest limiting factor.
Peter's suggestion of backend functions for getting info might be the way to
go. It would let the various interfaces get the info they need and would be
a step towards JDBC and ODBC compliance.
darrenk
Import Notes
Resolved by subject fallback
Here's a blast from the past. Shows how I keep those open issues in my
mailbox.Forwarded message:
Date: Wed, 29 Jan 1997 13:38:10 -0500
From: aixssd!darrenk@abs.net (Darren King)
To: hackers@postgreSQL.org
Subject: [HACKERS] Max size of data types and tuples.Still buried in my 'received' box here too. Can't imagine all the bugs
and/or issues you have kept in yours.
Not too bad now.
1. Clean up the #defines for the max block size. Currently,
there are at least four references to 8192...Think I found and fixed all of these up.
__These includes of storage/bufpage.h can be removed.__
Still _quite_ a few #includes that can be removed throughout. First,
"utils/elog.h" and "util/palloc.h" are include in "postgres.h", so are
unnecessary to include by themselves since "postgres.h" is include in
_every_ .c file, correct?
Yes, must be included. Period. Even 3rd party apps.
Also numerous #includes of "storage/bufpage.h" and "storage/fd.h" that are
unnecessary since the things they were included for (BLCKSZ and SEEK_*) are
now either in "config.h" or found in a system include file.2. Once the block size issue is taken care of, calculate the
maximum tuple size more accurately....
3. When #1 & #2 are resolved, let the textual fields have a max
of (MAX_TUPLE_SIZE - sizeof(int)).This could be done as soon as I come up with a way of defining the packet
size for the interfaces since this is the newest limiting factor.Peter's suggestion of backend functions for getting info might be the way to
go. It would let the various interfaces get the info they need and would be
a step towards JDBC and ODBC compliance.
Again, we could just set 3rd party apps to be the maximum tuple size we
will ever have to support.
--
Bruce Momjian
maillist@candle.pha.pa.us
On Wed, 14 Jan 1998, Bruce Momjian wrote:
This could be done as soon as I come up with a way of defining the packet
size for the interfaces since this is the newest limiting factor.Peter's suggestion of backend functions for getting info might be the way to
go. It would let the various interfaces get the info they need and would be
a step towards JDBC and ODBC compliance.Again, we could just set 3rd party apps to be the maximum tuple size we
will ever have to support.
Currently, were returning some defaults based on the 8K block size.
Probably for these, we may be able to get away with the values we are
setting. However, there are a few things that I think we will still need
to implement as functions.
--
Peter T Mount petermount@earthling.net or pmount@maidast.demon.co.uk
Main Homepage: http://www.demon.co.uk/finder
Work Homepage: http://www.maidstone.gov.uk Work EMail: peter@maidstone.gov.uk
On Wed, 14 Jan 1998, Bruce Momjian wrote:
This could be done as soon as I come up with a way of defining the packet
size for the interfaces since this is the newest limiting factor.Peter's suggestion of backend functions for getting info might be the way to
go. It would let the various interfaces get the info they need and would be
a step towards JDBC and ODBC compliance.Again, we could just set 3rd party apps to be the maximum tuple size we
will ever have to support.Currently, were returning some defaults based on the 8K block size.
Probably for these, we may be able to get away with the values we are
setting. However, there are a few things that I think we will still need
to implement as functions.
OK, let's decide soon, so people can be ready for Feb 1.
--
Bruce Momjian
maillist@candle.pha.pa.us
On Wed, 14 Jan 1998, Bruce Momjian wrote:
On Wed, 14 Jan 1998, Bruce Momjian wrote:
This could be done as soon as I come up with a way of defining the packet
size for the interfaces since this is the newest limiting factor.Peter's suggestion of backend functions for getting info might be the way to
go. It would let the various interfaces get the info they need and would be
a step towards JDBC and ODBC compliance.Again, we could just set 3rd party apps to be the maximum tuple size we
will ever have to support.Currently, were returning some defaults based on the 8K block size.
Probably for these, we may be able to get away with the values we are
setting. However, there are a few things that I think we will still need
to implement as functions.OK, let's decide soon, so people can be ready for Feb 1.
I'm going to sort out what needs to be done to get us as close to
compliance as possible over the next couple of days. Hopefully, we can
decide on some of them then.
--
Peter T Mount petermount@earthling.net or pmount@maidast.demon.co.uk
Main Homepage: http://www.demon.co.uk/finder
Work Homepage: http://www.maidstone.gov.uk Work EMail: peter@maidstone.gov.uk
Are we done with these issues, or are you still working on them, or is
Peter working on this?
Here's a blast from the past. Shows how I keep those open issues in my
mailbox.Forwarded message:
Date: Wed, 29 Jan 1997 13:38:10 -0500
From: aixssd!darrenk@abs.net (Darren King)
To: hackers@postgreSQL.org
Subject: [HACKERS] Max size of data types and tuples.Still buried in my 'received' box here too. Can't imagine all the bugs
and/or issues you have kept in yours.Not too bad now.
1. Clean up the #defines for the max block size. Currently,
there are at least four references to 8192...Think I found and fixed all of these up.
__These includes of storage/bufpage.h can be removed.__
Still _quite_ a few #includes that can be removed throughout. First,
"utils/elog.h" and "util/palloc.h" are include in "postgres.h", so are
unnecessary to include by themselves since "postgres.h" is include in
_every_ .c file, correct?Yes, must be included. Period. Even 3rd party apps.
Also numerous #includes of "storage/bufpage.h" and "storage/fd.h" that are
unnecessary since the things they were included for (BLCKSZ and SEEK_*) are
now either in "config.h" or found in a system include file.2. Once the block size issue is taken care of, calculate the
maximum tuple size more accurately....
3. When #1 & #2 are resolved, let the textual fields have a max
of (MAX_TUPLE_SIZE - sizeof(int)).This could be done as soon as I come up with a way of defining the packet
size for the interfaces since this is the newest limiting factor.Peter's suggestion of backend functions for getting info might be the way to
go. It would let the various interfaces get the info they need and would be
a step towards JDBC and ODBC compliance.Again, we could just set 3rd party apps to be the maximum tuple size we
will ever have to support.--
Bruce Momjian
maillist@candle.pha.pa.us
--
Bruce Momjian
maillist@candle.pha.pa.us