charset/collation in values
I've looked into storing charset/collation in the string values. This
means that we change varchar/text/BpChar to be structures that have a
charset oid field and a collation oid field, the rest of the Datum is the
string data.
Coercability I think one don't need to put in the Datum and it can be
stored in the Nodes. Charset/Collation need to be in the Datum since we
send that into functions as arguments.
Since we are changing what's stored in the Datum and the normal code saves
that on disk then we will end up with charset/collation stored on disk for
each value. If we want to avoid storing charset/collation both in the
column type and in each row, we would need an extra layer that transforms
the Datums before they are stored. As a first implementation it's easier
to just store everything.
For each type we need to have convertion functions to and from strings.
Any suggestion of how to represent these as strings now when it's a string
plus two oid's? This is a though one..
I have more comments/questions later on, but these are enough for one
mail.
--
/Dennis Bj�rklund
Dennis Bjorklund wrote:
I've looked into storing charset/collation in the string values. This
means that we change varchar/text/BpChar to be structures that have a
charset oid field and a collation oid field, the rest of the Datum is the
string data.
I think the number of charset/collation combinations will be relatively
few so perhaps it would be space efficient to maintain a table where
each combination is given an oid and have string values store that
rather than two separate oid's?
Regards,
Thomas Hallgren
Thomas Hallgren <thhal@mailblocks.com> writes:
I think the number of charset/collation combinations will be relatively
few so perhaps it would be space efficient to maintain a table where
each combination is given an oid and have string values store that
rather than two separate oid's?
In fact, we should do our best to get the overhead down to 1 or 2 bytes.
Two OIDs (8 bytes) is ridiculous.
I'm not sure if 1 byte is enough or not --- there might be more than 256
charsets/collations to support. 2 ought to be plenty though.
regards, tom lane
On Mon, 1 Nov 2004, Tom Lane wrote:
I think the number of charset/collation combinations will be relatively
few so perhaps it would be space efficient to maintain a table where
each combination is given an oid and have string values store that
rather than two separate oid's?In fact, we should do our best to get the overhead down to 1 or 2 bytes.
Two OIDs (8 bytes) is ridiculous.
Just to be clear, we don't want to store it on disk no matter what since
it should be enough to store it once for each column. As a first solution
we could store it just to keep it simple until we have tried it out.
--
/Dennis Bj�rklund
On Mon, 1 Nov 2004, Tom Lane wrote:
I think the number of charset/collation combinations will be relatively
few so perhaps it would be space efficient to maintain a table where
each combination is given an oid and have string values store that
rather than two separate oid's?In fact, we should do our best to get the overhead down to 1 or 2 bytes.
Two OIDs (8 bytes) is ridiculous.Just to be clear, we don't want to store it on disk no matter what since
it should be enough to store it once for each column. As a first solution
we could store it just to keep it simple until we have tried it out.
Right. AFAIK nobody has proposed charsets/collations onto disk.
--
Tatsuo Ishii
Tatsuo Ishii wrote:
Right. AFAIK nobody has proposed charsets/collations onto disk.
--
My apologies in that case. I triggered on Dennis wording "If we want to
avoid storing charset/collation both in the column type and in each row,
we would need an extra layer that transforms the Datums before they are
stored. As a first implementation it's easier to just store everything."
Regards,
Thomas Hallgren
Tatsuo Ishii <t-ishii@sra.co.jp> writes:
Right. AFAIK nobody has proposed charsets/collations onto disk.
Oh?
Personally, I'd much sooner eat those few bytes than try to impose a
regime where in-memory representation is different from on-disk.
regards, tom lane
Am Montag, 1. November 2004 07:41 schrieb Dennis Bjorklund:
For each type we need to have convertion functions to and from strings.
Any suggestion of how to represent these as strings now when it's a string
plus two oid's? This is a though one..
A collation implies a character set, so you only need to store one piece of
information anyway.
--
Peter Eisentraut
http://developer.postgresql.org/~petere/
On Tue, 2 Nov 2004, Peter Eisentraut wrote:
A collation implies a character set, so you only need to store one piece of
information anyway.
No, a collation implies a character repertoire like UCS (unicode), it can
apply to several character sets like UTF8 and UTF16.
One can enumerate all combinations if one want to, as suggested
previously.
--
/Dennis Bj�rklund
Am Montag, 1. November 2004 07:41 schrieb Dennis Bjorklund:
For each type we need to have convertion functions to and from strings.
Any suggestion of how to represent these as strings now when it's a string
plus two oid's? This is a though one..A collation implies a character set, so you only need to store one piece of
information anyway.
In my understanding the relation between charset and collation is
1:N. Thus storing only a collation is sufficient to determine the
charset. However a charset cannot determine a collation.
--
Tatsuo Ishii
Am Dienstag, 2. November 2004 13:15 schrieb Dennis Bjorklund:
On Tue, 2 Nov 2004, Peter Eisentraut wrote:
A collation implies a character set, so you only need to store one piece
of information anyway.No, a collation implies a character repertoire like UCS (unicode), it can
apply to several character sets like UTF8 and UTF16.
For the theoretical specification of a collation, it might suffice to know the
character repertoire. But I think in practice, the implementation of a
collation will require knowing the specific character encoding.
--
Peter Eisentraut
http://developer.postgresql.org/~petere/
Am Dienstag, 2. November 2004 13:53 schrieb Tatsuo Ishii:
In my understanding the relation between charset and collation is
1:N. Thus storing only a collation is sufficient to determine the
charset. However a charset cannot determine a collation.
Exactly.
--
Peter Eisentraut
http://developer.postgresql.org/~petere/
On Tue, 2 Nov 2004, Peter Eisentraut wrote:
For the theoretical specification of a collation, it might suffice to
know the character repertoire. But I think in practice, the
implementation of a collation will require knowing the specific
character encoding.
The named entity that is called a collation works for a character
repertoire. It would need to handle different charsets for that repertoire
of course. So there would be one collation called say ucs_sv and not
utf8_sv, utf16_sv, utf32_sv.
Anyway, this is not a problem.
--
/Dennis Bj�rklund
Dennis Bjorklund wrote:
The named entity that is called a collation works for a character
repertoire. It would need to handle different charsets for that
repertoire of course. So there would be one collation called say
ucs_sv and not utf8_sv, utf16_sv, utf32_sv.
Again, theoretically, this might work, but I doubt that this is a
practical implementation. Moreover, since Unicode is more or less the
only chararacter repertoire that have more than one encoding in use,
and neither UTF-16 nor UTF-32 can be used inside the PostgreSQL server
(embedded zero bytes etc.), this is really a nonissue.
--
Peter Eisentraut
http://developer.postgresql.org/~petere/
Dennis Bjorklund wrote:
The named entity that is called a collation works for a character
repertoire. It would need to handle different charsets for that
repertoire of course. So there would be one collation called say
ucs_sv and not utf8_sv, utf16_sv, utf32_sv.Again, theoretically, this might work, but I doubt that this is a
practical implementation. Moreover, since Unicode is more or less the
only chararacter repertoire that have more than one encoding in use,
and neither UTF-16 nor UTF-32 can be used inside the PostgreSQL server
(embedded zero bytes etc.), this is really a nonissue.
I agree with Peter.
--
Tatsuo Ishii