utf8 COPY DELIMITER?

Andrew Dunstan wrote:

Mark Dilger wrote:

The \COPY command rejects multibyte delimiters. Is this intentional
behavior?

It is certainly a known limitation, and I suspect removing it could add
non-trivial overhead to the input processing.

What is the use case for using such a delimiter?

I'm working on fixing bugs relating to multibyte character encodings. I
wasn't sure whether this was a bug or not. I don't think we should
use the phrasing "COPY delimiter must be a single character" when, in
utf8 land, I did in fact use a single character. We might say "a single
byte", or we might extend the functionality to handle multibyte characters.

mark

Mark Dilger wrote:

The \COPY command rejects multibyte delimiters. Is this intentional
behavior?

It is certainly a known limitation, and I suspect removing it could add
non-trivial overhead to the input processing.

What is the use case for using such a delimiter?

cheers

andrew

Mark Dilger wrote:

Andrew Dunstan wrote:

Mark Dilger wrote:

The \COPY command rejects multibyte delimiters. Is this intentional
behavior?

It is certainly a known limitation, and I suspect removing it could
add non-trivial overhead to the input processing.

What is the use case for using such a delimiter?

I'm working on fixing bugs relating to multibyte character encodings.
I wasn't sure whether this was a bug or not. I don't think we should
use the phrasing "COPY delimiter must be a single character" when, in
utf8 land, I did in fact use a single character. We might say "a
single byte", or we might extend the functionality to handle multibyte
characters.

Doing the latter would be a feature, and so is of course right off the
table for this release. Changing the error messages to be clearer should
be fine.

cheers

andrew

Andrew Dunstan <andrew@dunslane.net> writes:

Mark Dilger wrote:

I'm working on fixing bugs relating to multibyte character encodings.
I wasn't sure whether this was a bug or not. I don't think we should
use the phrasing "COPY delimiter must be a single character" when, in
utf8 land, I did in fact use a single character. We might say "a
single byte", or we might extend the functionality to handle multibyte
characters.

Doing the latter would be a feature, and so is of course right off the
table for this release. Changing the error messages to be clearer should
be fine.

+1 on changing the message: "character" is clearly less correct than "byte"
here.

I doubt that supporting a single multibyte character would be an
interesting extension --- if we wanted to do anything at all there, we'd
just generalize the delimiter to be an arbitrary string. But it would
certainly slow down COPY by some amount, which is an area where you'll
get push-back for performance losses, so you'd need to make a convincing
use-case for it.

regards, tom lane

On looking at the code, there's another issue: the CSV escape and quote
characters are assumed to be the same in client and server encodings,
because they're checked for before we do transcoding. This pretty much
restricts them to be ASCII.

regards, tom lane

On looking at the code, there's another issue: the CSV escape and quote
characters are assumed to be the same in client and server encodings,
because they're checked for before we do transcoding. This pretty much
restricts them to be ASCII.

regards, tom lane

+1.

The message in question should be something like:

"COPY delimiter must be a single ASCII character"
--
Tatsuo Ishii
SRA OSS, Inc. Japan

Tatsuo Ishii <ishii@postgresql.org> writes:

The message in question should be something like:
"COPY delimiter must be a single ASCII character"

If we phrase it like that we should enforce it like that --- ie, reject
high-bit-set characters.

But I'm a bit hesitant to do so, because it actually does work fine to
use a high-bit-set character as a delimiter as long as client and server
encodings are the same LATINx set. We'd be taking away functionality
for European users for no very good reason.

Is it worth going to the trouble of distinguish same-encoding and
different-encoding cases and applying a looser check for the former
case?

regards, tom lane

Tatsuo Ishii wrote:

On looking at the code, there's another issue: the CSV escape and quote
characters are assumed to be the same in client and server encodings,
because they're checked for before we do transcoding. This pretty much
restricts them to be ASCII.

regards, tom lane

+1.

The message in question should be something like:

"COPY delimiter must be a single ASCII character"

New text is:

The single ASCII character that separates columns within each row

Backpatched to 8.2.X.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Tatsuo Ishii <ishii@postgresql.org> writes:

The message in question should be something like:
"COPY delimiter must be a single ASCII character"

If we phrase it like that we should enforce it like that --- ie, reject
high-bit-set characters.

But I'm a bit hesitant to do so, because it actually does work fine to
use a high-bit-set character as a delimiter as long as client and server
encodings are the same LATINx set. We'd be taking away functionality
for European users for no very good reason.

Is it worth going to the trouble of distinguish same-encoding and
different-encoding cases and applying a looser check for the former
case?

I think yes. Seems a good idea.

Even better, however, is fixing the CVS escaping and quoting I
think. Clearly it's a bug.
--
Tatsuo Ishii
SRA OSS, Inc. Japan

On Tue, Apr 17, 2007 at 02:28:18PM -0400, Tom Lane wrote:

I doubt that supporting a single multibyte character would be an
interesting extension --- if we wanted to do anything at all there, we'd
just generalize the delimiter to be an arbitrary string. But it would
certainly slow down COPY by some amount, which is an area where you'll
get push-back for performance losses, so you'd need to make a convincing
use-case for it.

Couldn't we use a fast code path (what we have now) for the case when
the delimiter is a single byte? That would allow for multi-character
delimiters without penalizing those that don't use them.

As for use case, I worked on migrating some stuff out of a MySQL
database a while ago, and having arbitrary string delimiters would have
made life easier.
--
Jim Nasby jim@nasby.net
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)

Jim C. Nasby wrote:

On Tue, Apr 17, 2007 at 02:28:18PM -0400, Tom Lane wrote:

I doubt that supporting a single multibyte character would be an
interesting extension --- if we wanted to do anything at all there, we'd
just generalize the delimiter to be an arbitrary string. But it would
certainly slow down COPY by some amount, which is an area where you'll
get push-back for performance losses, so you'd need to make a convincing
use-case for it.

Couldn't we use a fast code path (what we have now) for the case when
the delimiter is a single byte? That would allow for multi-character
delimiters without penalizing those that don't use them.

As for use case, I worked on migrating some stuff out of a MySQL
database a while ago, and having arbitrary string delimiters would have
made life easier.

The first thing to note is that the COPY code is quite complex and
fragile. Personally, I'd want a heck of a lot of convincing to see it
changed, and your use case looks to me like it would be better handled
by preprocessing using a perl script.

Also, if we accept string delimiters on input, we should also allow them
on output.

cheers

andrew