storing binary data

Jason,

BLOBs as you have correctly inferred do not get automatically deleted.
You can add triggers to your tables to delete them automatically if you
so desire.

However 'bytea' is the datatype that is most appropriate for your needs.
It has been around for a long time, but not well documented. I have
been using it in my code since 7.0 of postgres and it works fine. In
fact many of the internal postgres tables use it.

The problem with bytea is that many of the client interfaces don't
support it well or at all. So depending on how you intend to access the
data you may not be able to use the bytea datatype. The situation is
much improved in 7.2 with bytea documented and better support for it in
the client interfaces (jdbc especially).

Encoding the data into a text format will certainly work, if you can't
work around the current limitations of the above two options. And I
believe there is some contrib code to help in this area.

thanks,
--Barry

Jason Orendorff wrote:

"Jason Orendorff" <jason@jorendorff.com> writes:

Reply-To: sender

Just to be nice, I'll do this. ;)

Hi. I was surprised to discover today that postgres's
character types don't support zero bytes. That is,
Postgres isn't 8-bit clean. Why is that?

As I understand it, the storage system itself is 8-bit clean; it's the
parser layer that isn't (as it uses C strings everywhere).

More to the point, I need to store about 1k bytes per row
of varying-length 8-bit binary data. I have a few options:

+ BLOBs. PostgreSQL BLOBs make me nervous. I worry about
the BLOB not being deleted when the corresponding row in
the table is deleted. The documentation is vague.

This is an issue. There is definitely no automatic deletion of
LOs. There is a 'vacuumlo' program in contrib/ that may be useful, or
you can roll your own, or you can use triggers to make sure LOs get
deleted.

FWIW, I've been using LOBs in a couple of applications and haven't had
too much trouble.

+ What I really need is a binary *short* object type.
I have heard rumors of a legendary "bytea" type that might
help me, but it doesn't appear to be documented anywhere,
so I hesitate to use it.

It is in 7.1, but is more fully documented in 7.2 (which is entering
beta). See:

http://candle.pha.pa.us/main/writings/pgsql/sgml/datatype-binary.html

+ I can base64-encode the data and store it in a "text"
field. But postgres is a great big data-storage system;
surely it can store binary data without resorting to
this kind of hack.

Since the only way to store or retrieve non-LOB data is to go through
the SQL parser, you always have to do some escaping. The link above
tells you how to do it for 'bytea' without having to go the base64
route.

-Doug
--
Let us cross over the river, and rest under the shade of the trees.
--T. J. Jackson, 1863

Use bytea. Search archives.

On Sun, 21 Oct 2001, Jason Orendorff wrote:

"Jason Orendorff" <jason@jorendorff.com> writes:

Hi. I was surprised to discover today that postgres's
character types don't support zero bytes. That is,
Postgres isn't 8-bit clean. Why is that?

(a) because all our datatype I/O interfaces are based on C-style
(null terminated) strings

(b) because comparison of character datatypes is based on strcoll()
(at least if you compiled with locale support)

Fixing either of these is far more pain than is justified to allow
people to store non-textual data in textual datatypes. I don't foresee
it happening.

+ What I really need is a binary *short* object type.
I have heard rumors of a legendary "bytea" type that might
help me, but it doesn't appear to be documented anywhere,
so I hesitate to use it.

It's real and it's not going away. It is pretty poorly documented
and doesn't have a wide variety of functions ... but hey, you can help
improve that situation. This is an open source project after all ;-)

regards, tom lane

+ What I really need is a binary *short* object type.
I have heard rumors of a legendary "bytea" type that might
help me, but it doesn't appear to be documented anywhere,
so I hesitate to use it.

It's real and it's not going away. It is pretty poorly documented
and doesn't have a wide variety of functions ... but hey, you can help
improve that situation. This is an open source project after all ;-)

regards, tom lane

I'll take a shot at improving the documentation for bytea. I'm hoping
documentation patches are accepted during beta though ;-)

Also, FWIW, 7.2 includes bytea support for LIKE, NOT LIKE, LIKE ESCAPE,
||, trim(), substring(), position(), length(), indexing, and various
comparators.

Joe

...

I'll take a shot at improving the documentation for bytea. I'm hoping
documentation patches are accepted during beta though ;-)

Always. At least up until a week or so before release, when we need to
firm up the docs and work on final cleanup etc. There are several
announcements leading up to that point, so it will not be a suprise.

- Thomas

Joe Conway <joseph.conway@home.com> writes:

I'll take a shot at improving the documentation for bytea. I'm hoping
documentation patches are accepted during beta though ;-)

Of course. The only limitation we place during beta is "no new features
added". I plan to spend a good deal of time on the docs during beta
myself.

regards, tom lane

Jason Orendorff writes:

Hi. I was surprised to discover today that postgres's
character types don't support zero bytes. That is,
Postgres isn't 8-bit clean. Why is that?

PostgreSQL is 8-bit clean. The character types don't support zero bytes
because the character types store characters, not bytes.

--
Peter Eisentraut peter_e@gmx.net http://funkturm.homeip.net/~peter

Quick question - I couldn't find this in the docs:

What exactly is the advantage in using VIEWs? I get the impression that the
SELECT query it is based on is cached (ie. a cached query plan).

But, is this cached between db restarts, between connections, etc. Is it
cached upon the first use of the view for a db instance for a particular
connection, etc?

Chris

"Christopher Kings-Lynne" <chriskl@familyhealth.com.au> writes:

What exactly is the advantage in using VIEWs?

A level of logical indirection between the application and the physical
data schema. There are no performance benefits.

I get the impression that the
SELECT query it is based on is cached (ie. a cached query plan).

Nope. If there's something in the docs that makes you think so,
point out so I can fix it ;-)

regards, tom lane

I get the impression that the
SELECT query it is based on is cached (ie. a cached query plan).

Nope. If there's something in the docs that makes you think so,
point out so I can fix it ;-)

Hmmm...I could have sworn that you mentioned in passing something about
cached query plans and VIEWs - I must have been in dream land.

Chris

I'll take a shot at improving the documentation for bytea. I'm hoping
documentation patches are accepted during beta though ;-)

Also, FWIW, 7.2 includes bytea support for LIKE, NOT LIKE, LIKE ESCAPE,
||, trim(), substring(), position(), length(), indexing, and various
comparators.

Cool!

Would it be practical to use substring for retrieving chunks of binary data
in manageable sizes? Or would the overheads be too high?

Cheerio,
Link.

Christopher Kings-Lynne wrote:

What exactly is the advantage in using VIEWs? I get the impression that the
SELECT query it is based on is cached (ie. a cached query plan).

I had the same impression but I've been told (with explanations) that
the query plan for a view is not cached in any way.

--
Alessio F. Bragadini alessio@albourne.com
APL Financial Services http://village.albourne.com
Nicosia, Cyprus phone: +357-2-755750

"It is more complicated than you think"
-- The Eighth Networking Truth from RFC 1925

Lincoln Yeoh wrote:

Also, FWIW, 7.2 includes bytea support for LIKE, NOT LIKE, LIKE ESCAPE,
||, trim(), substring(), position(), length(), indexing, and various
comparators.

Cool!

Would it be practical to use substring for retrieving chunks of binary data
in manageable sizes? Or would the overheads be too high?

Cheerio,
Link.

I haven't done any performance testing, but it should be no different
than the substring function used on TEXT fields. Try it out and let us
know ;-)

-- Joe