overhead of "small" large objects
Hi
I'm putting lots of small (~10Kb) chunks of binary seismic data into large
objects in postgres 7.0.2. Basically just arrays of 2500 or so ints that
represent about a minutes worth of data. I put in the data at the rate of
about 1.5Mb per hour, but the disk usage of the database is growing at
about 6Mb per hour! A factor of 4 seems a bit excessive.
Is there significant overhead involoved in using large objects that aren't
very large?
What might I be doing wrong?
Is there a better way to store these chunks?
thanks,
Philip
Philip Crotwell <crotwell@seis.sc.edu> writes:
Is there significant overhead involoved in using large objects that aren't
very large?
Yes, since each large object is a separate table in 7.0.* and before.
The allocation unit for table space is 8K, so your 10K objects chew up
16K of table space. What's worse, each LO table has a btree index, and
the minimum size of a btree index is 16K --- so your objects take 32K
apiece.
That accounts for a factor of 3. I'm not sure where the other 8K went.
Each LO table will require entries in pg_class, pg_attribute, pg_type,
and pg_index, plus the indexes on those tables, but that doesn't seem
like it'd amount to anything close to 8K per LO.
7.1 avoids this problem by keeping all LOs in one big table.
regards, tom lane
Is there significant overhead involoved in using large objects that
aren't very large?Yes, since each large object is a separate table in 7.0.* and before.
The allocation unit for table space is 8K, so your 10K objects chew up
16K of table space. What's worse, each LO table has a btree index, and
the minimum size of a btree index is 16K --- so your objects take 32K
apiece.That accounts for a factor of 3. I'm not sure where the other 8K went.
Each LO table will require entries in pg_class, pg_attribute, pg_type,
and pg_index, plus the indexes on those tables, but that doesn't seem
like it'd amount to anything close to 8K per LO.7.1 avoids this problem by keeping all LOs in one big table.
Or you can use my patch for the same functionality in 7.0.x.
You can get it at: http://www.perchine.com/dyp/pg/
--
Sincerely Yours,
Denis Perchine
----------------------------------
E-Mail: dyp@perchine.com
HomePage: http://www.perchine.com/dyp/
FidoNet: 2:5000/120.5
----------------------------------