[QUESTIONS] lo_write cannot > 640Kb? memory leaks?

Hi all,

Can anybody pinpoint where is wrong on lo_write(and possibly
lo_read)? I tried to lo_import(and lo_write) few MegaBytes of data
with large_object but it was painfully SLOW! So I simplified the
problem, just test "testlo.c" under test/examples directory

with few MegaBytes of file X.dat

and printout debug messages like this with lo_import ()->
importFile() in main()

IT CANNOT LO_WRITE > 640KB !!! WHY?

testlo.c-------------------
...
in importFile()
{
...
// with BUFSIZE -> 1024 or 8*1024 or 64*1024 i.e. 1K, 8K, 64K, ...
/*
* read in from the Unix file and write to the inversion file
*/
while ((nbytes = read(fd, buf, BUFSIZE)) > 0)
{
printf("COUNT [%d] %d Kb\n", ++count, count*BUFSIZE/1024);
tmp = lo_write(conn, lobj_fd, buf, nbytes);
...
if (tmp < nbytes)
{
fprintf(stderr, "error while reading \"%s\"", filename);
}
printf("READ %d , WRITE %d\n", nbytes, tmp);
_________________________

./testlo template1 X.dat out

....
importing file "in" ...
COUNT [1] 0 Kb
READ 65536 , WRITE 65536
COUNT [2] 64 Kb
...
READ 65536 , WRITE 65536
COUNT [9] 512 Kb
error while reading "in"READ 65536 , WRITE -1
COUNT [10] 576 Kb
error while reading "in"READ 65536 , WRITE -1
COUNT [11] 640 Kb
________________________________________________

and "strace ./testlo template1 in out" shows

...
write(3, "F \0", 3) = 3
write(3, "\0\0\3\273\0\0\0\2\0\0\0\4\0\0\0"..., 1024) = 1024
write(3, "\331\5\200\t\317\t\331]\334\331E"..., 64512) = 64512
write(3, "\213\r\30Lv\10I\211\312\1\322\1\312"..., 20) = 20
read(4, "EERROR: no empty local buffer.\n"..., 1024) = 33
write(2, "error while reading \"in\"", 24error while reading "in") = 24
read(5, "\0\0\0\0\1\310\272\300\276\345\t"..., 65536) = 65536
write(1, "COUNT [10] 576 Kb\n", 20COUNT [10] 576 Kb
) = 20
write(3, "F \0", 3) = 3
write(3, "\0\0\3\273\0\0\0\2\0\0\0\4\0\0\0"..., 1024) = 1024
write(3, "\'j\4j\4h\4\273j\10\241\30Lv\10H"..., 64512) = 64512
write(3, "\331\311\331]\370\331E\370\336\311"..., 20) = 20
read(4, "EERROR: cannot read block 62 of"..., 1024) = 45
write(2, "error while reading \"in\"", 24error while reading "in") = 24
read(5, "u5hp\366v\10\350pk\374\377\203\304"..., 65536) = 65536
write(1, "COUNT [11] 640 Kb\n", 20COUNT [11] 640 Kb
...
___________________________________________________________

So I lo_open/lo_lseek(SEEK_END)/lo_write/lo_close() INSIDE of the loop,
it solved the problem, which means MEM LEAKS or buffer size problem?
And just
lobj_fd = lo_open(conn, lobjId, INV_WRITE| INV_READ);
...
/*
* read in from the Unix file and write to the inversion file
*/
while ((nbytes = read(fd, buf, BUFSIZE)) > 0)
{
printf("COUNT [%d] %d Kb\n", ++count, count*BUFSIZE/1024);
o_lseek(conn, lobj_fd, 0, SEEK_END);
tmp = lo_write(conn, lobj_fd, buf, nbytes);
...

Also solved the problem... But I want reliable(lo_read/lo_write) large
object with BIG data.

Please Help me out.

C.S.Park

Have you tried increasing the number of postgres buffer cache buffers? I am
speculating that the lo_write() is using these buffers but perhaps not
flushing them. As the default configuration has way too few buffers anyway
this might be the problem. Could you try increasing the buffers to say 1024 or
so and rerun your test and post the results?

Thanks,
-dg

David Gould dg@illustra.com 510.628.3783 or 510.305.9468
Informix Software (No, really) 300 Lakeside Drive Oakland, CA 94612
"Of course, someone who knows more about this will correct me if I'm wrong,
and someone who knows less will correct me if I'm right."
--David Palmer (palmer@tybalt.caltech.edu)

Hi,

Have you tried increasing the number of postgres buffer cache buffers? I am
speculating that the lo_write() is using these buffers but perhaps not
flushing them. As the default configuration has way too few buffers anyway
this might be the problem. Could you try increasing the buffers to say 1024 or
so and rerun your test and post the results?

-dg

Thanks for your reply, but as I posted I tested many different size of
buffers in lotest.c, e.g. 1Kb, 4Kb, 8Kb, 9Kb, 10Kb, 16Kb, 32Kb, 64Kb, 640Kb,
1Mb, 10Mb, ... or your "cache" or "buffer" size have some diffferent meaning?

but lo_write cannot exceed 640kb always, only open/write(less than
640kb)/close looping OR open LOOP lo_write(buffer<640Kb)) , lo_lseek to
SEEK_END, write, ... seems to solve this problem(as in my last email), and I
found that 16Kb buffer seems to be the optimal buffer size for the SPEED(not
8Kb) for network & socket connections why?...

and I think that above lo_lseek write loop is not a real solution, because
lo_read seems does not have such defects(but 16kb buffer seems to be the
optimal size also), there must be some problems in lo_write or buffer
managing... does anybody have some guess or fixes?

c.s.park

Hi,

Have you tried increasing the number of postgres buffer cache buffers? I am
speculating that the lo_write() is using these buffers but perhaps not
flushing them. As the default configuration has way too few buffers anyway
this might be the problem. Could you try increasing the buffers to say 1024 or
so and rerun your test and post the results?

-dg

Thanks for your reply, but as I posted I tested many different size of
buffers in lotest.c, e.g. 1Kb, 4Kb, 8Kb, 9Kb, 10Kb, 16Kb, 32Kb, 64Kb, 640Kb,
1Mb, 10Mb, ... or your "cache" or "buffer" size have some diffferent meaning?

Yes, my "cache" or "buffer" size has a different meaning. I am referring to
the size of the postgres buffer pool which can be changed by a command line
option when you start the postmaster. I forget what the exact option is but
it is something like "-o -B <number_of_buffers>". It is documented I think.

-dg

David Gould dg@illustra.com 510.628.3783 or 510.305.9468
Informix Software (No, really) 300 Lakeside Drive Oakland, CA 94612
"Of course, someone who knows more about this will correct me if I'm wrong,
and someone who knows less will correct me if I'm right."
--David Palmer (palmer@tybalt.caltech.edu)