no way in LargeObject API to detect short read?

Anyone able to fix this?

Paul M. Aoki (aoki@acm.org) reports a bug with a severity of 3
The lower the number the more severe it is.

Short Description no way in LargeObject API to detect short
read?

Long Description
org.postgresql.largeobject.LargeObject.read(byte[],int,int)
returns void instead of int. there's therefore no reliable,
non-exceptional way to detect short reads.

the natural way to write blocked-read loops would be to assume
that read(byte[],int,int) returned the number of bytes read or
(e.g.) -1 on EOF.

Sample Code

No file was uploaded with this report

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Anyone able to fix this?

here's a hack we've been using in-house (written by Jun Gabayan,
<jgabayan@pahv.xerox.com>).

you may not like the style but it's a stab at a solution.
--
Paul M. Aoki / Xerox Palo Alto Research Center / 3333 Coyote Hill Road
aoki@acm.org / Computer Science Laboratory / Palo Alto, CA 94304-1314

Index: LargeObject.java
===================================================================
RCS file: /project/placeless/cvsroot/placeless2/src/org/postgresql/largeobject/LargeObject.java,v
retrieving revision 1.1
retrieving revision 1.3
diff -r1.1 -r1.3
64c64,67
<   
---

private int pos = 0; //current position
private int size = 0;

85a89,90

pos = tell();
size = size();

102a108

if(fd == 0) return;

105a112

fd = 0;

118a126,132

// calculate available data to read to avoid reading pass the end
// to avoid an exception
pos = tell();
int avail = size - pos;
if(avail == 0) return null;
if(avail < len) len = avail;
try {

123c137,141
<
---

}catch(SQLException se) {
System.out.println("***LargeObject.read: Caught SQLException: " + se.getMessage());
return null;
}

157c175
< public void read(byte buf[],int off,int len) throws SQLException
---

public int read(byte buf[],int off,int len) throws SQLException

159c177,180
< System.arraycopy(read(len),0,buf,off,len);
---

Paul, I need a context diff. Thanks.

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Anyone able to fix this?

here's a hack we've been using in-house (written by Jun Gabayan,
<jgabayan@pahv.xerox.com>).

you may not like the style but it's a stab at a solution.
--
Paul M. Aoki / Xerox Palo Alto Research Center / 3333 Coyote Hill Road
aoki@acm.org / Computer Science Laboratory / Palo Alto, CA 94304-1314

Index: LargeObject.java
===================================================================
RCS file: /project/placeless/cvsroot/placeless2/src/org/postgresql/largeobject/LargeObject.java,v
retrieving revision 1.1
retrieving revision 1.3
diff -r1.1 -r1.3
64c64,67
<   
---

private int pos = 0; //current position
private int size = 0;

85a89,90

pos = tell();
size = size();

102a108

if(fd == 0) return;

105a112

fd = 0;

118a126,132

// calculate available data to read to avoid reading pass the end
// to avoid an exception
pos = tell();
int avail = size - pos;
if(avail == 0) return null;
if(avail < len) len = avail;
try {

123c137,141
<
---

}catch(SQLException se) {
System.out.println("***LargeObject.read: Caught SQLException: " + se.getMessage());
return null;
}

157c175
< public void read(byte buf[],int off,int len) throws SQLException
---

public int read(byte buf[],int off,int len) throws SQLException

159c177,180
< System.arraycopy(read(len),0,buf,off,len);
---

byte mybuf[] = read(len);
int sz = (mybuf != null) ? mybuf.length : -1; //must return -1 for end of data
if(sz > 0) System.arraycopy(mybuf,0,buf,off,sz);
return sz;

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Quoting pgsql-bugs@postgresql.org:

Paul M. Aoki (aoki@acm.org) reports a bug with a severity of 3
The lower the number the more severe it is.

Short Description
no way in LargeObject API to detect short read?

Will check to see how libpq (which this API is based on) handles this. I've got
on my list of Saturday items the stream support, which will need this.

Peter

Long Description
org.postgresql.largeobject.LargeObject.read(byte[],int,int) returns void
instead of int. there's therefore no reliable, non-exceptional way to
detect short reads.

the natural way to write blocked-read loops would be to assume that
read(byte[],int,int) returned the number of bytes read or (e.g.) -1 on
EOF.

Sample Code

No file was uploaded with this report

--
Peter Mount peter@retep.org.uk
PostgreSQL JDBC Driver: http://www.retep.org.uk/postgres/
RetepPDF PDF library for Java: http://www.retep.org.uk/pdf/

Hmmm, what's the performance issues with this? Is there going to be a problem
with very large LargeObject's?

Quoting "Paul M. Aoki" <aoki@acm.org>:

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Anyone able to fix this?

here's a hack we've been using in-house (written by Jun Gabayan,
<jgabayan@pahv.xerox.com>).

you may not like the style but it's a stab at a solution.
--
Paul M. Aoki / Xerox Palo Alto Research Center / 3333 Coyote Hill Road
aoki@acm.org / Computer Science Laboratory / Palo Alto, CA
94304-1314

Index: LargeObject.java
===================================================================
RCS file:
/project/placeless/cvsroot/placeless2/src/org/postgresql/largeobject/LargeObje

ct.java,v

retrieving revision 1.1
retrieving revision 1.3
diff -r1.1 -r1.3
64c64,67
<
---

private int pos = 0; //current position
private int size = 0;

85a89,90

pos = tell();
size = size();

102a108

if(fd == 0) return;

105a112

fd = 0;

118a126,132

// calculate available data to read to avoid reading pass the

end

// to avoid an exception
pos = tell();
int avail = size - pos;
if(avail == 0) return null;
if(avail < len) len = avail;
try {

123c137,141
<
---

}catch(SQLException se) {
System.out.println("***LargeObject.read: Caught

SQLException: " + se.getMessage());

return null;
}

157c175
< public void read(byte buf[],int off,int len) throws SQLException
---

public int read(byte buf[],int off,int len) throws SQLException

159c177,180
< System.arraycopy(read(len),0,buf,off,len);
---

byte mybuf[] = read(len);
int sz = (mybuf != null) ? mybuf.length : -1; //must return -1 for

end of data

if(sz > 0) System.arraycopy(mybuf,0,buf,off,sz);
return sz;

--
Peter Mount peter@retep.org.uk
PostgreSQL JDBC Driver: http://www.retep.org.uk/postgres/
RetepPDF PDF library for Java: http://www.retep.org.uk/pdf/

Peter T Mount <peter@retep.org.uk> writes:

Hmmm, what's the performance issues with this? Is there going to be a problem
with very large LargeObject's?

you could probably be smarter about caching previous tell() state, if
that's what you mean.

jun's hack doesn't actually add any extra buffers or copies.
--
Paul M. Aoki / Xerox Palo Alto Research Center / 3333 Coyote Hill Road
aoki@acm.org / Computer Science Laboratory / Palo Alto, CA 94304-1314

Peter,

Given the current implementation of size(), I don't think this is a good
solution to the problem at hand. Since size() is an expensive call
(especially on large files), using it in this way wouldn't be a
performant solution. Using size() also requires additional round trips
to the database to get this information. There needs to be a better
solution that doesn't require the additional overhead.

thanks,
--Barry

Peter T Mount wrote:

At 15:37 26/01/01 -0800, Barry Lind wrote:

Peter,

Given the current implementation of size(), I don't think this is a good
solution to the problem at hand. Since size() is an expensive call
(especially on large files), using it in this way wouldn't be a
performant solution. Using size() also requires additional round trips
to the database to get this information. There needs to be a better
solution that doesn't require the additional overhead.

That's exactly what I was thinking of. Most of the largeobject API is based
on libpq hence where this came from. I don't think there is
anything available elsewhere for getting the size.

Yes, there needs to be a better way.

Peter

At 01:58 14/11/00 -0500, pgsql-bugs@postgresql.org wrote:

Paul M. Aoki (aoki@acm.org) reports a bug with a severity of 3
The lower the number the more severe it is.

Short Description
no way in LargeObject API to detect short read?

Long Description
org.postgresql.largeobject.LargeObject.read(byte[],int,int) returns void
instead of int. there's therefore no reliable, non-exceptional way to
detect short reads.

the natural way to write blocked-read loops would be to assume that
read(byte[],int,int) returned the number of bytes read or (e.g.) -1 on EOF.

Sample Code

No file was uploaded with this report

This method now returns the number of bytes read. It will be in CVS by
0900GMT tomorrow.

Peter

This is fixed in the current 7.1 snapshot.

Paul M. Aoki (aoki@acm.org) reports a bug with a severity of 3
The lower the number the more severe it is.

Short Description
no way in LargeObject API to detect short read?

Long Description
org.postgresql.largeobject.LargeObject.read(byte[],int,int) returns void instead of int. there's therefore no reliable, non-exceptional way to detect short reads.

the natural way to write blocked-read loops would be to assume that read(byte[],int,int) returned the number of bytes read or (e.g.) -1 on EOF.

Sample Code

No file was uploaded with this report

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026