Large queries - again...

Well, I tested the query you sent, and I got these results accessing the
data:

1) libpq from Windows (freshly compiled from 7.1.2 sources): Error:
pqReadData() -- read() failed: errno=0
No error
2) ODBC from Windows: It works ok.

Steve Howe <howe@carcass.dhs.org> escreveu nas not�cias de
mensagem:9j8jce$ddo$1@news.tht.net...

Hello all,

I've tried again sending large queries using libpq on Windows
environment, without success.
I downloaded the PostgreSQL v7.12 sources, compiled libpq.dll

using

Microsoft's Visual C++ 6.0, and tried sending a large query.
The problem is, when the query is > 8192 large, a NULL pointer is
returned from PQexec().
I have tried using ZDE (http://www.zeoslib.org), which I helped
develop, and pgAccess. ZDE is based on the Zeos Database Objects library,
which provides full access to PostgreSQL to Borland Delphi and Borland C++
Builder compilers.
Could anyone please try this query:
ftp://carcass.dhs.org/pub/test.zip on windows (using libpq) and confirm it
suceed ? This archive contains a test.sql source, which will create a dumb
table with a text filed and then try to insert in it a large data (>8192
bytes) on it, and the libpq.dll I just compiled, for who want a fresh

libpq

(it's virus free, don't worry... ). All my current PostgreSQL driver
implementation is depending on this. I'm sure the libpq will fail, unless
something very weird is happening in here... :)
Other friends have confirmed this behaviour.
I tried to look at the libpq sources to find out where's the

error,

"Steve Howe" <howe@carcass.dhs.org> writes:

I downloaded the PostgreSQL v7.12 sources, compiled libpq.dll using
Microsoft's Visual C++ 6.0, and tried sending a large query.
The problem is, when the query is > 8192 large, a NULL pointer is
returned from PQexec().

It sure sounds to me like you are invoking an old (6.5 or before) libpq.
Perhaps you should check around to see if there are multiple libpq.dll
files on your system ...

regards, tom lane

Hello Tom,

Nope, I'm 100% sure that the libpq.dll used is the one I just
compiled. And I never installed an older libpq.dll on this system.
My application loads specifically the libpq.dll I compiled (I use
the full library path on the call to LoadLibrary() call, so there is no way
it is an older library.
Eduardo Stern from dbExperts (http://www.dbexperts.com.br), a
PostgreSQL specialized consulting company, got the same results.
The ODBC driver, however, do not suffer from the same problem, once
it does not use libpq.
Maybe I should write a C sample that proves libpq under windows has
this bug ???

Best Regards,
Steve Howe

----- Original Message -----
From: "Tom Lane" <tgl@sss.pgh.pa.us>
To: "Steve Howe" <howe@carcass.dhs.org>
Cc: <pgsql-hackers@postgresql.org>
Sent: Friday, July 20, 2001 12:35 PM
Subject: Re: [HACKERS] Large queries - again...

"Steve Howe" <howe@carcass.dhs.org> writes:

I downloaded the PostgreSQL v7.12 sources, compiled libpq.dll

using

Microsoft's Visual C++ 6.0, and tried sending a large query.
The problem is, when the query is > 8192 large, a NULL pointer

is

"Steve Howe" <howe@carcass.dhs.org> writes:

Nope, I'm 100% sure that the libpq.dll used is the one I just
compiled. And I never installed an older libpq.dll on this system.

Hmph. So what is left in PQerrorMessage() after the failure?

regards, tom lane

Hello Tom,

It returns "Error: pqReadData() -- read() failed: errno=0 No error
" as expected when a nil pointer is returned.

Best Regards,
Steve Howe

----- Original Message -----
From: "Tom Lane" <tgl@sss.pgh.pa.us>
To: "Steve Howe" <howe@carcass.dhs.org>
Cc: <pgsql-hackers@postgresql.org>
Sent: Friday, July 20, 2001 2:42 PM
Subject: Re: [HACKERS] Large queries - again...

"Steve Howe" <howe@carcass.dhs.org> writes:

It returns "Error: pqReadData() -- read() failed: errno=0 No error
" as expected when a nil pointer is returned.

"As expected"? That's not what I'd expect, especially not for a
behavior that's dependent on the size of an *outgoing* message.

(Thinks for awhile...) You're not using PQsetnonblocking() are you,
by any chance?

regards, tom lane

----- Original Message -----
From: "Tom Lane" <tgl@sss.pgh.pa.us>
To: "Steve Howe" <howe@carcass.dhs.org>
Cc: <pgsql-hackers@postgresql.org>
Sent: Friday, July 20, 2001 5:17 PM
Subject: Re: [HACKERS] Large queries - again...

"Steve Howe" <howe@carcass.dhs.org> writes:

It returns "Error: pqReadData() -- read() failed: errno=0 No

error

" as expected when a nil pointer is returned.

"As expected"? That's not what I'd expect, especially not for a
behavior that's dependent on the size of an *outgoing* message.

It is expected, because it's the default message when a PQexec() query
returns NULL: pqReadData() will return nothing yet no error is signed.
Of course, the "really expected" would be a sucessfull exec :-)

(Thinks for awhile...) You're not using PQsetnonblocking() are you,
by any chance?

No, I'm not. Asynchronous libpq connections on Windows are still not
realiable (althought I read someone submitted a patch recently), so I'm
keeping synchronous queries for a while. I'm not also using any non-standard
functions; just plain PQconnectdb() and PQexec()...

Best Regards,
Steve

"As expected"? That's not what I'd expect, especially not for a
behavior that's dependent on the size of an *outgoing* message.

It is expected, because it's the default message when a PQexec() query
returns NULL: pqReadData() will return nothing yet no error is signed.
Of course, the "really expected" would be a sucessfull exec :-)

(Thinks for awhile...) You're not using PQsetnonblocking() are you,
by any chance?

No, I'm not. Asynchronous libpq connections on Windows are still not
realiable (althought I read someone submitted a patch recently), so I'm
keeping synchronous queries for a while. I'm not also using any non-standard
functions; just plain PQconnectdb() and PQexec()...

Yes, just applied. I will have another one next week.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

"Steve Howe" <howe@carcass.dhs.org> writes:

(Thinks for awhile...) You're not using PQsetnonblocking() are you,
by any chance?

No, I'm not.

Drat, another perfectly good theory down the drain :-(.

Well, we're not going to find out anymore until we discover what the
error code actually is --- the "errno=0" bogosity isn't helping.
As Bruce mentioned, we did just commit a patch that #defines errno
as WSAGetLastError() on WIN32, so that you can get at least something
useful about socket errors. I'd suggest pulling the current CVS sources
(or a nightly snapshot tarball dated after today) and building libpq
from that. Then maybe we can learn more.

regards, tom lane

"Steve Howe" <howe@carcass.dhs.org> writes:

(Thinks for awhile...) You're not using PQsetnonblocking() are you,
by any chance?

No, I'm not.

Drat, another perfectly good theory down the drain :-(.

Well, we're not going to find out anymore until we discover what the
error code actually is --- the "errno=0" bogosity isn't helping.
As Bruce mentioned, we did just commit a patch that #defines errno
as WSAGetLastError() on WIN32, so that you can get at least something
useful about socket errors. I'd suggest pulling the current CVS sources
(or a nightly snapshot tarball dated after today) and building libpq
from that. Then maybe we can learn more.

Unhappyly, there are unresolved externals and it won't build...
I'll try to fix it.
The log follows right below...

Best regards,
Steve Howe

----------------------------------------------------------------------------
----------
Microsoft (R) Program Maintenance Utility Version 6.00.8168.0
Copyright (C) Microsoft Corp 1988-1998. All rights reserved.

cd include
if not exist config.h copy config.h.win32 config.h
cd ..
cd interfaces\libpq
nmake /f win32.mak

Microsoft (R) Program Maintenance Utility Version 6.00.8168.0
Copyright (C) Microsoft Corp 1988-1998. All rights reserved.

link.exe @C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\nma01588.
Creating library .\Release\libpqdll.lib and object .\Release\libpqdll.exp
libpq.lib(fe-exec.obj) : error LNK2001: unresolved external symbol _snprintf
libpq.lib(fe-misc.obj) : error LNK2001: unresolved external symbol _snprintf
libpq.lib(fe-auth.obj) : error LNK2001: unresolved external symbol _snprintf
libpq.lib(dllist.obj) : error LNK2001: unresolved external symbol _elog
.\Release\libpq.dll : fatal error LNK1120: 2 unresolved externals
NMAKE : fatal error U1077: 'link.exe' : return code '0x460'
Stop.
NMAKE : fatal error U1077: '"C:\Program Files\Microsoft Visual
Studio\VC98\bin\NMAKE.EXE"' : return
code '0x2'
Stop.

OK, I just applied a patch to add the final fixes to Win32 libpq.
Please try the CVS or later snapshot to see how it works. The patch
suggested adding

#define snprintf _snprintf

to win32.h and I have done that. There was already one there for
vsnprintf. I am quite confused about the elog() mention. I don't see
where we added a call to elog() in the past day. I only see two
mentions of elog in the code, both it dllist.c. They don't use elog()
if you define FRONTEND. Please do -DFRONTEND on the compile line. I
think this will give you a good library binary.

Let us know how the new code works. The most recent patch I just
applied was tested by a user and it worked well for him. Nice to have
this resolved. I can mark this TODO item as done:

* -Fix libpq to properly handle socket failures under native MS
Win32 [libpq]

"Steve Howe" <howe@carcass.dhs.org> writes:

(Thinks for awhile...) You're not using PQsetnonblocking() are you,
by any chance?

No, I'm not.

Drat, another perfectly good theory down the drain :-(.

Well, we're not going to find out anymore until we discover what the
error code actually is --- the "errno=0" bogosity isn't helping.
As Bruce mentioned, we did just commit a patch that #defines errno
as WSAGetLastError() on WIN32, so that you can get at least something
useful about socket errors. I'd suggest pulling the current CVS sources
(or a nightly snapshot tarball dated after today) and building libpq
from that. Then maybe we can learn more.

Unhappyly, there are unresolved externals and it won't build...
I'll try to fix it.
The log follows right below...

Best regards,
Steve Howe

----------------------------------------------------------------------------
----------
Microsoft (R) Program Maintenance Utility Version 6.00.8168.0
Copyright (C) Microsoft Corp 1988-1998. All rights reserved.

cd include
if not exist config.h copy config.h.win32 config.h
cd ..
cd interfaces\libpq
nmake /f win32.mak

Microsoft (R) Program Maintenance Utility Version 6.00.8168.0
Copyright (C) Microsoft Corp 1988-1998. All rights reserved.

link.exe @C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\nma01588.
Creating library .\Release\libpqdll.lib and object .\Release\libpqdll.exp
libpq.lib(fe-exec.obj) : error LNK2001: unresolved external symbol _snprintf
libpq.lib(fe-misc.obj) : error LNK2001: unresolved external symbol _snprintf
libpq.lib(fe-auth.obj) : error LNK2001: unresolved external symbol _snprintf
libpq.lib(dllist.obj) : error LNK2001: unresolved external symbol _elog
.\Release\libpq.dll : fatal error LNK1120: 2 unresolved externals
NMAKE : fatal error U1077: 'link.exe' : return code '0x460'
Stop.
NMAKE : fatal error U1077: '"C:\Program Files\Microsoft Visual
Studio\VC98\bin\NMAKE.EXE"' : return
code '0x2'
Stop.

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://www.postgresql.org/search.mpl

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

----- Original Message -----
From: "Bruce Momjian" <pgman@candle.pha.pa.us>
To: "Steve Howe" <howe@carcass.dhs.org>
Cc: "Tom Lane" <tgl@sss.pgh.pa.us>; <pgsql-hackers@postgresql.org>
Sent: Saturday, July 21, 2001 1:39 AM
Subject: Re: [HACKERS] Large queries - again...

OK, I just applied a patch to add the final fixes to Win32 libpq.
Please try the CVS or later snapshot to see how it works. The patch
suggested adding

#define snprintf _snprintf

to win32.h and I have done that. There was already one there for
vsnprintf. I am quite confused about the elog() mention. I don't see
where we added a call to elog() in the past day. I only see two
mentions of elog in the code, both it dllist.c. They don't use elog()
if you define FRONTEND. Please do -DFRONTEND on the compile line. I
think this will give you a good library binary.

I did it, but that brings other dependency problems (see below). I think
it's better to properly fix the elog issue... :-)
----------------------------------------------------------------------------
--------------------------------------------
C:\ttt\src>nmake -f win32.mak

Microsoft (R) Program Maintenance Utility Version 6.00.8168.0
Copyright (C) Microsoft Corp 1988-1998. All rights reserved.

cd include
if not exist config.h copy config.h.win32 config.h
cd ..
cd interfaces\libpq
nmake /f win32.mak

Microsoft (R) Program Maintenance Utility Version 6.00.8168.0
Copyright (C) Microsoft Corp 1988-1998. All rights reserved.

cl.exe @C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\nma01700.
dllist.c
..\..\backend\lib\dllist.c(20) : fatal error C1083: Cannot open include
file: 'sysexits.h': No such
file or directory
----------------------------------------------------------------------------
--------------------------------------------

Let us know how the new code works. The most recent patch I just
applied was tested by a user and it worked well for him. Nice to have
this resolved. I can mark this TODO item as done:

* -Fix libpq to properly handle socket failures under native MS
Win32 [libpq]

I want this fixed more then anybody else i the world, believe me :-)

Best Regards,
Steve Howe

"Steve Howe" <howe@carcass.dhs.org> writes:

..\..\backend\lib\dllist.c(20) : fatal error C1083: Cannot open include
file: 'sysexits.h': No such file or directory

Jan added that recently. I was wondering if it was portable or not ...
looks like now we know :-(.

For the moment, just take out the include --- you may also need to
replace "exit(EX_UNAVAILABLE)" by plain "exit(1)".

regards, tom lane

OK, I just applied a patch to add the final fixes to Win32 libpq.
Please try the CVS or later snapshot to see how it works. The patch
suggested adding

#define snprintf _snprintf

to win32.h and I have done that. There was already one there for
vsnprintf. I am quite confused about the elog() mention. I don't see
where we added a call to elog() in the past day. I only see two
mentions of elog in the code, both it dllist.c. They don't use elog()
if you define FRONTEND. Please do -DFRONTEND on the compile line. I
think this will give you a good library binary.

I did it, but that brings other dependency problems (see below). I think
it's better to properly fix the elog issue... :-)

Shouldn't we be defining FRONTEND in the win32.mak file?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tom Lane wrote:

"Steve Howe" <howe@carcass.dhs.org> writes:

..\..\backend\lib\dllist.c(20) : fatal error C1083: Cannot open include
file: 'sysexits.h': No such file or directory

Jan added that recently. I was wondering if it was portable or not ...
looks like now we know :-(.

Grmbl - tell me why I don't like Windows...

Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com

Tom Lane wrote:

"Steve Howe" <howe@carcass.dhs.org> writes:

..\..\backend\lib\dllist.c(20) : fatal error C1083: Cannot open

include

file: 'sysexits.h': No such file or directory

Jan added that recently. I was wondering if it was portable or not ...
looks like now we know :-(.

Grmbl - tell me why I don't like Windows...

Please notify me when it's fixed so that I can test it.
I'll also test the error messages returned from libpq on Windows, as
requested on another thread.

Best Regards,
Steve Howe