pg_restore fails with a custom backup file

Hi. Asaba-san.

From: "Yoshiyuki Asaba"

Win32 does not implement fseeko() and ftello(). So I think it limit to
handle a 2GB file. Is this a specification?

Yes, Magnus-san suggested the problem. It is present TODO. The entire
adjustment was still difficult though I had tried it. SetFilePointer might be
able to be saved. However, I think it might be an attempt of 8.3...

Regards,
Hiroshi Saito

On Fri, Dec 15, 2006 at 12:57:50AM +0900, Hiroshi Saito wrote:

Win32 does not implement fseeko() and ftello(). So I think it limit to
handle a 2GB file. Is this a specification?

Yes, Magnus-san suggested the problem. It is present TODO. The entire
adjustment was still difficult though I had tried it. SetFilePointer might
be able to be saved. However, I think it might be an attempt of 8.3...

I've been looking at a fix for this, and I think I have it. The solution
looks to be to redefine off_t to 64-bit (the standard headers *always*
define it as 32-bit, and there is no way to change that - at least not
that I can find).

I have the fix made for just bin/pg_dump for now (in pg_dump.h), and I'm
testing that. (So far only on MSVC builds)

A question though - is there any *gain* from using 64-bit offsets in the
actual backend? The change could of course be done in port.h, but that
will affect the whole backend (and require a few more functions than
just fseeko/ftello to be redefined) which could have larger
consequences.

So - provided that this works after my test is completed, is the better
place to do this for just pg_dump/pg_restore, or attempt to do it for
the whole backend?

//Magnus

Magnus Hagander wrote:

On Fri, Dec 15, 2006 at 12:57:50AM +0900, Hiroshi Saito wrote:

Win32 does not implement fseeko() and ftello(). So I think it limit to
handle a 2GB file. Is this a specification?

Yes, Magnus-san suggested the problem. It is present TODO. The entire
adjustment was still difficult though I had tried it. SetFilePointer might
be able to be saved. However, I think it might be an attempt of 8.3...

I've been looking at a fix for this, and I think I have it. The solution
looks to be to redefine off_t to 64-bit (the standard headers *always*
define it as 32-bit, and there is no way to change that - at least not
that I can find).

I have the fix made for just bin/pg_dump for now (in pg_dump.h), and I'm
testing that. (So far only on MSVC builds)

A question though - is there any *gain* from using 64-bit offsets in the
actual backend? The change could of course be done in port.h, but that

No, not really. All files are kept < 1gig for the backend. We had code
for that from Berkeley, so we have just kept it.

--
Bruce Momjian bruce@momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Hi.

Oh, your great trust confidence.:-)

I have the fix made for just bin/pg_dump for now (in pg_dump.h), and I'm
testing that. (So far only on MSVC builds)

However, MinGW+gcc be able to be saved?
I was wishing it....

Regards,
Hiroshi Saito

On Mon, Dec 18, 2006 at 09:50:12AM -0500, Bruce Momjian wrote:

Yes, Magnus-san suggested the problem. It is present TODO. The entire
adjustment was still difficult though I had tried it. SetFilePointer might
be able to be saved. However, I think it might be an attempt of 8.3...

I've been looking at a fix for this, and I think I have it. The solution
looks to be to redefine off_t to 64-bit (the standard headers *always*
define it as 32-bit, and there is no way to change that - at least not
that I can find).

I have the fix made for just bin/pg_dump for now (in pg_dump.h), and I'm
testing that. (So far only on MSVC builds)

A question though - is there any *gain* from using 64-bit offsets in the
actual backend? The change could of course be done in port.h, but that

No, not really. All files are kept < 1gig for the backend. We had code
for that from Berkeley, so we have just kept it.

Ok, based on this, here's a patch that *appears* to fix the problem on
Win32. I tried the suggested repro to dump a pgbench database of
appropriate size, and it does restore properly and give me the proper
amount of rows in all the tables. But that's all I've tested so far.

Also, it compiles fine on MSVC. I still haven't managed to get the MingW
build environment working properly on Win64 even for building Win32
apps, so I haven't been able to build it on MingW yet. It *should* work
since it's all standard functions, but might require further hacks. I'll
try to get around to that later, and Dave has promised to give it a
compile-try as well, but if someone wants to test that, please do ;)

So, does it seem reasonable, and the right place to stick it? Oh, and
please do *not* apply until someone confirms it works on mingw!

//Magnus

Bruce Momjian <bruce@momjian.us> writes:

Magnus Hagander wrote:

A question though - is there any *gain* from using 64-bit offsets in the
actual backend? The change could of course be done in port.h, but that

No, not really. All files are kept < 1gig for the backend.

Not so: consider a backend COPY reading or writing a multi-gig table.
This will fail if the platform hasn't got largefile support.

regards, tom lane

Tom Lane wrote:

Bruce Momjian <bruce@momjian.us> writes:

Magnus Hagander wrote:

A question though - is there any *gain* from using 64-bit offsets in the
actual backend? The change could of course be done in port.h, but that

No, not really. All files are kept < 1gig for the backend.

Not so: consider a backend COPY reading or writing a multi-gig table.
This will fail if the platform hasn't got largefile support.

Good point --- but do we do any seeks in COPY files? I don't think so,
so I don't see how we would benefit from large file support there.
Certainly people are dumping >2 gig files.

--
Bruce Momjian bruce@momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian <bruce@momjian.us> writes:

Tom Lane wrote:

Not so: consider a backend COPY reading or writing a multi-gig table.

Good point --- but do we do any seeks in COPY files? I don't think so,

True, so if the problem is limited to whether we can seek or not, then
we don't need to fix the backend. Magnus said something about other
issues though?

regards, tom lane

Tom Lane wrote:

Bruce Momjian <bruce@momjian.us> writes:

Tom Lane wrote:

Not so: consider a backend COPY reading or writing a multi-gig table.

Good point --- but do we do any seeks in COPY files? I don't think so,

True, so if the problem is limited to whether we can seek or not, then
we don't need to fix the backend. Magnus said something about other
issues though?

Yes, MinGW. I think he is researching that.

--
Bruce Momjian bruce@momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Magnus Hagander wrote:

Index: src/bin/pg_dump/pg_dump.h
===================================================================
RCS file: /projects/cvsroot/pgsql/src/bin/pg_dump/pg_dump.h,v
retrieving revision 1.130
diff -c -r1.130 pg_dump.h
*** src/bin/pg_dump/pg_dump.h	9 Oct 2006 23:36:59 -0000	1.130
--- src/bin/pg_dump/pg_dump.h	18 Dec 2006 14:33:16 -0000
***************
*** 16,21 ****
--- 16,33 ----

#include "postgres_fe.h"

+ #ifdef WIN32
+ /*
+  * WIN32 does not provide a 64-bit off_t, but it does provide functions operating
+  * with 64-bit offsets. Redefine off_t to what's always a 64-bit int, and redefine
+  * the functions that accept off_t to be the 64-bit only ones.
+  */
+ #define off_t __int64
+ #undef fseeko
+ #define fseeko(stream, offset, origin) _fseeki64(stream, offset, origin)
+ #undef ftello
+ #define ftello(stream) _ftelli64(stream)
+ #endif

/*
* pg_dump uses two different mechanisms for identifying database objects:

This patch appears to be broken on my MinGW setup:

gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Winline
-Wdeclaration-after-statement -Wendif-labels -fno-strict-aliasing -g
-DFRONTEND -I../../../src/interfaces/libpq -I../.
./../src/include -I./src/include/port/win32 -DEXEC_BACKEND
-I/c/tcl/include "-I../../../src/include/port/win32" -c -o common.o
common.c -MMD -MP -MF .deps/common.Po
In file included from
c:/mingw/bin/../lib/gcc/mingw32/3.4.2/../../../../include/zconf.h:289,
from
c:/mingw/bin/../lib/gcc/mingw32/3.4.2/../../../../include/zlib.h:34,
from pg_backup_archiver.h:44,
from common.c:20:
c:/mingw/bin/../lib/gcc/mingw32/3.4.2/../../../../include/unistd.h:23:
error: conflicting types for 'chsize'
c:/mingw/bin/../lib/gcc/mingw32/3.4.2/../../../../include/io.h:271:
error: previous declaration of 'chsize' was here
c:/mingw/bin/../lib/gcc/mingw32/3.4.2/../../../../include/unistd.h:23:
error: conflicting types for 'chsize'
c:/mingw/bin/../lib/gcc/mingw32/3.4.2/../../../../include/io.h:271:
error: previous declaration of 'chsize' was here

I suspect we might need to create a pg_off_t type or some such gadget.

Bleah.

But it does need to be fixed.

cheers

andrew

Andrew Dunstan wrote:

Magnus Hagander wrote:

Index: src/bin/pg_dump/pg_dump.h
===================================================================
RCS file: /projects/cvsroot/pgsql/src/bin/pg_dump/pg_dump.h,v
retrieving revision 1.130
diff -c -r1.130 pg_dump.h
*** src/bin/pg_dump/pg_dump.h    9 Oct 2006 23:36:59 -0000    1.130
--- src/bin/pg_dump/pg_dump.h    18 Dec 2006 14:33:16 -0000
***************
*** 16,21 ****
--- 16,33 ----
#include "postgres_fe.h"
+ #ifdef WIN32
+ /*
+  * WIN32 does not provide a 64-bit off_t, but it does provide
functions operating
+  * with 64-bit offsets. Redefine off_t to what's always a 64-bit
int, and redefine
+  * the functions that accept off_t to be the 64-bit only ones.
+  */
+ #define off_t __int64
+ #undef fseeko
+ #define fseeko(stream, offset, origin) _fseeki64(stream, offset,
origin)
+ #undef ftello
+ #define ftello(stream) _ftelli64(stream)
+ #endif
/*
* pg_dump uses two different mechanisms for identifying database
objects:

This patch appears to be broken on my MinGW setup:

gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Winline
-Wdeclaration-after-statement -Wendif-labels -fno-strict-aliasing -g
-DFRONTEND -I../../../src/interfaces/libpq -I../.
./../src/include -I./src/include/port/win32 -DEXEC_BACKEND
-I/c/tcl/include "-I../../../src/include/port/win32" -c -o common.o
common.c -MMD -MP -MF .deps/common.Po
In file included from
c:/mingw/bin/../lib/gcc/mingw32/3.4.2/../../../../include/zconf.h:289,
from
c:/mingw/bin/../lib/gcc/mingw32/3.4.2/../../../../include/zlib.h:34,
from pg_backup_archiver.h:44,
from common.c:20:
c:/mingw/bin/../lib/gcc/mingw32/3.4.2/../../../../include/unistd.h:23:
error: conflicting types for 'chsize'
c:/mingw/bin/../lib/gcc/mingw32/3.4.2/../../../../include/io.h:271:
error: previous declaration of 'chsize' was here
c:/mingw/bin/../lib/gcc/mingw32/3.4.2/../../../../include/unistd.h:23:
error: conflicting types for 'chsize'
c:/mingw/bin/../lib/gcc/mingw32/3.4.2/../../../../include/io.h:271:
error: previous declaration of 'chsize' was here

I suspect we might need to create a pg_off_t type or some such gadget.

Bleah.

But it does need to be fixed.

Bummer. That might be what's needed, but I'm going to at least try to
find some neater way first. I wonder why it didn't happen on MSVC...

//Magnus

Magnus Hagander <magnus@hagander.net> writes:

Andrew Dunstan wrote:

I suspect we might need to create a pg_off_t type or some such gadget.
Bleah.

Bummer. That might be what's needed, but I'm going to at least try to
find some neater way first. I wonder why it didn't happen on MSVC...

Seems like this can only work if the extra #define's appear *after* all
system header files, which might or might not be practical --- but
you'll have as much trouble with the function #define's as the typedef
one if you include headers defining the functions after the macros appear.

regards, tom lane

Tom Lane wrote:

Magnus Hagander <magnus@hagander.net> writes:

Andrew Dunstan wrote:

I suspect we might need to create a pg_off_t type or some such gadget.
Bleah.

Bummer. That might be what's needed, but I'm going to at least try to
find some neater way first. I wonder why it didn't happen on MSVC...

Seems like this can only work if the extra #define's appear *after* all
system header files, which might or might not be practical --- but
you'll have as much trouble with the function #define's as the typedef
one if you include headers defining the functions after the macros appear.

The functions aren't really a problem I think - they don't exist on
win32. They're #defined from fseeko() to fseek() etc in port.h.

//Magnus

I suspect we might need to create a pg_off_t type or some such gadget.

Bleah.

But it does need to be fixed.

Bummer. That might be what's needed, but I'm going to at least try to
find some neater way first. I wonder why it didn't happen on MSVC...

Hmm. This was even worse than I thought :-(

I got it building most of the way by following Andrews suggestion and
greating a pgoff_t, just to check it out. That done, it seems that mingw
doesn't include these 64-bit functions in their import library *at all*.
That gives us basically two options that I can see, to proceed:

1) Set up pg_dump* to dynamically load these functions from msvcrt.dll
at startup. This will require a different codepath from the MSVC build
of course, since Microsoft have been shipping these functions in their
libraries since NT4. Should work, but nor particularly pretty.

2) Just say that the mingw compiled versions of pg_dump* can't deal with
2Gb+ files. IIRC, we've built pg_dump with both the "new vc build
system" on VS2005 and with the "old win32.mak style build system" with
VC++ 6.0, so if we're comfortable enough with that we could just ship
binaries built with VC++ for those utilities (even if we don't go to
shipping completely MSVC built binaries for 8.3).

Thoughts on these options?

//Magnus

Magnus Hagander wrote:

I suspect we might need to create a pg_off_t type or some such gadget.

Bleah.

But it does need to be fixed.

Bummer. That might be what's needed, but I'm going to at least try to
find some neater way first. I wonder why it didn't happen on MSVC...

Hmm. This was even worse than I thought :-(

I got it building most of the way by following Andrews suggestion and
greating a pgoff_t, just to check it out. That done, it seems that mingw
doesn't include these 64-bit functions in their import library *at all*.
That gives us basically two options that I can see, to proceed:

1) Set up pg_dump* to dynamically load these functions from msvcrt.dll
at startup. This will require a different codepath from the MSVC build
of course, since Microsoft have been shipping these functions in their
libraries since NT4. Should work, but nor particularly pretty.

2) Just say that the mingw compiled versions of pg_dump* can't deal with
2Gb+ files. IIRC, we've built pg_dump with both the "new vc build
system" on VS2005 and with the "old win32.mak style build system" with
VC++ 6.0, so if we're comfortable enough with that we could just ship
binaries built with VC++ for those utilities (even if we don't go to
shipping completely MSVC built binaries for 8.3).

Thoughts on these options?

//Magnus

Triple bleah. It is not acceptable to say that our only open source tool
chain can't build fundamentally required functionality.

A little googling on "mingw ftello64" came up with this link, which
looked somewhat promising:

http://www.nabble.com/RE:-ftello64-returning-wrong-values-p703470.html

cheers

andrew

Hmm. This was even worse than I thought :-(

I got it building most of the way by following Andrews suggestion and
greating a pgoff_t, just to check it out. That done, it seems that mingw
doesn't include these 64-bit functions in their import library *at all*.
That gives us basically two options that I can see, to proceed:

1) Set up pg_dump* to dynamically load these functions from msvcrt.dll
at startup. This will require a different codepath from the MSVC build
of course, since Microsoft have been shipping these functions in their
libraries since NT4. Should work, but nor particularly pretty.

2) Just say that the mingw compiled versions of pg_dump* can't deal with
2Gb+ files. IIRC, we've built pg_dump with both the "new vc build
system" on VS2005 and with the "old win32.mak style build system" with
VC++ 6.0, so if we're comfortable enough with that we could just ship
binaries built with VC++ for those utilities (even if we don't go to
shipping completely MSVC built binaries for 8.3).

Thoughts on these options?

//Magnus

Triple bleah. It is not acceptable to say that our only open source tool
chain can't build fundamentally required functionality.

A little googling on "mingw ftello64" came up with this link, which
looked somewhat promising:

http://www.nabble.com/RE:-ftello64-returning-wrong-values-p703470.html

What the heck. So it seems mingw went ahead and implemented their own
ftello64 function *without* using the 64-bit functions as provided by
the standard libraires. Argh. There's also fseeko64(). These are of
course mingw only, so we'll need one codepath for mingw and one for
MSVC, but it does at least seem doable.

We're still going to have to change from off_t to pgoff_t or similar,
since MSVC will need int64 and mingw will need off64_t.. Right?

I'll try to take a look at this sometime the next couple of days (out of
time for today) unless beaten to it.

//Magnus

Magnus Hagander wrote:

Hmm. This was even worse than I thought :-(

I got it building most of the way by following Andrews suggestion and
greating a pgoff_t, just to check it out. That done, it seems that mingw
doesn't include these 64-bit functions in their import library *at all*.
That gives us basically two options that I can see, to proceed:

1) Set up pg_dump* to dynamically load these functions from msvcrt.dll
at startup. This will require a different codepath from the MSVC build
of course, since Microsoft have been shipping these functions in their
libraries since NT4. Should work, but nor particularly pretty.

2) Just say that the mingw compiled versions of pg_dump* can't deal with
2Gb+ files. IIRC, we've built pg_dump with both the "new vc build
system" on VS2005 and with the "old win32.mak style build system" with
VC++ 6.0, so if we're comfortable enough with that we could just ship
binaries built with VC++ for those utilities (even if we don't go to
shipping completely MSVC built binaries for 8.3).

Thoughts on these options?

//Magnus

Triple bleah. It is not acceptable to say that our only open source tool
chain can't build fundamentally required functionality.

A little googling on "mingw ftello64" came up with this link, which
looked somewhat promising:

http://www.nabble.com/RE:-ftello64-returning-wrong-values-p703470.html

What the heck. So it seems mingw went ahead and implemented their own
ftello64 function *without* using the 64-bit functions as provided by
the standard libraires. Argh. There's also fseeko64(). These are of
course mingw only, so we'll need one codepath for mingw and one for
MSVC, but it does at least seem doable.

We're still going to have to change from off_t to pgoff_t or similar,
since MSVC will need int64 and mingw will need off64_t.. Right?

I'll try to take a look at this sometime the next couple of days (out of
time for today) unless beaten to it.

Actually, there's another option that Hiroshi mentioned off-list, that I
forgot.

We can implement the Microsoft functions _fseeki64() and _ftelli64()
ourselves, based on win32 API functions. There are examples available
for this.

Not sure which is the cleanest method, too late for more thinking ;-)

//Magnus

Magnus Hagander wrote:

I'll try to take a look at this sometime the next couple of days (out of
time for today) unless beaten to it.

Actually, there's another option that Hiroshi mentioned off-list, that I
forgot.

We can implement the Microsoft functions _fseeki64() and _ftelli64()
ourselves, based on win32 API functions. There are examples available
for this.

Not sure which is the cleanest method, too late for more thinking ;-)

See src/port/fseeko.c for an example for NetBSD and BSD/OS.

--
Bruce Momjian bruce@momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Magnus Hagander wrote:

Also, it compiles fine on MSVC. I still haven't managed to get the MingW
build environment working properly on Win64 even for building Win32
apps, so I haven't been able to build it on MingW yet. It *should* work
since it's all standard functions, but might require further hacks. I'll
try to get around to that later, and Dave has promised to give it a
compile-try as well, but if someone wants to test that, please do ;)

:-(

gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Winline
-Wdeclaration-after-statement -Wendif-labels -fno-strict-aliasing
pg_dump.o common.o pg_dump_sort.o pg_backup_archiver.o pg_backup_db.o
pg_backup_custom.o pg_backup_files.o pg_backup_null.o pg_backup_tar.o
dumputils.o win32ver.o ../../../src/backend/parser/keywords.o
-L../../../src/port -lpgport -L../../../src/interfaces/libpq -lpq
-L../../../src/port -Wl,--allow-multiple-definition -lpgport -lintl
-lssleay32 -leay32 -lcomerr32 -lkrb5_32 -lz -lm -lws2_32 -lshfolder -o
pg_dump.exe
pg_backup_archiver.o(.text+0x2017):pg_backup_archiver.c: undefined
reference to `_fseeki64'
pg_backup_archiver.o(.text+0x3dac):pg_backup_archiver.c: undefined
reference to `_fseeki64'
pg_backup_custom.o(.text+0x773):pg_backup_custom.c: undefined reference
to `_fseeki64'
pg_backup_custom.o(.text+0xaaa):pg_backup_custom.c: undefined reference
to `_ftelli64'
pg_backup_custom.o(.text+0xed2):pg_backup_custom.c: undefined reference
to `_ftelli64'
pg_backup_custom.o(.text+0xf21):pg_backup_custom.c: undefined reference
to `_fseeki64'
pg_backup_tar.o(.text+0x845):pg_backup_tar.c: undefined reference to
`_ftelli64'
pg_backup_tar.o(.text+0x10f9):pg_backup_tar.c: undefined reference to
`_fseeki64'
pg_backup_tar.o(.text+0x1107):pg_backup_tar.c: undefined reference to
`_ftelli64'
pg_backup_tar.o(.text+0x1162):pg_backup_tar.c: undefined reference to
`_fseeki64'
collect2: ld returned 1 exit status
make[3]: *** [pg_dump] Error 1
make[3]: Leaving directory `/cvs/pgsql/src/bin/pg_dump'

Regards, Dave.