pg_restore [archiver] file offset in dump file is too large

"Kevin Grittner" <Kevin.Grittner@wicourts.gov> writes:

File dumped from 8.1beta3 using pg_dump -Fc on Linux box.
This dump restored successfully onto 8.1RC1 on Linux box.
File FTP'd to Windows box; attempt to restore onto 8.1RC1 fails with:
pg_restore [archiver] file offset in dump file is too large

[ itch... ] This sure as heck sounds like the file was corrupted during
the FTP transfer. Did you remember to select binary transfer mode?

On Windows, the file size and md5sum -b result is identical.

That observation does seem like evidence against my theory, but ...

regards, tom lane

Hmm. Windows reports an offset size of 4 bytes on a dump I just made ...
is that relevant? What governs it?

cheers

andrew

Kevin Grittner wrote:

Andrew Dunstan <andrew@dunslane.net> writes:

Hmm. Windows reports an offset size of 4 bytes on a dump I just made ...
is that relevant? What governs it?

[ looks again... ] Hm, that is a 40Gb dump Kevin is complaining of,
isn't it. Do our Windows builds have LARGEFILE support?

regards, tom lane

Tom Lane said:

Andrew Dunstan <andrew@dunslane.net> writes:

Hmm. Windows reports an offset size of 4 bytes on a dump I just made
... is that relevant? What governs it?

[ looks again... ] Hm, that is a 40Gb dump Kevin is complaining of,
isn't it. Do our Windows builds have LARGEFILE support?

I think from a cursory investigation the short answer is no, but they
probably could. If so, that should definitely go on the TODO list. Windows
gurus, any thoughts?

Meanwhile, the answer seems to be to use text dumps if you run into this
problem.

cheers

andrew

Andrew Dunstan <andrew@dunslane.net> writes:

Hmm. Windows reports an offset size of 4 bytes on a dump I

just made

... is that relevant? What governs it?

[ looks again... ] Hm, that is a 40Gb dump Kevin is

complaining of,

isn't it. Do our Windows builds have LARGEFILE support?

I think from a cursory investigation the short answer is no,
but they probably could. If so, that should definitely go on
the TODO list. Windows gurus, any thoughts?

Windows certainly supports large files. I don't see why we wouldn't pick
this up in autoconf, perhaps the mingw libraries don't?
Definitly worth investigating, no time for 8.1, so putting it on TODO
seems very good :-)

//Magnus

Magnus Hagander wrote:

Andrew Dunstan <andrew@dunslane.net> writes:

Hmm. Windows reports an offset size of 4 bytes on a dump I

just made

... is that relevant? What governs it?

[ looks again... ] Hm, that is a 40Gb dump Kevin is

complaining of,

isn't it. Do our Windows builds have LARGEFILE support?

I think from a cursory investigation the short answer is no,
but they probably could. If so, that should definitely go on
the TODO list. Windows gurus, any thoughts?

Windows certainly supports large files. I don't see why we wouldn't pick
this up in autoconf, perhaps the mingw libraries don't?
Definitly worth investigating, no time for 8.1, so putting it on TODO
seems very good :-)

There is no fseeko in the Windows libraries, nor any provision in the
mingw headers that I can see for a 64 bit off_t. So we would need to
roll our own to some extent - I think we need more than just a bit of
configure cleverness.

However, there is a Windows library routine to do a 64bit seek and
return the file position, so we could fairly easily implement fseeko and
ftello based on that. See
http://msdn.microsoft.com/library/en-us/vclib/html/_crt__lseek.2c_._lseeki64.asp

cheers

andrew

Andrew Dunstan <andrew@dunslane.net> writes:

There is no fseeko in the Windows libraries, nor any provision in the
mingw headers that I can see for a 64 bit off_t. So we would need to
roll our own to some extent - I think we need more than just a bit of
configure cleverness.

However, there is a Windows library routine to do a 64bit seek and
return the file position, so we could fairly easily implement fseeko and
ftello based on that. See
http://msdn.microsoft.com/library/en-us/vclib/html/_crt__lseek.2c_._lseeki64.asp

Is there any risk that the mingw libraries would fail when manipulating
a file whose current offset exceeds 32 bits? I'm wondering if we'd have
to roll our own stdio in toto, not just fseeko/ftello :-(

regards, tom lane

Tom Lane wrote:

Andrew Dunstan <andrew@dunslane.net> writes:

There is no fseeko in the Windows libraries, nor any provision in the
mingw headers that I can see for a 64 bit off_t. So we would need to
roll our own to some extent - I think we need more than just a bit of
configure cleverness.

However, there is a Windows library routine to do a 64bit seek and
return the file position, so we could fairly easily implement fseeko and
ftello based on that. See
http://msdn.microsoft.com/library/en-us/vclib/html/_crt__lseek.2c_._lseeki64.asp

Is there any risk that the mingw libraries would fail when manipulating
a file whose current offset exceeds 32 bits? I'm wondering if we'd have
to roll our own stdio in toto, not just fseeko/ftello :-(

AFAIK all this is coming straight from the base Windows libraries. I
guess we'll have to test it when we get around to it.

cheers

andrew

Andrew Dunstan wrote:

There is no fseeko in the Windows libraries, nor any provision in the
mingw headers that I can see for a 64 bit off_t. So we would need to
roll our own to some extent - I think we need more than just a bit of
configure cleverness.

However, there is a Windows library routine to do a 64bit seek and
return the file position, so we could fairly easily implement fseeko and
ftello based on that. See
http://msdn.microsoft.com/library/en-us/vclib/html/_crt__lseek.2c_._lseeki64.asp

See src/port/fseeko.c for a version built on fsetpos().

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian wrote:

Andrew Dunstan wrote:

There is no fseeko in the Windows libraries, nor any provision in the
mingw headers that I can see for a 64 bit off_t. So we would need to
roll our own to some extent - I think we need more than just a bit of
configure cleverness.

However, there is a Windows library routine to do a 64bit seek and
return the file position, so we could fairly easily implement fseeko and
ftello based on that. See
http://msdn.microsoft.com/library/en-us/vclib/html/_crt__lseek.2c_._lseeki64.asp

See src/port/fseeko.c for a version built on fsetpos().

Yeah, I'm not made warm and fuzzy by the comments about fpos_t in
mingw's stdio.h, though:

/*
* An opaque data type used for storing file positions... The contents of
* this type are unknown, but we (the compiler) need to know the size
* because the programmer using fgetpos and fsetpos will be setting aside
* storage for fpos_t structres. Actually I tested using a byte array and
* it is fairly evident that the fpos_t type is a long (in CRTDLL.DLL).
* Perhaps an unsigned long? TODO? It's definitely a 64-bit number in
* MSVCRT however, and for now `long long' will do.
*/

But the example program on MSDN contains "pos = 14", which leads one to
assume that it really is some simple int underneath.

cheers

andrew

cheers

andrew

Added to TODO:

o Add long file support for binary pg_dump output

While Win32 supports 64-bit files, the MinGW API does not,
meaning we have to build an fseeko replacement on top of the
Win32 API, and we have to make sure MinGW handles it.

---------------------------------------------------------------------------

Andrew Dunstan wrote:

Bruce Momjian wrote:

Andrew Dunstan wrote:

There is no fseeko in the Windows libraries, nor any provision in the
mingw headers that I can see for a 64 bit off_t. So we would need to
roll our own to some extent - I think we need more than just a bit of
configure cleverness.

However, there is a Windows library routine to do a 64bit seek and
return the file position, so we could fairly easily implement fseeko and
ftello based on that. See
http://msdn.microsoft.com/library/en-us/vclib/html/_crt__lseek.2c_._lseeki64.asp

See src/port/fseeko.c for a version built on fsetpos().

Yeah, I'm not made warm and fuzzy by the comments about fpos_t in
mingw's stdio.h, though:

/*
* An opaque data type used for storing file positions... The contents of
* this type are unknown, but we (the compiler) need to know the size
* because the programmer using fgetpos and fsetpos will be setting aside
* storage for fpos_t structres. Actually I tested using a byte array and
* it is fairly evident that the fpos_t type is a long (in CRTDLL.DLL).
* Perhaps an unsigned long? TODO? It's definitely a 64-bit number in
* MSVCRT however, and for now `long long' will do.
*/

But the example program on MSDN contains "pos = 14", which leads one to
assume that it really is some simple int underneath.

cheers

andrew

cheers

andrew

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Added to TODO:

o Add long file support for binary pg_dump output

While Win32 supports 64-bit files, the MinGW API does not,
meaning we have to build an fseeko replacement on top of the
Win32 API, and we have to make sure MinGW handles it.

Wouldn't it be better to lobby the MinGW folk to fix their problem?
Or even help them with it? I can't see the rationale for implementing
a workaround that helps only us.

regards, tom lane

Tom Lane wrote:

While Win32 supports 64-bit files, the MinGW API does not,
meaning we have to build an fseeko replacement on top of the
Win32 API, and we have to make sure MinGW handles it.

Wouldn't it be better to lobby the MinGW folk to fix their problem?
Or even help them with it? I can't see the rationale for implementing
a workaround that helps only us.

There is a library available from the gnuwin32 project that advertises
fseeko and fseeko64. So we probably have a choice of requiring this
library or doing it ourselves.

see http://gnuwin32.sourceforge.net/packages/libgw32c.htm

cheers

andrew

Andrew Dunstan wrote:

Tom Lane wrote:

While Win32 supports 64-bit files, the MinGW API does not,
meaning we have to build an fseeko replacement on top of the
Win32 API, and we have to make sure MinGW handles it.

Wouldn't it be better to lobby the MinGW folk to fix their problem?
Or even help them with it? I can't see the rationale for implementing
a workaround that helps only us.

There is a library available from the gnuwin32 project that advertises
fseeko and fseeko64. So we probably have a choice of requiring this
library or doing it ourselves.

see http://gnuwin32.sourceforge.net/packages/libgw32c.htm

OK, updated:

o Add long file support for binary pg_dump output

While Win32 supports 64-bit files, the MinGW API does not,
meaning we have to build an fseeko replacement on top of the
Win32 API, and we have to make sure MinGW handles it. Another
option is to wait for the MinGW project to fix it, or use the
code from the LibGW32C project as a guide.

My bet is that the license for LibGW32C might require us to use it as a
guide rather than the code itself.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian wrote:

Andrew Dunstan wrote:

Tom Lane wrote:

While Win32 supports 64-bit files, the MinGW API does not,
meaning we have to build an fseeko replacement on top of the
Win32 API, and we have to make sure MinGW handles it.

Wouldn't it be better to lobby the MinGW folk to fix their problem?
Or even help them with it? I can't see the rationale for implementing
a workaround that helps only us.

There is a library available from the gnuwin32 project that advertises
fseeko and fseeko64. So we probably have a choice of requiring this
library or doing it ourselves.

see http://gnuwin32.sourceforge.net/packages/libgw32c.htm

OK, updated:

o Add long file support for binary pg_dump output

While Win32 supports 64-bit files, the MinGW API does not,
meaning we have to build an fseeko replacement on top of the
Win32 API, and we have to make sure MinGW handles it. Another
option is to wait for the MinGW project to fix it, or use the
code from the LibGW32C project as a guide.

My bet is that the license for LibGW32C might require us to use it as a
guide rather than the code itself.

[after examining source]

It's LGPL. But basically it just uses fsetpos() just like ours does. So
maybe a little adjustment to our config so we can use ours in a 64 bit
manner would be the way to go.

cheers

andrew