ftell mismatch with expected position

On 10/29/2015 02:51 AM, Eelke Klein wrote:

I am getting the following warning when our program runs pg_dump.exe and
the output is in custom format and send to standard out which is
connected to a pipe (Windows platform).

pg_dump: [custom archiver] WARNING: ftell mismatch with expected
position -- ftell used

Can you show us the exact command, with arguments, that you are using?

Also what is the size of the dump file you are creating?

The output of pg_dump is received by another process that copies it (and
some other data) into a zip file. When I extract the db backup from the
zip the backup is fine and I can restore it.

I have both tried the pg_dump.exe from the 9.5 alpha and a stable 9.4
both generate this warning.
When I change the format to plain the warning goes away.
When I call pg_dump from the commandline and let it send to standard out
and redirect it to a file the warning does not occur but it does occur
when I redirect to nul.

Is there a way to prevent this warning? I would think pg_dump shouldn't
call ftell on a pipe.

Regards, Eelke

--
Adrian Klaver
adrian.klaver@aklaver.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Normally we call this from within our windows program where a lot of code
is involved for setting up the environment, and creating the pipes and
redirecting stdout, stderr and stdin. However I believe it is the same
problem that can be caused with the following commandline in cmd.exe but
you will need to have Coreutils for windows installed to get the cat
command.

pg_dump.exe -U postgres --format=c mydb | cat > backupfile

The size of the backup doesn't matter. As long as there is atleast one
table in the database the warning occurs. When I use --verbose it looks
like I get one warning for every table in the database while it is dumping
the data in the tables.

Eelke Klein

2015-10-30 14:53 GMT+01:00 Adrian Klaver <adrian.klaver@aklaver.com>:

Please refrain from top-posting.

On 2 November 2015 at 10:48, Eelke Klein <eelke@bolt.nl> wrote:

Normally we call this from within our windows program where a lot of code is
involved for setting up the environment, and creating the pipes and
redirecting stdout, stderr and stdin. However I believe it is the same
problem that can be caused with the following commandline in cmd.exe but you
will need to have Coreutils for windows installed to get the cat command.

pg_dump.exe -U postgres --format=c mydb | cat > backupfile

Do you also get the warning if you split those commands up, and if so,
in which one?

pg_dump.exe -U postgres --format=c mydb > backupfile
cat backupfile > someotherfile

The size of the backup doesn't matter. As long as there is atleast one table
in the database the warning occurs. When I use --verbose it looks like I get
one warning for every table in the database while it is dumping the data in
the tables.

Eelke Klein

2015-10-30 14:53 GMT+01:00 Adrian Klaver <adrian.klaver@aklaver.com>:

On 10/29/2015 02:51 AM, Eelke Klein wrote:

I am getting the following warning when our program runs pg_dump.exe and
the output is in custom format and send to standard out which is
connected to a pipe (Windows platform).

pg_dump: [custom archiver] WARNING: ftell mismatch with expected
position -- ftell used

Can you show us the exact command, with arguments, that you are using?

Also what is the size of the dump file you are creating?

The output of pg_dump is received by another process that copies it (and
some other data) into a zip file. When I extract the db backup from the
zip the backup is fine and I can restore it.

I have both tried the pg_dump.exe from the 9.5 alpha and a stable 9.4
both generate this warning.
When I change the format to plain the warning goes away.
When I call pg_dump from the commandline and let it send to standard out
and redirect it to a file the warning does not occur but it does occur
when I redirect to nul.

Is there a way to prevent this warning? I would think pg_dump shouldn't
call ftell on a pipe.

Regards, Eelke

--
Adrian Klaver
adrian.klaver@aklaver.com

--
If you can't see the forest for the trees,
Cut the trees and you'll see there is no forest.

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Alban Hertroys <haramrae@gmail.com> writes:

Please refrain from top-posting.
On 2 November 2015 at 10:48, Eelke Klein <eelke@bolt.nl> wrote:

Normally we call this from within our windows program where a lot of code is
involved for setting up the environment, and creating the pipes and
redirecting stdout, stderr and stdin. However I believe it is the same
problem that can be caused with the following commandline in cmd.exe but you
will need to have Coreutils for windows installed to get the cat command.

pg_dump.exe -U postgres --format=c mydb | cat > backupfile

Do you also get the warning if you split those commands up, and if so,
in which one?

Presumably it doesn't happen without the use of the pipe notation.

I suspect that what's happening is that stdout isn't getting put into
binary mode, so that Microsoft's CR/NL translation corrupts the data.
If that's true, though, the resulting backup file should be corrupt;
is it valid according to "pg_restore backupfile"?

regards, tom lane

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Presumably it doesn't happen without the use of the pipe notation.

Indeed, it only happens when a pipe is involved.

I suspect that what's happening is that stdout isn't getting put into
binary mode, so that Microsoft's CR/NL translation corrupts the data.
If that's true, though, the resulting backup file should be corrupt;
is it valid according to "pg_restore backupfile"?

Yes pg_restore restores it without problems. However I did notice that it
is larger then normal.

I have also tested what happens on freebsd when using a pipe and there are
no warnings and backup is expected size.

I dug into the source code of pg_dump and found that the warning is given
by _getFilePos(ArchiveHandle *AH, lclContext *ctx) in pg_backup_custom.c
but it won't if ctx->hasSeek is set to false. Which is initialized by a
call to checkSeek in pg_backup_archiver.c. Unfortunatly I'm not yet able to
build and debug ton windows to verify if a correction to this function
would help.

regards, Eelke