How to create directory format backup

On Wed, Feb 08, 2023 at 05:00:10PM +0200, Andrus wrote:

Hi!

Creating backup in directory format using

    pg_dump -f "sba" -Fdirectory --jobs=32 --verbose sba

throws error

    pg_dump: error: could not stat file "sba/282168.data.gz": value too
large

How to fix it ?

Server is Postgres 12 running in Debian Linux 10 under WSL
Client is pg_dump from Postgres 15 running in Windows 11

Looks like your filesystem on client is having limits on file sizes.

Use better filesystem, or just dump on linux, it's filesystems usually
don't hit these limits.

Best regards,

depesz

Hi!

Looks like your filesystem on client is having limits on file sizes.

Use better filesystem, or just dump on linux, it's filesystems usually
don't hit these limits.

This file size is only 6.2 GB. If custom format is used, pg_dump creates
large file without problems. There are no file size limits. Error
message is not about this.

Backup client is running in Windows 11 and this cannot changed.

How to create backup in format from which tables can selectively restored?

Posted also in
https://stackoverflow.com/questions/75387616/how-to-make-directory-format-backup

Andrus.

On 08/02/2023 21:59 CET Andrus <kobruleht2@hot.ee> wrote:

How to create backup in format from which tables can selectively restored?

Dump as custom-format archive (-F custom) and use that with pg_restore and
options --table or --list/--use-list to select what should be restored.

--
Erik

Hi!

How to create backup in format from which tables can selectively restored?

Dump as custom-format archive (-F custom) and use that with pg_restore and
options --table or --list/--use-list to select what should be restored.

How to select tables interactively like pgAdmin allows to select when
directory format is used ?

Database contains hundreds of schemas. I need to restore public and
other other schema.

Whole backup file is scanned to restore only two schemas. It takes lot
of time.

Also directory format allows to use all cores with --jobs=32 parameter.
Dump and partial restore using custom format are much slower.

Andrus.

On 08/02/2023 22:37 CET Andrus <kobruleht2@hot.ee> wrote:

How to create backup in format from which tables can selectively
restored?

Dump as custom-format archive (-F custom) and use that with pg_restore
and options --table or --list/--use-list to select what should be
restored.

How to select tables interactively like pgAdmin allows to select when
directory format is used ?

Database contains hundreds of schemas. I need to restore public and other
other schema.

Whole backup file is scanned to restore only two schemas. It takes lot of
time.

pg_dump also accepts options --table and --schema to only dump what you need.

Also directory format allows to use all cores with --jobs=32 parameter.
Dump and partial restore using custom format are much slower.

Run multiple pg_dump processes in parallel where each processes a subset of
tables with the options mentioned above.

--
Erik

Andrus <kobruleht2@hot.ee> writes:

This file size is only 6.2 GB. If custom format is used, pg_dump creates
large file without problems. There are no file size limits. Error
message is not about this.

Are you certain that the pg_dump you're using is v15, and not something
pre-v14? We got rid of the 4GB limit on stat() results in v14,
or so we thought.

Anyway, as far as I can find pg_dump can only reach that error message
via fsync_dir_recurse(), which suggests that a workaround might be
to add --no-sync so it doesn't try to fsync its output.

regards, tom lane