Restoring only a subset of schemas

On 3/17/25 07:57, Sylvain Cuaz wrote:

Hi all,

    I have a DB with one schema named "Common" holding data referenced
by other schemas. All other schemas have the same structure (tables and
fields) and are named "cXXX" where XXX is just an int. Thus the only
cross-schema foreign keys are in "cXXX" pointing to "Common", and each
"cXXX" is completely independent of other "cXXX" schemas.
    Now if I want to restore from a full dump of this DB, but with only
one "cXXX" and the "Common" schema :
- if I pass --create --schema=Common, then the CREATE SCHEMA is missing,
i.e. it only emits data inside "Common" and the restore fails.

I am not seeing that.

For:

pg_dump -d test -U postgres -s --create --schema=other_sch --schema=public

In the output I get:

[...]

CREATE SCHEMA other_sch;

ALTER SCHEMA other_sch OWNER TO postgres;

--
-- Name: public; Type: SCHEMA; Schema: -; Owner: pg_database_owner
--

CREATE SCHEMA public;

ALTER SCHEMA public OWNER TO pg_database_owner;

[...]

What is the complete command you are using for the pg_dump?

What Postgres version(s) are you using?

Cheers,

Sylvain

--
Adrian Klaver
adrian.klaver@aklaver.com

Sylvain Cuaz <sylvain@ilm-informatique.fr> writes:

    Now if I want to restore from a full dump of this DB, but with only one "cXXX" and the "Common"
schema :
- if I pass --create --schema=Common, then the CREATE SCHEMA is missing, i.e. it only emits data
inside "Common" and the restore fails.
- if I could pass --create --exclude-schema='c*' (fictional notation as patterns are only recognized
by pg_dump), then all schemas would be created, with no data inside except for "Common". Creating
all schemas is a waste of time, but more importantly would make restoring other schemas more
difficult (e.g. rows should be inserted before creating foreign keys).

In general, the solution for edge-case restore selection needs is to
make a list of the dump's contents with "pg_restore -l", edit out what
you don't want using any method you like, then use the edited list with
"pg_restore -L".

While I'd be in favor of improving pg_restore to accept wild-card
patterns, I'm very hesitant to start inventing new kinds of selection
switches for it. The interactions between such switches would be a
mess.

regards, tom lane

Le 17/03/2025 à 16:21, Adrian Klaver a écrit :

On 3/17/25 07:57, Sylvain Cuaz wrote:

Hi all,

     I have a DB with one schema named "Common" holding data referenced by other schemas. All
other schemas have the same structure (tables and fields) and are named "cXXX" where XXX is just
an int. Thus the only cross-schema foreign keys are in "cXXX" pointing to "Common", and each
"cXXX" is completely independent of other "cXXX" schemas.
     Now if I want to restore from a full dump of this DB, but with only one "cXXX" and the
"Common" schema :
- if I pass --create --schema=Common, then the CREATE SCHEMA is missing, i.e. it only emits data
inside "Common" and the restore fails.

I am not seeing that.

For:

pg_dump -d test -U postgres -s --create --schema=other_sch --schema=public

What is the complete command you are using for the pg_dump?

Hi,

    As I said I'm restoring, not dumping. I make daily full backups and sometimes need to restore a
specific day, but the full database is quite big and I would like to only restore one or two schemas.

As you said, if one passes --create --schema to pg_dump then a valid SQL is produced with CREATE
DATABASE, CREATE SCHEMA, CREATE TABLE and all objects inside the schema.

But if one passes --create --schema to pg_restore then an invalid SQL is produced because it
contains CREATE DATABASE, CREATE TABLE but it doesn't contain the CREATE SCHEMA needed for the
tables. Is there any reason for that discrepancy between dump & restore and for outputting invalid
SQL ?

My proposed --include-create-schema would just add the CREATE SCHEMA so that pg_restore behaves the
same as pg_dump, and would allow to output valid SQL. But ideally this option shouldn't even be
needed because pg_restore would just emit CREATE SCHEMA like pg_dump.

What Postgres version(s) are you using?

A lot :-) But for this problem I'm using 13 & 15.

Cheers,

Sylvain.

Le 17/03/2025 à 16:29, Tom Lane a écrit :

Sylvain Cuaz <sylvain@ilm-informatique.fr> writes:

    Now if I want to restore from a full dump of this DB, but with only one "cXXX" and the "Common"
schema :
- if I pass --create --schema=Common, then the CREATE SCHEMA is missing, i.e. it only emits data
inside "Common" and the restore fails.
- if I could pass --create --exclude-schema='c*' (fictional notation as patterns are only recognized
by pg_dump), then all schemas would be created, with no data inside except for "Common". Creating
all schemas is a waste of time, but more importantly would make restoring other schemas more
difficult (e.g. rows should be inserted before creating foreign keys).

In general, the solution for edge-case restore selection needs is to
make a list of the dump's contents with "pg_restore -l", edit out what
you don't want using any method you like, then use the edited list with
"pg_restore -L".

Hi,

    I am aware of that feature, but that forces me to know every type of entry that pertains to a
schema or database (e.g. DEFAULT ACL, ACL, COMMENT, DATABASE PROPERTIES, etc.) and what about new
ones that will be added in the future ?

Further, I don't see how it's an edge-case, at the core I just want to restore some but not all the
schemas. This is possible for pg_dump, see my response to Adrian Klaver.

While I'd be in favor of improving pg_restore to accept wild-card
patterns,

That would definitely be appreciated.

I'm very hesitant to start inventing new kinds of selection
switches for it. The interactions between such switches would be a
mess.

Which interactions ? It seems to me that the name of the schema should be used as the namespace to
check in _tocEntryRequired() in pg_backup_archiver.c, and then the dependent entries (e.g. ACL,
COMMENT) would be handled around line 3050. I've attached a patch with some pseudo-code. In fact,
were it not for compatibility, I'd argue that my proposed options should be the default, at least
with --create, so as to neither output invalid SQL (for -n) nor extra unwanted ones (for -N) and to
behave like pg_dump.

Cheers,

Sylvain

Sylvain Cuaz <sylvain@ilm-informatique.fr> writes:

Further, I don't see how it's an edge-case, at the core I just want to restore some but not all the
schemas. This is possible for pg_dump, see my response to Adrian Klaver.

You have a very good point that it's annoying that pg_restore's
--schema switch doesn't act like pg_dump's --schema switch.
I could support changing that, but only if we also did something
about the fact that pg_restore's switch doesn't handle patterns.

The pattern aspect is kind of a mess, because pg_dump implements
that through a server-side regex, which is a facility pg_restore
doesn't have access to. Maybe it'd be good enough to implement
the "*" and "?" wildcards and stop there, but I'm not sure.

regards, tom lane