New option in pg_basebackup to exclude pg_log files during base backup

On Wed, Apr 9, 2014 at 2:06 AM, Prabakaran, Vaishnavi <
vaishnavip@fast.au.fujitsu.com> wrote:

Hi all,

Following the discussion in message id -
CAHGQGwFFMOr4EcugWHZpAaPYQbsEKDg66VmJ1rveJ6Z-EgaqAg@mail.gmail.com , I
have developed the patch which gives option to user to exclude pg_log
directory contents in pg_basebackup.

[Current situation]

During pg_basebackup, all files in pg_log directory will be copied to new
backup directory.

[Design]

- Added new non-mandatory option "-S/--skip-log-dir" to pg_basebackup .

- If "skip-log-dir" is specified in pg_basebackup command, then in
basebackup, exclude copying log files from standard "pg_log" directory and
any other directory specified in Log_directory guc variable. (Still empty
folder "pg_log"/$Log_directory will be created)

- In case, pg_log/$Log_directory is symbolic link, then an empty folder
will be created

[Advantage]

It gives an option to user to avoid copying of large log files if they
doesn't wish to and hence can save memory space.

While pg_log is definitely the most common one being the default on many
platforms, we'll still be missing other ones. Should we really hardcode it,
or should we somehow derive it from the settings for log_directory instead?

As a more general discussion, is this something we might want to expose as
a more general facility rather than hardcode it to the log directory only?
And is it perhaps something we'd rather have configured at the server than
specified in pg_basebackup - like a guc saying which directories should
always be excluded from a basebackup? So you don't have to remember it
every time?

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

Magnus Hagander wrote:

While pg_log is definitely the most common one being the default on many
platforms, we'll still be missing other ones. Should we really hardcode it,
or should we somehow derive it from the settings for log_directory instead?

As a more general discussion, is this something we might want to expose as
a more general facility rather than hardcode it to the log directory only?
And is it perhaps something we'd rather have configured at the server than
specified in pg_basebackup - like a guc saying which directories should
always be excluded from a basebackup? So you don't have to remember it
every time?

So it'd be an array, and by default you'd have something like:
basebackup_skip_path = $log_directory
?

Maybe use it to skip backup labels by default as well.
basebackup_skip_path = $log_directory, $backup_label_files

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Apr 9, 2014 at 4:55 PM, Alvaro Herrera <alvherre@2ndquadrant.com>wrote:

Magnus Hagander wrote:

While pg_log is definitely the most common one being the default on many
platforms, we'll still be missing other ones. Should we really hardcode

it,

or should we somehow derive it from the settings for log_directory

instead?

As a more general discussion, is this something we might want to expose

as

a more general facility rather than hardcode it to the log directory

only?

And is it perhaps something we'd rather have configured at the server

than

specified in pg_basebackup - like a guc saying which directories should
always be excluded from a basebackup? So you don't have to remember it
every time?

So it'd be an array, and by default you'd have something like:
basebackup_skip_path = $log_directory
?

Maybe use it to skip backup labels by default as well.
basebackup_skip_path = $log_directory, $backup_label_files

I hadn't considered any details, but yes, someting along that line. And
then you could also include arbitrary filenames or directories should you
want. E.g. if you use the data directory to store your torrents or
something.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

Magnus Hagander wrote:

On Wed, Apr 9, 2014 at 4:55 PM, Alvaro Herrera <alvherre@2ndquadrant.com>wrote:

So it'd be an array, and by default you'd have something like:
basebackup_skip_path = $log_directory
?

Maybe use it to skip backup labels by default as well.
basebackup_skip_path = $log_directory, $backup_label_files

I hadn't considered any details, but yes, someting along that line. And
then you could also include arbitrary filenames or directories should you
want. E.g. if you use the data directory to store your torrents or
something.

Man, that's a great idea. Database servers have lots of diskspace in
that partition, so it should work really well. Thanks!

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Thursday, Apr 10,2014 at 1:15Am, Álvaro Herrera wrote:

Magnus Hagander wrote:

On Wed, Apr 9, 2014 at 4:55 PM, Alvaro Herrera <alvherre@2ndquadrant.com>wrote:

So it'd be an array, and by default you'd have something like:
basebackup_skip_path = $log_directory ?

Maybe use it to skip backup labels by default as well.
basebackup_skip_path = $log_directory, $backup_label_files

I hadn't considered any details, but yes, someting along that line.
And then you could also include arbitrary filenames or directories
should you want. E.g. if you use the data directory to store your
torrents or something.

Man, that's a great idea. Database servers have lots of diskspace in that partition, so it should work really well. Thanks!

Yes, It sounds like a good idea. I will look into this and start working in sometime.

Thanks & Regards,
Vaishnavi
Fujitsu Australia

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 4/9/14, 10:57 AM, Magnus Hagander wrote:

So it'd be an array, and by default you'd have something like:
basebackup_skip_path = $log_directory
?

Maybe use it to skip backup labels by default as well.
basebackup_skip_path = $log_directory, $backup_label_files

I hadn't considered any details, but yes, someting along that line. And
then you could also include arbitrary filenames or directories should
you want.

What are the use cases for excluding anything else?

pg_basebackup ought to have some intelligence about what files are
appropriate to include or exclude, depending on what the user is trying
to do. It shouldn't become a general file copying tool.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers