pgsql: Allow pg_basebackup to stream transaction log in tar mode

That broke some buildfarm. Looking into it.

//Magnus

On Sun, Oct 23, 2016 at 3:28 PM, Magnus Hagander <magnus@hagander.net>
wrote:

Allow pg_basebackup to stream transaction log in tar mode

This will write the received transaction log into a file called
pg_wal.tar(.gz) next to the other tarfiles instead of writing it to
base.tar. When using fetch mode, the transaction log is still written to
base.tar like before, and when used against a pre-10 server, the file
is named pg_xlog.tar.

To do this, implement a new concept of a "walmethod", which is
responsible for writing the WAL. Two implementations exist, one that
writes to a plain directory (which is also used by pg_receivexlog) and
one that writes to a tar file with optional compression.

Reviewed by Michael Paquier

Branch
------
master

Details
-------
http://git.postgresql.org/pg/commitdiff/56c7d8d4552180fd66fe48423bb2a9
bb767c2d87

Modified Files
--------------
doc/src/sgml/ref/pg_basebackup.sgml | 18 +-
src/bin/pg_basebackup/Makefile | 2 +-
src/bin/pg_basebackup/pg_basebackup.c | 62 +-
src/bin/pg_basebackup/pg_receivexlog.c | 10 +-
src/bin/pg_basebackup/receivelog.c | 316 ++++------
src/bin/pg_basebackup/receivelog.h | 3 +-
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 6 +-
src/bin/pg_basebackup/walmethods.c | 886
+++++++++++++++++++++++++++
src/bin/pg_basebackup/walmethods.h | 45 ++
src/include/pgtar.h | 1 +
src/port/tar.c | 2 +-
11 files changed, 1107 insertions(+), 244 deletions(-)

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

The remaining windows buildfarm build issue looks like this:

src/bin/pg_basebackup/receivelog.c(135): error C2039: '_commit' : is
not a member of 'WalWriteMethod'
[C:\buildfarm\buildenv\HEAD\pgsql.build\pg_basebackup.vcxproj]

AFAICT this comes from using walmethod->fsync(), and we have a #define
changing fsync() to _commit (port/win32.h, line 70).

The easiest might just be to rename walmethod->fsync() to
walmethod->do_fsync(), so we don't have to mess with any other code.

Thoughts?

//Magnus

On Sun, Oct 23, 2016 at 3:55 PM, Magnus Hagander <magnus@hagander.net>
wrote:

Magnus Hagander <magnus@hagander.net> writes:

The remaining windows buildfarm build issue looks like this:
src/bin/pg_basebackup/receivelog.c(135): error C2039: '_commit' : is
not a member of 'WalWriteMethod'
[C:\buildfarm\buildenv\HEAD\pgsql.build\pg_basebackup.vcxproj]
AFAICT this comes from using walmethod->fsync(), and we have a #define
changing fsync() to _commit (port/win32.h, line 70).

Yeah.

The easiest might just be to rename walmethod->fsync() to
walmethod->do_fsync(), so we don't have to mess with any other code.

How about calling the method just "sync()"? Seems a bit less
artificial.

regards, tom lane

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

On Sun, Oct 23, 2016 at 5:48 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Magnus Hagander <magnus@hagander.net> writes:

The remaining windows buildfarm build issue looks like this:
src/bin/pg_basebackup/receivelog.c(135): error C2039: '_commit' : is
not a member of 'WalWriteMethod'
[C:\buildfarm\buildenv\HEAD\pgsql.build\pg_basebackup.vcxproj]
AFAICT this comes from using walmethod->fsync(), and we have a #define
changing fsync() to _commit (port/win32.h, line 70).

Yeah.

The easiest might just be to rename walmethod->fsync() to
walmethod->do_fsync(), so we don't have to mess with any other code.

How about calling the method just "sync()"? Seems a bit less
artificial.

Sure, that should work too. Seems we're not defining that one to something
else, so let's use that.

I'll go fix.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

I have following warning while compiling with clang.

walmethods.c:437:9: warning: comparison of unsigned expression < 0 is
always false [-Wtautological-compare]
if (r < 0)
~ ^ ~
1 warning generated.

It seems that 'r' should be 'ssize_t' instead of 'size_t'.

------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

On Sun, Oct 23, 2016 at 4:28 PM, Magnus Hagander <magnus@hagander.net>
wrote:

On Mon, Oct 24, 2016 at 2:58 PM, Alexander Korotkov <
a.korotkov@postgrespro.ru> wrote:

I have following warning while compiling with clang.

walmethods.c:437:9: warning: comparison of unsigned expression < 0 is
always false [-Wtautological-compare]
if (r < 0)
~ ^ ~
1 warning generated.

It seems that 'r' should be 'ssize_t' instead of 'size_t'.

Yup. The result from tar_write() is ssize_t so it definitely should be.

Thanks, I've applied a fix.

//Magnus