Teach pg_receivewal to use lz4 compression
Started by Nonameover 4 years ago55 messages
Hi,
The program pg_receivewal can use gzip compression to store the received WAL.
This patch teaches it to be able to use lz4 compression if the binary is build
using the -llz4 flag.
Previously, the user had to use the option --compress with a value between [0-9]
to denote that gzip compression was requested. This specific behaviour is
maintained. A newly introduced option --compress-program=lz4 can be used to ask
for the logs to be compressed using lz4 instead. In that case, no compression
values can be selected as it does not seem too useful.
Under the hood there is nothing exceptional to be noted. Tar based archives have
not yet been taught to use lz4 compression. Those are used by pg_basebackup. If
is is felt useful, then it is easy to be added in a new patch.
Cheers,
//Georgios
Attachments:
v1-0001-Teach-pg_receivewal-to-use-lz4-compression.patchapplication/octet-stream; name=v1-0001-Teach-pg_receivewal-to-use-lz4-compression.patchDownload
From 857f48859aa8ebbe6daa5b80e2f51bfb96e3979c Mon Sep 17 00:00:00 2001
From: Georgios Kokolatos <gkokolatos@pm.me>
Date: Tue, 29 Jun 2021 14:27:51 +0000
Subject: Teach pg_receivewal to use lz4 compression
---
src/bin/pg_basebackup/pg_basebackup.c | 7 +-
src/bin/pg_basebackup/pg_receivewal.c | 68 ++++++-
src/bin/pg_basebackup/t/020_pg_receivewal.pl | 38 +++-
src/bin/pg_basebackup/walmethods.c | 175 +++++++++++++++++--
src/bin/pg_basebackup/walmethods.h | 12 +-
5 files changed, 278 insertions(+), 22 deletions(-)
diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c
index 16d8929b23..6b8734d8ba 100644
--- a/src/bin/pg_basebackup/pg_basebackup.c
+++ b/src/bin/pg_basebackup/pg_basebackup.c
@@ -553,10 +553,13 @@ LogStreamerMain(logstreamer_param *param)
stream.replication_slot = replication_slot;
if (format == 'p')
- stream.walmethod = CreateWalDirectoryMethod(param->xlog, 0,
+ stream.walmethod = CreateWalDirectoryMethod(param->xlog,
+ COMPRESSION_NONE, 0,
stream.do_sync);
else
- stream.walmethod = CreateWalTarMethod(param->xlog, compresslevel,
+ stream.walmethod = CreateWalTarMethod(param->xlog,
+ COMPRESSION_NONE /* argument is ignored */,
+ compresslevel,
stream.do_sync);
if (!ReceiveXlogStream(param->bgconn, &stream))
diff --git a/src/bin/pg_basebackup/pg_receivewal.c b/src/bin/pg_basebackup/pg_receivewal.c
index 0d15012c29..6759e3e747 100644
--- a/src/bin/pg_basebackup/pg_receivewal.c
+++ b/src/bin/pg_basebackup/pg_receivewal.c
@@ -43,6 +43,7 @@ static bool do_drop_slot = false;
static bool do_sync = true;
static bool synchronous = false;
static char *replication_slot = NULL;
+static WalCompressionProgram compression_program = COMPRESSION_NONE;
static XLogRecPtr endpos = InvalidXLogRecPtr;
@@ -90,7 +91,8 @@ usage(void)
printf(_(" --synchronous flush write-ahead log immediately after writing\n"));
printf(_(" -v, --verbose output verbose messages\n"));
printf(_(" -V, --version output version information, then exit\n"));
- printf(_(" -Z, --compress=0-9 compress logs with given compression level\n"));
+ printf(_(" -I, --compress-program use this program for compression\n"));
+ printf(_(" -Z, --compress=0-9 compress logs with given compression level (available only with compress-program=zlib)\n"));
printf(_(" -?, --help show this help, then exit\n"));
printf(_("\nConnection options:\n"));
printf(_(" -d, --dbname=CONNSTR connection string\n"));
@@ -429,7 +431,9 @@ StreamLog(void)
stream.synchronous = synchronous;
stream.do_sync = do_sync;
stream.mark_done = false;
- stream.walmethod = CreateWalDirectoryMethod(basedir, compresslevel,
+ stream.walmethod = CreateWalDirectoryMethod(basedir,
+ compression_program,
+ compresslevel,
stream.do_sync);
stream.partial_suffix = ".partial";
stream.replication_slot = replication_slot;
@@ -482,6 +486,7 @@ main(int argc, char **argv)
{"status-interval", required_argument, NULL, 's'},
{"slot", required_argument, NULL, 'S'},
{"verbose", no_argument, NULL, 'v'},
+ {"compress-program", required_argument, NULL, 'I'},
{"compress", required_argument, NULL, 'Z'},
/* action */
{"create-slot", no_argument, NULL, 1},
@@ -573,6 +578,21 @@ main(int argc, char **argv)
case 'v':
verbose++;
break;
+ case 'I':
+ if (strcmp(optarg, "zlib") == 0)
+ {
+ compression_program = COMPRESSION_ZLIB;
+ }
+ else if (strcmp(optarg, "lz4") == 0)
+ {
+ compression_program = COMPRESSION_LZ4;
+ }
+ else
+ {
+ pg_log_error("invalid compress-program \"%s\"", optarg);
+ exit(1);
+ }
+ break;
case 'Z':
compresslevel = atoi(optarg);
if (compresslevel < 0 || compresslevel > 9)
@@ -657,14 +677,56 @@ main(int argc, char **argv)
exit(1);
}
+ if (compression_program != COMPRESSION_NONE)
+ {
+#ifndef HAVE_LIBZ
+ if (compression_program == COMPRESSION_ZLIB)
+ {
+ pg_log_error("this build does not support compression via zlib");
+ exit(1);
+ }
+#endif
+#ifndef HAVE_LIBLZ4
+ if (compression_program == COMPRESSION_LZ4)
+ {
+ pg_log_error("this build does not support compression via lz4");
+ exit(1);
+ }
+#endif
+ }
+
#ifndef HAVE_LIBZ
if (compresslevel != 0)
{
- pg_log_error("this build does not support compression");
+ pg_log_error("this build does not support compression via zlib");
exit(1);
}
#endif
+ if (compresslevel != 0)
+ {
+ if (compression_program == COMPRESSION_NONE)
+ {
+ compression_program = COMPRESSION_ZLIB;
+ }
+ if (compression_program != COMPRESSION_ZLIB)
+ {
+ pg_log_error("cannot use --compress when "
+ "--compress_program is not zlib");
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+ progname);
+ exit(1);
+ }
+ }
+ else if (compression_program == COMPRESSION_ZLIB)
+ {
+ pg_log_error("cannot use --compress_program zlib when "
+ "--compression is 0");
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+ progname);
+ exit(1);
+ }
+
/*
* Check existence of destination folder.
*/
diff --git a/src/bin/pg_basebackup/t/020_pg_receivewal.pl b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
index a547c97ef1..0e27cf030c 100644
--- a/src/bin/pg_basebackup/t/020_pg_receivewal.pl
+++ b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
@@ -5,7 +5,7 @@ use strict;
use warnings;
use TestLib;
use PostgresNode;
-use Test::More tests => 19;
+use Test::More tests => 22;
program_help_ok('pg_receivewal');
program_version_ok('pg_receivewal');
@@ -33,6 +33,13 @@ $primary->command_fails(
$primary->command_fails(
[ 'pg_receivewal', '-D', $stream_dir, '--synchronous', '--no-sync' ],
'failure if --synchronous specified with --no-sync');
+$primary->command_fails(
+ [
+ 'pg_receivewal', '-D', $stream_dir, '--compress_program', 'lz4',
+ '--compress', '0'
+ ],
+ 'failure if --compress_program=lz4 specified with --compress');
+
# Slot creation and drop
my $slot_name = 'test';
@@ -66,6 +73,35 @@ $primary->command_ok(
],
'streaming some WAL with --synchronous');
+# Check lz4 compression if available
+SKIP:
+{
+ skip "postgres was not build with LZ4 support", 2
+ if (!check_pg_config("#define HAVE_LIBLZ4 1"));
+
+ # Generate some WAL.
+ $primary->psql('postgres', 'SELECT pg_switch_wal();');
+ $nextlsn =
+ $primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
+ chomp($nextlsn);
+ $primary->psql('postgres',
+ 'INSERT INTO test_table VALUES (generate_series(100,200));');
+ $primary->psql('postgres', 'SELECT pg_switch_wal();');
+
+ # Stream up to the given position
+ $primary->command_ok(
+ [
+ 'pg_receivewal', '-D', $stream_dir, '--verbose',
+ '--endpos', $nextlsn, '--compress-program=lz4'
+ ],
+ 'streaming some WAL with --compress-program=lz4');
+
+ # Verify that the stored file is compressed and readable
+ my @lz4_wals = glob "$stream_dir/*.lz4";
+ is(scalar(@lz4_wals), 1, 'one lz4 compressed WAL was created');
+ system_or_bail('lz4', '-t', $lz4_wals[0]);
+}
+
# Permissions on WAL files should be default
SKIP:
{
diff --git a/src/bin/pg_basebackup/walmethods.c b/src/bin/pg_basebackup/walmethods.c
index a15bbb20e7..18d5bf3e59 100644
--- a/src/bin/pg_basebackup/walmethods.c
+++ b/src/bin/pg_basebackup/walmethods.c
@@ -17,6 +17,10 @@
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
+
+#ifdef HAVE_LIBLZ4
+#include <lz4frame.h>
+#endif
#ifdef HAVE_LIBZ
#include <zlib.h>
#endif
@@ -30,6 +34,9 @@
/* Size of zlib buffer for .tar.gz */
#define ZLIB_OUT_SIZE 4096
+/* Size of lz4 input chunk for .lz4 */
+#define LZ4_IN_SIZE 4096
+
/*-------------------------------------------------------------------------
* WalDirectoryMethod - write wal to a directory looking like pg_wal
*-------------------------------------------------------------------------
@@ -40,9 +47,10 @@
*/
typedef struct DirectoryMethodData
{
- char *basedir;
- int compression;
- bool sync;
+ char *basedir;
+ WalCompressionProgram compression_program;
+ int compression;
+ bool sync;
} DirectoryMethodData;
static DirectoryMethodData *dir_data = NULL;
@@ -59,6 +67,11 @@ typedef struct DirectoryMethodFile
#ifdef HAVE_LIBZ
gzFile gzfp;
#endif
+#ifdef HAVE_LIBLZ4
+ LZ4F_compressionContext_t ctx;
+ size_t outbufCapacity;
+ void *outbuf;
+#endif
} DirectoryMethodFile;
static const char *
@@ -77,10 +90,16 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
#ifdef HAVE_LIBZ
gzFile gzfp = NULL;
#endif
+#ifdef HAVE_LIBLZ4
+ LZ4F_compressionContext_t ctx = NULL;
+ size_t outbufCapacity;
+ void *outbuf = NULL;
+#endif
snprintf(tmppath, sizeof(tmppath), "%s/%s%s%s",
dir_data->basedir, pathname,
- dir_data->compression > 0 ? ".gz" : "",
+ dir_data->compression_program == COMPRESSION_ZLIB ? ".gz" :
+ dir_data->compression_program == COMPRESSION_LZ4 ? ".lz4": "",
temp_suffix ? temp_suffix : "");
/*
@@ -94,7 +113,7 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
return NULL;
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_program == COMPRESSION_ZLIB)
{
gzfp = gzdopen(fd, "wb");
if (gzfp == NULL)
@@ -111,9 +130,48 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
}
}
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_program == COMPRESSION_LZ4)
+ {
+ size_t ctx_out;
+ size_t header_size;
+
+ ctx_out = LZ4F_createCompressionContext(&ctx, LZ4F_VERSION);
+ outbufCapacity = LZ4F_compressBound(LZ4_IN_SIZE, NULL /* default preferences */);
+ if (LZ4F_isError(ctx_out))
+ {
+ close(fd);
+ return NULL;
+ }
+
+ outbuf = pg_malloc0(outbufCapacity);
+
+ /* add the header */
+ header_size = LZ4F_compressBegin(ctx, outbuf, outbufCapacity, NULL);
+ if (LZ4F_isError(header_size))
+ {
+ close(fd);
+ return NULL;
+ }
+
+ errno = 0;
+ if (write(fd, outbuf, header_size) != header_size)
+ {
+ int save_errno = errno;
+
+ close(fd);
+
+ /*
+ * If write didn't set errno, assume problem is no disk space.
+ */
+ errno = save_errno ? save_errno : ENOSPC;
+ return NULL;
+ }
+ }
+#endif
/* Do pre-padding on non-compressed files */
- if (pad_to_size && dir_data->compression == 0)
+ if (pad_to_size && dir_data->compression_program == COMPRESSION_NONE)
{
PGAlignedXLogBlock zerobuf;
int bytes;
@@ -158,7 +216,7 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
fsync_parent_path(tmppath) != 0)
{
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_program == COMPRESSION_ZLIB)
gzclose(gzfp);
else
#endif
@@ -169,9 +227,18 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
f = pg_malloc0(sizeof(DirectoryMethodFile));
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_program == COMPRESSION_ZLIB)
f->gzfp = gzfp;
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_program == COMPRESSION_LZ4)
+ {
+ f->ctx = ctx;
+ f->outbuf = outbuf;
+ f->outbufCapacity = outbufCapacity;
+ }
+#endif
+
f->fd = fd;
f->currpos = 0;
f->pathname = pg_strdup(pathname);
@@ -191,9 +258,46 @@ dir_write(Walfile f, const void *buf, size_t count)
Assert(f != NULL);
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_program == COMPRESSION_ZLIB)
r = (ssize_t) gzwrite(df->gzfp, buf, count);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_program == COMPRESSION_LZ4)
+ {
+ size_t chunk;
+ size_t remaining;
+ const void *inbuf = buf;
+
+ remaining = count;
+ while (remaining > 0)
+ {
+ size_t compressed;
+
+ if (remaining > LZ4_IN_SIZE)
+ chunk = LZ4_IN_SIZE;
+ else
+ chunk = remaining;
+
+ remaining -= chunk;
+ compressed = LZ4F_compressUpdate(df->ctx,
+ df->outbuf, df->outbufCapacity,
+ inbuf, chunk,
+ NULL);
+
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->outbuf, compressed) != compressed)
+ return -1;
+
+ inbuf = ((char *)inbuf) + chunk;
+ }
+
+ /* XXX: This is what our caller expects, but it is not nice at all */
+ r = (ssize_t)count;
+ }
+ else
#endif
r = write(df->fd, buf, count);
if (r > 0)
@@ -221,9 +325,30 @@ dir_close(Walfile f, WalCloseMethod method)
Assert(f != NULL);
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_program == COMPRESSION_ZLIB)
r = gzclose(df->gzfp);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_program == COMPRESSION_LZ4)
+ {
+ /* Flush any internal buffers */
+ size_t compressed = LZ4F_compressEnd(df->ctx,
+ df->outbuf, df->outbufCapacity,
+ NULL);
+ if (LZ4F_isError(compressed))
+ {
+ return -1;
+ }
+
+ if (write(df->fd, df->outbuf, compressed) != compressed)
+ {
+ return -1;
+ }
+
+ r = close(df->fd);
+ }
+ else
#endif
r = close(df->fd);
@@ -238,11 +363,13 @@ dir_close(Walfile f, WalCloseMethod method)
*/
snprintf(tmppath, sizeof(tmppath), "%s/%s%s%s",
dir_data->basedir, df->pathname,
- dir_data->compression > 0 ? ".gz" : "",
+ dir_data->compression_program == COMPRESSION_ZLIB ? ".gz" :
+ dir_data->compression_program == COMPRESSION_LZ4 ? ".lz4": "",
df->temp_suffix);
snprintf(tmppath2, sizeof(tmppath2), "%s/%s%s",
dir_data->basedir, df->pathname,
- dir_data->compression > 0 ? ".gz" : "");
+ dir_data->compression_program == COMPRESSION_ZLIB ? ".gz" :
+ dir_data->compression_program == COMPRESSION_LZ4 ? ".lz4": "");
r = durable_rename(tmppath, tmppath2);
}
else if (method == CLOSE_UNLINK)
@@ -250,7 +377,8 @@ dir_close(Walfile f, WalCloseMethod method)
/* Unlink the file once it's closed */
snprintf(tmppath, sizeof(tmppath), "%s/%s%s%s",
dir_data->basedir, df->pathname,
- dir_data->compression > 0 ? ".gz" : "",
+ dir_data->compression_program == COMPRESSION_ZLIB ? ".gz" :
+ dir_data->compression_program == COMPRESSION_LZ4 ? ".lz4": "",
df->temp_suffix ? df->temp_suffix : "");
r = unlink(tmppath);
}
@@ -270,6 +398,12 @@ dir_close(Walfile f, WalCloseMethod method)
}
}
+#ifdef HAVE_LIBLZ4
+ pg_free(df->outbuf);
+ /* supports free on NULL */
+ LZ4F_freeCompressionContext(df->ctx);
+#endif
+
pg_free(df->pathname);
pg_free(df->fullpath);
if (df->temp_suffix)
@@ -346,7 +480,9 @@ dir_finish(void)
WalWriteMethod *
-CreateWalDirectoryMethod(const char *basedir, int compression, bool sync)
+CreateWalDirectoryMethod(const char *basedir,
+ WalCompressionProgram compression_program,
+ int compression, bool sync)
{
WalWriteMethod *method;
@@ -362,6 +498,7 @@ CreateWalDirectoryMethod(const char *basedir, int compression, bool sync)
method->getlasterror = dir_getlasterror;
dir_data = pg_malloc0(sizeof(DirectoryMethodData));
+ dir_data->compression_program = compression_program;
dir_data->compression = compression;
dir_data->basedir = pg_strdup(basedir);
dir_data->sync = sync;
@@ -983,8 +1120,16 @@ tar_finish(void)
return true;
}
+/*
+ * The argument compression_program is currently ignored. It is in place for
+ * symmetry with CreateWalDirectoryMethod which uses it for distinguishing
+ * between the different compression methods. CreateWalTarMethod and its family
+ * of functions handle only zlib compression.
+ */
WalWriteMethod *
-CreateWalTarMethod(const char *tarbase, int compression, bool sync)
+CreateWalTarMethod(const char *tarbase,
+ WalCompressionProgram compression_program,
+ int compression, bool sync)
{
WalWriteMethod *method;
const char *suffix = (compression != 0) ? ".tar.gz" : ".tar";
diff --git a/src/bin/pg_basebackup/walmethods.h b/src/bin/pg_basebackup/walmethods.h
index fc4bb52cb7..f7d8582aad 100644
--- a/src/bin/pg_basebackup/walmethods.h
+++ b/src/bin/pg_basebackup/walmethods.h
@@ -19,6 +19,13 @@ typedef enum
CLOSE_NO_RENAME
} WalCloseMethod;
+typedef enum
+{
+ COMPRESSION_LZ4,
+ COMPRESSION_ZLIB,
+ COMPRESSION_NONE
+} WalCompressionProgram;
+
/*
* A WalWriteMethod structure represents the different methods used
* to write the streaming WAL as it's received.
@@ -86,8 +93,11 @@ struct WalWriteMethod
* not all those required for pg_receivewal)
*/
WalWriteMethod *CreateWalDirectoryMethod(const char *basedir,
+ WalCompressionProgram compression_program,
int compression, bool sync);
-WalWriteMethod *CreateWalTarMethod(const char *tarbase, int compression, bool sync);
+WalWriteMethod *CreateWalTarMethod(const char *tarbase,
+ WalCompressionProgram compression_program,
+ int compression, bool sync);
/* Cleanup routines for previously-created methods */
void FreeWalDirectoryMethod(void);
--
2.25.1
Re: Teach pg_receivewal to use lz4 compression
On Tue, Jun 29, 2021 at 02:45:17PM +0000, gkokolatos@pm.me wrote:
The program pg_receivewal can use gzip compression to store the received WAL.
This patch teaches it to be able to use lz4 compression if the binary is build
using the -llz4 flag.
Nice.
Previously, the user had to use the option --compress with a value between [0-9]
to denote that gzip compression was requested. This specific behaviour is
maintained. A newly introduced option --compress-program=lz4 can be used to ask
for the logs to be compressed using lz4 instead. In that case, no compression
values can be selected as it does not seem too useful.
Yes, I am not convinced either that we should care about making the
acceleration customizable.
Under the hood there is nothing exceptional to be noted. Tar based archives have
not yet been taught to use lz4 compression. Those are used by pg_basebackup. If
is is felt useful, then it is easy to be added in a new patch.
Documentation is missing from the patch.
+ LZ4F_compressionContext_t ctx;
+ size_t outbufCapacity;
+ void *outbuf;
It may be cleaner to refer to lz4 in the name of those variables?+ ctx_out = LZ4F_createCompressionContext(&ctx, LZ4F_VERSION);
+ outbufCapacity = LZ4F_compressBound(LZ4_IN_SIZE, NULL /* default preferences */);
Interesting. So this cannot be done at compilation time because of
the auto-flush mode looking at the LZ4 code. That looks about right.getopt_long() is forgotting the new option 'I'.
+ system_or_bail('lz4', '-t', $lz4_wals[0]);
I think that you should just drop this part of the test. The only
part of LZ4 that we require to be present when Postgres is built with
--with-lz4 is its library liblz4. Commands associated to it may not
be around, causing this test to fail. The test checking that one .lz4
file has been created is good to have. It may be worth adding a test
with a .lz4.partial segment generated and --endpos pointing to a LSN
that does not finish the segment that gets switched.It seems to me that you are missing some logic in
FindStreamingStart() to handle LZ4-compressed segments, in relation
with IsCompressXLogFileName() and IsPartialCompressXLogFileName().
+ pg_log_error("invalid compress-program \"%s\"", optarg);
"compress-program" sounds weird. Shouldn't that just say "invalid
compression method" or similar?
+ printf(_(" -Z, --compress=0-9 compress logs with given
compression level (available only with compress-program=zlib)\n"));
This line is too long.
Should we have more tests for ZLIB, while on it? That seems like a
good addition as long as we can skip the tests conditionally when
that's not supported.
--
Michael
Re: Teach pg_receivewal to use lz4 compression
On Tue, Jun 29, 2021 at 8:15 PM <gkokolatos@pm.me> wrote:
Hi,
The program pg_receivewal can use gzip compression to store the received WAL.
This patch teaches it to be able to use lz4 compression if the binary is build
using the -llz4 flag.
+1 for the idea
Some comments/suggestions on the patch
1.
@@ -90,7 +91,8 @@ usage(void)
printf(_(" --synchronous flush write-ahead log immediately
after writing\n"));
printf(_(" -v, --verbose output verbose messages\n"));
printf(_(" -V, --version output version information, then exit\n"));
- printf(_(" -Z, --compress=0-9 compress logs with given
compression level\n"));
+ printf(_(" -I, --compress-program use this program for compression\n"));Wouldn't it be better to call it compression method instead of
compression program?
2.
+ printf(_(" -Z, --compress=0-9 compress logs with given
compression level (available only with compress-program=zlib)\n"));
I think we can somehow use "acceleration" parameter of lz4 compression
to map on compression level, It is not direct mapping but
can't we create some internal mapping instead of completely ignoring
this option for lz4, or we can provide another option for lz4?
3. Should we also support LZ4 compression using dictionary?
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Re: Teach pg_receivewal to use lz4 compression
On Wed, Jun 30, 2021 at 8:34 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Tue, Jun 29, 2021 at 8:15 PM <gkokolatos@pm.me> wrote:
Hi,
The program pg_receivewal can use gzip compression to store the received WAL.
This patch teaches it to be able to use lz4 compression if the binary is build
using the -llz4 flag.+1 for the idea
Some comments/suggestions on the patch
1. @@ -90,7 +91,8 @@ usage(void) printf(_(" --synchronous flush write-ahead log immediately after writing\n")); printf(_(" -v, --verbose output verbose messages\n")); printf(_(" -V, --version output version information, then exit\n")); - printf(_(" -Z, --compress=0-9 compress logs with given compression level\n")); + printf(_(" -I, --compress-program use this program for compression\n"));Wouldn't it be better to call it compression method instead of
compression program?
I came here to say exactly that, just had to think up what I thought
was the better name first. Either method or algorithm, but method
seems like the much simpler choice and therefore better in this case.
Should is also then not be --compression-method, rather than --compress-method?
--
Magnus Hagander
Me: https://www.hagander.net/
Work: https://www.redpill-linpro.com/
Re: Teach pg_receivewal to use lz4 compression
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday, July 1st, 2021 at 12:28, Magnus Hagander <magnus@hagander.net> wrote:
On Wed, Jun 30, 2021 at 8:34 AM Dilip Kumar dilipbalaut@gmail.com wrote:
On Tue, Jun 29, 2021 at 8:15 PM gkokolatos@pm.me wrote:
Hi,
The program pg_receivewal can use gzip compression to store the received WAL.
This patch teaches it to be able to use lz4 compression if the binary is build
using the -llz4 flag.
+1 for the idea
Some comments/suggestions on the patch
@@ -90,7 +91,8 @@ usage(void)
printf((" --synchronous flush write-ahead log immediately
after writing\n"));
printf((" -v, --verbose output verbose messages\n"));
printf(_(" -V, --version output version information, then exit\n"));
- printf(_(" -Z, --compress=0-9 compress logs with given
compression level\n"));
- printf(_(" -I, --compress-program use this program for compression\n"));
Wouldn't it be better to call it compression method instead of
compression program?
I came here to say exactly that, just had to think up what I thought
was the better name first. Either method or algorithm, but method
seems like the much simpler choice and therefore better in this case.
Should is also then not be --compression-method, rather than --compress-method?
Not a problem. To be very transparent, I first looked what was already out there.
For example `tar` is using
-I, --use-compress-program=PROG
yet the 'use-' bit would push the alignment of the --help output, so I removed it.
To me, as a non native English speaker, `--compression-method` does sound better.
I can just re-align the rest of the help output.
Updated patch is on the making.
Show quoted text
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Magnus Hagander
Re: Teach pg_receivewal to use lz4 compression
On Thu, Jul 1, 2021 at 3:39 PM <gkokolatos@pm.me> wrote:
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday, July 1st, 2021 at 12:28, Magnus Hagander <magnus@hagander.net> wrote:
On Wed, Jun 30, 2021 at 8:34 AM Dilip Kumar dilipbalaut@gmail.com wrote:
On Tue, Jun 29, 2021 at 8:15 PM gkokolatos@pm.me wrote:
Hi,
The program pg_receivewal can use gzip compression to store the received WAL.
This patch teaches it to be able to use lz4 compression if the binary is build
using the -llz4 flag.
+1 for the idea
Some comments/suggestions on the patch
@@ -90,7 +91,8 @@ usage(void)
printf((" --synchronous flush write-ahead log immediately
after writing\n"));
printf((" -v, --verbose output verbose messages\n"));
printf(_(" -V, --version output version information, then exit\n"));
- printf(_(" -Z, --compress=0-9 compress logs with given
compression level\n"));
- printf(_(" -I, --compress-program use this program for compression\n"));
Wouldn't it be better to call it compression method instead of
compression program?
I came here to say exactly that, just had to think up what I thought
was the better name first. Either method or algorithm, but method
seems like the much simpler choice and therefore better in this case.
Should is also then not be --compression-method, rather than --compress-method?
Not a problem. To be very transparent, I first looked what was already out there.
For example `tar` is using
-I, --use-compress-program=PROG
yet the 'use-' bit would push the alignment of the --help output, so I removed it.
I think the difference there is that tar actually calls an external
program to do the work... And we are using the built-in library,
right?
--
Magnus Hagander
Me: https://www.hagander.net/
Work: https://www.redpill-linpro.com/
Re: Teach pg_receivewal to use lz4 compression
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday, July 1st, 2021 at 15:58, Magnus Hagander <magnus@hagander.net> wrote:
On Thu, Jul 1, 2021 at 3:39 PM gkokolatos@pm.me wrote:
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday, July 1st, 2021 at 12:28, Magnus Hagander magnus@hagander.net wrote:
On Wed, Jun 30, 2021 at 8:34 AM Dilip Kumar dilipbalaut@gmail.com wrote:
On Tue, Jun 29, 2021 at 8:15 PM gkokolatos@pm.me wrote:
Hi,
The program pg_receivewal can use gzip compression to store the received WAL.
This patch teaches it to be able to use lz4 compression if the binary is build
using the -llz4 flag.
+1 for the idea
Some comments/suggestions on the patch
@@ -90,7 +91,8 @@ usage(void)
printf((" --synchronous flush write-ahead log immediately
after writing\n"));
printf((" -v, --verbose output verbose messages\n"));
printf(_(" -V, --version output version information, then exit\n"));
- printf(_(" -Z, --compress=0-9 compress logs with given
compression level\n"));
- printf(_(" -I, --compress-program use this program for compression\n"));
Wouldn't it be better to call it compression method instead of
compression program?
I came here to say exactly that, just had to think up what I thought
was the better name first. Either method or algorithm, but method
seems like the much simpler choice and therefore better in this case.
Should is also then not be --compression-method, rather than --compress-method?
Not a problem. To be very transparent, I first looked what was already out there.
For example `tar` is using
-I, --use-compress-program=PROG
yet the 'use-' bit would push the alignment of the --help output, so I removed it.
I think the difference there is that tar actually calls an external
program to do the work... And we are using the built-in library,
right?
You are very correct :) I am not objecting the change at all. Just let you know
how I chose that. You know, naming is dead easy and all...
On a more serious note, what about the `-I` short flag? Should we keep it or
is there a better one to be used?
Micheal suggested on the same thread to move my entry in the help output so that
the output remains ordered. I would like the options for the compression method and
the already existing compression level to next to each other if possible. Then it
should be either 'X' or 'Y'.
Thoughts?
Show quoted text
------------------------------------------------------------------------------------------------------------------------------------------------
Magnus Hagander
Re: Teach pg_receivewal to use lz4 compression
On Thu, Jul 01, 2021 at 02:10:17PM +0000, gkokolatos@pm.me wrote:
Micheal suggested on the same thread to move my entry in the help output so that
the output remains ordered. I would like the options for the compression method and
the already existing compression level to next to each other if possible. Then it
should be either 'X' or 'Y'.
Hmm. Grouping these together makes sense for the user. One choice
that we have here is to drop the short option, and just use a long
one. What I think is important for the user when it comes to this
option is the consistency of its naming across all the tools
supporting it. pg_dump and pg_basebackup, where we could plug LZ4,
already use most of the short options you could use for pg_receivewal,
having only a long one gives a bit more flexibility.
--
Michael
Re: Teach pg_receivewal to use lz4 compression
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, July 2nd, 2021 at 03:10, Michael Paquier <michael@paquier.xyz> wrote:
On Thu, Jul 01, 2021 at 02:10:17PM +0000, gkokolatos@pm.me wrote:
Micheal suggested on the same thread to move my entry in the help output so that
the output remains ordered. I would like the options for the compression method and
the already existing compression level to next to each other if possible. Then it
should be either 'X' or 'Y'.
Hmm. Grouping these together makes sense for the user. One choice
that we have here is to drop the short option, and just use a long
one. What I think is important for the user when it comes to this
option is the consistency of its naming across all the tools
supporting it. pg_dump and pg_basebackup, where we could plug LZ4,
already use most of the short options you could use for pg_receivewal,
having only a long one gives a bit more flexibility.
Good point. I am going with that one.
Show quoted text
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Michael
Re: Teach pg_receivewal to use lz4 compression
Hi,
please find v2 of the patch which tries to address the commends received so far.
Thank you all for your comments.
Michael Paquier wrote:
Documentation is missing from the patch.
It has now been added.
+ LZ4F_compressionContext_t ctx; + size_t outbufCapacity; + void *outbuf; It may be cleaner to refer to lz4 in the name of those variables?
Agreed and done
+ ctx_out = LZ4F_createCompressionContext(&ctx, LZ4F_VERSION); + outbufCapacity = LZ4F_compressBound(LZ4_IN_SIZE, NULL /* default preferences */); Interesting. So this cannot be done at compilation time because of the auto-flush mode looking at the LZ4 code. That looks about right.
This is also my understanding.
+ system_or_bail('lz4', '-t', $lz4_wals[0]); I think that you should just drop this part of the test. The only part of LZ4 that we require to be present when Postgres is built with --with-lz4 is its library liblz4. Commands associated to it may not be around, causing this test to fail. The test checking that one .lz4 file has been created is good to have. It may be worth adding a test with a .lz4.partial segment generated and --endpos pointing to a LSN that does not finish the segment that gets switched.
I humbly disagree with the need for the test. It is rather easily possible
to generate a file that can not be decoded, thus becoming useless. Having the
test will provide some guarantee for the fact. In the current patch, there
is code to find out if the program lz4 is available in the system. If it is
not available, then that specific test is skipped. The rest remains as it
were. Also `system_or_bail` is not used anymore in favour of the `system_log`
so that the test counted and the execution of tests continues upon failure.
It seems to me that you are missing some logic in
FindStreamingStart() to handle LZ4-compressed segments, in relation
with IsCompressXLogFileName() and IsPartialCompressXLogFileName().
Very correct. The logic is now added. Given the lz4 api, one has to fill
in the uncompressed size during creation time. Then one can read the
headers and verify the expectations.
Should we have more tests for ZLIB, while on it? That seems like a
good addition as long as we can skip the tests conditionally when
that's not supported.
Agreed. Please allow me to provide a distinct patch for this code.
Dilip Kumar wrote:
Wouldn't it be better to call it compression method instead of
compression program?
Agreed. This is inline with the suggestions of other reviewers.
Find the change in the attached patch.
I think we can somehow use "acceleration" parameter of lz4 compression
to map on compression level, It is not direct mapping but
can't we create some internal mapping instead of completely ignoring
this option for lz4, or we can provide another option for lz4?
We can, though I am not in favour of doing so. There is seemingly
little benefit for added complexity.
Should we also support LZ4 compression using dictionary?
I would we should not do that. If my understanding is correct,
decompression would require the dictionary to be passed along.
The algorithm seems to be very competitive to the current compression
as is.
Magnus Hagander wrote:
I came here to say exactly that, just had to think up what I thought
was the better name first. Either method or algorithm, but method
seems like the much simpler choice and therefore better in this case.Should is also then not be --compression-method, rather than --compress-method?
Agreed and changed throughout.
Michael Paquier wrote:
What I think is important for the user when it comes to this
option is the consistency of its naming across all the tools
supporting it. pg_dump and pg_basebackup, where we could plug LZ4,
already use most of the short options you could use for pg_receivewal,
having only a long one gives a bit more flexibility.
Done.
Cheers,
//Georgios
Attachments:
v2-0001-Teach-pg_receivewal-to-use-lz4-compression.patchapplication/octet-stream; name=v2-0001-Teach-pg_receivewal-to-use-lz4-compression.patchDownload
From 8f44cb1723fb6ca749ac5d107152b95e2319eda8 Mon Sep 17 00:00:00 2001
From: Georgios Kokolatos <gkokolatos@pm.me>
Date: Thu, 8 Jul 2021 13:47:37 +0000
Subject: [PATCH v2] Teach pg_receivewal to use lz4 compression
The program pg_receivewal can use gzip compression to store the received WAL.
This commit teaches it to also be able to use lz4 compression. It is required
that the binary is build using the -llz4 flag. It is enabled via the --with-lz4
flag on configuration time.
Previously, the user had to use the option --compress with a value between [0-9]
to denote that gzip compression was required. This specific behaviour is
maintained. A newly introduced option --compression-method=lz4 can be used to ask
for the logs to be compressed with lz4. In that case, no compression values can
be selected as it did not seem too useful.
Under the hood there is nothing exceptional to be noted. Tar based archives have
not yet been taught to use lz4 compression. If that is felt useful, then it is
easy to be added in the future.
Tests have been added to verify the creation and correctness of the generated
lz4 files. The later is achieved by the use of lz4 program. Autoconf has been
taught to unconditionally recognize the existance of the program and propagate
the information to the tests.
---
configure | 55 ++++++
configure.ac | 2 +
doc/src/sgml/ref/pg_receivewal.sgml | 28 ++-
src/Makefile.global.in | 1 +
src/bin/pg_basebackup/Makefile | 3 +-
src/bin/pg_basebackup/pg_basebackup.c | 7 +-
src/bin/pg_basebackup/pg_receivewal.c | 196 +++++++++++++++++--
src/bin/pg_basebackup/t/020_pg_receivewal.pl | 46 ++++-
src/bin/pg_basebackup/walmethods.c | 188 ++++++++++++++++--
src/bin/pg_basebackup/walmethods.h | 12 +-
10 files changed, 495 insertions(+), 43 deletions(-)
diff --git a/configure b/configure
index e468def49e..4fd6652d1c 100755
--- a/configure
+++ b/configure
@@ -699,6 +699,7 @@ with_gnu_ld
LD
LDFLAGS_SL
LDFLAGS_EX
+LZ4
LZ4_LIBS
LZ4_CFLAGS
with_lz4
@@ -9563,6 +9564,60 @@ $as_echo_n "checking for TAR... " >&6; }
$as_echo "$TAR" >&6; }
fi
+if test -z "$LZ4"; then
+ for ac_prog in lz4
+do
+ # Extract the first word of "$ac_prog", so it can be a program name with args.
+set dummy $ac_prog; ac_word=$2
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
+$as_echo_n "checking for $ac_word... " >&6; }
+if ${ac_cv_path_LZ4+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ case $LZ4 in
+ [\\/]* | ?:[\\/]*)
+ ac_cv_path_LZ4="$LZ4" # Let the user override the test with a path.
+ ;;
+ *)
+ as_save_IFS=$IFS; IFS=$PATH_SEPARATOR
+for as_dir in $PATH
+do
+ IFS=$as_save_IFS
+ test -z "$as_dir" && as_dir=.
+ for ac_exec_ext in '' $ac_executable_extensions; do
+ if as_fn_executable_p "$as_dir/$ac_word$ac_exec_ext"; then
+ ac_cv_path_LZ4="$as_dir/$ac_word$ac_exec_ext"
+ $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5
+ break 2
+ fi
+done
+ done
+IFS=$as_save_IFS
+
+ ;;
+esac
+fi
+LZ4=$ac_cv_path_LZ4
+if test -n "$LZ4"; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: $LZ4" >&5
+$as_echo "$LZ4" >&6; }
+else
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+fi
+
+
+ test -n "$LZ4" && break
+done
+
+else
+ # Report the value of LZ4 in configure's output in all cases.
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for lz4" >&5
+$as_echo_n "checking for lz4... " >&6; }
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: $LZ4" >&5
+$as_echo "$LZ4" >&6; }
+fi
+
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether ln -s works" >&5
$as_echo_n "checking whether ln -s works... " >&6; }
LN_S=$as_ln_s
diff --git a/configure.ac b/configure.ac
index 39666f9727..d6db7d1b80 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1053,6 +1053,8 @@ case $MKDIR_P in
*install-sh*) MKDIR_P='\${SHELL} \${top_srcdir}/config/install-sh -c -d';;
esac
+PGAC_PATH_PROGS(LZ4, lz4)
+
PGAC_PATH_BISON
PGAC_PATH_FLEX
diff --git a/doc/src/sgml/ref/pg_receivewal.sgml b/doc/src/sgml/ref/pg_receivewal.sgml
index 45b544cf49..1b49884247 100644
--- a/doc/src/sgml/ref/pg_receivewal.sgml
+++ b/doc/src/sgml/ref/pg_receivewal.sgml
@@ -229,15 +229,35 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--compression-method=<replaceable class="parameter">level</replaceable></option></term>
+ <listitem>
+ <para>
+ Enables compression of write-ahead logs using the specified method.
+ Supported methods are <literal>lz4</literal> and
+ <literal>gzip</literal>.
+ The suffix <filename>.lz4</filename> or <filename>.gz</filename> will
+ automatically be added to all filenames for each method respectevilly.
+ For the <literal>lz4</literal> method to be available,
+ <productname>PostgreSQL</productname> must have been have been compiled
+ with <option>--with-lz4</option>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-Z <replaceable class="parameter">level</replaceable></option></term>
<term><option>--compress=<replaceable class="parameter">level</replaceable></option></term>
<listitem>
<para>
- Enables gzip compression of write-ahead logs, and specifies the
- compression level (0 through 9, 0 being no compression and 9 being best
- compression). The suffix <filename>.gz</filename> will
- automatically be added to all filenames.
+ Specifies the compression level (0 through 9, 0 being no compression and
+ 9 being best compression). If no <option>--compression-method</option>
+ is specified, it implies <literal>gzip</literal> compression method.
+ </para>
+
+ <para>
+ This option is not available when <option>--compression-method</option>
+ is specified as <literal>lz4</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 6e2f224cc4..91ba2240a2 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -341,6 +341,7 @@ perl_embed_ldflags = @perl_embed_ldflags@
AWK = @AWK@
LN_S = @LN_S@
+LZ4 = @LZ4@
MSGFMT = @MSGFMT@
MSGFMT_FLAGS = @MSGFMT_FLAGS@
MSGMERGE = @MSGMERGE@
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index 66e0070f1a..7950d2843e 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -18,8 +18,9 @@ subdir = src/bin/pg_basebackup
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-# make this available to TAP test scripts
+# make these available to TAP test scripts
export TAR
+export LZ4
override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport)
diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c
index 8bb0acf498..8cc73718a4 100644
--- a/src/bin/pg_basebackup/pg_basebackup.c
+++ b/src/bin/pg_basebackup/pg_basebackup.c
@@ -553,10 +553,13 @@ LogStreamerMain(logstreamer_param *param)
stream.replication_slot = replication_slot;
if (format == 'p')
- stream.walmethod = CreateWalDirectoryMethod(param->xlog, 0,
+ stream.walmethod = CreateWalDirectoryMethod(param->xlog,
+ COMPRESSION_NONE, 0,
stream.do_sync);
else
- stream.walmethod = CreateWalTarMethod(param->xlog, compresslevel,
+ stream.walmethod = CreateWalTarMethod(param->xlog,
+ COMPRESSION_NONE /* argument is ignored */,
+ compresslevel,
stream.do_sync);
if (!ReceiveXlogStream(param->bgconn, &stream))
diff --git a/src/bin/pg_basebackup/pg_receivewal.c b/src/bin/pg_basebackup/pg_receivewal.c
index c1334fad35..673b34df51 100644
--- a/src/bin/pg_basebackup/pg_receivewal.c
+++ b/src/bin/pg_basebackup/pg_receivewal.c
@@ -27,6 +27,10 @@
#include "receivelog.h"
#include "streamutil.h"
+#ifdef HAVE_LIBLZ4
+#include "lz4frame.h"
+#endif
+
/* Time to sleep between reconnection attempts */
#define RECONNECT_SLEEP_TIME 5
@@ -43,6 +47,7 @@ static bool do_drop_slot = false;
static bool do_sync = true;
static bool synchronous = false;
static char *replication_slot = NULL;
+static WalCompressionMethod compression_method = COMPRESSION_NONE;
static XLogRecPtr endpos = InvalidXLogRecPtr;
@@ -62,14 +67,22 @@ disconnect_atexit(void)
}
/* Routines to evaluate segment file format */
-#define IsCompressXLogFileName(fname) \
+#define IsZlibCompressXLogFileName(fname) \
(strlen(fname) == XLOG_FNAME_LEN + strlen(".gz") && \
strspn(fname, "0123456789ABCDEF") == XLOG_FNAME_LEN && \
strcmp((fname) + XLOG_FNAME_LEN, ".gz") == 0)
-#define IsPartialCompressXLogFileName(fname) \
+#define IsZlibPartialCompressXLogFileName(fname) \
(strlen(fname) == XLOG_FNAME_LEN + strlen(".gz.partial") && \
strspn(fname, "0123456789ABCDEF") == XLOG_FNAME_LEN && \
strcmp((fname) + XLOG_FNAME_LEN, ".gz.partial") == 0)
+#define IsLZ4CompressXLogFileName(fname) \
+ (strlen(fname) == XLOG_FNAME_LEN + strlen(".lz4") && \
+ strspn(fname, "0123456789ABCDEF") == XLOG_FNAME_LEN && \
+ strcmp((fname) + XLOG_FNAME_LEN, ".lz4") == 0)
+#define IsLZ4PartialCompressXLogFileName(fname) \
+ (strlen(fname) == XLOG_FNAME_LEN + strlen(".lz4.partial") && \
+ strspn(fname, "0123456789ABCDEF") == XLOG_FNAME_LEN && \
+ strcmp((fname) + XLOG_FNAME_LEN, ".lz4.partial") == 0)
static void
usage(void)
@@ -90,7 +103,10 @@ usage(void)
printf(_(" --synchronous flush write-ahead log immediately after writing\n"));
printf(_(" -v, --verbose output verbose messages\n"));
printf(_(" -V, --version output version information, then exit\n"));
- printf(_(" -Z, --compress=0-9 compress logs with given compression level\n"));
+ printf(_(" --compression-method=METHOD\n"
+ " use this method for compression\n"));
+ printf(_(" -Z, --compress=0-9 compress logs with given compression level\n"
+ " (available only with --compression-method=gzip)\n"));
printf(_(" -?, --help show this help, then exit\n"));
printf(_("\nConnection options:\n"));
printf(_(" -d, --dbname=CONNSTR connection string\n"));
@@ -212,7 +228,8 @@ FindStreamingStart(uint32 *tli)
uint32 tli;
XLogSegNo segno;
bool ispartial;
- bool iscompress;
+ bool iszlibcompress;
+ bool islz4compress;
/*
* Check if the filename looks like an xlog file, or a .partial file.
@@ -220,22 +237,38 @@ FindStreamingStart(uint32 *tli)
if (IsXLogFileName(dirent->d_name))
{
ispartial = false;
- iscompress = false;
+ iszlibcompress = false;
+ islz4compress = false;
}
else if (IsPartialXLogFileName(dirent->d_name))
{
ispartial = true;
- iscompress = false;
+ iszlibcompress = false;
+ islz4compress = false;
}
- else if (IsCompressXLogFileName(dirent->d_name))
+ else if (IsZlibCompressXLogFileName(dirent->d_name))
{
ispartial = false;
- iscompress = true;
+ iszlibcompress = true;
+ islz4compress = false;
}
- else if (IsPartialCompressXLogFileName(dirent->d_name))
+ else if (IsZlibPartialCompressXLogFileName(dirent->d_name))
{
ispartial = true;
- iscompress = true;
+ iszlibcompress = true;
+ islz4compress = false;
+ }
+ else if (IsLZ4CompressXLogFileName(dirent->d_name))
+ {
+ ispartial = false;
+ islz4compress = true;
+ iszlibcompress = false;
+ }
+ else if (IsLZ4PartialCompressXLogFileName(dirent->d_name))
+ {
+ ispartial = true;
+ islz4compress = true;
+ iszlibcompress = false;
}
else
continue;
@@ -248,14 +281,15 @@ FindStreamingStart(uint32 *tli)
/*
* Check that the segment has the right size, if it's supposed to be
* completed. For non-compressed segments just check the on-disk size
- * and see if it matches a completed segment. For compressed segments,
- * look at the last 4 bytes of the compressed file, which is where the
- * uncompressed size is located for gz files with a size lower than
- * 4GB, and then compare it to the size of a completed segment. The 4
- * last bytes correspond to the ISIZE member according to
+ * and see if it matches a completed segment. For zlib compressed
+ * segments, look at the last 4 bytes of the compressed file, which is
+ * where the uncompressed size is located for gz files with a size lower
+ * than 4GB, and then compare it to the size of a completed segment.
+ * The 4 last bytes correspond to the ISIZE member according to
* http://www.zlib.org/rfc-gzip.html.
+ * For lz4 compressed segments
*/
- if (!ispartial && !iscompress)
+ if (!ispartial && !iszlibcompress && !islz4compress)
{
struct stat statbuf;
char fullpath[MAXPGPATH * 2];
@@ -274,7 +308,7 @@ FindStreamingStart(uint32 *tli)
continue;
}
}
- else if (!ispartial && iscompress)
+ else if (!ispartial && iszlibcompress)
{
int fd;
char buf[4];
@@ -320,6 +354,70 @@ FindStreamingStart(uint32 *tli)
continue;
}
}
+ else if (!ispartial && islz4compress)
+ {
+#ifdef HAVE_LIBLZ4
+ int fd;
+ int r;
+ size_t consumed_len = LZ4F_HEADER_SIZE_MAX;
+ char buf[LZ4F_HEADER_SIZE_MAX];
+ char fullpath[MAXPGPATH * 2];
+ LZ4F_frameInfo_t frame_info = { 0 };
+ LZ4F_decompressionContext_t ctx = NULL;
+
+ snprintf(fullpath, sizeof(fullpath), "%s/%s", basedir, dirent->d_name);
+
+ fd = open(fullpath, O_RDONLY | PG_BINARY, 0);
+ if (fd < 0)
+ {
+ pg_log_error("could not open compressed file \"%s\": %m",
+ fullpath);
+ exit(1);
+ }
+
+ r = read(fd, buf, sizeof(buf));
+ if (r != sizeof(buf))
+ {
+ if (r < 0)
+ pg_log_error("could not read compressed file \"%s\": %m",
+ fullpath);
+ else
+ pg_log_error("could not read compressed file \"%s\": read %d of %lu",
+ fullpath, r, sizeof(buf));
+ exit(1);
+ }
+
+ if (LZ4F_isError(LZ4F_createDecompressionContext(&ctx, LZ4F_VERSION)))
+ {
+ pg_log_error("lz4 internal error");
+ exit(1);
+ }
+
+ LZ4F_getFrameInfo(ctx, &frame_info, (void *)buf, &consumed_len);
+ if (consumed_len <= LZ4F_HEADER_SIZE_MIN ||
+ consumed_len >= LZ4F_HEADER_SIZE_MAX)
+ {
+ pg_log_warning("compressed segment file \"%s\" has incorrect header size %lu, skipping",
+ dirent->d_name, consumed_len);
+ LZ4F_freeDecompressionContext(ctx);
+ continue;
+ }
+
+ if (frame_info.contentSize != WalSegSz)
+ {
+ pg_log_warning("compressed segment file \"%s\" has incorrect uncompressed size %lld, skipping",
+ dirent->d_name, frame_info.contentSize);
+ LZ4F_freeDecompressionContext(ctx);
+ continue;
+ }
+
+ LZ4F_freeDecompressionContext(ctx);
+#else
+ pg_log_error("cannot verify lz4 compressed segment file \"%s\", "
+ "this program was not build with lz4 support");
+ exit(1);
+#endif
+ }
/* Looks like a valid segment. Remember that we saw it. */
if ((segno > high_segno) ||
@@ -429,7 +527,9 @@ StreamLog(void)
stream.synchronous = synchronous;
stream.do_sync = do_sync;
stream.mark_done = false;
- stream.walmethod = CreateWalDirectoryMethod(basedir, compresslevel,
+ stream.walmethod = CreateWalDirectoryMethod(basedir,
+ compression_method,
+ compresslevel,
stream.do_sync);
stream.partial_suffix = ".partial";
stream.replication_slot = replication_slot;
@@ -482,6 +582,7 @@ main(int argc, char **argv)
{"status-interval", required_argument, NULL, 's'},
{"slot", required_argument, NULL, 'S'},
{"verbose", no_argument, NULL, 'v'},
+ {"compression-method", required_argument, NULL, 'I'},
{"compress", required_argument, NULL, 'Z'},
/* action */
{"create-slot", no_argument, NULL, 1},
@@ -573,6 +674,21 @@ main(int argc, char **argv)
case 'v':
verbose++;
break;
+ case 'I':
+ if (strcmp(optarg, "gzip") == 0)
+ {
+ compression_method = COMPRESSION_ZLIB;
+ }
+ else if (strcmp(optarg, "lz4") == 0)
+ {
+ compression_method = COMPRESSION_LZ4;
+ }
+ else
+ {
+ pg_log_error("invalid compression-method \"%s\"", optarg);
+ exit(1);
+ }
+ break;
case 'Z':
compresslevel = atoi(optarg);
if (compresslevel < 0 || compresslevel > 9)
@@ -657,14 +773,56 @@ main(int argc, char **argv)
exit(1);
}
+ if (compression_method != COMPRESSION_NONE)
+ {
+#ifndef HAVE_LIBZ
+ if (compression_method == COMPRESSION_ZLIB)
+ {
+ pg_log_error("this build does not support compression via gzip");
+ exit(1);
+ }
+#endif
+#ifndef HAVE_LIBLZ4
+ if (compression_method == COMPRESSION_LZ4)
+ {
+ pg_log_error("this build does not support compression via lz4");
+ exit(1);
+ }
+#endif
+ }
+
#ifndef HAVE_LIBZ
if (compresslevel != 0)
{
- pg_log_error("this build does not support compression");
+ pg_log_error("this build does not support compression via gzip");
exit(1);
}
#endif
+ if (compresslevel != 0)
+ {
+ if (compression_method == COMPRESSION_NONE)
+ {
+ compression_method = COMPRESSION_ZLIB;
+ }
+ if (compression_method != COMPRESSION_ZLIB)
+ {
+ pg_log_error("cannot use --compress when "
+ "--compression-method is not gzip");
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+ progname);
+ exit(1);
+ }
+ }
+ else if (compression_method == COMPRESSION_ZLIB)
+ {
+ pg_log_error("cannot use --compression-method gzip when "
+ "--compression is 0");
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+ progname);
+ exit(1);
+ }
+
/*
* Check existence of destination folder.
*/
diff --git a/src/bin/pg_basebackup/t/020_pg_receivewal.pl b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
index a547c97ef1..d688cb1899 100644
--- a/src/bin/pg_basebackup/t/020_pg_receivewal.pl
+++ b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
@@ -5,7 +5,7 @@ use strict;
use warnings;
use TestLib;
use PostgresNode;
-use Test::More tests => 19;
+use Test::More tests => 23;
program_help_ok('pg_receivewal');
program_version_ok('pg_receivewal');
@@ -33,6 +33,13 @@ $primary->command_fails(
$primary->command_fails(
[ 'pg_receivewal', '-D', $stream_dir, '--synchronous', '--no-sync' ],
'failure if --synchronous specified with --no-sync');
+$primary->command_fails(
+ [
+ 'pg_receivewal', '-D', $stream_dir, '--compression-method', 'lz4',
+ '--compress', '1'
+ ],
+ 'failure if --compression-method=lz4 specified with --compress');
+
# Slot creation and drop
my $slot_name = 'test';
@@ -66,6 +73,43 @@ $primary->command_ok(
],
'streaming some WAL with --synchronous');
+# Check lz4 compression if available
+SKIP:
+{
+ my $lz4 = $ENV{LZ4};
+
+ skip "postgres was not build with LZ4 support", 3
+ if (!check_pg_config("#define HAVE_LIBLZ4 1"));
+
+ # Generate some WAL.
+ $primary->psql('postgres', 'SELECT pg_switch_wal();');
+ $nextlsn =
+ $primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
+ chomp($nextlsn);
+ $primary->psql('postgres',
+ 'INSERT INTO test_table VALUES (generate_series(100,200));');
+ $primary->psql('postgres', 'SELECT pg_switch_wal();');
+
+ # Stream up to the given position
+ $primary->command_ok(
+ [
+ 'pg_receivewal', '-D', $stream_dir, '--verbose',
+ '--endpos', $nextlsn, '--compression-method=lz4'
+ ],
+ 'streaming some WAL with --compression-method=lz4');
+
+ # Verify that the stored file is compressed
+ my @lz4_wals = glob "$stream_dir/*.lz4";
+ is(scalar(@lz4_wals), 1, 'one lz4 compressed WAL was created');
+
+ # Verify that the stored file is readable if program lz4 is available
+ skip "program lz4 is not found in your system", 1
+ if (!defined $lz4 || $lz4 eq '');
+
+ my $can_decode = system_log($lz4, '-t', $lz4_wals[0]);
+ is($can_decode, 0, "program lz4 can decode compressed WAL");
+}
+
# Permissions on WAL files should be default
SKIP:
{
diff --git a/src/bin/pg_basebackup/walmethods.c b/src/bin/pg_basebackup/walmethods.c
index a15bbb20e7..640abc83aa 100644
--- a/src/bin/pg_basebackup/walmethods.c
+++ b/src/bin/pg_basebackup/walmethods.c
@@ -17,6 +17,10 @@
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
+
+#ifdef HAVE_LIBLZ4
+#include <lz4frame.h>
+#endif
#ifdef HAVE_LIBZ
#include <zlib.h>
#endif
@@ -30,6 +34,9 @@
/* Size of zlib buffer for .tar.gz */
#define ZLIB_OUT_SIZE 4096
+/* Size of lz4 input chunk for .lz4 */
+#define LZ4_IN_SIZE 4096
+
/*-------------------------------------------------------------------------
* WalDirectoryMethod - write wal to a directory looking like pg_wal
*-------------------------------------------------------------------------
@@ -40,9 +47,10 @@
*/
typedef struct DirectoryMethodData
{
- char *basedir;
- int compression;
- bool sync;
+ char *basedir;
+ WalCompressionMethod compression_method;
+ int compression;
+ bool sync;
} DirectoryMethodData;
static DirectoryMethodData *dir_data = NULL;
@@ -59,6 +67,11 @@ typedef struct DirectoryMethodFile
#ifdef HAVE_LIBZ
gzFile gzfp;
#endif
+#ifdef HAVE_LIBLZ4
+ LZ4F_compressionContext_t ctx;
+ size_t lz4bufsize;
+ void *lz4buf;
+#endif
} DirectoryMethodFile;
static const char *
@@ -77,10 +90,16 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
#ifdef HAVE_LIBZ
gzFile gzfp = NULL;
#endif
+#ifdef HAVE_LIBLZ4
+ LZ4F_compressionContext_t ctx = NULL;
+ size_t lz4bufsize = 0;
+ void *lz4buf = NULL;
+#endif
snprintf(tmppath, sizeof(tmppath), "%s/%s%s%s",
dir_data->basedir, pathname,
- dir_data->compression > 0 ? ".gz" : "",
+ dir_data->compression_method == COMPRESSION_ZLIB ? ".gz" :
+ dir_data->compression_method == COMPRESSION_LZ4 ? ".lz4": "",
temp_suffix ? temp_suffix : "");
/*
@@ -94,7 +113,7 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
return NULL;
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
{
gzfp = gzdopen(fd, "wb");
if (gzfp == NULL)
@@ -111,9 +130,61 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
}
}
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ LZ4F_preferences_t lz4preferences = { 0 };
+ size_t ctx_out;
+ size_t header_size;
+
+ /*
+ * Set all the preferences to default but do note contentSize. It will
+ * be needed in FindStreamingStart.
+ */
+ memset(&lz4preferences, 0, sizeof(LZ4F_frameInfo_t));
+ lz4preferences.frameInfo.contentSize = (unsigned long long)WalSegSz;
+ ctx_out = LZ4F_createCompressionContext(&ctx, LZ4F_VERSION);
+ lz4bufsize = LZ4F_compressBound(LZ4_IN_SIZE, &lz4preferences);
+ if (LZ4F_isError(ctx_out))
+ {
+ close(fd);
+ return NULL;
+ }
+
+ lz4buf = pg_malloc0(lz4bufsize);
+
+ /*
+ * XXX: this is crap... lz4preferences now does show the uncompressed
+ * size via lz4 --list <filename> but the compression goes down the
+ * window... also it is not very helpfull to have it at the startm, does
+ * it?
+ */
+ /* add the header */
+ header_size = LZ4F_compressBegin(ctx, lz4buf, lz4bufsize, &lz4preferences);
+ if (LZ4F_isError(header_size))
+ {
+ close(fd);
+ return NULL;
+ }
+
+ errno = 0;
+ if (write(fd, lz4buf, header_size) != header_size)
+ {
+ int save_errno = errno;
+
+ close(fd);
+
+ /*
+ * If write didn't set errno, assume problem is no disk space.
+ */
+ errno = save_errno ? save_errno : ENOSPC;
+ return NULL;
+ }
+ }
+#endif
/* Do pre-padding on non-compressed files */
- if (pad_to_size && dir_data->compression == 0)
+ if (pad_to_size && dir_data->compression_method == COMPRESSION_NONE)
{
PGAlignedXLogBlock zerobuf;
int bytes;
@@ -158,7 +229,7 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
fsync_parent_path(tmppath) != 0)
{
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
gzclose(gzfp);
else
#endif
@@ -169,9 +240,18 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
f = pg_malloc0(sizeof(DirectoryMethodFile));
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
f->gzfp = gzfp;
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ f->ctx = ctx;
+ f->lz4buf = lz4buf;
+ f->lz4bufsize = lz4bufsize;
+ }
+#endif
+
f->fd = fd;
f->currpos = 0;
f->pathname = pg_strdup(pathname);
@@ -191,9 +271,46 @@ dir_write(Walfile f, const void *buf, size_t count)
Assert(f != NULL);
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
r = (ssize_t) gzwrite(df->gzfp, buf, count);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ size_t chunk;
+ size_t remaining;
+ const void *inbuf = buf;
+
+ remaining = count;
+ while (remaining > 0)
+ {
+ size_t compressed;
+
+ if (remaining > LZ4_IN_SIZE)
+ chunk = LZ4_IN_SIZE;
+ else
+ chunk = remaining;
+
+ remaining -= chunk;
+ compressed = LZ4F_compressUpdate(df->ctx,
+ df->lz4buf, df->lz4bufsize,
+ inbuf, chunk,
+ NULL);
+
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+
+ inbuf = ((char *)inbuf) + chunk;
+ }
+
+ /* XXX: This is what our caller expects, but it is not nice at all */
+ r = (ssize_t)count;
+ }
+ else
#endif
r = write(df->fd, buf, count);
if (r > 0)
@@ -221,9 +338,30 @@ dir_close(Walfile f, WalCloseMethod method)
Assert(f != NULL);
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
r = gzclose(df->gzfp);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ /* Flush any internal buffers */
+ size_t compressed = LZ4F_compressEnd(df->ctx,
+ df->lz4buf, df->lz4bufsize,
+ NULL);
+ if (LZ4F_isError(compressed))
+ {
+ return -1;
+ }
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ {
+ return -1;
+ }
+
+ r = close(df->fd);
+ }
+ else
#endif
r = close(df->fd);
@@ -238,11 +376,13 @@ dir_close(Walfile f, WalCloseMethod method)
*/
snprintf(tmppath, sizeof(tmppath), "%s/%s%s%s",
dir_data->basedir, df->pathname,
- dir_data->compression > 0 ? ".gz" : "",
+ dir_data->compression_method == COMPRESSION_ZLIB ? ".gz" :
+ dir_data->compression_method == COMPRESSION_LZ4 ? ".lz4": "",
df->temp_suffix);
snprintf(tmppath2, sizeof(tmppath2), "%s/%s%s",
dir_data->basedir, df->pathname,
- dir_data->compression > 0 ? ".gz" : "");
+ dir_data->compression_method == COMPRESSION_ZLIB ? ".gz" :
+ dir_data->compression_method == COMPRESSION_LZ4 ? ".lz4": "");
r = durable_rename(tmppath, tmppath2);
}
else if (method == CLOSE_UNLINK)
@@ -250,7 +390,8 @@ dir_close(Walfile f, WalCloseMethod method)
/* Unlink the file once it's closed */
snprintf(tmppath, sizeof(tmppath), "%s/%s%s%s",
dir_data->basedir, df->pathname,
- dir_data->compression > 0 ? ".gz" : "",
+ dir_data->compression_method == COMPRESSION_ZLIB ? ".gz" :
+ dir_data->compression_method == COMPRESSION_LZ4 ? ".lz4": "",
df->temp_suffix ? df->temp_suffix : "");
r = unlink(tmppath);
}
@@ -270,6 +411,12 @@ dir_close(Walfile f, WalCloseMethod method)
}
}
+#ifdef HAVE_LIBLZ4
+ pg_free(df->lz4buf);
+ /* supports free on NULL */
+ LZ4F_freeCompressionContext(df->ctx);
+#endif
+
pg_free(df->pathname);
pg_free(df->fullpath);
if (df->temp_suffix)
@@ -346,7 +493,9 @@ dir_finish(void)
WalWriteMethod *
-CreateWalDirectoryMethod(const char *basedir, int compression, bool sync)
+CreateWalDirectoryMethod(const char *basedir,
+ WalCompressionMethod compression_method,
+ int compression, bool sync)
{
WalWriteMethod *method;
@@ -362,6 +511,7 @@ CreateWalDirectoryMethod(const char *basedir, int compression, bool sync)
method->getlasterror = dir_getlasterror;
dir_data = pg_malloc0(sizeof(DirectoryMethodData));
+ dir_data->compression_method = compression_method;
dir_data->compression = compression;
dir_data->basedir = pg_strdup(basedir);
dir_data->sync = sync;
@@ -983,8 +1133,16 @@ tar_finish(void)
return true;
}
+/*
+ * The argument compression_method is currently ignored. It is in place for
+ * symmetry with CreateWalDirectoryMethod which uses it for distinguishing
+ * between the different compression methods. CreateWalTarMethod and its family
+ * of functions handle only zlib compression.
+ */
WalWriteMethod *
-CreateWalTarMethod(const char *tarbase, int compression, bool sync)
+CreateWalTarMethod(const char *tarbase,
+ WalCompressionMethod compression_method,
+ int compression, bool sync)
{
WalWriteMethod *method;
const char *suffix = (compression != 0) ? ".tar.gz" : ".tar";
diff --git a/src/bin/pg_basebackup/walmethods.h b/src/bin/pg_basebackup/walmethods.h
index fc4bb52cb7..b5998a08bc 100644
--- a/src/bin/pg_basebackup/walmethods.h
+++ b/src/bin/pg_basebackup/walmethods.h
@@ -19,6 +19,13 @@ typedef enum
CLOSE_NO_RENAME
} WalCloseMethod;
+typedef enum
+{
+ COMPRESSION_LZ4,
+ COMPRESSION_ZLIB,
+ COMPRESSION_NONE
+} WalCompressionMethod;
+
/*
* A WalWriteMethod structure represents the different methods used
* to write the streaming WAL as it's received.
@@ -86,8 +93,11 @@ struct WalWriteMethod
* not all those required for pg_receivewal)
*/
WalWriteMethod *CreateWalDirectoryMethod(const char *basedir,
+ WalCompressionMethod compression_method,
int compression, bool sync);
-WalWriteMethod *CreateWalTarMethod(const char *tarbase, int compression, bool sync);
+WalWriteMethod *CreateWalTarMethod(const char *tarbase,
+ WalCompressionMethod compression_method,
+ int compression, bool sync);
/* Cleanup routines for previously-created methods */
void FreeWalDirectoryMethod(void);
--
2.25.1
Re: Teach pg_receivewal to use lz4 compression
On Thu, Jul 08, 2021 at 02:18:40PM +0000, gkokolatos@pm.me wrote:
please find v2 of the patch which tries to address the commends
received so far.
Thanks!
Michael Paquier wrote:
+ system_or_bail('lz4', '-t', $lz4_wals[0]); I think that you should just drop this part of the test. The only part of LZ4 that we require to be present when Postgres is built with --with-lz4 is its library liblz4. Commands associated to it may not be around, causing this test to fail. The test checking that one .lz4 file has been created is good to have. It may be worth adding a test with a .lz4.partial segment generated and --endpos pointing to a LSN that does not finish the segment that gets switched.I humbly disagree with the need for the test. It is rather easily possible
to generate a file that can not be decoded, thus becoming useless. Having the
test will provide some guarantee for the fact. In the current patch, there
is code to find out if the program lz4 is available in the system. If it is
not available, then that specific test is skipped. The rest remains as it
were. Also `system_or_bail` is not used anymore in favour of the `system_log`
so that the test counted and the execution of tests continues upon failure.
Check. I can see what you are using in the new patch. We could live
with that.
It seems to me that you are missing some logic in
FindStreamingStart() to handle LZ4-compressed segments, in relation
with IsCompressXLogFileName() and IsPartialCompressXLogFileName().Very correct. The logic is now added. Given the lz4 api, one has to fill
in the uncompressed size during creation time. Then one can read the
headers and verify the expectations.
Yeah, I read that as well with lz4 --list and the kind. That's weird
compared to how ZLIB gives an easy access to it. We may want to do an
effort and tell about more lz4 --content-size/--list, telling that we
add the size in the segment compressed because we have to and LZ4 does
not do it by default?
Should we have more tests for ZLIB, while on it? That seems like a
good addition as long as we can skip the tests conditionally when
that's not supported.Agreed. Please allow me to provide a distinct patch for this code.
Thanks. Looking forward to seeing it. That may be better on a
separate thread, even if I digressed in this thread :)
I think we can somehow use "acceleration" parameter of lz4 compression
to map on compression level, It is not direct mapping but
can't we create some internal mapping instead of completely ignoring
this option for lz4, or we can provide another option for lz4?We can, though I am not in favour of doing so. There is seemingly
little benefit for added complexity.
Agreed.
What I think is important for the user when it comes to this
option is the consistency of its naming across all the tools
supporting it. pg_dump and pg_basebackup, where we could plug LZ4,
already use most of the short options you could use for pg_receivewal,
having only a long one gives a bit more flexibility.Done.
* http://www.zlib.org/rfc-gzip.html.
+ * For lz4 compressed segments
*/
This comment is incomplete.
+#define IsLZ4CompressXLogFileName(fname) \
+ (strlen(fname) == XLOG_FNAME_LEN + strlen(".lz4") && \
+ strspn(fname, "0123456789ABCDEF") == XLOG_FNAME_LEN && \
+ strcmp((fname) + XLOG_FNAME_LEN, ".lz4") == 0)
+#define IsLZ4PartialCompressXLogFileName(fname) \
+ (strlen(fname) == XLOG_FNAME_LEN + strlen(".lz4.partial") && \
+ strspn(fname, "0123456789ABCDEF") == XLOG_FNAME_LEN && \
+ strcmp((fname) + XLOG_FNAME_LEN, ".lz4.partial") == 0)
This is getting complicated. Would it be better to change this stuff
and switch to a routine that checks if a segment has a valid name, is
partial, and the type of compression that applied to it? It seems to
me that we should group iszlibcompress and islz4compress together with
the options available through compression_method.+ if (compresslevel != 0)
+ {
+ if (compression_method == COMPRESSION_NONE)
+ {
+ compression_method = COMPRESSION_ZLIB;
+ }
+ if (compression_method != COMPRESSION_ZLIB)
+ {
+ pg_log_error("cannot use --compress when "
+ "--compression-method is not gzip");
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+ progname);
+ exit(1);
+ }
+ }
For compatibility where --compress enforces the use of zlib that would
work, but this needs a comment explaining the goal of this block. I
am wondering if it would be better to break the will and just complain
when specifying --compress without --compression-method though. That
would cause compatibility issues, but this would make folks aware of
the presence of LZ4, which does not sound bad to me either as ZLIB is
slower than LZ4 on all fronts.+ else if (compression_method == COMPRESSION_ZLIB)
+ {
+ pg_log_error("cannot use --compression-method gzip when "
+ "--compression is 0");
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+ progname);
+ exit(1);
+ }
Hmm. It would be more natural to enforce a default compression level
in this case? The user is asking for a compression with zlib here.+ my $lz4 = $ENV{LZ4};
[...]
+ # Verify that the stored file is readable if program lz4 is available
+ skip "program lz4 is not found in your system", 1
+ if (!defined $lz4 || $lz4 eq '');
Okay, this is acceptable. Didn't know the existing trick with TAR
either.+ /*
+ * XXX: this is crap... lz4preferences now does show the uncompressed
+ * size via lz4 --list <filename> but the compression goes down the
+ * window... also it is not very helpfull to have it at the startm, does
+ * it?
+ */
What do you mean here by "the compression goes out the window"?
--
MichaelRe: Teach pg_receivewal to use lz4 compression
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, July 9th, 2021 at 04:49, Michael Paquier <michael@paquier.xyz> wrote:
On Thu, Jul 08, 2021 at 02:18:40PM +0000, gkokolatos@pm.me wrote:
please find v2 of the patch which tries to address the commends
received so far.
Thanks!
Michael Paquier wrote:
- system_or_bail('lz4', '-t', $lz4_wals[0]);
I think that you should just drop this part of the test. The only
part of LZ4 that we require to be present when Postgres is built with
--with-lz4 is its library liblz4. Commands associated to it may not
be around, causing this test to fail. The test checking that one .lz4
file has been created is good to have. It may be worth adding a test
with a .lz4.partial segment generated and --endpos pointing to a LSN
that does not finish the segment that gets switched.
I humbly disagree with the need for the test. It is rather easily possible
to generate a file that can not be decoded, thus becoming useless. Having the
test will provide some guarantee for the fact. In the current patch, there
is code to find out if the program lz4 is available in the system. If it is
not available, then that specific test is skipped. The rest remains as it
were. Also `system_or_bail` is not used anymore in favour of the `system_log`
so that the test counted and the execution of tests continues upon failure.
Check. I can see what you are using in the new patch. We could live
with that.
Great. Thank you.
It seems to me that you are missing some logic in
FindStreamingStart() to handle LZ4-compressed segments, in relation
with IsCompressXLogFileName() and IsPartialCompressXLogFileName().
Very correct. The logic is now added. Given the lz4 api, one has to fill
in the uncompressed size during creation time. Then one can read the
headers and verify the expectations.
Yeah, I read that as well with lz4 --list and the kind. That's weird
compared to how ZLIB gives an easy access to it. We may want to do an
effort and tell about more lz4 --content-size/--list, telling that we
add the size in the segment compressed because we have to and LZ4 does
not do it by default?
I am afraid I do not follow. In the patch we do add the uncompressed size
and then, the uncompressed size is checked against a known value WalSegSz.
What the compressed size will be checked against?
Should we have more tests for ZLIB, while on it? That seems like a
good addition as long as we can skip the tests conditionally when
that's not supported.
Agreed. Please allow me to provide a distinct patch for this code.
Thanks. Looking forward to seeing it. That may be better on a
separate thread, even if I digressed in this thread :)
Thank you for the comments. I will sent in a separate thread.
I think we can somehow use "acceleration" parameter of lz4 compression
to map on compression level, It is not direct mapping but
can't we create some internal mapping instead of completely ignoring
this option for lz4, or we can provide another option for lz4?
We can, though I am not in favour of doing so. There is seemingly
little benefit for added complexity.
Agreed.
What I think is important for the user when it comes to this
option is the consistency of its naming across all the tools
supporting it. pg_dump and pg_basebackup, where we could plug LZ4,
already use most of the short options you could use for pg_receivewal,
having only a long one gives a bit more flexibility.
Done.
* http://www.zlib.org/rfc-gzip.html.
- - For lz4 compressed segments
*/
This comment is incomplete.
It is. I will fix.
+#define IsLZ4CompressXLogFileName(fname) \ - (strlen(fname) == XLOG_FNAME_LEN + strlen(".lz4") && \ - strspn(fname, "0123456789ABCDEF") == XLOG_FNAME_LEN && \ - strcmp((fname) + XLOG_FNAME_LEN, ".lz4") == 0)+#define IsLZ4PartialCompressXLogFileName(fname) \ - (strlen(fname) == XLOG_FNAME_LEN + strlen(".lz4.partial") && \ - strspn(fname, "0123456789ABCDEF") == XLOG_FNAME_LEN && \ - strcmp((fname) + XLOG_FNAME_LEN, ".lz4.partial") == 0)This is getting complicated. Would it be better to change this stuff
and switch to a routine that checks if a segment has a valid name, is
partial, and the type of compression that applied to it? It seems to
me that we should group iszlibcompress and islz4compress together with
the options available through compression_method.
I agree with you. I will refactor.
- if (compresslevel != 0)
- {
- if (compression_method == COMPRESSION_NONE)- {
- compression_method = COMPRESSION_ZLIB;
- }
- if (compression_method != COMPRESSION_ZLIB)
- {
- pg_log_error("cannot use --compress when "
- "--compression-method is not gzip");
- fprintf(stderr, _("Try \\"%s --help\\" for more information.\\n"),
- progname);
- exit(1);
- }
- }
For compatibility where --compress enforces the use of zlib that would
work, but this needs a comment explaining the goal of this block. I
am wondering if it would be better to break the will and just complain
when specifying --compress without --compression-method though. That
would cause compatibility issues, but this would make folks aware of
the presence of LZ4, which does not sound bad to me either as ZLIB is
slower than LZ4 on all fronts.
I would vote to break the compatibility if that is an option. I chose the
less invasive approach thinking that breaking the compatibility would not
be an option.
Unless others object, I will include --compress as a complimentary option
to --compression-method in updated version of the patch.
- else if (compression_method == COMPRESSION_ZLIB)
- {
- pg_log_error("cannot use --compression-method gzip when "- "--compression is 0");
- fprintf(stderr, _("Try \\"%s --help\\" for more information.\\n"),
- progname);
- exit(1);
- }
Hmm. It would be more natural to enforce a default compression level
in this case? The user is asking for a compression with zlib here.
You are correct, in the current patch passing --compression-method=gzip alone
is equivalent to passing --compression=0 in the current master version. This
behaviour may be confusing for the user. What should the default compression
be then? I am inclined to say '5' as a compromise between speed and compression
ration.
- my $lz4 = $ENV{LZ4};
[...]
- Verify that the stored file is readable if program lz4 is available
===================================================================- skip "program lz4 is not found in your system", 1
- if (!defined $lz4 || $lz4 eq '');Okay, this is acceptable. Didn't know the existing trick with TAR
either.
Thank you.
- /*
- * XXX: this is crap... lz4preferences now does show the uncompressed
- * size via lz4 --list <filename> but the compression goes down the
- * window... also it is not very helpfull to have it at the startm, does
- * it?
- */
What do you mean here by "the compression goes out the window"?
Please consider me adequately embarrassed. This was a personal comment while I was
working on the code. It is not correct and it should have never seen the public
light.
Cheers,
//Georgios
Show quoted text
---------------------------------------------------------------
Michael
Re: Teach pg_receivewal to use lz4 compression
On Thu, Jul 8, 2021 at 7:48 PM <gkokolatos@pm.me> wrote:
Dilip Kumar wrote:
Wouldn't it be better to call it compression method instead of
compression program?Agreed. This is inline with the suggestions of other reviewers.
Find the change in the attached patch.
Thanks, I will have a look.
I think we can somehow use "acceleration" parameter of lz4 compression
to map on compression level, It is not direct mapping but
can't we create some internal mapping instead of completely ignoring
this option for lz4, or we can provide another option for lz4?We can, though I am not in favour of doing so. There is seemingly
little benefit for added complexity.
I am really not sure what complexity you are talking about, do you
mean since with pglz we were already providing the compression level
so let it be as it is but with the new compression method you don't
see much benefit of providing compression level or speed?
Should we also support LZ4 compression using dictionary?
I would we should not do that. If my understanding is correct,
decompression would require the dictionary to be passed along.
The algorithm seems to be very competitive to the current compression
as is.
I agree, we might not go for a dictionary because we would need to
dictionary to decompress as well. So that will add an extra
complexity for user.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Re: Teach pg_receivewal to use lz4 compression
On Mon, Jul 12, 2021 at 11:10:24AM +0530, Dilip Kumar wrote:
On Thu, Jul 8, 2021 at 7:48 PM <gkokolatos@pm.me> wrote:
We can, though I am not in favour of doing so. There is seemingly
little benefit for added complexity.I am really not sure what complexity you are talking about, do you
mean since with pglz we were already providing the compression level
so let it be as it is but with the new compression method you don't
see much benefit of providing compression level or speed?
You mean s/pglz/zlib/ here perhaps? I am not sure what Georgios has
in mind, but my opinion stands on the latter: there is little benefit
in making lz4 faster than the default and reduce compression, as the
default is already a rather low CPU user.
--
Michael
Re: Teach pg_receivewal to use lz4 compression
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Monday, July 12th, 2021 at 07:56, Michael Paquier <michael@paquier.xyz> wrote:
On Mon, Jul 12, 2021 at 11:10:24AM +0530, Dilip Kumar wrote:
On Thu, Jul 8, 2021 at 7:48 PM gkokolatos@pm.me wrote:
We can, though I am not in favour of doing so. There is seemingly
little benefit for added complexity.
I am really not sure what complexity you are talking about, do you
mean since with pglz we were already providing the compression level
so let it be as it is but with the new compression method you don't
see much benefit of providing compression level or speed?
You mean s/pglz/zlib/ here perhaps? I am not sure what Georgios has
in mind, but my opinion stands on the latter: there is little benefit
in making lz4 faster than the default and reduce compression, as the
default is already a rather low CPU user.
Thank you all for your comments. I am sitting on the same side as Micheal
on this one. The complexity is not huge, yet there will need to be code to
pass this option to the lz4 api and various test cases to verify for
correctness and integrity. The burden of maintenance of this code vs the
benefit of the option, tilt the scale towards not including the option.
Of course, I will happily provide whatever the community finds beneficial.
Cheers,
//Georgios
Show quoted text
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Michael
Re: Teach pg_receivewal to use lz4 compression
On Mon, Jul 12, 2021 at 11:33 AM <gkokolatos@pm.me> wrote:
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Monday, July 12th, 2021 at 07:56, Michael Paquier <michael@paquier.xyz> wrote:
On Mon, Jul 12, 2021 at 11:10:24AM +0530, Dilip Kumar wrote:
On Thu, Jul 8, 2021 at 7:48 PM gkokolatos@pm.me wrote:
We can, though I am not in favour of doing so. There is seemingly
little benefit for added complexity.
I am really not sure what complexity you are talking about, do you
mean since with pglz we were already providing the compression level
so let it be as it is but with the new compression method you don't
see much benefit of providing compression level or speed?
You mean s/pglz/zlib/ here perhaps? I am not sure what Georgios has
in mind, but my opinion stands on the latter: there is little benefit
in making lz4 faster than the default and reduce compression, as the
default is already a rather low CPU user.
Thank you all for your comments. I am sitting on the same side as Micheal
on this one. The complexity is not huge, yet there will need to be code to
pass this option to the lz4 api and various test cases to verify for
correctness and integrity. The burden of maintenance of this code vs the
benefit of the option, tilt the scale towards not including the option.
+1 for skipping that one, at least for now, and sticking to
default-only for lz4.
--
Magnus Hagander
Me: https://www.hagander.net/
Work: https://www.redpill-linpro.com/
Re: Teach pg_receivewal to use lz4 compression
On Mon, Jul 12, 2021 at 3:15 PM Magnus Hagander <magnus@hagander.net> wrote:
On Mon, Jul 12, 2021 at 11:33 AM <gkokolatos@pm.me> wrote:
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Monday, July 12th, 2021 at 07:56, Michael Paquier <michael@paquier.xyz> wrote:
On Mon, Jul 12, 2021 at 11:10:24AM +0530, Dilip Kumar wrote:
On Thu, Jul 8, 2021 at 7:48 PM gkokolatos@pm.me wrote:
We can, though I am not in favour of doing so. There is seemingly
little benefit for added complexity.
I am really not sure what complexity you are talking about, do you
mean since with pglz we were already providing the compression level
so let it be as it is but with the new compression method you don't
see much benefit of providing compression level or speed?
You mean s/pglz/zlib/ here perhaps? I am not sure what Georgios has
in mind, but my opinion stands on the latter: there is little benefit
in making lz4 faster than the default and reduce compression, as the
default is already a rather low CPU user.
Thank you all for your comments. I am sitting on the same side as Micheal
on this one. The complexity is not huge, yet there will need to be code to
pass this option to the lz4 api and various test cases to verify for
correctness and integrity. The burden of maintenance of this code vs the
benefit of the option, tilt the scale towards not including the option.+1 for skipping that one, at least for now, and sticking to
default-only for lz4.
Okay, fine with me as well.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Re: Teach pg_receivewal to use lz4 compression
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, July 9th, 2021 at 04:49, Michael Paquier <michael@paquier.xyz> wrote:
Hi,
please find v3 of the patch attached, rebased to the current head.
Michael Paquier wrote:
* http://www.zlib.org/rfc-gzip.html.
- - For lz4 compressed segments
*/
This comment is incomplete.
Fixed.
+#define IsLZ4CompressXLogFileName(fname) \ - (strlen(fname) == XLOG_FNAME_LEN + strlen(".lz4") && \ - strspn(fname, "0123456789ABCDEF") == XLOG_FNAME_LEN && \ - strcmp((fname) + XLOG_FNAME_LEN, ".lz4") == 0)+#define IsLZ4PartialCompressXLogFileName(fname) \ - (strlen(fname) == XLOG_FNAME_LEN + strlen(".lz4.partial") && \ - strspn(fname, "0123456789ABCDEF") == XLOG_FNAME_LEN && \ - strcmp((fname) + XLOG_FNAME_LEN, ".lz4.partial") == 0)This is getting complicated. Would it be better to change this stuff
and switch to a routine that checks if a segment has a valid name, is
partial, and the type of compression that applied to it? It seems to
me that we should group iszlibcompress and islz4compress together with
the options available through compression_method.
Agreed and done.
- if (compresslevel != 0)
- {
- if (compression_method == COMPRESSION_NONE)
- {
- compression_method = COMPRESSION_ZLIB;
- }
- if (compression_method != COMPRESSION_ZLIB)
- {
- pg_log_error("cannot use --compress when "
- "--compression-method is not gzip");
- fprintf(stderr, _("Try \\"%s --help\\" for more information.\\n"),
- progname);
- exit(1);
- }
- }For compatibility where --compress enforces the use of zlib that would
work, but this needs a comment explaining the goal of this block. I
am wondering if it would be better to break the will and just complain
when specifying --compress without --compression-method though. That
would cause compatibility issues, but this would make folks aware of
the presence of LZ4, which does not sound bad to me either as ZLIB is
slower than LZ4 on all fronts.
Fair point. In v3 of the patch --compress requires --compression-method. Passing
0 as value errors out.
- else if (compression_method == COMPRESSION_ZLIB)
- {
- pg_log_error("cannot use --compression-method gzip when "
- "--compression is 0");
- fprintf(stderr, _("Try \\"%s --help\\" for more information.\\n"),
- progname);
- exit(1);
- }Hmm. It would be more natural to enforce a default compression level
in this case? The user is asking for a compression with zlib here.
Agreed. A default value of 5, which is in the middle point of options, has been
defined and used.
In addition, the tests have been adjusted to mimic the newly added gzip tests.
Cheers,
//Georgios
Show quoted text
---------------------------------------------------------------
Michael
Attachments:
v3-0001-Teach-pg_receivewal-to-use-lz4-compression.patchapplication/octet-stream; name=v3-0001-Teach-pg_receivewal-to-use-lz4-compression.patchDownload
From 43ce0965f6b87336f57cb2fe9bfb3dbda54b2f60 Mon Sep 17 00:00:00 2001
From: Georgios Kokolatos <gkokolatos@pm.me>
Date: Fri, 10 Sep 2021 08:02:09 +0000
Subject: [PATCH v3] Teach pg_receivewal to use lz4 compression
The program pg_receivewal can use gzip compression to store the received WAL.
This commit teaches it to also be able to use lz4 compression. It is required
that the binary is build using the -llz4 flag. It is enabled via the --with-lz4
flag on configuration time.
Previously, the user had to use the option --compress with a value between [0-9]
to denote that gzip compression was required. This specific behaviour has not
maintained. A newly introduced option --compression-method=[lz4|gzip] can be
used to ask for the logs to be compressed. Compression values can be selected
only when the compression method is gzip. A compression value of 0 now returns
an error.
Under the hood there is nothing exceptional to be noted. Tar based archives have
not yet been taught to use lz4 compression. If that is felt useful, then it is
easy to be added in the future.
Tests have been added to verify the creation and correctness of the generated
lz4 files. The later is achieved by the use of lz4 program. Autoconf has been
taught to unconditionally recognize the existance of the program and propagate
the information to the tests.
---
configure | 55 ++++
configure.ac | 2 +
doc/src/sgml/ref/pg_receivewal.sgml | 28 +-
src/Makefile.global.in | 1 +
src/bin/pg_basebackup/Makefile | 1 +
src/bin/pg_basebackup/pg_basebackup.c | 7 +-
src/bin/pg_basebackup/pg_receivewal.c | 267 +++++++++++++++----
src/bin/pg_basebackup/t/020_pg_receivewal.pl | 74 ++++-
src/bin/pg_basebackup/walmethods.c | 170 +++++++++++-
src/bin/pg_basebackup/walmethods.h | 12 +-
10 files changed, 545 insertions(+), 72 deletions(-)
diff --git a/configure b/configure
index 7542fe30a1..be55df9e0b 100755
--- a/configure
+++ b/configure
@@ -699,6 +699,7 @@ with_gnu_ld
LD
LDFLAGS_SL
LDFLAGS_EX
+LZ4
LZ4_LIBS
LZ4_CFLAGS
with_lz4
@@ -9661,6 +9662,60 @@ $as_echo_n "checking for TAR... " >&6; }
$as_echo "$TAR" >&6; }
fi
+if test -z "$LZ4"; then
+ for ac_prog in lz4
+do
+ # Extract the first word of "$ac_prog", so it can be a program name with args.
+set dummy $ac_prog; ac_word=$2
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
+$as_echo_n "checking for $ac_word... " >&6; }
+if ${ac_cv_path_LZ4+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ case $LZ4 in
+ [\\/]* | ?:[\\/]*)
+ ac_cv_path_LZ4="$LZ4" # Let the user override the test with a path.
+ ;;
+ *)
+ as_save_IFS=$IFS; IFS=$PATH_SEPARATOR
+for as_dir in $PATH
+do
+ IFS=$as_save_IFS
+ test -z "$as_dir" && as_dir=.
+ for ac_exec_ext in '' $ac_executable_extensions; do
+ if as_fn_executable_p "$as_dir/$ac_word$ac_exec_ext"; then
+ ac_cv_path_LZ4="$as_dir/$ac_word$ac_exec_ext"
+ $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5
+ break 2
+ fi
+done
+ done
+IFS=$as_save_IFS
+
+ ;;
+esac
+fi
+LZ4=$ac_cv_path_LZ4
+if test -n "$LZ4"; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: $LZ4" >&5
+$as_echo "$LZ4" >&6; }
+else
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+fi
+
+
+ test -n "$LZ4" && break
+done
+
+else
+ # Report the value of LZ4 in configure's output in all cases.
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for lz4" >&5
+$as_echo_n "checking for lz4... " >&6; }
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: $LZ4" >&5
+$as_echo "$LZ4" >&6; }
+fi
+
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether ln -s works" >&5
$as_echo_n "checking whether ln -s works... " >&6; }
LN_S=$as_ln_s
diff --git a/configure.ac b/configure.ac
index ed3cdb9a8e..3580e4a19c 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1062,6 +1062,8 @@ case $MKDIR_P in
*install-sh*) MKDIR_P='\${SHELL} \${top_srcdir}/config/install-sh -c -d';;
esac
+PGAC_PATH_PROGS(LZ4, lz4)
+
PGAC_PATH_BISON
PGAC_PATH_FLEX
diff --git a/doc/src/sgml/ref/pg_receivewal.sgml b/doc/src/sgml/ref/pg_receivewal.sgml
index 45b544cf49..654f2011d7 100644
--- a/doc/src/sgml/ref/pg_receivewal.sgml
+++ b/doc/src/sgml/ref/pg_receivewal.sgml
@@ -229,15 +229,35 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--compression-method=<replaceable class="parameter">level</replaceable></option></term>
+ <listitem>
+ <para>
+ Enables compression of write-ahead logs using the specified method.
+ Supported methods are <literal>lz4</literal> and
+ <literal>gzip</literal>.
+ The suffix <filename>.lz4</filename> or <filename>.gz</filename> will
+ automatically be added to all filenames for each method respectevilly.
+ For the <literal>lz4</literal> method to be available,
+ <productname>PostgreSQL</productname> must have been have been compiled
+ with <option>--with-lz4</option>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-Z <replaceable class="parameter">level</replaceable></option></term>
<term><option>--compress=<replaceable class="parameter">level</replaceable></option></term>
<listitem>
<para>
- Enables gzip compression of write-ahead logs, and specifies the
- compression level (0 through 9, 0 being no compression and 9 being best
- compression). The suffix <filename>.gz</filename> will
- automatically be added to all filenames.
+ Specifies the compression level (1 through 9, 1 being least compression
+ and 9 being most compression) for gzip compressed write-ahead logs. The
+ default value is 5.
+ </para>
+
+ <para>
+ It requires for <option>--compression-method</option> to be specified
+ as <literal>gzip</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 6e2f224cc4..91ba2240a2 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -341,6 +341,7 @@ perl_embed_ldflags = @perl_embed_ldflags@
AWK = @AWK@
LN_S = @LN_S@
+LZ4 = @LZ4@
MSGFMT = @MSGFMT@
MSGFMT_FLAGS = @MSGFMT_FLAGS@
MSGMERGE = @MSGMERGE@
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index 459d514183..387d728345 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -24,6 +24,7 @@ export TAR
# used by the command "gzip" to pass down options, so stick with a different
# name.
export GZIP_PROGRAM=$(GZIP)
+export LZ4
override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport)
diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c
index 7296eb97d0..964f4dfa84 100644
--- a/src/bin/pg_basebackup/pg_basebackup.c
+++ b/src/bin/pg_basebackup/pg_basebackup.c
@@ -555,10 +555,13 @@ LogStreamerMain(logstreamer_param *param)
stream.replication_slot = replication_slot;
if (format == 'p')
- stream.walmethod = CreateWalDirectoryMethod(param->xlog, 0,
+ stream.walmethod = CreateWalDirectoryMethod(param->xlog,
+ COMPRESSION_NONE, 0,
stream.do_sync);
else
- stream.walmethod = CreateWalTarMethod(param->xlog, compresslevel,
+ stream.walmethod = CreateWalTarMethod(param->xlog,
+ COMPRESSION_NONE /* argument is ignored */,
+ compresslevel,
stream.do_sync);
if (!ReceiveXlogStream(param->bgconn, &stream))
diff --git a/src/bin/pg_basebackup/pg_receivewal.c b/src/bin/pg_basebackup/pg_receivewal.c
index 9d1843728d..0847e42f90 100644
--- a/src/bin/pg_basebackup/pg_receivewal.c
+++ b/src/bin/pg_basebackup/pg_receivewal.c
@@ -29,9 +29,16 @@
#include "receivelog.h"
#include "streamutil.h"
+#ifdef HAVE_LIBLZ4
+#include "lz4frame.h"
+#endif
+
/* Time to sleep between reconnection attempts */
#define RECONNECT_SLEEP_TIME 5
+/* Default compression level for gzip compression method */
+#define DEFAULT_ZLIB_COMPRESSLEVEL 5
+
/* Global options */
static char *basedir = NULL;
static int verbose = 0;
@@ -45,6 +52,7 @@ static bool do_drop_slot = false;
static bool do_sync = true;
static bool synchronous = false;
static char *replication_slot = NULL;
+static WalCompressionMethod compression_method = COMPRESSION_NONE;
static XLogRecPtr endpos = InvalidXLogRecPtr;
@@ -63,16 +71,6 @@ disconnect_atexit(void)
PQfinish(conn);
}
-/* Routines to evaluate segment file format */
-#define IsCompressXLogFileName(fname) \
- (strlen(fname) == XLOG_FNAME_LEN + strlen(".gz") && \
- strspn(fname, "0123456789ABCDEF") == XLOG_FNAME_LEN && \
- strcmp((fname) + XLOG_FNAME_LEN, ".gz") == 0)
-#define IsPartialCompressXLogFileName(fname) \
- (strlen(fname) == XLOG_FNAME_LEN + strlen(".gz.partial") && \
- strspn(fname, "0123456789ABCDEF") == XLOG_FNAME_LEN && \
- strcmp((fname) + XLOG_FNAME_LEN, ".gz.partial") == 0)
-
static void
usage(void)
{
@@ -92,7 +90,10 @@ usage(void)
printf(_(" --synchronous flush write-ahead log immediately after writing\n"));
printf(_(" -v, --verbose output verbose messages\n"));
printf(_(" -V, --version output version information, then exit\n"));
- printf(_(" -Z, --compress=0-9 compress logs with given compression level\n"));
+ printf(_(" --compression-method=METHOD\n"
+ " use this method for compression\n"));
+ printf(_(" -Z, --compress=1-9 compress logs with given compression level (default: %d)\n"
+ " available only with --compression-method=gzip\n"), DEFAULT_ZLIB_COMPRESSLEVEL);
printf(_(" -?, --help show this help, then exit\n"));
printf(_("\nConnection options:\n"));
printf(_(" -d, --dbname=CONNSTR connection string\n"));
@@ -108,6 +109,79 @@ usage(void)
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
}
+
+/*
+ * Check if the filename looks like an xlog file. Also note if it is partial
+ * and/or compressed file.
+ */
+static bool
+is_xlogfilename(const char *filename, bool *ispartial,
+ WalCompressionMethod *wal_compression_method)
+{
+ size_t fname_len = strlen(filename);
+ size_t xlog_pattern_len = strspn(filename, "0123456789ABCDEF");
+
+ /* File does not look like a XLOG file */
+ if (xlog_pattern_len != XLOG_FNAME_LEN)
+ return false;
+
+ /* File looks like a complete uncompressed XLOG file */
+ if (fname_len == XLOG_FNAME_LEN)
+ {
+ *ispartial = false;
+ *wal_compression_method = COMPRESSION_NONE;
+ return true;
+ }
+
+ /* File looks like a complete zlib compressed XLOG file */
+ if ((fname_len == XLOG_FNAME_LEN + strlen(".gz")) &&
+ strcmp(filename + XLOG_FNAME_LEN, ".gz") == 0)
+ {
+ *ispartial = false;
+ *wal_compression_method = COMPRESSION_ZLIB;
+ return true;
+ }
+
+ /* File looks like a complete lz4 compressed XLOG file */
+ if ((fname_len == XLOG_FNAME_LEN + strlen(".lz4")) &&
+ strcmp(filename + XLOG_FNAME_LEN, ".lz4") == 0)
+ {
+ *ispartial = false;
+ *wal_compression_method = COMPRESSION_LZ4;
+ return true;
+ }
+
+ /* File looks like a partial uncompressed XLOG file */
+ if ((fname_len == XLOG_FNAME_LEN + strlen(".partial")) &&
+ strcmp(filename + XLOG_FNAME_LEN, ".partial") == 0)
+ {
+ *ispartial = true;
+ *wal_compression_method = COMPRESSION_NONE;
+ return true;
+ }
+
+ /* File looks like a partial zlib compressed XLOG file */
+ if ((fname_len == XLOG_FNAME_LEN + strlen(".gz.partial")) &&
+ strcmp(filename + XLOG_FNAME_LEN, ".gz.partial") == 0)
+ {
+ *ispartial = true;
+ *wal_compression_method = COMPRESSION_ZLIB;
+ return true;
+ }
+
+ /* File looks like a partial lz4 compressed XLOG file */
+ if ((fname_len == XLOG_FNAME_LEN + strlen(".lz4.partial")) &&
+ strcmp(filename + XLOG_FNAME_LEN, ".lz4.partial") == 0)
+ {
+ *ispartial = true;
+ *wal_compression_method = COMPRESSION_LZ4;
+ return true;
+ }
+
+ /* File does not look like something we recognise */
+ return false;
+}
+
static bool
stop_streaming(XLogRecPtr xlogpos, uint32 timeline, bool segment_finished)
{
@@ -213,33 +287,11 @@ FindStreamingStart(uint32 *tli)
{
uint32 tli;
XLogSegNo segno;
+ WalCompressionMethod wal_compression_method;
bool ispartial;
- bool iscompress;
- /*
- * Check if the filename looks like an xlog file, or a .partial file.
- */
- if (IsXLogFileName(dirent->d_name))
- {
- ispartial = false;
- iscompress = false;
- }
- else if (IsPartialXLogFileName(dirent->d_name))
- {
- ispartial = true;
- iscompress = false;
- }
- else if (IsCompressXLogFileName(dirent->d_name))
- {
- ispartial = false;
- iscompress = true;
- }
- else if (IsPartialCompressXLogFileName(dirent->d_name))
- {
- ispartial = true;
- iscompress = true;
- }
- else
+ if (!is_xlogfilename(dirent->d_name,
+ &ispartial, &wal_compression_method))
continue;
/*
@@ -250,14 +302,18 @@ FindStreamingStart(uint32 *tli)
/*
* Check that the segment has the right size, if it's supposed to be
* completed. For non-compressed segments just check the on-disk size
- * and see if it matches a completed segment. For compressed segments,
- * look at the last 4 bytes of the compressed file, which is where the
- * uncompressed size is located for gz files with a size lower than
- * 4GB, and then compare it to the size of a completed segment. The 4
- * last bytes correspond to the ISIZE member according to
+ * and see if it matches a completed segment. For zlib compressed
+ * segments, look at the last 4 bytes of the compressed file, which is
+ * where the uncompressed size is located for gz files with a size lower
+ * than 4GB, and then compare it to the size of a completed segment.
+ * The 4 last bytes correspond to the ISIZE member according to
* http://www.zlib.org/rfc-gzip.html.
+ *
+ * For lz4 compressed segments read the header using the exposed API and
+ * compare the uncompressed file size, stored in
+ * LZ4F_frameInfo_t{.contentSize}, to that of a completed segment.
*/
- if (!ispartial && !iscompress)
+ if (!ispartial && wal_compression_method == COMPRESSION_NONE)
{
struct stat statbuf;
char fullpath[MAXPGPATH * 2];
@@ -276,7 +332,7 @@ FindStreamingStart(uint32 *tli)
continue;
}
}
- else if (!ispartial && iscompress)
+ else if (!ispartial && wal_compression_method == COMPRESSION_ZLIB)
{
int fd;
char buf[4];
@@ -322,6 +378,70 @@ FindStreamingStart(uint32 *tli)
continue;
}
}
+ else if (!ispartial && compression_method == COMPRESSION_LZ4)
+ {
+#ifdef HAVE_LIBLZ4
+ int fd;
+ int r;
+ size_t consumed_len = LZ4F_HEADER_SIZE_MAX;
+ char buf[LZ4F_HEADER_SIZE_MAX];
+ char fullpath[MAXPGPATH * 2];
+ LZ4F_frameInfo_t frame_info = { 0 };
+ LZ4F_decompressionContext_t ctx = NULL;
+
+ snprintf(fullpath, sizeof(fullpath), "%s/%s", basedir, dirent->d_name);
+
+ fd = open(fullpath, O_RDONLY | PG_BINARY, 0);
+ if (fd < 0)
+ {
+ pg_log_error("could not open compressed file \"%s\": %m",
+ fullpath);
+ exit(1);
+ }
+
+ r = read(fd, buf, sizeof(buf));
+ if (r != sizeof(buf))
+ {
+ if (r < 0)
+ pg_log_error("could not read compressed file \"%s\": %m",
+ fullpath);
+ else
+ pg_log_error("could not read compressed file \"%s\": read %d of %lu",
+ fullpath, r, sizeof(buf));
+ exit(1);
+ }
+
+ if (LZ4F_isError(LZ4F_createDecompressionContext(&ctx, LZ4F_VERSION)))
+ {
+ pg_log_error("lz4 internal error");
+ exit(1);
+ }
+
+ LZ4F_getFrameInfo(ctx, &frame_info, (void *)buf, &consumed_len);
+ if (consumed_len <= LZ4F_HEADER_SIZE_MIN ||
+ consumed_len >= LZ4F_HEADER_SIZE_MAX)
+ {
+ pg_log_warning("compressed segment file \"%s\" has incorrect header size %lu, skipping",
+ dirent->d_name, consumed_len);
+ LZ4F_freeDecompressionContext(ctx);
+ continue;
+ }
+
+ if (frame_info.contentSize != WalSegSz)
+ {
+ pg_log_warning("compressed segment file \"%s\" has incorrect uncompressed size %lld, skipping",
+ dirent->d_name, frame_info.contentSize);
+ LZ4F_freeDecompressionContext(ctx);
+ continue;
+ }
+
+ LZ4F_freeDecompressionContext(ctx);
+#else
+ pg_log_error("cannot verify lz4 compressed segment file \"%s\", "
+ "this program was not build with lz4 support");
+ exit(1);
+#endif
+ }
/* Looks like a valid segment. Remember that we saw it. */
if ((segno > high_segno) ||
@@ -431,7 +551,9 @@ StreamLog(void)
stream.synchronous = synchronous;
stream.do_sync = do_sync;
stream.mark_done = false;
- stream.walmethod = CreateWalDirectoryMethod(basedir, compresslevel,
+ stream.walmethod = CreateWalDirectoryMethod(basedir,
+ compression_method,
+ compresslevel,
stream.do_sync);
stream.partial_suffix = ".partial";
stream.replication_slot = replication_slot;
@@ -482,6 +604,7 @@ main(int argc, char **argv)
{"status-interval", required_argument, NULL, 's'},
{"slot", required_argument, NULL, 'S'},
{"verbose", no_argument, NULL, 'v'},
+ {"compression-method", required_argument, NULL, 'I'},
{"compress", required_argument, NULL, 'Z'},
/* action */
{"create-slot", no_argument, NULL, 1},
@@ -567,8 +690,23 @@ main(int argc, char **argv)
case 'v':
verbose++;
break;
+ case 'I':
+ if (strcmp(optarg, "gzip") == 0)
+ {
+ compression_method = COMPRESSION_ZLIB;
+ }
+ else if (strcmp(optarg, "lz4") == 0)
+ {
+ compression_method = COMPRESSION_LZ4;
+ }
+ else
+ {
+ pg_log_error("invalid compression-method \"%s\"", optarg);
+ exit(1);
+ }
+ break;
case 'Z':
- if (!option_parse_int(optarg, "-Z/--compress", 0, 9,
+ if (!option_parse_int(optarg, "-Z/--compress", 1, 9,
&compresslevel))
exit(1);
break;
@@ -648,13 +786,46 @@ main(int argc, char **argv)
exit(1);
}
+
+ /*
+ * Compression related arguments
+ */
+ if (compression_method != COMPRESSION_NONE)
+ {
#ifndef HAVE_LIBZ
- if (compresslevel != 0)
+ if (compression_method == COMPRESSION_ZLIB)
+ {
+ pg_log_error("this build does not support compression via gzip");
+ exit(1);
+ }
+#endif
+#ifndef HAVE_LIBLZ4
+ if (compression_method == COMPRESSION_LZ4)
+ {
+ pg_log_error("this build does not support compression via lz4");
+ exit(1);
+ }
+#endif
+ }
+
+ if (compression_method != COMPRESSION_ZLIB && compresslevel != 0)
{
- pg_log_error("this build does not support compression");
+ pg_log_error("can only use --compress together with "
+ "--compression-method=gzip");
+#ifndef HAVE_LIBLZ4
+ pg_log_error("this build does not support compression via gzip");
+#endif
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+ progname);
exit(1);
}
-#endif
+
+ if (compression_method == COMPRESSION_ZLIB && compresslevel == 0)
+ {
+ pg_log_info("no --compression specified, will be using %d",
+ DEFAULT_ZLIB_COMPRESSLEVEL);
+ compresslevel = DEFAULT_ZLIB_COMPRESSLEVEL;
+ }
/*
* Check existence of destination folder.
diff --git a/src/bin/pg_basebackup/t/020_pg_receivewal.pl b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
index 0b33d73900..e01ae4c354 100644
--- a/src/bin/pg_basebackup/t/020_pg_receivewal.pl
+++ b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
@@ -5,7 +5,7 @@ use strict;
use warnings;
use TestLib;
use PostgresNode;
-use Test::More tests => 27;
+use Test::More tests => 33;
program_help_ok('pg_receivewal');
program_version_ok('pg_receivewal');
@@ -33,6 +33,13 @@ $primary->command_fails(
$primary->command_fails(
[ 'pg_receivewal', '-D', $stream_dir, '--synchronous', '--no-sync' ],
'failure if --synchronous specified with --no-sync');
+$primary->command_fails(
+ [
+ 'pg_receivewal', '-D', $stream_dir, '--compression-method', 'lz4',
+ '--compress', '1'
+ ],
+ 'failure if --compression-method=lz4 specified with --compress');
+
# Slot creation and drop
my $slot_name = 'test';
@@ -91,7 +98,9 @@ SKIP:
$primary->command_ok(
[
'pg_receivewal', '-D', $stream_dir, '--verbose',
- '--endpos', $nextlsn, '--compress', '1 ',
+ '--endpos', $nextlsn,
+ '--compression-method', 'gzip',
+ '--compress', '1 ',
'--no-loop'
],
"streaming some WAL using ZLIB compression");
@@ -128,14 +137,69 @@ SKIP:
"gzip verified the integrity of compressed WAL segments");
}
+# Check lz4 compression if available
+SKIP:
+{
+ skip "postgres was not built with LZ4 support", 5
+ if (!check_pg_config("#define HAVE_LIBLZ4 1"));
+
+ # Generate more WAL including one completed, compressed segment.
+ $primary->psql('postgres', 'SELECT pg_switch_wal();');
+ $nextlsn =
+ $primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
+ chomp($nextlsn);
+ $primary->psql('postgres',
+ 'INSERT INTO test_table VALUES (generate_series(201,300));');
+
+ # Stream up to the given position
+ $primary->command_ok(
+ [
+ 'pg_receivewal', '-D', $stream_dir, '--verbose',
+ '--endpos', $nextlsn, '--no-loop',
+ '--compression-method', 'lz4'
+ ],
+ 'streaming some WAL using --compression-method=lz4');
+
+ # Verify that the stored files are generated with their expected
+ # names.
+ my @lz4_wals = glob "$stream_dir/*.lz4";
+ is(scalar(@lz4_wals), 1,
+ "one WAL segment compressed with LZ4 was created");
+ my @lz4_partial_wals = glob "$stream_dir/*.lz4.partial";
+ is(scalar(@lz4_partial_wals),
+ 1, "one partial WAL segment compressed with LZ4 was created");
+
+ # Verify that the start streaming position is computed correctly by
+ # comparing it with the partial file generated previously. The name
+ # of the previous partial, now-completed WAL segment is updated, keeping
+ # its base number.
+ $partial_wals[0] =~ s/(\.gz)?\.partial$/.lz4/;
+ is($lz4_wals[0] eq $partial_wals[0],
+ 1, "one partial WAL segment is now completed");
+ # Update the list of partial wals with the current one.
+ @partial_wals = @lz4_partial_wals;
+
+ # Check the integrity of the completed segment, if lz4 is an available
+ # command.
+ my $lz4 = $ENV{LZ4};
+ skip "program lz4 is not found in your system", 1
+ if ( !defined $lz4
+ || $lz4 eq ''
+ || system_log($lz4, '--version') != 0);
+
+ my $lz4_is_valid = system_log($lz4, '-t', @lz4_wals);
+ is($lz4_is_valid, 0,
+ "lz4 verified the integrity of compressed WAL segments");
+}
+
# Verify that the start streaming position is computed and that the value is
-# correct regardless of whether ZLIB is available.
+# correct regardless of whether any compression is available.
$primary->psql('postgres', 'SELECT pg_switch_wal();');
$nextlsn =
$primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
chomp($nextlsn);
$primary->psql('postgres',
- 'INSERT INTO test_table VALUES (generate_series(200,300));');
+ 'INSERT INTO test_table VALUES (generate_series(301,400));');
$primary->command_ok(
[
'pg_receivewal', '-D', $stream_dir, '--verbose',
@@ -143,7 +207,7 @@ $primary->command_ok(
],
"streaming some WAL");
-$partial_wals[0] =~ s/(\.gz)?.partial//;
+$partial_wals[0] =~ s/(\.gz|\.lz4)?.partial//;
ok(-e $partial_wals[0], "check that previously partial WAL is now complete");
# Permissions on WAL files should be default
diff --git a/src/bin/pg_basebackup/walmethods.c b/src/bin/pg_basebackup/walmethods.c
index 8695647db4..684120ee8d 100644
--- a/src/bin/pg_basebackup/walmethods.c
+++ b/src/bin/pg_basebackup/walmethods.c
@@ -17,6 +17,10 @@
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
+
+#ifdef HAVE_LIBLZ4
+#include <lz4frame.h>
+#endif
#ifdef HAVE_LIBZ
#include <zlib.h>
#endif
@@ -30,6 +34,9 @@
/* Size of zlib buffer for .tar.gz */
#define ZLIB_OUT_SIZE 4096
+/* Size of lz4 input chunk for .lz4 */
+#define LZ4_IN_SIZE 4096
+
/*-------------------------------------------------------------------------
* WalDirectoryMethod - write wal to a directory looking like pg_wal
*-------------------------------------------------------------------------
@@ -40,9 +47,10 @@
*/
typedef struct DirectoryMethodData
{
- char *basedir;
- int compression;
- bool sync;
+ char *basedir;
+ WalCompressionMethod compression_method;
+ int compression;
+ bool sync;
} DirectoryMethodData;
static DirectoryMethodData *dir_data = NULL;
@@ -59,6 +67,11 @@ typedef struct DirectoryMethodFile
#ifdef HAVE_LIBZ
gzFile gzfp;
#endif
+#ifdef HAVE_LIBLZ4
+ LZ4F_compressionContext_t ctx;
+ size_t lz4bufsize;
+ void *lz4buf;
+#endif
} DirectoryMethodFile;
static const char *
@@ -74,7 +87,9 @@ dir_get_file_name(const char *pathname, const char *temp_suffix)
char *filename = pg_malloc0(MAXPGPATH * sizeof(char));
snprintf(filename, MAXPGPATH, "%s%s%s",
- pathname, dir_data->compression > 0 ? ".gz" : "",
+ pathname,
+ dir_data->compression_method == COMPRESSION_ZLIB ? ".gz" :
+ dir_data->compression_method == COMPRESSION_LZ4 ? ".lz4": "",
temp_suffix ? temp_suffix : "");
return filename;
@@ -90,6 +105,11 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
#ifdef HAVE_LIBZ
gzFile gzfp = NULL;
#endif
+#ifdef HAVE_LIBLZ4
+ LZ4F_compressionContext_t ctx = NULL;
+ size_t lz4bufsize = 0;
+ void *lz4buf = NULL;
+#endif
filename = dir_get_file_name(pathname, temp_suffix);
snprintf(tmppath, sizeof(tmppath), "%s/%s",
@@ -107,7 +127,7 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
return NULL;
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
{
gzfp = gzdopen(fd, "wb");
if (gzfp == NULL)
@@ -124,9 +144,55 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
}
}
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ LZ4F_preferences_t lz4preferences = { 0 };
+ size_t ctx_out;
+ size_t header_size;
+
+ /*
+ * Set all the preferences to default but do note contentSize. It will
+ * be needed in FindStreamingStart.
+ */
+ memset(&lz4preferences, 0, sizeof(LZ4F_frameInfo_t));
+ lz4preferences.frameInfo.contentSize = (unsigned long long)WalSegSz;
+ ctx_out = LZ4F_createCompressionContext(&ctx, LZ4F_VERSION);
+ lz4bufsize = LZ4F_compressBound(LZ4_IN_SIZE, &lz4preferences);
+ if (LZ4F_isError(ctx_out))
+ {
+ close(fd);
+ return NULL;
+ }
+
+ lz4buf = pg_malloc0(lz4bufsize);
+
+ /* add the header */
+ header_size = LZ4F_compressBegin(ctx, lz4buf, lz4bufsize, &lz4preferences);
+ if (LZ4F_isError(header_size))
+ {
+ close(fd);
+ return NULL;
+ }
+
+ errno = 0;
+ if (write(fd, lz4buf, header_size) != header_size)
+ {
+ int save_errno = errno;
+
+ close(fd);
+
+ /*
+ * If write didn't set errno, assume problem is no disk space.
+ */
+ errno = save_errno ? save_errno : ENOSPC;
+ return NULL;
+ }
+ }
+#endif
/* Do pre-padding on non-compressed files */
- if (pad_to_size && dir_data->compression == 0)
+ if (pad_to_size && dir_data->compression_method == COMPRESSION_NONE)
{
PGAlignedXLogBlock zerobuf;
int bytes;
@@ -171,7 +237,7 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
fsync_parent_path(tmppath) != 0)
{
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
gzclose(gzfp);
else
#endif
@@ -182,9 +248,18 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
f = pg_malloc0(sizeof(DirectoryMethodFile));
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
f->gzfp = gzfp;
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ f->ctx = ctx;
+ f->lz4buf = lz4buf;
+ f->lz4bufsize = lz4bufsize;
+ }
+#endif
+
f->fd = fd;
f->currpos = 0;
f->pathname = pg_strdup(pathname);
@@ -204,9 +279,46 @@ dir_write(Walfile f, const void *buf, size_t count)
Assert(f != NULL);
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
r = (ssize_t) gzwrite(df->gzfp, buf, count);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ size_t chunk;
+ size_t remaining;
+ const void *inbuf = buf;
+
+ remaining = count;
+ while (remaining > 0)
+ {
+ size_t compressed;
+
+ if (remaining > LZ4_IN_SIZE)
+ chunk = LZ4_IN_SIZE;
+ else
+ chunk = remaining;
+
+ remaining -= chunk;
+ compressed = LZ4F_compressUpdate(df->ctx,
+ df->lz4buf, df->lz4bufsize,
+ inbuf, chunk,
+ NULL);
+
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+
+ inbuf = ((char *)inbuf) + chunk;
+ }
+
+ /* Our caller keeps track of the uncompressed size. */
+ r = (ssize_t)count;
+ }
+ else
#endif
r = write(df->fd, buf, count);
if (r > 0)
@@ -234,9 +346,26 @@ dir_close(Walfile f, WalCloseMethod method)
Assert(f != NULL);
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
r = gzclose(df->gzfp);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ /* Flush any internal buffers */
+ size_t compressed = LZ4F_compressEnd(df->ctx,
+ df->lz4buf, df->lz4bufsize,
+ NULL);
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+
+ r = close(df->fd);
+ }
+ else
#endif
r = close(df->fd);
@@ -291,6 +420,12 @@ dir_close(Walfile f, WalCloseMethod method)
}
}
+#ifdef HAVE_LIBLZ4
+ pg_free(df->lz4buf);
+ /* supports free on NULL */
+ LZ4F_freeCompressionContext(df->ctx);
+#endif
+
pg_free(df->pathname);
pg_free(df->fullpath);
if (df->temp_suffix)
@@ -373,7 +508,9 @@ dir_finish(void)
WalWriteMethod *
-CreateWalDirectoryMethod(const char *basedir, int compression, bool sync)
+CreateWalDirectoryMethod(const char *basedir,
+ WalCompressionMethod compression_method,
+ int compression, bool sync)
{
WalWriteMethod *method;
@@ -391,6 +528,7 @@ CreateWalDirectoryMethod(const char *basedir, int compression, bool sync)
method->getlasterror = dir_getlasterror;
dir_data = pg_malloc0(sizeof(DirectoryMethodData));
+ dir_data->compression_method = compression_method;
dir_data->compression = compression;
dir_data->basedir = pg_strdup(basedir);
dir_data->sync = sync;
@@ -1031,8 +1169,16 @@ tar_finish(void)
return true;
}
+/*
+ * The argument compression_method is currently ignored. It is in place for
+ * symmetry with CreateWalDirectoryMethod which uses it for distinguishing
+ * between the different compression methods. CreateWalTarMethod and its family
+ * of functions handle only zlib compression.
+ */
WalWriteMethod *
-CreateWalTarMethod(const char *tarbase, int compression, bool sync)
+CreateWalTarMethod(const char *tarbase,
+ WalCompressionMethod compression_method,
+ int compression, bool sync)
{
WalWriteMethod *method;
const char *suffix = (compression != 0) ? ".tar.gz" : ".tar";
diff --git a/src/bin/pg_basebackup/walmethods.h b/src/bin/pg_basebackup/walmethods.h
index 4abdfd8333..872b677da5 100644
--- a/src/bin/pg_basebackup/walmethods.h
+++ b/src/bin/pg_basebackup/walmethods.h
@@ -19,6 +19,13 @@ typedef enum
CLOSE_NO_RENAME
} WalCloseMethod;
+typedef enum
+{
+ COMPRESSION_LZ4,
+ COMPRESSION_ZLIB,
+ COMPRESSION_NONE
+} WalCompressionMethod;
+
/*
* A WalWriteMethod structure represents the different methods used
* to write the streaming WAL as it's received.
@@ -95,8 +102,11 @@ struct WalWriteMethod
* not all those required for pg_receivewal)
*/
WalWriteMethod *CreateWalDirectoryMethod(const char *basedir,
+ WalCompressionMethod compression_method,
int compression, bool sync);
-WalWriteMethod *CreateWalTarMethod(const char *tarbase, int compression, bool sync);
+WalWriteMethod *CreateWalTarMethod(const char *tarbase,
+ WalCompressionMethod compression_method,
+ int compression, bool sync);
/* Cleanup routines for previously-created methods */
void FreeWalDirectoryMethod(void);
--
2.25.1
Re: Teach pg_receivewal to use lz4 compression
@@ -250,14 +302,18 @@ FindStreamingStart(uint32 *tli)
/*
* Check that the segment has the right size, if it's supposed to be
* completed. For non-compressed segments just check the on-disk size
- * and see if it matches a completed segment. For compressed segments,
- * look at the last 4 bytes of the compressed file, which is where the
- * uncompressed size is located for gz files with a size lower than
- * 4GB, and then compare it to the size of a completed segment. The 4
- * last bytes correspond to the ISIZE member according to
+ * and see if it matches a completed segment. For zlib compressed
+ * segments, look at the last 4 bytes of the compressed file, which is
+ * where the uncompressed size is located for gz files with a size lower
+ * than 4GB, and then compare it to the size of a completed segment.
+ * The 4 last bytes correspond to the ISIZE member according to
* http://www.zlib.org/rfc-gzip.html.
+ *
+ * For lz4 compressed segments read the header using the exposed API and
+ * compare the uncompressed file size, stored in
+ * LZ4F_frameInfo_t{.contentSize}, to that of a completed segment.
*/
- if (!ispartial && !iscompress)
+ if (!ispartial && wal_compression_method == COMPRESSION_NONE)
{
struct stat statbuf;
char fullpath[MAXPGPATH * 2];
@@ -276,7 +332,7 @@ FindStreamingStart(uint32 *tli)
continue;
}
}
- else if (!ispartial && iscompress)
+ else if (!ispartial && wal_compression_method == COMPRESSION_ZLIB)
{
int fd;
char buf[4];
@@ -322,6 +378,70 @@ FindStreamingStart(uint32 *tli)
continue;
}
}
+ else if (!ispartial && compression_method == COMPRESSION_LZ4)
+ {
+#ifdef HAVE_LIBLZ4
+ int fd;
+ int r;
+ size_t consumed_len = LZ4F_HEADER_SIZE_MAX;
+ char buf[LZ4F_HEADER_SIZE_MAX];
+ char fullpath[MAXPGPATH * 2];
+ LZ4F_frameInfo_t frame_info = { 0 };
+ LZ4F_decompressionContext_t ctx = NULL;
+
+ snprintf(fullpath, sizeof(fullpath), "%s/%s", basedir, dirent->d_name);
+
+ fd = open(fullpath, O_RDONLY | PG_BINARY, 0);Should close the fd before exit or abort.
Re: Teach pg_receivewal to use lz4 compression
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Saturday, September 11th, 2021 at 07:02, Jian Guo <gjian@vmware.com> wrote:
Hi,
thank you for looking at the patch.
- LZ4F_decompressionContext_t ctx = NULL;
- snprintf(fullpath, sizeof(fullpath), "%s/%s", basedir, dirent->d_name);
- fd = open(fullpath, O_RDONLY | PG_BINARY, 0);Should close the fd before exit or abort.
You are correct. Please find version 4 attached.
Cheers,
//Georgios
Attachments:
v4-0001-Teach-pg_receivewal-to-use-lz4-compression.patchapplication/octet-stream; name=v4-0001-Teach-pg_receivewal-to-use-lz4-compression.patchDownload
From 0d09ff375e2f6ea252fe76eb292466983d28deab Mon Sep 17 00:00:00 2001
From: Georgios Kokolatos <gkokolatos@pm.me>
Date: Mon, 13 Sep 2021 08:18:07 +0000
Subject: [PATCH v4] Teach pg_receivewal to use lz4 compression
The program pg_receivewal can use gzip compression to store the received WAL.
This commit teaches it to also be able to use lz4 compression. It is required
that the binary is build using the -llz4 flag. It is enabled via the --with-lz4
flag on configuration time.
Previously, the user had to use the option --compress with a value between [0-9]
to denote that gzip compression was required. This specific behaviour has not
maintained. A newly introduced option --compression-method=[lz4|gzip] can be
used to ask for the logs to be compressed. Compression values can be selected
only when the compression method is gzip. A compression value of 0 now returns
an error.
Under the hood there is nothing exceptional to be noted. Tar based archives have
not yet been taught to use lz4 compression. If that is felt useful, then it is
easy to be added in the future.
Tests have been added to verify the creation and correctness of the generated
lz4 files. The later is achieved by the use of lz4 program. Autoconf has been
taught to unconditionally recognize the existance of the program and propagate
the information to the tests.
---
configure | 55 ++++
configure.ac | 2 +
doc/src/sgml/ref/pg_receivewal.sgml | 28 +-
src/Makefile.global.in | 1 +
src/bin/pg_basebackup/Makefile | 1 +
src/bin/pg_basebackup/pg_basebackup.c | 7 +-
src/bin/pg_basebackup/pg_receivewal.c | 268 +++++++++++++++----
src/bin/pg_basebackup/receivelog.c | 2 +-
src/bin/pg_basebackup/t/020_pg_receivewal.pl | 74 ++++-
src/bin/pg_basebackup/walmethods.c | 170 +++++++++++-
src/bin/pg_basebackup/walmethods.h | 12 +-
11 files changed, 547 insertions(+), 73 deletions(-)
diff --git a/configure b/configure
index 7542fe30a1..be55df9e0b 100755
--- a/configure
+++ b/configure
@@ -699,6 +699,7 @@ with_gnu_ld
LD
LDFLAGS_SL
LDFLAGS_EX
+LZ4
LZ4_LIBS
LZ4_CFLAGS
with_lz4
@@ -9661,6 +9662,60 @@ $as_echo_n "checking for TAR... " >&6; }
$as_echo "$TAR" >&6; }
fi
+if test -z "$LZ4"; then
+ for ac_prog in lz4
+do
+ # Extract the first word of "$ac_prog", so it can be a program name with args.
+set dummy $ac_prog; ac_word=$2
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
+$as_echo_n "checking for $ac_word... " >&6; }
+if ${ac_cv_path_LZ4+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ case $LZ4 in
+ [\\/]* | ?:[\\/]*)
+ ac_cv_path_LZ4="$LZ4" # Let the user override the test with a path.
+ ;;
+ *)
+ as_save_IFS=$IFS; IFS=$PATH_SEPARATOR
+for as_dir in $PATH
+do
+ IFS=$as_save_IFS
+ test -z "$as_dir" && as_dir=.
+ for ac_exec_ext in '' $ac_executable_extensions; do
+ if as_fn_executable_p "$as_dir/$ac_word$ac_exec_ext"; then
+ ac_cv_path_LZ4="$as_dir/$ac_word$ac_exec_ext"
+ $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5
+ break 2
+ fi
+done
+ done
+IFS=$as_save_IFS
+
+ ;;
+esac
+fi
+LZ4=$ac_cv_path_LZ4
+if test -n "$LZ4"; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: $LZ4" >&5
+$as_echo "$LZ4" >&6; }
+else
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+fi
+
+
+ test -n "$LZ4" && break
+done
+
+else
+ # Report the value of LZ4 in configure's output in all cases.
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for lz4" >&5
+$as_echo_n "checking for lz4... " >&6; }
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: $LZ4" >&5
+$as_echo "$LZ4" >&6; }
+fi
+
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether ln -s works" >&5
$as_echo_n "checking whether ln -s works... " >&6; }
LN_S=$as_ln_s
diff --git a/configure.ac b/configure.ac
index ed3cdb9a8e..3580e4a19c 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1062,6 +1062,8 @@ case $MKDIR_P in
*install-sh*) MKDIR_P='\${SHELL} \${top_srcdir}/config/install-sh -c -d';;
esac
+PGAC_PATH_PROGS(LZ4, lz4)
+
PGAC_PATH_BISON
PGAC_PATH_FLEX
diff --git a/doc/src/sgml/ref/pg_receivewal.sgml b/doc/src/sgml/ref/pg_receivewal.sgml
index 45b544cf49..654f2011d7 100644
--- a/doc/src/sgml/ref/pg_receivewal.sgml
+++ b/doc/src/sgml/ref/pg_receivewal.sgml
@@ -229,15 +229,35 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--compression-method=<replaceable class="parameter">level</replaceable></option></term>
+ <listitem>
+ <para>
+ Enables compression of write-ahead logs using the specified method.
+ Supported methods are <literal>lz4</literal> and
+ <literal>gzip</literal>.
+ The suffix <filename>.lz4</filename> or <filename>.gz</filename> will
+ automatically be added to all filenames for each method respectevilly.
+ For the <literal>lz4</literal> method to be available,
+ <productname>PostgreSQL</productname> must have been have been compiled
+ with <option>--with-lz4</option>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-Z <replaceable class="parameter">level</replaceable></option></term>
<term><option>--compress=<replaceable class="parameter">level</replaceable></option></term>
<listitem>
<para>
- Enables gzip compression of write-ahead logs, and specifies the
- compression level (0 through 9, 0 being no compression and 9 being best
- compression). The suffix <filename>.gz</filename> will
- automatically be added to all filenames.
+ Specifies the compression level (1 through 9, 1 being least compression
+ and 9 being most compression) for gzip compressed write-ahead logs. The
+ default value is 5.
+ </para>
+
+ <para>
+ It requires for <option>--compression-method</option> to be specified
+ as <literal>gzip</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 6e2f224cc4..91ba2240a2 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -341,6 +341,7 @@ perl_embed_ldflags = @perl_embed_ldflags@
AWK = @AWK@
LN_S = @LN_S@
+LZ4 = @LZ4@
MSGFMT = @MSGFMT@
MSGFMT_FLAGS = @MSGFMT_FLAGS@
MSGMERGE = @MSGMERGE@
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index 459d514183..387d728345 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -24,6 +24,7 @@ export TAR
# used by the command "gzip" to pass down options, so stick with a different
# name.
export GZIP_PROGRAM=$(GZIP)
+export LZ4
override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport)
diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c
index 7296eb97d0..964f4dfa84 100644
--- a/src/bin/pg_basebackup/pg_basebackup.c
+++ b/src/bin/pg_basebackup/pg_basebackup.c
@@ -555,10 +555,13 @@ LogStreamerMain(logstreamer_param *param)
stream.replication_slot = replication_slot;
if (format == 'p')
- stream.walmethod = CreateWalDirectoryMethod(param->xlog, 0,
+ stream.walmethod = CreateWalDirectoryMethod(param->xlog,
+ COMPRESSION_NONE, 0,
stream.do_sync);
else
- stream.walmethod = CreateWalTarMethod(param->xlog, compresslevel,
+ stream.walmethod = CreateWalTarMethod(param->xlog,
+ COMPRESSION_NONE /* argument is ignored */,
+ compresslevel,
stream.do_sync);
if (!ReceiveXlogStream(param->bgconn, &stream))
diff --git a/src/bin/pg_basebackup/pg_receivewal.c b/src/bin/pg_basebackup/pg_receivewal.c
index 9d1843728d..1447d16270 100644
--- a/src/bin/pg_basebackup/pg_receivewal.c
+++ b/src/bin/pg_basebackup/pg_receivewal.c
@@ -29,9 +29,16 @@
#include "receivelog.h"
#include "streamutil.h"
+#ifdef HAVE_LIBLZ4
+#include "lz4frame.h"
+#endif
+
/* Time to sleep between reconnection attempts */
#define RECONNECT_SLEEP_TIME 5
+/* Default compression level for gzip compression method */
+#define DEFAULT_ZLIB_COMPRESSLEVEL 5
+
/* Global options */
static char *basedir = NULL;
static int verbose = 0;
@@ -45,6 +52,7 @@ static bool do_drop_slot = false;
static bool do_sync = true;
static bool synchronous = false;
static char *replication_slot = NULL;
+static WalCompressionMethod compression_method = COMPRESSION_NONE;
static XLogRecPtr endpos = InvalidXLogRecPtr;
@@ -63,16 +71,6 @@ disconnect_atexit(void)
PQfinish(conn);
}
-/* Routines to evaluate segment file format */
-#define IsCompressXLogFileName(fname) \
- (strlen(fname) == XLOG_FNAME_LEN + strlen(".gz") && \
- strspn(fname, "0123456789ABCDEF") == XLOG_FNAME_LEN && \
- strcmp((fname) + XLOG_FNAME_LEN, ".gz") == 0)
-#define IsPartialCompressXLogFileName(fname) \
- (strlen(fname) == XLOG_FNAME_LEN + strlen(".gz.partial") && \
- strspn(fname, "0123456789ABCDEF") == XLOG_FNAME_LEN && \
- strcmp((fname) + XLOG_FNAME_LEN, ".gz.partial") == 0)
-
static void
usage(void)
{
@@ -92,7 +90,10 @@ usage(void)
printf(_(" --synchronous flush write-ahead log immediately after writing\n"));
printf(_(" -v, --verbose output verbose messages\n"));
printf(_(" -V, --version output version information, then exit\n"));
- printf(_(" -Z, --compress=0-9 compress logs with given compression level\n"));
+ printf(_(" --compression-method=METHOD\n"
+ " use this method for compression\n"));
+ printf(_(" -Z, --compress=1-9 compress logs with given compression level (default: %d)\n"
+ " available only with --compression-method=gzip\n"), DEFAULT_ZLIB_COMPRESSLEVEL);
printf(_(" -?, --help show this help, then exit\n"));
printf(_("\nConnection options:\n"));
printf(_(" -d, --dbname=CONNSTR connection string\n"));
@@ -108,6 +109,79 @@ usage(void)
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
}
+
+/*
+ * Check if the filename looks like an xlog file. Also note if it is partial
+ * and/or compressed file.
+ */
+static bool
+is_xlogfilename(const char *filename, bool *ispartial,
+ WalCompressionMethod *wal_compression_method)
+{
+ size_t fname_len = strlen(filename);
+ size_t xlog_pattern_len = strspn(filename, "0123456789ABCDEF");
+
+ /* File does not look like a XLOG file */
+ if (xlog_pattern_len != XLOG_FNAME_LEN)
+ return false;
+
+ /* File looks like a complete uncompressed XLOG file */
+ if (fname_len == XLOG_FNAME_LEN)
+ {
+ *ispartial = false;
+ *wal_compression_method = COMPRESSION_NONE;
+ return true;
+ }
+
+ /* File looks like a complete zlib compressed XLOG file */
+ if ((fname_len == XLOG_FNAME_LEN + strlen(".gz")) &&
+ strcmp(filename + XLOG_FNAME_LEN, ".gz") == 0)
+ {
+ *ispartial = false;
+ *wal_compression_method = COMPRESSION_ZLIB;
+ return true;
+ }
+
+ /* File looks like a complete lz4 compressed XLOG file */
+ if ((fname_len == XLOG_FNAME_LEN + strlen(".lz4")) &&
+ strcmp(filename + XLOG_FNAME_LEN, ".lz4") == 0)
+ {
+ *ispartial = false;
+ *wal_compression_method = COMPRESSION_LZ4;
+ return true;
+ }
+
+ /* File looks like a partial uncompressed XLOG file */
+ if ((fname_len == XLOG_FNAME_LEN + strlen(".partial")) &&
+ strcmp(filename + XLOG_FNAME_LEN, ".partial") == 0)
+ {
+ *ispartial = true;
+ *wal_compression_method = COMPRESSION_NONE;
+ return true;
+ }
+
+ /* File looks like a partial zlib compressed XLOG file */
+ if ((fname_len == XLOG_FNAME_LEN + strlen(".gz.partial")) &&
+ strcmp(filename + XLOG_FNAME_LEN, ".gz.partial") == 0)
+ {
+ *ispartial = true;
+ *wal_compression_method = COMPRESSION_ZLIB;
+ return true;
+ }
+
+ /* File looks like a partial lz4 compressed XLOG file */
+ if ((fname_len == XLOG_FNAME_LEN + strlen(".lz4.partial")) &&
+ strcmp(filename + XLOG_FNAME_LEN, ".lz4.partial") == 0)
+ {
+ *ispartial = true;
+ *wal_compression_method = COMPRESSION_LZ4;
+ return true;
+ }
+
+ /* File does not look like something we recognise */
+ return false;
+}
+
static bool
stop_streaming(XLogRecPtr xlogpos, uint32 timeline, bool segment_finished)
{
@@ -213,33 +287,11 @@ FindStreamingStart(uint32 *tli)
{
uint32 tli;
XLogSegNo segno;
+ WalCompressionMethod wal_compression_method;
bool ispartial;
- bool iscompress;
- /*
- * Check if the filename looks like an xlog file, or a .partial file.
- */
- if (IsXLogFileName(dirent->d_name))
- {
- ispartial = false;
- iscompress = false;
- }
- else if (IsPartialXLogFileName(dirent->d_name))
- {
- ispartial = true;
- iscompress = false;
- }
- else if (IsCompressXLogFileName(dirent->d_name))
- {
- ispartial = false;
- iscompress = true;
- }
- else if (IsPartialCompressXLogFileName(dirent->d_name))
- {
- ispartial = true;
- iscompress = true;
- }
- else
+ if (!is_xlogfilename(dirent->d_name,
+ &ispartial, &wal_compression_method))
continue;
/*
@@ -250,14 +302,18 @@ FindStreamingStart(uint32 *tli)
/*
* Check that the segment has the right size, if it's supposed to be
* completed. For non-compressed segments just check the on-disk size
- * and see if it matches a completed segment. For compressed segments,
- * look at the last 4 bytes of the compressed file, which is where the
- * uncompressed size is located for gz files with a size lower than
- * 4GB, and then compare it to the size of a completed segment. The 4
- * last bytes correspond to the ISIZE member according to
+ * and see if it matches a completed segment. For zlib compressed
+ * segments, look at the last 4 bytes of the compressed file, which is
+ * where the uncompressed size is located for gz files with a size lower
+ * than 4GB, and then compare it to the size of a completed segment.
+ * The 4 last bytes correspond to the ISIZE member according to
* http://www.zlib.org/rfc-gzip.html.
+ *
+ * For lz4 compressed segments read the header using the exposed API and
+ * compare the uncompressed file size, stored in
+ * LZ4F_frameInfo_t{.contentSize}, to that of a completed segment.
*/
- if (!ispartial && !iscompress)
+ if (!ispartial && wal_compression_method == COMPRESSION_NONE)
{
struct stat statbuf;
char fullpath[MAXPGPATH * 2];
@@ -276,7 +332,7 @@ FindStreamingStart(uint32 *tli)
continue;
}
}
- else if (!ispartial && iscompress)
+ else if (!ispartial && wal_compression_method == COMPRESSION_ZLIB)
{
int fd;
char buf[4];
@@ -322,6 +378,71 @@ FindStreamingStart(uint32 *tli)
continue;
}
}
+ else if (!ispartial && compression_method == COMPRESSION_LZ4)
+ {
+#ifdef HAVE_LIBLZ4
+ int fd;
+ int r;
+ size_t consumed_len = LZ4F_HEADER_SIZE_MAX;
+ char buf[LZ4F_HEADER_SIZE_MAX];
+ char fullpath[MAXPGPATH * 2];
+ LZ4F_frameInfo_t frame_info = { 0 };
+ LZ4F_decompressionContext_t ctx = NULL;
+
+ snprintf(fullpath, sizeof(fullpath), "%s/%s", basedir, dirent->d_name);
+
+ fd = open(fullpath, O_RDONLY | PG_BINARY, 0);
+ if (fd < 0)
+ {
+ pg_log_error("could not open compressed file \"%s\": %m",
+ fullpath);
+ exit(1);
+ }
+
+ r = read(fd, buf, sizeof(buf));
+ if (r != sizeof(buf))
+ {
+ if (r < 0)
+ pg_log_error("could not read compressed file \"%s\": %m",
+ fullpath);
+ else
+ pg_log_error("could not read compressed file \"%s\": read %d of %lu",
+ fullpath, r, sizeof(buf));
+ exit(1);
+ }
+ close(fd);
+
+ if (LZ4F_isError(LZ4F_createDecompressionContext(&ctx, LZ4F_VERSION)))
+ {
+ pg_log_error("lz4 internal error");
+ exit(1);
+ }
+
+ LZ4F_getFrameInfo(ctx, &frame_info, (void *)buf, &consumed_len);
+ if (consumed_len <= LZ4F_HEADER_SIZE_MIN ||
+ consumed_len >= LZ4F_HEADER_SIZE_MAX)
+ {
+ pg_log_warning("compressed segment file \"%s\" has incorrect header size %lu, skipping",
+ dirent->d_name, consumed_len);
+ LZ4F_freeDecompressionContext(ctx);
+ continue;
+ }
+
+ if (frame_info.contentSize != WalSegSz)
+ {
+ pg_log_warning("compressed segment file \"%s\" has incorrect uncompressed size %lld, skipping",
+ dirent->d_name, frame_info.contentSize);
+ LZ4F_freeDecompressionContext(ctx);
+ continue;
+ }
+
+ LZ4F_freeDecompressionContext(ctx);
+#else
+ pg_log_error("cannot verify lz4 compressed segment file \"%s\", "
+ "this program was not build with lz4 support");
+ exit(1);
+#endif
+ }
/* Looks like a valid segment. Remember that we saw it. */
if ((segno > high_segno) ||
@@ -431,7 +552,9 @@ StreamLog(void)
stream.synchronous = synchronous;
stream.do_sync = do_sync;
stream.mark_done = false;
- stream.walmethod = CreateWalDirectoryMethod(basedir, compresslevel,
+ stream.walmethod = CreateWalDirectoryMethod(basedir,
+ compression_method,
+ compresslevel,
stream.do_sync);
stream.partial_suffix = ".partial";
stream.replication_slot = replication_slot;
@@ -482,6 +605,7 @@ main(int argc, char **argv)
{"status-interval", required_argument, NULL, 's'},
{"slot", required_argument, NULL, 'S'},
{"verbose", no_argument, NULL, 'v'},
+ {"compression-method", required_argument, NULL, 'I'},
{"compress", required_argument, NULL, 'Z'},
/* action */
{"create-slot", no_argument, NULL, 1},
@@ -567,8 +691,23 @@ main(int argc, char **argv)
case 'v':
verbose++;
break;
+ case 'I':
+ if (strcmp(optarg, "gzip") == 0)
+ {
+ compression_method = COMPRESSION_ZLIB;
+ }
+ else if (strcmp(optarg, "lz4") == 0)
+ {
+ compression_method = COMPRESSION_LZ4;
+ }
+ else
+ {
+ pg_log_error("invalid compression-method \"%s\"", optarg);
+ exit(1);
+ }
+ break;
case 'Z':
- if (!option_parse_int(optarg, "-Z/--compress", 0, 9,
+ if (!option_parse_int(optarg, "-Z/--compress", 1, 9,
&compresslevel))
exit(1);
break;
@@ -648,13 +787,46 @@ main(int argc, char **argv)
exit(1);
}
+
+ /*
+ * Compression related arguments
+ */
+ if (compression_method != COMPRESSION_NONE)
+ {
#ifndef HAVE_LIBZ
- if (compresslevel != 0)
+ if (compression_method == COMPRESSION_ZLIB)
+ {
+ pg_log_error("this build does not support compression via gzip");
+ exit(1);
+ }
+#endif
+#ifndef HAVE_LIBLZ4
+ if (compression_method == COMPRESSION_LZ4)
+ {
+ pg_log_error("this build does not support compression via lz4");
+ exit(1);
+ }
+#endif
+ }
+
+ if (compression_method != COMPRESSION_ZLIB && compresslevel != 0)
{
- pg_log_error("this build does not support compression");
+ pg_log_error("can only use --compress together with "
+ "--compression-method=gzip");
+#ifndef HAVE_LIBLZ4
+ pg_log_error("this build does not support compression via gzip");
+#endif
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+ progname);
exit(1);
}
-#endif
+
+ if (compression_method == COMPRESSION_ZLIB && compresslevel == 0)
+ {
+ pg_log_info("no --compression specified, will be using %d",
+ DEFAULT_ZLIB_COMPRESSLEVEL);
+ compresslevel = DEFAULT_ZLIB_COMPRESSLEVEL;
+ }
/*
* Check existence of destination folder.
diff --git a/src/bin/pg_basebackup/receivelog.c b/src/bin/pg_basebackup/receivelog.c
index 9601fd8d9c..71c80fd3cb 100644
--- a/src/bin/pg_basebackup/receivelog.c
+++ b/src/bin/pg_basebackup/receivelog.c
@@ -109,7 +109,7 @@ open_walfile(StreamCtl *stream, XLogRecPtr startpoint)
* When streaming to tar, no file with this name will exist before, so we
* never have to verify a size.
*/
- if (stream->walmethod->compression() == 0 &&
+ if (stream->walmethod->compression() == COMPRESSION_NONE &&
stream->walmethod->existsfile(fn))
{
size = stream->walmethod->get_file_size(fn);
diff --git a/src/bin/pg_basebackup/t/020_pg_receivewal.pl b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
index 0b33d73900..e01ae4c354 100644
--- a/src/bin/pg_basebackup/t/020_pg_receivewal.pl
+++ b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
@@ -5,7 +5,7 @@ use strict;
use warnings;
use TestLib;
use PostgresNode;
-use Test::More tests => 27;
+use Test::More tests => 33;
program_help_ok('pg_receivewal');
program_version_ok('pg_receivewal');
@@ -33,6 +33,13 @@ $primary->command_fails(
$primary->command_fails(
[ 'pg_receivewal', '-D', $stream_dir, '--synchronous', '--no-sync' ],
'failure if --synchronous specified with --no-sync');
+$primary->command_fails(
+ [
+ 'pg_receivewal', '-D', $stream_dir, '--compression-method', 'lz4',
+ '--compress', '1'
+ ],
+ 'failure if --compression-method=lz4 specified with --compress');
+
# Slot creation and drop
my $slot_name = 'test';
@@ -91,7 +98,9 @@ SKIP:
$primary->command_ok(
[
'pg_receivewal', '-D', $stream_dir, '--verbose',
- '--endpos', $nextlsn, '--compress', '1 ',
+ '--endpos', $nextlsn,
+ '--compression-method', 'gzip',
+ '--compress', '1 ',
'--no-loop'
],
"streaming some WAL using ZLIB compression");
@@ -128,14 +137,69 @@ SKIP:
"gzip verified the integrity of compressed WAL segments");
}
+# Check lz4 compression if available
+SKIP:
+{
+ skip "postgres was not built with LZ4 support", 5
+ if (!check_pg_config("#define HAVE_LIBLZ4 1"));
+
+ # Generate more WAL including one completed, compressed segment.
+ $primary->psql('postgres', 'SELECT pg_switch_wal();');
+ $nextlsn =
+ $primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
+ chomp($nextlsn);
+ $primary->psql('postgres',
+ 'INSERT INTO test_table VALUES (generate_series(201,300));');
+
+ # Stream up to the given position
+ $primary->command_ok(
+ [
+ 'pg_receivewal', '-D', $stream_dir, '--verbose',
+ '--endpos', $nextlsn, '--no-loop',
+ '--compression-method', 'lz4'
+ ],
+ 'streaming some WAL using --compression-method=lz4');
+
+ # Verify that the stored files are generated with their expected
+ # names.
+ my @lz4_wals = glob "$stream_dir/*.lz4";
+ is(scalar(@lz4_wals), 1,
+ "one WAL segment compressed with LZ4 was created");
+ my @lz4_partial_wals = glob "$stream_dir/*.lz4.partial";
+ is(scalar(@lz4_partial_wals),
+ 1, "one partial WAL segment compressed with LZ4 was created");
+
+ # Verify that the start streaming position is computed correctly by
+ # comparing it with the partial file generated previously. The name
+ # of the previous partial, now-completed WAL segment is updated, keeping
+ # its base number.
+ $partial_wals[0] =~ s/(\.gz)?\.partial$/.lz4/;
+ is($lz4_wals[0] eq $partial_wals[0],
+ 1, "one partial WAL segment is now completed");
+ # Update the list of partial wals with the current one.
+ @partial_wals = @lz4_partial_wals;
+
+ # Check the integrity of the completed segment, if lz4 is an available
+ # command.
+ my $lz4 = $ENV{LZ4};
+ skip "program lz4 is not found in your system", 1
+ if ( !defined $lz4
+ || $lz4 eq ''
+ || system_log($lz4, '--version') != 0);
+
+ my $lz4_is_valid = system_log($lz4, '-t', @lz4_wals);
+ is($lz4_is_valid, 0,
+ "lz4 verified the integrity of compressed WAL segments");
+}
+
# Verify that the start streaming position is computed and that the value is
-# correct regardless of whether ZLIB is available.
+# correct regardless of whether any compression is available.
$primary->psql('postgres', 'SELECT pg_switch_wal();');
$nextlsn =
$primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
chomp($nextlsn);
$primary->psql('postgres',
- 'INSERT INTO test_table VALUES (generate_series(200,300));');
+ 'INSERT INTO test_table VALUES (generate_series(301,400));');
$primary->command_ok(
[
'pg_receivewal', '-D', $stream_dir, '--verbose',
@@ -143,7 +207,7 @@ $primary->command_ok(
],
"streaming some WAL");
-$partial_wals[0] =~ s/(\.gz)?.partial//;
+$partial_wals[0] =~ s/(\.gz|\.lz4)?.partial//;
ok(-e $partial_wals[0], "check that previously partial WAL is now complete");
# Permissions on WAL files should be default
diff --git a/src/bin/pg_basebackup/walmethods.c b/src/bin/pg_basebackup/walmethods.c
index 8695647db4..684120ee8d 100644
--- a/src/bin/pg_basebackup/walmethods.c
+++ b/src/bin/pg_basebackup/walmethods.c
@@ -17,6 +17,10 @@
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
+
+#ifdef HAVE_LIBLZ4
+#include <lz4frame.h>
+#endif
#ifdef HAVE_LIBZ
#include <zlib.h>
#endif
@@ -30,6 +34,9 @@
/* Size of zlib buffer for .tar.gz */
#define ZLIB_OUT_SIZE 4096
+/* Size of lz4 input chunk for .lz4 */
+#define LZ4_IN_SIZE 4096
+
/*-------------------------------------------------------------------------
* WalDirectoryMethod - write wal to a directory looking like pg_wal
*-------------------------------------------------------------------------
@@ -40,9 +47,10 @@
*/
typedef struct DirectoryMethodData
{
- char *basedir;
- int compression;
- bool sync;
+ char *basedir;
+ WalCompressionMethod compression_method;
+ int compression;
+ bool sync;
} DirectoryMethodData;
static DirectoryMethodData *dir_data = NULL;
@@ -59,6 +67,11 @@ typedef struct DirectoryMethodFile
#ifdef HAVE_LIBZ
gzFile gzfp;
#endif
+#ifdef HAVE_LIBLZ4
+ LZ4F_compressionContext_t ctx;
+ size_t lz4bufsize;
+ void *lz4buf;
+#endif
} DirectoryMethodFile;
static const char *
@@ -74,7 +87,9 @@ dir_get_file_name(const char *pathname, const char *temp_suffix)
char *filename = pg_malloc0(MAXPGPATH * sizeof(char));
snprintf(filename, MAXPGPATH, "%s%s%s",
- pathname, dir_data->compression > 0 ? ".gz" : "",
+ pathname,
+ dir_data->compression_method == COMPRESSION_ZLIB ? ".gz" :
+ dir_data->compression_method == COMPRESSION_LZ4 ? ".lz4": "",
temp_suffix ? temp_suffix : "");
return filename;
@@ -90,6 +105,11 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
#ifdef HAVE_LIBZ
gzFile gzfp = NULL;
#endif
+#ifdef HAVE_LIBLZ4
+ LZ4F_compressionContext_t ctx = NULL;
+ size_t lz4bufsize = 0;
+ void *lz4buf = NULL;
+#endif
filename = dir_get_file_name(pathname, temp_suffix);
snprintf(tmppath, sizeof(tmppath), "%s/%s",
@@ -107,7 +127,7 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
return NULL;
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
{
gzfp = gzdopen(fd, "wb");
if (gzfp == NULL)
@@ -124,9 +144,55 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
}
}
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ LZ4F_preferences_t lz4preferences = { 0 };
+ size_t ctx_out;
+ size_t header_size;
+
+ /*
+ * Set all the preferences to default but do note contentSize. It will
+ * be needed in FindStreamingStart.
+ */
+ memset(&lz4preferences, 0, sizeof(LZ4F_frameInfo_t));
+ lz4preferences.frameInfo.contentSize = (unsigned long long)WalSegSz;
+ ctx_out = LZ4F_createCompressionContext(&ctx, LZ4F_VERSION);
+ lz4bufsize = LZ4F_compressBound(LZ4_IN_SIZE, &lz4preferences);
+ if (LZ4F_isError(ctx_out))
+ {
+ close(fd);
+ return NULL;
+ }
+
+ lz4buf = pg_malloc0(lz4bufsize);
+
+ /* add the header */
+ header_size = LZ4F_compressBegin(ctx, lz4buf, lz4bufsize, &lz4preferences);
+ if (LZ4F_isError(header_size))
+ {
+ close(fd);
+ return NULL;
+ }
+
+ errno = 0;
+ if (write(fd, lz4buf, header_size) != header_size)
+ {
+ int save_errno = errno;
+
+ close(fd);
+
+ /*
+ * If write didn't set errno, assume problem is no disk space.
+ */
+ errno = save_errno ? save_errno : ENOSPC;
+ return NULL;
+ }
+ }
+#endif
/* Do pre-padding on non-compressed files */
- if (pad_to_size && dir_data->compression == 0)
+ if (pad_to_size && dir_data->compression_method == COMPRESSION_NONE)
{
PGAlignedXLogBlock zerobuf;
int bytes;
@@ -171,7 +237,7 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
fsync_parent_path(tmppath) != 0)
{
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
gzclose(gzfp);
else
#endif
@@ -182,9 +248,18 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
f = pg_malloc0(sizeof(DirectoryMethodFile));
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
f->gzfp = gzfp;
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ f->ctx = ctx;
+ f->lz4buf = lz4buf;
+ f->lz4bufsize = lz4bufsize;
+ }
+#endif
+
f->fd = fd;
f->currpos = 0;
f->pathname = pg_strdup(pathname);
@@ -204,9 +279,46 @@ dir_write(Walfile f, const void *buf, size_t count)
Assert(f != NULL);
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
r = (ssize_t) gzwrite(df->gzfp, buf, count);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ size_t chunk;
+ size_t remaining;
+ const void *inbuf = buf;
+
+ remaining = count;
+ while (remaining > 0)
+ {
+ size_t compressed;
+
+ if (remaining > LZ4_IN_SIZE)
+ chunk = LZ4_IN_SIZE;
+ else
+ chunk = remaining;
+
+ remaining -= chunk;
+ compressed = LZ4F_compressUpdate(df->ctx,
+ df->lz4buf, df->lz4bufsize,
+ inbuf, chunk,
+ NULL);
+
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+
+ inbuf = ((char *)inbuf) + chunk;
+ }
+
+ /* Our caller keeps track of the uncompressed size. */
+ r = (ssize_t)count;
+ }
+ else
#endif
r = write(df->fd, buf, count);
if (r > 0)
@@ -234,9 +346,26 @@ dir_close(Walfile f, WalCloseMethod method)
Assert(f != NULL);
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
r = gzclose(df->gzfp);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ /* Flush any internal buffers */
+ size_t compressed = LZ4F_compressEnd(df->ctx,
+ df->lz4buf, df->lz4bufsize,
+ NULL);
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+
+ r = close(df->fd);
+ }
+ else
#endif
r = close(df->fd);
@@ -291,6 +420,12 @@ dir_close(Walfile f, WalCloseMethod method)
}
}
+#ifdef HAVE_LIBLZ4
+ pg_free(df->lz4buf);
+ /* supports free on NULL */
+ LZ4F_freeCompressionContext(df->ctx);
+#endif
+
pg_free(df->pathname);
pg_free(df->fullpath);
if (df->temp_suffix)
@@ -373,7 +508,9 @@ dir_finish(void)
WalWriteMethod *
-CreateWalDirectoryMethod(const char *basedir, int compression, bool sync)
+CreateWalDirectoryMethod(const char *basedir,
+ WalCompressionMethod compression_method,
+ int compression, bool sync)
{
WalWriteMethod *method;
@@ -391,6 +528,7 @@ CreateWalDirectoryMethod(const char *basedir, int compression, bool sync)
method->getlasterror = dir_getlasterror;
dir_data = pg_malloc0(sizeof(DirectoryMethodData));
+ dir_data->compression_method = compression_method;
dir_data->compression = compression;
dir_data->basedir = pg_strdup(basedir);
dir_data->sync = sync;
@@ -1031,8 +1169,16 @@ tar_finish(void)
return true;
}
+/*
+ * The argument compression_method is currently ignored. It is in place for
+ * symmetry with CreateWalDirectoryMethod which uses it for distinguishing
+ * between the different compression methods. CreateWalTarMethod and its family
+ * of functions handle only zlib compression.
+ */
WalWriteMethod *
-CreateWalTarMethod(const char *tarbase, int compression, bool sync)
+CreateWalTarMethod(const char *tarbase,
+ WalCompressionMethod compression_method,
+ int compression, bool sync)
{
WalWriteMethod *method;
const char *suffix = (compression != 0) ? ".tar.gz" : ".tar";
diff --git a/src/bin/pg_basebackup/walmethods.h b/src/bin/pg_basebackup/walmethods.h
index 4abdfd8333..872b677da5 100644
--- a/src/bin/pg_basebackup/walmethods.h
+++ b/src/bin/pg_basebackup/walmethods.h
@@ -19,6 +19,13 @@ typedef enum
CLOSE_NO_RENAME
} WalCloseMethod;
+typedef enum
+{
+ COMPRESSION_LZ4,
+ COMPRESSION_ZLIB,
+ COMPRESSION_NONE
+} WalCompressionMethod;
+
/*
* A WalWriteMethod structure represents the different methods used
* to write the streaming WAL as it's received.
@@ -95,8 +102,11 @@ struct WalWriteMethod
* not all those required for pg_receivewal)
*/
WalWriteMethod *CreateWalDirectoryMethod(const char *basedir,
+ WalCompressionMethod compression_method,
int compression, bool sync);
-WalWriteMethod *CreateWalTarMethod(const char *tarbase, int compression, bool sync);
+WalWriteMethod *CreateWalTarMethod(const char *tarbase,
+ WalCompressionMethod compression_method,
+ int compression, bool sync);
/* Cleanup routines for previously-created methods */
void FreeWalDirectoryMethod(void);
--
2.25.1
Re: Teach pg_receivewal to use lz4 compression
On Fri, Sep 10, 2021 at 08:21:51AM +0000, gkokolatos@pm.me wrote:
Agreed. A default value of 5, which is in the middle point of options, has been
defined and used.In addition, the tests have been adjusted to mimic the newly added gzip tests.
Looking at lz4frame.h, there is LZ4F_flush() that allows to compress
immediately any data buffered in the frame context but not compressed
yet. It seems to me that dir_sync() should be extended to support
LZ4.
export GZIP_PROGRAM=$(GZIP)
+export LZ4
[...]
+PGAC_PATH_PROGS(LZ4, lz4)
+
PGAC_PATH_BISON
The part of the test assigning LZ4 is fine, but I'd rather switch to a
logic à-la-gzip, where we just save "lz4" in Makefile.global.in,
saving cycles in ./configure.
+static bool
+is_xlogfilename(const char *filename, bool *ispartial,
+ WalCompressionMethod *wal_compression_method)
I like the set of simplifications you have done here to detection if a
segment is partial and which compression method applies to it.+ if (compression_method != COMPRESSION_ZLIB && compresslevel != 0)
+ {
+ pg_log_error("can only use --compress together with "
+ "--compression-method=gzip");
+#ifndef HAVE_LIBLZ4
+ pg_log_error("this build does not support compression via gzip");
+#endifs/HAVE_LIBLZ4/HAVE_LIBZ/.
+$primary->command_fails(
+ [
+ 'pg_receivewal', '-D', $stream_dir, '--compression-method', 'lz4',
+ '--compress', '1'
+ ],
+ 'failure if --compression-method=lz4 specified with --compress');This would fail when the code is not built with LZ4 with a non-zero
error code but with an error that is not what we expect. I think that
you should use $primary->command_fails_like() instead. That's quite
new, as of de1d4fe. The matching error pattern will need to change
depending on if we build the code with LZ4 or not. A simpler method
is to use --compression-method=none, to bypass the first round of
errors and make that build-independent, but that feels incomplete if
you want to tie that to LZ4.
+ pg_log_warning("compressed segment file \"%s\" has incorrect header size %lu, skipping",
+ dirent->d_name, consumed_len);
+ LZ4F_freeDecompressionContext(ctx);
I agree that skipping all those cases when calculating the streaming
start point is more consistent.+ if (r < 0)
+ pg_log_error("could not read compressed file \"%s\": %m",
+ fullpath);
+ else
+ pg_log_error("could not read compressed file \"%s\": read %d of %lu",
+ fullpath, r, sizeof(buf));
Let's same in translation effort here by just using "could not read",
etc. by removing the term "compressed".+ pg_log_error("can only use --compress together with "
+ "--compression-method=gzip");
Better to keep these in a single line to ease grepping. We don't care
if error strings are longer than the 72-80 character limit.+/* Size of lz4 input chunk for .lz4 */
+#define LZ4_IN_SIZE 4096
Why this choice? Does it need to use LZ4_COMPRESSBOUND?- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
gzclose(gzfp);
else
Hm. The addition of the header in dir_open_for_write() uses
LZ4F_compressBegin. Shouldn't we use LZ4F_compressEnd() if
fsync_fname() or fsync_parent_path() fail on top of closing the fd?
That would be more consistent IMO to do so. The patch does that in
dir_close(). You should do that additionally if there is a failure
when writing the header.+ pg_log_error("invalid compression-method \"%s\"", optarg);
+ exit(1);
This could be "invalid value \"%s\" for option %s", see
option_parse_int() in fe_utils/option_utils.c.After running the TAP tests, the LZ4 section is failing as follows:
pg_receivewal: stopped log streaming at 0/4001950 (timeline 1)
pg_receivewal: not renaming "000000010000000000000004.partial", segment is not complete
pg_receivewal: error: could not close file "000000010000000000000004": Undefined error: 0
ok 26 - streaming some WAL using --compression-method=lz4
The third log line I am quoting here looks unexpected to me. Saying
that, the tests integrate nicely with the existing code.
As mentioned upthread, LZ4-compressed files don't store the file size
by default. I think that we should document that better in the code
and the documentation, in two ways at least:
- Add some comments mentioning lz4 --content-size, with at least one
in FindStreamingStart().
- Add a new paragraph in the documentation of --compression-method.
The name of the compression method is "LZ4" with upper-case
characters. Some comments in the code and the tests, as well as the
docs, are not careful about that.
--
Michael
Re: Teach pg_receivewal to use lz4 compression
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, September 15th, 2021 at 08:46, Michael Paquier michael@paquier.xyz wrote:
Hi,
thank you for the review.
On Fri, Sep 10, 2021 at 08:21:51AM +0000, gkokolatos@pm.me wrote:
Agreed. A default value of 5, which is in the middle point of options, has been
defined and used.
In addition, the tests have been adjusted to mimic the newly added gzip tests.Looking at lz4frame.h, there is LZ4F_flush() that allows to compress
immediately any data buffered in the frame context but not compressed
yet. It seems to me that dir_sync() should be extended to support
LZ4.
Agreed. LZ4F_flush() calls have been added where appropriate.
export GZIP_PROGRAM=$(GZIP) +export LZ4 [...] +PGAC_PATH_PROGS(LZ4, lz4) - PGAC_PATH_BISONThe part of the test assigning LZ4 is fine, but I'd rather switch to a
logic à-la-gzip, where we just save "lz4" in Makefile.global.in,
saving cycles in ./configure.
Reluctantly agreed.
+static bool +is_xlogfilename(const char *filename, bool *ispartial, - WalCompressionMethod *wal_compression_method)I like the set of simplifications you have done here to detection if a
segment is partial and which compression method applies to it.
Thank you very much.
+ if (compression_method != COMPRESSION_ZLIB && compresslevel != 0) + { + pg_log_error("can only use --compress together with " + "--compression-method=gzip"); +#ifndef HAVE_LIBLZ4 + pg_log_error("this build does not support compression via gzip"); +#endifs/HAVE_LIBLZ4/HAVE_LIBZ/.
Fixed.
+$primary->command_fails( + [ + 'pg_receivewal', '-D', $stream_dir, '--compression-method', 'lz4', + '--compress', '1' + ], + 'failure if --compression-method=lz4 specified with --compress'); This would fail when the code is not built with LZ4 with a non-zero error code but with an error that is not what we expect. I think that you should use $primary->command_fails_like() instead. That's quite new, as of de1d4fe. The matching error pattern will need to change depending on if we build the code with LZ4 or not. A simpler method is to use --compression-method=none, to bypass the first round of errors and make that build-independent, but that feels incomplete if you want to tie that to LZ4.
Fixed. Now a regex has been added to address both builds.
+ pg_log_warning("compressed segment file \\\\"%s\\\\" has incorrect header size %lu, skipping", + dirent->d_name, consumed_len); + LZ4F_freeDecompressionContext(ctx);I agree that skipping all those cases when calculating the streaming
start point is more consistent.
Thanks.
+ if (r < 0) + pg_log_error("could not read compressed file \\\\"%s\\\\": %m", + fullpath); + else + pg_log_error("could not read compressed file \\\\"%s\\\\": read %d of %lu", + fullpath, r, sizeof(buf));Let's same in translation effort here by just using "could not read",
etc. by removing the term "compressed".
The string is also present in the gzip compressed case, i.e. prior to this patch.
The translation effort will not be reduced by changing this string only.
+ pg_log_error("can only use --compress together with " + "--compression-method=gzip");Better to keep these in a single line to ease grepping. We don't care
if error strings are longer than the 72-80 character limit.
Fixed.
+/* Size of lz4 input chunk for .lz4 */
+#define LZ4_IN_SIZE 4096Why this choice? Does it need to use LZ4_COMPRESSBOUND?
This value is used in order to calculate the bound, before any buffer is
received. Then when we receive buffer, we consume them in LZ4_IN_SIZE chunks.
Note the call to LZ4F_compressBound() in dir_open_for_write().
+ ctx_out = LZ4F_createCompressionContext(&ctx, LZ4F_VERSION);
+ lz4bufsize = LZ4F_compressBound(LZ4_IN_SIZE, &lz4preferences);- if (dir_data->compression > 0) + if (dir_data->compression_method == COMPRESSION_ZLIB) gzclose(gzfp); elseHm. The addition of the header in dir_open_for_write() uses
LZ4F_compressBegin. Shouldn't we use LZ4F_compressEnd() if
fsync_fname() or fsync_parent_path() fail on top of closing the fd?
That would be more consistent IMO to do so. The patch does that in
dir_close(). You should do that additionally if there is a failure
when writing the header.
Fixed. LZ4_flush() have been added where appropriate.
+ pg_log_error("invalid compression-method \"%s\", optarg); + exit(1);This could be "invalid value \"%s\" for option %s", see
option_parse_int() in fe_utils/option_utils.c.
Fixed.
After running the TAP tests, the LZ4 section is failing as follows:
pg_receivewal: stopped log streaming at 0/4001950 (timeline 1)
pg_receivewal: not renaming "000000010000000000000004.partial", segment is not complete
pg_receivewal: error: could not close file "000000010000000000000004": Undefined error: 0
ok 26 - streaming some WAL using --compression-method=lz4
The third log line I am quoting here looks unexpected to me. Saying
that, the tests integrate nicely with the existing code.
Strange that you got an undefined error. I managed to _almost_ reproduce
with the log line looking like:
pg_receivewal: error: could not close file "000000010000000000000004": Success
This was due to a call to LZ4F_compressEnd() on a partial file. In v5 of
the patch, LZ4F_compressEnd() is called when the WalCloseMethod is CLOSE_NORMAL
otherwise LZ4F_flush is used. This seems to remove the log line and a
more consistent behaviour overall.
In passing, close_walfile() has been taught to consider compression in
the filename, via get_file_name().
As mentioned upthread, LZ4-compressed files don't store the file size
by default. I think that we should document that better in the code
and the documentation, in two ways at least:- Add some comments mentioning lz4 --content-size, with at least one
in FindStreamingStart().
- Add a new paragraph in the documentation of --compression-method.
Apologies, I didn't understood what you meant upstream. Now I do.
How about:
By default, LZ4-compressed files don't store the uncompressed file size.
However, the program pg_receivewal, does store that information. As a
consequence, the file does not need to be decompressed if the external
program is used, e.g. lz4 -t --content-size <file>, will report the
uncompressed file size.
The name of the compression method is "LZ4" with upper-case
characters. Some comments in the code and the tests, as well as the
docs, are not careful about that.
Hopefully fixed.
Cheers,
//Georgios
Show quoted text
--
Michael
Attachments:
v5-0001-Teach-pg_receivewal-to-use-LZ4-compression.patchapplication/octet-stream; name=v5-0001-Teach-pg_receivewal-to-use-LZ4-compression.patchDownload
From 2aa50aabbab687afaf1960cafb31b9a76490d0f6 Mon Sep 17 00:00:00 2001
From: Georgios Kokolatos <gkokolatos@pm.me>
Date: Thu, 16 Sep 2021 08:10:51 +0000
Subject: [PATCH v5] Teach pg_receivewal to use LZ4 compression
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
The program pg_receivewal can use gzip compression to store the received WAL.
This commit teaches it to also be able to use LZ4 compression. It is required
that the binary is build using the -llz4 flag. It is enabled via the --with-lz4
flag on configuration time.
Previously, the user had to use the option --compress with a value between [0-9]
to denote that gzip compression was required. This specific behaviour has not
maintained. A newly introduced option --compression-method=[LZ4|gzip] can be
used to ask for the logs to be compressed. Compression values can be selected
only when the compression method is gzip. A compression value of 0 now returns
an error.
Under the hood there is nothing exceptional to be noted. Tar based archives have
not yet been taught to use LZ4 compression. If that is felt useful, then it is
easy to be added in the future.
Tests have been added to verify the creation and correctness of the generated
LZ4 files. The later is achieved by the use of LZ4 program, if present in the
installation.
---
doc/src/sgml/ref/pg_receivewal.sgml | 28 +-
src/Makefile.global.in | 1 +
src/bin/pg_basebackup/Makefile | 1 +
src/bin/pg_basebackup/pg_basebackup.c | 7 +-
src/bin/pg_basebackup/pg_receivewal.c | 269 +++++++++++++++----
src/bin/pg_basebackup/receivelog.c | 19 +-
src/bin/pg_basebackup/t/020_pg_receivewal.pl | 75 +++++-
src/bin/pg_basebackup/walmethods.c | 209 +++++++++++++-
src/bin/pg_basebackup/walmethods.h | 12 +-
9 files changed, 543 insertions(+), 78 deletions(-)
diff --git a/doc/src/sgml/ref/pg_receivewal.sgml b/doc/src/sgml/ref/pg_receivewal.sgml
index 45b544cf49..aa7dae7c76 100644
--- a/doc/src/sgml/ref/pg_receivewal.sgml
+++ b/doc/src/sgml/ref/pg_receivewal.sgml
@@ -229,15 +229,35 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--compression-method=<replaceable class="parameter">level</replaceable></option></term>
+ <listitem>
+ <para>
+ Enables compression of write-ahead logs using the specified method.
+ Supported methods are <literal>LZ4</literal> and
+ <literal>gzip</literal>.
+ The suffix <filename>.lz4</filename> or <filename>.gz</filename> will
+ automatically be added to all filenames for each method respectevilly.
+ For the <literal>LZ4</literal> method to be available,
+ <productname>PostgreSQL</productname> must have been have been compiled
+ with <option>--with-lz4</option>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-Z <replaceable class="parameter">level</replaceable></option></term>
<term><option>--compress=<replaceable class="parameter">level</replaceable></option></term>
<listitem>
<para>
- Enables gzip compression of write-ahead logs, and specifies the
- compression level (0 through 9, 0 being no compression and 9 being best
- compression). The suffix <filename>.gz</filename> will
- automatically be added to all filenames.
+ Specifies the compression level (1 through 9, 1 being least compression
+ and 9 being most compression) for gzip compressed write-ahead logs. The
+ default value is 5.
+ </para>
+
+ <para>
+ It requires for <option>--compression-method</option> to be specified
+ as <literal>gzip</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index e4fd7b5290..555d66903d 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -350,6 +350,7 @@ XGETTEXT = @XGETTEXT@
GZIP = gzip
BZIP2 = bzip2
+LZ4 = lz4
DOWNLOAD = wget -O $@ --no-use-server-timestamps
#DOWNLOAD = curl -o $@
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index 459d514183..387d728345 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -24,6 +24,7 @@ export TAR
# used by the command "gzip" to pass down options, so stick with a different
# name.
export GZIP_PROGRAM=$(GZIP)
+export LZ4
override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport)
diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c
index 669aa207a3..1e204f6862 100644
--- a/src/bin/pg_basebackup/pg_basebackup.c
+++ b/src/bin/pg_basebackup/pg_basebackup.c
@@ -555,10 +555,13 @@ LogStreamerMain(logstreamer_param *param)
stream.replication_slot = replication_slot;
if (format == 'p')
- stream.walmethod = CreateWalDirectoryMethod(param->xlog, 0,
+ stream.walmethod = CreateWalDirectoryMethod(param->xlog,
+ COMPRESSION_NONE, 0,
stream.do_sync);
else
- stream.walmethod = CreateWalTarMethod(param->xlog, compresslevel,
+ stream.walmethod = CreateWalTarMethod(param->xlog,
+ COMPRESSION_NONE /* argument is ignored */,
+ compresslevel,
stream.do_sync);
if (!ReceiveXlogStream(param->bgconn, &stream))
diff --git a/src/bin/pg_basebackup/pg_receivewal.c b/src/bin/pg_basebackup/pg_receivewal.c
index 9d1843728d..eaf1489c6f 100644
--- a/src/bin/pg_basebackup/pg_receivewal.c
+++ b/src/bin/pg_basebackup/pg_receivewal.c
@@ -29,9 +29,16 @@
#include "receivelog.h"
#include "streamutil.h"
+#ifdef HAVE_LIBLZ4
+#include "lz4frame.h"
+#endif
+
/* Time to sleep between reconnection attempts */
#define RECONNECT_SLEEP_TIME 5
+/* Default compression level for gzip compression method */
+#define DEFAULT_ZLIB_COMPRESSLEVEL 5
+
/* Global options */
static char *basedir = NULL;
static int verbose = 0;
@@ -45,6 +52,7 @@ static bool do_drop_slot = false;
static bool do_sync = true;
static bool synchronous = false;
static char *replication_slot = NULL;
+static WalCompressionMethod compression_method = COMPRESSION_NONE;
static XLogRecPtr endpos = InvalidXLogRecPtr;
@@ -63,16 +71,6 @@ disconnect_atexit(void)
PQfinish(conn);
}
-/* Routines to evaluate segment file format */
-#define IsCompressXLogFileName(fname) \
- (strlen(fname) == XLOG_FNAME_LEN + strlen(".gz") && \
- strspn(fname, "0123456789ABCDEF") == XLOG_FNAME_LEN && \
- strcmp((fname) + XLOG_FNAME_LEN, ".gz") == 0)
-#define IsPartialCompressXLogFileName(fname) \
- (strlen(fname) == XLOG_FNAME_LEN + strlen(".gz.partial") && \
- strspn(fname, "0123456789ABCDEF") == XLOG_FNAME_LEN && \
- strcmp((fname) + XLOG_FNAME_LEN, ".gz.partial") == 0)
-
static void
usage(void)
{
@@ -92,7 +90,10 @@ usage(void)
printf(_(" --synchronous flush write-ahead log immediately after writing\n"));
printf(_(" -v, --verbose output verbose messages\n"));
printf(_(" -V, --version output version information, then exit\n"));
- printf(_(" -Z, --compress=0-9 compress logs with given compression level\n"));
+ printf(_(" --compression-method=METHOD\n"
+ " use this method for compression\n"));
+ printf(_(" -Z, --compress=1-9 compress logs with given compression level (default: %d)\n"
+ " available only with --compression-method=gzip\n"), DEFAULT_ZLIB_COMPRESSLEVEL);
printf(_(" -?, --help show this help, then exit\n"));
printf(_("\nConnection options:\n"));
printf(_(" -d, --dbname=CONNSTR connection string\n"));
@@ -108,6 +109,79 @@ usage(void)
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
}
+
+/*
+ * Check if the filename looks like an xlog file. Also note if it is partial
+ * and/or compressed file.
+ */
+static bool
+is_xlogfilename(const char *filename, bool *ispartial,
+ WalCompressionMethod *wal_compression_method)
+{
+ size_t fname_len = strlen(filename);
+ size_t xlog_pattern_len = strspn(filename, "0123456789ABCDEF");
+
+ /* File does not look like a XLOG file */
+ if (xlog_pattern_len != XLOG_FNAME_LEN)
+ return false;
+
+ /* File looks like a complete uncompressed XLOG file */
+ if (fname_len == XLOG_FNAME_LEN)
+ {
+ *ispartial = false;
+ *wal_compression_method = COMPRESSION_NONE;
+ return true;
+ }
+
+ /* File looks like a complete zlib compressed XLOG file */
+ if ((fname_len == XLOG_FNAME_LEN + strlen(".gz")) &&
+ strcmp(filename + XLOG_FNAME_LEN, ".gz") == 0)
+ {
+ *ispartial = false;
+ *wal_compression_method = COMPRESSION_ZLIB;
+ return true;
+ }
+
+ /* File looks like a complete LZ4 compressed XLOG file */
+ if ((fname_len == XLOG_FNAME_LEN + strlen(".lz4")) &&
+ strcmp(filename + XLOG_FNAME_LEN, ".lz4") == 0)
+ {
+ *ispartial = false;
+ *wal_compression_method = COMPRESSION_LZ4;
+ return true;
+ }
+
+ /* File looks like a partial uncompressed XLOG file */
+ if ((fname_len == XLOG_FNAME_LEN + strlen(".partial")) &&
+ strcmp(filename + XLOG_FNAME_LEN, ".partial") == 0)
+ {
+ *ispartial = true;
+ *wal_compression_method = COMPRESSION_NONE;
+ return true;
+ }
+
+ /* File looks like a partial zlib compressed XLOG file */
+ if ((fname_len == XLOG_FNAME_LEN + strlen(".gz.partial")) &&
+ strcmp(filename + XLOG_FNAME_LEN, ".gz.partial") == 0)
+ {
+ *ispartial = true;
+ *wal_compression_method = COMPRESSION_ZLIB;
+ return true;
+ }
+
+ /* File looks like a partial LZ4 compressed XLOG file */
+ if ((fname_len == XLOG_FNAME_LEN + strlen(".lz4.partial")) &&
+ strcmp(filename + XLOG_FNAME_LEN, ".lz4.partial") == 0)
+ {
+ *ispartial = true;
+ *wal_compression_method = COMPRESSION_LZ4;
+ return true;
+ }
+
+ /* File does not look like something we recognise */
+ return false;
+}
+
static bool
stop_streaming(XLogRecPtr xlogpos, uint32 timeline, bool segment_finished)
{
@@ -213,33 +287,11 @@ FindStreamingStart(uint32 *tli)
{
uint32 tli;
XLogSegNo segno;
+ WalCompressionMethod wal_compression_method;
bool ispartial;
- bool iscompress;
- /*
- * Check if the filename looks like an xlog file, or a .partial file.
- */
- if (IsXLogFileName(dirent->d_name))
- {
- ispartial = false;
- iscompress = false;
- }
- else if (IsPartialXLogFileName(dirent->d_name))
- {
- ispartial = true;
- iscompress = false;
- }
- else if (IsCompressXLogFileName(dirent->d_name))
- {
- ispartial = false;
- iscompress = true;
- }
- else if (IsPartialCompressXLogFileName(dirent->d_name))
- {
- ispartial = true;
- iscompress = true;
- }
- else
+ if (!is_xlogfilename(dirent->d_name,
+ &ispartial, &wal_compression_method))
continue;
/*
@@ -250,14 +302,18 @@ FindStreamingStart(uint32 *tli)
/*
* Check that the segment has the right size, if it's supposed to be
* completed. For non-compressed segments just check the on-disk size
- * and see if it matches a completed segment. For compressed segments,
- * look at the last 4 bytes of the compressed file, which is where the
- * uncompressed size is located for gz files with a size lower than
- * 4GB, and then compare it to the size of a completed segment. The 4
- * last bytes correspond to the ISIZE member according to
+ * and see if it matches a completed segment. For zlib compressed
+ * segments, look at the last 4 bytes of the compressed file, which is
+ * where the uncompressed size is located for gz files with a size lower
+ * than 4GB, and then compare it to the size of a completed segment.
+ * The 4 last bytes correspond to the ISIZE member according to
* http://www.zlib.org/rfc-gzip.html.
+ *
+ * For LZ4 compressed segments read the header using the exposed API and
+ * compare the uncompressed file size, stored in
+ * LZ4F_frameInfo_t{.contentSize}, to that of a completed segment.
*/
- if (!ispartial && !iscompress)
+ if (!ispartial && wal_compression_method == COMPRESSION_NONE)
{
struct stat statbuf;
char fullpath[MAXPGPATH * 2];
@@ -276,7 +332,7 @@ FindStreamingStart(uint32 *tli)
continue;
}
}
- else if (!ispartial && iscompress)
+ else if (!ispartial && wal_compression_method == COMPRESSION_ZLIB)
{
int fd;
char buf[4];
@@ -322,6 +378,72 @@ FindStreamingStart(uint32 *tli)
continue;
}
}
+ else if (!ispartial && compression_method == COMPRESSION_LZ4)
+ {
+#ifdef HAVE_LIBLZ4
+ int fd;
+ int r;
+ size_t consumed_len = LZ4F_HEADER_SIZE_MAX;
+ char buf[LZ4F_HEADER_SIZE_MAX];
+ char fullpath[MAXPGPATH * 2];
+ LZ4F_frameInfo_t frame_info = { 0 };
+ LZ4F_decompressionContext_t ctx = NULL;
+
+ snprintf(fullpath, sizeof(fullpath), "%s/%s", basedir, dirent->d_name);
+
+ fd = open(fullpath, O_RDONLY | PG_BINARY, 0);
+ if (fd < 0)
+ {
+ pg_log_error("could not open compressed file \"%s\": %m",
+ fullpath);
+ exit(1);
+ }
+
+ r = read(fd, buf, sizeof(buf));
+ if (r != sizeof(buf))
+ {
+ if (r < 0)
+ pg_log_error("could not read compressed file \"%s\": %m",
+ fullpath);
+ else
+ pg_log_error("could not read compressed file \"%s\": read %d of %lu",
+ fullpath, r, sizeof(buf));
+ exit(1);
+ }
+ close(fd);
+
+ if (LZ4F_isError(LZ4F_createDecompressionContext(&ctx, LZ4F_VERSION)))
+ {
+ pg_log_error("LZ4 internal error");
+ exit(1);
+ }
+
+ LZ4F_getFrameInfo(ctx, &frame_info, (void *)buf, &consumed_len);
+ if (consumed_len <= LZ4F_HEADER_SIZE_MIN ||
+ consumed_len >= LZ4F_HEADER_SIZE_MAX)
+ {
+ pg_log_warning("compressed segment file \"%s\" has incorrect header size %lu, skipping",
+ dirent->d_name, consumed_len);
+ LZ4F_freeDecompressionContext(ctx);
+ continue;
+ }
+
+ if (frame_info.contentSize != WalSegSz)
+ {
+ pg_log_warning("compressed segment file \"%s\" has incorrect uncompressed size %lld, skipping",
+ dirent->d_name, frame_info.contentSize);
+ LZ4F_freeDecompressionContext(ctx);
+ continue;
+ }
+
+ LZ4F_freeDecompressionContext(ctx);
+#else
+ pg_log_error("cannot verify LZ4 compressed segment file \"%s\", "
+ "this program was not build with LZ4 support",
+ dirent->d_name);
+ exit(1);
+#endif
+ }
/* Looks like a valid segment. Remember that we saw it. */
if ((segno > high_segno) ||
@@ -431,7 +553,9 @@ StreamLog(void)
stream.synchronous = synchronous;
stream.do_sync = do_sync;
stream.mark_done = false;
- stream.walmethod = CreateWalDirectoryMethod(basedir, compresslevel,
+ stream.walmethod = CreateWalDirectoryMethod(basedir,
+ compression_method,
+ compresslevel,
stream.do_sync);
stream.partial_suffix = ".partial";
stream.replication_slot = replication_slot;
@@ -482,6 +606,7 @@ main(int argc, char **argv)
{"status-interval", required_argument, NULL, 's'},
{"slot", required_argument, NULL, 'S'},
{"verbose", no_argument, NULL, 'v'},
+ {"compression-method", required_argument, NULL, 'I'},
{"compress", required_argument, NULL, 'Z'},
/* action */
{"create-slot", no_argument, NULL, 1},
@@ -567,8 +692,24 @@ main(int argc, char **argv)
case 'v':
verbose++;
break;
+ case 'I':
+ if (pg_strcasecmp(optarg, "gzip") == 0)
+ {
+ compression_method = COMPRESSION_ZLIB;
+ }
+ else if (pg_strcasecmp(optarg, "lz4") == 0)
+ {
+ compression_method = COMPRESSION_LZ4;
+ }
+ else
+ {
+ pg_log_error("invalid value \"%s\" for option %s",
+ optarg, "--compress-method");
+ exit(1);
+ }
+ break;
case 'Z':
- if (!option_parse_int(optarg, "-Z/--compress", 0, 9,
+ if (!option_parse_int(optarg, "-Z/--compress", 1, 9,
&compresslevel))
exit(1);
break;
@@ -648,13 +789,45 @@ main(int argc, char **argv)
exit(1);
}
+
+ /*
+ * Compression related arguments
+ */
+ if (compression_method != COMPRESSION_NONE)
+ {
#ifndef HAVE_LIBZ
- if (compresslevel != 0)
+ if (compression_method == COMPRESSION_ZLIB)
+ {
+ pg_log_error("this build does not support compression via gzip");
+ exit(1);
+ }
+#endif
+#ifndef HAVE_LIBLZ4
+ if (compression_method == COMPRESSION_LZ4)
+ {
+ pg_log_error("this build does not support compression via LZ4");
+ exit(1);
+ }
+#endif
+ }
+
+ if (compression_method != COMPRESSION_ZLIB && compresslevel != 0)
{
- pg_log_error("this build does not support compression");
+ pg_log_error("can only use --compress together with --compression-method=gzip");
+#ifndef HAVE_LIBZ
+ pg_log_error("this build does not support compression via gzip");
+#endif
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+ progname);
exit(1);
}
-#endif
+
+ if (compression_method == COMPRESSION_ZLIB && compresslevel == 0)
+ {
+ pg_log_info("no --compression specified, will be using %d",
+ DEFAULT_ZLIB_COMPRESSLEVEL);
+ compresslevel = DEFAULT_ZLIB_COMPRESSLEVEL;
+ }
/*
* Check existence of destination folder.
diff --git a/src/bin/pg_basebackup/receivelog.c b/src/bin/pg_basebackup/receivelog.c
index 9601fd8d9c..a5a0161d04 100644
--- a/src/bin/pg_basebackup/receivelog.c
+++ b/src/bin/pg_basebackup/receivelog.c
@@ -109,7 +109,7 @@ open_walfile(StreamCtl *stream, XLogRecPtr startpoint)
* When streaming to tar, no file with this name will exist before, so we
* never have to verify a size.
*/
- if (stream->walmethod->compression() == 0 &&
+ if (stream->walmethod->compression() == COMPRESSION_NONE &&
stream->walmethod->existsfile(fn))
{
size = stream->walmethod->get_file_size(fn);
@@ -185,6 +185,7 @@ open_walfile(StreamCtl *stream, XLogRecPtr startpoint)
static bool
close_walfile(StreamCtl *stream, XLogRecPtr pos)
{
+ char *fn;
off_t currpos;
int r;
@@ -192,13 +193,18 @@ close_walfile(StreamCtl *stream, XLogRecPtr pos)
return true;
currpos = stream->walmethod->get_current_pos(walfile);
+
+ /* Note that this considers the compression used if necessary */
+ fn = stream->walmethod->get_file_name(current_walfile_name,
+ stream->partial_suffix);
if (currpos == -1)
{
pg_log_error("could not determine seek position in file \"%s\": %s",
- current_walfile_name, stream->walmethod->getlasterror());
+ fn, stream->walmethod->getlasterror());
stream->walmethod->close(walfile, CLOSE_UNLINK);
walfile = NULL;
+ pg_free(fn);
return false;
}
@@ -208,8 +214,7 @@ close_walfile(StreamCtl *stream, XLogRecPtr pos)
r = stream->walmethod->close(walfile, CLOSE_NORMAL);
else
{
- pg_log_info("not renaming \"%s%s\", segment is not complete",
- current_walfile_name, stream->partial_suffix);
+ pg_log_info("not renaming \"%s\", segment is not complete", fn);
r = stream->walmethod->close(walfile, CLOSE_NO_RENAME);
}
}
@@ -221,10 +226,14 @@ close_walfile(StreamCtl *stream, XLogRecPtr pos)
if (r != 0)
{
pg_log_error("could not close file \"%s\": %s",
- current_walfile_name, stream->walmethod->getlasterror());
+ fn, stream->walmethod->getlasterror());
+
+ pg_free(fn);
return false;
}
+ pg_free(fn);
+
/*
* Mark file as archived if requested by the caller - pg_basebackup needs
* to do so as files can otherwise get archived again after promotion of a
diff --git a/src/bin/pg_basebackup/t/020_pg_receivewal.pl b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
index 0b33d73900..af9eada534 100644
--- a/src/bin/pg_basebackup/t/020_pg_receivewal.pl
+++ b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
@@ -5,7 +5,7 @@ use strict;
use warnings;
use TestLib;
use PostgresNode;
-use Test::More tests => 27;
+use Test::More tests => 34;
program_help_ok('pg_receivewal');
program_version_ok('pg_receivewal');
@@ -33,6 +33,14 @@ $primary->command_fails(
$primary->command_fails(
[ 'pg_receivewal', '-D', $stream_dir, '--synchronous', '--no-sync' ],
'failure if --synchronous specified with --no-sync');
+$primary->command_fails_like(
+ [
+ 'pg_receivewal', '-D', $stream_dir, '--compression-method', 'lz4',
+ '--compress', '1'
+ ],
+ qr/\Qpg_receivewal: error: \E(can only use --compress together with --compression-method=gzip|this build does not support compression via LZ4)/,
+ 'failure if --compression-method=lz4 specified with --compress');
+
# Slot creation and drop
my $slot_name = 'test';
@@ -91,7 +99,9 @@ SKIP:
$primary->command_ok(
[
'pg_receivewal', '-D', $stream_dir, '--verbose',
- '--endpos', $nextlsn, '--compress', '1 ',
+ '--endpos', $nextlsn,
+ '--compression-method', 'gzip',
+ '--compress', '1 ',
'--no-loop'
],
"streaming some WAL using ZLIB compression");
@@ -128,14 +138,69 @@ SKIP:
"gzip verified the integrity of compressed WAL segments");
}
+# Check LZ4 compression if available
+SKIP:
+{
+ skip "postgres was not built with LZ4 support", 5
+ if (!check_pg_config("#define HAVE_LIBLZ4 1"));
+
+ # Generate more WAL including one completed, compressed segment.
+ $primary->psql('postgres', 'SELECT pg_switch_wal();');
+ $nextlsn =
+ $primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
+ chomp($nextlsn);
+ $primary->psql('postgres',
+ 'INSERT INTO test_table VALUES (generate_series(201,300));');
+
+ # Stream up to the given position
+ $primary->command_ok(
+ [
+ 'pg_receivewal', '-D', $stream_dir, '--verbose',
+ '--endpos', $nextlsn, '--no-loop',
+ '--compression-method', 'lz4'
+ ],
+ 'streaming some WAL using --compression-method=lz4');
+
+ # Verify that the stored files are generated with their expected
+ # names.
+ my @lz4_wals = glob "$stream_dir/*.lz4";
+ is(scalar(@lz4_wals), 1,
+ "one WAL segment compressed with LZ4 was created");
+ my @lz4_partial_wals = glob "$stream_dir/*.lz4.partial";
+ is(scalar(@lz4_partial_wals),
+ 1, "one partial WAL segment compressed with LZ4 was created");
+
+ # Verify that the start streaming position is computed correctly by
+ # comparing it with the partial file generated previously. The name
+ # of the previous partial, now-completed WAL segment is updated, keeping
+ # its base number.
+ $partial_wals[0] =~ s/(\.gz)?\.partial$/.lz4/;
+ is($lz4_wals[0] eq $partial_wals[0],
+ 1, "one partial WAL segment is now completed");
+ # Update the list of partial wals with the current one.
+ @partial_wals = @lz4_partial_wals;
+
+ # Check the integrity of the completed segment, if LZ4 is an available
+ # command.
+ my $lz4 = $ENV{LZ4};
+ skip "program lz4 is not found in your system", 1
+ if ( !defined $lz4
+ || $lz4 eq ''
+ || system_log($lz4, '--version') != 0);
+
+ my $lz4_is_valid = system_log($lz4, '-t', @lz4_wals);
+ is($lz4_is_valid, 0,
+ "lz4 verified the integrity of compressed WAL segments");
+}
+
# Verify that the start streaming position is computed and that the value is
-# correct regardless of whether ZLIB is available.
+# correct regardless of whether any compression is available.
$primary->psql('postgres', 'SELECT pg_switch_wal();');
$nextlsn =
$primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
chomp($nextlsn);
$primary->psql('postgres',
- 'INSERT INTO test_table VALUES (generate_series(200,300));');
+ 'INSERT INTO test_table VALUES (generate_series(301,400));');
$primary->command_ok(
[
'pg_receivewal', '-D', $stream_dir, '--verbose',
@@ -143,7 +208,7 @@ $primary->command_ok(
],
"streaming some WAL");
-$partial_wals[0] =~ s/(\.gz)?.partial//;
+$partial_wals[0] =~ s/(\.gz|\.lz4)?.partial//;
ok(-e $partial_wals[0], "check that previously partial WAL is now complete");
# Permissions on WAL files should be default
diff --git a/src/bin/pg_basebackup/walmethods.c b/src/bin/pg_basebackup/walmethods.c
index 8695647db4..a4bcc1a9df 100644
--- a/src/bin/pg_basebackup/walmethods.c
+++ b/src/bin/pg_basebackup/walmethods.c
@@ -17,6 +17,10 @@
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
+
+#ifdef HAVE_LIBLZ4
+#include <lz4frame.h>
+#endif
#ifdef HAVE_LIBZ
#include <zlib.h>
#endif
@@ -30,6 +34,9 @@
/* Size of zlib buffer for .tar.gz */
#define ZLIB_OUT_SIZE 4096
+/* Size of lz4 input chunk for .lz4 */
+#define LZ4_IN_SIZE 4096
+
/*-------------------------------------------------------------------------
* WalDirectoryMethod - write wal to a directory looking like pg_wal
*-------------------------------------------------------------------------
@@ -40,9 +47,10 @@
*/
typedef struct DirectoryMethodData
{
- char *basedir;
- int compression;
- bool sync;
+ char *basedir;
+ WalCompressionMethod compression_method;
+ int compression;
+ bool sync;
} DirectoryMethodData;
static DirectoryMethodData *dir_data = NULL;
@@ -59,6 +67,11 @@ typedef struct DirectoryMethodFile
#ifdef HAVE_LIBZ
gzFile gzfp;
#endif
+#ifdef HAVE_LIBLZ4
+ LZ4F_compressionContext_t ctx;
+ size_t lz4bufsize;
+ void *lz4buf;
+#endif
} DirectoryMethodFile;
static const char *
@@ -74,7 +87,9 @@ dir_get_file_name(const char *pathname, const char *temp_suffix)
char *filename = pg_malloc0(MAXPGPATH * sizeof(char));
snprintf(filename, MAXPGPATH, "%s%s%s",
- pathname, dir_data->compression > 0 ? ".gz" : "",
+ pathname,
+ dir_data->compression_method == COMPRESSION_ZLIB ? ".gz" :
+ dir_data->compression_method == COMPRESSION_LZ4 ? ".lz4": "",
temp_suffix ? temp_suffix : "");
return filename;
@@ -90,6 +105,11 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
#ifdef HAVE_LIBZ
gzFile gzfp = NULL;
#endif
+#ifdef HAVE_LIBLZ4
+ LZ4F_compressionContext_t ctx = NULL;
+ size_t lz4bufsize = 0;
+ void *lz4buf = NULL;
+#endif
filename = dir_get_file_name(pathname, temp_suffix);
snprintf(tmppath, sizeof(tmppath), "%s/%s",
@@ -107,7 +127,7 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
return NULL;
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
{
gzfp = gzdopen(fd, "wb");
if (gzfp == NULL)
@@ -124,9 +144,59 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
}
}
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ LZ4F_preferences_t lz4preferences = { 0 };
+ size_t ctx_out;
+ size_t header_size;
+
+ /*
+ * Set all the preferences to default but do note contentSize. It will
+ * be needed in FindStreamingStart.
+ */
+ memset(&lz4preferences, 0, sizeof(LZ4F_frameInfo_t));
+ lz4preferences.frameInfo.contentSize = (unsigned long long)WalSegSz;
+ ctx_out = LZ4F_createCompressionContext(&ctx, LZ4F_VERSION);
+ lz4bufsize = LZ4F_compressBound(LZ4_IN_SIZE, &lz4preferences);
+ if (LZ4F_isError(ctx_out))
+ {
+ close(fd);
+ return NULL;
+ }
+
+ lz4buf = pg_malloc0(lz4bufsize);
+
+ /* add the header */
+ header_size = LZ4F_compressBegin(ctx, lz4buf, lz4bufsize, &lz4preferences);
+ if (LZ4F_isError(header_size))
+ {
+ pg_free(lz4buf);
+ close(fd);
+ return NULL;
+ }
+
+ errno = 0;
+ if (write(fd, lz4buf, header_size) != header_size)
+ {
+ int save_errno = errno;
+
+ (void) LZ4F_flush(ctx, lz4buf, lz4bufsize, NULL);
+ LZ4F_freeCompressionContext(ctx);
+ pg_free(lz4buf);
+ close(fd);
+
+ /*
+ * If write didn't set errno, assume problem is no disk space.
+ */
+ errno = save_errno ? save_errno : ENOSPC;
+ return NULL;
+ }
+ }
+#endif
/* Do pre-padding on non-compressed files */
- if (pad_to_size && dir_data->compression == 0)
+ if (pad_to_size && dir_data->compression_method == COMPRESSION_NONE)
{
PGAlignedXLogBlock zerobuf;
int bytes;
@@ -171,9 +241,19 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
fsync_parent_path(tmppath) != 0)
{
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
gzclose(gzfp);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ (void) LZ4F_flush(ctx, lz4buf, lz4bufsize, NULL);
+ LZ4F_freeCompressionContext(ctx);
+ pg_free(lz4buf);
+ close(fd);
+ }
+ else
#endif
close(fd);
return NULL;
@@ -182,9 +262,18 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
f = pg_malloc0(sizeof(DirectoryMethodFile));
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
f->gzfp = gzfp;
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ f->ctx = ctx;
+ f->lz4buf = lz4buf;
+ f->lz4bufsize = lz4bufsize;
+ }
+#endif
+
f->fd = fd;
f->currpos = 0;
f->pathname = pg_strdup(pathname);
@@ -204,9 +293,46 @@ dir_write(Walfile f, const void *buf, size_t count)
Assert(f != NULL);
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
r = (ssize_t) gzwrite(df->gzfp, buf, count);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ size_t chunk;
+ size_t remaining;
+ const void *inbuf = buf;
+
+ remaining = count;
+ while (remaining > 0)
+ {
+ size_t compressed;
+
+ if (remaining > LZ4_IN_SIZE)
+ chunk = LZ4_IN_SIZE;
+ else
+ chunk = remaining;
+
+ remaining -= chunk;
+ compressed = LZ4F_compressUpdate(df->ctx,
+ df->lz4buf, df->lz4bufsize,
+ inbuf, chunk,
+ NULL);
+
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+
+ inbuf = ((char *)inbuf) + chunk;
+ }
+
+ /* Our caller keeps track of the uncompressed size. */
+ r = (ssize_t)count;
+ }
+ else
#endif
r = write(df->fd, buf, count);
if (r > 0)
@@ -234,9 +360,34 @@ dir_close(Walfile f, WalCloseMethod method)
Assert(f != NULL);
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
r = gzclose(df->gzfp);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ /* Flush any internal buffers */
+ size_t compressed;
+
+ if (method == CLOSE_NORMAL)
+ compressed = LZ4F_compressEnd(df->ctx,
+ df->lz4buf, df->lz4bufsize,
+ NULL);
+ else
+ compressed = LZ4F_flush(df->ctx,
+ df->lz4buf, df->lz4bufsize,
+ NULL);
+
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+
+ r = close(df->fd);
+ }
+ else
#endif
r = close(df->fd);
@@ -291,6 +442,12 @@ dir_close(Walfile f, WalCloseMethod method)
}
}
+#ifdef HAVE_LIBLZ4
+ pg_free(df->lz4buf);
+ /* supports free on NULL */
+ LZ4F_freeCompressionContext(df->ctx);
+#endif
+
pg_free(df->pathname);
pg_free(df->fullpath);
if (df->temp_suffix)
@@ -309,12 +466,27 @@ dir_sync(Walfile f)
return 0;
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
{
if (gzflush(((DirectoryMethodFile *) f)->gzfp, Z_SYNC_FLUSH) != Z_OK)
return -1;
}
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ DirectoryMethodFile *df = (DirectoryMethodFile *) f;
+ size_t compressed;
+
+ /* Flush any internal buffers */
+ compressed = LZ4F_flush(df->ctx, df->lz4buf, df->lz4bufsize, NULL);
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+ }
+#endif
return fsync(((DirectoryMethodFile *) f)->fd);
}
@@ -373,7 +545,9 @@ dir_finish(void)
WalWriteMethod *
-CreateWalDirectoryMethod(const char *basedir, int compression, bool sync)
+CreateWalDirectoryMethod(const char *basedir,
+ WalCompressionMethod compression_method,
+ int compression, bool sync)
{
WalWriteMethod *method;
@@ -391,6 +565,7 @@ CreateWalDirectoryMethod(const char *basedir, int compression, bool sync)
method->getlasterror = dir_getlasterror;
dir_data = pg_malloc0(sizeof(DirectoryMethodData));
+ dir_data->compression_method = compression_method;
dir_data->compression = compression;
dir_data->basedir = pg_strdup(basedir);
dir_data->sync = sync;
@@ -1031,8 +1206,16 @@ tar_finish(void)
return true;
}
+/*
+ * The argument compression_method is currently ignored. It is in place for
+ * symmetry with CreateWalDirectoryMethod which uses it for distinguishing
+ * between the different compression methods. CreateWalTarMethod and its family
+ * of functions handle only zlib compression.
+ */
WalWriteMethod *
-CreateWalTarMethod(const char *tarbase, int compression, bool sync)
+CreateWalTarMethod(const char *tarbase,
+ WalCompressionMethod compression_method,
+ int compression, bool sync)
{
WalWriteMethod *method;
const char *suffix = (compression != 0) ? ".tar.gz" : ".tar";
diff --git a/src/bin/pg_basebackup/walmethods.h b/src/bin/pg_basebackup/walmethods.h
index 4abdfd8333..872b677da5 100644
--- a/src/bin/pg_basebackup/walmethods.h
+++ b/src/bin/pg_basebackup/walmethods.h
@@ -19,6 +19,13 @@ typedef enum
CLOSE_NO_RENAME
} WalCloseMethod;
+typedef enum
+{
+ COMPRESSION_LZ4,
+ COMPRESSION_ZLIB,
+ COMPRESSION_NONE
+} WalCompressionMethod;
+
/*
* A WalWriteMethod structure represents the different methods used
* to write the streaming WAL as it's received.
@@ -95,8 +102,11 @@ struct WalWriteMethod
* not all those required for pg_receivewal)
*/
WalWriteMethod *CreateWalDirectoryMethod(const char *basedir,
+ WalCompressionMethod compression_method,
int compression, bool sync);
-WalWriteMethod *CreateWalTarMethod(const char *tarbase, int compression, bool sync);
+WalWriteMethod *CreateWalTarMethod(const char *tarbase,
+ WalCompressionMethod compression_method,
+ int compression, bool sync);
/* Cleanup routines for previously-created methods */
void FreeWalDirectoryMethod(void);
--
2.25.1
Re: Teach pg_receivewal to use lz4 compression
On Thu, Sep 16, 2021 at 03:17:15PM +0000, gkokolatos@pm.me wrote:
Hopefully fixed.
Thanks for the new version. I have put my hands on the patch, and
began reviewing its internals with LZ4. I am not done with it yet,
and I have noticed some places that could be improved (error handling,
some uses of LZ4F_flush() that should be replaced LZ4F_compressEnd(),
and more tweaks). I'll send an updated version once I complete my
review, but that looks rather solid overall.
The changes done in close_walfile()@receivelog.c are useful taken
independently, so I have applied these separately.
--
Michael
Re: Teach pg_receivewal to use lz4 compression
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, September 17th, 2021 at 09:39, Michael Paquier <michael@paquier.xyz> wrote:
On Thu, Sep 16, 2021 at 03:17:15PM +0000, gkokolatos@pm.me wrote:
Hopefully fixed.
Thanks for the new version. I have put my hands on the patch, and
began reviewing its internals with LZ4. I am not done with it yet,
and I have noticed some places that could be improved (error handling,
some uses of LZ4F_flush() that should be replaced LZ4F_compressEnd(),
and more tweaks). I'll send an updated version once I complete my
review, but that looks rather solid overall.
Thanks! Looking forward to seeing it!
The changes done in close_walfile()@receivelog.c are useful taken
independently, so I have applied these separately.
Yeah, I was considering it to split them over to a separate commit,
then decided against it. Will do so in the future.
Cheers,
//Georgios
Show quoted text
--------------------------------------------------------------------
Michael
Re: Teach pg_receivewal to use lz4 compression
On Fri, Sep 17, 2021 at 08:12:42AM +0000, gkokolatos@pm.me wrote:
Yeah, I was considering it to split them over to a separate commit,
then decided against it. Will do so in the future.
I have been digging into the issue I saw in the TAP tests when closing
a segment, and found the problem. The way you manipulate
frameInfo.contentSize by just setting it to WalSegSz when *opening*
a segment causes problems on LZ4F_compressEnd(), making the code
throw a ERROR_frameSize_wrong. In lz4frame.c, the end of
LZ4F_compressEnd() triggers this check and the error:
if (cctxPtr->prefs.frameInfo.contentSize) {
if (cctxPtr->prefs.frameInfo.contentSize != cctxPtr->totalInSize)
return err0r(LZ4F_ERROR_frameSize_wrong);
}
We don't really care about contentSize as long as a segment is not
completed. Rather than filling contentSize all the time we write
something, we'd better update frameInfo once the segment is
completed and closed. That would also take take of the error as this
is not checked if contentSize is 0. It seems to me that we should
fill in the information when doing a CLOSE_NORMAL.
- if (stream->walmethod->compression() == 0 &&
+ if (stream->walmethod->compression() == COMPRESSION_NONE &&
stream->walmethod->existsfile(fn))
This one was a more serious issue, as the compression() callback would
return an integer for the compression level but v5 compared it to a
WalCompressionMethod. In order to take care of this issue, mainly for
pg_basebackup, I think that we have to update the compression()
callback to compression_method(), and it is cleaner to save the
compression method as well as the compression level for the tar data.I am attaching a new patch, on which I have done many tweaks and
adjustments while reviewing it. The attached patch fixes the second
issue, and I have done nothing about the first issue yet, but that
should be simple enough to address as this needs an update of the
frame info when closing a completed segment. Could you look at it?
Thanks,
--
Michael
Attachments:
v6-0001-Teach-pg_receivewal-to-use-LZ4-compression.patchtext/plain; charset=us-asciiDownload
From 37e3800d279566445864ed82f29e8d650c72d8cd Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Sat, 18 Sep 2021 15:11:49 +0900
Subject: [PATCH v6] Teach pg_receivewal to use LZ4 compression
The program pg_receivewal can use gzip compression to store the received WAL.
This commit teaches it to also be able to use LZ4 compression. It is required
that the binary is build using the -llz4 flag. It is enabled via the --with-lz4
flag on configuration time.
Previously, the user had to use the option --compress with a value between [0-9]
to denote that gzip compression was required. This specific behaviour has not
maintained. A newly introduced option --compression-method=[LZ4|gzip] can be
used to ask for the logs to be compressed. Compression values can be selected
only when the compression method is gzip. A compression value of 0 now returns
an error.
Under the hood there is nothing exceptional to be noted. Tar based archives have
not yet been taught to use LZ4 compression. If that is felt useful, then it is
easy to be added in the future.
Tests have been added to verify the creation and correctness of the generated
LZ4 files. The later is achieved by the use of LZ4 program, if present in the
installation.
---
src/bin/pg_basebackup/Makefile | 1 +
src/bin/pg_basebackup/pg_basebackup.c | 7 +-
src/bin/pg_basebackup/pg_receivewal.c | 278 +++++++++++++++----
src/bin/pg_basebackup/receivelog.c | 2 +-
src/bin/pg_basebackup/t/020_pg_receivewal.pl | 81 +++++-
src/bin/pg_basebackup/walmethods.c | 216 ++++++++++++--
src/bin/pg_basebackup/walmethods.h | 16 +-
doc/src/sgml/ref/pg_receivewal.sgml | 28 +-
src/Makefile.global.in | 1 +
src/tools/pgindent/typedefs.list | 1 +
10 files changed, 546 insertions(+), 85 deletions(-)
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index 459d514183..387d728345 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -24,6 +24,7 @@ export TAR
# used by the command "gzip" to pass down options, so stick with a different
# name.
export GZIP_PROGRAM=$(GZIP)
+export LZ4
override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport)
diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c
index 669aa207a3..18c6a93cec 100644
--- a/src/bin/pg_basebackup/pg_basebackup.c
+++ b/src/bin/pg_basebackup/pg_basebackup.c
@@ -555,10 +555,13 @@ LogStreamerMain(logstreamer_param *param)
stream.replication_slot = replication_slot;
if (format == 'p')
- stream.walmethod = CreateWalDirectoryMethod(param->xlog, 0,
+ stream.walmethod = CreateWalDirectoryMethod(param->xlog,
+ COMPRESSION_NONE, 0,
stream.do_sync);
else
- stream.walmethod = CreateWalTarMethod(param->xlog, compresslevel,
+ stream.walmethod = CreateWalTarMethod(param->xlog,
+ COMPRESSION_NONE, /* ignored */
+ compresslevel,
stream.do_sync);
if (!ReceiveXlogStream(param->bgconn, &stream))
diff --git a/src/bin/pg_basebackup/pg_receivewal.c b/src/bin/pg_basebackup/pg_receivewal.c
index d5140a79fe..48fd9491c9 100644
--- a/src/bin/pg_basebackup/pg_receivewal.c
+++ b/src/bin/pg_basebackup/pg_receivewal.c
@@ -29,9 +29,16 @@
#include "receivelog.h"
#include "streamutil.h"
+#ifdef HAVE_LIBLZ4
+#include "lz4frame.h"
+#endif
+
/* Time to sleep between reconnection attempts */
#define RECONNECT_SLEEP_TIME 5
+/* Default compression level for gzip compression method */
+#define DEFAULT_ZLIB_COMPRESSLEVEL 5
+
/* Global options */
static char *basedir = NULL;
static int verbose = 0;
@@ -45,6 +52,7 @@ static bool do_drop_slot = false;
static bool do_sync = true;
static bool synchronous = false;
static char *replication_slot = NULL;
+static WalCompressionMethod compression_method = COMPRESSION_NONE;
static XLogRecPtr endpos = InvalidXLogRecPtr;
@@ -63,16 +71,6 @@ disconnect_atexit(void)
PQfinish(conn);
}
-/* Routines to evaluate segment file format */
-#define IsCompressXLogFileName(fname) \
- (strlen(fname) == XLOG_FNAME_LEN + strlen(".gz") && \
- strspn(fname, "0123456789ABCDEF") == XLOG_FNAME_LEN && \
- strcmp((fname) + XLOG_FNAME_LEN, ".gz") == 0)
-#define IsPartialCompressXLogFileName(fname) \
- (strlen(fname) == XLOG_FNAME_LEN + strlen(".gz.partial") && \
- strspn(fname, "0123456789ABCDEF") == XLOG_FNAME_LEN && \
- strcmp((fname) + XLOG_FNAME_LEN, ".gz.partial") == 0)
-
static void
usage(void)
{
@@ -92,7 +90,10 @@ usage(void)
printf(_(" --synchronous flush write-ahead log immediately after writing\n"));
printf(_(" -v, --verbose output verbose messages\n"));
printf(_(" -V, --version output version information, then exit\n"));
- printf(_(" -Z, --compress=0-9 compress logs with given compression level\n"));
+ printf(_(" --compression-method=METHOD\n"
+ " method to compress logs\n"));
+ printf(_(" -Z, --compress=1-9 compress logs with given compression level (default: %d)"),
+ DEFAULT_ZLIB_COMPRESSLEVEL);
printf(_(" -?, --help show this help, then exit\n"));
printf(_("\nConnection options:\n"));
printf(_(" -d, --dbname=CONNSTR connection string\n"));
@@ -108,6 +109,79 @@ usage(void)
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
}
+
+/*
+ * Check if the filename looks like an xlog file. Also note if it is partial
+ * and/or compressed file.
+ */
+static bool
+is_xlogfilename(const char *filename, bool *ispartial,
+ WalCompressionMethod *wal_compression_method)
+{
+ size_t fname_len = strlen(filename);
+ size_t xlog_pattern_len = strspn(filename, "0123456789ABCDEF");
+
+ /* File does not look like a XLOG file */
+ if (xlog_pattern_len != XLOG_FNAME_LEN)
+ return false;
+
+ /* File looks like a complete uncompressed XLOG file */
+ if (fname_len == XLOG_FNAME_LEN)
+ {
+ *ispartial = false;
+ *wal_compression_method = COMPRESSION_NONE;
+ return true;
+ }
+
+ /* File looks like a complete zlib compressed XLOG file */
+ if (fname_len == XLOG_FNAME_LEN + strlen(".gz") &&
+ strcmp(filename + XLOG_FNAME_LEN, ".gz") == 0)
+ {
+ *ispartial = false;
+ *wal_compression_method = COMPRESSION_ZLIB;
+ return true;
+ }
+
+ /* File looks like a complete LZ4 compressed XLOG file */
+ if (fname_len == XLOG_FNAME_LEN + strlen(".lz4") &&
+ strcmp(filename + XLOG_FNAME_LEN, ".lz4") == 0)
+ {
+ *ispartial = false;
+ *wal_compression_method = COMPRESSION_LZ4;
+ return true;
+ }
+
+ /* File looks like a partial uncompressed XLOG file */
+ if (fname_len == XLOG_FNAME_LEN + strlen(".partial") &&
+ strcmp(filename + XLOG_FNAME_LEN, ".partial") == 0)
+ {
+ *ispartial = true;
+ *wal_compression_method = COMPRESSION_NONE;
+ return true;
+ }
+
+ /* File looks like a partial zlib compressed XLOG file */
+ if (fname_len == XLOG_FNAME_LEN + strlen(".gz.partial") &&
+ strcmp(filename + XLOG_FNAME_LEN, ".gz.partial") == 0)
+ {
+ *ispartial = true;
+ *wal_compression_method = COMPRESSION_ZLIB;
+ return true;
+ }
+
+ /* File looks like a partial LZ4 compressed XLOG file */
+ if (fname_len == XLOG_FNAME_LEN + strlen(".lz4.partial") &&
+ strcmp(filename + XLOG_FNAME_LEN, ".lz4.partial") == 0)
+ {
+ *ispartial = true;
+ *wal_compression_method = COMPRESSION_LZ4;
+ return true;
+ }
+
+ /* File does not look like something we recognise */
+ return false;
+}
+
static bool
stop_streaming(XLogRecPtr xlogpos, uint32 timeline, bool segment_finished)
{
@@ -213,33 +287,11 @@ FindStreamingStart(uint32 *tli)
{
uint32 tli;
XLogSegNo segno;
+ WalCompressionMethod wal_compression_method;
bool ispartial;
- bool iscompress;
- /*
- * Check if the filename looks like an xlog file, or a .partial file.
- */
- if (IsXLogFileName(dirent->d_name))
- {
- ispartial = false;
- iscompress = false;
- }
- else if (IsPartialXLogFileName(dirent->d_name))
- {
- ispartial = true;
- iscompress = false;
- }
- else if (IsCompressXLogFileName(dirent->d_name))
- {
- ispartial = false;
- iscompress = true;
- }
- else if (IsPartialCompressXLogFileName(dirent->d_name))
- {
- ispartial = true;
- iscompress = true;
- }
- else
+ if (!is_xlogfilename(dirent->d_name,
+ &ispartial, &wal_compression_method))
continue;
/*
@@ -250,14 +302,18 @@ FindStreamingStart(uint32 *tli)
/*
* Check that the segment has the right size, if it's supposed to be
* completed. For non-compressed segments just check the on-disk size
- * and see if it matches a completed segment. For compressed segments,
- * look at the last 4 bytes of the compressed file, which is where the
- * uncompressed size is located for gz files with a size lower than
- * 4GB, and then compare it to the size of a completed segment. The 4
- * last bytes correspond to the ISIZE member according to
- * http://www.zlib.org/rfc-gzip.html.
+ * and see if it matches a completed segment. For zlib compressed
+ * segments, look at the last 4 bytes of the compressed file, which is
+ * where the uncompressed size is located for gz files with a size
+ * lower than 4GB, and then compare it to the size of a completed
+ * segment. The 4 last bytes correspond to the ISIZE member according
+ * to http://www.zlib.org/rfc-gzip.html.
+ *
+ * For LZ4 compressed segments read the header using the exposed API
+ * and compare the uncompressed file size, stored in
+ * LZ4F_frameInfo_t{.contentSize}, to that of a completed segment.
*/
- if (!ispartial && !iscompress)
+ if (!ispartial && wal_compression_method == COMPRESSION_NONE)
{
struct stat statbuf;
char fullpath[MAXPGPATH * 2];
@@ -276,7 +332,7 @@ FindStreamingStart(uint32 *tli)
continue;
}
}
- else if (!ispartial && iscompress)
+ else if (!ispartial && wal_compression_method == COMPRESSION_ZLIB)
{
int fd;
char buf[4];
@@ -322,6 +378,80 @@ FindStreamingStart(uint32 *tli)
continue;
}
}
+ else if (!ispartial && compression_method == COMPRESSION_LZ4)
+ {
+#ifdef HAVE_LIBLZ4
+ int fd;
+ int r;
+ size_t consumed_len = LZ4F_HEADER_SIZE_MAX;
+ char buf[LZ4F_HEADER_SIZE_MAX];
+ char fullpath[MAXPGPATH * 2];
+ LZ4F_frameInfo_t frame_info = {0};
+ LZ4F_decompressionContext_t ctx = NULL;
+ LZ4F_errorCode_t status;
+
+ snprintf(fullpath, sizeof(fullpath), "%s/%s", basedir, dirent->d_name);
+
+ fd = open(fullpath, O_RDONLY | PG_BINARY, 0);
+ if (fd < 0)
+ {
+ pg_log_error("could not open file \"%s\": %m", fullpath);
+ exit(1);
+ }
+
+ r = read(fd, buf, sizeof(buf));
+ if (r != sizeof(buf))
+ {
+ if (r < 0)
+ pg_log_error("could not read file \"%s\": %m", fullpath);
+ else
+ pg_log_error("could not read file \"%s\": read %d of %zu",
+ fullpath, r, sizeof(buf));
+ exit(1);
+ }
+ close(fd);
+
+ status = LZ4F_createDecompressionContext(&ctx, LZ4F_VERSION);
+ if (LZ4F_isError(status))
+ {
+ pg_log_error("could not create LZ4 decompression context: %s",
+ LZ4F_getErrorName(status));
+ exit(1);
+ }
+
+ LZ4F_getFrameInfo(ctx, &frame_info, (void *) buf, &consumed_len);
+ if (consumed_len <= LZ4F_HEADER_SIZE_MIN ||
+ consumed_len >= LZ4F_HEADER_SIZE_MAX)
+ {
+ pg_log_warning("compressed segment file \"%s\" has incorrect header size %lu, skipping",
+ dirent->d_name, consumed_len);
+ (void) LZ4F_freeDecompressionContext(ctx);
+ continue;
+ }
+
+ if (frame_info.contentSize != WalSegSz)
+ {
+ pg_log_warning("compressed segment file \"%s\" has incorrect uncompressed size %lld, skipping",
+ dirent->d_name, frame_info.contentSize);
+ (void) LZ4F_freeDecompressionContext(ctx);
+ continue;
+ }
+
+ status = LZ4F_freeDecompressionContext(ctx);
+ if (LZ4F_isError(status))
+ {
+ pg_log_error("could not free LZ4 decompression context: %s",
+ LZ4F_getErrorName(status));
+ exit(1);
+ }
+#else
+ pg_log_error("could not check segment file \"%s\" compressed with LZ4",
+ dirent->d_name);
+ pg_log_error("this build does not support compression with %s",
+ "LZ4");
+ exit(1);
+#endif
+ }
/* Looks like a valid segment. Remember that we saw it. */
if ((segno > high_segno) ||
@@ -432,7 +562,9 @@ StreamLog(void)
stream.synchronous = synchronous;
stream.do_sync = do_sync;
stream.mark_done = false;
- stream.walmethod = CreateWalDirectoryMethod(basedir, compresslevel,
+ stream.walmethod = CreateWalDirectoryMethod(basedir,
+ compression_method,
+ compresslevel,
stream.do_sync);
stream.partial_suffix = ".partial";
stream.replication_slot = replication_slot;
@@ -485,6 +617,7 @@ main(int argc, char **argv)
{"status-interval", required_argument, NULL, 's'},
{"slot", required_argument, NULL, 'S'},
{"verbose", no_argument, NULL, 'v'},
+ {"compression-method", required_argument, NULL, 'I'},
{"compress", required_argument, NULL, 'Z'},
/* action */
{"create-slot", no_argument, NULL, 1},
@@ -570,8 +703,22 @@ main(int argc, char **argv)
case 'v':
verbose++;
break;
+ case 'I':
+ if (pg_strcasecmp(optarg, "gzip") == 0)
+ compression_method = COMPRESSION_ZLIB;
+ else if (pg_strcasecmp(optarg, "lz4") == 0)
+ compression_method = COMPRESSION_LZ4;
+ else if (pg_strcasecmp(optarg, "none") == 0)
+ compression_method = COMPRESSION_NONE;
+ else
+ {
+ pg_log_error("invalid value \"%s\" for option %s",
+ optarg, "--compress-method");
+ exit(1);
+ }
+ break;
case 'Z':
- if (!option_parse_int(optarg, "-Z/--compress", 0, 9,
+ if (!option_parse_int(optarg, "-Z/--compress", 1, 9,
&compresslevel))
exit(1);
break;
@@ -651,13 +798,44 @@ main(int argc, char **argv)
exit(1);
}
-#ifndef HAVE_LIBZ
- if (compresslevel != 0)
+
+ /*
+ * Compression related arguments
+ */
+ if (compression_method != COMPRESSION_NONE)
{
- pg_log_error("this build does not support compression");
+#ifndef HAVE_LIBZ
+ if (compression_method == COMPRESSION_ZLIB)
+ {
+ pg_log_error("this build does not support compression with %s",
+ "gzip");
+ exit(1);
+ }
+#endif
+#ifndef HAVE_LIBLZ4
+ if (compression_method == COMPRESSION_LZ4)
+ {
+ pg_log_error("this build does not support compression with %s",
+ "LZ4");
+ exit(1);
+ }
+#endif
+ }
+
+ if (compression_method != COMPRESSION_ZLIB && compresslevel != 0)
+ {
+ pg_log_error("can only use --compress with --compression-method=gzip");
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+ progname);
exit(1);
}
-#endif
+
+ if (compression_method == COMPRESSION_ZLIB && compresslevel == 0)
+ {
+ pg_log_info("no --compression specified, will be using %d",
+ DEFAULT_ZLIB_COMPRESSLEVEL);
+ compresslevel = DEFAULT_ZLIB_COMPRESSLEVEL;
+ }
/*
* Check existence of destination folder.
diff --git a/src/bin/pg_basebackup/receivelog.c b/src/bin/pg_basebackup/receivelog.c
index 72b8d9e315..2d4f660daa 100644
--- a/src/bin/pg_basebackup/receivelog.c
+++ b/src/bin/pg_basebackup/receivelog.c
@@ -109,7 +109,7 @@ open_walfile(StreamCtl *stream, XLogRecPtr startpoint)
* When streaming to tar, no file with this name will exist before, so we
* never have to verify a size.
*/
- if (stream->walmethod->compression() == 0 &&
+ if (stream->walmethod->compression_method() == COMPRESSION_NONE &&
stream->walmethod->existsfile(fn))
{
size = stream->walmethod->get_file_size(fn);
diff --git a/src/bin/pg_basebackup/t/020_pg_receivewal.pl b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
index 0b33d73900..1bd8fd0d3e 100644
--- a/src/bin/pg_basebackup/t/020_pg_receivewal.pl
+++ b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
@@ -5,7 +5,7 @@ use strict;
use warnings;
use TestLib;
use PostgresNode;
-use Test::More tests => 27;
+use Test::More tests => 34;
program_help_ok('pg_receivewal');
program_version_ok('pg_receivewal');
@@ -33,6 +33,13 @@ $primary->command_fails(
$primary->command_fails(
[ 'pg_receivewal', '-D', $stream_dir, '--synchronous', '--no-sync' ],
'failure if --synchronous specified with --no-sync');
+$primary->command_fails_like(
+ [
+ 'pg_receivewal', '-D', $stream_dir, '--compression-method', 'none',
+ '--compress', '1'
+ ],
+ qr/\Qpg_receivewal: error: can only use --compress with --compression-method=gzip/,
+ 'failure if --compression-method=none specified with --compress');
# Slot creation and drop
my $slot_name = 'test';
@@ -41,7 +48,7 @@ $primary->command_ok(
'creating a replication slot');
my $slot = $primary->slot($slot_name);
is($slot->{'slot_type'}, 'physical', 'physical replication slot was created');
-is($slot->{'restart_lsn'}, '', 'restart LSN of new slot is null');
+is($slot->{'restart_lsn'}, '', 'restart LSN of new slot is null');
$primary->command_ok([ 'pg_receivewal', '--slot', $slot_name, '--drop-slot' ],
'dropping a replication slot');
is($primary->slot($slot_name)->{'slot_type'},
@@ -90,8 +97,11 @@ SKIP:
# a valid value.
$primary->command_ok(
[
- 'pg_receivewal', '-D', $stream_dir, '--verbose',
- '--endpos', $nextlsn, '--compress', '1 ',
+ 'pg_receivewal', '-D',
+ $stream_dir, '--verbose',
+ '--endpos', $nextlsn,
+ '--compression-method', 'gzip',
+ '--compress', '1 ',
'--no-loop'
],
"streaming some WAL using ZLIB compression");
@@ -128,14 +138,71 @@ SKIP:
"gzip verified the integrity of compressed WAL segments");
}
+# Check LZ4 compression if available
+SKIP:
+{
+ skip "postgres was not built with LZ4 support", 5
+ if (!check_pg_config("#define HAVE_LIBLZ4 1"));
+
+ # Generate more WAL including one completed, compressed segment.
+ $primary->psql('postgres', 'SELECT pg_switch_wal();');
+ $nextlsn =
+ $primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
+ chomp($nextlsn);
+ $primary->psql('postgres',
+ 'INSERT INTO test_table VALUES (generate_series(201,300));');
+
+ # Stream up to the given position
+ $primary->command_ok(
+ [
+ 'pg_receivewal', '-D',
+ $stream_dir, '--verbose',
+ '--endpos', $nextlsn,
+ '--no-loop', '--compression-method',
+ 'lz4'
+ ],
+ 'streaming some WAL using --compression-method=lz4');
+
+ # Verify that the stored files are generated with their expected
+ # names.
+ my @lz4_wals = glob "$stream_dir/*.lz4";
+ is(scalar(@lz4_wals), 1,
+ "one WAL segment compressed with LZ4 was created");
+ my @lz4_partial_wals = glob "$stream_dir/*.lz4.partial";
+ is(scalar(@lz4_partial_wals),
+ 1, "one partial WAL segment compressed with LZ4 was created");
+
+ # Verify that the start streaming position is computed correctly by
+ # comparing it with the partial file generated previously. The name
+ # of the previous partial, now-completed WAL segment is updated, keeping
+ # its base number.
+ $partial_wals[0] =~ s/(\.gz)?\.partial$/.lz4/;
+ is($lz4_wals[0] eq $partial_wals[0],
+ 1, "one partial WAL segment is now completed");
+ # Update the list of partial wals with the current one.
+ @partial_wals = @lz4_partial_wals;
+
+ # Check the integrity of the completed segment, if LZ4 is an available
+ # command.
+ my $lz4 = $ENV{LZ4};
+ skip "program lz4 is not found in your system", 1
+ if ( !defined $lz4
+ || $lz4 eq ''
+ || system_log($lz4, '--version') != 0);
+
+ my $lz4_is_valid = system_log($lz4, '-t', @lz4_wals);
+ is($lz4_is_valid, 0,
+ "lz4 verified the integrity of compressed WAL segments");
+}
+
# Verify that the start streaming position is computed and that the value is
-# correct regardless of whether ZLIB is available.
+# correct regardless of whether any compression is available.
$primary->psql('postgres', 'SELECT pg_switch_wal();');
$nextlsn =
$primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
chomp($nextlsn);
$primary->psql('postgres',
- 'INSERT INTO test_table VALUES (generate_series(200,300));');
+ 'INSERT INTO test_table VALUES (generate_series(301,400));');
$primary->command_ok(
[
'pg_receivewal', '-D', $stream_dir, '--verbose',
@@ -143,7 +210,7 @@ $primary->command_ok(
],
"streaming some WAL");
-$partial_wals[0] =~ s/(\.gz)?.partial//;
+$partial_wals[0] =~ s/(\.gz|\.lz4)?.partial//;
ok(-e $partial_wals[0], "check that previously partial WAL is now complete");
# Permissions on WAL files should be default
diff --git a/src/bin/pg_basebackup/walmethods.c b/src/bin/pg_basebackup/walmethods.c
index 8695647db4..6c43164e38 100644
--- a/src/bin/pg_basebackup/walmethods.c
+++ b/src/bin/pg_basebackup/walmethods.c
@@ -17,6 +17,10 @@
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
+
+#ifdef HAVE_LIBLZ4
+#include <lz4frame.h>
+#endif
#ifdef HAVE_LIBZ
#include <zlib.h>
#endif
@@ -30,6 +34,9 @@
/* Size of zlib buffer for .tar.gz */
#define ZLIB_OUT_SIZE 4096
+/* Size of lz4 input chunk for .lz4 */
+#define LZ4_IN_SIZE 4096
+
/*-------------------------------------------------------------------------
* WalDirectoryMethod - write wal to a directory looking like pg_wal
*-------------------------------------------------------------------------
@@ -41,6 +48,7 @@
typedef struct DirectoryMethodData
{
char *basedir;
+ WalCompressionMethod compression_method;
int compression;
bool sync;
} DirectoryMethodData;
@@ -59,6 +67,11 @@ typedef struct DirectoryMethodFile
#ifdef HAVE_LIBZ
gzFile gzfp;
#endif
+#ifdef HAVE_LIBLZ4
+ LZ4F_compressionContext_t ctx;
+ size_t lz4bufsize;
+ void *lz4buf;
+#endif
} DirectoryMethodFile;
static const char *
@@ -74,7 +87,9 @@ dir_get_file_name(const char *pathname, const char *temp_suffix)
char *filename = pg_malloc0(MAXPGPATH * sizeof(char));
snprintf(filename, MAXPGPATH, "%s%s%s",
- pathname, dir_data->compression > 0 ? ".gz" : "",
+ pathname,
+ dir_data->compression_method == COMPRESSION_ZLIB ? ".gz" :
+ dir_data->compression_method == COMPRESSION_LZ4 ? ".lz4" : "",
temp_suffix ? temp_suffix : "");
return filename;
@@ -90,6 +105,11 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
#ifdef HAVE_LIBZ
gzFile gzfp = NULL;
#endif
+#ifdef HAVE_LIBLZ4
+ LZ4F_compressionContext_t ctx = NULL;
+ size_t lz4bufsize = 0;
+ void *lz4buf = NULL;
+#endif
filename = dir_get_file_name(pathname, temp_suffix);
snprintf(tmppath, sizeof(tmppath), "%s/%s",
@@ -107,7 +127,7 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
return NULL;
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
{
gzfp = gzdopen(fd, "wb");
if (gzfp == NULL)
@@ -124,9 +144,59 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
}
}
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ LZ4F_preferences_t lz4preferences = {0};
+ size_t ctx_out;
+ size_t header_size;
+
+ /*
+ * Set all the preferences to default but do note contentSize. It will
+ * be needed in FindStreamingStart.
+ */
+ memset(&lz4preferences, 0, sizeof(LZ4F_frameInfo_t));
+ lz4preferences.frameInfo.contentSize = (unsigned long long) WalSegSz;
+ ctx_out = LZ4F_createCompressionContext(&ctx, LZ4F_VERSION);
+ lz4bufsize = LZ4F_compressBound(LZ4_IN_SIZE, &lz4preferences);
+ if (LZ4F_isError(ctx_out))
+ {
+ close(fd);
+ return NULL;
+ }
+
+ lz4buf = pg_malloc0(lz4bufsize);
+
+ /* add the header */
+ header_size = LZ4F_compressBegin(ctx, lz4buf, lz4bufsize, &lz4preferences);
+ if (LZ4F_isError(header_size))
+ {
+ pg_free(lz4buf);
+ close(fd);
+ return NULL;
+ }
+
+ errno = 0;
+ if (write(fd, lz4buf, header_size) != header_size)
+ {
+ int save_errno = errno;
+
+ (void) LZ4F_compressEnd(ctx, lz4buf, lz4bufsize, NULL);
+ (void) LZ4F_freeCompressionContext(ctx);
+ pg_free(lz4buf);
+ close(fd);
+
+ /*
+ * If write didn't set errno, assume problem is no disk space.
+ */
+ errno = save_errno ? save_errno : ENOSPC;
+ return NULL;
+ }
+ }
+#endif
/* Do pre-padding on non-compressed files */
- if (pad_to_size && dir_data->compression == 0)
+ if (pad_to_size && dir_data->compression_method == COMPRESSION_NONE)
{
PGAlignedXLogBlock zerobuf;
int bytes;
@@ -171,9 +241,19 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
fsync_parent_path(tmppath) != 0)
{
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
gzclose(gzfp);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ (void) LZ4F_compressEnd(ctx, lz4buf, lz4bufsize, NULL);
+ (void) LZ4F_freeCompressionContext(ctx);
+ pg_free(lz4buf);
+ close(fd);
+ }
+ else
#endif
close(fd);
return NULL;
@@ -182,9 +262,18 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
f = pg_malloc0(sizeof(DirectoryMethodFile));
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
f->gzfp = gzfp;
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ f->ctx = ctx;
+ f->lz4buf = lz4buf;
+ f->lz4bufsize = lz4bufsize;
+ }
+#endif
+
f->fd = fd;
f->currpos = 0;
f->pathname = pg_strdup(pathname);
@@ -204,9 +293,46 @@ dir_write(Walfile f, const void *buf, size_t count)
Assert(f != NULL);
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
r = (ssize_t) gzwrite(df->gzfp, buf, count);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ size_t chunk;
+ size_t remaining;
+ const void *inbuf = buf;
+
+ remaining = count;
+ while (remaining > 0)
+ {
+ size_t compressed;
+
+ if (remaining > LZ4_IN_SIZE)
+ chunk = LZ4_IN_SIZE;
+ else
+ chunk = remaining;
+
+ remaining -= chunk;
+ compressed = LZ4F_compressUpdate(df->ctx,
+ df->lz4buf, df->lz4bufsize,
+ inbuf, chunk,
+ NULL);
+
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+
+ inbuf = ((char *) inbuf) + chunk;
+ }
+
+ /* Our caller keeps track of the uncompressed size. */
+ r = (ssize_t) count;
+ }
+ else
#endif
r = write(df->fd, buf, count);
if (r > 0)
@@ -234,9 +360,29 @@ dir_close(Walfile f, WalCloseMethod method)
Assert(f != NULL);
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
r = gzclose(df->gzfp);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ /* Flush any internal buffers */
+ size_t compressed;
+
+ compressed = LZ4F_compressEnd(df->ctx,
+ df->lz4buf, df->lz4bufsize,
+ NULL);
+
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+
+ r = close(df->fd);
+ }
+ else
#endif
r = close(df->fd);
@@ -291,6 +437,12 @@ dir_close(Walfile f, WalCloseMethod method)
}
}
+#ifdef HAVE_LIBLZ4
+ pg_free(df->lz4buf);
+ /* supports free on NULL */
+ LZ4F_freeCompressionContext(df->ctx);
+#endif
+
pg_free(df->pathname);
pg_free(df->fullpath);
if (df->temp_suffix)
@@ -309,12 +461,27 @@ dir_sync(Walfile f)
return 0;
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
{
if (gzflush(((DirectoryMethodFile *) f)->gzfp, Z_SYNC_FLUSH) != Z_OK)
return -1;
}
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ DirectoryMethodFile *df = (DirectoryMethodFile *) f;
+ size_t compressed;
+
+ /* Flush any internal buffers */
+ compressed = LZ4F_flush(df->ctx, df->lz4buf, df->lz4bufsize, NULL);
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+ }
+#endif
return fsync(((DirectoryMethodFile *) f)->fd);
}
@@ -334,10 +501,10 @@ dir_get_file_size(const char *pathname)
return statbuf.st_size;
}
-static int
-dir_compression(void)
+static WalCompressionMethod
+dir_compression_method(void)
{
- return dir_data->compression;
+ return dir_data->compression_method;
}
static bool
@@ -373,7 +540,9 @@ dir_finish(void)
WalWriteMethod *
-CreateWalDirectoryMethod(const char *basedir, int compression, bool sync)
+CreateWalDirectoryMethod(const char *basedir,
+ WalCompressionMethod compression_method,
+ int compression, bool sync)
{
WalWriteMethod *method;
@@ -383,7 +552,7 @@ CreateWalDirectoryMethod(const char *basedir, int compression, bool sync)
method->get_current_pos = dir_get_current_pos;
method->get_file_size = dir_get_file_size;
method->get_file_name = dir_get_file_name;
- method->compression = dir_compression;
+ method->compression_method = dir_compression_method;
method->close = dir_close;
method->sync = dir_sync;
method->existsfile = dir_existsfile;
@@ -391,6 +560,7 @@ CreateWalDirectoryMethod(const char *basedir, int compression, bool sync)
method->getlasterror = dir_getlasterror;
dir_data = pg_malloc0(sizeof(DirectoryMethodData));
+ dir_data->compression_method = compression_method;
dir_data->compression = compression;
dir_data->basedir = pg_strdup(basedir);
dir_data->sync = sync;
@@ -424,6 +594,7 @@ typedef struct TarMethodData
{
char *tarfilename;
int fd;
+ WalCompressionMethod compression_method;
int compression;
bool sync;
TarMethodFile *currentfile;
@@ -731,10 +902,10 @@ tar_get_file_size(const char *pathname)
return -1;
}
-static int
-tar_compression(void)
+static WalCompressionMethod
+tar_compression_method(void)
{
- return tar_data->compression;
+ return tar_data->compression_method;
}
static off_t
@@ -1031,8 +1202,16 @@ tar_finish(void)
return true;
}
+/*
+ * The argument compression_method is currently ignored. It is in place for
+ * symmetry with CreateWalDirectoryMethod which uses it for distinguishing
+ * between the different compression methods. CreateWalTarMethod and its family
+ * of functions handle only zlib compression.
+ */
WalWriteMethod *
-CreateWalTarMethod(const char *tarbase, int compression, bool sync)
+CreateWalTarMethod(const char *tarbase,
+ WalCompressionMethod compression_method,
+ int compression, bool sync)
{
WalWriteMethod *method;
const char *suffix = (compression != 0) ? ".tar.gz" : ".tar";
@@ -1043,7 +1222,7 @@ CreateWalTarMethod(const char *tarbase, int compression, bool sync)
method->get_current_pos = tar_get_current_pos;
method->get_file_size = tar_get_file_size;
method->get_file_name = tar_get_file_name;
- method->compression = tar_compression;
+ method->compression_method = tar_compression_method;
method->close = tar_close;
method->sync = tar_sync;
method->existsfile = tar_existsfile;
@@ -1054,6 +1233,7 @@ CreateWalTarMethod(const char *tarbase, int compression, bool sync)
tar_data->tarfilename = pg_malloc0(strlen(tarbase) + strlen(suffix) + 1);
sprintf(tar_data->tarfilename, "%s%s", tarbase, suffix);
tar_data->fd = -1;
+ tar_data->compression_method = compression_method;
tar_data->compression = compression;
tar_data->sync = sync;
#ifdef HAVE_LIBZ
diff --git a/src/bin/pg_basebackup/walmethods.h b/src/bin/pg_basebackup/walmethods.h
index 4abdfd8333..3e378c87b6 100644
--- a/src/bin/pg_basebackup/walmethods.h
+++ b/src/bin/pg_basebackup/walmethods.h
@@ -19,6 +19,13 @@ typedef enum
CLOSE_NO_RENAME
} WalCloseMethod;
+typedef enum
+{
+ COMPRESSION_LZ4,
+ COMPRESSION_ZLIB,
+ COMPRESSION_NONE
+} WalCompressionMethod;
+
/*
* A WalWriteMethod structure represents the different methods used
* to write the streaming WAL as it's received.
@@ -58,8 +65,8 @@ struct WalWriteMethod
*/
char *(*get_file_name) (const char *pathname, const char *temp_suffix);
- /* Return the level of compression */
- int (*compression) (void);
+ /* Returns the compression method */
+ WalCompressionMethod (*compression_method) (void);
/*
* Write count number of bytes to the file, and return the number of bytes
@@ -95,8 +102,11 @@ struct WalWriteMethod
* not all those required for pg_receivewal)
*/
WalWriteMethod *CreateWalDirectoryMethod(const char *basedir,
+ WalCompressionMethod compression_method,
int compression, bool sync);
-WalWriteMethod *CreateWalTarMethod(const char *tarbase, int compression, bool sync);
+WalWriteMethod *CreateWalTarMethod(const char *tarbase,
+ WalCompressionMethod compression_method,
+ int compression, bool sync);
/* Cleanup routines for previously-created methods */
void FreeWalDirectoryMethod(void);
diff --git a/doc/src/sgml/ref/pg_receivewal.sgml b/doc/src/sgml/ref/pg_receivewal.sgml
index 45b544cf49..f6c710bfe3 100644
--- a/doc/src/sgml/ref/pg_receivewal.sgml
+++ b/doc/src/sgml/ref/pg_receivewal.sgml
@@ -229,15 +229,35 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--compression-method=<replaceable class="parameter">level</replaceable></option></term>
+ <listitem>
+ <para>
+ Enables compression of write-ahead logs using the specified method.
+ Supported values are <literal>lz4</literal>, <literal>gzip</literal>
+ and <literal>none</literal>.
+ For the <productname>LZ4</productname> method to be available,
+ <productname>PostgreSQL</productname> must have been have been compiled
+ with <option>--with-lz4</option>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-Z <replaceable class="parameter">level</replaceable></option></term>
<term><option>--compress=<replaceable class="parameter">level</replaceable></option></term>
<listitem>
<para>
- Enables gzip compression of write-ahead logs, and specifies the
- compression level (0 through 9, 0 being no compression and 9 being best
- compression). The suffix <filename>.gz</filename> will
- automatically be added to all filenames.
+ Specifies the compression level (<literal>1</literal> through
+ <literal>9</literal>, <literal>1</literal> being worst compression
+ and <literal>9</literal> being best compression) for
+ <application>gzip</application> compressed WAL segments. The
+ default value is <literal>5</literal>.
+ </para>
+
+ <para>
+ This option requires <option>--compression-method</option> to be
+ specified with <literal>gzip</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index e4fd7b5290..c03b646727 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -350,6 +350,7 @@ XGETTEXT = @XGETTEXT@
GZIP = gzip
BZIP2 = bzip2
+LZ4 = lz4
DOWNLOAD = wget -O $@ --no-use-server-timestamps
#DOWNLOAD = curl -o $@
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 402a6617a9..434db61fdf 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2851,6 +2851,7 @@ WaitEventTimeout
WaitPMResult
WalCloseMethod
WalCompression
+WalCompressionMethod
WalLevel
WalRcvData
WalRcvExecResult
--
2.33.0
Re: Teach pg_receivewal to use lz4 compression
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Saturday, September 18th, 2021 at 8:18 AM, Michael Paquier <michael@paquier.xyz> wrote:
On Fri, Sep 17, 2021 at 08:12:42AM +0000, gkokolatos@pm.me wrote:
I have been digging into the issue I saw in the TAP tests when closing
a segment, and found the problem. The way you manipulate
frameInfo.contentSize by just setting it to WalSegSz when *opening*
a segment causes problems on LZ4F_compressEnd(), making the code
throw a ERROR_frameSize_wrong. In lz4frame.c, the end of
LZ4F_compressEnd() triggers this check and the error:
if (cctxPtr->prefs.frameInfo.contentSize) {
if (cctxPtr->prefs.frameInfo.contentSize != cctxPtr->totalInSize)
return err0r(LZ4F_ERROR_frameSize_wrong);
}We don't really care about contentSize as long as a segment is not
completed. Rather than filling contentSize all the time we write
something, we'd better update frameInfo once the segment is
completed and closed. That would also take take of the error as this
is not checked if contentSize is 0. It seems to me that we should
fill in the information when doing a CLOSE_NORMAL.
Thank you for the comment. I think that the opposite should be done. At the time
that the file is closed, the header is already written to disk. We have no way
to know that is not. If we need to go back to refill the information, we will
have to ask for the API to produce a new header. There is little guarantee that
the header size will be the same and as a consequence we will have to shift
the actual data around.
In the attached, the header is rewritten only when closing an incomplete
segment. For all intents and purposes that segment is not usable. However there
might be custom scripts that might want to attempt to parse even an otherwise
unusable file.
A different and easier approach would be to simply prepare the LZ4 context for
future actions and simply ignore the file.
- if (stream->walmethod->compression() == 0 && + if (stream->walmethod->compression() == COMPRESSION_NONE && stream->walmethod->existsfile(fn)) This one was a more serious issue, as the compression() callback would return an integer for the compression level but v5 compared it to a WalCompressionMethod. In order to take care of this issue, mainly for pg_basebackup, I think that we have to update the compression() callback to compression_method(), and it is cleaner to save the compression method as well as the compression level for the tar data.
Agreed.
I am attaching a new patch, on which I have done many tweaks and
adjustments while reviewing it. The attached patch fixes the second
issue, and I have done nothing about the first issue yet, but that
should be simple enough to address as this needs an update of the
frame info when closing a completed segment. Could you look at it?
Thank you. Find v7 attached, rebased to the current head.
Cheers,
//Georgios
Show quoted text
Thanks,
--
Michae
Attachments:
v7-0001-Teach-pg_receivewal-to-use-LZ4-compression.patchtext/x-patch; name=v7-0001-Teach-pg_receivewal-to-use-LZ4-compression.patchDownload
From c3c2eca22102cd0186eb1975339248a200e1ceb9 Mon Sep 17 00:00:00 2001
From: Georgios Kokolatos <gkokolatos@pm.me>
Date: Fri, 22 Oct 2021 13:14:15 +0000
Subject: [PATCH v7] Teach pg_receivewal to use LZ4 compression
The program pg_receivewal can use gzip compression to store the received WAL.
This commit teaches it to also be able to use LZ4 compression. It is required
that the binary is build using the -llz4 flag. It is enabled via the --with-lz4
flag on configuration time.
Previously, the user had to use the option --compress with a value between [0-9]
to denote that gzip compression was required. This specific behaviour has not
maintained. A newly introduced option --compression-method=[LZ4|gzip] can be
used to ask for the logs to be compressed. Compression values can be selected
only when the compression method is gzip. A compression value of 0 now returns
an error.
Under the hood there is nothing exceptional to be noted. Tar based archives have
not yet been taught to use LZ4 compression. If that is felt useful, then it is
easy to be added in the future.
Tests have been added to verify the creation and correctness of the generated
LZ4 files. The later is achieved by the use of LZ4 program, if present in the
installation.
---
doc/src/sgml/ref/pg_receivewal.sgml | 28 +-
src/Makefile.global.in | 1 +
src/bin/pg_basebackup/Makefile | 1 +
src/bin/pg_basebackup/pg_basebackup.c | 7 +-
src/bin/pg_basebackup/pg_receivewal.c | 274 +++++++++++++++----
src/bin/pg_basebackup/receivelog.c | 2 +-
src/bin/pg_basebackup/t/020_pg_receivewal.pl | 79 +++++-
src/bin/pg_basebackup/walmethods.c | 263 ++++++++++++++++--
src/bin/pg_basebackup/walmethods.h | 16 +-
src/tools/pgindent/typedefs.list | 1 +
10 files changed, 589 insertions(+), 83 deletions(-)
diff --git a/doc/src/sgml/ref/pg_receivewal.sgml b/doc/src/sgml/ref/pg_receivewal.sgml
index 9fde2fd2ef..bc2710131a 100644
--- a/doc/src/sgml/ref/pg_receivewal.sgml
+++ b/doc/src/sgml/ref/pg_receivewal.sgml
@@ -263,15 +263,35 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--compression-method=<replaceable class="parameter">level</replaceable></option></term>
+ <listitem>
+ <para>
+ Enables compression of write-ahead logs using the specified method.
+ Supported values are <literal>lz4</literal>, <literal>gzip</literal>
+ and <literal>none</literal>.
+ For the <productname>LZ4</productname> method to be available,
+ <productname>PostgreSQL</productname> must have been have been compiled
+ with <option>--with-lz4</option>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-Z <replaceable class="parameter">level</replaceable></option></term>
<term><option>--compress=<replaceable class="parameter">level</replaceable></option></term>
<listitem>
<para>
- Enables gzip compression of write-ahead logs, and specifies the
- compression level (0 through 9, 0 being no compression and 9 being best
- compression). The suffix <filename>.gz</filename> will
- automatically be added to all filenames.
+ Specifies the compression level (<literal>1</literal> through
+ <literal>9</literal>, <literal>1</literal> being worst compression
+ and <literal>9</literal> being best compression) for
+ <application>gzip</application> compressed WAL segments. The
+ default value is <literal>5</literal>.
+ </para>
+
+ <para>
+ This option requires <option>--compression-method</option> to be
+ specified with <literal>gzip</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 533c12fef9..05c54b27de 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -350,6 +350,7 @@ XGETTEXT = @XGETTEXT@
GZIP = gzip
BZIP2 = bzip2
+LZ4 = lz4
DOWNLOAD = wget -O $@ --no-use-server-timestamps
#DOWNLOAD = curl -o $@
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index 459d514183..387d728345 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -24,6 +24,7 @@ export TAR
# used by the command "gzip" to pass down options, so stick with a different
# name.
export GZIP_PROGRAM=$(GZIP)
+export LZ4
override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport)
diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c
index 27ee6394cf..cdea3711b7 100644
--- a/src/bin/pg_basebackup/pg_basebackup.c
+++ b/src/bin/pg_basebackup/pg_basebackup.c
@@ -555,10 +555,13 @@ LogStreamerMain(logstreamer_param *param)
stream.replication_slot = replication_slot;
if (format == 'p')
- stream.walmethod = CreateWalDirectoryMethod(param->xlog, 0,
+ stream.walmethod = CreateWalDirectoryMethod(param->xlog,
+ COMPRESSION_NONE, 0,
stream.do_sync);
else
- stream.walmethod = CreateWalTarMethod(param->xlog, compresslevel,
+ stream.walmethod = CreateWalTarMethod(param->xlog,
+ COMPRESSION_NONE, /* ignored */
+ compresslevel,
stream.do_sync);
if (!ReceiveXlogStream(param->bgconn, &stream))
diff --git a/src/bin/pg_basebackup/pg_receivewal.c b/src/bin/pg_basebackup/pg_receivewal.c
index 04ba20b197..d22fd63770 100644
--- a/src/bin/pg_basebackup/pg_receivewal.c
+++ b/src/bin/pg_basebackup/pg_receivewal.c
@@ -29,9 +29,16 @@
#include "receivelog.h"
#include "streamutil.h"
+#ifdef HAVE_LIBLZ4
+#include "lz4frame.h"
+#endif
+
/* Time to sleep between reconnection attempts */
#define RECONNECT_SLEEP_TIME 5
+/* this is just the redefinition of a libz constant */
+#define Z_DEFAULT_COMPRESSION (-1)
+
/* Global options */
static char *basedir = NULL;
static int verbose = 0;
@@ -45,6 +52,7 @@ static bool do_drop_slot = false;
static bool do_sync = true;
static bool synchronous = false;
static char *replication_slot = NULL;
+static WalCompressionMethod compression_method = COMPRESSION_NONE;
static XLogRecPtr endpos = InvalidXLogRecPtr;
@@ -63,16 +71,6 @@ disconnect_atexit(void)
PQfinish(conn);
}
-/* Routines to evaluate segment file format */
-#define IsCompressXLogFileName(fname) \
- (strlen(fname) == XLOG_FNAME_LEN + strlen(".gz") && \
- strspn(fname, "0123456789ABCDEF") == XLOG_FNAME_LEN && \
- strcmp((fname) + XLOG_FNAME_LEN, ".gz") == 0)
-#define IsPartialCompressXLogFileName(fname) \
- (strlen(fname) == XLOG_FNAME_LEN + strlen(".gz.partial") && \
- strspn(fname, "0123456789ABCDEF") == XLOG_FNAME_LEN && \
- strcmp((fname) + XLOG_FNAME_LEN, ".gz.partial") == 0)
-
static void
usage(void)
{
@@ -92,7 +90,9 @@ usage(void)
printf(_(" --synchronous flush write-ahead log immediately after writing\n"));
printf(_(" -v, --verbose output verbose messages\n"));
printf(_(" -V, --version output version information, then exit\n"));
- printf(_(" -Z, --compress=0-9 compress logs with given compression level\n"));
+ printf(_(" --compression-method=METHOD\n"
+ " method to compress logs\n"));
+ printf(_(" -Z, --compress=1-9 compress logs with given compression level\n"));
printf(_(" -?, --help show this help, then exit\n"));
printf(_("\nConnection options:\n"));
printf(_(" -d, --dbname=CONNSTR connection string\n"));
@@ -108,6 +108,79 @@ usage(void)
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
}
+
+/*
+ * Check if the filename looks like an xlog file. Also note if it is partial
+ * and/or compressed file.
+ */
+static bool
+is_xlogfilename(const char *filename, bool *ispartial,
+ WalCompressionMethod *wal_compression_method)
+{
+ size_t fname_len = strlen(filename);
+ size_t xlog_pattern_len = strspn(filename, "0123456789ABCDEF");
+
+ /* File does not look like a XLOG file */
+ if (xlog_pattern_len != XLOG_FNAME_LEN)
+ return false;
+
+ /* File looks like a complete uncompressed XLOG file */
+ if (fname_len == XLOG_FNAME_LEN)
+ {
+ *ispartial = false;
+ *wal_compression_method = COMPRESSION_NONE;
+ return true;
+ }
+
+ /* File looks like a complete zlib compressed XLOG file */
+ if (fname_len == XLOG_FNAME_LEN + strlen(".gz") &&
+ strcmp(filename + XLOG_FNAME_LEN, ".gz") == 0)
+ {
+ *ispartial = false;
+ *wal_compression_method = COMPRESSION_ZLIB;
+ return true;
+ }
+
+ /* File looks like a complete LZ4 compressed XLOG file */
+ if (fname_len == XLOG_FNAME_LEN + strlen(".lz4") &&
+ strcmp(filename + XLOG_FNAME_LEN, ".lz4") == 0)
+ {
+ *ispartial = false;
+ *wal_compression_method = COMPRESSION_LZ4;
+ return true;
+ }
+
+ /* File looks like a partial uncompressed XLOG file */
+ if (fname_len == XLOG_FNAME_LEN + strlen(".partial") &&
+ strcmp(filename + XLOG_FNAME_LEN, ".partial") == 0)
+ {
+ *ispartial = true;
+ *wal_compression_method = COMPRESSION_NONE;
+ return true;
+ }
+
+ /* File looks like a partial zlib compressed XLOG file */
+ if (fname_len == XLOG_FNAME_LEN + strlen(".gz.partial") &&
+ strcmp(filename + XLOG_FNAME_LEN, ".gz.partial") == 0)
+ {
+ *ispartial = true;
+ *wal_compression_method = COMPRESSION_ZLIB;
+ return true;
+ }
+
+ /* File looks like a partial LZ4 compressed XLOG file */
+ if (fname_len == XLOG_FNAME_LEN + strlen(".lz4.partial") &&
+ strcmp(filename + XLOG_FNAME_LEN, ".lz4.partial") == 0)
+ {
+ *ispartial = true;
+ *wal_compression_method = COMPRESSION_LZ4;
+ return true;
+ }
+
+ /* File does not look like something we recognise */
+ return false;
+}
+
static bool
stop_streaming(XLogRecPtr xlogpos, uint32 timeline, bool segment_finished)
{
@@ -213,33 +286,11 @@ FindStreamingStart(uint32 *tli)
{
uint32 tli;
XLogSegNo segno;
+ WalCompressionMethod wal_compression_method;
bool ispartial;
- bool iscompress;
- /*
- * Check if the filename looks like an xlog file, or a .partial file.
- */
- if (IsXLogFileName(dirent->d_name))
- {
- ispartial = false;
- iscompress = false;
- }
- else if (IsPartialXLogFileName(dirent->d_name))
- {
- ispartial = true;
- iscompress = false;
- }
- else if (IsCompressXLogFileName(dirent->d_name))
- {
- ispartial = false;
- iscompress = true;
- }
- else if (IsPartialCompressXLogFileName(dirent->d_name))
- {
- ispartial = true;
- iscompress = true;
- }
- else
+ if (!is_xlogfilename(dirent->d_name,
+ &ispartial, &wal_compression_method))
continue;
/*
@@ -250,14 +301,18 @@ FindStreamingStart(uint32 *tli)
/*
* Check that the segment has the right size, if it's supposed to be
* completed. For non-compressed segments just check the on-disk size
- * and see if it matches a completed segment. For compressed segments,
- * look at the last 4 bytes of the compressed file, which is where the
- * uncompressed size is located for gz files with a size lower than
- * 4GB, and then compare it to the size of a completed segment. The 4
- * last bytes correspond to the ISIZE member according to
- * http://www.zlib.org/rfc-gzip.html.
+ * and see if it matches a completed segment. For zlib compressed
+ * segments, look at the last 4 bytes of the compressed file, which is
+ * where the uncompressed size is located for gz files with a size
+ * lower than 4GB, and then compare it to the size of a completed
+ * segment. The 4 last bytes correspond to the ISIZE member according
+ * to http://www.zlib.org/rfc-gzip.html.
+ *
+ * For LZ4 compressed segments read the header using the exposed API
+ * and compare the uncompressed file size, stored in
+ * LZ4F_frameInfo_t{.contentSize}, to that of a completed segment.
*/
- if (!ispartial && !iscompress)
+ if (!ispartial && wal_compression_method == COMPRESSION_NONE)
{
struct stat statbuf;
char fullpath[MAXPGPATH * 2];
@@ -276,7 +331,7 @@ FindStreamingStart(uint32 *tli)
continue;
}
}
- else if (!ispartial && iscompress)
+ else if (!ispartial && wal_compression_method == COMPRESSION_ZLIB)
{
int fd;
char buf[4];
@@ -322,6 +377,80 @@ FindStreamingStart(uint32 *tli)
continue;
}
}
+ else if (!ispartial && compression_method == COMPRESSION_LZ4)
+ {
+#ifdef HAVE_LIBLZ4
+ int fd;
+ int r;
+ size_t consumed_len = LZ4F_HEADER_SIZE_MAX;
+ char buf[LZ4F_HEADER_SIZE_MAX];
+ char fullpath[MAXPGPATH * 2];
+ LZ4F_frameInfo_t frame_info = {0};
+ LZ4F_decompressionContext_t ctx = NULL;
+ LZ4F_errorCode_t status;
+
+ snprintf(fullpath, sizeof(fullpath), "%s/%s", basedir, dirent->d_name);
+
+ fd = open(fullpath, O_RDONLY | PG_BINARY, 0);
+ if (fd < 0)
+ {
+ pg_log_error("could not open file \"%s\": %m", fullpath);
+ exit(1);
+ }
+
+ r = read(fd, buf, sizeof(buf));
+ if (r != sizeof(buf))
+ {
+ if (r < 0)
+ pg_log_error("could not read file \"%s\": %m", fullpath);
+ else
+ pg_log_error("could not read file \"%s\": read %d of %zu",
+ fullpath, r, sizeof(buf));
+ exit(1);
+ }
+ close(fd);
+
+ status = LZ4F_createDecompressionContext(&ctx, LZ4F_VERSION);
+ if (LZ4F_isError(status))
+ {
+ pg_log_error("could not create LZ4 decompression context: %s",
+ LZ4F_getErrorName(status));
+ exit(1);
+ }
+
+ LZ4F_getFrameInfo(ctx, &frame_info, (void *) buf, &consumed_len);
+ if (consumed_len <= LZ4F_HEADER_SIZE_MIN ||
+ consumed_len >= LZ4F_HEADER_SIZE_MAX)
+ {
+ pg_log_warning("compressed segment file \"%s\" has incorrect header size %lu, skipping",
+ dirent->d_name, consumed_len);
+ (void) LZ4F_freeDecompressionContext(ctx);
+ continue;
+ }
+
+ if (frame_info.contentSize != WalSegSz)
+ {
+ pg_log_warning("compressed segment file \"%s\" has incorrect uncompressed size %lld, skipping",
+ dirent->d_name, frame_info.contentSize);
+ (void) LZ4F_freeDecompressionContext(ctx);
+ continue;
+ }
+
+ status = LZ4F_freeDecompressionContext(ctx);
+ if (LZ4F_isError(status))
+ {
+ pg_log_error("could not free LZ4 decompression context: %s",
+ LZ4F_getErrorName(status));
+ exit(1);
+ }
+#else
+ pg_log_error("could not check segment file \"%s\" compressed with LZ4",
+ dirent->d_name);
+ pg_log_error("this build does not support compression with %s",
+ "LZ4");
+ exit(1);
+#endif
+ }
/* Looks like a valid segment. Remember that we saw it. */
if ((segno > high_segno) ||
@@ -457,7 +586,9 @@ StreamLog(void)
stream.synchronous = synchronous;
stream.do_sync = do_sync;
stream.mark_done = false;
- stream.walmethod = CreateWalDirectoryMethod(basedir, compresslevel,
+ stream.walmethod = CreateWalDirectoryMethod(basedir,
+ compression_method,
+ compresslevel,
stream.do_sync);
stream.partial_suffix = ".partial";
stream.replication_slot = replication_slot;
@@ -510,6 +641,7 @@ main(int argc, char **argv)
{"status-interval", required_argument, NULL, 's'},
{"slot", required_argument, NULL, 'S'},
{"verbose", no_argument, NULL, 'v'},
+ {"compression-method", required_argument, NULL, 'I'},
{"compress", required_argument, NULL, 'Z'},
/* action */
{"create-slot", no_argument, NULL, 1},
@@ -595,8 +727,22 @@ main(int argc, char **argv)
case 'v':
verbose++;
break;
+ case 'I':
+ if (pg_strcasecmp(optarg, "gzip") == 0)
+ compression_method = COMPRESSION_ZLIB;
+ else if (pg_strcasecmp(optarg, "lz4") == 0)
+ compression_method = COMPRESSION_LZ4;
+ else if (pg_strcasecmp(optarg, "none") == 0)
+ compression_method = COMPRESSION_NONE;
+ else
+ {
+ pg_log_error("invalid value \"%s\" for option %s",
+ optarg, "--compress-method");
+ exit(1);
+ }
+ break;
case 'Z':
- if (!option_parse_int(optarg, "-Z/--compress", 0, 9,
+ if (!option_parse_int(optarg, "-Z/--compress", 1, 9,
&compresslevel))
exit(1);
break;
@@ -676,13 +822,43 @@ main(int argc, char **argv)
exit(1);
}
+
+ /*
+ * Compression related arguments
+ */
+ if (compression_method != COMPRESSION_NONE)
+ {
#ifndef HAVE_LIBZ
- if (compresslevel != 0)
+ if (compression_method == COMPRESSION_ZLIB)
+ {
+ pg_log_error("this build does not support compression with %s",
+ "gzip");
+ exit(1);
+ }
+#endif
+#ifndef HAVE_LIBLZ4
+ if (compression_method == COMPRESSION_LZ4)
+ {
+ pg_log_error("this build does not support compression with %s",
+ "LZ4");
+ exit(1);
+ }
+#endif
+ }
+
+ if (compression_method != COMPRESSION_ZLIB && compresslevel != 0)
{
- pg_log_error("this build does not support compression");
+ pg_log_error("can only use --compress with --compression-method=gzip");
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+ progname);
exit(1);
}
-#endif
+
+ if (compression_method == COMPRESSION_ZLIB && compresslevel == 0)
+ {
+ pg_log_info("no --compression specified, will be using the library default");
+ compresslevel = Z_DEFAULT_COMPRESSION;
+ }
/*
* Check existence of destination folder.
diff --git a/src/bin/pg_basebackup/receivelog.c b/src/bin/pg_basebackup/receivelog.c
index 72b8d9e315..2d4f660daa 100644
--- a/src/bin/pg_basebackup/receivelog.c
+++ b/src/bin/pg_basebackup/receivelog.c
@@ -109,7 +109,7 @@ open_walfile(StreamCtl *stream, XLogRecPtr startpoint)
* When streaming to tar, no file with this name will exist before, so we
* never have to verify a size.
*/
- if (stream->walmethod->compression() == 0 &&
+ if (stream->walmethod->compression_method() == COMPRESSION_NONE &&
stream->walmethod->existsfile(fn))
{
size = stream->walmethod->get_file_size(fn);
diff --git a/src/bin/pg_basebackup/t/020_pg_receivewal.pl b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
index 2da200396e..a37b92d6af 100644
--- a/src/bin/pg_basebackup/t/020_pg_receivewal.pl
+++ b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
@@ -5,7 +5,7 @@ use strict;
use warnings;
use PostgreSQL::Test::Utils;
use PostgreSQL::Test::Cluster;
-use Test::More tests => 31;
+use Test::More tests => 38;
program_help_ok('pg_receivewal');
program_version_ok('pg_receivewal');
@@ -33,6 +33,13 @@ $primary->command_fails(
$primary->command_fails(
[ 'pg_receivewal', '-D', $stream_dir, '--synchronous', '--no-sync' ],
'failure if --synchronous specified with --no-sync');
+$primary->command_fails_like(
+ [
+ 'pg_receivewal', '-D', $stream_dir, '--compression-method', 'none',
+ '--compress', '1'
+ ],
+ qr/\Qpg_receivewal: error: can only use --compress with --compression-method=gzip/,
+ 'failure if --compression-method=none specified with --compress');
# Slot creation and drop
my $slot_name = 'test';
@@ -41,7 +48,7 @@ $primary->command_ok(
'creating a replication slot');
my $slot = $primary->slot($slot_name);
is($slot->{'slot_type'}, 'physical', 'physical replication slot was created');
-is($slot->{'restart_lsn'}, '', 'restart LSN of new slot is null');
+is($slot->{'restart_lsn'}, '', 'restart LSN of new slot is null');
$primary->command_ok([ 'pg_receivewal', '--slot', $slot_name, '--drop-slot' ],
'dropping a replication slot');
is($primary->slot($slot_name)->{'slot_type'},
@@ -90,8 +97,11 @@ SKIP:
# a valid value.
$primary->command_ok(
[
- 'pg_receivewal', '-D', $stream_dir, '--verbose',
- '--endpos', $nextlsn, '--compress', '1 ',
+ 'pg_receivewal', '-D',
+ $stream_dir, '--verbose',
+ '--endpos', $nextlsn,
+ '--compression-method', 'gzip',
+ '--compress', '1 ',
'--no-loop'
],
"streaming some WAL using ZLIB compression");
@@ -128,8 +138,65 @@ SKIP:
"gzip verified the integrity of compressed WAL segments");
}
+# Check LZ4 compression if available
+SKIP:
+{
+ skip "postgres was not built with LZ4 support", 5
+ if (!check_pg_config("#define HAVE_LIBLZ4 1"));
+
+ # Generate more WAL including one completed, compressed segment.
+ $primary->psql('postgres', 'SELECT pg_switch_wal();');
+ $nextlsn =
+ $primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
+ chomp($nextlsn);
+ $primary->psql('postgres',
+ 'INSERT INTO test_table VALUES (generate_series(201,300));');
+
+ # Stream up to the given position
+ $primary->command_ok(
+ [
+ 'pg_receivewal', '-D',
+ $stream_dir, '--verbose',
+ '--endpos', $nextlsn,
+ '--no-loop', '--compression-method',
+ 'lz4'
+ ],
+ 'streaming some WAL using --compression-method=lz4');
+
+ # Verify that the stored files are generated with their expected
+ # names.
+ my @lz4_wals = glob "$stream_dir/*.lz4";
+ is(scalar(@lz4_wals), 1,
+ "one WAL segment compressed with LZ4 was created");
+ my @lz4_partial_wals = glob "$stream_dir/*.lz4.partial";
+ is(scalar(@lz4_partial_wals),
+ 1, "one partial WAL segment compressed with LZ4 was created");
+
+ # Verify that the start streaming position is computed correctly by
+ # comparing it with the partial file generated previously. The name
+ # of the previous partial, now-completed WAL segment is updated, keeping
+ # its base number.
+ $partial_wals[0] =~ s/(\.gz)?\.partial$/.lz4/;
+ is($lz4_wals[0] eq $partial_wals[0],
+ 1, "one partial WAL segment is now completed");
+ # Update the list of partial wals with the current one.
+ @partial_wals = @lz4_partial_wals;
+
+ # Check the integrity of the completed segment, if LZ4 is an available
+ # command.
+ my $lz4 = $ENV{LZ4};
+ skip "program lz4 is not found in your system", 1
+ if ( !defined $lz4
+ || $lz4 eq ''
+ || system_log($lz4, '--version') != 0);
+
+ my $lz4_is_valid = system_log($lz4, '-t', @lz4_wals);
+ is($lz4_is_valid, 0,
+ "lz4 verified the integrity of compressed WAL segments");
+}
+
# Verify that the start streaming position is computed and that the value is
-# correct regardless of whether ZLIB is available.
+# correct regardless of whether any compression is available.
$primary->psql('postgres', 'SELECT pg_switch_wal();');
$nextlsn =
$primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
@@ -142,7 +209,7 @@ $primary->command_ok(
],
"streaming some WAL");
-$partial_wals[0] =~ s/(\.gz)?.partial//;
+$partial_wals[0] =~ s/(\.gz|\.lz4)?.partial//;
ok(-e $partial_wals[0], "check that previously partial WAL is now complete");
# Permissions on WAL files should be default
diff --git a/src/bin/pg_basebackup/walmethods.c b/src/bin/pg_basebackup/walmethods.c
index 8695647db4..bc26a71b75 100644
--- a/src/bin/pg_basebackup/walmethods.c
+++ b/src/bin/pg_basebackup/walmethods.c
@@ -17,6 +17,10 @@
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
+
+#ifdef HAVE_LIBLZ4
+#include <lz4frame.h>
+#endif
#ifdef HAVE_LIBZ
#include <zlib.h>
#endif
@@ -30,6 +34,9 @@
/* Size of zlib buffer for .tar.gz */
#define ZLIB_OUT_SIZE 4096
+/* Size of lz4 input chunk for .lz4 */
+#define LZ4_IN_SIZE 4096
+
/*-------------------------------------------------------------------------
* WalDirectoryMethod - write wal to a directory looking like pg_wal
*-------------------------------------------------------------------------
@@ -41,6 +48,7 @@
typedef struct DirectoryMethodData
{
char *basedir;
+ WalCompressionMethod compression_method;
int compression;
bool sync;
} DirectoryMethodData;
@@ -59,6 +67,11 @@ typedef struct DirectoryMethodFile
#ifdef HAVE_LIBZ
gzFile gzfp;
#endif
+#ifdef HAVE_LIBLZ4
+ LZ4F_compressionContext_t ctx;
+ size_t lz4bufsize;
+ void *lz4buf;
+#endif
} DirectoryMethodFile;
static const char *
@@ -74,7 +87,9 @@ dir_get_file_name(const char *pathname, const char *temp_suffix)
char *filename = pg_malloc0(MAXPGPATH * sizeof(char));
snprintf(filename, MAXPGPATH, "%s%s%s",
- pathname, dir_data->compression > 0 ? ".gz" : "",
+ pathname,
+ dir_data->compression_method == COMPRESSION_ZLIB ? ".gz" :
+ dir_data->compression_method == COMPRESSION_LZ4 ? ".lz4" : "",
temp_suffix ? temp_suffix : "");
return filename;
@@ -90,6 +105,11 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
#ifdef HAVE_LIBZ
gzFile gzfp = NULL;
#endif
+#ifdef HAVE_LIBLZ4
+ LZ4F_compressionContext_t ctx = NULL;
+ size_t lz4bufsize = 0;
+ void *lz4buf = NULL;
+#endif
filename = dir_get_file_name(pathname, temp_suffix);
snprintf(tmppath, sizeof(tmppath), "%s/%s",
@@ -107,7 +127,7 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
return NULL;
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
{
gzfp = gzdopen(fd, "wb");
if (gzfp == NULL)
@@ -124,9 +144,59 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
}
}
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ LZ4F_preferences_t lz4preferences = {0};
+ size_t ctx_out;
+ size_t header_size;
+
+ /*
+ * Set all the preferences to default but do note contentSize. It will
+ * be needed in FindStreamingStart.
+ */
+ memset(&lz4preferences, 0, sizeof(LZ4F_frameInfo_t));
+ lz4preferences.frameInfo.contentSize = (unsigned long long) WalSegSz;
+ ctx_out = LZ4F_createCompressionContext(&ctx, LZ4F_VERSION);
+ lz4bufsize = LZ4F_compressBound(LZ4_IN_SIZE, &lz4preferences);
+ if (LZ4F_isError(ctx_out))
+ {
+ close(fd);
+ return NULL;
+ }
+
+ lz4buf = pg_malloc0(lz4bufsize);
+
+ /* add the header */
+ header_size = LZ4F_compressBegin(ctx, lz4buf, lz4bufsize, &lz4preferences);
+ if (LZ4F_isError(header_size))
+ {
+ pg_free(lz4buf);
+ close(fd);
+ return NULL;
+ }
+
+ errno = 0;
+ if (write(fd, lz4buf, header_size) != header_size)
+ {
+ int save_errno = errno;
+
+ (void) LZ4F_compressEnd(ctx, lz4buf, lz4bufsize, NULL);
+ (void) LZ4F_freeCompressionContext(ctx);
+ pg_free(lz4buf);
+ close(fd);
+
+ /*
+ * If write didn't set errno, assume problem is no disk space.
+ */
+ errno = save_errno ? save_errno : ENOSPC;
+ return NULL;
+ }
+ }
+#endif
/* Do pre-padding on non-compressed files */
- if (pad_to_size && dir_data->compression == 0)
+ if (pad_to_size && dir_data->compression_method == COMPRESSION_NONE)
{
PGAlignedXLogBlock zerobuf;
int bytes;
@@ -171,9 +241,19 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
fsync_parent_path(tmppath) != 0)
{
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
gzclose(gzfp);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ (void) LZ4F_compressEnd(ctx, lz4buf, lz4bufsize, NULL);
+ (void) LZ4F_freeCompressionContext(ctx);
+ pg_free(lz4buf);
+ close(fd);
+ }
+ else
#endif
close(fd);
return NULL;
@@ -182,9 +262,18 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
f = pg_malloc0(sizeof(DirectoryMethodFile));
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
f->gzfp = gzfp;
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ f->ctx = ctx;
+ f->lz4buf = lz4buf;
+ f->lz4bufsize = lz4bufsize;
+ }
+#endif
+
f->fd = fd;
f->currpos = 0;
f->pathname = pg_strdup(pathname);
@@ -204,9 +293,46 @@ dir_write(Walfile f, const void *buf, size_t count)
Assert(f != NULL);
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
r = (ssize_t) gzwrite(df->gzfp, buf, count);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ size_t chunk;
+ size_t remaining;
+ const void *inbuf = buf;
+
+ remaining = count;
+ while (remaining > 0)
+ {
+ size_t compressed;
+
+ if (remaining > LZ4_IN_SIZE)
+ chunk = LZ4_IN_SIZE;
+ else
+ chunk = remaining;
+
+ remaining -= chunk;
+ compressed = LZ4F_compressUpdate(df->ctx,
+ df->lz4buf, df->lz4bufsize,
+ inbuf, chunk,
+ NULL);
+
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+
+ inbuf = ((char *) inbuf) + chunk;
+ }
+
+ /* Our caller keeps track of the uncompressed size. */
+ r = (ssize_t) count;
+ }
+ else
#endif
r = write(df->fd, buf, count);
if (r > 0)
@@ -234,9 +360,76 @@ dir_close(Walfile f, WalCloseMethod method)
Assert(f != NULL);
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
r = gzclose(df->gzfp);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ size_t compressed;
+
+ /*
+ * When closing an incomplete LZ4 compressed WAL segment file, since
+ * there are no means available to resume in a later stage. The file
+ * should be considered as invalid. Conversely, when closing a complete
+ * WAL segment all the care is taken to close it properly.
+ */
+ if (dir_get_current_pos(f) == WalSegSz)
+ {
+ compressed = LZ4F_compressEnd(df->ctx,
+ df->lz4buf, df->lz4bufsize,
+ NULL);
+
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+ }
+ else
+ {
+ /*
+ * Instead of simply preparing the compression context for future
+ * work, a best effort attempt is made to write the file. Any data
+ * still left in the compression buffer is written. The contentSize
+ * information stored in the header is overwritten, for the benefit
+ * of any inspecting application. See FindStreamingStart() for an
+ * example of such a case. No effort is made to verify that the new
+ * header has not overgrown, overwritting existing data.
+ */
+ LZ4F_preferences_t lz4preferences = {0};
+ size_t unused pg_attribute_unused();
+ size_t header_size;
+ off_t curr;
+
+ compressed = LZ4F_flush(df->ctx,
+ df->lz4buf, df->lz4bufsize,
+ NULL);
+
+ unused = write(df->fd, df->lz4buf, compressed);
+
+ header_size = LZ4F_compressBegin(df->ctx,
+ df->lz4buf, df->lz4bufsize,
+ &lz4preferences);
+
+ if ((curr = lseek(df->fd, 0, SEEK_CUR)) < 0 ||
+ lseek(df->fd, 0, SEEK_SET) < 0 ||
+ write(df->fd, df->lz4buf, header_size) != header_size ||
+ lseek(df->fd, curr, SEEK_SET) < 0)
+ return -1;
+
+ compressed = LZ4F_compressEnd(df->ctx,
+ df->lz4buf, df->lz4bufsize,
+ NULL);
+
+ if (LZ4F_isError(compressed))
+ return -1;
+ }
+
+ r = close(df->fd);
+ }
+ else
#endif
r = close(df->fd);
@@ -291,6 +484,12 @@ dir_close(Walfile f, WalCloseMethod method)
}
}
+#ifdef HAVE_LIBLZ4
+ pg_free(df->lz4buf);
+ /* supports free on NULL */
+ LZ4F_freeCompressionContext(df->ctx);
+#endif
+
pg_free(df->pathname);
pg_free(df->fullpath);
if (df->temp_suffix)
@@ -309,12 +508,27 @@ dir_sync(Walfile f)
return 0;
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
{
if (gzflush(((DirectoryMethodFile *) f)->gzfp, Z_SYNC_FLUSH) != Z_OK)
return -1;
}
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ DirectoryMethodFile *df = (DirectoryMethodFile *) f;
+ size_t compressed;
+
+ /* Flush any internal buffers */
+ compressed = LZ4F_flush(df->ctx, df->lz4buf, df->lz4bufsize, NULL);
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+ }
+#endif
return fsync(((DirectoryMethodFile *) f)->fd);
}
@@ -334,10 +548,10 @@ dir_get_file_size(const char *pathname)
return statbuf.st_size;
}
-static int
-dir_compression(void)
+static WalCompressionMethod
+dir_compression_method(void)
{
- return dir_data->compression;
+ return dir_data->compression_method;
}
static bool
@@ -373,7 +587,9 @@ dir_finish(void)
WalWriteMethod *
-CreateWalDirectoryMethod(const char *basedir, int compression, bool sync)
+CreateWalDirectoryMethod(const char *basedir,
+ WalCompressionMethod compression_method,
+ int compression, bool sync)
{
WalWriteMethod *method;
@@ -383,7 +599,7 @@ CreateWalDirectoryMethod(const char *basedir, int compression, bool sync)
method->get_current_pos = dir_get_current_pos;
method->get_file_size = dir_get_file_size;
method->get_file_name = dir_get_file_name;
- method->compression = dir_compression;
+ method->compression_method = dir_compression_method;
method->close = dir_close;
method->sync = dir_sync;
method->existsfile = dir_existsfile;
@@ -391,6 +607,7 @@ CreateWalDirectoryMethod(const char *basedir, int compression, bool sync)
method->getlasterror = dir_getlasterror;
dir_data = pg_malloc0(sizeof(DirectoryMethodData));
+ dir_data->compression_method = compression_method;
dir_data->compression = compression;
dir_data->basedir = pg_strdup(basedir);
dir_data->sync = sync;
@@ -424,6 +641,7 @@ typedef struct TarMethodData
{
char *tarfilename;
int fd;
+ WalCompressionMethod compression_method;
int compression;
bool sync;
TarMethodFile *currentfile;
@@ -731,10 +949,10 @@ tar_get_file_size(const char *pathname)
return -1;
}
-static int
-tar_compression(void)
+static WalCompressionMethod
+tar_compression_method(void)
{
- return tar_data->compression;
+ return tar_data->compression_method;
}
static off_t
@@ -1031,8 +1249,16 @@ tar_finish(void)
return true;
}
+/*
+ * The argument compression_method is currently ignored. It is in place for
+ * symmetry with CreateWalDirectoryMethod which uses it for distinguishing
+ * between the different compression methods. CreateWalTarMethod and its family
+ * of functions handle only zlib compression.
+ */
WalWriteMethod *
-CreateWalTarMethod(const char *tarbase, int compression, bool sync)
+CreateWalTarMethod(const char *tarbase,
+ WalCompressionMethod compression_method,
+ int compression, bool sync)
{
WalWriteMethod *method;
const char *suffix = (compression != 0) ? ".tar.gz" : ".tar";
@@ -1043,7 +1269,7 @@ CreateWalTarMethod(const char *tarbase, int compression, bool sync)
method->get_current_pos = tar_get_current_pos;
method->get_file_size = tar_get_file_size;
method->get_file_name = tar_get_file_name;
- method->compression = tar_compression;
+ method->compression_method = tar_compression_method;
method->close = tar_close;
method->sync = tar_sync;
method->existsfile = tar_existsfile;
@@ -1054,6 +1280,7 @@ CreateWalTarMethod(const char *tarbase, int compression, bool sync)
tar_data->tarfilename = pg_malloc0(strlen(tarbase) + strlen(suffix) + 1);
sprintf(tar_data->tarfilename, "%s%s", tarbase, suffix);
tar_data->fd = -1;
+ tar_data->compression_method = compression_method;
tar_data->compression = compression;
tar_data->sync = sync;
#ifdef HAVE_LIBZ
diff --git a/src/bin/pg_basebackup/walmethods.h b/src/bin/pg_basebackup/walmethods.h
index 4abdfd8333..3e378c87b6 100644
--- a/src/bin/pg_basebackup/walmethods.h
+++ b/src/bin/pg_basebackup/walmethods.h
@@ -19,6 +19,13 @@ typedef enum
CLOSE_NO_RENAME
} WalCloseMethod;
+typedef enum
+{
+ COMPRESSION_LZ4,
+ COMPRESSION_ZLIB,
+ COMPRESSION_NONE
+} WalCompressionMethod;
+
/*
* A WalWriteMethod structure represents the different methods used
* to write the streaming WAL as it's received.
@@ -58,8 +65,8 @@ struct WalWriteMethod
*/
char *(*get_file_name) (const char *pathname, const char *temp_suffix);
- /* Return the level of compression */
- int (*compression) (void);
+ /* Returns the compression method */
+ WalCompressionMethod (*compression_method) (void);
/*
* Write count number of bytes to the file, and return the number of bytes
@@ -95,8 +102,11 @@ struct WalWriteMethod
* not all those required for pg_receivewal)
*/
WalWriteMethod *CreateWalDirectoryMethod(const char *basedir,
+ WalCompressionMethod compression_method,
int compression, bool sync);
-WalWriteMethod *CreateWalTarMethod(const char *tarbase, int compression, bool sync);
+WalWriteMethod *CreateWalTarMethod(const char *tarbase,
+ WalCompressionMethod compression_method,
+ int compression, bool sync);
/* Cleanup routines for previously-created methods */
void FreeWalDirectoryMethod(void);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 7bbbb34e2f..da6ac8ed83 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2858,6 +2858,7 @@ WaitEventTimeout
WaitPMResult
WalCloseMethod
WalCompression
+WalCompressionMethod
WalLevel
WalRcvData
WalRcvExecResult
--
2.25.1
Re: Teach pg_receivewal to use lz4 compression
On Fri, Oct 29, 2021 at 09:45:41AM +0000, gkokolatos@pm.me wrote:
On Saturday, September 18th, 2021 at 8:18 AM, Michael Paquier <michael@paquier.xyz> wrote:
We don't really care about contentSize as long as a segment is not
completed. Rather than filling contentSize all the time we write
something, we'd better update frameInfo once the segment is
completed and closed. That would also take take of the error as this
is not checked if contentSize is 0. It seems to me that we should
fill in the information when doing a CLOSE_NORMAL.Thank you for the comment. I think that the opposite should be done. At the time
that the file is closed, the header is already written to disk. We have no way
to know that is not. If we need to go back to refill the information, we will
have to ask for the API to produce a new header. There is little guarantee that
the header size will be the same and as a consequence we will have to shift
the actual data around.
Why would the header size change between the moment the segment is
begun and it is finished? We could store it in memory and write it
again when the segment is closed instead, even if it means to fseek()
back to the beginning of the file once the segment is completed.
Storing WalSegSz from the moment a segment is opened makes the code
weaker to SIGINTs and the kind, so this does not fix the problem I
mentioned previously :/
In the attached, the header is rewritten only when closing an incomplete
segment. For all intents and purposes that segment is not usable. However there
might be custom scripts that might want to attempt to parse even an otherwise
unusable file.A different and easier approach would be to simply prepare the LZ4 context for
future actions and simply ignore the file.
I am not sure what you mean by "ignore" here. Do you mean to store 0
in contentSize when opening the segment and rewriting again the header
once the segment is completed?
--
Michael
Re: Teach pg_receivewal to use lz4 compression
On Fri, Oct 29, 2021 at 08:38:33PM +0900, Michael Paquier wrote:
Why would the header size change between the moment the segment is
begun and it is finished? We could store it in memory and write it
again when the segment is closed instead, even if it means to fseek()
back to the beginning of the file once the segment is completed.
Storing WalSegSz from the moment a segment is opened makes the code
weaker to SIGINTs and the kind, so this does not fix the problem I
mentioned previously :/
I got to think more on this one, and another argument against storing
an incorrect contentSize while the segment is not completed would
break the case of partial segments with --synchronous, where we should
still be able to recover as much data flushed as possible. Like zlib,
where one has to complete the partial segment with zeros after
decompression until the WAL segment size is reached, we should be able
to support that with LZ4. (I have saved some customer data in the
past thanks to this property, btw.)
It is proves to be too fancy to rewrite the header with a correct
contentSize once the segment is completed, another way would be to
enforce a decompression of each segment in-memory. The advantage of
this method is that we would be a maximum portable. For example, if
one begins to use pg_receivewal on an archive directory where we used
an archive_command, we would be able to grab the starting LSN. That's
more costly of course, but the LZ4 protocol does not make that easy
either with its chunk protocol. By the way, you are right that we
should worry about the variability in size of the header as we only
have the guarantee that it can be within a give window. I missed
that and lz4frame.h mentions that around LZ4F_headerSize :/
It would be good to test with many segments, but could we think about
just relying on LZ4F_decompress() with a frame and compute the
decompressed size by ourselves? At least that will never break, and
that would work in all the cases aimed by pg_receivewal.
--
Michael
Re: Teach pg_receivewal to use lz4 compression
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Monday, November 1st, 2021 at 9:09 AM, Michael Paquier <michael@paquier.xyz> wrote:
On Fri, Oct 29, 2021 at 08:38:33PM +0900, Michael Paquier wrote:
It would be good to test with many segments, but could we think about
just relying on LZ4F_decompress() with a frame and compute the
decompressed size by ourselves? At least that will never break, and
that would work in all the cases aimed by pg_receivewal.
Agreed.
I have already started on v8 of the patch with that technique. I should
be able to update the thread soon.
Show quoted text
Michael
Re: Teach pg_receivewal to use lz4 compression
On Mon, Nov 01, 2021 at 08:39:59AM +0000, gkokolatos@pm.me wrote:
Agreed.
I have already started on v8 of the patch with that technique. I should
be able to update the thread soon.
Nice, thanks!
--
Michael
Re: Teach pg_receivewal to use lz4 compression
On Tue, Nov 02, 2021 at 07:27:50AM +0900, Michael Paquier wrote:
On Mon, Nov 01, 2021 at 08:39:59AM +0000, gkokolatos@pm.me wrote:
Agreed.
I have already started on v8 of the patch with that technique. I should
be able to update the thread soon.Nice, thanks!
By the way, I was reading the last version of the patch today, and
it seems to me that we could make the commit history if we split the
patch into two parts:
- One that introduces the new option --compression-method and
is_xlogfilename(), while reworking --compress (including documentation
changes).
- One to have LZ4 support.
v7 has been using "gzip" for the option name, but I think that it
would be more consistent to use "zlib" instead.
--
Michael
Re: Teach pg_receivewal to use lz4 compression
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Tuesday, November 2nd, 2021 at 9:51 AM, Michael Paquier <michael@paquier.xyz> wrote:
On Tue, Nov 02, 2021 at 07:27:50AM +0900, Michael Paquier wrote:
On Mon, Nov 01, 2021 at 08:39:59AM +0000, gkokolatos@pm.me wrote:
Agreed.
I have already started on v8 of the patch with that technique. I should
be able to update the thread soon.Nice, thanks!
A pleasure. Please find it in the attached v8-0002 patch.
By the way, I was reading the last version of the patch today, and
it seems to me that we could make the commit history if we split the
patch into two parts:
- One that introduces the new option --compression-method and
is_xlogfilename(), while reworking --compress (including documentation
changes).
- One to have LZ4 support.
Agreed.
v7 has been using "gzip" for the option name, but I think that it
would be more consistent to use "zlib" instead.
Done.
Cheers,
//Georgios
Show quoted text
--
Michael
Attachments:
v8-0001-Teach-pg_receivewal-to-use-LZ4-compression.patchtext/x-patch; name=v8-0001-Teach-pg_receivewal-to-use-LZ4-compression.patchDownload
From dc33f7ea2930dfc5a7bace52eb0086c759a7d86e Mon Sep 17 00:00:00 2001
From: Georgios Kokolatos <gkokolatos@pm.me>
Date: Tue, 2 Nov 2021 10:39:42 +0000
Subject: [PATCH v8 1/2] Refactor pg_receivewal in preparation for introducing
lz4 compression
The program pg_receivewal can use gzip compression to store the received WAL.
The option `--compress` with a value [1, 9] was used to denote that gzip
compression was required. When `--compress` with a value of `0` was used, then
no compression would take place.
This commit introduces a new option, `--compression-method`. Valid values are
[none|zlib]. The option `--compress` requires for `--compression-method` with
value other than `none`. Also `--compress=0` now returns an error.
Under the hood, there are no surprising changes. A new enum WalCompressionMethod
has been introduced and is used throughout the relevant codepaths to explicitly
note which compression method to use.
Last, the macros IsXLogFileName and friends, have been replaced by the function
is_xlogfilename(). This will allow for easier expansion of the available
compression methods that can be recognised.
---
doc/src/sgml/ref/pg_receivewal.sgml | 24 ++-
src/bin/pg_basebackup/pg_basebackup.c | 7 +-
src/bin/pg_basebackup/pg_receivewal.c | 164 +++++++++++++------
src/bin/pg_basebackup/receivelog.c | 2 +-
src/bin/pg_basebackup/t/020_pg_receivewal.pl | 16 +-
src/bin/pg_basebackup/walmethods.c | 51 ++++--
src/bin/pg_basebackup/walmethods.h | 15 +-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 200 insertions(+), 80 deletions(-)
diff --git a/doc/src/sgml/ref/pg_receivewal.sgml b/doc/src/sgml/ref/pg_receivewal.sgml
index 9fde2fd2ef..f95cffcd5e 100644
--- a/doc/src/sgml/ref/pg_receivewal.sgml
+++ b/doc/src/sgml/ref/pg_receivewal.sgml
@@ -263,15 +263,31 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--compression-method=<replaceable class="parameter">level</replaceable></option></term>
+ <listitem>
+ <para>
+ Enables compression of write-ahead logs using the specified method.
+ Supported values <literal>zlib</literal>,
+ and <literal>none</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-Z <replaceable class="parameter">level</replaceable></option></term>
<term><option>--compress=<replaceable class="parameter">level</replaceable></option></term>
<listitem>
<para>
- Enables gzip compression of write-ahead logs, and specifies the
- compression level (0 through 9, 0 being no compression and 9 being best
- compression). The suffix <filename>.gz</filename> will
- automatically be added to all filenames.
+ Specifies the compression level (<literal>1</literal> through
+ <literal>9</literal>, <literal>1</literal> being worst compression
+ and <literal>9</literal> being best compression) for
+ <application>zlib</application> compressed WAL segments.
+ </para>
+
+ <para>
+ This option requires <option>--compression-method</option> to be
+ specified with <literal>zlib</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c
index 27ee6394cf..cdea3711b7 100644
--- a/src/bin/pg_basebackup/pg_basebackup.c
+++ b/src/bin/pg_basebackup/pg_basebackup.c
@@ -555,10 +555,13 @@ LogStreamerMain(logstreamer_param *param)
stream.replication_slot = replication_slot;
if (format == 'p')
- stream.walmethod = CreateWalDirectoryMethod(param->xlog, 0,
+ stream.walmethod = CreateWalDirectoryMethod(param->xlog,
+ COMPRESSION_NONE, 0,
stream.do_sync);
else
- stream.walmethod = CreateWalTarMethod(param->xlog, compresslevel,
+ stream.walmethod = CreateWalTarMethod(param->xlog,
+ COMPRESSION_NONE, /* ignored */
+ compresslevel,
stream.do_sync);
if (!ReceiveXlogStream(param->bgconn, &stream))
diff --git a/src/bin/pg_basebackup/pg_receivewal.c b/src/bin/pg_basebackup/pg_receivewal.c
index 04ba20b197..9641f4a2f4 100644
--- a/src/bin/pg_basebackup/pg_receivewal.c
+++ b/src/bin/pg_basebackup/pg_receivewal.c
@@ -32,6 +32,9 @@
/* Time to sleep between reconnection attempts */
#define RECONNECT_SLEEP_TIME 5
+/* This is just the redefinition of a libz constant */
+#define Z_DEFAULT_COMPRESSION (-1)
+
/* Global options */
static char *basedir = NULL;
static int verbose = 0;
@@ -45,6 +48,7 @@ static bool do_drop_slot = false;
static bool do_sync = true;
static bool synchronous = false;
static char *replication_slot = NULL;
+static WalCompressionMethod compression_method = COMPRESSION_NONE;
static XLogRecPtr endpos = InvalidXLogRecPtr;
@@ -63,16 +67,6 @@ disconnect_atexit(void)
PQfinish(conn);
}
-/* Routines to evaluate segment file format */
-#define IsCompressXLogFileName(fname) \
- (strlen(fname) == XLOG_FNAME_LEN + strlen(".gz") && \
- strspn(fname, "0123456789ABCDEF") == XLOG_FNAME_LEN && \
- strcmp((fname) + XLOG_FNAME_LEN, ".gz") == 0)
-#define IsPartialCompressXLogFileName(fname) \
- (strlen(fname) == XLOG_FNAME_LEN + strlen(".gz.partial") && \
- strspn(fname, "0123456789ABCDEF") == XLOG_FNAME_LEN && \
- strcmp((fname) + XLOG_FNAME_LEN, ".gz.partial") == 0)
-
static void
usage(void)
{
@@ -92,7 +86,9 @@ usage(void)
printf(_(" --synchronous flush write-ahead log immediately after writing\n"));
printf(_(" -v, --verbose output verbose messages\n"));
printf(_(" -V, --version output version information, then exit\n"));
- printf(_(" -Z, --compress=0-9 compress logs with given compression level\n"));
+ printf(_(" --compression-method=METHOD\n"
+ " method to compress logs\n"));
+ printf(_(" -Z, --compress=1-9 compress logs with given compression level\n"));
printf(_(" -?, --help show this help, then exit\n"));
printf(_("\nConnection options:\n"));
printf(_(" -d, --dbname=CONNSTR connection string\n"));
@@ -108,6 +104,61 @@ usage(void)
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
}
+
+/*
+ * Check if the filename looks like an xlog file. Also note if it is partial
+ * and/or compressed file.
+ */
+static bool
+is_xlogfilename(const char *filename, bool *ispartial,
+ WalCompressionMethod *wal_compression_method)
+{
+ size_t fname_len = strlen(filename);
+ size_t xlog_pattern_len = strspn(filename, "0123456789ABCDEF");
+
+ /* File does not look like a XLOG file */
+ if (xlog_pattern_len != XLOG_FNAME_LEN)
+ return false;
+
+ /* File looks like a complete uncompressed XLOG file */
+ if (fname_len == XLOG_FNAME_LEN)
+ {
+ *ispartial = false;
+ *wal_compression_method = COMPRESSION_NONE;
+ return true;
+ }
+
+ /* File looks like a complete zlib compressed XLOG file */
+ if (fname_len == XLOG_FNAME_LEN + strlen(".gz") &&
+ strcmp(filename + XLOG_FNAME_LEN, ".gz") == 0)
+ {
+ *ispartial = false;
+ *wal_compression_method = COMPRESSION_ZLIB;
+ return true;
+ }
+
+ /* File looks like a partial uncompressed XLOG file */
+ if (fname_len == XLOG_FNAME_LEN + strlen(".partial") &&
+ strcmp(filename + XLOG_FNAME_LEN, ".partial") == 0)
+ {
+ *ispartial = true;
+ *wal_compression_method = COMPRESSION_NONE;
+ return true;
+ }
+
+ /* File looks like a partial zlib compressed XLOG file */
+ if (fname_len == XLOG_FNAME_LEN + strlen(".gz.partial") &&
+ strcmp(filename + XLOG_FNAME_LEN, ".gz.partial") == 0)
+ {
+ *ispartial = true;
+ *wal_compression_method = COMPRESSION_ZLIB;
+ return true;
+ }
+
+ /* File does not look like something we recognise */
+ return false;
+}
+
static bool
stop_streaming(XLogRecPtr xlogpos, uint32 timeline, bool segment_finished)
{
@@ -213,33 +264,11 @@ FindStreamingStart(uint32 *tli)
{
uint32 tli;
XLogSegNo segno;
+ WalCompressionMethod wal_compression_method;
bool ispartial;
- bool iscompress;
- /*
- * Check if the filename looks like an xlog file, or a .partial file.
- */
- if (IsXLogFileName(dirent->d_name))
- {
- ispartial = false;
- iscompress = false;
- }
- else if (IsPartialXLogFileName(dirent->d_name))
- {
- ispartial = true;
- iscompress = false;
- }
- else if (IsCompressXLogFileName(dirent->d_name))
- {
- ispartial = false;
- iscompress = true;
- }
- else if (IsPartialCompressXLogFileName(dirent->d_name))
- {
- ispartial = true;
- iscompress = true;
- }
- else
+ if (!is_xlogfilename(dirent->d_name,
+ &ispartial, &wal_compression_method))
continue;
/*
@@ -250,14 +279,14 @@ FindStreamingStart(uint32 *tli)
/*
* Check that the segment has the right size, if it's supposed to be
* completed. For non-compressed segments just check the on-disk size
- * and see if it matches a completed segment. For compressed segments,
- * look at the last 4 bytes of the compressed file, which is where the
- * uncompressed size is located for gz files with a size lower than
- * 4GB, and then compare it to the size of a completed segment. The 4
- * last bytes correspond to the ISIZE member according to
- * http://www.zlib.org/rfc-gzip.html.
+ * and see if it matches a completed segment. For zlib compressed
+ * segments, look at the last 4 bytes of the compressed file, which is
+ * where the uncompressed size is located for gz files with a size
+ * lower than 4GB, and then compare it to the size of a completed
+ * segment. The 4 last bytes correspond to the ISIZE member according
+ * to http://www.zlib.org/rfc-gzip.html.
*/
- if (!ispartial && !iscompress)
+ if (!ispartial && wal_compression_method == COMPRESSION_NONE)
{
struct stat statbuf;
char fullpath[MAXPGPATH * 2];
@@ -276,7 +305,7 @@ FindStreamingStart(uint32 *tli)
continue;
}
}
- else if (!ispartial && iscompress)
+ else if (!ispartial && wal_compression_method == COMPRESSION_ZLIB)
{
int fd;
char buf[4];
@@ -457,7 +486,9 @@ StreamLog(void)
stream.synchronous = synchronous;
stream.do_sync = do_sync;
stream.mark_done = false;
- stream.walmethod = CreateWalDirectoryMethod(basedir, compresslevel,
+ stream.walmethod = CreateWalDirectoryMethod(basedir,
+ compression_method,
+ compresslevel,
stream.do_sync);
stream.partial_suffix = ".partial";
stream.replication_slot = replication_slot;
@@ -510,6 +541,7 @@ main(int argc, char **argv)
{"status-interval", required_argument, NULL, 's'},
{"slot", required_argument, NULL, 'S'},
{"verbose", no_argument, NULL, 'v'},
+ {"compression-method", required_argument, NULL, 'I'},
{"compress", required_argument, NULL, 'Z'},
/* action */
{"create-slot", no_argument, NULL, 1},
@@ -595,8 +627,20 @@ main(int argc, char **argv)
case 'v':
verbose++;
break;
+ case 'I':
+ if (pg_strcasecmp(optarg, "zlib") == 0)
+ compression_method = COMPRESSION_ZLIB;
+ else if (pg_strcasecmp(optarg, "none") == 0)
+ compression_method = COMPRESSION_NONE;
+ else
+ {
+ pg_log_error("invalid value \"%s\" for option %s",
+ optarg, "--compress-method");
+ exit(1);
+ }
+ break;
case 'Z':
- if (!option_parse_int(optarg, "-Z/--compress", 0, 9,
+ if (!option_parse_int(optarg, "-Z/--compress", 1, 9,
&compresslevel))
exit(1);
break;
@@ -676,13 +720,35 @@ main(int argc, char **argv)
exit(1);
}
+
+ /*
+ * Compression related arguments
+ */
+ if (compression_method != COMPRESSION_NONE)
+ {
#ifndef HAVE_LIBZ
- if (compresslevel != 0)
+ if (compression_method == COMPRESSION_ZLIB)
+ {
+ pg_log_error("this build does not support compression with %s",
+ "gzip");
+ exit(1);
+ }
+#endif
+ }
+
+ if (compression_method != COMPRESSION_ZLIB && compresslevel != 0)
{
- pg_log_error("this build does not support compression");
+ pg_log_error("can only use --compress with --compression-method=zlib");
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+ progname);
exit(1);
}
-#endif
+
+ if (compression_method == COMPRESSION_ZLIB && compresslevel == 0)
+ {
+ pg_log_info("no --compression specified, will be using the library default");
+ compresslevel = Z_DEFAULT_COMPRESSION;
+ }
/*
* Check existence of destination folder.
diff --git a/src/bin/pg_basebackup/receivelog.c b/src/bin/pg_basebackup/receivelog.c
index 72b8d9e315..2d4f660daa 100644
--- a/src/bin/pg_basebackup/receivelog.c
+++ b/src/bin/pg_basebackup/receivelog.c
@@ -109,7 +109,7 @@ open_walfile(StreamCtl *stream, XLogRecPtr startpoint)
* When streaming to tar, no file with this name will exist before, so we
* never have to verify a size.
*/
- if (stream->walmethod->compression() == 0 &&
+ if (stream->walmethod->compression_method() == COMPRESSION_NONE &&
stream->walmethod->existsfile(fn))
{
size = stream->walmethod->get_file_size(fn);
diff --git a/src/bin/pg_basebackup/t/020_pg_receivewal.pl b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
index ab05f9e91e..56c4a0d2af 100644
--- a/src/bin/pg_basebackup/t/020_pg_receivewal.pl
+++ b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
@@ -5,7 +5,7 @@ use strict;
use warnings;
use PostgreSQL::Test::Utils;
use PostgreSQL::Test::Cluster;
-use Test::More tests => 35;
+use Test::More tests => 37;
program_help_ok('pg_receivewal');
program_version_ok('pg_receivewal');
@@ -33,6 +33,13 @@ $primary->command_fails(
$primary->command_fails(
[ 'pg_receivewal', '-D', $stream_dir, '--synchronous', '--no-sync' ],
'failure if --synchronous specified with --no-sync');
+$primary->command_fails_like(
+ [
+ 'pg_receivewal', '-D', $stream_dir, '--compression-method', 'none',
+ '--compress', '1'
+ ],
+ qr/\Qpg_receivewal: error: can only use --compress with --compression-method=zlib/,
+ 'failure if --compression-method=none specified with --compress');
# Slot creation and drop
my $slot_name = 'test';
@@ -90,8 +97,11 @@ SKIP:
# a valid value.
$primary->command_ok(
[
- 'pg_receivewal', '-D', $stream_dir, '--verbose',
- '--endpos', $nextlsn, '--compress', '1 ',
+ 'pg_receivewal', '-D',
+ $stream_dir, '--verbose',
+ '--endpos', $nextlsn,
+ '--compression-method', 'zlib',
+ '--compress', '1 ',
'--no-loop'
],
"streaming some WAL using ZLIB compression");
diff --git a/src/bin/pg_basebackup/walmethods.c b/src/bin/pg_basebackup/walmethods.c
index 8695647db4..068b276251 100644
--- a/src/bin/pg_basebackup/walmethods.c
+++ b/src/bin/pg_basebackup/walmethods.c
@@ -41,6 +41,7 @@
typedef struct DirectoryMethodData
{
char *basedir;
+ WalCompressionMethod compression_method;
int compression;
bool sync;
} DirectoryMethodData;
@@ -74,7 +75,8 @@ dir_get_file_name(const char *pathname, const char *temp_suffix)
char *filename = pg_malloc0(MAXPGPATH * sizeof(char));
snprintf(filename, MAXPGPATH, "%s%s%s",
- pathname, dir_data->compression > 0 ? ".gz" : "",
+ pathname,
+ dir_data->compression_method == COMPRESSION_ZLIB ? ".gz" : "",
temp_suffix ? temp_suffix : "");
return filename;
@@ -107,7 +109,7 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
return NULL;
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
{
gzfp = gzdopen(fd, "wb");
if (gzfp == NULL)
@@ -126,7 +128,7 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
#endif
/* Do pre-padding on non-compressed files */
- if (pad_to_size && dir_data->compression == 0)
+ if (pad_to_size && dir_data->compression_method == COMPRESSION_NONE)
{
PGAlignedXLogBlock zerobuf;
int bytes;
@@ -171,7 +173,7 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
fsync_parent_path(tmppath) != 0)
{
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
gzclose(gzfp);
else
#endif
@@ -182,7 +184,7 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
f = pg_malloc0(sizeof(DirectoryMethodFile));
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
f->gzfp = gzfp;
#endif
f->fd = fd;
@@ -204,7 +206,7 @@ dir_write(Walfile f, const void *buf, size_t count)
Assert(f != NULL);
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
r = (ssize_t) gzwrite(df->gzfp, buf, count);
else
#endif
@@ -234,7 +236,7 @@ dir_close(Walfile f, WalCloseMethod method)
Assert(f != NULL);
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
r = gzclose(df->gzfp);
else
#endif
@@ -309,7 +311,7 @@ dir_sync(Walfile f)
return 0;
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_ZLIB)
{
if (gzflush(((DirectoryMethodFile *) f)->gzfp, Z_SYNC_FLUSH) != Z_OK)
return -1;
@@ -334,10 +336,10 @@ dir_get_file_size(const char *pathname)
return statbuf.st_size;
}
-static int
-dir_compression(void)
+static WalCompressionMethod
+dir_compression_method(void)
{
- return dir_data->compression;
+ return dir_data->compression_method;
}
static bool
@@ -373,7 +375,9 @@ dir_finish(void)
WalWriteMethod *
-CreateWalDirectoryMethod(const char *basedir, int compression, bool sync)
+CreateWalDirectoryMethod(const char *basedir,
+ WalCompressionMethod compression_method,
+ int compression, bool sync)
{
WalWriteMethod *method;
@@ -383,7 +387,7 @@ CreateWalDirectoryMethod(const char *basedir, int compression, bool sync)
method->get_current_pos = dir_get_current_pos;
method->get_file_size = dir_get_file_size;
method->get_file_name = dir_get_file_name;
- method->compression = dir_compression;
+ method->compression_method = dir_compression_method;
method->close = dir_close;
method->sync = dir_sync;
method->existsfile = dir_existsfile;
@@ -391,6 +395,7 @@ CreateWalDirectoryMethod(const char *basedir, int compression, bool sync)
method->getlasterror = dir_getlasterror;
dir_data = pg_malloc0(sizeof(DirectoryMethodData));
+ dir_data->compression_method = compression_method;
dir_data->compression = compression;
dir_data->basedir = pg_strdup(basedir);
dir_data->sync = sync;
@@ -424,6 +429,7 @@ typedef struct TarMethodData
{
char *tarfilename;
int fd;
+ WalCompressionMethod compression_method;
int compression;
bool sync;
TarMethodFile *currentfile;
@@ -731,10 +737,10 @@ tar_get_file_size(const char *pathname)
return -1;
}
-static int
-tar_compression(void)
+static WalCompressionMethod
+tar_compression_method(void)
{
- return tar_data->compression;
+ return tar_data->compression_method;
}
static off_t
@@ -1031,8 +1037,16 @@ tar_finish(void)
return true;
}
+/*
+ * The argument compression_method is currently ignored. It is in place for
+ * symmetry with CreateWalDirectoryMethod which uses it for distinguishing
+ * between the different compression methods. CreateWalTarMethod and its family
+ * of functions handle only zlib compression.
+ */
WalWriteMethod *
-CreateWalTarMethod(const char *tarbase, int compression, bool sync)
+CreateWalTarMethod(const char *tarbase,
+ WalCompressionMethod compression_method,
+ int compression, bool sync)
{
WalWriteMethod *method;
const char *suffix = (compression != 0) ? ".tar.gz" : ".tar";
@@ -1043,7 +1057,7 @@ CreateWalTarMethod(const char *tarbase, int compression, bool sync)
method->get_current_pos = tar_get_current_pos;
method->get_file_size = tar_get_file_size;
method->get_file_name = tar_get_file_name;
- method->compression = tar_compression;
+ method->compression_method = tar_compression_method;
method->close = tar_close;
method->sync = tar_sync;
method->existsfile = tar_existsfile;
@@ -1054,6 +1068,7 @@ CreateWalTarMethod(const char *tarbase, int compression, bool sync)
tar_data->tarfilename = pg_malloc0(strlen(tarbase) + strlen(suffix) + 1);
sprintf(tar_data->tarfilename, "%s%s", tarbase, suffix);
tar_data->fd = -1;
+ tar_data->compression_method = compression_method;
tar_data->compression = compression;
tar_data->sync = sync;
#ifdef HAVE_LIBZ
diff --git a/src/bin/pg_basebackup/walmethods.h b/src/bin/pg_basebackup/walmethods.h
index 4abdfd8333..4fc7b3d2a3 100644
--- a/src/bin/pg_basebackup/walmethods.h
+++ b/src/bin/pg_basebackup/walmethods.h
@@ -19,6 +19,12 @@ typedef enum
CLOSE_NO_RENAME
} WalCloseMethod;
+typedef enum
+{
+ COMPRESSION_ZLIB,
+ COMPRESSION_NONE
+} WalCompressionMethod;
+
/*
* A WalWriteMethod structure represents the different methods used
* to write the streaming WAL as it's received.
@@ -58,8 +64,8 @@ struct WalWriteMethod
*/
char *(*get_file_name) (const char *pathname, const char *temp_suffix);
- /* Return the level of compression */
- int (*compression) (void);
+ /* Returns the compression method */
+ WalCompressionMethod (*compression_method) (void);
/*
* Write count number of bytes to the file, and return the number of bytes
@@ -95,8 +101,11 @@ struct WalWriteMethod
* not all those required for pg_receivewal)
*/
WalWriteMethod *CreateWalDirectoryMethod(const char *basedir,
+ WalCompressionMethod compression_method,
int compression, bool sync);
-WalWriteMethod *CreateWalTarMethod(const char *tarbase, int compression, bool sync);
+WalWriteMethod *CreateWalTarMethod(const char *tarbase,
+ WalCompressionMethod compression_method,
+ int compression, bool sync);
/* Cleanup routines for previously-created methods */
void FreeWalDirectoryMethod(void);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 7bbbb34e2f..da6ac8ed83 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2858,6 +2858,7 @@ WaitEventTimeout
WaitPMResult
WalCloseMethod
WalCompression
+WalCompressionMethod
WalLevel
WalRcvData
WalRcvExecResult
--
2.25.1
v8-0002-Teach-pg_receivewal-to-use-LZ4-compression.patchtext/x-patch; name=v8-0002-Teach-pg_receivewal-to-use-LZ4-compression.patchDownload
From d936a8621339a86e9151bd5b942e25afd1dabc64 Mon Sep 17 00:00:00 2001
From: Georgios Kokolatos <gkokolatos@pm.me>
Date: Tue, 2 Nov 2021 11:26:51 +0000
Subject: [PATCH v8 2/2] Teach pg_receivewal to use LZ4 compression
The program pg_receivewal can use gzip compression to store the received WAL.
This commit teaches it to also be able to use LZ4 compression. It is required
that the binary is build using the -llz4 flag. It is enabled via the --with-lz4
flag on configuration time.
The option `--compression-method` has been expanded to inlude the value [LZ4].
The option `--compress` can not be used with LZ4 compression.
Under the hood there is nothing exceptional to be noted. Tar based archives have
not yet been taught to use LZ4 compression. If that is felt useful, then it is
easy to be added in the future.
Tests have been added to verify the creation and correctness of the generated
LZ4 files. The later is achieved by the use of LZ4 program, if present in the
installation.
---
doc/src/sgml/ref/pg_receivewal.sgml | 5 +-
src/Makefile.global.in | 1 +
src/bin/pg_basebackup/Makefile | 1 +
src/bin/pg_basebackup/pg_receivewal.c | 129 +++++++++++++++
src/bin/pg_basebackup/t/020_pg_receivewal.pl | 72 ++++++++-
src/bin/pg_basebackup/walmethods.c | 159 ++++++++++++++++++-
src/bin/pg_basebackup/walmethods.h | 1 +
7 files changed, 358 insertions(+), 10 deletions(-)
diff --git a/doc/src/sgml/ref/pg_receivewal.sgml b/doc/src/sgml/ref/pg_receivewal.sgml
index f95cffcd5e..2fb2431b3d 100644
--- a/doc/src/sgml/ref/pg_receivewal.sgml
+++ b/doc/src/sgml/ref/pg_receivewal.sgml
@@ -268,8 +268,11 @@ PostgreSQL documentation
<listitem>
<para>
Enables compression of write-ahead logs using the specified method.
- Supported values <literal>zlib</literal>,
+ Supported values are <literal>lz4</literal>, <literal>zlib</literal>,
and <literal>none</literal>.
+ For the <productname>LZ4</productname> method to be available,
+ <productname>PostgreSQL</productname> must have been have been compiled
+ with <option>--with-lz4</option>.
</para>
</listitem>
</varlistentry>
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 533c12fef9..05c54b27de 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -350,6 +350,7 @@ XGETTEXT = @XGETTEXT@
GZIP = gzip
BZIP2 = bzip2
+LZ4 = lz4
DOWNLOAD = wget -O $@ --no-use-server-timestamps
#DOWNLOAD = curl -o $@
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index 459d514183..387d728345 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -24,6 +24,7 @@ export TAR
# used by the command "gzip" to pass down options, so stick with a different
# name.
export GZIP_PROGRAM=$(GZIP)
+export LZ4
override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport)
diff --git a/src/bin/pg_basebackup/pg_receivewal.c b/src/bin/pg_basebackup/pg_receivewal.c
index 9641f4a2f4..87f8589a71 100644
--- a/src/bin/pg_basebackup/pg_receivewal.c
+++ b/src/bin/pg_basebackup/pg_receivewal.c
@@ -29,6 +29,10 @@
#include "receivelog.h"
#include "streamutil.h"
+#ifdef HAVE_LIBLZ4
+#include "lz4frame.h"
+#endif
+
/* Time to sleep between reconnection attempts */
#define RECONNECT_SLEEP_TIME 5
@@ -137,6 +141,15 @@ is_xlogfilename(const char *filename, bool *ispartial,
return true;
}
+ /* File looks like a complete LZ4 compressed XLOG file */
+ if (fname_len == XLOG_FNAME_LEN + strlen(".lz4") &&
+ strcmp(filename + XLOG_FNAME_LEN, ".lz4") == 0)
+ {
+ *ispartial = false;
+ *wal_compression_method = COMPRESSION_LZ4;
+ return true;
+ }
+
/* File looks like a partial uncompressed XLOG file */
if (fname_len == XLOG_FNAME_LEN + strlen(".partial") &&
strcmp(filename + XLOG_FNAME_LEN, ".partial") == 0)
@@ -155,6 +168,15 @@ is_xlogfilename(const char *filename, bool *ispartial,
return true;
}
+ /* File looks like a partial LZ4 compressed XLOG file */
+ if (fname_len == XLOG_FNAME_LEN + strlen(".lz4.partial") &&
+ strcmp(filename + XLOG_FNAME_LEN, ".lz4.partial") == 0)
+ {
+ *ispartial = true;
+ *wal_compression_method = COMPRESSION_LZ4;
+ return true;
+ }
+
/* File does not look like something we recognise */
return false;
}
@@ -285,6 +307,10 @@ FindStreamingStart(uint32 *tli)
* lower than 4GB, and then compare it to the size of a completed
* segment. The 4 last bytes correspond to the ISIZE member according
* to http://www.zlib.org/rfc-gzip.html.
+ *
+ * For LZ4 compressed segments, uncompress the file in a throw away
+ * buffer keeping track of the uncompressed size. Then compare it to
+ * the size of a completed segment.
*/
if (!ispartial && wal_compression_method == COMPRESSION_NONE)
{
@@ -351,6 +377,99 @@ FindStreamingStart(uint32 *tli)
continue;
}
}
+ else if (!ispartial && compression_method == COMPRESSION_LZ4)
+ {
+#ifdef HAVE_LIBLZ4
+#define LZ4_CHUNK_SZ 4096
+ int fd;
+ int r;
+ size_t uncompressed_size = 0;
+ char fullpath[MAXPGPATH * 2];
+ char readbuf[LZ4_CHUNK_SZ];
+ char outbuf[LZ4_CHUNK_SZ];
+ LZ4F_decompressionContext_t ctx = NULL;
+ LZ4F_errorCode_t status;
+
+ snprintf(fullpath, sizeof(fullpath), "%s/%s", basedir, dirent->d_name);
+
+ fd = open(fullpath, O_RDONLY | PG_BINARY, 0);
+ if (fd < 0)
+ {
+ pg_log_error("could not open file \"%s\": %m", fullpath);
+ exit(1);
+ }
+
+ status = LZ4F_createDecompressionContext(&ctx, LZ4F_VERSION);
+ if (LZ4F_isError(status))
+ {
+ pg_log_error("could not create LZ4 decompression context: %s",
+ LZ4F_getErrorName(status));
+ exit(1);
+ }
+
+ while (1)
+ {
+ char *readp;
+ char *readend;
+
+ r = read(fd, readbuf, sizeof(readbuf));
+ if (r < 0)
+ {
+ pg_log_error("could not read file \"%s\": %m", fullpath);
+ exit(1);
+ }
+
+ /* Done reading */
+ if (r == 0)
+ break;
+
+ readp = readbuf;
+ readend = readbuf + r;
+ while (readp < readend)
+ {
+ size_t read_size = 1;
+ size_t out_size = 1;
+
+ status = LZ4F_decompress(ctx, outbuf, &out_size,
+ readbuf, &read_size, NULL);
+ if (LZ4F_isError(status))
+ {
+ pg_log_error("could not decompress file \"%s\": %s",
+ fullpath,
+ LZ4F_getErrorName(status));
+ exit(1);
+ }
+
+ readp += read_size;
+ uncompressed_size += out_size;
+ }
+ }
+
+ close(fd);
+
+ status = LZ4F_freeDecompressionContext(ctx);
+ if (LZ4F_isError(status))
+ {
+ pg_log_error("could not free LZ4 decompression context: %s",
+ LZ4F_getErrorName(status));
+ exit(1);
+ }
+
+ if (uncompressed_size != WalSegSz)
+ {
+ pg_log_warning("compressed segment file \"%s\" has incorrect uncompressed size %ld, skipping",
+ dirent->d_name, uncompressed_size);
+ (void) LZ4F_freeDecompressionContext(ctx);
+ continue;
+ }
+#else
+ pg_log_error("could not check segment file \"%s\" compressed with LZ4",
+ dirent->d_name);
+ pg_log_error("this build does not support compression with %s",
+ "LZ4");
+ exit(1);
+#endif
+ }
/* Looks like a valid segment. Remember that we saw it. */
if ((segno > high_segno) ||
@@ -630,6 +749,8 @@ main(int argc, char **argv)
case 'I':
if (pg_strcasecmp(optarg, "zlib") == 0)
compression_method = COMPRESSION_ZLIB;
+ else if (pg_strcasecmp(optarg, "lz4") == 0)
+ compression_method = COMPRESSION_LZ4;
else if (pg_strcasecmp(optarg, "none") == 0)
compression_method = COMPRESSION_NONE;
else
@@ -733,6 +854,14 @@ main(int argc, char **argv)
"gzip");
exit(1);
}
+#endif
+#ifndef HAVE_LIBLZ4
+ if (compression_method == COMPRESSION_LZ4)
+ {
+ pg_log_error("this build does not support compression with %s",
+ "LZ4");
+ exit(1);
+ }
#endif
}
diff --git a/src/bin/pg_basebackup/t/020_pg_receivewal.pl b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
index 56c4a0d2af..ef4464556a 100644
--- a/src/bin/pg_basebackup/t/020_pg_receivewal.pl
+++ b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
@@ -5,7 +5,7 @@ use strict;
use warnings;
use PostgreSQL::Test::Utils;
use PostgreSQL::Test::Cluster;
-use Test::More tests => 37;
+use Test::More tests => 42;
program_help_ok('pg_receivewal');
program_version_ok('pg_receivewal');
@@ -138,13 +138,69 @@ SKIP:
"gzip verified the integrity of compressed WAL segments");
}
+# Check LZ4 compression if available
+SKIP:
+{
+ skip "postgres was not built with LZ4 support", 5
+ if (!check_pg_config("#define HAVE_LIBLZ4 1"));
+
+ # Generate more WAL including one completed, compressed segment.
+ $primary->psql('postgres', 'SELECT pg_switch_wal();');
+ $nextlsn =
+ $primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
+ chomp($nextlsn);
+ $primary->psql('postgres', 'INSERT INTO test_table VALUES (3);');
+
+ # Stream up to the given position
+ $primary->command_ok(
+ [
+ 'pg_receivewal', '-D',
+ $stream_dir, '--verbose',
+ '--endpos', $nextlsn,
+ '--no-loop', '--compression-method',
+ 'lz4'
+ ],
+ 'streaming some WAL using --compression-method=lz4');
+
+ # Verify that the stored files are generated with their expected
+ # names.
+ my @lz4_wals = glob "$stream_dir/*.lz4";
+ is(scalar(@lz4_wals), 1,
+ "one WAL segment compressed with LZ4 was created");
+ my @lz4_partial_wals = glob "$stream_dir/*.lz4.partial";
+ is(scalar(@lz4_partial_wals),
+ 1, "one partial WAL segment compressed with LZ4 was created");
+
+ # Verify that the start streaming position is computed correctly by
+ # comparing it with the partial file generated previously. The name
+ # of the previous partial, now-completed WAL segment is updated, keeping
+ # its base number.
+ $partial_wals[0] =~ s/(\.gz)?\.partial$/.lz4/;
+ is($lz4_wals[0] eq $partial_wals[0],
+ 1, "one partial WAL segment is now completed");
+ # Update the list of partial wals with the current one.
+ @partial_wals = @lz4_partial_wals;
+
+ # Check the integrity of the completed segment, if LZ4 is an available
+ # command.
+ my $lz4 = $ENV{LZ4};
+ skip "program lz4 is not found in your system", 1
+ if ( !defined $lz4
+ || $lz4 eq ''
+ || system_log($lz4, '--version') != 0);
+
+ my $lz4_is_valid = system_log($lz4, '-t', @lz4_wals);
+ is($lz4_is_valid, 0,
+ "lz4 verified the integrity of compressed WAL segments");
+}
+
# Verify that the start streaming position is computed and that the value is
-# correct regardless of whether ZLIB is available.
+# correct regardless of whether any compression is available.
$primary->psql('postgres', 'SELECT pg_switch_wal();');
$nextlsn =
$primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
chomp($nextlsn);
-$primary->psql('postgres', 'INSERT INTO test_table VALUES (3);');
+$primary->psql('postgres', 'INSERT INTO test_table VALUES (4);');
$primary->command_ok(
[
'pg_receivewal', '-D', $stream_dir, '--verbose',
@@ -152,7 +208,7 @@ $primary->command_ok(
],
"streaming some WAL");
-$partial_wals[0] =~ s/(\.gz)?.partial//;
+$partial_wals[0] =~ s/(\.gz|\.lz4)?.partial//;
ok(-e $partial_wals[0], "check that previously partial WAL is now complete");
# Permissions on WAL files should be default
@@ -190,7 +246,7 @@ my $walfile_streamed = $primary->safe_psql(
# Switch to a new segment, to make sure that the segment retained by the
# slot is still streamed. This may not be necessary, but play it safe.
-$primary->psql('postgres', 'INSERT INTO test_table VALUES (4);');
+$primary->psql('postgres', 'INSERT INTO test_table VALUES (5);');
$primary->psql('postgres', 'SELECT pg_switch_wal();');
$nextlsn =
$primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
@@ -198,7 +254,7 @@ chomp($nextlsn);
# Add a bit more data to accelerate the end of the next pg_receivewal
# commands.
-$primary->psql('postgres', 'INSERT INTO test_table VALUES (5);');
+$primary->psql('postgres', 'INSERT INTO test_table VALUES (6);');
# Check case where the slot does not exist.
$primary->command_fails_like(
@@ -253,13 +309,13 @@ $standby->promote;
# on the new timeline.
my $walfile_after_promotion = $standby->safe_psql('postgres',
"SELECT pg_walfile_name(pg_current_wal_insert_lsn());");
-$standby->psql('postgres', 'INSERT INTO test_table VALUES (6);');
+$standby->psql('postgres', 'INSERT INTO test_table VALUES (7);');
$standby->psql('postgres', 'SELECT pg_switch_wal();');
$nextlsn =
$standby->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
chomp($nextlsn);
# This speeds up the operation.
-$standby->psql('postgres', 'INSERT INTO test_table VALUES (7);');
+$standby->psql('postgres', 'INSERT INTO test_table VALUES (8);');
# Now try to resume from the slot after the promotion.
my $timeline_dir = $primary->basedir . '/timeline_wal';
diff --git a/src/bin/pg_basebackup/walmethods.c b/src/bin/pg_basebackup/walmethods.c
index 068b276251..1ec89f1d2b 100644
--- a/src/bin/pg_basebackup/walmethods.c
+++ b/src/bin/pg_basebackup/walmethods.c
@@ -17,6 +17,10 @@
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
+
+#ifdef HAVE_LIBLZ4
+#include <lz4frame.h>
+#endif
#ifdef HAVE_LIBZ
#include <zlib.h>
#endif
@@ -30,6 +34,9 @@
/* Size of zlib buffer for .tar.gz */
#define ZLIB_OUT_SIZE 4096
+/* Size of LZ4 input chunk for .lz4 */
+#define LZ4_IN_SIZE 4096
+
/*-------------------------------------------------------------------------
* WalDirectoryMethod - write wal to a directory looking like pg_wal
*-------------------------------------------------------------------------
@@ -60,6 +67,11 @@ typedef struct DirectoryMethodFile
#ifdef HAVE_LIBZ
gzFile gzfp;
#endif
+#ifdef HAVE_LIBLZ4
+ LZ4F_compressionContext_t ctx;
+ size_t lz4bufsize;
+ void *lz4buf;
+#endif
} DirectoryMethodFile;
static const char *
@@ -76,7 +88,8 @@ dir_get_file_name(const char *pathname, const char *temp_suffix)
snprintf(filename, MAXPGPATH, "%s%s%s",
pathname,
- dir_data->compression_method == COMPRESSION_ZLIB ? ".gz" : "",
+ dir_data->compression_method == COMPRESSION_ZLIB ? ".gz" :
+ dir_data->compression_method == COMPRESSION_LZ4 ? ".lz4" : "",
temp_suffix ? temp_suffix : "");
return filename;
@@ -92,6 +105,11 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
#ifdef HAVE_LIBZ
gzFile gzfp = NULL;
#endif
+#ifdef HAVE_LIBLZ4
+ LZ4F_compressionContext_t ctx = NULL;
+ size_t lz4bufsize = 0;
+ void *lz4buf = NULL;
+#endif
filename = dir_get_file_name(pathname, temp_suffix);
snprintf(tmppath, sizeof(tmppath), "%s/%s",
@@ -126,6 +144,49 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
}
}
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ size_t ctx_out;
+ size_t header_size;
+
+ ctx_out = LZ4F_createCompressionContext(&ctx, LZ4F_VERSION);
+ lz4bufsize = LZ4F_compressBound(LZ4_IN_SIZE, NULL);
+ if (LZ4F_isError(ctx_out))
+ {
+ close(fd);
+ return NULL;
+ }
+
+ lz4buf = pg_malloc0(lz4bufsize);
+
+ /* add the header */
+ header_size = LZ4F_compressBegin(ctx, lz4buf, lz4bufsize, NULL);
+ if (LZ4F_isError(header_size))
+ {
+ pg_free(lz4buf);
+ close(fd);
+ return NULL;
+ }
+
+ errno = 0;
+ if (write(fd, lz4buf, header_size) != header_size)
+ {
+ int save_errno = errno;
+
+ (void) LZ4F_compressEnd(ctx, lz4buf, lz4bufsize, NULL);
+ (void) LZ4F_freeCompressionContext(ctx);
+ pg_free(lz4buf);
+ close(fd);
+
+ /*
+ * If write didn't set errno, assume problem is no disk space.
+ */
+ errno = save_errno ? save_errno : ENOSPC;
+ return NULL;
+ }
+ }
+#endif
/* Do pre-padding on non-compressed files */
if (pad_to_size && dir_data->compression_method == COMPRESSION_NONE)
@@ -176,6 +237,16 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
if (dir_data->compression_method == COMPRESSION_ZLIB)
gzclose(gzfp);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ (void) LZ4F_compressEnd(ctx, lz4buf, lz4bufsize, NULL);
+ (void) LZ4F_freeCompressionContext(ctx);
+ pg_free(lz4buf);
+ close(fd);
+ }
+ else
#endif
close(fd);
return NULL;
@@ -187,6 +258,15 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
if (dir_data->compression_method == COMPRESSION_ZLIB)
f->gzfp = gzfp;
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ f->ctx = ctx;
+ f->lz4buf = lz4buf;
+ f->lz4bufsize = lz4bufsize;
+ }
+#endif
+
f->fd = fd;
f->currpos = 0;
f->pathname = pg_strdup(pathname);
@@ -209,6 +289,43 @@ dir_write(Walfile f, const void *buf, size_t count)
if (dir_data->compression_method == COMPRESSION_ZLIB)
r = (ssize_t) gzwrite(df->gzfp, buf, count);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ size_t chunk;
+ size_t remaining;
+ const void *inbuf = buf;
+
+ remaining = count;
+ while (remaining > 0)
+ {
+ size_t compressed;
+
+ if (remaining > LZ4_IN_SIZE)
+ chunk = LZ4_IN_SIZE;
+ else
+ chunk = remaining;
+
+ remaining -= chunk;
+ compressed = LZ4F_compressUpdate(df->ctx,
+ df->lz4buf, df->lz4bufsize,
+ inbuf, chunk,
+ NULL);
+
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+
+ inbuf = ((char *) inbuf) + chunk;
+ }
+
+ /* Our caller keeps track of the uncompressed size. */
+ r = (ssize_t) count;
+ }
+ else
#endif
r = write(df->fd, buf, count);
if (r > 0)
@@ -239,6 +356,25 @@ dir_close(Walfile f, WalCloseMethod method)
if (dir_data->compression_method == COMPRESSION_ZLIB)
r = gzclose(df->gzfp);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ size_t compressed;
+
+ compressed = LZ4F_compressEnd(df->ctx,
+ df->lz4buf, df->lz4bufsize,
+ NULL);
+
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+
+ r = close(df->fd);
+ }
+ else
#endif
r = close(df->fd);
@@ -293,6 +429,12 @@ dir_close(Walfile f, WalCloseMethod method)
}
}
+#ifdef HAVE_LIBLZ4
+ pg_free(df->lz4buf);
+ /* supports free on NULL */
+ LZ4F_freeCompressionContext(df->ctx);
+#endif
+
pg_free(df->pathname);
pg_free(df->fullpath);
if (df->temp_suffix)
@@ -317,6 +459,21 @@ dir_sync(Walfile f)
return -1;
}
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ DirectoryMethodFile *df = (DirectoryMethodFile *) f;
+ size_t compressed;
+
+ /* Flush any internal buffers */
+ compressed = LZ4F_flush(df->ctx, df->lz4buf, df->lz4bufsize, NULL);
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+ }
+#endif
return fsync(((DirectoryMethodFile *) f)->fd);
}
diff --git a/src/bin/pg_basebackup/walmethods.h b/src/bin/pg_basebackup/walmethods.h
index 4fc7b3d2a3..3e378c87b6 100644
--- a/src/bin/pg_basebackup/walmethods.h
+++ b/src/bin/pg_basebackup/walmethods.h
@@ -21,6 +21,7 @@ typedef enum
typedef enum
{
+ COMPRESSION_LZ4,
COMPRESSION_ZLIB,
COMPRESSION_NONE
} WalCompressionMethod;
--
2.25.1
Re: Teach pg_receivewal to use lz4 compression
On Tue, Nov 2, 2021 at 9:51 AM Michael Paquier <michael@paquier.xyz> wrote:
On Tue, Nov 02, 2021 at 07:27:50AM +0900, Michael Paquier wrote:
On Mon, Nov 01, 2021 at 08:39:59AM +0000, gkokolatos@pm.me wrote:
Agreed.
I have already started on v8 of the patch with that technique. I should
be able to update the thread soon.Nice, thanks!
By the way, I was reading the last version of the patch today, and
it seems to me that we could make the commit history if we split the
patch into two parts:
- One that introduces the new option --compression-method and
is_xlogfilename(), while reworking --compress (including documentation
changes).
- One to have LZ4 support.v7 has been using "gzip" for the option name, but I think that it
would be more consistent to use "zlib" instead.
Um, why?
That we are using zlib to provide the compression is an implementation
detail. Whereas AFAIK "gzip" refers to both the program and the format. And
we specifically use the gzxxx() functions in zlib, in order to produce gzip
format.
I think for the end user, it is strictly better to name it "gzip", and
given that the target of this option is the end user we should do so. (It'd
be different it we were talking about a build-time parameter to configure).
--
Magnus Hagander
Me: https://www.hagander.net/ <http://www.hagander.net/>
Work: https://www.redpill-linpro.com/ <http://www.redpill-linpro.com/>
Re: Teach pg_receivewal to use lz4 compression
On Tue, Nov 2, 2021 at 8:17 AM Magnus Hagander <magnus@hagander.net> wrote:
Um, why?
That we are using zlib to provide the compression is an implementation detail. Whereas AFAIK "gzip" refers to both the program and the format. And we specifically use the gzxxx() functions in zlib, in order to produce gzip format.
I think for the end user, it is strictly better to name it "gzip", and given that the target of this option is the end user we should do so. (It'd be different it we were talking about a build-time parameter to configure).
I agree. Also, I think there's actually a file format called "zlib"
which is slightly different from the "gzip" format, and you have to be
careful not to generate the wrong one.
--
Robert Haas
EDB: http://www.enterprisedb.com
Re: Teach pg_receivewal to use lz4 compression
On Tue, Nov 02, 2021 at 12:31:47PM -0400, Robert Haas wrote:
On Tue, Nov 2, 2021 at 8:17 AM Magnus Hagander <magnus@hagander.net> wrote:
I think for the end user, it is strictly better to name it "gzip",
and given that the target of this option is the end user we should
do so. (It'd be different it we were talking about a build-time
parameter to configure).I agree. Also, I think there's actually a file format called "zlib"
which is slightly different from the "gzip" format, and you have to be
careful not to generate the wrong one.
Okay, fine by me. It would be better to be also consistent in
WalCompressionMethods once we switch to this option value, then.
--
Michael
Re: Teach pg_receivewal to use lz4 compression
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, November 3rd, 2021 at 12:23 AM, Michael Paquier <michael@paquier.xyz> wrote:
On Tue, Nov 02, 2021 at 12:31:47PM -0400, Robert Haas wrote:
On Tue, Nov 2, 2021 at 8:17 AM Magnus Hagander magnus@hagander.net wrote:
I think for the end user, it is strictly better to name it "gzip",
and given that the target of this option is the end user we should
do so. (It'd be different it we were talking about a build-time
parameter to configure).I agree. Also, I think there's actually a file format called "zlib"
which is slightly different from the "gzip" format, and you have to be
careful not to generate the wrong one.Okay, fine by me. It would be better to be also consistent in
WalCompressionMethods once we switch to this option value, then.
I will revert to gzip for version 9. Should be out shortly.
Cheers,
//Georgios
Show quoted text
Michael
Re: Teach pg_receivewal to use lz4 compression
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, November 3rd, 2021 at 9:05 AM, <gkokolatos@pm.me> wrote:
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, November 3rd, 2021 at 12:23 AM, Michael Paquier michael@paquier.xyz wrote:
On Tue, Nov 02, 2021 at 12:31:47PM -0400, Robert Haas wrote:
On Tue, Nov 2, 2021 at 8:17 AM Magnus Hagander magnus@hagander.net wrote:
I think for the end user, it is strictly better to name it "gzip",
and given that the target of this option is the end user we should
do so. (It'd be different it we were talking about a build-time
parameter to configure).I agree. Also, I think there's actually a file format called "zlib"
which is slightly different from the "gzip" format, and you have to be
careful not to generate the wrong one.Okay, fine by me. It would be better to be also consistent in
WalCompressionMethods once we switch to this option value, then.I will revert to gzip for version 9. Should be out shortly.
Please find v9 attached.
Cheers,
//Georgios
Show quoted text
Michael
Attachments:
v9-0002-Teach-pg_receivewal-to-use-LZ4-compression.patchtext/x-patch; name=v9-0002-Teach-pg_receivewal-to-use-LZ4-compression.patchDownload
From 8e33136f81c3197020053cba0f7f070d594f056e Mon Sep 17 00:00:00 2001
From: Georgios Kokolatos <gkokolatos@pm.me>
Date: Wed, 3 Nov 2021 08:59:58 +0000
Subject: [PATCH v9 2/2] Teach pg_receivewal to use LZ4 compression
The program pg_receivewal can use gzip compression to store the received WAL.
This commit teaches it to also be able to use LZ4 compression. It is required
that the binary is build using the -llz4 flag. It is enabled via the --with-lz4
flag on configuration time.
The option `--compression-method` has been expanded to inlude the value [LZ4].
The option `--compress` can not be used with LZ4 compression.
Under the hood there is nothing exceptional to be noted. Tar based archives have
not yet been taught to use LZ4 compression. If that is felt useful, then it is
easy to be added in the future.
Tests have been added to verify the creation and correctness of the generated
LZ4 files. The later is achieved by the use of LZ4 program, if present in the
installation.
---
doc/src/sgml/ref/pg_receivewal.sgml | 5 +-
src/Makefile.global.in | 1 +
src/bin/pg_basebackup/Makefile | 1 +
src/bin/pg_basebackup/pg_receivewal.c | 129 +++++++++++++++
src/bin/pg_basebackup/t/020_pg_receivewal.pl | 72 ++++++++-
src/bin/pg_basebackup/walmethods.c | 159 ++++++++++++++++++-
src/bin/pg_basebackup/walmethods.h | 1 +
7 files changed, 358 insertions(+), 10 deletions(-)
diff --git a/doc/src/sgml/ref/pg_receivewal.sgml b/doc/src/sgml/ref/pg_receivewal.sgml
index cf2eaa1486..411b275de0 100644
--- a/doc/src/sgml/ref/pg_receivewal.sgml
+++ b/doc/src/sgml/ref/pg_receivewal.sgml
@@ -268,8 +268,11 @@ PostgreSQL documentation
<listitem>
<para>
Enables compression of write-ahead logs using the specified method.
- Supported values <literal>gzip</literal>,
+ Supported values are <literal>lz4</literal>, <literal>gzip</literal>,
and <literal>none</literal>.
+ For the <productname>LZ4</productname> method to be available,
+ <productname>PostgreSQL</productname> must have been have been compiled
+ with <option>--with-lz4</option>.
</para>
</listitem>
</varlistentry>
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 533c12fef9..05c54b27de 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -350,6 +350,7 @@ XGETTEXT = @XGETTEXT@
GZIP = gzip
BZIP2 = bzip2
+LZ4 = lz4
DOWNLOAD = wget -O $@ --no-use-server-timestamps
#DOWNLOAD = curl -o $@
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index 459d514183..387d728345 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -24,6 +24,7 @@ export TAR
# used by the command "gzip" to pass down options, so stick with a different
# name.
export GZIP_PROGRAM=$(GZIP)
+export LZ4
override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport)
diff --git a/src/bin/pg_basebackup/pg_receivewal.c b/src/bin/pg_basebackup/pg_receivewal.c
index 9449b50868..af3eba8845 100644
--- a/src/bin/pg_basebackup/pg_receivewal.c
+++ b/src/bin/pg_basebackup/pg_receivewal.c
@@ -29,6 +29,10 @@
#include "receivelog.h"
#include "streamutil.h"
+#ifdef HAVE_LIBLZ4
+#include "lz4frame.h"
+#endif
+
/* Time to sleep between reconnection attempts */
#define RECONNECT_SLEEP_TIME 5
@@ -137,6 +141,15 @@ is_xlogfilename(const char *filename, bool *ispartial,
return true;
}
+ /* File looks like a complete LZ4 compressed XLOG file */
+ if (fname_len == XLOG_FNAME_LEN + strlen(".lz4") &&
+ strcmp(filename + XLOG_FNAME_LEN, ".lz4") == 0)
+ {
+ *ispartial = false;
+ *wal_compression_method = COMPRESSION_LZ4;
+ return true;
+ }
+
/* File looks like a partial uncompressed XLOG file */
if (fname_len == XLOG_FNAME_LEN + strlen(".partial") &&
strcmp(filename + XLOG_FNAME_LEN, ".partial") == 0)
@@ -155,6 +168,15 @@ is_xlogfilename(const char *filename, bool *ispartial,
return true;
}
+ /* File looks like a partial LZ4 compressed XLOG file */
+ if (fname_len == XLOG_FNAME_LEN + strlen(".lz4.partial") &&
+ strcmp(filename + XLOG_FNAME_LEN, ".lz4.partial") == 0)
+ {
+ *ispartial = true;
+ *wal_compression_method = COMPRESSION_LZ4;
+ return true;
+ }
+
/* File does not look like something we recognise */
return false;
}
@@ -285,6 +307,10 @@ FindStreamingStart(uint32 *tli)
* lower than 4GB, and then compare it to the size of a completed
* segment. The 4 last bytes correspond to the ISIZE member according
* to http://www.zlib.org/rfc-gzip.html.
+ *
+ * For LZ4 compressed segments, uncompress the file in a throw away
+ * buffer keeping track of the uncompressed size. Then compare it to
+ * the size of a completed segment.
*/
if (!ispartial && wal_compression_method == COMPRESSION_NONE)
{
@@ -351,6 +377,99 @@ FindStreamingStart(uint32 *tli)
continue;
}
}
+ else if (!ispartial && compression_method == COMPRESSION_LZ4)
+ {
+#ifdef HAVE_LIBLZ4
+#define LZ4_CHUNK_SZ 4096
+ int fd;
+ int r;
+ size_t uncompressed_size = 0;
+ char fullpath[MAXPGPATH * 2];
+ char readbuf[LZ4_CHUNK_SZ];
+ char outbuf[LZ4_CHUNK_SZ];
+ LZ4F_decompressionContext_t ctx = NULL;
+ LZ4F_errorCode_t status;
+
+ snprintf(fullpath, sizeof(fullpath), "%s/%s", basedir, dirent->d_name);
+
+ fd = open(fullpath, O_RDONLY | PG_BINARY, 0);
+ if (fd < 0)
+ {
+ pg_log_error("could not open file \"%s\": %m", fullpath);
+ exit(1);
+ }
+
+ status = LZ4F_createDecompressionContext(&ctx, LZ4F_VERSION);
+ if (LZ4F_isError(status))
+ {
+ pg_log_error("could not create LZ4 decompression context: %s",
+ LZ4F_getErrorName(status));
+ exit(1);
+ }
+
+ while (1)
+ {
+ char *readp;
+ char *readend;
+
+ r = read(fd, readbuf, sizeof(readbuf));
+ if (r < 0)
+ {
+ pg_log_error("could not read file \"%s\": %m", fullpath);
+ exit(1);
+ }
+
+ /* Done reading */
+ if (r == 0)
+ break;
+
+ readp = readbuf;
+ readend = readbuf + r;
+ while (readp < readend)
+ {
+ size_t read_size = 1;
+ size_t out_size = 1;
+
+ status = LZ4F_decompress(ctx, outbuf, &out_size,
+ readbuf, &read_size, NULL);
+ if (LZ4F_isError(status))
+ {
+ pg_log_error("could not decompress file \"%s\": %s",
+ fullpath,
+ LZ4F_getErrorName(status));
+ exit(1);
+ }
+
+ readp += read_size;
+ uncompressed_size += out_size;
+ }
+ }
+
+ close(fd);
+
+ status = LZ4F_freeDecompressionContext(ctx);
+ if (LZ4F_isError(status))
+ {
+ pg_log_error("could not free LZ4 decompression context: %s",
+ LZ4F_getErrorName(status));
+ exit(1);
+ }
+
+ if (uncompressed_size != WalSegSz)
+ {
+ pg_log_warning("compressed segment file \"%s\" has incorrect uncompressed size %ld, skipping",
+ dirent->d_name, uncompressed_size);
+ (void) LZ4F_freeDecompressionContext(ctx);
+ continue;
+ }
+#else
+ pg_log_error("could not check segment file \"%s\" compressed with LZ4",
+ dirent->d_name);
+ pg_log_error("this build does not support compression with %s",
+ "LZ4");
+ exit(1);
+#endif
+ }
/* Looks like a valid segment. Remember that we saw it. */
if ((segno > high_segno) ||
@@ -630,6 +749,8 @@ main(int argc, char **argv)
case 'I':
if (pg_strcasecmp(optarg, "gzip") == 0)
compression_method = COMPRESSION_GZIP;
+ else if (pg_strcasecmp(optarg, "lz4") == 0)
+ compression_method = COMPRESSION_LZ4;
else if (pg_strcasecmp(optarg, "none") == 0)
compression_method = COMPRESSION_NONE;
else
@@ -733,6 +854,14 @@ main(int argc, char **argv)
"gzip");
exit(1);
}
+#endif
+#ifndef HAVE_LIBLZ4
+ if (compression_method == COMPRESSION_LZ4)
+ {
+ pg_log_error("this build does not support compression with %s",
+ "LZ4");
+ exit(1);
+ }
#endif
}
diff --git a/src/bin/pg_basebackup/t/020_pg_receivewal.pl b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
index 251ac247d8..998f40773b 100644
--- a/src/bin/pg_basebackup/t/020_pg_receivewal.pl
+++ b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
@@ -5,7 +5,7 @@ use strict;
use warnings;
use PostgreSQL::Test::Utils;
use PostgreSQL::Test::Cluster;
-use Test::More tests => 37;
+use Test::More tests => 42;
program_help_ok('pg_receivewal');
program_version_ok('pg_receivewal');
@@ -138,13 +138,69 @@ SKIP:
"gzip verified the integrity of compressed WAL segments");
}
+# Check LZ4 compression if available
+SKIP:
+{
+ skip "postgres was not built with LZ4 support", 5
+ if (!check_pg_config("#define HAVE_LIBLZ4 1"));
+
+ # Generate more WAL including one completed, compressed segment.
+ $primary->psql('postgres', 'SELECT pg_switch_wal();');
+ $nextlsn =
+ $primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
+ chomp($nextlsn);
+ $primary->psql('postgres', 'INSERT INTO test_table VALUES (3);');
+
+ # Stream up to the given position
+ $primary->command_ok(
+ [
+ 'pg_receivewal', '-D',
+ $stream_dir, '--verbose',
+ '--endpos', $nextlsn,
+ '--no-loop', '--compression-method',
+ 'lz4'
+ ],
+ 'streaming some WAL using --compression-method=lz4');
+
+ # Verify that the stored files are generated with their expected
+ # names.
+ my @lz4_wals = glob "$stream_dir/*.lz4";
+ is(scalar(@lz4_wals), 1,
+ "one WAL segment compressed with LZ4 was created");
+ my @lz4_partial_wals = glob "$stream_dir/*.lz4.partial";
+ is(scalar(@lz4_partial_wals),
+ 1, "one partial WAL segment compressed with LZ4 was created");
+
+ # Verify that the start streaming position is computed correctly by
+ # comparing it with the partial file generated previously. The name
+ # of the previous partial, now-completed WAL segment is updated, keeping
+ # its base number.
+ $partial_wals[0] =~ s/(\.gz)?\.partial$/.lz4/;
+ is($lz4_wals[0] eq $partial_wals[0],
+ 1, "one partial WAL segment is now completed");
+ # Update the list of partial wals with the current one.
+ @partial_wals = @lz4_partial_wals;
+
+ # Check the integrity of the completed segment, if LZ4 is an available
+ # command.
+ my $lz4 = $ENV{LZ4};
+ skip "program lz4 is not found in your system", 1
+ if ( !defined $lz4
+ || $lz4 eq ''
+ || system_log($lz4, '--version') != 0);
+
+ my $lz4_is_valid = system_log($lz4, '-t', @lz4_wals);
+ is($lz4_is_valid, 0,
+ "lz4 verified the integrity of compressed WAL segments");
+}
+
# Verify that the start streaming position is computed and that the value is
-# correct regardless of whether ZLIB is available.
+# correct regardless of whether any compression is available.
$primary->psql('postgres', 'SELECT pg_switch_wal();');
$nextlsn =
$primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
chomp($nextlsn);
-$primary->psql('postgres', 'INSERT INTO test_table VALUES (3);');
+$primary->psql('postgres', 'INSERT INTO test_table VALUES (4);');
$primary->command_ok(
[
'pg_receivewal', '-D', $stream_dir, '--verbose',
@@ -152,7 +208,7 @@ $primary->command_ok(
],
"streaming some WAL");
-$partial_wals[0] =~ s/(\.gz)?.partial//;
+$partial_wals[0] =~ s/(\.gz|\.lz4)?.partial//;
ok(-e $partial_wals[0], "check that previously partial WAL is now complete");
# Permissions on WAL files should be default
@@ -190,7 +246,7 @@ my $walfile_streamed = $primary->safe_psql(
# Switch to a new segment, to make sure that the segment retained by the
# slot is still streamed. This may not be necessary, but play it safe.
-$primary->psql('postgres', 'INSERT INTO test_table VALUES (4);');
+$primary->psql('postgres', 'INSERT INTO test_table VALUES (5);');
$primary->psql('postgres', 'SELECT pg_switch_wal();');
$nextlsn =
$primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
@@ -198,7 +254,7 @@ chomp($nextlsn);
# Add a bit more data to accelerate the end of the next pg_receivewal
# commands.
-$primary->psql('postgres', 'INSERT INTO test_table VALUES (5);');
+$primary->psql('postgres', 'INSERT INTO test_table VALUES (6);');
# Check case where the slot does not exist.
$primary->command_fails_like(
@@ -253,13 +309,13 @@ $standby->promote;
# on the new timeline.
my $walfile_after_promotion = $standby->safe_psql('postgres',
"SELECT pg_walfile_name(pg_current_wal_insert_lsn());");
-$standby->psql('postgres', 'INSERT INTO test_table VALUES (6);');
+$standby->psql('postgres', 'INSERT INTO test_table VALUES (7);');
$standby->psql('postgres', 'SELECT pg_switch_wal();');
$nextlsn =
$standby->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
chomp($nextlsn);
# This speeds up the operation.
-$standby->psql('postgres', 'INSERT INTO test_table VALUES (7);');
+$standby->psql('postgres', 'INSERT INTO test_table VALUES (8);');
# Now try to resume from the slot after the promotion.
my $timeline_dir = $primary->basedir . '/timeline_wal';
diff --git a/src/bin/pg_basebackup/walmethods.c b/src/bin/pg_basebackup/walmethods.c
index b710b1ef36..f271eea821 100644
--- a/src/bin/pg_basebackup/walmethods.c
+++ b/src/bin/pg_basebackup/walmethods.c
@@ -17,6 +17,10 @@
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
+
+#ifdef HAVE_LIBLZ4
+#include <lz4frame.h>
+#endif
#ifdef HAVE_LIBZ
#include <zlib.h>
#endif
@@ -30,6 +34,9 @@
/* Size of zlib buffer for .tar.gz */
#define ZLIB_OUT_SIZE 4096
+/* Size of LZ4 input chunk for .lz4 */
+#define LZ4_IN_SIZE 4096
+
/*-------------------------------------------------------------------------
* WalDirectoryMethod - write wal to a directory looking like pg_wal
*-------------------------------------------------------------------------
@@ -60,6 +67,11 @@ typedef struct DirectoryMethodFile
#ifdef HAVE_LIBZ
gzFile gzfp;
#endif
+#ifdef HAVE_LIBLZ4
+ LZ4F_compressionContext_t ctx;
+ size_t lz4bufsize;
+ void *lz4buf;
+#endif
} DirectoryMethodFile;
static const char *
@@ -76,7 +88,8 @@ dir_get_file_name(const char *pathname, const char *temp_suffix)
snprintf(filename, MAXPGPATH, "%s%s%s",
pathname,
- dir_data->compression_method == COMPRESSION_GZIP ? ".gz" : "",
+ dir_data->compression_method == COMPRESSION_GZIP ? ".gz" :
+ dir_data->compression_method == COMPRESSION_LZ4 ? ".lz4" : "",
temp_suffix ? temp_suffix : "");
return filename;
@@ -92,6 +105,11 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
#ifdef HAVE_LIBZ
gzFile gzfp = NULL;
#endif
+#ifdef HAVE_LIBLZ4
+ LZ4F_compressionContext_t ctx = NULL;
+ size_t lz4bufsize = 0;
+ void *lz4buf = NULL;
+#endif
filename = dir_get_file_name(pathname, temp_suffix);
snprintf(tmppath, sizeof(tmppath), "%s/%s",
@@ -126,6 +144,49 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
}
}
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ size_t ctx_out;
+ size_t header_size;
+
+ ctx_out = LZ4F_createCompressionContext(&ctx, LZ4F_VERSION);
+ lz4bufsize = LZ4F_compressBound(LZ4_IN_SIZE, NULL);
+ if (LZ4F_isError(ctx_out))
+ {
+ close(fd);
+ return NULL;
+ }
+
+ lz4buf = pg_malloc0(lz4bufsize);
+
+ /* add the header */
+ header_size = LZ4F_compressBegin(ctx, lz4buf, lz4bufsize, NULL);
+ if (LZ4F_isError(header_size))
+ {
+ pg_free(lz4buf);
+ close(fd);
+ return NULL;
+ }
+
+ errno = 0;
+ if (write(fd, lz4buf, header_size) != header_size)
+ {
+ int save_errno = errno;
+
+ (void) LZ4F_compressEnd(ctx, lz4buf, lz4bufsize, NULL);
+ (void) LZ4F_freeCompressionContext(ctx);
+ pg_free(lz4buf);
+ close(fd);
+
+ /*
+ * If write didn't set errno, assume problem is no disk space.
+ */
+ errno = save_errno ? save_errno : ENOSPC;
+ return NULL;
+ }
+ }
+#endif
/* Do pre-padding on non-compressed files */
if (pad_to_size && dir_data->compression_method == COMPRESSION_NONE)
@@ -176,6 +237,16 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
if (dir_data->compression_method == COMPRESSION_GZIP)
gzclose(gzfp);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ (void) LZ4F_compressEnd(ctx, lz4buf, lz4bufsize, NULL);
+ (void) LZ4F_freeCompressionContext(ctx);
+ pg_free(lz4buf);
+ close(fd);
+ }
+ else
#endif
close(fd);
return NULL;
@@ -187,6 +258,15 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
if (dir_data->compression_method == COMPRESSION_GZIP)
f->gzfp = gzfp;
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ f->ctx = ctx;
+ f->lz4buf = lz4buf;
+ f->lz4bufsize = lz4bufsize;
+ }
+#endif
+
f->fd = fd;
f->currpos = 0;
f->pathname = pg_strdup(pathname);
@@ -209,6 +289,43 @@ dir_write(Walfile f, const void *buf, size_t count)
if (dir_data->compression_method == COMPRESSION_GZIP)
r = (ssize_t) gzwrite(df->gzfp, buf, count);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ size_t chunk;
+ size_t remaining;
+ const void *inbuf = buf;
+
+ remaining = count;
+ while (remaining > 0)
+ {
+ size_t compressed;
+
+ if (remaining > LZ4_IN_SIZE)
+ chunk = LZ4_IN_SIZE;
+ else
+ chunk = remaining;
+
+ remaining -= chunk;
+ compressed = LZ4F_compressUpdate(df->ctx,
+ df->lz4buf, df->lz4bufsize,
+ inbuf, chunk,
+ NULL);
+
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+
+ inbuf = ((char *) inbuf) + chunk;
+ }
+
+ /* Our caller keeps track of the uncompressed size. */
+ r = (ssize_t) count;
+ }
+ else
#endif
r = write(df->fd, buf, count);
if (r > 0)
@@ -239,6 +356,25 @@ dir_close(Walfile f, WalCloseMethod method)
if (dir_data->compression_method == COMPRESSION_GZIP)
r = gzclose(df->gzfp);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ size_t compressed;
+
+ compressed = LZ4F_compressEnd(df->ctx,
+ df->lz4buf, df->lz4bufsize,
+ NULL);
+
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+
+ r = close(df->fd);
+ }
+ else
#endif
r = close(df->fd);
@@ -293,6 +429,12 @@ dir_close(Walfile f, WalCloseMethod method)
}
}
+#ifdef HAVE_LIBLZ4
+ pg_free(df->lz4buf);
+ /* supports free on NULL */
+ LZ4F_freeCompressionContext(df->ctx);
+#endif
+
pg_free(df->pathname);
pg_free(df->fullpath);
if (df->temp_suffix)
@@ -317,6 +459,21 @@ dir_sync(Walfile f)
return -1;
}
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ DirectoryMethodFile *df = (DirectoryMethodFile *) f;
+ size_t compressed;
+
+ /* Flush any internal buffers */
+ compressed = LZ4F_flush(df->ctx, df->lz4buf, df->lz4bufsize, NULL);
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+ }
+#endif
return fsync(((DirectoryMethodFile *) f)->fd);
}
diff --git a/src/bin/pg_basebackup/walmethods.h b/src/bin/pg_basebackup/walmethods.h
index 41b83dfdfe..ce21167eb7 100644
--- a/src/bin/pg_basebackup/walmethods.h
+++ b/src/bin/pg_basebackup/walmethods.h
@@ -21,6 +21,7 @@ typedef enum
typedef enum
{
+ COMPRESSION_LZ4,
COMPRESSION_GZIP,
COMPRESSION_NONE
} WalCompressionMethod;
--
2.25.1
v9-0001-Refactor-pg_receivewal-in-preparation-for-introdu.patchtext/x-patch; name=v9-0001-Refactor-pg_receivewal-in-preparation-for-introdu.patchDownload
From 6682123d7122ac8ea574a3c77a7637e9f81ecbce Mon Sep 17 00:00:00 2001
From: Georgios Kokolatos <gkokolatos@pm.me>
Date: Wed, 3 Nov 2021 08:51:13 +0000
Subject: [PATCH v9 1/2] Refactor pg_receivewal in preparation for introducing
lz4 compression
The program pg_receivewal can use gzip compression to store the received WAL.
The option `--compress` with a value [1, 9] was used to denote that gzip
compression was required. When `--compress` with a value of `0` was used, then
no compression would take place.
This commit introduces a new option, `--compression-method`. Valid values are
[none|gzip]. The option `--compress` requires for `--compression-method` with
value other than `none`. Also `--compress=0` now returns an error.
Under the hood, there are no surprising changes. A new enum WalCompressionMethod
has been introduced and is used throughout the relevant codepaths to explicitly
note which compression method to use.
Last, the macros IsXLogFileName and friends, have been replaced by the function
is_xlogfilename(). This will allow for easier expansion of the available
compression methods that can be recognised.
---
doc/src/sgml/ref/pg_receivewal.sgml | 24 ++-
src/bin/pg_basebackup/pg_basebackup.c | 7 +-
src/bin/pg_basebackup/pg_receivewal.c | 164 +++++++++++++------
src/bin/pg_basebackup/receivelog.c | 2 +-
src/bin/pg_basebackup/t/020_pg_receivewal.pl | 16 +-
src/bin/pg_basebackup/walmethods.c | 51 ++++--
src/bin/pg_basebackup/walmethods.h | 15 +-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 200 insertions(+), 80 deletions(-)
diff --git a/doc/src/sgml/ref/pg_receivewal.sgml b/doc/src/sgml/ref/pg_receivewal.sgml
index 9fde2fd2ef..cf2eaa1486 100644
--- a/doc/src/sgml/ref/pg_receivewal.sgml
+++ b/doc/src/sgml/ref/pg_receivewal.sgml
@@ -263,15 +263,31 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--compression-method=<replaceable class="parameter">level</replaceable></option></term>
+ <listitem>
+ <para>
+ Enables compression of write-ahead logs using the specified method.
+ Supported values <literal>gzip</literal>,
+ and <literal>none</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-Z <replaceable class="parameter">level</replaceable></option></term>
<term><option>--compress=<replaceable class="parameter">level</replaceable></option></term>
<listitem>
<para>
- Enables gzip compression of write-ahead logs, and specifies the
- compression level (0 through 9, 0 being no compression and 9 being best
- compression). The suffix <filename>.gz</filename> will
- automatically be added to all filenames.
+ Specifies the compression level (<literal>1</literal> through
+ <literal>9</literal>, <literal>1</literal> being worst compression
+ and <literal>9</literal> being best compression) for
+ <application>gzip</application> compressed WAL segments.
+ </para>
+
+ <para>
+ This option requires <option>--compression-method</option> to be
+ specified with <literal>gzip</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c
index 27ee6394cf..cdea3711b7 100644
--- a/src/bin/pg_basebackup/pg_basebackup.c
+++ b/src/bin/pg_basebackup/pg_basebackup.c
@@ -555,10 +555,13 @@ LogStreamerMain(logstreamer_param *param)
stream.replication_slot = replication_slot;
if (format == 'p')
- stream.walmethod = CreateWalDirectoryMethod(param->xlog, 0,
+ stream.walmethod = CreateWalDirectoryMethod(param->xlog,
+ COMPRESSION_NONE, 0,
stream.do_sync);
else
- stream.walmethod = CreateWalTarMethod(param->xlog, compresslevel,
+ stream.walmethod = CreateWalTarMethod(param->xlog,
+ COMPRESSION_NONE, /* ignored */
+ compresslevel,
stream.do_sync);
if (!ReceiveXlogStream(param->bgconn, &stream))
diff --git a/src/bin/pg_basebackup/pg_receivewal.c b/src/bin/pg_basebackup/pg_receivewal.c
index 04ba20b197..9449b50868 100644
--- a/src/bin/pg_basebackup/pg_receivewal.c
+++ b/src/bin/pg_basebackup/pg_receivewal.c
@@ -32,6 +32,9 @@
/* Time to sleep between reconnection attempts */
#define RECONNECT_SLEEP_TIME 5
+/* This is just the redefinition of a libz constant */
+#define Z_DEFAULT_COMPRESSION (-1)
+
/* Global options */
static char *basedir = NULL;
static int verbose = 0;
@@ -45,6 +48,7 @@ static bool do_drop_slot = false;
static bool do_sync = true;
static bool synchronous = false;
static char *replication_slot = NULL;
+static WalCompressionMethod compression_method = COMPRESSION_NONE;
static XLogRecPtr endpos = InvalidXLogRecPtr;
@@ -63,16 +67,6 @@ disconnect_atexit(void)
PQfinish(conn);
}
-/* Routines to evaluate segment file format */
-#define IsCompressXLogFileName(fname) \
- (strlen(fname) == XLOG_FNAME_LEN + strlen(".gz") && \
- strspn(fname, "0123456789ABCDEF") == XLOG_FNAME_LEN && \
- strcmp((fname) + XLOG_FNAME_LEN, ".gz") == 0)
-#define IsPartialCompressXLogFileName(fname) \
- (strlen(fname) == XLOG_FNAME_LEN + strlen(".gz.partial") && \
- strspn(fname, "0123456789ABCDEF") == XLOG_FNAME_LEN && \
- strcmp((fname) + XLOG_FNAME_LEN, ".gz.partial") == 0)
-
static void
usage(void)
{
@@ -92,7 +86,9 @@ usage(void)
printf(_(" --synchronous flush write-ahead log immediately after writing\n"));
printf(_(" -v, --verbose output verbose messages\n"));
printf(_(" -V, --version output version information, then exit\n"));
- printf(_(" -Z, --compress=0-9 compress logs with given compression level\n"));
+ printf(_(" --compression-method=METHOD\n"
+ " method to compress logs\n"));
+ printf(_(" -Z, --compress=1-9 compress logs with given compression level\n"));
printf(_(" -?, --help show this help, then exit\n"));
printf(_("\nConnection options:\n"));
printf(_(" -d, --dbname=CONNSTR connection string\n"));
@@ -108,6 +104,61 @@ usage(void)
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
}
+
+/*
+ * Check if the filename looks like an xlog file. Also note if it is partial
+ * and/or compressed file.
+ */
+static bool
+is_xlogfilename(const char *filename, bool *ispartial,
+ WalCompressionMethod *wal_compression_method)
+{
+ size_t fname_len = strlen(filename);
+ size_t xlog_pattern_len = strspn(filename, "0123456789ABCDEF");
+
+ /* File does not look like a XLOG file */
+ if (xlog_pattern_len != XLOG_FNAME_LEN)
+ return false;
+
+ /* File looks like a complete uncompressed XLOG file */
+ if (fname_len == XLOG_FNAME_LEN)
+ {
+ *ispartial = false;
+ *wal_compression_method = COMPRESSION_NONE;
+ return true;
+ }
+
+ /* File looks like a complete gzip compressed XLOG file */
+ if (fname_len == XLOG_FNAME_LEN + strlen(".gz") &&
+ strcmp(filename + XLOG_FNAME_LEN, ".gz") == 0)
+ {
+ *ispartial = false;
+ *wal_compression_method = COMPRESSION_GZIP;
+ return true;
+ }
+
+ /* File looks like a partial uncompressed XLOG file */
+ if (fname_len == XLOG_FNAME_LEN + strlen(".partial") &&
+ strcmp(filename + XLOG_FNAME_LEN, ".partial") == 0)
+ {
+ *ispartial = true;
+ *wal_compression_method = COMPRESSION_NONE;
+ return true;
+ }
+
+ /* File looks like a partial gzip compressed XLOG file */
+ if (fname_len == XLOG_FNAME_LEN + strlen(".gz.partial") &&
+ strcmp(filename + XLOG_FNAME_LEN, ".gz.partial") == 0)
+ {
+ *ispartial = true;
+ *wal_compression_method = COMPRESSION_GZIP;
+ return true;
+ }
+
+ /* File does not look like something we recognise */
+ return false;
+}
+
static bool
stop_streaming(XLogRecPtr xlogpos, uint32 timeline, bool segment_finished)
{
@@ -213,33 +264,11 @@ FindStreamingStart(uint32 *tli)
{
uint32 tli;
XLogSegNo segno;
+ WalCompressionMethod wal_compression_method;
bool ispartial;
- bool iscompress;
- /*
- * Check if the filename looks like an xlog file, or a .partial file.
- */
- if (IsXLogFileName(dirent->d_name))
- {
- ispartial = false;
- iscompress = false;
- }
- else if (IsPartialXLogFileName(dirent->d_name))
- {
- ispartial = true;
- iscompress = false;
- }
- else if (IsCompressXLogFileName(dirent->d_name))
- {
- ispartial = false;
- iscompress = true;
- }
- else if (IsPartialCompressXLogFileName(dirent->d_name))
- {
- ispartial = true;
- iscompress = true;
- }
- else
+ if (!is_xlogfilename(dirent->d_name,
+ &ispartial, &wal_compression_method))
continue;
/*
@@ -250,14 +279,14 @@ FindStreamingStart(uint32 *tli)
/*
* Check that the segment has the right size, if it's supposed to be
* completed. For non-compressed segments just check the on-disk size
- * and see if it matches a completed segment. For compressed segments,
- * look at the last 4 bytes of the compressed file, which is where the
- * uncompressed size is located for gz files with a size lower than
- * 4GB, and then compare it to the size of a completed segment. The 4
- * last bytes correspond to the ISIZE member according to
- * http://www.zlib.org/rfc-gzip.html.
+ * and see if it matches a completed segment. For gzip compressed
+ * segments, look at the last 4 bytes of the compressed file, which is
+ * where the uncompressed size is located for gz files with a size
+ * lower than 4GB, and then compare it to the size of a completed
+ * segment. The 4 last bytes correspond to the ISIZE member according
+ * to http://www.zlib.org/rfc-gzip.html.
*/
- if (!ispartial && !iscompress)
+ if (!ispartial && wal_compression_method == COMPRESSION_NONE)
{
struct stat statbuf;
char fullpath[MAXPGPATH * 2];
@@ -276,7 +305,7 @@ FindStreamingStart(uint32 *tli)
continue;
}
}
- else if (!ispartial && iscompress)
+ else if (!ispartial && wal_compression_method == COMPRESSION_GZIP)
{
int fd;
char buf[4];
@@ -457,7 +486,9 @@ StreamLog(void)
stream.synchronous = synchronous;
stream.do_sync = do_sync;
stream.mark_done = false;
- stream.walmethod = CreateWalDirectoryMethod(basedir, compresslevel,
+ stream.walmethod = CreateWalDirectoryMethod(basedir,
+ compression_method,
+ compresslevel,
stream.do_sync);
stream.partial_suffix = ".partial";
stream.replication_slot = replication_slot;
@@ -510,6 +541,7 @@ main(int argc, char **argv)
{"status-interval", required_argument, NULL, 's'},
{"slot", required_argument, NULL, 'S'},
{"verbose", no_argument, NULL, 'v'},
+ {"compression-method", required_argument, NULL, 'I'},
{"compress", required_argument, NULL, 'Z'},
/* action */
{"create-slot", no_argument, NULL, 1},
@@ -595,8 +627,20 @@ main(int argc, char **argv)
case 'v':
verbose++;
break;
+ case 'I':
+ if (pg_strcasecmp(optarg, "gzip") == 0)
+ compression_method = COMPRESSION_GZIP;
+ else if (pg_strcasecmp(optarg, "none") == 0)
+ compression_method = COMPRESSION_NONE;
+ else
+ {
+ pg_log_error("invalid value \"%s\" for option %s",
+ optarg, "--compress-method");
+ exit(1);
+ }
+ break;
case 'Z':
- if (!option_parse_int(optarg, "-Z/--compress", 0, 9,
+ if (!option_parse_int(optarg, "-Z/--compress", 1, 9,
&compresslevel))
exit(1);
break;
@@ -676,13 +720,35 @@ main(int argc, char **argv)
exit(1);
}
+
+ /*
+ * Compression related arguments
+ */
+ if (compression_method != COMPRESSION_NONE)
+ {
#ifndef HAVE_LIBZ
- if (compresslevel != 0)
+ if (compression_method == COMPRESSION_GZIP)
+ {
+ pg_log_error("this build does not support compression with %s",
+ "gzip");
+ exit(1);
+ }
+#endif
+ }
+
+ if (compression_method != COMPRESSION_GZIP && compresslevel != 0)
{
- pg_log_error("this build does not support compression");
+ pg_log_error("can only use --compress with --compression-method=gzip");
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+ progname);
exit(1);
}
-#endif
+
+ if (compression_method == COMPRESSION_GZIP && compresslevel == 0)
+ {
+ pg_log_info("no --compression specified, will be using the library default");
+ compresslevel = Z_DEFAULT_COMPRESSION;
+ }
/*
* Check existence of destination folder.
diff --git a/src/bin/pg_basebackup/receivelog.c b/src/bin/pg_basebackup/receivelog.c
index 72b8d9e315..2d4f660daa 100644
--- a/src/bin/pg_basebackup/receivelog.c
+++ b/src/bin/pg_basebackup/receivelog.c
@@ -109,7 +109,7 @@ open_walfile(StreamCtl *stream, XLogRecPtr startpoint)
* When streaming to tar, no file with this name will exist before, so we
* never have to verify a size.
*/
- if (stream->walmethod->compression() == 0 &&
+ if (stream->walmethod->compression_method() == COMPRESSION_NONE &&
stream->walmethod->existsfile(fn))
{
size = stream->walmethod->get_file_size(fn);
diff --git a/src/bin/pg_basebackup/t/020_pg_receivewal.pl b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
index ab05f9e91e..251ac247d8 100644
--- a/src/bin/pg_basebackup/t/020_pg_receivewal.pl
+++ b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
@@ -5,7 +5,7 @@ use strict;
use warnings;
use PostgreSQL::Test::Utils;
use PostgreSQL::Test::Cluster;
-use Test::More tests => 35;
+use Test::More tests => 37;
program_help_ok('pg_receivewal');
program_version_ok('pg_receivewal');
@@ -33,6 +33,13 @@ $primary->command_fails(
$primary->command_fails(
[ 'pg_receivewal', '-D', $stream_dir, '--synchronous', '--no-sync' ],
'failure if --synchronous specified with --no-sync');
+$primary->command_fails_like(
+ [
+ 'pg_receivewal', '-D', $stream_dir, '--compression-method', 'none',
+ '--compress', '1'
+ ],
+ qr/\Qpg_receivewal: error: can only use --compress with --compression-method=gzip/,
+ 'failure if --compression-method=none specified with --compress');
# Slot creation and drop
my $slot_name = 'test';
@@ -90,8 +97,11 @@ SKIP:
# a valid value.
$primary->command_ok(
[
- 'pg_receivewal', '-D', $stream_dir, '--verbose',
- '--endpos', $nextlsn, '--compress', '1 ',
+ 'pg_receivewal', '-D',
+ $stream_dir, '--verbose',
+ '--endpos', $nextlsn,
+ '--compression-method', 'gzip',
+ '--compress', '1 ',
'--no-loop'
],
"streaming some WAL using ZLIB compression");
diff --git a/src/bin/pg_basebackup/walmethods.c b/src/bin/pg_basebackup/walmethods.c
index 8695647db4..b710b1ef36 100644
--- a/src/bin/pg_basebackup/walmethods.c
+++ b/src/bin/pg_basebackup/walmethods.c
@@ -41,6 +41,7 @@
typedef struct DirectoryMethodData
{
char *basedir;
+ WalCompressionMethod compression_method;
int compression;
bool sync;
} DirectoryMethodData;
@@ -74,7 +75,8 @@ dir_get_file_name(const char *pathname, const char *temp_suffix)
char *filename = pg_malloc0(MAXPGPATH * sizeof(char));
snprintf(filename, MAXPGPATH, "%s%s%s",
- pathname, dir_data->compression > 0 ? ".gz" : "",
+ pathname,
+ dir_data->compression_method == COMPRESSION_GZIP ? ".gz" : "",
temp_suffix ? temp_suffix : "");
return filename;
@@ -107,7 +109,7 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
return NULL;
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_GZIP)
{
gzfp = gzdopen(fd, "wb");
if (gzfp == NULL)
@@ -126,7 +128,7 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
#endif
/* Do pre-padding on non-compressed files */
- if (pad_to_size && dir_data->compression == 0)
+ if (pad_to_size && dir_data->compression_method == COMPRESSION_NONE)
{
PGAlignedXLogBlock zerobuf;
int bytes;
@@ -171,7 +173,7 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
fsync_parent_path(tmppath) != 0)
{
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_GZIP)
gzclose(gzfp);
else
#endif
@@ -182,7 +184,7 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
f = pg_malloc0(sizeof(DirectoryMethodFile));
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_GZIP)
f->gzfp = gzfp;
#endif
f->fd = fd;
@@ -204,7 +206,7 @@ dir_write(Walfile f, const void *buf, size_t count)
Assert(f != NULL);
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_GZIP)
r = (ssize_t) gzwrite(df->gzfp, buf, count);
else
#endif
@@ -234,7 +236,7 @@ dir_close(Walfile f, WalCloseMethod method)
Assert(f != NULL);
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_GZIP)
r = gzclose(df->gzfp);
else
#endif
@@ -309,7 +311,7 @@ dir_sync(Walfile f)
return 0;
#ifdef HAVE_LIBZ
- if (dir_data->compression > 0)
+ if (dir_data->compression_method == COMPRESSION_GZIP)
{
if (gzflush(((DirectoryMethodFile *) f)->gzfp, Z_SYNC_FLUSH) != Z_OK)
return -1;
@@ -334,10 +336,10 @@ dir_get_file_size(const char *pathname)
return statbuf.st_size;
}
-static int
-dir_compression(void)
+static WalCompressionMethod
+dir_compression_method(void)
{
- return dir_data->compression;
+ return dir_data->compression_method;
}
static bool
@@ -373,7 +375,9 @@ dir_finish(void)
WalWriteMethod *
-CreateWalDirectoryMethod(const char *basedir, int compression, bool sync)
+CreateWalDirectoryMethod(const char *basedir,
+ WalCompressionMethod compression_method,
+ int compression, bool sync)
{
WalWriteMethod *method;
@@ -383,7 +387,7 @@ CreateWalDirectoryMethod(const char *basedir, int compression, bool sync)
method->get_current_pos = dir_get_current_pos;
method->get_file_size = dir_get_file_size;
method->get_file_name = dir_get_file_name;
- method->compression = dir_compression;
+ method->compression_method = dir_compression_method;
method->close = dir_close;
method->sync = dir_sync;
method->existsfile = dir_existsfile;
@@ -391,6 +395,7 @@ CreateWalDirectoryMethod(const char *basedir, int compression, bool sync)
method->getlasterror = dir_getlasterror;
dir_data = pg_malloc0(sizeof(DirectoryMethodData));
+ dir_data->compression_method = compression_method;
dir_data->compression = compression;
dir_data->basedir = pg_strdup(basedir);
dir_data->sync = sync;
@@ -424,6 +429,7 @@ typedef struct TarMethodData
{
char *tarfilename;
int fd;
+ WalCompressionMethod compression_method;
int compression;
bool sync;
TarMethodFile *currentfile;
@@ -731,10 +737,10 @@ tar_get_file_size(const char *pathname)
return -1;
}
-static int
-tar_compression(void)
+static WalCompressionMethod
+tar_compression_method(void)
{
- return tar_data->compression;
+ return tar_data->compression_method;
}
static off_t
@@ -1031,8 +1037,16 @@ tar_finish(void)
return true;
}
+/*
+ * The argument compression_method is currently ignored. It is in place for
+ * symmetry with CreateWalDirectoryMethod which uses it for distinguishing
+ * between the different compression methods. CreateWalTarMethod and its family
+ * of functions handle only zlib compression.
+ */
WalWriteMethod *
-CreateWalTarMethod(const char *tarbase, int compression, bool sync)
+CreateWalTarMethod(const char *tarbase,
+ WalCompressionMethod compression_method,
+ int compression, bool sync)
{
WalWriteMethod *method;
const char *suffix = (compression != 0) ? ".tar.gz" : ".tar";
@@ -1043,7 +1057,7 @@ CreateWalTarMethod(const char *tarbase, int compression, bool sync)
method->get_current_pos = tar_get_current_pos;
method->get_file_size = tar_get_file_size;
method->get_file_name = tar_get_file_name;
- method->compression = tar_compression;
+ method->compression_method = tar_compression_method;
method->close = tar_close;
method->sync = tar_sync;
method->existsfile = tar_existsfile;
@@ -1054,6 +1068,7 @@ CreateWalTarMethod(const char *tarbase, int compression, bool sync)
tar_data->tarfilename = pg_malloc0(strlen(tarbase) + strlen(suffix) + 1);
sprintf(tar_data->tarfilename, "%s%s", tarbase, suffix);
tar_data->fd = -1;
+ tar_data->compression_method = compression_method;
tar_data->compression = compression;
tar_data->sync = sync;
#ifdef HAVE_LIBZ
diff --git a/src/bin/pg_basebackup/walmethods.h b/src/bin/pg_basebackup/walmethods.h
index 4abdfd8333..41b83dfdfe 100644
--- a/src/bin/pg_basebackup/walmethods.h
+++ b/src/bin/pg_basebackup/walmethods.h
@@ -19,6 +19,12 @@ typedef enum
CLOSE_NO_RENAME
} WalCloseMethod;
+typedef enum
+{
+ COMPRESSION_GZIP,
+ COMPRESSION_NONE
+} WalCompressionMethod;
+
/*
* A WalWriteMethod structure represents the different methods used
* to write the streaming WAL as it's received.
@@ -58,8 +64,8 @@ struct WalWriteMethod
*/
char *(*get_file_name) (const char *pathname, const char *temp_suffix);
- /* Return the level of compression */
- int (*compression) (void);
+ /* Returns the compression method */
+ WalCompressionMethod (*compression_method) (void);
/*
* Write count number of bytes to the file, and return the number of bytes
@@ -95,8 +101,11 @@ struct WalWriteMethod
* not all those required for pg_receivewal)
*/
WalWriteMethod *CreateWalDirectoryMethod(const char *basedir,
+ WalCompressionMethod compression_method,
int compression, bool sync);
-WalWriteMethod *CreateWalTarMethod(const char *tarbase, int compression, bool sync);
+WalWriteMethod *CreateWalTarMethod(const char *tarbase,
+ WalCompressionMethod compression_method,
+ int compression, bool sync);
/* Cleanup routines for previously-created methods */
void FreeWalDirectoryMethod(void);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 7bbbb34e2f..da6ac8ed83 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2858,6 +2858,7 @@ WaitEventTimeout
WaitPMResult
WalCloseMethod
WalCompression
+WalCompressionMethod
WalLevel
WalRcvData
WalRcvExecResult
--
2.25.1
Re: Teach pg_receivewal to use lz4 compression
On Wed, Nov 03, 2021 at 09:11:24AM +0000, gkokolatos@pm.me wrote:
Please find v9 attached.
Thanks. I have looked at 0001 today, and applied it after fixing a
couple of issues. From memory:
- zlib.h was missing from pg_receivewal.c, issue that I noticed after
removing the redefinition of Z_DEFAULT_COMPRESSION because there was
no need for it (did a run with a --without-zlib as well).
- Simplified a bit the error handling for incorrect option
combinations, using a switch/case while on it.
- Renamed the existing variable "compression" in walmethods.c to
compression_level, to reduce any confusion with the introduction of
compression_method. One thing I have noticed is about the tar method,
where we rely on the compression level to decide if compression should
be used or not. There should be some simplifications possible there
but there is a huge take in receivelog.c where we use COMPRESSION_NONE
to track down that we still want to zero a new segment when using tar
method.
- Use of 'I' as short option name, err... After applying the first
batch..
Based on the work of 0001, there were some conflicts with 0002. I
have solved them while reviewing it, and adapted the code to what got
already applied.
+ header_size = LZ4F_compressBegin(ctx, lz4buf, lz4bufsize, NULL);
+ if (LZ4F_isError(header_size))
+ {
+ pg_free(lz4buf);
+ close(fd);
+ return NULL;
+ }
In dir_open_for_write(), I guess that this one is missing one
LZ4F_freeCompressionContext().+ status = LZ4F_freeDecompressionContext(ctx);
+ if (LZ4F_isError(status))
+ {
+ pg_log_error("could not free LZ4 decompression context: %s",
+ LZ4F_getErrorName(status));
+ exit(1);
+ }
+
+ if (uncompressed_size != WalSegSz)
+ {
+ pg_log_warning("compressed segment file \"%s\" has
incorrect uncompressed size %ld, skipping",
+ dirent->d_name, uncompressed_size);
+ (void) LZ4F_freeDecompressionContext(ctx);
+ continue;
+ }
When the uncompressed size does not match out expected size, the
second LZ4F_freeDecompressionContext() looks unnecessary to me because
we have already one a couple of lines above.+ ctx_out = LZ4F_createCompressionContext(&ctx, LZ4F_VERSION);
+ lz4bufsize = LZ4F_compressBound(LZ4_IN_SIZE, NULL);
+ if (LZ4F_isError(ctx_out))
+ {
+ close(fd);
+ return NULL;
+ }
LZ4F_compressBound() can be after the check on ctx_out, here.+ while (1)
+ {
+ char *readp;
+ char *readend;
Simply looping when decompressing a segment to check its size looks
rather unsafe to me. We should leave the loop once uncompressed_size
is strictly more than WalSegSz.The amount of TAP tests looks fine, and that's consistent with what we
do for zlib^D^D^D^Dgzip. Now, when testing manually pg_receivewal
with various combinations of gzip, lz4 and none, I can see the
following failure in the code that calculates the streaming start
point:
pg_receivewal: error: could not decompress file
"wals//000000010000000000000006.lz4": ERROR_frameType_unknown
In the LZ4 code, this points to lib/lz4frame.c, where we read an
incorrect header (see the part that does not match LZ4F_MAGICNUMBER).
The segments written by pg_receivewal are clean. Please note that
this shows up as well when manually compressing some segments with a
simple lz4 command, to simulate for example the case where a user
compressed some segments by himself/herself before running
pg_receivewal.
So, tour code does LZ4F_createDecompressionContext() followed by a
loop on read() and LZ4F_decompress() that relies on an input and an
output buffer of a fixed 4kB size (we could use 64kB at least here
actually?). So this set of loops looks rather correct to me.
Now, this part is weird:
+ while (readp < readend)
+ {
+ size_t read_size = 1;
+ size_t out_size = 1;I would have expected read_size to be (readend - readp) to match with
the remaining data in the read buffer that we still need to read.
Shouldn't out_size also be LZ4_CHUNK_SZ to match with the size of the
output buffer where all the contents are read? By setting it to 1, I
think that this is doing more loops into LZ4F_decompress() than really
necessary. It would not hurt either to memset(0) those buffers before
they are used, IMO. I am not completely sure either, but should we
use the number of bytes returned by LZ4F_decompress() as a hint for
the follow-up loops?
Attached is an updated patch, which includes fixes for most of the
issues I am mentioning above. Please note that I have not changed
FindStreamingStart(), so this part is the same as v9.
Thanks,
--
Michael
Attachments:
v10-0001-Teach-pg_receivewal-to-use-LZ4-compression.patchtext/x-diff; charset=us-asciiDownload
From 51a2352d2e543795ec201ed444d4e74014c3a2d3 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 4 Nov 2021 16:24:48 +0900
Subject: [PATCH v10] Teach pg_receivewal to use LZ4 compression
The program pg_receivewal can use gzip compression to store the received
WAL. This commit teaches it to also be able to use LZ4 compression. It
is required that the binary is build using the -llz4 flag. It is enabled
via the --with-lz4 flag on configuration time.
The option `--compression-method` has been expanded to inlude the value
[LZ4]. The option `--compress` can not be used with LZ4 compression.
Under the hood there is nothing exceptional to be noted. Tar based
archives have not yet been taught to use LZ4 compression. If that is
felt useful, then it is easy to be added in the future.
Tests have been added to verify the creation and correctness of the
generated LZ4 files. The later is achieved by the use of LZ4 program, if
present in the installation.
---
src/bin/pg_basebackup/Makefile | 1 +
src/bin/pg_basebackup/pg_receivewal.c | 144 +++++++++++++++++
src/bin/pg_basebackup/t/020_pg_receivewal.pl | 72 ++++++++-
src/bin/pg_basebackup/walmethods.c | 160 ++++++++++++++++++-
src/bin/pg_basebackup/walmethods.h | 1 +
doc/src/sgml/ref/pg_receivewal.sgml | 8 +-
src/Makefile.global.in | 1 +
7 files changed, 375 insertions(+), 12 deletions(-)
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index 459d514183..fd920fc197 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -19,6 +19,7 @@ top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
# make these available to TAP test scripts
+export LZ4
export TAR
# Note that GZIP cannot be used directly as this environment variable is
# used by the command "gzip" to pass down options, so stick with a different
diff --git a/src/bin/pg_basebackup/pg_receivewal.c b/src/bin/pg_basebackup/pg_receivewal.c
index 8acc0fc009..6d090d4c50 100644
--- a/src/bin/pg_basebackup/pg_receivewal.c
+++ b/src/bin/pg_basebackup/pg_receivewal.c
@@ -32,6 +32,10 @@
#include "receivelog.h"
#include "streamutil.h"
+#ifdef HAVE_LIBLZ4
+#include "lz4frame.h"
+#endif
+
/* Time to sleep between reconnection attempts */
#define RECONNECT_SLEEP_TIME 5
@@ -136,6 +140,15 @@ is_xlogfilename(const char *filename, bool *ispartial,
return true;
}
+ /* File looks like a completed LZ4-compressed WAL file */
+ if (fname_len == XLOG_FNAME_LEN + strlen(".lz4") &&
+ strcmp(filename + XLOG_FNAME_LEN, ".lz4") == 0)
+ {
+ *ispartial = false;
+ *wal_compression_method = COMPRESSION_LZ4;
+ return true;
+ }
+
/* File looks like a partial uncompressed WAL file */
if (fname_len == XLOG_FNAME_LEN + strlen(".partial") &&
strcmp(filename + XLOG_FNAME_LEN, ".partial") == 0)
@@ -154,6 +167,15 @@ is_xlogfilename(const char *filename, bool *ispartial,
return true;
}
+ /* File looks like a partial LZ4-compressed WAL file */
+ if (fname_len == XLOG_FNAME_LEN + strlen(".lz4.partial") &&
+ strcmp(filename + XLOG_FNAME_LEN, ".lz4.partial") == 0)
+ {
+ *ispartial = true;
+ *wal_compression_method = COMPRESSION_LZ4;
+ return true;
+ }
+
/* File does not look like something we know */
return false;
}
@@ -284,6 +306,14 @@ FindStreamingStart(uint32 *tli)
* than 4GB, and then compare it to the size of a completed segment.
* The 4 last bytes correspond to the ISIZE member according to
* http://www.zlib.org/rfc-gzip.html.
+ *
+ * For LZ4 compressed segments, uncompress the file in a throw-away
+ * buffer keeping track of the uncompressed size, then compare it to
+ * the size of a completed segment. Per its protocol, LZ4 does not
+ * store the uncompressed size of an object by default. contentSize
+ * is one possible way to do that, but we need to rely on a method
+ * where WAL segments could have been compressed by a different
+ * source than pg_receivewal, like an archive_command.
*/
if (!ispartial && wal_compression_method == COMPRESSION_NONE)
{
@@ -315,6 +345,7 @@ FindStreamingStart(uint32 *tli)
snprintf(fullpath, sizeof(fullpath), "%s/%s", basedir, dirent->d_name);
fd = open(fullpath, O_RDONLY | PG_BINARY, 0);
+
if (fd < 0)
{
pg_log_error("could not open compressed file \"%s\": %m",
@@ -350,6 +381,98 @@ FindStreamingStart(uint32 *tli)
continue;
}
}
+ else if (!ispartial && compression_method == COMPRESSION_LZ4)
+ {
+#ifdef HAVE_LIBLZ4
+#define LZ4_CHUNK_SZ 4096
+ int fd;
+ int r;
+ size_t uncompressed_size = 0;
+ char fullpath[MAXPGPATH * 2];
+ char readbuf[LZ4_CHUNK_SZ];
+ char outbuf[LZ4_CHUNK_SZ];
+ LZ4F_decompressionContext_t ctx = NULL;
+ LZ4F_errorCode_t status;
+
+ snprintf(fullpath, sizeof(fullpath), "%s/%s", basedir, dirent->d_name);
+
+ fd = open(fullpath, O_RDONLY | PG_BINARY, 0);
+ if (fd < 0)
+ {
+ pg_log_error("could not open file \"%s\": %m", fullpath);
+ exit(1);
+ }
+
+ status = LZ4F_createDecompressionContext(&ctx, LZ4F_VERSION);
+ if (LZ4F_isError(status))
+ {
+ pg_log_error("could not create LZ4 decompression context: %s",
+ LZ4F_getErrorName(status));
+ exit(1);
+ }
+
+ while (1)
+ {
+ char *readp;
+ char *readend;
+
+ r = read(fd, readbuf, sizeof(readbuf));
+ if (r < 0)
+ {
+ pg_log_error("could not read file \"%s\": %m", fullpath);
+ exit(1);
+ }
+
+ /* Done reading */
+ if (r == 0)
+ break;
+
+ readp = readbuf;
+ readend = readbuf + r;
+ while (readp < readend)
+ {
+ size_t read_size = 1;
+ size_t out_size = 1;
+
+ status = LZ4F_decompress(ctx, outbuf, &out_size,
+ readbuf, &read_size, NULL);
+ if (LZ4F_isError(status))
+ {
+ pg_log_error("could not decompress file \"%s\": %s",
+ fullpath,
+ LZ4F_getErrorName(status));
+ exit(1);
+ }
+
+ readp += read_size;
+ uncompressed_size += out_size;
+ }
+ }
+
+ close(fd);
+
+ status = LZ4F_freeDecompressionContext(ctx);
+ if (LZ4F_isError(status))
+ {
+ pg_log_error("could not free LZ4 decompression context: %s",
+ LZ4F_getErrorName(status));
+ exit(1);
+ }
+
+ if (uncompressed_size != WalSegSz)
+ {
+ pg_log_warning("compressed segment file \"%s\" has incorrect uncompressed size %ld, skipping",
+ dirent->d_name, uncompressed_size);
+ continue;
+ }
+#else
+ pg_log_error("could not check segment file \"%s\" compressed with LZ4",
+ dirent->d_name);
+ pg_log_error("this build does not support compression with %s",
+ "LZ4");
+ exit(1);
+#endif
+ }
/* Looks like a valid segment. Remember that we saw it. */
if ((segno > high_segno) ||
@@ -650,6 +773,8 @@ main(int argc, char **argv)
case 6:
if (pg_strcasecmp(optarg, "gzip") == 0)
compression_method = COMPRESSION_GZIP;
+ else if (pg_strcasecmp(optarg, "lz4") == 0)
+ compression_method = COMPRESSION_LZ4;
else if (pg_strcasecmp(optarg, "none") == 0)
compression_method = COMPRESSION_NONE;
else
@@ -748,6 +873,25 @@ main(int argc, char **argv)
exit(1);
#endif
break;
+ case COMPRESSION_LZ4:
+#ifdef HAVE_LIBLZ4
+ if (compresslevel != 0)
+ {
+ pg_log_error("cannot use --compress with --compression-method=%s",
+ "lz4");
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+ progname);
+ exit(1);
+ }
+#else
+ if (compression_method == COMPRESSION_LZ4)
+ {
+ pg_log_error("this build does not support compression with %s",
+ "LZ4");
+ exit(1);
+ }
+ break;
+#endif
}
diff --git a/src/bin/pg_basebackup/t/020_pg_receivewal.pl b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
index 94786f0815..43599d832b 100644
--- a/src/bin/pg_basebackup/t/020_pg_receivewal.pl
+++ b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
@@ -5,7 +5,7 @@ use strict;
use warnings;
use PostgreSQL::Test::Utils;
use PostgreSQL::Test::Cluster;
-use Test::More tests => 37;
+use Test::More tests => 42;
program_help_ok('pg_receivewal');
program_version_ok('pg_receivewal');
@@ -138,13 +138,69 @@ SKIP:
"gzip verified the integrity of compressed WAL segments");
}
+# Check LZ4 compression if available
+SKIP:
+{
+ skip "postgres was not built with LZ4 support", 5
+ if (!check_pg_config("#define HAVE_LIBLZ4 1"));
+
+ # Generate more WAL including one completed, compressed segment.
+ $primary->psql('postgres', 'SELECT pg_switch_wal();');
+ $nextlsn =
+ $primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
+ chomp($nextlsn);
+ $primary->psql('postgres', 'INSERT INTO test_table VALUES (3);');
+
+ # Stream up to the given position.
+ $primary->command_ok(
+ [
+ 'pg_receivewal', '-D',
+ $stream_dir, '--verbose',
+ '--endpos', $nextlsn,
+ '--no-loop', '--compression-method',
+ 'lz4'
+ ],
+ 'streaming some WAL using --compression-method=lz4');
+
+ # Verify that the stored files are generated with their expected
+ # names.
+ my @lz4_wals = glob "$stream_dir/*.lz4";
+ is(scalar(@lz4_wals), 1,
+ "one WAL segment compressed with LZ4 was created");
+ my @lz4_partial_wals = glob "$stream_dir/*.lz4.partial";
+ is(scalar(@lz4_partial_wals),
+ 1, "one partial WAL segment compressed with LZ4 was created");
+
+ # Verify that the start streaming position is computed correctly by
+ # comparing it with the partial file generated previously. The name
+ # of the previous partial, now-completed WAL segment is updated, keeping
+ # its base number.
+ $partial_wals[0] =~ s/(\.gz)?\.partial$/.lz4/;
+ is($lz4_wals[0] eq $partial_wals[0],
+ 1, "one partial WAL segment is now completed");
+ # Update the list of partial wals with the current one.
+ @partial_wals = @lz4_partial_wals;
+
+ # Check the integrity of the completed segment, if LZ4 is an available
+ # command.
+ my $lz4 = $ENV{LZ4};
+ skip "program lz4 is not found in your system", 1
+ if ( !defined $lz4
+ || $lz4 eq ''
+ || system_log($lz4, '--version') != 0);
+
+ my $lz4_is_valid = system_log($lz4, '-t', @lz4_wals);
+ is($lz4_is_valid, 0,
+ "lz4 verified the integrity of compressed WAL segments");
+}
+
# Verify that the start streaming position is computed and that the value is
-# correct regardless of whether ZLIB is available.
+# correct regardless of whether any compression is available.
$primary->psql('postgres', 'SELECT pg_switch_wal();');
$nextlsn =
$primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
chomp($nextlsn);
-$primary->psql('postgres', 'INSERT INTO test_table VALUES (3);');
+$primary->psql('postgres', 'INSERT INTO test_table VALUES (4);');
$primary->command_ok(
[
'pg_receivewal', '-D', $stream_dir, '--verbose',
@@ -152,7 +208,7 @@ $primary->command_ok(
],
"streaming some WAL");
-$partial_wals[0] =~ s/(\.gz)?.partial//;
+$partial_wals[0] =~ s/(\.gz|\.lz4)?.partial//;
ok(-e $partial_wals[0], "check that previously partial WAL is now complete");
# Permissions on WAL files should be default
@@ -190,7 +246,7 @@ my $walfile_streamed = $primary->safe_psql(
# Switch to a new segment, to make sure that the segment retained by the
# slot is still streamed. This may not be necessary, but play it safe.
-$primary->psql('postgres', 'INSERT INTO test_table VALUES (4);');
+$primary->psql('postgres', 'INSERT INTO test_table VALUES (5);');
$primary->psql('postgres', 'SELECT pg_switch_wal();');
$nextlsn =
$primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
@@ -198,7 +254,7 @@ chomp($nextlsn);
# Add a bit more data to accelerate the end of the next pg_receivewal
# commands.
-$primary->psql('postgres', 'INSERT INTO test_table VALUES (5);');
+$primary->psql('postgres', 'INSERT INTO test_table VALUES (6);');
# Check case where the slot does not exist.
$primary->command_fails_like(
@@ -253,13 +309,13 @@ $standby->promote;
# on the new timeline.
my $walfile_after_promotion = $standby->safe_psql('postgres',
"SELECT pg_walfile_name(pg_current_wal_insert_lsn());");
-$standby->psql('postgres', 'INSERT INTO test_table VALUES (6);');
+$standby->psql('postgres', 'INSERT INTO test_table VALUES (7);');
$standby->psql('postgres', 'SELECT pg_switch_wal();');
$nextlsn =
$standby->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
chomp($nextlsn);
# This speeds up the operation.
-$standby->psql('postgres', 'INSERT INTO test_table VALUES (7);');
+$standby->psql('postgres', 'INSERT INTO test_table VALUES (8);');
# Now try to resume from the slot after the promotion.
my $timeline_dir = $primary->basedir . '/timeline_wal';
diff --git a/src/bin/pg_basebackup/walmethods.c b/src/bin/pg_basebackup/walmethods.c
index 52f314af3b..f1ba2a828a 100644
--- a/src/bin/pg_basebackup/walmethods.c
+++ b/src/bin/pg_basebackup/walmethods.c
@@ -17,6 +17,10 @@
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
+
+#ifdef HAVE_LIBLZ4
+#include <lz4frame.h>
+#endif
#ifdef HAVE_LIBZ
#include <zlib.h>
#endif
@@ -30,6 +34,9 @@
/* Size of zlib buffer for .tar.gz */
#define ZLIB_OUT_SIZE 4096
+/* Size of LZ4 input chunk for .lz4 */
+#define LZ4_IN_SIZE 4096
+
/*-------------------------------------------------------------------------
* WalDirectoryMethod - write wal to a directory looking like pg_wal
*-------------------------------------------------------------------------
@@ -60,6 +67,11 @@ typedef struct DirectoryMethodFile
#ifdef HAVE_LIBZ
gzFile gzfp;
#endif
+#ifdef HAVE_LIBLZ4
+ LZ4F_compressionContext_t ctx;
+ size_t lz4bufsize;
+ void *lz4buf;
+#endif
} DirectoryMethodFile;
static const char *
@@ -76,7 +88,8 @@ dir_get_file_name(const char *pathname, const char *temp_suffix)
snprintf(filename, MAXPGPATH, "%s%s%s",
pathname,
- dir_data->compression_method == COMPRESSION_GZIP ? ".gz" : "",
+ dir_data->compression_method == COMPRESSION_GZIP ? ".gz" :
+ dir_data->compression_method == COMPRESSION_LZ4 ? ".lz4" : "",
temp_suffix ? temp_suffix : "");
return filename;
@@ -92,6 +105,11 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
#ifdef HAVE_LIBZ
gzFile gzfp = NULL;
#endif
+#ifdef HAVE_LIBLZ4
+ LZ4F_compressionContext_t ctx = NULL;
+ size_t lz4bufsize = 0;
+ void *lz4buf = NULL;
+#endif
filename = dir_get_file_name(pathname, temp_suffix);
snprintf(tmppath, sizeof(tmppath), "%s/%s",
@@ -126,6 +144,50 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
}
}
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ size_t ctx_out;
+ size_t header_size;
+
+ ctx_out = LZ4F_createCompressionContext(&ctx, LZ4F_VERSION);
+ if (LZ4F_isError(ctx_out))
+ {
+ close(fd);
+ return NULL;
+ }
+
+ lz4bufsize = LZ4F_compressBound(LZ4_IN_SIZE, NULL);
+ lz4buf = pg_malloc0(lz4bufsize);
+
+ /* add the header */
+ header_size = LZ4F_compressBegin(ctx, lz4buf, lz4bufsize, NULL);
+ if (LZ4F_isError(header_size))
+ {
+ (void) LZ4F_freeCompressionContext(ctx);
+ pg_free(lz4buf);
+ close(fd);
+ return NULL;
+ }
+
+ errno = 0;
+ if (write(fd, lz4buf, header_size) != header_size)
+ {
+ int save_errno = errno;
+
+ (void) LZ4F_compressEnd(ctx, lz4buf, lz4bufsize, NULL);
+ (void) LZ4F_freeCompressionContext(ctx);
+ pg_free(lz4buf);
+ close(fd);
+
+ /*
+ * If write didn't set errno, assume problem is no disk space.
+ */
+ errno = save_errno ? save_errno : ENOSPC;
+ return NULL;
+ }
+ }
+#endif
/* Do pre-padding on non-compressed files */
if (pad_to_size && dir_data->compression_method == COMPRESSION_NONE)
@@ -176,6 +238,16 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
if (dir_data->compression_method == COMPRESSION_GZIP)
gzclose(gzfp);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ (void) LZ4F_compressEnd(ctx, lz4buf, lz4bufsize, NULL);
+ (void) LZ4F_freeCompressionContext(ctx);
+ pg_free(lz4buf);
+ close(fd);
+ }
+ else
#endif
close(fd);
return NULL;
@@ -187,6 +259,15 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
if (dir_data->compression_method == COMPRESSION_GZIP)
f->gzfp = gzfp;
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ f->ctx = ctx;
+ f->lz4buf = lz4buf;
+ f->lz4bufsize = lz4bufsize;
+ }
+#endif
+
f->fd = fd;
f->currpos = 0;
f->pathname = pg_strdup(pathname);
@@ -209,6 +290,43 @@ dir_write(Walfile f, const void *buf, size_t count)
if (dir_data->compression_method == COMPRESSION_GZIP)
r = (ssize_t) gzwrite(df->gzfp, buf, count);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ size_t chunk;
+ size_t remaining;
+ const void *inbuf = buf;
+
+ remaining = count;
+ while (remaining > 0)
+ {
+ size_t compressed;
+
+ if (remaining > LZ4_IN_SIZE)
+ chunk = LZ4_IN_SIZE;
+ else
+ chunk = remaining;
+
+ remaining -= chunk;
+ compressed = LZ4F_compressUpdate(df->ctx,
+ df->lz4buf, df->lz4bufsize,
+ inbuf, chunk,
+ NULL);
+
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+
+ inbuf = ((char *) inbuf) + chunk;
+ }
+
+ /* Our caller keeps track of the uncompressed size. */
+ r = (ssize_t) count;
+ }
+ else
#endif
r = write(df->fd, buf, count);
if (r > 0)
@@ -239,6 +357,25 @@ dir_close(Walfile f, WalCloseMethod method)
if (dir_data->compression_method == COMPRESSION_GZIP)
r = gzclose(df->gzfp);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ size_t compressed;
+
+ compressed = LZ4F_compressEnd(df->ctx,
+ df->lz4buf, df->lz4bufsize,
+ NULL);
+
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+
+ r = close(df->fd);
+ }
+ else
#endif
r = close(df->fd);
@@ -293,6 +430,12 @@ dir_close(Walfile f, WalCloseMethod method)
}
}
+#ifdef HAVE_LIBLZ4
+ pg_free(df->lz4buf);
+ /* supports free on NULL */
+ LZ4F_freeCompressionContext(df->ctx);
+#endif
+
pg_free(df->pathname);
pg_free(df->fullpath);
if (df->temp_suffix)
@@ -317,6 +460,21 @@ dir_sync(Walfile f)
return -1;
}
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ DirectoryMethodFile *df = (DirectoryMethodFile *) f;
+ size_t compressed;
+
+ /* Flush any internal buffers */
+ compressed = LZ4F_flush(df->ctx, df->lz4buf, df->lz4bufsize, NULL);
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+ }
+#endif
return fsync(((DirectoryMethodFile *) f)->fd);
}
diff --git a/src/bin/pg_basebackup/walmethods.h b/src/bin/pg_basebackup/walmethods.h
index 5dfe330ea5..f9b6a1646a 100644
--- a/src/bin/pg_basebackup/walmethods.h
+++ b/src/bin/pg_basebackup/walmethods.h
@@ -22,6 +22,7 @@ typedef enum
/* Types of compression supported */
typedef enum
{
+ COMPRESSION_LZ4,
COMPRESSION_GZIP,
COMPRESSION_NONE
} WalCompressionMethod;
diff --git a/doc/src/sgml/ref/pg_receivewal.sgml b/doc/src/sgml/ref/pg_receivewal.sgml
index 79a4436ab9..5147928cce 100644
--- a/doc/src/sgml/ref/pg_receivewal.sgml
+++ b/doc/src/sgml/ref/pg_receivewal.sgml
@@ -268,13 +268,15 @@ PostgreSQL documentation
<listitem>
<para>
Enables compression of write-ahead logs using the specified method.
- Supported values <literal>gzip</literal>, and
- <literal>none</literal>.
+ Supported values <literal>gzip</literal>, <literal>lz4</literal>
+ (if <productname>PostgreSQL</productname> was compiled with
+ <option>--with-lz4</option>) and <literal>none</literal>.
</para>
<para>
The suffix <filename>.gz</filename> will automatically be added to
- all filenames when using <literal>gzip</literal>
+ all filenames when using <literal>gzip</literal>, and the suffix
+ <filename>.lz4</filename> is added when using <literal>lz4</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 533c12fef9..05c54b27de 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -350,6 +350,7 @@ XGETTEXT = @XGETTEXT@
GZIP = gzip
BZIP2 = bzip2
+LZ4 = lz4
DOWNLOAD = wget -O $@ --no-use-server-timestamps
#DOWNLOAD = curl -o $@
--
2.33.1
Re: Teach pg_receivewal to use lz4 compression
On Thu, Nov 04, 2021 at 04:31:48PM +0900, Michael Paquier wrote:
I would have expected read_size to be (readend - readp) to match with
the remaining data in the read buffer that we still need to read.
Shouldn't out_size also be LZ4_CHUNK_SZ to match with the size of the
output buffer where all the contents are read? By setting it to 1, I
think that this is doing more loops into LZ4F_decompress() than really
necessary. It would not hurt either to memset(0) those buffers before
they are used, IMO. I am not completely sure either, but should we
use the number of bytes returned by LZ4F_decompress() as a hint for
the follow-up loops?+#ifdef HAVE_LIBLZ4 + while (readp < readend) + { + size_t read_size = 1; + size_t out_size = 1; + + status = LZ4F_decompress(ctx, outbuf, &out_size, + readbuf, &read_size, NULL);
And... It happens that the error from v9 is here, where we need to
read the amount of remaining data from "readp", and not "readbuf" :)
It is already late here, I'll continue on this stuff tomorrow, but
this looks rather committable overall.
--
Michael
Attachments:
v11-0001-Teach-pg_receivewal-to-use-LZ4-compression.patchtext/x-diff; charset=us-asciiDownload
From d2d76d8fefb1bad5db611746cf3d0ac89a67de4b Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 4 Nov 2021 17:19:41 +0900
Subject: [PATCH v11] Teach pg_receivewal to use LZ4 compression
The program pg_receivewal can use gzip compression to store the received
WAL. This commit teaches it to also be able to use LZ4 compression. It
is required that the binary is build using the -llz4 flag. It is enabled
via the --with-lz4 flag on configuration time.
The option `--compression-method` has been expanded to inlude the value
[LZ4]. The option `--compress` can not be used with LZ4 compression.
Under the hood there is nothing exceptional to be noted. Tar based
archives have not yet been taught to use LZ4 compression. If that is
felt useful, then it is easy to be added in the future.
Tests have been added to verify the creation and correctness of the
generated LZ4 files. The later is achieved by the use of LZ4 program, if
present in the installation.
---
src/bin/pg_basebackup/Makefile | 1 +
src/bin/pg_basebackup/pg_receivewal.c | 153 ++++++++++++++++++
src/bin/pg_basebackup/t/020_pg_receivewal.pl | 72 ++++++++-
src/bin/pg_basebackup/walmethods.c | 160 ++++++++++++++++++-
src/bin/pg_basebackup/walmethods.h | 1 +
doc/src/sgml/ref/pg_receivewal.sgml | 8 +-
src/Makefile.global.in | 1 +
7 files changed, 384 insertions(+), 12 deletions(-)
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index 459d514183..fd920fc197 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -19,6 +19,7 @@ top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
# make these available to TAP test scripts
+export LZ4
export TAR
# Note that GZIP cannot be used directly as this environment variable is
# used by the command "gzip" to pass down options, so stick with a different
diff --git a/src/bin/pg_basebackup/pg_receivewal.c b/src/bin/pg_basebackup/pg_receivewal.c
index 8acc0fc009..b5c0a98b82 100644
--- a/src/bin/pg_basebackup/pg_receivewal.c
+++ b/src/bin/pg_basebackup/pg_receivewal.c
@@ -32,6 +32,10 @@
#include "receivelog.h"
#include "streamutil.h"
+#ifdef HAVE_LIBLZ4
+#include "lz4frame.h"
+#endif
+
/* Time to sleep between reconnection attempts */
#define RECONNECT_SLEEP_TIME 5
@@ -136,6 +140,15 @@ is_xlogfilename(const char *filename, bool *ispartial,
return true;
}
+ /* File looks like a completed LZ4-compressed WAL file */
+ if (fname_len == XLOG_FNAME_LEN + strlen(".lz4") &&
+ strcmp(filename + XLOG_FNAME_LEN, ".lz4") == 0)
+ {
+ *ispartial = false;
+ *wal_compression_method = COMPRESSION_LZ4;
+ return true;
+ }
+
/* File looks like a partial uncompressed WAL file */
if (fname_len == XLOG_FNAME_LEN + strlen(".partial") &&
strcmp(filename + XLOG_FNAME_LEN, ".partial") == 0)
@@ -154,6 +167,15 @@ is_xlogfilename(const char *filename, bool *ispartial,
return true;
}
+ /* File looks like a partial LZ4-compressed WAL file */
+ if (fname_len == XLOG_FNAME_LEN + strlen(".lz4.partial") &&
+ strcmp(filename + XLOG_FNAME_LEN, ".lz4.partial") == 0)
+ {
+ *ispartial = true;
+ *wal_compression_method = COMPRESSION_LZ4;
+ return true;
+ }
+
/* File does not look like something we know */
return false;
}
@@ -284,6 +306,14 @@ FindStreamingStart(uint32 *tli)
* than 4GB, and then compare it to the size of a completed segment.
* The 4 last bytes correspond to the ISIZE member according to
* http://www.zlib.org/rfc-gzip.html.
+ *
+ * For LZ4 compressed segments, uncompress the file in a throw-away
+ * buffer keeping track of the uncompressed size, then compare it to
+ * the size of a completed segment. Per its protocol, LZ4 does not
+ * store the uncompressed size of an object by default. contentSize
+ * is one possible way to do that, but we need to rely on a method
+ * where WAL segments could have been compressed by a different
+ * source than pg_receivewal, like an archive_command.
*/
if (!ispartial && wal_compression_method == COMPRESSION_NONE)
{
@@ -315,6 +345,7 @@ FindStreamingStart(uint32 *tli)
snprintf(fullpath, sizeof(fullpath), "%s/%s", basedir, dirent->d_name);
fd = open(fullpath, O_RDONLY | PG_BINARY, 0);
+
if (fd < 0)
{
pg_log_error("could not open compressed file \"%s\": %m",
@@ -350,6 +381,107 @@ FindStreamingStart(uint32 *tli)
continue;
}
}
+ else if (!ispartial && compression_method == COMPRESSION_LZ4)
+ {
+#ifdef HAVE_LIBLZ4
+#define LZ4_CHUNK_SZ 64 * 1024 /* 64kB as maximum chunk size read */
+ int fd;
+ int r;
+ size_t uncompressed_size = 0;
+ char fullpath[MAXPGPATH * 2];
+ LZ4F_decompressionContext_t ctx = NULL;
+ LZ4F_errorCode_t status;
+ LZ4F_decompressOptions_t dec_opt;
+
+ memset(&dec_opt, 0, sizeof(dec_opt));
+ snprintf(fullpath, sizeof(fullpath), "%s/%s", basedir, dirent->d_name);
+
+ fd = open(fullpath, O_RDONLY | PG_BINARY, 0);
+ if (fd < 0)
+ {
+ pg_log_error("could not open file \"%s\": %m", fullpath);
+ exit(1);
+ }
+
+ status = LZ4F_createDecompressionContext(&ctx, LZ4F_VERSION);
+ if (LZ4F_isError(status))
+ {
+ pg_log_error("could not create LZ4 decompression context: %s",
+ LZ4F_getErrorName(status));
+ exit(1);
+ }
+
+ /*
+ * Once we have read enough data to cover one segment, we are
+ * done, there is no need to do more.
+ */
+ while (uncompressed_size <= WalSegSz)
+ {
+ char *readp;
+ char *readend;
+ char readbuf[LZ4_CHUNK_SZ];
+
+ r = read(fd, readbuf, sizeof(readbuf));
+ if (r < 0)
+ {
+ pg_log_error("could not read file \"%s\": %m", fullpath);
+ exit(1);
+ }
+
+ /* Done reading the file */
+ if (r == 0)
+ break;
+
+ /* Process one chunk */
+ readp = readbuf;
+ readend = readbuf + r;
+ while (readp < readend)
+ {
+ size_t read_size = readend - readp;
+ char outbuf[LZ4_CHUNK_SZ];
+ size_t out_size = sizeof(outbuf);
+
+ memset(outbuf, 0, sizeof(outbuf));
+
+ status = LZ4F_decompress(ctx, outbuf, &out_size,
+ readp, &read_size, &dec_opt);
+ if (LZ4F_isError(status))
+ {
+ pg_log_error("could not decompress file \"%s\": %s",
+ fullpath,
+ LZ4F_getErrorName(status));
+ exit(1);
+ }
+
+ readp += read_size;
+ uncompressed_size += out_size;
+ }
+ }
+
+ close(fd);
+
+ status = LZ4F_freeDecompressionContext(ctx);
+ if (LZ4F_isError(status))
+ {
+ pg_log_error("could not free LZ4 decompression context: %s",
+ LZ4F_getErrorName(status));
+ exit(1);
+ }
+
+ if (uncompressed_size != WalSegSz)
+ {
+ pg_log_warning("compressed segment file \"%s\" has incorrect uncompressed size %ld, skipping",
+ dirent->d_name, uncompressed_size);
+ continue;
+ }
+#else
+ pg_log_error("could not check segment file \"%s\" compressed with LZ4",
+ dirent->d_name);
+ pg_log_error("this build does not support compression with %s",
+ "LZ4");
+ exit(1);
+#endif
+ }
/* Looks like a valid segment. Remember that we saw it. */
if ((segno > high_segno) ||
@@ -650,6 +782,8 @@ main(int argc, char **argv)
case 6:
if (pg_strcasecmp(optarg, "gzip") == 0)
compression_method = COMPRESSION_GZIP;
+ else if (pg_strcasecmp(optarg, "lz4") == 0)
+ compression_method = COMPRESSION_LZ4;
else if (pg_strcasecmp(optarg, "none") == 0)
compression_method = COMPRESSION_NONE;
else
@@ -748,6 +882,25 @@ main(int argc, char **argv)
exit(1);
#endif
break;
+ case COMPRESSION_LZ4:
+#ifdef HAVE_LIBLZ4
+ if (compresslevel != 0)
+ {
+ pg_log_error("cannot use --compress with --compression-method=%s",
+ "lz4");
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+ progname);
+ exit(1);
+ }
+#else
+ if (compression_method == COMPRESSION_LZ4)
+ {
+ pg_log_error("this build does not support compression with %s",
+ "LZ4");
+ exit(1);
+ }
+ break;
+#endif
}
diff --git a/src/bin/pg_basebackup/t/020_pg_receivewal.pl b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
index 94786f0815..43599d832b 100644
--- a/src/bin/pg_basebackup/t/020_pg_receivewal.pl
+++ b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
@@ -5,7 +5,7 @@ use strict;
use warnings;
use PostgreSQL::Test::Utils;
use PostgreSQL::Test::Cluster;
-use Test::More tests => 37;
+use Test::More tests => 42;
program_help_ok('pg_receivewal');
program_version_ok('pg_receivewal');
@@ -138,13 +138,69 @@ SKIP:
"gzip verified the integrity of compressed WAL segments");
}
+# Check LZ4 compression if available
+SKIP:
+{
+ skip "postgres was not built with LZ4 support", 5
+ if (!check_pg_config("#define HAVE_LIBLZ4 1"));
+
+ # Generate more WAL including one completed, compressed segment.
+ $primary->psql('postgres', 'SELECT pg_switch_wal();');
+ $nextlsn =
+ $primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
+ chomp($nextlsn);
+ $primary->psql('postgres', 'INSERT INTO test_table VALUES (3);');
+
+ # Stream up to the given position.
+ $primary->command_ok(
+ [
+ 'pg_receivewal', '-D',
+ $stream_dir, '--verbose',
+ '--endpos', $nextlsn,
+ '--no-loop', '--compression-method',
+ 'lz4'
+ ],
+ 'streaming some WAL using --compression-method=lz4');
+
+ # Verify that the stored files are generated with their expected
+ # names.
+ my @lz4_wals = glob "$stream_dir/*.lz4";
+ is(scalar(@lz4_wals), 1,
+ "one WAL segment compressed with LZ4 was created");
+ my @lz4_partial_wals = glob "$stream_dir/*.lz4.partial";
+ is(scalar(@lz4_partial_wals),
+ 1, "one partial WAL segment compressed with LZ4 was created");
+
+ # Verify that the start streaming position is computed correctly by
+ # comparing it with the partial file generated previously. The name
+ # of the previous partial, now-completed WAL segment is updated, keeping
+ # its base number.
+ $partial_wals[0] =~ s/(\.gz)?\.partial$/.lz4/;
+ is($lz4_wals[0] eq $partial_wals[0],
+ 1, "one partial WAL segment is now completed");
+ # Update the list of partial wals with the current one.
+ @partial_wals = @lz4_partial_wals;
+
+ # Check the integrity of the completed segment, if LZ4 is an available
+ # command.
+ my $lz4 = $ENV{LZ4};
+ skip "program lz4 is not found in your system", 1
+ if ( !defined $lz4
+ || $lz4 eq ''
+ || system_log($lz4, '--version') != 0);
+
+ my $lz4_is_valid = system_log($lz4, '-t', @lz4_wals);
+ is($lz4_is_valid, 0,
+ "lz4 verified the integrity of compressed WAL segments");
+}
+
# Verify that the start streaming position is computed and that the value is
-# correct regardless of whether ZLIB is available.
+# correct regardless of whether any compression is available.
$primary->psql('postgres', 'SELECT pg_switch_wal();');
$nextlsn =
$primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
chomp($nextlsn);
-$primary->psql('postgres', 'INSERT INTO test_table VALUES (3);');
+$primary->psql('postgres', 'INSERT INTO test_table VALUES (4);');
$primary->command_ok(
[
'pg_receivewal', '-D', $stream_dir, '--verbose',
@@ -152,7 +208,7 @@ $primary->command_ok(
],
"streaming some WAL");
-$partial_wals[0] =~ s/(\.gz)?.partial//;
+$partial_wals[0] =~ s/(\.gz|\.lz4)?.partial//;
ok(-e $partial_wals[0], "check that previously partial WAL is now complete");
# Permissions on WAL files should be default
@@ -190,7 +246,7 @@ my $walfile_streamed = $primary->safe_psql(
# Switch to a new segment, to make sure that the segment retained by the
# slot is still streamed. This may not be necessary, but play it safe.
-$primary->psql('postgres', 'INSERT INTO test_table VALUES (4);');
+$primary->psql('postgres', 'INSERT INTO test_table VALUES (5);');
$primary->psql('postgres', 'SELECT pg_switch_wal();');
$nextlsn =
$primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
@@ -198,7 +254,7 @@ chomp($nextlsn);
# Add a bit more data to accelerate the end of the next pg_receivewal
# commands.
-$primary->psql('postgres', 'INSERT INTO test_table VALUES (5);');
+$primary->psql('postgres', 'INSERT INTO test_table VALUES (6);');
# Check case where the slot does not exist.
$primary->command_fails_like(
@@ -253,13 +309,13 @@ $standby->promote;
# on the new timeline.
my $walfile_after_promotion = $standby->safe_psql('postgres',
"SELECT pg_walfile_name(pg_current_wal_insert_lsn());");
-$standby->psql('postgres', 'INSERT INTO test_table VALUES (6);');
+$standby->psql('postgres', 'INSERT INTO test_table VALUES (7);');
$standby->psql('postgres', 'SELECT pg_switch_wal();');
$nextlsn =
$standby->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
chomp($nextlsn);
# This speeds up the operation.
-$standby->psql('postgres', 'INSERT INTO test_table VALUES (7);');
+$standby->psql('postgres', 'INSERT INTO test_table VALUES (8);');
# Now try to resume from the slot after the promotion.
my $timeline_dir = $primary->basedir . '/timeline_wal';
diff --git a/src/bin/pg_basebackup/walmethods.c b/src/bin/pg_basebackup/walmethods.c
index 52f314af3b..f1ba2a828a 100644
--- a/src/bin/pg_basebackup/walmethods.c
+++ b/src/bin/pg_basebackup/walmethods.c
@@ -17,6 +17,10 @@
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
+
+#ifdef HAVE_LIBLZ4
+#include <lz4frame.h>
+#endif
#ifdef HAVE_LIBZ
#include <zlib.h>
#endif
@@ -30,6 +34,9 @@
/* Size of zlib buffer for .tar.gz */
#define ZLIB_OUT_SIZE 4096
+/* Size of LZ4 input chunk for .lz4 */
+#define LZ4_IN_SIZE 4096
+
/*-------------------------------------------------------------------------
* WalDirectoryMethod - write wal to a directory looking like pg_wal
*-------------------------------------------------------------------------
@@ -60,6 +67,11 @@ typedef struct DirectoryMethodFile
#ifdef HAVE_LIBZ
gzFile gzfp;
#endif
+#ifdef HAVE_LIBLZ4
+ LZ4F_compressionContext_t ctx;
+ size_t lz4bufsize;
+ void *lz4buf;
+#endif
} DirectoryMethodFile;
static const char *
@@ -76,7 +88,8 @@ dir_get_file_name(const char *pathname, const char *temp_suffix)
snprintf(filename, MAXPGPATH, "%s%s%s",
pathname,
- dir_data->compression_method == COMPRESSION_GZIP ? ".gz" : "",
+ dir_data->compression_method == COMPRESSION_GZIP ? ".gz" :
+ dir_data->compression_method == COMPRESSION_LZ4 ? ".lz4" : "",
temp_suffix ? temp_suffix : "");
return filename;
@@ -92,6 +105,11 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
#ifdef HAVE_LIBZ
gzFile gzfp = NULL;
#endif
+#ifdef HAVE_LIBLZ4
+ LZ4F_compressionContext_t ctx = NULL;
+ size_t lz4bufsize = 0;
+ void *lz4buf = NULL;
+#endif
filename = dir_get_file_name(pathname, temp_suffix);
snprintf(tmppath, sizeof(tmppath), "%s/%s",
@@ -126,6 +144,50 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
}
}
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ size_t ctx_out;
+ size_t header_size;
+
+ ctx_out = LZ4F_createCompressionContext(&ctx, LZ4F_VERSION);
+ if (LZ4F_isError(ctx_out))
+ {
+ close(fd);
+ return NULL;
+ }
+
+ lz4bufsize = LZ4F_compressBound(LZ4_IN_SIZE, NULL);
+ lz4buf = pg_malloc0(lz4bufsize);
+
+ /* add the header */
+ header_size = LZ4F_compressBegin(ctx, lz4buf, lz4bufsize, NULL);
+ if (LZ4F_isError(header_size))
+ {
+ (void) LZ4F_freeCompressionContext(ctx);
+ pg_free(lz4buf);
+ close(fd);
+ return NULL;
+ }
+
+ errno = 0;
+ if (write(fd, lz4buf, header_size) != header_size)
+ {
+ int save_errno = errno;
+
+ (void) LZ4F_compressEnd(ctx, lz4buf, lz4bufsize, NULL);
+ (void) LZ4F_freeCompressionContext(ctx);
+ pg_free(lz4buf);
+ close(fd);
+
+ /*
+ * If write didn't set errno, assume problem is no disk space.
+ */
+ errno = save_errno ? save_errno : ENOSPC;
+ return NULL;
+ }
+ }
+#endif
/* Do pre-padding on non-compressed files */
if (pad_to_size && dir_data->compression_method == COMPRESSION_NONE)
@@ -176,6 +238,16 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
if (dir_data->compression_method == COMPRESSION_GZIP)
gzclose(gzfp);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ (void) LZ4F_compressEnd(ctx, lz4buf, lz4bufsize, NULL);
+ (void) LZ4F_freeCompressionContext(ctx);
+ pg_free(lz4buf);
+ close(fd);
+ }
+ else
#endif
close(fd);
return NULL;
@@ -187,6 +259,15 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
if (dir_data->compression_method == COMPRESSION_GZIP)
f->gzfp = gzfp;
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ f->ctx = ctx;
+ f->lz4buf = lz4buf;
+ f->lz4bufsize = lz4bufsize;
+ }
+#endif
+
f->fd = fd;
f->currpos = 0;
f->pathname = pg_strdup(pathname);
@@ -209,6 +290,43 @@ dir_write(Walfile f, const void *buf, size_t count)
if (dir_data->compression_method == COMPRESSION_GZIP)
r = (ssize_t) gzwrite(df->gzfp, buf, count);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ size_t chunk;
+ size_t remaining;
+ const void *inbuf = buf;
+
+ remaining = count;
+ while (remaining > 0)
+ {
+ size_t compressed;
+
+ if (remaining > LZ4_IN_SIZE)
+ chunk = LZ4_IN_SIZE;
+ else
+ chunk = remaining;
+
+ remaining -= chunk;
+ compressed = LZ4F_compressUpdate(df->ctx,
+ df->lz4buf, df->lz4bufsize,
+ inbuf, chunk,
+ NULL);
+
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+
+ inbuf = ((char *) inbuf) + chunk;
+ }
+
+ /* Our caller keeps track of the uncompressed size. */
+ r = (ssize_t) count;
+ }
+ else
#endif
r = write(df->fd, buf, count);
if (r > 0)
@@ -239,6 +357,25 @@ dir_close(Walfile f, WalCloseMethod method)
if (dir_data->compression_method == COMPRESSION_GZIP)
r = gzclose(df->gzfp);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ size_t compressed;
+
+ compressed = LZ4F_compressEnd(df->ctx,
+ df->lz4buf, df->lz4bufsize,
+ NULL);
+
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+
+ r = close(df->fd);
+ }
+ else
#endif
r = close(df->fd);
@@ -293,6 +430,12 @@ dir_close(Walfile f, WalCloseMethod method)
}
}
+#ifdef HAVE_LIBLZ4
+ pg_free(df->lz4buf);
+ /* supports free on NULL */
+ LZ4F_freeCompressionContext(df->ctx);
+#endif
+
pg_free(df->pathname);
pg_free(df->fullpath);
if (df->temp_suffix)
@@ -317,6 +460,21 @@ dir_sync(Walfile f)
return -1;
}
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ DirectoryMethodFile *df = (DirectoryMethodFile *) f;
+ size_t compressed;
+
+ /* Flush any internal buffers */
+ compressed = LZ4F_flush(df->ctx, df->lz4buf, df->lz4bufsize, NULL);
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+ }
+#endif
return fsync(((DirectoryMethodFile *) f)->fd);
}
diff --git a/src/bin/pg_basebackup/walmethods.h b/src/bin/pg_basebackup/walmethods.h
index 5dfe330ea5..f9b6a1646a 100644
--- a/src/bin/pg_basebackup/walmethods.h
+++ b/src/bin/pg_basebackup/walmethods.h
@@ -22,6 +22,7 @@ typedef enum
/* Types of compression supported */
typedef enum
{
+ COMPRESSION_LZ4,
COMPRESSION_GZIP,
COMPRESSION_NONE
} WalCompressionMethod;
diff --git a/doc/src/sgml/ref/pg_receivewal.sgml b/doc/src/sgml/ref/pg_receivewal.sgml
index 79a4436ab9..5147928cce 100644
--- a/doc/src/sgml/ref/pg_receivewal.sgml
+++ b/doc/src/sgml/ref/pg_receivewal.sgml
@@ -268,13 +268,15 @@ PostgreSQL documentation
<listitem>
<para>
Enables compression of write-ahead logs using the specified method.
- Supported values <literal>gzip</literal>, and
- <literal>none</literal>.
+ Supported values <literal>gzip</literal>, <literal>lz4</literal>
+ (if <productname>PostgreSQL</productname> was compiled with
+ <option>--with-lz4</option>) and <literal>none</literal>.
</para>
<para>
The suffix <filename>.gz</filename> will automatically be added to
- all filenames when using <literal>gzip</literal>
+ all filenames when using <literal>gzip</literal>, and the suffix
+ <filename>.lz4</filename> is added when using <literal>lz4</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 533c12fef9..05c54b27de 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -350,6 +350,7 @@ XGETTEXT = @XGETTEXT@
GZIP = gzip
BZIP2 = bzip2
+LZ4 = lz4
DOWNLOAD = wget -O $@ --no-use-server-timestamps
#DOWNLOAD = curl -o $@
--
2.33.1
Re: Teach pg_receivewal to use lz4 compression
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday, November 4th, 2021 at 9:21 AM, Michael Paquier <michael@paquier.xyz> wrote:
On Thu, Nov 04, 2021 at 04:31:48PM +0900, Michael Paquier wrote:
Thanks. I have looked at 0001 today, and applied it after fixing a
couple of issues.
Great! Thank you very much.
From memory:
- zlib.h was missing from pg_receivewal.c, issue that I noticed after
removing the redefinition of Z_DEFAULT_COMPRESSION because there was
no need for it (did a run with a --without-zlib as well).
Yeah, I simply wanted to avoid adding a header. Either way works really.
- Simplified a bit the error handling for incorrect option
combinations, using a switch/case while on it.
Much cleaner done this way.
- Renamed the existing variable "compression" in walmethods.c to
compression_level, to reduce any confusion with the introduction of
compression_method. One thing I have noticed is about the tar method,
where we rely on the compression level to decide if compression should
be used or not. There should be some simplifications possible there
but there is a huge take in receivelog.c where we use COMPRESSION_NONE
to track down that we still want to zero a new segment when using tar
method.
Agreed.
- Use of 'I' as short option name, err... After applying the first
batch..
I left that in just to have the two compression related options next to each
other when switching. I assumed it might help with readability for the next
developer looking at it.
Removing it, is cleaner for the option definifion though, thanks.
Based on the work of 0001, there were some conflicts with 0002. I
have solved them while reviewing it, and adapted the code to what got
already applied.
Thank you very much.
+ header_size = LZ4F_compressBegin(ctx, lz4buf, lz4bufsize, NULL); + if (LZ4F_isError(header_size)) + { + pg_free(lz4buf); + close(fd); + return NULL; + } In dir_open_for_write(), I guess that this one is missing one LZ4F_freeCompressionContext().
Agreed.
+ status = LZ4F_freeDecompressionContext(ctx); + if (LZ4F_isError(status)) + { + pg_log_error("could not free LZ4 decompression context: %s", + LZ4F_getErrorName(status)); + exit(1); + } + + if (uncompressed_size != WalSegSz) + { + pg_log_warning("compressed segment file \"%s\" has incorrect uncompressed size %ld, skipping", + dirent->d_name, uncompressed_size); + (void) LZ4F_freeDecompressionContext(ctx); + continue; + } When the uncompressed size does not match out expected size, the second LZ4F_freeDecompressionContext() looks unnecessary to me because we have already one a couple of lines above.
Agreed.
+ ctx_out = LZ4F_createCompressionContext(&ctx, LZ4F_VERSION); + lz4bufsize = LZ4F_compressBound(LZ4_IN_SIZE, NULL); + if (LZ4F_isError(ctx_out)) + { + close(fd); + return NULL; + } LZ4F_compressBound() can be after the check on ctx_out, here.+ while (1) + { + char *readp; + char *readend; Simply looping when decompressing a segment to check its size looks rather unsafe to me. We should leave the loop once uncompressed_size is strictly more than WalSegSz.
The loop exits when done reading or when it failed to read:
+ r = read(fd, readbuf, sizeof(readbuf));
+ if (r < 0)
+ {
+ pg_log_error("could not read file \"%s\": %m", fullpath);
+ exit(1);
+ }
+
+ /* Done reading */
+ if (r == 0)
+ break;Although I do agree that it can exit before that, if the uncompressed size is
greater than WalSegSz.
The amount of TAP tests looks fine, and that's consistent with what we
do for zlib^D^D^D^Dgzip. Now, when testing manually pg_receivewal
with various combinations of gzip, lz4 and none, I can see the
following failure in the code that calculates the streaming start
point:
pg_receivewal: error: could not decompress file
"wals//000000010000000000000006.lz4": ERROR_frameType_unknown
Hmmm.... I will look into that.
In the LZ4 code, this points to lib/lz4frame.c, where we read an
incorrect header (see the part that does not match LZ4F_MAGICNUMBER).
The segments written by pg_receivewal are clean. Please note that
this shows up as well when manually compressing some segments with a
simple lz4 command, to simulate for example the case where a user
compressed some segments by himself/herself before running
pg_receivewal.
Rights, thank you for investigating. I will look further.
So, tour code does LZ4F_createDecompressionContext() followed by a
loop on read() and LZ4F_decompress() that relies on an input and an
output buffer of a fixed 4kB size (we could use 64kB at least here
actually?). So this set of loops looks rather correct to me.
For what is worth, in a stand alone program I wrote while investigating, I did
not notice any noteworthy performance gain, when decompressing files of original
size similar to common WalSegSz values, using 4kB, 8kB, 16kB and 32kB buffers.
I did not try 64kB though. This was by no means exhaustive performance testing,
though good enough to propose a value. I chose 4kB because it is small enough to
have in the stack. I thought anything bigger should be heap alloced and that
would add a bit more distraction in the code with the pg_free() calls.
I will re-write to use 64kB in the heap.
Now, this part is weird: + while (readp < readend) + { + size_t read_size = 1; + size_t out_size = 1;I would have expected read_size to be (readend - readp) to match with
the remaining data in the read buffer that we still need to read.
Shouldn't out_size also be LZ4_CHUNK_SZ to match with the size of the
output buffer where all the contents are read? By setting it to 1, I
think that this is doing more loops into LZ4F_decompress() than really
necessary.
You are very correct. An oversight when moving code over from my program and
renaming variables. Consider me embarrassed.
It would not hurt either to memset(0) those buffers before
they are used, IMO.
It does not hurt, yet I do not think that is necessary because one buffer is
throw away, i.e. the program writes to it but we never read it, and the other is
overwritten during the read call.
I am not completely sure either, but should we
use the number of bytes returned by LZ4F_decompress() as a hint for
the follow-up loops?
It is possible, though in my humble opinion it adds some code and has no
measurable effect in the code.
+ status = LZ4F_decompress(ctx, outbuf, &out_size, + readbuf, &read_size, NULL);
And... It happens that the error from v9 is here, where we need to
read the amount of remaining data from "readp", and not "readbuf" :)
Agreed.
I really suck at renaming all the things... I am really embarrassed.
Attached is an updated patch, which includes fixes for most of the
issues I am mentioning above. Please note that I have not changed
FindStreamingStart(), so this part is the same as v9.
Thanks!
Cheers,
//Georgios
Show quoted text
Thanks,
--
Michael
Re: Teach pg_receivewal to use lz4 compression
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday, November 4th, 2021 at 9:21 AM, Michael Paquier <michael@paquier.xyz> wrote:
+#ifdef HAVE_LIBLZ4 + while (readp < readend) + { + size_t read_size = 1; + size_t out_size = 1; + + status = LZ4F_decompress(ctx, outbuf, &out_size, + readbuf, &read_size, NULL);And... It happens that the error from v9 is here, where we need to
read the amount of remaining data from "readp", and not "readbuf" :)It is already late here, I'll continue on this stuff tomorrow, but
this looks rather committable overall.
Thank you for v11 of the patch. Please find attached v12 which addresses a few
minor points.
Added an Oxford comma since the list now contains three or more terms:
- <option>--with-lz4</option>) and <literal>none</literal>.
+ <option>--with-lz4</option>), and <literal>none</literal>.Removed an extra condinional check while switching over compression_method.
Instead of:
+ case COMPRESSION_LZ4:
+#ifdef HAVE_LIBLZ4
+ if (compresslevel != 0)
+ {
+ pg_log_error("cannot use --compress with
--compression-method=%s",
+ "lz4");
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+ progname);
+ exit(1);
+ }
+#else
+ if (compression_method == COMPRESSION_LZ4)
+ {
+ pg_log_error("this build does not support compression with %s",
+ "LZ4");
+ exit(1);
+ }
+ break;
+#endifI opted for:
+ case COMPRESSION_LZ4:
+#ifdef HAVE_LIBLZ4
+ if (compresslevel != 0)
+ {
+ pg_log_error("cannot use --compress with
--compression-method=%s",
+ "lz4");
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+ progname);
+ exit(1);
+ }
+#else
+ pg_log_error("this build does not support compression with %s",
+ "LZ4");
+ exit(1);
+ #endifThere was an error while trying to find the streaming start. The code read:
+ else if (!ispartial && compression_method == COMPRESSION_LZ4)
where it should be instead:
+ else if (!ispartial && wal_compression_method == COMPRESSION_LZ4)
because compression_method is the global option exposed to the whereas
wal_compression_method is the local variable used to figure out what kind of
file the function is currently working with. This error was existing at least in
v9-0002 of $subject.
The variables readbuf and outbuf, used in the decompression of LZ4 files, are
now heap allocated.
Last, while the following is correct:
+ /*
+ * Once we have read enough data to cover one segment, we are
+ * done, there is no need to do more.
+ */
+ while (uncompressed_size <= WalSegSz)I felt that converting it a do {} while () loop instead, will help with
readability:
+ do
+ {
<snip>
+ /*
+ * No need to continue reading the file when the uncompressed_size
+ * exceeds WalSegSz, even if there are still data left to read.
+ * However, if uncompressed_size is equal to WalSegSz, it should
+ * verify that there is no more data to read.
+ */
+ } while (r > 0 && uncompressed_size <= WalSegSz);of course the check:
+ /* Done reading the file */
+ if (r == 0)
+ break;
midway the loop is no longer needed and thus removed.I would like to have a bit more test coverage in the case for FindStreamingStart().
Specifically for the case that a lz4-compressed segment larger than WalSegSz exists.
The attached patch does not contain such test case. I am not very certain on how to
create such a test case reliably as it would be mostly based on a warning emitted
during the parsing of such a file.
Cheers,
//Georgios
Show quoted text
--
Michael
Attachments:
v12-0001-Teach-pg_receivewal-to-use-LZ4-compression.patchtext/x-patch; name=v12-0001-Teach-pg_receivewal-to-use-LZ4-compression.patchDownload
From 48720e7c6ba771c45d43dc9f5e6833f8bb6715e6 Mon Sep 17 00:00:00 2001
From: Georgios Kokolatos <gkokolatos@pm.me>
Date: Thu, 4 Nov 2021 16:05:21 +0000
Subject: [PATCH v12] Teach pg_receivewal to use LZ4 compression
The program pg_receivewal can use gzip compression to store the received
WAL. This commit teaches it to also be able to use LZ4 compression. It
is required that the binary is build using the -llz4 flag. It is enabled
via the --with-lz4 flag on configuration time.
The option `--compression-method` has been expanded to inlude the value
[LZ4]. The option `--compress` can not be used with LZ4 compression.
Under the hood there is nothing exceptional to be noted. Tar based
archives have not yet been taught to use LZ4 compression. If that is
felt useful, then it is easy to be added in the future.
Tests have been added to verify the creation and correctness of the
generated LZ4 files. The later is achieved by the use of LZ4 program, if
present in the installation.
---
doc/src/sgml/ref/pg_receivewal.sgml | 8 +-
src/Makefile.global.in | 1 +
src/bin/pg_basebackup/Makefile | 1 +
src/bin/pg_basebackup/pg_receivewal.c | 156 ++++++++++++++++++
src/bin/pg_basebackup/t/020_pg_receivewal.pl | 72 ++++++++-
src/bin/pg_basebackup/walmethods.c | 160 ++++++++++++++++++-
src/bin/pg_basebackup/walmethods.h | 1 +
7 files changed, 387 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/ref/pg_receivewal.sgml b/doc/src/sgml/ref/pg_receivewal.sgml
index 79a4436ab9..5de80f8c64 100644
--- a/doc/src/sgml/ref/pg_receivewal.sgml
+++ b/doc/src/sgml/ref/pg_receivewal.sgml
@@ -268,13 +268,15 @@ PostgreSQL documentation
<listitem>
<para>
Enables compression of write-ahead logs using the specified method.
- Supported values <literal>gzip</literal>, and
- <literal>none</literal>.
+ Supported values <literal>gzip</literal>, <literal>lz4</literal>
+ (if <productname>PostgreSQL</productname> was compiled with
+ <option>--with-lz4</option>), and <literal>none</literal>.
</para>
<para>
The suffix <filename>.gz</filename> will automatically be added to
- all filenames when using <literal>gzip</literal>
+ all filenames when using <literal>gzip</literal>, and the suffix
+ <filename>.lz4</filename> is added when using <literal>lz4</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 533c12fef9..05c54b27de 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -350,6 +350,7 @@ XGETTEXT = @XGETTEXT@
GZIP = gzip
BZIP2 = bzip2
+LZ4 = lz4
DOWNLOAD = wget -O $@ --no-use-server-timestamps
#DOWNLOAD = curl -o $@
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index 459d514183..fd920fc197 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -19,6 +19,7 @@ top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
# make these available to TAP test scripts
+export LZ4
export TAR
# Note that GZIP cannot be used directly as this environment variable is
# used by the command "gzip" to pass down options, so stick with a different
diff --git a/src/bin/pg_basebackup/pg_receivewal.c b/src/bin/pg_basebackup/pg_receivewal.c
index 8acc0fc009..1a943231ae 100644
--- a/src/bin/pg_basebackup/pg_receivewal.c
+++ b/src/bin/pg_basebackup/pg_receivewal.c
@@ -32,6 +32,10 @@
#include "receivelog.h"
#include "streamutil.h"
+#ifdef HAVE_LIBLZ4
+#include "lz4frame.h"
+#endif
+
/* Time to sleep between reconnection attempts */
#define RECONNECT_SLEEP_TIME 5
@@ -136,6 +140,15 @@ is_xlogfilename(const char *filename, bool *ispartial,
return true;
}
+ /* File looks like a completed LZ4-compressed WAL file */
+ if (fname_len == XLOG_FNAME_LEN + strlen(".lz4") &&
+ strcmp(filename + XLOG_FNAME_LEN, ".lz4") == 0)
+ {
+ *ispartial = false;
+ *wal_compression_method = COMPRESSION_LZ4;
+ return true;
+ }
+
/* File looks like a partial uncompressed WAL file */
if (fname_len == XLOG_FNAME_LEN + strlen(".partial") &&
strcmp(filename + XLOG_FNAME_LEN, ".partial") == 0)
@@ -154,6 +167,15 @@ is_xlogfilename(const char *filename, bool *ispartial,
return true;
}
+ /* File looks like a partial LZ4-compressed WAL file */
+ if (fname_len == XLOG_FNAME_LEN + strlen(".lz4.partial") &&
+ strcmp(filename + XLOG_FNAME_LEN, ".lz4.partial") == 0)
+ {
+ *ispartial = true;
+ *wal_compression_method = COMPRESSION_LZ4;
+ return true;
+ }
+
/* File does not look like something we know */
return false;
}
@@ -284,6 +306,14 @@ FindStreamingStart(uint32 *tli)
* than 4GB, and then compare it to the size of a completed segment.
* The 4 last bytes correspond to the ISIZE member according to
* http://www.zlib.org/rfc-gzip.html.
+ *
+ * For LZ4 compressed segments, uncompress the file in a throw-away
+ * buffer keeping track of the uncompressed size, then compare it to
+ * the size of a completed segment. Per its protocol, LZ4 does not
+ * store the uncompressed size of an object by default. contentSize
+ * is one possible way to do that, but we need to rely on a method
+ * where WAL segments could have been compressed by a different
+ * source than pg_receivewal, like an archive_command.
*/
if (!ispartial && wal_compression_method == COMPRESSION_NONE)
{
@@ -315,6 +345,7 @@ FindStreamingStart(uint32 *tli)
snprintf(fullpath, sizeof(fullpath), "%s/%s", basedir, dirent->d_name);
fd = open(fullpath, O_RDONLY | PG_BINARY, 0);
+
if (fd < 0)
{
pg_log_error("could not open compressed file \"%s\": %m",
@@ -350,6 +381,113 @@ FindStreamingStart(uint32 *tli)
continue;
}
}
+ else if (!ispartial && wal_compression_method == COMPRESSION_LZ4)
+ {
+#ifdef HAVE_LIBLZ4
+#define LZ4_CHUNK_SZ 64 * 1024 /* 64kB as maximum chunk size read */
+ int fd;
+ ssize_t r;
+ size_t uncompressed_size = 0;
+ char fullpath[MAXPGPATH * 2];
+ char *outbuf;
+ char *readbuf;
+ LZ4F_decompressionContext_t ctx = NULL;
+ LZ4F_decompressOptions_t dec_opt;
+ LZ4F_errorCode_t status;
+
+ memset(&dec_opt, 0, sizeof(dec_opt));
+ snprintf(fullpath, sizeof(fullpath), "%s/%s", basedir, dirent->d_name);
+
+ fd = open(fullpath, O_RDONLY | PG_BINARY, 0);
+ if (fd < 0)
+ {
+ pg_log_error("could not open file \"%s\": %m", fullpath);
+ exit(1);
+ }
+
+ status = LZ4F_createDecompressionContext(&ctx, LZ4F_VERSION);
+ if (LZ4F_isError(status))
+ {
+ pg_log_error("could not create LZ4 decompression context: %s",
+ LZ4F_getErrorName(status));
+ exit(1);
+ }
+
+ outbuf = pg_malloc0(LZ4_CHUNK_SZ);
+ readbuf = pg_malloc0(LZ4_CHUNK_SZ);
+ do
+ {
+ char *readp;
+ char *readend;
+
+ r = read(fd, readbuf, LZ4_CHUNK_SZ);
+ if (r < 0)
+ {
+ pg_log_error("could not read file \"%s\": %m", fullpath);
+ exit(1);
+ }
+
+ /* Done reading the file */
+ if (r == 0)
+ break;
+
+ /* Process one chunk */
+ readp = readbuf;
+ readend = readbuf + r;
+ while (readp < readend)
+ {
+ size_t out_size = LZ4_CHUNK_SZ;
+ size_t read_size = readend - readp;
+
+ memset(outbuf, 0, LZ4_CHUNK_SZ);
+ status = LZ4F_decompress(ctx, outbuf, &out_size,
+ readp, &read_size, &dec_opt);
+ if (LZ4F_isError(status))
+ {
+ pg_log_error("could not decompress file \"%s\": %s",
+ fullpath,
+ LZ4F_getErrorName(status));
+ exit(1);
+ }
+
+ readp += read_size;
+ uncompressed_size += out_size;
+ }
+
+ /*
+ * No need to continue reading the file when the uncompressed_size
+ * exceeds WalSegSz, even if there are still data left to read.
+ * However, if uncompressed_size is equal to WalSegSz, it should
+ * verify that there is no more data to read.
+ */
+ } while (r > 0 && uncompressed_size <= WalSegSz);
+
+ close(fd);
+ pg_free(outbuf);
+ pg_free(readbuf);
+
+ status = LZ4F_freeDecompressionContext(ctx);
+ if (LZ4F_isError(status))
+ {
+ pg_log_error("could not free LZ4 decompression context: %s",
+ LZ4F_getErrorName(status));
+ exit(1);
+ }
+
+ if (uncompressed_size != WalSegSz)
+ {
+ pg_log_warning("compressed segment file \"%s\" has incorrect uncompressed size %ld, skipping",
+ dirent->d_name, uncompressed_size);
+ continue;
+ }
+#else
+ pg_log_error("could not check segment file \"%s\" compressed with LZ4",
+ dirent->d_name);
+ pg_log_error("this build does not support compression with %s",
+ "LZ4");
+ exit(1);
+#endif
+ }
/* Looks like a valid segment. Remember that we saw it. */
if ((segno > high_segno) ||
@@ -650,6 +788,8 @@ main(int argc, char **argv)
case 6:
if (pg_strcasecmp(optarg, "gzip") == 0)
compression_method = COMPRESSION_GZIP;
+ else if (pg_strcasecmp(optarg, "lz4") == 0)
+ compression_method = COMPRESSION_LZ4;
else if (pg_strcasecmp(optarg, "none") == 0)
compression_method = COMPRESSION_NONE;
else
@@ -746,6 +886,22 @@ main(int argc, char **argv)
pg_log_error("this build does not support compression with %s",
"gzip");
exit(1);
+#endif
+ break;
+ case COMPRESSION_LZ4:
+#ifdef HAVE_LIBLZ4
+ if (compresslevel != 0)
+ {
+ pg_log_error("cannot use --compress with --compression-method=%s",
+ "lz4");
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+ progname);
+ exit(1);
+ }
+#else
+ pg_log_error("this build does not support compression with %s",
+ "LZ4");
+ exit(1);
#endif
break;
}
diff --git a/src/bin/pg_basebackup/t/020_pg_receivewal.pl b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
index 94786f0815..43599d832b 100644
--- a/src/bin/pg_basebackup/t/020_pg_receivewal.pl
+++ b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
@@ -5,7 +5,7 @@ use strict;
use warnings;
use PostgreSQL::Test::Utils;
use PostgreSQL::Test::Cluster;
-use Test::More tests => 37;
+use Test::More tests => 42;
program_help_ok('pg_receivewal');
program_version_ok('pg_receivewal');
@@ -138,13 +138,69 @@ SKIP:
"gzip verified the integrity of compressed WAL segments");
}
+# Check LZ4 compression if available
+SKIP:
+{
+ skip "postgres was not built with LZ4 support", 5
+ if (!check_pg_config("#define HAVE_LIBLZ4 1"));
+
+ # Generate more WAL including one completed, compressed segment.
+ $primary->psql('postgres', 'SELECT pg_switch_wal();');
+ $nextlsn =
+ $primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
+ chomp($nextlsn);
+ $primary->psql('postgres', 'INSERT INTO test_table VALUES (3);');
+
+ # Stream up to the given position.
+ $primary->command_ok(
+ [
+ 'pg_receivewal', '-D',
+ $stream_dir, '--verbose',
+ '--endpos', $nextlsn,
+ '--no-loop', '--compression-method',
+ 'lz4'
+ ],
+ 'streaming some WAL using --compression-method=lz4');
+
+ # Verify that the stored files are generated with their expected
+ # names.
+ my @lz4_wals = glob "$stream_dir/*.lz4";
+ is(scalar(@lz4_wals), 1,
+ "one WAL segment compressed with LZ4 was created");
+ my @lz4_partial_wals = glob "$stream_dir/*.lz4.partial";
+ is(scalar(@lz4_partial_wals),
+ 1, "one partial WAL segment compressed with LZ4 was created");
+
+ # Verify that the start streaming position is computed correctly by
+ # comparing it with the partial file generated previously. The name
+ # of the previous partial, now-completed WAL segment is updated, keeping
+ # its base number.
+ $partial_wals[0] =~ s/(\.gz)?\.partial$/.lz4/;
+ is($lz4_wals[0] eq $partial_wals[0],
+ 1, "one partial WAL segment is now completed");
+ # Update the list of partial wals with the current one.
+ @partial_wals = @lz4_partial_wals;
+
+ # Check the integrity of the completed segment, if LZ4 is an available
+ # command.
+ my $lz4 = $ENV{LZ4};
+ skip "program lz4 is not found in your system", 1
+ if ( !defined $lz4
+ || $lz4 eq ''
+ || system_log($lz4, '--version') != 0);
+
+ my $lz4_is_valid = system_log($lz4, '-t', @lz4_wals);
+ is($lz4_is_valid, 0,
+ "lz4 verified the integrity of compressed WAL segments");
+}
+
# Verify that the start streaming position is computed and that the value is
-# correct regardless of whether ZLIB is available.
+# correct regardless of whether any compression is available.
$primary->psql('postgres', 'SELECT pg_switch_wal();');
$nextlsn =
$primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
chomp($nextlsn);
-$primary->psql('postgres', 'INSERT INTO test_table VALUES (3);');
+$primary->psql('postgres', 'INSERT INTO test_table VALUES (4);');
$primary->command_ok(
[
'pg_receivewal', '-D', $stream_dir, '--verbose',
@@ -152,7 +208,7 @@ $primary->command_ok(
],
"streaming some WAL");
-$partial_wals[0] =~ s/(\.gz)?.partial//;
+$partial_wals[0] =~ s/(\.gz|\.lz4)?.partial//;
ok(-e $partial_wals[0], "check that previously partial WAL is now complete");
# Permissions on WAL files should be default
@@ -190,7 +246,7 @@ my $walfile_streamed = $primary->safe_psql(
# Switch to a new segment, to make sure that the segment retained by the
# slot is still streamed. This may not be necessary, but play it safe.
-$primary->psql('postgres', 'INSERT INTO test_table VALUES (4);');
+$primary->psql('postgres', 'INSERT INTO test_table VALUES (5);');
$primary->psql('postgres', 'SELECT pg_switch_wal();');
$nextlsn =
$primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
@@ -198,7 +254,7 @@ chomp($nextlsn);
# Add a bit more data to accelerate the end of the next pg_receivewal
# commands.
-$primary->psql('postgres', 'INSERT INTO test_table VALUES (5);');
+$primary->psql('postgres', 'INSERT INTO test_table VALUES (6);');
# Check case where the slot does not exist.
$primary->command_fails_like(
@@ -253,13 +309,13 @@ $standby->promote;
# on the new timeline.
my $walfile_after_promotion = $standby->safe_psql('postgres',
"SELECT pg_walfile_name(pg_current_wal_insert_lsn());");
-$standby->psql('postgres', 'INSERT INTO test_table VALUES (6);');
+$standby->psql('postgres', 'INSERT INTO test_table VALUES (7);');
$standby->psql('postgres', 'SELECT pg_switch_wal();');
$nextlsn =
$standby->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
chomp($nextlsn);
# This speeds up the operation.
-$standby->psql('postgres', 'INSERT INTO test_table VALUES (7);');
+$standby->psql('postgres', 'INSERT INTO test_table VALUES (8);');
# Now try to resume from the slot after the promotion.
my $timeline_dir = $primary->basedir . '/timeline_wal';
diff --git a/src/bin/pg_basebackup/walmethods.c b/src/bin/pg_basebackup/walmethods.c
index 52f314af3b..f1ba2a828a 100644
--- a/src/bin/pg_basebackup/walmethods.c
+++ b/src/bin/pg_basebackup/walmethods.c
@@ -17,6 +17,10 @@
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
+
+#ifdef HAVE_LIBLZ4
+#include <lz4frame.h>
+#endif
#ifdef HAVE_LIBZ
#include <zlib.h>
#endif
@@ -30,6 +34,9 @@
/* Size of zlib buffer for .tar.gz */
#define ZLIB_OUT_SIZE 4096
+/* Size of LZ4 input chunk for .lz4 */
+#define LZ4_IN_SIZE 4096
+
/*-------------------------------------------------------------------------
* WalDirectoryMethod - write wal to a directory looking like pg_wal
*-------------------------------------------------------------------------
@@ -60,6 +67,11 @@ typedef struct DirectoryMethodFile
#ifdef HAVE_LIBZ
gzFile gzfp;
#endif
+#ifdef HAVE_LIBLZ4
+ LZ4F_compressionContext_t ctx;
+ size_t lz4bufsize;
+ void *lz4buf;
+#endif
} DirectoryMethodFile;
static const char *
@@ -76,7 +88,8 @@ dir_get_file_name(const char *pathname, const char *temp_suffix)
snprintf(filename, MAXPGPATH, "%s%s%s",
pathname,
- dir_data->compression_method == COMPRESSION_GZIP ? ".gz" : "",
+ dir_data->compression_method == COMPRESSION_GZIP ? ".gz" :
+ dir_data->compression_method == COMPRESSION_LZ4 ? ".lz4" : "",
temp_suffix ? temp_suffix : "");
return filename;
@@ -92,6 +105,11 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
#ifdef HAVE_LIBZ
gzFile gzfp = NULL;
#endif
+#ifdef HAVE_LIBLZ4
+ LZ4F_compressionContext_t ctx = NULL;
+ size_t lz4bufsize = 0;
+ void *lz4buf = NULL;
+#endif
filename = dir_get_file_name(pathname, temp_suffix);
snprintf(tmppath, sizeof(tmppath), "%s/%s",
@@ -126,6 +144,50 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
}
}
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ size_t ctx_out;
+ size_t header_size;
+
+ ctx_out = LZ4F_createCompressionContext(&ctx, LZ4F_VERSION);
+ if (LZ4F_isError(ctx_out))
+ {
+ close(fd);
+ return NULL;
+ }
+
+ lz4bufsize = LZ4F_compressBound(LZ4_IN_SIZE, NULL);
+ lz4buf = pg_malloc0(lz4bufsize);
+
+ /* add the header */
+ header_size = LZ4F_compressBegin(ctx, lz4buf, lz4bufsize, NULL);
+ if (LZ4F_isError(header_size))
+ {
+ (void) LZ4F_freeCompressionContext(ctx);
+ pg_free(lz4buf);
+ close(fd);
+ return NULL;
+ }
+
+ errno = 0;
+ if (write(fd, lz4buf, header_size) != header_size)
+ {
+ int save_errno = errno;
+
+ (void) LZ4F_compressEnd(ctx, lz4buf, lz4bufsize, NULL);
+ (void) LZ4F_freeCompressionContext(ctx);
+ pg_free(lz4buf);
+ close(fd);
+
+ /*
+ * If write didn't set errno, assume problem is no disk space.
+ */
+ errno = save_errno ? save_errno : ENOSPC;
+ return NULL;
+ }
+ }
+#endif
/* Do pre-padding on non-compressed files */
if (pad_to_size && dir_data->compression_method == COMPRESSION_NONE)
@@ -176,6 +238,16 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
if (dir_data->compression_method == COMPRESSION_GZIP)
gzclose(gzfp);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ (void) LZ4F_compressEnd(ctx, lz4buf, lz4bufsize, NULL);
+ (void) LZ4F_freeCompressionContext(ctx);
+ pg_free(lz4buf);
+ close(fd);
+ }
+ else
#endif
close(fd);
return NULL;
@@ -187,6 +259,15 @@ dir_open_for_write(const char *pathname, const char *temp_suffix, size_t pad_to_
if (dir_data->compression_method == COMPRESSION_GZIP)
f->gzfp = gzfp;
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ f->ctx = ctx;
+ f->lz4buf = lz4buf;
+ f->lz4bufsize = lz4bufsize;
+ }
+#endif
+
f->fd = fd;
f->currpos = 0;
f->pathname = pg_strdup(pathname);
@@ -209,6 +290,43 @@ dir_write(Walfile f, const void *buf, size_t count)
if (dir_data->compression_method == COMPRESSION_GZIP)
r = (ssize_t) gzwrite(df->gzfp, buf, count);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ size_t chunk;
+ size_t remaining;
+ const void *inbuf = buf;
+
+ remaining = count;
+ while (remaining > 0)
+ {
+ size_t compressed;
+
+ if (remaining > LZ4_IN_SIZE)
+ chunk = LZ4_IN_SIZE;
+ else
+ chunk = remaining;
+
+ remaining -= chunk;
+ compressed = LZ4F_compressUpdate(df->ctx,
+ df->lz4buf, df->lz4bufsize,
+ inbuf, chunk,
+ NULL);
+
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+
+ inbuf = ((char *) inbuf) + chunk;
+ }
+
+ /* Our caller keeps track of the uncompressed size. */
+ r = (ssize_t) count;
+ }
+ else
#endif
r = write(df->fd, buf, count);
if (r > 0)
@@ -239,6 +357,25 @@ dir_close(Walfile f, WalCloseMethod method)
if (dir_data->compression_method == COMPRESSION_GZIP)
r = gzclose(df->gzfp);
else
+#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ size_t compressed;
+
+ compressed = LZ4F_compressEnd(df->ctx,
+ df->lz4buf, df->lz4bufsize,
+ NULL);
+
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+
+ r = close(df->fd);
+ }
+ else
#endif
r = close(df->fd);
@@ -293,6 +430,12 @@ dir_close(Walfile f, WalCloseMethod method)
}
}
+#ifdef HAVE_LIBLZ4
+ pg_free(df->lz4buf);
+ /* supports free on NULL */
+ LZ4F_freeCompressionContext(df->ctx);
+#endif
+
pg_free(df->pathname);
pg_free(df->fullpath);
if (df->temp_suffix)
@@ -317,6 +460,21 @@ dir_sync(Walfile f)
return -1;
}
#endif
+#ifdef HAVE_LIBLZ4
+ if (dir_data->compression_method == COMPRESSION_LZ4)
+ {
+ DirectoryMethodFile *df = (DirectoryMethodFile *) f;
+ size_t compressed;
+
+ /* Flush any internal buffers */
+ compressed = LZ4F_flush(df->ctx, df->lz4buf, df->lz4bufsize, NULL);
+ if (LZ4F_isError(compressed))
+ return -1;
+
+ if (write(df->fd, df->lz4buf, compressed) != compressed)
+ return -1;
+ }
+#endif
return fsync(((DirectoryMethodFile *) f)->fd);
}
diff --git a/src/bin/pg_basebackup/walmethods.h b/src/bin/pg_basebackup/walmethods.h
index 5dfe330ea5..f9b6a1646a 100644
--- a/src/bin/pg_basebackup/walmethods.h
+++ b/src/bin/pg_basebackup/walmethods.h
@@ -22,6 +22,7 @@ typedef enum
/* Types of compression supported */
typedef enum
{
+ COMPRESSION_LZ4,
COMPRESSION_GZIP,
COMPRESSION_NONE
} WalCompressionMethod;
--
2.25.1
Re: Teach pg_receivewal to use lz4 compression
On Thu, Nov 04, 2021 at 05:02:28PM +0000, gkokolatos@pm.me wrote:
Removed an extra condinional check while switching over compression_method.
Indeed. My rebase was a bit sloppy here.
because compression_method is the global option exposed to the whereas
wal_compression_method is the local variable used to figure out what kind of
file the function is currently working with. This error was existing at least in
v9-0002 of $subject.
Right.
I felt that converting it a do {} while () loop instead, will help with readability: + do + { <snip> + /* + * No need to continue reading the file when the uncompressed_size + * exceeds WalSegSz, even if there are still data left to read. + * However, if uncompressed_size is equal to WalSegSz, it should + * verify that there is no more data to read. + */ + } while (r > 0 && uncompressed_size <= WalSegSz);
No objections from me to do that. This makes the code a bit easier to
follow, indeed.
I would like to have a bit more test coverage in the case for FindStreamingStart().
Specifically for the case that a lz4-compressed segment larger than WalSegSz exists.
The same could be said for gzip. I am not sure that this is worth the
extra I/O and pg_receivewal commands, though.
I have spent an extra couple of hours staring at the code, and the
whole looked fine, so applied. While on it, I have tested the new TAP
tests with all the possible combinations of --without-zlib and
--with-lz4.
--
Michael
Re: Teach pg_receivewal to use lz4 compression
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, November 5th, 2021 at 3:47 AM, Michael Paquier <michael@paquier.xyz> wrote:
I have spent an extra couple of hours staring at the code, and the
whole looked fine, so applied. While on it, I have tested the new TAP
tests with all the possible combinations of --without-zlib and
--with-lz4.
Great news. Thank you very much.
Cheers,
//Georgios
Show quoted text
--
Michael
Re: Teach pg_receivewal to use lz4 compression
In dir_open_for_write() I observe that we are writing the header
and then calling LZ4F_compressEnd() in case there is an error
while writing the buffer to the file, and the output buffer of
LZ4F_compressEnd() is not written anywhere. Why should this be
necessary? To flush the contents of the internal buffer? But, then we
are calling LZ4F_freeCompressionContext() immediately after the
LZ4F_compressEnd() call. I might be missing something, will be
happy to get more insights.
Regards,
Jeevan Ladhe
On Fri, Nov 5, 2021 at 1:21 PM <gkokolatos@pm.me> wrote:
Show quoted text
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, November 5th, 2021 at 3:47 AM, Michael Paquier <
michael@paquier.xyz> wrote:I have spent an extra couple of hours staring at the code, and the
whole looked fine, so applied. While on it, I have tested the new TAP
tests with all the possible combinations of --without-zlib and
--with-lz4.Great news. Thank you very much.
Cheers,
//Georgios--
Michael
Re: Teach pg_receivewal to use lz4 compression
On Thu, Nov 18, 2021 at 07:54:37PM +0530, Jeevan Ladhe wrote:
In dir_open_for_write() I observe that we are writing the header
and then calling LZ4F_compressEnd() in case there is an error
while writing the buffer to the file, and the output buffer of
LZ4F_compressEnd() is not written anywhere. Why should this be
necessary? To flush the contents of the internal buffer? But, then we
are calling LZ4F_freeCompressionContext() immediately after the
LZ4F_compressEnd() call. I might be missing something, will be
happy to get more insights.
My concern here was symmetry, where IMO it makes sense to have a
compressEnd call each time there is a successful compressBegin call
done for the LZ4 state data, as there is no way to know if in the
future LZ4 won't change some of its internals to do more than just an
internal flush.
--
Michael
Re: Teach pg_receivewal to use lz4 compression
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, November 19th, 2021 at 3:07 AM, Michael Paquier <michael@paquier.xyz> wrote:
On Thu, Nov 18, 2021 at 07:54:37PM +0530, Jeevan Ladhe wrote:
In dir_open_for_write() I observe that we are writing the header
and then calling LZ4F_compressEnd() in case there is an error
while writing the buffer to the file, and the output buffer of
LZ4F_compressEnd() is not written anywhere. Why should this be
necessary? To flush the contents of the internal buffer? But, then we
are calling LZ4F_freeCompressionContext() immediately after the
LZ4F_compressEnd() call. I might be missing something, will be
happy to get more insights.My concern here was symmetry, where IMO it makes sense to have a
compressEnd call each time there is a successful compressBegin call
done for the LZ4 state data, as there is no way to know if in the
future LZ4 won't change some of its internals to do more than just an
internal flush.
Agreed.
Although the library does provide an interface for simply flushing contents, it
also assumes that each initializing call will have a finilizing call. If my
memory serves me right, earlier versions of the patch, did not have this
summetry, but that got ammended.
Cheers,
//Georgios
Show quoted text
---
Michael
Re: Teach pg_receivewal to use lz4 compression
On Fri, Nov 19, 2021 at 7:37 AM Michael Paquier <michael@paquier.xyz> wrote:
On Thu, Nov 18, 2021 at 07:54:37PM +0530, Jeevan Ladhe wrote:
In dir_open_for_write() I observe that we are writing the header
and then calling LZ4F_compressEnd() in case there is an error
while writing the buffer to the file, and the output buffer of
LZ4F_compressEnd() is not written anywhere. Why should this be
necessary? To flush the contents of the internal buffer? But, then we
are calling LZ4F_freeCompressionContext() immediately after the
LZ4F_compressEnd() call. I might be missing something, will be
happy to get more insights.My concern here was symmetry, where IMO it makes sense to have a
compressEnd call each time there is a successful compressBegin call
done for the LZ4 state data, as there is no way to know if in the
future LZ4 won't change some of its internals to do more than just an
internal flush.
Fair enough. But, still I have a doubt in mind what benefit would that
really bring to us here, because we are immediately also freeing the
lz4buf without using it anywhere.
Regards,
Jeevan
Re: Teach pg_receivewal to use lz4 compression
On Mon, Nov 22, 2021 at 12:46 AM Jeevan Ladhe
<jeevan.ladhe@enterprisedb.com> wrote:
Fair enough. But, still I have a doubt in mind what benefit would that
really bring to us here, because we are immediately also freeing the
lz4buf without using it anywhere.
Yeah, I'm also doubtful about that. If we're freeng the compression
context, we shouldn't need to guarantee that it's in any particular
state before doing so. Why would any critical cleanup be part of
LZ4F_compressEnd() rather than LZ4F_freeCompressionContext()? The
point of LZ4F_compressEnd() is to make sure all of the output bytes
get written, and it would be stupid to force people to write the
output bytes even when they've decided that they no longer care about
them due to some error.
--
Robert Haas
EDB: http://www.enterprisedb.com
Re: Teach pg_receivewal to use lz4 compression
On Mon, Nov 22, 2021 at 09:02:47AM -0500, Robert Haas wrote:
On Mon, Nov 22, 2021 at 12:46 AM Jeevan Ladhe
<jeevan.ladhe@enterprisedb.com> wrote:Fair enough. But, still I have a doubt in mind what benefit would that
really bring to us here, because we are immediately also freeing the
lz4buf without using it anywhere.Yeah, I'm also doubtful about that. If we're freeng the compression
context, we shouldn't need to guarantee that it's in any particular
state before doing so. Why would any critical cleanup be part of
LZ4F_compressEnd() rather than LZ4F_freeCompressionContext()? The
point of LZ4F_compressEnd() is to make sure all of the output bytes
get written, and it would be stupid to force people to write the
output bytes even when they've decided that they no longer care about
them due to some error.
Hmm. I have double-checked all that, and I agree that we could just
skip LZ4F_compressEnd() in this error code path. From what I can see
in the upstream code, what we have now is not broken either, but the
compressEnd() call does some work that's not needed here.
--
Michael
Re: Teach pg_receivewal to use lz4 compression
On Wed, Nov 24, 2021 at 10:55 AM Michael Paquier <michael@paquier.xyz>
wrote:
On Mon, Nov 22, 2021 at 09:02:47AM -0500, Robert Haas wrote:
On Mon, Nov 22, 2021 at 12:46 AM Jeevan Ladhe
<jeevan.ladhe@enterprisedb.com> wrote:Fair enough. But, still I have a doubt in mind what benefit would that
really bring to us here, because we are immediately also freeing the
lz4buf without using it anywhere.Yeah, I'm also doubtful about that. If we're freeng the compression
context, we shouldn't need to guarantee that it's in any particular
state before doing so. Why would any critical cleanup be part of
LZ4F_compressEnd() rather than LZ4F_freeCompressionContext()? The
point of LZ4F_compressEnd() is to make sure all of the output bytes
get written, and it would be stupid to force people to write the
output bytes even when they've decided that they no longer care about
them due to some error.Hmm. I have double-checked all that, and I agree that we could just
skip LZ4F_compressEnd() in this error code path. From what I can see
in the upstream code, what we have now is not broken either, but the
compressEnd() call does some work that's not needed here.
Yes I agree that we are not broken, but as you said we are doing some
an extra bit of work here.
Regards,
Jeevan Ladhe
Re: Teach pg_receivewal to use lz4 compression
On Thu, Nov 4, 2021 at 10:47 PM Michael Paquier <michael@paquier.xyz> wrote:
Indeed. My rebase was a bit sloppy here.
Hi!
Over in /messages/by-id/CA+TgmoYUDEJga2qV_XbAZ=pGEBaOsgFmzZ6Ac4_sRwOm_+UeHA@mail.gmail.com
I was noticing that CreateWalTarMethod doesn't support LZ4
compression. It would be nice if it did. I thought maybe the patch on
this thread would fix that, but I think maybe it doesn't, because it
looks like that's touching the WalDirectoryMethod part of that file,
rather than the WalTarMethod part. Is that correct? And, on a related
note, Michael, do you plan to get something committed here?
--
Robert Haas
EDB: http://www.enterprisedb.com
Re: Teach pg_receivewal to use lz4 compression
On Fri, Feb 11, 2022 at 10:07:49AM -0500, Robert Haas wrote:
Over in /messages/by-id/CA+TgmoYUDEJga2qV_XbAZ=pGEBaOsgFmzZ6Ac4_sRwOm_+UeHA@mail.gmail.com
I was noticing that CreateWalTarMethod doesn't support LZ4
compression. It would be nice if it did. I thought maybe the patch on
this thread would fix that, but I think maybe it doesn't, because it
looks like that's touching the WalDirectoryMethod part of that file,
rather than the WalTarMethod part. Is that correct?
Correct. pg_receivewal only cares about the directory method, so this
thread was limited to this part. Yes, it would be nice to extend
fully the tar method of walmethods.c to support LZ4, but I was not
sure what needed to be done, and I am still not sure based on what has
just been done as of 751b8d23.
And, on a related note, Michael, do you plan to get something
committed here?
Apart from f79962d, babbbb5 and 50e1441, I don't think that there was
something left to do for this thread. Perhaps I am missing something?
--
Michael
Re: Teach pg_receivewal to use lz4 compression
On Fri, Feb 11, 2022 at 10:52 PM Michael Paquier <michael@paquier.xyz> wrote:
And, on a related note, Michael, do you plan to get something
committed here?Apart from f79962d, babbbb5 and 50e1441, I don't think that there was
something left to do for this thread. Perhaps I am missing something?
Oh, my mistake. I didn't realize you'd already committed it.
--
Robert Haas
EDB: http://www.enterprisedb.com
Re: Teach pg_receivewal to use lz4 compression
On Sat, Feb 12, 2022 at 12:52:40PM +0900, Michael Paquier wrote:
On Fri, Feb 11, 2022 at 10:07:49AM -0500, Robert Haas wrote:
Over in /messages/by-id/CA+TgmoYUDEJga2qV_XbAZ=pGEBaOsgFmzZ6Ac4_sRwOm_+UeHA@mail.gmail.com
I was noticing that CreateWalTarMethod doesn't support LZ4
compression. It would be nice if it did. I thought maybe the patch on
this thread would fix that, but I think maybe it doesn't, because it
looks like that's touching the WalDirectoryMethod part of that file,
rather than the WalTarMethod part. Is that correct?Correct. pg_receivewal only cares about the directory method, so this
thread was limited to this part. Yes, it would be nice to extend
fully the tar method of walmethods.c to support LZ4, but I was not
sure what needed to be done, and I am still not sure based on what has
just been done as of 751b8d23.And, on a related note, Michael, do you plan to get something
committed here?Apart from f79962d, babbbb5 and 50e1441, I don't think that there was
something left to do for this thread. Perhaps I am missing something?
I think this should use <lz4frame.h>
+#include "lz4frame.h"
commit babbbb595d2322da095a1e6703171b3f1f2815cb
Author: Michael Paquier <michael@paquier.xyz>
Date: Fri Nov 5 11:33:25 2021 +0900
Add support for LZ4 compression in pg_receivewal
--
Justin
Re: Teach pg_receivewal to use lz4 compression
On Thu, Mar 17, 2022 at 06:12:20AM -0500, Justin Pryzby wrote:
I think this should use <lz4frame.h>
+#include "lz4frame.h"
commit babbbb595d2322da095a1e6703171b3f1f2815cb
Author: Michael Paquier <michael@paquier.xyz>
Date: Fri Nov 5 11:33:25 2021 +0900Add support for LZ4 compression in pg_receivewal
Yes, you are right. A second thing is that should be declared before
the PG headers.
--
Michael