adding 'zstd' as a compression algorithm

Started by Robert Haasalmost 4 years ago32 messages
#1Robert Haas
robertmhaas@gmail.com
1 attachment(s)

Hi,

Over on the rather-long "refactoring basebackup.c" thread, there is a
proposal, which I endorse, to add base-backup compression via zstd. To
do that, we'd need to patch configure.ac to create a new --with-zstd
flag and appropriate supporting infrastructure. My colleague Jeevan
Ladhe included that in a patch posted over there; I've extracted just
the part adding configure support for libzstd and attach it here. I
thought it would be good to have a new thread specifically devoted to
the topic of whether zstd is a thing that PostgreSQL ought to support
in general.

In general, deciding on new compression algorithms can feel a bit like
debating the merits of vi vs. emacs, or one political party vs.
another. Everyone has their own favorites, for reasons that can often
seem idiosyncratic. One advantage of zstd is that it is already being
used by other prominent open-source projects, including the Linux
kernel.[1]https://en.wikipedia.org/wiki/Zstd#Usage This means that it is unlikely to just dry up and vanish,
and it also reduces the risk of legal issues. On a technical level,
zstd offers compression ratios similar to or better than gzip, but
with much faster compression speed. Furthermore, the zstd library has
built-in multi-threaded compression which we may be able to leverage
for even better performance. In fact, parallel zstd might be able to
compress faster than lz4, which is already extremely fast.

What I imagine if this patch is accepted is that we (or our users)
will end up using lz4 for places where compression needs to be very
lightweight, and zstd for places where it's acceptable or even
desirable to spend more CPU cycles in exchange for better compression.
I think that gzip and pglz are really only of historical interest -
and I don't say that to mean that we shouldn't continue to support
them or that they won't get use. Lots of people are perfectly happy
with TOAST compression using pglz, and I'm perfectly happy if they
continue to do that forever, even though I'm glad LZ4 is now an
option. Likewise, I still download the .tar.gz version of anything
that gives me that option, basically because I'm familiar with the
format and it's easy for me to just carry on using it -- and in a
similar way I expect a lot of people will be happy to continue to
compress backups with gzip for many years to come. But I think there
is value in supporting newer and better technology, too. I realize
that we don't want to support every new and shiny thing that shows up,
but I don't think that's what I am proposing here.

Anyway, those are my thoughts. What are yours?

Thanks,

[1]: https://en.wikipedia.org/wiki/Zstd#Usage

--
Robert Haas
EDB: http://www.enterprisedb.com

Attachments:

0001-Add-support-for-building-with-ZSTD.patchapplication/octet-stream; name=0001-Add-support-for-building-with-ZSTD.patchDownload
From f26057174a00be4deacf4bfd26aa80cf829fd300 Mon Sep 17 00:00:00 2001
From: Robert Haas <rhaas@postgresql.org>
Date: Tue, 15 Feb 2022 11:38:39 -0500
Subject: [PATCH] Add support for building with ZSTD.

This commit doesn't actually add anything that uses ZSTD; that will be
done separately. It just puts the basic infrastructure into place.

Jeevan Ladhe. I extracted these changes from a larger patch by Jeevan
and reran 'autoconf' as the version on Jeevan's machine included some
unwanted changes in configure, but otherwise made no changes.
---
 configure                  | 271 +++++++++++++++++++++++++++++++++++++
 configure.ac               |  33 +++++
 src/Makefile.global.in     |   1 +
 src/include/pg_config.h.in |   9 ++
 4 files changed, 314 insertions(+)

diff --git a/configure b/configure
index 9305555658..f07f689f1a 100755
--- a/configure
+++ b/configure
@@ -650,6 +650,7 @@ CFLAGS_ARMV8_CRC32C
 CFLAGS_SSE42
 have_win32_dbghelp
 LIBOBJS
+ZSTD
 LZ4
 UUID_LIBS
 LDAP_LIBS_BE
@@ -700,6 +701,9 @@ with_gnu_ld
 LD
 LDFLAGS_SL
 LDFLAGS_EX
+ZSTD_LIBS
+ZSTD_CFLAGS
+with_zstd
 LZ4_LIBS
 LZ4_CFLAGS
 with_lz4
@@ -869,6 +873,7 @@ with_libxslt
 with_system_tzdata
 with_zlib
 with_lz4
+with_zstd
 with_gnu_ld
 with_ssl
 with_openssl
@@ -898,6 +903,8 @@ XML2_CFLAGS
 XML2_LIBS
 LZ4_CFLAGS
 LZ4_LIBS
+ZSTD_CFLAGS
+ZSTD_LIBS
 LDFLAGS_EX
 LDFLAGS_SL
 PERL
@@ -1577,6 +1584,7 @@ Optional Packages:
                           use system time zone data in DIR
   --without-zlib          do not use Zlib
   --with-lz4              build with LZ4 support
+  --with-zstd             build with ZSTD support
   --with-gnu-ld           assume the C compiler uses GNU ld [default=no]
   --with-ssl=LIB          use LIB for SSL/TLS support (openssl)
   --with-openssl          obsolete spelling of --with-ssl=openssl
@@ -1606,6 +1614,8 @@ Some influential environment variables:
   XML2_LIBS   linker flags for XML2, overriding pkg-config
   LZ4_CFLAGS  C compiler flags for LZ4, overriding pkg-config
   LZ4_LIBS    linker flags for LZ4, overriding pkg-config
+  ZSTD_CFLAGS C compiler flags for ZSTD, overriding pkg-config
+  ZSTD_LIBS   linker flags for ZSTD, overriding pkg-config
   LDFLAGS_EX  extra linker flags for linking executables only
   LDFLAGS_SL  extra linker flags for linking shared libraries only
   PERL        Perl program
@@ -9034,6 +9044,146 @@ fi
   done
 fi
 
+#
+# ZSTD
+#
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with ZSTD support" >&5
+$as_echo_n "checking whether to build with ZSTD support... " >&6; }
+
+
+
+# Check whether --with-zstd was given.
+if test "${with_zstd+set}" = set; then :
+  withval=$with_zstd;
+  case $withval in
+    yes)
+
+$as_echo "#define USE_ZSTD 1" >>confdefs.h
+
+      ;;
+    no)
+      :
+      ;;
+    *)
+      as_fn_error $? "no argument expected for --with-zstd option" "$LINENO" 5
+      ;;
+  esac
+
+else
+  with_zstd=no
+
+fi
+
+
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_zstd" >&5
+$as_echo "$with_zstd" >&6; }
+
+
+if test "$with_zstd" = yes; then
+
+pkg_failed=no
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for libzstd" >&5
+$as_echo_n "checking for libzstd... " >&6; }
+
+if test -n "$ZSTD_CFLAGS"; then
+    pkg_cv_ZSTD_CFLAGS="$ZSTD_CFLAGS"
+ elif test -n "$PKG_CONFIG"; then
+    if test -n "$PKG_CONFIG" && \
+    { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"libzstd\""; } >&5
+  ($PKG_CONFIG --exists --print-errors "libzstd") 2>&5
+  ac_status=$?
+  $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+  test $ac_status = 0; }; then
+  pkg_cv_ZSTD_CFLAGS=`$PKG_CONFIG --cflags "libzstd" 2>/dev/null`
+		      test "x$?" != "x0" && pkg_failed=yes
+else
+  pkg_failed=yes
+fi
+ else
+    pkg_failed=untried
+fi
+if test -n "$ZSTD_LIBS"; then
+    pkg_cv_ZSTD_LIBS="$ZSTD_LIBS"
+ elif test -n "$PKG_CONFIG"; then
+    if test -n "$PKG_CONFIG" && \
+    { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"libzstd\""; } >&5
+  ($PKG_CONFIG --exists --print-errors "libzstd") 2>&5
+  ac_status=$?
+  $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+  test $ac_status = 0; }; then
+  pkg_cv_ZSTD_LIBS=`$PKG_CONFIG --libs "libzstd" 2>/dev/null`
+		      test "x$?" != "x0" && pkg_failed=yes
+else
+  pkg_failed=yes
+fi
+ else
+    pkg_failed=untried
+fi
+
+
+
+if test $pkg_failed = yes; then
+        { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+
+if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then
+        _pkg_short_errors_supported=yes
+else
+        _pkg_short_errors_supported=no
+fi
+        if test $_pkg_short_errors_supported = yes; then
+	        ZSTD_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "libzstd" 2>&1`
+        else
+	        ZSTD_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "libzstd" 2>&1`
+        fi
+	# Put the nasty error message in config.log where it belongs
+	echo "$ZSTD_PKG_ERRORS" >&5
+
+	as_fn_error $? "Package requirements (libzstd) were not met:
+
+$ZSTD_PKG_ERRORS
+
+Consider adjusting the PKG_CONFIG_PATH environment variable if you
+installed software in a non-standard prefix.
+
+Alternatively, you may set the environment variables ZSTD_CFLAGS
+and ZSTD_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details." "$LINENO" 5
+elif test $pkg_failed = untried; then
+        { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+	{ { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
+$as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
+as_fn_error $? "The pkg-config script could not be found or is too old.  Make sure it
+is in your PATH or set the PKG_CONFIG environment variable to the full
+path to pkg-config.
+
+Alternatively, you may set the environment variables ZSTD_CFLAGS
+and ZSTD_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details.
+
+To get pkg-config, see <http://pkg-config.freedesktop.org/>.
+See \`config.log' for more details" "$LINENO" 5; }
+else
+	ZSTD_CFLAGS=$pkg_cv_ZSTD_CFLAGS
+	ZSTD_LIBS=$pkg_cv_ZSTD_LIBS
+        { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
+$as_echo "yes" >&6; }
+
+fi
+  # We only care about -I, -D, and -L switches;
+  # note that -lzstd will be added by AC_CHECK_LIB below.
+  for pgac_option in $ZSTD_CFLAGS; do
+    case $pgac_option in
+      -I*|-D*) CPPFLAGS="$CPPFLAGS $pgac_option";;
+    esac
+  done
+  for pgac_option in $ZSTD_LIBS; do
+    case $pgac_option in
+      -L*) LDFLAGS="$LDFLAGS $pgac_option";;
+    esac
+  done
+fi
 #
 # Assignments
 #
@@ -13130,6 +13280,56 @@ fi
 
 fi
 
+if test "$with_zstd" = yes ; then
+  { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ZSTD_compress in -lzstd" >&5
+$as_echo_n "checking for ZSTD_compress in -lzstd... " >&6; }
+if ${ac_cv_lib_zstd_ZSTD_compress+:} false; then :
+  $as_echo_n "(cached) " >&6
+else
+  ac_check_lib_save_LIBS=$LIBS
+LIBS="-lzstd  $LIBS"
+cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h.  */
+
+/* Override any GCC internal prototype to avoid an error.
+   Use char because int might match the return type of a GCC
+   builtin and then its argument prototype would still apply.  */
+#ifdef __cplusplus
+extern "C"
+#endif
+char ZSTD_compress ();
+int
+main ()
+{
+return ZSTD_compress ();
+  ;
+  return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+  ac_cv_lib_zstd_ZSTD_compress=yes
+else
+  ac_cv_lib_zstd_ZSTD_compress=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+    conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_zstd_ZSTD_compress" >&5
+$as_echo "$ac_cv_lib_zstd_ZSTD_compress" >&6; }
+if test "x$ac_cv_lib_zstd_ZSTD_compress" = xyes; then :
+  cat >>confdefs.h <<_ACEOF
+#define HAVE_LIBZSTD 1
+_ACEOF
+
+  LIBS="-lzstd $LIBS"
+
+else
+  as_fn_error $? "library 'zstd' is required for ZSTD support" "$LINENO" 5
+fi
+
+fi
+
 # Note: We can test for libldap_r only after we know PTHREAD_LIBS;
 # also, on AIX, we may need to have openssl in LIBS for this step.
 if test "$with_ldap" = yes ; then
@@ -13904,6 +14104,77 @@ done
 
 fi
 
+if test -z "$ZSTD"; then
+  for ac_prog in zstd
+do
+  # Extract the first word of "$ac_prog", so it can be a program name with args.
+set dummy $ac_prog; ac_word=$2
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
+$as_echo_n "checking for $ac_word... " >&6; }
+if ${ac_cv_path_ZSTD+:} false; then :
+  $as_echo_n "(cached) " >&6
+else
+  case $ZSTD in
+  [\\/]* | ?:[\\/]*)
+  ac_cv_path_ZSTD="$ZSTD" # Let the user override the test with a path.
+  ;;
+  *)
+  as_save_IFS=$IFS; IFS=$PATH_SEPARATOR
+for as_dir in $PATH
+do
+  IFS=$as_save_IFS
+  test -z "$as_dir" && as_dir=.
+    for ac_exec_ext in '' $ac_executable_extensions; do
+  if as_fn_executable_p "$as_dir/$ac_word$ac_exec_ext"; then
+    ac_cv_path_ZSTD="$as_dir/$ac_word$ac_exec_ext"
+    $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5
+    break 2
+  fi
+done
+  done
+IFS=$as_save_IFS
+
+  ;;
+esac
+fi
+ZSTD=$ac_cv_path_ZSTD
+if test -n "$ZSTD"; then
+  { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ZSTD" >&5
+$as_echo "$ZSTD" >&6; }
+else
+  { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+fi
+
+
+  test -n "$ZSTD" && break
+done
+
+else
+  # Report the value of ZSTD in configure's output in all cases.
+  { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ZSTD" >&5
+$as_echo_n "checking for ZSTD... " >&6; }
+  { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ZSTD" >&5
+$as_echo "$ZSTD" >&6; }
+fi
+
+if test "$with_zstd" = yes; then
+  for ac_header in zstd.h
+do :
+  ac_fn_c_check_header_mongrel "$LINENO" "zstd.h" "ac_cv_header_zstd_h" "$ac_includes_default"
+if test "x$ac_cv_header_zstd_h" = xyes; then :
+  cat >>confdefs.h <<_ACEOF
+#define HAVE_ZSTD_H 1
+_ACEOF
+
+else
+  as_fn_error $? "zstd.h header file is required for ZSTD" "$LINENO" 5
+fi
+
+done
+
+fi
+
 if test "$with_gssapi" = yes ; then
   for ac_header in gssapi/gssapi.h
 do :
diff --git a/configure.ac b/configure.ac
index 16167329fc..729b23fbea 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1056,6 +1056,30 @@ if test "$with_lz4" = yes; then
   done
 fi
 
+#
+# ZSTD
+#
+AC_MSG_CHECKING([whether to build with ZSTD support])
+PGAC_ARG_BOOL(with, zstd, no, [build with ZSTD support],
+              [AC_DEFINE([USE_ZSTD], 1, [Define to 1 to build with ZSTD support. (--with-zstd)])])
+AC_MSG_RESULT([$with_zstd])
+AC_SUBST(with_zstd)
+
+if test "$with_zstd" = yes; then
+  PKG_CHECK_MODULES(ZSTD, libzstd)
+  # We only care about -I, -D, and -L switches;
+  # note that -lzstd will be added by AC_CHECK_LIB below.
+  for pgac_option in $ZSTD_CFLAGS; do
+    case $pgac_option in
+      -I*|-D*) CPPFLAGS="$CPPFLAGS $pgac_option";;
+    esac
+  done
+  for pgac_option in $ZSTD_LIBS; do
+    case $pgac_option in
+      -L*) LDFLAGS="$LDFLAGS $pgac_option";;
+    esac
+  done
+fi
 #
 # Assignments
 #
@@ -1325,6 +1349,10 @@ if test "$with_lz4" = yes ; then
   AC_CHECK_LIB(lz4, LZ4_compress_default, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])])
 fi
 
+if test "$with_zstd" = yes ; then
+  AC_CHECK_LIB(zstd, ZSTD_compress, [], [AC_MSG_ERROR([library 'zstd' is required for ZSTD support])])
+fi
+
 # Note: We can test for libldap_r only after we know PTHREAD_LIBS;
 # also, on AIX, we may need to have openssl in LIBS for this step.
 if test "$with_ldap" = yes ; then
@@ -1490,6 +1518,11 @@ if test "$with_lz4" = yes; then
   AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])
 fi
 
+PGAC_PATH_PROGS(ZSTD, zstd)
+if test "$with_zstd" = yes; then
+  AC_CHECK_HEADERS(zstd.h, [], [AC_MSG_ERROR([zstd.h header file is required for ZSTD])])
+fi
+
 if test "$with_gssapi" = yes ; then
   AC_CHECK_HEADERS(gssapi/gssapi.h, [],
 	[AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])])
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 9dcd54fcbd..c980444233 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -351,6 +351,7 @@ XGETTEXT = @XGETTEXT@
 GZIP	= gzip
 BZIP2	= bzip2
 LZ4	= @LZ4@
+ZSTD	= @ZSTD@
 
 DOWNLOAD = wget -O $@ --no-use-server-timestamps
 #DOWNLOAD = curl -o $@
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 28a1f0e9f0..1912cf35de 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -352,6 +352,9 @@
 /* Define to 1 if you have the `z' library (-lz). */
 #undef HAVE_LIBZ
 
+/* Define to 1 if you have the `zstd' library (-lzstd). */
+#undef HAVE_LIBZSTD
+
 /* Define to 1 if you have the `link' function. */
 #undef HAVE_LINK
 
@@ -718,6 +721,9 @@
 /* Define to 1 if the assembler supports X86_64's POPCNTQ instruction. */
 #undef HAVE_X86_64_POPCNTQ
 
+/* Define to 1 if you have the <zstd.h> header file. */
+#undef HAVE_ZSTD_H
+
 /* Define to 1 if the system has the type `_Bool'. */
 #undef HAVE__BOOL
 
@@ -949,6 +955,9 @@
 /* Define to select Win32-style shared memory. */
 #undef USE_WIN32_SHARED_MEMORY
 
+/* Define to 1 to build with ZSTD support. (--with-zstd) */
+#undef USE_ZSTD
+
 /* Define to 1 if `wcstombs_l' requires <xlocale.h>. */
 #undef WCSTOMBS_L_IN_XLOCALE
 
-- 
2.24.3 (Apple Git-128)

#2Justin Pryzby
pryzby@telsasoft.com
In reply to: Robert Haas (#1)
Re: adding 'zstd' as a compression algorithm

I think the WAL patch (4035cd5d4) should support zstd if library support is
added. A supplementary patch for that already exists.
/messages/by-id/YNqWd2GSMrnqWIfx@paquier.xyz

There's also some in-progress patches:

- Konstantin and Daniil have a patch to add libpq compression, which ultimately
wants to use zstd.
https://commitfest.postgresql.org/37/3499/

- I had a patch to add zstd to pg_dump, but I ended up storing backups to
ZFS+zstd rather than waiting to progress the patch.
https://commitfest.postgresql.org/32/2888/

Regarding the patch:

I suppose it should support windows, and whatever patches use zstd should
update the install instructions. See 9ca40dcd4, 02a93e7ef, 4035cd5d4.

In reply to: Robert Haas (#1)
Re: adding 'zstd' as a compression algorithm

On Tue, Feb 15, 2022 at 10:20 AM Robert Haas <robertmhaas@gmail.com> wrote:

In general, deciding on new compression algorithms can feel a bit like
debating the merits of vi vs. emacs, or one political party vs.
another.

Really? I don't get that impression myself. (That's not important, though.)

What I imagine if this patch is accepted is that we (or our users)
will end up using lz4 for places where compression needs to be very
lightweight, and zstd for places where it's acceptable or even
desirable to spend more CPU cycles in exchange for better compression.

Sounds reasonable to me.

Likewise, I still download the .tar.gz version of anything
that gives me that option, basically because I'm familiar with the
format and it's easy for me to just carry on using it -- and in a
similar way I expect a lot of people will be happy to continue to
compress backups with gzip for many years to come.

Maybe, but it seems more likely that they'll usually do whatever the
easiest reasonable-seeming thing is. Perhaps we need to distinguish
between "container format" (e.g., a backup that is produced by a tool
like pgBackrest, including backup manifest) and compression algorithm.

Isn't it an incontrovertible fact that LZ4 is superior to pglz in
every way? LZ4 is pretty much its successor. And so it seems totally
fine to assume that users will always want to use the clearly better
option, and that that option will be generally available going
forward. TOAST compression is applied selectively already, based on
various obscure implementation details, some of which are quite
arbitrary.

I know less about zstd, but I imagine that a similar dynamic exists
there. Overall, everything that you've said makes sense to me.

--
Peter Geoghegan

#4Andres Freund
andres@anarazel.de
In reply to: Peter Geoghegan (#3)
Re: adding 'zstd' as a compression algorithm

Hi,

On 2022-02-15 11:40:20 -0800, Peter Geoghegan wrote:

On Tue, Feb 15, 2022 at 10:20 AM Robert Haas <robertmhaas@gmail.com> wrote:

Likewise, I still download the .tar.gz version of anything
that gives me that option, basically because I'm familiar with the
format and it's easy for me to just carry on using it -- and in a
similar way I expect a lot of people will be happy to continue to
compress backups with gzip for many years to come.

There's a difference between downloading a source tarball of 1.5x the size,
and 3x the decompression cost (made up numbers), because the cost of either is
not a relevant factor. I think basebackups are a different story.

Isn't it an incontrovertible fact that LZ4 is superior to pglz in
every way? LZ4 is pretty much its successor. And so it seems totally
fine to assume that users will always want to use the clearly better
option, and that that option will be generally available going
forward. TOAST compression is applied selectively already, based on
various obscure implementation details, some of which are quite
arbitrary.

Yea, we should really default to lz4 in initdb when available. Expecting users
to know about magic incantation to get often vastly superior performance isn't
a good approach. And it even makes initdb itself faster, because it turns out
we compress enough that the cpu cycles for it are a measurable factor.

I know less about zstd, but I imagine that a similar dynamic exists
there. Overall, everything that you've said makes sense to me.

IMO zstd has pretty much won the "gzip" type usecase of compression. Even
prior lz4 uses have even been converted to it because it's often cheap enough.

Greetings,

Andres Freund

#5Robert Haas
robertmhaas@gmail.com
In reply to: Peter Geoghegan (#3)
Re: adding 'zstd' as a compression algorithm

On Tue, Feb 15, 2022 at 2:40 PM Peter Geoghegan <pg@bowt.ie> wrote:

On Tue, Feb 15, 2022 at 10:20 AM Robert Haas <robertmhaas@gmail.com> wrote:

In general, deciding on new compression algorithms can feel a bit like
debating the merits of vi vs. emacs, or one political party vs.
another.

Really? I don't get that impression myself. (That's not important, though.)

Well, that's a good thing. Maybe we're getting better at keeping
things constructive...

What I imagine if this patch is accepted is that we (or our users)
will end up using lz4 for places where compression needs to be very
lightweight, and zstd for places where it's acceptable or even
desirable to spend more CPU cycles in exchange for better compression.

Sounds reasonable to me.

Cool.

Likewise, I still download the .tar.gz version of anything
that gives me that option, basically because I'm familiar with the
format and it's easy for me to just carry on using it -- and in a
similar way I expect a lot of people will be happy to continue to
compress backups with gzip for many years to come.

Maybe, but it seems more likely that they'll usually do whatever the
easiest reasonable-seeming thing is. Perhaps we need to distinguish
between "container format" (e.g., a backup that is produced by a tool
like pgBackrest, including backup manifest) and compression algorithm.

I'm not sure I completely follow this. There are cases where we use
compression algorithms for internal purposes, and we can change the
defaults without users knowing or caring. For example, we could decide
that LZ4-compressing FPIs in WAL records is a sensible default and
just start doing it, and users need not care. But backups are
different, because when you pg_basebackup -Ft, you get .tar or .tar.gz
or .tar.lz4 files which we don't give you tools to extract. You have
to be able to extract them with OS tools. In cases like that, people
are going to be more likely to pick formats with which they are
familiar even if they are not technically best, and I don't think we
really dare change the defaults. That seems OK, though.

Isn't it an incontrovertible fact that LZ4 is superior to pglz in
every way? LZ4 is pretty much its successor. And so it seems totally
fine to assume that users will always want to use the clearly better
option, and that that option will be generally available going
forward. TOAST compression is applied selectively already, based on
various obscure implementation details, some of which are quite
arbitrary.

I pretty much agree with all of that. Nevertheless, there's a lot of
pglz-compressed data that's already on a disk someplace which people
are not necessarily keen to rewrite, and that will probably remain
true for the foreseable future. If we change the default to something
that's not pglz, and then wait 10 years, we MIGHT be able to get rid
of it without pushback. But I wouldn't be surprised to learn that even
then there are a lot of people who have just pg_upgrade'd the same
database over and over again. And honestly I think that's fine. Yeah,
lz4 is better. But, on some data sets, there's very little real
difference in the compression ratio you get. Until keeping pglz around
starts costing us something tangible, there's no reason to spend any
effort worrying about it.

I know less about zstd, but I imagine that a similar dynamic exists
there. Overall, everything that you've said makes sense to me.

Great, thanks for chiming in.

--
Robert Haas
EDB: http://www.enterprisedb.com

#6Robert Haas
robertmhaas@gmail.com
In reply to: Andres Freund (#4)
Re: adding 'zstd' as a compression algorithm

On Tue, Feb 15, 2022 at 2:54 PM Andres Freund <andres@anarazel.de> wrote:

There's a difference between downloading a source tarball of 1.5x the size,
and 3x the decompression cost (made up numbers), because the cost of either is
not a relevant factor. I think basebackups are a different story.

To be clear, I'm not saying that people shouldn't choose to adopt the
new stuff. I'm just saying that many of them won't, for various
reasons, including inertia. There may come a point when we want to
make a push to remove obsolete stuff, but I think what is far more
important right now is making the new stuff available.

Yea, we should really default to lz4 in initdb when available. Expecting users
to know about magic incantation to get often vastly superior performance isn't
a good approach. And it even makes initdb itself faster, because it turns out
we compress enough that the cpu cycles for it are a measurable factor.

I don't mind if you want to propose a patch for that, but it seems
like a separate topic from this thread.

IMO zstd has pretty much won the "gzip" type usecase of compression. Even
prior lz4 uses have even been converted to it because it's often cheap enough.

You know more about this than I do...

--
Robert Haas
EDB: http://www.enterprisedb.com

#7Robert Haas
robertmhaas@gmail.com
In reply to: Justin Pryzby (#2)
Re: adding 'zstd' as a compression algorithm

On Tue, Feb 15, 2022 at 1:55 PM Justin Pryzby <pryzby@telsasoft.com> wrote:

Regarding the patch:

I suppose it should support windows, and whatever patches use zstd should
update the install instructions. See 9ca40dcd4, 02a93e7ef, 4035cd5d4.

Thanks, this is helpful. To me, it looks like we need something
similar to 9ca40dcd4, plus something like the last hunk of 02a93e7ef.
The rest of 02a93e7ef and all of 4035cd5d4 seem like they address the
application of lz4 rather than just adding support for it, so I don't
see the need for those elements in this patch. Am I missing something?

--
Robert Haas
EDB: http://www.enterprisedb.com

#8Justin Pryzby
pryzby@telsasoft.com
In reply to: Robert Haas (#7)
Re: adding 'zstd' as a compression algorithm

On Tue, Feb 15, 2022 at 03:23:30PM -0500, Robert Haas wrote:

On Tue, Feb 15, 2022 at 1:55 PM Justin Pryzby <pryzby@telsasoft.com> wrote:

Regarding the patch:

I suppose it should support windows, and whatever patches use zstd should
update the install instructions. See 9ca40dcd4, 02a93e7ef, 4035cd5d4.

Thanks, this is helpful. To me, it looks like we need something
similar to 9ca40dcd4, plus something like the last hunk of 02a93e7ef.

If you mean a patch which only adds the configure options, I don't think 02a9
is applicable at all.

What I mean is that any patches which *use* the zstd support should update
those for completeness.

doc/src/sgml/config.sgml | 14 +++++++++-----
doc/src/sgml/install-windows.sgml | 2 +-
doc/src/sgml/installation.sgml | 5 +++--

Show quoted text

The rest of 02a93e7ef and all of 4035cd5d4 seem like they address the
application of lz4 rather than just adding support for it, so I don't
see the need for those elements in this patch. Am I missing something?

#9Andres Freund
andres@anarazel.de
In reply to: Robert Haas (#6)
Re: adding 'zstd' as a compression algorithm

Hi,

On 2022-02-15 15:09:37 -0500, Robert Haas wrote:

On Tue, Feb 15, 2022 at 2:54 PM Andres Freund <andres@anarazel.de> wrote:

There's a difference between downloading a source tarball of 1.5x the size,
and 3x the decompression cost (made up numbers), because the cost of either is
not a relevant factor. I think basebackups are a different story.

To be clear, I'm not saying that people shouldn't choose to adopt the
new stuff. I'm just saying that many of them won't, for various
reasons, including inertia. There may come a point when we want to
make a push to remove obsolete stuff, but I think what is far more
important right now is making the new stuff available.

I think well before removing stuff we should default to decent compression
algorithms. E.g. -Z/--compress should probably not continue to use gzip if
better things are available.

Greetings,

Andres Freund

In reply to: Robert Haas (#5)
Re: adding 'zstd' as a compression algorithm

On Tue, Feb 15, 2022 at 12:00 PM Robert Haas <robertmhaas@gmail.com> wrote:

I'm not sure I completely follow this. There are cases where we use
compression algorithms for internal purposes, and we can change the
defaults without users knowing or caring. For example, we could decide
that LZ4-compressing FPIs in WAL records is a sensible default and
just start doing it, and users need not care. But backups are
different, because when you pg_basebackup -Ft, you get .tar or .tar.gz
or .tar.lz4 files which we don't give you tools to extract.

What I meant is that you're buying into an ecosystem by choosing to
use a tool like pgBackrest. That might not be the right thing to do
universally, but it comes pretty close. You as a user are largely
deferring to the maintainers and their choice of defaults. Not
entirely, of course, but to a significant degree, especially in
matters like this. There aren't that many dimensions to the problem
once the question of compatibility is settled (it's settled here
because you've already bought into a tool that requires the library as
standard, say).

A lot of things are *not* like that, but ISTM that backup compression
really is -- we have the luxury of a constrained problem space, for
once. There aren't that many metrics to consider, because it must be
lossless compression in any case, and because the requirements are
relatively homogenous. The truly important thing (once compatibility
is accounted for) is to use something basically reasonable, with no
noticeable weaknesses relative to any of the alternatives.

I pretty much agree with all of that. Nevertheless, there's a lot of
pglz-compressed data that's already on a disk someplace which people
are not necessarily keen to rewrite, and that will probably remain
true for the foreseable future. If we change the default to something
that's not pglz, and then wait 10 years, we MIGHT be able to get rid
of it without pushback. But I wouldn't be surprised to learn that even
then there are a lot of people who have just pg_upgrade'd the same
database over and over again. And honestly I think that's fine.

I think that it's fine too. It's unlikely that anybody is going to go
to any trouble to get better compression, certainly, but we should
give them every opportunity to use better alternatives. Ideally
without anybody having to even think about it.

--
Peter Geoghegan

#11David Steele
david@pgmasters.net
In reply to: Peter Geoghegan (#10)
Re: adding 'zstd' as a compression algorithm

On 2/15/22 16:10, Peter Geoghegan wrote:

On Tue, Feb 15, 2022 at 12:00 PM Robert Haas <robertmhaas@gmail.com> wrote:

I'm not sure I completely follow this. There are cases where we use
compression algorithms for internal purposes, and we can change the
defaults without users knowing or caring. For example, we could decide
that LZ4-compressing FPIs in WAL records is a sensible default and
just start doing it, and users need not care.

I'm not sure that this is entirely true. lz4 is available on most
non-EOL distros but you don't need to look to far to find a distro that
does not have it. So, choosing lz4 by default could impact binary
portability. Of course, there are lots of other factors that impact
binary portability, e.g. collations, but this would add one more.

This is even more true for zstd since it is not as ubiquitous as lz4. In
fact, it is not even included with base RHEL8. You need to install EPEL
to get zstd.

But backups are
different, because when you pg_basebackup -Ft, you get .tar or .tar.gz
or .tar.lz4 files which we don't give you tools to extract.

What I meant is that you're buying into an ecosystem by choosing to
use a tool like pgBackrest. That might not be the right thing to do
universally, but it comes pretty close. You as a user are largely
deferring to the maintainers and their choice of defaults. Not
entirely, of course, but to a significant degree, especially in
matters like this. There aren't that many dimensions to the problem
once the question of compatibility is settled (it's settled here
because you've already bought into a tool that requires the library as
standard, say).

As much as we would like to, we have not changed the default compression
for pgBackRest. lz4 and zstd are both optional at build time since they
are not always available (though lz4 is getting close). So we have left
gzip as the default, though a lot of that has to do with maintaining
compatibility with prior versions of pgBackRest.

Having said all that, I'm absolutely in favor of adding zstd to Postgres
as an optional compression format.

Regards,
David

#12Michael Paquier
michael@paquier.xyz
In reply to: Andres Freund (#9)
Re: adding 'zstd' as a compression algorithm

On Tue, Feb 15, 2022 at 01:42:55PM -0800, Andres Freund wrote:

I think well before removing stuff we should default to decent compression
algorithms. E.g. -Z/--compress should probably not continue to use gzip if
better things are available.

Which one of lz4 or zstd would it be though? LZ4 is light on CPU, and
compresses less than zstd which is more CPU intensive with more
compression, so the choice does not look that straight-forward to me.
Both of them are better than zlib, still they are less available on
the platforms Postgres needs to provide support for. It does not seem
really intuitive to me to have different defaults depending on the
build options provided, either :/

Saying that, I'd like to think that zstd would still be a better
default choice than LZ4.
--
Michael

#13Robert Haas
robertmhaas@gmail.com
In reply to: David Steele (#11)
Re: adding 'zstd' as a compression algorithm

On Tue, Feb 15, 2022 at 7:09 PM David Steele <david@pgmasters.net> wrote:

I'm not sure that this is entirely true. lz4 is available on most
non-EOL distros but you don't need to look to far to find a distro that
does not have it. So, choosing lz4 by default could impact binary
portability. Of course, there are lots of other factors that impact
binary portability, e.g. collations, but this would add one more.

This is even more true for zstd since it is not as ubiquitous as lz4. In
fact, it is not even included with base RHEL8. You need to install EPEL
to get zstd.

Yeah, I agree. One thing I was thinking about but didn't include in
the previous email is that if we did choose to make something like LZ4
the default, it would presumably only be the default on builds that
include LZ4 support. Other builds would need to use something else,
unless we just chose to make LZ4 a hard requirement, which would be
bolder than we usually are. And that has the consequence that you
mention. It's something we should consider as we think about changing
defaults.

Having said all that, I'm absolutely in favor of adding zstd to Postgres
as an optional compression format.

Cool.

--
Robert Haas
EDB: http://www.enterprisedb.com

#14Tom Lane
tgl@sss.pgh.pa.us
In reply to: Robert Haas (#13)
Re: adding 'zstd' as a compression algorithm

Robert Haas <robertmhaas@gmail.com> writes:

On Tue, Feb 15, 2022 at 7:09 PM David Steele <david@pgmasters.net> wrote:

This is even more true for zstd since it is not as ubiquitous as lz4. In
fact, it is not even included with base RHEL8. You need to install EPEL
to get zstd.

Yeah, I agree. One thing I was thinking about but didn't include in
the previous email is that if we did choose to make something like LZ4
the default, it would presumably only be the default on builds that
include LZ4 support. Other builds would need to use something else,
unless we just chose to make LZ4 a hard requirement, which would be
bolder than we usually are. And that has the consequence that you
mention. It's something we should consider as we think about changing
defaults.

Yeah. I'm +1 on adding zstd as an option, but I think we need to
move *very* slowly on changing any defaults for user-accessible
data (like backup files). Maybe we have a bit more flexibility
for TOAST, not sure.

regards, tom lane

#15Andres Freund
andres@anarazel.de
In reply to: Michael Paquier (#12)
Re: adding 'zstd' as a compression algorithm

Hi,

On 2022-02-16 09:58:44 +0900, Michael Paquier wrote:

On Tue, Feb 15, 2022 at 01:42:55PM -0800, Andres Freund wrote:

I think well before removing stuff we should default to decent compression
algorithms. E.g. -Z/--compress should probably not continue to use gzip if
better things are available.

Which one of lz4 or zstd would it be though? LZ4 is light on CPU, and
compresses less than zstd which is more CPU intensive with more
compression, so the choice does not look that straight-forward to me.

For backups it's pretty obviously zstd imo. At the lower levels it achieves
considerably higher compression ratios while still being vastly faster than
gzip. Without even using the threaded compression support the library has.

For something like wal_compression it'd be a harder question.

Greetings,

Andres Freund

#16Michael Paquier
michael@paquier.xyz
In reply to: Justin Pryzby (#8)
Re: adding 'zstd' as a compression algorithm

On Tue, Feb 15, 2022 at 02:33:32PM -0600, Justin Pryzby wrote:

What I mean is that any patches which *use* the zstd support should update
those for completeness.

doc/src/sgml/config.sgml | 14 +++++++++-----
doc/src/sgml/install-windows.sgml | 2 +-
doc/src/sgml/installation.sgml | 5 +++--

Yes, the patch misses the fact that each ./configure switch is
documented. For consistency, I think that you should also add that in
the MSVC scripts in the first version. It is really straight-forward
to do so, and it should be just a matter of grepping for the code
paths of lz4, then adjust things for zstd.
--
Michael

#17Andres Freund
andres@anarazel.de
In reply to: Tom Lane (#14)
Re: adding 'zstd' as a compression algorithm

Hi,m

On 2022-02-15 20:56:15 -0500, Tom Lane wrote:

Maybe we have a bit more flexibility for TOAST, not sure.

It'd be nice to at least add it as an option for initdb. Afaics there's no way
to change the default at that point. initdb itself is measurably
faster. Although sadly it's a bigger difference for optimized builds.

I think it matter a bit even after initdb, pg_rewrite is most of the
compressed data and some of the views are regularly used...

- Andres

#18Robert Haas
robertmhaas@gmail.com
In reply to: Michael Paquier (#16)
1 attachment(s)
Re: adding 'zstd' as a compression algorithm

On Tue, Feb 15, 2022 at 9:33 PM Michael Paquier <michael@paquier.xyz> wrote:

On Tue, Feb 15, 2022 at 02:33:32PM -0600, Justin Pryzby wrote:

What I mean is that any patches which *use* the zstd support should update
those for completeness.

doc/src/sgml/config.sgml | 14 +++++++++-----
doc/src/sgml/install-windows.sgml | 2 +-
doc/src/sgml/installation.sgml | 5 +++--

Yes, the patch misses the fact that each ./configure switch is
documented. For consistency, I think that you should also add that in
the MSVC scripts in the first version. It is really straight-forward
to do so, and it should be just a matter of grepping for the code
paths of lz4, then adjust things for zstd.

Makes sense. Here's an attempt by me to do that.

--
Robert Haas
EDB: http://www.enterprisedb.com

Attachments:

v2-0001-Add-support-for-building-with-ZSTD.patchapplication/octet-stream; name=v2-0001-Add-support-for-building-with-ZSTD.patchDownload
From c46df1ebef3000227251a7870d62fa49e1822e2c Mon Sep 17 00:00:00 2001
From: Robert Haas <rhaas@postgresql.org>
Date: Wed, 16 Feb 2022 10:36:36 -0500
Subject: [PATCH v2] Add support for building with ZSTD.

This commit doesn't actually add anything that uses ZSTD; that will be
done separately. It just puts the basic infrastructure into place.

Jeevan Ladhe and Robert Haas
---
 configure                         | 271 ++++++++++++++++++++++++++++++
 configure.ac                      |  33 ++++
 doc/src/sgml/install-windows.sgml |   9 +
 doc/src/sgml/installation.sgml    |   9 +
 src/Makefile.global.in            |   1 +
 src/include/pg_config.h.in        |   9 +
 src/tools/msvc/Solution.pm        |  12 ++
 src/tools/msvc/config_default.pl  |   1 +
 8 files changed, 345 insertions(+)

diff --git a/configure b/configure
index 9305555658..f07f689f1a 100755
--- a/configure
+++ b/configure
@@ -650,6 +650,7 @@ CFLAGS_ARMV8_CRC32C
 CFLAGS_SSE42
 have_win32_dbghelp
 LIBOBJS
+ZSTD
 LZ4
 UUID_LIBS
 LDAP_LIBS_BE
@@ -700,6 +701,9 @@ with_gnu_ld
 LD
 LDFLAGS_SL
 LDFLAGS_EX
+ZSTD_LIBS
+ZSTD_CFLAGS
+with_zstd
 LZ4_LIBS
 LZ4_CFLAGS
 with_lz4
@@ -869,6 +873,7 @@ with_libxslt
 with_system_tzdata
 with_zlib
 with_lz4
+with_zstd
 with_gnu_ld
 with_ssl
 with_openssl
@@ -898,6 +903,8 @@ XML2_CFLAGS
 XML2_LIBS
 LZ4_CFLAGS
 LZ4_LIBS
+ZSTD_CFLAGS
+ZSTD_LIBS
 LDFLAGS_EX
 LDFLAGS_SL
 PERL
@@ -1577,6 +1584,7 @@ Optional Packages:
                           use system time zone data in DIR
   --without-zlib          do not use Zlib
   --with-lz4              build with LZ4 support
+  --with-zstd             build with ZSTD support
   --with-gnu-ld           assume the C compiler uses GNU ld [default=no]
   --with-ssl=LIB          use LIB for SSL/TLS support (openssl)
   --with-openssl          obsolete spelling of --with-ssl=openssl
@@ -1606,6 +1614,8 @@ Some influential environment variables:
   XML2_LIBS   linker flags for XML2, overriding pkg-config
   LZ4_CFLAGS  C compiler flags for LZ4, overriding pkg-config
   LZ4_LIBS    linker flags for LZ4, overriding pkg-config
+  ZSTD_CFLAGS C compiler flags for ZSTD, overriding pkg-config
+  ZSTD_LIBS   linker flags for ZSTD, overriding pkg-config
   LDFLAGS_EX  extra linker flags for linking executables only
   LDFLAGS_SL  extra linker flags for linking shared libraries only
   PERL        Perl program
@@ -9034,6 +9044,146 @@ fi
   done
 fi
 
+#
+# ZSTD
+#
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with ZSTD support" >&5
+$as_echo_n "checking whether to build with ZSTD support... " >&6; }
+
+
+
+# Check whether --with-zstd was given.
+if test "${with_zstd+set}" = set; then :
+  withval=$with_zstd;
+  case $withval in
+    yes)
+
+$as_echo "#define USE_ZSTD 1" >>confdefs.h
+
+      ;;
+    no)
+      :
+      ;;
+    *)
+      as_fn_error $? "no argument expected for --with-zstd option" "$LINENO" 5
+      ;;
+  esac
+
+else
+  with_zstd=no
+
+fi
+
+
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_zstd" >&5
+$as_echo "$with_zstd" >&6; }
+
+
+if test "$with_zstd" = yes; then
+
+pkg_failed=no
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for libzstd" >&5
+$as_echo_n "checking for libzstd... " >&6; }
+
+if test -n "$ZSTD_CFLAGS"; then
+    pkg_cv_ZSTD_CFLAGS="$ZSTD_CFLAGS"
+ elif test -n "$PKG_CONFIG"; then
+    if test -n "$PKG_CONFIG" && \
+    { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"libzstd\""; } >&5
+  ($PKG_CONFIG --exists --print-errors "libzstd") 2>&5
+  ac_status=$?
+  $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+  test $ac_status = 0; }; then
+  pkg_cv_ZSTD_CFLAGS=`$PKG_CONFIG --cflags "libzstd" 2>/dev/null`
+		      test "x$?" != "x0" && pkg_failed=yes
+else
+  pkg_failed=yes
+fi
+ else
+    pkg_failed=untried
+fi
+if test -n "$ZSTD_LIBS"; then
+    pkg_cv_ZSTD_LIBS="$ZSTD_LIBS"
+ elif test -n "$PKG_CONFIG"; then
+    if test -n "$PKG_CONFIG" && \
+    { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"libzstd\""; } >&5
+  ($PKG_CONFIG --exists --print-errors "libzstd") 2>&5
+  ac_status=$?
+  $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+  test $ac_status = 0; }; then
+  pkg_cv_ZSTD_LIBS=`$PKG_CONFIG --libs "libzstd" 2>/dev/null`
+		      test "x$?" != "x0" && pkg_failed=yes
+else
+  pkg_failed=yes
+fi
+ else
+    pkg_failed=untried
+fi
+
+
+
+if test $pkg_failed = yes; then
+        { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+
+if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then
+        _pkg_short_errors_supported=yes
+else
+        _pkg_short_errors_supported=no
+fi
+        if test $_pkg_short_errors_supported = yes; then
+	        ZSTD_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "libzstd" 2>&1`
+        else
+	        ZSTD_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "libzstd" 2>&1`
+        fi
+	# Put the nasty error message in config.log where it belongs
+	echo "$ZSTD_PKG_ERRORS" >&5
+
+	as_fn_error $? "Package requirements (libzstd) were not met:
+
+$ZSTD_PKG_ERRORS
+
+Consider adjusting the PKG_CONFIG_PATH environment variable if you
+installed software in a non-standard prefix.
+
+Alternatively, you may set the environment variables ZSTD_CFLAGS
+and ZSTD_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details." "$LINENO" 5
+elif test $pkg_failed = untried; then
+        { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+	{ { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
+$as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
+as_fn_error $? "The pkg-config script could not be found or is too old.  Make sure it
+is in your PATH or set the PKG_CONFIG environment variable to the full
+path to pkg-config.
+
+Alternatively, you may set the environment variables ZSTD_CFLAGS
+and ZSTD_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details.
+
+To get pkg-config, see <http://pkg-config.freedesktop.org/>.
+See \`config.log' for more details" "$LINENO" 5; }
+else
+	ZSTD_CFLAGS=$pkg_cv_ZSTD_CFLAGS
+	ZSTD_LIBS=$pkg_cv_ZSTD_LIBS
+        { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
+$as_echo "yes" >&6; }
+
+fi
+  # We only care about -I, -D, and -L switches;
+  # note that -lzstd will be added by AC_CHECK_LIB below.
+  for pgac_option in $ZSTD_CFLAGS; do
+    case $pgac_option in
+      -I*|-D*) CPPFLAGS="$CPPFLAGS $pgac_option";;
+    esac
+  done
+  for pgac_option in $ZSTD_LIBS; do
+    case $pgac_option in
+      -L*) LDFLAGS="$LDFLAGS $pgac_option";;
+    esac
+  done
+fi
 #
 # Assignments
 #
@@ -13130,6 +13280,56 @@ fi
 
 fi
 
+if test "$with_zstd" = yes ; then
+  { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ZSTD_compress in -lzstd" >&5
+$as_echo_n "checking for ZSTD_compress in -lzstd... " >&6; }
+if ${ac_cv_lib_zstd_ZSTD_compress+:} false; then :
+  $as_echo_n "(cached) " >&6
+else
+  ac_check_lib_save_LIBS=$LIBS
+LIBS="-lzstd  $LIBS"
+cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h.  */
+
+/* Override any GCC internal prototype to avoid an error.
+   Use char because int might match the return type of a GCC
+   builtin and then its argument prototype would still apply.  */
+#ifdef __cplusplus
+extern "C"
+#endif
+char ZSTD_compress ();
+int
+main ()
+{
+return ZSTD_compress ();
+  ;
+  return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+  ac_cv_lib_zstd_ZSTD_compress=yes
+else
+  ac_cv_lib_zstd_ZSTD_compress=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+    conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_zstd_ZSTD_compress" >&5
+$as_echo "$ac_cv_lib_zstd_ZSTD_compress" >&6; }
+if test "x$ac_cv_lib_zstd_ZSTD_compress" = xyes; then :
+  cat >>confdefs.h <<_ACEOF
+#define HAVE_LIBZSTD 1
+_ACEOF
+
+  LIBS="-lzstd $LIBS"
+
+else
+  as_fn_error $? "library 'zstd' is required for ZSTD support" "$LINENO" 5
+fi
+
+fi
+
 # Note: We can test for libldap_r only after we know PTHREAD_LIBS;
 # also, on AIX, we may need to have openssl in LIBS for this step.
 if test "$with_ldap" = yes ; then
@@ -13904,6 +14104,77 @@ done
 
 fi
 
+if test -z "$ZSTD"; then
+  for ac_prog in zstd
+do
+  # Extract the first word of "$ac_prog", so it can be a program name with args.
+set dummy $ac_prog; ac_word=$2
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
+$as_echo_n "checking for $ac_word... " >&6; }
+if ${ac_cv_path_ZSTD+:} false; then :
+  $as_echo_n "(cached) " >&6
+else
+  case $ZSTD in
+  [\\/]* | ?:[\\/]*)
+  ac_cv_path_ZSTD="$ZSTD" # Let the user override the test with a path.
+  ;;
+  *)
+  as_save_IFS=$IFS; IFS=$PATH_SEPARATOR
+for as_dir in $PATH
+do
+  IFS=$as_save_IFS
+  test -z "$as_dir" && as_dir=.
+    for ac_exec_ext in '' $ac_executable_extensions; do
+  if as_fn_executable_p "$as_dir/$ac_word$ac_exec_ext"; then
+    ac_cv_path_ZSTD="$as_dir/$ac_word$ac_exec_ext"
+    $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5
+    break 2
+  fi
+done
+  done
+IFS=$as_save_IFS
+
+  ;;
+esac
+fi
+ZSTD=$ac_cv_path_ZSTD
+if test -n "$ZSTD"; then
+  { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ZSTD" >&5
+$as_echo "$ZSTD" >&6; }
+else
+  { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+fi
+
+
+  test -n "$ZSTD" && break
+done
+
+else
+  # Report the value of ZSTD in configure's output in all cases.
+  { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ZSTD" >&5
+$as_echo_n "checking for ZSTD... " >&6; }
+  { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ZSTD" >&5
+$as_echo "$ZSTD" >&6; }
+fi
+
+if test "$with_zstd" = yes; then
+  for ac_header in zstd.h
+do :
+  ac_fn_c_check_header_mongrel "$LINENO" "zstd.h" "ac_cv_header_zstd_h" "$ac_includes_default"
+if test "x$ac_cv_header_zstd_h" = xyes; then :
+  cat >>confdefs.h <<_ACEOF
+#define HAVE_ZSTD_H 1
+_ACEOF
+
+else
+  as_fn_error $? "zstd.h header file is required for ZSTD" "$LINENO" 5
+fi
+
+done
+
+fi
+
 if test "$with_gssapi" = yes ; then
   for ac_header in gssapi/gssapi.h
 do :
diff --git a/configure.ac b/configure.ac
index 16167329fc..729b23fbea 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1056,6 +1056,30 @@ if test "$with_lz4" = yes; then
   done
 fi
 
+#
+# ZSTD
+#
+AC_MSG_CHECKING([whether to build with ZSTD support])
+PGAC_ARG_BOOL(with, zstd, no, [build with ZSTD support],
+              [AC_DEFINE([USE_ZSTD], 1, [Define to 1 to build with ZSTD support. (--with-zstd)])])
+AC_MSG_RESULT([$with_zstd])
+AC_SUBST(with_zstd)
+
+if test "$with_zstd" = yes; then
+  PKG_CHECK_MODULES(ZSTD, libzstd)
+  # We only care about -I, -D, and -L switches;
+  # note that -lzstd will be added by AC_CHECK_LIB below.
+  for pgac_option in $ZSTD_CFLAGS; do
+    case $pgac_option in
+      -I*|-D*) CPPFLAGS="$CPPFLAGS $pgac_option";;
+    esac
+  done
+  for pgac_option in $ZSTD_LIBS; do
+    case $pgac_option in
+      -L*) LDFLAGS="$LDFLAGS $pgac_option";;
+    esac
+  done
+fi
 #
 # Assignments
 #
@@ -1325,6 +1349,10 @@ if test "$with_lz4" = yes ; then
   AC_CHECK_LIB(lz4, LZ4_compress_default, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])])
 fi
 
+if test "$with_zstd" = yes ; then
+  AC_CHECK_LIB(zstd, ZSTD_compress, [], [AC_MSG_ERROR([library 'zstd' is required for ZSTD support])])
+fi
+
 # Note: We can test for libldap_r only after we know PTHREAD_LIBS;
 # also, on AIX, we may need to have openssl in LIBS for this step.
 if test "$with_ldap" = yes ; then
@@ -1490,6 +1518,11 @@ if test "$with_lz4" = yes; then
   AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])
 fi
 
+PGAC_PATH_PROGS(ZSTD, zstd)
+if test "$with_zstd" = yes; then
+  AC_CHECK_HEADERS(zstd.h, [], [AC_MSG_ERROR([zstd.h header file is required for ZSTD])])
+fi
+
 if test "$with_gssapi" = yes ; then
   AC_CHECK_HEADERS(gssapi/gssapi.h, [],
 	[AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])])
diff --git a/doc/src/sgml/install-windows.sgml b/doc/src/sgml/install-windows.sgml
index 30dd0c7f75..d2f63db3f2 100644
--- a/doc/src/sgml/install-windows.sgml
+++ b/doc/src/sgml/install-windows.sgml
@@ -307,6 +307,15 @@ $ENV{MSBFLAGS}="/m";
      </para></listitem>
     </varlistentry>
 
+    <varlistentry>
+     <term><productname>ZSTD</productname></term>
+     <listitem><para>
+      Required for supporting <productname>ZSTD</productname> compression
+      method. Binaries and source can be downloaded from
+      <ulink url="https://github.com/facebook/zstd/releases"></ulink>.
+     </para></listitem>
+    </varlistentry>
+
     <varlistentry>
      <term><productname>OpenSSL</productname></term>
      <listitem><para>
diff --git a/doc/src/sgml/installation.sgml b/doc/src/sgml/installation.sgml
index 655095f3b1..c6190f6955 100644
--- a/doc/src/sgml/installation.sgml
+++ b/doc/src/sgml/installation.sgml
@@ -989,6 +989,15 @@ build-postgresql:
        </listitem>
       </varlistentry>
 
+      <varlistentry>
+       <term><option>--with-zstd</option></term>
+       <listitem>
+        <para>
+         Build with <productname>ZSTD</productname> compression support.
+        </para>
+       </listitem>
+      </varlistentry>
+
       <varlistentry>
        <term><option>--with-ssl=<replaceable>LIBRARY</replaceable></option>
        <indexterm>
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 9dcd54fcbd..c980444233 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -351,6 +351,7 @@ XGETTEXT = @XGETTEXT@
 GZIP	= gzip
 BZIP2	= bzip2
 LZ4	= @LZ4@
+ZSTD	= @ZSTD@
 
 DOWNLOAD = wget -O $@ --no-use-server-timestamps
 #DOWNLOAD = curl -o $@
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 28a1f0e9f0..1912cf35de 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -352,6 +352,9 @@
 /* Define to 1 if you have the `z' library (-lz). */
 #undef HAVE_LIBZ
 
+/* Define to 1 if you have the `zstd' library (-lzstd). */
+#undef HAVE_LIBZSTD
+
 /* Define to 1 if you have the `link' function. */
 #undef HAVE_LINK
 
@@ -718,6 +721,9 @@
 /* Define to 1 if the assembler supports X86_64's POPCNTQ instruction. */
 #undef HAVE_X86_64_POPCNTQ
 
+/* Define to 1 if you have the <zstd.h> header file. */
+#undef HAVE_ZSTD_H
+
 /* Define to 1 if the system has the type `_Bool'. */
 #undef HAVE__BOOL
 
@@ -949,6 +955,9 @@
 /* Define to select Win32-style shared memory. */
 #undef USE_WIN32_SHARED_MEMORY
 
+/* Define to 1 to build with ZSTD support. (--with-zstd) */
+#undef USE_ZSTD
+
 /* Define to 1 if `wcstombs_l' requires <xlocale.h>. */
 #undef WCSTOMBS_L_IN_XLOCALE
 
diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm
index e6f20679dc..087acfbaa1 100644
--- a/src/tools/msvc/Solution.pm
+++ b/src/tools/msvc/Solution.pm
@@ -539,6 +539,12 @@ sub GenerateFiles
 		$define{HAVE_LZ4_H}  = 1;
 		$define{USE_LZ4}     = 1;
 	}
+	if ($self->{options}->{zstd})
+	{
+		$define{HAVE_LIBZSTD} = 1;
+		$define{HAVE_ZSTD_H}  = 1;
+		$define{USE_ZSTD}     = 1;
+	}
 	if ($self->{options}->{openssl})
 	{
 		$define{USE_OPENSSL} = 1;
@@ -1081,6 +1087,11 @@ sub AddProject
 		$proj->AddIncludeDir($self->{options}->{lz4} . '\include');
 		$proj->AddLibrary($self->{options}->{lz4} . '\lib\liblz4.lib');
 	}
+	if ($self->{options}->{zstd})
+	{
+		$proj->AddIncludeDir($self->{options}->{zstd} . '\include');
+		$proj->AddLibrary($self->{options}->{zstd} . '\lib\libzstd.lib');
+	}
 	if ($self->{options}->{uuid})
 	{
 		$proj->AddIncludeDir($self->{options}->{uuid} . '\include');
@@ -1193,6 +1204,7 @@ sub GetFakeConfigure
 	$cfg .= ' --with-libxml'        if ($self->{options}->{xml});
 	$cfg .= ' --with-libxslt'       if ($self->{options}->{xslt});
 	$cfg .= ' --with-lz4'           if ($self->{options}->{lz4});
+	$cfg .= ' --with-zstd'          if ($self->{options}->{zstd});
 	$cfg .= ' --with-gssapi'        if ($self->{options}->{gss});
 	$cfg .= ' --with-icu'           if ($self->{options}->{icu});
 	$cfg .= ' --with-tcl'           if ($self->{options}->{tcl});
diff --git a/src/tools/msvc/config_default.pl b/src/tools/msvc/config_default.pl
index 7a9b00be72..186849a09a 100644
--- a/src/tools/msvc/config_default.pl
+++ b/src/tools/msvc/config_default.pl
@@ -15,6 +15,7 @@ our $config = {
 	gss       => undef,    # --with-gssapi=<path>
 	icu       => undef,    # --with-icu=<path>
 	lz4       => undef,    # --with-lz4=<path>
+	zstd      => undef,    # --with-zstd=<path>
 	nls       => undef,    # --enable-nls=<path>
 	tap_tests => undef,    # --enable-tap-tests
 	tcl       => undef,    # --with-tcl=<path>
-- 
2.24.3 (Apple Git-128)

#19Justin Pryzby
pryzby@telsasoft.com
In reply to: Andres Freund (#4)
Re: adding 'zstd' as a compression algorithm (initdb/lz4)

On Tue, Feb 15, 2022 at 11:54:10AM -0800, Andres Freund wrote:

Isn't it an incontrovertible fact that LZ4 is superior to pglz in
every way? LZ4 is pretty much its successor. And so it seems totally
fine to assume that users will always want to use the clearly better
option, and that that option will be generally available going
forward. TOAST compression is applied selectively already, based on
various obscure implementation details, some of which are quite
arbitrary.

Yea, we should really default to lz4 in initdb when available.

This patch intends to implement that. I have no particular interest in this,
but if anyone wants, I will add it to the next CF (or the one after that).

commit 2a3c5950e625ccfaebc49bbf71b8db16dc143cd2
Author: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Tue Feb 15 19:14:33 2022 -0600

initdb: default_toast_compression=lz4 if available

TODO: consider same for wal_compression

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index fc63172efde..f9cd2ef7229 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8537,7 +8537,9 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
         The supported compression methods are <literal>pglz</literal> and
         (if <productname>PostgreSQL</productname> was compiled with
         <option>--with-lz4</option>) <literal>lz4</literal>.
-        The default is <literal>pglz</literal>.
+        The default is <literal>lz4</literal> if available at the time 
+        <productname>PostgreSQL</productname> was compiled, otherwise
+        <literal>pglz</literal>.
        </para>
       </listitem>
      </varlistentry>
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index f505413a7f9..6b6f6efaba1 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -4726,7 +4726,11 @@ static struct config_enum ConfigureNamesEnum[] =
 			NULL
 		},
 		&default_toast_compression,
+#ifdef USE_LZ4
+		TOAST_LZ4_COMPRESSION,
+#else
 		TOAST_PGLZ_COMPRESSION,
+#endif
 		default_toast_compression_options,
 		NULL, NULL, NULL
 	},
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index d78e8e67b8d..bb7c57e00fa 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -1185,6 +1185,12 @@ setup_config(void)
 							  "#update_process_title = off");
 #endif
+#ifdef USE_LZ4
+	conflines = replace_token(conflines,
+							  "#default_toast_compression = 'pglz'",
+							  "#default_toast_compression = 'lz4'");
+#endif
+
 	/*
 	 * Change password_encryption setting to md5 if md5 was chosen as an
 	 * authentication method, unless scram-sha-256 was also chosen.
#20Michael Paquier
michael@paquier.xyz
In reply to: Andres Freund (#15)
Re: adding 'zstd' as a compression algorithm

On Tue, Feb 15, 2022 at 06:24:13PM -0800, Andres Freund wrote:

For backups it's pretty obviously zstd imo. At the lower levels it achieves
considerably higher compression ratios while still being vastly faster than
gzip. Without even using the threaded compression support the library has.

Noted.

For something like wal_compression it'd be a harder question.

FWIW, I have done some measurements for wal_compression with zstd, as
of:
/messages/by-id/YMmlvyVyAFlxZ+/H@paquier.xyz

The result is not surprising, a bit more CPU for zstd with more
compression compared to LZ4, both outclassing easily zlib. I am not
sure which one would be more adapted as default as FPI patterns depend
on the workload, for one, and this is just one corner case.
--
Michael

#21Michael Paquier
michael@paquier.xyz
In reply to: Robert Haas (#18)
1 attachment(s)
Re: adding 'zstd' as a compression algorithm

On Wed, Feb 16, 2022 at 10:40:01AM -0500, Robert Haas wrote:

On Tue, Feb 15, 2022 at 9:33 PM Michael Paquier <michael@paquier.xyz> wrote:

Yes, the patch misses the fact that each ./configure switch is
documented. For consistency, I think that you should also add that in
the MSVC scripts in the first version. It is really straight-forward
to do so, and it should be just a matter of grepping for the code
paths of lz4, then adjust things for zstd.

Makes sense. Here's an attempt by me to do that.

Thanks. This looks pretty much right, except for two things that I
have taken the freedom to fix as of the v3 attached.

%define needs to include HAVE_LIBZSTD, HAVE_ZSTD_H and USE_ZSTD, so
this version fails the sanity check between pg_config.h.in and the
MSVC scripts checking that all flags exist.

@@ -351,6 +351,7 @@ XGETTEXT = @XGETTEXT@
GZIP = gzip
BZIP2 = bzip2
LZ4 = @LZ4@
+ZSTD = @ZSTD@
A similar refresh is needed in vcregress.pl.

+       $proj->AddLibrary($self->{options}->{zstd} . '\lib\libzstd.lib');
The upstream code is also using this library name, so that should be
fine.
--
Michael

Attachments:

v3-0001-Add-support-for-building-with-ZSTD.patchtext/x-diff; charset=us-asciiDownload
From 54485782e931c9536de2fdf7dbdfd0bb5c0c632d Mon Sep 17 00:00:00 2001
From: Robert Haas <rhaas@postgresql.org>
Date: Wed, 16 Feb 2022 10:36:36 -0500
Subject: [PATCH v3] Add support for building with ZSTD.

This commit doesn't actually add anything that uses ZSTD; that will be
done separately. It just puts the basic infrastructure into place.

Jeevan Ladhe and Robert Haas
---
 src/include/pg_config.h.in        |   9 +
 doc/src/sgml/install-windows.sgml |   9 +
 doc/src/sgml/installation.sgml    |   9 +
 configure                         | 271 ++++++++++++++++++++++++++++++
 configure.ac                      |  33 ++++
 src/Makefile.global.in            |   1 +
 src/tools/msvc/Solution.pm        |  15 ++
 src/tools/msvc/config_default.pl  |   1 +
 src/tools/msvc/vcregress.pl       |   1 +
 9 files changed, 349 insertions(+)

diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 28a1f0e9f0..1912cf35de 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -352,6 +352,9 @@
 /* Define to 1 if you have the `z' library (-lz). */
 #undef HAVE_LIBZ
 
+/* Define to 1 if you have the `zstd' library (-lzstd). */
+#undef HAVE_LIBZSTD
+
 /* Define to 1 if you have the `link' function. */
 #undef HAVE_LINK
 
@@ -718,6 +721,9 @@
 /* Define to 1 if the assembler supports X86_64's POPCNTQ instruction. */
 #undef HAVE_X86_64_POPCNTQ
 
+/* Define to 1 if you have the <zstd.h> header file. */
+#undef HAVE_ZSTD_H
+
 /* Define to 1 if the system has the type `_Bool'. */
 #undef HAVE__BOOL
 
@@ -949,6 +955,9 @@
 /* Define to select Win32-style shared memory. */
 #undef USE_WIN32_SHARED_MEMORY
 
+/* Define to 1 to build with ZSTD support. (--with-zstd) */
+#undef USE_ZSTD
+
 /* Define to 1 if `wcstombs_l' requires <xlocale.h>. */
 #undef WCSTOMBS_L_IN_XLOCALE
 
diff --git a/doc/src/sgml/install-windows.sgml b/doc/src/sgml/install-windows.sgml
index 30dd0c7f75..d2f63db3f2 100644
--- a/doc/src/sgml/install-windows.sgml
+++ b/doc/src/sgml/install-windows.sgml
@@ -307,6 +307,15 @@ $ENV{MSBFLAGS}="/m";
      </para></listitem>
     </varlistentry>
 
+    <varlistentry>
+     <term><productname>ZSTD</productname></term>
+     <listitem><para>
+      Required for supporting <productname>ZSTD</productname> compression
+      method. Binaries and source can be downloaded from
+      <ulink url="https://github.com/facebook/zstd/releases"></ulink>.
+     </para></listitem>
+    </varlistentry>
+
     <varlistentry>
      <term><productname>OpenSSL</productname></term>
      <listitem><para>
diff --git a/doc/src/sgml/installation.sgml b/doc/src/sgml/installation.sgml
index 655095f3b1..c6190f6955 100644
--- a/doc/src/sgml/installation.sgml
+++ b/doc/src/sgml/installation.sgml
@@ -989,6 +989,15 @@ build-postgresql:
        </listitem>
       </varlistentry>
 
+      <varlistentry>
+       <term><option>--with-zstd</option></term>
+       <listitem>
+        <para>
+         Build with <productname>ZSTD</productname> compression support.
+        </para>
+       </listitem>
+      </varlistentry>
+
       <varlistentry>
        <term><option>--with-ssl=<replaceable>LIBRARY</replaceable></option>
        <indexterm>
diff --git a/configure b/configure
index 9305555658..f07f689f1a 100755
--- a/configure
+++ b/configure
@@ -650,6 +650,7 @@ CFLAGS_ARMV8_CRC32C
 CFLAGS_SSE42
 have_win32_dbghelp
 LIBOBJS
+ZSTD
 LZ4
 UUID_LIBS
 LDAP_LIBS_BE
@@ -700,6 +701,9 @@ with_gnu_ld
 LD
 LDFLAGS_SL
 LDFLAGS_EX
+ZSTD_LIBS
+ZSTD_CFLAGS
+with_zstd
 LZ4_LIBS
 LZ4_CFLAGS
 with_lz4
@@ -869,6 +873,7 @@ with_libxslt
 with_system_tzdata
 with_zlib
 with_lz4
+with_zstd
 with_gnu_ld
 with_ssl
 with_openssl
@@ -898,6 +903,8 @@ XML2_CFLAGS
 XML2_LIBS
 LZ4_CFLAGS
 LZ4_LIBS
+ZSTD_CFLAGS
+ZSTD_LIBS
 LDFLAGS_EX
 LDFLAGS_SL
 PERL
@@ -1577,6 +1584,7 @@ Optional Packages:
                           use system time zone data in DIR
   --without-zlib          do not use Zlib
   --with-lz4              build with LZ4 support
+  --with-zstd             build with ZSTD support
   --with-gnu-ld           assume the C compiler uses GNU ld [default=no]
   --with-ssl=LIB          use LIB for SSL/TLS support (openssl)
   --with-openssl          obsolete spelling of --with-ssl=openssl
@@ -1606,6 +1614,8 @@ Some influential environment variables:
   XML2_LIBS   linker flags for XML2, overriding pkg-config
   LZ4_CFLAGS  C compiler flags for LZ4, overriding pkg-config
   LZ4_LIBS    linker flags for LZ4, overriding pkg-config
+  ZSTD_CFLAGS C compiler flags for ZSTD, overriding pkg-config
+  ZSTD_LIBS   linker flags for ZSTD, overriding pkg-config
   LDFLAGS_EX  extra linker flags for linking executables only
   LDFLAGS_SL  extra linker flags for linking shared libraries only
   PERL        Perl program
@@ -9034,6 +9044,146 @@ fi
   done
 fi
 
+#
+# ZSTD
+#
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with ZSTD support" >&5
+$as_echo_n "checking whether to build with ZSTD support... " >&6; }
+
+
+
+# Check whether --with-zstd was given.
+if test "${with_zstd+set}" = set; then :
+  withval=$with_zstd;
+  case $withval in
+    yes)
+
+$as_echo "#define USE_ZSTD 1" >>confdefs.h
+
+      ;;
+    no)
+      :
+      ;;
+    *)
+      as_fn_error $? "no argument expected for --with-zstd option" "$LINENO" 5
+      ;;
+  esac
+
+else
+  with_zstd=no
+
+fi
+
+
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_zstd" >&5
+$as_echo "$with_zstd" >&6; }
+
+
+if test "$with_zstd" = yes; then
+
+pkg_failed=no
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for libzstd" >&5
+$as_echo_n "checking for libzstd... " >&6; }
+
+if test -n "$ZSTD_CFLAGS"; then
+    pkg_cv_ZSTD_CFLAGS="$ZSTD_CFLAGS"
+ elif test -n "$PKG_CONFIG"; then
+    if test -n "$PKG_CONFIG" && \
+    { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"libzstd\""; } >&5
+  ($PKG_CONFIG --exists --print-errors "libzstd") 2>&5
+  ac_status=$?
+  $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+  test $ac_status = 0; }; then
+  pkg_cv_ZSTD_CFLAGS=`$PKG_CONFIG --cflags "libzstd" 2>/dev/null`
+		      test "x$?" != "x0" && pkg_failed=yes
+else
+  pkg_failed=yes
+fi
+ else
+    pkg_failed=untried
+fi
+if test -n "$ZSTD_LIBS"; then
+    pkg_cv_ZSTD_LIBS="$ZSTD_LIBS"
+ elif test -n "$PKG_CONFIG"; then
+    if test -n "$PKG_CONFIG" && \
+    { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"libzstd\""; } >&5
+  ($PKG_CONFIG --exists --print-errors "libzstd") 2>&5
+  ac_status=$?
+  $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+  test $ac_status = 0; }; then
+  pkg_cv_ZSTD_LIBS=`$PKG_CONFIG --libs "libzstd" 2>/dev/null`
+		      test "x$?" != "x0" && pkg_failed=yes
+else
+  pkg_failed=yes
+fi
+ else
+    pkg_failed=untried
+fi
+
+
+
+if test $pkg_failed = yes; then
+        { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+
+if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then
+        _pkg_short_errors_supported=yes
+else
+        _pkg_short_errors_supported=no
+fi
+        if test $_pkg_short_errors_supported = yes; then
+	        ZSTD_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "libzstd" 2>&1`
+        else
+	        ZSTD_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "libzstd" 2>&1`
+        fi
+	# Put the nasty error message in config.log where it belongs
+	echo "$ZSTD_PKG_ERRORS" >&5
+
+	as_fn_error $? "Package requirements (libzstd) were not met:
+
+$ZSTD_PKG_ERRORS
+
+Consider adjusting the PKG_CONFIG_PATH environment variable if you
+installed software in a non-standard prefix.
+
+Alternatively, you may set the environment variables ZSTD_CFLAGS
+and ZSTD_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details." "$LINENO" 5
+elif test $pkg_failed = untried; then
+        { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+	{ { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
+$as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
+as_fn_error $? "The pkg-config script could not be found or is too old.  Make sure it
+is in your PATH or set the PKG_CONFIG environment variable to the full
+path to pkg-config.
+
+Alternatively, you may set the environment variables ZSTD_CFLAGS
+and ZSTD_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details.
+
+To get pkg-config, see <http://pkg-config.freedesktop.org/>.
+See \`config.log' for more details" "$LINENO" 5; }
+else
+	ZSTD_CFLAGS=$pkg_cv_ZSTD_CFLAGS
+	ZSTD_LIBS=$pkg_cv_ZSTD_LIBS
+        { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
+$as_echo "yes" >&6; }
+
+fi
+  # We only care about -I, -D, and -L switches;
+  # note that -lzstd will be added by AC_CHECK_LIB below.
+  for pgac_option in $ZSTD_CFLAGS; do
+    case $pgac_option in
+      -I*|-D*) CPPFLAGS="$CPPFLAGS $pgac_option";;
+    esac
+  done
+  for pgac_option in $ZSTD_LIBS; do
+    case $pgac_option in
+      -L*) LDFLAGS="$LDFLAGS $pgac_option";;
+    esac
+  done
+fi
 #
 # Assignments
 #
@@ -13130,6 +13280,56 @@ fi
 
 fi
 
+if test "$with_zstd" = yes ; then
+  { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ZSTD_compress in -lzstd" >&5
+$as_echo_n "checking for ZSTD_compress in -lzstd... " >&6; }
+if ${ac_cv_lib_zstd_ZSTD_compress+:} false; then :
+  $as_echo_n "(cached) " >&6
+else
+  ac_check_lib_save_LIBS=$LIBS
+LIBS="-lzstd  $LIBS"
+cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h.  */
+
+/* Override any GCC internal prototype to avoid an error.
+   Use char because int might match the return type of a GCC
+   builtin and then its argument prototype would still apply.  */
+#ifdef __cplusplus
+extern "C"
+#endif
+char ZSTD_compress ();
+int
+main ()
+{
+return ZSTD_compress ();
+  ;
+  return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+  ac_cv_lib_zstd_ZSTD_compress=yes
+else
+  ac_cv_lib_zstd_ZSTD_compress=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+    conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_zstd_ZSTD_compress" >&5
+$as_echo "$ac_cv_lib_zstd_ZSTD_compress" >&6; }
+if test "x$ac_cv_lib_zstd_ZSTD_compress" = xyes; then :
+  cat >>confdefs.h <<_ACEOF
+#define HAVE_LIBZSTD 1
+_ACEOF
+
+  LIBS="-lzstd $LIBS"
+
+else
+  as_fn_error $? "library 'zstd' is required for ZSTD support" "$LINENO" 5
+fi
+
+fi
+
 # Note: We can test for libldap_r only after we know PTHREAD_LIBS;
 # also, on AIX, we may need to have openssl in LIBS for this step.
 if test "$with_ldap" = yes ; then
@@ -13904,6 +14104,77 @@ done
 
 fi
 
+if test -z "$ZSTD"; then
+  for ac_prog in zstd
+do
+  # Extract the first word of "$ac_prog", so it can be a program name with args.
+set dummy $ac_prog; ac_word=$2
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
+$as_echo_n "checking for $ac_word... " >&6; }
+if ${ac_cv_path_ZSTD+:} false; then :
+  $as_echo_n "(cached) " >&6
+else
+  case $ZSTD in
+  [\\/]* | ?:[\\/]*)
+  ac_cv_path_ZSTD="$ZSTD" # Let the user override the test with a path.
+  ;;
+  *)
+  as_save_IFS=$IFS; IFS=$PATH_SEPARATOR
+for as_dir in $PATH
+do
+  IFS=$as_save_IFS
+  test -z "$as_dir" && as_dir=.
+    for ac_exec_ext in '' $ac_executable_extensions; do
+  if as_fn_executable_p "$as_dir/$ac_word$ac_exec_ext"; then
+    ac_cv_path_ZSTD="$as_dir/$ac_word$ac_exec_ext"
+    $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5
+    break 2
+  fi
+done
+  done
+IFS=$as_save_IFS
+
+  ;;
+esac
+fi
+ZSTD=$ac_cv_path_ZSTD
+if test -n "$ZSTD"; then
+  { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ZSTD" >&5
+$as_echo "$ZSTD" >&6; }
+else
+  { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+fi
+
+
+  test -n "$ZSTD" && break
+done
+
+else
+  # Report the value of ZSTD in configure's output in all cases.
+  { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ZSTD" >&5
+$as_echo_n "checking for ZSTD... " >&6; }
+  { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ZSTD" >&5
+$as_echo "$ZSTD" >&6; }
+fi
+
+if test "$with_zstd" = yes; then
+  for ac_header in zstd.h
+do :
+  ac_fn_c_check_header_mongrel "$LINENO" "zstd.h" "ac_cv_header_zstd_h" "$ac_includes_default"
+if test "x$ac_cv_header_zstd_h" = xyes; then :
+  cat >>confdefs.h <<_ACEOF
+#define HAVE_ZSTD_H 1
+_ACEOF
+
+else
+  as_fn_error $? "zstd.h header file is required for ZSTD" "$LINENO" 5
+fi
+
+done
+
+fi
+
 if test "$with_gssapi" = yes ; then
   for ac_header in gssapi/gssapi.h
 do :
diff --git a/configure.ac b/configure.ac
index 16167329fc..729b23fbea 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1056,6 +1056,30 @@ if test "$with_lz4" = yes; then
   done
 fi
 
+#
+# ZSTD
+#
+AC_MSG_CHECKING([whether to build with ZSTD support])
+PGAC_ARG_BOOL(with, zstd, no, [build with ZSTD support],
+              [AC_DEFINE([USE_ZSTD], 1, [Define to 1 to build with ZSTD support. (--with-zstd)])])
+AC_MSG_RESULT([$with_zstd])
+AC_SUBST(with_zstd)
+
+if test "$with_zstd" = yes; then
+  PKG_CHECK_MODULES(ZSTD, libzstd)
+  # We only care about -I, -D, and -L switches;
+  # note that -lzstd will be added by AC_CHECK_LIB below.
+  for pgac_option in $ZSTD_CFLAGS; do
+    case $pgac_option in
+      -I*|-D*) CPPFLAGS="$CPPFLAGS $pgac_option";;
+    esac
+  done
+  for pgac_option in $ZSTD_LIBS; do
+    case $pgac_option in
+      -L*) LDFLAGS="$LDFLAGS $pgac_option";;
+    esac
+  done
+fi
 #
 # Assignments
 #
@@ -1325,6 +1349,10 @@ if test "$with_lz4" = yes ; then
   AC_CHECK_LIB(lz4, LZ4_compress_default, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])])
 fi
 
+if test "$with_zstd" = yes ; then
+  AC_CHECK_LIB(zstd, ZSTD_compress, [], [AC_MSG_ERROR([library 'zstd' is required for ZSTD support])])
+fi
+
 # Note: We can test for libldap_r only after we know PTHREAD_LIBS;
 # also, on AIX, we may need to have openssl in LIBS for this step.
 if test "$with_ldap" = yes ; then
@@ -1490,6 +1518,11 @@ if test "$with_lz4" = yes; then
   AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])
 fi
 
+PGAC_PATH_PROGS(ZSTD, zstd)
+if test "$with_zstd" = yes; then
+  AC_CHECK_HEADERS(zstd.h, [], [AC_MSG_ERROR([zstd.h header file is required for ZSTD])])
+fi
+
 if test "$with_gssapi" = yes ; then
   AC_CHECK_HEADERS(gssapi/gssapi.h, [],
 	[AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])])
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 9dcd54fcbd..c980444233 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -351,6 +351,7 @@ XGETTEXT = @XGETTEXT@
 GZIP	= gzip
 BZIP2	= bzip2
 LZ4	= @LZ4@
+ZSTD	= @ZSTD@
 
 DOWNLOAD = wget -O $@ --no-use-server-timestamps
 #DOWNLOAD = curl -o $@
diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm
index e6f20679dc..85f706b49f 100644
--- a/src/tools/msvc/Solution.pm
+++ b/src/tools/msvc/Solution.pm
@@ -311,6 +311,7 @@ sub GenerateFiles
 		HAVE_LIBXML2                                => undef,
 		HAVE_LIBXSLT                                => undef,
 		HAVE_LIBZ                   => $self->{options}->{zlib} ? 1 : undef,
+		HAVE_LIBZSTD                => undef,
 		HAVE_LINK                   => undef,
 		HAVE_LOCALE_T               => 1,
 		HAVE_LONG_INT_64            => undef,
@@ -432,6 +433,7 @@ sub GenerateFiles
 		HAVE_WRITEV                              => undef,
 		HAVE_X509_GET_SIGNATURE_NID              => 1,
 		HAVE_X86_64_POPCNTQ                      => undef,
+		HAVE_ZSTD_H                              => undef,
 		HAVE__BOOL                               => undef,
 		HAVE__BUILTIN_BSWAP16                    => undef,
 		HAVE__BUILTIN_BSWAP32                    => undef,
@@ -506,6 +508,7 @@ sub GenerateFiles
 		USE_UNNAMED_POSIX_SEMAPHORES        => undef,
 		USE_WIN32_SEMAPHORES                => 1,
 		USE_WIN32_SHARED_MEMORY             => 1,
+		USE_ZSTD                            => undef,
 		WCSTOMBS_L_IN_XLOCALE               => undef,
 		WORDS_BIGENDIAN                     => undef,
 		XLOG_BLCKSZ       => 1024 * $self->{options}->{wal_blocksize},
@@ -539,6 +542,12 @@ sub GenerateFiles
 		$define{HAVE_LZ4_H}  = 1;
 		$define{USE_LZ4}     = 1;
 	}
+	if ($self->{options}->{zstd})
+	{
+		$define{HAVE_LIBZSTD} = 1;
+		$define{HAVE_ZSTD_H}  = 1;
+		$define{USE_ZSTD}     = 1;
+	}
 	if ($self->{options}->{openssl})
 	{
 		$define{USE_OPENSSL} = 1;
@@ -1081,6 +1090,11 @@ sub AddProject
 		$proj->AddIncludeDir($self->{options}->{lz4} . '\include');
 		$proj->AddLibrary($self->{options}->{lz4} . '\lib\liblz4.lib');
 	}
+	if ($self->{options}->{zstd})
+	{
+		$proj->AddIncludeDir($self->{options}->{zstd} . '\include');
+		$proj->AddLibrary($self->{options}->{zstd} . '\lib\libzstd.lib');
+	}
 	if ($self->{options}->{uuid})
 	{
 		$proj->AddIncludeDir($self->{options}->{uuid} . '\include');
@@ -1193,6 +1207,7 @@ sub GetFakeConfigure
 	$cfg .= ' --with-libxml'        if ($self->{options}->{xml});
 	$cfg .= ' --with-libxslt'       if ($self->{options}->{xslt});
 	$cfg .= ' --with-lz4'           if ($self->{options}->{lz4});
+	$cfg .= ' --with-zstd'          if ($self->{options}->{zstd});
 	$cfg .= ' --with-gssapi'        if ($self->{options}->{gss});
 	$cfg .= ' --with-icu'           if ($self->{options}->{icu});
 	$cfg .= ' --with-tcl'           if ($self->{options}->{tcl});
diff --git a/src/tools/msvc/config_default.pl b/src/tools/msvc/config_default.pl
index 7a9b00be72..186849a09a 100644
--- a/src/tools/msvc/config_default.pl
+++ b/src/tools/msvc/config_default.pl
@@ -15,6 +15,7 @@ our $config = {
 	gss       => undef,    # --with-gssapi=<path>
 	icu       => undef,    # --with-icu=<path>
 	lz4       => undef,    # --with-lz4=<path>
+	zstd      => undef,    # --with-zstd=<path>
 	nls       => undef,    # --enable-nls=<path>
 	tap_tests => undef,    # --enable-tap-tests
 	tcl       => undef,    # --with-tcl=<path>
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index a994626239..e2b0db0879 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -36,6 +36,7 @@ do './src/tools/msvc/config.pl' if (-f 'src/tools/msvc/config.pl');
 $ENV{GZIP_PROGRAM} ||= 'gzip';
 $ENV{LZ4} ||= 'lz4';
 $ENV{TAR} ||= 'tar';
+$ENV{ZSTD} ||= 'zstd';
 
 # buildenv.pl is for specifying the build environment settings
 # it should contain lines like:
-- 
2.34.1

#22Andrey Borodin
x4mmm@yandex-team.ru
In reply to: Robert Haas (#1)
Re: adding 'zstd' as a compression algorithm

15 февр. 2022 г., в 23:20, Robert Haas <robertmhaas@gmail.com> написал(а):

Anyway, those are my thoughts. What are yours?

+1 for adding Zstd everywhere. Quite similar patches are already a part of "libpq compression" and "zstd for pg_dump" commitfest entries, so pushing this will simplify those entries IMV.

Also I like the idea of defaulting FPI compression to lz4 and adding Zstd as wal_compression option.
I don't think Zstd is obvious choice for default FPI compression: there are HW setups when disks are almost as fast as RAM. In this case lz4 can improve performance, while Zstd might make things slower.

Thanks!

Best regards, Andrey Borodin.

#23Robert Haas
robertmhaas@gmail.com
In reply to: Michael Paquier (#21)
Re: adding 'zstd' as a compression algorithm

On Wed, Feb 16, 2022 at 11:34 PM Michael Paquier <michael@paquier.xyz> wrote:

Thanks. This looks pretty much right, except for two things that I
have taken the freedom to fix as of the v3 attached.

Ah, OK, cool. It seems like we have consensus on this being a good
direction, so I plan to commit this later today unless objections or
additional review comments turn up.

--
Robert Haas
EDB: http://www.enterprisedb.com

#24Michael Paquier
michael@paquier.xyz
In reply to: Robert Haas (#23)
Re: adding 'zstd' as a compression algorithm

On Thu, Feb 17, 2022 at 09:40:13AM -0500, Robert Haas wrote:

Ah, OK, cool. It seems like we have consensus on this being a good
direction, so I plan to commit this later today unless objections or
additional review comments turn up.

So, will there be a part of the system where we'll make use of that in
15? pg_basebackup for server-side and client-side compression, at
least, as far as I can guess?
--
Michael

#25Andres Freund
andres@anarazel.de
In reply to: Michael Paquier (#21)
Re: adding 'zstd' as a compression algorithm

Hi,

On 2022-02-17 13:34:08 +0900, Michael Paquier wrote:

%define needs to include HAVE_LIBZSTD, HAVE_ZSTD_H and USE_ZSTD, so
this version fails the sanity check between pg_config.h.in and the
MSVC scripts checking that all flags exist.

Do we really need all three defines? How about using AC_CHECK_HEADER() instead
of AC_CHECK_HEADERS()? That wouldn't define HAVE_ZSTD_H. Cases where we error
out if a header isn't found make it a bit pointless to then still define
HAVE_*_H. Plenty other cases in configure.ac just use AC_CHECK_HEADER.

Greetings,

Andres Freund

#26Robert Haas
robertmhaas@gmail.com
In reply to: Michael Paquier (#24)
Re: adding 'zstd' as a compression algorithm

On Thu, Feb 17, 2022 at 8:18 PM Michael Paquier <michael@paquier.xyz> wrote:

On Thu, Feb 17, 2022 at 09:40:13AM -0500, Robert Haas wrote:

Ah, OK, cool. It seems like we have consensus on this being a good
direction, so I plan to commit this later today unless objections or
additional review comments turn up.

So, will there be a part of the system where we'll make use of that in
15? pg_basebackup for server-side and client-side compression, at
least, as far as I can guess?

That's my plan, yes. I think the patches only need a small amount of
work at this point, but I'd like to get this part committed first.

--
Robert Haas
EDB: http://www.enterprisedb.com

#27Robert Haas
robertmhaas@gmail.com
In reply to: Andres Freund (#25)
Re: adding 'zstd' as a compression algorithm

On Thu, Feb 17, 2022 at 8:36 PM Andres Freund <andres@anarazel.de> wrote:

On 2022-02-17 13:34:08 +0900, Michael Paquier wrote:

%define needs to include HAVE_LIBZSTD, HAVE_ZSTD_H and USE_ZSTD, so
this version fails the sanity check between pg_config.h.in and the
MSVC scripts checking that all flags exist.

Do we really need all three defines? How about using AC_CHECK_HEADER() instead
of AC_CHECK_HEADERS()? That wouldn't define HAVE_ZSTD_H. Cases where we error
out if a header isn't found make it a bit pointless to then still define
HAVE_*_H. Plenty other cases in configure.ac just use AC_CHECK_HEADER.

I have to admit to being somewhat confused by the apparent lack of
consistency in the way we do configure checks. The ZSTD check we added
here is just based on the LZ4 check just above it, which was the
result of my commit of Dilip's patch to add LZ4 TOAST compression. So
if we want to do something different we should change them both. But
that just begs the question of why the LZ4 support looks the way it
does, and to be honest I don't recall. The zlib and XSLT formulas just
above are much simpler, but for some reason what we're doing here
seems to be based on the more-complex formula we use for XML support
instead of either of those.

But having said that, the proposed patch adds no new call to
AC_CHECK_HEADER(), and does add a new call to AC_CHECK_HEADERS(), so I
don't understand the specifics of your complaint here.

--
Robert Haas
EDB: http://www.enterprisedb.com

#28Robert Haas
robertmhaas@gmail.com
In reply to: Robert Haas (#27)
Re: adding 'zstd' as a compression algorithm

On Fri, Feb 18, 2022 at 8:48 AM Robert Haas <robertmhaas@gmail.com> wrote:

On Thu, Feb 17, 2022 at 8:36 PM Andres Freund <andres@anarazel.de> wrote:

On 2022-02-17 13:34:08 +0900, Michael Paquier wrote:

%define needs to include HAVE_LIBZSTD, HAVE_ZSTD_H and USE_ZSTD, so
this version fails the sanity check between pg_config.h.in and the
MSVC scripts checking that all flags exist.

Do we really need all three defines? How about using AC_CHECK_HEADER() instead
of AC_CHECK_HEADERS()? That wouldn't define HAVE_ZSTD_H. Cases where we error
out if a header isn't found make it a bit pointless to then still define
HAVE_*_H. Plenty other cases in configure.ac just use AC_CHECK_HEADER.

I have to admit to being somewhat confused by the apparent lack of
consistency in the way we do configure checks. The ZSTD check we added
here is just based on the LZ4 check just above it, which was the
result of my commit of Dilip's patch to add LZ4 TOAST compression. So
if we want to do something different we should change them both. But
that just begs the question of why the LZ4 support looks the way it
does, and to be honest I don't recall. The zlib and XSLT formulas just
above are much simpler, but for some reason what we're doing here
seems to be based on the more-complex formula we use for XML support
instead of either of those.

But having said that, the proposed patch adds no new call to
AC_CHECK_HEADER(), and does add a new call to AC_CHECK_HEADERS(), so I
don't understand the specifics of your complaint here.

Oh wait ... you want it the other way. Yeah, that seems harmless to
change. I wonder how many others there are that could be changed
similarly...

--
Robert Haas
EDB: http://www.enterprisedb.com

#29Robert Haas
robertmhaas@gmail.com
In reply to: Robert Haas (#28)
Re: adding 'zstd' as a compression algorithm

On Fri, Feb 18, 2022 at 9:02 AM Robert Haas <robertmhaas@gmail.com> wrote:

Oh wait ... you want it the other way. Yeah, that seems harmless to
change. I wonder how many others there are that could be changed
similarly...

I went through configure.ac looking for instances of
AC_CHECK_HEADERS() where the corresponding symbol was not used. I
found four:

AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is
required for LZ4])])
AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_MSG_ERROR([gssapi.h header
file is required for GSSAPI])])])
AC_CHECK_HEADERS(ldap.h, [], [AC_MSG_ERROR([header file <ldap.h> is
required for LDAP])]
AC_CHECK_HEADER(winldap.h, [], ...stuff...)

I guess we could clean all of those up similarly.

--
Robert Haas
EDB: http://www.enterprisedb.com

#30Robert Haas
robertmhaas@gmail.com
In reply to: Robert Haas (#29)
2 attachment(s)
Re: adding 'zstd' as a compression algorithm

On Fri, Feb 18, 2022 at 9:24 AM Robert Haas <robertmhaas@gmail.com> wrote:

On Fri, Feb 18, 2022 at 9:02 AM Robert Haas <robertmhaas@gmail.com> wrote:

Oh wait ... you want it the other way. Yeah, that seems harmless to
change. I wonder how many others there are that could be changed
similarly...

I went through configure.ac looking for instances of
AC_CHECK_HEADERS() where the corresponding symbol was not used. I
found four:

AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is
required for LZ4])])
AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_MSG_ERROR([gssapi.h header
file is required for GSSAPI])])])
AC_CHECK_HEADERS(ldap.h, [], [AC_MSG_ERROR([header file <ldap.h> is
required for LDAP])]
AC_CHECK_HEADER(winldap.h, [], ...stuff...)

I guess we could clean all of those up similarly.

So here's a revised patch for zstd support that uses
AC_CHECK_HEADER(), plus a second patch to change the occurrences above
to use AC_CHECK_HEADER() and remove all traces of the corresponding
preprocessor symbol.

--
Robert Haas
EDB: http://www.enterprisedb.com

Attachments:

v4-0001-Add-support-for-building-with-ZSTD.patchapplication/octet-stream; name=v4-0001-Add-support-for-building-with-ZSTD.patchDownload
From 2f45795ae6d67c4952f0cc60ba9d2c60842309bf Mon Sep 17 00:00:00 2001
From: Robert Haas <rhaas@postgresql.org>
Date: Fri, 18 Feb 2022 09:40:34 -0500
Subject: [PATCH v4 1/2] Add support for building with ZSTD.

This commit doesn't actually add anything that uses ZSTD; that will be
done separately. It just puts the basic infrastructure into place.

Jeevan Ladhe, Robert Haas, and Michael Paquier. Reviewed by Justin
Pryzby and Andres Freund.

Discussion: http://postgr.es/m/CA+TgmoatQKGd+8SjcV+bzvw4XaoEwminHjU83yG12+NXtQzTTQ@mail.gmail.com
---
 configure                         | 265 ++++++++++++++++++++++++++++++
 configure.ac                      |  33 ++++
 doc/src/sgml/install-windows.sgml |   9 +
 doc/src/sgml/installation.sgml    |   9 +
 src/Makefile.global.in            |   1 +
 src/include/pg_config.h.in        |   6 +
 src/tools/msvc/Solution.pm        |  13 ++
 src/tools/msvc/config_default.pl  |   1 +
 src/tools/msvc/vcregress.pl       |   1 +
 9 files changed, 338 insertions(+)

diff --git a/configure b/configure
index df72560277..ca890b8b07 100755
--- a/configure
+++ b/configure
@@ -650,6 +650,7 @@ CFLAGS_ARMV8_CRC32C
 CFLAGS_SSE42
 have_win32_dbghelp
 LIBOBJS
+ZSTD
 LZ4
 UUID_LIBS
 LDAP_LIBS_BE
@@ -700,6 +701,9 @@ with_gnu_ld
 LD
 LDFLAGS_SL
 LDFLAGS_EX
+ZSTD_LIBS
+ZSTD_CFLAGS
+with_zstd
 LZ4_LIBS
 LZ4_CFLAGS
 with_lz4
@@ -869,6 +873,7 @@ with_libxslt
 with_system_tzdata
 with_zlib
 with_lz4
+with_zstd
 with_gnu_ld
 with_ssl
 with_openssl
@@ -898,6 +903,8 @@ XML2_CFLAGS
 XML2_LIBS
 LZ4_CFLAGS
 LZ4_LIBS
+ZSTD_CFLAGS
+ZSTD_LIBS
 LDFLAGS_EX
 LDFLAGS_SL
 PERL
@@ -1577,6 +1584,7 @@ Optional Packages:
                           use system time zone data in DIR
   --without-zlib          do not use Zlib
   --with-lz4              build with LZ4 support
+  --with-zstd             build with ZSTD support
   --with-gnu-ld           assume the C compiler uses GNU ld [default=no]
   --with-ssl=LIB          use LIB for SSL/TLS support (openssl)
   --with-openssl          obsolete spelling of --with-ssl=openssl
@@ -1606,6 +1614,8 @@ Some influential environment variables:
   XML2_LIBS   linker flags for XML2, overriding pkg-config
   LZ4_CFLAGS  C compiler flags for LZ4, overriding pkg-config
   LZ4_LIBS    linker flags for LZ4, overriding pkg-config
+  ZSTD_CFLAGS C compiler flags for ZSTD, overriding pkg-config
+  ZSTD_LIBS   linker flags for ZSTD, overriding pkg-config
   LDFLAGS_EX  extra linker flags for linking executables only
   LDFLAGS_SL  extra linker flags for linking shared libraries only
   PERL        Perl program
@@ -9034,6 +9044,146 @@ fi
   done
 fi
 
+#
+# ZSTD
+#
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with ZSTD support" >&5
+$as_echo_n "checking whether to build with ZSTD support... " >&6; }
+
+
+
+# Check whether --with-zstd was given.
+if test "${with_zstd+set}" = set; then :
+  withval=$with_zstd;
+  case $withval in
+    yes)
+
+$as_echo "#define USE_ZSTD 1" >>confdefs.h
+
+      ;;
+    no)
+      :
+      ;;
+    *)
+      as_fn_error $? "no argument expected for --with-zstd option" "$LINENO" 5
+      ;;
+  esac
+
+else
+  with_zstd=no
+
+fi
+
+
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_zstd" >&5
+$as_echo "$with_zstd" >&6; }
+
+
+if test "$with_zstd" = yes; then
+
+pkg_failed=no
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for libzstd" >&5
+$as_echo_n "checking for libzstd... " >&6; }
+
+if test -n "$ZSTD_CFLAGS"; then
+    pkg_cv_ZSTD_CFLAGS="$ZSTD_CFLAGS"
+ elif test -n "$PKG_CONFIG"; then
+    if test -n "$PKG_CONFIG" && \
+    { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"libzstd\""; } >&5
+  ($PKG_CONFIG --exists --print-errors "libzstd") 2>&5
+  ac_status=$?
+  $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+  test $ac_status = 0; }; then
+  pkg_cv_ZSTD_CFLAGS=`$PKG_CONFIG --cflags "libzstd" 2>/dev/null`
+		      test "x$?" != "x0" && pkg_failed=yes
+else
+  pkg_failed=yes
+fi
+ else
+    pkg_failed=untried
+fi
+if test -n "$ZSTD_LIBS"; then
+    pkg_cv_ZSTD_LIBS="$ZSTD_LIBS"
+ elif test -n "$PKG_CONFIG"; then
+    if test -n "$PKG_CONFIG" && \
+    { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"libzstd\""; } >&5
+  ($PKG_CONFIG --exists --print-errors "libzstd") 2>&5
+  ac_status=$?
+  $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+  test $ac_status = 0; }; then
+  pkg_cv_ZSTD_LIBS=`$PKG_CONFIG --libs "libzstd" 2>/dev/null`
+		      test "x$?" != "x0" && pkg_failed=yes
+else
+  pkg_failed=yes
+fi
+ else
+    pkg_failed=untried
+fi
+
+
+
+if test $pkg_failed = yes; then
+        { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+
+if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then
+        _pkg_short_errors_supported=yes
+else
+        _pkg_short_errors_supported=no
+fi
+        if test $_pkg_short_errors_supported = yes; then
+	        ZSTD_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "libzstd" 2>&1`
+        else
+	        ZSTD_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "libzstd" 2>&1`
+        fi
+	# Put the nasty error message in config.log where it belongs
+	echo "$ZSTD_PKG_ERRORS" >&5
+
+	as_fn_error $? "Package requirements (libzstd) were not met:
+
+$ZSTD_PKG_ERRORS
+
+Consider adjusting the PKG_CONFIG_PATH environment variable if you
+installed software in a non-standard prefix.
+
+Alternatively, you may set the environment variables ZSTD_CFLAGS
+and ZSTD_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details." "$LINENO" 5
+elif test $pkg_failed = untried; then
+        { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+	{ { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
+$as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
+as_fn_error $? "The pkg-config script could not be found or is too old.  Make sure it
+is in your PATH or set the PKG_CONFIG environment variable to the full
+path to pkg-config.
+
+Alternatively, you may set the environment variables ZSTD_CFLAGS
+and ZSTD_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details.
+
+To get pkg-config, see <http://pkg-config.freedesktop.org/>.
+See \`config.log' for more details" "$LINENO" 5; }
+else
+	ZSTD_CFLAGS=$pkg_cv_ZSTD_CFLAGS
+	ZSTD_LIBS=$pkg_cv_ZSTD_LIBS
+        { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
+$as_echo "yes" >&6; }
+
+fi
+  # We only care about -I, -D, and -L switches;
+  # note that -lzstd will be added by AC_CHECK_LIB below.
+  for pgac_option in $ZSTD_CFLAGS; do
+    case $pgac_option in
+      -I*|-D*) CPPFLAGS="$CPPFLAGS $pgac_option";;
+    esac
+  done
+  for pgac_option in $ZSTD_LIBS; do
+    case $pgac_option in
+      -L*) LDFLAGS="$LDFLAGS $pgac_option";;
+    esac
+  done
+fi
 #
 # Assignments
 #
@@ -13130,6 +13280,56 @@ fi
 
 fi
 
+if test "$with_zstd" = yes ; then
+  { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ZSTD_compress in -lzstd" >&5
+$as_echo_n "checking for ZSTD_compress in -lzstd... " >&6; }
+if ${ac_cv_lib_zstd_ZSTD_compress+:} false; then :
+  $as_echo_n "(cached) " >&6
+else
+  ac_check_lib_save_LIBS=$LIBS
+LIBS="-lzstd  $LIBS"
+cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h.  */
+
+/* Override any GCC internal prototype to avoid an error.
+   Use char because int might match the return type of a GCC
+   builtin and then its argument prototype would still apply.  */
+#ifdef __cplusplus
+extern "C"
+#endif
+char ZSTD_compress ();
+int
+main ()
+{
+return ZSTD_compress ();
+  ;
+  return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+  ac_cv_lib_zstd_ZSTD_compress=yes
+else
+  ac_cv_lib_zstd_ZSTD_compress=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+    conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_zstd_ZSTD_compress" >&5
+$as_echo "$ac_cv_lib_zstd_ZSTD_compress" >&6; }
+if test "x$ac_cv_lib_zstd_ZSTD_compress" = xyes; then :
+  cat >>confdefs.h <<_ACEOF
+#define HAVE_LIBZSTD 1
+_ACEOF
+
+  LIBS="-lzstd $LIBS"
+
+else
+  as_fn_error $? "library 'zstd' is required for ZSTD support" "$LINENO" 5
+fi
+
+fi
+
 # Note: We can test for libldap_r only after we know PTHREAD_LIBS;
 # also, on AIX, we may need to have openssl in LIBS for this step.
 if test "$with_ldap" = yes ; then
@@ -13902,6 +14102,71 @@ fi
 
 done
 
+fi
+
+if test -z "$ZSTD"; then
+  for ac_prog in zstd
+do
+  # Extract the first word of "$ac_prog", so it can be a program name with args.
+set dummy $ac_prog; ac_word=$2
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
+$as_echo_n "checking for $ac_word... " >&6; }
+if ${ac_cv_path_ZSTD+:} false; then :
+  $as_echo_n "(cached) " >&6
+else
+  case $ZSTD in
+  [\\/]* | ?:[\\/]*)
+  ac_cv_path_ZSTD="$ZSTD" # Let the user override the test with a path.
+  ;;
+  *)
+  as_save_IFS=$IFS; IFS=$PATH_SEPARATOR
+for as_dir in $PATH
+do
+  IFS=$as_save_IFS
+  test -z "$as_dir" && as_dir=.
+    for ac_exec_ext in '' $ac_executable_extensions; do
+  if as_fn_executable_p "$as_dir/$ac_word$ac_exec_ext"; then
+    ac_cv_path_ZSTD="$as_dir/$ac_word$ac_exec_ext"
+    $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5
+    break 2
+  fi
+done
+  done
+IFS=$as_save_IFS
+
+  ;;
+esac
+fi
+ZSTD=$ac_cv_path_ZSTD
+if test -n "$ZSTD"; then
+  { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ZSTD" >&5
+$as_echo "$ZSTD" >&6; }
+else
+  { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+fi
+
+
+  test -n "$ZSTD" && break
+done
+
+else
+  # Report the value of ZSTD in configure's output in all cases.
+  { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ZSTD" >&5
+$as_echo_n "checking for ZSTD... " >&6; }
+  { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ZSTD" >&5
+$as_echo "$ZSTD" >&6; }
+fi
+
+if test "$with_zstd" = yes; then
+  ac_fn_c_check_header_mongrel "$LINENO" "zstd.h" "ac_cv_header_zstd_h" "$ac_includes_default"
+if test "x$ac_cv_header_zstd_h" = xyes; then :
+
+else
+  as_fn_error $? "zstd.h header file is required for ZSTD" "$LINENO" 5
+fi
+
+
 fi
 
 if test "$with_gssapi" = yes ; then
diff --git a/configure.ac b/configure.ac
index 91a28cb50b..331683b336 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1056,6 +1056,30 @@ if test "$with_lz4" = yes; then
   done
 fi
 
+#
+# ZSTD
+#
+AC_MSG_CHECKING([whether to build with ZSTD support])
+PGAC_ARG_BOOL(with, zstd, no, [build with ZSTD support],
+              [AC_DEFINE([USE_ZSTD], 1, [Define to 1 to build with ZSTD support. (--with-zstd)])])
+AC_MSG_RESULT([$with_zstd])
+AC_SUBST(with_zstd)
+
+if test "$with_zstd" = yes; then
+  PKG_CHECK_MODULES(ZSTD, libzstd)
+  # We only care about -I, -D, and -L switches;
+  # note that -lzstd will be added by AC_CHECK_LIB below.
+  for pgac_option in $ZSTD_CFLAGS; do
+    case $pgac_option in
+      -I*|-D*) CPPFLAGS="$CPPFLAGS $pgac_option";;
+    esac
+  done
+  for pgac_option in $ZSTD_LIBS; do
+    case $pgac_option in
+      -L*) LDFLAGS="$LDFLAGS $pgac_option";;
+    esac
+  done
+fi
 #
 # Assignments
 #
@@ -1325,6 +1349,10 @@ if test "$with_lz4" = yes ; then
   AC_CHECK_LIB(lz4, LZ4_compress_default, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])])
 fi
 
+if test "$with_zstd" = yes ; then
+  AC_CHECK_LIB(zstd, ZSTD_compress, [], [AC_MSG_ERROR([library 'zstd' is required for ZSTD support])])
+fi
+
 # Note: We can test for libldap_r only after we know PTHREAD_LIBS;
 # also, on AIX, we may need to have openssl in LIBS for this step.
 if test "$with_ldap" = yes ; then
@@ -1490,6 +1518,11 @@ if test "$with_lz4" = yes; then
   AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])
 fi
 
+PGAC_PATH_PROGS(ZSTD, zstd)
+if test "$with_zstd" = yes; then
+  AC_CHECK_HEADER(zstd.h, [], [AC_MSG_ERROR([zstd.h header file is required for ZSTD])])
+fi
+
 if test "$with_gssapi" = yes ; then
   AC_CHECK_HEADERS(gssapi/gssapi.h, [],
 	[AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])])
diff --git a/doc/src/sgml/install-windows.sgml b/doc/src/sgml/install-windows.sgml
index b3435eabc4..e08c9514d4 100644
--- a/doc/src/sgml/install-windows.sgml
+++ b/doc/src/sgml/install-windows.sgml
@@ -307,6 +307,15 @@ $ENV{MSBFLAGS}="/m";
      </para></listitem>
     </varlistentry>
 
+    <varlistentry>
+     <term><productname>ZSTD</productname></term>
+     <listitem><para>
+      Required for supporting <productname>ZSTD</productname> compression
+      method. Binaries and source can be downloaded from
+      <ulink url="https://github.com/facebook/zstd/releases"></ulink>.
+     </para></listitem>
+    </varlistentry>
+
     <varlistentry>
      <term><productname>OpenSSL</productname></term>
      <listitem><para>
diff --git a/doc/src/sgml/installation.sgml b/doc/src/sgml/installation.sgml
index 094d23c292..311f7f261d 100644
--- a/doc/src/sgml/installation.sgml
+++ b/doc/src/sgml/installation.sgml
@@ -985,6 +985,15 @@ build-postgresql:
        </listitem>
       </varlistentry>
 
+      <varlistentry>
+       <term><option>--with-zstd</option></term>
+       <listitem>
+        <para>
+         Build with <productname>ZSTD</productname> compression support.
+        </para>
+       </listitem>
+      </varlistentry>
+
       <varlistentry>
        <term><option>--with-ssl=<replaceable>LIBRARY</replaceable></option>
        <indexterm>
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 9dcd54fcbd..c980444233 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -351,6 +351,7 @@ XGETTEXT = @XGETTEXT@
 GZIP	= gzip
 BZIP2	= bzip2
 LZ4	= @LZ4@
+ZSTD	= @ZSTD@
 
 DOWNLOAD = wget -O $@ --no-use-server-timestamps
 #DOWNLOAD = curl -o $@
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 12aac8616e..635fbb2181 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -352,6 +352,9 @@
 /* Define to 1 if you have the `z' library (-lz). */
 #undef HAVE_LIBZ
 
+/* Define to 1 if you have the `zstd' library (-lzstd). */
+#undef HAVE_LIBZSTD
+
 /* Define to 1 if you have the `link' function. */
 #undef HAVE_LINK
 
@@ -952,6 +955,9 @@
 /* Define to select Win32-style shared memory. */
 #undef USE_WIN32_SHARED_MEMORY
 
+/* Define to 1 to build with ZSTD support. (--with-zstd) */
+#undef USE_ZSTD
+
 /* Define to 1 if `wcstombs_l' requires <xlocale.h>. */
 #undef WCSTOMBS_L_IN_XLOCALE
 
diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm
index 439809fcd0..a21ea9bef9 100644
--- a/src/tools/msvc/Solution.pm
+++ b/src/tools/msvc/Solution.pm
@@ -311,6 +311,7 @@ sub GenerateFiles
 		HAVE_LIBXML2                                => undef,
 		HAVE_LIBXSLT                                => undef,
 		HAVE_LIBZ                   => $self->{options}->{zlib} ? 1 : undef,
+		HAVE_LIBZSTD                => undef,
 		HAVE_LINK                   => undef,
 		HAVE_LOCALE_T               => 1,
 		HAVE_LONG_INT_64            => undef,
@@ -507,6 +508,7 @@ sub GenerateFiles
 		USE_UNNAMED_POSIX_SEMAPHORES        => undef,
 		USE_WIN32_SEMAPHORES                => 1,
 		USE_WIN32_SHARED_MEMORY             => 1,
+		USE_ZSTD                            => undef,
 		WCSTOMBS_L_IN_XLOCALE               => undef,
 		WORDS_BIGENDIAN                     => undef,
 		XLOG_BLCKSZ       => 1024 * $self->{options}->{wal_blocksize},
@@ -540,6 +542,11 @@ sub GenerateFiles
 		$define{HAVE_LZ4_H}  = 1;
 		$define{USE_LZ4}     = 1;
 	}
+	if ($self->{options}->{zstd})
+	{
+		$define{HAVE_LIBZSTD} = 1;
+		$define{USE_ZSTD}     = 1;
+	}
 	if ($self->{options}->{openssl})
 	{
 		$define{USE_OPENSSL} = 1;
@@ -1082,6 +1089,11 @@ sub AddProject
 		$proj->AddIncludeDir($self->{options}->{lz4} . '\include');
 		$proj->AddLibrary($self->{options}->{lz4} . '\lib\liblz4.lib');
 	}
+	if ($self->{options}->{zstd})
+	{
+		$proj->AddIncludeDir($self->{options}->{zstd} . '\include');
+		$proj->AddLibrary($self->{options}->{zstd} . '\lib\libzstd.lib');
+	}
 	if ($self->{options}->{uuid})
 	{
 		$proj->AddIncludeDir($self->{options}->{uuid} . '\include');
@@ -1194,6 +1206,7 @@ sub GetFakeConfigure
 	$cfg .= ' --with-libxml'        if ($self->{options}->{xml});
 	$cfg .= ' --with-libxslt'       if ($self->{options}->{xslt});
 	$cfg .= ' --with-lz4'           if ($self->{options}->{lz4});
+	$cfg .= ' --with-zstd'          if ($self->{options}->{zstd});
 	$cfg .= ' --with-gssapi'        if ($self->{options}->{gss});
 	$cfg .= ' --with-icu'           if ($self->{options}->{icu});
 	$cfg .= ' --with-tcl'           if ($self->{options}->{tcl});
diff --git a/src/tools/msvc/config_default.pl b/src/tools/msvc/config_default.pl
index 7a9b00be72..186849a09a 100644
--- a/src/tools/msvc/config_default.pl
+++ b/src/tools/msvc/config_default.pl
@@ -15,6 +15,7 @@ our $config = {
 	gss       => undef,    # --with-gssapi=<path>
 	icu       => undef,    # --with-icu=<path>
 	lz4       => undef,    # --with-lz4=<path>
+	zstd      => undef,    # --with-zstd=<path>
 	nls       => undef,    # --enable-nls=<path>
 	tap_tests => undef,    # --enable-tap-tests
 	tcl       => undef,    # --with-tcl=<path>
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index a994626239..e2b0db0879 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -36,6 +36,7 @@ do './src/tools/msvc/config.pl' if (-f 'src/tools/msvc/config.pl');
 $ENV{GZIP_PROGRAM} ||= 'gzip';
 $ENV{LZ4} ||= 'lz4';
 $ENV{TAR} ||= 'tar';
+$ENV{ZSTD} ||= 'zstd';
 
 # buildenv.pl is for specifying the build environment settings
 # it should contain lines like:
-- 
2.24.3 (Apple Git-128)

v4-0002-Demote-AC_CHECK_HEADERS-calls-to-AC_CHECK_HEADER-.patchapplication/octet-stream; name=v4-0002-Demote-AC_CHECK_HEADERS-calls-to-AC_CHECK_HEADER-.patchDownload
From 53d3cbe8b52f537fc5896cb3de143fc361276ce9 Mon Sep 17 00:00:00 2001
From: Robert Haas <rhaas@postgresql.org>
Date: Fri, 18 Feb 2022 09:47:20 -0500
Subject: [PATCH v4 2/2] Demote AC_CHECK_HEADERS calls to AC_CHECK_HEADER were
 we can.

The difference between these two autoconf macros is that the former
defines a preprocessor symbol while the latter does not. So we should
prefer the latter where the preprocessor symbol is not used anywhere.
Most of the tests in configure.ac follow this rule already, but these
few escaped previous notice.

Discussion: http://postgr.es/m/20220218013612.2fpzbbb7wnfogblj@alap3.anarazel.de
---
 configure                  | 28 ++--------------------------
 configure.ac               |  8 ++++----
 src/include/pg_config.h.in | 12 ------------
 src/tools/msvc/Solution.pm |  5 -----
 4 files changed, 6 insertions(+), 47 deletions(-)

diff --git a/configure b/configure
index ca890b8b07..eed516404f 100755
--- a/configure
+++ b/configure
@@ -14088,19 +14088,13 @@ $as_echo "$LZ4" >&6; }
 fi
 
 if test "$with_lz4" = yes; then
-  for ac_header in lz4.h
-do :
   ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default"
 if test "x$ac_cv_header_lz4_h" = xyes; then :
-  cat >>confdefs.h <<_ACEOF
-#define HAVE_LZ4_H 1
-_ACEOF
 
 else
   as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5
 fi
 
-done
 
 fi
 
@@ -14170,13 +14164,8 @@ fi
 fi
 
 if test "$with_gssapi" = yes ; then
-  for ac_header in gssapi/gssapi.h
-do :
   ac_fn_c_check_header_mongrel "$LINENO" "gssapi/gssapi.h" "ac_cv_header_gssapi_gssapi_h" "$ac_includes_default"
 if test "x$ac_cv_header_gssapi_gssapi_h" = xyes; then :
-  cat >>confdefs.h <<_ACEOF
-#define HAVE_GSSAPI_GSSAPI_H 1
-_ACEOF
 
 else
   for ac_header in gssapi.h
@@ -14195,7 +14184,6 @@ done
 
 fi
 
-done
 
 fi
 
@@ -14294,19 +14282,13 @@ fi
 
 if test "$with_ldap" = yes ; then
   if test "$PORTNAME" != "win32"; then
-     for ac_header in ldap.h
-do :
-  ac_fn_c_check_header_mongrel "$LINENO" "ldap.h" "ac_cv_header_ldap_h" "$ac_includes_default"
+     ac_fn_c_check_header_mongrel "$LINENO" "ldap.h" "ac_cv_header_ldap_h" "$ac_includes_default"
 if test "x$ac_cv_header_ldap_h" = xyes; then :
-  cat >>confdefs.h <<_ACEOF
-#define HAVE_LDAP_H 1
-_ACEOF
 
 else
   as_fn_error $? "header file <ldap.h> is required for LDAP" "$LINENO" 5
 fi
 
-done
 
      { $as_echo "$as_me:${as_lineno-$LINENO}: checking for compatible LDAP implementation" >&5
 $as_echo_n "checking for compatible LDAP implementation... " >&6; }
@@ -14350,22 +14332,16 @@ $as_echo "$as_me: WARNING:
 *** also uses LDAP will crash on exit." >&2;}
 fi
   else
-     for ac_header in winldap.h
-do :
-  ac_fn_c_check_header_compile "$LINENO" "winldap.h" "ac_cv_header_winldap_h" "$ac_includes_default
+     ac_fn_c_check_header_compile "$LINENO" "winldap.h" "ac_cv_header_winldap_h" "$ac_includes_default
 #include <windows.h>
 
 "
 if test "x$ac_cv_header_winldap_h" = xyes; then :
-  cat >>confdefs.h <<_ACEOF
-#define HAVE_WINLDAP_H 1
-_ACEOF
 
 else
   as_fn_error $? "header file <winldap.h> is required for LDAP" "$LINENO" 5
 fi
 
-done
 
   fi
 fi
diff --git a/configure.ac b/configure.ac
index 331683b336..c485831d46 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1515,7 +1515,7 @@ fi
 
 PGAC_PATH_PROGS(LZ4, lz4)
 if test "$with_lz4" = yes; then
-  AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])
+  AC_CHECK_HEADER(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])
 fi
 
 PGAC_PATH_PROGS(ZSTD, zstd)
@@ -1524,7 +1524,7 @@ if test "$with_zstd" = yes; then
 fi
 
 if test "$with_gssapi" = yes ; then
-  AC_CHECK_HEADERS(gssapi/gssapi.h, [],
+  AC_CHECK_HEADER(gssapi/gssapi.h, [],
 	[AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])])
 fi
 
@@ -1557,11 +1557,11 @@ fi
 
 if test "$with_ldap" = yes ; then
   if test "$PORTNAME" != "win32"; then
-     AC_CHECK_HEADERS(ldap.h, [],
+     AC_CHECK_HEADER(ldap.h, [],
                       [AC_MSG_ERROR([header file <ldap.h> is required for LDAP])])
      PGAC_LDAP_SAFE
   else
-     AC_CHECK_HEADERS(winldap.h, [],
+     AC_CHECK_HEADER(winldap.h, [],
                       [AC_MSG_ERROR([header file <winldap.h> is required for LDAP])],
                       [AC_INCLUDES_DEFAULT
 #include <windows.h>
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 635fbb2181..f3cfdd3a35 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -256,9 +256,6 @@
 /* Define to 1 if you have the `gettimeofday' function. */
 #undef HAVE_GETTIMEOFDAY
 
-/* Define to 1 if you have the <gssapi/gssapi.h> header file. */
-#undef HAVE_GSSAPI_GSSAPI_H
-
 /* Define to 1 if you have the <gssapi.h> header file. */
 #undef HAVE_GSSAPI_H
 
@@ -310,9 +307,6 @@
 /* Define to 1 if you have the <langinfo.h> header file. */
 #undef HAVE_LANGINFO_H
 
-/* Define to 1 if you have the <ldap.h> header file. */
-#undef HAVE_LDAP_H
-
 /* Define to 1 if you have the `ldap_initialize' function. */
 #undef HAVE_LDAP_INITIALIZE
 
@@ -367,9 +361,6 @@
 /* Define to 1 if `long long int' works and is 64 bits. */
 #undef HAVE_LONG_LONG_INT_64
 
-/* Define to 1 if you have the <lz4.h> header file. */
-#undef HAVE_LZ4_H
-
 /* Define to 1 if you have the <mbarrier.h> header file. */
 #undef HAVE_MBARRIER_H
 
@@ -709,9 +700,6 @@
 /* Define to 1 if you have the <wctype.h> header file. */
 #undef HAVE_WCTYPE_H
 
-/* Define to 1 if you have the <winldap.h> header file. */
-#undef HAVE_WINLDAP_H
-
 /* Define to 1 if you have the `writev' function. */
 #undef HAVE_WRITEV
 
diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm
index a21ea9bef9..21e575d1bf 100644
--- a/src/tools/msvc/Solution.pm
+++ b/src/tools/msvc/Solution.pm
@@ -279,7 +279,6 @@ sub GenerateFiles
 		HAVE_GETRLIMIT                              => undef,
 		HAVE_GETRUSAGE                              => undef,
 		HAVE_GETTIMEOFDAY                           => undef,
-		HAVE_GSSAPI_GSSAPI_H                        => undef,
 		HAVE_GSSAPI_H                               => undef,
 		HAVE_HMAC_CTX_FREE                          => undef,
 		HAVE_HMAC_CTX_NEW                           => undef,
@@ -297,7 +296,6 @@ sub GenerateFiles
 		HAVE_I_CONSTRAINT__BUILTIN_CONSTANT_P       => undef,
 		HAVE_KQUEUE                                 => undef,
 		HAVE_LANGINFO_H                             => undef,
-		HAVE_LDAP_H                                 => undef,
 		HAVE_LDAP_INITIALIZE                        => undef,
 		HAVE_LIBCRYPTO                              => undef,
 		HAVE_LIBLDAP                                => undef,
@@ -316,7 +314,6 @@ sub GenerateFiles
 		HAVE_LOCALE_T               => 1,
 		HAVE_LONG_INT_64            => undef,
 		HAVE_LONG_LONG_INT_64       => 1,
-		HAVE_LZ4_H                  => undef,
 		HAVE_MBARRIER_H             => undef,
 		HAVE_MBSTOWCS_L             => 1,
 		HAVE_MEMORY_H               => 1,
@@ -427,7 +424,6 @@ sub GenerateFiles
 		HAVE_UUID_OSSP                           => undef,
 		HAVE_UUID_H                              => undef,
 		HAVE_UUID_UUID_H                         => undef,
-		HAVE_WINLDAP_H                           => undef,
 		HAVE_WCSTOMBS_L                          => 1,
 		HAVE_WCTYPE_H                            => 1,
 		HAVE_WRITEV                              => undef,
@@ -539,7 +535,6 @@ sub GenerateFiles
 	if ($self->{options}->{lz4})
 	{
 		$define{HAVE_LIBLZ4} = 1;
-		$define{HAVE_LZ4_H}  = 1;
 		$define{USE_LZ4}     = 1;
 	}
 	if ($self->{options}->{zstd})
-- 
2.24.3 (Apple Git-128)

#31Robert Haas
robertmhaas@gmail.com
In reply to: Robert Haas (#30)
Re: adding 'zstd' as a compression algorithm

On Fri, Feb 18, 2022 at 9:52 AM Robert Haas <robertmhaas@gmail.com> wrote:

So here's a revised patch for zstd support that uses
AC_CHECK_HEADER(), plus a second patch to change the occurrences above
to use AC_CHECK_HEADER() and remove all traces of the corresponding
preprocessor symbol.

And I've now committed the first of these.

--
Robert Haas
EDB: http://www.enterprisedb.com

#32Andres Freund
andres@anarazel.de
In reply to: Robert Haas (#30)
Re: adding 'zstd' as a compression algorithm

On 2022-02-18 09:52:52 -0500, Robert Haas wrote:

plus a second patch to change the occurrences above to use AC_CHECK_HEADER()
and remove all traces of the corresponding preprocessor symbol.

LGTM. I'm not entirely sure the gssapi one is a real improvement, because we
kind of test for that branch, via the #else in HAVE_GSSAPI_H. But on balance,
I'd probably go for it.