Replication: GZIP compression in WAL sender/receiver processes communication?
Hello.
Is there a way to compress the traffic between master and slave during the
replication?.. The streaming gzip would be quite efficient for that.
(WAL archiving is not too good for this purpose because of high lag. I just
need to minimize the cross-datacenter traffic keeping the replication lag
low.)
On Tue, Dec 10, 2013 at 8:13 AM, Dmitry Koterov <dmitry@koterov.ru> wrote:
Hello.
Is there a way to compress the traffic between master and slave during the
replication?.. The streaming gzip would be quite efficient for that.(WAL archiving is not too good for this purpose because of high lag. I just
need to minimize the cross-datacenter traffic keeping the replication lag
low.)
Not directly AFAIK, but this would be a good argument for the
implementation of hooks in walsender and walreceiver that could allow
to use such customization of the stream sent and received.
Note that there is also a patch pending for 9.4 that would make
possible the compression of full page writes, reducing globally the
amount of WAL produced by server in exchange of some CPU work to
compress and decompress the data.
--
Michael
--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
On 12/9/2013 3:13 PM, Dmitry Koterov wrote:
Is there a way to compress the traffic between master and slave during
the replication?.. The streaming gzip would be quite efficient for that.(WAL archiving is not too good for this purpose because of high lag. I
just need to minimize the cross-datacenter traffic keeping the
replication lag low.)
run it through a ssh tunnel that has compression turned on ?
--
john r pierce 37N 122W
somewhere on the middle of the left coast
--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
Hello,
Yes gzip compression can be used for compressing WAL traffic during
streaming replication Following tools can be used in this regard.
SSL compression-SSL support is built in PostgreSQL. You need to ensure you
have OpenSSL library support in your PostgreSQL installation.
Also, you can compress WAL traffic by setting up SSH tunneling between
master and standby and turn on compression while setting up SSH tunnel.
Following link can be followed for the same.
http://www.postgresql.org/docs/9.3/static/ssh-tunnels.html
Ofcourse, these are the solutions outside PostgreSQL.
On Tue, Dec 10, 2013 at 4:43 AM, Dmitry Koterov <dmitry@koterov.ru> wrote:
Show quoted text
Hello.
Is there a way to compress the traffic between master and slave during the
replication?.. The streaming gzip would be quite efficient for that.(WAL archiving is not too good for this purpose because of high lag. I
just need to minimize the cross-datacenter traffic keeping the replication
lag low.)
Hello,
Yes, gzip compression can be used for compressing WAL traffic during
streaming replication Following tools can be used in this regard.
SSL compression-SSL support is built in PostgreSQL. You need to ensure
you have OpenSSL library support in your PostgreSQL installation.
Also, you can compress WAL traffic by setting up SSH tunneling between
master and standby and turn on compression while setting up SSH
tunnel. Following link can be followed for the same.
http://www.postgresql.org/docs/9.3/static/ssh-tunnels.html
Ofcourse, these are the solutions outside PostgreSQL.
On Tue, Dec 10, 2013 at 4:43 AM, Dmitry Koterov <dmitry@koterov.ru> wrote:
Hello.
Is there a way to compress the traffic between master and slave during the replication?.. The streaming gzip would be quite efficient for that.
(WAL archiving is not too good for this purpose because of high lag. I just need to minimize the cross-datacenter traffic keeping the replication lag low.)
--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
On Mon, Dec 9, 2013 at 3:13 PM, Dmitry Koterov <dmitry@koterov.ru> wrote:
Is there a way to compress the traffic between master and slave during the
replication?.. The streaming gzip would be quite efficient for that.
Take a look at the ssh_tunnel.sh [1]https://github.com/grayhemp/pgcookbook/blob/master/ssh_tunnel_with_compression_setup.md tool. This is a wrapper around
SSH tunnel with compression, a watchdog and lock management. Very
useful for cross data center streaming.
[1]: https://github.com/grayhemp/pgcookbook/blob/master/ssh_tunnel_with_compression_setup.md
--
Kind regards,
Sergey Konoplev
PostgreSQL Consultant and DBA
http://www.linkedin.com/in/grayhemp
+1 (415) 867-9984, +7 (901) 903-0499, +7 (988) 888-1979
gray.ru@gmail.com
--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general