MPTCP - multiplexing many TCP connections through one socket to get better bandwidth

On Thu, Sep 4, 2025 at 12:56 PM Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:

Hi -hackers,

With the attached patch PostgreSQL could possibly gain built-in MPTCP
support which would allow multiplexing (aggregating) multiple
kernel-based TCP streams into one MPTCP socket. This allows bypassing
any "chokepoints" on the network transparently for libpq, especially
if having *multiple* TCP streams could achieve higher bandwidth than
single one. One can think of transparent aggregation of bandwidth over
multiple WAN links/tunnels and so. In short it works like this:
libpq_client <--MPTCP--> client_kernel <==multiple TCP
connections==> server_kernel <--MPTCP--> server_kernel

Without much rework of PostgreSQL, this means accelerating any
libpq-based use case. Most obvious beneficiaries could be any
libpq-based heavy network transfers, especially in enterprise
networks. Those come to my mind:
- pg_basebackup (over e.g. WAN or multiple interfaces; but also one
can think of using 2x 10GigE over LAN)
- streaming replication or logical replication [years ago I've was
able to use MPTCP with colleagues on production to bypass single TCP
stream limitation of streaming replication]
- COPY (both upload and download)
- postgres_fdw/dblinks?

MPTCP is IETF standard and included from Linux kernels from some time
(realistically 5.16+?) and it's *enabled* by default in most modern
distributions. One could use it with mptcpize (LD_PRELOAD wrapper to
hijack socket()), but it's not elegant and would require altering
systemd startup scripts (the same story like with NUMA: literally
nobody hacking those to just include numactl --interleave there or
with adjusting ulimits).

The patch right now just assumes IPPROTO_MPTCP is there, so it is not
portable, but not that many OSes support it at all -- I think #ifdef
would be good enough for now. I dont have access to MacOS to develop
this more there, nor I think it would add benefit there, but I may be
wrong. So as such the proposed patch is trivial and Linux-only,
although there is RFC8684[1][2]. I suspect it is way easier and
simpler to support it , rather than try to solve the same problem for
each of the listed use-cases.

Simulation, basic-use and tests:

1. Strictly for demo purposes here, we need to ARTIFICIALLY limit
outbound bandwidth for each new flow (TCP connection) to 10 Mbit/s
using `tc` on the server where PostgreSQL is going to be running later
on (this simulates some chokepoints, multiple WAN paths):
DEV=enp0s31f6
tc qdisc add dev $DEV root handle 1: htb
tc class add dev $DEV parent 1: classid 1:1 htb rate 100mbit
for i in `seq 1 9`; do
tc class add dev $DEV parent 1:1 classid 1:$i htb rate 10mbit
ceil 10mbit
done
# see tc-flow(8) for details, classify each flow with port into
separate class (1:X)
tc filter add dev $DEV parent 1: protocol ip prio 1 handle 1 flow
hash keys src,dst,proto,proto-src,proto-dst divisor 8 baseclass 1

2. From client, verify that single TCP bandwidth is really limited:
verify using iperf3 -P 1 -R -c <server> # if you really getting
limited single-stream TCP connection instead of full
verify using iperf3 -P 8 -R -c <server> # if you really getting
more bandwidth than above

3. Check if MPTCP is enabled and configured on both sides
uname -r # at least 5.10+ according [4] to get this balancing
working, but 6.1+ LTS highly recommended (I've used 6.14.x)
sysctl net.mptcp.enabled # should be 1 on both sides by default
ip mptcp limits set subflows 8 add_addr_accepted 8 # but feel
free to setup max limits

4. Configure MPTCP endpoints on the server (registers some dedicated
listening ports for MPTCP use so that there's no need to use multiple
IP aliases or PBR):
ps uaxw | grep -i mptcpd # check if mptcp daemon (path manager is
running or not), it is NOT required in this case
ip addr ls # let's assume 10.0.1.240 is my main IP on eno1 device,
no need to add new IPs thanks to below trick:
ip mptcp endpoint show # to verify
#ip mptcp endpoint flush # if necessary
# below registers ports 5202..5205 as LISTENing by kernel and
dedicated for MPTCP subflows
ip mptcp endpoint add 10.0.1.240 dev eno1 port 5202 signal
ip mptcp endpoint add 10.0.1.240 dev eno1 port 5203 signal
ip mptcp endpoint add 10.0.1.240 dev eno1 port 5204 signal
ip mptcp endpoint add 10.0.1.240 dev eno1 port 5205 signal
ip mptcp endpoint show # to verify

5. Configure the client:
ip addr ls # here I got 10.0.1.250
ip mptcp endpoint show
ip mptcp endpoint add 10.0.1.250 dev enp0s31f6 subflow fullmesh #
not sure fullmesh is necessary, probably not
ip mptcp limits set add_addr_accepted 8 subflows 8

6. Verify that MPTCP works, rerun tests with mptcpize, e.g.:
on server: mptcpize run iperf3 -s
on client: mptcpize run -d iperf3 -P 1 -R -c <server> # should get
better bandwidth but using just 1 MPTCP connection
on server run PostgreSQL with listen_mptcp='on'
on server: ss -Mtlnp sport 5432 # mptcp should be displayed
on client: run basebackup/psql/..

Sample results for 82MB table copy, it's 3x:
$ time PGMPTCP=0 /usr/pgsql19/bin/psql -h 10.0.1.240 -c '\copy
pgbench_accounts TO '/dev/null';'
COPY 500000
real 0m42.123s

$ time PGMPTCP=1 /usr/pgsql19/bin/psql -h 10.0.1.240 -c '\copy
pgbench_accounts TO '/dev/null';'
enabling MPTCP client
COPY 500000
real 0m14.416s

Sample results for pgbench of DB created with: pgbench -i -s 5,
~1076MB total due to WALs
$ time /usr/pgsql19/bin/pg_basebackup -h 10.0.1.240 -c fast -D /tmp/test -v
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
[..]
pg_basebackup: base backup completed
real 1m26.786s

With PGMPTCP=1 set, it gets ~3x
$ time PGMPTCP=1 /usr/pgsql19/bin/pg_basebackup -h 10.0.1.240 -c
fast -D /tmp/test -v
enabling MPTCP client
pg_basebackup: initiating base backup, waiting for checkpoint to complete
[..]
pg_basebackup: starting background WAL receiver
enabling MPTCP client
[..]
pg_basebackup: base backup completed
real 0m30.460s

Because in the above case, we have advertised 4 IP addresses/port of
server to the client, we got the bump on a single socket (note: flows
end up being hashed into various HTB classes is random depending on
ports used you can get usually 2x .. 4x here). Also as there are two
independent application-based connections here in basebackup (transfer
+ WALs), both get multiplexed (each with 4 subflows). If I would add
more ip mptcp ports (server-side), we could get even more juice of
course there, but it assumes one has that many paths. Some more
advanced setups including separate policy-based-routed (ip rule)
things are possible, and stuff like keeping the TCP connection highly
available 0 even across ISP/interface (WiFi?) outages - is possible.
It works transparently with SSL/TLS too - tested. Of course it won't
remove the single CPU limitation of the tools involved (that's
completely different problem).

If it sounds interesting I was thinking about adding to the patch
something like contrib/mptcpinfo (pg_stat_mptcp view to mimic
pg_stat_ssl). Also as for the patch there were some places where
socket() is being created (libpq cancel packet), but there's no
purpose of adding MPTCP there I think.

It is important to mention there are two implementations of MPTCP on
Linux, so when someone will be googling there's lots of conflicting
information:
1) Earlier one, required kernel patching up to <= 5.6, had
"ndiffports" multiplexer built-in which worked mostly out of the box.
2) Newer one [3], already merged one into kernel today, a little bit
different does not come with built-in ndiffports path manager. In this
newer one, as shown above some more manual steps (ip mptcp endpoints)
may be required, but mptcpd daemon which is managing (sub)flows seems
to be evolving as the usage of this protocol is rising. So I hope in
future all of those mptcp commands would be probably optional.

Thoughts?

-Jakub Wartak.

[1] - https://en.wikipedia.org/wiki/Multipath_TCP
[2] - https://www.rfc-editor.org/rfc/rfc8684.html
[3] - https://www.mptcp.dev/
[4] - https://github.com/multipath-tcp/mptcp_net-next/wiki/#changelog

This is rebased (v2) and revived attempt at (the very simple) MPTCP patch. It
allows bypassing single TCP connection limit speed present on most (all?)
clouds, as they are pretty constrainted in terms of single TCP connection
bandwidth available:

Hardware type | Max aggr bw | 1x TCP LAN | 1x TCP MAN| 1x TCP WAN
Physical server | <= 200 GbE | 1 CPU | 1-10Gbps | 1-10Gbps?
VM (Xen, VMware, etc) | <= 100 GbE? | 1 CPU | 1-10Gbps | 1-10Gbps?
lowend AWS VM | 5-40 Gbps | 5 Gbps | 5 Gbps | <= 5 Gbps
lowend AWS VM Cluster Pl. | <= 200 Gbps | 10 Gbps | 5 Gbps | <= 5 Gbps
highend AWS VM | <= 200 Gbps | 5 Gbps | 5 Gbps | <= 5 Gbps
highend AWS VM+ENA Express| <= 200 Gbps | 25 Gbps | 25 Gbps | <= 5 Gbps
lowend Azure VM | 3(!)-40 Gbps| 3(!) Gbps | 1.5-3 Gbps| 1.5-3 Gbps
highend Azure VM+acc. net.| <= 200 Gbps | 10-12.5Gbps| 10 Gbps | 1.5-3 Gbps

"1 CPU" is of couse not bandwidth, but single CPU core constraint. MAN is just
kinda shortcut for zone/local DC to fit into 80 char limit with the table.

How to use the patch

server:
sysctl net.mptcp.enabled # should be 1 on both sides by default
ip addr ls # to get main IP and device of server
ip mptcp limits set subflows 8 add_addr_accepted 8
# add two addtional listening TCP ports (for subflows)
ip mptcp endpoint add <myIP> dev <mydev> >port 5202 signal # +open it on fw
ip mptcp endpoint add <myIP> dev <mydev> >port 5203 signal # +open it on fw
set listen_mptcp=on in postgresql.auto.conf

client:
sysctl net.mptcp.enabled # should be 1 on both sides by default
ip mptcp limits set subflows 8 add_addr_accepted 8
PGMPTCP=1 pg_basebackup ... (or psql/COPY/whatever over TCP/libpq)

For sample raw performance numbers how much this can help please see recent
pg_basebackup patch [1]/messages/by-id/CAKZiRmwwW-hDc3B6ERJB+paX7RNSBcQLheq1KdsTf42cGuRvuA@mail.gmail.com.

Sample MPTCP results:
iperf3 single TCP stream: 15-23Gps with ENA Express (max what AWS provides)
iperf3 multiple TCP streams (-P 8): 50Gbps (max aggr. bandwidth)
iperf3 single MPTCP stream(!): ~41Gbps
so if there's app that uses 1 socket (literally anything libpq related), you
can easily get ~41Gbps (or much more with this, while on the network supports
maximum X Gbps per single TCP connection speed - as per table, affects every
major cloud infrastructure provider).

To best of my knowledge bandwidth is limited only the by the MPTCP kernel-side
stream (subflow) reassembly on the CPU core where the app is running (Recieve
Steer Side is working fine for all of the subflows so they get different cores,
it's just the final reassembly part that needs to happen before gets it's data
using recv()/recvfrom()).

-J.

[1]: /messages/by-id/CAKZiRmwwW-hDc3B6ERJB+paX7RNSBcQLheq1KdsTf42cGuRvuA@mail.gmail.com

I like the idea a lot. Have not tested yet, but the patch looks okay as a
POC. I'm wondering if rather than a run-time check for IPPROTO_MCP
in src/backend/libpq/pqcomm.c, we shouldn't check right away as soon as it
is enabled via a guc check - see check_bonjour as a good example.

Cheers,
Greg

On Thu, Aug 13, 2026 at 9:59 PM Greg Sabino Mullane <htamfids@gmail.com> wrote:

I like the idea a lot. Have not tested yet, but the patch looks okay as a POC. I'm wondering if rather than a run-time check for IPPROTO_MCP in src/backend/libpq/pqcomm.c, we shouldn't check right away as soon as it is enabled via a guc check - see check_bonjour as a good example.

Hi Greg, thanks for taking a look. I've attached v3 that added
check_listen_mptcp() (GUC check) that enhanced the situation just as You
have indicated.

-J.

On Mon, Aug 17, 2026 at 10:19 AM Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:

On Thu, Aug 13, 2026 at 9:59 PM Greg Sabino Mullane <htamfids@gmail.com> wrote:

I like the idea a lot. Have not tested yet, but the patch looks okay as a POC. I'm wondering if rather than a run-time check for IPPROTO_MCP in src/backend/libpq/pqcomm.c, we shouldn't check right away as soon as it is enabled via a guc check - see check_bonjour as a good example.

Hi Greg, thanks for taking a look. I've attached v3 that added
check_listen_mptcp() (GUC check) that enhanced the situation just as You
have indicated.

Oops, I completely forgot about properly guarding this for non-Linux OS,
e.g. on FreeBSD complained that IPPROTO_MPTCP is unknown. Attached v4
fixes that.

-J.

Thanks for v4! Took a heavier look. Code applies cleanly to
033f39e694872d8d74e670a20093db781eb0bf61, is indented properly, and passes
make check.

doc/src/sgml/libpq.sgml

<term><literal>MPTCP</literal><indexterm><primary>MultiPath

TCP</primary></indexterm></term>

s/MultiPath/Multipath/ (other places like the commit message too)

(multiplexing) over mulitple network paths, provided that remote also

s/mulitple/multiple/

src/backend/libpq/pqcomm.c

if (addr->ai_family != AF_UNIX)
#ifdef IPPROTO_MPTCP
ipprotocol = ListenMPTCP ? IPPROTO_MPTCP : 0;
#else
ipprotocol = 0;
#endif

It's already 0 from the top of the loop, so it should be enough to do:

#ifdef IPPROTO_MPTCP
if (ListenMPTCP && addr->ai_family != AF_UNIX)
socket_protocol = IPPROTO_MPTCP;
#endif

src/backend/utils/misc/postgresql.conf.sample

#listen_mptcp = off # whether to enable Multipathing

TCP or not

s/Multipathing/Multipath/

Or just: # enable Multipath TCP

Can we move to a less prominent place - perhaps under TCP, after
client_connection_check_interval?

src/interfaces/libpq/fe-connect.c

{"mptcp", "PGMPTCP", "0", NULL,

Does this need freeing in freePGconn()?

src/interfaces/libpq/fe-connect.c

if (addr_cur->family != AF_UNIX && conn->mptcp && conn->mptcp[0] == '1')
{
#ifdef IPPROTO_MPTCP
fprintf(stderr, "enabling MPTCP client\n");
ip_protocol = IPPROTO_MPTCP;
#else
fprintf(stderr, "MPTCP client is not supported on this platform\n");
#endif

We should throw a proper message to the user if they attempt mptcp via tcp,
but don't have it enabled by using the libpq_append_conn_error function.
Don't know if a warning for attempting mptcp on via unix socket is worth it.

I'm not clear on the mptcp[0] == '1' bit - does that mean the only way to
invoke it is exactly this?:

PGMPTCP=1

(or I suppose, any other string starting with "1")

Big picture: is it worth making this more generic, in case other protocols
appear some time in the future?

listen_mptcp = on -> listen_protocol = mptcp
PGMPTCP=1 -> PGSOCKETPROTOCOL=mptcp

On re-reading this email, that doesn't allow us to handle different
protocols for different families, so listen_mptcp is fine.

This new ENV should be added to the lists at
src/test/perl/PostgreSQL/Test/Utils.pm
and src/test/regress/pg_regress.c

Ideally also some tests.

Cheers,
Greg