psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol"

Markus KARG <markus@headcrashing.eu> writes:

Summary: The TLS behavior of psql changed between 16.3 and 16.4 as shown
below.

Have you tried the current release (16.6)?

regards, tom lane

Markus KARG<markus@headcrashing.eu> writes:

Summary: The TLS behavior of psql changed between 16.3 and 16.4 as shown
below.

Have you tried the current release (16.6)?

regards, tom lane

Yes, but psql 16.6 fails in the same way:

root@hetzner-2:~# docker run -it postgres:16.6 psql
"host=headcrashing.eu port=5432 dbname=postgres user=postgres
password=HeadcrashingPostgres2022 sslmode=require"
psql: error: connection to server at "headcrashing.eu" (49.13.53.107),
port 5432 failed: SSL error: tlsv1 alert no application protocol

-Markus

On 25/12/2024 19:05, Markus KARG wrote:

I am running the official PostgreSQL 17.2 Docker Container (https://
hub.docker.com/layers/library/postgres/17.2/images/sha256-
c063081175f45f4a3a5ac03c234e060e67618ebe75b49e2a7ffb79f8357bd1e6)
proxied by a TLSv1.3 proxy (official Traefik 3.2.3 Docker Container
https://hub.docker.com/layers/library/traefik/v3.2.3/images/
sha256-06966a9ba1747ad724a490b8f27df1434c64e8eee5d681df03c4761c9653f62c). Traefik utilizes ACME with Let's Encrypt to produce the TLS certificate.

In v17, libpq requests the ALPN extension in the TLS handshake. Looks
like the proxy doesn't know about the "postgresql" ALPN protocol, and
rejects the connection.

I guess Traefik needs some configuration changes to tell it that the
"postgresql" protocol is expected. Or code changes.

Using the official PostgresSQL Docker Container (16.3 vs 16.4+), I am
asking psql to connect to my server. While psql 16.3 and earlier
versions successfully connect via the TLS proxy to the PostgreSQL
server, psql 16.4 and later versions fail doing so:

root@hetzner-2:~# docker run -it postgres:16.3 psql "host=headcrashing.eu port=5432
dbname=postgres user=postgres password=... sslmode=require"
psql (16.3 (Debian 16.3-1.pgdg120+1), server 17.2 (Debian 17.2-1.pgdg120+1))
WARNING: psql major version 16, server major version 17.
Some psql features might not work.
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_128_GCM_SHA256, compression: off)
Type"help" for help.

postgres=# \q
root@hetzner-2:~# docker run -it postgres:16.4 psql "host=headcrashing.eu port=5432
dbname=postgres user=postgres password=... sslmode=require"
psql: error: connection to server at"headcrashing.eu" (49.13.53.107), port 5432 failed: SSL error: tlsv1 alert no application protocol

There were no changes between 16.3 and 16.4 to explain this. When I test
that with v16 client that I built from sources, I don't get that error.

The error message suggests that you're actually using libpq v17. And
indeed I get that error when connecting with v17 client. Perhaps the
postgres:16.4 docker image was built with v17 libpq?

--
Heikki Linnakangas
Neon (https://neon.tech)

On 25/12/2024 19:05, Markus KARG wrote:

I am running the official PostgreSQL 17.2 Docker Container (https://
hub.docker.com/layers/library/postgres/17.2/images/sha256-
c063081175f45f4a3a5ac03c234e060e67618ebe75b49e2a7ffb79f8357bd1e6)
proxied by a TLSv1.3 proxy (official Traefik 3.2.3 Docker Container
https://hub.docker.com/layers/library/traefik/v3.2.3/images/
sha256-06966a9ba1747ad724a490b8f27df1434c64e8eee5d681df03c4761c9653f62c).
Traefik utilizes ACME with Let's Encrypt to produce the TLS certificate.

In v17, libpq requests the ALPN extension in the TLS handshake. Looks
like the proxy doesn't know about the "postgresql" ALPN protocol, and
rejects the connection.

I guess Traefik needs some configuration changes to tell it that the
"postgresql" protocol is expected. Or code changes.

Traefik does NOT REJECT the connection (if it would, the error message
from psql would be different).

Traefik is "postgres-aware" already since 3.0.0, while I am running 3.2.3.

Note that psql v16.3 works fine but psql v16.4 is not, so a change
introduced by v17 this CANNOT be the cause of the current problem.

Using the official PostgresSQL Docker Container (16.3 vs 16.4+), I am
asking psql to connect to my server. While psql 16.3 and earlier
versions successfully connect via the TLS proxy to the PostgreSQL
server, psql 16.4 and later versions fail doing so:

root@hetzner-2:~# docker run -it postgres:16.3 psql
"host=headcrashing.eu port=5432 dbname=postgres user=postgres
password=... sslmode=require"
psql (16.3 (Debian 16.3-1.pgdg120+1), server 17.2 (Debian
17.2-1.pgdg120+1))
WARNING: psql major version 16, server major version 17.
          Some psql features might not work.
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_128_GCM_SHA256,
compression: off)
Type"help" for help.

postgres=# \q
root@hetzner-2:~# docker run -it postgres:16.4 psql
"host=headcrashing.eu port=5432 dbname=postgres user=postgres
password=... sslmode=require"
psql: error: connection to server at"headcrashing.eu" (49.13.53.107),
port 5432 failed: SSL error: tlsv1 alert no application protocol

There were no changes between 16.3 and 16.4 to explain this. When I
test that with v16 client that I built from sources, I don't get that
error.

The error message suggests that you're actually using libpq v17. And
indeed I get that error when connecting with v17 client. Perhaps the
postgres:16.4 docker image was built with v17 libpq?

I am using the original, pre-built container images found on Docker Hub
and have NOT built them on my own. I am not a PostgresSQL committer
either. So I cannot answer your question.

On 25/12/2024 19:05, Markus KARG wrote:

I am running the official PostgreSQL 17.2 Docker Container (https://
hub.docker.com/layers/library/postgres/17.2/images/sha256-
c063081175f45f4a3a5ac03c234e060e67618ebe75b49e2a7ffb79f8357bd1e6)
proxied by a TLSv1.3 proxy (official Traefik 3.2.3 Docker Container
https://hub.docker.com/layers/library/traefik/v3.2.3/images/
sha256-06966a9ba1747ad724a490b8f27df1434c64e8eee5d681df03c4761c9653f62c).
Traefik utilizes ACME with Let's Encrypt to produce the TLS
certificate.

In v17, libpq requests the ALPN extension in the TLS handshake. Looks
like the proxy doesn't know about the "postgresql" ALPN protocol, and
rejects the connection.

I guess Traefik needs some configuration changes to tell it that the
"postgresql" protocol is expected. Or code changes.

Following this theory, sslnegotiation=postgres used with psql v17.x
should solve the problem, as it falls back to the same mechanism
supported in v16.3, correct? Unfortunately it fails with the same error
message:

root@hetzner-2:~# docker run -it postgres:17.2 psql
"host=headcrashing.eu port=5432 dbname=postgres user=postgres
password=... sslmode=require sslnegotiation=postgres"

psql: error: connection to server at "headcrashing.eu" (49.13.53.107),
port 5432 failed: SSL error: tlsv1 alert no application protocol

The same happens when using sslnegotication=direct.

Regarding your question: I have tried "sslnegotiation=postgres" and
"ssqlnegotiation=direct" with the original official PostgreSQL Docker
Container found on Docker Hub. Using v16.3 psql says it doesn't know the
option "sslnegotiation", and 16.4...17 say "tlsv1 alert no application
protocol". Does PROOFS your claim that the original images on Docker Hub
are using libpq v17 even for psql v16.4+!

-Markus

-Markus

Markus KARG <markus@headcrashing.eu> writes:

Following this theory, sslnegotiation=postgres used with psql v17.x
should solve the problem, as it falls back to the same mechanism
supported in v16.3, correct?

No, sorry, it looks like we insert the ALPN extension into the SSL
request packet regardless of that. Maybe there should have been
a way to suppress that, but v17 libpq doesn't provide one.

Regarding your question: I have tried "sslnegotiation=postgres" and
"ssqlnegotiation=direct" with the original official PostgreSQL Docker
Container found on Docker Hub.

Let's clarify something here: there is nothing "official" about either
that docker container or anything else you might find on DockerHub.
The Postgres community does not produce any such packaging. You'd
have to discussion the inclusion of v17 libpq with whoever did build
that container. I know that Debian thinks it's a good idea to use
latest libpq with older servers, and this builder might be following
their lead. (The timing would be about right, since v17.0 came out
about the same time as 16.4.) This example does show that there
are pitfalls in that policy.

regards, tom lane

Markus KARG<markus@headcrashing.eu> writes:

Following this theory, sslnegotiation=postgres used with psql v17.x
should solve the problem, as it falls back to the same mechanism
supported in v16.3, correct?

No, sorry, it looks like we insert the ALPN extension into the SSL
request packet regardless of that. Maybe there should have been
a way to suppress that, but v17 libpq doesn't provide one.

Thank you, Tom, you saved my day! I have explicitly added the name
`postgresql` to the ALPN configuration on Traefik, and immediately psql
v16.4+ (including psql v17) worked like a charm!

As a reference for others, in `traefik-dynamic.yaml` add the follwing to
make it work (as long as you patch the `default` TLS options, take care
to keep all those protocols contained; alternative path a per-router tls
config instead to ONLY have postgresql in that list):

    ## Dynamic configuration
    tls:
      options:
        default:
          alpnProtocols:
            - acme-tls/1
            - http/1.1
            - h2
            - postgresql

Regarding your question: I have tried "sslnegotiation=postgres" and
"ssqlnegotiation=direct" with the original official PostgreSQL Docker
Container found on Docker Hub.

Let's clarify something here: there is nothing "official" about either
that docker container or anything else you might find on DockerHub.
The Postgres community does not produce any such packaging. You'd
have to discussion the inclusion of v17 libpq with whoever did build
that container. I know that Debian thinks it's a good idea to use
latest libpq with older servers, and this builder might be following
their lead. (The timing would be about right, since v17.0 came out
about the same time as 16.4.) This example does show that there
are pitfalls in that policy.

Thanks a lot for this really essential information! In fact, the authors
of that official docker image (they really call it that way) call
themselves "The PostgreSQL Docker Community". This is totally
misleading, as one understands that to be "The Docker Community withing
the PostgreSQL Community", which it (apparently) is not. Instead, it
seems that it actually is "The PostgreSQL Community at Docker, Inc.",
people totally unrelated to the "real" PostgreSQL Community!

Thanks a lot, Tom, for your kind help. You provided the best Christmas
present this year! :-)