[POC] Implement async DNS with getaddrinfo_a

Started by Jacob Championabout 2 months ago2 messageshackers
Beta feature

Hackorum builds and tests every patch posted to the lists, not only commitfest submissions. This is Hackorum's own CI rather than the PostgreSQL project's, and it is still under testing - please report anything that looks wrong.

appliesbuild failedCI history

Every patchset is also pushed to a branch of our PostgreSQL fork, so you can check out the same tree CI built. Without a PostgreSQL checkout:

git clone --branch t253017_1 https://github.com/hackorum-dev/postgres.git

In a checkout you already have, add the fork once:

git remote add hackorum https://github.com/hackorum-dev/postgres.git

then, for this patchset and every later one:

git fetch hackorum t253017_1 && git checkout t253017_1

Patchset v1 (message #1) is on t253017_1

Jump to latest
#1Jacob Champion
jacob.champion@enterprisedb.com

Hi all,

With several recent discussions focusing on DNS features [1, 2], I
coded up an experiment during PGConf.dev to see if we could get simple
async DNS support going. I cleaned it up a bit today, and it looks
like all the architectural bones are there; we'd just need to do the
(hard) work of deciding what OSes to support and what the behavior
should be in the corner cases.

This is just a proof-of-concept, with multiple TODOs. It hacks up the
existing altsock support in libpq to handle connection establishment,
using glibc's getaddrinfo_a as a sample (the BSDs have their own
distinct APIs, I think). getaddrinfo_a is fundamentally a
background-thread architecture, so this implementation might not be
something we want to pursue in the end. But it does pass
libpq/t/004_load_balance_dns, which is a decent start.

I don't plan to make DNS a primary focus of mine for PG20, but anyone
who's interested is welcome to take this and play with it. Note that
our code coverage for DNS behavior is currently very poor: the
overwhelming majority of our tests use Unix sockets and/or loopback IP
addresses.

Thanks,
--Jacob

[1]: /messages/by-id/8398C22D-429A-4980-9028-4F941F2B7483@yandex-team.ru
[2]: /messages/by-id/AM9PR09MB49008B02CDF003054D5D4E00977DA@AM9PR09MB4900.eurprd09.prod.outlook.com

Attachments:

t253017_1
0001-WIP-Switch-to-getaddrinfo_a-where-available.patchapplication/octet-stream; name=0001-WIP-Switch-to-getaddrinfo_a-where-available.patchDownload+27-4
0002-libpq-Break-address-resolution-into-its-own-helper.patchapplication/octet-stream; name=0002-libpq-Break-address-resolution-into-its-own-helper.patchDownload+140-119
0003-WIP-libpq-Implement-async-DNS-with-green-tests.patchapplication/octet-stream; name=0003-WIP-libpq-Implement-async-DNS-with-green-tests.patchDownload+405-15
#2Andrey Borodin
amborodin@acm.org
In reply to: Jacob Champion (#1)
Re: [POC] Implement async DNS with getaddrinfo_a

Hi Jacob,

On Tue, Jul 7, 2026, Jacob Champion wrote:

This is just a proof-of-concept, with multiple TODOs. It hacks up the
existing altsock support in libpq to handle connection establishment,
using glibc's getaddrinfo_a as a sample (the BSDs have their own
distinct APIs, I think).

Thanks for working this out. Sorry it took me so long to follow up. I
spent more time than I expected thinking about where the abstraction
boundary should be, and I will try to respond much faster in the next
round.

I tried to fit my SRV/SVCB work around your patches, and I think the new
PQconnectPoll() state is the right foundation. You already describe
getaddrinfo_a as a sample and point out that its background-thread
architecture may not be something we want to keep. I agree. My question
is which of the POC's architectural bones we should preserve if we take
it forward.

The POC already hides gaicb and sigevent in src/common/ip.c. What still
seems specific to the getaddrinfo_a implementation is the contract
visible to fe-connect.c: it creates a self-pipe, exposes one end through
altsock, keeps an opaque async_dns_ctx, and knows when to finish and
release the operation. That works for this implementation, but Windows
has GetAddrInfoExW(), the BSDs have different resolver APIs, and c-ares
has multiple sockets plus a timer rather than one completion descriptor.

Could we make the address-resolution extraction establish an internal
resolver interface before putting an asynchronous implementation behind
it? I think there are two useful boundaries here:

1. PQconnectPoll() owns a generic asynchronous operation. It can start
or advance the operation, ask what the caller should wait for, and
cancel and free it. It should not know whether the operation owns a
gaicb, an OVERLAPPED object, or a DNS channel.

2. The resolver produces connection endpoints. An endpoint has a host,
port, and optionally already-resolved addresses. Plain A/AAAA
resolution produces one endpoint, while SRV or SVCB discovery may
produce several. The existing multi-host and target_session_attrs
machinery can then try them.

I am not suggesting that SRV/SVCB support, c-ares, or a public resolver
API has to be part of this patch. They are useful tests of the internal
boundary. If the result is only struct addrinfo, service discovery will
have to bypass the interface or replace it later. Conversely, treating
ordinary resolution as the one-endpoint case does not seem to add much
complexity.

I would avoid fixing the exact wait contract around the first backend.
One completion descriptor is sufficient for this self-pipe, but not for
a resolver with several sockets and a timeout. We could keep that part
private until a second implementation tells us what it actually needs.
The important first step is that fe-connect.c does not own
backend-specific state or cleanup rules.

Your commit message already calls out the need to merge or refactor the
DNS and authentication cleanup. A generic asynchronous-operation object
seems like one way to resolve that TODO while keeping resolver-specific
lifetime rules out of pqDropConnection(). It may also make the DNS test
coverage issue you mentioned easier to address, by giving tests a place
to control resolution without depending entirely on the host resolver.

Your concern about the background-thread architecture may matter even
more when libpq is loaded into a PostgreSQL backend through dblink or
postgres_fdw. Threads created internally by libraries are not
unprecedented, and there is active work on making PostgreSQL capable of
using threads, so the existence of a thread is not by itself an
objection.

However, in this case libpq supplies the callback and manages the
objects it accesses. Before adopting SIGEV_THREAD, I think we need to
understand the callback's signal mask, whether PostgreSQL signals can be
delivered to it, and precisely which operations it may perform while
racing with connection cancellation and destruction.

If someone takes the POC forward, my suggested order would be:

1. Keep CONNECTION_AWAITING_HOST and the extraction of host resolution
from the main connection loop.
2. Put the existing synchronous getaddrinfo implementation behind the
internal resolver interface, preserving current behavior.
3. Use getaddrinfo_a, if its thread model is safe in a backend, and a
second implementation to settle the asynchronous wait interface.
4. Only then decide whether any part of the resolver interface should be
public.

I have a c-ares SVCB prototype that returned multiple endpoints through
a PQsetResolver() hook. That experiment convinced me that endpoint
results are useful, but also that publishing the hook before we have a
sound asynchronous wait model would be premature. I can rebase the
prototype onto an internal interface and help test where the boundary
needs to be, without asking this patch to implement service discovery.

WDYT?

Thank you!

Best regards, Andrey Borodin.