Coverage with make coverage-html is broken on latest Debian using lcov v2

Hi Narek,

On 2026-Apr-21, Narek Galstyan wrote:

On Debian 13 (trixie), `make coverage-html` command triggers an lcov
failure.

[...]

Without applying any of the patches, with --enable-coverage and in-tree
build (no vpath), make check && make coverage-html results in the following
error:

```
/usr/bin/lcov --gcov-tool /usr/bin/gcov -q --no-external -c -i -d . -d . -o
lcov_base.info
lcov: ERROR: (usage) duplicate file ./src/backend/access/table/tableam.gcno
in both . and .

Handling this part with your 0001 seems reasonable to me. I think we
should backpatch that one.

I'm not sure sure about the 0002 patch though. It builds in the
assumption that lcov is broken and that we're going to ignore these
warnings by default [forever]. Do we really want to bake those flags
into our build system?

--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"El número de instalaciones de UNIX se ha elevado a 10,
y se espera que este número aumente" (UPM, 1972)

[ This thread went quiet, but you reminded me of it in answering
Pierre Forstmann's nearby question ]

=?utf-8?Q?=C3=81lvaro?= Herrera <alvherre@kurilemu.de> writes:

Handling this part with your 0001 seems reasonable to me. I think we
should backpatch that one.

Agreed, and it doesn't look like that got done, so I'll go do it now.

I'm not sure sure about the 0002 patch though. It builds in the
assumption that lcov is broken and that we're going to ignore these
warnings by default [forever]. Do we really want to bake those flags
into our build system?

That bothers me too, mainly because I foresee a risk of the switches
hiding genuine problems somewhere down the road. Also the switches
Narek proposes don't match what I've found to be necessary on my
own installation (so maybe there is a gcov version dependency here
too?).

For the moment I'm content to insert the --ignore-errors flags
manually. The 0001 patch should at least reduce the noise level
a bit.

regards, tom lane

I wrote:

=?utf-8?Q?=C3=81lvaro?= Herrera <alvherre@kurilemu.de> writes:

Handling this part with your 0001 seems reasonable to me. I think we
should backpatch that one.

Agreed, and it doesn't look like that got done, so I'll go do it now.

Testing this locally, it initially didn't seem to be doing anything.
But that turns out to be because Red Hat has stuck with lcov 2.0-1,
which does not produce the "duplicate file" error. What it does do
is blindly read every foo.gcda file twice, resulting in counts twice
what they should be :-(. Now I wonder whether pre-2.0 lcov did the
same ... but I don't have an old executable laying about to test with.

I gather that coverage.postgresql.org is running a VPATH build,
because it shows some odd coverage counts as well as even ones.
That's impossible in an in-tree build with this bug, if I've
diagnosed it correctly.

Fix pushed.

regards, tom lane

Hi,

On 2026-10-03 12:39:29 -0400, Tom Lane wrote:

I'm not sure sure about the 0002 patch though. It builds in the
assumption that lcov is broken and that we're going to ignore these
warnings by default [forever]. Do we really want to bake those flags
into our build system?

That bothers me too, mainly because I foresee a risk of the switches
hiding genuine problems somewhere down the road. Also the switches
Narek proposes don't match what I've found to be necessary on my
own installation (so maybe there is a gcov version dependency here
too?).

For the moment I'm content to insert the --ignore-errors flags
manually. The 0001 patch should at least reduce the noise level
a bit.

Yea, I think we really need to fix the sources of some of these corruptions. I
don't think it's primarily lcov's problem, I think it's that we end up with
actually corrupted coverage data and that lcov got *better* at surfacing that.

I know of a few problems:

- We build some code without -pthread that is then used in a threaded
environment. One problem is that gcc uses the presence of -pthread to
influence what -fprofile-update=method gets chosen.

Which, I think, means that any multi-threaded execution of any of that code
ends up using the non-atomic counter updates, which allows them to become
inconsistent.

For autoconf we end up with -pthread for libpq, pgbench, and ecpg. But not
for pgport, pgcommon, which means none of them are safe.

For meson it's similar, except that the backend will typically be built with
-pthread as well. But that doesn't help the fact that pgport, pgcommon can
end up being completely inconsistent.

- Sometimes we can interrupt a process in the middle of a normal exit, while
coverage data being written out, with a SIGQUIT/SIGKILL, which then leads to
corrupted coverage files.

Those coverage files then end up being corrupted.

I unfortunately don't know of a way of fixing that short of teaching libgcov
to update the coverage files with an atomic rename - except that it uses
flock for locking, which probably would be incompatible with that :(.

- code compiled multiple times ends triggering errors

That's the new lcov-2.5 thing where it complains about functions being in
different places if they're built multiple times.

I suspect that a lot of those we could fix with relatively minimal effort,
e.g. by moving the #ifdef SH_RAW_ALLOCATOR around SH_CREATE to inside the
argument list and instead of defining use_builtin_flow() in three places
depending on how things are built, do it in one, moving the ifdefs inside.

- flex generates wrong file locations

This has been an open flex bug for a long time:
https://github.com/westes/flex/issues/235

Kinda wonder if we should just strip line numbers from the file. Or perhaps
we should compile with explicit options to not generate a profile?
Unfortunately the lines are only wrong starting with the first rule,
otherwise it'd not be too hard to just fix that.

Until then I guess it might make sense to add --exclude '*.l' or such? That
does seem to avoid these problems.

- The intentional use of SIGQUIT shutdowns in a lot of tests leads to
incomplete coverage

That doesn't trigger gcov / lcov / genhtml errors, but it leads to things
being assumed uncovered that aren't.

Because we shut down a lot of tap test clusters with immediate mode, this
actually has pretty large impact.

I've experimented with replacing all the _exit() uses in the backend with
something that triggers flushing of the coverage information in some cases
that can be considered kinda maybe safe.

I think there are definitely some bugs in lcov around multi-line statements
that contain branches. That's where it seems to very often get confused and
claims the data is inconsistent, but afaict it's due to it misunderstanding
the data that gcov spits out. I'll try to make a bug report out of that.

Greetings,

Andres Freund

On 2026-Oct-03, Tom Lane wrote:

Testing this locally, it initially didn't seem to be doing anything.
But that turns out to be because Red Hat has stuck with lcov 2.0-1,
which does not produce the "duplicate file" error. What it does do
is blindly read every foo.gcda file twice, resulting in counts twice
what they should be :-(. Now I wonder whether pre-2.0 lcov did the
same ... but I don't have an old executable laying about to test with.

I gather that coverage.postgresql.org is running a VPATH build,
because it shows some odd coverage counts as well as even ones.
That's impossible in an in-tree build with this bug, if I've
diagnosed it correctly.

Hmm, no, it's running an in-tree build,

./configure --cache-file=/home/coverage/pgsrc/configure.cache --enable-depend --enable-coverage --enable-tap-tests --enable-nls --with-python --with-perl --with-tcl --with-openssl --with-libxml --with-ldap --with-pam --with-llvm --with-lz4 --enable-injection-points CFLAGS=-O0 'CPPFLAGS=-DCOPY_PARSE_PLAN_TREES -DWRITE_READ_PARSE_PLAN_TREES -DRAW_EXPRESSION_COVERAGE_TEST' LLVM_CONFIG=/usr/bin/llvm-config-19 CLANG=clang-19 >> $LOG 2>&1

So, I was going to say that Debian packages the same version:

coverage@galvin:~$ lcov --version
lcov: LCOV version 2.0-1

but I found out that the packaging files have a patch file that modifies
the version string, so the code that's actually running is 2.3, not 2.0.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1142382

--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
"La espina, desde que nace, ya pincha" (Proverbio africano)

On 2026-Oct-03, Andres Freund wrote:

- The intentional use of SIGQUIT shutdowns in a lot of tests leads to
incomplete coverage

That doesn't trigger gcov / lcov / genhtml errors, but it leads to things
being assumed uncovered that aren't.

Because we shut down a lot of tap test clusters with immediate mode, this
actually has pretty large impact.

I've experimented with replacing all the _exit() uses in the backend with
something that triggers flushing of the coverage information in some cases
that can be considered kinda maybe safe.

We had these threads about this problem
/messages/by-id/58ce1068-0cfc-3394-df69-ee0d03aaebb5@2ndquadrant.com
/messages/by-id/816a4de5-3f15-4d66-1ec1-791cab2ca6df@gmail.com
but I don't think we did anything about that.

--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/

Hi,

On October 3, 2026 5:03:27 PM EDT, Andres Freund <andres@anarazel.de> wrote:

I think there are definitely some bugs in lcov around multi-line statements
that contain branches. That's where it seems to very often get confused and
claims the data is inconsistent, but afaict it's due to it misunderstanding
the data that gcov spits out. I'll try to make a bug report out of that.

Oh, huh. At least one of the two bugs is actually a behavioral difference between gcc versions, not lcov versions. The gcov data starting with gcc 15 reports

var =
cond ? a : b;

i.e. a variable assignment with a conditional in a new line, different than before. And the new output actually triggers errors in lcov back to at least lcov 2.0. It's actually an error in geninfo.

I found another small repro for multi line conditionals where some branches are never reached triggering errors in lcov itself, and that one is not gcc version dependent...

Will report bugs later today or tomorrow, food is more important now.

Greetings,

Andres
--
Sent from my phone. Please excuse my brevity.

Hi,

On 2026-10-03 19:07:02 -0400, Andres Freund wrote:

On October 3, 2026 5:03:27 PM EDT, Andres Freund <andres@anarazel.de> wrote:

I think there are definitely some bugs in lcov around multi-line statements
that contain branches. That's where it seems to very often get confused and
claims the data is inconsistent, but afaict it's due to it misunderstanding
the data that gcov spits out. I'll try to make a bug report out of that.

Oh, huh. At least one of the two bugs is actually a behavioral difference between gcc versions, not lcov versions. The gcov data starting with gcc 15 reports

var =
cond ? a : b;

i.e. a variable assignment with a conditional in a new line, different than before. And the new output actually triggers errors in lcov back to at least lcov 2.0. It's actually an error in geninfo.

I found another small repro for multi line conditionals where some branches are never reached triggering errors in lcov itself, and that one is not gcc version dependent...

Will report bugs later today or tomorrow, food is more important now.

Bug triggering at least with lcov >= 2.0:
https://github.com/linux-test-project/lcov/issues/554

At first I though this was only reachable with gcc 15, but with some LLM help
I found a version that triggered down to gcc 12.

Bug triggering with lcov >= 2.2:
https://github.com/linux-test-project/lcov/issues/555

this seems independent of the gcc version.

It's kinda interesting that both only trigger for multi-line statements.
Which may be related to why we're seeing them more - we use more multi-line
statements than we used to.

Greetings,

Andres Freund