Fix locking issue with fixed-size stats template in injection_points

On Sep 29, 2025, at 08:48, Michael Paquier <michael@paquier.xyz> wrote:

The failure is not surprising, because the stats reports can happen in
a concurrent fashion when a point is run for example, contrary to
other fixed-sized stats kind where the reports are only done by a
single process (archiver, bgwriter, checkpointer). So this is just a
matter of acquiring a lock that was forgotten, to make sure that the
changes are consistent. Far from critical as this is template code,
still embarrassing.

Thoughts or comments?

I saw pg_state_begin_changecount_write() is called multiple places, as you mention, for example bgwriter. But there are not the same lock acquired in other places, for example, in bgwriter:

void
pgstat_report_bgwriter(void)
{
PgStatShared_BgWriter *stats_shmem = &pgStatLocal.shmem->bgwriter;

Assert(!pgStatLocal.shmem->is_shutdown);
pgstat_assert_is_up();

/*
* This function can be called even if nothing at all has happened. In
* this case, avoid unnecessarily modifying the stats entry.
*/
if (pg_memory_is_all_zeros(&PendingBgWriterStats,
sizeof(struct PgStat_BgWriterStats)))
return;

pgstat_begin_changecount_write(&stats_shmem->changecount);

#define BGWRITER_ACC(fld) stats_shmem->stats.fld += PendingBgWriterStats.fld
BGWRITER_ACC(buf_written_clean);
BGWRITER_ACC(maxwritten_clean);
BGWRITER_ACC(buf_alloc);
#undef BGWRITER_ACC

pgstat_end_changecount_write(&stats_shmem->changecount);

Only adding the lock in pg_report_inj_fixed() won’t prevent the race conditions from bgwriter. So I wonder, do we need to add the same lock in the other places?

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

On Mon, Sep 29, 2025 at 09:46:05AM +0800, Chao Li wrote:

I saw pg_state_begin_changecount_write() is called multiple places,
as you mention, for example bgwriter.

I've mentioned that in my first email, and put in details:
- pgstat_report_bgwriter() is called once, by the bgwriter.
- pgstat_report_checkpointer() is called three time, all by the
checkpointer.
- pgstat_report_archiver() is called twice, all by pgarch.c.

So all of them don't have a problem, two calls cannot happen
concurrently.
--
Michael

Hi,

On Mon, Sep 29, 2025 at 09:48:44AM +0900, Michael Paquier wrote:

Then something like the following command:
$ cat create_inj.sql
\set id random(1,100000)
select injection_points_attach('popo:id', 'notice');
select injection_points_run('popo:id');
select injection_points_detach('popo:id');
$ pgbench -n -T 300 -f create_inj.sql -c 10

The failure is not surprising, because the stats reports can happen in
a concurrent fashion when a point is run for example, contrary to
other fixed-sized stats kind where the reports are only done by a
single process (archiver, bgwriter, checkpointer). So this is just a
matter of acquiring a lock that was forgotten, to make sure that the
changes are consistent. Far from critical as this is template code,
still embarrassing.

Thoughts or comments?

Patch LGTM.

Remark: I like the "popo" prefix in your test ;-)

Regards,

--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

On Sep 29, 2025, at 14:27, Michael Paquier <michael@paquier.xyz> wrote:

On Mon, Sep 29, 2025 at 09:46:05AM +0800, Chao Li wrote:

I saw pg_state_begin_changecount_write() is called multiple places,
as you mention, for example bgwriter.

I've mentioned that in my first email, and put in details:
- pgstat_report_bgwriter() is called once, by the bgwriter.
- pgstat_report_checkpointer() is called three time, all by the
checkpointer.
- pgstat_report_archiver() is called twice, all by pgarch.c.

So all of them don't have a problem, two calls cannot happen
concurrently.
--
Michael

Thanks for the clarification. Then the patch looks good to me.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

HI
This patch looks good to me.

The issue is clear: unlike other fixed-size stats kinds (archiver,
bgwriter, checkpointer), the injection_points stats can be updated
concurrently by multiple backends. Without synchronization, this can lead
to inconsistent changecount state and assertion failures in
pgstat_begin_changecount_write(), as shown in your reproduction.

Thanks

On Mon, Sep 29, 2025 at 3:29 PM Chao Li <li.evan.chao@gmail.com> wrote:

On Mon, Sep 29, 2025 at 06:52:31AM +0000, Bertrand Drouvot wrote:

Remark: I like the "popo" prefix in your test ;-)

It's that or popopop. "glop" and "pas-glop" were my other candidates,
but I doubt we can use them freely, and you may be the only one around
here to know what this refers to. :D
--
Michael

On Mon, Sep 29, 2025 at 05:01:37PM +0900, Michael Paquier wrote:

It's that or popopop. "glop" and "pas-glop" were my other candidates,
but I doubt we can use them freely, and you may be the only one around
here to know what this refers to. :D

Putting that aside, fixed down to v18.
--
Michael