Injection point: fix race condition between attach and cleanup
Hi hackers,
(CC: Sawada-san and Alexander, who attended in the original thread)
This is a fork thread from [1]postgresql.org/message-id/flat/OS7PR01MB183172A425382C8A781DB153FF56D2%40OS7PR01MB18317.jpnprd01.prod.outlook.com. I found a race condition between the exiting
backend and another one, while using the injection points.
Found issue
==========
Injection points attached after injection_points_set_local() are removed via
before_shmem_exit callback. This cleanup identifies an injection point only by name.
If another backend detached the point and attached a new callback with the same
name before the original backend exited, the cleanup could detach the new callback
instead.
0001 reproduces the same issue.
Proposed fix
==========
The fix idea is to remember the indicator of attached injection points by the
backend. When the backend is in the local injection point mode and points, the
slot id and its generation are stored in the local list. At the cleanup phase,
the backend iterates all listed injection points and removes it if the generation
is the same. See 0002.
Alternative fix
===========
So far, a regression test 010_index_concurrently_upsert can fail by this issue [2]/messages/by-id/8ccd0497-c914-47e2-804d-299b678863bf@gmail.com.
If we only focus on fixing this test, there is another approach to wait till
cleanups are done. The patch with the prefix "nocfbot" implements the same.
How do you feel? Feedbacks are very welcome.
[1]: postgresql.org/message-id/flat/OS7PR01MB183172A425382C8A781DB153FF56D2%40OS7PR01MB18317.jpnprd01.prod.outlook.com
[2]: /messages/by-id/8ccd0497-c914-47e2-804d-299b678863bf@gmail.com
Best regards,
Hayato Kuroda
FUJITSU LIMITED
Attachments:
nocfbot-0001-Stabilize-010_index_concurrently_upsert.patchapplication/octet-stream; name=nocfbot-0001-Stabilize-010_index_concurrently_upsert.patchDownload+39-17
0001-injection-points-reproducer-for-the-race-condition-b.patchapplication/octet-stream; name=0001-injection-points-reproducer-for-the-race-condition-b.patchDownload
0002-injection-points-Add-handles-for-detaching-callbacks.patchapplication/octet-stream; name=0002-injection-points-Add-handles-for-detaching-callbacks.patchDownload
Hi,
I found that cfbot did not accept the patch. Also, the typedefs.list was not included.
PSA new version. I did not attach for alternative approach not to confuse cfbot.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
Attachments:
v2-0001-injection-points-reproducer-for-the-race-conditio.patchapplication/octet-stream; name=v2-0001-injection-points-reproducer-for-the-race-conditio.patchDownload+111-2
v2-0002-injection-points-Add-handles-for-detaching-callba.patchapplication/octet-stream; name=v2-0002-injection-points-Add-handles-for-detaching-callba.patchDownload
Hi,
Sorry for posting many times. My mailer attached an invalid file.
See the proper version.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
Attachments:
t258485_3v2-0001-injection-points-reproducer-for-the-race-conditio.patchapplication/octet-stream; name=v2-0001-injection-points-reproducer-for-the-race-conditio.patchDownload+111-2
v2-0002-injection-points-Add-handles-for-detaching-callba.patchapplication/octet-stream; name=v2-0002-injection-points-Add-handles-for-detaching-callba.patchDownload+181-40
On Wed, Nov 11, 2026 at 12:26:41PM +0000, Hayato Kuroda (Fujitsu) wrote:
Found issue
==========
Injection points attached after injection_points_set_local() are removed via
before_shmem_exit callback. This cleanup identifies an injection point only by name.
If another backend detached the point and attached a new callback with the same
name before the original backend exited, the cleanup could detach the new callback
instead.
Yeah, sure. Or just make sure that the first backend has exited
before attaching something else to the same point. Or just use a
differently-named point for the second backend, in the same path.
There are a couple of options.
FWIW, I've read your patch, and I am absolutely unconvinced about the
fact that we need to complicate the server side injpoint functionality
in core for only this purpose. Test facilities like this one should
remain simple in design, even if they require tweaks sometimes.
Complicating them would lead to bugs in the facility. I know that I
am pretty adamant about this part of the code, but injpoints are used
for many cases across the whole tree, and what you are doing is
changing one property for another, complicating the code even further,
while on it.
--
Michael
Hi Michael,
Yeah, sure. Or just make sure that the first backend has exited
before attaching something else to the same point. Or just use a
differently-named point for the second backend, in the same path.
There are a couple of options.
You meant everything can be handled on the test code side, right? Per our finding,
010_index_concurrently_upsert can fail due to this issue, and alternative fix
patch for it was to ensure background sessions were exit. See "nocfbot" patch in [1]/messages/by-id/avVNVbUeLngeqiEM@paquier.xyz.
Remained other patches tries to fix the root cause for future developers.
Sawada-san suggested the approach in the original thread [2]/messages/by-id/CAD21AoDbakvmxM2Ggn-80VCNej_0f2-_0E0teyMnPN_3T3YWQw@mail.gmail.com Best regards, Hayato Kuroda FUJITSU LIMITED, he have not seen
the implementation yet though.
FWIW, I've read your patch, and I am absolutely unconvinced about the
fact that we need to complicate the server side injpoint functionality
in core for only this purpose. Test facilities like this one should
remain simple in design, even if they require tweaks sometimes.
Complicating them would lead to bugs in the facility. I know that I
am pretty adamant about this part of the code, but injpoints are used
for many cases across the whole tree, and what you are doing is
changing one property for another, complicating the code even further,
while on it.
Hmm, thank you for the feedback. My alternative idea was to match the MyProcPid
and InjectionPointCondition::pid, but it was also complex. InjectionPointDetach()
is implemented in the core, but the condition is controlled in the extension side
thus it's opaque from the core.
Do some others have simpler ideas? Otherwise, we might choose fixing the test code
to prevent the future BF failure.
[1]: /messages/by-id/avVNVbUeLngeqiEM@paquier.xyz
[2]: /messages/by-id/CAD21AoDbakvmxM2Ggn-80VCNej_0f2-_0E0teyMnPN_3T3YWQw@mail.gmail.com Best regards, Hayato Kuroda FUJITSU LIMITED
Best regards,
Hayato Kuroda
FUJITSU LIMITED
Kuroda-san
InjectionPointDetach()
is implemented in the core, but the condition is controlled in the extension side
thus it's opaque from the core.
As an alternative, could we let the test module supply a callback,
similar to a hook, that checks whether an injection point may be detached?
The callback would run under the same lock as the removal, keeping
checks such as PID matching in the test module.
However, this would still make the core API more complicated. Since I
cannot think of another concrete use case, I am not sure the benefit
justifies adding this mechanism. Would it nevertheless be worth
considering as an alternative?
Koshi Shibagaki
FUJITSU LIMITED
https://www.fujitsu.com/
On Fri, Nov 13, 2026 at 1:43 AM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
Hi Michael,
Yeah, sure. Or just make sure that the first backend has exited
before attaching something else to the same point. Or just use a
differently-named point for the second backend, in the same path.
There are a couple of options.You meant everything can be handled on the test code side, right? Per our finding,
010_index_concurrently_upsert can fail due to this issue, and alternative fix
patch for it was to ensure background sessions were exit. See "nocfbot" patch in [1].
Remained other patches tries to fix the root cause for future developers.
Sawada-san suggested the approach in the original thread [2], he have not seen
the implementation yet though.FWIW, I've read your patch, and I am absolutely unconvinced about the
fact that we need to complicate the server side injpoint functionality
in core for only this purpose. Test facilities like this one should
remain simple in design, even if they require tweaks sometimes.
Complicating them would lead to bugs in the facility. I know that I
am pretty adamant about this part of the code, but injpoints are used
for many cases across the whole tree, and what you are doing is
changing one property for another, complicating the code even further,
while on it.
Agreed. I suggested the tracking idea in the original thread, but
having looked at the patch, it's more complex than I expected.
Hmm, thank you for the feedback. My alternative idea was to match the MyProcPid
and InjectionPointCondition::pid, but it was also complex. InjectionPointDetach()
is implemented in the core, but the condition is controlled in the extension side
thus it's opaque from the core.Do some others have simpler ideas? Otherwise, we might choose fixing the test code
to prevent the future BF failure.
Alternative idea would be to pass an optional bool to
BackgroundPsql.quit so that it quits the session and waits for the
corresponding backend to exit too. I think that would be enough for
010_index_concurrently_upsert. It's close to the alternative patch
Kuroda-san posted, in a reusable form. We can use it also in
051_effective_wal_level.pl.
The same detach-and-attach race can happen in principle for isolation
tests too, if a spec leaves a local point to be cleaned up on exit
while another spec reuses the same name. But I don't think that needs
handling. Using distinct names (or detaching explicitly) avoids it,
and it doesn't seem worth special handling on the injection point
side.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
On Wed, Nov 18, 2026 at 04:33:12PM -0800, Masahiko Sawada wrote:
Alternative idea would be to pass an optional bool to
BackgroundPsql.quit so that it quits the session and waits for the
corresponding backend to exit too. I think that would be enough for
010_index_concurrently_upsert. It's close to the alternative patch
Kuroda-san posted, in a reusable form. We can use it also in
051_effective_wal_level.pl.
I suspect that just offering this guarantee by default would have more
value in the long-run for the whole testing infrastructure. We have
also tests that depend on the timing where the shmem exit callbacks
are called, and backends could be reported as exited back to the
client before they have actually plugged off from the server and run
all the exit/cleanup code...
--
Michael
Hello,
At Wed, 18 Nov 2026 16:33:12 -0800, Masahiko Sawada <sawada.mshk@gmail.com> wrote in
FWIW, I've read your patch, and I am absolutely unconvinced about the
fact that we need to complicate the server side injpoint functionality
in core for only this purpose. Test facilities like this one should
remain simple in design, even if they require tweaks sometimes.
Complicating them would lead to bugs in the facility. I know that I
am pretty adamant about this part of the code, but injpoints are used
for many cases across the whole tree, and what you are doing is
changing one property for another, complicating the code even further,
while on it.Agreed. I suggested the tracking idea in the original thread, but
having looked at the patch, it's more complex than I expected.
...
The same detach-and-attach race can happen in principle for isolation
tests too, if a spec leaves a local point to be cleaned up on exit
while another spec reuses the same name. But I don't think that needs
handling. Using distinct names (or detaching explicitly) avoids it,
and it doesn't seem worth special handling on the injection point
side.
As I understand it, local injection points are a private feature of
the injection_points test module, and they are not designed to support
multiple sessions independently managing injection points with the
same name.
I see this primarily as an issue with how injection points are used in
tests, rather than a race condition that needs to be handled by the
underlying injection point infrastructure or even by the local
injection point implementation. Therefore, I'm also opposed to adding
machinery to either of them to address it.
I haven't looked into this very deeply, but I agree with Michael and
Sawada-san that using distinct injection point names in the test
scripts seems to be the simplest solution and should be sufficient.
Regards,
--
Kyotaro Horiguchi
NTT Open Source Software Center
Hi Michael, Sawada-san and Horiguchi-san,
Thanks for the feedback. OK, let's go ahead to fix the perl module.
My besic idea is to 1) get OID of the backend then 2) do kill(0, $pid) to check
the existence. There were several approaches to get the backend PID to wait for,
I chose to remember the backend PID at wait_connect() and ensure its exit in quit().
This needs additional work for all background sessions but it looks simpler for me.
Another idea I came up with:
* Specify the backend PID by an argument. This can minimize changes on the module
but developers must take care. It may be difficult to make it default.
* Obtain the backend PID in quit(). This must consider the case queries input
before the quit() returns results asynchronously thus stdout/stderr can be
mixture.
0001 implemented the idea and 0002 used it. 0003 patch made sure the waiting is done
by default. Per my quick check, 013_temp_obj_multisession tried to ensure
shmem_cleanup can be blocked, so quit(0) must be used here.
Best regards,
Hayato Kuroda
FUJITSU LIMITED