Race conditions in shm_mq.c
During my experiments with parallel workers I sometimes saw the "master" and
worker process blocked. The master uses shm queue to send data to the worker,
both sides nowait==false. I concluded that the following happened:
The worker process set itself as a receiver on the queue after
shm_mq_wait_internal() has completed its first check of "ptr", so this
function left sender's procLatch in reset state. But before the procLatch was
reset, the receiver still managed to read some data and set sender's procLatch
to signal the reading, and eventually called its (receiver's) WaitLatch().
So sender has effectively missed the receiver's notification and called
WaitLatch() too (if the receiver already waits on its latch, it does not help
for sender to call shm_mq_notify_receiver(): receiver won't do anything
because there's no new data in the queue).
Below is my patch proposal.
--
Antonin Houska
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt
Web: http://www.postgresql-support.de, http://www.cybertec.at
Attachments:
shm_mq_race.patchtext/x-diffDownload+16-0
On Thu, Aug 6, 2015 at 10:10 AM, Antonin Houska <ah@cybertec.at> wrote:
During my experiments with parallel workers I sometimes saw the "master" and
worker process blocked. The master uses shm queue to send data to the worker,
both sides nowait==false. I concluded that the following happened:The worker process set itself as a receiver on the queue after
shm_mq_wait_internal() has completed its first check of "ptr", so this
function left sender's procLatch in reset state. But before the procLatch was
reset, the receiver still managed to read some data and set sender's procLatch
to signal the reading, and eventually called its (receiver's) WaitLatch().So sender has effectively missed the receiver's notification and called
WaitLatch() too (if the receiver already waits on its latch, it does not help
for sender to call shm_mq_notify_receiver(): receiver won't do anything
because there's no new data in the queue).Below is my patch proposal.
Another good catch. However, I would prefer to fix this without
introducing a "continue" as I think that will make the control flow
clearer. Therefore, I propose the attached variant of your idea.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
fix-shm-mq-attach-race.patchtext/x-diff; charset=US-ASCII; name=fix-shm-mq-attach-race.patchDownload+8-1
On Thu, Aug 6, 2015 at 2:38 PM, Robert Haas <robertmhaas@gmail.com> wrote:
On Thu, Aug 6, 2015 at 10:10 AM, Antonin Houska <ah@cybertec.at> wrote:
During my experiments with parallel workers I sometimes saw the "master" and
worker process blocked. The master uses shm queue to send data to the worker,
both sides nowait==false. I concluded that the following happened:The worker process set itself as a receiver on the queue after
shm_mq_wait_internal() has completed its first check of "ptr", so this
function left sender's procLatch in reset state. But before the procLatch was
reset, the receiver still managed to read some data and set sender's procLatch
to signal the reading, and eventually called its (receiver's) WaitLatch().So sender has effectively missed the receiver's notification and called
WaitLatch() too (if the receiver already waits on its latch, it does not help
for sender to call shm_mq_notify_receiver(): receiver won't do anything
because there's no new data in the queue).Below is my patch proposal.
Another good catch. However, I would prefer to fix this without
introducing a "continue" as I think that will make the control flow
clearer. Therefore, I propose the attached variant of your idea.
Err, that doesn't work at all. Have a look at this version instead.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
fix-shm-mq-attach-race-v2.patchtext/x-diff; charset=US-ASCII; name=fix-shm-mq-attach-race-v2.patchDownload+21-17
Robert Haas <robertmhaas@gmail.com> wrote:
On Thu, Aug 6, 2015 at 2:38 PM, Robert Haas <robertmhaas@gmail.com> wrote:
On Thu, Aug 6, 2015 at 10:10 AM, Antonin Houska <ah@cybertec.at> wrote:
During my experiments with parallel workers I sometimes saw the "master" and
worker process blocked. The master uses shm queue to send data to the worker,
both sides nowait==false. I concluded that the following happened:The worker process set itself as a receiver on the queue after
shm_mq_wait_internal() has completed its first check of "ptr", so this
function left sender's procLatch in reset state. But before the procLatch was
reset, the receiver still managed to read some data and set sender's procLatch
to signal the reading, and eventually called its (receiver's) WaitLatch().So sender has effectively missed the receiver's notification and called
WaitLatch() too (if the receiver already waits on its latch, it does not help
for sender to call shm_mq_notify_receiver(): receiver won't do anything
because there's no new data in the queue).Below is my patch proposal.
Another good catch. However, I would prefer to fix this without
introducing a "continue" as I think that will make the control flow
clearer. Therefore, I propose the attached variant of your idea.Err, that doesn't work at all. Have a look at this version instead.
This makes sense to me.
One advantage of "continue" was that I could apply the patch to my test code
(containing the appropriate sleep() calls, to simulate the race conditions)
with no conflicts and see the difference. The restructuring you do does not
allow for such a "mechanical" testing, but it's clear enough.
--
Antonin Houska
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt
Web: http://www.postgresql-support.de, http://www.cybertec.at
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Thu, Aug 6, 2015 at 5:59 PM, Antonin Houska <ah@cybertec.at> wrote:
Robert Haas <robertmhaas@gmail.com> wrote:
On Thu, Aug 6, 2015 at 2:38 PM, Robert Haas <robertmhaas@gmail.com> wrote:
On Thu, Aug 6, 2015 at 10:10 AM, Antonin Houska <ah@cybertec.at> wrote:
During my experiments with parallel workers I sometimes saw the "master" and
worker process blocked. The master uses shm queue to send data to the worker,
both sides nowait==false. I concluded that the following happened:The worker process set itself as a receiver on the queue after
shm_mq_wait_internal() has completed its first check of "ptr", so this
function left sender's procLatch in reset state. But before the procLatch was
reset, the receiver still managed to read some data and set sender's procLatch
to signal the reading, and eventually called its (receiver's) WaitLatch().So sender has effectively missed the receiver's notification and called
WaitLatch() too (if the receiver already waits on its latch, it does not help
for sender to call shm_mq_notify_receiver(): receiver won't do anything
because there's no new data in the queue).Below is my patch proposal.
Another good catch. However, I would prefer to fix this without
introducing a "continue" as I think that will make the control flow
clearer. Therefore, I propose the attached variant of your idea.Err, that doesn't work at all. Have a look at this version instead.
This makes sense to me.
One advantage of "continue" was that I could apply the patch to my test code
(containing the appropriate sleep() calls, to simulate the race conditions)
with no conflicts and see the difference. The restructuring you do does not
allow for such a "mechanical" testing, but it's clear enough.
OK, I've committed that and back-patched it to 9.5 and 9.4.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers