[patch] Imporve pqmq
Hi,
When I use the 'pqmq' recently, I found some issues, just fix them.
Allow the param 'dsm_segment *seg' to be NULL in function
'pq_redirect_to_shm_mq'. As sometimes the shm_mq is created
in shared memory instead of DSM.
Add function 'pq_leave_shm_mq' to allow the process to go
back to the previous pq environment.
Attachments:
0001-Improve-pqmq.patchapplication/octet-stream; name=0001-Improve-pqmq.patchDownload
From 00c9628aa8475e7e9215b5887bb71614cffba56b Mon Sep 17 00:00:00 2001
From: wangxiaoran <fanfuxiaoran@gmail.com>
Date: Thu, 8 Aug 2024 11:02:20 +0800
Subject: [PATCH] Improve pqmq
Allow the param 'dsm_segment *seg' to be NULL in function
'pq_redirect_to_shm_mq'. As sometimes the shm_mq is created
in shared memory instead of DSM.
Add function 'pq_leave_shm_mq' to allow the process to go
back to the previous pq environment.
---
src/backend/libpq/pqmq.c | 22 +++++++++++++++++++++-
src/include/libpq/pqmq.h | 1 +
2 files changed, 22 insertions(+), 1 deletion(-)
diff --git a/src/backend/libpq/pqmq.c b/src/backend/libpq/pqmq.c
index fd735e2fea..7609099629 100644
--- a/src/backend/libpq/pqmq.c
+++ b/src/backend/libpq/pqmq.c
@@ -28,6 +28,10 @@ static bool pq_mq_busy = false;
static pid_t pq_mq_parallel_leader_pid = 0;
static ProcNumber pq_mq_parallel_leader_proc_number = INVALID_PROC_NUMBER;
+static PQcommMethods *pre_PqCommMethods = NULL;
+static CommandDest pre_whereToSendOutput = DestNone;
+static ProtocolVersion pre_FrontendProtocol = PG_PROTOCOL_LATEST;
+
static void pq_cleanup_redirect_to_shm_mq(dsm_segment *seg, Datum arg);
static void mq_comm_reset(void);
static int mq_flush(void);
@@ -52,11 +56,27 @@ static const PQcommMethods PqCommMqMethods = {
void
pq_redirect_to_shm_mq(dsm_segment *seg, shm_mq_handle *mqh)
{
+ pre_PqCommMethods = PqCommMethods;
+ pre_whereToSendOutput = whereToSendOutput;
+ pre_FrontendProtocol = FrontendProtocol;
+
PqCommMethods = &PqCommMqMethods;
pq_mq_handle = mqh;
whereToSendOutput = DestRemote;
FrontendProtocol = PG_PROTOCOL_LATEST;
- on_dsm_detach(seg, pq_cleanup_redirect_to_shm_mq, (Datum) 0);
+ if (seg != NULL)
+ on_dsm_detach(seg, pq_cleanup_redirect_to_shm_mq, (Datum) 0);
+}
+
+void
+pq_leave_shm_mq()
+{
+ if (pq_mq_handle == NULL)
+ return;
+ pq_mq_handle = NULL;
+ PqCommMethods = pre_PqCommMethods;
+ whereToSendOutput = pre_whereToSendOutput;
+ FrontendProtocol = pre_FrontendProtocol;
}
/*
diff --git a/src/include/libpq/pqmq.h b/src/include/libpq/pqmq.h
index 227df8976f..30543582c2 100644
--- a/src/include/libpq/pqmq.h
+++ b/src/include/libpq/pqmq.h
@@ -17,6 +17,7 @@
#include "storage/shm_mq.h"
extern void pq_redirect_to_shm_mq(dsm_segment *seg, shm_mq_handle *mqh);
+extern void pq_leave_shm_mq();
extern void pq_set_parallel_leader(pid_t pid, ProcNumber procNumber);
extern void pq_parse_errornotice(StringInfo msg, ErrorData *edata);
--
2.39.3 (Apple Git-146)
On Wed, Aug 7, 2024 at 11:24 PM Xiaoran Wang <fanfuxiaoran@gmail.com> wrote:
When I use the 'pqmq' recently, I found some issues, just fix them.
Allow the param 'dsm_segment *seg' to be NULL in function
'pq_redirect_to_shm_mq'. As sometimes the shm_mq is created
in shared memory instead of DSM.
Under what circumstances does this happen?
Add function 'pq_leave_shm_mq' to allow the process to go
back to the previous pq environment.
In the code as it currently exists, a parallel worker never has a
connected client, and it talks to a shm_mq instead. So there's no need
for this. If a backend needs to communicate with both a connected
client and also a shm_mq, it probably should not use pqmq but rather
decide explicitly which messages should be sent to the client and
which to the shm_mq. Otherwise, it seems hard to avoid possible loss
of protocol sync.
--
Robert Haas
EDB: http://www.enterprisedb.com
Robert Haas <robertmhaas@gmail.com>
03:24 (6小时前)
发送至 我、 pgsql-hackers
On Wed, Aug 7, 2024 at 11:24 PM Xiaoran Wang <fanfuxiaoran@gmail.com> wrote:
When I use the 'pqmq' recently, I found some issues, just fix them.
Allow the param 'dsm_segment *seg' to be NULL in function
'pq_redirect_to_shm_mq'. As sometimes the shm_mq is created
in shared memory instead of DSM.
. Under what circumstances does this happen?
I just create a shm_mq in shared memory, compared with DSM, it is easier.
And don't need to attach and detach to the DSM.
This shm_mq in shared memory can meet my requirement, which is used
in two different sessions, one session A dumps some information into
another
session B through the shm_mq. Session B is actually a monitor session,
user can
use it to monitor the state of slow queries, such as queries in session A.
Yes, I can choose to use DSM in such situation. But I think it's better
to let the 'pqmq'
to support the shm_mq not in DSM.
Add function 'pq_leave_shm_mq' to allow the process to go
back to the previous pq environment.
. In the code as it currently exists, a parallel worker never has a
. connected client, and it talks to a shm_mq instead. So there's no need
. for this. If a backend needs to communicate with both a connected
client and also a shm_mq, it probably should not use pqmq but rather
decide explicitly which messages should be sent to the client and
which to the shm_mq. Otherwise, it seems hard to avoid possible loss
of protocol sync.
As described above, session B will send a signal to session A, then
session A handle the signal and send the message into the shm_mq.
The message is sent by pq protocol. So session A will firstly call
'pq_redirect_to_shm_mq' and then call 'pq_leave_shm_mq' to
continue to do its work.
Robert Haas <robertmhaas@gmail.com> 于2024年8月9日周五 03:24写道：
Show quoted text
On Wed, Aug 7, 2024 at 11:24 PM Xiaoran Wang <fanfuxiaoran@gmail.com>
wrote:When I use the 'pqmq' recently, I found some issues, just fix them.
Allow the param 'dsm_segment *seg' to be NULL in function
'pq_redirect_to_shm_mq'. As sometimes the shm_mq is created
in shared memory instead of DSM.Under what circumstances does this happen?
Add function 'pq_leave_shm_mq' to allow the process to go
back to the previous pq environment.In the code as it currently exists, a parallel worker never has a
connected client, and it talks to a shm_mq instead. So there's no need
for this. If a backend needs to communicate with both a connected
client and also a shm_mq, it probably should not use pqmq but rather
decide explicitly which messages should be sent to the client and
which to the shm_mq. Otherwise, it seems hard to avoid possible loss
of protocol sync.--
Robert Haas
EDB: http://www.enterprisedb.com
On Thu, Aug 8, 2024 at 10:27 PM Xiaoran Wang <fanfuxiaoran@gmail.com> wrote:
Add function 'pq_leave_shm_mq' to allow the process to go
back to the previous pq environment.. In the code as it currently exists, a parallel worker never has a
. connected client, and it talks to a shm_mq instead. So there's no need
. for this. If a backend needs to communicate with both a connected
client and also a shm_mq, it probably should not use pqmq but rather
decide explicitly which messages should be sent to the client and
which to the shm_mq. Otherwise, it seems hard to avoid possible loss
of protocol sync.As described above, session B will send a signal to session A, then
session A handle the signal and send the message into the shm_mq.
The message is sent by pq protocol. So session A will firstly call
'pq_redirect_to_shm_mq' and then call 'pq_leave_shm_mq' to
continue to do its work.
In this kind of use case, there is really no reason to use the libpq
protocol at all. You would be better off just using a shm_mq directly,
and then you don't need this patch. See tqueue.c for an example of
such a coding pattern.
Using pqmq is very error-prone here. In particular, if a backend
unexpectedly hits an ERROR while the direct is in place, the error
will be sent to the other session rather than to the connected client.
This breaks wire protocol synchronization.
--
Robert Haas
EDB: http://www.enterprisedb.com
Robert Haas <robertmhaas@gmail.com> 于2024年8月13日周二 00:28写道：
On Thu, Aug 8, 2024 at 10:27 PM Xiaoran Wang <fanfuxiaoran@gmail.com>
wrote:Add function 'pq_leave_shm_mq' to allow the process to go
back to the previous pq environment.. In the code as it currently exists, a parallel worker never has a
. connected client, and it talks to a shm_mq instead. So there's noneed
. for this. If a backend needs to communicate with both a connected
client and also a shm_mq, it probably should not use pqmq butrather
decide explicitly which messages should be sent to the client and
which to the shm_mq. Otherwise, it seems hard to avoid possibleloss
of protocol sync.
As described above, session B will send a signal to session A, then
session A handle the signal and send the message into the shm_mq.
The message is sent by pq protocol. So session A will firstly call
'pq_redirect_to_shm_mq' and then call 'pq_leave_shm_mq' to
continue to do its work.In this kind of use case, there is really no reason to use the libpq
protocol at all. You would be better off just using a shm_mq directly,
and then you don't need this patch. See tqueue.c for an example of
such a coding pattern.
Thanks for your reply and suggestion, I will look into that.
Using pqmq is very error-prone here. In particular, if a backend
unexpectedly hits an ERROR while the direct is in place, the error
will be sent to the other session rather than to the connected client.
This breaks wire protocol synchronization.
Yes, I found this problem too. Between the 'pq_beginmessage' and
'pq_endmessage',
any log should not be emitted to the client as it will be sent to the shm_mq
instead of client. Such as I sometimes set client_min_messages='debug1'
in psql, then it will go totally wrong. It maybe better to firstly write
the 'msg'
into a StringInfoData, then send the 'msg' by libpq.
I agree that it is not good way to communicate between tow sessions.
--
Robert Haas
EDB: http://www.enterprisedb.com
--
Best regards !
Xiaoran Wang