PATCH: Batch/pipelining support for libpq
Started by Craig Ringerover 9 years ago129 messages
Hi all
Following on from the foreign table batch inserts thread[1]/messages/by-id/CAMsr+YFgDUiJ37DEfPRk8WDBuZ58psdAYJd8iNFSaGxtw=wU3g@mail.gmail.com, here's a patch
to add support for pipelining queries into asynchronous batches in libpq.
Attached, and also available at
https://github.com/2ndQuadrant/postgres/tree/dev/libpq-async-batch (subject
to rebasing and force pushes).
It's cleaned up over the draft I posted on that thread and has error
recovery implemented. I've written and included the SGML docs for it. The
test program is now pretty comprehensive, more so than for anything else in
libpq anyway. I'll submit it to the next CF as a 9.7/10.0 candidate.
I'm measuring 300x (not %) performance improvements doing batches on
servers over the Internet, so this seems pretty worthwhile. It turned out
to be way less invasive than I expected too.
(I intentionally didn't add any way for clients to annotate each work-item
in a batch with their own private data. I think that'd be really useful and
would make implementing clients easier, but should be a separate patch).
This should be very useful for optimising FDWs, Postgres-XC, etc.
[1]: /messages/by-id/CAMsr+YFgDUiJ37DEfPRk8WDBuZ58psdAYJd8iNFSaGxtw=wU3g@mail.gmail.com
/messages/by-id/CAMsr+YFgDUiJ37DEfPRk8WDBuZ58psdAYJd8iNFSaGxtw=wU3g@mail.gmail.com
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Attachments:
0001-Pipelining-batch-support-for-libpq.patchtext/x-patch; charset=US-ASCII; name=0001-Pipelining-batch-support-for-libpq.patchDownload
From 711a6951f57d84309bd2d7477a145ee3412b7967 Mon Sep 17 00:00:00 2001
From: Craig Ringer <craig@2ndquadrant.com>
Date: Fri, 20 May 2016 12:45:18 +0800
Subject: [PATCH] Pipelining (batch) support for libpq
Allow libpq clients to avoid excessive round trips by pipelining multiple
commands into batches. A sync is only sent at the end of a batch. Commands
in a batch succeed or fail together.
Includes a test program in src/test/examples/testlibpqbatch.c
---
doc/src/sgml/libpq.sgml | 478 +++++++++++++
src/interfaces/libpq/exports.txt | 7 +
src/interfaces/libpq/fe-connect.c | 16 +
src/interfaces/libpq/fe-exec.c | 572 +++++++++++++++-
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 17 +-
src/interfaces/libpq/libpq-fe.h | 13 +-
src/interfaces/libpq/libpq-int.h | 37 +-
src/test/examples/.gitignore | 1 +
src/test/examples/Makefile | 2 +-
src/test/examples/testlibpqbatch.c | 1254 +++++++++++++++++++++++++++++++++++
11 files changed, 2360 insertions(+), 43 deletions(-)
create mode 100644 src/test/examples/testlibpqbatch.c
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 3829a14..d6e896f 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4568,6 +4568,484 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up mulitiple queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/examples/libpqbatch.c</filename>.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It somewhat increased client application
+ complexity and extra caution is required to prevent client/server network
+ deadlocks, but can offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ ("ping time") is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 9.6, but clients using it can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ <application>libpq</application> into batch mode. Enter batch mode with <link
+ linkend="libpq-pqbeginbatchmode"><function>PQbeginBatchMode(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-pqisinbatchmode"><function>PQisInBatchMode(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not allowed. (The restriction on <literal>COPY</literal> is an implementation
+ limit; the PostgreSQL protocol and server can support batched <literal>COPY</literal>).
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQsendEndBatch</function>.
+ Concurrently, it uses <function>PQgetResult</function> and
+ <function>PQgetNextQuery</function> to get results. It may eventually exit
+ batch mode with <function>PQendBatchMode</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it's already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions like
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ etc. The asynchronous requests are followed by a <link
+ linkend="libpq-pqsendendbatch"><function>PQsendEndBatch(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server usually begins executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing with sending batch queries</link>, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-pqgetnextquery"><function>PQgetNextQuery</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null (or would return null if
+ called). The result from the next batch entry may then be retrieved using
+ <function>PQgetNextQuery</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQsendEndBatch</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQsendEndBatch</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetNextQuery(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/examples/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-pqendbatchmode"><function>PQendBatchMode(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-pqisinbatchmode">
+ <term>
+ <function>PQisInBatchMode</function>
+ <indexterm>
+ <primary>PQisInBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns 1 if a <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>, otherwise 0.
+
+<synopsis>
+int PQisInBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqbeginbatchmode">
+ <term>
+ <function>PQbeginBatchMode</function>
+ <indexterm>
+ <primary>PQbeginBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode and returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+<synopsis>
+int PQbeginBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqendbatchmode">
+ <term>
+ <function>PQendBatchMode</function>
+ <indexterm>
+ <primary>PQendBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results and returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQgetNextQuery</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+<synopsis>
+int PQendBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqsendendbatch">
+ <term>
+ <function>PQsendEndBatch</function>
+ <indexterm>
+ <primary>PQsendEndBatch</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQsendEndBatch(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqgetnextquery">
+ <term>
+ <function>PQgetNextQuery</function>
+ <indexterm>
+ <primary>PQgetNextQuery</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results. Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. See <link
+ linkend="libpq-batch-results">processing results</link>.
+
+<synopsis>
+int PQgetNextQuery(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqqueriesinbatch">
+ <term>
+ <function>PQqueriesInBatch</function>
+ <indexterm>
+ <primary>PQqueriesInBatch</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processsed.
+ This is the number of times <function>PQgetNextQuery</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQqueriesInBatch(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqbatchisaborted">
+ <term>
+ <function>PQbatchIsAborted</function>
+ <indexterm>
+ <primary>PQbatchIsAborted</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns 1 if the batch curently being received on a
+ <application>libpq</application> connection in <link
+ linkend="libpq-batch-mode">batch mode</link> is
+ <link linkend="libpq-batch-errors">aborted</link>, 0
+ otherwise. The aborted flag is cleared as soon as the result of the
+ <function>PQsendEndBatch</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function as they can tell
+ that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+
+<synopsis>
+int PQbatchIsAborted(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 21dd772..1b0b8c5 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -171,3 +171,10 @@ PQsslAttributeNames 168
PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
+PQisInBatchMode 172
+PQqueriesInBatch 173
+PQbeginBatchMode 174
+PQendBatchMode 175
+PQsendEndBatch 176
+PQgetNextQuery 177
+PQbatchIsAborted 178
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 9b2839b..78154e2 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -2949,6 +2949,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
/*
@@ -2995,6 +2996,21 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+ queue = queue->next;
+ free(prev);
+ }
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+ queue = conn->cmd_queue_recycle;
+ {
+ PGcommandQueueEntry *prev = queue;
+ queue = queue->next;
+ free(prev);
+ }
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 2621767..319c63e 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -69,6 +71,9 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry* PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry);
/* ----------------
@@ -1090,7 +1095,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1113,6 +1118,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->in_batch)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1211,9 +1223,29 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
+ if (conn->in_batch)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
/* check the arguments */
if (!stmtName)
{
@@ -1269,18 +1301,21 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (!conn->in_batch)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1290,10 +1325,14 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->in_batch)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1340,6 +1379,81 @@ PQsendQueryPrepared(PGconn *conn,
resultFormat);
}
+/* Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry*
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry*) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/* Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/* Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, "tried to recycle non-dangling command queue entry");
+ abort();
+ }
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/* Set up for processing a new query's results */
+static void
+PQstartProcessingNewQuery(PGconn *conn)
+{
+ /* initialize async result-accumulation state */
+ conn->result = NULL;
+ conn->next_result = NULL;
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+}
+
/*
* Common startup code for PQsendQuery and sibling routines
*/
@@ -1359,20 +1473,52 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && !conn->in_batch)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- conn->result = NULL;
- conn->next_result = NULL;
-
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ if (conn->in_batch)
+ {
+ /* When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQgetNextQuery(...) advances
+ * to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ case PGASYNC_IDLE:
+ fprintf(stderr, "internal error, idle state in batch mode");
+ abort();
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately */
+ PQstartProcessingNewQuery(conn);
+ }
/* ready to send command message */
return true;
@@ -1397,6 +1543,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1406,6 +1556,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->in_batch)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1518,22 +1685,25 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (!conn->in_batch)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1543,10 +1713,15 @@ PQsendQueryGuts(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->in_batch)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
+
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1673,6 +1848,282 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/* PQisInBatchMode
+ * Return true if currently in batch mode
+ */
+int
+PQisInBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->in_batch;
+}
+
+/* PQqueriesInBatch
+ * Return true if there are queries currently pending in batch mode
+ */
+int
+PQqueriesInBatch(PGconn *conn)
+{
+ if (!PQisInBatchMode(conn))
+ return false;
+
+ return conn->cmd_queue_head != NULL;
+}
+
+/* PQbatchIsAborted
+ * Batch being processed is aborted, results discarded until next sync
+ */
+int
+PQbatchIsAborted(PGconn *conn)
+{
+ if (!PQisInBatchMode(conn))
+ return false;
+
+ return conn->batch_aborted;
+}
+
+/* Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * COPY is not permitted in batch mode.
+ *
+ * A set of commands is terminated by a PQsendEndBatch. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQendBatchMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQbeginBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->in_batch)
+ return true;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return false;
+
+ conn->in_batch = true;
+ conn->batch_aborted = false;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return true;
+}
+
+/* End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns true if batch mode ended.
+ */
+int
+PQendBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (!conn->in_batch)
+ return true;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ fprintf(stderr, "internal error, IDLE in batch mode");
+ abort();
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ fprintf(stderr, "internal error, COPY in batch mode");
+ abort();
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ return false;
+ case PGASYNC_QUEUED:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ return false;
+
+ conn->in_batch = false;
+ conn->batch_aborted = false;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ return true;
+}
+
+/* End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new non-batch commands until all
+ * results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQsendEndBatch command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQsendEndBatch(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return false;
+
+ if (!conn->in_batch)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ fprintf(stderr, "internal error, IDLE in batch mode");
+ abort();
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ fprintf(stderr, "internal error, COPY in batch mode");
+ abort();
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /* Should try to flush immediately if there's room */
+ PQflush(conn);
+
+ return true;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/* PQgetNextQuery
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns true if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns false if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQgetNextQuery(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return FALSE;
+
+ if (!conn->in_batch)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ fprintf(stderr, "internal error, COPY in batch mode");
+ abort();
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ fprintf(stderr, "internal error, idle in batch mode");
+ abort();
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /* In batch mode but nothing left on the queue; caller can submit
+ * more work or PQendBatchMode() now. */
+ return false;
+ }
+
+ /* Pop the next query from the queue and set up the connection state
+ * as if it'd just been dispatched from a non-batched call */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ PQstartProcessingNewQuery(conn);
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_aborted && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+
+ /* Parse any available data */
+ parseInput(conn);
+ }
+
+ return true;
+}
+
/*
* PQgetResult
@@ -1680,7 +2131,6 @@ PQisBusy(PGconn *conn)
* query work remains or an error has occurred (e.g. out of
* memory).
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1732,10 +2182,31 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->in_batch)
+ {
+ /* batched queries aren't followed by a Sync to put us back in
+ * PGASYNC_IDLE state, and when we do get a sync we could still
+ * have another batch coming after this one.
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1915,6 +2386,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->in_batch)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQexec in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2109,6 +2587,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2124,6 +2605,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->in_batch)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2132,15 +2627,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (!conn->in_batch)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && !conn->in_batch)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2154,10 +2652,14 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->in_batch)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index f1b90f3..e824f36 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->in_batch)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 0b8c62f..3d6e6c9 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->in_batch)
+ {
+ conn->batch_aborted = false;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -305,7 +313,7 @@ pqParseInput3(PGconn *conn)
* parsing until the application accepts the current
* result.
*/
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
return;
}
break;
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->in_batch)
+ conn->batch_aborted = true;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 9ca0756..a05298f 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -91,7 +91,9 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort earlier in a batch */
} ExecStatusType;
typedef enum
@@ -421,6 +423,15 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQisInBatchMode(PGconn *conn);
+extern int PQbatchIsAborted(PGconn *conn);
+extern int PQqueriesInBatch(PGconn *conn);
+extern int PQbeginBatchMode(PGconn *conn);
+extern int PQendBatchMode(PGconn *conn);
+extern int PQsendEndBatch(PGconn *conn);
+extern int PQgetNextQuery(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 1183323..a21e07c 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,10 +217,13 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch result,
+ More results expected from this query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -229,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -292,6 +296,22 @@ typedef struct pgDataValue
const char *value; /* data value, without zero-termination */
} PGdataValue;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* PGconn stores all the state data associated with a single connection
* to a backend.
@@ -356,6 +376,8 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ bool in_batch; /* connection is in batch (pipelined) mode */
+ bool batch_aborted; /* current batch is aborted, discarding until next Sync */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY
@@ -363,6 +385,15 @@ struct pg_conn
PGnotify *notifyHead; /* oldest unreported Notify msg */
PGnotify *notifyTail; /* newest unreported Notify msg */
+ /* The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
/* See PQconnectPoll() for how we use 'int' and not 'pgsocket'. */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
diff --git a/src/test/examples/.gitignore b/src/test/examples/.gitignore
index 1957ec1..7a72420 100644
--- a/src/test/examples/.gitignore
+++ b/src/test/examples/.gitignore
@@ -4,3 +4,4 @@
/testlibpq4
/testlo
/testlo64
+/testlibpqbatch
diff --git a/src/test/examples/Makefile b/src/test/examples/Makefile
index 31da210..92a6faf 100644
--- a/src/test/examples/Makefile
+++ b/src/test/examples/Makefile
@@ -14,7 +14,7 @@ override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
override LDLIBS := $(libpq_pgport) $(LDLIBS)
-PROGS = testlibpq testlibpq2 testlibpq3 testlibpq4 testlo testlo64
+PROGS = testlibpq testlibpq2 testlibpq3 testlibpq4 testlo testlo64 testlibpqbatch
all: $(PROGS)
diff --git a/src/test/examples/testlibpqbatch.c b/src/test/examples/testlibpqbatch.c
new file mode 100644
index 0000000..b137d6f
--- /dev/null
+++ b/src/test/examples/testlibpqbatch.c
@@ -0,0 +1,1254 @@
+/*
+ * src/test/examples/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution funtionality
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+
+#ifndef VERBOSE
+#define VERBOSE 0
+#endif
+
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+ fflush(stderr);
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode\n");
+ goto fail;
+ }
+
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQendBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode with work in progress should fail, but succeeded\n");
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending a batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * in batch mode we have to ask for the first result to be processed;
+ * until we do PQgetResult will return null:
+ */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQgetNextQuery() call\n");
+ goto fail;
+ }
+
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* We can't PQgetNextQuery when there might still be pending results */
+ if (PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() should've failed with pending results: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQgetNextQuery() call\n");
+ goto fail;
+ }
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQendBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ /* should now get an explicit sync result */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after end batch call\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+ fflush(stderr);
+
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode: %u\n", __LINE__);
+ goto fail;
+ }
+
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "Unable to enter batch mode\n");
+ goto fail;
+ }
+
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Batch mode not activated properly\n");
+ goto fail;
+ }
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "PQexec should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ {
+ fprintf(stderr, "PQsendQuery should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "re-entering batch mode should be a no-op but failed\n");
+ goto fail;
+ }
+
+ if (PQisBusy(conn))
+ {
+ fprintf(stderr, "PQisBusy should return false when idle in batch, returned true\n");
+ goto fail;
+ }
+
+ /* ok, back to normal command mode */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "couldn't exit idle empty batch mode\n");
+ goto fail;
+ }
+
+ if (PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Batch mode not terminated properly\n");
+ goto fail;
+ }
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "batch mode exit when not in batch mode should succeed but failed\n");
+ goto fail;
+ }
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+ fflush(stderr);
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* OK, start processing the batch results */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQgetNextQuery() call\n");
+ goto fail;
+ }
+
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQgetNextQuery() call\n");
+ goto fail;
+ }
+
+ if (PQendBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ /* second batch */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+ fflush(stderr);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "1";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching error select failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "2";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "3";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* second query, caused error */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (!PQbatchIsAborted(conn))
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* third query in batch, the second insert */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at third batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (!PQbatchIsAborted(conn))
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the batch sync */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchIsAborted(conn))
+ {
+ fprintf(stderr, "sync should've cleared the aborted flag but didn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the insert from the second batch */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first entry in second batch: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first item in second batch\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* the second batch sync */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch sync\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ {
+ fprintf(stderr, "expected only insert with value 3, got %s", val);
+ goto fail;
+ }
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ fprintf(stderr, "expected 1 result, got %d", PQntuples(res));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "xact start failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "sent BEGIN\n");
+#endif
+ send_step = BI_DROP_TABLE;
+
+ if (!PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "sent DROP\n");
+#endif
+ send_step = BI_CREATE_TABLE;
+
+ if (!PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "sent CREATE\n");
+#endif
+ send_step = BI_PREPARE;
+
+ if (!PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids))
+ {
+ fprintf(stderr, "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "sent PREPARE\n");
+#endif
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ {
+ fprintf(stderr, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "Expected next query result but unable to dequeue: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "next query!\n");
+#endif
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+#if VERBOSE
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+#endif
+
+ if (PQresultStatus(res) != status)
+ {
+ fprintf(stderr, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ {
+ fprintf(stderr, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+#endif
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+#if VERBOSE
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+#endif
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+#if VERBOSE
+ fprintf(stdout, "sent COMMIT\n");
+#endif
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQsendEndBatch(conn))
+ {
+#if VERBOSE
+ fprintf(stdout, "Dispatched end batch message\n");
+#endif
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ {
+ fprintf(stderr, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "prepare failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows]]\n", progname);
+ exit(1);
+}
+
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ struct timeval start_time,
+ end_time,
+ elapsed_time;
+ int number_of_rows = 10000;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 3)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ test_disallowed_in_batch(conn);
+ simple_batch(conn);
+ multi_batch(conn);
+ test_batch_abort(conn);
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_pipelined(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("batch insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_sequential(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("sequential insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, elapsed_time.tv_usec);
+
+ fprintf(stderr, "Done.\n");
+
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
--
2.5.5
Re: PATCH: Batch/pipelining support for libpq
Hi,
On 2016-05-23 17:19:09 +0800, Craig Ringer wrote:
Hi all
Following on from the foreign table batch inserts thread[1], here's a patch
to add support for pipelining queries into asynchronous batches in libpq.
Yay!
I'm measuring 300x (not %) performance improvements doing batches on
servers over the Internet, so this seems pretty worthwhile. It turned out
to be way less invasive than I expected too.
yay^2.
(I intentionally didn't add any way for clients to annotate each work-item
in a batch with their own private data. I think that'd be really useful and
would make implementing clients easier, but should be a separate patch).This should be very useful for optimising FDWs, Postgres-XC, etc.
And optimizing normal clients.
Not easy, but I'd be very curious how much psql's performance improves
when replaying a .sql style dump, and replaying from a !tty fd, after
hacking it up to use the batch API.
- Andres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On Mon, May 23, 2016 at 8:50 AM, Andres Freund <andres@anarazel.de> wrote:
On 2016-05-23 17:19:09 +0800, Craig Ringer wrote:
Following on from the foreign table batch inserts thread[1], here's a patch
to add support for pipelining queries into asynchronous batches in libpq.Yay!
I'm measuring 300x (not %) performance improvements doing batches on
servers over the Internet, so this seems pretty worthwhile. It turned out
to be way less invasive than I expected too.yay^2.
I'll follow this mood. Yeha.
(I intentionally didn't add any way for clients to annotate each work-item
in a batch with their own private data. I think that'd be really useful and
would make implementing clients easier, but should be a separate patch).This should be very useful for optimising FDWs, Postgres-XC, etc.
And optimizing normal clients.
Not easy, but I'd be very curious how much psql's performance improves
when replaying a .sql style dump, and replaying from a !tty fd, after
hacking it up to use the batch API.
Did you consider the use of simple_list.c instead of introducing a new
mimic as PGcommandQueueEntry? It would be cool avoiding adding new
list emulations on frontends.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On 24 May 2016 at 00:00, Michael Paquier <michael.paquier@gmail.com> wrote:
On Mon, May 23, 2016 at 8:50 AM, Andres Freund <andres@anarazel.de>
wrote:This should be very useful for optimising FDWs, Postgres-XC, etc.
And optimizing normal clients.
Not easy, but I'd be very curious how much psql's performance improves
when replaying a .sql style dump, and replaying from a !tty fd, after
hacking it up to use the batch API.
I didn't, but agree it'd be interesting. So would pg_restore, for that
matter, though its use of COPY for the bulk of its work means it wouldn't
make tons of difference.
I think it'd be safe to enable it automatically in psql's
--single-transaction mode. It's also safe to send anything after an
explicit BEGIN and until the next COMMIT as a batch from libpq, and since
it parses the SQL enough to detect statement boundaries already that
shouldn't be too hard to handle.
However: psql is synchronous, using the PQexec family of blocking calls.
It's all fairly well contained in SendQuery and PSQLexec, but making it use
batching still require restructuring those to use the asynchronous
nonblocking API and append the query to a pending-list, plus the addition
of a select() loop to handle results and dispatch more work. MainLoop()
isn't structured around a select or poll, it loops over gets. So while it'd
be interesting to see what difference batching made the changes to make
psql use it would be a bit more invasive. Far from a rewrite, but to avoid
lots of code duplication it'd have to change everything to use nonblocking
mode and a select loop, which is a big change for such a core tool.
This is a bit of a side-project and I've got to get back to "real work" so
I don't expect to do a proper patch for psql any time soon. I'd rather not
try to build too much on this until it's seen some review and I know the
API won't need a drastic rewrite anyway. I'll see if I can do a hacked-up
version one evening to see what it does for performance though.
Did you consider the use of simple_list.c instead of introducing a new
mimic as PGcommandQueueEntry? It would be cool avoiding adding new
list emulations on frontends.
Nope. I didn't realise it was there; I've done very little on the C client
and library side so far. So thanks, I'll update it accordingly.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Re: PATCH: Batch/pipelining support for libpq
On 24 May 2016 at 00:00, Michael Paquier <michael.paquier@gmail.com> wrote:
On Mon, May 23, 2016 at 8:50 AM, Andres Freund <andres@anarazel.de> wrote:
yay^2.
I'll follow this mood. Yeha.
BTW, I've publushed the HTML-ified SGML docs to
http://2ndquadrant.github.io/postgres/libpq-batch-mode.html as a preview.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Re: PATCH: Batch/pipelining support for libpq
On 24 May 2016 at 00:00, Michael Paquier <michael.paquier@gmail.com> wrote:
Did you consider the use of simple_list.c instead of introducing a new
mimic as PGcommandQueueEntry? It would be cool avoiding adding new
list emulations on frontends.
I'd have to extend simple_list to add a generic object version, like
struct my_list_elem
{
PG_SIMPLE_LIST_ATTRS;
mytype mycol;
myothertype myothercol;
}
Objections?
I could add a void* version that's a simple clone of the string version,
but having to malloc both a list cell and its contents separately is
annoying.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Re: PATCH: Batch/pipelining support for libpq
Craig Ringer <craig@2ndquadrant.com> writes:
On 24 May 2016 at 00:00, Michael Paquier <michael.paquier@gmail.com> wrote:
Did you consider the use of simple_list.c instead of introducing a new
mimic as PGcommandQueueEntry? It would be cool avoiding adding new
list emulations on frontends.
I'd have to extend simple_list to add a generic object version, like
struct my_list_elem
{
PG_SIMPLE_LIST_ATTRS;
mytype mycol;
myothertype myothercol;
}
Objections?
That doesn't look exactly "generic".
I could add a void* version that's a simple clone of the string version,
but having to malloc both a list cell and its contents separately is
annoying.
I'd be okay with a void* version, but I'm not sure about this.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On 5/23/16 4:19 AM, Craig Ringer wrote:
+ Batching less useful when information from one operation is required by the
SB "Batching is less useful".
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532) mobile: 512-569-9461
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
From: pgsql-hackers-owner@postgresql.org [mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Craig Ringer
I'll follow this mood. Yeha.
BTW, I've publushed the HTML-ified SGML docs to http://2ndquadrant.github.io/postgres/libpq-batch-mode.html as a preview.
Sorry for my late reply. Fantastic performance improvement, nice documentation, and amazing rapid development! I think I’ll join the review & testing in 2016/9 CF.
Regards
Takayuki Tsunakawa
Re: PATCH: Batch/pipelining support for libpq
2016-05-24 5:01 GMT+03:00 Craig Ringer <craig@2ndquadrant.com>:
On 24 May 2016 at 00:00, Michael Paquier <michael.paquier@gmail.com> wrote:
On Mon, May 23, 2016 at 8:50 AM, Andres Freund <andres@anarazel.de> wrote:
yay^2.
I'll follow this mood. Yeha.
BTW, I've publushed the HTML-ified SGML docs to
http://2ndquadrant.github.io/postgres/libpq-batch-mode.html as a preview.
Typo detected: "Returns 1 if the batch curently being received" -- "curently".
--
// Dmitry.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On Fri, Jun 3, 2016 at 8:51 PM, Dmitry Igrishin <dmitigr@gmail.com> wrote:
BTW, I've publushed the HTML-ified SGML docs to
http://2ndquadrant.github.io/postgres/libpq-batch-mode.html as a preview.Typo detected: "Returns 1 if the batch curently being received" -- "curently".
I am looking a bit more seriously at this patch and assigned myself as
a reviewer.
testlibpqbatch.c:1239:73: warning: format specifies type 'long' but
the argument has type '__darwin_suseconds_t' (aka 'int') [-Wformat]
printf("batch insert elapsed: %ld.%06lds\n",
elapsed_time.tv_sec, elapsed_time.tv_usec);
macos complains here. You may want to replace %06lds by just %06d.
(lldb) bt
* thread #1: tid = 0x0000, 0x0000000108bcdd56
libpq.5.dylib`closePGconn(conn=0x00007fd642d002d0) + 438 at
fe-connect.c:3010, stop reason = signal SIGSTOP
* frame #0: 0x0000000108bcdd56
libpq.5.dylib`closePGconn(conn=0x00007fd642d002d0) + 438 at
fe-connect.c:3010
frame #1: 0x0000000108bc9db0
libpq.5.dylib`PQfinish(conn=0x00007fd642d002d0) + 32 at
fe-connect.c:3072
frame #2: 0x0000000108bc9ede
libpq.5.dylib`PQping(conninfo="dbname=postgres port=5432 host='/tmp'
connect_timeout=5") + 46 at fe-connect.c:539
frame #3: 0x0000000108bb5210
pg_ctl`test_postmaster_connection(pm_pid=78218, do_checkpoint='\0') +
976 at pg_ctl.c:681
frame #4: 0x0000000108bb388e pg_ctl`do_start + 302 at pg_ctl.c:915
frame #5: 0x0000000108bb29b4 pg_ctl`main(argc=5,
argv=0x00007fff5704e5c0) + 2836 at pg_ctl.c:2416
frame #6: 0x00007fff8b8b65ad libdyld.dylib`start + 1
(lldb) down 1
frame #0: 0x0000000108bcdd56
libpq.5.dylib`closePGconn(conn=0x00007fd642d002d0) + 438 at
fe-connect.c:3010
3007 queue = conn->cmd_queue_recycle;
3008 {
3009 PGcommandQueueEntry *prev = queue;
-> 3010 queue = queue->next;
3011 free(prev);
3012 }
3013 conn->cmd_queue_recycle = NULL;
This patch generates a core dump, use for example pg_ctl start -w and
you'll bump into the trace above. There is something wrong with the
queue handling.
Do you have plans for a more generic structure for the command queue list?
+ <application>libpq</application> supports queueing up mulitiple queries into
s/mulitiple/multiple/.
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/examples/libpqbatch.c</filename>.
+ </para>
You mean testlibpqbatch.c here.+ <para>
+ Batching less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
"Batching *is* less useful".src/test/examples/.gitignore needs a new entry for the new test binary.
+ fprintf(stderr, "internal error, COPY in batch mode");
+ abort();
I don't think that's a good idea. defaultNoticeProcessor can be
overridden to allow applications to have error messages sent
elsewhere. Error messages should also use libpq_gettext, and perhaps
be stored in conn->errorMessage as we do so for OOMs happening on
client-side and reporting them back even if they are not expected
(those are blocked PQsendQueryStart in your patch).src/test/examples is a good idea to show people what this new API can
do, but this is never getting compiled. It could as well be possible
to include tests in src/test/modules/, in the same shape as what
postgres_fdw is doing by connecting to itself and link it to libpq. As
this patch complicates quote a lot fe-exec.c, I think that this would
be worth it. Thoughts?
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On 10 August 2016 at 14:44, Michael Paquier <michael.paquier@gmail.com>
wrote:
On Fri, Jun 3, 2016 at 8:51 PM, Dmitry Igrishin <dmitigr@gmail.com> wrote:
BTW, I've publushed the HTML-ified SGML docs to
http://2ndquadrant.github.io/postgres/libpq-batch-mode.html as apreview.
Typo detected: "Returns 1 if the batch curently being received" --
"curently".
I am looking a bit more seriously at this patch and assigned myself as
a reviewer.
Much appreciated.
testlibpqbatch.c:1239:73: warning: format specifies type 'long' but
the argument has type '__darwin_suseconds_t' (aka 'int') [-Wformat]
printf("batch insert elapsed: %ld.%06lds\n",
elapsed_time.tv_sec, elapsed_time.tv_usec);
macos complains here. You may want to replace %06lds by just %06d.
Yeah, or cast to a type known to be big enough. Will amend.
This patch generates a core dump, use for example pg_ctl start -w and
you'll bump into the trace above. There is something wrong with the
queue handling.
Huh. I didn't see that here (Fedora 23). I'll look more closely.
Do you have plans for a more generic structure for the command queue list?
No plans, no. This was a weekend experiment that turned into a useful patch
and I'm having to scrape up time for it amongst much more important things
like logical failover / sequence decoding and various other replication
work.
Thanks for the docs review too, will amend.
+ fprintf(stderr, "internal error, COPY in batch mode"); + abort(); I don't think that's a good idea. defaultNoticeProcessor can be overridden to allow applications to have error messages sent elsewhere. Error messages should also use libpq_gettext, and perhaps be stored in conn->errorMessage as we do so for OOMs happening on client-side and reporting them back even if they are not expected (those are blocked PQsendQueryStart in your patch).src/test/examples is a good idea to show people what this new API can
do, but this is never getting compiled. It could as well be possible
to include tests in src/test/modules/, in the same shape as what
postgres_fdw is doing by connecting to itself and link it to libpq. As
this patch complicates quote a lot fe-exec.c, I think that this would
be worth it. Thoughts?
I didn't think it added much complexity to fe-exec.c personally. A lot of
the appeal is that it has very minor impact on anything that isn't using it.
I think it makes sense to (ab)use the recovery module tests for this,
invoking the test program from there.
Ideally I'd like to teach pgsql and pg_restore how to use async mode, but
that's a whole separate patch.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Re: PATCH: Batch/pipelining support for libpq
On 23 August 2016 at 08:27, Craig Ringer <craig@2ndquadrant.com> wrote:
On 10 August 2016 at 14:44, Michael Paquier <michael.paquier@gmail.com>
wrote:
I am looking a bit more seriously at this patch and assigned myself as
a reviewer.
Much appreciated.
macos complains here. You may want to replace %06lds by just %06d.
Yeah, or cast to a type known to be big enough. Will amend.
I used an upcast to (long), because on Linux it's a long. I don't see the
point of messing about adding a configure test for something this trivial.
This patch generates a core dump, use for example pg_ctl start -w and
you'll bump into the trace above. There is something wrong with the
queue handling.Huh. I didn't see that here (Fedora 23). I'll look more closely.
Do you have plans for a more generic structure for the command queue list?
No plans, no. This was a weekend experiment that turned into a useful
patch and I'm having to scrape up time for it amongst much more important
things like logical failover / sequence decoding and various other
replication work.Thanks for the docs review too, will amend.
+ fprintf(stderr, "internal error, COPY in batch mode"); + abort(); I don't think that's a good idea.
My thinking there was that it's a "shouldn't happen" case. It's a problem
in library logic. I'd use an Assert() here in the backend.
I could printfPQExpBuffer(...) an error and return failure instead if you
think that's more appropriate. I'm not sure how the app would handle it
correctly, but OTOH it's generally better for libraries not to call
abort(). So I'll do that, but since it's an internal error that's not meant
to happen I'll skip the gettext calls.
Error messages should also use libpq_gettext, and perhaps
be stored in conn->errorMessage as we do so for OOMs happening on
client-side and reporting them back even if they are not expected
(those are blocked PQsendQueryStart in your patch).
I didn't get that last part, re PQsendQueryStart.
src/test/examples is a good idea to show people what this new API can
do, but this is never getting compiled. It could as well be possible
to include tests in src/test/modules/, in the same shape as what
postgres_fdw is doing by connecting to itself and link it to libpq. As
this patch complicates quote a lot fe-exec.c, I think that this would
be worth it. Thoughts?
I think it makes sense to use the TAP framework. Added
src/test/modules/test_libpq/ with a test for async mode. Others can be
added/migrated based on that. I thought it made more sense for the tests to
live there than in src/interfaces//libpq/ since they need test client
programs and shouldn't pollute the library directory.
I've made the docs changes too. Thanks.
I fixed the list handling error. I'm amazed it appears to run fine, and
without complaint from valgrind, here, since it was an accidentally
_deleted_ line.
Re lists, I looked at simple_list.c and it's exceedingly primitive. Using
it would mean more malloc()ing since we'll have a list cell and then a
struct pointed to it, and won't recycle members, but... whatever. It's not
going to matter a great deal. The reason I did it with an embedded list
originally was because that's how it's done for PGnotify, but that's not
exactly new code
The bigger problem is that simple_list also uses pg_malloc, which won't set
conn->errorMessage, it'll just fprintf() and exit(1). I'm not convinced
it's appropriate to use that for libpq.
For now I've left list handling unchanged. If it's to move to a generic
list, it should probably be one that knows how to use
pg_malloc_extended(size, MCXT_ALLOC_NO_OOM) and emit its own
libpq-error-handling-aware error. I'm not sure whether that list should use
cell heads embedded in the structures it manages or pointing to them,
either.
Updated patch attached.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Attachments:
0001-Pipelining-batch-support-for-libpq-v2.patchtext/x-patch; charset=US-ASCII; name=0001-Pipelining-batch-support-for-libpq-v2.patchDownload
From fad0def5570907f5c8e3b6d65d57e5e7678e7383 Mon Sep 17 00:00:00 2001
From: Craig Ringer <craig@2ndquadrant.com>
Date: Fri, 20 May 2016 12:45:18 +0800
Subject: [PATCH] Pipelining (batch) support for libpq
Allow libpq clients to avoid excessive round trips by pipelining multiple
commands into batches. A sync is only sent at the end of a batch. Commands
in a batch succeed or fail together.
Adds TAP tests for libpq at src/test/modules/test_libpq .
Includes a test program in src/test/modules/test_libpq/testlibpqbatch.c
---
doc/src/sgml/libpq.sgml | 478 ++++++++
src/interfaces/libpq/.gitignore | 1 +
src/interfaces/libpq/Makefile | 5 +
src/interfaces/libpq/exports.txt | 7 +
src/interfaces/libpq/fe-connect.c | 17 +
src/interfaces/libpq/fe-exec.c | 572 ++++++++-
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 17 +-
src/interfaces/libpq/libpq-fe.h | 13 +-
src/interfaces/libpq/libpq-int.h | 39 +-
src/interfaces/libpq/t/001_libpq_async.pl | 15 +
src/test/examples/.gitignore | 1 +
src/test/examples/Makefile | 2 +-
src/test/modules/test_libpq/.gitignore | 5 +
src/test/modules/test_libpq/Makefile | 23 +
src/test/modules/test_libpq/README | 1 +
src/test/modules/test_libpq/t/001_libpq_async.pl | 26 +
src/test/modules/test_libpq/testlibpqbatch.c | 1386 ++++++++++++++++++++++
18 files changed, 2571 insertions(+), 43 deletions(-)
create mode 100644 src/interfaces/libpq/t/001_libpq_async.pl
create mode 100644 src/test/modules/test_libpq/.gitignore
create mode 100644 src/test/modules/test_libpq/Makefile
create mode 100644 src/test/modules/test_libpq/README
create mode 100644 src/test/modules/test_libpq/t/001_libpq_async.pl
create mode 100644 src/test/modules/test_libpq/testlibpqbatch.c
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index f22e3da..c565e18 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4568,6 +4568,484 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up multiple queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It somewhat increased client application
+ complexity and extra caution is required to prevent client/server network
+ deadlocks, but can offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ ("ping time") is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 9.6, but clients using it can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ <application>libpq</application> into batch mode. Enter batch mode with <link
+ linkend="libpq-pqbeginbatchmode"><function>PQbeginBatchMode(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-pqisinbatchmode"><function>PQisInBatchMode(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not allowed. (The restriction on <literal>COPY</literal> is an implementation
+ limit; the PostgreSQL protocol and server can support batched <literal>COPY</literal>).
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQsendEndBatch</function>.
+ Concurrently, it uses <function>PQgetResult</function> and
+ <function>PQgetNextQuery</function> to get results. It may eventually exit
+ batch mode with <function>PQendBatchMode</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it's already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions like
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ etc. The asynchronous requests are followed by a <link
+ linkend="libpq-pqsendendbatch"><function>PQsendEndBatch(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server usually begins executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing with sending batch queries</link>, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-pqgetnextquery"><function>PQgetNextQuery</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null (or would return null if
+ called). The result from the next batch entry may then be retrieved using
+ <function>PQgetNextQuery</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQsendEndBatch</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQsendEndBatch</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetNextQuery(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-pqendbatchmode"><function>PQendBatchMode(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-pqisinbatchmode">
+ <term>
+ <function>PQisInBatchMode</function>
+ <indexterm>
+ <primary>PQisInBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns 1 if a <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>, otherwise 0.
+
+<synopsis>
+int PQisInBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqbeginbatchmode">
+ <term>
+ <function>PQbeginBatchMode</function>
+ <indexterm>
+ <primary>PQbeginBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode and returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+<synopsis>
+int PQbeginBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqendbatchmode">
+ <term>
+ <function>PQendBatchMode</function>
+ <indexterm>
+ <primary>PQendBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results and returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQgetNextQuery</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+<synopsis>
+int PQendBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqsendendbatch">
+ <term>
+ <function>PQsendEndBatch</function>
+ <indexterm>
+ <primary>PQsendEndBatch</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQsendEndBatch(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqgetnextquery">
+ <term>
+ <function>PQgetNextQuery</function>
+ <indexterm>
+ <primary>PQgetNextQuery</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results. Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. See <link
+ linkend="libpq-batch-results">processing results</link>.
+
+<synopsis>
+int PQgetNextQuery(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqqueriesinbatch">
+ <term>
+ <function>PQqueriesInBatch</function>
+ <indexterm>
+ <primary>PQqueriesInBatch</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processsed.
+ This is the number of times <function>PQgetNextQuery</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQqueriesInBatch(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqbatchisaborted">
+ <term>
+ <function>PQbatchIsAborted</function>
+ <indexterm>
+ <primary>PQbatchIsAborted</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns 1 if the batch curently being received on a
+ <application>libpq</application> connection in <link
+ linkend="libpq-batch-mode">batch mode</link> is
+ <link linkend="libpq-batch-errors">aborted</link>, 0
+ otherwise. The aborted flag is cleared as soon as the result of the
+ <function>PQsendEndBatch</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function as they can tell
+ that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+
+<synopsis>
+int PQbatchIsAborted(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
diff --git a/src/interfaces/libpq/.gitignore b/src/interfaces/libpq/.gitignore
index cb96af7..4c0d934 100644
--- a/src/interfaces/libpq/.gitignore
+++ b/src/interfaces/libpq/.gitignore
@@ -22,3 +22,4 @@
/encnames.c
/wchar.c
/libpq.rc
+/tmp_check
diff --git a/src/interfaces/libpq/Makefile b/src/interfaces/libpq/Makefile
index 0b4065e..573b265 100644
--- a/src/interfaces/libpq/Makefile
+++ b/src/interfaces/libpq/Makefile
@@ -129,6 +129,11 @@ install: all installdirs install-lib
installcheck:
$(MAKE) -C test $@
+check: prove-check
+
+prove-check:
+ $(prove_check)
+
installdirs: installdirs-lib
$(MKDIR_P) '$(DESTDIR)$(includedir)' '$(DESTDIR)$(includedir_internal)' '$(DESTDIR)$(datadir)'
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 21dd772..1b0b8c5 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -171,3 +171,10 @@ PQsslAttributeNames 168
PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
+PQisInBatchMode 172
+PQqueriesInBatch 173
+PQbeginBatchMode 174
+PQendBatchMode 175
+PQsendEndBatch 176
+PQgetNextQuery 177
+PQbatchIsAborted 178
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 76b61bd..8725dc6 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -2949,6 +2949,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
/*
@@ -2995,6 +2996,22 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+ queue = queue->next;
+ free(prev);
+ }
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+ queue = conn->cmd_queue_recycle;
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+ queue = queue->next;
+ free(prev);
+ }
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index d1b91c8..023edab 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -69,6 +71,9 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry* PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry);
/* ----------------
@@ -1107,7 +1112,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1130,6 +1135,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->in_batch)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1228,9 +1240,29 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
+ if (conn->in_batch)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
/* check the arguments */
if (!stmtName)
{
@@ -1286,18 +1318,21 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (!conn->in_batch)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1307,10 +1342,14 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->in_batch)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1357,6 +1396,81 @@ PQsendQueryPrepared(PGconn *conn,
resultFormat);
}
+/* Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry*
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry*) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/* Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/* Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, "tried to recycle non-dangling command queue entry");
+ abort();
+ }
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/* Set up for processing a new query's results */
+static void
+PQstartProcessingNewQuery(PGconn *conn)
+{
+ /* initialize async result-accumulation state */
+ conn->result = NULL;
+ conn->next_result = NULL;
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+}
+
/*
* Common startup code for PQsendQuery and sibling routines
*/
@@ -1376,20 +1490,52 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && !conn->in_batch)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- conn->result = NULL;
- conn->next_result = NULL;
-
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ if (conn->in_batch)
+ {
+ /* When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQgetNextQuery(...) advances
+ * to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately */
+ PQstartProcessingNewQuery(conn);
+ }
/* ready to send command message */
return true;
@@ -1414,6 +1560,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1423,6 +1573,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->in_batch)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1535,22 +1702,25 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (!conn->in_batch)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1560,10 +1730,15 @@ PQsendQueryGuts(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->in_batch)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
+
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1690,6 +1865,282 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/* PQisInBatchMode
+ * Return true if currently in batch mode
+ */
+int
+PQisInBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->in_batch;
+}
+
+/* PQqueriesInBatch
+ * Return true if there are queries currently pending in batch mode
+ */
+int
+PQqueriesInBatch(PGconn *conn)
+{
+ if (!PQisInBatchMode(conn))
+ return false;
+
+ return conn->cmd_queue_head != NULL;
+}
+
+/* PQbatchIsAborted
+ * Batch being processed is aborted, results discarded until next sync
+ */
+int
+PQbatchIsAborted(PGconn *conn)
+{
+ if (!PQisInBatchMode(conn))
+ return false;
+
+ return conn->batch_aborted;
+}
+
+/* Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * COPY is not permitted in batch mode.
+ *
+ * A set of commands is terminated by a PQsendEndBatch. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQendBatchMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQbeginBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->in_batch)
+ return true;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return false;
+
+ conn->in_batch = true;
+ conn->batch_aborted = false;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return true;
+}
+
+/* End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns true if batch mode ended.
+ */
+int
+PQendBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (!conn->in_batch)
+ return true;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ return false;
+ case PGASYNC_QUEUED:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ return false;
+
+ conn->in_batch = false;
+ conn->batch_aborted = false;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ return true;
+}
+
+/* End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new non-batch commands until all
+ * results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQsendEndBatch command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQsendEndBatch(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return false;
+
+ if (!conn->in_batch)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /* Should try to flush immediately if there's room */
+ PQflush(conn);
+
+ return true;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/* PQgetNextQuery
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns true if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns false if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQgetNextQuery(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return FALSE;
+
+ if (!conn->in_batch)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /* In batch mode but nothing left on the queue; caller can submit
+ * more work or PQendBatchMode() now. */
+ return false;
+ }
+
+ /* Pop the next query from the queue and set up the connection state
+ * as if it'd just been dispatched from a non-batched call */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ PQstartProcessingNewQuery(conn);
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_aborted && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+
+ /* Parse any available data */
+ parseInput(conn);
+ }
+
+ return true;
+}
+
/*
* PQgetResult
@@ -1697,7 +2148,6 @@ PQisBusy(PGconn *conn)
* query work remains or an error has occurred (e.g. out of
* memory).
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1749,10 +2199,31 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->in_batch)
+ {
+ /* batched queries aren't followed by a Sync to put us back in
+ * PGASYNC_IDLE state, and when we do get a sync we could still
+ * have another batch coming after this one.
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1932,6 +2403,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->in_batch)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQexec in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2126,6 +2604,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2141,6 +2622,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->in_batch)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2149,15 +2644,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (!conn->in_batch)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && !conn->in_batch)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2171,10 +2669,14 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->in_batch)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index f1b90f3..e824f36 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->in_batch)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index a8a987a..78ffc1e 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->in_batch)
+ {
+ conn->batch_aborted = false;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -305,7 +313,7 @@ pqParseInput3(PGconn *conn)
* parsing until the application accepts the current
* result.
*/
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
return;
}
break;
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->in_batch)
+ conn->batch_aborted = true;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 9ca0756..a05298f 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -91,7 +91,9 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort earlier in a batch */
} ExecStatusType;
typedef enum
@@ -421,6 +423,15 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQisInBatchMode(PGconn *conn);
+extern int PQbatchIsAborted(PGconn *conn);
+extern int PQqueriesInBatch(PGconn *conn);
+extern int PQbeginBatchMode(PGconn *conn);
+extern int PQendBatchMode(PGconn *conn);
+extern int PQsendEndBatch(PGconn *conn);
+extern int PQgetNextQuery(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 1183323..0954239 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,10 +217,13 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch result,
+ More results expected from this query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -229,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -292,6 +296,22 @@ typedef struct pgDataValue
const char *value; /* data value, without zero-termination */
} PGdataValue;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* PGconn stores all the state data associated with a single connection
* to a backend.
@@ -356,6 +376,8 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ bool in_batch; /* connection is in batch (pipelined) mode */
+ bool batch_aborted; /* current batch is aborted, discarding until next Sync */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY
@@ -363,6 +385,15 @@ struct pg_conn
PGnotify *notifyHead; /* oldest unreported Notify msg */
PGnotify *notifyTail; /* newest unreported Notify msg */
+ /* The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
/* See PQconnectPoll() for how we use 'int' and not 'pgsocket'. */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
@@ -658,6 +689,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
diff --git a/src/interfaces/libpq/t/001_libpq_async.pl b/src/interfaces/libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000..d8f153e
--- /dev/null
+++ b/src/interfaces/libpq/t/001_libpq_async.pl
@@ -0,0 +1,15 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+$node->stop('fast');
diff --git a/src/test/examples/.gitignore b/src/test/examples/.gitignore
index 1957ec1..7a72420 100644
--- a/src/test/examples/.gitignore
+++ b/src/test/examples/.gitignore
@@ -4,3 +4,4 @@
/testlibpq4
/testlo
/testlo64
+/testlibpqbatch
diff --git a/src/test/examples/Makefile b/src/test/examples/Makefile
index 31da210..92a6faf 100644
--- a/src/test/examples/Makefile
+++ b/src/test/examples/Makefile
@@ -14,7 +14,7 @@ override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
override LDLIBS := $(libpq_pgport) $(LDLIBS)
-PROGS = testlibpq testlibpq2 testlibpq3 testlibpq4 testlo testlo64
+PROGS = testlibpq testlibpq2 testlibpq3 testlibpq4 testlo testlo64 testlibpqbatch
all: $(PROGS)
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000..11e8463
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000..7f1b8b0
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,23 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+check: prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000..d8174dd
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000..28c3eb5
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,26 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 5;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_batch simple_batch multi_batch batch_abort timings);
+
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(['testlibpqbatch', 'dbname=postgres', "$numrows", "$testname"],
+ "testlibpqbatch $testname");
+}
+
+$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000..2dc485d
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1386 @@
+/*
+ * src/test/examples/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution funtionality
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+
+#ifndef VERBOSE
+#define VERBOSE 0
+#endif
+
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+ fflush(stderr);
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode\n");
+ goto fail;
+ }
+
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQendBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode with work in progress should fail, but succeeded\n");
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending a batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * in batch mode we have to ask for the first result to be processed;
+ * until we do PQgetResult will return null:
+ */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQgetNextQuery() call\n");
+ goto fail;
+ }
+
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* We can't PQgetNextQuery when there might still be pending results */
+ if (PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() should've failed with pending results: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQgetNextQuery() call\n");
+ goto fail;
+ }
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQendBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ /* should now get an explicit sync result */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after end batch call\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+ fflush(stderr);
+
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode: %u\n", __LINE__);
+ goto fail;
+ }
+
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "Unable to enter batch mode\n");
+ goto fail;
+ }
+
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Batch mode not activated properly\n");
+ goto fail;
+ }
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "PQexec should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ {
+ fprintf(stderr, "PQsendQuery should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "re-entering batch mode should be a no-op but failed\n");
+ goto fail;
+ }
+
+ if (PQisBusy(conn))
+ {
+ fprintf(stderr, "PQisBusy should return false when idle in batch, returned true\n");
+ goto fail;
+ }
+
+ /* ok, back to normal command mode */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "couldn't exit idle empty batch mode\n");
+ goto fail;
+ }
+
+ if (PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Batch mode not terminated properly\n");
+ goto fail;
+ }
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "batch mode exit when not in batch mode should succeed but failed\n");
+ goto fail;
+ }
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+ fflush(stderr);
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* OK, start processing the batch results */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQgetNextQuery() call\n");
+ goto fail;
+ }
+
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQgetNextQuery() call\n");
+ goto fail;
+ }
+
+ if (PQendBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ /* second batch */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+ fflush(stderr);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "1";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching error select failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "2";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "3";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* second query, caused error */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (!PQbatchIsAborted(conn))
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* third query in batch, the second insert */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at third batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (!PQbatchIsAborted(conn))
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the batch sync */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchIsAborted(conn))
+ {
+ fprintf(stderr, "sync should've cleared the aborted flag but didn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the insert from the second batch */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first entry in second batch: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first item in second batch\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* the second batch sync */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch sync\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ {
+ fprintf(stderr, "expected only insert with value 3, got %s", val);
+ goto fail;
+ }
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ fprintf(stderr, "expected 1 result, got %d", PQntuples(res));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "xact start failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "sent BEGIN\n");
+#endif
+ send_step = BI_DROP_TABLE;
+
+ if (!PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "sent DROP\n");
+#endif
+ send_step = BI_CREATE_TABLE;
+
+ if (!PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "sent CREATE\n");
+#endif
+ send_step = BI_PREPARE;
+
+ if (!PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids))
+ {
+ fprintf(stderr, "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "sent PREPARE\n");
+#endif
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ {
+ fprintf(stderr, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "Expected next query result but unable to dequeue: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "next query!\n");
+#endif
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+#if VERBOSE
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+#endif
+
+ if (PQresultStatus(res) != status)
+ {
+ fprintf(stderr, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ {
+ fprintf(stderr, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+#endif
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+#if VERBOSE
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+#endif
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+#if VERBOSE
+ fprintf(stdout, "sent COMMIT\n");
+#endif
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQsendEndBatch(conn))
+ {
+#if VERBOSE
+ fprintf(stdout, "Dispatched end batch message\n");
+#endif
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ {
+ fprintf(stderr, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "prepare failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "COPY: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[MAXINTLEN+2];
+ int formatted = snprintf(&buf[0], MAXINTLEN+1, "%d\n", nrows);
+ if (formatted >= MAXINTLEN+1)
+ {
+ fprintf(stderr, "Buffer write truncated somehow\n");
+ goto fail;
+ }
+
+ if (PQputCopyData(conn, buf, formatted) != 1)
+ {
+ fprintf(stderr, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ nrows--;
+ }
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ struct timeval start_time,
+ end_time,
+ elapsed_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_pipelined(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("batch insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long)elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_sequential(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("sequential insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long)elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_copy(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("COPY elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long)elapsed_time.tv_usec);
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings\n");
+ exit(1);
+}
+
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognised test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
--
2.5.5
Re: PATCH: Batch/pipelining support for libpq
Craig Ringer wrote:
Updated patch attached.
Please find attached a couple fixes for typos I've came across in
the doc part.
Also it appears that PQqueriesInBatch() doesn't work as documented.
It says:
"Returns the number of queries still in the queue for this batch"
but in fact it's implemented as a boolean:
+/* PQqueriesInBatch
+ * Return true if there are queries currently pending in batch mode
+ */+int
+PQqueriesInBatch(PGconn *conn)
+{
+ if (!PQisInBatchMode(conn))
+ return false;
+
+ return conn->cmd_queue_head != NULL;
+}However, is this function really needed? It doesn't seem essential to
the API. You don't call it in the test program either.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
Attachments:
diff-typos.txttext/plain; name=diff-typos.txtDownload
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 73c6c03..af4f922 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4653,7 +4653,7 @@ int PQflush(PGconn *conn);
</programlisting>
could be much more efficiently done with:
<programlisting>
- UPDATE mytable SET x = x + 1;
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
</programlisting>
</para>
@@ -4696,7 +4696,7 @@ int PQflush(PGconn *conn);
<link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
blocking mode it is possible for a client/server deadlock to occur. The
client will block trying to send queries to the server, but the server will
- block trying to send results from queries it's already processed to the
+ block trying to send results from queries it has already processed to the
client. This only occurs when the client sends enough queries to fill its
output buffer and the server's receive buffer before switching to
processing input from the server, but it's hard to predict exactly when
@@ -5015,7 +5015,7 @@ int PQgetNextQuery(PGconn *conn);
<listitem>
<para>
Returns the number of queries still in the queue for this batch, not
- including any query that's currently having results being processsed.
+ including any query that's currently having results being processed.
This is the number of times <function>PQgetNextQuery</function> has to be
called before the query queue is empty again.
@@ -5037,7 +5037,7 @@ int PQqueriesInBatch(PGconn *conn);
<listitem>
<para>
- Returns 1 if the batch curently being received on a
+ Returns 1 if the batch currently being received on a
<application>libpq</application> connection in <link
linkend="libpq-batch-mode">batch mode</link> is
<link linkend="libpq-batch-errors">aborted</link>, 0
Re: PATCH: Batch/pipelining support for libpq
On 6 September 2016 at 16:10, Daniel Verite <daniel@manitou-mail.org> wrote:
Craig Ringer wrote:
Updated patch attached.
Please find attached a couple fixes for typos I've came across in
the doc part.
Thanks, will apply and post a rebased patch soon, or if someone picks
this up in the mean time they can apply your diff on top of the patch.
Also it appears that PQqueriesInBatch() doesn't work as documented.
It says:
"Returns the number of queries still in the queue for this batch"
but in fact it's implemented as a boolean:
Whoops. Will fix.
I think the function is useful and necessary. There's no reason not to
expose that, but also it's good for when your query dispatch isn't as
tightly coupled to your query handling as in the example, so your app
might need to keep processing until it sees the end of queued results.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On Tue, Sep 6, 2016 at 8:01 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
On 6 September 2016 at 16:10, Daniel Verite <daniel@manitou-mail.org> wrote:
Craig Ringer wrote:
Updated patch attached.
Please find attached a couple fixes for typos I've came across in
the doc part.Thanks, will apply and post a rebased patch soon, or if someone picks
this up in the mean time they can apply your diff on top of the patch.
Could you send an updated patch then? At the same time I am noticing
that git --check is complaining... This patch has tests and a
well-documented feature, so I'll take a look at it soon at the top of
my list. Moved to next CF for now.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On 3 October 2016 at 10:10, Michael Paquier <michael.paquier@gmail.com> wrote:
On Tue, Sep 6, 2016 at 8:01 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
On 6 September 2016 at 16:10, Daniel Verite <daniel@manitou-mail.org> wrote:
Craig Ringer wrote:
Updated patch attached.
Please find attached a couple fixes for typos I've came across in
the doc part.Thanks, will apply and post a rebased patch soon, or if someone picks
this up in the mean time they can apply your diff on top of the patch.Could you send an updated patch then? At the same time I am noticing
that git --check is complaining... This patch has tests and a
well-documented feature, so I'll take a look at it soon at the top of
my list. Moved to next CF for now.
Thanks.
I'd really like to teach psql in non-interactive mode to use it, but
(a) I'm concerned about possible subtle behaviour differences arising
if we do that and (b) I won't have the time. I think it's mostly of
interest to app authors and driver developers and that's what it's
aimed at. pg_restore could benefit a lot too.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
Craig Ringer wrote:
I think it's mostly of interest to app authors and driver developers
and that's what it's aimed at. pg_restore could benefit a lot too.
Wouldn't pgbench benefit from it?
It was mentioned some time ago [1]/messages/by-id/alpine.DEB.2.20.1607140925400.1962@sto, in relationship to the
\into construct, how client-server latency was important enough to
justify the use of a "\;" separator between statements, to send them
as a group.
But with the libpq batch API, maybe this could be modernized
with meta-commands like this:
\startbatch
...
\endbatch
which would essentially call PQbeginBatchMode() and PQsendEndBatch().
Inside the batch section, collecting results would be interleaved
with sending queries.
Interdepencies between results and subsequent queries could
be handled or ignored, depending on how sophisticated we'd want
this.
This might also draw more users to the batch API, because
it would make it easier to check how exactly it affects
the performance of specific sequences of SQL statements to be
grouped in a batch.
For instance it would make sense for programmers to benchmark mock-ups
of their code with pgbench with/without batching, before embarking on
adapting it from blocking mode to asynchronous/non-blocking mode.
[1]: /messages/by-id/alpine.DEB.2.20.1607140925400.1962@sto
/messages/by-id/alpine.DEB.2.20.1607140925400.1962@sto
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On Mon, Oct 3, 2016 at 11:52 PM, Daniel Verite <daniel@manitou-mail.org> wrote:
Wouldn't pgbench benefit from it?
It was mentioned some time ago [1], in relationship to the
\into construct, how client-server latency was important enough to
justify the use of a "\;" separator between statements, to send them
as a group.But with the libpq batch API, maybe this could be modernized
with meta-commands like this:
\startbatch
...
\endbatch
Or just \batch [on|off], which sounds like a damn good idea to me for
some users willing to test some workloads before integrating it in an
application.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On 4 Oct. 2016 15:15, "Michael Paquier" <michael.paquier@gmail.com> wrote:
On Mon, Oct 3, 2016 at 11:52 PM, Daniel Verite <daniel@manitou-mail.org>
wrote:
Wouldn't pgbench benefit from it?
It was mentioned some time ago [1], in relationship to the
\into construct, how client-server latency was important enough to
justify the use of a "\;" separator between statements, to send them
as a group.But with the libpq batch API, maybe this could be modernized
with meta-commands like this:
\startbatch
...
\endbatchOr just \batch [on|off], which sounds like a damn good idea to me for
some users willing to test some workloads before integrating it in an
application.
A batch jsnt necessarily terminated by a commit, so I'm more keen on
start/end batch. It's more in line with begin/commit. Batch is not only a
mode, you also have to delineate batches.
Re: PATCH: Batch/pipelining support for libpq
On 04/10/16 20:15, Michael Paquier wrote:
On Mon, Oct 3, 2016 at 11:52 PM, Daniel Verite <daniel@manitou-mail.org> wrote:
Wouldn't pgbench benefit from it?
It was mentioned some time ago [1], in relationship to the
\into construct, how client-server latency was important enough to
justify the use of a "\;" separator between statements, to send them
as a group.But with the libpq batch API, maybe this could be modernized
with meta-commands like this:
\startbatch
...
\endbatchOr just \batch [on|off], which sounds like a damn good idea to me for
some users willing to test some workloads before integrating it in an
application.
+1
'\batch' is a bit easier, to find, & to remember than '\startbatch'
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On Mon, Oct 3, 2016 at 12:48 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
On 3 October 2016 at 10:10, Michael Paquier <michael.paquier@gmail.com> wrote:
On Tue, Sep 6, 2016 at 8:01 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
On 6 September 2016 at 16:10, Daniel Verite <daniel@manitou-mail.org> wrote:
Craig Ringer wrote:
Updated patch attached.
Please find attached a couple fixes for typos I've came across in
the doc part.Thanks, will apply and post a rebased patch soon, or if someone picks
this up in the mean time they can apply your diff on top of the patch.Could you send an updated patch then? At the same time I am noticing
that git --check is complaining... This patch has tests and a
well-documented feature, so I'll take a look at it soon at the top of
my list. Moved to next CF for now.Thanks.
I'd really like to teach psql in non-interactive mode to use it, but
(a) I'm concerned about possible subtle behaviour differences arising
if we do that and (b) I won't have the time. I think it's mostly of
interest to app authors and driver developers and that's what it's
aimed at. pg_restore could benefit a lot too.
Looking at it now... The most interesting comments are first.
I wanted to congratulate you. I barely see a patch with this level of
details done within the first versions. Anybody can review the patch
just by looking at the code and especially the docs without looking at
the thread. There are even tests to show what this does, for the
client.
+PQisInBatchMode 172
+PQqueriesInBatch 173
+PQbeginBatchMode 174
+PQendBatchMode 175
+PQsendEndBatch 176
+PQgetNextQuery 177
+PQbatchIsAborted 178
This set of routines is a bit inconsistent. Why not just prefixing
them with PQbatch? Like that for example:
PQbatchStatus(): consists of disabled/inactive/none, active, error.
This covers both PQbatchIsAborted() and PQisInBatchMode().
PQbatchBegin()
PQbatchEnd()
PQbatchQueueSync() or PQbatchQueueFlush, same as PQsendEndBatch() to
add and process a sync message into the queue.
PQbatchQueueCount(): returns N>0 if there are N entries, 0 if empty,
-1 on failure
PQbatchQueueProcess(): returns 1 if process can begin, 0 if not, -1 on
failure (OOM)
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It somewhat increased client application
+ complexity and extra caution is required to prevent client/server network
+ deadlocks, but can offer considerable performance improvements.
+ </para>
I would reword that a bit "it increases client application complexity
and extra caution is required to prevent client/server deadlocks but
offers considerable performance improvements".+ ("ping time") is high, and when many small operations are being
performed in
Nit: should use <quote> here. Still not quoting it would be fine.
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions like
+ <function>PQsendQueryParams</function>,
<function>PQsendPrepare</function>,
+ etc. The asynchronous requests are followed by a <link
It may be better to list all the functions here, PQSendQuery * query work remains or an error has occurred (e.g. out of
* memory).
*/
-
PGresult *
PQgetResult(PGconn *conn)
Some noise in the patch.
git diff --check complains:
usage_exit(argv[0]);
warning: 1 line adds whitespace errors.
+++ b/src/interfaces/libpq/t/001_libpq_async.pl
@@ -0,0 +1,15 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+$node->stop('fast');
Er... This does nothing..The changes in src/test/examples/ are not necessary anymore. You moved
all the tests to test_libpq (for the best actually).
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+ queue = queue->next;
+ free(prev);
+ }
This should free prev->query./* blah comment
* blah2 comment
*/
A lot of comment blocks are like that, those should be reformated.
Running directly make check in src/test/modules/test_libpq/ does not work:
# Postmaster PID for node "main" is 10225
# Running: testlibpqbatch dbname=postgres 10000 disallowed_in_batch
Command 'testlibpqbatch' not found in [...PATH list ...]
The problem is that testlibpqbatch is not getting compiled but I think
it should.
+ * testlibpqbatch.c
+ * Test of batch execution funtionality
[...]
+ fprintf(stderr, "%s is not a recognised test name\n", argv[3]);
s/funtionality/functionality/
s/recognised/recognized/+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first SELECT failed: %s\n",
PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n",
PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second SELECT failed: %s\n",
PQerrorMessage(conn));
+ goto fail;
+ }
May be better to use a loop here and set in the queue a bunch of queries..You could just remove the VERBOSE flag in the tests, having a test
more talkative is always better.
+ <para>
+ The batch API was introduced in PostgreSQL 9.6, but clients using it can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
Postgres 10, not 9.6.It may be a good idea to check for PG_PROTOCOL_MAJOR < 3 and issue an
error for the new routines.
+int
+PQgetNextQuery(PGconn *conn)
[...]
+ return false;
The new routines had better return explicitly 0 or 1, not a boolean
for consistency with the rest.+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. See <link
It would be useful for the user to make a difference between those
different statuses. As of your patch, PQgetNextQuery() returns false/0
in all those cases so it is a bit hard to know what's going on. Maybe
PQgetNextQuery() could be renamed to PQbatchGetNext, PQbatchQueueNext
to outline the fact that it is beginning the next query in the queue.
This helps also to understand that this can just be used with the
batch mode. Actually no, I am wrong. It is possible to guess each one
of those errors with respectively PQgetResult == NULL,
PQisInBatchMode() and PQqueriesInBatch(). Definitely it should be
mentioned in the docs that it is possible to make a difference between
all those states.+ entry = PQmakePipelinedCommand(conn);
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
PQmakePipelinedCommand() returns NULL, and boom.+ bool in_batch; /* connection is in batch (pipelined) mode */
+ bool batch_aborted; /* current batch is aborted,
discarding until next Sync */
Having only one flag would be fine. batch_aborted is set of used only
when in_batch is used, so both have a strong link. /* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->in_batch)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
No, it is put in the queue.+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processsed.
+ This is the number of times <function>PQgetNextQuery</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQqueriesInBatch(PGconn *conn);
+</synopsis>
This is not true. It does not return a count, just 0 or 1.It may be a good idea to add a test for COPY and trigger a failure.
If I read the code correctly, it seems to me that it is possible to
enable the batch mode, and then to use PQexec(), PQsendPrepare will
just happily process queue the command. Shouldn't PQexec() be
prevented in batch mode? Similar remark for PQexecParams(),
PQexecPrepared() PQdescribePrepared and PQprepare(). In short
everything calling PQexecFinish().
I haven't run the tests directly on Windows wiht MSVC, but they won't
run as vcregress modulescheck lacks knowledge about modules using TAP.
I cannot blame you for that..
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
Hi all. I thought I'd share some experience from Npgsql regarding
batching/pipelining - hope this isn't off-topic.
Npgsql has supported batching for quite a while, similar to what this patch
proposes - with a single Sync message is sent at the end.
It has recently come to my attention that this implementation is
problematic because it forces the batch to occur within a transaction; in
other words, there's no option for a non-transactional batch. This can be a
problem for several reasons: users may want to sent off a batch of inserts,
not caring whether one of them fails (e.g. because of a unique constraint
violation). In other words, in some scenarios it may be appropriate for
later batched statements to be executed when an earlier batched statement
raised an error. If Sync is only sent at the very end, this isn't possible.
Another example of a problem (which actually happened) is that transactions
acquire row-level locks, and so may trigger deadlocks if two different
batches update the same rows in reverse order. Both of these issues
wouldn't occur if the batch weren't implicitly batched.
My current plan is to modify the batch implementation based on whether
we're in an (explicit) transaction or not. If we're in a transaction, then
it makes perfect sense to send a single Sync at the end as is being
proposed here - any failure would cause the transaction to fail anyway, so
skipping all subsequent statements until the batch's end makes sense.
However, if we're not in an explicit transaction, I plan to insert a Sync
message after each individual Execute, making non-transactional batched
statements more or less identical in behavior to non-transactional
unbatched statements. Note that this mean that a batch can generate
multiple errors, not just one.
I'm sharing this since it may be relevant to the libpq batching
implementation as well, and also to get any feedback regarding how Npgsql
should act.
Re: PATCH: Batch/pipelining support for libpq
On 10/4/16 11:54 PM, Michael Paquier wrote:
+ <para> + Much like asynchronous query mode, there is no performance disadvantage to + using batching and pipelining. It somewhat increased client application + complexity and extra caution is required to prevent client/server network + deadlocks, but can offer considerable performance improvements. + </para> I would reword that a bit "it increases client application complexity and extra caution is required to prevent client/server deadlocks but offers considerable performance improvements".
Unrelated, but another doc bug, on line 4647:
+ The batch API was introduced in PostgreSQL 9.6, but clients
using it can
That should read 10.0 (or just 10?)
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532) mobile: 512-569-9461
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On 12 October 2016 at 19:51, Shay Rojansky <roji@roji.org> wrote:
Hi all. I thought I'd share some experience from Npgsql regarding
batching/pipelining - hope this isn't off-topic.
Not at all.
Npgsql has supported batching for quite a while, similar to what this patch
proposes - with a single Sync message is sent at the end.
PgJDBC does too. The benefits of it there are what prompted me to do
this, not only so libpq users can use it directly but so psqlODBC,
psycopg2, etc can benefit from it if they choose to expose it.
It has recently come to my attention that this implementation is problematic
because it forces the batch to occur within a transaction; in other words,
there's no option for a non-transactional batch.
That's not strictly the case. If you explicitly BEGIN and COMMIT,
those operations are honoured within the batch.
What's missing is implicit transactions. Usually if you send a series
of messages they will each get their own implicit transaction. If you
batch them, the whole lot gets an implicit transaction. This is
because the PostgreSQL v3 protocol conflates transaction delineation
with protocol error recovery into a single Sync message type.
It's very similar to the behaviour of multi-statements, where:
psql -c "CREATE TABLE fred(x integer); DROP TABLE notexists;"
doesn't create "fred", but
psql -c "BEGIN; CREATE TABLE fred(x integer); COMMIT; BEGIN; DROP
TABLE notexists; COMMMIT;"
... does. And in fact it's for almost the same reason. They're sent as
a single SimpleQuery message by psql and split up client side, but the
effect is the same as two separate Query messages followed by a Sync.
It isn't simple to manage this client-side, because libpq doesn't know
whether a given command string may contain transaction delimiting
statements or not and can't reliably look for them without client-side
parsing of the SQL. So it can't dispatch its own BEGIN/COMMIT around
statements in a batch that it thinks might be intended to be
autocommit, and anyway that'd result in sending 3 queries for every 1
client query, which would suck.
If the mythical v4 protocol ever happens I'd want to split Sync into
two messages, one which is a protocol synchronisation message and
another that is a transaction delimiter. Or give it flags or whatever.
In the mean time:
This can be a problem for
several reasons: users may want to sent off a batch of inserts, not caring
whether one of them fails (e.g. because of a unique constraint violation).
In other words, in some scenarios it may be appropriate for later batched
statements to be executed when an earlier batched statement raised an error.
If Sync is only sent at the very end, this isn't possible.
Right, and that remains the case even with explicit transaction
delineation, because the first ERROR causes processing of all
subsequent messages to be skipped.
The design I have in libpq allows for this by allowing the client to
delimit batches without ending batch mode, concurrently consuming a
stream of multiple batches. Each endbatch is a Sync. So a client that
wants autocommit-like behavour can send a series of 1-query batches.
I think I'll need to document this a bit better since it's more subtle
than I properly explained.
Another example
of a problem (which actually happened) is that transactions acquire
row-level locks, and so may trigger deadlocks if two different batches
update the same rows in reverse order. Both of these issues wouldn't occur
if the batch weren't implicitly batched.
Same solution as above.
My current plan is to modify the batch implementation based on whether we're
in an (explicit) transaction or not. If we're in a transaction, then it
makes perfect sense to send a single Sync at the end as is being proposed
here - any failure would cause the transaction to fail anyway, so skipping
all subsequent statements until the batch's end makes sense. However, if
we're not in an explicit transaction, I plan to insert a Sync message after
each individual Execute, making non-transactional batched statements more or
less identical in behavior to non-transactional unbatched statements. Note
that this mean that a batch can generate multiple errors, not just one.
Yes, that's what I suggest, and basically what the libpq batch
interface does, though it expects the client to deal with the
transaction boundaries.
You will need to think hard about transaction boundaries as they
relate to multi-statements unless nPgSQL parses out each statement
from multi-statement strings like PgJDBC does. Otherwise a user can
sneak in:
somestatement; BEGIN; someotherstatement;
or
somestatement; CoMMiT; otherstatement;
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
It has recently come to my attention that this implementation is
problematic
because it forces the batch to occur within a transaction; in other
words,
there's no option for a non-transactional batch.
That's not strictly the case. If you explicitly BEGIN and COMMIT,
those operations are honoured within the batch.
I wasn't precise in my formulation (although I think we understand each
other)... The problem I'm trying to address here is the fact that in the
"usual" batching implementation (i.e. where a single Sync message is sent
at the end of the batch), there's no support for batches which have no
transactions whatsoever (i.e. where each statement is auto-committed and
errors in earlier statements don't trigger skipping of later statements).
What's missing is implicit transactions. Usually if you send a series
of messages they will each get their own implicit transaction. If you
batch them, the whole lot gets an implicit transaction. This is
because the PostgreSQL v3 protocol conflates transaction delineation
with protocol error recovery into a single Sync message type.
That's right. I sent a message complaining about this conflation a while
ago:
/messages/by-id/CADT4RqDdo9EcFbxwB_YO2H3BVZ0t-1qqZ=++dVMnYaN6BpyUGQ@mail.gmail.com.
There weren't any responses, although I'll add a note on the wiki on this
as a requested feature for the v4 protocol.
If the mythical v4 protocol ever happens I'd want to split Sync into
two messages, one which is a protocol synchronisation message and
another that is a transaction delimiter. Or give it flags or whatever.
Totally agree.
In the mean time:
This can be a problem for
several reasons: users may want to sent off a batch of inserts, notcaring
whether one of them fails (e.g. because of a unique constraint
violation).
In other words, in some scenarios it may be appropriate for later batched
statements to be executed when an earlier batched statement raised anerror.
If Sync is only sent at the very end, this isn't possible.
Right, and that remains the case even with explicit transaction
delineation, because the first ERROR causes processing of all
subsequent messages to be skipped.The design I have in libpq allows for this by allowing the client to
delimit batches without ending batch mode, concurrently consuming a
stream of multiple batches. Each endbatch is a Sync. So a client that
wants autocommit-like behavour can send a series of 1-query batches.
I think I'll need to document this a bit better since it's more subtle
than I properly explained.
Ah, I see. libpq's API is considerably more low-level than what Npgsql
needs to provide. If I understand correctly, you allow users to specify
exactly where to insert Sync messages (if at all), so that any number of
statements arbitrarily interleaved with Sync messages can be sent without
starting to read any results. If so, then the user indeed has everything
they need to control the exact transactional behavior they want (including
full auto-commit) without compromising on performance in any way (i.e. by
increasing roundtrips).
The only minor problem I can see is that PQsendEndBatch not only adds a
Sync message to the buffer, but also flushes it. This means that you may be
forcing users to needlessly flush the buffer just because they wanted to
insert a Sync. In other words, users can't send the following messages in a
single buffer/packet:
Prepare1/Bind1/Describe1/Execute1/Sync1/Prepare2/Bind2/Describe2/Execute2/Sync2
- they have to be split into different packets. Of course, this is a
relatively minor performance issue (especially when compared to the overall
performance benefits provided by batching), and providing an API
distinction between adding a Sync and flushing the buffer may
over-complicate the API. I just thought I'd mention it.
Yes, that's what I suggest, and basically what the libpq batch
interface does, though it expects the client to deal with the
transaction boundaries.You will need to think hard about transaction boundaries as they
relate to multi-statements unless nPgSQL parses out each statement
from multi-statement strings like PgJDBC does. Otherwise a user can
sneak in:somestatement; BEGIN; someotherstatement;
or
somestatement; CoMMiT; otherstatement;
That's a good point. I definitely don't want to depend on client-side
parsing of SQL in any way (in fact a planned feature is to allow using
Npgsql without any sort of client-side parsing/manipulation of SQL).
However, the fact that BEGIN/COMMIT can be sent in batches doesn't appear
too problematic to me.
When it's about to send a batch, Npgsql knows whether it's in an (explicit)
transaction or not (by examining the transaction indicator on the last
ReadyForQuery message it received). If it's not in an (explicit)
transaction, it automatically inserts a Sync message after every Execute.
If some statement happens to be a BEGIN, it will be executed as a normal
statement and so on. The only issue is that if an error occurs after a
sneaked-in BEGIN, all subsequent statements will fail, and all have the
Sync messages Npgsql inserted. The end result will be a series of errors
that will be raised up to the user, but this isn't fundamentally different
from the case of a simple auto-commit batch containing multiple failures
(because of unique constraint violation or whatever) - multiple errors is
something that will have to be handled in any case.
Thanks for all your comments. Npgsql's support of batches needs to be more
complicated than libpq's since it's a more high-level interface - whereas
libpq offloads some of the sending/processing complexity to the user,
Npgsql needs to take care of most of it internally (another good example is
deadlock avoidance...).
Re: PATCH: Batch/pipelining support for libpq
On 14 October 2016 at 18:09, Shay Rojansky <roji@roji.org> wrote:
It has recently come to my attention that this implementation is
problematic
because it forces the batch to occur within a transaction; in other
words,
there's no option for a non-transactional batch.That's not strictly the case. If you explicitly BEGIN and COMMIT,
those operations are honoured within the batch.I wasn't precise in my formulation (although I think we understand each
other)... The problem I'm trying to address here is the fact that in the
"usual" batching implementation (i.e. where a single Sync message is sent at
the end of the batch), there's no support for batches which have no
transactions whatsoever (i.e. where each statement is auto-committed and
errors in earlier statements don't trigger skipping of later statements).
Right, you can't use implicit transactions delimited by protocol
message boundaries when batching.
The design I have in libpq allows for this by allowing the client to
delimit batches without ending batch mode, concurrently consuming a
stream of multiple batches. Each endbatch is a Sync. So a client that
wants autocommit-like behavour can send a series of 1-query batches.Ah, I see. libpq's API is considerably more low-level than what Npgsql needs
to provide. If I understand correctly, you allow users to specify exactly
where to insert Sync messages (if at all)
They don't get total control, but they can cause a Sync to be emitted
after any given statement when in batch mode.
so that any number of statements
arbitrarily interleaved with Sync messages can be sent without starting to
read any results.
Right.
if so, then the user indeed has everything they need to
control the exact transactional behavior they want (including full
auto-commit) without compromising on performance in any way (i.e. by
increasing roundtrips).
Right.
The only minor problem I can see is that PQsendEndBatch not only adds a Sync
message to the buffer, but also flushes it.
It only does a non-blocking flush.
This means that you may be
forcing users to needlessly flush the buffer just because they wanted to
insert a Sync. In other words, users can't send the following messages in a
single buffer/packet:
Prepare1/Bind1/Describe1/Execute1/Sync1/Prepare2/Bind2/Describe2/Execute2/Sync2
- they have to be split into different packets.
Oh, right. That's true, but I'm not really sure we care. I suspect
that the OS will tend to coalesce them anyway, since we're not
actually waiting until the socket sends the message. At least when the
socket isn't able to send as fast as input comes in. It might matter
for local performance in incredibly high throughput applications, but
I suspect there will be a great many other things that come first.
Anyway, the client can already control this with TCP_CORK.
Of course, this is a
relatively minor performance issue (especially when compared to the overall
performance benefits provided by batching), and providing an API distinction
between adding a Sync and flushing the buffer may over-complicate the API. I
just thought I'd mention it.
Unnecessary IMO. If we really want to add it later we'd probably do so
by setting a "no flush on endbatch" mode and adding an explicit flush
call. But I expect TCP_CORK will satisfy all realistic needs.
That's a good point. I definitely don't want to depend on client-side
parsing of SQL in any way (in fact a planned feature is to allow using
Npgsql without any sort of client-side parsing/manipulation of SQL).
However, the fact that BEGIN/COMMIT can be sent in batches doesn't appear
too problematic to me.When it's about to send a batch, Npgsql knows whether it's in an (explicit)
transaction or not (by examining the transaction indicator on the last
ReadyForQuery message it received). If it's not in an (explicit)
transaction, it automatically inserts a Sync message after every Execute. If
some statement happens to be a BEGIN, it will be executed as a normal
statement and so on. The only issue is that if an error occurs after a
sneaked-in BEGIN, all subsequent statements will fail, and all have the Sync
messages Npgsql inserted. The end result will be a series of errors that
will be raised up to the user, but this isn't fundamentally different from
the case of a simple auto-commit batch containing multiple failures (because
of unique constraint violation or whatever) - multiple errors is something
that will have to be handled in any case.
I'm a bit hesitant about how this will interact with multi-statements
containing embedded BEGIN or COMMIT, where a single protocol message
contains multiple ; delimited queries. But at least at this time of
night I can't give a concrete problematic example.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
Of course, this is a
relatively minor performance issue (especially when compared to theoverall
performance benefits provided by batching), and providing an API
distinction
between adding a Sync and flushing the buffer may over-complicate the
API. I
just thought I'd mention it.
Unnecessary IMO. If we really want to add it later we'd probably do so
by setting a "no flush on endbatch" mode and adding an explicit flush
call. But I expect TCP_CORK will satisfy all realistic needs.
Unless I'm mistaken TCP_CORK is not necessarily going to work across all
platforms (e.g. Windows), although SO_LINGER (which is more standard) also
helps here.
When it's about to send a batch, Npgsql knows whether it's in an
(explicit)transaction or not (by examining the transaction indicator on the last
ReadyForQuery message it received). If it's not in an (explicit)
transaction, it automatically inserts a Sync message after everyExecute. If
some statement happens to be a BEGIN, it will be executed as a normal
statement and so on. The only issue is that if an error occurs after a
sneaked-in BEGIN, all subsequent statements will fail, and all have theSync
messages Npgsql inserted. The end result will be a series of errors that
will be raised up to the user, but this isn't fundamentally differentfrom
the case of a simple auto-commit batch containing multiple failures
(because
of unique constraint violation or whatever) - multiple errors is
something
that will have to be handled in any case.
I'm a bit hesitant about how this will interact with multi-statements
containing embedded BEGIN or COMMIT, where a single protocol message
contains multiple ; delimited queries. But at least at this time of
night I can't give a concrete problematic example.
Unless I'm mistaken, in the extended protocol you can't combine multiple ;
delimited queries in a single Parse - that's a feature supported only by
the Query message of the simple protocol. But then, if you're in the simple
protocol Sync doesn't play any role, does it? I mean, a Query message
behaves as though it's implicitly followed by a Sync message - an error in
a previous Query message doesn't cause later messages to be skipped...
Note that Npgsql only rarely uses the simple protocol for user messages.
Npgsql is a binary-only driver, and the simple protocol doesn't support
requesting binary results. So only the extended protocol is used, except
for some edge cases where it's possible to use the simple protocol for
efficiency - statements with no parameters and where the ExecuteNonQuery
API is used (i.e. the user won't access any results).
Re: PATCH: Batch/pipelining support for libpq
On 14 October 2016 at 22:15, Shay Rojansky <roji@roji.org> wrote:
Unless I'm mistaken TCP_CORK is not necessarily going to work across all
platforms (e.g. Windows), although SO_LINGER (which is more standard) also
helps here.
Yeah, true.
You can also twiddle TCP_NODELAY on and off on other platforms.
Anyway, my point is that it's not likely to be crucial, especially
since even without socket options the host will often do packet
combining if the queue isn't empty.
Unless I'm mistaken, in the extended protocol you can't combine multiple ;
delimited queries in a single Parse - that's a feature supported only by the
Query message of the simple protocol. But then, if you're in the simple
protocol Sync doesn't play any role, does it? I mean, a Query message
behaves as though it's implicitly followed by a Sync message - an error in a
previous Query message doesn't cause later messages to be skipped.
Right, good point. So that concern isn't relevant.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
Hello, Craig,
I'm sorry to be late to review your patch. I've just been able to read the HTML doc first. Can I get the latest .patch file for reading and running the code?
Here are some comments and questions. I tried to avoid the same point as other reviewers, but there may be an overlap.
(1)
The example
UPDATE mytable SET x = x + 1;
should be
UPDATE mytable SET x = x + 1 WHERE id = 42;
(2)
"The server usually begins executing the batch before all commands in the batch are queued and the end of batch command is sent."
Does this mean that the app developer cannot control or predict how many TCP transmissions a batch is sent with? For example, if I want to insert 10 rows into a table in bulk, can I send those 10 rows (and the end of batch command) efficiently in one TCP transmission, or are they split by libpq into multiple TCP transmissions?
(3)
"To avoid deadlocks on large batches the client should be structured around a nonblocking I/O loop using a function like select, poll, epoll, WaitForMultipleObjectEx, etc."
Can't we use some (new) platform-independent API instead of using poll() or WaitForMultipleObject()? e.g. some thin wrapper around pqWait(). It seems a bit burdonsome to have to use an OS-specific API to just wait for libpq. Apart from that, it does not seem possible to wait for the socket in 64-bit apps on Windows, because SOCKET is 64-bit while PQsocket() returns int.
[winsock2.h]
/*
* The new type to be used in all
* instances which refer to sockets.
*/
typedef UINT_PTR SOCKET;
Regards
Takayuki Tsunakawa
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On 18 November 2016 at 14:04, Tsunakawa, Takayuki
<tsunakawa.takay@jp.fujitsu.com> wrote:
Hello, Craig,
I'm sorry to be late to review your patch. I've just been able to read the HTML doc first. Can I get the latest .patch file for reading and running the code?
The latest is what's attached upthread and what's in the git repo also
referenced upthread.
I haven't been able to update in response to more recent review due to
other development commitments. At this point I doubt I'll be able to
get update it again in time for v10, so anyone who wants to adopt it
is welcome.
Here are some comments and questions. I tried to avoid the same point as other reviewers, but there may be an overlap.
(1)
The example
UPDATE mytable SET x = x + 1;
should be
UPDATE mytable SET x = x + 1 WHERE id = 42;
Good catch.
(2)
"The server usually begins executing the batch before all commands in the batch are queued and the end of batch command is sent."Does this mean that the app developer cannot control or predict how many TCP transmissions a batch is sent with?
That's not what that sentence means since the TCP layer is much lower
level, but what you say is also true.
All the docs are saying there is that there's no explicit control over
when we start sending the batch to the server. How that translates to
individual TCP packets, etc, is not its problem.
For example, if I want to insert 10 rows into a table in bulk, can I send those 10 rows (and the end of batch command) efficiently in one TCP transmission, or are they split by libpq into multiple TCP transmissions?
libpq neither knows nor cares about individual TCP packets. It sends
things to the kernel and lets the kernel deal with that.
That said, you can use socket options TCP_CORK and TCP_NODELAY to get
some control over how and when data is sent. If you know you're about
to send more, you might cork the socket to give the kernel a hint that
it should expect more data to send.
(3)
"To avoid deadlocks on large batches the client should be structured around a nonblocking I/O loop using a function like select, poll, epoll, WaitForMultipleObjectEx, etc."Can't we use some (new) platform-independent API instead of using poll() or WaitForMultipleObject()? e.g. some thin wrapper around pqWait(). It seems a bit burdonsome to have to use an OS-specific API to just wait for libpq. Apart from that, it does not seem possible to wait for the socket in 64-bit apps on Windows, because SOCKET is 64-bit while PQsocket() returns int.
IMO this problem is out of scope for this patch. A wait abstraction
might be nice, but next thing we know we'll be reinventing APR or
NSPR, I think that's a totally different problem.
Not being able to get a win32 SOCKET from libpq seems like a bit of an
oversight, and it'd definitely be good to address that to make async
mode more usable on Windows. There's some other ugliness in PQsocket
already per the comments there. I think it should be a separate patch,
though.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On Fri, Nov 18, 2016 at 7:18 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
The latest is what's attached upthread and what's in the git repo also
referenced upthread.I haven't been able to update in response to more recent review due to
other development commitments. At this point I doubt I'll be able to
get update it again in time for v10, so anyone who wants to adopt it
is welcome.
Currently patch status is marked as "returned with feedback" in the 11-2016
commitfest. Anyone who wants to work on it can submit the updated patch
by taking care of all review comments and change the status of the patch
at any time.
Thanks for the patch.
Regards,
Hari Babu
Fujitsu Australia
Re: PATCH: Batch/pipelining support for libpq
On 22 November 2016 at 15:14, Haribabu Kommi <kommi.haribabu@gmail.com> wrote:
On Fri, Nov 18, 2016 at 7:18 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
The latest is what's attached upthread and what's in the git repo also
referenced upthread.I haven't been able to update in response to more recent review due to
other development commitments. At this point I doubt I'll be able to
get update it again in time for v10, so anyone who wants to adopt it
is welcome.Currently patch status is marked as "returned with feedback" in the 11-2016
commitfest. Anyone who wants to work on it can submit the updated patch
by taking care of all review comments and change the status of the patch
at any time.Thanks for the patch.
Thanks. Sorry I haven't had time to work on it. Priorities.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
Hi,
On 18 November 2016 at 08:18, Craig Ringer wrote:
At this point I doubt I'll be able to
get update it again in time for v10, so anyone who wants to adopt it
is welcome.
I am interested in pipeline/batch support for ECPG, and found this thread.
I updated Craig's patch [1]https://github.com/2ndQuadrant/postgres/tree/dev/libpq-async-batch to apply this one to HEAD. Moreover, I fixed an easy typo.
First, I'm changing PQqueriesInBatch() to work as documented.
After that, I plan to reflect contents of reviews in the patch.
On Mon, Oct 3, 2016 at 11:52 PM, Daniel Verite wrote:
Wouldn't pgbench benefit from it?
It was mentioned some time ago [1], in relationship to the
\into construct, how client-server latency was important enough to
justify the use of a "\;" separator between statements, to send them
as a group.
But with the libpq batch API, maybe this could be modernized
with meta-commands like this:
\startbatch
...
\endbatch
I'm planning to work on meta-commands to support batch mode after committing this patch successfully.
[1]: https://github.com/2ndQuadrant/postgres/tree/dev/libpq-async-batch
Regards,
Aya Iwata
FUJITSU
Attachments:
0001-Pipelining-batch-support-for-libpq-v3.patchapplication/octet-stream; name=0001-Pipelining-batch-support-for-libpq-v3.patchDownload
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index ea7e7da..ac5d4fa 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4637,6 +4637,484 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up multiple queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It somewhat increased client application
+ complexity and extra caution is required to prevent client/server network
+ deadlocks, but can offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ ("ping time") is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 9.6, but clients using it can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ <application>libpq</application> into batch mode. Enter batch mode with <link
+ linkend="libpq-pqbeginbatchmode"><function>PQbeginBatchMode(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-pqisinbatchmode"><function>PQisInBatchMode(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not allowed. (The restriction on <literal>COPY</literal> is an implementation
+ limit; the PostgreSQL protocol and server can support batched <literal>COPY</literal>).
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQsendEndBatch</function>.
+ Concurrently, it uses <function>PQgetResult</function> and
+ <function>PQgetNextQuery</function> to get results. It may eventually exit
+ batch mode with <function>PQendBatchMode</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it's already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions like
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ etc. The asynchronous requests are followed by a <link
+ linkend="libpq-pqsendendbatch"><function>PQsendEndBatch(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server usually begins executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing with sending batch queries</link>, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-pqgetnextquery"><function>PQgetNextQuery</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null (or would return null if
+ called). The result from the next batch entry may then be retrieved using
+ <function>PQgetNextQuery</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQsendEndBatch</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQsendEndBatch</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetNextQuery(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-pqendbatchmode"><function>PQendBatchMode(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-pqisinbatchmode">
+ <term>
+ <function>PQisInBatchMode</function>
+ <indexterm>
+ <primary>PQisInBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns 1 if a <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>, otherwise 0.
+
+<synopsis>
+int PQisInBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqbeginbatchmode">
+ <term>
+ <function>PQbeginBatchMode</function>
+ <indexterm>
+ <primary>PQbeginBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode and returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+<synopsis>
+int PQbeginBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqendbatchmode">
+ <term>
+ <function>PQendBatchMode</function>
+ <indexterm>
+ <primary>PQendBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results and returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQgetNextQuery</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+<synopsis>
+int PQendBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqsendendbatch">
+ <term>
+ <function>PQsendEndBatch</function>
+ <indexterm>
+ <primary>PQsendEndBatch</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQsendEndBatch(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqgetnextquery">
+ <term>
+ <function>PQgetNextQuery</function>
+ <indexterm>
+ <primary>PQgetNextQuery</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results. Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. See <link
+ linkend="libpq-batch-results">processing results</link>.
+
+<synopsis>
+int PQgetNextQuery(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqqueriesinbatch">
+ <term>
+ <function>PQqueriesInBatch</function>
+ <indexterm>
+ <primary>PQqueriesInBatch</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processsed.
+ This is the number of times <function>PQgetNextQuery</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQqueriesInBatch(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqbatchisaborted">
+ <term>
+ <function>PQbatchIsAborted</function>
+ <indexterm>
+ <primary>PQbatchIsAborted</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns 1 if the batch curently being received on a
+ <application>libpq</application> connection in <link
+ linkend="libpq-batch-mode">batch mode</link> is
+ <link linkend="libpq-batch-errors">aborted</link>, 0
+ otherwise. The aborted flag is cleared as soon as the result of the
+ <function>PQsendEndBatch</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function as they can tell
+ that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+
+<synopsis>
+int PQbatchIsAborted(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
diff --git a/src/interfaces/libpq/.gitignore b/src/interfaces/libpq/.gitignore
index cb96af7..4c0d934 100644
--- a/src/interfaces/libpq/.gitignore
+++ b/src/interfaces/libpq/.gitignore
@@ -22,3 +22,4 @@
/encnames.c
/wchar.c
/libpq.rc
+/tmp_check
diff --git a/src/interfaces/libpq/Makefile b/src/interfaces/libpq/Makefile
index 4b1e552..8d5cf21 100644
--- a/src/interfaces/libpq/Makefile
+++ b/src/interfaces/libpq/Makefile
@@ -129,6 +129,11 @@ install: all installdirs install-lib
installcheck:
$(MAKE) -C test $@
+check: prove-check
+
+prove-check:
+ $(prove_check)
+
installdirs: installdirs-lib
$(MKDIR_P) '$(DESTDIR)$(includedir)' '$(DESTDIR)$(includedir_internal)' '$(DESTDIR)$(datadir)'
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 21dd772..1b0b8c5 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -171,3 +171,10 @@ PQsslAttributeNames 168
PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
+PQisInBatchMode 172
+PQqueriesInBatch 173
+PQbeginBatchMode 174
+PQendBatchMode 175
+PQsendEndBatch 176
+PQgetNextQuery 177
+PQbatchIsAborted 178
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index b83af64..71418dd 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3396,6 +3396,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3428,6 +3429,22 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+ queue = queue->next;
+ free(prev);
+ }
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+ queue = conn->cmd_queue_recycle;
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+ queue = queue->next;
+ free(prev);
+ }
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index b551875..df48767 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -69,6 +71,9 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry* PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry);
/* ----------------
@@ -1108,7 +1113,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1131,6 +1136,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->in_batch)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1229,9 +1241,29 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
+ if (conn->in_batch)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
/* check the arguments */
if (!stmtName)
{
@@ -1287,18 +1319,21 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (!conn->in_batch)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1308,10 +1343,14 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->in_batch)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1358,6 +1397,81 @@ PQsendQueryPrepared(PGconn *conn,
resultFormat);
}
+/* Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry*
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry*) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/* Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/* Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, "tried to recycle non-dangling command queue entry");
+ abort();
+ }
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/* Set up for processing a new query's results */
+static void
+PQstartProcessingNewQuery(PGconn *conn)
+{
+ /* initialize async result-accumulation state */
+ conn->result = NULL;
+ conn->next_result = NULL;
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+}
+
/*
* Common startup code for PQsendQuery and sibling routines
*/
@@ -1377,19 +1491,52 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && !conn->in_batch)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
-
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ if (conn->in_batch)
+ {
+ /* When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQgetNextQuery(...) advances
+ * to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately */
+ PQstartProcessingNewQuery(conn);
+ }
/* ready to send command message */
return true;
@@ -1414,6 +1561,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1423,6 +1574,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->in_batch)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1535,22 +1703,25 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (!conn->in_batch)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1560,10 +1731,15 @@ PQsendQueryGuts(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->in_batch)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
+
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1690,6 +1866,282 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/* PQisInBatchMode
+ * Return true if currently in batch mode
+ */
+int
+PQisInBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->in_batch;
+}
+
+/* PQqueriesInBatch
+ * Return true if there are queries currently pending in batch mode
+ */
+int
+PQqueriesInBatch(PGconn *conn)
+{
+ if (!PQisInBatchMode(conn))
+ return false;
+
+ return conn->cmd_queue_head != NULL;
+}
+
+/* PQbatchIsAborted
+ * Batch being processed is aborted, results discarded until next sync
+ */
+int
+PQbatchIsAborted(PGconn *conn)
+{
+ if (!PQisInBatchMode(conn))
+ return false;
+
+ return conn->batch_aborted;
+}
+
+/* Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * COPY is not permitted in batch mode.
+ *
+ * A set of commands is terminated by a PQsendEndBatch. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQendBatchMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQbeginBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->in_batch)
+ return true;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return false;
+
+ conn->in_batch = true;
+ conn->batch_aborted = false;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return true;
+}
+
+/* End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns true if batch mode ended.
+ */
+int
+PQendBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (!conn->in_batch)
+ return true;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ return false;
+ case PGASYNC_QUEUED:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ return false;
+
+ conn->in_batch = false;
+ conn->batch_aborted = false;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ return true;
+}
+
+/* End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new non-batch commands until all
+ * results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQsendEndBatch command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQsendEndBatch(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return false;
+
+ if (!conn->in_batch)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /* Should try to flush immediately if there's room */
+ PQflush(conn);
+
+ return true;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/* PQgetNextQuery
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns true if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns false if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQgetNextQuery(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return FALSE;
+
+ if (!conn->in_batch)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /* In batch mode but nothing left on the queue; caller can submit
+ * more work or PQendBatchMode() now. */
+ return false;
+ }
+
+ /* Pop the next query from the queue and set up the connection state
+ * as if it'd just been dispatched from a non-batched call */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ PQstartProcessingNewQuery(conn);
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_aborted && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+
+ /* Parse any available data */
+ parseInput(conn);
+ }
+
+ return true;
+}
+
/*
* PQgetResult
@@ -1697,7 +2149,6 @@ PQisBusy(PGconn *conn)
* query work remains or an error has occurred (e.g. out of
* memory).
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1749,10 +2200,31 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->in_batch)
+ {
+ /* batched queries aren't followed by a Sync to put us back in
+ * PGASYNC_IDLE state, and when we do get a sync we could still
+ * have another batch coming after this one.
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1932,6 +2404,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->in_batch)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQexec in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2126,6 +2605,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2141,6 +2623,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->in_batch)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2149,15 +2645,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (!conn->in_batch)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && !conn->in_batch)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2171,10 +2670,14 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->in_batch)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 3b0500f..32e8c79 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->in_batch)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 53776e2..026453b 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->in_batch)
+ {
+ conn->batch_aborted = false;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -305,7 +313,7 @@ pqParseInput3(PGconn *conn)
* parsing until the application accepts the current
* result.
*/
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
return;
}
break;
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->in_batch)
+ conn->batch_aborted = true;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 1b53d0e..ffdc220 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -93,7 +93,9 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort earlier in a batch */
} ExecStatusType;
typedef enum
@@ -423,6 +425,15 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQisInBatchMode(PGconn *conn);
+extern int PQbatchIsAborted(PGconn *conn);
+extern int PQqueriesInBatch(PGconn *conn);
+extern int PQbeginBatchMode(PGconn *conn);
+extern int PQendBatchMode(PGconn *conn);
+extern int PQsendEndBatch(PGconn *conn);
+extern int PQgetNextQuery(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index e9b73a9..d1ccab0 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,10 +217,13 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch result,
+ More results expected from this query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -229,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -316,6 +320,22 @@ typedef struct pg_conn_host
struct addrinfo *addrlist; /* list of possible backend addresses */
} pg_conn_host;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* PGconn stores all the state data associated with a single connection
* to a backend.
@@ -386,6 +406,8 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ bool in_batch; /* connection is in batch (pipelined) mode */
+ bool batch_aborted; /* current batch is aborted, discarding until next Sync */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY
@@ -393,6 +415,15 @@ struct pg_conn
PGnotify *notifyHead; /* oldest unreported Notify msg */
PGnotify *notifyTail; /* newest unreported Notify msg */
+ /* The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Support for multiple hosts in connection string */
int nconnhost; /* # of possible hosts */
int whichhost; /* host we're currently considering */
@@ -690,6 +721,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
diff --git a/src/interfaces/libpq/t/001_libpq_async.pl b/src/interfaces/libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000..d8f153e
--- /dev/null
+++ b/src/interfaces/libpq/t/001_libpq_async.pl
@@ -0,0 +1,15 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+$node->stop('fast');
diff --git a/src/test/examples/.gitignore b/src/test/examples/.gitignore
index 1957ec1..7a72420 100644
--- a/src/test/examples/.gitignore
+++ b/src/test/examples/.gitignore
@@ -4,3 +4,4 @@
/testlibpq4
/testlo
/testlo64
+/testlibpqbatch
diff --git a/src/test/examples/Makefile b/src/test/examples/Makefile
index 31da210..92a6faf 100644
--- a/src/test/examples/Makefile
+++ b/src/test/examples/Makefile
@@ -14,7 +14,7 @@ override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
override LDLIBS := $(libpq_pgport) $(LDLIBS)
-PROGS = testlibpq testlibpq2 testlibpq3 testlibpq4 testlo testlo64
+PROGS = testlibpq testlibpq2 testlibpq3 testlibpq4 testlo testlo64 testlibpqbatch
all: $(PROGS)
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000..11e8463
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000..7f1b8b0
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,23 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+check: prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000..d8174dd
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000..28c3eb5
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,26 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 5;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_batch simple_batch multi_batch batch_abort timings);
+
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(['testlibpqbatch', 'dbname=postgres', "$numrows", "$testname"],
+ "testlibpqbatch $testname");
+}
+
+$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000..a3c0a33
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1386 @@
+/*
+ * src/test/examples/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution funtionality
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+
+#ifndef VERBOSE
+#define VERBOSE 0
+#endif
+
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+ fflush(stderr);
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode\n");
+ goto fail;
+ }
+
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQendBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode with work in progress should fail, but succeeded\n");
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending a batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * in batch mode we have to ask for the first result to be processed;
+ * until we do PQgetResult will return null:
+ */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQgetNextQuery() call\n");
+ goto fail;
+ }
+
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* We can't PQgetNextQuery when there might still be pending results */
+ if (PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() should've failed with pending results: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQgetNextQuery() call\n");
+ goto fail;
+ }
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQendBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ /* should now get an explicit sync result */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after end batch call\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+ fflush(stderr);
+
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode: %u\n", __LINE__);
+ goto fail;
+ }
+
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "Unable to enter batch mode\n");
+ goto fail;
+ }
+
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Batch mode not activated properly\n");
+ goto fail;
+ }
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "PQexec should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ {
+ fprintf(stderr, "PQsendQuery should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "re-entering batch mode should be a no-op but failed\n");
+ goto fail;
+ }
+
+ if (PQisBusy(conn))
+ {
+ fprintf(stderr, "PQisBusy should return false when idle in batch, returned true\n");
+ goto fail;
+ }
+
+ /* ok, back to normal command mode */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "couldn't exit idle empty batch mode\n");
+ goto fail;
+ }
+
+ if (PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Batch mode not terminated properly\n");
+ goto fail;
+ }
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "batch mode exit when not in batch mode should succeed but failed\n");
+ goto fail;
+ }
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+ fflush(stderr);
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* OK, start processing the batch results */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQgetNextQuery() call\n");
+ goto fail;
+ }
+
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQgetNextQuery() call\n");
+ goto fail;
+ }
+
+ if (PQendBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ /* second batch */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+ fflush(stderr);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "1";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching error select failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "2";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "3";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* second query, caused error */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (!PQbatchIsAborted(conn))
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* third query in batch, the second insert */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at third batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (!PQbatchIsAborted(conn))
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the batch sync */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchIsAborted(conn))
+ {
+ fprintf(stderr, "sync should've cleared the aborted flag but didn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the insert from the second batch */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first entry in second batch: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first item in second batch\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* the second batch sync */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch sync\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ {
+ fprintf(stderr, "expected only insert with value 3, got %s", val);
+ goto fail;
+ }
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ fprintf(stderr, "expected 1 result, got %d", PQntuples(res));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "xact start failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "sent BEGIN\n");
+#endif
+ send_step = BI_DROP_TABLE;
+
+ if (!PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "sent DROP\n");
+#endif
+ send_step = BI_CREATE_TABLE;
+
+ if (!PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "sent CREATE\n");
+#endif
+ send_step = BI_PREPARE;
+
+ if (!PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids))
+ {
+ fprintf(stderr, "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "sent PREPARE\n");
+#endif
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ {
+ fprintf(stderr, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "Expected next query result but unable to dequeue: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "next query!\n");
+#endif
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+#if VERBOSE
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+#endif
+
+ if (PQresultStatus(res) != status)
+ {
+ fprintf(stderr, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ {
+ fprintf(stderr, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+#endif
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+#if VERBOSE
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+#endif
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+#if VERBOSE
+ fprintf(stdout, "sent COMMIT\n");
+#endif
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQsendEndBatch(conn))
+ {
+#if VERBOSE
+ fprintf(stdout, "Dispatched end batch message\n");
+#endif
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ {
+ fprintf(stderr, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "prepare failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "COPY: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[MAXINTLEN+2];
+ int formatted = snprintf(&buf[0], MAXINTLEN+1, "%d\n", nrows);
+ if (formatted >= MAXINTLEN+1)
+ {
+ fprintf(stderr, "Buffer write truncated somehow\n");
+ goto fail;
+ }
+
+ if (PQputCopyData(conn, buf, formatted) != 1)
+ {
+ fprintf(stderr, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ nrows--;
+ }
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ struct timeval start_time,
+ end_time,
+ elapsed_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_pipelined(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("batch insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long)elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_sequential(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("sequential insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long)elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_copy(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("COPY elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long)elapsed_time.tv_usec);
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings\n");
+ exit(1);
+}
+
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognised test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
Re: PATCH: Batch/pipelining support for libpq
On 22 November 2016 at 18:32, Craig Ringer<craig@2ndquadrant.com> wrote:
On 22 November 2016 at 15:14, Haribabu Kommi
<kommi.haribabu@gmail.com> wrote:On Fri, Nov 18, 2016 at 7:18 PM, Craig Ringer <craig@2ndquadrant.com>
wrote:
The latest is what's attached upthread and what's in the git repo
also referenced upthread.I haven't been able to update in response to more recent review due
to other development commitments. At this point I doubt I'll be able
to get update it again in time for v10, so anyone who wants to adopt
it is welcome.Currently patch status is marked as "returned with feedback" in the
11-2016 commitfest. Anyone who wants to work on it can submit the
updated patch by taking care of all review comments and change the
status of the patch at any time.Thanks for the patch.
Thanks. Sorry I haven't had time to work on it. Priorities.
Hi,
I am interested in this patch and addressed various below comments from reviewers. And, I have separated out code and test module into 2 patches. So, If needed, test patch can be enhanced more, meanwhile code patch can be committed.
Renaming and refactoring new APIs +PQisInBatchMode 172 +PQqueriesInBatch 173 +PQbeginBatchMode 174 +PQendBatchMode 175 +PQsendEndBatch 176 +PQgetNextQuery 177 +PQbatchIsAborted 178 This set of routines is a bit inconsistent. Why not just prefixing them with PQbatch? Like that for example: PQbatchStatus(): consists of disabled/inactive/none, active, error. This covers both PQbatchIsAborted() and PQisInBatchMode(). PQbatchBegin() PQbatchEnd() PQbatchQueueSync() or PQbatchQueueFlush, same as PQsendEndBatch() to add and process a sync message into the queue.
Renamed and modified batch status APIs as below
PQisInBatchMode & PQbatchIsAborted ==> PQbatchStatus
PQqueriesInBatch ==> PQbatchQueueCount
PQbeginBatchMode ==> PQbatchBegin
PQendBatchMode ==> PQbatchEnd
PQsendEndBatch ==> PQbatchQueueSync
PQgetNextQuery ==> PQbatchQueueProcess
PQbatchQueueCount(): returns N>0 if there are N entries, 0 if empty,-1 on failure
PQbatchQueueProcess(): returns 1 if process can begin, 0 if not, -1 on failure (OOM)
I think it is still ok to keep the current behaviour like other ones present in the same file. E.g:"PQsendPrepare" "PQsendQueryGuts"
PQqueriesInBatch() (Newname(NN):PQbatchQueueCount)doesn't work as documented.
It says:
"Returns the number of queries still in the queue for this batch"
but in fact it's implemented as a Boolean.
Modified the logic to count number of entries in pending queue and return the count
The changes in src/test/examples/ are not necessary anymore. You moved all the tests to test_libpq (for the best actually).
Removed these unnecessary changes from src/test/examples folder and corrected the path mentioned in comments section of testlibpqbatch.c
+ while (queue != NULL) + { PGcommandQueueEntry *prev = queue; + queue = queue->next; + free(prev); + } This should free prev->query.
Both prev->query and prev is freed. Also, this applies to "cmd_queue_recycle" too.
Running directly make check in src/test/modules/test_libpq/ does not work
Modified "check" rule in makefile
You could just remove the VERBOSE flag in the tests, having a test more talkative is always better.
Removed ifdef VERBOSE checks.
But with the libpq batch API, maybe this could be modernized
with meta-commands like this:
\startbatch
...
\endbatch
I think it is a separate patch candidate.
It is possible to guess each one of those errors(occurred in PQbatchQueueProcess API) with respectively
PQgetResult == NULL, PQisInBatchMode() and PQqueriesInBatch().
Definitely it should be mentioned in the docs that it is possible to make a difference between all those states.
Updated documentation section of PQbatchQueueProcess() with these details.
+ entry = PQmakePipelinedCommand(conn); + entry->queryclass = PGQUERY_SYNC; + entry->query = NULL; PQmakePipelinedCommand() returns NULL, and boom.
Corrected to return false if PQmakePipelinedCommand() returns NULL.
+ bool in_batch; /* connection is in batch (pipelined) mode */ + bool batch_aborted; /* current batch is aborted, discarding until next Sync */ Having only one flag would be fine. batch_aborted is set of used only when in_batch is used, so both have a strong link
Yes, agree that tracking the batch status via one flag is more clean
So, Added new enum typedef enum
{
PQBATCH_MODE_OFF,
PQBATCH_MODE_ON,
PQBATCH_MODE_ABORTED
} PQBatchStatus;
and " PQBatchStatus batch_status" member of pg_conn is used to track the status of batch mode.
/* OK, it's launched! */ - conn->asyncStatus = PGASYNC_BUSY; + if (conn->in_batch) + PQappendPipelinedCommand(conn, pipeCmd); + else + conn->asyncStatus = PGASYNC_BUSY; No, it is put in the queue
Though it is put in queue, technically the command is sent to server and server might have started executing it.
So it is ok to use launched I think.
It may be a good idea to add a test for COPY and trigger a failure.
Added new test - "copy_failure" to trigger failures during copy state.
Please note that COPY is allowed in batch mode, but only syncing batch/queuing any command while copy is in progress is blocked due to potential failure of entire batch.
So updated the documentation for more clarity in this case.
If I read the code correctly, it seems to me that it is possible to enable the batch mode, and then to use PQexec(), PQsendPrepare will
just happily process queue the command. Shouldn't PQexec() be prevented in batch mode? Similar remark for PQexecParams(),
PQexecPrepared() PQdescribePrepared and PQprepare(). In short everything calling PQexecFinish().
Check to verify whether the connection is in batch mode or not is already present in PQexecStart().
And, all the functions calling PQexecFinish() will not be allowed in batch mode anyways. So, no new check is needed I think.
It may be a good idea to check for PG_PROTOCOL_MAJOR < 3 and issue an error for the new routines.
All the APIs which supports asynchronous command processing can be executed only if PG_PROTOCOL_MAJOR >= 3. So, adding it to new routines are not really needed.
+ After entering batch mode the application dispatches requests + using normal asynchronous <application>libpq</application> functions like + <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>, + etc. It may be better to list all the functions here, PQSendQuery
Documentation updated with list of functions - PQsendQueryParams,PQsendPrepare,
PQsendQueryPrepared,PQdescribePortal,PQdescribePrepared,
PQsendDescribePortal,PQsendDescribePrepared.
1. Typos in document part - diff-typos.txt file contains the typos specified here 2. git --check is complaining 3. s/funtionality/functionality/ and s/recognised/recognized/ typos in testlibpqbatch.c file 4. s/Batching less useful/Batching is less useful in libpq.sgml 5. + <para> + Much like asynchronous query mode, there is no performance disadvantage to + using batching and pipelining. It somewhat increased client application + complexity and extra caution is required to prevent client/server network + deadlocks, but can offer considerable performance improvements. + </para> I would reword that a bit "it increases client application complexity and extra caution is required to prevent client/server deadlocks but offers considerable performance improvements". 6. + ("ping time") is high, and when many small operations are being performed in Nit: should use <quote> here. Still not quoting it would be fine. 7. Postgres 10, not 9.6.
Corrected.
Thanks & Regards,
Vaishnavi
Fujitsu Australia
Disclaimer
The information in this e-mail is confidential and may contain content that is subject to copyright and/or is commercial-in-confidence and is intended only for the use of the above named addressee. If you are not the intended recipient, you are hereby notified that dissemination, copying or use of the information is strictly prohibited. If you have received this e-mail in error, please telephone Fujitsu Australia Software Technology Pty Ltd on + 61 2 9452 9000 or by reply e-mail to the sender and delete the document and all copies thereof.
Whereas Fujitsu Australia Software Technology Pty Ltd would not knowingly transmit a virus within an email communication, it is the receiver’s responsibility to scan all communication and any files attached for computer viruses and other defects. Fujitsu Australia Software Technology Pty Ltd does not accept liability for any loss or damage (whether direct, indirect, consequential or economic) however caused, and whether by negligence or otherwise, which may result directly or indirectly from this communication or any files attached.
If you do not wish to receive commercial and/or marketing email messages from Fujitsu Australia Software Technology Pty Ltd, please email unsubscribe@fast.au.fujitsu.com
Attachments:
0001-Pipelining-batch-support-for-libpq-code-v3.patchapplication/octet-stream; name=0001-Pipelining-batch-support-for-libpq-code-v3.patchDownload
---
doc/src/sgml/libpq.sgml | 463 ++++++++++++++++++++++++++++
src/interfaces/libpq/.gitignore | 1 +
src/interfaces/libpq/Makefile | 5 +
src/interfaces/libpq/exports.txt | 6 +
src/interfaces/libpq/fe-connect.c | 23 ++
src/interfaces/libpq/fe-exec.c | 590 +++++++++++++++++++++++++++++++++---
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 17 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 41 ++-
10 files changed, 1133 insertions(+), 43 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 4bc5bf3..c80d0d1 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4655,6 +4655,469 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up multiple queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ offers considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ <application>libpq</application> into batch mode. Enter batch mode with <link
+ linkend="libpq-PQbatchBegin"><function>PQbatchBegin(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ (The restriction on <literal>COPY</literal> is an implementation
+ limit; the PostgreSQL protocol and server can support batched <literal>COPY</literal>).
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchQueueSync</function>.
+ Concurrently, it uses <function>PQgetResult</function> and
+ <function>PQbatchQueueProcess</function> to get results. It may eventually exit
+ batch mode with <function>PQbatchEnd</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQdescribePortal</function>,
+ <function>PQdescribePrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchQueueSync"><function>PQbatchQueueSync(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server usually begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing with sending batch queries</link>, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchQueueProcess"><function>PQbatchQueueProcess</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null (or would return null if
+ called). The result from the next batch entry may then be retrieved using
+ <function>PQbatchQueueProcess</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchQueueSync</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchQueueSync</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchQueueProcess(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQbatchEnd"><function>PQbatchEnd(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns PQBATCH_MODE_OFF or PQBATCH_MODE_ON based on a <application>libpq</application> connection's <link
+ linkend="libpq-batch-mode">batch mode</link> status.
+ Returns PQBATCH_MODE_ABORTED if a <application>libpq</application> connection is in aborted status.
+ The aborted flag is cleared as soon as the result of the
+ <function>PQbatchQueueSync</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchBegin">
+ <term>
+ <function>PQbatchBegin</function>
+ <indexterm>
+ <primary>PQbatchBegin</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode and returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+<synopsis>
+int PQbatchBegin(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchEnd">
+ <term>
+ <function>PQbatchEnd</function>
+ <indexterm>
+ <primary>PQbatchEnd</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results and returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchQueueProcess</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+<synopsis>
+int PQbatchEnd(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueSync">
+ <term>
+ <function>PQbatchQueueSync</function>
+ <indexterm>
+ <primary>PQbatchQueueSync</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchQueueSync(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueProcess">
+ <term>
+ <function>PQbatchQueueProcess</function>
+ <indexterm>
+ <primary>PQbatchQueueProcess</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results. Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchQueueCount</function>, <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+<synopsis>
+int PQbatchQueueProcess(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueCount">
+ <term>
+ <function>PQbatchQueueCount</function>
+ <indexterm>
+ <primary>PQbatchQueueCount</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processed.
+ This is the number of times <function>PQbatchQueueProcess</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQbatchQueueCount(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
diff --git a/src/interfaces/libpq/.gitignore b/src/interfaces/libpq/.gitignore
index cb96af7..4c0d934 100644
--- a/src/interfaces/libpq/.gitignore
+++ b/src/interfaces/libpq/.gitignore
@@ -22,3 +22,4 @@
/encnames.c
/wchar.c
/libpq.rc
+/tmp_check
diff --git a/src/interfaces/libpq/Makefile b/src/interfaces/libpq/Makefile
index 4b1e552..8d5cf21 100644
--- a/src/interfaces/libpq/Makefile
+++ b/src/interfaces/libpq/Makefile
@@ -129,6 +129,11 @@ install: all installdirs install-lib
installcheck:
$(MAKE) -C test $@
+check: prove-check
+
+prove-check:
+ $(prove_check)
+
installdirs: installdirs-lib
$(MKDIR_P) '$(DESTDIR)$(includedir)' '$(DESTDIR)$(includedir_internal)' '$(DESTDIR)$(datadir)'
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 21dd772..e9f81b3 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -171,3 +171,9 @@ PQsslAttributeNames 168
PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
+PQbatchQueueCount 172
+PQbatchBegin 173
+PQbatchEnd 174
+PQbatchQueueSync 175
+PQbatchQueueProcess 176
+PQbatchStatus 177
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 65b7c31..4d49f6e 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3428,6 +3428,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3460,6 +3461,28 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+ queue = conn->cmd_queue_recycle;
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index b551875..2aef492 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -69,6 +71,9 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
/* ----------------
@@ -1108,7 +1113,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1131,6 +1136,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1229,9 +1241,29 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
/* check the arguments */
if (!stmtName)
{
@@ -1287,18 +1319,21 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1308,10 +1343,14 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1358,8 +1397,90 @@ PQsendQueryPrepared(PGconn *conn,
resultFormat);
}
-/*
- * Common startup code for PQsendQuery and sibling routines
+/* PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/* PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/* PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, "tried to recycle non-dangling command queue entry");
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/* PQstartProcessingNewQuery
+ * Set up for processing a new query's results
+ */
+static void
+PQstartProcessingNewQuery(PGconn *conn)
+{
+ /* initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+}
+
+/* PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1377,20 +1498,54 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
-
- /* reset single-row processing mode */
- conn->singleRowMode = false;
-
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately */
+ PQstartProcessingNewQuery(conn);
+ }
/* ready to send command message */
return true;
}
@@ -1414,6 +1569,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1423,6 +1582,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1535,22 +1711,25 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1560,10 +1739,14 @@ PQsendQueryGuts(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1690,6 +1873,288 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/* PQbatchQueueCount
+ * Return number of queries currently pending in batch mode
+ */
+int
+PQbatchQueueCount(PGconn *conn)
+{
+ int count;
+ PGcommandQueueEntry *entry;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ return 0;
+
+ entry = conn->cmd_queue_head;
+ while (entry != NULL)
+ {
+ ++count;
+ entry = entry->next;
+ }
+ return count;
+}
+
+/* PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->batch_status;
+}
+
+/* PQbatchBegin
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQbatchBegin(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return true;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return true;
+}
+
+/* PQbatchEnd
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns true if batch mode ended.
+ */
+int
+PQbatchEnd(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return true;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ return false;
+ case PGASYNC_QUEUED:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ return true;
+}
+
+/* PQbatchQueueSync
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new non-batch commands until all
+ * results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchQueueSync(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return false; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /* Should try to flush immediately if there's room */
+ PQflush(conn);
+
+ return true;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/* PQbatchQueueProcess
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns true if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns false if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchQueueProcess(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return false;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ PQstartProcessingNewQuery(conn);
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+
+ /* Parse any available data */
+ parseInput(conn);
+ }
+
+ return true;
+}
+
/*
* PQgetResult
@@ -1749,10 +2214,32 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * batched queries aren't followed by a Sync to put us back in
+ * PGASYNC_IDLE state, and when we do get a sync we could
+ * still have another batch coming after this one.
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1932,6 +2419,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQexec in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2126,6 +2620,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2141,6 +2638,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2149,15 +2660,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2171,10 +2685,14 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 3b0500f..32e8c79 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->in_batch)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 53776e2..026453b 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->in_batch)
+ {
+ conn->batch_aborted = false;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -305,7 +313,7 @@ pqParseInput3(PGconn *conn)
* parsing until the application accepts the current
* result.
*/
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
return;
}
break;
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->in_batch)
+ conn->batch_aborted = true;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 635af5b..737264d 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQbatchBegin(PGconn *conn);
+extern int PQbatchEnd(PGconn *conn);
+extern int PQbatchQueueSync(PGconn *conn);
+extern int PQbatchQueueProcess(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index e9b73a9..5792247 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,10 +217,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -229,7 +234,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -299,6 +305,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -386,6 +408,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY
@@ -398,6 +421,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -690,6 +723,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.7.4.windows.1
0002-Pipelining-batch-support-for-libpq-test-v3.patchapplication/octet-stream; name=0002-Pipelining-batch-support-for-libpq-test-v3.patchDownload
---
src/test/modules/test_libpq/.gitignore | 5 +
src/test/modules/test_libpq/Makefile | 25 +
src/test/modules/test_libpq/README | 1 +
src/test/modules/test_libpq/t/001_libpq_async.pl | 26 +
src/test/modules/test_libpq/testlibpqbatch.c | 1608 ++++++++++++++++++++++
5 files changed, 1665 insertions(+)
create mode 100644 src/test/modules/test_libpq/.gitignore
create mode 100644 src/test/modules/test_libpq/Makefile
create mode 100644 src/test/modules/test_libpq/README
create mode 100644 src/test/modules/test_libpq/t/001_libpq_async.pl
create mode 100644 src/test/modules/test_libpq/testlibpqbatch.c
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000..11e8463
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000..d907063
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,25 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+testlibpqbatch.o: testlibpqbatch.c
+testlibpqbatch: testlibpqbatch.o
+check: testlibpqbatch prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000..d8174dd
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000..706b2de
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,26 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_batch simple_batch multi_batch batch_abort timings copyfailure);
+
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(['testlibpqbatch', 'dbname=postgres', "$numrows", "$testname"],
+ "testlibpqbatch $testname");
+}
+
+#$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000..9b72960
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1608 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution functionality
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+static void test_copyfailure(PGconn *conn);
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+ fflush(stderr);
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode\n");
+ goto fail;
+ }
+
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode with work in progress should fail, but succeeded\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending a batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * in batch mode we have to ask for the first result to be processed;
+ * until we do PQgetResult will return null:
+ */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* We can't PQbatchQueueProcess when there might still be pending results */
+ if (PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() should've failed with pending results: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ /* should now get an explicit sync result */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after end batch call\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+ fflush(stderr);
+
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode: %u\n", __LINE__);
+ goto fail;
+ }
+
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Unable to enter batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not activated properly\n");
+ goto fail;
+ }
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "PQexec should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ {
+ fprintf(stderr, "PQsendQuery should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "re-entering batch mode should be a no-op but failed\n");
+ goto fail;
+ }
+
+ if (PQisBusy(conn))
+ {
+ fprintf(stderr, "PQisBusy should return false when idle in batch, returned true\n");
+ goto fail;
+ }
+
+ /* ok, back to normal command mode */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "couldn't exit idle empty batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not terminated properly\n");
+ goto fail;
+ }
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "batch mode exit when not in batch mode should succeed but failed\n");
+ goto fail;
+ }
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+static void
+test_copyfailure(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *create_sql = "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer); INSERT INTO batch_demo VALUES(5,10); ";
+ const char *select_sql = "select id from batch_demo;";
+ const char *copy_sql = "copy batch_demo(id) to stdout;";
+ const char *copyfrom_sql = "copy batch_demo(itemno) FROM stdin;";
+ int ret;
+ char *copybuf;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Failed to enter batch mode first time\n");
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, copy_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching COPY TO query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed while processing COPY TO command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_OUT)
+ {
+ fprintf(stderr, "Wrong state during COPY TO command processing: %s %s\n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, select_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching SELECT query failed: %s\n", PQerrorMessage(conn));
+ }
+
+ for (;;)
+ {
+ ret = PQgetCopyData(conn, ©buf, 0);
+ if (ret < 0)
+ break; /* done or error */
+
+ if (copybuf)
+ {
+ fprintf(stderr, "COPYBUF: %s \n", copybuf);
+ PQfreemem(copybuf);
+ }
+ }
+
+ PQclear(res);
+ res = NULL;
+ res = PQgetResult(conn);
+
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Finish of COPY TO command failed %s :%s", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed in sync command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+ PQclear(res);
+ PQsetnonblocking(conn, 1);
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Failed to enter batch mode\n");
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, copyfrom_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching COPY FROM query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, select_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching SELECT query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* Start processing the batch results */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed while processing COPY FROM command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "Wrong state during COPY command processing: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY FROM command failed: %s \n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ /* Expect a failure here */
+ if (PQresultStatus(res) == PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpectedly COPY FROM finished with %s: %s",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "Batch mode is not aborted after failure\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at SELECT command after copy : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ /* Select query after copy should also fail */
+ if (PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "SELECT - Expected failure, got %s: %s \n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ /* Clean up all the error responses after COPY failure */
+ do
+ {
+ PQbatchQueueProcess(conn);
+ res = PQgetResult(conn);
+ fprintf(stderr, "Error status and message got from server due to COPY command failure are : %s %s \n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ } while (res != NULL);
+
+ PQclear(res);
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "Attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = PQexec(conn, select_sql);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "\nExpected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ return;
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+ fflush(stderr);
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* OK, start processing the batch results */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ /* second batch */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+ fflush(stderr);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "1";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching error select failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "2";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "3";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* second query, caused error */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* third query in batch, the second insert */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at third batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the batch sync */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "sync should've cleared the aborted flag but didn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the insert from the second batch */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first entry in second batch: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first item in second batch\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* the second batch sync */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch sync\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ {
+ fprintf(stderr, "expected only insert with value 3, got %s", val);
+ goto fail;
+ }
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ fprintf(stderr, "expected 1 result, got %d", PQntuples(res));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "xact start failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent BEGIN\n");
+
+ send_step = BI_DROP_TABLE;
+
+ if (!PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent DROP\n");
+
+ send_step = BI_CREATE_TABLE;
+
+ if (!PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent CREATE\n");
+
+ send_step = BI_PREPARE;
+
+ if (!PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids))
+ {
+ fprintf(stderr, "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent PREPARE\n");
+
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ {
+ fprintf(stderr, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "Expected next query result but unable to dequeue: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ fprintf(stdout, "next query!\n");
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+
+ if (PQresultStatus(res) != status)
+ {
+ fprintf(stderr, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ {
+ fprintf(stderr, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+ goto fail;
+ }
+
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchQueueSync(conn))
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ {
+ fprintf(stderr, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "prepare failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "COPY: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[12 + 2];
+ int formatted = snprintf(&buf[0], 12 + 1, "%d\n", nrows);
+
+ if (formatted >= 12 + 1)
+ {
+ fprintf(stderr, "Buffer write truncated somehow\n");
+ goto fail;
+ }
+
+ if (PQputCopyData(conn, buf, formatted) != 1)
+ {
+ fprintf(stderr, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ nrows--;
+ }
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ struct timeval start_time,
+ end_time,
+ elapsed_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_pipelined(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("batch insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_sequential(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("sequential insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_copy(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("COPY elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings\n");
+ exit(1);
+}
+
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1,
+ run_copyfailure = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ run_copyfailure = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else if (strcmp(argv[3], "copyfailure") == 0)
+ run_copyfailure = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognized test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+ if (run_copyfailure)
+ test_copyfailure(conn);
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
--
2.7.4.windows.1
Re: PATCH: Batch/pipelining support for libpq
On Fri, Feb 17, 2017 at 2:17 PM, Prabakaran, Vaishnavi
<VaishnaviP@fast.au.fujitsu.com> wrote:
On 22 November 2016 at 18:32, Craig Ringer<craig@2ndquadrant.com> wrote:
I am interested in this patch and addressed various below comments from reviewers. And, I have separated out code and test module into 2 patches. So, If needed, test patch can be enhanced more, meanwhile code patch can be committed.
Cool. Nice to see a new version of this patch.
(You may want to your replies breath a bit, for example by adding an
extra newline to the paragraph you are answering to.)
Renaming and refactoring new APIs +PQisInBatchMode 172 +PQqueriesInBatch 173 +PQbeginBatchMode 174 +PQendBatchMode 175 +PQsendEndBatch 176 +PQgetNextQuery 177 +PQbatchIsAborted 178 This set of routines is a bit inconsistent. Why not just prefixing them with PQbatch? Like that for example: PQbatchStatus(): consists of disabled/inactive/none, active, error. This covers both PQbatchIsAborted() and PQisInBatchMode(). PQbatchBegin() PQbatchEnd() PQbatchQueueSync() or PQbatchQueueFlush, same as PQsendEndBatch() to add and process a sync message into the queue.Renamed and modified batch status APIs as below
PQisInBatchMode & PQbatchIsAborted ==> PQbatchStatus
PQqueriesInBatch ==> PQbatchQueueCount
PQbeginBatchMode ==> PQbatchBegin
PQendBatchMode ==> PQbatchEnd
PQsendEndBatch ==> PQbatchQueueSync
PQgetNextQuery ==> PQbatchQueueProcess
Thanks. This seems a way cleaner interface to me. Others may have a
different opinion so let's see if there are some.
PQbatchQueueCount(): returns N>0 if there are N entries, 0 if empty,-1 on failure
PQbatchQueueProcess(): returns 1 if process can begin, 0 if not, -1 on failure (OOM)I think it is still ok to keep the current behaviour like other ones present in the same file. E.g:"PQsendPrepare" "PQsendQueryGuts"
PQqueriesInBatch() (Newname(NN):PQbatchQueueCount)doesn't work as documented.
It says:
"Returns the number of queries still in the queue for this batch"
but in fact it's implemented as a Boolean.Modified the logic to count number of entries in pending queue and return the count
The changes in src/test/examples/ are not necessary anymore. You moved all the tests to test_libpq (for the best actually).
Removed these unnecessary changes from src/test/examples folder and corrected the path mentioned in comments section of testlibpqbatch.c
But with the libpq batch API, maybe this could be modernized
with meta-commands like this:
\startbatch
...
\endbatchI think it is a separate patch candidate.
Definitely agreed on that. Let's not complicate things further more.
It may be a good idea to check for PG_PROTOCOL_MAJOR < 3 and issue an error for the new routines.
All the APIs which supports asynchronous command processing can be executed only if PG_PROTOCOL_MAJOR >= 3. So, adding it to new routines are not really needed.
OK. Let's not complicate the patch more than necessary.
After an extra lookup at the patch, here are some comments.
In the docs when listing any of the PQBATCH_MODE_* variables, you
should have a markup <literal> around them. It would be cleaner to
make a list of the potential values that can be returned by
PQbatchStatus() using <variablelist>.
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+ conn->cmd_queue_recycle = NULL
This could be useful as a separate routine.+/* PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
This comment block is not project-like. Please use an empty line as
first line with only "/*". The same comment applies to a bunch of the
routines introduced by this patch.Not sure there is much point in having PQstartProcessingNewQuery. It
only does two things in two places, so that's not worth the loss in
readability.
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
This setup should happen further down.conn->in_batch is undefined, causing the patch to fail to compile. And
actually you should not need it.
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
return;
This should really not be changed, or it should be changed to a status
dedicated to batching only if batch mode is activated.If you are planning for integrating this patch into Postres 10, please
be sure that this is registered in the last commit fest that will
begin next week:
https://commitfest.postgresql.org/13/
I'll try to book a couple of cycles to look at it if you are able to
register it into the CF and provide a new version.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On Tue, Feb 21, 2017 at 6:51 PM, Michael Paquier <michael.paquier@gmail.com>
wrote:
On Fri, Feb 17, 2017 at 2:17 PM, Prabakaran, Vaishnavi
<VaishnaviP@fast.au.fujitsu.com> wrote:On 22 November 2016 at 18:32, Craig Ringer<craig@2ndquadrant.com> wrote:
I am interested in this patch and addressed various below comments fromreviewers. And, I have separated out code and test module into 2 patches.
So, If needed, test patch can be enhanced more, meanwhile code patch can be
committed.Cool. Nice to see a new version of this patch.
(You may want to your replies breath a bit, for example by adding an
extra newline to the paragraph you are answering to.)Renaming and refactoring new APIs +PQisInBatchMode 172 +PQqueriesInBatch 173 +PQbeginBatchMode 174 +PQendBatchMode 175 +PQsendEndBatch 176 +PQgetNextQuery 177 +PQbatchIsAborted 178 This set of routines is a bit inconsistent. Why not just prefixing themwith PQbatch? Like that for example:
PQbatchStatus(): consists of disabled/inactive/none, active, error.
This covers both PQbatchIsAborted() and PQisInBatchMode().
PQbatchBegin()
PQbatchEnd()
PQbatchQueueSync() or PQbatchQueueFlush, same as PQsendEndBatch() to addand process a sync message into the queue.
Renamed and modified batch status APIs as below
PQisInBatchMode & PQbatchIsAborted ==> PQbatchStatus
PQqueriesInBatch ==> PQbatchQueueCount
PQbeginBatchMode ==> PQbatchBegin
PQendBatchMode ==> PQbatchEnd
PQsendEndBatch ==> PQbatchQueueSync
PQgetNextQuery ==> PQbatchQueueProcessThanks. This seems a way cleaner interface to me. Others may have a
different opinion so let's see if there are some.
PQbatchQueueCount(): returns N>0 if there are N entries, 0 if empty,-1 on
failure
PQbatchQueueProcess(): returns 1 if process can begin, 0 if not, -1 on
failure (OOM)
I think it is still ok to keep the current behaviour like other ones
present in the same file. E.g:"PQsendPrepare" "PQsendQueryGuts"
PQqueriesInBatch() (Newname(NN):PQbatchQueueCount)doesn't work as
documented.
It says:
"Returns the number of queries still in the queue for this batch"
but in fact it's implemented as a Boolean.Modified the logic to count number of entries in pending queue and
return the count
The changes in src/test/examples/ are not necessary anymore. You moved
all the tests to test_libpq (for the best actually).
Removed these unnecessary changes from src/test/examples folder and
corrected the path mentioned in comments section of testlibpqbatch.c
But with the libpq batch API, maybe this could be modernized
with meta-commands like this:
\startbatch
...
\endbatchI think it is a separate patch candidate.
Definitely agreed on that. Let's not complicate things further more.
It may be a good idea to check for PG_PROTOCOL_MAJOR < 3 and issue an
error for the new routines.
All the APIs which supports asynchronous command processing can be
executed only if PG_PROTOCOL_MAJOR >= 3. So, adding it to new routines are
not really needed.OK. Let's not complicate the patch more than necessary.
After an extra lookup at the patch, here are some comments.
Thanks for reviewing the patch.
In the docs when listing any of the PQBATCH_MODE_* variables, you
should have a markup <literal> around them. It would be cleaner to
make a list of the potential values that can be returned by
PQbatchStatus() using <variablelist>.
Modified the format of PQbatchStatus() and other batch APIs too in
documentation along with addition of <literal> tags wherever required.
+ while (queue != NULL) + { + PGcommandQueueEntry *prev = queue; + + queue = queue->next; + if (prev->query) + free(prev->query); + free(prev); + } + conn->cmd_queue_recycle = NULL This could be useful as a separate routine.
Ok, Moved this code to new function - PQfreeCommandQueue().
+/* PQmakePipelinedCommand + * Get a new command queue entry, allocating it if required. Doesn't add it to + * the tail of the queue yet, use PQappendPipelinedCommand once the command has + * been written for that. If a command fails once it's called this, it should + * use PQrecyclePipelinedCommand to put it on the freelist or release it. + * This comment block is not project-like. Please use an empty line as first line with only "/*". The same comment applies to a bunch of the routines introduced by this patch.
Corrected this.
Not sure there is much point in having PQstartProcessingNewQuery. It
only does two things in two places, so that's not worth the loss in
readability.
Yes, removed this function and placed those two things in line.
+ if (conn->batch_status != PQBATCH_MODE_OFF) + { + pipeCmd = PQmakePipelinedCommand(conn); + + if (pipeCmd == NULL) + return 0; /* error msg already set */ + + last_query = &pipeCmd->query; + queryclass = &pipeCmd->queryclass; + } + else + { + last_query = &conn->last_query; + queryclass = &conn->queryclass; + } This setup should happen further down.
Moved it down post other validations in this function.
conn->in_batch is undefined, causing the patch to fail to compile. And
actually you should not need it.
That is an update missed in my previous patch, corrected in the new patch.
- conn->asyncStatus = PGASYNC_READY; + conn->asyncStatus = PGASYNC_READY_MORE; return; This should really not be changed, or it should be changed to a status dedicated to batching only if batch mode is activated.
Since pqParseInput3() reads all the input data post "Row description"
message, yes, this change is not needed here.
If you are planning for integrating this patch into Postres 10, please
be sure that this is registered in the last commit fest that will
begin next week:
https://commitfest.postgresql.org/13/
Yes, I have created a new patch entry into the commitfest 2017-03 and
attached the latest patch with this e-mail.
I'll try to book a couple of cycles to look at it if you are able to
register it into the CF and provide a new version.
Thanks again.
Regards,
Vaishnavi,
Fujitsu Australia.
Attachments:
0001-Pipelining-batch-support-for-libpq-code-v4.patchapplication/octet-stream; name=0001-Pipelining-batch-support-for-libpq-code-v4.patchDownload
---
doc/src/sgml/libpq.sgml | 511 +++++++++++++++++++++++++++++++
src/interfaces/libpq/.gitignore | 1 +
src/interfaces/libpq/Makefile | 5 +
src/interfaces/libpq/exports.txt | 6 +
src/interfaces/libpq/fe-connect.c | 28 ++
src/interfaces/libpq/fe-exec.c | 590 ++++++++++++++++++++++++++++++++++--
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 15 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 41 ++-
10 files changed, 1188 insertions(+), 39 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 4bc5bf3..899cc5d 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4655,6 +4655,517 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up multiple queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ offers considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ <application>libpq</application> into batch mode. Enter batch mode with <link
+ linkend="libpq-PQbatchBegin"><function>PQbatchBegin(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ (The restriction on <literal>COPY</literal> is an implementation
+ limit; the PostgreSQL protocol and server can support batched <literal>COPY</literal>).
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchQueueSync</function>.
+ Concurrently, it uses <function>PQgetResult</function> and
+ <function>PQbatchQueueProcess</function> to get results. It may eventually exit
+ batch mode with <function>PQbatchEnd</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQdescribePortal</function>,
+ <function>PQdescribePrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchQueueSync"><function>PQbatchQueueSync(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing with sending batch queries</link>, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchQueueProcess"><function>PQbatchQueueProcess</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null (or would return null if
+ called). The result from the next batch entry may then be retrieved using
+ <function>PQbatchQueueProcess</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchQueueSync</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchQueueSync</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchQueueProcess(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQbatchEnd"><function>PQbatchEnd(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchQueueSync</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchBegin">
+ <term>
+ <function>PQbatchBegin</function>
+ <indexterm>
+ <primary>PQbatchBegin</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQbatchBegin(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchEnd">
+ <term>
+ <function>PQbatchEnd</function>
+ <indexterm>
+ <primary>PQbatchEnd</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQbatchEnd(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchQueueProcess</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueSync">
+ <term>
+ <function>PQbatchQueueSync</function>
+ <indexterm>
+ <primary>PQbatchQueueSync</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchQueueSync(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueProcess">
+ <term>
+ <function>PQbatchQueueProcess</function>
+ <indexterm>
+ <primary>PQbatchQueueProcess</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchQueueProcess(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchQueueCount</function>, <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueCount">
+ <term>
+ <function>PQbatchQueueCount</function>
+ <indexterm>
+ <primary>PQbatchQueueCount</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processed.
+ This is the number of times <function>PQbatchQueueProcess</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQbatchQueueCount(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
diff --git a/src/interfaces/libpq/.gitignore b/src/interfaces/libpq/.gitignore
index cb96af7..4c0d934 100644
--- a/src/interfaces/libpq/.gitignore
+++ b/src/interfaces/libpq/.gitignore
@@ -22,3 +22,4 @@
/encnames.c
/wchar.c
/libpq.rc
+/tmp_check
diff --git a/src/interfaces/libpq/Makefile b/src/interfaces/libpq/Makefile
index 4b1e552..8d5cf21 100644
--- a/src/interfaces/libpq/Makefile
+++ b/src/interfaces/libpq/Makefile
@@ -129,6 +129,11 @@ install: all installdirs install-lib
installcheck:
$(MAKE) -C test $@
+check: prove-check
+
+prove-check:
+ $(prove_check)
+
installdirs: installdirs-lib
$(MKDIR_P) '$(DESTDIR)$(includedir)' '$(DESTDIR)$(includedir_internal)' '$(DESTDIR)$(datadir)'
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 21dd772..e9f81b3 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -171,3 +171,9 @@ PQsslAttributeNames 168
PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
+PQbatchQueueCount 172
+PQbatchBegin 173
+PQbatchEnd 174
+PQbatchQueueSync 175
+PQbatchQueueProcess 176
+PQbatchStatus 177
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 65b7c31..f6530e7 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3417,6 +3417,25 @@ sendTerminateConn(PGconn *conn)
}
/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
+
+/*
* closePGconn
* - properly close a connection to the backend
*
@@ -3428,6 +3447,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3460,6 +3480,14 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index b551875..abae83c 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -69,6 +71,9 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
/* ----------------
@@ -1108,7 +1113,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1131,6 +1136,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1229,6 +1241,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1287,18 +1303,34 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1308,10 +1340,14 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1359,7 +1395,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, "tried to recycle non-dangling command queue entry");
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1377,20 +1486,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1414,6 +1563,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1423,6 +1576,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1535,22 +1705,25 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1560,10 +1733,14 @@ PQsendQueryGuts(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1690,6 +1867,300 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/*
+ * PQbatchQueueCount
+ * Return number of queries currently pending in batch mode
+ */
+int
+PQbatchQueueCount(PGconn *conn)
+{
+ int count;
+ PGcommandQueueEntry *entry;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ return 0;
+
+ entry = conn->cmd_queue_head;
+ while (entry != NULL)
+ {
+ ++count;
+ entry = entry->next;
+ }
+ return count;
+}
+
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQbatchBegin
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQbatchBegin(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return true;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return true;
+}
+
+/*
+ * PQbatchEnd
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns true if batch mode ended.
+ */
+int
+PQbatchEnd(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return true;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ return false;
+ case PGASYNC_QUEUED:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ return true;
+}
+
+/*
+ * PQbatchQueueSync
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new non-batch commands until all
+ * results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchQueueSync(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return false; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /* Should try to flush immediately if there's room */
+ PQflush(conn);
+
+ return true;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/*
+ * PQbatchQueueProcess
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns true if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns false if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchQueueProcess(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return false;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+
+ /* Parse any available data */
+ parseInput(conn);
+ }
+
+ return true;
+}
+
/*
* PQgetResult
@@ -1749,10 +2220,32 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * batched queries aren't followed by a Sync to put us back in
+ * PGASYNC_IDLE state, and when we do get a sync we could
+ * still have another batch coming after this one.
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1932,6 +2425,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQexec in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2126,6 +2626,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2141,6 +2644,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2149,15 +2666,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2171,10 +2691,14 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 3b0500f..c01f1a2 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 53776e2..e24d7ce 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 635af5b..737264d 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQbatchBegin(PGconn *conn);
+extern int PQbatchEnd(PGconn *conn);
+extern int PQbatchQueueSync(PGconn *conn);
+extern int PQbatchQueueProcess(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index e9b73a9..5792247 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,10 +217,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -229,7 +234,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -299,6 +305,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -386,6 +408,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY
@@ -398,6 +421,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -690,6 +723,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.7.4.windows.1
0002-Pipelining-batch-support-for-libpq-test-v3.patchapplication/octet-stream; name=0002-Pipelining-batch-support-for-libpq-test-v3.patchDownload
---
src/test/modules/test_libpq/.gitignore | 5 +
src/test/modules/test_libpq/Makefile | 25 +
src/test/modules/test_libpq/README | 1 +
src/test/modules/test_libpq/t/001_libpq_async.pl | 26 +
src/test/modules/test_libpq/testlibpqbatch.c | 1608 ++++++++++++++++++++++
5 files changed, 1665 insertions(+)
create mode 100644 src/test/modules/test_libpq/.gitignore
create mode 100644 src/test/modules/test_libpq/Makefile
create mode 100644 src/test/modules/test_libpq/README
create mode 100644 src/test/modules/test_libpq/t/001_libpq_async.pl
create mode 100644 src/test/modules/test_libpq/testlibpqbatch.c
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000..11e8463
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000..d907063
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,25 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+testlibpqbatch.o: testlibpqbatch.c
+testlibpqbatch: testlibpqbatch.o
+check: testlibpqbatch prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000..d8174dd
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000..706b2de
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,26 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_batch simple_batch multi_batch batch_abort timings copyfailure);
+
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(['testlibpqbatch', 'dbname=postgres', "$numrows", "$testname"],
+ "testlibpqbatch $testname");
+}
+
+#$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000..9b72960
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1608 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution functionality
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+static void test_copyfailure(PGconn *conn);
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+ fflush(stderr);
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode\n");
+ goto fail;
+ }
+
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode with work in progress should fail, but succeeded\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending a batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * in batch mode we have to ask for the first result to be processed;
+ * until we do PQgetResult will return null:
+ */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* We can't PQbatchQueueProcess when there might still be pending results */
+ if (PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() should've failed with pending results: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ /* should now get an explicit sync result */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after end batch call\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+ fflush(stderr);
+
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode: %u\n", __LINE__);
+ goto fail;
+ }
+
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Unable to enter batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not activated properly\n");
+ goto fail;
+ }
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "PQexec should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ {
+ fprintf(stderr, "PQsendQuery should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "re-entering batch mode should be a no-op but failed\n");
+ goto fail;
+ }
+
+ if (PQisBusy(conn))
+ {
+ fprintf(stderr, "PQisBusy should return false when idle in batch, returned true\n");
+ goto fail;
+ }
+
+ /* ok, back to normal command mode */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "couldn't exit idle empty batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not terminated properly\n");
+ goto fail;
+ }
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "batch mode exit when not in batch mode should succeed but failed\n");
+ goto fail;
+ }
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+static void
+test_copyfailure(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *create_sql = "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer); INSERT INTO batch_demo VALUES(5,10); ";
+ const char *select_sql = "select id from batch_demo;";
+ const char *copy_sql = "copy batch_demo(id) to stdout;";
+ const char *copyfrom_sql = "copy batch_demo(itemno) FROM stdin;";
+ int ret;
+ char *copybuf;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Failed to enter batch mode first time\n");
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, copy_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching COPY TO query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed while processing COPY TO command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_OUT)
+ {
+ fprintf(stderr, "Wrong state during COPY TO command processing: %s %s\n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, select_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching SELECT query failed: %s\n", PQerrorMessage(conn));
+ }
+
+ for (;;)
+ {
+ ret = PQgetCopyData(conn, ©buf, 0);
+ if (ret < 0)
+ break; /* done or error */
+
+ if (copybuf)
+ {
+ fprintf(stderr, "COPYBUF: %s \n", copybuf);
+ PQfreemem(copybuf);
+ }
+ }
+
+ PQclear(res);
+ res = NULL;
+ res = PQgetResult(conn);
+
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Finish of COPY TO command failed %s :%s", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed in sync command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+ PQclear(res);
+ PQsetnonblocking(conn, 1);
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Failed to enter batch mode\n");
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, copyfrom_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching COPY FROM query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, select_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching SELECT query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* Start processing the batch results */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed while processing COPY FROM command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "Wrong state during COPY command processing: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY FROM command failed: %s \n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ /* Expect a failure here */
+ if (PQresultStatus(res) == PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpectedly COPY FROM finished with %s: %s",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "Batch mode is not aborted after failure\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at SELECT command after copy : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ /* Select query after copy should also fail */
+ if (PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "SELECT - Expected failure, got %s: %s \n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ /* Clean up all the error responses after COPY failure */
+ do
+ {
+ PQbatchQueueProcess(conn);
+ res = PQgetResult(conn);
+ fprintf(stderr, "Error status and message got from server due to COPY command failure are : %s %s \n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ } while (res != NULL);
+
+ PQclear(res);
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "Attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = PQexec(conn, select_sql);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "\nExpected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ return;
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+ fflush(stderr);
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* OK, start processing the batch results */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ /* second batch */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+ fflush(stderr);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "1";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching error select failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "2";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "3";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* second query, caused error */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* third query in batch, the second insert */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at third batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the batch sync */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "sync should've cleared the aborted flag but didn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the insert from the second batch */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first entry in second batch: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first item in second batch\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* the second batch sync */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch sync\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ {
+ fprintf(stderr, "expected only insert with value 3, got %s", val);
+ goto fail;
+ }
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ fprintf(stderr, "expected 1 result, got %d", PQntuples(res));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "xact start failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent BEGIN\n");
+
+ send_step = BI_DROP_TABLE;
+
+ if (!PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent DROP\n");
+
+ send_step = BI_CREATE_TABLE;
+
+ if (!PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent CREATE\n");
+
+ send_step = BI_PREPARE;
+
+ if (!PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids))
+ {
+ fprintf(stderr, "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent PREPARE\n");
+
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ {
+ fprintf(stderr, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "Expected next query result but unable to dequeue: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ fprintf(stdout, "next query!\n");
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+
+ if (PQresultStatus(res) != status)
+ {
+ fprintf(stderr, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ {
+ fprintf(stderr, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+ goto fail;
+ }
+
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchQueueSync(conn))
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ {
+ fprintf(stderr, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "prepare failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "COPY: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[12 + 2];
+ int formatted = snprintf(&buf[0], 12 + 1, "%d\n", nrows);
+
+ if (formatted >= 12 + 1)
+ {
+ fprintf(stderr, "Buffer write truncated somehow\n");
+ goto fail;
+ }
+
+ if (PQputCopyData(conn, buf, formatted) != 1)
+ {
+ fprintf(stderr, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ nrows--;
+ }
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ struct timeval start_time,
+ end_time,
+ elapsed_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_pipelined(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("batch insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_sequential(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("sequential insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_copy(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("COPY elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings\n");
+ exit(1);
+}
+
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1,
+ run_copyfailure = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ run_copyfailure = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else if (strcmp(argv[3], "copyfailure") == 0)
+ run_copyfailure = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognized test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+ if (run_copyfailure)
+ test_copyfailure(conn);
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
--
2.7.4.windows.1
Re: PATCH: Batch/pipelining support for libpq
On 22 Feb. 2017 14:14, "Vaishnavi Prabakaran" <vaishnaviprabakaran@gmail.com>
wrote:
Thanks for reviewing the patch.
Thanks for picking it up! I've wanted to see this process for some time,
but just haven't had the bandwidth for it.
Re: PATCH: Batch/pipelining support for libpq
Vaishnavi Prabakaran wrote:
Yes, I have created a new patch entry into the commitfest 2017-03 and
attached the latest patch with this e-mail.
Please find attached a companion patch implementing the batch API in
pgbench, exposed as \beginbatch and \endbatch meta-commands
(without documentation).
The idea for now is to make it easier to exercise the API and test
how batching performs. I guess I'll submit the patch separately in
a future CF, depending on when/if the libpq patch goes in.
While developing this, I noted a few things with 0001-v4:
1. lack of initialization for count in PQbatchQueueCount.
Trivial fix:
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -1874,7 +1874,7 @@ PQisBusy(PGconn *conn)
int
PQbatchQueueCount(PGconn *conn)
{
- int count;
+ int count = 0;
PGcommandQueueEntry *entry;2. misleading error message in PQexecStart. It gets called by a few other
functions than PQexec, such as PQprepare. As I understand it, the intent
here is to forbid the synchronous functions in batch mode, so this error
message should not single out PQexec.
@@ -1932,6 +2425,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status !=
PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot
PQexec in batch mode\n"));
+ return false;
+ }
+3. In relation to #2, PQsendQuery() is not forbidden in batch mode
although I don't think it can work with it, as it's based on the old
protocol.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
Attachments:
pgbench-batch-mode-v1.patchtext/plainDownload
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index f6cb5d4..9b2fce8 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -269,7 +269,8 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
+ * CSTATE_WAIT_RESULT state unless in batch mode.
+ * On a \sleep meta-command, the timer is set,
* and we enter the CSTATE_SLEEP state to wait for it to expire. Other
* meta-commands are executed immediately.
*
@@ -1882,11 +1883,24 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- fprintf(stderr, "%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ fprintf(stderr, "%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In batch mode, we use asynchronous functions. If a server-side
+ * error occurs, it will be processed later among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ fprintf(stderr, "%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2165,7 +2179,13 @@ doCustom(TState *thread, CState *st, StatsData *agg)
return;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in batch mode */
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -2207,7 +2227,47 @@ doCustom(TState *thread, CState *st, StatsData *agg)
}
else
{
- if (pg_strcasecmp(argv[0], "set") == 0)
+ if (pg_strcasecmp(argv[0], "beginbatch") == 0)
+ {
+ /*
+ * In batch mode, we use a workflow based on libpq batch
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "cannot use batch mode with the simple query protocol");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_OFF)
+ {
+ commandFailed(st, "already in batch mode");
+ break;
+ }
+ PQbatchBegin(st->con);
+ }
+ else if (pg_strcasecmp(argv[0], "endbatch") == 0)
+ {
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ {
+ commandFailed(st, "not in batch mode");
+ break;
+ }
+ if (!PQbatchQueueSync(st->con))
+ {
+ commandFailed(st, "failed to end the batch");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ if (PQbatchEnd(st->con) == 0)
+ {
+ /* collect pending results before getting out of batch mode */
+ st->state = CSTATE_WAIT_RESULT;
+ break;
+ }
+ }
+ else if (pg_strcasecmp(argv[0], "set") == 0)
{
PgBenchExpr *expr = command->expr;
PgBenchValue result;
@@ -2279,6 +2339,7 @@ doCustom(TState *thread, CState *st, StatsData *agg)
}
break;
+
/*
* Wait for the current SQL command to complete
*/
@@ -2295,6 +2356,13 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
+ if (PQbatchStatus(st->con) == PQBATCH_MODE_ON &&
+ !PQbatchQueueProcess(st->con))
+ {
+ /* no complete result yet in batch mode*/
+ return;
+ }
+
/*
* Read and discard the query result;
*/
@@ -2307,7 +2375,22 @@ doCustom(TState *thread, CState *st, StatsData *agg)
/* OK */
PQclear(res);
discard_response(st);
- st->state = CSTATE_END_COMMAND;
+ /*
+ * In non-batch mode, only one result per command is expected.
+ * In batch mode, keep waiting for results until getting
+ * PGRES_BATCH_END.
+ */
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ st->state = CSTATE_END_COMMAND;
+ break;
+ case PGRES_BATCH_END:
+ if (PQbatchEnd(st->con) == 1)
+ {
+ /* all results collected, exit out of command and batch mode */
+ st->state = CSTATE_END_COMMAND;
+ }
+ else
+ fprintf(stderr, "client %d to exit batch mode", st->id);
break;
default:
commandFailed(st, PQerrorMessage(st->con));
@@ -3173,6 +3256,13 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->line, my_command->argv[0],
"missing command", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "beginbatch") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "endbatch") == 0 )
+ {
+ if (my_command->argc > 1)
+ syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ "unexpected argument", NULL, -1);
+ }
else
{
syntax_error(source, lineno, my_command->line, my_command->argv[0],
Re: PATCH: Batch/pipelining support for libpq
On 8 March 2017 at 00:52, Daniel Verite <daniel@manitou-mail.org> wrote:
Vaishnavi Prabakaran wrote:
Yes, I have created a new patch entry into the commitfest 2017-03 and
attached the latest patch with this e-mail.Please find attached a companion patch implementing the batch API in
pgbench, exposed as \beginbatch and \endbatch meta-commands
(without documentation).The idea for now is to make it easier to exercise the API and test
how batching performs. I guess I'll submit the patch separately in
a future CF, depending on when/if the libpq patch goes in.
That's great, thanks, and thanks also for the fixes. Any chance you
can attach your updated patch?
I looked at modifying psql to support batching when run
non-interactively, but it would've required major restructuring of its
control loop and I ran out of time. I didn't think of modifying
pgbench. Great to see.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On Wed, Mar 8, 2017 at 3:52 AM, Daniel Verite <daniel@manitou-mail.org>
wrote:
Vaishnavi Prabakaran wrote:
Yes, I have created a new patch entry into the commitfest 2017-03 and
attached the latest patch with this e-mail.Please find attached a companion patch implementing the batch API in
pgbench, exposed as \beginbatch and \endbatch meta-commands
(without documentation).The idea for now is to make it easier to exercise the API and test
how batching performs. I guess I'll submit the patch separately in
a future CF, depending on when/if the libpq patch goes in.
Thanks for the companion patch and here are some comments:
1. I see, below check is used to verify if the connection is not in batch
mode:
if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
But, it is better to use if (PQbatchStatus(st->con) ==
PQBATCH_MODE_OFF) for this verification. Reason is there is one more state
in batch mode - PQBATCH_MODE_ABORTED. So, if the batch mode is in aborted
status, this check will still assume that the connection is not in batch
mode.
In a same way, "if(PQbatchStatus(st->con) == PQBATCH_MODE_ON)" check is
better to be modified as "PQbatchStatus(st->con) != PQBATCH_MODE_OFF".
2. @@ -2207,7 +2227,47 @@ doCustom(TState *thread, CState *st, StatsData
*agg)
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_OFF)
+ {
+ commandFailed(st, "already in batch mode");
+ break;
+ }This is not required as below PQbatchBegin() internally does this
verification.
+ PQbatchBegin(st->con);
3. It is better to check the return value of PQbatchBegin() rather than
assuming success. E.g: PQbatchBegin() will return false if the connection
is in copy in/out/both states.
While developing this, I noted a few things with 0001-v4:
1. lack of initialization for count in PQbatchQueueCount.
Trivial fix:--- a/src/interfaces/libpq/fe-exec.c +++ b/src/interfaces/libpq/fe-exec.c @@ -1874,7 +1874,7 @@ PQisBusy(PGconn *conn) int PQbatchQueueCount(PGconn *conn) { - int count; + int count = 0; PGcommandQueueEntry *entry;
Thanks for your review and yes, Corrected.
2. misleading error message in PQexecStart. It gets called by a few other
functions than PQexec, such as PQprepare. As I understand it, the intent
here is to forbid the synchronous functions in batch mode, so this error
message should not single out PQexec.@@ -1932,6 +2425,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF) + { + printfPQExpBuffer(&conn->errorMessage, + libpq_gettext("cannot PQexec in batch mode\n")); + return false; + } +
Hmm, this error message goes with the flow of other error messages in the
same function. E.g: "PQexec not allowed during COPY BOTH" . But, anyways I
modified the
error message to be more generic.
3. In relation to #2, PQsendQuery() is not forbidden in batch mode
although I don't think it can work with it, as it's based on the old
protocol.
It is actually forbidden and you can see the error message "cannot
PQsendQuery in batch mode, use PQsendQueryParams" when you use
PQsendQuery().
Attached the updated patch.
Thanks & Regards,
Vaishnavi
Fujitsu Australia.
Attachments:
0001-Pipelining-batch-support-for-libpq-code-v5.patchapplication/octet-stream; name=0001-Pipelining-batch-support-for-libpq-code-v5.patchDownload
---
doc/src/sgml/libpq.sgml | 511 +++++++++++++++++++++++++++++++
src/interfaces/libpq/.gitignore | 1 +
src/interfaces/libpq/Makefile | 5 +
src/interfaces/libpq/exports.txt | 6 +
src/interfaces/libpq/fe-connect.c | 28 ++
src/interfaces/libpq/fe-exec.c | 590 ++++++++++++++++++++++++++++++++++--
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 15 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 41 ++-
10 files changed, 1188 insertions(+), 39 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 4bc5bf3..899cc5d 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4655,6 +4655,517 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up multiple queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ offers considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ <application>libpq</application> into batch mode. Enter batch mode with <link
+ linkend="libpq-PQbatchBegin"><function>PQbatchBegin(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ (The restriction on <literal>COPY</literal> is an implementation
+ limit; the PostgreSQL protocol and server can support batched <literal>COPY</literal>).
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchQueueSync</function>.
+ Concurrently, it uses <function>PQgetResult</function> and
+ <function>PQbatchQueueProcess</function> to get results. It may eventually exit
+ batch mode with <function>PQbatchEnd</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQdescribePortal</function>,
+ <function>PQdescribePrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchQueueSync"><function>PQbatchQueueSync(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing with sending batch queries</link>, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchQueueProcess"><function>PQbatchQueueProcess</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null (or would return null if
+ called). The result from the next batch entry may then be retrieved using
+ <function>PQbatchQueueProcess</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchQueueSync</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchQueueSync</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchQueueProcess(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQbatchEnd"><function>PQbatchEnd(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchQueueSync</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchBegin">
+ <term>
+ <function>PQbatchBegin</function>
+ <indexterm>
+ <primary>PQbatchBegin</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQbatchBegin(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchEnd">
+ <term>
+ <function>PQbatchEnd</function>
+ <indexterm>
+ <primary>PQbatchEnd</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQbatchEnd(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchQueueProcess</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueSync">
+ <term>
+ <function>PQbatchQueueSync</function>
+ <indexterm>
+ <primary>PQbatchQueueSync</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchQueueSync(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueProcess">
+ <term>
+ <function>PQbatchQueueProcess</function>
+ <indexterm>
+ <primary>PQbatchQueueProcess</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchQueueProcess(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchQueueCount</function>, <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueCount">
+ <term>
+ <function>PQbatchQueueCount</function>
+ <indexterm>
+ <primary>PQbatchQueueCount</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processed.
+ This is the number of times <function>PQbatchQueueProcess</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQbatchQueueCount(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
diff --git a/src/interfaces/libpq/.gitignore b/src/interfaces/libpq/.gitignore
index cb96af7..4c0d934 100644
--- a/src/interfaces/libpq/.gitignore
+++ b/src/interfaces/libpq/.gitignore
@@ -22,3 +22,4 @@
/encnames.c
/wchar.c
/libpq.rc
+/tmp_check
diff --git a/src/interfaces/libpq/Makefile b/src/interfaces/libpq/Makefile
index 4b1e552..8d5cf21 100644
--- a/src/interfaces/libpq/Makefile
+++ b/src/interfaces/libpq/Makefile
@@ -129,6 +129,11 @@ install: all installdirs install-lib
installcheck:
$(MAKE) -C test $@
+check: prove-check
+
+prove-check:
+ $(prove_check)
+
installdirs: installdirs-lib
$(MKDIR_P) '$(DESTDIR)$(includedir)' '$(DESTDIR)$(includedir_internal)' '$(DESTDIR)$(datadir)'
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 21dd772..e9f81b3 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -171,3 +171,9 @@ PQsslAttributeNames 168
PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
+PQbatchQueueCount 172
+PQbatchBegin 173
+PQbatchEnd 174
+PQbatchQueueSync 175
+PQbatchQueueProcess 176
+PQbatchStatus 177
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 65b7c31..f6530e7 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3417,6 +3417,25 @@ sendTerminateConn(PGconn *conn)
}
/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
+
+/*
* closePGconn
* - properly close a connection to the backend
*
@@ -3428,6 +3447,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3460,6 +3480,14 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index b551875..abae83c 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -69,6 +71,9 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
/* ----------------
@@ -1108,7 +1113,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1131,6 +1136,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1229,6 +1241,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1287,18 +1303,34 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1308,10 +1340,14 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1359,7 +1395,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, "tried to recycle non-dangling command queue entry");
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1377,20 +1486,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1414,6 +1563,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1423,6 +1576,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1535,22 +1705,25 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1560,10 +1733,14 @@ PQsendQueryGuts(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1690,6 +1867,300 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/*
+ * PQbatchQueueCount
+ * Return number of queries currently pending in batch mode
+ */
+int
+PQbatchQueueCount(PGconn *conn)
+{
+ int count = 0;
+ PGcommandQueueEntry *entry;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ return 0;
+
+ entry = conn->cmd_queue_head;
+ while (entry != NULL)
+ {
+ ++count;
+ entry = entry->next;
+ }
+ return count;
+}
+
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQbatchBegin
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQbatchBegin(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return true;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return true;
+}
+
+/*
+ * PQbatchEnd
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns true if batch mode ended.
+ */
+int
+PQbatchEnd(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return true;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ return false;
+ case PGASYNC_QUEUED:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ return true;
+}
+
+/*
+ * PQbatchQueueSync
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new non-batch commands until all
+ * results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchQueueSync(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return false; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /* Should try to flush immediately if there's room */
+ PQflush(conn);
+
+ return true;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/*
+ * PQbatchQueueProcess
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns true if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns false if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchQueueProcess(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return false;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+
+ /* Parse any available data */
+ parseInput(conn);
+ }
+
+ return true;
+}
+
/*
* PQgetResult
@@ -1749,10 +2220,32 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * batched queries aren't followed by a Sync to put us back in
+ * PGASYNC_IDLE state, and when we do get a sync we could
+ * still have another batch coming after this one.
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1932,6 +2425,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2126,6 +2626,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2141,6 +2644,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2149,15 +2666,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2171,10 +2691,14 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 3b0500f..c01f1a2 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 53776e2..e24d7ce 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 635af5b..737264d 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQbatchBegin(PGconn *conn);
+extern int PQbatchEnd(PGconn *conn);
+extern int PQbatchQueueSync(PGconn *conn);
+extern int PQbatchQueueProcess(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index e9b73a9..5792247 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,10 +217,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -229,7 +234,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -299,6 +305,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -386,6 +408,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY
@@ -398,6 +421,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -690,6 +723,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.7.4.windows.1
0002-Pipelining-batch-support-for-libpq-test-v3.patchapplication/octet-stream; name=0002-Pipelining-batch-support-for-libpq-test-v3.patchDownload
---
src/test/modules/test_libpq/.gitignore | 5 +
src/test/modules/test_libpq/Makefile | 25 +
src/test/modules/test_libpq/README | 1 +
src/test/modules/test_libpq/t/001_libpq_async.pl | 26 +
src/test/modules/test_libpq/testlibpqbatch.c | 1608 ++++++++++++++++++++++
5 files changed, 1665 insertions(+)
create mode 100644 src/test/modules/test_libpq/.gitignore
create mode 100644 src/test/modules/test_libpq/Makefile
create mode 100644 src/test/modules/test_libpq/README
create mode 100644 src/test/modules/test_libpq/t/001_libpq_async.pl
create mode 100644 src/test/modules/test_libpq/testlibpqbatch.c
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000..11e8463
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000..d907063
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,25 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+testlibpqbatch.o: testlibpqbatch.c
+testlibpqbatch: testlibpqbatch.o
+check: testlibpqbatch prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000..d8174dd
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000..706b2de
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,26 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_batch simple_batch multi_batch batch_abort timings copyfailure);
+
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(['testlibpqbatch', 'dbname=postgres', "$numrows", "$testname"],
+ "testlibpqbatch $testname");
+}
+
+#$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000..9b72960
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1608 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution functionality
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+static void test_copyfailure(PGconn *conn);
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+ fflush(stderr);
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode\n");
+ goto fail;
+ }
+
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode with work in progress should fail, but succeeded\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending a batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * in batch mode we have to ask for the first result to be processed;
+ * until we do PQgetResult will return null:
+ */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* We can't PQbatchQueueProcess when there might still be pending results */
+ if (PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() should've failed with pending results: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ /* should now get an explicit sync result */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after end batch call\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+ fflush(stderr);
+
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode: %u\n", __LINE__);
+ goto fail;
+ }
+
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Unable to enter batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not activated properly\n");
+ goto fail;
+ }
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "PQexec should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ {
+ fprintf(stderr, "PQsendQuery should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "re-entering batch mode should be a no-op but failed\n");
+ goto fail;
+ }
+
+ if (PQisBusy(conn))
+ {
+ fprintf(stderr, "PQisBusy should return false when idle in batch, returned true\n");
+ goto fail;
+ }
+
+ /* ok, back to normal command mode */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "couldn't exit idle empty batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not terminated properly\n");
+ goto fail;
+ }
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "batch mode exit when not in batch mode should succeed but failed\n");
+ goto fail;
+ }
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+static void
+test_copyfailure(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *create_sql = "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer); INSERT INTO batch_demo VALUES(5,10); ";
+ const char *select_sql = "select id from batch_demo;";
+ const char *copy_sql = "copy batch_demo(id) to stdout;";
+ const char *copyfrom_sql = "copy batch_demo(itemno) FROM stdin;";
+ int ret;
+ char *copybuf;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Failed to enter batch mode first time\n");
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, copy_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching COPY TO query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed while processing COPY TO command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_OUT)
+ {
+ fprintf(stderr, "Wrong state during COPY TO command processing: %s %s\n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, select_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching SELECT query failed: %s\n", PQerrorMessage(conn));
+ }
+
+ for (;;)
+ {
+ ret = PQgetCopyData(conn, ©buf, 0);
+ if (ret < 0)
+ break; /* done or error */
+
+ if (copybuf)
+ {
+ fprintf(stderr, "COPYBUF: %s \n", copybuf);
+ PQfreemem(copybuf);
+ }
+ }
+
+ PQclear(res);
+ res = NULL;
+ res = PQgetResult(conn);
+
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Finish of COPY TO command failed %s :%s", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed in sync command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+ PQclear(res);
+ PQsetnonblocking(conn, 1);
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Failed to enter batch mode\n");
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, copyfrom_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching COPY FROM query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, select_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching SELECT query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* Start processing the batch results */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed while processing COPY FROM command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "Wrong state during COPY command processing: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY FROM command failed: %s \n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ /* Expect a failure here */
+ if (PQresultStatus(res) == PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpectedly COPY FROM finished with %s: %s",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "Batch mode is not aborted after failure\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at SELECT command after copy : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ /* Select query after copy should also fail */
+ if (PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "SELECT - Expected failure, got %s: %s \n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ /* Clean up all the error responses after COPY failure */
+ do
+ {
+ PQbatchQueueProcess(conn);
+ res = PQgetResult(conn);
+ fprintf(stderr, "Error status and message got from server due to COPY command failure are : %s %s \n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ } while (res != NULL);
+
+ PQclear(res);
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "Attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = PQexec(conn, select_sql);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "\nExpected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ return;
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+ fflush(stderr);
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* OK, start processing the batch results */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ /* second batch */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+ fflush(stderr);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "1";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching error select failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "2";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "3";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* second query, caused error */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* third query in batch, the second insert */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at third batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the batch sync */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "sync should've cleared the aborted flag but didn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the insert from the second batch */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first entry in second batch: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first item in second batch\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* the second batch sync */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch sync\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ {
+ fprintf(stderr, "expected only insert with value 3, got %s", val);
+ goto fail;
+ }
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ fprintf(stderr, "expected 1 result, got %d", PQntuples(res));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "xact start failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent BEGIN\n");
+
+ send_step = BI_DROP_TABLE;
+
+ if (!PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent DROP\n");
+
+ send_step = BI_CREATE_TABLE;
+
+ if (!PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent CREATE\n");
+
+ send_step = BI_PREPARE;
+
+ if (!PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids))
+ {
+ fprintf(stderr, "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent PREPARE\n");
+
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ {
+ fprintf(stderr, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "Expected next query result but unable to dequeue: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ fprintf(stdout, "next query!\n");
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+
+ if (PQresultStatus(res) != status)
+ {
+ fprintf(stderr, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ {
+ fprintf(stderr, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+ goto fail;
+ }
+
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchQueueSync(conn))
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ {
+ fprintf(stderr, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "prepare failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "COPY: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[12 + 2];
+ int formatted = snprintf(&buf[0], 12 + 1, "%d\n", nrows);
+
+ if (formatted >= 12 + 1)
+ {
+ fprintf(stderr, "Buffer write truncated somehow\n");
+ goto fail;
+ }
+
+ if (PQputCopyData(conn, buf, formatted) != 1)
+ {
+ fprintf(stderr, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ nrows--;
+ }
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ struct timeval start_time,
+ end_time,
+ elapsed_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_pipelined(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("batch insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_sequential(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("sequential insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_copy(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("COPY elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings\n");
+ exit(1);
+}
+
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1,
+ run_copyfailure = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ run_copyfailure = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else if (strcmp(argv[3], "copyfailure") == 0)
+ run_copyfailure = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognized test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+ if (run_copyfailure)
+ test_copyfailure(conn);
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
--
2.7.4.windows.1
Re: PATCH: Batch/pipelining support for libpq
Vaishnavi Prabakaran wrote:
if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
But, it is better to use if (PQbatchStatus(st->con) ==
PQBATCH_MODE_OFF) for this verification. Reason is there is one more state
in batch mode - PQBATCH_MODE_ABORTED. So, if the batch mode is in aborted
status, this check will still assume that the connection is not in batch
mode.
In a same way, "if(PQbatchStatus(st->con) == PQBATCH_MODE_ON)" check is
better to be modified as "PQbatchStatus(st->con) != PQBATCH_MODE_OFF".
Agreed, these two tests need to be changed to account for the
PQBATCH_MODE_ABORTED case. Fixed in new patch.
2. @@ -2207,7 +2227,47 @@ doCustom(TState *thread, CState *st, StatsData *agg) + if (PQbatchStatus(st->con) != PQBATCH_MODE_OFF) + { + commandFailed(st, "already in batch mode"); + break; + }This is not required as below PQbatchBegin() internally does this
verification.+ PQbatchBegin(st->con);
The point of this test is to specifically forbid a sequence like this
\beginbatch
...(no \endbatch)
\beginbatch
...
and according to the doc for PQbatchBegin, it looks like the return
code won't tell us:
"Causes a connection to enter batch mode if it is currently idle or
already in batch mode.
int PQbatchBegin(PGconn *conn);
Returns 1 for success"
My understanding is that "already in batch mode" is not an error.
"Returns 0 and has no effect if the connection is not currently
idle, i.e. it has a result ready, is waiting for more input from
the server, etc. This function does not actually send anything to
the server, it just changes the libpq connection state"
3. It is better to check the return value of PQbatchBegin() rather than
assuming success. E.g: PQbatchBegin() will return false if the connection
is in copy in/out/both states.
Given point #2 above, I think both tests are needed:
if (PQbatchStatus(st->con) != PQBATCH_MODE_OFF)
{
commandFailed(st, "already in batch mode");
break;
}
if (PQbatchBegin(st->con) == 0)
{
commandFailed(st, "failed to start a batch");
break;
}
3. In relation to #2, PQsendQuery() is not forbidden in batch mode
although I don't think it can work with it, as it's based on the old
protocol.It is actually forbidden and you can see the error message "cannot
PQsendQuery in batch mode, use PQsendQueryParams" when you use
PQsendQuery().
Sorry for blaming the innocent piece of code, looking closer
it's pgbench that does not display the message.
With the simple query protocol, sendCommand() does essentially:
r = PQsendQuery(st->con, sql);
... other stuff...
if (r == 0)
{
if (debug)
fprintf(stderr, "client %d could not send %s\n",
st->id, command->argv[0]);
st->ecnt++;
return false;
}
So only in debug mode does it inform the user that it failed, and
even then, it does not display PQerrorMessage(st->con).
In non-debug mode it's opaque to the user. If it always fail, it appears
to loop forever. The caller has this comment that is relevant to the problem:
if (!sendCommand(st, command))
{
/*
* Failed. Stay in CSTATE_START_COMMAND state, to
* retry. ??? What the point or retrying? Should
* rather abort?
*/
return;
}
I think it doesn't bother anyone up to now because the immediate
failure modes of PQsendQuery() are resource allocation or protocol
failures that tend to never happen.
Anyway currently \beginbatch avoids the problem by checking
whether querymode == QUERY_SIMPLE, to fail gracefully
instead of letting the endless loop happen.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
Attachments:
pgbench-batch-mode-v2.patchtext/plainDownload
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index f6cb5d4..f93673e 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -269,7 +269,8 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
+ * CSTATE_WAIT_RESULT state unless in batch mode.
+ * On a \sleep meta-command, the timer is set,
* and we enter the CSTATE_SLEEP state to wait for it to expire. Other
* meta-commands are executed immediately.
*
@@ -1882,11 +1883,24 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- fprintf(stderr, "%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQbatchStatus(st->con) == PQBATCH_MODE_OFF)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ fprintf(stderr, "%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In batch mode, we use asynchronous functions. If a server-side
+ * error occurs, it will be processed later among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ fprintf(stderr, "%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2165,7 +2179,13 @@ doCustom(TState *thread, CState *st, StatsData *agg)
return;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in batch mode */
+ if (PQbatchStatus(st->con) == PQBATCH_MODE_OFF)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -2207,7 +2227,51 @@ doCustom(TState *thread, CState *st, StatsData *agg)
}
else
{
- if (pg_strcasecmp(argv[0], "set") == 0)
+ if (pg_strcasecmp(argv[0], "beginbatch") == 0)
+ {
+ /*
+ * In batch mode, we use a workflow based on libpq batch
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "cannot use batch mode with the simple query protocol");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_OFF)
+ {
+ commandFailed(st, "already in batch mode");
+ break;
+ }
+ if (PQbatchBegin(st->con) == 0)
+ {
+ commandFailed(st, "failed to start a batch");
+ break;
+ }
+ }
+ else if (pg_strcasecmp(argv[0], "endbatch") == 0)
+ {
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ {
+ commandFailed(st, "not in batch mode");
+ break;
+ }
+ if (!PQbatchQueueSync(st->con))
+ {
+ commandFailed(st, "failed to end the batch");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ if (PQbatchEnd(st->con) == 0)
+ {
+ /* collect pending results before getting out of batch mode */
+ st->state = CSTATE_WAIT_RESULT;
+ break;
+ }
+ }
+ else if (pg_strcasecmp(argv[0], "set") == 0)
{
PgBenchExpr *expr = command->expr;
PgBenchValue result;
@@ -2279,6 +2343,7 @@ doCustom(TState *thread, CState *st, StatsData *agg)
}
break;
+
/*
* Wait for the current SQL command to complete
*/
@@ -2295,6 +2360,13 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
+ if (PQbatchStatus(st->con) == PQBATCH_MODE_ON &&
+ !PQbatchQueueProcess(st->con))
+ {
+ /* no complete result yet in batch mode*/
+ return;
+ }
+
/*
* Read and discard the query result;
*/
@@ -2307,7 +2379,22 @@ doCustom(TState *thread, CState *st, StatsData *agg)
/* OK */
PQclear(res);
discard_response(st);
- st->state = CSTATE_END_COMMAND;
+ /*
+ * In non-batch mode, only one result per command is expected.
+ * In batch mode, keep waiting for results until getting
+ * PGRES_BATCH_END.
+ */
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ st->state = CSTATE_END_COMMAND;
+ break;
+ case PGRES_BATCH_END:
+ if (PQbatchEnd(st->con) == 1)
+ {
+ /* all results collected, exit out of command and batch mode */
+ st->state = CSTATE_END_COMMAND;
+ }
+ else
+ fprintf(stderr, "client %d to exit batch mode", st->id);
break;
default:
commandFailed(st, PQerrorMessage(st->con));
@@ -3173,6 +3260,13 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->line, my_command->argv[0],
"missing command", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "beginbatch") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "endbatch") == 0 )
+ {
+ if (my_command->argc > 1)
+ syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ "unexpected argument", NULL, -1);
+ }
else
{
syntax_error(source, lineno, my_command->line, my_command->argv[0],
Re: PATCH: Batch/pipelining support for libpq
Hi,
I notice that PQsetSingleRowMode() doesn't work when getting batch results.
The function is documented as:
" int PQsetSingleRowMode(PGconn *conn);
This function can only be called immediately after PQsendQuery or one
of its sibling functions, before any other operation on the connection
such as PQconsumeInput or PQgetResult"
But PQbatchQueueProcess() unconditionally clears conn->singleRowMode,
so whatever it was when sending the query gets lost, and besides
other queries might have been submitted in the meantime.
Also if trying to set that mode when fetching like this
while (QbatchQueueProcess(conn)) {
r = PQsetSingleRowMode(conn);
if (r!=1) {
fprintf(stderr, "PQsetSingleRowMode() failed");
}
..
it might work the first time, but on the next iterations, conn->asyncStatus
might be PGASYNC_READY, which is a failure condition for
PQsetSingleRowMode(), so that won't do.
ISTM that the simplest fix would be that when in batch mode,
PQsetSingleRowMode() should register that the last submitted query
does request that mode, and when later QbatchQueueProcess dequeues
the batch entry for the corresponding query, this flag would be popped off
and set as the current mode.
Please find attached the suggested fix, against the v5 of the patch.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
Attachments:
fix-for-singlerowmode.difftext/x-patch; name=fix-for-singlerowmode.diffDownload
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 3c0be46..8ddf63d 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -1425,7 +1425,7 @@ PQmakePipelinedCommand(PGconn *conn)
}
entry->next = NULL;
entry->query = NULL;
-
+ entry->singleRowMode = false;
return entry;
}
@@ -1783,16 +1783,28 @@ PQsetSingleRowMode(PGconn *conn)
/*
* Only allow setting the flag when we have launched a query and not yet
* received any results.
+ * In batch mode, store the flag in the queue for applying it later.
*/
if (!conn)
return 0;
- if (conn->asyncStatus != PGASYNC_BUSY)
- return 0;
if (conn->queryclass != PGQUERY_SIMPLE &&
conn->queryclass != PGQUERY_EXTENDED)
return 0;
if (conn->result)
return 0;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (conn->asyncStatus != PGASYNC_BUSY)
+ return 0;
+ }
+ else
+ {
+ /* apply to the last submitted query in the batch, or fail */
+ if (conn->cmd_queue_tail != NULL)
+ conn->cmd_queue_tail->singleRowMode = true;
+ else
+ return 0;
+ }
/* OK, set flag */
conn->singleRowMode = true;
@@ -2120,9 +2132,8 @@ PQbatchQueueProcess(PGconn *conn)
* Initialize async result-accumulation state */
pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
-
+ /* Set single-row processing mode */
+ conn->singleRowMode = next_query->singleRowMode;
conn->last_query = next_query->query;
next_query->query = NULL;
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 33f212f..af4f753 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -315,6 +315,7 @@ typedef struct pgCommandQueueEntry
{
PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
char *query; /* SQL command, or NULL if unknown */
+ bool singleRowMode; /* apply single row mode to this query */
struct pgCommandQueueEntry *next;
} PGcommandQueueEntry;
Re: PATCH: Batch/pipelining support for libpq
On Sat, Mar 11, 2017 at 12:52 AM, Daniel Verite <daniel@manitou-mail.org>
wrote:
Hi,
I notice that PQsetSingleRowMode() doesn't work when getting batch results.
The function is documented as:
" int PQsetSingleRowMode(PGconn *conn);This function can only be called immediately after PQsendQuery or one
of its sibling functions, before any other operation on the connection
such as PQconsumeInput or PQgetResult"But PQbatchQueueProcess() unconditionally clears conn->singleRowMode,
so whatever it was when sending the query gets lost, and besides
other queries might have been submitted in the meantime.Also if trying to set that mode when fetching like this
while (QbatchQueueProcess(conn)) {
r = PQsetSingleRowMode(conn);
if (r!=1) {
fprintf(stderr, "PQsetSingleRowMode() failed");
}
..it might work the first time, but on the next iterations, conn->asyncStatus
might be PGASYNC_READY, which is a failure condition for
PQsetSingleRowMode(), so that won't do.
Thanks for investigating the problem, and could you kindly explain what
"next iteration" you mean here? Because I don't see any problem in
following sequence of calls - PQbatchQueueProcess(),PQsetSingleRowMode()
, PQgetResult(). Am I missing anything?
Please note that it is MUST to call PQgetResult immediately after
PQbatchQueueProcess() as documented.
ISTM that the simplest fix would be that when in batch mode,
PQsetSingleRowMode() should register that the last submitted query
does request that mode, and when later QbatchQueueProcess dequeues
the batch entry for the corresponding query, this flag would be popped off
and set as the current mode.Please find attached the suggested fix, against the v5 of the patch.
Before going with this fix, I would like you to consider the option of
asking batch processing users(via documentation) to set single-row mode
before calling PQgetResult().
Either way we need to fix the documentation part, letting users know how
they can activate single-row mode while using batch processing.
Let me know your thoughts.
Best Regards,
Vaishnavi,
Fujitsu Australia.
Re: PATCH: Batch/pipelining support for libpq
On 13 March 2017 at 08:54, Vaishnavi Prabakaran
<vaishnaviprabakaran@gmail.com> wrote:
Before going with this fix, I would like you to consider the option of
asking batch processing users(via documentation) to set single-row mode
before calling PQgetResult().
Either way we need to fix the documentation part, letting users know how
they can activate single-row mode while using batch processing.
Let me know your thoughts.
Thanks for looking into this, it's clear that I didn't cover the combo
of single-row-mode and batch mode adequately.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
Vaishnavi Prabakaran wrote:
while (QbatchQueueProcess(conn)) {
r = PQsetSingleRowMode(conn);
if (r!=1) {
fprintf(stderr, "PQsetSingleRowMode() failed");
}
..
Thanks for investigating the problem, and could you kindly explain what
"next iteration" you mean here? Because I don't see any problem in
following sequence of calls - PQbatchQueueProcess(),PQsetSingleRowMode()
, PQgetResult()
I mean the next iteration of the above while statement. Referring
to the doc, that would be the "next batch entry":
" To get the result of the first batch entry the client must call
PQbatchQueueProcess. It must then call PQgetResult and handle the
results until PQgetResult returns null (or would return null if
called). The result from the next batch entry may then be retrieved
using PQbatchQueueProcess and the cycle repeated"
Attached is a bare-bones testcase showing the problem.
As it is, it works, retrieving results for three "SELECT 1"
in the same batch. Now in order to use the single-row
fetch mode, consider adding this:
r = PQsetSingleRowMode(conn);
if (r!=1) {
fprintf(stderr, "PQsetSingleRowMode() failed for i=%d\n", i);
}
When inserted after the call to PQbatchQueueProcess,
which is what I understand you're saying works for you,
it fails for me when starting to get the results of the 2nd query
and after.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
Attachments:
Re: PATCH: Batch/pipelining support for libpq
On Tue, Mar 14, 2017 at 4:19 AM, Daniel Verite <daniel@manitou-mail.org>
wrote:
I mean the next iteration of the above while statement. Referring
to the doc, that would be the "next batch entry":" To get the result of the first batch entry the client must call
PQbatchQueueProcess. It must then call PQgetResult and handle the
results until PQgetResult returns null (or would return null if
called). The result from the next batch entry may then be retrieved
using PQbatchQueueProcess and the cycle repeated"Attached is a bare-bones testcase showing the problem.
As it is, it works, retrieving results for three "SELECT 1"
in the same batch. Now in order to use the single-row
fetch mode, consider adding this:r = PQsetSingleRowMode(conn);
if (r!=1) {
fprintf(stderr, "PQsetSingleRowMode() failed for i=%d\n", i);
}When inserted after the call to PQbatchQueueProcess,
which is what I understand you're saying works for you,
it fails for me when starting to get the results of the 2nd query
and after.
Thanks for explaining the issue in detail and yes, I do see the issue using
your attached test file.
And I think the problem is with the "parseInput" call at the end of
PQbatchQueueProcess().
I don't see the need for "parseInput" to cover the scope of
PQbatchQueueProcess(), also anyways we do it in PQgetResult().
This function call changes the asyncstatus of connection to READY(following
command complete message from backend), so eventually PQsetSingleRowMode()
is failing. So, attached the alternative fix for this issue.
Please share me your thoughts.
I would also like to hear Craig's opinion on it before applying this fix to
the original patch, just to make sure am not missing anything here.
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
Attachments:
0002-single-row-mode-fix.patchapplication/octet-stream; name=0002-single-row-mode-fix.patchDownload
From bc320c1f03afe2c614dc75e47aa10b39256df0a7 Mon Sep 17 00:00:00 2001
From: Vaishnavi Prabakaran <vaishnavip@fast.au.fujitsu.com>
Date: Tue, 14 Mar 2017 17:34:32 +1100
Subject: [PATCH 2/2] single row mode fix
---
src/interfaces/libpq/fe-exec.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 3c0be46..129b49a 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -2155,7 +2155,7 @@ PQbatchQueueProcess(PGconn *conn)
conn->asyncStatus = PGASYNC_BUSY;
/* Parse any available data */
- parseInput(conn);
+ /* parseInput(conn); */
}
return true;
--
2.7.4.windows.1
Re: PATCH: Batch/pipelining support for libpq
On Tue, Mar 14, 2017 at 5:50 PM, Vaishnavi Prabakaran <
vaishnaviprabakaran@gmail.com> wrote:
On Tue, Mar 14, 2017 at 4:19 AM, Daniel Verite <daniel@manitou-mail.org>
wrote:I mean the next iteration of the above while statement. Referring
to the doc, that would be the "next batch entry":" To get the result of the first batch entry the client must call
PQbatchQueueProcess. It must then call PQgetResult and handle the
results until PQgetResult returns null (or would return null if
called). The result from the next batch entry may then be retrieved
using PQbatchQueueProcess and the cycle repeated"Attached is a bare-bones testcase showing the problem.
As it is, it works, retrieving results for three "SELECT 1"
in the same batch. Now in order to use the single-row
fetch mode, consider adding this:r = PQsetSingleRowMode(conn);
if (r!=1) {
fprintf(stderr, "PQsetSingleRowMode() failed for i=%d\n", i);
}When inserted after the call to PQbatchQueueProcess,
which is what I understand you're saying works for you,
it fails for me when starting to get the results of the 2nd query
and after.Thanks for explaining the issue in detail and yes, I do see the issue
using your attached test file.And I think the problem is with the "parseInput" call at the end of
PQbatchQueueProcess().
I don't see the need for "parseInput" to cover the scope of
PQbatchQueueProcess(), also anyways we do it in PQgetResult().
This function call changes the asyncstatus of connection to
READY(following command complete message from backend), so eventually
PQsetSingleRowMode() is failing. So, attached the alternative fix for this
issue.
Please share me your thoughts.
I would also like to hear Craig's opinion on it before applying this fix
to the original patch, just to make sure am not missing anything here.
Attached the code patch applied with the fix explained above and moving the
CF status seeking review.
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
Attachments:
0001-Pipelining-batch-support-for-libpq-code-v6.patchapplication/octet-stream; name=0001-Pipelining-batch-support-for-libpq-code-v6.patchDownload
From 1d6efd751051399555ec8cb902500421e8f7468b Mon Sep 17 00:00:00 2001
From: Vaishnavi Prabakaran <vaishnavip@fast.au.fujitsu.com>
Date: Thu, 16 Mar 2017 13:32:08 +1100
Subject: [PATCH] Pipelining-batch-support-for-libpq-code-v6
---
doc/src/sgml/libpq.sgml | 519 +++++++++++++++++++++++++++++++
src/interfaces/libpq/exports.txt | 6 +
src/interfaces/libpq/fe-connect.c | 28 ++
src/interfaces/libpq/fe-exec.c | 587 ++++++++++++++++++++++++++++++++++--
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 15 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 41 ++-
8 files changed, 1187 insertions(+), 39 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 4bc5bf3..e82721e 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4655,6 +4655,525 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up multiple queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ offers considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ <application>libpq</application> into batch mode. Enter batch mode with <link
+ linkend="libpq-PQbatchBegin"><function>PQbatchBegin(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ (The restriction on <literal>COPY</literal> is an implementation
+ limit; the PostgreSQL protocol and server can support batched <literal>COPY</literal>).
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchQueueSync</function>.
+ Concurrently, it uses <function>PQgetResult</function> and
+ <function>PQbatchQueueProcess</function> to get results. It may eventually exit
+ batch mode with <function>PQbatchEnd</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQdescribePortal</function>,
+ <function>PQdescribePrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchQueueSync"><function>PQbatchQueueSync(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing with sending batch queries</link>, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchQueueProcess"><function>PQbatchQueueProcess</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null (or would return null if
+ called). The result from the next batch entry may then be retrieved using
+ <function>PQbatchQueueProcess</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
+ successful call of <function>PQbatchQueueProcess</function>. This mode selection is effective
+ only for the currently executing query. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode">.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchQueueSync</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchQueueSync</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchQueueProcess(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQbatchEnd"><function>PQbatchEnd(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchQueueSync</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchBegin">
+ <term>
+ <function>PQbatchBegin</function>
+ <indexterm>
+ <primary>PQbatchBegin</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQbatchBegin(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchEnd">
+ <term>
+ <function>PQbatchEnd</function>
+ <indexterm>
+ <primary>PQbatchEnd</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQbatchEnd(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchQueueProcess</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueSync">
+ <term>
+ <function>PQbatchQueueSync</function>
+ <indexterm>
+ <primary>PQbatchQueueSync</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchQueueSync(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueProcess">
+ <term>
+ <function>PQbatchQueueProcess</function>
+ <indexterm>
+ <primary>PQbatchQueueProcess</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchQueueProcess(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchQueueCount</function>, <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueCount">
+ <term>
+ <function>PQbatchQueueCount</function>
+ <indexterm>
+ <primary>PQbatchQueueCount</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processed.
+ This is the number of times <function>PQbatchQueueProcess</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQbatchQueueCount(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 21dd772..e9f81b3 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -171,3 +171,9 @@ PQsslAttributeNames 168
PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
+PQbatchQueueCount 172
+PQbatchBegin 173
+PQbatchEnd 174
+PQbatchQueueSync 175
+PQbatchQueueProcess 176
+PQbatchStatus 177
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 27155f8..bb10b89 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3459,6 +3459,25 @@ sendTerminateConn(PGconn *conn)
}
/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
+
+/*
* closePGconn
* - properly close a connection to the backend
*
@@ -3470,6 +3489,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3502,6 +3522,14 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9decd53..7c02f00 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -69,6 +71,9 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
/* ----------------
@@ -1108,7 +1113,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1131,6 +1136,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1229,6 +1241,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1287,18 +1303,34 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1308,10 +1340,14 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1359,7 +1395,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, "tried to recycle non-dangling command queue entry");
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1377,20 +1486,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1414,6 +1563,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1423,6 +1576,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1535,22 +1705,25 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1560,10 +1733,14 @@ PQsendQueryGuts(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1690,6 +1867,297 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/*
+ * PQbatchQueueCount
+ * Return number of queries currently pending in batch mode
+ */
+int
+PQbatchQueueCount(PGconn *conn)
+{
+ int count = 0;
+ PGcommandQueueEntry *entry;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ return 0;
+
+ entry = conn->cmd_queue_head;
+ while (entry != NULL)
+ {
+ ++count;
+ entry = entry->next;
+ }
+ return count;
+}
+
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQbatchBegin
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQbatchBegin(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return true;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return true;
+}
+
+/*
+ * PQbatchEnd
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns true if batch mode ended.
+ */
+int
+PQbatchEnd(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return true;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ return false;
+ case PGASYNC_QUEUED:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ return true;
+}
+
+/*
+ * PQbatchQueueSync
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new non-batch commands until all
+ * results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchQueueSync(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return false; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /* Should try to flush immediately if there's room */
+ PQflush(conn);
+
+ return true;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/*
+ * PQbatchQueueProcess
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns true if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns false if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchQueueProcess(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return false;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return true;
+}
+
/*
* PQgetResult
@@ -1749,10 +2217,32 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * batched queries aren't followed by a Sync to put us back in
+ * PGASYNC_IDLE state, and when we do get a sync we could
+ * still have another batch coming after this one.
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1932,6 +2422,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2126,6 +2623,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2141,6 +2641,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2149,15 +2663,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2171,10 +2688,14 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 3b0500f..c01f1a2 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 53776e2..e24d7ce 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 635af5b..737264d 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQbatchBegin(PGconn *conn);
+extern int PQbatchEnd(PGconn *conn);
+extern int PQbatchQueueSync(PGconn *conn);
+extern int PQbatchQueueProcess(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index b8ec341..33f212f 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -215,10 +215,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -227,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -297,6 +303,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -384,6 +406,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY
@@ -396,6 +419,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -693,6 +726,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.7.4.windows.1
0002-Pipelining-batch-support-for-libpq-test-v3.patchapplication/octet-stream; name=0002-Pipelining-batch-support-for-libpq-test-v3.patchDownload
---
src/test/modules/test_libpq/.gitignore | 5 +
src/test/modules/test_libpq/Makefile | 25 +
src/test/modules/test_libpq/README | 1 +
src/test/modules/test_libpq/t/001_libpq_async.pl | 26 +
src/test/modules/test_libpq/testlibpqbatch.c | 1608 ++++++++++++++++++++++
5 files changed, 1665 insertions(+)
create mode 100644 src/test/modules/test_libpq/.gitignore
create mode 100644 src/test/modules/test_libpq/Makefile
create mode 100644 src/test/modules/test_libpq/README
create mode 100644 src/test/modules/test_libpq/t/001_libpq_async.pl
create mode 100644 src/test/modules/test_libpq/testlibpqbatch.c
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000..11e8463
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000..d907063
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,25 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+testlibpqbatch.o: testlibpqbatch.c
+testlibpqbatch: testlibpqbatch.o
+check: testlibpqbatch prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000..d8174dd
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000..706b2de
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,26 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_batch simple_batch multi_batch batch_abort timings copyfailure);
+
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(['testlibpqbatch', 'dbname=postgres', "$numrows", "$testname"],
+ "testlibpqbatch $testname");
+}
+
+#$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000..9b72960
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1608 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution functionality
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+static void test_copyfailure(PGconn *conn);
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+ fflush(stderr);
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode\n");
+ goto fail;
+ }
+
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode with work in progress should fail, but succeeded\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending a batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * in batch mode we have to ask for the first result to be processed;
+ * until we do PQgetResult will return null:
+ */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* We can't PQbatchQueueProcess when there might still be pending results */
+ if (PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() should've failed with pending results: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ /* should now get an explicit sync result */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after end batch call\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+ fflush(stderr);
+
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode: %u\n", __LINE__);
+ goto fail;
+ }
+
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Unable to enter batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not activated properly\n");
+ goto fail;
+ }
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "PQexec should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ {
+ fprintf(stderr, "PQsendQuery should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "re-entering batch mode should be a no-op but failed\n");
+ goto fail;
+ }
+
+ if (PQisBusy(conn))
+ {
+ fprintf(stderr, "PQisBusy should return false when idle in batch, returned true\n");
+ goto fail;
+ }
+
+ /* ok, back to normal command mode */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "couldn't exit idle empty batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not terminated properly\n");
+ goto fail;
+ }
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "batch mode exit when not in batch mode should succeed but failed\n");
+ goto fail;
+ }
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+static void
+test_copyfailure(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *create_sql = "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer); INSERT INTO batch_demo VALUES(5,10); ";
+ const char *select_sql = "select id from batch_demo;";
+ const char *copy_sql = "copy batch_demo(id) to stdout;";
+ const char *copyfrom_sql = "copy batch_demo(itemno) FROM stdin;";
+ int ret;
+ char *copybuf;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Failed to enter batch mode first time\n");
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, copy_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching COPY TO query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed while processing COPY TO command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_OUT)
+ {
+ fprintf(stderr, "Wrong state during COPY TO command processing: %s %s\n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, select_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching SELECT query failed: %s\n", PQerrorMessage(conn));
+ }
+
+ for (;;)
+ {
+ ret = PQgetCopyData(conn, ©buf, 0);
+ if (ret < 0)
+ break; /* done or error */
+
+ if (copybuf)
+ {
+ fprintf(stderr, "COPYBUF: %s \n", copybuf);
+ PQfreemem(copybuf);
+ }
+ }
+
+ PQclear(res);
+ res = NULL;
+ res = PQgetResult(conn);
+
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Finish of COPY TO command failed %s :%s", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed in sync command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+ PQclear(res);
+ PQsetnonblocking(conn, 1);
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Failed to enter batch mode\n");
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, copyfrom_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching COPY FROM query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, select_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching SELECT query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* Start processing the batch results */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed while processing COPY FROM command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "Wrong state during COPY command processing: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY FROM command failed: %s \n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ /* Expect a failure here */
+ if (PQresultStatus(res) == PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpectedly COPY FROM finished with %s: %s",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "Batch mode is not aborted after failure\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at SELECT command after copy : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ /* Select query after copy should also fail */
+ if (PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "SELECT - Expected failure, got %s: %s \n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ /* Clean up all the error responses after COPY failure */
+ do
+ {
+ PQbatchQueueProcess(conn);
+ res = PQgetResult(conn);
+ fprintf(stderr, "Error status and message got from server due to COPY command failure are : %s %s \n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ } while (res != NULL);
+
+ PQclear(res);
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "Attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = PQexec(conn, select_sql);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "\nExpected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ return;
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+ fflush(stderr);
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* OK, start processing the batch results */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ /* second batch */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+ fflush(stderr);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "1";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching error select failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "2";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "3";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* second query, caused error */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* third query in batch, the second insert */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at third batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the batch sync */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "sync should've cleared the aborted flag but didn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the insert from the second batch */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first entry in second batch: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first item in second batch\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* the second batch sync */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch sync\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ {
+ fprintf(stderr, "expected only insert with value 3, got %s", val);
+ goto fail;
+ }
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ fprintf(stderr, "expected 1 result, got %d", PQntuples(res));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "xact start failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent BEGIN\n");
+
+ send_step = BI_DROP_TABLE;
+
+ if (!PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent DROP\n");
+
+ send_step = BI_CREATE_TABLE;
+
+ if (!PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent CREATE\n");
+
+ send_step = BI_PREPARE;
+
+ if (!PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids))
+ {
+ fprintf(stderr, "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent PREPARE\n");
+
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ {
+ fprintf(stderr, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "Expected next query result but unable to dequeue: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ fprintf(stdout, "next query!\n");
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+
+ if (PQresultStatus(res) != status)
+ {
+ fprintf(stderr, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ {
+ fprintf(stderr, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+ goto fail;
+ }
+
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchQueueSync(conn))
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ {
+ fprintf(stderr, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "prepare failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "COPY: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[12 + 2];
+ int formatted = snprintf(&buf[0], 12 + 1, "%d\n", nrows);
+
+ if (formatted >= 12 + 1)
+ {
+ fprintf(stderr, "Buffer write truncated somehow\n");
+ goto fail;
+ }
+
+ if (PQputCopyData(conn, buf, formatted) != 1)
+ {
+ fprintf(stderr, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ nrows--;
+ }
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ struct timeval start_time,
+ end_time,
+ elapsed_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_pipelined(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("batch insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_sequential(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("sequential insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_copy(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("COPY elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings\n");
+ exit(1);
+}
+
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1,
+ run_copyfailure = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ run_copyfailure = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else if (strcmp(argv[3], "copyfailure") == 0)
+ run_copyfailure = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognized test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+ if (run_copyfailure)
+ test_copyfailure(conn);
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
--
2.7.4.windows.1
Re: PATCH: Batch/pipelining support for libpq
Vaishnavi Prabakaran wrote:
So, attached the alternative fix for this issue.
Please share me your thoughts.
I assume you prefer the alternative fix because it's simpler.
I would also like to hear Craig's opinion on it before applying this fix
to the original patch, just to make sure am not missing anything here.
+1
The main question is whether the predicates enforced
by PQsetSingleRowMode() apply in batch mode in all cases
when it's legit to call that function. Two predicates
that may be problematic are:
if (conn->asyncStatus != PGASYNC_BUSY)
return 0;
and
if (conn->result)
return 0;
The general case with batch mode is that, from the doc:
"The client interleaves result processing with sending batch queries"
Note that I've not even tested that here, I've tested
batching a bunch of queries in a first step and getting the results
in a second step.
I am not confident that the above predicates will be true
in all cases. Also your alternative fix assumes that we add
a user-visible exception to PQsetSingleRowMode in batch mode,
whereby it must not be called as currently documented:
"call PQsetSingleRowMode immediately after a successful call of
PQsendQuery (or a sibling function)"
My gut feeling is that it's not the right direction, I prefer making
the single-row a per-query attribute internally and keep
PQsetSingleRowMode's contract unchanged from the
user's perspective.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On Fri, Mar 17, 2017 at 12:37 AM, Daniel Verite <daniel@manitou-mail.org>
wrote:
Vaishnavi Prabakaran wrote:
So, attached the alternative fix for this issue.
Please share me your thoughts.I assume you prefer the alternative fix because it's simpler.
I would like add one more reason for this fix, I think "PQsetSingleRowMode"
should be called only when the result is ready to be processed and before
starting to consume result as it is documented currently as follows -
"To enter single-row mode, call PQsetSingleRowMode immediately after a
successful call of PQsendQuery (or a sibling function). This mode selection
is effective only for the currently executing query. Then call PQgetResult
repeatedly..."
I agree that first fix (you shared) will allow user to set single-row mode
after PQsendQuery, but it also allows user to set this mode at any time of
batch processing(not necessarily "immediately after PQsendQuery"), also
"mode selection is effective only for the currently executing query" will
be false. Please note that I don't see any problem with this deviation. I
like to outline that documentation here anyways needs an update/note.
Before going further, I would like to mention that I have modified the
documentation of batch processing( in v6 code patch) as below:
"To enter single-row mode, call PQsetSingleRowMode immediately after a
successful call of PQbatchQueueProcess. This mode selection is effective
only for the currently executing query. For more information on the
use of PQsetSingleRowMode
, refer to Section 33.6, “Retrieving Query Results Row-By-Row”. "
Please let me know if you think this is not enough but wanted to update
section 33.6 also?
I would also like to hear Craig's opinion on it before applying this fix
to the original patch, just to make sure am not missing anything here.+1
The main question is whether the predicates enforced
by PQsetSingleRowMode() apply in batch mode in all cases
when it's legit to call that function. Two predicates
that may be problematic are:
if (conn->asyncStatus != PGASYNC_BUSY)
return 0;
and
if (conn->result)
return 0;The general case with batch mode is that, from the doc:
"The client interleaves result processing with sending batch queries"
While sending batch queries in middle of result processing, only the query
is appended to the list of queries maintained for batch processing and no
current connection attribute impacting result processing will be changed.
So, calling the PQsetSingleRowMode in-between result processing will fail
as it tries to set single-row mode for currently executing query for which
result processing is already started.
Note that I've not even tested that here,
I've tested
batching a bunch of queries in a first step and getting the results
in a second step.
I am not confident that the above predicates will be true
in all cases.
Also your alternative fix assumes that we add
a user-visible exception to PQsetSingleRowMode in batch mode,
whereby it must not be called as currently documented:
"call PQsetSingleRowMode immediately after a successful call of
PQsendQuery (or a sibling function)"
My gut feeling is that it's not the right direction, I prefer making
the single-row a per-query attribute internally and keep
PQsetSingleRowMode's contract unchanged from the
user's perspective.
Am going to include the test which you shared in the test patch. Please let
me know if you want to cover anymore specific cases to gain confidence.
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
Re: PATCH: Batch/pipelining support for libpq
Hi Vaishnavi,
On 3/19/17 9:32 PM, Vaishnavi Prabakaran wrote:
On Fri, Mar 17, 2017 at 12:37 AM, Daniel Verite <daniel@manitou-mail.org
Please let me know if you think this is not enough but wanted to update
section 33.6 also?
Daniel, any input here?
I would also like to hear Craig's opinion on it before applying this fix
to the original patch, just to make sure am not missing anything here.
Craig?
Am going to include the test which you shared in the test patch. Please
let me know if you want to cover anymore specific cases to gain
confidence.
I have marked this submission "Waiting for Author" since it appears a
new patch is required based on input and adding a new test.
--
-David
david@pgmasters.net
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On 24 March 2017 at 23:21, David Steele <david@pgmasters.net> wrote:
Hi Vaishnavi,
On 3/19/17 9:32 PM, Vaishnavi Prabakaran wrote:
On Fri, Mar 17, 2017 at 12:37 AM, Daniel Verite <daniel@manitou-mail.org
Please let me know if you think this is not enough but wanted to update
section 33.6 also?Daniel, any input here?
I would also like to hear Craig's opinion on it before applying this
fix
to the original patch, just to make sure am not missing anything
here.
Craig?
I'm fairly confident that I overlooked single row mode entirely in the
original patch, though it's long enough ago that it's hard for me to
remember exactly.
I don't really have much of an opinion on the best handling of it.
I would expect to be setting single-row mode just before I requested
the *result* from the next pending query, since it relates to result
processing rather than query dispatch. But that's about all the
opinion I have here.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On Sat, Mar 25, 2017 at 9:50 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
I'm fairly confident that I overlooked single row mode entirely in the
original patch, though it's long enough ago that it's hard for me to
remember exactly.I don't really have much of an opinion on the best handling of it.
I would expect to be setting single-row mode just before I requested
the *result* from the next pending query, since it relates to result
processing rather than query dispatch. But that's about all the
opinion I have here.
Yeah, I think that it makes sense to allow users to switch to single
row mode before requesting a result in the queue. It seems to me that
this should also be effective only during the fetching of one single
result set. When the client moves on to the next item in the queue we
should make necessary again a call to PQsetSingleRowMode().
Regarding the patch, I have spotted the following things in the last version:
- src/test/modules/Makefile should include test_libpq.
- Regression tests from 0002 are failing:
ok 1 - testlibpqbatch disallowed_in_batch
ok 2 - testlibpqbatch simple_batch
ok 3 - testlibpqbatch multi_batch
ok 4 - testlibpqbatch batch_abort
ok 5 - testlibpqbatch timings
not ok 6 - testlibpqbatch copyfailure
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On 27 March 2017 at 15:26, Michael Paquier <michael.paquier@gmail.com> wrote:
On Sat, Mar 25, 2017 at 9:50 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
I'm fairly confident that I overlooked single row mode entirely in the
original patch, though it's long enough ago that it's hard for me to
remember exactly.I don't really have much of an opinion on the best handling of it.
I would expect to be setting single-row mode just before I requested
the *result* from the next pending query, since it relates to result
processing rather than query dispatch. But that's about all the
opinion I have here.Yeah, I think that it makes sense to allow users to switch to single
row mode before requesting a result in the queue. It seems to me that
this should also be effective only during the fetching of one single
result set. When the client moves on to the next item in the queue we
should make necessary again a call to PQsetSingleRowMode().Regarding the patch, I have spotted the following things in the last version:
- src/test/modules/Makefile should include test_libpq.
- Regression tests from 0002 are failing:
ok 1 - testlibpqbatch disallowed_in_batch
ok 2 - testlibpqbatch simple_batch
ok 3 - testlibpqbatch multi_batch
ok 4 - testlibpqbatch batch_abort
ok 5 - testlibpqbatch timings
not ok 6 - testlibpqbatch copyfailure
There are only a few more days left of this commit fest.
Things are sounding pretty ready here, with some final cleanups
pending. It'd be cool to get this into Pg 10 :)
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On Mon, Mar 27, 2017 at 4:42 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
On 27 March 2017 at 15:26, Michael Paquier <michael.paquier@gmail.com> wrote:
On Sat, Mar 25, 2017 at 9:50 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
I'm fairly confident that I overlooked single row mode entirely in the
original patch, though it's long enough ago that it's hard for me to
remember exactly.I don't really have much of an opinion on the best handling of it.
I would expect to be setting single-row mode just before I requested
the *result* from the next pending query, since it relates to result
processing rather than query dispatch. But that's about all the
opinion I have here.Yeah, I think that it makes sense to allow users to switch to single
row mode before requesting a result in the queue. It seems to me that
this should also be effective only during the fetching of one single
result set. When the client moves on to the next item in the queue we
should make necessary again a call to PQsetSingleRowMode().Regarding the patch, I have spotted the following things in the last version:
- src/test/modules/Makefile should include test_libpq.
- Regression tests from 0002 are failing:
ok 1 - testlibpqbatch disallowed_in_batch
ok 2 - testlibpqbatch simple_batch
ok 3 - testlibpqbatch multi_batch
ok 4 - testlibpqbatch batch_abort
ok 5 - testlibpqbatch timings
not ok 6 - testlibpqbatch copyfailureThere are only a few more days left of this commit fest.
Things are sounding pretty ready here, with some final cleanups
pending. It'd be cool to get this into Pg 10 :)
Yes, that's one of the items I'd like to help with by the end of the CF.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
Vaishnavi Prabakaran wrote:
Please let me know if you think this is not enough but wanted to update
section 33.6 also?
Yes, if the right place to call PQsetSingleRowMode() is immediately
after PQbatchQueueProcess(), I think we need to update
"33.6. Retrieving Query Results Row-By-Row"
with that information, otherwise what it says is just wrong
when in batch mode.
Also, in 33.5, I suggest to not use the "currently executing
query" as a synonym for the "query currently being processed"
to avoid any confusion between what the backend is executing
and a prior query whose results are being collected by the client
at the same moment.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
Vaishnavi Prabakaran wrote:
Am going to include the test which you shared in the test patch. Please let
me know if you want to cover anymore specific cases to gain confidence.
One bit of functionality that does not work in batch mode and is left
as is by the patch is the PQfn() fast path interface and the large object
functions that use it.
Currently, calling lo_* functions inside a batch will fail with a message
that depends on whether the internal lo_initialize() has been successfully
called before.
If it hasn't, PQerrorMessage() will be:
"Synchronous command execution functions are not allowed in batch mode"
which is fine, but it comes by happenstance from lo_initialize()
calling PQexec() to fetch the function OIDs from pg_catalog.pg_proc.
OTOH, if lo_initialize() has succeeded before, a call to a large
object function will fail with a different message:
"connection in wrong state"
which is emitted by PQfn() based on conn->asyncStatus != PGASYNC_IDLE
Having an unified error message would be more user friendly.
Concerning the doc, when saying in 33.5.2:
"In batch mode only asynchronous operations are permitted"
the server-side functions are indeed ruled out, since PQfn() is
synchronous, but maybe we should be a bit more explicit
about that?
Chapter 34.3 (Large Objects / Client Interfaces) could also
mention the incompatibility with batch mode.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
Thanks Craig and Michael for confirming that "PQsetSingleRowMode" should be
called right after "PQbatchQueueProcess".
Michael Paquier wrote:
It seems to me that
this should also be effective only during the fetching of one single
result set. When the client moves on to the next item in the queue we
should make necessary again a call to PQsetSingleRowMode().
Yes, the current behavior(with V6 code patch) is exactly the same as you
described above. PQsetSingleRowMode() should be called each time after
"PQbatchQueueProcess"
to set result processing to single-row mode.
src/test/modules/Makefile should include test_libpq.
Yes, added test_libpq to the list in Makefile.
Daniel Verite wrote:
Please let me know if you think this is not enough but wanted to update
section 33.6 also?
Yes, if the right place to call PQsetSingleRowMode() is immediately
after PQbatchQueueProcess(), I think we need to update
"33.6. Retrieving Query Results Row-By-Row"
with that information, otherwise what it says is just wrong
when in batch mode.
Yes, I have updated Chapter 33.6 by adding note for batch mode.
Also, in 33.5, I suggest to not use the "currently executing
query" as a synonym for the "query currently being processed"
to avoid any confusion between what the backend is executing
and a prior query whose results are being collected by the client
at the same moment.
Yes correct, I modified the words to "query currently being processed" as
suggested.
One bit of functionality that does not work in batch mode and is left
as is by the patch is the PQfn() fast path interface and the large object
functions that use it.Currently, calling lo_* functions inside a batch will fail with a message
that depends on whether the internal lo_initialize() has been successfully
called before.If it hasn't, PQerrorMessage() will be:
"Synchronous command execution functions are not allowed in batch mode"
which is fine, but it comes by happenstance from lo_initialize()
calling PQexec() to fetch the function OIDs from pg_catalog.pg_proc.OTOH, if lo_initialize() has succeeded before, a call to a large
object function will fail with a different message:
"connection in wrong state"
which is emitted by PQfn() based on conn->asyncStatus != PGASYNC_IDLEHaving an unified error message would be more user friendly.
Thanks for finding out this and yes, added a new check in PQfn() to give
the same error message - "Synchronous command execution functions are not
allowed in batch mode" when called in batch mode.
Concerning the doc, when saying in 33.5.2:
"In batch mode only asynchronous operations are permitted"
the server-side functions are indeed ruled out, since PQfn() is
synchronous, but maybe we should be a bit more explicit
about that?
Chapter 34.3 (Large Objects / Client Interfaces) could also
mention the incompatibility with batch mode.
Updated 33.5.2 to be more clear about what functions are allowed and what
are not allowed. Updated Chapter 33.3(Large Objects/ Client Interfaces) to
let the user know about the incompatibility with batch mode .
Attached the latest patch and here is the RT run result:
ok 1 - testlibpqbatch disallowed_in_batch
ok 2 - testlibpqbatch simple_batch
ok 3 - testlibpqbatch multi_batch
ok 4 - testlibpqbatch batch_abort
ok 5 - testlibpqbatch timings
ok 6 - testlibpqbatch copyfailure
ok 7 - testlibpqbatch test_singlerowmode
ok
All tests successful.
Files=1, Tests=7, 5 wallclock secs ( 0.01 usr 0.00 sys + 1.79 cusr 0.35
csys = 2.15 CPU)
Result: PASS
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
Attachments:
0001-Pipelining-batch-support-for-libpq-code-v7.patchapplication/octet-stream; name=0001-Pipelining-batch-support-for-libpq-code-v7.patchDownload
---
doc/src/sgml/libpq.sgml | 529 ++++++++++++++++++++++++++++++++
doc/src/sgml/lobj.sgml | 4 +
src/interfaces/libpq/exports.txt | 6 +
src/interfaces/libpq/fe-connect.c | 28 ++
src/interfaces/libpq/fe-exec.c | 594 ++++++++++++++++++++++++++++++++++--
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 15 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 41 ++-
9 files changed, 1208 insertions(+), 39 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 4bc5bf3..3b7e37b 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4655,6 +4655,527 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up multiple queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ offers considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ <application>libpq</application> into batch mode. Enter batch mode with <link
+ linkend="libpq-PQbatchBegin"><function>PQbatchBegin(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ (The restriction on <literal>COPY</literal> is an implementation
+ limit; the PostgreSQL protocol and server can support batched <literal>COPY</literal>).
+ Using any synchronous command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error conditions.
+ Functions allowed in batch mode are described in <xref linkend="libpq-batch-sending">.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchQueueSync</function>.
+ Concurrently, it uses <function>PQgetResult</function> and
+ <function>PQbatchQueueProcess</function> to get results. It may eventually exit
+ batch mode with <function>PQbatchEnd</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchQueueSync"><function>PQbatchQueueSync(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing with sending batch queries</link>, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchQueueProcess"><function>PQbatchQueueProcess</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null (or would return null if
+ called). The result from the next batch entry may then be retrieved using
+ <function>PQbatchQueueProcess</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
+ successful call of <function>PQbatchQueueProcess</function>. This mode selection is effective
+ only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode">.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchQueueSync</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchQueueSync</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchQueueProcess(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQbatchEnd"><function>PQbatchEnd(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchQueueSync</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchBegin">
+ <term>
+ <function>PQbatchBegin</function>
+ <indexterm>
+ <primary>PQbatchBegin</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQbatchBegin(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchEnd">
+ <term>
+ <function>PQbatchEnd</function>
+ <indexterm>
+ <primary>PQbatchEnd</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQbatchEnd(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchQueueProcess</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueSync">
+ <term>
+ <function>PQbatchQueueSync</function>
+ <indexterm>
+ <primary>PQbatchQueueSync</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchQueueSync(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueProcess">
+ <term>
+ <function>PQbatchQueueProcess</function>
+ <indexterm>
+ <primary>PQbatchQueueProcess</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchQueueProcess(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchQueueCount</function>, <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueCount">
+ <term>
+ <function>PQbatchQueueCount</function>
+ <indexterm>
+ <primary>PQbatchQueueCount</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processed.
+ This is the number of times <function>PQbatchQueueProcess</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQbatchQueueCount(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
@@ -4695,6 +5216,14 @@ int PQflush(PGconn *conn);
Each object should be freed with <function>PQclear</function> as usual.
</para>
+ <note>
+ <para>
+ On using batch mode, call <function>PQsetSingleRowMode</function>
+ immediately after a successful call of <function>PQbatchQueueProcess</function>
+ See <xref linkend="libpq-batch-mode"> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-pqsetsinglerowmode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 7757e1e..db8523d 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 21dd772..e9f81b3 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -171,3 +171,9 @@ PQsslAttributeNames 168
PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
+PQbatchQueueCount 172
+PQbatchBegin 173
+PQbatchEnd 174
+PQbatchQueueSync 175
+PQbatchQueueProcess 176
+PQbatchStatus 177
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 27155f8..bb10b89 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3459,6 +3459,25 @@ sendTerminateConn(PGconn *conn)
}
/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
+
+/*
* closePGconn
* - properly close a connection to the backend
*
@@ -3470,6 +3489,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3502,6 +3522,14 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9decd53..b0f5ce7 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -69,6 +71,9 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
/* ----------------
@@ -1108,7 +1113,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1131,6 +1136,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1229,6 +1241,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1287,18 +1303,34 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1308,10 +1340,14 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1359,7 +1395,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, "tried to recycle non-dangling command queue entry");
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1377,20 +1486,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1414,6 +1563,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1423,6 +1576,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1535,22 +1705,25 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1560,10 +1733,14 @@ PQsendQueryGuts(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1690,6 +1867,297 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/*
+ * PQbatchQueueCount
+ * Return number of queries currently pending in batch mode
+ */
+int
+PQbatchQueueCount(PGconn *conn)
+{
+ int count = 0;
+ PGcommandQueueEntry *entry;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ return 0;
+
+ entry = conn->cmd_queue_head;
+ while (entry != NULL)
+ {
+ ++count;
+ entry = entry->next;
+ }
+ return count;
+}
+
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQbatchBegin
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQbatchBegin(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return true;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return true;
+}
+
+/*
+ * PQbatchEnd
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns true if batch mode ended.
+ */
+int
+PQbatchEnd(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return true;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ return false;
+ case PGASYNC_QUEUED:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ return true;
+}
+
+/*
+ * PQbatchQueueSync
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new non-batch commands until all
+ * results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchQueueSync(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return false; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /* Should try to flush immediately if there's room */
+ PQflush(conn);
+
+ return true;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/*
+ * PQbatchQueueProcess
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns true if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns false if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchQueueProcess(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return false;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return true;
+}
+
/*
* PQgetResult
@@ -1749,10 +2217,32 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * batched queries aren't followed by a Sync to put us back in
+ * PGASYNC_IDLE state, and when we do get a sync we could
+ * still have another batch coming after this one.
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1932,6 +2422,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2126,6 +2623,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2141,6 +2641,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2149,15 +2663,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2171,10 +2688,14 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -2569,6 +3090,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 3b0500f..c01f1a2 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 53776e2..e24d7ce 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 635af5b..737264d 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQbatchBegin(PGconn *conn);
+extern int PQbatchEnd(PGconn *conn);
+extern int PQbatchQueueSync(PGconn *conn);
+extern int PQbatchQueueProcess(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index b8ec341..33f212f 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -215,10 +215,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -227,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -297,6 +303,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -384,6 +406,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY
@@ -396,6 +419,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -693,6 +726,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.7.4.windows.1
0002-Pipelining-batch-support-for-libpq-test-v4.patchapplication/octet-stream; name=0002-Pipelining-batch-support-for-libpq-test-v4.patchDownload
---
src/test/modules/Makefile | 1 +
src/test/modules/test_libpq/.gitignore | 5 +
src/test/modules/test_libpq/Makefile | 25 +
src/test/modules/test_libpq/README | 1 +
src/test/modules/test_libpq/t/001_libpq_async.pl | 26 +
src/test/modules/test_libpq/testlibpqbatch.c | 1661 ++++++++++++++++++++++
6 files changed, 1719 insertions(+)
create mode 100644 src/test/modules/test_libpq/.gitignore
create mode 100644 src/test/modules/test_libpq/Makefile
create mode 100644 src/test/modules/test_libpq/README
create mode 100644 src/test/modules/test_libpq/t/001_libpq_async.pl
create mode 100644 src/test/modules/test_libpq/testlibpqbatch.c
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 3ce9904..310d3ba 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -11,6 +11,7 @@ SUBDIRS = \
snapshot_too_old \
test_ddl_deparse \
test_extensions \
+ test_libpq \
test_parser \
test_pg_dump \
test_rls_hooks \
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000..11e8463
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000..d907063
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,25 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+testlibpqbatch.o: testlibpqbatch.c
+testlibpqbatch: testlibpqbatch.o
+check: testlibpqbatch prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000..d8174dd
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000..add5d23
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,26 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 7;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_batch simple_batch multi_batch batch_abort timings copyfailure test_singlerowmode);
+#my @tests = qw(test_singlerowmode);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(['testlibpqbatch', 'dbname=postgres', "$numrows", "$testname"],
+ "testlibpqbatch $testname");
+}
+
+#$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000..1134bd1
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1661 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution functionality
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+static void test_copyfailure(PGconn *conn);
+static void test_singlerowmode(PGconn *conn);
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+ fflush(stderr);
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode\n");
+ goto fail;
+ }
+
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode with work in progress should fail, but succeeded\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending a batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * in batch mode we have to ask for the first result to be processed;
+ * until we do PQgetResult will return null:
+ */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* We can't PQbatchQueueProcess when there might still be pending results */
+ if (PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() should've failed with pending results: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ /* should now get an explicit sync result */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after end batch call\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+ fflush(stderr);
+
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode: %u\n", __LINE__);
+ goto fail;
+ }
+
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Unable to enter batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not activated properly\n");
+ goto fail;
+ }
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "PQexec should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ {
+ fprintf(stderr, "PQsendQuery should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "re-entering batch mode should be a no-op but failed\n");
+ goto fail;
+ }
+
+ if (PQisBusy(conn))
+ {
+ fprintf(stderr, "PQisBusy should return false when idle in batch, returned true\n");
+ goto fail;
+ }
+
+ /* ok, back to normal command mode */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "couldn't exit idle empty batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not terminated properly\n");
+ goto fail;
+ }
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "batch mode exit when not in batch mode should succeed but failed\n");
+ goto fail;
+ }
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+static void
+test_copyfailure(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *create_sql = "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer); INSERT INTO batch_demo VALUES(5,10); ";
+ const char *select_sql = "select id from batch_demo;";
+ const char *copy_sql = "copy batch_demo(id) to stdout;";
+ const char *copyfrom_sql = "copy batch_demo(itemno) FROM stdin;";
+ int ret;
+ char *copybuf;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Failed to enter batch mode first time\n");
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, copy_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching COPY TO query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed while processing COPY TO command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_OUT)
+ {
+ fprintf(stderr, "Wrong state during COPY TO command processing: %s %s\n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, select_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching SELECT query failed: %s\n", PQerrorMessage(conn));
+ }
+
+ for (;;)
+ {
+ ret = PQgetCopyData(conn, ©buf, 0);
+ if (ret < 0)
+ break; /* done or error */
+
+ if (copybuf)
+ {
+ fprintf(stderr, "COPYBUF: %s \n", copybuf);
+ PQfreemem(copybuf);
+ }
+ }
+
+ PQclear(res);
+ res = NULL;
+ res = PQgetResult(conn);
+
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Finish of COPY TO command failed %s :%s", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed in sync command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+ PQclear(res);
+ PQsetnonblocking(conn, 1);
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Failed to enter batch mode\n");
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, copyfrom_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching COPY FROM query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, select_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching SELECT query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* Start processing the batch results */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed while processing COPY FROM command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "Wrong state during COPY command processing: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY FROM command failed: %s \n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ /* Expect a failure here */
+ if (PQresultStatus(res) == PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpectedly COPY FROM finished with %s: %s",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "Batch mode is not aborted after failure\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at SELECT command after copy : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ /* Select query after copy should also fail */
+ if (PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "SELECT - Expected failure, got %s: %s \n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ /* Clean up all the error responses after COPY failure */
+ do
+ {
+ PQbatchQueueProcess(conn);
+ res = PQgetResult(conn);
+ fprintf(stderr, "Error status and message got from server due to COPY command failure are : %s %s \n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ } while (res != NULL);
+
+ PQclear(res);
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "Attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = PQexec(conn, select_sql);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "\nExpected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ return;
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+ fflush(stderr);
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* OK, start processing the batch results */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ /* second batch */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+ fflush(stderr);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "1";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching error select failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "2";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "3";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* second query, caused error */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* third query in batch, the second insert */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at third batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the batch sync */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "sync should've cleared the aborted flag but didn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the insert from the second batch */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first entry in second batch: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first item in second batch\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* the second batch sync */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch sync\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ {
+ fprintf(stderr, "expected only insert with value 3, got %s", val);
+ goto fail;
+ }
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ fprintf(stderr, "expected 1 result, got %d", PQntuples(res));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "xact start failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent BEGIN\n");
+
+ send_step = BI_DROP_TABLE;
+
+ if (!PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent DROP\n");
+
+ send_step = BI_CREATE_TABLE;
+
+ if (!PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent CREATE\n");
+
+ send_step = BI_PREPARE;
+
+ if (!PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids))
+ {
+ fprintf(stderr, "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent PREPARE\n");
+
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ {
+ fprintf(stderr, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "Expected next query result but unable to dequeue: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ fprintf(stdout, "next query!\n");
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+
+ if (PQresultStatus(res) != status)
+ {
+ fprintf(stderr, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ {
+ fprintf(stderr, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+ goto fail;
+ }
+
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchQueueSync(conn))
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ {
+ fprintf(stderr, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "prepare failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "COPY: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[12 + 2];
+ int formatted = snprintf(&buf[0], 12 + 1, "%d\n", nrows);
+
+ if (formatted >= 12 + 1)
+ {
+ fprintf(stderr, "Buffer write truncated somehow\n");
+ goto fail;
+ }
+
+ if (PQputCopyData(conn, buf, formatted) != 1)
+ {
+ fprintf(stderr, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ nrows--;
+ }
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ struct timeval start_time,
+ end_time,
+ elapsed_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_pipelined(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("batch insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_sequential(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("sequential insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_copy(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("COPY elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings\n");
+ exit(1);
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i,r;
+
+ /* 1 batch, 3 queries in it */
+ r = PQbatchBegin(conn);
+
+ for (i=0; i < 3; i++) {
+ r = PQsendQueryParams(conn,
+ "SELECT 1",
+ 0,
+ NULL,
+ NULL,
+ NULL,
+ NULL,
+ 0);
+ }
+ PQbatchQueueSync(conn);
+
+ i=0;
+ while (PQbatchQueueProcess(conn))
+ {
+ r = PQsetSingleRowMode(conn);
+ if (r!=1)
+ {
+ fprintf(stderr, "PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+ fprintf(stderr, "Result status: %d (%s) for i=%d", est, PQresStatus(est), i);
+ if (est == PGRES_TUPLES_OK)
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ else if (est == PGRES_SINGLE_TUPLE)
+ fprintf(stderr, ", single tuple: %d\n", PQntuples(res));
+ else if (est == PGRES_BATCH_END)
+ fprintf(stderr, ", end of batch reached\n");
+ else if (est != PGRES_COMMAND_OK)
+ fprintf(stderr, ", error: %s\n", PQresultErrorMessage(res));
+ PQclear(res);
+ }
+ i++;
+ }
+ PQbatchEnd(conn);
+}
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1,
+ run_copyfailure = 1,
+ run_singlerowmode = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ run_copyfailure = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else if (strcmp(argv[3], "copyfailure") == 0)
+ run_copyfailure = 1;
+ else if (strcmp(argv[3], "test_singlerowmode") == 0)
+ run_singlerowmode = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognized test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+ if (run_copyfailure)
+ test_copyfailure(conn);
+
+ if(run_singlerowmode)
+ test_singlerowmode(conn);
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
--
2.7.4.windows.1
Re: PATCH: Batch/pipelining support for libpq
On Tue, Mar 28, 2017 at 1:57 PM, Vaishnavi Prabakaran
<vaishnaviprabakaran@gmail.com> wrote:
Thanks Craig and Michael for confirming that "PQsetSingleRowMode" should be
called right after "PQbatchQueueProcess".Michael Paquier wrote:
It seems to me that
this should also be effective only during the fetching of one single
result set. When the client moves on to the next item in the queue we
should make necessary again a call to PQsetSingleRowMode().Yes, the current behavior(with V6 code patch) is exactly the same as you
described above. PQsetSingleRowMode() should be called each time after
"PQbatchQueueProcess" to set result processing to single-row mode.
Okay, that's fine for me then.
Attached the latest patch and here is the RT run result:
ok 1 - testlibpqbatch disallowed_in_batch
ok 2 - testlibpqbatch simple_batch
ok 3 - testlibpqbatch multi_batch
ok 4 - testlibpqbatch batch_abort
ok 5 - testlibpqbatch timings
ok 6 - testlibpqbatch copyfailure
ok 7 - testlibpqbatch test_singlerowmode
ok
All tests successful.
Files=1, Tests=7, 5 wallclock secs ( 0.01 usr 0.00 sys + 1.79 cusr 0.35
csys = 2.15 CPU)
Result: PASS
ok 1 - testlibpqbatch disallowed_in_batch
ok 2 - testlibpqbatch simple_batch
ok 3 - testlibpqbatch multi_batch
ok 4 - testlibpqbatch batch_abort
ok 5 - testlibpqbatch timings
not ok 6 - testlibpqbatch copyfailure
Hm... Not here. I saw something like that a couple of days ago on my
macos laptop and that was related to a variable not initialized. From
001_libpq_async_main.log:
7-03-28 17:05:49.159 JST [31553] t/001_libpq_async.pl LOG: execute
<unnamed>: copy batch_demo(id) to stdout;
2017-03-28 17:05:49.159 JST [31553] t/001_libpq_async.pl LOG: execute
<unnamed>: copy batch_demo(itemno) FROM stdin;
2017-03-28 17:05:49.160 JST [31553] t/001_libpq_async.pl ERROR:
unexpected message type 0x50 during COPY from stdin
2017-03-28 17:05:49.160 JST [31553] t/001_libpq_async.pl CONTEXT:
COPY batch_demo, line 1
From regress_log_001_libpq_async :
ok 5 - testlibpqbatch timings
# Running: testlibpqbatch dbname=postgres 10000 copyfailure
dispatching SELECT query failed: cannot queue commands during COPY
COPYBUF: 5
Error status and message got from server due to COPY command failure
are : PGRES_FATAL_ERROR ERROR: unexpected message type 0x50 during
COPY from stdin
CONTEXT: COPY batch_demo, line 1
So it seems to me that you are still missing something..
src/test/modules/Makefile | 1 +
src/test/modules/test_libpq/.gitignore | 5 +
src/test/modules/test_libpq/Makefile | 25 +
src/test/modules/test_libpq/README | 1 +
src/test/modules/test_libpq/t/001_libpq_async.pl | 26 +
src/test/modules/test_libpq/testlibpqbatch.c | 1661 ++++++++++++++++++++++
6 files changed, 1719 insertions(+)
Could you as well update src/tools/msvc/vcregress.pl, aka the routine
modulescheck so as this new test is skipped. I am sure that nobody
will scream if this test is not run on Windows, but the buildfarm will
complain if the patch is let in its current state.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
Michael Paquier wrote:
# Running: testlibpqbatch dbname=postgres 10000 copyfailure
dispatching SELECT query failed: cannot queue commands during COPYCOPYBUF: 5
Error status and message got from server due to COPY command failure
are : PGRES_FATAL_ERROR ERROR: unexpected message type 0x50 during
COPY from stdin
CONTEXT: COPY batch_demo, line 1
Same result here.
BTW the doc says:
"In batch mode only asynchronous operations are permitted, and COPY is
not recommended as it most likely will trigger failure in batch
processing"
Yet it seems that the test assumes that it should work nonetheless.
I don't quite understand what we expect of this test, given what's
documented. Or what am I missing?
While looking at the regress log, I noticed multiple spurious
test_singlerowmode tests among the others. Should be fixed with:
--- a/src/test/modules/test_libpq/testlibpqbatch.c
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -1578,6 +1578,7 @@ main(int argc, char **argv)
run_batch_abort = 0;
run_timings = 0;
run_copyfailure = 0;
+ run_singlerowmode = 0;
if (strcmp(argv[3], "disallowed_in_batch") == 0)
run_disallowed_in_batch = 1;
else if (strcmp(argv[3], "simple_batch") == 0)There's also the fact that this test doesn't care whether it fails
(compare with all the "goto fail" of the other tests).
And it happens that PQsetSingleRowMode() fails after the call to
PQbatchQueueProcess() that instantiates a PGresult
of status PGRES_BATCH_END to indicate the end of the batch,
To me this shows how this way of setting the single row mode is not ideal.
In order to avoid the failure, the loop should know in advance
what kind of result it's going to dequeue before calling PQgetResult(),
which doesn't feel right.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
Michael Paquier wrote:
Could you as well update src/tools/msvc/vcregress.pl, aka the routine
modulescheck so as this new test is skipped. I am sure that nobody
will scream if this test is not run on Windows, but the buildfarm will
complain if the patch is let in its current state.
Thanks for the finding. Hmm, modulescheck will not pick up test_libpq as
"subdircheck" will fail for this module(because it does not have
"sql"/"expected" folders in it) and hence it will be skipped.
But, Modified "Mkvcbuild.pm" to exclude "test_libpq" module, as this test
anyways will not be run in Windows and also because it uses linux specific
APIs(gettimeofday , timersub) .
On Wed, Mar 29, 2017 at 12:28 AM, Daniel Verite <daniel@manitou-mail.org>
wrote:
Michael Paquier wrote:
# Running: testlibpqbatch dbname=postgres 10000 copyfailure
dispatching SELECT query failed: cannot queue commands during COPYCOPYBUF: 5
Error status and message got from server due to COPY command failure
are : PGRES_FATAL_ERROR ERROR: unexpected message type 0x50 during
COPY from stdin
CONTEXT: COPY batch_demo, line 1Same result here.
BTW the doc says:
"In batch mode only asynchronous operations are permitted, and COPY is
not recommended as it most likely will trigger failure in batch
processing"
Yet it seems that the test assumes that it should work nonetheless.
I don't quite understand what we expect of this test, given what's
documented. Or what am I missing?
copyfailure test is basically tests the error scenarios arising in batch
mode on using copy command. And, test expects error(not that it should
work) and the test is made to pass if expected error message is received.
So, it is error case testing behaves like:
expects error -> receives error = test pass
expects error -> no error = test fail
There are 2 cases tested here:
1. Example for the case - Using COPY command in batch mode will trigger
failure. (Specified in documentation)
Failure can be seen only when processing the copy command's results. That
is, after dispatching COPY command, server goes into COPY state , but the
client dispatched another query in batch mode.
Below error messages belongs to this case :
Error status and message got from server due to COPY command failure are :
PGRES_FATAL_ERROR ERROR: unexpected message type 0x50 during COPY from
stdin
CONTEXT: COPY batch_demo, line 1
message type 0x5a arrived from server while idle
2. When connection is is copy state, that is, while processing copy
command's result, Cannot queue any query in batch mode.
Below error message belongs to this case:
dispatching SELECT query failed: cannot queue commands during COPY
I added this test following Michael's review comment - "It may be a good
idea to add a test for COPY and trigger a failure."
Hope this clarifies the presence of error message in log file and why the
test is made to pass.
While looking at the regress log, I noticed multiple spurious
test_singlerowmode tests among the others. Should be fixed with:--- a/src/test/modules/test_libpq/testlibpqbatch.c +++ b/src/test/modules/test_libpq/testlibpqbatch.c @@ -1578,6 +1578,7 @@ main(int argc, char **argv) run_batch_abort = 0; run_timings = 0; run_copyfailure = 0; + run_singlerowmode = 0; if (strcmp(argv[3], "disallowed_in_batch") == 0) run_disallowed_in_batch = 1; else if (strcmp(argv[3], "simple_batch") == 0)
Thanks for finding it out and yes corrected.
There's also the fact that this test doesn't care whether it fails
(compare with all the "goto fail" of the other tests).
Modified the test to check for failure condition.
And it happens that PQsetSingleRowMode() fails after the call to
PQbatchQueueProcess() that instantiates a PGresult
of status PGRES_BATCH_END to indicate the end of the batch,To me this shows how this way of setting the single row mode is not ideal.
In order to avoid the failure, the loop should know in advance
what kind of result it's going to dequeue before calling PQgetResult(),
which doesn't feel right.
Yes, it is one of the requirement that the batch mode application should
know the order of queries it sent and the expected results. (Specified in
"Processing results" section of batch mode documentation)
Attached the latest code and test patches.
Thanks & Regards,
Vaishnavi
Fujitusu Australia.
Attachments:
0001-Pipelining-batch-support-for-libpq-code-v8.patchapplication/octet-stream; name=0001-Pipelining-batch-support-for-libpq-code-v8.patchDownload
From 04c69538495ffa10ce89c198d12df042ce5f7084 Mon Sep 17 00:00:00 2001
From: Vaishnavi Prabakaran <vaishnavip@fast.au.fujitsu.com>
Date: Wed, 29 Mar 2017 13:38:47 +1100
Subject: [PATCH 1/2] Pipelining-batch-support-for-libpq-code-v8
---
doc/src/sgml/libpq.sgml | 529 ++++++++++++++++++++++++++++++++
doc/src/sgml/lobj.sgml | 4 +
src/interfaces/libpq/exports.txt | 6 +
src/interfaces/libpq/fe-connect.c | 28 ++
src/interfaces/libpq/fe-exec.c | 594 ++++++++++++++++++++++++++++++++++--
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 15 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 41 ++-
9 files changed, 1208 insertions(+), 39 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 4bc5bf3..3b7e37b 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4655,6 +4655,527 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up multiple queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ offers considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ <application>libpq</application> into batch mode. Enter batch mode with <link
+ linkend="libpq-PQbatchBegin"><function>PQbatchBegin(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ (The restriction on <literal>COPY</literal> is an implementation
+ limit; the PostgreSQL protocol and server can support batched <literal>COPY</literal>).
+ Using any synchronous command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error conditions.
+ Functions allowed in batch mode are described in <xref linkend="libpq-batch-sending">.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchQueueSync</function>.
+ Concurrently, it uses <function>PQgetResult</function> and
+ <function>PQbatchQueueProcess</function> to get results. It may eventually exit
+ batch mode with <function>PQbatchEnd</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchQueueSync"><function>PQbatchQueueSync(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing with sending batch queries</link>, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchQueueProcess"><function>PQbatchQueueProcess</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null (or would return null if
+ called). The result from the next batch entry may then be retrieved using
+ <function>PQbatchQueueProcess</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
+ successful call of <function>PQbatchQueueProcess</function>. This mode selection is effective
+ only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode">.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchQueueSync</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchQueueSync</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchQueueProcess(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQbatchEnd"><function>PQbatchEnd(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchQueueSync</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchBegin">
+ <term>
+ <function>PQbatchBegin</function>
+ <indexterm>
+ <primary>PQbatchBegin</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQbatchBegin(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchEnd">
+ <term>
+ <function>PQbatchEnd</function>
+ <indexterm>
+ <primary>PQbatchEnd</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQbatchEnd(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchQueueProcess</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueSync">
+ <term>
+ <function>PQbatchQueueSync</function>
+ <indexterm>
+ <primary>PQbatchQueueSync</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchQueueSync(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueProcess">
+ <term>
+ <function>PQbatchQueueProcess</function>
+ <indexterm>
+ <primary>PQbatchQueueProcess</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchQueueProcess(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchQueueCount</function>, <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueCount">
+ <term>
+ <function>PQbatchQueueCount</function>
+ <indexterm>
+ <primary>PQbatchQueueCount</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processed.
+ This is the number of times <function>PQbatchQueueProcess</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQbatchQueueCount(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
@@ -4695,6 +5216,14 @@ int PQflush(PGconn *conn);
Each object should be freed with <function>PQclear</function> as usual.
</para>
+ <note>
+ <para>
+ On using batch mode, call <function>PQsetSingleRowMode</function>
+ immediately after a successful call of <function>PQbatchQueueProcess</function>
+ See <xref linkend="libpq-batch-mode"> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-pqsetsinglerowmode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 7757e1e..db8523d 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 21dd772..e9f81b3 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -171,3 +171,9 @@ PQsslAttributeNames 168
PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
+PQbatchQueueCount 172
+PQbatchBegin 173
+PQbatchEnd 174
+PQbatchQueueSync 175
+PQbatchQueueProcess 176
+PQbatchStatus 177
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 27155f8..bb10b89 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3459,6 +3459,25 @@ sendTerminateConn(PGconn *conn)
}
/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
+
+/*
* closePGconn
* - properly close a connection to the backend
*
@@ -3470,6 +3489,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3502,6 +3522,14 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9decd53..b0f5ce7 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -69,6 +71,9 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
/* ----------------
@@ -1108,7 +1113,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1131,6 +1136,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1229,6 +1241,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1287,18 +1303,34 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1308,10 +1340,14 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1359,7 +1395,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, "tried to recycle non-dangling command queue entry");
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1377,20 +1486,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1414,6 +1563,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1423,6 +1576,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1535,22 +1705,25 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1560,10 +1733,14 @@ PQsendQueryGuts(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1690,6 +1867,297 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/*
+ * PQbatchQueueCount
+ * Return number of queries currently pending in batch mode
+ */
+int
+PQbatchQueueCount(PGconn *conn)
+{
+ int count = 0;
+ PGcommandQueueEntry *entry;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ return 0;
+
+ entry = conn->cmd_queue_head;
+ while (entry != NULL)
+ {
+ ++count;
+ entry = entry->next;
+ }
+ return count;
+}
+
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQbatchBegin
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQbatchBegin(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return true;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return true;
+}
+
+/*
+ * PQbatchEnd
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns true if batch mode ended.
+ */
+int
+PQbatchEnd(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return true;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ return false;
+ case PGASYNC_QUEUED:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ return true;
+}
+
+/*
+ * PQbatchQueueSync
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new non-batch commands until all
+ * results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchQueueSync(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return false; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /* Should try to flush immediately if there's room */
+ PQflush(conn);
+
+ return true;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/*
+ * PQbatchQueueProcess
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns true if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns false if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchQueueProcess(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return false;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return true;
+}
+
/*
* PQgetResult
@@ -1749,10 +2217,32 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * batched queries aren't followed by a Sync to put us back in
+ * PGASYNC_IDLE state, and when we do get a sync we could
+ * still have another batch coming after this one.
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1932,6 +2422,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2126,6 +2623,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2141,6 +2641,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2149,15 +2663,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2171,10 +2688,14 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -2569,6 +3090,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 3b0500f..c01f1a2 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 53776e2..e24d7ce 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 635af5b..737264d 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQbatchBegin(PGconn *conn);
+extern int PQbatchEnd(PGconn *conn);
+extern int PQbatchQueueSync(PGconn *conn);
+extern int PQbatchQueueProcess(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index b8ec341..33f212f 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -215,10 +215,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -227,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -297,6 +303,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -384,6 +406,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY
@@ -396,6 +419,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -693,6 +726,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.7.4.windows.1
0002-Pipelining-batch-support-for-libpq-test-v5.patchapplication/octet-stream; name=0002-Pipelining-batch-support-for-libpq-test-v5.patchDownload
From c9985df208c0cf14b5e99d21aa85f133f10572e1 Mon Sep 17 00:00:00 2001
From: Vaishnavi Prabakaran <vaishnavip@fast.au.fujitsu.com>
Date: Wed, 29 Mar 2017 13:39:05 +1100
Subject: [PATCH 2/2] Pipelining-batch-support-for-libpq-test-v5
---
src/test/modules/Makefile | 1 +
src/test/modules/test_libpq/.gitignore | 5 +
src/test/modules/test_libpq/Makefile | 25 +
src/test/modules/test_libpq/README | 1 +
src/test/modules/test_libpq/t/001_libpq_async.pl | 25 +
src/test/modules/test_libpq/testlibpqbatch.c | 1688 ++++++++++++++++++++++
src/tools/msvc/Mkvcbuild.pm | 2 +-
7 files changed, 1746 insertions(+), 1 deletion(-)
create mode 100644 src/test/modules/test_libpq/.gitignore
create mode 100644 src/test/modules/test_libpq/Makefile
create mode 100644 src/test/modules/test_libpq/README
create mode 100644 src/test/modules/test_libpq/t/001_libpq_async.pl
create mode 100644 src/test/modules/test_libpq/testlibpqbatch.c
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 3ce9904..310d3ba 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -11,6 +11,7 @@ SUBDIRS = \
snapshot_too_old \
test_ddl_deparse \
test_extensions \
+ test_libpq \
test_parser \
test_pg_dump \
test_rls_hooks \
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000..11e8463
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000..d907063
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,25 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+testlibpqbatch.o: testlibpqbatch.c
+testlibpqbatch: testlibpqbatch.o
+check: testlibpqbatch prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000..d8174dd
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000..bff2c0d
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,25 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 7;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_batch simple_batch multi_batch batch_abort timings copyfailure singlerowmode);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(['testlibpqbatch', 'dbname=postgres', "$numrows", "$testname"],
+ "testlibpqbatch $testname");
+}
+
+$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000..7a1c004
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1688 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution functionality
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+static void test_copyfailure(PGconn *conn);
+static void test_singlerowmode(PGconn *conn);
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+ fflush(stderr);
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode\n");
+ goto fail;
+ }
+
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode with work in progress should fail, but succeeded\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending a batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * in batch mode we have to ask for the first result to be processed;
+ * until we do PQgetResult will return null:
+ */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* We can't PQbatchQueueProcess when there might still be pending results */
+ if (PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() should've failed with pending results: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ /* should now get an explicit sync result */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after end batch call\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+ fflush(stderr);
+
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode: %u\n", __LINE__);
+ goto fail;
+ }
+
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Unable to enter batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not activated properly\n");
+ goto fail;
+ }
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "PQexec should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ {
+ fprintf(stderr, "PQsendQuery should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "re-entering batch mode should be a no-op but failed\n");
+ goto fail;
+ }
+
+ if (PQisBusy(conn))
+ {
+ fprintf(stderr, "PQisBusy should return false when idle in batch, returned true\n");
+ goto fail;
+ }
+
+ /* ok, back to normal command mode */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "couldn't exit idle empty batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not terminated properly\n");
+ goto fail;
+ }
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "batch mode exit when not in batch mode should succeed but failed\n");
+ goto fail;
+ }
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+static void
+test_copyfailure(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *create_sql = "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer); INSERT INTO batch_demo VALUES(5,10); ";
+ const char *select_sql = "select id from batch_demo;";
+ const char *copy_sql = "copy batch_demo(id) to stdout;";
+ const char *copyfrom_sql = "copy batch_demo(itemno) FROM stdin;";
+ int ret;
+ char *copybuf;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Failed to enter batch mode first time\n");
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, copy_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching COPY TO query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed while processing COPY TO command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_OUT)
+ {
+ fprintf(stderr, "Wrong state during COPY TO command processing: %s %s\n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, select_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching SELECT query failed: %s\n", PQerrorMessage(conn));
+ }
+
+ for (;;)
+ {
+ ret = PQgetCopyData(conn, ©buf, 0);
+ if (ret < 0)
+ break; /* done or error */
+
+ if (copybuf)
+ {
+ fprintf(stderr, "COPYBUF: %s \n", copybuf);
+ PQfreemem(copybuf);
+ }
+ }
+
+ PQclear(res);
+ res = NULL;
+ res = PQgetResult(conn);
+
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Finish of COPY TO command failed %s :%s", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed in sync command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+ PQclear(res);
+ PQsetnonblocking(conn, 1);
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Failed to enter batch mode\n");
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, copyfrom_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching COPY FROM query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, select_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching SELECT query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* Start processing the batch results */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed while processing COPY FROM command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "Wrong state during COPY command processing: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY FROM command failed: %s \n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ /* Expect a failure here */
+ if (PQresultStatus(res) == PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpectedly COPY FROM finished with %s: %s",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "Batch mode is not aborted after failure\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at SELECT command after copy : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ /* Select query after copy should also fail */
+ if (PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "SELECT - Expected failure, got %s: %s \n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ /* Clean up all the error responses after COPY failure */
+ do
+ {
+ PQbatchQueueProcess(conn);
+ res = PQgetResult(conn);
+ fprintf(stderr, "Error status and message got from server due to COPY command failure are : %s %s \n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ } while (res != NULL);
+
+ PQclear(res);
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "Attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = PQexec(conn, select_sql);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "\nExpected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ return;
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+ fflush(stderr);
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* OK, start processing the batch results */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ /* second batch */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+ fflush(stderr);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "1";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching error select failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "2";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "3";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* second query, caused error */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* third query in batch, the second insert */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at third batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the batch sync */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "sync should've cleared the aborted flag but didn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the insert from the second batch */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first entry in second batch: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first item in second batch\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* the second batch sync */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch sync\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ {
+ fprintf(stderr, "expected only insert with value 3, got %s", val);
+ goto fail;
+ }
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ fprintf(stderr, "expected 1 result, got %d", PQntuples(res));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "xact start failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent BEGIN\n");
+
+ send_step = BI_DROP_TABLE;
+
+ if (!PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent DROP\n");
+
+ send_step = BI_CREATE_TABLE;
+
+ if (!PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent CREATE\n");
+
+ send_step = BI_PREPARE;
+
+ if (!PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids))
+ {
+ fprintf(stderr, "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent PREPARE\n");
+
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ {
+ fprintf(stderr, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "Expected next query result but unable to dequeue: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ fprintf(stdout, "next query!\n");
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+
+ if (PQresultStatus(res) != status)
+ {
+ fprintf(stderr, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ {
+ fprintf(stderr, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+ goto fail;
+ }
+
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchQueueSync(conn))
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ {
+ fprintf(stderr, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "prepare failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "COPY: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[12 + 2];
+ int formatted = snprintf(&buf[0], 12 + 1, "%d\n", nrows);
+
+ if (formatted >= 12 + 1)
+ {
+ fprintf(stderr, "Buffer write truncated somehow\n");
+ goto fail;
+ }
+
+ if (PQputCopyData(conn, buf, formatted) != 1)
+ {
+ fprintf(stderr, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ nrows--;
+ }
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ struct timeval start_time,
+ end_time,
+ elapsed_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_pipelined(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("batch insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_sequential(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("sequential insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_copy(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("COPY elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings|copyfailure|singlerowmode\n");
+ exit(1);
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i,r;
+
+ /* 1 batch, 3 queries in it */
+ r = PQbatchBegin(conn);
+
+ for (i=0; i < 3; i++) {
+ r = PQsendQueryParams(conn,
+ "SELECT 1",
+ 0,
+ NULL,
+ NULL,
+ NULL,
+ NULL,
+ 0);
+ }
+ PQbatchQueueSync(conn);
+
+ i=0;
+ while (PQbatchQueueProcess(conn))
+ {
+ int isSingleTuple = 0;
+ /* Set single row mode for only first 3 SELECT queries */
+ if(i < 3)
+ r = PQsetSingleRowMode(conn);
+ if (r!=1)
+ {
+ fprintf(stderr, "PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+ fprintf(stderr, "Result status: %d (%s) for i=%d", est, PQresStatus(est), i);
+ if (est == PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ if(!isSingleTuple)
+ {
+ fprintf(stderr, " Expected to follow PGREG_SINGLE_TUPLE, but received PGRES_TUPLES_OK directly instead\n");
+ goto fail;
+ }
+ isSingleTuple=0;
+ }
+ else if (est == PGRES_SINGLE_TUPLE)
+ {
+ isSingleTuple = 1;
+ fprintf(stderr, ", single tuple: %d\n", PQntuples(res));
+ }
+ else if (est == PGRES_BATCH_END)
+ {
+ fprintf(stderr, ", end of batch reached\n");
+ }
+ else if (est != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, ", error: %s\n", PQresultErrorMessage(res));
+ }
+ PQclear(res);
+ }
+ i++;
+ }
+ PQbatchEnd(conn);
+ PQclear(res);
+ res = NULL;
+ return;
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+
+}
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1,
+ run_copyfailure = 1,
+ run_singlerowmode = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ run_copyfailure = 0;
+ run_singlerowmode = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else if (strcmp(argv[3], "copyfailure") == 0)
+ run_copyfailure = 1;
+ else if (strcmp(argv[3], "singlerowmode") == 0)
+ run_singlerowmode = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognized test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+ if (run_copyfailure)
+ test_copyfailure(conn);
+
+ if(run_singlerowmode)
+ test_singlerowmode(conn);
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index ba1bf6d..4a2a4dd 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -46,7 +46,7 @@ my @contrib_excludes = (
'ltree_plpython', 'pgcrypto',
'sepgsql', 'brin',
'test_extensions', 'test_pg_dump',
- 'snapshot_too_old');
+ 'snapshot_too_old', 'test_libpq');
# Set of variables for frontend modules
my $frontend_defines = { 'initdb' => 'FRONTEND' };
--
2.7.4.windows.1
Re: PATCH: Batch/pipelining support for libpq
On Wed, Mar 29, 2017 at 12:40 PM, Vaishnavi Prabakaran
<vaishnaviprabakaran@gmail.com> wrote:
Michael Paquier wrote:
Could you as well update src/tools/msvc/vcregress.pl, aka the routine
modulescheck so as this new test is skipped. I am sure that nobody
will scream if this test is not run on Windows, but the buildfarm will
complain if the patch is let in its current state.Thanks for the finding. Hmm, modulescheck will not pick up test_libpq as
"subdircheck" will fail for this module(because it does not have
"sql"/"expected" folders in it) and hence it will be skipped.
But, Modified "Mkvcbuild.pm" to exclude "test_libpq" module, as this test
anyways will not be run in Windows and also because it uses linux specific
APIs(gettimeofday , timersub).
src/port/gettimeofday.c extends things on Windows, and we could
consider to just drop the timing portion for simplicity (except
Windows I am not seeing any platform missing timersub). But that's
just a point of detail. Or we could just drop those tests from the
final patch, and re-implement them after having some psql-level
subcommands. That would far better for portability.
There are 2 cases tested here:
1. Example for the case - Using COPY command in batch mode will trigger
failure. (Specified in documentation)
Failure can be seen only when processing the copy command's results. That
is, after dispatching COPY command, server goes into COPY state , but the
client dispatched another query in batch mode.Below error messages belongs to this case :
Error status and message got from server due to COPY command failure are :
PGRES_FATAL_ERROR ERROR: unexpected message type 0x50 during COPY from
stdin
CONTEXT: COPY batch_demo, line 1message type 0x5a arrived from server while idle
Such messages are really confusing to the user as they refer to the
protocol going out of sync. I would think that something like "cannot
process COPY results during a batch processing" would be cleaner.
Isn't some more error handling necessary in GetCopyStart()?
Attached the latest code and test patches.
For me the test is still broken:
ok 1 - testlibpqbatch disallowed_in_batch
ok 2 - testlibpqbatch simple_batch
ok 3 - testlibpqbatch multi_batch
ok 4 - testlibpqbatch batch_abort
ok 5 - testlibpqbatch timings
not ok 6 - testlibpqbatch copyfailure
Still broken here. I can see that this patch is having enough
safeguards in PQbatchBegin() and PQsendQueryStart(), but this issue is
really pointing out at something...
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On Thu, Mar 30, 2017 at 12:08 PM, Michael Paquier <michael.paquier@gmail.com
wrote:
On Wed, Mar 29, 2017 at 12:40 PM, Vaishnavi Prabakaran
<vaishnaviprabakaran@gmail.com> wrote:Michael Paquier wrote:
Could you as well update src/tools/msvc/vcregress.pl, aka the routine
modulescheck so as this new test is skipped. I am sure that nobody
will scream if this test is not run on Windows, but the buildfarm will
complain if the patch is let in its current state.Thanks for the finding. Hmm, modulescheck will not pick up test_libpq as
"subdircheck" will fail for this module(because it does not have
"sql"/"expected" folders in it) and hence it will be skipped.
But, Modified "Mkvcbuild.pm" to exclude "test_libpq" module, as this test
anyways will not be run in Windows and also because it uses linuxspecific
APIs(gettimeofday , timersub).
src/port/gettimeofday.c extends things on Windows, and we could
consider to just drop the timing portion for simplicity (except
Windows I am not seeing any platform missing timersub). But that's
just a point of detail. Or we could just drop those tests from the
final patch, and re-implement them after having some psql-level
subcommands. That would far better for portability.
Yes agree, having tests with psql meta commands will be more easy to
understand also. And, that is one of the reason I separated the patch into
two - code and test . So, yes, we can drop the test patch for now.
There are 2 cases tested here:
1. Example for the case - Using COPY command in batch mode will trigger
failure. (Specified in documentation)
Failure can be seen only when processing the copy command's results. That
is, after dispatching COPY command, server goes into COPY state , but the
client dispatched another query in batch mode.Below error messages belongs to this case :
Error status and message got from server due to COPY command failure are:
PGRES_FATAL_ERROR ERROR: unexpected message type 0x50 during COPY from
stdin
CONTEXT: COPY batch_demo, line 1message type 0x5a arrived from server while idle
Such messages are really confusing to the user as they refer to the
protocol going out of sync. I would think that something like "cannot
process COPY results during a batch processing" would be cleaner.
Isn't some more error handling necessary in GetCopyStart()?
Hmm, With batch mode, after sending COPY command to server(and server
started processing the query and goes into COPY state) , client does not
immediately read the result , but it keeps sending other queries to the
server. By this time, server already encountered the error
scenario(Receiving different message during COPY state) and sent error
messages. So, when client starts reading the result, already error messages
sent by server are present in socket. I think trying to handle during
result processing is more like masking the server's error message with new
error message and I think it is not appropriate.
Attached the latest code and test patches.
For me the test is still broken:
ok 1 - testlibpqbatch disallowed_in_batch
ok 2 - testlibpqbatch simple_batch
ok 3 - testlibpqbatch multi_batch
ok 4 - testlibpqbatch batch_abort
ok 5 - testlibpqbatch timings
not ok 6 - testlibpqbatch copyfailureStill broken here. I can see that this patch is having enough
safeguards in PQbatchBegin() and PQsendQueryStart(), but this issue is
really pointing out at something...
I will check this one with fresh setup again and I guess that this should
not block the code patch.
Thanks & Regards,
Vaishnavi
Fujitsu Australia.
Re: PATCH: Batch/pipelining support for libpq
Vaishnavi Prabakaran wrote:
Hmm, With batch mode, after sending COPY command to server(and server
started processing the query and goes into COPY state) , client does not
immediately read the result , but it keeps sending other queries to the
server. By this time, server already encountered the error
scenario(Receiving different message during COPY state) and sent error
messages
IOW, the test intentionally violates the protocol and then all goes wonky
because of that.
That's why I was wondering upthread what's it's supposed to test.
I mean, regression tests are meant to warn against a desirable behavior
being unknowingly changed by new code into an undesirable behavior.
Here we have the undesirable behavior to start with.
What kind of regression could we fear from that?
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
Hi,
On 3/30/17 2:12 PM, Daniel Verite wrote:
Vaishnavi Prabakaran wrote:
Hmm, With batch mode, after sending COPY command to server(and server
started processing the query and goes into COPY state) , client does not
immediately read the result , but it keeps sending other queries to the
server. By this time, server already encountered the error
scenario(Receiving different message during COPY state) and sent error
messagesIOW, the test intentionally violates the protocol and then all goes wonky
because of that.
That's why I was wondering upthread what's it's supposed to test.
I mean, regression tests are meant to warn against a desirable behavior
being unknowingly changed by new code into an undesirable behavior.
Here we have the undesirable behavior to start with.
What kind of regression could we fear from that?
The CF has been extended until April 7 but time is still growing short.
Please respond with a new patch by 2017-04-04 00:00 AoE (UTC-12) or this
submission will be marked "Returned with Feedback".
Thanks,
--
-David
david@pgmasters.net
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On Sat, Apr 1, 2017 at 2:03 AM, David Steele <david@pgmasters.net> wrote:
Hi,
On 3/30/17 2:12 PM, Daniel Verite wrote:
Vaishnavi Prabakaran wrote:
Hmm, With batch mode, after sending COPY command to server(and server
started processing the query and goes into COPY state) , client does not
immediately read the result , but it keeps sending other queries to the
server. By this time, server already encountered the error
scenario(Receiving different message during COPY state) and sent error
messagesIOW, the test intentionally violates the protocol and then all goes wonky
because of that.
That's why I was wondering upthread what's it's supposed to test.
I mean, regression tests are meant to warn against a desirable behavior
being unknowingly changed by new code into an undesirable behavior.
Here we have the undesirable behavior to start with.
What kind of regression could we fear from that?
Yes, completely agree, demonstrating the undesirable behavior is not needed
as documentation gives enough warning to user.
The test patch is decided not to go in for now, but will be re-implemented
with PSQL commands later. So, during the re-implementation of test patch, I
will remove this test. Thanks .
The CF has been extended until April 7 but time is still growing short.
Please respond with a new patch by 2017-04-04 00:00 AoE (UTC-12) or this
submission will be marked "Returned with Feedback".
Thanks for the information, attached the latest patch resolving one
compilation warning. And, please discard the test patch as it will be
re-implemented later separately.
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
Attachments:
0001-Pipelining-batch-support-for-libpq-code-v9.patchapplication/octet-stream; name=0001-Pipelining-batch-support-for-libpq-code-v9.patchDownload
---
doc/src/sgml/libpq.sgml | 529 ++++++++++++++++++
doc/src/sgml/lobj.sgml | 4 +
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/interfaces/libpq/exports.txt | 6 +
src/interfaces/libpq/fe-connect.c | 28 +
src/interfaces/libpq/fe-exec.c | 594 +++++++++++++++++++--
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 15 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 41 +-
10 files changed, 1211 insertions(+), 39 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 4bc5bf3..3b7e37b 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4655,6 +4655,527 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up multiple queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ offers considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ <application>libpq</application> into batch mode. Enter batch mode with <link
+ linkend="libpq-PQbatchBegin"><function>PQbatchBegin(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ (The restriction on <literal>COPY</literal> is an implementation
+ limit; the PostgreSQL protocol and server can support batched <literal>COPY</literal>).
+ Using any synchronous command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error conditions.
+ Functions allowed in batch mode are described in <xref linkend="libpq-batch-sending">.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchQueueSync</function>.
+ Concurrently, it uses <function>PQgetResult</function> and
+ <function>PQbatchQueueProcess</function> to get results. It may eventually exit
+ batch mode with <function>PQbatchEnd</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchQueueSync"><function>PQbatchQueueSync(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing with sending batch queries</link>, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchQueueProcess"><function>PQbatchQueueProcess</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null (or would return null if
+ called). The result from the next batch entry may then be retrieved using
+ <function>PQbatchQueueProcess</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
+ successful call of <function>PQbatchQueueProcess</function>. This mode selection is effective
+ only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode">.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchQueueSync</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchQueueSync</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchQueueProcess(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQbatchEnd"><function>PQbatchEnd(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchQueueSync</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchBegin">
+ <term>
+ <function>PQbatchBegin</function>
+ <indexterm>
+ <primary>PQbatchBegin</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQbatchBegin(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchEnd">
+ <term>
+ <function>PQbatchEnd</function>
+ <indexterm>
+ <primary>PQbatchEnd</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQbatchEnd(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchQueueProcess</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueSync">
+ <term>
+ <function>PQbatchQueueSync</function>
+ <indexterm>
+ <primary>PQbatchQueueSync</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchQueueSync(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueProcess">
+ <term>
+ <function>PQbatchQueueProcess</function>
+ <indexterm>
+ <primary>PQbatchQueueProcess</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchQueueProcess(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchQueueCount</function>, <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueCount">
+ <term>
+ <function>PQbatchQueueCount</function>
+ <indexterm>
+ <primary>PQbatchQueueCount</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processed.
+ This is the number of times <function>PQbatchQueueProcess</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQbatchQueueCount(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
@@ -4695,6 +5216,14 @@ int PQflush(PGconn *conn);
Each object should be freed with <function>PQclear</function> as usual.
</para>
+ <note>
+ <para>
+ On using batch mode, call <function>PQsetSingleRowMode</function>
+ immediately after a successful call of <function>PQbatchQueueProcess</function>
+ See <xref linkend="libpq-batch-mode"> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-pqsetsinglerowmode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 7757e1e..db8523d 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 9d7bb25..a30a032 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -917,6 +917,9 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->status = WALRCV_ERROR;
walres->err = pchomp(PQerrorMessage(conn->streamConn));
break;
+ default:
+ /* This is just to keep compiler quiet */
+ break;
}
PQclear(pgres);
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 21dd772..e9f81b3 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -171,3 +171,9 @@ PQsslAttributeNames 168
PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
+PQbatchQueueCount 172
+PQbatchBegin 173
+PQbatchEnd 174
+PQbatchQueueSync 175
+PQbatchQueueProcess 176
+PQbatchStatus 177
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 27155f8..bb10b89 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3459,6 +3459,25 @@ sendTerminateConn(PGconn *conn)
}
/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
+
+/*
* closePGconn
* - properly close a connection to the backend
*
@@ -3470,6 +3489,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3502,6 +3522,14 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9decd53..b0f5ce7 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -69,6 +71,9 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
/* ----------------
@@ -1108,7 +1113,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1131,6 +1136,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1229,6 +1241,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1287,18 +1303,34 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1308,10 +1340,14 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1359,7 +1395,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, "tried to recycle non-dangling command queue entry");
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1377,20 +1486,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1414,6 +1563,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1423,6 +1576,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1535,22 +1705,25 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1560,10 +1733,14 @@ PQsendQueryGuts(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1690,6 +1867,297 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/*
+ * PQbatchQueueCount
+ * Return number of queries currently pending in batch mode
+ */
+int
+PQbatchQueueCount(PGconn *conn)
+{
+ int count = 0;
+ PGcommandQueueEntry *entry;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ return 0;
+
+ entry = conn->cmd_queue_head;
+ while (entry != NULL)
+ {
+ ++count;
+ entry = entry->next;
+ }
+ return count;
+}
+
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQbatchBegin
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQbatchBegin(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return true;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return true;
+}
+
+/*
+ * PQbatchEnd
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns true if batch mode ended.
+ */
+int
+PQbatchEnd(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return true;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ return false;
+ case PGASYNC_QUEUED:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ return true;
+}
+
+/*
+ * PQbatchQueueSync
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new non-batch commands until all
+ * results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchQueueSync(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return false; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /* Should try to flush immediately if there's room */
+ PQflush(conn);
+
+ return true;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/*
+ * PQbatchQueueProcess
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns true if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns false if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchQueueProcess(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return false;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return true;
+}
+
/*
* PQgetResult
@@ -1749,10 +2217,32 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * batched queries aren't followed by a Sync to put us back in
+ * PGASYNC_IDLE state, and when we do get a sync we could
+ * still have another batch coming after this one.
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1932,6 +2422,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2126,6 +2623,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2141,6 +2641,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2149,15 +2663,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2171,10 +2688,14 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -2569,6 +3090,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 3b0500f..c01f1a2 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 53776e2..e24d7ce 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 635af5b..737264d 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQbatchBegin(PGconn *conn);
+extern int PQbatchEnd(PGconn *conn);
+extern int PQbatchQueueSync(PGconn *conn);
+extern int PQbatchQueueProcess(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index b8ec341..33f212f 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -215,10 +215,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -227,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -297,6 +303,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -384,6 +406,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY
@@ -396,6 +419,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -693,6 +726,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.7.4.windows.1
Re: PATCH: Batch/pipelining support for libpq
On 2017-04-03 14:10:47 +1000, Vaishnavi Prabakaran wrote:
The CF has been extended until April 7 but time is still growing short.
Please respond with a new patch by 2017-04-04 00:00 AoE (UTC-12) or this
submission will be marked "Returned with Feedback".Thanks for the information, attached the latest patch resolving one
compilation warning. And, please discard the test patch as it will be
re-implemented later separately.
Hm. If the tests aren't ready yet, it seems we'll have to move this to
the next CF.
+ <sect1 id="libpq-batch-mode"> + <title>Batch mode and query pipelining</title> + + <indexterm zone="libpq-batch-mode"> + <primary>libpq</primary> + <secondary>batch mode</secondary> + </indexterm> + + <indexterm zone="libpq-batch-mode"> + <primary>libpq</primary> + <secondary>pipelining</secondary> + </indexterm> + + <para> + <application>libpq</application> supports queueing up multiple queries into + a pipeline to be executed as a batch on the server. Batching queries allows + applications to avoid a client/server round-trip after each query to get + the results before issuing the next query. + </para>
"queueing .. into a pipeline" sounds weird to me - but I'm not a native
speaker. Also batching != pipelining.
+ <sect2> + <title>When to use batching</title> + + <para> + Much like asynchronous query mode, there is no performance disadvantage to + using batching and pipelining. It increases client application complexity + and extra caution is required to prevent client/server deadlocks but + offers considerable performance improvements. + </para>
s/offers/can sometimes offer/
+ <sect2 id="libpq-batch-using"> + <title>Using batch mode</title> + + <para> + To issue batches the application must switch + <application>libpq</application> into batch mode.
s/libpq/a connection/?
Enter batch mode with <link + linkend="libpq-PQbatchBegin"><function>PQbatchBegin(conn)</function></link> or test + whether batch mode is active with <link + linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link + linkend="libpq-async">asynchronous operations</link> are permitted, and + <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing. + (The restriction on <literal>COPY</literal> is an implementation + limit; the PostgreSQL protocol and server can support batched <literal>COPY</literal>).
Hm, I'm unconvinced that that's a useful parenthetical in the libpq
docs.
+ <para> + The client uses libpq's asynchronous query functions to dispatch work, + marking the end of each batch with <function>PQbatchQueueSync</function>. + Concurrently, it uses <function>PQgetResult</function> and + <function>PQbatchQueueProcess</function> to get results.
"Concurrently" imo is a dangerous word, somewhat implying multi-threading.
+ <note> + <para> + It is best to use batch mode with <application>libpq</application> in + <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in + blocking mode it is possible for a client/server deadlock to occur. The + client will block trying to send queries to the server, but the server will + block trying to send results from queries it has already processed to the + client. This only occurs when the client sends enough queries to fill its + output buffer and the server's receive buffer before switching to + processing input from the server, but it's hard to predict exactly when + that'll happen so it's best to always use non-blocking mode. + </para> + </note>
Such deadlocks are possible just as well with non-blocking mode, unless
one can avoid sending queries and switching to receiving results anytime
in the application code.
+ <para> + Batched operations will be executed by the server in the order the client + sends them. The server will send the results in the order the statements + executed. The server may begin executing the batch before all commands + in the batch are queued and the end of batch command is sent. If any + statement encounters an error the server aborts the current transaction and + skips processing the rest of the batch. Query processing resumes after the + end of the failed batch. + </para>
What if a batch contains transaction boundaries?
+ <sect3 id="libpq-batch-results"> + <title>Processing results</title> + + <para> + The client <link linkend="libpq-batch-interleave">interleaves result + processing with sending batch queries</link>, or for small batches may + process all results after sending the whole batch. + </para>
That's a very long <link> text, is it not?
+ <para> + To get the result of the first batch entry the client must call <link + linkend="libpq-PQbatchQueueProcess"><function>PQbatchQueueProcess</function></link>. It must then call
What does 'QueueProcess' mean? Shouldn't it be 'ProcessQueue'? You're
not enquing a process or processing, right?
+ <function>PQgetResult</function> and handle the results until + <function>PQgetResult</function> returns null (or would return null if + called).
What is that parenthetical referring to? IIRC we don't provide any
external way to determine PQgetResult would return NULL.
Have you checked how this API works with PQsetSingleRowMode()?
+ <para> + To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a + successful call of <function>PQbatchQueueProcess</function>. This mode selection is effective + only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode + </function>, refer to <xref linkend="libpq-single-row-mode">.
Hah ;).
+ <para> + The client must not assume that work is committed when it + <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the + corresponding result is received to confirm the commit is complete. + Because errors arrive asynchronously the application needs to be able to + restart from the last <emphasis>received</emphasis> committed change and + resend work done after that point if something goes wrong. + </para>
That seems like a batch independent thing, right? If so, maybe make it
a <note>?
+ <listitem> + <para> + Causes a connection to enter batch mode if it is currently idle or + already in batch mode. + +<synopsis> +int PQbatchBegin(PGconn *conn); +</synopsis> + + </para> + <para> + Returns 1 for success. Returns 0 and has no + effect if the connection is not currently idle, i.e. it has a result + ready, is waiting for more input from the server, etc. This function + does not actually send anything to the server, it just changes the + <application>libpq</application> connection state. + + </para> + </listitem> + </varlistentry>
That function name sounds a bit too much like it'd be relevant for a
single batch, not something that can send many batches. enterBatchMode?
+ <varlistentry id="libpq-PQbatchEnd"> + <term> + <function>PQbatchEnd</function> + <indexterm> + <primary>PQbatchEnd</primary> + </indexterm> + </term> + + <listitem> + <para> + Causes a connection to exit batch mode if it is currently in batch mode + with an empty queue and no pending results. +<synopsis> +int PQbatchEnd(PGconn *conn); +</synopsis> + </para> + <para>Returns 1 for success. + Returns 1 and takes no action if not in batch mode. If the connection has + pending batch items in the queue for reading with + <function>PQbatchQueueProcess</function>, the current statement isn't finished + processing or there are results pending for collection with + <function>PQgetResult</function>, returns 0 and does nothing. + + </para> + </listitem> + </varlistentry>
""
+ <varlistentry id="libpq-PQbatchQueueSync"> + <term> + <function>PQbatchQueueSync</function> + <function>PQbatchQueueProcess</function>
As said above, I'm not a fan of these, because it sounds like you're
queueing a sync/process.
/* ---------------- @@ -1108,7 +1113,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp) conn->next_result = conn->result; conn->result = res; /* And mark the result ready to return */ - conn->asyncStatus = PGASYNC_READY; + conn->asyncStatus = PGASYNC_READY_MORE; }
Uhm, isn't that an API/ABI breakage issue?
/* - * Common startup code for PQsendQuery and sibling routines + * PQmakePipelinedCommand + * Get a new command queue entry, allocating it if required. Doesn't add it to + * the tail of the queue yet, use PQappendPipelinedCommand once the command has + * been written for that. If a command fails once it's called this, it should + * use PQrecyclePipelinedCommand to put it on the freelist or release it.
"command fails once it's called this"?
+/* + * PQrecyclePipelinedCommand + * Push a command queue entry onto the freelist. It must be a dangling entry + * with null next pointer and not referenced by any other entry's next pointer. + */ +static void +PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry) +{ + if (entry == NULL) + return; + if (entry->next != NULL) + { + fprintf(stderr, "tried to recycle non-dangling command queue entry"); + abort();
Needs a libpq_gettext()?
+/* + * PQbatchEnd + * End batch mode and return to normal command mode. + * + * Has no effect unless the client has processed all results + * from all outstanding batches and the connection is idle. + * + * Returns true if batch mode ended. + */ +int +PQbatchEnd(PGconn *conn) +{ + if (!conn) + return false; + + if (conn->batch_status == PQBATCH_MODE_OFF) + return true; + + switch (conn->asyncStatus) + { + case PGASYNC_IDLE: + printfPQExpBuffer(&conn->errorMessage, + libpq_gettext_noop("internal error, IDLE in batch mode")); + break; + case PGASYNC_COPY_IN: + case PGASYNC_COPY_OUT: + case PGASYNC_COPY_BOTH: + printfPQExpBuffer(&conn->errorMessage, + libpq_gettext_noop("internal error, COPY in batch mode")); + break;
Why aren't you returning false here,
+ case PGASYNC_READY: + case PGASYNC_READY_MORE: + case PGASYNC_BUSY: + /* can't end batch while busy */ + return false;
but are here?
+ case PGASYNC_QUEUED: + break; + } +
+int +PQbatchQueueSync(PGconn *conn) +{ + PGcommandQueueEntry *entry; + + if (!conn) + return false; + + if (conn->batch_status == PQBATCH_MODE_OFF) + return false; + + switch (conn->asyncStatus) + { + case PGASYNC_IDLE: + printfPQExpBuffer(&conn->errorMessage, + libpq_gettext_noop("internal error, IDLE in batch mode")); + break; + case PGASYNC_COPY_IN: + case PGASYNC_COPY_OUT: + case PGASYNC_COPY_BOTH: + printfPQExpBuffer(&conn->errorMessage, + libpq_gettext_noop("internal error, COPY in batch mode")); + break; + case PGASYNC_READY: + case PGASYNC_READY_MORE: + case PGASYNC_BUSY: + case PGASYNC_QUEUED: + /* can send sync to end this batch of cmds */ + break; + }
Uhm, what is that switch actually achieving? We're not returning an
error code, so ...?
+ /* Should try to flush immediately if there's room */ + PQflush(conn);
"room"?
Also, don't we need to process PQflush's return value?
+/* + * PQbatchQueueProcess + * In batch mode, start processing the next query in the queue. + * + * Returns true if the next query was popped from the queue and can + * be processed by PQconsumeInput, PQgetResult, etc. + * + * Returns false if the current query isn't done yet, the connection + * is not in a batch, or there are no more queries to process. + */ +int +PQbatchQueueProcess(PGconn *conn) +{ + PGcommandQueueEntry *next_query; + + if (!conn) + return false; + + if (conn->batch_status == PQBATCH_MODE_OFF) + return false; + + switch (conn->asyncStatus) + { + case PGASYNC_COPY_IN: + case PGASYNC_COPY_OUT: + case PGASYNC_COPY_BOTH: + printfPQExpBuffer(&conn->errorMessage, + libpq_gettext_noop("internal error, COPY in batch mode")); + break; + case PGASYNC_READY: + case PGASYNC_READY_MORE: + case PGASYNC_BUSY: + /* client still has to process current query or results */ + return false; + break; + case PGASYNC_IDLE: + printfPQExpBuffer(&conn->errorMessage, + libpq_gettext_noop("internal error, IDLE in batch mode")); + break; + case PGASYNC_QUEUED: + /* next query please */ + break; + }
Once more, I'm very unconvinced by the switch. Unless you do anything
with the errors, this seems pointless.
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC) + { + /* + * In an aborted batch we don't get anything from the server for each + * result; we're just discarding input until we get to the next sync + * from the server. The client needs to know its queries got aborted + * so we create a fake PGresult to return immediately from + * PQgetResult. + */ + conn->result = PQmakeEmptyPGresult(conn, + PGRES_BATCH_ABORTED); + if (!conn->result) + { + printfPQExpBuffer(&conn->errorMessage, + libpq_gettext("out of memory")); + pqSaveErrorResult(conn); + } + conn->asyncStatus = PGASYNC_READY;
So we still return true in the OOM case?
Greetings,
Andres Freund
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
Hi,
On Tue, Apr 4, 2017 at 7:05 AM, Andres Freund <andres@anarazel.de> wrote:
On 2017-04-03 14:10:47 +1000, Vaishnavi Prabakaran wrote:
The CF has been extended until April 7 but time is still growing short.
Please respond with a new patch by 2017-04-04 00:00 AoE (UTC-12) orthis
submission will be marked "Returned with Feedback".
Thanks for the information, attached the latest patch resolving one
compilation warning. And, please discard the test patch as it will be
re-implemented later separately.Hm. If the tests aren't ready yet, it seems we'll have to move this to
the next CF.
Thanks for your review and I will address your review comments and send the
newer version of patch shortly.
Just quickly, Is it not ok to consider only the code patch for this CF
without test patch?
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
Re: PATCH: Batch/pipelining support for libpq
On 2017-04-04 09:24:23 +1000, Vaishnavi Prabakaran wrote:
Hi,
On Tue, Apr 4, 2017 at 7:05 AM, Andres Freund <andres@anarazel.de> wrote:On 2017-04-03 14:10:47 +1000, Vaishnavi Prabakaran wrote:
The CF has been extended until April 7 but time is still growing short.
Please respond with a new patch by 2017-04-04 00:00 AoE (UTC-12) orthis
submission will be marked "Returned with Feedback".
Thanks for the information, attached the latest patch resolving one
compilation warning. And, please discard the test patch as it will be
re-implemented later separately.Hm. If the tests aren't ready yet, it seems we'll have to move this to
the next CF.Thanks for your review and I will address your review comments and send the
newer version of patch shortly.
Cool.
Just quickly, Is it not ok to consider only the code patch for this CF
without test patch?
I'd say no, it's not acceptable. This is too much new code for it not
to be tested.
Andres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On Tue, Apr 4, 2017 at 8:26 AM, Andres Freund <andres@anarazel.de> wrote:
On 2017-04-04 09:24:23 +1000, Vaishnavi Prabakaran wrote:
Just quickly, Is it not ok to consider only the code patch for this CF
without test patch?I'd say no, it's not acceptable. This is too much new code for it not
to be tested.
Doesn't it depend actually? I would think that the final patch may not
include all the tests implemented if:
- The thread on which a patch has been developed had a set of tests
done and posted with it.
- Including them does not make sense if we have a way to run those
tests more efficiently. Sometimes a bunch of benchmarks or tests are
run on a patch bu for the final result keeping them around does not
make much sense.
In the case of this patch, it seems to me that we would have a far
better portable set of tests if we had a dedicated set of subcommands
available at psql level, particularly for Windows/MSVC. If that's a
requirement for this patch so let it be. I am not saying that tests
are not necessary. They are of course, but in this case having a bit
more infrastructure would be more be more helpful for users and the
tests themselves.
Note that I won't complain either if this set of C tests are included
at the end.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On 2017-04-04 08:57:33 +0900, Michael Paquier wrote:
On Tue, Apr 4, 2017 at 8:26 AM, Andres Freund <andres@anarazel.de> wrote:
On 2017-04-04 09:24:23 +1000, Vaishnavi Prabakaran wrote:
Just quickly, Is it not ok to consider only the code patch for this CF
without test patch?I'd say no, it's not acceptable. This is too much new code for it not
to be tested.Doesn't it depend actually?
Well, I didn't make a general statement, I made one about this patch.
And this would add a significant bunch of untested code, and it'll likely
take years till it gets decent coverage outside.
In the case of this patch, it seems to me that we would have a far
better portable set of tests if we had a dedicated set of subcommands
available at psql level, particularly for Windows/MSVC.
That's a really large scope creep imo. Adding a bunch of user-facing
psql stuff doesn't compare in complexity to running a test across
platforms. We can just do that from regess.c or such, if that ends up
being a problem..
If that's a requirement for this patch so let it be. I am not saying that tests
are not necessary. They are of course, but in this case having a bit
more infrastructure would be more be more helpful for users and the
tests themselves.
I'm not following.
Greetings,
Andres Freund
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
Andres Freund <andres@anarazel.de> wrote:
+ + <para> + <application>libpq</application> supports queueing up multiplequeries into
+ a pipeline to be executed as a batch on the server. Batching queries
allows
+ applications to avoid a client/server round-trip after each query to
get
+ the results before issuing the next query. + </para>"queueing .. into a pipeline" sounds weird to me - but I'm not a native
speaker. Also batching != pipelining.
Re-phrased the sentence as "libpq supports queries to be queued up in
batches and pipeline them to the server, where it will be executed as a
batch" . Pipelining queries allows applications to avoid a client/server
round-trip after each query to get the results before issuing the next
query.
+ <sect2> + <title>When to use batching</title> + + <para> + Much like asynchronous query mode, there is no performancedisadvantage to
+ using batching and pipelining. It increases client application
complexity
+ and extra caution is required to prevent client/server deadlocks but + offers considerable performance improvements. + </para>s/offers/can sometimes offer/
Corrected.
+ <sect2 id="libpq-batch-using"> + <title>Using batch mode</title> + + <para> + To issue batches the application must switch + <application>libpq</application> into batch mode.s/libpq/a connection/?
Corrected.
Enter batch mode with <link
+ linkend="libpq-PQbatchBegin"><function>PQbatchBegin(conn)</function></link>or test
+ whether batch mode is active with <link + linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>.In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are
permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will
trigger failure in batch processing.
+ (The restriction on <literal>COPY</literal> is an implementation + limit; the PostgreSQL protocol and server can support batched <literal>COPY</literal>).Hm, I'm unconvinced that that's a useful parenthetical in the libpq
docs.
Removed the parenthesis as the line before gives enough warning.
+ <para> + The client uses libpq's asynchronous query functions to dispatchwork,
+ marking the end of each batch with <function>PQbatchQueueSync</
function>.
+ Concurrently, it uses <function>PQgetResult</function> and + <function>PQbatchQueueProcess</function> to get results."Concurrently" imo is a dangerous word, somewhat implying multi-threading.
Corrected by replacing "concurrently" with "And to get results"
+ <note> + <para> + It is best to use batch mode with <application>libpq</application>in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>.
If used in
+ blocking mode it is possible for a client/server deadlock to
occur. The
+ client will block trying to send queries to the server, but the
server will
+ block trying to send results from queries it has already processed
to the
+ client. This only occurs when the client sends enough queries to
fill its
+ output buffer and the server's receive buffer before switching to + processing input from the server, but it's hard to predict exactlywhen
+ that'll happen so it's best to always use non-blocking mode. + </para> + </note>Such deadlocks are possible just as well with non-blocking mode, unless
one can avoid sending queries and switching to receiving results anytime
in the application code.
True, and in the documentation , how to interleave result processing and
query dispatch is given.
+ <para> + Batched operations will be executed by the server in the order theclient
+ sends them. The server will send the results in the order the
statements
+ executed. The server may begin executing the batch before all
commands
+ in the batch are queued and the end of batch command is sent. If
any
+ statement encounters an error the server aborts the current
transaction and
+ skips processing the rest of the batch. Query processing resumes
after the
+ end of the failed batch.
+ </para>What if a batch contains transaction boundaries?
Regardless of transaction boundaries, failed command will make the entire
batch aborted and no commands will be processed in server until Batch sync
is sent. And, if the command inside the transaction fails, then the batch
will be aborted, but transaction will still be open in the server. Client
explicitly needs to close the transaction. This is explained further down
in the documentation section "Error handling" , paragraph starting with "If
the batch used an implicit transaction.." and "Note" in that section gives
warning too.
+ <sect3 id="libpq-batch-results"> + <title>Processing results</title> + + <para> + The client <link linkend="libpq-batch-interleave">interleavesresult
+ processing with sending batch queries</link>, or for small batches
may
+ process all results after sending the whole batch. + </para>That's a very long <link> text, is it not?
Corrected.
+ <para> + To get the result of the first batch entry the client must call<link
+ linkend="libpq-PQbatchQueueProcess"><
function>PQbatchQueueProcess</function></link>. It must then call
What does 'QueueProcess' mean? Shouldn't it be 'ProcessQueue'? You're
not enquing a process or processing, right?
Changed to PQbatchProcessQueue.
+ <function>PQgetResult</function> and handle the results until + <function>PQgetResult</function> returns null (or would returnnull if
+ called).
What is that parenthetical referring to? IIRC we don't provide any
external way to determine PQgetResult would return NULL.
True, it is just application's extra knowledge. Removed the parenthesis.
+ <para> + The client must not assume that work is committed when it + <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only whenthe
+ corresponding result is received to confirm the commit is complete. + Because errors arrive asynchronously the application needs to beable to
+ restart from the last <emphasis>received</emphasis> committed
change and
+ resend work done after that point if something goes wrong. + </para>That seems like a batch independent thing, right? If so, maybe make it
a <note>?
Yes, made it as Note.
+ +<synopsis> +int PQbatchBegin(PGconn *conn);...
That function name sounds a bit too much like it'd be relevant for a
single batch, not something that can send many batches. enterBatchMode?+int PQbatchEnd(PGconn *conn);
...
""
Changed the function names to PQenterBatchMode and PQexitBatchMode.
+ <varlistentry id="libpq-PQbatchQueueSync"> + <term> + <function>PQbatchQueueSync</function> + <function>PQbatchQueueProcess</function>As said above, I'm not a fan of these, because it sounds like you're
queueing a sync/process.
Changed it to PQbatchSyncQueue.
/* ---------------- @@ -1108,7 +1113,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp) conn->next_result = conn->result; conn->result = res; /* And mark the result ready to return */ - conn->asyncStatus = PGASYNC_READY; + conn->asyncStatus = PGASYNC_READY_MORE; }Uhm, isn't that an API/ABI breakage issue?
I think no, asynstatus is in libpq-int.h which is meant to be used only by
frontend libpq, applications should use libpq-fe.h . Also, we don't have
any API to return the async status of connection to application. So, I
think it will not break any existing applications.
/* - * Common startup code for PQsendQuery and sibling routines + * PQmakePipelinedCommand + * Get a new command queue entry, allocating it if required. Doesn'tadd it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the
command has
+ * been written for that. If a command fails once it's called this,
it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release
it.
"command fails once it's called this"?
Corrected.- "Command sending fails"
+/* + * PQrecyclePipelinedCommand + * Push a command queue entry onto the freelist. It must be adangling entry
+ * with null next pointer and not referenced by any other entry's
next pointer.
+ */ +static void +PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry) +{ + if (entry == NULL) + return; + if (entry->next != NULL) + { + fprintf(stderr, "tried to recycle non-dangling commandqueue entry");
+ abort();
Needs a libpq_gettext()?
Corrected.
+/* + * PQbatchEnd + * End batch mode and return to normal command mode. + * + * Has no effect unless the client has processed all results + * from all outstanding batches and the connection is idle. + * + * Returns true if batch mode ended. + */ +int +PQbatchEnd(PGconn *conn) +{ + if (!conn) + return false; + + if (conn->batch_status == PQBATCH_MODE_OFF) + return true; + + switch (conn->asyncStatus) + { + case PGASYNC_IDLE: + printfPQExpBuffer(&conn->errorMessage, + libpq_gettext_noop("internal error,IDLE in batch mode"));
+ break; + case PGASYNC_COPY_IN: + case PGASYNC_COPY_OUT: + case PGASYNC_COPY_BOTH: + printfPQExpBuffer(&conn->errorMessage, + libpq_gettext_noop("internal error,COPY in batch mode"));
+ break;
Why aren't you returning false here,
Using copy command in batch mode is the error case, so instead of giving
opportunity to the application to finish the error scenario, we set the
error message and allows it to quit the batch mode.
+ case PGASYNC_READY: + case PGASYNC_READY_MORE: + case PGASYNC_BUSY: + /* can't end batch while busy */ + return false;but are here?
Application gets a chance to finish the result processing and exit the
batch mode.
+int +PQbatchQueueSync(PGconn *conn) +{ + PGcommandQueueEntry *entry; + + if (!conn) + return false; + + if (conn->batch_status == PQBATCH_MODE_OFF) + return false; + + switch (conn->asyncStatus) + { + case PGASYNC_IDLE: + printfPQExpBuffer(&conn->errorMessage, + libpq_gettext_noop("internal error,IDLE in batch mode"));
+ break; + case PGASYNC_COPY_IN: + case PGASYNC_COPY_OUT: + case PGASYNC_COPY_BOTH: + printfPQExpBuffer(&conn->errorMessage, + libpq_gettext_noop("internal error,COPY in batch mode"));
+ break; + case PGASYNC_READY: + case PGASYNC_READY_MORE: + case PGASYNC_BUSY: + case PGASYNC_QUEUED: + /* can send sync to end this batch of cmds */ + break; + }Uhm, what is that switch actually achieving? We're not returning an
error code, so ...?
These errors are marked here, so that the application later when reading
the results for batch queries, will have more information about what went
wrong. This is just an additional information to user only, but doesn't
stop the error scenario.
+ /* Should try to flush immediately if there's room */ + PQflush(conn);"room"?
Also, don't we need to process PQflush's return value?
Corrected both .
+/* + * PQbatchQueueProcess + * In batch mode, start processing the next query in the queue. + * + * Returns true if the next query was popped from the queue and can + * be processed by PQconsumeInput, PQgetResult, etc. + * + * Returns false if the current query isn't done yet, the connection + * is not in a batch, or there are no more queries to process. + */ +int +PQbatchQueueProcess(PGconn *conn) +{ + PGcommandQueueEntry *next_query; + + if (!conn) + return false; + + if (conn->batch_status == PQBATCH_MODE_OFF) + return false; + + switch (conn->asyncStatus) + { + case PGASYNC_COPY_IN: + case PGASYNC_COPY_OUT: + case PGASYNC_COPY_BOTH: + printfPQExpBuffer(&conn->errorMessage, + libpq_gettext_noop("internal error,COPY in batch mode"));
+ break; + case PGASYNC_READY: + case PGASYNC_READY_MORE: + case PGASYNC_BUSY: + /* client still has to process current query orresults */
+ return false; + break; + case PGASYNC_IDLE: + printfPQExpBuffer(&conn->errorMessage, + libpq_gettext_noop("internal error,IDLE in batch mode"));
+ break; + case PGASYNC_QUEUED: + /* next query please */ + break; + }Once more, I'm very unconvinced by the switch. Unless you do anything
with the errors, this seems pointless.
Same answer as above switch question.
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass
!= PGQUERY_SYNC)
+ { + /* + * In an aborted batch we don't get anything from theserver for each
+ * result; we're just discarding input until we get to the
next sync
+ * from the server. The client needs to know its queries
got aborted
+ * so we create a fake PGresult to return immediately from + * PQgetResult. + */ + conn->result = PQmakeEmptyPGresult(conn, +PGRES_BATCH_ABORTED);
+ if (!conn->result) + { + printfPQExpBuffer(&conn->errorMessage, +libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn); + } + conn->asyncStatus = PGASYNC_READY;So we still return true in the OOM case?
Corrected to return false, as further calls to this function will also
fail. And , application as well can check for this failure reason from
PQgetResult.
Regarding test patch, I have corrected the test suite after David Steele's
comments.
Also, I would like to mention that a companion patch was submitted by David
Steele up-thread.
Attached the latest code and test patch.
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
Attachments:
0001-Pipelining-batch-support-for-libpq-code-v10.patchapplication/octet-stream; name=0001-Pipelining-batch-support-for-libpq-code-v10.patchDownload
---
doc/src/sgml/libpq.sgml | 528 ++++++++++++++++++
doc/src/sgml/lobj.sgml | 4 +
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/interfaces/libpq/exports.txt | 6 +
src/interfaces/libpq/fe-connect.c | 28 +
src/interfaces/libpq/fe-exec.c | 599 +++++++++++++++++++--
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 15 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 41 +-
10 files changed, 1215 insertions(+), 39 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 4bc5bf3..8aec588 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4655,6 +4655,526 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ can sometimes offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ a connection into batch mode. Enter batch mode with <link
+ linkend="libpq-PQenterBatchMode"><function>PQenterBatchMode(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ Using any synchronous command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error conditions.
+ Functions allowed in batch mode are described in <xref linkend="libpq-batch-sending">.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchSyncQueue</function>.
+ And to get results, it uses <function>PQgetResult</function> and
+ <function>PQbatchProcessQueue</function>. It may eventually exit
+ batch mode with <function>PQexitBatchMode</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchSyncQueue"><function>PQbatchSyncQueue(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing</link> with sending batch queries, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchProcessQueue"><function>PQbatchProcessQueue</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null. The result from the next batch entry
+ may then be retrieved using <function>PQbatchProcessQueue</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
+ successful call of <function>PQbatchProcessQueue</function>. This mode selection is effective
+ only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode">.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchSyncQueue</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchSyncQueue</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchProcessQueue(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQexitBatchMode"><function>PQexitBatchMode(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchSyncQueue</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterBatchMode">
+ <term>
+ <function>PQenterBatchMode</function>
+ <indexterm>
+ <primary>PQenterBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQenterBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitBatchMode">
+ <term>
+ <function>PQexitBatchMode</function>
+ <indexterm>
+ <primary>PQexitBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitBatchMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchProcessQueue</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchSyncQueue">
+ <term>
+ <function>PQbatchSyncQueue</function>
+ <indexterm>
+ <primary>PQbatchSyncQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchSyncQueue(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchProcessQueue">
+ <term>
+ <function>PQbatchProcessQueue</function>
+ <indexterm>
+ <primary>PQbatchProcessQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchProcessQueue(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchQueueCount</function>, <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueCount">
+ <term>
+ <function>PQbatchQueueCount</function>
+ <indexterm>
+ <primary>PQbatchQueueCount</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processed.
+ This is the number of times <function>PQbatchProcessQueue</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQbatchQueueCount(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
@@ -4695,6 +5215,14 @@ int PQflush(PGconn *conn);
Each object should be freed with <function>PQclear</function> as usual.
</para>
+ <note>
+ <para>
+ On using batch mode, call <function>PQsetSingleRowMode</function>
+ immediately after a successful call of <function>PQbatchProcessQueue</function>
+ See <xref linkend="libpq-batch-mode"> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-pqsetsinglerowmode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 7757e1e..db8523d 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 9d7bb25..a30a032 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -917,6 +917,9 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->status = WALRCV_ERROR;
walres->err = pchomp(PQerrorMessage(conn->streamConn));
break;
+ default:
+ /* This is just to keep compiler quiet */
+ break;
}
PQclear(pgres);
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 21dd772..3d7ec85 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -171,3 +171,9 @@ PQsslAttributeNames 168
PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
+PQbatchQueueCount 172
+PQenterBatchMode 173
+PQexitBatchMode 174
+PQbatchSyncQueue 175
+PQbatchProcessQueue 176
+PQbatchStatus 177
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index d79a1cf..ea33543 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3459,6 +3459,25 @@ sendTerminateConn(PGconn *conn)
}
/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
+
+/*
* closePGconn
* - properly close a connection to the backend
*
@@ -3470,6 +3489,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3502,6 +3522,14 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9decd53..2a3928b 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -69,6 +71,9 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
/* ----------------
@@ -1108,7 +1113,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1131,6 +1136,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1229,6 +1241,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1287,18 +1303,34 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1308,10 +1340,14 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1359,7 +1395,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, libpq_gettext("tried to recycle non-dangling command queue entry"));
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1377,20 +1486,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1414,6 +1563,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1423,6 +1576,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1535,22 +1705,25 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1560,10 +1733,14 @@ PQsendQueryGuts(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1690,6 +1867,302 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/*
+ * PQbatchQueueCount
+ * Return number of queries currently pending in batch mode
+ */
+int
+PQbatchQueueCount(PGconn *conn)
+{
+ int count = 0;
+ PGcommandQueueEntry *entry;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ return 0;
+
+ entry = conn->cmd_queue_head;
+ while (entry != NULL)
+ {
+ ++count;
+ entry = entry->next;
+ }
+ return count;
+}
+
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQbatchBegin
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return true;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return true;
+}
+
+/*
+ * PQbatchEnd
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns true if batch mode ended.
+ */
+int
+PQexitBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return true;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ return false;
+ case PGASYNC_QUEUED:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ return true;
+}
+
+/*
+ * PQbatchQueueSync
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new non-batch commands until all
+ * results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchSyncQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return false; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ return true;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/*
+ * PQbatchQueueProcess
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns true if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns false if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return false;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ return false;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return true;
+}
+
/*
* PQgetResult
@@ -1749,10 +2222,32 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * batched queries aren't followed by a Sync to put us back in
+ * PGASYNC_IDLE state, and when we do get a sync we could
+ * still have another batch coming after this one.
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1932,6 +2427,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2126,6 +2628,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2141,6 +2646,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2149,15 +2668,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2171,10 +2693,14 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -2569,6 +3095,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 3b0500f..c01f1a2 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 53776e2..e24d7ce 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 635af5b..04aaded 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQenterBatchMode(PGconn *conn);
+extern int PQexitBatchMode(PGconn *conn);
+extern int PQbatchSyncQueue(PGconn *conn);
+extern int PQbatchProcessQueue(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index b8ec341..33f212f 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -215,10 +215,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -227,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -297,6 +303,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -384,6 +406,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY
@@ -396,6 +419,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -693,6 +726,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.7.4.windows.1
0002-Pipelining-batch-support-for-libpq-test-v6.patchapplication/octet-stream; name=0002-Pipelining-batch-support-for-libpq-test-v6.patchDownload
---
src/test/modules/Makefile | 1 +
src/test/modules/test_libpq/.gitignore | 5 +
src/test/modules/test_libpq/Makefile | 25 +
src/test/modules/test_libpq/README | 1 +
src/test/modules/test_libpq/t/001_libpq_async.pl | 25 +
src/test/modules/test_libpq/testlibpqbatch.c | 1452 ++++++++++++++++++++++
src/tools/msvc/Mkvcbuild.pm | 2 +-
7 files changed, 1510 insertions(+), 1 deletion(-)
create mode 100644 src/test/modules/test_libpq/.gitignore
create mode 100644 src/test/modules/test_libpq/Makefile
create mode 100644 src/test/modules/test_libpq/README
create mode 100644 src/test/modules/test_libpq/t/001_libpq_async.pl
create mode 100644 src/test/modules/test_libpq/testlibpqbatch.c
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 3ce9904..310d3ba 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -11,6 +11,7 @@ SUBDIRS = \
snapshot_too_old \
test_ddl_deparse \
test_extensions \
+ test_libpq \
test_parser \
test_pg_dump \
test_rls_hooks \
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000..11e8463
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000..d907063
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,25 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+testlibpqbatch.o: testlibpqbatch.c
+testlibpqbatch: testlibpqbatch.o
+check: testlibpqbatch prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000..d8174dd
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000..25f9ff0
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,25 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_batch simple_batch multi_batch batch_abort timings singlerowmode);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(['testlibpqbatch', 'dbname=postgres', "$numrows", "$testname"],
+ "testlibpqbatch $testname");
+}
+
+$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000..88d6d60
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1452 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution functionality
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+static void test_singlerowmode(PGconn *conn);
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+ fflush(stderr);
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode\n");
+ goto fail;
+ }
+
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode with work in progress should fail, but succeeded\n");
+ goto fail;
+ }
+
+ if (!PQbatchSyncQueue(conn))
+ {
+ fprintf(stderr, "Ending a batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * in batch mode we have to ask for the first result to be processed;
+ * until we do PQgetResult will return null:
+ */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchProcessQueue() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* We can't PQbatchProcessQueue when there might still be pending results */
+ if (PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() should've failed with pending results: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchProcessQueue() call\n");
+ goto fail;
+ }
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ /* should now get an explicit sync result */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after end batch call\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+ fflush(stderr);
+
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode: %u\n", __LINE__);
+ goto fail;
+ }
+
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "Unable to enter batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not activated properly\n");
+ goto fail;
+ }
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "PQexec should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ {
+ fprintf(stderr, "PQsendQuery should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "re-entering batch mode should be a no-op but failed\n");
+ goto fail;
+ }
+
+ if (PQisBusy(conn))
+ {
+ fprintf(stderr, "PQisBusy should return false when idle in batch, returned true\n");
+ goto fail;
+ }
+
+ /* ok, back to normal command mode */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "couldn't exit idle empty batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not terminated properly\n");
+ goto fail;
+ }
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "batch mode exit when not in batch mode should succeed but failed\n");
+ goto fail;
+ }
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+ fflush(stderr);
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchSyncQueue(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchSyncQueue(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* OK, start processing the batch results */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchProcessQueue() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchProcessQueue() call\n");
+ goto fail;
+ }
+
+ if (PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ /* second batch */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+ fflush(stderr);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "1";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching error select failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "2";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchSyncQueue(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "3";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchSyncQueue(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* second query, caused error */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* third query in batch, the second insert */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at third batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the batch sync */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "sync should've cleared the aborted flag but didn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the insert from the second batch */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first entry in second batch: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first item in second batch\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* the second batch sync */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch sync\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ {
+ fprintf(stderr, "expected only insert with value 3, got %s", val);
+ goto fail;
+ }
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ fprintf(stderr, "expected 1 result, got %d", PQntuples(res));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "xact start failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent BEGIN\n");
+
+ send_step = BI_DROP_TABLE;
+
+ if (!PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent DROP\n");
+
+ send_step = BI_CREATE_TABLE;
+
+ if (!PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent CREATE\n");
+
+ send_step = BI_PREPARE;
+
+ if (!PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids))
+ {
+ fprintf(stderr, "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent PREPARE\n");
+
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ {
+ fprintf(stderr, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "Expected next query result but unable to dequeue: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ fprintf(stdout, "next query!\n");
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+
+ if (PQresultStatus(res) != status)
+ {
+ fprintf(stderr, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ {
+ fprintf(stderr, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+ goto fail;
+ }
+
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchSyncQueue(conn))
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ {
+ fprintf(stderr, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "prepare failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "COPY: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[12 + 2];
+ int formatted = snprintf(&buf[0], 12 + 1, "%d\n", nrows);
+
+ if (formatted >= 12 + 1)
+ {
+ fprintf(stderr, "Buffer write truncated somehow\n");
+ goto fail;
+ }
+
+ if (PQputCopyData(conn, buf, formatted) != 1)
+ {
+ fprintf(stderr, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ nrows--;
+ }
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ struct timeval start_time,
+ end_time,
+ elapsed_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_pipelined(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("batch insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_sequential(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("sequential insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_copy(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("COPY elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings|singlerowmode\n");
+ exit(1);
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i,r;
+
+ /* 1 batch, 3 queries in it */
+ r = PQenterBatchMode(conn);
+
+ for (i=0; i < 3; i++) {
+ r = PQsendQueryParams(conn,
+ "SELECT 1",
+ 0,
+ NULL,
+ NULL,
+ NULL,
+ NULL,
+ 0);
+ }
+ PQbatchSyncQueue(conn);
+
+ i=0;
+ while (PQbatchProcessQueue(conn))
+ {
+ int isSingleTuple = 0;
+ /* Set single row mode for only first 3 SELECT queries */
+ if(i < 3)
+ r = PQsetSingleRowMode(conn);
+ if (r!=1)
+ {
+ fprintf(stderr, "PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+ fprintf(stderr, "Result status: %d (%s) for i=%d", est, PQresStatus(est), i);
+ if (est == PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ if(!isSingleTuple)
+ {
+ fprintf(stderr, " Expected to follow PGREG_SINGLE_TUPLE, but received PGRES_TUPLES_OK directly instead\n");
+ goto fail;
+ }
+ isSingleTuple=0;
+ }
+ else if (est == PGRES_SINGLE_TUPLE)
+ {
+ isSingleTuple = 1;
+ fprintf(stderr, ", single tuple: %d\n", PQntuples(res));
+ }
+ else if (est == PGRES_BATCH_END)
+ {
+ fprintf(stderr, ", end of batch reached\n");
+ }
+ else if (est != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, ", error: %s\n", PQresultErrorMessage(res));
+ }
+ PQclear(res);
+ }
+ i++;
+ }
+ PQexitBatchMode(conn);
+ PQclear(res);
+ res = NULL;
+ return;
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+
+}
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1,
+ run_singlerowmode = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ run_singlerowmode = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else if (strcmp(argv[3], "singlerowmode") == 0)
+ run_singlerowmode = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognized test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+ if(run_singlerowmode)
+ test_singlerowmode(conn);
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index ba1bf6d..4a2a4dd 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -46,7 +46,7 @@ my @contrib_excludes = (
'ltree_plpython', 'pgcrypto',
'sepgsql', 'brin',
'test_extensions', 'test_pg_dump',
- 'snapshot_too_old');
+ 'snapshot_too_old', 'test_libpq');
# Set of variables for frontend modules
my $frontend_defines = { 'initdb' => 'FRONTEND' };
--
2.7.4.windows.1
Re: PATCH: Batch/pipelining support for libpq
Hi,
On 2017-04-05 17:00:42 +1000, Vaishnavi Prabakaran wrote:
Regarding test patch, I have corrected the test suite after David Steele's
comments.
Also, I would like to mention that a companion patch was submitted by David
Steele up-thread.Attached the latest code and test patch.
My impression is that this'll need a couple more rounds of review. Given
that this'll establish API we'll pretty much ever going to be able to
change/remove, I think it'd be a bad idea to rush this into v10.
Therefore I propose moving this to the next CF.
- Andres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On 2017-04-05 15:45:26 -0700, Andres Freund wrote:
Hi,
On 2017-04-05 17:00:42 +1000, Vaishnavi Prabakaran wrote:
Regarding test patch, I have corrected the test suite after David Steele's
comments.
Also, I would like to mention that a companion patch was submitted by David
Steele up-thread.Attached the latest code and test patch.
My impression is that this'll need a couple more rounds of review. Given
that this'll establish API we'll pretty much ever going to be able to
change/remove, I think it'd be a bad idea to rush this into v10.
Therefore I propose moving this to the next CF.
Craig, Vaishnavi, everyone else: Are you planning to continue to work on
this for v11? I'm willing to do another round, but only if it's
worthwhile.
FWIW, I still think this needs a pgbench or similar example integration,
so we can actually properly measure the benefits.
- Andres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On Tue, Jun 20, 2017 at 8:49 AM, Andres Freund <andres@anarazel.de> wrote:
On 2017-04-05 15:45:26 -0700, Andres Freund wrote:
Hi,
On 2017-04-05 17:00:42 +1000, Vaishnavi Prabakaran wrote:
Regarding test patch, I have corrected the test suite after David
Steele's
comments.
Also, I would like to mention that a companion patch was submitted byDavid
Steele up-thread.
Attached the latest code and test patch.
My impression is that this'll need a couple more rounds of review. Given
that this'll establish API we'll pretty much ever going to be able to
change/remove, I think it'd be a bad idea to rush this into v10.
Therefore I propose moving this to the next CF.Craig, Vaishnavi, everyone else: Are you planning to continue to work on
this for v11? I'm willing to do another round, but only if it's
worthwhile.
Yes, am willing to continue working on this patch for v11.
FWIW, I still think this needs a pgbench or similar example integration,
so we can actually properly measure the benefits.
I will investigate on this further.
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
Re: PATCH: Batch/pipelining support for libpq
On 20 June 2017 at 06:49, Andres Freund <andres@anarazel.de> wrote:
On 2017-04-05 15:45:26 -0700, Andres Freund wrote:
Hi,
On 2017-04-05 17:00:42 +1000, Vaishnavi Prabakaran wrote:
Regarding test patch, I have corrected the test suite after David Steele's
comments.
Also, I would like to mention that a companion patch was submitted by David
Steele up-thread.Attached the latest code and test patch.
My impression is that this'll need a couple more rounds of review. Given
that this'll establish API we'll pretty much ever going to be able to
change/remove, I think it'd be a bad idea to rush this into v10.
Therefore I propose moving this to the next CF.Craig, Vaishnavi, everyone else: Are you planning to continue to work on
this for v11? I'm willing to do another round, but only if it's
worthwhile.
I'm happy to work on review, and will try to make some time, but have
to focus primarily on logical rep infrastructure. This patch was a
proof of concept and fun hack for me and while I'm glad folks are
interested, it's not something I can dedicate much time to. Especially
with a 6-week-old baby now....
FWIW, I still think this needs a pgbench or similar example integration,
so we can actually properly measure the benefits.
I agree. I originally wanted to patch psql, but it's pretty intrusive.
pgbench is likely a better target. Also pg_restore.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On Tue, Jun 20, 2017 at 10:43 AM, Craig Ringer <craig@2ndquadrant.com> wrote:
Especially with a 6-week-old baby now....
Congratulations!
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
Andres Freund wrote:
FWIW, I still think this needs a pgbench or similar example integration,
so we can actually properly measure the benefits.
Here's an updated version of the patch I made during review,
adding \beginbatch and \endbatch to pgbench.
The performance improvement appears clearly
with a custom script of this kind:
\beginbatch
UPDATE pgbench_branches SET bbalance = bbalance + 1 WHERE bid = 0;
..above repeated 1000 times...
\endbatch
versus the same with a BEGIN; END; pair instead of \beginbatch \endbatch
On localhost on my desktop I tend to see a 30% difference in favor
of the batch mode with that kind of test.
On slower networks there are much bigger differences.
The latest main patch (v10) must also be slightly updated for HEAD,
because of this:
error: patch failed: src/interfaces/libpq/exports.txt:171
v11 attached without any other change.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
Attachments:
0001-Pipelining-batch-support-for-libpq-code-v11.patchtext/plainDownload
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index c3bd4f9..366f278 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4656,6 +4656,526 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ can sometimes offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ a connection into batch mode. Enter batch mode with <link
+ linkend="libpq-PQenterBatchMode"><function>PQenterBatchMode(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ Using any synchronous command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error conditions.
+ Functions allowed in batch mode are described in <xref linkend="libpq-batch-sending">.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchSyncQueue</function>.
+ And to get results, it uses <function>PQgetResult</function> and
+ <function>PQbatchProcessQueue</function>. It may eventually exit
+ batch mode with <function>PQexitBatchMode</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchSyncQueue"><function>PQbatchSyncQueue(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing</link> with sending batch queries, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchProcessQueue"><function>PQbatchProcessQueue</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null. The result from the next batch entry
+ may then be retrieved using <function>PQbatchProcessQueue</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
+ successful call of <function>PQbatchProcessQueue</function>. This mode selection is effective
+ only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode">.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchSyncQueue</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchSyncQueue</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchProcessQueue(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQexitBatchMode"><function>PQexitBatchMode(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchSyncQueue</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterBatchMode">
+ <term>
+ <function>PQenterBatchMode</function>
+ <indexterm>
+ <primary>PQenterBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQenterBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitBatchMode">
+ <term>
+ <function>PQexitBatchMode</function>
+ <indexterm>
+ <primary>PQexitBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitBatchMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchProcessQueue</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchSyncQueue">
+ <term>
+ <function>PQbatchSyncQueue</function>
+ <indexterm>
+ <primary>PQbatchSyncQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchSyncQueue(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchProcessQueue">
+ <term>
+ <function>PQbatchProcessQueue</function>
+ <indexterm>
+ <primary>PQbatchProcessQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchProcessQueue(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchQueueCount</function>, <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueCount">
+ <term>
+ <function>PQbatchQueueCount</function>
+ <indexterm>
+ <primary>PQbatchQueueCount</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processed.
+ This is the number of times <function>PQbatchProcessQueue</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQbatchQueueCount(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
@@ -4696,6 +5216,14 @@ int PQflush(PGconn *conn);
Each object should be freed with <function>PQclear</function> as usual.
</para>
+ <note>
+ <para>
+ On using batch mode, call <function>PQsetSingleRowMode</function>
+ immediately after a successful call of <function>PQbatchProcessQueue</function>
+ See <xref linkend="libpq-batch-mode"> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-pqsetsinglerowmode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 7757e1e..db8523d 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index f6fa0e4..d4ee576 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -936,6 +936,9 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->status = WALRCV_ERROR;
walres->err = pchomp(PQerrorMessage(conn->streamConn));
break;
+ default:
+ /* This is just to keep compiler quiet */
+ break;
}
PQclear(pgres);
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index d6a38d0..49871f5 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -172,3 +172,9 @@ PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
PQencryptPasswordConn 172
+PQbatchQueueCount 173
+PQenterBatchMode 174
+PQexitBatchMode 175
+PQbatchSyncQueue 176
+PQbatchProcessQueue 177
+PQbatchStatus 178
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 02ec8f0..a92f4bc 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3483,6 +3483,25 @@ sendTerminateConn(PGconn *conn)
}
/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
+
+/*
* closePGconn
* - properly close a connection to the backend
*
@@ -3494,6 +3513,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3526,6 +3546,14 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9decd53..2a3928b 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -69,6 +71,9 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
/* ----------------
@@ -1108,7 +1113,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1131,6 +1136,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1229,6 +1241,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1287,18 +1303,34 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1308,10 +1340,14 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1359,7 +1395,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, libpq_gettext("tried to recycle non-dangling command queue entry"));
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1377,20 +1486,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1414,6 +1563,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1423,6 +1576,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1535,22 +1705,25 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1560,10 +1733,14 @@ PQsendQueryGuts(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1690,6 +1867,302 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/*
+ * PQbatchQueueCount
+ * Return number of queries currently pending in batch mode
+ */
+int
+PQbatchQueueCount(PGconn *conn)
+{
+ int count = 0;
+ PGcommandQueueEntry *entry;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ return 0;
+
+ entry = conn->cmd_queue_head;
+ while (entry != NULL)
+ {
+ ++count;
+ entry = entry->next;
+ }
+ return count;
+}
+
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQbatchBegin
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return true;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return true;
+}
+
+/*
+ * PQbatchEnd
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns true if batch mode ended.
+ */
+int
+PQexitBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return true;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ return false;
+ case PGASYNC_QUEUED:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ return true;
+}
+
+/*
+ * PQbatchQueueSync
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new non-batch commands until all
+ * results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchSyncQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return false; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ return true;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/*
+ * PQbatchQueueProcess
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns true if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns false if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return false;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ return false;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return true;
+}
+
/*
* PQgetResult
@@ -1749,10 +2222,32 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * batched queries aren't followed by a Sync to put us back in
+ * PGASYNC_IDLE state, and when we do get a sync we could
+ * still have another batch coming after this one.
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1932,6 +2427,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2126,6 +2628,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2141,6 +2646,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2149,15 +2668,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2171,10 +2693,14 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -2569,6 +3095,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 3b0500f..c01f1a2 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 53776e2..e24d7ce 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index e7496c5..321fc60 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQenterBatchMode(PGconn *conn);
+extern int PQexitBatchMode(PGconn *conn);
+extern int PQbatchSyncQueue(PGconn *conn);
+extern int PQbatchProcessQueue(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 335568b..619f5c0 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -215,10 +215,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -227,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -297,6 +303,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -384,6 +406,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY
@@ -396,6 +419,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -684,6 +717,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
pgbench-batch-mode-v3.patchtext/plainDownload
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 14aa587..016455e 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -269,7 +269,8 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
+ * CSTATE_WAIT_RESULT state unless in batch mode.
+ * On a \sleep meta-command, the timer is set,
* and we enter the CSTATE_SLEEP state to wait for it to expire. Other
* meta-commands are executed immediately.
*
@@ -1882,11 +1883,24 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- fprintf(stderr, "%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQbatchStatus(st->con) == PQBATCH_MODE_OFF)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ fprintf(stderr, "%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In batch mode, we use asynchronous functions. If a server-side
+ * error occurs, it will be processed later among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ fprintf(stderr, "%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2165,7 +2179,13 @@ doCustom(TState *thread, CState *st, StatsData *agg)
return;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in batch mode */
+ if (PQbatchStatus(st->con) == PQBATCH_MODE_OFF)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -2207,7 +2227,51 @@ doCustom(TState *thread, CState *st, StatsData *agg)
}
else
{
- if (pg_strcasecmp(argv[0], "set") == 0)
+ if (pg_strcasecmp(argv[0], "beginbatch") == 0)
+ {
+ /*
+ * In batch mode, we use a workflow based on libpq batch
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "cannot use batch mode with the simple query protocol");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_OFF)
+ {
+ commandFailed(st, "already in batch mode");
+ break;
+ }
+ if (PQenterBatchMode(st->con) == 0)
+ {
+ commandFailed(st, "failed to start a batch");
+ break;
+ }
+ }
+ else if (pg_strcasecmp(argv[0], "endbatch") == 0)
+ {
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ {
+ commandFailed(st, "not in batch mode");
+ break;
+ }
+ if (!PQbatchSyncQueue(st->con))
+ {
+ commandFailed(st, "failed to end the batch");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ if (PQexitBatchMode(st->con) == 0)
+ {
+ /* collect pending results before getting out of batch mode */
+ st->state = CSTATE_WAIT_RESULT;
+ break;
+ }
+ }
+ else if (pg_strcasecmp(argv[0], "set") == 0)
{
PgBenchExpr *expr = command->expr;
PgBenchValue result;
@@ -2279,6 +2343,7 @@ doCustom(TState *thread, CState *st, StatsData *agg)
}
break;
+
/*
* Wait for the current SQL command to complete
*/
@@ -2295,6 +2360,13 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
+ if (PQbatchStatus(st->con) == PQBATCH_MODE_ON &&
+ !PQbatchProcessQueue(st->con))
+ {
+ /* no complete result yet in batch mode*/
+ return;
+ }
+
/*
* Read and discard the query result;
*/
@@ -2307,7 +2379,22 @@ doCustom(TState *thread, CState *st, StatsData *agg)
/* OK */
PQclear(res);
discard_response(st);
- st->state = CSTATE_END_COMMAND;
+ /*
+ * In non-batch mode, only one result per command is expected.
+ * In batch mode, keep waiting for results until getting
+ * PGRES_BATCH_END.
+ */
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ st->state = CSTATE_END_COMMAND;
+ break;
+ case PGRES_BATCH_END:
+ if (PQexitBatchMode(st->con) == 1)
+ {
+ /* all results collected, exit out of command and batch mode */
+ st->state = CSTATE_END_COMMAND;
+ }
+ else
+ fprintf(stderr, "client %d to exit batch mode", st->id);
break;
default:
commandFailed(st, PQerrorMessage(st->con));
@@ -3173,6 +3260,13 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->line, my_command->argv[0],
"missing command", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "beginbatch") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "endbatch") == 0 )
+ {
+ if (my_command->argc > 1)
+ syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ "unexpected argument", NULL, -1);
+ }
else
{
syntax_error(source, lineno, my_command->line, my_command->argv[0],
Re: PATCH: Batch/pipelining support for libpq
On 2017-06-20 17:51:23 +0200, Daniel Verite wrote:
Andres Freund wrote:
FWIW, I still think this needs a pgbench or similar example integration,
so we can actually properly measure the benefits.Here's an updated version of the patch I made during review,
adding \beginbatch and \endbatch to pgbench.
The performance improvement appears clearly
with a custom script of this kind:
\beginbatch
UPDATE pgbench_branches SET bbalance = bbalance + 1 WHERE bid = 0;
..above repeated 1000 times...
\endbatchversus the same with a BEGIN; END; pair instead of \beginbatch \endbatch
On localhost on my desktop I tend to see a 30% difference in favor
of the batch mode with that kind of test.
On slower networks there are much bigger differences.
This is seriously impressive. Just using the normal pgbench mixed
workload, wrapping a whole transaction into a batch *doubles* the
throughput. And that's locally over a unix socket - the gain over
actual network will be larger.
\set nbranches 1 * :scale
\set ntellers 10 * :scale
\set naccounts 100000 * :scale
\set aid random(1, :naccounts)
\set bid random(1, :nbranches)
\set tid random(1, :ntellers)
\set delta random(-5000, 5000)
\beginbatch
BEGIN;
UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
END;
\endbatch
- Andres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On 22 Jun. 2017 07:40, "Andres Freund" <andres@anarazel.de> wrote:
On 2017-06-20 17:51:23 +0200, Daniel Verite wrote:
Andres Freund wrote:
FWIW, I still think this needs a pgbench or similar example integration,
so we can actually properly measure the benefits.Here's an updated version of the patch I made during review,
adding \beginbatch and \endbatch to pgbench.
The performance improvement appears clearly
with a custom script of this kind:
\beginbatch
UPDATE pgbench_branches SET bbalance = bbalance + 1 WHERE bid = 0;
..above repeated 1000 times...
\endbatchversus the same with a BEGIN; END; pair instead of \beginbatch \endbatch
On localhost on my desktop I tend to see a 30% difference in favor
of the batch mode with that kind of test.
On slower networks there are much bigger differences.
This is seriously impressive. Just using the normal pgbench mixed
workload, wrapping a whole transaction into a batch *doubles* the
throughput. And that's locally over a unix socket - the gain over
actual network will be larger.
In my original tests I got over a 300x improvement on WAN :) . I should
check if the same applies with pgbench.
Re: PATCH: Batch/pipelining support for libpq
On 2017-06-21 16:40:48 -0700, Andres Freund wrote:
On 2017-06-20 17:51:23 +0200, Daniel Verite wrote:
Andres Freund wrote:
FWIW, I still think this needs a pgbench or similar example integration,
so we can actually properly measure the benefits.Here's an updated version of the patch I made during review,
adding \beginbatch and \endbatch to pgbench.
The performance improvement appears clearly
with a custom script of this kind:
\beginbatch
UPDATE pgbench_branches SET bbalance = bbalance + 1 WHERE bid = 0;
..above repeated 1000 times...
\endbatchversus the same with a BEGIN; END; pair instead of \beginbatch \endbatch
On localhost on my desktop I tend to see a 30% difference in favor
of the batch mode with that kind of test.
On slower networks there are much bigger differences.This is seriously impressive. Just using the normal pgbench mixed
workload, wrapping a whole transaction into a batch *doubles* the
throughput. And that's locally over a unix socket - the gain over
actual network will be larger.
I've not analyzed this further, but something with the way network is
done isn't yet quite right either in the pgbench patch or in the libpq
patch. You'll currently get IO like:
sendto(3, "B\0\0\0\22\0P1_2\0\0\0\0\0\0\1\0\0D\0\0\0\6P\0E\0\0\0\t\0"..., 36, MSG_NOSIGNAL, NULL, 0) = 36
sendto(3, "B\0\0\0\35\0P1_4\0\0\0\0\1\0\0\0\0073705952\0\1\0\0D\0"..., 47, MSG_NOSIGNAL, NULL, 0) = 47
sendto(3, "B\0\0\0\35\0P1_6\0\0\0\0\1\0\0\0\0077740854\0\1\0\0D\0"..., 47, MSG_NOSIGNAL, NULL, 0) = 47
sendto(3, "B\0\0\0\35\0P1_8\0\0\0\0\1\0\0\0\0071570280\0\1\0\0D\0"..., 47, MSG_NOSIGNAL, NULL, 0) = 47
sendto(3, "B\0\0\0\36\0P1_10\0\0\0\0\1\0\0\0\0072634305\0\1\0\0D"..., 48, MSG_NOSIGNAL, NULL, 0) = 48
sendto(3, "B\0\0\0\36\0P1_12\0\0\0\0\1\0\0\0\0078960656\0\1\0\0D"..., 48, MSG_NOSIGNAL, NULL, 0) = 48
sendto(3, "B\0\0\0\36\0P1_14\0\0\0\0\1\0\0\0\0073030370\0\1\0\0D"..., 48, MSG_NOSIGNAL, NULL, 0) = 48
sendto(3, "B\0\0\0\35\0P1_16\0\0\0\0\1\0\0\0\006376125\0\1\0\0D\0"..., 47, MSG_NOSIGNAL, NULL, 0) = 47
sendto(3, "B\0\0\0\36\0P1_18\0\0\0\0\1\0\0\0\0072982423\0\1\0\0D"..., 48, MSG_NOSIGNAL, NULL, 0) = 48
sendto(3, "B\0\0\0\36\0P1_20\0\0\0\0\1\0\0\0\0073860195\0\1\0\0D"..., 48, MSG_NOSIGNAL, NULL, 0) = 48
sendto(3, "B\0\0\0\36\0P1_22\0\0\0\0\1\0\0\0\0072794433\0\1\0\0D"..., 48, MSG_NOSIGNAL, NULL, 0) = 48
sendto(3, "B\0\0\0\36\0P1_24\0\0\0\0\1\0\0\0\0075475271\0\1\0\0D"..., 48, MSG_NOSIGNAL, NULL, 0) = 48
sendto(3, "B\0\0\0\23\0P1_25\0\0\0\0\0\0\1\0\0D\0\0\0\6P\0E\0\0\0\t"..., 37, MSG_NOSIGNAL, NULL, 0) = 37
sendto(3, "S\0\0\0\4", 5, MSG_NOSIGNAL, NULL, 0) = 5
recvfrom(3, "2\0\0\0\4n\0\0\0\4C\0\0\0\nBEGIN\0002\0\0\0\4T\0\0\0!\0"..., 16384, 0, NULL, NULL) = 775
recvfrom(3, 0x559a02667ff2, 15630, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(3, 0x559a02667fb1, 15695, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(3, 0x559a02667f6c, 15764, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(3, 0x559a02667f2b, 15829, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(3, 0x559a02667eea, 15894, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(3, 0x559a02667ea9, 15959, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(3, 0x559a02667e68, 16024, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(3, 0x559a02667e24, 16092, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(3, 0x559a02667de3, 16157, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(3, 0x559a02667da2, 16222, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(3, 0x559a02667d5d, 16291, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(3, 0x559a02667d1c, 16356, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(3, 0x559a02667d06, 16378, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
I.e. we're doing tiny write send() syscalls (they should be coalesced)
and then completely unnecessarily call recv() over and over again
without polling. To me it looks very much like we're just doing either
exactly once per command...
- Andres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On 22 June 2017 at 08:29, Andres Freund <andres@anarazel.de> wrote:
I.e. we're doing tiny write send() syscalls (they should be coalesced)
That's likely worth doing, but can probably wait for a separate patch.
The kernel will usually do some packet aggregation unless we use
TCP_NODELAY (which we don't and shouldn't), and the syscall overhead
is IMO not worth worrying about just yet.
and then completely unnecessarily call recv() over and over again
without polling. To me it looks very much like we're just doing either
exactly once per command...
Yeah, that looks suspect.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On 2017-06-22 09:03:05 +0800, Craig Ringer wrote:
On 22 June 2017 at 08:29, Andres Freund <andres@anarazel.de> wrote:
I.e. we're doing tiny write send() syscalls (they should be coalesced)
That's likely worth doing, but can probably wait for a separate patch.
I don't think so, we should get this right, it could have API influence.
The kernel will usually do some packet aggregation unless we use
TCP_NODELAY (which we don't and shouldn't), and the syscall overhead
is IMO not worth worrying about just yet.
1)
/*
* Select socket options: no delay of outgoing data for
* TCP sockets, nonblock mode, close-on-exec. Fail if any
* of this fails.
*/
if (!IS_AF_UNIX(addr_cur->ai_family))
{
if (!connectNoDelay(conn))
{
pqDropConnection(conn, true);
conn->addr_cur = addr_cur->ai_next;
continue;
}
}
2) Even if nodelay weren't set, this can still lead to smaller packets
being sent, because you start sending normal sized tcp packets,
rather than jumbo ones, even if configured (pretty common these
days).
3) Syscall overhead is actually quite significant.
- Andres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On 22 June 2017 at 09:07, Andres Freund <andres@anarazel.de> wrote:
On 2017-06-22 09:03:05 +0800, Craig Ringer wrote:
On 22 June 2017 at 08:29, Andres Freund <andres@anarazel.de> wrote:
I.e. we're doing tiny write send() syscalls (they should be coalesced)
That's likely worth doing, but can probably wait for a separate patch.
I don't think so, we should get this right, it could have API influence.
The kernel will usually do some packet aggregation unless we use
TCP_NODELAY (which we don't and shouldn't), and the syscall overhead
is IMO not worth worrying about just yet.1)
/*
* Select socket options: no delay of outgoing data for
* TCP sockets, nonblock mode, close-on-exec. Fail if any
* of this fails.
*/
if (!IS_AF_UNIX(addr_cur->ai_family))
{
if (!connectNoDelay(conn))
{
pqDropConnection(conn, true);
conn->addr_cur = addr_cur->ai_next;
continue;
}
}2) Even if nodelay weren't set, this can still lead to smaller packets
being sent, because you start sending normal sized tcp packets,
rather than jumbo ones, even if configured (pretty common these
days).3) Syscall overhead is actually quite significant.
Fair enough, and *headdesk* re not checking NODELAY. I thought I'd
checked for our use of that before, but I must've remembered wrong.
We could use TCP_CORK but it's not portable and it'd be better to just
collect up a buffer to dispatch.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On 2017-06-21 18:07:21 -0700, Andres Freund wrote:
On 2017-06-22 09:03:05 +0800, Craig Ringer wrote:
On 22 June 2017 at 08:29, Andres Freund <andres@anarazel.de> wrote:
I.e. we're doing tiny write send() syscalls (they should be coalesced)
That's likely worth doing, but can probably wait for a separate patch.
I don't think so, we should get this right, it could have API influence.
The kernel will usually do some packet aggregation unless we use
TCP_NODELAY (which we don't and shouldn't), and the syscall overhead
is IMO not worth worrying about just yet.1)
/*
* Select socket options: no delay of outgoing data for
* TCP sockets, nonblock mode, close-on-exec. Fail if any
* of this fails.
*/
if (!IS_AF_UNIX(addr_cur->ai_family))
{
if (!connectNoDelay(conn))
{
pqDropConnection(conn, true);
conn->addr_cur = addr_cur->ai_next;
continue;
}
}2) Even if nodelay weren't set, this can still lead to smaller packets
being sent, because you start sending normal sized tcp packets,
rather than jumbo ones, even if configured (pretty common these
days).3) Syscall overhead is actually quite significant.
Proof of the pudding:
pgbench of 10 pgbench select statements in a batch:
as submitted by Daniel:
pgbench -h localhost -M prepared -S -n -c 16 -j 16 -T 10000 -P 1 -f ~/tmp/pgbench-select-only-batch.sq
progress: 1.0 s, 24175.5 tps, lat 0.647 ms stddev 0.782
progress: 2.0 s, 27737.6 tps, lat 0.577 ms stddev 0.625
progress: 3.0 s, 28853.3 tps, lat 0.554 ms stddev 0.619
progress: 4.0 s, 26660.8 tps, lat 0.600 ms stddev 0.776
progress: 5.0 s, 30023.8 tps, lat 0.533 ms stddev 0.484
progress: 6.0 s, 29959.3 tps, lat 0.534 ms stddev 0.450
progress: 7.0 s, 29944.9 tps, lat 0.534 ms stddev 0.536
progress: 8.0 s, 30137.7 tps, lat 0.531 ms stddev 0.533
progress: 9.0 s, 30285.2 tps, lat 0.528 ms stddev 0.479
progress: 10.0 s, 30228.7 tps, lat 0.529 ms stddev 0.460
progress: 11.0 s, 29921.4 tps, lat 0.534 ms stddev 0.613
progress: 12.0 s, 29982.4 tps, lat 0.533 ms stddev 0.510
progress: 13.0 s, 29247.4 tps, lat 0.547 ms stddev 0.526
progress: 14.0 s, 28757.3 tps, lat 0.556 ms stddev 0.635
progress: 15.0 s, 29035.3 tps, lat 0.551 ms stddev 0.523
^C
sample vmstat:
r b swpd free buff cache si so bi bo in cs us sy id wa st
19 0 0 488992 787332 23558676 0 0 0 0 9720 455099 65 35 0 0 0
(i.e. ~450k context switches)
hackily patched:
pgbench -h localhost -M prepared -S -n -c 16 -j 16 -T 10000 -P 1 -f ~/tmp/pgbench-select-only-batch.sq
progress: 1.0 s, 40545.2 tps, lat 0.386 ms stddev 0.625
progress: 2.0 s, 48158.0 tps, lat 0.332 ms stddev 0.277
progress: 3.0 s, 50125.7 tps, lat 0.319 ms stddev 0.204
progress: 4.0 s, 50740.6 tps, lat 0.315 ms stddev 0.250
progress: 5.0 s, 50795.6 tps, lat 0.315 ms stddev 0.246
progress: 6.0 s, 51195.6 tps, lat 0.312 ms stddev 0.207
progress: 7.0 s, 50746.7 tps, lat 0.315 ms stddev 0.264
progress: 8.0 s, 50619.1 tps, lat 0.316 ms stddev 0.250
progress: 9.0 s, 50619.4 tps, lat 0.316 ms stddev 0.228
progress: 10.0 s, 46967.8 tps, lat 0.340 ms stddev 0.499
progress: 11.0 s, 50480.1 tps, lat 0.317 ms stddev 0.239
progress: 12.0 s, 50242.5 tps, lat 0.318 ms stddev 0.286
progress: 13.0 s, 49912.7 tps, lat 0.320 ms stddev 0.266
progress: 14.0 s, 49841.7 tps, lat 0.321 ms stddev 0.271
progress: 15.0 s, 49807.1 tps, lat 0.321 ms stddev 0.248
^C
sample vmstat:
r b swpd free buff cache si so bi bo in cs us sy id wa st
23 0 0 482008 787312 23558996 0 0 0 0 8219 105097 87 14 0 0 0
(i.e. ~100k context switches)
That's *localhost*.
It's completely possible that I've screwed something up here, I didn't
test it besides running pgbench, but the send/recv'd data looks like
it's similar amounts of data, just fewer syscalls.
Greetings,
Andres Freund
Attachments:
fewer-syscalls.difftext/x-diff; charset=us-asciiDownload
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index e498ad61e5..aeed1649ce 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -2352,14 +2352,17 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (debug)
fprintf(stderr, "client %d receiving\n", st->id);
- if (!PQconsumeInput(st->con))
- { /* there's something wrong */
- commandFailed(st, "perhaps the backend died while processing");
- st->state = CSTATE_ABORTED;
- break;
- }
if (PQisBusy(st->con))
- return; /* don't have the whole result yet */
+ {
+ if (!PQconsumeInput(st->con))
+ { /* there's something wrong */
+ commandFailed(st, "perhaps the backend died while processing");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ if (PQisBusy(st->con))
+ return; /* don't have the whole result yet */
+ }
if (PQbatchStatus(st->con) == PQBATCH_MODE_ON &&
!PQbatchProcessQueue(st->con))
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 4cb87a4393..210410a92c 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -1725,12 +1725,15 @@ PQsendQueryGuts(PGconn *conn,
else
*last_query = NULL;
- /*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
- */
- if (pqFlush(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (pqFlush(conn) < 0)
+ goto sendFailed;
+ }
/* OK, it's launched! */
if (conn->batch_status != PQBATCH_MODE_OFF)
Re: PATCH: Batch/pipelining support for libpq
Craig Ringer wrote:
The kernel will usually do some packet aggregation unless we use
TCP_NODELAY (which we don't and shouldn't)
Not sure. As a point of comparison, Oracle has it as a tunable
parameter (TCP.NODELAY), and they changed its default from
No to Yes starting from their 10g R2.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
Andres Freund wrote:
- if (pqFlush(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /*
+ * Give the data a push. In nonblock mode, don't complain if
we're unable
+ * to send it all; PQgetResult() will do any additional
flushing needed.
+ */
+ if (pqFlush(conn) < 0)
+ goto sendFailed;
+ }Seems to be responsible for roughly an 1.7x speedup in tps and equivalent
decrease in latency, based on the "progress" info.
I wonder how much of that corresponds to a decrease in the number of
packets versus the number of syscalls. Both matter, I guess.
But OTOH there are certainly batch workloads where it will be preferrable
for the first query to reach the server ASAP, rather than waiting to be
coalesced with the next ones.
libpq is not going to know what's best.
One option may be to leave that decision to the user by providing a
PQBatchAutoFlush(true|false) property, along with a PQBatchFlush()
function. Maybe we could even let the user set the size of the sending
buffer, so those who really want to squeeze performance may tune it
for their network and workload.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
Hi,
On 2017-06-22 13:43:35 +0200, Daniel Verite wrote:
But OTOH there are certainly batch workloads where it will be preferrable
for the first query to reach the server ASAP, rather than waiting to be
coalesced with the next ones.
Is that really something people expect from a batch API? I suspect it's
not really, and nothing would stop one from adding PQflush() or similar
calls if desirable anyway.
FWIW, the way I did that in the hack clearly isn't ok: If you were to
send a gigabyte of queries, it'd buffer them all up in memory... So some
more intelligence is going to be needed.
libpq is not going to know what's best.
One option may be to leave that decision to the user by providing a
PQBatchAutoFlush(true|false) property, along with a PQBatchFlush()
function.
What'd be the difference between PQflush() and PQbatchFlush()?
Greetings,
Andres Freund
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
Andres Freund wrote:
One option may be to leave that decision to the user by providing a
PQBatchAutoFlush(true|false) property, along with a PQBatchFlush()
function.What'd be the difference between PQflush() and PQbatchFlush()?
I guess no difference, I was just not seeing that libpq already provides
this functionality...
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
Andres Freund wrote :
If you were to send a gigabyte of queries, it'd buffer them all up in
memory... So some
more intelligence is going to be needed.
Firstly, sorry for the delayed response as I got busy with other
activities.
To buffer up the queries before flushing them to the socket, I think
"conn->outCount>=65536" is ok to use, as 64k is considered to be safe in
Windows as per comments in pqSendSome() API.
Attached the code patch to replace pqFlush calls with pqBatchFlush in the
asynchronous libpq function calls flow.
Still pqFlush is used in "PQbatchSyncQueue" and
"PQexitBatchMode" functions.
Am failing to see the benefit in allowing user to set
PQBatchAutoFlush(true|false) property? Is it really needed?
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
Attachments:
0001-Pipelining-batch-support-for-libpq-code-v12.patchapplication/octet-stream; name=0001-Pipelining-batch-support-for-libpq-code-v12.patchDownload
From ea300052d23b7ee311b83a9a8fa3cc63e04e0ae6 Mon Sep 17 00:00:00 2001
From: Vaishnavi Prabakaran <vaishnavip@fast.au.fujitsu.com>
Date: Thu, 10 Aug 2017 10:10:51 +1000
Subject: [PATCH] Pipelining-batch-support-for-libpq-code-v12
---
doc/src/sgml/libpq.sgml | 528 +++++++++++++++++
doc/src/sgml/lobj.sgml | 4 +
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/interfaces/libpq/exports.txt | 6 +
src/interfaces/libpq/fe-connect.c | 28 +
src/interfaces/libpq/fe-exec.c | 624 +++++++++++++++++++--
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 15 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 41 +-
10 files changed, 1236 insertions(+), 43 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index ad5e9b9..435df5b 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4723,6 +4723,526 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ can sometimes offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ a connection into batch mode. Enter batch mode with <link
+ linkend="libpq-PQenterBatchMode"><function>PQenterBatchMode(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ Using any synchronous command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error conditions.
+ Functions allowed in batch mode are described in <xref linkend="libpq-batch-sending">.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchSyncQueue</function>.
+ And to get results, it uses <function>PQgetResult</function> and
+ <function>PQbatchProcessQueue</function>. It may eventually exit
+ batch mode with <function>PQexitBatchMode</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchSyncQueue"><function>PQbatchSyncQueue(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing</link> with sending batch queries, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchProcessQueue"><function>PQbatchProcessQueue</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null. The result from the next batch entry
+ may then be retrieved using <function>PQbatchProcessQueue</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
+ successful call of <function>PQbatchProcessQueue</function>. This mode selection is effective
+ only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode">.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchSyncQueue</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchSyncQueue</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchProcessQueue(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQexitBatchMode"><function>PQexitBatchMode(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchSyncQueue</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterBatchMode">
+ <term>
+ <function>PQenterBatchMode</function>
+ <indexterm>
+ <primary>PQenterBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQenterBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitBatchMode">
+ <term>
+ <function>PQexitBatchMode</function>
+ <indexterm>
+ <primary>PQexitBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitBatchMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchProcessQueue</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchSyncQueue">
+ <term>
+ <function>PQbatchSyncQueue</function>
+ <indexterm>
+ <primary>PQbatchSyncQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchSyncQueue(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchProcessQueue">
+ <term>
+ <function>PQbatchProcessQueue</function>
+ <indexterm>
+ <primary>PQbatchProcessQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchProcessQueue(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchQueueCount</function>, <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueCount">
+ <term>
+ <function>PQbatchQueueCount</function>
+ <indexterm>
+ <primary>PQbatchQueueCount</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processed.
+ This is the number of times <function>PQbatchProcessQueue</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQbatchQueueCount(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
@@ -4763,6 +5283,14 @@ int PQflush(PGconn *conn);
Each object should be freed with <function>PQclear</function> as usual.
</para>
+ <note>
+ <para>
+ On using batch mode, call <function>PQsetSingleRowMode</function>
+ immediately after a successful call of <function>PQbatchProcessQueue</function>
+ See <xref linkend="libpq-batch-mode"> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-pqsetsinglerowmode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 7757e1e..db8523d 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index de03362..ba45ec2 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -939,6 +939,9 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->status = WALRCV_ERROR;
walres->err = pchomp(PQerrorMessage(conn->streamConn));
break;
+ default:
+ /* This is just to keep compiler quiet */
+ break;
}
PQclear(pgres);
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index d6a38d0..49871f5 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -172,3 +172,9 @@ PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
PQencryptPasswordConn 172
+PQbatchQueueCount 173
+PQenterBatchMode 174
+PQexitBatchMode 175
+PQbatchSyncQueue 176
+PQbatchProcessQueue 177
+PQbatchStatus 178
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 5a964bf..780ab35 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3566,6 +3566,25 @@ sendTerminateConn(PGconn *conn)
}
/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
+
+/*
* closePGconn
* - properly close a connection to the backend
*
@@ -3577,6 +3596,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3608,6 +3628,14 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index e1e2d18..db9b7c9 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -69,7 +71,10 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
-
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static int pqBatchFlush(PGconn *conn);
/* ----------------
* Space management for PGresult.
@@ -1108,7 +1113,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1131,6 +1136,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1229,6 +1241,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1287,31 +1303,51 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1359,7 +1395,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, libpq_gettext("tried to recycle non-dangling command queue entry"));
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1377,20 +1486,59 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1414,6 +1562,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1423,6 +1575,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1535,35 +1704,42 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1690,6 +1866,309 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/*
+ * PQbatchQueueCount
+ * Return number of queries currently pending in batch mode
+ */
+int
+PQbatchQueueCount(PGconn *conn)
+{
+ int count = 0;
+ PGcommandQueueEntry *entry;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ return 0;
+
+ entry = conn->cmd_queue_head;
+ while (entry != NULL)
+ {
+ ++count;
+ entry = entry->next;
+ }
+ return count;
+}
+
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQbatchBegin
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return true;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return true;
+}
+
+/*
+ * PQbatchEnd
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns true if batch mode ended.
+ */
+int
+PQexitBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return true;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ return false;
+ case PGASYNC_QUEUED:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ goto sendFailed;
+ return true;
+
+sendFailed:
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/*
+ * PQbatchQueueSync
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new non-batch commands until all
+ * results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchSyncQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return false; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ return true;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/*
+ * PQbatchQueueProcess
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns true if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns false if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return false;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ return false;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return true;
+}
+
/*
* PQgetResult
@@ -1749,10 +2228,32 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * batched queries aren't followed by a Sync to put us back in
+ * PGASYNC_IDLE state, and when we do get a sync we could
+ * still have another batch coming after this one.
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1932,6 +2433,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2126,6 +2634,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2141,6 +2652,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2149,15 +2674,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2167,14 +2695,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -2569,6 +3101,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -3762,3 +4301,14 @@ PQunescapeBytea(const unsigned char *strtext, size_t *retbuflen)
*retbuflen = buflen;
return tmpbuf;
}
+/* pqBatchFlush
+ * In batch mode, data will be flushed only when the out buffer reaches the threshold value.
+ * In non-batch mode, data will be flushed all the time.
+ */
+static int
+pqBatchFlush(PGconn *conn)
+{
+ if ((conn->batch_status == PQBATCH_MODE_OFF)||(conn->outCount>=65536))
+ return(pqFlush(conn));
+ return 0; /* Just to keep compiler quiet */
+}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index a58f701..ffd6737 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index a484fe8..6d90ae7 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 1d915e7..7eb465a 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQenterBatchMode(PGconn *conn);
+extern int PQexitBatchMode(PGconn *conn);
+extern int PQbatchSyncQueue(PGconn *conn);
+extern int PQbatchProcessQueue(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 4291360..425e318 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -215,10 +215,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -227,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -297,6 +303,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -385,6 +407,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -396,6 +419,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -683,6 +716,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.7.4.windows.1
Re: PATCH: Batch/pipelining support for libpq
Hi,
On 2017-08-10 15:23:06 +1000, Vaishnavi Prabakaran wrote:
Andres Freund wrote :
If you were to send a gigabyte of queries, it'd buffer them all up in
memory... So some
more intelligence is going to be needed.
Firstly, sorry for the delayed response as I got busy with other
activities.
No worries - development of new features was slowed down anyway, due to
the v10 feature freeze.
To buffer up the queries before flushing them to the socket, I think
"conn->outCount>=65536" is ok to use, as 64k is considered to be safe in
Windows as per comments in pqSendSome() API.Attached the code patch to replace pqFlush calls with pqBatchFlush in the
asynchronous libpq function calls flow.
Still pqFlush is used in "PQbatchSyncQueue" and
"PQexitBatchMode" functions.
Am failing to see the benefit in allowing user to set
PQBatchAutoFlush(true|false) property? Is it really needed?
I'm inclined not to introduce that for now. If somebody comes up with a
convincing usecase and numbers, we can add it later. Libpq API is set in
stone, so I'd rather not introduce unnecessary stuff...
+ <para> + Much like asynchronous query mode, there is no performance disadvantage to + using batching and pipelining. It increases client application complexity + and extra caution is required to prevent client/server deadlocks but + can sometimes offer considerable performance improvements. + </para>
That's not necessarily true, is it? Unless you count always doing
batches of exactly size 1.
+ <para> + Batching is most useful when the server is distant, i.e. network latency + (<quote>ping time</quote>) is high, and when many small operations are being performed in + rapid sequence. There is usually less benefit in using batches when each + query takes many multiples of the client/server round-trip time to execute. + A 100-statement operation run on a server 300ms round-trip-time away would take + 30 seconds in network latency alone without batching; with batching it may spend + as little as 0.3s waiting for results from the server. + </para>
I'd add a remark that this is frequently beneficial even in cases of
minimal latency - as e.g. shown by the numbers I presented upthread.
+ <para> + Use batches when your application does lots of small + <literal>INSERT</literal>, <literal>UPDATE</literal> and + <literal>DELETE</literal> operations that can't easily be transformed into + operations on sets or into a + <link linkend="libpq-copy"><literal>COPY</literal></link> operation. + </para>
Aren't SELECTs also a major beneficiarry of this?
+ <para> + Batching is less useful when information from one operation is required by the + client before it knows enough to send the next operation.
s/less/not/
+ <note> + <para> + The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can + use batches on server versions 8.4 and newer. Batching works on any server + that supports the v3 extended query protocol. + </para> + </note>
Where's the 8.4 coming from?
+ <para> + The client uses libpq's asynchronous query functions to dispatch work, + marking the end of each batch with <function>PQbatchSyncQueue</function>. + And to get results, it uses <function>PQgetResult</function> and + <function>PQbatchProcessQueue</function>. It may eventually exit + batch mode with <function>PQexitBatchMode</function> once all results are + processed. + </para> + + <note> + <para> + It is best to use batch mode with <application>libpq</application> in + <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in + blocking mode it is possible for a client/server deadlock to occur. The + client will block trying to send queries to the server, but the server will + block trying to send results from queries it has already processed to the + client. This only occurs when the client sends enough queries to fill its + output buffer and the server's receive buffer before switching to + processing input from the server, but it's hard to predict exactly when + that'll happen so it's best to always use non-blocking mode. + </para> + </note>
Mention that nonblocking only actually helps if send/recv is done as
required, and can essentially require unbound memory? We probably
should either document or implement some smarts about when to signal
read/write readyness. Otherwise we e.g. might be receiving tons of
result data without having sent the next query - or the other way round.
+ <sect3 id="libpq-batch-sending"> + <title>Issuing queries</title> + + <para> + After entering batch mode the application dispatches requests + using normal asynchronous <application>libpq</application> functions such as + <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>, + <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>, + <function>PQsendDescribePrepared</function>. + The asynchronous requests are followed by a <link + linkend="libpq-PQbatchSyncQueue"><function>PQbatchSyncQueue(conn)</function></link> call to mark + the end of the batch. The client <emphasis>does not</emphasis> need to call + <function>PQgetResult</function> immediately after dispatching each + operation. <link linkend="libpq-batch-results">Result processing</link> + is handled separately. + </para> + + <para> + Batched operations will be executed by the server in the order the client + sends them. The server will send the results in the order the statements + executed. The server may begin executing the batch before all commands + in the batch are queued and the end of batch command is sent. If any + statement encounters an error the server aborts the current transaction and + skips processing the rest of the batch. Query processing resumes after the + end of the failed batch. + </para>
Maybe note that multiple batches can be "in flight"?
I.e. PQbatchSyncQueue() is about error handling, nothing else? Don't
have a great idea, but we might want to rename...
+ <note> + <para> + The client must not assume that work is committed when it + <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the + corresponding result is received to confirm the commit is complete. + Because errors arrive asynchronously the application needs to be able to + restart from the last <emphasis>received</emphasis> committed change and + resend work done after that point if something goes wrong. + </para> + </note>
This seems fairly independent of batching.
+ </sect3> + + <sect3 id="libpq-batch-interleave"> + <title>Interleaving result processing and query dispatch</title> + + <para> + To avoid deadlocks on large batches the client should be structured around + a nonblocking I/O loop using a function like <function>select</function>, + <function>poll</function>, <function>epoll</function>, + <function>WaitForMultipleObjectEx</function>, etc. + </para> + + <para> + The client application should generally maintain a queue of work still to + be dispatched and a queue of work that has been dispatched but not yet had + its results processed.
Hm. Why? If queries are just issued, no such queue is required?
When the socket is writable it should dispatch more + work. When the socket is readable it should read results and process them, + matching them up to the next entry in its expected results queue. Batches + should be scoped to logical units of work, usually (but not always) one + transaction per batch. There's no need to exit batch mode and re-enter it + between batches or to wait for one batch to finish before sending the next. + </para>
This really needs to take memory usage into account.
+ </variablelist> + + </listitem> + </varlistentry> + + <varlistentry id="libpq-PQenterBatchMode"> + <term> + <function>PQenterBatchMode</function> + <indexterm> + <primary>PQenterBatchMode</primary> + </indexterm> + </term> + + <listitem> + <para> + Causes a connection to enter batch mode if it is currently idle or + already in batch mode. + +<synopsis> +int PQenterBatchMode(PGconn *conn); +</synopsis> + + </para> + <para> + Returns 1 for success. Returns 0 and has no + effect if the connection is not currently idle, i.e. it has a result + ready, is waiting for more input from the server, etc. This function + does not actually send anything to the server, it just changes the + <application>libpq</application> connection state. + + </para> + </listitem> + </varlistentry> + + <varlistentry id="libpq-PQexitBatchMode"> + <term> + <function>PQexitBatchMode</function> + <indexterm> + <primary>PQexitBatchMode</primary> + </indexterm> + </term> + + <listitem> + <para> + Causes a connection to exit batch mode if it is currently in batch mode + with an empty queue and no pending results. +<synopsis> +int PQexitBatchMode(PGconn *conn); +</synopsis> + </para> + <para>Returns 1 for success. + Returns 1 and takes no action if not in batch mode. If the connection has
"returns 1"?
+ <varlistentry id="libpq-PQbatchSyncQueue"> + <term> + <function>PQbatchSyncQueue</function> + <indexterm> + <primary>PQbatchSyncQueue</primary> + </indexterm> + </term> + + <listitem> + <para> + Delimits the end of a set of a batched commands by sending a <link + linkend="protocol-flow-ext-query">sync message</link> and flushing + the send buffer. The end of a batch serves as + the delimiter of an implicit transaction and + an error recovery point; see <link linkend="libpq-batch-errors"> + error handling</link>.
I wonder why this isn't framed as PQbatchIssue/Send/...()? Syncing seems
to mostly make sense from a protocol POV.
+ <varlistentry id="libpq-PQbatchQueueCount"> + <term> + <function>PQbatchQueueCount</function> + <indexterm> + <primary>PQbatchQueueCount</primary> + </indexterm> + </term> + + <listitem> + <para> + Returns the number of queries still in the queue for this batch, not + including any query that's currently having results being processed. + This is the number of times <function>PQbatchProcessQueue</function> has to be + called before the query queue is empty again. + +<synopsis> +int PQbatchQueueCount(PGconn *conn); +</synopsis> + + </para> + </listitem> + </varlistentry>
Given that apps are supposed to track this, I'm not sure why we have
this?
+/* + * PQrecyclePipelinedCommand + * Push a command queue entry onto the freelist. It must be a dangling entry + * with null next pointer and not referenced by any other entry's next pointer. + */ +static void +PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry) +{ + if (entry == NULL) + return; + if (entry->next != NULL) + { + fprintf(stderr, libpq_gettext("tried to recycle non-dangling command queue entry")); + abort();
Don't think we use abort() in libpq like that. There's some Assert()s
tho.
static bool PQsendQueryStart(PGconn *conn) @@ -1377,20 +1486,59 @@ PQsendQueryStart(PGconn *conn) libpq_gettext("no connection to the server\n")); return false; } - /* Can't send while already busy, either. */ - if (conn->asyncStatus != PGASYNC_IDLE) + /* Can't send while already busy, either, unless enqueuing for later */ + if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF) { printfPQExpBuffer(&conn->errorMessage, libpq_gettext("another command is already in progress\n")); return false; }- /* initialize async result-accumulation state */ - pqClearAsyncResult(conn); + if (conn->batch_status != PQBATCH_MODE_OFF) + { + /*
Weirdly indented.
+ if (conn->batch_status != PQBATCH_MODE_OFF) + { + pipeCmd = PQmakePipelinedCommand(conn); + + if (pipeCmd == NULL) + return 0; /* error msg already set */ + + last_query = &pipeCmd->query; + queryclass = &pipeCmd->queryclass; + } + else + { + last_query = &conn->last_query; + queryclass = &conn->queryclass; + }
The amount of complexity / branches we're adding to all of these is more
than a bit unsightly.
+/* + * PQbatchQueueCount + * Return number of queries currently pending in batch mode + */ +int +PQbatchQueueCount(PGconn *conn) +{ + int count = 0; + PGcommandQueueEntry *entry; + + if (PQbatchStatus(conn) == PQBATCH_MODE_OFF) + return 0; + + entry = conn->cmd_queue_head; + while (entry != NULL) + { + ++count; + entry = entry->next; + } + return count; +}
Ugh, O(N)? In that case I'd rather just remove this.
+/*
+ * PQbatchBegin
Mismatched w/ actual function name.
+ * Put an idle connection in batch mode. Commands submitted after this + * can be pipelined on the connection, there's no requirement to wait for + * one to finish before the next is dispatched. + * + * Queuing of new query or syncing during COPY is not allowed.
+"a"?
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched + * commands may be sent while in batch mode. Batch mode can be exited by + * calling PQbatchEnd() once all results are processed. + * + * This doesn't actually send anything on the wire, it just puts libpq + * into a state where it can pipeline work. + */ +int +PQenterBatchMode(PGconn *conn) +{ + if (!conn) + return false;
true/false isn't quite in line with int return code.
+/*
+ * PQbatchEnd
wrong name.
+ * End batch mode and return to normal command mode. + * + * Has no effect unless the client has processed all results + * from all outstanding batches and the connection is idle.
That seems wrong - will lead to hard to diagnose errors.
+ * Returns true if batch mode ended. + */ +int +PQexitBatchMode(PGconn *conn) +{ + if (!conn) + return false; + + if (conn->batch_status == PQBATCH_MODE_OFF) + return true; + + switch (conn->asyncStatus) + { + case PGASYNC_IDLE: + printfPQExpBuffer(&conn->errorMessage, + libpq_gettext_noop("internal error, IDLE in batch mode")); + break; + case PGASYNC_COPY_IN: + case PGASYNC_COPY_OUT: + case PGASYNC_COPY_BOTH: + printfPQExpBuffer(&conn->errorMessage, + libpq_gettext_noop("internal error, COPY in batch mode")); + break;
So we'll still check the queue in this case, that's a bit weird?
+/*
+ * PQbatchQueueSync
Out of sync.
+ * End a batch submission by sending a protocol sync. The connection will + * remain in batch mode and unavailable for new non-batch commands until all + * results from the batch are processed by the client.
"unavailable for new non-batch commands" - that's hard to follow, and
seems pretty redundant with PQendBatchMode (or however it's called).
+ * It's legal to start submitting another batch immediately, without waiting + * for the results of the current batch. There's no need to end batch mode + * and start it again. + * + * If a command in a batch fails, every subsequent command up to and including + * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the + * whole batch is processed without error, a PGresult with PGRES_BATCH_END is + * produced.
Hm, should probably mention that that's only true for commands since the
last PQbatchQueueSync?
+/*
+ * PQbatchQueueProcess
Out of sync.
+ * Returns false if the current query isn't done yet, the connection + * is not in a batch, or there are no more queries to process.
Last complaint about this - think this forgiving mode is a mistake.
+ */ +int +PQbatchProcessQueue(PGconn *conn) +{
+ /* This command's results will come in immediately. + * Initialize async result-accumulation state */ + pqClearAsyncResult(conn);
I'm not following?
/*
* PQgetResult
@@ -1749,10 +2228,32 @@ PQgetResult(PGconn *conn)
+ if (conn->batch_status != PQBATCH_MODE_OFF) + { + /* + * batched queries aren't followed by a Sync to put us back in + * PGASYNC_IDLE state, and when we do get a sync we could + * still have another batch coming after this one.
This needs rephrasing.
+ * The connection isn't idle since we can't submit new + * nonbatched commands. It isn't also busy since the current + * command is done and we need to process a new one. + */ + conn->asyncStatus = PGASYNC_QUEUED;
Not sure I like the name.
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF) + { + printfPQExpBuffer(&conn->errorMessage, + libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n")); + return false; + }
Why do we need the PGASYNC_QUEUED test here?
+/* pqBatchFlush + * In batch mode, data will be flushed only when the out buffer reaches the threshold value. + * In non-batch mode, data will be flushed all the time. + */ +static int +pqBatchFlush(PGconn *conn) +{ + if ((conn->batch_status == PQBATCH_MODE_OFF)||(conn->outCount>=65536)) + return(pqFlush(conn)); + return 0; /* Just to keep compiler quiet */ +}
This should be defined in a macro or such, rather than hardcoded.
Falling over now. This seems like enough feedback for a bit of work
anyway.
Regards,
Andres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On Wed, Aug 23, 2017 at 7:40 PM, Andres Freund <andres@anarazel.de> wrote:
Am failing to see the benefit in allowing user to set
PQBatchAutoFlush(true|false) property? Is it really needed?I'm inclined not to introduce that for now. If somebody comes up with a
convincing usecase and numbers, we can add it later. Libpq API is set in
stone, so I'd rather not introduce unnecessary stuff...
Thanks for reviewing the patch and yes ok.
+ <para> + Much like asynchronous query mode, there is no performancedisadvantage to
+ using batching and pipelining. It increases client application
complexity
+ and extra caution is required to prevent client/server deadlocks but + can sometimes offer considerable performance improvements. + </para>That's not necessarily true, is it? Unless you count always doing
batches of exactly size 1.
Client application complexity is increased in batch mode,because
application needs to remember the query queue status. Results processing
can be done at anytime, so the application needs to know till what query,
the results are consumed.
+ <para>
+ Use batches when your application does lots of small + <literal>INSERT</literal>, <literal>UPDATE</literal> and + <literal>DELETE</literal> operations that can't easily betransformed into
+ operations on sets or into a + <link linkend="libpq-copy"><literal>COPY</literal></link>operation.
+ </para>
Aren't SELECTs also a major beneficiarry of this?
Hmm, though SELECTs also benefit from batch mode, doing multiple selects in
batch mode will fill up the memory rapidly and might not be as beneficial
as other operations listed.
+ <para> + Batching is less useful when information from one operation isrequired by the
+ client before it knows enough to send the next operation.
s/less/not/
Corrected.
+ <note> + <para> + The batch API was introduced in PostgreSQL 10.0, but clients usingPostgresSQL 10.0 version of libpq can
+ use batches on server versions 8.4 and newer. Batching works on
any server
+ that supports the v3 extended query protocol. + </para> + </note>Where's the 8.4 coming from?
I guess it is 7.4 where "PQsendQueryParams" is introduced, and not 8.4.
Corrected.
+ <note>
+ <para> + It is best to use batch mode with <application>libpq</application>in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>.
If used in
+ blocking mode it is possible for a client/server deadlock to
occur. The
+ client will block trying to send queries to the server, but the
server will
+ block trying to send results from queries it has already processed
to the
+ client. This only occurs when the client sends enough queries to
fill its
+ output buffer and the server's receive buffer before switching to + processing input from the server, but it's hard to predict exactlywhen
+ that'll happen so it's best to always use non-blocking mode. + </para> + </note>Mention that nonblocking only actually helps if send/recv is done as
required, and can essentially require unbound memory? We probably
should either document or implement some smarts about when to signal
read/write readyness. Otherwise we e.g. might be receiving tons of
result data without having sent the next query - or the other way round.
Added a statement for caution in documentation and again this is one of the
reason why SELECT query is not so beneficial in batch mode.
Maybe note that multiple batches can be "in flight"?
I.e. PQbatchSyncQueue() is about error handling, nothing else? Don't
have a great idea, but we might want to rename...
This function not only does error handling, but also sends the "Sync"
message to backend. In batch mode, "Sync" message is not sent with every
query but will
be sent only via this function to mark the end of implicit transaction.
Renamed it to PQbatchCommitQueue. Kindly let me know if you think of any
other better name.
+ <note> + <para> + The client must not assume that work is committed when it + <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only whenthe
+ corresponding result is received to confirm the commit is
complete.
+ Because errors arrive asynchronously the application needs to be
able to
+ restart from the last <emphasis>received</emphasis> committed
change and
+ resend work done after that point if something goes wrong. + </para> + </note>This seems fairly independent of batching.
Yes and the reason why is it explicitly specified for batch mode is that if
more than one explicit transactions are used in Single batch, then failure
of one transaction will lead to skipping the consequent transactions until
the end of current batch is reached. This behavior is specific to batch
mode, so adding a precautionary note here is needed I think.
+ </sect3> + + <sect3 id="libpq-batch-interleave"> + <title>Interleaving result processing and query dispatch</title> + + <para> + To avoid deadlocks on large batches the client should bestructured around
+ a nonblocking I/O loop using a function like
<function>select</function>,
+ <function>poll</function>, <function>epoll</function>, + <function>WaitForMultipleObjectEx</function>, etc. + </para> + + <para> + The client application should generally maintain a queue of workstill to
+ be dispatched and a queue of work that has been dispatched but not
yet had
+ its results processed.
Hm. Why? If queries are just issued, no such queue is required?
This is essentially telling that client application should remember the
order of queries sent , so that result processing will be based on what
results are expected. For e.g, if the order of queries are 1)SELECT,
2)INSERT, then in result processing, "PGRES_TUPLES_OK" and later
"PGRES_COMMAND_OK" checks are required. Above mentioned queue of work means
the intelligent to remember the queue of queries.
When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results andprocess them,
+ matching them up to the next entry in its expected results queue.
Batches
+ should be scoped to logical units of work, usually (but not
always) one
+ transaction per batch. There's no need to exit batch mode and
re-enter it
+ between batches or to wait for one batch to finish before sending
the next.
+ </para>
This really needs to take memory usage into account.
Modified the para to include that frequency of result processing should be
based on available memory.
+<synopsis> +int PQenterBatchMode(PGconn *conn); +</synopsis> +<synopsis> +int PQexitBatchMode(PGconn *conn); +</synopsis> + </para> + <para>Returns 1 for success. + Returns 1 and takes no action if not in batch mode. If theconnection has
"returns 1"?
Modified the code to return 1/0.
+ <varlistentry id="libpq-PQbatchSyncQueue"> + <term> + <function>PQbatchSyncQueue</function> + <indexterm> + <primary>PQbatchSyncQueue</primary> + </indexterm> + </term>I wonder why this isn't framed as PQbatchIssue/Send/...()? Syncing seems
to mostly make sense from a protocol POV.
Renamed to PQbatchCommitQueue.
+<synopsis> +int PQbatchQueueCount(PGconn *conn); +</synopsis> +Given that apps are supposed to track this, I'm not sure why we have
this?
Removed this function considering your another comment about O(N) as well.
+static void +PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry) +{ + if (entry == NULL) + return; + if (entry->next != NULL) + { + fprintf(stderr, libpq_gettext("tried to recyclenon-dangling command queue entry"));
+ abort();
Don't think we use abort() in libpq like that. There's some Assert()s
tho.
For out-of-memory cases, we do abort(), and here abort will happen only
during some memory corruption. So, I think it is ok to abort here. Please
let me know if you think otherwise.
static bool
PQsendQueryStart(PGconn *conn).....
Weirdly indented.
Corrected.
+/*
+ * PQbatchBeginMismatched w/ actual function name.
Corrected.
+ * Put an idle connection in batch mode. Commands submitted after this + * can be pipelined on the connection, there's no requirement to waitfor
+ * one to finish before the next is dispatched. + * + * Queuing of new query or syncing during COPY is not allowed.+"a"?
Hmm, Can you explain the question please. I don't understand.
+/*
+ * PQbatchEndwrong name.
Corrected.
+ * End batch mode and return to normal command mode. + * + * Has no effect unless the client has processed all results + * from all outstanding batches and the connection is idle.That seems wrong - will lead to hard to diagnose errors.
I have now added a error message when the batch end is failed to ease the
error diagnose. And, this behavior is documented. So the application should
not assume that the batch mode ended without checking the return value.
+ * Returns true if batch mode ended. + */ +int +PQexitBatchMode(PGconn *conn) +{ + if (!conn) + return false; + + if (conn->batch_status == PQBATCH_MODE_OFF) + return true; + + switch (conn->asyncStatus) + { + case PGASYNC_IDLE: + printfPQExpBuffer(&conn->errorMessage, + libpq_gettext_noop("internal error,IDLE in batch mode"));
+ break; + case PGASYNC_COPY_IN: + case PGASYNC_COPY_OUT: + case PGASYNC_COPY_BOTH: + printfPQExpBuffer(&conn->errorMessage, + libpq_gettext_noop("internal error,COPY in batch mode"));
+ break;
So we'll still check the queue in this case, that's a bit weird?
Removed above cases as adding error here is not of much help as well.
+/*
+ * PQbatchQueueSyncOut of sync.
Corrected.
+ * End a batch submission by sending a protocol sync. The connection
will
+ * remain in batch mode and unavailable for new non-batch commands
until all
+ * results from the batch are processed by the client.
"unavailable for new non-batch commands" - that's hard to follow, and
seems pretty redundant with PQendBatchMode (or however it's called).
Modified documentation to be more precise and difference between
PQexitBatchMode and this one is that to exit batch, application should have
consumed all the results pending in buffer and post exit batch, an
application can call synchronous command execution functions. Whereas with
batch sync/commit, application cannot issue synchronous commands but can
issue asynchronous commands without reading all the results pending in
buffer. This function basically marks the end of implicit transaction.
+ * It's legal to start submitting another batch immediately, without
waiting
+ * for the results of the current batch. There's no need to end batch
mode
+ * and start it again. + * + * If a command in a batch fails, every subsequent command up to andincluding
+ * the PQbatchQueueSync command result gets set to
PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with
PGRES_BATCH_END is
+ * produced.
Hm, should probably mention that that's only true for commands since the
last PQbatchQueueSync?
"Batch" means the set of commands between PQbatchCommitQueue(newly renamed
name) calls except first batch which count from beginning to
PQbatchCommitQueue. And in documentation it is mentioned that "The result
for <function>PQbatchSyncQueue</function> is reported as
<literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
and resumption of normal result processing."
+/*
+ * PQbatchQueueProcessOut of sync.
Corrected .
+ * Returns false if the current query isn't done yet, the connection + * is not in a batch, or there are no more queries to process.Last complaint about this - think this forgiving mode is a mistake.
Now added an error message using printfPQExpBuffer(). I think aborting here
is not really needed. Please let me know if you prefer to abort() rather
than this solution.
+ */ +int +PQbatchProcessQueue(PGconn *conn) +{+ /* This command's results will come in immediately. + * Initialize async result-accumulation state */ + pqClearAsyncResult(conn);I'm not following?
First line of comment is removed. It is copy/paste error.
/*
* PQgetResult
@@ -1749,10 +2228,32 @@ PQgetResult(PGconn *conn)+ if (conn->batch_status != PQBATCH_MODE_OFF) + { + /* + * batched queries aren't followed by aSync to put us back in
+ * PGASYNC_IDLE state, and when we do get
a sync we could
+ * still have another batch coming after
this one.
This needs rephrasing.
Rephrased as "In batch mode, query execution state cannot be IDLE as there
can be other queries or results waiting in the queue ..." . Hope this is
simple and enough.
+ * The connection isn't idle since we
can't submit new
+ * nonbatched commands. It isn't also busy
since the current
+ * command is done and we need to process
a new one.
+ */ + conn->asyncStatus = PGASYNC_QUEUED;Not sure I like the name.
Changed the name to PGASYNC_BATCH to make it more align to batch mode as
this status will not be set anywhere in non-batch mode.
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status !=
PQBATCH_MODE_OFF)
+ { + printfPQExpBuffer(&conn->errorMessage, +libpq_gettext("Synchronous command execution functions are not allowed in
batch mode\n"));+ return false;
+ }Why do we need the PGASYNC_QUEUED test here?
Yes, we don't need this check here. It was removed in PQfn and similarly it
is not needed here and in pqParseInput2 too.
+static int +pqBatchFlush(PGconn *conn) +{ + if ((conn->batch_status == PQBATCH_MODE_OFF)||(conn->outCount>=65536))
+ return(pqFlush(conn)); + return 0; /* Just to keep compiler quiet */ +}This should be defined in a macro or such, rather than hardcoded.
Added macro "OUTBUFFER_THRESHOLD"
Falling over now. This seems like enough feedback for a bit of work
anyway.
Thanks once again for reviewing the patch. Attached the patch with your
review comments incorporation.
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
Attachments:
0001-Pipelining-batch-support-for-libpq-code-v13.patchapplication/octet-stream; name=0001-Pipelining-batch-support-for-libpq-code-v13.patchDownload
From f1c14fbf4368feb514126c9a82f2c33e59c011d8 Mon Sep 17 00:00:00 2001
From: Vaishnavi Prabakaran <vaishnavip@fast.au.fujitsu.com>
Date: Wed, 13 Sep 2017 14:08:25 +1000
Subject: [PATCH] Pipelining-batch-support-for-libpq-code-v13
---
doc/src/sgml/libpq.sgml | 502 +++++++++++++++++
doc/src/sgml/lobj.sgml | 4 +
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/interfaces/libpq/exports.txt | 5 +
src/interfaces/libpq/fe-connect.c | 28 +
src/interfaces/libpq/fe-exec.c | 595 +++++++++++++++++++--
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 15 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 47 +-
10 files changed, 1186 insertions(+), 43 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 096a8be..5e8a2e6 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4726,6 +4726,500 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ can sometimes offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is not useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 7.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ a connection into batch mode. Enter batch mode with <link
+ linkend="libpq-PQenterBatchMode"><function>PQenterBatchMode(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ Using any synchronous command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error conditions.
+ Functions allowed in batch mode are described in <xref linkend="libpq-batch-sending">.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchCommitQueue</function>.
+ And to get results, it uses <function>PQgetResult</function> and
+ <function>PQbatchProcessQueue</function>. It may eventually exit
+ batch mode with <function>PQexitBatchMode</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ Batch mode consumes more memory when send/recv is not done as required even in non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchCommitQueue"><function>PQbatchCommitQueue(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing</link> with sending batch queries, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchProcessQueue"><function>PQbatchProcessQueue</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null. The result from the next batch entry
+ may then be retrieved using <function>PQbatchProcessQueue</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
+ successful call of <function>PQbatchProcessQueue</function>. This mode selection is effective
+ only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode">.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchCommitQueue</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchCommitQueue</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchProcessQueue(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue.
+ Based on available memory, results from socket should be read frequently and
+ there's no need to wait till the batch end to read the results. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQexitBatchMode"><function>PQexitBatchMode(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchCommitQueue</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterBatchMode">
+ <term>
+ <function>PQenterBatchMode</function>
+ <indexterm>
+ <primary>PQenterBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQenterBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitBatchMode">
+ <term>
+ <function>PQexitBatchMode</function>
+ <indexterm>
+ <primary>PQexitBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitBatchMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchProcessQueue</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchCommitQueue">
+ <term>
+ <function>PQbatchCommitQueue</function>
+ <indexterm>
+ <primary>PQbatchCommitQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchCommitQueue(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchProcessQueue">
+ <term>
+ <function>PQbatchProcessQueue</function>
+ <indexterm>
+ <primary>PQbatchProcessQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchProcessQueue(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
@@ -4766,6 +5260,14 @@ int PQflush(PGconn *conn);
Each object should be freed with <function>PQclear</function> as usual.
</para>
+ <note>
+ <para>
+ On using batch mode, call <function>PQsetSingleRowMode</function>
+ immediately after a successful call of <function>PQbatchProcessQueue</function>
+ See <xref linkend="libpq-batch-mode"> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-pqsetsinglerowmode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 7757e1e..db8523d 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 3957bd3..00c6340 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -944,6 +944,9 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->status = WALRCV_ERROR;
walres->err = pchomp(PQerrorMessage(conn->streamConn));
break;
+ default:
+ /* This is just to keep compiler quiet */
+ break;
}
PQclear(pgres);
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index d6a38d0..f907baa 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -172,3 +172,8 @@ PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
PQencryptPasswordConn 172
+PQenterBatchMode 173
+PQexitBatchMode 174
+PQbatchCommitQueue 175
+PQbatchProcessQueue 176
+PQbatchStatus 177
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index c580d91..8e043f7 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3566,6 +3566,25 @@ sendTerminateConn(PGconn *conn)
}
/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
+
+/*
* closePGconn
* - properly close a connection to the backend
*
@@ -3577,6 +3596,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3608,6 +3628,14 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index c24bce6..d6a5f13 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -40,7 +40,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -71,7 +73,10 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
-
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static int pqBatchFlush(PGconn *conn);
/* ----------------
* Space management for PGresult.
@@ -1159,7 +1164,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1182,6 +1187,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1280,6 +1292,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1338,31 +1354,51 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1410,7 +1446,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, libpq_gettext("tried to recycle non-dangling command queue entry"));
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1428,20 +1537,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_BATCH:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1465,6 +1614,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1474,6 +1627,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1586,35 +1756,42 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1741,6 +1918,280 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQenterBatchMode
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return 0;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_BATCH;
+
+ return 1;
+}
+
+/*
+ * PQexitBatchMode
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns 1 if batch mode ended.
+ */
+int
+PQexitBatchMode(PGconn *conn)
+{
+ if (!conn)
+ goto exitFailed;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ goto exitFailed;
+ default:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ goto exitFailed;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ goto sendFailed;
+ return 1;
+
+sendFailed:
+ pqHandleSendFailure(conn);
+
+exitFailed:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, Failed to exit batch mode"));
+ return 0;
+}
+
+/*
+ * PQbatchCommitQueue
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new synchronous command execution
+ * functions until all results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchCommitQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_BATCH:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ return 1;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return 0;
+}
+
+/*
+ * PQbatchProcessQueue
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns 1 if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns 0 if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return 0;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_BATCH:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return 0;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ return 0;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return 1;
+}
+
/*
* PQgetResult
@@ -1800,10 +2251,31 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_BATCH:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, query execution state cannot be IDLE as there
+ * can be other queries or results waiting in the queue
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_BATCH;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1983,6 +2455,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2177,6 +2656,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2192,6 +2674,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2200,15 +2696,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2218,14 +2717,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -2620,6 +3123,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -3813,3 +4323,14 @@ PQunescapeBytea(const unsigned char *strtext, size_t *retbuflen)
*retbuflen = buflen;
return tmpbuf;
}
+/* pqBatchFlush
+ * In batch mode, data will be flushed only when the out buffer reaches the threshold value.
+ * In non-batch mode, data will be flushed all the time.
+ */
+static int
+pqBatchFlush(PGconn *conn)
+{
+ if ((conn->batch_status == PQBATCH_MODE_OFF)||(conn->outCount >= OUTBUFFER_THRESHOLD))
+ return(pqFlush(conn));
+ return 0; /* Just to keep compiler quiet */
+}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 83f74f3..4104cfe 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 7da5fb2..bd99287 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 1d915e7..7eb465a 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQenterBatchMode(PGconn *conn);
+extern int PQexitBatchMode(PGconn *conn);
+extern int PQbatchCommitQueue(PGconn *conn);
+extern int PQbatchProcessQueue(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 4291360..1d47077 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -215,10 +215,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_BATCH /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -227,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -297,6 +303,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -385,6 +407,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -396,6 +419,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -675,6 +708,12 @@ extern ssize_t pgtls_write(PGconn *conn, const void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold is set to 64k as it is safe
+ * in Windows as per comments in pqSendSome() API.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
@@ -683,6 +722,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.7.4.windows.1
Re: PATCH: Batch/pipelining support for libpq
On 13 September 2017 at 13:06, Vaishnavi Prabakaran <
vaishnaviprabakaran@gmail.com> wrote:
On Wed, Aug 23, 2017 at 7:40 PM, Andres Freund <andres@anarazel.de> wrote:
Am failing to see the benefit in allowing user to set
PQBatchAutoFlush(true|false) property? Is it really needed?I'm inclined not to introduce that for now. If somebody comes up with a
convincing usecase and numbers, we can add it later. Libpq API is set in
stone, so I'd rather not introduce unnecessary stuff...Thanks for reviewing the patch and yes ok.
+ <para> + Much like asynchronous query mode, there is no performancedisadvantage to
+ using batching and pipelining. It increases client application
complexity
+ and extra caution is required to prevent client/server deadlocks
but
+ can sometimes offer considerable performance improvements. + </para>That's not necessarily true, is it? Unless you count always doing
batches of exactly size 1.Client application complexity is increased in batch mode,because
application needs to remember the query queue status. Results processing
can be done at anytime, so the application needs to know till what query,
the results are consumed.
Yep. Also, the client/server deadlocks at issue here are a buffer
management issue, and deadlock is probably not exactly the right word. Your
app has to process replies from the server while it's sending queries,
otherwise it can get into a state where it has no room left in its send
buffer, but the server isn't consuming its receive buffer because the
server's send buffer is full. To allow the system to make progress, the
client must read from the client receive buffer.
This isn't an issue when using libpq normally.
PgJDBC has similar issues with its batch mode, but in PgJDBC it's much
worse because there's no non-blocking send available. In libpq you can at
least set your sending socket to non-blocking.
+ <para>
+ Use batches when your application does lots of small + <literal>INSERT</literal>, <literal>UPDATE</literal> and + <literal>DELETE</literal> operations that can't easily betransformed into
+ operations on sets or into a + <link linkend="libpq-copy"><literal>COPY</literal></link>operation.
+ </para>
Aren't SELECTs also a major beneficiarry of this?
Yes, many individual SELECTs that cannot be assembled into a single more
efficient query would definitely also benefit.
Hmm, though SELECTs also benefit from batch mode, doing multiple selects
in batch mode will fill up the memory rapidly and might not be as
beneficial as other operations listed.
Depends on the SELECT. With wide results you'll get less benefit, but even
then you can gain if you're on a high latency network. With "n+1" patterns
and similar, you'll see huge gains.
Maybe note that multiple batches can be "in flight"?
I.e. PQbatchSyncQueue() is about error handling, nothing else? Don't
have a great idea, but we might want to rename...This function not only does error handling, but also sends the "Sync"
message to backend. In batch mode, "Sync" message is not sent with every
query but will
be sent only via this function to mark the end of implicit transaction.
Renamed it to PQbatchCommitQueue. Kindly let me know if you think of any
other better name.
I really do not like calling it "commit" as that conflates with a database
commit.
A batch can embed multiple BEGINs and COMMITs. It's entirely possible for
an earlier part of the batch to succeed and commit, then a later part to
fail, if that's the case. So that name is IMO wrong.
+ <varlistentry id="libpq-PQbatchSyncQueue"> + <term> + <function>PQbatchSyncQueue</function> + <indexterm> + <primary>PQbatchSyncQueue</primary> + </indexterm> + </term>I wonder why this isn't framed as PQbatchIssue/Send/...()? Syncing seems
to mostly make sense from a protocol POV.Renamed to PQbatchCommitQueue.
Per above, strong -1 on that. But SendQueue seems OK, or FlushQueue?
+ * Put an idle connection in batch mode. Commands submitted after
this
+ * can be pipelined on the connection, there's no requirement to
wait for
+ * one to finish before the next is dispatched. + * + * Queuing of new query or syncing during COPY is not allowed.+"a"?
Hmm, Can you explain the question please. I don't understand.
s/of new query/of a new query/
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Re: PATCH: Batch/pipelining support for libpq
On Wed, Sep 13, 2017 at 3:33 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
I really do not like calling it "commit" as that conflates with a database
commit.A batch can embed multiple BEGINs and COMMITs. It's entirely possible for
an earlier part of the batch to succeed and commit, then a later part to
fail, if that's the case. So that name is IMO wrong.
Ok, SendQueue seems ok to me as well. Will change it in next version.
+"a"?
Hmm, Can you explain the question please. I don't understand.
s/of new query/of a new query/
Thanks for explaining. Will change this too in next version.
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
Re: PATCH: Batch/pipelining support for libpq
On 13 September 2017 at 13:44, Vaishnavi Prabakaran <
vaishnaviprabakaran@gmail.com> wrote:
Thanks for explaining. Will change this too in next version.
Thankyou, a lot, for picking up this patch.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Re: PATCH: Batch/pipelining support for libpq
On 13 Sep 2017, at 07:44, Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com> wrote:
On Wed, Sep 13, 2017 at 3:33 PM, Craig Ringer <craig@2ndquadrant.com <mailto:craig@2ndquadrant.com>> wrote:
I really do not like calling it "commit" as that conflates with a database commit.
A batch can embed multiple BEGINs and COMMITs. It's entirely possible for an earlier part of the batch to succeed and commit, then a later part to fail, if that's the case. So that name is IMO wrong.
Ok, SendQueue seems ok to me as well. Will change it in next version.
+"a"?
Hmm, Can you explain the question please. I don't understand.
s/of new query/of a new query/
Thanks for explaining. Will change this too in next version.
Based on the discussions in this thread, and that a new version hasn’t been
submitted, I’m marking this Returned with Feedback. Please re-submit the new
version in an upcoming commitfest when ready.
cheers ./daniel
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: PATCH: Batch/pipelining support for libpq
On Mon, Oct 2, 2017 at 8:31 PM, Daniel Gustafsson <daniel@yesql.se> wrote:
On 13 Sep 2017, at 07:44, Vaishnavi Prabakaran <
vaishnaviprabakaran@gmail.com> wrote:
On Wed, Sep 13, 2017 at 3:33 PM, Craig Ringer <craig@2ndquadrant.com
<mailto:craig@2ndquadrant.com>> wrote:
I really do not like calling it "commit" as that conflates with a
database commit.
A batch can embed multiple BEGINs and COMMITs. It's entirely possible
for an earlier part of the batch to succeed and commit, then a later part
to fail, if that's the case. So that name is IMO wrong.Ok, SendQueue seems ok to me as well. Will change it in next version.
+"a"?
Hmm, Can you explain the question please. I don't understand.
s/of new query/of a new query/
Thanks for explaining. Will change this too in next version.
Based on the discussions in this thread, and that a new version hasn’t been
submitted, I’m marking this Returned with Feedback. Please re-submit the
new
version in an upcoming commitfest when ready.
Thanks for the suggestion and, OK I will create a new patch in upcoming
commitfest with attached patch addressing above review comments.
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
Attachments:
0001-Pipelining-batch-support-for-libpq-code-v14.patchapplication/octet-stream; name=0001-Pipelining-batch-support-for-libpq-code-v14.patchDownload
From 0b2fbd49c73f9783ffa09cb75c425d75852c4d56 Mon Sep 17 00:00:00 2001
From: Prabakaran <Vaishnavi.Prabakaran@au.fujitsu.com>
Date: Thu, 5 Oct 2017 11:55:50 +1100
Subject: [PATCH] Pipelining-batch-support-for-libpq-code-v14
---
doc/src/sgml/libpq.sgml | 502 +++++++++++++++++
doc/src/sgml/lobj.sgml | 4 +
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/interfaces/libpq/exports.txt | 5 +
src/interfaces/libpq/fe-connect.c | 28 +
src/interfaces/libpq/fe-exec.c | 595 +++++++++++++++++++--
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 15 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 47 +-
10 files changed, 1186 insertions(+), 43 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 0aedd83..e353658 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4728,6 +4728,500 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ can sometimes offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is not useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 7.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ a connection into batch mode. Enter batch mode with <link
+ linkend="libpq-PQenterBatchMode"><function>PQenterBatchMode(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ Using any synchronous command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error conditions.
+ Functions allowed in batch mode are described in <xref linkend="libpq-batch-sending">.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchSendQueue</function>.
+ And to get results, it uses <function>PQgetResult</function> and
+ <function>PQbatchProcessQueue</function>. It may eventually exit
+ batch mode with <function>PQexitBatchMode</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ Batch mode consumes more memory when send/recv is not done as required even in non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchSendQueue"><function>PQbatchSendQueue(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing</link> with sending batch queries, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchProcessQueue"><function>PQbatchProcessQueue</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null. The result from the next batch entry
+ may then be retrieved using <function>PQbatchProcessQueue</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
+ successful call of <function>PQbatchProcessQueue</function>. This mode selection is effective
+ only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode">.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchSendQueue</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchSendQueue</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchProcessQueue(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue.
+ Based on available memory, results from socket should be read frequently and
+ there's no need to wait till the batch end to read the results. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQexitBatchMode"><function>PQexitBatchMode(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchSendQueue</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterBatchMode">
+ <term>
+ <function>PQenterBatchMode</function>
+ <indexterm>
+ <primary>PQenterBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQenterBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitBatchMode">
+ <term>
+ <function>PQexitBatchMode</function>
+ <indexterm>
+ <primary>PQexitBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitBatchMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchProcessQueue</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchSendQueue">
+ <term>
+ <function>PQbatchSendQueue</function>
+ <indexterm>
+ <primary>PQbatchSendQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchSendQueue(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchProcessQueue">
+ <term>
+ <function>PQbatchProcessQueue</function>
+ <indexterm>
+ <primary>PQbatchProcessQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchProcessQueue(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
@@ -4768,6 +5262,14 @@ int PQflush(PGconn *conn);
Each object should be freed with <function>PQclear</function> as usual.
</para>
+ <note>
+ <para>
+ On using batch mode, call <function>PQsetSingleRowMode</function>
+ immediately after a successful call of <function>PQbatchProcessQueue</function>
+ See <xref linkend="libpq-batch-mode"> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-pqsetsinglerowmode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 7757e1e..db8523d 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 3957bd3..00c6340 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -944,6 +944,9 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->status = WALRCV_ERROR;
walres->err = pchomp(PQerrorMessage(conn->streamConn));
break;
+ default:
+ /* This is just to keep compiler quiet */
+ break;
}
PQclear(pgres);
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index d6a38d0..625e74e 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -172,3 +172,8 @@ PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
PQencryptPasswordConn 172
+PQenterBatchMode 173
+PQexitBatchMode 174
+PQbatchSendQueue 175
+PQbatchProcessQueue 176
+PQbatchStatus 177
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 5f79803..ee8c3b4 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3566,6 +3566,25 @@ sendTerminateConn(PGconn *conn)
}
/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
+
+/*
* closePGconn
* - properly close a connection to the backend
*
@@ -3577,6 +3596,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3608,6 +3628,14 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index c24bce6..99c0b49 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -40,7 +40,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -71,7 +73,10 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
-
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static int pqBatchFlush(PGconn *conn);
/* ----------------
* Space management for PGresult.
@@ -1159,7 +1164,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1182,6 +1187,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1280,6 +1292,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1338,31 +1354,51 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1410,7 +1446,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, libpq_gettext("tried to recycle non-dangling command queue entry"));
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1428,20 +1537,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1465,6 +1614,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1474,6 +1627,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1586,35 +1756,42 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1741,6 +1918,280 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQenterBatchMode
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return 0;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return 1;
+}
+
+/*
+ * PQexitBatchMode
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns 1 if batch mode ended.
+ */
+int
+PQexitBatchMode(PGconn *conn)
+{
+ if (!conn)
+ goto exitFailed;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ goto exitFailed;
+ default:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ goto exitFailed;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ goto sendFailed;
+ return 1;
+
+sendFailed:
+ pqHandleSendFailure(conn);
+
+exitFailed:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, Failed to exit batch mode"));
+ return 0;
+}
+
+/*
+ * PQbatchSendQueue
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new synchronous command execution
+ * functions until all results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchSendQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ return 1;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return 0;
+}
+
+/*
+ * PQbatchProcessQueue
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns 1 if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns 0 if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return 0;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return 0;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ return 0;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return 1;
+}
+
/*
* PQgetResult
@@ -1800,10 +2251,31 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, query execution state cannot be IDLE as there
+ * can be other queries or results waiting in the queue
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1983,6 +2455,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2177,6 +2656,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2192,6 +2674,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2200,15 +2696,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2218,14 +2717,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -2620,6 +3123,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -3813,3 +4323,14 @@ PQunescapeBytea(const unsigned char *strtext, size_t *retbuflen)
*retbuflen = buflen;
return tmpbuf;
}
+/* pqBatchFlush
+ * In batch mode, data will be flushed only when the out buffer reaches the threshold value.
+ * In non-batch mode, data will be flushed all the time.
+ */
+static int
+pqBatchFlush(PGconn *conn)
+{
+ if ((conn->batch_status == PQBATCH_MODE_OFF)||(conn->outCount >= OUTBUFFER_THRESHOLD))
+ return(pqFlush(conn));
+ return 0; /* Just to keep compiler quiet */
+}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 1320d18..7ce6885 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -411,6 +411,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 21fb8f2..7d05af3 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -219,10 +219,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -879,6 +887,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 1d915e7..98993fa 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQenterBatchMode(PGconn *conn);
+extern int PQexitBatchMode(PGconn *conn);
+extern int PQbatchSendQueue(PGconn *conn);
+extern int PQbatchProcessQueue(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 4291360..1d47077 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -215,10 +215,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -227,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -297,6 +303,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -385,6 +407,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -396,6 +419,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -675,6 +708,12 @@ extern ssize_t pgtls_write(PGconn *conn, const void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold is set to 64k as it is safe
+ * in Windows as per comments in pqSendSome() API.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
@@ -683,6 +722,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.7.4.windows.1
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
On Thu, Oct 5, 2017 at 9:58 AM, Vaishnavi Prabakaran
<vaishnaviprabakaran@gmail.com> wrote:
Thanks for the suggestion and, OK I will create a new patch in upcoming
commitfest with attached patch addressing above review comments.
The patch still applies and there has been no updates for the last
month, as well as no reviews. I am bumping it to next CF.
--
Michael
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
On Tue, Nov 28, 2017 at 12:57 PM, Michael Paquier <michael.paquier@gmail.com
wrote:
On Thu, Oct 5, 2017 at 9:58 AM, Vaishnavi Prabakaran
<vaishnaviprabakaran@gmail.com> wrote:Thanks for the suggestion and, OK I will create a new patch in upcoming
commitfest with attached patch addressing above review comments.The patch still applies and there has been no updates for the last
month, as well as no reviews. I am bumping it to next CF.
Thank you, I see the patch generates a compilation error due to usage of
"FALSE" with latest postgres code, Hence attaching the patch with
correction.
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
Attachments:
0001-Pipelining-batch-support-for-libpq-code-v15.patchapplication/octet-stream; name=0001-Pipelining-batch-support-for-libpq-code-v15.patchDownload
From 40208e16e478d0a95bb81b7a28dfe8332ddd8b94 Mon Sep 17 00:00:00 2001
From: Vaishnavi Prabakaran <vaishnavip@fast.au.fujitsu.com>
Date: Fri, 5 Jan 2018 16:52:02 +1100
Subject: [PATCH] Pipelining-batch-support-for-libpq-code-v15
---
doc/src/sgml/libpq.sgml | 502 +++++++++++++++++
doc/src/sgml/lobj.sgml | 4 +
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/interfaces/libpq/exports.txt | 5 +
src/interfaces/libpq/fe-connect.c | 28 +
src/interfaces/libpq/fe-exec.c | 595 +++++++++++++++++++--
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 15 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 47 +-
10 files changed, 1186 insertions(+), 43 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 4e46451..b540e75 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4752,6 +4752,500 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ can sometimes offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is not useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 7.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ a connection into batch mode. Enter batch mode with <link
+ linkend="libpq-PQenterBatchMode"><function>PQenterBatchMode(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ Using any synchronous command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error conditions.
+ Functions allowed in batch mode are described in <xref linkend="libpq-batch-sending">.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchSendQueue</function>.
+ And to get results, it uses <function>PQgetResult</function> and
+ <function>PQbatchProcessQueue</function>. It may eventually exit
+ batch mode with <function>PQexitBatchMode</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ Batch mode consumes more memory when send/recv is not done as required even in non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchSendQueue"><function>PQbatchSendQueue(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing</link> with sending batch queries, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchProcessQueue"><function>PQbatchProcessQueue</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null. The result from the next batch entry
+ may then be retrieved using <function>PQbatchProcessQueue</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
+ successful call of <function>PQbatchProcessQueue</function>. This mode selection is effective
+ only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode">.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchSendQueue</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchSendQueue</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchProcessQueue(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue.
+ Based on available memory, results from socket should be read frequently and
+ there's no need to wait till the batch end to read the results. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQexitBatchMode"><function>PQexitBatchMode(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchSendQueue</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterBatchMode">
+ <term>
+ <function>PQenterBatchMode</function>
+ <indexterm>
+ <primary>PQenterBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQenterBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitBatchMode">
+ <term>
+ <function>PQexitBatchMode</function>
+ <indexterm>
+ <primary>PQexitBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitBatchMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchProcessQueue</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchSendQueue">
+ <term>
+ <function>PQbatchSendQueue</function>
+ <indexterm>
+ <primary>PQbatchSendQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchSendQueue(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchProcessQueue">
+ <term>
+ <function>PQbatchProcessQueue</function>
+ <indexterm>
+ <primary>PQbatchProcessQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchProcessQueue(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
@@ -4792,6 +5286,14 @@ int PQflush(PGconn *conn);
Each object should be freed with <function>PQclear</function> as usual.
</para>
+ <note>
+ <para>
+ On using batch mode, call <function>PQsetSingleRowMode</function>
+ immediately after a successful call of <function>PQbatchProcessQueue</function>
+ See <xref linkend="libpq-batch-mode"> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-pqsetsinglerowmode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 086cb8d..7de9e30 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 3957bd3..00c6340 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -944,6 +944,9 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->status = WALRCV_ERROR;
walres->err = pchomp(PQerrorMessage(conn->streamConn));
break;
+ default:
+ /* This is just to keep compiler quiet */
+ break;
}
PQclear(pgres);
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index d6a38d0..625e74e 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -172,3 +172,8 @@ PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
PQencryptPasswordConn 172
+PQenterBatchMode 173
+PQexitBatchMode 174
+PQbatchSendQueue 175
+PQbatchProcessQueue 176
+PQbatchStatus 177
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 68fb9a1..037bda2 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3574,6 +3574,25 @@ sendTerminateConn(PGconn *conn)
}
/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
+
+/*
* closePGconn
* - properly close a connection to the backend
*
@@ -3585,6 +3604,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3616,6 +3636,14 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 6653087..961ff41 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -40,7 +40,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -71,7 +73,10 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
-
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static int pqBatchFlush(PGconn *conn);
/* ----------------
* Space management for PGresult.
@@ -1159,7 +1164,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1182,6 +1187,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1280,6 +1292,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1338,31 +1354,51 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1410,7 +1446,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, libpq_gettext("tried to recycle non-dangling command queue entry"));
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1428,20 +1537,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1465,6 +1614,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1474,6 +1627,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1586,35 +1756,42 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1741,6 +1918,280 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQenterBatchMode
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return 0;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return 1;
+}
+
+/*
+ * PQexitBatchMode
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns 1 if batch mode ended.
+ */
+int
+PQexitBatchMode(PGconn *conn)
+{
+ if (!conn)
+ goto exitFailed;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ goto exitFailed;
+ default:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ goto exitFailed;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ goto sendFailed;
+ return 1;
+
+sendFailed:
+ pqHandleSendFailure(conn);
+
+exitFailed:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, Failed to exit batch mode"));
+ return 0;
+}
+
+/*
+ * PQbatchSendQueue
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new synchronous command execution
+ * functions until all results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchSendQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ return 1;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return 0;
+}
+
+/*
+ * PQbatchProcessQueue
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns 1 if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns 0 if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return 0;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return 0;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ return 0;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return 1;
+}
+
/*
* PQgetResult
@@ -1800,10 +2251,31 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, query execution state cannot be IDLE as there
+ * can be other queries or results waiting in the queue
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1983,6 +2455,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2177,6 +2656,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2192,6 +2674,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2200,15 +2696,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2218,14 +2717,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -2620,6 +3123,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -3813,3 +4323,14 @@ PQunescapeBytea(const unsigned char *strtext, size_t *retbuflen)
*retbuflen = buflen;
return tmpbuf;
}
+/* pqBatchFlush
+ * In batch mode, data will be flushed only when the out buffer reaches the threshold value.
+ * In non-batch mode, data will be flushed all the time.
+ */
+static int
+pqBatchFlush(PGconn *conn)
+{
+ if ((conn->batch_status == PQBATCH_MODE_OFF)||(conn->outCount >= OUTBUFFER_THRESHOLD))
+ return(pqFlush(conn));
+ return 0; /* Just to keep compiler quiet */
+}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 5335a91..eac7714 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -411,6 +411,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 0c5099c..be64000 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -219,10 +219,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -879,6 +887,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 1d915e7..98993fa 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQenterBatchMode(PGconn *conn);
+extern int PQexitBatchMode(PGconn *conn);
+extern int PQbatchSendQueue(PGconn *conn);
+extern int PQbatchProcessQueue(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index f6c1023..edf4a7a 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -215,10 +215,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -227,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -297,6 +303,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -386,6 +408,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -397,6 +420,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -679,6 +712,12 @@ extern char *pgtls_get_finished(PGconn *conn, size_t *len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold is set to 64k as it is safe
+ * in Windows as per comments in pqSendSome() API.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
@@ -687,6 +726,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.7.4.windows.1
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
On Fri, Jan 5, 2018 at 4:55 PM, Vaishnavi Prabakaran <
vaishnaviprabakaran@gmail.com> wrote:
On Tue, Nov 28, 2017 at 12:57 PM, Michael Paquier <
michael.paquier@gmail.com> wrote:On Thu, Oct 5, 2017 at 9:58 AM, Vaishnavi Prabakaran
<vaishnaviprabakaran@gmail.com> wrote:Thanks for the suggestion and, OK I will create a new patch in upcoming
commitfest with attached patch addressing above review comments.The patch still applies and there has been no updates for the last
month, as well as no reviews. I am bumping it to next CF.Thank you, I see the patch generates a compilation error due to usage of
"FALSE" with latest postgres code, Hence attaching the patch with
correction.
Corrected compilation error in documentation portion of patch with latest
postgres code. Attached the corrected patch.
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
Attachments:
0001-Pipelining-batch-support-for-libpq-code-v16.patchapplication/octet-stream; name=0001-Pipelining-batch-support-for-libpq-code-v16.patchDownload
From ba93ae02eca024997f2ce6a9c2c2987aea4a77b8 Mon Sep 17 00:00:00 2001
From: Prabakaran <Vaishnavi.Prabakaran@au.fujitsu.com>
Date: Fri, 12 Jan 2018 10:09:09 +1100
Subject: [PATCH] Pipelining-batch-support-for-libpq-code-v16
---
doc/src/sgml/libpq.sgml | 502 +++++++++++++++++
doc/src/sgml/lobj.sgml | 4 +
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/interfaces/libpq/exports.txt | 5 +
src/interfaces/libpq/fe-connect.c | 28 +
src/interfaces/libpq/fe-exec.c | 595 +++++++++++++++++++--
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 15 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 47 +-
10 files changed, 1186 insertions(+), 43 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 4e46451..6aae637 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4752,6 +4752,500 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ can sometimes offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is not useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 7.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ a connection into batch mode. Enter batch mode with <link
+ linkend="libpq-PQenterBatchMode"><function>PQenterBatchMode(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ Using any synchronous command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error conditions.
+ Functions allowed in batch mode are described in <xref linkend="libpq-batch-sending"/>.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchSendQueue</function>.
+ And to get results, it uses <function>PQgetResult</function> and
+ <function>PQbatchProcessQueue</function>. It may eventually exit
+ batch mode with <function>PQexitBatchMode</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ Batch mode consumes more memory when send/recv is not done as required even in non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchSendQueue"><function>PQbatchSendQueue(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing</link> with sending batch queries, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchProcessQueue"><function>PQbatchProcessQueue</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null. The result from the next batch entry
+ may then be retrieved using <function>PQbatchProcessQueue</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
+ successful call of <function>PQbatchProcessQueue</function>. This mode selection is effective
+ only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode"/>.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchSendQueue</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchSendQueue</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchProcessQueue(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue.
+ Based on available memory, results from socket should be read frequently and
+ there's no need to wait till the batch end to read the results. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQexitBatchMode"><function>PQexitBatchMode(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchSendQueue</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterBatchMode">
+ <term>
+ <function>PQenterBatchMode</function>
+ <indexterm>
+ <primary>PQenterBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQenterBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitBatchMode">
+ <term>
+ <function>PQexitBatchMode</function>
+ <indexterm>
+ <primary>PQexitBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitBatchMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchProcessQueue</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchSendQueue">
+ <term>
+ <function>PQbatchSendQueue</function>
+ <indexterm>
+ <primary>PQbatchSendQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchSendQueue(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchProcessQueue">
+ <term>
+ <function>PQbatchProcessQueue</function>
+ <indexterm>
+ <primary>PQbatchProcessQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchProcessQueue(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
@@ -4792,6 +5286,14 @@ int PQflush(PGconn *conn);
Each object should be freed with <function>PQclear</function> as usual.
</para>
+ <note>
+ <para>
+ On using batch mode, call <function>PQsetSingleRowMode</function>
+ immediately after a successful call of <function>PQbatchProcessQueue</function>
+ See <xref linkend="libpq-batch-mode"/> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-pqsetsinglerowmode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 6b5aaeb..cf22f3f 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index f9aec05..32151ff 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -944,6 +944,9 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->status = WALRCV_ERROR;
walres->err = pchomp(PQerrorMessage(conn->streamConn));
break;
+ default:
+ /* This is just to keep compiler quiet */
+ break;
}
PQclear(pgres);
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index d6a38d0..625e74e 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -172,3 +172,8 @@ PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
PQencryptPasswordConn 172
+PQenterBatchMode 173
+PQexitBatchMode 174
+PQbatchSendQueue 175
+PQbatchProcessQueue 176
+PQbatchStatus 177
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 8d54333..c2df669 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3574,6 +3574,25 @@ sendTerminateConn(PGconn *conn)
}
/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
+
+/*
* closePGconn
* - properly close a connection to the backend
*
@@ -3585,6 +3604,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3616,6 +3636,14 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 4c0114c..2049035 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -40,7 +40,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -71,7 +73,10 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
-
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static int pqBatchFlush(PGconn *conn);
/* ----------------
* Space management for PGresult.
@@ -1159,7 +1164,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1182,6 +1187,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1280,6 +1292,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1338,31 +1354,51 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1410,7 +1446,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, libpq_gettext("tried to recycle non-dangling command queue entry"));
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1428,20 +1537,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1465,6 +1614,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1474,6 +1627,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1586,35 +1756,42 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1741,6 +1918,280 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQenterBatchMode
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return 0;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return 1;
+}
+
+/*
+ * PQexitBatchMode
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns 1 if batch mode ended.
+ */
+int
+PQexitBatchMode(PGconn *conn)
+{
+ if (!conn)
+ goto exitFailed;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ goto exitFailed;
+ default:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ goto exitFailed;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ goto sendFailed;
+ return 1;
+
+sendFailed:
+ pqHandleSendFailure(conn);
+
+exitFailed:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, Failed to exit batch mode"));
+ return 0;
+}
+
+/*
+ * PQbatchSendQueue
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new synchronous command execution
+ * functions until all results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchSendQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ return 1;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return 0;
+}
+
+/*
+ * PQbatchProcessQueue
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns 1 if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns 0 if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return 0;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return 0;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ return 0;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return 1;
+}
+
/*
* PQgetResult
@@ -1800,10 +2251,31 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, query execution state cannot be IDLE as there
+ * can be other queries or results waiting in the queue
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1983,6 +2455,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2177,6 +2656,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2192,6 +2674,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2200,15 +2696,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2218,14 +2717,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -2620,6 +3123,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -3813,3 +4323,14 @@ PQunescapeBytea(const unsigned char *strtext, size_t *retbuflen)
*retbuflen = buflen;
return tmpbuf;
}
+/* pqBatchFlush
+ * In batch mode, data will be flushed only when the out buffer reaches the threshold value.
+ * In non-batch mode, data will be flushed all the time.
+ */
+static int
+pqBatchFlush(PGconn *conn)
+{
+ if ((conn->batch_status == PQBATCH_MODE_OFF)||(conn->outCount >= OUTBUFFER_THRESHOLD))
+ return(pqFlush(conn));
+ return 0; /* Just to keep compiler quiet */
+}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 7dcef80..26c9a9e 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -411,6 +411,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index d3ca5d2..f699607 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -219,10 +219,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -879,6 +887,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index ed9c806..7136980 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQenterBatchMode(PGconn *conn);
+extern int PQexitBatchMode(PGconn *conn);
+extern int PQbatchSendQueue(PGconn *conn);
+extern int PQbatchProcessQueue(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 4e35409..fc5376f 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -215,10 +215,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -227,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -297,6 +303,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -386,6 +408,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -397,6 +420,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -680,6 +713,12 @@ extern char *pgtls_get_peer_certificate_hash(PGconn *conn, size_t *len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold is set to 64k as it is safe
+ * in Windows as per comments in pqSendSome() API.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
@@ -688,6 +727,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.7.4.windows.1
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
Hello.
At Fri, 12 Jan 2018 10:12:35 +1100, Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com> wrote in <CAOoUkxRRTBm8ztzVXL45EVk=BC8AjfaFM5=ZMtrOMMfm+dnbwA@mail.gmail.com>
Corrected compilation error in documentation portion of patch with latest
postgres code. Attached the corrected patch.
My understanding of this patch is that it intends to do the
following things without changing the protocol.
A. Refrain from sending sync message and socket flush during batching.
B. Set required information to PGconn before receiving a result.
Multi statement query does the a similar thing but a bit
different. Queries are not needed to be built and mreged as a
string in advance with this feature.
Since I'm not sure what is the current stage of this patch and I
haven't fully recognize the disucssion so far, I might make
stupid or duplicate comments but would like to make some
comments.
- Previous comments
A disucssion on psql batch mode was held in another branch of
this thread. How do we treat that?
- Interface complteness
I think PQenter/exitBatchMode() is undoubtedly needed.
PQbatchSendQueue() seems to be required to make sure to send
queries before calling PQgetResult.
PQbatchProcessQueue() doesn't seem essential. If there's no case
where it is called without following PQgetResult, it can be
included in PQgetResult. Client code may be simpler if it is not
required.
The latest patch has extern definition of PQbatchQueueCount, the
body and doc of which seems to have been removed, or haven't
exist since the beginning.
- Some comments on the code
-- PQbatchProcessQueue()
PQbatchProcessQueue() returns 0 for several undistinguishable
reasons. If asyncStatus is PGASYNC_BUSY at the time, the
connection is out of sync and doesn't accept further
operations. So it should be distinguished from the end-of-queue
case. (It can reutrn an error result if combining with
PQgetResult.)
The function handles COPY_* in inconsistent way. It sets an error
message to conn in hardly noticeable way but allows to go
further. The document is telling as the follows.
COPY is not recommended as it most likely will trigger failure
in batch processing
I don't think the desciription is informative, but the reason for
the description would be the fact that we cannot detect a command
leads to protocol failure before the failure actually
happens. The test code suggests that that is the case where any
command (other than Sync) following "COPY FROM" breaks the
protocol. We get an error messages like below in the case. (The
header part is of my test program.)
ABORTED: (PGRES_BATCH_ABORTED) ERROR: unexpected message type 0x42 during COPY from stdin
CONTEXT: COPY t, line 1
I haven't investigated further, but it seems unable to restore
the sync of connection until disconnection. It would need a
side-channel for COPY_IN to avoid the failure but we don't have
it now. The possible choices here are:
A. Make the description more precisely, by adding the condition
to cause the error and how it looks.
B. The situation seems detectable in PGgetResult. Return an error
result containing any meaningful error. But this doesn't stop
the following protocol error messages.
ABORTED: (PGRES_FATAL_ERROR) COPY_IN cannot be followed by any command in a batch
ABORTED: (PGRES_FATAL_ERROR) ERROR: 08P01: unexpected message type 0x42 during COPY from stdin
ABORTED: (PGRES_BATCH_ABORTED) ERROR: unexpected message type 0x42 during COPY from stdin
ABORTED: (PGRES_BATCH_ABORTED) ERROR: unexpected message type 0x42 during COPY from stdin
C. Cancel COPY on the server and return error to the client for
the situation. This might let us be able to avoid
unrecoverable desync of the protocol in the case. (I'm not
sure this can be done cleanly.)
D. Wait for a protocol change that can solve this problem.
-- PQbatchFlush()
static int pqBatchFlush(Pgconn *conn)
...
if (...)
return(pqFlush(conn));
return 0; /* Just to keep compiler quiet */
The comment in the last line is wrong.
+ if ((conn->batch_status == PQBATCH_MODE_OFF)||(conn->outCount >= OUTBUFFER_T
Isn't this a bit too long?
-- PGcommandQueueEntry
The resason for the fact that it is a linked list seems to me to
allow queueing and result processing happen alternately in one
batch period. But I coundn't find a description about the
behavior in the documentation. (State transition chart is
required?)
Aside from the above, the interface to handle the command queue
seems to be divided into too-small pieces.
PQsendQueryGuts()
el = PQmakePipelinedCommand(conn)
el->members = hoge;
..
PQappendPipelinedCommand(conn, el)
Isn't the following enough instead of the above?
PQappendPipelinedCommand(conn, conn->queryclass, conn->last_query);
regards,
--
Kyotaro Horiguchi
NTT Open Source Software Center
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
Kyotaro HORIGUCHI wrote:
A disucssion on psql batch mode was held in another branch of
this thread. How do we treat that?
There's a batch mode for pgbench in a patch posted in [1]/messages/by-id/b4e34135-2bd9-4b8a-94ca-27d760da26d7@manitou-mail.org,
with \beginbatch and \endbatch commands, but nothing
for psql AFAICS.
psql is more complicated because currently it uses a
blocking PQexec() call at its core. Craig mentioned psql
integration in [2]/messages/by-id/CAMsr+YGLhaDkjymLuNVQy4MrSKQoA=F1vO=aN8XQf30N=aQuVA@mail.gmail.com and [3]/messages/by-id/CAMsr+YE6BK4iAaQz=nY3xDnbLhnNZ_4tp-PTJqbNNpsZMgoo8Q@mail.gmail.com.
Also a script can have inter-query dependencies such as in
insert into table(...) returning id \gset
update othertable set col= :id where ...;
which is a problem in batch mode, as we don't want
to send the update before the right value for :id is
known. Whether we want to support these dependencies
and how needs discussion.
For instance we might not support them at all, or
create a synchronization command that collects
all results of queries sent so far, or do it implicitly
when a variable is injected into a query...
This looks like substantial work that might be best
done separately from the libpq patch.
[1]: /messages/by-id/b4e34135-2bd9-4b8a-94ca-27d760da26d7@manitou-mail.org
/messages/by-id/b4e34135-2bd9-4b8a-94ca-27d760da26d7@manitou-mail.org
[2]: /messages/by-id/CAMsr+YGLhaDkjymLuNVQy4MrSKQoA=F1vO=aN8XQf30N=aQuVA@mail.gmail.com
/messages/by-id/CAMsr+YGLhaDkjymLuNVQy4MrSKQoA=F1vO=aN8XQf30N=aQuVA@mail.gmail.com
[3]: /messages/by-id/CAMsr+YE6BK4iAaQz=nY3xDnbLhnNZ_4tp-PTJqbNNpsZMgoo8Q@mail.gmail.com
/messages/by-id/CAMsr+YE6BK4iAaQz=nY3xDnbLhnNZ_4tp-PTJqbNNpsZMgoo8Q@mail.gmail.com
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
On Fri, Mar 23, 2018 at 02:18:09PM +0100, Daniel Verite wrote:
There's a batch mode for pgbench in a patch posted in [1],
with \beginbatch and \endbatch commands, but nothing
for psql AFAICS.
psql is more complicated because currently it uses a
blocking PQexec() call at its core. Craig mentioned psql
integration in [2] and [3].
This patch has been around for some time now, the last version fails to
apply cleanly and in-depth reviews have happened. I am moving that to
the next CF, waiting on its author.
--
Michael
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
On Mon, Oct 1, 2018 at 8:34 AM Michael Paquier <michael@paquier.xyz> wrote:
On Fri, Mar 23, 2018 at 02:18:09PM +0100, Daniel Verite wrote:
There's a batch mode for pgbench in a patch posted in [1],
with \beginbatch and \endbatch commands, but nothing
for psql AFAICS.
psql is more complicated because currently it uses a
blocking PQexec() call at its core. Craig mentioned psql
integration in [2] and [3].This patch has been around for some time now, the last version fails to
apply cleanly and in-depth reviews have happened. I am moving that to
the next CF, waiting on its author.
Unfortunately, nothing was changed since then, so there is already some amount
of unaddressed review feedback. I'll move this patch to "Returned with
feedback".
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
Hi,
This patch has been around for some time now, the last version fails to
apply cleanly and in-depth reviews have happened. I am moving that to
the next CF, waiting on its author.Unfortunately, nothing was changed since then, so there is already some amount
of unaddressed review feedback. I'll move this patch to "Returned with
feedback".
Craig Ringer mentioned about this thread to me recently.
This effort has seen decent reviews from Craig, Andres and Michael
already. So, I thought of refreshing it to work against latest master
HEAD.
PFA, main patch as well as the test patch (I named the test patch v17
to be consistent with the main patch). The major grouse with the test
patch AFAICS was the use of non-Windows compliant timersub() function.
I have now used INSTR_TIME_SET_CURRENT/INSTR_TIME_SUBTRACT family of
portable macros for the same.
Please let me know on what we think of the above.
Regards,
Nikhil
--
Nikhil Sontakke
2ndQuadrant - PostgreSQL Solutions for the Enterprise
https://www.2ndQuadrant.com/
Attachments:
0002-libpq_batch_tests_community_master.v17.patchapplication/octet-stream; name=0002-libpq_batch_tests_community_master.v17.patchDownload
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 953905d1ef..6eeec4f397 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -20,6 +20,7 @@ SUBDIRS = \
test_rbtree \
test_rls_hooks \
test_shm_mq \
+ test_libpq \
unsafe_tests \
worker_spi
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000000..11e8463984
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000000..d907063f65
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,25 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+testlibpqbatch.o: testlibpqbatch.c
+testlibpqbatch: testlibpqbatch.o
+check: testlibpqbatch prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000000..25f9ff0f89
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,25 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_batch simple_batch multi_batch batch_abort timings singlerowmode);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(['testlibpqbatch', 'dbname=postgres', "$numrows", "$testname"],
+ "testlibpqbatch $testname");
+}
+
+$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000000..bcebde1a01
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1456 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution functionality
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "c.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+static void test_singlerowmode(PGconn *conn);
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+ fflush(stderr);
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode\n");
+ goto fail;
+ }
+
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode with work in progress should fail, but succeeded\n");
+ goto fail;
+ }
+
+ if (!PQbatchSendQueue(conn))
+ {
+ fprintf(stderr, "Ending a batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * in batch mode we have to ask for the first result to be processed;
+ * until we do PQgetResult will return null:
+ */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchProcessQueue() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* We can't PQbatchProcessQueue when there might still be pending results */
+ if (PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() should've failed with pending results: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchProcessQueue() call\n");
+ goto fail;
+ }
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ /* should now get an explicit sync result */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after end batch call\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+ fflush(stderr);
+
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode: %u\n", __LINE__);
+ goto fail;
+ }
+
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "Unable to enter batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not activated properly\n");
+ goto fail;
+ }
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "PQexec should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ {
+ fprintf(stderr, "PQsendQuery should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "re-entering batch mode should be a no-op but failed\n");
+ goto fail;
+ }
+
+ if (PQisBusy(conn))
+ {
+ fprintf(stderr, "PQisBusy should return false when idle in batch, returned true\n");
+ goto fail;
+ }
+
+ /* ok, back to normal command mode */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "couldn't exit idle empty batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not terminated properly\n");
+ goto fail;
+ }
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "batch mode exit when not in batch mode should succeed but failed\n");
+ goto fail;
+ }
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+ fflush(stderr);
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchSendQueue(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchSendQueue(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* OK, start processing the batch results */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchProcessQueue() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchProcessQueue() call\n");
+ goto fail;
+ }
+
+ if (PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ /* second batch */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+ fflush(stderr);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "1";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching error select failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "2";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchSendQueue(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "3";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchSendQueue(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* second query, caused error */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* third query in batch, the second insert */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at third batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the batch sync */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "sync should've cleared the aborted flag but didn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the insert from the second batch */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first entry in second batch: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first item in second batch\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* the second batch sync */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch sync\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ {
+ fprintf(stderr, "expected only insert with value 3, got %s", val);
+ goto fail;
+ }
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ fprintf(stderr, "expected 1 result, got %d", PQntuples(res));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "xact start failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent BEGIN\n");
+
+ send_step = BI_DROP_TABLE;
+
+ if (!PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent DROP\n");
+
+ send_step = BI_CREATE_TABLE;
+
+ if (!PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent CREATE\n");
+
+ send_step = BI_PREPARE;
+
+ if (!PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids))
+ {
+ fprintf(stderr, "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent PREPARE\n");
+
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ {
+ fprintf(stderr, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "Expected next query result but unable to dequeue: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ fprintf(stdout, "next query!\n");
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+
+ if (PQresultStatus(res) != status)
+ {
+ fprintf(stderr, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ {
+ fprintf(stderr, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+ goto fail;
+ }
+
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchSendQueue(conn))
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ {
+ fprintf(stderr, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "prepare failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "COPY: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[12 + 2];
+ int formatted = snprintf(&buf[0], 12 + 1, "%d\n", nrows);
+
+ if (formatted >= 12 + 1)
+ {
+ fprintf(stderr, "Buffer write truncated somehow\n");
+ goto fail;
+ }
+
+ if (PQputCopyData(conn, buf, formatted) != 1)
+ {
+ fprintf(stderr, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ nrows--;
+ }
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ instr_time start_time, end_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_pipelined(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+
+ printf("batch insert elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_sequential(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+ printf("sequential insert elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_copy(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+ printf("COPY elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings|singlerowmode\n");
+ exit(1);
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i,r;
+
+ /* 1 batch, 3 queries in it */
+ r = PQenterBatchMode(conn);
+
+ for (i=0; i < 3; i++) {
+ r = PQsendQueryParams(conn,
+ "SELECT 1",
+ 0,
+ NULL,
+ NULL,
+ NULL,
+ NULL,
+ 0);
+ }
+ PQbatchSendQueue(conn);
+
+ i=0;
+ while (PQbatchProcessQueue(conn))
+ {
+ int isSingleTuple = 0;
+ /* Set single row mode for only first 3 SELECT queries */
+ if(i < 3)
+ r = PQsetSingleRowMode(conn);
+ if (r!=1)
+ {
+ fprintf(stderr, "PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+ fprintf(stderr, "Result status: %d (%s) for i=%d", est, PQresStatus(est), i);
+ if (est == PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ if(!isSingleTuple)
+ {
+ fprintf(stderr, " Expected to follow PGREG_SINGLE_TUPLE, but received PGRES_TUPLES_OK directly instead\n");
+ goto fail;
+ }
+ isSingleTuple=0;
+ }
+ else if (est == PGRES_SINGLE_TUPLE)
+ {
+ isSingleTuple = 1;
+ fprintf(stderr, ", single tuple: %d\n", PQntuples(res));
+ }
+ else if (est == PGRES_BATCH_END)
+ {
+ fprintf(stderr, ", end of batch reached\n");
+ }
+ else if (est != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, ", error: %s\n", PQresultErrorMessage(res));
+ }
+ PQclear(res);
+ }
+ i++;
+ }
+ PQexitBatchMode(conn);
+ PQclear(res);
+ res = NULL;
+ return;
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+
+}
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1,
+ run_singlerowmode = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ run_singlerowmode = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else if (strcmp(argv[3], "singlerowmode") == 0)
+ run_singlerowmode = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognized test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+ if(run_singlerowmode)
+ test_singlerowmode(conn);
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 2eab635898..7c67fc9330 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -48,7 +48,8 @@ my @contrib_excludes = (
'ltree_plpython', 'pgcrypto',
'sepgsql', 'brin',
'test_extensions', 'test_pg_dump',
- 'snapshot_too_old', 'unsafe_tests');
+ 'snapshot_too_old', 'unsafe_tests',
+ 'test_libpq');
# Set of variables for frontend modules
my $frontend_defines = { 'initdb' => 'FRONTEND' };
0001-libpq_batch_support_commmunity_master.v17.patchapplication/octet-stream; name=0001-libpq_batch_support_commmunity_master.v17.patchDownload
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index b7c3d96b01..50d4cd279c 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4663,6 +4663,500 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ can sometimes offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is not useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 7.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ a connection into batch mode. Enter batch mode with <link
+ linkend="libpq-PQenterBatchMode"><function>PQenterBatchMode(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ Using any synchronous command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error conditions.
+ Functions allowed in batch mode are described in <xref linkend="libpq-batch-sending"/>.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchSendQueue</function>.
+ And to get results, it uses <function>PQgetResult</function> and
+ <function>PQbatchProcessQueue</function>. It may eventually exit
+ batch mode with <function>PQexitBatchMode</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ Batch mode consumes more memory when send/recv is not done as required even in non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchSendQueue"><function>PQbatchSendQueue(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing</link> with sending batch queries, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchProcessQueue"><function>PQbatchProcessQueue</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null. The result from the next batch entry
+ may then be retrieved using <function>PQbatchProcessQueue</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
+ successful call of <function>PQbatchProcessQueue</function>. This mode selection is effective
+ only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode"/>.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchSendQueue</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchSendQueue</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchProcessQueue(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue.
+ Based on available memory, results from socket should be read frequently and
+ there's no need to wait till the batch end to read the results. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQexitBatchMode"><function>PQexitBatchMode(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchSendQueue</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterBatchMode">
+ <term>
+ <function>PQenterBatchMode</function>
+ <indexterm>
+ <primary>PQenterBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQenterBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitBatchMode">
+ <term>
+ <function>PQexitBatchMode</function>
+ <indexterm>
+ <primary>PQexitBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitBatchMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchProcessQueue</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchSendQueue">
+ <term>
+ <function>PQbatchSendQueue</function>
+ <indexterm>
+ <primary>PQbatchSendQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchSendQueue(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchProcessQueue">
+ <term>
+ <function>PQbatchProcessQueue</function>
+ <indexterm>
+ <primary>PQbatchProcessQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchProcessQueue(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
@@ -4703,6 +5197,14 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <note>
+ <para>
+ On using batch mode, call <function>PQsetSingleRowMode</function>
+ immediately after a successful call of <function>PQbatchProcessQueue</function>
+ See <xref linkend="libpq-batch-mode"/> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 34dcfabe73..6f527267e0 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 6eba08a920..d41164edfa 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -984,6 +984,9 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->status = WALRCV_ERROR;
walres->err = pchomp(PQerrorMessage(conn->streamConn));
break;
+ default:
+ /* This is just to keep compiler quiet */
+ break;
}
PQclear(pgres);
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 7c808e5215..ffcbe50c61 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -176,3 +176,8 @@ PQresultMemorySize 173
PQhostaddr 174
PQgssEncInUse 175
PQgetgssctx 176
+PQenterBatchMode 177
+PQexitBatchMode 178
+PQbatchSendQueue 179
+PQbatchProcessQueue 180
+PQbatchStatus 181
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 7f1fd2f45e..1a9a552e53 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -502,6 +502,24 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
/*
* pqDropServerData
@@ -521,6 +539,7 @@ pqDropServerData(PGconn *conn)
{
PGnotify *notify;
pgParameterStatus *pstatus;
+ PGcommandQueueEntry *queue;
/* Forget pending notifies */
notify = conn->notifyHead;
@@ -533,6 +552,14 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index b3c59a0992..db79015100 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -40,7 +40,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -71,7 +73,10 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
-
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static int pqBatchFlush(PGconn *conn);
/* ----------------
* Space management for PGresult.
@@ -1211,7 +1216,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1234,6 +1239,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1332,6 +1344,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1390,31 +1406,51 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1462,7 +1498,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, libpq_gettext("tried to recycle non-dangling command queue entry"));
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1480,20 +1589,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1517,6 +1666,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1526,6 +1679,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1638,35 +1808,42 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1767,6 +1944,280 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQenterBatchMode
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return 0;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return 1;
+}
+
+/*
+ * PQexitBatchMode
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns 1 if batch mode ended.
+ */
+int
+PQexitBatchMode(PGconn *conn)
+{
+ if (!conn)
+ goto exitFailed;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ goto exitFailed;
+ default:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ goto exitFailed;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ goto sendFailed;
+ return 1;
+
+sendFailed:
+ /* error message should be set up already */
+
+exitFailed:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, Failed to exit batch mode"));
+ return 0;
+}
+
+/*
+ * PQbatchSendQueue
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new synchronous command execution
+ * functions until all results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchSendQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ return 1;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
+
+/*
+ * PQbatchProcessQueue
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns 1 if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns 0 if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return 0;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return 0;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ return 0;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return 1;
+}
+
/*
* PQgetResult
@@ -1843,9 +2294,30 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
+ res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, query execution state cannot be IDLE as there
+ * can be other queries or results waiting in the queue
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -2026,6 +2498,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2220,6 +2699,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2235,6 +2717,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2243,15 +2739,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ /* reset last-query string (not relevant now) */
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2261,14 +2760,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -2666,6 +3169,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -3859,3 +4369,14 @@ PQunescapeBytea(const unsigned char *strtext, size_t *retbuflen)
*retbuflen = buflen;
return tmpbuf;
}
+/* pqBatchFlush
+ * In batch mode, data will be flushed only when the out buffer reaches the threshold value.
+ * In non-batch mode, data will be flushed all the time.
+ */
+static int
+pqBatchFlush(PGconn *conn)
+{
+ if ((conn->batch_status == PQBATCH_MODE_OFF)||(conn->outCount >= OUTBUFFER_THRESHOLD))
+ return(pqFlush(conn));
+ return 0; /* Just to keep compiler quiet */
+}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 0e36974fc4..8357334027 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -411,6 +411,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index b04f7ec123..0cb1ac03f9 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -219,10 +219,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -879,6 +887,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 22c4954f2b..bb452f28c5 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -96,7 +96,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -136,6 +139,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -434,6 +448,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQenterBatchMode(PGconn *conn);
+extern int PQexitBatchMode(PGconn *conn);
+extern int PQbatchSendQueue(PGconn *conn);
+extern int PQbatchProcessQueue(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index d37bb3ce40..eed6945cff 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,10 +217,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -229,7 +234,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the pqSetenv state machine */
@@ -301,6 +307,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -389,6 +411,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -401,6 +424,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -781,6 +814,12 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold is set to 64k as it is safe
+ * in Windows as per comments in pqSendSome() API.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
@@ -789,6 +828,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
On Fri, Aug 30, 2019 at 7:06 PM Nikhil Sontakke <nikhils@2ndquadrant.com> wrote:
Hi,
This patch has been around for some time now, the last version fails to
apply cleanly and in-depth reviews have happened. I am moving that to
the next CF, waiting on its author.Unfortunately, nothing was changed since then, so there is already some amount
of unaddressed review feedback. I'll move this patch to "Returned with
feedback".Craig Ringer mentioned about this thread to me recently.
This effort has seen decent reviews from Craig, Andres and Michael
already. So, I thought of refreshing it to work against latest master
HEAD.
Thanks for picking up this. However, I noticed that previously
Horiguchi-San has given some comments on this patch [1]/messages/by-id/20180322.211148.187821341.horiguchi.kyotaro@lab.ntt.co.jp which doesn't
seem to be addressed or at least not all of them are addressed. It is
possible that you would have already addressed those, but in that
case, it would be good if you respond to his email as well. If those
are not addressed, then it will be good to address those.
[1]: /messages/by-id/20180322.211148.187821341.horiguchi.kyotaro@lab.ntt.co.jp
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
On 2019-Sep-09, Amit Kapila wrote:
Thanks for picking up this. However, I noticed that previously
Horiguchi-San has given some comments on this patch [1] which doesn't
seem to be addressed or at least not all of them are addressed. It is
possible that you would have already addressed those, but in that
case, it would be good if you respond to his email as well. If those
are not addressed, then it will be good to address those.
Totally unasked for, here's a rebase of this patch series. I didn't do
anything other than rebasing to current master, solving a couple of very
trivial conflicts, fixing some whitespace complaints by git apply, and
running tests to verify everthing works.
I don't foresee working on this at all, so if anyone is interested in
seeing this feature in, I encourage them to read and address
Horiguchi-san's feedback.
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
v18-0001-libpq-batch-support.patchtext/x-diff; charset=us-asciiDownload
From c2de4e2a2cbc301b48a7946a9ee0b9acceae7233 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Fri, 10 Jul 2020 11:29:11 -0400
Subject: [PATCH v18 1/2] libpq batch support
---
doc/src/sgml/libpq.sgml | 502 +++++++++++++++
doc/src/sgml/lobj.sgml | 4 +
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/interfaces/libpq/exports.txt | 5 +
src/interfaces/libpq/fe-connect.c | 27 +
src/interfaces/libpq/fe-exec.c | 597 ++++++++++++++++--
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 15 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 47 +-
10 files changed, 1186 insertions(+), 44 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index d1ccaa775a..ad168727c7 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4826,6 +4826,500 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ can sometimes offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is not useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 7.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ a connection into batch mode. Enter batch mode with <link
+ linkend="libpq-PQenterBatchMode"><function>PQenterBatchMode(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ Using any synchronous command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error conditions.
+ Functions allowed in batch mode are described in <xref linkend="libpq-batch-sending"/>.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchSendQueue</function>.
+ And to get results, it uses <function>PQgetResult</function> and
+ <function>PQbatchProcessQueue</function>. It may eventually exit
+ batch mode with <function>PQexitBatchMode</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ Batch mode consumes more memory when send/recv is not done as required even in non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchSendQueue"><function>PQbatchSendQueue(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing</link> with sending batch queries, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchProcessQueue"><function>PQbatchProcessQueue</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null. The result from the next batch entry
+ may then be retrieved using <function>PQbatchProcessQueue</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
+ successful call of <function>PQbatchProcessQueue</function>. This mode selection is effective
+ only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode"/>.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchSendQueue</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchSendQueue</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchProcessQueue(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue.
+ Based on available memory, results from socket should be read frequently and
+ there's no need to wait till the batch end to read the results. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQexitBatchMode"><function>PQexitBatchMode(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchSendQueue</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterBatchMode">
+ <term>
+ <function>PQenterBatchMode</function>
+ <indexterm>
+ <primary>PQenterBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQenterBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitBatchMode">
+ <term>
+ <function>PQexitBatchMode</function>
+ <indexterm>
+ <primary>PQexitBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitBatchMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchProcessQueue</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchSendQueue">
+ <term>
+ <function>PQbatchSendQueue</function>
+ <indexterm>
+ <primary>PQbatchSendQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchSendQueue(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchProcessQueue">
+ <term>
+ <function>PQbatchProcessQueue</function>
+ <indexterm>
+ <primary>PQbatchProcessQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchProcessQueue(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4866,6 +5360,14 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <note>
+ <para>
+ On using batch mode, call <function>PQsetSingleRowMode</function>
+ immediately after a successful call of <function>PQbatchProcessQueue</function>
+ See <xref linkend="libpq-batch-mode"/> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index cf4653fe0f..a1b46c83ed 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index e4fd1f9bb6..5cac751d99 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1001,6 +1001,9 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->status = WALRCV_ERROR;
walres->err = pchomp(PQerrorMessage(conn->streamConn));
break;
+ default:
+ /* This is just to keep compiler quiet */
+ break;
}
PQclear(pgres);
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..ffb18fcf47 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,8 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterBatchMode 180
+PQexitBatchMode 181
+PQbatchSendQueue 182
+PQbatchProcessQueue 183
+PQbatchStatus 184
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 27c9bb46ee..0e665b3df4 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -529,6 +529,24 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
/*
* pqDropServerData
@@ -548,6 +566,7 @@ pqDropServerData(PGconn *conn)
{
PGnotify *notify;
pgParameterStatus *pstatus;
+ PGcommandQueueEntry *queue;
/* Forget pending notifies */
notify = conn->notifyHead;
@@ -560,6 +579,14 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index eea0237c3a..eb9b8fc8c5 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -70,7 +72,10 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
-
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static int pqBatchFlush(PGconn *conn);
/* ----------------
* Space management for PGresult.
@@ -1210,7 +1215,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1233,6 +1238,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1331,6 +1343,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1389,31 +1405,51 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1461,7 +1497,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, libpq_gettext("tried to recycle non-dangling command queue entry"));
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1479,20 +1588,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1516,6 +1665,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1525,6 +1678,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1637,35 +1807,42 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1766,6 +1943,280 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQenterBatchMode
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return 0;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return 1;
+}
+
+/*
+ * PQexitBatchMode
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns 1 if batch mode ended.
+ */
+int
+PQexitBatchMode(PGconn *conn)
+{
+ if (!conn)
+ goto exitFailed;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ goto exitFailed;
+ default:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ goto exitFailed;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ goto sendFailed;
+ return 1;
+
+sendFailed:
+ /* error message should be set up already */
+
+exitFailed:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, Failed to exit batch mode"));
+ return 0;
+}
+
+/*
+ * PQbatchSendQueue
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new synchronous command execution
+ * functions until all results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchSendQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ return 1;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
+
+/*
+ * PQbatchProcessQueue
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns 1 if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns 0 if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return 0;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return 0;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ return 0;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return 1;
+}
+
/*
* PQgetResult
@@ -1842,9 +2293,30 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
+ res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, query execution state cannot be IDLE as there
+ * can be other queries or results waiting in the queue
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -2025,6 +2497,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2219,6 +2698,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2234,6 +2716,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2242,15 +2738,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ /* reset last-query string (not relevant now) */
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2260,14 +2759,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -2665,6 +3168,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -3858,3 +4368,14 @@ PQunescapeBytea(const unsigned char *strtext, size_t *retbuflen)
*retbuflen = buflen;
return tmpbuf;
}
+/* pqBatchFlush
+ * In batch mode, data will be flushed only when the out buffer reaches the threshold value.
+ * In non-batch mode, data will be flushed all the time.
+ */
+static int
+pqBatchFlush(PGconn *conn)
+{
+ if ((conn->batch_status == PQBATCH_MODE_OFF)||(conn->outCount >= OUTBUFFER_THRESHOLD))
+ return(pqFlush(conn));
+ return 0; /* Just to keep compiler quiet */
+}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 9360c541be..2ff3fa4883 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -406,6 +406,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 1696525475..9c37a9c4ff 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -217,10 +217,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -875,6 +883,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 3b6a9fbce3..fc0e590985 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -97,7 +97,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -137,6 +140,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -435,6 +449,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQenterBatchMode(PGconn *conn);
+extern int PQexitBatchMode(PGconn *conn);
+extern int PQbatchSendQueue(PGconn *conn);
+extern int PQbatchProcessQueue(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 1de91ae295..88d168f804 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,10 +217,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -229,7 +234,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the pqSetenv state machine */
@@ -301,6 +307,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -394,6 +416,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -406,6 +429,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -798,6 +831,12 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold is set to 64k as it is safe
+ * in Windows as per comments in pqSendSome() API.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
@@ -806,6 +845,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.20.1
v18-0002-libpq-batch-tests.patchtext/x-diff; charset=us-asciiDownload
From 47c4e9c0af9dc5abe6dd17101945ab4f10291958 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Fri, 10 Jul 2020 11:30:38 -0400
Subject: [PATCH v18 2/2] libpq batch tests
---
src/test/modules/Makefile | 1 +
src/test/modules/test_libpq/.gitignore | 5 +
src/test/modules/test_libpq/Makefile | 25 +
src/test/modules/test_libpq/README | 1 +
.../modules/test_libpq/t/001_libpq_async.pl | 27 +
src/test/modules/test_libpq/testlibpqbatch.c | 1456 +++++++++++++++++
src/tools/msvc/Mkvcbuild.pm | 16 +-
7 files changed, 1523 insertions(+), 8 deletions(-)
create mode 100644 src/test/modules/test_libpq/.gitignore
create mode 100644 src/test/modules/test_libpq/Makefile
create mode 100644 src/test/modules/test_libpq/README
create mode 100644 src/test/modules/test_libpq/t/001_libpq_async.pl
create mode 100644 src/test/modules/test_libpq/testlibpqbatch.c
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 29de73c060..a4ba8c04c1 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -22,6 +22,7 @@ SUBDIRS = \
test_rbtree \
test_rls_hooks \
test_shm_mq \
+ test_libpq \
unsafe_tests \
worker_spi
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000000..11e8463984
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000000..d907063f65
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,25 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+testlibpqbatch.o: testlibpqbatch.c
+testlibpqbatch: testlibpqbatch.o
+check: testlibpqbatch prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000000..2c361880fa
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,27 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests =
+ qw(disallowed_in_batch simple_batch multi_batch batch_abort timings singlerowmode);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'testlibpqbatch', 'dbname=postgres', "$numrows", "$testname" ],
+ "testlibpqbatch $testname");
+}
+
+$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000000..4d6ba266e5
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1456 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution functionality
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "c.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+static void test_singlerowmode(PGconn *conn);
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+ fflush(stderr);
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode\n");
+ goto fail;
+ }
+
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode with work in progress should fail, but succeeded\n");
+ goto fail;
+ }
+
+ if (!PQbatchSendQueue(conn))
+ {
+ fprintf(stderr, "Ending a batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * in batch mode we have to ask for the first result to be processed;
+ * until we do PQgetResult will return null:
+ */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchProcessQueue() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* We can't PQbatchProcessQueue when there might still be pending results */
+ if (PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() should've failed with pending results: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchProcessQueue() call\n");
+ goto fail;
+ }
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ /* should now get an explicit sync result */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after end batch call\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+ fflush(stderr);
+
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode: %u\n", __LINE__);
+ goto fail;
+ }
+
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "Unable to enter batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not activated properly\n");
+ goto fail;
+ }
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "PQexec should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ {
+ fprintf(stderr, "PQsendQuery should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "re-entering batch mode should be a no-op but failed\n");
+ goto fail;
+ }
+
+ if (PQisBusy(conn))
+ {
+ fprintf(stderr, "PQisBusy should return false when idle in batch, returned true\n");
+ goto fail;
+ }
+
+ /* ok, back to normal command mode */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "couldn't exit idle empty batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not terminated properly\n");
+ goto fail;
+ }
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "batch mode exit when not in batch mode should succeed but failed\n");
+ goto fail;
+ }
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+ fflush(stderr);
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchSendQueue(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchSendQueue(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* OK, start processing the batch results */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchProcessQueue() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchProcessQueue() call\n");
+ goto fail;
+ }
+
+ if (PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ /* second batch */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+ fflush(stderr);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "1";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching error select failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "2";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchSendQueue(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "3";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchSendQueue(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* second query, caused error */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* third query in batch, the second insert */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at third batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the batch sync */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "sync should've cleared the aborted flag but didn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the insert from the second batch */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first entry in second batch: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first item in second batch\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* the second batch sync */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch sync\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ {
+ fprintf(stderr, "expected only insert with value 3, got %s", val);
+ goto fail;
+ }
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ fprintf(stderr, "expected 1 result, got %d", PQntuples(res));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "xact start failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent BEGIN\n");
+
+ send_step = BI_DROP_TABLE;
+
+ if (!PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent DROP\n");
+
+ send_step = BI_CREATE_TABLE;
+
+ if (!PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent CREATE\n");
+
+ send_step = BI_PREPARE;
+
+ if (!PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids))
+ {
+ fprintf(stderr, "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent PREPARE\n");
+
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ {
+ fprintf(stderr, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "Expected next query result but unable to dequeue: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ fprintf(stdout, "next query!\n");
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+
+ if (PQresultStatus(res) != status)
+ {
+ fprintf(stderr, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ {
+ fprintf(stderr, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+ goto fail;
+ }
+
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchSendQueue(conn))
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ {
+ fprintf(stderr, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "prepare failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "COPY: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[12 + 2];
+ int formatted = snprintf(&buf[0], 12 + 1, "%d\n", nrows);
+
+ if (formatted >= 12 + 1)
+ {
+ fprintf(stderr, "Buffer write truncated somehow\n");
+ goto fail;
+ }
+
+ if (PQputCopyData(conn, buf, formatted) != 1)
+ {
+ fprintf(stderr, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ nrows--;
+ }
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ instr_time start_time, end_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_pipelined(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+
+ printf("batch insert elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_sequential(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+ printf("sequential insert elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_copy(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+ printf("COPY elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings|singlerowmode\n");
+ exit(1);
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i,r;
+
+ /* 1 batch, 3 queries in it */
+ r = PQenterBatchMode(conn);
+
+ for (i=0; i < 3; i++) {
+ r = PQsendQueryParams(conn,
+ "SELECT 1",
+ 0,
+ NULL,
+ NULL,
+ NULL,
+ NULL,
+ 0);
+ }
+ PQbatchSendQueue(conn);
+
+ i=0;
+ while (PQbatchProcessQueue(conn))
+ {
+ int isSingleTuple = 0;
+ /* Set single row mode for only first 3 SELECT queries */
+ if(i < 3)
+ r = PQsetSingleRowMode(conn);
+ if (r!=1)
+ {
+ fprintf(stderr, "PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+ fprintf(stderr, "Result status: %d (%s) for i=%d", est, PQresStatus(est), i);
+ if (est == PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ if(!isSingleTuple)
+ {
+ fprintf(stderr, " Expected to follow PGREG_SINGLE_TUPLE, but received PGRES_TUPLES_OK directly instead\n");
+ goto fail;
+ }
+ isSingleTuple=0;
+ }
+ else if (est == PGRES_SINGLE_TUPLE)
+ {
+ isSingleTuple = 1;
+ fprintf(stderr, ", single tuple: %d\n", PQntuples(res));
+ }
+ else if (est == PGRES_BATCH_END)
+ {
+ fprintf(stderr, ", end of batch reached\n");
+ }
+ else if (est != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, ", error: %s\n", PQresultErrorMessage(res));
+ }
+ PQclear(res);
+ }
+ i++;
+ }
+ PQexitBatchMode(conn);
+ PQclear(res);
+ res = NULL;
+ return;
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+
+}
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1,
+ run_singlerowmode = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ run_singlerowmode = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else if (strcmp(argv[3], "singlerowmode") == 0)
+ run_singlerowmode = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognized test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+ if(run_singlerowmode)
+ test_singlerowmode(conn);
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 20da7985c1..0054a394ea 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -43,14 +43,14 @@ my $contrib_extrasource = {
'seg' => [ 'contrib/seg/segscan.l', 'contrib/seg/segparse.y' ],
};
my @contrib_excludes = (
- 'bool_plperl', 'commit_ts',
- 'hstore_plperl', 'hstore_plpython',
- 'intagg', 'jsonb_plperl',
- 'jsonb_plpython', 'ltree_plpython',
- 'pgcrypto', 'sepgsql',
- 'brin', 'test_extensions',
- 'test_misc', 'test_pg_dump',
- 'snapshot_too_old', 'unsafe_tests');
+ 'bool_plperl', 'commit_ts',
+ 'hstore_plperl', 'hstore_plpython',
+ 'intagg', 'jsonb_plperl',
+ 'jsonb_plpython', 'ltree_plpython',
+ 'pgcrypto', 'sepgsql',
+ 'brin', 'test_extensions',
+ 'test_pg_dump', 'snapshot_too_old',
+ 'unsafe_tests', 'test_libpq');
# Set of variables for frontend modules
my $frontend_defines = { 'initdb' => 'FRONTEND' };
--
2.20.1
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
Hi,
On 2020-07-10 19:01:49 -0400, Alvaro Herrera wrote:
Totally unasked for, here's a rebase of this patch series. I didn't do
anything other than rebasing to current master, solving a couple of very
trivial conflicts, fixing some whitespace complaints by git apply, and
running tests to verify everthing works.I don't foresee working on this at all, so if anyone is interested in
seeing this feature in, I encourage them to read and address
Horiguchi-san's feedback.
Nor am I planning to do so, but I do think its a pretty important
improvement.
+/* + * PQrecyclePipelinedCommand + * Push a command queue entry onto the freelist. It must be a dangling entry + * with null next pointer and not referenced by any other entry's next pointer. + */
Dangling sounds a bit like it's already freed.
+/* + * PQbatchSendQueue + * End a batch submission by sending a protocol sync. The connection will + * remain in batch mode and unavailable for new synchronous command execution + * functions until all results from the batch are processed by the client.
I feel like the reference to the protocol sync is a bit too low level
for an external API. It should first document what the function does
from a user's POV.
I think it'd also be good to document whether / whether not queries can
already have been sent before PQbatchSendQueue is called or not.
+/* + * PQbatchProcessQueue + * In batch mode, start processing the next query in the queue. + * + * Returns 1 if the next query was popped from the queue and can + * be processed by PQconsumeInput, PQgetResult, etc. + * + * Returns 0 if the current query isn't done yet, the connection + * is not in a batch, or there are no more queries to process. + */ +int +PQbatchProcessQueue(PGconn *conn) +{ + PGcommandQueueEntry *next_query; + + if (!conn) + return 0; + + if (conn->batch_status == PQBATCH_MODE_OFF) + return 0; + + switch (conn->asyncStatus) + { + case PGASYNC_COPY_IN: + case PGASYNC_COPY_OUT: + case PGASYNC_COPY_BOTH: + printfPQExpBuffer(&conn->errorMessage, + libpq_gettext_noop("internal error, COPY in batch mode")); + break;
Shouldn't there be a return 0 here?
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC) + { + /* + * In an aborted batch we don't get anything from the server for each + * result; we're just discarding input until we get to the next sync + * from the server. The client needs to know its queries got aborted + * so we create a fake PGresult to return immediately from + * PQgetResult. + */ + conn->result = PQmakeEmptyPGresult(conn, + PGRES_BATCH_ABORTED); + if (!conn->result) + { + printfPQExpBuffer(&conn->errorMessage, + libpq_gettext("out of memory")); + pqSaveErrorResult(conn); + return 0;
Is there any way an application can recover at this point? ISTM we'd be
stuck in the previous asyncStatus, no?
+/* pqBatchFlush + * In batch mode, data will be flushed only when the out buffer reaches the threshold value. + * In non-batch mode, data will be flushed all the time. + */ +static int +pqBatchFlush(PGconn *conn) +{ + if ((conn->batch_status == PQBATCH_MODE_OFF)||(conn->outCount >= OUTBUFFER_THRESHOLD)) + return(pqFlush(conn)); + return 0; /* Just to keep compiler quiet */ +}
unnecessarily long line.
+/* + * Connection's outbuffer threshold is set to 64k as it is safe + * in Windows as per comments in pqSendSome() API. + */ +#define OUTBUFFER_THRESHOLD 65536
I don't think the comment explains much. It's fine to send more than 64k
with pqSendSome(), they'll just be send with separate pgsecure_write()
invocations. And only on windows.
It clearly makes sense to start sending out data at a certain
granularity to avoid needing unnecessary amounts of memory, and to make
more efficient use of latency / serer side compute.
It's not implausible that 64k is the right amount for that, I just don't
think the explanation above is good.
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c new file mode 100644 index 0000000000..4d6ba266e5 --- /dev/null +++ b/src/test/modules/test_libpq/testlibpqbatch.c @@ -0,0 +1,1456 @@ +/* + * src/test/modules/test_libpq/testlibpqbatch.c + * + * + * testlibpqbatch.c + * Test of batch execution functionality + */ + +#ifdef WIN32 +#include <windows.h> +#endif
ISTM that this shouldn't be needed in a test program like this?
Shouldn't libpq abstract all of this away?
+static void +simple_batch(PGconn *conn) +{
ISTM that all or at least several of these should include tests of
transactional behaviour with pipelining (e.g. using both implicit and
explicit transactions inside a single batch, using transactions across
batches, multiple explicit transactions inside a batch).
+ PGresult *res = NULL; + const char *dummy_params[1] = {"1"}; + Oid dummy_param_oids[1] = {INT4OID}; + + fprintf(stderr, "simple batch... "); + fflush(stderr);
Why do we need fflush()? IMO that shouldn't be needed in a use like
this? Especially not on stderr, which ought to be unbuffered?
+ /* + * Enter batch mode and dispatch a set of operations, which we'll then + * process the results of as they come in. + * + * For a simple case we should be able to do this without interim + * processing of results since our out buffer will give us enough slush to + * work with and we won't block on sending. So blocking mode is fine. + */ + if (PQisnonblocking(conn)) + { + fprintf(stderr, "Expected blocking connection mode\n"); + goto fail; + }
Perhaps worth adding a helper for this?
#define EXPECT(condition, explanation) \
...
or such?
Greetings,
Andres Freund
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
On Wed, Jul 15, 2020 at 12:18 AM Andres Freund <andres@anarazel.de> wrote:
Hi,
On 2020-07-10 19:01:49 -0400, Alvaro Herrera wrote:
Totally unasked for, here's a rebase of this patch series. I didn't do
anything other than rebasing to current master, solving a couple of very
trivial conflicts, fixing some whitespace complaints by git apply, and
running tests to verify everthing works.I don't foresee working on this at all, so if anyone is interested in
seeing this feature in, I encourage them to read and address
Horiguchi-san's feedback.Nor am I planning to do so, but I do think its a pretty important
improvement.+/* + * PQrecyclePipelinedCommand + * Push a command queue entry onto the freelist. It must be adangling entry
+ * with null next pointer and not referenced by any other entry's
next pointer.
+ */
Dangling sounds a bit like it's already freed.
+/* + * PQbatchSendQueue + * End a batch submission by sending a protocol sync. The connectionwill
+ * remain in batch mode and unavailable for new synchronous command
execution
+ * functions until all results from the batch are processed by the
client.
I feel like the reference to the protocol sync is a bit too low level
for an external API. It should first document what the function does
from a user's POV.I think it'd also be good to document whether / whether not queries can
already have been sent before PQbatchSendQueue is called or not.+/* + * PQbatchProcessQueue + * In batch mode, start processing the next query in the queue. + * + * Returns 1 if the next query was popped from the queue and can + * be processed by PQconsumeInput, PQgetResult, etc. + * + * Returns 0 if the current query isn't done yet, the connection + * is not in a batch, or there are no more queries to process. + */ +int +PQbatchProcessQueue(PGconn *conn) +{ + PGcommandQueueEntry *next_query; + + if (!conn) + return 0; + + if (conn->batch_status == PQBATCH_MODE_OFF) + return 0; + + switch (conn->asyncStatus) + { + case PGASYNC_COPY_IN: + case PGASYNC_COPY_OUT: + case PGASYNC_COPY_BOTH: + printfPQExpBuffer(&conn->errorMessage, + libpq_gettext_noop("internal error,COPY in batch mode"));
+ break;
Shouldn't there be a return 0 here?
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass
!= PGQUERY_SYNC)
+ { + /* + * In an aborted batch we don't get anything from theserver for each
+ * result; we're just discarding input until we get to the
next sync
+ * from the server. The client needs to know its queries
got aborted
+ * so we create a fake PGresult to return immediately from + * PQgetResult. + */ + conn->result = PQmakeEmptyPGresult(conn, +PGRES_BATCH_ABORTED);
+ if (!conn->result) + { + printfPQExpBuffer(&conn->errorMessage, +libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn); + return 0;Is there any way an application can recover at this point? ISTM we'd be
stuck in the previous asyncStatus, no?+/* pqBatchFlush + * In batch mode, data will be flushed only when the out buffer reachesthe threshold value.
+ * In non-batch mode, data will be flushed all the time. + */ +static int +pqBatchFlush(PGconn *conn) +{ + if ((conn->batch_status == PQBATCH_MODE_OFF)||(conn->outCount >=OUTBUFFER_THRESHOLD))
+ return(pqFlush(conn)); + return 0; /* Just to keep compiler quiet */ +}unnecessarily long line.
+/* + * Connection's outbuffer threshold is set to 64k as it is safe + * in Windows as per comments in pqSendSome() API. + */ +#define OUTBUFFER_THRESHOLD 65536I don't think the comment explains much. It's fine to send more than 64k
with pqSendSome(), they'll just be send with separate pgsecure_write()
invocations. And only on windows.It clearly makes sense to start sending out data at a certain
granularity to avoid needing unnecessary amounts of memory, and to make
more efficient use of latency / serer side compute.It's not implausible that 64k is the right amount for that, I just don't
think the explanation above is good.diff --git a/src/test/modules/test_libpq/testlibpqbatch.cb/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644 index 0000000000..4d6ba266e5 --- /dev/null +++ b/src/test/modules/test_libpq/testlibpqbatch.c @@ -0,0 +1,1456 @@ +/* + * src/test/modules/test_libpq/testlibpqbatch.c + * + * + * testlibpqbatch.c + * Test of batch execution functionality + */ + +#ifdef WIN32 +#include <windows.h> +#endifISTM that this shouldn't be needed in a test program like this?
Shouldn't libpq abstract all of this away?+static void +simple_batch(PGconn *conn) +{ISTM that all or at least several of these should include tests of
transactional behaviour with pipelining (e.g. using both implicit and
explicit transactions inside a single batch, using transactions across
batches, multiple explicit transactions inside a batch).+ PGresult *res = NULL; + const char *dummy_params[1] = {"1"}; + Oid dummy_param_oids[1] = {INT4OID}; + + fprintf(stderr, "simple batch... "); + fflush(stderr);Why do we need fflush()? IMO that shouldn't be needed in a use like
this? Especially not on stderr, which ought to be unbuffered?+ /* + * Enter batch mode and dispatch a set of operations, which we'llthen
+ * process the results of as they come in. + * + * For a simple case we should be able to do this without interim + * processing of results since our out buffer will give us enoughslush to
+ * work with and we won't block on sending. So blocking mode is
fine.
+ */ + if (PQisnonblocking(conn)) + { + fprintf(stderr, "Expected blocking connection mode\n"); + goto fail; + }Perhaps worth adding a helper for this?
#define EXPECT(condition, explanation) \
...
or such?Greetings,
Andres Freund
The build is failing for this patch, can you please take a look at this?
https://cirrus-ci.com/task/4568547922804736
https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.129221
I am marking the patch "Waiting on Author"
--
Ibrar Ahmed
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
On 2021-Mar-04, Ibrar Ahmed wrote:
The build is failing for this patch, can you please take a look at this?
https://cirrus-ci.com/task/4568547922804736
https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.129221I am marking the patch "Waiting on Author"
I don't know why you chose to respond to such an old message, but here's
a rebase which also includes the workaround I suggested last night for
the problem of the outdated query printed by error messages, as well as
Justin's suggested fixes. (I also made a few tweaks to the TAP test).
--
�lvaro Herrera Valdivia, Chile
"No renuncies a nada. No te aferres a nada."
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
Apparently, the archives system or the commitfest system is not picking
up new messages to the thread, so the CF app is trying to apply a
very old patch version. I'm not sure what's up with that. Thomas, any
clues on where to look?
--
�lvaro Herrera Valdivia, Chile
"Oh, great altar of passive entertainment, bestow upon me thy discordant images
at such speed as to render linear thought impossible" (Calvin a la TV)
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
On Thu, Mar 4, 2021 at 8:01 PM Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:
Apparently, the archives system or the commitfest system is not picking
up new messages to the thread, so the CF app is trying to apply a
very old patch version. I'm not sure what's up with that. Thomas, any
clues on where to look?--
Álvaro Herrera Valdivia, Chile
"Oh, great altar of passive entertainment, bestow upon me thy discordant
images
at such speed as to render linear thought impossible" (Calvin a la TV)
Hi Alvaro,
The thread splits and CF app still has the previous thread. The old thread
has the v18 as the latest patch which is failing. The new thread which (
/messages/by-id/CAJkzx4T5E-2cQe3dtv2R78dYFvz+in8PY7A8MArvLhs_pg75gg@mail.gmail.com
)
has the new patch.
--
Ibrar Ahmed
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
On Thu, Mar 04, 2021 at 12:01:37PM -0300, Alvaro Herrera wrote:
Apparently, the archives system or the commitfest system is not picking
up new messages to the thread, so the CF app is trying to apply a
very old patch version. I'm not sure what's up with that. Thomas, any
clues on where to look?
I think it's just because you forgot the patch.
/messages/by-id/20210304142627.GA5978@alvherre.pgsql
--
Justin
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
I think it's just because you forgot the patch.
/messages/by-id/20210304142627.GA5978@alvherre.pgsql
--
�lvaro Herrera 39�49'30"S 73�17'W
Attachments:
v29-libpq-pipeline.patchtext/x-diff; charset=us-asciiDownload
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 0553279314..c16befa314 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -3173,6 +3173,33 @@ ExecStatusType PQresultStatus(const PGresult *res);
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-sync">
+ <term><literal>PGRES_PIPELINE_SYNC</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a
+ synchronization point in pipeline mode, requested by
+ <xref linkend="libpq-PQsendPipeline"/>.
+ This status occurs only when pipeline mode has been selected.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-aborted">
+ <term><literal>PGRES_PIPELINE_ABORTED</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a pipeline that has
+ received an error from the server. <function>PQgetResult</function>
+ must be called repeatedly, and each time it will return this status code
+ until the end of the current pipeline, at which point it will return
+ <literal>PGRES_PIPELINE_SYNC</literal> and normal processing can
+ resume.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
If the result status is <literal>PGRES_TUPLES_OK</literal> or
@@ -4919,6 +4946,498 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-pipeline-mode">
+ <title>Pipeline Mode</title>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>libpq</primary>
+ <secondary>pipeline mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>pipelining</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>batch mode</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> pipeline mode allows applications to
+ send a query without having to read the result of the previously
+ sent query. Taking advantage of the pipeline mode, a client will wait
+ less for the server, since multiple queries/results can be sent/
+ received in a single network transaction.
+ </para>
+
+ <para>
+ While pipeline mode provides a significant performance boost, writing
+ clients using the pipeline mode is more complex because it involves
+ managing a queue of pending queries and finding which result
+ corresponds to which query in the queue.
+ </para>
+
+ <para>
+ Pipeline mode also generally consumes more memory on both the client and server,
+ though careful and aggressive management of the send/receive queue can mitigate
+ this. This applies whether or not the connection is in blocking or non-blocking
+ mode.
+ </para>
+
+ <sect2 id="libpq-pipeline-using">
+ <title>Using Pipeline Mode</title>
+
+ <para>
+ To issue pipelines, the application must switch a connection into pipeline mode.
+ Enter pipeline mode with <xref linkend="libpq-PQenterPipelineMode"/>
+ or test whether pipeline mode is active with
+ <xref linkend="libpq-PQpipelineStatus"/>.
+ In pipeline mode, only <link linkend="libpq-async">asynchronous operations</link>
+ are permitted, and <literal>COPY</literal> is disallowed.
+ Using any synchronous command execution functions,
+ such as <function>PQfn</function>, or <function>PQexec</function>
+ and its sibling functions, is an error condition.
+ </para>
+
+ <note>
+ <para>
+ It is best to use pipeline mode with <application>libpq</application> in
+ <link linkend="libpq-PQsetnonblocking">non-blocking mode</link>. If used
+ in blocking mode it is possible for a client/server deadlock to occur.
+ <footnote>
+ <para>
+ The client will block trying to send queries to the server, but the
+ server will block trying to send results to the client from queries
+ it has already processed. This only occurs when the client sends
+ enough queries to fill both its output buffer and the server's receive
+ buffer before it switches to processing input from the server,
+ but it's hard to predict exactly when that will happen.
+ </para>
+ </footnote>
+ </para>
+ </note>
+
+ <sect3 id="libpq-pipeline-sending">
+ <title>Issuing Queries</title>
+
+ <para>
+ After entering pipeline mode, the application dispatches requests using
+ <xref linkend="libpq-PQsendQueryParams"/>,
+ or its prepared-query sibling
+ <xref linkend="libpq-PQsendQueryPrepared"/>.
+ These requests are queued on the client-side until flushed to the server;
+ this occurs when <xref linkend="libpq-PQsendPipeline"/> is used to
+ establish a synchronization point in the pipeline,
+ or when <xref linkend="libpq-PQflush"/> is called.
+ The functions <xref linkend="libpq-PQsendPrepare"/>,
+ <xref linkend="libpq-PQsendDescribePrepared"/>, and
+ <xref linkend="libpq-PQsendDescribePortal"/> also work in pipeline mode.
+ Result processing is described below.
+ </para>
+
+ <para>
+ The server executes statements, and returns results, in the order the
+ client sends them. The server will begin executing the commands in the
+ pipeline immediately, not waiting for the end of the pipeline.
+ If any statement encounters an error, the server aborts the current
+ transaction and skips processing commands in the pipeline until the
+ next synchronization point established by <function>PQsendPipeline</function>.
+ (This remains true even if the commands in the pipeline would rollback
+ the transaction.)
+ Query processing resumes after the synchronization point.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one; for example, one query may define a table that the next
+ query in the same pipeline uses. Similarly, an application may
+ create a named prepared statement and execute it with later
+ statements in the same pipeline.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-results">
+ <title>Processing Results</title>
+
+ <para>
+ To process the result of one query in a pipeline, the application calls
+ <function>PQgetResult</function> repeatedly and handles each result
+ until <function>PQgetResult</function> returns null.
+ The result from the next query in the pipeline may then be retrieved using
+ <function>PQgetResult</function> again and the cycle repeated.
+ The application handles individual statement results as normal.
+ When the results of all the queries in the pipeline have been
+ returned, <function>PQgetResult</function> returns a result
+ containing the status value <literal>PGRES_PIPELINE_SYNC</literal>.
+ </para>
+
+ <para>
+ The client may choose to defer result processing until the complete
+ pipeline has been sent, or interleave that with sending further
+ queries in the pipeline; see <xref linkend="libpq-pipeline-interleave" />.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ This mode selection is effective only for the query currently
+ being processed. For more information on the use of
+ <function>PQsetSingleRowMode</function>,
+ refer to <xref linkend="libpq-single-row-mode"/>.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal
+ asynchronous processing except that it may contain the new
+ <type>PGresult</type> types <literal>PGRES_PIPELINE_SYNC</literal>
+ and <literal>PGRES_PIPELINE_ABORTED</literal>.
+ <literal>PGRES_PIPELINE_SYNC</literal> is reported exactly once for each
+ <function>PQsendPipeline</function> after retrieving results for all
+ queries in the pipeline.
+ <literal>PGRES_PIPELINE_ABORTED</literal> is emitted in place of a normal
+ stream result for the first error and all subsequent results
+ except <literal>PGRES_PIPELINE_SYNC</literal> and null;
+ see <xref linkend="libpq-pipeline-errors"/>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing pipeline results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed (except that
+ <function>PQgetResult</function> returns null to indicate that we start
+ returning the results of next query). The application must keep track
+ of the order in which it sent queries, to associate them with their
+ corresponding results.
+ Applications will typically use a state machine or a FIFO queue for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-pipeline-errors">
+ <title>Error Handling</title>
+
+ <para>
+ When a query in a pipeline causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the pipeline
+ synchronization message. The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after <function>PQresultStatus</function>
+ returns <literal>PGRES_FATAL_ERROR</literal>,
+ the pipeline is flagged as aborted.
+ <function>PQresultStatus</function>, will report a
+ <literal>PGRES_PIPELINE_ABORTED</literal> result for each remaining queued
+ operation in an aborted pipeline. The result for
+ <function>PQsendPipeline</function> is reported as
+ <literal>PGRES_PIPELINE_SYNC</literal> to signal the end of the aborted pipeline
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the pipeline used an implicit transaction, then operations that have
+ already executed are rolled back and operations that were queued to follow
+ the failed operation are skipped entirely. The same behaviour holds if the
+ pipeline starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the pipeline. If a pipeline contains
+ <emphasis>multiple explicit transactions</emphasis>, all transactions that
+ committed prior to the error remain committed, the currently in-progress
+ transaction is aborted, and all subsequent operations are skipped completely,
+ including subsequent transactions.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal> — only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously, the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-interleave">
+ <title>Interleaving Result Processing and Query Dispatch</title>
+
+ <para>
+ To avoid deadlocks on large pipelines the client should be structured
+ around a non-blocking event loop using operating system facilities
+ such as <function>select</function>, <function>poll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work
+ remaining to be dispatched and a queue of work that has been dispatched
+ but not yet had its results processed. When the socket is writable
+ it should dispatch more work. When the socket is readable it should
+ read results and process them, matching them up to the next entry in
+ its expected results queue. Based on available memory, results from the
+ socket should be read frequently: there's no need to wait until the
+ pipeline end to read the results. Pipelines should be scoped to logical
+ units of work, usually (but not necessarily) one transaction per pipeline.
+ There's no need to exit pipeline mode and re-enter it between pipelines,
+ or to wait for one pipeline to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state
+ machine to track sent and received work is in
+ <filename>src/test/modules/test_libpq/pipeline.c</filename>
+ in the PostgreSQL source distribution.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-end">
+ <title>Ending Pipeline Mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed, and
+ the end pipeline result has been consumed, the application may return
+ to non-pipelined mode with <xref linkend="libpq-PQexitPipelineMode"/>.
+ </para>
+ </sect3>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-functions">
+ <title>Functions Associated with Pipeline Mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQpipelineStatus">
+ <term>
+ <function>PQpipelineStatus</function>
+ <indexterm>
+ <primary>PQpipelineStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the current pipeline mode status of the
+ <application>libpq</application> connection.
+<synopsis>
+PGpipelineStatus PQpipelineStatus(const PGconn *conn);
+</synopsis>
+ </para>
+
+ <para>
+ <function>PQpipelineStatus</function> can return one of the following values:
+
+ <variablelist>
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ON</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in
+ pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_OFF</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is
+ <emphasis>not</emphasis> in pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in pipeline
+ mode and an error occurred while processing the current pipeline.
+ The aborted flag is cleared when <function>PQresultStatus</function>
+ returns PGRES_PIPELINE_SYNC at the end of the pipeline.
+ Clients don't usually need this function to
+ verify aborted status, as they can tell that the pipeline is aborted
+ from the <literal>PGRES_PIPELINE_ABORTED</literal> result code.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterPipelineMode">
+ <term>
+ <function>PQenterPipelineMode</function>
+ <indexterm>
+ <primary>PQenterPipelineMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter pipeline mode if it is currently idle or
+ already in pipeline mode.
+
+<synopsis>
+int PQenterPipelineMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success.
+ Returns 0 and has no effect if the connection is not currently
+ idle, i.e., it has a result ready, or it is waiting for more
+ input from the server, etc.
+ This function does not actually send anything to the server,
+ it just changes the <application>libpq</application> connection
+ state.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitPipelineMode">
+ <term>
+ <function>PQexitPipelineMode</function>
+ <indexterm>
+ <primary>PQexitPipelineMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit pipeline mode if it is currently in pipeline mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitPipelineMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 1 and takes no action if not in
+ pipeline mode. If the current statement isn't finished processing
+ or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQsendPipeline">
+ <term>
+ <function>PQsendPipeline</function>
+ <indexterm>
+ <primary>PQsendPipeline</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Marks a synchronization point in a pipeline by sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ and flushing the send buffer. This serves as
+ the delimiter of an implicit transaction and an error recovery
+ point; see <xref linkend="libpq-pipeline-errors"/>.
+
+<synopsis>
+int PQsendPipeline(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 if the connection is not in
+ pipeline mode or sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ failed.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-tips">
+ <title>When to Use Pipeline Mode</title>
+
+ <para>
+ Much like asynchronous query mode, there is no meaningful performance
+ overhead when using pipeline mode. It increases client application complexity,
+ and extra caution is required to prevent client/server deadlocks, but
+ pipeline mode can offer considerable performance improvements, in exchange for
+ increased memory usage from leaving state around longer.
+ </para>
+
+ <para>
+ Pipeline mode is most useful when the server is distant, i.e., network latency
+ (<quote>ping time</quote>) is high, and also when many small operations
+ are being performed in rapid succession. There is usually less benefit
+ in using pipelined commands when each query takes many multiples of the client/server
+ round-trip time to execute. A 100-statement operation run on a server
+ 300ms round-trip-time away would take 30 seconds in network latency alone
+ without pipelining; with pipelining it may spend as little as 0.3s waiting for
+ results from the server.
+ </para>
+
+ <para>
+ Use pipelined commands when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed
+ into operations on sets, or into a <literal>COPY</literal> operation.
+ </para>
+
+ <para>
+ Pipeline mode is not useful when information from one operation is required by
+ the client to produce the next operation. In such cases, the client
+ would have to introduce a synchronization point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+BEGIN;
+SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+-- result: x=2
+-- client adds 1 to x:
+UPDATE mytable SET x = 3 WHERE id = 42;
+COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <para>
+ Pipelining is less useful, and more complex, when a single pipeline contains
+ multiple transactions (see <xref linkend="libpq-pipeline-errors"/>).
+ </para>
+
+ <note>
+ <para>
+ The pipeline API was introduced in <productname>PostgreSQL</productname> 14.
+ Pipeline mode is a client-side feature which doesn't require server
+ support, and works on any server that supports the v3 extended query
+ protocol.
+ </para>
+ </note>
+ </sect2>
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4959,6 +5478,13 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <para>
+ When using pipeline mode, single-row mode needs to be activated for each
+ query in the pipeline before retrieving results for that query
+ with <function>PQgetResult</function>.
+ See <xref linkend="libpq-pipeline-mode"/> for more information.
+ </para>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 6d46da42e2..012e44c736 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while a libpq connection is in pipeline mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 2ec0580a79..75448162ca 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -1109,6 +1109,12 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
row, the last value is kept.
</para>
+ <para>
+ <literal>\gset</literal> and <literal>\aset</literal> cannot be used
+ pipeline mode, since query results are not immediately
+ fetched in this mode.
+ </para>
+
<para>
The following example puts the final account balance from the first query
into variable <replaceable>abalance</replaceable>, and fills variables
@@ -1269,6 +1275,21 @@ SELECT 4 AS four \; SELECT 5 AS five \aset
</programlisting></para>
</listitem>
</varlistentry>
+
+ <varlistentry id='pgbench-metacommand-pipeline'>
+ <term><literal>\startpipeline</literal></term>
+ <term><literal>\endpipeline</literal></term>
+
+ <listitem>
+ <para>
+ These commands delimit the start and end of a pipeline of SQL statements.
+ In a pipeline, statements are sent to server without waiting for the results
+ of previous statements (see <xref linkend="libpq-pipeline-mode"/>).
+ Pipeline mode requires the extended query protocol.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect2>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 5272eed9ab..f74378110a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1019,6 +1019,12 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->err = _("empty query");
break;
+ case PGRES_PIPELINE_SYNC:
+ case PGRES_PIPELINE_ABORTED:
+ walres->status = WALRCV_ERROR;
+ walres->err = _("unexpected pipeline mode");
+ break;
+
case PGRES_NONFATAL_ERROR:
case PGRES_FATAL_ERROR:
case PGRES_BAD_RESPONSE:
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 31a4df45f5..fbbe270654 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -351,10 +351,11 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
- * and we enter the CSTATE_SLEEP state to wait for it to expire. Other
- * meta-commands are executed immediately. If the command about to start
- * is actually beyond the end of the script, advance to CSTATE_END_TX.
+ * CSTATE_WAIT_RESULT state unless in pipeline mode. On a \sleep
+ * meta-command, the timer is set, and we enter the CSTATE_SLEEP state to
+ * wait for it to expire. Other meta-commands are executed immediately. If
+ * the command about to start is actually beyond the end of the script,
+ * advance to CSTATE_END_TX.
*
* CSTATE_WAIT_RESULT waits until we get a result set back from the server
* for the current command.
@@ -484,7 +485,9 @@ typedef enum MetaCommand
META_IF, /* \if */
META_ELIF, /* \elif */
META_ELSE, /* \else */
- META_ENDIF /* \endif */
+ META_ENDIF, /* \endif */
+ META_STARTPIPELINE, /* \startpipeline */
+ META_ENDPIPELINE /* \endpipeline */
} MetaCommand;
typedef enum QueryMode
@@ -2504,6 +2507,10 @@ getMetaCommand(const char *cmd)
mc = META_GSET;
else if (pg_strcasecmp(cmd, "aset") == 0)
mc = META_ASET;
+ else if (pg_strcasecmp(cmd, "startpipeline") == 0)
+ mc = META_STARTPIPELINE;
+ else if (pg_strcasecmp(cmd, "endpipeline") == 0)
+ mc = META_ENDPIPELINE;
else
mc = META_NONE;
return mc;
@@ -2693,11 +2700,25 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In pipeline mode, we use asynchronous functions. If a
+ * server-side error occurs, it will be processed later
+ * among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ pg_log_error("%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2741,8 +2762,10 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
* varprefix should be set only with \gset or \aset, and SQL commands do
* not need it.
*/
+#if 0
Assert((meta == META_NONE && varprefix == NULL) ||
((meta == META_GSET || meta == META_ASET) && varprefix != NULL));
+#endif
res = PQgetResult(st->con);
@@ -2810,6 +2833,12 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* otherwise the result is simply thrown away by PQclear below */
break;
+ case PGRES_PIPELINE_SYNC:
+ pg_log_debug("client %d pipeline ending", st->id);
+ if (PQexitPipelineMode(st->con) != 1)
+ pg_log_error("client %d failed to exit pipeline mode", st->id);
+ break;
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3066,13 +3095,36 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
/* Execute the command */
if (command->type == SQL_COMMAND)
{
+ /* disallow \aset and \gset in pipeline mode */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ if (command->meta == META_GSET)
+ {
+ commandFailed(st, "gset", "\\gset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ else if (command->meta == META_ASET)
+ {
+ commandFailed(st, "aset", "\\aset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ }
+
if (!sendCommand(st, command))
{
commandFailed(st, "SQL", "SQL command send failed");
st->state = CSTATE_ABORTED;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in pipeline mode */
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -3212,7 +3264,15 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (readCommandResponse(st,
sql_script[st->use_file].commands[st->command]->meta,
sql_script[st->use_file].commands[st->command]->varprefix))
- st->state = CSTATE_END_COMMAND;
+ {
+ /*
+ * outside of pipeline mode: stop reading results.
+ * pipeline mode: continue reading results until an
+ * end-of-pipeline response.
+ */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ st->state = CSTATE_END_COMMAND;
+ }
else
st->state = CSTATE_ABORTED;
break;
@@ -3456,6 +3516,49 @@ executeMetaCommand(CState *st, instr_time *now)
return CSTATE_ABORTED;
}
}
+ else if (command->meta == META_STARTPIPELINE)
+ {
+ /*
+ * In pipeline mode, we use a workflow based on libpq pipeline
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "startpipeline", "cannot use pipeline mode with the simple query protocol");
+ return CSTATE_ABORTED;
+ }
+
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ commandFailed(st, "startpipeline", "already in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (PQenterPipelineMode(st->con) == 0)
+ {
+ commandFailed(st, "startpipeline", "failed to enter pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ }
+ else if (command->meta == META_ENDPIPELINE)
+ {
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ {
+ commandFailed(st, "endpipeline", "not in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (!PQsendPipeline(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to send the pipeline");
+ return CSTATE_ABORTED;
+ }
+ if (!PQexitPipelineMode(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to exit pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ /* collect pending results before getting out of pipeline mode */
+ return CSTATE_WAIT_RESULT;
+ }
/*
* executing the expression or shell command might have taken a
@@ -4683,7 +4786,9 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
- else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
+ else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF ||
+ my_command->meta == META_STARTPIPELINE ||
+ my_command->meta == META_ENDPIPELINE)
{
if (my_command->argc != 1)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
@@ -6808,4 +6913,5 @@ pthread_join(pthread_t th, void **thread_return)
return 0;
}
+
#endif /* WIN32 */
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..60d09e6d63 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,7 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterPipelineMode 180
+PQexitPipelineMode 181
+PQsendPipeline 182
+PQpipelineStatus 183
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index f83af03d0a..4c97e8a078 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -540,6 +540,23 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * pqFreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+pqFreeCommandQueue(PGcommandQueueEntry *queue)
+{
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *cur = queue;
+
+ queue = cur->next;
+ if (cur->query)
+ free(cur->query);
+ free(cur);
+ }
+}
/*
* pqDropServerData
@@ -571,6 +588,12 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ pqFreeCommandQueue(conn->cmd_queue_head);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ pqFreeCommandQueue(conn->cmd_queue_recycle);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
@@ -2486,6 +2509,7 @@ keep_going: /* We will come back to here until there is
/* Drop any PGresult we might have, too */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn);
/* Reset conn->status to put the state machine in the right state */
@@ -3928,6 +3952,7 @@ makeEmptyPGconn(void)
conn->status = CONNECTION_BAD;
conn->asyncStatus = PGASYNC_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
conn->xactStatus = PQTRANS_IDLE;
conn->options_valid = false;
conn->nonblocking = false;
@@ -4187,6 +4212,7 @@ closePGconn(PGconn *conn)
conn->status = CONNECTION_BAD; /* Well, not really _bad_ - just absent */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn); /* deallocate result */
resetPQExpBuffer(&conn->errorMessage);
release_conn_addrinfo(conn);
@@ -6735,6 +6761,15 @@ PQbackendPID(const PGconn *conn)
return conn->be_pid;
}
+PGpipelineStatus
+PQpipelineStatus(const PGconn *conn)
+{
+ if (!conn)
+ return PQ_PIPELINE_OFF;
+
+ return conn->pipelineStatus;
+}
+
int
PQconnectionNeedsPassword(const PGconn *conn)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9a038043b2..0fb790fdfa 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_PIPELINE_SYNC",
+ "PGRES_PIPELINE_ABORTED"
};
/*
@@ -71,6 +73,11 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *pqMakePipelineCmd(PGconn *conn);
+static void pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqPipelineProcessQueue(PGconn *conn);
+static int pqPipelineFlush(PGconn *conn);
/* ----------------
@@ -1171,7 +1178,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1197,18 +1204,37 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot use %s in pipeline mode, use PQsendQueryParams\n"),
+ "PQsendQuery");
+ return 0;
+ }
+
return PQsendQueryInternal(conn, query, true);
}
int
PQsendQueryContinue(PGconn *conn, const char *query)
{
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot use %s in pipeline mode, use PQsendQueryParams\n"),
+ "PQsendQueryContinue");
+ return 0;
+ }
+
return PQsendQueryInternal(conn, query, false);
}
static int
PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
{
+ /* pipeline mode requires extended query protocol */
+ Assert(conn->pipelineStatus == PQ_PIPELINE_OFF);
+
if (!PQsendQueryStart(conn, newQuery))
return 0;
@@ -1307,6 +1333,8 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+
if (!PQsendQueryStart(conn, true))
return 0;
@@ -1330,6 +1358,15 @@ PQsendPrepare(PGconn *conn,
return 0;
}
+ /* Alloc pipeline memory before doing anything */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+ }
+
/* construct the Parse message */
if (pqPutMsgStart('P', conn) < 0 ||
pqPuts(stmtName, conn) < 0 ||
@@ -1356,32 +1393,46 @@ PQsendPrepare(PGconn *conn,
if (pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /*
+ * In non-pipeline mode, add a Sync and prepare to send. In pipeline mode
+ * we just keep track of the new message.
+ */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ /* remember we are doing just a Parse */
+ conn->queryclass = PGQUERY_PREPARE;
- /* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
-
- /* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ /* and remember the query text too, if possible */
+ /* if insufficient memory, last_query just winds up NULL */
+ if (conn->last_query)
+ free(conn->last_query);
+ conn->last_query = strdup(query);
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ else
+ {
+ pipeCmd->queryclass = PGQUERY_PREPARE;
+ /* as above, if insufficient memory, query winds up NULL */
+ pipeCmd->query = strdup(query);
+ pqAppendPipelineCmd(conn, pipeCmd);
+ }
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
- /* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1429,7 +1480,8 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn, bool newQuery)
@@ -1450,20 +1502,57 @@ PQsendQueryStart(PGconn *conn, bool newQuery)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE &&
+ conn->pipelineStatus == PQ_PIPELINE_OFF)
{
appendPQExpBufferStr(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * When enqueuing commands we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when pqPipelineProcessQueue()
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ }
+ }
+ else
+ {
+ /*
+ * This command's results will come in immediately. Initialize async
+ * result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1487,10 +1576,34 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **query;
+ PGQueryClass *queryclass;
/*
- * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
- * using specified statement name and the unnamed portal.
+ * Decide where the query is going to be stored. In pipeline mode, we
+ * allocate a new pipeline element; in non-pipeline mode, it's simply the
+ * connection's last query.
+ */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+ /*
+ * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync
+ * (if not in pipeline mode), using specified statement name and the
+ * unnamed portal.
*/
if (command)
@@ -1600,35 +1713,43 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /* construct the Sync message if not in pipeline mode */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ /* if insufficient memory, query just winds up NULL */
+ if (*query)
+ free(*query);
if (command)
- conn->last_query = strdup(command);
+ *query = strdup(command);
else
- conn->last_query = NULL;
+ *query = NULL;
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1726,14 +1847,17 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
-
/*
* PQgetResult
* Get the next PGresult produced by a query. Returns NULL if no
* query work remains or an error has occurred (e.g. out of
* memory).
+ *
+ * In pipeline mode, once all the result of a query have been returned,
+ * PQgetResult returns NULL to let the user know that the next
+ * query is being processed. At the end of the pipeline, returns a
+ * result with PQresultStatus(result) == PGRES_PIPELINE_SYNC.
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1803,8 +1927,49 @@ PQgetResult(PGconn *conn)
{
case PGASYNC_IDLE:
res = NULL; /* query is complete */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to return the NULL that terminates the round of
+ * results from the current query; prepare to send the results
+ * of the next query when we're called next. Also, since this
+ * is the start of the results of the next query, clear any
+ * prior error message.
+ */
+ resetPQExpBuffer(&conn->errorMessage);
+ pqPipelineProcessQueue(conn);
+ }
break;
case PGASYNC_READY:
+ res = pqPrepareAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to send the results of the current query. Set
+ * us idle now, and ...
+ */
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /*
+ * ... in cases when we're sending a pipeline-related result,
+ * move queue processing forwards immediately, so that next
+ * time we're called, we're prepared to return the next result
+ * received from the server.
+ *
+ * In all other cases, leave the queue state change for next
+ * time, so that a terminating NULL result is sent.
+ */
+ if (res && (res->resultStatus == PGRES_PIPELINE_ABORTED ||
+ res->resultStatus == PGRES_PIPELINE_SYNC))
+ pqPipelineProcessQueue(conn);
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -1985,6 +2150,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("synchronous command execution functions are not allowed in pipeline mode\n"));
+ return false;
+ }
+
/*
* Since this is the beginning of a query cycle, reset the error buffer.
*/
@@ -2148,6 +2320,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2155,6 +2330,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
if (!PQsendQueryStart(conn, true))
return 0;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ queryclass = &conn->queryclass;
+
/* construct the Describe message */
if (pqPutMsgStart('D', conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2163,32 +2350,40 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ /* reset last-query string (not relevant now) */
+ if (conn->last_query && conn->pipelineStatus != PQ_PIPELINE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
}
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -2541,6 +2736,13 @@ PQfn(PGconn *conn,
*/
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("PQfn not allowed in pipeline mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -2555,6 +2757,359 @@ PQfn(PGconn *conn,
args, nargs);
}
+/* ====== Pipeline mode support ======== */
+
+/*
+ * PQenterPipelineMode
+ * Put an idle connection in pipeline mode.
+ *
+ * Returns 1 on success. On failure, errorMessage is set and 0 is returned.
+ *
+ * Commands submitted after this can be pipelined on the connection;
+ * there's no requirement to wait for one to finish before the next is
+ * dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQsendPipeline. Multiple pipelines
+ * can be sent while in pipeline mode. Pipeline mode can be exited
+ * by calling PQexitPipelineMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ /* succeed with no action if already in pipeline mode */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot enter pipeline mode, connection not idle\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+
+ return 1;
+}
+
+/*
+ * PQexitPipelineMode
+ * End pipeline mode and return to normal command mode.
+ *
+ * Returns 1 in success (pipeline mode successfully ended, or not in pipeline
+ * mode).
+ *
+ * Returns 0 if in pipeline mode and cannot be ended yet. Error message will
+ * be set.
+ */
+int
+PQexitPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ /* there are some uncollected results */
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+
+ case PGASYNC_BUSY:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode while busy\n"));
+ return 0;
+
+ default:
+ /* OK */
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ return 0; /* error message is setup already */
+ return 1;
+}
+
+/*
+ * pqPipelineProcessQueue: subroutine for PQgetResult
+ * In pipeline mode, start processing the results of the next query in the queue.
+ */
+static void
+pqPipelineProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ Assert(conn->pipelineStatus != PQ_PIPELINE_OFF);
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: COPY in pipeline mode\n");
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return;
+ case PGASYNC_IDLE:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In pipeline mode but nothing left on the queue; caller can submit
+ * more work or PQexitPipelineMode() now.
+ */
+ return;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-pipeline call.
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /*
+ * Reset single-row processing mode. (Client has to set it up for each
+ * query, if desired.)
+ */
+ conn->singleRowMode = false;
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ pqRecyclePipelineCmd(conn, next_query);
+
+ if (conn->pipelineStatus == PQ_PIPELINE_ABORTED &&
+ conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted pipeline we don't get anything from the server for
+ * each result; we're just discarding input until we get to the next
+ * sync from the server. The client needs to know its queries got
+ * aborted so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn, PGRES_PIPELINE_ABORTED);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ pqSaveErrorResult(conn);
+ return;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+}
+
+/*
+ * PQsendPipeline
+ * Send a Sync message as part of a pipeline, and flush to server
+ *
+ * It's legal to start submitting more commands in the pipeline immediately,
+ * without waiting for the results of the current pipeline. There's no need to
+ * end pipeline mode and start it again.
+ *
+ * If a command in a pipeline fails, every subsequent command up to and including
+ * the result to the Sync message sent by PQsendPipeline gets set to
+ * PGRES_PIPELINE_ABORTED state. If the whole pipeline is processed without
+ * error, a PGresult with PGRES_PIPELINE_SYNC is produced.
+ *
+ * Queries can already have been sent before PQsendPipeline is called, but
+ * PQsendPipeline need to be called before retrieving command results.
+ *
+ * The connection will remain in pipeline mode and unavailable for new
+ * synchronous command execution functions until all results from the pipeline
+ * are processed by the client.
+ */
+int
+PQsendPipeline(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot send pipeline when not in pipeline mode\n"));
+ return 0;
+ }
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: cannot send pipeline while in COPY\n");
+ return 0;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_IDLE:
+ /* OK to send sync */
+ break;
+ }
+
+ entry = pqMakePipelineCmd(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ pqAppendPipelineCmd(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ /*
+ * Call pqPipelineProcessQueue so the user can call start calling
+ * PQgetResult.
+ */
+ pqPipelineProcessQueue(conn);
+
+ return 1;
+
+sendFailed:
+ pqRecyclePipelineCmd(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
+
+/*
+ * pqMakePipelineCmd
+ * Get a command queue entry for caller to fill.
+ *
+ * If the recycle queue has a free element, that is returned; if not, a
+ * fresh one is allocated. Caller is responsible for adding it to the
+ * command queue (pqAppendPipelineCmd) once the struct is filled in, or
+ * releasing the memory (pqRecyclePipelineCmd) if an error occurs.
+ *
+ * If allocation fails, sets the error message and returns NULL.
+ */
+static PGcommandQueueEntry *
+pqMakePipelineCmd(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * pqAppendPipelineCmd
+ * Append a caller-allocated command queue entry to the queue.
+ *
+ * The query itself must already have been put in the output buffer by the
+ * caller.
+ */
+static void
+pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ PGcommandQueueEntry **tail;
+
+ if (conn->cmd_queue_head == NULL)
+ tail = &conn->cmd_queue_head;
+ else
+ tail = &conn->cmd_queue_tail->next;
+
+ *tail = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * pqRecyclePipelineCmd
+ * Push a command queue entry onto the freelist. It must be an entry
+ * with null next pointer and not referenced by any other entry's next
+ * pointer.
+ */
+static void
+pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+
+ Assert(entry->next == NULL);
+
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
/* ====== accessor funcs for PGresult ======== */
@@ -3152,6 +3707,23 @@ PQflush(PGconn *conn)
return pqFlush(conn);
}
+/*
+ * pqPipelineFlush
+ *
+ * In pipeline mode, data will be flushed only when the out buffer reaches the
+ * threshold value. In non-pipeline mode, it behaves as stock pqFlush.
+ *
+ * Returns 0 on success.
+ */
+static int
+pqPipelineFlush(PGconn *conn)
+{
+ if ((conn->pipelineStatus == PQ_PIPELINE_OFF) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return pqFlush(conn);
+ return 0;
+}
+
/*
* PQfreemem - safely frees memory allocated
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 2ca8c057b9..b131995974 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -177,14 +177,24 @@ pqParseInput3(PGconn *conn)
if (getParameterStatus(conn))
return;
}
- else
- {
- pqInternalNotice(&conn->noticeHooks,
- "message type 0x%02x arrived from server while idle",
- id);
- /* Discard the unexpected message */
- conn->inCursor += msgLength;
- }
+
+ /*
+ * We're also IDLE when in pipeline mode we have completed
+ * processing the results of one query and are waiting for the
+ * next one in the pipeline. In this case, as above, just wait to
+ * see what's next.
+ */
+ if (conn->asyncStatus == PGASYNC_IDLE &&
+ conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->cmd_queue_head != NULL)
+ return;
+
+ /* Any other case is unexpected and we summarily skip it */
+ pqInternalNotice(&conn->noticeHooks,
+ "message type 0x%02x arrived from server while idle",
+ id);
+ /* Discard the unexpected message */
+ conn->inCursor += msgLength;
}
else
{
@@ -217,10 +227,28 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new
+ * query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_PIPELINE_SYNC);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ else
+ {
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -450,7 +478,7 @@ handleSyncLoss(PGconn *conn, char id, int msgLength)
id, msgLength);
/* build an error result holding the error message */
pqSaveErrorResult(conn);
- conn->asyncStatus = PGASYNC_READY; /* drop out of GetResult wait loop */
+ conn->asyncStatus = PGASYNC_READY; /* drop out of PQgetResult wait loop */
/* flush input data since we're giving up on processing it */
pqDropConnection(conn, true);
conn->status = CONNECTION_BAD; /* No more connection to backend */
@@ -875,6 +903,10 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ /* If in pipeline mode, set error indicator for it */
+ if (isError && conn->pipelineStatus != PQ_PIPELINE_OFF)
+ conn->pipelineStatus = PQ_PIPELINE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
@@ -930,9 +962,21 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
* Save the active query text, if any, into res as well; but only if we
* might need it for an error cursor display, which is only true if there
* is a PG_DIAG_STATEMENT_POSITION field.
+ *
+ * Note that in pipeline mode, we have not yet advanced the query pointer
+ * to the next query, so we have to look at that.
*/
- if (have_position && conn->last_query && res)
- res->errQuery = pqResultStrdup(res, conn->last_query);
+ if (have_position && res)
+ {
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->asyncStatus == PGASYNC_IDLE)
+ {
+ if (conn->cmd_queue_head)
+ res->errQuery = pqResultStrdup(res, conn->cmd_queue_head->query);
+ }
+ else if (conn->last_query)
+ res->errQuery = pqResultStrdup(res, conn->last_query);
+ }
/*
* Now build the "overall" error message for PQresultErrorMessage.
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index fa9b62a844..fb31a49fab 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -96,7 +96,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_PIPELINE_SYNC, /* pipeline synchronization point */
+ PGRES_PIPELINE_ABORTED, /* Command didn't run because of an abort
+ * earlier in a pipeline */
} ExecStatusType;
typedef enum
@@ -136,6 +139,16 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PGpipelineStatus - Current status of pipeline mode
+ */
+typedef enum
+{
+ PQ_PIPELINE_OFF,
+ PQ_PIPELINE_ON,
+ PQ_PIPELINE_ABORTED
+} PGpipelineStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -327,6 +340,7 @@ extern int PQserverVersion(const PGconn *conn);
extern char *PQerrorMessage(const PGconn *conn);
extern int PQsocket(const PGconn *conn);
extern int PQbackendPID(const PGconn *conn);
+extern PGpipelineStatus PQpipelineStatus(const PGconn *conn);
extern int PQconnectionNeedsPassword(const PGconn *conn);
extern int PQconnectionUsedPassword(const PGconn *conn);
extern int PQclientEncoding(const PGconn *conn);
@@ -434,6 +448,11 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for pipeline mode management */
+extern int PQenterPipelineMode(PGconn *conn);
+extern int PQexitPipelineMode(PGconn *conn);
+extern int PQsendPipeline(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 8d51e6ed9f..3e9f99f8fb 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,7 +217,11 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, more results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
@@ -229,7 +233,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* Sync at end of a pipeline */
} PGQueryClass;
/* Target server type (decoded value of target_session_attrs) */
@@ -305,6 +310,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by pipeline mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct PGcommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct PGcommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -397,6 +418,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PGpipelineStatus pipelineStatus; /* status of pipeline mode */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -409,6 +431,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue, for pipeline mode.
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -790,6 +822,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 5391f461a2..6e31458be2 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -17,6 +17,7 @@ SUBDIRS = \
test_extensions \
test_ginpostinglist \
test_integerset \
+ test_libpq \
test_misc \
test_parser \
test_pg_dump \
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000000..4fbf97a5b0
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/pipeline
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000000..64a3a4af5e
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,20 @@
+# src/test/modules/test_libpq/Makefile
+
+PROGRAM = pipeline
+OBJS = pipeline.o
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+TAP_TESTS = 1
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/pipeline.c b/src/test/modules/test_libpq/pipeline.c
new file mode 100644
index 0000000000..bb217e6b08
--- /dev/null
+++ b/src/test/modules/test_libpq/pipeline.c
@@ -0,0 +1,1141 @@
+/*
+ * src/test/modules/test_libpq/pipeline.c
+ * Verify libpq pipeline execution functionality
+ */
+#include "postgres_fe.h"
+
+#include <sys/time.h>
+
+#include "catalog/pg_type_d.h"
+#include "common/fe_memutils.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+
+static void exit_nicely(PGconn *conn);
+
+const char *const progname = "pipeline";
+
+
+#define DEBUG
+#ifdef DEBUG
+#define pg_debug(...) do { fprintf(stderr, __VA_ARGS__); } while (0)
+#else
+#define pg_debug(...)
+#endif
+
+static const char *const drop_table_sql =
+"DROP TABLE IF EXISTS pq_pipeline_demo";
+static const char *const create_table_sql =
+"CREATE UNLOGGED TABLE pq_pipeline_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql =
+"INSERT INTO pq_pipeline_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+/*
+ * Print an error to stderr and terminate the program.
+ */
+#define pg_fatal(...) pg_fatal_impl(__LINE__, __VA_ARGS__)
+static void
+pg_fatal_impl(int line, const char *fmt,...)
+{
+ va_list args;
+
+ fprintf(stderr, "\n"); /* XXX hack */
+ fprintf(stderr, "%s:%d: ", progname, line);
+
+ va_start(args, fmt);
+ vfprintf(stderr, fmt, args);
+ va_end(args);
+ printf("Failure, exiting\n");
+ exit(1);
+}
+
+static void
+test_disallowed(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("Unable to enter pipeline mode\n");
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not activated properly\n");
+
+ /* PQexec should fail in pipeline mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("PQexec should fail in pipeline mode but succeeded\n");
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ pg_fatal("PQsendQuery should fail in pipeline mode but succeeded\n");
+
+ /* Entering pipeline mode when already in pipeline mode is OK */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("re-entering pipeline mode should be a no-op but failed\n");
+
+ if (PQisBusy(conn) != 0)
+ pg_fatal("PQisBusy should return 0 when idle in pipeline mode, returned 1\n");
+
+ /* ok, back to normal command mode */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("couldn't exit idle empty pipeline mode\n");
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not terminated properly\n");
+
+ /* exiting pipeline mode when not in pipeline mode should be a no-op */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("pipeline mode exit when not in pipeline mode should succeed but failed\n");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("PQexec should succeed after exiting pipeline mode but failed with: %s\n",
+ PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_simple_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple pipeline... ");
+
+ /*
+ * Enter pipeline mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our output buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode with work in progress should fail, but succeeded\n");
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending pipeline failed: %s\n", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first query result.\n");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit pipeline mode yet
+ */
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result PGRES_PIPELINE_SYNC expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of PGRES_PIPELINE_SYNC, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after pipeline end: %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* ... until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_multi_pipelines(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi pipeline... ");
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending first pipeline failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s\n", PQerrorMessage(conn));
+
+ /* OK, start processing the results */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first result\n");
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ PQclear(res);
+
+#if 0
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+#endif
+
+ /* second pipeline */
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from second pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode ... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+/*
+ * When an operation in a pipeline fails the rest of the pipeline is flushed. We
+ * still have to get results for each pipeline item, but the item will just be
+ * a PGRES_PIPELINE_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the pipeline. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_aborted_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted pipeline... ");
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * pipeline ERRORs.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT no_such_function($1)",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching error select failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending first pipeline failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second-pipeline insert failed: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the pipeline results.
+ *
+ * We should get a command-ok for the first query, then a fatal error and
+ * a pipeline aborted message for the second insert, a pipeline-end, then
+ * a command-ok and a pipeline-ok for the second pipeline operation.
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result status %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* Second query caused error, so we expect an error next */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("Unexpected result code -- expected PGRES_FATAL_ERROR, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /*
+ * pipeline should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third pipeline since we've already sent a
+ * second. The aborted flag relates only to the pipeline being received.
+ */
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't\n");
+
+ /* third query in pipeline, the second insert */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_ABORTED)
+ pg_fatal("Unexpected result code -- expected PGRES_PIPELINE_ABORTED, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+#if 0
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+#endif
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't\n");
+
+ /* Ensure we're still in pipeline */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /*
+ * The end of a failed pipeline is a PGRES_PIPELINE_SYNC.
+ *
+ * (This is so clients know to start processing results normally again and
+ * can tell the difference between skipped commands and the sync.)
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code from first pipeline sync\n"
+ "Expected PGRES_PIPELINE_SYNC, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+#if 0
+ /* after the synchronization point we get a NULL result */
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+#endif
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_ABORTED)
+ pg_fatal("sync should've cleared the aborted flag but didn't\n");
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* the insert from the second pipeline */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result code %s from first item in second pipeline\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* Read the NULL result at the end of the command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* the second pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQerrorMessage(conn));
+
+ /* Test single-row mode with an error partways */
+ if (PQsendQueryParams(conn, "SELECT 1/g FROM generate_series(5, -1, -1) g",
+ 0, NULL, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to send query: %s", PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ PQsetSingleRowMode(conn);
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ if (PQresultStatus(res) == PGRES_FATAL_ERROR)
+ printf("got error: {{%s}} (expected: division by zero)\n", PQerrorMessage(conn));
+ else if (PQresultStatus(res) == PGRES_SINGLE_TUPLE)
+ printf("got row: %s\n", PQgetvalue(res, 0, 0));
+ else
+ pg_fatal("got unexpected result %s\n", PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ }
+ /* the third pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from third pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ /*-
+ * Since we fired the pipelines off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st pipeline
+ * - First insert applied
+ * - Second statement aborted xact
+ * - Third insert skipped
+ * - Sync rolled back first implicit xact
+ * - Implicit xact created by server around 2nd pipeline
+ * - insert applied from 2nd pipeline
+ * - Sync commits 2nd xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM pq_pipeline_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Expected tuples, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("expected 1 result, got %d\n", PQntuples(res));
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ pg_fatal("expected only insert with value 3, got %s", val);
+ }
+
+ PQclear(res);
+}
+
+/* State machine enum for test_pipelined_insert */
+typedef enum PipelineInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} PipelineInsertStep;
+
+static void
+test_pipelined_insert(PGconn *conn, int n_rows)
+{
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ PipelineInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a pipelined insert into a table created at the start of the pipeline
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ while (send_step != BI_PREPARE)
+ {
+ const char *sql;
+
+ switch (send_step)
+ {
+ case BI_BEGIN_TX:
+ sql = "BEGIN TRANSACTION";
+ send_step = BI_DROP_TABLE;
+ break;
+
+ case BI_DROP_TABLE:
+ sql = drop_table_sql;
+ send_step = BI_CREATE_TABLE;
+ break;
+
+ case BI_CREATE_TABLE:
+ sql = create_table_sql;
+ send_step = BI_PREPARE;
+ break;
+
+ default:
+ pg_fatal("invalid state");
+ }
+
+ pg_debug("sending: %s\n", sql);
+ if (PQsendQueryParams(conn, sql,
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching %s failed: %s\n", sql, PQerrorMessage(conn));
+ }
+
+ Assert(send_step == BI_PREPARE);
+ pg_debug("sending: %s\n", insert_sql);
+ if (PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids) != 1)
+ pg_fatal("dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our output buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the pipeline before processing
+ * results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ pg_fatal("failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's output buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ PGresult *res;
+ const char *cmdtag;
+ const char *description = "";
+ int status;
+
+ /*
+ * Read next result. If no more results from this query,
+ * advance to the next query
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ continue;
+
+ status = PGRES_COMMAND_OK;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ recv_step++;
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ recv_step++;
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ recv_step++;
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ recv_step++;
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive == 0)
+ recv_step++;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ recv_step++;
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_PIPELINE_SYNC;
+ recv_step++;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ description = "";
+ abort();
+ }
+
+ if (PQresultStatus(res) != status)
+ pg_fatal("%s reported status %s, expected %s\n"
+ "Error message: \"%s\"\n",
+ description, PQresStatus(PQresultStatus(res)),
+ PQresStatus(status), PQerrorMessage(conn));
+
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ pg_fatal("%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+
+ pg_debug("Got %s OK\n", cmdtag[0] != '\0' ? cmdtag : description);
+
+ PQclear(res);
+ }
+ }
+
+ /* Write more rows and/or the end pipeline message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQsendPipeline(conn) == 1)
+ {
+ fprintf(stdout, "Sent pipeline\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending pipeline failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+ }
+
+ /* We've got the sync message and the pipeline should be done */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ pg_fatal("failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i;
+ bool pipeline_ended = false;
+
+ /* 1 pipeline, 3 queries in it */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n",
+ PQerrorMessage(conn));
+
+ for (i = 0; i < 3; i++)
+ {
+ char *param[1];
+
+ param[0] = psprintf("%d", 44 + i);
+
+ if (PQsendQueryParams(conn,
+ "SELECT generate_series(42, $1)",
+ 1,
+ NULL,
+ (const char **) param,
+ NULL,
+ NULL,
+ 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ pfree(param[0]);
+ }
+ PQsendPipeline(conn);
+
+ for (i = 0; !pipeline_ended; i++)
+ {
+ bool first = true;
+ bool saw_ending_tuplesok;
+ bool isSingleTuple = false;
+
+ /* Set single row mode for only first 2 SELECT queries */
+ if (i < 2)
+ {
+ if (PQsetSingleRowMode(conn) != 1)
+ pg_fatal("PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+
+ /* Consume rows for this query */
+ saw_ending_tuplesok = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+
+ if (est == PGRES_PIPELINE_SYNC)
+ {
+ fprintf(stderr, "end of pipeline reached\n");
+ pipeline_ended = true;
+ PQclear(res);
+ if (i != 3)
+ pg_fatal("Expected three results, got %d\n", i);
+ break;
+ }
+
+ /* Expect SINGLE_TUPLE for queries 0 and 1, TUPLES_OK for 2 */
+ if (first)
+ {
+ if (i <= 1 && est != PGRES_SINGLE_TUPLE)
+ pg_fatal("Expected PGRES_SINGLE_TUPLE for query %d, got %s\n",
+ i, PQresStatus(est));
+ if (i >= 2 && est != PGRES_TUPLES_OK)
+ pg_fatal("Expected PGRES_TUPLES_OK for query %d, got %s\n",
+ i, PQresStatus(est));
+ first = false;
+ }
+
+ fprintf(stderr, "Result status %s for query %d", PQresStatus(est), i);
+ switch (est)
+ {
+ case PGRES_TUPLES_OK:
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ saw_ending_tuplesok = true;
+ if (isSingleTuple)
+ {
+ if (PQntuples(res) == 0)
+ fprintf(stderr, "all tuples received in query %d\n", i);
+ else
+ pg_fatal("Expected to follow PGRES_SINGLE_TUPLE, "
+ "but received PGRES_TUPLES_OK directly instead\n");
+ }
+ break;
+
+ case PGRES_SINGLE_TUPLE:
+ isSingleTuple = true;
+ fprintf(stderr, ", %d tuple: %s\n", PQntuples(res), PQgetvalue(res, 0, 0));
+ break;
+
+ default:
+ pg_fatal("unexpected\n");
+ }
+ PQclear(res);
+ }
+ if (!pipeline_ended && !saw_ending_tuplesok)
+ pg_fatal("didn't get expected terminating TUPLES_OK\n");
+ }
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s\n", PQerrorMessage(conn));
+}
+
+/*
+ * Simple test to verify that a pipeline is discarded as a whole when there's
+ * an error, ignoring transaction commands.
+ */
+static void
+test_transaction(PGconn *conn)
+{
+ PGresult *res;
+ int num_sends = 0;
+
+ res = PQexec(conn, "DROP TABLE IF EXISTS pq_pipeline_tst;"
+ "CREATE TABLE pq_pipeline_tst (id int)");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to create test table: %s",
+ PQerrorMessage(conn));
+ PQclear(res);
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n",
+ PQerrorMessage(conn));
+ if (PQsendPrepare(conn, "rollback", "ROLLBACK", 0, NULL) != 1)
+ pg_fatal("could not send prepare on pipeline: %s",
+ PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn,
+ "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ if (PQsendQueryParams(conn,
+ "SELECT 0/0",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+
+ /*
+ * send a ROLLBACK using a prepared stmt. Doesn't work because we need to
+ * get out of the pipeline-aborted state first.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /* This insert fails because we're in pipeline-aborted state */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (1)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * This insert fails even though the pipeline got a SYNC, because we're in
+ * an aborted transaction
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (2)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * Send ROLLBACK using prepared stmt. This one works because we just did
+ * PQsendPipeline above.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /*
+ * Now that we're out of a transaction and in pipeline-good mode, this
+ * insert works
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (3)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+ PQsendPipeline(conn);
+ num_sends++;
+
+ for (int i = 0;; i++)
+ {
+ ExecStatusType restype;
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "%d: got NULL result\n", i);
+ continue;
+ }
+ restype = PQresultStatus(res);
+ fprintf(stderr, "%d: got status %s", i, PQresStatus(restype));
+ if (restype == PGRES_FATAL_ERROR)
+ fprintf(stderr, "; error: %s", PQerrorMessage(conn));
+ else if (restype == PGRES_PIPELINE_ABORTED)
+ {
+ fprintf(stderr, ": command didn't run because pipeline aborted\n");
+ }
+ else
+ fprintf(stderr, "\n");
+ PQclear(res);
+
+ if (restype == PGRES_PIPELINE_SYNC)
+ num_sends--;
+ if (num_sends <= 0)
+ break;
+ }
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("returned something extra after all the syncs: %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s\n", PQerrorMessage(conn));
+
+ /* We expect to find one tuple containing the value "3" */
+ res = PQexec(conn, "SELECT * FROM pq_pipeline_tst");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("failed to obtain result: %s", PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("did not get 1 tuple\n");
+ if (strcmp(PQgetvalue(res, 0, 0), "3") != 0)
+ pg_fatal("did not get expected tuple\n");
+ PQclear(res);
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+usage(const char *progname)
+{
+ fprintf(stderr, "%s tests libpq's pipeline mode.\n\n", progname);
+ fprintf(stderr, "Usage:\n");
+ fprintf(stderr, " %s testname [conninfo [number_of_rows]]\n", progname);
+ fprintf(stderr, "Tests:\n");
+ fprintf(stderr, " disallowed_in_pipeline\n");
+ fprintf(stderr, " simple_pipeline\n");
+ fprintf(stderr, " multi_pipeline\n");
+ fprintf(stderr, " pipeline_abort\n");
+ fprintf(stderr, " singlerow\n");
+ fprintf(stderr, " pipeline_insert\n");
+ fprintf(stderr, " transaction\n");
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo = "";
+ PGconn *conn;
+ int numrows = 10000;
+ PGresult *res;
+
+ /*
+ * The testname parameter is mandatory; it can be followed by a conninfo
+ * string and number of rows.
+ */
+ if (argc < 2 || argc > 4)
+ {
+ usage(argv[0]);
+ exit(1);
+ }
+
+ if (argc >= 3)
+ conninfo = pg_strdup(argv[2]);
+
+ if (argc >= 4)
+ {
+ errno = 0;
+ numrows = strtol(argv[3], NULL, 10);
+ if (errno != 0 || numrows <= 0)
+ {
+ fprintf(stderr, "couldn't parse \"%s\" as a positive integer\n", argv[3]);
+ exit(1);
+ }
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+ res = PQexec(conn, "SET lc_messages TO \"C\"");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to set lc_messages: %s", PQerrorMessage(conn));
+
+ if (strcmp(argv[1], "disallowed_in_pipeline") == 0)
+ test_disallowed(conn);
+ else if (strcmp(argv[1], "simple_pipeline") == 0)
+ test_simple_pipeline(conn);
+ else if (strcmp(argv[1], "multi_pipeline") == 0)
+ test_multi_pipelines(conn);
+ else if (strcmp(argv[1], "pipeline_abort") == 0)
+ test_aborted_pipeline(conn);
+ else if (strcmp(argv[1], "pipeline_insert") == 0)
+ test_pipelined_insert(conn, numrows);
+ else if (strcmp(argv[1], "singlerow") == 0)
+ test_singlerowmode(conn);
+ else if (strcmp(argv[1], "transaction") == 0)
+ test_transaction(conn);
+ else
+ {
+ fprintf(stderr, "\"%s\" is not a recognized test name\n", argv[1]);
+ usage(argv[0]);
+ exit(1);
+ }
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+ return 0;
+}
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000000..c7d4b1feec
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,31 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 7;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $numrows = 10000;
+my @tests =
+ qw(disallowed_in_pipeline
+ simple_pipeline
+ multi_pipeline
+ pipeline_abort
+ pipeline_insert
+ singlerow
+ transaction);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'pipeline', $testname, $node->connstr('postgres'), $numrows ],
+ "pipeline $testname");
+}
+
+$node->stop('fast');
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 8bd95aefa1..1bd23eec6d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1561,10 +1561,12 @@ PG_Locale_Strategy
PG_Lock_Status
PG_init_t
PGcancel
+PGcommandQueueEntry
PGconn
PGdataValue
PGlobjfuncs
PGnotify
+PGpipelineStatus
PGresAttDesc
PGresAttValue
PGresParamDesc
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
v30 contains changes to hopefully make it build on MSVC.
--
�lvaro Herrera Valdivia, Chile
Attachments:
v30-libpq-pipeline.patchtext/x-diff; charset=us-asciiDownload
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 0553279314..c16befa314 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -3173,6 +3173,33 @@ ExecStatusType PQresultStatus(const PGresult *res);
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-sync">
+ <term><literal>PGRES_PIPELINE_SYNC</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a
+ synchronization point in pipeline mode, requested by
+ <xref linkend="libpq-PQsendPipeline"/>.
+ This status occurs only when pipeline mode has been selected.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-aborted">
+ <term><literal>PGRES_PIPELINE_ABORTED</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a pipeline that has
+ received an error from the server. <function>PQgetResult</function>
+ must be called repeatedly, and each time it will return this status code
+ until the end of the current pipeline, at which point it will return
+ <literal>PGRES_PIPELINE_SYNC</literal> and normal processing can
+ resume.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
If the result status is <literal>PGRES_TUPLES_OK</literal> or
@@ -4919,6 +4946,498 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-pipeline-mode">
+ <title>Pipeline Mode</title>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>libpq</primary>
+ <secondary>pipeline mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>pipelining</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>batch mode</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> pipeline mode allows applications to
+ send a query without having to read the result of the previously
+ sent query. Taking advantage of the pipeline mode, a client will wait
+ less for the server, since multiple queries/results can be sent/
+ received in a single network transaction.
+ </para>
+
+ <para>
+ While pipeline mode provides a significant performance boost, writing
+ clients using the pipeline mode is more complex because it involves
+ managing a queue of pending queries and finding which result
+ corresponds to which query in the queue.
+ </para>
+
+ <para>
+ Pipeline mode also generally consumes more memory on both the client and server,
+ though careful and aggressive management of the send/receive queue can mitigate
+ this. This applies whether or not the connection is in blocking or non-blocking
+ mode.
+ </para>
+
+ <sect2 id="libpq-pipeline-using">
+ <title>Using Pipeline Mode</title>
+
+ <para>
+ To issue pipelines, the application must switch a connection into pipeline mode.
+ Enter pipeline mode with <xref linkend="libpq-PQenterPipelineMode"/>
+ or test whether pipeline mode is active with
+ <xref linkend="libpq-PQpipelineStatus"/>.
+ In pipeline mode, only <link linkend="libpq-async">asynchronous operations</link>
+ are permitted, and <literal>COPY</literal> is disallowed.
+ Using any synchronous command execution functions,
+ such as <function>PQfn</function>, or <function>PQexec</function>
+ and its sibling functions, is an error condition.
+ </para>
+
+ <note>
+ <para>
+ It is best to use pipeline mode with <application>libpq</application> in
+ <link linkend="libpq-PQsetnonblocking">non-blocking mode</link>. If used
+ in blocking mode it is possible for a client/server deadlock to occur.
+ <footnote>
+ <para>
+ The client will block trying to send queries to the server, but the
+ server will block trying to send results to the client from queries
+ it has already processed. This only occurs when the client sends
+ enough queries to fill both its output buffer and the server's receive
+ buffer before it switches to processing input from the server,
+ but it's hard to predict exactly when that will happen.
+ </para>
+ </footnote>
+ </para>
+ </note>
+
+ <sect3 id="libpq-pipeline-sending">
+ <title>Issuing Queries</title>
+
+ <para>
+ After entering pipeline mode, the application dispatches requests using
+ <xref linkend="libpq-PQsendQueryParams"/>,
+ or its prepared-query sibling
+ <xref linkend="libpq-PQsendQueryPrepared"/>.
+ These requests are queued on the client-side until flushed to the server;
+ this occurs when <xref linkend="libpq-PQsendPipeline"/> is used to
+ establish a synchronization point in the pipeline,
+ or when <xref linkend="libpq-PQflush"/> is called.
+ The functions <xref linkend="libpq-PQsendPrepare"/>,
+ <xref linkend="libpq-PQsendDescribePrepared"/>, and
+ <xref linkend="libpq-PQsendDescribePortal"/> also work in pipeline mode.
+ Result processing is described below.
+ </para>
+
+ <para>
+ The server executes statements, and returns results, in the order the
+ client sends them. The server will begin executing the commands in the
+ pipeline immediately, not waiting for the end of the pipeline.
+ If any statement encounters an error, the server aborts the current
+ transaction and skips processing commands in the pipeline until the
+ next synchronization point established by <function>PQsendPipeline</function>.
+ (This remains true even if the commands in the pipeline would rollback
+ the transaction.)
+ Query processing resumes after the synchronization point.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one; for example, one query may define a table that the next
+ query in the same pipeline uses. Similarly, an application may
+ create a named prepared statement and execute it with later
+ statements in the same pipeline.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-results">
+ <title>Processing Results</title>
+
+ <para>
+ To process the result of one query in a pipeline, the application calls
+ <function>PQgetResult</function> repeatedly and handles each result
+ until <function>PQgetResult</function> returns null.
+ The result from the next query in the pipeline may then be retrieved using
+ <function>PQgetResult</function> again and the cycle repeated.
+ The application handles individual statement results as normal.
+ When the results of all the queries in the pipeline have been
+ returned, <function>PQgetResult</function> returns a result
+ containing the status value <literal>PGRES_PIPELINE_SYNC</literal>.
+ </para>
+
+ <para>
+ The client may choose to defer result processing until the complete
+ pipeline has been sent, or interleave that with sending further
+ queries in the pipeline; see <xref linkend="libpq-pipeline-interleave" />.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ This mode selection is effective only for the query currently
+ being processed. For more information on the use of
+ <function>PQsetSingleRowMode</function>,
+ refer to <xref linkend="libpq-single-row-mode"/>.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal
+ asynchronous processing except that it may contain the new
+ <type>PGresult</type> types <literal>PGRES_PIPELINE_SYNC</literal>
+ and <literal>PGRES_PIPELINE_ABORTED</literal>.
+ <literal>PGRES_PIPELINE_SYNC</literal> is reported exactly once for each
+ <function>PQsendPipeline</function> after retrieving results for all
+ queries in the pipeline.
+ <literal>PGRES_PIPELINE_ABORTED</literal> is emitted in place of a normal
+ stream result for the first error and all subsequent results
+ except <literal>PGRES_PIPELINE_SYNC</literal> and null;
+ see <xref linkend="libpq-pipeline-errors"/>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing pipeline results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed (except that
+ <function>PQgetResult</function> returns null to indicate that we start
+ returning the results of next query). The application must keep track
+ of the order in which it sent queries, to associate them with their
+ corresponding results.
+ Applications will typically use a state machine or a FIFO queue for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-pipeline-errors">
+ <title>Error Handling</title>
+
+ <para>
+ When a query in a pipeline causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the pipeline
+ synchronization message. The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after <function>PQresultStatus</function>
+ returns <literal>PGRES_FATAL_ERROR</literal>,
+ the pipeline is flagged as aborted.
+ <function>PQresultStatus</function>, will report a
+ <literal>PGRES_PIPELINE_ABORTED</literal> result for each remaining queued
+ operation in an aborted pipeline. The result for
+ <function>PQsendPipeline</function> is reported as
+ <literal>PGRES_PIPELINE_SYNC</literal> to signal the end of the aborted pipeline
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the pipeline used an implicit transaction, then operations that have
+ already executed are rolled back and operations that were queued to follow
+ the failed operation are skipped entirely. The same behaviour holds if the
+ pipeline starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the pipeline. If a pipeline contains
+ <emphasis>multiple explicit transactions</emphasis>, all transactions that
+ committed prior to the error remain committed, the currently in-progress
+ transaction is aborted, and all subsequent operations are skipped completely,
+ including subsequent transactions.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal> — only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously, the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-interleave">
+ <title>Interleaving Result Processing and Query Dispatch</title>
+
+ <para>
+ To avoid deadlocks on large pipelines the client should be structured
+ around a non-blocking event loop using operating system facilities
+ such as <function>select</function>, <function>poll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work
+ remaining to be dispatched and a queue of work that has been dispatched
+ but not yet had its results processed. When the socket is writable
+ it should dispatch more work. When the socket is readable it should
+ read results and process them, matching them up to the next entry in
+ its expected results queue. Based on available memory, results from the
+ socket should be read frequently: there's no need to wait until the
+ pipeline end to read the results. Pipelines should be scoped to logical
+ units of work, usually (but not necessarily) one transaction per pipeline.
+ There's no need to exit pipeline mode and re-enter it between pipelines,
+ or to wait for one pipeline to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state
+ machine to track sent and received work is in
+ <filename>src/test/modules/test_libpq/pipeline.c</filename>
+ in the PostgreSQL source distribution.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-end">
+ <title>Ending Pipeline Mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed, and
+ the end pipeline result has been consumed, the application may return
+ to non-pipelined mode with <xref linkend="libpq-PQexitPipelineMode"/>.
+ </para>
+ </sect3>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-functions">
+ <title>Functions Associated with Pipeline Mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQpipelineStatus">
+ <term>
+ <function>PQpipelineStatus</function>
+ <indexterm>
+ <primary>PQpipelineStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the current pipeline mode status of the
+ <application>libpq</application> connection.
+<synopsis>
+PGpipelineStatus PQpipelineStatus(const PGconn *conn);
+</synopsis>
+ </para>
+
+ <para>
+ <function>PQpipelineStatus</function> can return one of the following values:
+
+ <variablelist>
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ON</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in
+ pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_OFF</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is
+ <emphasis>not</emphasis> in pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in pipeline
+ mode and an error occurred while processing the current pipeline.
+ The aborted flag is cleared when <function>PQresultStatus</function>
+ returns PGRES_PIPELINE_SYNC at the end of the pipeline.
+ Clients don't usually need this function to
+ verify aborted status, as they can tell that the pipeline is aborted
+ from the <literal>PGRES_PIPELINE_ABORTED</literal> result code.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterPipelineMode">
+ <term>
+ <function>PQenterPipelineMode</function>
+ <indexterm>
+ <primary>PQenterPipelineMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter pipeline mode if it is currently idle or
+ already in pipeline mode.
+
+<synopsis>
+int PQenterPipelineMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success.
+ Returns 0 and has no effect if the connection is not currently
+ idle, i.e., it has a result ready, or it is waiting for more
+ input from the server, etc.
+ This function does not actually send anything to the server,
+ it just changes the <application>libpq</application> connection
+ state.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitPipelineMode">
+ <term>
+ <function>PQexitPipelineMode</function>
+ <indexterm>
+ <primary>PQexitPipelineMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit pipeline mode if it is currently in pipeline mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitPipelineMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 1 and takes no action if not in
+ pipeline mode. If the current statement isn't finished processing
+ or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQsendPipeline">
+ <term>
+ <function>PQsendPipeline</function>
+ <indexterm>
+ <primary>PQsendPipeline</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Marks a synchronization point in a pipeline by sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ and flushing the send buffer. This serves as
+ the delimiter of an implicit transaction and an error recovery
+ point; see <xref linkend="libpq-pipeline-errors"/>.
+
+<synopsis>
+int PQsendPipeline(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 if the connection is not in
+ pipeline mode or sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ failed.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-tips">
+ <title>When to Use Pipeline Mode</title>
+
+ <para>
+ Much like asynchronous query mode, there is no meaningful performance
+ overhead when using pipeline mode. It increases client application complexity,
+ and extra caution is required to prevent client/server deadlocks, but
+ pipeline mode can offer considerable performance improvements, in exchange for
+ increased memory usage from leaving state around longer.
+ </para>
+
+ <para>
+ Pipeline mode is most useful when the server is distant, i.e., network latency
+ (<quote>ping time</quote>) is high, and also when many small operations
+ are being performed in rapid succession. There is usually less benefit
+ in using pipelined commands when each query takes many multiples of the client/server
+ round-trip time to execute. A 100-statement operation run on a server
+ 300ms round-trip-time away would take 30 seconds in network latency alone
+ without pipelining; with pipelining it may spend as little as 0.3s waiting for
+ results from the server.
+ </para>
+
+ <para>
+ Use pipelined commands when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed
+ into operations on sets, or into a <literal>COPY</literal> operation.
+ </para>
+
+ <para>
+ Pipeline mode is not useful when information from one operation is required by
+ the client to produce the next operation. In such cases, the client
+ would have to introduce a synchronization point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+BEGIN;
+SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+-- result: x=2
+-- client adds 1 to x:
+UPDATE mytable SET x = 3 WHERE id = 42;
+COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <para>
+ Pipelining is less useful, and more complex, when a single pipeline contains
+ multiple transactions (see <xref linkend="libpq-pipeline-errors"/>).
+ </para>
+
+ <note>
+ <para>
+ The pipeline API was introduced in <productname>PostgreSQL</productname> 14.
+ Pipeline mode is a client-side feature which doesn't require server
+ support, and works on any server that supports the v3 extended query
+ protocol.
+ </para>
+ </note>
+ </sect2>
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4959,6 +5478,13 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <para>
+ When using pipeline mode, single-row mode needs to be activated for each
+ query in the pipeline before retrieving results for that query
+ with <function>PQgetResult</function>.
+ See <xref linkend="libpq-pipeline-mode"/> for more information.
+ </para>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 6d46da42e2..012e44c736 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while a libpq connection is in pipeline mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 2ec0580a79..75448162ca 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -1109,6 +1109,12 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
row, the last value is kept.
</para>
+ <para>
+ <literal>\gset</literal> and <literal>\aset</literal> cannot be used
+ pipeline mode, since query results are not immediately
+ fetched in this mode.
+ </para>
+
<para>
The following example puts the final account balance from the first query
into variable <replaceable>abalance</replaceable>, and fills variables
@@ -1269,6 +1275,21 @@ SELECT 4 AS four \; SELECT 5 AS five \aset
</programlisting></para>
</listitem>
</varlistentry>
+
+ <varlistentry id='pgbench-metacommand-pipeline'>
+ <term><literal>\startpipeline</literal></term>
+ <term><literal>\endpipeline</literal></term>
+
+ <listitem>
+ <para>
+ These commands delimit the start and end of a pipeline of SQL statements.
+ In a pipeline, statements are sent to server without waiting for the results
+ of previous statements (see <xref linkend="libpq-pipeline-mode"/>).
+ Pipeline mode requires the extended query protocol.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect2>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 5272eed9ab..f74378110a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1019,6 +1019,12 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->err = _("empty query");
break;
+ case PGRES_PIPELINE_SYNC:
+ case PGRES_PIPELINE_ABORTED:
+ walres->status = WALRCV_ERROR;
+ walres->err = _("unexpected pipeline mode");
+ break;
+
case PGRES_NONFATAL_ERROR:
case PGRES_FATAL_ERROR:
case PGRES_BAD_RESPONSE:
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 31a4df45f5..fbbe270654 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -351,10 +351,11 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
- * and we enter the CSTATE_SLEEP state to wait for it to expire. Other
- * meta-commands are executed immediately. If the command about to start
- * is actually beyond the end of the script, advance to CSTATE_END_TX.
+ * CSTATE_WAIT_RESULT state unless in pipeline mode. On a \sleep
+ * meta-command, the timer is set, and we enter the CSTATE_SLEEP state to
+ * wait for it to expire. Other meta-commands are executed immediately. If
+ * the command about to start is actually beyond the end of the script,
+ * advance to CSTATE_END_TX.
*
* CSTATE_WAIT_RESULT waits until we get a result set back from the server
* for the current command.
@@ -484,7 +485,9 @@ typedef enum MetaCommand
META_IF, /* \if */
META_ELIF, /* \elif */
META_ELSE, /* \else */
- META_ENDIF /* \endif */
+ META_ENDIF, /* \endif */
+ META_STARTPIPELINE, /* \startpipeline */
+ META_ENDPIPELINE /* \endpipeline */
} MetaCommand;
typedef enum QueryMode
@@ -2504,6 +2507,10 @@ getMetaCommand(const char *cmd)
mc = META_GSET;
else if (pg_strcasecmp(cmd, "aset") == 0)
mc = META_ASET;
+ else if (pg_strcasecmp(cmd, "startpipeline") == 0)
+ mc = META_STARTPIPELINE;
+ else if (pg_strcasecmp(cmd, "endpipeline") == 0)
+ mc = META_ENDPIPELINE;
else
mc = META_NONE;
return mc;
@@ -2693,11 +2700,25 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In pipeline mode, we use asynchronous functions. If a
+ * server-side error occurs, it will be processed later
+ * among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ pg_log_error("%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2741,8 +2762,10 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
* varprefix should be set only with \gset or \aset, and SQL commands do
* not need it.
*/
+#if 0
Assert((meta == META_NONE && varprefix == NULL) ||
((meta == META_GSET || meta == META_ASET) && varprefix != NULL));
+#endif
res = PQgetResult(st->con);
@@ -2810,6 +2833,12 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* otherwise the result is simply thrown away by PQclear below */
break;
+ case PGRES_PIPELINE_SYNC:
+ pg_log_debug("client %d pipeline ending", st->id);
+ if (PQexitPipelineMode(st->con) != 1)
+ pg_log_error("client %d failed to exit pipeline mode", st->id);
+ break;
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3066,13 +3095,36 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
/* Execute the command */
if (command->type == SQL_COMMAND)
{
+ /* disallow \aset and \gset in pipeline mode */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ if (command->meta == META_GSET)
+ {
+ commandFailed(st, "gset", "\\gset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ else if (command->meta == META_ASET)
+ {
+ commandFailed(st, "aset", "\\aset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ }
+
if (!sendCommand(st, command))
{
commandFailed(st, "SQL", "SQL command send failed");
st->state = CSTATE_ABORTED;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in pipeline mode */
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -3212,7 +3264,15 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (readCommandResponse(st,
sql_script[st->use_file].commands[st->command]->meta,
sql_script[st->use_file].commands[st->command]->varprefix))
- st->state = CSTATE_END_COMMAND;
+ {
+ /*
+ * outside of pipeline mode: stop reading results.
+ * pipeline mode: continue reading results until an
+ * end-of-pipeline response.
+ */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ st->state = CSTATE_END_COMMAND;
+ }
else
st->state = CSTATE_ABORTED;
break;
@@ -3456,6 +3516,49 @@ executeMetaCommand(CState *st, instr_time *now)
return CSTATE_ABORTED;
}
}
+ else if (command->meta == META_STARTPIPELINE)
+ {
+ /*
+ * In pipeline mode, we use a workflow based on libpq pipeline
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "startpipeline", "cannot use pipeline mode with the simple query protocol");
+ return CSTATE_ABORTED;
+ }
+
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ commandFailed(st, "startpipeline", "already in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (PQenterPipelineMode(st->con) == 0)
+ {
+ commandFailed(st, "startpipeline", "failed to enter pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ }
+ else if (command->meta == META_ENDPIPELINE)
+ {
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ {
+ commandFailed(st, "endpipeline", "not in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (!PQsendPipeline(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to send the pipeline");
+ return CSTATE_ABORTED;
+ }
+ if (!PQexitPipelineMode(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to exit pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ /* collect pending results before getting out of pipeline mode */
+ return CSTATE_WAIT_RESULT;
+ }
/*
* executing the expression or shell command might have taken a
@@ -4683,7 +4786,9 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
- else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
+ else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF ||
+ my_command->meta == META_STARTPIPELINE ||
+ my_command->meta == META_ENDPIPELINE)
{
if (my_command->argc != 1)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
@@ -6808,4 +6913,5 @@ pthread_join(pthread_t th, void **thread_return)
return 0;
}
+
#endif /* WIN32 */
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..60d09e6d63 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,7 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterPipelineMode 180
+PQexitPipelineMode 181
+PQsendPipeline 182
+PQpipelineStatus 183
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index f83af03d0a..4c97e8a078 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -540,6 +540,23 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * pqFreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+pqFreeCommandQueue(PGcommandQueueEntry *queue)
+{
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *cur = queue;
+
+ queue = cur->next;
+ if (cur->query)
+ free(cur->query);
+ free(cur);
+ }
+}
/*
* pqDropServerData
@@ -571,6 +588,12 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ pqFreeCommandQueue(conn->cmd_queue_head);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ pqFreeCommandQueue(conn->cmd_queue_recycle);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
@@ -2486,6 +2509,7 @@ keep_going: /* We will come back to here until there is
/* Drop any PGresult we might have, too */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn);
/* Reset conn->status to put the state machine in the right state */
@@ -3928,6 +3952,7 @@ makeEmptyPGconn(void)
conn->status = CONNECTION_BAD;
conn->asyncStatus = PGASYNC_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
conn->xactStatus = PQTRANS_IDLE;
conn->options_valid = false;
conn->nonblocking = false;
@@ -4187,6 +4212,7 @@ closePGconn(PGconn *conn)
conn->status = CONNECTION_BAD; /* Well, not really _bad_ - just absent */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn); /* deallocate result */
resetPQExpBuffer(&conn->errorMessage);
release_conn_addrinfo(conn);
@@ -6735,6 +6761,15 @@ PQbackendPID(const PGconn *conn)
return conn->be_pid;
}
+PGpipelineStatus
+PQpipelineStatus(const PGconn *conn)
+{
+ if (!conn)
+ return PQ_PIPELINE_OFF;
+
+ return conn->pipelineStatus;
+}
+
int
PQconnectionNeedsPassword(const PGconn *conn)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9a038043b2..0fb790fdfa 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_PIPELINE_SYNC",
+ "PGRES_PIPELINE_ABORTED"
};
/*
@@ -71,6 +73,11 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *pqMakePipelineCmd(PGconn *conn);
+static void pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqPipelineProcessQueue(PGconn *conn);
+static int pqPipelineFlush(PGconn *conn);
/* ----------------
@@ -1171,7 +1178,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1197,18 +1204,37 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot use %s in pipeline mode, use PQsendQueryParams\n"),
+ "PQsendQuery");
+ return 0;
+ }
+
return PQsendQueryInternal(conn, query, true);
}
int
PQsendQueryContinue(PGconn *conn, const char *query)
{
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot use %s in pipeline mode, use PQsendQueryParams\n"),
+ "PQsendQueryContinue");
+ return 0;
+ }
+
return PQsendQueryInternal(conn, query, false);
}
static int
PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
{
+ /* pipeline mode requires extended query protocol */
+ Assert(conn->pipelineStatus == PQ_PIPELINE_OFF);
+
if (!PQsendQueryStart(conn, newQuery))
return 0;
@@ -1307,6 +1333,8 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+
if (!PQsendQueryStart(conn, true))
return 0;
@@ -1330,6 +1358,15 @@ PQsendPrepare(PGconn *conn,
return 0;
}
+ /* Alloc pipeline memory before doing anything */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+ }
+
/* construct the Parse message */
if (pqPutMsgStart('P', conn) < 0 ||
pqPuts(stmtName, conn) < 0 ||
@@ -1356,32 +1393,46 @@ PQsendPrepare(PGconn *conn,
if (pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /*
+ * In non-pipeline mode, add a Sync and prepare to send. In pipeline mode
+ * we just keep track of the new message.
+ */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ /* remember we are doing just a Parse */
+ conn->queryclass = PGQUERY_PREPARE;
- /* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
-
- /* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ /* and remember the query text too, if possible */
+ /* if insufficient memory, last_query just winds up NULL */
+ if (conn->last_query)
+ free(conn->last_query);
+ conn->last_query = strdup(query);
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ else
+ {
+ pipeCmd->queryclass = PGQUERY_PREPARE;
+ /* as above, if insufficient memory, query winds up NULL */
+ pipeCmd->query = strdup(query);
+ pqAppendPipelineCmd(conn, pipeCmd);
+ }
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
- /* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1429,7 +1480,8 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn, bool newQuery)
@@ -1450,20 +1502,57 @@ PQsendQueryStart(PGconn *conn, bool newQuery)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE &&
+ conn->pipelineStatus == PQ_PIPELINE_OFF)
{
appendPQExpBufferStr(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * When enqueuing commands we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when pqPipelineProcessQueue()
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ }
+ }
+ else
+ {
+ /*
+ * This command's results will come in immediately. Initialize async
+ * result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1487,10 +1576,34 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **query;
+ PGQueryClass *queryclass;
/*
- * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
- * using specified statement name and the unnamed portal.
+ * Decide where the query is going to be stored. In pipeline mode, we
+ * allocate a new pipeline element; in non-pipeline mode, it's simply the
+ * connection's last query.
+ */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+ /*
+ * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync
+ * (if not in pipeline mode), using specified statement name and the
+ * unnamed portal.
*/
if (command)
@@ -1600,35 +1713,43 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /* construct the Sync message if not in pipeline mode */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ /* if insufficient memory, query just winds up NULL */
+ if (*query)
+ free(*query);
if (command)
- conn->last_query = strdup(command);
+ *query = strdup(command);
else
- conn->last_query = NULL;
+ *query = NULL;
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1726,14 +1847,17 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
-
/*
* PQgetResult
* Get the next PGresult produced by a query. Returns NULL if no
* query work remains or an error has occurred (e.g. out of
* memory).
+ *
+ * In pipeline mode, once all the result of a query have been returned,
+ * PQgetResult returns NULL to let the user know that the next
+ * query is being processed. At the end of the pipeline, returns a
+ * result with PQresultStatus(result) == PGRES_PIPELINE_SYNC.
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1803,8 +1927,49 @@ PQgetResult(PGconn *conn)
{
case PGASYNC_IDLE:
res = NULL; /* query is complete */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to return the NULL that terminates the round of
+ * results from the current query; prepare to send the results
+ * of the next query when we're called next. Also, since this
+ * is the start of the results of the next query, clear any
+ * prior error message.
+ */
+ resetPQExpBuffer(&conn->errorMessage);
+ pqPipelineProcessQueue(conn);
+ }
break;
case PGASYNC_READY:
+ res = pqPrepareAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to send the results of the current query. Set
+ * us idle now, and ...
+ */
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /*
+ * ... in cases when we're sending a pipeline-related result,
+ * move queue processing forwards immediately, so that next
+ * time we're called, we're prepared to return the next result
+ * received from the server.
+ *
+ * In all other cases, leave the queue state change for next
+ * time, so that a terminating NULL result is sent.
+ */
+ if (res && (res->resultStatus == PGRES_PIPELINE_ABORTED ||
+ res->resultStatus == PGRES_PIPELINE_SYNC))
+ pqPipelineProcessQueue(conn);
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -1985,6 +2150,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("synchronous command execution functions are not allowed in pipeline mode\n"));
+ return false;
+ }
+
/*
* Since this is the beginning of a query cycle, reset the error buffer.
*/
@@ -2148,6 +2320,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2155,6 +2330,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
if (!PQsendQueryStart(conn, true))
return 0;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ queryclass = &conn->queryclass;
+
/* construct the Describe message */
if (pqPutMsgStart('D', conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2163,32 +2350,40 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ /* reset last-query string (not relevant now) */
+ if (conn->last_query && conn->pipelineStatus != PQ_PIPELINE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
}
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -2541,6 +2736,13 @@ PQfn(PGconn *conn,
*/
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("PQfn not allowed in pipeline mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -2555,6 +2757,359 @@ PQfn(PGconn *conn,
args, nargs);
}
+/* ====== Pipeline mode support ======== */
+
+/*
+ * PQenterPipelineMode
+ * Put an idle connection in pipeline mode.
+ *
+ * Returns 1 on success. On failure, errorMessage is set and 0 is returned.
+ *
+ * Commands submitted after this can be pipelined on the connection;
+ * there's no requirement to wait for one to finish before the next is
+ * dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQsendPipeline. Multiple pipelines
+ * can be sent while in pipeline mode. Pipeline mode can be exited
+ * by calling PQexitPipelineMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ /* succeed with no action if already in pipeline mode */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot enter pipeline mode, connection not idle\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+
+ return 1;
+}
+
+/*
+ * PQexitPipelineMode
+ * End pipeline mode and return to normal command mode.
+ *
+ * Returns 1 in success (pipeline mode successfully ended, or not in pipeline
+ * mode).
+ *
+ * Returns 0 if in pipeline mode and cannot be ended yet. Error message will
+ * be set.
+ */
+int
+PQexitPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ /* there are some uncollected results */
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+
+ case PGASYNC_BUSY:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode while busy\n"));
+ return 0;
+
+ default:
+ /* OK */
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ return 0; /* error message is setup already */
+ return 1;
+}
+
+/*
+ * pqPipelineProcessQueue: subroutine for PQgetResult
+ * In pipeline mode, start processing the results of the next query in the queue.
+ */
+static void
+pqPipelineProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ Assert(conn->pipelineStatus != PQ_PIPELINE_OFF);
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: COPY in pipeline mode\n");
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return;
+ case PGASYNC_IDLE:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In pipeline mode but nothing left on the queue; caller can submit
+ * more work or PQexitPipelineMode() now.
+ */
+ return;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-pipeline call.
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /*
+ * Reset single-row processing mode. (Client has to set it up for each
+ * query, if desired.)
+ */
+ conn->singleRowMode = false;
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ pqRecyclePipelineCmd(conn, next_query);
+
+ if (conn->pipelineStatus == PQ_PIPELINE_ABORTED &&
+ conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted pipeline we don't get anything from the server for
+ * each result; we're just discarding input until we get to the next
+ * sync from the server. The client needs to know its queries got
+ * aborted so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn, PGRES_PIPELINE_ABORTED);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ pqSaveErrorResult(conn);
+ return;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+}
+
+/*
+ * PQsendPipeline
+ * Send a Sync message as part of a pipeline, and flush to server
+ *
+ * It's legal to start submitting more commands in the pipeline immediately,
+ * without waiting for the results of the current pipeline. There's no need to
+ * end pipeline mode and start it again.
+ *
+ * If a command in a pipeline fails, every subsequent command up to and including
+ * the result to the Sync message sent by PQsendPipeline gets set to
+ * PGRES_PIPELINE_ABORTED state. If the whole pipeline is processed without
+ * error, a PGresult with PGRES_PIPELINE_SYNC is produced.
+ *
+ * Queries can already have been sent before PQsendPipeline is called, but
+ * PQsendPipeline need to be called before retrieving command results.
+ *
+ * The connection will remain in pipeline mode and unavailable for new
+ * synchronous command execution functions until all results from the pipeline
+ * are processed by the client.
+ */
+int
+PQsendPipeline(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot send pipeline when not in pipeline mode\n"));
+ return 0;
+ }
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: cannot send pipeline while in COPY\n");
+ return 0;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_IDLE:
+ /* OK to send sync */
+ break;
+ }
+
+ entry = pqMakePipelineCmd(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ pqAppendPipelineCmd(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ /*
+ * Call pqPipelineProcessQueue so the user can call start calling
+ * PQgetResult.
+ */
+ pqPipelineProcessQueue(conn);
+
+ return 1;
+
+sendFailed:
+ pqRecyclePipelineCmd(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
+
+/*
+ * pqMakePipelineCmd
+ * Get a command queue entry for caller to fill.
+ *
+ * If the recycle queue has a free element, that is returned; if not, a
+ * fresh one is allocated. Caller is responsible for adding it to the
+ * command queue (pqAppendPipelineCmd) once the struct is filled in, or
+ * releasing the memory (pqRecyclePipelineCmd) if an error occurs.
+ *
+ * If allocation fails, sets the error message and returns NULL.
+ */
+static PGcommandQueueEntry *
+pqMakePipelineCmd(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * pqAppendPipelineCmd
+ * Append a caller-allocated command queue entry to the queue.
+ *
+ * The query itself must already have been put in the output buffer by the
+ * caller.
+ */
+static void
+pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ PGcommandQueueEntry **tail;
+
+ if (conn->cmd_queue_head == NULL)
+ tail = &conn->cmd_queue_head;
+ else
+ tail = &conn->cmd_queue_tail->next;
+
+ *tail = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * pqRecyclePipelineCmd
+ * Push a command queue entry onto the freelist. It must be an entry
+ * with null next pointer and not referenced by any other entry's next
+ * pointer.
+ */
+static void
+pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+
+ Assert(entry->next == NULL);
+
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
/* ====== accessor funcs for PGresult ======== */
@@ -3152,6 +3707,23 @@ PQflush(PGconn *conn)
return pqFlush(conn);
}
+/*
+ * pqPipelineFlush
+ *
+ * In pipeline mode, data will be flushed only when the out buffer reaches the
+ * threshold value. In non-pipeline mode, it behaves as stock pqFlush.
+ *
+ * Returns 0 on success.
+ */
+static int
+pqPipelineFlush(PGconn *conn)
+{
+ if ((conn->pipelineStatus == PQ_PIPELINE_OFF) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return pqFlush(conn);
+ return 0;
+}
+
/*
* PQfreemem - safely frees memory allocated
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 2ca8c057b9..b131995974 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -177,14 +177,24 @@ pqParseInput3(PGconn *conn)
if (getParameterStatus(conn))
return;
}
- else
- {
- pqInternalNotice(&conn->noticeHooks,
- "message type 0x%02x arrived from server while idle",
- id);
- /* Discard the unexpected message */
- conn->inCursor += msgLength;
- }
+
+ /*
+ * We're also IDLE when in pipeline mode we have completed
+ * processing the results of one query and are waiting for the
+ * next one in the pipeline. In this case, as above, just wait to
+ * see what's next.
+ */
+ if (conn->asyncStatus == PGASYNC_IDLE &&
+ conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->cmd_queue_head != NULL)
+ return;
+
+ /* Any other case is unexpected and we summarily skip it */
+ pqInternalNotice(&conn->noticeHooks,
+ "message type 0x%02x arrived from server while idle",
+ id);
+ /* Discard the unexpected message */
+ conn->inCursor += msgLength;
}
else
{
@@ -217,10 +227,28 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new
+ * query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_PIPELINE_SYNC);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ else
+ {
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -450,7 +478,7 @@ handleSyncLoss(PGconn *conn, char id, int msgLength)
id, msgLength);
/* build an error result holding the error message */
pqSaveErrorResult(conn);
- conn->asyncStatus = PGASYNC_READY; /* drop out of GetResult wait loop */
+ conn->asyncStatus = PGASYNC_READY; /* drop out of PQgetResult wait loop */
/* flush input data since we're giving up on processing it */
pqDropConnection(conn, true);
conn->status = CONNECTION_BAD; /* No more connection to backend */
@@ -875,6 +903,10 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ /* If in pipeline mode, set error indicator for it */
+ if (isError && conn->pipelineStatus != PQ_PIPELINE_OFF)
+ conn->pipelineStatus = PQ_PIPELINE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
@@ -930,9 +962,21 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
* Save the active query text, if any, into res as well; but only if we
* might need it for an error cursor display, which is only true if there
* is a PG_DIAG_STATEMENT_POSITION field.
+ *
+ * Note that in pipeline mode, we have not yet advanced the query pointer
+ * to the next query, so we have to look at that.
*/
- if (have_position && conn->last_query && res)
- res->errQuery = pqResultStrdup(res, conn->last_query);
+ if (have_position && res)
+ {
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->asyncStatus == PGASYNC_IDLE)
+ {
+ if (conn->cmd_queue_head)
+ res->errQuery = pqResultStrdup(res, conn->cmd_queue_head->query);
+ }
+ else if (conn->last_query)
+ res->errQuery = pqResultStrdup(res, conn->last_query);
+ }
/*
* Now build the "overall" error message for PQresultErrorMessage.
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index fa9b62a844..fb31a49fab 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -96,7 +96,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_PIPELINE_SYNC, /* pipeline synchronization point */
+ PGRES_PIPELINE_ABORTED, /* Command didn't run because of an abort
+ * earlier in a pipeline */
} ExecStatusType;
typedef enum
@@ -136,6 +139,16 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PGpipelineStatus - Current status of pipeline mode
+ */
+typedef enum
+{
+ PQ_PIPELINE_OFF,
+ PQ_PIPELINE_ON,
+ PQ_PIPELINE_ABORTED
+} PGpipelineStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -327,6 +340,7 @@ extern int PQserverVersion(const PGconn *conn);
extern char *PQerrorMessage(const PGconn *conn);
extern int PQsocket(const PGconn *conn);
extern int PQbackendPID(const PGconn *conn);
+extern PGpipelineStatus PQpipelineStatus(const PGconn *conn);
extern int PQconnectionNeedsPassword(const PGconn *conn);
extern int PQconnectionUsedPassword(const PGconn *conn);
extern int PQclientEncoding(const PGconn *conn);
@@ -434,6 +448,11 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for pipeline mode management */
+extern int PQenterPipelineMode(PGconn *conn);
+extern int PQexitPipelineMode(PGconn *conn);
+extern int PQsendPipeline(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 8d51e6ed9f..3e9f99f8fb 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,7 +217,11 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, more results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
@@ -229,7 +233,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* Sync at end of a pipeline */
} PGQueryClass;
/* Target server type (decoded value of target_session_attrs) */
@@ -305,6 +310,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by pipeline mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct PGcommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct PGcommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -397,6 +418,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PGpipelineStatus pipelineStatus; /* status of pipeline mode */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -409,6 +431,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue, for pipeline mode.
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -790,6 +822,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 5391f461a2..6e31458be2 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -17,6 +17,7 @@ SUBDIRS = \
test_extensions \
test_ginpostinglist \
test_integerset \
+ test_libpq \
test_misc \
test_parser \
test_pg_dump \
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000000..4fbf97a5b0
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/pipeline
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000000..2b00877c93
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,20 @@
+# src/test/modules/test_libpq/Makefile
+
+PROGRAM = pipeline
+OBJS = pipeline.o
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS_INTERNAL += $(libpq_pgport)
+
+TAP_TESTS = 1
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/pipeline.c b/src/test/modules/test_libpq/pipeline.c
new file mode 100644
index 0000000000..bb217e6b08
--- /dev/null
+++ b/src/test/modules/test_libpq/pipeline.c
@@ -0,0 +1,1141 @@
+/*
+ * src/test/modules/test_libpq/pipeline.c
+ * Verify libpq pipeline execution functionality
+ */
+#include "postgres_fe.h"
+
+#include <sys/time.h>
+
+#include "catalog/pg_type_d.h"
+#include "common/fe_memutils.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+
+static void exit_nicely(PGconn *conn);
+
+const char *const progname = "pipeline";
+
+
+#define DEBUG
+#ifdef DEBUG
+#define pg_debug(...) do { fprintf(stderr, __VA_ARGS__); } while (0)
+#else
+#define pg_debug(...)
+#endif
+
+static const char *const drop_table_sql =
+"DROP TABLE IF EXISTS pq_pipeline_demo";
+static const char *const create_table_sql =
+"CREATE UNLOGGED TABLE pq_pipeline_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql =
+"INSERT INTO pq_pipeline_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+/*
+ * Print an error to stderr and terminate the program.
+ */
+#define pg_fatal(...) pg_fatal_impl(__LINE__, __VA_ARGS__)
+static void
+pg_fatal_impl(int line, const char *fmt,...)
+{
+ va_list args;
+
+ fprintf(stderr, "\n"); /* XXX hack */
+ fprintf(stderr, "%s:%d: ", progname, line);
+
+ va_start(args, fmt);
+ vfprintf(stderr, fmt, args);
+ va_end(args);
+ printf("Failure, exiting\n");
+ exit(1);
+}
+
+static void
+test_disallowed(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("Unable to enter pipeline mode\n");
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not activated properly\n");
+
+ /* PQexec should fail in pipeline mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("PQexec should fail in pipeline mode but succeeded\n");
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ pg_fatal("PQsendQuery should fail in pipeline mode but succeeded\n");
+
+ /* Entering pipeline mode when already in pipeline mode is OK */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("re-entering pipeline mode should be a no-op but failed\n");
+
+ if (PQisBusy(conn) != 0)
+ pg_fatal("PQisBusy should return 0 when idle in pipeline mode, returned 1\n");
+
+ /* ok, back to normal command mode */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("couldn't exit idle empty pipeline mode\n");
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not terminated properly\n");
+
+ /* exiting pipeline mode when not in pipeline mode should be a no-op */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("pipeline mode exit when not in pipeline mode should succeed but failed\n");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("PQexec should succeed after exiting pipeline mode but failed with: %s\n",
+ PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_simple_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple pipeline... ");
+
+ /*
+ * Enter pipeline mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our output buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode with work in progress should fail, but succeeded\n");
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending pipeline failed: %s\n", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first query result.\n");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit pipeline mode yet
+ */
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result PGRES_PIPELINE_SYNC expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of PGRES_PIPELINE_SYNC, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after pipeline end: %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* ... until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_multi_pipelines(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi pipeline... ");
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending first pipeline failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s\n", PQerrorMessage(conn));
+
+ /* OK, start processing the results */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first result\n");
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ PQclear(res);
+
+#if 0
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+#endif
+
+ /* second pipeline */
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from second pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode ... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+/*
+ * When an operation in a pipeline fails the rest of the pipeline is flushed. We
+ * still have to get results for each pipeline item, but the item will just be
+ * a PGRES_PIPELINE_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the pipeline. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_aborted_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted pipeline... ");
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * pipeline ERRORs.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT no_such_function($1)",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching error select failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending first pipeline failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second-pipeline insert failed: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the pipeline results.
+ *
+ * We should get a command-ok for the first query, then a fatal error and
+ * a pipeline aborted message for the second insert, a pipeline-end, then
+ * a command-ok and a pipeline-ok for the second pipeline operation.
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result status %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* Second query caused error, so we expect an error next */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("Unexpected result code -- expected PGRES_FATAL_ERROR, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /*
+ * pipeline should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third pipeline since we've already sent a
+ * second. The aborted flag relates only to the pipeline being received.
+ */
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't\n");
+
+ /* third query in pipeline, the second insert */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_ABORTED)
+ pg_fatal("Unexpected result code -- expected PGRES_PIPELINE_ABORTED, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+#if 0
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+#endif
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't\n");
+
+ /* Ensure we're still in pipeline */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /*
+ * The end of a failed pipeline is a PGRES_PIPELINE_SYNC.
+ *
+ * (This is so clients know to start processing results normally again and
+ * can tell the difference between skipped commands and the sync.)
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code from first pipeline sync\n"
+ "Expected PGRES_PIPELINE_SYNC, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+#if 0
+ /* after the synchronization point we get a NULL result */
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+#endif
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_ABORTED)
+ pg_fatal("sync should've cleared the aborted flag but didn't\n");
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* the insert from the second pipeline */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result code %s from first item in second pipeline\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* Read the NULL result at the end of the command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* the second pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQerrorMessage(conn));
+
+ /* Test single-row mode with an error partways */
+ if (PQsendQueryParams(conn, "SELECT 1/g FROM generate_series(5, -1, -1) g",
+ 0, NULL, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to send query: %s", PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ PQsetSingleRowMode(conn);
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ if (PQresultStatus(res) == PGRES_FATAL_ERROR)
+ printf("got error: {{%s}} (expected: division by zero)\n", PQerrorMessage(conn));
+ else if (PQresultStatus(res) == PGRES_SINGLE_TUPLE)
+ printf("got row: %s\n", PQgetvalue(res, 0, 0));
+ else
+ pg_fatal("got unexpected result %s\n", PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ }
+ /* the third pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from third pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ /*-
+ * Since we fired the pipelines off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st pipeline
+ * - First insert applied
+ * - Second statement aborted xact
+ * - Third insert skipped
+ * - Sync rolled back first implicit xact
+ * - Implicit xact created by server around 2nd pipeline
+ * - insert applied from 2nd pipeline
+ * - Sync commits 2nd xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM pq_pipeline_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Expected tuples, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("expected 1 result, got %d\n", PQntuples(res));
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ pg_fatal("expected only insert with value 3, got %s", val);
+ }
+
+ PQclear(res);
+}
+
+/* State machine enum for test_pipelined_insert */
+typedef enum PipelineInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} PipelineInsertStep;
+
+static void
+test_pipelined_insert(PGconn *conn, int n_rows)
+{
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ PipelineInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a pipelined insert into a table created at the start of the pipeline
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ while (send_step != BI_PREPARE)
+ {
+ const char *sql;
+
+ switch (send_step)
+ {
+ case BI_BEGIN_TX:
+ sql = "BEGIN TRANSACTION";
+ send_step = BI_DROP_TABLE;
+ break;
+
+ case BI_DROP_TABLE:
+ sql = drop_table_sql;
+ send_step = BI_CREATE_TABLE;
+ break;
+
+ case BI_CREATE_TABLE:
+ sql = create_table_sql;
+ send_step = BI_PREPARE;
+ break;
+
+ default:
+ pg_fatal("invalid state");
+ }
+
+ pg_debug("sending: %s\n", sql);
+ if (PQsendQueryParams(conn, sql,
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching %s failed: %s\n", sql, PQerrorMessage(conn));
+ }
+
+ Assert(send_step == BI_PREPARE);
+ pg_debug("sending: %s\n", insert_sql);
+ if (PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids) != 1)
+ pg_fatal("dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our output buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the pipeline before processing
+ * results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ pg_fatal("failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's output buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ PGresult *res;
+ const char *cmdtag;
+ const char *description = "";
+ int status;
+
+ /*
+ * Read next result. If no more results from this query,
+ * advance to the next query
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ continue;
+
+ status = PGRES_COMMAND_OK;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ recv_step++;
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ recv_step++;
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ recv_step++;
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ recv_step++;
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive == 0)
+ recv_step++;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ recv_step++;
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_PIPELINE_SYNC;
+ recv_step++;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ description = "";
+ abort();
+ }
+
+ if (PQresultStatus(res) != status)
+ pg_fatal("%s reported status %s, expected %s\n"
+ "Error message: \"%s\"\n",
+ description, PQresStatus(PQresultStatus(res)),
+ PQresStatus(status), PQerrorMessage(conn));
+
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ pg_fatal("%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+
+ pg_debug("Got %s OK\n", cmdtag[0] != '\0' ? cmdtag : description);
+
+ PQclear(res);
+ }
+ }
+
+ /* Write more rows and/or the end pipeline message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQsendPipeline(conn) == 1)
+ {
+ fprintf(stdout, "Sent pipeline\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending pipeline failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+ }
+
+ /* We've got the sync message and the pipeline should be done */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ pg_fatal("failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i;
+ bool pipeline_ended = false;
+
+ /* 1 pipeline, 3 queries in it */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n",
+ PQerrorMessage(conn));
+
+ for (i = 0; i < 3; i++)
+ {
+ char *param[1];
+
+ param[0] = psprintf("%d", 44 + i);
+
+ if (PQsendQueryParams(conn,
+ "SELECT generate_series(42, $1)",
+ 1,
+ NULL,
+ (const char **) param,
+ NULL,
+ NULL,
+ 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ pfree(param[0]);
+ }
+ PQsendPipeline(conn);
+
+ for (i = 0; !pipeline_ended; i++)
+ {
+ bool first = true;
+ bool saw_ending_tuplesok;
+ bool isSingleTuple = false;
+
+ /* Set single row mode for only first 2 SELECT queries */
+ if (i < 2)
+ {
+ if (PQsetSingleRowMode(conn) != 1)
+ pg_fatal("PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+
+ /* Consume rows for this query */
+ saw_ending_tuplesok = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+
+ if (est == PGRES_PIPELINE_SYNC)
+ {
+ fprintf(stderr, "end of pipeline reached\n");
+ pipeline_ended = true;
+ PQclear(res);
+ if (i != 3)
+ pg_fatal("Expected three results, got %d\n", i);
+ break;
+ }
+
+ /* Expect SINGLE_TUPLE for queries 0 and 1, TUPLES_OK for 2 */
+ if (first)
+ {
+ if (i <= 1 && est != PGRES_SINGLE_TUPLE)
+ pg_fatal("Expected PGRES_SINGLE_TUPLE for query %d, got %s\n",
+ i, PQresStatus(est));
+ if (i >= 2 && est != PGRES_TUPLES_OK)
+ pg_fatal("Expected PGRES_TUPLES_OK for query %d, got %s\n",
+ i, PQresStatus(est));
+ first = false;
+ }
+
+ fprintf(stderr, "Result status %s for query %d", PQresStatus(est), i);
+ switch (est)
+ {
+ case PGRES_TUPLES_OK:
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ saw_ending_tuplesok = true;
+ if (isSingleTuple)
+ {
+ if (PQntuples(res) == 0)
+ fprintf(stderr, "all tuples received in query %d\n", i);
+ else
+ pg_fatal("Expected to follow PGRES_SINGLE_TUPLE, "
+ "but received PGRES_TUPLES_OK directly instead\n");
+ }
+ break;
+
+ case PGRES_SINGLE_TUPLE:
+ isSingleTuple = true;
+ fprintf(stderr, ", %d tuple: %s\n", PQntuples(res), PQgetvalue(res, 0, 0));
+ break;
+
+ default:
+ pg_fatal("unexpected\n");
+ }
+ PQclear(res);
+ }
+ if (!pipeline_ended && !saw_ending_tuplesok)
+ pg_fatal("didn't get expected terminating TUPLES_OK\n");
+ }
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s\n", PQerrorMessage(conn));
+}
+
+/*
+ * Simple test to verify that a pipeline is discarded as a whole when there's
+ * an error, ignoring transaction commands.
+ */
+static void
+test_transaction(PGconn *conn)
+{
+ PGresult *res;
+ int num_sends = 0;
+
+ res = PQexec(conn, "DROP TABLE IF EXISTS pq_pipeline_tst;"
+ "CREATE TABLE pq_pipeline_tst (id int)");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to create test table: %s",
+ PQerrorMessage(conn));
+ PQclear(res);
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n",
+ PQerrorMessage(conn));
+ if (PQsendPrepare(conn, "rollback", "ROLLBACK", 0, NULL) != 1)
+ pg_fatal("could not send prepare on pipeline: %s",
+ PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn,
+ "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ if (PQsendQueryParams(conn,
+ "SELECT 0/0",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+
+ /*
+ * send a ROLLBACK using a prepared stmt. Doesn't work because we need to
+ * get out of the pipeline-aborted state first.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /* This insert fails because we're in pipeline-aborted state */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (1)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * This insert fails even though the pipeline got a SYNC, because we're in
+ * an aborted transaction
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (2)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * Send ROLLBACK using prepared stmt. This one works because we just did
+ * PQsendPipeline above.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /*
+ * Now that we're out of a transaction and in pipeline-good mode, this
+ * insert works
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (3)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+ PQsendPipeline(conn);
+ num_sends++;
+
+ for (int i = 0;; i++)
+ {
+ ExecStatusType restype;
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "%d: got NULL result\n", i);
+ continue;
+ }
+ restype = PQresultStatus(res);
+ fprintf(stderr, "%d: got status %s", i, PQresStatus(restype));
+ if (restype == PGRES_FATAL_ERROR)
+ fprintf(stderr, "; error: %s", PQerrorMessage(conn));
+ else if (restype == PGRES_PIPELINE_ABORTED)
+ {
+ fprintf(stderr, ": command didn't run because pipeline aborted\n");
+ }
+ else
+ fprintf(stderr, "\n");
+ PQclear(res);
+
+ if (restype == PGRES_PIPELINE_SYNC)
+ num_sends--;
+ if (num_sends <= 0)
+ break;
+ }
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("returned something extra after all the syncs: %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s\n", PQerrorMessage(conn));
+
+ /* We expect to find one tuple containing the value "3" */
+ res = PQexec(conn, "SELECT * FROM pq_pipeline_tst");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("failed to obtain result: %s", PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("did not get 1 tuple\n");
+ if (strcmp(PQgetvalue(res, 0, 0), "3") != 0)
+ pg_fatal("did not get expected tuple\n");
+ PQclear(res);
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+usage(const char *progname)
+{
+ fprintf(stderr, "%s tests libpq's pipeline mode.\n\n", progname);
+ fprintf(stderr, "Usage:\n");
+ fprintf(stderr, " %s testname [conninfo [number_of_rows]]\n", progname);
+ fprintf(stderr, "Tests:\n");
+ fprintf(stderr, " disallowed_in_pipeline\n");
+ fprintf(stderr, " simple_pipeline\n");
+ fprintf(stderr, " multi_pipeline\n");
+ fprintf(stderr, " pipeline_abort\n");
+ fprintf(stderr, " singlerow\n");
+ fprintf(stderr, " pipeline_insert\n");
+ fprintf(stderr, " transaction\n");
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo = "";
+ PGconn *conn;
+ int numrows = 10000;
+ PGresult *res;
+
+ /*
+ * The testname parameter is mandatory; it can be followed by a conninfo
+ * string and number of rows.
+ */
+ if (argc < 2 || argc > 4)
+ {
+ usage(argv[0]);
+ exit(1);
+ }
+
+ if (argc >= 3)
+ conninfo = pg_strdup(argv[2]);
+
+ if (argc >= 4)
+ {
+ errno = 0;
+ numrows = strtol(argv[3], NULL, 10);
+ if (errno != 0 || numrows <= 0)
+ {
+ fprintf(stderr, "couldn't parse \"%s\" as a positive integer\n", argv[3]);
+ exit(1);
+ }
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+ res = PQexec(conn, "SET lc_messages TO \"C\"");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to set lc_messages: %s", PQerrorMessage(conn));
+
+ if (strcmp(argv[1], "disallowed_in_pipeline") == 0)
+ test_disallowed(conn);
+ else if (strcmp(argv[1], "simple_pipeline") == 0)
+ test_simple_pipeline(conn);
+ else if (strcmp(argv[1], "multi_pipeline") == 0)
+ test_multi_pipelines(conn);
+ else if (strcmp(argv[1], "pipeline_abort") == 0)
+ test_aborted_pipeline(conn);
+ else if (strcmp(argv[1], "pipeline_insert") == 0)
+ test_pipelined_insert(conn, numrows);
+ else if (strcmp(argv[1], "singlerow") == 0)
+ test_singlerowmode(conn);
+ else if (strcmp(argv[1], "transaction") == 0)
+ test_transaction(conn);
+ else
+ {
+ fprintf(stderr, "\"%s\" is not a recognized test name\n", argv[1]);
+ usage(argv[0]);
+ exit(1);
+ }
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+ return 0;
+}
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000000..c7d4b1feec
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,31 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 7;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $numrows = 10000;
+my @tests =
+ qw(disallowed_in_pipeline
+ simple_pipeline
+ multi_pipeline
+ pipeline_abort
+ pipeline_insert
+ singlerow
+ transaction);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'pipeline', $testname, $node->connstr('postgres'), $numrows ],
+ "pipeline $testname");
+}
+
+$node->stop('fast');
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 49614106dc..b2546f0809 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -33,7 +33,7 @@ my @unlink_on_exit;
# Set of variables for modules in contrib/ and src/test/modules/
my $contrib_defines = { 'refint' => 'REFINT_VERBOSE' };
-my @contrib_uselibpq = ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo');
+my @contrib_uselibpq = ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo', 'test_libpq');
my @contrib_uselibpgport = ('oid2name', 'vacuumlo');
my @contrib_uselibpgcommon = ('oid2name', 'vacuumlo');
my $contrib_extralibs = undef;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 8bd95aefa1..1bd23eec6d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1561,10 +1561,12 @@ PG_Locale_Strategy
PG_Lock_Status
PG_init_t
PGcancel
+PGcommandQueueEntry
PGconn
PGdataValue
PGlobjfuncs
PGnotify
+PGpipelineStatus
PGresAttDesc
PGresAttValue
PGresParamDesc
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
On 2021-Mar-04, Alvaro Herrera wrote:
v30 contains changes to hopefully make it build on MSVC.
Hm, that didn't work -- appveyor still says:
Project "C:\projects\postgresql\pgsql.sln" (1) is building "C:\projects\postgresql\pipeline.vcxproj" (75) on node 1 (default targets).
PrepareForBuild:
Creating directory ".\Release\pipeline\".
Creating directory ".\Release\pipeline\pipeline.tlog\".
InitializeBuildStatus:
Creating ".\Release\pipeline\pipeline.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified.
ClCompile:
C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\x86_amd64\CL.exe /c /Isrc/include /Isrc/include/port/win32 /Isrc/include/port/win32_msvc /Zi /nologo /W3 /WX- /Ox /D WIN32 /D _WINDOWS /D __WINDOWS__ /D __WIN32__ /D WIN32_STACK_RLIMIT=4194304 /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _MBCS /GF /Gm- /EHsc /MD /GS /fp:precise /Zc:wchar_t /Zc:forScope /Fo".\Release\pipeline\\" /Fd".\Release\pipeline\vc120.pdb" /Gd /TC /wd4018 /wd4244 /wd4273 /wd4102 /wd4090 /wd4267 /errorReport:queue /MP src/test/modules/test_libpq/pipeline.c
pipeline.c
src/test/modules/test_libpq/pipeline.c(11): fatal error C1083: Cannot open include file: 'libpq-fe.h': No such file or directory [C:\projects\postgresql\pipeline.vcxproj]
Done Building Project "C:\projects\postgresql\pipeline.vcxproj" (default targets) -- FAILED.
Project "C:\projects\postgresql\pgsql.sln" (1) is building "C:\projects\postgresql\test_parser.vcxproj" (76) on node 1 (default targets).
I think the problem is that the project is called pipeline and not test_libpq,
so there's no match in the name. I'm going to rename the whole thing to
src/test/modules/libpq_pipeline/ and see if the msvc tooling likes that
better.
--
�lvaro Herrera 39�49'30"S 73�17'W
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
On 2021-Mar-04, Alvaro Herrera wrote:
I think the problem is that the project is called pipeline and not
test_libpq, so there's no match in the name. I'm going to rename the
whole thing to src/test/modules/libpq_pipeline/ and see if the msvc
tooling likes that better.
v31.
--
�lvaro Herrera Valdivia, Chile
"I'm impressed how quickly you are fixing this obscure issue. I came from
MS SQL and it would be hard for me to put into words how much of a better job
you all are doing on [PostgreSQL]."
Steve Midgley, http://archives.postgresql.org/pgsql-sql/2008-08/msg00000.php
Attachments:
v31-libpq-pipeline.patchtext/x-diff; charset=us-asciiDownload
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 0553279314..c87b0ce911 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -3173,6 +3173,33 @@ ExecStatusType PQresultStatus(const PGresult *res);
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-sync">
+ <term><literal>PGRES_PIPELINE_SYNC</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a
+ synchronization point in pipeline mode, requested by
+ <xref linkend="libpq-PQsendPipeline"/>.
+ This status occurs only when pipeline mode has been selected.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-aborted">
+ <term><literal>PGRES_PIPELINE_ABORTED</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a pipeline that has
+ received an error from the server. <function>PQgetResult</function>
+ must be called repeatedly, and each time it will return this status code
+ until the end of the current pipeline, at which point it will return
+ <literal>PGRES_PIPELINE_SYNC</literal> and normal processing can
+ resume.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
If the result status is <literal>PGRES_TUPLES_OK</literal> or
@@ -4919,6 +4946,498 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-pipeline-mode">
+ <title>Pipeline Mode</title>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>libpq</primary>
+ <secondary>pipeline mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>pipelining</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>batch mode</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> pipeline mode allows applications to
+ send a query without having to read the result of the previously
+ sent query. Taking advantage of the pipeline mode, a client will wait
+ less for the server, since multiple queries/results can be sent/
+ received in a single network transaction.
+ </para>
+
+ <para>
+ While pipeline mode provides a significant performance boost, writing
+ clients using the pipeline mode is more complex because it involves
+ managing a queue of pending queries and finding which result
+ corresponds to which query in the queue.
+ </para>
+
+ <para>
+ Pipeline mode also generally consumes more memory on both the client and server,
+ though careful and aggressive management of the send/receive queue can mitigate
+ this. This applies whether or not the connection is in blocking or non-blocking
+ mode.
+ </para>
+
+ <sect2 id="libpq-pipeline-using">
+ <title>Using Pipeline Mode</title>
+
+ <para>
+ To issue pipelines, the application must switch a connection into pipeline mode.
+ Enter pipeline mode with <xref linkend="libpq-PQenterPipelineMode"/>
+ or test whether pipeline mode is active with
+ <xref linkend="libpq-PQpipelineStatus"/>.
+ In pipeline mode, only <link linkend="libpq-async">asynchronous operations</link>
+ are permitted, and <literal>COPY</literal> is disallowed.
+ Using any synchronous command execution functions,
+ such as <function>PQfn</function>, or <function>PQexec</function>
+ and its sibling functions, is an error condition.
+ </para>
+
+ <note>
+ <para>
+ It is best to use pipeline mode with <application>libpq</application> in
+ <link linkend="libpq-PQsetnonblocking">non-blocking mode</link>. If used
+ in blocking mode it is possible for a client/server deadlock to occur.
+ <footnote>
+ <para>
+ The client will block trying to send queries to the server, but the
+ server will block trying to send results to the client from queries
+ it has already processed. This only occurs when the client sends
+ enough queries to fill both its output buffer and the server's receive
+ buffer before it switches to processing input from the server,
+ but it's hard to predict exactly when that will happen.
+ </para>
+ </footnote>
+ </para>
+ </note>
+
+ <sect3 id="libpq-pipeline-sending">
+ <title>Issuing Queries</title>
+
+ <para>
+ After entering pipeline mode, the application dispatches requests using
+ <xref linkend="libpq-PQsendQueryParams"/>,
+ or its prepared-query sibling
+ <xref linkend="libpq-PQsendQueryPrepared"/>.
+ These requests are queued on the client-side until flushed to the server;
+ this occurs when <xref linkend="libpq-PQsendPipeline"/> is used to
+ establish a synchronization point in the pipeline,
+ or when <xref linkend="libpq-PQflush"/> is called.
+ The functions <xref linkend="libpq-PQsendPrepare"/>,
+ <xref linkend="libpq-PQsendDescribePrepared"/>, and
+ <xref linkend="libpq-PQsendDescribePortal"/> also work in pipeline mode.
+ Result processing is described below.
+ </para>
+
+ <para>
+ The server executes statements, and returns results, in the order the
+ client sends them. The server will begin executing the commands in the
+ pipeline immediately, not waiting for the end of the pipeline.
+ If any statement encounters an error, the server aborts the current
+ transaction and skips processing commands in the pipeline until the
+ next synchronization point established by <function>PQsendPipeline</function>.
+ (This remains true even if the commands in the pipeline would rollback
+ the transaction.)
+ Query processing resumes after the synchronization point.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one; for example, one query may define a table that the next
+ query in the same pipeline uses. Similarly, an application may
+ create a named prepared statement and execute it with later
+ statements in the same pipeline.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-results">
+ <title>Processing Results</title>
+
+ <para>
+ To process the result of one query in a pipeline, the application calls
+ <function>PQgetResult</function> repeatedly and handles each result
+ until <function>PQgetResult</function> returns null.
+ The result from the next query in the pipeline may then be retrieved using
+ <function>PQgetResult</function> again and the cycle repeated.
+ The application handles individual statement results as normal.
+ When the results of all the queries in the pipeline have been
+ returned, <function>PQgetResult</function> returns a result
+ containing the status value <literal>PGRES_PIPELINE_SYNC</literal>.
+ </para>
+
+ <para>
+ The client may choose to defer result processing until the complete
+ pipeline has been sent, or interleave that with sending further
+ queries in the pipeline; see <xref linkend="libpq-pipeline-interleave" />.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ This mode selection is effective only for the query currently
+ being processed. For more information on the use of
+ <function>PQsetSingleRowMode</function>,
+ refer to <xref linkend="libpq-single-row-mode"/>.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal
+ asynchronous processing except that it may contain the new
+ <type>PGresult</type> types <literal>PGRES_PIPELINE_SYNC</literal>
+ and <literal>PGRES_PIPELINE_ABORTED</literal>.
+ <literal>PGRES_PIPELINE_SYNC</literal> is reported exactly once for each
+ <function>PQsendPipeline</function> after retrieving results for all
+ queries in the pipeline.
+ <literal>PGRES_PIPELINE_ABORTED</literal> is emitted in place of a normal
+ stream result for the first error and all subsequent results
+ except <literal>PGRES_PIPELINE_SYNC</literal> and null;
+ see <xref linkend="libpq-pipeline-errors"/>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing pipeline results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed (except that
+ <function>PQgetResult</function> returns null to indicate that we start
+ returning the results of next query). The application must keep track
+ of the order in which it sent queries, to associate them with their
+ corresponding results.
+ Applications will typically use a state machine or a FIFO queue for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-pipeline-errors">
+ <title>Error Handling</title>
+
+ <para>
+ When a query in a pipeline causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the pipeline
+ synchronization message. The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after <function>PQresultStatus</function>
+ returns <literal>PGRES_FATAL_ERROR</literal>,
+ the pipeline is flagged as aborted.
+ <function>PQresultStatus</function>, will report a
+ <literal>PGRES_PIPELINE_ABORTED</literal> result for each remaining queued
+ operation in an aborted pipeline. The result for
+ <function>PQsendPipeline</function> is reported as
+ <literal>PGRES_PIPELINE_SYNC</literal> to signal the end of the aborted pipeline
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the pipeline used an implicit transaction, then operations that have
+ already executed are rolled back and operations that were queued to follow
+ the failed operation are skipped entirely. The same behaviour holds if the
+ pipeline starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the pipeline. If a pipeline contains
+ <emphasis>multiple explicit transactions</emphasis>, all transactions that
+ committed prior to the error remain committed, the currently in-progress
+ transaction is aborted, and all subsequent operations are skipped completely,
+ including subsequent transactions.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal> — only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously, the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-interleave">
+ <title>Interleaving Result Processing and Query Dispatch</title>
+
+ <para>
+ To avoid deadlocks on large pipelines the client should be structured
+ around a non-blocking event loop using operating system facilities
+ such as <function>select</function>, <function>poll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work
+ remaining to be dispatched and a queue of work that has been dispatched
+ but not yet had its results processed. When the socket is writable
+ it should dispatch more work. When the socket is readable it should
+ read results and process them, matching them up to the next entry in
+ its expected results queue. Based on available memory, results from the
+ socket should be read frequently: there's no need to wait until the
+ pipeline end to read the results. Pipelines should be scoped to logical
+ units of work, usually (but not necessarily) one transaction per pipeline.
+ There's no need to exit pipeline mode and re-enter it between pipelines,
+ or to wait for one pipeline to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state
+ machine to track sent and received work is in
+ <filename>src/test/modules/libpq_pipeline/libpq_pipeline.c</filename>
+ in the PostgreSQL source distribution.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-end">
+ <title>Ending Pipeline Mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed, and
+ the end pipeline result has been consumed, the application may return
+ to non-pipelined mode with <xref linkend="libpq-PQexitPipelineMode"/>.
+ </para>
+ </sect3>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-functions">
+ <title>Functions Associated with Pipeline Mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQpipelineStatus">
+ <term>
+ <function>PQpipelineStatus</function>
+ <indexterm>
+ <primary>PQpipelineStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the current pipeline mode status of the
+ <application>libpq</application> connection.
+<synopsis>
+PGpipelineStatus PQpipelineStatus(const PGconn *conn);
+</synopsis>
+ </para>
+
+ <para>
+ <function>PQpipelineStatus</function> can return one of the following values:
+
+ <variablelist>
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ON</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in
+ pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_OFF</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is
+ <emphasis>not</emphasis> in pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in pipeline
+ mode and an error occurred while processing the current pipeline.
+ The aborted flag is cleared when <function>PQresultStatus</function>
+ returns PGRES_PIPELINE_SYNC at the end of the pipeline.
+ Clients don't usually need this function to
+ verify aborted status, as they can tell that the pipeline is aborted
+ from the <literal>PGRES_PIPELINE_ABORTED</literal> result code.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterPipelineMode">
+ <term>
+ <function>PQenterPipelineMode</function>
+ <indexterm>
+ <primary>PQenterPipelineMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter pipeline mode if it is currently idle or
+ already in pipeline mode.
+
+<synopsis>
+int PQenterPipelineMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success.
+ Returns 0 and has no effect if the connection is not currently
+ idle, i.e., it has a result ready, or it is waiting for more
+ input from the server, etc.
+ This function does not actually send anything to the server,
+ it just changes the <application>libpq</application> connection
+ state.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitPipelineMode">
+ <term>
+ <function>PQexitPipelineMode</function>
+ <indexterm>
+ <primary>PQexitPipelineMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit pipeline mode if it is currently in pipeline mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitPipelineMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 1 and takes no action if not in
+ pipeline mode. If the current statement isn't finished processing
+ or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQsendPipeline">
+ <term>
+ <function>PQsendPipeline</function>
+ <indexterm>
+ <primary>PQsendPipeline</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Marks a synchronization point in a pipeline by sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ and flushing the send buffer. This serves as
+ the delimiter of an implicit transaction and an error recovery
+ point; see <xref linkend="libpq-pipeline-errors"/>.
+
+<synopsis>
+int PQsendPipeline(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 if the connection is not in
+ pipeline mode or sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ failed.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-tips">
+ <title>When to Use Pipeline Mode</title>
+
+ <para>
+ Much like asynchronous query mode, there is no meaningful performance
+ overhead when using pipeline mode. It increases client application complexity,
+ and extra caution is required to prevent client/server deadlocks, but
+ pipeline mode can offer considerable performance improvements, in exchange for
+ increased memory usage from leaving state around longer.
+ </para>
+
+ <para>
+ Pipeline mode is most useful when the server is distant, i.e., network latency
+ (<quote>ping time</quote>) is high, and also when many small operations
+ are being performed in rapid succession. There is usually less benefit
+ in using pipelined commands when each query takes many multiples of the client/server
+ round-trip time to execute. A 100-statement operation run on a server
+ 300ms round-trip-time away would take 30 seconds in network latency alone
+ without pipelining; with pipelining it may spend as little as 0.3s waiting for
+ results from the server.
+ </para>
+
+ <para>
+ Use pipelined commands when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed
+ into operations on sets, or into a <literal>COPY</literal> operation.
+ </para>
+
+ <para>
+ Pipeline mode is not useful when information from one operation is required by
+ the client to produce the next operation. In such cases, the client
+ would have to introduce a synchronization point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+BEGIN;
+SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+-- result: x=2
+-- client adds 1 to x:
+UPDATE mytable SET x = 3 WHERE id = 42;
+COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <para>
+ Pipelining is less useful, and more complex, when a single pipeline contains
+ multiple transactions (see <xref linkend="libpq-pipeline-errors"/>).
+ </para>
+
+ <note>
+ <para>
+ The pipeline API was introduced in <productname>PostgreSQL</productname> 14.
+ Pipeline mode is a client-side feature which doesn't require server
+ support, and works on any server that supports the v3 extended query
+ protocol.
+ </para>
+ </note>
+ </sect2>
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4959,6 +5478,13 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <para>
+ When using pipeline mode, single-row mode needs to be activated for each
+ query in the pipeline before retrieving results for that query
+ with <function>PQgetResult</function>.
+ See <xref linkend="libpq-pipeline-mode"/> for more information.
+ </para>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 6d46da42e2..012e44c736 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while a libpq connection is in pipeline mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 2ec0580a79..75448162ca 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -1109,6 +1109,12 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
row, the last value is kept.
</para>
+ <para>
+ <literal>\gset</literal> and <literal>\aset</literal> cannot be used
+ pipeline mode, since query results are not immediately
+ fetched in this mode.
+ </para>
+
<para>
The following example puts the final account balance from the first query
into variable <replaceable>abalance</replaceable>, and fills variables
@@ -1269,6 +1275,21 @@ SELECT 4 AS four \; SELECT 5 AS five \aset
</programlisting></para>
</listitem>
</varlistentry>
+
+ <varlistentry id='pgbench-metacommand-pipeline'>
+ <term><literal>\startpipeline</literal></term>
+ <term><literal>\endpipeline</literal></term>
+
+ <listitem>
+ <para>
+ These commands delimit the start and end of a pipeline of SQL statements.
+ In a pipeline, statements are sent to server without waiting for the results
+ of previous statements (see <xref linkend="libpq-pipeline-mode"/>).
+ Pipeline mode requires the extended query protocol.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect2>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 5272eed9ab..f74378110a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1019,6 +1019,12 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->err = _("empty query");
break;
+ case PGRES_PIPELINE_SYNC:
+ case PGRES_PIPELINE_ABORTED:
+ walres->status = WALRCV_ERROR;
+ walres->err = _("unexpected pipeline mode");
+ break;
+
case PGRES_NONFATAL_ERROR:
case PGRES_FATAL_ERROR:
case PGRES_BAD_RESPONSE:
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 31a4df45f5..fbbe270654 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -351,10 +351,11 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
- * and we enter the CSTATE_SLEEP state to wait for it to expire. Other
- * meta-commands are executed immediately. If the command about to start
- * is actually beyond the end of the script, advance to CSTATE_END_TX.
+ * CSTATE_WAIT_RESULT state unless in pipeline mode. On a \sleep
+ * meta-command, the timer is set, and we enter the CSTATE_SLEEP state to
+ * wait for it to expire. Other meta-commands are executed immediately. If
+ * the command about to start is actually beyond the end of the script,
+ * advance to CSTATE_END_TX.
*
* CSTATE_WAIT_RESULT waits until we get a result set back from the server
* for the current command.
@@ -484,7 +485,9 @@ typedef enum MetaCommand
META_IF, /* \if */
META_ELIF, /* \elif */
META_ELSE, /* \else */
- META_ENDIF /* \endif */
+ META_ENDIF, /* \endif */
+ META_STARTPIPELINE, /* \startpipeline */
+ META_ENDPIPELINE /* \endpipeline */
} MetaCommand;
typedef enum QueryMode
@@ -2504,6 +2507,10 @@ getMetaCommand(const char *cmd)
mc = META_GSET;
else if (pg_strcasecmp(cmd, "aset") == 0)
mc = META_ASET;
+ else if (pg_strcasecmp(cmd, "startpipeline") == 0)
+ mc = META_STARTPIPELINE;
+ else if (pg_strcasecmp(cmd, "endpipeline") == 0)
+ mc = META_ENDPIPELINE;
else
mc = META_NONE;
return mc;
@@ -2693,11 +2700,25 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In pipeline mode, we use asynchronous functions. If a
+ * server-side error occurs, it will be processed later
+ * among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ pg_log_error("%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2741,8 +2762,10 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
* varprefix should be set only with \gset or \aset, and SQL commands do
* not need it.
*/
+#if 0
Assert((meta == META_NONE && varprefix == NULL) ||
((meta == META_GSET || meta == META_ASET) && varprefix != NULL));
+#endif
res = PQgetResult(st->con);
@@ -2810,6 +2833,12 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* otherwise the result is simply thrown away by PQclear below */
break;
+ case PGRES_PIPELINE_SYNC:
+ pg_log_debug("client %d pipeline ending", st->id);
+ if (PQexitPipelineMode(st->con) != 1)
+ pg_log_error("client %d failed to exit pipeline mode", st->id);
+ break;
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3066,13 +3095,36 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
/* Execute the command */
if (command->type == SQL_COMMAND)
{
+ /* disallow \aset and \gset in pipeline mode */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ if (command->meta == META_GSET)
+ {
+ commandFailed(st, "gset", "\\gset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ else if (command->meta == META_ASET)
+ {
+ commandFailed(st, "aset", "\\aset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ }
+
if (!sendCommand(st, command))
{
commandFailed(st, "SQL", "SQL command send failed");
st->state = CSTATE_ABORTED;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in pipeline mode */
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -3212,7 +3264,15 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (readCommandResponse(st,
sql_script[st->use_file].commands[st->command]->meta,
sql_script[st->use_file].commands[st->command]->varprefix))
- st->state = CSTATE_END_COMMAND;
+ {
+ /*
+ * outside of pipeline mode: stop reading results.
+ * pipeline mode: continue reading results until an
+ * end-of-pipeline response.
+ */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ st->state = CSTATE_END_COMMAND;
+ }
else
st->state = CSTATE_ABORTED;
break;
@@ -3456,6 +3516,49 @@ executeMetaCommand(CState *st, instr_time *now)
return CSTATE_ABORTED;
}
}
+ else if (command->meta == META_STARTPIPELINE)
+ {
+ /*
+ * In pipeline mode, we use a workflow based on libpq pipeline
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "startpipeline", "cannot use pipeline mode with the simple query protocol");
+ return CSTATE_ABORTED;
+ }
+
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ commandFailed(st, "startpipeline", "already in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (PQenterPipelineMode(st->con) == 0)
+ {
+ commandFailed(st, "startpipeline", "failed to enter pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ }
+ else if (command->meta == META_ENDPIPELINE)
+ {
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ {
+ commandFailed(st, "endpipeline", "not in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (!PQsendPipeline(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to send the pipeline");
+ return CSTATE_ABORTED;
+ }
+ if (!PQexitPipelineMode(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to exit pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ /* collect pending results before getting out of pipeline mode */
+ return CSTATE_WAIT_RESULT;
+ }
/*
* executing the expression or shell command might have taken a
@@ -4683,7 +4786,9 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
- else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
+ else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF ||
+ my_command->meta == META_STARTPIPELINE ||
+ my_command->meta == META_ENDPIPELINE)
{
if (my_command->argc != 1)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
@@ -6808,4 +6913,5 @@ pthread_join(pthread_t th, void **thread_return)
return 0;
}
+
#endif /* WIN32 */
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..60d09e6d63 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,7 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterPipelineMode 180
+PQexitPipelineMode 181
+PQsendPipeline 182
+PQpipelineStatus 183
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index f83af03d0a..4c97e8a078 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -540,6 +540,23 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * pqFreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+pqFreeCommandQueue(PGcommandQueueEntry *queue)
+{
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *cur = queue;
+
+ queue = cur->next;
+ if (cur->query)
+ free(cur->query);
+ free(cur);
+ }
+}
/*
* pqDropServerData
@@ -571,6 +588,12 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ pqFreeCommandQueue(conn->cmd_queue_head);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ pqFreeCommandQueue(conn->cmd_queue_recycle);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
@@ -2486,6 +2509,7 @@ keep_going: /* We will come back to here until there is
/* Drop any PGresult we might have, too */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn);
/* Reset conn->status to put the state machine in the right state */
@@ -3928,6 +3952,7 @@ makeEmptyPGconn(void)
conn->status = CONNECTION_BAD;
conn->asyncStatus = PGASYNC_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
conn->xactStatus = PQTRANS_IDLE;
conn->options_valid = false;
conn->nonblocking = false;
@@ -4187,6 +4212,7 @@ closePGconn(PGconn *conn)
conn->status = CONNECTION_BAD; /* Well, not really _bad_ - just absent */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn); /* deallocate result */
resetPQExpBuffer(&conn->errorMessage);
release_conn_addrinfo(conn);
@@ -6735,6 +6761,15 @@ PQbackendPID(const PGconn *conn)
return conn->be_pid;
}
+PGpipelineStatus
+PQpipelineStatus(const PGconn *conn)
+{
+ if (!conn)
+ return PQ_PIPELINE_OFF;
+
+ return conn->pipelineStatus;
+}
+
int
PQconnectionNeedsPassword(const PGconn *conn)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9a038043b2..0fb790fdfa 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_PIPELINE_SYNC",
+ "PGRES_PIPELINE_ABORTED"
};
/*
@@ -71,6 +73,11 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *pqMakePipelineCmd(PGconn *conn);
+static void pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqPipelineProcessQueue(PGconn *conn);
+static int pqPipelineFlush(PGconn *conn);
/* ----------------
@@ -1171,7 +1178,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1197,18 +1204,37 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot use %s in pipeline mode, use PQsendQueryParams\n"),
+ "PQsendQuery");
+ return 0;
+ }
+
return PQsendQueryInternal(conn, query, true);
}
int
PQsendQueryContinue(PGconn *conn, const char *query)
{
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot use %s in pipeline mode, use PQsendQueryParams\n"),
+ "PQsendQueryContinue");
+ return 0;
+ }
+
return PQsendQueryInternal(conn, query, false);
}
static int
PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
{
+ /* pipeline mode requires extended query protocol */
+ Assert(conn->pipelineStatus == PQ_PIPELINE_OFF);
+
if (!PQsendQueryStart(conn, newQuery))
return 0;
@@ -1307,6 +1333,8 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+
if (!PQsendQueryStart(conn, true))
return 0;
@@ -1330,6 +1358,15 @@ PQsendPrepare(PGconn *conn,
return 0;
}
+ /* Alloc pipeline memory before doing anything */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+ }
+
/* construct the Parse message */
if (pqPutMsgStart('P', conn) < 0 ||
pqPuts(stmtName, conn) < 0 ||
@@ -1356,32 +1393,46 @@ PQsendPrepare(PGconn *conn,
if (pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /*
+ * In non-pipeline mode, add a Sync and prepare to send. In pipeline mode
+ * we just keep track of the new message.
+ */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ /* remember we are doing just a Parse */
+ conn->queryclass = PGQUERY_PREPARE;
- /* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
-
- /* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ /* and remember the query text too, if possible */
+ /* if insufficient memory, last_query just winds up NULL */
+ if (conn->last_query)
+ free(conn->last_query);
+ conn->last_query = strdup(query);
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ else
+ {
+ pipeCmd->queryclass = PGQUERY_PREPARE;
+ /* as above, if insufficient memory, query winds up NULL */
+ pipeCmd->query = strdup(query);
+ pqAppendPipelineCmd(conn, pipeCmd);
+ }
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
- /* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1429,7 +1480,8 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn, bool newQuery)
@@ -1450,20 +1502,57 @@ PQsendQueryStart(PGconn *conn, bool newQuery)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE &&
+ conn->pipelineStatus == PQ_PIPELINE_OFF)
{
appendPQExpBufferStr(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * When enqueuing commands we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when pqPipelineProcessQueue()
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ }
+ }
+ else
+ {
+ /*
+ * This command's results will come in immediately. Initialize async
+ * result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1487,10 +1576,34 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **query;
+ PGQueryClass *queryclass;
/*
- * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
- * using specified statement name and the unnamed portal.
+ * Decide where the query is going to be stored. In pipeline mode, we
+ * allocate a new pipeline element; in non-pipeline mode, it's simply the
+ * connection's last query.
+ */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+ /*
+ * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync
+ * (if not in pipeline mode), using specified statement name and the
+ * unnamed portal.
*/
if (command)
@@ -1600,35 +1713,43 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /* construct the Sync message if not in pipeline mode */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ /* if insufficient memory, query just winds up NULL */
+ if (*query)
+ free(*query);
if (command)
- conn->last_query = strdup(command);
+ *query = strdup(command);
else
- conn->last_query = NULL;
+ *query = NULL;
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1726,14 +1847,17 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
-
/*
* PQgetResult
* Get the next PGresult produced by a query. Returns NULL if no
* query work remains or an error has occurred (e.g. out of
* memory).
+ *
+ * In pipeline mode, once all the result of a query have been returned,
+ * PQgetResult returns NULL to let the user know that the next
+ * query is being processed. At the end of the pipeline, returns a
+ * result with PQresultStatus(result) == PGRES_PIPELINE_SYNC.
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1803,8 +1927,49 @@ PQgetResult(PGconn *conn)
{
case PGASYNC_IDLE:
res = NULL; /* query is complete */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to return the NULL that terminates the round of
+ * results from the current query; prepare to send the results
+ * of the next query when we're called next. Also, since this
+ * is the start of the results of the next query, clear any
+ * prior error message.
+ */
+ resetPQExpBuffer(&conn->errorMessage);
+ pqPipelineProcessQueue(conn);
+ }
break;
case PGASYNC_READY:
+ res = pqPrepareAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to send the results of the current query. Set
+ * us idle now, and ...
+ */
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /*
+ * ... in cases when we're sending a pipeline-related result,
+ * move queue processing forwards immediately, so that next
+ * time we're called, we're prepared to return the next result
+ * received from the server.
+ *
+ * In all other cases, leave the queue state change for next
+ * time, so that a terminating NULL result is sent.
+ */
+ if (res && (res->resultStatus == PGRES_PIPELINE_ABORTED ||
+ res->resultStatus == PGRES_PIPELINE_SYNC))
+ pqPipelineProcessQueue(conn);
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -1985,6 +2150,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("synchronous command execution functions are not allowed in pipeline mode\n"));
+ return false;
+ }
+
/*
* Since this is the beginning of a query cycle, reset the error buffer.
*/
@@ -2148,6 +2320,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2155,6 +2330,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
if (!PQsendQueryStart(conn, true))
return 0;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ queryclass = &conn->queryclass;
+
/* construct the Describe message */
if (pqPutMsgStart('D', conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2163,32 +2350,40 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ /* reset last-query string (not relevant now) */
+ if (conn->last_query && conn->pipelineStatus != PQ_PIPELINE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
}
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -2541,6 +2736,13 @@ PQfn(PGconn *conn,
*/
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("PQfn not allowed in pipeline mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -2555,6 +2757,359 @@ PQfn(PGconn *conn,
args, nargs);
}
+/* ====== Pipeline mode support ======== */
+
+/*
+ * PQenterPipelineMode
+ * Put an idle connection in pipeline mode.
+ *
+ * Returns 1 on success. On failure, errorMessage is set and 0 is returned.
+ *
+ * Commands submitted after this can be pipelined on the connection;
+ * there's no requirement to wait for one to finish before the next is
+ * dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQsendPipeline. Multiple pipelines
+ * can be sent while in pipeline mode. Pipeline mode can be exited
+ * by calling PQexitPipelineMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ /* succeed with no action if already in pipeline mode */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot enter pipeline mode, connection not idle\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+
+ return 1;
+}
+
+/*
+ * PQexitPipelineMode
+ * End pipeline mode and return to normal command mode.
+ *
+ * Returns 1 in success (pipeline mode successfully ended, or not in pipeline
+ * mode).
+ *
+ * Returns 0 if in pipeline mode and cannot be ended yet. Error message will
+ * be set.
+ */
+int
+PQexitPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ /* there are some uncollected results */
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+
+ case PGASYNC_BUSY:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode while busy\n"));
+ return 0;
+
+ default:
+ /* OK */
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ return 0; /* error message is setup already */
+ return 1;
+}
+
+/*
+ * pqPipelineProcessQueue: subroutine for PQgetResult
+ * In pipeline mode, start processing the results of the next query in the queue.
+ */
+static void
+pqPipelineProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ Assert(conn->pipelineStatus != PQ_PIPELINE_OFF);
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: COPY in pipeline mode\n");
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return;
+ case PGASYNC_IDLE:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In pipeline mode but nothing left on the queue; caller can submit
+ * more work or PQexitPipelineMode() now.
+ */
+ return;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-pipeline call.
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /*
+ * Reset single-row processing mode. (Client has to set it up for each
+ * query, if desired.)
+ */
+ conn->singleRowMode = false;
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ pqRecyclePipelineCmd(conn, next_query);
+
+ if (conn->pipelineStatus == PQ_PIPELINE_ABORTED &&
+ conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted pipeline we don't get anything from the server for
+ * each result; we're just discarding input until we get to the next
+ * sync from the server. The client needs to know its queries got
+ * aborted so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn, PGRES_PIPELINE_ABORTED);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ pqSaveErrorResult(conn);
+ return;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+}
+
+/*
+ * PQsendPipeline
+ * Send a Sync message as part of a pipeline, and flush to server
+ *
+ * It's legal to start submitting more commands in the pipeline immediately,
+ * without waiting for the results of the current pipeline. There's no need to
+ * end pipeline mode and start it again.
+ *
+ * If a command in a pipeline fails, every subsequent command up to and including
+ * the result to the Sync message sent by PQsendPipeline gets set to
+ * PGRES_PIPELINE_ABORTED state. If the whole pipeline is processed without
+ * error, a PGresult with PGRES_PIPELINE_SYNC is produced.
+ *
+ * Queries can already have been sent before PQsendPipeline is called, but
+ * PQsendPipeline need to be called before retrieving command results.
+ *
+ * The connection will remain in pipeline mode and unavailable for new
+ * synchronous command execution functions until all results from the pipeline
+ * are processed by the client.
+ */
+int
+PQsendPipeline(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot send pipeline when not in pipeline mode\n"));
+ return 0;
+ }
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: cannot send pipeline while in COPY\n");
+ return 0;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_IDLE:
+ /* OK to send sync */
+ break;
+ }
+
+ entry = pqMakePipelineCmd(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ pqAppendPipelineCmd(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ /*
+ * Call pqPipelineProcessQueue so the user can call start calling
+ * PQgetResult.
+ */
+ pqPipelineProcessQueue(conn);
+
+ return 1;
+
+sendFailed:
+ pqRecyclePipelineCmd(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
+
+/*
+ * pqMakePipelineCmd
+ * Get a command queue entry for caller to fill.
+ *
+ * If the recycle queue has a free element, that is returned; if not, a
+ * fresh one is allocated. Caller is responsible for adding it to the
+ * command queue (pqAppendPipelineCmd) once the struct is filled in, or
+ * releasing the memory (pqRecyclePipelineCmd) if an error occurs.
+ *
+ * If allocation fails, sets the error message and returns NULL.
+ */
+static PGcommandQueueEntry *
+pqMakePipelineCmd(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * pqAppendPipelineCmd
+ * Append a caller-allocated command queue entry to the queue.
+ *
+ * The query itself must already have been put in the output buffer by the
+ * caller.
+ */
+static void
+pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ PGcommandQueueEntry **tail;
+
+ if (conn->cmd_queue_head == NULL)
+ tail = &conn->cmd_queue_head;
+ else
+ tail = &conn->cmd_queue_tail->next;
+
+ *tail = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * pqRecyclePipelineCmd
+ * Push a command queue entry onto the freelist. It must be an entry
+ * with null next pointer and not referenced by any other entry's next
+ * pointer.
+ */
+static void
+pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+
+ Assert(entry->next == NULL);
+
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
/* ====== accessor funcs for PGresult ======== */
@@ -3152,6 +3707,23 @@ PQflush(PGconn *conn)
return pqFlush(conn);
}
+/*
+ * pqPipelineFlush
+ *
+ * In pipeline mode, data will be flushed only when the out buffer reaches the
+ * threshold value. In non-pipeline mode, it behaves as stock pqFlush.
+ *
+ * Returns 0 on success.
+ */
+static int
+pqPipelineFlush(PGconn *conn)
+{
+ if ((conn->pipelineStatus == PQ_PIPELINE_OFF) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return pqFlush(conn);
+ return 0;
+}
+
/*
* PQfreemem - safely frees memory allocated
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 2ca8c057b9..b131995974 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -177,14 +177,24 @@ pqParseInput3(PGconn *conn)
if (getParameterStatus(conn))
return;
}
- else
- {
- pqInternalNotice(&conn->noticeHooks,
- "message type 0x%02x arrived from server while idle",
- id);
- /* Discard the unexpected message */
- conn->inCursor += msgLength;
- }
+
+ /*
+ * We're also IDLE when in pipeline mode we have completed
+ * processing the results of one query and are waiting for the
+ * next one in the pipeline. In this case, as above, just wait to
+ * see what's next.
+ */
+ if (conn->asyncStatus == PGASYNC_IDLE &&
+ conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->cmd_queue_head != NULL)
+ return;
+
+ /* Any other case is unexpected and we summarily skip it */
+ pqInternalNotice(&conn->noticeHooks,
+ "message type 0x%02x arrived from server while idle",
+ id);
+ /* Discard the unexpected message */
+ conn->inCursor += msgLength;
}
else
{
@@ -217,10 +227,28 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new
+ * query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_PIPELINE_SYNC);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ else
+ {
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -450,7 +478,7 @@ handleSyncLoss(PGconn *conn, char id, int msgLength)
id, msgLength);
/* build an error result holding the error message */
pqSaveErrorResult(conn);
- conn->asyncStatus = PGASYNC_READY; /* drop out of GetResult wait loop */
+ conn->asyncStatus = PGASYNC_READY; /* drop out of PQgetResult wait loop */
/* flush input data since we're giving up on processing it */
pqDropConnection(conn, true);
conn->status = CONNECTION_BAD; /* No more connection to backend */
@@ -875,6 +903,10 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ /* If in pipeline mode, set error indicator for it */
+ if (isError && conn->pipelineStatus != PQ_PIPELINE_OFF)
+ conn->pipelineStatus = PQ_PIPELINE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
@@ -930,9 +962,21 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
* Save the active query text, if any, into res as well; but only if we
* might need it for an error cursor display, which is only true if there
* is a PG_DIAG_STATEMENT_POSITION field.
+ *
+ * Note that in pipeline mode, we have not yet advanced the query pointer
+ * to the next query, so we have to look at that.
*/
- if (have_position && conn->last_query && res)
- res->errQuery = pqResultStrdup(res, conn->last_query);
+ if (have_position && res)
+ {
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->asyncStatus == PGASYNC_IDLE)
+ {
+ if (conn->cmd_queue_head)
+ res->errQuery = pqResultStrdup(res, conn->cmd_queue_head->query);
+ }
+ else if (conn->last_query)
+ res->errQuery = pqResultStrdup(res, conn->last_query);
+ }
/*
* Now build the "overall" error message for PQresultErrorMessage.
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index fa9b62a844..fb31a49fab 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -96,7 +96,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_PIPELINE_SYNC, /* pipeline synchronization point */
+ PGRES_PIPELINE_ABORTED, /* Command didn't run because of an abort
+ * earlier in a pipeline */
} ExecStatusType;
typedef enum
@@ -136,6 +139,16 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PGpipelineStatus - Current status of pipeline mode
+ */
+typedef enum
+{
+ PQ_PIPELINE_OFF,
+ PQ_PIPELINE_ON,
+ PQ_PIPELINE_ABORTED
+} PGpipelineStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -327,6 +340,7 @@ extern int PQserverVersion(const PGconn *conn);
extern char *PQerrorMessage(const PGconn *conn);
extern int PQsocket(const PGconn *conn);
extern int PQbackendPID(const PGconn *conn);
+extern PGpipelineStatus PQpipelineStatus(const PGconn *conn);
extern int PQconnectionNeedsPassword(const PGconn *conn);
extern int PQconnectionUsedPassword(const PGconn *conn);
extern int PQclientEncoding(const PGconn *conn);
@@ -434,6 +448,11 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for pipeline mode management */
+extern int PQenterPipelineMode(PGconn *conn);
+extern int PQexitPipelineMode(PGconn *conn);
+extern int PQsendPipeline(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 8d51e6ed9f..3e9f99f8fb 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,7 +217,11 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, more results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
@@ -229,7 +233,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* Sync at end of a pipeline */
} PGQueryClass;
/* Target server type (decoded value of target_session_attrs) */
@@ -305,6 +310,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by pipeline mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct PGcommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct PGcommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -397,6 +418,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PGpipelineStatus pipelineStatus; /* status of pipeline mode */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -409,6 +431,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue, for pipeline mode.
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -790,6 +822,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 5391f461a2..93e7829c67 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -10,6 +10,7 @@ SUBDIRS = \
delay_execution \
dummy_index_am \
dummy_seclabel \
+ libpq_pipeline \
plsample \
snapshot_too_old \
test_bloomfilter \
diff --git a/src/test/modules/libpq_pipeline/.gitignore b/src/test/modules/libpq_pipeline/.gitignore
new file mode 100644
index 0000000000..3a11e786b8
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/libpq_pipeline
diff --git a/src/test/modules/libpq_pipeline/Makefile b/src/test/modules/libpq_pipeline/Makefile
new file mode 100644
index 0000000000..b798f5fbbc
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/Makefile
@@ -0,0 +1,20 @@
+# src/test/modules/libpq_pipeline/Makefile
+
+PROGRAM = libpq_pipeline
+OBJS = libpq_pipeline.o
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS_INTERNAL += $(libpq_pgport)
+
+TAP_TESTS = 1
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/libpq_pipeline
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/libpq_pipeline/README b/src/test/modules/libpq_pipeline/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/libpq_pipeline/libpq_pipeline.c b/src/test/modules/libpq_pipeline/libpq_pipeline.c
new file mode 100644
index 0000000000..41706b514c
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/libpq_pipeline.c
@@ -0,0 +1,1141 @@
+/*
+ * src/test/modules/libpq_pipeline/libpq_pipeline.c
+ * Verify libpq pipeline execution functionality
+ */
+#include "postgres_fe.h"
+
+#include <sys/time.h>
+
+#include "catalog/pg_type_d.h"
+#include "common/fe_memutils.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+
+static void exit_nicely(PGconn *conn);
+
+const char *const progname = "libpq_pipeline";
+
+
+#define DEBUG
+#ifdef DEBUG
+#define pg_debug(...) do { fprintf(stderr, __VA_ARGS__); } while (0)
+#else
+#define pg_debug(...)
+#endif
+
+static const char *const drop_table_sql =
+"DROP TABLE IF EXISTS pq_pipeline_demo";
+static const char *const create_table_sql =
+"CREATE UNLOGGED TABLE pq_pipeline_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql =
+"INSERT INTO pq_pipeline_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+/*
+ * Print an error to stderr and terminate the program.
+ */
+#define pg_fatal(...) pg_fatal_impl(__LINE__, __VA_ARGS__)
+static void
+pg_fatal_impl(int line, const char *fmt,...)
+{
+ va_list args;
+
+ fprintf(stderr, "\n"); /* XXX hack */
+ fprintf(stderr, "%s:%d: ", progname, line);
+
+ va_start(args, fmt);
+ vfprintf(stderr, fmt, args);
+ va_end(args);
+ printf("Failure, exiting\n");
+ exit(1);
+}
+
+static void
+test_disallowed(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("Unable to enter pipeline mode\n");
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not activated properly\n");
+
+ /* PQexec should fail in pipeline mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("PQexec should fail in pipeline mode but succeeded\n");
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ pg_fatal("PQsendQuery should fail in pipeline mode but succeeded\n");
+
+ /* Entering pipeline mode when already in pipeline mode is OK */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("re-entering pipeline mode should be a no-op but failed\n");
+
+ if (PQisBusy(conn) != 0)
+ pg_fatal("PQisBusy should return 0 when idle in pipeline mode, returned 1\n");
+
+ /* ok, back to normal command mode */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("couldn't exit idle empty pipeline mode\n");
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not terminated properly\n");
+
+ /* exiting pipeline mode when not in pipeline mode should be a no-op */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("pipeline mode exit when not in pipeline mode should succeed but failed\n");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("PQexec should succeed after exiting pipeline mode but failed with: %s\n",
+ PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_simple_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple pipeline... ");
+
+ /*
+ * Enter pipeline mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our output buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode with work in progress should fail, but succeeded\n");
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending pipeline failed: %s\n", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first query result.\n");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit pipeline mode yet
+ */
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result PGRES_PIPELINE_SYNC expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of PGRES_PIPELINE_SYNC, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after pipeline end: %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* ... until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_multi_pipelines(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi pipeline... ");
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending first pipeline failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s\n", PQerrorMessage(conn));
+
+ /* OK, start processing the results */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first result\n");
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ PQclear(res);
+
+#if 0
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+#endif
+
+ /* second pipeline */
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from second pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode ... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+/*
+ * When an operation in a pipeline fails the rest of the pipeline is flushed. We
+ * still have to get results for each pipeline item, but the item will just be
+ * a PGRES_PIPELINE_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the pipeline. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_aborted_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted pipeline... ");
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * pipeline ERRORs.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT no_such_function($1)",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching error select failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending first pipeline failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second-pipeline insert failed: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the pipeline results.
+ *
+ * We should get a command-ok for the first query, then a fatal error and
+ * a pipeline aborted message for the second insert, a pipeline-end, then
+ * a command-ok and a pipeline-ok for the second pipeline operation.
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result status %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* Second query caused error, so we expect an error next */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("Unexpected result code -- expected PGRES_FATAL_ERROR, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /*
+ * pipeline should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third pipeline since we've already sent a
+ * second. The aborted flag relates only to the pipeline being received.
+ */
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't\n");
+
+ /* third query in pipeline, the second insert */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_ABORTED)
+ pg_fatal("Unexpected result code -- expected PGRES_PIPELINE_ABORTED, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+#if 0
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+#endif
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't\n");
+
+ /* Ensure we're still in pipeline */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /*
+ * The end of a failed pipeline is a PGRES_PIPELINE_SYNC.
+ *
+ * (This is so clients know to start processing results normally again and
+ * can tell the difference between skipped commands and the sync.)
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code from first pipeline sync\n"
+ "Expected PGRES_PIPELINE_SYNC, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+#if 0
+ /* after the synchronization point we get a NULL result */
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+#endif
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_ABORTED)
+ pg_fatal("sync should've cleared the aborted flag but didn't\n");
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* the insert from the second pipeline */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result code %s from first item in second pipeline\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* Read the NULL result at the end of the command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* the second pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQerrorMessage(conn));
+
+ /* Test single-row mode with an error partways */
+ if (PQsendQueryParams(conn, "SELECT 1/g FROM generate_series(5, -1, -1) g",
+ 0, NULL, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to send query: %s", PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ PQsetSingleRowMode(conn);
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ if (PQresultStatus(res) == PGRES_FATAL_ERROR)
+ printf("got error: {{%s}} (expected: division by zero)\n", PQerrorMessage(conn));
+ else if (PQresultStatus(res) == PGRES_SINGLE_TUPLE)
+ printf("got row: %s\n", PQgetvalue(res, 0, 0));
+ else
+ pg_fatal("got unexpected result %s\n", PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ }
+ /* the third pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from third pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ /*-
+ * Since we fired the pipelines off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st pipeline
+ * - First insert applied
+ * - Second statement aborted xact
+ * - Third insert skipped
+ * - Sync rolled back first implicit xact
+ * - Implicit xact created by server around 2nd pipeline
+ * - insert applied from 2nd pipeline
+ * - Sync commits 2nd xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM pq_pipeline_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Expected tuples, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("expected 1 result, got %d\n", PQntuples(res));
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ pg_fatal("expected only insert with value 3, got %s", val);
+ }
+
+ PQclear(res);
+}
+
+/* State machine enum for test_pipelined_insert */
+typedef enum PipelineInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} PipelineInsertStep;
+
+static void
+test_pipelined_insert(PGconn *conn, int n_rows)
+{
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ PipelineInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a pipelined insert into a table created at the start of the pipeline
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ while (send_step != BI_PREPARE)
+ {
+ const char *sql;
+
+ switch (send_step)
+ {
+ case BI_BEGIN_TX:
+ sql = "BEGIN TRANSACTION";
+ send_step = BI_DROP_TABLE;
+ break;
+
+ case BI_DROP_TABLE:
+ sql = drop_table_sql;
+ send_step = BI_CREATE_TABLE;
+ break;
+
+ case BI_CREATE_TABLE:
+ sql = create_table_sql;
+ send_step = BI_PREPARE;
+ break;
+
+ default:
+ pg_fatal("invalid state");
+ }
+
+ pg_debug("sending: %s\n", sql);
+ if (PQsendQueryParams(conn, sql,
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching %s failed: %s\n", sql, PQerrorMessage(conn));
+ }
+
+ Assert(send_step == BI_PREPARE);
+ pg_debug("sending: %s\n", insert_sql);
+ if (PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids) != 1)
+ pg_fatal("dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our output buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the pipeline before processing
+ * results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ pg_fatal("failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's output buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ PGresult *res;
+ const char *cmdtag;
+ const char *description = "";
+ int status;
+
+ /*
+ * Read next result. If no more results from this query,
+ * advance to the next query
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ continue;
+
+ status = PGRES_COMMAND_OK;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ recv_step++;
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ recv_step++;
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ recv_step++;
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ recv_step++;
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive == 0)
+ recv_step++;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ recv_step++;
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_PIPELINE_SYNC;
+ recv_step++;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ description = "";
+ abort();
+ }
+
+ if (PQresultStatus(res) != status)
+ pg_fatal("%s reported status %s, expected %s\n"
+ "Error message: \"%s\"\n",
+ description, PQresStatus(PQresultStatus(res)),
+ PQresStatus(status), PQerrorMessage(conn));
+
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ pg_fatal("%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+
+ pg_debug("Got %s OK\n", cmdtag[0] != '\0' ? cmdtag : description);
+
+ PQclear(res);
+ }
+ }
+
+ /* Write more rows and/or the end pipeline message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQsendPipeline(conn) == 1)
+ {
+ fprintf(stdout, "Sent pipeline\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending pipeline failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+ }
+
+ /* We've got the sync message and the pipeline should be done */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ pg_fatal("failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i;
+ bool pipeline_ended = false;
+
+ /* 1 pipeline, 3 queries in it */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n",
+ PQerrorMessage(conn));
+
+ for (i = 0; i < 3; i++)
+ {
+ char *param[1];
+
+ param[0] = psprintf("%d", 44 + i);
+
+ if (PQsendQueryParams(conn,
+ "SELECT generate_series(42, $1)",
+ 1,
+ NULL,
+ (const char **) param,
+ NULL,
+ NULL,
+ 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ pfree(param[0]);
+ }
+ PQsendPipeline(conn);
+
+ for (i = 0; !pipeline_ended; i++)
+ {
+ bool first = true;
+ bool saw_ending_tuplesok;
+ bool isSingleTuple = false;
+
+ /* Set single row mode for only first 2 SELECT queries */
+ if (i < 2)
+ {
+ if (PQsetSingleRowMode(conn) != 1)
+ pg_fatal("PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+
+ /* Consume rows for this query */
+ saw_ending_tuplesok = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+
+ if (est == PGRES_PIPELINE_SYNC)
+ {
+ fprintf(stderr, "end of pipeline reached\n");
+ pipeline_ended = true;
+ PQclear(res);
+ if (i != 3)
+ pg_fatal("Expected three results, got %d\n", i);
+ break;
+ }
+
+ /* Expect SINGLE_TUPLE for queries 0 and 1, TUPLES_OK for 2 */
+ if (first)
+ {
+ if (i <= 1 && est != PGRES_SINGLE_TUPLE)
+ pg_fatal("Expected PGRES_SINGLE_TUPLE for query %d, got %s\n",
+ i, PQresStatus(est));
+ if (i >= 2 && est != PGRES_TUPLES_OK)
+ pg_fatal("Expected PGRES_TUPLES_OK for query %d, got %s\n",
+ i, PQresStatus(est));
+ first = false;
+ }
+
+ fprintf(stderr, "Result status %s for query %d", PQresStatus(est), i);
+ switch (est)
+ {
+ case PGRES_TUPLES_OK:
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ saw_ending_tuplesok = true;
+ if (isSingleTuple)
+ {
+ if (PQntuples(res) == 0)
+ fprintf(stderr, "all tuples received in query %d\n", i);
+ else
+ pg_fatal("Expected to follow PGRES_SINGLE_TUPLE, "
+ "but received PGRES_TUPLES_OK directly instead\n");
+ }
+ break;
+
+ case PGRES_SINGLE_TUPLE:
+ isSingleTuple = true;
+ fprintf(stderr, ", %d tuple: %s\n", PQntuples(res), PQgetvalue(res, 0, 0));
+ break;
+
+ default:
+ pg_fatal("unexpected\n");
+ }
+ PQclear(res);
+ }
+ if (!pipeline_ended && !saw_ending_tuplesok)
+ pg_fatal("didn't get expected terminating TUPLES_OK\n");
+ }
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s\n", PQerrorMessage(conn));
+}
+
+/*
+ * Simple test to verify that a pipeline is discarded as a whole when there's
+ * an error, ignoring transaction commands.
+ */
+static void
+test_transaction(PGconn *conn)
+{
+ PGresult *res;
+ int num_sends = 0;
+
+ res = PQexec(conn, "DROP TABLE IF EXISTS pq_pipeline_tst;"
+ "CREATE TABLE pq_pipeline_tst (id int)");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to create test table: %s",
+ PQerrorMessage(conn));
+ PQclear(res);
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n",
+ PQerrorMessage(conn));
+ if (PQsendPrepare(conn, "rollback", "ROLLBACK", 0, NULL) != 1)
+ pg_fatal("could not send prepare on pipeline: %s",
+ PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn,
+ "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ if (PQsendQueryParams(conn,
+ "SELECT 0/0",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+
+ /*
+ * send a ROLLBACK using a prepared stmt. Doesn't work because we need to
+ * get out of the pipeline-aborted state first.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /* This insert fails because we're in pipeline-aborted state */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (1)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * This insert fails even though the pipeline got a SYNC, because we're in
+ * an aborted transaction
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (2)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * Send ROLLBACK using prepared stmt. This one works because we just did
+ * PQsendPipeline above.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /*
+ * Now that we're out of a transaction and in pipeline-good mode, this
+ * insert works
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (3)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+ PQsendPipeline(conn);
+ num_sends++;
+
+ for (int i = 0;; i++)
+ {
+ ExecStatusType restype;
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "%d: got NULL result\n", i);
+ continue;
+ }
+ restype = PQresultStatus(res);
+ fprintf(stderr, "%d: got status %s", i, PQresStatus(restype));
+ if (restype == PGRES_FATAL_ERROR)
+ fprintf(stderr, "; error: %s", PQerrorMessage(conn));
+ else if (restype == PGRES_PIPELINE_ABORTED)
+ {
+ fprintf(stderr, ": command didn't run because pipeline aborted\n");
+ }
+ else
+ fprintf(stderr, "\n");
+ PQclear(res);
+
+ if (restype == PGRES_PIPELINE_SYNC)
+ num_sends--;
+ if (num_sends <= 0)
+ break;
+ }
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("returned something extra after all the syncs: %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s\n", PQerrorMessage(conn));
+
+ /* We expect to find one tuple containing the value "3" */
+ res = PQexec(conn, "SELECT * FROM pq_pipeline_tst");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("failed to obtain result: %s", PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("did not get 1 tuple\n");
+ if (strcmp(PQgetvalue(res, 0, 0), "3") != 0)
+ pg_fatal("did not get expected tuple\n");
+ PQclear(res);
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+usage(const char *progname)
+{
+ fprintf(stderr, "%s tests libpq's pipeline mode.\n\n", progname);
+ fprintf(stderr, "Usage:\n");
+ fprintf(stderr, " %s testname [conninfo [number_of_rows]]\n", progname);
+ fprintf(stderr, "Tests:\n");
+ fprintf(stderr, " disallowed_in_pipeline\n");
+ fprintf(stderr, " simple_pipeline\n");
+ fprintf(stderr, " multi_pipeline\n");
+ fprintf(stderr, " pipeline_abort\n");
+ fprintf(stderr, " singlerow\n");
+ fprintf(stderr, " pipeline_insert\n");
+ fprintf(stderr, " transaction\n");
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo = "";
+ PGconn *conn;
+ int numrows = 10000;
+ PGresult *res;
+
+ /*
+ * The testname parameter is mandatory; it can be followed by a conninfo
+ * string and number of rows.
+ */
+ if (argc < 2 || argc > 4)
+ {
+ usage(argv[0]);
+ exit(1);
+ }
+
+ if (argc >= 3)
+ conninfo = pg_strdup(argv[2]);
+
+ if (argc >= 4)
+ {
+ errno = 0;
+ numrows = strtol(argv[3], NULL, 10);
+ if (errno != 0 || numrows <= 0)
+ {
+ fprintf(stderr, "couldn't parse \"%s\" as a positive integer\n", argv[3]);
+ exit(1);
+ }
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+ res = PQexec(conn, "SET lc_messages TO \"C\"");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to set lc_messages: %s", PQerrorMessage(conn));
+
+ if (strcmp(argv[1], "disallowed_in_pipeline") == 0)
+ test_disallowed(conn);
+ else if (strcmp(argv[1], "simple_pipeline") == 0)
+ test_simple_pipeline(conn);
+ else if (strcmp(argv[1], "multi_pipeline") == 0)
+ test_multi_pipelines(conn);
+ else if (strcmp(argv[1], "pipeline_abort") == 0)
+ test_aborted_pipeline(conn);
+ else if (strcmp(argv[1], "pipeline_insert") == 0)
+ test_pipelined_insert(conn, numrows);
+ else if (strcmp(argv[1], "singlerow") == 0)
+ test_singlerowmode(conn);
+ else if (strcmp(argv[1], "transaction") == 0)
+ test_transaction(conn);
+ else
+ {
+ fprintf(stderr, "\"%s\" is not a recognized test name\n", argv[1]);
+ usage(argv[0]);
+ exit(1);
+ }
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+ return 0;
+}
diff --git a/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
new file mode 100644
index 0000000000..a12f2dd47b
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
@@ -0,0 +1,31 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 7;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_pipeline
+ simple_pipeline
+ multi_pipeline
+ pipeline_abort
+ pipeline_insert
+ singlerow
+ transaction);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'libpq_pipeline', $testname, $node->connstr('postgres'), $numrows ],
+ "libp_pipeline $testname");
+}
+
+$node->stop('fast');
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 49614106dc..ed65a61a62 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -33,7 +33,8 @@ my @unlink_on_exit;
# Set of variables for modules in contrib/ and src/test/modules/
my $contrib_defines = { 'refint' => 'REFINT_VERBOSE' };
-my @contrib_uselibpq = ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo');
+my @contrib_uselibpq =
+ ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo', 'libpq_pipeline');
my @contrib_uselibpgport = ('oid2name', 'vacuumlo');
my @contrib_uselibpgcommon = ('oid2name', 'vacuumlo');
my $contrib_extralibs = undef;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 8bd95aefa1..1bd23eec6d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1561,10 +1561,12 @@ PG_Locale_Strategy
PG_Lock_Status
PG_init_t
PGcancel
+PGcommandQueueEntry
PGconn
PGdataValue
PGlobjfuncs
PGnotify
+PGpipelineStatus
PGresAttDesc
PGresAttValue
PGresParamDesc
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
On 2021-Mar-04, Alvaro Herrera wrote:
v31.
Got this:
libpq_pipeline.obj : error LNK2019: unresolved external symbol __WSAFDIsSet referenced in function test_pipelined_insert [C:\projects\postgresql\libpq_pipeline.vcxproj]
5019libpq_pipeline.obj : error LNK2019: unresolved external symbol __imp_select referenced in function test_pipelined_insert [C:\projects\postgresql\libpq_pipeline.vcxproj]
5020libpq_pipeline.obj : error LNK2019: unresolved external symbol pg_snprintf referenced in function test_pipelined_insert [C:\projects\postgresql\libpq_pipeline.vcxproj]
5021libpq_pipeline.obj : error LNK2019: unresolved external symbol pg_vfprintf referenced in function pg_fatal_impl [C:\projects\postgresql\libpq_pipeline.vcxproj]
5022libpq_pipeline.obj : error LNK2019: unresolved external symbol pg_fprintf referenced in function pg_fatal_impl [C:\projects\postgresql\libpq_pipeline.vcxproj]
5023libpq_pipeline.obj : error LNK2019: unresolved external symbol pg_printf referenced in function pg_fatal_impl [C:\projects\postgresql\libpq_pipeline.vcxproj]
5024libpq_pipeline.obj : error LNK2019: unresolved external symbol pg_strerror referenced in function test_pipelined_insert [C:\projects\postgresql\libpq_pipeline.vcxproj]
5025libpq_pipeline.obj : error LNK2019: unresolved external symbol pg_strdup referenced in function main [C:\projects\postgresql\libpq_pipeline.vcxproj]
5026libpq_pipeline.obj : error LNK2019: unresolved external symbol pfree referenced in function test_singlerowmode [C:\projects\postgresql\libpq_pipeline.vcxproj]
5027libpq_pipeline.obj : error LNK2019: unresolved external symbol psprintf referenced in function test_singlerowmode [C:\projects\postgresql\libpq_pipeline.vcxproj]
pg_snprintf, pg_vfprintf, pg_fprintf, pg_printf, pg_strerror are in pgport.
pg_strdup and pfree, psprintf are in pgcommon.
I don't know where do __WSAFDIsSet and __imp_select come from or what to
do about them. Let's see if adding pgport and pgcommon fixes things.
--
�lvaro Herrera 39�49'30"S 73�17'W
Attachments:
v32-libpq-pipeline.patchtext/x-diff; charset=us-asciiDownload
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 0553279314..c87b0ce911 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -3173,6 +3173,33 @@ ExecStatusType PQresultStatus(const PGresult *res);
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-sync">
+ <term><literal>PGRES_PIPELINE_SYNC</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a
+ synchronization point in pipeline mode, requested by
+ <xref linkend="libpq-PQsendPipeline"/>.
+ This status occurs only when pipeline mode has been selected.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-aborted">
+ <term><literal>PGRES_PIPELINE_ABORTED</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a pipeline that has
+ received an error from the server. <function>PQgetResult</function>
+ must be called repeatedly, and each time it will return this status code
+ until the end of the current pipeline, at which point it will return
+ <literal>PGRES_PIPELINE_SYNC</literal> and normal processing can
+ resume.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
If the result status is <literal>PGRES_TUPLES_OK</literal> or
@@ -4919,6 +4946,498 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-pipeline-mode">
+ <title>Pipeline Mode</title>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>libpq</primary>
+ <secondary>pipeline mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>pipelining</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>batch mode</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> pipeline mode allows applications to
+ send a query without having to read the result of the previously
+ sent query. Taking advantage of the pipeline mode, a client will wait
+ less for the server, since multiple queries/results can be sent/
+ received in a single network transaction.
+ </para>
+
+ <para>
+ While pipeline mode provides a significant performance boost, writing
+ clients using the pipeline mode is more complex because it involves
+ managing a queue of pending queries and finding which result
+ corresponds to which query in the queue.
+ </para>
+
+ <para>
+ Pipeline mode also generally consumes more memory on both the client and server,
+ though careful and aggressive management of the send/receive queue can mitigate
+ this. This applies whether or not the connection is in blocking or non-blocking
+ mode.
+ </para>
+
+ <sect2 id="libpq-pipeline-using">
+ <title>Using Pipeline Mode</title>
+
+ <para>
+ To issue pipelines, the application must switch a connection into pipeline mode.
+ Enter pipeline mode with <xref linkend="libpq-PQenterPipelineMode"/>
+ or test whether pipeline mode is active with
+ <xref linkend="libpq-PQpipelineStatus"/>.
+ In pipeline mode, only <link linkend="libpq-async">asynchronous operations</link>
+ are permitted, and <literal>COPY</literal> is disallowed.
+ Using any synchronous command execution functions,
+ such as <function>PQfn</function>, or <function>PQexec</function>
+ and its sibling functions, is an error condition.
+ </para>
+
+ <note>
+ <para>
+ It is best to use pipeline mode with <application>libpq</application> in
+ <link linkend="libpq-PQsetnonblocking">non-blocking mode</link>. If used
+ in blocking mode it is possible for a client/server deadlock to occur.
+ <footnote>
+ <para>
+ The client will block trying to send queries to the server, but the
+ server will block trying to send results to the client from queries
+ it has already processed. This only occurs when the client sends
+ enough queries to fill both its output buffer and the server's receive
+ buffer before it switches to processing input from the server,
+ but it's hard to predict exactly when that will happen.
+ </para>
+ </footnote>
+ </para>
+ </note>
+
+ <sect3 id="libpq-pipeline-sending">
+ <title>Issuing Queries</title>
+
+ <para>
+ After entering pipeline mode, the application dispatches requests using
+ <xref linkend="libpq-PQsendQueryParams"/>,
+ or its prepared-query sibling
+ <xref linkend="libpq-PQsendQueryPrepared"/>.
+ These requests are queued on the client-side until flushed to the server;
+ this occurs when <xref linkend="libpq-PQsendPipeline"/> is used to
+ establish a synchronization point in the pipeline,
+ or when <xref linkend="libpq-PQflush"/> is called.
+ The functions <xref linkend="libpq-PQsendPrepare"/>,
+ <xref linkend="libpq-PQsendDescribePrepared"/>, and
+ <xref linkend="libpq-PQsendDescribePortal"/> also work in pipeline mode.
+ Result processing is described below.
+ </para>
+
+ <para>
+ The server executes statements, and returns results, in the order the
+ client sends them. The server will begin executing the commands in the
+ pipeline immediately, not waiting for the end of the pipeline.
+ If any statement encounters an error, the server aborts the current
+ transaction and skips processing commands in the pipeline until the
+ next synchronization point established by <function>PQsendPipeline</function>.
+ (This remains true even if the commands in the pipeline would rollback
+ the transaction.)
+ Query processing resumes after the synchronization point.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one; for example, one query may define a table that the next
+ query in the same pipeline uses. Similarly, an application may
+ create a named prepared statement and execute it with later
+ statements in the same pipeline.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-results">
+ <title>Processing Results</title>
+
+ <para>
+ To process the result of one query in a pipeline, the application calls
+ <function>PQgetResult</function> repeatedly and handles each result
+ until <function>PQgetResult</function> returns null.
+ The result from the next query in the pipeline may then be retrieved using
+ <function>PQgetResult</function> again and the cycle repeated.
+ The application handles individual statement results as normal.
+ When the results of all the queries in the pipeline have been
+ returned, <function>PQgetResult</function> returns a result
+ containing the status value <literal>PGRES_PIPELINE_SYNC</literal>.
+ </para>
+
+ <para>
+ The client may choose to defer result processing until the complete
+ pipeline has been sent, or interleave that with sending further
+ queries in the pipeline; see <xref linkend="libpq-pipeline-interleave" />.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ This mode selection is effective only for the query currently
+ being processed. For more information on the use of
+ <function>PQsetSingleRowMode</function>,
+ refer to <xref linkend="libpq-single-row-mode"/>.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal
+ asynchronous processing except that it may contain the new
+ <type>PGresult</type> types <literal>PGRES_PIPELINE_SYNC</literal>
+ and <literal>PGRES_PIPELINE_ABORTED</literal>.
+ <literal>PGRES_PIPELINE_SYNC</literal> is reported exactly once for each
+ <function>PQsendPipeline</function> after retrieving results for all
+ queries in the pipeline.
+ <literal>PGRES_PIPELINE_ABORTED</literal> is emitted in place of a normal
+ stream result for the first error and all subsequent results
+ except <literal>PGRES_PIPELINE_SYNC</literal> and null;
+ see <xref linkend="libpq-pipeline-errors"/>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing pipeline results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed (except that
+ <function>PQgetResult</function> returns null to indicate that we start
+ returning the results of next query). The application must keep track
+ of the order in which it sent queries, to associate them with their
+ corresponding results.
+ Applications will typically use a state machine or a FIFO queue for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-pipeline-errors">
+ <title>Error Handling</title>
+
+ <para>
+ When a query in a pipeline causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the pipeline
+ synchronization message. The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after <function>PQresultStatus</function>
+ returns <literal>PGRES_FATAL_ERROR</literal>,
+ the pipeline is flagged as aborted.
+ <function>PQresultStatus</function>, will report a
+ <literal>PGRES_PIPELINE_ABORTED</literal> result for each remaining queued
+ operation in an aborted pipeline. The result for
+ <function>PQsendPipeline</function> is reported as
+ <literal>PGRES_PIPELINE_SYNC</literal> to signal the end of the aborted pipeline
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the pipeline used an implicit transaction, then operations that have
+ already executed are rolled back and operations that were queued to follow
+ the failed operation are skipped entirely. The same behaviour holds if the
+ pipeline starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the pipeline. If a pipeline contains
+ <emphasis>multiple explicit transactions</emphasis>, all transactions that
+ committed prior to the error remain committed, the currently in-progress
+ transaction is aborted, and all subsequent operations are skipped completely,
+ including subsequent transactions.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal> — only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously, the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-interleave">
+ <title>Interleaving Result Processing and Query Dispatch</title>
+
+ <para>
+ To avoid deadlocks on large pipelines the client should be structured
+ around a non-blocking event loop using operating system facilities
+ such as <function>select</function>, <function>poll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work
+ remaining to be dispatched and a queue of work that has been dispatched
+ but not yet had its results processed. When the socket is writable
+ it should dispatch more work. When the socket is readable it should
+ read results and process them, matching them up to the next entry in
+ its expected results queue. Based on available memory, results from the
+ socket should be read frequently: there's no need to wait until the
+ pipeline end to read the results. Pipelines should be scoped to logical
+ units of work, usually (but not necessarily) one transaction per pipeline.
+ There's no need to exit pipeline mode and re-enter it between pipelines,
+ or to wait for one pipeline to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state
+ machine to track sent and received work is in
+ <filename>src/test/modules/libpq_pipeline/libpq_pipeline.c</filename>
+ in the PostgreSQL source distribution.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-end">
+ <title>Ending Pipeline Mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed, and
+ the end pipeline result has been consumed, the application may return
+ to non-pipelined mode with <xref linkend="libpq-PQexitPipelineMode"/>.
+ </para>
+ </sect3>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-functions">
+ <title>Functions Associated with Pipeline Mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQpipelineStatus">
+ <term>
+ <function>PQpipelineStatus</function>
+ <indexterm>
+ <primary>PQpipelineStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the current pipeline mode status of the
+ <application>libpq</application> connection.
+<synopsis>
+PGpipelineStatus PQpipelineStatus(const PGconn *conn);
+</synopsis>
+ </para>
+
+ <para>
+ <function>PQpipelineStatus</function> can return one of the following values:
+
+ <variablelist>
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ON</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in
+ pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_OFF</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is
+ <emphasis>not</emphasis> in pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in pipeline
+ mode and an error occurred while processing the current pipeline.
+ The aborted flag is cleared when <function>PQresultStatus</function>
+ returns PGRES_PIPELINE_SYNC at the end of the pipeline.
+ Clients don't usually need this function to
+ verify aborted status, as they can tell that the pipeline is aborted
+ from the <literal>PGRES_PIPELINE_ABORTED</literal> result code.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterPipelineMode">
+ <term>
+ <function>PQenterPipelineMode</function>
+ <indexterm>
+ <primary>PQenterPipelineMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter pipeline mode if it is currently idle or
+ already in pipeline mode.
+
+<synopsis>
+int PQenterPipelineMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success.
+ Returns 0 and has no effect if the connection is not currently
+ idle, i.e., it has a result ready, or it is waiting for more
+ input from the server, etc.
+ This function does not actually send anything to the server,
+ it just changes the <application>libpq</application> connection
+ state.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitPipelineMode">
+ <term>
+ <function>PQexitPipelineMode</function>
+ <indexterm>
+ <primary>PQexitPipelineMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit pipeline mode if it is currently in pipeline mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitPipelineMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 1 and takes no action if not in
+ pipeline mode. If the current statement isn't finished processing
+ or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQsendPipeline">
+ <term>
+ <function>PQsendPipeline</function>
+ <indexterm>
+ <primary>PQsendPipeline</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Marks a synchronization point in a pipeline by sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ and flushing the send buffer. This serves as
+ the delimiter of an implicit transaction and an error recovery
+ point; see <xref linkend="libpq-pipeline-errors"/>.
+
+<synopsis>
+int PQsendPipeline(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 if the connection is not in
+ pipeline mode or sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ failed.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-tips">
+ <title>When to Use Pipeline Mode</title>
+
+ <para>
+ Much like asynchronous query mode, there is no meaningful performance
+ overhead when using pipeline mode. It increases client application complexity,
+ and extra caution is required to prevent client/server deadlocks, but
+ pipeline mode can offer considerable performance improvements, in exchange for
+ increased memory usage from leaving state around longer.
+ </para>
+
+ <para>
+ Pipeline mode is most useful when the server is distant, i.e., network latency
+ (<quote>ping time</quote>) is high, and also when many small operations
+ are being performed in rapid succession. There is usually less benefit
+ in using pipelined commands when each query takes many multiples of the client/server
+ round-trip time to execute. A 100-statement operation run on a server
+ 300ms round-trip-time away would take 30 seconds in network latency alone
+ without pipelining; with pipelining it may spend as little as 0.3s waiting for
+ results from the server.
+ </para>
+
+ <para>
+ Use pipelined commands when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed
+ into operations on sets, or into a <literal>COPY</literal> operation.
+ </para>
+
+ <para>
+ Pipeline mode is not useful when information from one operation is required by
+ the client to produce the next operation. In such cases, the client
+ would have to introduce a synchronization point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+BEGIN;
+SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+-- result: x=2
+-- client adds 1 to x:
+UPDATE mytable SET x = 3 WHERE id = 42;
+COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <para>
+ Pipelining is less useful, and more complex, when a single pipeline contains
+ multiple transactions (see <xref linkend="libpq-pipeline-errors"/>).
+ </para>
+
+ <note>
+ <para>
+ The pipeline API was introduced in <productname>PostgreSQL</productname> 14.
+ Pipeline mode is a client-side feature which doesn't require server
+ support, and works on any server that supports the v3 extended query
+ protocol.
+ </para>
+ </note>
+ </sect2>
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4959,6 +5478,13 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <para>
+ When using pipeline mode, single-row mode needs to be activated for each
+ query in the pipeline before retrieving results for that query
+ with <function>PQgetResult</function>.
+ See <xref linkend="libpq-pipeline-mode"/> for more information.
+ </para>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 6d46da42e2..012e44c736 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while a libpq connection is in pipeline mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 2ec0580a79..75448162ca 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -1109,6 +1109,12 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
row, the last value is kept.
</para>
+ <para>
+ <literal>\gset</literal> and <literal>\aset</literal> cannot be used
+ pipeline mode, since query results are not immediately
+ fetched in this mode.
+ </para>
+
<para>
The following example puts the final account balance from the first query
into variable <replaceable>abalance</replaceable>, and fills variables
@@ -1269,6 +1275,21 @@ SELECT 4 AS four \; SELECT 5 AS five \aset
</programlisting></para>
</listitem>
</varlistentry>
+
+ <varlistentry id='pgbench-metacommand-pipeline'>
+ <term><literal>\startpipeline</literal></term>
+ <term><literal>\endpipeline</literal></term>
+
+ <listitem>
+ <para>
+ These commands delimit the start and end of a pipeline of SQL statements.
+ In a pipeline, statements are sent to server without waiting for the results
+ of previous statements (see <xref linkend="libpq-pipeline-mode"/>).
+ Pipeline mode requires the extended query protocol.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect2>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 5272eed9ab..f74378110a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1019,6 +1019,12 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->err = _("empty query");
break;
+ case PGRES_PIPELINE_SYNC:
+ case PGRES_PIPELINE_ABORTED:
+ walres->status = WALRCV_ERROR;
+ walres->err = _("unexpected pipeline mode");
+ break;
+
case PGRES_NONFATAL_ERROR:
case PGRES_FATAL_ERROR:
case PGRES_BAD_RESPONSE:
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 31a4df45f5..fbbe270654 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -351,10 +351,11 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
- * and we enter the CSTATE_SLEEP state to wait for it to expire. Other
- * meta-commands are executed immediately. If the command about to start
- * is actually beyond the end of the script, advance to CSTATE_END_TX.
+ * CSTATE_WAIT_RESULT state unless in pipeline mode. On a \sleep
+ * meta-command, the timer is set, and we enter the CSTATE_SLEEP state to
+ * wait for it to expire. Other meta-commands are executed immediately. If
+ * the command about to start is actually beyond the end of the script,
+ * advance to CSTATE_END_TX.
*
* CSTATE_WAIT_RESULT waits until we get a result set back from the server
* for the current command.
@@ -484,7 +485,9 @@ typedef enum MetaCommand
META_IF, /* \if */
META_ELIF, /* \elif */
META_ELSE, /* \else */
- META_ENDIF /* \endif */
+ META_ENDIF, /* \endif */
+ META_STARTPIPELINE, /* \startpipeline */
+ META_ENDPIPELINE /* \endpipeline */
} MetaCommand;
typedef enum QueryMode
@@ -2504,6 +2507,10 @@ getMetaCommand(const char *cmd)
mc = META_GSET;
else if (pg_strcasecmp(cmd, "aset") == 0)
mc = META_ASET;
+ else if (pg_strcasecmp(cmd, "startpipeline") == 0)
+ mc = META_STARTPIPELINE;
+ else if (pg_strcasecmp(cmd, "endpipeline") == 0)
+ mc = META_ENDPIPELINE;
else
mc = META_NONE;
return mc;
@@ -2693,11 +2700,25 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In pipeline mode, we use asynchronous functions. If a
+ * server-side error occurs, it will be processed later
+ * among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ pg_log_error("%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2741,8 +2762,10 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
* varprefix should be set only with \gset or \aset, and SQL commands do
* not need it.
*/
+#if 0
Assert((meta == META_NONE && varprefix == NULL) ||
((meta == META_GSET || meta == META_ASET) && varprefix != NULL));
+#endif
res = PQgetResult(st->con);
@@ -2810,6 +2833,12 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* otherwise the result is simply thrown away by PQclear below */
break;
+ case PGRES_PIPELINE_SYNC:
+ pg_log_debug("client %d pipeline ending", st->id);
+ if (PQexitPipelineMode(st->con) != 1)
+ pg_log_error("client %d failed to exit pipeline mode", st->id);
+ break;
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3066,13 +3095,36 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
/* Execute the command */
if (command->type == SQL_COMMAND)
{
+ /* disallow \aset and \gset in pipeline mode */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ if (command->meta == META_GSET)
+ {
+ commandFailed(st, "gset", "\\gset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ else if (command->meta == META_ASET)
+ {
+ commandFailed(st, "aset", "\\aset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ }
+
if (!sendCommand(st, command))
{
commandFailed(st, "SQL", "SQL command send failed");
st->state = CSTATE_ABORTED;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in pipeline mode */
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -3212,7 +3264,15 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (readCommandResponse(st,
sql_script[st->use_file].commands[st->command]->meta,
sql_script[st->use_file].commands[st->command]->varprefix))
- st->state = CSTATE_END_COMMAND;
+ {
+ /*
+ * outside of pipeline mode: stop reading results.
+ * pipeline mode: continue reading results until an
+ * end-of-pipeline response.
+ */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ st->state = CSTATE_END_COMMAND;
+ }
else
st->state = CSTATE_ABORTED;
break;
@@ -3456,6 +3516,49 @@ executeMetaCommand(CState *st, instr_time *now)
return CSTATE_ABORTED;
}
}
+ else if (command->meta == META_STARTPIPELINE)
+ {
+ /*
+ * In pipeline mode, we use a workflow based on libpq pipeline
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "startpipeline", "cannot use pipeline mode with the simple query protocol");
+ return CSTATE_ABORTED;
+ }
+
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ commandFailed(st, "startpipeline", "already in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (PQenterPipelineMode(st->con) == 0)
+ {
+ commandFailed(st, "startpipeline", "failed to enter pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ }
+ else if (command->meta == META_ENDPIPELINE)
+ {
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ {
+ commandFailed(st, "endpipeline", "not in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (!PQsendPipeline(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to send the pipeline");
+ return CSTATE_ABORTED;
+ }
+ if (!PQexitPipelineMode(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to exit pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ /* collect pending results before getting out of pipeline mode */
+ return CSTATE_WAIT_RESULT;
+ }
/*
* executing the expression or shell command might have taken a
@@ -4683,7 +4786,9 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
- else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
+ else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF ||
+ my_command->meta == META_STARTPIPELINE ||
+ my_command->meta == META_ENDPIPELINE)
{
if (my_command->argc != 1)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
@@ -6808,4 +6913,5 @@ pthread_join(pthread_t th, void **thread_return)
return 0;
}
+
#endif /* WIN32 */
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..60d09e6d63 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,7 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterPipelineMode 180
+PQexitPipelineMode 181
+PQsendPipeline 182
+PQpipelineStatus 183
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index f83af03d0a..4c97e8a078 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -540,6 +540,23 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * pqFreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+pqFreeCommandQueue(PGcommandQueueEntry *queue)
+{
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *cur = queue;
+
+ queue = cur->next;
+ if (cur->query)
+ free(cur->query);
+ free(cur);
+ }
+}
/*
* pqDropServerData
@@ -571,6 +588,12 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ pqFreeCommandQueue(conn->cmd_queue_head);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ pqFreeCommandQueue(conn->cmd_queue_recycle);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
@@ -2486,6 +2509,7 @@ keep_going: /* We will come back to here until there is
/* Drop any PGresult we might have, too */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn);
/* Reset conn->status to put the state machine in the right state */
@@ -3928,6 +3952,7 @@ makeEmptyPGconn(void)
conn->status = CONNECTION_BAD;
conn->asyncStatus = PGASYNC_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
conn->xactStatus = PQTRANS_IDLE;
conn->options_valid = false;
conn->nonblocking = false;
@@ -4187,6 +4212,7 @@ closePGconn(PGconn *conn)
conn->status = CONNECTION_BAD; /* Well, not really _bad_ - just absent */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn); /* deallocate result */
resetPQExpBuffer(&conn->errorMessage);
release_conn_addrinfo(conn);
@@ -6735,6 +6761,15 @@ PQbackendPID(const PGconn *conn)
return conn->be_pid;
}
+PGpipelineStatus
+PQpipelineStatus(const PGconn *conn)
+{
+ if (!conn)
+ return PQ_PIPELINE_OFF;
+
+ return conn->pipelineStatus;
+}
+
int
PQconnectionNeedsPassword(const PGconn *conn)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9a038043b2..0fb790fdfa 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_PIPELINE_SYNC",
+ "PGRES_PIPELINE_ABORTED"
};
/*
@@ -71,6 +73,11 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *pqMakePipelineCmd(PGconn *conn);
+static void pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqPipelineProcessQueue(PGconn *conn);
+static int pqPipelineFlush(PGconn *conn);
/* ----------------
@@ -1171,7 +1178,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1197,18 +1204,37 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot use %s in pipeline mode, use PQsendQueryParams\n"),
+ "PQsendQuery");
+ return 0;
+ }
+
return PQsendQueryInternal(conn, query, true);
}
int
PQsendQueryContinue(PGconn *conn, const char *query)
{
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot use %s in pipeline mode, use PQsendQueryParams\n"),
+ "PQsendQueryContinue");
+ return 0;
+ }
+
return PQsendQueryInternal(conn, query, false);
}
static int
PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
{
+ /* pipeline mode requires extended query protocol */
+ Assert(conn->pipelineStatus == PQ_PIPELINE_OFF);
+
if (!PQsendQueryStart(conn, newQuery))
return 0;
@@ -1307,6 +1333,8 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+
if (!PQsendQueryStart(conn, true))
return 0;
@@ -1330,6 +1358,15 @@ PQsendPrepare(PGconn *conn,
return 0;
}
+ /* Alloc pipeline memory before doing anything */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+ }
+
/* construct the Parse message */
if (pqPutMsgStart('P', conn) < 0 ||
pqPuts(stmtName, conn) < 0 ||
@@ -1356,32 +1393,46 @@ PQsendPrepare(PGconn *conn,
if (pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /*
+ * In non-pipeline mode, add a Sync and prepare to send. In pipeline mode
+ * we just keep track of the new message.
+ */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ /* remember we are doing just a Parse */
+ conn->queryclass = PGQUERY_PREPARE;
- /* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
-
- /* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ /* and remember the query text too, if possible */
+ /* if insufficient memory, last_query just winds up NULL */
+ if (conn->last_query)
+ free(conn->last_query);
+ conn->last_query = strdup(query);
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ else
+ {
+ pipeCmd->queryclass = PGQUERY_PREPARE;
+ /* as above, if insufficient memory, query winds up NULL */
+ pipeCmd->query = strdup(query);
+ pqAppendPipelineCmd(conn, pipeCmd);
+ }
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
- /* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1429,7 +1480,8 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn, bool newQuery)
@@ -1450,20 +1502,57 @@ PQsendQueryStart(PGconn *conn, bool newQuery)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE &&
+ conn->pipelineStatus == PQ_PIPELINE_OFF)
{
appendPQExpBufferStr(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * When enqueuing commands we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when pqPipelineProcessQueue()
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ }
+ }
+ else
+ {
+ /*
+ * This command's results will come in immediately. Initialize async
+ * result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1487,10 +1576,34 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **query;
+ PGQueryClass *queryclass;
/*
- * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
- * using specified statement name and the unnamed portal.
+ * Decide where the query is going to be stored. In pipeline mode, we
+ * allocate a new pipeline element; in non-pipeline mode, it's simply the
+ * connection's last query.
+ */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+ /*
+ * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync
+ * (if not in pipeline mode), using specified statement name and the
+ * unnamed portal.
*/
if (command)
@@ -1600,35 +1713,43 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /* construct the Sync message if not in pipeline mode */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ /* if insufficient memory, query just winds up NULL */
+ if (*query)
+ free(*query);
if (command)
- conn->last_query = strdup(command);
+ *query = strdup(command);
else
- conn->last_query = NULL;
+ *query = NULL;
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1726,14 +1847,17 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
-
/*
* PQgetResult
* Get the next PGresult produced by a query. Returns NULL if no
* query work remains or an error has occurred (e.g. out of
* memory).
+ *
+ * In pipeline mode, once all the result of a query have been returned,
+ * PQgetResult returns NULL to let the user know that the next
+ * query is being processed. At the end of the pipeline, returns a
+ * result with PQresultStatus(result) == PGRES_PIPELINE_SYNC.
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1803,8 +1927,49 @@ PQgetResult(PGconn *conn)
{
case PGASYNC_IDLE:
res = NULL; /* query is complete */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to return the NULL that terminates the round of
+ * results from the current query; prepare to send the results
+ * of the next query when we're called next. Also, since this
+ * is the start of the results of the next query, clear any
+ * prior error message.
+ */
+ resetPQExpBuffer(&conn->errorMessage);
+ pqPipelineProcessQueue(conn);
+ }
break;
case PGASYNC_READY:
+ res = pqPrepareAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to send the results of the current query. Set
+ * us idle now, and ...
+ */
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /*
+ * ... in cases when we're sending a pipeline-related result,
+ * move queue processing forwards immediately, so that next
+ * time we're called, we're prepared to return the next result
+ * received from the server.
+ *
+ * In all other cases, leave the queue state change for next
+ * time, so that a terminating NULL result is sent.
+ */
+ if (res && (res->resultStatus == PGRES_PIPELINE_ABORTED ||
+ res->resultStatus == PGRES_PIPELINE_SYNC))
+ pqPipelineProcessQueue(conn);
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -1985,6 +2150,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("synchronous command execution functions are not allowed in pipeline mode\n"));
+ return false;
+ }
+
/*
* Since this is the beginning of a query cycle, reset the error buffer.
*/
@@ -2148,6 +2320,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2155,6 +2330,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
if (!PQsendQueryStart(conn, true))
return 0;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ queryclass = &conn->queryclass;
+
/* construct the Describe message */
if (pqPutMsgStart('D', conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2163,32 +2350,40 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ /* reset last-query string (not relevant now) */
+ if (conn->last_query && conn->pipelineStatus != PQ_PIPELINE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
}
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -2541,6 +2736,13 @@ PQfn(PGconn *conn,
*/
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("PQfn not allowed in pipeline mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -2555,6 +2757,359 @@ PQfn(PGconn *conn,
args, nargs);
}
+/* ====== Pipeline mode support ======== */
+
+/*
+ * PQenterPipelineMode
+ * Put an idle connection in pipeline mode.
+ *
+ * Returns 1 on success. On failure, errorMessage is set and 0 is returned.
+ *
+ * Commands submitted after this can be pipelined on the connection;
+ * there's no requirement to wait for one to finish before the next is
+ * dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQsendPipeline. Multiple pipelines
+ * can be sent while in pipeline mode. Pipeline mode can be exited
+ * by calling PQexitPipelineMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ /* succeed with no action if already in pipeline mode */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot enter pipeline mode, connection not idle\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+
+ return 1;
+}
+
+/*
+ * PQexitPipelineMode
+ * End pipeline mode and return to normal command mode.
+ *
+ * Returns 1 in success (pipeline mode successfully ended, or not in pipeline
+ * mode).
+ *
+ * Returns 0 if in pipeline mode and cannot be ended yet. Error message will
+ * be set.
+ */
+int
+PQexitPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ /* there are some uncollected results */
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+
+ case PGASYNC_BUSY:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode while busy\n"));
+ return 0;
+
+ default:
+ /* OK */
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ return 0; /* error message is setup already */
+ return 1;
+}
+
+/*
+ * pqPipelineProcessQueue: subroutine for PQgetResult
+ * In pipeline mode, start processing the results of the next query in the queue.
+ */
+static void
+pqPipelineProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ Assert(conn->pipelineStatus != PQ_PIPELINE_OFF);
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: COPY in pipeline mode\n");
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return;
+ case PGASYNC_IDLE:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In pipeline mode but nothing left on the queue; caller can submit
+ * more work or PQexitPipelineMode() now.
+ */
+ return;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-pipeline call.
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /*
+ * Reset single-row processing mode. (Client has to set it up for each
+ * query, if desired.)
+ */
+ conn->singleRowMode = false;
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ pqRecyclePipelineCmd(conn, next_query);
+
+ if (conn->pipelineStatus == PQ_PIPELINE_ABORTED &&
+ conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted pipeline we don't get anything from the server for
+ * each result; we're just discarding input until we get to the next
+ * sync from the server. The client needs to know its queries got
+ * aborted so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn, PGRES_PIPELINE_ABORTED);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ pqSaveErrorResult(conn);
+ return;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+}
+
+/*
+ * PQsendPipeline
+ * Send a Sync message as part of a pipeline, and flush to server
+ *
+ * It's legal to start submitting more commands in the pipeline immediately,
+ * without waiting for the results of the current pipeline. There's no need to
+ * end pipeline mode and start it again.
+ *
+ * If a command in a pipeline fails, every subsequent command up to and including
+ * the result to the Sync message sent by PQsendPipeline gets set to
+ * PGRES_PIPELINE_ABORTED state. If the whole pipeline is processed without
+ * error, a PGresult with PGRES_PIPELINE_SYNC is produced.
+ *
+ * Queries can already have been sent before PQsendPipeline is called, but
+ * PQsendPipeline need to be called before retrieving command results.
+ *
+ * The connection will remain in pipeline mode and unavailable for new
+ * synchronous command execution functions until all results from the pipeline
+ * are processed by the client.
+ */
+int
+PQsendPipeline(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot send pipeline when not in pipeline mode\n"));
+ return 0;
+ }
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: cannot send pipeline while in COPY\n");
+ return 0;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_IDLE:
+ /* OK to send sync */
+ break;
+ }
+
+ entry = pqMakePipelineCmd(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ pqAppendPipelineCmd(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ /*
+ * Call pqPipelineProcessQueue so the user can call start calling
+ * PQgetResult.
+ */
+ pqPipelineProcessQueue(conn);
+
+ return 1;
+
+sendFailed:
+ pqRecyclePipelineCmd(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
+
+/*
+ * pqMakePipelineCmd
+ * Get a command queue entry for caller to fill.
+ *
+ * If the recycle queue has a free element, that is returned; if not, a
+ * fresh one is allocated. Caller is responsible for adding it to the
+ * command queue (pqAppendPipelineCmd) once the struct is filled in, or
+ * releasing the memory (pqRecyclePipelineCmd) if an error occurs.
+ *
+ * If allocation fails, sets the error message and returns NULL.
+ */
+static PGcommandQueueEntry *
+pqMakePipelineCmd(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * pqAppendPipelineCmd
+ * Append a caller-allocated command queue entry to the queue.
+ *
+ * The query itself must already have been put in the output buffer by the
+ * caller.
+ */
+static void
+pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ PGcommandQueueEntry **tail;
+
+ if (conn->cmd_queue_head == NULL)
+ tail = &conn->cmd_queue_head;
+ else
+ tail = &conn->cmd_queue_tail->next;
+
+ *tail = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * pqRecyclePipelineCmd
+ * Push a command queue entry onto the freelist. It must be an entry
+ * with null next pointer and not referenced by any other entry's next
+ * pointer.
+ */
+static void
+pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+
+ Assert(entry->next == NULL);
+
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
/* ====== accessor funcs for PGresult ======== */
@@ -3152,6 +3707,23 @@ PQflush(PGconn *conn)
return pqFlush(conn);
}
+/*
+ * pqPipelineFlush
+ *
+ * In pipeline mode, data will be flushed only when the out buffer reaches the
+ * threshold value. In non-pipeline mode, it behaves as stock pqFlush.
+ *
+ * Returns 0 on success.
+ */
+static int
+pqPipelineFlush(PGconn *conn)
+{
+ if ((conn->pipelineStatus == PQ_PIPELINE_OFF) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return pqFlush(conn);
+ return 0;
+}
+
/*
* PQfreemem - safely frees memory allocated
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 2ca8c057b9..b131995974 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -177,14 +177,24 @@ pqParseInput3(PGconn *conn)
if (getParameterStatus(conn))
return;
}
- else
- {
- pqInternalNotice(&conn->noticeHooks,
- "message type 0x%02x arrived from server while idle",
- id);
- /* Discard the unexpected message */
- conn->inCursor += msgLength;
- }
+
+ /*
+ * We're also IDLE when in pipeline mode we have completed
+ * processing the results of one query and are waiting for the
+ * next one in the pipeline. In this case, as above, just wait to
+ * see what's next.
+ */
+ if (conn->asyncStatus == PGASYNC_IDLE &&
+ conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->cmd_queue_head != NULL)
+ return;
+
+ /* Any other case is unexpected and we summarily skip it */
+ pqInternalNotice(&conn->noticeHooks,
+ "message type 0x%02x arrived from server while idle",
+ id);
+ /* Discard the unexpected message */
+ conn->inCursor += msgLength;
}
else
{
@@ -217,10 +227,28 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new
+ * query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_PIPELINE_SYNC);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ else
+ {
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -450,7 +478,7 @@ handleSyncLoss(PGconn *conn, char id, int msgLength)
id, msgLength);
/* build an error result holding the error message */
pqSaveErrorResult(conn);
- conn->asyncStatus = PGASYNC_READY; /* drop out of GetResult wait loop */
+ conn->asyncStatus = PGASYNC_READY; /* drop out of PQgetResult wait loop */
/* flush input data since we're giving up on processing it */
pqDropConnection(conn, true);
conn->status = CONNECTION_BAD; /* No more connection to backend */
@@ -875,6 +903,10 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ /* If in pipeline mode, set error indicator for it */
+ if (isError && conn->pipelineStatus != PQ_PIPELINE_OFF)
+ conn->pipelineStatus = PQ_PIPELINE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
@@ -930,9 +962,21 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
* Save the active query text, if any, into res as well; but only if we
* might need it for an error cursor display, which is only true if there
* is a PG_DIAG_STATEMENT_POSITION field.
+ *
+ * Note that in pipeline mode, we have not yet advanced the query pointer
+ * to the next query, so we have to look at that.
*/
- if (have_position && conn->last_query && res)
- res->errQuery = pqResultStrdup(res, conn->last_query);
+ if (have_position && res)
+ {
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->asyncStatus == PGASYNC_IDLE)
+ {
+ if (conn->cmd_queue_head)
+ res->errQuery = pqResultStrdup(res, conn->cmd_queue_head->query);
+ }
+ else if (conn->last_query)
+ res->errQuery = pqResultStrdup(res, conn->last_query);
+ }
/*
* Now build the "overall" error message for PQresultErrorMessage.
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index fa9b62a844..fb31a49fab 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -96,7 +96,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_PIPELINE_SYNC, /* pipeline synchronization point */
+ PGRES_PIPELINE_ABORTED, /* Command didn't run because of an abort
+ * earlier in a pipeline */
} ExecStatusType;
typedef enum
@@ -136,6 +139,16 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PGpipelineStatus - Current status of pipeline mode
+ */
+typedef enum
+{
+ PQ_PIPELINE_OFF,
+ PQ_PIPELINE_ON,
+ PQ_PIPELINE_ABORTED
+} PGpipelineStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -327,6 +340,7 @@ extern int PQserverVersion(const PGconn *conn);
extern char *PQerrorMessage(const PGconn *conn);
extern int PQsocket(const PGconn *conn);
extern int PQbackendPID(const PGconn *conn);
+extern PGpipelineStatus PQpipelineStatus(const PGconn *conn);
extern int PQconnectionNeedsPassword(const PGconn *conn);
extern int PQconnectionUsedPassword(const PGconn *conn);
extern int PQclientEncoding(const PGconn *conn);
@@ -434,6 +448,11 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for pipeline mode management */
+extern int PQenterPipelineMode(PGconn *conn);
+extern int PQexitPipelineMode(PGconn *conn);
+extern int PQsendPipeline(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 8d51e6ed9f..3e9f99f8fb 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,7 +217,11 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, more results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
@@ -229,7 +233,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* Sync at end of a pipeline */
} PGQueryClass;
/* Target server type (decoded value of target_session_attrs) */
@@ -305,6 +310,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by pipeline mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct PGcommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct PGcommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -397,6 +418,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PGpipelineStatus pipelineStatus; /* status of pipeline mode */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -409,6 +431,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue, for pipeline mode.
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -790,6 +822,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 5391f461a2..93e7829c67 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -10,6 +10,7 @@ SUBDIRS = \
delay_execution \
dummy_index_am \
dummy_seclabel \
+ libpq_pipeline \
plsample \
snapshot_too_old \
test_bloomfilter \
diff --git a/src/test/modules/libpq_pipeline/.gitignore b/src/test/modules/libpq_pipeline/.gitignore
new file mode 100644
index 0000000000..3a11e786b8
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/libpq_pipeline
diff --git a/src/test/modules/libpq_pipeline/Makefile b/src/test/modules/libpq_pipeline/Makefile
new file mode 100644
index 0000000000..b798f5fbbc
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/Makefile
@@ -0,0 +1,20 @@
+# src/test/modules/libpq_pipeline/Makefile
+
+PROGRAM = libpq_pipeline
+OBJS = libpq_pipeline.o
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS_INTERNAL += $(libpq_pgport)
+
+TAP_TESTS = 1
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/libpq_pipeline
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/libpq_pipeline/README b/src/test/modules/libpq_pipeline/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/libpq_pipeline/libpq_pipeline.c b/src/test/modules/libpq_pipeline/libpq_pipeline.c
new file mode 100644
index 0000000000..41706b514c
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/libpq_pipeline.c
@@ -0,0 +1,1141 @@
+/*
+ * src/test/modules/libpq_pipeline/libpq_pipeline.c
+ * Verify libpq pipeline execution functionality
+ */
+#include "postgres_fe.h"
+
+#include <sys/time.h>
+
+#include "catalog/pg_type_d.h"
+#include "common/fe_memutils.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+
+static void exit_nicely(PGconn *conn);
+
+const char *const progname = "libpq_pipeline";
+
+
+#define DEBUG
+#ifdef DEBUG
+#define pg_debug(...) do { fprintf(stderr, __VA_ARGS__); } while (0)
+#else
+#define pg_debug(...)
+#endif
+
+static const char *const drop_table_sql =
+"DROP TABLE IF EXISTS pq_pipeline_demo";
+static const char *const create_table_sql =
+"CREATE UNLOGGED TABLE pq_pipeline_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql =
+"INSERT INTO pq_pipeline_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+/*
+ * Print an error to stderr and terminate the program.
+ */
+#define pg_fatal(...) pg_fatal_impl(__LINE__, __VA_ARGS__)
+static void
+pg_fatal_impl(int line, const char *fmt,...)
+{
+ va_list args;
+
+ fprintf(stderr, "\n"); /* XXX hack */
+ fprintf(stderr, "%s:%d: ", progname, line);
+
+ va_start(args, fmt);
+ vfprintf(stderr, fmt, args);
+ va_end(args);
+ printf("Failure, exiting\n");
+ exit(1);
+}
+
+static void
+test_disallowed(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("Unable to enter pipeline mode\n");
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not activated properly\n");
+
+ /* PQexec should fail in pipeline mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("PQexec should fail in pipeline mode but succeeded\n");
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ pg_fatal("PQsendQuery should fail in pipeline mode but succeeded\n");
+
+ /* Entering pipeline mode when already in pipeline mode is OK */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("re-entering pipeline mode should be a no-op but failed\n");
+
+ if (PQisBusy(conn) != 0)
+ pg_fatal("PQisBusy should return 0 when idle in pipeline mode, returned 1\n");
+
+ /* ok, back to normal command mode */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("couldn't exit idle empty pipeline mode\n");
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not terminated properly\n");
+
+ /* exiting pipeline mode when not in pipeline mode should be a no-op */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("pipeline mode exit when not in pipeline mode should succeed but failed\n");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("PQexec should succeed after exiting pipeline mode but failed with: %s\n",
+ PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_simple_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple pipeline... ");
+
+ /*
+ * Enter pipeline mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our output buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode with work in progress should fail, but succeeded\n");
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending pipeline failed: %s\n", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first query result.\n");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit pipeline mode yet
+ */
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result PGRES_PIPELINE_SYNC expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of PGRES_PIPELINE_SYNC, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after pipeline end: %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* ... until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_multi_pipelines(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi pipeline... ");
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending first pipeline failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s\n", PQerrorMessage(conn));
+
+ /* OK, start processing the results */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first result\n");
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ PQclear(res);
+
+#if 0
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+#endif
+
+ /* second pipeline */
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from second pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode ... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+/*
+ * When an operation in a pipeline fails the rest of the pipeline is flushed. We
+ * still have to get results for each pipeline item, but the item will just be
+ * a PGRES_PIPELINE_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the pipeline. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_aborted_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted pipeline... ");
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * pipeline ERRORs.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT no_such_function($1)",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching error select failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending first pipeline failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second-pipeline insert failed: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the pipeline results.
+ *
+ * We should get a command-ok for the first query, then a fatal error and
+ * a pipeline aborted message for the second insert, a pipeline-end, then
+ * a command-ok and a pipeline-ok for the second pipeline operation.
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result status %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* Second query caused error, so we expect an error next */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("Unexpected result code -- expected PGRES_FATAL_ERROR, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /*
+ * pipeline should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third pipeline since we've already sent a
+ * second. The aborted flag relates only to the pipeline being received.
+ */
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't\n");
+
+ /* third query in pipeline, the second insert */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_ABORTED)
+ pg_fatal("Unexpected result code -- expected PGRES_PIPELINE_ABORTED, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+#if 0
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+#endif
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't\n");
+
+ /* Ensure we're still in pipeline */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /*
+ * The end of a failed pipeline is a PGRES_PIPELINE_SYNC.
+ *
+ * (This is so clients know to start processing results normally again and
+ * can tell the difference between skipped commands and the sync.)
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code from first pipeline sync\n"
+ "Expected PGRES_PIPELINE_SYNC, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+#if 0
+ /* after the synchronization point we get a NULL result */
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+#endif
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_ABORTED)
+ pg_fatal("sync should've cleared the aborted flag but didn't\n");
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* the insert from the second pipeline */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result code %s from first item in second pipeline\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* Read the NULL result at the end of the command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* the second pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQerrorMessage(conn));
+
+ /* Test single-row mode with an error partways */
+ if (PQsendQueryParams(conn, "SELECT 1/g FROM generate_series(5, -1, -1) g",
+ 0, NULL, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to send query: %s", PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ PQsetSingleRowMode(conn);
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ if (PQresultStatus(res) == PGRES_FATAL_ERROR)
+ printf("got error: {{%s}} (expected: division by zero)\n", PQerrorMessage(conn));
+ else if (PQresultStatus(res) == PGRES_SINGLE_TUPLE)
+ printf("got row: %s\n", PQgetvalue(res, 0, 0));
+ else
+ pg_fatal("got unexpected result %s\n", PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ }
+ /* the third pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from third pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ /*-
+ * Since we fired the pipelines off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st pipeline
+ * - First insert applied
+ * - Second statement aborted xact
+ * - Third insert skipped
+ * - Sync rolled back first implicit xact
+ * - Implicit xact created by server around 2nd pipeline
+ * - insert applied from 2nd pipeline
+ * - Sync commits 2nd xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM pq_pipeline_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Expected tuples, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("expected 1 result, got %d\n", PQntuples(res));
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ pg_fatal("expected only insert with value 3, got %s", val);
+ }
+
+ PQclear(res);
+}
+
+/* State machine enum for test_pipelined_insert */
+typedef enum PipelineInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} PipelineInsertStep;
+
+static void
+test_pipelined_insert(PGconn *conn, int n_rows)
+{
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ PipelineInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a pipelined insert into a table created at the start of the pipeline
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ while (send_step != BI_PREPARE)
+ {
+ const char *sql;
+
+ switch (send_step)
+ {
+ case BI_BEGIN_TX:
+ sql = "BEGIN TRANSACTION";
+ send_step = BI_DROP_TABLE;
+ break;
+
+ case BI_DROP_TABLE:
+ sql = drop_table_sql;
+ send_step = BI_CREATE_TABLE;
+ break;
+
+ case BI_CREATE_TABLE:
+ sql = create_table_sql;
+ send_step = BI_PREPARE;
+ break;
+
+ default:
+ pg_fatal("invalid state");
+ }
+
+ pg_debug("sending: %s\n", sql);
+ if (PQsendQueryParams(conn, sql,
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching %s failed: %s\n", sql, PQerrorMessage(conn));
+ }
+
+ Assert(send_step == BI_PREPARE);
+ pg_debug("sending: %s\n", insert_sql);
+ if (PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids) != 1)
+ pg_fatal("dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our output buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the pipeline before processing
+ * results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ pg_fatal("failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's output buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ PGresult *res;
+ const char *cmdtag;
+ const char *description = "";
+ int status;
+
+ /*
+ * Read next result. If no more results from this query,
+ * advance to the next query
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ continue;
+
+ status = PGRES_COMMAND_OK;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ recv_step++;
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ recv_step++;
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ recv_step++;
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ recv_step++;
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive == 0)
+ recv_step++;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ recv_step++;
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_PIPELINE_SYNC;
+ recv_step++;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ description = "";
+ abort();
+ }
+
+ if (PQresultStatus(res) != status)
+ pg_fatal("%s reported status %s, expected %s\n"
+ "Error message: \"%s\"\n",
+ description, PQresStatus(PQresultStatus(res)),
+ PQresStatus(status), PQerrorMessage(conn));
+
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ pg_fatal("%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+
+ pg_debug("Got %s OK\n", cmdtag[0] != '\0' ? cmdtag : description);
+
+ PQclear(res);
+ }
+ }
+
+ /* Write more rows and/or the end pipeline message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQsendPipeline(conn) == 1)
+ {
+ fprintf(stdout, "Sent pipeline\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending pipeline failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+ }
+
+ /* We've got the sync message and the pipeline should be done */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ pg_fatal("failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i;
+ bool pipeline_ended = false;
+
+ /* 1 pipeline, 3 queries in it */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n",
+ PQerrorMessage(conn));
+
+ for (i = 0; i < 3; i++)
+ {
+ char *param[1];
+
+ param[0] = psprintf("%d", 44 + i);
+
+ if (PQsendQueryParams(conn,
+ "SELECT generate_series(42, $1)",
+ 1,
+ NULL,
+ (const char **) param,
+ NULL,
+ NULL,
+ 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ pfree(param[0]);
+ }
+ PQsendPipeline(conn);
+
+ for (i = 0; !pipeline_ended; i++)
+ {
+ bool first = true;
+ bool saw_ending_tuplesok;
+ bool isSingleTuple = false;
+
+ /* Set single row mode for only first 2 SELECT queries */
+ if (i < 2)
+ {
+ if (PQsetSingleRowMode(conn) != 1)
+ pg_fatal("PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+
+ /* Consume rows for this query */
+ saw_ending_tuplesok = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+
+ if (est == PGRES_PIPELINE_SYNC)
+ {
+ fprintf(stderr, "end of pipeline reached\n");
+ pipeline_ended = true;
+ PQclear(res);
+ if (i != 3)
+ pg_fatal("Expected three results, got %d\n", i);
+ break;
+ }
+
+ /* Expect SINGLE_TUPLE for queries 0 and 1, TUPLES_OK for 2 */
+ if (first)
+ {
+ if (i <= 1 && est != PGRES_SINGLE_TUPLE)
+ pg_fatal("Expected PGRES_SINGLE_TUPLE for query %d, got %s\n",
+ i, PQresStatus(est));
+ if (i >= 2 && est != PGRES_TUPLES_OK)
+ pg_fatal("Expected PGRES_TUPLES_OK for query %d, got %s\n",
+ i, PQresStatus(est));
+ first = false;
+ }
+
+ fprintf(stderr, "Result status %s for query %d", PQresStatus(est), i);
+ switch (est)
+ {
+ case PGRES_TUPLES_OK:
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ saw_ending_tuplesok = true;
+ if (isSingleTuple)
+ {
+ if (PQntuples(res) == 0)
+ fprintf(stderr, "all tuples received in query %d\n", i);
+ else
+ pg_fatal("Expected to follow PGRES_SINGLE_TUPLE, "
+ "but received PGRES_TUPLES_OK directly instead\n");
+ }
+ break;
+
+ case PGRES_SINGLE_TUPLE:
+ isSingleTuple = true;
+ fprintf(stderr, ", %d tuple: %s\n", PQntuples(res), PQgetvalue(res, 0, 0));
+ break;
+
+ default:
+ pg_fatal("unexpected\n");
+ }
+ PQclear(res);
+ }
+ if (!pipeline_ended && !saw_ending_tuplesok)
+ pg_fatal("didn't get expected terminating TUPLES_OK\n");
+ }
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s\n", PQerrorMessage(conn));
+}
+
+/*
+ * Simple test to verify that a pipeline is discarded as a whole when there's
+ * an error, ignoring transaction commands.
+ */
+static void
+test_transaction(PGconn *conn)
+{
+ PGresult *res;
+ int num_sends = 0;
+
+ res = PQexec(conn, "DROP TABLE IF EXISTS pq_pipeline_tst;"
+ "CREATE TABLE pq_pipeline_tst (id int)");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to create test table: %s",
+ PQerrorMessage(conn));
+ PQclear(res);
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n",
+ PQerrorMessage(conn));
+ if (PQsendPrepare(conn, "rollback", "ROLLBACK", 0, NULL) != 1)
+ pg_fatal("could not send prepare on pipeline: %s",
+ PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn,
+ "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ if (PQsendQueryParams(conn,
+ "SELECT 0/0",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+
+ /*
+ * send a ROLLBACK using a prepared stmt. Doesn't work because we need to
+ * get out of the pipeline-aborted state first.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /* This insert fails because we're in pipeline-aborted state */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (1)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * This insert fails even though the pipeline got a SYNC, because we're in
+ * an aborted transaction
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (2)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * Send ROLLBACK using prepared stmt. This one works because we just did
+ * PQsendPipeline above.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /*
+ * Now that we're out of a transaction and in pipeline-good mode, this
+ * insert works
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (3)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+ PQsendPipeline(conn);
+ num_sends++;
+
+ for (int i = 0;; i++)
+ {
+ ExecStatusType restype;
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "%d: got NULL result\n", i);
+ continue;
+ }
+ restype = PQresultStatus(res);
+ fprintf(stderr, "%d: got status %s", i, PQresStatus(restype));
+ if (restype == PGRES_FATAL_ERROR)
+ fprintf(stderr, "; error: %s", PQerrorMessage(conn));
+ else if (restype == PGRES_PIPELINE_ABORTED)
+ {
+ fprintf(stderr, ": command didn't run because pipeline aborted\n");
+ }
+ else
+ fprintf(stderr, "\n");
+ PQclear(res);
+
+ if (restype == PGRES_PIPELINE_SYNC)
+ num_sends--;
+ if (num_sends <= 0)
+ break;
+ }
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("returned something extra after all the syncs: %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s\n", PQerrorMessage(conn));
+
+ /* We expect to find one tuple containing the value "3" */
+ res = PQexec(conn, "SELECT * FROM pq_pipeline_tst");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("failed to obtain result: %s", PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("did not get 1 tuple\n");
+ if (strcmp(PQgetvalue(res, 0, 0), "3") != 0)
+ pg_fatal("did not get expected tuple\n");
+ PQclear(res);
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+usage(const char *progname)
+{
+ fprintf(stderr, "%s tests libpq's pipeline mode.\n\n", progname);
+ fprintf(stderr, "Usage:\n");
+ fprintf(stderr, " %s testname [conninfo [number_of_rows]]\n", progname);
+ fprintf(stderr, "Tests:\n");
+ fprintf(stderr, " disallowed_in_pipeline\n");
+ fprintf(stderr, " simple_pipeline\n");
+ fprintf(stderr, " multi_pipeline\n");
+ fprintf(stderr, " pipeline_abort\n");
+ fprintf(stderr, " singlerow\n");
+ fprintf(stderr, " pipeline_insert\n");
+ fprintf(stderr, " transaction\n");
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo = "";
+ PGconn *conn;
+ int numrows = 10000;
+ PGresult *res;
+
+ /*
+ * The testname parameter is mandatory; it can be followed by a conninfo
+ * string and number of rows.
+ */
+ if (argc < 2 || argc > 4)
+ {
+ usage(argv[0]);
+ exit(1);
+ }
+
+ if (argc >= 3)
+ conninfo = pg_strdup(argv[2]);
+
+ if (argc >= 4)
+ {
+ errno = 0;
+ numrows = strtol(argv[3], NULL, 10);
+ if (errno != 0 || numrows <= 0)
+ {
+ fprintf(stderr, "couldn't parse \"%s\" as a positive integer\n", argv[3]);
+ exit(1);
+ }
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+ res = PQexec(conn, "SET lc_messages TO \"C\"");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to set lc_messages: %s", PQerrorMessage(conn));
+
+ if (strcmp(argv[1], "disallowed_in_pipeline") == 0)
+ test_disallowed(conn);
+ else if (strcmp(argv[1], "simple_pipeline") == 0)
+ test_simple_pipeline(conn);
+ else if (strcmp(argv[1], "multi_pipeline") == 0)
+ test_multi_pipelines(conn);
+ else if (strcmp(argv[1], "pipeline_abort") == 0)
+ test_aborted_pipeline(conn);
+ else if (strcmp(argv[1], "pipeline_insert") == 0)
+ test_pipelined_insert(conn, numrows);
+ else if (strcmp(argv[1], "singlerow") == 0)
+ test_singlerowmode(conn);
+ else if (strcmp(argv[1], "transaction") == 0)
+ test_transaction(conn);
+ else
+ {
+ fprintf(stderr, "\"%s\" is not a recognized test name\n", argv[1]);
+ usage(argv[0]);
+ exit(1);
+ }
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+ return 0;
+}
diff --git a/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
new file mode 100644
index 0000000000..a12f2dd47b
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
@@ -0,0 +1,31 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 7;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_pipeline
+ simple_pipeline
+ multi_pipeline
+ pipeline_abort
+ pipeline_insert
+ singlerow
+ transaction);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'libpq_pipeline', $testname, $node->connstr('postgres'), $numrows ],
+ "libp_pipeline $testname");
+}
+
+$node->stop('fast');
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 49614106dc..e3b14af9dd 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -33,9 +33,10 @@ my @unlink_on_exit;
# Set of variables for modules in contrib/ and src/test/modules/
my $contrib_defines = { 'refint' => 'REFINT_VERBOSE' };
-my @contrib_uselibpq = ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo');
-my @contrib_uselibpgport = ('oid2name', 'vacuumlo');
-my @contrib_uselibpgcommon = ('oid2name', 'vacuumlo');
+my @contrib_uselibpq =
+ ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo', 'libpq_pipeline');
+my @contrib_uselibpgport = ('libpq_pipeline', 'oid2name', 'vacuumlo');
+my @contrib_uselibpgcommon = ('libpq_pipeline', 'oid2name', 'vacuumlo');
my $contrib_extralibs = undef;
my $contrib_extraincludes = { 'dblink' => ['src/backend'] };
my $contrib_extrasource = {
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 8bd95aefa1..1bd23eec6d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1561,10 +1561,12 @@ PG_Locale_Strategy
PG_Lock_Status
PG_init_t
PGcancel
+PGcommandQueueEntry
PGconn
PGdataValue
PGlobjfuncs
PGnotify
+PGpipelineStatus
PGresAttDesc
PGresAttValue
PGresParamDesc
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
On 2021-Mar-04, Alvaro Herrera wrote:
I don't know where do __WSAFDIsSet and __imp_select come from or what to
do about them. Let's see if adding pgport and pgcommon fixes things.
Indeed all those other problems were fixed and these remain. New
failure is:
"C:\projects\postgresql\pgsql.sln" (default target) (1) ->
6007"C:\projects\postgresql\libpq_pipeline.vcxproj" (default target) (55) ->
6008(Link target) ->
6009 libpq_pipeline.obj : error LNK2019: unresolved external symbol __WSAFDIsSet referenced in function test_pipelined_insert [C:\projects\postgresql\libpq_pipeline.vcxproj]
6010 libpq_pipeline.obj : error LNK2019: unresolved external symbol __imp_select referenced in function test_pipelined_insert [C:\projects\postgresql\libpq_pipeline.vcxproj]
6011 .\Release\libpq_pipeline\libpq_pipeline.exe : fatal error LNK1120: 2 unresolved externals [C:\projects\postgresql\libpq_pipeline.vcxproj]
I did notice that isolationtester.c is using select(), and one
difference is that it includes <sys/select.h> which libpq_pipeline.c
does not -- and it also pulls in ws2_32.lib. Let's see if those two
changes fix things.
--
�lvaro Herrera Valdivia, Chile
Attachments:
v33-libpq-pipeline.patchtext/x-diff; charset=us-asciiDownload
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 0553279314..c87b0ce911 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -3173,6 +3173,33 @@ ExecStatusType PQresultStatus(const PGresult *res);
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-sync">
+ <term><literal>PGRES_PIPELINE_SYNC</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a
+ synchronization point in pipeline mode, requested by
+ <xref linkend="libpq-PQsendPipeline"/>.
+ This status occurs only when pipeline mode has been selected.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-aborted">
+ <term><literal>PGRES_PIPELINE_ABORTED</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a pipeline that has
+ received an error from the server. <function>PQgetResult</function>
+ must be called repeatedly, and each time it will return this status code
+ until the end of the current pipeline, at which point it will return
+ <literal>PGRES_PIPELINE_SYNC</literal> and normal processing can
+ resume.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
If the result status is <literal>PGRES_TUPLES_OK</literal> or
@@ -4919,6 +4946,498 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-pipeline-mode">
+ <title>Pipeline Mode</title>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>libpq</primary>
+ <secondary>pipeline mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>pipelining</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>batch mode</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> pipeline mode allows applications to
+ send a query without having to read the result of the previously
+ sent query. Taking advantage of the pipeline mode, a client will wait
+ less for the server, since multiple queries/results can be sent/
+ received in a single network transaction.
+ </para>
+
+ <para>
+ While pipeline mode provides a significant performance boost, writing
+ clients using the pipeline mode is more complex because it involves
+ managing a queue of pending queries and finding which result
+ corresponds to which query in the queue.
+ </para>
+
+ <para>
+ Pipeline mode also generally consumes more memory on both the client and server,
+ though careful and aggressive management of the send/receive queue can mitigate
+ this. This applies whether or not the connection is in blocking or non-blocking
+ mode.
+ </para>
+
+ <sect2 id="libpq-pipeline-using">
+ <title>Using Pipeline Mode</title>
+
+ <para>
+ To issue pipelines, the application must switch a connection into pipeline mode.
+ Enter pipeline mode with <xref linkend="libpq-PQenterPipelineMode"/>
+ or test whether pipeline mode is active with
+ <xref linkend="libpq-PQpipelineStatus"/>.
+ In pipeline mode, only <link linkend="libpq-async">asynchronous operations</link>
+ are permitted, and <literal>COPY</literal> is disallowed.
+ Using any synchronous command execution functions,
+ such as <function>PQfn</function>, or <function>PQexec</function>
+ and its sibling functions, is an error condition.
+ </para>
+
+ <note>
+ <para>
+ It is best to use pipeline mode with <application>libpq</application> in
+ <link linkend="libpq-PQsetnonblocking">non-blocking mode</link>. If used
+ in blocking mode it is possible for a client/server deadlock to occur.
+ <footnote>
+ <para>
+ The client will block trying to send queries to the server, but the
+ server will block trying to send results to the client from queries
+ it has already processed. This only occurs when the client sends
+ enough queries to fill both its output buffer and the server's receive
+ buffer before it switches to processing input from the server,
+ but it's hard to predict exactly when that will happen.
+ </para>
+ </footnote>
+ </para>
+ </note>
+
+ <sect3 id="libpq-pipeline-sending">
+ <title>Issuing Queries</title>
+
+ <para>
+ After entering pipeline mode, the application dispatches requests using
+ <xref linkend="libpq-PQsendQueryParams"/>,
+ or its prepared-query sibling
+ <xref linkend="libpq-PQsendQueryPrepared"/>.
+ These requests are queued on the client-side until flushed to the server;
+ this occurs when <xref linkend="libpq-PQsendPipeline"/> is used to
+ establish a synchronization point in the pipeline,
+ or when <xref linkend="libpq-PQflush"/> is called.
+ The functions <xref linkend="libpq-PQsendPrepare"/>,
+ <xref linkend="libpq-PQsendDescribePrepared"/>, and
+ <xref linkend="libpq-PQsendDescribePortal"/> also work in pipeline mode.
+ Result processing is described below.
+ </para>
+
+ <para>
+ The server executes statements, and returns results, in the order the
+ client sends them. The server will begin executing the commands in the
+ pipeline immediately, not waiting for the end of the pipeline.
+ If any statement encounters an error, the server aborts the current
+ transaction and skips processing commands in the pipeline until the
+ next synchronization point established by <function>PQsendPipeline</function>.
+ (This remains true even if the commands in the pipeline would rollback
+ the transaction.)
+ Query processing resumes after the synchronization point.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one; for example, one query may define a table that the next
+ query in the same pipeline uses. Similarly, an application may
+ create a named prepared statement and execute it with later
+ statements in the same pipeline.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-results">
+ <title>Processing Results</title>
+
+ <para>
+ To process the result of one query in a pipeline, the application calls
+ <function>PQgetResult</function> repeatedly and handles each result
+ until <function>PQgetResult</function> returns null.
+ The result from the next query in the pipeline may then be retrieved using
+ <function>PQgetResult</function> again and the cycle repeated.
+ The application handles individual statement results as normal.
+ When the results of all the queries in the pipeline have been
+ returned, <function>PQgetResult</function> returns a result
+ containing the status value <literal>PGRES_PIPELINE_SYNC</literal>.
+ </para>
+
+ <para>
+ The client may choose to defer result processing until the complete
+ pipeline has been sent, or interleave that with sending further
+ queries in the pipeline; see <xref linkend="libpq-pipeline-interleave" />.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ This mode selection is effective only for the query currently
+ being processed. For more information on the use of
+ <function>PQsetSingleRowMode</function>,
+ refer to <xref linkend="libpq-single-row-mode"/>.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal
+ asynchronous processing except that it may contain the new
+ <type>PGresult</type> types <literal>PGRES_PIPELINE_SYNC</literal>
+ and <literal>PGRES_PIPELINE_ABORTED</literal>.
+ <literal>PGRES_PIPELINE_SYNC</literal> is reported exactly once for each
+ <function>PQsendPipeline</function> after retrieving results for all
+ queries in the pipeline.
+ <literal>PGRES_PIPELINE_ABORTED</literal> is emitted in place of a normal
+ stream result for the first error and all subsequent results
+ except <literal>PGRES_PIPELINE_SYNC</literal> and null;
+ see <xref linkend="libpq-pipeline-errors"/>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing pipeline results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed (except that
+ <function>PQgetResult</function> returns null to indicate that we start
+ returning the results of next query). The application must keep track
+ of the order in which it sent queries, to associate them with their
+ corresponding results.
+ Applications will typically use a state machine or a FIFO queue for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-pipeline-errors">
+ <title>Error Handling</title>
+
+ <para>
+ When a query in a pipeline causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the pipeline
+ synchronization message. The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after <function>PQresultStatus</function>
+ returns <literal>PGRES_FATAL_ERROR</literal>,
+ the pipeline is flagged as aborted.
+ <function>PQresultStatus</function>, will report a
+ <literal>PGRES_PIPELINE_ABORTED</literal> result for each remaining queued
+ operation in an aborted pipeline. The result for
+ <function>PQsendPipeline</function> is reported as
+ <literal>PGRES_PIPELINE_SYNC</literal> to signal the end of the aborted pipeline
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the pipeline used an implicit transaction, then operations that have
+ already executed are rolled back and operations that were queued to follow
+ the failed operation are skipped entirely. The same behaviour holds if the
+ pipeline starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the pipeline. If a pipeline contains
+ <emphasis>multiple explicit transactions</emphasis>, all transactions that
+ committed prior to the error remain committed, the currently in-progress
+ transaction is aborted, and all subsequent operations are skipped completely,
+ including subsequent transactions.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal> — only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously, the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-interleave">
+ <title>Interleaving Result Processing and Query Dispatch</title>
+
+ <para>
+ To avoid deadlocks on large pipelines the client should be structured
+ around a non-blocking event loop using operating system facilities
+ such as <function>select</function>, <function>poll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work
+ remaining to be dispatched and a queue of work that has been dispatched
+ but not yet had its results processed. When the socket is writable
+ it should dispatch more work. When the socket is readable it should
+ read results and process them, matching them up to the next entry in
+ its expected results queue. Based on available memory, results from the
+ socket should be read frequently: there's no need to wait until the
+ pipeline end to read the results. Pipelines should be scoped to logical
+ units of work, usually (but not necessarily) one transaction per pipeline.
+ There's no need to exit pipeline mode and re-enter it between pipelines,
+ or to wait for one pipeline to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state
+ machine to track sent and received work is in
+ <filename>src/test/modules/libpq_pipeline/libpq_pipeline.c</filename>
+ in the PostgreSQL source distribution.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-end">
+ <title>Ending Pipeline Mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed, and
+ the end pipeline result has been consumed, the application may return
+ to non-pipelined mode with <xref linkend="libpq-PQexitPipelineMode"/>.
+ </para>
+ </sect3>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-functions">
+ <title>Functions Associated with Pipeline Mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQpipelineStatus">
+ <term>
+ <function>PQpipelineStatus</function>
+ <indexterm>
+ <primary>PQpipelineStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the current pipeline mode status of the
+ <application>libpq</application> connection.
+<synopsis>
+PGpipelineStatus PQpipelineStatus(const PGconn *conn);
+</synopsis>
+ </para>
+
+ <para>
+ <function>PQpipelineStatus</function> can return one of the following values:
+
+ <variablelist>
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ON</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in
+ pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_OFF</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is
+ <emphasis>not</emphasis> in pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in pipeline
+ mode and an error occurred while processing the current pipeline.
+ The aborted flag is cleared when <function>PQresultStatus</function>
+ returns PGRES_PIPELINE_SYNC at the end of the pipeline.
+ Clients don't usually need this function to
+ verify aborted status, as they can tell that the pipeline is aborted
+ from the <literal>PGRES_PIPELINE_ABORTED</literal> result code.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterPipelineMode">
+ <term>
+ <function>PQenterPipelineMode</function>
+ <indexterm>
+ <primary>PQenterPipelineMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter pipeline mode if it is currently idle or
+ already in pipeline mode.
+
+<synopsis>
+int PQenterPipelineMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success.
+ Returns 0 and has no effect if the connection is not currently
+ idle, i.e., it has a result ready, or it is waiting for more
+ input from the server, etc.
+ This function does not actually send anything to the server,
+ it just changes the <application>libpq</application> connection
+ state.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitPipelineMode">
+ <term>
+ <function>PQexitPipelineMode</function>
+ <indexterm>
+ <primary>PQexitPipelineMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit pipeline mode if it is currently in pipeline mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitPipelineMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 1 and takes no action if not in
+ pipeline mode. If the current statement isn't finished processing
+ or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQsendPipeline">
+ <term>
+ <function>PQsendPipeline</function>
+ <indexterm>
+ <primary>PQsendPipeline</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Marks a synchronization point in a pipeline by sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ and flushing the send buffer. This serves as
+ the delimiter of an implicit transaction and an error recovery
+ point; see <xref linkend="libpq-pipeline-errors"/>.
+
+<synopsis>
+int PQsendPipeline(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 if the connection is not in
+ pipeline mode or sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ failed.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-tips">
+ <title>When to Use Pipeline Mode</title>
+
+ <para>
+ Much like asynchronous query mode, there is no meaningful performance
+ overhead when using pipeline mode. It increases client application complexity,
+ and extra caution is required to prevent client/server deadlocks, but
+ pipeline mode can offer considerable performance improvements, in exchange for
+ increased memory usage from leaving state around longer.
+ </para>
+
+ <para>
+ Pipeline mode is most useful when the server is distant, i.e., network latency
+ (<quote>ping time</quote>) is high, and also when many small operations
+ are being performed in rapid succession. There is usually less benefit
+ in using pipelined commands when each query takes many multiples of the client/server
+ round-trip time to execute. A 100-statement operation run on a server
+ 300ms round-trip-time away would take 30 seconds in network latency alone
+ without pipelining; with pipelining it may spend as little as 0.3s waiting for
+ results from the server.
+ </para>
+
+ <para>
+ Use pipelined commands when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed
+ into operations on sets, or into a <literal>COPY</literal> operation.
+ </para>
+
+ <para>
+ Pipeline mode is not useful when information from one operation is required by
+ the client to produce the next operation. In such cases, the client
+ would have to introduce a synchronization point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+BEGIN;
+SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+-- result: x=2
+-- client adds 1 to x:
+UPDATE mytable SET x = 3 WHERE id = 42;
+COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <para>
+ Pipelining is less useful, and more complex, when a single pipeline contains
+ multiple transactions (see <xref linkend="libpq-pipeline-errors"/>).
+ </para>
+
+ <note>
+ <para>
+ The pipeline API was introduced in <productname>PostgreSQL</productname> 14.
+ Pipeline mode is a client-side feature which doesn't require server
+ support, and works on any server that supports the v3 extended query
+ protocol.
+ </para>
+ </note>
+ </sect2>
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4959,6 +5478,13 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <para>
+ When using pipeline mode, single-row mode needs to be activated for each
+ query in the pipeline before retrieving results for that query
+ with <function>PQgetResult</function>.
+ See <xref linkend="libpq-pipeline-mode"/> for more information.
+ </para>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 6d46da42e2..012e44c736 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while a libpq connection is in pipeline mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 2ec0580a79..75448162ca 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -1109,6 +1109,12 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
row, the last value is kept.
</para>
+ <para>
+ <literal>\gset</literal> and <literal>\aset</literal> cannot be used
+ pipeline mode, since query results are not immediately
+ fetched in this mode.
+ </para>
+
<para>
The following example puts the final account balance from the first query
into variable <replaceable>abalance</replaceable>, and fills variables
@@ -1269,6 +1275,21 @@ SELECT 4 AS four \; SELECT 5 AS five \aset
</programlisting></para>
</listitem>
</varlistentry>
+
+ <varlistentry id='pgbench-metacommand-pipeline'>
+ <term><literal>\startpipeline</literal></term>
+ <term><literal>\endpipeline</literal></term>
+
+ <listitem>
+ <para>
+ These commands delimit the start and end of a pipeline of SQL statements.
+ In a pipeline, statements are sent to server without waiting for the results
+ of previous statements (see <xref linkend="libpq-pipeline-mode"/>).
+ Pipeline mode requires the extended query protocol.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect2>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 5272eed9ab..f74378110a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1019,6 +1019,12 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->err = _("empty query");
break;
+ case PGRES_PIPELINE_SYNC:
+ case PGRES_PIPELINE_ABORTED:
+ walres->status = WALRCV_ERROR;
+ walres->err = _("unexpected pipeline mode");
+ break;
+
case PGRES_NONFATAL_ERROR:
case PGRES_FATAL_ERROR:
case PGRES_BAD_RESPONSE:
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 31a4df45f5..fbbe270654 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -351,10 +351,11 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
- * and we enter the CSTATE_SLEEP state to wait for it to expire. Other
- * meta-commands are executed immediately. If the command about to start
- * is actually beyond the end of the script, advance to CSTATE_END_TX.
+ * CSTATE_WAIT_RESULT state unless in pipeline mode. On a \sleep
+ * meta-command, the timer is set, and we enter the CSTATE_SLEEP state to
+ * wait for it to expire. Other meta-commands are executed immediately. If
+ * the command about to start is actually beyond the end of the script,
+ * advance to CSTATE_END_TX.
*
* CSTATE_WAIT_RESULT waits until we get a result set back from the server
* for the current command.
@@ -484,7 +485,9 @@ typedef enum MetaCommand
META_IF, /* \if */
META_ELIF, /* \elif */
META_ELSE, /* \else */
- META_ENDIF /* \endif */
+ META_ENDIF, /* \endif */
+ META_STARTPIPELINE, /* \startpipeline */
+ META_ENDPIPELINE /* \endpipeline */
} MetaCommand;
typedef enum QueryMode
@@ -2504,6 +2507,10 @@ getMetaCommand(const char *cmd)
mc = META_GSET;
else if (pg_strcasecmp(cmd, "aset") == 0)
mc = META_ASET;
+ else if (pg_strcasecmp(cmd, "startpipeline") == 0)
+ mc = META_STARTPIPELINE;
+ else if (pg_strcasecmp(cmd, "endpipeline") == 0)
+ mc = META_ENDPIPELINE;
else
mc = META_NONE;
return mc;
@@ -2693,11 +2700,25 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In pipeline mode, we use asynchronous functions. If a
+ * server-side error occurs, it will be processed later
+ * among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ pg_log_error("%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2741,8 +2762,10 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
* varprefix should be set only with \gset or \aset, and SQL commands do
* not need it.
*/
+#if 0
Assert((meta == META_NONE && varprefix == NULL) ||
((meta == META_GSET || meta == META_ASET) && varprefix != NULL));
+#endif
res = PQgetResult(st->con);
@@ -2810,6 +2833,12 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* otherwise the result is simply thrown away by PQclear below */
break;
+ case PGRES_PIPELINE_SYNC:
+ pg_log_debug("client %d pipeline ending", st->id);
+ if (PQexitPipelineMode(st->con) != 1)
+ pg_log_error("client %d failed to exit pipeline mode", st->id);
+ break;
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3066,13 +3095,36 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
/* Execute the command */
if (command->type == SQL_COMMAND)
{
+ /* disallow \aset and \gset in pipeline mode */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ if (command->meta == META_GSET)
+ {
+ commandFailed(st, "gset", "\\gset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ else if (command->meta == META_ASET)
+ {
+ commandFailed(st, "aset", "\\aset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ }
+
if (!sendCommand(st, command))
{
commandFailed(st, "SQL", "SQL command send failed");
st->state = CSTATE_ABORTED;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in pipeline mode */
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -3212,7 +3264,15 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (readCommandResponse(st,
sql_script[st->use_file].commands[st->command]->meta,
sql_script[st->use_file].commands[st->command]->varprefix))
- st->state = CSTATE_END_COMMAND;
+ {
+ /*
+ * outside of pipeline mode: stop reading results.
+ * pipeline mode: continue reading results until an
+ * end-of-pipeline response.
+ */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ st->state = CSTATE_END_COMMAND;
+ }
else
st->state = CSTATE_ABORTED;
break;
@@ -3456,6 +3516,49 @@ executeMetaCommand(CState *st, instr_time *now)
return CSTATE_ABORTED;
}
}
+ else if (command->meta == META_STARTPIPELINE)
+ {
+ /*
+ * In pipeline mode, we use a workflow based on libpq pipeline
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "startpipeline", "cannot use pipeline mode with the simple query protocol");
+ return CSTATE_ABORTED;
+ }
+
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ commandFailed(st, "startpipeline", "already in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (PQenterPipelineMode(st->con) == 0)
+ {
+ commandFailed(st, "startpipeline", "failed to enter pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ }
+ else if (command->meta == META_ENDPIPELINE)
+ {
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ {
+ commandFailed(st, "endpipeline", "not in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (!PQsendPipeline(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to send the pipeline");
+ return CSTATE_ABORTED;
+ }
+ if (!PQexitPipelineMode(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to exit pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ /* collect pending results before getting out of pipeline mode */
+ return CSTATE_WAIT_RESULT;
+ }
/*
* executing the expression or shell command might have taken a
@@ -4683,7 +4786,9 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
- else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
+ else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF ||
+ my_command->meta == META_STARTPIPELINE ||
+ my_command->meta == META_ENDPIPELINE)
{
if (my_command->argc != 1)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
@@ -6808,4 +6913,5 @@ pthread_join(pthread_t th, void **thread_return)
return 0;
}
+
#endif /* WIN32 */
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..60d09e6d63 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,7 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterPipelineMode 180
+PQexitPipelineMode 181
+PQsendPipeline 182
+PQpipelineStatus 183
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index f83af03d0a..4c97e8a078 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -540,6 +540,23 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * pqFreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+pqFreeCommandQueue(PGcommandQueueEntry *queue)
+{
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *cur = queue;
+
+ queue = cur->next;
+ if (cur->query)
+ free(cur->query);
+ free(cur);
+ }
+}
/*
* pqDropServerData
@@ -571,6 +588,12 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ pqFreeCommandQueue(conn->cmd_queue_head);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ pqFreeCommandQueue(conn->cmd_queue_recycle);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
@@ -2486,6 +2509,7 @@ keep_going: /* We will come back to here until there is
/* Drop any PGresult we might have, too */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn);
/* Reset conn->status to put the state machine in the right state */
@@ -3928,6 +3952,7 @@ makeEmptyPGconn(void)
conn->status = CONNECTION_BAD;
conn->asyncStatus = PGASYNC_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
conn->xactStatus = PQTRANS_IDLE;
conn->options_valid = false;
conn->nonblocking = false;
@@ -4187,6 +4212,7 @@ closePGconn(PGconn *conn)
conn->status = CONNECTION_BAD; /* Well, not really _bad_ - just absent */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn); /* deallocate result */
resetPQExpBuffer(&conn->errorMessage);
release_conn_addrinfo(conn);
@@ -6735,6 +6761,15 @@ PQbackendPID(const PGconn *conn)
return conn->be_pid;
}
+PGpipelineStatus
+PQpipelineStatus(const PGconn *conn)
+{
+ if (!conn)
+ return PQ_PIPELINE_OFF;
+
+ return conn->pipelineStatus;
+}
+
int
PQconnectionNeedsPassword(const PGconn *conn)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9a038043b2..0fb790fdfa 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_PIPELINE_SYNC",
+ "PGRES_PIPELINE_ABORTED"
};
/*
@@ -71,6 +73,11 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *pqMakePipelineCmd(PGconn *conn);
+static void pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqPipelineProcessQueue(PGconn *conn);
+static int pqPipelineFlush(PGconn *conn);
/* ----------------
@@ -1171,7 +1178,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1197,18 +1204,37 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot use %s in pipeline mode, use PQsendQueryParams\n"),
+ "PQsendQuery");
+ return 0;
+ }
+
return PQsendQueryInternal(conn, query, true);
}
int
PQsendQueryContinue(PGconn *conn, const char *query)
{
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot use %s in pipeline mode, use PQsendQueryParams\n"),
+ "PQsendQueryContinue");
+ return 0;
+ }
+
return PQsendQueryInternal(conn, query, false);
}
static int
PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
{
+ /* pipeline mode requires extended query protocol */
+ Assert(conn->pipelineStatus == PQ_PIPELINE_OFF);
+
if (!PQsendQueryStart(conn, newQuery))
return 0;
@@ -1307,6 +1333,8 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+
if (!PQsendQueryStart(conn, true))
return 0;
@@ -1330,6 +1358,15 @@ PQsendPrepare(PGconn *conn,
return 0;
}
+ /* Alloc pipeline memory before doing anything */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+ }
+
/* construct the Parse message */
if (pqPutMsgStart('P', conn) < 0 ||
pqPuts(stmtName, conn) < 0 ||
@@ -1356,32 +1393,46 @@ PQsendPrepare(PGconn *conn,
if (pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /*
+ * In non-pipeline mode, add a Sync and prepare to send. In pipeline mode
+ * we just keep track of the new message.
+ */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ /* remember we are doing just a Parse */
+ conn->queryclass = PGQUERY_PREPARE;
- /* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
-
- /* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ /* and remember the query text too, if possible */
+ /* if insufficient memory, last_query just winds up NULL */
+ if (conn->last_query)
+ free(conn->last_query);
+ conn->last_query = strdup(query);
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ else
+ {
+ pipeCmd->queryclass = PGQUERY_PREPARE;
+ /* as above, if insufficient memory, query winds up NULL */
+ pipeCmd->query = strdup(query);
+ pqAppendPipelineCmd(conn, pipeCmd);
+ }
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
- /* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1429,7 +1480,8 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn, bool newQuery)
@@ -1450,20 +1502,57 @@ PQsendQueryStart(PGconn *conn, bool newQuery)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE &&
+ conn->pipelineStatus == PQ_PIPELINE_OFF)
{
appendPQExpBufferStr(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * When enqueuing commands we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when pqPipelineProcessQueue()
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ }
+ }
+ else
+ {
+ /*
+ * This command's results will come in immediately. Initialize async
+ * result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1487,10 +1576,34 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **query;
+ PGQueryClass *queryclass;
/*
- * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
- * using specified statement name and the unnamed portal.
+ * Decide where the query is going to be stored. In pipeline mode, we
+ * allocate a new pipeline element; in non-pipeline mode, it's simply the
+ * connection's last query.
+ */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+ /*
+ * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync
+ * (if not in pipeline mode), using specified statement name and the
+ * unnamed portal.
*/
if (command)
@@ -1600,35 +1713,43 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /* construct the Sync message if not in pipeline mode */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ /* if insufficient memory, query just winds up NULL */
+ if (*query)
+ free(*query);
if (command)
- conn->last_query = strdup(command);
+ *query = strdup(command);
else
- conn->last_query = NULL;
+ *query = NULL;
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1726,14 +1847,17 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
-
/*
* PQgetResult
* Get the next PGresult produced by a query. Returns NULL if no
* query work remains or an error has occurred (e.g. out of
* memory).
+ *
+ * In pipeline mode, once all the result of a query have been returned,
+ * PQgetResult returns NULL to let the user know that the next
+ * query is being processed. At the end of the pipeline, returns a
+ * result with PQresultStatus(result) == PGRES_PIPELINE_SYNC.
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1803,8 +1927,49 @@ PQgetResult(PGconn *conn)
{
case PGASYNC_IDLE:
res = NULL; /* query is complete */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to return the NULL that terminates the round of
+ * results from the current query; prepare to send the results
+ * of the next query when we're called next. Also, since this
+ * is the start of the results of the next query, clear any
+ * prior error message.
+ */
+ resetPQExpBuffer(&conn->errorMessage);
+ pqPipelineProcessQueue(conn);
+ }
break;
case PGASYNC_READY:
+ res = pqPrepareAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to send the results of the current query. Set
+ * us idle now, and ...
+ */
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /*
+ * ... in cases when we're sending a pipeline-related result,
+ * move queue processing forwards immediately, so that next
+ * time we're called, we're prepared to return the next result
+ * received from the server.
+ *
+ * In all other cases, leave the queue state change for next
+ * time, so that a terminating NULL result is sent.
+ */
+ if (res && (res->resultStatus == PGRES_PIPELINE_ABORTED ||
+ res->resultStatus == PGRES_PIPELINE_SYNC))
+ pqPipelineProcessQueue(conn);
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -1985,6 +2150,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("synchronous command execution functions are not allowed in pipeline mode\n"));
+ return false;
+ }
+
/*
* Since this is the beginning of a query cycle, reset the error buffer.
*/
@@ -2148,6 +2320,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2155,6 +2330,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
if (!PQsendQueryStart(conn, true))
return 0;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ queryclass = &conn->queryclass;
+
/* construct the Describe message */
if (pqPutMsgStart('D', conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2163,32 +2350,40 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ /* reset last-query string (not relevant now) */
+ if (conn->last_query && conn->pipelineStatus != PQ_PIPELINE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
}
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -2541,6 +2736,13 @@ PQfn(PGconn *conn,
*/
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("PQfn not allowed in pipeline mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -2555,6 +2757,359 @@ PQfn(PGconn *conn,
args, nargs);
}
+/* ====== Pipeline mode support ======== */
+
+/*
+ * PQenterPipelineMode
+ * Put an idle connection in pipeline mode.
+ *
+ * Returns 1 on success. On failure, errorMessage is set and 0 is returned.
+ *
+ * Commands submitted after this can be pipelined on the connection;
+ * there's no requirement to wait for one to finish before the next is
+ * dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQsendPipeline. Multiple pipelines
+ * can be sent while in pipeline mode. Pipeline mode can be exited
+ * by calling PQexitPipelineMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ /* succeed with no action if already in pipeline mode */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot enter pipeline mode, connection not idle\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+
+ return 1;
+}
+
+/*
+ * PQexitPipelineMode
+ * End pipeline mode and return to normal command mode.
+ *
+ * Returns 1 in success (pipeline mode successfully ended, or not in pipeline
+ * mode).
+ *
+ * Returns 0 if in pipeline mode and cannot be ended yet. Error message will
+ * be set.
+ */
+int
+PQexitPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ /* there are some uncollected results */
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+
+ case PGASYNC_BUSY:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode while busy\n"));
+ return 0;
+
+ default:
+ /* OK */
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ return 0; /* error message is setup already */
+ return 1;
+}
+
+/*
+ * pqPipelineProcessQueue: subroutine for PQgetResult
+ * In pipeline mode, start processing the results of the next query in the queue.
+ */
+static void
+pqPipelineProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ Assert(conn->pipelineStatus != PQ_PIPELINE_OFF);
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: COPY in pipeline mode\n");
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return;
+ case PGASYNC_IDLE:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In pipeline mode but nothing left on the queue; caller can submit
+ * more work or PQexitPipelineMode() now.
+ */
+ return;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-pipeline call.
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /*
+ * Reset single-row processing mode. (Client has to set it up for each
+ * query, if desired.)
+ */
+ conn->singleRowMode = false;
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ pqRecyclePipelineCmd(conn, next_query);
+
+ if (conn->pipelineStatus == PQ_PIPELINE_ABORTED &&
+ conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted pipeline we don't get anything from the server for
+ * each result; we're just discarding input until we get to the next
+ * sync from the server. The client needs to know its queries got
+ * aborted so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn, PGRES_PIPELINE_ABORTED);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ pqSaveErrorResult(conn);
+ return;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+}
+
+/*
+ * PQsendPipeline
+ * Send a Sync message as part of a pipeline, and flush to server
+ *
+ * It's legal to start submitting more commands in the pipeline immediately,
+ * without waiting for the results of the current pipeline. There's no need to
+ * end pipeline mode and start it again.
+ *
+ * If a command in a pipeline fails, every subsequent command up to and including
+ * the result to the Sync message sent by PQsendPipeline gets set to
+ * PGRES_PIPELINE_ABORTED state. If the whole pipeline is processed without
+ * error, a PGresult with PGRES_PIPELINE_SYNC is produced.
+ *
+ * Queries can already have been sent before PQsendPipeline is called, but
+ * PQsendPipeline need to be called before retrieving command results.
+ *
+ * The connection will remain in pipeline mode and unavailable for new
+ * synchronous command execution functions until all results from the pipeline
+ * are processed by the client.
+ */
+int
+PQsendPipeline(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot send pipeline when not in pipeline mode\n"));
+ return 0;
+ }
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: cannot send pipeline while in COPY\n");
+ return 0;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_IDLE:
+ /* OK to send sync */
+ break;
+ }
+
+ entry = pqMakePipelineCmd(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ pqAppendPipelineCmd(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ /*
+ * Call pqPipelineProcessQueue so the user can call start calling
+ * PQgetResult.
+ */
+ pqPipelineProcessQueue(conn);
+
+ return 1;
+
+sendFailed:
+ pqRecyclePipelineCmd(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
+
+/*
+ * pqMakePipelineCmd
+ * Get a command queue entry for caller to fill.
+ *
+ * If the recycle queue has a free element, that is returned; if not, a
+ * fresh one is allocated. Caller is responsible for adding it to the
+ * command queue (pqAppendPipelineCmd) once the struct is filled in, or
+ * releasing the memory (pqRecyclePipelineCmd) if an error occurs.
+ *
+ * If allocation fails, sets the error message and returns NULL.
+ */
+static PGcommandQueueEntry *
+pqMakePipelineCmd(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * pqAppendPipelineCmd
+ * Append a caller-allocated command queue entry to the queue.
+ *
+ * The query itself must already have been put in the output buffer by the
+ * caller.
+ */
+static void
+pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ PGcommandQueueEntry **tail;
+
+ if (conn->cmd_queue_head == NULL)
+ tail = &conn->cmd_queue_head;
+ else
+ tail = &conn->cmd_queue_tail->next;
+
+ *tail = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * pqRecyclePipelineCmd
+ * Push a command queue entry onto the freelist. It must be an entry
+ * with null next pointer and not referenced by any other entry's next
+ * pointer.
+ */
+static void
+pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+
+ Assert(entry->next == NULL);
+
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
/* ====== accessor funcs for PGresult ======== */
@@ -3152,6 +3707,23 @@ PQflush(PGconn *conn)
return pqFlush(conn);
}
+/*
+ * pqPipelineFlush
+ *
+ * In pipeline mode, data will be flushed only when the out buffer reaches the
+ * threshold value. In non-pipeline mode, it behaves as stock pqFlush.
+ *
+ * Returns 0 on success.
+ */
+static int
+pqPipelineFlush(PGconn *conn)
+{
+ if ((conn->pipelineStatus == PQ_PIPELINE_OFF) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return pqFlush(conn);
+ return 0;
+}
+
/*
* PQfreemem - safely frees memory allocated
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 2ca8c057b9..b131995974 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -177,14 +177,24 @@ pqParseInput3(PGconn *conn)
if (getParameterStatus(conn))
return;
}
- else
- {
- pqInternalNotice(&conn->noticeHooks,
- "message type 0x%02x arrived from server while idle",
- id);
- /* Discard the unexpected message */
- conn->inCursor += msgLength;
- }
+
+ /*
+ * We're also IDLE when in pipeline mode we have completed
+ * processing the results of one query and are waiting for the
+ * next one in the pipeline. In this case, as above, just wait to
+ * see what's next.
+ */
+ if (conn->asyncStatus == PGASYNC_IDLE &&
+ conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->cmd_queue_head != NULL)
+ return;
+
+ /* Any other case is unexpected and we summarily skip it */
+ pqInternalNotice(&conn->noticeHooks,
+ "message type 0x%02x arrived from server while idle",
+ id);
+ /* Discard the unexpected message */
+ conn->inCursor += msgLength;
}
else
{
@@ -217,10 +227,28 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new
+ * query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_PIPELINE_SYNC);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ else
+ {
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -450,7 +478,7 @@ handleSyncLoss(PGconn *conn, char id, int msgLength)
id, msgLength);
/* build an error result holding the error message */
pqSaveErrorResult(conn);
- conn->asyncStatus = PGASYNC_READY; /* drop out of GetResult wait loop */
+ conn->asyncStatus = PGASYNC_READY; /* drop out of PQgetResult wait loop */
/* flush input data since we're giving up on processing it */
pqDropConnection(conn, true);
conn->status = CONNECTION_BAD; /* No more connection to backend */
@@ -875,6 +903,10 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ /* If in pipeline mode, set error indicator for it */
+ if (isError && conn->pipelineStatus != PQ_PIPELINE_OFF)
+ conn->pipelineStatus = PQ_PIPELINE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
@@ -930,9 +962,21 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
* Save the active query text, if any, into res as well; but only if we
* might need it for an error cursor display, which is only true if there
* is a PG_DIAG_STATEMENT_POSITION field.
+ *
+ * Note that in pipeline mode, we have not yet advanced the query pointer
+ * to the next query, so we have to look at that.
*/
- if (have_position && conn->last_query && res)
- res->errQuery = pqResultStrdup(res, conn->last_query);
+ if (have_position && res)
+ {
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->asyncStatus == PGASYNC_IDLE)
+ {
+ if (conn->cmd_queue_head)
+ res->errQuery = pqResultStrdup(res, conn->cmd_queue_head->query);
+ }
+ else if (conn->last_query)
+ res->errQuery = pqResultStrdup(res, conn->last_query);
+ }
/*
* Now build the "overall" error message for PQresultErrorMessage.
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index fa9b62a844..fb31a49fab 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -96,7 +96,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_PIPELINE_SYNC, /* pipeline synchronization point */
+ PGRES_PIPELINE_ABORTED, /* Command didn't run because of an abort
+ * earlier in a pipeline */
} ExecStatusType;
typedef enum
@@ -136,6 +139,16 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PGpipelineStatus - Current status of pipeline mode
+ */
+typedef enum
+{
+ PQ_PIPELINE_OFF,
+ PQ_PIPELINE_ON,
+ PQ_PIPELINE_ABORTED
+} PGpipelineStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -327,6 +340,7 @@ extern int PQserverVersion(const PGconn *conn);
extern char *PQerrorMessage(const PGconn *conn);
extern int PQsocket(const PGconn *conn);
extern int PQbackendPID(const PGconn *conn);
+extern PGpipelineStatus PQpipelineStatus(const PGconn *conn);
extern int PQconnectionNeedsPassword(const PGconn *conn);
extern int PQconnectionUsedPassword(const PGconn *conn);
extern int PQclientEncoding(const PGconn *conn);
@@ -434,6 +448,11 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for pipeline mode management */
+extern int PQenterPipelineMode(PGconn *conn);
+extern int PQexitPipelineMode(PGconn *conn);
+extern int PQsendPipeline(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 8d51e6ed9f..3e9f99f8fb 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,7 +217,11 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, more results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
@@ -229,7 +233,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* Sync at end of a pipeline */
} PGQueryClass;
/* Target server type (decoded value of target_session_attrs) */
@@ -305,6 +310,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by pipeline mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct PGcommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct PGcommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -397,6 +418,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PGpipelineStatus pipelineStatus; /* status of pipeline mode */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -409,6 +431,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue, for pipeline mode.
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -790,6 +822,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 5391f461a2..93e7829c67 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -10,6 +10,7 @@ SUBDIRS = \
delay_execution \
dummy_index_am \
dummy_seclabel \
+ libpq_pipeline \
plsample \
snapshot_too_old \
test_bloomfilter \
diff --git a/src/test/modules/libpq_pipeline/.gitignore b/src/test/modules/libpq_pipeline/.gitignore
new file mode 100644
index 0000000000..3a11e786b8
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/libpq_pipeline
diff --git a/src/test/modules/libpq_pipeline/Makefile b/src/test/modules/libpq_pipeline/Makefile
new file mode 100644
index 0000000000..b798f5fbbc
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/Makefile
@@ -0,0 +1,20 @@
+# src/test/modules/libpq_pipeline/Makefile
+
+PROGRAM = libpq_pipeline
+OBJS = libpq_pipeline.o
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS_INTERNAL += $(libpq_pgport)
+
+TAP_TESTS = 1
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/libpq_pipeline
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/libpq_pipeline/README b/src/test/modules/libpq_pipeline/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/libpq_pipeline/libpq_pipeline.c b/src/test/modules/libpq_pipeline/libpq_pipeline.c
new file mode 100644
index 0000000000..782e106e33
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/libpq_pipeline.c
@@ -0,0 +1,1144 @@
+/*
+ * src/test/modules/libpq_pipeline/libpq_pipeline.c
+ * Verify libpq pipeline execution functionality
+ */
+#include "postgres_fe.h"
+
+#include <sys/time.h>
+#ifdef HAVE_SYS_SELECT_H
+#include <sys/select.h>
+#endif
+
+#include "catalog/pg_type_d.h"
+#include "common/fe_memutils.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+
+static void exit_nicely(PGconn *conn);
+
+const char *const progname = "libpq_pipeline";
+
+
+#define DEBUG
+#ifdef DEBUG
+#define pg_debug(...) do { fprintf(stderr, __VA_ARGS__); } while (0)
+#else
+#define pg_debug(...)
+#endif
+
+static const char *const drop_table_sql =
+"DROP TABLE IF EXISTS pq_pipeline_demo";
+static const char *const create_table_sql =
+"CREATE UNLOGGED TABLE pq_pipeline_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql =
+"INSERT INTO pq_pipeline_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+/*
+ * Print an error to stderr and terminate the program.
+ */
+#define pg_fatal(...) pg_fatal_impl(__LINE__, __VA_ARGS__)
+static void
+pg_fatal_impl(int line, const char *fmt,...)
+{
+ va_list args;
+
+ fprintf(stderr, "\n"); /* XXX hack */
+ fprintf(stderr, "%s:%d: ", progname, line);
+
+ va_start(args, fmt);
+ vfprintf(stderr, fmt, args);
+ va_end(args);
+ printf("Failure, exiting\n");
+ exit(1);
+}
+
+static void
+test_disallowed(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("Unable to enter pipeline mode\n");
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not activated properly\n");
+
+ /* PQexec should fail in pipeline mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("PQexec should fail in pipeline mode but succeeded\n");
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ pg_fatal("PQsendQuery should fail in pipeline mode but succeeded\n");
+
+ /* Entering pipeline mode when already in pipeline mode is OK */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("re-entering pipeline mode should be a no-op but failed\n");
+
+ if (PQisBusy(conn) != 0)
+ pg_fatal("PQisBusy should return 0 when idle in pipeline mode, returned 1\n");
+
+ /* ok, back to normal command mode */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("couldn't exit idle empty pipeline mode\n");
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not terminated properly\n");
+
+ /* exiting pipeline mode when not in pipeline mode should be a no-op */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("pipeline mode exit when not in pipeline mode should succeed but failed\n");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("PQexec should succeed after exiting pipeline mode but failed with: %s\n",
+ PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_simple_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple pipeline... ");
+
+ /*
+ * Enter pipeline mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our output buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode with work in progress should fail, but succeeded\n");
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending pipeline failed: %s\n", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first query result.\n");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit pipeline mode yet
+ */
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result PGRES_PIPELINE_SYNC expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of PGRES_PIPELINE_SYNC, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after pipeline end: %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* ... until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_multi_pipelines(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi pipeline... ");
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending first pipeline failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s\n", PQerrorMessage(conn));
+
+ /* OK, start processing the results */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first result\n");
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ PQclear(res);
+
+#if 0
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+#endif
+
+ /* second pipeline */
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from second pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode ... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+/*
+ * When an operation in a pipeline fails the rest of the pipeline is flushed. We
+ * still have to get results for each pipeline item, but the item will just be
+ * a PGRES_PIPELINE_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the pipeline. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_aborted_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted pipeline... ");
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * pipeline ERRORs.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT no_such_function($1)",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching error select failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending first pipeline failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second-pipeline insert failed: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the pipeline results.
+ *
+ * We should get a command-ok for the first query, then a fatal error and
+ * a pipeline aborted message for the second insert, a pipeline-end, then
+ * a command-ok and a pipeline-ok for the second pipeline operation.
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result status %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* Second query caused error, so we expect an error next */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("Unexpected result code -- expected PGRES_FATAL_ERROR, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /*
+ * pipeline should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third pipeline since we've already sent a
+ * second. The aborted flag relates only to the pipeline being received.
+ */
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't\n");
+
+ /* third query in pipeline, the second insert */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_ABORTED)
+ pg_fatal("Unexpected result code -- expected PGRES_PIPELINE_ABORTED, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+#if 0
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+#endif
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't\n");
+
+ /* Ensure we're still in pipeline */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /*
+ * The end of a failed pipeline is a PGRES_PIPELINE_SYNC.
+ *
+ * (This is so clients know to start processing results normally again and
+ * can tell the difference between skipped commands and the sync.)
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code from first pipeline sync\n"
+ "Expected PGRES_PIPELINE_SYNC, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+#if 0
+ /* after the synchronization point we get a NULL result */
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+#endif
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_ABORTED)
+ pg_fatal("sync should've cleared the aborted flag but didn't\n");
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* the insert from the second pipeline */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result code %s from first item in second pipeline\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* Read the NULL result at the end of the command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* the second pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQerrorMessage(conn));
+
+ /* Test single-row mode with an error partways */
+ if (PQsendQueryParams(conn, "SELECT 1/g FROM generate_series(5, -1, -1) g",
+ 0, NULL, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to send query: %s", PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ PQsetSingleRowMode(conn);
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ if (PQresultStatus(res) == PGRES_FATAL_ERROR)
+ printf("got error: {{%s}} (expected: division by zero)\n", PQerrorMessage(conn));
+ else if (PQresultStatus(res) == PGRES_SINGLE_TUPLE)
+ printf("got row: %s\n", PQgetvalue(res, 0, 0));
+ else
+ pg_fatal("got unexpected result %s\n", PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ }
+ /* the third pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from third pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ /*-
+ * Since we fired the pipelines off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st pipeline
+ * - First insert applied
+ * - Second statement aborted xact
+ * - Third insert skipped
+ * - Sync rolled back first implicit xact
+ * - Implicit xact created by server around 2nd pipeline
+ * - insert applied from 2nd pipeline
+ * - Sync commits 2nd xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM pq_pipeline_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Expected tuples, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("expected 1 result, got %d\n", PQntuples(res));
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ pg_fatal("expected only insert with value 3, got %s", val);
+ }
+
+ PQclear(res);
+}
+
+/* State machine enum for test_pipelined_insert */
+typedef enum PipelineInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} PipelineInsertStep;
+
+static void
+test_pipelined_insert(PGconn *conn, int n_rows)
+{
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ PipelineInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a pipelined insert into a table created at the start of the pipeline
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ while (send_step != BI_PREPARE)
+ {
+ const char *sql;
+
+ switch (send_step)
+ {
+ case BI_BEGIN_TX:
+ sql = "BEGIN TRANSACTION";
+ send_step = BI_DROP_TABLE;
+ break;
+
+ case BI_DROP_TABLE:
+ sql = drop_table_sql;
+ send_step = BI_CREATE_TABLE;
+ break;
+
+ case BI_CREATE_TABLE:
+ sql = create_table_sql;
+ send_step = BI_PREPARE;
+ break;
+
+ default:
+ pg_fatal("invalid state");
+ }
+
+ pg_debug("sending: %s\n", sql);
+ if (PQsendQueryParams(conn, sql,
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching %s failed: %s\n", sql, PQerrorMessage(conn));
+ }
+
+ Assert(send_step == BI_PREPARE);
+ pg_debug("sending: %s\n", insert_sql);
+ if (PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids) != 1)
+ pg_fatal("dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our output buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the pipeline before processing
+ * results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ pg_fatal("failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's output buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ PGresult *res;
+ const char *cmdtag;
+ const char *description = "";
+ int status;
+
+ /*
+ * Read next result. If no more results from this query,
+ * advance to the next query
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ continue;
+
+ status = PGRES_COMMAND_OK;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ recv_step++;
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ recv_step++;
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ recv_step++;
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ recv_step++;
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive == 0)
+ recv_step++;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ recv_step++;
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_PIPELINE_SYNC;
+ recv_step++;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ description = "";
+ abort();
+ }
+
+ if (PQresultStatus(res) != status)
+ pg_fatal("%s reported status %s, expected %s\n"
+ "Error message: \"%s\"\n",
+ description, PQresStatus(PQresultStatus(res)),
+ PQresStatus(status), PQerrorMessage(conn));
+
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ pg_fatal("%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+
+ pg_debug("Got %s OK\n", cmdtag[0] != '\0' ? cmdtag : description);
+
+ PQclear(res);
+ }
+ }
+
+ /* Write more rows and/or the end pipeline message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQsendPipeline(conn) == 1)
+ {
+ fprintf(stdout, "Sent pipeline\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending pipeline failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+ }
+
+ /* We've got the sync message and the pipeline should be done */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ pg_fatal("failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i;
+ bool pipeline_ended = false;
+
+ /* 1 pipeline, 3 queries in it */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n",
+ PQerrorMessage(conn));
+
+ for (i = 0; i < 3; i++)
+ {
+ char *param[1];
+
+ param[0] = psprintf("%d", 44 + i);
+
+ if (PQsendQueryParams(conn,
+ "SELECT generate_series(42, $1)",
+ 1,
+ NULL,
+ (const char **) param,
+ NULL,
+ NULL,
+ 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ pfree(param[0]);
+ }
+ PQsendPipeline(conn);
+
+ for (i = 0; !pipeline_ended; i++)
+ {
+ bool first = true;
+ bool saw_ending_tuplesok;
+ bool isSingleTuple = false;
+
+ /* Set single row mode for only first 2 SELECT queries */
+ if (i < 2)
+ {
+ if (PQsetSingleRowMode(conn) != 1)
+ pg_fatal("PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+
+ /* Consume rows for this query */
+ saw_ending_tuplesok = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+
+ if (est == PGRES_PIPELINE_SYNC)
+ {
+ fprintf(stderr, "end of pipeline reached\n");
+ pipeline_ended = true;
+ PQclear(res);
+ if (i != 3)
+ pg_fatal("Expected three results, got %d\n", i);
+ break;
+ }
+
+ /* Expect SINGLE_TUPLE for queries 0 and 1, TUPLES_OK for 2 */
+ if (first)
+ {
+ if (i <= 1 && est != PGRES_SINGLE_TUPLE)
+ pg_fatal("Expected PGRES_SINGLE_TUPLE for query %d, got %s\n",
+ i, PQresStatus(est));
+ if (i >= 2 && est != PGRES_TUPLES_OK)
+ pg_fatal("Expected PGRES_TUPLES_OK for query %d, got %s\n",
+ i, PQresStatus(est));
+ first = false;
+ }
+
+ fprintf(stderr, "Result status %s for query %d", PQresStatus(est), i);
+ switch (est)
+ {
+ case PGRES_TUPLES_OK:
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ saw_ending_tuplesok = true;
+ if (isSingleTuple)
+ {
+ if (PQntuples(res) == 0)
+ fprintf(stderr, "all tuples received in query %d\n", i);
+ else
+ pg_fatal("Expected to follow PGRES_SINGLE_TUPLE, "
+ "but received PGRES_TUPLES_OK directly instead\n");
+ }
+ break;
+
+ case PGRES_SINGLE_TUPLE:
+ isSingleTuple = true;
+ fprintf(stderr, ", %d tuple: %s\n", PQntuples(res), PQgetvalue(res, 0, 0));
+ break;
+
+ default:
+ pg_fatal("unexpected\n");
+ }
+ PQclear(res);
+ }
+ if (!pipeline_ended && !saw_ending_tuplesok)
+ pg_fatal("didn't get expected terminating TUPLES_OK\n");
+ }
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s\n", PQerrorMessage(conn));
+}
+
+/*
+ * Simple test to verify that a pipeline is discarded as a whole when there's
+ * an error, ignoring transaction commands.
+ */
+static void
+test_transaction(PGconn *conn)
+{
+ PGresult *res;
+ int num_sends = 0;
+
+ res = PQexec(conn, "DROP TABLE IF EXISTS pq_pipeline_tst;"
+ "CREATE TABLE pq_pipeline_tst (id int)");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to create test table: %s",
+ PQerrorMessage(conn));
+ PQclear(res);
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n",
+ PQerrorMessage(conn));
+ if (PQsendPrepare(conn, "rollback", "ROLLBACK", 0, NULL) != 1)
+ pg_fatal("could not send prepare on pipeline: %s",
+ PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn,
+ "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ if (PQsendQueryParams(conn,
+ "SELECT 0/0",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+
+ /*
+ * send a ROLLBACK using a prepared stmt. Doesn't work because we need to
+ * get out of the pipeline-aborted state first.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /* This insert fails because we're in pipeline-aborted state */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (1)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * This insert fails even though the pipeline got a SYNC, because we're in
+ * an aborted transaction
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (2)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * Send ROLLBACK using prepared stmt. This one works because we just did
+ * PQsendPipeline above.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /*
+ * Now that we're out of a transaction and in pipeline-good mode, this
+ * insert works
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (3)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+ PQsendPipeline(conn);
+ num_sends++;
+
+ for (int i = 0;; i++)
+ {
+ ExecStatusType restype;
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "%d: got NULL result\n", i);
+ continue;
+ }
+ restype = PQresultStatus(res);
+ fprintf(stderr, "%d: got status %s", i, PQresStatus(restype));
+ if (restype == PGRES_FATAL_ERROR)
+ fprintf(stderr, "; error: %s", PQerrorMessage(conn));
+ else if (restype == PGRES_PIPELINE_ABORTED)
+ {
+ fprintf(stderr, ": command didn't run because pipeline aborted\n");
+ }
+ else
+ fprintf(stderr, "\n");
+ PQclear(res);
+
+ if (restype == PGRES_PIPELINE_SYNC)
+ num_sends--;
+ if (num_sends <= 0)
+ break;
+ }
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("returned something extra after all the syncs: %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s\n", PQerrorMessage(conn));
+
+ /* We expect to find one tuple containing the value "3" */
+ res = PQexec(conn, "SELECT * FROM pq_pipeline_tst");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("failed to obtain result: %s", PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("did not get 1 tuple\n");
+ if (strcmp(PQgetvalue(res, 0, 0), "3") != 0)
+ pg_fatal("did not get expected tuple\n");
+ PQclear(res);
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+usage(const char *progname)
+{
+ fprintf(stderr, "%s tests libpq's pipeline mode.\n\n", progname);
+ fprintf(stderr, "Usage:\n");
+ fprintf(stderr, " %s testname [conninfo [number_of_rows]]\n", progname);
+ fprintf(stderr, "Tests:\n");
+ fprintf(stderr, " disallowed_in_pipeline\n");
+ fprintf(stderr, " simple_pipeline\n");
+ fprintf(stderr, " multi_pipeline\n");
+ fprintf(stderr, " pipeline_abort\n");
+ fprintf(stderr, " singlerow\n");
+ fprintf(stderr, " pipeline_insert\n");
+ fprintf(stderr, " transaction\n");
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo = "";
+ PGconn *conn;
+ int numrows = 10000;
+ PGresult *res;
+
+ /*
+ * The testname parameter is mandatory; it can be followed by a conninfo
+ * string and number of rows.
+ */
+ if (argc < 2 || argc > 4)
+ {
+ usage(argv[0]);
+ exit(1);
+ }
+
+ if (argc >= 3)
+ conninfo = pg_strdup(argv[2]);
+
+ if (argc >= 4)
+ {
+ errno = 0;
+ numrows = strtol(argv[3], NULL, 10);
+ if (errno != 0 || numrows <= 0)
+ {
+ fprintf(stderr, "couldn't parse \"%s\" as a positive integer\n", argv[3]);
+ exit(1);
+ }
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+ res = PQexec(conn, "SET lc_messages TO \"C\"");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to set lc_messages: %s", PQerrorMessage(conn));
+
+ if (strcmp(argv[1], "disallowed_in_pipeline") == 0)
+ test_disallowed(conn);
+ else if (strcmp(argv[1], "simple_pipeline") == 0)
+ test_simple_pipeline(conn);
+ else if (strcmp(argv[1], "multi_pipeline") == 0)
+ test_multi_pipelines(conn);
+ else if (strcmp(argv[1], "pipeline_abort") == 0)
+ test_aborted_pipeline(conn);
+ else if (strcmp(argv[1], "pipeline_insert") == 0)
+ test_pipelined_insert(conn, numrows);
+ else if (strcmp(argv[1], "singlerow") == 0)
+ test_singlerowmode(conn);
+ else if (strcmp(argv[1], "transaction") == 0)
+ test_transaction(conn);
+ else
+ {
+ fprintf(stderr, "\"%s\" is not a recognized test name\n", argv[1]);
+ usage(argv[0]);
+ exit(1);
+ }
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+ return 0;
+}
diff --git a/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
new file mode 100644
index 0000000000..a12f2dd47b
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
@@ -0,0 +1,31 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 7;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_pipeline
+ simple_pipeline
+ multi_pipeline
+ pipeline_abort
+ pipeline_insert
+ singlerow
+ transaction);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'libpq_pipeline', $testname, $node->connstr('postgres'), $numrows ],
+ "libp_pipeline $testname");
+}
+
+$node->stop('fast');
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 49614106dc..44b1a43f30 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -33,10 +33,11 @@ my @unlink_on_exit;
# Set of variables for modules in contrib/ and src/test/modules/
my $contrib_defines = { 'refint' => 'REFINT_VERBOSE' };
-my @contrib_uselibpq = ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo');
-my @contrib_uselibpgport = ('oid2name', 'vacuumlo');
-my @contrib_uselibpgcommon = ('oid2name', 'vacuumlo');
-my $contrib_extralibs = undef;
+my @contrib_uselibpq =
+ ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo', 'libpq_pipeline');
+my @contrib_uselibpgport = ('libpq_pipeline', 'oid2name', 'vacuumlo');
+my @contrib_uselibpgcommon = ('libpq_pipeline', 'oid2name', 'vacuumlo');
+my $contrib_extralibs = { 'libpq_pipeline' => ['ws2_32.lib'] };
my $contrib_extraincludes = { 'dblink' => ['src/backend'] };
my $contrib_extrasource = {
'cube' => [ 'contrib/cube/cubescan.l', 'contrib/cube/cubeparse.y' ],
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 8bd95aefa1..1bd23eec6d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1561,10 +1561,12 @@ PG_Locale_Strategy
PG_Lock_Status
PG_init_t
PGcancel
+PGcommandQueueEntry
PGconn
PGdataValue
PGlobjfuncs
PGnotify
+PGpipelineStatus
PGresAttDesc
PGresAttValue
PGresParamDesc
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
Hi,
+ <literal>\gset</literal> and <literal>\aset</literal> cannot be used
+ pipeline mode, since query results are not immediately'used pipeline mode' -> 'used in pipeline mode'
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/libpq_pipeline.c
@@ -0,0 +1,1144 @@
+/*
+ * src/test/modules/libpq_pipeline/libpq_pipeline.cLooks like license information is missing from the header.
Cheers
On Thu, Mar 4, 2021 at 1:40 PM Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:
Show quoted text
On 2021-Mar-04, Alvaro Herrera wrote:
I don't know where do __WSAFDIsSet and __imp_select come from or what to
do about them. Let's see if adding pgport and pgcommon fixes things.Indeed all those other problems were fixed and these remain. New
failure is:"C:\projects\postgresql\pgsql.sln" (default target) (1) ->
6007"C:\projects\postgresql\libpq_pipeline.vcxproj" (default target) (55)
->
6008(Link target) ->
6009 libpq_pipeline.obj : error LNK2019: unresolved external symbol
__WSAFDIsSet referenced in function test_pipelined_insert
[C:\projects\postgresql\libpq_pipeline.vcxproj]
6010 libpq_pipeline.obj : error LNK2019: unresolved external symbol
__imp_select referenced in function test_pipelined_insert
[C:\projects\postgresql\libpq_pipeline.vcxproj]
6011 .\Release\libpq_pipeline\libpq_pipeline.exe : fatal error LNK1120: 2
unresolved externals [C:\projects\postgresql\libpq_pipeline.vcxproj]I did notice that isolationtester.c is using select(), and one
difference is that it includes <sys/select.h> which libpq_pipeline.c
does not -- and it also pulls in ws2_32.lib. Let's see if those two
changes fix things.--
Álvaro Herrera Valdivia, Chile
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
v33 was indeed marked a pass by cfbot. However, it only *builds* the
test program, it does not *run* it. I guess we'll have to wait for the
buildfarm to tell us more.
In the meantime, I implemented PQsendQuery() as callable in pipeline
mode; it does that by using the extended-query protocol directly rather
than sending 'Q' as in non-pipeline mode. I also adjusted the docs a
little bit more. That's what you see here as v34.
I'll take the weekend to think about the issue with conn->last_query and
conn->queryclass that I mentioned yesterday; other than that detail my
feeling is that this is committable, so I'll be looking at getting this
pushed early next weeks, barring opinions from others.
--
�lvaro Herrera Valdivia, Chile
"El sabio habla porque tiene algo que decir;
el tonto, porque tiene que decir algo" (Platon).
Attachments:
v34-libpq-pipeline.patchtext/x-diff; charset=us-asciiDownload
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 0553279314..e3cd5c377b 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -3173,6 +3173,33 @@ ExecStatusType PQresultStatus(const PGresult *res);
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-sync">
+ <term><literal>PGRES_PIPELINE_SYNC</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a
+ synchronization point in pipeline mode, requested by
+ <xref linkend="libpq-PQsendPipeline"/>.
+ This status occurs only when pipeline mode has been selected.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-aborted">
+ <term><literal>PGRES_PIPELINE_ABORTED</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a pipeline that has
+ received an error from the server. <function>PQgetResult</function>
+ must be called repeatedly, and each time it will return this status code
+ until the end of the current pipeline, at which point it will return
+ <literal>PGRES_PIPELINE_SYNC</literal> and normal processing can
+ resume.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
If the result status is <literal>PGRES_TUPLES_OK</literal> or
@@ -4919,6 +4946,473 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-pipeline-mode">
+ <title>Pipeline Mode</title>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>libpq</primary>
+ <secondary>pipeline mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>pipelining</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>batch mode</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> pipeline mode allows applications to
+ send a query without having to read the result of the previously
+ sent query. Taking advantage of the pipeline mode, a client will wait
+ less for the server, since multiple queries/results can be
+ sent/received in a single network transaction.
+ </para>
+
+ <para>
+ While pipeline mode provides a significant performance boost, writing
+ clients using the pipeline mode is more complex because it involves
+ managing a queue of pending queries and finding which result
+ corresponds to which query in the queue.
+ </para>
+
+ <para>
+ Pipeline mode also generally consumes more memory on both the client and server,
+ though careful and aggressive management of the send/receive queue can mitigate
+ this. This applies whether or not the connection is in blocking or non-blocking
+ mode.
+ </para>
+
+ <sect2 id="libpq-pipeline-using">
+ <title>Using Pipeline Mode</title>
+
+ <para>
+ To issue pipelines, the application must switch the connection
+ into pipeline mode,
+ which is done with <xref linkend="libpq-PQenterPipelineMode"/>.
+ <xref linkend="libpq-PQpipelineStatus"/> can be used
+ to test whether pipeline mode is active.
+ In pipeline mode, only <link linkend="libpq-async">asynchronous operations</link>
+ are permitted, and <literal>COPY</literal> is disallowed.
+ Using synchronous command execution functions
+ such as <function>PQfn</function>,
+ <function>PQexec</function>,
+ <function>PQexecParams</function>,
+ <function>PQprepare</function>,
+ <function>PQexecPrepared</function>,
+ <function>PQdescribePrepared</function>,
+ <function>PQdescribePortal</function>,
+ is an error condition.
+ Once all dispatched commands have had their results processed, and
+ the end pipeline result has been consumed, the application may return
+ to non-pipelined mode with <xref linkend="libpq-PQexitPipelineMode"/>.
+ </para>
+
+ <note>
+ <para>
+ It is best to use pipeline mode with <application>libpq</application> in
+ <link linkend="libpq-PQsetnonblocking">non-blocking mode</link>. If used
+ in blocking mode it is possible for a client/server deadlock to occur.
+ <footnote>
+ <para>
+ The client will block trying to send queries to the server, but the
+ server will block trying to send results to the client from queries
+ it has already processed. This only occurs when the client sends
+ enough queries to fill both its output buffer and the server's receive
+ buffer before it switches to processing input from the server,
+ but it's hard to predict exactly when that will happen.
+ </para>
+ </footnote>
+ </para>
+ </note>
+
+ <sect3 id="libpq-pipeline-sending">
+ <title>Issuing Queries</title>
+
+ <para>
+ After entering pipeline mode, the application dispatches requests using
+ <xref linkend="libpq-PQsendQuery"/>,
+ <xref linkend="libpq-PQsendQueryParams"/>,
+ or its prepared-query sibling
+ <xref linkend="libpq-PQsendQueryPrepared"/>.
+ These requests are queued on the client-side until flushed to the server;
+ this occurs when <xref linkend="libpq-PQsendPipeline"/> is used to
+ establish a synchronization point in the pipeline,
+ or when <xref linkend="libpq-PQflush"/> is called.
+ The functions <xref linkend="libpq-PQsendPrepare"/>,
+ <xref linkend="libpq-PQsendDescribePrepared"/>, and
+ <xref linkend="libpq-PQsendDescribePortal"/> also work in pipeline mode.
+ Result processing is described below.
+ </para>
+
+ <para>
+ The server executes statements, and returns results, in the order the
+ client sends them. The server will begin executing the commands in the
+ pipeline immediately, not waiting for the end of the pipeline.
+ If any statement encounters an error, the server aborts the current
+ transaction and skips processing commands in the pipeline until the
+ next synchronization point established by <function>PQsendPipeline</function>.
+ (This remains true even if the commands in the pipeline would rollback
+ the transaction.)
+ Query processing resumes after the synchronization point.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one; for example, one query may define a table that the next
+ query in the same pipeline uses. Similarly, an application may
+ create a named prepared statement and execute it with later
+ statements in the same pipeline.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-results">
+ <title>Processing Results</title>
+
+ <para>
+ To process the result of one query in a pipeline, the application calls
+ <function>PQgetResult</function> repeatedly and handles each result
+ until <function>PQgetResult</function> returns null.
+ The result from the next query in the pipeline may then be retrieved using
+ <function>PQgetResult</function> again and the cycle repeated.
+ The application handles individual statement results as normal.
+ When the results of all the queries in the pipeline have been
+ returned, <function>PQgetResult</function> returns a result
+ containing the status value <literal>PGRES_PIPELINE_SYNC</literal>
+ </para>
+
+ <para>
+ The client may choose to defer result processing until the complete
+ pipeline has been sent, or interleave that with sending further
+ queries in the pipeline; see <xref linkend="libpq-pipeline-interleave"/>.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ This mode selection is effective only for the query currently
+ being processed. For more information on the use of
+ <function>PQsetSingleRowMode</function>,
+ refer to <xref linkend="libpq-single-row-mode"/>.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal
+ asynchronous processing except that it may contain the new
+ <type>PGresult</type> types <literal>PGRES_PIPELINE_SYNC</literal>
+ and <literal>PGRES_PIPELINE_ABORTED</literal>.
+ <literal>PGRES_PIPELINE_SYNC</literal> is reported exactly once for each
+ <function>PQsendPipeline</function> after retrieving results for all
+ queries in the pipeline.
+ <literal>PGRES_PIPELINE_ABORTED</literal> is emitted in place of a normal
+ stream result for the first error and all subsequent results
+ until <literal>PGRES_PIPELINE_SYNC</literal>;
+ see <xref linkend="libpq-pipeline-errors"/>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing pipeline results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed (except that
+ <function>PQgetResult</function> returns null to indicate that we start
+ returning the results of next query). The application must keep track
+ of the order in which it sent queries, to associate them with their
+ corresponding results.
+ Applications will typically use a state machine or a FIFO queue for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-pipeline-errors">
+ <title>Error Handling</title>
+
+ <para>
+ From the client perspective, after <function>PQresultStatus</function>
+ returns <literal>PGRES_FATAL_ERROR</literal>,
+ the pipeline is flagged as aborted.
+ <function>PQresultStatus</function> will report a
+ <literal>PGRES_PIPELINE_ABORTED</literal> result for each remaining queued
+ operation in an aborted pipeline. The result for
+ <function>PQsendPipeline</function> is reported as
+ <literal>PGRES_PIPELINE_SYNC</literal> to signal the end of the aborted pipeline
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the pipeline used an implicit transaction, then operations that have
+ already executed are rolled back and operations that were queued to follow
+ the failed operation are skipped entirely. The same behaviour holds if the
+ pipeline starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the pipeline. If a pipeline contains
+ <emphasis>multiple explicit transactions</emphasis>, all transactions that
+ committed prior to the error remain committed, the currently in-progress
+ transaction is aborted, and all subsequent operations are skipped completely,
+ including subsequent transactions.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal> — only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously, the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-interleave">
+ <title>Interleaving Result Processing and Query Dispatch</title>
+
+ <para>
+ To avoid deadlocks on large pipelines the client should be structured
+ around a non-blocking event loop using operating system facilities
+ such as <function>select</function>, <function>poll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work
+ remaining to be dispatched and a queue of work that has been dispatched
+ but not yet had its results processed. When the socket is writable
+ it should dispatch more work. When the socket is readable it should
+ read results and process them, matching them up to the next entry in
+ its expected results queue. Based on available memory, results from the
+ socket should be read frequently: there's no need to wait until the
+ pipeline end to read the results. Pipelines should be scoped to logical
+ units of work, usually (but not necessarily) one transaction per pipeline.
+ There's no need to exit pipeline mode and re-enter it between pipelines,
+ or to wait for one pipeline to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state
+ machine to track sent and received work is in
+ <filename>src/test/modules/libpq_pipeline/libpq_pipeline.c</filename>
+ in the PostgreSQL source distribution.
+ </para>
+ </sect3>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-functions">
+ <title>Functions Associated with Pipeline Mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQpipelineStatus">
+ <term><function>PQpipelineStatus</function><indexterm><primary>PQpipelineStatus</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Returns the current pipeline mode status of the
+ <application>libpq</application> connection.
+<synopsis>
+PGpipelineStatus PQpipelineStatus(const PGconn *conn);
+</synopsis>
+ </para>
+
+ <para>
+ <function>PQpipelineStatus</function> can return one of the following values:
+
+ <variablelist>
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ON</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in
+ pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_OFF</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is
+ <emphasis>not</emphasis> in pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in pipeline
+ mode and an error occurred while processing the current pipeline.
+ The aborted flag is cleared when <function>PQresultStatus</function>
+ returns PGRES_PIPELINE_SYNC at the end of the pipeline.
+ Clients don't usually need this function to
+ verify aborted status, as they can tell that the pipeline is aborted
+ from the <literal>PGRES_PIPELINE_ABORTED</literal> result code.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterPipelineMode">
+ <term><function>PQenterPipelineMode</function><indexterm><primary>PQenterPipelineMode</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter pipeline mode if it is currently idle or
+ already in pipeline mode.
+
+<synopsis>
+int PQenterPipelineMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success.
+ Returns 0 and has no effect if the connection is not currently
+ idle, i.e., it has a result ready, or it is waiting for more
+ input from the server, etc.
+ This function does not actually send anything to the server,
+ it just changes the <application>libpq</application> connection
+ state.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitPipelineMode">
+ <term><function>PQexitPipelineMode</function><indexterm><primary>PQexitPipelineMode</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit pipeline mode if it is currently in pipeline mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitPipelineMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 1 and takes no action if not in
+ pipeline mode. If the current statement isn't finished processing
+ or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQsendPipeline">
+ <term><function>PQsendPipeline</function><indexterm><primary>PQsendPipeline</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Marks a synchronization point in a pipeline by sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ and flushing the send buffer. This serves as
+ the delimiter of an implicit transaction and an error recovery
+ point; see <xref linkend="libpq-pipeline-errors"/>.
+
+<synopsis>
+int PQsendPipeline(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 if the connection is not in
+ pipeline mode or sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ failed.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-tips">
+ <title>When to Use Pipeline Mode</title>
+
+ <para>
+ Much like asynchronous query mode, there is no meaningful performance
+ overhead when using pipeline mode. It increases client application complexity,
+ and extra caution is required to prevent client/server deadlocks, but
+ pipeline mode can offer considerable performance improvements, in exchange for
+ increased memory usage from leaving state around longer.
+ </para>
+
+ <para>
+ Pipeline mode is most useful when the server is distant, i.e., network latency
+ (<quote>ping time</quote>) is high, and also when many small operations
+ are being performed in rapid succession. There is usually less benefit
+ in using pipelined commands when each query takes many multiples of the client/server
+ round-trip time to execute. A 100-statement operation run on a server
+ 300ms round-trip-time away would take 30 seconds in network latency alone
+ without pipelining; with pipelining it may spend as little as 0.3s waiting for
+ results from the server.
+ </para>
+
+ <para>
+ Use pipelined commands when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed
+ into operations on sets, or into a <literal>COPY</literal> operation.
+ </para>
+
+ <para>
+ Pipeline mode is not useful when information from one operation is required by
+ the client to produce the next operation. In such cases, the client
+ would have to introduce a synchronization point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+BEGIN;
+SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+-- result: x=2
+-- client adds 1 to x:
+UPDATE mytable SET x = 3 WHERE id = 42;
+COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <para>
+ Pipelining is less useful, and more complex, when a single pipeline contains
+ multiple transactions (see <xref linkend="libpq-pipeline-errors"/>).
+ </para>
+
+ <note>
+ <para>
+ The pipeline API was introduced in <productname>PostgreSQL</productname> 14.
+ Pipeline mode is a client-side feature which doesn't require server
+ support, and works on any server that supports the v3 extended query
+ protocol.
+ </para>
+ </note>
+ </sect2>
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4959,6 +5453,13 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <para>
+ When using pipeline mode, single-row mode needs to be activated for each
+ query in the pipeline before retrieving results for that query
+ with <function>PQgetResult</function>.
+ See <xref linkend="libpq-pipeline-mode"/> for more information.
+ </para>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 6d46da42e2..012e44c736 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while a libpq connection is in pipeline mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 2ec0580a79..75448162ca 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -1109,6 +1109,12 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
row, the last value is kept.
</para>
+ <para>
+ <literal>\gset</literal> and <literal>\aset</literal> cannot be used
+ pipeline mode, since query results are not immediately
+ fetched in this mode.
+ </para>
+
<para>
The following example puts the final account balance from the first query
into variable <replaceable>abalance</replaceable>, and fills variables
@@ -1269,6 +1275,21 @@ SELECT 4 AS four \; SELECT 5 AS five \aset
</programlisting></para>
</listitem>
</varlistentry>
+
+ <varlistentry id='pgbench-metacommand-pipeline'>
+ <term><literal>\startpipeline</literal></term>
+ <term><literal>\endpipeline</literal></term>
+
+ <listitem>
+ <para>
+ These commands delimit the start and end of a pipeline of SQL statements.
+ In a pipeline, statements are sent to server without waiting for the results
+ of previous statements (see <xref linkend="libpq-pipeline-mode"/>).
+ Pipeline mode requires the extended query protocol.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect2>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 5272eed9ab..f74378110a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1019,6 +1019,12 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->err = _("empty query");
break;
+ case PGRES_PIPELINE_SYNC:
+ case PGRES_PIPELINE_ABORTED:
+ walres->status = WALRCV_ERROR;
+ walres->err = _("unexpected pipeline mode");
+ break;
+
case PGRES_NONFATAL_ERROR:
case PGRES_FATAL_ERROR:
case PGRES_BAD_RESPONSE:
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 31a4df45f5..fbbe270654 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -351,10 +351,11 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
- * and we enter the CSTATE_SLEEP state to wait for it to expire. Other
- * meta-commands are executed immediately. If the command about to start
- * is actually beyond the end of the script, advance to CSTATE_END_TX.
+ * CSTATE_WAIT_RESULT state unless in pipeline mode. On a \sleep
+ * meta-command, the timer is set, and we enter the CSTATE_SLEEP state to
+ * wait for it to expire. Other meta-commands are executed immediately. If
+ * the command about to start is actually beyond the end of the script,
+ * advance to CSTATE_END_TX.
*
* CSTATE_WAIT_RESULT waits until we get a result set back from the server
* for the current command.
@@ -484,7 +485,9 @@ typedef enum MetaCommand
META_IF, /* \if */
META_ELIF, /* \elif */
META_ELSE, /* \else */
- META_ENDIF /* \endif */
+ META_ENDIF, /* \endif */
+ META_STARTPIPELINE, /* \startpipeline */
+ META_ENDPIPELINE /* \endpipeline */
} MetaCommand;
typedef enum QueryMode
@@ -2504,6 +2507,10 @@ getMetaCommand(const char *cmd)
mc = META_GSET;
else if (pg_strcasecmp(cmd, "aset") == 0)
mc = META_ASET;
+ else if (pg_strcasecmp(cmd, "startpipeline") == 0)
+ mc = META_STARTPIPELINE;
+ else if (pg_strcasecmp(cmd, "endpipeline") == 0)
+ mc = META_ENDPIPELINE;
else
mc = META_NONE;
return mc;
@@ -2693,11 +2700,25 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In pipeline mode, we use asynchronous functions. If a
+ * server-side error occurs, it will be processed later
+ * among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ pg_log_error("%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2741,8 +2762,10 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
* varprefix should be set only with \gset or \aset, and SQL commands do
* not need it.
*/
+#if 0
Assert((meta == META_NONE && varprefix == NULL) ||
((meta == META_GSET || meta == META_ASET) && varprefix != NULL));
+#endif
res = PQgetResult(st->con);
@@ -2810,6 +2833,12 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* otherwise the result is simply thrown away by PQclear below */
break;
+ case PGRES_PIPELINE_SYNC:
+ pg_log_debug("client %d pipeline ending", st->id);
+ if (PQexitPipelineMode(st->con) != 1)
+ pg_log_error("client %d failed to exit pipeline mode", st->id);
+ break;
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3066,13 +3095,36 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
/* Execute the command */
if (command->type == SQL_COMMAND)
{
+ /* disallow \aset and \gset in pipeline mode */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ if (command->meta == META_GSET)
+ {
+ commandFailed(st, "gset", "\\gset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ else if (command->meta == META_ASET)
+ {
+ commandFailed(st, "aset", "\\aset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ }
+
if (!sendCommand(st, command))
{
commandFailed(st, "SQL", "SQL command send failed");
st->state = CSTATE_ABORTED;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in pipeline mode */
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -3212,7 +3264,15 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (readCommandResponse(st,
sql_script[st->use_file].commands[st->command]->meta,
sql_script[st->use_file].commands[st->command]->varprefix))
- st->state = CSTATE_END_COMMAND;
+ {
+ /*
+ * outside of pipeline mode: stop reading results.
+ * pipeline mode: continue reading results until an
+ * end-of-pipeline response.
+ */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ st->state = CSTATE_END_COMMAND;
+ }
else
st->state = CSTATE_ABORTED;
break;
@@ -3456,6 +3516,49 @@ executeMetaCommand(CState *st, instr_time *now)
return CSTATE_ABORTED;
}
}
+ else if (command->meta == META_STARTPIPELINE)
+ {
+ /*
+ * In pipeline mode, we use a workflow based on libpq pipeline
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "startpipeline", "cannot use pipeline mode with the simple query protocol");
+ return CSTATE_ABORTED;
+ }
+
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ commandFailed(st, "startpipeline", "already in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (PQenterPipelineMode(st->con) == 0)
+ {
+ commandFailed(st, "startpipeline", "failed to enter pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ }
+ else if (command->meta == META_ENDPIPELINE)
+ {
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ {
+ commandFailed(st, "endpipeline", "not in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (!PQsendPipeline(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to send the pipeline");
+ return CSTATE_ABORTED;
+ }
+ if (!PQexitPipelineMode(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to exit pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ /* collect pending results before getting out of pipeline mode */
+ return CSTATE_WAIT_RESULT;
+ }
/*
* executing the expression or shell command might have taken a
@@ -4683,7 +4786,9 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
- else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
+ else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF ||
+ my_command->meta == META_STARTPIPELINE ||
+ my_command->meta == META_ENDPIPELINE)
{
if (my_command->argc != 1)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
@@ -6808,4 +6913,5 @@ pthread_join(pthread_t th, void **thread_return)
return 0;
}
+
#endif /* WIN32 */
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..60d09e6d63 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,7 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterPipelineMode 180
+PQexitPipelineMode 181
+PQsendPipeline 182
+PQpipelineStatus 183
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index f83af03d0a..4c97e8a078 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -540,6 +540,23 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * pqFreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+pqFreeCommandQueue(PGcommandQueueEntry *queue)
+{
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *cur = queue;
+
+ queue = cur->next;
+ if (cur->query)
+ free(cur->query);
+ free(cur);
+ }
+}
/*
* pqDropServerData
@@ -571,6 +588,12 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ pqFreeCommandQueue(conn->cmd_queue_head);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ pqFreeCommandQueue(conn->cmd_queue_recycle);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
@@ -2486,6 +2509,7 @@ keep_going: /* We will come back to here until there is
/* Drop any PGresult we might have, too */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn);
/* Reset conn->status to put the state machine in the right state */
@@ -3928,6 +3952,7 @@ makeEmptyPGconn(void)
conn->status = CONNECTION_BAD;
conn->asyncStatus = PGASYNC_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
conn->xactStatus = PQTRANS_IDLE;
conn->options_valid = false;
conn->nonblocking = false;
@@ -4187,6 +4212,7 @@ closePGconn(PGconn *conn)
conn->status = CONNECTION_BAD; /* Well, not really _bad_ - just absent */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn); /* deallocate result */
resetPQExpBuffer(&conn->errorMessage);
release_conn_addrinfo(conn);
@@ -6735,6 +6761,15 @@ PQbackendPID(const PGconn *conn)
return conn->be_pid;
}
+PGpipelineStatus
+PQpipelineStatus(const PGconn *conn)
+{
+ if (!conn)
+ return PQ_PIPELINE_OFF;
+
+ return conn->pipelineStatus;
+}
+
int
PQconnectionNeedsPassword(const PGconn *conn)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9a038043b2..0cdba4260e 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_PIPELINE_SYNC",
+ "PGRES_PIPELINE_ABORTED"
};
/*
@@ -71,6 +73,11 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *pqMakePipelineCmd(PGconn *conn);
+static void pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqPipelineProcessQueue(PGconn *conn);
+static int pqPipelineFlush(PGconn *conn);
/* ----------------
@@ -1171,7 +1178,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1209,9 +1216,23 @@ PQsendQueryContinue(PGconn *conn, const char *query)
static int
PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+
if (!PQsendQueryStart(conn, newQuery))
return 0;
+ /*
+ * Decide where the query is going to be stored. In pipeline mode, we
+ * allocate a new pipeline element; in non-pipeline mode, it's simply the
+ * connection's last query.
+ */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+ }
+
/* check the argument */
if (!query)
{
@@ -1220,37 +1241,79 @@ PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
return 0;
}
- /* construct the outgoing Query message */
- if (pqPutMsgStart('Q', conn) < 0 ||
- pqPuts(query, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
+ /* Send the query message(s) */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
{
- /* error message should be set up already */
- return 0;
+ /* construct the outgoing Query message */
+ if (pqPutMsgStart('Q', conn) < 0 ||
+ pqPuts(query, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ {
+ /* error message should be set up already */
+ return 0;
+ }
+
+ /* remember we are using simple query protocol */
+ conn->queryclass = PGQUERY_SIMPLE;
+ /* and remember the query text too, if possible */
+ /* if insufficient memory, last_query just winds up NULL */
+ if (conn->last_query)
+ free(conn->last_query);
+ conn->last_query = strdup(query);
}
+ else
+ {
+ /*
+ * In pipeline mode, we cannot use the simple protocol, so we send
+ * Parse, Bind, Describe Portal, Execute.
+ */
+ if (pqPutMsgStart('P', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPuts(query, conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ if (pqPutMsgStart('B', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ if (pqPutMsgStart('D', conn) < 0 ||
+ pqPutc('P', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ if (pqPutMsgStart('E', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPutInt(0, 4, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
- /* remember we are using simple query protocol */
- conn->queryclass = PGQUERY_SIMPLE;
-
- /* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ pipeCmd->queryclass = PGQUERY_EXTENDED;
+ pipeCmd->query = strdup(query);
+ }
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
- {
- /* error message should be set up already */
- return 0;
- }
+ if (pqPipelineFlush(conn) < 0)
+ goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
+
+sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
+ /* error message should be set up already */
+ return 0;
}
/*
@@ -1307,6 +1370,8 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+
if (!PQsendQueryStart(conn, true))
return 0;
@@ -1330,6 +1395,15 @@ PQsendPrepare(PGconn *conn,
return 0;
}
+ /* Alloc pipeline memory before doing anything */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+ }
+
/* construct the Parse message */
if (pqPutMsgStart('P', conn) < 0 ||
pqPuts(stmtName, conn) < 0 ||
@@ -1356,32 +1430,46 @@ PQsendPrepare(PGconn *conn,
if (pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /*
+ * In non-pipeline mode, add a Sync and prepare to send. In pipeline mode
+ * we just keep track of the new message.
+ */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ /* remember we are doing just a Parse */
+ conn->queryclass = PGQUERY_PREPARE;
- /* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
-
- /* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ /* and remember the query text too, if possible */
+ /* if insufficient memory, last_query just winds up NULL */
+ if (conn->last_query)
+ free(conn->last_query);
+ conn->last_query = strdup(query);
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ else
+ {
+ pipeCmd->queryclass = PGQUERY_PREPARE;
+ /* as above, if insufficient memory, query winds up NULL */
+ pipeCmd->query = strdup(query);
+ pqAppendPipelineCmd(conn, pipeCmd);
+ }
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
- /* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1429,7 +1517,8 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn, bool newQuery)
@@ -1450,20 +1539,57 @@ PQsendQueryStart(PGconn *conn, bool newQuery)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE &&
+ conn->pipelineStatus == PQ_PIPELINE_OFF)
{
appendPQExpBufferStr(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * When enqueuing commands we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when pqPipelineProcessQueue()
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ }
+ }
+ else
+ {
+ /*
+ * This command's results will come in immediately. Initialize async
+ * result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1487,10 +1613,34 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **query;
+ PGQueryClass *queryclass;
/*
- * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
- * using specified statement name and the unnamed portal.
+ * Decide where the query is going to be stored. In pipeline mode, we
+ * allocate a new pipeline element; in non-pipeline mode, it's simply the
+ * connection's last query.
+ */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+ /*
+ * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync
+ * (if not in pipeline mode), using specified statement name and the
+ * unnamed portal.
*/
if (command)
@@ -1600,35 +1750,43 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /* construct the Sync message if not in pipeline mode */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ /* if insufficient memory, query just winds up NULL */
+ if (*query)
+ free(*query);
if (command)
- conn->last_query = strdup(command);
+ *query = strdup(command);
else
- conn->last_query = NULL;
+ *query = NULL;
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1726,14 +1884,17 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
-
/*
* PQgetResult
* Get the next PGresult produced by a query. Returns NULL if no
* query work remains or an error has occurred (e.g. out of
* memory).
+ *
+ * In pipeline mode, once all the result of a query have been returned,
+ * PQgetResult returns NULL to let the user know that the next
+ * query is being processed. At the end of the pipeline, returns a
+ * result with PQresultStatus(result) == PGRES_PIPELINE_SYNC.
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1803,8 +1964,49 @@ PQgetResult(PGconn *conn)
{
case PGASYNC_IDLE:
res = NULL; /* query is complete */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to return the NULL that terminates the round of
+ * results from the current query; prepare to send the results
+ * of the next query when we're called next. Also, since this
+ * is the start of the results of the next query, clear any
+ * prior error message.
+ */
+ resetPQExpBuffer(&conn->errorMessage);
+ pqPipelineProcessQueue(conn);
+ }
break;
case PGASYNC_READY:
+ res = pqPrepareAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to send the results of the current query. Set
+ * us idle now, and ...
+ */
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /*
+ * ... in cases when we're sending a pipeline-related result,
+ * move queue processing forwards immediately, so that next
+ * time we're called, we're prepared to return the next result
+ * received from the server.
+ *
+ * In all other cases, leave the queue state change for next
+ * time, so that a terminating NULL result is sent.
+ */
+ if (res && (res->resultStatus == PGRES_PIPELINE_ABORTED ||
+ res->resultStatus == PGRES_PIPELINE_SYNC))
+ pqPipelineProcessQueue(conn);
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -1985,6 +2187,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("synchronous command execution functions are not allowed in pipeline mode\n"));
+ return false;
+ }
+
/*
* Since this is the beginning of a query cycle, reset the error buffer.
*/
@@ -2148,6 +2357,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2155,6 +2367,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
if (!PQsendQueryStart(conn, true))
return 0;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ queryclass = &conn->queryclass;
+
/* construct the Describe message */
if (pqPutMsgStart('D', conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2163,32 +2387,40 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ /* reset last-query string (not relevant now) */
+ if (conn->last_query && conn->pipelineStatus != PQ_PIPELINE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
}
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -2541,6 +2773,13 @@ PQfn(PGconn *conn,
*/
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("PQfn not allowed in pipeline mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -2555,6 +2794,359 @@ PQfn(PGconn *conn,
args, nargs);
}
+/* ====== Pipeline mode support ======== */
+
+/*
+ * PQenterPipelineMode
+ * Put an idle connection in pipeline mode.
+ *
+ * Returns 1 on success. On failure, errorMessage is set and 0 is returned.
+ *
+ * Commands submitted after this can be pipelined on the connection;
+ * there's no requirement to wait for one to finish before the next is
+ * dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQsendPipeline. Multiple pipelines
+ * can be sent while in pipeline mode. Pipeline mode can be exited
+ * by calling PQexitPipelineMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ /* succeed with no action if already in pipeline mode */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot enter pipeline mode, connection not idle\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+
+ return 1;
+}
+
+/*
+ * PQexitPipelineMode
+ * End pipeline mode and return to normal command mode.
+ *
+ * Returns 1 in success (pipeline mode successfully ended, or not in pipeline
+ * mode).
+ *
+ * Returns 0 if in pipeline mode and cannot be ended yet. Error message will
+ * be set.
+ */
+int
+PQexitPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ /* there are some uncollected results */
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+
+ case PGASYNC_BUSY:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode while busy\n"));
+ return 0;
+
+ default:
+ /* OK */
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ return 0; /* error message is setup already */
+ return 1;
+}
+
+/*
+ * pqPipelineProcessQueue: subroutine for PQgetResult
+ * In pipeline mode, start processing the results of the next query in the queue.
+ */
+static void
+pqPipelineProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ Assert(conn->pipelineStatus != PQ_PIPELINE_OFF);
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: COPY in pipeline mode\n");
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return;
+ case PGASYNC_IDLE:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In pipeline mode but nothing left on the queue; caller can submit
+ * more work or PQexitPipelineMode() now.
+ */
+ return;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-pipeline call.
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /*
+ * Reset single-row processing mode. (Client has to set it up for each
+ * query, if desired.)
+ */
+ conn->singleRowMode = false;
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ pqRecyclePipelineCmd(conn, next_query);
+
+ if (conn->pipelineStatus == PQ_PIPELINE_ABORTED &&
+ conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted pipeline we don't get anything from the server for
+ * each result; we're just discarding input until we get to the next
+ * sync from the server. The client needs to know its queries got
+ * aborted so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn, PGRES_PIPELINE_ABORTED);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ pqSaveErrorResult(conn);
+ return;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+}
+
+/*
+ * PQsendPipeline
+ * Send a Sync message as part of a pipeline, and flush to server
+ *
+ * It's legal to start submitting more commands in the pipeline immediately,
+ * without waiting for the results of the current pipeline. There's no need to
+ * end pipeline mode and start it again.
+ *
+ * If a command in a pipeline fails, every subsequent command up to and including
+ * the result to the Sync message sent by PQsendPipeline gets set to
+ * PGRES_PIPELINE_ABORTED state. If the whole pipeline is processed without
+ * error, a PGresult with PGRES_PIPELINE_SYNC is produced.
+ *
+ * Queries can already have been sent before PQsendPipeline is called, but
+ * PQsendPipeline need to be called before retrieving command results.
+ *
+ * The connection will remain in pipeline mode and unavailable for new
+ * synchronous command execution functions until all results from the pipeline
+ * are processed by the client.
+ */
+int
+PQsendPipeline(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot send pipeline when not in pipeline mode\n"));
+ return 0;
+ }
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: cannot send pipeline while in COPY\n");
+ return 0;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_IDLE:
+ /* OK to send sync */
+ break;
+ }
+
+ entry = pqMakePipelineCmd(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ pqAppendPipelineCmd(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ /*
+ * Call pqPipelineProcessQueue so the user can call start calling
+ * PQgetResult.
+ */
+ pqPipelineProcessQueue(conn);
+
+ return 1;
+
+sendFailed:
+ pqRecyclePipelineCmd(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
+
+/*
+ * pqMakePipelineCmd
+ * Get a command queue entry for caller to fill.
+ *
+ * If the recycle queue has a free element, that is returned; if not, a
+ * fresh one is allocated. Caller is responsible for adding it to the
+ * command queue (pqAppendPipelineCmd) once the struct is filled in, or
+ * releasing the memory (pqRecyclePipelineCmd) if an error occurs.
+ *
+ * If allocation fails, sets the error message and returns NULL.
+ */
+static PGcommandQueueEntry *
+pqMakePipelineCmd(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * pqAppendPipelineCmd
+ * Append a caller-allocated command queue entry to the queue.
+ *
+ * The query itself must already have been put in the output buffer by the
+ * caller.
+ */
+static void
+pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ PGcommandQueueEntry **tail;
+
+ if (conn->cmd_queue_head == NULL)
+ tail = &conn->cmd_queue_head;
+ else
+ tail = &conn->cmd_queue_tail->next;
+
+ *tail = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * pqRecyclePipelineCmd
+ * Push a command queue entry onto the freelist. It must be an entry
+ * with null next pointer and not referenced by any other entry's next
+ * pointer.
+ */
+static void
+pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+
+ Assert(entry->next == NULL);
+
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
/* ====== accessor funcs for PGresult ======== */
@@ -3152,6 +3744,23 @@ PQflush(PGconn *conn)
return pqFlush(conn);
}
+/*
+ * pqPipelineFlush
+ *
+ * In pipeline mode, data will be flushed only when the out buffer reaches the
+ * threshold value. In non-pipeline mode, it behaves as stock pqFlush.
+ *
+ * Returns 0 on success.
+ */
+static int
+pqPipelineFlush(PGconn *conn)
+{
+ if ((conn->pipelineStatus == PQ_PIPELINE_OFF) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return pqFlush(conn);
+ return 0;
+}
+
/*
* PQfreemem - safely frees memory allocated
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 2ca8c057b9..b131995974 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -177,14 +177,24 @@ pqParseInput3(PGconn *conn)
if (getParameterStatus(conn))
return;
}
- else
- {
- pqInternalNotice(&conn->noticeHooks,
- "message type 0x%02x arrived from server while idle",
- id);
- /* Discard the unexpected message */
- conn->inCursor += msgLength;
- }
+
+ /*
+ * We're also IDLE when in pipeline mode we have completed
+ * processing the results of one query and are waiting for the
+ * next one in the pipeline. In this case, as above, just wait to
+ * see what's next.
+ */
+ if (conn->asyncStatus == PGASYNC_IDLE &&
+ conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->cmd_queue_head != NULL)
+ return;
+
+ /* Any other case is unexpected and we summarily skip it */
+ pqInternalNotice(&conn->noticeHooks,
+ "message type 0x%02x arrived from server while idle",
+ id);
+ /* Discard the unexpected message */
+ conn->inCursor += msgLength;
}
else
{
@@ -217,10 +227,28 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new
+ * query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_PIPELINE_SYNC);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ else
+ {
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -450,7 +478,7 @@ handleSyncLoss(PGconn *conn, char id, int msgLength)
id, msgLength);
/* build an error result holding the error message */
pqSaveErrorResult(conn);
- conn->asyncStatus = PGASYNC_READY; /* drop out of GetResult wait loop */
+ conn->asyncStatus = PGASYNC_READY; /* drop out of PQgetResult wait loop */
/* flush input data since we're giving up on processing it */
pqDropConnection(conn, true);
conn->status = CONNECTION_BAD; /* No more connection to backend */
@@ -875,6 +903,10 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ /* If in pipeline mode, set error indicator for it */
+ if (isError && conn->pipelineStatus != PQ_PIPELINE_OFF)
+ conn->pipelineStatus = PQ_PIPELINE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
@@ -930,9 +962,21 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
* Save the active query text, if any, into res as well; but only if we
* might need it for an error cursor display, which is only true if there
* is a PG_DIAG_STATEMENT_POSITION field.
+ *
+ * Note that in pipeline mode, we have not yet advanced the query pointer
+ * to the next query, so we have to look at that.
*/
- if (have_position && conn->last_query && res)
- res->errQuery = pqResultStrdup(res, conn->last_query);
+ if (have_position && res)
+ {
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->asyncStatus == PGASYNC_IDLE)
+ {
+ if (conn->cmd_queue_head)
+ res->errQuery = pqResultStrdup(res, conn->cmd_queue_head->query);
+ }
+ else if (conn->last_query)
+ res->errQuery = pqResultStrdup(res, conn->last_query);
+ }
/*
* Now build the "overall" error message for PQresultErrorMessage.
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index fa9b62a844..fb31a49fab 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -96,7 +96,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_PIPELINE_SYNC, /* pipeline synchronization point */
+ PGRES_PIPELINE_ABORTED, /* Command didn't run because of an abort
+ * earlier in a pipeline */
} ExecStatusType;
typedef enum
@@ -136,6 +139,16 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PGpipelineStatus - Current status of pipeline mode
+ */
+typedef enum
+{
+ PQ_PIPELINE_OFF,
+ PQ_PIPELINE_ON,
+ PQ_PIPELINE_ABORTED
+} PGpipelineStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -327,6 +340,7 @@ extern int PQserverVersion(const PGconn *conn);
extern char *PQerrorMessage(const PGconn *conn);
extern int PQsocket(const PGconn *conn);
extern int PQbackendPID(const PGconn *conn);
+extern PGpipelineStatus PQpipelineStatus(const PGconn *conn);
extern int PQconnectionNeedsPassword(const PGconn *conn);
extern int PQconnectionUsedPassword(const PGconn *conn);
extern int PQclientEncoding(const PGconn *conn);
@@ -434,6 +448,11 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for pipeline mode management */
+extern int PQenterPipelineMode(PGconn *conn);
+extern int PQexitPipelineMode(PGconn *conn);
+extern int PQsendPipeline(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 8d51e6ed9f..3e9f99f8fb 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,7 +217,11 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, more results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
@@ -229,7 +233,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* Sync at end of a pipeline */
} PGQueryClass;
/* Target server type (decoded value of target_session_attrs) */
@@ -305,6 +310,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by pipeline mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct PGcommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct PGcommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -397,6 +418,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PGpipelineStatus pipelineStatus; /* status of pipeline mode */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -409,6 +431,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue, for pipeline mode.
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -790,6 +822,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 5391f461a2..93e7829c67 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -10,6 +10,7 @@ SUBDIRS = \
delay_execution \
dummy_index_am \
dummy_seclabel \
+ libpq_pipeline \
plsample \
snapshot_too_old \
test_bloomfilter \
diff --git a/src/test/modules/libpq_pipeline/.gitignore b/src/test/modules/libpq_pipeline/.gitignore
new file mode 100644
index 0000000000..3a11e786b8
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/libpq_pipeline
diff --git a/src/test/modules/libpq_pipeline/Makefile b/src/test/modules/libpq_pipeline/Makefile
new file mode 100644
index 0000000000..b798f5fbbc
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/Makefile
@@ -0,0 +1,20 @@
+# src/test/modules/libpq_pipeline/Makefile
+
+PROGRAM = libpq_pipeline
+OBJS = libpq_pipeline.o
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS_INTERNAL += $(libpq_pgport)
+
+TAP_TESTS = 1
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/libpq_pipeline
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/libpq_pipeline/README b/src/test/modules/libpq_pipeline/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/libpq_pipeline/libpq_pipeline.c b/src/test/modules/libpq_pipeline/libpq_pipeline.c
new file mode 100644
index 0000000000..1074634384
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/libpq_pipeline.c
@@ -0,0 +1,1152 @@
+/*
+ * src/test/modules/libpq_pipeline/libpq_pipeline.c
+ * Verify libpq pipeline execution functionality
+ */
+#include "postgres_fe.h"
+
+#include <sys/time.h>
+#ifdef HAVE_SYS_SELECT_H
+#include <sys/select.h>
+#endif
+
+#include "catalog/pg_type_d.h"
+#include "common/fe_memutils.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+
+static void exit_nicely(PGconn *conn);
+
+const char *const progname = "libpq_pipeline";
+
+
+#define DEBUG
+#ifdef DEBUG
+#define pg_debug(...) do { fprintf(stderr, __VA_ARGS__); } while (0)
+#else
+#define pg_debug(...)
+#endif
+
+static const char *const drop_table_sql =
+"DROP TABLE IF EXISTS pq_pipeline_demo";
+static const char *const create_table_sql =
+"CREATE UNLOGGED TABLE pq_pipeline_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql =
+"INSERT INTO pq_pipeline_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+/*
+ * Print an error to stderr and terminate the program.
+ */
+#define pg_fatal(...) pg_fatal_impl(__LINE__, __VA_ARGS__)
+static void
+pg_fatal_impl(int line, const char *fmt,...)
+{
+ va_list args;
+
+ fprintf(stderr, "\n"); /* XXX hack */
+ fprintf(stderr, "%s:%d: ", progname, line);
+
+ va_start(args, fmt);
+ vfprintf(stderr, fmt, args);
+ va_end(args);
+ printf("Failure, exiting\n");
+ exit(1);
+}
+
+static void
+test_disallowed(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("Unable to enter pipeline mode\n");
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not activated properly\n");
+
+ /* PQexec should fail in pipeline mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("PQexec should fail in pipeline mode but succeeded\n");
+
+ /* Entering pipeline mode when already in pipeline mode is OK */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("re-entering pipeline mode should be a no-op but failed\n");
+
+ if (PQisBusy(conn) != 0)
+ pg_fatal("PQisBusy should return 0 when idle in pipeline mode, returned 1\n");
+
+ /* ok, back to normal command mode */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("couldn't exit idle empty pipeline mode\n");
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not terminated properly\n");
+
+ /* exiting pipeline mode when not in pipeline mode should be a no-op */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("pipeline mode exit when not in pipeline mode should succeed but failed\n");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("PQexec should succeed after exiting pipeline mode but failed with: %s\n",
+ PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_simple_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple pipeline... ");
+
+ /*
+ * Enter pipeline mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our output buffer will give us enough slush
+ * to work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode with work in progress should fail, but succeeded\n");
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending pipeline failed: %s\n", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first query result.\n");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit pipeline mode yet
+ */
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result PGRES_PIPELINE_SYNC expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of PGRES_PIPELINE_SYNC, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after pipeline end: %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* ... until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_multi_pipelines(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi pipeline... ");
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending first pipeline failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s\n", PQerrorMessage(conn));
+
+ /* OK, start processing the results */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first result\n");
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ PQclear(res);
+
+#if 0
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+#endif
+
+ /* second pipeline */
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from second pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode ... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+/*
+ * When an operation in a pipeline fails the rest of the pipeline is flushed. We
+ * still have to get results for each pipeline item, but the item will just be
+ * a PGRES_PIPELINE_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the pipeline. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_aborted_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+ bool goterror;
+
+ fprintf(stderr, "aborted pipeline... ");
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * pipeline ERRORs.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT no_such_function($1)",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching error select failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending first pipeline failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second-pipeline insert failed: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the pipeline results.
+ *
+ * We should get a command-ok for the first query, then a fatal error and
+ * a pipeline aborted message for the second insert, a pipeline-end, then
+ * a command-ok and a pipeline-ok for the second pipeline operation.
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result status %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* Second query caused error, so we expect an error next */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("Unexpected result code -- expected PGRES_FATAL_ERROR, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /*
+ * pipeline should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third pipeline since we've already sent a
+ * second. The aborted flag relates only to the pipeline being received.
+ */
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't\n");
+
+ /* third query in pipeline, the second insert */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_ABORTED)
+ pg_fatal("Unexpected result code -- expected PGRES_PIPELINE_ABORTED, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+#if 0
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+#endif
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't\n");
+
+ /* Ensure we're still in pipeline */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /*
+ * The end of a failed pipeline is a PGRES_PIPELINE_SYNC.
+ *
+ * (This is so clients know to start processing results normally again and
+ * can tell the difference between skipped commands and the sync.)
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code from first pipeline sync\n"
+ "Expected PGRES_PIPELINE_SYNC, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+#if 0
+ /* after the synchronization point we get a NULL result */
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+#endif
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_ABORTED)
+ pg_fatal("sync should've cleared the aborted flag but didn't\n");
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* the insert from the second pipeline */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result code %s from first item in second pipeline\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* Read the NULL result at the end of the command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* the second pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQerrorMessage(conn));
+
+ /* Test single-row mode with an error partways */
+ if (PQsendQuery(conn, "SELECT 1.0/g FROM generate_series(3, -1, -1) g") != 1)
+ pg_fatal("failed to send query: %s", PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ PQsetSingleRowMode(conn);
+ goterror = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_SINGLE_TUPLE:
+ printf("got row: %s\n", PQgetvalue(res, 0, 0));
+ break;
+ case PGRES_FATAL_ERROR:
+ if (strcmp(PQresultErrorField(res, PG_DIAG_SQLSTATE), "22012") != 0)
+ pg_fatal("expected division-by-zero, got: %s\n",
+ PQresultErrorField(res, PG_DIAG_SQLSTATE));
+ printf("got expected division-by-zero\n");
+ goterror = true;
+ break;
+ default:
+ pg_fatal("got unexpected result %s\n", PQresStatus(PQresultStatus(res)));
+ }
+ PQclear(res);
+ }
+ if (!goterror)
+ pg_fatal("did not get division-by-zero error\n");
+ /* the third pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from third pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ /*-
+ * Since we fired the pipelines off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st pipeline
+ * - First insert applied
+ * - Second statement aborted xact
+ * - Third insert skipped
+ * - Sync rolled back first implicit xact
+ * - Implicit xact created by server around 2nd pipeline
+ * - insert applied from 2nd pipeline
+ * - Sync commits 2nd xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM pq_pipeline_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Expected tuples, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("expected 1 result, got %d\n", PQntuples(res));
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ pg_fatal("expected only insert with value 3, got %s", val);
+ }
+
+ PQclear(res);
+}
+
+/* State machine enum for test_pipelined_insert */
+typedef enum PipelineInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} PipelineInsertStep;
+
+static void
+test_pipelined_insert(PGconn *conn, int n_rows)
+{
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ PipelineInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a pipelined insert into a table created at the start of the pipeline
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ while (send_step != BI_PREPARE)
+ {
+ const char *sql;
+
+ switch (send_step)
+ {
+ case BI_BEGIN_TX:
+ sql = "BEGIN TRANSACTION";
+ send_step = BI_DROP_TABLE;
+ break;
+
+ case BI_DROP_TABLE:
+ sql = drop_table_sql;
+ send_step = BI_CREATE_TABLE;
+ break;
+
+ case BI_CREATE_TABLE:
+ sql = create_table_sql;
+ send_step = BI_PREPARE;
+ break;
+
+ default:
+ pg_fatal("invalid state");
+ }
+
+ pg_debug("sending: %s\n", sql);
+ if (PQsendQueryParams(conn, sql,
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching %s failed: %s\n", sql, PQerrorMessage(conn));
+ }
+
+ Assert(send_step == BI_PREPARE);
+ pg_debug("sending: %s\n", insert_sql);
+ if (PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids) != 1)
+ pg_fatal("dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our output buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the pipeline before processing
+ * results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ pg_fatal("failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's output buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ PGresult *res;
+ const char *cmdtag;
+ const char *description = "";
+ int status;
+
+ /*
+ * Read next result. If no more results from this query,
+ * advance to the next query
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ continue;
+
+ status = PGRES_COMMAND_OK;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ recv_step++;
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ recv_step++;
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ recv_step++;
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ recv_step++;
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive == 0)
+ recv_step++;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ recv_step++;
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_PIPELINE_SYNC;
+ recv_step++;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ description = "";
+ abort();
+ }
+
+ if (PQresultStatus(res) != status)
+ pg_fatal("%s reported status %s, expected %s\n"
+ "Error message: \"%s\"\n",
+ description, PQresStatus(PQresultStatus(res)),
+ PQresStatus(status), PQerrorMessage(conn));
+
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ pg_fatal("%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+
+ pg_debug("Got %s OK\n", cmdtag[0] != '\0' ? cmdtag : description);
+
+ PQclear(res);
+ }
+ }
+
+ /* Write more rows and/or the end pipeline message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQsendPipeline(conn) == 1)
+ {
+ fprintf(stdout, "Sent pipeline\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending pipeline failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+ }
+
+ /* We've got the sync message and the pipeline should be done */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ pg_fatal("failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i;
+ bool pipeline_ended = false;
+
+ /* 1 pipeline, 3 queries in it */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n",
+ PQerrorMessage(conn));
+
+ for (i = 0; i < 3; i++)
+ {
+ char *param[1];
+
+ param[0] = psprintf("%d", 44 + i);
+
+ if (PQsendQueryParams(conn,
+ "SELECT generate_series(42, $1)",
+ 1,
+ NULL,
+ (const char **) param,
+ NULL,
+ NULL,
+ 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ pfree(param[0]);
+ }
+ PQsendPipeline(conn);
+
+ for (i = 0; !pipeline_ended; i++)
+ {
+ bool first = true;
+ bool saw_ending_tuplesok;
+ bool isSingleTuple = false;
+
+ /* Set single row mode for only first 2 SELECT queries */
+ if (i < 2)
+ {
+ if (PQsetSingleRowMode(conn) != 1)
+ pg_fatal("PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+
+ /* Consume rows for this query */
+ saw_ending_tuplesok = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+
+ if (est == PGRES_PIPELINE_SYNC)
+ {
+ fprintf(stderr, "end of pipeline reached\n");
+ pipeline_ended = true;
+ PQclear(res);
+ if (i != 3)
+ pg_fatal("Expected three results, got %d\n", i);
+ break;
+ }
+
+ /* Expect SINGLE_TUPLE for queries 0 and 1, TUPLES_OK for 2 */
+ if (first)
+ {
+ if (i <= 1 && est != PGRES_SINGLE_TUPLE)
+ pg_fatal("Expected PGRES_SINGLE_TUPLE for query %d, got %s\n",
+ i, PQresStatus(est));
+ if (i >= 2 && est != PGRES_TUPLES_OK)
+ pg_fatal("Expected PGRES_TUPLES_OK for query %d, got %s\n",
+ i, PQresStatus(est));
+ first = false;
+ }
+
+ fprintf(stderr, "Result status %s for query %d", PQresStatus(est), i);
+ switch (est)
+ {
+ case PGRES_TUPLES_OK:
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ saw_ending_tuplesok = true;
+ if (isSingleTuple)
+ {
+ if (PQntuples(res) == 0)
+ fprintf(stderr, "all tuples received in query %d\n", i);
+ else
+ pg_fatal("Expected to follow PGRES_SINGLE_TUPLE, "
+ "but received PGRES_TUPLES_OK directly instead\n");
+ }
+ break;
+
+ case PGRES_SINGLE_TUPLE:
+ isSingleTuple = true;
+ fprintf(stderr, ", %d tuple: %s\n", PQntuples(res), PQgetvalue(res, 0, 0));
+ break;
+
+ default:
+ pg_fatal("unexpected\n");
+ }
+ PQclear(res);
+ }
+ if (!pipeline_ended && !saw_ending_tuplesok)
+ pg_fatal("didn't get expected terminating TUPLES_OK\n");
+ }
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s\n", PQerrorMessage(conn));
+}
+
+/*
+ * Simple test to verify that a pipeline is discarded as a whole when there's
+ * an error, ignoring transaction commands.
+ */
+static void
+test_transaction(PGconn *conn)
+{
+ PGresult *res;
+ int num_sends = 0;
+
+ res = PQexec(conn, "DROP TABLE IF EXISTS pq_pipeline_tst;"
+ "CREATE TABLE pq_pipeline_tst (id int)");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to create test table: %s",
+ PQerrorMessage(conn));
+ PQclear(res);
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n",
+ PQerrorMessage(conn));
+ if (PQsendPrepare(conn, "rollback", "ROLLBACK", 0, NULL) != 1)
+ pg_fatal("could not send prepare on pipeline: %s",
+ PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn,
+ "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ if (PQsendQueryParams(conn,
+ "SELECT 0/0",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+
+ /*
+ * send a ROLLBACK using a prepared stmt. Doesn't work because we need to
+ * get out of the pipeline-aborted state first.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /* This insert fails because we're in pipeline-aborted state */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (1)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * This insert fails even though the pipeline got a SYNC, because we're in
+ * an aborted transaction
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (2)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * Send ROLLBACK using prepared stmt. This one works because we just did
+ * PQsendPipeline above.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /*
+ * Now that we're out of a transaction and in pipeline-good mode, this
+ * insert works
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (3)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+ PQsendPipeline(conn);
+ num_sends++;
+
+ for (int i = 0;; i++)
+ {
+ ExecStatusType restype;
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "%d: got NULL result\n", i);
+ continue;
+ }
+ restype = PQresultStatus(res);
+ fprintf(stderr, "%d: got status %s", i, PQresStatus(restype));
+ if (restype == PGRES_FATAL_ERROR)
+ fprintf(stderr, "; error: %s", PQerrorMessage(conn));
+ else if (restype == PGRES_PIPELINE_ABORTED)
+ {
+ fprintf(stderr, ": command didn't run because pipeline aborted\n");
+ }
+ else
+ fprintf(stderr, "\n");
+ PQclear(res);
+
+ if (restype == PGRES_PIPELINE_SYNC)
+ num_sends--;
+ if (num_sends <= 0)
+ break;
+ }
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("returned something extra after all the syncs: %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s\n", PQerrorMessage(conn));
+
+ /* We expect to find one tuple containing the value "3" */
+ res = PQexec(conn, "SELECT * FROM pq_pipeline_tst");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("failed to obtain result: %s", PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("did not get 1 tuple\n");
+ if (strcmp(PQgetvalue(res, 0, 0), "3") != 0)
+ pg_fatal("did not get expected tuple\n");
+ PQclear(res);
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+usage(const char *progname)
+{
+ fprintf(stderr, "%s tests libpq's pipeline mode.\n\n", progname);
+ fprintf(stderr, "Usage:\n");
+ fprintf(stderr, " %s testname [conninfo [number_of_rows]]\n", progname);
+ fprintf(stderr, "Tests:\n");
+ fprintf(stderr, " disallowed_in_pipeline\n");
+ fprintf(stderr, " simple_pipeline\n");
+ fprintf(stderr, " multi_pipeline\n");
+ fprintf(stderr, " pipeline_abort\n");
+ fprintf(stderr, " singlerow\n");
+ fprintf(stderr, " pipeline_insert\n");
+ fprintf(stderr, " transaction\n");
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo = "";
+ PGconn *conn;
+ int numrows = 10000;
+ PGresult *res;
+
+ /*
+ * The testname parameter is mandatory; it can be followed by a conninfo
+ * string and number of rows.
+ */
+ if (argc < 2 || argc > 4)
+ {
+ usage(argv[0]);
+ exit(1);
+ }
+
+ if (argc >= 3)
+ conninfo = pg_strdup(argv[2]);
+
+ if (argc >= 4)
+ {
+ errno = 0;
+ numrows = strtol(argv[3], NULL, 10);
+ if (errno != 0 || numrows <= 0)
+ {
+ fprintf(stderr, "couldn't parse \"%s\" as a positive integer\n", argv[3]);
+ exit(1);
+ }
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+ res = PQexec(conn, "SET lc_messages TO \"C\"");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to set lc_messages: %s", PQerrorMessage(conn));
+
+ if (strcmp(argv[1], "disallowed_in_pipeline") == 0)
+ test_disallowed(conn);
+ else if (strcmp(argv[1], "simple_pipeline") == 0)
+ test_simple_pipeline(conn);
+ else if (strcmp(argv[1], "multi_pipeline") == 0)
+ test_multi_pipelines(conn);
+ else if (strcmp(argv[1], "pipeline_abort") == 0)
+ test_aborted_pipeline(conn);
+ else if (strcmp(argv[1], "pipeline_insert") == 0)
+ test_pipelined_insert(conn, numrows);
+ else if (strcmp(argv[1], "singlerow") == 0)
+ test_singlerowmode(conn);
+ else if (strcmp(argv[1], "transaction") == 0)
+ test_transaction(conn);
+ else
+ {
+ fprintf(stderr, "\"%s\" is not a recognized test name\n", argv[1]);
+ usage(argv[0]);
+ exit(1);
+ }
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+ return 0;
+}
diff --git a/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
new file mode 100644
index 0000000000..a12f2dd47b
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
@@ -0,0 +1,31 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 7;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_pipeline
+ simple_pipeline
+ multi_pipeline
+ pipeline_abort
+ pipeline_insert
+ singlerow
+ transaction);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'libpq_pipeline', $testname, $node->connstr('postgres'), $numrows ],
+ "libp_pipeline $testname");
+}
+
+$node->stop('fast');
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 49614106dc..44b1a43f30 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -33,10 +33,11 @@ my @unlink_on_exit;
# Set of variables for modules in contrib/ and src/test/modules/
my $contrib_defines = { 'refint' => 'REFINT_VERBOSE' };
-my @contrib_uselibpq = ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo');
-my @contrib_uselibpgport = ('oid2name', 'vacuumlo');
-my @contrib_uselibpgcommon = ('oid2name', 'vacuumlo');
-my $contrib_extralibs = undef;
+my @contrib_uselibpq =
+ ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo', 'libpq_pipeline');
+my @contrib_uselibpgport = ('libpq_pipeline', 'oid2name', 'vacuumlo');
+my @contrib_uselibpgcommon = ('libpq_pipeline', 'oid2name', 'vacuumlo');
+my $contrib_extralibs = { 'libpq_pipeline' => ['ws2_32.lib'] };
my $contrib_extraincludes = { 'dblink' => ['src/backend'] };
my $contrib_extrasource = {
'cube' => [ 'contrib/cube/cubescan.l', 'contrib/cube/cubeparse.y' ],
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 8bd95aefa1..1bd23eec6d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1561,10 +1561,12 @@ PG_Locale_Strategy
PG_Lock_Status
PG_init_t
PGcancel
+PGcommandQueueEntry
PGconn
PGdataValue
PGlobjfuncs
PGnotify
+PGpipelineStatus
PGresAttDesc
PGresAttValue
PGresParamDesc
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
On 2021-Mar-05, Alvaro Herrera wrote:
I'll take the weekend to think about the issue with conn->last_query and
conn->queryclass that I mentioned yesterday; other than that detail my
feeling is that this is committable, so I'll be looking at getting this
pushed early next weeks, barring opinions from others.
It took longer than I expected, but it works well now. conn->last_query
is gone; all commands, both in pipeline mode and in no-pipeline mode, go
via the command queue. This is cleaner all around; we don't have to
have the pipeline code "cheat" so that it looks like each command is
"last" at each point.
I have not absorbed David Johnston's latest doc suggestions yet.
I'm going to give the code a last renaming pass, on the idea that the
command queue is no longer exclusively for the pipeline mode, so some
things need less exclusionary names. But functionality wise AFAICS this
patch has the shape it ought to have.
--
�lvaro Herrera 39�49'30"S 73�17'W
Attachments:
v35-libpq-pipeline.patchtext/x-diff; charset=us-asciiDownload
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 910e9a81ea..0bffb92462 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -3180,6 +3180,33 @@ ExecStatusType PQresultStatus(const PGresult *res);
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-sync">
+ <term><literal>PGRES_PIPELINE_SYNC</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a
+ synchronization point in pipeline mode, requested by
+ <xref linkend="libpq-PQsendPipeline"/>.
+ This status occurs only when pipeline mode has been selected.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-aborted">
+ <term><literal>PGRES_PIPELINE_ABORTED</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a pipeline that has
+ received an error from the server. <function>PQgetResult</function>
+ must be called repeatedly, and each time it will return this status code
+ until the end of the current pipeline, at which point it will return
+ <literal>PGRES_PIPELINE_SYNC</literal> and normal processing can
+ resume.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
If the result status is <literal>PGRES_TUPLES_OK</literal> or
@@ -4926,6 +4953,473 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-pipeline-mode">
+ <title>Pipeline Mode</title>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>libpq</primary>
+ <secondary>pipeline mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>pipelining</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>batch mode</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> pipeline mode allows applications to
+ send a query without having to read the result of the previously
+ sent query. Taking advantage of the pipeline mode, a client will wait
+ less for the server, since multiple queries/results can be
+ sent/received in a single network transaction.
+ </para>
+
+ <para>
+ While pipeline mode provides a significant performance boost, writing
+ clients using the pipeline mode is more complex because it involves
+ managing a queue of pending queries and finding which result
+ corresponds to which query in the queue.
+ </para>
+
+ <para>
+ Pipeline mode also generally consumes more memory on both the client and server,
+ though careful and aggressive management of the send/receive queue can mitigate
+ this. This applies whether or not the connection is in blocking or non-blocking
+ mode.
+ </para>
+
+ <sect2 id="libpq-pipeline-using">
+ <title>Using Pipeline Mode</title>
+
+ <para>
+ To issue pipelines, the application must switch the connection
+ into pipeline mode,
+ which is done with <xref linkend="libpq-PQenterPipelineMode"/>.
+ <xref linkend="libpq-PQpipelineStatus"/> can be used
+ to test whether pipeline mode is active.
+ In pipeline mode, only <link linkend="libpq-async">asynchronous operations</link>
+ are permitted, and <literal>COPY</literal> is disallowed.
+ Using synchronous command execution functions
+ such as <function>PQfn</function>,
+ <function>PQexec</function>,
+ <function>PQexecParams</function>,
+ <function>PQprepare</function>,
+ <function>PQexecPrepared</function>,
+ <function>PQdescribePrepared</function>,
+ <function>PQdescribePortal</function>,
+ is an error condition.
+ Once all dispatched commands have had their results processed, and
+ the end pipeline result has been consumed, the application may return
+ to non-pipelined mode with <xref linkend="libpq-PQexitPipelineMode"/>.
+ </para>
+
+ <note>
+ <para>
+ It is best to use pipeline mode with <application>libpq</application> in
+ <link linkend="libpq-PQsetnonblocking">non-blocking mode</link>. If used
+ in blocking mode it is possible for a client/server deadlock to occur.
+ <footnote>
+ <para>
+ The client will block trying to send queries to the server, but the
+ server will block trying to send results to the client from queries
+ it has already processed. This only occurs when the client sends
+ enough queries to fill both its output buffer and the server's receive
+ buffer before it switches to processing input from the server,
+ but it's hard to predict exactly when that will happen.
+ </para>
+ </footnote>
+ </para>
+ </note>
+
+ <sect3 id="libpq-pipeline-sending">
+ <title>Issuing Queries</title>
+
+ <para>
+ After entering pipeline mode, the application dispatches requests using
+ <xref linkend="libpq-PQsendQuery"/>,
+ <xref linkend="libpq-PQsendQueryParams"/>,
+ or its prepared-query sibling
+ <xref linkend="libpq-PQsendQueryPrepared"/>.
+ These requests are queued on the client-side until flushed to the server;
+ this occurs when <xref linkend="libpq-PQsendPipeline"/> is used to
+ establish a synchronization point in the pipeline,
+ or when <xref linkend="libpq-PQflush"/> is called.
+ The functions <xref linkend="libpq-PQsendPrepare"/>,
+ <xref linkend="libpq-PQsendDescribePrepared"/>, and
+ <xref linkend="libpq-PQsendDescribePortal"/> also work in pipeline mode.
+ Result processing is described below.
+ </para>
+
+ <para>
+ The server executes statements, and returns results, in the order the
+ client sends them. The server will begin executing the commands in the
+ pipeline immediately, not waiting for the end of the pipeline.
+ If any statement encounters an error, the server aborts the current
+ transaction and skips processing commands in the pipeline until the
+ next synchronization point established by <function>PQsendPipeline</function>.
+ (This remains true even if the commands in the pipeline would rollback
+ the transaction.)
+ Query processing resumes after the synchronization point.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one; for example, one query may define a table that the next
+ query in the same pipeline uses. Similarly, an application may
+ create a named prepared statement and execute it with later
+ statements in the same pipeline.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-results">
+ <title>Processing Results</title>
+
+ <para>
+ To process the result of one query in a pipeline, the application calls
+ <function>PQgetResult</function> repeatedly and handles each result
+ until <function>PQgetResult</function> returns null.
+ The result from the next query in the pipeline may then be retrieved using
+ <function>PQgetResult</function> again and the cycle repeated.
+ The application handles individual statement results as normal.
+ When the results of all the queries in the pipeline have been
+ returned, <function>PQgetResult</function> returns a result
+ containing the status value <literal>PGRES_PIPELINE_SYNC</literal>
+ </para>
+
+ <para>
+ The client may choose to defer result processing until the complete
+ pipeline has been sent, or interleave that with sending further
+ queries in the pipeline; see <xref linkend="libpq-pipeline-interleave"/>.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ This mode selection is effective only for the query currently
+ being processed. For more information on the use of
+ <function>PQsetSingleRowMode</function>,
+ refer to <xref linkend="libpq-single-row-mode"/>.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal
+ asynchronous processing except that it may contain the new
+ <type>PGresult</type> types <literal>PGRES_PIPELINE_SYNC</literal>
+ and <literal>PGRES_PIPELINE_ABORTED</literal>.
+ <literal>PGRES_PIPELINE_SYNC</literal> is reported exactly once for each
+ <function>PQsendPipeline</function> after retrieving results for all
+ queries in the pipeline.
+ <literal>PGRES_PIPELINE_ABORTED</literal> is emitted in place of a normal
+ stream result for the first error and all subsequent results
+ until <literal>PGRES_PIPELINE_SYNC</literal>;
+ see <xref linkend="libpq-pipeline-errors"/>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing pipeline results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed (except that
+ <function>PQgetResult</function> returns null to indicate that we start
+ returning the results of next query). The application must keep track
+ of the order in which it sent queries, to associate them with their
+ corresponding results.
+ Applications will typically use a state machine or a FIFO queue for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-pipeline-errors">
+ <title>Error Handling</title>
+
+ <para>
+ From the client perspective, after <function>PQresultStatus</function>
+ returns <literal>PGRES_FATAL_ERROR</literal>,
+ the pipeline is flagged as aborted.
+ <function>PQresultStatus</function> will report a
+ <literal>PGRES_PIPELINE_ABORTED</literal> result for each remaining queued
+ operation in an aborted pipeline. The result for
+ <function>PQsendPipeline</function> is reported as
+ <literal>PGRES_PIPELINE_SYNC</literal> to signal the end of the aborted pipeline
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the pipeline used an implicit transaction, then operations that have
+ already executed are rolled back and operations that were queued to follow
+ the failed operation are skipped entirely. The same behaviour holds if the
+ pipeline starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the pipeline. If a pipeline contains
+ <emphasis>multiple explicit transactions</emphasis>, all transactions that
+ committed prior to the error remain committed, the currently in-progress
+ transaction is aborted, and all subsequent operations are skipped completely,
+ including subsequent transactions.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal> — only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously, the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-interleave">
+ <title>Interleaving Result Processing and Query Dispatch</title>
+
+ <para>
+ To avoid deadlocks on large pipelines the client should be structured
+ around a non-blocking event loop using operating system facilities
+ such as <function>select</function>, <function>poll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work
+ remaining to be dispatched and a queue of work that has been dispatched
+ but not yet had its results processed. When the socket is writable
+ it should dispatch more work. When the socket is readable it should
+ read results and process them, matching them up to the next entry in
+ its expected results queue. Based on available memory, results from the
+ socket should be read frequently: there's no need to wait until the
+ pipeline end to read the results. Pipelines should be scoped to logical
+ units of work, usually (but not necessarily) one transaction per pipeline.
+ There's no need to exit pipeline mode and re-enter it between pipelines,
+ or to wait for one pipeline to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state
+ machine to track sent and received work is in
+ <filename>src/test/modules/libpq_pipeline/libpq_pipeline.c</filename>
+ in the PostgreSQL source distribution.
+ </para>
+ </sect3>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-functions">
+ <title>Functions Associated with Pipeline Mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQpipelineStatus">
+ <term><function>PQpipelineStatus</function><indexterm><primary>PQpipelineStatus</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Returns the current pipeline mode status of the
+ <application>libpq</application> connection.
+<synopsis>
+PGpipelineStatus PQpipelineStatus(const PGconn *conn);
+</synopsis>
+ </para>
+
+ <para>
+ <function>PQpipelineStatus</function> can return one of the following values:
+
+ <variablelist>
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ON</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in
+ pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_OFF</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is
+ <emphasis>not</emphasis> in pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in pipeline
+ mode and an error occurred while processing the current pipeline.
+ The aborted flag is cleared when <function>PQresultStatus</function>
+ returns PGRES_PIPELINE_SYNC at the end of the pipeline.
+ Clients don't usually need this function to
+ verify aborted status, as they can tell that the pipeline is aborted
+ from the <literal>PGRES_PIPELINE_ABORTED</literal> result code.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterPipelineMode">
+ <term><function>PQenterPipelineMode</function><indexterm><primary>PQenterPipelineMode</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter pipeline mode if it is currently idle or
+ already in pipeline mode.
+
+<synopsis>
+int PQenterPipelineMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success.
+ Returns 0 and has no effect if the connection is not currently
+ idle, i.e., it has a result ready, or it is waiting for more
+ input from the server, etc.
+ This function does not actually send anything to the server,
+ it just changes the <application>libpq</application> connection
+ state.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitPipelineMode">
+ <term><function>PQexitPipelineMode</function><indexterm><primary>PQexitPipelineMode</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit pipeline mode if it is currently in pipeline mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitPipelineMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 1 and takes no action if not in
+ pipeline mode. If the current statement isn't finished processing
+ or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQsendPipeline">
+ <term><function>PQsendPipeline</function><indexterm><primary>PQsendPipeline</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Marks a synchronization point in a pipeline by sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ and flushing the send buffer. This serves as
+ the delimiter of an implicit transaction and an error recovery
+ point; see <xref linkend="libpq-pipeline-errors"/>.
+
+<synopsis>
+int PQsendPipeline(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 if the connection is not in
+ pipeline mode or sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ failed.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-tips">
+ <title>When to Use Pipeline Mode</title>
+
+ <para>
+ Much like asynchronous query mode, there is no meaningful performance
+ overhead when using pipeline mode. It increases client application complexity,
+ and extra caution is required to prevent client/server deadlocks, but
+ pipeline mode can offer considerable performance improvements, in exchange for
+ increased memory usage from leaving state around longer.
+ </para>
+
+ <para>
+ Pipeline mode is most useful when the server is distant, i.e., network latency
+ (<quote>ping time</quote>) is high, and also when many small operations
+ are being performed in rapid succession. There is usually less benefit
+ in using pipelined commands when each query takes many multiples of the client/server
+ round-trip time to execute. A 100-statement operation run on a server
+ 300ms round-trip-time away would take 30 seconds in network latency alone
+ without pipelining; with pipelining it may spend as little as 0.3s waiting for
+ results from the server.
+ </para>
+
+ <para>
+ Use pipelined commands when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed
+ into operations on sets, or into a <literal>COPY</literal> operation.
+ </para>
+
+ <para>
+ Pipeline mode is not useful when information from one operation is required by
+ the client to produce the next operation. In such cases, the client
+ would have to introduce a synchronization point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+BEGIN;
+SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+-- result: x=2
+-- client adds 1 to x:
+UPDATE mytable SET x = 3 WHERE id = 42;
+COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <para>
+ Pipelining is less useful, and more complex, when a single pipeline contains
+ multiple transactions (see <xref linkend="libpq-pipeline-errors"/>).
+ </para>
+
+ <note>
+ <para>
+ The pipeline API was introduced in <productname>PostgreSQL</productname> 14.
+ Pipeline mode is a client-side feature which doesn't require server
+ support, and works on any server that supports the v3 extended query
+ protocol.
+ </para>
+ </note>
+ </sect2>
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4966,6 +5460,13 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <para>
+ When using pipeline mode, single-row mode needs to be activated for each
+ query in the pipeline before retrieving results for that query
+ with <function>PQgetResult</function>.
+ See <xref linkend="libpq-pipeline-mode"/> for more information.
+ </para>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 6d46da42e2..012e44c736 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while a libpq connection is in pipeline mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 299d93b241..5dd1e9e936 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -1110,6 +1110,12 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
row, the last value is kept.
</para>
+ <para>
+ <literal>\gset</literal> and <literal>\aset</literal> cannot be used
+ pipeline mode, since query results are not immediately
+ fetched in this mode.
+ </para>
+
<para>
The following example puts the final account balance from the first query
into variable <replaceable>abalance</replaceable>, and fills variables
@@ -1270,6 +1276,21 @@ SELECT 4 AS four \; SELECT 5 AS five \aset
</programlisting></para>
</listitem>
</varlistentry>
+
+ <varlistentry id='pgbench-metacommand-pipeline'>
+ <term><literal>\startpipeline</literal></term>
+ <term><literal>\endpipeline</literal></term>
+
+ <listitem>
+ <para>
+ These commands delimit the start and end of a pipeline of SQL statements.
+ In a pipeline, statements are sent to server without waiting for the results
+ of previous statements (see <xref linkend="libpq-pipeline-mode"/>).
+ Pipeline mode requires the extended query protocol.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect2>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 5272eed9ab..f74378110a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1019,6 +1019,12 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->err = _("empty query");
break;
+ case PGRES_PIPELINE_SYNC:
+ case PGRES_PIPELINE_ABORTED:
+ walres->status = WALRCV_ERROR;
+ walres->err = _("unexpected pipeline mode");
+ break;
+
case PGRES_NONFATAL_ERROR:
case PGRES_FATAL_ERROR:
case PGRES_BAD_RESPONSE:
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index f6a214669c..b7400708fc 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -395,10 +395,11 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
- * and we enter the CSTATE_SLEEP state to wait for it to expire. Other
- * meta-commands are executed immediately. If the command about to start
- * is actually beyond the end of the script, advance to CSTATE_END_TX.
+ * CSTATE_WAIT_RESULT state unless in pipeline mode. On a \sleep
+ * meta-command, the timer is set, and we enter the CSTATE_SLEEP state to
+ * wait for it to expire. Other meta-commands are executed immediately. If
+ * the command about to start is actually beyond the end of the script,
+ * advance to CSTATE_END_TX.
*
* CSTATE_WAIT_RESULT waits until we get a result set back from the server
* for the current command.
@@ -530,7 +531,9 @@ typedef enum MetaCommand
META_IF, /* \if */
META_ELIF, /* \elif */
META_ELSE, /* \else */
- META_ENDIF /* \endif */
+ META_ENDIF, /* \endif */
+ META_STARTPIPELINE, /* \startpipeline */
+ META_ENDPIPELINE /* \endpipeline */
} MetaCommand;
typedef enum QueryMode
@@ -2568,6 +2571,10 @@ getMetaCommand(const char *cmd)
mc = META_GSET;
else if (pg_strcasecmp(cmd, "aset") == 0)
mc = META_ASET;
+ else if (pg_strcasecmp(cmd, "startpipeline") == 0)
+ mc = META_STARTPIPELINE;
+ else if (pg_strcasecmp(cmd, "endpipeline") == 0)
+ mc = META_ENDPIPELINE;
else
mc = META_NONE;
return mc;
@@ -2757,11 +2764,25 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In pipeline mode, we use asynchronous functions. If a
+ * server-side error occurs, it will be processed later
+ * among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ pg_log_error("%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2805,8 +2826,10 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
* varprefix should be set only with \gset or \aset, and SQL commands do
* not need it.
*/
+#if 0
Assert((meta == META_NONE && varprefix == NULL) ||
((meta == META_GSET || meta == META_ASET) && varprefix != NULL));
+#endif
res = PQgetResult(st->con);
@@ -2874,6 +2897,12 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* otherwise the result is simply thrown away by PQclear below */
break;
+ case PGRES_PIPELINE_SYNC:
+ pg_log_debug("client %d pipeline ending", st->id);
+ if (PQexitPipelineMode(st->con) != 1)
+ pg_log_error("client %d failed to exit pipeline mode", st->id);
+ break;
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3127,13 +3156,36 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
/* Execute the command */
if (command->type == SQL_COMMAND)
{
+ /* disallow \aset and \gset in pipeline mode */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ if (command->meta == META_GSET)
+ {
+ commandFailed(st, "gset", "\\gset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ else if (command->meta == META_ASET)
+ {
+ commandFailed(st, "aset", "\\aset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ }
+
if (!sendCommand(st, command))
{
commandFailed(st, "SQL", "SQL command send failed");
st->state = CSTATE_ABORTED;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in pipeline mode */
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -3273,7 +3325,15 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (readCommandResponse(st,
sql_script[st->use_file].commands[st->command]->meta,
sql_script[st->use_file].commands[st->command]->varprefix))
- st->state = CSTATE_END_COMMAND;
+ {
+ /*
+ * outside of pipeline mode: stop reading results.
+ * pipeline mode: continue reading results until an
+ * end-of-pipeline response.
+ */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ st->state = CSTATE_END_COMMAND;
+ }
else
st->state = CSTATE_ABORTED;
break;
@@ -3516,6 +3576,49 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
return CSTATE_ABORTED;
}
}
+ else if (command->meta == META_STARTPIPELINE)
+ {
+ /*
+ * In pipeline mode, we use a workflow based on libpq pipeline
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "startpipeline", "cannot use pipeline mode with the simple query protocol");
+ return CSTATE_ABORTED;
+ }
+
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ commandFailed(st, "startpipeline", "already in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (PQenterPipelineMode(st->con) == 0)
+ {
+ commandFailed(st, "startpipeline", "failed to enter pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ }
+ else if (command->meta == META_ENDPIPELINE)
+ {
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ {
+ commandFailed(st, "endpipeline", "not in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (!PQsendPipeline(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to send the pipeline");
+ return CSTATE_ABORTED;
+ }
+ if (!PQexitPipelineMode(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to exit pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ /* collect pending results before getting out of pipeline mode */
+ return CSTATE_WAIT_RESULT;
+ }
/*
* executing the expression or shell command might have taken a
@@ -4725,7 +4828,9 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
- else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
+ else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF ||
+ my_command->meta == META_STARTPIPELINE ||
+ my_command->meta == META_ENDPIPELINE)
{
if (my_command->argc != 1)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..60d09e6d63 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,7 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterPipelineMode 180
+PQexitPipelineMode 181
+PQsendPipeline 182
+PQpipelineStatus 183
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 4e21057d0f..a714e9fc53 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -522,6 +522,23 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * pqFreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+pqFreeCommandQueue(PGcommandQueueEntry *queue)
+{
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *cur = queue;
+
+ queue = cur->next;
+ if (cur->query)
+ free(cur->query);
+ free(cur);
+ }
+}
/*
* pqDropServerData
@@ -553,6 +570,12 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ pqFreeCommandQueue(conn->cmd_queue_head);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ pqFreeCommandQueue(conn->cmd_queue_recycle);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
@@ -2459,6 +2482,7 @@ keep_going: /* We will come back to here until there is
/* Drop any PGresult we might have, too */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn);
/* Reset conn->status to put the state machine in the right state */
@@ -3917,6 +3941,7 @@ makeEmptyPGconn(void)
conn->status = CONNECTION_BAD;
conn->asyncStatus = PGASYNC_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
conn->xactStatus = PQTRANS_IDLE;
conn->options_valid = false;
conn->nonblocking = false;
@@ -4084,8 +4109,6 @@ freePGconn(PGconn *conn)
if (conn->connip)
free(conn->connip);
/* Note that conn->Pfdebug is not ours to close or free */
- if (conn->last_query)
- free(conn->last_query);
if (conn->write_err_msg)
free(conn->write_err_msg);
if (conn->inBuffer)
@@ -4174,6 +4197,7 @@ closePGconn(PGconn *conn)
conn->status = CONNECTION_BAD; /* Well, not really _bad_ - just absent */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn); /* deallocate result */
resetPQExpBuffer(&conn->errorMessage);
release_conn_addrinfo(conn);
@@ -6726,6 +6750,15 @@ PQbackendPID(const PGconn *conn)
return conn->be_pid;
}
+PGpipelineStatus
+PQpipelineStatus(const PGconn *conn)
+{
+ if (!conn)
+ return PQ_PIPELINE_OFF;
+
+ return conn->pipelineStatus;
+}
+
int
PQconnectionNeedsPassword(const PGconn *conn)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9a038043b2..0e964979bc 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_PIPELINE_SYNC",
+ "PGRES_PIPELINE_ABORTED"
};
/*
@@ -71,6 +73,12 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *pqMakePipelineCmd(PGconn *conn);
+static void pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+void pqCommandQueueAdvance(PGconn *conn);
+static void pqPipelineProcessQueue(PGconn *conn);
+static int pqPipelineFlush(PGconn *conn);
/* ----------------
@@ -1171,7 +1179,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1209,9 +1217,15 @@ PQsendQueryContinue(PGconn *conn, const char *query)
static int
PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
{
+ PGcommandQueueEntry *entry = NULL;
+
if (!PQsendQueryStart(conn, newQuery))
return 0;
+ entry = pqMakePipelineCmd(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
/* check the argument */
if (!query)
{
@@ -1220,37 +1234,75 @@ PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
return 0;
}
- /* construct the outgoing Query message */
- if (pqPutMsgStart('Q', conn) < 0 ||
- pqPuts(query, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
+ /* Send the query message(s) */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
{
- /* error message should be set up already */
- return 0;
+ /* construct the outgoing Query message */
+ if (pqPutMsgStart('Q', conn) < 0 ||
+ pqPuts(query, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ {
+ /* error message should be set up already */
+ return 0;
+ }
+
+ /* remember we are using simple query protocol */
+ entry->queryclass = PGQUERY_SIMPLE;
+ /* and remember the query text too, if possible */
+ entry->query = strdup(query);
}
+ else
+ {
+ /*
+ * In pipeline mode, we cannot use the simple protocol, so we send
+ * Parse, Bind, Describe Portal, Execute.
+ */
+ if (pqPutMsgStart('P', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPuts(query, conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ if (pqPutMsgStart('B', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ if (pqPutMsgStart('D', conn) < 0 ||
+ pqPutc('P', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ if (pqPutMsgStart('E', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPutInt(0, 4, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
- /* remember we are using simple query protocol */
- conn->queryclass = PGQUERY_SIMPLE;
-
- /* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ entry->queryclass = PGQUERY_EXTENDED;
+ entry->query = strdup(query);
+ }
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
- {
- /* error message should be set up already */
- return 0;
- }
+ if (pqPipelineFlush(conn) < 0)
+ goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ pqAppendPipelineCmd(conn, entry);
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
+
+sendFailed:
+ pqRecyclePipelineCmd(conn, entry);
+ /* error message should be set up already */
+ return 0;
}
/*
@@ -1307,6 +1359,8 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *entry = NULL;
+
if (!PQsendQueryStart(conn, true))
return 0;
@@ -1330,6 +1384,10 @@ PQsendPrepare(PGconn *conn,
return 0;
}
+ entry = pqMakePipelineCmd(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
/* construct the Parse message */
if (pqPutMsgStart('P', conn) < 0 ||
pqPuts(stmtName, conn) < 0 ||
@@ -1356,32 +1414,42 @@ PQsendPrepare(PGconn *conn,
if (pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /*
+ * In non-pipeline mode, add a Sync and prepare to send. In pipeline mode
+ * we just keep track of the new message.
+ */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ entry->queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ /* if insufficient memory, query just winds up NULL */
+ entry->query = strdup(query);
+
+ pqAppendPipelineCmd(conn, entry);
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ conn->asyncStatus = PGASYNC_BUSY;
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
- /* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, entry);
/* error message should be set up already */
return 0;
}
@@ -1429,7 +1497,8 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn, bool newQuery)
@@ -1450,20 +1519,57 @@ PQsendQueryStart(PGconn *conn, bool newQuery)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE &&
+ conn->pipelineStatus == PQ_PIPELINE_OFF)
{
appendPQExpBufferStr(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * When enqueuing commands we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when pqPipelineProcessQueue()
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ }
+ }
+ else
+ {
+ /*
+ * This command's results will come in immediately. Initialize async
+ * result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1487,10 +1593,16 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *entry;
+
+ entry = pqMakePipelineCmd(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
/*
- * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
- * using specified statement name and the unnamed portal.
+ * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync
+ * (if not in pipeline mode), using specified statement name and the
+ * unnamed portal.
*/
if (command)
@@ -1600,35 +1712,38 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /* construct the Sync message if not in pipeline mode */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ entry->queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ /* if insufficient memory, query just winds up NULL */
if (command)
- conn->last_query = strdup(command);
- else
- conn->last_query = NULL;
+ entry->query = strdup(command);
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ pqAppendPipelineCmd(conn, entry);
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, entry);
/* error message should be set up already */
return 0;
}
@@ -1647,8 +1762,9 @@ PQsetSingleRowMode(PGconn *conn)
return 0;
if (conn->asyncStatus != PGASYNC_BUSY)
return 0;
- if (conn->queryclass != PGQUERY_SIMPLE &&
- conn->queryclass != PGQUERY_EXTENDED)
+ if (!conn->cmd_queue_head ||
+ (conn->cmd_queue_head->queryclass != PGQUERY_SIMPLE &&
+ conn->cmd_queue_head->queryclass != PGQUERY_EXTENDED))
return 0;
if (conn->result)
return 0;
@@ -1726,14 +1842,17 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
-
/*
* PQgetResult
* Get the next PGresult produced by a query. Returns NULL if no
* query work remains or an error has occurred (e.g. out of
* memory).
+ *
+ * In pipeline mode, once all the result of a query have been returned,
+ * PQgetResult returns NULL to let the user know that the next
+ * query is being processed. At the end of the pipeline, returns a
+ * result with PQresultStatus(result) == PGRES_PIPELINE_SYNC.
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1803,8 +1922,60 @@ PQgetResult(PGconn *conn)
{
case PGASYNC_IDLE:
res = NULL; /* query is complete */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to return the NULL that terminates the round of
+ * results from the current query; prepare to send the results
+ * of the next query when we're called next. Also, since this
+ * is the start of the results of the next query, clear any
+ * prior error message.
+ */
+ resetPQExpBuffer(&conn->errorMessage);
+ pqPipelineProcessQueue(conn);
+ }
break;
case PGASYNC_READY:
+ /*
+ * For any query type other than simple query protocol, we advance
+ * the command queue here. For simple query protocol, we can get
+ * the READY state multiple times before the command is actually
+ * complete, since the string can contain many queries -- so we
+ * wait till we receive ReadyForQuery.
+ */
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass != PGQUERY_SIMPLE)
+ pqCommandQueueAdvance(conn);
+ res = pqPrepareAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to send the results of the current query. Set
+ * us idle now, and ...
+ */
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /*
+ * ... in cases when we're sending a pipeline-sync result,
+ * move queue processing forwards immediately, so that next
+ * time we're called, we're prepared to return the next result
+ * received from the server. In all other cases, leave the
+ * queue state change for next time, so that a terminating NULL
+ * result is sent.
+ *
+ * In other words: we don't return a NULL after a pipeline
+ * sync.
+ */
+ if (res && res->resultStatus == PGRES_PIPELINE_SYNC)
+ pqPipelineProcessQueue(conn);
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -1985,6 +2156,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("synchronous command execution functions are not allowed in pipeline mode\n"));
+ return false;
+ }
+
/*
* Since this is the beginning of a query cycle, reset the error buffer.
*/
@@ -2148,6 +2326,8 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *entry = NULL;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2155,6 +2335,10 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
if (!PQsendQueryStart(conn, true))
return 0;
+ entry = pqMakePipelineCmd(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
/* construct the Describe message */
if (pqPutMsgStart('D', conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2163,32 +2347,32 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
-
- /* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
-
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
{
- free(conn->last_query);
- conn->last_query = NULL;
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
}
+ /* remember we are doing a Describe */
+ entry->queryclass = PGQUERY_DESCRIBE;
+
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ pqAppendPipelineCmd(conn, entry);
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, entry);
/* error message should be set up already */
return 0;
}
@@ -2327,7 +2511,8 @@ PQputCopyEnd(PGconn *conn, const char *errormsg)
* If we sent the COPY command in extended-query mode, we must issue a
* Sync as well.
*/
- if (conn->queryclass != PGQUERY_SIMPLE)
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass != PGQUERY_SIMPLE)
{
if (pqPutMsgStart('S', conn) < 0 ||
pqPutMsgEnd(conn) < 0)
@@ -2541,6 +2726,13 @@ PQfn(PGconn *conn,
*/
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("PQfn not allowed in pipeline mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -2555,6 +2747,365 @@ PQfn(PGconn *conn,
args, nargs);
}
+/* ====== Pipeline mode support ======== */
+
+/*
+ * PQenterPipelineMode
+ * Put an idle connection in pipeline mode.
+ *
+ * Returns 1 on success. On failure, errorMessage is set and 0 is returned.
+ *
+ * Commands submitted after this can be pipelined on the connection;
+ * there's no requirement to wait for one to finish before the next is
+ * dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQsendPipeline. Multiple pipelines
+ * can be sent while in pipeline mode. Pipeline mode can be exited
+ * by calling PQexitPipelineMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ /* succeed with no action if already in pipeline mode */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot enter pipeline mode, connection not idle\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+
+ return 1;
+}
+
+/*
+ * PQexitPipelineMode
+ * End pipeline mode and return to normal command mode.
+ *
+ * Returns 1 in success (pipeline mode successfully ended, or not in pipeline
+ * mode).
+ *
+ * Returns 0 if in pipeline mode and cannot be ended yet. Error message will
+ * be set.
+ */
+int
+PQexitPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ /* there are some uncollected results */
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+
+ case PGASYNC_BUSY:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode while busy\n"));
+ return 0;
+
+ default:
+ /* OK */
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ return 0; /* error message is setup already */
+ return 1;
+}
+
+/*
+ * pqCommandQueueAdvance
+ * Remove one query from the command queue, when we receive
+ * all results from the server that pertain to it.
+ */
+void
+pqCommandQueueAdvance(PGconn *conn)
+{
+ PGcommandQueueEntry *prevquery;
+
+ if (conn->cmd_queue_head == NULL)
+ return;
+
+ /* delink from queue */
+ prevquery = conn->cmd_queue_head;
+ conn->cmd_queue_head = conn->cmd_queue_head->next;
+
+ /* and make it recyclable */
+ prevquery->next = NULL;
+ pqRecyclePipelineCmd(conn, prevquery);
+}
+
+/*
+ * pqPipelineProcessQueue: subroutine for PQgetResult
+ * In pipeline mode, start processing the results of the next query in the queue.
+ */
+void
+pqPipelineProcessQueue(PGconn *conn)
+{
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return;
+ case PGASYNC_IDLE:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * XXX rewrite this comment. In pipeline mode but nothing left on the
+ * queue; caller can submit more work or PQexitPipelineMode() now.
+ */
+ return;
+ }
+
+ /* In non-pipeline mode, we're done here */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ return;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /*
+ * Reset single-row processing mode. (Client has to set it up for each
+ * query, if desired.)
+ */
+ conn->singleRowMode = false;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_ABORTED &&
+ conn->cmd_queue_head->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted pipeline we don't get anything from the server for
+ * each result; we're just discarding commands from the queue until we
+ * get to the next sync from the server.
+ *
+ * The PGRES_PIPELINE_ABORTED results tell the client that its queries
+ * got aborted.
+ */
+ conn->result = PQmakeEmptyPGresult(conn, PGRES_PIPELINE_ABORTED);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ pqSaveErrorResult(conn);
+ return;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+}
+
+/*
+ * PQsendPipeline
+ * Send a Sync message as part of a pipeline, and flush to server
+ *
+ * It's legal to start submitting more commands in the pipeline immediately,
+ * without waiting for the results of the current pipeline. There's no need to
+ * end pipeline mode and start it again.
+ *
+ * If a command in a pipeline fails, every subsequent command up to and including
+ * the result to the Sync message sent by PQsendPipeline gets set to
+ * PGRES_PIPELINE_ABORTED state. If the whole pipeline is processed without
+ * error, a PGresult with PGRES_PIPELINE_SYNC is produced.
+ *
+ * Queries can already have been sent before PQsendPipeline is called, but
+ * PQsendPipeline need to be called before retrieving command results.
+ *
+ * The connection will remain in pipeline mode and unavailable for new
+ * synchronous command execution functions until all results from the pipeline
+ * are processed by the client.
+ */
+int
+PQsendPipeline(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot send pipeline when not in pipeline mode\n"));
+ return 0;
+ }
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: cannot send pipeline while in COPY\n");
+ return 0;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_IDLE:
+ /* OK to send sync */
+ break;
+ }
+
+ entry = pqMakePipelineCmd(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ pqAppendPipelineCmd(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ /*
+ * Call pqPipelineProcessQueue so the user can call start calling
+ * PQgetResult.
+ */
+ pqPipelineProcessQueue(conn);
+
+ return 1;
+
+sendFailed:
+ pqRecyclePipelineCmd(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
+
+/*
+ * pqMakePipelineCmd
+ * Get a command queue entry for caller to fill.
+ *
+ * If the recycle queue has a free element, that is returned; if not, a
+ * fresh one is allocated. Caller is responsible for adding it to the
+ * command queue (pqAppendPipelineCmd) once the struct is filled in, or
+ * releasing the memory (pqRecyclePipelineCmd) if an error occurs.
+ *
+ * If allocation fails, sets the error message and returns NULL.
+ */
+static PGcommandQueueEntry *
+pqMakePipelineCmd(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * pqAppendPipelineCmd
+ * Append a caller-allocated command queue entry to the queue.
+ *
+ * The query itself must already have been put in the output buffer by the
+ * caller.
+ */
+static void
+pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ Assert(entry->next == NULL);
+
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * pqRecyclePipelineCmd
+ * Push a command queue entry onto the freelist.
+ */
+static void
+pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+
+ /* recyclable entries should not have a follow-on command */
+ Assert(entry->next == NULL);
+
+ if (entry->query)
+ {
+ free(entry->query);
+ entry->query = NULL;
+ }
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
/* ====== accessor funcs for PGresult ======== */
@@ -2569,7 +3120,7 @@ PQresultStatus(const PGresult *res)
char *
PQresStatus(ExecStatusType status)
{
- if ((unsigned int) status >= sizeof pgresStatus / sizeof pgresStatus[0])
+ if ((unsigned int) status >= lengthof(pgresStatus))
return libpq_gettext("invalid ExecStatusType code");
return pgresStatus[status];
}
@@ -3152,6 +3703,23 @@ PQflush(PGconn *conn)
return pqFlush(conn);
}
+/*
+ * pqPipelineFlush
+ *
+ * In pipeline mode, data will be flushed only when the out buffer reaches the
+ * threshold value. In non-pipeline mode, it behaves as stock pqFlush.
+ *
+ * Returns 0 on success.
+ */
+static int
+pqPipelineFlush(PGconn *conn)
+{
+ if ((conn->pipelineStatus == PQ_PIPELINE_OFF) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return pqFlush(conn);
+ return 0;
+}
+
/*
* PQfreemem - safely frees memory allocated
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 2ca8c057b9..38dc1e4efa 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -177,14 +177,24 @@ pqParseInput3(PGconn *conn)
if (getParameterStatus(conn))
return;
}
- else
- {
- pqInternalNotice(&conn->noticeHooks,
- "message type 0x%02x arrived from server while idle",
- id);
- /* Discard the unexpected message */
- conn->inCursor += msgLength;
- }
+
+ /*
+ * We're notionally not-IDLE when in pipeline mode we have
+ * completed processing the results of one query and are waiting
+ * for the next one in the pipeline. In this case, as above, just
+ * wait.
+ */
+ if (conn->asyncStatus == PGASYNC_IDLE &&
+ conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->cmd_queue_head != NULL)
+ return;
+
+ /* Any other case is unexpected and we summarily skip it */
+ pqInternalNotice(&conn->noticeHooks,
+ "message type 0x%02x arrived from server while idle",
+ id);
+ /* Discard the unexpected message */
+ conn->inCursor += msgLength;
}
else
{
@@ -217,10 +227,37 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new
+ * query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_PIPELINE_SYNC);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ else
+ {
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ }
+ else
+ {
+ /*
+ * In simple query protocol, advance the command queue
+ * (see PQgetResult).
+ */
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_SIMPLE)
+ pqCommandQueueAdvance(conn);
+ conn->asyncStatus = PGASYNC_IDLE;
+ }
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -238,7 +275,8 @@ pqParseInput3(PGconn *conn)
break;
case '1': /* Parse Complete */
/* If we're doing PQprepare, we're done; else ignore */
- if (conn->queryclass == PGQUERY_PREPARE)
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_PREPARE)
{
if (conn->result == NULL)
{
@@ -285,7 +323,8 @@ pqParseInput3(PGconn *conn)
conn->inCursor += msgLength;
}
else if (conn->result == NULL ||
- conn->queryclass == PGQUERY_DESCRIBE)
+ (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_DESCRIBE))
{
/* First 'T' in a query sequence */
if (getRowDescriptions(conn, msgLength))
@@ -318,7 +357,8 @@ pqParseInput3(PGconn *conn)
* instead of PGRES_TUPLES_OK. Otherwise we can just
* ignore this message.
*/
- if (conn->queryclass == PGQUERY_DESCRIBE)
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_DESCRIBE)
{
if (conn->result == NULL)
{
@@ -450,7 +490,7 @@ handleSyncLoss(PGconn *conn, char id, int msgLength)
id, msgLength);
/* build an error result holding the error message */
pqSaveErrorResult(conn);
- conn->asyncStatus = PGASYNC_READY; /* drop out of GetResult wait loop */
+ conn->asyncStatus = PGASYNC_READY; /* drop out of PQgetResult wait loop */
/* flush input data since we're giving up on processing it */
pqDropConnection(conn, true);
conn->status = CONNECTION_BAD; /* No more connection to backend */
@@ -477,7 +517,9 @@ getRowDescriptions(PGconn *conn, int msgLength)
* PGresult created by getParamDescriptions, and we should fill data into
* that. Otherwise, create a new, empty PGresult.
*/
- if (conn->queryclass == PGQUERY_DESCRIBE)
+ if (!conn->cmd_queue_head ||
+ (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_DESCRIBE))
{
if (conn->result)
result = conn->result;
@@ -584,7 +626,9 @@ getRowDescriptions(PGconn *conn, int msgLength)
* If we're doing a Describe, we're done, and ready to pass the result
* back to the client.
*/
- if (conn->queryclass == PGQUERY_DESCRIBE)
+ if ((!conn->cmd_queue_head) ||
+ (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_DESCRIBE))
{
conn->asyncStatus = PGASYNC_READY;
return 0;
@@ -875,6 +919,10 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ /* If in pipeline mode, set error indicator for it */
+ if (isError && conn->pipelineStatus != PQ_PIPELINE_OFF)
+ conn->pipelineStatus = PQ_PIPELINE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
@@ -931,8 +979,8 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
* might need it for an error cursor display, which is only true if there
* is a PG_DIAG_STATEMENT_POSITION field.
*/
- if (have_position && conn->last_query && res)
- res->errQuery = pqResultStrdup(res, conn->last_query);
+ if (have_position && res && conn->cmd_queue_head && conn->cmd_queue_head->query)
+ res->errQuery = pqResultStrdup(res, conn->cmd_queue_head->query);
/*
* Now build the "overall" error message for PQresultErrorMessage.
@@ -1851,7 +1899,8 @@ pqEndcopy3(PGconn *conn)
* If we sent the COPY command in extended-query mode, we must issue a
* Sync as well.
*/
- if (conn->queryclass != PGQUERY_SIMPLE)
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass != PGQUERY_SIMPLE)
{
if (pqPutMsgStart('S', conn) < 0 ||
pqPutMsgEnd(conn) < 0)
@@ -1931,6 +1980,9 @@ pqFunctionCall3(PGconn *conn, Oid fnid,
int avail;
int i;
+ /* already validated by PQfn */
+ Assert(conn->pipelineStatus == PQ_PIPELINE_OFF);
+
/* PQfn already validated connection state */
if (pqPutMsgStart('F', conn) < 0 || /* function call msg */
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index fa9b62a844..fb31a49fab 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -96,7 +96,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_PIPELINE_SYNC, /* pipeline synchronization point */
+ PGRES_PIPELINE_ABORTED, /* Command didn't run because of an abort
+ * earlier in a pipeline */
} ExecStatusType;
typedef enum
@@ -136,6 +139,16 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PGpipelineStatus - Current status of pipeline mode
+ */
+typedef enum
+{
+ PQ_PIPELINE_OFF,
+ PQ_PIPELINE_ON,
+ PQ_PIPELINE_ABORTED
+} PGpipelineStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -327,6 +340,7 @@ extern int PQserverVersion(const PGconn *conn);
extern char *PQerrorMessage(const PGconn *conn);
extern int PQsocket(const PGconn *conn);
extern int PQbackendPID(const PGconn *conn);
+extern PGpipelineStatus PQpipelineStatus(const PGconn *conn);
extern int PQconnectionNeedsPassword(const PGconn *conn);
extern int PQconnectionUsedPassword(const PGconn *conn);
extern int PQclientEncoding(const PGconn *conn);
@@ -434,6 +448,11 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for pipeline mode management */
+extern int PQenterPipelineMode(PGconn *conn);
+extern int PQexitPipelineMode(PGconn *conn);
+extern int PQsendPipeline(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 2f052f61f8..9b02e54f0d 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,7 +217,11 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, more results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
@@ -229,7 +233,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* Sync at end of a pipeline */
} PGQueryClass;
/* Target server type (decoded value of target_session_attrs) */
@@ -305,6 +310,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by pipeline mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct PGcommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type */
+ char *query; /* SQL command, or NULL if none/unknown/OOM */
+ struct PGcommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -389,12 +410,11 @@ struct pg_conn
ConnStatusType status;
PGAsyncStatusType asyncStatus;
PGTransactionStatusType xactStatus; /* never changes to ACTIVE */
- PGQueryClass queryclass;
- char *last_query; /* last SQL command, or NULL if unknown */
char last_sqlstate[6]; /* last reported SQLSTATE */
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PGpipelineStatus pipelineStatus; /* status of pipeline mode */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -407,6 +427,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue, for pipeline mode.
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -795,6 +825,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 5391f461a2..93e7829c67 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -10,6 +10,7 @@ SUBDIRS = \
delay_execution \
dummy_index_am \
dummy_seclabel \
+ libpq_pipeline \
plsample \
snapshot_too_old \
test_bloomfilter \
diff --git a/src/test/modules/libpq_pipeline/.gitignore b/src/test/modules/libpq_pipeline/.gitignore
new file mode 100644
index 0000000000..3a11e786b8
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/libpq_pipeline
diff --git a/src/test/modules/libpq_pipeline/Makefile b/src/test/modules/libpq_pipeline/Makefile
new file mode 100644
index 0000000000..b798f5fbbc
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/Makefile
@@ -0,0 +1,20 @@
+# src/test/modules/libpq_pipeline/Makefile
+
+PROGRAM = libpq_pipeline
+OBJS = libpq_pipeline.o
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS_INTERNAL += $(libpq_pgport)
+
+TAP_TESTS = 1
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/libpq_pipeline
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/libpq_pipeline/README b/src/test/modules/libpq_pipeline/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/libpq_pipeline/libpq_pipeline.c b/src/test/modules/libpq_pipeline/libpq_pipeline.c
new file mode 100644
index 0000000000..8819df4d92
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/libpq_pipeline.c
@@ -0,0 +1,1160 @@
+/*
+ * src/test/modules/libpq_pipeline/libpq_pipeline.c
+ * Verify libpq pipeline execution functionality
+ */
+#include "postgres_fe.h"
+
+#include <sys/time.h>
+#ifdef HAVE_SYS_SELECT_H
+#include <sys/select.h>
+#endif
+
+#include "catalog/pg_type_d.h"
+#include "common/fe_memutils.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+
+static void exit_nicely(PGconn *conn);
+
+const char *const progname = "libpq_pipeline";
+
+
+#define DEBUG
+#ifdef DEBUG
+#define pg_debug(...) do { fprintf(stderr, __VA_ARGS__); } while (0)
+#else
+#define pg_debug(...)
+#endif
+
+static const char *const drop_table_sql =
+"DROP TABLE IF EXISTS pq_pipeline_demo";
+static const char *const create_table_sql =
+"CREATE UNLOGGED TABLE pq_pipeline_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql =
+"INSERT INTO pq_pipeline_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+/*
+ * Print an error to stderr and terminate the program.
+ */
+#define pg_fatal(...) pg_fatal_impl(__LINE__, __VA_ARGS__)
+static void
+pg_fatal_impl(int line, const char *fmt,...)
+{
+ va_list args;
+
+
+ fflush(stdout);
+
+ fprintf(stderr, "\n%s:%d: ", progname, line);
+ va_start(args, fmt);
+ vfprintf(stderr, fmt, args);
+ va_end(args);
+ Assert(fmt[strlen(fmt) - 1] != '\n');
+ fprintf(stderr, "\n");
+ exit(1);
+}
+
+static void
+test_disallowed(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("Unable to enter pipeline mode");
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not activated properly");
+
+ /* PQexec should fail in pipeline mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("PQexec should fail in pipeline mode but succeeded");
+
+ /* Entering pipeline mode when already in pipeline mode is OK */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("re-entering pipeline mode should be a no-op but failed");
+
+ if (PQisBusy(conn) != 0)
+ pg_fatal("PQisBusy should return 0 when idle in pipeline mode, returned 1");
+
+ /* ok, back to normal command mode */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("couldn't exit idle empty pipeline mode");
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not terminated properly");
+
+ /* exiting pipeline mode when not in pipeline mode should be a no-op */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("pipeline mode exit when not in pipeline mode should succeed but failed");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("PQexec should succeed after exiting pipeline mode but failed with: %s",
+ PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_simple_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple pipeline... ");
+
+ /*
+ * Enter pipeline mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our output buffer will give us enough slush
+ * to work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching SELECT failed: %s", PQerrorMessage(conn));
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode with work in progress should fail, but succeeded");
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending pipeline failed: %s", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item",
+ PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first query result.");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit pipeline mode yet
+ */
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result PGRES_PIPELINE_SYNC expected: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of PGRES_PIPELINE_SYNC, error: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after pipeline end: %s",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /* ... until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Exiting pipeline mode didn't seem to work");
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_multi_pipelines(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi pipeline... ");
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first SELECT failed: %s", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending first pipeline failed: %s", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second SELECT failed: %s", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s", PQerrorMessage(conn));
+
+ /* OK, start processing the results */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first result");
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result expected: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of sync result, error: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ PQclear(res);
+
+#if 0
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s",
+ PQresStatus(PQresultStatus(res)));
+#endif
+
+ /* second pipeline */
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from second pipeline item",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode ... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work");
+
+ fprintf(stderr, "ok\n");
+}
+
+/*
+ * When an operation in a pipeline fails the rest of the pipeline is flushed. We
+ * still have to get results for each pipeline item, but the item will just be
+ * a PGRES_PIPELINE_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the pipeline. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_aborted_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+ bool goterror;
+
+ fprintf(stderr, "aborted pipeline... ");
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching DROP TABLE failed: %s", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching CREATE TABLE failed: %s", PQerrorMessage(conn));
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * pipeline ERRORs.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first insert failed: %s", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT no_such_function($1)",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching error select failed: %s", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second insert failed: %s", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending first pipeline failed: %s", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second-pipeline insert failed: %s",
+ PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the pipeline results.
+ *
+ * We should get a command-ok for the first query, then a fatal error and
+ * a pipeline aborted message for the second insert, a pipeline-end, then
+ * a command-ok and a pipeline-ok for the second pipeline operation.
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result status %s: %s",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s",
+ PQresStatus(PQresultStatus(res)));
+
+ /* Second query caused error, so we expect an error next */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("Unexpected result code -- expected PGRES_FATAL_ERROR, got %s",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s",
+ PQresStatus(PQresultStatus(res)));
+
+ /*
+ * pipeline should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third pipeline since we've already sent a
+ * second. The aborted flag relates only to the pipeline being received.
+ */
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't");
+
+ /* third query in pipeline, the second insert */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_ABORTED)
+ pg_fatal("Unexpected result code -- expected PGRES_PIPELINE_ABORTED, got %s",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s", PQresStatus(PQresultStatus(res)));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't");
+
+ /* Ensure we're still in pipeline */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /*
+ * The end of a failed pipeline is a PGRES_PIPELINE_SYNC.
+ *
+ * (This is so clients know to start processing results normally again and
+ * can tell the difference between skipped commands and the sync.)
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code from first pipeline sync\n"
+ "Expected PGRES_PIPELINE_SYNC, got %s",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+#if 0
+ /* after the synchronization point we get a NULL result */
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s", PQresStatus(PQresultStatus(res)));
+#endif
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_ABORTED)
+ pg_fatal("sync should've cleared the aborted flag but didn't");
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /* the insert from the second pipeline */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result code %s from first item in second pipeline",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* Read the NULL result at the end of the command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s", PQresStatus(PQresultStatus(res)));
+
+ /* the second pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s: %s",
+ PQresStatus(PQresultStatus(res)),
+ PQerrorMessage(conn));
+
+ /* Test single-row mode with an error partways */
+ if (PQsendQuery(conn, "SELECT 1.0/g FROM generate_series(3, -1, -1) g") != 1)
+ pg_fatal("failed to send query: %s", PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ PQsetSingleRowMode(conn);
+ goterror = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_SINGLE_TUPLE:
+ printf("got row: %s\n", PQgetvalue(res, 0, 0));
+ break;
+ case PGRES_FATAL_ERROR:
+ if (strcmp(PQresultErrorField(res, PG_DIAG_SQLSTATE), "22012") != 0)
+ pg_fatal("expected division-by-zero, got: %s",
+ PQresultErrorField(res, PG_DIAG_SQLSTATE));
+ printf("got expected division-by-zero\n");
+ goterror = true;
+ break;
+ default:
+ pg_fatal("got unexpected result %s", PQresStatus(PQresultStatus(res)));
+ }
+ PQclear(res);
+ }
+ if (!goterror)
+ pg_fatal("did not get division-by-zero error");
+ /* the third pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from third pipeline sync",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work");
+
+ fprintf(stderr, "ok\n");
+
+ /*-
+ * Since we fired the pipelines off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st pipeline
+ * - First insert applied
+ * - Second statement aborted xact
+ * - Third insert skipped
+ * - Sync rolled back first implicit xact
+ * - Implicit xact created by server around 2nd pipeline
+ * - insert applied from 2nd pipeline
+ * - Sync commits 2nd xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM pq_pipeline_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("expected 1 result, got %d", PQntuples(res));
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ pg_fatal("expected only insert with value 3, got %s", val);
+ }
+
+ PQclear(res);
+}
+
+/* State machine enum for test_pipelined_insert */
+typedef enum PipelineInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} PipelineInsertStep;
+
+static void
+test_pipelined_insert(PGconn *conn, int n_rows)
+{
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ PipelineInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a pipelined insert into a table created at the start of the pipeline
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s", PQerrorMessage(conn));
+
+ while (send_step != BI_PREPARE)
+ {
+ const char *sql;
+
+ switch (send_step)
+ {
+ case BI_BEGIN_TX:
+ sql = "BEGIN TRANSACTION";
+ send_step = BI_DROP_TABLE;
+ break;
+
+ case BI_DROP_TABLE:
+ sql = drop_table_sql;
+ send_step = BI_CREATE_TABLE;
+ break;
+
+ case BI_CREATE_TABLE:
+ sql = create_table_sql;
+ send_step = BI_PREPARE;
+ break;
+
+ default:
+ pg_fatal("invalid state");
+ }
+
+ pg_debug("sending: %s\n", sql);
+ if (PQsendQueryParams(conn, sql,
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching %s failed: %s", sql, PQerrorMessage(conn));
+ }
+
+ Assert(send_step == BI_PREPARE);
+ pg_debug("sending: %s\n", insert_sql);
+ if (PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids) != 1)
+ pg_fatal("dispatching PREPARE failed: %s", PQerrorMessage(conn));
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our output buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the pipeline before processing
+ * results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ pg_fatal("failed to set nonblocking mode: %s", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's output buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ PGresult *res;
+ const char *cmdtag;
+ const char *description = "";
+ int status;
+
+ /*
+ * Read next result. If no more results from this query,
+ * advance to the next query
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ continue;
+
+ status = PGRES_COMMAND_OK;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ recv_step++;
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ recv_step++;
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ recv_step++;
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ recv_step++;
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive == 0)
+ recv_step++;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ recv_step++;
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_PIPELINE_SYNC;
+ recv_step++;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ description = "";
+ abort();
+ }
+
+ if (PQresultStatus(res) != status)
+ pg_fatal("%s reported status %s, expected %s\n"
+ "Error message: \"%s\"",
+ description, PQresStatus(PQresultStatus(res)),
+ PQresStatus(status), PQerrorMessage(conn));
+
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ pg_fatal("%s expected command tag '%s', got '%s'",
+ description, cmdtag, PQcmdStatus(res));
+
+ pg_debug("Got %s OK\n", cmdtag[0] != '\0' ? cmdtag : description);
+
+ PQclear(res);
+ }
+ }
+
+ /* Write more rows and/or the end pipeline message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQsendPipeline(conn) == 1)
+ {
+ fprintf(stdout, "Sent pipeline\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending pipeline failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+ }
+
+ /* We've got the sync message and the pipeline should be done */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s",
+ PQerrorMessage(conn));
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ pg_fatal("failed to clear nonblocking mode: %s", PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i;
+ bool pipeline_ended = false;
+
+ /* 1 pipeline, 3 queries in it */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s",
+ PQerrorMessage(conn));
+
+ for (i = 0; i < 3; i++)
+ {
+ char *param[1];
+
+ param[0] = psprintf("%d", 44 + i);
+
+ if (PQsendQueryParams(conn,
+ "SELECT generate_series(42, $1)",
+ 1,
+ NULL,
+ (const char **) param,
+ NULL,
+ NULL,
+ 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ pfree(param[0]);
+ }
+ PQsendPipeline(conn);
+
+ for (i = 0; !pipeline_ended; i++)
+ {
+ bool first = true;
+ bool saw_ending_tuplesok;
+ bool isSingleTuple = false;
+
+ /* Set single row mode for only first 2 SELECT queries */
+ if (i < 2)
+ {
+ if (PQsetSingleRowMode(conn) != 1)
+ pg_fatal("PQsetSingleRowMode() failed for i=%d", i);
+ }
+
+ /* Consume rows for this query */
+ saw_ending_tuplesok = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+
+ if (est == PGRES_PIPELINE_SYNC)
+ {
+ fprintf(stderr, "end of pipeline reached\n");
+ pipeline_ended = true;
+ PQclear(res);
+ if (i != 3)
+ pg_fatal("Expected three results, got %d", i);
+ break;
+ }
+
+ /* Expect SINGLE_TUPLE for queries 0 and 1, TUPLES_OK for 2 */
+ if (first)
+ {
+ if (i <= 1 && est != PGRES_SINGLE_TUPLE)
+ pg_fatal("Expected PGRES_SINGLE_TUPLE for query %d, got %s",
+ i, PQresStatus(est));
+ if (i >= 2 && est != PGRES_TUPLES_OK)
+ pg_fatal("Expected PGRES_TUPLES_OK for query %d, got %s",
+ i, PQresStatus(est));
+ first = false;
+ }
+
+ fprintf(stderr, "Result status %s for query %d", PQresStatus(est), i);
+ switch (est)
+ {
+ case PGRES_TUPLES_OK:
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ saw_ending_tuplesok = true;
+ if (isSingleTuple)
+ {
+ if (PQntuples(res) == 0)
+ fprintf(stderr, "all tuples received in query %d\n", i);
+ else
+ pg_fatal("Expected to follow PGRES_SINGLE_TUPLE, but received PGRES_TUPLES_OK directly instead");
+ }
+ break;
+
+ case PGRES_SINGLE_TUPLE:
+ isSingleTuple = true;
+ fprintf(stderr, ", %d tuple: %s\n", PQntuples(res), PQgetvalue(res, 0, 0));
+ break;
+
+ default:
+ pg_fatal("unexpected");
+ }
+ PQclear(res);
+ }
+ if (!pipeline_ended && !saw_ending_tuplesok)
+ pg_fatal("didn't get expected terminating TUPLES_OK");
+ }
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s", PQerrorMessage(conn));
+}
+
+/*
+ * Simple test to verify that a pipeline is discarded as a whole when there's
+ * an error, ignoring transaction commands.
+ */
+static void
+test_transaction(PGconn *conn)
+{
+ PGresult *res;
+ bool expect_null;
+ int num_sends = 0;
+
+ res = PQexec(conn, "DROP TABLE IF EXISTS pq_pipeline_tst;"
+ "CREATE TABLE pq_pipeline_tst (id int)");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to create test table: %s",
+ PQerrorMessage(conn));
+ PQclear(res);
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s",
+ PQerrorMessage(conn));
+ if (PQsendPrepare(conn, "rollback", "ROLLBACK", 0, NULL) != 1)
+ pg_fatal("could not send prepare on pipeline: %s",
+ PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn,
+ "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ if (PQsendQueryParams(conn,
+ "SELECT 0/0",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+
+ /*
+ * send a ROLLBACK using a prepared stmt. Doesn't work because we need to
+ * get out of the pipeline-aborted state first.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /* This insert fails because we're in pipeline-aborted state */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (1)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * This insert fails even though the pipeline got a SYNC, because we're in
+ * an aborted transaction
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (2)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * Send ROLLBACK using prepared stmt. This one works because we just did
+ * PQsendPipeline above.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /*
+ * Now that we're out of a transaction and in pipeline-good mode, this
+ * insert works
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (3)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+ PQsendPipeline(conn);
+ num_sends++;
+
+ expect_null = false;
+ for (int i = 0;; i++)
+ {
+ ExecStatusType restype;
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ printf("%d: got NULL result\n", i);
+ if (!expect_null)
+ pg_fatal("did not expect NULL here");
+ expect_null = false;
+ continue;
+ }
+ restype = PQresultStatus(res);
+ printf("%d: got status %s", i, PQresStatus(restype));
+ if (expect_null)
+ pg_fatal("expected NULL");
+ if (restype == PGRES_FATAL_ERROR)
+ printf("; error: %s", PQerrorMessage(conn));
+ else if (restype == PGRES_PIPELINE_ABORTED)
+ {
+ printf(": command didn't run because pipeline aborted\n");
+ }
+ else
+ printf("\n");
+ PQclear(res);
+
+ if (restype == PGRES_PIPELINE_SYNC)
+ num_sends--;
+ else
+ expect_null = true;
+ if (num_sends <= 0)
+ break;
+ }
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("returned something extra after all the syncs: %s",
+ PQresStatus(PQresultStatus(res)));
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s", PQerrorMessage(conn));
+
+ /* We expect to find one tuple containing the value "3" */
+ res = PQexec(conn, "SELECT * FROM pq_pipeline_tst");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("failed to obtain result: %s", PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("did not get 1 tuple");
+ if (strcmp(PQgetvalue(res, 0, 0), "3") != 0)
+ pg_fatal("did not get expected tuple");
+ PQclear(res);
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+usage(const char *progname)
+{
+ fprintf(stderr, "%s tests libpq's pipeline mode.\n\n", progname);
+ fprintf(stderr, "Usage:\n");
+ fprintf(stderr, " %s testname [conninfo [number_of_rows]]\n", progname);
+ fprintf(stderr, "Tests:\n");
+ fprintf(stderr, " disallowed_in_pipeline\n");
+ fprintf(stderr, " simple_pipeline\n");
+ fprintf(stderr, " multi_pipeline\n");
+ fprintf(stderr, " pipeline_abort\n");
+ fprintf(stderr, " singlerow\n");
+ fprintf(stderr, " pipeline_insert\n");
+ fprintf(stderr, " transaction\n");
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo = "";
+ PGconn *conn;
+ int numrows = 10000;
+ PGresult *res;
+
+ /*
+ * The testname parameter is mandatory; it can be followed by a conninfo
+ * string and number of rows.
+ */
+ if (argc < 2 || argc > 4)
+ {
+ usage(argv[0]);
+ exit(1);
+ }
+
+ if (argc >= 3)
+ conninfo = pg_strdup(argv[2]);
+
+ if (argc >= 4)
+ {
+ errno = 0;
+ numrows = strtol(argv[3], NULL, 10);
+ if (errno != 0 || numrows <= 0)
+ {
+ fprintf(stderr, "couldn't parse \"%s\" as a positive integer\n", argv[3]);
+ exit(1);
+ }
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+ res = PQexec(conn, "SET lc_messages TO \"C\"");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to set lc_messages: %s", PQerrorMessage(conn));
+
+ if (strcmp(argv[1], "disallowed_in_pipeline") == 0)
+ test_disallowed(conn);
+ else if (strcmp(argv[1], "simple_pipeline") == 0)
+ test_simple_pipeline(conn);
+ else if (strcmp(argv[1], "multi_pipeline") == 0)
+ test_multi_pipelines(conn);
+ else if (strcmp(argv[1], "pipeline_abort") == 0)
+ test_aborted_pipeline(conn);
+ else if (strcmp(argv[1], "pipeline_insert") == 0)
+ test_pipelined_insert(conn, numrows);
+ else if (strcmp(argv[1], "singlerow") == 0)
+ test_singlerowmode(conn);
+ else if (strcmp(argv[1], "transaction") == 0)
+ test_transaction(conn);
+ else
+ {
+ fprintf(stderr, "\"%s\" is not a recognized test name\n", argv[1]);
+ usage(argv[0]);
+ exit(1);
+ }
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+ return 0;
+}
diff --git a/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
new file mode 100644
index 0000000000..a12f2dd47b
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
@@ -0,0 +1,31 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 7;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_pipeline
+ simple_pipeline
+ multi_pipeline
+ pipeline_abort
+ pipeline_insert
+ singlerow
+ transaction);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'libpq_pipeline', $testname, $node->connstr('postgres'), $numrows ],
+ "libp_pipeline $testname");
+}
+
+$node->stop('fast');
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 49614106dc..44b1a43f30 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -33,10 +33,11 @@ my @unlink_on_exit;
# Set of variables for modules in contrib/ and src/test/modules/
my $contrib_defines = { 'refint' => 'REFINT_VERBOSE' };
-my @contrib_uselibpq = ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo');
-my @contrib_uselibpgport = ('oid2name', 'vacuumlo');
-my @contrib_uselibpgcommon = ('oid2name', 'vacuumlo');
-my $contrib_extralibs = undef;
+my @contrib_uselibpq =
+ ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo', 'libpq_pipeline');
+my @contrib_uselibpgport = ('libpq_pipeline', 'oid2name', 'vacuumlo');
+my @contrib_uselibpgcommon = ('libpq_pipeline', 'oid2name', 'vacuumlo');
+my $contrib_extralibs = { 'libpq_pipeline' => ['ws2_32.lib'] };
my $contrib_extraincludes = { 'dblink' => ['src/backend'] };
my $contrib_extrasource = {
'cube' => [ 'contrib/cube/cubescan.l', 'contrib/cube/cubeparse.y' ],
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 574a8a94fa..ca68e88a72 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1562,10 +1562,12 @@ PG_Locale_Strategy
PG_Lock_Status
PG_init_t
PGcancel
+PGcommandQueueEntry
PGconn
PGdataValue
PGlobjfuncs
PGnotify
+PGpipelineStatus
PGresAttDesc
PGresAttValue
PGresParamDesc
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
[ v35-libpq-pipeline.patch ]
I think the changes in pqParseInput3() are broken. You should have
kept the else-structure as-is and inserted the check for "not really
idle" inside the else-clause that reports an error. As it stands,
after successfully processing an asynchronously-received error or
ParameterStatus message, the added code will cause us to return without
advancing inStart, creating an infinite loop of reprocessing that message.
It's possible that we should redefine the way things happen so that if
we're waiting for another pipeline event, we should hold off processing
of async error & ParameterStatus; but in that case you should have added
the pre-emptive return ahead of that if/else structure, where the existing
"If not IDLE state, just wait ..." test is. My guess though is that we
do need to process error messages in that state, so that the correct
patch looks more like
else
{
+ /*
+ * We're notionally not-IDLE when in pipeline mode we have
+ * completed processing the results of one query and are waiting
+ * for the next one in the pipeline. In this case, as above, just
+ * wait.
+ */
+ if (conn->asyncStatus == PGASYNC_IDLE &&
+ conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->cmd_queue_head != NULL)
+ return;
+
pqInternalNotice(&conn->noticeHooks,
"message type 0x%02x arrived from server while idle",
id);
/* Discard the unexpected message */
conn->inCursor += msgLength;
}It'd be appropriate to do more than nothing to the comment block above
this if/else chain, too, because really that one ought to explain why we
should consume ERROR when in pipeline state.
(I've not looked at the rest of this patch, just scanned what you did
in fe-protocol3.c, because I wondered if there would be any interaction
with the where-to-advance-inStart changes I'm about to commit. Looks
okay modulo this issue.)
regards, tom lane
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
On 2021-Mar-11, Tom Lane wrote:
I think the changes in pqParseInput3() are broken. You should have
kept the else-structure as-is and inserted the check for "not really
idle" inside the else-clause that reports an error. As it stands,
after successfully processing an asynchronously-received error or
ParameterStatus message, the added code will cause us to return without
advancing inStart, creating an infinite loop of reprocessing that message.It's possible that we should redefine the way things happen so that if
we're waiting for another pipeline event, we should hold off processing
of async error & ParameterStatus; but in that case you should have added
the pre-emptive return ahead of that if/else structure, where the existing
"If not IDLE state, just wait ..." test is.
I think I agree that holding off 'E' and 'S' messages when in between
processing results for different queries in a pipeline, so keeping the
original if/else structure is correct. An error would be correctly
dealt with in the BUSY state immediately afterwards; and the fact that
we pass 'inError' false at that point causes the wrong reaction (namely
that the pipeline is not put in aborted state).
I made a number of other changes: documentation adjustments per comments
from David Johnston, some function renaming as previously noted, and
added test code for PQsendDescribePrepared, PQsendDescribePortal. Also
rebased on latest changes. I also absorbed one change that I already
had when I submitted v35, but hadn't done "git add" on (which caused a
compile failure for CF bot).
--
�lvaro Herrera Valdivia, Chile
Attachments:
v36-libpq-pipeline.patchtext/x-diff; charset=us-asciiDownload
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 910e9a81ea..7b938c106c 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -3180,6 +3180,33 @@ ExecStatusType PQresultStatus(const PGresult *res);
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-sync">
+ <term><literal>PGRES_PIPELINE_SYNC</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a
+ synchronization point in pipeline mode, requested by
+ <xref linkend="libpq-PQsendPipeline"/>.
+ This status occurs only when pipeline mode has been selected.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-aborted">
+ <term><literal>PGRES_PIPELINE_ABORTED</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a pipeline that has
+ received an error from the server. <function>PQgetResult</function>
+ must be called repeatedly, and each time it will return this status code
+ until the end of the current pipeline, at which point it will return
+ <literal>PGRES_PIPELINE_SYNC</literal> and normal processing can
+ resume.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
If the result status is <literal>PGRES_TUPLES_OK</literal> or
@@ -4926,6 +4953,479 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-pipeline-mode">
+ <title>Pipeline Mode</title>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>libpq</primary>
+ <secondary>pipeline mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>pipelining</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>batch mode</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> pipeline mode allows applications to
+ send a query without having to read the result of the previously
+ sent query. Taking advantage of the pipeline mode, a client will wait
+ less for the server, since multiple queries/results can be
+ sent/received in a single network transaction.
+ </para>
+
+ <para>
+ While pipeline mode provides a significant performance boost, writing
+ clients using the pipeline mode is more complex because it involves
+ managing a queue of pending queries and finding which result
+ corresponds to which query in the queue.
+ </para>
+
+ <para>
+ Pipeline mode also generally consumes more memory on both the client and server,
+ though careful and aggressive management of the send/receive queue can mitigate
+ this. This applies whether or not the connection is in blocking or non-blocking
+ mode.
+ </para>
+
+ <para>
+ While the pipeline API was introduced in
+ <productname>PostgreSQL</productname> 14, it is a client-side feature
+ which doesn't require special server support, and works on any server
+ that supports the v3 extended query protocol.
+ </para>
+
+ <sect2 id="libpq-pipeline-using">
+ <title>Using Pipeline Mode</title>
+
+ <para>
+ To issue pipelines, the application must switch the connection
+ into pipeline mode,
+ which is done with <xref linkend="libpq-PQenterPipelineMode"/>.
+ <xref linkend="libpq-PQpipelineStatus"/> can be used
+ to test whether pipeline mode is active.
+ In pipeline mode, only <link linkend="libpq-async">asynchronous operations</link>
+ are permitted, and <literal>COPY</literal> is disallowed.
+ Using synchronous command execution functions
+ such as <function>PQfn</function>,
+ <function>PQexec</function>,
+ <function>PQexecParams</function>,
+ <function>PQprepare</function>,
+ <function>PQexecPrepared</function>,
+ <function>PQdescribePrepared</function>,
+ <function>PQdescribePortal</function>,
+ is an error condition.
+ Once all dispatched commands have had their results processed, and
+ the end pipeline result has been consumed, the application may return
+ to non-pipelined mode with <xref linkend="libpq-PQexitPipelineMode"/>.
+ </para>
+
+ <note>
+ <para>
+ It is best to use pipeline mode with <application>libpq</application> in
+ <link linkend="libpq-PQsetnonblocking">non-blocking mode</link>. If used
+ in blocking mode it is possible for a client/server deadlock to occur.
+ <footnote>
+ <para>
+ The client will block trying to send queries to the server, but the
+ server will block trying to send results to the client from queries
+ it has already processed. This only occurs when the client sends
+ enough queries to fill both its output buffer and the server's receive
+ buffer before it switches to processing input from the server,
+ but it's hard to predict exactly when that will happen.
+ </para>
+ </footnote>
+ </para>
+ </note>
+
+ <sect3 id="libpq-pipeline-sending">
+ <title>Issuing Queries</title>
+
+ <para>
+ After entering pipeline mode, the application dispatches requests using
+ <xref linkend="libpq-PQsendQuery"/>,
+ <xref linkend="libpq-PQsendQueryParams"/>,
+ or its prepared-query sibling
+ <xref linkend="libpq-PQsendQueryPrepared"/>.
+ These requests are queued on the client-side until flushed to the server;
+ this occurs when <xref linkend="libpq-PQsendPipeline"/> is used to
+ establish a synchronization point in the pipeline,
+ or when <xref linkend="libpq-PQflush"/> is called.
+ The functions <xref linkend="libpq-PQsendPrepare"/>,
+ <xref linkend="libpq-PQsendDescribePrepared"/>, and
+ <xref linkend="libpq-PQsendDescribePortal"/> also work in pipeline mode.
+ Result processing is described below.
+ </para>
+
+ <para>
+ The server executes statements, and returns results, in the order the
+ client sends them. The server will begin executing the commands in the
+ pipeline immediately, not waiting for the end of the pipeline.
+ If any statement encounters an error, the server aborts the current
+ transaction and does not execute any subsequent command in the queue
+ until the next synchronization point established by
+ <function>PQsendPipeline</function>;
+ a <literal>PGRES_PIPELINE_ABORTED</literal> result is produced for
+ each such command.
+ (This remains true even if the commands in the pipeline would rollback
+ the transaction.)
+ Query processing resumes after the synchronization point.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one; for example, one query may define a table that the next
+ query in the same pipeline uses. Similarly, an application may
+ create a named prepared statement and execute it with later
+ statements in the same pipeline.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-results">
+ <title>Processing Results</title>
+
+ <para>
+ To process the result of one query in a pipeline, the application calls
+ <function>PQgetResult</function> repeatedly and handles each result
+ until <function>PQgetResult</function> returns null.
+ The result from the next query in the pipeline may then be retrieved using
+ <function>PQgetResult</function> again and the cycle repeated.
+ The application handles individual statement results as normal.
+ When the results of all the queries in the pipeline have been
+ returned, <function>PQgetResult</function> returns a result
+ containing the status value <literal>PGRES_PIPELINE_SYNC</literal>
+ </para>
+
+ <para>
+ The client may choose to defer result processing until the complete
+ pipeline has been sent, or interleave that with sending further
+ queries in the pipeline; see <xref linkend="libpq-pipeline-interleave"/>.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ This mode selection is effective only for the query currently
+ being processed. For more information on the use of
+ <function>PQsetSingleRowMode</function>,
+ refer to <xref linkend="libpq-single-row-mode"/>.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal
+ asynchronous processing except that it may contain the new
+ <type>PGresult</type> types <literal>PGRES_PIPELINE_SYNC</literal>
+ and <literal>PGRES_PIPELINE_ABORTED</literal>.
+ <literal>PGRES_PIPELINE_SYNC</literal> is reported exactly once for each
+ <function>PQsendPipeline</function> after retrieving results for all
+ queries in the pipeline.
+ <literal>PGRES_PIPELINE_ABORTED</literal> is emitted in place of a normal
+ stream result for the first error and all subsequent results
+ until <literal>PGRES_PIPELINE_SYNC</literal>;
+ see <xref linkend="libpq-pipeline-errors"/>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing pipeline results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed (except that
+ <function>PQgetResult</function> returns null to indicate that we start
+ returning the results of next query). The application must keep track
+ of the order in which it sent queries, to associate them with their
+ corresponding results.
+ Applications will typically use a state machine or a FIFO queue for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-pipeline-errors">
+ <title>Error Handling</title>
+
+ <para>
+ From the client's perspective, after <function>PQresultStatus</function>
+ returns <literal>PGRES_FATAL_ERROR</literal>,
+ the pipeline is flagged as aborted.
+ <function>PQresultStatus</function> will report a
+ <literal>PGRES_PIPELINE_ABORTED</literal> result for each remaining queued
+ operation in an aborted pipeline. The result for
+ <function>PQsendPipeline</function> is reported as
+ <literal>PGRES_PIPELINE_SYNC</literal> to signal the end of the aborted pipeline
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the pipeline used an implicit transaction, then operations that have
+ already executed are rolled back and operations that were queued to follow
+ the failed operation are skipped entirely. The same behavior holds if the
+ pipeline starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the pipeline. If a pipeline contains
+ <emphasis>multiple explicit transactions</emphasis>, all transactions that
+ committed prior to the error remain committed, the currently in-progress
+ transaction is aborted, and all subsequent operations are skipped completely,
+ including subsequent transactions. If a pipeline synchronization point
+ occurs with an explicit transaction block in aborted state, the next pipeline
+ will become aborted immediately unless the next command puts the transaction
+ in normal mode with <command>ROLLBACK</command>.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal> — only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously, the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-interleave">
+ <title>Interleaving Result Processing and Query Dispatch</title>
+
+ <para>
+ To avoid deadlocks on large pipelines the client should be structured
+ around a non-blocking event loop using operating system facilities
+ such as <function>select</function>, <function>poll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work
+ remaining to be dispatched and a queue of work that has been dispatched
+ but not yet had its results processed. When the socket is writable
+ it should dispatch more work. When the socket is readable it should
+ read results and process them, matching them up to the next entry in
+ its corresponding results queue. Based on available memory, results from the
+ socket should be read frequently: there's no need to wait until the
+ pipeline end to read the results. Pipelines should be scoped to logical
+ units of work, usually (but not necessarily) one transaction per pipeline.
+ There's no need to exit pipeline mode and re-enter it between pipelines,
+ or to wait for one pipeline to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state
+ machine to track sent and received work is in
+ <filename>src/test/modules/libpq_pipeline/libpq_pipeline.c</filename>
+ in the PostgreSQL source distribution.
+ </para>
+ </sect3>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-functions">
+ <title>Functions Associated with Pipeline Mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQpipelineStatus">
+ <term><function>PQpipelineStatus</function><indexterm><primary>PQpipelineStatus</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Returns the current pipeline mode status of the
+ <application>libpq</application> connection.
+<synopsis>
+PGpipelineStatus PQpipelineStatus(const PGconn *conn);
+</synopsis>
+ </para>
+
+ <para>
+ <function>PQpipelineStatus</function> can return one of the following values:
+
+ <variablelist>
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ON</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in
+ pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_OFF</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is
+ <emphasis>not</emphasis> in pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in pipeline
+ mode and an error occurred while processing the current pipeline.
+ The aborted flag is cleared when <function>PQresultStatus</function>
+ returns PGRES_PIPELINE_SYNC at the end of the pipeline.
+ Clients don't usually need this function to
+ verify aborted status, as they can tell that the pipeline is aborted
+ from the <literal>PGRES_PIPELINE_ABORTED</literal> result code.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterPipelineMode">
+ <term><function>PQenterPipelineMode</function><indexterm><primary>PQenterPipelineMode</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter pipeline mode if it is currently idle or
+ already in pipeline mode.
+
+<synopsis>
+int PQenterPipelineMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success.
+ Returns 0 and has no effect if the connection is not currently
+ idle, i.e., it has a result ready, or it is waiting for more
+ input from the server, etc.
+ This function does not actually send anything to the server,
+ it just changes the <application>libpq</application> connection
+ state.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitPipelineMode">
+ <term><function>PQexitPipelineMode</function><indexterm><primary>PQexitPipelineMode</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit pipeline mode if it is currently in pipeline mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitPipelineMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 1 and takes no action if not in
+ pipeline mode. If the current statement isn't finished processing,
+ or <function>PQgetResult</function> has not been called to collect
+ results from all previously sent query, returns 0 (in which case,
+ use <xref linkend="libpq-PQerrorMessage"/> to get more information
+ about the failure).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQsendPipeline">
+ <term><function>PQsendPipeline</function><indexterm><primary>PQsendPipeline</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Marks a synchronization point in a pipeline by sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ and flushing the send buffer. This serves as
+ the delimiter of an implicit transaction and an error recovery
+ point; see <xref linkend="libpq-pipeline-errors"/>.
+
+<synopsis>
+int PQsendPipeline(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 if the connection is not in
+ pipeline mode or sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ failed.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-tips">
+ <title>When to Use Pipeline Mode</title>
+
+ <para>
+ Much like asynchronous query mode, there is no meaningful performance
+ overhead when using pipeline mode. It increases client application complexity,
+ and extra caution is required to prevent client/server deadlocks, but
+ pipeline mode can offer considerable performance improvements, in exchange for
+ increased memory usage from leaving state around longer.
+ </para>
+
+ <para>
+ Pipeline mode is most useful when the server is distant, i.e., network latency
+ (<quote>ping time</quote>) is high, and also when many small operations
+ are being performed in rapid succession. There is usually less benefit
+ in using pipelined commands when each query takes many multiples of the client/server
+ round-trip time to execute. A 100-statement operation run on a server
+ 300ms round-trip-time away would take 30 seconds in network latency alone
+ without pipelining; with pipelining it may spend as little as 0.3s waiting for
+ results from the server.
+ </para>
+
+ <para>
+ Use pipelined commands when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed
+ into operations on sets, or into a <literal>COPY</literal> operation.
+ </para>
+
+ <para>
+ Pipeline mode is not useful when information from one operation is required by
+ the client to produce the next operation. In such cases, the client
+ would have to introduce a synchronization point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+BEGIN;
+SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+-- result: x=2
+-- client adds 1 to x:
+UPDATE mytable SET x = 3 WHERE id = 42;
+COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <para>
+ Pipelining is less useful, and more complex, when a single pipeline contains
+ multiple transactions (see <xref linkend="libpq-pipeline-errors"/>).
+ </para>
+ </sect2>
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4966,6 +5466,13 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <para>
+ When using pipeline mode, single-row mode needs to be activated for each
+ query in the pipeline before retrieving results for that query
+ with <function>PQgetResult</function>.
+ See <xref linkend="libpq-pipeline-mode"/> for more information.
+ </para>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 6d46da42e2..012e44c736 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while a libpq connection is in pipeline mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 299d93b241..5dd1e9e936 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -1110,6 +1110,12 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
row, the last value is kept.
</para>
+ <para>
+ <literal>\gset</literal> and <literal>\aset</literal> cannot be used
+ pipeline mode, since query results are not immediately
+ fetched in this mode.
+ </para>
+
<para>
The following example puts the final account balance from the first query
into variable <replaceable>abalance</replaceable>, and fills variables
@@ -1270,6 +1276,21 @@ SELECT 4 AS four \; SELECT 5 AS five \aset
</programlisting></para>
</listitem>
</varlistentry>
+
+ <varlistentry id='pgbench-metacommand-pipeline'>
+ <term><literal>\startpipeline</literal></term>
+ <term><literal>\endpipeline</literal></term>
+
+ <listitem>
+ <para>
+ These commands delimit the start and end of a pipeline of SQL statements.
+ In a pipeline, statements are sent to server without waiting for the results
+ of previous statements (see <xref linkend="libpq-pipeline-mode"/>).
+ Pipeline mode requires the extended query protocol.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect2>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 5272eed9ab..f74378110a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1019,6 +1019,12 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->err = _("empty query");
break;
+ case PGRES_PIPELINE_SYNC:
+ case PGRES_PIPELINE_ABORTED:
+ walres->status = WALRCV_ERROR;
+ walres->err = _("unexpected pipeline mode");
+ break;
+
case PGRES_NONFATAL_ERROR:
case PGRES_FATAL_ERROR:
case PGRES_BAD_RESPONSE:
diff --git a/src/bin/pg_amcheck/pg_amcheck.c b/src/bin/pg_amcheck/pg_amcheck.c
index 008a75d207..c9d9900693 100644
--- a/src/bin/pg_amcheck/pg_amcheck.c
+++ b/src/bin/pg_amcheck/pg_amcheck.c
@@ -929,6 +929,8 @@ should_processing_continue(PGresult *res)
case PGRES_COPY_IN:
case PGRES_COPY_BOTH:
case PGRES_SINGLE_TUPLE:
+ case PGRES_PIPELINE_SYNC:
+ case PGRES_PIPELINE_ABORTED:
return false;
}
return true;
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index f6a214669c..b7400708fc 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -395,10 +395,11 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
- * and we enter the CSTATE_SLEEP state to wait for it to expire. Other
- * meta-commands are executed immediately. If the command about to start
- * is actually beyond the end of the script, advance to CSTATE_END_TX.
+ * CSTATE_WAIT_RESULT state unless in pipeline mode. On a \sleep
+ * meta-command, the timer is set, and we enter the CSTATE_SLEEP state to
+ * wait for it to expire. Other meta-commands are executed immediately. If
+ * the command about to start is actually beyond the end of the script,
+ * advance to CSTATE_END_TX.
*
* CSTATE_WAIT_RESULT waits until we get a result set back from the server
* for the current command.
@@ -530,7 +531,9 @@ typedef enum MetaCommand
META_IF, /* \if */
META_ELIF, /* \elif */
META_ELSE, /* \else */
- META_ENDIF /* \endif */
+ META_ENDIF, /* \endif */
+ META_STARTPIPELINE, /* \startpipeline */
+ META_ENDPIPELINE /* \endpipeline */
} MetaCommand;
typedef enum QueryMode
@@ -2568,6 +2571,10 @@ getMetaCommand(const char *cmd)
mc = META_GSET;
else if (pg_strcasecmp(cmd, "aset") == 0)
mc = META_ASET;
+ else if (pg_strcasecmp(cmd, "startpipeline") == 0)
+ mc = META_STARTPIPELINE;
+ else if (pg_strcasecmp(cmd, "endpipeline") == 0)
+ mc = META_ENDPIPELINE;
else
mc = META_NONE;
return mc;
@@ -2757,11 +2764,25 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In pipeline mode, we use asynchronous functions. If a
+ * server-side error occurs, it will be processed later
+ * among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ pg_log_error("%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2805,8 +2826,10 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
* varprefix should be set only with \gset or \aset, and SQL commands do
* not need it.
*/
+#if 0
Assert((meta == META_NONE && varprefix == NULL) ||
((meta == META_GSET || meta == META_ASET) && varprefix != NULL));
+#endif
res = PQgetResult(st->con);
@@ -2874,6 +2897,12 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* otherwise the result is simply thrown away by PQclear below */
break;
+ case PGRES_PIPELINE_SYNC:
+ pg_log_debug("client %d pipeline ending", st->id);
+ if (PQexitPipelineMode(st->con) != 1)
+ pg_log_error("client %d failed to exit pipeline mode", st->id);
+ break;
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3127,13 +3156,36 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
/* Execute the command */
if (command->type == SQL_COMMAND)
{
+ /* disallow \aset and \gset in pipeline mode */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ if (command->meta == META_GSET)
+ {
+ commandFailed(st, "gset", "\\gset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ else if (command->meta == META_ASET)
+ {
+ commandFailed(st, "aset", "\\aset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ }
+
if (!sendCommand(st, command))
{
commandFailed(st, "SQL", "SQL command send failed");
st->state = CSTATE_ABORTED;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in pipeline mode */
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -3273,7 +3325,15 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (readCommandResponse(st,
sql_script[st->use_file].commands[st->command]->meta,
sql_script[st->use_file].commands[st->command]->varprefix))
- st->state = CSTATE_END_COMMAND;
+ {
+ /*
+ * outside of pipeline mode: stop reading results.
+ * pipeline mode: continue reading results until an
+ * end-of-pipeline response.
+ */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ st->state = CSTATE_END_COMMAND;
+ }
else
st->state = CSTATE_ABORTED;
break;
@@ -3516,6 +3576,49 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
return CSTATE_ABORTED;
}
}
+ else if (command->meta == META_STARTPIPELINE)
+ {
+ /*
+ * In pipeline mode, we use a workflow based on libpq pipeline
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "startpipeline", "cannot use pipeline mode with the simple query protocol");
+ return CSTATE_ABORTED;
+ }
+
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ commandFailed(st, "startpipeline", "already in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (PQenterPipelineMode(st->con) == 0)
+ {
+ commandFailed(st, "startpipeline", "failed to enter pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ }
+ else if (command->meta == META_ENDPIPELINE)
+ {
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ {
+ commandFailed(st, "endpipeline", "not in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (!PQsendPipeline(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to send the pipeline");
+ return CSTATE_ABORTED;
+ }
+ if (!PQexitPipelineMode(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to exit pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ /* collect pending results before getting out of pipeline mode */
+ return CSTATE_WAIT_RESULT;
+ }
/*
* executing the expression or shell command might have taken a
@@ -4725,7 +4828,9 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
- else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
+ else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF ||
+ my_command->meta == META_STARTPIPELINE ||
+ my_command->meta == META_ENDPIPELINE)
{
if (my_command->argc != 1)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..60d09e6d63 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,7 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterPipelineMode 180
+PQexitPipelineMode 181
+PQsendPipeline 182
+PQpipelineStatus 183
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 4e21057d0f..53b354abb2 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -522,6 +522,23 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * pqFreeCommandQueue
+ * Free all the entries of PGcmdQueueEntry queue passed.
+ */
+static void
+pqFreeCommandQueue(PGcmdQueueEntry *queue)
+{
+ while (queue != NULL)
+ {
+ PGcmdQueueEntry *cur = queue;
+
+ queue = cur->next;
+ if (cur->query)
+ free(cur->query);
+ free(cur);
+ }
+}
/*
* pqDropServerData
@@ -553,6 +570,12 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ pqFreeCommandQueue(conn->cmd_queue_head);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ pqFreeCommandQueue(conn->cmd_queue_recycle);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
@@ -2459,6 +2482,7 @@ keep_going: /* We will come back to here until there is
/* Drop any PGresult we might have, too */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn);
/* Reset conn->status to put the state machine in the right state */
@@ -3917,6 +3941,7 @@ makeEmptyPGconn(void)
conn->status = CONNECTION_BAD;
conn->asyncStatus = PGASYNC_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
conn->xactStatus = PQTRANS_IDLE;
conn->options_valid = false;
conn->nonblocking = false;
@@ -4084,8 +4109,6 @@ freePGconn(PGconn *conn)
if (conn->connip)
free(conn->connip);
/* Note that conn->Pfdebug is not ours to close or free */
- if (conn->last_query)
- free(conn->last_query);
if (conn->write_err_msg)
free(conn->write_err_msg);
if (conn->inBuffer)
@@ -4174,6 +4197,7 @@ closePGconn(PGconn *conn)
conn->status = CONNECTION_BAD; /* Well, not really _bad_ - just absent */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn); /* deallocate result */
resetPQExpBuffer(&conn->errorMessage);
release_conn_addrinfo(conn);
@@ -6726,6 +6750,15 @@ PQbackendPID(const PGconn *conn)
return conn->be_pid;
}
+PGpipelineStatus
+PQpipelineStatus(const PGconn *conn)
+{
+ if (!conn)
+ return PQ_PIPELINE_OFF;
+
+ return conn->pipelineStatus;
+}
+
int
PQconnectionNeedsPassword(const PGconn *conn)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9a038043b2..ab867b5f33 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_PIPELINE_SYNC",
+ "PGRES_PIPELINE_ABORTED"
};
/*
@@ -71,6 +73,8 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static void pqPipelineProcessQueue(PGconn *conn);
+static int pqPipelineFlush(PGconn *conn);
/* ----------------
@@ -1171,7 +1175,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1184,6 +1188,87 @@ fail:
}
+/*
+ * pqAllocCmdQueueEntry
+ * Get a command queue entry for caller to fill.
+ *
+ * If the recycle queue has a free element, that is returned; if not, a
+ * fresh one is allocated. Caller is responsible for adding it to the
+ * command queue (pqAppendCmdQueueEntry) once the struct is filled in, or
+ * releasing the memory (pqRecycleCmdQueueEntry) if an error occurs.
+ *
+ * If allocation fails, sets the error message and returns NULL.
+ */
+static PGcmdQueueEntry *
+pqAllocCmdQueueEntry(PGconn *conn)
+{
+ PGcmdQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcmdQueueEntry *) malloc(sizeof(PGcmdQueueEntry));
+ if (entry == NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * pqAppendCmdQueueEntry
+ * Append a caller-allocated command queue entry to the queue.
+ *
+ * The query itself must already have been put in the output buffer by the
+ * caller.
+ */
+static void
+pqAppendCmdQueueEntry(PGconn *conn, PGcmdQueueEntry *entry)
+{
+ Assert(entry->next == NULL);
+
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * pqRecycleCmdQueueEntry
+ * Push a command queue entry onto the freelist.
+ */
+static void
+pqRecycleCmdQueueEntry(PGconn *conn, PGcmdQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+
+ /* recyclable entries should not have a follow-on command */
+ Assert(entry->next == NULL);
+
+ if (entry->query)
+ {
+ free(entry->query);
+ entry->query = NULL;
+ }
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+
/*
* PQsendQuery
* Submit a query, but don't wait for it to finish
@@ -1209,9 +1294,15 @@ PQsendQueryContinue(PGconn *conn, const char *query)
static int
PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
{
+ PGcmdQueueEntry *entry = NULL;
+
if (!PQsendQueryStart(conn, newQuery))
return 0;
+ entry = pqAllocCmdQueueEntry(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
/* check the argument */
if (!query)
{
@@ -1220,37 +1311,75 @@ PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
return 0;
}
- /* construct the outgoing Query message */
- if (pqPutMsgStart('Q', conn) < 0 ||
- pqPuts(query, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
+ /* Send the query message(s) */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
{
- /* error message should be set up already */
- return 0;
+ /* construct the outgoing Query message */
+ if (pqPutMsgStart('Q', conn) < 0 ||
+ pqPuts(query, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ {
+ /* error message should be set up already */
+ return 0;
+ }
+
+ /* remember we are using simple query protocol */
+ entry->queryclass = PGQUERY_SIMPLE;
+ /* and remember the query text too, if possible */
+ entry->query = strdup(query);
}
+ else
+ {
+ /*
+ * In pipeline mode we cannot use the simple protocol, so we send
+ * Parse, Bind, Describe Portal, Execute.
+ */
+ if (pqPutMsgStart('P', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPuts(query, conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ if (pqPutMsgStart('B', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ if (pqPutMsgStart('D', conn) < 0 ||
+ pqPutc('P', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ if (pqPutMsgStart('E', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPutInt(0, 4, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
- /* remember we are using simple query protocol */
- conn->queryclass = PGQUERY_SIMPLE;
-
- /* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ entry->queryclass = PGQUERY_EXTENDED;
+ entry->query = strdup(query);
+ }
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
- {
- /* error message should be set up already */
- return 0;
- }
+ if (pqPipelineFlush(conn) < 0)
+ goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ pqAppendCmdQueueEntry(conn, entry);
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
+
+sendFailed:
+ pqRecycleCmdQueueEntry(conn, entry);
+ /* error message should be set up already */
+ return 0;
}
/*
@@ -1307,6 +1436,8 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcmdQueueEntry *entry = NULL;
+
if (!PQsendQueryStart(conn, true))
return 0;
@@ -1330,6 +1461,10 @@ PQsendPrepare(PGconn *conn,
return 0;
}
+ entry = pqAllocCmdQueueEntry(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
/* construct the Parse message */
if (pqPutMsgStart('P', conn) < 0 ||
pqPuts(stmtName, conn) < 0 ||
@@ -1356,32 +1491,38 @@ PQsendPrepare(PGconn *conn,
if (pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /* Add a Sync, unless in pipeline mode. */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ entry->queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ /* if insufficient memory, query just winds up NULL */
+ entry->query = strdup(query);
+
+ pqAppendCmdQueueEntry(conn, entry);
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ conn->asyncStatus = PGASYNC_BUSY;
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
- /* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecycleCmdQueueEntry(conn, entry);
/* error message should be set up already */
return 0;
}
@@ -1429,7 +1570,8 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn, bool newQuery)
@@ -1450,20 +1592,57 @@ PQsendQueryStart(PGconn *conn, bool newQuery)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE &&
+ conn->pipelineStatus == PQ_PIPELINE_OFF)
{
appendPQExpBufferStr(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * When enqueuing commands we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when pqPipelineProcessQueue()
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ }
+ }
+ else
+ {
+ /*
+ * This command's results will come in immediately. Initialize async
+ * result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1487,10 +1666,16 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcmdQueueEntry *entry;
+
+ entry = pqAllocCmdQueueEntry(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
/*
- * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
- * using specified statement name and the unnamed portal.
+ * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync
+ * (if not in pipeline mode), using specified statement name and the
+ * unnamed portal.
*/
if (command)
@@ -1600,35 +1785,38 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /* construct the Sync message if not in pipeline mode */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ entry->queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ /* if insufficient memory, query just winds up NULL */
if (command)
- conn->last_query = strdup(command);
- else
- conn->last_query = NULL;
+ entry->query = strdup(command);
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ pqAppendCmdQueueEntry(conn, entry);
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecycleCmdQueueEntry(conn, entry);
/* error message should be set up already */
return 0;
}
@@ -1647,8 +1835,9 @@ PQsetSingleRowMode(PGconn *conn)
return 0;
if (conn->asyncStatus != PGASYNC_BUSY)
return 0;
- if (conn->queryclass != PGQUERY_SIMPLE &&
- conn->queryclass != PGQUERY_EXTENDED)
+ if (!conn->cmd_queue_head ||
+ (conn->cmd_queue_head->queryclass != PGQUERY_SIMPLE &&
+ conn->cmd_queue_head->queryclass != PGQUERY_EXTENDED))
return 0;
if (conn->result)
return 0;
@@ -1726,14 +1915,17 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
-
/*
* PQgetResult
* Get the next PGresult produced by a query. Returns NULL if no
* query work remains or an error has occurred (e.g. out of
* memory).
+ *
+ * In pipeline mode, once all the result of a query have been returned,
+ * PQgetResult returns NULL to let the user know that the next
+ * query is being processed. At the end of the pipeline, returns a
+ * result with PQresultStatus(result) == PGRES_PIPELINE_SYNC.
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1803,8 +1995,61 @@ PQgetResult(PGconn *conn)
{
case PGASYNC_IDLE:
res = NULL; /* query is complete */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to return the NULL that terminates the round of
+ * results from the current query; prepare to send the results
+ * of the next query when we're called next. Also, since this
+ * is the start of the results of the next query, clear any
+ * prior error message.
+ */
+ resetPQExpBuffer(&conn->errorMessage);
+ pqPipelineProcessQueue(conn);
+ }
break;
case PGASYNC_READY:
+
+ /*
+ * For any query type other than simple query protocol, we advance
+ * the command queue here. For simple query protocol, we can get
+ * the READY state multiple times before the command is actually
+ * complete, since the command string can contain many queries --
+ * so we wait till we receive ReadyForQuery.
+ */
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass != PGQUERY_SIMPLE)
+ pqCommandQueueAdvance(conn);
+ res = pqPrepareAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to send the results of the current query. Set
+ * us idle now, and ...
+ */
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /*
+ * ... in cases when we're sending a pipeline-sync result,
+ * move queue processing forwards immediately, so that next
+ * time we're called, we're prepared to return the next result
+ * received from the server. In all other cases, leave the
+ * queue state change for next time, so that a terminating
+ * NULL result is sent.
+ *
+ * In other words: we don't return a NULL after a pipeline
+ * sync.
+ */
+ if (res && res->resultStatus == PGRES_PIPELINE_SYNC)
+ pqPipelineProcessQueue(conn);
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -1985,6 +2230,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("synchronous command execution functions are not allowed in pipeline mode\n"));
+ return false;
+ }
+
/*
* Since this is the beginning of a query cycle, reset the error buffer.
*/
@@ -2148,6 +2400,8 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcmdQueueEntry *entry = NULL;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2155,6 +2409,10 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
if (!PQsendQueryStart(conn, true))
return 0;
+ entry = pqAllocCmdQueueEntry(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
/* construct the Describe message */
if (pqPutMsgStart('D', conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2163,32 +2421,32 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
-
- /* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
-
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
{
- free(conn->last_query);
- conn->last_query = NULL;
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
}
+ /* remember we are doing a Describe */
+ entry->queryclass = PGQUERY_DESCRIBE;
+
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ pqAppendCmdQueueEntry(conn, entry);
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecycleCmdQueueEntry(conn, entry);
/* error message should be set up already */
return 0;
}
@@ -2327,7 +2585,8 @@ PQputCopyEnd(PGconn *conn, const char *errormsg)
* If we sent the COPY command in extended-query mode, we must issue a
* Sync as well.
*/
- if (conn->queryclass != PGQUERY_SIMPLE)
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass != PGQUERY_SIMPLE)
{
if (pqPutMsgStart('S', conn) < 0 ||
pqPutMsgEnd(conn) < 0)
@@ -2541,6 +2800,13 @@ PQfn(PGconn *conn,
*/
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("PQfn not allowed in pipeline mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -2555,6 +2821,285 @@ PQfn(PGconn *conn,
args, nargs);
}
+/* ====== Pipeline mode support ======== */
+
+/*
+ * PQenterPipelineMode
+ * Put an idle connection in pipeline mode.
+ *
+ * Returns 1 on success. On failure, errorMessage is set and 0 is returned.
+ *
+ * Commands submitted after this can be pipelined on the connection;
+ * there's no requirement to wait for one to finish before the next is
+ * dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQsendPipeline. Multiple pipelines
+ * can be sent while in pipeline mode. Pipeline mode can be exited
+ * by calling PQexitPipelineMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ /* succeed with no action if already in pipeline mode */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot enter pipeline mode, connection not idle\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+
+ return 1;
+}
+
+/*
+ * PQexitPipelineMode
+ * End pipeline mode and return to normal command mode.
+ *
+ * Returns 1 in success (pipeline mode successfully ended, or not in pipeline
+ * mode).
+ *
+ * Returns 0 if in pipeline mode and cannot be ended yet. Error message will
+ * be set.
+ */
+int
+PQexitPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ /* there are some uncollected results */
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+
+ case PGASYNC_BUSY:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode while busy\n"));
+ return 0;
+
+ default:
+ /* OK */
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ return 0; /* error message is setup already */
+ return 1;
+}
+
+/*
+ * pqCommandQueueAdvance
+ * Remove one query from the command queue, when we receive
+ * all results from the server that pertain to it.
+ */
+void
+pqCommandQueueAdvance(PGconn *conn)
+{
+ PGcmdQueueEntry *prevquery;
+
+ if (conn->cmd_queue_head == NULL)
+ return;
+
+ /* delink from queue */
+ prevquery = conn->cmd_queue_head;
+ conn->cmd_queue_head = conn->cmd_queue_head->next;
+
+ /* and make it recyclable */
+ prevquery->next = NULL;
+ pqRecycleCmdQueueEntry(conn, prevquery);
+}
+
+/*
+ * pqPipelineProcessQueue: subroutine for PQgetResult
+ * In pipeline mode, start processing the results of the next query in the queue.
+ */
+void
+pqPipelineProcessQueue(PGconn *conn)
+{
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return;
+ case PGASYNC_IDLE:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * XXX rewrite this comment. In pipeline mode but nothing left on the
+ * queue; caller can submit more work or PQexitPipelineMode() now.
+ */
+ return;
+ }
+
+ /* In non-pipeline mode, we're done here */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ return;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /*
+ * Reset single-row processing mode. (Client has to set it up for each
+ * query, if desired.)
+ */
+ conn->singleRowMode = false;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_ABORTED &&
+ conn->cmd_queue_head->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted pipeline we don't get anything from the server for
+ * each result; we're just discarding commands from the queue until we
+ * get to the next sync from the server.
+ *
+ * The PGRES_PIPELINE_ABORTED results tell the client that its queries
+ * got aborted.
+ */
+ conn->result = PQmakeEmptyPGresult(conn, PGRES_PIPELINE_ABORTED);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ pqSaveErrorResult(conn);
+ return;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+}
+
+/*
+ * PQsendPipeline
+ * Send a Sync message as part of a pipeline, and flush to server
+ *
+ * It's legal to start submitting more commands in the pipeline immediately,
+ * without waiting for the results of the current pipeline. There's no need to
+ * end pipeline mode and start it again.
+ *
+ * If a command in a pipeline fails, every subsequent command up to and including
+ * the result to the Sync message sent by PQsendPipeline gets set to
+ * PGRES_PIPELINE_ABORTED state. If the whole pipeline is processed without
+ * error, a PGresult with PGRES_PIPELINE_SYNC is produced.
+ *
+ * Queries can already have been sent before PQsendPipeline is called, but
+ * PQsendPipeline need to be called before retrieving command results.
+ *
+ * The connection will remain in pipeline mode and unavailable for new
+ * synchronous command execution functions until all results from the pipeline
+ * are processed by the client.
+ */
+int
+PQsendPipeline(PGconn *conn)
+{
+ PGcmdQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot send pipeline when not in pipeline mode\n"));
+ return 0;
+ }
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: cannot send pipeline while in COPY\n");
+ return 0;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_IDLE:
+ /* OK to send sync */
+ break;
+ }
+
+ entry = pqAllocCmdQueueEntry(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ pqAppendCmdQueueEntry(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ /*
+ * Call pqPipelineProcessQueue so the user can call start calling
+ * PQgetResult.
+ */
+ pqPipelineProcessQueue(conn);
+
+ return 1;
+
+sendFailed:
+ pqRecycleCmdQueueEntry(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
+
/* ====== accessor funcs for PGresult ======== */
@@ -2569,7 +3114,7 @@ PQresultStatus(const PGresult *res)
char *
PQresStatus(ExecStatusType status)
{
- if ((unsigned int) status >= sizeof pgresStatus / sizeof pgresStatus[0])
+ if ((unsigned int) status >= lengthof(pgresStatus))
return libpq_gettext("invalid ExecStatusType code");
return pgresStatus[status];
}
@@ -3152,6 +3697,23 @@ PQflush(PGconn *conn)
return pqFlush(conn);
}
+/*
+ * pqPipelineFlush
+ *
+ * In pipeline mode, data will be flushed only when the out buffer reaches the
+ * threshold value. In non-pipeline mode, it behaves as stock pqFlush.
+ *
+ * Returns 0 on success.
+ */
+static int
+pqPipelineFlush(PGconn *conn)
+{
+ if ((conn->pipelineStatus == PQ_PIPELINE_OFF) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return pqFlush(conn);
+ return 0;
+}
+
/*
* PQfreemem - safely frees memory allocated
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index eb55d528fb..306e89acfd 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -158,6 +158,18 @@ pqParseInput3(PGconn *conn)
if (conn->asyncStatus != PGASYNC_IDLE)
return;
+ /*
+ * We're also notionally not-IDLE when in pipeline mode the state
+ * says "idle" (so we have completed receiving the results of one
+ * query from the server and dispatched them to the application)
+ * but another query is queued; yield back control to caller so
+ * that they can initiate processing of the next query in the
+ * queue.
+ */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->cmd_queue_head != NULL)
+ return;
+
/*
* Unexpected message in IDLE state; need to recover somehow.
* ERROR messages are handled using the notice processor;
@@ -179,6 +191,7 @@ pqParseInput3(PGconn *conn)
}
else
{
+ /* Any other case is unexpected and we summarily skip it */
pqInternalNotice(&conn->noticeHooks,
"message type 0x%02x arrived from server while idle",
id);
@@ -217,10 +230,37 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new
+ * query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_PIPELINE_SYNC);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ else
+ {
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ }
+ else
+ {
+ /*
+ * In simple query protocol, advance the command queue
+ * (see PQgetResult).
+ */
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_SIMPLE)
+ pqCommandQueueAdvance(conn);
+ conn->asyncStatus = PGASYNC_IDLE;
+ }
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -238,7 +278,8 @@ pqParseInput3(PGconn *conn)
break;
case '1': /* Parse Complete */
/* If we're doing PQprepare, we're done; else ignore */
- if (conn->queryclass == PGQUERY_PREPARE)
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_PREPARE)
{
if (conn->result == NULL)
{
@@ -285,7 +326,8 @@ pqParseInput3(PGconn *conn)
conn->inCursor += msgLength;
}
else if (conn->result == NULL ||
- conn->queryclass == PGQUERY_DESCRIBE)
+ (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_DESCRIBE))
{
/* First 'T' in a query sequence */
if (getRowDescriptions(conn, msgLength))
@@ -316,7 +358,8 @@ pqParseInput3(PGconn *conn)
* instead of PGRES_TUPLES_OK. Otherwise we can just
* ignore this message.
*/
- if (conn->queryclass == PGQUERY_DESCRIBE)
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_DESCRIBE)
{
if (conn->result == NULL)
{
@@ -445,7 +488,7 @@ handleSyncLoss(PGconn *conn, char id, int msgLength)
id, msgLength);
/* build an error result holding the error message */
pqSaveErrorResult(conn);
- conn->asyncStatus = PGASYNC_READY; /* drop out of GetResult wait loop */
+ conn->asyncStatus = PGASYNC_READY; /* drop out of PQgetResult wait loop */
/* flush input data since we're giving up on processing it */
pqDropConnection(conn, true);
conn->status = CONNECTION_BAD; /* No more connection to backend */
@@ -471,7 +514,9 @@ getRowDescriptions(PGconn *conn, int msgLength)
* PGresult created by getParamDescriptions, and we should fill data into
* that. Otherwise, create a new, empty PGresult.
*/
- if (conn->queryclass == PGQUERY_DESCRIBE)
+ if (!conn->cmd_queue_head ||
+ (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_DESCRIBE))
{
if (conn->result)
result = conn->result;
@@ -568,7 +613,9 @@ getRowDescriptions(PGconn *conn, int msgLength)
* If we're doing a Describe, we're done, and ready to pass the result
* back to the client.
*/
- if (conn->queryclass == PGQUERY_DESCRIBE)
+ if ((!conn->cmd_queue_head) ||
+ (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_DESCRIBE))
{
conn->asyncStatus = PGASYNC_READY;
return 0;
@@ -841,6 +888,10 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ /* If in pipeline mode, set error indicator for it */
+ if (isError && conn->pipelineStatus != PQ_PIPELINE_OFF)
+ conn->pipelineStatus = PQ_PIPELINE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
@@ -897,8 +948,8 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
* might need it for an error cursor display, which is only true if there
* is a PG_DIAG_STATEMENT_POSITION field.
*/
- if (have_position && conn->last_query && res)
- res->errQuery = pqResultStrdup(res, conn->last_query);
+ if (have_position && res && conn->cmd_queue_head && conn->cmd_queue_head->query)
+ res->errQuery = pqResultStrdup(res, conn->cmd_queue_head->query);
/*
* Now build the "overall" error message for PQresultErrorMessage.
@@ -1817,7 +1868,8 @@ pqEndcopy3(PGconn *conn)
* If we sent the COPY command in extended-query mode, we must issue a
* Sync as well.
*/
- if (conn->queryclass != PGQUERY_SIMPLE)
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass != PGQUERY_SIMPLE)
{
if (pqPutMsgStart('S', conn) < 0 ||
pqPutMsgEnd(conn) < 0)
@@ -1897,6 +1949,9 @@ pqFunctionCall3(PGconn *conn, Oid fnid,
int avail;
int i;
+ /* already validated by PQfn */
+ Assert(conn->pipelineStatus == PQ_PIPELINE_OFF);
+
/* PQfn already validated connection state */
if (pqPutMsgStart('F', conn) < 0 || /* function call msg */
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index fa9b62a844..fb31a49fab 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -96,7 +96,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_PIPELINE_SYNC, /* pipeline synchronization point */
+ PGRES_PIPELINE_ABORTED, /* Command didn't run because of an abort
+ * earlier in a pipeline */
} ExecStatusType;
typedef enum
@@ -136,6 +139,16 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PGpipelineStatus - Current status of pipeline mode
+ */
+typedef enum
+{
+ PQ_PIPELINE_OFF,
+ PQ_PIPELINE_ON,
+ PQ_PIPELINE_ABORTED
+} PGpipelineStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -327,6 +340,7 @@ extern int PQserverVersion(const PGconn *conn);
extern char *PQerrorMessage(const PGconn *conn);
extern int PQsocket(const PGconn *conn);
extern int PQbackendPID(const PGconn *conn);
+extern PGpipelineStatus PQpipelineStatus(const PGconn *conn);
extern int PQconnectionNeedsPassword(const PGconn *conn);
extern int PQconnectionUsedPassword(const PGconn *conn);
extern int PQclientEncoding(const PGconn *conn);
@@ -434,6 +448,11 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for pipeline mode management */
+extern int PQenterPipelineMode(PGconn *conn);
+extern int PQexitPipelineMode(PGconn *conn);
+extern int PQsendPipeline(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 2f052f61f8..60f604d685 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,21 +217,16 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, more results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
} PGAsyncStatusType;
-/* PGQueryClass tracks which query protocol we are now executing */
-typedef enum
-{
- PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
- PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
- PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
-} PGQueryClass;
-
/* Target server type (decoded value of target_session_attrs) */
typedef enum
{
@@ -305,6 +300,29 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/*
+ * PGQueryClass tracks which query protocol is in use for each command queue
+ * entry, or special operation in execution
+ */
+typedef enum
+{
+ PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
+ PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
+ PGQUERY_PREPARE, /* Parse only (PQprepare) */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* Sync (at end of a pipeline) */
+} PGQueryClass;
+
+/*
+ * An entry in the pending command queue.
+ */
+typedef struct PGcmdQueueEntry
+{
+ PGQueryClass queryclass; /* Query type */
+ char *query; /* SQL command, or NULL if none/unknown/OOM */
+ struct PGcmdQueueEntry *next; /* list link */
+} PGcmdQueueEntry;
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -389,12 +407,11 @@ struct pg_conn
ConnStatusType status;
PGAsyncStatusType asyncStatus;
PGTransactionStatusType xactStatus; /* never changes to ACTIVE */
- PGQueryClass queryclass;
- char *last_query; /* last SQL command, or NULL if unknown */
char last_sqlstate[6]; /* last reported SQLSTATE */
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PGpipelineStatus pipelineStatus; /* status of pipeline mode */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -407,6 +424,19 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The pending command queue as a singly-linked list. Head is the command
+ * currently in execution, tail is where new commands are added.
+ */
+ PGcmdQueueEntry *cmd_queue_head;
+ PGcmdQueueEntry *cmd_queue_tail;
+
+ /*
+ * To save malloc traffic, we don't free entries right away; instead we
+ * save them in this list for possible reuse.
+ */
+ PGcmdQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -622,6 +652,7 @@ extern void pqSaveMessageField(PGresult *res, char code,
extern void pqSaveParameterStatus(PGconn *conn, const char *name,
const char *value);
extern int pqRowProcessor(PGconn *conn, const char **errmsgp);
+extern void pqCommandQueueAdvance(PGconn *conn);
extern int PQsendQueryContinue(PGconn *conn, const char *query);
/* === in fe-protocol3.c === */
@@ -795,6 +826,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 5391f461a2..93e7829c67 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -10,6 +10,7 @@ SUBDIRS = \
delay_execution \
dummy_index_am \
dummy_seclabel \
+ libpq_pipeline \
plsample \
snapshot_too_old \
test_bloomfilter \
diff --git a/src/test/modules/libpq_pipeline/.gitignore b/src/test/modules/libpq_pipeline/.gitignore
new file mode 100644
index 0000000000..3a11e786b8
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/libpq_pipeline
diff --git a/src/test/modules/libpq_pipeline/Makefile b/src/test/modules/libpq_pipeline/Makefile
new file mode 100644
index 0000000000..b798f5fbbc
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/Makefile
@@ -0,0 +1,20 @@
+# src/test/modules/libpq_pipeline/Makefile
+
+PROGRAM = libpq_pipeline
+OBJS = libpq_pipeline.o
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS_INTERNAL += $(libpq_pgport)
+
+TAP_TESTS = 1
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/libpq_pipeline
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/libpq_pipeline/README b/src/test/modules/libpq_pipeline/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/libpq_pipeline/libpq_pipeline.c b/src/test/modules/libpq_pipeline/libpq_pipeline.c
new file mode 100644
index 0000000000..cb79979a95
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/libpq_pipeline.c
@@ -0,0 +1,1263 @@
+/*-------------------------------------------------------------------------
+ *
+ * libpq_pipeline.c
+ * Verify libpq pipeline execution functionality
+ *
+ * Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/test/modules/libpq_pipeline/libpq_pipeline.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include <sys/time.h>
+#ifdef HAVE_SYS_SELECT_H
+#include <sys/select.h>
+#endif
+
+#include "catalog/pg_type_d.h"
+#include "common/fe_memutils.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+
+static void exit_nicely(PGconn *conn);
+
+const char *const progname = "libpq_pipeline";
+
+
+#define DEBUG
+#ifdef DEBUG
+#define pg_debug(...) do { fprintf(stderr, __VA_ARGS__); } while (0)
+#else
+#define pg_debug(...)
+#endif
+
+static const char *const drop_table_sql =
+"DROP TABLE IF EXISTS pq_pipeline_demo";
+static const char *const create_table_sql =
+"CREATE UNLOGGED TABLE pq_pipeline_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql =
+"INSERT INTO pq_pipeline_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+/*
+ * Print an error to stderr and terminate the program.
+ */
+#define pg_fatal(...) pg_fatal_impl(__LINE__, __VA_ARGS__)
+static void
+pg_fatal_impl(int line, const char *fmt,...)
+{
+ va_list args;
+
+
+ fflush(stdout);
+
+ fprintf(stderr, "\n%s:%d: ", progname, line);
+ va_start(args, fmt);
+ vfprintf(stderr, fmt, args);
+ va_end(args);
+ Assert(fmt[strlen(fmt) - 1] != '\n');
+ fprintf(stderr, "\n");
+ exit(1);
+}
+
+static void
+test_disallowed_in_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("Unable to enter pipeline mode");
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not activated properly");
+
+ /* PQexec should fail in pipeline mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("PQexec should fail in pipeline mode but succeeded");
+
+ /* Entering pipeline mode when already in pipeline mode is OK */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("re-entering pipeline mode should be a no-op but failed");
+
+ if (PQisBusy(conn) != 0)
+ pg_fatal("PQisBusy should return 0 when idle in pipeline mode, returned 1");
+
+ /* ok, back to normal command mode */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("couldn't exit idle empty pipeline mode");
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not terminated properly");
+
+ /* exiting pipeline mode when not in pipeline mode should be a no-op */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("pipeline mode exit when not in pipeline mode should succeed but failed");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("PQexec should succeed after exiting pipeline mode but failed with: %s",
+ PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_multi_pipelines(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi pipeline... ");
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first SELECT failed: %s", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending first pipeline failed: %s", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second SELECT failed: %s", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s", PQerrorMessage(conn));
+
+ /* OK, start processing the results */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first result");
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result expected: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of sync result, error: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ PQclear(res);
+
+ /* second pipeline */
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from second pipeline item",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode ... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work");
+
+ fprintf(stderr, "ok\n");
+}
+
+/*
+ * When an operation in a pipeline fails the rest of the pipeline is flushed. We
+ * still have to get results for each pipeline item, but the item will just be
+ * a PGRES_PIPELINE_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the pipeline. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_pipeline_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+ bool goterror;
+
+ fprintf(stderr, "aborted pipeline... ");
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching DROP TABLE failed: %s", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching CREATE TABLE failed: %s", PQerrorMessage(conn));
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * pipeline ERRORs.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first insert failed: %s", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT no_such_function($1)",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching error select failed: %s", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second insert failed: %s", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending first pipeline failed: %s", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second-pipeline insert failed: %s",
+ PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the pipeline results.
+ *
+ * We should get a command-ok for the first query, then a fatal error and
+ * a pipeline aborted message for the second insert, a pipeline-end, then
+ * a command-ok and a pipeline-ok for the second pipeline operation.
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result status %s: %s",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s",
+ PQresStatus(PQresultStatus(res)));
+
+ /* Second query caused error, so we expect an error next */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("Unexpected result code -- expected PGRES_FATAL_ERROR, got %s",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s",
+ PQresStatus(PQresultStatus(res)));
+
+ /*
+ * pipeline should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third pipeline since we've already sent a
+ * second. The aborted flag relates only to the pipeline being received.
+ */
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't");
+
+ /* third query in pipeline, the second insert */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_ABORTED)
+ pg_fatal("Unexpected result code -- expected PGRES_PIPELINE_ABORTED, got %s",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s", PQresStatus(PQresultStatus(res)));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't");
+
+ /* Ensure we're still in pipeline */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /*
+ * The end of a failed pipeline is a PGRES_PIPELINE_SYNC.
+ *
+ * (This is so clients know to start processing results normally again and
+ * can tell the difference between skipped commands and the sync.)
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code from first pipeline sync\n"
+ "Expected PGRES_PIPELINE_SYNC, got %s",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_ABORTED)
+ pg_fatal("sync should've cleared the aborted flag but didn't");
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /* the insert from the second pipeline */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result code %s from first item in second pipeline",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* Read the NULL result at the end of the command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s", PQresStatus(PQresultStatus(res)));
+
+ /* the second pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s: %s",
+ PQresStatus(PQresultStatus(res)),
+ PQerrorMessage(conn));
+
+ /* Test single-row mode with an error partways */
+ if (PQsendQuery(conn, "SELECT 1.0/g FROM generate_series(3, -1, -1) g") != 1)
+ pg_fatal("failed to send query: %s", PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ PQsetSingleRowMode(conn);
+ goterror = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_SINGLE_TUPLE:
+ printf("got row: %s\n", PQgetvalue(res, 0, 0));
+ break;
+ case PGRES_FATAL_ERROR:
+ if (strcmp(PQresultErrorField(res, PG_DIAG_SQLSTATE), "22012") != 0)
+ pg_fatal("expected division-by-zero, got: %s",
+ PQresultErrorField(res, PG_DIAG_SQLSTATE));
+ printf("got expected division-by-zero\n");
+ goterror = true;
+ break;
+ default:
+ pg_fatal("got unexpected result %s", PQresStatus(PQresultStatus(res)));
+ }
+ PQclear(res);
+ }
+ if (!goterror)
+ pg_fatal("did not get division-by-zero error");
+ /* the third pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from third pipeline sync",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work");
+
+ fprintf(stderr, "ok\n");
+
+ /*-
+ * Since we fired the pipelines off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st pipeline
+ * - First insert applied
+ * - Second statement aborted xact
+ * - Third insert skipped
+ * - Sync rolled back first implicit xact
+ * - Implicit xact created by server around 2nd pipeline
+ * - insert applied from 2nd pipeline
+ * - Sync commits 2nd xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM pq_pipeline_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("expected 1 result, got %d", PQntuples(res));
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ pg_fatal("expected only insert with value 3, got %s", val);
+ }
+
+ PQclear(res);
+}
+
+/* State machine enum for test_pipelined_insert */
+enum PipelineInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+};
+
+static void
+test_pipelined_insert(PGconn *conn, int n_rows)
+{
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ enum PipelineInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a pipelined insert into a table created at the start of the pipeline
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s", PQerrorMessage(conn));
+
+ while (send_step != BI_PREPARE)
+ {
+ const char *sql;
+
+ switch (send_step)
+ {
+ case BI_BEGIN_TX:
+ sql = "BEGIN TRANSACTION";
+ send_step = BI_DROP_TABLE;
+ break;
+
+ case BI_DROP_TABLE:
+ sql = drop_table_sql;
+ send_step = BI_CREATE_TABLE;
+ break;
+
+ case BI_CREATE_TABLE:
+ sql = create_table_sql;
+ send_step = BI_PREPARE;
+ break;
+
+ default:
+ pg_fatal("invalid state");
+ }
+
+ pg_debug("sending: %s\n", sql);
+ if (PQsendQueryParams(conn, sql,
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching %s failed: %s", sql, PQerrorMessage(conn));
+ }
+
+ Assert(send_step == BI_PREPARE);
+ pg_debug("sending: %s\n", insert_sql);
+ if (PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids) != 1)
+ pg_fatal("dispatching PREPARE failed: %s", PQerrorMessage(conn));
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our output buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the pipeline before processing
+ * results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ pg_fatal("failed to set nonblocking mode: %s", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's output buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ PGresult *res;
+ const char *cmdtag;
+ const char *description = "";
+ int status;
+
+ /*
+ * Read next result. If no more results from this query,
+ * advance to the next query
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ continue;
+
+ status = PGRES_COMMAND_OK;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ recv_step++;
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ recv_step++;
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ recv_step++;
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ recv_step++;
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive == 0)
+ recv_step++;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ recv_step++;
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_PIPELINE_SYNC;
+ recv_step++;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ description = "";
+ abort();
+ }
+
+ if (PQresultStatus(res) != status)
+ pg_fatal("%s reported status %s, expected %s\n"
+ "Error message: \"%s\"",
+ description, PQresStatus(PQresultStatus(res)),
+ PQresStatus(status), PQerrorMessage(conn));
+
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ pg_fatal("%s expected command tag '%s', got '%s'",
+ description, cmdtag, PQcmdStatus(res));
+
+ pg_debug("Got %s OK\n", cmdtag[0] != '\0' ? cmdtag : description);
+
+ PQclear(res);
+ }
+ }
+
+ /* Write more rows and/or the end pipeline message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQsendPipeline(conn) == 1)
+ {
+ fprintf(stdout, "Sent pipeline\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending pipeline failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+ }
+
+ /* We've got the sync message and the pipeline should be done */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s",
+ PQerrorMessage(conn));
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ pg_fatal("failed to clear nonblocking mode: %s", PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_prepared(PGconn *conn)
+{
+ PGresult *res = NULL;
+ Oid param_oids[1] = {INT4OID};
+ Oid expected_oids[4];
+ Oid typ;
+
+ fprintf(stderr, "prepared... ");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s", PQerrorMessage(conn));
+ if (PQsendPrepare(conn, "select_one", "SELECT $1, '42', $1::numeric, "
+ "interval '1 sec'",
+ 1, param_oids) != 1)
+ pg_fatal("preparing query failed: %s", PQerrorMessage(conn));
+ expected_oids[0] = INT4OID;
+ expected_oids[1] = TEXTOID;
+ expected_oids[2] = NUMERICOID;
+ expected_oids[3] = INTERVALOID;
+ if (PQsendDescribePrepared(conn, "select_one") != 1)
+ pg_fatal("failed to send describePrepared: %s", PQerrorMessage(conn));
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("failed to send pipeline: %s", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("expected COMMAND_OK, got %s", PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("expected NULL result");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned NULL");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("expected COMMAND_OK, got %s", PQresStatus(PQresultStatus(res)));
+ if (PQnfields(res) != lengthof(expected_oids))
+ pg_fatal("expected %d columns, got %d",
+ lengthof(expected_oids), PQnfields(res));
+ for (int i = 0; i < PQnfields(res); i++)
+ {
+ typ = PQftype(res, i);
+ if (typ != expected_oids[i])
+ pg_fatal("field %d: expected type %u, got %u",
+ i, expected_oids[i], typ);
+ }
+ PQclear(res);
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("expected NULL result");
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("expected PGRES_PIPELINE_SYNC, got %s", PQresStatus(PQresultStatus(res)));
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("could not exit pipeline mode: %s", PQerrorMessage(conn));
+
+ PQexec(conn, "BEGIN");
+ PQexec(conn, "DECLARE cursor_one CURSOR FOR SELECT 1");
+ PQenterPipelineMode(conn);
+ if (PQsendDescribePortal(conn, "cursor_one") != 1)
+ pg_fatal("PQsendDescribePortal failed: %s", PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("expected NULL result");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("expected COMMAND_OK, got %s", PQresStatus(PQresultStatus(res)));
+
+ typ = PQftype(res, 0);
+ if (typ != INT4OID)
+ pg_fatal("portal: expected type %u, got %u",
+ INT4OID, typ);
+ PQclear(res);
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("expected NULL result");
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("expected PGRES_PIPELINE_SYNC, got %s", PQresStatus(PQresultStatus(res)));
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("could not exit pipeline mode: %s", PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_simple_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple pipeline... ");
+
+ /*
+ * Enter pipeline mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our output buffer will give us enough slush
+ * to work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching SELECT failed: %s", PQerrorMessage(conn));
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode with work in progress should fail, but succeeded");
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending pipeline failed: %s", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item",
+ PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first query result.");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit pipeline mode yet
+ */
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result PGRES_PIPELINE_SYNC expected: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of PGRES_PIPELINE_SYNC, error: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after pipeline end: %s",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /* ... until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Exiting pipeline mode didn't seem to work");
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i;
+ bool pipeline_ended = false;
+
+ /* 1 pipeline, 3 queries in it */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s",
+ PQerrorMessage(conn));
+
+ for (i = 0; i < 3; i++)
+ {
+ char *param[1];
+
+ param[0] = psprintf("%d", 44 + i);
+
+ if (PQsendQueryParams(conn,
+ "SELECT generate_series(42, $1)",
+ 1,
+ NULL,
+ (const char **) param,
+ NULL,
+ NULL,
+ 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ pfree(param[0]);
+ }
+ PQsendPipeline(conn);
+
+ for (i = 0; !pipeline_ended; i++)
+ {
+ bool first = true;
+ bool saw_ending_tuplesok;
+ bool isSingleTuple = false;
+
+ /* Set single row mode for only first 2 SELECT queries */
+ if (i < 2)
+ {
+ if (PQsetSingleRowMode(conn) != 1)
+ pg_fatal("PQsetSingleRowMode() failed for i=%d", i);
+ }
+
+ /* Consume rows for this query */
+ saw_ending_tuplesok = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+
+ if (est == PGRES_PIPELINE_SYNC)
+ {
+ fprintf(stderr, "end of pipeline reached\n");
+ pipeline_ended = true;
+ PQclear(res);
+ if (i != 3)
+ pg_fatal("Expected three results, got %d", i);
+ break;
+ }
+
+ /* Expect SINGLE_TUPLE for queries 0 and 1, TUPLES_OK for 2 */
+ if (first)
+ {
+ if (i <= 1 && est != PGRES_SINGLE_TUPLE)
+ pg_fatal("Expected PGRES_SINGLE_TUPLE for query %d, got %s",
+ i, PQresStatus(est));
+ if (i >= 2 && est != PGRES_TUPLES_OK)
+ pg_fatal("Expected PGRES_TUPLES_OK for query %d, got %s",
+ i, PQresStatus(est));
+ first = false;
+ }
+
+ fprintf(stderr, "Result status %s for query %d", PQresStatus(est), i);
+ switch (est)
+ {
+ case PGRES_TUPLES_OK:
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ saw_ending_tuplesok = true;
+ if (isSingleTuple)
+ {
+ if (PQntuples(res) == 0)
+ fprintf(stderr, "all tuples received in query %d\n", i);
+ else
+ pg_fatal("Expected to follow PGRES_SINGLE_TUPLE, but received PGRES_TUPLES_OK directly instead");
+ }
+ break;
+
+ case PGRES_SINGLE_TUPLE:
+ isSingleTuple = true;
+ fprintf(stderr, ", %d tuple: %s\n", PQntuples(res), PQgetvalue(res, 0, 0));
+ break;
+
+ default:
+ pg_fatal("unexpected");
+ }
+ PQclear(res);
+ }
+ if (!pipeline_ended && !saw_ending_tuplesok)
+ pg_fatal("didn't get expected terminating TUPLES_OK");
+ }
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s", PQerrorMessage(conn));
+}
+
+/*
+ * Simple test to verify that a pipeline is discarded as a whole when there's
+ * an error, ignoring transaction commands.
+ */
+static void
+test_transaction(PGconn *conn)
+{
+ PGresult *res;
+ bool expect_null;
+ int num_sends = 0;
+
+ res = PQexec(conn, "DROP TABLE IF EXISTS pq_pipeline_tst;"
+ "CREATE TABLE pq_pipeline_tst (id int)");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to create test table: %s",
+ PQerrorMessage(conn));
+ PQclear(res);
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s",
+ PQerrorMessage(conn));
+ if (PQsendPrepare(conn, "rollback", "ROLLBACK", 0, NULL) != 1)
+ pg_fatal("could not send prepare on pipeline: %s",
+ PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn,
+ "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ if (PQsendQueryParams(conn,
+ "SELECT 0/0",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+
+ /*
+ * send a ROLLBACK using a prepared stmt. Doesn't work because we need to
+ * get out of the pipeline-aborted state first.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /* This insert fails because we're in pipeline-aborted state */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (1)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * This insert fails even though the pipeline got a SYNC, because we're in
+ * an aborted transaction
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (2)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * Send ROLLBACK using prepared stmt. This one works because we just did
+ * PQsendPipeline above.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /*
+ * Now that we're out of a transaction and in pipeline-good mode, this
+ * insert works
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (3)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+ PQsendPipeline(conn);
+ num_sends++;
+
+ expect_null = false;
+ for (int i = 0;; i++)
+ {
+ ExecStatusType restype;
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ printf("%d: got NULL result\n", i);
+ if (!expect_null)
+ pg_fatal("did not expect NULL here");
+ expect_null = false;
+ continue;
+ }
+ restype = PQresultStatus(res);
+ printf("%d: got status %s", i, PQresStatus(restype));
+ if (expect_null)
+ pg_fatal("expected NULL");
+ if (restype == PGRES_FATAL_ERROR)
+ printf("; error: %s", PQerrorMessage(conn));
+ else if (restype == PGRES_PIPELINE_ABORTED)
+ {
+ printf(": command didn't run because pipeline aborted\n");
+ }
+ else
+ printf("\n");
+ PQclear(res);
+
+ if (restype == PGRES_PIPELINE_SYNC)
+ num_sends--;
+ else
+ expect_null = true;
+ if (num_sends <= 0)
+ break;
+ }
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("returned something extra after all the syncs: %s",
+ PQresStatus(PQresultStatus(res)));
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s", PQerrorMessage(conn));
+
+ /* We expect to find one tuple containing the value "3" */
+ res = PQexec(conn, "SELECT * FROM pq_pipeline_tst");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("failed to obtain result: %s", PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("did not get 1 tuple");
+ if (strcmp(PQgetvalue(res, 0, 0), "3") != 0)
+ pg_fatal("did not get expected tuple");
+ PQclear(res);
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+usage(const char *progname)
+{
+ fprintf(stderr, "%s tests libpq's pipeline mode.\n\n", progname);
+ fprintf(stderr, "Usage:\n");
+ fprintf(stderr, " %s tests", progname);
+ fprintf(stderr, " %s testname [conninfo [number_of_rows]]\n", progname);
+}
+
+static void
+print_test_list(void)
+{
+ printf("disallowed_in_pipeline\n");
+ printf("multi_pipelines\n");
+ printf("pipeline_abort\n");
+ printf("pipelined_insert\n");
+ printf("prepared\n");
+ printf("simple_pipeline\n");
+ printf("singlerow\n");
+ printf("transaction\n");
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo = "";
+ PGconn *conn;
+ int numrows = 10000;
+ PGresult *res;
+
+ if (strcmp(argv[1], "tests") == 0)
+ {
+ print_test_list();
+ exit(0);
+ }
+
+ /*
+ * The testname parameter is mandatory; it can be followed by a conninfo
+ * string and number of rows.
+ */
+ if (argc < 2 || argc > 4)
+ {
+ usage(argv[0]);
+ exit(1);
+ }
+
+ if (argc >= 3)
+ conninfo = pg_strdup(argv[2]);
+
+ if (argc >= 4)
+ {
+ errno = 0;
+ numrows = strtol(argv[3], NULL, 10);
+ if (errno != 0 || numrows <= 0)
+ {
+ fprintf(stderr, "couldn't parse \"%s\" as a positive integer\n", argv[3]);
+ exit(1);
+ }
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+ res = PQexec(conn, "SET lc_messages TO \"C\"");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to set lc_messages: %s", PQerrorMessage(conn));
+
+ if (strcmp(argv[1], "disallowed_in_pipeline") == 0)
+ test_disallowed_in_pipeline(conn);
+ else if (strcmp(argv[1], "multi_pipelines") == 0)
+ test_multi_pipelines(conn);
+ else if (strcmp(argv[1], "pipeline_abort") == 0)
+ test_pipeline_abort(conn);
+ else if (strcmp(argv[1], "pipelined_insert") == 0)
+ test_pipelined_insert(conn, numrows);
+ else if (strcmp(argv[1], "prepared") == 0)
+ test_prepared(conn);
+ else if (strcmp(argv[1], "simple_pipeline") == 0)
+ test_simple_pipeline(conn);
+ else if (strcmp(argv[1], "singlerow") == 0)
+ test_singlerowmode(conn);
+ else if (strcmp(argv[1], "transaction") == 0)
+ test_transaction(conn);
+ else
+ {
+ fprintf(stderr, "\"%s\" is not a recognized test name\n", argv[1]);
+ usage(argv[0]);
+ exit(1);
+ }
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+ return 0;
+}
diff --git a/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
new file mode 100644
index 0000000000..ba15b64ca7
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
@@ -0,0 +1,28 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 8;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $numrows = 10000;
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+
+my ($out, $err) = run_command(['libpq_pipeline', 'tests']);
+die "oops: $err" unless $err eq '';
+my @tests = split(/\s/, $out);
+
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'libpq_pipeline', $testname, $node->connstr('postgres'), $numrows ],
+ "libpq_pipeline $testname");
+}
+
+$node->stop('fast');
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 74fde40e3a..a184404e21 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -33,10 +33,11 @@ my @unlink_on_exit;
# Set of variables for modules in contrib/ and src/test/modules/
my $contrib_defines = { 'refint' => 'REFINT_VERBOSE' };
-my @contrib_uselibpq = ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo');
-my @contrib_uselibpgport = ('oid2name', 'vacuumlo');
-my @contrib_uselibpgcommon = ('oid2name', 'vacuumlo');
-my $contrib_extralibs = undef;
+my @contrib_uselibpq =
+ ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo', 'libpq_pipeline');
+my @contrib_uselibpgport = ('libpq_pipeline', 'oid2name', 'vacuumlo');
+my @contrib_uselibpgcommon = ('libpq_pipeline', 'oid2name', 'vacuumlo');
+my $contrib_extralibs = { 'libpq_pipeline' => ['ws2_32.lib'] };
my $contrib_extraincludes = { 'dblink' => ['src/backend'] };
my $contrib_extrasource = {
'cube' => [ 'contrib/cube/cubescan.l', 'contrib/cube/cubeparse.y' ],
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index e017557e3e..41a62be994 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1563,10 +1563,12 @@ PG_Locale_Strategy
PG_Lock_Status
PG_init_t
PGcancel
+PGcmdQueueEntry
PGconn
PGdataValue
PGlobjfuncs
PGnotify
+PGpipelineStatus
PGresAttDesc
PGresAttValue
PGresParamDesc
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
Here's what seems a final version of the patch. I renamed one more
function: PQsendPipeline is now PQpipelineSync. I also reworded the
docs in a couple of places, added a few tests to the pgbench patch, and
made it work.
Note the pgbench results in pipeline mode:
./pgbench -r -Mextended -n -f /home/alvherre/Code/pgsql-build/pipeline/src/bin/pgbench/tmp_check/t_001_pgbench_with_server_main_data/001_pgbench_pipeline -c 100 -t10000
pgbench (PostgreSQL) 14.0
transaction type: /home/alvherre/Code/pgsql-build/pipeline/src/bin/pgbench/tmp_check/t_001_pgbench_with_server_main_data/001_pgbench_pipeline
scaling factor: 1
query mode: extended
number of clients: 100
number of threads: 1
number of transactions per client: 10000
number of transactions actually processed: 1000000/1000000
latency average = 2.316 ms
initial connection time = 113.859 ms
tps = 43182.438635 (without initial connection time)
statement latencies in milliseconds:
0.000 \startpipeline
0.000 select 1;
0.000 select 1;
0.000 select 1;
0.000 select 1;
0.000 select 1;
0.000 select 1;
0.000 select 1;
0.000 select 1;
0.000 select 1;
0.000 select 1;
1.624 \endpipeline
If I just replace the \startpipeline and \endpipeline lines with BEGIN
and COMMIT respectively, I get this:
tps = 10220.259051 (without initial connection time)
0.830 begin;
0.765 select 1;
0.752 select 1;
0.753 select 1;
0.755 select 1;
0.754 select 1;
0.755 select 1;
0.757 select 1;
0.756 select 1;
0.756 select 1;
0.756 select 1;
0.750 commit;
Yes, you could say that this is a liiiitle bit unfair -- but it seems
quite impressive nonetheless.
--
�lvaro Herrera 39�49'30"S 73�17'W
Attachments:
v37-0001-Implement-pipeline-mode-in-libpq.patchtext/x-diff; charset=iso-8859-1Download
From 5e4fdd5246d559caf0d75ad74001f09a48ec4c0e Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Mon, 15 Mar 2021 15:05:22 -0300
Subject: [PATCH v37 1/2] Implement pipeline mode in libpq
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Pipeline mode in libpq lets an application avoid the Sync messages in
the FE/BE protocol that are implicit in the old libpq API after each
query. The application can then insert Sync at its leisure with a new
libpq function PQpipelineSync. This can lead to substantial reductions
in query latency.
Co-authored-by: Craig Ringer <craig.ringer@enterprisedb.com>
Co-authored-by: Matthieu Garrigues <matthieu.garrigues@gmail.com>
Co-authored-by: Ãlvaro Herrera <alvherre@alvh.no-ip.org>
Reviewed-by: Andres Freund <andres@anarazel.de>
Reviewed-by: Aya Iwata <iwata.aya@jp.fujitsu.com>
Reviewed-by: Daniel Vérité <daniel@manitou-mail.org>
Reviewed-by: David G. Johnston <david.g.johnston@gmail.com>
Reviewed-by: Justin Pryzby <pryzby@telsasoft.com>
Reviewed-by: Kirk Jamison <k.jamison@fujitsu.com>
Reviewed-by: Michael Paquier <michael.paquier@gmail.com>
Reviewed-by: Nikhil Sontakke <nikhils@2ndquadrant.com>
Reviewed-by: Vaishnavi Prabakaran <VaishnaviP@fast.au.fujitsu.com>
Reviewed-by: Zhihong Yu <zyu@yugabyte.com>
Discussion: https://postgr.es/m/CAMsr+YFUjJytRyV4J-16bEoiZyH=4nj+sQ7JP9ajwz=B4dMMZw@mail.gmail.com
Discussion: https://postgr.es/m/CAJkzx4T5E-2cQe3dtv2R78dYFvz+in8PY7A8MArvLhs_pg75gg@mail.gmail.com
---
doc/src/sgml/libpq.sgml | 522 ++++++-
doc/src/sgml/lobj.sgml | 4 +
doc/src/sgml/ref/pgbench.sgml | 21 +
.../libpqwalreceiver/libpqwalreceiver.c | 6 +
src/bin/pg_amcheck/pg_amcheck.c | 2 +
src/interfaces/libpq/exports.txt | 4 +
src/interfaces/libpq/fe-connect.c | 37 +-
src/interfaces/libpq/fe-exec.c | 717 ++++++++-
src/interfaces/libpq/fe-protocol3.c | 77 +-
src/interfaces/libpq/libpq-fe.h | 21 +-
src/interfaces/libpq/libpq-int.h | 60 +-
src/test/modules/Makefile | 1 +
src/test/modules/libpq_pipeline/.gitignore | 5 +
src/test/modules/libpq_pipeline/Makefile | 20 +
src/test/modules/libpq_pipeline/README | 1 +
.../modules/libpq_pipeline/libpq_pipeline.c | 1303 +++++++++++++++++
.../libpq_pipeline/t/001_libpq_pipeline.pl | 28 +
src/tools/msvc/Mkvcbuild.pm | 9 +-
src/tools/pgindent/typedefs.list | 2 +
19 files changed, 2727 insertions(+), 113 deletions(-)
create mode 100644 src/test/modules/libpq_pipeline/.gitignore
create mode 100644 src/test/modules/libpq_pipeline/Makefile
create mode 100644 src/test/modules/libpq_pipeline/README
create mode 100644 src/test/modules/libpq_pipeline/libpq_pipeline.c
create mode 100644 src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 910e9a81ea..be674fbaa9 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -3180,6 +3180,33 @@ ExecStatusType PQresultStatus(const PGresult *res);
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-sync">
+ <term><literal>PGRES_PIPELINE_SYNC</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a
+ synchronization point in pipeline mode, requested by
+ <xref linkend="libpq-PQpipelineSync"/>.
+ This status occurs only when pipeline mode has been selected.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-aborted">
+ <term><literal>PGRES_PIPELINE_ABORTED</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a pipeline that has
+ received an error from the server. <function>PQgetResult</function>
+ must be called repeatedly, and each time it will return this status code
+ until the end of the current pipeline, at which point it will return
+ <literal>PGRES_PIPELINE_SYNC</literal> and normal processing can
+ resume.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
If the result status is <literal>PGRES_TUPLES_OK</literal> or
@@ -4677,8 +4704,9 @@ int PQsendDescribePortal(PGconn *conn, const char *portalName);
<xref linkend="libpq-PQsendQueryParams"/>,
<xref linkend="libpq-PQsendPrepare"/>,
<xref linkend="libpq-PQsendQueryPrepared"/>,
- <xref linkend="libpq-PQsendDescribePrepared"/>, or
- <xref linkend="libpq-PQsendDescribePortal"/>
+ <xref linkend="libpq-PQsendDescribePrepared"/>,
+ <xref linkend="libpq-PQsendDescribePortal"/>, or
+ <xref linkend="libpq-PQpipelineSync"/>
call, and returns it.
A null pointer is returned when the command is complete and there
will be no more results.
@@ -4702,6 +4730,19 @@ PGresult *PQgetResult(PGconn *conn);
<xref linkend="libpq-PQconsumeInput"/>.
</para>
+ <para>
+ In pipeline mode, <function>PQgetResult</function> will return normally
+ unless an error occurs; for any subsequent query sent after the one
+ that caused the error until (and excluding) the next synchronization point,
+ a special result of type <literal>PGRES_PIPELINE_ABORTED</literal> will
+ be returned, and a null pointer will be returned after it.
+ When the pipeline synchronization point is reached, a result of type
+ <literal>PGRES_PIPELINE_SYNC</literal> will be returned.
+ The result of the next query after the synchronization point follows
+ immediately (that is, no null pointer is returned after
+ the synchronization point.)
+ </para>
+
<note>
<para>
Even when <xref linkend="libpq-PQresultStatus"/> indicates a fatal
@@ -4926,6 +4967,476 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-pipeline-mode">
+ <title>Pipeline Mode</title>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>libpq</primary>
+ <secondary>pipeline mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>pipelining</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>batch mode</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> pipeline mode allows applications to
+ send a query without having to read the result of the previously
+ sent query. Taking advantage of the pipeline mode, a client will wait
+ less for the server, since multiple queries/results can be
+ sent/received in a single network transaction.
+ </para>
+
+ <para>
+ While pipeline mode provides a significant performance boost, writing
+ clients using the pipeline mode is more complex because it involves
+ managing a queue of pending queries and finding which result
+ corresponds to which query in the queue.
+ </para>
+
+ <para>
+ Pipeline mode also generally consumes more memory on both the client and server,
+ though careful and aggressive management of the send/receive queue can mitigate
+ this. This applies whether or not the connection is in blocking or non-blocking
+ mode.
+ </para>
+
+ <para>
+ While the pipeline API was introduced in
+ <productname>PostgreSQL</productname> 14, it is a client-side feature
+ which doesn't require special server support, and works on any server
+ that supports the v3 extended query protocol.
+ </para>
+
+ <sect2 id="libpq-pipeline-using">
+ <title>Using Pipeline Mode</title>
+
+ <para>
+ To issue pipelines, the application must switch the connection
+ into pipeline mode,
+ which is done with <xref linkend="libpq-PQenterPipelineMode"/>.
+ <xref linkend="libpq-PQpipelineStatus"/> can be used
+ to test whether pipeline mode is active.
+ In pipeline mode, only <link linkend="libpq-async">asynchronous operations</link>
+ are permitted, and <literal>COPY</literal> is disallowed.
+ Using synchronous command execution functions
+ such as <function>PQfn</function>,
+ <function>PQexec</function>,
+ <function>PQexecParams</function>,
+ <function>PQprepare</function>,
+ <function>PQexecPrepared</function>,
+ <function>PQdescribePrepared</function>,
+ <function>PQdescribePortal</function>,
+ is an error condition.
+ Once all dispatched commands have had their results processed, and
+ the end pipeline result has been consumed, the application may return
+ to non-pipelined mode with <xref linkend="libpq-PQexitPipelineMode"/>.
+ </para>
+
+ <note>
+ <para>
+ It is best to use pipeline mode with <application>libpq</application> in
+ <link linkend="libpq-PQsetnonblocking">non-blocking mode</link>. If used
+ in blocking mode it is possible for a client/server deadlock to occur.
+ <footnote>
+ <para>
+ The client will block trying to send queries to the server, but the
+ server will block trying to send results to the client from queries
+ it has already processed. This only occurs when the client sends
+ enough queries to fill both its output buffer and the server's receive
+ buffer before it switches to processing input from the server,
+ but it's hard to predict exactly when that will happen.
+ </para>
+ </footnote>
+ </para>
+ </note>
+
+ <sect3 id="libpq-pipeline-sending">
+ <title>Issuing Queries</title>
+
+ <para>
+ After entering pipeline mode, the application dispatches requests using
+ <xref linkend="libpq-PQsendQuery"/>,
+ <xref linkend="libpq-PQsendQueryParams"/>,
+ or its prepared-query sibling
+ <xref linkend="libpq-PQsendQueryPrepared"/>.
+ These requests are queued on the client-side until flushed to the server;
+ this occurs when <xref linkend="libpq-PQpipelineSync"/> is used to
+ establish a synchronization point in the pipeline,
+ or when <xref linkend="libpq-PQflush"/> is called.
+ The functions <xref linkend="libpq-PQsendPrepare"/>,
+ <xref linkend="libpq-PQsendDescribePrepared"/>, and
+ <xref linkend="libpq-PQsendDescribePortal"/> also work in pipeline mode.
+ Result processing is described below.
+ </para>
+
+ <para>
+ The server executes statements, and returns results, in the order the
+ client sends them. The server will begin executing the commands in the
+ pipeline immediately, not waiting for the end of the pipeline.
+ If any statement encounters an error, the server aborts the current
+ transaction and does not execute any subsequent command in the queue
+ until the next synchronization point established by
+ <function>PQpipelineSync</function>;
+ a <literal>PGRES_PIPELINE_ABORTED</literal> result is produced for
+ each such command.
+ (This remains true even if the commands in the pipeline would rollback
+ the transaction.)
+ Query processing resumes after the synchronization point.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one; for example, one query may define a table that the next
+ query in the same pipeline uses. Similarly, an application may
+ create a named prepared statement and execute it with later
+ statements in the same pipeline.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-results">
+ <title>Processing Results</title>
+
+ <para>
+ To process the result of one query in a pipeline, the application calls
+ <function>PQgetResult</function> repeatedly and handles each result
+ until <function>PQgetResult</function> returns null.
+ The result from the next query in the pipeline may then be retrieved using
+ <function>PQgetResult</function> again and the cycle repeated.
+ The application handles individual statement results as normal.
+ When the results of all the queries in the pipeline have been
+ returned, <function>PQgetResult</function> returns a result
+ containing the status value <literal>PGRES_PIPELINE_SYNC</literal>
+ </para>
+
+ <para>
+ The client may choose to defer result processing until the complete
+ pipeline has been sent, or interleave that with sending further
+ queries in the pipeline; see <xref linkend="libpq-pipeline-interleave"/>.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ This mode selection is effective only for the query currently
+ being processed. For more information on the use of
+ <function>PQsetSingleRowMode</function>,
+ refer to <xref linkend="libpq-single-row-mode"/>.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal
+ asynchronous processing except that it may contain the new
+ <type>PGresult</type> types <literal>PGRES_PIPELINE_SYNC</literal>
+ and <literal>PGRES_PIPELINE_ABORTED</literal>.
+ <literal>PGRES_PIPELINE_SYNC</literal> is reported exactly once for each
+ <function>PQpipelineSync</function> at the corresponding point
+ in the pipeline.
+ <literal>PGRES_PIPELINE_ABORTED</literal> is emitted in place of a normal
+ query result for the first error and all subsequent results
+ until the next <literal>PGRES_PIPELINE_SYNC</literal>;
+ see <xref linkend="libpq-pipeline-errors"/>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing pipeline results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed (except that
+ <function>PQgetResult</function> returns null to indicate that we start
+ returning the results of next query). The application must keep track
+ of the order in which it sent queries, to associate them with their
+ corresponding results.
+ Applications will typically use a state machine or a FIFO queue for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-pipeline-errors">
+ <title>Error Handling</title>
+
+ <para>
+ From the client's perspective, after <function>PQresultStatus</function>
+ returns <literal>PGRES_FATAL_ERROR</literal>,
+ the pipeline is flagged as aborted.
+ <function>PQresultStatus</function> will report a
+ <literal>PGRES_PIPELINE_ABORTED</literal> result for each remaining queued
+ operation in an aborted pipeline. The result for
+ <function>PQpipelineSync</function> is reported as
+ <literal>PGRES_PIPELINE_SYNC</literal> to signal the end of the aborted pipeline
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the pipeline used an implicit transaction, then operations that have
+ already executed are rolled back and operations that were queued to follow
+ the failed operation are skipped entirely. The same behavior holds if the
+ pipeline starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the pipeline. If a pipeline contains
+ <emphasis>multiple explicit transactions</emphasis>, all transactions that
+ committed prior to the error remain committed, the currently in-progress
+ transaction is aborted, and all subsequent operations are skipped completely,
+ including subsequent transactions. If a pipeline synchronization point
+ occurs with an explicit transaction block in aborted state, the next pipeline
+ will become aborted immediately unless the next command puts the transaction
+ in normal mode with <command>ROLLBACK</command>.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal> — only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously, the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-interleave">
+ <title>Interleaving Result Processing and Query Dispatch</title>
+
+ <para>
+ To avoid deadlocks on large pipelines the client should be structured
+ around a non-blocking event loop using operating system facilities
+ such as <function>select</function>, <function>poll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work
+ remaining to be dispatched and a queue of work that has been dispatched
+ but not yet had its results processed. When the socket is writable
+ it should dispatch more work. When the socket is readable it should
+ read results and process them, matching them up to the next entry in
+ its corresponding results queue. Based on available memory, results from the
+ socket should be read frequently: there's no need to wait until the
+ pipeline end to read the results. Pipelines should be scoped to logical
+ units of work, usually (but not necessarily) one transaction per pipeline.
+ There's no need to exit pipeline mode and re-enter it between pipelines,
+ or to wait for one pipeline to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state
+ machine to track sent and received work is in
+ <filename>src/test/modules/libpq_pipeline/libpq_pipeline.c</filename>
+ in the PostgreSQL source distribution.
+ </para>
+ </sect3>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-functions">
+ <title>Functions Associated with Pipeline Mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQpipelineStatus">
+ <term><function>PQpipelineStatus</function><indexterm><primary>PQpipelineStatus</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Returns the current pipeline mode status of the
+ <application>libpq</application> connection.
+<synopsis>
+PGpipelineStatus PQpipelineStatus(const PGconn *conn);
+</synopsis>
+ </para>
+
+ <para>
+ <function>PQpipelineStatus</function> can return one of the following values:
+
+ <variablelist>
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ON</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in
+ pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_OFF</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is
+ <emphasis>not</emphasis> in pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in pipeline
+ mode and an error occurred while processing the current pipeline.
+ The aborted flag is cleared when <function>PQgetResult</function>
+ returns a result of type <literal>PGRES_PIPELINE_SYNC</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterPipelineMode">
+ <term><function>PQenterPipelineMode</function><indexterm><primary>PQenterPipelineMode</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter pipeline mode if it is currently idle or
+ already in pipeline mode.
+
+<synopsis>
+int PQenterPipelineMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success.
+ Returns 0 and has no effect if the connection is not currently
+ idle, i.e., it has a result ready, or it is waiting for more
+ input from the server, etc.
+ This function does not actually send anything to the server,
+ it just changes the <application>libpq</application> connection
+ state.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitPipelineMode">
+ <term><function>PQexitPipelineMode</function><indexterm><primary>PQexitPipelineMode</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit pipeline mode if it is currently in pipeline mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitPipelineMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 1 and takes no action if not in
+ pipeline mode. If the current statement isn't finished processing,
+ or <function>PQgetResult</function> has not been called to collect
+ results from all previously sent query, returns 0 (in which case,
+ use <xref linkend="libpq-PQerrorMessage"/> to get more information
+ about the failure).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQpipelineSync">
+ <term><function>PQpipelineSync</function><indexterm><primary>PQpipelineSync</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Marks a synchronization point in a pipeline by sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ and flushing the send buffer. This serves as
+ the delimiter of an implicit transaction and an error recovery
+ point; see <xref linkend="libpq-pipeline-errors"/>.
+
+<synopsis>
+int PQpipelineSync(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 if the connection is not in
+ pipeline mode or sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ failed.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-tips">
+ <title>When to Use Pipeline Mode</title>
+
+ <para>
+ Much like asynchronous query mode, there is no meaningful performance
+ overhead when using pipeline mode. It increases client application complexity,
+ and extra caution is required to prevent client/server deadlocks, but
+ pipeline mode can offer considerable performance improvements, in exchange for
+ increased memory usage from leaving state around longer.
+ </para>
+
+ <para>
+ Pipeline mode is most useful when the server is distant, i.e., network latency
+ (<quote>ping time</quote>) is high, and also when many small operations
+ are being performed in rapid succession. There is usually less benefit
+ in using pipelined commands when each query takes many multiples of the client/server
+ round-trip time to execute. A 100-statement operation run on a server
+ 300ms round-trip-time away would take 30 seconds in network latency alone
+ without pipelining; with pipelining it may spend as little as 0.3s waiting for
+ results from the server.
+ </para>
+
+ <para>
+ Use pipelined commands when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed
+ into operations on sets, or into a <literal>COPY</literal> operation.
+ </para>
+
+ <para>
+ Pipeline mode is not useful when information from one operation is required by
+ the client to produce the next operation. In such cases, the client
+ would have to introduce a synchronization point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+BEGIN;
+SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+-- result: x=2
+-- client adds 1 to x:
+UPDATE mytable SET x = 3 WHERE id = 42;
+COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <para>
+ Pipelining is less useful, and more complex, when a single pipeline contains
+ multiple transactions (see <xref linkend="libpq-pipeline-errors"/>).
+ </para>
+ </sect2>
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4966,6 +5477,13 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <para>
+ When using pipeline mode, single-row mode needs to be activated for each
+ query in the pipeline before retrieving results for that query
+ with <function>PQgetResult</function>.
+ See <xref linkend="libpq-pipeline-mode"/> for more information.
+ </para>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 6d46da42e2..012e44c736 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while a libpq connection is in pipeline mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 299d93b241..5dd1e9e936 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -1110,6 +1110,12 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
row, the last value is kept.
</para>
+ <para>
+ <literal>\gset</literal> and <literal>\aset</literal> cannot be used
+ pipeline mode, since query results are not immediately
+ fetched in this mode.
+ </para>
+
<para>
The following example puts the final account balance from the first query
into variable <replaceable>abalance</replaceable>, and fills variables
@@ -1270,6 +1276,21 @@ SELECT 4 AS four \; SELECT 5 AS five \aset
</programlisting></para>
</listitem>
</varlistentry>
+
+ <varlistentry id='pgbench-metacommand-pipeline'>
+ <term><literal>\startpipeline</literal></term>
+ <term><literal>\endpipeline</literal></term>
+
+ <listitem>
+ <para>
+ These commands delimit the start and end of a pipeline of SQL statements.
+ In a pipeline, statements are sent to server without waiting for the results
+ of previous statements (see <xref linkend="libpq-pipeline-mode"/>).
+ Pipeline mode requires the extended query protocol.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect2>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 5272eed9ab..f74378110a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1019,6 +1019,12 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->err = _("empty query");
break;
+ case PGRES_PIPELINE_SYNC:
+ case PGRES_PIPELINE_ABORTED:
+ walres->status = WALRCV_ERROR;
+ walres->err = _("unexpected pipeline mode");
+ break;
+
case PGRES_NONFATAL_ERROR:
case PGRES_FATAL_ERROR:
case PGRES_BAD_RESPONSE:
diff --git a/src/bin/pg_amcheck/pg_amcheck.c b/src/bin/pg_amcheck/pg_amcheck.c
index 008a75d207..c9d9900693 100644
--- a/src/bin/pg_amcheck/pg_amcheck.c
+++ b/src/bin/pg_amcheck/pg_amcheck.c
@@ -929,6 +929,8 @@ should_processing_continue(PGresult *res)
case PGRES_COPY_IN:
case PGRES_COPY_BOTH:
case PGRES_SINGLE_TUPLE:
+ case PGRES_PIPELINE_SYNC:
+ case PGRES_PIPELINE_ABORTED:
return false;
}
return true;
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..5c48c14191 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,7 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterPipelineMode 180
+PQexitPipelineMode 181
+PQpipelineSync 182
+PQpipelineStatus 183
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 4e21057d0f..53b354abb2 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -522,6 +522,23 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * pqFreeCommandQueue
+ * Free all the entries of PGcmdQueueEntry queue passed.
+ */
+static void
+pqFreeCommandQueue(PGcmdQueueEntry *queue)
+{
+ while (queue != NULL)
+ {
+ PGcmdQueueEntry *cur = queue;
+
+ queue = cur->next;
+ if (cur->query)
+ free(cur->query);
+ free(cur);
+ }
+}
/*
* pqDropServerData
@@ -553,6 +570,12 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ pqFreeCommandQueue(conn->cmd_queue_head);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ pqFreeCommandQueue(conn->cmd_queue_recycle);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
@@ -2459,6 +2482,7 @@ keep_going: /* We will come back to here until there is
/* Drop any PGresult we might have, too */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn);
/* Reset conn->status to put the state machine in the right state */
@@ -3917,6 +3941,7 @@ makeEmptyPGconn(void)
conn->status = CONNECTION_BAD;
conn->asyncStatus = PGASYNC_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
conn->xactStatus = PQTRANS_IDLE;
conn->options_valid = false;
conn->nonblocking = false;
@@ -4084,8 +4109,6 @@ freePGconn(PGconn *conn)
if (conn->connip)
free(conn->connip);
/* Note that conn->Pfdebug is not ours to close or free */
- if (conn->last_query)
- free(conn->last_query);
if (conn->write_err_msg)
free(conn->write_err_msg);
if (conn->inBuffer)
@@ -4174,6 +4197,7 @@ closePGconn(PGconn *conn)
conn->status = CONNECTION_BAD; /* Well, not really _bad_ - just absent */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn); /* deallocate result */
resetPQExpBuffer(&conn->errorMessage);
release_conn_addrinfo(conn);
@@ -6726,6 +6750,15 @@ PQbackendPID(const PGconn *conn)
return conn->be_pid;
}
+PGpipelineStatus
+PQpipelineStatus(const PGconn *conn)
+{
+ if (!conn)
+ return PQ_PIPELINE_OFF;
+
+ return conn->pipelineStatus;
+}
+
int
PQconnectionNeedsPassword(const PGconn *conn)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9a038043b2..f3443708a6 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_PIPELINE_SYNC",
+ "PGRES_PIPELINE_ABORTED"
};
/*
@@ -71,6 +73,8 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static void pqPipelineProcessQueue(PGconn *conn);
+static int pqPipelineFlush(PGconn *conn);
/* ----------------
@@ -1171,7 +1175,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1184,6 +1188,87 @@ fail:
}
+/*
+ * pqAllocCmdQueueEntry
+ * Get a command queue entry for caller to fill.
+ *
+ * If the recycle queue has a free element, that is returned; if not, a
+ * fresh one is allocated. Caller is responsible for adding it to the
+ * command queue (pqAppendCmdQueueEntry) once the struct is filled in, or
+ * releasing the memory (pqRecycleCmdQueueEntry) if an error occurs.
+ *
+ * If allocation fails, sets the error message and returns NULL.
+ */
+static PGcmdQueueEntry *
+pqAllocCmdQueueEntry(PGconn *conn)
+{
+ PGcmdQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcmdQueueEntry *) malloc(sizeof(PGcmdQueueEntry));
+ if (entry == NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * pqAppendCmdQueueEntry
+ * Append a caller-allocated command queue entry to the queue.
+ *
+ * The query itself must already have been put in the output buffer by the
+ * caller.
+ */
+static void
+pqAppendCmdQueueEntry(PGconn *conn, PGcmdQueueEntry *entry)
+{
+ Assert(entry->next == NULL);
+
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * pqRecycleCmdQueueEntry
+ * Push a command queue entry onto the freelist.
+ */
+static void
+pqRecycleCmdQueueEntry(PGconn *conn, PGcmdQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+
+ /* recyclable entries should not have a follow-on command */
+ Assert(entry->next == NULL);
+
+ if (entry->query)
+ {
+ free(entry->query);
+ entry->query = NULL;
+ }
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+
/*
* PQsendQuery
* Submit a query, but don't wait for it to finish
@@ -1209,9 +1294,15 @@ PQsendQueryContinue(PGconn *conn, const char *query)
static int
PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
{
+ PGcmdQueueEntry *entry = NULL;
+
if (!PQsendQueryStart(conn, newQuery))
return 0;
+ entry = pqAllocCmdQueueEntry(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
/* check the argument */
if (!query)
{
@@ -1220,37 +1311,75 @@ PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
return 0;
}
- /* construct the outgoing Query message */
- if (pqPutMsgStart('Q', conn) < 0 ||
- pqPuts(query, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
+ /* Send the query message(s) */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
{
- /* error message should be set up already */
- return 0;
+ /* construct the outgoing Query message */
+ if (pqPutMsgStart('Q', conn) < 0 ||
+ pqPuts(query, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ {
+ /* error message should be set up already */
+ return 0;
+ }
+
+ /* remember we are using simple query protocol */
+ entry->queryclass = PGQUERY_SIMPLE;
+ /* and remember the query text too, if possible */
+ entry->query = strdup(query);
}
+ else
+ {
+ /*
+ * In pipeline mode we cannot use the simple protocol, so we send
+ * Parse, Bind, Describe Portal, Execute.
+ */
+ if (pqPutMsgStart('P', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPuts(query, conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ if (pqPutMsgStart('B', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ if (pqPutMsgStart('D', conn) < 0 ||
+ pqPutc('P', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ if (pqPutMsgStart('E', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPutInt(0, 4, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
- /* remember we are using simple query protocol */
- conn->queryclass = PGQUERY_SIMPLE;
-
- /* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ entry->queryclass = PGQUERY_EXTENDED;
+ entry->query = strdup(query);
+ }
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
- {
- /* error message should be set up already */
- return 0;
- }
+ if (pqPipelineFlush(conn) < 0)
+ goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ pqAppendCmdQueueEntry(conn, entry);
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
+
+sendFailed:
+ pqRecycleCmdQueueEntry(conn, entry);
+ /* error message should be set up already */
+ return 0;
}
/*
@@ -1307,6 +1436,8 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcmdQueueEntry *entry = NULL;
+
if (!PQsendQueryStart(conn, true))
return 0;
@@ -1330,6 +1461,10 @@ PQsendPrepare(PGconn *conn,
return 0;
}
+ entry = pqAllocCmdQueueEntry(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
/* construct the Parse message */
if (pqPutMsgStart('P', conn) < 0 ||
pqPuts(stmtName, conn) < 0 ||
@@ -1356,32 +1491,38 @@ PQsendPrepare(PGconn *conn,
if (pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /* Add a Sync, unless in pipeline mode. */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ entry->queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ /* if insufficient memory, query just winds up NULL */
+ entry->query = strdup(query);
+
+ pqAppendCmdQueueEntry(conn, entry);
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ conn->asyncStatus = PGASYNC_BUSY;
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
- /* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecycleCmdQueueEntry(conn, entry);
/* error message should be set up already */
return 0;
}
@@ -1429,7 +1570,8 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn, bool newQuery)
@@ -1450,20 +1592,57 @@ PQsendQueryStart(PGconn *conn, bool newQuery)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE &&
+ conn->pipelineStatus == PQ_PIPELINE_OFF)
{
appendPQExpBufferStr(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * When enqueuing commands we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when pqPipelineProcessQueue()
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ }
+ }
+ else
+ {
+ /*
+ * This command's results will come in immediately. Initialize async
+ * result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1487,10 +1666,16 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcmdQueueEntry *entry;
+
+ entry = pqAllocCmdQueueEntry(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
/*
- * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
- * using specified statement name and the unnamed portal.
+ * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync
+ * (if not in pipeline mode), using specified statement name and the
+ * unnamed portal.
*/
if (command)
@@ -1600,35 +1785,38 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /* construct the Sync message if not in pipeline mode */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ entry->queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ /* if insufficient memory, query just winds up NULL */
if (command)
- conn->last_query = strdup(command);
- else
- conn->last_query = NULL;
+ entry->query = strdup(command);
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ pqAppendCmdQueueEntry(conn, entry);
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecycleCmdQueueEntry(conn, entry);
/* error message should be set up already */
return 0;
}
@@ -1647,8 +1835,9 @@ PQsetSingleRowMode(PGconn *conn)
return 0;
if (conn->asyncStatus != PGASYNC_BUSY)
return 0;
- if (conn->queryclass != PGQUERY_SIMPLE &&
- conn->queryclass != PGQUERY_EXTENDED)
+ if (!conn->cmd_queue_head ||
+ (conn->cmd_queue_head->queryclass != PGQUERY_SIMPLE &&
+ conn->cmd_queue_head->queryclass != PGQUERY_EXTENDED))
return 0;
if (conn->result)
return 0;
@@ -1726,14 +1915,17 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
-
/*
* PQgetResult
* Get the next PGresult produced by a query. Returns NULL if no
* query work remains or an error has occurred (e.g. out of
* memory).
+ *
+ * In pipeline mode, once all the result of a query have been returned,
+ * PQgetResult returns NULL to let the user know that the next
+ * query is being processed. At the end of the pipeline, returns a
+ * result with PQresultStatus(result) == PGRES_PIPELINE_SYNC.
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1803,8 +1995,62 @@ PQgetResult(PGconn *conn)
{
case PGASYNC_IDLE:
res = NULL; /* query is complete */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to return the NULL that terminates the round of
+ * results from the current query; prepare to send the results
+ * of the next query when we're called next. Also, since this
+ * is the start of the results of the next query, clear any
+ * prior error message.
+ */
+ resetPQExpBuffer(&conn->errorMessage);
+ pqPipelineProcessQueue(conn);
+ }
break;
case PGASYNC_READY:
+
+ /*
+ * For any query type other than simple query protocol, we advance
+ * the command queue here. This is because for simple query
+ * protocol we can get the READY state multiple times before the
+ * command is actually complete, since the command string can
+ * contain many queries. In simple query protocol, the queue
+ * advance is done by fe-protocol3 when it receives ReadyForQuery.
+ */
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass != PGQUERY_SIMPLE)
+ pqCommandQueueAdvance(conn);
+ res = pqPrepareAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to send the results of the current query. Set
+ * us idle now, and ...
+ */
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /*
+ * ... in cases when we're sending a pipeline-sync result,
+ * move queue processing forwards immediately, so that next
+ * time we're called, we're prepared to return the next result
+ * received from the server. In all other cases, leave the
+ * queue state change for next time, so that a terminating
+ * NULL result is sent.
+ *
+ * (In other words: we don't return a NULL after a pipeline
+ * sync.)
+ */
+ if (res && res->resultStatus == PGRES_PIPELINE_SYNC)
+ pqPipelineProcessQueue(conn);
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -1985,6 +2231,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("synchronous command execution functions are not allowed in pipeline mode\n"));
+ return false;
+ }
+
/*
* Since this is the beginning of a query cycle, reset the error buffer.
*/
@@ -2148,6 +2401,8 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcmdQueueEntry *entry = NULL;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2155,6 +2410,10 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
if (!PQsendQueryStart(conn, true))
return 0;
+ entry = pqAllocCmdQueueEntry(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
/* construct the Describe message */
if (pqPutMsgStart('D', conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2163,32 +2422,32 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
-
- /* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
-
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
{
- free(conn->last_query);
- conn->last_query = NULL;
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
}
+ /* remember we are doing a Describe */
+ entry->queryclass = PGQUERY_DESCRIBE;
+
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ pqAppendCmdQueueEntry(conn, entry);
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecycleCmdQueueEntry(conn, entry);
/* error message should be set up already */
return 0;
}
@@ -2327,7 +2586,8 @@ PQputCopyEnd(PGconn *conn, const char *errormsg)
* If we sent the COPY command in extended-query mode, we must issue a
* Sync as well.
*/
- if (conn->queryclass != PGQUERY_SIMPLE)
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass != PGQUERY_SIMPLE)
{
if (pqPutMsgStart('S', conn) < 0 ||
pqPutMsgEnd(conn) < 0)
@@ -2541,6 +2801,13 @@ PQfn(PGconn *conn,
*/
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("PQfn not allowed in pipeline mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -2555,6 +2822,277 @@ PQfn(PGconn *conn,
args, nargs);
}
+/* ====== Pipeline mode support ======== */
+
+/*
+ * PQenterPipelineMode
+ * Put an idle connection in pipeline mode.
+ *
+ * Returns 1 on success. On failure, errorMessage is set and 0 is returned.
+ *
+ * Commands submitted after this can be pipelined on the connection;
+ * there's no requirement to wait for one to finish before the next is
+ * dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQpipelineSync. Multiple sync
+ * points can be established while in pipeline mode. Pipeline mode can
+ * be exited by calling PQexitPipelineMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ /* succeed with no action if already in pipeline mode */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot enter pipeline mode, connection not idle\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+
+ return 1;
+}
+
+/*
+ * PQexitPipelineMode
+ * End pipeline mode and return to normal command mode.
+ *
+ * Returns 1 in success (pipeline mode successfully ended, or not in pipeline
+ * mode).
+ *
+ * Returns 0 if in pipeline mode and cannot be ended yet. Error message will
+ * be set.
+ */
+int
+PQexitPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ /* there are some uncollected results */
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+
+ case PGASYNC_BUSY:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode while busy\n"));
+ return 0;
+
+ default:
+ /* OK */
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ return 0; /* error message is setup already */
+ return 1;
+}
+
+/*
+ * pqCommandQueueAdvance
+ * Remove one query from the command queue, when we receive
+ * all results from the server that pertain to it.
+ */
+void
+pqCommandQueueAdvance(PGconn *conn)
+{
+ PGcmdQueueEntry *prevquery;
+
+ if (conn->cmd_queue_head == NULL)
+ return;
+
+ /* delink from queue */
+ prevquery = conn->cmd_queue_head;
+ conn->cmd_queue_head = conn->cmd_queue_head->next;
+
+ /* and make it recyclable */
+ prevquery->next = NULL;
+ pqRecycleCmdQueueEntry(conn, prevquery);
+}
+
+/*
+ * pqPipelineProcessQueue: subroutine for PQgetResult
+ * In pipeline mode, start processing the results of the next query in the queue.
+ */
+void
+pqPipelineProcessQueue(PGconn *conn)
+{
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return;
+ case PGASYNC_IDLE:
+ /* next query please */
+ break;
+ }
+
+ /* Nothing to do if not in pipeline mode, or queue is empty */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF ||
+ conn->cmd_queue_head == NULL)
+ return;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /*
+ * Reset single-row processing mode. (Client has to set it up for each
+ * query, if desired.)
+ */
+ conn->singleRowMode = false;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_ABORTED &&
+ conn->cmd_queue_head->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted pipeline we don't get anything from the server for
+ * each result; we're just discarding commands from the queue until we
+ * get to the next sync from the server.
+ *
+ * The PGRES_PIPELINE_ABORTED results tell the client that its queries
+ * got aborted.
+ */
+ conn->result = PQmakeEmptyPGresult(conn, PGRES_PIPELINE_ABORTED);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ pqSaveErrorResult(conn);
+ return;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+}
+
+/*
+ * PQpipelineSync
+ * Send a Sync message as part of a pipeline, and flush to server
+ *
+ * It's legal to start submitting more commands in the pipeline immediately,
+ * without waiting for the results of the current pipeline. There's no need to
+ * end pipeline mode and start it again.
+ *
+ * If a command in a pipeline fails, every subsequent command up to and including
+ * the result to the Sync message sent by PQpipelineSync gets set to
+ * PGRES_PIPELINE_ABORTED state. If the whole pipeline is processed without
+ * error, a PGresult with PGRES_PIPELINE_SYNC is produced.
+ *
+ * Queries can already have been sent before PQpipelineSync is called, but
+ * PQpipelineSync need to be called before retrieving command results.
+ *
+ * The connection will remain in pipeline mode and unavailable for new
+ * synchronous command execution functions until all results from the pipeline
+ * are processed by the client.
+ */
+int
+PQpipelineSync(PGconn *conn)
+{
+ PGcmdQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot send pipeline when not in pipeline mode\n"));
+ return 0;
+ }
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: cannot send pipeline while in COPY\n");
+ return 0;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_IDLE:
+ /* OK to send sync */
+ break;
+ }
+
+ entry = pqAllocCmdQueueEntry(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ pqAppendCmdQueueEntry(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ /*
+ * Call pqPipelineProcessQueue so the user can call start calling
+ * PQgetResult.
+ */
+ pqPipelineProcessQueue(conn);
+
+ return 1;
+
+sendFailed:
+ pqRecycleCmdQueueEntry(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
+
/* ====== accessor funcs for PGresult ======== */
@@ -2569,7 +3107,7 @@ PQresultStatus(const PGresult *res)
char *
PQresStatus(ExecStatusType status)
{
- if ((unsigned int) status >= sizeof pgresStatus / sizeof pgresStatus[0])
+ if ((unsigned int) status >= lengthof(pgresStatus))
return libpq_gettext("invalid ExecStatusType code");
return pgresStatus[status];
}
@@ -3152,6 +3690,23 @@ PQflush(PGconn *conn)
return pqFlush(conn);
}
+/*
+ * pqPipelineFlush
+ *
+ * In pipeline mode, data will be flushed only when the out buffer reaches the
+ * threshold value. In non-pipeline mode, it behaves as stock pqFlush.
+ *
+ * Returns 0 on success.
+ */
+static int
+pqPipelineFlush(PGconn *conn)
+{
+ if ((conn->pipelineStatus != PQ_PIPELINE_ON) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return pqFlush(conn);
+ return 0;
+}
+
/*
* PQfreemem - safely frees memory allocated
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index eb55d528fb..306e89acfd 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -158,6 +158,18 @@ pqParseInput3(PGconn *conn)
if (conn->asyncStatus != PGASYNC_IDLE)
return;
+ /*
+ * We're also notionally not-IDLE when in pipeline mode the state
+ * says "idle" (so we have completed receiving the results of one
+ * query from the server and dispatched them to the application)
+ * but another query is queued; yield back control to caller so
+ * that they can initiate processing of the next query in the
+ * queue.
+ */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->cmd_queue_head != NULL)
+ return;
+
/*
* Unexpected message in IDLE state; need to recover somehow.
* ERROR messages are handled using the notice processor;
@@ -179,6 +191,7 @@ pqParseInput3(PGconn *conn)
}
else
{
+ /* Any other case is unexpected and we summarily skip it */
pqInternalNotice(&conn->noticeHooks,
"message type 0x%02x arrived from server while idle",
id);
@@ -217,10 +230,37 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new
+ * query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_PIPELINE_SYNC);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ else
+ {
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ }
+ else
+ {
+ /*
+ * In simple query protocol, advance the command queue
+ * (see PQgetResult).
+ */
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_SIMPLE)
+ pqCommandQueueAdvance(conn);
+ conn->asyncStatus = PGASYNC_IDLE;
+ }
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -238,7 +278,8 @@ pqParseInput3(PGconn *conn)
break;
case '1': /* Parse Complete */
/* If we're doing PQprepare, we're done; else ignore */
- if (conn->queryclass == PGQUERY_PREPARE)
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_PREPARE)
{
if (conn->result == NULL)
{
@@ -285,7 +326,8 @@ pqParseInput3(PGconn *conn)
conn->inCursor += msgLength;
}
else if (conn->result == NULL ||
- conn->queryclass == PGQUERY_DESCRIBE)
+ (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_DESCRIBE))
{
/* First 'T' in a query sequence */
if (getRowDescriptions(conn, msgLength))
@@ -316,7 +358,8 @@ pqParseInput3(PGconn *conn)
* instead of PGRES_TUPLES_OK. Otherwise we can just
* ignore this message.
*/
- if (conn->queryclass == PGQUERY_DESCRIBE)
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_DESCRIBE)
{
if (conn->result == NULL)
{
@@ -445,7 +488,7 @@ handleSyncLoss(PGconn *conn, char id, int msgLength)
id, msgLength);
/* build an error result holding the error message */
pqSaveErrorResult(conn);
- conn->asyncStatus = PGASYNC_READY; /* drop out of GetResult wait loop */
+ conn->asyncStatus = PGASYNC_READY; /* drop out of PQgetResult wait loop */
/* flush input data since we're giving up on processing it */
pqDropConnection(conn, true);
conn->status = CONNECTION_BAD; /* No more connection to backend */
@@ -471,7 +514,9 @@ getRowDescriptions(PGconn *conn, int msgLength)
* PGresult created by getParamDescriptions, and we should fill data into
* that. Otherwise, create a new, empty PGresult.
*/
- if (conn->queryclass == PGQUERY_DESCRIBE)
+ if (!conn->cmd_queue_head ||
+ (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_DESCRIBE))
{
if (conn->result)
result = conn->result;
@@ -568,7 +613,9 @@ getRowDescriptions(PGconn *conn, int msgLength)
* If we're doing a Describe, we're done, and ready to pass the result
* back to the client.
*/
- if (conn->queryclass == PGQUERY_DESCRIBE)
+ if ((!conn->cmd_queue_head) ||
+ (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_DESCRIBE))
{
conn->asyncStatus = PGASYNC_READY;
return 0;
@@ -841,6 +888,10 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ /* If in pipeline mode, set error indicator for it */
+ if (isError && conn->pipelineStatus != PQ_PIPELINE_OFF)
+ conn->pipelineStatus = PQ_PIPELINE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
@@ -897,8 +948,8 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
* might need it for an error cursor display, which is only true if there
* is a PG_DIAG_STATEMENT_POSITION field.
*/
- if (have_position && conn->last_query && res)
- res->errQuery = pqResultStrdup(res, conn->last_query);
+ if (have_position && res && conn->cmd_queue_head && conn->cmd_queue_head->query)
+ res->errQuery = pqResultStrdup(res, conn->cmd_queue_head->query);
/*
* Now build the "overall" error message for PQresultErrorMessage.
@@ -1817,7 +1868,8 @@ pqEndcopy3(PGconn *conn)
* If we sent the COPY command in extended-query mode, we must issue a
* Sync as well.
*/
- if (conn->queryclass != PGQUERY_SIMPLE)
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass != PGQUERY_SIMPLE)
{
if (pqPutMsgStart('S', conn) < 0 ||
pqPutMsgEnd(conn) < 0)
@@ -1897,6 +1949,9 @@ pqFunctionCall3(PGconn *conn, Oid fnid,
int avail;
int i;
+ /* already validated by PQfn */
+ Assert(conn->pipelineStatus == PQ_PIPELINE_OFF);
+
/* PQfn already validated connection state */
if (pqPutMsgStart('F', conn) < 0 || /* function call msg */
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index fa9b62a844..cee42d4843 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -96,7 +96,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_PIPELINE_SYNC, /* pipeline synchronization point */
+ PGRES_PIPELINE_ABORTED, /* Command didn't run because of an abort
+ * earlier in a pipeline */
} ExecStatusType;
typedef enum
@@ -136,6 +139,16 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PGpipelineStatus - Current status of pipeline mode
+ */
+typedef enum
+{
+ PQ_PIPELINE_OFF,
+ PQ_PIPELINE_ON,
+ PQ_PIPELINE_ABORTED
+} PGpipelineStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -327,6 +340,7 @@ extern int PQserverVersion(const PGconn *conn);
extern char *PQerrorMessage(const PGconn *conn);
extern int PQsocket(const PGconn *conn);
extern int PQbackendPID(const PGconn *conn);
+extern PGpipelineStatus PQpipelineStatus(const PGconn *conn);
extern int PQconnectionNeedsPassword(const PGconn *conn);
extern int PQconnectionUsedPassword(const PGconn *conn);
extern int PQclientEncoding(const PGconn *conn);
@@ -434,6 +448,11 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for pipeline mode management */
+extern int PQenterPipelineMode(PGconn *conn);
+extern int PQexitPipelineMode(PGconn *conn);
+extern int PQpipelineSync(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 2f052f61f8..6374ec657a 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,21 +217,16 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, more results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
} PGAsyncStatusType;
-/* PGQueryClass tracks which query protocol we are now executing */
-typedef enum
-{
- PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
- PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
- PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
-} PGQueryClass;
-
/* Target server type (decoded value of target_session_attrs) */
typedef enum
{
@@ -305,6 +300,29 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/*
+ * PGQueryClass tracks which query protocol is in use for each command queue
+ * entry, or special operation in execution
+ */
+typedef enum
+{
+ PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
+ PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
+ PGQUERY_PREPARE, /* Parse only (PQprepare) */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* Sync (at end of a pipeline) */
+} PGQueryClass;
+
+/*
+ * An entry in the pending command queue.
+ */
+typedef struct PGcmdQueueEntry
+{
+ PGQueryClass queryclass; /* Query type */
+ char *query; /* SQL command, or NULL if none/unknown/OOM */
+ struct PGcmdQueueEntry *next; /* list link */
+} PGcmdQueueEntry;
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -389,12 +407,11 @@ struct pg_conn
ConnStatusType status;
PGAsyncStatusType asyncStatus;
PGTransactionStatusType xactStatus; /* never changes to ACTIVE */
- PGQueryClass queryclass;
- char *last_query; /* last SQL command, or NULL if unknown */
char last_sqlstate[6]; /* last reported SQLSTATE */
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PGpipelineStatus pipelineStatus; /* status of pipeline mode */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -407,6 +424,19 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The pending command queue as a singly-linked list. Head is the command
+ * currently in execution, tail is where new commands are added.
+ */
+ PGcmdQueueEntry *cmd_queue_head;
+ PGcmdQueueEntry *cmd_queue_tail;
+
+ /*
+ * To save malloc traffic, we don't free entries right away; instead we
+ * save them in this list for possible reuse.
+ */
+ PGcmdQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -622,6 +652,7 @@ extern void pqSaveMessageField(PGresult *res, char code,
extern void pqSaveParameterStatus(PGconn *conn, const char *name,
const char *value);
extern int pqRowProcessor(PGconn *conn, const char **errmsgp);
+extern void pqCommandQueueAdvance(PGconn *conn);
extern int PQsendQueryContinue(PGconn *conn, const char *query);
/* === in fe-protocol3.c === */
@@ -795,6 +826,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold, for pipeline mode.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 5391f461a2..93e7829c67 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -10,6 +10,7 @@ SUBDIRS = \
delay_execution \
dummy_index_am \
dummy_seclabel \
+ libpq_pipeline \
plsample \
snapshot_too_old \
test_bloomfilter \
diff --git a/src/test/modules/libpq_pipeline/.gitignore b/src/test/modules/libpq_pipeline/.gitignore
new file mode 100644
index 0000000000..3a11e786b8
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/libpq_pipeline
diff --git a/src/test/modules/libpq_pipeline/Makefile b/src/test/modules/libpq_pipeline/Makefile
new file mode 100644
index 0000000000..b798f5fbbc
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/Makefile
@@ -0,0 +1,20 @@
+# src/test/modules/libpq_pipeline/Makefile
+
+PROGRAM = libpq_pipeline
+OBJS = libpq_pipeline.o
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS_INTERNAL += $(libpq_pgport)
+
+TAP_TESTS = 1
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/libpq_pipeline
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/libpq_pipeline/README b/src/test/modules/libpq_pipeline/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/libpq_pipeline/libpq_pipeline.c b/src/test/modules/libpq_pipeline/libpq_pipeline.c
new file mode 100644
index 0000000000..03eb3df504
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/libpq_pipeline.c
@@ -0,0 +1,1303 @@
+/*-------------------------------------------------------------------------
+ *
+ * libpq_pipeline.c
+ * Verify libpq pipeline execution functionality
+ *
+ * Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/test/modules/libpq_pipeline/libpq_pipeline.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include <sys/time.h>
+#ifdef HAVE_SYS_SELECT_H
+#include <sys/select.h>
+#endif
+
+#include "catalog/pg_type_d.h"
+#include "common/fe_memutils.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+
+static void exit_nicely(PGconn *conn);
+
+const char *const progname = "libpq_pipeline";
+
+
+#define DEBUG
+#ifdef DEBUG
+#define pg_debug(...) do { fprintf(stderr, __VA_ARGS__); } while (0)
+#else
+#define pg_debug(...)
+#endif
+
+static const char *const drop_table_sql =
+"DROP TABLE IF EXISTS pq_pipeline_demo";
+static const char *const create_table_sql =
+"CREATE UNLOGGED TABLE pq_pipeline_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql =
+"INSERT INTO pq_pipeline_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+/*
+ * Print an error to stderr and terminate the program.
+ */
+#define pg_fatal(...) pg_fatal_impl(__LINE__, __VA_ARGS__)
+static void
+pg_fatal_impl(int line, const char *fmt,...)
+{
+ va_list args;
+
+
+ fflush(stdout);
+
+ fprintf(stderr, "\n%s:%d: ", progname, line);
+ va_start(args, fmt);
+ vfprintf(stderr, fmt, args);
+ va_end(args);
+ Assert(fmt[strlen(fmt) - 1] != '\n');
+ fprintf(stderr, "\n");
+ exit(1);
+}
+
+static void
+test_disallowed_in_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("Unable to enter pipeline mode");
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not activated properly");
+
+ /* PQexec should fail in pipeline mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("PQexec should fail in pipeline mode but succeeded");
+
+ /* Entering pipeline mode when already in pipeline mode is OK */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("re-entering pipeline mode should be a no-op but failed");
+
+ if (PQisBusy(conn) != 0)
+ pg_fatal("PQisBusy should return 0 when idle in pipeline mode, returned 1");
+
+ /* ok, back to normal command mode */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("couldn't exit idle empty pipeline mode");
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not terminated properly");
+
+ /* exiting pipeline mode when not in pipeline mode should be a no-op */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("pipeline mode exit when not in pipeline mode should succeed but failed");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("PQexec should succeed after exiting pipeline mode but failed with: %s",
+ PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_multi_pipelines(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi pipeline... ");
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first SELECT failed: %s", PQerrorMessage(conn));
+
+ if (PQpipelineSync(conn) != 1)
+ pg_fatal("Pipeline sync failed: %s", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second SELECT failed: %s", PQerrorMessage(conn));
+
+ if (PQpipelineSync(conn) != 1)
+ pg_fatal("pipeline sync failed: %s", PQerrorMessage(conn));
+
+ /* OK, start processing the results */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first result");
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result expected: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of sync result, error: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ PQclear(res);
+
+ /* second pipeline */
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from second pipeline item",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode ... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work");
+
+ fprintf(stderr, "ok\n");
+}
+
+/*
+ * When an operation in a pipeline fails the rest of the pipeline is flushed. We
+ * still have to get results for each pipeline item, but the item will just be
+ * a PGRES_PIPELINE_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the pipeline. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_pipeline_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+ bool goterror;
+
+ fprintf(stderr, "aborted pipeline... ");
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching DROP TABLE failed: %s", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching CREATE TABLE failed: %s", PQerrorMessage(conn));
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * pipeline ERRORs.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first insert failed: %s", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT no_such_function($1)",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching error select failed: %s", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second insert failed: %s", PQerrorMessage(conn));
+
+ if (PQpipelineSync(conn) != 1)
+ pg_fatal("pipeline sync failed: %s", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second-pipeline insert failed: %s",
+ PQerrorMessage(conn));
+
+ if (PQpipelineSync(conn) != 1)
+ pg_fatal("pipeline sync failed: %s", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the pipeline results.
+ *
+ * We should get a command-ok for the first query, then a fatal error and
+ * a pipeline aborted message for the second insert, a pipeline-end, then
+ * a command-ok and a pipeline-ok for the second pipeline operation.
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result status %s: %s",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s",
+ PQresStatus(PQresultStatus(res)));
+
+ /* Second query caused error, so we expect an error next */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("Unexpected result code -- expected PGRES_FATAL_ERROR, got %s",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s",
+ PQresStatus(PQresultStatus(res)));
+
+ /*
+ * pipeline should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third pipeline since we've already sent a
+ * second. The aborted flag relates only to the pipeline being received.
+ */
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't");
+
+ /* third query in pipeline, the second insert */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_ABORTED)
+ pg_fatal("Unexpected result code -- expected PGRES_PIPELINE_ABORTED, got %s",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s", PQresStatus(PQresultStatus(res)));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't");
+
+ /* Ensure we're still in pipeline */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /*
+ * The end of a failed pipeline is a PGRES_PIPELINE_SYNC.
+ *
+ * (This is so clients know to start processing results normally again and
+ * can tell the difference between skipped commands and the sync.)
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code from first pipeline sync\n"
+ "Expected PGRES_PIPELINE_SYNC, got %s",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_ABORTED)
+ pg_fatal("sync should've cleared the aborted flag but didn't");
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /* the insert from the second pipeline */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result code %s from first item in second pipeline",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* Read the NULL result at the end of the command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s", PQresStatus(PQresultStatus(res)));
+
+ /* the second pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s: %s",
+ PQresStatus(PQresultStatus(res)),
+ PQerrorMessage(conn));
+
+ /* Try to send two queries in one command */
+ if (PQsendQuery(conn, "SELECT 1; SELECT 2") != 1)
+ pg_fatal("failed to send query: %s", PQerrorMessage(conn));
+ if (PQpipelineSync(conn) != 1)
+ pg_fatal("pipeline sync failed: %s", PQerrorMessage(conn));
+ goterror = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_FATAL_ERROR:
+ if (strcmp(PQresultErrorField(res, PG_DIAG_SQLSTATE), "42601") != 0)
+ pg_fatal("expected error about multiple commands, got %s",
+ PQerrorMessage(conn));
+ printf("got expected %s", PQerrorMessage(conn));
+ goterror = true;
+ break;
+ default:
+ pg_fatal("got unexpected status %s", PQresStatus(PQresultStatus(res)));
+ break;
+ }
+ }
+ if (!goterror)
+ pg_fatal("did not get cannot-insert-multiple-commands error");
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("got NULL result");
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from pipeline sync",
+ PQresStatus(PQresultStatus(res)));
+
+ /* Test single-row mode with an error partways */
+ if (PQsendQuery(conn, "SELECT 1.0/g FROM generate_series(3, -1, -1) g") != 1)
+ pg_fatal("failed to send query: %s", PQerrorMessage(conn));
+ if (PQpipelineSync(conn) != 1)
+ pg_fatal("pipeline sync failed: %s", PQerrorMessage(conn));
+ PQsetSingleRowMode(conn);
+ goterror = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_SINGLE_TUPLE:
+ printf("got row: %s\n", PQgetvalue(res, 0, 0));
+ break;
+ case PGRES_FATAL_ERROR:
+ if (strcmp(PQresultErrorField(res, PG_DIAG_SQLSTATE), "22012") != 0)
+ pg_fatal("expected division-by-zero, got: %s (%s)",
+ PQerrorMessage(conn),
+ PQresultErrorField(res, PG_DIAG_SQLSTATE));
+ printf("got expected division-by-zero\n");
+ goterror = true;
+ break;
+ default:
+ pg_fatal("got unexpected result %s", PQresStatus(PQresultStatus(res)));
+ }
+ PQclear(res);
+ }
+ if (!goterror)
+ pg_fatal("did not get division-by-zero error");
+ /* the third pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from third pipeline sync",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work");
+
+ fprintf(stderr, "ok\n");
+
+ /*-
+ * Since we fired the pipelines off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st pipeline
+ * - First insert applied
+ * - Second statement aborted xact
+ * - Third insert skipped
+ * - Sync rolled back first implicit xact
+ * - Implicit xact created by server around 2nd pipeline
+ * - insert applied from 2nd pipeline
+ * - Sync commits 2nd xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM pq_pipeline_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("expected 1 result, got %d", PQntuples(res));
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ pg_fatal("expected only insert with value 3, got %s", val);
+ }
+
+ PQclear(res);
+}
+
+/* State machine enum for test_pipelined_insert */
+enum PipelineInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+};
+
+static void
+test_pipelined_insert(PGconn *conn, int n_rows)
+{
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ enum PipelineInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a pipelined insert into a table created at the start of the pipeline
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s", PQerrorMessage(conn));
+
+ while (send_step != BI_PREPARE)
+ {
+ const char *sql;
+
+ switch (send_step)
+ {
+ case BI_BEGIN_TX:
+ sql = "BEGIN TRANSACTION";
+ send_step = BI_DROP_TABLE;
+ break;
+
+ case BI_DROP_TABLE:
+ sql = drop_table_sql;
+ send_step = BI_CREATE_TABLE;
+ break;
+
+ case BI_CREATE_TABLE:
+ sql = create_table_sql;
+ send_step = BI_PREPARE;
+ break;
+
+ default:
+ pg_fatal("invalid state");
+ }
+
+ pg_debug("sending: %s\n", sql);
+ if (PQsendQueryParams(conn, sql,
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching %s failed: %s", sql, PQerrorMessage(conn));
+ }
+
+ Assert(send_step == BI_PREPARE);
+ pg_debug("sending: %s\n", insert_sql);
+ if (PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids) != 1)
+ pg_fatal("dispatching PREPARE failed: %s", PQerrorMessage(conn));
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our output buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the pipeline before processing
+ * results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ pg_fatal("failed to set nonblocking mode: %s", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's output buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ PGresult *res;
+ const char *cmdtag;
+ const char *description = "";
+ int status;
+
+ /*
+ * Read next result. If no more results from this query,
+ * advance to the next query
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ continue;
+
+ status = PGRES_COMMAND_OK;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ recv_step++;
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ recv_step++;
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ recv_step++;
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ recv_step++;
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive == 0)
+ recv_step++;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ recv_step++;
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_PIPELINE_SYNC;
+ recv_step++;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ description = "";
+ abort();
+ }
+
+ if (PQresultStatus(res) != status)
+ pg_fatal("%s reported status %s, expected %s\n"
+ "Error message: \"%s\"",
+ description, PQresStatus(PQresultStatus(res)),
+ PQresStatus(status), PQerrorMessage(conn));
+
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ pg_fatal("%s expected command tag '%s', got '%s'",
+ description, cmdtag, PQcmdStatus(res));
+
+ pg_debug("Got %s OK\n", cmdtag[0] != '\0' ? cmdtag : description);
+
+ PQclear(res);
+ }
+ }
+
+ /* Write more rows and/or the end pipeline message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step++;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent COMMIT\n");
+ send_step++;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQpipelineSync(conn) == 1)
+ {
+ fprintf(stdout, "pipeline sync sent\n");
+ send_step++;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: pipeline sync failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+ }
+
+ /* We've got the sync message and the pipeline should be done */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s",
+ PQerrorMessage(conn));
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ pg_fatal("failed to clear nonblocking mode: %s", PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_prepared(PGconn *conn)
+{
+ PGresult *res = NULL;
+ Oid param_oids[1] = {INT4OID};
+ Oid expected_oids[4];
+ Oid typ;
+
+ fprintf(stderr, "prepared... ");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s", PQerrorMessage(conn));
+ if (PQsendPrepare(conn, "select_one", "SELECT $1, '42', $1::numeric, "
+ "interval '1 sec'",
+ 1, param_oids) != 1)
+ pg_fatal("preparing query failed: %s", PQerrorMessage(conn));
+ expected_oids[0] = INT4OID;
+ expected_oids[1] = TEXTOID;
+ expected_oids[2] = NUMERICOID;
+ expected_oids[3] = INTERVALOID;
+ if (PQsendDescribePrepared(conn, "select_one") != 1)
+ pg_fatal("failed to send describePrepared: %s", PQerrorMessage(conn));
+ if (PQpipelineSync(conn) != 1)
+ pg_fatal("pipeline sync failed: %s", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("expected COMMAND_OK, got %s", PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("expected NULL result");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned NULL");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("expected COMMAND_OK, got %s", PQresStatus(PQresultStatus(res)));
+ if (PQnfields(res) != lengthof(expected_oids))
+ pg_fatal("expected %d columns, got %d",
+ lengthof(expected_oids), PQnfields(res));
+ for (int i = 0; i < PQnfields(res); i++)
+ {
+ typ = PQftype(res, i);
+ if (typ != expected_oids[i])
+ pg_fatal("field %d: expected type %u, got %u",
+ i, expected_oids[i], typ);
+ }
+ PQclear(res);
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("expected NULL result");
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("expected PGRES_PIPELINE_SYNC, got %s", PQresStatus(PQresultStatus(res)));
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("could not exit pipeline mode: %s", PQerrorMessage(conn));
+
+ PQexec(conn, "BEGIN");
+ PQexec(conn, "DECLARE cursor_one CURSOR FOR SELECT 1");
+ PQenterPipelineMode(conn);
+ if (PQsendDescribePortal(conn, "cursor_one") != 1)
+ pg_fatal("PQsendDescribePortal failed: %s", PQerrorMessage(conn));
+ if (PQpipelineSync(conn) != 1)
+ pg_fatal("pipeline sync failed: %s", PQerrorMessage(conn));
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("expected NULL result");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("expected COMMAND_OK, got %s", PQresStatus(PQresultStatus(res)));
+
+ typ = PQftype(res, 0);
+ if (typ != INT4OID)
+ pg_fatal("portal: expected type %u, got %u",
+ INT4OID, typ);
+ PQclear(res);
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("expected NULL result");
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("expected PGRES_PIPELINE_SYNC, got %s", PQresStatus(PQresultStatus(res)));
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("could not exit pipeline mode: %s", PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_simple_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple pipeline... ");
+
+ /*
+ * Enter pipeline mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our output buffer will give us enough slush
+ * to work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching SELECT failed: %s", PQerrorMessage(conn));
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode with work in progress should fail, but succeeded");
+
+ if (PQpipelineSync(conn) != 1)
+ pg_fatal("pipeline sync failed: %s", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item",
+ PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first query result.");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit pipeline mode yet
+ */
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result PGRES_PIPELINE_SYNC expected: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of PGRES_PIPELINE_SYNC, error: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after pipeline end: %s",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /* ... until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Exiting pipeline mode didn't seem to work");
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i;
+ bool pipeline_ended = false;
+
+ /* 1 pipeline, 3 queries in it */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s",
+ PQerrorMessage(conn));
+
+ for (i = 0; i < 3; i++)
+ {
+ char *param[1];
+
+ param[0] = psprintf("%d", 44 + i);
+
+ if (PQsendQueryParams(conn,
+ "SELECT generate_series(42, $1)",
+ 1,
+ NULL,
+ (const char **) param,
+ NULL,
+ NULL,
+ 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ pfree(param[0]);
+ }
+ if (PQpipelineSync(conn) != 1)
+ pg_fatal("pipeline sync failed: %s", PQerrorMessage(conn));
+
+ for (i = 0; !pipeline_ended; i++)
+ {
+ bool first = true;
+ bool saw_ending_tuplesok;
+ bool isSingleTuple = false;
+
+ /* Set single row mode for only first 2 SELECT queries */
+ if (i < 2)
+ {
+ if (PQsetSingleRowMode(conn) != 1)
+ pg_fatal("PQsetSingleRowMode() failed for i=%d", i);
+ }
+
+ /* Consume rows for this query */
+ saw_ending_tuplesok = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+
+ if (est == PGRES_PIPELINE_SYNC)
+ {
+ fprintf(stderr, "end of pipeline reached\n");
+ pipeline_ended = true;
+ PQclear(res);
+ if (i != 3)
+ pg_fatal("Expected three results, got %d", i);
+ break;
+ }
+
+ /* Expect SINGLE_TUPLE for queries 0 and 1, TUPLES_OK for 2 */
+ if (first)
+ {
+ if (i <= 1 && est != PGRES_SINGLE_TUPLE)
+ pg_fatal("Expected PGRES_SINGLE_TUPLE for query %d, got %s",
+ i, PQresStatus(est));
+ if (i >= 2 && est != PGRES_TUPLES_OK)
+ pg_fatal("Expected PGRES_TUPLES_OK for query %d, got %s",
+ i, PQresStatus(est));
+ first = false;
+ }
+
+ fprintf(stderr, "Result status %s for query %d", PQresStatus(est), i);
+ switch (est)
+ {
+ case PGRES_TUPLES_OK:
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ saw_ending_tuplesok = true;
+ if (isSingleTuple)
+ {
+ if (PQntuples(res) == 0)
+ fprintf(stderr, "all tuples received in query %d\n", i);
+ else
+ pg_fatal("Expected to follow PGRES_SINGLE_TUPLE, but received PGRES_TUPLES_OK directly instead");
+ }
+ break;
+
+ case PGRES_SINGLE_TUPLE:
+ isSingleTuple = true;
+ fprintf(stderr, ", %d tuple: %s\n", PQntuples(res), PQgetvalue(res, 0, 0));
+ break;
+
+ default:
+ pg_fatal("unexpected");
+ }
+ PQclear(res);
+ }
+ if (!pipeline_ended && !saw_ending_tuplesok)
+ pg_fatal("didn't get expected terminating TUPLES_OK");
+ }
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s", PQerrorMessage(conn));
+}
+
+/*
+ * Simple test to verify that a pipeline is discarded as a whole when there's
+ * an error, ignoring transaction commands.
+ */
+static void
+test_transaction(PGconn *conn)
+{
+ PGresult *res;
+ bool expect_null;
+ int num_syncs = 0;
+
+ res = PQexec(conn, "DROP TABLE IF EXISTS pq_pipeline_tst;"
+ "CREATE TABLE pq_pipeline_tst (id int)");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to create test table: %s",
+ PQerrorMessage(conn));
+ PQclear(res);
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s",
+ PQerrorMessage(conn));
+ if (PQsendPrepare(conn, "rollback", "ROLLBACK", 0, NULL) != 1)
+ pg_fatal("could not send prepare on pipeline: %s",
+ PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn,
+ "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ if (PQsendQueryParams(conn,
+ "SELECT 0/0",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+
+ /*
+ * send a ROLLBACK using a prepared stmt. Doesn't work because we need to
+ * get out of the pipeline-aborted state first.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /* This insert fails because we're in pipeline-aborted state */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (1)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ if (PQpipelineSync(conn) != 1)
+ pg_fatal("pipeline sync failed: %s", PQerrorMessage(conn));
+ num_syncs++;
+
+ /*
+ * This insert fails even though the pipeline got a SYNC, because we're in
+ * an aborted transaction
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (2)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ if (PQpipelineSync(conn) != 1)
+ pg_fatal("pipeline sync failed: %s", PQerrorMessage(conn));
+ num_syncs++;
+
+ /*
+ * Send ROLLBACK using prepared stmt. This one works because we just did
+ * PQpipelineSync above.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /*
+ * Now that we're out of a transaction and in pipeline-good mode, this
+ * insert works
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (3)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ /* Send two syncs now -- match up to SYNC messages below */
+ if (PQpipelineSync(conn) != 1)
+ pg_fatal("pipeline sync failed: %s", PQerrorMessage(conn));
+ num_syncs++;
+ if (PQpipelineSync(conn) != 1)
+ pg_fatal("pipeline sync failed: %s", PQerrorMessage(conn));
+ num_syncs++;
+
+ expect_null = false;
+ for (int i = 0;; i++)
+ {
+ ExecStatusType restype;
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ printf("%d: got NULL result\n", i);
+ if (!expect_null)
+ pg_fatal("did not expect NULL here");
+ expect_null = false;
+ continue;
+ }
+ restype = PQresultStatus(res);
+ printf("%d: got status %s", i, PQresStatus(restype));
+ if (expect_null)
+ pg_fatal("expected NULL");
+ if (restype == PGRES_FATAL_ERROR)
+ printf("; error: %s", PQerrorMessage(conn));
+ else if (restype == PGRES_PIPELINE_ABORTED)
+ {
+ printf(": command didn't run because pipeline aborted\n");
+ }
+ else
+ printf("\n");
+ PQclear(res);
+
+ if (restype == PGRES_PIPELINE_SYNC)
+ num_syncs--;
+ else
+ expect_null = true;
+ if (num_syncs <= 0)
+ break;
+ }
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("returned something extra after all the syncs: %s",
+ PQresStatus(PQresultStatus(res)));
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s", PQerrorMessage(conn));
+
+ /* We expect to find one tuple containing the value "3" */
+ res = PQexec(conn, "SELECT * FROM pq_pipeline_tst");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("failed to obtain result: %s", PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("did not get 1 tuple");
+ if (strcmp(PQgetvalue(res, 0, 0), "3") != 0)
+ pg_fatal("did not get expected tuple");
+ PQclear(res);
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+usage(const char *progname)
+{
+ fprintf(stderr, "%s tests libpq's pipeline mode.\n\n", progname);
+ fprintf(stderr, "Usage:\n");
+ fprintf(stderr, " %s tests", progname);
+ fprintf(stderr, " %s testname [conninfo [number_of_rows]]\n", progname);
+}
+
+static void
+print_test_list(void)
+{
+ printf("disallowed_in_pipeline\n");
+ printf("multi_pipelines\n");
+ printf("pipeline_abort\n");
+ printf("pipelined_insert\n");
+ printf("prepared\n");
+ printf("simple_pipeline\n");
+ printf("singlerow\n");
+ printf("transaction\n");
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo = "";
+ PGconn *conn;
+ int numrows = 10000;
+ PGresult *res;
+
+ if (strcmp(argv[1], "tests") == 0)
+ {
+ print_test_list();
+ exit(0);
+ }
+
+ /*
+ * The testname parameter is mandatory; it can be followed by a conninfo
+ * string and number of rows.
+ */
+ if (argc < 2 || argc > 4)
+ {
+ usage(argv[0]);
+ exit(1);
+ }
+
+ if (argc >= 3)
+ conninfo = pg_strdup(argv[2]);
+
+ if (argc >= 4)
+ {
+ errno = 0;
+ numrows = strtol(argv[3], NULL, 10);
+ if (errno != 0 || numrows <= 0)
+ {
+ fprintf(stderr, "couldn't parse \"%s\" as a positive integer\n", argv[3]);
+ exit(1);
+ }
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+ res = PQexec(conn, "SET lc_messages TO \"C\"");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to set lc_messages: %s", PQerrorMessage(conn));
+
+ if (strcmp(argv[1], "disallowed_in_pipeline") == 0)
+ test_disallowed_in_pipeline(conn);
+ else if (strcmp(argv[1], "multi_pipelines") == 0)
+ test_multi_pipelines(conn);
+ else if (strcmp(argv[1], "pipeline_abort") == 0)
+ test_pipeline_abort(conn);
+ else if (strcmp(argv[1], "pipelined_insert") == 0)
+ test_pipelined_insert(conn, numrows);
+ else if (strcmp(argv[1], "prepared") == 0)
+ test_prepared(conn);
+ else if (strcmp(argv[1], "simple_pipeline") == 0)
+ test_simple_pipeline(conn);
+ else if (strcmp(argv[1], "singlerow") == 0)
+ test_singlerowmode(conn);
+ else if (strcmp(argv[1], "transaction") == 0)
+ test_transaction(conn);
+ else
+ {
+ fprintf(stderr, "\"%s\" is not a recognized test name\n", argv[1]);
+ usage(argv[0]);
+ exit(1);
+ }
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+ return 0;
+}
diff --git a/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
new file mode 100644
index 0000000000..ba15b64ca7
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
@@ -0,0 +1,28 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 8;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $numrows = 10000;
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+
+my ($out, $err) = run_command(['libpq_pipeline', 'tests']);
+die "oops: $err" unless $err eq '';
+my @tests = split(/\s/, $out);
+
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'libpq_pipeline', $testname, $node->connstr('postgres'), $numrows ],
+ "libpq_pipeline $testname");
+}
+
+$node->stop('fast');
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 74fde40e3a..a184404e21 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -33,10 +33,11 @@ my @unlink_on_exit;
# Set of variables for modules in contrib/ and src/test/modules/
my $contrib_defines = { 'refint' => 'REFINT_VERBOSE' };
-my @contrib_uselibpq = ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo');
-my @contrib_uselibpgport = ('oid2name', 'vacuumlo');
-my @contrib_uselibpgcommon = ('oid2name', 'vacuumlo');
-my $contrib_extralibs = undef;
+my @contrib_uselibpq =
+ ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo', 'libpq_pipeline');
+my @contrib_uselibpgport = ('libpq_pipeline', 'oid2name', 'vacuumlo');
+my @contrib_uselibpgcommon = ('libpq_pipeline', 'oid2name', 'vacuumlo');
+my $contrib_extralibs = { 'libpq_pipeline' => ['ws2_32.lib'] };
my $contrib_extraincludes = { 'dblink' => ['src/backend'] };
my $contrib_extrasource = {
'cube' => [ 'contrib/cube/cubescan.l', 'contrib/cube/cubeparse.y' ],
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 61cf4eae1f..9e6777e9d0 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1563,10 +1563,12 @@ PG_Locale_Strategy
PG_Lock_Status
PG_init_t
PGcancel
+PGcmdQueueEntry
PGconn
PGdataValue
PGlobjfuncs
PGnotify
+PGpipelineStatus
PGresAttDesc
PGresAttValue
PGresParamDesc
--
2.20.1
v37-0002-Add-libpq-pipeline-mode-support-to-pgbench.patchtext/x-diff; charset=iso-8859-1Download
From 2c50bf03d46fe2767c271dafb22c20e8ead5e510 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Mon, 15 Mar 2021 15:07:59 -0300
Subject: [PATCH v37 2/2] Add libpq pipeline mode support to pgbench
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Author: Daniel Vérité <daniel@manitou-mail.org>
Discussion: https://postgr.es/m/b4e34135-2bd9-4b8a-94ca-27d760da26d7@manitou-mail.org
---
src/bin/pgbench/pgbench.c | 128 +++++++++++++++++--
src/bin/pgbench/t/001_pgbench_with_server.pl | 61 +++++++++
2 files changed, 176 insertions(+), 13 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index f6a214669c..ba7b35d83c 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -395,10 +395,11 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
- * and we enter the CSTATE_SLEEP state to wait for it to expire. Other
- * meta-commands are executed immediately. If the command about to start
- * is actually beyond the end of the script, advance to CSTATE_END_TX.
+ * CSTATE_WAIT_RESULT state unless in pipeline mode. On a \sleep
+ * meta-command, the timer is set, and we enter the CSTATE_SLEEP state to
+ * wait for it to expire. Other meta-commands are executed immediately. If
+ * the command about to start is actually beyond the end of the script,
+ * advance to CSTATE_END_TX.
*
* CSTATE_WAIT_RESULT waits until we get a result set back from the server
* for the current command.
@@ -530,7 +531,9 @@ typedef enum MetaCommand
META_IF, /* \if */
META_ELIF, /* \elif */
META_ELSE, /* \else */
- META_ENDIF /* \endif */
+ META_ENDIF, /* \endif */
+ META_STARTPIPELINE, /* \startpipeline */
+ META_ENDPIPELINE /* \endpipeline */
} MetaCommand;
typedef enum QueryMode
@@ -2568,6 +2571,10 @@ getMetaCommand(const char *cmd)
mc = META_GSET;
else if (pg_strcasecmp(cmd, "aset") == 0)
mc = META_ASET;
+ else if (pg_strcasecmp(cmd, "startpipeline") == 0)
+ mc = META_STARTPIPELINE;
+ else if (pg_strcasecmp(cmd, "endpipeline") == 0)
+ mc = META_ENDPIPELINE;
else
mc = META_NONE;
return mc;
@@ -2757,11 +2764,25 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In pipeline mode, we use asynchronous functions. If a
+ * server-side error occurs, it will be processed later
+ * among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ pg_log_error("%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2805,8 +2826,10 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
* varprefix should be set only with \gset or \aset, and SQL commands do
* not need it.
*/
+#if 0
Assert((meta == META_NONE && varprefix == NULL) ||
((meta == META_GSET || meta == META_ASET) && varprefix != NULL));
+#endif
res = PQgetResult(st->con);
@@ -2874,6 +2897,13 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* otherwise the result is simply thrown away by PQclear below */
break;
+ case PGRES_PIPELINE_SYNC:
+ pg_log_debug("client %d pipeline ending", st->id);
+ if (PQexitPipelineMode(st->con) != 1)
+ pg_log_error("client %d failed to exit pipeline mode: %s", st->id,
+ PQerrorMessage(st->con));
+ break;
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3127,13 +3157,36 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
/* Execute the command */
if (command->type == SQL_COMMAND)
{
+ /* disallow \aset and \gset in pipeline mode */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ if (command->meta == META_GSET)
+ {
+ commandFailed(st, "gset", "\\gset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ else if (command->meta == META_ASET)
+ {
+ commandFailed(st, "aset", "\\aset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ }
+
if (!sendCommand(st, command))
{
commandFailed(st, "SQL", "SQL command send failed");
st->state = CSTATE_ABORTED;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in pipeline mode */
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -3273,7 +3326,15 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (readCommandResponse(st,
sql_script[st->use_file].commands[st->command]->meta,
sql_script[st->use_file].commands[st->command]->varprefix))
- st->state = CSTATE_END_COMMAND;
+ {
+ /*
+ * outside of pipeline mode: stop reading results.
+ * pipeline mode: continue reading results until an
+ * end-of-pipeline response.
+ */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ st->state = CSTATE_END_COMMAND;
+ }
else
st->state = CSTATE_ABORTED;
break;
@@ -3516,6 +3577,45 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
return CSTATE_ABORTED;
}
}
+ else if (command->meta == META_STARTPIPELINE)
+ {
+ /*
+ * In pipeline mode, we use a workflow based on libpq pipeline
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "startpipeline", "cannot use pipeline mode with the simple query protocol");
+ return CSTATE_ABORTED;
+ }
+
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ commandFailed(st, "startpipeline", "already in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (PQenterPipelineMode(st->con) == 0)
+ {
+ commandFailed(st, "startpipeline", "failed to enter pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ }
+ else if (command->meta == META_ENDPIPELINE)
+ {
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ {
+ commandFailed(st, "endpipeline", "not in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (!PQpipelineSync(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to send a pipeline sync");
+ return CSTATE_ABORTED;
+ }
+ /* Now wait for the PGRES_PIPELINE_SYNC and exit pipeline mode there */
+ /* collect pending results before getting out of pipeline mode */
+ return CSTATE_WAIT_RESULT;
+ }
/*
* executing the expression or shell command might have taken a
@@ -4725,7 +4825,9 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
- else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
+ else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF ||
+ my_command->meta == META_STARTPIPELINE ||
+ my_command->meta == META_ENDPIPELINE)
{
if (my_command->argc != 1)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index daffc18e52..d07b36faa7 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -755,6 +755,67 @@ pgbench(
}
});
+# Working \startpipeline
+pgbench(
+ '-t 1 -n -M extended',
+ 0,
+ [ qr{type: .*/001_pgbench_pipeline}, qr{processed: 1/1} ],
+ [],
+ 'pgbench startpipeline command',
+ {
+ '001_pgbench_pipeline' => q{
+-- test startpipeline
+\startpipeline
+} . "select 1;\n" x 10 . q{
+\endpipeline
+}
+ });
+
+# Try \startpipeline twice
+pgbench(
+ '-t 1 -n -M extended',
+ 2,
+ [],
+ [qr{already in pipeline mode}],
+ 'pgbench startpipeline command',
+ {
+ '001_pgbench_pipeline_2' => q{
+-- startpipeline twice
+\startpipeline
+\startpipeline
+}
+ });
+
+# Try to end a pipeline that hasn't started
+pgbench(
+ '-t 1 -n -M extended',
+ 2,
+ [],
+ [qr{not in pipeline mode}],
+ 'pgbench startpipeline command',
+ {
+ '001_pgbench_pipeline_3' => q{
+-- pipeline not started
+\endpipeline
+}
+ });
+
+# Try \gset in pipeline mode
+pgbench(
+ '-t 1 -n -M extended',
+ 2,
+ [],
+ [qr{gset is not allowed in pipeline mode}],
+ 'pgbench startpipeline command',
+ {
+ '001_pgbench_pipeline_gset' => q{
+\startpipeline
+select 1 \gset f
+\endpipeline
+}
+ });
+
+
# trigger many expression errors
my @errors = (
--
2.20.1
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
Are you going to update the assertion ?
+#if 0
Assert((meta == META_NONE && varprefix == NULL) ||
((meta == META_GSET || meta == META_ASET) && varprefix != NULL));
+#endif Re: [HACKERS] PATCH: Batch/pipelining support for libpq
On 2021-Mar-15, Justin Pryzby wrote:
Are you going to update the assertion ?
+#if 0 Assert((meta == META_NONE && varprefix == NULL) || ((meta == META_GSET || meta == META_ASET) && varprefix != NULL)); +#endif
Yeah, caught that just after sending. Here's a notpatch.
--
�lvaro Herrera 39�49'30"S 73�17'W
"La virtud es el justo medio entre dos defectos" (Arist�teles)
Attachments:
fixassert.notpatchtext/plain; charset=us-asciiDownload
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index ba7b35d83c..e69d43b26b 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -2823,13 +2823,12 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
int qrynum = 0;
/*
- * varprefix should be set only with \gset or \aset, and SQL commands do
- * not need it.
+ * varprefix should be set only with \gset or \aset, and \endpipeline and
+ * SQL commands do not need it.
*/
-#if 0
Assert((meta == META_NONE && varprefix == NULL) ||
+ ((meta == META_ENDPIPELINE) && varprefix == NULL) ||
((meta == META_GSET || meta == META_ASET) && varprefix != NULL));
-#endif
res = PQgetResult(st->con);
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
Hi,
On 2021-03-05 21:35:59 -0300, Alvaro Herrera wrote:
I'll take the weekend to think about the issue with conn->last_query and
conn->queryclass that I mentioned yesterday; other than that detail my
feeling is that this is committable, so I'll be looking at getting this
pushed early next weeks, barring opinions from others.
It is *very* exciting to see this being merged. Thanks for all the work
to all that contributed!
Greetings,
Andres Freund
Re: [HACKERS] PATCH: Batch/pipelining support for libpq
Thanks a lot for the merge. I did some tests and the master branch
runs up to 15% faster than the last
patch I tried (v22). Amazing!
Cheers,
Matthieu Garrigues
Show quoted text
On Tue, Mar 16, 2021 at 9:00 PM Andres Freund <andres@anarazel.de> wrote:
Hi,
On 2021-03-05 21:35:59 -0300, Alvaro Herrera wrote:
I'll take the weekend to think about the issue with conn->last_query and
conn->queryclass that I mentioned yesterday; other than that detail my
feeling is that this is committable, so I'll be looking at getting this
pushed early next weeks, barring opinions from others.It is *very* exciting to see this being merged. Thanks for all the work
to all that contributed!Greetings,
Andres Freund