Patching dblink.c to avoid warning about open transaction

Jonathan Beit-Aharon <jbeitaharon@intrusic.com> writes:

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if (!conn)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; DBLINK_CONN_NOT_AVAIL;<br>
+<br>
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if (rcon)<br>
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; rcon-&gt;remoteTrFlag = (PQtransactionStatus(conn) !=
PQTRANS_IDLE);<br>
<br>
-&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; res = PQexec(conn, "BEGIN");<br>
-&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if (PQresultStatus(res) != PGRES_COMMAND_OK)<br>

[etc]

Could we see this in a less broken format?

regards, tom lane

Tom Lane wrote:

Jonathan Beit-Aharon <jbeitaharon@intrusic.com> writes:

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if (!conn)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; DBLINK_CONN_NOT_AVAIL;<br>

Could we see this in a less broken format?

Here is the patch in text format.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

[ Joe, would you review this? ]

Your patch has been added to the PostgreSQL unapplied patches list at:

http://momjian.postgresql.org/cgi-bin/pgpatches

It will be applied as soon as one of the PostgreSQL committers reviews
and approves it.

---------------------------------------------------------------------------

Bruce Momjian wrote:

Tom Lane wrote:

Jonathan Beit-Aharon <jbeitaharon@intrusic.com> writes:

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if (!conn)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; DBLINK_CONN_NOT_AVAIL;<br>

Could we see this in a less broken format?

Here is the patch in text format.

-- 
Bruce Momjian                        |  http://candle.pha.pa.us
pgman@candle.pha.pa.us               |  (610) 359-1001
+  If your life is a hard drive,     |  13 Roberts Road
+  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

--- dblink.c    Sat Jan  1 00:43:05 2005
+++
/home/jbeitaharon/dev/third/postgreSQL/contrib/dblink/dblink.c
Thu Sep 22 16:10:20 2005
@@ -329,12 +329,16 @@
if (!conn)
DBLINK_CONN_NOT_AVAIL;
+
+       if (rcon)
+               rcon->remoteTrFlag = (PQtransactionStatus(conn) !=
PQTRANS_IDLE);
-       res = PQexec(conn, "BEGIN");
-       if (PQresultStatus(res) != PGRES_COMMAND_OK)
-               DBLINK_RES_INTERNALERROR("begin error");
-
-       PQclear(res);
+       if ((!rcon) || (!(rcon->remoteTrFlag))) {
+               res = PQexec(conn, "BEGIN");
+               if (PQresultStatus(res) != PGRES_COMMAND_OK)
+                       DBLINK_RES_INTERNALERROR("begin error");
+               PQclear(res);
+       }
appendStringInfo(str, "DECLARE %s CURSOR FOR %s", curname,
sql);
res = PQexec(conn, str->data);
@@ -424,12 +428,13 @@
PQclear(res);
-       /* commit the transaction */
-       res = PQexec(conn, "COMMIT");
-       if (PQresultStatus(res) != PGRES_COMMAND_OK)
-               DBLINK_RES_INTERNALERROR("commit error");
-
-       PQclear(res);
+       if ((!rcon) || (!(rcon->remoteTrFlag))) {
+               /* commit the transaction */
+               res = PQexec(conn, "COMMIT");
+               if (PQresultStatus(res) != PGRES_COMMAND_OK)
+                       DBLINK_RES_INTERNALERROR("commit error");
+               PQclear(res);
+       }
PG_RETURN_TEXT_P(GET_TEXT("OK"));
}

---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian wrote:

[ Joe, would you review this? ]

Your patch has been added to the PostgreSQL unapplied patches list at:

http://momjian.postgresql.org/cgi-bin/pgpatches

It will be applied as soon as one of the PostgreSQL committers reviews
and approves it.

The patch itself is pretty simple, but I'm unclear on the use case.
Jonathan, can you elaborate a bit?

Thanks,

Joe

p.s. I'm on a business trip in Asia again, so my responses may be
delayed a bit.

Joe Conway wrote:

Bruce Momjian wrote:

[ Joe, would you review this? ]

Your patch has been added to the PostgreSQL unapplied patches list at:

http://momjian.postgresql.org/cgi-bin/pgpatches

It will be applied as soon as one of the PostgreSQL committers reviews
and approves it.

The patch itself is pretty simple, but I'm unclear on the use case.
Jonathan, can you elaborate a bit?

Thanks,

Joe

p.s. I'm on a business trip in Asia again, so my responses may be
delayed a bit.

Hi Joe,
We are using the dblink module on Sensor servers to provide
summarization services to a Central server. Sensors are in the business
of populating certain Postgres databases, and the Central is in the
business of populating a summary Postgres database. The issue in our
situation is that the Central server does an explicit BEGIN transaction
some time before it calls the dblink_open() routine. On the Sensors, we
were getting many "there is already a transaction in progress" warnings,
and overflowing our log storage. Is this patch the right way to go
about this?
Thanks,
Jonathan

I'm not a member of this list (yet), so please CC me on responses and
discussion. The patch below seems to be completion of work already
started, because the boolean remoteTrFlag was already defined, and all I
had to add was its setting and two references. I hope someone will find
it useful,

Jonathan

I have worked on this issue and have an extensive patch to dblink to fix
it.

The reported problem is that dblink_open/dblink_close() (for cursor
reads) do a BEGIN/COMMIT regardless of the transaction state of the
remote connection. There was code in dblink.c to track the remote
transaction state (rconn), but it was not being maintained or used.

This patch fixes that by routing all connections through an rconn
structure and using the transaction status properly. I removed the
global persistent connection and function-local 'conn' structures in
favor of using rconn consistently. This cleans up a lot of error-prone
code that tried to track what type of connection was being used.

I don't know if people want this for 8.1 or 8.2.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

On Thu, Oct 06, 2005 at 10:38:54PM -0400, Bruce Momjian wrote:

I'm not a member of this list (yet), so please CC me on responses
and discussion. The patch below seems to be completion of work
already started, because the boolean remoteTrFlag was already
defined, and all I had to add was its setting and two references.
I hope someone will find it useful,

Jonathan

I have worked on this issue and have an extensive patch to dblink to
fix it.

The reported problem is that dblink_open/dblink_close() (for cursor
reads) do a BEGIN/COMMIT regardless of the transaction state of the
remote connection. There was code in dblink.c to track the remote
transaction state (rconn), but it was not being maintained or used.

This patch fixes that by routing all connections through an rconn
structure and using the transaction status properly. I removed the
global persistent connection and function-local 'conn' structures in
favor of using rconn consistently. This cleans up a lot of
error-prone code that tried to track what type of connection was
being used.

I don't know if people want this for 8.1 or 8.2.

8.1, IMHO. It's a bug fix. :)

Cheers,
D
--
David Fetter david@fetter.org http://fetter.org/
phone: +1 510 893 6100 mobile: +1 415 235 3778

Remember to vote!

David Fetter <david@fetter.org> writes:

On Thu, Oct 06, 2005 at 10:38:54PM -0400, Bruce Momjian wrote:

I don't know if people want this for 8.1 or 8.2.

8.1, IMHO. It's a bug fix. :)

Not unless Joe Conway signs off on it ...

regards, tom lane

On Thu, Oct 06, 2005 at 11:31:46PM -0400, Tom Lane wrote:

David Fetter <david@fetter.org> writes:

On Thu, Oct 06, 2005 at 10:38:54PM -0400, Bruce Momjian wrote:

I don't know if people want this for 8.1 or 8.2.

8.1, IMHO. It's a bug fix. :)

Not unless Joe Conway signs off on it ...

Of course it's his to sign or not :)

Cheers,
D
--
David Fetter david@fetter.org http://fetter.org/
phone: +1 510 893 6100 mobile: +1 415 235 3778

Remember to vote!

Tom Lane wrote:

David Fetter <david@fetter.org> writes:

On Thu, Oct 06, 2005 at 10:38:54PM -0400, Bruce Momjian wrote:

I don't know if people want this for 8.1 or 8.2.

8.1, IMHO. It's a bug fix. :)

Not unless Joe Conway signs off on it ...

I had asked on the original simple patch, and I'll ask again now -- why
is this needed? I don't have a clear understanding of the problem that
this (or the earlier) patch is trying to solve. Without that, it is
impossible to say whether it is a bug fix or feature.

Joe

Joe Conway wrote:

Tom Lane wrote:

David Fetter <david@fetter.org> writes:

On Thu, Oct 06, 2005 at 10:38:54PM -0400, Bruce Momjian wrote:

I don't know if people want this for 8.1 or 8.2.

8.1, IMHO. It's a bug fix. :)

Not unless Joe Conway signs off on it ...

I had asked on the original simple patch, and I'll ask again now -- why
is this needed? I don't have a clear understanding of the problem that
this (or the earlier) patch is trying to solve. Without that, it is
impossible to say whether it is a bug fix or feature.

Well, as I said in the patch email:

The reported problem is that dblink_open/dblink_close() (for cursor
reads) do a BEGIN/COMMIT regardless of the transaction state of the
remote connection. There was code in dblink.c to track the remote
transaction state (rconn), but it was not being maintained or used.

If a remote transactions has already been started by the user,
dblink_open is going to call BEGIN, which is going to fail because there
is already an open transaction, right?

Here is an crude example:

test=> BEGIN;
BEGIN
test=> begin;
WARNING: there is already a transaction in progress
BEGIN
test=> SELECT 1;
?column?
----------
1
(1 row)

test=> COMMIT;
COMMIT
test=> COMMIT;
WARNING: there is no transaction in progress
COMMIT

The BEGIN and COMMIT are only a warning though, but the early COMMIT by
the dblink_close() is a serious issue.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Well, as I said in the patch email:

The reported problem is that dblink_open/dblink_close() (for cursor
reads) do a BEGIN/COMMIT regardless of the transaction state of the
remote connection. There was code in dblink.c to track the remote
transaction state (rconn), but it was not being maintained or used.

You should lose the remoteXactOpen flag entirely, in favor of just
testing PQtransactionStatus() on-the-fly when necessary. Simpler,
more reliable, not notably slower.

With that change, the separate remoteConn struct could be dropped
altogether in favor of just using the PGconn pointer. This would
make things notationally simpler, and in fact perhaps allow undoing
the bulk of the edits in your patch. As-is I think the patch is
pretty risky to apply during beta.

regards, tom lane

Tom Lane wrote:

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Well, as I said in the patch email:

The reported problem is that dblink_open/dblink_close() (for cursor
reads) do a BEGIN/COMMIT regardless of the transaction state of the
remote connection. There was code in dblink.c to track the remote
transaction state (rconn), but it was not being maintained or used.

You should lose the remoteXactOpen flag entirely, in favor of just
testing PQtransactionStatus() on-the-fly when necessary. Simpler,
more reliable, not notably slower.

With that change, the separate remoteConn struct could be dropped
altogether in favor of just using the PGconn pointer. This would
make things notationally simpler, and in fact perhaps allow undoing
the bulk of the edits in your patch. As-is I think the patch is
pretty risky to apply during beta.

The problem with not using rconn is that we are not saving the
transaction status at the _start_ of the cursor open. If we don't do
that, we have no way of knowing on close if _we_ opened the transaction
or whether it was opened by the user.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian <pgman@candle.pha.pa.us> writes:

The problem with not using rconn is that we are not saving the
transaction status at the _start_ of the cursor open. If we don't do
that, we have no way of knowing on close if _we_ opened the transaction
or whether it was opened by the user.

Oh, I see. In that case the flag is horribly misnamed and miscommented.
Something like "newXactForCursor" would be clearer.

regards, tom lane

Tom Lane wrote:

Bruce Momjian <pgman@candle.pha.pa.us> writes:

The problem with not using rconn is that we are not saving the
transaction status at the _start_ of the cursor open. If we don't do
that, we have no way of knowing on close if _we_ opened the transaction
or whether it was opened by the user.

Oh, I see. In that case the flag is horribly misnamed and miscommented.
Something like "newXactForCursor" would be clearer.

OK, renamed to isClientXact.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian wrote:

There was also a problem in that if two cursors were opened, the first
close would close the transaction. I have fixed that code by changing
the xact variable in to a counter that keeps track of the number of
opened cursors and commits only when they are all closed.

Both the dblink.c patch and the regression patch are at:

ftp://candle.pha.pa.us/pub/postgresql/mypatches

OK, I'll take a look, but I won't have time for a couple of days (I'm
not at home -- visiting my dad for his 80th birthday -- and have no
broadband access).

Joe

Joe Conway wrote:

Bruce Momjian wrote:

There was also a problem in that if two cursors were opened, the first
close would close the transaction. I have fixed that code by changing
the xact variable in to a counter that keeps track of the number of
opened cursors and commits only when they are all closed.

Both the dblink.c patch and the regression patch are at:

ftp://candle.pha.pa.us/pub/postgresql/mypatches

OK, I'll take a look, but I won't have time for a couple of days (I'm
not at home -- visiting my dad for his 80th birthday -- and have no
broadband access).

No problem -- thanks. I have slimmed down the patch by applying the
cosmetic parts to CVS. Use the URL above to get the newest versions of
the dblink.c and regression changes.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

[ reposted]

Tom Lane wrote:

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Well, as I said in the patch email:

The reported problem is that dblink_open/dblink_close() (for cursor
reads) do a BEGIN/COMMIT regardless of the transaction state of the
remote connection. There was code in dblink.c to track the remote
transaction state (rconn), but it was not being maintained or used.

You should lose the remoteXactOpen flag entirely, in favor of just
testing PQtransactionStatus() on-the-fly when necessary. Simpler,
more reliable, not notably slower.

With that change, the separate remoteConn struct could be dropped
altogether in favor of just using the PGconn pointer. This would
make things notationally simpler, and in fact perhaps allow undoing
the bulk of the edits in your patch. As-is I think the patch is
pretty risky to apply during beta.

I have developed a dblink regression patch that tests for the bug. It
opens a transaction in the client code, opens/closes a cursor, then
tries a DECLARE. Without my patch, this fails because the
dblink_close() closes the transaction, but with my patch it succeeds.
Here is the test:

-- test opening cursor in a transaction
SELECT dblink_exec('myconn','BEGIN');

-- an open transaction will prevent dblink_open() from opening its own
SELECT dblink_open('myconn','rmt_foo_cursor','SELECT * FROM foo');

-- this should not close the cursor because the client opened it
SELECT dblink_close('myconn','rmt_foo_cursor');

-- this should succeed because we have an open transaction
SELECT dblink_exec('myconn','DECLARE xact_test CURSOR FOR SELECT * FROM foo');

-- commit remote transaction
SELECT dblink_exec('myconn','COMMIT');

Here is the failure reported by the current code:

-- this should succeed because we have an open transaction
SELECT dblink_exec('myconn','DECLARE xact_test CURSOR FOR SELECT * FROM foo');
ERROR: sql error
DETAIL: ERROR: DECLARE CURSOR may only be used in transaction blocks

There was also a problem in that if two cursors were opened, the first
close would close the transaction. I have fixed that code by changing
the xact variable in to a counter that keeps track of the number of
opened cursors and commits only when they are all closed.

Both the dblink.c patch and the regression patch are at:

ftp://candle.pha.pa.us/pub/postgresql/mypatches

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian wrote:

No problem -- thanks. I have slimmed down the patch by applying the
cosmetic parts to CVS. Use the URL above to get the newest versions of
the dblink.c and regression changes.

Here is my counter-proposal to Bruce's dblink patch. Any comments?

Is it too late to apply this for 8.1? I tend to agree with calling this
a bugfix.

Thanks,

Joe