Method to pass data between queries in a multi-statement transaction

On Wed, Apr 17, 2019 at 3:04 PM Souvik Bhattacherjee <kivuosb@gmail.com>
wrote:

Hello,

I'm trying to pass some values between queries in a multi-statement
transaction. For example, consider the following representative
multi-statement transaction:

begin;
select * from table1 t1, table2 t2 where t1.cid = t2.cid;
delete from table1 where cid in
(values-to-be-populated-from-the-previous-query);
commit;

There are few if any situations where you need to immediately and
completely pass all values from one query to another in the same
transaction where the queries cannot just be combined into a single
statement. Your representative example is one that is easily combined into
a single statement.

Now, the stuff you are trying seems to indicate you are trying to do
something in C, inside the engine itself, with all of this. If that is the
case you may want to be more clear as to what you are attempting to do.
But as far as server SQL goes the only persistence area are
tables/relations - including temporary ones.

David J.

On 4/17/19 3:04 PM, Souvik Bhattacherjee wrote:

Hello,

I'm trying to pass some values between queries in a multi-statement
transaction. For example, consider the following representative
multi-statement transaction:

begin;
select * from table1 t1, table2 t2 where t1.cid = t2.cid;
delete from table1 where cid in
(values-to-be-populated-from-the-previous-query);
commit;

https://www.postgresql.org/docs/11/sql-delete.html
"DELETE FROM films
WHERE producer_id IN (SELECT id FROM producers WHERE name = 'foo');"

Now, assume that I'm able to get the cid values from table1 that
satisfies the equi-join condition and I want to pass those values in the
IN condition in the subsequent delete query. Is there a straightforward
way to achieve this by modifying the postgresql source code?

I tried doing this by creating a hash table (by following this example:
https://wiki.postgresql.org/wiki/HashTable). The hash key in this case
is the current transactionid (which I presume should remain unchanged
for the entire duration of the transaction) and the hash value is
dynamically allocated. Within the query I can perform a lookup and
insert without any problem. However, when I try to do a lookup of the
hash value from a different query that did not insert the value
originally, I do not get any value. The hash table is able to tell me
that the key exists (due to the fact that the key is not dynamically
allocated) but doesn't return any value. My guess is that after each
query in the multi-statement txn block, the storage for that query is
deallocated, which results in the following behaviour.

The hash table approach (although it didn't work) above, IMO, has the
drawback that it needs to be locked since there can be other txns that
can try to access the hash table as well.

The other approach here would be the serialize the values into a file
and then read those values later from the subsequent query. However this
is not efficient.

Thus, I'm looking for a method of passing values between queries in a
multi-statement block that avoids the disk and does not need locking. I
was wondering if there is a way to define a hash table (or any data
structure) which is visible only to the current transaction.

-SB

--
Adrian Klaver
adrian.klaver@aklaver.com

There are few if any situations where you need to immediately and
completely pass all values from one query to another in the same
transaction where the queries cannot just be combined into a single
statement. Your representative example is one that is easily combined into
a single statement.

What if I need the result of the join to be stored into table3 as well as

the tuples that participated in the query to be deleted from table1. The
following can be done without the need to transfer values from the previous
query into the next:

begin;
insert into table3 (id, attr1, attr2) (select t1.cid, t1.empname, t2.dept
from table1 t1, table2 t2 where t1.cid = t2.cid);
delete from table1 where cid in (select c.cid from table1 t1, table2 t2
where t1.cid = t2.cid);
commit;

However note that we have to perform the join twice, which is not
efficient. Now to make things worse, increase the number of tables to join
while imposing the requirement of tuple deletion to apply to all or to a
subset of the tables that participate in join.

Now, the stuff you are trying seems to indicate you are trying to do
something in C, inside the engine itself, with all of this. If that is the
case you may want to be more clear as to what you are attempting to do.
But as far as server SQL goes the only persistence area are
tables/relations - including temporary ones.

I'm trying to modify the engine here.

-SB

On Wed, Apr 17, 2019 at 6:16 PM David G. Johnston <
david.g.johnston@gmail.com> wrote:

One can argue here that we can use the results from table3 to perform the
delete operation instead of joining again. But table3 may be subject to
updates which makes the situation tricky.

On Wed, Apr 17, 2019 at 8:30 PM Souvik Bhattacherjee <kivuosb@gmail.com>
wrote:

On Wed, Apr 17, 2019 at 5:30 PM Souvik Bhattacherjee <kivuosb@gmail.com>
wrote:

What if I need the result of the join to be stored into table3 as well

as the tuples that participated in the query to be deleted from table1. The
following can be done without the need to transfer values from the previous
query into the next:

begin;
insert into table3 (id, attr1, attr2) (select t1.cid, t1.empname, t2.dept
from table1 t1, table2 t2 where t1.cid = t2.cid);
delete from table1 where cid in (select c.cid from table1 t1, table2 t2
where t1.cid = t2.cid);
commit;

You can use INSERT...RETURNING in a WITH query:

postgres=# create table foo (a integer);
CREATE TABLE
postgres=# create table bar (b integer);
CREATE TABLE
postgres=# insert into bar values (42);
INSERT 0 1
postgres=# with i as (insert into foo values (42) returning a) delete from
bar where b = (select a from i);
DELETE 1

-Michel

On 4/17/19 5:30 PM, Souvik Bhattacherjee wrote:

There are few if any situations where you need to immediately and
completely pass all values from one query to another in the same
transaction where the queries cannot just be combined into a single
statement.  Your representative example is one that is easily combined
into a single statement.

What if I need the result of the join to be stored into table3 as

well as the tuples that participated in the query to be deleted from
table1. The following can be done without the need to transfer values
from the previous query into the next:

begin;
insert into table3 (id, attr1, attr2) (select t1.cid, t1.empname,
t2.dept from table1 t1, table2 t2 where t1.cid = t2.cid);
delete from table1 where cid in (select c.cid from table1 t1, table2 t2
where t1.cid = t2.cid);

Well the DELETE is not going to work as c.cid will error as undefined.

commit;

However note that we have to perform the join twice, which is not
efficient. Now to make things worse, increase the number of tables to
join while imposing the requirement of tuple deletion to apply to all or
to a subset of the tables that participate in join.

You might want to take a look at CTE's:

https://www.postgresql.org/docs/11/queries-with.html

Now, the stuff you are trying seems to indicate you are trying to do
something in C, inside the engine itself, with all of this.  If that is
the case you may want to be more clear as to what you are attempting to
do.  But as far as server SQL goes the only persistence area are
tables/relations - including temporary ones.

I'm trying to modify the engine here.

-SB

On Wed, Apr 17, 2019 at 6:16 PM David G. Johnston
<david.g.johnston@gmail.com <mailto:david.g.johnston@gmail.com>> wrote:

On Wed, Apr 17, 2019 at 3:04 PM Souvik Bhattacherjee
<kivuosb@gmail.com <mailto:kivuosb@gmail.com>> wrote:

Hello,

I'm trying to pass some values between queries in a
multi-statement transaction. For example, consider the following
representative multi-statement transaction:

begin;
select * from table1 t1, table2 t2 where t1.cid = t2.cid;
delete from table1 where cid in
(values-to-be-populated-from-the-previous-query);
commit;

There are few if any situations where you need to immediately and
completely pass all values from one query to another in the same
transaction where the queries cannot just be combined into a single
statement.  Your representative example is one that is easily
combined into a single statement.

Now, the stuff you are trying seems to indicate you are trying to do
something in C, inside the engine itself, with all of this.  If that
is the case you may want to be more clear as to what you are
attempting to do.  But as far as server SQL goes the only
persistence area are tables/relations - including temporary ones.

David J.

--
Adrian Klaver
adrian.klaver@aklaver.com

Thanks Michel.

However this only works if a is an unique attribute in the table that would
help us to identify tuples that participated in the join. Consider the
following join:

insert into table3 (id, level, empname, salary)
(select t0.cid, t0.level, t1.empname, t2.salary from table0 t0, table1 t1,
table2 t2 where t0.cid = t1.cid and t1.pid = t2.pid);

Now if I want to delete those tuples from table2 that satisfied the join
condition, I need to execute the join again with additional attributes.
Also note that based on query plan, i.e. whether table0 and table1 were
joined first followed by table1 and table2, we have to execute one
additional join to get the tuples in table2 that satisfied the join
condition (t1.pid = t2.pid).

Getting that information while the query is executed may not be difficult.
There are other use cases in my application that require me to transfer the
data from one query to the next within a transaction.

Thus, what I'm looking for here is way to store the information and then
pass that information to the next query efficiently.
For example, is it possible to define a struct of my choice, private to the
current transaction, that would store the data and then pass it around to
the next query in the transaction without having to materialize that struct
(or deal with concurrency issues as in the hash table approach mentioned
earlier) .

-SB

On Wed, Apr 17, 2019 at 10:47 PM Michel Pelletier <
pelletier.michel@gmail.com> wrote:

Well the DELETE is not going to work as c.cid will error as undefined.

Yes, that's a typo. I haven't tested it out before typing; just wanted to

convey the general idea.

-SB

On Thu, Apr 18, 2019 at 10:50 AM Adrian Klaver <adrian.klaver@aklaver.com>
wrote:

Thus, what I'm looking for here is way to store the information and then
pass that information to the next query efficiently.
For example, is it possible to define a struct of my choice, private to
the current transaction, that would store the data and then pass it around
to the next query in the transaction without having to materialize that
struct (or deal with concurrency issues as in the hash table approach
mentioned earlier) .

Perhaps I am missing something obvious, but why not use a temp table?

On Thu, Apr 18, 2019 at 9:06 AM Michael Lewis <mlewis@entrata.com> wrote:

Thus, what I'm looking for here is way to store the information and then

pass that information to the next query efficiently.
For example, is it possible to define a struct of my choice, private to
the current transaction, that would store the data and then pass it around
to the next query in the transaction without having to materialize that
struct (or deal with concurrency issues as in the hash table approach
mentioned earlier) .

Perhaps I am missing something obvious, but why not use a temp table?

Right, or as Adrian and I pointed out, use a CTE (WITH) query, which will
materialize any results you want for the query. Perhaps if you give us a
working, reproducible, self contained example of what you expect we can
help you better.

On Thu, Apr 18, 2019 at 9:03 AM Souvik Bhattacherjee <kivuosb@gmail.com>
wrote:

Thanks Michel.

However this only works if a is an unique attribute in the table that
would help us to identify tuples that participated in the join. Consider
the following join:

insert into table3 (id, level, empname, salary)
(select t0.cid, t0.level, t1.empname, t2.salary from table0 t0, table1
t1, table2 t2 where t0.cid = t1.cid and t1.pid = t2.pid);

Now if I want to delete those tuples from table2 that satisfied the join
condition, I need to execute the join again with additional attributes.

Or just add pid to table3...

Also note that based on query plan, i.e. whether table0 and table1 were
joined first followed by table1 and table2, we have to execute one
additional join to get the tuples in table2 that satisfied the join
condition (t1.pid = t2.pid).

???

Getting that information while the query is executed may not be difficult.
There are other use cases in my application that require me to transfer the
data from one query to the next within a transaction.

There may be some that benefit to some degree but its likely that you can
write the application and queries in such a way to avoid a hard requirement.

Thus, what I'm looking for here is way to store the information and then

pass that information to the next query efficiently.

For example, is it possible to define a struct of my choice, private to the

current transaction, that would store the data and then pass it around to
the next query in the transaction without having to materialize that struct
(or deal with concurrency issues as in the hash table approach mentioned
earlier) .

How much development and maintenance effort are you willing to spend here
to gain what is likely to amount to only a bit of efficiency? Many things
are possible if you are going to modify the server code but why add grief?

David J.

Or just add pid to table3...

That's an application requirement. So pid cannot be added at will to

table3.

How much development and maintenance effort are you willing to spend here
to gain what is likely to amount to only a bit of efficiency? Many things
are possible if you are going to modify the server code but why add grief?

How much development / maintenance effort do you anticipate for

implementing this feature? This is something that my application would need
so I'm willing to dedicate some time to it.

By the way, I figured that the hash table deallocation issue could be
resolved (although in an inefficient way) by serializing the data to a
string and then copying that value into the hash table during insertion.
However the hash table is still visible to all the transactions I suppose
and as a result needs to be locked. Just wanted to let you know that I have
initialized the hash table within the PostgresMain() method and the hash
table is declared as an extern variable, which I anticipate to be accessed
by many methods. How difficult is it to make a hash table (or any data
structure) private to the current transaction so that I do not have to
resort to locking?

-SB

On Thu, Apr 18, 2019 at 5:34 PM David G. Johnston <
david.g.johnston@gmail.com> wrote: