Another oddity in handling of WCO constraints in postgres_fdw

On Thu, Sep 28, 2017 at 5:45 PM, Etsuro Fujita
<fujita.etsuro@lab.ntt.co.jp> wrote:

Hi,

Commit 7086be6e3627c1ad797e32ebbdd232905b5f577f addressed mishandling of WCO
in direct foreign table modification by disabling it when we have WCO, but I
noticed another oddity in postgres_fdw:

postgres=# create table base_tbl (a int, b int);
postgres=# create function row_before_insupd_trigfunc() returns trigger as
$$begin new.a := new.a + 10; return new; end$$ language plpgsql;
postgres=# create trigger row_before_insupd_trigger before insert or update
on base_tbl for each row execute procedure row_before_insupd_trigfunc();
postgres=# create server loopback foreign data wrapper postgres_fdw options
(dbname 'postgres');
postgres=# create user mapping for CURRENT_USER server loopback;
postgres=# create foreign table foreign_tbl (a int, b int) server loopback
options (table_name 'base_tbl');
postgres=# create view rw_view as select * from foreign_tbl where a < b with
check option;

So, this should fail, but

postgres=# insert into rw_view values (0, 5);
INSERT 0 1

The reason for that is: this is processed using postgres_fdw's non-direct
foreign table modification (ie. ForeignModify), but unlike the RETURNING or
local after trigger case, the ForeignModify doesn't take care that remote
triggers might change the data in that case, so the WCO is evaluated using
the data supplied, not the data actually inserted, which I think is wrong.
(I should have noticed that as well while working on the fix, though.) So,
I'd propose to fix that by modifying postgresPlanForeignModify so that it
handles WCO the same way as for the RETURNING case. Attached is a patch for
that. I'll add the patch to the next commitfest.

Enforcing WCO constraints imposed by the local server on the row/DML
being passed to the foreign server is fine, but trying to impose them
on the row being inserted/updated at the foreign server looks odd. May
be we should just leave this case as it is. I am comparing this case
with the way we handle constraints on a foreign table.

--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2017/10/03 18:16, Ashutosh Bapat wrote:

Enforcing WCO constraints imposed by the local server on the row/DML
being passed to the foreign server is fine, but trying to impose them
on the row being inserted/updated at the foreign server looks odd. May
be we should just leave this case as it is. I am comparing this case
with the way we handle constraints on a foreign table.

Hmm, I think that would be okay in the case where WCO constraints match
constraints on the foreign table, but I'm not sure that would be okay
even in the case where WCO constraints don't match? Consider:

create table bt (a int check (a % 2 = 0));
create foreign table ft (a int check (a % 2 = 0)) server loopback
options (table_name 'bt');
create view rw_view_2 as select * from ft where a % 2 = 0 with check option;

In that case the WCO constraint matches the constraint on the foreign
table, so there would be no need to ensure the WCO constraint locally
(to make the explanation simple, we assume here that we don't have
triggers on the remote end). BUT: for another auto-updatable view
defined using the same foreign table like this:

create view rw_view_4 as select * from ft where a % 4 = 0 with check option;

how is the WCO constraint (ie, a % 4 = 0) ensured remotely, which is
different from the constraint on the foreign table (ie, a % 2 = 0)?
Maybe I'm missing something, though.

Best regards,
Etsuro Fujita

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Oct 4, 2017 at 3:45 PM, Etsuro Fujita
<fujita.etsuro@lab.ntt.co.jp> wrote:

On 2017/10/03 18:16, Ashutosh Bapat wrote:

Enforcing WCO constraints imposed by the local server on the row/DML
being passed to the foreign server is fine, but trying to impose them
on the row being inserted/updated at the foreign server looks odd. May
be we should just leave this case as it is. I am comparing this case
with the way we handle constraints on a foreign table.

Hmm, I think that would be okay in the case where WCO constraints match
constraints on the foreign table, but I'm not sure that would be okay even
in the case where WCO constraints don't match? Consider:

create table bt (a int check (a % 2 = 0));
create foreign table ft (a int check (a % 2 = 0)) server loopback options
(table_name 'bt');
create view rw_view_2 as select * from ft where a % 2 = 0 with check option;

In that case the WCO constraint matches the constraint on the foreign table,
so there would be no need to ensure the WCO constraint locally (to make the
explanation simple, we assume here that we don't have triggers on the remote
end). BUT: for another auto-updatable view defined using the same foreign
table like this:

create view rw_view_4 as select * from ft where a % 4 = 0 with check option;

how is the WCO constraint (ie, a % 4 = 0) ensured remotely, which is
different from the constraint on the foreign table (ie, a % 2 = 0)? Maybe
I'm missing something, though.

Just like the local constraints on a foreign table are not ensured on
remote table (unless user takes steps to make that sure), WCO defined
locally need not be (and probably can not be) ensured remotely. We can
check whether a row being sent from the local server to the foreign
server obeys WCO, but what foreign server does to that row is beyond
local server's scope.

--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Oct 4, 2017 at 6:40 AM, Ashutosh Bapat
<ashutosh.bapat@enterprisedb.com> wrote:

Just like the local constraints on a foreign table are not ensured on
remote table (unless user takes steps to make that sure), WCO defined
locally need not be (and probably can not be) ensured remotely. We can
check whether a row being sent from the local server to the foreign
server obeys WCO, but what foreign server does to that row is beyond
local server's scope.

But I think right now we're not checking the row being sent from the
local server, either. The WCO that is being ignored isn't a
constraint on the foreign table; it's a constraint on a view which
happens to reference the foreign table. It seems quite odd for the
"assume constraints are valid" property of the foreign table to
propagate back up into the view that references it.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Oct 4, 2017 at 5:32 PM, Robert Haas <robertmhaas@gmail.com> wrote:

On Wed, Oct 4, 2017 at 6:40 AM, Ashutosh Bapat
<ashutosh.bapat@enterprisedb.com> wrote:

Just like the local constraints on a foreign table are not ensured on
remote table (unless user takes steps to make that sure), WCO defined
locally need not be (and probably can not be) ensured remotely. We can
check whether a row being sent from the local server to the foreign
server obeys WCO, but what foreign server does to that row is beyond
local server's scope.

But I think right now we're not checking the row being sent from the
local server, either.

Didn't 7086be6e3627c1ad797e32ebbdd232905b5f577f fix that?

The WCO that is being ignored isn't a
constraint on the foreign table; it's a constraint on a view which
happens to reference the foreign table. It seems quite odd for the
"assume constraints are valid" property of the foreign table to
propagate back up into the view that references it.

The view with WCO is local but the modification which violates WCO is
being made on remote server by a trigger on remote table. Trying to
control that doesn't seem to be a good idea, just like we can't
control what rows get inserted on the foreign server when they violate
local constraints. I am using local constraints as an example of
precedence where we ignore what's happening on remote side and enforce
whatever we could enforce locally. Local server should make sure that
any rows sent from local server to the remote server do not violate
any local WCO. But once it's handed over to the foreign server, we
shouldn't worry about what happens there. That behaviour is ensured by
the above commit, isn't it? I am not suggesting that we use local
constraints to enforce WCO or something like that.

--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2017/10/04 21:28, Ashutosh Bapat wrote:

On Wed, Oct 4, 2017 at 5:32 PM, Robert Haas <robertmhaas@gmail.com> wrote:

On Wed, Oct 4, 2017 at 6:40 AM, Ashutosh Bapat
<ashutosh.bapat@enterprisedb.com> wrote:

We can
check whether a row being sent from the local server to the foreign
server obeys WCO, but what foreign server does to that row is beyond
local server's scope.

But I think right now we're not checking the row being sent from the
local server, either.

We don't check the row *before* sending it to the remote server, but
check the row returned by ExecForeignInsert/ExecForeignUpdate, which is
allowed to have been changed by the remote server. In postgres_fdw, we
currently return the data actually inserted/updated if RETURNING/AFTER
TRIGGER present, but not if WCO only presents. So, for the postgres_fdw
foreign table, WCO is enforced on the data that was actually
inserted/updated if RETURNING/AFTER TRIGGER present and on the original
data core supplied if WCO only presents, which is inconsistent behavior.

Didn't 7086be6e3627c1ad797e32ebbdd232905b5f577f fix that?

No. The commit addressed another issue.

The WCO that is being ignored isn't a
constraint on the foreign table; it's a constraint on a view which
happens to reference the foreign table. It seems quite odd for the
"assume constraints are valid" property of the foreign table to
propagate back up into the view that references it.

Agreed.

The view with WCO is local but the modification which violates WCO is
being made on remote server by a trigger on remote table. Trying to
control that doesn't seem to be a good idea, just like we can't
control what rows get inserted on the foreign server when they violate
local constraints. I am using local constraints as an example of
precedence where we ignore what's happening on remote side and enforce
whatever we could enforce locally. Local server should make sure that
any rows sent from local server to the remote server do not violate
any local WCO.

Seems odd (and too restrictive) to me too.

Best regards,
Etsuro Fujita

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

At Thu, 5 Oct 2017 18:08:50 +0900, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote in <60e94494-4e5d-afed-e482-b9ad1986bbf6@lab.ntt.co.jp>

On 2017/10/04 21:28, Ashutosh Bapat wrote:

On Wed, Oct 4, 2017 at 5:32 PM, Robert Haas <robertmhaas@gmail.com>
wrote:

On Wed, Oct 4, 2017 at 6:40 AM, Ashutosh Bapat
<ashutosh.bapat@enterprisedb.com> wrote:

We can
check whether a row being sent from the local server to the foreign
server obeys WCO, but what foreign server does to that row is beyond
local server's scope.

But I think right now we're not checking the row being sent from the
local server, either.

We don't check the row *before* sending it to the remote server, but
check the row returned by ExecForeignInsert/ExecForeignUpdate, which
is allowed to have been changed by the remote server. In
postgres_fdw, we currently return the data actually inserted/updated
if RETURNING/AFTER TRIGGER present, but not if WCO only presents. So,
for the postgres_fdw foreign table, WCO is enforced on the data that
was actually inserted/updated if RETURNING/AFTER TRIGGER present and
on the original data core supplied if WCO only presents, which is
inconsistent behavior.

Didn't 7086be6e3627c1ad797e32ebbdd232905b5f577f fix that?

No. The commit addressed another issue.

The WCO that is being ignored isn't a
constraint on the foreign table; it's a constraint on a view which
happens to reference the foreign table. It seems quite odd for the
"assume constraints are valid" property of the foreign table to
propagate back up into the view that references it.

Agreed.

The view with WCO is local but the modification which violates WCO is
being made on remote server by a trigger on remote table. Trying to
control that doesn't seem to be a good idea, just like we can't
control what rows get inserted on the foreign server when they violate
local constraints. I am using local constraints as an example of
precedence where we ignore what's happening on remote side and enforce
whatever we could enforce locally. Local server should make sure that
any rows sent from local server to the remote server do not violate
any local WCO.

Seems odd (and too restrictive) to me too.

Since WCO ensures finally inserted values, we can't do other than
acturally requesting for the values. So just merging WCO columns
to RETURNING in deparsed query is ok. But can't we concatenate
returningList and withCheckOptionList at more higher level?
Specifically, just passing calculated used_attr to
deparse(Insert|Update)Sql instead of returningList and
withCheckOptionList separately. Deparsed queries anyway forget
the origin of requested columns.

regards,

--
Kyotaro Horiguchi
NTT Open Source Software Center

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2017/10/05 20:06, Kyotaro HORIGUCHI wrote:

Since WCO ensures finally inserted values, we can't do other than
acturally requesting for the values.

I think so too.

So just merging WCO columns
to RETURNING in deparsed query is ok. But can't we concatenate
returningList and withCheckOptionList at more higher level?
Specifically, just passing calculated used_attr to
deparse(Insert|Update)Sql instead of returningList and
withCheckOptionList separately. Deparsed queries anyway forget
the origin of requested columns.

We could do that, but I think that would need a bit more code to
postgresPlanForeignModify including changes to the deparseDeleteSql API
in addition to the deparse(Insert|Update)Sql APIs. I prefer making high
level functions simple, so I'd vote for just passing withCheckOptionList
separately to deparse(Insert|Update)Sql, as proposed in the patch.

Best regards,
Etsuro Fujita

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Oct 4, 2017 at 5:58 PM, Ashutosh Bapat
<ashutosh.bapat@enterprisedb.com> wrote:

The view with WCO is local but the modification which violates WCO is
being made on remote server by a trigger on remote table. Trying to
control that doesn't seem to be a good idea, just like we can't
control what rows get inserted on the foreign server when they violate
local constraints.

I think that's a fair point.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

(2017/11/01 11:16), Robert Haas wrote:

On Wed, Oct 4, 2017 at 5:58 PM, Ashutosh Bapat
<ashutosh.bapat@enterprisedb.com> wrote:

The view with WCO is local but the modification which violates WCO is
being made on remote server by a trigger on remote table. Trying to
control that doesn't seem to be a good idea, just like we can't
control what rows get inserted on the foreign server when they violate
local constraints.

I think that's a fair point.

For local constraints on foreign tables, it's the user's responsibility
to ensure that those constraints matches the remote side, so we don't
need to ensure those constraints locally. But I'm not sure if the same
thing applies to WCOs on views defined on foreign tables, because in
some case it's not possible to impose constraints on the remote side
that match those WCOs, as I explained before.

Best regards,
Etsuro Fujita

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Fri, Nov 10, 2017 at 8:36 PM, Etsuro Fujita
<fujita.etsuro@lab.ntt.co.jp> wrote:

For local constraints on foreign tables, it's the user's responsibility to
ensure that those constraints matches the remote side, so we don't need to
ensure those constraints locally. But I'm not sure if the same thing
applies to WCOs on views defined on foreign tables, because in some case
it's not possible to impose constraints on the remote side that match those
WCOs, as I explained before.

Moved to CF 2018-01.
--
Michael

Greetings Etsuro, Robert, all,

* Etsuro Fujita (fujita.etsuro@lab.ntt.co.jp) wrote:

(2017/11/01 11:16), Robert Haas wrote:

On Wed, Oct 4, 2017 at 5:58 PM, Ashutosh Bapat
<ashutosh.bapat@enterprisedb.com> wrote:

The view with WCO is local but the modification which violates WCO is
being made on remote server by a trigger on remote table. Trying to
control that doesn't seem to be a good idea, just like we can't
control what rows get inserted on the foreign server when they violate
local constraints.

I think that's a fair point.

For local constraints on foreign tables, it's the user's responsibility to
ensure that those constraints matches the remote side, so we don't need to
ensure those constraints locally. But I'm not sure if the same thing
applies to WCOs on views defined on foreign tables, because in some case
it's not possible to impose constraints on the remote side that match those
WCOs, as I explained before.

Reviewing this thread, I tend to agree with Etsuro and I'm not sure I
see where there's a good argument for having a foreign table under a
view behave differently than a local table under a view for WCO (which
is an option of the view- not about the table underneath it or if it's
local or remote). I've not done a detailed review of the patch but it
seems pretty reasonable and pretty small.

Thanks!

Stephen

(2018/01/17 22:00), Stephen Frost wrote:

Reviewing this thread, I tend to agree with Etsuro and I'm not sure I
see where there's a good argument for having a foreign table under a
view behave differently than a local table under a view for WCO (which
is an option of the view- not about the table underneath it or if it's
local or remote). I've not done a detailed review of the patch but it
seems pretty reasonable and pretty small.

Thanks for the comments!

I noticed the patch doesn't apply. Attached is a rebased patch.

Best regards,
Etsuro Fujita

(2018/01/18 16:16), Etsuro Fujita wrote:

Attached is a rebased patch.

I rebased the patch over HEAD and revised comments/docs a little bit.
Please find attached a new version of the patch.

Best regards,
Etsuro Fujita

Hello,

On Wed, Feb 28, 2018 at 05:22:42PM +0900, Etsuro Fujita wrote:

I rebased the patch over HEAD and revised comments/docs a little bit. Please
find attached a new version of the patch.

I've reviewed the patch.

The code is good, clear and it is pretty small. There are documentation
fixes and additional regression tests.

Unfortunately the patch is outdated and it needs rebasing. Outdated
files are regression tests files.

After rebasing regression tests they pass.

--
Arthur Zakirov
Postgres Professional: http://www.postgrespro.com
Russian Postgres Company

Hi Arthur,

(2018/03/03 18:51), Arthur Zakirov wrote:

On Wed, Feb 28, 2018 at 05:22:42PM +0900, Etsuro Fujita wrote:

I rebased the patch over HEAD and revised comments/docs a little bit. Please
find attached a new version of the patch.

I've reviewed the patch.

The code is good, clear and it is pretty small. There are documentation
fixes and additional regression tests.

Unfortunately the patch is outdated and it needs rebasing. Outdated
files are regression tests files.

After rebasing regression tests they pass.

I rebased the patch over HEAD. Please find attached an updated patch.

Thank you for the review!

Best regards,
Etsuro Fujita

On Mon, Mar 05, 2018 at 09:44:37PM +0900, Etsuro Fujita wrote:

I rebased the patch over HEAD. Please find attached an updated patch.

Thank you!

IMHO, it is worth to add more explaining comment into
deparseReturningList, why it is necessary to merge WCO attributes to
RETURNING clause. You already noted it in the thread. I think it could
confuse someone who not very familiar how RETURNING is related with WITH
CHECK OPTION.

--
Arthur Zakirov
Postgres Professional: http://www.postgrespro.com
Russian Postgres Company

(2018/03/06 1:57), Arthur Zakirov wrote:

IMHO, it is worth to add more explaining comment into
deparseReturningList, why it is necessary to merge WCO attributes to
RETURNING clause. You already noted it in the thread. I think it could
confuse someone who not very familiar how RETURNING is related with WITH
CHECK OPTION.

Agreed. I added a comment to that function. I think that that comment
in combination with changes to the FDW docs in the patch would help FDW
authors understand why that is needed. Please find attached an updated
version of the patch.

Thanks for the comments!

Best regards,
Etsuro Fujita

On Tue, Mar 06, 2018 at 08:09:50PM +0900, Etsuro Fujita wrote:

Agreed. I added a comment to that function. I think that that comment in
combination with changes to the FDW docs in the patch would help FDW authors
understand why that is needed. Please find attached an updated version of
the patch.

Thank you.

All tests pass, the documentation builds. There was the suggestion [1]
of different approach. But the patch fix the issue in much more simple
way.

Marked as "Ready for Commiter".

1 - /messages/by-id/20171005.200631.134118679.horiguchi.kyotaro@lab.ntt.co.jp

--
Arthur Zakirov
Postgres Professional: http://www.postgrespro.com
Russian Postgres Company