validating foreign tables

Andrew Dunstan <andrew@dunslane.net> writes:

The API for FDW validators doesn't appear to have any way that the
validator function can check that the defined foreign table shape
matches the FDW options sanely.

Huh? The options ought to be orthogonal to the table column info.
If they're not, maybe you need to rethink your option definitions.

regards, tom lane

On 02/21/2011 06:56 PM, Tom Lane wrote:

Andrew Dunstan<andrew@dunslane.net> writes:

The API for FDW validators doesn't appear to have any way that the
validator function can check that the defined foreign table shape
matches the FDW options sanely.

Huh? The options ought to be orthogonal to the table column info.
If they're not, maybe you need to rethink your option definitions.

Well, let's take a couple of cases.

1. My old favorite, file as a text array.
2. A hypothetical RSS feed, where the options specify which RSS fields
we want.

Of course, we could just let these break or give odd results when we run
a SELECT if the foreign table doesn't match what's expected. file_fdw
will presumably break if the input file has rows with the wrong number
of columns, just as COPY does. But there will be cases, like the two
above, where a sanity check on the table shape could usefully be done at
validation time as opposed to run time, and it would be nice to be able
to do such a check.

cheers

andrew

On Tue, Feb 22, 2011 at 10:12, Andrew Dunstan <andrew@dunslane.net> wrote:

The API for FDW validators doesn't appear to have any way that the
validator function can check that the defined foreign table shape
matches the FDW options sanely.

Huh?  The options ought to be orthogonal to the table column info.
If they're not, maybe you need to rethink your option definitions.

Well, let's take a couple of cases.

1. My old favorite, file as a text array.
2. A hypothetical RSS feed, where the options specify which RSS fields we
want.

I think we need to overhaul validators in 9.2 listening to FDW developers'
opinions anyway. The text array is an example, but there should be many
other requirements. Personally, I'd like to have a method to list available
options from SQL. We should also consider column-level options for foreign
tables then.

--
Itagaki Takahiro

On 02/21/2011 08:59 PM, Itagaki Takahiro wrote:

I think we need to overhaul validators in 9.2 listening to FDW developers'
opinions anyway. The text array is an example, but there should be many
other requirements. Personally, I'd like to have a method to list available
options from SQL. We should also consider column-level options for foreign
tables then.

Ok, I guess. It just seems to me like it will be harder to extend the
API later than now, so if we can reasonably foresee a likely need we
should try to provide for it.

cheers

andrew

Andrew Dunstan <andrew@dunslane.net> writes:

On 02/21/2011 06:56 PM, Tom Lane wrote:

Huh? The options ought to be orthogonal to the table column info.
If they're not, maybe you need to rethink your option definitions.

Well, let's take a couple of cases.

1. My old favorite, file as a text array.
2. A hypothetical RSS feed, where the options specify which RSS fields
we want.

As above, I claim that an FDW that has such options is badly designed to
begin with. Why can't you generate the RSS command on-the-fly from the
table rowtype?

Of course, we could just let these break or give odd results when we run
a SELECT if the foreign table doesn't match what's expected. file_fdw
will presumably break if the input file has rows with the wrong number
of columns, just as COPY does. But there will be cases, like the two
above, where a sanity check on the table shape could usefully be done at
validation time as opposed to run time, and it would be nice to be able
to do such a check.

I can't get excited about this. For one thing, you'd then need to worry
about involving the validator in random ALTER TABLE situations, not just
when changing the options it's supposed to be checking.

regards, tom lane

Andrew Dunstan <andrew@dunslane.net> writes:

On 02/21/2011 08:59 PM, Itagaki Takahiro wrote:

I think we need to overhaul validators in 9.2 listening to FDW developers'
opinions anyway.

Ok, I guess. It just seems to me like it will be harder to extend the
API later than now, so if we can reasonably foresee a likely need we
should try to provide for it.

Perhaps we should put a large friendly "EXPERIMENTAL, SUBJECT TO CHANGE"
notice on all the FDW API stuff? Just tell people up front that we're
not prepared to promise any API stability yet. There's stuff we *know*
is lacking (it's read-only, the optimization support sucks) in addition
to whatever we may later realize is misdesigned.

regards, tom lane

2011/2/22 Tom Lane <tgl@sss.pgh.pa.us>:

Andrew Dunstan <andrew@dunslane.net> writes:

On 02/21/2011 08:59 PM, Itagaki Takahiro wrote:

I think we need to overhaul validators in 9.2 listening to FDW developers'
opinions anyway.

Ok, I guess. It just seems to me like it will be harder to extend the
API later than now, so if we can reasonably foresee a likely need we
should try to provide for it.

Perhaps we should put a large friendly "EXPERIMENTAL, SUBJECT TO CHANGE"
notice on all the FDW API stuff?  Just tell people up front that we're
not prepared to promise any API stability yet.  There's stuff we *know*
is lacking (it's read-only, the optimization support sucks) in addition
to whatever we may later realize is misdesigned.

                       regards, tom lane

+1

regards

Pavel Stehule