ALTER OBJECT any_name SET SCHEMA name

Dimitri Fontaine <dimitri@2ndQuadrant.fr> writes:

As soon as we have that ability, we are able to provide for relocatable
extensions with the following command:

ALTER EXTENSION ext SET SCHEMA name;
ALTER EXTENSION ext SET SCHEMA foo TO bar;

I think that would end the open debate about search_path vs extension,
because each user would be able to relocate his local extensions easily,
wherever the main script has installed them (often enough, public).

I'm not sure whether that really fixes anything, or just provides people
with a larger-caliber foot-gun. See for example recent complaints about
citext misbehaving if it's not in the public schema (or more generally,
any schema not in the search path). I think we'd need to think a bit
harder about the behavior of objects that aren't in the search path
before creating a facility like this, since it seems to be tantamount
to promising that extensions won't break when pushed around to different
schemas.

I'm also a bit less than enthused about the implementation approach.
If we're going to have a policy that every object type must support
ALTER SET SCHEMA, I think it might be time to refactor, rather than
copying-and-pasting similar boilerplate code for every one.

regards, tom lane

Hi,

Thanks for your review!

Tom Lane <tgl@sss.pgh.pa.us> writes:

Dimitri Fontaine <dimitri@2ndQuadrant.fr> writes:

ALTER EXTENSION ext SET SCHEMA name;
ALTER EXTENSION ext SET SCHEMA foo TO bar;

I think that would end the open debate about search_path vs extension,
because each user would be able to relocate his local extensions easily,
wherever the main script has installed them (often enough, public).

I'm not sure whether that really fixes anything, or just provides people
with a larger-caliber foot-gun. See for example recent complaints about
citext misbehaving if it's not in the public schema (or more generally,
any schema not in the search path). I think we'd need to think a bit
harder about the behavior of objects that aren't in the search path
before creating a facility like this, since it seems to be tantamount
to promising that extensions won't break when pushed around to different
schemas.

Well AFAIK we have only two choices here. Either we impose the schema
where to find any extension's object or we offer more facilities for
users to support their choices.

In the former case, we have problems with upgrades and hosting several
versions of the same extension at the same time (think PostGIS 1.4 and
1.5 e.g.). Not being able to choose a schema where to host an
extension's objects only makes sense when the user can't set the
search_path, which is what we do with pg_catalog.

In the latter case, following your example, all it would take the user
to fix his setup would be a SET SCHEMA command on the citext extension.
The only goal of this proposal is not to have to rethink object
visibility, by offering the tools users need to manage the situation.

All in all, I don't think it'll be ever possible to both support
search_path flexibility and extension robustness when search_path
changes.

What we could do is offer extension's author a way to find their
operator or functions or whatever dynamically in SQL, so that writing
robust pure-SQL functions is possible. What comes to mind now would be a
way to call a function/operator/... by OID at the SQL level. Not pretty
but with the pg_extension_objects() SRF and maybe a layer atop that,
that would do the trick. Brain dumping still.

I'm also a bit less than enthused about the implementation approach.
If we're going to have a policy that every object type must support
ALTER SET SCHEMA, I think it might be time to refactor, rather than
copying-and-pasting similar boilerplate code for every one.

I've begun the effort with the CheckSetNamespace() function, about half
of the rest of the code could receive some abstraction too (syscache
searches, tuple access and editing, dependency changing, but ACL checks
looks harder. How much easier would it be with defmacro...

I'll have a try at it soon, happy to hear what you have in mind here
before I start, so that I follow your guidance.

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

--On 30. Oktober 2010 18:59:30 -0400 Tom Lane <tgl@sss.pgh.pa.us> wrote:

I'm not sure whether that really fixes anything, or just provides people
with a larger-caliber foot-gun. See for example recent complaints about
citext misbehaving if it's not in the public schema (or more generally,
any schema not in the search path). I think we'd need to think a bit
harder about the behavior of objects that aren't in the search path
before creating a facility like this, since it seems to be tantamount
to promising that extensions won't break when pushed around to different
schemas.

I'm also a bit less than enthused about the implementation approach.
If we're going to have a policy that every object type must support
ALTER SET SCHEMA, I think it might be time to refactor, rather than
copying-and-pasting similar boilerplate code for every one.

This reminds me of a small discussion we had some years ago when i targeted
this for the sake of completeness of ASS (see
<http://archives.postgresql.org/pgsql-patches/2006-06/msg00021.php&gt;).

I didn't follow the previous discussions about EXTENSION very closely, but
to amend the idea made in the mentioned thread, couldn't we just invent a
facility to move classes of objects belonging to an extension, only?

--
Thanks

Bernd

Bernd Helmle <mailings@oopsware.de> writes:

This reminds me of a small discussion we had some years ago when i targeted
this for the sake of completeness of ASS (see
<http://archives.postgresql.org/pgsql-patches/2006-06/msg00021.php&gt;).

Discovered it, thanks for the pointer.

I didn't follow the previous discussions about EXTENSION very closely, but
to amend the idea made in the mentioned thread, couldn't we just invent a
facility to move classes of objects belonging to an extension, only?

Well under the hood you still need about the same code. In the current
patch version I have, that would mean forgetting about a ~20 lines long
function and keeping the other two, 10+30 lines about.

Now if someone has a idea on how to have a single routine that knows how
to deal with any object type and change its namespace, I'm all ears, and
willing to have a try at it. Coding in C is for me a twice a year
exercise, so whatever I come up with will be... improvable, I fear.

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

On 31.10.2010 14:46, Dimitri Fontaine wrote:

What we could do is offer extension's author a way to find their
operator or functions or whatever dynamically in SQL, so that writing
robust pure-SQL functions is possible. What comes to mind now would be a
way to call a function/operator/... by OID at the SQL level. Not pretty
but with the pg_extension_objects() SRF and maybe a layer atop that,
that would do the trick. Brain dumping still.

How about something like:

CREATE EXTENSION myextension ... SCHEMA myschema;

And in the .sql file in the extension you could have special markers for
the schema, something like:

CREATE FUNCTION otherfunction() AS ...;
CREATE FUNCTION foo() AS $$ SELECT 'foo', @extschema@.otherfunction() $$;

@extschema@ would be search&replaced at CREATE EXTENSION time with the
schema specified by the user.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> writes:

CREATE EXTENSION myextension ... SCHEMA myschema;

And in the .sql file in the extension you could have special markers for the
schema, something like:

That's exactly the road I want to avoid, because of the script parsing issues.

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

Sorry for the interruption, our program now continues...

Dimitri Fontaine <dimitri@2ndQuadrant.fr> writes:

That's exactly the road I want to avoid, because of the script parsing issues.

In particular, embedded and/or dynamic calls in PLs will get hairy if
not turing complete and outright impossible to solve.

--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

On 31.10.2010 19:38, Dimitri Fontaine wrote:

Sorry for the interruption, our program now continues...

Dimitri Fontaine<dimitri@2ndQuadrant.fr> writes:

That's exactly the road I want to avoid, because of the script parsing issues.

In particular, embedded and/or dynamic calls in PLs will get hairy if
not turing complete and outright impossible to solve.

Sorry, I don't follow. Got an example?

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> writes:

In particular, embedded and/or dynamic calls in PLs will get hairy if
not turing complete and outright impossible to solve.

Sorry, I don't follow. Got an example?

Well, who's to say the following hypothetical plpgsql example should be
forgiven only in an exception's script?

v_sql := 'SELECT * FROM ' || p_fun || '()';
FOR rec in EXECUTE v_sql
LOOP
…
END LOOP;

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

On Sun, Oct 31, 2010 at 12:45 PM, Dimitri Fontaine
<dimitri@2ndquadrant.fr> wrote:

Bernd Helmle <mailings@oopsware.de> writes:

This reminds me of a small discussion we had some years ago when i targeted
this for the sake of completeness of ASS (see
<http://archives.postgresql.org/pgsql-patches/2006-06/msg00021.php&gt;).

Discovered it, thanks for the pointer.

I didn't follow the previous discussions about EXTENSION very closely, but
to amend the idea made in the mentioned thread, couldn't we just invent a
facility to move classes of objects belonging to an extension, only?

Well under the hood you still need about the same code. In the current
patch version I have, that would mean forgetting about a ~20 lines long
function and keeping the other two, 10+30 lines about.

Now if someone has a idea on how to have a single routine that knows how
to deal with any object type and change its namespace, I'm all ears, and
willing to have a try at it. Coding in C is for me a twice a year
exercise, so whatever I come up with will be... improvable, I fear.

I guess it's a question of how much special case code there is. The
new objectaddress.c code knows how to deal with lots of different
object types, but for example in the case of ALTER TABLE .. SET
SCHEMA, what's there now is enmeshed in a complex structure so as to
allow multiple ALTER TABLE subcommands in a single statement, and it
has further special-case logic to handle moving the rowtype and
related indexes, sequences, and constraints. It seems hard to fit
that into a general framework, but maybe it could be done for other
object types.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 31.10.2010 20:19, Dimitri Fontaine wrote:

Heikki Linnakangas<heikki.linnakangas@enterprisedb.com> writes:

In particular, embedded and/or dynamic calls in PLs will get hairy if
not turing complete and outright impossible to solve.

Sorry, I don't follow. Got an example?

Well, who's to say the following hypothetical plpgsql example should be
forgiven only in an exception's script?

v_sql := 'SELECT * FROM ' || p_fun || '()';
FOR rec in EXECUTE v_sql
LOOP
…
END LOOP;

If I understand that correctly, the idea is that p_fun holds the name of
a function that's in the same schema as the extension? You would write
that as

v_sql := 'SELECT * FROM @extschema@.' || p_fun || '()';
FOR rec in EXECUTE v_sql
LOOP
…
END LOOP;

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Robert Haas <robertmhaas@gmail.com> writes:
...

related indexes, sequences, and constraints. It seems hard to fit
that into a general framework, but maybe it could be done for other
object types.

My guess is that we're talking about having a generic code that would
get exercised directly from src/backend/commands/alter.c in the function
ExecAlterObjectSchemaStmt(AlterObjectSchemaStmt *stmt), rather than
having a switch() statement and very similar functions implemented all
over the place to do the actual bit editing.

Let's see some details, if we're getting there.

This line of code is repeated in lots of places, but always a little different:

+ conversionOid = get_conversion_oid(name, false);

+ 	operOid = LookupOperNameTypeNames(NULL, operatorName,
+ 									  typeName1, typeName2,
+ 									  false, -1);

+ prsId = get_ts_parser_oid(name, false);

+ dictId = get_ts_dict_oid(name, false);

And for operator class and operator family, we're dealing with HeapTuple
objects directly, rather than Oids, because the lookup ain't so
easy. But all in all that code could get moved into another layer in the
generic function, in a object type switch, if we get to HeapTuple based
internal API, because next example shows that in most cases we get the
tuple from the Oid in the same way.

In the following code part, I can see how to get generic code with some
parameters for the first 3 lines, but not so much for the other 2 lines:

+ 	tup = SearchSysCacheCopy1(CONVOID, ObjectIdGetDatum(conversionOid));
+ 	if (!HeapTupleIsValid(tup)) /* should not happen */
+ 		elog(ERROR, "cache lookup failed for conversion %u", conversionOid);
+ 
+ 	convForm = (Form_pg_conversion) GETSTRUCT(tup);
+ 	oldNspOid = convForm->connamespace;

Then there's this code which looks even harder to get generic:

+ 		/* Otherwise, must be owner of the existing object */
+ 		if (!pg_conversion_ownercheck(HeapTupleGetOid(tup), GetUserId()))
+ 			aclcheck_error(ACLCHECK_NOT_OWNER, ACL_KIND_CONVERSION,
+ 						   NameStr(convForm->conname));

Now the following part could get moved easily to the CheckSetNamespace()
function introduced in the patch, with a boolean to trigger the check,
because as it is not all objects are open to non-superuser changes.

+ 		/* New owner must have CREATE privilege on namespace */
+ 		aclresult = pg_namespace_aclcheck(nspOid, GetUserId(), ACL_CREATE);
+ 		if (aclresult != ACLCHECK_OK)
+ 			aclcheck_error(aclresult, ACL_KIND_NAMESPACE,
+ 						   get_namespace_name(nspOid));

Then in the end of the function, we have the real work, where I can see
a generic function for the heap update and the dependency tracking, but
not for the tuple editing itself:

+ 	/* tup is a copy, so we can scribble directly on it */
+ 	opfForm->opfnamespace = nspOid;
+ 
+ 	simple_heap_update(rel, &tup->t_self, tup);
+ 	CatalogUpdateIndexes(rel, tup);
+ 
+ 	/* update dependencies to point to the new schema */
+ 	changeDependencyFor(OperatorFamilyRelationId, HeapTupleGetOid(tup),
+ 						NamespaceRelationId, oldNspOid, nspOid);

So it seems to get down to C'fu related to handling Form_pg_* pointers
to get to values and change them then simple_heap_update the tuple. The
other parts are all about the same code with different RelationId and
such.

Well of course you think you could just have a pointer as argument, but
how to get the pointer is something we'd prefer generic too. Maybe we
need a 3-step code here, in aforementioned ExecAlterObjectSchemaStmt:

1. call a generic function that switch on stmt->objectType and
returns an HeapTuple

2. switch on stmt->objectType to get oldNspOid from the tuple and to
check for permissions

3. call another generic function to do the namespace related checks
then the heap update etc

This way the only specific code we need to maintain are the second step
here, object dependent, and in the first function too, but that's not an
entire API like in current patch.

Well of course there's the question of how to do that in a way that
allows some other backend's code to get the action done with Oids as
input rather than a statement, but that looks workable: get from classid
to objectype (aren't they equal?) then call the step 1. function to get
the HeapTyple, and proceed. So in fact the 3 steps are a separate
function that is called either from ExecAlterObjectSchemaStmt or from
the extension's code.

And there's also the question whether that has anything to do with what
Tom would want to see happening, too (or I'll be talking about those
ideas with gcc rather than with the list) :)

Comments?
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> writes:

If I understand that correctly, the idea is that p_fun holds the name of a
function that's in the same schema as the extension? You would write that as

v_sql := 'SELECT * FROM @extschema@.' || p_fun || '()';

Fair enough. Now what about the citext example, where IIRC the failure
is not on function names but operators and opclass not found, etc.

Forcing extension's author to get to always use the following notation
seems to me like pushing it:

- WHERE foo = bar
+ WHERE foo operator(@extschema@.=) bar

Also, those examples are plpgsql but extensions are free to depend on
plperl or plpython, or even some pljs or plscheme out there.

The alternative is to give DBAs the tools to move their local extensions
in some schema that is part of their edited search_path, should they
remove public from there.

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

On 31.10.2010 21:42, Dimitri Fontaine wrote:

Heikki Linnakangas<heikki.linnakangas@enterprisedb.com> writes:

If I understand that correctly, the idea is that p_fun holds the name of a
function that's in the same schema as the extension? You would write that as

v_sql := 'SELECT * FROM @extschema@.' || p_fun || '()';

Fair enough. Now what about the citext example, where IIRC the failure
is not on function names but operators and opclass not found, etc.

Just do "SET search_path=@extschema@" at the beginning of the install
script, just like we have "SET search_path=public" there now.

Forcing extension's author to get to always use the following notation
seems to me like pushing it:

- WHERE foo = bar
+ WHERE foo operator(@extschema@.=) bar

Also, those examples are plpgsql but extensions are free to depend on
plperl or plpython, or even some pljs or plscheme out there.

Well, in case of functions you can always do "CREATE FUNCTION ... AS $$
... $$ SET search_path=@extschema".

"ALTER ... SET SCHEMA" wouldn't do anything for SQL statements embedded
in plperl or plpython anyway.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> writes:

Just do "SET search_path=@extschema@" at the beginning of the install
script, just like we have "SET search_path=public" there now.

Well there's the installation itself then the "runtime", as you say
later...

Well, in case of functions you can always do "CREATE FUNCTION ... AS $$
... $$ SET search_path=@extschema".

Fair enough.

"ALTER ... SET SCHEMA" wouldn't do anything for SQL statements embedded in
plperl or plpython anyway.

That's why I was thinking about adding the possibility to:
- easily find your function's etc OID, that's already mainly done
- be able to call/use those objects per OID

Ok that sucks somehow. I think it's better than @extschema@ replacing in
the extension's script parsing, though.

Maybe we should just shut down this attempt at working on search_path
and extensions together, again. I though it was a simple and good enough
solution though, and that it would avoid the usual rat holes. But we're
deep in them already.

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

On Sun, Oct 31, 2010 at 5:46 PM, Dimitri Fontaine
<dimitri@2ndquadrant.fr> wrote:

Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> writes:

Just do "SET search_path=@extschema@" at the beginning of the install
script, just like we have "SET search_path=public" there now.

Well there's the installation itself then the "runtime", as you say
later...

Well, in case of functions you can always do "CREATE FUNCTION ... AS $$
... $$ SET search_path=@extschema".

Fair enough.

"ALTER ... SET SCHEMA" wouldn't do anything for SQL statements embedded in
plperl or plpython anyway.

That's why I was thinking about adding the possibility to:
 - easily find your function's etc OID, that's already mainly done
 - be able to call/use those objects per OID

Ok that sucks somehow.

Yeah, I think that sucks a lot. I don't see what's wrong with
Heikki's solution, actually.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas <robertmhaas@gmail.com> writes:

Yeah, I think that sucks a lot. I don't see what's wrong with
Heikki's solution, actually.

Coding the parser and replace. If all it takes is calling our replace
function on the all-in-memory query string that we have in
pg_execute_from_file() function, I can have a try at it.

Main wrong point is that it puts the burden on the extension's authors
rather than on the one who manages the search_path for its
applications...
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> writes:

How about something like:

CREATE EXTENSION myextension ... SCHEMA myschema;

And in the .sql file in the extension you could have special markers for the
schema, something like:

CREATE FUNCTION otherfunction() AS ...;
CREATE FUNCTION foo() AS $$ SELECT 'foo', @extschema@.otherfunction() $$;

@extschema@ would be search&replaced at CREATE EXTENSION time with the
schema specified by the user.

Please find attached v12 of the patch, which implements that idea.

And a new pg_execute_from_file patch version too: the function now has a
second (documented) variant accepting a VARIADIC text[] argument where
to put pairs of name and value for the placeholders in the script.

I guess it would be cleaner with hstore in core, but we're not there
yet, so meanwhile it's a variable length array.

The CREATE EXTENSION ... WITH SCHEMA ... command will then use the
variadic form of pg_execute_from_file() with a single variable in there,
the proposed @extschema@. When the option is not used, the placeholder
is still set, hard-coded to 'public'.

Contrib scripts have been all changed this way:

- SET search_path = public;
+ SET search_path = @extschema@;

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

Dimitri Fontaine <dimitri@2ndQuadrant.fr> writes:

Please find attached v12 of the patch, which implements that idea.

And v13 now. v12 was intended to see what you think about the new
pg_execute_from_file placeholder API and replace_text usage, v13 fixes
the pg_dump support by adding dependencies.

Also, I've been changing the \dx output to show the schema where an
extension's objects are living rather than the custom_variable_classes
that most users won't care about, I think.

Then, I think the ALTER EXTENSION foo SET SCHEMA name still has a use
case, so I've prepared a simple patch to show the API usage before we
get to refactor it all following Tom's asking. So there's a initial
patch to see that in action. I had to rework AlterFunctionNamespace()
API so that I can call it from elsewhere in the backend where I have
Oids, so here's an updated set_schema.4.patch. We will have to extend
the APIs for relations and types the same way, but it's already possible
to test the patch with some extensions this way.

Producing those patches (the alter_extension patch is an incremental
patch that sits atop both the extension and the set_schema one) is made
easy enough with git, I'm impressed by this tool.

http://git.postgresql.org/gitweb?p=postgresql-extension.git;a=summary

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support