idea, proposal: only preloadable libraries (conditional load)

On Wed, 2009-03-11 at 11:09 +0100, Pavel Stehule wrote:

I am searching way to extensible parser (actually not bison, only
transformations). I propose parserHook (transformation part). One
Tom's objection is difference between heooked and unhooked parser. It
serious problem.

Do you mean hooking the whole parser? That sounds more useful and less
hassle than trying to hook parts of it. That would be just one check to
see if the hook exists per statement, rather than potentially thousands
of times per statement.

I'd go for an implementation that uses pg_language to store new
languages, just with lanispl = false. We can then have a new parameter
session_language (TEXT) with 'internal' as default. session_language
cannot be reset while connected. That would allow us to have multiple
session languages in use at one time and to add new ones (or modify
existing ones) without changing core behaviour.

In the longer term it will be very useful to have the ability to support
multiple language variants, including older PostgreSQL syntax to allow
legacy systems to work with Postgres at the same time as allowing new
development to continue.

I have one idea - only preloadable libraries. These libs have to be
specified in local_preload_libraries variable, and cannot be
initialised by LOAD statement. Statement LOAD should by used only for
reload library. Because pre loaded library is initialised before first
SQL statement should be executed, then hook is installed on time and
an behave is consistent. One technique is an calling library's
function "canload" before initialising. When canload returns false,
then dfmgr unloads lib. I thing so this behave complements current
functions PG_init and PG_finit. What I can understand, PG_init, cannot
throw exception or signalise any problems with initialisation.

I remember I had some differences between the way loading occurs at
session start and as a result of a LOAD command. I think there's
probably already a way of doing this - probably by checking for
something that would only be there *after* having read reloadable
libraries but before main session starts.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

On Wed, Mar 11, 2009 at 12:56 PM, Simon Riggs <simon@2ndquadrant.com> wrote:

In the longer term it will be very useful to have the ability to support
multiple language variants, including older PostgreSQL syntax to allow
legacy systems to work with Postgres at the same time as allowing new
development to continue.

So I think having multiple parsers for different versions of Pg
backwards compatibility is an awful idea. It would be a huge
maintenance headache since every time we change a structure that the
parser works someone would have to maintain all those compatibility
parsers. And it's very rare that we make non-backwards compatible
changes to the grammar with the exception of adding new reserved
keywords.

However that last thought led me to an interesting idea. We could
fairly easily support SQL which used keywords which we later reserved.
We could do this by marking each keyword in keywords.c with a version
number that introduced it. Then have a guc which tells the lexer which
version to target -- any keywords introduced after the desired version
can just be passed up as regular urecognized identifiers.

That would allow us to add new keywords more freely -- I think still
not liberally since we would rather people not be forced to decide
between new features and a working application.

Hm, actually I see a fly in the ointment -- we often upgrade keywords
from one kind of reservedness to another. That would mean we wouldn't
be able to handle something like WITH which was previously some
flavour of unreserved keyword and later became reserved.

--
greg

2009/3/11 Simon Riggs <simon@2ndquadrant.com>:

On Wed, 2009-03-11 at 11:09 +0100, Pavel Stehule wrote:

I am searching way to extensible parser (actually not bison, only
transformations). I propose parserHook (transformation part). One
Tom's objection is difference between heooked and unhooked parser. It
serious problem.

Do you mean hooking the whole parser? That sounds more useful and less
hassle than trying to hook parts of it. That would be just one check to
see if the hook exists per statement, rather than potentially thousands
of times per statement.

No, now I want to add hook only to current parser - concretely to
transformations, although I can imagine any hook over whole parser. It
could help with modules, that adds non sql statements like "show
statistic", "show ... ", "explain " and others - all service
statements, some extensions ... Nearest goal is support for some smart
functions like decode, greatest, xmlelement, ...

I'd go for an implementation that uses pg_language to store new
languages, just with lanispl = false. We can then have a new parameter
session_language (TEXT) with 'internal' as default. session_language
cannot be reset while connected. That would allow us to have multiple
session languages in use at one time and to add new ones (or modify
existing ones) without changing core behaviour.

In the longer term it will be very useful to have the ability to support
multiple language variants, including older PostgreSQL syntax to allow
legacy systems to work with Postgres at the same time as allowing new
development to continue.

I have one idea - only preloadable libraries. These libs have to be
specified in local_preload_libraries variable, and cannot be
initialised by LOAD statement. Statement LOAD should by used only for
reload library. Because pre loaded library is initialised before first
SQL statement should be executed, then hook is installed on time and
an behave is consistent. One technique is an calling library's
function "canload" before initialising. When canload returns false,
then dfmgr unloads lib. I thing so this behave complements current
functions PG_init and PG_finit. What I can understand, PG_init, cannot
throw exception or signalise any problems with initialisation.

I remember I had some differences between the way loading occurs at
session start and as a result of a LOAD command. I think there's
probably already a way of doing this - probably by checking for
something that would only be there *after* having read reloadable
libraries but before main session starts.

I can test debug_query_string, but main problem is impossibility throw
exception inside PG_init.

regards
Pavel Stehule

On Wed, 2009-03-11 at 13:14 +0000, Greg Stark wrote:

On Wed, Mar 11, 2009 at 12:56 PM, Simon Riggs <simon@2ndquadrant.com> wrote:

In the longer term it will be very useful to have the ability to support
multiple language variants, including older PostgreSQL syntax to allow
legacy systems to work with Postgres at the same time as allowing new
development to continue.

So I think having multiple parsers for different versions of Pg
backwards compatibility is an awful idea.

There are a number of people interested in producing open source
compatibility layers. I realise you may not be one of them, but that's
no reason to stop the idea from taking root.

It would be a huge
maintenance headache since every time we change a structure that the
parser works someone would have to maintain all those compatibility
parsers.

If it's a plugin that "someone" isn't any concern of ours. External
projects can keep up with releases, or specific customer implementations
may simply choose to standardise on one release and go with that.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

On Wed, Mar 11, 2009 at 2:18 PM, Simon Riggs <simon@2ndquadrant.com> wrote:

It would be a huge
maintenance headache since every time we change a structure that the
parser works someone would have to maintain all those compatibility
parsers.

If it's a plugin that "someone" isn't any concern of ours. External
projects can keep up with releases, or specific customer implementations
may simply choose to standardise on one release and go with that.

That's not what I mean. I mean, for example, if someone adds a field
to any of the structss in parsenodes.h to implement a new feature. The
old parser would have to know how to initialize that field correctly
to avoid triggering that new feature or trigger it in a manner
compatible with the old version's implicit behaviour.

The last few commits to that file include Tom's commit to handle ALTER
TABLE SET WITHOUT OIDS, Alvaro's commit to handle reloptions with
qualifiers, Stephen Frost's patch to support column-level privileges,
Heikki's commit to handle vacuum_freeze_table_age, etc.

Every one of these commits would have had to adjust every single old
parser to generate the correct data for the changed nodes.

The parser isn't a separable module interacting with the rest of the
system through a static interface. It's closely in bed with the rest
of the system implementing the syntax it's parsing. Every feature the
parser can parse has to be communicated to the backend code
implementing the feature so it has to have a corresponding knob in the
interface between the parser and the rest of the system.

--
greg

On Wed, 2009-03-11 at 14:45 +0000, Greg Stark wrote:

On Wed, Mar 11, 2009 at 2:18 PM, Simon Riggs <simon@2ndquadrant.com> wrote:

It would be a huge
maintenance headache since every time we change a structure that the
parser works someone would have to maintain all those compatibility
parsers.

If it's a plugin that "someone" isn't any concern of ours. External
projects can keep up with releases, or specific customer implementations
may simply choose to standardise on one release and go with that.

That's not what I mean. I mean, for example, if someone adds a field
to any of the structss in parsenodes.h to implement a new feature. The
old parser would have to know how to initialize that field correctly
to avoid triggering that new feature or trigger it in a manner
compatible with the old version's implicit behaviour.

The last few commits to that file include Tom's commit to handle ALTER
TABLE SET WITHOUT OIDS, Alvaro's commit to handle reloptions with
qualifiers, Stephen Frost's patch to support column-level privileges,
Heikki's commit to handle vacuum_freeze_table_age, etc.

Every one of these commits would have had to adjust every single old
parser to generate the correct data for the changed nodes.

The parser isn't a separable module interacting with the rest of the
system through a static interface. It's closely in bed with the rest
of the system implementing the syntax it's parsing. Every feature the
parser can parse has to be communicated to the backend code
implementing the feature so it has to have a corresponding knob in the
interface between the parser and the rest of the system.

It would be a matter for a plugin designer how they did that. If the new
parser involved just some changes in specific areas, then presumably you
would design it as a drop through parser: handle any special cases or
drop through to normal Postgres parser.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Pavel Stehule <pavel.stehule@gmail.com> writes:

2009/3/11 Simon Riggs <simon@2ndquadrant.com>:

I remember I had some differences between the way loading occurs at
session start and as a result of a LOAD command. I think there's
probably already a way of doing this - probably by checking for
something that would only be there *after* having read reloadable
libraries but before main session starts.

I can test debug_query_string, but main problem is impossibility throw
exception inside PG_init.

If we can't support throwing an error there, I think we need to fix that.
There's no way a "precheck" function can completely guarantee that no
error will happen in the real "do it" function; at least not for
interesting values of "do it".

regards, tom lane