keyword list/ecpg

Michael Meskes <meskes@postgresql.org> writes:

I recently (on my flight to Ottawa) changed ecpg to use the keyword list
of the backend instead of its own. This means that there is one less
file to sync manually. However, it also means that an additional keyword
defined in the backend will break compilation as ecpg doesn't have this
definition in its parser.

Does anyone object to moving the symbol definition part to a file of its
own that's inluded by both parser? Or does anyone have a better idea?

AFAIK bison doesn't have an "include" capability, so I'm not sure how
you plan to make that work?

regards, tom lane

On Mon, May 26, 2008 at 04:15:18PM -0400, Tom Lane wrote:

Does anyone object to moving the symbol definition part to a file of its
own that's inluded by both parser? Or does anyone have a better idea?

AFAIK bison doesn't have an "include" capability, so I'm not sure how
you plan to make that work?

You found the weak part in my idea. :-)

How about a small perl script that is run before bison and creates the
real grammar file?

Michael
--
Michael Meskes
Email: Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
ICQ: 179140304, AIM/Yahoo: michaelmeskes, Jabber: meskes@jabber.org
Go VfL Borussia! Go SF 49ers! Use Debian GNU/Linux! Use PostgreSQL!

On Tue, 2008-05-27 at 10:03 +0200, Michael Meskes wrote:

On Mon, May 26, 2008 at 04:15:18PM -0400, Tom Lane wrote:

Does anyone object to moving the symbol definition part to a file of its
own that's inluded by both parser? Or does anyone have a better idea?

AFAIK bison doesn't have an "include" capability, so I'm not sure how
you plan to make that work?

You found the weak part in my idea. :-)

How about a small perl script that is run before bison and creates the
real grammar file?

man cpp ?

-------------
Hannu

On Tue, May 27, 2008 at 11:23:28AM +0300, Hannu Krosing wrote:

How about a small perl script that is run before bison and creates the
real grammar file?

man cpp ?

As this would handle all includes and defines I'm unsure whether it is
desirable.

Michael
--
Michael Meskes
Email: Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
ICQ: 179140304, AIM/Yahoo: michaelmeskes, Jabber: meskes@jabber.org
Go VfL Borussia! Go SF 49ers! Use Debian GNU/Linux! Use PostgreSQL!

Michael Meskes <meskes@postgresql.org> writes:

On Mon, May 26, 2008 at 04:15:18PM -0400, Tom Lane wrote:

AFAIK bison doesn't have an "include" capability, so I'm not sure how
you plan to make that work?

You found the weak part in my idea. :-)

How about a small perl script that is run before bison and creates the
real grammar file?

If we were going to do that, I'd want it to go all the way and somehow
generate the common parts of the two .y files from a single source.
That'd be enough of a step forward that it would be worth whatever
ugliness is needed to make it happen ...

regards, tom lane

On Tue, May 27, 2008 at 09:44:02AM -0400, Tom Lane wrote:

If we were going to do that, I'd want it to go all the way and somehow
generate the common parts of the two .y files from a single source.
That'd be enough of a step forward that it would be worth whatever
ugliness is needed to make it happen ...

Any idea how to make this happen? We could of course do it with a lot of
ifdef like sequences:

rule {
IF ECPGCOMPILE
ecpg stuff
ELSE
backend stuff
ENDIF
}

Would you go into this direction?

Michael
--
Michael Meskes
Email: Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
ICQ: 179140304, AIM/Yahoo: michaelmeskes, Jabber: meskes@jabber.org
Go VfL Borussia! Go SF 49ers! Use Debian GNU/Linux! Use PostgreSQL!

Michael Meskes <meskes@postgresql.org> writes:

On Tue, May 27, 2008 at 09:44:02AM -0400, Tom Lane wrote:

If we were going to do that, I'd want it to go all the way and somehow
generate the common parts of the two .y files from a single source.

Any idea how to make this happen? We could of course do it with a lot of
ifdef like sequences:

rule {
IF ECPGCOMPILE
ecpg stuff
ELSE
backend stuff
ENDIF
}

Would you go into this direction?

Ugh :-(.

I have not spent much time looking at the ecpg grammar, so feel free to
laugh this off, but I had the impression that all the rules derived from
the backend grammar have boilerplate action sections (ie, just join the
strings together). So I was hoping that we could leave the backend's
.y file more or less as-is, and write a perl script that would go
through it and replace each { ... } action with a suitable cat_str call,
which it could build on-the-fly by counting the number of rule tokens.
Then combine that output with the ecpg-specific rules taken from a
separate source file. Obviously there would have to be a few small
kluges, but an #if-like mechanism might suffice for those. An #if
in every rule is way past my tolerance for ugliness though, especially
when one arm of it is essentially mechanically-generatable code.

regards, tom lane

On Wed, Jun 04, 2008 at 10:21:19AM -0400, Tom Lane wrote:

Ugh :-(.

This is why I didn't want to go that route. :-)

I have not spent much time looking at the ecpg grammar, so feel free to
laugh this off, but I had the impression that all the rules derived from
the backend grammar have boilerplate action sections (ie, just join the

This is true.

strings together). So I was hoping that we could leave the backend's
.y file more or less as-is, and write a perl script that would go
through it and replace each { ... } action with a suitable cat_str call,
which it could build on-the-fly by counting the number of rule tokens.

There is some small magic to know when to have blanks in between and
when not, but that should be doable.

Then combine that output with the ecpg-specific rules taken from a
separate source file. Obviously there would have to be a few small

This might work. Anyone with good perl knowledge interested?

Michael

--
Michael Meskes
Email: Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
ICQ: 179140304, AIM/Yahoo: michaelmeskes, Jabber: meskes@jabber.org
Go VfL Borussia! Go SF 49ers! Use Debian GNU/Linux! Use PostgreSQL!

I wouldn't mind having a stab at this if you can expand on the 'magic'
required.

(I'm interested because I might be able to use the same logic to roll a third
version of the .y for Aubit4GL outside of the Postgresql tree)

On Wednesday 04 June 2008 16:11:49 Michael Meskes wrote:

On Wed, Jun 04, 2008 at 10:21:19AM -0400, Tom Lane wrote:

Ugh :-(.

This is why I didn't want to go that route. :-)

I have not spent much time looking at the ecpg grammar, so feel free to
laugh this off, but I had the impression that all the rules derived from
the backend grammar have boilerplate action sections (ie, just join the

This is true.

strings together). So I was hoping that we could leave the backend's
.y file more or less as-is, and write a perl script that would go
through it and replace each { ... } action with a suitable cat_str call,
which it could build on-the-fly by counting the number of rule tokens.

There is some small magic to know when to have blanks in between and
when not, but that should be doable.

Then combine that output with the ecpg-specific rules taken from a
separate source file. Obviously there would have to be a few small

This might work. Anyone with good perl knowledge interested?

Michael

--
Michael Meskes
Email: Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
ICQ: 179140304, AIM/Yahoo: michaelmeskes, Jabber: meskes@jabber.org
Go VfL Borussia! Go SF 49ers! Use Debian GNU/Linux! Use PostgreSQL!

--
Mike Aubury

http://www.aubit.com/
Aubit Computing Ltd is registered in England and Wales, Number: 3112827
Registered Address : Clayton House,59 Piccadilly,Manchester,M1 2AQ

It might depend on the tokens..
Are ">=", "++" etc single tokens ?

On Wednesday 04 June 2008 17:06:44 Tom Lane wrote:

Mike Aubury <mike.aubury@aubit.com> writes:

On Wednesday 04 June 2008 16:11:49 Michael Meskes wrote:

There is some small magic to know when to have blanks in between and
when not, but that should be doable.

I wouldn't mind having a stab at this if you can expand on the 'magic'
required.

Wouldn't it work to just always insert a space between tokens, no matter
whether there was one originally?

regards, tom lane

--
Mike Aubury

http://www.aubit.com/
Aubit Computing Ltd is registered in England and Wales, Number: 3112827
Registered Address : Clayton House,59 Piccadilly,Manchester,M1 2AQ

Mike Aubury <mike.aubury@aubit.com> writes:

On Wednesday 04 June 2008 16:11:49 Michael Meskes wrote:

There is some small magic to know when to have blanks in between and
when not, but that should be doable.

I wouldn't mind having a stab at this if you can expand on the 'magic'
required.

Wouldn't it work to just always insert a space between tokens, no matter
whether there was one originally?

regards, tom lane

[Sorry, just noticed that I didn't answer this email. ]

On Wed, Jun 04, 2008 at 05:06:41PM +0100, Mike Aubury wrote:

It might depend on the tokens..
Are ">=", "++" etc single tokens ?
...

Wouldn't it work to just always insert a space between tokens, no matter
whether there was one originally?

There are a few cases where you must not enter a blank, but I'm not sure
whethere these are all in ecpg specific rules anyway. One example that
comes to my mind is the handling of ":<port>" in the connect statement.

Michael
--
Michael Meskes
Email: Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
ICQ: 179140304, AIM/Yahoo: michaelmeskes, Jabber: meskes@jabber.org
Go VfL Borussia! Go SF 49ers! Use Debian GNU/Linux! Use PostgreSQL!

I took a quick look at this - would it be ok to add some small amounts
of 'markup' to the gram.y ?

eg :

/* ECPGCOPYON */
....
/* ECPGCOPYOFF */

/* ECPGMODE=NOSPACE */
...
/* ECPGMODE=USESPACE */

etc ?

On Friday 13 June 2008 10:47:55 Michael Meskes wrote:

[Sorry, just noticed that I didn't answer this email. ]

On Wed, Jun 04, 2008 at 05:06:41PM +0100, Mike Aubury wrote:

It might depend on the tokens..
Are ">=", "++" etc single tokens ?
...

Wouldn't it work to just always insert a space between tokens, no
matter whether there was one originally?

There are a few cases where you must not enter a blank, but I'm not sure
whethere these are all in ecpg specific rules anyway. One example that
comes to my mind is the handling of ":<port>" in the connect statement.

Michael
--
Michael Meskes
Email: Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
ICQ: 179140304, AIM/Yahoo: michaelmeskes, Jabber: meskes@jabber.org
Go VfL Borussia! Go SF 49ers! Use Debian GNU/Linux! Use PostgreSQL!

--
Mike Aubury

http://www.aubit.com/
Aubit Computing Ltd is registered in England and Wales, Number: 3112827
Registered Address : Clayton House,59 Piccadilly,Manchester,M1 2AQ

Mike Aubury <mike.aubury@aubit.com> writes:

I took a quick look at this - would it be ok to add some small amounts
of 'markup' to the gram.y ?
eg :

/* ECPGCOPYON */
....
/* ECPGCOPYOFF */

/* ECPGMODE=NOSPACE */
...
/* ECPGMODE=USESPACE */

We're almost certainly going to need some kluges of that sort, so as
long as they're not all over the place I won't object.

But ... I've seen no evidence that those specific examples are needed.
Why wouldn't we copy all the backend rules? And based on Michael's last
comment it's unclear that we need to avoid adding spaces in the
mechanically generated actions, either (which squares with my gut
feeling about SQL syntax). You'll probably need to get into specific
cases before finding out what kluges you need.

regards, tom lane

We're almost certainly going to need some kluges of that sort, so as
long as they're not all over the place I won't object.

But ... I've seen no evidence that those specific examples are needed.
Why wouldn't we copy all the backend rules? And based on Michael's last
comment it's unclear that we need to avoid adding spaces in the
mechanically generated actions, either (which squares with my gut
feeling about SQL syntax). You'll probably need to get into specific
cases before finding out what kluges you need.

I think this was more an 'in principle' - if thats route is ok, then I'll
start hacking away properly...

I was thinking about the copy on/copy off for more the header info (before
the %%) - so we can have a really dumb script that just gets told what blocks
to copy - and what to ignore..

There will also be some grammer in the original which we'll need to replace
with some ecpg specifics - eg adding grammer for the variables etc.

Might be easier to just turn 'off' the original rules and have some custom
ecpg stuff appended to the generated code..

Theres also another thing that needs to be decided, which is if the generated
ecpg grammer should be developer generated (ie. Michael Meskes runs a script
and commits the output), or should be generated for each and every source
based installation. I personally would stongly favour the script being a tool
for ecpg tool developers and not used as part of a normal installation.

--
Mike Aubury

http://www.aubit.com/
Aubit Computing Ltd is registered in England and Wales, Number: 3112827
Registered Address : Clayton House,59 Piccadilly,Manchester,M1 2AQ

Mike Aubury wrote:

Theres also another thing that needs to be decided, which is if the generated
ecpg grammer should be developer generated (ie. Michael Meskes runs a script
and commits the output), or should be generated for each and every source
based installation. I personally would stongly favour the script being a tool
for ecpg tool developers and not used as part of a normal installation.

What happens when a non-Michael developer changes the original gram.y?
Is he expected to run the script before committing too? That sounds
brittle to me.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

The same as happens at the moment - nothing...

The grammer for the ecpg needs to be re-generated when the grammer in the main
parser is changed - whether its a manual or (mostly) automatic task is
largely irrelevant.

The only downside is that if its not regenerated then the change to gram.y
simply wont be reflected in the grammer for ecpg.

I personally think its down to the ecpg developers (of which I believe Michael
is the main developer) to decide when to do this and to check that its
worked.

Its just otherwise - there could be a serious case for 'unintended
consequences'...

Just my 2 pence worth...

On Friday 13 June 2008 15:39:48 Alvaro Herrera wrote:

I personally would stongly favour
the script being a tool for ecpg tool developers and not used as part of
a normal installation.

What happens when a non-Michael developer changes the original gram.y?
Is he expected to run the script before committing too? That sounds
brittle to me.

--
Mike Aubury

http://www.aubit.com/
Aubit Computing Ltd is registered in England and Wales, Number: 3112827
Registered Address : Clayton House,59 Piccadilly,Manchester,M1 2AQ

Alvaro Herrera <alvherre@commandprompt.com> writes:

Mike Aubury wrote:

Theres also another thing that needs to be decided, which is if the generated
ecpg grammer should be developer generated (ie. Michael Meskes runs a script
and commits the output), or should be generated for each and every source
based installation. I personally would stongly favour the script being a tool
for ecpg tool developers and not used as part of a normal installation.

What happens when a non-Michael developer changes the original gram.y?
Is he expected to run the script before committing too? That sounds
brittle to me.

As long as the script is written in Perl and not exceedingly slow,
I see no reason it shouldn't be required to run as part of a build from
CVS. We already require Perl use elsewhere in the build if you're not
working from a tarball.

regards, tom lane

Mike Aubury <mike.aubury@aubit.com> writes:

I was thinking about the copy on/copy off for more the header info (before
the %%) - so we can have a really dumb script that just gets told what blocks
to copy - and what to ignore..

I think you'll find that doesn't work very well; the script will need at
least some understanding of the bison %-declarations, and it can just
ignore everything else before %%. The reason is that (1) you're going
to need to alter the %type declarations and (2) you're going to need to
merge these declarations with ones from ecpg.

It might be that the merging is best handled via directive-like
comments, but I suspect they'll be in the ecpg-side file not in the
main grammar: you'll have something indicating "insert all the main
%type declarations here", one for "insert precedence declarations
here", etc.

But of course this is just speculation...

regards, tom lane

Mike Aubury <mike.aubury@aubit.com> writes:

What happens when a non-Michael developer changes the original gram.y?

The same as happens at the moment - nothing...

Wrong answer. The entire point of this work is to get rid of manual
work in updating the ecpg grammar when the main changes. The above
seems about like saying that the build process shouldn't regenerate
gram.c from gram.y --- you're supposed to do that manually.

One of the things I want to have come out of this is that we find out
right away if a main-grammar change breaks ecpg. Right now, any
conflict is not discovered until Michael gets around to syncing the
files, which is often weeks or months later.

regards, tom lane

On Fri, Jun 13, 2008 at 02:57:54PM +0100, Mike Aubury wrote:

based installation. I personally would stongly favour the script being a tool
for ecpg tool developers and not used as part of a normal installation.

Why?

Michael
--
Michael Meskes
Email: Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
ICQ: 179140304, AIM/Yahoo: michaelmeskes, Jabber: meskes@jabber.org
Go VfL Borussia! Go SF 49ers! Use Debian GNU/Linux! Use PostgreSQL!

On Fri, Jun 13, 2008 at 10:39:48AM -0400, Alvaro Herrera wrote:

What happens when a non-Michael developer changes the original gram.y?
Is he expected to run the script before committing too? That sounds
brittle to me.

The situation used to be that this only caused ecpg to be out-of-sync.
No big deal. But nowadays it might break compilation. If you add a new
keyword to keywords.c but not to preproc.y you'll get an undefined
symbol during build process.

To make sure stuff like this cannot happen in the future I'd prefer to
have the script run automatically, albeit having a defined, but maybe
non-functional, default.

Michael
--
Michael Meskes
Email: Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
ICQ: 179140304, AIM/Yahoo: michaelmeskes, Jabber: meskes@jabber.org
Go VfL Borussia! Go SF 49ers! Use Debian GNU/Linux! Use PostgreSQL!