Coping with 'C' vs 'newC' function language names

Tom Lane writes:

Philip pointed out awhile back that it does not work to load a 7.0.*
dump into current sources if the dumped database contains any
procedural language definitions. The dumped handler-function
definitions will look like

CREATE FUNCTION "plpgsql_call_handler" ( ) RETURNS opaque AS
'/opt/postgres/lib/plpgsql.sl' LANGUAGE 'C';

which was correct for 7.0.* ... but under 7.1 the handler functions
use the new-style function manager API and therefore need to be
declared as LANGUAGE 'newC'.

I don't really have a better idea, but consider if you installed 7.1 into
/opt/postgres71: then this dump will load the old version of plpgsql.sl.
Assuming that that would work in the first place, LANGUAGE 'C' is correct.

Btw., could we use something other than 'newC'? It's going to get old
really fast (pun intended). Maybe 'Cv2' or something along these lines?

--
Peter Eisentraut peter_e@gmx.net http://yi.org/peter-e/

Peter Eisentraut <peter_e@gmx.net> writes:

I don't really have a better idea, but consider if you installed 7.1 into
/opt/postgres71: then this dump will load the old version of plpgsql.sl.

True, but absolute paths in a dump file are a different (and
long-standing) issue.

Assuming that that would work in the first place, LANGUAGE 'C' is correct.

It wouldn't work, so that's irrelevant. The PL handlers know way more
than the average user-defined function about backend innards, and aren't
usually cross-version compatible. They won't be this time, for sure.

Btw., could we use something other than 'newC'? It's going to get old
really fast (pun intended). Maybe 'Cv2' or something along these lines?

Where were you six months ago? ;-( It's a bit late in the dev cycle to
be running around renaming this kind of stuff...

regards, tom lane

So new-style C functions are language "newC"?

Peter Eisentraut <peter_e@gmx.net> writes:

I don't really have a better idea, but consider if you installed 7.1 into
/opt/postgres71: then this dump will load the old version of plpgsql.sl.

True, but absolute paths in a dump file are a different (and
long-standing) issue.

Assuming that that would work in the first place, LANGUAGE 'C' is correct.

It wouldn't work, so that's irrelevant. The PL handlers know way more
than the average user-defined function about backend innards, and aren't
usually cross-version compatible. They won't be this time, for sure.

Btw., could we use something other than 'newC'? It's going to get old
really fast (pun intended). Maybe 'Cv2' or something along these lines?

Where were you six months ago? ;-( It's a bit late in the dev cycle to
be running around renaming this kind of stuff...

regards, tom lane

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

At 12:07 10/11/00 -0500, Tom Lane wrote:

Something's got to be done about this --- not being able to load
7.0 dump files will not do.

I assume modifying pg_dump for 7.0.x is not an option, since we really need
to support loading databases from historical dump files. If so, then would
there be any value in modifying the CREATE LANGUAGE code to output a
warning for non-newC handlers? Or just updating the function definition
(with a WARNING) when the language is created?

At one time I had toyed with the idea of using 'C' to specify
dynamically-loaded functions that use the new-style fmgr API, and
'oldC' for old-style fmgr API. That would be nicer in the long run,
and it'd fix this particular problem, but it'd break dumped definitions
for user-defined C functions that haven't yet been updated to new-style
API. That doesn't seem like an acceptable tradeoff.

Another option in the long run would be to use language 'C' in all cases
and add an attribute which allows people to specify the function manager to
use. For 7.1 this would default to 'fmgr71' or some such. Then only
old-style functions would break as per your example (I dislike the idea of
the language changing when it is just the interface we modify).

In the even longer term, maybe the function manager should be able to
interrogate functions when they are first loaded, and ask them about the
interface they expect to use.

----------------------------------------------------------------
Philip Warner | __---_____
Albatross Consulting Pty. Ltd. |----/ - \
(A.B.N. 75 008 659 498) | /(@) ______---_
Tel: (+61) 0500 83 82 81 | _________ \
Fax: (+61) 0500 83 82 82 | ___________ |
Http://www.rhyme.com.au | / \|
| --________--
PGP key available upon request, | /
and from pgp5.ai.mit.edu:11371 |/

Philip Warner <pjw@rhyme.com.au> writes:

At 12:07 10/11/00 -0500, Tom Lane wrote:

Something's got to be done about this --- not being able to load
7.0 dump files will not do.

I assume modifying pg_dump for 7.0.x is not an option,

Not really ...

would
there be any value in modifying the CREATE LANGUAGE code to output a
warning for non-newC handlers? Or just updating the function definition
(with a WARNING) when the language is created?

Just outputting a warning is possible, but it still leaves you with a
broken database after you reload your 7.0 dump file :-(. (And there
isn't any supported way to fix it, short of reloading all your function
definitions...) I thought about the fix-handler-at-CREATE-LANGUAGE-time
option, but it doesn't seem noticeably cleaner than hacking CREATE
FUNCTION.

At one time I had toyed with the idea of using 'C' to specify
dynamically-loaded functions that use the new-style fmgr API, and
'oldC' for old-style fmgr API. That would be nicer in the long run,
and it'd fix this particular problem, but it'd break dumped definitions
for user-defined C functions that haven't yet been updated to new-style
API. That doesn't seem like an acceptable tradeoff.

Another option in the long run would be to use language 'C' in all cases
and add an attribute which allows people to specify the function manager to
use. For 7.1 this would default to 'fmgr71' or some such.

No, that just adds complexity without really accomplishing anything.

From the function manager's point of view, the "language" setting *is*

the interface, they're not separately twiddlable concepts.

More to the point, I think we have to assume old-style interface if we
see ... LANGUAGE 'C' with no other decoration, because any other
assumption is guaranteed to break all existing user-defined functions.

regards, tom lane

At 10:55 11/11/00 -0500, Tom Lane wrote:

Just outputting a warning is possible, but it still leaves you with a
broken database after you reload your 7.0 dump file :-(. (And there
isn't any supported way to fix it, short of reloading all your function
definitions...) I thought about the fix-handler-at-CREATE-LANGUAGE-time
option, but it doesn't seem noticeably cleaner than hacking CREATE
FUNCTION.

Is it the PL manager stuff that requires the new interface for all
languages, or is it up to the PL implementor to choose the language for
their handler?

Another option in the long run would be to use language 'C' in all cases
and add an attribute which allows people to specify the function manager to
use. For 7.1 this would default to 'fmgr71' or some such.

No, that just adds complexity without really accomplishing anything.

From the function manager's point of view, the "language" setting *is*

the interface, they're not separately twiddlable concepts.

It does avoid "language 'evenNewerC'" in the furture. By allowing optional
attributes to use older fmgr interfaces we preserve compatibility without
cluttering the list of languages. From the developer/user point of view,
the language is 'C' if it's written in C.

More to the point, I think we have to assume old-style interface if we
see ... LANGUAGE 'C' with no other decoration, because any other
assumption is guaranteed to break all existing user-defined functions.

Quite right. The (unspecified) plan was to allow an environmental setting
that set the default attr for "language 'C'" functions. Then restoring a
dump file would be a matter of setting this variable to use the old
interface at the start of the script.

What about (not for 7.1) adding the ability for the fmgr to query object
modules. eg. look for an entry point 'pg_fmgr_info' or similar, and if
found call it to get attrs of functions. Then the person defining the
function in the database is freed of the need for a large amount of
information about the functions. Might even be able to implement a 'CREATE
MODULE' which loads an object, enquires about functions in the module, and
creates function entries based on information returned from pg_fmgr_info...

----------------------------------------------------------------
Philip Warner | __---_____
Albatross Consulting Pty. Ltd. |----/ - \
(A.B.N. 75 008 659 498) | /(@) ______---_
Tel: (+61) 0500 83 82 81 | _________ \
Fax: (+61) 0500 83 82 82 | ___________ |
Http://www.rhyme.com.au | / \|
| --________--
PGP key available upon request, | /
and from pgp5.ai.mit.edu:11371 |/

Philip Warner <pjw@rhyme.com.au> writes:

Is it the PL manager stuff that requires the new interface for all
languages, or is it up to the PL implementor to choose the language for
their handler?

The current fmgr implementation requires PL handlers to be newC or
newInternal. That could be relaxed, but I don't see any reason to do
so, since an old-style handler would be incapable of handling null
arguments/results or supporting triggers.

No, that just adds complexity without really accomplishing anything.

From the function manager's point of view, the "language" setting *is*

the interface, they're not separately twiddlable concepts.

It does avoid "language 'evenNewerC'" in the furture.

How so? It appears to me it would just move the 'evenNewerFoo'
dirtiness to a different keyword, which would still be essential
for the user to write. Net result: no gain, just more writing.

By allowing optional
attributes to use older fmgr interfaces we preserve compatibility without
cluttering the list of languages. From the developer/user point of view,
the language is 'C' if it's written in C.

No. 'C' is really a misnomer, since it does NOT imply anything about
whether the code is in C or not --- in theory you could use any language
that's link-compatible with C. What LANGUAGE 'C' really implies is
"dynamically linked, compiled function following fmgr interface
convention #1"", as opposed to (for example) LANGUAGE 'internal' which
implies "statically linked, compiled function following fmgr interface
convention #1". Nothing about language at all.

Quite right. The (unspecified) plan was to allow an environmental setting
that set the default attr for "language 'C'" functions. Then restoring a
dump file would be a matter of setting this variable to use the old
interface at the start of the script.

Interesting idea, but it'll fall down as soon as there's more than one
possible value. You don't think that people are going to update all
their functions to a new interface style at the same time, do you?

Might even be able to implement a 'CREATE MODULE' which loads an
object, enquires about functions in the module, and creates function
entries based on information returned from pg_fmgr_info...

That could work ... something to think about for the future, anyway.

regards, tom lane

On Sat, Nov 11, 2000 at 08:30:48PM -0500, Tom Lane wrote:

Philip Warner <pjw@rhyme.com.au> writes:

...

It does avoid "language 'evenNewerC'" in the furture.

How so? It appears to me it would just move the 'evenNewerFoo'
dirtiness to a different keyword, which would still be essential
for the user to write. Net result: no gain, just more writing.

By allowing optional
attributes to use older fmgr interfaces we preserve compatibility without
cluttering the list of languages. From the developer/user point of view,
the language is 'C' if it's written in C.

No. 'C' is really a misnomer, since it does NOT imply anything about
whether the code is in C or not --- in theory you could use any language
that's link-compatible with C. What LANGUAGE 'C' really implies is
"dynamically linked, compiled function following fmgr interface
convention #1"", as opposed to (for example) LANGUAGE 'internal' which
implies "statically linked, compiled function following fmgr interface
convention #1". Nothing about language at all.

Maybe the construct

CREATE FUNCTION foo(..) RETURNS ...
AS '../foo.so' LANGUAGE 'C';

would be cleaner as

CREATE FUNCTION foo(..) RETURNS ...
FROM '../foo.so', 'pg_foo' [[WITH] VERSION abi_ver];

or with more noise:

FROM [LIBRARY] '../foo.so' AS 'pg_foo' [[WITH] VERSION abi_ver];

because as said, it can be any other language besides C and also
the 'AS file' is weird.

--
marko

No. 'C' is really a misnomer, since it does NOT imply anything about
whether the code is in C or not --- in theory you could use any language
that's link-compatible with C. What LANGUAGE 'C' really implies is
"dynamically linked, compiled function following fmgr interface
convention #1"", as opposed to (for example) LANGUAGE 'internal' which
implies "statically linked, compiled function following fmgr interface
convention #1". Nothing about language at all.

Maybe the construct

CREATE FUNCTION foo(..) RETURNS ...
AS '../foo.so' LANGUAGE 'C';

would be cleaner as

CREATE FUNCTION foo(..) RETURNS ...
FROM '../foo.so', 'pg_foo' [[WITH] VERSION abi_ver];

or with more noise:

FROM [LIBRARY] '../foo.so' AS 'pg_foo' [[WITH] VERSION abi_ver];

because as said, it can be any other language besides C and also
the 'AS file' is weird.

This is interesting. It allows us to control the default behavour of
"C". I would vote to default to 7.0-style when no version is used for
7.1, then default to 7.1 style in 7.2 and later. We don't need
backward C function compatibility for more than one release, I think.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

At 21:15 11/11/00 -0500, Bruce Momjian wrote:

This is interesting. It allows us to control the default behavour of
"C". I would vote to default to 7.0-style when no version is used for
7.1, then default to 7.1 style in 7.2 and later. We don't need
backward C function compatibility for more than one release, I think.

The reason I'd like to see the 'pg_fmgr_info' method in 7.2 is that it will
completely remove any ambiguity from function definitions in the future.

eg. something like:

CREATE FUNCTION foo(..) RETURNS ...
FROM LIBRARY '../foolib.so';

When a backend *loads* the library file, it would call pg_fmgr_info,
possibly passing the desired function name, and would get in return the
calling semantics (including version information). This would not be stored
in the database, since updates to the object file may change the semantics.

This would make all functions upward compatible. It also has what I
consider a big advantage in that the person who writes the function knows
if it is cacheable, strict etc, and it removes the effective duplication of
effort, not to mention possible errors.

As a final bonus, it makes the following possible:

CREATE MODULE foomod FROM LIBRARY '../foolib.so';

when this statement is executed, pg_fmgr_info could be called to return a
list of definied functions...for functions created this way, pg_dump would
only need to dump the module definition.

----------------------------------------------------------------
Philip Warner | __---_____
Albatross Consulting Pty. Ltd. |----/ - \
(A.B.N. 75 008 659 498) | /(@) ______---_
Tel: (+61) 0500 83 82 81 | _________ \
Fax: (+61) 0500 83 82 82 | ___________ |
Http://www.rhyme.com.au | / \|
| --________--
PGP key available upon request, | /
and from pgp5.ai.mit.edu:11371 |/

At 20:30 11/11/00 -0500, Tom Lane wrote:

The current fmgr implementation requires PL handlers to be newC or
newInternal. That could be relaxed, but I don't see any reason to do
so, since an old-style handler would be incapable of handling null
arguments/results or supporting triggers.

OK, then given this, the CREATE LANGUAGE statement really should output a
warning and/or force a change in the language.

It does avoid "language 'evenNewerC'" in the furture.

How so? It appears to me it would just move the 'evenNewerFoo'
dirtiness to a different keyword, which would still be essential
for the user to write. Net result: no gain, just more writing.

Well the gain is that by version 8.3 we won't be requiring all new
functions to use 'veryVeryExtraSpecialC' as their language name. The
clutter has no negative effect on updated and new functions. Equally well,
if we use pg_fmgr_info or similar the problem will go away.

Quite right. The (unspecified) plan was to allow an environmental setting
that set the default attr for "language 'C'" functions. Then restoring a
dump file would be a matter of setting this variable to use the old
interface at the start of the script.

Interesting idea, but it'll fall down as soon as there's more than one
possible value. You don't think that people are going to update all
their functions to a new interface style at the same time, do you?

No, but when restoring from pg_dump, they will (initially, I assume) be
working with old function definitions. Newer versions of pg_dump would dump
the appropriate attribute.

Might even be able to implement a 'CREATE MODULE' which loads an
object, enquires about functions in the module, and creates function
entries based on information returned from pg_fmgr_info...

That could work ... something to think about for the future, anyway.

Yep.

----------------------------------------------------------------
Philip Warner | __---_____
Albatross Consulting Pty. Ltd. |----/ - \
(A.B.N. 75 008 659 498) | /(@) ______---_
Tel: (+61) 0500 83 82 81 | _________ \
Fax: (+61) 0500 83 82 82 | ___________ |
Http://www.rhyme.com.au | / \|
| --________--
PGP key available upon request, | /
and from pgp5.ai.mit.edu:11371 |/

because as said, it can be any other language besides C and also
the 'AS file' is weird.

This is interesting. It allows us to control the default behavour of
"C". I would vote to default to 7.0-style when no version is used for
7.1, then default to 7.1 style in 7.2 and later. We don't need
backward C function compatibility for more than one release, I think.

We need the 7.0 style for compatibility with other DB's. Postgres was
"the" pioneer in this area, but similar functionality is now available in other DB's.

I think it would be worthwhile, to at least peek at what Informix and DB/2
does here.

Andreas

At 20:33 13/11/00 -0500, Mark Hollomon wrote:

On Saturday 11 November 2000 20:30, Tom Lane wrote:

Philip Warner <pjw@rhyme.com.au> writes:

Might even be able to implement a 'CREATE MODULE' which loads an
object, enquires about functions in the module, and creates function
entries based on information returned from pg_fmgr_info...

That could work ... something to think about for the future, anyway.

I apporached the other way. LOAD MODULE would call a well defined
entry point which could then use SPI or some other facility to create
whatever.

The problem with this suggestion (in the current context) is it does not
make the function manager interface version independant.

I saw it as a way to distribute types and even whole applications.

I don't know the details, but why not write an installer of some kind (RPM,
etc)?

----------------------------------------------------------------
Philip Warner | __---_____
Albatross Consulting Pty. Ltd. |----/ - \
(A.B.N. 75 008 659 498) | /(@) ______---_
Tel: (+61) 0500 83 82 81 | _________ \
Fax: (+61) 0500 83 82 82 | ___________ |
Http://www.rhyme.com.au | / \|
| --________--
PGP key available upon request, | /
and from pgp5.ai.mit.edu:11371 |/

On Saturday 11 November 2000 20:30, Tom Lane wrote:

Philip Warner <pjw@rhyme.com.au> writes:

Might even be able to implement a 'CREATE MODULE' which loads an
object, enquires about functions in the module, and creates function
entries based on information returned from pg_fmgr_info...

That could work ... something to think about for the future, anyway.

I proposed this some time ago, even wrote up a semiformal proposal.
I was told at the time it was a bad idea.

The only objection I remember of the top of my head had to do with set-uid.

I apporached the other way. LOAD MODULE would call a well defined
entry point which could then use SPI or some other facility to create
whatever. I saw it as a way to distribute types and even whole applications.

--
Mark Hollomon

Tom Lane wrote:

More to the point, I think we have to assume old-style interface if we
see ... LANGUAGE 'C' with no other decoration, because any other
assumption is guaranteed to break all existing user-defined functions.

Just successfully loading an old-style C function doesn't
guarantee that it works anyway. I pointed out before that the
changes due to TOAST require each function that takes
arguments of varlen types to expect toasted values. Worst
case a dump might reload and anything works fine, but a month
later the first toasted value appears and the old-style C
function corrupts the data without a single warning.

We need to WARN, WARN and explicitly WARN users of selfmade C
functions about this in any possible place!

Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

On Mon, Nov 13, 2000 at 09:58:30AM +0100, Zeugswetter Andreas SB wrote:

because as said, it can be any other language besides C and also
the 'AS file' is weird.

This is interesting. It allows us to control the default behavour of
"C". I would vote to default to 7.0-style when no version is used for
7.1, then default to 7.1 style in 7.2 and later. We don't need
backward C function compatibility for more than one release, I think.

We need the 7.0 style for compatibility with other DB's. Postgres was
"the" pioneer in this area, but similar functionality is now available in other DB's.

Could you explain? PostgreSQL cant be compatible in C level, why
the SQL compatibility? (I mean the LANGUAGE 'C' specifically)

I see already three different C interfaces:

1) 7.0.x
2) 7.1.x
3) > 7.1 where is possible to give a generic funtion/struct where
the backend can guess the actual params, also with some
docstrings, etc... Per-function interface needs not to change.

How do you see it possible to solve with current LANGUAGE 'fooC'
approach?

--
marko

We need the 7.0 style for compatibility with other DB's. Postgres was
"the" pioneer in this area, but similar functionality is now available in other DB's.

Could you explain? PostgreSQL cant be compatible in C level, why
the SQL compatibility? (I mean the LANGUAGE 'C' specifically)

C code compatible with Informix:

int32 intadd (int32 a, int32 b)
{
return a + b;
}

This is the same code that was standard in PostgreSQL 7.0

Andreas

On Wed, Nov 15, 2000 at 02:42:24PM +0100, Zeugswetter Andreas SB wrote:

We need the 7.0 style for compatibility with other DB's. Postgres was
"the" pioneer in this area, but similar functionality is now available in other DB's.

Could you explain? PostgreSQL cant be compatible in C level, why
the SQL compatibility? (I mean the LANGUAGE 'C' specifically)

C code compatible with Informix:

int32 intadd (int32 a, int32 b)
{
return a + b;
}

This is the same code that was standard in PostgreSQL 7.0

Hmm, I have not actually researched if 7.1 supports 7.0 'C' code
or not. Butthe 'newC' is anyway incompatible with 'C'. So:

* CREATE FUNCTION .. AS 'foo.so', .. LANGUAGE 'C';

creates the old� 'C', 7.0 and ifnormix compatible funtion.

And it is documented as deprecated, for-compatibility.

* CREATE FUNCTION .. FROM LIBRARY 'foo.so.2' ..{name in .so}
[WITH VERSION abi_ver]
{the actual syntax needs polishing}

creates by default the newC style fn's
but WITH VERSION 0 (e.g.) you can create the old style
functions too.

Comments?

--
marko

'Marko Kreen' wrote:

On Wed, Nov 15, 2000 at 02:42:24PM +0100, Zeugswetter Andreas SB wrote:

We need the 7.0 style for compatibility with other DB's. Postgres was
"the" pioneer in this area, but similar functionality is now available in other DB's.

Could you explain? PostgreSQL cant be compatible in C level, why
the SQL compatibility? (I mean the LANGUAGE 'C' specifically)

C code compatible with Informix:

int32 intadd (int32 a, int32 b)
{
return a + b;
}

This is the same code that was standard in PostgreSQL 7.0

Hmm, I have not actually researched if 7.1 supports 7.0 'C' code
or not. Butthe 'newC' is anyway incompatible with 'C'. So:

* CREATE FUNCTION .. AS 'foo.so', .. LANGUAGE 'C';

creates the old� 'C', 7.0 and ifnormix compatible funtion.

And it is documented as deprecated, for-compatibility.

* CREATE FUNCTION .. FROM LIBRARY 'foo.so.2' ..{name in .so}
[WITH VERSION abi_ver]
{the actual syntax needs polishing}

creates by default the newC style fn's
but WITH VERSION 0 (e.g.) you can create the old style
functions too.

Comments?

I generally like the idea, but I am working on a text index/search
project that will rely heavily on C interfacing with Postgres.

I'm not sure what "NewC" is, nor do I understand what problem it is
attempting to fix.

--
http://www.mohawksoft.com