Coping with 'C' vs 'newC' function language names

Tom Lane writes:

Philip pointed out awhile back that it does not work to load a 7.0.*
dump into current sources if the dumped database contains any
procedural language definitions. The dumped handler-function
definitions will look like

CREATE FUNCTION "plpgsql_call_handler" ( ) RETURNS opaque AS
'/opt/postgres/lib/plpgsql.sl' LANGUAGE 'C';

which was correct for 7.0.* ... but under 7.1 the handler functions
use the new-style function manager API and therefore need to be
declared as LANGUAGE 'newC'.

I don't really have a better idea, but consider if you installed 7.1 into
/opt/postgres71: then this dump will load the old version of plpgsql.sl.
Assuming that that would work in the first place, LANGUAGE 'C' is correct.

Btw., could we use something other than 'newC'? It's going to get old
really fast (pun intended). Maybe 'Cv2' or something along these lines?

--
Peter Eisentraut peter_e@gmx.net http://yi.org/peter-e/

Peter Eisentraut <peter_e@gmx.net> writes:

I don't really have a better idea, but consider if you installed 7.1 into
/opt/postgres71: then this dump will load the old version of plpgsql.sl.

True, but absolute paths in a dump file are a different (and
long-standing) issue.

Assuming that that would work in the first place, LANGUAGE 'C' is correct.

It wouldn't work, so that's irrelevant. The PL handlers know way more
than the average user-defined function about backend innards, and aren't
usually cross-version compatible. They won't be this time, for sure.

Btw., could we use something other than 'newC'? It's going to get old
really fast (pun intended). Maybe 'Cv2' or something along these lines?

Where were you six months ago? ;-( It's a bit late in the dev cycle to
be running around renaming this kind of stuff...

regards, tom lane

So new-style C functions are language "newC"?

Peter Eisentraut <peter_e@gmx.net> writes:

I don't really have a better idea, but consider if you installed 7.1 into
/opt/postgres71: then this dump will load the old version of plpgsql.sl.

True, but absolute paths in a dump file are a different (and
long-standing) issue.

Assuming that that would work in the first place, LANGUAGE 'C' is correct.

It wouldn't work, so that's irrelevant. The PL handlers know way more
than the average user-defined function about backend innards, and aren't
usually cross-version compatible. They won't be this time, for sure.

Btw., could we use something other than 'newC'? It's going to get old
really fast (pun intended). Maybe 'Cv2' or something along these lines?

Where were you six months ago? ;-( It's a bit late in the dev cycle to
be running around renaming this kind of stuff...

regards, tom lane

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

At 12:07 10/11/00 -0500, Tom Lane wrote:

Something's got to be done about this --- not being able to load
7.0 dump files will not do.

I assume modifying pg_dump for 7.0.x is not an option, since we really need
to support loading databases from historical dump files. If so, then would
there be any value in modifying the CREATE LANGUAGE code to output a
warning for non-newC handlers? Or just updating the function definition
(with a WARNING) when the language is created?

At one time I had toyed with the idea of using 'C' to specify
dynamically-loaded functions that use the new-style fmgr API, and
'oldC' for old-style fmgr API. That would be nicer in the long run,
and it'd fix this particular problem, but it'd break dumped definitions
for user-defined C functions that haven't yet been updated to new-style
API. That doesn't seem like an acceptable tradeoff.

Another option in the long run would be to use language 'C' in all cases
and add an attribute which allows people to specify the function manager to
use. For 7.1 this would default to 'fmgr71' or some such. Then only
old-style functions would break as per your example (I dislike the idea of
the language changing when it is just the interface we modify).

In the even longer term, maybe the function manager should be able to
interrogate functions when they are first loaded, and ask them about the
interface they expect to use.

----------------------------------------------------------------
Philip Warner | __---_____
Albatross Consulting Pty. Ltd. |----/ - \
(A.B.N. 75 008 659 498) | /(@) ______---_
Tel: (+61) 0500 83 82 81 | _________ \
Fax: (+61) 0500 83 82 82 | ___________ |
Http://www.rhyme.com.au | / \|
| --________--
PGP key available upon request, | /
and from pgp5.ai.mit.edu:11371 |/

Philip Warner <pjw@rhyme.com.au> writes:

At 12:07 10/11/00 -0500, Tom Lane wrote:

Something's got to be done about this --- not being able to load
7.0 dump files will not do.

I assume modifying pg_dump for 7.0.x is not an option,

Not really ...

would
there be any value in modifying the CREATE LANGUAGE code to output a
warning for non-newC handlers? Or just updating the function definition
(with a WARNING) when the language is created?

Just outputting a warning is possible, but it still leaves you with a
broken database after you reload your 7.0 dump file :-(. (And there
isn't any supported way to fix it, short of reloading all your function
definitions...) I thought about the fix-handler-at-CREATE-LANGUAGE-time
option, but it doesn't seem noticeably cleaner than hacking CREATE
FUNCTION.

At one time I had toyed with the idea of using 'C' to specify
dynamically-loaded functions that use the new-style fmgr API, and
'oldC' for old-style fmgr API. That would be nicer in the long run,
and it'd fix this particular problem, but it'd break dumped definitions
for user-defined C functions that haven't yet been updated to new-style
API. That doesn't seem like an acceptable tradeoff.

Another option in the long run would be to use language 'C' in all cases
and add an attribute which allows people to specify the function manager to
use. For 7.1 this would default to 'fmgr71' or some such.

No, that just adds complexity without really accomplishing anything.

From the function manager's point of view, the "language" setting *is*

the interface, they're not separately twiddlable concepts.

More to the point, I think we have to assume old-style interface if we
see ... LANGUAGE 'C' with no other decoration, because any other
assumption is guaranteed to break all existing user-defined functions.

regards, tom lane

At 10:55 11/11/00 -0500, Tom Lane wrote:

Just outputting a warning is possible, but it still leaves you with a
broken database after you reload your 7.0 dump file :-(. (And there
isn't any supported way to fix it, short of reloading all your function
definitions...) I thought about the fix-handler-at-CREATE-LANGUAGE-time
option, but it doesn't seem noticeably cleaner than hacking CREATE
FUNCTION.

Is it the PL manager stuff that requires the new interface for all
languages, or is it up to the PL implementor to choose the language for
their handler?

Another option in the long run would be to use language 'C' in all cases
and add an attribute which allows people to specify the function manager to
use. For 7.1 this would default to 'fmgr71' or some such.

No, that just adds complexity without really accomplishing anything.

From the function manager's point of view, the "language" setting *is*

the interface, they're not separately twiddlable concepts.

It does avoid "language 'evenNewerC'" in the furture. By allowing optional
attributes to use older fmgr interfaces we preserve compatibility without
cluttering the list of languages. From the developer/user point of view,
the language is 'C' if it's written in C.

More to the point, I think we have to assume old-style interface if we
see ... LANGUAGE 'C' with no other decoration, because any other
assumption is guaranteed to break all existing user-defined functions.

Quite right. The (unspecified) plan was to allow an environmental setting
that set the default attr for "language 'C'" functions. Then restoring a
dump file would be a matter of setting this variable to use the old
interface at the start of the script.

What about (not for 7.1) adding the ability for the fmgr to query object
modules. eg. look for an entry point 'pg_fmgr_info' or similar, and if
found call it to get attrs of functions. Then the person defining the
function in the database is freed of the need for a large amount of
information about the functions. Might even be able to implement a 'CREATE
MODULE' which loads an object, enquires about functions in the module, and
creates function entries based on information returned from pg_fmgr_info...

----------------------------------------------------------------
Philip Warner | __---_____
Albatross Consulting Pty. Ltd. |----/ - \
(A.B.N. 75 008 659 498) | /(@) ______---_
Tel: (+61) 0500 83 82 81 | _________ \
Fax: (+61) 0500 83 82 82 | ___________ |
Http://www.rhyme.com.au | / \|
| --________--
PGP key available upon request, | /
and from pgp5.ai.mit.edu:11371 |/

Philip Warner <pjw@rhyme.com.au> writes:

Is it the PL manager stuff that requires the new interface for all
languages, or is it up to the PL implementor to choose the language for
their handler?

The current fmgr implementation requires PL handlers to be newC or
newInternal. That could be relaxed, but I don't see any reason to do
so, since an old-style handler would be incapable of handling null
arguments/results or supporting triggers.

No, that just adds complexity without really accomplishing anything.

From the function manager's point of view, the "language" setting *is*

the interface, they're not separately twiddlable concepts.

It does avoid "language 'evenNewerC'" in the furture.

How so? It appears to me it would just move the 'evenNewerFoo'
dirtiness to a different keyword, which would still be essential
for the user to write. Net result: no gain, just more writing.

By allowing optional
attributes to use older fmgr interfaces we preserve compatibility without
cluttering the list of languages. From the developer/user point of view,
the language is 'C' if it's written in C.

No. 'C' is really a misnomer, since it does NOT imply anything about
whether the code is in C or not --- in theory you could use any language
that's link-compatible with C. What LANGUAGE 'C' really implies is
"dynamically linked, compiled function following fmgr interface
convention #1"", as opposed to (for example) LANGUAGE 'internal' which
implies "statically linked, compiled function following fmgr interface
convention #1". Nothing about language at all.

Quite right. The (unspecified) plan was to allow an environmental setting
that set the default attr for "language 'C'" functions. Then restoring a
dump file would be a matter of setting this variable to use the old
interface at the start of the script.

Interesting idea, but it'll fall down as soon as there's more than one
possible value. You don't think that people are going to update all
their functions to a new interface style at the same time, do you?

Might even be able to implement a 'CREATE MODULE' which loads an
object, enquires about functions in the module, and creates function
entries based on information returned from pg_fmgr_info...

That could work ... something to think about for the future, anyway.

regards, tom lane

On Sat, Nov 11, 2000 at 08:30:48PM -0500, Tom Lane wrote:

Philip Warner <pjw@rhyme.com.au> writes:

...

It does avoid "language 'evenNewerC'" in the furture.

How so? It appears to me it would just move the 'evenNewerFoo'
dirtiness to a different keyword, which would still be essential
for the user to write. Net result: no gain, just more writing.

By allowing optional
attributes to use older fmgr interfaces we preserve compatibility without
cluttering the list of languages. From the developer/user point of view,
the language is 'C' if it's written in C.

No. 'C' is really a misnomer, since it does NOT imply anything about
whether the code is in C or not --- in theory you could use any language
that's link-compatible with C. What LANGUAGE 'C' really implies is
"dynamically linked, compiled function following fmgr interface
convention #1"", as opposed to (for example) LANGUAGE 'internal' which
implies "statically linked, compiled function following fmgr interface
convention #1". Nothing about language at all.

Maybe the construct

CREATE FUNCTION foo(..) RETURNS ...
AS '../foo.so' LANGUAGE 'C';

would be cleaner as

CREATE FUNCTION foo(..) RETURNS ...
FROM '../foo.so', 'pg_foo' [[WITH] VERSION abi_ver];

or with more noise:

FROM [LIBRARY] '../foo.so' AS 'pg_foo' [[WITH] VERSION abi_ver];

because as said, it can be any other language besides C and also
the 'AS file' is weird.

--
marko

No. 'C' is really a misnomer, since it does NOT imply anything about
whether the code is in C or not --- in theory you could use any language
that's link-compatible with C. What LANGUAGE 'C' really implies is
"dynamically linked, compiled function following fmgr interface
convention #1"", as opposed to (for example) LANGUAGE 'internal' which
implies "statically linked, compiled function following fmgr interface
convention #1". Nothing about language at all.

Maybe the construct

CREATE FUNCTION foo(..) RETURNS ...
AS '../foo.so' LANGUAGE 'C';

would be cleaner as

CREATE FUNCTION foo(..) RETURNS ...
FROM '../foo.so', 'pg_foo' [[WITH] VERSION abi_ver];

or with more noise:

FROM [LIBRARY] '../foo.so' AS 'pg_foo' [[WITH] VERSION abi_ver];

because as said, it can be any other language besides C and also
the 'AS file' is weird.

This is interesting. It allows us to control the default behavour of
"C". I would vote to default to 7.0-style when no version is used for
7.1, then default to 7.1 style in 7.2 and later. We don't need
backward C function compatibility for more than one release, I think.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

At 21:15 11/11/00 -0500, Bruce Momjian wrote:

This is interesting. It allows us to control the default behavour of
"C". I would vote to default to 7.0-style when no version is used for
7.1, then default to 7.1 style in 7.2 and later. We don't need
backward C function compatibility for more than one release, I think.

The reason I'd like to see the 'pg_fmgr_info' method in 7.2 is that it will
completely remove any ambiguity from function definitions in the future.

eg. something like:

CREATE FUNCTION foo(..) RETURNS ...
FROM LIBRARY '../foolib.so';

When a backend *loads* the library file, it would call pg_fmgr_info,
possibly passing the desired function name, and would get in return the
calling semantics (including version information). This would not be stored
in the database, since updates to the object file may change the semantics.

This would make all functions upward compatible. It also has what I
consider a big advantage in that the person who writes the function knows
if it is cacheable, strict etc, and it removes the effective duplication of
effort, not to mention possible errors.

As a final bonus, it makes the following possible:

CREATE MODULE foomod FROM LIBRARY '../foolib.so';

when this statement is executed, pg_fmgr_info could be called to return a
list of definied functions...for functions created this way, pg_dump would
only need to dump the module definition.

----------------------------------------------------------------
Philip Warner | __---_____
Albatross Consulting Pty. Ltd. |----/ - \
(A.B.N. 75 008 659 498) | /(@) ______---_
Tel: (+61) 0500 83 82 81 | _________ \
Fax: (+61) 0500 83 82 82 | ___________ |
Http://www.rhyme.com.au | / \|
| --________--
PGP key available upon request, | /
and from pgp5.ai.mit.edu:11371 |/

At 20:30 11/11/00 -0500, Tom Lane wrote:

The current fmgr implementation requires PL handlers to be newC or
newInternal. That could be relaxed, but I don't see any reason to do
so, since an old-style handler would be incapable of handling null
arguments/results or supporting triggers.

OK, then given this, the CREATE LANGUAGE statement really should output a
warning and/or force a change in the language.

It does avoid "language 'evenNewerC'" in the furture.

How so? It appears to me it would just move the 'evenNewerFoo'
dirtiness to a different keyword, which would still be essential
for the user to write. Net result: no gain, just more writing.

Well the gain is that by version 8.3 we won't be requiring all new
functions to use 'veryVeryExtraSpecialC' as their language name. The
clutter has no negative effect on updated and new functions. Equally well,
if we use pg_fmgr_info or similar the problem will go away.

Quite right. The (unspecified) plan was to allow an environmental setting
that set the default attr for "language 'C'" functions. Then restoring a
dump file would be a matter of setting this variable to use the old
interface at the start of the script.

Interesting idea, but it'll fall down as soon as there's more than one
possible value. You don't think that people are going to update all
their functions to a new interface style at the same time, do you?

No, but when restoring from pg_dump, they will (initially, I assume) be
working with old function definitions. Newer versions of pg_dump would dump
the appropriate attribute.

Might even be able to implement a 'CREATE MODULE' which loads an
object, enquires about functions in the module, and creates function
entries based on information returned from pg_fmgr_info...

That could work ... something to think about for the future, anyway.

Yep.

----------------------------------------------------------------
Philip Warner | __---_____
Albatross Consulting Pty. Ltd. |----/ - \
(A.B.N. 75 008 659 498) | /(@) ______---_
Tel: (+61) 0500 83 82 81 | _________ \
Fax: (+61) 0500 83 82 82 | ___________ |
Http://www.rhyme.com.au | / \|
| --________--
PGP key available upon request, | /
and from pgp5.ai.mit.edu:11371 |/

because as said, it can be any other language besides C and also
the 'AS file' is weird.

This is interesting. It allows us to control the default behavour of
"C". I would vote to default to 7.0-style when no version is used for
7.1, then default to 7.1 style in 7.2 and later. We don't need
backward C function compatibility for more than one release, I think.

We need the 7.0 style for compatibility with other DB's. Postgres was
"the" pioneer in this area, but similar functionality is now available in other DB's.

I think it would be worthwhile, to at least peek at what Informix and DB/2
does here.

Andreas

At 20:33 13/11/00 -0500, Mark Hollomon wrote:

On Saturday 11 November 2000 20:30, Tom Lane wrote:

Philip Warner <pjw@rhyme.com.au> writes:

Might even be able to implement a 'CREATE MODULE' which loads an
object, enquires about functions in the module, and creates function
entries based on information returned from pg_fmgr_info...

That could work ... something to think about for the future, anyway.

I apporached the other way. LOAD MODULE would call a well defined
entry point which could then use SPI or some other facility to create
whatever.

The problem with this suggestion (in the current context) is it does not
make the function manager interface version independant.

I saw it as a way to distribute types and even whole applications.

I don't know the details, but why not write an installer of some kind (RPM,
etc)?

----------------------------------------------------------------
Philip Warner | __---_____
Albatross Consulting Pty. Ltd. |----/ - \
(A.B.N. 75 008 659 498) | /(@) ______---_
Tel: (+61) 0500 83 82 81 | _________ \
Fax: (+61) 0500 83 82 82 | ___________ |
Http://www.rhyme.com.au | / \|
| --________--
PGP key available upon request, | /
and from pgp5.ai.mit.edu:11371 |/

On Saturday 11 November 2000 20:30, Tom Lane wrote:

Philip Warner <pjw@rhyme.com.au> writes:

Might even be able to implement a 'CREATE MODULE' which loads an
object, enquires about functions in the module, and creates function
entries based on information returned from pg_fmgr_info...

That could work ... something to think about for the future, anyway.

I proposed this some time ago, even wrote up a semiformal proposal.
I was told at the time it was a bad idea.

The only objection I remember of the top of my head had to do with set-uid.

I apporached the other way. LOAD MODULE would call a well defined
entry point which could then use SPI or some other facility to create
whatever. I saw it as a way to distribute types and even whole applications.

--
Mark Hollomon

Tom Lane wrote:

More to the point, I think we have to assume old-style interface if we
see ... LANGUAGE 'C' with no other decoration, because any other
assumption is guaranteed to break all existing user-defined functions.

Just successfully loading an old-style C function doesn't
guarantee that it works anyway. I pointed out before that the
changes due to TOAST require each function that takes
arguments of varlen types to expect toasted values. Worst
case a dump might reload and anything works fine, but a month
later the first toasted value appears and the old-style C
function corrupts the data without a single warning.

We need to WARN, WARN and explicitly WARN users of selfmade C
functions about this in any possible place!

Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

On Mon, Nov 13, 2000 at 09:58:30AM +0100, Zeugswetter Andreas SB wrote:

because as said, it can be any other language besides C and also
the 'AS file' is weird.

This is interesting. It allows us to control the default behavour of
"C". I would vote to default to 7.0-style when no version is used for
7.1, then default to 7.1 style in 7.2 and later. We don't need
backward C function compatibility for more than one release, I think.

We need the 7.0 style for compatibility with other DB's. Postgres was
"the" pioneer in this area, but similar functionality is now available in other DB's.

Could you explain? PostgreSQL cant be compatible in C level, why
the SQL compatibility? (I mean the LANGUAGE 'C' specifically)

I see already three different C interfaces:

1) 7.0.x
2) 7.1.x
3) > 7.1 where is possible to give a generic funtion/struct where
the backend can guess the actual params, also with some
docstrings, etc... Per-function interface needs not to change.

How do you see it possible to solve with current LANGUAGE 'fooC'
approach?

--
marko

We need the 7.0 style for compatibility with other DB's. Postgres was
"the" pioneer in this area, but similar functionality is now available in other DB's.

Could you explain? PostgreSQL cant be compatible in C level, why
the SQL compatibility? (I mean the LANGUAGE 'C' specifically)

C code compatible with Informix:

int32 intadd (int32 a, int32 b)
{
return a + b;
}

This is the same code that was standard in PostgreSQL 7.0

Andreas

On Wed, Nov 15, 2000 at 02:42:24PM +0100, Zeugswetter Andreas SB wrote:

We need the 7.0 style for compatibility with other DB's. Postgres was
"the" pioneer in this area, but similar functionality is now available in other DB's.

Could you explain? PostgreSQL cant be compatible in C level, why
the SQL compatibility? (I mean the LANGUAGE 'C' specifically)

C code compatible with Informix:

int32 intadd (int32 a, int32 b)
{
return a + b;
}

This is the same code that was standard in PostgreSQL 7.0

Hmm, I have not actually researched if 7.1 supports 7.0 'C' code
or not. Butthe 'newC' is anyway incompatible with 'C'. So:

* CREATE FUNCTION .. AS 'foo.so', .. LANGUAGE 'C';

creates the old� 'C', 7.0 and ifnormix compatible funtion.

And it is documented as deprecated, for-compatibility.

* CREATE FUNCTION .. FROM LIBRARY 'foo.so.2' ..{name in .so}
[WITH VERSION abi_ver]
{the actual syntax needs polishing}

creates by default the newC style fn's
but WITH VERSION 0 (e.g.) you can create the old style
functions too.

Comments?

--
marko

'Marko Kreen' wrote:

On Wed, Nov 15, 2000 at 02:42:24PM +0100, Zeugswetter Andreas SB wrote:

We need the 7.0 style for compatibility with other DB's. Postgres was
"the" pioneer in this area, but similar functionality is now available in other DB's.

Could you explain? PostgreSQL cant be compatible in C level, why
the SQL compatibility? (I mean the LANGUAGE 'C' specifically)

C code compatible with Informix:

int32 intadd (int32 a, int32 b)
{
return a + b;
}

This is the same code that was standard in PostgreSQL 7.0

Hmm, I have not actually researched if 7.1 supports 7.0 'C' code
or not. Butthe 'newC' is anyway incompatible with 'C'. So:

* CREATE FUNCTION .. AS 'foo.so', .. LANGUAGE 'C';

creates the old� 'C', 7.0 and ifnormix compatible funtion.

And it is documented as deprecated, for-compatibility.

* CREATE FUNCTION .. FROM LIBRARY 'foo.so.2' ..{name in .so}
[WITH VERSION abi_ver]
{the actual syntax needs polishing}

creates by default the newC style fn's
but WITH VERSION 0 (e.g.) you can create the old style
functions too.

Comments?

I generally like the idea, but I am working on a text index/search
project that will rely heavily on C interfacing with Postgres.

I'm not sure what "NewC" is, nor do I understand what problem it is
attempting to fix.

--
http://www.mohawksoft.com

mlw <markw@mohawksoft.com> writes:

I'm not sure what "NewC" is, nor do I understand what problem it is
attempting to fix.

I haven't gotten around to updating the main documentation yet (my bad)
but the design document about the fmgr rewrite is in current sources
as src/backend/utils/fmgr/README --- you can read it on the web at
http://www.postgresql.org/cgi/cvsweb.cgi/pgsql/src/backend/utils/fmgr/README
if you don't have a local copy.

The key issues are summarized in that document as:

: We know that the existing mechanism for calling Postgres functions needs
: to be redesigned. It has portability problems because it makes
: assumptions about parameter passing that violate ANSI C; it fails to
: handle NULL arguments and results cleanly; and "function handlers" that
: support a class of functions (such as fmgr_pl) can only be done via a
: really ugly, non-reentrant kluge. (Global variable set during every
: function call, forsooth.) Here is a proposal for fixing these problems.

To answer another misconception that I saw in this thread:

: The old language names "internal" and "C" will continue to refer to
: functions with the old calling convention. We should deprecate
: old-style functions because of their portability problems, but the
: support for them will only be one small function handler routine,
: so we can leave them in place for as long as necessary.

regards, tom lane

To answer another misconception that I saw in this thread:

: The old language names "internal" and "C" will continue to refer to
: functions with the old calling convention. We should deprecate
: old-style functions because of their portability problems, but the
: support for them will only be one small function handler routine,
: so we can leave them in place for as long as necessary.

My question is can we drop newC and use just plain C in 7.2 or 7.3?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

At 23:42 15/11/00 -0500, Bruce Momjian wrote:

To answer another misconception that I saw in this thread:

: The old language names "internal" and "C" will continue to refer to
: functions with the old calling convention. We should deprecate
: old-style functions because of their portability problems, but the
: support for them will only be one small function handler routine,
: so we can leave them in place for as long as necessary.

My question is can we drop newC and use just plain C in 7.2 or 7.3?

I plan to work on a a proposal for a (hopefully) version independant
function manager interface; the idea behind the proposal is to allow PGSQL
to query the modules for information about the functions, but calling a
single known entry point. The information stored in the database relation
would be substantially reduced, and the backends would load the module then
enquire about the functions, storing the results in memory. The kind of SQL
required would be:

Create Function foo(int4, int4) from library 'path-to-lib';

and possibly,

Create Module foo_mod from library 'path-to-lib';

The idea being to only store whe function signature and enough details to
get to the info-function. If 'Create Module' were allowed, then it would
automatically create appropriate function definitions when the statement
was executed.

The info-function would return data in a struct passed from the backend,
and part of the struct would include version information. The backend would
then be responsible for handling new & old protocols.

----------------------------------------------------------------
Philip Warner | __---_____
Albatross Consulting Pty. Ltd. |----/ - \
(A.B.N. 75 008 659 498) | /(@) ______---_
Tel: (+61) 0500 83 82 81 | _________ \
Fax: (+61) 0500 83 82 82 | ___________ |
Http://www.rhyme.com.au | / \|
| --________--
PGP key available upon request, | /
and from pgp5.ai.mit.edu:11371 |/

To answer another misconception that I saw in this thread:

: The old language names "internal" and "C" will continue to refer to
: functions with the old calling convention. We should deprecate
: old-style functions because of their portability problems, but the
: support for them will only be one small function handler routine,
: so we can leave them in place for as long as necessary.

My question is can we drop newC and use just plain C in 7.2 or 7.3?

Has anybody had time to look at how this is done in DB/2, Oracle ? Philip ?

In Informix there is an additional keyword "parameter style".

Thus you have:
create function foo (a int, b int) return{s|ing} int
external name '/path/libmod.so(symbol)' language C
[parameter style informix] [not variant];

We could have "parameter style postgresql" and map that to
some arbitrary string that would not be something the user sees.

As you see this is really very close to what we have or want
and I am really unhappy that there has been no effort at all
to look at what others do. Not that we want to copy some stupidity,
but if it is sane .... These are also the companies that
have the most influence on future ANSI specs, and thus if we keep
close we will have a better position to stay conformant.

Actually my proposal would be to not advertise "newC" in 7.1 and do
some more research in that area until we have a solid and maybe compatible
interface that also makes the missing features possible
(multiple columns and rows for return, enter the function more than once
to retrieve only part of the result if it consists of many rows).

Andreas

On Thu, Nov 16, 2000 at 04:16:26PM +1100, Philip Warner wrote:

and possibly,

Create Module foo_mod from library 'path-to-lib';

The idea being to only store whe function signature and enough details to
get to the info-function. If 'Create Module' were allowed, then it would
automatically create appropriate function definitions when the statement
was executed.

Phil - be careful with the nomenclature. We've got another naming collision,
here. SQL9[29] talk about modules, which may or may not be related to what
your suggesting here.

Ross
--
Open source code is like a natural resource, it's the result of providing
food and sunshine to programmers, and then staying out of their way.
[...] [It] is not going away because it has utility for both the developers
and users independent of economic motivations. Jim Flynn, Sunnyvale, Calif.

On Thu, Nov 16, 2000 at 09:32:43AM -0600, Ross J. Reedstrom wrote:

On Thu, Nov 16, 2000 at 04:16:26PM +1100, Philip Warner wrote:

Create Module foo_mod from library 'path-to-lib';

Phil - be careful with the nomenclature. We've got another naming collision,
here. SQL9[29] talk about modules, which may or may not be related to what
your suggesting here.

Do you know any url's where the SQL* standards could be looked
up?

Mark Hollomon's idea was to use 'package' not 'module', but
ofcourse it would be nice to be SQL* conforming.

--
marko

At 05:51 PM 11/16/00 +0200, 'Marko Kreen' wrote:

On Thu, Nov 16, 2000 at 09:32:43AM -0600, Ross J. Reedstrom wrote:

On Thu, Nov 16, 2000 at 04:16:26PM +1100, Philip Warner wrote:

Create Module foo_mod from library 'path-to-lib';

Phil - be careful with the nomenclature. We've got another naming

collision,

here. SQL9[29] talk about modules, which may or may not be related to what
your suggesting here.

Do you know any url's where the SQL* standards could be looked
up?

I have a copy of the SQL92 draft (the one that's circulated among this
group in
the past) at dsl-dhogaza.pacifier.net. Just use anonymous ftp, it's in the
pub
directory with an obvious name (sql1992.txt???)

- Don Baccus, Portland OR <dhogaza@pacifier.com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.

On Thu, Nov 16, 2000 at 05:51:07PM +0200, 'Marko Kreen' wrote:

On Thu, Nov 16, 2000 at 09:32:43AM -0600, Ross J. Reedstrom wrote:

On Thu, Nov 16, 2000 at 04:16:26PM +1100, Philip Warner wrote:

Create Module foo_mod from library 'path-to-lib';

Phil - be careful with the nomenclature. We've got another naming collision,
here. SQL9[29] talk about modules, which may or may not be related to what
your suggesting here.

Do you know any url's where the SQL* standards could be looked
up?

Mark Hollomon's idea was to use 'package' not 'module', but
ofcourse it would be nice to be SQL* conforming.

Well, the 1999 standards seem to live at:

ftp://jerry.ece.umassd.edu/isowg3/x3h2/Standards/

Which is the working repository for the ANSI database committee (x3h2)

Ross
--
Open source code is like a natural resource, it's the result of providing
food and sunshine to programmers, and then staying out of their way.
[...] [It] is not going away because it has utility for both the developers
and users independent of economic motivations. Jim Flynn, Sunnyvale, Calif.

Actually my proposal would be to not advertise "newC" in 7.1 and do
some more research in that area until we have a solid and maybe compatible
interface that also makes the missing features possible
(multiple columns and rows for return, enter the function more than once
to retrieve only part of the result if it consists of many rows).

My problem with newC is that I think it is going to cause confusing by
people who create new-style functions and call the language "C". I
recommend making our current code "C" style, and calling pre-7.1
functions "C70", that way, we can still enable old functions to work,
they just have to use "C70" to make them work, and all our new code is
the clean "C" type.

Comments?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

At 10:39 16/11/00 +0100, Zeugswetter Andreas SB wrote:

Has anybody had time to look at how this is done in DB/2, Oracle ? Philip ?

Don't know about Oracle or DB2, but Dec/RDB has:

Create Function <name1> [Stored Name Is <name2>] (...) Returns <type>;
[ External Name <name3> ] [ Location <libfile> ]
[Language {ADA|C|Fortran|Pascal|General]
General Parameter Style [Not] Variant

where <name1> is the function name, the 'Stored Name' relates to
cross-schema functions, the 'External Name' is the name of the entry point,
'location' is the file location and the 'Variant' part corresponds to our
'iscacheable' attribute.

Functions themselves require no special coding. This is pretty much what I
think 'Language C' does for us now - just calling a shared library. There
is definitely a case to be made for coninuing to support standard calls to
general libraries unmodified. In Tom's current proposal, I think this will
be the case if there is no 'function-info' function available.

----------------------------------------------------------------
Philip Warner | __---_____
Albatross Consulting Pty. Ltd. |----/ - \
(A.B.N. 75 008 659 498) | /(@) ______---_
Tel: (+61) 0500 83 82 81 | _________ \
Fax: (+61) 0500 83 82 82 | ___________ |
Http://www.rhyme.com.au | / \|
| --________--
PGP key available upon request, | /
and from pgp5.ai.mit.edu:11371 |/

More to the point, I think we have to assume old-style interface if we
see ... LANGUAGE 'C' with no other decoration, because any other
assumption is guaranteed to break all existing user-defined functions.

Just successfully loading an old-style C function doesn't
guarantee that it works anyway. I pointed out before that the
changes due to TOAST require each function that takes
arguments of varlen types to expect toasted values. Worst
case a dump might reload and anything works fine, but a month
later the first toasted value appears and the old-style C
function corrupts the data without a single warning.

We need to WARN, WARN and explicitly WARN users of selfmade C
functions about this in any possible place!

Imho the better solution would be, to always detoast values before we pass
them to old-style C-functions. And toast the return value when the function returns.

Andreas

Jan Wieck writes:

Just successfully loading an old-style C function doesn't
guarantee that it works anyway. I pointed out before that the
changes due to TOAST require each function that takes
arguments of varlen types to expect toasted values.

Couldn't the function handler detoast the values before handing them to
the function? That would be slower, but it would allow people to continue
to use the "simpler" interface.

--
Peter Eisentraut peter_e@gmx.net http://yi.org/peter-e/

Peter Eisentraut <peter_e@gmx.net> writes:

Couldn't the function handler detoast the values before handing them to
the function? That would be slower, but it would allow people to continue
to use the "simpler" interface.

This would be a moderately expensive thing to do --- not impossible,
but relatively expensive. The old-style function handler would need
to look up all the function-argument datatypes during fmgr_info()
so that it could save a bool array telling it which argument positions
need detoasting. (You can't just detoast every Datum --- you have to
at least verify that it's of a varlena type first.)

On a per-invocation basis, however, it probably wouldn't be very costly.

I didn't want to do this during development, but now that there are no
more old-style internal functions left, I suppose you could make a good
argument that this is worth doing for old-style dynamically loaded
functions. Will put it on the to-do list.

Are people satisfied with the notion of requiring an info function
to go with each dynamically loaded new-style function? If so, I'll
start working on that too.

regards, tom lane

Just successfully loading an old-style C function doesn't
guarantee that it works anyway. I pointed out before that the
changes due to TOAST require each function that takes
arguments of varlen types to expect toasted values.

Couldn't the function handler detoast the values before handing them to
the function? That would be slower, but it would allow people to continue
to use the "simpler" interface.

Are there really that many things you can do with a toasted value ?
Off hand I can only think of equality functions. All others should need
a detoasted value anyways.

Andreas

I didn't want to do this during development, but now that there are no
more old-style internal functions left, I suppose you could make a good
argument that this is worth doing for old-style dynamically loaded
functions. Will put it on the to-do list.

Are people satisfied with the notion of requiring an info function
to go with each dynamically loaded new-style function? If so, I'll
start working on that too.

I think we need to balance portability with inconvenence for new users.

I think mixing new/old function types in the same object file is pretty
rare, and the confusion for programmers of having to label every
function seems much more error-prone.

I would support a single symbol to mark the entire object file. In
fact, I would require old-style functions to add a symbol, and have
new-style functions left alone.

There are not that many functions out there, are there? People are
having to recompile their C files anyway for the upgrade, don't they?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tom Lane writes:

Are people satisfied with the notion of requiring an info function
to go with each dynamically loaded new-style function? If so, I'll
start working on that too.

Sounds good to me.

--
Peter Eisentraut peter_e@gmx.net http://yi.org/peter-e/

Bruce Momjian writes:

I think mixing new/old function types in the same object file is pretty
rare,

What about several object files linked into a shared library? Won't work.

and the confusion for programmers of having to label every function
seems much more error-prone.

I think of it as having to declare your function.

--
Peter Eisentraut peter_e@gmx.net http://yi.org/peter-e/

On Fri, Nov 17, 2000 at 09:43:28PM +1100, Philip Warner wrote:

At 10:39 16/11/00 +0100, Zeugswetter Andreas SB wrote:

Has anybody had time to look at how this is done in DB/2, Oracle ? Philip ?

Don't know about Oracle or DB2, but Dec/RDB has:

Well, I don't know 'nuthing about Oracle, but I _did_ sign up for the
OTN web site some time ago, specifically to get at Oracle docs. ;-)

After clicking around there for a bit I came up with this, which is my
interpretation of a 'flowchart' style language diagram:

CREATE [OR REPLACE] FUNCTION [<schema>.]<name>
[( <argument1> [IN|OUT|IN OUT] [NOCOPY] <datatype1> [, <...>] )]
RETURN <datatype>
[{AUTHID {CURRENT_USER|DEFINER} | DETERMINISTIC | PARALLEL_ENABLE} <...>]
{IS|AS}
{ <pl/sql_function_body>|
LANGUAGE {JAVA NAME '<java_method_name>'|
C [NAME <c_func_name>] LIBRARY <lib_name>
[WITH CONTEXT] [PARAMETERS (...)]
}
}

The actual filesystem path to the DLL or .so is defined with a CREATE
LIBRARY command.

The WITH CONTEXT bit is a pointer to an opaque structure that the
underlying function is supposed to pass on to service routines it might
call, for their use.

It seems the parameters can be defined either in place with the name,
or with the PARAMETERS keyword.

Philip's comments about this directly calling into a C shared library
seem to apply, as well.

Ross
--
Open source code is like a natural resource, it's the result of providing
food and sunshine to programmers, and then staying out of their way.
[...] [It] is not going away because it has utility for both the developers
and users independent of economic motivations. Jim Flynn, Sunnyvale, Calif.

Bruce Momjian <pgman@candle.pha.pa.us> writes:

I would support a single symbol to mark the entire object file. In
fact, I would require old-style functions to add a symbol, and have
new-style functions left alone.

That won't fly.

There are not that many functions out there, are there? People are
having to recompile their C files anyway for the upgrade, don't they?

There's a big difference between having to recompile and having to
change your source code.

For that matter, I think past version updates haven't even forced
recompiles of user-defined functions, at least not ones that didn't poke
into system innards. We can't get away with requiring a source code
change --- people will scream about it.

The nice thing about the info-marker idea is that we'll be able to
extend it later, so that more info about the function is stored right
where the function text is, and you don't have such a problem with
keeping an SQL script file in sync with the function's real definition.

regards, tom lane

Bruce Momjian wrote:

I didn't want to do this during development, but now that there are no
more old-style internal functions left, I suppose you could make a good
argument that this is worth doing for old-style dynamically loaded
functions. Will put it on the to-do list.

Are people satisfied with the notion of requiring an info function
to go with each dynamically loaded new-style function? If so, I'll
start working on that too.

I think we need to balance portability with inconvenence for new users.

I think mixing new/old function types in the same object file is pretty
rare, and the confusion for programmers of having to label every
function seems much more error-prone.

I would support a single symbol to mark the entire object file. In
fact, I would require old-style functions to add a symbol, and have
new-style functions left alone.

There are not that many functions out there, are there? People are
having to recompile their C files anyway for the upgrade, don't they?

Can't we insert that magic variable automatically using some
#includ/#define tricks ?

So that people need just to recompile, but the result has the variable
nonetheless ?

-----------
Hannu

For that matter, I think past version updates haven't even forced
recompiles of user-defined functions, at least not ones that didn't poke
into system innards. We can't get away with requiring a source code
change --- people will scream about it.

imho that is *not* sufficient to keep from Doing The Right Thing. And
DTRT usually means designing correctly for the future. I would support a
design that is cleanest for the next release, and put backward
compatibility a (no so) distant second.

Agreed.

The nice thing about the info-marker idea is that we'll be able to
extend it later, so that more info about the function is stored right
where the function text is, and you don't have such a problem with
keeping an SQL script file in sync with the function's real definition.

I've been wanting to think about the "package" or "module" idea (which
others have brought up recently). If this kind of hook gets us more
options to support that (e.g. running a routine when the module is first
loaded to install new options into "SET key=val") then even better. Lack
of hooks of this nature lead us to push datatypes down into the core
when if we had full-up module support we could make more things
loadable.

One idea we had was to define a SET variable or psql command-line switch
so that LANGUAGE "C" would mean either old or new style. That way,
pg_dump loads from old versions could set the switch before load without
changing their code, and we could call all functions C.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Bruce Momjian wrote:

I didn't want to do this during development, but now that there are no
more old-style internal functions left, I suppose you could make a good
argument that this is worth doing for old-style dynamically loaded
functions. Will put it on the to-do list.

Are people satisfied with the notion of requiring an info function
to go with each dynamically loaded new-style function? If so, I'll
start working on that too.

I think we need to balance portability with inconvenence for new users.

I think mixing new/old function types in the same object file is pretty
rare, and the confusion for programmers of having to label every
function seems much more error-prone.

I would support a single symbol to mark the entire object file. In
fact, I would require old-style functions to add a symbol, and have
new-style functions left alone.

There are not that many functions out there, are there? People are
having to recompile their C files anyway for the upgrade, don't they?

Can't we insert that magic variable automatically using some
#includ/#define tricks ?

So that people need just to recompile, but the result has the variable
nonetheless ?

We thought of that. The problem is some new-style functions do not need
to call any macros.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

For that matter, I think past version updates haven't even forced
recompiles of user-defined functions, at least not ones that didn't poke
into system innards. We can't get away with requiring a source code
change --- people will scream about it.

imho that is *not* sufficient to keep from Doing The Right Thing. And
DTRT usually means designing correctly for the future. I would support a
design that is cleanest for the next release, and put backward
compatibility a (no so) distant second.

The nice thing about the info-marker idea is that we'll be able to
extend it later, so that more info about the function is stored right
where the function text is, and you don't have such a problem with
keeping an SQL script file in sync with the function's real definition.

I've been wanting to think about the "package" or "module" idea (which
others have brought up recently). If this kind of hook gets us more
options to support that (e.g. running a routine when the module is first
loaded to install new options into "SET key=val") then even better. Lack
of hooks of this nature lead us to push datatypes down into the core
when if we had full-up module support we could make more things
loadable.

- Thomas

Jan Wieck <janwieck@Yahoo.com> writes:

Just successfully loading an old-style C function doesn't
guarantee that it works anyway. I pointed out before that the
changes due to TOAST require each function that takes
arguments of varlen types to expect toasted values. Worst
case a dump might reload and anything works fine, but a month
later the first toasted value appears and the old-style C
function corrupts the data without a single warning.

We need to WARN, WARN and explicitly WARN users of selfmade C
functions about this in any possible place!

As of now, not.

regards, tom lane