SPI & file locations

Ron Peterson <rpeterson@yellowbank.com> writes:

After installation, shouldn't everything you need be in
/usr/local/pgsql?

Yeah, it should really. We've had this discussion before. The real
problem is that no one wants to install the entire pgsql include tree,
but it's hard to draw the line at what an SPI extension might need or
not need. It doesn't help that we've been sloppy about what lives in
which include file, too :-(. Sooner or later someone should go through
the whole include tree and try to rearrange things so that there's a
fairly compact set of files that need to be exported.

In the meantime, pointing at the source tree is a good way to build SPI
extensions...

regards, tom lane

Ron Peterson wrote:

I'm afraid this might sound rather dumb. I'm hoping I can just get a
little clarification about file locations.

I've just started playing w/ SPI. As a first stab, I thought I'd
compile a couple of the test applications in /contrib.

I pointed gcc to include files from /usr/local/pgsql - i.e. 'gcc ...
-I/usr/local/pgsql/include ...'. This of course didn't work.
/usr/local/pgsql/include/executor/spi.h attempts to include files which
don't exist in the install directory. They only exist in
/usr/local/src/postgresql-7.0/src/include (or wherever you put the
source).

After installation, shouldn't everything you need be in
/usr/local/pgsql?

I too have run into this dependency problem. The number of
headers required to compile an SPI code module is around 80, if I
recall correctly. Lamar Owen was good enough to include those
headers as apart of the RPM distribution and they would go into
/usr/include/pgsql. I believe he got the dependency list from
Oliver Elphick who manages the Debian package, so that should be
correct as well. If you're not using RedHat or Debian
distrubitions, I think you're stuck with keeping the backend
source tree lying around. :-(

Mike Mascari

So I got the moddatetime trigger example in /contrib working. A couple
of notes:

moddatetime.c makes reference to DATETIME. This needs to be changed to
TIMESTAMP. I've done this, if you want the source.

I need to make .so files in two steps: first make a regular object file,
then compile that w/ -fpic and -shared and output an .so file. If I try
to do this in one step, it doesn't work. This may very well be the way
the compiler is _supposed_ to work, I dunno. RH6.1, kernel 2.2.13, gcc
version egcs-2.91.66_19990314/Linux (egcs-1.1.2 release).

-Ron-

On Fri, 26 May 2000, Mike Mascari wrote:

Ron Peterson wrote:

After installation, shouldn't everything you need be in
/usr/local/pgsql?

Too many assumptions are in the source that the source will always be there.
Not necessarily true!

I too have run into this dependency problem. The number of
headers required to compile an SPI code module is around 80, if I
recall correctly. Lamar Owen was good enough to include those
headers as apart of the RPM distribution and they would go into
/usr/include/pgsql. I believe he got the dependency list from
Oliver Elphick who manages the Debian package, so that should be

For PostgreSQL 7.0-2 and above, the SPI header list is dynamically generated
during package build and manually copied into place using the following
one-liner:

/lib/cpp -M -I. -I../backend executor/spi.h |xargs -n 1|grep \\W|grep -v ^/|grep -v spi.h | sort |cpio -pdu $RPM_BUILD_ROOT/usr/include/pgsql

Replace $RPM_BUILD_ROOT/usr/include/pgsql with the directory of your choice,
and run this one-liner when the cwd is in src/include in the PostgreSQL source
tree. The sort is optional, of course.

This could easily enough be included in the make install, couldn't it?
(Tom? Anyone?) I realize that GNU grepisms (or is \W common?) are used above,
but, after all, I _know_ my one-liner in the RPM spec file installation section
is going to be running on RedHat with a complete development environment, so it
wasn't designed to be portable. I also realize the regexps could be tuned to
only need a single grep invocation, but, it works and works nicely as-is in the
RPM building environment.

Oh, BTW, there's the same number of header includes as there were with 6.5.3,
but there are differences in the list.... SPI development CAN be done without
26+ MB of source code taking up space -- Mike is _doing_ it, IIRC. In fact,
Mike is the one who prodded me to get it working in the RPMs.

--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

On Fri, 26 May 2000, Tom Lane wrote:

Ron Peterson <rpeterson@yellowbank.com> writes:

After installation, shouldn't everything you need be in
/usr/local/pgsql?

Yeah, it should really. We've had this discussion before. The real
problem is that no one wants to install the entire pgsql include tree,
but it's hard to draw the line at what an SPI extension might need or

If it only needs what spi.h depends upon, then the line is drawn by the
one-liner in my other message.

--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

On Fri, 26 May 2000, Tom Lane wrote:

Lamar Owen <lamar.owen@wgcr.org> writes:

/lib/cpp -M -I. -I../backend executor/spi.h |xargs -n 1|grep \\W|grep -v ^/|grep -v spi.h | sort |cpio -pdu $RPM_BUILD_ROOT/usr/include/pgsql

This could easily enough be included in the make install, couldn't it?
(Tom? Anyone?) I realize that GNU grepisms (or is \W common?) are
used above,

That's just the tip of the iceberg of the portability problems in the
above. If you want to propose that we actually do something like that
during 'make install', you're gonna have to work a lot harder.

:-) I expect no less (than to work harder, of course....). Why not do
something like that, other than the 'out-of-sight, out-of-mind' problem? Or,
to put it far more bluntly, the SPI installation is very broken if the SPI
developer has to go around manually installing header files that make install
should have automatically taken care of.

(However, as a non-portable trick for getting a list of files that need
to be included in a hand-generated makefile, it's not bad.)

I take that as high praise :-). No, really it IS a quick hack -- I got tired
of generating the list by hand, so I automated it. And, when the list shrinks
down to nearly nothing (10-15 includes, maybe?), I won't have to change a thing.

The more serious problem is "what else might an SPI module need besides
spi.h".

What else indeed. Isn't spi.h the exported SPI itself? We seem to have a very
poorly defined line between exported and private interfaces -- back to my point
about the assumption that the whole source tree is available for any little
program's whim (such as the regression test suite, which was _interesting_ to
get working without a bunch of the source tree around.) Mike Mascari may be
able to answer more about what SPI programs require, as he is doing SPI
development.

Also, it disturbs me a lot that spi.h pulls in 80+ include
files in the first place --- there's got to be stuff in there that
needn't/shouldn't be exported.

Such as lztext.h?

rather than fall back on automation that will let the list bloat even
more without anyone much noticing...

Part of that would be to think more like a packager or user, rather than a
developer. That was the first thing I learned when tackling the RPM's, was,
that I was a packager -- while that does involve knowledge of the guts of the
package, it also requires a mindset very different from development -- more
user-oriented, I suppose.

A packager for RedHat will start thinking about what can be partitioned out,
saving the user's HD space. A packager will start thinking about how the
package _can_ be partitioned, and logical partitioning (for instance, the
absolutely worst possible packaging for PostgreSQL in RPM form would be a
single RPM with everything in it -- thus, requiring a database server to have a
full-bore X11 installation for pgaccess (which requires tk and tcl, as well),
when that database server may have absolutely no need for pgaccess.)). I am
now even considering splitting out pltcl into a separate package -- as pltcl
implicitly requires the server package, it makes the whole tcl package require
the server package -- and someone may have legitimate need for a postgresql
client machine _without_ the server AND need the tcl client. Maybe a
pgaccess-based administration client, perhaps?).

Those who build from the source have the configure options to eliminate (in
their opinion) the cruft they don't need -- RPM users don't have that option --
so, I build everything, and split the distribution.

Some issues I have seen along these lines are:
1.) The header situation -- assumes a source tree somewhere so that you
can pull in things....
2.) The regression tests -- you know, someone MIGHT want to _run_ those
tests on a database server in a production situation where there is no make, no
compiler, and no PostgreSQL source tree (where config.guess resides). The
installation can and should prebuild all necessary binaries and should
preconfigure the results of a config.guess -- and the shell script should not
need to be invoked by make. (Yes, the RPMset does all this (except eliminate
the need for config.guess), and it's packaged as the postgresql-test rpm, which
is an optional 1.5MB package -- I wimped out and packaged the whole test
subtree....)

While I am all for source availability and the whole open source (free
software) model, I am also for easily installed and upgraded packages for users
who simply want to use the package in a production environment. I've been
there and done that -- and was more than a little frustrated at the state of
the RPM packaging. Thus my current situation :-).

--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Lamar Owen <lamar.owen@wgcr.org> writes:

/lib/cpp -M -I. -I../backend executor/spi.h |xargs -n 1|grep \\W|grep -v ^/|grep -v spi.h | sort |cpio -pdu $RPM_BUILD_ROOT/usr/include/pgsql

This could easily enough be included in the make install, couldn't it?
(Tom? Anyone?) I realize that GNU grepisms (or is \W common?) are
used above,

That's just the tip of the iceberg of the portability problems in the
above. If you want to propose that we actually do something like that
during 'make install', you're gonna have to work a lot harder.
(However, as a non-portable trick for getting a list of files that need
to be included in a hand-generated makefile, it's not bad.)

The more serious problem is "what else might an SPI module need besides
spi.h". Also, it disturbs me a lot that spi.h pulls in 80+ include
files in the first place --- there's got to be stuff in there that
needn't/shouldn't be exported. I know that an SPI developer who's just
trying to get some work done couldn't care less, but I'd like to see us
make some effort to actually clean up the list of files to be exported,
rather than fall back on automation that will let the list bloat even
more without anyone much noticing...

regards, tom lane

Lamar Owen <lamar.owen@wgcr.org> writes:

:-) I expect no less (than to work harder, of course....). Why not do
something like that, other than the 'out-of-sight, out-of-mind'
problem? Or, to put it far more bluntly, the SPI installation is very
broken if the SPI developer has to go around manually installing
header files that make install should have automatically taken care
of.

Agreed, but I'm worried about the 'out-of-sight, out-of-mind' aspect.

The more serious problem is "what else might an SPI module need besides
spi.h".

What else indeed. Isn't spi.h the exported SPI itself? We seem to
have a very poorly defined line between exported and private
interfaces

Ah-hah, I think you and I are on exactly the same wavelength there.

My whole problem with the spi.h-imports-88-headers business is that it
exposes in gory detail the fact that *we have absolutely no idea* what
we consider an exported backend interface and what we don't. I don't
like the idea of an automatic tool that exports whatever it thinks might
be needed, because if we let ourselves go down that path we will very
soon find ourselves trying to preserve backwards compatibility with
something that should never have been exported at all.

Basically I feel that this is a problem that requires some actual
thought and design judgment. I don't want to see us substituting
a one-liner script hack for design judgment. OTOH, I don't really
want to do the legwork myself (lame excuse: never having written
an SPI module myself, I have little idea what they need to see).
I'm just concerned about the long-term consequences of taking the
easy way out.

regards, tom lane

On Sat, 27 May 2000, Tom Lane wrote:

Lamar Owen <lamar.owen@wgcr.org> writes:

What else indeed. Isn't spi.h the exported SPI itself? We seem to
have a very poorly defined line between exported and private
interfaces

Ah-hah, I think you and I are on exactly the same wavelength there.

My whole problem with the spi.h-imports-88-headers business is that it
exposes in gory detail the fact that *we have absolutely no idea* what
we consider an exported backend interface and what we don't. I don't

Yes, this is a problem. And, yes, it needs to be taken care of in a designed,
methodical manner -- the spi.h header should only need a few things (I, like
you, have not done any SPI development yet to see what really _is_ needed....).

Basically I feel that this is a problem that requires some actual
thought and design judgment.

Yes, most certainly. Can someone who has actual SPI experience look at this?

I'm just concerned about the long-term consequences of taking the
easy way out.

I'll have to admit to taking the easy way out a little in my RPM solution --
but, it's only there so that an advertised development interface can be
actually used. Once a thorough look is taken at the whole header mess for SPI,
I still won't have to change anything, which is good both for me and for RPM
users, as I really don't keep track of every header file dependency change --
thus, RPM releases won't happen (again, as 7.0-1 was in error in this regard)
with broken header deps, requiring a bugfix package.

If no one else is forthcoming with a solution to this problem, I'll take a look
at it (possibly during the development cycle for 7.1 after the 7.0.x series has
stabilized, when the RPM cycle is at idle).

--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Hmmm...

I'm attached a small patch. This patch includes the list of dirs and
headers refered by spi.h (And _not_ more)

On Sat, 27 May 2000, Lamar Owen wrote:

On Sat, 27 May 2000, Tom Lane wrote:

My whole problem with the spi.h-imports-88-headers business is that it
exposes in gory detail the fact that *we have absolutely no idea* what
we consider an exported backend interface and what we don't. I don't

Yes, but _before_ somebody create the new (more structural?) header layout,
must all spi-developer search for this files?

Yes, this is a problem. And, yes, it needs to be taken care of in a
designed, methodical manner -- the spi.h header should only need a few
things (I, like you, have not done any SPI development yet to see what
really _is_ needed....).

--
nek;(