add server include files to default installation?

Fabien COELHO <coelho@cri.ensmp.fr> writes:

I wish to submit a small patch so that server includes and
all necessary configuration files could be installed *by default*.

There is a reason why install-all-headers is not the default. It is
that it shouldn't be the default: most people do not need it, because
they will never build any C extensions for themselves. It would just
be a waste of disk space for them.

If we did do such a thing it would have zero impact on the majority of
users anyway, because the RPM and other packagers will still put these
files into separate postgresql-devel packages, which would still not be
installed by default.

regards, tom lane

Fabien COELHO wrote:

Dear hackers,

I wish to submit a small patch so that server includes and
all necessary configuration files could be installed *by default*.

The current status is that server includes files are only installed
if explicitly required (make install-all-headers).

The idea is that external extension modules (new types or whatever)
could then *rely* on the fact that these files are available, which
would make developping and installing these extensions quite easier.

As an illustration, the apache software installs by default all
necessary includes and even a special tailored command (apxs) so as
to help extension modules to be compiled, installed and even configured
easilly.

Is there any opposition to this move? Silence will mean consent;-)

Agreed. I would also like to see Makefile.global installed.
pg_config.h has C-level configs, and Makefile.global has the
Makefile-level configs.

We have identified that we need to make configuration easier for 3rd
party apps.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fabien COELHO wrote:

Dear hackers,

I wish to submit a small patch so that server includes and
all necessary configuration files could be installed *by default*.

The current status is that server includes files are only installed
if explicitly required (make install-all-headers).

The idea is that external extension modules (new types or whatever)
could then *rely* on the fact that these files are available, which
would make developping and installing these extensions quite easier.

As an illustration, the apache software installs by default all
necessary includes and even a special tailored command (apxs) so as
to help extension modules to be compiled, installed and even configured
easilly.

Is there any opposition to this move? Silence will mean consent;-)

Thanks in advance, have a nice day,

If the intention is that external extension types should be able to
build using the default installation you must also include the
src/Makefile.global and the src/Makefile.shlib. Cygwin ports are
dependent on the presence of src/utils/dllinit.c also (perhaps true for
win32 too). A dllinit.o would be sufficient of course.

Kind regards,

Thomas Hallgren

On Sat, 15 May 2004, Bruce Momjian wrote:

Fabien COELHO wrote:

Dear hackers,

I wish to submit a small patch so that server includes and
all necessary configuration files could be installed *by default*.

The current status is that server includes files are only installed
if explicitly required (make install-all-headers).

The idea is that external extension modules (new types or whatever)
could then *rely* on the fact that these files are available, which
would make developping and installing these extensions quite easier.

As an illustration, the apache software installs by default all
necessary includes and even a special tailored command (apxs) so as
to help extension modules to be compiled, installed and even configured
easilly.

Is there any opposition to this move? Silence will mean consent;-)

Agreed. I would also like to see Makefile.global installed.
pg_config.h has C-level configs, and Makefile.global has the
Makefile-level configs.

Don't agree with Makefile.global, cause then you have to also install the
physical template files for the various operating system too, no? What
else does Makefile.global rely on?

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664

Marc G. Fournier wrote:

As an illustration, the apache software installs by default all
necessary includes and even a special tailored command (apxs) so as
to help extension modules to be compiled, installed and even configured
easilly.

Is there any opposition to this move? Silence will mean consent;-)

Agreed. I would also like to see Makefile.global installed.
pg_config.h has C-level configs, and Makefile.global has the
Makefile-level configs.

Don't agree with Makefile.global, cause then you have to also install the
physical template files for the various operating system too, no? What
else does Makefile.global rely on?

I think you would need the src/makefile files. The template files are
only used during configure. You might need some more include stuff too,
not sure. I see:

include $(top_builddir)/src/Makefile.port

and this gets confusing because Makefile.port is a symlink to the actual
file. I wonder if we could copy the actual file as Makefile.port. I
think that would work.

Anyway, I can also see packagers greping the Makefile for information.
For example, it is hard to know the thread flags except from the file.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Dear Tom,

I wish to submit a small patch so that server includes and
all necessary configuration files could be installed *by default*.

There is a reason why install-all-headers is not the default.

Sure. I hope so!

I'm questionning the pros and cons. I'm arguing that the cons against the
current situation are strong. Thanks for providing the pros, so it can be
discussed.

It is that it shouldn't be the default: most people do not need it,

How will people know that they will not need it afterwards?

As far as I'm concerned, I noticed it afterwards:-(
I'm a pretty stupid user, but I'm sure I'm not the only one.

For instance several weeks after installing a pg, I wanted to test
tsearch2 on one database. What do I need to do? Well, basically I need a
full recompile, or at least a full extract of the sources, re-configure
and compilation of the relevant contrib part. When reconfiguring, I need
to remember what options I used.

Now, as someone trying to develop extensions, I need to rely on some sound
and extendable default installation.

because they will never build any C extensions for themselves.

Hey, how do they know that before hand? They might need extensions. If
they need extensions, it is likely that they may have a C part. Most
contrib have a C part. External projects are likely to have a C part.
Some in user land (e.g. SPI_*), but not necessarilly.

Say aclitem accessor functions or new aggregates that may be needed for
some project that may be of interest to people. If there are not in, you
have to compile them. If the project cannot rely on the fact that
everything needed will be provided, it is no good.

It would just be a waste of disk space for them.

2MB... that is less than 0.002 EUR.

Moreover, I do not think that it is wasted, especially as postgresql is
moving things outside the main project (pgfoundry, contrib, gborg). You
must help external projects to live if you want them to leave;-)

As a compromise, we can have two makefile targets, say "install" and
"light-install".

I argue that the default should be the one which enable later extensions.

If we did do such a thing it would have zero impact on the majority of
users anyway, because the RPM and other packagers will still put these
files into separate postgresql-devel packages, which would still not be
installed by default.

Thanks for this argument! If packagers do it (I know of BSD ports) it mean
that they think it is useful. So it should be the default for people who
*bother* to install pg by hand, so that they don't have to do it twice
because they did not notice this installation subtlety.

IMHO, it is the core business of postgresql to help external projects by
providing a usable default infrastructure for them. As another example of
this bad behavior, having external projects (pgadmin or phppgadmin) to
"parse" aclitem descriptors because you want to keep things opaque and
they need it anyway is damaging.

Thanks for you answer, have a nice day,

--
Fabien Coelho - coelho@cri.ensmp.fr

Hi all,

Attached is a patch against HEAD implementing tablespaces.

I've done some testing on Linux and BSD. I've also compiled without
HAVE_SYMLINK defined -- which determines whether or not tablespaces are
available.

The reason for this is that symlinks are used extensively to simplify
access to relations in hairy places.

This is extensive discussion of this in the archives.

There are some outstanding TODOs. These include

pg_dump
documentation
regression tests
comment on

Chris KL has offered to tackle these.

Other outstanding things which need to be resolved:

XLog entries:

Within a tablespace, the files for different databases are isolated into
directories. Those directory names are the database oid for those files.
Since we create those files, we should log them. The question is, what is
the best way: appending it to an xlrec and storing its length in the xlrec
or adding another rdata with that in it?

The same goes for symlinks, where we need to store the oldpath and
newpath.

Alternative database location:

Should this code be removed now?

Drop tablespace:

I haven't added an interlock here (as suggested by Tom) to handle
situations where one transaction schedules the removal of a tablespace and
concurrently another transaction makes use of that tablespace. I'd
appreciate some advice on what this would look like: should a command
creating an object touch a CREATE a file and a DROP TABLESPACE block until
the file goes away, then check if it can drop the tablespace (ie, that the
tablespace is empty)? Likewise when DROP TABLESPACE proceeds, should it
block creates?

The other thing is handling DROP TABLESPACE in a transaction where the
tablespace has been emptied (using DROP TABLE, etc) during that
transaction. Because we only schedule the unlink() of the relation, the
data files will (and must) still be around. Is it reasonable to look
inside PendingRelDeletes to see if the tablespace will be empty at COMMIT?

Basic usage rundown

initdb sets up a new directory pg_tablespaces with some symlinks with in
it. To create a tablespace, you must first up create a tablespace
directory. Then you can create a tablespace inside postgres using:

CREATE TABLESPACE <name> LOCATION '/path/to/tbl/dir'

Then, the DDL commands can have a tablespace associatged with it:

CREATE DATABASE ... TABLESPACE <name>
CREATE SCHEMA ... TABLESPACE <name>
CREATE TABLE ... TABLESPACE <name>
CREATE INDEX ... TABLESPACE <name>
CREATE SEQUENCE ... TABLESPACE <name>

Comments, criticisms, etc, all welcome, of course.

Thanks,

Gavin

Dear hackers,

Agreed. I would also like to see Makefile.global installed.
pg_config.h has C-level configs, and Makefile.global has the
Makefile-level configs.

There is also "config.status" which is definitely of interest as it
allows to recreate the build tree, and which is not installed by default.

Independently of the actual file list to be included, where
could these makefiles be installed?

- share is for "architecture independent" files. Not really the case.

- include is rather for C files... but Makefile.global and others are
actually included, so it may make sense? If so, should it be directly
in the include/postgresql subdir, or some special subdir, say
"include/postgresql/config"?

- some other directory?

I would tend to put everything in "include/postgresql/config":
- Makefile.global
- a copy of Makefile.port (the actual file, not the link)
- config.status
- possibly other included files (NLS? win32? cygwin?)

Another issue is that Makefile.global seems hardwired to be in
the "src" subdirectory of a postgresql source tree, and to find other
files there:

# Pull in platform-specific magic
include $(top_builddir)/src/Makefile.port

This is an issue for external tools that would like to include
Makefile.global so as to be in the same environment as the server
compilation.

Any idea? The best I have would be to create a "src" subdir in the
installation, so that top_builddir could be set to
".../include/postgresql/config" and everything would work from that
point of view.

--
Fabien COELHO _ http://www.cri.ensmp.fr/~coelho _ Fabien.Coelho@ensmp.fr
CRI-ENSMP, 35, rue Saint-Honoré, 77305 Fontainebleau cedex, France
phone: (+33|0) 1 64 69 {voice: 48 52, fax: 47 09, standard: 47 08}
________ All opinions expressed here are mine _________

Gavin Sherry wrote:

Alternative database location:

Should this code be removed now?

Yes, I believe we agreed on this. One of the committers will take care
of that.

The only downside to removal is that folks without symlinks (I believe
Win32 only) will loose that functionality with nothing to replace it.
However, I think the clarity of removing it is worth it. Also, I think
someone had a special way to do symlinks on Win32 and we should look
into that.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Alternative database location:

Should this code be removed now?

Yes, I believe we agreed on this. One of the committers will
take care of that.

The only downside to removal is that folks without symlinks (I believe
Win32 only) will loose that functionality with nothing to replace it.
However, I think the clarity of removing it is worth it.
Also, I think someone had a special way to do symlinks on
Win32 and we should look into that.

If we're talking *directory symlinks only*, it can be done on Win32.
It's pretty ugly, the native UI doesn't support showing them etc, but it
can be done. (This is only Windows 2000+, not NT4, btw)

I still would like a non-symlink solution on Unix as well, but I've
noticed I'm with the minority side there, and this is still a lot better
than nothing...

//Magnus

I would tend to put everything in "include/postgresql/config":

On second thought, I would hesitate with "lib/...", as config file are
plateform specific so they cannot be shared with other plateforms,
although it could be the case for include files that they could be shared.

--
Fabien Coelho - coelho@cri.ensmp.fr

Bruce Momjian wrote:

The only downside to removal is that folks without symlinks (I believe
Win32 only) will loose that functionality with nothing to replace it.
However, I think the clarity of removing it is worth it. Also, I think
someone had a special way to do symlinks on Win32 and we should look
into that.

Windows 2000 and later support mount points - you can attach a new
partition as C:\pgsql\data\xlog instead of D:\. That might be enough for
most users. IIRC there was a tool to create arbitrary links, but it was
removed just before W2K final.

--
Manfred

"Manfred Spraul" <manfred@colorfullife.com> wrote in message
news:40A8EEE9.3000405@colorfullife.com...

Bruce Momjian wrote:

The only downside to removal is that folks without symlinks (I believe
Win32 only) will loose that functionality with nothing to replace it.
However, I think the clarity of removing it is worth it. Also, I think
someone had a special way to do symlinks on Win32 and we should look
into that.

Windows 2000 and later support mount points - you can attach a new
partition as C:\pgsql\data\xlog instead of D:\. That might be enough for
most users. IIRC there was a tool to create arbitrary links, but it was
removed just before W2K final.

If you run NTFS, it's still possible to use arbitrary links. In the Windows
world, they are called junctions. Microsoft does not provide a junction tool
for some reason (perhaps because it's limited to NTFS). A good tool, free
and with source, can be found here
http://www.sysinternals.com/ntw2k/source/misc.shtml#junction I use this tool
myself. Works like a charm.

Kind regards,

Thomas Hallgren

Thomas Hallgren wrote:

"Manfred Spraul" <manfred@colorfullife.com> wrote in message
news:40A8EEE9.3000405@colorfullife.com...

Bruce Momjian wrote:

The only downside to removal is that folks without symlinks (I believe
Win32 only) will loose that functionality with nothing to replace it.
However, I think the clarity of removing it is worth it. Also, I think
someone had a special way to do symlinks on Win32 and we should look
into that.

Windows 2000 and later support mount points - you can attach a new
partition as C:\pgsql\data\xlog instead of D:\. That might be enough for
most users. IIRC there was a tool to create arbitrary links, but it was
removed just before W2K final.

If you run NTFS, it's still possible to use arbitrary links. In the Windows
world, they are called junctions. Microsoft does not provide a junction tool
for some reason (perhaps because it's limited to NTFS). A good tool, free
and with source, can be found here
http://www.sysinternals.com/ntw2k/source/misc.shtml#junction I use this tool
myself. Works like a charm.

We've looked at it before. Apart from anything else I don't think its
license is compatible with PostgreSQL's.

Also, IIRC NTFS junctions also have some severe limitations.

cheers

andrew

The only downside to removal is that folks without symlinks

(I believe

Win32 only) will loose that functionality with nothing to

replace it.

However, I think the clarity of removing it is worth it.

Also, I think

someone had a special way to do symlinks on Win32 and we should look
into that.

Windows 2000 and later support mount points - you can attach a new
partition as C:\pgsql\data\xlog instead of D:\. That might

be enough for

most users. IIRC there was a tool to create arbitrary links,

but it was

removed just before W2K final.

If you run NTFS, it's still possible to use arbitrary links.

In the Windows

world, they are called junctions. Microsoft does not provide

a junction tool

for some reason (perhaps because it's limited to NTFS). A

good tool, free

and with source, can be found here
http://www.sysinternals.com/ntw2k/source/misc.shtml#junction

I use this tool

myself. Works like a charm.

We've looked at it before. Apart from anything else I don't think its
license is compatible with PostgreSQL's.

Well, people can still use it. We just can't distribute it... We can
always link to it.
But unless there is a GUI tool (actually, unless it shows up in the
*default* GUI tool), expect there to be questions. An

Also, IIRC NTFS junctions also have some severe limitations.

The main being they can't do files, and there are few tools for them.
Also, most win32 admins are *NOT* experienced with them. Sure, they are
used for the NETLOGON directory, but how many admins know that...

//Magnus

Magnus Hagander wrote:

If you run NTFS, it's still possible to use arbitrary links.

In the Windows

world, they are called junctions. Microsoft does not provide

a junction tool

for some reason (perhaps because it's limited to NTFS). A

good tool, free

and with source, can be found here
http://www.sysinternals.com/ntw2k/source/misc.shtml#junction

I use this tool

myself. Works like a charm.

We've looked at it before. Apart from anything else I don't think its
license is compatible with PostgreSQL's.

Well, people can still use it. We just can't distribute it... We can
always link to it.
But unless there is a GUI tool (actually, unless it shows up in the
*default* GUI tool), expect there to be questions. An

I assume we can just look at the source and write our own version
bypassing any license.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Alternative database location:

Should this code be removed now?

I think that this:

CREATE DATABASE blah LOCATION 'xyz';

Should now be interpreted to mean:

CREATE TABLESPACE blah_tbsp LOCATION 'xyz';
CREATE DATABSE blah TABLESPACE blah_tbsp;

Or something like that...

Chris

If you run NTFS, it's still possible to use arbitrary links. In the Windows
world, they are called junctions. Microsoft does not provide a junction tool
for some reason (perhaps because it's limited to NTFS). A good tool, free
and with source, can be found here
http://www.sysinternals.com/ntw2k/source/misc.shtml#junction

I use this tool myself. Works like a charm.

We've looked at it before. Apart from anything else I don't think its
license is compatible with PostgreSQL's.

The tool was suggested for use as is, not for inclusion in pg source.
It can be used to e.g. symlink xlog manually.

If you want something in pg source, the relevant lines (about 20)
can be copied from MSDN.

Andreas

Marc G. Fournier wrote:

On Sat, 15 May 2004, Bruce Momjian wrote:

Fabien COELHO wrote:

Dear hackers,

I wish to submit a small patch so that server includes and
all necessary configuration files could be installed *by default*.

The current status is that server includes files are only installed
if explicitly required (make install-all-headers).

The idea is that external extension modules (new types or whatever)
could then *rely* on the fact that these files are available, which
would make developping and installing these extensions quite easier.

As an illustration, the apache software installs by default all
necessary includes and even a special tailored command (apxs) so as
to help extension modules to be compiled, installed and even configured
easilly.

Is there any opposition to this move? Silence will mean consent;-)

Agreed. I would also like to see Makefile.global installed.
pg_config.h has C-level configs, and Makefile.global has the
Makefile-level configs.

Don't agree with Makefile.global, cause then you have to also install the
physical template files for the various operating system too, no? What
else does Makefile.global rely on?

Depending on the OS you also need funny things like mkldexport shell
scripts and the like. I thought we had a consensus of providing a
template build environment for external modules. This half-baked attempt
is not doing it. The makefiles make assumptions about their location
that aren't true any more when they're installed somewhere else.

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #