Linking backend in one piece

On Fri, Feb 22, 2008 at 7:48 AM, Peter Eisentraut <peter_e@gmx.net> wrote:

Here is a patch so that the backend is linked in one piece instead of using
the SUBSYS.o files.

The question is how we want to activate that. I currently used make
BIGLINK=1, which is obviously just for testing. Should we just turn it on by
default and see if anyone complains?

I say go for it. We'll soon know if it kills the buildfarm.

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com
The Oracle-compatible database company

Peter Eisentraut wrote:

Here is a patch so that the backend is linked in one piece instead of using
the SUBSYS.o files.

The question is how we want to activate that. I currently used make
BIGLINK=1, which is obviously just for testing. Should we just turn it on by
default and see if anyone complains?

Hmm. Do we need the text file? I was kinda hoping we could just aggregate
each subdir's OBJS into a big variable listing all the needed files, and then
invoking the linker with that.

Also, the concatenating thing looks weird. Did you try using
.SECOND_EXPANSION? Sometime ago I came up with this trick to link
files from the backend into a src/bin/ subdirectory, which we use to
build a Replicator utility that links some files from the backend. It
looks like this:

# Common makefile to symlink files from the backend directory. The subdir
# makefile should define a LN_OBJS variable, listing the files it wants to
# compile from any particular backend directory, which is defined in the
# BACKEND_DIR variable. If those files are in turn inside another directory
# then use the name including that directory, and replace the / with __.
# For example, files in src/backend/access/ should be listed with
# BACKEND_DIR = access
# and the files themselves would be
# LN_OBJS = transam__xact.o gist__gistget.o
#
# Then include this file. Extra files may be declared in the variable OBJS.

mammoth_includedir=$(top_srcdir)/src/bin/mammoth/include

OBJS += $(LN_OBJS)

LN_SRC = $(LN_OBJS:.o=.c)

.SECONDEXPANSION:
$(LN_SRC): %: $$(subst __,/,$(top_srcdir)/src/backend/$(BACKEND_DIR)/%)
rm -f $@ && $(LN_S) $(subst __,/,$<) $@

I think we should be able to use something like this here.

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Peter Eisentraut <peter_e@gmx.net> writes:

Here is a patch so that the backend is linked in one piece instead of using
the SUBSYS.o files.

The question is how we want to activate that. I currently used make
BIGLINK=1, which is obviously just for testing. Should we just turn it on by
default and see if anyone complains?

What is the build time like with vs without this?

regards, tom lane

Am Freitag, 22. Februar 2008 schrieb Tom Lane:

What is the build time like with vs without this?

It's virtually indistinguishable. The big linker call doesn't take any
measurable extra time.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Am Freitag, 22. Februar 2008 schrieb Alvaro Herrera:

Hmm. �Do we need the text file? �I was kinda hoping we could just aggregate
each subdir's OBJS into a big variable listing all the needed files, and
then invoking the linker with that.

Well, my goal here was that we could use both ways of building for a while
because we have no experience with how long command lines and argument lists
we can handle portably.

Eventually, we could ideally transform the subdirectory Makefiles from
independently callable Makefiles to mere include files so we have only one
big dependency tree at the top, which would get rid of the annoying and
time-consuming directory traversal. This, however, raises other issues, such
as that some compilers do not support creating output files in
subdirectories. I tried to support all of this in parallel but didn't
succeeed without creating a big mess. We can always come back to that later.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut wrote:

Eventually, we could ideally transform the subdirectory Makefiles from
independently callable Makefiles to mere include files so we have only one
big dependency tree at the top, which would get rid of the annoying and
time-consuming directory traversal. This, however, raises other issues, such
as that some compilers do not support creating output files in
subdirectories. I tried to support all of this in parallel but didn't
succeeed without creating a big mess. We can always come back to that later.

Ah, right, that was what I was thinking about. Sure, we can investigate
that later.

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Peter Eisentraut <peter_e@gmx.net> writes:

Well, my goal here was that we could use both ways of building for a while
because we have no experience with how long command lines and argument lists
we can handle portably.

Yeah, I think it would be folly to assume that we can name all the
individual object files in one big command line. But isn't the current
patch trying to do exactly that?

regards, tom lane

Am Freitag, 22. Februar 2008 schrieb Tom Lane:

Yeah, I think it would be folly to assume that we can name all the
individual object files in one big command line. But isn't the current
patch trying to do exactly that?

The current patch assumes that it works in most environments and offers the
old way as a fallback.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut <peter_e@gmx.net> writes:

Am Freitag, 22. Februar 2008 schrieb Tom Lane:

Yeah, I think it would be folly to assume that we can name all the
individual object files in one big command line. But isn't the current
patch trying to do exactly that?

The current patch assumes that it works in most environments and offers the
old way as a fallback.

I see, that's why you wanted the conditional in there. It would need to
be documented in the build instructions.

Some experimentation with "find" suggests that the actual command length
will be under 12K, which is probably going to work on most modern
platforms, so maybe I'm worried over nothing. Let's put it in and see
what the buildfarm says ...

regards, tom lane