Faster install-sh in C

This has been saved for the 8.1 release:

http:/momjian.postgresql.org/cgi-bin/pgpatches2

---------------------------------------------------------------------------

Alvaro Herrera wrote:

Patchers,

I wrote an "install" program in C. It's supposed to replace the
config/install-sh script, limited to the functionality we need, i.e.
what is in Makefiles in the Pg main source tree. The main objective of
this exercise is to reduce "make install" execution time; a part of that
is being able to install multiple files with one command.

Portability testing right now is limited to my machine, which is Linux
with glibc 2.3.2 (Debian Sid).

With this in place, "make install" in src/include takes 17 seconds on
this machine, where the script version takes more than a minute. I
think this is a useful improvement.

Right now I'm missing a Makefile rule for it. It needs the pg_progname
symbol from src/port, and the includes from $(srcdir)/src/include and
$(builddir)/src/include. From the config directory, this works:

$(CC) $(CFLAGS) -I$(top_srcdir)/src/include -I$(top_builddir)/src/include ../src/port/path.o ../src/port/exec.o $< -o $@

Also, I don't know how to force it to build before executing "install",
short of putting it as a dependency in every makefile (which sounds
silly to me).

I attach the source code, and a src/include/Makefile modification to
use the multi-file install feature.

--
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"Hay quien adquiere la mala costumbre de ser infeliz" (M. A. Evans)

[ Attachment, skipping... ]

[ Attachment, skipping... ]

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Alvaro Herrera wrote:

I wrote an "install" program in C. It's supposed to replace the
config/install-sh script, limited to the functionality we need, i.e.
what is in Makefiles in the Pg main source tree. The main objective of
this exercise is to reduce "make install" execution time; a part of that
is being able to install multiple files with one command.

What's the status of this patch?

-Neil

Am Freitag, 4. M�rz 2005 01:11 schrieb Neil Conway:

Alvaro Herrera wrote:

I wrote an "install" program in C. It's supposed to replace the
config/install-sh script, limited to the functionality we need, i.e.
what is in Makefiles in the Pg main source tree. The main objective of
this exercise is to reduce "make install" execution time; a part of that
is being able to install multiple files with one command.

What's the status of this patch?

I'm not convinced that this is the best approach. Apparently, PostgreSQL is
about the only project that is somehow bothered by this. It would be more
productive to find out why we can't use the system install and find fixes for
that.

Anyway, if you want to speed up your installation today, just override the
INSTALL variable by hand.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

On Fri, Mar 04, 2005 at 11:20:07AM +0100, Peter Eisentraut wrote:

Am Freitag, 4. M�rz 2005 01:11 schrieb Neil Conway:

Alvaro Herrera wrote:

I wrote an "install" program in C. It's supposed to replace the
config/install-sh script, limited to the functionality we need, i.e.
what is in Makefiles in the Pg main source tree. The main objective of
this exercise is to reduce "make install" execution time; a part of that
is being able to install multiple files with one command.

What's the status of this patch?

I was playing a little with Make just before leaving, but since I'm
hardly an expert I was having difficulty making it all work. Also,
because I need to include bits from both the source dir and the build
dir (to get get_progname from the port dir and signal handling from
libpq), it was getting somewhat ugly. With a little more effort it can
be made to work cleanly.

I'm not convinced that this is the best approach. Apparently, PostgreSQL is
about the only project that is somehow bothered by this. It would be more
productive to find out why we can't use the system install and find fixes for
that.

Well, apparently everyone says the system install is not portable and
the discussion stops there. Probably some projects need a sh install
because they don't require a compiler, but we do; we won't require
anything beyond what we already do to have a C install. Maybe most
projects don't install a lot of files like we do, or maybe people just
don't care. (Neither did I care back when installing all headers wasn't
the default behaviour.) I do care now.

And given that the work is mostly already done, I don't see why we
couldn't use it.

Anyway, if you want to speed up your installation today, just override the
INSTALL variable by hand.

This doesn't solve the fact that the INSTALL program is called once for
each file in the include directories.

--
Alvaro Herrera (<alvherre[@]dcc.uchile.cl>)
Y una voz del caos me habl� y me dijo
"Sonr�e y s� feliz, podr�a ser peor".
Y sonre�. Y fui feliz.
Y fue peor.

Alvaro Herrera wrote:

Well, apparently everyone says the system install is not portable and
the discussion stops there.

Well, who actually says that? I know that I was the one who actually
coded up the current avoid-system-install-at-all-cost behavior, but
only because we were too annoyed at the time to do more research. Now,
as long as someone's spending time on it, it may be worth investigating
what is wrong and provide an explanation for the benefit of the entire
autotools-using community.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

On Mon, Mar 07, 2005 at 07:50:23PM +0100, Peter Eisentraut wrote:

Alvaro Herrera wrote:

Well, apparently everyone says the system install is not portable and
the discussion stops there.

Well, who actually says that? I know that I was the one who actually
coded up the current avoid-system-install-at-all-cost behavior, but
only because we were too annoyed at the time to do more research. Now,
as long as someone's spending time on it, it may be worth investigating
what is wrong and provide an explanation for the benefit of the entire
autotools-using community.

Well, the problem with this approach is that while I did have time to
write the C replacement, I didn't (and still don't) have access to a lot
of systems in order to check what the differences between the install
programs are.

Also, keep in my that this C install program has the extra feature of
being able to install multiple files on one invocation, per suggestion
from Tom Lane. This allows us to save the nested for-loop in
src/include/Makefile. GNU install (available on my system) also has
this capability, but would we be able to use the trick if we had to
cater for the lowest common denominator found on other, non-GNU-enabled
systems?

--
Alvaro Herrera (<alvherre[@]dcc.uchile.cl>)
"Llegar� una �poca en la que una investigaci�n diligente y prolongada sacar�
a la luz cosas que hoy est�n ocultas" (S�neca, siglo I)

Alvaro Herrera <alvherre@dcc.uchile.cl> writes:

Also, keep in my that this C install program has the extra feature of
being able to install multiple files on one invocation, per suggestion
from Tom Lane. This allows us to save the nested for-loop in
src/include/Makefile. GNU install (available on my system) also has
this capability, but would we be able to use the trick if we had to
cater for the lowest common denominator found on other, non-GNU-enabled
systems?

We would definitely not ... and my suspicion is that that is the largest
single component of the available speedup.

regards, tom lane

Tom Lane wrote:

Alvaro Herrera <alvherre@dcc.uchile.cl> writes:

Also, keep in my that this C install program has the extra feature of
being able to install multiple files on one invocation, per suggestion
from Tom Lane. This allows us to save the nested for-loop in
src/include/Makefile. GNU install (available on my system) also has
this capability, but would we be able to use the trick if we had to
cater for the lowest common denominator found on other, non-GNU-enabled
systems?

We would definitely not ... and my suspicion is that that is the largest
single component of the available speedup.

OK, what is 'install' doing for us that 'cp' and 'chmod' would not
already do? The following patch reduces the 'gmake install' in
/src/include from 8 seconds to 0.8 seconds.

Oh, and for testing I use a sourceforge shell farm account, and a HP
testdrive shell acount that gives me access to almost every operating
system.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian <pgman@candle.pha.pa.us> writes:

OK, what is 'install' doing for us that 'cp' and 'chmod' would not
already do?

Quite a lot of things, such as coping with busy target files --- not too
important for headers, but very important for executables and shlibs.

We might be able to get away with this for just the headers, though,
and that's certainly the bulk of the install work now.

regards, tom lane

Alvaro Herrera wrote:

Also, keep in my that this C install program has the extra feature of
being able to install multiple files on one invocation, per suggestion
from Tom Lane. This allows us to save the nested for-loop in
src/include/Makefile. GNU install (available on my system) also has
this capability, but would we be able to use the trick if we had to
cater for the lowest common denominator found on other, non-GNU-enabled
systems?

Why not check for GNU install in configure, and only enable the
optimization if it is available?

-Neil

Tom Lane wrote:

Bruce Momjian <pgman@candle.pha.pa.us> writes:

OK, what is 'install' doing for us that 'cp' and 'chmod' would not
already do?

Quite a lot of things, such as coping with busy target files --- not too
important for headers, but very important for executables and shlibs.

We might be able to get away with this for just the headers, though,
and that's certainly the bulk of the install work now.

Here is my next version of the patch that uses 'cp' and 'chmod' to
install multiple header files rather than 'install'.

I moved the file modes into variables so any changes are propogated to
src/include/Makefile.

This is 20 times faster than what we have now, 8 seconds vs 0.40 seconds.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Tom Lane wrote:

We might be able to get away with this for just the headers, though,
and that's certainly the bulk of the install work now.

Here is my next version of the patch that uses 'cp' and 'chmod' to
install multiple header files rather than 'install'.

I moved the file modes into variables so any changes are propogated to
src/include/Makefile.

This is 20 times faster than what we have now, 8 seconds vs 0.40 seconds.

I like it. Given that we are installing into directories we'd have just
created ourselves, which should have no extraneous files in them, a lot
of the objections one might raise go away.

Why don't you go ahead and apply that to HEAD for the time being, even
if we end up replacing it later? Might as well start saving developers'
time immediately.

regards, tom lane

Tom Lane wrote:

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Tom Lane wrote:

We might be able to get away with this for just the headers, though,
and that's certainly the bulk of the install work now.

Here is my next version of the patch that uses 'cp' and 'chmod' to
install multiple header files rather than 'install'.

I moved the file modes into variables so any changes are propogated to
src/include/Makefile.

This is 20 times faster than what we have now, 8 seconds vs 0.40 seconds.

I like it. Given that we are installing into directories we'd have just
created ourselves, which should have no extraneous files in them, a lot
of the objections one might raise go away.

Why don't you go ahead and apply that to HEAD for the time being, even
if we end up replacing it later? Might as well start saving developers'
time immediately.

Done.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

I am using 'cp' in current CVS for this. I assume it is as fast as the
C implementation, but if not, please let me know.

---------------------------------------------------------------------------

Alvaro Herrera wrote:

Patchers,

I wrote an "install" program in C. It's supposed to replace the
config/install-sh script, limited to the functionality we need, i.e.
what is in Makefiles in the Pg main source tree. The main objective of
this exercise is to reduce "make install" execution time; a part of that
is being able to install multiple files with one command.

Portability testing right now is limited to my machine, which is Linux
with glibc 2.3.2 (Debian Sid).

With this in place, "make install" in src/include takes 17 seconds on
this machine, where the script version takes more than a minute. I
think this is a useful improvement.

Right now I'm missing a Makefile rule for it. It needs the pg_progname
symbol from src/port, and the includes from $(srcdir)/src/include and
$(builddir)/src/include. From the config directory, this works:

$(CC) $(CFLAGS) -I$(top_srcdir)/src/include -I$(top_builddir)/src/include ../src/port/path.o ../src/port/exec.o $< -o $@

Also, I don't know how to force it to build before executing "install",
short of putting it as a dependency in every makefile (which sounds
silly to me).

I attach the source code, and a src/include/Makefile modification to
use the multi-file install feature.

--
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"Hay quien adquiere la mala costumbre de ser infeliz" (M. A. Evans)

[ Attachment, skipping... ]

[ Attachment, skipping... ]

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

On Sat, Jun 04, 2005 at 08:42:29PM -0400, Bruce Momjian wrote:

I am using 'cp' in current CVS for this. I assume it is as fast as the
C implementation, but if not, please let me know.

Yes, your change did make the installation considerably faster so I
abandoned the binary install idea. Thanks.

--
Alvaro Herrera (<alvherre[@]dcc.uchile.cl>)
"Lo esencial es invisible para los ojos" (A. de Saint Ex�pery)