DTrace build dependency rules

Started by Mark Johnstonover 10 years ago8 messages

markj@FreeBSD.org

over 10 years ago

1 attachment(s)

Hi,

There seems to be a bug in the make rules when DTrace is enabled. It
causes dtrace -G to be invoked twice when building PostgreSQL as a
FreeBSD port: once during the build itself, and once during
installation. For a long time this has been worked around on FreeBSD
with a change to libdtrace itself, but this workaround is proving
problematic and I'd like to fix the problem properly. I'm not sure
whether the problem has been observed on other operating systems that
support DTrace.

The bug is in src/backend/Makefile. probes.o, the dtrace(1)-generated
object file, depends on the objfiles.txt for each of the backend
subdirs. These files depend in turn on the object files themselves; if
objfiles.txt is out of date with respect to one of its object files, the
mtime of objfiles.txt is updated with "touch" (see backend/common.mk).
The problem is that dtrace -G, which runs at the end of the build,
modifies a number of object files (it overwrites their probe sites with
NOPs), thus making their corresponding objfiles.txt out of date. Then,
when "make install" traverses the backend subdirs, it updates
objfiles.txt, which causes probes.o to be rebuilt, resulting in an error
from dtrace(1).

The attached patch fixes the problem by having probes.o depend on object
files directly, rather than on objfiles.txt. I've tested it with
PostgreSQL 9.0-9.4 on FreeBSD CURRENT.

Thanks!
-Mark

Attachments:

postgres_dtrace_rule.difftext/x-diff; charset=us-asciiDownload

diff --git a/src/backend/Makefile b/src/backend/Makefile
index 98b978f..b1e3969 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -180,8 +180,8 @@ $(top_builddir)/src/include/utils/probes.h: utils/probes.h
 	    $(LN_S) "../../../$(subdir)/utils/probes.h" .
 
 
-utils/probes.o: utils/probes.d $(SUBDIROBJS)
-	$(DTRACE) $(DTRACEFLAGS) -C -G -s $(call expand_subsys,$^) -o $@
+utils/probes.o: utils/probes.d $(call expand_subsys,$(SUBDIROBJS))
+	$(DTRACE) $(DTRACEFLAGS) -C -G -s $< $(call expand_subsys,$(SUBDIROBJS)) -o $@
 
 
 ##########################################################################

Robert Haas

robertmhaas@gmail.com

over 10 years ago

In reply to: Mark Johnston (#1)

Re: DTrace build dependency rules

On Sat, Aug 15, 2015 at 6:45 PM, Mark Johnston <markj@freebsd.org> wrote:

There seems to be a bug in the make rules when DTrace is enabled. It
causes dtrace -G to be invoked twice when building PostgreSQL as a
FreeBSD port: once during the build itself, and once during
installation. For a long time this has been worked around on FreeBSD
with a change to libdtrace itself, but this workaround is proving
problematic and I'd like to fix the problem properly. I'm not sure
whether the problem has been observed on other operating systems that
support DTrace.

The bug is in src/backend/Makefile. probes.o, the dtrace(1)-generated
object file, depends on the objfiles.txt for each of the backend
subdirs. These files depend in turn on the object files themselves; if
objfiles.txt is out of date with respect to one of its object files, the
mtime of objfiles.txt is updated with "touch" (see backend/common.mk).
The problem is that dtrace -G, which runs at the end of the build,
modifies a number of object files (it overwrites their probe sites with
NOPs), thus making their corresponding objfiles.txt out of date. Then,
when "make install" traverses the backend subdirs, it updates
objfiles.txt, which causes probes.o to be rebuilt, resulting in an error
from dtrace(1).

Gosh, that's pretty ugly. I would have thought it would be a real
no-no to update the .o file once it got generated. If nothing else, a
modification to the .c file concurrent with a make invocation might
lead to the .o not getting rebuilt the next time make is run.

The attached patch fixes the problem by having probes.o depend on object
files directly, rather than on objfiles.txt. I've tested it with
PostgreSQL 9.0-9.4 on FreeBSD CURRENT.

I don't see a particular reason not to make this change, though.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Alvaro Herrera

alvherre@2ndquadrant.com

over 10 years ago

In reply to: Robert Haas (#2)

Re: DTrace build dependency rules

Robert Haas wrote:

On Sat, Aug 15, 2015 at 6:45 PM, Mark Johnston <markj@freebsd.org> wrote:

The bug is in src/backend/Makefile. probes.o, the dtrace(1)-generated
object file, depends on the objfiles.txt for each of the backend
subdirs. These files depend in turn on the object files themselves; if
objfiles.txt is out of date with respect to one of its object files, the
mtime of objfiles.txt is updated with "touch" (see backend/common.mk).
The problem is that dtrace -G, which runs at the end of the build,
modifies a number of object files (it overwrites their probe sites with
NOPs), thus making their corresponding objfiles.txt out of date. Then,
when "make install" traverses the backend subdirs, it updates
objfiles.txt, which causes probes.o to be rebuilt, resulting in an error
from dtrace(1).

Gosh, that's pretty ugly. I would have thought it would be a real
no-no to update the .o file once it got generated. If nothing else, a
modification to the .c file concurrent with a make invocation might
lead to the .o not getting rebuilt the next time make is run.

I had the same thought, and wondered for a bit whether we should instead
have the compilation rules produce some intermediate file (prior to
dtrace fumbling), then emit the .o from dtrace -G. OTOH this might be
more trouble than is worth for a feature that doesn't see a lot of use.

The attached patch fixes the problem by having probes.o depend on object
files directly, rather than on objfiles.txt. I've tested it with
PostgreSQL 9.0-9.4 on FreeBSD CURRENT.

I don't see a particular reason not to make this change, though.

I wanted to test it on Linux yesterday but didn't get any further than
installing a couple of packages.

--
ï¿½lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Robert Haas

robertmhaas@gmail.com

about 10 years ago

In reply to: Alvaro Herrera (#3)

Re: DTrace build dependency rules

On Tue, Aug 18, 2015 at 12:08 PM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:

Robert Haas wrote:

On Sat, Aug 15, 2015 at 6:45 PM, Mark Johnston <markj@freebsd.org> wrote:

The bug is in src/backend/Makefile. probes.o, the dtrace(1)-generated
object file, depends on the objfiles.txt for each of the backend
subdirs. These files depend in turn on the object files themselves; if
objfiles.txt is out of date with respect to one of its object files, the
mtime of objfiles.txt is updated with "touch" (see backend/common.mk).
The problem is that dtrace -G, which runs at the end of the build,
modifies a number of object files (it overwrites their probe sites with
NOPs), thus making their corresponding objfiles.txt out of date. Then,
when "make install" traverses the backend subdirs, it updates
objfiles.txt, which causes probes.o to be rebuilt, resulting in an error
from dtrace(1).

Gosh, that's pretty ugly. I would have thought it would be a real
no-no to update the .o file once it got generated. If nothing else, a
modification to the .c file concurrent with a make invocation might
lead to the .o not getting rebuilt the next time make is run.

I had the same thought, and wondered for a bit whether we should instead
have the compilation rules produce some intermediate file (prior to
dtrace fumbling), then emit the .o from dtrace -G. OTOH this might be
more trouble than is worth for a feature that doesn't see a lot of use.

Given the lack of further interest from the PostgreSQL community,
that's my guess. I've pushed this patch to master; let's see if we
get any complaints. If it makes life better for FreeBSD without
making life worse for anyone else, I suppose we might as well do it
until something better comes along.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Enrique Escobar

ekurth@live.com.mx

about 10 years ago

In reply to: Robert Haas (#4)

Duda

Hola
Tengo una duda, que tan pesado es poner ssl en una base. (me refiero si
es mas pesado para el equipo usar esta conexión o es igual a una con ip
en hba?
Mil Gracias

--
Saludos Enrique

Rodolfo Campero

rodolfo.campero@anachronics.com

about 10 years ago

In reply to: Enrique Escobar (#5)

Re: Duda

La lista de correo apropiada para tu consulta es pgsql-es-ayuda
(pgsql-es-ayuda(at)postgresql(dot)org).
Saludos,

El 13 de octubre de 2015, 16:51, Enrique Escobar <ekurth@live.com.mx>
escribió:

Hola
Tengo una duda, que tan pesado es poner ssl en una base. (me refiero si es
mas pesado para el equipo usar esta conexión o es igual a una con ip en hba?
Mil Gracias

--
Saludos Enrique

--
Rodolfo Campero

Robert Haas

robertmhaas@gmail.com

about 10 years ago

In reply to: Robert Haas (#4)

Re: DTrace build dependency rules

On Tue, Oct 13, 2015 at 3:47 PM, Robert Haas <robertmhaas@gmail.com> wrote:

On Tue, Aug 18, 2015 at 12:08 PM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:

Robert Haas wrote:

On Sat, Aug 15, 2015 at 6:45 PM, Mark Johnston <markj@freebsd.org> wrote:

The bug is in src/backend/Makefile. probes.o, the dtrace(1)-generated
object file, depends on the objfiles.txt for each of the backend
subdirs. These files depend in turn on the object files themselves; if
objfiles.txt is out of date with respect to one of its object files, the
mtime of objfiles.txt is updated with "touch" (see backend/common.mk).
The problem is that dtrace -G, which runs at the end of the build,
modifies a number of object files (it overwrites their probe sites with
NOPs), thus making their corresponding objfiles.txt out of date. Then,
when "make install" traverses the backend subdirs, it updates
objfiles.txt, which causes probes.o to be rebuilt, resulting in an error
from dtrace(1).

Gosh, that's pretty ugly. I would have thought it would be a real
no-no to update the .o file once it got generated. If nothing else, a
modification to the .c file concurrent with a make invocation might
lead to the .o not getting rebuilt the next time make is run.

I had the same thought, and wondered for a bit whether we should instead
have the compilation rules produce some intermediate file (prior to
dtrace fumbling), then emit the .o from dtrace -G. OTOH this might be
more trouble than is worth for a feature that doesn't see a lot of use.

Given the lack of further interest from the PostgreSQL community,
that's my guess. I've pushed this patch to master; let's see if we
get any complaints. If it makes life better for FreeBSD without
making life worse for anyone else, I suppose we might as well do it
until something better comes along.

Per /messages/by-id/24541.1444928519@sss.pgh.pa.us
I have had to revert this patch, because it's breaking parallel builds
even for non-dtrace users.

Sorry for the inconvenience.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Claudio Freire

klaussfreire@gmail.com

about 10 years ago

In reply to: Rodolfo Campero (#6)

Re: Duda

2015-10-13 17:22 GMT-03:00 Rodolfo Campero <rodolfo.campero@anachronics.com>:

La lista de correo apropiada para tu consulta es pgsql-es-ayuda
(pgsql-es-ayuda(at)postgresql(dot)org).
Saludos,

El 13 de octubre de 2015, 16:51, Enrique Escobar <ekurth@live.com.mx>
escribió:

Hola
Tengo una duda, que tan pesado es poner ssl en una base. (me refiero si es
mas pesado para el equipo usar esta conexión o es igual a una con ip en hba?
Mil Gracias

A tu pregunta, sí, es más pesado. Considerablemente, pero el impacto
depende mucho de tu carga.

Lo más pesado de SSL es establecer la conexión. Si son conexiones con
larga vida, no hay tanto impacto. Menos cuanto menos transferencia de
datos tengas (ej: resultados pequeños). Pero si van a ser muchas
conexiones de corta duración, ahí sí el costo de establecer la
conexión te va a impactar mucho.

Como siempre, bencharkeá.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers