MacPorts xsltproc is very slow?

Hello Thomas,

25.11.2017 06:38, Thomas Munro wrote:

Hi,

Does anyone know why I'd see this difference in "make docs" performance?

Can you show the output of
make XSLTPROCFLAGS="--profile" -C doc/src/sgml/html
or
make XSLTPROCFLAGS="--profile" docs
?

------
Alexander Lakhin
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Thomas Munro <thomas.munro@enterprisedb.com> writes:

Does anyone know why I'd see this difference in "make docs" performance?

1. On macOS using Apple's /usr/bin/xsltproc (--version says libxml
20902, libxslt 10128 and libexslt 817), it builds in a few minutes but
produces warnings like this:

postgres.sgml:408: element biblioentry: validity error : ID ston89b
already defined
postgres.sgml:426: element biblioentry: validity error : ID ston90a
already defined
postgres.sgml:454: element biblioentry: validity error : ID ston90b
already defined

FWIW, I see no such warnings using the system xsltproc on either Sierra
or High Sierra. Both of them show

$ xsltproc --version
Using libxml 20904, libxslt 10129 and libexslt 817
xsltproc was compiled against libxml 20904, libxslt 10129 and libexslt 817
libxslt 10129 was compiled against libxml 20904
libexslt 817 was compiled against libxml 20904

and build the HTML docs in about a minute and a half.

I noticed that the Apple version is using libxslt 1.1.28 (for context,
that's the same as Debian Jessie used; Stretch/Buster/Sid are on
1.1.29 -- I'm guessing many of you are using that?), whereas MacPorts
is shipping libxslt 1.1.32. I know next to nothing about these tools
but I wonder if something we're doing gets horribly slow in future
libxslt versions that will come down the pipeline on other
distributions. Or if the MacPorts port is just borked somehow.

This seems vaguely reminiscent of this recent discussion:

/messages/by-id/8F33D0C7-E12E-4996-990C-3CF0C5ED0437@filmlance.se

in which the conclusion seemed to be that Apple's build of zlib was
considerably faster than another build (of uncertain provenance, mind you,
so maybe that is unrelated). I wonder whether Apple are using more
aggressive optimization flags than other people. OTOH, while it would
not surprise me if Apple put some work into making zlib go fast, it
seems less likely that they'd expend effort or risk on xsltproc.

regards, tom lane

On Sat, Nov 25, 2017 at 4:57 PM, Alexander Lakhin <exclusion@gmail.com> wrote:

Can you show the output of
make XSLTPROCFLAGS="--profile" -C doc/src/sgml/html
or
make XSLTPROCFLAGS="--profile" docs
?

Hi Alexander, please see attached.

--
Thomas Munro
http://www.enterprisedb.com

I wrote:

... I wonder whether Apple are using more
aggressive optimization flags than other people. OTOH, while it would
not surprise me if Apple put some work into making zlib go fast, it
seems less likely that they'd expend effort or risk on xsltproc.

I checked Fedora 26 and found that it's using an identical source
version of xsltproc:

$ xsltproc --version
Using libxml 20904, libxslt 10129 and libexslt 817
xsltproc was compiled against libxml 20904, libxslt 10129 and libexslt 817
libxslt 10129 was compiled against libxml 20904
libexslt 817 was compiled against libxml 20904

Its time to build the HTML docs ... 46 seconds. This is on a machine
intermediate in speed between the Sierra and High Sierra machines whose
timings I quoted before. So we can definitively discard the idea that
Apple sprinkled any magic pixie dust on their xsltproc build. Your
MacPorts build is the outlier, leaving us with the theories that it was
built at -O0 or there's a performance bug in the newer source releases.
I have no doubt that xmlsoft.org would be interested if you can narrow
it down to the latter.

regards, tom lane

25.11.2017 07:49, Thomas Munro wrote:

On Sat, Nov 25, 2017 at 4:57 PM, Alexander Lakhin
<exclusion@gmail.com> wrote:

Can you show the output of make XSLTPROCFLAGS="--profile" -C
doc/src/sgml/html or make XSLTPROCFLAGS="--profile" docs ?

Hi Alexander, please see attached.

Thanks! Just to compare your results with my:
number               match                name      mode  Calls Tot
100us  Avg
    0               d:appendix             label.markup 22503 70018988 
3111
    1               chunk-all-sections 1394 36361616 26084
    2               d:chapter              label.markup 24918
20792834   834
    3                                      href.target    23895
18556477   776
    4                                   footer.navigation  1394
5276429  3785
    5                        header.navigation             1394
5095535  3655
    6                         gentext.template           237071 857659 
   3

vs

number               match                name      mode  Calls Tot
100us Avg

    0                         gentext.template 247659    742878    2
    1                       chunk-all-sections 1394    705195  505
    2                              href.target 35446    663090   18

I wonder, what version of docbook-xsl are you using?
(I have 1.79.1+dfsg-1).
Can you check with 1.79+ (if yours is older)?

------
Alexander Lakhin
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

On Sat, Nov 25, 2017 at 7:18 PM, Alexander Lakhin <exclusion@gmail.com> wrote:

Thanks! Just to compare your results with my:

number match name mode Calls Tot 100us
Avg
0 d:appendix label.markup 22503 70018988
3111
1 chunk-all-sections 1394 36361616
26084
2 d:chapter label.markup 24918 20792834
834
3 href.target 23895 18556477
776
4 footer.navigation 1394 5276429
3785
5 header.navigation 1394 5095535
3655
6 gentext.template 237071 857659
3

vs

number match name mode Calls Tot 100us
Avg

0 gentext.template 247659 742878
2
1 chunk-all-sections 1394 705195
505
2 href.target 35446 663090
18

Hmm. Well, this is all new to me but I'd have expected the numbers in
the "Calls" column to be entirely deterministic. Perhaps that
business about conditional use of UnwrapLinks and other things like it
change the numbers. It's interesting that "gentext.template" is in
the same ballpark on our two systems in terms of calls and CPU time,
but the top templates are massive outliers on my system. I have no
idea what I'm even looking at really but I couldn't help noticing that
templates with match="chapter" and match="appendix" appear in our tree
in sgml/stylesheet-speedup-common.xsl with a comment
"Performance-optimized versions of some upstream templates from
common/ directory". Could it be that whatever performance-enhancing
trick they perform doesn't work on 1.1.32, or alternatively they are
not being reached so we're falling back to non-optimised versions
instead of these?

I wonder, what version of docbook-xsl are you using?
(I have 1.79.1+dfsg-1).
Can you check with 1.79+ (if yours is older)?

docbook-xsl version 1.79.2_1.

--
Thomas Munro
http://www.enterprisedb.com

25.11.2017 11:03, Thomas Munro wrote:

Hmm. Well, this is all new to me but I'd have expected the numbers in
the "Calls" column to be entirely deterministic.

I think, calls are depending on the XSL templates and it seems we have
different templates.
(I couldn't find 'd:appendix' in my docbook-xsl installation, that's why
I asked about version number.)
Maybe it's another case then, your version is new.
Now I see 'd:appendix' appeared in
https://github.com/docbook/xslt10-stylesheets/blob/master/xsl/html/chunktoc.xsl

Perhaps that
business about conditional use of UnwrapLinks and other things like it
change the numbers. It's interesting that "gentext.template" is in
the same ballpark on our two systems in terms of calls and CPU time,
but the top templates are massive outliers on my system. I have no
idea what I'm even looking at really but I couldn't help noticing that
templates with match="chapter" and match="appendix" appear in our tree
in sgml/stylesheet-speedup-common.xsl with a comment
"Performance-optimized versions of some upstream templates from
common/ directory". Could it be that whatever performance-enhancing
trick they perform doesn't work on 1.1.32, or alternatively they are
not being reached so we're falling back to non-optimised versions
instead of these?

I wonder, what version of docbook-xsl are you using?
(I have 1.79.1+dfsg-1).
Can you check with 1.79+ (if yours is older)?

docbook-xsl version 1.79.2_1.

I'll try to install 1.79.2 version and check the performance on my side.

------
Alexander Lakhin
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

25.11.2017 11:21, Alexander Lakhin wrote:

I wonder, what version of docbook-xsl are you using?

(I have 1.79.1+dfsg-1).
Can you check with 1.79+ (if yours is older)?

docbook-xsl version 1.79.2_1.

I'll try to install 1.79.2 version and check the performance on my side.

I installed docbook-style-xsl-1.79.2-5 in Fedora 27 and didn't noticed
performance drop.
In fact in this version I see XSL templates without namespaces
("appendix" instead of "d:appendix").
I looked at the spec of the package and found that it's building from the
"https://github.com/docbook/xslt10-stylesheets/releases/download/release%2F{%version}/docbook-xsl-nons-%{version}.tar.bz2&quot;
nons - is for "no namespace"

Indeed, you can see both variations of the source packages at:
https://github.com/docbook/xslt10-stylesheets/releases/tag/release%2F1.79.2

It seems that your package is built from "ns" version.
(I couldn't find docbook-xsl-nons in Macports.)
If it's the only available version for Mac, it seems we need to adjust
our XSL templates to work with namespaces too.

Best regards,
------
Alexander Lakhin
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Thomas Munro <thomas.munro@enterprisedb.com> writes:

... I couldn't help noticing that
templates with match="chapter" and match="appendix" appear in our tree
in sgml/stylesheet-speedup-common.xsl with a comment
"Performance-optimized versions of some upstream templates from
common/ directory". Could it be that whatever performance-enhancing
trick they perform doesn't work on 1.1.32, or alternatively they are
not being reached so we're falling back to non-optimised versions
instead of these?

If you're suspicious of that, you could try removing those parts of
stylesheet-speedup-common.xsl and see what happens ...

regards, tom lane

On Sun, Nov 26, 2017 at 4:21 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Thomas Munro <thomas.munro@enterprisedb.com> writes:

... I couldn't help noticing that
templates with match="chapter" and match="appendix" appear in our tree
in sgml/stylesheet-speedup-common.xsl with a comment
"Performance-optimized versions of some upstream templates from
common/ directory". Could it be that whatever performance-enhancing
trick they perform doesn't work on 1.1.32, or alternatively they are
not being reached so we're falling back to non-optimised versions
instead of these?

If you're suspicious of that, you could try removing those parts of
stylesheet-speedup-common.xsl and see what happens ...

Good idea. Removing them didn't help (though removing them makes
Apple xsltproc similarly slow). Adding "d:" namespace to "chapter"
and "appendix", declared in the xsl:stylesheet element as
xmlns:d="http://docbook.org/ns/docbook&quot; didn't help either. I suspect
that could be made to work with some more tweaking, but I lack the XSL
knowledge. I found another way forward though:

On Sun, Nov 26, 2017 at 4:09 AM, Alexander Lakhin <exclusion@gmail.com> wrote:

It seems that your package is built from "ns" version.
(I couldn't find docbook-xsl-nons in Macports.)
If it's the only available version for Mac, it seems we need to adjust our
XSL templates to work with namespaces too.

Aha, you're right! MacPorts does actually have two different ports
(packages): docbook-xsl and docbook-xsl-ns, and the first one should
be the no-namespace variant. But I can clearly see that the
docbook-xsl packages installs stylesheets *with* namespaces. I
compared this with a Debian system and found that common/labels.xsl
(the file that defines the templates that our
stylesheet-speedup-common.xsl seems to want to replace) has the "d:"
prefix on "chapter" and "appendix", but doesn't on the Debian system.
Presumably this interferes with the interposing technique. Perhaps
that is a packaging error that should be reported upstream.

That got me wondering... why does the Apple xsltproc in /usr/bin work
then? Where is it even getting docbook-xsl from? I ran it with
--profile and http://docbook.sourceforge.net instead of file:// URLs,
and I could see outgoing connections with netstat. It believe that's
because it doesn't find it locally in /etc/xml/catalog, whereas the
MacPorts xsltproc looks in /opt/locl/etc/xml/catalog where it has been
listed by the docbook-xsl package.

So one solution is simply to uninstall the docbook-xsl package. That
gets me back to fast documentation builds! Incidentally, uninstalling
the docbooks-xsl package also works for FreeBSD which currently ships
a too-old DocBook version. I believed until now that it couldn't
build the PostgreSQL docs, so I'm very happy to discover that it can,
but (1) it needs the network (2) it's using HTTP instead of HTTPS so
Alice could mess with Bob's documentation.

Thanks both for your help figuring this out. That's quite enough XML
for one day.

--
Thomas Munro
http://www.enterprisedb.com

On 11/26/17 17:03, Thomas Munro wrote:

So one solution is simply to uninstall the docbook-xsl package. That
gets me back to fast documentation builds! Incidentally, uninstalling
the docbooks-xsl package also works for FreeBSD which currently ships
a too-old DocBook version.

This is actually documented:
https://www.postgresql.org/docs/devel/static/docguide-toolsets.html

I believed until now that it couldn't
build the PostgreSQL docs, so I'm very happy to discover that it can,
but (1) it needs the network (2) it's using HTTP instead of HTTPS so
Alice could mess with Bob's documentation.

Good point. I have filed a bug about this.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Sun, Nov 26, 2017 at 5:03 PM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:

That got me wondering... why does the Apple xsltproc in /usr/bin work
then? Where is it even getting docbook-xsl from? I ran it with
--profile and http://docbook.sourceforge.net instead of file:// URLs,
and I could see outgoing connections with netstat. It believe that's
because it doesn't find it locally in /etc/xml/catalog, whereas the
MacPorts xsltproc looks in /opt/locl/etc/xml/catalog where it has been
listed by the docbook-xsl package.

So one solution is simply to uninstall the docbook-xsl package. That
gets me back to fast documentation builds!

For me, the documentation build fails without docbook-xsl. I wonder
why it works for you.

It also fails for me if I follow the instructions in the documentation:

If you use MacPorts, the following will get you set up:
<programlisting>
sudo port install docbook-xml-4.2 docbook-xsl fop
</programlisting>

I have to also 'port install libxslt'. Otherwise, /usr/bin/xsltproc
is used, and it won't use files installed by MacPorts packages.

And then it takes, like you reported originally, an insanely long time.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Sat, Mar 3, 2018 at 8:51 AM, Robert Haas <robertmhaas@gmail.com> wrote:

On Sun, Nov 26, 2017 at 5:03 PM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:

So one solution is simply to uninstall the docbook-xsl package. That
gets me back to fast documentation builds!

For me, the documentation build fails without docbook-xsl. I wonder
why it works for you.

Recently I've been unable to build the documentation intermittently,
and I think it's because sourceforge.net has become flakey. Let me
try right now...

$ make docs XSLTPROC=/usr/bin/xsltproc
...blah blah blah...
warning: failed to load external entity
"http://docbook.sourceforge.net/release/xsl/current/xhtml/chunk.xsl&quot;
...blah blah blah...

So, yeah, it looks like we might need a local docbooks installation currently.

It also fails for me if I follow the instructions in the documentation:

If you use MacPorts, the following will get you set up:
<programlisting>
sudo port install docbook-xml-4.2 docbook-xsl fop
</programlisting>

I have to also 'port install libxslt'. Otherwise, /usr/bin/xsltproc
is used, and it won't use files installed by MacPorts packages.

And then it takes, like you reported originally, an insanely long time.

I think we should complain to the MacPorts packager about the
namespace vs non-namespace stuff being possibly confused in the
packages.

--
Thomas Munro
http://www.enterprisedb.com

On Sat, Mar 3, 2018 at 9:34 AM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:

I think we should complain to the MacPorts packager about the
namespace vs non-namespace stuff being possibly confused in the
packages.

https://trac.macports.org/ticket/55946

--
Thomas Munro
http://www.enterprisedb.com

On Sat, Mar 03, 2018 at 09:34:58AM +1300, Thomas Munro wrote:

Recently I've been unable to build the documentation intermittently,
and I think it's because sourceforge.net has become flakey. Let me
try right now...

$ make docs XSLTPROC=/usr/bin/xsltproc
...blah blah blah...
warning: failed to load external entity
"http://docbook.sourceforge.net/release/xsl/current/xhtml/chunk.xsl&quot;
...blah blah blah...

I am seeing that as well lately on my Linux box as well with Debian.
And the build failure consists in a mountain of warnings and errors
where you need to dig up to the top to see the real problem. That's
annoying :(
--
Michael

On Fri, Mar 2, 2018 at 3:34 PM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:

I think we should complain to the MacPorts packager about the
namespace vs non-namespace stuff being possibly confused in the
packages.

Any ideas about a workaround for the meantime? Having to wait half an
hour for the documentation to build is pretty annoying.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Fri, Mar 23, 2018 at 3:27 AM, Robert Haas <robertmhaas@gmail.com> wrote:

On Fri, Mar 2, 2018 at 3:34 PM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:

I think we should complain to the MacPorts packager about the
namespace vs non-namespace stuff being possibly confused in the
packages.

Any ideas about a workaround for the meantime? Having to wait half an
hour for the documentation to build is pretty annoying.

This worked for me:

1. Steal a working installation of docbook-xsl from some other
system. Or clone the repo from github.com/docbook and figure out how
to build it so that there is an xhtml directory (that was beyond my
attention span). I just did this on a Debian system:

tar czvf docbook-xsl.tgz -C /usr/share/xml/docbook/stylesheet/ docbook-xsl

I could give you that file off-list if you want, but it's 2.2MB so I
won't post it here. You could also grab the Debian (or other) package
file and unpack that.

I created a directory ~/docbook-stuff/ and then unpacked that tarball there.

2. Tell xsltproc to rewrite any references to sourceforge.net to use
this local stuff instead. Either you can create/edit the system-wide
/etc/xml/catalog file (or equivalent under /opt if you're using
MacPorts tools), or you can create a new file somewhere and point to
it with the environment variable XML_CATALOG_FILES.

I went with the env variable approach because I didn't want to mess
with system-wide configuration. So I created a file
~/docbook-stuff/catalog.xml with this in it:

<?xml version="1.0" encoding="utf-8"?>
<catalog xmlns="urn:oasis:names:tc:entity:xmlns:xml:catalog">
<!-- redirect references to docbook.sourceforge.net to local files -->
<rewriteURI
uriStartString="http://docbook.sourceforge.net/release/xsl/current/&quot;
rewritePrefix="file:///Users/munro/docbook-stuff/docbook-xsl/"/>
<rewriteSystem
systemIdStartString="http://docbook.sourceforge.net/release/xsl/current/&quot;
rewritePrefix="file:///Users/munro/docbook-stuff/docbook-xsl/"/>
</catalog>

Obviously that needs to be adjusted to point to wherever you unpacked
that tarball.

I did wonder about simply changing our documentation's sourceforge.net
references to point to docbook.org instead, apparently the new home
(?) of this stuff:

-<xsl:import href="http://docbook.sourceforge.net/release/xsl/current/xhtml/chunk.xsl"/>
+<xsl:import href="https://cdn.docbook.org/release/xsl-nons/current/xhtml/chunk.xsl"/>

(plus similar changes elsewhere).

But it didn't seem to be able to fetch stylesheets that way. Perhaps
my tools don't like speaking HTTPS. I didn't try to dig any further
because I'd rather go and hack on C code today.

--
Thomas Munro
http://www.enterprisedb.com

On 3/22/18 18:56, Thomas Munro wrote:

I did wonder about simply changing our documentation's sourceforge.net
references to point to docbook.org instead, apparently the new home
(?) of this stuff:

-<xsl:import href="http://docbook.sourceforge.net/release/xsl/current/xhtml/chunk.xsl"/>
+<xsl:import href="https://cdn.docbook.org/release/xsl-nons/current/xhtml/chunk.xsl"/>

That would affect those who are currently relying on catalog entries
redirecting the old URLs to their local copy.

I think we will at least need to wait for a new release with new
catalogs before doing this.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Sat, Mar 3, 2018 at 3:53 PM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:

On Sat, Mar 3, 2018 at 9:34 AM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:

I think we should complain to the MacPorts packager about the
namespace vs non-namespace stuff being possibly confused in the
packages.

https://trac.macports.org/ticket/55946

This was fixed yesterday. After "sudo port update" and "sudo port
upgrade outdated" I see "docbook-xsl @1.79.2_3" being installed and
then I get:

$ time make docs
...
real 1m47.777s
user 1m43.627s
sys 0m2.958s

--
Thomas Munro
http://www.enterprisedb.com