meson documentation build open issues

Hi,

On 2023-03-15 08:14:09 +0100, Peter Eisentraut wrote:

I have identified several open issues with the documentation build under
Meson (approximately in priority order):

1. Image files are not handled at all, so they don't show up in the final
product.

Hm. Somehow I thought I'd tackled that at some point. Ah. I got there for the
PDF output, but didn't realize it's also an issue for the html output.

For FO it sufficed to set the img.src.path param. For HTML that's not enough,
because that just adjusts the link to the file - but we don't want to link to
the source file. We actually solved this for the single-page html version - we
just embed the svg. I wonder if we should just do that as well.

Another way would be to emit the files into the desired place as part of the
stylesheet. While it requires touching xslt, it does seems somewhat more
elegant than just copying files around. I did implement that, curious what you
think.

2. Defaults to website stylesheet, no way to configure. This should be
adjusted to match the make build.

Should we add a meson option?

3. The various build targets and their combinations are mismatching and
incomplete. For example:

Top-level GNUmakefile has these targets:

- docs (builds html and man)
- html
- man

(Those are the formats that are part of a distribution build.)

doc/src/sgml/Makefile has these documented targets:

- default target is html
- all (builds html and man, maps to top-level "docs")
- html
- man
- postgres-A4.pdf
- postgres-US.pdf
- check

as well as (undocumented):

- htmlhelp
- postgres.html
- postgres.txt
- epub
- postgres.epub
- postgres.info

meson has the following documented targets:

- docs (builds only html)
- alldocs (builds all formats, including obscure ones)

as well as the following undocumented targets:

- html
- man
- html_help [sic]

renamed in the attached patch.

- postgres-A4.pdf
- postgres-US.pdf
- postgres.epub

Note that these are actually named doc/src/sgml/{html,man,...}, not top-level
targets.

- [info is not implemented at all]

Would be easy to implement, but not sure it's worth doing.

- [didn't find an equivalent of check]

That's probably worth doing - should it be run as an actual test, or be a
target?

4. There doesn't appear to be a way to install the documentation.
(There are also some open questions in the top-level meson.build about
the installation directories, but I suppose if we can't install them
then exactly where to install them hasn't been thought about too
much.)

WIP patch for that attached. There's now
install-doc-man
install-doc-html
run targets and a
install-docs
alias target.

I did end up getting stuck when hacking on this, and ended up adding css
support for nochunk and support for the website style for htmlhelp and
nochunk, as well as obsoleting the need for copying the css files... But
perhaps that's a bit too much.

Greetings,

Andres Freund

Hi,

On 2023-03-15 20:55:33 -0700, Andres Freund wrote:

WIP patch for that attached. There's now
install-doc-man
install-doc-html
run targets and a
install-docs
alias target.

I did end up getting stuck when hacking on this, and ended up adding css
support for nochunk and support for the website style for htmlhelp and
nochunk, as well as obsoleting the need for copying the css files... But
perhaps that's a bit too much.

Updated set of patches attached. This one works in older meson versions too
and adds install-world and install-quiet targets.

I also ended up getting so frustrated at the docs build speed that I started
to hack a bit on that. I attached a patch shaving a few seconds off the
buildtime.

I think we can make the docs build in parallel and incrementally, by building
the different parts of the docs in parallel, using --stringparam rootid,
e.g. building each 'part' separately.

A very very rough draft attached:

parallel with parts:
real 0m10.831s
user 0m58.295s
sys 0m1.402s

normal:
real 0m32.215s
user 0m31.876s
sys 0m0.328s

1/3 of the build time at 2x the cost is nothing to sneeze at.

Greetings,

Andres Freund

On 20.03.23 03:33, Andres Freund wrote:

I did end up getting stuck when hacking on this, and ended up adding css
support for nochunk and support for the website style for htmlhelp and
nochunk, as well as obsoleting the need for copying the css files... But
perhaps that's a bit too much.

Updated set of patches attached. This one works in older meson versions too
and adds install-world and install-quiet targets.

Oh, this patch set grew quite quickly. ;-)

[PATCH v2 1/8] meson: rename html_help target to htmlhelp

This is obvious.

[PATCH v2 5/8] docs: html: copy images to output as part of xslt build

Making the XSLT stylesheets do the copying has some appeal. I think it
would only work for SVG (or other XML) files, which I guess is okay, but
maybe the templates should have a filter on format="SVG" or something.
Also, this copying actually modifies the files in some XML-equivalent
way. Also okay, I think, but worth noting.

Note sure why you removed this comment

-<!-- strip directory name from image filerefs -->

since the code still exists.

[PATCH v2 6/8] wip: docs: copy or inline css

This seems pretty complicated compared to just copying a file?

Hi,

On 2023-03-20 11:58:08 +0100, Peter Eisentraut wrote:

Oh, this patch set grew quite quickly. ;-)

Yep :)

[PATCH v2 5/8] docs: html: copy images to output as part of xslt build

Making the XSLT stylesheets do the copying has some appeal. I think it
would only work for SVG (or other XML) files, which I guess is okay, but
maybe the templates should have a filter on format="SVG" or something. Also,
this copying actually modifies the files in some XML-equivalent way. Also
okay, I think, but worth noting.

I think it can be made work for non-xml files with xinclude too. But the
restriction around only working in top-level stylesheets (vs everywhere for
documents) is quite annoying.

[PATCH v2 6/8] wip: docs: copy or inline css

This seems pretty complicated compared to just copying a file?

Mainly that it works correctly for the standalone file.

Greetings,

Andres Freund

Hi,

On 2023-03-19 19:33:38 -0700, Andres Freund wrote:

I think we can make the docs build in parallel and incrementally, by building
the different parts of the docs in parallel, using --stringparam rootid,
e.g. building each 'part' separately.

A very very rough draft attached:

parallel with parts:
real 0m10.831s
user 0m58.295s
sys 0m1.402s

normal:
real 0m32.215s
user 0m31.876s
sys 0m0.328s

1/3 of the build time at 2x the cost is nothing to sneeze at.

I could not make myself stop trying to figure out where the big constant time
factor comes from. Every invocation costs about 2s, even if not much is
rendered. Turns out, that's solely spent building all the <xsl:key>s. The
first time *any* key() is invoked for a document, all the keys are computed in
a single pass over the document.

A single reasonable key doesn't take that much time, even for the size of our
docs. But there are several redundant keys being built. Some of them somewhat
expensive. E.g. each
<xsl:key name="genid" match="*" use="generate-id()"/>
takes about 300ms. There's one in chunk-common and one in
docbook-no-doctype.xsl.

I'm going to cry now.

Greetings,

Andres Freund

Hi,

On 2023-03-20 10:32:49 -0700, Andres Freund wrote:

On 2023-03-20 11:58:08 +0100, Peter Eisentraut wrote:

Oh, this patch set grew quite quickly. ;-)

Yep :)

Unless somebody sees a reason to wait, I am planning to commit:
meson: add install-{quiet, world} targets
meson: add install-{docs,doc-html,doc-man} targets
meson: make install_test_files more generic, rename to install_files

While I don't think we have necessarily the path forward around .css and .svg,
the above are independent of that.

For the .svg: I wonder if we should just inline the images in the chunked
html, just like we do in the single page one. It's not like we reuse one image
across a lot of pages, so there's no bandwidth saved from having the images
separate...

For the .css: docbook-xsl actually has support for writing the .css: [1]https://docbook.sourceforge.net/release/xsl/current/doc/html/custom.css.source.html - but
it requires the .css file be valid xml. I wonder if the cleanest approch would
be to have a build step to create .css.xml - then the non-chunked build's
generate.css.header would do the right thing.

I'll start a new thread for
docs: speed up docs build by special-casing the gentext.template
VERY WIP: parallel doc generation
after the feature freeze.

After looking into it a tiny bit more, it seems we should use neither pandoc
nor dbtoepub for epub generation.

All the dbtoepub does is to invoke the docbook-xsl support for epubs and zip
the result - except it doesn't use our stylesheets, so it looks randomly
different and doesn't use our speedups. At the very least we should use our
customizations, if we want epub support. Or we should just remove it.

Pandoc unfortunately doesn't do docbook well enough to be usable for now to
directly parse our docbook.

Regards,

Andres

[1]: https://docbook.sourceforge.net/release/xsl/current/doc/html/custom.css.source.html

Hi,

On 2023-03-22 11:59:17 -0700, Andres Freund wrote:

Unless somebody sees a reason to wait, I am planning to commit:
meson: add install-{quiet, world} targets
meson: add install-{docs,doc-html,doc-man} targets
meson: make install_test_files more generic, rename to install_files

I've done that now.

For the .css: docbook-xsl actually has support for writing the .css: [1] - but
it requires the .css file be valid xml. I wonder if the cleanest approch would
be to have a build step to create .css.xml - then the non-chunked build's
generate.css.header would do the right thing.

We don't even need to do that! The attached patch just creates a wrapper
css.xml that loads the .css via an entity reference.

I think this looks reasonably complicated, given that it gives us a working
stylesheet for the non-chunked output?

I don't know if my hack of putting the paramters in stylesheet-common.xsl is
reasonable. Perhaps we should just include stylesheet-html-common.xsl in
stylesheet-hh.xsl, then this uglyness wouldn't be required.

Greetings,

Andres Freund

On 24.03.23 08:26, Andres Freund wrote:

For the .css: docbook-xsl actually has support for writing the .css: [1] - but
it requires the .css file be valid xml. I wonder if the cleanest approch would
be to have a build step to create .css.xml - then the non-chunked build's
generate.css.header would do the right thing.

We don't even need to do that! The attached patch just creates a wrapper
css.xml that loads the .css via an entity reference.

That looks like a better solution.

I don't know if my hack of putting the paramters in stylesheet-common.xsl is
reasonable. Perhaps we should just include stylesheet-html-common.xsl in
stylesheet-hh.xsl, then this uglyness wouldn't be required.

Maybe, but it's not clear whether all the customizations in there are
applicable to htmlhelp.

Another option here is to remove support for htmlhelp.

Hi,

On 2023-03-24 11:59:23 +0100, Peter Eisentraut wrote:

Another option here is to remove support for htmlhelp.

That might actually be the best path - it certainly doesn't look like anybody
has been actively using it. Or otherwise somebody would have complained about
there not being any instructions on how to actually compile a .chm file. And
perhaps complained that it takes next to forever to build.

I also have the impression that people don't use the .chm stuff much anymore,
but that might just be me not using windows.

Greetings,

Andres

On 24 Mar 2023, at 17:58, Andres Freund <andres@anarazel.de> wrote:
On 2023-03-24 11:59:23 +0100, Peter Eisentraut wrote:

Another option here is to remove support for htmlhelp.

That might actually be the best path - it certainly doesn't look like anybody
has been actively using it.

I had no idea we had support for building a .chm until reading this, but I've
also never come across anyone asking for such a docset. FWIW, no objections to
it going.

--
Daniel Gustafsson

On 24.03.23 17:58, Andres Freund wrote:

On 2023-03-24 11:59:23 +0100, Peter Eisentraut wrote:

Another option here is to remove support for htmlhelp.

That might actually be the best path - it certainly doesn't look like anybody
has been actively using it. Or otherwise somebody would have complained about
there not being any instructions on how to actually compile a .chm file. And
perhaps complained that it takes next to forever to build.

I also have the impression that people don't use the .chm stuff much anymore,
but that might just be me not using windows.

I think in ancient times, pgadmin used it for its internal help.

But I have heard less about htmlhelp over the years than about the info
format.

On Tue, 28 Mar 2023 at 10:46, Peter Eisentraut <
peter.eisentraut@enterprisedb.com> wrote:

On 24.03.23 17:58, Andres Freund wrote:

On 2023-03-24 11:59:23 +0100, Peter Eisentraut wrote:

Another option here is to remove support for htmlhelp.

That might actually be the best path - it certainly doesn't look like

anybody

has been actively using it. Or otherwise somebody would have complained

about

there not being any instructions on how to actually compile a .chm file.

And

perhaps complained that it takes next to forever to build.

I also have the impression that people don't use the .chm stuff much

anymore,

but that might just be me not using windows.

I think in ancient times, pgadmin used it for its internal help.

Yes, very ancient :-). We use Sphinx now.

But I have heard less about htmlhelp over the years than about the info
format.

--
Dave Page
Blog: https://pgsnake.blogspot.com
Twitter: @pgsnake

EDB: https://www.enterprisedb.com

On 15.03.23 08:14, Peter Eisentraut wrote:

I have identified several open issues with the documentation build under
Meson (approximately in priority order):

Some work has been done on this. Here is my current assessment.

1. Image files are not handled at all, so they don't show up in the
final product.

This is fixed.

2. Defaults to website stylesheet, no way to configure.  This should be
adjusted to match the make build.

This is fixed.

3. The various build targets and their combinations are mismatching and
incomplete.

This has been improved, and I see there is documentation.

I think it's still an issue that "make docs" builds html and man but
"ninja docs" only builds html. For some reason the wiki page actually
claims that ninja docs builds both, but this does not happen for me.

4. There doesn't appear to be a way to install the documentation.

This has been addressed.

5. There doesn't appear to be an equivalent of "make world" and "make
install-world" that includes documentation builds.

This has been addressed with the additional meson auto options. But it
seems that these options only control building, not installing, so there
is no "install-world" aspect yet.

Hi,

On 2023-04-05 12:24:04 +0200, Peter Eisentraut wrote:

On 15.03.23 08:14, Peter Eisentraut wrote:

3. The various build targets and their combinations are mismatching and
incomplete.

This has been improved, and I see there is documentation.

I think it's still an issue that "make docs" builds html and man but "ninja
docs" only builds html. For some reason the wiki page actually claims that
ninja docs builds both, but this does not happen for me.

It used to, but Tom insisted that it should not. I'm afraid that it's not
quite possible to emulate make here. 'make docs' at the toplevel builds both
HTML and manpages. But 'make -C doc/src/sgml', only builds HTML.

5. There doesn't appear to be an equivalent of "make world" and "make
install-world" that includes documentation builds.

This has been addressed with the additional meson auto options. But it
seems that these options only control building, not installing, so there is
no "install-world" aspect yet.

I'm not following - install-world install docs if the docs feature is
available, and not if not?

Greetings,

Andres Freund

On 05.04.23 16:45, Andres Freund wrote:

I think it's still an issue that "make docs" builds html and man but "ninja
docs" only builds html. For some reason the wiki page actually claims that
ninja docs builds both, but this does not happen for me.

It used to, but Tom insisted that it should not. I'm afraid that it's not
quite possible to emulate make here. 'make docs' at the toplevel builds both
HTML and manpages. But 'make -C doc/src/sgml', only builds HTML.

Ok, not a topic for this thread then.

5. There doesn't appear to be an equivalent of "make world" and "make
install-world" that includes documentation builds.

This has been addressed with the additional meson auto options. But it
seems that these options only control building, not installing, so there is
no "install-world" aspect yet.

I'm not following - install-world install docs if the docs feature is
available, and not if not?

I had expected that if meson setup enables the 'docs' feature, then
meson compile will build the documentation, which happens, and meson
install will install it, which does not happen.

On 2023-04-06 Th 05:11, Peter Eisentraut wrote:

On 05.04.23 16:45, Andres Freund wrote:

I think it's still an issue that "make docs" builds html and man but
"ninja
docs" only builds html.  For some reason the wiki page actually
claims that
ninja docs builds both, but this does not happen for me.

It used to, but Tom insisted that it should not. I'm afraid that it's
not
quite possible to emulate make here. 'make docs' at the toplevel
builds both
HTML and manpages. But 'make -C doc/src/sgml', only builds HTML.

Ok, not a topic for this thread then.

5. There doesn't appear to be an equivalent of "make world" and "make
install-world" that includes documentation builds.

This has been addressed with the additional meson auto options.  But it
seems that these options only control building, not installing, so
there is
no "install-world" aspect yet.

I'm not following - install-world install docs if the docs feature is
available, and not if not?

I had expected that if meson setup enables the 'docs' feature, then
meson compile will build the documentation, which happens, and meson
install will install it, which does not happen.

"meson compile" doesn't seem to build the docs by default ( see
<https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=crake&amp;dt=2023-04-06%2018%3A17%3A04&amp;stg=build&gt;),
and I'd rather it didn't, building the docs is a separate and optional
step for the buildfarm.

cheers

andrew

--
Andrew Dunstan
EDB:https://www.enterprisedb.com

On 07.04.23 16:39, Andrew Dunstan wrote:

5. There doesn't appear to be an equivalent of "make world" and "make
install-world" that includes documentation builds.

This has been addressed with the additional meson auto options.  But it
seems that these options only control building, not installing, so
there is
no "install-world" aspect yet.

I'm not following - install-world install docs if the docs feature is
available, and not if not?

I had expected that if meson setup enables the 'docs' feature, then
meson compile will build the documentation, which happens, and meson
install will install it, which does not happen.

"meson compile" doesn't seem to build the docs by default ( see
<https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=crake&amp;dt=2023-04-06%2018%3A17%3A04&amp;stg=build&gt;), and I'd rather it didn't, building the docs is a separate and optional step for the buildfarm.

You can control this with the "docs" option for meson, as of recently.

Re: Peter Eisentraut

"meson compile" doesn't seem to build the docs by default ( see <https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=crake&amp;dt=2023-04-06%2018%3A17%3A04&amp;stg=build&gt;),
and I'd rather it didn't, building the docs is a separate and optional
step for the buildfarm.

You can control this with the "docs" option for meson, as of recently.

I've been looking into switching the Debian PG 17 build to meson, but
I'm running into several problems.

* The docs are still not built by default, and -Ddocs=enabled doesn't
change that

* None of the "build docs" targets are documented in install-meson.html

* "ninja -C build alldocs" works, but it's impossible to see what
flavors it's actually building. Everything is autodetected, and
perhaps I would like to no build the .txt/something variants,
but I have no idea what switch that is, or what package I have to
uninstall so it's not autodetected (only html and pdf are
documented.)

Are there any other targets for the individual formats? (I could
probably use one for the manpages only, without the html.)

Non-doc issues:

* LLVM is off by default (ok), when I enable it with -Dllvm=auto, it
gets detected, but no .bc files are built, nor installed

* selinux is not autodetected. It needs -Dselinux=auto, but that's not
documented in install-meson.html

* There is no split between libdir and pkglibdir. We had used that in
the past for libpq -> /usr/lib/x86_64-linux-gnu and PG stuff ->
/usr/lib/postgresql/17/lib.

Christoph

Hi,

On 2023-11-03 15:26:05 +0100, Christoph Berg wrote:

Re: Peter Eisentraut

"meson compile" doesn't seem to build the docs by default ( see <https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=crake&amp;dt=2023-04-06%2018%3A17%3A04&amp;stg=build&gt;),
and I'd rather it didn't, building the docs is a separate and optional
step for the buildfarm.

You can control this with the "docs" option for meson, as of recently.

I've been looking into switching the Debian PG 17 build to meson, but
I'm running into several problems.

* The docs are still not built by default, and -Ddocs=enabled doesn't
change that

Maybe I am missing something - they aren't built by default in autoconf
either?

* None of the "build docs" targets are documented in install-meson.html

Hm, odd, I thought they were, but you are right. There were some docs patches
that we never really could find agreement upon :/

* "ninja -C build alldocs" works, but it's impossible to see what
flavors it's actually building. Everything is autodetected, and
perhaps I would like to no build the .txt/something variants,
but I have no idea what switch that is, or what package I have to
uninstall so it's not autodetected (only html and pdf are
documented.)

I think a package build should probably turn off auto-detection (
meson setup --auto-features=disabled) and enable specific features that are
desired - in which case you get errors if they are not available. Which
presumably is the behaviour you'd like?

Are there any other targets for the individual formats? (I could
probably use one for the manpages only, without the html.)

Yes, there are.
ninja doc/src/sgml/{postgres-A4.pdf,html,postgres.html,man1}

Perhaps more interesting for your purposes, there are the
install-doc-{html,man} targets.

I remember discussing adding doc-{html,man} targets alongside
install-doc-{html,man}, not sure why we ended up not doing that. I'd be in
favor of adding them.

I've also been wondering about a 'help' target that documents important
targets in a interactively usable way.

Non-doc issues:

* LLVM is off by default (ok), when I enable it with -Dllvm=auto, it
gets detected, but no .bc files are built, nor installed

Support for that has not yet been merged.

* selinux is not autodetected. It needs -Dselinux=auto, but that's not
documented in install-meson.html

Uh, huh. There's no documentation for --with-selinux in the installation.sgml
either, just in sepgsql.sgml. So when the relevant docs got translated to
meson, -Dselinux= wasn't documented either.

* There is no split between libdir and pkglibdir. We had used that in
the past for libpq -> /usr/lib/x86_64-linux-gnu and PG stuff ->
/usr/lib/postgresql/17/lib.

I don't think the autoconf build currently exposes separately configuring
pkglibdir either, I think that's a debian patch? I'm entirely open to adding
an explicit configuration option for this though.

Thanks for looking at this, it's quite helpful!

Greetings,

Andres Freund