documentation is now XML

On Thu, Nov 23, 2017 at 6:01 PM, Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:

The documentation sources are now DocBook XML, not SGML. (The files are
still named *.sgml. That's something to think about separately.)

Congratulations to you and Alexander ! That is what I waited for a long time.
Now we could think about diagrams :)

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:

The documentation sources are now DocBook XML, not SGML. (The files are
still named *.sgml. That's something to think about separately.)

I think we should have a discussion about whether it'd be smart
to convert the back branches' documentation to XML as well.

The main reason that I want to consider this is that back-patching
documentation fixes is going to be a huge problem if we don't.

Using the same doc-building toolchain across all branches seems like a win
as well. You could argue that switching build requirements in a minor
release is unfriendly to packagers; but those who build their own docs
have already had to adapt to the xsltproc/fop toolchain for v10, so
standardizing on that for 9.3-9.6 as well doesn't seem like it complicates
their lives. (Possibly we should canvass opinions on pgsql-packagers to
be sure of that.)

Also, we're way overdue for getting out from under the creaky TeX-based
toolchain for producing PDFs. Every time we make releases, I worry
whether we're going to get blindsided by its bugs with hotlinks that get
split across pages, since page breaks tend to vary in position depending
on exactly whose version of the toolchain and style sheets you build with.
I've also been living in fear of the day we hit some hardwired TeX limit
that we can't increase or work around. We've had to hack around such
limits repeatedly in the past (eg commit 944b41fc0). Now, it's probably
unlikely that growth of the release notes would be enough to put us over
the top in any back branch --- but if it did happen, we'd find out about
it at a point in the release cycle where there's very little margin for
error.

As against all that, there's our traditional conservatism about making
not-strictly-necessary changes in the back branches. Which is a strong
factor, for sure, but I think maybe here's a case for overriding it.

regards, tom lane

On Fri, Nov 24, 2017 at 12:21 AM, Oleg Bartunov <obartunov@gmail.com> wrote:

On Thu, Nov 23, 2017 at 6:01 PM, Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:

The documentation sources are now DocBook XML, not SGML. (The files are
still named *.sgml. That's something to think about separately.)

Congratulations to you and Alexander !

+1.
-- 
Michael

On Fri, Nov 24, 2017 at 5:39 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:

The documentation sources are now DocBook XML, not SGML. (The files are
still named *.sgml. That's something to think about separately.)

I think we should have a discussion about whether it'd be smart
to convert the back branches' documentation to XML as well.

The main reason that I want to consider this is that back-patching
documentation fixes is going to be a huge problem if we don't.

Things like c29c578 and 1ff01b3 only found their way on HEAD. There is
a bit of work needed here for back-branches. At the end I would vote
for having consistent documentation on all active branches.

Using the same doc-building toolchain across all branches seems like a win
as well. You could argue that switching build requirements in a minor
release is unfriendly to packagers; but those who build their own docs
have already had to adapt to the xsltproc/fop toolchain for v10, so
standardizing on that for 9.3-9.6 as well doesn't seem like it complicates
their lives. (Possibly we should canvass opinions on pgsql-packagers to
be sure of that.)

My own packaging is going to need some tweaks as well, but there is
nothing difficult. A discussion is definitely deserved on -packagers,
all don't have the same toolchain set.
--
Michael

On 11/23/17 15:39, Tom Lane wrote:

I think we should have a discussion about whether it'd be smart
to convert the back branches' documentation to XML as well.

My short answer to that is, I don't have time for that. I don't know if
anyone else wants to investigate it. But it took us years to get to
this point, and backpatching and back-testing all of that is just a lot
of work that was not planned. Also, we will want to keep moving
forward. As the title of the thread on -docs shows, some people want to
move to DocBook 5. If every move like that will require backpatching
everything, nobody is going to want to sign up for it anymore.

The main reason that I want to consider this is that back-patching
documentation fixes is going to be a huge problem if we don't.

I understand that. I would like to think about a way, maybe a git hook
or wrapper or something, to make that simpler. I haven't found any
promising approach so far, however.

Using the same doc-building toolchain across all branches seems like a win
as well. You could argue that switching build requirements in a minor
release is unfriendly to packagers; but those who build their own docs
have already had to adapt to the xsltproc/fop toolchain for v10, so
standardizing on that for 9.3-9.6 as well doesn't seem like it complicates
their lives. (Possibly we should canvass opinions on pgsql-packagers to
be sure of that.)

xsltproc has been required since 9.0 for the man pages, so that wouldn't
be a problem.

Also, we're way overdue for getting out from under the creaky TeX-based
toolchain for producing PDFs.

Makefile rules to build via FOP have been available since 9.4, so there
is a backup plan there. However, there has also been a fair amount of
tweaking to make things look good before switching in 10, so that would
all have to be collected and analyzed.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:

On 11/23/17 15:39, Tom Lane wrote:

I think we should have a discussion about whether it'd be smart
to convert the back branches' documentation to XML as well.

My short answer to that is, I don't have time for that. I don't know if
anyone else wants to investigate it. But it took us years to get to
this point, and backpatching and back-testing all of that is just a lot
of work that was not planned.

I thought that might be your answer :-(. I can't argue with it ---
if it's not a simple thing to back-patch, then it's unclear whether
the net annoyance over the next five years would be enough to justify
doing the work.

The main reason that I want to consider this is that back-patching
documentation fixes is going to be a huge problem if we don't.

I understand that. I would like to think about a way, maybe a git hook
or wrapper or something, to make that simpler. I haven't found any
promising approach so far, however.

One thing we'd definitely better do is enable some buildfarm coverage.
AFAIK, the only buildfarm animal that's building the docs is guaibasaurus,
and it only seems to be doing that on HEAD. Since this has considerably
increased the risks of back-patching creating busted docs in the back
branches, we'd better spin up some back-branch testing as well.

regards, tom lane

On 11/28/2017 12:06 PM, Tom Lane wrote:

One thing we'd definitely better do is enable some buildfarm coverage.
AFAIK, the only buildfarm animal that's building the docs is guaibasaurus,
and it only seems to be doing that on HEAD. Since this has considerably
increased the risks of back-patching creating busted docs in the back
branches, we'd better spin up some back-branch testing as well.

crake builds the docs on all branches pretty regularly. I have changed
its config so it builds them pretty much every build.

Of course, it's only building the HTML docs. Do we need more than that?
Continuously making PDFs seems a bit excessive.

cheers

andrew

--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:

On 11/28/2017 12:06 PM, Tom Lane wrote:

One thing we'd definitely better do is enable some buildfarm coverage.
AFAIK, the only buildfarm animal that's building the docs is guaibasaurus,
and it only seems to be doing that on HEAD. Since this has considerably
increased the risks of back-patching creating busted docs in the back
branches, we'd better spin up some back-branch testing as well.

crake builds the docs on all branches pretty regularly. I have changed
its config so it builds them pretty much every build.

Oh, excellent, thanks.

Of course, it's only building the HTML docs. Do we need more than that?
Continuously making PDFs seems a bit excessive.

I think sanity-checking the SGML/XML is the only part we need to do
regularly, so building HTML seems like plenty. The only point of
doing test PDF builds is to check for the link-crosses-a-page issue,
and I do not think it's worth worrying about that except very near
to a release. A problem we fix today might be gone tomorrow anyway.

regards, tom lane

Hello,

28.11.2017 20:06, Tom Lane wrote:

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:

On 11/23/17 15:39, Tom Lane wrote:

I think we should have a discussion about whether it'd be smart
to convert the back branches' documentation to XML as well.

My short answer to that is, I don't have time for that. I don't know if
anyone else wants to investigate it. But it took us years to get to
this point, and backpatching and back-testing all of that is just a lot
of work that was not planned.

I thought that might be your answer :-(. I can't argue with it ---
if it's not a simple thing to back-patch, then it's unclear whether
the net annoyance over the next five years would be enough to justify
doing the work.

I can prepare such patches (scripts to generate them). In fact we
(Postgres Pro) perform such conversion (SGML->XML) on-fly when building
docs starting from 9.6. So it's not problem to convert *.sgml and
replace Makefile and *.xsl.
But I would prefer to perform the conversion when we finish the move on
11devel (renaming sgml to xml, maybe optimizing xsl's...).

Best regards,
------
Alexander Lakhin
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

On Thu, Nov 23, 2017 at 03:39:24PM -0500, Tom Lane wrote:

Also, we're way overdue for getting out from under the creaky TeX-based
toolchain for producing PDFs. Every time we make releases, I worry
whether we're going to get blindsided by its bugs with hotlinks that get
split across pages, since page breaks tend to vary in position depending
on exactly whose version of the toolchain and style sheets you build with.
I've also been living in fear of the day we hit some hardwired TeX limit
that we can't increase or work around. We've had to hack around such
limits repeatedly in the past (eg commit 944b41fc0). Now, it's probably
unlikely that growth of the release notes would be enough to put us over
the top in any back branch --- but if it did happen, we'd find out about
it at a point in the release cycle where there's very little margin for
error.

I am coming in late here, but I am not aware of any open source
professional typesetting software that has output quality as good as
TeX.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On 01/23/2018 04:59 PM, Bruce Momjian wrote:

On Thu, Nov 23, 2017 at 03:39:24PM -0500, Tom Lane wrote:

Also, we're way overdue for getting out from under the creaky TeX-based
toolchain for producing PDFs. Every time we make releases, I worry
whether we're going to get blindsided by its bugs with hotlinks that get
split across pages, since page breaks tend to vary in position depending

I am coming in late here, but I am not aware of any open source
professional typesetting software that has output quality as good as
TeX.

It has been ages since I have dealt with this but as I recall but moving
to XML we can use XSLT to push to PDF. The resulting product is as good
(if not better in some ways) to old fashion SGML->TEX->PDF.
Here are a couple of articles:

https://www.codeproject.com/Articles/595650/Creating-PDF-documents-from-XML
https://www.ibm.com/developerworks/library/x-xmlxsltpdf/index.html

JD

--
Command Prompt, Inc. || http://the.postgres.company/ || @cmdpromptinc

PostgreSQL centered full stack support, consulting and development.
Advocate: @amplifypostgres || Learn: https://postgresconf.org
***** Unless otherwise stated, opinions are my own. *****

On Tue, Jan 23, 2018 at 05:25:30PM -0800, Joshua Drake wrote:

On 01/23/2018 04:59 PM, Bruce Momjian wrote:

On Thu, Nov 23, 2017 at 03:39:24PM -0500, Tom Lane wrote:

Also, we're way overdue for getting out from under the creaky TeX-based
toolchain for producing PDFs. Every time we make releases, I worry
whether we're going to get blindsided by its bugs with hotlinks that get
split across pages, since page breaks tend to vary in position depending

I am coming in late here, but I am not aware of any open source
professional typesetting software that has output quality as good as
TeX.

It has been ages since I have dealt with this but as I recall but moving to
XML we can use XSLT to push to PDF. The resulting product is as good (if not
better in some ways) to old fashion SGML->TEX->PDF.
Here are a couple of articles:

https://www.codeproject.com/Articles/595650/Creating-PDF-documents-from-XML
https://www.ibm.com/developerworks/library/x-xmlxsltpdf/index.html

Uh, the first article says it uses FO.NET, and on their webpage they
say:

https://fonet.codeplex.com/
XSL-FO to PDF renderer written in managed C# code for the .NET framework

FO.NET has been written for the Microsoft .NET Framework and is suitable
for use from any .NET compatible language such as C#, VB.NET or C#.

That sounds like a boatload of dependencies.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

Bruce Momjian <bruce@momjian.us> writes:

On Thu, Nov 23, 2017 at 03:39:24PM -0500, Tom Lane wrote:

Also, we're way overdue for getting out from under the creaky TeX-based
toolchain for producing PDFs.

I am coming in late here, but I am not aware of any open source
professional typesetting software that has output quality as good as
TeX.

I like TeX as much as the next guy --- I wrote my thesis with it,
a long time ago --- but there's no denying that (a) it's got hard
limits we're approaching, (b) the downstream conversion to PDF is
buggy, and (c) nobody is working on fixing it.

regards, tom lane

On Tue, Jan 23, 2018 at 10:22:53PM -0500, Tom Lane wrote:

Bruce Momjian <bruce@momjian.us> writes:

On Thu, Nov 23, 2017 at 03:39:24PM -0500, Tom Lane wrote:

Also, we're way overdue for getting out from under the creaky TeX-based
toolchain for producing PDFs.

I am coming in late here, but I am not aware of any open source
professional typesetting software that has output quality as good as
TeX.

I like TeX as much as the next guy --- I wrote my thesis with it,
a long time ago --- but there's no denying that (a) it's got hard
limits we're approaching, (b) the downstream conversion to PDF is
buggy, and (c) nobody is working on fixing it.

All agreed, but what alternatives are being developed?

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Tuesday, January 23, 2018, Bruce Momjian <bruce@momjian.us> wrote:

On Tue, Jan 23, 2018 at 10:22:53PM -0500, Tom Lane wrote:
(a) it's got hard

limits we're approaching,

All agreed, but what alternatives are being developed?

I seem to recall a proposal a while back to gain margin on some of the
limits by pruning the release notes section down to at least this century
and archiving putting the older ones elsewhere.

David J.

"David G. Johnston" <david.g.johnston@gmail.com> writes:

On Tuesday, January 23, 2018, Bruce Momjian <bruce@momjian.us> wrote:

All agreed, but what alternatives are being developed?

I seem to recall a proposal a while back to gain margin on some of the
limits by pruning the release notes section down to at least this century
and archiving putting the older ones elsewhere.

Yeah; I did and still do think that's a good idea. But so far as the
toolchain is concerned, that's just a band-aid.

Anyway, we're on XML now, and it seems to be working fairly well.
I don't feel a need to revisit that. It's probably true that the
TeX-based toolchain was potentially capable of producing finer
typesetting results than the XML chain ... but, honestly, who's
printing the PG manual on dead trees anymore? I find the PDF output
to be mostly a legacy thing in the first place.

regards, tom lane

On 1/23/18 22:24, Bruce Momjian wrote:

I like TeX as much as the next guy --- I wrote my thesis with it,
a long time ago --- but there's no denying that (a) it's got hard
limits we're approaching, (b) the downstream conversion to PDF is
buggy, and (c) nobody is working on fixing it.

All agreed, but what alternatives are being developed?

We are using FOP now, which appears to give reasonable results. There
are also commercial/proprietary/expensive XSL-FO processors that give
even better results. Some of those are used in professional publishing.

That's where things are now. The next wave in this area will be doing
the whole thing with HTML and CSS.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Wed, Jan 24, 2018 at 08:21:21AM -0500, Peter Eisentraut wrote:

On 1/23/18 22:24, Bruce Momjian wrote:

I like TeX as much as the next guy --- I wrote my thesis with it,
a long time ago --- but there's no denying that (a) it's got hard
limits we're approaching, (b) the downstream conversion to PDF is
buggy, and (c) nobody is working on fixing it.

All agreed, but what alternatives are being developed?

We are using FOP now, which appears to give reasonable results. There
are also commercial/proprietary/expensive XSL-FO processors that give
even better results. Some of those are used in professional publishing.

That's where things are now. The next wave in this area will be doing
the whole thing with HTML and CSS.

Ah, so we are using fop now:

https://xmlgraphics.apache.org/fop/

and it seems I have fop installed on my server already. So we are not
using TeX anymore for PG11+ docs?

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Tue, Jan 23, 2018 at 11:16:23PM -0500, Tom Lane wrote:

"David G. Johnston" <david.g.johnston@gmail.com> writes:

On Tuesday, January 23, 2018, Bruce Momjian <bruce@momjian.us> wrote:

All agreed, but what alternatives are being developed?

I seem to recall a proposal a while back to gain margin on some of
the limits by pruning the release notes section down to at least
this century and archiving putting the older ones elsewhere.

Yeah; I did and still do think that's a good idea. But so far as
the toolchain is concerned, that's just a band-aid.

Anyway, we're on XML now, and it seems to be working fairly well. I
don't feel a need to revisit that. It's probably true that the
TeX-based toolchain was potentially capable of producing finer
typesetting results than the XML chain ... but, honestly, who's
printing the PG manual on dead trees anymore? I find the PDF output
to be mostly a legacy thing in the first place.

Speaking of legacy things, do we want to see about deprecating LaTeX
output for tables in psql? I haven't seen such a thing in production,
but maybe I'm missing something important.

Best,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate