Switching to XML

On Fri, 2006-12-08 at 09:09 -0500, David Blewett wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi All:

I read a thread from July explaining the current status of moving to XML
[1].

To add to this, it would take minutes to transform XML versus the days
it takes the SGML.

Joshua D. Drake

Peter, you mention that if someone has a clear need you would be open to
switching. Let me explain the problems I've encountered with SGML.

A few months ago, I split up the 8.1 pdf into 2 volumes so that I could
get a hardbound copy from Lulu. You can get those here [2-3] (I'm not
making any money, it's the pure cost of production). For 8.2, I wanted
to go deeper and be able to distribute the final version to bookstores.
In order to do that, each volume needs to be under 700 pages.

I started work on modifying the SGML to use a set of 3 volumes, split at
roughly 500 page intervals. I wanted to generate individual ToC's and
indexes for each volume. I started to modify the SGML to include a
"role" attribute for each indexterm, to tell what volume it was part of.
This was done with a simple sed script. When I went to generate the
actual indexes, I hit a brick wall. Apparently, the SGML toolchain
cannot handle typed indexes as described here [4]. Only the XML
toolchain currently handles them.

I am working with Joshua Drake on creating a Lulu account for the
fundraising group. We would then move the volumes I did for 8.1 to their
account, and raise the price so that any profit went to them. The
ability to order the 8.2 manual from bookstores could increase the
visibility of the project as a whole.

David Blewett

1.
http://groups.google.com/group/pgsql.docs/browse_thread/thread/37ff3a011bb705d7
2. http://www.lulu.com/content/455020
3. http://www.lulu.com/content/464949
4. http://www.xml.com/pub/a/2004/07/14/dbndx.html?page=3
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFFeXIOZmlc6wNjtLYRAo2yAKCZT1NbsklCd8djADdv48MuLELG7wCfYYG3
ypUw8LU3g++GaY6Dz2vUi+I=
=lC+c
-----END PGP SIGNATURE-----

---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?

http://archives.postgresql.org

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate

David Blewett wrote:

I started work on modifying the SGML to use a set of 3 volumes, split
at roughly 500 page intervals. I wanted to generate individual ToC's
and indexes for each volume. I started to modify the SGML to include
a "role" attribute for each indexterm, to tell what volume it was
part of. This was done with a simple sed script. When I went to
generate the actual indexes, I hit a brick wall. Apparently, the SGML
toolchain cannot handle typed indexes as described here [4]. Only the
XML toolchain currently handles them.

But no one is forcing you to use "the SGML toolchain". I requote the
message you cited:

First of all, moving to DocBook XML will not do anything in the way of
improving our output processing abilities. Any tool that you can use
on DocBook SGML can also be used on DocBook XML and vice versa.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter,

First of all, moving to DocBook XML will not do anything in the way of
improving our output processing abilities. Any tool that you can use
on DocBook SGML can also be used on DocBook XML and vice versa.

As I said then, this is absolutely untrue. OpenOffice.org, for example,
works with DocBook XML but not SGML. There are also a plethora of XML
editing and publishing tools which can been used for Docbook XML which
are not available for SGML. A simple look at this page:
http://wiki.docbook.org/topic/DocBookAuthoringTools
.... shows that there are more than twice as many authoring tools which
support only XML as support SGML -- and that most of the tools which
support SGML are out-of-maintenance.

--Josh Berkus

On Fri, 2006-12-08 at 13:36 -0500, Josh Berkus wrote:

Peter,

First of all, moving to DocBook XML will not do anything in the way of
improving our output processing abilities. Any tool that you can use
on DocBook SGML can also be used on DocBook XML and vice versa.

As I said then, this is absolutely untrue.

You are correct Josh. I am not sure why Peter ignores this simple fact.

OpenOffice.org, for example,
works with DocBook XML but not SGML. There are also a plethora of XML
editing and publishing tools which can been used for Docbook XML which
are not available for SGML. A simple look at this page:
http://wiki.docbook.org/topic/DocBookAuthoringTools
.... shows that there are more than twice as many authoring tools which
support only XML as support SGML -- and that most of the tools which
support SGML are out-of-maintenance.

Yep.

Sincerely,

Joshua D. Drake

--Josh Berkus

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate

Josh Berkus wrote:

As I said then, this is absolutely untrue. OpenOffice.org, for
example, works with DocBook XML but not SGML.

What does "works" mean?

There are also a
plethora of XML editing and publishing tools which can been used for
Docbook XML which are not available for SGML.

That has nothing to do with processing.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

On Fri, 2006-12-08 at 21:15 +0100, Peter Eisentraut wrote:

Josh Berkus wrote:

As I said then, this is absolutely untrue. OpenOffice.org, for
example, works with DocBook XML but not SGML.

What does "works" mean?

You can create, edit, convert, save, and open docbook xml in
OpenOfice.org.

There are also a
plethora of XML editing and publishing tools which can been used for
Docbook XML which are not available for SGML.

That has nothing to do with processing.

You are correct but XML does give us more modern and efficient toolsets
for processing. XSLT, FOP + Xerces for example.

Sincerely,

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate

Joshua D. Drake wrote:

You can create, edit, convert, save, and open docbook xml in
OpenOfice.org.

Sure, there are more editing options with DocBook XML. No one disputes
that. But the question at hand was about processing the DocBook.

You are correct but XML does give us more modern and efficient
toolsets for processing. XSLT, FOP + Xerces for example.

You can do that right now.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

On Fri, 2006-12-08 at 21:58 +0100, Peter Eisentraut wrote:

Joshua D. Drake wrote:

You can create, edit, convert, save, and open docbook xml in
OpenOfice.org.

Sure, there are more editing options with DocBook XML. No one disputes
that. But the question at hand was about processing the DocBook.

Yes which is generated from our use of SGML which is the core of this
problem and the core of the question as a whole.

SGML is making working with the documentation *harder*.

We have people that *DO NOT* contribute because of this SGML
requirement. They have what I consider extremely valid reasons, namely
it is dumb to require a writer to use emacs or write tags explictly.

Hell, the only reason I have even bothered to contribute what little I
have to the docs is because I wrote a book in SGML, thus it is a no
brainer to me. Others aren't so tortured as to have done the same.

There is a long standing support within the community to move to XML
including:

Josh Berkus
Josh Drake
Robert Treat
Andrew Dunslane
David Blewett
David Fetter
Devrim Gunduz
Darcy Buskermolen

And that is just from #postgresql

The french team also uses Docbook XML and they can generate a PDF in 30
minutes... it takes us DAYS because of the SGML.

How about we stopping using assembling and move to C shall we?

Sincerely,

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate

On Fri, 2006-12-08 at 15:26, Joshua D. Drake wrote:

On Fri, 2006-12-08 at 21:58 +0100, Peter Eisentraut wrote:

Joshua D. Drake wrote:

You can create, edit, convert, save, and open docbook xml in
OpenOfice.org.

Sure, there are more editing options with DocBook XML. No one disputes
that. But the question at hand was about processing the DocBook.

Yes which is generated from our use of SGML which is the core of this
problem and the core of the question as a whole.

SGML is making working with the documentation *harder*.

We have people that *DO NOT* contribute because of this SGML
requirement. They have what I consider extremely valid reasons, namely
it is dumb to require a writer to use emacs or write tags explictly.

I'm one. I had the vacuuming stuff about written, but couldn't figure
out the tool set this summer, and didn't have the time. Does OO make it
pretty much painless to work with? I couldn't find any tools that I
felt comfortable using for SGML / Docbooks.

On Fri, 2006-12-08 at 15:45 -0600, Scott Marlowe wrote:

On Fri, 2006-12-08 at 15:26, Joshua D. Drake wrote:

On Fri, 2006-12-08 at 21:58 +0100, Peter Eisentraut wrote:

Joshua D. Drake wrote:

You can create, edit, convert, save, and open docbook xml in
OpenOfice.org.

Sure, there are more editing options with DocBook XML. No one disputes
that. But the question at hand was about processing the DocBook.

Yes which is generated from our use of SGML which is the core of this
problem and the core of the question as a whole.

SGML is making working with the documentation *harder*.

We have people that *DO NOT* contribute because of this SGML
requirement. They have what I consider extremely valid reasons, namely
it is dumb to require a writer to use emacs or write tags explictly.

I'm one. I had the vacuuming stuff about written, but couldn't figure
out the tool set this summer, and didn't have the time. Does OO make it
pretty much painless to work with? I couldn't find any tools that I
felt comfortable using for SGML / Docbooks.

OO treats Docbook like a normal document. You will however loose styles
(like bold, italic). It does support tables, it understands
transformation from things like sect1 (from OO heading1) etc...

The style loss is to be expected because Docbook doesn't contain
representation data. That belongs to a style sheet.

Sincerely,

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate

On Fri, 2006-12-08 at 15:51, Joshua D. Drake wrote:

On Fri, 2006-12-08 at 15:45 -0600, Scott Marlowe wrote:

On Fri, 2006-12-08 at 15:26, Joshua D. Drake wrote:

On Fri, 2006-12-08 at 21:58 +0100, Peter Eisentraut wrote:

Joshua D. Drake wrote:

You can create, edit, convert, save, and open docbook xml in
OpenOfice.org.

Sure, there are more editing options with DocBook XML. No one disputes
that. But the question at hand was about processing the DocBook.

Yes which is generated from our use of SGML which is the core of this
problem and the core of the question as a whole.

SGML is making working with the documentation *harder*.

We have people that *DO NOT* contribute because of this SGML
requirement. They have what I consider extremely valid reasons, namely
it is dumb to require a writer to use emacs or write tags explictly.

I'm one. I had the vacuuming stuff about written, but couldn't figure
out the tool set this summer, and didn't have the time. Does OO make it
pretty much painless to work with? I couldn't find any tools that I
felt comfortable using for SGML / Docbooks.

OO treats Docbook like a normal document. You will however loose styles
(like bold, italic). It does support tables, it understands
transformation from things like sect1 (from OO heading1) etc...

The style loss is to be expected because Docbook doesn't contain
representation data. That belongs to a style sheet.

and let me add that I'm not really anti-sgml docbook, I just couldn't
find a "starter set" for editing the stuff. It seemed like everything I
found on docbook xml was written for people who already use docbook xml.

OO treats Docbook like a normal document. You will however loose styles
(like bold, italic). It does support tables, it understands
transformation from things like sect1 (from OO heading1) etc...

The style loss is to be expected because Docbook doesn't contain
representation data. That belongs to a style sheet.

and let me add that I'm not really anti-sgml docbook, I just couldn't
find a "starter set" for editing the stuff. It seemed like everything I
found on docbook xml was written for people who already use docbook xml.

Nor am I anti-sgml. I am however anti-noncontribution, if people are not
contributing because of our sgml but would if it is xml, that is a no
brainer.

Further, here is a real world problem that our toolset creates...

I take 5 minutes, change the stylesheet for SGML. I want to see what my
changes will look like... 3 days later, I will know.

That is stupid. If it was XML, it would be 30 minutes. That is a
workable timeframe.

Sincerely,

Joshua D. Drake

---------------------------(end of broadcast)---------------------------
TIP 7: You can help support the PostgreSQL project by donating at

http://www.postgresql.org/about/donate

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate

Joshua D. Drake wrote:

OO treats Docbook like a normal document. You will however loose styles
(like bold, italic). It does support tables, it understands
transformation from things like sect1 (from OO heading1) etc...

The style loss is to be expected because Docbook doesn't contain
representation data. That belongs to a style sheet.

and let me add that I'm not really anti-sgml docbook, I just couldn't
find a "starter set" for editing the stuff. It seemed like everything I
found on docbook xml was written for people who already use docbook xml.

Nor am I anti-sgml. I am however anti-noncontribution, if people are not
contributing because of our sgml but would if it is xml, that is a no
brainer.

Further, here is a real world problem that our toolset creates...

I take 5 minutes, change the stylesheet for SGML. I want to see what my
changes will look like... 3 days later, I will know.

That is stupid. If it was XML, it would be 30 minutes. That is a
workable timeframe.

Sincerely,

Joshua D. Drake

---------------------------(end of broadcast)---------------------------
TIP 7: You can help support the PostgreSQL project by donating at

http://www.postgresql.org/about/donate

I am not an expert in any of this but would say the one time I had
something I would have added to a doc (windows Vs *nix differences in
copy command) I was in knots with the whole SGML thing and figured it
would be quicker to just post an answer on the mailing list so it would
be immortalized, it should not be so hard to contribute. I feel there
are people who maybe cannot code or add in that way but could add to
docs if it was easier. I had an easier time setting up a pgAdmin build
than figuring out the SGML stuff.

Oisin

On 08/12/06, Joshua D. Drake <jd@commandprompt.com> wrote:

There is a long standing support within the community to move to XML
including:

Josh Berkus
Josh Drake
Robert Treat
Andrew Dunslane
David Blewett
David Fetter
Devrim Gunduz
Darcy Buskermolen

And that is just from #postgresql

And now in Chile, I wrote an app to translate postgres documentation
(l10n.postgresql.cl). I took SGML sources, transform to XML and to POT
files

The results are good! but is it not the best. To move from SGML to
XML is a hard work.

--
http://www.advogato.org/person/mgonzalez/

Joshua D. Drake a �crit :

On Fri, 2006-12-08 at 21:58 +0100, Peter Eisentraut wrote:

Joshua D. Drake wrote:

You can create, edit, convert, save, and open docbook xml in
OpenOfice.org.

Sure, there are more editing options with DocBook XML. No one disputes
that. But the question at hand was about processing the DocBook.

Yes which is generated from our use of SGML which is the core of this
problem and the core of the question as a whole.

SGML is making working with the documentation *harder*.

+1

We have people that *DO NOT* contribute because of this SGML
requirement. They have what I consider extremely valid reasons, namely
it is dumb to require a writer to use emacs or write tags explictly.

Hell, the only reason I have even bothered to contribute what little I
have to the docs is because I wrote a book in SGML, thus it is a no
brainer to me. Others aren't so tortured as to have done the same.

I'm not so sure it will help you find more contributors. I'm part of a
project which aims to translate HOWTO from TLDP. They don't find
contributors and we too have really hard times to find contributors
despite the fact we try to only use DocBook XML (TLDP use DocBook SGML,
DocBook XML and LinuxDoc formats).

Did you try to use OpenOffice.org with DocBook ? I tried once and it was
a complete disaster. But it was a long time ago. I will try again this
week-end.

There is a long standing support within the community to move to XML
including:

Josh Berkus
Josh Drake
Robert Treat
Andrew Dunslane
David Blewett
David Fetter
Devrim Gunduz
Darcy Buskermolen

And that is just from #postgresql

The french team also uses Docbook XML and they can generate a PDF in 30
minutes... it takes us DAYS because of the SGML.

In fact, we need 15 minutes to build HTML files and 10 minutes to build
PDF file. To be completely honest, I don't seem to be able to build PDF
file for 8.2.0 release. I must have made a mistake (or perhaps a lot of
:) ).

Regards.

--
Guillaume.
<!-- http://abs.traduc.org/
http://lfs.traduc.org/
http://docs.postgresqlfr.org/ -->

On Fri, 2006-12-08 at 13:26 -0800, Joshua D. Drake wrote:
<snip>

Yes which is generated from our use of SGML which is the core of this
problem and the core of the question as a whole.

SGML is making working with the documentation *harder*.

From a total outsider's point of view I have to disagree. It took me a

couple of minutes to figure out how to make the tiny change I did the
other day by looking at the rest of the sgml (managed to get the diff
the wrong way around but not the sgml :/).

We have people that *DO NOT* contribute because of this SGML
requirement. They have what I consider extremely valid reasons, namely
it is dumb to require a writer to use emacs or write tags explictly.

Again would have to disagree - surely if someone really wants to
contribute they could provide their input in plain text, and someone on
the list could then integrate those contribs.

Hell, the only reason I have even bothered to contribute what little I
have to the docs is because I wrote a book in SGML, thus it is a no
brainer to me. Others aren't so tortured as to have done the same.

I would hate to hand edit the stuff generated by something like
OpenOffice.org.

There is a long standing support within the community to move to XML
including:

Josh Berkus
Josh Drake
Robert Treat
Andrew Dunslane
David Blewett
David Fetter
Devrim Gunduz
Darcy Buskermolen

And that is just from #postgresql

The french team also uses Docbook XML and they can generate a PDF in 30
minutes... it takes us DAYS because of the SGML.

Here I agree - 30 minutes vs days is a good reason - as long as editing
with an ascii editor is not taken away.

--
Regards
Theo

Joshua D. Drake wrote:

Further, here is a real world problem that our toolset creates...

I take 5 minutes, change the stylesheet for SGML. I want to see what
my changes will look like... 3 days later, I will know.

That is stupid. If it was XML, it would be 30 minutes. That is a
workable timeframe.

I feel like I'm talkling to a wall here, but I'm going to say this one
last time:

Any processing toolchain that you can use with DocBook XML you can use
with DocBook SGML and vice versa.

If you know of a way to create a PDF off DocBook XML in 30 minutes,
please tell us.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Josh Berkus <josh@agliodbs.com> writes:

As I said then, this is absolutely untrue. OpenOffice.org, for example,
works with DocBook XML but not SGML. There are also a plethora of XML
editing and publishing tools which can been used for Docbook XML which
are not available for SGML. A simple look at this page:
http://wiki.docbook.org/topic/DocBookAuthoringTools
.... shows that there are more than twice as many authoring tools which
support only XML as support SGML -- and that most of the tools which
support SGML are out-of-maintenance.

This is confusing authoring tools (ie, stuff for more or less WYSIWYG
editing of the document source) with output generation tools.

As for authoring tools, show me one that produces SGML or XML that's
reasonably readable, and I might worry about allowing people to use it.
Most of the ones I've seen would render the doc sources unreadable for
anyone not using an authoring tool (possibly even the very same
authoring tool). We are not going to move in that direction
because it would piss off the people who do the bulk of the work now.

regards, tom lane

The french team also uses Docbook XML and they can generate a PDF in 30
minutes... it takes us DAYS because of the SGML.

Has anyone looked into actually fixing the performance problem?

oprofile results for jade trying to produce tex output from our docs are
suggestive of a localized performance issue:

samples % symbol name
2082917 98.5829 OpenJade_DSSSL::PairNodeListObj::nodeListFirst(OpenJade_DSSSL::EvalContext&, OpenJade_DSSSL::Interpreter&)
9713 0.4597 OpenJade_DSSSL::PairNodeListObj::nodeListRest(OpenJade_DSSSL::EvalContext&, OpenJade_DSSSL::Interpreter&)
5019 0.2375 OpenJade_DSSSL::AppendSosofoObj::traceSubObjects(Collector&) const
3571 0.1690 Collector::collect()
1938 0.0917 OpenJade_DSSSL::FlowObj::traceSubObjects(Collector&) const

I attached to the process with gdb and found it nested four thousand (!)
call levels deep in OpenJade_DSSSL::PairNodeListObj::nodeListFirst and
OpenJade_DSSSL::PairNodeListObj::nodeListRest calls. Meanwhile, looking
at the output-so-far-emitted makes me think it was working on a fairly
large <programlisting> example. The last little bit is:

{asis}\def\InputWhitespaceTreatment%
{preserve}}\Seq%
{}\Seq%
{}~~~~\endSeq{}/*
\Seq%
{}~~~~\endSeq{}~*~testlibpq2.c
\Seq%
{}~~~~\endSeq{}~*~~~~~~Test~of~the~asynchronous~notification~interface
\Seq%
{}~~~~\endSeq{}~*
\Seq%
{}~~~~\endSeq{}~*~Start~this~program,~then~from~psql~in~another~window~do
\Seq%
{}~~~~\endSeq{}~*~~~NOTIFY~TBL2;
\Seq%
{}~~~~\endSeq{}~*~Repeat~four~times~to~get~this~program~to~exit.
\Seq%
{}~~~~\endSeq{}~*
\Seq%
{}~~~~\endSeq{}~*~Or,~if~you~want~to~get~fancy,~try~this:
\Seq%
{}~~~~\endSeq{}~*~populate~a~database~with~th

What it looks like to me is that there is some bit of stupidity that is
producing a deeply nested list representation of a <programlisting>
section, probably one list level per character in the text, making the
runtime O(N^2) or worse in the length of the <programlisting>. (The
particular example it's stuck on here is about 10K characters.)

Since jade does not go into this kind of spiral when producing html
output from the same sources, I suggest that it's not jade's fault,
but rather crummy coding in the sgml-to-tex conversion scripts it's
using. I don't know enough about those to know where to look, but maybe
someone here does?

regards, tom lane