Images in the official documentation

Daniel Westermann <daniel.westermann@dbi-services.com> writes:

Is there an agreement not to include that pictures for any reason?

This has been discussed repeatedly (see the archives). We'd surely
like the ability to include figures, but we've not found any tools that
met the varying requirements people wanted to set --- mainly, that it
be possible to commit readable source text for figures into our git
repo and have updates that were reviewable (ie, didn't amount to a 100%
replacement of one set of gibberish with another one). In the distant
past, as I recall, we had a GIF or two; but we abandoned that on the
grounds that it was unmaintainable and also incompatible with some
documentation output formats. I'm not too sure what the state of
play is on the latter point, now that we've switched to XML.

regards, tom lane

On 2/23/18 11:21, Tom Lane wrote:

In the distant
past, as I recall, we had a GIF or two; but we abandoned that on the
grounds that it was unmaintainable and also incompatible with some
documentation output formats. I'm not too sure what the state of
play is on the latter point, now that we've switched to XML.

The complications with the image formats in the past were mainly around
what ((pdf)jade)tex would accept. The tools have shifted a bit now, and
the zoo formats is a different one. Nothing that a few make rules
couldn't address, though, I think.

The issue of how to manage the sources is still the same, though.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Sat, Feb 24, 2018 at 4:04 AM, Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:

On 2/23/18 11:21, Tom Lane wrote:

In the distant
past, as I recall, we had a GIF or two; but we abandoned that on the
grounds that it was unmaintainable and also incompatible with some
documentation output formats. I'm not too sure what the state of
play is on the latter point, now that we've switched to XML.

The complications with the image formats in the past were mainly around
what ((pdf)jade)tex would accept. The tools have shifted a bit now, and
the zoo formats is a different one. Nothing that a few make rules
couldn't address, though, I think.

The issue of how to manage the sources is still the same, though.

SVG format is ascii based vector format. We made experimental pdf with pictures
http://www.sai.msu.su/~megera/postgres/files/postgres-11-diagram.pdf
(GIN AM diagram, Appendix L).

Appendix L also demonstrates our sample database with step-by-step
introduction to Postgres for beginners. We have a separate book for beginners,
which we released under BSD license and it's available on
russian/english languages.
Our experience shows, that people really appreciate it. I hope we will
have time at PGCon
to discuss documentation somehow.

Show quoted text

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Oleg Bartunov <obartunov@gmail.com> writes:

On Sat, Feb 24, 2018 at 4:04 AM, Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:

The issue of how to manage the sources is still the same, though.

SVG format is ascii based vector format.

Yeah. I think where the last discussion left this was that we'd be
willing to accept SVG-format figures, but we were having a hard time
figuring out what tools to recommend for editing them, because available
editors have no respect for readability or preserving small deltas when
rewriting SVG files.

regards, tom lane

As an addition to my mail from January 2016 concerning graphics
(/messages/by-id/568A9148.30303@purtz.de) I
propose to use SVG (after switching to XML) - but not an SVG which is
generated by Inkscape or similar tools. Those editors generate very ugly
and chatty commands. This form is not easy to read or understand.
Therefore we shall use nothing but a simple text editor and write every
line by our self. The process is divided into two parts:

As a basis we shall develop an SVG library containing a bunch of
"atomic" symbols of simple graphical elements (rectangle, arrow, ...) up
to complex elements (magnetic disc, laptop, cloud, UML-elements, ...).
The SVG routines creating those symbols shall accept parameters for
position, size, rotation, colour, ... . This library shortens the
individual SVG files, it ensures a consistent rendering of common
graphical elements, it is diff-able, and it will reach a stable state -
some day.

The real graphics shall use the elements of the library and add
individual SVG elements. The rules for this part are the same as above:
create SVG commands with vi (or similar), store it in git.

If such an approach works (we must distribute the docs across a wide
range of different systems, a proof-of-system is necessary) and the
community accepts my proposal, I would like to work on the library-part
- starting after finishing my actual project in about 6 weeks from now.
The attached file contains a very first draft as of Jan. 2016.

Kind regards
Jürgen Purtz

Show quoted text

On 23.02.2018 22:14, Daniel Westermann wrote:

Hi %,

I am working with PostgreSQL documentation quite a few years now and I
am almost happy. What I think is completely missing (especially if you
compare to commercial product documentation) are pictures that
illustrate a topic, e.g. the relation of
instance->database->user/role->schema->objects.

Is there an agreement not to include that pictures for any reason? I
can not promise that I find time for that in the near future but if
that will be appreciated I am willing to spend time on that to make
the documentation even better.

Regards
Daniel

On Feb 25, 2018, at 4:00 AM, Jürgen Purtz <juergen@purtz.de> wrote:

As an addition to my mail from January 2016 concerning graphics (/messages/by-id/568A9148.30303@purtz.de) I propose to use SVG (after switching to XML) - but not an SVG which is generated by Inkscape or similar tools. Those editors generate very ugly and chatty commands. This form is not easy to read or understand. Therefore we shall use nothing but a simple text editor and write every line by our self. The process is divided into two parts:
As a basis we shall develop an SVG library containing a bunch of "atomic" symbols of simple graphical elements (rectangle, arrow, ...) up to complex elements (magnetic disc, laptop, cloud, UML-elements, ...). The SVG routines creating those symbols shall accept parameters for position, size, rotation, colour, ... . This library shortens the individual SVG files, it ensures a consistent rendering of common graphical elements, it is diff-able, and it will reach a stable state - some day.

The real graphics shall use the elements of the library and add individual SVG elements. The rules for this part are the same as above: create SVG commands with vi (or similar), store it in git.
If such an approach works (we must distribute the docs across a wide range of different systems, a proof-of-system is necessary) and the community accepts my proposal, I would like to work on the library-part - starting after finishing my actual project in about 6 weeks from now. The attached file contains a very first draft as of Jan. 2016.

Writing SVG by hand maybe doesn't seem the best idea.

I understand the attraction to people who want to store everything as diffable text, but images of this sort are unlikely to get updated by others, which means they're unlikely to be maintained as the things they're intended to document change. It also means that the people best suited to generating diagrams are the least likely to do so, and vice-versa.

Cheers,
Steve

On 26 February 2018 at 04:12, Steve Atkins <steve@blighty.com> wrote:

Writing SVG by hand maybe doesn't seem the best idea.

I understand the attraction to people who want to store everything as
diffable text, but images of this sort are unlikely to get updated by
others, which means they're unlikely to be maintained as the things they're
intended to document change. It also means that the people best suited to
generating diagrams are the least likely to do so, and vice-versa.

Yeah, I think it'd just effectively preserve the status quo by rendering
anyone who's willing to add images and designs to the docs unable - or
unlikely to be willing - to do so.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Feb 25, 2018, at 6:45 PM, Craig Ringer <craig@2ndquadrant.com> wrote:

On 26 February 2018 at 04:12, Steve Atkins <steve@blighty.com> wrote:

Writing SVG by hand maybe doesn't seem the best idea.

I understand the attraction to people who want to store everything as diffable text, but images of this sort are unlikely to get updated by others, which means they're unlikely to be maintained as the things they're intended to document change. It also means that the people best suited to generating diagrams are the least likely to do so, and vice-versa.

Yeah, I think it'd just effectively preserve the status quo by rendering anyone who's willing to add images and designs to the docs unable - or unlikely to be willing - to do so.

Yup. I do think that graphics would be nice in a few places, and that SVG is likely the best format for them.

There are quite a few tools that could be integrated with varying amounts of effort into the documentation generation workflow.

# ASCII language or ascii art to SVG

# asciitosvg

https://github.com/dhobsd/asciitosvg

Inspired by ditaa, similar in functionality

## blockdiag, seqdiag, actdiag, nwdiag

http://blockdiag.com

Generates various box and arrow diagrams from a DOT-ish input language.

## ditaa

It takes ascii art of box and arrow diagrams and turns them into nice svg. Also supports boxes that look like "storage" cylinders, documents, clouds and computers.

## erd

https://github.com/BurntSushi/erd

Entity relationship diagrams, from a plain text input. Uses DOT and graphviz under the covers.

## Markdeep

http://casual-effects.com/markdeep/

In-browser javascript rendering of ascii art, particularly boxes and arrows.

Alternate implementation, https://github.com/blampe/goat, converts to SVG via CLI.

## Mermaid

Flowcharts and sequence diagrams from a markdown-esque input.

## mscgen

Message sequence chart inputs in a DOT-ish language to SVG

## plantuml

http://plantuml.com

It supports a human-editable descriptive text input language and generates from it:

sequence diagrams
various box + arrow style diagrams
flowcharts
state diagrams
etc.

## shaape

https://github.com/christiangoltz/shaape

Converts ascii art to SVG. Rather nice.

## svgbob

https://github.com/ivanceras/svgbobrus

Ascii art to SVG. Likely good for boxes and arrows.

## syntrax

https://kevinpt.github.io/syntrax/

Railroad diagrams (like the ones SQLite docs are known for).

## umlet

http://www.umlet.com

Pointy-clicky editor, but also converts plain text to uml, sequence, activity diagrams

# Interactive editor

## SVG-Edit

https://github.com/SVG-Edit/svgedit

Open source, browser based interactive SVG editor. Seems to generate fairly clean SVG.

Cheers,
Steve

Craig Ringer <craig@2ndquadrant.com> writes:

On 26 February 2018 at 04:12, Steve Atkins <steve@blighty.com> wrote:

Writing SVG by hand maybe doesn't seem the best idea.
I understand the attraction to people who want to store everything as
diffable text, but images of this sort are unlikely to get updated by
others, which means they're unlikely to be maintained as the things they're
intended to document change. It also means that the people best suited to
generating diagrams are the least likely to do so, and vice-versa.

Yeah, I think it'd just effectively preserve the status quo by rendering
anyone who's willing to add images and designs to the docs unable - or
unlikely to be willing - to do so.

This is an entirely reasonable complaint. But I don't see how we'd cope
with patches that rewrite an entire SVG file because the patch author's
WYSIWG editor emits its output in a style randomly different from the tool
the previous patch author used. It seems like such patches would be
effectively unreviewable, and certainly incapable of being merged.

Maybe we could improve matters a bit by insisting that everyone use the
same version of the same SVG-editing tool. But that's not too practical.
Worse, from what I've seen, even that would not really fix the problem.
The tools simply don't give a damn about comparability of their dump
files. I don't blame their authors exactly (try diffing Postgres data
file changes :-() but that doesn't mean it isn't a problem for us.

How can we resolve these issues?

regards, tom lane

On 26 February 2018 at 12:16, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Craig Ringer <craig@2ndquadrant.com> writes:

Yeah, I think it'd just effectively preserve the status quo by rendering
anyone who's willing to add images and designs to the docs unable - or
unlikely to be willing - to do so.

This is an entirely reasonable complaint. But I don't see how we'd cope
with patches that rewrite an entire SVG file because the patch author's
WYSIWG editor emits its output in a style randomly different from the tool
the previous patch author used. It seems like such patches would be
effectively unreviewable, and certainly incapable of being merged.

Well, they definitely couldn't be merged in any situation entailing
conflicts, no.

Patch review would entail displaying the new and (if present) old SVGs,
possibly in the context of a build of the docs, or possibly standalone.

This is always going to be the case for anything but the most trivial SVG
changes anyway. After all, even small textual changes can cause elements to
overlap, break out of their expected bounaries, or otherwise look wrong.

So IMO whether it's SVG or a raster image format, the net effect isn't that
different: you have to review the rendered result not the source.
Personally I'd just mark svg as binary in .gitattributes to stop it from
spamming noise in diffs.

Github offers a cool tool for side-by-side compares of svg diffs (
https://github.com/blog/1902-svg-viewing-diffing) but that likely won't
help us much.

There's diffsvg (https://github.com/jrsmith3/diffsvg), which I haven't
tried but looks interesting. Combined with filters in .gitattributes this
might offer improved visibility into change history if we really need it.

Personally, I don't think images will be changing that often and they
should just be tracked as blobs.

Maybe we could improve matters a bit by insisting that everyone use the
same version of the same SVG-editing tool. But that's not too practical.
Worse, from what I've seen, even that would not really fix the problem.
The tools simply don't give a damn about comparability of their dump
files. I don't blame their authors exactly (try diffing Postgres data
file changes :-() but that doesn't mean it isn't a problem for us.

How can we resolve these issues?

Question the assumptions and requirements. Why do we actually _need_
diffable, mergeable images? Sure, it'd be *nice*, but what's the real world
impact if we don't have it?

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Craig Ringer <craig@2ndquadrant.com> writes:

On 26 February 2018 at 12:16, Tom Lane <tgl@sss.pgh.pa.us> wrote:

How can we resolve these issues?

Question the assumptions and requirements. Why do we actually _need_
diffable, mergeable images? Sure, it'd be *nice*, but what's the real world
impact if we don't have it?

Well, I'll tell you exactly why I'm being sticky about this: we've been
down this road before. We used to have some figures in .gif format,
and one of the problems with them was they were too hard to update.
I don't buy the "they won't need updates" argument for a second, either.
For instance, I recall that one of the images we had was a diagram of
the system catalog cross-references, and it was constantly out of date
because of the difficulty of updating it.

Admittedly, this was 15+ years ago. Maybe the state of the art in
figure editors has advanced to the point where it won't be so hard.
But color me suspicious.

regards, tom lane

On 27 February 2018 at 03:23, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Craig Ringer <craig@2ndquadrant.com> writes:

On 26 February 2018 at 12:16, Tom Lane <tgl@sss.pgh.pa.us> wrote:

How can we resolve these issues?

Question the assumptions and requirements. Why do we actually _need_
diffable, mergeable images? Sure, it'd be *nice*, but what's the real

world

impact if we don't have it?

Well, I'll tell you exactly why I'm being sticky about this: we've been
down this road before. We used to have some figures in .gif format,
and one of the problems with them was they were too hard to update.
I don't buy the "they won't need updates" argument for a second, either.
For instance, I recall that one of the images we had was a diagram of
the system catalog cross-references, and it was constantly out of date
because of the difficulty of updating it.

Yeah, that sounds painful. I can certainly see your objection when framed
in terms of things like illustrations of catalogs and catalog relationships.

If I were maintaining the docs in a vacuum, I'd use graphviz for something
like that, because it's a figure that does need regular updates and
changes. And because
the list of fun things to do in my life definitely does not include
hand-writing SVG. Not that tweaking GraphViz .dot is fun, but it's the
default tool for a reason.

I'd be awfully tempted to generate the node-map part of the catalog
relationship .dot file from a query, too.

I still advocate for relaxing the policy for images that are *not* likely
to need frequent updates, but also for being conservative about how/when we
include images. Does this add real value to the docs, is it worth any
maintenance hassle? Then, for things that will change more, like catalogs,
using a tool like graphviz. If we object to adding a docs build-dependency,
we could always commit generated files like we already do for the
'configure' script, and make sure there's a committer/maintainer Make
target that warns if the sources are newer than the docs.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Mon, Feb 26, 2018 at 10:23 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Craig Ringer <craig@2ndquadrant.com> writes:

On 26 February 2018 at 12:16, Tom Lane <tgl@sss.pgh.pa.us> wrote:

How can we resolve these issues?

Question the assumptions and requirements. Why do we actually _need_
diffable, mergeable images? Sure, it'd be *nice*, but what's the real world
impact if we don't have it?

Well, I'll tell you exactly why I'm being sticky about this: we've been
down this road before. We used to have some figures in .gif format,
and one of the problems with them was they were too hard to update.
I don't buy the "they won't need updates" argument for a second, either.
For instance, I recall that one of the images we had was a diagram of
the system catalog cross-references, and it was constantly out of date
because of the difficulty of updating it.

Admittedly, this was 15+ years ago. Maybe the state of the art in
figure editors has advanced to the point where it won't be so hard.
But color me suspicious.

In case you missed, a couple of years ago we discussed this on pgcon:

Heikki's version:
https://wiki.postgresql.org/wiki/Figures_%26_Pics_in_Docs

Emre suggested to use Markdeep (BSD license),
http://casual-effects.com/markdeep/
http://www.sai.msu.su/~megera/postgres/gin-ascii-v2.md.html

It looks good for small diagrams, but will not work for complex stuff,
such as pg_catalog structure.

Show quoted text

regards, tom lane

On 2/26/18 20:02, Craig Ringer wrote:

If I were maintaining the docs in a vacuum, I'd use graphviz for
something like that, because it's a figure that does need regular
updates and changes. And because
 the list of fun things to do in my life definitely does not include
hand-writing SVG. Not that tweaking GraphViz .dot is fun, but it's the
default tool for a reason.

I'd be awfully tempted to generate the node-map part of the catalog
relationship .dot file from a query, too.

I think graphviz would be a great fit for what we are discussing here.
Certainly more so then some-person-on-github's latest idea for how to
convert ASCII art into diagrams.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Peter Eisentraut wrote:

On 2/26/18 20:02, Craig Ringer wrote:

If I were maintaining the docs in a vacuum, I'd use graphviz for
something like that, because it's a figure that does need regular
updates and changes. And because
�the list of fun things to do in my life definitely does not include
hand-writing SVG. Not that tweaking GraphViz .dot is fun, but it's the
default tool for a reason.

I'd be awfully tempted to generate the node-map part of the catalog
relationship .dot file from a query, too.

I think graphviz would be a great fit for what we are discussing here.
Certainly more so then some-person-on-github's latest idea for how to
convert ASCII art into diagrams.

... particularly so if said idea involves PHP, Haskell, Go, Python, or
any other language that we don't currently have as requirement in our
build chain.

GraphViz gets my vote, too. It may not produce the most elegant
diagrams in the universe, but the source format is as good as we can
get.

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Our discussion about grafics in the documentation reached to the
conclusion that we shall use SVG, the importance to 'diff-ability' is
rated differently, and there is no consensus about tools.

To push the issue forward I modify my original proposal to use plain svg
files in a standard editor as follows:

* We define a 'Simplified SVG format' (SSVG)
* We create libraries where complex elements are predefined and can be
referenced
* We write the source in ssvg-format
* A compiler (written in bison or xslt) converts ssvg-files to svg-files
* We extend the sgml-files to include the svg-files
* The ssvg and svg-files are located in a new svg directory, Makefile
copies them to sgml and html directory
* A proof-of-concept is performed in 11beta2 for HTML and PDF generation.
* The ssvg-format may be XML (as used in the examples), JSON, C-style
function calls

PRO:

* SVG 1.x has many restrictions and SVG 2.x does not make progress in
the last years. Tools and Browsers support different ranges of the
specification. The planned compiler cuts everything down to the
basic language level, where a broad support is possible.
* Predefinded elements and default values reduce the ssvg file to a
small and clear source file.
* You can embed original svg commands into ssvg files.
* When you use an editor and a browser in parallel, you get the visual
result with few clicks.
* Everything is diff-able.
* The Makefile needs only slightly amendments: additional cp commands
and some target-dependencies. We need no new tool.

CON:

* The development is done in a non-wysiwyg editor and without mouse.
* You have to count pixel.

Example:
PageLayout.ssvg: written in the new language
PageLayout.svg: the generated svg file (actually by hand, the compiler
is not yet implemented)
storage.sgml: an additional paragraph to refer to the svg-file

<para>
  <mediaobject id="PageLayoutSVG">
    <imageobject role="html">
      <imagedata fileref="PageLayout.svg" format="SVG"/>
    </imageobject>
    <imageobject role="fo">
      <imagedata fileref="PageLayout.svg" format="SVG" scalefit="1"
width="100%" contentdepth="100%"/>
    </imageobject>
  </mediaobject>
</para>

PageLayoutHtml.png: the HTML result
PageLayoutPdf.png: the PDF result

A second example: pgDump.svg within backup.sgml

Kind regards, Jürgen Purtz

Hello, Jürgen.

You wrote:

JP> Our discussion about grafics in the documentation reached to the
JP> conclusion that we shall use SVG, the importance to 'diff-ability'
JP> is rated differently, and there is no consensus about tools.

I disagree. From what I heard, GraphViz is the winner for now. I can
give you my two cents: plantuml is another good choice

JP> To push the issue forward I modify my original proposal to use
JP> plain svg files in a standard editor as follows:

JP> We define a 'Simplified SVG format' (SSVG) We create libraries
JP> where complex elements are predefined and can be
JP> referenced We write the source in ssvg-format A compiler
JP> (written in bison or xslt) converts ssvg-files to
JP> svg-files We extend the sgml-files to include the svg-files The
JP> ssvg and svg-files are located in a new svg directory,
JP> Makefile copies them to sgml and html directory A
JP> proof-of-concept is performed in 11beta2 for HTML and PDF
JP> generation. The ssvg-format may be XML (as used in the examples),
JP> JSON, C-style function calls
JP>

JP> PRO:
JP>
JP>
JP> SVG 1.x has many restrictions and SVG 2.x does not make
JP> progress in the last years. Tools and Browsers support different
JP> ranges of the specification. The planned compiler cuts
JP> everything down to the basic language level, where a broad
JP> support is possible. Predefinded elements and default values
JP> reduce the ssvg file to a small and clear source file.
JP> You can embed original svg commands into ssvg files. When you use
JP> an editor and a browser in parallel, you get the visual
JP> result with few clicks. Everything is diff-able. The Makefile
JP> needs only slightly amendments: additional cp commands and
JP> some target-dependencies. We need no new tool.
JP>
JP> CON:
JP>
JP>
JP> The development is done in a non-wysiwyg editor and without
JP> mouse. You have to count pixel.
JP>

JP> Example:
JP> PageLayout.ssvg: written in the new language
JP> PageLayout.svg: the generated svg file (actually by hand,
JP> the compiler is not yet implemented)
JP> storage.sgml: an additional paragraph to refer to the svg-file
JP>
JP> <para>
JP>   <mediaobject id="PageLayoutSVG">
JP>     <imageobject role="html">
JP>       <imagedata fileref="PageLayout.svg" format="SVG"/>
JP>     </imageobject>
JP>     <imageobject role="fo">
JP>       <imagedata fileref="PageLayout.svg" format="SVG"
JP> scalefit="1" width="100%" contentdepth="100%"/>
JP>     </imageobject>
JP>   </mediaobject>
JP> </para>
JP>
JP> PageLayoutHtml.png: the HTML result
JP> PageLayoutPdf.png: the PDF result
JP>
JP> A second example: pgDump.svg within backup.sgml
JP>
JP>
JP> Kind regards, Jürgen Purtz
JP>
JP>

JP>
JP>

--
With best wishes,
Pavel mailto:pavel@gf.microolap.com

On 19.07.2018 14:06, Pavel Golub wrote:

I disagree. From what I heard, GraphViz is the winner for now. I can
give you my two cents: plantuml is another good choice

Ok, please give us an example - possibly the two previous graphics.

Kind regards, Jürgen

Hello, Jürgen.

You wrote:

JP> On 19.07.2018 14:06, Pavel Golub wrote:

I disagree. From what I heard, GraphViz is the winner for now. I can
give you my two cents: plantuml is another good choice

JP> Ok, please give us an example - possibly the two previous graphics.

Fair enough. Let's try different formats. Sorry, what exactly previous
graphics we're talking about?

JP> Kind regards, Jürgen

--
With best wishes,
Pavel mailto:pavel@gf.microolap.com