split func.sgml to separated individual sgml files

The following is step-by-step logic.

The end result (one file per section) seems good to me.

I suspect that reviewer burden may be the biggest barrier to going forward.
Perhaps breaking up the changes so that each new sect1 file gets its own
commit, allowing the reviewer to more easily (if not programmatically)
verify that the text that moved out of func.sgml moved into
func-sect-foo.sgml.

Granted, the committer will likely squash all of those commits down into
one big one, but by the the hard work of reviewing is done by then.

On Wed, Nov 13, 2024 at 1:11 PM Corey Huinker <corey.huinker@gmail.com>
wrote:

The following is step-by-step logic.

The end result (one file per section) seems good to me.

I suspect that reviewer burden may be the biggest barrier to going
forward. Perhaps breaking up the changes so that each new sect1 file gets
its own commit, allowing the reviewer to more easily (if not
programmatically) verify that the text that moved out of func.sgml moved
into func-sect-foo.sgml.

Granted, the committer will likely squash all of those commits down into
one big one, but by the the hard work of reviewing is done by then.

Validation is pretty trivial. I just built the before and after HTML files
and confirmed they are exactly the same size.

I suppose we might have lost some comments or something that wouldn't end
up visible in the HTML (seems unlikely) but this is basically one-and-done
so long as you don't let other commits happen (that touch this area) while
you extract and build HEAD and then compare it to the patched build
results. The git diff will let us know the script didn't affect any source
files it wasn't supposed to.

In short, ready to commit (see last paragraph below however), but the
committer will need to run the python script at the time of commit on the
then-current tree.

In my recent patch touching filelist.sgml I would be placing this new
%allfiles_func; line pairing at the top just beneath %allfiles; which is
the first child element. But the choice made here makes sense should this
go in first.

There is little downside, though, to renaming the existing %allfiles; to
%allfiles_ref; It's a local-only name.

David J.

On Thu, Mar 20, 2025 at 10:16 AM David G. Johnston
<david.g.johnston@gmail.com> wrote:

In short, ready to commit (see last paragraph below however), but the committer will need to run the python script at the time of commit on the then-current tree.

hi.
more explanation, since the python script seems quite large...

each <sect1 id="functions-XXX"> in doc/src/sgml/func.sgml
corresponds to each individual section in [1]https://www.postgresql.org/docs/current/functions.html.

each <sect1 id="functions-XXX"> within func.sgml is unique.
if you try to rename it, having two <sect1 id="functions-logical">
will error out saying something like:
../../Desktop/pg_src/src6/postgres/doc/src/sgml/postgres.sgml:199:
element sect1: validity error : ID functions-logical already defined
see [2]https://en.wikipedia.org/wiki/Document_type_definition also.

Based on this, we can use the literal string <sect1 id="functions-XXX"> to
perform pattern matching and identify the line numbers that mark the start and
end of each <sect1> section.

The polished v2 python script use the following steps for splitting func.sgml
into several pieces:

0. For each 9.X section listed in [1]https://www.postgresql.org/docs/current/functions.html, create an empty SGML file to hold the
corresponding content.

1. Use the pattern <sect1 id="functions-XXX"> to locate the starting and ending
line number of each section in func.sgml

2. Copy func.sgml all the content block (<sect1>)

<sect1 id="functions-XXX">
...main content
</sect1>

into the newly created SGML files.

3. Remove the copied content from func.sgml.
4. In func.sgml, insert general entity references [3]https://www.gnu.org/software/sed/manual/html_node/Command_002dLine-Options.html#index-_002di to include the newly
created SGML files.

because PG18, and PG17, Chapter 9. Functions and Operators
have the same amount of section (31),

so v1-0001-split_func_sgml.py will work just fine.
but I did some minor changes, therefore v2 attached.

----------------------------------------------------
I used the sed --in-place option [3]https://www.gnu.org/software/sed/manual/html_node/Command_002dLine-Options.html#index-_002di to modify and truncate the original large
func.sgml file directly.
I also used the -n and -p options with sed to extract lines from func.sgml
between line X and line Y, as shown in reference [4]https://www.gnu.org/software/sed/manual/html_node/Common-Commands.html#index-n-_0028next_002dline_0029.

for the attach file:
first run ``python3 v2-0001-split_func_sgml.py``
then run ``git apply v2-0001-update-filelist.sgml-allfiles.sgml.no-cfbot``
(`git am` won't work, need to use `git apply`).

[1]: https://www.postgresql.org/docs/current/functions.html
[2]: https://en.wikipedia.org/wiki/Document_type_definition
[3]: https://www.gnu.org/software/sed/manual/html_node/Command_002dLine-Options.html#index-_002di
[4]: https://www.gnu.org/software/sed/manual/html_node/Common-Commands.html#index-n-_0028next_002dline_0029

hi.

after run the v2 python script and ``git apply
v2-0001-update-filelist.sgml-allfiles.sgml.no-cfbot``
git status -u
shows:

Changes not staged for commit:
(use "git add/rm <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: doc/src/sgml/filelist.sgml
deleted: doc/src/sgml/func.sgml

That means to verify the changes, we only need to verify html files
related to "functions".

I use GNU diff to compare the HTML output of doc/src/sgml/func.sgml generated
from the master branch against the HTML file produced by the patch.
For example, $DOC9 is the PATCH (split func.sgml) html file directory, $DOC5 is
the master branch html file directory. and no message produced while running
diff, which means the patch (with the script) produced output is the
same as the master branch.

diff $DOC5/functions.html $DOC9/functions.html
diff $DOC5/functions-logical.html $DOC9/functions-logical.html
diff $DOC5/functions-comparison.html $DOC9/functions-comparison.html
diff $DOC5/functions-math.html $DOC9/functions-math.html
diff $DOC5/functions-string.html $DOC9/functions-string.html
diff $DOC5/functions-binarystring.html $DOC9/functions-binarystring.html
diff $DOC5/functions-matching.html $DOC9/functions-matching.html
diff $DOC5/functions-formatting.html $DOC9/functions-formatting.html
diff $DOC5/functions-datetime.html $DOC9/functions-datetime.html
diff $DOC5/functions-enum.html $DOC9/functions-enum.html
diff $DOC5/functions-geometry.html $DOC9/functions-geometry.html
diff $DOC5/functions-net.html $DOC9/functions-net.html
diff $DOC5/functions-textsearch.html $DOC9/functions-textsearch.html
diff $DOC5/functions-uuid.html $DOC9/functions-uuid.html
diff $DOC5/functions-xml.html $DOC9/functions-xml.html
diff $DOC5/functions-json.html $DOC9/functions-json.html
diff $DOC5/functions-sequence.html $DOC9/functions-sequence.html
diff $DOC5/functions-conditional.html $DOC9/functions-conditional.html
diff $DOC5/functions-array.html $DOC9/functions-array.html
diff $DOC5/functions-range.html $DOC9/functions-range.html
diff $DOC5/functions-aggregate.html $DOC9/functions-aggregate.html
diff $DOC5/functions-window.html $DOC9/functions-window.html
diff $DOC5/functions-merge-support.html $DOC9/functions-merge-support.html
diff $DOC5/functions-subquery.html $DOC9/functions-subquery.html
diff $DOC5/functions-comparisons.html $DOC9/functions-comparisons.html
diff $DOC5/functions-srf.html $DOC9/functions-srf.html
diff $DOC5/functions-info.html $DOC9/functions-info.html
diff $DOC5/functions-admin.html $DOC9/functions-admin.html
diff $DOC5/functions-trigger.html $DOC9/functions-trigger.html
diff $DOC5/functions-event-triggers.html $DOC9/functions-event-triggers.html
diff $DOC5/functions-statistics.html $DOC9/functions-statistics.html

On 2025-07-29 Tu 2:15 AM, jian he wrote:

hi.

after run the v2 python script and ``git apply
v2-0001-update-filelist.sgml-allfiles.sgml.no-cfbot``
git status -u
shows:

Changes not staged for commit:
(use "git add/rm <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: doc/src/sgml/filelist.sgml
deleted: doc/src/sgml/func.sgml

That means to verify the changes, we only need to verify html files
related to "functions".

I use GNU diff to compare the HTML output of doc/src/sgml/func.sgml generated
from the master branch against the HTML file produced by the patch.
For example, $DOC9 is the PATCH (split func.sgml) html file directory, $DOC5 is
the master branch html file directory. and no message produced while running
diff, which means the patch (with the script) produced output is the
same as the master branch.

[snip]

OK. I'm inclined to do this after the CF finishes, to avoid collisions
with other patches. I assume it's going to make the CFbot fairly unhappy.

cheers

andrew

--
Andrew Dunstan
EDB:https://www.enterprisedb.com

Andrew Dunstan <andrew@dunslane.net> writes:

OK. I'm inclined to do this after the CF finishes, to avoid collisions
with other patches. I assume it's going to make the CFbot fairly unhappy.

+1 for proceeding that way. (I did not look at whether the proposed
changes are sane, but I agree that this'll inevitably break a lot of
pending patches.)

regards, tom lane

On 2025-07-29 Tu 11:40 AM, Tom Lane wrote:

Andrew Dunstan <andrew@dunslane.net> writes:

OK. I'm inclined to do this after the CF finishes, to avoid collisions
with other patches. I assume it's going to make the CFbot fairly unhappy.

+1 for proceeding that way. (I did not look at whether the proposed
changes are sane, but I agree that this'll inevitably break a lot of
pending patches.)

Done.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

On 4 Aug 2025, at 4:09 PM, Andrew Dunstan <andrew@dunslane.net> wrote:

On 2025-07-29 Tu 11:40 AM, Tom Lane wrote:

Andrew Dunstan <andrew@dunslane.net> writes:

OK. I'm inclined to do this after the CF finishes, to avoid collisions
with other patches. I assume it's going to make the CFbot fairly unhappy.

+1 for proceeding that way. (I did not look at whether the proposed
changes are sane, but I agree that this'll inevitably break a lot of
pending patches.)

Done.

While working on this https://commitfest.postgresql.org/patch/6020/
I discovered that when changing for func/func-aggregate.sgml, the HTML
wasn’t marked for update.

IIUC the doc/Makefile should be updated as attached, right ?

On Mon, Sep 1, 2025, at 7:35 AM, Florents Tselai wrote:

While working on this https://commitfest.postgresql.org/patch/6020/
I discovered that when changing for func/func-aggregate.sgml, the HTML
wasn’t marked for update.

IIUC the doc/Makefile should be updated as attached, right ?

Good catch.

However, your patch doesn't fix all issues. The check target (check-tabs and
check-nbsp) is broken; these targets should also include the func files.

--
Euler Taveira
EDB https://www.enterprisedb.com/

On 1 Sep 2025, at 4:35 PM, Euler Taveira <euler@eulerto.com> wrote:

On Mon, Sep 1, 2025, at 7:35 AM, Florents Tselai wrote:

While working on this https://commitfest.postgresql.org/patch/6020/
I discovered that when changing for func/func-aggregate.sgml, the HTML
wasn’t marked for update.

IIUC the doc/Makefile should be updated as attached, right ?

Good catch.

However, your patch doesn't fix all issues. The check target (check-tabs and
check-nbsp) is broken; these targets should also include the func files.

Ah, you’re right, but then again, I’d expect ALL_SGML to be used
consistently, but it isn't and I didn't check.
v3 does that.
Note that GENERATED_SGML where'te included in these two targets but I think
there's no harm in checking them too.

On 2025-09-01 Mo 11:44 AM, Florents Tselai wrote:

On 1 Sep 2025, at 4:35 PM, Euler Taveira <euler@eulerto.com> wrote:

On Mon, Sep 1, 2025, at 7:35 AM, Florents Tselai wrote:

While working on this https://commitfest.postgresql.org/patch/6020/
I discovered that when changing for func/func-aggregate.sgml, the HTML
wasn’t marked for update.

IIUC the doc/Makefile should be updated as attached, right ?

Good catch.

However, your patch doesn't fix all issues. The check target
(check-tabs and
check-nbsp) is broken; these targets should also include the func files.

Ah, you’re right, but then again,  I’d expect ALL_SGML to be used
consistently, but it isn't and I didn't check.
v3 does that.
Note that GENERATED_SGML where'te included in these two targets but I
think there's no harm in checking them too.

Do we actually care about those? I don't want to add needless cycles
anywhere. I note that the meson.build doesn't appear to have a check
target at all, or anything that looks for hard tabs or nbsps.Those
checks were added to the Makefile back in October in commit 5b7da5c261d,
but that got missed even though Daniel had mentioned it in the
discussion thread.[1]/messages/by-id/F7102912-0BDA-42A3-BDCF-8A4CBD1CC688@yesql.se

cheers

andrew

[1]: /messages/by-id/F7102912-0BDA-42A3-BDCF-8A4CBD1CC688@yesql.se
/messages/by-id/F7102912-0BDA-42A3-BDCF-8A4CBD1CC688@yesql.se

--
Andrew Dunstan
EDB:https://www.enterprisedb.com

On Tue, Sep 2, 2025 at 5:54 PM Andrew Dunstan <andrew@dunslane.net> wrote:

On 2025-09-01 Mo 11:44 AM, Florents Tselai wrote:

On 1 Sep 2025, at 4:35 PM, Euler Taveira <euler@eulerto.com> wrote:

On Mon, Sep 1, 2025, at 7:35 AM, Florents Tselai wrote:

While working on this https://commitfest.postgresql.org/patch/6020/
I discovered that when changing for func/func-aggregate.sgml, the HTML
wasn’t marked for update.

IIUC the doc/Makefile should be updated as attached, right ?

Good catch.

However, your patch doesn't fix all issues. The check target (check-tabs
and
check-nbsp) is broken; these targets should also include the func files.

Ah, you’re right, but then again, I’d expect ALL_SGML to be used
consistently, but it isn't and I didn't check.
v3 does that.
Note that GENERATED_SGML where'te included in these two targets but I
think there's no harm in checking them too.

Do we actually care about those? I don't want to add needless cycles
anywhere. I note that the meson.build doesn't appear to have a check target
at all, or anything that looks for hard tabs or nbsps.Those checks were
added to the Makefile back in October in commit 5b7da5c261d, but that got
missed even though Daniel had mentioned it in the discussion thread.[1]

From the message and discussion in 5b7da5c261d it looks like we do;
and I've seen some messages here and there that people have indeed trouble
applying patches due to spurious whitespace
and special chars.
So I assume the better solution would be having such checks in meson too,

Hi,

On Tue, 2 Sept 2025 at 17:54, Andrew Dunstan <andrew@dunslane.net> wrote:

Ah, you’re right, but then again, I’d expect ALL_SGML to be used consistently, but it isn't and I didn't check.
v3 does that.
Note that GENERATED_SGML where'te included in these two targets but I think there's no harm in checking them too.

Do we actually care about those? I don't want to add needless cycles anywhere. I note that the meson.build doesn't appear to have a check target at all, or anything that looks for hard tabs or nbsps.Those checks were added to the Makefile back in October in commit 5b7da5c261d, but that got missed even though Daniel had mentioned it in the discussion thread.[1]

I have been working on running these checks under the Meson build
system. To do this, I converted the checks into a Perl script
(sgml_syntax_check) and ran it against both the Makefile and Meson.
Test's name is 'sgml_syntax_check' in the Meson. One difference I
noticed: I could not find a way in Meson to create a test that does
not run by default. As a result, this syntax test runs every time you
run the 'meson test'. This behaviour differs from Autoconf, but I
think it is acceptable.

Additionally, some of the CI OSes were missing docbook-xml; but it has
now been installed.

I did not create a new thread for that, I can create one if you think
that it would be better.

CI run with the attached patch applied:
https://cirrus-ci.com/build/6610354173640704

--
Regards,
Nazir Bilal Yavuz
Microsoft

On 2025-09-12 Fr 10:12 AM, Nazir Bilal Yavuz wrote:

Hi,

On Tue, 2 Sept 2025 at 17:54, Andrew Dunstan <andrew@dunslane.net> wrote:

Ah, you’re right, but then again, I’d expect ALL_SGML to be used consistently, but it isn't and I didn't check.
v3 does that.
Note that GENERATED_SGML where'te included in these two targets but I think there's no harm in checking them too.

Do we actually care about those? I don't want to add needless cycles anywhere. I note that the meson.build doesn't appear to have a check target at all, or anything that looks for hard tabs or nbsps.Those checks were added to the Makefile back in October in commit 5b7da5c261d, but that got missed even though Daniel had mentioned it in the discussion thread.[1]

I have been working on running these checks under the Meson build
system.

Thanks for this!

To do this, I converted the checks into a Perl script
(sgml_syntax_check) and ran it against both the Makefile and Meson.
Test's name is 'sgml_syntax_check' in the Meson. One difference I
noticed: I could not find a way in Meson to create a test that does
not run by default. As a result, this syntax test runs every time you
run the 'meson test'. This behaviour differs from Autoconf, but I
think it is acceptable.

Yes, I think so too.

Additionally, some of the CI OSes were missing docbook-xml; but it has
now been installed.

I did not create a new thread for that, I can create one if you think
that it would be better.

CI run with the attached patch applied:
https://cirrus-ci.com/build/6610354173640704

I am away this coming week, will check it out in detail when I return.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

On 2025-09-12 Fr 10:12 AM, Nazir Bilal Yavuz wrote:

Hi,

On Tue, 2 Sept 2025 at 17:54, Andrew Dunstan <andrew@dunslane.net> wrote:

Ah, you’re right, but then again, I’d expect ALL_SGML to be used consistently, but it isn't and I didn't check.
v3 does that.
Note that GENERATED_SGML where'te included in these two targets but I think there's no harm in checking them too.

Do we actually care about those? I don't want to add needless cycles anywhere. I note that the meson.build doesn't appear to have a check target at all, or anything that looks for hard tabs or nbsps.Those checks were added to the Makefile back in October in commit 5b7da5c261d, but that got missed even though Daniel had mentioned it in the discussion thread.[1]

I have been working on running these checks under the Meson build
system. To do this, I converted the checks into a Perl script
(sgml_syntax_check) and ran it against both the Makefile and Meson.
Test's name is 'sgml_syntax_check' in the Meson. One difference I
noticed: I could not find a way in Meson to create a test that does
not run by default. As a result, this syntax test runs every time you
run the 'meson test'. This behaviour differs from Autoconf, but I
think it is acceptable.

Additionally, some of the CI OSes were missing docbook-xml; but it has
now been installed.

I did not create a new thread for that, I can create one if you think
that it would be better.

CI run with the attached patch applied:
https://cirrus-ci.com/build/6610354173640704

Hi Bilal,

This got preempted slightly by Tom's commit 170a8a3f460, but I think
it's worth doing. I tried to simplify it some. See attached. There
doesn't seem to me to be any point in using a different set of files for
the tab tests and the NBSP tests. If we use the same set of files we can
improve the efficiency easily by opening them only once. Here we just
look for all the sgml files and all the xsl files and process them all.

WDYT?

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Andrew Dunstan <andrew@dunslane.net> writes:

On 2025-09-12 Fr 10:12 AM, Nazir Bilal Yavuz wrote:

Test's name is 'sgml_syntax_check' in the Meson. One difference I
noticed: I could not find a way in Meson to create a test that does
not run by default. As a result, this syntax test runs every time you
run the 'meson test'. This behaviour differs from Autoconf, but I
think it is acceptable.

Actually, I've been meaning to complain about the fact that these
checks aren't run by the default Makefile target. I never remember
that there is a separate "check" target, and even if I did remember
it's mostly useless to me because I always want to look at the
rendered HTML. So when I'm working on the docs I always just say
"make" in the doc/src/sgml directory. It'd be helpful, at least to
me, if the default target ran the tabs and nbsp checks. It already
does run xmllint, so that change could probably be integrated with
what you've done here without too much trouble.

This got preempted slightly by Tom's commit 170a8a3f460, but I think
it's worth doing. I tried to simplify it some. See attached. There
doesn't seem to me to be any point in using a different set of files for
the tab tests and the NBSP tests. If we use the same set of files we can
improve the efficiency easily by opening them only once. Here we just
look for all the sgml files and all the xsl files and process them all.

+1 for merging those two checks into one pass, especially if we're
to run them by default.

regards, tom lane

Hi,

On Tue, 30 Sept 2025 at 22:48, Andrew Dunstan <andrew@dunslane.net> wrote:

Hi Bilal,

This got preempted slightly by Tom's commit 170a8a3f460, but I think
it's worth doing. I tried to simplify it some. See attached. There
doesn't seem to me to be any point in using a different set of files for
the tab tests and the NBSP tests. If we use the same set of files we can
improve the efficiency easily by opening them only once. Here we just
look for all the sgml files and all the xsl files and process them all.

WDYT?

It looks good to me. I made 2 changes to your patch:

1- Declaration of $line_no is lost, I re-added it.
2- s/.cirrus.tasks,yml/.cirrus.tasks.yml/ in the commit message.

--
Regards,
Nazir Bilal Yavuz
Microsoft

Hi,

On Wed, 1 Oct 2025 at 15:09, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:

On Tue, 30 Sept 2025 at 22:48, Andrew Dunstan <andrew@dunslane.net> wrote:

Hi Bilal,

This got preempted slightly by Tom's commit 170a8a3f460, but I think
it's worth doing. I tried to simplify it some. See attached. There
doesn't seem to me to be any point in using a different set of files for
the tab tests and the NBSP tests. If we use the same set of files we can
improve the efficiency easily by opening them only once. Here we just
look for all the sgml files and all the xsl files and process them all.

WDYT?

It looks good to me. I made 2 changes to your patch:

1- Declaration of $line_no is lost, I re-added it.
2- s/.cirrus.tasks,yml/.cirrus.tasks.yml/ in the commit message.

Two more minor changes that I missed in the v2:

1- I added $line_no and removed $_ from the tab check's warning
message. I think it is better this way, otherwise if the line only
contains tab character; $_ will print an empty looking line.
2- s/Tabsand/Tabs and/

--
Regards,
Nazir Bilal Yavuz
Microsoft

On 2025-10-01 We 8:27 AM, Nazir Bilal Yavuz wrote:

Hi,

On Wed, 1 Oct 2025 at 15:09, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:

On Tue, 30 Sept 2025 at 22:48, Andrew Dunstan <andrew@dunslane.net> wrote:

Hi Bilal,

This got preempted slightly by Tom's commit 170a8a3f460, but I think
it's worth doing. I tried to simplify it some. See attached. There
doesn't seem to me to be any point in using a different set of files for
the tab tests and the NBSP tests. If we use the same set of files we can
improve the efficiency easily by opening them only once. Here we just
look for all the sgml files and all the xsl files and process them all.

WDYT?

It looks good to me. I made 2 changes to your patch:

1- Declaration of $line_no is lost, I re-added it.
2- s/.cirrus.tasks,yml/.cirrus.tasks.yml/ in the commit message.

Two more minor changes that I missed in the v2:

1- I added $line_no and removed $_ from the tab check's warning
message. I think it is better this way, otherwise if the line only
contains tab character; $_ will print an empty looking line.
2- s/Tabsand/Tabs and/

OK, thanks, looks good. How do we go about doing what Tom wants (i.e.
running the tests by default) under meson. I think in the Makefile we
could just add it to the html target.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Hi,

On Wed, 1 Oct 2025 at 23:02, Andrew Dunstan <andrew@dunslane.net> wrote:

On 2025-10-01 We 8:27 AM, Nazir Bilal Yavuz wrote:

Hi,

On Wed, 1 Oct 2025 at 15:09, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:

On Tue, 30 Sept 2025 at 22:48, Andrew Dunstan <andrew@dunslane.net> wrote:

Hi Bilal,

This got preempted slightly by Tom's commit 170a8a3f460, but I think
it's worth doing. I tried to simplify it some. See attached. There
doesn't seem to me to be any point in using a different set of files for
the tab tests and the NBSP tests. If we use the same set of files we can
improve the efficiency easily by opening them only once. Here we just
look for all the sgml files and all the xsl files and process them all.

WDYT?

It looks good to me. I made 2 changes to your patch:

1- Declaration of $line_no is lost, I re-added it.
2- s/.cirrus.tasks,yml/.cirrus.tasks.yml/ in the commit message.

Two more minor changes that I missed in the v2:

1- I added $line_no and removed $_ from the tab check's warning
message. I think it is better this way, otherwise if the line only
contains tab character; $_ will print an empty looking line.
2- s/Tabsand/Tabs and/

OK, thanks, looks good. How do we go about doing what Tom wants (i.e.
running the tests by default) under meson. I think in the Makefile we
could just add it to the html target.

I might be misunderstanding, but these syntax checks already run by
default under meson build with this patch. Would we just need to add
this test to the HTML target in the Makefile?

--
Regards,
Nazir Bilal Yavuz
Microsoft

On 2025-10-02 Th 2:58 AM, Nazir Bilal Yavuz wrote:

Hi,

On Wed, 1 Oct 2025 at 23:02, Andrew Dunstan <andrew@dunslane.net> wrote:

On 2025-10-01 We 8:27 AM, Nazir Bilal Yavuz wrote:

Hi,

On Wed, 1 Oct 2025 at 15:09, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:

On Tue, 30 Sept 2025 at 22:48, Andrew Dunstan <andrew@dunslane.net> wrote:

Hi Bilal,

This got preempted slightly by Tom's commit 170a8a3f460, but I think
it's worth doing. I tried to simplify it some. See attached. There
doesn't seem to me to be any point in using a different set of files for
the tab tests and the NBSP tests. If we use the same set of files we can
improve the efficiency easily by opening them only once. Here we just
look for all the sgml files and all the xsl files and process them all.

WDYT?

It looks good to me. I made 2 changes to your patch:

1- Declaration of $line_no is lost, I re-added it.
2- s/.cirrus.tasks,yml/.cirrus.tasks.yml/ in the commit message.

Two more minor changes that I missed in the v2:

1- I added $line_no and removed $_ from the tab check's warning
message. I think it is better this way, otherwise if the line only
contains tab character; $_ will print an empty looking line.
2- s/Tabsand/Tabs and/

OK, thanks, looks good. How do we go about doing what Tom wants (i.e.
running the tests by default) under meson. I think in the Makefile we
could just add it to the html target.

I might be misunderstanding, but these syntax checks already run by
default under meson build with this patch. Would we just need to add
this test to the HTML target in the Makefile?

Oh, ok, I missed that about meson. I will adjust the Makefile.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Hi,

On Thu, 2 Oct 2025 at 15:27, Andrew Dunstan <andrew@dunslane.net> wrote:

Oh, ok, I missed that about meson. I will adjust the Makefile.

I think there is one more problem that we need to think about. This
test runs when the xmllint is enabled but it also requires docbook
(docbook-xml on some OSes) to be installed, otherwise the test fails
with 'I/O error : Attempt to load network entity
http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd&#39;. I think that
we need to skip this test if the docbook can not be found in the
system. Otherwise that would be a hassle for most of the people and
buildfarm members. What do you think about this?

--
Regards,
Nazir Bilal Yavuz
Microsoft

On 2025-10-02 Th 8:52 AM, Nazir Bilal Yavuz wrote:

Hi,

On Thu, 2 Oct 2025 at 15:27, Andrew Dunstan<andrew@dunslane.net> wrote:

Oh, ok, I missed that about meson. I will adjust the Makefile.

I think there is one more problem that we need to think about. This
test runs when the xmllint is enabled but it also requires docbook
(docbook-xml on some OSes) to be installed, otherwise the test fails
with 'I/O error : Attempt to load network entity
http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd&#39;. I think that
we need to skip this test if the docbook can not be found in the
system. Otherwise that would be a hassle for most of the people and
buildfarm members. What do you think about this?

Oops, missed seeing this earlier. Yes, I think we need to skip the test
in the meson case. Probably nothing more needed for the Makefile.

cheers

andrew

--
Andrew Dunstan
EDB:https://www.enterprisedb.com

On 01.10.25 22:02, Andrew Dunstan wrote:

(Maybe these discussions could have been in a new thread and not hidden
under some unrelated thing.)

OK, thanks, looks good. How do we go about doing what Tom wants (i.e.
running the tests by default) under meson. I think in the Makefile we
could just add it to the html target.

-html: html-stamp
+html: check html-stamp

This is not a good solution. This means the html target is never up to
date. Compare PostgreSQL 18:

$ make html
make: Nothing to be done for 'html'.
$ make -q html; echo $?
0

And master:

$ make html
perl ...
$ make -q html; echo $?
1

Also, consider the postgres-full.xml target:

# Run validation only once, common to all subsequent targets. While
# we're at it, also resolve all entities (that is, copy all included
# files into one big file). This helps tools that don't understand
# vpath builds (such as dbtoepub).
postgres-full.xml: postgres.sgml $(ALL_SGML)
$(XMLLINT) $(XMLINCLUDE) --output $@ --noent --valid $<

Note that this already does validation. The way this is structured now
is that it runs the validation once when you create postgres-full.xml,
which is than later input into the HTML generation, and then you run the
validation again, on the already-processed input files, which doesn't
make any sense.

I suspect what you're really after here is the functionality of the
check-tabs and check-nbsp targets. So the new Perl script really just
has to cover those two and doesn't have to bother with xmllint. And
then you just call that script as part of the postgres-full.xml target.

Peter Eisentraut <peter@eisentraut.org> writes:

I suspect what you're really after here is the functionality of the
check-tabs and check-nbsp targets. So the new Perl script really just
has to cover those two and doesn't have to bother with xmllint. And
then you just call that script as part of the postgres-full.xml target.

Yeah, that's what I was imagining: replace the xmllint call in
postgres-full.xml with this new script that will also run the
tab/nbsp checks.

regards, tom lane

Hi,

On Thu, 2 Oct 2025 at 21:43, Andrew Dunstan <andrew@dunslane.net> wrote:

On 2025-10-02 Th 8:52 AM, Nazir Bilal Yavuz wrote:

I think there is one more problem that we need to think about. This
test runs when the xmllint is enabled but it also requires docbook
(docbook-xml on some OSes) to be installed, otherwise the test fails
with 'I/O error : Attempt to load network entity
http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd&#39;. I think that
we need to skip this test if the docbook can not be found in the
system. Otherwise that would be a hassle for most of the people and
buildfarm members. What do you think about this?

Oops, missed seeing this earlier. Yes, I think we need to skip the test in the meson case. Probably nothing more needed for the Makefile.

Here is the patch which does that. It has a basic check for the
docbook and if the docbook can not be found, then meson skips the
test.

--
Regards,
Nazir Bilal Yavuz
Microsoft

Hi,

On Thu, 2 Oct 2025 at 23:16, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Peter Eisentraut <peter@eisentraut.org> writes:

I suspect what you're really after here is the functionality of the
check-tabs and check-nbsp targets. So the new Perl script really just
has to cover those two and doesn't have to bother with xmllint. And
then you just call that script as part of the postgres-full.xml target.

Yeah, that's what I was imagining: replace the xmllint call in
postgres-full.xml with this new script that will also run the
tab/nbsp checks.

Does not this mean we can not run the syntax check by itself in the
make builds? If I understand correctly, we need to create
postgres-full.xml each time we want to run the syntax check, right?

I was under the impression that the sgml_syntax_check.pl test would be
a lightweight way to do a syntax check, so that we could easily use it
by itself or in the CI.

--
Regards,
Nazir Bilal Yavuz
Microsoft

On 03.10.25 13:48, Nazir Bilal Yavuz wrote:

On Thu, 2 Oct 2025 at 23:16, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Peter Eisentraut <peter@eisentraut.org> writes:

I suspect what you're really after here is the functionality of the
check-tabs and check-nbsp targets. So the new Perl script really just
has to cover those two and doesn't have to bother with xmllint. And
then you just call that script as part of the postgres-full.xml target.

Yeah, that's what I was imagining: replace the xmllint call in
postgres-full.xml with this new script that will also run the
tab/nbsp checks.

Does not this mean we can not run the syntax check by itself in the
make builds? If I understand correctly, we need to create
postgres-full.xml each time we want to run the syntax check, right?

If you look at this more closely, creating postgres-full.xml and running
the syntax check perform the same operations, except that the latter
throws away the output. So it seems redundant to build a whole new code
path for this. I think you can make the check target dependent on
postgres-full.xml and be done, kind of like this (starting from
pre-b2922562726):

diff --git a/doc/src/sgml/Makefile b/doc/src/sgml/Makefile
index b53b2694a6b..574ae7b3984 100644
--- a/doc/src/sgml/Makefile
+++ b/doc/src/sgml/Makefile
@@ -69,8 +69,12 @@ ALL_IMAGES := $(wildcard $(srcdir)/images/*.svg)
  # files into one big file).  This helps tools that don't understand
  # vpath builds (such as dbtoepub).
  postgres-full.xml: postgres.sgml $(ALL_SGML)
+	$(MAKE) check-tabs check-nbsp
  	$(XMLLINT) $(XMLINCLUDE) --output $@ --noent --valid $<

+# Quick syntax check without style processing
+check: postgres-full.xml
+

##
## Man pages
@@ -195,15 +199,6 @@ MAKEINFO = makeinfo
$(MAKEINFO) --enable-encoding --no-split --no-validate $< -o $@

-##
-## Check
-##
-
-# Quick syntax check without style processing
-check: postgres.sgml $(ALL_SGML) check-tabs check-nbsp
- $(XMLLINT) $(XMLINCLUDE) --noout --valid $<
-
-
##
## Install
##

Peter Eisentraut <peter@eisentraut.org> writes:

If you look at this more closely, creating postgres-full.xml and running
the syntax check perform the same operations, except that the latter
throws away the output. So it seems redundant to build a whole new code
path for this. I think you can make the check target dependent on
postgres-full.xml and be done, kind of like this (starting from
pre-b2922562726):

Would it be unreasonable to discard the "check" target altogether?
It made sense back in the day when actually building the html docs
took many minutes. But I haven't used it in years, so I wonder
if anyone else has either.

regards, tom lane

On 2025-10-03 Fr 10:41 AM, Tom Lane wrote:

Peter Eisentraut<peter@eisentraut.org> writes:

If you look at this more closely, creating postgres-full.xml and running
the syntax check perform the same operations, except that the latter
throws away the output. So it seems redundant to build a whole new code
path for this. I think you can make the check target dependent on
postgres-full.xml and be done, kind of like this (starting from
pre-b2922562726):

Would it be unreasonable to discard the "check" target altogether?
It made sense back in the day when actually building the html docs
took many minutes. But I haven't used it in years, so I wonder
if anyone else has either.

I have no objection. We'll need to work out what we're doing on the
meson side, which is kinda where we came in ...

cheers

andrew

--
Andrew Dunstan
EDB:https://www.enterprisedb.com

Hi,

On Fri, 3 Oct 2025 at 18:47, Andrew Dunstan <andrew@dunslane.net> wrote:

On 2025-10-03 Fr 10:41 AM, Tom Lane wrote:

Peter Eisentraut <peter@eisentraut.org> writes:

If you look at this more closely, creating postgres-full.xml and running
the syntax check perform the same operations, except that the latter
throws away the output. So it seems redundant to build a whole new code
path for this. I think you can make the check target dependent on
postgres-full.xml and be done, kind of like this (starting from
pre-b2922562726):

Would it be unreasonable to discard the "check" target altogether?
It made sense back in the day when actually building the html docs
took many minutes. But I haven't used it in years, so I wonder
if anyone else has either.

I have no objection. We'll need to work out what we're doing on the meson side, which is kinda where we came in ...

I can work on this but I want to clarify it first. Which one do you prefer:

1- We won't have any command to do syntax checks (including tab and
nbsp), these checks will automatically run when we generate docs.

2- We will have a 'check' target but it will only do tab and nbsp
checks; xmllint will run only when generating the docs.

--
Regards,
Nazir Bilal Yavuz
Microsoft

On 06.10.25 10:29, Nazir Bilal Yavuz wrote:

Hi,

On Fri, 3 Oct 2025 at 18:47, Andrew Dunstan <andrew@dunslane.net> wrote:

On 2025-10-03 Fr 10:41 AM, Tom Lane wrote:

Peter Eisentraut <peter@eisentraut.org> writes:

If you look at this more closely, creating postgres-full.xml and running
the syntax check perform the same operations, except that the latter
throws away the output. So it seems redundant to build a whole new code
path for this. I think you can make the check target dependent on
postgres-full.xml and be done, kind of like this (starting from
pre-b2922562726):

Would it be unreasonable to discard the "check" target altogether?
It made sense back in the day when actually building the html docs
took many minutes. But I haven't used it in years, so I wonder
if anyone else has either.

I have no objection. We'll need to work out what we're doing on the meson side, which is kinda where we came in ...

I can work on this but I want to clarify it first. Which one do you prefer:

1- We won't have any command to do syntax checks (including tab and
nbsp), these checks will automatically run when we generate docs.

2- We will have a 'check' target but it will only do tab and nbsp
checks; xmllint will run only when generating the docs.

I don't know, people have a lot of individual workflows, and they are
not reading this thread. I still don't know what we are actually trying
to fix here, I just noticed that what was committed is flawed.

I would prefer that b2922562726 be reverted, and then someone start a
new thread with a descriptive change proposal.

Hi,

On Mon, 6 Oct 2025 at 11:54, Peter Eisentraut <peter@eisentraut.org> wrote:

On 06.10.25 10:29, Nazir Bilal Yavuz wrote:

I can work on this but I want to clarify it first. Which one do you prefer:

1- We won't have any command to do syntax checks (including tab and
nbsp), these checks will automatically run when we generate docs.

2- We will have a 'check' target but it will only do tab and nbsp
checks; xmllint will run only when generating the docs.

I don't know, people have a lot of individual workflows, and they are
not reading this thread. I still don't know what we are actually trying
to fix here, I just noticed that what was committed is flawed.

The problem was meson build doesn't have tab and nbsp checks [1]/messages/by-id/7020df24-1d5f-41e5-8948-2e8d5da57935@dunslane.net. We
were trying to enable these checks on meson build by moving these
checks to the perl script so that we can run this script on both build
systems.

I would prefer that b2922562726 be reverted, and then someone start a
new thread with a descriptive change proposal.

Sounds good to me. I can create a new thread if it gets reverted.

[1]: /messages/by-id/7020df24-1d5f-41e5-8948-2e8d5da57935@dunslane.net

--
Regards,
Nazir Bilal Yavuz
Microsoft

On 2025-10-06 Mo 6:44 AM, Nazir Bilal Yavuz wrote:

Hi,

On Mon, 6 Oct 2025 at 11:54, Peter Eisentraut <peter@eisentraut.org> wrote:

On 06.10.25 10:29, Nazir Bilal Yavuz wrote:

I can work on this but I want to clarify it first. Which one do you prefer:

1- We won't have any command to do syntax checks (including tab and
nbsp), these checks will automatically run when we generate docs.

2- We will have a 'check' target but it will only do tab and nbsp
checks; xmllint will run only when generating the docs.

I don't know, people have a lot of individual workflows, and they are
not reading this thread. I still don't know what we are actually trying
to fix here, I just noticed that what was committed is flawed.

The problem was meson build doesn't have tab and nbsp checks [1]. We
were trying to enable these checks on meson build by moving these
checks to the perl script so that we can run this script on both build
systems.

I would prefer that b2922562726 be reverted, and then someone start a
new thread with a descriptive change proposal.

Sounds good to me. I can create a new thread if it gets reverted.

[1] /messages/by-id/7020df24-1d5f-41e5-8948-2e8d5da57935@dunslane.net

OK, reverted.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

On Fri, Oct 3, 2025 at 10:41:56AM -0400, Tom Lane wrote:

Peter Eisentraut <peter@eisentraut.org> writes:

If you look at this more closely, creating postgres-full.xml and running
the syntax check perform the same operations, except that the latter
throws away the output. So it seems redundant to build a whole new code
path for this. I think you can make the check target dependent on
postgres-full.xml and be done, kind of like this (starting from
pre-b2922562726):

Would it be unreasonable to discard the "check" target altogether?
It made sense back in the day when actually building the html docs
took many minutes. But I haven't used it in years, so I wonder
if anyone else has either.

I run 'make check' on the SGML every time I build the C code.

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com

Do not let urgent matters crowd out time for investment in the future.

On Mon, Oct 6, 2025 at 10:55:53AM -0400, Bruce Momjian wrote:

On Fri, Oct 3, 2025 at 10:41:56AM -0400, Tom Lane wrote:

Peter Eisentraut <peter@eisentraut.org> writes:

If you look at this more closely, creating postgres-full.xml and running
the syntax check perform the same operations, except that the latter
throws away the output. So it seems redundant to build a whole new code
path for this. I think you can make the check target dependent on
postgres-full.xml and be done, kind of like this (starting from
pre-b2922562726):

Would it be unreasonable to discard the "check" target altogether?
It made sense back in the day when actually building the html docs
took many minutes. But I haven't used it in years, so I wonder
if anyone else has either.

I run 'make check' on the SGML every time I build the C code.

Uh, more accurately I run:

make --silent postgres.sgml
make --silent check
make check-tabs

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com

Do not let urgent matters crowd out time for investment in the future.

Bruce Momjian <bruce@momjian.us> writes:

Uh, more accurately I run:

make --silent postgres.sgml
make --silent check
make check-tabs

If we included the tabs/nbsp checks in the normal build, then the
first of those would cover everything. Even as it is, I don't
think the "make check" step is adding anything.

regards, tom lane

On 2025-Oct-03, Tom Lane wrote:

Would it be unreasonable to discard the "check" target altogether?
It made sense back in the day when actually building the html docs
took many minutes. But I haven't used it in years, so I wonder
if anyone else has either.

I wouldn't particularly appreciate that. Doing "make check" takes 0.6
seconds for me, while the HTML build is 28 seconds. It's quite a
difference.

--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
"This is a foot just waiting to be shot" (Andrew Dunstan)

On Mon, Oct 6, 2025 at 11:13:24AM -0400, Tom Lane wrote:

Bruce Momjian <bruce@momjian.us> writes:

Uh, more accurately I run:

make --silent postgres.sgml
make --silent check
make check-tabs

If we included the tabs/nbsp checks in the normal build, then the
first of those would cover everything. Even as it is, I don't
think the "make check" step is adding anything.

Looking at my test code, I do

$ make postgres.sgml
make: Nothing to be done for 'postgres.sgml'.

and my shell comment says it is so configure runs and can check that
works first, but it looks like it now does nothing.

I agree the "make --silent check-tabs" doesn't add anything because that
is already part of 'make check'.

I still would like to run checks without building the HTML.

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com

Do not let urgent matters crowd out time for investment in the future.

On 2025-10-06 Mo 12:00 PM, Álvaro Herrera wrote:

On 2025-Oct-03, Tom Lane wrote:

Would it be unreasonable to discard the "check" target altogether?
It made sense back in the day when actually building the html docs
took many minutes. But I haven't used it in years, so I wonder
if anyone else has either.

I wouldn't particularly appreciate that. Doing "make check" takes 0.6
seconds for me, while the HTML build is 28 seconds. It's quite a
difference.

OK, so I think that one's not going to fly. We could keep the check
target and also run the checks as part of building postgres-full.sgml.

It's less clear to me how to do that in meson, though, since you can
only have a single command in a custom target.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Andrew Dunstan <andrew@dunslane.net> writes:

OK, so I think that one's not going to fly. We could keep the check
target and also run the checks as part of building postgres-full.sgml.

Works for me.

regards, tom lane

Hi,

On 2025-10-07 14:39:44 -0400, Andrew Dunstan wrote:

It's less clear to me how to do that in meson, though, since you can only
have a single command in a custom target.

Create a stamp file for the check success and make that a dependency of
the main build too.

Greetings,

Andres Freund

Hi,

On Mon, 6 Oct 2025 at 13:44, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:

Sounds good to me. I can create a new thread if it gets reverted.

I created a new thread [1]/messages/by-id/CAN55FZ1qzoDcaKqsR3DwE=X6FL+wpm+=KLvH6ahrRXNhjU53DQ@mail.gmail.com and tried to apply recent feedback on this thread.

[1]: /messages/by-id/CAN55FZ1qzoDcaKqsR3DwE=X6FL+wpm+=KLvH6ahrRXNhjU53DQ@mail.gmail.com

--
Regards,
Nazir Bilal Yavuz
Microsoft