fulltext parser strange behave

Started by Pavel Stehuleabout 18 years ago18 messages
#1Pavel Stehule
pavel.stehule@gmail.com

Hello

I am writing tsearch2 wrapper and I testing functionality. I found
some little bit strange on default parser. It can't parse tags with
numbers:

test=# select * from parse('<h1>zluty kun se napil <b>zlute</b> vody</h2>');
tokid | token
-------+-------
12 | <
3 | h1
12 | >
1 | zluty
12 |
1 | kun
12 |
1 | se
12 |
1 | napil
12 |
13 | <b>
1 | zlute
13 | </b>
12 |
1 | vody
12 | < <=====
19 | /h2
12 | > <=====
(19 rows)

It is correct?

Regards
Pavel Stehule

#2Tom Lane
tgl@sss.pgh.pa.us
In reply to: Pavel Stehule (#1)
Re: fulltext parser strange behave

"Pavel Stehule" <pavel.stehule@gmail.com> writes:

I am writing tsearch2 wrapper and I testing functionality. I found
some little bit strange on default parser. It can't parse tags with
numbers:

Well, the state machine definitely thinks that tag names should contain
only ASCII letters (with possibly a leading or trailing '/'). Given the
HTML examples I suppose we should allow non-first digits too. Is there
anything else that should be considered a tag? What about dash and
underscore for instance?

regards, tom lane

#3Andrew Dunstan
andrew@dunslane.net
In reply to: Tom Lane (#2)
Re: fulltext parser strange behave

Tom Lane wrote:

"Pavel Stehule" <pavel.stehule@gmail.com> writes:

I am writing tsearch2 wrapper and I testing functionality. I found
some little bit strange on default parser. It can't parse tags with
numbers:

Well, the state machine definitely thinks that tag names should contain
only ASCII letters (with possibly a leading or trailing '/'). Given the
HTML examples I suppose we should allow non-first digits too. Is there
anything else that should be considered a tag? What about dash and
underscore for instance?

The docs say we specifically accept HTML tags. Are we really just
accepting anything that is a string of ASCII letters as the tag name?
Then we should adjust the docs. <foo> and <foo1234> are not HTML tags.

cheers

andrew

#4Tom Lane
tgl@sss.pgh.pa.us
In reply to: Andrew Dunstan (#3)
Re: fulltext parser strange behave

Andrew Dunstan <andrew@dunslane.net> writes:

Tom Lane wrote:

Well, the state machine definitely thinks that tag names should contain
only ASCII letters (with possibly a leading or trailing '/'). Given the
HTML examples I suppose we should allow non-first digits too. Is there
anything else that should be considered a tag? What about dash and
underscore for instance?

The docs say we specifically accept HTML tags. Are we really just
accepting anything that is a string of ASCII letters as the tag name?
Then we should adjust the docs. <foo> and <foo1234> are not HTML tags.

I don't think I want to try to maintain a list of exactly which
identifiers are considered valid tag names ... and if I did, I wouldn't
put it into the parser. It would be a dictionary's job to tell valid
from invalid tag names, no?

regards, tom lane

#5Andrew Dunstan
andrew@dunslane.net
In reply to: Tom Lane (#4)
Re: fulltext parser strange behave

Tom Lane wrote:

Andrew Dunstan <andrew@dunslane.net> writes:

Tom Lane wrote:

Well, the state machine definitely thinks that tag names should contain
only ASCII letters (with possibly a leading or trailing '/'). Given the
HTML examples I suppose we should allow non-first digits too. Is there
anything else that should be considered a tag? What about dash and
underscore for instance?

The docs say we specifically accept HTML tags. Are we really just
accepting anything that is a string of ASCII letters as the tag name?
Then we should adjust the docs. <foo> and <foo1234> are not HTML tags.

I don't think I want to try to maintain a list of exactly which
identifiers are considered valid tag names ... and if I did, I wouldn't
put it into the parser. It would be a dictionary's job to tell valid
from invalid tag names, no?

I don't have a quarrel with that. But then we should be more clear about
what we are recognizing. We could describe the thing as an HTML-like
tag, possibly. I think the same probably goes for entities too.

cheers

andrew

#6Oleg Bartunov
oleg@sai.msu.su
In reply to: Tom Lane (#4)
Re: fulltext parser strange behave

On Wed, 7 Nov 2007, Tom Lane wrote:

Andrew Dunstan <andrew@dunslane.net> writes:

Tom Lane wrote:

Well, the state machine definitely thinks that tag names should contain
only ASCII letters (with possibly a leading or trailing '/'). Given the
HTML examples I suppose we should allow non-first digits too. Is there
anything else that should be considered a tag? What about dash and
underscore for instance?

The docs say we specifically accept HTML tags. Are we really just
accepting anything that is a string of ASCII letters as the tag name?
Then we should adjust the docs. <foo> and <foo1234> are not HTML tags.

I don't think I want to try to maintain a list of exactly which
identifiers are considered valid tag names ... and if I did, I wouldn't
put it into the parser. It would be a dictionary's job to tell valid
from invalid tag names, no?

it'd be nice to know in dictionary the parser state, but I think it's
too much knowledge for dictionary and the only possibility is to
let <foo1234> pass to dictionary. Currently we have three separate tokens.

regards, tom lane

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

#7Andrew Dunstan
andrew@dunslane.net
In reply to: Andrew Dunstan (#5)
Re: fulltext parser strange behave

Andrew Dunstan wrote:

Tom Lane wrote:

Andrew Dunstan <andrew@dunslane.net> writes:

Tom Lane wrote:

Well, the state machine definitely thinks that tag names should
contain
only ASCII letters (with possibly a leading or trailing '/').
Given the
HTML examples I suppose we should allow non-first digits too. Is
there
anything else that should be considered a tag? What about dash and
underscore for instance?

The docs say we specifically accept HTML tags. Are we really just
accepting anything that is a string of ASCII letters as the tag
name? Then we should adjust the docs. <foo> and <foo1234> are not
HTML tags.

I don't think I want to try to maintain a list of exactly which
identifiers are considered valid tag names ... and if I did, I wouldn't
put it into the parser. It would be a dictionary's job to tell valid
from invalid tag names, no?

I don't have a quarrel with that. But then we should be more clear
about what we are recognizing. We could describe the thing as an
HTML-like tag, possibly. I think the same probably goes for entities too.

I've just been looking at the state machine in wparser_def.c. I think
the processing for entities is also a few bob short in the pound. It
recognises decimal numeric character references, but nor hexadecimal
numeric character references. That's fairly silly since the HTML spec
specifically says the latter are "particularly useful". The rules for
named entities are also deficient w.r.t. digits, just like the case of
tags that Tom noticed. This isn't academic: HTML features a number of
named entities with digits in the name (sup2, frac14 for example).

In XML at least, legal names are defined by the following rules from the
spec:

[4]: NameStartChar ::= ":" | [A-Z] | "_" | [a-z] [4a] NameChar ::= NameStartChar | "-" | "." | [0-9]
[#xC0-#xD6] | [#xD8-#xF6] | [#xF8-#x2FF] | [#x370-#x37D] |
[#x37F-#x1FFF] | [#x200C-#x200D] | [#x2070-#x218F] | [#x2C00-#x2FEF] |
[#x3001-#xD7FF] | [#xF900-#xFDCF] | [#xFDF0-#xFFFD] | [#x10000-#xEFFFF]
[4a] NameChar ::= NameStartChar | "-" | "." | [0-9] |
#xB7 | [#x0300-#x036F] | [#x203F-#x2040]
[5]: Name ::= NameStartChar (NameChar)*

Restricting this to ASCII, we get:

[4]: NameStartChar ::= ":" | [A-Z] | "_" | [a-z] [4a] NameChar ::= NameStartChar | "-" | "." | [0-9]
[4a] NameChar ::= NameStartChar | "-" | "." | [0-9]
[5]: Name ::= NameStartChar (NameChar)*

or this regex for Name:

[A-Za-z:_][A-Za-z0-9:_.-]*

I suggest we use that or something very close to it as the rule for
names in these patterns.

cheers

andrew

#8Tom Lane
tgl@sss.pgh.pa.us
In reply to: Andrew Dunstan (#7)
Re: fulltext parser strange behave

Andrew Dunstan <andrew@dunslane.net> writes:

I've just been looking at the state machine in wparser_def.c. I think
the processing for entities is also a few bob short in the pound. It
recognises decimal numeric character references, but nor hexadecimal
numeric character references. That's fairly silly since the HTML spec
specifically says the latter are "particularly useful". The rules for
named entities are also deficient w.r.t. digits, just like the case of
tags that Tom noticed. This isn't academic: HTML features a number of
named entities with digits in the name (sup2, frac14 for example).

In XML at least, legal names are defined by the following rules from the
spec:
...
[A-Za-z:_][A-Za-z0-9:_.-]*

I suggest we use that or something very close to it as the rule for
names in these patterns.

No objections here. Who wants to patch wparser_def?

regards, tom lane

#9Andrew Dunstan
andrew@dunslane.net
In reply to: Tom Lane (#8)
Re: fulltext parser strange behave

Tom Lane wrote:

Andrew Dunstan <andrew@dunslane.net> writes:

I've just been looking at the state machine in wparser_def.c. I think
the processing for entities is also a few bob short in the pound. It
recognises decimal numeric character references, but nor hexadecimal
numeric character references. That's fairly silly since the HTML spec
specifically says the latter are "particularly useful". The rules for
named entities are also deficient w.r.t. digits, just like the case of
tags that Tom noticed. This isn't academic: HTML features a number of
named entities with digits in the name (sup2, frac14 for example).

In XML at least, legal names are defined by the following rules from the
spec:
...
[A-Za-z:_][A-Za-z0-9:_.-]*

I suggest we use that or something very close to it as the rule for
names in these patterns.

No objections here. Who wants to patch wparser_def?

I can get to it some time in the next week. - rather snowed under right now.

BTW, I'm also suspicious of the clause that allows <?xml ... it appears
that it will allow <?xfoo and <?XFOO also, which seems quite odd,
especially the latter.

cheers

andrew

#10Andrew Dunstan
andrew@dunslane.net
In reply to: Andrew Dunstan (#9)
1 attachment(s)
Re: [HACKERS] fulltext parser strange behave

Andrew Dunstan wrote:

Tom Lane wrote:

Andrew Dunstan <andrew@dunslane.net> writes:

I've just been looking at the state machine in wparser_def.c. I
think the processing for entities is also a few bob short in the
pound. It recognises decimal numeric character references, but nor
hexadecimal numeric character references. That's fairly silly since
the HTML spec specifically says the latter are "particularly
useful". The rules for named entities are also deficient w.r.t.
digits, just like the case of tags that Tom noticed. This isn't
academic: HTML features a number of named entities with digits in
the name (sup2, frac14 for example).

In XML at least, legal names are defined by the following rules from
the spec:
...
[A-Za-z:_][A-Za-z0-9:_.-]*

I suggest we use that or something very close to it as the rule for
names in these patterns.

No objections here. Who wants to patch wparser_def?

I can get to it some time in the next week. - rather snowed under
right now.

BTW, I'm also suspicious of the clause that allows <?xml ... it
appears that it will allow <?xfoo and <?XFOO also, which seems quite
odd, especially the latter.

Here's a patch that fixes the patterns for numeric entities, tag names,
and removes the upper case 'X' case in the special case for an XML
prolog. There are still some oddities, but I decided against making
heroic efforts to fix them. It's probably less important if the patterns
are slightly too liberal (e.g. accepting <a href="qwe<qwe>"> ) than if
they don't recognize what they are alleged to recognize.

cheers

andrew

Attachments:

tsfix.patchtext/x-patch; charset=iso-8859-1; name=tsfix.patchDownload
Index: doc/src/sgml/textsearch.sgml
===================================================================
RCS file: /cvsroot/pgsql/doc/src/sgml/textsearch.sgml,v
retrieving revision 1.36
diff -c -r1.36 textsearch.sgml
*** doc/src/sgml/textsearch.sgml	16 Nov 2007 03:23:07 -0000	1.36
--- doc/src/sgml/textsearch.sgml	19 Nov 2007 13:22:11 -0000
***************
*** 1862,1873 ****
       </row>
       <row>
        <entry><literal>tag</></entry>
!       <entry>HTML tag</entry>
        <entry><literal>&lt;A HREF="dictionaries.html"&gt;</literal></entry>
       </row>
       <row>
        <entry><literal>entity</></entry>
!       <entry>HTML entity</entry>
        <entry><literal>&amp;amp;</literal></entry>
       </row>
       <row>
--- 1862,1873 ----
       </row>
       <row>
        <entry><literal>tag</></entry>
!       <entry>HTML-type tag</entry>
        <entry><literal>&lt;A HREF="dictionaries.html"&gt;</literal></entry>
       </row>
       <row>
        <entry><literal>entity</></entry>
!       <entry>HTML-type entity</entry>
        <entry><literal>&amp;amp;</literal></entry>
       </row>
       <row>
Index: src/backend/tsearch/wparser_def.c
===================================================================
RCS file: /cvsroot/pgsql/src/backend/tsearch/wparser_def.c,v
retrieving revision 1.10
diff -c -r1.10 wparser_def.c
*** src/backend/tsearch/wparser_def.c	15 Nov 2007 22:25:16 -0000	1.10
--- src/backend/tsearch/wparser_def.c	19 Nov 2007 13:22:11 -0000
***************
*** 95,101 ****
  	"Hyphenated word part, all letters",
  	"Hyphenated word part, all ASCII",
  	"Space symbols",
! 	"HTML tag",
  	"Protocol head",
  	"Hyphenated word, letters and digits",
  	"Hyphenated word, all ASCII",
--- 95,101 ----
  	"Hyphenated word part, all letters",
  	"Hyphenated word part, all ASCII",
  	"Space symbols",
! 	"HTML-type tag",
  	"Protocol head",
  	"Hyphenated word, letters and digits",
  	"Hyphenated word, all ASCII",
***************
*** 105,111 ****
  	"Decimal notation",
  	"Signed integer",
  	"Unsigned integer",
! 	"HTML entity"
  };
  
  
--- 105,111 ----
  	"Decimal notation",
  	"Signed integer",
  	"Unsigned integer",
! 	"HTML-type entity"
  };
  
  
***************
*** 136,141 ****
--- 136,143 ----
  	TPS_InHTMLEntity,
  	TPS_InHTMLEntityNumFirst,
  	TPS_InHTMLEntityNum,
+ 	TPS_InHTMLEntityHexNumFirst,
+ 	TPS_InHTMLEntityHexNum,
  	TPS_InHTMLEntityEnd,
  	TPS_InTagFirst,
  	TPS_InXMLBegin,
***************
*** 815,836 ****
  	{p_isEOF, 0, A_POP, TPS_Null, 0, NULL},
  	{p_iseqC, '#', A_NEXT, TPS_InHTMLEntityNumFirst, 0, NULL},
  	{p_isasclet, 0, A_NEXT, TPS_InHTMLEntity, 0, NULL},
  	{NULL, 0, A_POP, TPS_Null, 0, NULL}
  };
  
  static const TParserStateActionItem actionTPS_InHTMLEntity[] = {
  	{p_isEOF, 0, A_POP, TPS_Null, 0, NULL},
! 	{p_isasclet, 0, A_NEXT, TPS_InHTMLEntity, 0, NULL},
  	{p_iseqC, ';', A_NEXT, TPS_InHTMLEntityEnd, 0, NULL},
  	{NULL, 0, A_POP, TPS_Null, 0, NULL}
  };
  
  static const TParserStateActionItem actionTPS_InHTMLEntityNumFirst[] = {
  	{p_isEOF, 0, A_POP, TPS_Null, 0, NULL},
  	{p_isdigit, 0, A_NEXT, TPS_InHTMLEntityNum, 0, NULL},
  	{NULL, 0, A_POP, TPS_Null, 0, NULL}
  };
  
  static const TParserStateActionItem actionTPS_InHTMLEntityNum[] = {
  	{p_isEOF, 0, A_POP, TPS_Null, 0, NULL},
  	{p_isdigit, 0, A_NEXT, TPS_InHTMLEntityNum, 0, NULL},
--- 817,852 ----
  	{p_isEOF, 0, A_POP, TPS_Null, 0, NULL},
  	{p_iseqC, '#', A_NEXT, TPS_InHTMLEntityNumFirst, 0, NULL},
  	{p_isasclet, 0, A_NEXT, TPS_InHTMLEntity, 0, NULL},
+ 	{p_iseqC, ':', A_NEXT, TPS_InHTMLEntity, 0, NULL},
+ 	{p_iseqC, '_', A_NEXT, TPS_InHTMLEntity, 0, NULL},
  	{NULL, 0, A_POP, TPS_Null, 0, NULL}
  };
  
  static const TParserStateActionItem actionTPS_InHTMLEntity[] = {
  	{p_isEOF, 0, A_POP, TPS_Null, 0, NULL},
! 	{p_isalnum, 0, A_NEXT, TPS_InHTMLEntity, 0, NULL},
! 	{p_iseqC, ':', A_NEXT, TPS_InHTMLEntity, 0, NULL},
! 	{p_iseqC, '_', A_NEXT, TPS_InHTMLEntity, 0, NULL},
! 	{p_iseqC, ':', A_NEXT, TPS_InHTMLEntity, 0, NULL},
! 	{p_iseqC, '.', A_NEXT, TPS_InHTMLEntity, 0, NULL},
! 	{p_iseqC, '-', A_NEXT, TPS_InHTMLEntity, 0, NULL},
  	{p_iseqC, ';', A_NEXT, TPS_InHTMLEntityEnd, 0, NULL},
  	{NULL, 0, A_POP, TPS_Null, 0, NULL}
  };
  
  static const TParserStateActionItem actionTPS_InHTMLEntityNumFirst[] = {
  	{p_isEOF, 0, A_POP, TPS_Null, 0, NULL},
+ 	{p_iseqC, 'x', A_NEXT, TPS_InHTMLEntityHexNumFirst, 0, NULL},
  	{p_isdigit, 0, A_NEXT, TPS_InHTMLEntityNum, 0, NULL},
  	{NULL, 0, A_POP, TPS_Null, 0, NULL}
  };
  
+ static const TParserStateActionItem actionTPS_InHTMLEntityHexNumFirst[] = {
+ 	{p_isEOF, 0, A_POP, TPS_Null, 0, NULL},
+ 	{p_isxdigit, 0, A_NEXT, TPS_InHTMLEntityHexNum, 0, NULL},
+ 	{NULL, 0, A_POP, TPS_Null, 0, NULL}
+ };
+ 
  static const TParserStateActionItem actionTPS_InHTMLEntityNum[] = {
  	{p_isEOF, 0, A_POP, TPS_Null, 0, NULL},
  	{p_isdigit, 0, A_NEXT, TPS_InHTMLEntityNum, 0, NULL},
***************
*** 838,843 ****
--- 854,866 ----
  	{NULL, 0, A_POP, TPS_Null, 0, NULL}
  };
  
+ static const TParserStateActionItem actionTPS_InHTMLEntityHexNum[] = {
+ 	{p_isEOF, 0, A_POP, TPS_Null, 0, NULL},
+ 	{p_isxdigit, 0, A_NEXT, TPS_InHTMLEntityHexNum, 0, NULL},
+ 	{p_iseqC, ';', A_NEXT, TPS_InHTMLEntityEnd, 0, NULL},
+ 	{NULL, 0, A_POP, TPS_Null, 0, NULL}
+ };
+ 
  static const TParserStateActionItem actionTPS_InHTMLEntityEnd[] = {
  	{NULL, 0, A_BINGO | A_CLEAR, TPS_Base, HTMLENTITY, NULL}
  };
***************
*** 854,861 ****
  static const TParserStateActionItem actionTPS_InXMLBegin[] = {
  	{p_isEOF, 0, A_POP, TPS_Null, 0, NULL},
  	/* <?xml ... */
  	{p_iseqC, 'x', A_NEXT, TPS_InTag, 0, NULL},
- 	{p_iseqC, 'X', A_NEXT, TPS_InTag, 0, NULL},
  	{NULL, 0, A_POP, TPS_Null, 0, NULL}
  };
  
--- 877,884 ----
  static const TParserStateActionItem actionTPS_InXMLBegin[] = {
  	{p_isEOF, 0, A_POP, TPS_Null, 0, NULL},
  	/* <?xml ... */
+     /* XXX do we wants states for the m and l ?  Right now this accepts <?xZ */
  	{p_iseqC, 'x', A_NEXT, TPS_InTag, 0, NULL},
  	{NULL, 0, A_POP, TPS_Null, 0, NULL}
  };
  
***************
*** 1282,1287 ****
--- 1305,1312 ----
  	TPARSERSTATEACTION(TPS_InHTMLEntity),
  	TPARSERSTATEACTION(TPS_InHTMLEntityNumFirst),
  	TPARSERSTATEACTION(TPS_InHTMLEntityNum),
+ 	TPARSERSTATEACTION(TPS_InHTMLEntityHexNumFirst),
+ 	TPARSERSTATEACTION(TPS_InHTMLEntityHexNum),
  	TPARSERSTATEACTION(TPS_InHTMLEntityEnd),
  	TPARSERSTATEACTION(TPS_InTagFirst),
  	TPARSERSTATEACTION(TPS_InXMLBegin),
Index: src/test/regress/expected/tsearch.out
===================================================================
RCS file: /cvsroot/pgsql/src/test/regress/expected/tsearch.out,v
retrieving revision 1.8
diff -c -r1.8 tsearch.out
*** src/test/regress/expected/tsearch.out	27 Oct 2007 19:03:45 -0000	1.8
--- src/test/regress/expected/tsearch.out	19 Nov 2007 13:22:12 -0000
***************
*** 222,228 ****
      10 | hword_part      | Hyphenated word part, all letters
      11 | hword_asciipart | Hyphenated word part, all ASCII
      12 | blank           | Space symbols
!     13 | tag             | HTML tag
      14 | protocol        | Protocol head
      15 | numhword        | Hyphenated word, letters and digits
      16 | asciihword      | Hyphenated word, all ASCII
--- 222,228 ----
      10 | hword_part      | Hyphenated word part, all letters
      11 | hword_asciipart | Hyphenated word part, all ASCII
      12 | blank           | Space symbols
!     13 | tag             | HTML-type tag
      14 | protocol        | Protocol head
      15 | numhword        | Hyphenated word, letters and digits
      16 | asciihword      | Hyphenated word, all ASCII
***************
*** 232,238 ****
      20 | float           | Decimal notation
      21 | int             | Signed integer
      22 | uint            | Unsigned integer
!     23 | entity          | HTML entity
  (23 rows)
  
  SELECT * FROM ts_parse('default', '345 qwe@efd.r '' http://www.com/ http://aew.werc.ewr/?ad=qwe&dw 1aew.werc.ewr/?ad=qwe&dw 2aew.werc.ewr http://3aew.werc.ewr/?ad=qwe&dw http://4aew.werc.ewr http://5aew.werc.ewr:8100/?  ad=qwe&dw 6aew.werc.ewr:8100/?ad=qwe&dw 7aew.werc.ewr:8100/?ad=qwe&dw=%20%32 +4.0e-10 qwe qwe qwqwe 234.435 455 5.005 teodor@stack.net qwe-wer asdf <fr>qwer jf sdjk<we hjwer <werrwe> ewr1> ewri2 <a href="qwe<qwe>">
--- 232,238 ----
      20 | float           | Decimal notation
      21 | int             | Signed integer
      22 | uint            | Unsigned integer
!     23 | entity          | HTML-type entity
  (23 rows)
  
  SELECT * FROM ts_parse('default', '345 qwe@efd.r '' http://www.com/ http://aew.werc.ewr/?ad=qwe&dw 1aew.werc.ewr/?ad=qwe&dw 2aew.werc.ewr http://3aew.werc.ewr/?ad=qwe&dw http://4aew.werc.ewr http://5aew.werc.ewr:8100/?  ad=qwe&dw 6aew.werc.ewr:8100/?ad=qwe&dw 7aew.werc.ewr:8100/?ad=qwe&dw=%20%32 +4.0e-10 qwe qwe qwqwe 234.435 455 5.005 teodor@stack.net qwe-wer asdf <fr>qwer jf sdjk<we hjwer <werrwe> ewr1> ewri2 <a href="qwe<qwe>">
#11Tom Lane
tgl@sss.pgh.pa.us
In reply to: Andrew Dunstan (#10)
Re: [HACKERS] fulltext parser strange behave

Andrew Dunstan <andrew@dunslane.net> writes:

Here's a patch that fixes the patterns for numeric entities, tag names,
and removes the upper case 'X' case in the special case for an XML
prolog. There are still some oddities, but I decided against making
heroic efforts to fix them. It's probably less important if the patterns
are slightly too liberal (e.g. accepting <a href="qwe<qwe>"> ) than if
they don't recognize what they are alleged to recognize.

I don't approve of the changes to the exposed token type names, but
the state machine changes seem sane first-glance.

regards, tom lane

#12Andrew Dunstan
andrew@dunslane.net
In reply to: Tom Lane (#11)
Re: [HACKERS] fulltext parser strange behave

Tom Lane wrote:

I don't approve of the changes to the exposed token type names, but
the state machine changes seem sane first-glance.

Well, I think it's just plain wrong to describe as HTML tags and
entities things that just aren't. In any case, what I changed was not
the name (or alias, to be more precise), but the exposed description.
The aliases (tag, entity) would remain the same.

cheers

andrew

#13Tom Lane
tgl@sss.pgh.pa.us
In reply to: Andrew Dunstan (#12)
Re: [HACKERS] fulltext parser strange behave

Andrew Dunstan <andrew@dunslane.net> writes:

Tom Lane wrote:

I don't approve of the changes to the exposed token type names, but
the state machine changes seem sane first-glance.

Well, I think it's just plain wrong to describe as HTML tags and
entities things that just aren't.

Maybe, but "HTML-type" is an unhelpful description. Isn't there a more
general markup standard that subsumes both HTML and XML? (I seem to
recall that SGML might be that, but not sure.)

regards, tom lane

#14Andrew Dunstan
andrew@dunslane.net
In reply to: Tom Lane (#13)
Re: [HACKERS] fulltext parser strange behave

Tom Lane wrote:

Andrew Dunstan <andrew@dunslane.net> writes:

Tom Lane wrote:

I don't approve of the changes to the exposed token type names, but
the state machine changes seem sane first-glance.

Well, I think it's just plain wrong to describe as HTML tags and
entities things that just aren't.

Maybe, but "HTML-type" is an unhelpful description. Isn't there a more
general markup standard that subsumes both HTML and XML? (I seem to
recall that SGML might be that, but not sure.)

Most people haven't heard of SGML. I'd settle for "XML tag" or maybe
"XML/HTML tag".

Any other bids?

cheers

andrew

#15Joshua D. Drake
jd@commandprompt.com
In reply to: Andrew Dunstan (#14)
Re: [HACKERS] fulltext parser strange behave

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mon, 19 Nov 2007 11:49:35 -0500
Andrew Dunstan <andrew@dunslane.net> wrote:

Most people haven't heard of SGML. I'd settle for "XML tag" or maybe
"XML/HTML tag".

Any other bids?

XML tag is fine, imo.

Sincerely,

Joshua D. Drake

cheers

andrew

---------------------------(end of
broadcast)--------------------------- TIP 5: don't forget to increase
your free space map settings

- --

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997 http://www.commandprompt.com/
UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHQcFEATb/zqfZUUQRAnpqAKCRPpvG/AQmI5qqkokC1u13gdGw2ACcCC8J
o9F/VjTRIPrLynuJQnJB0L8=
=X0Kn
-----END PGP SIGNATURE-----

#16Peter Eisentraut
peter_e@gmx.net
In reply to: Tom Lane (#13)
Re: [HACKERS] fulltext parser strange behave

Am Montag, 19. November 2007 schrieb Tom Lane:

Maybe, but "HTML-type" is an unhelpful description. Isn't there a more
general markup standard that subsumes both HTML and XML? (I seem to
recall that SGML might be that, but not sure.)

I think "XML tag" would actually cover anything that would be valid as an HTML
tag. (As opposed to the fact that an XML document is not a superset of an
HTML document.)

SGML might be too broad. It would require us to recognize "</>" and "<>" and
perhaps a few other odd things.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

#17Tom Lane
tgl@sss.pgh.pa.us
In reply to: Peter Eisentraut (#16)
Re: [HACKERS] fulltext parser strange behave

Peter Eisentraut <peter_e@gmx.net> writes:

Am Montag, 19. November 2007 schrieb Tom Lane:

Maybe, but "HTML-type" is an unhelpful description. Isn't there a more
general markup standard that subsumes both HTML and XML? (I seem to
recall that SGML might be that, but not sure.)

I think "XML tag" would actually cover anything that would be valid as an HTML
tag.

+1 for "XML tag", then.

regards, tom lane

#18Andrew Dunstan
andrew@dunslane.net
In reply to: Tom Lane (#17)
Re: [HACKERS] fulltext parser strange behave

Tom Lane wrote:

Peter Eisentraut <peter_e@gmx.net> writes:

Am Montag, 19. November 2007 schrieb Tom Lane:

Maybe, but "HTML-type" is an unhelpful description. Isn't there a more
general markup standard that subsumes both HTML and XML? (I seem to
recall that SGML might be that, but not sure.)

I think "XML tag" would actually cover anything that would be valid as an HTML
tag.

+1 for "XML tag", then.

Changed to XML tag and XML entity. Code names adjusted accordingly.
Committed.

cheers

andrew