BUG #15420: Server crash. Segmentation fault when parsing xml file

On Thu, Oct 04, 2018 at 08:57:34AM +0000, PG Bug reporting form wrote:

Steps to reproduce
1. Place this file into PG_DATA directory
https://www.dropbox.com/s/upteflaye9r3fz7/EGRUL_FULL_2018-01-01_X.XML?dl=1

If you can, could you please attach this file to this thread? This is
important for the archives.
--
Michael

Also this query works just fine against 9.4.1 on Windows

And xml_is_well_formed function returns true

postgres=# select xml_is_well_formed(d) from
convert_from(pg_read_binary_file('EGRUL_FULL_2018-01-01_X.XML'),'windows-1251')
g(d);

xml_is_well_formed
--------------------
t

On Thu, Oct 4, 2018 at 2:11 PM Michael Paquier <michael@paquier.xyz> wrote:

On Thu, Oct 04, 2018 at 08:57:34AM +0000, PG Bug reporting form wrote:

Steps to reproduce
1. Place this file into PG_DATA directory

https://www.dropbox.com/s/upteflaye9r3fz7/EGRUL_FULL_2018-01-01_X.XML?dl=1

If you can, could you please attach this file to this thread? This is
important for the archives.
--
Michael

--
--Regards, Sergey Mirvoda

On Thu, Oct 4, 2018 at 2:11 PM Michael Paquier <michael@paquier.xyz> wrote:

If you can, could you please attach this file to this thread? This is
important for the archives.
--
Michael

Looks like it is too big to send uncompressed, here it is in zip archive

--
--Regards, Sergey Mirvoda

Hi

čt 4. 10. 2018 v 12:12 odesílatel Sergey Mirvoda <sergey@mirvoda.com>
napsal:

On Thu, Oct 4, 2018 at 2:11 PM Michael Paquier <michael@paquier.xyz>
wrote:

If you can, could you please attach this file to this thread? This is
important for the archives.
--
Michael

Looks like it is too big to send uncompressed, here it is in zip archive

I am try to import this xml to Postgres with pgimportdoc

https://github.com/okbob/pgimportdoc

and looks like some libxml2 issue.

pgimportdoc: Unexpected result status: PGRES_FATAL_ERROR
pgimportdoc: Error: ERROR: invalid XML content
DETAIL: line 178950: internal error: Huge input lookup
� органе Пенсионного фонда Российской Федер

^
line 178950: attributes construct error

On Thu, Oct 04, 2018 at 12:18:05PM +0200, Pavel Stehule wrote:

čt 4. 10. 2018 v 12:12 odesílatel Sergey Mirvoda <sergey@mirvoda.com>
I am try to import this xml to Postgres with pgimportdoc

https://github.com/okbob/pgimportdoc

and looks like some libxml2 issue.

pgimportdoc: Unexpected result status: PGRES_FATAL_ERROR
pgimportdoc: Error: ERROR: invalid XML content
DETAIL: line 178950: internal error: Huge input lookup
� органе Пенсионного фонда Российской Федер

^
line 178950: attributes construct error

Sergey, what is the version of libxml2 bundled with the Windows
installer and Ubuntu? You should normally be able to see which version
it is on Windows by looking at the version number of the libxml2 dll...
However libxml2 upstream is not smart enough to produce a version file
if I recall correctly (please note last time I looked at that I had to
add to the upstream tree a custom patch to associate a correct version
to the produced dll, making sure that minor MSI upgrades were able to
update the version of libxml2 bundled in a Postgres MSI installer I
lately maintain).
--
Michael

čt 4. 10. 2018 v 12:18 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

Hi

čt 4. 10. 2018 v 12:12 odesílatel Sergey Mirvoda <sergey@mirvoda.com>
napsal:

On Thu, Oct 4, 2018 at 2:11 PM Michael Paquier <michael@paquier.xyz>
wrote:

If you can, could you please attach this file to this thread? This is
important for the archives.
--
Michael

Looks like it is too big to send uncompressed, here it is in zip archive

I am try to import this xml to Postgres with pgimportdoc

https://github.com/okbob/pgimportdoc

and looks like some libxml2 issue.

pgimportdoc: Unexpected result status: PGRES_FATAL_ERROR
pgimportdoc: Error: ERROR: invalid XML content
DETAIL: line 178950: internal error: Huge input lookup
� органе Пенсионного фонда Российской Федер

^
line 178950: attributes construct error

I checked Sergey's example, and it doesn't crash on Linux - The error is
displayed correctly. Looks like MS Windows issue of libxml2

postgres=# select xml_is_well_formed(d) from
convert_from(pg_read_binary_file('error.xml'),'windows-1251') g(d);
┌────────────────────┐
│ xml_is_well_formed │
╞════════════════════╡
│ f │
└────────────────────┘
(1 row)

This issue can be enforced by relatively new libxml2 limits

https://mail.gnome.org/archives/commits-list/2012-August/msg00645.html

Unfortunately, default configuration uses xmlParseBalancedChunkMemory for
parsing content, and this function cannot to get option like

XML_PARSE_HUGE

So it is hard to fix it.

Regards

Pavel

чт, 4 окт. 2018, 16:20 Pavel Stehule <pavel.stehule@gmail.com>:

čt 4. 10. 2018 v 12:18 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

Hi

čt 4. 10. 2018 v 12:12 odesílatel Sergey Mirvoda <sergey@mirvoda.com>
napsal:

On Thu, Oct 4, 2018 at 2:11 PM Michael Paquier <michael@paquier.xyz>
wrote:

If you can, could you please attach this file to this thread? This is
important for the archives.
--
Michael

Looks like it is too big to send uncompressed, here it is in zip archive

I am try to import this xml to Postgres with pgimportdoc

https://github.com/okbob/pgimportdoc

and looks like some libxml2 issue.

pgimportdoc: Unexpected result status: PGRES_FATAL_ERROR
pgimportdoc: Error: ERROR: invalid XML content
DETAIL: line 178950: internal error: Huge input lookup
� органе Пенсионного фонда Российской Федер

^
line 178950: attributes construct error

I checked Sergey's example, and it doesn't crash on Linux - The error is
displayed correctly. Looks like MS Windows issue of libxml2

postgres=# select xml_is_well_formed(d) from
convert_from(pg_read_binary_file('error.xml'),'windows-1251') g(d);
┌────────────────────┐
│ xml_is_well_formed │
╞════════════════════╡
│ f │
└────────────────────┘
(1 row)

This issue can be enforced by relatively new libxml2 limits

https://mail.gnome.org/archives/commits-list/2012-August/msg00645.html

Unfortunately, default configuration uses xmlParseBalancedChunkMemory for
parsing content, and this function cannot to get option like

XML_PARSE_HUGE

So it is hard to fix it.

Regards

Pavel

Actually we found this error in very fresh intatallation of Ubuntu 16.04
and postgres 10.5
After that we checked every configuration we have.
And only postgres 9.4 works as expected.

Additionally Andrey just reproduced this on his dev box.

--Regards, Sergey Mirvoda

čt 4. 10. 2018 v 13:20 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

čt 4. 10. 2018 v 12:18 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

Hi

čt 4. 10. 2018 v 12:12 odesílatel Sergey Mirvoda <sergey@mirvoda.com>
napsal:

On Thu, Oct 4, 2018 at 2:11 PM Michael Paquier <michael@paquier.xyz>
wrote:

If you can, could you please attach this file to this thread? This is
important for the archives.
--
Michael

Looks like it is too big to send uncompressed, here it is in zip archive

I am try to import this xml to Postgres with pgimportdoc

https://github.com/okbob/pgimportdoc

and looks like some libxml2 issue.

pgimportdoc: Unexpected result status: PGRES_FATAL_ERROR
pgimportdoc: Error: ERROR: invalid XML content
DETAIL: line 178950: internal error: Huge input lookup
� органе Пенсионного фонда Российской Федер

^
line 178950: attributes construct error

I checked Sergey's example, and it doesn't crash on Linux - The error is
displayed correctly. Looks like MS Windows issue of libxml2

postgres=# select xml_is_well_formed(d) from
convert_from(pg_read_binary_file('error.xml'),'windows-1251') g(d);
┌────────────────────┐
│ xml_is_well_formed │
╞════════════════════╡
│ f │
└────────────────────┘
(1 row)

This issue can be enforced by relatively new libxml2 limits

https://mail.gnome.org/archives/commits-list/2012-August/msg00645.html

Unfortunately, default configuration uses xmlParseBalancedChunkMemory for
parsing content, and this function cannot to get option like

XML_PARSE_HUGE

So it is hard to fix it.

It probably requires refactoring of parsing xml like
http://xmlsoft.org/examples/parse4.c

Regards

Pavel

Hi!

4 окт. 2018 г., в 16:35, Sergey Mirvoda <sergey@mirvoda.com> написал(а):

Additionally Andrey just reproduced this on his dev box.

Yep, crashes on my laptop with MacOS and libxml2/2.9.7 from brew on current master.

Best regards, Andrey Borodin.

Actually we found this error in very fresh intatallation of Ubuntu 16.04

and postgres 10.5
After that we checked every configuration we have.
And only postgres 9.4 works as expected.

This issue is related to libxml2 limits - and it cannot to work with modern
libxml2 libraries.

Pavel

čt 4. 10. 2018 v 12:18 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

Hi

čt 4. 10. 2018 v 12:12 odesílatel Sergey Mirvoda <sergey@mirvoda.com>
napsal:

On Thu, Oct 4, 2018 at 2:11 PM Michael Paquier <michael@paquier.xyz>
wrote:

If you can, could you please attach this file to this thread? This is
important for the archives.
--
Michael

Looks like it is too big to send uncompressed, here it is in zip archive

I am try to import this xml to Postgres with pgimportdoc

https://github.com/okbob/pgimportdoc

and looks like some libxml2 issue.

pgimportdoc: Unexpected result status: PGRES_FATAL_ERROR
pgimportdoc: Error: ERROR: invalid XML content
DETAIL: line 178950: internal error: Huge input lookup
� органе Пенсионного фонда Российской Федер

^
line 178950: attributes construct error

probably it is same issue like

/messages/by-id/CA+cVU8M2FuK3XrHRSypW1y3gk+xgGjAgrT0k+sLVWsRJsW=3+A@mail.gmail.com

Regards

Pavel

4 окт. 2018 г., в 16:38, Pavel Stehule <pavel.stehule@gmail.com> написал(а):

Actually we found this error in very fresh intatallation of Ubuntu 16.04 and postgres 10.5
After that we checked every configuration we have.
And only postgres 9.4 works as expected.

This issue is related to libxml2 limits - and it cannot to work with modern libxml2 libraries.

Yes, root cause is inside libxml2 code.

Can we protect postmaster from crashing from libxml2 error? There is a bunch of PG_TRY there, but it does not help.

Best regards, Andrey Borodin.

čt 4. 10. 2018 v 13:43 odesílatel Andrey Borodin <x4mmm@yandex-team.ru>
napsal:

4 окт. 2018 г., в 16:38, Pavel Stehule <pavel.stehule@gmail.com>
написал(а):

Actually we found this error in very fresh intatallation of Ubuntu 16.04

and postgres 10.5
After that we checked every configuration we have.
And only postgres 9.4 works as expected.

This issue is related to libxml2 limits - and it cannot to work with
modern libxml2 libraries.

Yes, root cause is inside libxml2 code.

Can we protect postmaster from crashing from libxml2 error? There is a
bunch of PG_TRY there, but it does not help.

Unfortunately, no. You cannot to handle crash. PostgreSQL doesn't start
separate process for libxml2 calls, and fault there is fatal.

Regards

Pavel

čt 4. 10. 2018 v 13:47 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

čt 4. 10. 2018 v 13:43 odesílatel Andrey Borodin <x4mmm@yandex-team.ru>
napsal:

4 окт. 2018 г., в 16:38, Pavel Stehule <pavel.stehule@gmail.com>
написал(а):

Actually we found this error in very fresh intatallation of Ubuntu 16.04

and postgres 10.5
After that we checked every configuration we have.
And only postgres 9.4 works as expected.

This issue is related to libxml2 limits - and it cannot to work with
modern libxml2 libraries.

Yes, root cause is inside libxml2 code.

Can we protect postmaster from crashing from libxml2 error? There is a
bunch of PG_TRY there, but it does not help.

Unfortunately, no. You cannot to handle crash. PostgreSQL doesn't start
separate process for libxml2 calls, and fault there is fatal.

I played with it, and it looks on some problems with libxml2 and your
specific document (maybe too much multibyte chars, .. I don't know)

I imported 200MB long xml document with 1M items. So it has not sense to
limit xml size of PostgreSQL side.

It looks so your xml document hits some corner case of libxml2 where it is
extremely memory expensive. What I can see, there is lot of long content
inside attributes.

Regards

Pavel

чт, 4 окт. 2018, 19:03 Pavel Stehule <pavel.stehule@gmail.com>:

čt 4. 10. 2018 v 13:47 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

čt 4. 10. 2018 v 13:43 odesílatel Andrey Borodin <x4mmm@yandex-team.ru>
napsal:

4 окт. 2018 г., в 16:38, Pavel Stehule <pavel.stehule@gmail.com>
написал(а):

Actually we found this error in very fresh intatallation of Ubuntu 16.04

and postgres 10.5
After that we checked every configuration we have.
And only postgres 9.4 works as expected.

This issue is related to libxml2 limits - and it cannot to work with
modern libxml2 libraries.

Yes, root cause is inside libxml2 code.

Can we protect postmaster from crashing from libxml2 error? There is a
bunch of PG_TRY there, but it does not help.

Unfortunately, no. You cannot to handle crash. PostgreSQL doesn't start
separate process for libxml2 calls, and fault there is fatal.

I played with it, and it looks on some problems with libxml2 and your
specific document (maybe too much multibyte chars, .. I don't know)

I imported 200MB long xml document with 1M items. So it has not sense to
limit xml size of PostgreSQL side.

It looks so your xml document hits some corner case of libxml2 where it is
extremely memory expensive. What I can see, there is lot of long content
inside attributes.

Regards

Pavel, thank you for your interest.
It is definitely something inside this document.

Actually we loaded about 10k different documents like this one. About 10Gb
of content and crash is only on this one.

But every other parser we tried (.net, Java, python) handled this just
fine.

For now we ended with custom plpython function for parsing xml and this is
slow as hell.

This is looks like regression, pg 9.4 load this document without any
problem.

On 2018-Oct-04, Sergey Mirvoda wrote:

чт, 4 окт. 2018, 19:03 Pavel Stehule <pavel.stehule@gmail.com>:

Pavel, thank you for your interest.
It is definitely something inside this document.

Actually we loaded about 10k different documents like this one. About 10Gb
of content and crash is only on this one.

It's probably a good idea to report this to libxml2 then.

--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

чт, 4 окт. 2018, 19:31 Alvaro Herrera <alvherre@2ndquadrant.com>:

On 2018-Oct-04, Sergey Mirvoda wrote:

чт, 4 окт. 2018, 19:03 Pavel Stehule <pavel.stehule@gmail.com>:

Pavel, thank you for your interest.
It is definitely something inside this document.

Actually we loaded about 10k different documents like this one. About

10Gb

of content and crash is only on this one.

It's probably a good idea to report this to libxml2 then.

--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Sure, but bug is mostly about unhandled server crash. Is it normal?

Also, as far as I understand Pavel, lbxml2 interface somehow changed for
handling 'huge' documents, but postgres don't handle this correctly.

čt 4. 10. 2018 v 16:42 odesílatel Sergey Mirvoda <sergey@mirvoda.com>
napsal:

чт, 4 окт. 2018, 19:31 Alvaro Herrera <alvherre@2ndquadrant.com>:

On 2018-Oct-04, Sergey Mirvoda wrote:

чт, 4 окт. 2018, 19:03 Pavel Stehule <pavel.stehule@gmail.com>:

Pavel, thank you for your interest.
It is definitely something inside this document.

Actually we loaded about 10k different documents like this one. About

10Gb

of content and crash is only on this one.

It's probably a good idea to report this to libxml2 then.

--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Sure, but bug is mostly about unhandled server crash. Is it normal?

Nobody can handle process crash. This should be fixed on libxml2 side.

Also, as far as I understand Pavel, lbxml2 interface somehow changed for
handling 'huge' documents, but postgres don't handle this correctly.

This can be fixed only partially - the authors of libxml2 introduced new
limit and new option, but there is not possible apply new option on all
API.

If I understand well to this behave - libxml2 implemented new limits as
safeguards against strange attacker documents. Is it question if it is good
idea disable this safeguards by default. Second issue is impossibility to
set this option for functionality that we use from libxml2 - and there are
not alternatives. I have not any idea, how these issues can be fixed on
postgresql side.

The mentioned limit is 10 000 000 bytes - if you will generate documents
less than this size, then probably don't hit this issue.

Regards

Pavel

On Thu, Oct 04, 2018 at 05:02:05PM +0200, Pavel Stehule wrote:

čt 4. 10. 2018 v 16:42 odesílatel Sergey Mirvoda <sergey@mirvoda.com>
napsal:

Sure, but bug is mostly about unhandled server crash. Is it normal?

Nobody can handle process crash. This should be fixed on libxml2 side.

If some code crashes in glibc or within a system call, impacting
Postgres backends, there is usually little to do, and the correct call
is to fix the place where the problem happens. If libxml2 has
integrated with a new API which is considered safer to use, then we
could consider working on improving Postgres regarding that aspect.
--
Michael