Regression failure with PostgreSQL 8beta1 and Intel Itanium 2 C compiler

Robert,

Are you planning to use Intel's C compiler in production?
We tried that some time ago and corrupted our database cluster almost
instantly (for some reason we have not investigated any further).
I highly recommend to do some stress testing to see if everything works
nicely.
I'd be pleased to get some feedback.

Regard,

Hans

Robert E. Bruccoleri wrote:

Dear All,
I built PG 8.0 beta1 on an Itanium 2 platform using the Intel compilers
version 8, and got one real difference in the regression tests that affected
int2, int4, union, and numerology. Here's the key difference:

horta postgres 177 > diff -c int4.out ../expected/
*** int4.out    Tue Aug 10 18:41:48 2004
--- ../expected/int4.out        Wed Mar 10 21:11:13 2004
***************
*** 22,27 ****
--- 22,28 ----
INSERT INTO INT4_TBL(f1) VALUES ('   asdf   ');
ERROR:  invalid input syntax for integer: "   asdf   "
INSERT INTO INT4_TBL(f1) VALUES ('- 1234');
+ ERROR:  invalid input syntax for integer: "- 1234"
INSERT INTO INT4_TBL(f1) VALUES ('123       5');
ERROR:  invalid input syntax for integer: "123       5"
INSERT INTO INT4_TBL(f1) VALUES ('');

PG 8.0 beta1 is accepting "- 1234" as a valid integer. Further investigation
reveals that this is a peculiarity of the Intel compilers. The following
program,

#include <stdio.h>
#include <stdlib.h>

main() {
char st[] = "- 1234";
int l;
char *endp;

l = strtol(st, &endp, 10);
printf("l = %d st = %lx endp = %lx\n", l, st, endp);
}

using the Intel compiler provided libraries prints

l = -1234 st = 60000fffffffb720 endp = 60000fffffffb726

whereas gcc and glibc yields

l = 0 st = 60000fffffffb710 endp = 60000fffffffb710

Boo hiss...

+-----------------------------+------------------------------------+
| Robert E. Bruccoleri, Ph.D. | email: bruc@acm.org                |
| President, Congenair LLC    | URL:   http://www.congen.com/~bruc |
| P.O. Box 314                | Phone: 609 818 7251                | 
| Pennington, NJ 08534        |                                    |
+-----------------------------+------------------------------------+

---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

--
Cybertec Geschwinde u Schoenig
Schoengrabern 134, A-2020 Hollabrunn, Austria
Tel: +43/720/10 1234567 or +43/660/816 40 77
www.cybertec.at, www.postgresql.at, kernel.cybertec.at

Your name :
Your email address :

System Configuration
---------------------
Architecture (example: Intel Pentium) : Intel Pentium

Operating System (example: Linux 2.4.18) : Debian unstable
Linux 2.6.6-1-k7

PostgreSQL version (example: PostgreSQL-8.0): PostgreSQL-8.0 CVS HEAD

Compiler used (example: gcc 2.95.2) : gcc 3.3.4

Please enter a FULL description of your problem:
------------------------------------------------

Problems with Turkish locale are widely known to developers.
Another one, now in PL/pgSQL have reared it's ugly head.
Regression tests are failing at triggers, plpgsql, copy2
and rangefuncs. Examienation of regression.diff showed that
the failures were due to unrecognised statements like
BEGIN, RAISE and IF in PL/pgSQL functions. Replacing
capital "I" with lower-case "i" (BEGiN, RAiSE, iF) completely
sloves the problem.

If you know how this problem might be fixed, list the solution below:
---------------------------------------------------------------------

Apparently problem is caused by the following directive:

%option case-insensitive

on line 76 in file src/pl/plpgsql/src/scan.l

flex (flex version 2.5.4) incorporates case-insensitivity in it's
state tables because if I run flex stage with LANG=C everything
works fine. A quick and dirty fix could be implemented by placing

LANG=C
export LANG

in file src/pl/plpgsql/src/Makefile before calling flex.

A long term fix can be done by implementing a function
for keyword lookup like ScanKeywordLookup() in
src/backend/parser/keywords.c.

I would gladly prepare a patch and send it for your consideration
tomorrow morning.

Best regards,
Nicolai Tufar

ntufar <ntufar@pisem.net> writes:

flex (flex version 2.5.4) incorporates case-insensitivity in it's
state tables because if I run flex stage with LANG=C everything
works fine.

Ick. That is of course why it worked for me when I tested it :-(

A quick and dirty fix could be implemented by placing
LANG=C
export LANG
in file src/pl/plpgsql/src/Makefile before calling flex.

This is probably what we'd better do. Otherwise we have
build-context-dependency in the system's behavior, which is bad.

Peter, any thoughts on this one way or the other? At the moment
plpgsql's scan.l seems to be the only use of '%option case-insensitive'
but we have enough flex lexers laying about that I wouldn't be surprised
to have this same risk elsewhere. Is it reasonable to try to force
LANG=C in some global fashion during the build?

regards, tom lane

Greetings,

12-08-2004 Per�embe g�n� saat 18:32 sular�nda, Tom Lane dedi ki:
ntufar <ntufar@pisem.net> writes:

flex (flex version 2.5.4) incorporates case-insensitivity in it's
state tables because if I run flex stage with LANG=C everything
works fine.

Ick. That is of course why it worked for me when I tested it :-(

A quick and dirty fix could be implemented by placing
LANG=C
export LANG
in file src/pl/plpgsql/src/Makefile before calling flex.

This is probably what we'd better do. Otherwise we have
build-context-dependency in the system's behavior, which is bad.

I attached a diff of fix that adds LANG=C; before call to $(FLEX).
Fixes the problem here but I don't know if adding environment variable
assignment like this is appropriate. I am not too fluent in PostgreSQL
build environment and do not know where one can put a global deffinition
you are talking below.

Peter, any thoughts on this one way or the other? At the moment

plpgsql's scan.l seems to be the only use of '%option

case-insensitive'

but we have enough flex lexers laying about that I wouldn't be

surprised

to have this same risk elsewhere. Is it reasonable to try to force
LANG=C in some global fashion during the build?

regards, tom lane

Best regards,
Nicolai Tufar

ntufar <ntufar@pisem.net> writes:

I attached a diff of fix that adds LANG=C; before call to $(FLEX).
Fixes the problem here but I don't know if adding environment variable
assignment like this is appropriate. I am not too fluent in PostgreSQL
build environment and do not know where one can put a global deffinition
you are talking below.

Um, the attachment was unreadable :-( but I get the idea.

As for the global solution, I was wondering if it would work to put
"LANG=C" right inside the definition of $(FLEX). That would ensure
the right behavior from all our flex builds without unnecessarily
messing up people's build environments otherwise. I don't know however
whether this would parse properly.

regards, tom lane

Dear Hans,

Robert,

Are you planning to use Intel's C compiler in production?
We tried that some time ago and corrupted our database cluster almost
instantly (for some reason we have not investigated any further).
I highly recommend to do some stress testing to see if everything works
nicely.
I'd be pleased to get some feedback.

We're using the Intel C compiler for all our code on the Altix. We
have encountered a few bugs with optimization, but on the whole, it
works pretty well, and it generates code that runs about 30% faster
than gcc (depending on the application of course). With regard to
PostgreSQL, we are not having massive problems with data
corruption. We have had problems with 7.4.3, and I'm currently testing
8.0.0beta1. I don't believe the problem with 7.4.3 is a compiler issue --
it only happens when we have multiple backends running at the same time.
Other people like Hubert Froehlich
(http://archives.postgresql.org/pgsql-general/2004-07/msg00670.php)
have run into the same problem using a completely different environment.
I'll report on 8.0.0beta1 when our testing is complete. --Bob

+-----------------------------+------------------------------------+
| Robert E. Bruccoleri, Ph.D. | email: bruc@acm.org                |
| President, Congenair LLC    | URL:   http://www.congen.com/~bruc |
| P.O. Box 314                | Phone: 609 818 7251                | 
| Pennington, NJ 08534        |                                    |
+-----------------------------+------------------------------------+

12-08-2004 Per�embe g�n� saat 22:27 sular�nda, Tom Lane dedi ki:

ntufar <ntufar@pisem.net> writes:

I attached a diff of fix that adds LANG=C; before call to $(FLEX).
Fixes the problem here but I don't know if adding environment variable
assignment like this is appropriate. I am not too fluent in PostgreSQL
build environment and do not know where one can put a global deffinition
you are talking below.

Um, the attachment was unreadable :-( but I get the idea.

Something to do with my mail provider, sorry.
in file src/pl/plpgsql/src/Makefile:
LANG=C;$(FLEX) $(FLEXFLAGS) -Pplpgsql_base_yy -o'$@' $<
instead of
$(FLEX) $(FLEXFLAGS) -Pplpgsql_base_yy -o'$@' $<

As for the global solution, I was wondering if it would work to put
"LANG=C" right inside the definition of $(FLEX). That would ensure
the right behavior from all our flex builds without unnecessarily
messing up people's build environments otherwise. I don't know however
whether this would parse properly.

The only thing that comest in mind is that it may break Win32 port.
Can someone comment on this?

regards, tom lane

Regards,
Nicolai Tufar

Tom Lane wrote:

Peter, any thoughts on this one way or the other? At the moment
plpgsql's scan.l seems to be the only use of '%option
case-insensitive' but we have enough flex lexers laying about that I
wouldn't be surprised to have this same risk elsewhere. Is it
reasonable to try to force LANG=C in some global fashion during the
build?

You'd have to set LC_ALL=C to be really sure to override everything.
But I would stay away from doing that globally, because all the
translation work in gcc and make would go to waste.

I would also suggest that Nicolai report this issue to the flex
developers. It's only bound to reappear everywhere case-insensitive
flex scanners are used.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

ntufar wrote:

Apparently problem is caused by the following directive:

%option case-insensitive

on line 76 in file src/pl/plpgsql/src/scan.l

flex (flex version 2.5.4) incorporates case-insensitivity in it's
state tables because if I run flex stage with LANG=C everything
works fine. A quick and dirty fix could be implemented by placing

LANG=C
export LANG

in file src/pl/plpgsql/src/Makefile before calling flex.

I have tried running flex (2.5.4) with a number of different locales
including tr_TR, but the output file is always the same. Can you show
us a diff of the generated files?

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut <peter_e@gmx.net> writes:

You'd have to set LC_ALL=C to be really sure to override everything.
But I would stay away from doing that globally, because all the
translation work in gcc and make would go to waste.

Agreed. I was toying with changing the FLEX variable to contain
"LC_ALL=C flex" but I'm a bit worried about breaking the build on
some platforms (especially Windows).

I would also suggest that Nicolai report this issue to the flex
developers. It's only bound to reappear everywhere case-insensitive
flex scanners are used.

True. Maybe we should just call it a flex bug and wait for them to
fix it. It's not going to affect builds from tarballs anyway, only
people who build from CVS.

regards, tom lane

Peter Eisentraut <peter_e@gmx.net> writes:

I have tried running flex (2.5.4) with a number of different locales
including tr_TR, but the output file is always the same. Can you show
us a diff of the generated files?

Hmm ... a quick look at the flex sources shows that flex does rely on
the <ctype.h> routines for case-folding, so I have no doubt that
ntufar's report is accurate. Maybe you used the wrong tr_TR locale?

(Just for the record, though, I can't see any change in the generated
pl_scan.c output in any of the tr_TR variants available on either HPUX
or OS X. I don't have a full set of locales installed on my Linux
machine so I can't try it there.)

regards, tom lane

Hi,

On Thu, 12 Aug 2004, Tom Lane wrote:

flex (flex version 2.5.4) incorporates case-insensitivity in it's
state tables because if I run flex stage with LANG=C everything
works fine.

Ick. That is of course why it worked for me when I tested it :-(

Nicolai is on holiday now. I tested on my Fedora Core 2 and RHEL 3 ES
systems and all regression tests passed:

======================
All 96 tests passed.
======================

I'm using the latest tr_TR locale of glibc, and flex-2.5.4a-29 (of RHEL)
and flex-2.5.4a-31 (of FC 2).

What am I missing?

Regards,
--
Devrim GUNDUZ
devrim~gunduz.org devrim.gunduz~linux.org.tr
http://www.tdmsoft.com
http://www.gunduz.org

Devrim GUNDUZ <devrim@gunduz.org> writes:

All 96 tests passed.

I'm using the latest tr_TR locale of glibc, and flex-2.5.4a-29 (of RHEL)
and flex-2.5.4a-31 (of FC 2).

What am I missing?

If you built from a tarball, then the flex run is already done for you.
Remove src/pl/plpgsql/src/pl_scan.c and rebuild to see if you see a
problem.

regards, tom lane

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

On Mon, 16 Aug 2004, Tom Lane wrote:

I'm using the latest tr_TR locale of glibc, and flex-2.5.4a-29 (of RHEL)
and flex-2.5.4a-31 (of FC 2).

What am I missing?

If you built from a tarball, then the flex run is already done for you.
Remove src/pl/plpgsql/src/pl_scan.c and rebuild to see if you see a
problem.

I tried beta1 and latest CVS snapshot before sending the mail. All
produced the same result.

A few minutes before I tried on a Debian unstable, as Nicolai reported.
But all the regression tests passed again, using the latest flex +
glibc... I can't reproduce the problem :( Or there is not a bug :)

Regards,
- --
Devrim GUNDUZ
devrim~gunduz.org devrim.gunduz~linux.org.tr
http://www.tdmsoft.com
http://www.gunduz.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQFBIMRTtl86P3SPfQ4RAhPVAJ9g+UZRyv6SROp9TOzzp2/UsD/W4gCfSKNI
3cdoGjAZ4WLLZHXWs0Wq5Lk=
=1TsT
-----END PGP SIGNATURE-----

Devrim GUNDUZ <devrim@gunduz.org> writes:

A few minutes before I tried on a Debian unstable, as Nicolai reported.
But all the regression tests passed again, using the latest flex +
glibc... I can't reproduce the problem :( Or there is not a bug :)

Hmph. Either Nicolai has a weird locale setting, or he made a mistake.

We'll have to put this on hold until he gets back, I guess. Fortunately
there's still lots of time till release.

regards, tom lane