UPDATE Query problem

"Josh Berkus" <josh@agliodbs.com> writes:

The problem is that I cannot figure out a subselect that will allow me to
select the last complete history record prior to the one being
updated.

Sure you can. You can't alias history in the UPDATE, but you can alias
it in the subselect, so:

UPDATE history SET fieldA =
(SELECT fieldA FROM history older
WHERE older.key = history.key AND
older.fieldA IS NOT NULL AND older.fieldB IS NOT NULL AND
older.timestamp =
(SELECT max(timestamp) FROM history oldest
WHERE oldest.key = history.key AND
oldest.fieldA IS NOT NULL AND oldest.fieldB IS NOT NULL)),
fieldB = (SELECT fieldB FROM ... repeat entire subselect above ...)
WHERE (history.fieldA IS NULL or
history.fieldB IS NULL);

This will work and (AFAIK) is fully SQL-compliant, but it will be
slower than the dickens because of all those subselects :-(. Might
be tolerable if the key field is near-unique and is indexed, but
heaven help you if not.

To further hamper things, for portability reasons, I can use neither SELECT
DISTINCT ON nor custom functions.

Too bad. SELECT DISTINCT ON would let you get rid of the bottom SELECT
max() and would let you exploit an index on (key,timestamp). By the
time the query above finishes running, very likely you could talk your
boss into accepting a nonstandard solution ;-)

Also, just because PG can handle the above doesn't mean every RDBMS does
(do I need to name names?). What products do you really need it to
be portable to?

regards, tom lane

On Thu, 17 Jan 2002, Josh Berkus wrote:

Folks,

I have a database that contains a chronological journal of activity. For
various reasons, this journal contains both complete and incomplete records,
and while all records are timestamped, the primary key is not strictly ordered
by timestamp.

What I want to do is update each incomplete record with the contents of the
last previous complete record. As a simple-minded test case:

CREATE TABLE history AS (
history_id SERIAL PRIMARY KEY,
period_date TIMESTAMP,
fieldA VARCHAR(30),
fieldB INT4 );

CREATE VIEW complete_history_records AS
SELECT history.*
FROM history WHERE fieldA IS NOT NULL
and fieldB IS NOT NULL

UPDATE history SET fieldA = chr.fieldA
fieldB = chr.fieldB
FROM (SELECT complete_history_records.*
WHERE ??? ) chr
WHERE (history.fieldA IS NULL or
history.fieldB IS NULL);

The problem is that I cannot figure out a subselect that will allow me to
select the last complete history record prior to the one being updated. It
seems like I need to reference a field in the main query in the subselect,
which can't be done.

To further hamper things, for portability reasons, I can use neither SELECT
DISTINCT ON nor custom functions.

Isn't update...from already an extension?

In any case, is performance really important? I think something like:
update history set fieldA=chr.fieldA, fieldB=chr.fieldB from
complete_history_records chr where (history.fieldA is null or
history.fieldB is null) and chr.period_date=(select max(period_date)
from complete_history_records where period_date<history.period_date);

might work if really slow.

Tom,

Sure you can. You can't alias history in the UPDATE, but you can alias
it in the subselect, so:

UPDATE history SET fieldA =
(SELECT fieldA FROM history older
WHERE older.key = history.key AND
older.fieldA IS NOT NULL AND older.fieldB IS NOT NULL AND
older.timestamp =
(SELECT max(timestamp) FROM history oldest
WHERE oldest.key = history.key AND
oldest.fieldA IS NOT NULL AND oldest.fieldB IS NOT NULL)),
fieldB = (SELECT fieldB FROM ... repeat entire subselect above ...)
WHERE (history.fieldA IS NULL or
history.fieldB IS NULL);

Interesting. however, it appears to give me the most recent record with
non-NULL values. What I want is the most recent record with non-NULL values
*before* the record I'm trying to update. In other words, if I have the
following data:

history
id timestamp fieldA fieldB
1341 6/30/00 KCKG 1
1345 7/31/00 KC 1
1402 8/31/00 NULL NULL
2799 9/30/00 NULL NULL
1581 10/31/00 KC 2
1673 11/30/00 KC 2

I want records 1402 and 2799 to be updated from record 1345, not from record
1673.

This will work and (AFAIK) is fully SQL-compliant, but it will be
slower than the dickens because of all those subselects :-(. Might
be tolerable if the key field is near-unique and is indexed, but
heaven help you if not.

The key field is unique. And slow is OK ... the history-correction program
runs overnight. I just can't afford to take a procedural approach and correct
one record at a time ... there are 200,000 records and growing at a rate of
8,000 records per month.

Also, just because PG can handle the above doesn't mean every RDBMS does
(do I need to name names?). What products do you really need it to
be portable to?

Yeah, you guessed it ... MS SQL Server 7. Which kills custom functions or
custom aggregates, something that would make this whole process a lot easier.

Thanks for the help!

-Josh Berkus

______AGLIO DATABASE SOLUTIONS___________________________
Josh Berkus
Complete information technology josh@agliodbs.com
and data management solutions (415) 565-7293
for law firms, small businesses fax 621-2533
and non-profit organizations. San Francisco

"Josh Berkus" <josh@agliodbs.com> writes:

Interesting. however, it appears to give me the most recent record with
non-NULL values. What I want is the most recent record with non-NULL values
*before* the record I'm trying to update.

Oh, I'm sorry: forgot the extra qualification on the innermost SELECT:

AND oldest.timestamp < history.timestamp

Yeah, you guessed it ... MS SQL Server 7.

I dunno, how good is SQL Server on subselects?

regards, tom lane

Tom,

Oh, I'm sorry: forgot the extra qualification on the innermost SELECT:

AND oldest.timestamp < history.timestamp

Hmmm ... I'll try both solutions tommorrow. That is, I'll see if they port
across databases ...

Yeah, you guessed it ... MS SQL Server 7.

I dunno, how good is SQL Server on subselects?

Not very good. A lot of stuff, like subselects in the SELECT line, is not
supported. And MS has gotten further from the SQL standard with each update
since SQL Server 7.0 ...

-Josh

______AGLIO DATABASE SOLUTIONS___________________________
Josh Berkus
Complete information technology josh@agliodbs.com
and data management solutions (415) 565-7293
for law firms, small businesses fax 621-2533
and non-profit organizations. San Francisco

I try this code

CREATE FUNCTION pltclu_call_handler() RETURNS OPAQUE AS
'/usr/lib/postgresql/pltcl.so' LANGUAGE 'C';

and get this error

ERROR: Load of file /usr/lib/postgresql/pltcl.so failed:
/usr/lib/postgresql/pltcl.so: undefined symbol: pg_get_enconv_by_encoding

anyone help me with this?

murray hobbs

ps here's my .configure command

./configure --enable-multibyte=UNICODE --enable-unicode-conversion --enable-locale --bindir=/usr/local/bin --libdir=/usr/lib --includedir=/usr/include --mandir=/usr/local/man --with-tcl --enable-odbc --with-unixodbc --enable-syslog

Murray Prior Hobbs <murray@efone.com> writes:

ERROR: Load of file /usr/lib/postgresql/pltcl.so failed:
/usr/lib/postgresql/pltcl.so: undefined symbol: pg_get_enconv_by_encoding

Looks like a multibyte-enabled pltcl and a non-multibyte-enabled
backend. Given that they were clearly built at different times and
with different configurations, one might also wonder if they're even
the same Postgres version.

regards, tom lane

you are so right

bugga, and i thought i was being both so clever and so careful

m

Tom Lane wrote:

Tom, Stephan,

Well, you'll be interested to know that Stephan's solution worked for both
PostgreSQL and MS SQL Server ... as far as parsing goes. On PostgreSQL, the
query took 14 minutes to complete.

On MS SQL Server, it never completed at all. Looks like I will have to take a
semi-procedural approach with MS SQL Server after all. Just another evidence
of the superiority of Postgres ...

-Josh Berkus

______AGLIO DATABASE SOLUTIONS___________________________
Josh Berkus
Complete information technology josh@agliodbs.com
and data management solutions (415) 565-7293
for law firms, small businesses fax 621-2533
and non-profit organizations. San Francisco

after i have succesfully used createlan script to load both the trusted and untrusted tlc languages i try some tests

i create a test function (right out of the docs)

CREATE FUNCTION tcl_max (int4, int4) RETURNS int4 AS '
if {$1 > $2} {return $1}
return $2
' LANGUAGE 'pltclu';

and i try to run this stest

select tcl_max(4,6);

but i get

ERROR: fmgr_info: function 17020: cache lookup failed

so i create trusted version

CREATE FUNCTION tcl_max (int4, int4) RETURNS int4 AS '
if {$1 > $2} {return $1}
return $2
' LANGUAGE 'pltcl';

and i again try to run this stest

select tcl_max(4,6);

but i get instead

server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
connection to server was lost

and if i look at the log

postgres: murray kale [local] SELECT: error while loading shared libraries: /usr/lib/postgresql/pltcl.so: undefined symbol: Tcl_CreateInterp
DEBUG: server process (pid 18415) exited with exit code 127
DEBUG: terminating any other active server processes
DEBUG: all server processes terminated; reinitializing shared memory and semaphores
DEBUG: database system was interrupted at 2002-01-19 15:01:29 EST
DEBUG: checkpoint record is at 0/4BAD10
DEBUG: redo record is at 0/4BAD10; undo record is at 0/0; shutdown TRUE
DEBUG: next transaction id: 2120; next oid: 49324
DEBUG: database system was not properly shut down; automatic recovery in progress
DEBUG: redo starts at 0/4BAD50
DEBUG: ReadRecord: record with zero length at 0/4C0FB4
DEBUG: redo done at 0/4C0F90
DEBUG: database system is ready

so what do i do now?

log it as a bug?

murray

so ok i go to the sources looking for test of pl/tlc or pl/tlcu (untrusted)

Tom Lane wrote:

Murray Prior Hobbs <murray@efone.com> writes:

postgres: murray kale [local] SELECT: error while loading shared libraries: /usr/lib/postgresql/pltcl.so: undefined symbol: Tcl_CreateInterp
DEBUG: server process (pid 18415) exited with exit code 127

Kinda looks like your dynamic loader doesn't know where to find
libtcl.so (and thinks that the appropriate way to fail is a hard exit(),
which is not my idea of friendly).

so what do i do now?

log it as a bug?

It's not a Postgres bug; you need to twiddle your shlib configuration.
But since you didn't mention your platform, I can't offer any pointers
beyond
http://www.ca.postgresql.org/users-lounge/docs/7.1/postgres/install-post.html#AEN11747
(note that this is talking about finding Postgres' libraries; alter
to suit wherever libtcl lives).

regards, tom lane

Thanks tom but i think there's more to it

error while loading shared libraries: /usr/lib/postgresql/pltcl.so: undefined symbol: Tcl_CreateInterp

as you can see it knows where the library is - what i think it's
complaining about is the undefined symbol

so i do a grep through the sources and find the call - the only call -
but there's no function declaration in the sources

i did follow your link and i had read the page before - i'm on RedHat
7.2 so should not have needed to do that - but i did anyway - it made no
difference

is there meant to be Tcl_CreateInterp anywhere in the sources?

murray

Tom Lane wrote:

[2002-01-19 18:58] Murray Prior Hobbs said:
|
| Thanks tom but i think there's more to it
|
| error while loading shared libraries: /usr/lib/postgresql/pltcl.so:
| undefined symbol: Tcl_CreateInterp
|
| as you can see it knows where the library is - what i think it's
| complaining about is the undefined symbol
|
| so i do a grep through the sources and find the call - the only call -
| but there's no function declaration in the sources
|
| i did follow your link and i had read the page before - i'm on RedHat
| 7.2 so should not have needed to do that - but i did anyway - it made no
| difference
|
| is there meant to be Tcl_CreateInterp anywhere in the sources?

No. This is provided by the tcl library:

bash$ grep Tcl_CreateInter /usr/include/tcl8.3/tclDecls.h
EXTERN Tcl_Interp * Tcl_CreateInterp _ANSI_ARGS_((void));

The problem is, as Tom said, that your tcl library is not being
found by the system's linker. Try this:

bash$ ldd /usr/lib/postgresql/pltcl.so

I suspect you'll see a line containing "not found".

brent

--
"Develop your talent, man, and leave the world something. Records are
really gifts from people. To think that an artist would love you enough
to share his music with anyone is a beautiful thing." -- Duane Allman

i have had no trouble loading and using the pgpgsql language - and it
lives in exactly the same place

i've done as you suggested though - here is the output

[murray@localhost dbSources]$ ldd /usr/lib/postgresql/pltcl.so
libdl.so.2 => /lib/libdl.so.2 (0x40020000)
libm.so.6 => /lib/i686/libm.so.6 (0x40024000)
libc.so.6 => /lib/i686/libc.so.6 (0x40047000)
/lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x80000000)

murray

Brent Verner wrote:

No. This is provided by the tcl library:

bash$ grep Tcl_CreateInter /usr/include/tcl8.3/tclDecls.h
EXTERN Tcl_Interp * Tcl_CreateInterp _ANSI_ARGS_((void));

The problem is, as Tom said, that your tcl library is not being
found by the system's linker. Try this:

bash$ ldd /usr/lib/postgresql/pltcl.so

I suspect you'll see a line containing "not found".

brent

[2002-01-19 18:58] Murray Prior Hobbs said:
|
| Thanks tom but i think there's more to it
|
| error while loading shared libraries: /usr/lib/postgresql/pltcl.so:
| undefined symbol: Tcl_CreateInterp
|
| as you can see it knows where the library is - what i think it's
| complaining about is the undefined symbol
|
| so i do a grep through the sources and find the call - the only call -
| but there's no function declaration in the sources
|
| i did follow your link and i had read the page before - i'm on RedHat
| 7.2 so should not have needed to do that - but i did anyway - it made no
| difference
|
| is there meant to be Tcl_CreateInterp anywhere in the sources?

maybe this is a dumb question

but are not all the tcl sources part of the source distribution?

like - am i to assume then that there are binaries in the distribution
for which there is no code?

like - if i go looking for somethng in the code should not i find it?

murray

Brent Verner wrote:

No. This is provided by the tcl library:

bash$ grep Tcl_CreateInter /usr/include/tcl8.3/tclDecls.h
EXTERN Tcl_Interp * Tcl_CreateInterp _ANSI_ARGS_((void));

The problem is, as Tom said, that your tcl library is not being
found by the system's linker. Try this:

bash$ ldd /usr/lib/postgresql/pltcl.so

I suspect you'll see a line containing "not found".

brent

[2002-01-19 18:58] Murray Prior Hobbs said:
|
| Thanks tom but i think there's more to it
|
| error while loading shared libraries: /usr/lib/postgresql/pltcl.so:
| undefined symbol: Tcl_CreateInterp
|
| as you can see it knows where the library is - what i think it's
| complaining about is the undefined symbol
|
| so i do a grep through the sources and find the call - the only call -
| but there's no function declaration in the sources
|
| i did follow your link and i had read the page before - i'm on RedHat
| 7.2 so should not have needed to do that - but i did anyway - it made no
| difference
|
| is there meant to be Tcl_CreateInterp anywhere in the sources?

Murray Prior Hobbs <murray@efone.com> writes:

maybe this is a dumb question
but are not all the tcl sources part of the source distribution?

I'm only going to say this one more time: Tcl is not part of Postgres.

pltcl depends on libtcl (note difference), and the loader is evidently
not finding libtcl.so. Which will absolutely NOT be in
/usr/lib/postgresql. The question for you is where it actually lives
(if it's installed at all), and next why the dynamic loader search path
isn't finding it.

regards, tom lane

ok ok - so i went off and read a LOT about Tcl - cool what a nice tool

and then i downloaded the tcl/tk sources and built with appropriate
configure options and installed it - in approprate places (in /usr/bin -
over the old copies that were already there) and ran the tests and
dowloaded some TCl samples and played with the apps

as i had been using the postgres 7.2 beta i reinstalled 7.13 over the
top - right from the start, reconfigured, recompiled, reinstalled,
reinitialised

and tried to call a tcl function yet again - but now i get this error

ERROR: pltcl: internal error - cannot create 'normal' interpreter

but hmm, that's further than i got before and at least the database does
not restart itself in the process

and in the code i have got one step further

any clues?

murray

if ((pltcl_hold_interp = Tcl_CreateInterp()) == NULL)
{
elog(ERROR, "pltcl: internal error - cannot create 'hold' "
"interpreter");
}

/************************************************************
* Create the two interpreters
************************************************************/
if ((pltcl_norm_interp =
Tcl_CreateSlave(pltcl_hold_interp, "norm", 0)) == NULL)
{
elog(ERROR,
"*pltcl: internal error - cannot create 'normal' interpreter*");
}

murray

Tom Lane wrote:

[2002-01-19 19:40] Murray Prior Hobbs said:
|
| i have had no trouble loading and using the pgpgsql language - and it
| lives in exactly the same place
|
| i've done as you suggested though - here is the output

Indeed. I just got finished installing a chroot image of
redhat-7.2 to test this. I am seeing the same Tcl_CreateInterp
problem you mentioned earlier. The pltcl language does not work
even from the 7.2b3 rpms. Can someone verify that pltcl works on
their stock redhat 7.2 system?

Are there a known bugs in the stock 7.2 binutils or any other part
of the toolchain that might be causing this problem? Most notably
is the absence of pltcl.so being linked to libtcl.so. Could this
be a problem with redhat's tcl package?

Monty, are you by chance running in a chroot?

confounded,
brent

--
"Develop your talent, man, and leave the world something. Records are
really gifts from people. To think that an artist would love you enough
to share his music with anyone is a beautiful thing." -- Duane Allman

[2002-01-20 23:24] Murray Prior Hobbs said:
| Brent Verner wrote:
|
| >[2002-01-19 19:40] Murray Prior Hobbs said:
| >|
| >| i have had no trouble loading and using the pgpgsql language - and it
| >| lives in exactly the same place
| >|
| >| i've done as you suggested though - here is the output
| >
| >Indeed. I just got finished installing a chroot image of
| >redhat-7.2 to test this. I am seeing the same Tcl_CreateInterp
| >problem you mentioned earlier. The pltcl language does not work
| >even from the 7.2b3 rpms. Can someone verify that pltcl works on
| >their stock redhat 7.2 system?
| >
| >Are there a known bugs in the stock 7.2 binutils or any other part
| >of the toolchain that might be causing this problem? Most notably
| >is the absence of pltcl.so being linked to libtcl.so. Could this
| >be a problem with redhat's tcl package?
| >
| >Monty, are you by chance running in a chroot?
| >
| if you mean me (Murray) nope - it's a bog standard RedHat 7.2 install

sorry! I know a guy named "Monty Hobbs"... I'm really too tired ;-)

| but i have installed Tcl from the sources from scratch - 8.3.4

Indeed I've tracked the problem down to the line that links
the pltcl.so library:

make[3]: Entering directory `/usr/local/cvs/pgsql/src/pl/tcl'
/bin/sh mkMakefile.tcldefs.sh '/usr/lib/tclConfig.sh' 'Makefile.tcldefs'
make[3]: Leaving directory `/usr/local/cvs/pgsql/src/pl/tcl'
make[3]: Entering directory `/usr/local/cvs/pgsql/src/pl/tcl'
gcc -pipe -O -D__NO_STRING_INLINES -D__NO_MATH_INLINES -fPIC -I../../../src/include -DHAVE_UNISTD_H=1 -DHAVE_LIMITS_H=1 -DHAVE_GETCWD=1 -DHAVE_OPENDIR=1 -DHAVE_STRSTR=1 -DHAVE_STRTOL=1 -DHAVE_TMPNAM=1 -DHAVE_WAITPID=1 -DHAVE_UNISTD_H=1 -DHAVE_SYS_PARAM_H=1 -DUSE_TERMIOS=1 -DHAVE_SYS_TIME_H=1 -DTIME_WITH_SYS_TIME=1 -DHAVE_TM_ZONE=1 -DHAVE_TM_GMTOFF=1 -DHAVE_TIMEZONE_VAR=1 -DHAVE_ST_BLKSIZE=1 -DSTDC_HEADERS=1 -DNEED_MATHERR=1 -DHAVE_SIGNED_CHAR=1 -DHAVE_SYS_IOCTL_H=1 -c -o pltcl.o pltcl.c
gcc -pipe -shared -Wl,-soname,libtcl.so.0 -o pltcl.so pltcl.o -L/usr/lib -ltcl -ldl -lieee -lm -lc
^^^^^^^^^^^

IIRC, this was changed to workaround another problem with the
tcl client library having name conflicts. This value (TCL_SHLIB_LD)
comes directly from the /usr/lib/tclConfig.sh file supplied by the
rpm. You can add the following line to src/pl/tcl/Makefile
below "-include Makefile.tcldefs"

TCL_SHLIB_LD = gcc -shared

to override the erronious value supplied by the system's tclConfig.sh.

| but just because i'm ignorant of many things - how would i check if i
| was running in chroot environment?

not sure. I always know when I am, because I setup the chroot.
Some web hosts will give you a chroot as well, but if you are
developing on your own workstation, there is little chance of
you being in a chroot and not knowing it.

hth.
b

--
"Develop your talent, man, and leave the world something. Records are
really gifts from people. To think that an artist would love you enough
to share his music with anyone is a beautiful thing." -- Duane Allman

Brent Verner <brent@rcfile.org> writes:

I am seeing the same Tcl_CreateInterp
problem you mentioned earlier. The pltcl language does not work
even from the 7.2b3 rpms. Can someone verify that pltcl works on
their stock redhat 7.2 system?

Hmm, what Tcl version are you using? pltcl does not appear broken
on my system, but I think what I've got installed is Tcl 8.0.5.

regards, tom lane

Brent Verner <brent@rcfile.org> writes:

Can someone verify that pltcl works on
their stock redhat 7.2 system?

Indeed it does not. On a straight-from-the-CD RH 7.2 install and
CVS-tip Postgres, I see both of the behaviors Murray complained of.

What I think is particularly nasty is that we get an exit(127) when
the symbol resolution fails, leading to database restart. This will
probably happen on *most* systems not only Linux, because we are
specifying RTLD_LAZY in our dlopen() calls, meaning that missing
symbols should be flagged when they are referenced at runtime --- and
if we call a function that should be there and isn't, there's not much
the dynamic loader can do except throw a signal or exit().

What we should be doing is specifying RTLD_NOW to dlopen(), so that
any unresolved symbol failure occurs during dlopen(), when we are
prepared to deal with it in a clean fashion.

I ran into this same behavior years ago on HPUX and fixed it by using
what they call BIND_IMMEDIATE mode; but I now see that most of the
other ports are specifying RTLD_LAZY, and thus have this problem.

Unless I hear a credible counter-argument, I am going to change
RTLD_LAZY to RTLD_NOW in src/backend/port/dynloader/linux.h. I have
tested that and it produces a clean error with no backend crash.

What I would *like* to do is make the same change in all the
port/dynloader files that reference RTLD_LAZY:
src/backend/port/dynloader/aix.h
src/backend/port/dynloader/bsdi.h
src/backend/port/dynloader/dgux.h
src/backend/port/dynloader/freebsd.h
src/backend/port/dynloader/irix5.h
src/backend/port/dynloader/linux.h
src/backend/port/dynloader/netbsd.h
src/backend/port/dynloader/openbsd.h
src/backend/port/dynloader/osf.h
src/backend/port/dynloader/sco.h
src/backend/port/dynloader/solaris.h
src/backend/port/dynloader/svr4.h
src/backend/port/dynloader/univel.h
src/backend/port/dynloader/unixware.h
src/backend/port/dynloader/win.h
However I'm a bit scared to do that at this late stage of the release
cycle, because perhaps some of these platforms don't support the full
dlopen() API. Comments? Can anyone test whether RTLD_NOW works on
any of the above-mentioned ports?

regards, tom lane

However I'm a bit scared to do that at this late stage of the release
cycle, because perhaps some of these platforms don't support the full
dlopen() API. Comments? Can anyone test whether RTLD_NOW works on
any of the above-mentioned ports?

I can confirm that RTLD_NOW exists on BSD/OS. Can we do:

#ifdef RTLD_NOW
use RTLD_NOW
#else
whatever_is_there_now
#endif

in those ports at least for 7.2 so we can be sure we don't get failures.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Brent Verner <brent@rcfile.org> writes:

Indeed I've tracked the problem down to the line that links
the pltcl.so library:

gcc -pipe -shared -Wl,-soname,libtcl.so.0 -o pltcl.so pltcl.o -L/usr/lib -ltcl -ldl -lieee -lm -lc
^^^^^^^^^^^

Yeah, removing the "-Wl,-soname,libtcl.so.0" switch produces a correctly
functioning pltcl.

IIRC, this was changed to workaround another problem with the
tcl client library having name conflicts. This value (TCL_SHLIB_LD)
comes directly from the /usr/lib/tclConfig.sh file supplied by the
rpm.

I seem to recall that this same problem was being debated a few weeks
back, but apparently we didn't actually do anything about it. Looks
like we have to.

Peter, didn't you have a proposal on the table to fix this?

regards, tom lane

Murray Prior Hobbs <murray@efone.com> writes:

Brent sent me a fix

----start Bent's fix ----

rpm. You can add the following line to src/pl/tcl/Makefile
below "-include Makefile.tcldefs"

TCL_SHLIB_LD = gcc -shared

---- end Brent's fix -----

which i tried but did not work

Works for me. Did you remove the pltcl.so file so it would be rebuilt?

regards, tom lane

Tom Lane writes:

Unless I hear a credible counter-argument, I am going to change
RTLD_LAZY to RTLD_NOW in src/backend/port/dynloader/linux.h. I have
tested that and it produces a clean error with no backend crash.

What I would *like* to do is make the same change in all the
port/dynloader files that reference RTLD_LAZY:

RTLD_LAZY allows you to load shared library modules that contain circular
references. I don't know if that's useful or just stupid, but on some
systems the shared library models are pretty, um, different so that the
need for this might arise from time to time.

However I'm a bit scared to do that at this late stage of the release
cycle, because perhaps some of these platforms don't support the full
dlopen() API. Comments? Can anyone test whether RTLD_NOW works on
any of the above-mentioned ports?

I really don't think this is a good change to make now, as we don't know
how well all of this is supported, and the failure scenario is annoying
but not really that harmful.

--
Peter Eisentraut peter_e@gmx.net

Tom Lane writes:

Peter, didn't you have a proposal on the table to fix this?

Yeah, complain loudly to whoever dared to package a broken Tcl like
that... Or we'll work with Andreas Zeugwetter's patches and eliminate the
use of tclConfig.sh mostly.

--
Peter Eisentraut peter_e@gmx.net

Peter Eisentraut <peter_e@gmx.net> writes:

Peter, didn't you have a proposal on the table to fix this?

Yeah, complain loudly to whoever dared to package a broken Tcl like
that... Or we'll work with Andreas Zeugwetter's patches and eliminate the
use of tclConfig.sh mostly.

Yeah, I was taking a second look at Andreas' patch myself. At the time,
the report was that we were only seeing a failure in RPM-packaged
Postgres and so I thought that the root problem was somewhere in our RPM
script. However, I have now tried it for myself and can confirm that
we fail in a Postgres source build too. The bogus soname switch might
be blamable on the Tcl RPM package and not on Tcl sources, but that
doesn't make a lot of difference to us either way.

I'm still quite nervous about making these changes so late in the cycle.
OTOH I suspect Andreas was right: we haven't been getting any pltcl
portability testing from our beta testers. If it's broken now, we
can hardly make it worse.

regards, tom lane

Peter Eisentraut <peter_e@gmx.net> writes:

Tom Lane writes:

Unless I hear a credible counter-argument, I am going to change
RTLD_LAZY to RTLD_NOW in src/backend/port/dynloader/linux.h. I have
tested that and it produces a clean error with no backend crash.

RTLD_LAZY allows you to load shared library modules that contain circular
references.

Does that not work with RTLD_NOW? I should think it would. In any
case, I'm doubtful that we care.

I really don't think this is a good change to make now, as we don't know
how well all of this is supported, and the failure scenario is annoying
but not really that harmful.

A database restart is always very bad news in my mind. You might be
right that it's too risky to make such a change now for 7.2, but I still
absolutely want to do it for 7.3.

regards, tom lane

Tom Lane writes:

I'm still quite nervous about making these changes so late in the cycle.
OTOH I suspect Andreas was right: we haven't been getting any pltcl
portability testing from our beta testers.

This logic can also be reversed: We haven't been getting any beta testing
from users of Red Hat 7.1.

If it's broken now, we can hardly make it worse.

You can surely make things a lot worse for those that are using other
operating systems. I certainly don't agree with making changes just
because Red Hat blew it.

--
Peter Eisentraut peter_e@gmx.net

Peter Eisentraut <peter_e@gmx.net> writes:

You can surely make things a lot worse for those that are using other
operating systems. I certainly don't agree with making changes just
because Red Hat blew it.

It does appear that the problem can be blamed entirely on the RPM
packaging of Tcl. I tried configuring from source on RHL 7.2, and
neither tcl 8.3.2 nor 8.3.4 produce a "soname" switch in TCL_SHLIB_LD.
In fact, grep can't find any occurrence of "soname" anywhere in the
Tcl source distribution.

Nonetheless, I'm not sure that "do nothing" is an acceptable response
on our part.

I tried setting up pltcl's makefile to dike out the offending switch:

override TCL_SHLIB_LD := $(patsubst %soname%, , $(TCL_SHLIB_LD))

but could not get it to work --- gmake's pattern matching logic seems
to be too brain-dead to cope with more than one % in a pattern. And

override TCL_SHLIB_LD := $(patsubst -Wl,-soname%, , $(TCL_SHLIB_LD))

doesn't work either; apparently there's no way to escape the comma.
Anyone know a cute hack to get gmake to do this?

regards, tom lane

[2002-01-20 17:52] Tom Lane said:
| Peter Eisentraut <peter_e@gmx.net> writes:
| > You can surely make things a lot worse for those that are using other
| > operating systems. I certainly don't agree with making changes just
| > because Red Hat blew it.
|
| It does appear that the problem can be blamed entirely on the RPM
| packaging of Tcl. I tried configuring from source on RHL 7.2, and
| neither tcl 8.3.2 nor 8.3.4 produce a "soname" switch in TCL_SHLIB_LD.
| In fact, grep can't find any occurrence of "soname" anywhere in the
| Tcl source distribution.
|
| Nonetheless, I'm not sure that "do nothing" is an acceptable response
| on our part.

Agreed. I think working around this borkenness in the Makefile is
the best solution; I don't think switching from RTLD_LAZY is good
right now.

| I tried setting up pltcl's makefile to dike out the offending switch:
|
| override TCL_SHLIB_LD := $(patsubst %soname%, , $(TCL_SHLIB_LD))
|
| but could not get it to work --- gmake's pattern matching logic seems
| to be too brain-dead to cope with more than one % in a pattern. And
|
| override TCL_SHLIB_LD := $(patsubst -Wl,-soname%, , $(TCL_SHLIB_LD))
|
| doesn't work either; apparently there's no way to escape the comma.
| Anyone know a cute hack to get gmake to do this?

It seems that substvar operates on each " " separated token in the
string. The following works for me.

override TCL_SHLIB_LD := $(shell echo $(TCL_SHLIB_LD) | sed 's/-Wl,-soname.*//')

cheers.
brent

--
"Develop your talent, man, and leave the world something. Records are
really gifts from people. To think that an artist would love you enough
to share his music with anyone is a beautiful thing." -- Duane Allman

Brent Verner <brent@rcfile.org> writes:

It seems that substvar operates on each " " separated token in the
string. The following works for me.

override TCL_SHLIB_LD := $(shell echo $(TCL_SHLIB_LD) | sed 's/-Wl,-soname.*//')

I suspect that the above works only because -Wl,-soname is the last
switch in TCL_SHLIB_LD; any following switches would be removed too.
Perhaps better

override TCL_SHLIB_LD := $(shell echo $(TCL_SHLIB_LD) | sed 's/-Wl,-soname[^ ]*//'

regards, tom lane

[2002-01-20 19:16] Tom Lane said:
| Brent Verner <brent@rcfile.org> writes:
| > It seems that substvar operates on each " " separated token in the
| > string. The following works for me.
|
| > override TCL_SHLIB_LD := $(shell echo $(TCL_SHLIB_LD) | sed 's/-Wl,-soname.*//')
|
| I suspect that the above works only because -Wl,-soname is the last
| switch in TCL_SHLIB_LD; any following switches would be removed too.
| Perhaps better
|
| override TCL_SHLIB_LD := $(shell echo $(TCL_SHLIB_LD) | sed 's/-Wl,-soname[^ ]*//'

Yes, much better.

cheers.
brent

--
"Develop your talent, man, and leave the world something. Records are
really gifts from people. To think that an artist would love you enough
to share his music with anyone is a beautiful thing." -- Duane Allman

Murray Prior Hobbs <murray@efone.com> writes:

tell me oh mighty guru's
what linux distribution could i use to make me a happy happy man

Just apply this patch and RHL should work.

*** src/pl/tcl/Makefile.orig	Sat Oct 13 00:23:50 2001
--- src/pl/tcl/Makefile	Sun Jan 20 21:57:45 2002
***************
*** 49,54 ****
--- 49,58 ----
  endif
  endif

+ # Suppress bogus soname switch that RedHat RPMs put into tclConfig.sh
+ override TCL_SHLIB_LD := $(shell echo "$(TCL_SHLIB_LD)" | sed 's/-Wl,-soname[^ ]*//')
+ 
+ 
  %$(TCL_SHLIB_SUFFIX): %.o
  	$(TCL_SHLIB_LD) -o $@ $< $(TCL_LIB_SPEC) $(SHLIB_EXTRA_LIBS)

regards, tom lane

Tom Lane writes:

Just apply this patch and RHL should work.

I'm OK with this patch. (Although you don't need the override.)

We should file a bug report with Red Hat, methinks.

*** src/pl/tcl/Makefile.orig	Sat Oct 13 00:23:50 2001
--- src/pl/tcl/Makefile	Sun Jan 20 21:57:45 2002
***************
*** 49,54 ****
--- 49,58 ----
endif
endif

+ # Suppress bogus soname switch that RedHat RPMs put into tclConfig.sh
+ override TCL_SHLIB_LD := $(shell echo "$(TCL_SHLIB_LD)" | sed 's/-Wl,-soname[^ ]*//')
+
+
%$(TCL_SHLIB_SUFFIX): %.o
$(TCL_SHLIB_LD) -o $@ $< $(TCL_LIB_SPEC) $(SHLIB_EXTRA_LIBS)

--
Peter Eisentraut peter_e@gmx.net

Peter Eisentraut <peter_e@gmx.net> writes:

I'm OK with this patch. (Although you don't need the override.)

Okay, committed. (I left in the override; it can't hurt can it?)

We should file a bug report with Red Hat, methinks.

Trond, I think this is your turf ... is it a bug?

regards, tom lane

On Sun, Jan 20, 2002 at 01:40:17PM -0500, Tom Lane wrote:

What I would *like* to do is make the same change in all the
port/dynloader files that reference RTLD_LAZY:
src/backend/port/dynloader/openbsd.h

I can't speak for other platforms but openbsd only has RTLD_LAZY.

--
David Terrell | "... a grandiose, wasteful drug war that will never
dbt@meat.net | be won as long as so many Americans need to
Nebcorp Prime Minister | anesthetize themselves to get through the day."
http://wwn.nebcorp.com/ | -Camille Paglia

On Sun, Jan 20, 2002 at 01:40:17PM -0500, Tom Lane wrote:

What I would *like* to do is make the same change in all the
port/dynloader files that reference RTLD_LAZY:
src/backend/port/dynloader/openbsd.h

I can't speak for other platforms but openbsd only has RTLD_LAZY.

FreeBSD supports both:

RTLD_LAZY Each external function reference is resolved when the func-
tion is first called.

RTLD_NOW All external function references are bound immediately by
dlopen().

RTLD_LAZY is normally preferred, for reasons of efficiency. However,
RTLD_NOW is useful to ensure that any undefined symbols are discovered

Chris

Christopher Kings-Lynne wrote:

On Sun, Jan 20, 2002 at 01:40:17PM -0500, Tom Lane wrote:

What I would *like* to do is make the same change in all the
port/dynloader files that reference RTLD_LAZY:
src/backend/port/dynloader/openbsd.h

I can't speak for other platforms but openbsd only has RTLD_LAZY.

FreeBSD supports both:

RTLD_LAZY Each external function reference is resolved when the func-
tion is first called.

RTLD_NOW All external function references are bound immediately by
dlopen().

RTLD_LAZY is normally preferred, for reasons of efficiency. However,
RTLD_NOW is useful to ensure that any undefined symbols are discovered

Interesting LAZY has better efficiency. Seems we should just keep LAZY
as our default for future releases and tell people if they link to bad
object files, they should expect trouble.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Bruce Momjian writes:

Interesting LAZY has better efficiency. Seems we should just keep LAZY
as our default for future releases and tell people if they link to bad
object files, they should expect trouble.

In practice, we load object files only if we call the function, so symbol
resolution happens either way briefly after loading. RTLD_NOW includes
some overhead because it checks symbols that we might not end up needing,
but for the typical PostgreSQL extension module, that should really not
matter.

--
Peter Eisentraut peter_e@gmx.net

Peter Eisentraut wrote:

Bruce Momjian writes:

Interesting LAZY has better efficiency. Seems we should just keep LAZY
as our default for future releases and tell people if they link to bad
object files, they should expect trouble.

In practice, we load object files only if we call the function, so symbol
resolution happens either way briefly after loading. RTLD_NOW includes
some overhead because it checks symbols that we might not end up needing,
but for the typical PostgreSQL extension module, that should really not
matter.

OK, I was just throwing out the point in case it was significant.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Interesting LAZY has better efficiency.

"Efficiency" by what measure? I would think that batch resolution of
symbols would be faster than taking a trap for each one.

Seems we should just keep LAZY
as our default for future releases and tell people if they link to bad
object files, they should expect trouble.

(a) How are they going to find out if the object files are bad, other
than by crashing their database? I *really* don't like the attitude
that a backend crash is okay. Under any circumstances, development
or not.

(b) Badness may depend on context, eg LD_LIBRARY_PATH. So it's not
really safe to assume that if it worked before then you don't have to
worry about it crashing you in production.

regards, tom lane

Peter Eisentraut <peter_e@gmx.net> writes:

Tom Lane writes:

I'm still quite nervous about making these changes so late in the cycle.
OTOH I suspect Andreas was right: we haven't been getting any pltcl
portability testing from our beta testers.

This logic can also be reversed: We haven't been getting any beta testing
from users of Red Hat 7.1.

I don't think the tcl there had a proper so-name (from memory, I don't
use tcl)

--
Trond Eivind Glomsr�d
Red Hat, Inc.

Tom Lane <tgl@sss.pgh.pa.us> writes:

Peter Eisentraut <peter_e@gmx.net> writes:

I'm OK with this patch. (Although you don't need the override.)

Okay, committed. (I left in the override; it can't hurt can it?)

We should file a bug report with Red Hat, methinks.

Trond, I think this is your turf ... is it a bug?

(Note, I don't do tcl.)

At least part of it is intentional - adding the so name to the tcl
library. That said, it looks like a bug to me too... at the minimum,
the soname should be removed from the tclConfig.sh script after use in
generating the tcl libraries. Adrian?

--
Trond Eivind Glomsr�d
Red Hat, Inc.

override TCL_SHLIB_LD := $(shell echo $(TCL_SHLIB_LD) | sed

's/-Wl,-soname.*//')

I suspect that the above works only because -Wl,-soname is the last
switch in TCL_SHLIB_LD; any following switches would be removed too.
Perhaps better

Just have to say: I can't beleive this.
Doublefixing such a grotty Makefile that is known to not work on
2 Platforms, and known to work on those 2 + at least 2 other platforms
with the patch Tom recently sent in.

Don't you all trust Makefile.shlib ? It does perfectly create shlibs
of nearly the same making without problems.

Andreas

"Zeugswetter Andreas SB SD" <ZeugswetterA@spardat.at> writes:

Don't you all trust Makefile.shlib ?

I do trust Makefile.shlib. I don't trust a large revision of the pltcl
build process on the day we intend to put out RC1. That's a recipe for
embarrassment.

Did you already forget my report yesterday that your patch failed on HPUX?
How many of our other platforms might have problems too? Minor ones,
sure, but we have no remaining window for port testing.

The patch I stuck in was certainly an ugly wart, but it fixes the
immediate problem with minimal risk. Switching pltcl to use
Makefile.shlib is something we need to do in an orderly way with full
provisions for a port testing cycle. In other words, hold it for 7.3.

There's also the possibility that we'll listen to Peter and start
statically linking the PLs into the backend, in which case this all
becomes moot. But, again, that's not happening for 7.2.

regards, tom lane

Don't you all trust Makefile.shlib ?

I do trust Makefile.shlib. I don't trust a large revision of the pltcl
build process on the day we intend to put out RC1. That's a recipe for
embarrassment.

Yes, I know, and do accept that :-) Just had to rid myself of a little
annoyance ... (probably wasn't good wording, but such is the krux of
non-native speakers)

Did you already forget my report yesterday that your patch failed on HPUX?

I hoped, you would test on hpux, but yes. sorry. The only diff to my latest
patch that Bruce keeps in queue is the "-lc", no ?

How many of our other platforms might have problems too? Minor ones,
sure, but we have no remaining window for port testing.

Note for future fix:
I think actually hpux only failed, because it has a broken Makefile.shlib.
It is Makefile.shlib's business to know that the link line for
a shlib needs -lc ? Other ports that still use ld to link seem to get
this right (mainly the BSD's). (Makefile.port might also be the place
to do this)

As a short test you could test plpython on hpux with current cvs.
My guess is, that it doesn't work.

Andreas

"Zeugswetter Andreas SB SD" <ZeugswetterA@spardat.at> writes:

I think actually hpux only failed, because it has a broken Makefile.shlib.
It is Makefile.shlib's business to know that the link line for
a shlib needs -lc ?

Possibly -lc should go into Makefile.shlib, but that doesn't eliminate
the need for a special case in pltcl's makefile --- at least as long as
it depends on tclConfig to tell it how to link. The problem is that
TCL_SHLIB_LD_LIBS comes up empty on HPUX, but you still need to put
$(TCL_LIBS) into the link command.

As a short test you could test plpython on hpux with current cvs.
My guess is, that it doesn't work.

Could be. I don't have python installed here ...

regards, tom lane

On Sun, Jan 20, 2002 at 01:40:17PM -0500, Tom Lane wrote:

cycle, because perhaps some of these platforms don't support the full
dlopen() API. Comments? Can anyone test whether RTLD_NOW works on
any of the above-mentioned ports?

Didn't check it *works*, but from $NetBSD: dlfcn.h,v 1.13 2000/06/13 01:21:53

/* Values for dlopen `mode'. */
#define RTLD_LAZY 1
#define RTLD_NOW 2
#define RTLD_GLOBAL 0x100 /* Allow global searches in object */
#define RTLD_LOCAL 0x200
#if !defined(_XOPEN_SOURCE)
#define DL_LAZY RTLD_LAZY /* Compat */
#endif

Cheers,

Patrick

On Sun, Jan 20, 2002 at 07:16:50PM -0500, Tom Lane wrote:

Brent Verner <brent@rcfile.org> writes:

It seems that substvar operates on each " " separated token in the
string. The following works for me.

override TCL_SHLIB_LD := $(shell echo $(TCL_SHLIB_LD) | sed 's/-Wl,-soname.*//')

I suspect that the above works only because -Wl,-soname is the last
switch in TCL_SHLIB_LD; any following switches would be removed too.
Perhaps better

override TCL_SHLIB_LD := $(shell echo $(TCL_SHLIB_LD) | sed 's/-Wl,-soname[^ ]*//'

Sorry, but by this you broke freebsd build which has:

TCL_SHLIB_LD = ld -shared -x -soname $@

and $@ gets substituted too early

can you restrict this hack by putting something like

ifeq ($(PORTNAME), linux)
override TCL_SHLIB_LD := $(shell echo $(TCL_SHLIB_LD) | sed 's/-Wl,-soname[^ ]*//'
endif

instead?

[I trimmed the CC list. It was getting out of hand. Boy, how I despise having
to use reply-all.... :-)]

On Wednesday 23 January 2002 04:44 am, Murray Prior Hobbs wrote:

and ran the check (make check) - 79 tests passed

then i ran the make installcheck

and get precisely the same failures as i got on my 686 over the last 3 days

could somebody else confirm these findings please or suggest what's going
on

This probably has nothing to do with the TCL issue. It is the locale setting
biting you. The first regression run using make check is using the C locale
-- which passes all tests. The second run isn't using the C locale,
apparently. And those tests fail when the locale is not 'C'.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Murray Prior Hobbs <murray@efone.com> writes:

i made and installed

and ran the check (make check) - 79 tests passed

then i ran the make installcheck

and get precisely the same failures as i got on my 686 over the last 3 days

PATH pointing at the wrong thing, or other conflicting environment
variables (eg PGPORT), I'd guess.

regards, tom lane

Vsevolod Lobko <seva@sevasoft.kiev.ua> writes:

Perhaps better

override TCL_SHLIB_LD := $(shell echo $(TCL_SHLIB_LD) | sed 's/-Wl,-soname[^ ]*//'

Sorry, but by this you broke freebsd build which has:

TCL_SHLIB_LD = ld -shared -x -soname $@

and $@ gets substituted too early

How annoying. I was debating whether to use single or double quotes
around the echo parameter --- looks like I made the wrong choice.
Will fix.

regards, tom lane

Vsevolod Lobko writes:

override TCL_SHLIB_LD := $(shell echo $(TCL_SHLIB_LD) | sed 's/-Wl,-soname[^ ]*//'

Sorry, but by this you broke freebsd build which has:

TCL_SHLIB_LD = ld -shared -x -soname $@

and $@ gets substituted too early

I've submitted a bug report to FreeBSD about this. Let's hope they fix it
soon.

--
Peter Eisentraut peter_e@gmx.net

Peter Eisentraut <peter_e@gmx.net> writes:

I've submitted a bug report to FreeBSD about this. Let's hope they fix it
soon.

Will it not work to do

$(shell echo '$(TCL_SHLIB_LD)' | sed ...

?

regards, tom lane

[2002-01-23 11:54] Tom Lane said:
| Peter Eisentraut <peter_e@gmx.net> writes:
| > I've submitted a bug report to FreeBSD about this. Let's hope they fix it
| > soon.
|
| Will it not work to do
|
| $(shell echo '$(TCL_SHLIB_LD)' | sed ...

No. I just tested this, and the $@ still got expanded too early.
We'll need to use the suggested ifeq($(PORTNAME),linux) test.

brent

--
"Develop your talent, man, and leave the world something. Records are
really gifts from people. To think that an artist would love you enough
to share his music with anyone is a beautiful thing." -- Duane Allman

Brent Verner <brent@rcfile.org> writes:

| Will it not work to do
|
| $(shell echo '$(TCL_SHLIB_LD)' | sed ...

No. I just tested this, and the $@ still got expanded too early.

[ scratches head ... ] Where is the expansion happening, then? Seems
weird.

We'll need to use the suggested ifeq($(PORTNAME),linux) test.

I don't much like that since it makes an inappropriate assumption,
viz that if you're on Linux you must have a TCL_SHLIB_LD value that
hasn't got any $variables in it. I'd prefer to figure out *why* we
are getting a premature evaluation.

regards, tom lane

Tom Lane wrote:

Brent Verner <brent@rcfile.org> writes:

| Will it not work to do
|
| $(shell echo '$(TCL_SHLIB_LD)' | sed ...

No. I just tested this, and the $@ still got expanded too early.

[ scratches head ... ] Where is the expansion happening, then? Seems
weird.

We'll need to use the suggested ifeq($(PORTNAME),linux) test.

I don't much like that since it makes an inappropriate assumption,
viz that if you're on Linux you must have a TCL_SHLIB_LD value that
hasn't got any $variables in it. I'd prefer to figure out *why* we
are getting a premature evaluation.

As a data point, now that FreeBSD is showing problems too, I see this on
BSD/OS with TCL 8.3 in tclConfig.sh:

TCL_SHLIB_LD='cc -shared'

Does this mean I don't have the problem here?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

[2002-01-23 12:06] Tom Lane said:
| Brent Verner <brent@rcfile.org> writes:
| > | Will it not work to do
| > |
| > | $(shell echo '$(TCL_SHLIB_LD)' | sed ...
|
| > No. I just tested this, and the $@ still got expanded too early.
|
| [ scratches head ... ] Where is the expansion happening, then? Seems
| weird.

apparently the $(shell ...) construct expands any shell-like
vars.

from make's info file:

The `shell' function performs the same function that backquotes
(``') perform in most shells: it does "command expansion".

| > We'll need to use the suggested ifeq($(PORTNAME),linux) test.
|
| I don't much like that since it makes an inappropriate assumption,
| viz that if you're on Linux you must have a TCL_SHLIB_LD value that
| hasn't got any $variables in it. I'd prefer to figure out *why* we
| are getting a premature evaluation.

maybe check for a '$' in the TCL_SHLIB_LD ? I've been trying
$(findstring ...) but have not gotten that to work right yet.

brent

--
"Develop your talent, man, and leave the world something. Records are
really gifts from people. To think that an artist would love you enough
to share his music with anyone is a beautiful thing." -- Duane Allman

Tom Lane writes:

[ scratches head ... ] Where is the expansion happening, then? Seems
weird.

The $@ is expanded as a make variable. Make does care whether you're
executing a $(shell) thing around it. However, it seems that $@ should
expand to nothing in that assignment, so where's the problem?

--
Peter Eisentraut peter_e@gmx.net

[2002-01-23 12:29] Brent Verner said:
| [2002-01-23 12:06] Tom Lane said:
| | Brent Verner <brent@rcfile.org> writes:
| | > | Will it not work to do
| | > |
| | > | $(shell echo '$(TCL_SHLIB_LD)' | sed ...
| |
| | > No. I just tested this, and the $@ still got expanded too early.
| |
| | [ scratches head ... ] Where is the expansion happening, then? Seems
| | weird.
|
| apparently the $(shell ...) construct expands any shell-like
| vars.
|
| from make's info file:
|
| The `shell' function performs the same function that backquotes
| (``') perform in most shells: it does "command expansion".
|
| | > We'll need to use the suggested ifeq($(PORTNAME),linux) test.
| |
| | I don't much like that since it makes an inappropriate assumption,
| | viz that if you're on Linux you must have a TCL_SHLIB_LD value that
| | hasn't got any $variables in it. I'd prefer to figure out *why* we
| | are getting a premature evaluation.
|
| maybe check for a '$' in the TCL_SHLIB_LD ? I've been trying
| $(findstring ...) but have not gotten that to work right yet.

The best I can come up with is checking for the offending string
'libtcl.so.0' in the TCL_SHLIB_LD.

ifneq (,$(findstring libtcl.so.0,$(TCL_SHLIB_LD)))
TCL_SHLIB_LD := $(shell echo '$(TCL_SHLIB_LD)' | sed 's/-Wl,-soname[^ ]*//')
endif

any better ideas out there? I've exhausted my PG time for today ;-)

brent

--
"Develop your talent, man, and leave the world something. Records are
really gifts from people. To think that an artist would love you enough
to share his music with anyone is a beautiful thing." -- Duane Allman

Tom Lane <tgl@sss.pgh.pa.us> writes:

Brent Verner <brent@rcfile.org> writes:

| Will it not work to do
|
| $(shell echo '$(TCL_SHLIB_LD)' | sed ...

No. I just tested this, and the $@ still got expanded too early.

[ scratches head ... ] Where is the expansion happening, then? Seems
weird.

The expansion is done by make. It does expand variables recursively,
even for :=. Using $$@ instead of $@ should help in this particular
case, I think.

Bernhard

--
Intevation GmbH http://intevation.de/
Sketch http://sketch.sourceforge.net/
MapIt! http://mapit.de/

Peter Eisentraut <peter_e@gmx.net> writes:

Tom Lane writes:

[ scratches head ... ] Where is the expansion happening, then? Seems
weird.

The $@ is expanded as a make variable. Make does care whether you're
executing a $(shell) thing around it. However, it seems that $@ should
expand to nothing in that assignment, so where's the problem?

I think the complaint is that we need it to still look like $@ when
TCL_SHLIB_LD is used in the shlib-building rule. If Make recursively
expands the variable before executing the $shell construct then we
got trouble.

Ugly as it is, the check on portname may be the best solution available.
I'm gonna think a little more, but I haven't got a better idea now.

regards, tom lane

Bernhard Herzog <bh@intevation.de> writes:

Using $$@ instead of $@ should help in this particular
case, I think.

But we haven't got control of what the initial value of TCL_SHLIB_LD is.

Hmm. Wait a minute; we're going about this all wrong. Instead of
hacking the Makefile, let's hack mkMakefile.tcldefs.sh. We can
definitely fix the TCL_SHLIB_LD value there, before Make sees it.

regards, tom lane

On Wed, Jan 23, 2002 at 12:59:00PM -0500, Peter Eisentraut wrote:

Tom Lane writes:

[ scratches head ... ] Where is the expansion happening, then? Seems
weird.

The $@ is expanded as a make variable. Make does care whether you're
executing a $(shell) thing around it. However, it seems that $@ should
expand to nothing in that assignment, so where's the problem?

exactly in that, so make triing to execute
ld -shared -x -soname -o pltcl.so pltcl.o
^ no arguments to -soname

ok i dropped the locale support and reconfigured, remade, reinstalled
etc etc on two machines

both machines failed - but with different failures

see below Lamar's mail for deatils of the two sets of failures

murray

ps - i'll remove configure options until nothing fails

Lamar Owen wrote:

[I trimmed the CC list. It was getting out of hand. Boy, how I despise having
to use reply-all.... :-)]

On Wednesday 23 January 2002 04:44 am, Murray Prior Hobbs wrote:

and ran the check (make check) - 79 tests passed

then i ran the make installcheck

and get precisely the same failures as i got on my 686 over the last 3 days

could somebody else confirm these findings please or suggest what's going
on

This probably has nothing to do with the TCL issue. It is the locale setting
biting you. The first regression run using make check is using the C locale
-- which passes all tests. The second run isn't using the C locale,
apparently. And those tests fail when the locale is not 'C'.

these are the make checkinstall failures for the following configuarion

./configure --enable-multibyte=UNICODE --enable-unicode-conversion --enable-locale --with-tcl --without-tk --enable-odbc --with-unixodbc --enable-syslog

/bin/sh ./pg_regress --schedule=./serial_schedule --multibyte=UNICODE
(using postmaster on Unix socket, default port)
============== dropping database "regression" ==============
ERROR: DROP DATABASE: database "regression" does not exist
dropdb: database removal failed
============== creating database "regression" ==============
CREATE DATABASE
============== dropping regression test user accounts ==============
============== installing PL/pgSQL ==============
============== running regression test queries ==============
test boolean ... ok
test char ... FAILED
test name ... ok
test varchar ... FAILED
test text ... ok
test int2 ... ok
test int4 ... ok
test int8 ... FAILED
test oid ... ok
test float4 ... ok
test float8 ... ok
test bit ... ok
test numeric ... FAILED
test strings ... ok
test numerology ... ok
test point ... ok
test lseg ... ok
test box ... ok
test path ... ok
test polygon ... ok
test circle ... ok
test date ... ok
test time ... ok
test timetz ... ok
test timestamp ... ok
test timestamptz ... ok
test interval ... ok
test abstime ... ok
test reltime ... ok
test tinterval ... ok
test inet ... ok
test comments ... ok
test oidjoins ... ok
test type_sanity ... ok
test opr_sanity ... ok
test geometry ... ok
test horology ... ok
test create_function_1 ... ok
test create_type ... ok
test create_table ... ok
test create_function_2 ... ok
test copy ... FAILED
test constraints ... ok
test triggers ... ok
test create_misc ... ok
test create_aggregate ... ok
test create_operator ... ok
test create_index ... ok
test inherit ... ok
test create_view ... ok
test sanity_check ... ok
test errors ... ok
test select ... ok
test select_into ... ok
test select_distinct ... ok
test select_distinct_on ... ok
test select_implicit ... FAILED
test select_having ... FAILED
test subselect ... ok
test union ... ok
test case ... ok
test join ... ok
test aggregates ... ok
test transactions ... ok
test random ... ok
test portals ... ok
test arrays ... ok
test btree_index ... ok
test hash_index ... ok
test privileges ... ok
test misc ... FAILED
test select_views ... FAILED
test alter_table ... ok
test portals_p2 ... ok
test rules ... ok
test foreign_key ... ok
test limit ... ok
test plpgsql ... ok
test temp ... ok

=======================
9 of 79 tests failed.
=======================

these are the make checkinstall failures for the following configuarion - the locale option is dropped

./configure --enable-multibyte=UNICODE --enable-unicode-conversion --with-tcl --without-tk --enable-odbc --with-unixodbc --enable-syslog

/bin/sh ./pg_regress --schedule=./serial_schedule --multibyte=UNICODE
(using postmaster on Unix socket, default port)
============== dropping database "regression" ==============
ERROR: DROP DATABASE: database "regression" does not exist
dropdb: database removal failed
============== creating database "regression" ==============
CREATE DATABASE
============== dropping regression test user accounts ==============
============== installing PL/pgSQL ==============
============== running regression test queries ==============
test boolean ... ok
test char ... ok
test name ... ok
test varchar ... ok
test text ... ok
test int2 ... ok
test int4 ... ok
test int8 ... ok
test oid ... ok
test float4 ... ok
test float8 ... ok
test bit ... ok
test numeric ... ok
test strings ... ok
test numerology ... ok
test point ... ok
test lseg ... ok
test box ... ok
test path ... ok
test polygon ... ok
test circle ... ok
test date ... ok
test time ... ok
test timetz ... ok
test timestamp ... ok
test timestamptz ... ok
test interval ... ok
test abstime ... ok
test reltime ... ok
test tinterval ... ok
test inet ... ok
test comments ... ok
test oidjoins ... ok
test type_sanity ... ok
test opr_sanity ... ok
test geometry ... ok
test horology ... ok
test create_function_1 ... ok
test create_type ... ok
test create_table ... ok
test create_function_2 ... ok
test copy ... FAILED
test constraints ... ok
test triggers ... ok
test create_misc ... ok
test create_aggregate ... ok
test create_operator ... ok
test create_index ... ok
test inherit ... ok
test create_view ... ok
test sanity_check ... ok
test errors ... ok
test select ... FAILED
test select_into ... ok
test select_distinct ... FAILED
test select_distinct_on ... FAILED
test select_implicit ... ok
test select_having ... ok
test subselect ... ok
test union ... ok
test case ... ok
test join ... ok
test aggregates ... FAILED
test transactions ... ok
test random ... failed (ignored)
test portals ... ok
test arrays ... ok
test btree_index ... ok
test hash_index ... ok
test privileges ... ok
test misc ... FAILED
test select_views ... ok
test alter_table ... ok
test portals_p2 ... FAILED
test rules ... ok
test foreign_key ... ok
test limit ... FAILED
test plpgsql ... ok
test temp ... ok

====================================================
9 of 79 tests failed, 1 of these failures ignored.
====================================================

Vsevolod Lobko writes:

Sorry, but by this you broke freebsd build which has:

TCL_SHLIB_LD = ld -shared -x -soname $@

and $@ gets substituted too early

FreeBSD has fixed this now.

--
Peter Eisentraut peter_e@gmx.net

I hate to sound like a broken record, but I want to re-open that
discussion about RTLD_LAZY binding that trailed off a week or two
ago. I have just noticed that the 7.0 and 7.1 versions of
src/backend/port/dynloader/linux.h have

#define pg_dlopen(f) dlopen(f, 2)

which in 7.2 has been changed to

#define pg_dlopen(f) dlopen((f), RTLD_LAZY | RTLD_GLOBAL)

But a quick look in /usr/include/bits/dlfcn.h shows that (at least
on RH Linux 7.2), the old coding was equivalent to RTLD_NOW.

I therefore assert that the current coding is effectively untested
on Linux, which is probably our most popular platform, and therefore
it should *NOT* be accorded the respect normally due to the status
quo. Arguably, 7.2 has introduced breakage here.

A grep through the 7.1 versions of src/backend/port/dynloader/*.h
shows the following rather motley assortment of dlopen flag choices:

aix.h:61:#define pg_dlopen(f) dlopen(f, RTLD_LAZY)
bsdi.h:23:#define pg_dlopen(f) dlopen(f, RTLD_LAZY)
dgux.h:26:#define pg_dlopen(f) dlopen(f,1)
freebsd.h:36:#define pg_dlopen(f) BSD44_derived_dlopen(f, 1)
irix5.h:29:#define pg_dlopen(f) dlopen(f,1)
linux.h:34:#define pg_dlopen(f) dlopen(f, 2)
netbsd.h:36:#define pg_dlopen(f) BSD44_derived_dlopen(f, 1)
openbsd.h:36:#define pg_dlopen(f) BSD44_derived_dlopen(f, 1)
osf.h:31:#define pg_dlopen(f) dlopen(f, RTLD_LAZY)
sco.h:29:#define pg_dlopen(f) dlopen(f,1)
solaris.h:9:#define pg_dlopen(f) dlopen(f,1)
sunos4.h:29:#define pg_dlopen(f) dlopen(f, 1)
svr4.h:29:#define pg_dlopen(f) dlopen(f,RTLD_LAZY)
univel.h:29:#define pg_dlopen(f) dlopen(f,RTLD_LAZY)
unixware.h:29:#define pg_dlopen(f) dlopen(f,RTLD_LAZY)
win.h:29:#define pg_dlopen(f) dlopen(f,1)

In 7.2 these have all been changed to "RTLD_LAZY | RTLD_GLOBAL", but
I am no longer willing to presume that that's equivalent to the
original coding. Could people who have these platforms look to see
what the numeric values mentioned above actually equate to on their
platforms?

regards, tom lane

[2002-02-10 20:00] Tom Lane said:

| freebsd.h:36:#define pg_dlopen(f) BSD44_derived_dlopen(f, 1)
| netbsd.h:36:#define pg_dlopen(f) BSD44_derived_dlopen(f, 1)

freebsd 4.5
netbsd 1.5.2

#define RTLD_LAZY 1
#define RTLD_GLOBAL 0x100

cheers.
brent

--
"Develop your talent, man, and leave the world something. Records are
really gifts from people. To think that an artist would love you enough
to share his music with anyone is a beautiful thing." -- Duane Allman

Brent Verner <brent@rcfile.org> writes:

freebsd 4.5
netbsd 1.5.2

#define RTLD_LAZY 1
#define RTLD_GLOBAL 0x100

Thanks. Is there an RTLD_NOW symbol on those platforms?

regards, tom lane

[2002-02-10 21:24] Tom Lane said:
| Brent Verner <brent@rcfile.org> writes:
| > freebsd 4.5
| > netbsd 1.5.2
|
| > #define RTLD_LAZY 1
| > #define RTLD_GLOBAL 0x100
|
| Thanks. Is there an RTLD_NOW symbol on those platforms?

yes. I've attached the dlfcn.h files from each incase there is
anything else in there that might be of interest.

cheers.
brent

--
"Develop your talent, man, and leave the world something. Records are
really gifts from people. To think that an artist would love you enough
to share his music with anyone is a beautiful thing." -- Duane Allman

I wrote:

I hate to sound like a broken record, but I want to re-open that
discussion about RTLD_LAZY binding that trailed off a week or two
ago.
... I therefore assert that the current coding is effectively untested
on Linux, which is probably our most popular platform, and therefore
it should *NOT* be accorded the respect normally due to the status
quo. Arguably, 7.2 has introduced breakage here.

After some further digging around on the net, I believe that coding in
the following style is safe and will work on all systems supporting
dlopen():

/*
* In older systems, like SunOS 4.1.3, the RTLD_NOW flag isn't defined
* and the mode argument to dlopen must always be 1. The RTLD_GLOBAL
* flag is wanted if available, but it doesn't exist everywhere.
* If it doesn't exist, set it to 0 so it has no effect.
*/
#ifndef RTLD_NOW
# define RTLD_NOW 1
#endif

#ifndef RTLD_GLOBAL
# define RTLD_GLOBAL 0
#endif

#define pg_dlopen(f) dlopen((f), RTLD_NOW | RTLD_GLOBAL)

I also believe that this will produce more consistent cross-platform
behavior: so far as I could learn from googling, systems that do not
define RTLD_NOW/RTLD_LAZY all act as though the mode were RTLD_NOW,
ie, immediate binding.

Any objections to modifying all the port/dynloader files this way?

regards, tom lane

I hate to sound like a broken record, but I want to re-open that
discussion about RTLD_LAZY binding that trailed off a week or two
ago. I have just noticed that the 7.0 and 7.1 versions of
src/backend/port/dynloader/linux.h have

#define pg_dlopen(f) dlopen(f, 2)

which in 7.2 has been changed to

#define pg_dlopen(f) dlopen((f), RTLD_LAZY | RTLD_GLOBAL)

One thing to really watch out for is that old style extensions,
that only need a few functions from a standard library will load
more efficiently with RTLD_LAZY. (might not even pull in dependent
libs, that are not needed)

Next thing to watch out for is, that RTLD_NOW will probably not load
a shared lib, that was not linked with a "no entry" flag. Arguably a bug
for a shared lib, but a recent report has shown that pg fails to supply
such a flag on all ports.

I am for keeping RTLD_LAZY :-)

Andreas

"Zeugswetter Andreas SB SD" <ZeugswetterA@spardat.at> writes:

One thing to really watch out for is that old style extensions,
that only need a few functions from a standard library will load
more efficiently with RTLD_LAZY.

I thought we had disposed of that argument. The issue here is not a
marginal efficiency gain, it is that an unresolved symbol will lead
to a backend crash (== database-wide restart) unless it is detected
at dlopen time. As a wise man once said, "I can make it arbitrarily
fast ... if it doesn't have to work."

Next thing to watch out for is, that RTLD_NOW will probably not load
a shared lib, that was not linked with a "no entry" flag.

Say again?

regards, tom lane

Next thing to watch out for is, that RTLD_NOW will probably not load
a shared lib, that was not linked with a "no entry" flag.

Say again?

On e.g. AIX (and Sun?) the Makefile.shlib does not use a "-bnoentry" flag
when linking shared libs. While the pg backend successfully loads
those libs, tclsh e.g. fails. Without reading the tcl code, I thought it
might be RTLD_NOW.

I just tested 7.2 on AIX with RTLD_NOW though, and it works fine,
so the issue must be something else.

Andreas

On Mon, Feb 11, 2002 at 07:49:57PM -0500, Tom Lane wrote:

I also believe that this will produce more consistent cross-platform
behavior: so far as I could learn from googling, systems that do not
define RTLD_NOW/RTLD_LAZY all act as though the mode were RTLD_NOW,
ie, immediate binding.

Any objections to modifying all the port/dynloader files this way?

OpenBSD:
The dlopen() function takes a name of a shared object as its first argu-
ment. The shared object is mapped into the address space, relocated, and
its external references are resolved in the same way as is done with the
implicitly loaded shared libraries at program startup.

The path argument can either be an absolute pathname or it can be of the
form ``lib<name>.so[.xx[.yy]]'' in which case the same library search
rules apply that are used for ``intrinsic'' shared library searches. The
second argument currently has no effect, but should be set to DL_LAZY for
future compatibility.

That last sentence being key....

--
David Terrell | "War is peace,
Prime Minister, Nebcorp | freedom is slavery,
dbt@meat.net | ignorance is strength
http://wwn.nebcorp.com/ | Dishes are clean." - Chris Fester