Threaded Python vs. PostGreSQL plpython

Mike Meyer writes:

The problem is that the ports system build python with thread
support. postmaster doesn't have thread support, so when the
libpython2.2.so is dynamically loaded, it fails to find the thread
functions, and the load fails.

What is "thread support", what are "the thread functions", why does it try
to find them in the postmaster, and what are the exact details anyway?

The first workaround I tried was to build a custom version of the
python library that doesn't have thread support. Given that plpython
won't let me import the thread modules, this isn't a problem. However,
it does mean I have a copy of libpython2.2.so where they dynamic
loader can find it, meaning the linker will find it, meaning that
future builds of other embedded software - like apache's mod_python -
will wind up with the non-threaded library. This is a bad thing, and
I'd like to avoid it.

With judicious use of rpath you can keep several versions of a shared
libpython around without the dynamic loader getting confused.

I tried building linking plpython.so against the static library
instead of the dynamica library, but that doesn't work properly when
loaded. I'm not sure what the problem is.

Possibly, the dynamic library would in turn automatically load some other
dynamic libraries, either because of implicit linkage or through the
run-time dynamic loading mechanism. If you have a static library, then
you miss all those things. Hard to tell without getting exact details,
though. I know on some platforms it has been shown to be possible to use
a static libpython for plpython.

--
Peter Eisentraut peter_e@gmx.net

On 24 Mar 2003 at 11:45, Mike Meyer wrote:

In <200303241039.04880.shridhar_daithankar@persistent.co.in>, "Shridhar Daithankar<shridhar_daithankar@persistent.co.in>" <shridhar_daithankar@persistent.co.in> typed:

Well, I believe, if you link a library against libc_r explicitly, then even if
it is loaded by a program which is not linked against it would do.

One problem on freeBSD is that the thread library is C library itself. So if
postgresql is linked against libc.so and libpython is linked against
libc_r.so, then you *might* get symbol collision.

I suggest you try following.

It's possible you mean something other than what I think you mean by
"link against", so my replies might not make sense. In which case,
please be more explicit.

I mean nothing more than adding -lc_r flag in link time option. That is link
against. Hope this is clear.

1) Link libpython.so against linuxthreads.

Ugh. I'd really rather not have a my python depending on
linuxthreads. That's marginally better than having no threads at all,
but that's about it. It's not clear it's worth installing the
linuxthreads port for.

OK let me explain.

2) Link libpython.so explicitly against libc_r.so, in case it depends upon
thread fucntions to be provided by loader program.. Check ldd libpython.so

I know it depends on thread functions provided by the loader
program. It lists one of those as missing when I try and do a
createlang on the shared library built from the installed python.

Building a shared library from the threaded version and explicitly
linking in libc_r.so - so that ldd shows libc_r as a requirement -
causes the server to terminate on a signal 11 when I try and do a
createlang on plpython.

OK. Let's try to explain what is happening here. You have linked libpython.so
against libc_r.so. Postgresql is linked against libc.so. These two libraries
have large number of symbols common between them. It is quite possible that the
SIGSEGV you are getting is due to some symbol collision.

Now if you link libpython.so against linuxthreads, it *should* work perfectly
with postgresql because linuxthreads library provides only thread functions.
libc.so has none of them. So there is no chance of collision and it *should*
work.

I hope you find this logical. That is the reason I abandoned linuxthreads in
one of my projects because I could not stably link linuxthreads and libc_r.so
in same program.

Anyway this is just to get things linked against and running. You can not run
threaded python apps with such a library because libc against which it is
linked is not threaded or thread safe.

I consider this as a bit of fault with freeBSD libc. Although I can live with
it.

OTOH, you can link postgresql and libpython.so with libc_r. That should work as
well and you can use threads in python too. I would recommend this approach.

HTH

Bye
Shridhar

--
Bubble Memory, n.: A derogatory term, usually referring to a person's
intelligence. See also "vacuum tube".

In <3E803616.19976.2E2BDDB@localhost>, Shridhar Daithankar <shridhar_daithankar@persistent.co.in> typed:

On 24 Mar 2003 at 11:45, Mike Meyer wrote:

Just to clarify, the current situation is that I have things working,
but I also have a libpython.so that isn't threaded. I'm not happy
about the latter, and worry about rebuilding an embedded applications
and suddenly having my threaded scripots quit working because it
picked up the wrong library. I also have problems in that some modules
I should be able to import with plpython don't, because they are
linked against the wrong libc.

In <200303241039.04880.shridhar_daithankar@persistent.co.in>, "Shridhar Daithankar<shridhar_daithankar@persistent.co.in>" <shridhar_daithankar@persistent.co.in> typed:

Now if you link libpython.so against linuxthreads, it *should* work perfectly
with postgresql because linuxthreads library provides only thread functions.
libc.so has none of them. So there is no chance of collision and it *should*
work.
Anyway this is just to get things linked against and running. You can not run
threaded python apps with such a library because libc against which it is
linked is not threaded or thread safe.

So this would be similar to my current solution - my real python would
be threaded and linked against libc_r, and I'd have a "special"
libpython.so that looked like it could be threaded, but in reality
couldn't because it uses linuxthreads and a not-thread-safe libc. This
is not an improvement - in fact, I'd say it's worse.

OTOH, you can link postgresql and libpython.so with libc_r. That should work as
well and you can use threads in python too. I would recommend this approach.

This was my second choice. I was hoping that someone would tell me
this was safe before I went to the trouble of trying it. Well, I was
hoping someone had a better solution than that, but if this works,
I'll do it this way.

Thank you,
<mike
--
Mike Meyer <mwm@mired.org> http://www.mired.org/consulting.html
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.