Thread configure flag

Bruce Momjian writes:

This applied patch adds --with-threads to configure. This will soon be
hooked into libpq. Doc additions made too.

Is there a reason why this isn't the default?

--
Peter Eisentraut peter_e@gmx.net

Peter Eisentraut wrote:

Bruce Momjian writes:

This applied patch adds --with-threads to configure. This will soon be
hooked into libpq. Doc additions made too.

Is there a reason why this isn't the default?

If it is the default, libpq is going to use the libc_r library, at least
on some platforms, and that may not be desired. Fortunately, the flag
has no effect on the backend code.

One funny thing is that on BSD/OS and NetBSD, because there are no
thread compile flags, function names, or libraries, libpq is the same
code with or without the flag. ecpg, however, has thread calls, and is
effected on every platform.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Peter Eisentraut <peter_e@gmx.net> writes:

Bruce Momjian writes:

This applied patch adds --with-threads to configure. This will soon be
hooked into libpq. Doc additions made too.

Is there a reason why this isn't the default?

When Bruce and I discussed this last week, we assumed you'd object if
we did not do it this way ;-). --with-threads is a feature that can't
be supported on some platforms. Usually you argue that configure
shouldn't auto-enable features in some places and not others.

regards, tom lane

Tom Lane writes:

When Bruce and I discussed this last week, we assumed you'd object if
we did not do it this way ;-). --with-threads is a feature that can't
be supported on some platforms. Usually you argue that configure
shouldn't auto-enable features in some places and not others.

But this isn't the same thing as supporting some external package. If
readline is missing, you install it and repeat. But if thread-safeness is
missing, there's nothing you can do. Therefore I think that the default
behavior should be to do as good as the operating system can.

Consider this: If an operating system did not support thread-safe
applications outright (do such operating systems exist?) then users would
never get the idea to write thread-safe libpq applications, so the option
to make a thread-safe libpq would be useless.

--
Peter Eisentraut peter_e@gmx.net

Bruce Momjian writes:

If it is the default, libpq is going to use the libc_r library, at least
on some platforms, and that may not be desired.

But what is the objective reason for this lack of desire?

One funny thing is that on BSD/OS and NetBSD, because there are no
thread compile flags, function names, or libraries, libpq is the same
code with or without the flag. ecpg, however, has thread calls, and is
effected on every platform.

Well, I'm pretty sure that the current thread options are overkill and can
be simplified. For instance, on Linux the template currently claims

THREAD_CFLAGS="-D_REENTRANT -D_THREAD_SAFE -D_POSIX_PTHREAD_SEMANTICS"

_POSIX_PTHREAD_SEMANTICS isn't used anywhere in the entire system, the
effect of _THREAD_SAFE is to define _REENTRANT, and the effect of
_REENTRANT is to declare getlogin_r(), which PostgreSQL sources don't use.
In other words, this can be removed.

--
Peter Eisentraut peter_e@gmx.net

Peter Eisentraut wrote:

Bruce Momjian writes:

If it is the default, libpq is going to use the libc_r library, at least
on some platforms, and that may not be desired.

But what is the objective reason for this lack of desire?

libc_r could behave differently than libc on some platforms --- this is
the reason we don't use some semaphore stuff in pthreads in backend code
--- we don't know the ramifications.

Also, --with-threads adds threading calls to ecpg, which I am not sure
everyone would want in the default case.

One funny thing is that on BSD/OS and NetBSD, because there are no
thread compile flags, function names, or libraries, libpq is the same
code with or without the flag. ecpg, however, has thread calls, and is
effected on every platform.

Well, I'm pretty sure that the current thread options are overkill and can
be simplified. For instance, on Linux the template currently claims

No question. I just grabbed all the flags suggested. Many OS's are
going to require such tweeking.

THREAD_CFLAGS="-D_REENTRANT -D_THREAD_SAFE -D_POSIX_PTHREAD_SEMANTICS"

_POSIX_PTHREAD_SEMANTICS isn't used anywhere in the entire system, the
effect of _THREAD_SAFE is to define _REENTRANT, and the effect of
_REENTRANT is to declare getlogin_r(), which PostgreSQL sources don't use.
In other words, this can be removed.

You mean none of them are needed? Why would _REENTRANT only control
getlogin_r()? Are all the other functions thread-safe already?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Peter Eisentraut <peter_e@gmx.net> writes:

Bruce Momjian writes:

If it is the default, libpq is going to use the libc_r library, at least
on some platforms, and that may not be desired.

But what is the objective reason for this lack of desire?

I have heard that the reentrant libc is significantly worse-performing
than the non-reentrant one on some platforms. This is not real hard
to believe, since functions as common as malloc() will need locking
overhead if they think they might be in a multithreaded environment.

However, that's only an argument that we must provide a
--without-threads option, it doesn't speak strongly to the question of
what the default choice should be (where there is a choice).

_POSIX_PTHREAD_SEMANTICS isn't used anywhere in the entire system, the
effect of _THREAD_SAFE is to define _REENTRANT, and the effect of
_REENTRANT is to declare getlogin_r(), which PostgreSQL sources don't use.

On which version of Linux are the above statements true? Are you sure
they are true on other versions?

regards, tom lane

Tom Lane writes:

I have heard that the reentrant libc is significantly worse-performing
than the non-reentrant one on some platforms. This is not real hard
to believe, since functions as common as malloc() will need locking
overhead if they think they might be in a multithreaded environment.

Then maybe we should make a libpq and a libpq_r, etc. Then users have the
choice when they build their applications, and don't already have to think
about what their applications may require when building PostgreSQL.
(There is a libmysqlclient and a libmysqlclient_r, FWIW.)

_POSIX_PTHREAD_SEMANTICS isn't used anywhere in the entire system, the
effect of _THREAD_SAFE is to define _REENTRANT, and the effect of
_REENTRANT is to declare getlogin_r(), which PostgreSQL sources don't use.

On which version of Linux are the above statements true? Are you sure
they are true on other versions?

The documentation tells:

- Macro: _REENTRANT
- Macro: _THREAD_SAFE
If you define one of these macros, reentrant versions of several
functions get declared.

However, no function is specifically documented to require these macros.
Also, according to the change log of glibc, this state has existed since
about 1996, just the set of xxx_r() functions that were covered by these
macros has changed now and then.

SUSv3 documents a number of xxx_r() functions, but it doesn't speak of
_REENTRANT or _THREAD_SAFE anywhere.

So what I'm looking for is some documentation that states that such and
such flags is required to get such and such effect.

--
Peter Eisentraut peter_e@gmx.net

--On Tuesday, June 17, 2003 20:02:00 +0200 Peter Eisentraut
<peter_e@gmx.net> wrote:

Tom Lane writes:

SUSv3 documents a number of xxx_r() functions, but it doesn't speak of
_REENTRANT or _THREAD_SAFE anywhere.

So what I'm looking for is some documentation that states that such and
such flags is required to get such and such effect.

On UnixWare, you NEED to use -Kthread or -Kpthread, to get a Threads
version.

There are a NUMBER of functions that get wrapped, etc when these flags are
specified.

They also call in libthread.so.

The doc at http://www.lerctr.org:8458/ might prove useful as a real-life
example of
a threaded kernel.

LER

--
Peter Eisentraut peter_e@gmx.net

---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to majordomo@postgresql.org)

--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749

Peter Eisentraut <peter_e@gmx.net> writes:

Tom Lane writes:

I have heard that the reentrant libc is significantly worse-performing
than the non-reentrant one on some platforms. This is not real hard
to believe, since functions as common as malloc() will need locking
overhead if they think they might be in a multithreaded environment.

Then maybe we should make a libpq and a libpq_r, etc.

That would be fine with me ... although it's only worth doing if the
platform has a libc and a libc_r. Shall we use the existence of libc_r
to drive this choice?

regards, tom lane

On 17 Jun 2003 at 14:14, Tom Lane wrote:

Then maybe we should make a libpq and a libpq_r, etc.

That would be fine with me ... although it's only worth doing if the
platform has a libc and a libc_r. Shall we use the existence of libc_r
to drive this choice?

I agree with this. If libpq is linked against libc_r for threading and mutex
functions etc. and postgresql is linked against libc, then things like dblink
which uses both postgresql and libc run into problem of symbol conflict. I
experienced this on freeBSD and work around was to switch to linuxthreads which
is not really good.

If there are two versions of libraries available, we should do both. But that
adds to bloat for certain.

Just a thought.

Bye
Shridhar

--
Lie, n.: A very poor substitute for the truth, but the only one discovered to
date.

Shridhar Daithankar wrote:

On 17 Jun 2003 at 14:14, Tom Lane wrote:

Then maybe we should make a libpq and a libpq_r, etc.

That would be fine with me ... although it's only worth doing if the
platform has a libc and a libc_r. Shall we use the existence of libc_r
to drive this choice?

I agree with this. If libpq is linked against libc_r for threading and mutex
functions etc. and postgresql is linked against libc, then things like dblink
which uses both postgresql and libc run into problem of symbol conflict. I
experienced this on freeBSD and work around was to switch to linuxthreads which
is not really good.

If there are two versions of libraries available, we should do both. But that
adds to bloat for certain.

There are a couple of issues here:

First, ecpg has actual thead creation function calls, so I believe we
will still need a --with-threads configure option, at least to control
ecpg, unless we make threading some kind of ecpg runtime flag.

Second, we have some platforms that generate threaded libpq without any
special flags --- libc is threaded by default and no special compile
flags are required.

So, if we move in the libpq_r direction, we will have some platforms
that don't have libpq_r because they don't need it, and others that
don't have it because they don't support threading or we don't know how
to do threading in our configure tests. This seems very confusing to
me. I wonder if we should just keep the current setup for 7.4 and
collect theading configure information, then revisit this for 7.5.

In the current code, if they ask for threads, they succeed or fail.
With an automatic setup, I am not sure how to communicate to the user
whether they are thread-safe or not. I believe this comes back to
Peter's idea that we don't do things automatically, and if they ask for
something, we should fail if we can't do it.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian writes:

First, ecpg has actual thead creation function calls, so I believe we
will still need a --with-threads configure option, at least to control
ecpg, unless we make threading some kind of ecpg runtime flag.

This is not an issue because it only affects the libecpg library.

So, if we move in the libpq_r direction, we will have some platforms
that don't have libpq_r because they don't need it, and others that
don't have it because they don't support threading or we don't know how
to do threading in our configure tests. This seems very confusing to
me.

On a system with a separate libc_r, you *already* need special compiler
flags to link thread-safe applications. So it is only consistent that
libpq applications also need special flags. For example, a normal
application is linked so:

cc -o prog prog1.o prog2.o -lpq -lc

(well, you don't write the -lc) and a thread-safe application is linked
so:

cc -o prog prog1.o prog2.o -lpq_r -lc_r

I wonder if we should just keep the current setup for 7.4 and
collect theading configure information, then revisit this for 7.5.

Well, when the packagers wrap up 7.4 they're going to need to make a call.
Either it's not going to be thread-safe, and the work was in vain, or it's
thread-safe and the concerns about the supposed performance hits are not
addressed.

--
Peter Eisentraut peter_e@gmx.net

Peter Eisentraut wrote:

Bruce Momjian writes:

First, ecpg has actual thead creation function calls, so I believe we
will still need a --with-threads configure option, at least to control
ecpg, unless we make threading some kind of ecpg runtime flag.

This is not an issue because it only affects the libecpg library.

So we would just have a libecpg and a libecpg_r? OK.

So, if we move in the libpq_r direction, we will have some platforms
that don't have libpq_r because they don't need it, and others that
don't have it because they don't support threading or we don't know how
to do threading in our configure tests. This seems very confusing to
me.

On a system with a separate libc_r, you *already* need special compiler
flags to link thread-safe applications. So it is only consistent that
libpq applications also need special flags. For example, a normal
application is linked so:

cc -o prog prog1.o prog2.o -lpq -lc

(well, you don't write the -lc) and a thread-safe application is linked
so:

cc -o prog prog1.o prog2.o -lpq_r -lc_r

OK, I am now wondering if we should just create non-thread libpq by
default, and create threaded libpq for non-libc_r platforms and libpq
and libpq_r for libc_r platforms, and of course fail on platforms where
we don't know how to do threading. How is that?

I wonder if we should just keep the current setup for 7.4 and
collect theading configure information, then revisit this for 7.5.

Well, when the packagers wrap up 7.4 they're going to need to make a call.
Either it's not going to be thread-safe, and the work was in vain, or it's
thread-safe and the concerns about the supposed performance hits are not
addressed.

Well, it will not be in vain because people can still compile their own
builds. I am not sure if we support enough platforms to enable all this
by default in 7.4, and it could cause confusion.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

--On Saturday, June 21, 2003 15:57:54 -0400 Bruce Momjian
<pgman@candle.pha.pa.us> wrote:

\

Well, when the packagers wrap up 7.4 they're going to need to make a
call. Either it's not going to be thread-safe, and the work was in vain,
or it's thread-safe and the concerns about the supposed performance hits
are not addressed.

Well, it will not be in vain because people can still compile their own
builds. I am not sure if we support enough platforms to enable all this
by default in 7.4, and it could cause confusion.

Did anyone see my comments about -Kthread/-Kpthread on UnixWare?

That throws another mix into it.

--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749