ECPG thread-safety

I have put a mailbox file at:

ftp://candle.pha.pa.us/pub/postgresql/mypatches

where there is a partial implementation of thread-safe ecpg. Maybe you
can work with Michael to complete it.

---------------------------------------------------------------------------

Philip Yarra wrote:

As I found out recently, ECPG is not thread-safe. This is a problem for us, as
we really need to be able to use ECPG* in a multi-threaded C application.
Michael mentioned that Lee is working on fixing these problems, but was not
sure when it would be complete.

So, my questions are:
Lee/anyone, do you have an estimated completion date?
If not, can we (myself plus a "real" C programmer) help you?

Regards, Philip Yarra.

* Yes, we could use libpq, but ECPG minimises our porting from Informix.
Rewriting everything to use libpq would be too expensive.

---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

On Fri, 28 Mar 2003 13:00, you wrote:

I have put a mailbox file at:

ftp://candle.pha.pa.us/pub/postgresql/mypatches

where there is a partial implementation of thread-safe ecpg. Maybe you
can work with Michael to complete it.

Okay, thanks for that. Have grabbed the file, and started reading. I'll also
wait to hear from Michael or Lee before going too far on this.

Everything I have read so far suggests the issue is related to SQLCA itself...
and that would be consistent with what happens in the backend (data is
inserted, but threads lock straight after). Can anyone enlighten me further?

Thanks and regards, Philip.

Philip,

You can find the "patch so far" at:

http://services.csl.co.uk/postgresql/

along with a libpq thread-safe patch.

The thread referenced by Bruce's email details some of the issues
still to be looked at. The main work is integration with the build
system (testing for and linking in POSIX threads) and testing.

I'm more than happy for you to look at at, I haven't had the time I
hoped to complete this.

Thanks, Lee.

Philip Yarra writes:

I have the libpq patch in my mailbox that just needs configure thread
testing code. I will get that in for 7.4.

---------------------------------------------------------------------------

Lee Kindness wrote:

Philip,

You can find the "patch so far" at:

http://services.csl.co.uk/postgresql/

along with a libpq thread-safe patch.

The thread referenced by Bruce's email details some of the issues
still to be looked at. The main work is integration with the build
system (testing for and linking in POSIX threads) and testing.

I'm more than happy for you to look at at, I haven't had the time I
hoped to complete this.

Thanks, Lee.

Philip Yarra writes:

As I found out recently, ECPG is not thread-safe. This is a problem for us, as
we really need to be able to use ECPG* in a multi-threaded C application.
Michael mentioned that Lee is working on fixing these problems, but was not
sure when it would be complete.

So, my questions are:
Lee/anyone, do you have an estimated completion date?
If not, can we (myself plus a "real" C programmer) help you?

Regards, Philip Yarra.

* Yes, we could use libpq, but ECPG minimises our porting from Informix.
Rewriting everything to use libpq would be too expensive.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

--On Friday, March 28, 2003 10:54:23 -0500 Bruce Momjian
<pgman@candle.pha.pa.us> wrote:

I have the libpq patch in my mailbox that just needs configure thread
testing code. I will get that in for 7.4.

I can help with the UnixWare threads stuff. Also, Micheal Meskes now has
an account
on my UnixWare box.

The threads are native on UW, but you need to invoke the cc -Kpthread
option.

LER

-------------------------------------------------------------------------
--

Lee Kindness wrote:

Philip,

You can find the "patch so far" at:

http://services.csl.co.uk/postgresql/

along with a libpq thread-safe patch.

The thread referenced by Bruce's email details some of the issues
still to be looked at. The main work is integration with the build
system (testing for and linking in POSIX threads) and testing.

I'm more than happy for you to look at at, I haven't had the time I
hoped to complete this.

Thanks, Lee.

Philip Yarra writes:

As I found out recently, ECPG is not thread-safe. This is a problem
for us, as we really need to be able to use ECPG* in a
multi-threaded C application. Michael mentioned that Lee is working
on fixing these problems, but was not sure when it would be
complete.

So, my questions are:
Lee/anyone, do you have an estimated completion date?
If not, can we (myself plus a "real" C programmer) help you?

Regards, Philip Yarra.

* Yes, we could use libpq, but ECPG minimises our porting from
Informix. Rewriting everything to use libpq would be too expensive.

--
Bruce Momjian                        |  http://candle.pha.pa.us
pgman@candle.pha.pa.us               |  (610) 359-1001
+  If your life is a hard drive,     |  13 Roberts Road
+  Christ can be your backup.        |  Newtown Square, Pennsylvania
19073

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749

Would someone take those patches and hunt around for proper 'configure'
tests? I can do the configure coding, but I don't know what tests to
make.

http://services.csl.co.uk/postgresql/

---------------------------------------------------------------------------

Larry Rosenman wrote:

--On Friday, March 28, 2003 10:54:23 -0500 Bruce Momjian
<pgman@candle.pha.pa.us> wrote:

I have the libpq patch in my mailbox that just needs configure thread
testing code. I will get that in for 7.4.

I can help with the UnixWare threads stuff. Also, Micheal Meskes now has
an account
on my UnixWare box.

The threads are native on UW, but you need to invoke the cc -Kpthread
option.

LER

-------------------------------------------------------------------------
--

Lee Kindness wrote:

Philip,

You can find the "patch so far" at:

http://services.csl.co.uk/postgresql/

along with a libpq thread-safe patch.

The thread referenced by Bruce's email details some of the issues
still to be looked at. The main work is integration with the build
system (testing for and linking in POSIX threads) and testing.

I'm more than happy for you to look at at, I haven't had the time I
hoped to complete this.

Thanks, Lee.

Philip Yarra writes:

As I found out recently, ECPG is not thread-safe. This is a problem
for us, as we really need to be able to use ECPG* in a
multi-threaded C application. Michael mentioned that Lee is working
on fixing these problems, but was not sure when it would be
complete.

So, my questions are:
Lee/anyone, do you have an estimated completion date?
If not, can we (myself plus a "real" C programmer) help you?

Regards, Philip Yarra.

* Yes, we could use libpq, but ECPG minimises our porting from
Informix. Rewriting everything to use libpq would be too expensive.

--
Bruce Momjian                        |  http://candle.pha.pa.us
pgman@candle.pha.pa.us               |  (610) 359-1001
+  If your life is a hard drive,     |  13 Roberts Road
+  Christ can be your backup.        |  Newtown Square, Pennsylvania
19073

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

--On Friday, March 28, 2003 11:00:47 -0500 Bruce Momjian
<pgman@candle.pha.pa.us> wrote:

Would someone take those patches and hunt around for proper 'configure'
tests? I can do the configure coding, but I don't know what tests to
make.

http://services.csl.co.uk/postgresql/

I do know there are configure tests built for UnixWare's pthreads, but I
don't know where. I have seen them invoked as I've compiled stuff.

Also, once you compile a piece with -Kpthread you must compile ALL of it
with -Kpthread.

You are welcome to read the UnixWare docs at http://www.lerctr.org:8458/ or
on the SCO site.

-------------------------------------------------------------------------
--

Larry Rosenman wrote:

--On Friday, March 28, 2003 10:54:23 -0500 Bruce Momjian
<pgman@candle.pha.pa.us> wrote:

I have the libpq patch in my mailbox that just needs configure thread
testing code. I will get that in for 7.4.

I can help with the UnixWare threads stuff. Also, Micheal Meskes now
has an account
on my UnixWare box.

The threads are native on UW, but you need to invoke the cc -Kpthread
option.

LER

----------------------------------------------------------------------
--- --

Lee Kindness wrote:

Philip,

You can find the "patch so far" at:

http://services.csl.co.uk/postgresql/

along with a libpq thread-safe patch.

The thread referenced by Bruce's email details some of the issues
still to be looked at. The main work is integration with the build
system (testing for and linking in POSIX threads) and testing.

I'm more than happy for you to look at at, I haven't had the time I
hoped to complete this.

Thanks, Lee.

Philip Yarra writes:

As I found out recently, ECPG is not thread-safe. This is a problem
for us, as we really need to be able to use ECPG* in a
multi-threaded C application. Michael mentioned that Lee is
working on fixing these problems, but was not sure when it would
be complete.

So, my questions are:
Lee/anyone, do you have an estimated completion date?
If not, can we (myself plus a "real" C programmer) help you?

Regards, Philip Yarra.

* Yes, we could use libpq, but ECPG minimises our porting from
Informix. Rewriting everything to use libpq would be too
expensive.

--
Bruce Momjian                        |  http://candle.pha.pa.us
pgman@candle.pha.pa.us               |  (610) 359-1001
+  If your life is a hard drive,     |  13 Roberts Road
+  Christ can be your backup.        |  Newtown Square, Pennsylvania
19073

---------------------------(end of
broadcast)--------------------------- TIP 6: Have you searched our
list archives?

http://archives.postgresql.org

--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749

--
Bruce Momjian                        |  http://candle.pha.pa.us
pgman@candle.pha.pa.us               |  (610) 359-1001
+  If your life is a hard drive,     |  13 Roberts Road
+  Christ can be your backup.        |  Newtown Square, Pennsylvania
19073

--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749

I must have spent at least a day looking at configure threads tests
the past few months. Look at the tests in Python, OpenLDAP, MySQL,
Apache 2.0, Perl, ... each and everyone of them is different!
Ouch. Way above my experience in configure.

One of the packages (sorry cannot find my notes on which one!) I
looked at down the test on a per-OS level - supplying the tread
compile and link flags used on that OS and CC to see if they
worked... Probably the simplest way to go?

L.

Bruce Momjian writes:

On Friday 28 March 2003 21:30, you wrote:

Would someone take those patches and hunt around for proper 'configure'
tests? I can do the configure coding, but I don't know what tests to
make.

http://services.csl.co.uk/postgresql/

I also need configure tests for thread for one of my other application and
don't know how to put them. So I have left those as additional ldflags
options to users.

I suggest we somehow test following config to start with

1) Linuxthreads on linux
Threads are usually installed in /usr/ So pthread.h and -lpthread should work
on most setups.

2) Native freeBSD threads
pthread.h in /usr/include and lc_r

3)linuxthreads on freeBSD, ports collection
/usr/local/include/pthreads/linuxthreads/pthread.h
/usr/local/lib/libpthread.so

For testing the library, I think it is enough to init an thread attr. like
pthread_init_attr, that should be enough to test header and library.

I don't how we are going to let people specify thread implementation where
there are multiple thread libraries are available .e.g. freeBSD.

HTH

Shridhar

On Friday 28 March 2003 21:45, you wrote:

For testing the library, I think it is enough to init an thread attr. like
pthread_init_attr, that should be enough to test header and library.

And compiler flags, I forgot.. -pthread on freeBSD/gcc and -D_REENTRANT on
linux at least. FreeBSD needs _THREAD_SAFE as well IIRC.

Shridhar

Yep. I need someone who knows these OS's to get some tests together.

---------------------------------------------------------------------------

Shridhar Daithankar wrote:

On Friday 28 March 2003 21:45, you wrote:

For testing the library, I think it is enough to init an thread attr. like
pthread_init_attr, that should be enough to test header and library.

And compiler flags, I forgot.. -pthread on freeBSD/gcc and -D_REENTRANT on
linux at least. FreeBSD needs _THREAD_SAFE as well IIRC.

Shridhar

---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to majordomo@postgresql.org)

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

On Fri, Mar 28, 2003 at 05:49:04PM +1100, Philip Yarra wrote:

Okay, thanks for that. Have grabbed the file, and started reading. I'll also
wait to hear from Michael or Lee before going too far on this.

Don't be surprised if I'm a bit late in answering emails on the list.
I'm currently very busy.

I'd be interested to get ecpg thread safe.

Is this the only problem you're facing with Informix? I'm currently
working on other Informix compatibility stuff. And I'd like to get that
tested as good as possible.

Michael
--
Michael Meskes
Email: Michael@Fam-Meskes.De
ICQ: 179140304
Go SF 49ers! Go Rhein Fire! Use Debian GNU/Linux! Use PostgreSQL!

On Mon, 31 Mar 2003 00:06, Michael Meskes wrote:

Is this the only problem you're facing with Informix? I'm currently
working on other Informix compatibility stuff. And I'd like to get that
tested as good as possible.

Yes, I recall reading that you were working on better Informix compatibility -
most welcome! Honestly, the changes we needed to make to port from Informix
were pretty minor, as I recall:
- some of the connection syntax differred
- figuring out that OID was not populated in sqlca if not auto-committing
(makes sense really)
- our original code used non-standard ways to name a PREPAREd statement which
threw ECPG (and according the Informix doco shouldn't really have worked for
that either!) so I fixed that in both development streams

You can probably deduce from this that we are not really using a great deal of
functionality in this app - what we have ported so far is a simple logging
server. We really won't be porting any more until ECPG is thread-safe. It's a
major show-stopper, since all of our apps are built on the same (pthreads)
framework.

That said, we would love to help test further Informix compatibility stuff -
anything that makes our future porting process easier would be welcome! Let
us know what you'd like us to test. Diffs are fine.

Regards, Philip.

On Mon, Mar 31, 2003 at 11:23:04AM +1100, Philip Yarra wrote:

That said, we would love to help test further Informix compatibility stuff -
anything that makes our future porting process easier would be welcome! Let
us know what you'd like us to test. Diffs are fine.

Can you take ecpg from CVS? I will not be able to answer email for about
a two week long vacation trip, but everything I've done so far is in
CVS.

Michael
--
Michael Meskes
Email: Michael@Fam-Meskes.De
ICQ: 179140304
Go SF 49ers! Go Rhein Fire! Use Debian GNU/Linux! Use PostgreSQL!

On Fri, 11 Apr 2003 03:20, Michael Meskes wrote:

Can you take ecpg from CVS? I will not be able to answer email for about
a two week long vacation trip, but everything I've done so far is in
CVS.

Hi Michael, I grabbed everything from CVS and tried building, but ran into a
few problems.

Bison 1.35 and 1.50 both have bugs that prevent successful compilation, so I
moved to 1.85. This appears to work okay. Is there some way during configure
to check for broken bison versions and complain loudly?

Once I compiled the source tree, and got it installed, I ran into further
problems as soon as I tried to link against the ecpg libs:

$ gcc -I/usr/local/pgsql/include -L/usr/local/pgsql/lib -lecpg test.c
/usr/local/pgsql/lib/libecpg.so: undefined reference to
`PGTYPESnumeric_from_asc'
/usr/local/pgsql/lib/libecpg.so: undefined reference to `PGTYPESdate_to_asc'
/usr/local/pgsql/lib/libecpg.so: undefined reference to
`PGTYPESinterval_from_asc'
/usr/local/pgsql/lib/libecpg.so: undefined reference to
`PGTYPESinterval_to_asc'
/usr/local/pgsql/lib/libecpg.so: undefined reference to
`PGTYPEStimestamp_from_asc'
/usr/local/pgsql/lib/libecpg.so: undefined reference to `PGTYPESdate_from_asc'
/usr/local/pgsql/lib/libecpg.so: undefined reference to `PGTYPESnumeric_copy'
/usr/local/pgsql/lib/libecpg.so: undefined reference to `PGTYPESinterval_copy'
/usr/local/pgsql/lib/libecpg.so: undefined reference to
`PGTYPEStimestamp_to_asc'
/usr/local/pgsql/lib/libecpg.so: undefined reference to
`PGTYPESnumeric_to_asc'
collect2: ld returned 1 exit status

Is there something obvious I am doing wrong here? Is there something borked in
my build environment? I've attached my test.pgc.

Regards, Philip Yarra.

On Tue, 22 Apr 2003 13:45, Philip Yarra wrote:

Bison 1.35 and 1.50 both have bugs that prevent successful compilation, so
I moved to 1.85.

Correction, bison 1.875. Sorry about that, folks.

Philip.

On Fri, 28 Mar 2003 17:49, Philip Yarra wrote:

On Fri, 28 Mar 2003 13:00, you wrote:

I have put a mailbox file at:

ftp://candle.pha.pa.us/pub/postgresql/mypatches

where there is a partial implementation of thread-safe ecpg. Maybe you
can work with Michael to complete it.

Okay, thanks for that. Have grabbed the file, and started reading. I'll
also wait to hear from Michael or Lee before going too far on this.

I've had a few tries at testing the ECPG threadsafe patch, but so far no luck,
probably due to my own lack of experience. Can someone help me out with how
to test this?

So far I have:
- run configure
- applied Lee's patch to the source tree for 7.3.2
- edited src/Makefile.global, adding -pthread to CFLAGS and and -lpthread to
LIBS
- make and make install
- make our own pthread'd app, linking against newly installed pgsql
Then when I run our app with 3 DB threads, I can watch 2 of the threads fail
to return from EXEC SQL, which is exactly the same behaviour we used to get.

Can anyone point me in the right direction? I'm building on RedHat Linux 7.3
on Intel, using all standard tools for that version (except updated bison
from 1.35 to 1.875). So gcc 2.96, gmake 3.79.1, etc. Let me know if more
details are required.

Regards, Philip Yarra.

Philip Yarra writes:

I've had a few tries at testing the ECPG threadsafe patch, but so
far no luck, probably due to my own lack of experience. Can someone
help me out with how to test this?

So far I have:
- run configure
- applied Lee's patch to the source tree for 7.3.2
- edited src/Makefile.global, adding -pthread to CFLAGS and and
-lpthread to LIBS
- make and make install
- make our own pthread'd app, linking against newly installed pgsql
Then when I run our app with 3 DB threads, I can watch 2 of the threads fai=
l=20
to return from EXEC SQL, which is exactly the same behaviour we used to get.

Can anyone point me in the right direction? I'm building on RedHat Linux 7.=
3=20
on Intel, using all standard tools for that version (except updated bison=
=20
from 1.35 to 1.875). So gcc 2.96, gmake 3.79.1, etc. Let me know if more=20
details are required.

Even with this patch you'll still be limited by the underlying
libpq. So for example you cannot share a connection between threads
without implementing locking to prevent multiple access to that
connection.

I guess the best way would be to open a named connection for EACH
thread. This SHOULD work, but again not tested...

L.

I have libpq thread-safe patch in my queue:

http://momjian.postgresql.org/cgi-bin/pgpatches

Once I have a way to test thread flags for various OS's, I will apply
it. Maybe you guys can help with that too.

---------------------------------------------------------------------------

Lee Kindness wrote:

Philip Yarra writes:

I've had a few tries at testing the ECPG threadsafe patch, but so
far no luck, probably due to my own lack of experience. Can someone
help me out with how to test this?

So far I have:
- run configure
- applied Lee's patch to the source tree for 7.3.2
- edited src/Makefile.global, adding -pthread to CFLAGS and and
-lpthread to LIBS
- make and make install
- make our own pthread'd app, linking against newly installed pgsql
Then when I run our app with 3 DB threads, I can watch 2 of the threads fai=
l=20
to return from EXEC SQL, which is exactly the same behaviour we used to get.

Can anyone point me in the right direction? I'm building on RedHat Linux 7.=
3=20
on Intel, using all standard tools for that version (except updated bison=
=20
from 1.35 to 1.875). So gcc 2.96, gmake 3.79.1, etc. Let me know if more=20
details are required.

Even with this patch you'll still be limited by the underlying
libpq. So for example you cannot share a connection between threads
without implementing locking to prevent multiple access to that
connection.

I guess the best way would be to open a named connection for EACH
thread. This SHOULD work, but again not tested...

L.

---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073