Cannot Restart PostgreSQL-8.1.4

Started by Rich Shepardabout 19 years ago13 messagesgeneral
Jump to latest
#1Rich Shepard
rshepard@appl-ecosys.com

I had a problem with SQL-Ledger running on the local httpd that traced
back to some crufty old libpg.so* from 2003 and 2004 in /usr/local/lib. I
removed those (saved them, actually), ran ldconfig, then restarted both
httpd and postgresql. Unfortunately, the latter really has not started
despite indicating on the console that it has.

Postgres-8.1.4 installed.

Here're the libaries in /usr/lib/:

[rshepard@salmo ~]$ ll /usr/lib/libpq*
-rw-r--r-- 1 root root 149728 2006-05-24 15:06 /usr/lib/libpq.a
lrwxrwxrwx 1 root root 12 2006-07-06 17:19 /usr/lib/libpq.so ->
libpq.so.4.1*
lrwxrwxrwx 1 root root 12 2006-01-27 10:22 /usr/lib/libpq.so.3 ->
libpq.so.3.1*
-rwxr-xr-x 1 root root 110586 2006-01-26 09:49 /usr/lib/libpq.so.3.1*
lrwxrwxrwx 1 root root 12 2006-07-06 17:19 /usr/lib/libpq.so.4 ->
libpq.so.4.1*
-rwxr-xr-x 1 root root 111532 2006-05-24 15:06 /usr/lib/libpq.so.4.1*

And, in case postgres is looking in /usr/local/lib/ it has:

[rshepard@salmo ~]$ ll /usr/local/lib/libpq*
lrwxrwxrwx 1 root root 21 2007-01-24 10:38 /usr/local/lib/libpq.so ->
/usr/lib/libpq.so.4.1*
lrwxrwxrwx 1 root root 21 2007-01-24 10:35 /usr/local/lib/libpq.so.3 ->
/usr/lib/libpq.so.3.1*
lrwxrwxrwx 1 root root 21 2007-01-24 10:38 /usr/local/lib/libpq.so.4 ->
/usr/lib/libpq.so.4.1*

When I run '/etc/rc.d/rc.postgresql start' it returns 'Starting
PostgreSQL: ok', but '/etc/rc.d/rc.postgresql status' returns 'pg_ctl:
neither postmaster nor postgres running' which is true.

I don't know what I did to break the installation, but I would greatly
appreciate help getting it running again ASAP.

TIA,

Rich

--
Richard B. Shepard, Ph.D. | The Environmental Permitting
Applied Ecosystem Services, Inc. | Accelerator(TM)
<http://www.appl-ecosys.com&gt; Voice: 503-667-4517 Fax: 503-667-8863

#2Tom Lane
tgl@sss.pgh.pa.us
In reply to: Rich Shepard (#1)
Re: Cannot Restart PostgreSQL-8.1.4

Rich Shepard <rshepard@appl-ecosys.com> writes:

When I run '/etc/rc.d/rc.postgresql start' it returns 'Starting
PostgreSQL: ok', but '/etc/rc.d/rc.postgresql status' returns 'pg_ctl:
neither postmaster nor postgres running' which is true.

Have you looked in the postmaster log? Under almost all circumstances,
a failing postmaster should emit some sort of bleat on stderr, or
perhaps to syslog if you've configured it that way (and it managed to
get as far as absorbing the contents of postgresql.conf).

Some PG start scripts redirect the postmaster's stderr to /dev/null;
if yours does, change it, at least till you've resolved the problem.

regards, tom lane

#3Rich Shepard
rshepard@appl-ecosys.com
In reply to: Tom Lane (#2)
Re: Cannot Restart PostgreSQL-8.1.4 -- SOLVED!

On Wed, 24 Jan 2007, Tom Lane wrote:

Have you looked in the postmaster log?

Tom,

I went looking for it before writing, but did not find it. Now I have.
The ownership and permissions of /var/lib/pgsql and /var/lib/postgresql were
FUBAR.

I changed the permissions and it's now running.

Thanks,

Rich

--
Richard B. Shepard, Ph.D. | The Environmental Permitting
Applied Ecosystem Services, Inc. | Accelerator(TM)
<http://www.appl-ecosys.com&gt; Voice: 503-667-4517 Fax: 503-667-8863

#4Rich Shepard
rshepard@appl-ecosys.com
In reply to: Tom Lane (#2)
Re: Cannot Restart PostgreSQL-8.1.4

On Wed, 24 Jan 2007, Tom Lane wrote:

Have you looked in the postmaster log?

Tom, et al.:

I got postmaster running now, but have been blocked by Bug #2701 when I
try to run SQL-Ledger. Google has 1,830 hits for it, so I'm far from the
first. :-)

I'm running -8.1.4 here. In /usr/lib/ I have:

lrwxrwxrwx 1 root root 12 2006-07-06 17:19 libpq.so -> libpq.so.4.1*
lrwxrwxrwx 1 root root 12 2006-01-27 10:22 libpq.so.3 -> libpq.so.3.1*
-rwxr-xr-x 1 root root 110586 2006-01-26 09:49 libpq.so.3.1*
lrwxrwxrwx 1 root root 12 2006-07-06 17:19 libpq.so.4 -> libpq.so.4.1*
-rwxr-xr-x 1 root root 111532 2006-05-24 15:06 libpq.so.4.1*

What version of libpq.so do I need?

Thanks,

Rich

--
Richard B. Shepard, Ph.D. | The Environmental Permitting
Applied Ecosystem Services, Inc. | Accelerator(TM)
<http://www.appl-ecosys.com&gt; Voice: 503-667-4517 Fax: 503-667-8863

#5Martijn van Oosterhout
kleptog@svana.org
In reply to: Rich Shepard (#4)
Re: Cannot Restart PostgreSQL-8.1.4

On Wed, Jan 24, 2007 at 12:02:30PM -0800, Rich Shepard wrote:

I got postmaster running now, but have been blocked by Bug #2701 when I
try to run SQL-Ledger. Google has 1,830 hits for it, so I'm far from the
first. :-)

I'm running -8.1.4 here. In /usr/lib/ I have:

<snip>

What version of libpq.so do I need?

The server doesn't care, what matter is what version the *client* (in
this case SQL-Ledger) expects. ldd should tell you that.

(It's ok to have multiple versions of the client library installed...)

Have a nice day,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/

Show quoted text

From each according to his ability. To each according to his ability to litigate.

#6Rich Shepard
rshepard@appl-ecosys.com
In reply to: Martijn van Oosterhout (#5)
Re: Cannot Restart PostgreSQL-8.1.4

On Wed, 24 Jan 2007, Martijn van Oosterhout wrote:

What version of libpq.so do I need?

The server doesn't care, what matter is what version the *client* (in this
case SQL-Ledger) expects. ldd should tell you that.

Martijn,

OK. I'll go back there (which is where I started this journey). I don't
know how to run ldd on a perl script.

(It's ok to have multiple versions of the client library installed...)

Sure. I do for many libraries.

Thanks,

Rich

--
Richard B. Shepard, Ph.D. | The Environmental Permitting
Applied Ecosystem Services, Inc. | Accelerator(TM)
<http://www.appl-ecosys.com&gt; Voice: 503-667-4517 Fax: 503-667-8863

#7Alan Hodgson
ahodgson@simkin.ca
In reply to: Rich Shepard (#6)
Re: Cannot Restart PostgreSQL-8.1.4

On Wednesday 24 January 2007 13:08, Rich Shepard
<rshepard@appl-ecosys.com> wrote:

OK. I'll go back there (which is where I started this journey). I
don't know how to run ldd on a perl script.

It'll be whatever version the DBD::Pg module is compiled to use. Just
rebuilding and installing an updated version of that package will
likely fix your problem.

--
"Pulling together is the aim of despotism and tyranny. Free men pull in
all kinds of directions." -- Terry Pratchett

#8Martijn van Oosterhout
kleptog@svana.org
In reply to: Rich Shepard (#6)
Re: Cannot Restart PostgreSQL-8.1.4

On Wed, Jan 24, 2007 at 01:08:04PM -0800, Rich Shepard wrote:

The server doesn't care, what matter is what version the *client* (in this
case SQL-Ledger) expects. ldd should tell you that.

Martijn,

OK. I'll go back there (which is where I started this journey). I don't
know how to run ldd on a perl script.

Method 1: Find the .so module that perl uses (try locate Pg.so and ldd
that).

Method 2: strace -e open <program>

And look at which version it tries to open.

How exactly is it failing?

Have a nice day,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/

Show quoted text

From each according to his ability. To each according to his ability to litigate.

#9Rich Shepard
rshepard@appl-ecosys.com
In reply to: Martijn van Oosterhout (#8)
Re: Cannot Restart PostgreSQL-8.1.4

On Wed, 24 Jan 2007, Martijn van Oosterhout wrote:

Method 1: Find the .so module that perl uses (try locate Pg.so and ldd
that).

Martijn,

Thank you for teaching me a new trick today. The results of method 1 are:

/lib/libsafe.so.2 (0xb7f76000)
linux-gate.so.1 => (0xffffe000)
libpq.so.3 => /usr/lib/libpq.so.3 (0xb7f5f000)
libc.so.6 => /lib/tls/libc.so.6 (0xb7e1a000)
libdl.so.2 => /lib/tls/libdl.so.2 (0xb7e16000)
libcrypt.so.1 => /lib/tls/libcrypt.so.1 (0xb7de8000)
libresolv.so.2 => /lib/tls/libresolv.so.2 (0xb7dd4000)
libnsl.so.1 => /lib/tls/libnsl.so.1 (0xb7dbd000)
/lib/ld-linux.so.2 (0x80000000)

So it's finding all the libraries it wants, and it's using the older
libpq.so. Interesting.

Method 2: strace -e open <program>

It's a perl script -- actually, a large family of scripts -- that act as
middleware between a web browser and an httpd. I don't see anything in the
login.pl output that looks suspicious.

How exactly is it failing?

I open a new firefox tab and open http://localhost/sql-ledger/login.pl.
After entering my username and password and pressing [Enter] I get an
internal server error:

The server encountered an internal error or misconfiguration and was unable
to complete your request.

Please contact the server administrator, rshepard@appl-ecosys.com and inform
them of the time the error occurred, and anything you might have done that
may have caused the error.

More information about this error may be available in the server error log.

The log file (/var/log/apache/error_log) shows:

[Wed Jan 24 15:01:20 2007] [error] [client 127.0.0.1] Premature end of
script headers: /usr/local/sql-ledger/login.pl
/usr/local/sql-ledger/login.pl: symbol lookup error:
/usr/lib/perl5/site_perl/5.8.8/i486-linux/auto/DBD/Pg/Pg.so: undefined symbol: PQserverVersion

The version of DBD::Pg is 1.4.9 which is the most recent one available.

Thanks,

Rich

--
Richard B. Shepard, Ph.D. | The Environmental Permitting
Applied Ecosystem Services, Inc. | Accelerator(TM)
<http://www.appl-ecosys.com&gt; Voice: 503-667-4517 Fax: 503-667-8863

#10Chad Wagner
chad.wagner@gmail.com
In reply to: Rich Shepard (#9)
Re: Cannot Restart PostgreSQL-8.1.4

On 1/24/07, Rich Shepard <rshepard@appl-ecosys.com> wrote:

libpq.so.3 => /usr/lib/libpq.so.3 (0xb7f5f000)

So it's finding all the libraries it wants, and it's using the older
libpq.so. Interesting.

Where is your other installation of PostgreSQL installed? I suspect this is
because somewhere along the line you had a partial installation of
PostgreSQL (newer pg_config, but missing libpq.so.4)

[Wed Jan 24 15:01:20 2007] [error] [client 127.0.0.1] Premature end of

script headers: /usr/local/sql-ledger/login.pl
/usr/local/sql-ledger/login.pl: symbol lookup error:
/usr/lib/perl5/site_perl/5.8.8/i486-linux/auto/DBD/Pg/Pg.so: undefined
symbol: PQserverVersion

The version of DBD::Pg is 1.4.9 which is the most recent one available.

The reason is when DBD::Pg probed the PostgreSQL configuration using
pg_config --version it received a response of at least 8.0.0 or later. Yet
when it attempted to link it could only find libpq.so.3. I would try
rebuilding DBD::Pg, and run "make test" to verify whether it works before
installing.

I don't know what the rules are for ld to locate libraries and link them,
but perhaps if ldconfig was not run and libpq.so.4 didn't link to the right
version it used libpq.so.3.

--
Chad
http://www.postgresqlforums.com/

#11Rich Shepard
rshepard@appl-ecosys.com
In reply to: Alan Hodgson (#7)
Re: Cannot Restart PostgreSQL-8.1.4

On Wed, 24 Jan 2007, Alan Hodgson wrote:

It'll be whatever version the DBD::Pg module is compiled to use. Just
rebuilding and installing an updated version of that package will likely
fix your problem.

Alan,

I'm not following you. DBD::Pg is the most current version (1.49 from May
2006). That I built and installed this morning. It's calling
/usr/lib/libpg.so.3.1, but I also have /usr/lib/ligpg.so.4.1. The perl
script, login.pl, hasn't changed in a couple of years.

Rich

--
Richard B. Shepard, Ph.D. | The Environmental Permitting
Applied Ecosystem Services, Inc. | Accelerator(TM)
<http://www.appl-ecosys.com&gt; Voice: 503-667-4517 Fax: 503-667-8863

#12Tom Lane
tgl@sss.pgh.pa.us
In reply to: Rich Shepard (#11)
Re: Cannot Restart PostgreSQL-8.1.4

Rich Shepard <rshepard@appl-ecosys.com> writes:

I'm not following you. DBD::Pg is the most current version (1.49 from May
2006). That I built and installed this morning. It's calling
/usr/lib/libpg.so.3.1, but I also have /usr/lib/ligpg.so.4.1.

Somehow DBD::Pg picked up the wrong (older) version of libpq.so;
you need to look into why. Wrong search path while building maybe?

regards, tom lane

#13Rich Shepard
rshepard@appl-ecosys.com
In reply to: Tom Lane (#12)
Re: Cannot Restart PostgreSQL-8.1.4

On Wed, 24 Jan 2007, Tom Lane wrote:

Somehow DBD::Pg picked up the wrong (older) version of libpq.so; you need
to look into why. Wrong search path while building maybe?

Tom,

I _thought_ that might be what happened, but I wanted to make sure before
I dug myself into a hole.

You are absolutely correct. I'll bet it looked first at /usr/local/lib and
found the really old versions of libpq.so. I had cleaned those out, but
after making and installing Pg.so.

I looked through the DBD::Pg source files and the paths looked OK to my
naive eyes. So I just re-ran the 'perl Makefile.PL, make, make test, and
make install' series. All tests passed, and SQL-Ledger now comes up. Whew!!

I'll bet those old libraries were left over from Red Hat 7.3; it was in
September of 2003 that I migrated to Slackware.

You have my sincere thanks for clearing up a mess that occupied me since
yesterday afternoon.

Rich

--
Richard B. Shepard, Ph.D. | The Environmental Permitting
Applied Ecosystem Services, Inc. | Accelerator(TM)
<http://www.appl-ecosys.com&gt; Voice: 503-667-4517 Fax: 503-667-8863