Broken postgres links need to find callers

On 10/31/18 10:18 AM, Rich Shepard wrote:

  I managed to mess up postgresql-10.3 on this Slackware-14.2 desktop
server/workstation. It worked OK until I tried adding access to an another
application.

  For a reason I don't know, adding that listening address revealed that
many sym links are looking for 10.2 directories. I've found and fixed many
of these and need help finding the rest (perhaps only one more).

What was the listening address you added?

What happens if you remove the listening address?

Did you recently upgrade from 10.2 --> 10.3?

  Running pg_ctl start fails:
$ pg_ctl start -D /var/lib/pgsql/10.3/data/
waiting for server to start....2018-10-31 10:02:01.312 PDT [1285] FATAL:
could not open directory "/usr/share/postgresql-10.2/timezonesets": No such
file or directory 2018-10-31 10:02:01.312 PDT [1285] HINT: This may
indicate
an incomplete PostgreSQL installation, or that the file
"/usr/lib/postgresql/10.3/bin/postgres" has been moved away from its proper
location.
 stopped waiting

  What file is looking for the timezonesets directory? It's using a
symlink
but the error message is not telling me where it is so I can remove it and
replace it with a symlink to the proper postgresql-10.3/timezonesets
directory.

TIA,

Rich

--
Adrian Klaver
adrian.klaver@aklaver.com

On Wed, 31 Oct 2018, Adrian Klaver wrote:

What was the listening address you added?

Adrian,

I added the host name.

What happens if you remove the listening address?

I don't think this makes a difference. pg_ctl is calling a program that
looks for timezonesets in the wrong directory

Did you recently upgrade from 10.2 --> 10.3?

On March 1st of this year.

Regards,

Rich

On 10/31/18 10:55 AM, Rich Shepard wrote:

On Wed, 31 Oct 2018, Adrian Klaver wrote:

What was the listening address you added?

Adrian,

  I added the host name.

What happens if you remove the listening address?

  I don't think this makes a difference. pg_ctl is calling a program that
looks for timezonesets in the wrong directory

You said it made a difference when you added it, just trying to figure
out if removing it also makes a difference. If not then we need to look
elsewhere for an explanation.

Did you recently upgrade from 10.2 --> 10.3?

  On March 1st of this year.

Regards,

Rich

--
Adrian Klaver
adrian.klaver@aklaver.com

On Wed, 31 Oct 2018, Adrian Klaver wrote:

You said it made a difference when you added it, just trying to figure out if
removing it also makes a difference. If not then we need to look elsewhere
for an explanation.

Adrian,

Each time I hit a broken symlink pg_ctl told me which link was broken. For
example, the libpgtypes.so* files were sought in
/usr/lib/postgresql/10.2/lib/ when they're actually now in
/usr/lib/postgresql/10.3/lib/. Removing broken links and creating new,
valid ones fixes them.

So, I'm seeking the library or executable file that's finding timezonesets
in the no-longer existing ../10.2/ directory.

Regards,

Rich

On 10/31/18 11:15 AM, Rich Shepard wrote:

On Wed, 31 Oct 2018, Adrian Klaver wrote:

You said it made a difference when you added it, just trying to figure
out if removing it also makes a difference. If not then we need to
look elsewhere for an explanation.

Adrian,

  Each time I hit a broken symlink pg_ctl told me which link was
broken. For
example, the libpgtypes.so* files were sought in
/usr/lib/postgresql/10.2/lib/ when they're actually now in
/usr/lib/postgresql/10.3/lib/. Removing broken links and creating new,
valid ones fixes them.

  So, I'm seeking the library or executable file that's finding
timezonesets
in the no-longer existing ../10.2/ directory.

If you refuse to implement the suggestions I asked for then I cannot
help you, as you are now off on a different tangent. One that on the
face of it is dangerous.

Regards,

Rich

--
Adrian Klaver
adrian.klaver@aklaver.com

On Wed, 31 Oct 2018, Adrian Klaver wrote:

If you refuse to implement the suggestions I asked for then I cannot help
you, as you are now off on a different tangent. One that on the face of it is
dangerous.

In var/lib/pgsql/10.3/data/postgresql.conf:

# - Connection Settings -

_addresses = 'localhost'

No listener_addresses present.

$ pg_ctl start -D /var/lib/pgsql/10.3/data/
waiting for server to start....2018-10-31 18:33:03.323 GMT [3352] LOG: unrecognized configuration parameter "_addresses" in file "/var/lib/pgsql/10.3/data/postgresql.conf" line 59
2018-10-31 18:33:03.323 GMT [3352] FATAL: configuration file "/var/lib/pgsql/10.3/data/postgresql.conf" contains errors
stopped waiting
pg_ctl: could not start server

Rich

"Rich" == Rich Shepard <rshepard@appl-ecosys.com> writes:

Rich> I managed to mess up postgresql-10.3 on this Slackware-14.2
Rich> desktop server/workstation. It worked OK until I tried adding
Rich> access to an another application.

Rich> waiting for server to start....2018-10-31 10:02:01.312 PDT [1285] FATAL:
Rich> could not open directory "/usr/share/postgresql-10.2/timezonesets": No such
Rich> file or directory 2018-10-31 10:02:01.312 PDT [1285] HINT: This may indicate
Rich> an incomplete PostgreSQL installation, or that the file
Rich> "/usr/lib/postgresql/10.3/bin/postgres" has been moved away from its proper
Rich> location.

Is there a pg_config binary in /usr/lib/postgresql/10.3/bin/ and if so,
what is the output of /usr/lib/postgresql/10.3/bin/pg_config --sharedir

Also what is the output of /usr/lib/postgresql/10.3/bin/postgres -V

The most plausible explanation I can see for what you're seeing there is
that what you have as /usr/lib/postgresql/10.3/bin/postgres is not
actually the 10.3 binary but rather the 10.2 one. There should be no
symlinks involved there - the path that is reported in the error message
is the one that the postgres binary actually did try to open.

--
Andrew (irc:RhodiumToad)

On Wed, 31 Oct 2018, Andrew Gierth wrote:

Is there a pg_config binary in /usr/lib/postgresql/10.3/bin/ and if so,
what is the output of /usr/lib/postgresql/10.3/bin/pg_config --sharedir

Andrew,

Yes, pg_config is present but pointing to the wrong directory:
# /usr/lib/postgresql/10.3/bin/pg_config --sharedir
/usr/share/postgresql-10.2

However, the file dates are that of the upgrade from 10.2 to 10.3:

-rwxr-xr-x 1 root root 7096448 Mar 1 2018 postgres*
lrwxrwxrwx 1 root root 8 Mar 1 2018 postmaster -> postgres*
-rwxr-xr-x 1 root root 514732 Mar 1 2018 psql*

and the postgres version is 10.3:

Also what is the output of /usr/lib/postgresql/10.3/bin/postgres -V

# /usr/lib/postgresql/10.3/bin/postgres -V
postgres (PostgreSQL) 10.3

The most plausible explanation I can see for what you're seeing there is
that what you have as /usr/lib/postgresql/10.3/bin/postgres is not
actually the 10.3 binary but rather the 10.2 one. There should be no
symlinks involved there - the path that is reported in the error message
is the one that the postgres binary actually did try to open.

Can pg_config be corrected independent of anything else. That seems to be
where the blockage is found.

Thanks,

Rich

On 10/31/18 11:33 AM, Rich Shepard wrote:

On Wed, 31 Oct 2018, Adrian Klaver wrote:

If you refuse to implement the suggestions I asked for then I cannot
help you, as you are now off on a different tangent. One that on the
face of it is dangerous.

  In var/lib/pgsql/10.3/data/postgresql.conf:

# - Connection Settings -

_addresses = 'localhost'

  No listener_addresses present.

As the below shows that is not a valid setting.

Try:

listen_addresses = ''

$ pg_ctl start -D /var/lib/pgsql/10.3/data/
waiting for server to start....2018-10-31 18:33:03.323 GMT [3352] LOG:
unrecognized configuration parameter "_addresses" in file
"/var/lib/pgsql/10.3/data/postgresql.conf" line 59
2018-10-31 18:33:03.323 GMT [3352] FATAL:  configuration file
"/var/lib/pgsql/10.3/data/postgresql.conf" contains errors
 stopped waiting
pg_ctl: could not start server

Rich

--
Adrian Klaver
adrian.klaver@aklaver.com

On 10/31/18 11:48 AM, Rich Shepard wrote:

On Wed, 31 Oct 2018, Andrew Gierth wrote:

Is there a pg_config binary in /usr/lib/postgresql/10.3/bin/ and if so,
what is the output of /usr/lib/postgresql/10.3/bin/pg_config --sharedir

Andrew,

  Yes, pg_config is present but pointing to the wrong directory:
# /usr/lib/postgresql/10.3/bin/pg_config --sharedir
/usr/share/postgresql-10.2

What does:

/usr/lib/postgresql/10.3/bin/pg_config --version

show?

What does:

ps ax | grep post

show?

  However, the file dates are that of the upgrade from 10.2 to 10.3:

-rwxr-xr-x 1 root root 7096448 Mar  1  2018 postgres*
lrwxrwxrwx 1 root root       8 Mar  1  2018 postmaster -> postgres*
-rwxr-xr-x 1 root root  514732 Mar  1  2018 psql*

and the postgres version is 10.3:

Also what is the output of  /usr/lib/postgresql/10.3/bin/postgres -V

# /usr/lib/postgresql/10.3/bin/postgres -V
postgres (PostgreSQL) 10.3

The most plausible explanation I can see for what you're seeing there is
that what you have as /usr/lib/postgresql/10.3/bin/postgres is not
actually the 10.3 binary but rather the 10.2 one. There should be no
symlinks involved there - the path that is reported in the error message
is the one that the postgres binary actually did try to open.

  Can pg_config be corrected independent of anything else. That seems
to be
where the blockage is found.

Thanks,

Rich

--
Adrian Klaver
adrian.klaver@aklaver.com

On Wed, 31 Oct 2018, Adrian Klaver wrote:

listen_addresses = ''

Adrian,

#listen_addresses = ''

$ pg_ctl start -D /var/lib/pgsql/10.3/data/
waiting for server to start....2018-10-31 12:12:39.530 PDT [4398] FATAL: could not open directory "/usr/share/postgresql-10.2/timezonesets": No such file or directory
2018-10-31 12:12:39.530 PDT [4398] HINT: This may indicate an incomplete PostgreSQL installation, or that the file "/usr/lib/postgresql/10.3/bin/postgres" has been moved away from its proper location.
stopped waiting
pg_ctl: could not start server

listen_addresses = ''

$ pg_ctl start -D /var/lib/pgsql/10.3/data/
waiting for server to start....2018-10-31 12:13:28.141 PDT [4413] FATAL: could not open directory "/usr/share/postgresql-10.2/timezonesets": No such file or directory
2018-10-31 12:13:28.141 PDT [4413] HINT: This may indicate an incomplete PostgreSQL installation, or that the file "/usr/lib/postgresql/10.3/bin/postgres" has been moved away from its proper location.
stopped waiting
pg_ctl: could not start server

Thanks,

Rich

On Wed, 31 Oct 2018, Adrian Klaver wrote:

What does:

/usr/lib/postgresql/10.3/bin/pg_config --version

show?

# /usr/lib/postgresql/10.3/bin/pg_config --version
PostgreSQL 10.3

What does:

ps ax | grep post

show?

# ps ax | grep post
1307 ? Ss 1:29 /usr/libexec/postfix/master -w
4445 ? S 0:00 spawn -n policy -t unix user=nobody argv=/usr/bin/perl /usr/lib/postfix/postfix-policyd-spf-perl
4446 ? Ss 0:00 /usr/bin/perl /usr/lib/postfix/postfix-policyd-spf-perl
4456 ? S 0:00 spawn -n policy -t unix user=nobody argv=/usr/bin/perl /usr/lib/postfix/postfix-policyd-spf-perl
4457 ? Ss 0:00 /usr/bin/perl /usr/lib/postfix/postfix-policyd-spf-perl
4483 pts/2 S+ 0:00 grep post

Rich

On 10/31/18 12:14 PM, Rich Shepard wrote:

On Wed, 31 Oct 2018, Adrian Klaver wrote:

listen_addresses = ''

Adrian,

#listen_addresses = ''

$ pg_ctl start -D /var/lib/pgsql/10.3/data/
waiting for server to start....2018-10-31 12:12:39.530 PDT [4398]
FATAL:  could not open directory
"/usr/share/postgresql-10.2/timezonesets": No such file or directory
2018-10-31 12:12:39.530 PDT [4398] HINT:  This may indicate an
incomplete PostgreSQL installation, or that the file
"/usr/lib/postgresql/10.3/bin/postgres" has been moved away from its
proper location.
 stopped waiting
pg_ctl: could not start server

What does:

pg_ctl --version

show?

listen_addresses = ''

$ pg_ctl start -D /var/lib/pgsql/10.3/data/
waiting for server to start....2018-10-31 12:13:28.141 PDT [4413]
FATAL:  could not open directory
"/usr/share/postgresql-10.2/timezonesets": No such file or directory
2018-10-31 12:13:28.141 PDT [4413] HINT:  This may indicate an
incomplete PostgreSQL installation, or that the file
"/usr/lib/postgresql/10.3/bin/postgres" has been moved away from its
proper location.
 stopped waiting
pg_ctl: could not start server

So it seems listen_addresses is not the issue at this point.

So when you added the new application did you make any other changes?

At this point you need to get back to two discreet Postgres installs
10.2 and 10.3.

1) Remove all the symlinks you made.

2) In the 10.3/ verify that the programs in bin/ are actually the 10.3
ones using prgm_name --version.

3) Try pg_ctl start.

Thanks,

Rich

--
Adrian Klaver
adrian.klaver@aklaver.com

On Wed, 31 Oct 2018, Adrian Klaver wrote:

What does:
pg_ctl --version
show?

# pg_ctl --version
pg_ctl (PostgreSQL) 10.3

So when you added the new application did you make any other changes?

I did not add another application; grass has been installed here for
decades. Because I could not connect to the postgres database for a spatial
project it was suggested that I expand the listen_addresses to include the
server name, too.

At this point you need to get back to two discreet Postgres installs 10.2 and
10.3.

I did not have two distinct installations of postgres, only the 10.3
version. It was some of the directories labeled 10.2/ that seem to be the
issue.

When I run pg_config --configure this is the result:
# ./pg_config --configure
'--prefix=/usr/lib/postgresql/10.2' '--sysconfdir=/etc/postgresql/10.2'
'--includedir=/usr/include' '--datarootdir=/usr/share' '--mandir=/usr/man'
'--docdir=/usr/doc/postgresql-10.3' '--datadir=/usr/share/postgresql-10.2'
'--with-openssl' '--with-tcl' '--with-perl' '--with-python' '--with-libxml'
'--with-libxslt' '--enable-thread-safety'
'--with-system-tzdata=/usr/share/zoneinfo' '--disable-nls'
'--build=i586-slackware-linux' 'build_alias=i586-slackware-linux'
'CFLAGS=-O2 -march=i586 -mtune=i686' 'PKG_CONFIG_PATH=/usr/lib/pkgconfig

Notice that the prefix, sysconfdir, and datadir specify 10.2. But, the
build script shows the version as 10.3 and uses that to configure the build
(entire script will be provided on request):

PRGNAM=postgresql
VERSION=${VERSION:-10.3}
BUILD=${BUILD:-1}
TAG=${TAG:-_SBo}

PG_VERSION=${PG_VERSION:-10.3}
PG_PORT=${PG_PORT:-5432}
PG_UID=${PG_UID:-209}
PG_GID=${PG_GID:-209}
...
./configure \
--prefix=/usr/lib${LIBDIRSUFFIX}/$PRGNAM/$PG_VERSION \
--sysconfdir=/etc/$PRGNAM/$PG_VERSION \
--includedir=/usr/include \
--datarootdir=/usr/share \
--mandir=/usr/man \
--docdir=/usr/doc/$PRGNAM-$VERSION \
--datadir=/usr/share/$PRGNAM-$PG_VERSION \

The build script sets the prefix, sysconfdir, and datadir to $PG_VERSION
which is defined as 10.3 so why 10.2 shows up in pg_config makes no sense to
me. And why psql worked without issue from the upgrade date of March 1 to
today with this inconsistency also puzzles me.

If it matters, there's no /etc/postgresql/ and none in the backups since
the beginning of August.

Regards,

Rich

On 10/31/18 1:09 PM, Rich Shepard wrote:

On Wed, 31 Oct 2018, Adrian Klaver wrote:

What does:
pg_ctl --version
show?

# pg_ctl --version
pg_ctl (PostgreSQL) 10.3

So when you added the new application did you make any other changes?

  I did not add another application; grass has been installed here for
decades. Because I could not connect to the postgres database for a spatial
project it was suggested that I expand the listen_addresses to include the
server name, too.

At this point you need to get back to two discreet Postgres installs
10.2 and 10.3.

   I did not have two distinct installations of postgres, only the 10.3
version. It was some of the directories labeled 10.2/ that seem to be the
issue.

Are there actually 10.2/ directories or is that just what you are seeing
in the error messages and the pg_config output?

   When I run pg_config --configure this is the result:

Previously you used:

/usr/lib/postgresql/10.3/bin/pg_config

is that the same as below?:

# ./pg_config --configure

In other words what does:

./pg_config --version

show?

me. And why psql worked without issue from the upgrade date of March 1 to
today with this inconsistency also puzzles me.

Did the server been running continuously from the upgrade to the time
you made the listen_addresses change?

  If it matters, there's no /etc/postgresql/ and none in the backups since
the beginning of August.

Regards,

Rich

--
Adrian Klaver
adrian.klaver@aklaver.com

On Wed, 31 Oct 2018, Adrian Klaver wrote:

Are there actually 10.2/ directories or is that just what you are seeing in
the error messages and the pg_config output?

Adrian,

No 10.2/ directories, only what is shown in the error messages and
pg_config output.

Previously you used:
/usr/lib/postgresql/10.3/bin/pg_config
is that the same as below?:

# ./pg_config --configure

Yes. I was in that directory when I ran pg_config.[1]This prompted me to look for more pg_config files, and I found a symlink in /usr/bin/ that pointed to /usr/lib/postgresql/10.2/bin/pg_config which does not exist. I changed that symlink to point to the 10.3/ pg_config version but there's still a broken link somewhere because pg_ctl start cannot find the correct directory for timezonesets.

In other words what does:
./pg_config --version
show?

This shows 10.3

Did the server been running continuously from the upgrade to the time you
made the listen_addresses change?

Yes, other than a few kernel upgrades.

[1]: This prompted me to look for more pg_config files, and I found a symlink in /usr/bin/ that pointed to /usr/lib/postgresql/10.2/bin/pg_config which does not exist. I changed that symlink to point to the 10.3/ pg_config version but there's still a broken link somewhere because pg_ctl start cannot find the correct directory for timezonesets.
in /usr/bin/ that pointed to /usr/lib/postgresql/10.2/bin/pg_config which
does not exist. I changed that symlink to point to the 10.3/ pg_config
version but there's still a broken link somewhere because pg_ctl start
cannot find the correct directory for timezonesets.

Regards,

Rich

On Wed, 31 Oct 2018, Rich Shepard wrote:

[1] This prompted me to look for more pg_config files, and I found a symlink
in /usr/bin/ that pointed to /usr/lib/postgresql/10.2/bin/pg_config which
does not exist. I changed that symlink to point to the 10.3/ pg_config
version but there's still a broken link somewhere because pg_ctl start
cannot find the correct directory for timezonesets.

Thanks for pointing me to the links in /usr/bin/, Adrian. All of them are
pointing to the mythical 10.2/ directory. I'll fix those links and report
the results of running pg_ctl start.

Regards,

Rich

On Wed, 31 Oct 2018, Rich Shepard wrote:

I'll fix those links and report the results of running pg_ctl start.

Still bad links remaining. Some of those symlinks in /usr/bin/ dated back
to versons 9.4 and 9.6. Why they were not removed during upgrades remains a
mystery.

Rich

On Wed, 31 Oct 2018, Rich Shepard wrote:

Still bad links remaining.

Every pg_* not in /usr/lib/postgresql/10.3/bin/ now points to its namesake
there.

Question: if pg_dump, pg_dumpall, pg_restore, pg_ctl, and pg_controldata
have symlinks in /usr/bin/ do they also need symlinks in /bin/? The ones
found there dated back to 2010 and I don't know that they're needed now.

pg_ctl start still cannot find the proper timezonesets/ directory. If you
have suggestions what pg_ctl is calling that looks in the non-existent 10.2/
directory instead of the existing 10.3/ directory please let me know.

Rich