BUG #18179: Cluster History Error

On Thu, Nov 02, 2023 at 09:59:22AM +0000, PG Bug reporting form wrote:

I'm getting an error on my replica cluster. I could not find the cause or
solution of this error.
Error detail:could not restore file "00000004.history" from archive: command
not found

You will need to provide more evidence if you think that this is a bug
in the core code of Postgres. From what you are reporting, this is
just saying that a restore_command has failed before it could not find
the command to use. This depends on your configuration and the
environment setup where Postgres is running.
--
Michael

The log of the error I encountered is as follows:

```
● postgresql@13-mydb.service - PostgreSQL Cluster 13-mydb
Loaded: loaded (/lib/systemd/system/postgresql@.service;
enabled-runtime; vendor preset: enabled)
Active: failed (Result: protocol) since Thu 2023-11-09 17:01:44 +03;
1min 32s ago
Process: 767207 ExecStart=/usr/bin/pg_ctlcluster
--skip-systemctl-redirect 13-mydb start (code=exited, status=1/FAILURE)
Nov 09 17:01:44 se-prd-db02 postgresql@13-mydb[767207]: sh: 1: pgbackrest:
not found
Nov 09 17:01:44 se-prd-db02 postgresql@13-mydb[767207]: 2023-11-09
17:01:44.700 +03 [767228] FATAL: could not restore file "00000004.history"
from archive: command not found
Nov 09 17:01:44 se-prd-db02 postgresql@13-mydb[767207]: 2023-11-09
17:01:44.703 +03 [767227] LOG: startup process (PID 767228) exited with
exit code 1
Nov 09 17:01:44 se-prd-db02 postgresql@13-mydb[767207]: 2023-11-09
17:01:44.703 +03 [767227] LOG: aborting startup due to startup process
failure
Nov 09 17:01:44 se-prd-db02 postgresql@13-mydb[767207]: 2023-11-09
17:01:44.745 +03 [767227] LOG: database system is shut down
Nov 09 17:01:44 se-prd-db02 postgresql@13-mydb[767207]: pg_ctl: could not
start server
Nov 09 17:01:44 se-prd-db02 postgresql@13-mydb[767207]: Examine the log
output.
Nov 09 17:01:44 se-prd-db02 systemd[1]: postgresql@13-mydb.service: Can't
open PID file /run/postgresql/13-mydb.pid (yet?) after start: Operation not
permitted
Nov 09 17:01:44 se-prd-db02 systemd[1]: postgresql@13-mydb.service: Failed
with result 'protocol'.
Nov 09 17:01:44 se-prd-db02 systemd[1]: Failed to start PostgreSQL Cluster
13-mydb.
```

Based on this, I checked the `postgresql.auto.conf` on my master server,
and the restore information found there is as follows:

```
# Recovery settings generated by pgBackRest restore on 2021-09-29 01:04:40
restore_command = 'pgbackrest --stanza=mydb_stanza archive-get %f "%p'
```

The archive command information in `postgresql.conf` is as follows:

```
# - Archiving -
archive_mode = on
archive_command = 'pgbackrest --stanza=mydb_stanza archive-push %p'
```
Thank you.

On Mon, 2023-11-13 at 21:00 +0300, Abdullah Ergin wrote:

 Nov 09 17:01:44 se-prd-db02 postgresql@13-mydb[767207]: sh: 1: pgbackrest: not found
 Nov 09 17:01:44 se-prd-db02 postgresql@13-mydb[767207]: 2023-11-09 17:01:44.700 +03 [767228] FATAL: could not restore file "00000004.history" from archive: command not found
 Nov 09 17:01:44 se-prd-db02 postgresql@13-mydb[767207]: 2023-11-09 17:01:44.703 +03 [767227] LOG: startup process (PID 767228) exited with exit code 1
 Nov 09 17:01:44 se-prd-db02 postgresql@13-mydb[767207]: 2023-11-09 17:01:44.703 +03 [767227] LOG: aborting startup due to startup process failure
 Nov 09 17:01:44 se-prd-db02 postgresql@13-mydb[767207]: 2023-11-09 17:01:44.745 +03 [767227] LOG: database system is shut down

Based on this, I checked the `postgresql.auto.conf` on my master server, and the restore information found there is as follows:

# Recovery settings generated by pgBackRest restore on 2021-09-29 01:04:40
restore_command = 'pgbackrest --stanza=mydb_stanza archive-get %f "%p'

Use the absolute path for "pgbackrest".
Did you install pgBackRest on the standby?

Yours,
Laurenz Albe

Elbette, işte İngilizce çevirisi:

---

I'm relatively new to database expertise, are you referring to the absolute
path as follows:

postgresql.auto.conf;
#Recovery settings generated by pgBackRest restore on 2021-09-29 01:04:40
restore_command = '/usr/bin/pgbackrest --stanza=mydb_stanza archive-get %f
"%p"'

postgresql.conf;
# - Archiving -
archive_mode = on
archive_command = '/usr/bin/pgbackrest --stanza=mydb_stanza archive-push %p'

Will this operation affect my other clusters since I will perform it on the
master server?
Thanks

Laurenz Albe <laurenz.albe@cybertec.at>, 13 Kas 2023 Pzt, 22:05 tarihinde
şunu yazdı:

On Mon, 2023-11-13 at 22:42 +0300, Abdullah Ergin wrote:

I'm relatively new to database expertise, are you referring to the absolute path as follows:

postgresql.auto.conf;
#Recovery settings generated by pgBackRest restore on 2021-09-29 01:04:40
restore_command = '/usr/bin/pgbackrest --stanza=mydb_stanza archive-get %f "%p"'

Yes, precisely. But if te executable is in /usr/bin, it should be found
anyway... Is there really a /usr/bin/pgbackrest on the standby server?

Yours,
Laurenz Albe

pgBackRest is only installed on my master and backup servers. The directory
where it is installed is '/usr/bin/pgbackrest.'

However, pgBackRest is not installed on my replica server.

Laurenz Albe <laurenz.albe@cybertec.at>, 13 Kas 2023 Pzt, 23:13 tarihinde
şunu yazdı:

On Tue, 2023-11-14 at 00:04 +0300, Abdullah Ergin wrote:

Laurenz Albe <laurenz.albe@cybertec.at>, 13 Kas 2023 Pzt, 23:13 tarihinde şunu yazdı:

On Mon, 2023-11-13 at 22:42 +0300, Abdullah Ergin wrote:

I'm relatively new to database expertise, are you referring to the absolute path as follows:

postgresql.auto.conf;
#Recovery settings generated by pgBackRest restore on 2021-09-29 01:04:40
restore_command = '/usr/bin/pgbackrest --stanza=mydb_stanza archive-get %f "%p"'

Yes, precisely.  But if te executable is in /usr/bin, it should be found
anyway... Is there really a /usr/bin/pgbackrest on the standby server?

pgBackRest is only installed on my master and backup servers. The directory where
it is installed is '/usr/bin/pgbackrest.'

However, pgBackRest is not installed on my replica server.

Then you shouldn't use it in your "restore_command". Best comment that out.

This is no PostgreSQL bug.

Yours,
Laurenz Albe