Pgbackrest fails due after an ISP change
List,
I am running a pgbackrest(2.52.1) backup setup for a postgres(16)
server(RHEL9.4) to a repo server RHEL 9.4..
The pgbackrest was working fine and the remote repo server got regular
backups through cron scheduler.
Suddenly there was an ISP change at our end, the VPN tunnel between DB
server and Repo server was lost for 24 hours due to VPN tunnel crash.
later I restored the VPN tunnel
After that the cron scheduler does not perform the pgbackrest backup. It
fails!!!
So I performed a full backup to overcome this .. But it fails with the
cron command ( 38 14 * * 1 pgbackrest --type=full --stanza=My_Repo
backup)
ERROR: [082]: WAL segment 000000010000022000000038 was not archived before
the 60000ms timeout
HINT: check the archive_command to
ensure that all options are correct (especially --stanza).
HINT: check the PostgreSQL server log
for errors.
HINT: run the 'start' command if the
stanza was previously stopped.
--------------------------------------------------------------------
If SUBMITTING AN ISSUE please provide
the following information:
version: 2.52.1
command: backup
options: --delta
--exec-id=492671-6f2068fc --log-level-console=info --log-level-file=debug
--pg1-host=10.x.y.202 --pg1-host-user=enterprisedb
--pg1-path=/data/edb/as16/data --pg-version-force=16 --process-max=5
--repo1-block --repo1-bundle --repo1-cipher-pass=<redacted>
--repo1-cipher-type=aes-256-cbc --repo1-path=/DBBACKUP
--repo1-retention-diff=3 --repo1-retention-full=3 --stanza=KTA_Repo
--start-fast --type=full
2025-04-21 14:39:07.335 P00 INFO: backup command end: aborted with
exception [082]
2025-04-21 14:39:07.335 P00 DEBUG: command/exit::exitSafe: => 82
2025-04-21 14:39:07.936 P00 DEBUG: main::main: => 82
How can I make the full backup command not to check the WAL was archived
or not to the repo server for atleast once ?
Any hint appreciated ..
Thanks
Krishane
On Mon, Apr 21, 2025 at 9:03 AM KK CHN <kkchn.in@gmail.com> wrote:
ERROR: [082]: WAL segment 000000010000022000000038 was not archived before
the 60000ms timeout
...
How can I make the full backup command not to check the WAL was archived
or not to the repo server for atleast once ?
You cannot. WAL is an integral part of the backups. You should run the
pgbackrest "check" command and look at your Postgres logs to figure out why
WAL files are not being archived. Once that is solved, try the backup again.
https://pgbackrest.org/command.html#command-check
Cheers,
Greg
--
Crunchy Data - https://www.crunchydata.com
Enterprise Postgres Software Products & Tech Support
Hi List,
Thanks for the clarification.
If I want to start a fresh backup to the same repo server ( the one which
fails the pgbackrest backup due to the file system full outage) what all
are the steps I have to perform on the Repo Server and the DB server ?
AFAIK, These all are the steps required , correct me if I'm wrong.
1. Stop the stanza on both repo server as well as on DB server (
$sudo -u myuser pgbackrest --stanza=My_RepoX --force stop)
2. Then delete the stanza from either Repo Server or DB server ( by the
delete stanza option ) Is the deletion of the stanza needed to be
performed only at the Repo server is enough ?
3. After deletion of the stanza, delete the folders in the repo server
archive, backup respectively by ( rm -f archive , rm -rf backup
folders for My_RepoX ?)
4. Then execute the same stanza create command from the reposerver ? (
$ sudo -u myuser pgbackrest --stanza=My_RepoX --log-level-console=info
stanza-create)?
5. Then Check the stanza creation and config all perfect by ( sudo -u
myuser pgbackrest --stanza=My_RepoX --log-level-console=info check )
6. If the step 5 success, schedule the full backup from the cron
I hope this will help me to take backup afresh in case an existing
pgbackrest backup fails ( whatever the stupid reason what happened as in
my case ) .
( In my case pgbackrest fails due to the scenario I faced, I mean due to
the file system full and didn't notice it for 24 hours, even after made
space of the partition /root which got full, later on the pgbackrest
backups from cron scheduler unable to complete the backup)
If anything is wrong or not sufficient as in the 6 steps AFAIK (As this
a live Repo Server for other DB Clusters too, more than 10 nos, nothing
should go wrong) , please correct/guide me
Thank you,
Krishane
On Mon, Apr 21, 2025 at 9:49 PM Greg Sabino Mullane <htamfids@gmail.com>
wrote:
Show quoted text
On Mon, Apr 21, 2025 at 9:03 AM KK CHN <kkchn.in@gmail.com> wrote:
ERROR: [082]: WAL segment 000000010000022000000038 was not archived
before the 60000ms timeout...
How can I make the full backup command not to check the WAL was archived
or not to the repo server for atleast once ?You cannot. WAL is an integral part of the backups. You should run the
pgbackrest "check" command and look at your Postgres logs to figure out why
WAL files are not being archived. Once that is solved, try the backup again.https://pgbackrest.org/command.html#command-check
Cheers,
Greg--
Crunchy Data - https://www.crunchydata.com
Enterprise Postgres Software Products & Tech Support