could not fsync file ""pg_wal/xlogtemp.23229"": Input/output error
Hi All,
Postgres 11 on RHEL 7.9
We filled out the wal archive disk, and then those messages appear
2021-01-15 01:18:47.284
CST,"replicator","consprod",3510,"[local]",5fdebf22.db6,21,"SELECT",2020-12-19
21:04:02 CST,35/46792889,437317854,PANIC,58030,"could not fsync file
""pg_wal/xlogtemp.3510"": Input/output error",,,,,,"select
""_consprod_replication"".forwardConfirm(2, 3, '6233786669', '2021-01-15
00:32:32.828355-06'); ",,,"slon.remoteWorkerThread_2"
2021-01-15 01:18:47.508 CST,,,2461,,5fdebf20.99d,23,,2020-12-19 21:04:00
CST,,0,LOG,00000,"server process (PID 3510) was terminated by signal 6:
Aborted","Failed process was running: select
""_consprod_replication"".forwardConfirm(2, 3, '6233786669', '2021-01-15
00:32:32.828355-06'); ",,,,,,,,""
2021-01-15 01:18:47.508 CST,,,2461,,5fdebf20.99d,24,,2020-12-19 21:04:00
CST,,0,LOG,00000,"terminating any other active server processes",,,,,,,,,""
2021-01-15 01:18:47.508
CST,"monitoring","consprod",14606,"10.2.5.12:48018",6001416d.390e,3,"SELECT",2021-01-15
01:17:01 CST,122/339910,0,WARNING,57P02,"terminating connection because of
crash of another server process","The postmaster has commanded this server
process to roll back the current transaction and exit, because another
server process exited abnormally and possibly corrupted shared memory.","In
a moment you should be able to reconnect to the database and repeat your
command.",,,,,,,"psql"
And now I cannot start postgres
2021-01-15 03:04:19.972 CST,,,23229,,6001550d.5abd,6,,2021-01-15 02:40:45
CST,,0,PANIC,58030,"could not fsync file ""pg_wal/xlogtemp.23229"":
Input/output error",,,,,,,,,""2021-01-15 03:04:19.979
CST,,,23226,,6001550d.5aba,2,,2021-01-15 02:40:45 CST,,0,LOG,00000,"startup
process (PID 23229) was terminated by signal 6: Aborted",,,,,,,,,""
2021-01-15 03:04:19.979 CST,,,23226,,6001550d.5aba,3,,2021-01-15 02:40:45
CST,,0,LOG,00000,"aborting startup due to startup process
failure",,,,,,,,,""
2021-01-15 03:04:20.041 CST,,,23226,,6001550d.5aba,4,,2021-01-15 02:40:45
CST,,0,LOG,00000,"database system is shut down",,,,,,,,,""
Any idea to solve this ?
--
Regards,
Soni Maula Harriz
On Fri, 2021-01-15 at 16:13 +0700, Soni M wrote:
We filled out the wal archive disk, and then those messages appear
[...]PANIC,58030,"could not fsync file ""pg_wal/xlogtemp.3510"": Input/output error"
And now I cannot start postgres
[...]PANIC,58030,"could not fsync file ""pg_wal/xlogtemp.23229"": Input/output error"
Any idea to solve this ?
Did you add more disk space?
You won't be able to start PostgreSQL unless you do that.
And no, don't start deleting files in the data directory.
What you can do is move the *complete* data directory to another location
with more disk space and start it there.
Yours,
Laurenz Albe
--
Cybertec | https://www.cybertec-postgresql.com
More disk space now available, but still the error happens and postgres
won't start.
I think it could be corrupted shared buffers, as stated on the log
messages, and perhaps the corruption goes to disk.
On Fri, Jan 15, 2021 at 4:33 PM Laurenz Albe <laurenz.albe@cybertec.at>
wrote:
On Fri, 2021-01-15 at 16:13 +0700, Soni M wrote:
We filled out the wal archive disk, and then those messages appear
[...]PANIC,58030,"could not fsync file ""pg_wal/xlogtemp.3510"":
Input/output error"
And now I cannot start postgres
[...]PANIC,58030,"could not fsync file ""pg_wal/xlogtemp.23229"":
Input/output error"
Any idea to solve this ?
Did you add more disk space?
You won't be able to start PostgreSQL unless you do that.
And no, don't start deleting files in the data directory.
What you can do is move the *complete* data directory to another location
with more disk space and start it there.Yours,
Laurenz Albe
--
Cybertec | https://www.cybertec-postgresql.com
--
Regards,
Soni Maula Harriz
Fix it already, so it was the SAN and vmware issue.
Thanks Everyone.
On Fri, Jan 15, 2021 at 4:40 PM Soni M <diptatapa@gmail.com> wrote:
More disk space now available, but still the error happens and postgres
won't start.
I think it could be corrupted shared buffers, as stated on the log
messages, and perhaps the corruption goes to disk.On Fri, Jan 15, 2021 at 4:33 PM Laurenz Albe <laurenz.albe@cybertec.at>
wrote:On Fri, 2021-01-15 at 16:13 +0700, Soni M wrote:
We filled out the wal archive disk, and then those messages appear
[...]PANIC,58030,"could not fsync file ""pg_wal/xlogtemp.3510"":
Input/output error"
And now I cannot start postgres
[...]PANIC,58030,"could not fsync file ""pg_wal/xlogtemp.23229"":
Input/output error"
Any idea to solve this ?
Did you add more disk space?
You won't be able to start PostgreSQL unless you do that.
And no, don't start deleting files in the data directory.
What you can do is move the *complete* data directory to another location
with more disk space and start it there.Yours,
Laurenz Albe
--
Cybertec | https://www.cybertec-postgresql.com--
Regards,Soni Maula Harriz
--
Regards,
Soni Maula Harriz