PostgreSQL 10.20 crashes / Antivirus

On 4/6/22 03:59, Thomas, Richard wrote:

I’ve been having problems with several PostgreSQL crashes over the last
month (originally with 10.16, then more recently with 10.20, all on a
Windows Server 2012R2 platform), always with a low level permissions error:

"server process (PID 5664) was terminated by exception 0xC0000022"

(This NT error translates as "Access Denied: A process has requested
access to an object but has not been granted those access rights").

I have also got our IT to disable antivirus on our tablespace folder and
on the folder running our backup PowerShell script (which calls pg_dump).

Is either one of those directories where the dump file is being output to?

If not is the directory that file is being created in have AV checks
disabled?

--
Adrian Klaver
adrian.klaver@aklaver.com

Ah... no - the AV is still allowed to run riot in the folder where the dump file is written to. Do you think that could be the issue?

Richard

-----Original Message-----
From: Adrian Klaver <adrian.klaver@aklaver.com>
Sent: 06 April 2022 15:42
To: Thomas, Richard <Richard.Thomas@atkinsglobal.com>; pgsql-general@lists.postgresql.org
Subject: Re: PostgreSQL 10.20 crashes / Antivirus

On 4/6/22 03:59, Thomas, Richard wrote:

I’ve been having problems with several PostgreSQL crashes over the
last month (originally with 10.16, then more recently with 10.20, all
on a Windows Server 2012R2 platform), always with a low level permissions error:

"server process (PID 5664) was terminated by exception 0xC0000022"

(This NT error translates as "Access Denied: A process has requested
access to an object but has not been granted those access rights").

I have also got our IT to disable antivirus on our tablespace folder
and on the folder running our backup PowerShell script (which calls pg_dump).

Is either one of those directories where the dump file is being output to?

If not is the directory that file is being created in have AV checks disabled?

--
Adrian Klaver
adrian.klaver@aklaver.com
At Atkins - member of the SNC-Lavalin Group, we work flexible hours around the world. Although I have sent this email at a time convenient for me, I don't expect you to respond until it works for you.
NOTICE – This email message and any attachments may contain information or material that is confidential, privileged, and/or subject to copyright or other rights. Any unauthorized viewing, disclosure, retransmission, dissemination, or other use of or reliance on this message or anything contained therein is strictly prohibited and may be unlawful. If you believe you may have received this message in error, kindly inform the sender by return email and delete this message from your system. Thank you.

On 4/6/22 07:55, Thomas, Richard wrote:

Ah... no - the AV is still allowed to run riot in the folder where the dump file is written to. Do you think that could be the issue?

I would say that would be the most likely cause. Mainly because I can
not think of another for this, "Access Denied: A process has requested
access to an object but has not been granted those access rights",
happening after a previous dump file had been successfully saved in the
same location.

Richard

--
Adrian Klaver
adrian.klaver@aklaver.com

On 4/6/22 07:55, Thomas, Richard wrote:

Ah... no - the AV is still allowed to run riot in the folder where the dump file is written to. Do you think that could be the issue?

Should have added to previous post:

Try directing the pg_dump to one of the directories that have AV
scanning disabled(if that is possible) and see if it completes.

Richard

--
Adrian Klaver
adrian.klaver@aklaver.com

A request for further suggestions for how to fix/diagnose this PG crash (having just got back from holiday I discover that PG is still crashing even though I have changed the pgdump target output location to a folder excluded from the McAfee Antivirus scanning - but thanks for the suggestion Adrian). The last few crashes have occurred only during the backup process, when database pg_dump completes (successfully - including the disconnection message in the log) but before the next database pgdump can begin (that pg_dump command gets the error that it cannot connect to the database as the server is no longer responding). This will happen after dumping several databases - sometimes it manages the complete multiple database dump fine. One new thing I've noticed is that when it fails the file access 0xC0000022 error occurs a second time when PG is trying to restart itself after shutting down all the other processes, suggesting whatever caused the crash is still around (a later manual restart works fine):

2022-04-15 03:54:06 BST: [3720] WARNING: 57P02: terminating connection because of crash of another server process
2022-04-15 03:54:06 BST: [3720] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-04-15 03:54:06 BST: [3720] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2022-04-15 03:54:06 BST: [3720] LOCATION: quickdie, postgres.c:2622
2022-04-15 03:54:06 BST: [7900] LOG: 00000: all server processes terminated; reinitializing
2022-04-15 03:54:06 BST: [7900] LOCATION: PostmasterStateMachine, postmaster.c:3924
2022-04-15 03:54:08 BST: [7900] LOG: 00000: startup process (PID 3652) was terminated by exception 0xC0000022
2022-04-15 03:54:08 BST: [7900] HINT: See C include file "ntstatus.h" for a description of the hexadecimal value.
2022-04-15 03:54:08 BST: [7900] LOCATION: LogChildExit, postmaster.c:3650
2022-04-15 03:54:08 BST: [7900] LOG: 00000: aborting startup due to startup process failure
2022-04-15 03:54:08 BST: [7900] LOCATION: reaper, postmaster.c:2907
2022-04-15 03:54:09 BST: [7900] LOG: 00000: database system is shut down
2022-04-15 03:54:09 BST: [7900] LOCATION: UnlinkLockFiles, miscinit.c:765

Some details from my original email (as it was 2 weeks ago):

I’ve been having problems with several PostgreSQL crashes over the last month (originally with 10.16, then more recently with 10.20, all on a Windows Server 2012R2 platform), always with a low level permissions error:
"server process (PID 5664) was terminated by exception 0xC0000022"
(This NT error translates as "Access Denied: A process has requested access to an object but has not been granted those access rights").

I was certainly having issues earlier with McAfee Antivirus causing some of these problems but this has been reduced after following the advice here to prevent scanning of PGDATA folders and postgres.exe (if someone can confirm that reference to “postgresql.exe” is a typo on that page I will correct it, but don’t want to in case it refers to a different version of PostgreSQL):
https://wiki.postgresql.org/wiki/Running_%26_Installing_PostgreSQL_On_Native_Windows#Antivirus_software
I have also got our IT to disable antivirus on our tablespace folder and on the folder running our backup PowerShell script (which calls pg_dump).

Any suggestions for how to fix/diagnose? Some things I have tried already:
- Ran “sfc” (System File Checker) to check for/fix any Windows OS corruptions (none found that were relevant)
- Ran “amcheck” on several of the databases
- Unfortunately checksums are not turned on with the database (not an option with the EDB PostgreSQL Windows Installer) and can’t be turned on after initdb on version 10
- I hoped to run pg_catcheck but cannot find a binary version of this online and I think it would take me quite a lot of work to build it myself (I don’t have Visual Studio, only VSC).

Richard

2022-04-02 02:14:53 BST: [7480] user=scde_suad,db=a9sde,app=pg_dump,host=a9dualling.atkinsgeospatial.com LOG: 00000: duration: 0.816 ms statement: COPY sde.sde_xml_indexes (index_id, index_name, owner, index_type, description) TO stdout;
2022-04-02 02:14:53 BST: [7480] user=scde_suad,db=a9sde,app=pg_dump,host=a9dualling.atkinsgeospatial.com LOCATION: exec_simple_query, postgres.c:1200
2022-04-02 02:14:54 BST: [7480] user=scde_suad,db=a9sde,app=pg_dump,host=a9dualling.atkinsgeospatial.com LOG: 00000: disconnection: session time: 0:36:52.430 user=scde_suad database=a9sde host=a9dualling.atkinsgeospatial.com port=59467
2022-04-02 02:14:54 BST: [7480] user=scde_suad,db=a9sde,app=pg_dump,host=a9dualling.atkinsgeospatial.com LOCATION: log_disconnections, postgres.c:4577
2022-04-02 02:14:55 BST: [5016] LOG: 00000: server process (PID 100) was terminated by exception 0xC0000022

At Atkins - member of the SNC-Lavalin Group, we work flexible hours around the world. Although I have sent this email at a time convenient for me, I don't expect you to respond until it works for you.
NOTICE – This email message and any attachments may contain information or material that is confidential, privileged, and/or subject to copyright or other rights. Any unauthorized viewing, disclosure, retransmission, dissemination, or other use of or reliance on this message or anything contained therein is strictly prohibited and may be unlawful. If you believe you may have received this message in error, kindly inform the sender by return email and delete this message from your system. Thank you.

On 4/19/22 04:59, Thomas, Richard wrote:

A request for further suggestions for how to fix/diagnose this PG crash (having just got back from holiday I discover that PG is still crashing even though I have changed the pgdump target output location to a folder excluded from the McAfee Antivirus scanning - but thanks for the suggestion Adrian). The last few crashes have occurred only during the backup process, when database pg_dump completes (successfully - including the disconnection message in the log) but before the next database pgdump can begin (that pg_dump command gets the error that it cannot connect to the database as the server is no longer responding). This will happen after dumping several databases - sometimes it manages the complete multiple database dump fine. One new thing I've noticed is that when it fails the file access 0xC0000022 error occurs a second time when PG is trying to restart itself after shutting down all the other processes, suggesting whatever caused the crash is still around (a later manual restart works fine):

From you original post:

"... although the new pg_dump log file was created (is empty) it does
not get as far as creating the .backup file ..."

What are the actual commands you are using to do the above?

Richard

--
Adrian Klaver
adrian.klaver@aklaver.com

Adrian Klaver wrote:

What are the actual commands you are using to do the above?

The command used in a PowerShell script (run with Windows task scheduler) to dump each database should evaluate to:

"C:\Program Files\PostgreSQL\10\bin\pg_dump.exe" -b -v -F c -d $dbName -h localhost -p 6488 -U backup_su -f $backupFile 2`>`&1 | Out-File $pgdumpLogFile

where:
- "backup_su" is a superuser role (with password stored in the user's pgpass.conf file)
- $backupFile and $pgdumpLogFile are in folders excluded from McAfee scanning
- pg_dump.exe executable is not excluded from McAfee on-access scanning (although as recommended postgres.exe is)

The actual script segment building the command is:

$pgdumpCmd = "C:\Program Files\PostgreSQL\10\bin\pg_dump.exe"
$pgdumpArgs = @("-b", "-v",
"-F", "c",
"-d", $dbName,
"-h", "localhost",
"-p", "6488",
"-U", " backup_su",
"-f", $backupFile)
cmd /c $pgdumpCmd $pgdumpArgs 2`>`&1 | Out-File $pgdumpLogFile

Note that the .backup file for the first failing pg_dump (after several successful ones) is now being produced but is of length 0, with the associated pg_dump log file simply reading:

pg_dump: [archiver (db)] connection to database "dbexample" failed: could not connect to server: Connection refused (0x0000274D/10061)
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 6488?
could not connect to server: Connection refused (0x0000274D/10061)
Is the server running on host "localhost" (::1) and accepting
TCP/IP connections on port 6488?

Richard

-----Original Message-----
From: Adrian Klaver <adrian.klaver@aklaver.com>
Sent: 19 April 2022 18:11
To: Thomas, Richard <Richard.Thomas@atkinsglobal.com>; pgsql-general@lists.postgresql.org
Subject: Re: PostgreSQL 10.20 crashes / Antivirus

On 4/19/22 04:59, Thomas, Richard wrote:

A request for further suggestions for how to fix/diagnose this PG crash (having just got back from holiday I discover that PG is still crashing even though I have changed the pgdump target output location to a folder excluded from the McAfee Antivirus scanning - but thanks for the suggestion Adrian). The last few crashes have occurred only during the backup process, when database pg_dump completes (successfully - including the disconnection message in the log) but before the next database pgdump can begin (that pg_dump command gets the error that it cannot connect to the database as the server is no longer responding). This will happen after dumping several databases - sometimes it manages the complete multiple database dump fine. One new thing I've noticed is that when it fails the file access 0xC0000022 error occurs a second time when PG is trying to restart itself after shutting down all the other processes, suggesting whatever caused the crash is still around (a later manual restart works fine):

From you original post:

"... although the new pg_dump log file was created (is empty) it does not get as far as creating the .backup file ..."

What are the actual commands you are using to do the above?
At Atkins - member of the SNC-Lavalin Group, we work flexible hours around the world. Although I have sent this email at a time convenient for me, I don't expect you to respond until it works for you.
NOTICE – This email message and any attachments may contain information or material that is confidential, privileged, and/or subject to copyright or other rights. Any unauthorized viewing, disclosure, retransmission, dissemination, or other use of or reliance on this message or anything contained therein is strictly prohibited and may be unlawful. If you believe you may have received this message in error, kindly inform the sender by return email and delete this message from your system. Thank you.

On 4/20/22 01:06, Thomas, Richard wrote:

Adrian Klaver wrote:

What are the actual commands you are using to do the above?

The command used in a PowerShell script (run with Windows task scheduler) to dump each database should evaluate to:

"C:\Program Files\PostgreSQL\10\bin\pg_dump.exe" -b -v -F c -d $dbName -h localhost -p 6488 -U backup_su -f $backupFile 2`>`&1 | Out-File $pgdumpLogFile

FYI, -b is the default unless you are restricting the dump to a schema
or a table .

Do you have large objects(blobs) in the database?

where:
- "backup_su" is a superuser role (with password stored in the user's pgpass.conf file)
- $backupFile and $pgdumpLogFile are in folders excluded from McAfee scanning
- pg_dump.exe executable is not excluded from McAfee on-access scanning (although as recommended postgres.exe is)

Why not?

I would think the whole C:\Program Files\PostgreSQL\10\bin\ would be
excluded.

The actual script segment building the command is:

$pgdumpCmd = "C:\Program Files\PostgreSQL\10\bin\pg_dump.exe"
$pgdumpArgs = @("-b", "-v",
"-F", "c",
"-d", $dbName,
"-h", "localhost",
"-p", "6488",
"-U", " backup_su",
"-f", $backupFile)
cmd /c $pgdumpCmd $pgdumpArgs 2`>`&1 | Out-File $pgdumpLogFile

Note that the .backup file for the first failing pg_dump (after several successful ones) is now being produced but is of length 0, with the associated pg_dump log file simply reading:

What does the Windows event log show?

Same for the A/V software log.

pg_dump: [archiver (db)] connection to database "dbexample" failed: could not connect to server: Connection refused (0x0000274D/10061)
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 6488?
could not connect to server: Connection refused (0x0000274D/10061)
Is the server running on host "localhost" (::1) and accepting
TCP/IP connections on port 6488?

Richard

--
Adrian Klaver
adrian.klaver@aklaver.com

Adrian Klaver wrote:

On 4/20/22 01:06, Thomas, Richard wrote:

The command used in a PowerShell script (run with Windows task scheduler)

to dump each database should evaluate to:

"C:\Program Files\PostgreSQL\10\bin\pg_dump.exe" -b -v -F c -d $dbName
-h localhost -p 6488 -U backup_su -f $backupFile 2`>`&1 | Out-File
$pgdumpLogFile

Do you have large objects(blobs) in the database?

No, but do have PostGIS geometries in almost all tables.

- pg_dump.exe executable is not excluded from McAfee on-access
scanning (although as recommended postgres.exe is)

Why not?

I would think the whole C:\Program Files\PostgreSQL\10\bin\ would be
excluded.

I was following the instructions here:
https://wiki.postgresql.org/wiki/Running_%26_Installing_PostgreSQL_On_Native_Windows#Antivirus_software
I am not particularly familiar with how our McAfee is configured or operates (all under control of our IT, including it seems access to many of the logs). With the executable postgres.exe, they have specified that in McAfee as an executable not to include in "on-access scanning" (wherever the executable lives). This differs from the file read/write scanning where the folders to exclude are specified. I have put in a request earlier today to add exclusions for pg_dump.exe; maybe I'll hear back from IT in a day or so ;-(

What does the Windows event log show?

Only events:
- Info: "The Windows Error Reporting Service service entered the running state" (goes into a stopped state 2 minutes later)
- (Error event messages generated by my PowerShell script)

Not sure if WER is actually writing data somewhere (none of the "*\AppData\Local\Microsoft\Windows\WER\ReportArchive" folders on the server have any data from the last 2 years)

Same for the A/V software log.

I can't find much separate McAfee log information beyond the last hour in "Endpoint Security" app. Previously (before I got the PostgreSQL backup PowerShell script removed from scanning, McAfee used to put messages in the Windows Event Viewer reporting an error that it was a violating one of its rules, but it would allow the operation to continue (no longer get that).

Richard
At Atkins - member of the SNC-Lavalin Group, we work flexible hours around the world. Although I have sent this email at a time convenient for me, I don't expect you to respond until it works for you.
NOTICE – This email message and any attachments may contain information or material that is confidential, privileged, and/or subject to copyright or other rights. Any unauthorized viewing, disclosure, retransmission, dissemination, or other use of or reliance on this message or anything contained therein is strictly prohibited and may be unlawful. If you believe you may have received this message in error, kindly inform the sender by return email and delete this message from your system. Thank you.

On 4/20/22 10:23 AM, Thomas, Richard wrote:

Adrian Klaver wrote:

On 4/20/22 01:06, Thomas, Richard wrote:

- pg_dump.exe executable is not excluded from McAfee on-access
scanning (although as recommended postgres.exe is)

Why not?

I would think the whole C:\Program Files\PostgreSQL\10\bin\ would be
excluded.

I was following the instructions here:

I am not particularly familiar with how our McAfee is configured or operates (all under control of our IT, including it seems access to many of the logs). With the executable postgres.exe, they have specified that in McAfee as an executable not to include in "on-access scanning" (wherever the executable lives). This differs from the file read/write scanning where the folders to exclude are specified. I have put in a request earlier today to add exclusions for pg_dump.exe; maybe I'll hear back from IT in a day or so ;-(

That link also has:

"Specific issues have also been reported with McAfee and Panda
anti-virus software and NetLimiter network monitoring software. While
some people do have PostgreSQL working with these software packages,
there is no specific or even recommend solutions that have not worked in
some cases, so the issues would appear to be installation specific,
sometimes even requiring uninstallation. "

Now the last edit to the page was 2013 so not sure how relevant the
above is anymore. Also, I don't use Windows anymore(other then
occasionally working on peoples machines) so I am not the best person to
comment on current reality. Still the Postgres process seems to be
killed by an external program and my suspicion is that the AV software
is involved. Looks like this is something the Sys admins are going to
have to look into via the tools and files they have access to.

Richard

--
Adrian Klaver
adrian.klaver@aklaver.com

On 4/20/22 10:23 AM, Thomas, Richard wrote:

Adrian Klaver wrote:

On 4/20/22 01:06, Thomas, Richard wrote:

The command used in a PowerShell script (run with Windows task scheduler)

to dump each database should evaluate to:

"C:\Program Files\PostgreSQL\10\bin\pg_dump.exe" -b -v -F c -d $dbName
-h localhost -p 6488 -U backup_su -f $backupFile 2`>`&1 | Out-File
$pgdumpLogFile

Should have asked earlier:

Have you tried doing the pg_dumps as independent actions manually for
each database instead of iterating over a list of databases in a script?

--
Adrian Klaver
adrian.klaver@aklaver.com

Adrian (all),

Thanks for your suggestions. Running the equivalent pg_dump commands at the command line instead of in a script works fine and generates no errors/warnings/McAfee messages in the Windows Event Viewer or on stdout/stderr. (Most days the backup script itself completes running multiple pg_dump without problem). I too suspect McAfee of causing the problem - even if it isn't consciously taking any action, it must be processing all file I/O accesses in order to be able to filter which ones it should do virus scanning on. Unfortunately it is a very painful process for me to get our IT to modify virus scanning or report back on AV messages (even if not errors), so any diagnostics I can perform help speed the process. (And unfortunately, my hands are currently tied on having to use an existing AWS EC2 Windows Server, much as I would prefer to use a Linux system or a cloud-based PaaS.)

Richard

Adrian Klaver wrote:

On 4/20/22 01:06, Thomas, Richard wrote:

- pg_dump.exe executable is not excluded from McAfee on-access
scanning (although as recommended postgres.exe is)

Why not?

I would think the whole C:\Program Files\PostgreSQL\10\bin\ would be
excluded.

I was following the instructions here:

I am not particularly familiar with how our McAfee is configured or
operates (all under control of our IT, including it seems access to
many of the logs). With the executable postgres.exe, they have
specified that in McAfee as an executable not to include in "on-access
scanning" (wherever the executable lives). This differs from the file
read/write scanning where the folders to exclude are specified. I have
put in a request earlier today to add exclusions for pg_dump.exe;
maybe I'll hear back from IT in a day or so ;-(

That link also has:

"Specific issues have also been reported with McAfee and Panda anti-virus
software and NetLimiter network monitoring software. While some people do
have PostgreSQL working with these software packages, there is no specific or
even recommend solutions that have not worked in some cases, so the issues
would appear to be installation specific, sometimes even requiring
uninstallation. "

Now the last edit to the page was 2013 so not sure how relevant the above is
anymore. Also, I don't use Windows anymore(other then occasionally working
on peoples machines) so I am not the best person to comment on current
reality. Still the Postgres process seems to be killed by an external program and
my suspicion is that the AV software is involved. Looks like this is something the
Sys admins are going to have to look into via the tools and files they have
access to.

At Atkins - member of the SNC-Lavalin Group, we work flexible hours around the world. Although I have sent this email at a time convenient for me, I don't expect you to respond until it works for you.
NOTICE – This email message and any attachments may contain information or material that is confidential, privileged, and/or subject to copyright or other rights. Any unauthorized viewing, disclosure, retransmission, dissemination, or other use of or reliance on this message or anything contained therein is strictly prohibited and may be unlawful. If you believe you may have received this message in error, kindly inform the sender by return email and delete this message from your system. Thank you.

On 4/21/22 02:13, Thomas, Richard wrote:

Adrian (all),

Thanks for your suggestions. Running the equivalent pg_dump commands at the command line instead of in a script works fine and generates no errors/warnings/McAfee messages in the Windows Event Viewer or on stdout/stderr. (Most days the backup script itself completes running multiple pg_dump without problem).

For the time being a possible solution would be to insert a time out
between iterations of:

$pgdumpCmd = "C:\Program Files\PostgreSQL\10\bin\pg_dump.exe"
$pgdumpArgs = @("-b", "-v",
"-F", "c",
"-d", $dbName,
"-h", "localhost",
"-p", "6488",
"-U", " backup_su",
"-f", $backupFile)
cmd /c $pgdumpCmd $pgdumpArgs 2`>`&1 | Out-File $pgdumpLogFile

Richard

--
Adrian Klaver
adrian.klaver@aklaver.com