accumulating handles problem on machine running postgresql
I'm running Postgresql 9.0.3 on a Windows 7 box. There is one large database that gets inserted to thousands of times a day every day. The problem is the Handle count on the Windows System process of the server, image name c:\windows\system32\ntoskrnl.exe, accumulates over time and the handles never get released. This causes the handle count to go to about 130,000 at which time the machine starts malfunctioning and a reboot is needed. It takes about 6 days before a reboot is needed. This is causing our regression test machines, which do inserts into the database via a gsoap connection to a web service running on the database machine, to get bungled up every 6 days or so. Has anyone had a problem with their postgresql server where the System process accumulates handles and doesn't release them?
P.S. In addition to postgresql, the Windows 7 server is running IIS 7 and asp.net web services. The web services are reached via gSOAP from various test machines on the local network, some running windows and some running linux.
I really wish I knew what was causing this. I guess it could be postgresql, the devart bindings, gsoap or the web service code itself. I'm sure I'm closing all database connections in the web service code. The database and web services function perfectly except for the handle problem.
Thanks,
Adam
Adam Bruss
Senior Development Engineer
AWR Corporation
11520 N. Port Washington Rd., Suite 201
Mequon, WI 53092 USA
P: 1.262.240.0291 x104
F: 1.262.240.0294
E: abruss@awrcorp.com
W: http://www.awrcorp.com<http://www.awrcorp.com/>
On 02/27/12 6:34 PM, Adam Bruss wrote:
I’m running Postgresql 9.0.3 on a Windows 7 box. There is one large
database that gets inserted to thousands of times a day every day. The
problem is the Handle count on the Windows System process of the
server, image name c:\windows\system32\ntoskrnl.exe, accumulates over
time and the handles never get released. This causes the handle count
to go to about 130,000 at which time the machine starts malfunctioning
and a reboot is needed. It takes about 6 days before a reboot is
needed. This is causing our regression test machines, which do inserts
into the database via a gsoap connection to a web service running on
the database machine, to get bungled up every 6 days or so. Has anyone
had a problem with their postgresql server where the System process
accumulates handles and doesn’t release them?P.S. In addition to postgresql, the Windows 7 server is running IIS 7
and asp.net web services. The web services are reached via gSOAP from
various test machines on the local network, some running windows and
some running linux.I really wish I knew what was causing this. I guess it could be
postgresql, the devart bindings, gsoap or the web service code itself.
I’m sure I’m closing all database connections in the web service code.
The database and web services function perfectly except for the handle
problem.
IIS7 runs the websites half in the kernel, which is the 'system'
process... I'd be suspicious of something happening over there, and not
in the postgres database server.
you want to try a test? run postgres on a different system, and have
your webpile connect to it over the LAN. if you're still loosing
handles, its definitely not PG's fault.
--
john r pierce N 37, W 122
santa cruz ca mid-left coast
On Tue, Feb 28, 2012 at 04:34, Adam Bruss <abruss@awrcorp.com> wrote:
The problem is the
Handle count on the Windows System process of the server, image name
c:\windows\system32\ntoskrnl.exe, accumulates over time and the handles
never get released. This causes the handle count to go to about 130,000 at
which time the machine starts malfunctioning and a reboot is needed.
Do these handles go away when you restart any of these services? If
it's a user space code problem then the handles should get freed
automatically after the process exits.
Regards,
Marti
The handles persist through restarting the postgresql service and restarting the IIS server. The handles are accumulating on the System process. I think the handles are created when the web service is accessed but that would mean the IIS worker processes would have responsibility and they don't seem to. Recycling the worker processes in IIS does nothing. And the worker processes have their own process w3wp.exe which never accumulate handles. It's probably not a postgresql thing or other people would be seeing it.
Adam Bruss
Senior Development Engineer
AWR Corporation
11520 N. Port Washington Rd., Suite 201
Mequon, WI 53092 USA
P: 1.262.240.0291 x104
F: 1.262.240.0294
E: abruss@awrcorp.com
W: http://www.awrcorp.com
-----Original Message-----
From: Marti Raudsepp [mailto:marti@juffo.org]
Sent: Tuesday, February 28, 2012 8:23 AM
To: Adam Bruss
Cc: pgsql-general@postgresql.org
Subject: Re: [GENERAL] accumulating handles problem on machine running postgresql
On Tue, Feb 28, 2012 at 04:34, Adam Bruss <abruss@awrcorp.com> wrote:
The problem is the
Handle count on the Windows System process of the server, image name
c:\windows\system32\ntoskrnl.exe, accumulates over time and the handles
never get released. This causes the handle count to go to about 130,000 at
which time the machine starts malfunctioning and a reboot is needed.
Do these handles go away when you restart any of these services? If
it's a user space code problem then the handles should get freed
automatically after the process exits.
Regards,
Marti
On Tue, Feb 28, 2012 at 8:48 AM, Adam Bruss <abruss@awrcorp.com> wrote:
The handles persist through restarting the postgresql service and restarting the IIS server. The handles are accumulating on the System process. I think the handles are created when the web service is accessed but that would mean the IIS worker processes would have responsibility and they don't seem to. Recycling the worker processes in IIS does nothing. And the worker processes have their own process w3wp.exe which never accumulate handles. It's probably not a postgresql thing or other people would be seeing it.
Use "process explorer" from sysinternals / microsoft (google for it)
to see what these handles are for (pipes, files, events, mutants,
desktops, winstations (ok, probably not those), etc...
I ran process explorer and looked at the handles for the System process. The vast majority of the handles are of type "Key". I can find them in the registry. I took two at random from process explorer and exported the registry branch for them below.
## EXAMPLE 1: ##
Key Name: HKEY_CLASSES_ROOT\CLSID\{9F074EE2-E6E9-4d8a-A047-EB5B5C3C55DA}
Class Name: <NO CLASS>
Last Write Time: 2/28/2012 - 1:26 AM
Value 0
Name: <NO NAME>
Type: REG_SZ
Data: HwTextInsertion Class
Key Name: HKEY_CLASSES_ROOT\CLSID\{9F074EE2-E6E9-4d8a-A047-EB5B5C3C55DA}\InprocServer32
Class Name: <NO CLASS>
Last Write Time: 2/29/2012 - 4:05 AM
Value 0
Name: <NO NAME>
Type: REG_EXPAND_SZ
Data: %CommonProgramFiles%\microsoft shared\ink\tiptsf.dll
Value 1
Name: ThreadingModel
Type: REG_SZ
Data: Apartment
Key Name: HKEY_CLASSES_ROOT\CLSID\{9F074EE2-E6E9-4d8a-A047-EB5B5C3C55DA}\ProgID
Class Name: <NO CLASS>
Last Write Time: 2/29/2012 - 4:05 AM
Key Name: HKEY_CLASSES_ROOT\CLSID\{9F074EE2-E6E9-4d8a-A047-EB5B5C3C55DA}\Server
Class Name: <NO CLASS>
Last Write Time: 2/29/2012 - 4:05 AM
## EXAMPLE 2: ##
Key Name: HKEY_LOCAL_MACHINE\SOFTWARE\Classes\CLSID\{80FF6842-51A9-4959-B3B9-EE4DCBFD7740}
Class Name: <NO CLASS>
Last Write Time: 2/28/2012 - 12:07 AM
Key Name: HKEY_LOCAL_MACHINE\SOFTWARE\Classes\CLSID\{80FF6842-51A9-4959-B3B9-EE4DCBFD7740}\Programmable
Class Name: <NO CLASS>
Last Write Time: 12/13/2010 - 12:27 PM
Key Name: HKEY_LOCAL_MACHINE\SOFTWARE\Classes\CLSID\{80FF6842-51A9-4959-B3B9-EE4DCBFD7740}\InprocServer32
Class Name: <NO CLASS>
Last Write Time: 2/29/2012 - 3:05 AM
Key Name: HKEY_LOCAL_MACHINE\SOFTWARE\Classes\CLSID\{80FF6842-51A9-4959-B3B9-EE4DCBFD7740}\ProgID
Class Name: <NO CLASS>
Last Write Time: 2/29/2012 - 3:05 AM
Key Name: HKEY_LOCAL_MACHINE\SOFTWARE\Classes\CLSID\{80FF6842-51A9-4959-B3B9-EE4DCBFD7740}\Server
Class Name: <NO CLASS>
Last Write Time: 2/29/2012 - 3:05 AM
## END EXAMPLES ##
A common thread I notice when looking through the keys is InprocServer32.
Adam Bruss
Senior Development Engineer
AWR Corporation
11520 N. Port Washington Rd., Suite 201
Mequon, WI 53092 USA
P: 1.262.240.0291 x104
F: 1.262.240.0294
E: abruss@awrcorp.com
W: http://www.awrcorp.com
-----Original Message-----
From: pgsql-general-owner@postgresql.org [mailto:pgsql-general-owner@postgresql.org] On Behalf Of dennis jenkins
Sent: Tuesday, February 28, 2012 4:01 PM
To: pgsql-general@postgresql.org
Subject: Re: [GENERAL] accumulating handles problem on machine running postgresql
On Tue, Feb 28, 2012 at 8:48 AM, Adam Bruss <abruss@awrcorp.com> wrote:
The handles persist through restarting the postgresql service and restarting the IIS server. The handles are accumulating on the System process. I think the handles are created when the web service is accessed but that would mean the IIS worker processes would have responsibility and they don't seem to. Recycling the worker processes in IIS does nothing. And the worker processes have their own process w3wp.exe which never accumulate handles. It's probably not a postgresql thing or other people would be seeing it.
Use "process explorer" from sysinternals / microsoft (google for it)
to see what these handles are for (pipes, files, events, mutants,
desktops, winstations (ok, probably not those), etc...
--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
On Wed, Feb 29, 2012 at 8:54 AM, Adam Bruss <abruss@awrcorp.com> wrote:
I ran process explorer and looked at the handles for the System process. The vast majority of the handles are of type "Key". I can find them in the registry. I took two at random from process explorer and exported the registry branch for them below.
## EXAMPLE 1: ##
Key Name: HKEY_CLASSES_ROOT\CLSID\{9F074EE2-E6E9-4d8a-A047-EB5B5C3C55DA}
Class Name: <NO CLASS>
Last Write Time: 2/28/2012 - 1:26 AM
Value 0
Name: <NO NAME>
Type: REG_SZ
Data: HwTextInsertion ClassKey Name: HKEY_CLASSES_ROOT\CLSID\{9F074EE2-E6E9-4d8a-A047-EB5B5C3C55DA}\InprocServer32
Class Name: <NO CLASS>
Last Write Time: 2/29/2012 - 4:05 AM
Value 0
Name: <NO NAME>
Type: REG_EXPAND_SZ
Data: %CommonProgramFiles%\microsoft shared\ink\tiptsf.dllValue 1
Name: ThreadingModel
Type: REG_SZ
Data: Apartment
Seems like your web server is leaking registry keys used when loading
COM objects. The sample that you posted is for the "Tablet PC Input
Panel Text Services Framework" [1]http://systemexplorer.net/filereviews.php?fid=515344. However, I find it strange that
a) IIS needs this and b) that it would leak it.
Are you able to obtain a large statistical sample of the leaked
registry keys? 2 out of 130,000 seems like a small sample.
Try the command line "handle.exe" tool [2]http://technet.microsoft.com/en-us/sysinternals/bb896655. It can dump to a text
file that you can then analyze with perl, python, grep, etc... or your
own eyeballs. :) See if the handle list is dominated by a specific
set of registry keys.
[1]: http://systemexplorer.net/filereviews.php?fid=515344
[2]: http://technet.microsoft.com/en-us/sysinternals/bb896655
After doing some more work with Process Monitor I found the leaks to be stemming from a driver associated with a License Manager from SafeNet Inc. which runs on the machine.
Adam Bruss
Senior Development Engineer
AWR Corporation
11520 N. Port Washington Rd., Suite 201
Mequon, WI 53092 USA
P: 1.262.240.0291 x104
F: 1.262.240.0294
E: abruss@awrcorp.com
W: http://www.awrcorp.com
-----Original Message-----
From: dennis jenkins [mailto:dennis.jenkins.75@gmail.com]
Sent: Wednesday, February 29, 2012 12:25 PM
To: Adam Bruss
Cc: pgsql-general@postgresql.org
Subject: Re: [GENERAL] accumulating handles problem on machine running postgresql
On Wed, Feb 29, 2012 at 8:54 AM, Adam Bruss <abruss@awrcorp.com> wrote:
I ran process explorer and looked at the handles for the System process. The vast majority of the handles are of type "Key". I can find them in the registry. I took two at random from process explorer and exported the registry branch for them below.
## EXAMPLE 1: ##
Key Name: HKEY_CLASSES_ROOT\CLSID\{9F074EE2-E6E9-4d8a-A047-EB5B5C3C55DA}
Class Name: <NO CLASS>
Last Write Time: 2/28/2012 - 1:26 AM
Value 0
Name: <NO NAME>
Type: REG_SZ
Data: HwTextInsertion ClassKey Name: HKEY_CLASSES_ROOT\CLSID\{9F074EE2-E6E9-4d8a-A047-EB5B5C3C55DA}\InprocServer32
Class Name: <NO CLASS>
Last Write Time: 2/29/2012 - 4:05 AM
Value 0
Name: <NO NAME>
Type: REG_EXPAND_SZ
Data: %CommonProgramFiles%\microsoft shared\ink\tiptsf.dllValue 1
Name: ThreadingModel
Type: REG_SZ
Data: Apartment
Seems like your web server is leaking registry keys used when loading
COM objects. The sample that you posted is for the "Tablet PC Input
Panel Text Services Framework" [1]http://systemexplorer.net/filereviews.php?fid=515344. However, I find it strange that
a) IIS needs this and b) that it would leak it.
Are you able to obtain a large statistical sample of the leaked
registry keys? 2 out of 130,000 seems like a small sample.
Try the command line "handle.exe" tool [2]http://technet.microsoft.com/en-us/sysinternals/bb896655. It can dump to a text
file that you can then analyze with perl, python, grep, etc... or your
own eyeballs. :) See if the handle list is dominated by a specific
set of registry keys.
[1]: http://systemexplorer.net/filereviews.php?fid=515344
[2]: http://technet.microsoft.com/en-us/sysinternals/bb896655