resource leak in 7.2

"Tom Pfau" <T.Pfau@emCrit.com> writes:

There was a bug in 7.1.2 that was fixed in 7.1.3 that may be back in
7.2.
I downloaded 7.2 yesterday morning and started testing it. I left some
procedures running overnight performing random updates in a table.
Looking at task manager this morning, the three backend processes are
each consuming about 1.5e6 handles.

I see nothing in the CVS logs to make me think that any file-handle-leakage
bug was fixed between 7.1.2 and 7.1.3; and even less to make me think
that 7.1.3 contains any fixes that weren't also applied to the 7.2 line.

You're going to have to be a great deal more specific.

regards, tom lane

There was a cygwin postgresql 7.1.3-2 release to fix a file handle problem
(init_rels() not closing pg_internal.init):
http://archives.postgresql.org/pgsql-cygwin/2002-01/msg00034.php
which refers to:
http://archives.postgresql.org/pgsql-cygwin/2002-01/msg00029.php
Maybe this is it?
- Stuart

"Henshall, Stuart - WCP" <SHenshall@westcountrypublications.co.uk> writes:

There was a cygwin postgresql 7.1.3-2 release to fix a file handle problem
(init_rels() not closing pg_internal.init):
Maybe this is it?

No, because (a) that problem is also fixed in 7.2, and (b) that was
only responsible for leaking one file handle per backend launch,
not 1.5e6 handles...

regards, tom lane

We were initially using 7.1.2. Some overnight tests of our application
revealed that the backends were not releasing handles according to task
manager. We upgraded to 7.1.3 and the problem went away.

My initial testing of 7.2 shows the same symptom although I don't know
that it's the same cause. I just noted that we have seen the problem
before so that if anyone was aware of the prior problem, they might be
able to check that it hadn't been lost in the new version. I don't know
exactly what got changed between 7.1.2 and 7.1.3 to cause the problem to
go away.

In any case, if I run my test procedure which just performs updates to a
random column of a random row of a 1000 row table, I can watch the
handle usage continue to rise on the backend process.

Looking a bit more closely now, a single process doesn't seem to cause a
problem. Running two or more copies simultaneously causes the backends
to continuously consume handles. If I stop one of the processes, the
other stops losing handles.

I'm attaching the test procedure. BTW, the file it reads for data
contains a log of a vacuum session that has been reformatted slightly.
The lines are between 3 and 76 bytes.

-----Original Message-----
From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
Sent: Tuesday, February 05, 2002 11:21 AM
To: Tom Pfau
Cc: pgsql-bugs@postgresql.org; pgsql-cygwin@postgresql.org
Subject: Re: [BUGS] resource leak in 7.2

"Tom Pfau" <T.Pfau@emCrit.com> writes:

There was a bug in 7.1.2 that was fixed in 7.1.3 that may be back in
7.2.
I downloaded 7.2 yesterday morning and started testing it. I left

some

procedures running overnight performing random updates in a table.
Looking at task manager this morning, the three backend processes are
each consuming about 1.5e6 handles.

I see nothing in the CVS logs to make me think that any
file-handle-leakage
bug was fixed between 7.1.2 and 7.1.3; and even less to make me think
that 7.1.3 contains any fixes that weren't also applied to the 7.2 line.

You're going to have to be a great deal more specific.

regards, tom lane

No, that was a different problem related to recreating the
pg_internal.init file after a vacuum. This current leakage occurs
during normal usage.

-----Original Message-----
From: Henshall, Stuart - WCP
[mailto:SHenshall@westcountrypublications.co.uk]
Sent: Tuesday, February 05, 2002 11:49 AM
To: 'Tom Lane'; Tom Pfau
Cc: pgsql-bugs@postgresql.org; pgsql-cygwin@postgresql.org
Subject: RE: [CYGWIN] [BUGS] resource leak in 7.2

There was a cygwin postgresql 7.1.3-2 release to fix a file handle
problem
(init_rels() not closing pg_internal.init):
http://archives.postgresql.org/pgsql-cygwin/2002-01/msg00034.php
which refers to:
http://archives.postgresql.org/pgsql-cygwin/2002-01/msg00029.php
Maybe this is it?
- Stuart

"Tom Pfau" <T.Pfau@emCrit.com> writes:

In any case, if I run my test procedure which just performs updates to a
random column of a random row of a 1000 row table, I can watch the
handle usage continue to rise on the backend process.
Looking a bit more closely now, a single process doesn't seem to cause a
problem. Running two or more copies simultaneously causes the backends
to continuously consume handles. If I stop one of the processes, the
other stops losing handles.

Unsurprisingly, I don't see any such problem under Unix (HPUX to be
specific, but I'd be astonished to see PG on any Unix do that,
considering that each backend manages its open files independently).

I think you must be looking at some misbehavior of the Cygwin layer.
If the problem really did go away in 7.1.3, perhaps some fix was applied
by the Cygwin packager and not contributed back to the main code base.
But it's got to be a Cygwin bug anyway; the most we could do is find
some workaround to avoid triggering it.

regards, tom lane

On Tue, Feb 05, 2002 at 01:07:18PM -0500, Tom Lane wrote:

"Tom Pfau" <T.Pfau@emCrit.com> writes:

In any case, if I run my test procedure which just performs updates to a
random column of a random row of a 1000 row table, I can watch the
handle usage continue to rise on the backend process.
Looking a bit more closely now, a single process doesn't seem to cause a
problem. Running two or more copies simultaneously causes the backends
to continuously consume handles. If I stop one of the processes, the
other stops losing handles.

Unsurprisingly, I don't see any such problem under Unix (HPUX to be
specific, but I'd be astonished to see PG on any Unix do that,
considering that each backend manages its open files independently).

I think you must be looking at some misbehavior of the Cygwin layer.
If the problem really did go away in 7.1.3, perhaps some fix was applied
by the Cygwin packager and not contributed back to the main code base.

I only include patches to the standard Cygwin PostgreSQL distribution
that have been accepted into CVS. [1]To be more precise, there are one or two instances where the version of the patch in CVS was later tweaked for other platforms, but I continued to use the initial version. I'm too lazy to maintain private
patches for the long haul... :,) So, whatever is in Cygwin PostgreSQL
7.1.3 should be in 7.2.

[1]: To be more precise, there are one or two instances where the version of the patch in CVS was later tweaked for other platforms, but I continued to use the initial version.
of the patch in CVS was later tweaked for other platforms, but I continued
to use the initial version.

But it's got to be a Cygwin bug anyway; the most we could do is find
some workaround to avoid triggering it.

Without any investigation, I would tend to agree with the above. Tom
(Pfau), please try to find a minimal test for submission to the Cygwin
mailing list.

Thanks,
Jason

I'm attaching two sql scripts in a zip file. Run script1.sql from one
session. After it's done with the inserts, start script2.sql in another
session. Have task manager running in the background and watch the
backends eat handles.

If these don't run long enough, cut and paste some more updates to the
end.

One running by itself doesn't seem to be a problem. Two or more running
together cause a problem. Also, two interactive sessions updating the
same table don't appear to cause a problem. It only appears to be a
problem when two or more sessions hit the table with an update
simultaneously.

-----Original Message-----
From: Jason Tishler [mailto:jason@tishler.net]
Sent: Tuesday, February 05, 2002 3:03 PM
To: Tom Lane
Cc: Tom Pfau; pgsql-bugs@postgresql.org; pgsql-cygwin@postgresql.org
Subject: Re: [CYGWIN] [BUGS] resource leak in 7.2

On Tue, Feb 05, 2002 at 01:07:18PM -0500, Tom Lane wrote:

"Tom Pfau" <T.Pfau@emCrit.com> writes:

In any case, if I run my test procedure which just performs updates

to a

random column of a random row of a 1000 row table, I can watch the
handle usage continue to rise on the backend process.
Looking a bit more closely now, a single process doesn't seem to

cause a

problem. Running two or more copies simultaneously causes the

backends

to continuously consume handles. If I stop one of the processes,

the

other stops losing handles.

Unsurprisingly, I don't see any such problem under Unix (HPUX to be
specific, but I'd be astonished to see PG on any Unix do that,
considering that each backend manages its open files independently).

I think you must be looking at some misbehavior of the Cygwin layer.
If the problem really did go away in 7.1.3, perhaps some fix was

applied

by the Cygwin packager and not contributed back to the main code base.

I only include patches to the standard Cygwin PostgreSQL distribution
that have been accepted into CVS. [1]To be more precise, there are one or two instances where the version of the patch in CVS was later tweaked for other platforms, but I continued to use the initial version. I'm too lazy to maintain private
patches for the long haul... :,) So, whatever is in Cygwin PostgreSQL
7.1.3 should be in 7.2.

[1]: To be more precise, there are one or two instances where the version of the patch in CVS was later tweaked for other platforms, but I continued to use the initial version.
of the patch in CVS was later tweaked for other platforms, but I
continued
to use the initial version.

But it's got to be a Cygwin bug anyway; the most we could do is find
some workaround to avoid triggering it.

Without any investigation, I would tend to agree with the above. Tom
(Pfau), please try to find a minimal test for submission to the Cygwin
mailing list.

Thanks,
Jason

Tom,

On Tue, Feb 05, 2002 at 04:49:43PM -0500, Tom Pfau wrote:

I'm attaching two sql scripts in a zip file. Run script1.sql from one
session. After it's done with the inserts, start script2.sql in another
session. Have task manager running in the background and watch the
backends eat handles.

[snip]

I meant a C test case -- the Cygwin team will not consider anything else.

Jason

-----Original Message-----
From: Tom Pfau

I'm attaching two sql scripts in a zip file. Run script1.sql from one
session. After it's done with the inserts, start script2.sql in another
session. Have task manager running in the background and watch the
backends eat handles.

I tested the srcipt with both cygipc-1.09 and my patched cygipc-1.09
(Where have my postings on Feb 6 gone ?).

With about 20 concurrent clients running
1) cygipc-1.09
Handles 9000 ---> 23000 or so.
2) my patched
Handles 9000 ---> 15600 or so.

So at least one of the cause seems the semaphore handle leak
in the cygipc library.

regards,
Hiroshi Inoue