postgresql-v6.5beta2.tar.gz ...

The Hermit Hacker wrote:

I am just creating this now...this is to be considered the first *releaes*
candidate for v6.5 ...

I expect there to be *no more* commits to the source tree until after a
formal release is done...

Ah. I wanted to implement proper handling of case when tuple in
update-chain is updated by "too old" xaction (part of chain should
be deleted) but ok - it's not bug, just unimplemented feature.

(BTW, Mark, I have to update lock.sgml)

Hiroshi, please run your test. You'll get NOTICEs instead of ERRORs.
Unfortunately, I can't get your test finished locally - I always
get core and gdb says too little:

#0 0x2823452f in strcpy ()
(gdb) bt
#0 0x2823452f in strcpy ()
#1 0xefbf8944 in ?? ()
#2 0x812d19d in hash_search (hashp=0x1c518bfc,
keyPtr=0x8901428d <Address 0x8901428d out of bounds>, action=1233853505,
foundPtr=0x4d89410c <Address 0x4d89410c out of bounds>) at dynahash.c:651
#3 0x4d8b0000 in ?? ()
Cannot access memory at address 0x3c7.
(gdb) f 2
#2 0x812cfbd in hash_search (hashp=0x1c518bfc,
keyPtr=0x8901428d <Address 0x8901428d out of bounds>, action=1233853505,
foundPtr=0x4d89410c <Address 0x4d89410c out of bounds>) at dynahash.c:651
651 memmove(destAddr, keyPtr, hctl->keysize);
^^^^^^^
??? memmove shouldn't call strcpy. Something is broken,
but maybe due to FreeBSD 3.0 on my box... I'll run test
on 2.2.6 soon.

Vadim

On Mon, 7 Jun 1999, Vadim Mikheev wrote:

The Hermit Hacker wrote:

I am just creating this now...this is to be considered the first *releaes*
candidate for v6.5 ...

I expect there to be *no more* commits to the source tree until after a
formal release is done...

Ah. I wanted to implement proper handling of case when tuple in
update-chain is updated by "too old" xaction (part of chain should
be deleted) but ok - it's not bug, just unimplemented feature.

Can that wait until *after* 6.5 is released, for inclusion on v6.5.1?

(BTW, Mark, I have to update lock.sgml)

I have no major qualms about docs, if thomas doesn't ...

Marc G. Fournier ICQ#7615664 IRC Nick: Scrappy
Systems Administrator @ hub.org
primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org

The Hermit Hacker wrote:

On Mon, 7 Jun 1999, Vadim Mikheev wrote:

The Hermit Hacker wrote:

I am just creating this now...this is to be considered the first *releaes*
candidate for v6.5 ...

I expect there to be *no more* commits to the source tree until after a
formal release is done...

Ah. I wanted to implement proper handling of case when tuple in
update-chain is updated by "too old" xaction (part of chain should
be deleted) but ok - it's not bug, just unimplemented feature.

Can that wait until *after* 6.5 is released, for inclusion on v6.5.1?

Yes. And if I'll not implement it in 6.5.1 then I'll not have time
for this after -:)

Vadim

-----Original Message-----
From: owner-pgsql-hackers@postgreSQL.org
[mailto:owner-pgsql-hackers@postgreSQL.org]On Behalf Of Vadim Mikheev
Sent: Monday, June 07, 1999 1:12 PM
To: The Hermit Hacker
Cc: pgsql-hackers@postgreSQL.org
Subject: Re: [HACKERS] postgresql-v6.5beta2.tar.gz ...

The Hermit Hacker wrote:

I am just creating this now...this is to be considered the

first *releaes*

candidate for v6.5 ...

I expect there to be *no more* commits to the source tree until after a
formal release is done...

Ah. I wanted to implement proper handling of case when tuple in
update-chain is updated by "too old" xaction (part of chain should
be deleted) but ok - it's not bug, just unimplemented feature.

(BTW, Mark, I have to update lock.sgml)

Hiroshi, please run your test. You'll get NOTICEs instead of ERRORs.

I have run my test and got the following NOTICEs.

NOTICE: Child itemid in update-chain marked as unused - can't
continue vc_rpfheap

Thanks.

Hiroshi Inoue
Inoue@tpf.co.jp

Hiroshi Inoue wrote:

Hiroshi, please run your test. You'll get NOTICEs instead of ERRORs.

I have run my test and got the following NOTICEs.

NOTICE: Child itemid in update-chain marked as unused - can't
continue vc_rpfheap

I'm still getting troubles when running your test due to

1. bug in cache invalidation code: when we invalidate relcache
we forget to free MdfdVec in md.c!

Vacuum invalidates a relation tuple in pg_class and concurrent
xactions invalidate corresponding relcache entry, but don't
free MdfdVec and so allocate new one for the same relation
more and more. Each MdfdVed requires own fd.c:Vfd entry -> below

2. fd.c:pg_nofile()->sysconf(_SC_OPEN_MAX) returns in FreeBSD
near total number of files that can be opened in system
(by _all_ users/procs). With total number of opened files
~ 2000 I can run your test with 10-20 simultaneous
xactions for very short time, -:)

Should we limit fd.c:no_files to ~ 256?
This is port-specific, of course...

Vadim
P.S. Hiroshi test I run:

10-20 psql < file with 10000 xactions:

BEGIN;
UPDATE single row in table;
END;

A script run "vacuum table" each rand(10) + 1 sec.

Ah. I wanted to implement proper handling of case when tuple in
update-chain is updated by "too old" xaction (part of chain should
be deleted) but ok - it's not bug, just unimplemented feature.

Can that wait until *after* 6.5 is released, for inclusion on v6.5.1?

(BTW, Mark, I have to update lock.sgml)

I have no major qualms about docs, if thomas doesn't ...

No. If you want it, do it now, not in 6.5.1. Adding features in
minor releases is worse than adding stuff now.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

On Mon, 7 Jun 1999, Bruce Momjian wrote:

Ah. I wanted to implement proper handling of case when tuple in
update-chain is updated by "too old" xaction (part of chain should
be deleted) but ok - it's not bug, just unimplemented feature.

Can that wait until *after* 6.5 is released, for inclusion on v6.5.1?

(BTW, Mark, I have to update lock.sgml)

I have no major qualms about docs, if thomas doesn't ...

No. If you want it, do it now, not in 6.5.1. Adding features in
minor releases is worse than adding stuff now.

With Vadim himself uncertain as to the effect of doing the patch, then it
should bve left for 6.6 then...

Marc G. Fournier ICQ#7615664 IRC Nick: Scrappy
Systems Administrator @ hub.org
primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org

-----Original Message-----
From: root@sunpine.krs.ru [mailto:root@sunpine.krs.ru]On Behalf Of Vadim
Mikheev
Sent: Monday, June 07, 1999 7:49 PM
To: Hiroshi Inoue
Cc: The Hermit Hacker; pgsql-hackers@postgreSQL.org
Subject: Re: [HACKERS] postgresql-v6.5beta2.tar.gz ...

Hiroshi Inoue wrote:

Hiroshi, please run your test. You'll get NOTICEs instead of ERRORs.

I have run my test and got the following NOTICEs.

NOTICE: Child itemid in update-chain marked as unused - can't
continue vc_rpfheap

I'm still getting troubles when running your test due to

1. bug in cache invalidation code: when we invalidate relcache
we forget to free MdfdVec in md.c!

Vacuum invalidates a relation tuple in pg_class and concurrent
xactions invalidate corresponding relcache entry, but don't
free MdfdVec and so allocate new one for the same relation

I have noticed the same thing But at that point my concern was
truncation of segmented relations. AFAIC concurrent xactions
invalidate relcache entry when StartTransaction() or Command-
CounterIncrement() is called. Unfortunately vacuum couldn't stop
other sessions from executing StartTransaction() nor Command-
CounterIncrement().

more and more. Each MdfdVed requires own fd.c:Vfd entry -> below

2. fd.c:pg_nofile()->sysconf(_SC_OPEN_MAX) returns in FreeBSD
near total number of files that can be opened in system
(by _all_ users/procs). With total number of opened files
~ 2000 I can run your test with 10-20 simultaneous
xactions for very short time, -:)

I have remembered another thing.
Someone has already reported to ML in Japan and I was able to
find the cause easily thanks to him.

vacuum calls open() directly to create pg_vlock file.
Sometimes I was not able to open the file because of the lack of
file descriptors().
FileNameOpenFile()/FileClose() etc should be called instead of
open()/close() etc ?

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp

vacuum calls open() directly to create pg_vlock file.
Sometimes I was not able to open the file because of the lack of
file descriptors().
FileNameOpenFile()/FileClose() etc should be called instead of
open()/close() etc ?

Thanks Hiroshi! I forgot to forward his message regarding pg_vlock
issue. Also note that there are some codes that directly open files in
createdb/destroydb stuffs...
--
Tatsuo Ishii

Marc,

I would like you remind that the tar ball file name and the top source
directory name for the official release are better assigned in the
same convention as previoius release. I mean:

tar ball: postgresql-6.5.tar.gz
directory: postgresql-6.5

would be nice. If you alredy deicide to do that, please forgive me.
--
Tatsuo Ishii

On Tue, 8 Jun 1999, Tatsuo Ishii wrote:

Marc,

I would like you remind that the tar ball file name and the top source
directory name for the official release are better assigned in the
same convention as previoius release. I mean:

tar ball: postgresql-6.5.tar.gz
directory: postgresql-6.5

would be nice. If you alredy deicide to do that, please forgive me.

S'alright, it will be done for the release, but the reminder is/will be
appreciated. I created the beta2.tar.gz file and then clued in that I had
forgotten to rename the directory :(

Marc G. Fournier ICQ#7615664 IRC Nick: Scrappy
Systems Administrator @ hub.org
primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org

"Hiroshi Inoue" <Inoue@tpf.co.jp> writes:

vacuum calls open() directly to create pg_vlock file.
Sometimes I was not able to open the file because of the lack of
file descriptors().
FileNameOpenFile()/FileClose() etc should be called instead of
open()/close() etc ?

That is a good point, but actually I think it should not be a problem.
vacuum.c just does open() and immediately close() on pg_vlock; it
doesn't hold the file descriptor open.

When I fixed psort.c a few weeks ago, I looked through all the other
direct calls of open() and fopen() in the backend. There are still half
a dozen or so, but none of them open more than one file or will hold the
file descriptor for longer than the execution of the function they're
in. So I felt it was OK to leave them alone.

The reason it's OK is that fd.c doesn't use up all the available
file descriptors --- it tries to leave ten or so unused. That's
necessary to ensure that library functions like dlopen() will work,
because they don't know anything about using fd.c's routines.

So, the occasional short-term file opening in vacuum.c and similar
places should not matter. If those do fail for lack of FDs, then the
*real* problem is that fd.c is not estimating correctly how many file
descriptors it can safely use; that's what we need to fix.

But what I want to know right now is whether this behavior has been
seen with code from the last week or two. Maybe the report is just
a side-effect of the FD leaks that used to exist in several places...

regards, tom lane

That is a good point, but actually I think it should not be a problem.
vacuum.c just does open() and immediately close() on pg_vlock; it
doesn't hold the file descriptor open.

When I fixed psort.c a few weeks ago, I looked through all the other
direct calls of open() and fopen() in the backend. There are still half
a dozen or so, but none of them open more than one file or will hold the
file descriptor for longer than the execution of the function they're
in. So I felt it was OK to leave them alone.

The reason it's OK is that fd.c doesn't use up all the available
file descriptors --- it tries to leave ten or so unused. That's
necessary to ensure that library functions like dlopen() will work,
because they don't know anything about using fd.c's routines.

So, the occasional short-term file opening in vacuum.c and similar
places should not matter. If those do fail for lack of FDs, then the
*real* problem is that fd.c is not estimating correctly how many file
descriptors it can safely use; that's what we need to fix.

But what I want to know right now is whether this behavior has been
seen with code from the last week or two. Maybe the report is just
a side-effect of the FD leaks that used to exist in several places...

If I correctly remember, the report was regarding 6.4.2. I will check
if it happens with current, and report back soon.
---
Tatsuo Ishii

-----Original Message-----
From: owner-pgsql-hackers@postgreSQL.org
[mailto:owner-pgsql-hackers@postgreSQL.org]On Behalf Of Tom Lane
Sent: Tuesday, June 08, 1999 11:34 AM
To: pgsql-hackers@postgreSQL.org
Subject: Re: [HACKERS] postgresql-v6.5beta2.tar.gz ...

"Hiroshi Inoue" <Inoue@tpf.co.jp> writes:

vacuum calls open() directly to create pg_vlock file.
Sometimes I was not able to open the file because of the lack of
file descriptors().
FileNameOpenFile()/FileClose() etc should be called instead of
open()/close() etc ?

That is a good point, but actually I think it should not be a problem.
vacuum.c just does open() and immediately close() on pg_vlock; it
doesn't hold the file descriptor open.

When I fixed psort.c a few weeks ago, I looked through all the other
direct calls of open() and fopen() in the backend. There are still half
a dozen or so, but none of them open more than one file or will hold the
file descriptor for longer than the execution of the function they're
in. So I felt it was OK to leave them alone.

The reason it's OK is that fd.c doesn't use up all the available
file descriptors --- it tries to leave ten or so unused. That's
necessary to ensure that library functions like dlopen() will work,
because they don't know anything about using fd.c's routines.

So, the occasional short-term file opening in vacuum.c and similar
places should not matter. If those do fail for lack of FDs, then the
*real* problem is that fd.c is not estimating correctly how many file
descriptors it can safely use; that's what we need to fix.

But what I want to know right now is whether this behavior has been
seen with code from the last week or two. Maybe the report is just
a side-effect of the FD leaks that used to exist in several places...

I don't remember correctly how the phenomenon occured,sorry.
I couldn't reproduce it now.

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp

That is a good point, but actually I think it should not be a problem.
vacuum.c just does open() and immediately close() on pg_vlock; it
doesn't hold the file descriptor open.

When I fixed psort.c a few weeks ago, I looked through all the other
direct calls of open() and fopen() in the backend. There are still half
a dozen or so, but none of them open more than one file or will hold the
file descriptor for longer than the execution of the function they're
in. So I felt it was OK to leave them alone.

The reason it's OK is that fd.c doesn't use up all the available
file descriptors --- it tries to leave ten or so unused. That's
necessary to ensure that library functions like dlopen() will work,
because they don't know anything about using fd.c's routines.

So, the occasional short-term file opening in vacuum.c and similar
places should not matter. If those do fail for lack of FDs, then the
*real* problem is that fd.c is not estimating correctly how many file
descriptors it can safely use; that's what we need to fix.

But what I want to know right now is whether this behavior has been
seen with code from the last week or two. Maybe the report is just
a side-effect of the FD leaks that used to exist in several places...

If I correctly remember, the report was regarding 6.4.2. I will check
if it happens with current, and report back soon.

Done. With 32/64 users and 100 transactions each, I see no problem so
far.
--
Tatsuo Ishii