semaphore usage "port based"?

'k, an excerpt from a thread on the freebsd lists ... I'm not sure how to
answer:

----
On Sun, Apr 02, 2006 at 05:24:10PM -0300, Marc G. Fournier wrote:

On Sun, 2 Apr 2006, Kris Kennaway wrote:

Right, but why are they doing it *consistently* in FreeBSD 6.x, when

they

never did it in FreeBSD 4.x? I have postmaster processes running on

the

FreeBSD box as far back as November 27th, 2005 ... and have *never*
experienced this problem ... so it isn't PostgreSQL that has changed,
something in FreeBSD has changed :(

You'll need to do some debugging to find out which of the two causes
of EINVAL are true here (or some undocumented cause).

'k, right now, the checks in PostgreSQL are just seeing if the result of
semctl < 0 ... i see from the man page what 'two values' of EINVAL you

are

referring to ... but, if they both return the same ERRNO, how do I
determine which of the two is the cause of the problem? :(

Evaluate context: what other semaphore operations have been performed
previously?

Kris
------

is there any easy way to answer this? I'm getting the Invalid Argument
error for SETVAL and IPC_RMID ...

On Sun, 2 Apr 2006, Marc G. Fournier wrote:

I've got an odd issue that I'm not sure how to fix ... or, if fixing is even
possible ...

I just put into place a FreeBSD 6.x server ... it has 2 jails running on it,
and inside of each, I'm trying to run a PostgreSQL 7.4.12 server (OpenACS
requirement, no choice there) ...

Now, on my older FreeBSD 4.x servers, I have about 17 PostgreSQL servers
(some 7.2, some 7.4, some 8.x) ... and they all run fine, and they all run on
port 5432 ...

Now, something in FreeBSD has changed since 4.x that, if you start up a
second PostgreSQL server on port 5432, the first one starts to generate
"semctl: Invalid argument" errors ...

If I move one to port 5433, both run great ...

Now, since this *did* work fine with 4.x, the FreeBSD developers have
obviously changed something that is causing it not to work ... but, since
'changing port' appears to fix it, I'm wondering if there is something in our
Semaphore creation code that can be tweaked so that the semaphore side of
things *thinks* its running on a different port, but it still responses to
port 5432?

Or, more simply, I think ... is there somewhere in the Semaphore code that is
using the port # as a 'seed'?

I'm trying to attack things from the FreeBSD side too, to find out what has
changed, and how to fix it, but figured I might be able to come up with a
quicker fix from this group ...

Thx ...

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664

"Marc G. Fournier" <scrappy@postgresql.org> writes:

Or, more simply, I think ... is there somewhere in the Semaphore code that
is using the port # as a 'seed'?

We use the port number as a basis for selecting the semaphore key (see
semget(2)). There is code in there to pick a different key value if the
one we first selected appears to be in use; that has to work correctly
if you're going to run multi postmasters on the same port number. It
sounds like FBSD 6 has done something that broke the key-in-use check.

Look at IpcSemaphoreCreate and InternalIpcSemaphoreCreate in
src/backend/port/sysv_sema.c. It may be worth stepping through them
with gdb to see what the semget calls are returning.

regards, tom lane

I wrote:

Look at IpcSemaphoreCreate and InternalIpcSemaphoreCreate in
src/backend/port/sysv_sema.c. It may be worth stepping through them
with gdb to see what the semget calls are returning.

BTW, even before doing that, you should look at "ipcs -s" output to try
to get a clue what's going on. The EINVAL failures may be because the
second postmaster to start deletes the semaphores created by the first
one. You could easily see this happening in before-and-after ipcs data
if so.

strace'ing startup of the second postmaster is another approach that
might be easier than gdb'ing.

regards, tom lane

On Sun, 2 Apr 2006, Tom Lane wrote:

I wrote:

Look at IpcSemaphoreCreate and InternalIpcSemaphoreCreate in
src/backend/port/sysv_sema.c. It may be worth stepping through them
with gdb to see what the semget calls are returning.

BTW, even before doing that, you should look at "ipcs -s" output to try
to get a clue what's going on. The EINVAL failures may be because the
second postmaster to start deletes the semaphores created by the first
one. You could easily see this happening in before-and-after ipcs data
if so.

You are right ...

Before:

Semaphores:
T ID KEY MODE OWNER GROUP CREATOR CGROUP NSEMS OTIME CTIME
s 524288 5432001 --rw------- 70 70 70 70 17 14:44:19 14:44:19
s 524289 5432002 --rw------- 70 70 70 70 17 14:44:19 14:44:19
s 524290 5432003 --rw------- 70 70 70 70 17 14:44:19 14:44:19
s 524291 5432004 --rw------- 70 70 70 70 17 14:44:19 14:44:19
s 524292 5432005 --rw------- 70 70 70 70 17 14:44:19 14:44:19
s 524293 5432006 --rw------- 70 70 70 70 17 20:23:56 14:44:19
s 524294 5432007 --rw------- 70 70 70 70 17 20:23:58 14:44:19

After:

Semaphores:
T ID KEY MODE OWNER GROUP CREATOR CGROUP NSEMS OTIME CTIME
s 589824 5432001 --rw------- 70 70 70 70 17 21:38:03 21:38:03
s 589825 5432002 --rw------- 70 70 70 70 17 21:38:03 21:38:03
s 589826 5432003 --rw------- 70 70 70 70 17 21:38:03 21:38:03
s 589827 5432004 --rw------- 70 70 70 70 17 21:38:03 21:38:03
s 589828 5432005 --rw------- 70 70 70 70 17 21:38:03 21:38:03
s 589829 5432006 --rw------- 70 70 70 70 17 21:38:03 21:38:03
s 589830 5432007 --rw------- 70 70 70 70 17 21:38:03 21:38:03

So, our semget() is trying to acquire 5432001, FreeBSD's semget is
reporting back that its not in use, so the second instance if basically
'punting' the original one off of it ...

Kris, from the PostgreSQL sources, here is where we try and set the semId
to use ... is there something we are doing wrong with our code as far as
FreeBSD 6.x is concerned, such that semget is not returning a negative
value when the key is already in use? Or is there a problem with semget()
in a jail such that it is allowing for the KEY to be reused, instead of
returning a negative value?

========
static IpcSemaphoreId
InternalIpcSemaphoreCreate(IpcSemaphoreKey semKey, int numSems)
{
int semId;

semId = semget(semKey, numSems, IPC_CREAT | IPC_EXCL | IPCProtection);

if (semId < 0)
{
/*
* Fail quietly if error indicates a collision with existing set.
* One would expect EEXIST, given that we said IPC_EXCL, but
* perhaps we could get a permission violation instead? Also,
* EIDRM might occur if an old set is slated for destruction but
* not gone yet.
*/
if (errno == EEXIST || errno == EACCES
#ifdef EIDRM
|| errno == EIDRM
#endif
)
return -1;

/*
* Else complain and abort
*/
ereport(FATAL,
(errmsg("could not create semaphores: %m"),
errdetail("Failed system call was semget(%d, %d, 0%o).",
(int) semKey, numSems,
IPC_CREAT | IPC_EXCL | IPCProtection),
(errno == ENOSPC) ?
errhint("This error does *not* mean that you have run out of disk space.\n"
"It occurs when either the system limit for the maximum number of "
"semaphore sets (SEMMNI), or the system wide maximum number of "
"semaphores (SEMMNS), would be exceeded. You need to raise the "
"respective kernel parameter. Alternatively, reduce PostgreSQL's "
"consumption of semaphores by reducing its max_connections parameter "
"(currently %d).\n"
"The PostgreSQL documentation contains more information about "
"configuring your system for PostgreSQL.",
MaxBackends) : 0));
}

return semId;
}
========

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664

"Marc G. Fournier" <scrappy@postgresql.org> writes:

On Sun, 2 Apr 2006, Tom Lane wrote:

BTW, even before doing that, you should look at "ipcs -s" output to try
to get a clue what's going on. The EINVAL failures may be because the
second postmaster to start deletes the semaphores created by the first
one. You could easily see this happening in before-and-after ipcs data
if so.

You are right ...

OK, could we see strace (or whatever BSD calls it) output for the second
postmaster? I'd like to see exactly what results it's getting for the
kernel calls it makes during IpcSemaphoreCreate.

regards, tom lane

On Sun, 2 Apr 2006, Tom Lane wrote:

"Marc G. Fournier" <scrappy@postgresql.org> writes:

On Sun, 2 Apr 2006, Tom Lane wrote:

BTW, even before doing that, you should look at "ipcs -s" output to try
to get a clue what's going on. The EINVAL failures may be because the
second postmaster to start deletes the semaphores created by the first
one. You could easily see this happening in before-and-after ipcs data
if so.

You are right ...

OK, could we see strace (or whatever BSD calls it) output for the second
postmaster? I'd like to see exactly what results it's getting for the
kernel calls it makes during IpcSemaphoreCreate.

'k, dont' know what strace is ... we have ktrace and truss ... truss is
what I usually use, and is:

DESCRIPTION
The truss utility traces the system calls called by the specified process
or program. Output is to the specified output file, or standard error by
default. It does this by stopping and restarting the process being moni-
tored via procfs(5).

And shows output like:

# truss ls
ioctl(1,TIOCGETA,0x7fbff514) = 0 (0x0)
ioctl(1,TIOCGWINSZ,0x7fbff588) = 0 (0x0)
getuid() = 0 (0x0)
readlink("/etc/malloc.conf",0x7fbff470,63) ERR#2 'No such file or directory'
mmap(0x0,4096,0x3,0x1002,-1,0x0) = 671666176 (0x2808d000)
break(0x809b000) = 0 (0x0)
break(0x809c000) = 0 (0x0)
break(0x809d000) = 0 (0x0)
break(0x809e000) = 0 (0x0)
stat(".",0x7fbff470) = 0 (0x0)
open(".",0x0,00) = 3 (0x3)
fchdir(0x3) = 0 (0x0)
open(".",0x0,00) = 4 (0x4)
stat(".",0x7fbff430) = 0 (0x0)
open(".",0x4,00) = 5 (0x5)
fstat(5,0x7fbff430) = 0 (0x0)
fcntl(0x5,0x2,0x1) = 0 (0x0)
__sysctl(0x7fbff2e8,0x2,0x8098760,0x7fbff2e4,0x0,0x0) = 0 (0x0)
fstatfs(0x5,0x7fbff330) = 0 (0x0)
break(0x809f000) = 0 (0x0)
getdirentries(0x5,0x809e000,0x1000,0x809a0b4) = 512 (0x200)
getdirentries(0x5,0x809e000,0x1000,0x809a0b4) = 0 (0x0)
lseek(5,0x0,0) = 0 (0x0)
close(5) = 0 (0x0)
fchdir(0x4) = 0 (0x0)
close(4) = 0 (0x0)
fstat(1,0x7fbff270) = 0 (0x0)
break(0x80a0000) = 0 (0x0)
ioctl(1,TIOCGETA,0x7fbff2a4) = 0 (0x0)
.cshrc .cvspass .history .login .psql_history .ssh
write(1,0x809f000,53) = 53 (0x35)
.cshrc~ .emacs.d .klogin .profile .rnd ktrace.out
write(1,0x809f000,53) = 53 (0x35)
exit(0x0) process exit, rval = 0

ktrace is:

DESCRIPTION
The ktrace utility enables kernel trace logging for the specified pro-
cesses. Kernel trace data is logged to the file ktrace.out. The kernel
operations that are traced include system calls, namei translations, sig-
nal processing, and I/O.

And shows output like:

86523 ls RET __sysctl 0
86523 ls CALL fstatfs(0x5,0x7fbff330)
86523 ls RET fstatfs 0
86523 ls CALL break(0x809f000)
86523 ls RET break 0
86523 ls CALL getdirentries(0x5,0x809e000,0x1000,0x809a0b4)
86523 ls RET getdirentries 512/0x200
86523 ls CALL getdirentries(0x5,0x809e000,0x1000,0x809a0b4)
86523 ls RET getdirentries 0
86523 ls CALL lseek(0x5,0,0,0,0)

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664

"Marc G. Fournier" <scrappy@postgresql.org> writes:

On Sun, 2 Apr 2006, Tom Lane wrote:

OK, could we see strace (or whatever BSD calls it) output for the second
postmaster? I'd like to see exactly what results it's getting for the
kernel calls it makes during IpcSemaphoreCreate.

'k, dont' know what strace is ... we have ktrace and truss ... truss is
what I usually use, and is:

truss seems to have an output format closer to what I'm used to, but
either will do.

regards, tom lane

Sent offlist ...

On Sun, 2 Apr 2006, Tom Lane wrote:

"Marc G. Fournier" <scrappy@postgresql.org> writes:

On Sun, 2 Apr 2006, Tom Lane wrote:

OK, could we see strace (or whatever BSD calls it) output for the second
postmaster? I'd like to see exactly what results it's getting for the
kernel calls it makes during IpcSemaphoreCreate.

'k, dont' know what strace is ... we have ktrace and truss ... truss is
what I usually use, and is:

truss seems to have an output format closer to what I'm used to, but
either will do.

regards, tom lane
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664

"Marc G. Fournier" <scrappy@postgresql.org> writes:

'k, try this one ... looks better, actually has semget() calls in it :)

OK, here's our problem:

84250: semget(0x52e2c1,0x11,0x780) ERR#17 'File exists'

This is InternalIpcSemaphoreCreate failing because of key collision.
As it should.

84250: semget(0x52e2c1,0x11,0x0) = 1114112 (0x110000)

This is IpcSemaphoreCreate trying to see what's up. OK.

84250: __semctl(0x110000,0x10,0x5,0x0) = 537 (0x219)

IpcSemaphoreGetValue indicates it has the right "magic number" to be
a Postgres semaphore set. Still expected.

84250: __semctl(0x110000,0x10,0x4,0x0) = 83699 (0x146f3)

IpcSemaphoreGetLastPID says the sema set is last touched by pid 83699.
Looks reasonable (but do you want to double check that that matched the
first postmaster's PID?)

84250: getpid() = 84250 (0x1491a)

our pid ... as expected ...

84250: kill(0x146f3,0x0) ERR#3 'No such process'

Oops. Here is the problem: kill() is lying by claiming there is no such
process as 83699. It looks to me like there in fact is such a process,
but it's in a different jail.

I venture that FBSD 6 has decided to return ESRCH (no such process)
where FBSD 4 returned some other error that acknowledged that the
process did exist (EPERM would be a reasonable guess).

If this is the story, then FBSD have broken their system and must revert
their change. They do not have kernel behavior that totally hides the
existence of the other process, and therefore having some calls that
pretend it's not there is simply inconsistent.

regards, tom lane

On Sun, Apr 02, 2006 at 11:08:11PM -0400, Tom Lane wrote:

I venture that FBSD 6 has decided to return ESRCH (no such process)
where FBSD 4 returned some other error that acknowledged that the
process did exist (EPERM would be a reasonable guess).

If this is the story, then FBSD have broken their system and must revert
their change. They do not have kernel behavior that totally hides the
existence of the other process, and therefore having some calls that
pretend it's not there is simply inconsistent.

I'm guessing it's a deliberate change to prevent the information
leakage between jails.

Kris

Kris Kennaway <kris@obsecurity.org> writes:

On Sun, Apr 02, 2006 at 11:08:11PM -0400, Tom Lane wrote:

If this is the story, then FBSD have broken their system and must revert
their change. They do not have kernel behavior that totally hides the
existence of the other process, and therefore having some calls that
pretend it's not there is simply inconsistent.

I'm guessing it's a deliberate change to prevent the information
leakage between jails.

I have no objection to doing that, so long as you are actually doing it
correctly. This example shows that each jail must have its own SysV
semaphore key space, else information leaks anyway. The current
situation breaks Postgres, and therefore I suggest reverting the errno
change until you are prepared to fix the SysV IPC stuff to be per-jail.

regards, tom lane

On Sun, Apr 02, 2006 at 11:17:49PM -0400, Tom Lane wrote:

Kris Kennaway <kris@obsecurity.org> writes:

On Sun, Apr 02, 2006 at 11:08:11PM -0400, Tom Lane wrote:

If this is the story, then FBSD have broken their system and must revert
their change. They do not have kernel behavior that totally hides the
existence of the other process, and therefore having some calls that
pretend it's not there is simply inconsistent.

I'm guessing it's a deliberate change to prevent the information
leakage between jails.

I have no objection to doing that, so long as you are actually doing it
correctly. This example shows that each jail must have its own SysV
semaphore key space, else information leaks anyway.

By default SysV shared memory is disallowed in jails.

Kris

Kris Kennaway <kris@obsecurity.org> writes:

On Sun, Apr 02, 2006 at 11:17:49PM -0400, Tom Lane wrote:

I have no objection to doing that, so long as you are actually doing it
correctly. This example shows that each jail must have its own SysV
semaphore key space, else information leaks anyway.

By default SysV shared memory is disallowed in jails.

Hm, the present problem seems to be about semaphores not shared memory
... although I'd not be surprised to find that there's a similar issue
around shared memory. Anyway, if FBSD's position is that they are
uninterested in supporting SysV IPC in connection with jails, then I
think the Postgres project position has to be that we are uninterested
in supporting Postgres inside FBSD jails. Sorry Marc :-(

regards, tom lane

On Sun, 2 Apr 2006, Kris Kennaway wrote:

On Sun, Apr 02, 2006 at 11:17:49PM -0400, Tom Lane wrote:

Kris Kennaway <kris@obsecurity.org> writes:

On Sun, Apr 02, 2006 at 11:08:11PM -0400, Tom Lane wrote:

If this is the story, then FBSD have broken their system and must revert
their change. They do not have kernel behavior that totally hides the
existence of the other process, and therefore having some calls that
pretend it's not there is simply inconsistent.

I'm guessing it's a deliberate change to prevent the information
leakage between jails.

I have no objection to doing that, so long as you are actually doing it
correctly. This example shows that each jail must have its own SysV
semaphore key space, else information leaks anyway.

By default SysV shared memory is disallowed in jails.

'k, but how do I fix kill so that it has the proper behaviour if SysV is
enabled? Maybe a mount option for procfs that allows for pre-5.x
behaviour? I'm not the first one to point out that this is a problem, just
the first to follow it through to the cause ;( And I believe there is
more then just PostgreSQL that is affected by shared memory (ie. apache2
needs SysV IPC enabled, so anyone doing that in a jail has it enabled
also) ...

Basically, I don't care if 'default' is ultra-secure ... but some means to
bring it down a notch would be nice ... :(

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664

On Sun, Apr 02, 2006 at 11:26:52PM -0400, Tom Lane wrote:

Kris Kennaway <kris@obsecurity.org> writes:

On Sun, Apr 02, 2006 at 11:17:49PM -0400, Tom Lane wrote:

I have no objection to doing that, so long as you are actually doing it
correctly. This example shows that each jail must have its own SysV
semaphore key space, else information leaks anyway.

By default SysV shared memory is disallowed in jails.

Hm, the present problem seems to be about semaphores not shared memory

Sorry, I meant IPC.

... although I'd not be surprised to find that there's a similar issue
around shared memory. Anyway, if FBSD's position is that they are
uninterested in supporting SysV IPC in connection with jails, then I
think the Postgres project position has to be that we are uninterested
in supporting Postgres inside FBSD jails.

No-one is taking a position of being "uninterested", so please don't
be hasty to reciprocate.

Kris

On Sun, 2 Apr 2006, Kris Kennaway wrote:

No-one is taking a position of being "uninterested", so please don't
be hasty to reciprocate.

I just posted it off the -hackers list, but there is an ancient patch in
the FreeBSD queue for implementing Private IPCs for Jails ... not sure why
it was never committed, or what is involved in bring it up to speed with
the current 6.x and / or -current kernels though ... but, as I mentioned
in another thread, I know that *at least* Apache2 makes use of shared
memory for some of its stuff ...

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664

On Mon, Apr 03, 2006 at 12:30:58AM -0300, Marc G. Fournier wrote:

On Sun, 2 Apr 2006, Kris Kennaway wrote:

On Sun, Apr 02, 2006 at 11:17:49PM -0400, Tom Lane wrote:

Kris Kennaway <kris@obsecurity.org> writes:

On Sun, Apr 02, 2006 at 11:08:11PM -0400, Tom Lane wrote:

If this is the story, then FBSD have broken their system and must revert
their change. They do not have kernel behavior that totally hides the
existence of the other process, and therefore having some calls that
pretend it's not there is simply inconsistent.

I'm guessing it's a deliberate change to prevent the information
leakage between jails.

I have no objection to doing that, so long as you are actually doing it
correctly. This example shows that each jail must have its own SysV
semaphore key space, else information leaks anyway.

By default SysV shared memory is disallowed in jails.

'k, but how do I fix kill so that it has the proper behaviour if SysV is
enabled?

Check the source, perhaps there's already a way. If not, talk to
whoever made the change.

Maybe a mount option for procfs that allows for pre-5.x
behaviour?

procfs has nothing to do with this though.

I'm not the first one to point out that this is a problem, just
the first to follow it through to the cause ;( And I believe there is
more then just PostgreSQL that is affected by shared memory (ie. apache2
needs SysV IPC enabled, so anyone doing that in a jail has it enabled
also) ...

Also note that SysV IPC is not the problem here, it's the change in
the behaviour of kill() that is causing postgresql to become confused.
That's what you should investigate.

Kris

On Sun, Apr 02, 2006 at 11:41:01PM -0400, Kris Kennaway wrote:

On Mon, Apr 03, 2006 at 12:30:58AM -0300, Marc G. Fournier wrote:

'k, but how do I fix kill so that it has the proper behaviour if SysV is
enabled?

Check the source, perhaps there's already a way. If not, talk to
whoever made the change.

Maybe a mount option for procfs that allows for pre-5.x
behaviour?

procfs has nothing to do with this though.

I'm not the first one to point out that this is a problem, just
the first to follow it through to the cause ;( And I believe there is
more then just PostgreSQL that is affected by shared memory (ie. apache2
needs SysV IPC enabled, so anyone doing that in a jail has it enabled
also) ...

Also note that SysV IPC is not the problem here, it's the change in
the behaviour of kill() that is causing postgresql to become confused.
That's what you should investigate.

The ESRCH error is being returned from prison_check(), that would be a
good starting place.

Andrew

Thanks all ... have moved this to just the freebsd-stable list, since I
don't imagine most here are interested in FreeBSD :(

On Mon, 3 Apr 2006, Andrew Thompson wrote:

On Sun, Apr 02, 2006 at 11:41:01PM -0400, Kris Kennaway wrote:

On Mon, Apr 03, 2006 at 12:30:58AM -0300, Marc G. Fournier wrote:

'k, but how do I fix kill so that it has the proper behaviour if SysV is
enabled?

Check the source, perhaps there's already a way. If not, talk to
whoever made the change.

Maybe a mount option for procfs that allows for pre-5.x
behaviour?

procfs has nothing to do with this though.

I'm not the first one to point out that this is a problem, just
the first to follow it through to the cause ;( And I believe there is
more then just PostgreSQL that is affected by shared memory (ie. apache2
needs SysV IPC enabled, so anyone doing that in a jail has it enabled
also) ...

Also note that SysV IPC is not the problem here, it's the change in
the behaviour of kill() that is causing postgresql to become confused.
That's what you should investigate.

The ESRCH error is being returned from prison_check(), that would be a
good starting place.

Andrew
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664