Re: 8.2.3: Server crashes on Windows using Eclipse/Junit

"Magnus Hagander" <magnus@hagander.net> writes:

- Max_connections is set to 500.

There's your problem. 500 is way above what the windows version can
handle. IIRC the hard max is somewhere around 200 depending on some OS
factors that we don't entirely know.

Maybe we should put an #ifdef WIN32 into guc.c to limit max_connections
to something we know the platform can stand? It'd be more comfortable
if we understood exactly where the limit was, but I think I'd rather
have an "I'm sorry Dave, I can't do that" than random-seeming crashes.

regards, tom lane

On Wed, Oct 17, 2007 at 02:40:14AM -0400, Tom Lane wrote:

"Magnus Hagander" <magnus@hagander.net> writes:

- Max_connections is set to 500.

There's your problem. 500 is way above what the windows version can
handle. IIRC the hard max is somewhere around 200 depending on some OS
factors that we don't entirely know.

Maybe we should put an #ifdef WIN32 into guc.c to limit max_connections
to something we know the platform can stand? It'd be more comfortable
if we understood exactly where the limit was, but I think I'd rather
have an "I'm sorry Dave, I can't do that" than random-seeming crashes.

Yeayh, that's probably a good idea - except we never managed to figure out
where the limit is. It appears to vary pretty wildly between different
machines, for reasons we don't really know why (total RAM has some effect
on it, but that's not the only one, for example)

//Magnus

On Wed, 17 Oct 2007 09:22:11 +0200
Magnus Hagander <magnus@hagander.net> wrote:

Maybe we should put an #ifdef WIN32 into guc.c to limit
max_connections to something we know the platform can stand? It'd
be more comfortable if we understood exactly where the limit was,
but I think I'd rather have an "I'm sorry Dave, I can't do that"
than random-seeming crashes.

Yeayh, that's probably a good idea - except we never managed to
figure out where the limit is. It appears to vary pretty wildly
between different machines, for reasons we don't really know why
(total RAM has some effect on it, but that's not the only one, for
example)

How about we just emit a warning..

WARNING: Connections above 250 on Windows platforms may have
unpredictable results.

Joshua D. Drake

//Magnus

---------------------------(end of
broadcast)--------------------------- TIP 1: if posting/reading
through Usenet, please send an appropriate subscribe-nomail command
to majordomo@postgresql.org so that your message can get through to
the mailing list cleanly

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997 http://www.commandprompt.com/
UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

Maybe we should put an #ifdef WIN32 into guc.c to limit
max_connections to something we know the platform can stand? It'd
be more comfortable if we understood exactly where the limit was,
but I think I'd rather have an "I'm sorry Dave, I can't do that"
than random-seeming crashes.

Yeayh, that's probably a good idea - except we never managed to
figure out where the limit is. It appears to vary pretty wildly
between different machines, for reasons we don't really know why
(total RAM has some effect on it, but that's not the only one, for
example)

How about we just emit a warning..

WARNING: Connections above 250 on Windows platforms may have
unpredictable results.

That's probably a better idea. I'll go look at that unless people feel we should just stick it in docd/faq?

/Magnus

"Magnus Hagander" wrote:

- Max_connections is set to 500. I did that originally because I kept
seeing a message about no connection available and I thought it was
because I was not allocating enough connections. My machine has 2GB of RAM.

There's your problem. 500 is way above what the windows version can handle. IIRC the hard max is somewhere around 200 depending on some OS factors that we don't entirely know. I'd never recommend going above 100-150. With no more than 2Gb ram, not above 100.

My guess is that Windows is running out of handles. Each backend uses about
150 handles. 100 Backends means 15000 handles. Depending how many other
programs are currently running the no. of startable backends will vary
depending on the total handle limit Windows imposes.

Rainer

"Magnus Hagander" <magnus@hagander.net> writes:

How about we just emit a warning..

WARNING: Connections above 250 on Windows platforms may have
unpredictable results.

That's probably a better idea. I'll go look at that unless people feel we should just stick it in docd/faq?

Unless we've got some credible basis for citing a particular number,
I don't think this will help much.

Rainer Bauer <usenet@munnin.com> writes:

My guess is that Windows is running out of handles. Each backend uses about
150 handles. 100 Backends means 15000 handles. Depending how many other
programs are currently running the no. of startable backends will vary
depending on the total handle limit Windows imposes.

I find this theory very interesting; for one thing it explains the
reported variability of results, since the non-Postgres demand for
handles could be anything. Is there any way we could check it?
If it's accurate, what we ought to be whining about is some
combination of max_connections and max_files_per_process, rather
than only considering the former.

regards, tom lane

Tom Lane wrote:

"Magnus Hagander" <magnus@hagander.net> writes:

How about we just emit a warning..

WARNING: Connections above 250 on Windows platforms may have
unpredictable results.

That's probably a better idea. I'll go look at that unless people feel we should just stick it in docd/faq?

Unless we've got some credible basis for citing a particular number,
I don't think this will help much.

ok. Maybe a note in the docs or FAQ at least?

Rainer Bauer <usenet@munnin.com> writes:

My guess is that Windows is running out of handles. Each backend uses about
150 handles. 100 Backends means 15000 handles. Depending how many other
programs are currently running the no. of startable backends will vary
depending on the total handle limit Windows imposes.

I find this theory very interesting; for one thing it explains the
reported variability of results, since the non-Postgres demand for
handles could be anything. Is there any way we could check it?
If it's accurate, what we ought to be whining about is some
combination of max_connections and max_files_per_process, rather
than only considering the former.

It's not that simple. Merlin ran some checks, and drastically reducing
max_files_per_process made no measurable difference.

My best guess is it's due to the non-paged pool. Handles are a part of
what goes in there, but only a part.

//Magnus

On 10/17/07, Magnus Hagander <magnus@hagander.net> wrote:

On Wed, Oct 17, 2007 at 02:40:14AM -0400, Tom Lane wrote:

Maybe we should put an #ifdef WIN32 into guc.c to limit max_connections
to something we know the platform can stand? It'd be more comfortable
if we understood exactly where the limit was, but I think I'd rather
have an "I'm sorry Dave, I can't do that" than random-seeming crashes.

Yeayh, that's probably a good idea - except we never managed to figure out
where the limit is. It appears to vary pretty wildly between different
machines, for reasons we don't really know why (total RAM has some effect
on it, but that's not the only one, for example)

I tried generating idle connections in an effort to reproduce
Laurent's problem, but I ran into a local limit instead: for each
backend, postmaster creates a thread and burns 4MB of its 2GB address
space. It fails around 490.

Laurent's issue must depend on other load characteristics. It's
possible to get a trace of DLL loads, but I haven't found a
noninvasive way of doing that. It seems to require a debugger be
attached.

On 10/20/07, Rainer Bauer <usenet@munnin.com> wrote:

"Magnus Hagander" wrote:

- Max_connections is set to 500. I did that originally because I kept
seeing a message about no connection available and I thought it was
because I was not allocating enough connections. My machine has 2GB of RAM.

There's your problem. 500 is way above what the windows version can handle. IIRC the hard max is somewhere around 200 depending on some OS factors that we don't entirely know. I'd never recommend going above 100-150. With no more than 2Gb ram, not above 100.

My guess is that Windows is running out of handles. Each backend uses about
150 handles. 100 Backends means 15000 handles. Depending how many other
programs are currently running the no. of startable backends will vary
depending on the total handle limit Windows imposes.

Those are kernel object handles; the ceiling does depend on available
kernel memory, but they're cheap, and postgres is in no danger of
running into that limit. Most of the handle limits people talk about
are on USER (window etc) objects, which come from a single shared
pool.

--- "Joshua D. Drake" <jd@commandprompt.com> wrote:

How about we just emit a warning..

WARNING: Connections above 250 on Windows platforms may have
unpredictable results.

Joshua D. Drake

I'd personally vote for a lower warning limit like 175 as I can
consistently crash Postgresql on Windows system right around the 200th
connection.

Regards,

Shelby Cain

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

On 10/20/07, Shelby Cain <alyandon@yahoo.com> wrote:

I'd personally vote for a lower warning limit like 175 as I can
consistently crash Postgresql on Windows system right around the 200th
connection.

What error gets logged for your crashes?

"Trevor Talbot" wrote:

On 10/20/07, Rainer Bauer <usenet@munnin.com> wrote:

"Magnus Hagander" wrote:

- Max_connections is set to 500. I did that originally because I kept
seeing a message about no connection available and I thought it was
because I was not allocating enough connections. My machine has 2GB of RAM.

There's your problem. 500 is way above what the windows version can handle. IIRC the hard max is somewhere around 200 depending on some OS factors that we don't entirely know. I'd never recommend going above 100-150. With no more than 2Gb ram, not above 100.

My guess is that Windows is running out of handles. Each backend uses about
150 handles. 100 Backends means 15000 handles. Depending how many other
programs are currently running the no. of startable backends will vary
depending on the total handle limit Windows imposes.

Those are kernel object handles; the ceiling does depend on available
kernel memory, but they're cheap, and postgres is in no danger of
running into that limit. Most of the handle limits people talk about
are on USER (window etc) objects, which come from a single shared
pool.

You are right. I just did a quick test and depending on the handle type these
limits are quite high. I could create 5 millions events or 4 millions
semaphores or 3,5 millions mutexes before the system returned error 1816
ERROR_NOT_ENOUGH_QUOTA "Not enough quota is available to process this
command.".

Rainer

I wrote:

You are right. I just did a quick test and depending on the handle type these
limits are quite high. I could create 5 millions events or 4 millions
semaphores or 3,5 millions mutexes before the system returned error 1816
ERROR_NOT_ENOUGH_QUOTA "Not enough quota is available to process this
command.".

[Does some further testing] The limit is high, but nonetheless Postgres is
running out of handles. Setting <max_connections> to 10000 and starting
postgres _without_ any connection consumes 40000 handles. This correspodends
to the 4 Postgres processes running after the server was started. Every new
connection consumes another 10000 handles.

I don't know the Postgres code involved, but it seems that every backend
consumes at least <max_connections> handles. Hence increasing this value will
have the opposite effect once a certain threshold is met.

Rainer

--- Trevor Talbot <quension@gmail.com> wrote:

On 10/20/07, Shelby Cain <alyandon@yahoo.com> wrote:

I'd personally vote for a lower warning limit like 175 as I can
consistently crash Postgresql on Windows system right around the

200th

connection.

What error gets logged for your crashes?

It's been a while but IIRC there wasn't anything in the logs other than
an entry noting that a backend had crashed unexpectedly so the
postmaster was restarting all active backends. I can trivially
reproduce it at work on my workstation if you need the exact error
text.

Regards,

Shelby Cain

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

On 10/20/07, Shelby Cain <alyandon@yahoo.com> wrote:

--- Trevor Talbot <quension@gmail.com> wrote:

On 10/20/07, Shelby Cain <alyandon@yahoo.com> wrote:

I'd personally vote for a lower warning limit like 175 as I can
consistently crash Postgresql on Windows system right around the

200th

connection.

What error gets logged for your crashes?

It's been a while but IIRC there wasn't anything in the logs other than
an entry noting that a backend had crashed unexpectedly so the
postmaster was restarting all active backends. I can trivially
reproduce it at work on my workstation if you need the exact error
text.

I think it would be useful; if nothing else, maybe it'll tell us if
you can see the same problem Laruent does, or if it's a different
limit entirely.

Shelby Cain wrote:

--- Trevor Talbot <quension@gmail.com> wrote:

On 10/20/07, Shelby Cain <alyandon@yahoo.com> wrote:

I'd personally vote for a lower warning limit like 175 as I can
consistently crash Postgresql on Windows system right around the

200th

connection.

What error gets logged for your crashes?

It's been a while but IIRC there wasn't anything in the logs other than
an entry noting that a backend had crashed unexpectedly so the
postmaster was restarting all active backends. I can trivially
reproduce it at work on my workstation if you need the exact error
text.

I could reproduce this here:

Server closed the connection unexpectedly
This probaly means the server terminated abnormally before or while processing
the request

2007-10-20 23:33:42 LOG: server process (PID 5240) exited with exit code
-1073741502

Shelby, are you using the /3GB switch by chance? This will half the no. of
available handles on your system.

Rainer

On 10/20/07, Rainer Bauer <usenet@munnin.com> wrote:

I could reproduce this here:

Server closed the connection unexpectedly
This probaly means the server terminated abnormally before or while processing
the request

2007-10-20 23:33:42 LOG: server process (PID 5240) exited with exit code
-1073741502

How?

Hello Trevor,

Sunday, October 21, 2007, 12:15:25 AM, you wrote:

TT> On 10/20/07, Rainer Bauer <usenet@munnin.com> wrote:

I could reproduce this here:

Server closed the connection unexpectedly
This probaly means the server terminated abnormally before or while processing
the request

2007-10-20 23:33:42 LOG: server process (PID 5240) exited with exit code
-1073741502

TT> How?

Seems like the mailiming list is not catching up fast enough (I am
posting through usenet)...

Anyway, the problem are the no. of semaphores created by Postgres:
Every backend creates at least 4*<max_connections> semaphores. Just
increase <max_connections> to an unusual high value (say 10000) and
start creating new connections while monitoring the handle count.

Rainer

--- Rainer Bauer <usenet@munnin.com> wrote:

I could reproduce this here:

Server closed the connection unexpectedly
This probaly means the server terminated abnormally before or while
processing
the request

2007-10-20 23:33:42 LOG: server process (PID 5240) exited with exit
code
-1073741502

Shelby, are you using the /3GB switch by chance? This will half the
no. of
available handles on your system.

Rainer

Probably not although I haven't examined boot.ini. My workstation only
has 1.5 GB of ram so I'm highly doubtful that IBM would have configured
it to boot with the /3GB switch.

Regards,

Shelby Cain

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com