Out of Memory - 8.2.4

Started by Jeff Amielover 18 years ago52 messagesgeneral
Jump to latest
#1Jeff Amiel
becauseimjeff@yahoo.com

"PostgreSQL 8.2.4 on i386-pc-solaris2.10, compiled by
GCC gcc (GCC) 3.4.3 (csl-sol210-3_4-branch+sol_rpath)"

Week-old install....still tuning and tweaking this
thing.

Over last 2 days, have spotted 10 "Out of Memory"
errors in postgres logs (never saw before with same
app/usage patterns on tuned hardware/postgres under
FreeBSD)

Aug 22 18:08:24 db-1 postgres[16452]: [ID 748848
local0.warning] [6-1] 2007-08-22 18:08:24 CDT ERROR:
out of memory.
Aug 22 18:08:24 db-1 postgres[16452]: [ID 748848
local0.warning] [6-2] 2007-08-22 18:08:24 CDT
DETAIL: Failed on request of size 536870910.

What I found interesting is that It's ALWAYS the same
size....536870910

I am running autovacuum and slony.....but I see
nothing in the logs anywhere near the "out of memory"
errors related to either (autovacuum used to under
8.0.X log INFO messages every time it vacuumed which
came in handy...I assume it doesn't so this any more?)

The events are fairly spread out...and cannot (by
looking at app logs and rest of DB logs) correlate to
any specific query or activity.

Any help would be appreciated

Box is a Sun X4600 with 8 dual-core processors and 32
gig of ram.

# su - pgsql
Sun Microsystems Inc. SunOS 5.10 Generic
January 2005
-bash-3.00$ ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
open files (-n) 256
pipe size (512 bytes, -p) 10
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 16357
virtual memory (kbytes, -v) unlimited

shared_buffers = 3GB # min 128kB or
max_connections*16kB
temp_buffers = 1000 # min 800kB
was 8MB
max_prepared_transactions = 450 # can be 0 or
more
work_mem = 100MB # min
64kB
maintenance_work_mem = 512MB # min 1MB
#max_stack_depth = 2MB # min 100kB
max_fsm_pages = 208000 # min
max_fsm_relations*16, 6 bytes each
max_fsm_relations = 10000 # min 100, ~70
bytes each
#max_files_per_process = 1000 # min 25
#shared_preload_libraries = '' # (change
requires restart)
fsync = on # turns forced
synchronization on or off
wal_sync_method = fdatasync # the default
is the first option
full_page_writes = off # recover from
partial page writes
wal_buffers = 2300 # min 32kB
commit_delay = 10 # range
0-100000, in microseconds
#commit_siblings = 5 # range 1-1000
checkpoint_segments = 128 # in logfile
segments, min 1, 16MB each
checkpoint_timeout = 5min # range 30s-1h
checkpoint_warning = 99s # 0 is off

____________________________________________________________________________________
Be a better Globetrotter. Get better travel answers from someone who knows. Yahoo! Answers - Check it out.
http://answers.yahoo.com/dir/?link=list&sid=396545469

#2Erik Jones
erik@myemma.com
In reply to: Jeff Amiel (#1)
Re: Out of Memory - 8.2.4

On Aug 24, 2007, at 10:09 AM, Jeff Amiel wrote:

"PostgreSQL 8.2.4 on i386-pc-solaris2.10, compiled by
GCC gcc (GCC) 3.4.3 (csl-sol210-3_4-branch+sol_rpath)"

Week-old install....still tuning and tweaking this
thing.

Over last 2 days, have spotted 10 "Out of Memory"
errors in postgres logs (never saw before with same
app/usage patterns on tuned hardware/postgres under
FreeBSD)

Aug 22 18:08:24 db-1 postgres[16452]: [ID 748848
local0.warning] [6-1] 2007-08-22 18:08:24 CDT ERROR:
out of memory.
Aug 22 18:08:24 db-1 postgres[16452]: [ID 748848
local0.warning] [6-2] 2007-08-22 18:08:24 CDT
DETAIL: Failed on request of size 536870910.

What I found interesting is that It's ALWAYS the same
size....536870910

I am running autovacuum and slony.....but I see
nothing in the logs anywhere near the "out of memory"
errors related to either (autovacuum used to under
8.0.X log INFO messages every time it vacuumed which
came in handy...I assume it doesn't so this any more?)

The events are fairly spread out...and cannot (by
looking at app logs and rest of DB logs) correlate to
any specific query or activity.

Any help would be appreciated

Box is a Sun X4600 with 8 dual-core processors and 32
gig of ram.

# su - pgsql
Sun Microsystems Inc. SunOS 5.10 Generic
January 2005
-bash-3.00$ ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
open files (-n) 256
pipe size (512 bytes, -p) 10
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 16357
virtual memory (kbytes, -v) unlimited

shared_buffers = 3GB # min 128kB or
max_connections*16kB
temp_buffers = 1000 # min 800kB
was 8MB
max_prepared_transactions = 450 # can be 0 or
more
work_mem = 100MB # min
64kB
maintenance_work_mem = 512MB # min 1MB
#max_stack_depth = 2MB # min 100kB
max_fsm_pages = 208000 # min
max_fsm_relations*16, 6 bytes each
max_fsm_relations = 10000 # min 100, ~70
bytes each
#max_files_per_process = 1000 # min 25
#shared_preload_libraries = '' # (change
requires restart)
fsync = on # turns forced
synchronization on or off
wal_sync_method = fdatasync # the default
is the first option
full_page_writes = off # recover from
partial page writes
wal_buffers = 2300 # min 32kB
commit_delay = 10 # range
0-100000, in microseconds
#commit_siblings = 5 # range 1-1000
checkpoint_segments = 128 # in logfile
segments, min 1, 16MB each
checkpoint_timeout = 5min # range 30s-1h
checkpoint_warning = 99s # 0 is off

A few weeks ago I got the same error on the same server. In fact,
the only difference is our memory where you have 32GB and I have 16GB
and you have 512MB maintenance_work_mem and I have 256MB. I point
out the maintenance work memory setting as that is pretty much
exactly what the request size that your error pointed out as was mine
(yours/2). However, that was the only time I've seen this. Below is
the full context of the error report in my log. I see that there is
an Autovacuum context as well as references to a toast table so,
something to do with autovacuum?

TopMemoryContext: 14466424 total in 1758 blocks; 7160792 free (12578
chunks); 7305632 used
TopTransactionContext: 8192 total in 1 blocks; 7688 free (10 chunks);
504 used
Type information cache: 8192 total in 1 blocks; 1800 free (0 chunks);
6392 used
Operator class cache: 8192 total in 1 blocks; 4872 free (0 chunks);
3320 used
Autovacuum context: 16769024 total in 11 blocks; 6959320 free (11
chunks); 9809704 used
smgr relation table: 24576 total in 2 blocks; 11952 free (4 chunks);
12624 used
TransactionAbortContext: 32768 total in 1 blocks; 32752 free (0
chunks); 16 used
Portal hash: 8192 total in 1 blocks; 3912 free (0 chunks); 4280 used
PortalMemory: 0 total in 0 blocks; 0 free (0 chunks); 0 used
Relcache by OID: 8192 total in 1 blocks; 256 free (0 chunks); 7936 used
CacheMemoryContext: 1183288 total in 20 blocks; 889824 free (4094
chunks); 293464 used
pg_toast_356294_index: 1024 total in 1 blocks; 328 free (0 chunks);
696 used
pg_attrdef_adrelid_adnum_index: 1024 total in 1 blocks; 288 free (0
chunks); 736 used
pg_index_indrelid_index: 1024 total in 1 blocks; 352 free (0 chunks);
672 used
pg_namespace_oid_index: 1024 total in 1 blocks; 352 free (0 chunks);
672 used
pg_authid_oid_index: 1024 total in 1 blocks; 352 free (0 chunks); 672
used
pg_trigger_tgrelid_tgname_index: 1024 total in 1 blocks; 288 free (0
chunks); 736 used
pg_rewrite_rel_rulename_index: 1024 total in 1 blocks; 328 free (0
chunks); 696 used
pg_operator_oid_index: 1024 total in 1 blocks; 352 free (0 chunks);
672 used
pg_index_indexrelid_index: 1024 total in 1 blocks; 352 free (0
chunks); 672 used
pg_class_oid_index: 1024 total in 1 blocks; 352 free (0 chunks); 672
used
pg_attribute_relid_attnum_index: 1024 total in 1 blocks; 288 free (0
chunks); 736 used
pg_amproc_opc_proc_index: 1024 total in 1 blocks; 216 free (0
chunks); 808 used
pg_amop_opc_strat_index: 1024 total in 1 blocks; 216 free (0 chunks);
808 used
Per-database table: 57344 total in 3 blocks; 37592 free (13 chunks);
19752 used
Per-database table: 57344 total in 3 blocks; 37592 free (13 chunks);
19752 used
Per-database table: 57344 total in 3 blocks; 37592 free (13 chunks);
19752 used
Per-database table: 57344 total in 3 blocks; 37592 free (13 chunks);
19752 used
Per-database table: 24576 total in 2 blocks; 13040 free (5 chunks);
11536 used
Per-database table: 100671512 total in 22 blocks; 5803552 free (91
chunks); 94867960 used
Databases hash: 8192 total in 1 blocks; 4936 free (0 chunks); 3256 used
MdSmgr: 8192 total in 1 blocks; 7936 free (226 chunks); 256 used
LOCALLOCK hash: 24576 total in 2 blocks; 10000 free (4 chunks); 14576
used
Timezones: 48616 total in 2 blocks; 5968 free (0 chunks); 42648 used
Postmaster: 24576 total in 2 blocks; 20264 free (155 chunks); 4312 used
ErrorContext: 8192 total in 1 blocks; 8176 free (4 chunks); 16 used
2007-08-08 20:21:05 CDT 3716 :ERROR: out of memory
2007-08-08 20:21:05 CDT 3716 :DETAIL: Failed on request of size
268435452.

Erik Jones

Software Developer | Emma®
erik@myemma.com
800.595.4401 or 615.292.5888
615.292.0777 (fax)

Emma helps organizations everywhere communicate & market in style.
Visit us online at http://www.myemma.com

#3Tom Lane
tgl@sss.pgh.pa.us
In reply to: Jeff Amiel (#1)
Re: Out of Memory - 8.2.4

Jeff Amiel <becauseimjeff@yahoo.com> writes:

Aug 22 18:08:24 db-1 postgres[16452]: [ID 748848
local0.warning] [6-1] 2007-08-22 18:08:24 CDT ERROR:
out of memory.
Aug 22 18:08:24 db-1 postgres[16452]: [ID 748848
local0.warning] [6-2] 2007-08-22 18:08:24 CDT
DETAIL: Failed on request of size 536870910.

What I found interesting is that It's ALWAYS the same
size....536870910

maintenance_work_mem = 512MB # min 1MB

Apparently this maintenance_work_mem setting is higher than your system
can reliably provide. Knock it back a bit.

regards, tom lane

#4Erik Jones
erik@myemma.com
In reply to: Tom Lane (#3)
Re: Out of Memory - 8.2.4

On Aug 24, 2007, at 11:46 AM, Tom Lane wrote:

Jeff Amiel <becauseimjeff@yahoo.com> writes:

Aug 22 18:08:24 db-1 postgres[16452]: [ID 748848
local0.warning] [6-1] 2007-08-22 18:08:24 CDT ERROR:
out of memory.
Aug 22 18:08:24 db-1 postgres[16452]: [ID 748848
local0.warning] [6-2] 2007-08-22 18:08:24 CDT
DETAIL: Failed on request of size 536870910.

What I found interesting is that It's ALWAYS the same
size....536870910

maintenance_work_mem = 512MB # min 1MB

Apparently this maintenance_work_mem setting is higher than your
system
can reliably provide. Knock it back a bit.

regards, tom lane

I'm not so sure. In my case, the request size was only 256MB and we
maintain about 10 - 11 GB free of our 16 GB of memory (2GB
shared_buffers, 42MB work_mem, and 256 MB maintenance_work_mem). The
toast table that was involved in the error was pretty small and I was
able to successfully vacuum it myself virtually instantly. However,
in my case, this (so far) being a one time error I don't have much
more data to contribute. We constantly monitor and graph our
system's I/O, CPU, and memory usage and scan our logs for errors so
if anything else comes up I'll be sure to share.

Erik Jones

Software Developer | Emma®
erik@myemma.com
800.595.4401 or 615.292.5888
615.292.0777 (fax)

Emma helps organizations everywhere communicate & market in style.
Visit us online at http://www.myemma.com

#5Joshua D. Drake
jd@commandprompt.com
In reply to: Erik Jones (#4)
Re: Out of Memory - 8.2.4

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Erik Jones wrote:

On Aug 24, 2007, at 11:46 AM, Tom Lane wrote:

Jeff Amiel <becauseimjeff@yahoo.com> writes:

Aug 22 18:08:24 db-1 postgres[16452]: [ID 748848
local0.warning] [6-1] 2007-08-22 18:08:24 CDT ERROR:
out of memory.
Aug 22 18:08:24 db-1 postgres[16452]: [ID 748848
local0.warning] [6-2] 2007-08-22 18:08:24 CDT
DETAIL: Failed on request of size 536870910.

What I found interesting is that It's ALWAYS the same
size....536870910

maintenance_work_mem = 512MB # min 1MB

Apparently this maintenance_work_mem setting is higher than your system
can reliably provide. Knock it back a bit.

regards, tom lane

I'm not so sure. In my case, the request size was only 256MB and we
maintain about 10 - 11 GB free of our 16 GB of memory (2GB
shared_buffers, 42MB work_mem, and 256 MB maintenance_work_mem). The
toast table that was involved in the error was pretty small and I was
able to successfully vacuum it myself virtually instantly. However, in
my case, this (so far) being a one time error I don't have much more
data to contribute. We constantly monitor and graph our system's I/O,
CPU, and memory usage and scan our logs for errors so if anything else
comes up I'll be sure to share.

We are actually diagnosing a similar problem on this end, where we get a
failure at 1920... I am currently trying to get some DEBUG output.

Sincerely,

Joshua D. Drake

Erik Jones

Software Developer | Emma�
erik@myemma.com
800.595.4401 or 615.292.5888
615.292.0777 (fax)

Emma helps organizations everywhere communicate & market in style.
Visit us online at http://www.myemma.com

---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?

http://archives.postgresql.org/

- --

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997 http://www.commandprompt.com/
UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGzx/gATb/zqfZUUQRAg10AJ9bmIUZ8V99vVCDZfWH05PWckf49QCfa4ta
G1daeagQY2CMUR1QDMtuXTQ=
=HxG3
-----END PGP SIGNATURE-----

#6Jeff Amiel
becauseimjeff@yahoo.com
In reply to: Joshua D. Drake (#5)
Re: Out of Memory - 8.2.4
--- "Joshua D. Drake" <jd@commandprompt.com> wrote:

We are actually diagnosing a similar problem on this
end, where we get a
failure at 1920... I am currently trying to get some
DEBUG output.

We are actually getting it semi-regularly today (3
times already)....I would be happy to provide some
more info if somebody guides me (just set
log_min_messages to one of the debug settings?)

____________________________________________________________________________________
Park yourself in front of a world of choices in alternative vehicles. Visit the Yahoo! Auto Green Center.
http://autos.yahoo.com/green_center/

#7Joshua D. Drake
jd@commandprompt.com
In reply to: Jeff Amiel (#6)
Re: Out of Memory - 8.2.4

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jeff Amiel wrote:

--- "Joshua D. Drake" <jd@commandprompt.com> wrote:

We are actually diagnosing a similar problem on this
end, where we get a
failure at 1920... I am currently trying to get some
DEBUG output.

We are actually getting it semi-regularly today (3
times already)....I would be happy to provide some
more info if somebody guides me (just set
log_min_messages to one of the debug settings?)

Having log_line_prefix with at least %p and %m (or %t) plus a
log_min_messages of DEBUG2 would be great.

Joshua D. Drake

____________________________________________________________________________________
Park yourself in front of a world of choices in alternative vehicles. Visit the Yahoo! Auto Green Center.
http://autos.yahoo.com/green_center/

---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

- --

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997 http://www.commandprompt.com/
UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGzySaATb/zqfZUUQRAqxJAJwL8VcEjDJ1dwQYuvEPh4pORCRUQQCeIwAO
ajfjr7m1bTy9r5DFuNmUP6Y=
=zq4y
-----END PGP SIGNATURE-----

#8Jeff Amiel
becauseimjeff@yahoo.com
In reply to: Joshua D. Drake (#7)
Re: Out of Memory - 8.2.4
--- "Joshua D. Drake" <jd@commandprompt.com> wrote:

Having log_line_prefix with at least %p and %m (or
%t) plus a
log_min_messages of DEBUG2 would be great.

i am getting the additional timestampt/pid on my log
lines now....but no additional debug output...
is log_min_messages one of them that requires a
restart?

____________________________________________________________________________________
Be a better Heartthrob. Get better relationship answers from someone who knows. Yahoo! Answers - Check it out.
http://answers.yahoo.com/dir/?link=list&amp;sid=396545433

#9Tom Lane
tgl@sss.pgh.pa.us
In reply to: Jeff Amiel (#8)
Re: Out of Memory - 8.2.4

Jeff Amiel <becauseimjeff@yahoo.com> writes:

is log_min_messages one of them that requires a
restart?

No, SIGHUP (pg_ctl reload) should be sufficient.

regards, tom lane

#10Jeff Amiel
becauseimjeff@yahoo.com
In reply to: Tom Lane (#9)
Re: Out of Memory - 8.2.4
--- Tom Lane <tgl@sss.pgh.pa.us> wrote:

Jeff Amiel <becauseimjeff@yahoo.com> writes:

is log_min_messages one of them that requires a
restart?

No, SIGHUP (pg_ctl reload) should be sufficient.

Weird....
looks like some items are going to syslog and some to
my defined postgres logfile (from -L option).
Bizarre.
Anyway....I hope this helps someone.....

At 10:46, I find this in my syslog files..

Aug 27 10:46:01 db-1 postgres[27008]: [ID 748848
local0.warning] [85355-1] 2007-08-27 10:46:01.888 CDT
27008ERROR: out of memory
Aug 27 10:46:01 db-1 postgres[27008]: [ID 748848
local0.warning] [85355-2] 2007-08-27 10:46:01.888 CDT
27008DETAIL: Failed on request of size 536870910.

and at the same time in my postgres logfile I see this
(and only this)

TopMemoryContext: 169608 total in 10 blocks; 18832
free (34 chunks); 150776 used
TopTransactionContext: 8192 total in 1 blocks; 7648
free (9 chunks); 544 used
CFuncHash: 8192 total in 1 blocks; 4936 free (0
chunks); 3256 used
Type information cache: 8192 total in 1 blocks; 1800
free (0 chunks); 6392 used
Operator class cache: 8192 total in 1 blocks; 4872
free (0 chunks); 3320 used
Autovacuum context: 8192 total in 1 blocks; 5416 free
(8 chunks); 2776 used
smgr relation table: 8192 total in 1 blocks; 2808 free
(0 chunks); 5384 used
TransactionAbortContext: 32768 total in 1 blocks;
32752 free (0 chunks); 16 used
Portal hash: 8192 total in 1 blocks; 3912 free (0
chunks); 4280 used
PortalMemory: 0 total in 0 blocks; 0 free (0 chunks);
0 used
Relcache by OID: 8192 total in 1 blocks; 3376 free (0
chunks); 4816 used
CacheMemoryContext: 659000 total in 19 blocks; 264904
free (15 chunks); 394096 used
sl_seqlog_idx: 1024 total in 1 blocks; 256 free (0
chunks); 768 used
PartInd_istream_replication_cluster_sl_log_1-node-1:
1024 total in 1 blocks; 392 free (0 chunks); 632 used
sl_log_1_idx1: 1024 total in 1 blocks; 256 free (0
chunks); 768 used
pg_index_indrelid_index: 1024 total in 1 blocks; 352
free (0 chunks); 672 used
pg_attrdef_adrelid_adnum_index: 1024 total in 1
blocks; 288 free (0 chunks); 736 used
pg_autovacuum_vacrelid_index: 1024 total in 1 blocks;
392 free (0 chunks); 632 used
pg_type_typname_nsp_index: 1024 total in 1 blocks; 328
free (0 chunks); 696 used
pg_type_oid_index: 1024 total in 1 blocks; 352 free (0
chunks); 672 used
pg_trigger_tgrelid_tgname_index: 1024 total in 1
blocks; 288 free (0 chunks); 736 used
pg_statistic_relid_att_index: 1024 total in 1 blocks;
288 free (0 chunks); 736 used
pg_auth_members_member_role_index: 1024 total in 1
blocks; 328 free (0 chunks); 696 used
pg_auth_members_role_member_index: 1024 total in 1
blocks; 328 free (0 chunks); 696 used
pg_rewrite_rel_rulename_index: 1024 total in 1 blocks;
328 free (0 chunks); 696 used
pg_proc_proname_args_nsp_index: 1024 total in 1
blocks; 256 free (0 chunks); 768 used
pg_proc_oid_index: 1024 total in 1 blocks; 352 free (0
chunks); 672 used
pg_operator_oprname_l_r_n_index: 1024 total in 1
blocks; 192 free (0 chunks); 832 used
pg_operator_oid_index: 1024 total in 1 blocks; 352
free (0 chunks); 672 used
pg_opclass_oid_index: 1024 total in 1 blocks; 352 free
(0 chunks); 672 used
pg_opclass_am_name_nsp_index: 1024 total in 1 blocks;
216 free (0 chunks); 808 used
pg_namespace_oid_index: 1024 total in 1 blocks; 352
free (0 chunks); 672 used
pg_namespace_nspname_index: 1024 total in 1 blocks;
392 free (0 chunks); 632 used
pg_language_oid_index: 1024 total in 1 blocks; 392
free (0 chunks); 632 used
pg_language_name_index: 1024 total in 1 blocks; 392
free (0 chunks); 632 used
pg_inherits_relid_seqno_index: 1024 total in 1 blocks;
328 free (0 chunks); 696 used
pg_index_indexrelid_index: 1024 total in 1 blocks; 352
free (0 chunks); 672 used
pg_authid_oid_index: 1024 total in 1 blocks; 352 free
(0 chunks); 672 used
pg_authid_rolname_index: 1024 total in 1 blocks; 392
free (0 chunks); 632 used
pg_database_oid_index: 1024 total in 1 blocks; 352
free (0 chunks); 672 used
pg_conversion_oid_index: 1024 total in 1 blocks; 392
free (0 chunks); 632 used
pg_conversion_name_nsp_index: 1024 total in 1 blocks;
328 free (0 chunks); 696 used
pg_conversion_default_index: 1024 total in 1 blocks;
192 free (0 chunks); 832 used
pg_class_relname_nsp_index: 1024 total in 1 blocks;
328 free (0 chunks); 696 used
pg_class_oid_index: 1024 total in 1 blocks; 352 free
(0 chunks); 672 used
pg_cast_source_target_index: 1024 total in 1 blocks;
288 free (0 chunks); 736 used
pg_attribute_relid_attnum_index: 1024 total in 1
blocks; 288 free (0 chunks); 736 used
pg_attribute_relid_attnam_index: 1024 total in 1
blocks; 328 free (0 chunks); 696 used
pg_amproc_opc_proc_index: 1024 total in 1 blocks; 216
free (0 chunks); 808 used
pg_amop_opr_opc_index: 1024 total in 1 blocks; 288
free (0 chunks); 736 used
pg_amop_opc_strat_index: 1024 total in 1 blocks; 216
free (0 chunks); 808 used
pg_aggregate_fnoid_index: 1024 total in 1 blocks; 392
free (0 chunks); 632 used
Per-database table: 122880 total in 4 blocks; 44680
free (19 chunks); 78200 used
Per-database table: 24576 total in 2 blocks; 13040
free (5 chunks); 11536 used
Per-database table: 24576 total in 2 blocks; 13040
free (5 chunks); 11536 used
Per-database table: 24576 total in 2 blocks; 13040
free (5 chunks); 11536 used
Databases hash: 8192 total in 1 blocks; 4936 free (0
chunks); 3256 used
MdSmgr: 8192 total in 1 blocks; 8056 free (1 chunks);
136 used
LOCALLOCK hash: 8192 total in 1 blocks; 3912 free (0
chunks); 4280 used
Timezones: 48616 total in 2 blocks; 5968 free (0
chunks); 42648 used
Postmaster: 24576 total in 2 blocks; 13576 free (123
chunks); 11000 used
ErrorContext: 8192 total in 1 blocks; 8176 free (11
chunks); 16 used

____________________________________________________________________________________
Park yourself in front of a world of choices in alternative vehicles. Visit the Yahoo! Auto Green Center.
http://autos.yahoo.com/green_center/

#11Jeff Amiel
becauseimjeff@yahoo.com
In reply to: Tom Lane (#9)
Re: Out of Memory - 8.2.4
--- Tom Lane <tgl@sss.pgh.pa.us> wrote:

Jeff Amiel <becauseimjeff@yahoo.com> writes:

is log_min_messages one of them that requires a
restart?

No, SIGHUP (pg_ctl reload) should be sufficient.

Weird....
looks like some items are going to syslog and some to
my defined postgres logfile (from -L option).
Bizarre.
Anyway....I hope this helps someone.....

At 10:46, I find this in my syslog files..

Aug 27 10:46:01 db-1 postgres[27008]: [ID 748848
local0.warning] [85355-1] 2007-08-27 10:46:01.888 CDT
27008ERROR: out of memory
Aug 27 10:46:01 db-1 postgres[27008]: [ID 748848
local0.warning] [85355-2] 2007-08-27 10:46:01.888 CDT
27008DETAIL: Failed on request of size 536870910.

and at the same time in my postgres logfile I see this
(and only this)

TopMemoryContext: 169608 total in 10 blocks; 18832
free (34 chunks); 150776 used
TopTransactionContext: 8192 total in 1 blocks; 7648
free (9 chunks); 544 used
CFuncHash: 8192 total in 1 blocks; 4936 free (0
chunks); 3256 used
Type information cache: 8192 total in 1 blocks; 1800
free (0 chunks); 6392 used
Operator class cache: 8192 total in 1 blocks; 4872
free (0 chunks); 3320 used
Autovacuum context: 8192 total in 1 blocks; 5416 free
(8 chunks); 2776 used
smgr relation table: 8192 total in 1 blocks; 2808 free
(0 chunks); 5384 used
TransactionAbortContext: 32768 total in 1 blocks;
32752 free (0 chunks); 16 used
Portal hash: 8192 total in 1 blocks; 3912 free (0
chunks); 4280 used
PortalMemory: 0 total in 0 blocks; 0 free (0 chunks);
0 used
Relcache by OID: 8192 total in 1 blocks; 3376 free (0
chunks); 4816 used
CacheMemoryContext: 659000 total in 19 blocks; 264904
free (15 chunks); 394096 used
sl_seqlog_idx: 1024 total in 1 blocks; 256 free (0
chunks); 768 used
PartInd_istream_replication_cluster_sl_log_1-node-1:
1024 total in 1 blocks; 392 free (0 chunks); 632 used
sl_log_1_idx1: 1024 total in 1 blocks; 256 free (0
chunks); 768 used
pg_index_indrelid_index: 1024 total in 1 blocks; 352
free (0 chunks); 672 used
pg_attrdef_adrelid_adnum_index: 1024 total in 1
blocks; 288 free (0 chunks); 736 used
pg_autovacuum_vacrelid_index: 1024 total in 1 blocks;
392 free (0 chunks); 632 used
pg_type_typname_nsp_index: 1024 total in 1 blocks; 328
free (0 chunks); 696 used
pg_type_oid_index: 1024 total in 1 blocks; 352 free (0
chunks); 672 used
pg_trigger_tgrelid_tgname_index: 1024 total in 1
blocks; 288 free (0 chunks); 736 used
pg_statistic_relid_att_index: 1024 total in 1 blocks;
288 free (0 chunks); 736 used
pg_auth_members_member_role_index: 1024 total in 1
blocks; 328 free (0 chunks); 696 used
pg_auth_members_role_member_index: 1024 total in 1
blocks; 328 free (0 chunks); 696 used
pg_rewrite_rel_rulename_index: 1024 total in 1 blocks;
328 free (0 chunks); 696 used
pg_proc_proname_args_nsp_index: 1024 total in 1
blocks; 256 free (0 chunks); 768 used
pg_proc_oid_index: 1024 total in 1 blocks; 352 free (0
chunks); 672 used
pg_operator_oprname_l_r_n_index: 1024 total in 1
blocks; 192 free (0 chunks); 832 used
pg_operator_oid_index: 1024 total in 1 blocks; 352
free (0 chunks); 672 used
pg_opclass_oid_index: 1024 total in 1 blocks; 352 free
(0 chunks); 672 used
pg_opclass_am_name_nsp_index: 1024 total in 1 blocks;
216 free (0 chunks); 808 used
pg_namespace_oid_index: 1024 total in 1 blocks; 352
free (0 chunks); 672 used
pg_namespace_nspname_index: 1024 total in 1 blocks;
392 free (0 chunks); 632 used
pg_language_oid_index: 1024 total in 1 blocks; 392
free (0 chunks); 632 used
pg_language_name_index: 1024 total in 1 blocks; 392
free (0 chunks); 632 used
pg_inherits_relid_seqno_index: 1024 total in 1 blocks;
328 free (0 chunks); 696 used
pg_index_indexrelid_index: 1024 total in 1 blocks; 352
free (0 chunks); 672 used
pg_authid_oid_index: 1024 total in 1 blocks; 352 free
(0 chunks); 672 used
pg_authid_rolname_index: 1024 total in 1 blocks; 392
free (0 chunks); 632 used
pg_database_oid_index: 1024 total in 1 blocks; 352
free (0 chunks); 672 used
pg_conversion_oid_index: 1024 total in 1 blocks; 392
free (0 chunks); 632 used
pg_conversion_name_nsp_index: 1024 total in 1 blocks;
328 free (0 chunks); 696 used
pg_conversion_default_index: 1024 total in 1 blocks;
192 free (0 chunks); 832 used
pg_class_relname_nsp_index: 1024 total in 1 blocks;
328 free (0 chunks); 696 used
pg_class_oid_index: 1024 total in 1 blocks; 352 free
(0 chunks); 672 used
pg_cast_source_target_index: 1024 total in 1 blocks;
288 free (0 chunks); 736 used
pg_attribute_relid_attnum_index: 1024 total in 1
blocks; 288 free (0 chunks); 736 used
pg_attribute_relid_attnam_index: 1024 total in 1
blocks; 328 free (0 chunks); 696 used
pg_amproc_opc_proc_index: 1024 total in 1 blocks; 216
free (0 chunks); 808 used
pg_amop_opr_opc_index: 1024 total in 1 blocks; 288
free (0 chunks); 736 used
pg_amop_opc_strat_index: 1024 total in 1 blocks; 216
free (0 chunks); 808 used
pg_aggregate_fnoid_index: 1024 total in 1 blocks; 392
free (0 chunks); 632 used
Per-database table: 122880 total in 4 blocks; 44680
free (19 chunks); 78200 used
Per-database table: 24576 total in 2 blocks; 13040
free (5 chunks); 11536 used
Per-database table: 24576 total in 2 blocks; 13040
free (5 chunks); 11536 used
Per-database table: 24576 total in 2 blocks; 13040
free (5 chunks); 11536 used
Databases hash: 8192 total in 1 blocks; 4936 free (0
chunks); 3256 used
MdSmgr: 8192 total in 1 blocks; 8056 free (1 chunks);
136 used
LOCALLOCK hash: 8192 total in 1 blocks; 3912 free (0
chunks); 4280 used
Timezones: 48616 total in 2 blocks; 5968 free (0
chunks); 42648 used
Postmaster: 24576 total in 2 blocks; 13576 free (123
chunks); 11000 used
ErrorContext: 8192 total in 1 blocks; 8176 free (11
chunks); 16 used

____________________________________________________________________________________
Yahoo! oneSearch: Finally, mobile search
that gives answers, not web links.
http://mobile.yahoo.com/mobileweb/onesearch?refer=1ONXIC

#12Jeff Amiel
becauseimjeff@yahoo.com
In reply to: Joshua D. Drake (#5)
Re: Out of Memory - 8.2.4
--- "Joshua D. Drake" <jd@commandprompt.com> wrote:

We are actually diagnosing a similar problem on this
end, where we get a
failure at 1920... I am currently trying to get some
DEBUG output.

Tracking for last few days.
Does not appear to happen when little or no user
activity (like Saturday) I don't know if that rules
out autovacuum or not (if no update threshholds are
reached, no vacuuming will take place anyway)

Aug 23 11:11:51 db-1 postgres[8455]: [ID 748848
local0.warning] [2-1] 2007-08-23 11:11:51 CDT ERROR:
out of memory
Aug 23 11:11:51 db-1 postgres[8455]: [ID 748848
local0.warning] [2-2] 2007-08-23 11:11:51 CDT
DETAIL: Failed on request of size 536870910.
--
Aug 23 11:34:49 db-1 postgres[8910]: [ID 748848
local0.warning] [2-1] 2007-08-23 11:34:49 CDT ERROR:
out of memory
Aug 23 11:34:49 db-1 postgres[8910]: [ID 748848
local0.warning] [2-2] 2007-08-23 11:34:49 CDT
DETAIL: Failed on request of size 536870910.
--
Aug 23 12:06:47 db-1 postgres[9562]: [ID 748848
local0.warning] [2-1] 2007-08-23 12:06:47 CDT ERROR:
out of memory
Aug 23 12:06:47 db-1 postgres[9562]: [ID 748848
local0.warning] [2-2] 2007-08-23 12:06:47 CDT
DETAIL: Failed on request of size 536870910.
--
Aug 23 12:58:47 db-1 postgres[10617]: [ID 748848
local0.warning] [2-1] 2007-08-23 12:58:47 CDT ERROR:
out of memory
Aug 23 12:58:47 db-1 postgres[10617]: [ID 748848
local0.warning] [2-2] 2007-08-23 12:58:47 CDT
DETAIL: Failed on request of size 536870910.
--
Aug 23 15:15:35 db-1 postgres[13400]: [ID 748848
local0.warning] [2-1] 2007-08-23 15:15:35 CDT ERROR:
out of memory
Aug 23 15:15:35 db-1 postgres[13400]: [ID 748848
local0.warning] [2-2] 2007-08-23 15:15:35 CDT
DETAIL: Failed on request of size 536870910.
--
Aug 23 16:50:47 db-1 postgres[15422]: [ID 748848
local0.warning] [2-1] 2007-08-23 16:50:47 CDT ERROR:
out of memory
Aug 23 16:50:47 db-1 postgres[15422]: [ID 748848
local0.warning] [2-2] 2007-08-23 16:50:47 CDT
DETAIL: Failed on request of size 536870910.
--
Aug 24 10:46:46 db-1 postgres[10508]: [ID 748848
local0.warning] [2-1] 2007-08-24 10:46:46 CDT ERROR:
out of memory
Aug 24 10:46:46 db-1 postgres[10508]: [ID 748848
local0.warning] [2-2] 2007-08-24 10:46:46 CDT
DETAIL: Failed on request of size 536870910.
--
Aug 24 11:29:00 db-1 postgres[11539]: [ID 748848
local0.warning] [2-1] 2007-08-24 11:29:00 CDT ERROR:
out of memory
Aug 24 11:29:00 db-1 postgres[11539]: [ID 748848
local0.warning] [2-2] 2007-08-24 11:29:00 CDT
DETAIL: Failed on request of size 536870910.
--
Aug 24 11:50:04 db-1 postgres[12051]: [ID 748848
local0.warning] [2-1] 2007-08-24 11:50:04 CDT ERROR:
out of memory
Aug 24 11:50:04 db-1 postgres[12051]: [ID 748848
local0.warning] [2-2] 2007-08-24 11:50:04 CDT
DETAIL: Failed on request of size 536870910.
--
Aug 24 12:00:33 db-1 postgres[12310]: [ID 748848
local0.warning] [2-1] 2007-08-24 12:00:33 CDT ERROR:
out of memory
Aug 24 12:00:33 db-1 postgres[12310]: [ID 748848
local0.warning] [2-2] 2007-08-24 12:00:33 CDT
DETAIL: Failed on request of size 536870910.
--
Aug 24 16:03:19 db-1 postgres[18263]: [ID 748848
local0.warning] [2493-1] 2007-08-24 16:03:19.296 CDT
18263ERROR: out of memory
Aug 24 16:03:19 db-1 postgres[18263]: [ID 748848
local0.warning] [2493-2] 2007-08-24 16:03:19.296 CDT
18263DETAIL: Failed on request of size 536870910.
--
Aug 24 16:45:46 db-1 postgres[19313]: [ID 748848
local0.warning] [3356-1] 2007-08-24 16:45:46.804 CDT
19313ERROR: out of memory
Aug 24 16:45:46 db-1 postgres[19313]: [ID 748848
local0.warning] [3356-2] 2007-08-24 16:45:46.804 CDT
19313DETAIL: Failed on request of size 536870910.
--
Aug 24 17:29:16 db-1 postgres[20379]: [ID 748848
local0.warning] [4238-1] 2007-08-24 17:29:16.926 CDT
20379ERROR: out of memory
Aug 24 17:29:16 db-1 postgres[20379]: [ID 748848
local0.warning] [4238-2] 2007-08-24 17:29:16.926 CDT
20379DETAIL: Failed on request of size 536870910.
--
Aug 24 17:40:02 db-1 postgres[20651]: [ID 748848
local0.warning] [4452-1] 2007-08-24 17:40:02.682 CDT
20651ERROR: out of memory
Aug 24 17:40:02 db-1 postgres[20651]: [ID 748848
local0.warning] [4452-2] 2007-08-24 17:40:02.682 CDT
20651DETAIL: Failed on request of size 536870910.
--
Aug 26 11:14:56 db-1 postgres[22161]: [ID 748848
local0.warning] [56115-1] 2007-08-26 11:14:56.077 CDT
22161ERROR: out of memory
Aug 26 11:14:56 db-1 postgres[22161]: [ID 748848
local0.warning] [56115-2] 2007-08-26 11:14:56.077 CDT
22161DETAIL: Failed on request of size 536870910.
--
Aug 26 11:27:41 db-1 postgres[22477]: [ID 748848
local0.warning] [56381-1] 2007-08-26 11:27:41.141 CDT
22477ERROR: out of memory
Aug 26 11:27:41 db-1 postgres[22477]: [ID 748848
local0.warning] [56381-2] 2007-08-26 11:27:41.141 CDT
22477DETAIL: Failed on request of size 536870910.
--
Aug 26 11:37:27 db-1 postgres[22729]: [ID 748848
local0.warning] [56603-1] 2007-08-26 11:37:27.476 CDT
22729ERROR: out of memory
Aug 26 11:37:27 db-1 postgres[22729]: [ID 748848
local0.warning] [56603-2] 2007-08-26 11:37:27.476 CDT
22729DETAIL: Failed on request of size 536870910.
--
Aug 26 13:02:47 db-1 postgres[24831]: [ID 748848
local0.warning] [58357-1] 2007-08-26 13:02:47.721 CDT
24831ERROR: out of memory
Aug 26 13:02:47 db-1 postgres[24831]: [ID 748848
local0.warning] [58357-2] 2007-08-26 13:02:47.721 CDT
24831DETAIL: Failed on request of size 536870910.
--
Aug 26 14:15:54 db-1 postgres[26625]: [ID 748848
local0.warning] [59885-1] 2007-08-26 14:15:54.583 CDT
26625ERROR: out of memory
Aug 26 14:15:54 db-1 postgres[26625]: [ID 748848
local0.warning] [59885-2] 2007-08-26 14:15:54.583 CDT
26625DETAIL: Failed on request of size 536870910.
--
Aug 26 14:38:10 db-1 postgres[27167]: [ID 748848
local0.warning] [60334-1] 2007-08-26 14:38:10.817 CDT
27167ERROR: out of memory
Aug 26 14:38:10 db-1 postgres[27167]: [ID 748848
local0.warning] [60334-2] 2007-08-26 14:38:10.817 CDT
27167DETAIL: Failed on request of size 536870910.
--
Aug 26 14:57:42 db-1 postgres[27662]: [ID 748848
local0.warning] [60748-1] 2007-08-26 14:57:42.690 CDT
27662ERROR: out of memory
Aug 26 14:57:42 db-1 postgres[27662]: [ID 748848
local0.warning] [60748-2] 2007-08-26 14:57:42.690 CDT
27662DETAIL: Failed on request of size 536870910.
--
Aug 26 17:25:41 db-1 postgres[1352]: [ID 748848
local0.warning] [63840-1] 2007-08-26 17:25:41.189 CDT
1352ERROR: out of memory
Aug 26 17:25:41 db-1 postgres[1352]: [ID 748848
local0.warning] [63840-2] 2007-08-26 17:25:41.189 CDT
1352DETAIL: Failed on request of size 536870910.
--
Aug 26 18:10:21 db-1 postgres[2467]: [ID 748848
local0.warning] [64756-1] 2007-08-26 18:10:21.684 CDT
2467ERROR: out of memory
Aug 26 18:10:21 db-1 postgres[2467]: [ID 748848
local0.warning] [64756-2] 2007-08-26 18:10:21.684 CDT
2467DETAIL: Failed on request of size 536870910.
--
Aug 26 18:42:15 db-1 postgres[3246]: [ID 748848
local0.warning] [65420-1] 2007-08-26 18:42:15.973 CDT
3246ERROR: out of memory
Aug 26 18:42:15 db-1 postgres[3246]: [ID 748848
local0.warning] [65420-2] 2007-08-26 18:42:15.973 CDT
3246DETAIL: Failed on request of size 536870910.
--
Aug 27 08:05:48 db-1 postgres[23092]: [ID 748848
local0.warning] [82122-1] 2007-08-27 08:05:48.214 CDT
23092ERROR: out of memory
Aug 27 08:05:48 db-1 postgres[23092]: [ID 748848
local0.warning] [82122-2] 2007-08-27 08:05:48.214 CDT
23092DETAIL: Failed on request of size 536870910.
--
Aug 27 08:25:06 db-1 postgres[23569]: [ID 748848
local0.warning] [82520-1] 2007-08-27 08:25:06.407 CDT
23569ERROR: out of memory
Aug 27 08:25:06 db-1 postgres[23569]: [ID 748848
local0.warning] [82520-2] 2007-08-27 08:25:06.407 CDT
23569DETAIL: Failed on request of size 536870910.
--
Aug 27 08:38:05 db-1 postgres[23909]: [ID 748848
local0.warning] [82785-1] 2007-08-27 08:38:05.991 CDT
23909ERROR: out of memory
Aug 27 08:38:05 db-1 postgres[23909]: [ID 748848
local0.warning] [82785-2] 2007-08-27 08:38:05.991 CDT
23909DETAIL: Failed on request of size 536870910.
--
Aug 27 09:20:09 db-1 postgres[24945]: [ID 748848
local0.warning] [83640-1] 2007-08-27 09:20:09.331 CDT
24945ERROR: out of memory
Aug 27 09:20:09 db-1 postgres[24945]: [ID 748848
local0.warning] [83640-2] 2007-08-27 09:20:09.331 CDT
24945DETAIL: Failed on request of size 536870910.
--
Aug 27 09:30:08 db-1 postgres[25155]: [ID 748848
local0.warning] [83857-1] 2007-08-27 09:30:08.536 CDT
25155ERROR: out of memory
Aug 27 09:30:08 db-1 postgres[25155]: [ID 748848
local0.warning] [83857-2] 2007-08-27 09:30:08.536 CDT
25155DETAIL: Failed on request of size 536870910.
--
Aug 27 09:40:01 db-1 postgres[25396]: [ID 748848
local0.warning] [84040-1] 2007-08-27 09:40:01.195 CDT
25396ERROR: out of memory
Aug 27 09:40:01 db-1 postgres[25396]: [ID 748848
local0.warning] [84040-2] 2007-08-27 09:40:01.195 CDT
25396DETAIL: Failed on request of size 536870910.
--
Aug 27 09:53:16 db-1 postgres[25729]: [ID 748848
local0.warning] [84289-1] 2007-08-27 09:53:16.815 CDT
25729ERROR: out of memory
Aug 27 09:53:16 db-1 postgres[25729]: [ID 748848
local0.warning] [84289-2] 2007-08-27 09:53:16.815 CDT
25729DETAIL: Failed on request of size 536870910.
--
Aug 27 10:46:01 db-1 postgres[27008]: [ID 748848
local0.warning] [85355-1] 2007-08-27 10:46:01.888 CDT
27008ERROR: out of memory
Aug 27 10:46:01 db-1 postgres[27008]: [ID 748848
local0.warning] [85355-2] 2007-08-27 10:46:01.888 CDT
27008DETAIL: Failed on request of size 536870910.

____________________________________________________________________________________
Sick sense of humor? Visit Yahoo! TV's
Comedy with an Edge to see what's on, when.
http://tv.yahoo.com/collections/222

#13Tom Lane
tgl@sss.pgh.pa.us
In reply to: Jeff Amiel (#12)
Re: Out of Memory - 8.2.4

Jeff Amiel <becauseimjeff@yahoo.com> writes:

Tracking for last few days.
Does not appear to happen when little or no user
activity (like Saturday) I don't know if that rules
out autovacuum or not (if no update threshholds are
reached, no vacuuming will take place anyway)

Can you correlate these occurrences with anything in the regular system
logs (kernel log in particular)? The Postgres log shows nothing out of
the ordinary --- it's simply that the kernel won't give us 512M for some
reason. I'm guessing it's got something to do with overall system load.

regards, tom lane

#14Martijn van Oosterhout
kleptog@svana.org
In reply to: Jeff Amiel (#12)
Re: Out of Memory - 8.2.4

On Mon, Aug 27, 2007 at 09:12:17AM -0700, Jeff Amiel wrote:

Tracking for last few days.
Does not appear to happen when little or no user
activity (like Saturday) I don't know if that rules
out autovacuum or not (if no update threshholds are
reached, no vacuuming will take place anyway)

I don't think I've seen it so far this thread, but what are your memory
overcommit settings and allocated swap? At least on Linux you would
need a significant chunk of swap to be able to work with that much
memory, even with overcommit off. Check the rules for your system.

Another thing I havn't seen mentioned: you appear to be on a 32-bit
architecture and with 2GB shared_buffers you've lost half your address
space on that alone. Perhaps you simply don't have enough contiguous
address space to alloc 512MB.

Hope this helps,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/

Show quoted text

From each according to his ability. To each according to his ability to litigate.

#15Erik Jones
erik@myemma.com
In reply to: Martijn van Oosterhout (#14)
Re: Out of Memory - 8.2.4

On Aug 27, 2007, at 12:15 PM, Martijn van Oosterhout wrote:

On Mon, Aug 27, 2007 at 09:12:17AM -0700, Jeff Amiel wrote:

Tracking for last few days.
Does not appear to happen when little or no user
activity (like Saturday) I don't know if that rules
out autovacuum or not (if no update threshholds are
reached, no vacuuming will take place anyway)

I don't think I've seen it so far this thread, but what are your
memory
overcommit settings and allocated swap? At least on Linux you would
need a significant chunk of swap to be able to work with that much
memory, even with overcommit off. Check the rules for your system.

Another thing I havn't seen mentioned: you appear to be on a 32-bit
architecture and with 2GB shared_buffers you've lost half your address
space on that alone. Perhaps you simply don't have enough contiguous
address space to alloc 512MB.

The X4600 runs with 64-bit Dual Opterons.

Erik Jones

Software Developer | Emma®
erik@myemma.com
800.595.4401 or 615.292.5888
615.292.0777 (fax)

Emma helps organizations everywhere communicate & market in style.
Visit us online at http://www.myemma.com

#16Joshua D. Drake
jd@commandprompt.com
In reply to: Erik Jones (#15)
Re: Out of Memory - 8.2.4

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Erik Jones wrote:

On Aug 27, 2007, at 12:15 PM, Martijn van Oosterhout wrote:

On Mon, Aug 27, 2007 at 09:12:17AM -0700, Jeff Amiel wrote:

Tracking for last few days.
Does not appear to happen when little or no user
activity (like Saturday) I don't know if that rules
out autovacuum or not (if no update threshholds are
reached, no vacuuming will take place anyway)

I don't think I've seen it so far this thread, but what are your memory
overcommit settings and allocated swap? At least on Linux you would
need a significant chunk of swap to be able to work with that much
memory, even with overcommit off. Check the rules for your system.

Another thing I havn't seen mentioned: you appear to be on a 32-bit
architecture and with 2GB shared_buffers you've lost half your address
space on that alone. Perhaps you simply don't have enough contiguous
address space to alloc 512MB.

The X4600 runs with 64-bit Dual Opterons.

The machine we are tracking this problem on is also 64bit.

Joshua D. Drake

Erik Jones

Software Developer | Emma�
erik@myemma.com
800.595.4401 or 615.292.5888
615.292.0777 (fax)

Emma helps organizations everywhere communicate & market in style.
Visit us online at http://www.myemma.com

- --

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997 http://www.commandprompt.com/
UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFG0xomATb/zqfZUUQRAnfUAJ4jQhMV9vEqL8I7zyT59qo0vhbxuACeLH9d
+PpbVOWYxMkrNC/+V4meHSs=
=DK8s
-----END PGP SIGNATURE-----

#17Jeff Amiel
becauseimjeff@yahoo.com
In reply to: Joshua D. Drake (#16)
Re: Out of Memory - 8.2.4
--- "Joshua D. Drake" <jd@commandprompt.com> wrote:

The machine we are tracking this problem on is also 64bit.

Hmmmm.....looks like 3 different people are tracking a similar issue on 64 bit platforms.....you,
Erik and myself.

____________________________________________________________________________________Ready for the edge of your seat?
Check out tonight's top picks on Yahoo! TV.
http://tv.yahoo.com/

#18Erik Jones
erik@myemma.com
In reply to: Jeff Amiel (#17)
Re: Out of Memory - 8.2.4

Yes, but fortunately for me, unfortunately for the list, it's only
happened to me once so I don't really have anything to go on wrt
repeating the problem. I can only say, "Yep! It's happened!" I am
watching my db closely, though. Well, my monitoring scripts are :)

On Aug 27, 2007, at 1:56 PM, Jeff Amiel wrote:

--- "Joshua D. Drake" <jd@commandprompt.com> wrote:

The machine we are tracking this problem on is also 64bit.

Hmmmm.....looks like 3 different people are tracking a similar
issue on 64 bit platforms.....you,
Erik and myself.

______________________________________________________________________
______________Ready for the edge of your seat?
Check out tonight's top picks on Yahoo! TV.
http://tv.yahoo.com/

---------------------------(end of
broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that
your
message can get through to the mailing list cleanly

Erik Jones

Software Developer | Emma®
erik@myemma.com
800.595.4401 or 615.292.5888
615.292.0777 (fax)

Emma helps organizations everywhere communicate & market in style.
Visit us online at http://www.myemma.com

#19Jeff Amiel
becauseimjeff@yahoo.com
In reply to: Tom Lane (#13)
Re: Out of Memory - 8.2.4

I notice in the log entries for the out of memory events have no username, database name or host
identifier (while regular logged events do) Does that mean anything to anybody?

Aug 28 08:25:50 db-1 postgres[29019]: [ID 748848 local0.warning] [111900-1] 2007-08-28
08:25:50.081 CDT 29019ERROR: out of memory
Aug 28 08:25:50 db-1 postgres[29019]: [ID 748848 local0.warning] [111900-2] 2007-08-28
08:25:50.081 CDT 29019DETAIL: Failed on request of size 536870910.

(regular log entry)
Aug 28 08:26:45 db-1 postgres[28785]: [ID 748848 local0.info] [114999-1] 2007-08-28 08:26:45.413
CDT jboss prod 192.168.20.44 28785LOG: duration: 22606.146 ms execute <unnamed>: select

--- Tom Lane <tgl@sss.pgh.pa.us> wrote:

Can you correlate these occurrences with anything in the regular system
logs (kernel log in particular)? The Postgres log shows nothing out of
the ordinary --- it's simply that the kernel won't give us 512M for some
reason. I'm guessing it's got something to do with overall system load.

regards, tom lane

____________________________________________________________________________________
Moody friends. Drama queens. Your life? Nope! - their life, your story. Play Sims Stories at Yahoo! Games.
http://sims.yahoo.com/

#20Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Jeff Amiel (#19)
Re: Out of Memory - 8.2.4

Jeff Amiel wrote:

I notice in the log entries for the out of memory events have no username, database name or host
identifier (while regular logged events do) Does that mean anything to anybody?

Aug 28 08:25:50 db-1 postgres[29019]: [ID 748848 local0.warning] [111900-1] 2007-08-28
08:25:50.081 CDT 29019ERROR: out of memory
Aug 28 08:25:50 db-1 postgres[29019]: [ID 748848 local0.warning] [111900-2] 2007-08-28
08:25:50.081 CDT 29019DETAIL: Failed on request of size 536870910.

(regular log entry)
Aug 28 08:26:45 db-1 postgres[28785]: [ID 748848 local0.info] [114999-1] 2007-08-28 08:26:45.413
CDT jboss prod 192.168.20.44 28785LOG: duration: 22606.146 ms execute <unnamed>: select

Interesting. What's your log_line_prefix? Does it have "%q" somewhere?

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

#21Tom Lane
tgl@sss.pgh.pa.us
In reply to: Jeff Amiel (#19)
#22Jeff Amiel
becauseimjeff@yahoo.com
In reply to: Alvaro Herrera (#20)
#23Erik Jones
erik@myemma.com
In reply to: Tom Lane (#21)
#24Marko Kreen
markokr@gmail.com
In reply to: Jeff Amiel (#1)
#25Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Marko Kreen (#24)
#26Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Jeff Amiel (#1)
#27Tom Lane
tgl@sss.pgh.pa.us
In reply to: Alvaro Herrera (#25)
#28Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Tom Lane (#27)
#29Tom Lane
tgl@sss.pgh.pa.us
In reply to: Alvaro Herrera (#28)
#30Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Tom Lane (#29)
#31Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Tom Lane (#29)
#32Tom Lane
tgl@sss.pgh.pa.us
In reply to: Alvaro Herrera (#31)
#33Marko Kreen
markokr@gmail.com
In reply to: Tom Lane (#32)
#34Tom Lane
tgl@sss.pgh.pa.us
In reply to: Marko Kreen (#33)
#35Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Marko Kreen (#33)
#36Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Tom Lane (#34)
#37Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Tom Lane (#32)
#38Tom Lane
tgl@sss.pgh.pa.us
In reply to: Alvaro Herrera (#37)
#39Marko Kreen
markokr@gmail.com
In reply to: Tom Lane (#34)
#40Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Tom Lane (#38)
#41Tom Lane
tgl@sss.pgh.pa.us
In reply to: Alvaro Herrera (#40)
#42Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Tom Lane (#41)
#43Tom Lane
tgl@sss.pgh.pa.us
In reply to: Alvaro Herrera (#35)
#44Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Tom Lane (#43)
#45Tom Lane
tgl@sss.pgh.pa.us
In reply to: Alvaro Herrera (#44)
#46Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Tom Lane (#32)
#47Tom Lane
tgl@sss.pgh.pa.us
In reply to: Alvaro Herrera (#46)
#48Bruce Momjian
bruce@momjian.us
In reply to: Tom Lane (#45)
#49Tom Lane
tgl@sss.pgh.pa.us
In reply to: Alvaro Herrera (#40)
#50Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Tom Lane (#49)
#51Tom Lane
tgl@sss.pgh.pa.us
In reply to: Alvaro Herrera (#50)
#52Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Tom Lane (#47)