Memory issues

Started by Shiran Kleidermanover 13 years ago18 messagesgeneral

shirank1@gmail.com

over 13 years ago

Hi,
I'm using and Amazon ec2 instance with the following spec and the
application that I'm running uses a postgres DB 9.1.
The app has 3 main cron jobs.

*Ubuntu 12, High-Memory Extra Large Instance
17.1 GB of memory
6.5 EC2 Compute Units (2 virtual cores with 3.25 EC2 Compute Units each)
420 GB of instance storage
64-bit platform*

I've changed the main default values under file *postgresql.conf* to:
shared_buffers = 4GB
work_mem = 16MB
wal_buffers = 16MB
checkpoint_segments = 32
effective_cache_size = 8GB

When I run the app, after an hour or two, free -m looks like below ans the
crons can't run due to memory loss or similar (i'm new to postgres and db
admin).
Thanks!

free -m, errors:

total used free shared buffers cached
Mem: 17079 13742 3337 0 64 11882
-/+ buffers/cache: 1796 15283
Swap: 511 0 511

total used *free* shared buffers cached
Mem: 17079 16833 *245 *0 42 14583
-/+ buffers/cache: 2207 14871
Swap: 511 0 511

**free above stays low even when nothing is running.

**errors:
*DBI connect('database=---;host=localhost','postgres',...) failed: could
not fork new process for connection: Cannot allocate memory*
could not fork new process for connection: Cannot allocate memory

and
execute failed: ERROR: out of memory
DETAIL: Failed on request of size 968. [for Statement "
SELECT DISTINCT....

Thank you!

Scott Marlowe

scott.marlowe@gmail.com

over 13 years ago

In reply to: Shiran Kleiderman (#1)

Re: Memory issues

On Mon, Sep 24, 2012 at 12:45 AM, Shiran Kleiderman <shirank1@gmail.com> wrote:

Hi,
I'm using and Amazon ec2 instance with the following spec and the
application that I'm running uses a postgres DB 9.1.
The app has 3 main cron jobs.

Ubuntu 12, High-Memory Extra Large Instance
17.1 GB of memory
6.5 EC2 Compute Units (2 virtual cores with 3.25 EC2 Compute Units each)
420 GB of instance storage
64-bit platform

I've changed the main default values under file postgresql.conf to:
shared_buffers = 4GB
work_mem = 16MB
wal_buffers = 16MB
checkpoint_segments = 32
effective_cache_size = 8GB

When I run the app, after an hour or two, free -m looks like below ans the
crons can't run due to memory loss or similar (i'm new to postgres and db
admin).
Thanks!

free -m, errors:

total used free shared buffers cached
Mem: 17079 13742 3337 0 64 11882
-/+ buffers/cache: 1796 15283
Swap: 511 0 511

You have 11.8G cached, that's basically free memory on demand.

total used free shared buffers cached
Mem: 17079 16833 245 0 42 14583
-/+ buffers/cache: 2207 14871
Swap: 511 0 511

Here you have 14.5G cached, again that's free memory so to speak.
I.e. when something needs it it gets allocated.

**free above stays low even when nothing is running.

**errors:
DBI connect('database=---;host=localhost','postgres',...) failed: could not
fork new process for connection: Cannot allocate memory
could not fork new process for connection: Cannot allocate memory

This error is happening in your client process. Maybe it's 32 bit or
something and running out of local memory in its process space? Maybe
memory is so fragmented that no large blocks can get allocated or
something? Either way, your machine has plenty of memory according to
free. BTW, it's pretty common for folks new to unix to mis-read free
and not realize that cached memory + free memory is what's really
available.

Shiran Kleiderman

shirank1@gmail.com

over 13 years ago

In reply to: Scott Marlowe (#2)

Re: Memory issues

Hi
Thanks for your answer.
I understood that the server is ok memory wise.
What can I check on the client side or the DB queries?

Thank u.
On Wed, Sep 26, 2012 at 2:56 AM, Scott Marlowe <scott.marlowe@gmail.com>wrote:

On Mon, Sep 24, 2012 at 12:45 AM, Shiran Kleiderman <shirank1@gmail.com>
wrote:

Hi,
I'm using and Amazon ec2 instance with the following spec and the
application that I'm running uses a postgres DB 9.1.
The app has 3 main cron jobs.

Ubuntu 12, High-Memory Extra Large Instance
17.1 GB of memory
6.5 EC2 Compute Units (2 virtual cores with 3.25 EC2 Compute Units each)
420 GB of instance storage
64-bit platform

I've changed the main default values under file postgresql.conf to:
shared_buffers = 4GB
work_mem = 16MB
wal_buffers = 16MB
checkpoint_segments = 32
effective_cache_size = 8GB

When I run the app, after an hour or two, free -m looks like below ans

the

crons can't run due to memory loss or similar (i'm new to postgres and db
admin).
Thanks!

free -m, errors:

total used free shared buffers cached
Mem: 17079 13742 3337 0 64 11882
-/+ buffers/cache: 1796 15283
Swap: 511 0 511

You have 11.8G cached, that's basically free memory on demand.

total used free shared buffers cached
Mem: 17079 16833 245 0 42 14583
-/+ buffers/cache: 2207 14871
Swap: 511 0 511

Here you have 14.5G cached, again that's free memory so to speak.
I.e. when something needs it it gets allocated.

**free above stays low even when nothing is running.

**errors:
DBI connect('database=---;host=localhost','postgres',...) failed: could

not

fork new process for connection: Cannot allocate memory
could not fork new process for connection: Cannot allocate memory

This error is happening in your client process. Maybe it's 32 bit or
something and running out of local memory in its process space? Maybe
memory is so fragmented that no large blocks can get allocated or
something? Either way, your machine has plenty of memory according to
free. BTW, it's pretty common for folks new to unix to mis-read free
and not realize that cached memory + free memory is what's really
available.

--
Best,
Shiran Kleiderman
+972 - 542380838
Skype - shirank1

Scott Marlowe

scott.marlowe@gmail.com

over 13 years ago

In reply to: Shiran Kleiderman (#3)

Re: Memory issues

On Tue, Sep 25, 2012 at 7:00 PM, Shiran Kleiderman <shirank1@gmail.com> wrote:

Hi
Thanks for your answer.
I understood that the server is ok memory wise.
What can I check on the client side or the DB queries?

Well you're connecting to localhost so I'd expect you to show a memory
issue in free I'm not seeing. Are you really connecting to localhost
or not?

Shiran Kleiderman

shirank1@gmail.com

over 13 years ago

In reply to: Scott Marlowe (#4)

Re: Memory issues

Hi
Thanks again.
Right now, this is *free -m and ps aux* and non of the crons can run -
can't allocate memory.

cif@domU-12-31-39-08-06-20:~$ free -m
total used free shared buffers cached
Mem: 17079 12051 5028 0 270 9578
-/+ buffers/cache: 2202 14877
Swap: 511 0 511

cif@domU-12-31-39-08-06-20:~$ ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 24316 2280 ? Ss Sep24 0:00 /sbin/init
root 2 0.0 0.0 0 0 ? S Sep24 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S Sep24 0:00
[ksoftirqd/0]
root 4 0.0 0.0 0 0 ? S Sep24 0:00
[kworker/0:0]
root 5 0.0 0.0 0 0 ? S Sep24 0:00
[kworker/u:0]
root 6 0.0 0.0 0 0 ? S Sep24 0:00
[migration/0]
root 7 0.0 0.0 0 0 ? S Sep24 0:00
[watchdog/0]
root 8 0.0 0.0 0 0 ? S Sep24 0:00
[migration/1]
root 9 0.0 0.0 0 0 ? S Sep24 0:00
[kworker/1:0]
root 10 0.0 0.0 0 0 ? S Sep24 0:01
[ksoftirqd/1]
root 11 0.0 0.0 0 0 ? S Sep24 0:00
[watchdog/1]
root 12 0.0 0.0 0 0 ? S< Sep24 0:00 [cpuset]
root 13 0.0 0.0 0 0 ? S< Sep24 0:00 [khelper]
root 14 0.0 0.0 0 0 ? S Sep24 0:00 [kdevtmpfs]
root 15 0.0 0.0 0 0 ? S< Sep24 0:00 [netns]
root 16 0.0 0.0 0 0 ? S Sep24 0:00
[kworker/u:1]
root 17 0.0 0.0 0 0 ? S Sep24 0:00 [xenwatch]
root 18 0.0 0.0 0 0 ? S Sep24 0:00 [xenbus]
root 19 0.0 0.0 0 0 ? S Sep24 0:00
[sync_supers]
root 20 0.0 0.0 0 0 ? S Sep24 0:00
[bdi-default]
root 21 0.0 0.0 0 0 ? S< Sep24 0:00
[kintegrityd]
root 22 0.0 0.0 0 0 ? S< Sep24 0:00 [kblockd]
root 23 0.0 0.0 0 0 ? S< Sep24 0:00 [ata_sff]
root 24 0.0 0.0 0 0 ? S Sep24 0:00 [khubd]
root 25 0.0 0.0 0 0 ? S< Sep24 0:00 [md]
root 26 0.0 0.0 0 0 ? S Sep24 0:02
[kworker/0:1]
root 28 0.0 0.0 0 0 ? S Sep24 0:00
[khungtaskd]
root 29 0.0 0.0 0 0 ? S Sep24 0:00 [kswapd0]
root 30 0.0 0.0 0 0 ? SN Sep24 0:00 [ksmd]
root 31 0.0 0.0 0 0 ? S Sep24 0:00
[fsnotify_mark]
root 32 0.0 0.0 0 0 ? S Sep24 0:00
[ecryptfs-kthrea]
root 33 0.0 0.0 0 0 ? S< Sep24 0:00 [crypto]
root 41 0.0 0.0 0 0 ? S< Sep24 0:00 [kthrotld]
root 42 0.0 0.0 0 0 ? S Sep24 0:00 [khvcd]
root 43 0.0 0.0 0 0 ? S Sep24 0:01
[kworker/1:1]
root 62 0.0 0.0 0 0 ? S< Sep24 0:00
[devfreq_wq]
root 176 0.0 0.0 0 0 ? S< Sep24 0:00 [kdmflush]
root 187 0.0 0.0 0 0 ? S Sep24 0:01
[jbd2/xvda1-8]
root 188 0.0 0.0 0 0 ? S< Sep24 0:00
[ext4-dio-unwrit]
root 258 0.0 0.0 17224 640 ? S Sep24 0:00
upstart-udev-bridge --daemon
root 265 0.0 0.0 21460 1196 ? Ss Sep24 0:00
/sbin/udevd --daemon
root 328 0.0 0.0 21456 712 ? S Sep24 0:00
/sbin/udevd --daemon
root 329 0.0 0.0 21456 716 ? S Sep24 0:00
/sbin/udevd --daemon
root 389 0.0 0.0 15180 392 ? S Sep24 0:00
upstart-socket-bridge --daemon
root 419 0.0 0.0 7256 1008 ? Ss Sep24 0:00 dhclient3
-e IF_METRIC=100 -pf /var/run/dhclient.eth0.pid -lf
/var/lib/dhcp/dhclient.eth0.leases -1 eth
root 574 0.0 0.0 0 0 ? S Sep24 0:03
[jbd2/dm-0-8]
root 575 0.0 0.0 0 0 ? S< Sep24 0:00
[ext4-dio-unwrit]
root 610 0.0 0.0 49948 2880 ? Ss Sep24 0:00
/usr/sbin/sshd -D
syslog 625 0.0 0.0 253708 1552 ? Sl Sep24 0:11 rsyslogd
-c5
102 630 0.0 0.0 23808 944 ? Ss Sep24 0:00
dbus-daemon --system --fork --activation=upstart
root 687 0.0 0.0 14496 968 tty4 Ss+ Sep24 0:00
/sbin/getty -8 38400 tty4
root 696 0.0 0.0 14496 972 tty5 Ss+ Sep24 0:00
/sbin/getty -8 38400 tty5
root 708 0.0 0.0 14496 968 tty2 Ss+ Sep24 0:00
/sbin/getty -8 38400 tty2
root 710 0.0 0.0 14496 964 tty3 Ss+ Sep24 0:00
/sbin/getty -8 38400 tty3
root 715 0.0 0.0 14496 968 tty6 Ss+ Sep24 0:00
/sbin/getty -8 38400 tty6
root 720 0.0 0.0 4320 660 ? Ss Sep24 0:00 acpid -c
/etc/acpi/events -s /var/run/acpid.socket
root 728 0.0 11.9 2194848 2097324 ? Ss Sep24 0:09
/usr/bin/searchd --nodetach
root 733 0.0 0.0 19104 928 ? Ss Sep24 0:00 cron
daemon 735 0.0 0.0 16900 376 ? Ss Sep24 0:00 atd
bind 739 0.0 0.0 235540 13404 ? Ssl Sep24 0:00
/usr/sbin/named -u bind
mysql 755 0.0 0.2 558104 47940 ? Ssl Sep24 0:34
/usr/sbin/mysqld
whoopsie 790 0.0 0.0 187576 4236 ? Ssl Sep24 0:00 whoopsie
root 924 0.0 0.0 0 0 ? S Sep24 0:00
[flush-252:0]
root 999 0.0 0.0 99400 6496 ? Ss Sep24 0:04
/usr/sbin/apache2 -k start
www-data 1018 0.0 1.0 427080 185684 ? S Sep24 0:10
/usr/sbin/apache2 -k start
www-data 1019 0.0 1.0 427140 185852 ? S Sep24 0:33
/usr/sbin/apache2 -k start
root 1032 0.0 0.1 80220 21276 ? Ss Sep24 0:17 starman
master --port 5000 --daemonize -MMoose /usr/local/cif-rest-sphinx/CIF.psgi
root 1035 0.0 0.0 14496 968 tty1 Ss+ Sep24 0:00
/sbin/getty -8 38400 tty1
root 1037 0.0 0.1 184400 28532 ? S Sep24 0:00 starman
worker --port 5000 --daemonize -MMoose /usr/local/cif-rest-sphinx/CIF.psgi
root 1038 0.0 0.1 184444 28592 ? S Sep24 0:00 starman
worker --port 5000 --daemonize -MMoose /usr/local/cif-rest-sphinx/CIF.psgi
root 1039 0.0 0.1 184132 28040 ? S Sep24 0:00 starman
worker --port 5000 --daemonize -MMoose /usr/local/cif-rest-sphinx/CIF.psgi
root 1040 0.0 0.1 184408 28600 ? S Sep24 0:00 starman
worker --port 5000 --daemonize -MMoose /usr/local/cif-rest-sphinx/CIF.psgi
root 1041 0.0 0.1 184444 28588 ? S Sep24 0:00 starman
worker --port 5000 --daemonize -MMoose /usr/local/cif-rest-sphinx/CIF.psgi
www-data 1055 0.0 1.2 469948 225732 ? S Sep24 1:04
/usr/sbin/apache2 -k start
www-data 1056 0.0 1.0 427180 185924 ? S Sep24 0:28
/usr/sbin/apache2 -k start
www-data 3452 0.0 1.0 426964 185624 ? S Sep25 0:28
/usr/sbin/apache2 -k start
www-data 3775 0.0 1.0 426900 185696 ? S Sep25 0:14
/usr/sbin/apache2 -k start
postgres 4717 0.0 0.6 4411584 113372 ? S Sep25 0:01
/usr/lib/postgresql/9.1/bin/postgres -D /mnt/dbstorage/9.1/main -c
config_file=/etc/postgresql/9.1/main
postgres 4720 0.0 0.1 4413628 31392 ? Ss Sep25 0:09 postgres:
writer process
postgres 4721 0.0 0.0 4413636 1808 ? Ss Sep25 0:08 postgres:
wal writer process
postgres 4722 0.0 0.0 4414344 3044 ? Ss Sep25 1:36 postgres:
autovacuum launcher process
postgres 4723 0.0 0.0 94920 1752 ? Ss Sep25 0:41 postgres:
stats collector process
postgres 4738 0.0 7.0 4491580 1229488 ? Ss Sep25 0:58 postgres:
postgres cif 127.0.0.1(56867) idle
postgres 4740 0.0 6.9 4417748 1221100 ? Ss Sep25 0:18 postgres:
postgres cif 127.0.0.1(56869) idle
postgres 4741 0.0 7.0 4425720 1228856 ? Ss Sep25 0:31 postgres:
postgres cif 127.0.0.1(56870) idle
postgres 4742 0.0 6.9 4417376 1220464 ? Ss Sep25 0:08 postgres:
postgres cif 127.0.0.1(56871) idle
postgres 4743 0.0 7.0 4421104 1225328 ? Ss Sep25 0:24 postgres:
postgres cif 127.0.0.1(56872) idle
postgres 4745 0.0 7.0 4421124 1225040 ? Ss Sep25 0:27 postgres:
postgres cif 127.0.0.1(56874) idle
www-data 4746 0.0 1.2 455984 212284 ? S Sep25 0:30
/usr/sbin/apache2 -k start
postgres 4754 0.0 6.9 4417728 1220988 ? Ss Sep25 0:30 postgres:
postgres cif 127.0.0.1(56879) idle
www-data 4755 0.0 0.9 403836 163240 ? S Sep25 0:06
/usr/sbin/apache2 -k start
postgres 4765 0.0 6.9 4482528 1220704 ? Ss Sep25 0:23 postgres:
postgres cif 127.0.0.1(56881) idle
www-data 4995 0.0 0.9 403464 162872 ? S Sep25 0:03
/usr/sbin/apache2 -k start
www-data 4997 0.0 0.3 305460 64996 ? S Sep25 0:00
/usr/sbin/apache2 -k start
postgres 5002 0.0 6.9 4417384 1220172 ? Ss Sep25 0:09 postgres:
postgres cif 127.0.0.1(56895) idle
postgres 5003 0.0 7.1 4417920 1243772 ? Ss Sep25 0:06 postgres:
postgres cif 127.0.0.1(56896) idle
root 5218 0.0 0.0 0 0 ? S Sep25 0:00
[flush-202:1]
root 5820 0.0 0.0 73352 3568 ? Ss 16:37 0:00 sshd:
ubuntu [priv]
ubuntu 5950 0.0 0.0 73352 1676 ? S 16:37 0:00 sshd:
ubuntu@pts/0
ubuntu 5952 0.6 0.0 25872 8432 pts/0 Ss 16:37 0:00 -bash
root 6048 0.0 0.0 41896 1752 pts/0 S 16:38 0:00 sudo su -
cif
cif 6049 0.0 0.0 39516 1388 pts/0 S 16:38 0:00 su - cif
cif 6050 0.8 0.0 25912 8472 pts/0 S 16:38 0:00 -su
cif 6161 0.0 0.0 16872 1272 pts/0 R+ 16:38 0:00 ps aux

On Wed, Sep 26, 2012 at 6:29 PM, Scott Marlowe <scott.marlowe@gmail.com>wrote:

On Tue, Sep 25, 2012 at 7:00 PM, Shiran Kleiderman <shirank1@gmail.com>
wrote:

Hi
Thanks for your answer.
I understood that the server is ok memory wise.
What can I check on the client side or the DB queries?

Well you're connecting to localhost so I'd expect you to show a memory
issue in free I'm not seeing. Are you really connecting to localhost
or not?

--
Best,
Shiran Kleiderman
+972 - 542380838
Skype - shirank1

Scott Marlowe

scott.marlowe@gmail.com

over 13 years ago

In reply to: Shiran Kleiderman (#5)

Re: [GENERAL] Memory issues

On Wed, Sep 26, 2012 at 10:41 AM, Shiran Kleiderman <shirank1@gmail.com> wrote:

Hi
Thanks again.
Right now, this is free -m and ps aux and non of the crons can run - can't
allocate memory.

OK, so is the machine you're running free -m on the same as the one
running postgresql and the same one you're running cron jobs on and
the same one you're running apache on?

Also please don't remove the cc for the list, others might have an
insight I'd miss.

Show quoted text

cif@domU-12-31-39-08-06-20:~$ free -m
total used free shared buffers cached
Mem: 17079 12051 5028 0 270 9578
-/+ buffers/cache: 2202 14877
Swap: 511 0 511

Shiran Kleiderman

shirank1@gmail.com

over 13 years ago

In reply to: Scott Marlowe (#6)

Re: [GENERAL] Memory issues

Hi
Yes, same machine.

Thanks for your help.

On Wed, Sep 26, 2012 at 10:55 PM, Scott Marlowe <scott.marlowe@gmail.com>wrote:

On Wed, Sep 26, 2012 at 10:41 AM, Shiran Kleiderman <shirank1@gmail.com>
wrote:

Hi
Thanks again.
Right now, this is free -m and ps aux and non of the crons can run -

can't

allocate memory.

OK, so is the machine you're running free -m on the same as the one
running postgresql and the same one you're running cron jobs on and
the same one you're running apache on?

Also please don't remove the cc for the list, others might have an
insight I'd miss.

cif@domU-12-31-39-08-06-20:~$ free -m
total used free shared buffers cached
Mem: 17079 12051 5028 0 270 9578
-/+ buffers/cache: 2202 14877
Swap: 511 0 511

--
Best,
Shiran Kleiderman
+972 - 542380838
Skype - shirank1

Scott Marlowe

scott.marlowe@gmail.com

over 13 years ago

In reply to: Shiran Kleiderman (#7)

Re: [GENERAL] Memory issues

OK then I'm lost. It's got to either be a bug in how amazon ec2
instances work or severely fragmented memory because you've got a TON
of kernel cache available.

On Wed, Sep 26, 2012 at 3:00 PM, Shiran Kleiderman <shirank1@gmail.com> wrote:

Hi
Yes, same machine.

Thanks for your help.

On Wed, Sep 26, 2012 at 10:55 PM, Scott Marlowe <scott.marlowe@gmail.com>
wrote:

On Wed, Sep 26, 2012 at 10:41 AM, Shiran Kleiderman <shirank1@gmail.com>
wrote:

Hi
Thanks again.
Right now, this is free -m and ps aux and non of the crons can run -
can't
allocate memory.

OK, so is the machine you're running free -m on the same as the one
running postgresql and the same one you're running cron jobs on and
the same one you're running apache on?

Also please don't remove the cc for the list, others might have an
insight I'd miss.

cif@domU-12-31-39-08-06-20:~$ free -m
total used free shared buffers
cached
Mem: 17079 12051 5028 0 270
9578
-/+ buffers/cache: 2202 14877
Swap: 511 0 511

--
Best,
Shiran Kleiderman
+972 - 542380838
Skype - shirank1

--
To understand recursion, one must first understand recursion.

Shiran Kleiderman

shirank1@gmail.com

over 13 years ago

In reply to: Scott Marlowe (#8)

Re: [GENERAL] Memory issues

Hi
I contact amazon with this issue.
What can I check for the fragmented memory issue?

Thanks (:

On Wed, Sep 26, 2012 at 11:11 PM, Scott Marlowe <scott.marlowe@gmail.com>wrote:

OK then I'm lost. It's got to either be a bug in how amazon ec2
instances work or severely fragmented memory because you've got a TON
of kernel cache available.

On Wed, Sep 26, 2012 at 3:00 PM, Shiran Kleiderman <shirank1@gmail.com>
wrote:

Hi
Yes, same machine.

Thanks for your help.

On Wed, Sep 26, 2012 at 10:55 PM, Scott Marlowe <scott.marlowe@gmail.com

wrote:

On Wed, Sep 26, 2012 at 10:41 AM, Shiran Kleiderman <shirank1@gmail.com

wrote:

Hi
Thanks again.
Right now, this is free -m and ps aux and non of the crons can run -
can't
allocate memory.

OK, so is the machine you're running free -m on the same as the one
running postgresql and the same one you're running cron jobs on and
the same one you're running apache on?

Also please don't remove the cc for the list, others might have an
insight I'd miss.

cif@domU-12-31-39-08-06-20:~$ free -m
total used free shared buffers
cached
Mem: 17079 12051 5028 0 270
9578
-/+ buffers/cache: 2202 14877
Swap: 511 0 511

--
Best,
Shiran Kleiderman
+972 - 542380838
Skype - shirank1

--
To understand recursion, one must first understand recursion.

--
Best,
Shiran Kleiderman
+972 - 542380838
Skype - shirank1

#10

Shiran Kleiderman

shirank1@gmail.com

over 13 years ago

In reply to: Shiran Kleiderman (#9)

Re: [GENERAL] Memory issues

Hi
Another thing that may help,
When I restart the postgres db, then I have a little bit of "grace" time
(the memory free field is also lifted a bit).
I can run the crons and then after an hour or two the status returns to
regular... "out of memory" errors.

Thanks!

On Wed, Sep 26, 2012 at 11:36 PM, Shiran Kleiderman <shirank1@gmail.com>wrote:

Hi
I contact amazon with this issue.
What can I check for the fragmented memory issue?

Thanks (:

On Wed, Sep 26, 2012 at 11:11 PM, Scott Marlowe <scott.marlowe@gmail.com>wrote:

OK then I'm lost. It's got to either be a bug in how amazon ec2
instances work or severely fragmented memory because you've got a TON
of kernel cache available.

On Wed, Sep 26, 2012 at 3:00 PM, Shiran Kleiderman <shirank1@gmail.com>
wrote:

Hi
Yes, same machine.

Thanks for your help.

On Wed, Sep 26, 2012 at 10:55 PM, Scott Marlowe <

scott.marlowe@gmail.com>

wrote:

On Wed, Sep 26, 2012 at 10:41 AM, Shiran Kleiderman <

shirank1@gmail.com>

wrote:

Hi
Thanks again.
Right now, this is free -m and ps aux and non of the crons can run -
can't
allocate memory.

OK, so is the machine you're running free -m on the same as the one
running postgresql and the same one you're running cron jobs on and
the same one you're running apache on?

Also please don't remove the cc for the list, others might have an
insight I'd miss.

cif@domU-12-31-39-08-06-20:~$ free -m
total used free shared buffers
cached
Mem: 17079 12051 5028 0 270
9578
-/+ buffers/cache: 2202 14877
Swap: 511 0 511

--
Best,
Shiran Kleiderman
+972 - 542380838
Skype - shirank1

--
To understand recursion, one must first understand recursion.

--
Best,
Shiran Kleiderman
+972 - 542380838
Skype - shirank1

--
Best,
Shiran Kleiderman
+972 - 542380838
Skype - shirank1

#11

Andres Freund

andres@anarazel.de

over 13 years ago

In reply to: Shiran Kleiderman (#1)

Re: Memory issues

On Monday, September 24, 2012 08:45:06 AM Shiran Kleiderman wrote:

Hi,
I'm using and Amazon ec2 instance with the following spec and the
application that I'm running uses a postgres DB 9.1.
The app has 3 main cron jobs.

*Ubuntu 12, High-Memory Extra Large Instance
17.1 GB of memory
6.5 EC2 Compute Units (2 virtual cores with 3.25 EC2 Compute Units each)
420 GB of instance storage
64-bit platform*

I've changed the main default values under file *postgresql.conf* to:
shared_buffers = 4GB
work_mem = 16MB
wal_buffers = 16MB
checkpoint_segments = 32
effective_cache_size = 8GB

When I run the app, after an hour or two, free -m looks like below ans the
crons can't run due to memory loss or similar (i'm new to postgres and db
admin).
Thanks!

free -m, errors:

total used free shared buffers cached
Mem: 17079 13742 3337 0 64 11882
-/+ buffers/cache: 1796 15283
Swap: 511 0 511

total used *free* shared buffers cached
Mem: 17079 16833 *245 *0 42 14583
-/+ buffers/cache: 2207 14871
Swap: 511 0 511

**free above stays low even when nothing is running.

**errors:
*DBI connect('database=---;host=localhost','postgres',...) failed: could
not fork new process for connection: Cannot allocate memory*
could not fork new process for connection: Cannot allocate memory

and
execute failed: ERROR: out of memory
DETAIL: Failed on request of size 968. [for Statement "
SELECT DISTINCT....

could you show cat /proc/meminfo?

Greetings,

Andres
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

#12

Shiran Kleiderman

shirank1@gmail.com

over 13 years ago

In reply to: Andres Freund (#11)

Re: Memory issues

Hi
I've returned the memory configs to the default, erased data from my db and
am testing the system again.

This is the output of *cat /proc/meminfo*
Thanks

root@ip-10-194-167-240:~# cat /proc/meminfo
MemTotal: 7629508 kB
MemFree: 170368 kB
Buffers: 10272 kB
Cached: 6220848 kB
SwapCached: 0 kB
Active: 3249748 kB
Inactive: 3936960 kB
Active(anon): 971336 kB
Inactive(anon): 2103844 kB
Active(file): 2278412 kB
Inactive(file): 1833116 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 524284 kB
SwapFree: 522716 kB
Dirty: 83068 kB
Writeback: 3080 kB
AnonPages: 955856 kB
Mapped: 2132564 kB
Shmem: 2119424 kB
Slab: 157200 kB
SReclaimable: 144488 kB
SUnreclaim: 12712 kB
KernelStack: 1184 kB
PageTables: 21092 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 4339036 kB
Committed_AS: 3637424 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 26152 kB
VmallocChunk: 34359710052 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 7872512 kB
DirectMap2M: 0 kB

On Thu, Sep 27, 2012 at 8:59 AM, Andres Freund <andres@2ndquadrant.com>wrote:

On Monday, September 24, 2012 08:45:06 AM Shiran Kleiderman wrote:

Hi,
I'm using and Amazon ec2 instance with the following spec and the
application that I'm running uses a postgres DB 9.1.
The app has 3 main cron jobs.

*Ubuntu 12, High-Memory Extra Large Instance
17.1 GB of memory
6.5 EC2 Compute Units (2 virtual cores with 3.25 EC2 Compute Units each)
420 GB of instance storage
64-bit platform*

I've changed the main default values under file *postgresql.conf* to:
shared_buffers = 4GB
work_mem = 16MB
wal_buffers = 16MB
checkpoint_segments = 32
effective_cache_size = 8GB

When I run the app, after an hour or two, free -m looks like below ans

the

crons can't run due to memory loss or similar (i'm new to postgres and db
admin).
Thanks!

free -m, errors:

total used free shared buffers cached
Mem: 17079 13742 3337 0 64 11882
-/+ buffers/cache: 1796 15283
Swap: 511 0 511

total used *free* shared buffers cached
Mem: 17079 16833 *245 *0 42 14583
-/+ buffers/cache: 2207 14871
Swap: 511 0 511

**free above stays low even when nothing is running.

**errors:
*DBI connect('database=---;host=localhost','postgres',...) failed: could
not fork new process for connection: Cannot allocate memory*
could not fork new process for connection: Cannot allocate memory

and
execute failed: ERROR: out of memory
DETAIL: Failed on request of size 968. [for Statement "
SELECT DISTINCT....

could you show cat /proc/meminfo?

Greetings,

Andres
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Best,
Shiran Kleiderman
+972 - 542380838
Skype - shirank1

#13

Alexander Gataric

gataric@usa.net

over 13 years ago

In reply to: Shiran Kleiderman (#12)

Mapping PostgreSQL data types to DB2 Federated Server

Hello,

I'm on a project which requires adding PostgreSQL tables to DB2 Federated
Server. I'm getting an error with PostgreSQL data types boolean, text,
bytea, and XML. I believe this can be solved with the CREATE TYPE MAPPING in
Fed Server. Does anyone know which values to use? I'm not that familiar with
Fed Server.

Also, the Postgres data is being extracted and inserted from the same table
using Optim Archive. Does this pose an additional challenge with setting up
the mapping?

Thank you in advance

Alex

#14

John R Pierce

pierce@hogranch.com

over 13 years ago

In reply to: Alexander Gataric (#13)

Re: Mapping PostgreSQL data types to DB2 Federated Server

On 10/14/12 5:52 PM, Alexander Gataric wrote:

I'm on a project which requires adding PostgreSQL tables to DB2
Federated Server. I'm getting an error with PostgreSQL data types
boolean, text, bytea, and XML. I believe this can be solved with the
CREATE TYPE MAPPING in Fed Server. Does anyone know which values to
use? I'm not that familiar with Fed Server.

Also, the Postgres data is being extracted and inserted from the same
table using Optim Archive. Does this pose an additional challenge with
setting up the mapping?

I suggest you talk to your IBM support contacts for these issues, they
really have little to do with Postgres and are completely beyond
postgres' control.

--
john r pierce N 37, W 122
santa cruz ca mid-left coast

#15

Alexander Gataric

gataric@usa.net

over 13 years ago

In reply to: John R Pierce (#14)

Re: [GENERAL] Mapping PostgreSQL data types to DB2 Federated Server

The IBM people aren't being helpful so I thought I'd ask here.

Sent from my smartphone

----- Reply message -----
From: "John R Pierce" <pierce@hogranch.com>
To: <pgsql-general@postgresql.org>
Subject: [GENERAL] Mapping PostgreSQL data types to DB2 Federated Server
Date: Sun, Oct 14, 2012 8:05 pm

On 10/14/12 5:52 PM, Alexander Gataric wrote:

I'm on a project which requires adding PostgreSQL tables to DB2 > Federated Server. I'm getting an error with PostgreSQL data types > boolean, text, bytea, and XML. I believe this can be solved with the > CREATE TYPE MAPPING in Fed Server. Does anyone know which values to > use? I'm not that familiar with Fed Server.

Also, the Postgres data is being extracted and inserted from the same > table using Optim Archive. Does this pose an additional challenge with > setting up the mapping?

I suggest you talk to your IBM support contacts for these issues, they really have little to do with Postgres and are completely beyond postgres' control.

--
john r pierce N 37, W 122
santa cruz ca mid-left coast

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Import Notes

Resolved by subject fallback

#16

Craig Ringer

craig@2ndquadrant.com

over 13 years ago

In reply to: Alexander Gataric (#15)

Re: Re: [GENERAL] Mapping PostgreSQL data types to DB2 Federated Server

On 10/15/2012 09:37 AM, Alexander Gataric wrote:

The IBM people aren't being helpful so I thought I'd ask here.

Try dba.stackexchange.com .

--
Craig Ringer

#17

Shiran Kleiderman

shirank1@gmail.com

over 13 years ago

In reply to: Shiran Kleiderman (#12)

Re: Memory issues

Hi
This is the output of meminfo when the system is under some stress.
Thanks

cif@ip-10-194-167-240:/tmp$ cat /proc/meminfo
MemTotal: 7629508 kB
MemFree: 37820 kB
Buffers: 2108 kB
Cached: 5500200 kB
SwapCached: 332 kB
Active: 4172020 kB
Inactive: 3166244 kB
Active(anon): 1864040 kB
Inactive(anon): 1568760 kB
Active(file): 2307980 kB
Inactive(file): 1597484 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 524284 kB
SwapFree: 0 kB
Dirty: 23336 kB
Writeback: 0 kB
AnonPages: 1835716 kB
Mapped: 1610460 kB
Shmem: 1596916 kB
Slab: 136168 kB
SReclaimable: 123820 kB
SUnreclaim: 12348 kB
KernelStack: 1176 kB
PageTables: 23148 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 4339036 kB
Committed_AS: 4517524 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 26152 kB
VmallocChunk: 34359710052 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 7872512 kB
DirectMap2M: 0 kB

On Mon, Oct 15, 2012 at 2:45 AM, Shiran Kleiderman <shirank1@gmail.com>wrote:

Hi
I've returned the memory configs to the default, erased data from my db
and am testing the system again.

This is the output of *cat /proc/meminfo*
Thanks

root@ip-10-194-167-240:~# cat /proc/meminfo
MemTotal: 7629508 kB
MemFree: 170368 kB
Buffers: 10272 kB
Cached: 6220848 kB
SwapCached: 0 kB
Active: 3249748 kB
Inactive: 3936960 kB
Active(anon): 971336 kB
Inactive(anon): 2103844 kB
Active(file): 2278412 kB
Inactive(file): 1833116 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 524284 kB
SwapFree: 522716 kB
Dirty: 83068 kB
Writeback: 3080 kB
AnonPages: 955856 kB
Mapped: 2132564 kB
Shmem: 2119424 kB
Slab: 157200 kB
SReclaimable: 144488 kB
SUnreclaim: 12712 kB
KernelStack: 1184 kB
PageTables: 21092 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 4339036 kB
Committed_AS: 3637424 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 26152 kB
VmallocChunk: 34359710052 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 7872512 kB
DirectMap2M: 0 kB

On Thu, Sep 27, 2012 at 8:59 AM, Andres Freund <andres@2ndquadrant.com>wrote:

On Monday, September 24, 2012 08:45:06 AM Shiran Kleiderman wrote:

Hi,
I'm using and Amazon ec2 instance with the following spec and the
application that I'm running uses a postgres DB 9.1.
The app has 3 main cron jobs.

*Ubuntu 12, High-Memory Extra Large Instance
17.1 GB of memory
6.5 EC2 Compute Units (2 virtual cores with 3.25 EC2 Compute Units each)
420 GB of instance storage
64-bit platform*

I've changed the main default values under file *postgresql.conf* to:
shared_buffers = 4GB
work_mem = 16MB
wal_buffers = 16MB
checkpoint_segments = 32
effective_cache_size = 8GB

When I run the app, after an hour or two, free -m looks like below ans

the

crons can't run due to memory loss or similar (i'm new to postgres and

db

admin).
Thanks!

free -m, errors:

total used free shared buffers cached
Mem: 17079 13742 3337 0 64 11882
-/+ buffers/cache: 1796 15283
Swap: 511 0 511

total used *free* shared buffers cached
Mem: 17079 16833 *245 *0 42 14583
-/+ buffers/cache: 2207 14871
Swap: 511 0 511

**free above stays low even when nothing is running.

**errors:
*DBI connect('database=---;host=localhost','postgres',...) failed: could
not fork new process for connection: Cannot allocate memory*
could not fork new process for connection: Cannot allocate memory

and
execute failed: ERROR: out of memory
DETAIL: Failed on request of size 968. [for Statement "
SELECT DISTINCT....

could you show cat /proc/meminfo?

Greetings,

Andres
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Best,
Shiran Kleiderman
+972 - 542380838
Skype - shirank1

--
Best,
Shiran Kleiderman
+972 - 542380838
Skype - shirank1

#18

John R Pierce

pierce@hogranch.com

over 13 years ago

In reply to: Alexander Gataric (#15)

Re: Re: [GENERAL] Mapping PostgreSQL data types to DB2 Federated Server

On 10/14/12 6:37 PM, Alexander Gataric wrote:

On 10/14/12 5:52 PM, Alexander Gataric wrote:

I'm on a project which requires adding PostgreSQL tables to DB2 >

Federated Server. I'm getting an error with PostgreSQL data types >
boolean, text, bytea, and XML. I believe this can be solved with the >
CREATE TYPE MAPPING in Fed Server. Does anyone know which values to >
use? I'm not that familiar with Fed Server.

I googled db2 data types.
http://publib.boulder.ibm.com/infocenter/dzichelp/v2r2/index.jsp?topic=%2Fcom.ibm.db2z10.doc.intro%2Fsrc%2Ftpc%2Fdb2z_datatypes.htm

Postgres' TEXT would map to VARCHAR(n) but you'll have to decide on a
suitable max length for you n value. ther's nothing like boolean, so
probably mapping it either to a SMALLINT where 't' is 1 and 'f' is 0, or
mapping it to a char(1) and just storing it as t/f for true/false.
BYTEA is equivalent to a BLOB, whatever DB2 uses for that. and DB2 is
supposed to have an XML type, but I suppose you could just map it to
VARCHAR(n) again, as XML is really just a bunch of text formatted with
html-like entities.

--
john r pierce N 37, W 122
santa cruz ca mid-left coast