Swap on postgres master server

Hello,
we are running Postgres 10.5 with master slave replication.

These are our custom params

archive_command = 'pgbackrest --stanza=cmdprod archive-push %p' # command
to use to archive a logfile segment
archive_mode = on # enables archiving; off, on, or always
checkpoint_completion_target = 0.7 # checkpoint target duration, 0.0 - 1.0
default_text_search_config = 'pg_catalog.english' #
datestyle = 'iso, mdy' #
effective_cache_size = 120GB #
hot_standby = on # "on" allows queries during
recovery
lc_messages = 'en_US.UTF-8' # locale for system
error message
lc_monetary = 'en_US.UTF-8' # locale for monetary
formatting
lc_numeric = 'en_US.UTF-8' # locale for number
formatting
lc_time = 'en_US.UTF-8' # locale for time
formatting
listen_addresses = '*' # defaults to 'localhost', '*' = all
log_autovacuum_min_duration = 1000ms # -1 disables, 0 logs all
actions and
log_checkpoints = on #
log_line_prefix = '%t [%r] [%p]: [%l-1] db=%d,user=%u ' #
log_lock_waits = on # log lock waits >=
deadlock_timeout
log_min_duration_statement = 1000ms # -1 is disabled, 0 logs all
statements
log_statement = 'ddl' # none, ddl, mod, all
log_temp_files = 1024kB # log temporary files equal
or larger
maintenance_work_mem = 2GB #
max_connections = 220 #
max_parallel_workers_per_gather = 8 # taken from
max_worker_processes
max_wal_size = 2GB #
min_wal_size = 1GB #
pg_stat_statements.max = 10000 #
pg_stat_statements.track = all #
port = 5432 # port number which Postgres listen
shared_buffers = 10GB #
shared_preload_libraries = 'pg_stat_statements' # (change
requires restart)
synchronous_standby_names = '1 ( "usnyh2" )' # comment out during upgrade
track_activity_query_size = 16384 # (change requires restart)
track_io_timing = on #
wal_buffers = 16MB #
wal_keep_segments = 100 #
wal_level = replica # minimal, replica, or logical
work_mem = 600MB #

This server is on Centos 7 and the strange thing is that we see a lot of
swap usage :

[root@usnyh-cmd1 ~]# free -m
total used free shared buff/cache
available
Mem: 257652 7555 5559 12804 244536
236036
Swap: 16383 7326 9057

7GB used.

But can't see it from any of the commands like top etc.
I am sure it is postgres because it is the only service running on that
machine.

Is there anything we can do?
On the sync slave, the usage is just 400MB.

Any trick?

Thanks a lot,
Nicola

Are you sure that swap is used actively? Maybe it had just been used
during backup or something.

Look after SwapIn/SwapOut (si/so) it should be '0'
$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system--
------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id
wa st
1 0 12 89344 46608 586384 0 0 12 8 30 86 0 0
99 0 0

If you want to see the amount of ram used by each program with childs run
this:
ps -A --sort -rss -o comm,pmem | awk '
NR == 1 { print; next }
{ a[$1] += $2 }
END {
for (i in a) {
printf "%-15s\t%s\n", i, a[i];
}
}
'

On Tue, Oct 16, 2018 at 11:04 AM Nicola Contu <nicola.contu@gmail.com>
wrote:

Hello,
we are running Postgres 10.5 with master slave replication.

These are our custom params

archive_command = 'pgbackrest --stanza=cmdprod archive-push %p' #
command to use to archive a logfile segment
archive_mode = on # enables archiving; off, on, or always
checkpoint_completion_target = 0.7 # checkpoint target duration, 0.0 -
1.0
default_text_search_config = 'pg_catalog.english' #
datestyle = 'iso, mdy' #
effective_cache_size = 120GB #
hot_standby = on # "on" allows queries during
recovery
lc_messages = 'en_US.UTF-8' # locale for system
error message
lc_monetary = 'en_US.UTF-8' # locale for monetary
formatting
lc_numeric = 'en_US.UTF-8' # locale for number
formatting
lc_time = 'en_US.UTF-8' # locale for time
formatting
listen_addresses = '*' # defaults to 'localhost', '*' = all
log_autovacuum_min_duration = 1000ms # -1 disables, 0 logs all
actions and
log_checkpoints = on #
log_line_prefix = '%t [%r] [%p]: [%l-1] db=%d,user=%u ' #
log_lock_waits = on # log lock waits >=
deadlock_timeout
log_min_duration_statement = 1000ms # -1 is disabled, 0 logs all
statements
log_statement = 'ddl' # none, ddl, mod, all
log_temp_files = 1024kB # log temporary files equal
or larger
maintenance_work_mem = 2GB #
max_connections = 220 #
max_parallel_workers_per_gather = 8 # taken from
max_worker_processes
max_wal_size = 2GB #
min_wal_size = 1GB #
pg_stat_statements.max = 10000 #
pg_stat_statements.track = all #
port = 5432 # port number which Postgres
listen
shared_buffers = 10GB #
shared_preload_libraries = 'pg_stat_statements' # (change
requires restart)
synchronous_standby_names = '1 ( "usnyh2" )' # comment out during upgrade
track_activity_query_size = 16384 # (change requires restart)
track_io_timing = on #
wal_buffers = 16MB #
wal_keep_segments = 100 #
wal_level = replica # minimal, replica, or logical
work_mem = 600MB #

This server is on Centos 7 and the strange thing is that we see a lot of
swap usage :

[root@usnyh-cmd1 ~]# free -m
total used free shared buff/cache
available
Mem: 257652 7555 5559 12804 244536
236036
Swap: 16383 7326 9057

7GB used.

But can't see it from any of the commands like top etc.
I am sure it is postgres because it is the only service running on that
machine.

Is there anything we can do?
On the sync slave, the usage is just 400MB.

Any trick?

Thanks a lot,
Nicola

On Tue, 16 Oct 2018 at 10:48, Nicola Contu <nicola.contu@gmail.com> wrote:

No it is not probably used, because I can't find it in any way as I said.

I run your commands :

[root@usnyh-cmd1 ~]# vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
3 2 7505332 14545468 13692 241436784 0 0 47 115 0 0 11 2 84 2 0
3 2 7505332 14490408 13692 241436928 0 0 0 248 6153 2013 3 1 93 3 0
1 2 7505332 14474672 13692 241436800 0 0 0 1124 4905 1454 3 1 93 3 0
4 2 7505332 14381156 13692 241436832 0 0 0 96 5322 1782 2 1 94 3 0
^C
[root@usnyh-cmd1 ~]# ps -A --sort -rss -o comm,pmem | awk '

NR == 1 { print; next }
{ a[$1] += $2 }
END {
for (i in a) {
printf "%-15s\t%s\n", i, a[i];
}
}
'

COMMAND %MEM
kworker/42:2 0
kworker/60:6H 0
kworker/60:2H 0
kdmwork-253:2 0
ksoftirqd/60 0
postmaster 15.2
kworker/9:0H 0

So I'm just asking why it is still there with free -m and if there is any way to free that up if it is not used.

Thanks

Il giorno mar 16 ott 2018 alle ore 11:18 Hans Schou <hans.schou@gmail.com> ha scritto:

Are you sure that swap is used actively? Maybe it had just been used during backup or something.

Look after SwapIn/SwapOut (si/so) it should be '0'
$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 12 89344 46608 586384 0 0 12 8 30 86 0 0 99 0 0

If you want to see the amount of ram used by each program with childs run this:
ps -A --sort -rss -o comm,pmem | awk '
NR == 1 { print; next }
{ a[$1] += $2 }
END {
for (i in a) {
printf "%-15s\t%s\n", i, a[i];
}
}
'

On Tue, Oct 16, 2018 at 11:04 AM Nicola Contu <nicola.contu@gmail.com> wrote:

Hello,
we are running Postgres 10.5 with master slave replication.

These are our custom params

archive_command = 'pgbackrest --stanza=cmdprod archive-push %p' # command to use to archive a logfile segment
archive_mode = on # enables archiving; off, on, or always
checkpoint_completion_target = 0.7 # checkpoint target duration, 0.0 - 1.0
default_text_search_config = 'pg_catalog.english' #
datestyle = 'iso, mdy' #
effective_cache_size = 120GB #
hot_standby = on # "on" allows queries during recovery
lc_messages = 'en_US.UTF-8' # locale for system error message
lc_monetary = 'en_US.UTF-8' # locale for monetary formatting
lc_numeric = 'en_US.UTF-8' # locale for number formatting
lc_time = 'en_US.UTF-8' # locale for time formatting
listen_addresses = '*' # defaults to 'localhost', '*' = all
log_autovacuum_min_duration = 1000ms # -1 disables, 0 logs all actions and
log_checkpoints = on #
log_line_prefix = '%t [%r] [%p]: [%l-1] db=%d,user=%u ' #
log_lock_waits = on # log lock waits >= deadlock_timeout
log_min_duration_statement = 1000ms # -1 is disabled, 0 logs all statements
log_statement = 'ddl' # none, ddl, mod, all
log_temp_files = 1024kB # log temporary files equal or larger
maintenance_work_mem = 2GB #
max_connections = 220 #
max_parallel_workers_per_gather = 8 # taken from max_worker_processes
max_wal_size = 2GB #
min_wal_size = 1GB #
pg_stat_statements.max = 10000 #
pg_stat_statements.track = all #
port = 5432 # port number which Postgres listen
shared_buffers = 10GB #
shared_preload_libraries = 'pg_stat_statements' # (change requires restart)
synchronous_standby_names = '1 ( "usnyh2" )' # comment out during upgrade
track_activity_query_size = 16384 # (change requires restart)
track_io_timing = on #
wal_buffers = 16MB #
wal_keep_segments = 100 #
wal_level = replica # minimal, replica, or logical
work_mem = 600MB #

This server is on Centos 7 and the strange thing is that we see a lot of swap usage :

[root@usnyh-cmd1 ~]# free -m
total used free shared buff/cache available
Mem: 257652 7555 5559 12804 244536 236036
Swap: 16383 7326 9057

7GB used.

But can't see it from any of the commands like top etc.
I am sure it is postgres because it is the only service running on that machine.

Is there anything we can do?
On the sync slave, the usage is just 400MB.

Any trick?

Thanks a lot,
Nicola

I guess you can run swapoff (followed by swapon). That will free up
whatever is currently swapped. Beware if the system is actively
swapping then swapoff can take some time. But it seems not in your
case.
On Tue, 16 Oct 2018 at 10:48, Nicola Contu <nicola.contu@gmail.com> wrote:

No it is not probably used, because I can't find it in any way as I said.

I run your commands :

[root@usnyh-cmd1 ~]# vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system--

------cpu-----

r b swpd free buff cache si so bi bo in cs us sy

id wa st

3 2 7505332 14545468 13692 241436784 0 0 47 115 0 0

11 2 84 2 0

3 2 7505332 14490408 13692 241436928 0 0 0 248 6153 2013

3 1 93 3 0

1 2 7505332 14474672 13692 241436800 0 0 0 1124 4905 1454

3 1 93 3 0

4 2 7505332 14381156 13692 241436832 0 0 0 96 5322 1782

2 1 94 3 0

^C
[root@usnyh-cmd1 ~]# ps -A --sort -rss -o comm,pmem | awk '

NR == 1 { print; next }
{ a[$1] += $2 }
END {
for (i in a) {
printf "%-15s\t%s\n", i, a[i];
}
}
'

COMMAND %MEM
kworker/42:2 0
kworker/60:6H 0
kworker/60:2H 0
kdmwork-253:2 0
ksoftirqd/60 0
postmaster 15.2
kworker/9:0H 0

So I'm just asking why it is still there with free -m and if there is

any way to free that up if it is not used.

Thanks

Il giorno mar 16 ott 2018 alle ore 11:18 Hans Schou <

hans.schou@gmail.com> ha scritto:

Are you sure that swap is used actively? Maybe it had just been used

during backup or something.

Look after SwapIn/SwapOut (si/so) it should be '0'
$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system--

------cpu-----

r b swpd free buff cache si so bi bo in cs us sy

id wa st

1 0 12 89344 46608 586384 0 0 12 8 30 86 0 0

99 0 0

If you want to see the amount of ram used by each program with childs

run this:

ps -A --sort -rss -o comm,pmem | awk '
NR == 1 { print; next }
{ a[$1] += $2 }
END {
for (i in a) {
printf "%-15s\t%s\n", i, a[i];
}
}
'

On Tue, Oct 16, 2018 at 11:04 AM Nicola Contu <nicola.contu@gmail.com>

wrote:

Hello,
we are running Postgres 10.5 with master slave replication.

These are our custom params

archive_command = 'pgbackrest --stanza=cmdprod archive-push %p' #

command to use to archive a logfile segment

archive_mode = on # enables archiving; off, on, or always
checkpoint_completion_target = 0.7 # checkpoint target duration, 0.0 -

1.0

default_text_search_config = 'pg_catalog.english' #
datestyle = 'iso, mdy' #
effective_cache_size = 120GB #
hot_standby = on # "on" allows queries during

recovery

lc_messages = 'en_US.UTF-8' # locale for system

error message

lc_monetary = 'en_US.UTF-8' # locale for monetary

formatting

lc_numeric = 'en_US.UTF-8' # locale for number

formatting

lc_time = 'en_US.UTF-8' # locale for time

formatting

listen_addresses = '*' # defaults to 'localhost', '*' = all
log_autovacuum_min_duration = 1000ms # -1 disables, 0 logs all

actions and

log_checkpoints = on #
log_line_prefix = '%t [%r] [%p]: [%l-1] db=%d,user=%u ' #
log_lock_waits = on # log lock waits >=

deadlock_timeout

log_min_duration_statement = 1000ms # -1 is disabled, 0 logs

all statements

log_statement = 'ddl' # none, ddl, mod, all
log_temp_files = 1024kB # log temporary files

equal or larger

maintenance_work_mem = 2GB #
max_connections = 220 #
max_parallel_workers_per_gather = 8 # taken from

max_worker_processes

max_wal_size = 2GB #
min_wal_size = 1GB #
pg_stat_statements.max = 10000 #
pg_stat_statements.track = all #
port = 5432 # port number which Postgres

listen

shared_buffers = 10GB #
shared_preload_libraries = 'pg_stat_statements' # (change

requires restart)

synchronous_standby_names = '1 ( "usnyh2" )' # comment out during

upgrade

track_activity_query_size = 16384 # (change requires restart)
track_io_timing = on #
wal_buffers = 16MB #
wal_keep_segments = 100 #
wal_level = replica # minimal, replica, or logical
work_mem = 600MB #

This server is on Centos 7 and the strange thing is that we see a lot

of swap usage :

[root@usnyh-cmd1 ~]# free -m
total used free shared buff/cache

available

Mem: 257652 7555 5559 12804 244536

236036

Swap: 16383 7326 9057

7GB used.

But can't see it from any of the commands like top etc.
I am sure it is postgres because it is the only service running on

that machine.

Is there anything we can do?
On the sync slave, the usage is just 400MB.

Any trick?

Thanks a lot,
Nicola