strange 'vacuum verbose analyze' behaviour

Oleg Bartunov <oleg@sai.msu.su> writes:

Day ago we run 'vacuum verbose analyze;' and now we're observing
strange output (see below). We see many repeated passes through the
table 'usno' and all indices (2).

Nothing strange about it: that's how vacuum deals with large tables.
You can reduce the number of passes over the indexes by increasing
maintenance_work_mem (aka vacuum_mem). It looks like you have that
set to 16MB, which is enough to hold about 2.79M 6-byte CTIDs. When
that fills up, we have to go and clean out the corresponding index
entries before we resume scanning the heap.

It is a bit odd that nothing is getting deleted from "ipix_ind" ...
is that a partial index?

regards, tom lane

On Thu, 27 Jan 2005, Tom Lane wrote:

Oleg Bartunov <oleg@sai.msu.su> writes:

Day ago we run 'vacuum verbose analyze;' and now we're observing
strange output (see below). We see many repeated passes through the
table 'usno' and all indices (2).

Nothing strange about it: that's how vacuum deals with large tables.
You can reduce the number of passes over the indexes by increasing
maintenance_work_mem (aka vacuum_mem). It looks like you have that
set to 16MB, which is enough to hold about 2.79M 6-byte CTIDs. When
that fills up, we have to go and clean out the corresponding index
entries before we resume scanning the heap.

thanks for info, just increased maintenance_work_mem and will see

It is a bit odd that nothing is getting deleted from "ipix_ind" ...
is that a partial index?

no.

wsdb=# \d ipix_ind
Index "public.ipix_ind"
Column | Type
--------+--------
ipix | bigint
btree, for table "public.usno"

regards, tom lane

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

On Fri, 28 Jan 2005, Oleg Bartunov wrote:

On Thu, 27 Jan 2005, Tom Lane wrote:

Oleg Bartunov <oleg@sai.msu.su> writes:

Day ago we run 'vacuum verbose analyze;' and now we're observing
strange output (see below). We see many repeated passes through the
table 'usno' and all indices (2).

Nothing strange about it: that's how vacuum deals with large tables.
You can reduce the number of passes over the indexes by increasing
maintenance_work_mem (aka vacuum_mem). It looks like you have that
set to 16MB, which is enough to hold about 2.79M 6-byte CTIDs. When
that fills up, we have to go and clean out the corresponding index
entries before we resume scanning the heap.

I've increased maintenance_work_mem to 65536 (4x default value) and now I see
postmaster eats more than 1Gb memory !

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 20458
postgres 15 0 1350m 522m 204m D 8.0 51.6 1:08.26 postmaster

Is this normal ? I've also increased checkpoint_segments to 12,
but it shouldn't affect to memory.

Memory growth stoped at 1.8Gb

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20458 postgres 15 0 1902m 503m 204m D 5.9 49.7 13:59.61 postmaster

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

Oleg Bartunov <oleg@sai.msu.su> writes:

Memory growth stoped at 1.8Gb

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20458 postgres 15 0 1902m 503m 204m D 5.9 49.7 13:59.61 postmaster

Index-related memory leak maybe? What are the indexes on this table,
exactly?

regards, tom lane

On Fri, 28 Jan 2005, Tom Lane wrote:

Oleg Bartunov <oleg@sai.msu.su> writes:

Memory growth stoped at 1.8Gb

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20458 postgres 15 0 1902m 503m 204m D 5.9 49.7 13:59.61 postmaster

Index-related memory leak maybe? What are the indexes on this table,
exactly?

nothing special :)

CREATE TABLE usno (
ra real,
dec real,
bmag real,
rmag real);

COPY usno FROM stdin with delimiter '|';

CREATE INDEX radec_idx1 ON usno (ra,dec) ;

ALTER TABLE usno ADD COLUMN ipix bigint;

UPDATE usno SET ipix = q3c_ang2pix(ra,dec);

CREATE INDEX ipix_ind ON usno (ipix);

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

Tom,

After almost 20 hours running vacuum I see postmaster grew a little bit:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20458 postgres 15 0 2136m 553m 204m D 33.2 54.7 198:18.36 postmaster

It's strange that I see no output since starting
vacuumdb -v -z -f wsdb > vacuum-wsdb.log 2>&1&

-rw-r--r-- 1 postgres postgres 48784 Jan 28 15:03 vacuum-wsdb.log

Is there something I could do ?

Oleg

On Fri, 28 Jan 2005, Oleg Bartunov wrote:

On Fri, 28 Jan 2005, Tom Lane wrote:

Oleg Bartunov <oleg@sai.msu.su> writes:

Memory growth stoped at 1.8Gb

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20458 postgres 15 0 1902m 503m 204m D 5.9 49.7 13:59.61 postmaster

Index-related memory leak maybe? What are the indexes on this table,
exactly?

nothing special :)

CREATE TABLE usno (
ra real,
dec real,
bmag real,
rmag real);

COPY usno FROM stdin with delimiter '|';

CREATE INDEX radec_idx1 ON usno (ra,dec) ;

ALTER TABLE usno ADD COLUMN ipix bigint;

UPDATE usno SET ipix = q3c_ang2pix(ra,dec);

CREATE INDEX ipix_ind ON usno (ipix);

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83