Background writer committed

TODO updated:

* -Use background process to write dirty shared buffers to disk

---------------------------------------------------------------------------

Jan Wieck wrote:

I committed the first part of the background writer process. We had a
consensus on attempting to avoid write() calls from regular backends,
but did no come to any conclusions what to do to force the kernel to
actually do some IO.

Consequently, this patch is a separate process launched by postmaster,
that periodically write()'s out "some" dirty buffers in LRU order. This
causes the buffers returned for replacement (when a backend needs to
read in a page) to be clean allways. The process does no sync(), fsync()
or any other calls thus far. Nothing has changed in the checkpoint logic
either.

The configuration options controlling the process are all PGC_SIGHUP:

# - Background writer -
#bgwriter_delay = 200 # 10-5000 milliseconds
#bgwriter_percent = 1 # 0-100% of dirty buffers
#bgwriter_maxpages = 100 # 1-1000 buffers max at once

Delay is the number of milliseconds to wait between loops. If there was
nothing to do at all in one loop (all buffers clean), then the process
will sleep for 10 seconds.

Percent is the percentage of "dirty pages" to write per loop. This is
independant of the size of the buffer pool. If percent = 0 the
postmaster will not start the process at all.

Maxpages is an upper bound to prevent the background writer from
producing a write storm if a sequential operation causes all pages of a
large buffer pool to be dirtied at once.

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Jan Wieck wrote:

I committed the first part of the background writer process. We had a
consensus on attempting to avoid write() calls from regular backends,
but did no come to any conclusions what to do to force the kernel to
actually do some IO.

Consequently, this patch is a separate process launched by postmaster,
that periodically write()'s out "some" dirty buffers in LRU order. This
causes the buffers returned for replacement (when a backend needs to
read in a page) to be clean allways. The process does no sync(), fsync()
or any other calls thus far. Nothing has changed in the checkpoint logic
either.

Can we have some idea where to tweak sync routines for comparing results?

I mean I would like to run pgbench with same config all along and compare the
performance difference between sync, fsync and fdatasync etc.

If we could get to run any live world data test by that, it would be great as well.

Shridhar

Shridhar Daithankar wrote:

Jan Wieck wrote:

I committed the first part of the background writer process. We had a
consensus on attempting to avoid write() calls from regular backends,
but did no come to any conclusions what to do to force the kernel to
actually do some IO.

Consequently, this patch is a separate process launched by postmaster,
that periodically write()'s out "some" dirty buffers in LRU order. This
causes the buffers returned for replacement (when a backend needs to
read in a page) to be clean allways. The process does no sync(), fsync()
or any other calls thus far. Nothing has changed in the checkpoint logic
either.

Can we have some idea where to tweak sync routines for comparing results?

I mean I would like to run pgbench with same config all along and compare the
performance difference between sync, fsync and fdatasync etc.

pgbench is actually a very bad example to test any cache strategy.
Either 98% of your lookups result in cache hits, so basically your
entire database is cached, or it doesn't fit and every cache strategy
becomes useless. It doesn't have parts that fit and other parts that
don't. I think pgbench doesn't use non-uniform random access as real
world applications do (you have bestsellers and other items, you have
frequent customers and once-a-year visitors). So it's very hard to get
the system into a state where you have like 50% cache hitrate.

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #