Separating bgwriter and checkpointer
As discussed previously...
Currently the bgwriter process performs both background writing,
checkpointing and some other duties. This means that we can't perform
the final checkpoint fsync without stopping background writing, so
there is a negative performance effect from doing both things in one
process.
Additionally, our aim in 9.2 is to replace polling loops with latches
for power reduction. The complexity of the bgwriter loops is high and
it seems unlikely to come up with a clean approach using latches.
This patch splits bgwriter into 2 processes: checkpointer and
bgwriter, seeking to avoid contentious changes. Additional changes are
expected in this release to build upon these changes for both new
processes, though this patch stands on its own as both a performance
vehicle and in some ways a refcatoring to simplify the code.
Checkpointer does the important things, "new bgwriter" just does
background writing and so is much less important than before.
Current patch has a bug at shutdown I've not located yet, but seems
likely is a simple error. That is mainly because for personal reasons
I've not been able to work on the patch recently. I expect to be able
to fix that later in the CF.
--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Attachments:
bgwriter_split.v1.patchapplication/octet-stream; name=bgwriter_split.v1.patchDownload+320-1056
On Fri, Sep 16, 2011 at 7:53 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
This patch splits bgwriter into 2 processes: checkpointer and
bgwriter, seeking to avoid contentious changes. Additional changes are
expected in this release to build upon these changes for both new
processes, though this patch stands on its own as both a performance
vehicle and in some ways a refcatoring to simplify the code.
I like this idea to simplify the code. How much performance gain can we
expect by this patch?
Current patch has a bug at shutdown I've not located yet, but seems
likely is a simple error. That is mainly because for personal reasons
I've not been able to work on the patch recently. I expect to be able
to fix that later in the CF.
You seem to have forgotten to include checkpointor.c and .h in the patch.
Regards,
--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
On Fri, Sep 16, 2011 at 2:38 AM, Fujii Masao <masao.fujii@gmail.com> wrote:
On Fri, Sep 16, 2011 at 7:53 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
This patch splits bgwriter into 2 processes: checkpointer and
bgwriter, seeking to avoid contentious changes. Additional changes are
expected in this release to build upon these changes for both new
processes, though this patch stands on its own as both a performance
vehicle and in some ways a refcatoring to simplify the code.I like this idea to simplify the code. How much performance gain can we
expect by this patch?
On heavily I/O bound systems, this is likely to make a noticeable
difference, since bgwriter reduces I/O in user processes.
The overhead of sending signals between processes is much less than I
had previously thought, so I expect no problems there, even on highly
loaded systems.
Current patch has a bug at shutdown I've not located yet, but seems
likely is a simple error. That is mainly because for personal reasons
I've not been able to work on the patch recently. I expect to be able
to fix that later in the CF.You seem to have forgotten to include checkpointor.c and .h in the patch.
I confirm this error. I'll repost full patch later in the week when I
have more time.
--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
On 20.09.2011 10:48, Simon Riggs wrote:
On Fri, Sep 16, 2011 at 2:38 AM, Fujii Masao<masao.fujii@gmail.com> wrote:
On Fri, Sep 16, 2011 at 7:53 AM, Simon Riggs<simon@2ndquadrant.com> wrote:
This patch splits bgwriter into 2 processes: checkpointer and
bgwriter, seeking to avoid contentious changes. Additional changes are
expected in this release to build upon these changes for both new
processes, though this patch stands on its own as both a performance
vehicle and in some ways a refcatoring to simplify the code.I like this idea to simplify the code. How much performance gain can we
expect by this patch?On heavily I/O bound systems, this is likely to make a noticeable
difference, since bgwriter reduces I/O in user processes.
Hmm. If the system is I/O bound, it doesn't matter which process
performs the I/O. It's still the same amount of I/O in total, and in an
I/O bound system, that's what determines the overall throughput.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
On Tue, Sep 20, 2011 at 9:06 AM, Heikki Linnakangas
<heikki.linnakangas@enterprisedb.com> wrote:
On 20.09.2011 10:48, Simon Riggs wrote:
On Fri, Sep 16, 2011 at 2:38 AM, Fujii Masao<masao.fujii@gmail.com>
wrote:On Fri, Sep 16, 2011 at 7:53 AM, Simon Riggs<simon@2ndquadrant.com>
wrote:This patch splits bgwriter into 2 processes: checkpointer and
bgwriter, seeking to avoid contentious changes. Additional changes are
expected in this release to build upon these changes for both new
processes, though this patch stands on its own as both a performance
vehicle and in some ways a refcatoring to simplify the code.I like this idea to simplify the code. How much performance gain can we
expect by this patch?On heavily I/O bound systems, this is likely to make a noticeable
difference, since bgwriter reduces I/O in user processes.Hmm. If the system is I/O bound, it doesn't matter which process performs
the I/O. It's still the same amount of I/O in total, and in an I/O bound
system, that's what determines the overall throughput.
That's true, but not relevant.
The bgwriter avoids I/O, if it is operating correctly. This patch
ensures it continues to operate even during heavy checkpoints. So it
helps avoid extra I/O during a period of very high I/O activity.
--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
On 20.09.2011 11:18, Simon Riggs wrote:
The bgwriter avoids I/O, if it is operating correctly. This patch
ensures it continues to operate even during heavy checkpoints. So it
helps avoid extra I/O during a period of very high I/O activity.
I don't see what difference it makes which process does the I/O. If a
write() by checkpointer process blocks, any write()s by the separate
bgwriter process at that time will block too. If the I/O is not
saturated, and the checkpoint write()s don't block, then even without
this patch, the bgwriter process can handle its usual bgwriter duties
during checkpoint just fine. (And if the I/O is not saturated, it's not
an I/O bound system anyway.)
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
On Tue, Sep 20, 2011 at 10:03 AM, Heikki Linnakangas
<heikki.linnakangas@enterprisedb.com> wrote:
On 20.09.2011 11:18, Simon Riggs wrote:
The bgwriter avoids I/O, if it is operating correctly. This patch
ensures it continues to operate even during heavy checkpoints. So it
helps avoid extra I/O during a period of very high I/O activity.I don't see what difference it makes which process does the I/O. If a
write() by checkpointer process blocks, any write()s by the separate
bgwriter process at that time will block too. If the I/O is not saturated,
and the checkpoint write()s don't block, then even without this patch, the
bgwriter process can handle its usual bgwriter duties during checkpoint just
fine. (And if the I/O is not saturated, it's not an I/O bound system
anyway.)
Whatever value you assign to the bgwriter, then this patch makes sure
that happens during heavy fsyncs.
--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
On Tue, Sep 20, 2011 at 11:03 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
I don't see what difference it makes which process does the I/O. If a
write() by checkpointer process blocks, any write()s by the separate
bgwriter process at that time will block too. If the I/O is not saturated,
and the checkpoint write()s don't block, then even without this patch, the
bgwriter process can handle its usual bgwriter duties during checkpoint just
fine. (And if the I/O is not saturated, it's not an I/O bound system
anyway.)Whatever value you assign to the bgwriter, then this patch makes sure
that happens during heavy fsyncs.
I think his point is that it doesn't because if the heavy fsyncs cause
the system to be i/o bound it then bgwriter will just block issuing
the writes instead of the fsyncs.
I'm not actually convinced. Writes will only block if the kernel
decides to block. We don't really know how the kernel makes this
decision but it's entirely possible that having pending physical i/o
issued due to an fsync doesn't influence the decision if there is
still a reasonable number of dirty pages in the buffer cache. In a
sense, "I/O bound" means different things for write and fsync. Or to
put it another way fsync is latency sensitive but write is only
bandwidth sensitive.
All that said my question is which way is the code more legible and
easier to follow?
On 20.09.2011 16:29, Greg Stark wrote:
On Tue, Sep 20, 2011 at 11:03 AM, Simon Riggs<simon@2ndquadrant.com> wrote:
I don't see what difference it makes which process does the I/O. If a
write() by checkpointer process blocks, any write()s by the separate
bgwriter process at that time will block too. If the I/O is not saturated,
and the checkpoint write()s don't block, then even without this patch, the
bgwriter process can handle its usual bgwriter duties during checkpoint just
fine. (And if the I/O is not saturated, it's not an I/O bound system
anyway.)Whatever value you assign to the bgwriter, then this patch makes sure
that happens during heavy fsyncs.I think his point is that it doesn't because if the heavy fsyncs cause
the system to be i/o bound it then bgwriter will just block issuing
the writes instead of the fsyncs.I'm not actually convinced. Writes will only block if the kernel
decides to block. We don't really know how the kernel makes this
decision but it's entirely possible that having pending physical i/o
issued due to an fsync doesn't influence the decision if there is
still a reasonable number of dirty pages in the buffer cache. In a
sense, "I/O bound" means different things for write and fsync. Or to
put it another way fsync is latency sensitive but write is only
bandwidth sensitive.
Yeah, I was thinking of write()s, not fsyncs. I agree this might have
some effect during fsync phase.
All that said my question is which way is the code more legible and
easier to follow?
Hear hear. If we're going to give the bgwriter more responsibilities,
this might make sense even if it has no effect on performance.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
On Tue, Sep 20, 2011 at 15:35, Heikki Linnakangas
<heikki.linnakangas@enterprisedb.com> wrote:
On 20.09.2011 16:29, Greg Stark wrote:
On Tue, Sep 20, 2011 at 11:03 AM, Simon Riggs<simon@2ndquadrant.com>
wrote:I don't see what difference it makes which process does the I/O. If a
write() by checkpointer process blocks, any write()s by the separate
bgwriter process at that time will block too. If the I/O is not
saturated,
and the checkpoint write()s don't block, then even without this patch,
the
bgwriter process can handle its usual bgwriter duties during checkpoint
just
fine. (And if the I/O is not saturated, it's not an I/O bound system
anyway.)Whatever value you assign to the bgwriter, then this patch makes sure
that happens during heavy fsyncs.I think his point is that it doesn't because if the heavy fsyncs cause
the system to be i/o bound it then bgwriter will just block issuing
the writes instead of the fsyncs.I'm not actually convinced. Writes will only block if the kernel
decides to block. We don't really know how the kernel makes this
decision but it's entirely possible that having pending physical i/o
issued due to an fsync doesn't influence the decision if there is
still a reasonable number of dirty pages in the buffer cache. In a
sense, "I/O bound" means different things for write and fsync. Or to
put it another way fsync is latency sensitive but write is only
bandwidth sensitive.Yeah, I was thinking of write()s, not fsyncs. I agree this might have some
effect during fsync phase.All that said my question is which way is the code more legible and
easier to follow?Hear hear. If we're going to give the bgwriter more responsibilities, this
might make sense even if it has no effect on performance.
Isn't there also the advantage of that work put in two different
processes can use two different CPU cores? Or is that likely to never
ever come in play here?
--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/
On 20.09.2011 16:49, Magnus Hagander wrote:
Isn't there also the advantage of that work put in two different
processes can use two different CPU cores? Or is that likely to never
ever come in play here?
You would need one helluva I/O system to saturate even a single CPU,
just by doing write+fsync.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
2011/9/20 Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>:
On 20.09.2011 16:49, Magnus Hagander wrote:
Isn't there also the advantage of that work put in two different
processes can use two different CPU cores? Or is that likely to never
ever come in play here?You would need one helluva I/O system to saturate even a single CPU, just by
doing write+fsync.
The point of Magnus is valid. There are possible throttling done by
linux per node, per process/task.
Since ..2.6.37 (32 ?) I believe .. there are more temptation to have
have per cgroup io/sec limits, and there exists some promising work
done to have a better IO bandwith throttling per process.
IMO, splitting the type of IO workload per process allows the
administrators to have more control on the IO limits they want to have
(and it may help the kernels() to have a better strategy ?)
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
--
Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation
On 20.09.2011 17:31, C�dric Villemain wrote:
2011/9/20 Heikki Linnakangas<heikki.linnakangas@enterprisedb.com>:
On 20.09.2011 16:49, Magnus Hagander wrote:
Isn't there also the advantage of that work put in two different
processes can use two different CPU cores? Or is that likely to never
ever come in play here?You would need one helluva I/O system to saturate even a single CPU, just by
doing write+fsync.The point of Magnus is valid. There are possible throttling done by
linux per node, per process/task.
Since ..2.6.37 (32 ?) I believe .. there are more temptation to have
have per cgroup io/sec limits, and there exists some promising work
done to have a better IO bandwith throttling per process.IMO, splitting the type of IO workload per process allows the
administrators to have more control on the IO limits they want to have
(and it may help the kernels() to have a better strategy ?)
That is a separate issue from being able to use different CPU cores. But
cool! I didn't know Linux can do that nowadays. That could be highly
useful, if you can put e.g autovacuum on a different cgroup from regular
backends.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
On Tue, Sep 20, 2011 at 9:35 AM, Heikki Linnakangas
<heikki.linnakangas@enterprisedb.com> wrote:
All that said my question is which way is the code more legible and
easier to follow?Hear hear. If we're going to give the bgwriter more responsibilities, this
might make sense even if it has no effect on performance.
I agree. I don't think this change needs to be justified on
performance grounds; there are enough collateral benefits to make it
worthwhile. If the checkpoint process handles all the stuff with
highly variable latency (i.e. fsyncs), then the background writer work
will happen more regularly and predictably. The code will also be
simpler, which I think will open up opportunities for additional
optimizations such as (perhaps) making the background writer only wake
up when there are dirty buffers to write, which ties in to
longstanding concerns about power consumption.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Fri, Sep 16, 2011 at 01:53, Simon Riggs <simon@2ndquadrant.com> wrote:
This patch splits bgwriter into 2 processes: checkpointer and
bgwriter, seeking to avoid contentious changes. Additional changes are
expected in this release to build upon these changes for both new
processes, though this patch stands on its own as both a performance
vehicle and in some ways a refcatoring to simplify the code.
While you're already splitting up bgwriter, could there be any benefit
to spawning a separate bgwriter process for each tablespace?
If your database has one tablespace on a fast I/O system and another
on a slow one, the slow tablespace would also bog down background
writing for the fast tablespace. But I don't know whether that's
really a problem or not.
Regards,
Marti
On Tue, Sep 20, 2011 at 11:01 AM, Marti Raudsepp <marti@juffo.org> wrote:
On Fri, Sep 16, 2011 at 01:53, Simon Riggs <simon@2ndquadrant.com> wrote:
This patch splits bgwriter into 2 processes: checkpointer and
bgwriter, seeking to avoid contentious changes. Additional changes are
expected in this release to build upon these changes for both new
processes, though this patch stands on its own as both a performance
vehicle and in some ways a refcatoring to simplify the code.While you're already splitting up bgwriter, could there be any benefit
to spawning a separate bgwriter process for each tablespace?If your database has one tablespace on a fast I/O system and another
on a slow one, the slow tablespace would also bog down background
writing for the fast tablespace. But I don't know whether that's
really a problem or not.
I doubt it. Most of the time the writes are going to be absorbed by
the OS write cache anyway.
I think there's probably more performance to be squeezed out of the
background writer, but maybe not that exact thing, and in any case it
seems like material for a separate patch.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On 09/20/2011 09:35 AM, Heikki Linnakangas wrote:
Yeah, I was thinking of write()s, not fsyncs. I agree this might have
some effect during fsync phase.
Right; that's where the most serious problems seem to pop up at anyway
now. All the testing I did earlier this year suggested Linux at least
is happy to do a granular fsync, and it can also use things like
barriers when appropriate to schedule I/O. The hope here is that the
background writer work to clean ahead of the strategy point is helpful
to backends, and that should keep going even during the sync
phase--which currently doesn't pause for anything else once it's
started. The cleaner writes should all queue up into RAM in a lazy way
rather than block the true I/O, which is being driven by sync calls.
There is some risk here that the cleaner writes happen faster than the
true rate at which backends really need buffers, since it has a
predictive component it can be wrong about. Those could in theory
result in the write cache filling faster than it would in the current
environment, such that writes truly block that would have been cached in
the current code. If you're that close to the edge though, backends
should really benefit from the cleaner--that same write done by a client
would turn into a serious stall. From that perspective, when things
have completely filled the write cache, any writes the cleaner can get
out of the way in advance of when a backend needs it should be the
biggest win most of the time.
--
Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us
On Fri, Sep 16, 2011 at 2:38 AM, Fujii Masao <masao.fujii@gmail.com> wrote:
On Fri, Sep 16, 2011 at 7:53 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
This patch splits bgwriter into 2 processes: checkpointer and
bgwriter, seeking to avoid contentious changes. Additional changes are
expected in this release to build upon these changes for both new
processes, though this patch stands on its own as both a performance
vehicle and in some ways a refcatoring to simplify the code.I like this idea to simplify the code. How much performance gain can we
expect by this patch?Current patch has a bug at shutdown I've not located yet, but seems
likely is a simple error. That is mainly because for personal reasons
I've not been able to work on the patch recently. I expect to be able
to fix that later in the CF.You seem to have forgotten to include checkpointor.c and .h in the patch.
Original patch included here.
--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Attachments:
bgwriter_split.v1a.patchapplication/octet-stream; name=bgwriter_split.v1a.patchDownload+1347-985
On Thu, Sep 15, 2011 at 11:53 PM, Simon Riggs <simon@2ndquadrant.com> wrote:
Current patch has a bug at shutdown I've not located yet, but seems
likely is a simple error. That is mainly because for personal reasons
I've not been able to work on the patch recently. I expect to be able
to fix that later in the CF.
Full patch, with bug fixed. (v2)
I'm now free to take review comments and make changes.
--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Attachments:
bgwriter_split.v2.patchapplication/octet-stream; name=bgwriter_split.v2.patchDownload+1347-985
2011/10/2 Simon Riggs <simon@2ndquadrant.com>:
On Thu, Sep 15, 2011 at 11:53 PM, Simon Riggs <simon@2ndquadrant.com> wrote:
Current patch has a bug at shutdown I've not located yet, but seems
likely is a simple error. That is mainly because for personal reasons
I've not been able to work on the patch recently. I expect to be able
to fix that later in the CF.Full patch, with bug fixed. (v2)
I'm now free to take review comments and make changes.
Hi Simon,
I'm trying your patch, it was applied cleanly to master and compiled
ok. But since I started postgres I'm seeing a 99% of CPU usage:
guedes@betelgeuse:/srv/postgres/bgwriter_split$ ps -ef | grep postgres
guedes 14878 1 0 19:37 pts/0 00:00:00
/srv/postgres/bgwriter_split/bin/postgres -D data
guedes 14880 14878 0 19:37 ? 00:00:00 postgres: writer
process
guedes 14881 14878 99 19:37 ? 00:03:07 postgres: checkpointer
process
guedes 14882 14878 0 19:37 ? 00:00:00 postgres: wal writer
process
guedes 14883 14878 0 19:37 ? 00:00:00 postgres: autovacuum
launcher process
guedes 14884 14878 0 19:37 ? 00:00:00 postgres: stats
collector process
Best regards.
--
Dickson S. Guedes
mail/xmpp: guedes@guedesoft.net - skype: guediz
http://guedesoft.net - http://www.postgresql.org.br