rebased background worker reimplementation prototype
Hi,
I've talked a few times about a bgwriter replacement prototype I'd
written a few years back. That happened somewhere deep in another thread
[1]: /messages/by-id/20160204155458.jrw3crmyscusdqf6@alap3.anarazel.de
Tomas Vondra asked me for a link, but there was some considerable bitrot
since. Attached is a rebased and slightly improved version. It's also
available at [2]https://git.postgresql.org/gitweb/?p=users/andresfreund/postgres.git;a=shortlog;h=refs/heads/bgwriter-rewrite[3]https://github.com/anarazel/postgres/tree/bgwriter-rewrite.
The basic observation is that there's some fairly fundamental issues
with the current bgwriter implementation:
1) The pacing logic is complicated, but doesn't work well
2) If most/all buffers have a usagecount, it cannot do anything, because
it doesn't participate in the clock-sweep
3) Backends have to re-discover the now clean buffers.
The prototype is much simpler - in my opinion of course. It has a
ringbuffer of buffers it thinks are clean (which might be reused
concurrently though). It fills that ringbuffer by performing
clock-sweep, and if necessary cleaning, usagecount=pincount=0
buffers. Backends can then pop buffers from that ringbuffer.
Pacing works by bgwriter trying to keep the ringbuffer full, and
backends emptying the ringbuffer. If the ringbuffer is less than 1/4
full, backends wake up bgwriter using the existing latch mechanism.
The ringbuffer is a pretty simplistic lockless (but just obstruction
free, not lock free) implementation, with a lot of unneccessary
constraints.
I've had to improve the current instrumentation for pgwriter
(i.e. pg_stat_bgwriter) considerably - the details in there imo are not
even remotely good enough to actually understand the system (nor are the
names understandable). That needs to be split into a separate commit,
and the half dozen different implementations of the counters need to be
unified.
Obviously this is very prototype-stage code. But I think it's a good
starting point for going forward.
To enable it, one currently has to set the bgwriter_legacy = false GUC.
Some early benchmarks show that in IO heavy cases there's somewhere
between a very mild regression (close to noise), to a pretty
considerable improvement. To see a benefit one - fairly obviously -
needs a workload that is bigger than shared buffers, because otherwise
checkpointer is going to do all writes (and should, it can sort them
perfectly!).
It's quite possible to saturate what a single bgwriter can write out (as
it is before the replacement). I'm inclined to think the next solution
for that is asynchronous IO, and write-combining, rather than multiple
bgwriters.
Here's an example pg_stat_bgwriter from the middle of a pgbench run
(after resetting it a short while before):
┌─[ RECORD 1 ]───────────────┬───────────────────────────────┐
│ checkpoints_timed │ 1 │
│ checkpoints_req │ 0 │
│ checkpoint_write_time │ 179491 │
│ checkpoint_sync_time │ 266 │
│ buffers_written_checkpoint │ 172414 │
│ buffers_written_bgwriter │ 475802 │
│ buffers_written_backend │ 7140 │
│ buffers_written_ring │ 0 │
│ buffers_fsync_checkpointer │ 137 │
│ buffers_fsync_bgwriter │ 0 │
│ buffers_fsync_backend │ 0 │
│ buffers_bgwriter_clean │ 832616 │
│ buffers_alloc_preclean │ 1306572 │
│ buffers_alloc_free │ 0 │
│ buffers_alloc_sweep │ 4639 │
│ buffers_alloc_ring │ 767 │
│ buffers_ticks_bgwriter │ 4398290 │
│ buffers_ticks_backend │ 17098 │
│ maxwritten_clean │ 17 │
│ stats_reset │ 2019-06-10 20:17:56.087704-07 │
└────────────────────────────┴───────────────────────────────┘
Note that buffers_written_backend (as buffers_backend before) accounts
for file extensions too - which bgwriter can't offload. We should
replace that by a non-write (i.e. fallocate) anyway.
Greetings,
Andres Freund
[1]: /messages/by-id/20160204155458.jrw3crmyscusdqf6@alap3.anarazel.de
[2]: https://git.postgresql.org/gitweb/?p=users/andresfreund/postgres.git;a=shortlog;h=refs/heads/bgwriter-rewrite
[3]: https://github.com/anarazel/postgres/tree/bgwriter-rewrite
Attachments:
v7-0001-Basic-obstruction-free-single-producer-multiple-c.patchtext/x-diff; charset=us-asciiDownload
From 53094143e3c1fc9a8090cce66e73e26d58c67b93 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 19 Feb 2016 12:07:51 -0800
Subject: [PATCH v7 1/2] Basic obstruction-free single producer, multiple
consumer ringbuffer.
This is pretty darn limited, supporting only small queues - but could
easily be improved.
---
src/backend/lib/Makefile | 3 +-
src/backend/lib/ringbuf.c | 161 ++++++++++++++++++++++++++++++++++++++
src/include/lib/ringbuf.h | 72 +++++++++++++++++
3 files changed, 235 insertions(+), 1 deletion(-)
create mode 100644 src/backend/lib/ringbuf.c
create mode 100644 src/include/lib/ringbuf.h
diff --git a/src/backend/lib/Makefile b/src/backend/lib/Makefile
index 3c1ee1df83a..b0a63fba309 100644
--- a/src/backend/lib/Makefile
+++ b/src/backend/lib/Makefile
@@ -13,6 +13,7 @@ top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
OBJS = binaryheap.o bipartite_match.o bloomfilter.o dshash.o hyperloglog.o \
- ilist.o integerset.o knapsack.o pairingheap.o rbtree.o stringinfo.o
+ ilist.o integerset.o knapsack.o pairingheap.o rbtree.o ringbuf.o \
+ stringinfo.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/lib/ringbuf.c b/src/backend/lib/ringbuf.c
new file mode 100644
index 00000000000..3de2a4977d8
--- /dev/null
+++ b/src/backend/lib/ringbuf.c
@@ -0,0 +1,161 @@
+/*-------------------------------------------------------------------------
+ *
+ * ringbuf.c
+
+ * Single producer, multiple consumer ringbuffer where consumption is
+ * obstruction-free (i.e. no progress guarantee, but a consumer that is
+ * stopped will not block progress).
+ *
+ * Implemented by essentially using an optimistic lock on the read side.
+ *
+ * XXX: It'd be nice if we could modify this so there's variants for push/pop
+ * that work for different concurrency scenarios. E.g. having spsc_push(),
+ * spmc_push(), ... - that'd avoid having to use different interfaces for
+ * different needs.
+ *
+ * Copyright (c) 2015, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/lib/ringbuf.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/ringbuf.h"
+#include "storage/proc.h"
+
+static inline uint32
+ringbuf_backendid(ringbuf *rb, uint32 pos)
+{
+ return pos & 0xffff0000;
+}
+
+uint32
+ringbuf_elements(ringbuf *rb)
+{
+ uint32 read_off = ringbuf_pos(rb, pg_atomic_read_u32(&rb->read_state));
+ uint32 write_off = ringbuf_pos(rb, rb->write_off);
+
+ /* not wrapped around */
+ if (read_off <= write_off)
+ {
+ return write_off - read_off;
+ }
+
+ /* wrapped around */
+ return (rb->size - read_off) + write_off;
+}
+
+size_t
+ringbuf_size(size_t nelems)
+{
+ Assert(nelems <= 0x0000FFFF);
+ return sizeof(ringbuf) + sizeof(void *) * nelems;
+}
+
+/*
+ * Memory needs to be externally allocated and be at least
+ * ringbuf_size(nelems) large.
+ */
+ringbuf *
+ringbuf_create(void *target, size_t nelems)
+{
+ ringbuf *rb = (ringbuf *) target;
+
+ Assert(nelems <= 0x0000FFFF);
+
+ memset(target, 0, ringbuf_size(nelems));
+
+ rb->size = nelems;
+ pg_atomic_init_u32(&rb->read_state, 0);
+ rb->write_off = 0;
+
+ return rb;
+}
+
+bool
+ringbuf_push(ringbuf *rb, void *data)
+{
+ uint32 read_off = pg_atomic_read_u32(&rb->read_state);
+
+ /*
+ * Check if full - can be outdated, but that's ok. New readers are just
+ * going to further consume elements, never cause the buffer to become
+ * full.
+ */
+ if (ringbuf_pos(rb, read_off)
+ == ringbuf_pos(rb, ringbuf_advance_pos(rb, rb->write_off)))
+ {
+ return false;
+ }
+
+ rb->elements[ringbuf_pos(rb, rb->write_off)] = data;
+
+ /*
+ * The write adding the data needs to be visible before the corresponding
+ * increase of write_off is visible.
+ */
+ pg_write_barrier();
+
+ rb->write_off = ringbuf_advance_pos(rb, rb->write_off);
+
+ return true;
+}
+
+
+bool
+ringbuf_pop(ringbuf *rb, void **data)
+{
+ void *ret;
+ uint32 mybackend = MyProc->backendId;
+
+ Assert((mybackend & 0x0000ffff) == mybackend);
+
+ while (true)
+ {
+ uint32 read_state = pg_atomic_read_u32(&rb->read_state);
+ uint32 read_off = ringbuf_pos(rb, read_state);
+ uint32 old_read_state = read_state;
+
+ /* check if empty - can be outdated, but that's ok */
+ if (read_off == ringbuf_pos(rb, rb->write_off))
+ return false;
+
+ /*
+ * Add our backend id to the position, to detect wrap around.
+ * XXX
+ *
+ * XXX: Skip if the ID already is ours. That's probably likely enough
+ * to warrant the additional branch.
+ */
+ read_state = (read_state & 0x0000ffff) | mybackend << 16;
+
+ /*
+ * Mix the reader position into the current read_off, otherwise
+ * unchanged. If the offset changed since, retry from start.
+ *
+ * NB: This also serves as the read barrier pairing with the write
+ * barrier in ringbuf_push().
+ */
+ if (!pg_atomic_compare_exchange_u32(&rb->read_state, &old_read_state,
+ read_state))
+ continue;
+ old_read_state = read_state; /* with backend id mixed in */
+
+ /* finally read the data */
+ ret = rb->elements[read_off];
+
+ /* compute next offset */
+ read_state = ringbuf_advance_pos(rb, read_state);
+
+ if (pg_atomic_compare_exchange_u32(&rb->read_state, &old_read_state,
+ read_state))
+ break;
+ }
+
+ *data = ret;
+
+ return true;
+}
diff --git a/src/include/lib/ringbuf.h b/src/include/lib/ringbuf.h
new file mode 100644
index 00000000000..3be450bb8f8
--- /dev/null
+++ b/src/include/lib/ringbuf.h
@@ -0,0 +1,72 @@
+/*
+ * ringbuf.h
+ *
+ * Single writer.multiple reader lockless & obstruction free ringbuffer.
+ *
+ * Copyright (c) 2015, PostgreSQL Global Development Group
+ *
+ * src/include/lib/ringbuf.h
+ */
+#ifndef RINGBUF_H
+#define RINGBUF_H
+
+#include "port/atomics.h"
+
+typedef struct ringbuf
+{
+ uint32 size;
+
+ /* 16 bit reader id, 16 bit offset */
+ /* XXX: probably should be on separate cachelines */
+ pg_atomic_uint32 read_state;
+ uint32_t write_off;
+
+ void *elements[FLEXIBLE_ARRAY_MEMBER];
+} ringbuf;
+
+size_t ringbuf_size(size_t nelems);
+
+ringbuf *ringbuf_create(void *target, size_t nelems);
+
+static inline uint32
+ringbuf_pos(ringbuf *rb, uint32 pos)
+{
+ /*
+ * XXX: replacing rb->size with a bitmask op would avoid expensive
+ * divisions. Requiring a pow2 size seems ok.
+ */
+ return (pos & 0x0000ffff) % rb->size;
+}
+
+/*
+ * Compute the new offset, slightly complicated by the fact that we only want
+ * to modify the lower 16 bits.
+ */
+static inline uint32
+ringbuf_advance_pos(ringbuf *rb, uint32 pos)
+{
+ return ((ringbuf_pos(rb, pos) + 1) & 0x0000FFFF) | (pos & 0xFFFF0000);
+}
+
+static inline bool
+ringbuf_empty(ringbuf *rb)
+{
+ uint32 read_state = pg_atomic_read_u32(&rb->read_state);
+
+ return ringbuf_pos(rb, read_state) == ringbuf_pos(rb, rb->write_off);
+}
+
+static inline bool
+ringbuf_full(ringbuf *rb)
+{
+ uint32 read_state = pg_atomic_read_u32(&rb->read_state);
+
+ return ringbuf_pos(rb, read_state) ==
+ ringbuf_pos(rb, ringbuf_advance_pos(rb, rb->write_off));
+}
+
+uint32 ringbuf_elements(ringbuf *rb);
+bool ringbuf_push(ringbuf *rb, void *data);
+bool ringbuf_pop(ringbuf *rb, void **data);
+
+#endif
--
2.22.0.dirty
v7-0002-Rewrite-background-writer.patchtext/x-diff; charset=us-asciiDownload
From 7c5df799aa12dc43f4d8bcef78120225cda990e0 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 19 Feb 2016 12:07:51 -0800
Subject: [PATCH v7 2/2] Rewrite background writer.
This currently consists out of two major parts:
1) Add more statistics, to be able to even evaluate the effects of
bgwriter changes / problems. This should probably be split into a
separate commit.
It's remarkable how odd the set of current measurements is, and how
many different mechanisms for transporting those values we
currently have. The patch adds and replaces a few measurements, but
doesn't yet do enough cleanup (have fewer transport mechanisms,
split into different views).
2) A new bgwriter implementation (that can be turned on by setting the
bgwriter_legacy GUC to false). There's a few major differences:
a) bgwriter performs the clock sweep - that makes it a lot easier
to actually find buffers worthwhile to clean. It's quite
possible to get into situations where the old bgwriter can't do
anything for a while because all buffers have a usagecount > 0.
b) When a buffer is encountered by bgwriter, after performing clock
sweep, is clean and has usage/pin count of 0 (i.e. it can be
reclaimed), then we also push it onto the queue.
c) It just has a ringbuffer of clean buffers, that backends can
drain. Bgwriter pushes (without any locks) entries onto the
queue, backends can pop them of.
d) The pacing logic is a lot simpler. There's a ringbuffer that
bgwriter tries to fill. There's a low watermark that causes
backends to wake up bgwriter.
---
src/backend/access/transam/xlog.c | 2 +
src/backend/catalog/system_views.sql | 25 ++-
src/backend/postmaster/bgwriter.c | 9 +-
src/backend/postmaster/checkpointer.c | 38 ++--
src/backend/postmaster/pgstat.c | 20 ++-
src/backend/storage/buffer/buf_init.c | 22 ++-
src/backend/storage/buffer/bufmgr.c | 198 +++++++++++++++++++--
src/backend/storage/buffer/freelist.c | 240 +++++++++++++++++++-------
src/backend/utils/adt/pgstatfuncs.c | 69 +++++++-
src/backend/utils/misc/guc.c | 13 +-
src/include/catalog/pg_proc.dat | 74 ++++++--
src/include/pgstat.h | 51 +++++-
src/include/postmaster/bgwriter.h | 3 +
src/include/storage/buf_internals.h | 30 +++-
src/include/storage/bufmgr.h | 4 +-
src/test/regress/expected/rules.out | 19 +-
16 files changed, 670 insertions(+), 147 deletions(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 1c7dd51b9f1..78c1d786fa4 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -8376,6 +8376,8 @@ LogCheckpointEnd(bool restartpoint)
BgWriterStats.m_checkpoint_sync_time +=
sync_secs * 1000 + sync_usecs / 1000;
+ BgWriterStats.m_buf_fsync_checkpointer += CheckpointStats.ckpt_sync_rels;
+
/*
* All of the published timing statistics are accounted for. Only
* continue if a log message is to be written.
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 78a103cdb95..d15aed10ad2 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -898,12 +898,27 @@ CREATE VIEW pg_stat_bgwriter AS
pg_stat_get_bgwriter_requested_checkpoints() AS checkpoints_req,
pg_stat_get_checkpoint_write_time() AS checkpoint_write_time,
pg_stat_get_checkpoint_sync_time() AS checkpoint_sync_time,
- pg_stat_get_bgwriter_buf_written_checkpoints() AS buffers_checkpoint,
- pg_stat_get_bgwriter_buf_written_clean() AS buffers_clean,
+
+ pg_stat_get_buf_written_checkpoints() AS buffers_written_checkpoint,
+ pg_stat_get_buf_written_bgwriter() AS buffers_written_bgwriter,
+ pg_stat_get_buf_written_backend() AS buffers_written_backend,
+ pg_stat_get_buf_written_ring() AS buffers_written_ring,
+
+ pg_stat_get_buf_fsync_checkpointer() AS buffers_fsync_checkpointer,
+ pg_stat_get_buf_fsync_bgwriter() AS buffers_fsync_bgwriter,
+ pg_stat_get_buf_fsync_backend() AS buffers_fsync_backend,
+
+ pg_stat_get_buf_bgwriter_clean() AS buffers_bgwriter_clean,
+
+ pg_stat_get_buf_alloc_preclean() AS buffers_alloc_preclean,
+ pg_stat_get_buf_alloc_free() AS buffers_alloc_free,
+ pg_stat_get_buf_alloc_sweep() AS buffers_alloc_sweep,
+ pg_stat_get_buf_alloc_ring() AS buffers_alloc_ring,
+
+ pg_stat_get_buf_ticks_bgwriter() AS buffers_ticks_bgwriter,
+ pg_stat_get_buf_ticks_backend() AS buffers_ticks_backend,
+
pg_stat_get_bgwriter_maxwritten_clean() AS maxwritten_clean,
- pg_stat_get_buf_written_backend() AS buffers_backend,
- pg_stat_get_buf_fsync_backend() AS buffers_backend_fsync,
- pg_stat_get_buf_alloc() AS buffers_alloc,
pg_stat_get_bgwriter_stat_reset_time() AS stats_reset;
CREATE VIEW pg_stat_progress_vacuum AS
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index e6b6c549de5..526304fefc9 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -65,6 +65,7 @@
* GUC parameters
*/
int BgWriterDelay = 200;
+bool BgWriterLegacy = true;
/*
* Multiplier to apply to BgWriterDelay when we decide to hibernate.
@@ -264,7 +265,10 @@ BackgroundWriterMain(void)
/*
* Do one cycle of dirty-buffer writing.
*/
- can_hibernate = BgBufferSync(&wb_context);
+ if (BgWriterLegacy)
+ can_hibernate = BgBufferSyncLegacy(&wb_context);
+ else
+ can_hibernate = BgBufferSyncNew(&wb_context);
/*
* Send off activity statistics to the stats collector
@@ -366,7 +370,8 @@ BackgroundWriterMain(void)
BgWriterDelay * HIBERNATE_FACTOR,
WAIT_EVENT_BGWRITER_HIBERNATE);
/* Reset the notification request in case we timed out */
- StrategyNotifyBgWriter(-1);
+ if (BgWriterLegacy)
+ StrategyNotifyBgWriter(-1);
}
prev_hibernate = can_hibernate;
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index 13f152b4731..e5ecca1e3db 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -102,7 +102,7 @@
* The requests array holds fsync requests sent by backends and not yet
* absorbed by the checkpointer.
*
- * Unlike the checkpoint fields, num_backend_writes, num_backend_fsync, and
+ * Unlike the checkpoint fields, num_written_*, num_fsync_*, and
* the requests fields are protected by CheckpointerCommLock.
*----------
*/
@@ -127,8 +127,11 @@ typedef struct
ConditionVariable start_cv; /* signaled when ckpt_started advances */
ConditionVariable done_cv; /* signaled when ckpt_done advances */
- uint32 num_backend_writes; /* counts user backend buffer writes */
- uint32 num_backend_fsync; /* counts user backend fsync calls */
+ uint32 num_written_backend; /* counts user backend buffer writes */
+ uint32 num_written_ring; /* counts ring buffer writes */
+
+ uint32 num_fsync_bgwriter; /* counts bgwriter fsync calls */
+ uint32 num_fsync_backend; /* counts user backend fsync calls */
int num_requests; /* current # of requests */
int max_requests; /* allocated array size */
@@ -1119,7 +1122,7 @@ ForwardSyncRequest(const FileTag *ftag, SyncRequestType type)
/* Count all backend writes regardless of if they fit in the queue */
if (!AmBackgroundWriterProcess())
- CheckpointerShmem->num_backend_writes++;
+ CheckpointerShmem->num_written_backend++;
/*
* If the checkpointer isn't running or the request queue is full, the
@@ -1134,8 +1137,10 @@ ForwardSyncRequest(const FileTag *ftag, SyncRequestType type)
* Count the subset of writes where backends have to do their own
* fsync
*/
- if (!AmBackgroundWriterProcess())
- CheckpointerShmem->num_backend_fsync++;
+ if (AmBackgroundWriterProcess())
+ CheckpointerShmem->num_fsync_backend++;
+ else
+ CheckpointerShmem->num_fsync_bgwriter++;
LWLockRelease(CheckpointerCommLock);
return false;
}
@@ -1295,11 +1300,15 @@ AbsorbSyncRequests(void)
LWLockAcquire(CheckpointerCommLock, LW_EXCLUSIVE);
/* Transfer stats counts into pending pgstats message */
- BgWriterStats.m_buf_written_backend += CheckpointerShmem->num_backend_writes;
- BgWriterStats.m_buf_fsync_backend += CheckpointerShmem->num_backend_fsync;
+ BgWriterStats.m_buf_written_backend += CheckpointerShmem->num_written_backend;
+ BgWriterStats.m_buf_written_ring += CheckpointerShmem->num_written_ring;
+ BgWriterStats.m_buf_fsync_backend += CheckpointerShmem->num_fsync_backend;
+ BgWriterStats.m_buf_fsync_bgwriter += CheckpointerShmem->num_fsync_bgwriter;
- CheckpointerShmem->num_backend_writes = 0;
- CheckpointerShmem->num_backend_fsync = 0;
+ CheckpointerShmem->num_written_backend = 0;
+ CheckpointerShmem->num_written_ring = 0;
+ CheckpointerShmem->num_fsync_backend = 0;
+ CheckpointerShmem->num_fsync_bgwriter = 0;
/*
* We try to avoid holding the lock for a long time by copying the request
@@ -1373,3 +1382,12 @@ FirstCallSinceLastCheckpoint(void)
return FirstCall;
}
+
+// FIXME: crappy API
+void
+ReportRingWrite(void)
+{
+ LWLockAcquire(CheckpointerCommLock, LW_EXCLUSIVE);
+ CheckpointerShmem->num_written_ring++;
+ LWLockRelease(CheckpointerCommLock);
+}
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index b4f2b28b517..9aa7b9b8139 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -6313,12 +6313,26 @@ pgstat_recv_bgwriter(PgStat_MsgBgWriter *msg, int len)
globalStats.requested_checkpoints += msg->m_requested_checkpoints;
globalStats.checkpoint_write_time += msg->m_checkpoint_write_time;
globalStats.checkpoint_sync_time += msg->m_checkpoint_sync_time;
+
globalStats.buf_written_checkpoints += msg->m_buf_written_checkpoints;
- globalStats.buf_written_clean += msg->m_buf_written_clean;
- globalStats.maxwritten_clean += msg->m_maxwritten_clean;
+ globalStats.buf_written_bgwriter += msg->m_buf_written_bgwriter;
globalStats.buf_written_backend += msg->m_buf_written_backend;
+ globalStats.buf_written_ring += msg->m_buf_written_ring;
+
+ globalStats.buf_fsync_checkpointer += msg->m_buf_fsync_checkpointer;
+ globalStats.buf_fsync_bgwriter += msg->m_buf_fsync_bgwriter;
globalStats.buf_fsync_backend += msg->m_buf_fsync_backend;
- globalStats.buf_alloc += msg->m_buf_alloc;
+
+ globalStats.buf_alloc_preclean += msg->m_buf_alloc_preclean;
+ globalStats.buf_alloc_free += msg->m_buf_alloc_free;
+ globalStats.buf_alloc_sweep += msg->m_buf_alloc_sweep;
+ globalStats.buf_alloc_ring += msg->m_buf_alloc_ring;
+
+ globalStats.buf_ticks_bgwriter += msg->m_buf_ticks_bgwriter;
+ globalStats.buf_ticks_backend += msg->m_buf_ticks_backend;
+
+ globalStats.buf_clean_bgwriter += msg->m_buf_clean_bgwriter;
+ globalStats.maxwritten_clean += msg->m_maxwritten_clean;
}
/* ----------
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index ccd2c31c0b3..6154f75714f 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,6 +14,7 @@
*/
#include "postgres.h"
+#include "lib/ringbuf.h"
#include "storage/bufmgr.h"
#include "storage/buf_internals.h"
@@ -23,6 +24,7 @@ char *BufferBlocks;
LWLockMinimallyPadded *BufferIOLWLockArray = NULL;
WritebackContext BackendWritebackContext;
CkptSortItem *CkptBufferIds;
+ringbuf *VictimBuffers = NULL;
/*
@@ -70,7 +72,8 @@ InitBufferPool(void)
bool foundBufs,
foundDescs,
foundIOLocks,
- foundBufCkpt;
+ foundBufCkpt,
+ foundFreeBufs;
/* Align descriptors to a cacheline boundary. */
BufferDescriptors = (BufferDescPadded *)
@@ -91,6 +94,10 @@ InitBufferPool(void)
LWLockRegisterTranche(LWTRANCHE_BUFFER_IO_IN_PROGRESS, "buffer_io");
LWLockRegisterTranche(LWTRANCHE_BUFFER_CONTENT, "buffer_content");
+ VictimBuffers = ShmemInitStruct("Free Buffers",
+ ringbuf_size(VICTIM_BUFFER_PRECLEAN_SIZE),
+ &foundFreeBufs);
+
/*
* The array used to sort to-be-checkpointed buffer ids is located in
* shared memory, to avoid having to allocate significant amounts of
@@ -102,10 +109,11 @@ InitBufferPool(void)
ShmemInitStruct("Checkpoint BufferIds",
NBuffers * sizeof(CkptSortItem), &foundBufCkpt);
- if (foundDescs || foundBufs || foundIOLocks || foundBufCkpt)
+ if (foundDescs || foundBufs || foundIOLocks || foundBufCkpt || foundFreeBufs)
{
/* should find all of these, or none of them */
- Assert(foundDescs && foundBufs && foundIOLocks && foundBufCkpt);
+ Assert(foundDescs && foundBufs && foundIOLocks && foundBufCkpt && foundFreeBufs);
+
/* note: this path is only taken in EXEC_BACKEND case */
}
else
@@ -129,6 +137,7 @@ InitBufferPool(void)
/*
* Initially link all the buffers together as unused. Subsequent
* management of this list is done by freelist.c.
+ * FIXME: remove once legacy bgwriter is removed
*/
buf->freeNext = i + 1;
@@ -139,8 +148,10 @@ InitBufferPool(void)
LWTRANCHE_BUFFER_IO_IN_PROGRESS);
}
- /* Correct last entry of linked list */
+ /* Correct last entry of linked list: FIXME: remove */
GetBufferDescriptor(NBuffers - 1)->freeNext = FREENEXT_END_OF_LIST;
+ /* FIXME: could fill the first few free buffers? */
+ VictimBuffers = ringbuf_create(VictimBuffers, VICTIM_BUFFER_PRECLEAN_SIZE);
}
/* Init other shared buffer-management stuff */
@@ -189,5 +200,8 @@ BufferShmemSize(void)
/* size of checkpoint sort array in bufmgr.c */
size = add_size(size, mul_size(NBuffers, sizeof(CkptSortItem)));
+ /* FIXME: better ringbuffer size */
+ size = add_size(size, ringbuf_size(VICTIM_BUFFER_PRECLEAN_SIZE));
+
return size;
}
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 7332e6b5903..9d63244ba08 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -39,6 +39,7 @@
#include "catalog/storage.h"
#include "executor/instrument.h"
#include "lib/binaryheap.h"
+#include "lib/ringbuf.h"
#include "miscadmin.h"
#include "pg_trace.h"
#include "pgstat.h"
@@ -101,7 +102,7 @@ typedef struct CkptTsStatus
/* already processed pages in this tablespace */
int num_scanned;
- /* current offset in CkptBufferIds for this tablespace */
+ /* currentCheckpointerShmem->num_written_ring offset in CkptBufferIds for this tablespace */
int index;
} CkptTsStatus;
@@ -866,11 +867,29 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
if (isExtend)
{
+ instr_time io_start,
+ io_time;
+
+ if (track_io_timing)
+ INSTR_TIME_SET_CURRENT(io_start);
+
/* new buffers are zero-filled */
MemSet((char *) bufBlock, 0, BLCKSZ);
+
+ if (track_io_timing)
+ INSTR_TIME_SET_CURRENT(io_start);
+
/* don't set checksum for all-zero page */
smgrextend(smgr, forkNum, blockNum, (char *) bufBlock, false);
+ if (track_io_timing)
+ {
+ INSTR_TIME_SET_CURRENT(io_time);
+ INSTR_TIME_SUBTRACT(io_time, io_start);
+ pgstat_count_buffer_write_time(INSTR_TIME_GET_MICROSEC(io_time));
+ INSTR_TIME_ADD(pgBufferUsage.blk_write_time, io_time);
+ }
+
/*
* NB: we're *not* doing a ScheduleBufferTagForWriteback here;
* although we're essentially performing a write. At least on linux
@@ -1136,6 +1155,9 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
UnpinBuffer(buf, true);
continue;
}
+
+ // FIXME: crappy API
+ StrategyReportWrite(strategy, buf);
}
/* OK, do the I/O */
@@ -1352,6 +1374,8 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* trying to write it out. We have to let them finish before we can
* reclaim the buffer.
*
+ * FIXME: ^^^
+ *
* The buffer could get reclaimed by someone else while we are waiting
* to acquire the necessary locks; if so, don't mess it up.
*/
@@ -2038,7 +2062,119 @@ BufferSync(int flags)
}
/*
- * BgBufferSync -- Write out some dirty buffers in the pool.
+ * BgBufferSyncNew -- Write out some dirty buffers in the pool.
+ *
+ * This is called periodically by the background writer process.
+ *
+ * Returns true if it's appropriate for the bgwriter process to go into
+ * low-power hibernation mode.
+ */
+bool
+BgBufferSyncNew(WritebackContext *wb_context)
+{
+ uint32 recent_alloc_preclean;
+ uint32 recent_alloc_free;
+ uint32 recent_alloc_sweep;
+ uint32 recent_alloc_ring;
+ uint32 strategy_passes;
+ uint64 nticks;
+ uint64 nticks_sum = 0;
+
+ /* Make sure we can handle the pin inside SyncOneBuffer */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
+ /* Know where to start, and report buffer alloc counts to pgstat */
+ StrategySyncStart(&strategy_passes,
+ &recent_alloc_preclean,
+ &recent_alloc_free,
+ &recent_alloc_sweep,
+ &recent_alloc_ring,
+ &nticks);
+
+ /* Report buffer alloc counts to pgstat */
+ BgWriterStats.m_buf_alloc_preclean += recent_alloc_preclean;
+ BgWriterStats.m_buf_alloc_free += recent_alloc_free;
+ BgWriterStats.m_buf_alloc_sweep += recent_alloc_sweep;
+ BgWriterStats.m_buf_alloc_ring += recent_alloc_ring;
+ BgWriterStats.m_buf_ticks_backend += nticks;
+
+ /* go and populate freelist */
+ while (!ringbuf_full(VictimBuffers))
+ {
+ BufferDesc *bufHdr;
+ bool pushed;
+ bool dirty;
+ uint32 buf_state;
+
+ ReservePrivateRefCountEntry();
+
+ bufHdr = ClockSweep(NULL, &buf_state, &nticks);
+ nticks_sum += nticks;
+
+ dirty = buf_state & BM_DIRTY;
+
+ if (dirty)
+ {
+ SMgrRelation reln;
+ BufferTag tag;
+ LWLock *content_lock;
+
+
+ /*
+ * Pin it, share-lock it, write it. (FlushBuffer will do nothing if the
+ * buffer is clean by the time we've locked it.)
+ */
+ PinBuffer_Locked(bufHdr);
+
+ /* open relation before locking the page */
+ reln = smgropen(bufHdr->tag.rnode, InvalidBackendId);
+
+ content_lock = BufferDescriptorGetContentLock(bufHdr);
+
+ LWLockAcquire(content_lock, LW_SHARED);
+ FlushBuffer(bufHdr, reln);
+ LWLockRelease(content_lock);
+
+ /* copy tag before releasing pin */
+ tag = bufHdr->tag;
+
+ UnpinBuffer(bufHdr, true);
+
+ pushed = ringbuf_push(VictimBuffers, bufHdr);
+
+ Assert(wb_context);
+ ScheduleBufferTagForWriteback(wb_context, &tag);
+
+ BgWriterStats.m_buf_written_bgwriter++;
+ }
+ else
+ {
+ UnlockBufHdr(bufHdr, buf_state);
+ pushed = ringbuf_push(VictimBuffers, bufHdr);
+
+ BgWriterStats.m_buf_clean_bgwriter++;
+ }
+
+ /* full, shouldn't normally happen, we're the only writer */
+ if (!pushed)
+ break;
+
+ /* so we occasionally sleep, even if continually busy */
+ if (BgWriterStats.m_buf_written_bgwriter >= bgwriter_lru_maxpages)
+ {
+ BgWriterStats.m_maxwritten_clean++;
+ break;
+ }
+ }
+
+ BgWriterStats.m_buf_ticks_bgwriter += nticks_sum;
+
+ return BgWriterStats.m_buf_written_bgwriter == 0 &&
+ BgWriterStats.m_buf_clean_bgwriter == 0;
+}
+
+/*
+ * BgBufferSyncLegacy -- Write out some dirty buffers in the pool.
*
* This is called periodically by the background writer process.
*
@@ -2049,12 +2185,16 @@ BufferSync(int flags)
* bgwriter_lru_maxpages to 0.)
*/
bool
-BgBufferSync(WritebackContext *wb_context)
+BgBufferSyncLegacy(WritebackContext *wb_context)
{
/* info obtained from freelist.c */
int strategy_buf_id;
uint32 strategy_passes;
- uint32 recent_alloc;
+ uint32 recent_alloc_preclean;
+ uint32 recent_alloc_free;
+ uint32 recent_alloc_sweep;
+ uint32 recent_alloc_ring;
+ uint64 recent_ticks;
/*
* Information saved between calls so we can determine the strategy
@@ -2090,16 +2230,25 @@ BgBufferSync(WritebackContext *wb_context)
/* Variables for final smoothed_density update */
long new_strategy_delta;
- uint32 new_recent_alloc;
+ uint32 new_recent_alloc_sweep;
/*
* Find out where the freelist clock sweep currently is, and how many
* buffer allocations have happened since our last call.
*/
- strategy_buf_id = StrategySyncStart(&strategy_passes, &recent_alloc);
+ strategy_buf_id = StrategySyncStart(&strategy_passes,
+ &recent_alloc_preclean,
+ &recent_alloc_free,
+ &recent_alloc_sweep,
+ &recent_alloc_ring,
+ &recent_ticks);
/* Report buffer alloc counts to pgstat */
- BgWriterStats.m_buf_alloc += recent_alloc;
+ BgWriterStats.m_buf_alloc_preclean += recent_alloc_preclean;
+ BgWriterStats.m_buf_alloc_free += recent_alloc_free;
+ BgWriterStats.m_buf_alloc_sweep += recent_alloc_sweep;
+ BgWriterStats.m_buf_alloc_ring += recent_alloc_ring;
+ BgWriterStats.m_buf_ticks_backend += recent_ticks;
/*
* If we're not running the LRU scan, just stop after doing the stats
@@ -2196,9 +2345,9 @@ BgBufferSync(WritebackContext *wb_context)
*
* If the strategy point didn't move, we don't update the density estimate
*/
- if (strategy_delta > 0 && recent_alloc > 0)
+ if (strategy_delta > 0 && recent_alloc_sweep > 0)
{
- scans_per_alloc = (float) strategy_delta / (float) recent_alloc;
+ scans_per_alloc = (float) strategy_delta / (float) recent_alloc_sweep;
smoothed_density += (scans_per_alloc - smoothed_density) /
smoothing_samples;
}
@@ -2216,10 +2365,10 @@ BgBufferSync(WritebackContext *wb_context)
* a true average we want a fast-attack, slow-decline behavior: we
* immediately follow any increase.
*/
- if (smoothed_alloc <= (float) recent_alloc)
- smoothed_alloc = recent_alloc;
+ if (smoothed_alloc <= (float) recent_alloc_sweep)
+ smoothed_alloc = recent_alloc_sweep;
else
- smoothed_alloc += ((float) recent_alloc - smoothed_alloc) /
+ smoothed_alloc += ((float) recent_alloc_sweep - smoothed_alloc) /
smoothing_samples;
/* Scale the estimate by a GUC to allow more aggressive tuning. */
@@ -2297,7 +2446,7 @@ BgBufferSync(WritebackContext *wb_context)
reusable_buffers++;
}
- BgWriterStats.m_buf_written_clean += num_written;
+ BgWriterStats.m_buf_written_bgwriter += num_written;
#ifdef BGW_DEBUG
elog(DEBUG1, "bgwriter: recent_alloc=%u smoothed=%.2f delta=%ld ahead=%d density=%.2f reusable_est=%d upcoming_est=%d scanned=%d wrote=%d reusable=%d",
@@ -2317,22 +2466,22 @@ BgBufferSync(WritebackContext *wb_context)
* density estimates.
*/
new_strategy_delta = bufs_to_lap - num_to_scan;
- new_recent_alloc = reusable_buffers - reusable_buffers_est;
- if (new_strategy_delta > 0 && new_recent_alloc > 0)
+ new_recent_alloc_sweep = reusable_buffers - reusable_buffers_est;
+ if (new_strategy_delta > 0 && new_recent_alloc_sweep > 0)
{
- scans_per_alloc = (float) new_strategy_delta / (float) new_recent_alloc;
+ scans_per_alloc = (float) new_strategy_delta / (float) new_recent_alloc_sweep;
smoothed_density += (scans_per_alloc - smoothed_density) /
smoothing_samples;
#ifdef BGW_DEBUG
elog(DEBUG2, "bgwriter: cleaner density alloc=%u scan=%ld density=%.2f new smoothed=%.2f",
- new_recent_alloc, new_strategy_delta,
+ new_recent_alloc_sweep, new_strategy_delta,
scans_per_alloc, smoothed_density);
#endif
}
/* Return true if OK to hibernate */
- return (bufs_to_lap == 0 && recent_alloc == 0);
+ return (bufs_to_lap == 0 && new_recent_alloc_sweep == 0);
}
/*
@@ -4321,6 +4470,8 @@ void
IssuePendingWritebacks(WritebackContext *context)
{
int i;
+ instr_time io_start,
+ io_time;
if (context->nr_pending == 0)
return;
@@ -4332,6 +4483,9 @@ IssuePendingWritebacks(WritebackContext *context)
qsort(&context->pending_writebacks, context->nr_pending,
sizeof(PendingWriteback), buffertag_comparator);
+ if (track_io_timing)
+ INSTR_TIME_SET_CURRENT(io_start);
+
/*
* Coalesce neighbouring writes, but nothing else. For that we iterate
* through the, now sorted, array of pending flushes, and look forward to
@@ -4381,6 +4535,14 @@ IssuePendingWritebacks(WritebackContext *context)
smgrwriteback(reln, tag.forkNum, tag.blockNum, nblocks);
}
+ if (track_io_timing)
+ {
+ INSTR_TIME_SET_CURRENT(io_time);
+ INSTR_TIME_SUBTRACT(io_time, io_start);
+ pgstat_count_buffer_write_time(INSTR_TIME_GET_MICROSEC(io_time));
+ INSTR_TIME_ADD(pgBufferUsage.blk_write_time, io_time);
+ }
+
context->nr_pending = 0;
}
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 06659ab2653..6583f1c3815 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -15,7 +15,9 @@
*/
#include "postgres.h"
+#include "lib/ringbuf.h"
#include "port/atomics.h"
+#include "postmaster/bgwriter.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/proc.h"
@@ -51,7 +53,14 @@ typedef struct
* overflow during a single bgwriter cycle.
*/
uint32 completePasses; /* Complete cycles of the clock sweep */
- pg_atomic_uint32 numBufferAllocs; /* Buffers allocated since last reset */
+
+ /* Buffers allocated since last reset */
+ pg_atomic_uint32 numBufferAllocsPreclean;
+ pg_atomic_uint32 numBufferAllocsFree;
+ pg_atomic_uint32 numBufferAllocsSweep;
+ pg_atomic_uint32 numBufferAllocsRing;
+
+ pg_atomic_uint64 numBufferTicksBackend;
/*
* Bgworker process to be notified upon activity or -1 if none. See
@@ -168,6 +177,62 @@ ClockSweepTick(void)
return victim;
}
+BufferDesc *
+ClockSweep(BufferAccessStrategy strategy, uint32 *buf_state, uint64 *nticks)
+{
+ BufferDesc *buf;
+ int trycounter;
+ uint32 local_buf_state; /* to avoid repeated (de-)referencing */
+ uint64 local_nticks = 0;
+
+ trycounter = NBuffers;
+ for (;;)
+ {
+
+ buf = GetBufferDescriptor(ClockSweepTick());
+ local_nticks++;
+
+ /*
+ * If the buffer is pinned or has a nonzero usage_count, we cannot use
+ * it; decrement the usage_count (unless pinned) and keep scanning.
+ */
+ local_buf_state = LockBufHdr(buf);
+
+ if (BUF_STATE_GET_REFCOUNT(local_buf_state) == 0)
+ {
+ if (BUF_STATE_GET_USAGECOUNT(local_buf_state) != 0)
+ {
+ local_buf_state -= BUF_USAGECOUNT_ONE;
+
+ trycounter = NBuffers;
+ }
+ else
+ {
+ /* Found a usable buffer */
+ if (strategy != NULL)
+ AddBufferToRing(strategy, buf);
+ *buf_state = local_buf_state;
+ *nticks = local_nticks;
+
+ return buf;
+ }
+ }
+ else if (--trycounter == 0)
+ {
+ /*
+ * We've scanned all the buffers without making any state changes,
+ * so all the buffers are pinned (or were when we looked at them).
+ * We could hope that someone will free one eventually, but it's
+ * probably better to fail than to risk getting stuck in an
+ * infinite loop.
+ */
+ UnlockBufHdr(buf, local_buf_state);
+ elog(ERROR, "no unpinned buffers available");
+ }
+ UnlockBufHdr(buf, local_buf_state);
+ }
+}
+
/*
* have_free_buffer -- a lockless check to see if there is a free buffer in
* buffer pool.
@@ -202,8 +267,8 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state)
{
BufferDesc *buf;
int bgwprocno;
- int trycounter;
uint32 local_buf_state; /* to avoid repeated (de-)referencing */
+ uint64 nticks;
/*
* If given a strategy object, see whether it can select a buffer. We
@@ -229,26 +294,22 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state)
* some arbitrary process.
*/
bgwprocno = INT_ACCESS_ONCE(StrategyControl->bgwprocno);
- if (bgwprocno != -1)
+ if (BgWriterLegacy)
{
- /* reset bgwprocno first, before setting the latch */
- StrategyControl->bgwprocno = -1;
+ if (bgwprocno != -1)
+ {
+ /* reset bgwprocno first, before setting the latch */
+ StrategyControl->bgwprocno = -1;
- /*
- * Not acquiring ProcArrayLock here which is slightly icky. It's
- * actually fine because procLatch isn't ever freed, so we just can
- * potentially set the wrong process' (or no process') latch.
- */
- SetLatch(&ProcGlobal->allProcs[bgwprocno].procLatch);
+ /*
+ * Not acquiring ProcArrayLock here which is slightly icky. It's
+ * actually fine because procLatch isn't ever freed, so we just can
+ * potentially set the wrong process' (or no process') latch.
+ */
+ SetLatch(&ProcGlobal->allProcs[bgwprocno].procLatch);
+ }
}
- /*
- * We count buffer allocation requests so that the bgwriter can estimate
- * the rate of buffer consumption. Note that buffers recycled by a
- * strategy object are intentionally not counted here.
- */
- pg_atomic_fetch_add_u32(&StrategyControl->numBufferAllocs, 1);
-
/*
* First check, without acquiring the lock, whether there's buffers in the
* freelist. Since we otherwise don't require the spinlock in every
@@ -302,6 +363,9 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state)
if (BUF_STATE_GET_REFCOUNT(local_buf_state) == 0
&& BUF_STATE_GET_USAGECOUNT(local_buf_state) == 0)
{
+ // FIXME: possible to do outside of lock?
+ pg_atomic_fetch_add_u32(&StrategyControl->numBufferAllocsFree, 1);
+
if (strategy != NULL)
AddBufferToRing(strategy, buf);
*buf_state = local_buf_state;
@@ -312,51 +376,81 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state)
}
}
- /* Nothing on the freelist, so run the "clock sweep" algorithm */
- trycounter = NBuffers;
- for (;;)
+ if (!BgWriterLegacy)
{
- buf = GetBufferDescriptor(ClockSweepTick());
+ int i = 0;
/*
- * If the buffer is pinned or has a nonzero usage_count, we cannot use
- * it; decrement the usage_count (unless pinned) and keep scanning.
+ * Try to get a buffer from the clean buffer list.
*/
- local_buf_state = LockBufHdr(buf);
-
- if (BUF_STATE_GET_REFCOUNT(local_buf_state) == 0)
+ while (!ringbuf_empty(VictimBuffers))
{
- if (BUF_STATE_GET_USAGECOUNT(local_buf_state) != 0)
- {
- local_buf_state -= BUF_USAGECOUNT_ONE;
+ BufferDesc *buf;
+ bool found;
+ uint32 elements;
- trycounter = NBuffers;
+ found = ringbuf_pop(VictimBuffers, (void *)&buf);
+
+ /* If the ringbuffer is sufficiently depleted, wakeup the bgwriter. */
+ if (bgwprocno != -1 &&
+ (!found ||
+ (elements = ringbuf_elements(VictimBuffers)) < VICTIM_BUFFER_PRECLEAN_SIZE / 4))
+ {
+#if 0
+ if (!found)
+ elog(LOG, "signalling bgwriter: empty");
+ else
+ elog(LOG, "signalling bgwriter: watermark: %u %u/%u",
+ elements, VICTIM_BUFFER_PRECLEAN_SIZE / 4, VICTIM_BUFFER_PRECLEAN_SIZE);
+#endif
+ SetLatch(&ProcGlobal->allProcs[bgwprocno].procLatch);
+ }
+
+ if (!found)
+ break;
+
+ /* check if the buffer is still unused, done if so */
+ local_buf_state = LockBufHdr(buf);
+ if (BUF_STATE_GET_REFCOUNT(local_buf_state) == 0
+ && BUF_STATE_GET_USAGECOUNT(local_buf_state) == 0)
+ {
+ // FIXME: possible to do outside of lock?
+ pg_atomic_fetch_add_u32(&StrategyControl->numBufferAllocsPreclean, 1);
+
+ if (strategy != NULL)
+ AddBufferToRing(strategy, buf);
+ *buf_state = local_buf_state;
+ return buf;
}
else
{
- /* Found a usable buffer */
- if (strategy != NULL)
- AddBufferToRing(strategy, buf);
- *buf_state = local_buf_state;
- return buf;
+ UnlockBufHdr(buf, local_buf_state);
+ //ereport(LOG, (errmsg("buffer %u since reused (hand at %u)",
+ // buf->buf_id,
+ // pg_atomic_read_u32(&StrategyControl->nextVictimBuffer) % NBuffers),
+ // errhidestmt(true)));
}
+
+ i++;
}
- else if (--trycounter == 0)
- {
- /*
- * We've scanned all the buffers without making any state changes,
- * so all the buffers are pinned (or were when we looked at them).
- * We could hope that someone will free one eventually, but it's
- * probably better to fail than to risk getting stuck in an
- * infinite loop.
- */
- UnlockBufHdr(buf, local_buf_state);
- elog(ERROR, "no unpinned buffers available");
- }
- UnlockBufHdr(buf, local_buf_state);
+
+#if 0
+ ereport(LOG, (errmsg("ringbuf empty after %u cycles", i),
+ errhidestmt(true)));
+#endif
+
}
+
+ /* Nothing on the freelist, so run the "clock sweep" algorithm */
+ buf = ClockSweep(strategy, buf_state, &nticks);
+
+ pg_atomic_fetch_add_u32(&StrategyControl->numBufferAllocsSweep, 1);
+ pg_atomic_fetch_add_u64(&StrategyControl->numBufferTicksBackend, nticks);
+
+ return buf;
}
+
/*
* StrategyFreeBuffer: put a buffer on the freelist
*/
@@ -381,18 +475,22 @@ StrategyFreeBuffer(BufferDesc *buf)
}
/*
- * StrategySyncStart -- tell BufferSync where to start syncing
+ * StrategySyncStart -- tell BgBufferSync where to start syncing
*
- * The result is the buffer index of the best buffer to sync first.
- * BufferSync() will proceed circularly around the buffer array from there.
+ * The result is the buffer index below the current clock-hand. BgBufferSync()
+ * will proceed circularly around the buffer array from there.
*
- * In addition, we return the completed-pass count (which is effectively
- * the higher-order bits of nextVictimBuffer) and the count of recent buffer
- * allocs if non-NULL pointers are passed. The alloc count is reset after
- * being read.
+ * In addition, we return the completed-pass count (which is effectively the
+ * higher-order bits of nextVictimBuffer) and the counts of recent buffer
+ * allocations. The allocation counts are reset after being read.
*/
int
-StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc)
+StrategySyncStart(uint32 *complete_passes,
+ uint32 *alloc_preclean,
+ uint32 *alloc_free,
+ uint32 *alloc_sweep,
+ uint32 *alloc_ring,
+ uint64 *ticks_backend)
{
uint32 nextVictimBuffer;
int result;
@@ -410,13 +508,16 @@ StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc)
* completePasses could be incremented. C.f. ClockSweepTick().
*/
*complete_passes += nextVictimBuffer / NBuffers;
- }
- if (num_buf_alloc)
- {
- *num_buf_alloc = pg_atomic_exchange_u32(&StrategyControl->numBufferAllocs, 0);
}
SpinLockRelease(&StrategyControl->buffer_strategy_lock);
+
+ *alloc_preclean = pg_atomic_exchange_u32(&StrategyControl->numBufferAllocsPreclean, 0);
+ *alloc_free = pg_atomic_exchange_u32(&StrategyControl->numBufferAllocsFree, 0);
+ *alloc_sweep = pg_atomic_exchange_u32(&StrategyControl->numBufferAllocsSweep, 0);
+ *alloc_ring = pg_atomic_exchange_u32(&StrategyControl->numBufferAllocsRing, 0);
+ *ticks_backend = pg_atomic_exchange_u64(&StrategyControl->numBufferTicksBackend, 0);
+
return result;
}
@@ -517,7 +618,11 @@ StrategyInitialize(bool init)
/* Clear statistics */
StrategyControl->completePasses = 0;
- pg_atomic_init_u32(&StrategyControl->numBufferAllocs, 0);
+ pg_atomic_init_u32(&StrategyControl->numBufferAllocsPreclean, 0);
+ pg_atomic_init_u32(&StrategyControl->numBufferAllocsFree, 0);
+ pg_atomic_init_u32(&StrategyControl->numBufferAllocsSweep, 0);
+ pg_atomic_init_u32(&StrategyControl->numBufferAllocsRing, 0);
+ pg_atomic_init_u64(&StrategyControl->numBufferTicksBackend, 0);
/* No pending notification */
StrategyControl->bgwprocno = -1;
@@ -645,6 +750,9 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state)
if (BUF_STATE_GET_REFCOUNT(local_buf_state) == 0
&& BUF_STATE_GET_USAGECOUNT(local_buf_state) <= 1)
{
+ // FIXME: possible to do outside of lock?
+ pg_atomic_fetch_add_u32(&StrategyControl->numBufferAllocsRing, 1);
+
strategy->current_was_in_ring = true;
*buf_state = local_buf_state;
return buf;
@@ -702,3 +810,11 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf)
return true;
}
+
+void
+StrategyReportWrite(BufferAccessStrategy strategy,
+ BufferDesc *buf)
+{
+ if (strategy->current_was_in_ring)
+ ReportRingWrite();
+}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 05240bfd142..d0d163ea35a 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1604,15 +1604,45 @@ pg_stat_get_bgwriter_requested_checkpoints(PG_FUNCTION_ARGS)
}
Datum
-pg_stat_get_bgwriter_buf_written_checkpoints(PG_FUNCTION_ARGS)
+pg_stat_get_buf_written_checkpoints(PG_FUNCTION_ARGS)
{
PG_RETURN_INT64(pgstat_fetch_global()->buf_written_checkpoints);
}
Datum
-pg_stat_get_bgwriter_buf_written_clean(PG_FUNCTION_ARGS)
+pg_stat_get_buf_written_bgwriter(PG_FUNCTION_ARGS)
{
- PG_RETURN_INT64(pgstat_fetch_global()->buf_written_clean);
+ PG_RETURN_INT64(pgstat_fetch_global()->buf_written_bgwriter);
+}
+
+Datum
+pg_stat_get_buf_written_backend(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT64(pgstat_fetch_global()->buf_written_backend);
+}
+
+Datum
+pg_stat_get_buf_written_ring(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT64(pgstat_fetch_global()->buf_written_ring);
+}
+
+Datum
+pg_stat_get_buf_ticks_bgwriter(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT64(pgstat_fetch_global()->buf_ticks_bgwriter);
+}
+
+Datum
+pg_stat_get_buf_ticks_backend(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT64(pgstat_fetch_global()->buf_ticks_backend);
+}
+
+Datum
+pg_stat_get_buf_bgwriter_clean(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT64(pgstat_fetch_global()->buf_clean_bgwriter);
}
Datum
@@ -1641,10 +1671,17 @@ pg_stat_get_bgwriter_stat_reset_time(PG_FUNCTION_ARGS)
PG_RETURN_TIMESTAMPTZ(pgstat_fetch_global()->stat_reset_timestamp);
}
+// FIXME: name
Datum
-pg_stat_get_buf_written_backend(PG_FUNCTION_ARGS)
+pg_stat_get_buf_fsync_checkpointer(PG_FUNCTION_ARGS)
{
- PG_RETURN_INT64(pgstat_fetch_global()->buf_written_backend);
+ PG_RETURN_INT64(pgstat_fetch_global()->buf_fsync_checkpointer);
+}
+
+Datum
+pg_stat_get_buf_fsync_bgwriter(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT64(pgstat_fetch_global()->buf_fsync_backend);
}
Datum
@@ -1654,9 +1691,27 @@ pg_stat_get_buf_fsync_backend(PG_FUNCTION_ARGS)
}
Datum
-pg_stat_get_buf_alloc(PG_FUNCTION_ARGS)
+pg_stat_get_buf_alloc_preclean(PG_FUNCTION_ARGS)
{
- PG_RETURN_INT64(pgstat_fetch_global()->buf_alloc);
+ PG_RETURN_INT64(pgstat_fetch_global()->buf_alloc_preclean);
+}
+
+Datum
+pg_stat_get_buf_alloc_free(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT64(pgstat_fetch_global()->buf_alloc_free);
+}
+
+Datum
+pg_stat_get_buf_alloc_sweep(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT64(pgstat_fetch_global()->buf_alloc_sweep);
+}
+
+Datum
+pg_stat_get_buf_alloc_ring(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT64(pgstat_fetch_global()->buf_alloc_ring);
}
Datum
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 1208eb9a683..425d057a475 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -1434,6 +1434,17 @@ static struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"bgwriter_legacy", PGC_SIGHUP, RESOURCES_BGWRITER,
+ gettext_noop("Use legacy bgwriter algorithm."),
+ NULL,
+ GUC_UNIT_MS
+ },
+ &BgWriterLegacy,
+ true,
+ NULL, NULL, NULL
+ },
+
{
{"trace_notify", PGC_USERSET, DEVELOPER_OPTIONS,
gettext_noop("Generates debugging output for LISTEN and NOTIFY."),
@@ -2734,7 +2745,7 @@ static struct config_int ConfigureNamesInt[] =
GUC_UNIT_MS
},
&BgWriterDelay,
- 200, 10, 10000,
+ 200, 1, 10000,
NULL, NULL, NULL
},
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 87335248a03..464e088c346 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5325,16 +5325,70 @@
proname => 'pg_stat_get_bgwriter_requested_checkpoints', provolatile => 's',
proparallel => 'r', prorettype => 'int8', proargtypes => '',
prosrc => 'pg_stat_get_bgwriter_requested_checkpoints' },
+
{ oid => '2771',
descr => 'statistics: number of buffers written by the bgwriter during checkpoints',
- proname => 'pg_stat_get_bgwriter_buf_written_checkpoints', provolatile => 's',
+ proname => 'pg_stat_get_buf_written_checkpoints', provolatile => 's',
proparallel => 'r', prorettype => 'int8', proargtypes => '',
- prosrc => 'pg_stat_get_bgwriter_buf_written_checkpoints' },
+ prosrc => 'pg_stat_get_buf_written_checkpoints' },
{ oid => '2772',
descr => 'statistics: number of buffers written by the bgwriter for cleaning dirty buffers',
- proname => 'pg_stat_get_bgwriter_buf_written_clean', provolatile => 's',
+ proname => 'pg_stat_get_buf_written_bgwriter', provolatile => 's',
proparallel => 'r', prorettype => 'int8', proargtypes => '',
- prosrc => 'pg_stat_get_bgwriter_buf_written_clean' },
+ prosrc => 'pg_stat_get_buf_written_bgwriter' },
+
+{ oid => '2775',
+ descr => 'statistics: number of buffers written by backends while cleaning dirty buffers',
+ proname => 'pg_stat_get_buf_written_backend', provolatile => 's',
+ proparallel => 'r', prorettype => 'int8', proargtypes => '',
+ prosrc => 'pg_stat_get_buf_written_backend' },
+{ oid => '270',
+ descr => 'statistics: number of buffers written by backends when recycling ring entries',
+ proname => 'pg_stat_get_buf_written_ring', provolatile => 's',
+ proparallel => 'r', prorettype => 'int8', proargtypes => '',
+ prosrc => 'pg_stat_get_buf_written_ring' },
+
+{ oid => '271',
+ descr => 'statistics: number of fsync requests processed by checkpointer',
+ proname => 'pg_stat_get_buf_fsync_checkpointer', provolatile => 's',
+ proparallel => 'r', prorettype => 'int8', proargtypes => '',
+ prosrc => 'pg_stat_get_buf_fsync_checkpointer' },
+{ oid => '272',
+ descr => 'statistics: number of bgwriter buffer writes that did their own fsync',
+ proname => 'pg_stat_get_buf_fsync_bgwriter', provolatile => 's',
+ proparallel => 'r', prorettype => 'int8', proargtypes => '',
+ prosrc => 'pg_stat_get_buf_fsync_bgwriter' },
+{ oid => '3063',
+ descr => 'statistics: number of backend writes that did their own fsync',
+ proname => 'pg_stat_get_buf_fsync_backend', provolatile => 's',
+ proparallel => 'r', prorettype => 'int8', proargtypes => '',
+ prosrc => 'pg_stat_get_buf_fsync_backend' },
+
+{ oid => '273', descr => 'statistics: number of reusable clean buffers discovered by bgwriter',
+ proname => 'pg_stat_get_buf_bgwriter_clean', provolatile => 's', proparallel => 'r',
+ prorettype => 'int8', proargtypes => '', prosrc => 'pg_stat_get_buf_bgwriter_clean' },
+
+{ oid => '380', descr => 'statistics: number of backend buffer allocations via preclean list',
+ proname => 'pg_stat_get_buf_alloc_preclean', provolatile => 's', proparallel => 'r',
+ prorettype => 'int8', proargtypes => '', prosrc => 'pg_stat_get_buf_alloc_preclean' },
+{ oid => '2859', descr => 'statistics: number of backend buffer allocations via backend clock sweep',
+ proname => 'pg_stat_get_buf_alloc_sweep', provolatile => 's', proparallel => 'r',
+ prorettype => 'int8', proargtypes => '', prosrc => 'pg_stat_get_buf_alloc_sweep' },
+{ oid => '381', descr => 'statistics: number of backend buffer allocations via ring buffer',
+ proname => 'pg_stat_get_buf_alloc_ring', provolatile => 's', proparallel => 'r',
+ prorettype => 'int8', proargtypes => '', prosrc => 'pg_stat_get_buf_alloc_ring' },
+{ oid => '421', descr => 'statistics: number of backend buffer allocations via free list',
+ proname => 'pg_stat_get_buf_alloc_free', provolatile => 's', proparallel => 'r',
+ prorettype => 'int8', proargtypes => '', prosrc => 'pg_stat_get_buf_alloc_free' },
+
+{ oid => '560', descr => 'statistics: number of clock sweep ticks by bgwriter',
+ proname => 'pg_stat_get_buf_ticks_bgwriter', provolatile => 's', proparallel => 'r',
+ prorettype => 'int8', proargtypes => '', prosrc => 'pg_stat_get_buf_ticks_bgwriter' },
+{ oid => '561', descr => 'statistics: number of clock sweep ticks by backend',
+ proname => 'pg_stat_get_buf_ticks_backend', provolatile => 's', proparallel => 'r',
+ prorettype => 'int8', proargtypes => '', prosrc => 'pg_stat_get_buf_ticks_backend' },
+
+
{ oid => '2773',
descr => 'statistics: number of times the bgwriter stopped processing when it had written too many buffers while cleaning',
proname => 'pg_stat_get_bgwriter_maxwritten_clean', provolatile => 's',
@@ -5354,18 +5408,6 @@
proname => 'pg_stat_get_checkpoint_sync_time', provolatile => 's',
proparallel => 'r', prorettype => 'float8', proargtypes => '',
prosrc => 'pg_stat_get_checkpoint_sync_time' },
-{ oid => '2775', descr => 'statistics: number of buffers written by backends',
- proname => 'pg_stat_get_buf_written_backend', provolatile => 's',
- proparallel => 'r', prorettype => 'int8', proargtypes => '',
- prosrc => 'pg_stat_get_buf_written_backend' },
-{ oid => '3063',
- descr => 'statistics: number of backend buffer writes that did their own fsync',
- proname => 'pg_stat_get_buf_fsync_backend', provolatile => 's',
- proparallel => 'r', prorettype => 'int8', proargtypes => '',
- prosrc => 'pg_stat_get_buf_fsync_backend' },
-{ oid => '2859', descr => 'statistics: number of buffer allocations',
- proname => 'pg_stat_get_buf_alloc', provolatile => 's', proparallel => 'r',
- prorettype => 'int8', proargtypes => '', prosrc => 'pg_stat_get_buf_alloc' },
{ oid => '2978', descr => 'statistics: number of function calls',
proname => 'pg_stat_get_function_calls', provolatile => 's',
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 0a3ad3a1883..54c4765fb11 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -413,14 +413,30 @@ typedef struct PgStat_MsgBgWriter
PgStat_Counter m_timed_checkpoints;
PgStat_Counter m_requested_checkpoints;
- PgStat_Counter m_buf_written_checkpoints;
- PgStat_Counter m_buf_written_clean;
- PgStat_Counter m_maxwritten_clean;
- PgStat_Counter m_buf_written_backend;
- PgStat_Counter m_buf_fsync_backend;
- PgStat_Counter m_buf_alloc;
PgStat_Counter m_checkpoint_write_time; /* times in milliseconds */
PgStat_Counter m_checkpoint_sync_time;
+
+ PgStat_Counter m_buf_written_checkpoints;
+ PgStat_Counter m_buf_written_bgwriter;
+ PgStat_Counter m_buf_written_backend;
+ PgStat_Counter m_buf_written_ring;
+
+ PgStat_Counter m_buf_fsync_checkpointer;
+ PgStat_Counter m_buf_fsync_bgwriter;
+ PgStat_Counter m_buf_fsync_backend;
+
+ PgStat_Counter m_buf_clean_bgwriter;
+
+ PgStat_Counter m_buf_alloc_preclean;
+ PgStat_Counter m_buf_alloc_free;
+ PgStat_Counter m_buf_alloc_sweep;
+ PgStat_Counter m_buf_alloc_ring;
+
+ PgStat_Counter m_buf_ticks_bgwriter;
+ PgStat_Counter m_buf_ticks_backend;
+
+ PgStat_Counter m_maxwritten_clean;
+
} PgStat_MsgBgWriter;
/* ----------
@@ -699,16 +715,33 @@ typedef struct PgStat_ArchiverStats
typedef struct PgStat_GlobalStats
{
TimestampTz stats_timestamp; /* time of stats file update */
+
PgStat_Counter timed_checkpoints;
PgStat_Counter requested_checkpoints;
PgStat_Counter checkpoint_write_time; /* times in milliseconds */
PgStat_Counter checkpoint_sync_time;
+
PgStat_Counter buf_written_checkpoints;
- PgStat_Counter buf_written_clean;
- PgStat_Counter maxwritten_clean;
+ PgStat_Counter buf_written_bgwriter;
PgStat_Counter buf_written_backend;
+ PgStat_Counter buf_written_ring;
+
+ PgStat_Counter buf_fsync_checkpointer;
+ PgStat_Counter buf_fsync_bgwriter;
PgStat_Counter buf_fsync_backend;
- PgStat_Counter buf_alloc;
+
+ PgStat_Counter buf_clean_bgwriter;
+
+ PgStat_Counter buf_alloc_preclean;
+ PgStat_Counter buf_alloc_free;
+ PgStat_Counter buf_alloc_sweep;
+ PgStat_Counter buf_alloc_ring;
+
+ PgStat_Counter buf_ticks_bgwriter;
+ PgStat_Counter buf_ticks_backend;
+
+ PgStat_Counter maxwritten_clean;
+
TimestampTz stat_reset_timestamp;
} PgStat_GlobalStats;
diff --git a/src/include/postmaster/bgwriter.h b/src/include/postmaster/bgwriter.h
index 630366f49ef..892e24e0832 100644
--- a/src/include/postmaster/bgwriter.h
+++ b/src/include/postmaster/bgwriter.h
@@ -26,6 +26,7 @@ extern int BgWriterDelay;
extern int CheckPointTimeout;
extern int CheckPointWarning;
extern double CheckPointCompletionTarget;
+extern bool BgWriterLegacy;
extern void BackgroundWriterMain(void) pg_attribute_noreturn();
extern void CheckpointerMain(void) pg_attribute_noreturn();
@@ -40,6 +41,8 @@ extern void AbsorbSyncRequests(void);
extern Size CheckpointerShmemSize(void);
extern void CheckpointerShmemInit(void);
+extern void ReportRingWrite(void);
+
extern bool FirstCallSinceLastCheckpoint(void);
#endif /* _BGWRITER_H */
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index df2dda7e7e7..1b58b1db0df 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -142,7 +142,7 @@ typedef struct buftag
* single atomic operation, without actually acquiring and releasing spinlock;
* for instance, increase or decrease refcount. buf_id field never changes
* after initialization, so does not need locking. freeNext is protected by
- * the buffer_strategy_lock not buffer header lock. The LWLock can take care
+ * the buffer_strategy_lock not buffer header lock (XXX: remove). The LWLock can take care
* of itself. The buffer header lock is *not* used to control access to the
* data in the buffer!
*
@@ -184,7 +184,9 @@ typedef struct BufferDesc
pg_atomic_uint32 state;
int wait_backend_pid; /* backend PID of pin-count waiter */
- int freeNext; /* link in freelist chain */
+
+ /* link in freelist chain: only used with legacy bgwriter */
+ int freeNext;
LWLock content_lock; /* to lock access to buffer contents */
} BufferDesc;
@@ -232,10 +234,18 @@ extern PGDLLIMPORT LWLockMinimallyPadded *BufferIOLWLockArray;
/*
* The freeNext field is either the index of the next freelist entry,
* or one of these special values:
+ * XXX: Remove when removing legacy bgwriter
*/
#define FREENEXT_END_OF_LIST (-1)
#define FREENEXT_NOT_IN_LIST (-2)
+/*
+ * FIXME: Probably needs to depend on NBuffers or such.
+ */
+
+/* size of buffer free list */
+#define VICTIM_BUFFER_PRECLEAN_SIZE 4096
+
/*
* Functions for acquiring/releasing a shared buffer header's spinlock. Do
* not apply these to local buffers!
@@ -274,6 +284,7 @@ typedef struct WritebackContext
/* in buf_init.c */
extern PGDLLIMPORT BufferDescPadded *BufferDescriptors;
extern PGDLLIMPORT WritebackContext BackendWritebackContext;
+extern PGDLLIMPORT struct ringbuf *VictimBuffers;
/* in localbuf.c */
extern BufferDesc *LocalBufferDescriptors;
@@ -306,13 +317,24 @@ extern void IssuePendingWritebacks(WritebackContext *context);
extern void ScheduleBufferTagForWriteback(WritebackContext *context, BufferTag *tag);
/* freelist.c */
-extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
+extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategym,
uint32 *buf_state);
+extern BufferDesc *ClockSweep(BufferAccessStrategy strategy,
+ uint32 *buf_state, uint64 *nticks);
+
extern void StrategyFreeBuffer(BufferDesc *buf);
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
BufferDesc *buf);
-extern int StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc);
+extern void StrategyReportWrite(BufferAccessStrategy strategy,
+ BufferDesc *buf);
+
+extern int StrategySyncStart(uint32 *complete_passes,
+ uint32 *alloc_preclean,
+ uint32 *alloc_free,
+ uint32 *alloc_sweep,
+ uint32 *alloc_ring,
+ uint64 *ticks_backend);
extern void StrategyNotifyBgWriter(int bgwprocno);
extern Size StrategyShmemSize(void);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 509f4b7ef1c..9957b9c8c27 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -221,7 +221,9 @@ extern bool HoldingBufferPinThatDelaysRecovery(void);
extern void AbortBufferIO(void);
extern void BufmgrCommit(void);
-extern bool BgBufferSync(struct WritebackContext *wb_context);
+
+extern bool BgBufferSyncNew(struct WritebackContext *wb_context);
+extern bool BgBufferSyncLegacy(struct WritebackContext *wb_context);
extern void AtProcExit_LocalBuffers(void);
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 7d365c48d12..da436d982ab 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1796,12 +1796,21 @@ pg_stat_bgwriter| SELECT pg_stat_get_bgwriter_timed_checkpoints() AS checkpoints
pg_stat_get_bgwriter_requested_checkpoints() AS checkpoints_req,
pg_stat_get_checkpoint_write_time() AS checkpoint_write_time,
pg_stat_get_checkpoint_sync_time() AS checkpoint_sync_time,
- pg_stat_get_bgwriter_buf_written_checkpoints() AS buffers_checkpoint,
- pg_stat_get_bgwriter_buf_written_clean() AS buffers_clean,
+ pg_stat_get_buf_written_checkpoints() AS buffers_written_checkpoint,
+ pg_stat_get_buf_written_bgwriter() AS buffers_written_bgwriter,
+ pg_stat_get_buf_written_backend() AS buffers_written_backend,
+ pg_stat_get_buf_written_ring() AS buffers_written_ring,
+ pg_stat_get_buf_fsync_checkpointer() AS buffers_fsync_checkpointer,
+ pg_stat_get_buf_fsync_bgwriter() AS buffers_fsync_bgwriter,
+ pg_stat_get_buf_fsync_backend() AS buffers_fsync_backend,
+ pg_stat_get_buf_bgwriter_clean() AS buffers_bgwriter_clean,
+ pg_stat_get_buf_alloc_preclean() AS buffers_alloc_preclean,
+ pg_stat_get_buf_alloc_free() AS buffers_alloc_free,
+ pg_stat_get_buf_alloc_sweep() AS buffers_alloc_sweep,
+ pg_stat_get_buf_alloc_ring() AS buffers_alloc_ring,
+ pg_stat_get_buf_ticks_bgwriter() AS buffers_ticks_bgwriter,
+ pg_stat_get_buf_ticks_backend() AS buffers_ticks_backend,
pg_stat_get_bgwriter_maxwritten_clean() AS maxwritten_clean,
- pg_stat_get_buf_written_backend() AS buffers_backend,
- pg_stat_get_buf_fsync_backend() AS buffers_backend_fsync,
- pg_stat_get_buf_alloc() AS buffers_alloc,
pg_stat_get_bgwriter_stat_reset_time() AS stats_reset;
pg_stat_database| SELECT d.oid AS datid,
d.datname,
--
2.22.0.dirty
Hi,
I've done a bit of benchmarking / testing on this, so let me report some
basic results. I haven't done any significant code review, I've simply
ran a bunch of pgbench runs on different systems with different scales.
System #1
---------
* CPU: Intel i5
* RAM: 8GB
* storage: 6 x SATA SSD RAID0 (Intel S3700)
* autovacuum_analyze_scale_factor = 0.1
* autovacuum_vacuum_cost_delay = 2
* autovacuum_vacuum_cost_limit = 1000
* autovacuum_vacuum_scale_factor = 0.01
* bgwriter_delay = 100
* bgwriter_lru_maxpages = 10000
* checkpoint_timeout = 30min
* max_wal_size = 64GB
* shared_buffers = 1GB
System #2
---------
* CPU: 2x Xeon E5-2620v5
* RAM: 64GB
* storage: 3 x 7.2k SATA RAID0, 1x NVMe
* autovacuum_analyze_scale_factor = 0.1
* autovacuum_vacuum_cost_delay = 2
* autovacuum_vacuum_cost_limit = 1000
* autovacuum_vacuum_scale_factor = 0.01
* bgwriter_delay = 100
* bgwriter_lru_maxpages = 10000
* checkpoint_completion_target = 0.9
* checkpoint_timeout = 15min
* max_wal_size = 32GB
* shared_buffers = 8GB
For each config I've done tests with three scales - small (fits into
shared buffers), medium (fits into RAM) and large (at least 2x the RAM).
Aside from the basic metrics (throughput etc.) I've also sampled data
about 5% of transactions, to be able to look at latency stats.
The tests were done on master and patched code (both in the 'legacy' and
new mode).
I haven't done any temporal analysis yet (i.e. I'm only looking at global
summaries, not tps over time etc).
Attached is a spreadsheet with a summary of the results and a couple of
charts. Generally speaking, the patch has minimal impact on throughput,
especially when using SSD/NVMe storage. See the attached "tps" charts.
When running on the 7.2k SATA RAID, the throughput improves with the
medium scale - from ~340tps to ~439tps, which is a pretty significant
jump. But on the large scale this disappears (in fact, it seems to be a
bit lower than master/legacy cases). Of course, all this is just from a
single run (although 4h, so noise should even out).
I've also computed latency CDF (from the 5% sample) - I've attached this
for the two interesting cases mentioned in the previous paragraph. This
shows that with the medium scale the latencies move down (with the patch,
both in the legacy and "new" modes), while on large scale the "new" mode
moves a bit to the right to higher values).
And finally, I've looked at buffer stats, i.e. number of buffers written
in various ways (checkpoing, bgwriter, backends) etc. Interestingly
enough, these numbers did not change very much - especially on the flash
storage. Maybe that's expected, though.
The one case where it did change is the "medium" scale on SATA storage,
where the throughput improved with the patch. But the change is kinda
strange, because the number of buffers evicted by the bgwriter decreased
(and instead it got evicted by the checkpointer). Which might explain the
higher throughput, because checkpointer is probably more efficient.
results
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
perc.odsapplication/vnd.oasis.opendocument.spreadsheetDownload
PK pf�N�l9�. . mimetypeapplication/vnd.oasis.opendocument.spreadsheetPK pf�N��-p Thumbnails/thumbnail.png�PNG
IHDR � � �I� fPLTE"""...444;;;DDDLLLTTT[[[ccckkksss{{{������������������������������������������������ ���#��� �IDATx��]���8r�=�n��;���L��{��B��^To�� �N��m�����e3��7���t�����S, %��������d�5&M"<l�Z���s���+�������>��)�Y7�������c~��g'���>�HKt)������R�)���|��z6���c9��J���C������|1�1���������kV��:t���X[��Ch-��4�J�ux����n��#7��2Z��rn^�R����������z�{��I+1��{��?�����?�;E�������[��~������7�������$���*��0�V���q���)��v��3�q���r���t.mT�i �`��g���&9D�C"��6��<?
��Q@�L� ��� Vm���?�7������CF��/�Z���q������QQ�������������@nh�=��T?�z����3���9?|�+R���\�������SZ�9�nb���r�R�?�6��|W*�&��������`�����~^� �uo��J�F�����V�����������u����_���G����2��h�d�'1M
�U��1\�����X^^1�x����Y�� ��-X��Y^o���� ���_��}�}� �f�,��^�����/�����pmlS�3����-V��Q��JZ��Z��a9���R���:&N1�M7��?^�6n�"�I*�b�J.��C���S��k|F2�%����J�l�(c2�H���T�M��<j���� ��dg.~F�/M���$��2^�A7>��@f�}�Aj)f|u�?[=���F�%;&�k�}����C�+�[������?���Ex��r�2�A�uX=x3Fp�� ��fC�- V������nx7�?�?]��(>����N��i��N]�
�!)]��U����C��{>W
�G?�)Q����C������P
U
Z~�� ���z&9�!@jq�^-a�S���\;�'.>�_%����Sv!���c�IS,I ���e��� ���\�8�����
�/�r����������Uw$jU�����/���y����n�v�O�������x;����k��c���z�J��_�C�{#�\Ch3Q�����6�/||���z^RT���o�A�l��M�<@�"��0"����=���Oy<��c�����x�����lU��� R�S�5�F���x��p��Kj�IS������"�P�}�����H�wBE��/���
<:����e��U�q�Qd� ��4|'QO�����vd�^r�x�*���� �(���U���TeI[�������v��xmob�����Y����4$�%��!]�"7!Vx.e�R/%7�*��9HQ�R^U�t��c`:Q��BU�����d9���]2h��-'��xO��qu���u��(�IY8��D��H����@;�N:��,�qU(S�2� �'����<��lv���zz���������8}���5�t�U.�$C����'h�B����b'FHK�����>�9�*4^�����c���)�M7�_^8]�7YZ�m����&�%������Ah��k�����4+WD���lTU� TX��lQ%�^�G�X�(�-y\���3�� ��f����IAP�9�������Y�)|�K-LTR�I�=�F�D>Wy �����1���D�������Ux�(>�,�!K e?R�yu�
���x�:&-��)>/���/�J �:�h��nu�(���b|2;_��sZ<V.�/o����}���jY5ER����*0���`�{��z�����V�k���l�T�����Sb�;�N[���a�)G|V7��q91�������}'��sR���I����M�.���Vt�._�R�[C%��0h7I7 ��V6�{�
�s �����jD���a,,�"m�h�
���)q-#YWuk��&k� V/,^q����6�j���u#�R�oJ�5�+�����P�!��7���nx�0xC�tzr$�R�G�� �U!��YAmHRT���W��E{��Or�C y.�B�=/���{� W1�������O���(Y(Fe���������t��O����,�����"<���y�A2F��X��D�'|�mi���
[���v���
e�+��
W�+�;3R+�g�S<$���zNSH�.<��&��W�\~go�X������q��/����w������ ���KF��S"��2����k=V��o���Lv3����P�C����R���)�>�{5X�<�#L��l9�0�(��j��+�i]������"[9����IUcvj���Z�g���0�b�9����H�����|�����������_5F$�U�Qy���x�*�|t���X�h���z���Fw��_?K��1b�<���V"�E8R` h�>��u���c���y�~�
���W�o{�����h0��������vCxm�Y�'���������U������?C]�p���
Q'*�a�q�I�[��B�����S�a�!(�����������h(�����~
�ZI�cPM�)���VVq��U����P�E����Sl��9����9H�l&��le_���R_��9���U)�mY��
7�II��R��4S������t/��&��
��o����1����h����j%58-�TS+���D����y&#��$������0@t�S����@1��?9UJC����X����[��.#S��).��(���q�
�{zk%L�J-����M�V��E�#n��ZI���y=��cBk�+\��Y����J��i��;*�� e��{pY�m�9jix��T%YG[?5�S�z�
_����1��~AxV����
��t>[8X��p3��S�
�=>~=������s�9�|�+u�-�Tj"U��%7�R;�U�x�3xv����#��Tf���R��X��,����|����3�z�� ��dN��cz� �( ��������������c$V�"�^��am�n5��d��(,�r��ux�5��z��"���������;���e��o�����BY���S{���}��}kp�}����������8��3��c@����U�L
Rr�u����g ��W"�v�.������nm�:G' ��^����1�Q�|�������,���'�
xN�$�+2c�a@�*&�������8EW�RK%������5A�Zf2�>�{�h��zpI\��LI�
RQ���!��(s��f�Hl����7���cRdW�x��)��T"6!�"Yh���b�+��������nx7���N���IR|���Ja���`mRU)Y�;����
����~�"��b���bV�-nP�)��q�3v7��t��h�i�Q��_6��y����<��:O��j�r�r$�O���zR�U�1A����R�������4D���ucDV���3�E������J���"����a��Jy��+��>�^W�#c �S��+t��L2��S�yg�V������n�����sg�V��20�
*m�� E;�o�;��5{>k��6,p�14s��)GK�[��9)�e����<6�D��*c�������{�,e��{B��%O���0 Mr�����;0*3���,�?|sMj���Li�������;i����G��fa��L��-�2�5Aj�D2�w�Y�f|0���i.��1�Yz��O�I��]_�5�������l[��,����nb������������_�����c���o��3i�2<L�~�:�����b��rPF%sG{�d�e��62����/uv6��������w#R�����W����/�29?������E$Zw��ZQ##F�������]>[�V�����)���c !�}g� �1���#�Q�.�4����M���0��&�G�>J�;C�I���i�Z�A�����p��
������2l�bn��$=_%l�24�CWz8������E��M�\�g��c���(����������+_<���N���4�����I��p"�������Db�U1v*[�{���1�?�'-;<�0�N*�\�m
{/�`�#����O�ZT!���-b�.�3�hJ�n'��Y�q����q<Z�?H
�y��c��VSa�4���4.W�WK;j<zf��M7�_^8�04��1�6�_����R�s��+����h����z>���m=�J�c?^��"�T��u�t� � 9T��~9-�U+ y�0N)�h��lVN����G�K�����a��Y��z��?�/U!e���!-h����Fe�
ZGc@�
��x������)J�#�b�6�� �.�s��"v���o&�_��`g�EkGIlI��G��g/�����9L��%��p������N�����~�3�<�c�K>�Q�C��8��>��N`���"��R�DW���s��B��
�����c��P��k]��$9�6�)�u#�C83 ���Ux����T���Ck�M��i|�`�([�-�5+e4��z��{Q��&I-y���
N�By����WG��K�E���]��acV���N����.
\J?�$x�&�yG���pN!nb�����y��@g����!�@#9!�_K���+�$���W"�;�3�O��a ��'�� �5�V��Zv"�T������d�~�'(J$�Q���8�;Sr�.��J FoD�)�G�;kWNI27c?�������Oxf�"�����/U���E��L+�9�a&~)�k>��^*�(rh�~���|rb$�����gD������T���"]�=�ox�(����n�����X��p�S���������7kF'/�@�����"�h���J�&A:o�����v�n�L������(O����i�'�S�!A������@Z9{O~\+)�I���K�!�s.� ��"m���J���y�xY��� ��@��%�(M�q5��9�����D
v���ugF�Q)[�"�C�k�����X�%gL��q�f��\O��y��w��k�5��\wpQ,I�9/�S|ZVH1���X[�d�����
��c�ceN������Ap����N|���Zr����F������$Y�_"�d��cX!$�\�Z�U�9���cP��=E[C������fL��T�b(�ikv�*�Z\�W�����LW�!�:$����+q^�� ��R4Hl��,�W�&��
����e�Nk~��)�uV��VrB D-���\i�b��F}���!�y�A
�C{:�h�&e5�a��h�^���|Vu��
�o�Vz�Jb�i������^�3�Kb�/Z��k%�JZiE��2�a����
����!{>_-�[�6����Z�h��6Z�
��e;\2'��Db�z��k��`��c��6d�II����N�X|4L1�G
�n����=���� ��Lq�U���*��J���z� ��C#z�����>����R0Q����C%��(EF�*��Z��ch�:|�'>����&���0��"FT�y����c8/>j7����*iJ�������r�lYv�Yt~af�u���Y���q�c��h�
$�=Z>���>6�,�������c���$�5!�2D�y���R��>��n>�0��1��nx�0x��pF-{�$���|4���4�_�K��������`^��+���V�<H5nO+4��KR�%fE�m[$�Z���<��2vb$#��]&�P� ��|��0@��}[�S�1�����c��Ab��9�/L���ch\x�1(�p��:�G�5�w��������3�4<}pr�IN ;�������nx7�����2�l5��� IEND�B`�PK pf�N
styles.xml�Zko�6��_ax�E,-���������.� ������(�;�(�T�����K�,�Q�I��LF�����>��/��s6�#BR^\M�Y8��"� -�����?������K��4&���UN
��z`DN@��+�y5�D��XR�*pN�J�+^�������������pSZ�{5VXc[��v����N���X��)�����������%VtO�{F�/W��R�*���l��q��r�Lo�p\��J0�J��0�'�A4�����c����JE��1��p��� `���q5eZ�u�����l��x���83�v�,����H��9V��^������]\�|�\��*�m�E7�9���Z�&�Qw�'�}o���[A
x|c�����4�E ��!��B=8�i H���I�;`g^��F�l8Uu��f"Iz���"����Aw�l��j�a���5��A�(4�N�wWlEV/)�����%���DP���[�FhF�G������J ��V��;�(��d�\5�[�/�B����/��CzU����i,������)�U3�1A ��\_��W7O��V�j�3��i��|�)�%�){����\�mg�����2R���<rK�l!J�b(/wXP�a��N3��=���@pP�^�5�"�W��f��_�h��_M�`����n��- Iq������t5��b����K,p&p�A%�/����vF�%J�T�����Y�v�����5�0�+�������4���J�v�~����:�b������7��
�;~����iF��c�=E
���I�����8c���+h\����X��J�/Z�a��nqrv�O�]i��0V����i���0���
�w'"78�[�$�B`�������;�1�W����� ��Y�����*bU�81ZAbP�f�?-��,���=zq�-1�y������i������D�a�%)f��,�H�;��Yu8�+I��5Wfr�%*b���BEHG��s\��\���Ds�������u}!�04Y�����r������|x�����w�4� �f=�8���=��Zv��1lR
�5��E������Utl�� �{��{�ia66�%4�J��f��1}rt'xo������b��`}�p@xR'l}������['C��������m�w�w0�5o �6����,ix�R�]����"o��%����DG��y;���� ���c���~�F��_�������@��pE��(�X��kz�� }���)<q��"�x��eH �MA:}
��>=*��� g��q�l��]��gx��p��<'�/�������=l��`�]N��q|~j>����
�9G�{�Ck� �gt$c"f ����������R�R�IaU�����8O���uL�����1��gg=��%S��TJ�B��Hq�o)�����=~�(�t�$$�{W��$�?�j�
)(�~������1�N��,J����{��osdxq��,���#��XpL�R,<5=v�
5m��4�A���oJ�� m���-1O��`�:���}_�����Je�5�R-�?�y4�u/����L u�s�N��W#�-���i��Z6�U�h6?s�������:�v�����[�����|.�=m0����
�T�����O������L�k�������t�X~J��/������,�����i hLu�c��A$�1�z<���j�:r,�!���H�n\�y�S����K������B,z}}}}�7��r��=�k/���K���z�i[��V�^1�#?_�����E�A���Q����f�f�� ���K+�T�mC����OQ��P��]�����&�I{Z$Xy[�WC���
��j������a�D�9���0�o:-4p���W8|h
�CG����*gg����t~z��X�����0l��v��=4��6���)`�^(�����s�?PK�o#� �) PK pf�N Object 9/content.xml�]]��Hv}��h�yU���Y������,�������w����d���r[^��V%;x�?��{��xH���������a���^��]_��e�Z���^��/���?���7���z��������g�nw��
[�7�������M�8�7���=��7�C�K[��k������ys���������x��q]�����G�W��^��.�8���o~�]���a3��B������,�6���o����������O�'�t��9������ /����~��r�n�x���7|����������Oi�����/�����W��]���}!4�������W��+W�w�\��n��/���������\�o/=V\��j�_?\LsX;�����T��@���������������c��V_���r�YN�����������-����1�/l ���i������������}�]���������p\�^"����E�z�o��q
���f�-1���q���������~�:�j89�����n?���������+���g�~�|���q��!A^����t��w�B�r��k���:.Zl��n����t�.�+W�2���M/&Jgw�u��!�O�I�p�mM����e����V��{<&���<���_�:��f�����HYQ ���
`�x����^������0��C��^
�y<������OV�[���?����j}�>-����7�3�|�f���X����]l����|K������Bl��q��b~�]��_�v�������������
���CwX����m���;�97[��b9����b�����B�������"���d<�b����B�i�����Z�����������X��z�M�x�� '��B1���2����]o6i� �VmM�sq@�k=��%��������w�������W�v7[��M,n�C��J�e��9|������>���w�n?��_����\�O���+/�k�������,,_��j}x�,�g���vC�l��E�����%]�"�]TX"����
���]�c�O�����}��u����q�NwqF�0�������I7���,�#�=��'�_<�Yo�W��e-�������!���m���3 ~�$p�\��J�����{����n3;>?L��z�����^�y2k�[n�^��Q;s�7.)��9�����J�������~�����~XI���C���~����k����W?�$����2���Q��-�>?���������yj����������Y��$X#O�t���6t�h,���������}{�4�=��a�;e�r�8����c�olf"���t:��1F59d�m���L�c�K��v�8�U���������W�?��j~��Y���|\��<;Z:���������y����������n�+���l#~���O��t������g�OGLg�i����*od����z�����p:��D��p�}j���O��p[&tp{��UL�a����P��]1��O#�~f�0� #�j�ocG��"�����a\�#?,j��o.Y��������`_?]��������0F�~���/C���>M���L�=��SsT��f�Ti��Lp����n�]�_�w!�g��nOq8o��� Z������������c��XYO��m;��S�<$��~�]�b��|�����������N���~�:��;k��������r�������Lv��n~[l�������3r*b���"zuN1Y���z�}���u��Kr��y����4�|hCv�b�����������h�y�����KX�G� ��?|���.a��G����fs���}�n���Z2��Tj���'��=b�b���>$k���������q�;�e�m[��������%Z���;�J��99lH�TL~��a� 2pQ9�T3�)�������@�OY�cv�]B6�������O�6V�a���]���'���<�����O(����
�����!<~xxv������������sC�Y_"�K"��;�a�]���0]����������u�������-�]�..��������7�\Z����������Z}��F�K�������P�C�_�~��Z�F�l�O���aYN��7�5���F�vf�3��3�"��$ a��R6�$���)H8a����Fq�P�����4.�E&�f ���k)�k$��9B�pJN��5�N�0�����kx��T3�4*Q���o��H a�V���@ ���QB���W����6��\�CHM �V5d���y��nH�H�@�����P�6,��'f�m�
�� U��2j����"x�R4��Fs%G�d�`��$�8Q{cKd�#��������"x��T�sc�g������p��m5o�#����Y��ir�p��`� g�L�^��
�,����n'�m<��U��za5I� *�J(3�
%=�&�*��I��b���J�G�f�=���F� V��G9(V3#�����I� Tt(��������9-��~^��`�� �B��{F����E���8]+/H��v����q�'y?BE0C�=�=�^���'%d���I�C��
���T93��1��OP�P
@�(�n��(`D`�$�!P�"Lr9Q�U��K�4u5T3�P^�S�� NB`�$6"N�a�������yyc�m�`@����~�����e�(GO�%u-�%����,~c���g�������|������ X<��TwLx�%�nf�I�U.��.�*�Q>��[z�0A�x/�k��!����L0c��jh� ����z�H��mX$��S4�*��d�8Bu3<���UT3T��|o�w
I�AE��LF�%�J*E������R:��{R +T��b�����57A�h��h&}�)Au3�rai�%�f(
�������L�'O:�����>���P���`Z:�������;��d�a��
�$�q��Xb�OP���R���=LP�P��W'�W��x&O�$�t������ �����Iz�?AE0Ci ��p!�h���������8u[�rZ}�rN�3�?}�N��G�'\.��5A�x�h&�l��=��������]��`�� �E��K��A2A�x'�na����Um�*g&���q� *�J(G%��ah��� X<���S����LIP�����]��kw��`��y���w�<Q;A�xW�nn��'U�U�LI�
�������"��&
:!�h�k�;�,��!���Y)�������
��0G� e��2�&qq��^AG�U�L9+9��)�J��D2���k���I<E�����t�U�L��7�7T3�P���\��f&OzQ�U@FG3�<10B�3��YG����"��4 �V��>[� X<��S�p��~r�KP���Q��h`��`�� �a e���@�`�$.#N�L0��"�G*���q���G�^����+^h���
�$�q�����um�*gfx(��gdb�N���9*\9A�%O�*������P����_�$�$�f(
}b������O��Z�7��U��h�h�3"E���B9�Js+h�� X<��S��������s2�jI�����0?�x����$�} ��������T��x
����X�r��(�h���8��c.�'_���������w%�f���y'����6,��9�)Z2Ke�*gfs�Kz}�N����%"~�3��� ���x�0E+e�;���P���v�S�+AE0Ci ��()
}.>A�x�B�ne4����s��;��|�JP�P���L�`�;xCI�����x�#R9/���
��^��G�%�:i��\&O��4��st.�p_���89�E��<C�N��G9%�O�=A�x��h.e�#}�rNa@����IP�P��J8c�:� ���8�8]w��f�*g���������"��4�Q����2�/�$O���&��B�O��=T93g-�������~�wbW�!� *�d��e�%i�U3r��!�o���(���K��� �E�x�0%�������{�rf�k��]�.�*�Qn����l� ��IC�����C&}�rN^*���}�N���=����mX$�?��2gF���T93���7�T3T�����
��
�$�q�����;F�rf�2����$�^��G9%�{m�K8 �h��J�F�0 �i��T��{!h�H�P���K���1�a�$>!L�a�n���M���d�y����&�f��G�$VKF �mX$�K����+�{ ���d�Y��T3T��k��
�����I\B����JQ�3Au3�LyC*���� ��u��?zO�"$O�����K �v#T7�0�sg�?AE0Ci ��8.��VT4IA �����&�fV�8���
)�*�Q��������aq$�!L�V1E^����9I������"�����8)�%a� X<�oS����e��������vO�.�*�Q����;r�xD`�$~!L��*�������I����]� *�J��m;�{� �����0m[/5@l��Ir�%};a��`��~��p�XC�kK,��7������<
0"u���+��+vB�����/������6A�x��0e�����]7')�f��� *�*�Q���2��P�`�$�!L�NYq�u%�nfR���F������'����D#�'qa�vFKK����L�O�+�v%�f
X��������8"�h��m�43�'�����N��YT3T����s����cIr1��k,'o@�v���d���k\ )�*�Q� cZ���bG��4,��� ���L����y:&�^��G�',�D����ID�������O�%�nfRj�y�p��`�� �Ea��+g�����I<D�����qR��v���t��zd��"x�r��0�FR'L,��AD�Zx�������T\8���R/T���������!X<�{�S5W���]9'�X~-�EpB�=������� X<�o�S�������sR� GG����(�$���j?A�x��g�w_
��T�#R9/����9��� �_��v�<�3A�x��m����� ������9$�f
8��+3!x�x�#�'�q��&L�O�����i#���r H�P
@�(B������AE�'��NEX�H��R���G� V��G�'�r#����&qa���hN�%�rfF:u��[��`�� �Ca:��o|O,��A���3-�������)z
�"x��rR��N7D �&qa�J9G�����
48�g��"������\W�8�#�&qa��?����&�rf��E�+{D������"�t���$A�xO�l�
�=��T�� ���C�N����)VY��E�'qa���E,�T93g-�_p��"��4��S�T�~� ��I<E�����V� ���D�T3�`~����FBB`����Q�V���4A�3��qK�n%�f x�l�P'����I<E���$/����9)f����&�f��G�*V
�}gh�"I�{B@IC���H��W^��R +T��\'�2��L,��I���H�$�
T73%���<�7AE0Ci ��xaB����,��I�i�2�������+\�� �u'�^��U��i�<�����I|D�����w���;Au3S�]lC�MP�P
9)J0��3� �E�8�(eK��&_oOH����.�a��LH�P��F%�R��M4A�x7�k&,�d�8"��r*��c�kD����P=������dB`�$>"L���?�^���L3/�!�� *�J }k�$���=�&qa�V�3C� MP�����r���"�� ��l
g��w'O�#����u��r��ffe�%� *�H��,)������
%�T@�v�I���'�nf�)��M'�f����������2;�]Xa8y�g ����������`>���)r�Ph-���M�wh�>i~�V�/�U�|����l�������PK��3/� pX PK pf�N Object 9/meta.xml���n�0F�>r���Pb�:t�����E�}C���� y��)���������'�]��;�i�V9���x��R�9y{}�7dW<d�x�����QY�F���U�t*��3�j���*VcK-��A5��%M]��r���������}��@���v�:��_��3���pLh!
"�����R#�T�Z_�F|�vqq&0�g�4BT�0�k�e�Yb�8w�3�nH����sa�'�I1��(^dN�D��Ym���0�� H�8H���Ru���&=���@������z�d%�8�?;f�+n���PK @�z" K PK pf�N Object 9/styles.xml��K�� ��9�KYc,+��)���LM� $SA�
�#� KA��PYJ����/�/c#w=�F��d�����bP
U_�����[�r�t����X�pe��?%7;'V�L�K�iE�a��
7�2-W���4 ��?�Y�<�������=����������C���.����T�h$� 1hZj���P?.����`<�~(��k��N'���������*���`���������G-��2��������nT'�7��ez��ru|�E�z�c= ,�����C����<}G����\G8{�3*��h��q9v��/�Lk����b�[�v9H�>�.;��mo�����[g��e�������k>�>|^�����p�Vm�T�p������3KW���J��2��S�
n�����k�MTYy�{L���o���j- .��Y��� 5 ��Z�W�kLa������
�����=N�����L�
����Ct�PK�P�� � PK pf�N settings.xml��]w�8���Wp���bHB'�� i���#��l�F������~emJqJ���{��l�!����������9dQr^,��H�"2=/��O������d�Xw�x��#�����lNx}u��0R��#^'���.�:�!�4�?��[Y`D���3!�z������e�R�V�����KJ&h������MQJ���:�h�Ii��XXw���T���8l>~�lm`�� �EcSX��v^�&�s�o�V����6#����� P��9)��<��(6����"�������b�K��������/!��vv��������.��m���R�F�[��c^�[���I(6"�(�����V?@���:�c;�L�O�����q�z�#!g����>Mw+T~��|�N��2����c$n��o����2�����z��,'������;�a�n 2M�J:�M3\Y#)���m ��6���P���H�L=:l��-ki� La]_T��7g4�2��mJ1��,�)'q�`Bw �^fL�8�2��`�`l�����D�z�$7"TR
EEEEE����N\FX(k�P!e2��.����R��}��������o��������+����#������B�q�
��U�d(, %2�������b8VI�����@����jEEEE���o')<(<(<(<(<Dx�@������'
�������_��G
�������e�t�H:���w�{0m��z����:�Q�m���*��S.&)������&����`�?W
���������>��yI�3�#����t�J��R9�fp[s�K�e��4�����V���9�P�P����F�K����p���=�R�8J!dxl�(r&�������j .��Sea`%,?�jO��,�K�J�-*SF�m������f`�h�xb�3����_������E%�k������z���p(v��;�7]���5��A`a� �'�����n�����c��9�EG�S9����� ����C���+~��E���K����5$.)�i�}�����C�G� #:�3�'�V|�v�����w��&�t(�X���~_CFb�����������;GF�hN2����y��?�M0��������)��H�9q�����7 uk��SX_����
�;���u!���o�|�4 ?/��?$)�������rM09����j����p���L�S+���b�CS��br^���2�?5W���4�k��V���O�4$�YN�_���N|���\�Q=�������nBl?|����&"@��A���K��mo�������o;S���jng�������u�V��o����~��������h���h�������pU3*��z��?.�}����-[^M���
�kAo��;�X>������Nc����^�����k���Nexm�;K������}���H�Bo.t~�
�U�����B�;�����v.���`a[^'��dyv=��5��������
��a���WjQ_O������c+�j
����l�M�9��Z`�� L�������������V��aOoe����]�[�a����Y����x
r��=�CUT;?�������] r��,?��� @)*�����O�����o�PK�km�� 8h PK pf�N Object 8/content.xml�]]o9�}�_ax�}S��/o:��,0�3���*R����2$�c��_�U,�:�#Or<������-�X��\����z�������nX���.y�./���_�w7?]������?���w���z�]������g�~w��^�_�W���.����~qX�v�mw�:.���n�~u����u
���qs��c������x��CY�����k���_������8��F�~������fv�{�o�����x��w��ty{<�]���?n>������sn�N������7��j9�6]��0�
������8��B���v��O��l�,��/Z���������+�Y�.�gs#��+W�7�\���.��_i;��_����'.�����S-����a����}?=j�����qc�|����_,�y�>v��������f9Y���2�/������=�t"~0��+?���T����������my�mO���.<[�����2��_E������'�\��a���������{�����W��E����w}�x�����?���e>�y,D\��_p6���^�[?e�<A�����4
]��;��\�����n�����+r��������8rew ������b�t�6}?�<}���wW������>�w�������u��AO�������Y}�?����}� ����m��,�������~�������g���?���B��]��#�����7���e~9�n~����
7����-����z�T�o�'7�����;o�n\w���/�}������u������/W���a` ������1Z�����}��1rn����p���.�6��u��v��a�?� �w��a�z�;���k�@�����@-�A�_B}��j�:��C=/|P3j��ozL����]����M*;)�����mP��AO^|2({�A��e��n��_u���j��f�n�����bs��_`Zvi�����nq�u���c���f�������q��j�x�OMy.\}���p��Y�Y�h���p�Y<�6�O�����7{�n�Kz��@���a Xz�����n9�k�e� �)C7������F����?�����~.3�^��g���W�|�_>���a}�Z���_�� n�Q-N�a�
������$x{�WG$/�;�Z���wx�~�7����40���g����*��Y��r{r��SDe��1n�R`�s����[�N�����+���a%;��J������"�C�Y�~\H|�cc���^��-��/�h����x�S�z���L��
4����$X#O\��BD�P4�Em\���w�?]6M�~��������0����\�"�(fN���q��c������4���$������-��C�B��o�n�|������_��#g��������R���O�*7Bd����k�����~�����]���_���������S�� 6�M�[]�B�|���q/����O4�
��1��{�[��U�����o��w���_|w����
)�0����<�w ����������|��~����_�Q���o��Y|�N?�u�g<��)"���K�n�!�������aX�������8�w�b����F��+�����j����z��<�w7���C%��R�J�r�k�
�_�by���!���
�n$�C�<4��~�]�(��4���n�x�v��og���f�^�,b�u2�����1>=���~��4���t�����0��x}.��?�f��G;��]o������wy���'?�"z�]�����]�y�
��nh�3���j���N��[���3@����>�j� ���3�m�y���n����?<�.%�;O!��"���h������
YBRjv�-V�>�����^�������2=��@���5��y��v<��5���HA���=�Ly��3���O������� �OY��cn�]<��n''7~b�c�UwX��A#����;���J~�q>�p��`{�|���7D��q>�p�����c��:�s-3��'Ou'���>���-�;����\���P��� }�����HF����0��tn� ��hM�� ~W�qf<���z}K�����w�dgFo�o[�����t���r��kx��&
�>�0�5��`�Z��W�T�K1����p��"p��/A�4���"�������+�#�b���v2�b/�Du���P��03JbF��JB� ��H8��^,��
F� 1,h0��k�a���<�b����9"/�Du8�A��E�1#�y� ��kG����@���Q]bxfFM�s^�I����i��+�����mk�-����1#�y5���8b�"Kbu/�Du�7�-1#�y5��50V#D~"�#�b�@TGuM���t�^���;���X�u/V��X����A��:*M�C?:<���I~��� �����c���x� <(���-F��B�zN����ed].��#2��D����������d%k���I�q0���+�vD�P6e�����3dn�E�I:��V��/b�������h#�QI
�I�g������`�k�&��b��H�/�E Bq���C��#>l�����{[�o� b��H&)�E Bq���"_R�"��$#��^�H/�P[m����t�P\G�)�P��^�����p���a4j?�"jIy�D(���V82��"��$'��^����R�o��p�u�Z�|%"�*oj%��%0^'99��:�g_�X;"C���D(���/-�?eG�t�Z�v����X5�hD�p�+��X���
��P�������2BkGD�H�Ez"�a
K��-nZ�.���d��� ���"���P8KGG����d�`����W��X;"IV��X"�Q
�I$D�I6����$V�b����gSA,������-� ��H�q(��|�^��q�-���@��:*C�4�50$�8��
F^@�b����4����5�P\Ge(�1��^���d�P�+8�Y��1��&����\(��Fco����(��Hrr(��������6����j����@\G�P��V5���fM����� k2Q�k5m&]F��2��6�P9(P�$�[
�����X5f���Q, �� ?B�g�FfE�������5yb)i*��{ ��)��)������d*D�I�������F�rD\[GNrI�"p��gA���3�%0Q>lE����rDB��*?P#i���b<*7���$��40k�����a���4����y�6j���b?��O>s0[�\��%�6EFE���jmK�E�P��9dk'lI�5If����&�J�4u�FX���t���~���j�J�40k�,��[�r�nQo��"���{� ��x����2? 3�0;�,��5�_��b��|����%M�@�G�zm}����H�fM��Cy�4*�$��I���Q,����G�������#Y y��6�/���(�f��b9�{�U� #W��lI2|0�
��� �IS7.i�#^��"�xZ;T���QY�40k�<���r&�X9"��%kkIS.�A���B�&?8i`�$�>�?�� Cv�'M���2�JC�@��=(��ZgD�\�40k���_~�Z�_%ij������X"�Aq��:_��"��$�����OOx~����������E�P�E������(��H�z(/n�&]|kG��w�X"�A�����E�IF���1J���IS9.�m�h��6h����=(j���(7�s�40k����[����X9"����X"�Qg�j��e�(��H�z(V��k�Q��c�V�GM�P�e t��f�"��d:�Z�����l^���g
�X �A���*
��"����V �[������l���F�D(��z'mE�FfG��C�����Q�Q���]Q,���XZ;�L���40k�,���,g|kGd�"oc'M�P������6����Y�d�P����|}&��#r�2Z�@�b<���h��xav��!���J�������p���"������
/�{av$<�+��
dQ��&�� ��uP{�2GNSN�5I�������Q�s�!�Q,�� t�pI�{I�5I���Z(NO�M��qq��JJ���~
����J76��G
��$���j-m��N��q���Q,����e<_��"��$\��V�������jL��_��bxPL�_����E�����d�P�������4��6n�X"�A���"�vXav$y=�'k��"ov%M���r�f*��\(���f�:����I�&����Y;%$y�/ij����^�X"�A�������@���Ir|(6�|�-��#F�]sIS.�A��q�7������d�P�l$��q��o��+�u�Q,�� Z�� �_$
��tq ��Bj�In1i*��^%�)&M�P���������YI�&��������=8[#D-#�Z����B�b<�Ng����#���|�h%-���r\a1Y�J�"p�xo@lt��QM~B[���I�}(6�+A�q��r\����b�P��B8��"���
�-�������N��c�2
�4`B���:+����"�����r)9�
L����8��.�E B1�:�������&��$���e+9�41j*��9S�;{�D(����3��$��I�'���|��F82K��q ����"���k�ZIhd�%I���Vk�
�Q����^���U�Pl�~x�R��$
��$���bkU�F�rDJ�G�G�D(�����Y���M*�=I��������*IS7.��![�,���W!B1m��P�y������.��|� �/3G�nD��6�'�"�8j/�7y4e�%I�UO�J���Q��B�S���\ �[P6���m ���$�����k8Y�J��qqM��A,���Y��x:�9�
���AEl���74�55�rR:�+hT� �z6�����0�`�$�>�?3�Z�����S<��S�@��>*7+D��^>a�T0{�����v���$O���9�����X"�Q�Y�8��e�%I����EC�IU72g���@%M�P�G�g�
�-��7M*�=I��\����&U����� � �~T�V
�D>yJ�5I.���9��Lw�T�� �'-�K��"p�����J���Z2��$�h���T�#c�[C���*�����6���&��
fO�����mKW{��rd�h!���UE C� *��j���9�
fO���n&1��GU��8�~��gR���/Xp�87
���T0{��!����!g���IU92.�uds��*�P�!�� ��s���I��b��������������� T(��rC�0����&��$��h�Z�������`J�g3�QU2���)�g����
fO�MDy�����8�&U���hM��L(��� �E>�d�%I&���(A��IU72.�����*���|�h���A�Y�da�z�����IU72.��.���4E�Bq�g���m]�����Ir�0�VN����A����kG��J�"p���CD��o�c&��$��i�0K|z����+�kDlR�� >JO��A�,��T0{�<"�����r������G;���IU2���'�,g�~(3�`�$yD�wkf������W�
)5��O�"��<�3�'p��hg9�`���Q�����QU72������{����3[[���vFh�ajp�����C��V<�����G�?��4�9P�������*�����g��~��$������g�~w����_PK���� T PK pf�N Object 8/meta.xml���n�0F�>r���Pb�:t�����E�}C���� y��)���������'�]��;�i�V9���x��R�9y{}�7dW<d�x�����QY�F���U�t*��3�j���*VcK-��A5��%M]��r���������}��@���v�:��_��3���pLh!
"�����R#�T�Z_�F|�vqq&0�g�4BT�0�k�e�Yb�8w�3�nH����sa�'�I1��(^dN�D��Ym���0�� H�8H���Ru���&=���@������z�d%�8�?;f�+n���PK @�z" K PK pf�N Object 8/styles.xml��K�� ��9�KYc,+��)���LM� $SA�
�#� KA��PYJ����/�/c#w=�F��d�����bP
U_�����[�r�t����X�pe��?%7;'V�L�K�iE�a��
7�2-W���4 ��?�Y�<�������=����������C���.����T�h$� 1hZj���P?.����`<�~(��k��N'���������*���`���������G-��2��������nT'�7��ez��ru|�E�z�c= ,�����C����<}G����\G8{�3*��h��q9v��/�Lk����b�[�v9H�>�.;��mo�����[g��e�������k>�>|^�����p�Vm�T�p������3KW���J��2��S�
n�����k�MTYy�{L���o���j- .��Y��� 5 ��Z�W�kLa������
�����=N�����L�
����Ct�PK�P�� � PK pf�N Object 7/content.xml�]]s���}������M��&������>��3���h�=���������(c9�-�x�d����l�sm�
l`���?>n�_�z_V���b]^�e�*��/���=�.����>T77���ZU��M�mf�j��/����U{���}����}�����b�,��]����
����jK����������M���{�;�����?9�zU��^���N
/�����q���T���]��/��q]n�xy�4������a���U}����?z|�����}��g���b]���b.������}?wn�J���uQ���������������;�Y������'c������V������N�d�_�A��/y�B�9�Y�\p��.wg�l���������6@���(�������u�up������zy�x�9�4{�X�3f�WG�#��#���@.�������������w�&>�|��Y��7���3����"Mu��������L[[��nw�f��pwG�So������u����
����x������,�Ib�^!��?)l5����s|�%�s_�{���~k!������qW��;���eWp�����7���sw���,�]{q�t�6U5��-}l�T���j����y�s�Q�n^��E3���Us�
��������6��'I����e �}S���\�|�����m�M~[z����]?^�5�|q�����[�����Uy[66��������{~�����,�����&���'W���]r[��;��;���n�b��eoUW��l�m�<��������am��<7�U���^m����O]5�s�|{�����M{��Ix�����M�������5����g�A���;�:�u�x8��z7��^sa��C�}�ue���]+���=��\��s��S���9���k=y���Q?������v��_��r�*��e�^�����{���Z�aqSy3+6����cy���� uy{���\���b�\���MN�=�����,h6^��������l�_k���ns���m�%�,�r�"�V�v���O],[|��
6y.f��;���v����������Y�������jp��,��<_������}P[�����X!��U�C5l��7�������@���_�w�c�����is]�g����1�W�g��C�r#�he���N���,=��������W
77��N��##�����R2:]����G*�j]��/#>�����j�O������-�;�<��VOG����-
���-L��\I�q��p ��@9��/���Wwuq��r>�������w���:��.��/���v� ����{���uS6�����8f'I�2O��2��}�6n��������aq(���t\�Q�vuA~q]5��������*x�I_���p}�������ujq�e}/���+;f�y������Pw�Ym�6�V��p��}n_��E���������:ODQ{`�gq��)��x��?�7���s������r�?���7���|���t]=�$��QV�\�#{�f�D�J�K���C5TUu���zUn�����m�^�}������+��)�<R�<��S�m���]{L�7��'����wu��m��x�;�[�{:}����K��w[�����A���Y��!z~�c�,�����d�m����b?�-�>���~���}��F���\���m������h] ?���*��q��+,�W��"������43�J;�i�<P�}~�gr��h���O����/o����������L<��fm���g������f>�������ueA�Z-
������]vy����X[���������2�������$����n��l������r�on�{QrTs\��k6//���I�������}n�����������5�DbU����������~�#��5���56���O?�����\���]�-~�/g5���e�M4�j6N�9�|�7���6�ys Gl�����!���0=/�{O6�=#�~ ���N�t0�w�����n���~p�4
��Y?�v�����������4����
!M4���<)�G"�0k������9uD�L���-.���2�����47Jp#+�����#R:
Ys�HT�Yn��@,w%4g*p&-�u�����H h��9
D$��� �4�Qi�cp#+���5 ����4
Ys�HT'u\Be&���Iscn�ofRD����A�,�8s�HT'�L"VI
yZWBsf
�d�p�2�2^b6"�^���9
D$���N��L�3R�i�����d��9s������3G��DuR[���hx�`ndo"�0�q��%�( "����q4�����i���X��SGd�0�t�(����,3��d��
�4R�d�7��G��-r���X��)R�oL����H�E,���:��g���,��:��ic{�/z�����1��7%FoD,��4
����G�����q��M�ys��bh��9
D,��D���~�G�c�q���#dmT:"- ��)�����'�4����6� �������
�(�����@��:K�HS�nSbvZ1zu�� of Ddmg���,��r{�����G��Gr<�`(DG �8����u�B��s��/j1z
�T�OP�B�%�T�"�%K1�5��:�������4q��"�i:��G��nis�Z���6�:��"���xX,�yQF�3�2u j1nai�3'�(�:J�QG,��
��:��Gl2x�;X_?��1�2�^�X\�)Zao�x�=�q��EI�T����S����"�i
��qt�F��8b���
��2��l]���>�X\�) �J���8b�&� /z�B�
r���B��:O�@-]��t j1zS����C!�0��xyfD,�������j1zq����w�������� bq��Ph�����8b��w'����`�������AD���)��F�V�HP����a��%c(D~]���.����u�N�+E5o�(Lg���~2<���x+�z�a���P��
��W�����C� �L�e/����k1p�W�[��@��Eo��:s��$f/�����X\�)�����e�
�Gb�f�u^�2"\����;�B��:M�������]�5�� "��0C!��o�C/D,��
��0%nu�@�#F�����k@D1�T1������u�B1`�j3zq$��Q��:o��^�X\�)Y�#��7�5�������������@��:O���u2��:
�8f�"3xk6C�+3�
�^�H\g�F\��� o@j1zq������`�P�xp/D,��t��aR�<�ys)�Y����A��{�a���Ph�ky{O���1������
���.fx�X/D,��
�'(�����^��4-
������=��N���}�bP�����x+��BdP�0<��"�i
�����: �T�g����k��'"�i
��
�����SD�P���du��:M�0��������"z�B�1z5/z� bq��P�[�����A�cF/"��j�`�pb�#�>�X\�)g-q��8f���4�t D:��W�9
D$��v�s���G�|�q���:��>"�:6g���,�b@?���7�3��Z�>��T��Y
��@Gw&�����V`�������E���^�X\g):2)���V$���7FD������W�~��'"�Y
����3i~5�����!0�v�(��N��fH��l�s���~A��SG$��;�^�X\g)Z)��Q����p�C�^������P��A��� bq��P��Er&������W�>���:�4`g���4�"����7�����^�������(�����(��NS(�K{M��>5�� �� O_
���EW2
\$����EA��L�A���p
+��9uDGV�8�����u�Z��p�o�� !-�5�����4�L@^���a���Sd
V�9��E��xq+`8gN�u���@��:K��R�;��G��������%�7�3G���u�N�����4?�&���8�|��SG�a����@��:K�0
2[g���/z�Bf��pC!2�G�3G���u���hX�L�A��Eo=�3��H��^g����9b�p8�M�q�O�^e�����5�$�����@��:����g�I�#hr���]S�tzs��p�C�-�"�I�,���
�+�y49^��oN��d����@Db<K�����nx����&G��(
7���G�!���(��NB'��5�&���I�F!B�{zs�x���+�9<,���J�$T^�I�"hr����J���SG�H`z����"�I������4?�&G�^���\w��������Ua/D,���K�&�p���49Z��4�|��SG�8���@��:������4?�&G�^%�pZ��G$M�B�������Z���zI����M��h��R��������R��Q b1�5�O��WJ�I�#�s����M���
Q3L�"�Yj���������X��p1�7��(Q������"�Yj���z:o���-z���3��v���(����)�Y��I�#(s��MT\W����C$2V ���^�X\giF��h�M������F@hx�e
�'��K��I���`��7i^e�����GgNQbh�
k����X\'�*�U�����9V�J������8�4 w���(��N�/%�������M��X1,3
��3'�H�X��H]�(p�xO���4&�x��G��h�E�8���F$���J�JF���xR�)����7i~}��"Uq�qYs��d����zs�X\'�Sqnt�M�A��E������E��QV����^�X\'�SIj�y��iWB�&�t�V&�q����8
��Q b1��@�e,��*u%4o��.-�U�Q�7'�H �A�ex�����xCj��B���Pd���b8�a����#�e�A�7G���uc������,?�Y�0�;���4�T����9<,����*
Ex��E���q�����,�b�U��d�X�'�+�i�q�����M��x�����7'�H���)�"�I�����
b�+�y;^$g���7'�H���&o�����:�"ho��Z+��HE�h���Fd{]����+.�I}r��"�z�I�#(v���p��7��(6&��/�
F���wR�����I�#�v�6I���7'�H���}������I�0Y�&������7MM�@zs���L(3��+.�Is�Z$��]���7A�cE��Q�����6�4�BA�������3�����Is$��P6q��zs���*�����Q��Q������A
HO�Cz�}�8�@�P0U<&�*������h�%���=Yky������Z��0����������b�������~�PK�
�e2 �O PK pf�N Object 7/meta.xml���n�0F�>r���Pb�:t�����E�}C���� y��)���������'�]��;�i�V9���x��R�9y{}�7dW<d�x�����QY�F���U�t*��3�j���*VcK-��A5��%M]��r���������}��@���v�:��_��3���pLh!
"�����R#�T�Z_�F|�vqq&0�g�4BT�0�k�e�Yb�8w�3�nH����sa�'�I1��(^dN�D��Ym���0�� H�8H���Ru���&=���@������z�d%�8�?;f�+n���PK @�z" K PK pf�N Object 7/styles.xml��K�� ��9�KYc,+��)���LM� $SA�
�#� KA��PYJ����/�/c#w=�F��d�����bP
U_�����[�r�t����X�pe��?%7;'V�L�K�iE�a��
7�2-W���4 ��?�Y�<�������=����������C���.����T�h$� 1hZj���P?.����`<�~(��k��N'���������*���`���������G-��2��������nT'�7��ez��ru|�E�z�c= ,�����C����<}G����\G8{�3*��h��q9v��/�Lk����b�[�v9H�>�.;��mo�����[g��e�������k>�>|^�����p�Vm�T�p������3KW���J��2��S�
n�����k�MTYy�{L���o���j- .��Y��� 5 ��Z�W�kLa������
�����=N�����L�
����Ct�PK�P�� � PK pf�N Object 6/content.xml�]]s��}��p9Uy��k��:��V�r�����b ��"P^���@���������$w�v/b�8��{zh������8���M^�_���w~���j���/���g��wW���nn�Uv��Vw��lg��l��g�����~9����*m���L�Ys��.�]V�������_���}x?�n��v���X���z����x��N����5B��o��?4���2R���6vE^�����mw������>�W�f�k����
���vwu��Z�Y�uk��_�n�6{�X���n{���E����6�7��}��hV�i=��`[��z�z�5�M��Wt�?���_?����v������Vu�
s��WUu�����@��]z^��a��������j�zs�*-VG�W�SB3���1��w4=�D������8�Y�z������n�m�48���Y^6mZ>I����*�hQg��n����0����{�m������=�������v��1}cx��yv�k�����Y&������������A�\|�9�B7�]i ��k/��a��y�+-��.�3 ����r?s�,w�g��_!�<MU�����1�jwG��C�}w��4����3>s�� �Sl��_��Y79������C�����m�j�{���r��{�?�z6x��^����;6�l������7��u��[c�����rq��W��I_^`u����I�y������p��Nw���wF6Y��Ys������~6'(�2�/Wx�r��/�o�������lW5y�K�����y���97K�M�x���:���n*�W�5���@�����9�<3%-_��M�E�����:�0�'����}�����2a�������0�&/������Y��-P�-����$P�M��/h^���u6���������0fs�����`� n�,mg�v�>��o��>���m;�s=K��I�c�F�������g�6���:ovE�8+������&5Q��6_�;��sk�q7u���]�20�r6�6�����������]�1������n�gR�����]���g�N����i
�S�|��S
��P���`��r�����Mxc/�>�i����*f���8����HM���n9��=������J(R�� l�g����}j�� ���8b�g�j���F=���wZ��)������*������o��''~��V�/�h���r���Z?7�{��h������IK5��T���`��/������zq[g7_�����@�&��} ��H�f/��f������&�P��<�Msw��m'Ws��7����p7���ew������@dw����.�.�W,���\����Uk$z~��~��?��e����c��a�����H5;}��4=���Y����@���w��YmV����:�������o��<+�o��V������]�?�cy����r��~?�\_���7� ��U�rv�6C$kPW�G�x�(�Q=� �����y�$x0�I���J���XUU�����fu�1�<�`����L��2������������.�=���l������:��&�~8���G�{<}������������!�3#������/=����M��4Z�Q��|O�����2��&�q���3\����M�w������|�Y�A���9z�Uy��s���{�M�>�oP�����Q�����A���@��L��6e�5�m�o����(N��:�Ma&����p�c�a��|��^�Ee`����5jv�������m������q����i����������'�_� �v�_�]�P�>�8����/��Ng9�� ��r��~�����9�6m��N�/6f�a��u���>lB3k��9/��|�x�~�xL ���%�����x�}�x���]�����f��2O���F<���H�����M�skO�@��Q�>��}$'O���@y��4:p�C��M)�>�16����~��x�c6�����p���z��?��.9.7���k�
�U������ ��n6>9�)��D��$�$&M�KK�,cU����'G����xhH�9�"�w���I�-D��wBh&��I~5�4\#�4��Y�����'G)����ISS�v��o�,��UCpb6>9�nx���Im��b����bK�$3�=�hxs�����G4$z���x ��l�D�X"d�2@4��D�������
��$pq O��M��!�X�w�� �A��\��}�8D�<jK�,k
�}�<�-�f��a���Y�P�i�s+w���\��",�����G�q�������m���|�=G�B3A���M�e��]��vV^�f�12"�1B�u��7i�� ��������Z�UZ�*�f��a����b|�.Nh�_��������yo�H���nX�f�� �GX�1���;U1��*!4S�
������O�b<+7F�VLd��D�� ���oV� _;�y���Vr�g���S�T�� ���oV� _D�y/"�Vv�f�3;����B3A����z%$�����\�oe�x����E�Bh&��Y9!FX�XIO��*a}(&�e�za1��������4�wH�;�Rh&��IQ�#lA�V�hh�y)4�
��$t ����^^6R��c4k��(�!bh���
�����b���c4k�$\��(����nX�&=�c���Y+�F�� Rh&��I�c���Y+~`��>�B3E���M���ae�h���d���R
�u��7){ �+;�����r�Rh&��Y_��1�����U#��'Rh&��Y94���������<-��f�Sc�T,,f��X
"�U����N������B#d�.hX��y;|���X�1���<��!!4B�����oV� ��O��S`O�,k�� �>��)���oV� �/�ExVv�f�X�'�U��B#d�.hX�feB��B^�ge�h�������A
�����a���=1�y1���Y+���ORh&��Y��c���Y�1��b=��W�M�����a���=�0��x1���Yk�O�"�� !4B�����o��"�0��x����Y+��/�B��w���]�����!`���W��Z���7`����5H���:`a1��=�0��h�whe�hv��'^�I!4X�6���uB��7+{c��"������5��;�E�Rh���
���0��;�E����Y+��1o>B��7����Y������c�v�G�V��c^�-�F�Z]�����`E��WQ:��c4k��� �~�)kuA��7+{���^�������5��;���Bh0�Nx���Y�)�ge�x����>�!kuA��7+{��
^������g��7����;���.hH�f�����&�g�Vv�f�
g#���d�HY��Y9�����wXK�zL�|�h����y������(|��hOW#+;��S��-��B#d�.hX�fe�3M��LY�1��bg����@
�!Ix}H������(��-�����Z5F��y���A �k��Y�����c4k�8i�l$�F�Z]�����`����y)��04k�����C
���>;NhX�fe4�x��Y�1��*f#�����Y��Y�������Vv�e������H�����a����P��H�:EVv�e���
o6B�D4���
���ov,R��E���Y�B4��HM�hx�e4,~�<����B<V���i�;�!Ku��b6i^R�F ����Vv�f���F�:�H��T4,~�VI�����Ew���Y+v�Q��4Rh���
��$t�����Ew���Y+v�Q��4Bh���
��$�)#<��Y�1��bg��L#��������.hX�f�`���buL7R��c4k��4���F
���n��hX�fe�I�bu�5R���4kE���N#�cGV�XG4,~����P�:6�Y�1��bc
��O#�F�Z]������"cu4R��c4k������F
�����!��U�Aa���c�fe�x���e^7))4B�����oVK�+V�onM����C*V�K!,B�����lV� {)^�������/)^�)4��z 9�a���=�R����(��c<kE���!�F�Z]�����`�"����X�1�������uB
�!Q��*�hX�fe������I���Z��*��:!�F�Z]���������y)��c<kE���D"�W��^$NhX�f����
��x�X�1���KB�W�]
���NNhX�fe��H�������X��I8��].��V^o'4,~������uH���Z�Q����B�)E���:�!��U�Ra5{��f�X�1��b(�x�������������oVA���`�;�|�x�oE�`��xu��������*^-iee�hv��A�6���x��������`�]�������Z�������B��"^�X'4,~��R���c4k�j��WMS
�o���.hX�fe�ae�X���~���/�A����U�tB��7 �#���Z5�&���DRh������
�������!o�ne�h��\4����X�y�����MB�� A�e++;F�V�p�y.��D��;��a�����XG���(�+�Z�+s���\
�F��NhH�f� �K�xK����c4k�/H5�R!4����}3����o��!��\j�7��Y|�o.5��K,��������yI�wO������c4;�����RM�^'�y4,~�fY�_�h�W.������jV�b14�M��}�����oV.���������Y+~>�Y�B���wb5��X'4,~���Z��������Z�!�����B3A�����`�4�������Z}�u[���N��� �
��$t��E��-���b}w[����&���r���=,��m��h��xV�� ����'������b9����p�E���-�Y�o�+>1^���)��r��xH,�Y/��K�&�m��hgXf���5��g���St[��C�9ieD����6Ie��s��D����M���M��F����i�����6����o�������j�����Vw��lg��l��W�PK��� �J PK pf�N Object 6/meta.xml���n�0F�>r���Pb�:t�����E�}C���� y��)���������'�]��;�i�V9���x��R�9y{}�7dW<d�x�����QY�F���U�t*��3�j���*VcK-��A5��%M]��r���������}��@���v�:��_��3���pLh!
"�����R#�T�Z_�F|�vqq&0�g�4BT�0�k�e�Yb�8w�3�nH����sa�'�I1��(^dN�D��Ym���0�� H�8H���Ru���&=���@������z�d%�8�?;f�+n���PK @�z" K PK pf�N Object 6/styles.xml��K�� ��9�KYc,+��)���LM� $SA�
�#� KA��PYJ����/�/c#w=�F��d�����bP
U_�����[�r�t����X�pe��?%7;'V�L�K�iE�a��
7�2-W���4 ��?�Y�<�������=����������C���.����T�h$� 1hZj���P?.����`<�~(��k��N'���������*���`���������G-��2��������nT'�7��ez��ru|�E�z�c= ,�����C����<}G����\G8{�3*��h��q9v��/�Lk����b�[�v9H�>�.;��mo�����[g��e�������k>�>|^�����p�Vm�T�p������3KW���J��2��S�
n�����k�MTYy�{L���o���j- .��Y��� 5 ��Z�W�kLa������
�����=N�����L�
����Ct�PK�P�� � PK pf�N Object 11/content.xml��n����_ax�G�-Y2/����}h�E_i����D��c�_�CR�)E����� h8��~(���sE/����e����sV��|^����9�������~Or�)X~�p-����.��}^y�aH��Q��F��������-��y���5�M-�Y�%V�Z��/Y#��G���
�j���\��������I2��LI��yy���x��trO��x�Y�y�+��x��S�U��X ^�^�[a����pm��c��|�k�D��*^���R�pM~@|vnh�ax�b~x�����<��I�}�����5x5W���*���m����c�����V7���g�-��]�'s=���#��g��� /� ��/*M��W�7B��{dQ�d�����#?�
]����R���g�
�MKc���q�;f?�aB��^�����rW���E1�
�D�>��B���A?�����%v�"�=�dw��(�^$������B{v���\������(��l3�`�<eo`�N.����L�~��4��1��>P&��X������<v�4X�s��\�HNe�o��Q��+�Z
���� �Q2�m$wt��'����oN��+]�������}�H�U.�� %�P����>y<g
n *pN*D����p����`�6��.�����=�����:R����5�h��\,*��g_�A�j��Z�����O�03�Td�� "u�%?��gR����R�K����T�����DA��A��^�������[@5�)>�����d5�sz,�s E�k'��B���
���c,�1�>�#������#��;%j:N��4-tV�g������N�L� : 7��@e*.�����?W��l��������"��.F�������CE��%�>���8�M Z�^�a������?7�-��C%�.l���p1v�Z��Jv��(��L�~+�'�V��l!�)�����������U�$�Z1�c�?�<������ys��L��k�53���i>0����P�{�9q�(\qQb��=S�c���'�}�`��;�tu�m��D�>�u�
�}B�\��V�����+.���i�drNj-N�PoB��� ���Xm ��]��T���8����v���(��.�s��h]��!�����V�NI�r(H�|I�u��� G�V�z�b�n�U7�3f��[������k��<�'���������]�Z�K�yQp�
bU���� �t������ ��?����M�pH�Bv;&��l�]��ev����{n#dn�g�"�Py���x��P�HB��fK'&p�4�I�p,&u�(���u
g^�G5����Asc;wF�6pUD0-���[��Je=
M6�l��)�������Q��*������j�}�.�]�
���.9)&Q�Q�T�?��PI`~�DC6���3��g�'�~+/����$V*����4�9�0����U7\@�X��:��t����f{
�<�d�����N����!��1/�X�����k�U-�0s�=F1��$j���� '#�����bm`��������`�6pm����'Q SA�E���0�:�'�#�P�{e��_'m���)� ��\ j)�����������@H��C$�������y���,i7=>jbs=�f��h�v��,^�ifW�!��e����.�1*$$��'�����7�b�J`��0
�xeq�U��a�����`���@��B7��.IViYA~d�1xG��[�bD�+'Y�N�(N�bC�S(>d[Z�������qI`�byO��`� \
��l���A��Q>hs��8�4N�l�d�5'�������,�ShBii���1���`|�e�u����
�u�$v���0>����B���{�:z4�V���?(n�PK)�"Ey �( PK pf�N Object 11/meta.xml���n�0F�>r���Pb�:t�����E�}C���� y��)���������'�]��;�i�V9���x��R�9y{}�7dW<d�x�����QY�F���U�t*��3�j���*VcK-��A5��%M]��r���������}��@���v�:��_��3���pLh!
"�����R#�T�Z_�F|�vqq&0�g�4BT�0�k�e�Yb�8w�3�nH����sa�'�I1��(^dN�D��Ym���0�� H�8H���Ru���&=���@������z�d%�8�?;f�+n���PK @�z" K PK pf�N Object 11/styles.xml��K�� ��9�KYc,+��)���LM� $SA�
�#� KA��PYJ����/�/c#w=�F��d�����bP
U_�����[�r�t����X�pe��?%7;'V�L�K�iE�a��
7�2-W���4 ��?�Y�<�������=����������C���.����T�h$� 1hZj���P?.����`<�~(��k��N'���������*���`���������G-��2��������nT'�7��ez��ru|�E�z�c= ,�����C����<}G����\G8{�3*��h��q9v��/�Lk����b�[�v9H�>�.;��mo�����[g��e�������k>�>|^�����p�Vm�T�p������3KW���J��2��S�
n�����k�MTYy�{L���o���j- .��Y��� 5 ��Z�W�kLa������
�����=N�����L�
����Ct�PK�P�� � PK pf�N Object 12/content.xml��n�6�}��p�G�-�2�]tP����+-�2;�(�tl���!)��"{�t�M �����<�p*�� sAX�8\>�U�rR���?����?l���v;��u��C�+�d���w��X�����5C��u�J,�2[�W-���^kY�D�3�L��mj�Or*������t����9:N%V��T�|���uv�^�H��'J�O����������{�\�/H����N�����j�<�0�J��7�Z�K4U?�k�T�-��]�$zU�TL�����k�=��sC#������6m���JLV� ���\r��Se)���2N��fl��1���L�juC�_x���>�D?r"1������Y�qV�9
�0����K|�q� ��C�U�|���l�KtA&�GvH%$�.��*W-�=�k�e����� �
;������]A[����(*�yP�Px����z��v>��F���M���4��5?G�p: � ���n
���`r5���s�@�j�u�������e3�,�vN0m�Eg�(��R@�@��zmQ��/O����`�n�q�&2!"9�
��Q��#�Z
���� �A2�m$st������oNg��+]�����Y�T�J�\ ]N
"�����y�FxN\�����:5E!�=���e�]�����ys�C%�gu���%)8����b��$X�T�+�>��UX��$����laf���:5D��K~�-��� U�Z�|�����y��,�������2��
U��j�S|j�wX��jRe��cgO�WN�)�(�`�
�X c�},9F��e-�-G��v
T�8�{i�;��2�f�[3E[L�v�zBtn����T��-�-�.���j ��c��MW�D@���^}k)+l
��dK8V}C�q�� 4p���r��-��Swn*[�3��
�]����b�$�;�*����A0��+�V���-��M�B"SQ�n�������Q��b��*&�'�o<����M���M�L��s�51�_�i�1����P�y�9�j� pI�y���L��%��5&���m,�Q��o$��I��������/��#
e���B����lY~��N��s�Skq$�zc�nYi���jK���n��S�:g��n���>�s��NT��t�(�q�z�x{nW�ZM[U$��� �%q�d�j��R�8�B�t��7'���7�f�Y���f����n�������5�%�]S���Q���M�R�� �>�9��������1\������Y��}u����W���8{$L��-��y��]�K� ������W����>�\���6���`JI�~\��i�n��,�p(f��A2��:H^�G5��Q�Cs;�F��pqD0;O��\4�Je=Mn������.�����Q��*������b�m�����
��/8�GQ���)���_QI`~�K}6I�F�����e���f�h�+���j������`�M��{ �!q,��A-�Q:j���f;
�<�d�����N����!��1��zX��n��ye%�0s��G1��(j���� '���>cm`���u�����`"�npi�����R3
A�E���0�Z��%Q�{f��_Gm���1� ���� jE�/����e����u@H��C$.{�<��y?��,iW=>hbS=�rE~-?��`�&�5x��I���@/O���%���C�7S��F1\Ej�/�U����e� b+�mQ����B����(Z���X��W?����!���5�/�(N��b-�d���b���B�&S�n'i��*
S+�k
�kM�(u�.{a�~b��!��a�7���0t�4��x.���@^W0�d{
��4Y��r�'�e� �����$���8�A��������0OD��{���@m������i����PKh+S�� �( PK pf�N Object 12/meta.xml���n�0F�>r���Pb�:t�����E�}C���� y��)���������'�]��;�i�V9���x��R�9y{}�7dW<d�x�����QY�F���U�t*��3�j���*VcK-��A5��%M]��r���������}��@���v�:��_��3���pLh!
"�����R#�T�Z_�F|�vqq&0�g�4BT�0�k�e�Yb�8w�3�nH����sa�'�I1��(^dN�D��Ym���0�� H�8H���Ru���&=���@������z�d%�8�?;f�+n���PK @�z" K PK pf�N Object 12/styles.xml��K�� ��9�KYc,+��)���LM� $SA�
�#� KA��PYJ����/�/c#w=�F��d�����bP
U_�����[�r�t����X�pe��?%7;'V�L�K�iE�a��
7�2-W���4 ��?�Y�<�������=����������C���.����T�h$� 1hZj���P?.����`<�~(��k��N'���������*���`���������G-��2��������nT'�7��ez��ru|�E�z�c= ,�����C����<}G����\G8{�3*��h��q9v��/�Lk����b�[�v9H�>�.;��mo�����[g��e�������k>�>|^�����p�Vm�T�p������3KW���J��2��S�
n�����k�MTYy�{L���o���j- .��Y��� 5 ��Z�W�kLa������
�����=N�����L�
����Ct�PK�P�� � PK pf�N Object 13/content.xml�[Y���~��0���,Q�%���$;G��<�l���-+#�)����I�-��Ak��g��n�X��>�����c����)-�ha�g��h�������7�����<��6��*��>'EeD������R���=+V���
����-I�H�t����(�:e��%�.]�c5VX��d�f�f��K��
^�.��c��<3�����J��8fi��q���re���aqp�%&
������Q�W�Y&���$����lxsR��� ^��b�o\�����d4"��+��v����d��������u�W�+9 �O0)}��������PE,-G���uyJik�PT�k[�k���}��~`iE���d�p���P����a�'��"���m�����W����_?G;���9}��H^���I���g2RRV����/��-��mW����.f���� +�����a�O)9�����������nJ ��Lz�|N@�� ]�gI{
m�� ����G�%a����[�V�1���X�>��z�<%YS/Z�����9�� -W�t�t`�q�rbk�x{��Y��8w�!4���)�q8B��5iM�=_7 �Wj[�b�����u�W��������������i!MN@.N���������Xs���T���&lq�f'AxkD��X\Bl�R�g_��_`��D��4�Rg�V'G�2u���%�i%�Z�=i�a���3p��v���)�}�-��B���#r��5�!PPpqmR�9��wx����-�';��}^�/��yg���0>��L��q��\qo�,kx[B�J|�3:����Nv����UM�(�����qL
#"Y�b�3��@�6�\��#�2H^V�f���'b$�lxX��*����G��l�����&�3X~4�zy3q����oH�w)� ��k���Yx�@�e$R!����n ��@@��tu�U����4\_�m�xu���a�"�t���p�c��Y7w�u��BL/����ppw�q���j���s�����<x�F�����0�b0�R��I�__cj���y�um�|WA����� ��m}WA�r1y�A��a���*�������Y�\{?��g.2w�`E���E��������v mZ?(�����C�����pl�*z;"ZsH��XZ�*u�v�l��E3�N�{*��4*Q�9�C��`����!+4�Ti%"
Z1Ac,
�pH\<5��W����[�f��(���!����LMO���7}��5�h���5�N�<�f���g�?�2��o���Bm�������Lhm��V��yc���|_7�^ �� �Z�K\�J�N2l�+ �&������AF�E��1��f0���~�cFDZU��������d����PK�(����UN7c��j���v��j-\K��^8��)
���4\8���;J�*��A���<�@`%�(r{J</<��\x��V��6��oo9��l\V�,Y�c��S��da����~�Y�&~�g��)�L��{s%�H��mN�����Y��:ai<�������+&q�����L�,���=��-��u���0)1��.%�&q��5fJt�4-���X�O��<S�����q�s�;���s�=�$��s��>t�}���I|{��;��h��[�}���4�p�
��t�o�Ay�r���SNKY����^)��BU3�;���#8&L�@����9E���$6���!5���Jy��b�vFg@9S�m�������}k$@U��<B#�5"��D_d��Dv� t�p���G���Me3�`6"/d�&Q/�G��+K����/�4�6����6��(U�b�U�20�*��C�g�c =�$� \�{3�J�y
}#4�8:uN�OSy�}4 Ayo��\p�����35mW��o�^������t����=��r&��6
<=��0f��Z�oQ��O�$�����crK�h��c����B0���1�������k���/��u�:��X7�%7�@��M& �h������# �h����Ako�oU�CW�P��Ig���t��T@��
{�� �a���f^O��������G'���m[�^j jI�h�=� i����H�}�-
���N@����,/l�5i*�Z�B�>i(?l�Xc`���{(�S�sB;��5i*������XEy��E���d�l��[��N��e������5�i������4��o�A�-��o�0�
�K�z�P����x����U��o�A@��OP9��K� �`%e*���-{�U�W�s
��������o��
�w����2_�PK���t� �>