Optimising compactify_tuples()

Started by Thomas Munroover 5 years ago25 messages
#1Thomas Munro
thomas.munro@gmail.com
2 attachment(s)

Hi,

With [1]https://commitfest.postgresql.org/29/2669/ applied so that you can get crash recovery to be CPU bound
with a pgbench workload, we spend an awful lot of time in qsort(),
called from compactify_tuples(). I tried replacing that with a
specialised sort, and I got my test crash recovery time from ~55.5s
down to ~49.5s quite consistently.

I've attached a draft patch. The sort_utils.h thing (which I've
proposed before in another context where it didn't turn out to be
needed) probably needs better naming, and a few more parameterisations
so that it could entirely replace the existing copies of the algorithm
rather than adding yet one more. The header also contains some more
related algorithms that don't have a user right now; maybe I should
remove them.

While writing this email, I checked the archives and discovered that a
couple of other people have complained about this hot spot before and
proposed faster sorts already[2]/messages/by-id/3c6ff1d3a2ff429ee80d0031e6c69cb7@postgrespro.ru[3]/messages/by-id/546B89DE.7030906@vmware.com, and then there was a wide ranging
discussion of various options which ultimately seemed to conclude that
we should do what I'm now proposing ... and then it stalled. The
present work is independent; I wrote this for some other sorting
problem, and then tried it out here when perf told me that it was the
next thing to fix to make recovery go faster. So I guess what I'm
really providing here is the compelling workload and numbers that were
perhaps missing from that earlier thread, but I'm open to other
solutions too.

[1]: https://commitfest.postgresql.org/29/2669/
[2]: /messages/by-id/3c6ff1d3a2ff429ee80d0031e6c69cb7@postgrespro.ru
[3]: /messages/by-id/546B89DE.7030906@vmware.com

Attachments:

0001-Add-parameterized-sorting-searching-support.patchtext/x-patch; charset=US-ASCII; name=0001-Add-parameterized-sorting-searching-support.patchDownload
From 4f3e6ccab72b3241da9afebac311eb32400eaa61 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 17 Aug 2020 21:31:56 +1200
Subject: [PATCH 1/2] Add parameterized sorting/searching support.

Provide some simple sorting and searching algorithms for working with
sorted arrays, allowing for inlining of comparisons and object sizes.
---
 src/include/lib/sort_utils.h | 322 +++++++++++++++++++++++++++++++++++
 1 file changed, 322 insertions(+)
 create mode 100644 src/include/lib/sort_utils.h

diff --git a/src/include/lib/sort_utils.h b/src/include/lib/sort_utils.h
new file mode 100644
index 0000000000..8b92e00273
--- /dev/null
+++ b/src/include/lib/sort_utils.h
@@ -0,0 +1,322 @@
+/*-------------------------------------------------------------------------
+ *
+ * sort_utils.h
+ *
+ *	  Simple sorting-related algorithms specialized for arrays of
+ *	  paramaterized type, using inlined comparators.
+ *
+ * Copyright (c) 2020, PostgreSQL Global Development Group
+ *
+ * Usage notes:
+ *
+ *	  To generate functions specialized for a type, the following parameter
+ *	  macros should be #define'd before this file is included.
+ *
+ *	  - SA_PREFIX - prefix for all symbol names generated.
+ *	  - SA_ELEMENT_TYPE - type of the referenced elements
+ *	  - SA_DECLARE - if defined the functions and types are declared
+ *	  - SA_DEFINE - if defined the functions and types are defined
+ *	  - SA_SCOPE - scope (e.g. extern, static inline) for functions
+ *
+ *	  The following macros should be #define'd to enable functions:
+ *
+ *	  - SA_ENABLE_SORT - a PREFIX_sort function
+ *	  - SA_ENABLE_UNIQUE - a PREFIX_unique function
+ *	  - SA_ENABLE_BINARY_SEARCH - a PREFIX_binary_search function
+ *	  - SA_ENABLE_LOWER_BOUND - a PREFIX_lower_bound function
+ *
+ *	  The following are relevant only when SA_DEFINE is defined:
+ *
+ *	  - SA_COMPARE(a, b) - an expression to compare pointers to two values
+ *
+ * IDENTIFICATION
+ *		src/include/lib/sort_utils.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#define SA_MAKE_PREFIX(a) CppConcat(a,_)
+#define SA_MAKE_NAME(name) SA_MAKE_NAME_(SA_MAKE_PREFIX(SA_PREFIX),name)
+#define SA_MAKE_NAME_(a,b) CppConcat(a,b)
+
+/* function declarations */
+#define SA_SORT SA_MAKE_NAME(sort)
+#define SA_UNIQUE SA_MAKE_NAME(unique)
+#define SA_BINARY_SEARCH SA_MAKE_NAME(binary_search)
+#define SA_LOWER_BOUND SA_MAKE_NAME(lower_bound)
+
+#ifdef SA_DECLARE
+
+SA_SCOPE void SA_SORT(SA_ELEMENT_TYPE *first, size_t n);
+SA_SCOPE SA_ELEMENT_TYPE *SA_UNIQUE(SA_ELEMENT_TYPE *first,
+									SA_ELEMENT_TYPE *last);
+SA_SCOPE SA_ELEMENT_TYPE *SA_BINARY_SEARCH(SA_ELEMENT_TYPE *first,
+										   SA_ELEMENT_TYPE *last,
+										   SA_ELEMENT_TYPE *value);
+SA_SCOPE SA_ELEMENT_TYPE *SA_LOWER_BOUND(SA_ELEMENT_TYPE *first,
+										 SA_ELEMENT_TYPE *last,
+										 SA_ELEMENT_TYPE *value);
+
+#endif
+
+#ifdef SA_DEFINE
+
+#ifdef SA_ENABLE_SORT
+
+/* helper functions */
+#define SA_MED3 SA_MAKE_NAME(med3)
+#define SA_SWAP SA_MAKE_NAME(swap)
+#define SA_SWAPN SA_MAKE_NAME(swapn)
+
+static inline SA_ELEMENT_TYPE *
+SA_MED3(SA_ELEMENT_TYPE *a,
+		SA_ELEMENT_TYPE *b,
+		SA_ELEMENT_TYPE *c)
+{
+	return SA_COMPARE(a, b) < 0 ?
+		(SA_COMPARE(b, c) < 0 ? b : (SA_COMPARE(a, c) < 0 ? c : a))
+		: (SA_COMPARE(b, c) > 0 ? b : (SA_COMPARE(a, c) < 0 ? a : c));
+}
+
+static inline void
+SA_SWAP(SA_ELEMENT_TYPE *a, SA_ELEMENT_TYPE *b)
+{
+	SA_ELEMENT_TYPE tmp = *a;
+
+	*a = *b;
+	*b = tmp;
+}
+
+static inline void
+SA_SWAPN(SA_ELEMENT_TYPE *a, SA_ELEMENT_TYPE *b, size_t n)
+{
+	size_t		i;
+
+	for (i = 0; i < n; ++i)
+		SA_SWAP(&a[i], &b[i]);
+}
+
+/*
+ * Sort an array [first, last).  This is the same algorithm as
+ * src/port/qsort.c, parameterized at compile-time for comparison and element
+ * type.
+ */
+SA_SCOPE void
+SA_SORT(SA_ELEMENT_TYPE *first, size_t n)
+{
+	SA_ELEMENT_TYPE *a = first,
+			   *pa,
+			   *pb,
+			   *pc,
+			   *pd,
+			   *pl,
+			   *pm,
+			   *pn;
+	size_t		d1,
+				d2;
+	int			r,
+				presorted;
+
+loop:
+	if (n < 7)
+	{
+		for (pm = a + 1; pm < a + n; ++pm)
+			for (pl = pm; pl > a && SA_COMPARE(pl - 1, pl) > 0; --pl)
+				SA_SWAP(pl, pl - 1);
+		return;
+	}
+	presorted = 1;
+	for (pm = a + 1; pm < a + n; ++pm)
+	{
+		if (SA_COMPARE(pm - 1, pm) > 0)
+		{
+			presorted = 0;
+			break;
+		}
+	}
+	if (presorted)
+		return;
+	pm = a + (n / 2);
+	if (n > 7)
+	{
+		pl = a;
+		pn = a + (n - 1);
+		if (n > 40)
+		{
+			size_t		d = n / 8;
+
+			pl = SA_MED3(pl, pl + d, pl + 2 * d);
+			pm = SA_MED3(pm - d, pm, pm + d);
+			pn = SA_MED3(pn - 2 * d, pn - d, pn);
+		}
+		pm = SA_MED3(pl, pm, pn);
+	}
+	SA_SWAP(a, pm);
+	pa = pb = a + 1;
+	pc = pd = a + (n - 1);
+	for (;;)
+	{
+		while (pb <= pc && (r = SA_COMPARE(pb, a)) <= 0)
+		{
+			if (r == 0)
+			{
+				SA_SWAP(pa, pb);
+				++pa;
+			}
+			++pb;
+		}
+		while (pb <= pc && (r = SA_COMPARE(pc, a)) >= 0)
+		{
+			if (r == 0)
+			{
+				SA_SWAP(pc, pd);
+				--pd;
+			}
+			--pc;
+		}
+		if (pb > pc)
+			break;
+		SA_SWAP(pb, pc);
+		++pb;
+		--pc;
+	}
+	pn = a + n;
+	d1 = Min(pa - a, pb - pa);
+	SA_SWAPN(a, pb - d1, d1);
+	d1 = Min(pd - pc, pn - pd - 1);
+	SA_SWAPN(pb, pn - d1, d1);
+	d1 = pb - pa;
+	d2 = pd - pc;
+	if (d1 <= d2)
+	{
+		/* Recurse on left partition, then iterate on right partition */
+		if (d1 > 1)
+			SA_SORT(a, d1);
+		if (d2 > 1)
+		{
+			/* Iterate rather than recurse to save stack space */
+			/* SA_SORT(pn - d2, d2) */
+			a = pn - d2;
+			n = d2;
+			goto loop;
+		}
+	}
+	else
+	{
+		/* Recurse on right partition, then iterate on left partition */
+		if (d2 > 1)
+			SA_SORT(pn - d2, d2);
+		if (d1 > 1)
+		{
+			/* Iterate rather than recurse to save stack space */
+			/* SA_SORT(a, d1) */
+			n = d1;
+			goto loop;
+		}
+	}
+}
+#endif
+
+#ifdef SA_ENABLE_UNIQUE
+/*
+ * Remove duplicates from an array.  Return the new size.
+ */
+SA_SCOPE size_t
+SA_UNIQUE(SA_ELEMENT_TYPE *first, size_t n)
+{
+	SA_ELEMENT_TYPE *write_head;
+	SA_ELEMENT_TYPE *read_head;
+	SA_ELEMENT_TYPE *end = first + n;
+
+	if (n <= 1)
+		return n;
+
+	write_head = first;
+	read_head = first + 1;
+
+	while (read_head < end)
+	{
+		if (SA_COMPARE(read_head, write_head) != 0)
+			*++write_head = *read_head;
+		++read_head;
+	}
+	return (write_head - first) + 1;
+}
+#endif
+
+#ifdef SA_ENABLE_BINARY_SEARCH
+/*
+ * Find an element in the array of sorted values that is equal to a given
+ * value.  Return NULL if there is none.
+ */
+SA_SCOPE SA_ELEMENT_TYPE *
+SA_BINARY_SEARCH(SA_ELEMENT_TYPE *first, size_t n, SA_ELEMENT_TYPE *value)
+{
+	SA_ELEMENT_TYPE *lower = first;
+	SA_ELEMENT_TYPE *upper = first + n - 1;
+
+	while (lower <= upper)
+	{
+		SA_ELEMENT_TYPE *mid;
+		int			cmp;
+
+		mid = lower + (upper - lower) / 2;
+		cmp = SA_COMPARE(mid, value);
+		if (cmp < 0)
+			lower = mid + 1;
+		else if (cmp > 0)
+			upper = mid - 1;
+		else
+			return mid;
+	}
+
+	return NULL;
+}
+#endif
+
+#ifdef SA_ENABLE_LOWER_BOUND
+/*
+ * Find the first element in a sorted array that is not less than *value.
+ */
+SA_SCOPE SA_ELEMENT_TYPE *
+SA_LOWER_BOUND(SA_ELEMENT_TYPE *first, size_t n, SA_ELEMENT_TYPE *value)
+{
+	ptrdiff_t		count;
+
+	count = n;
+	while (count > 0)
+	{
+		SA_ELEMENT_TYPE *iter = first;
+		ptrdiff_t		step = count / 2;
+
+		iter += step;
+		if (SA_COMPARE(iter, value) < 0)
+		{
+			first = ++iter;
+			count -= step + 1;
+		}
+		else
+			count = step;
+	}
+	return first;
+}
+#endif
+
+#endif
+
+#undef SA_BINARY_SEARCH
+#undef SA_DECLARE
+#undef SA_DEFINE
+#undef SA_ENABLE_BINARY_SEARCH
+#undef SA_ENABLE_LOWER_BOUND
+#undef SA_ENABLE_SORT
+#undef SA_ENABLE_UNIQUE
+#undef SA_LOWER_BOUND
+#undef SA_MAKE_NAME
+#undef SA_MAKE_NAME
+#undef SA_MAKE_NAME_
+#undef SA_MAKE_PREFIX
+#undef SA_MED3
+#undef SA_SORT
+#undef SA_SWAP
+#undef SA_UNIQUE
-- 
2.20.1

0002-Use-specialized-sort-in-compactify_tuples.patchtext/x-patch; charset=US-ASCII; name=0002-Use-specialized-sort-in-compactify_tuples.patchDownload
From ffe50d5901a8f9e76200abf2c6ef43e250cc25f7 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 17 Aug 2020 20:59:32 +1200
Subject: [PATCH 2/2] Use specialized sort in compactify_tuples().

Micro-optimization:  by using a sort function with inlined comparison
and size we can improve page compaction performance.
---
 src/backend/storage/page/bufpage.c | 21 ++++++++++++---------
 1 file changed, 12 insertions(+), 9 deletions(-)

diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index d708117a40..33d783e5f6 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -421,13 +421,17 @@ typedef struct itemIdSortData
 } itemIdSortData;
 typedef itemIdSortData *itemIdSort;
 
-static int
-itemoffcompare(const void *itemidp1, const void *itemidp2)
-{
-	/* Sort in decreasing itemoff order */
-	return ((itemIdSort) itemidp2)->itemoff -
-		((itemIdSort) itemidp1)->itemoff;
-}
+/*
+ * Instantiate a fast itemid_sort() function that sorts in decreasing offset
+ * order.
+ */
+#define SA_PREFIX itemid
+#define SA_ELEMENT_TYPE itemIdSortData
+#define SA_DEFINE
+#define SA_SCOPE static
+#define SA_ENABLE_SORT
+#define SA_COMPARE(a, b) ((b)->itemoff - (a)->itemoff)
+#include "lib/sort_utils.h"
 
 /*
  * After removing or marking some line pointers unused, move the tuples to
@@ -441,8 +445,7 @@ compactify_tuples(itemIdSort itemidbase, int nitems, Page page)
 	int			i;
 
 	/* sort itemIdSortData array into decreasing itemoff order */
-	qsort((char *) itemidbase, nitems, sizeof(itemIdSortData),
-		  itemoffcompare);
+	itemid_sort(itemidbase, nitems);
 
 	upper = phdr->pd_special;
 	for (i = 0; i < nitems; i++)
-- 
2.20.1

In reply to: Thomas Munro (#1)
Re: Optimising compactify_tuples()

On Mon, Aug 17, 2020 at 4:01 AM Thomas Munro <thomas.munro@gmail.com> wrote:

While writing this email, I checked the archives and discovered that a
couple of other people have complained about this hot spot before and
proposed faster sorts already[2][3], and then there was a wide ranging
discussion of various options which ultimately seemed to conclude that
we should do what I'm now proposing ... and then it stalled.

I saw compactify_tuples() feature prominently in profiles when testing
the deduplication patch. We changed the relevant nbtdedup.c logic to
use a temp page rather than incrementally rewriting the authoritative
page in shared memory, which sidestepped the problem.

I definitely think that we should have something like this, though.
It's a relatively easy win. There are plenty of workloads that spend
lots of time on pruning.

--
Peter Geoghegan

#3Thomas Munro
thomas.munro@gmail.com
In reply to: Peter Geoghegan (#2)
4 attachment(s)
Re: Optimising compactify_tuples()

On Tue, Aug 18, 2020 at 6:53 AM Peter Geoghegan <pg@bowt.ie> wrote:

I definitely think that we should have something like this, though.
It's a relatively easy win. There are plenty of workloads that spend
lots of time on pruning.

Alright then, here's an attempt to flesh the idea out a bit more, and
replace the three other copies of qsort() while I'm at it.

Attachments:

0003-Use-sort_template.h-for-qsort-and-qsort_arg.patchtext/x-patch; charset=US-ASCII; name=0003-Use-sort_template.h-for-qsort-and-qsort_arg.patchDownload
From 66e33fbeb329bffe9d8dd2ef2c8594d20e1ad662 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Wed, 19 Aug 2020 19:34:45 +1200
Subject: [PATCH 3/4] Use sort_template.h for qsort() and qsort_arg().

Reduce duplication by using the new template.
---
 src/port/qsort.c     | 227 ++----------------------------------------
 src/port/qsort_arg.c | 228 ++-----------------------------------------
 2 files changed, 15 insertions(+), 440 deletions(-)

diff --git a/src/port/qsort.c b/src/port/qsort.c
index fa992e2081..7879e6cd56 100644
--- a/src/port/qsort.c
+++ b/src/port/qsort.c
@@ -1,229 +1,16 @@
 /*
  *	qsort.c: standard quicksort algorithm
- *
- *	Modifications from vanilla NetBSD source:
- *	  Add do ... while() macro fix
- *	  Remove __inline, _DIAGASSERTs, __P
- *	  Remove ill-considered "swap_cnt" switch to insertion sort,
- *	  in favor of a simple check for presorted input.
- *	  Take care to recurse on the smaller partition, to bound stack usage.
- *
- *	CAUTION: if you change this file, see also qsort_arg.c, gen_qsort_tuple.pl
- *
- *	src/port/qsort.c
- */
-
-/*	$NetBSD: qsort.c,v 1.13 2003/08/07 16:43:42 agc Exp $	*/
-
-/*-
- * Copyright (c) 1992, 1993
- *	The Regents of the University of California.  All rights reserved.
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *	  notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *	  notice, this list of conditions and the following disclaimer in the
- *	  documentation and/or other materials provided with the distribution.
- * 3. Neither the name of the University nor the names of its contributors
- *	  may be used to endorse or promote products derived from this software
- *	  without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
  */
 
 #include "c.h"
 
-
-static char *med3(char *a, char *b, char *c,
-				  int (*cmp) (const void *, const void *));
-static void swapfunc(char *, char *, size_t, int);
-
-/*
- * Qsort routine based on J. L. Bentley and M. D. McIlroy,
- * "Engineering a sort function",
- * Software--Practice and Experience 23 (1993) 1249-1265.
- *
- * We have modified their original by adding a check for already-sorted input,
- * which seems to be a win per discussions on pgsql-hackers around 2006-03-21.
- *
- * Also, we recurse on the smaller partition and iterate on the larger one,
- * which ensures we cannot recurse more than log(N) levels (since the
- * partition recursed to is surely no more than half of the input).  Bentley
- * and McIlroy explicitly rejected doing this on the grounds that it's "not
- * worth the effort", but we have seen crashes in the field due to stack
- * overrun, so that judgment seems wrong.
- */
-
-#define swapcode(TYPE, parmi, parmj, n) \
-do {		\
-	size_t i = (n) / sizeof (TYPE);			\
-	TYPE *pi = (TYPE *)(void *)(parmi);			\
-	TYPE *pj = (TYPE *)(void *)(parmj);			\
-	do {						\
-		TYPE	t = *pi;			\
-		*pi++ = *pj;				\
-		*pj++ = t;				\
-		} while (--i > 0);				\
-} while (0)
-
-#define SWAPINIT(a, es) swaptype = ((char *)(a) - (char *)0) % sizeof(long) || \
-	(es) % sizeof(long) ? 2 : (es) == sizeof(long)? 0 : 1
-
-static void
-swapfunc(char *a, char *b, size_t n, int swaptype)
-{
-	if (swaptype <= 1)
-		swapcode(long, a, b, n);
-	else
-		swapcode(char, a, b, n);
-}
-
-#define swap(a, b)						\
-	if (swaptype == 0) {					\
-		long t = *(long *)(void *)(a);			\
-		*(long *)(void *)(a) = *(long *)(void *)(b);	\
-		*(long *)(void *)(b) = t;			\
-	} else							\
-		swapfunc(a, b, es, swaptype)
-
-#define vecswap(a, b, n) if ((n) > 0) swapfunc(a, b, n, swaptype)
-
-static char *
-med3(char *a, char *b, char *c, int (*cmp) (const void *, const void *))
-{
-	return cmp(a, b) < 0 ?
-		(cmp(b, c) < 0 ? b : (cmp(a, c) < 0 ? c : a))
-		: (cmp(b, c) > 0 ? b : (cmp(a, c) < 0 ? a : c));
-}
-
-void
-pg_qsort(void *a, size_t n, size_t es, int (*cmp) (const void *, const void *))
-{
-	char	   *pa,
-			   *pb,
-			   *pc,
-			   *pd,
-			   *pl,
-			   *pm,
-			   *pn;
-	size_t		d1,
-				d2;
-	int			r,
-				swaptype,
-				presorted;
-
-loop:SWAPINIT(a, es);
-	if (n < 7)
-	{
-		for (pm = (char *) a + es; pm < (char *) a + n * es; pm += es)
-			for (pl = pm; pl > (char *) a && cmp(pl - es, pl) > 0;
-				 pl -= es)
-				swap(pl, pl - es);
-		return;
-	}
-	presorted = 1;
-	for (pm = (char *) a + es; pm < (char *) a + n * es; pm += es)
-	{
-		if (cmp(pm - es, pm) > 0)
-		{
-			presorted = 0;
-			break;
-		}
-	}
-	if (presorted)
-		return;
-	pm = (char *) a + (n / 2) * es;
-	if (n > 7)
-	{
-		pl = (char *) a;
-		pn = (char *) a + (n - 1) * es;
-		if (n > 40)
-		{
-			size_t		d = (n / 8) * es;
-
-			pl = med3(pl, pl + d, pl + 2 * d, cmp);
-			pm = med3(pm - d, pm, pm + d, cmp);
-			pn = med3(pn - 2 * d, pn - d, pn, cmp);
-		}
-		pm = med3(pl, pm, pn, cmp);
-	}
-	swap(a, pm);
-	pa = pb = (char *) a + es;
-	pc = pd = (char *) a + (n - 1) * es;
-	for (;;)
-	{
-		while (pb <= pc && (r = cmp(pb, a)) <= 0)
-		{
-			if (r == 0)
-			{
-				swap(pa, pb);
-				pa += es;
-			}
-			pb += es;
-		}
-		while (pb <= pc && (r = cmp(pc, a)) >= 0)
-		{
-			if (r == 0)
-			{
-				swap(pc, pd);
-				pd -= es;
-			}
-			pc -= es;
-		}
-		if (pb > pc)
-			break;
-		swap(pb, pc);
-		pb += es;
-		pc -= es;
-	}
-	pn = (char *) a + n * es;
-	d1 = Min(pa - (char *) a, pb - pa);
-	vecswap(a, pb - d1, d1);
-	d1 = Min(pd - pc, pn - pd - es);
-	vecswap(pb, pn - d1, d1);
-	d1 = pb - pa;
-	d2 = pd - pc;
-	if (d1 <= d2)
-	{
-		/* Recurse on left partition, then iterate on right partition */
-		if (d1 > es)
-			pg_qsort(a, d1 / es, es, cmp);
-		if (d2 > es)
-		{
-			/* Iterate rather than recurse to save stack space */
-			/* pg_qsort(pn - d2, d2 / es, es, cmp); */
-			a = pn - d2;
-			n = d2 / es;
-			goto loop;
-		}
-	}
-	else
-	{
-		/* Recurse on right partition, then iterate on left partition */
-		if (d2 > es)
-			pg_qsort(pn - d2, d2 / es, es, cmp);
-		if (d1 > es)
-		{
-			/* Iterate rather than recurse to save stack space */
-			/* pg_qsort(a, d1 / es, es, cmp); */
-			n = d1 / es;
-			goto loop;
-		}
-	}
-}
+#define ST_SORT pg_qsort
+#define ST_ELEMENT_TYPE_VOID
+#define ST_COMPARE_RUNTIME_POINTER
+#define ST_SCOPE
+#define ST_DECLARE
+#define ST_DEFINE
+#include "lib/sort_template.h"
 
 /*
  * qsort comparator wrapper for strcmp.
diff --git a/src/port/qsort_arg.c b/src/port/qsort_arg.c
index 6d54fbc2b4..fa7e11a3b8 100644
--- a/src/port/qsort_arg.c
+++ b/src/port/qsort_arg.c
@@ -1,226 +1,14 @@
 /*
  *	qsort_arg.c: qsort with a passthrough "void *" argument
- *
- *	Modifications from vanilla NetBSD source:
- *	  Add do ... while() macro fix
- *	  Remove __inline, _DIAGASSERTs, __P
- *	  Remove ill-considered "swap_cnt" switch to insertion sort,
- *	  in favor of a simple check for presorted input.
- *	  Take care to recurse on the smaller partition, to bound stack usage.
- *
- *	CAUTION: if you change this file, see also qsort.c, gen_qsort_tuple.pl
- *
- *	src/port/qsort_arg.c
- */
-
-/*	$NetBSD: qsort.c,v 1.13 2003/08/07 16:43:42 agc Exp $	*/
-
-/*-
- * Copyright (c) 1992, 1993
- *	The Regents of the University of California.  All rights reserved.
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *	  notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *	  notice, this list of conditions and the following disclaimer in the
- *	  documentation and/or other materials provided with the distribution.
- * 3. Neither the name of the University nor the names of its contributors
- *	  may be used to endorse or promote products derived from this software
- *	  without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
  */
 
 #include "c.h"
 
-
-static char *med3(char *a, char *b, char *c,
-				  qsort_arg_comparator cmp, void *arg);
-static void swapfunc(char *, char *, size_t, int);
-
-/*
- * Qsort routine based on J. L. Bentley and M. D. McIlroy,
- * "Engineering a sort function",
- * Software--Practice and Experience 23 (1993) 1249-1265.
- *
- * We have modified their original by adding a check for already-sorted input,
- * which seems to be a win per discussions on pgsql-hackers around 2006-03-21.
- *
- * Also, we recurse on the smaller partition and iterate on the larger one,
- * which ensures we cannot recurse more than log(N) levels (since the
- * partition recursed to is surely no more than half of the input).  Bentley
- * and McIlroy explicitly rejected doing this on the grounds that it's "not
- * worth the effort", but we have seen crashes in the field due to stack
- * overrun, so that judgment seems wrong.
- */
-
-#define swapcode(TYPE, parmi, parmj, n) \
-do {		\
-	size_t i = (n) / sizeof (TYPE);			\
-	TYPE *pi = (TYPE *)(void *)(parmi);			\
-	TYPE *pj = (TYPE *)(void *)(parmj);			\
-	do {						\
-		TYPE	t = *pi;			\
-		*pi++ = *pj;				\
-		*pj++ = t;				\
-		} while (--i > 0);				\
-} while (0)
-
-#define SWAPINIT(a, es) swaptype = ((char *)(a) - (char *)0) % sizeof(long) || \
-	(es) % sizeof(long) ? 2 : (es) == sizeof(long)? 0 : 1
-
-static void
-swapfunc(char *a, char *b, size_t n, int swaptype)
-{
-	if (swaptype <= 1)
-		swapcode(long, a, b, n);
-	else
-		swapcode(char, a, b, n);
-}
-
-#define swap(a, b)						\
-	if (swaptype == 0) {					\
-		long t = *(long *)(void *)(a);			\
-		*(long *)(void *)(a) = *(long *)(void *)(b);	\
-		*(long *)(void *)(b) = t;			\
-	} else							\
-		swapfunc(a, b, es, swaptype)
-
-#define vecswap(a, b, n) if ((n) > 0) swapfunc(a, b, n, swaptype)
-
-static char *
-med3(char *a, char *b, char *c, qsort_arg_comparator cmp, void *arg)
-{
-	return cmp(a, b, arg) < 0 ?
-		(cmp(b, c, arg) < 0 ? b : (cmp(a, c, arg) < 0 ? c : a))
-		: (cmp(b, c, arg) > 0 ? b : (cmp(a, c, arg) < 0 ? a : c));
-}
-
-void
-qsort_arg(void *a, size_t n, size_t es, qsort_arg_comparator cmp, void *arg)
-{
-	char	   *pa,
-			   *pb,
-			   *pc,
-			   *pd,
-			   *pl,
-			   *pm,
-			   *pn;
-	size_t		d1,
-				d2;
-	int			r,
-				swaptype,
-				presorted;
-
-loop:SWAPINIT(a, es);
-	if (n < 7)
-	{
-		for (pm = (char *) a + es; pm < (char *) a + n * es; pm += es)
-			for (pl = pm; pl > (char *) a && cmp(pl - es, pl, arg) > 0;
-				 pl -= es)
-				swap(pl, pl - es);
-		return;
-	}
-	presorted = 1;
-	for (pm = (char *) a + es; pm < (char *) a + n * es; pm += es)
-	{
-		if (cmp(pm - es, pm, arg) > 0)
-		{
-			presorted = 0;
-			break;
-		}
-	}
-	if (presorted)
-		return;
-	pm = (char *) a + (n / 2) * es;
-	if (n > 7)
-	{
-		pl = (char *) a;
-		pn = (char *) a + (n - 1) * es;
-		if (n > 40)
-		{
-			size_t		d = (n / 8) * es;
-
-			pl = med3(pl, pl + d, pl + 2 * d, cmp, arg);
-			pm = med3(pm - d, pm, pm + d, cmp, arg);
-			pn = med3(pn - 2 * d, pn - d, pn, cmp, arg);
-		}
-		pm = med3(pl, pm, pn, cmp, arg);
-	}
-	swap(a, pm);
-	pa = pb = (char *) a + es;
-	pc = pd = (char *) a + (n - 1) * es;
-	for (;;)
-	{
-		while (pb <= pc && (r = cmp(pb, a, arg)) <= 0)
-		{
-			if (r == 0)
-			{
-				swap(pa, pb);
-				pa += es;
-			}
-			pb += es;
-		}
-		while (pb <= pc && (r = cmp(pc, a, arg)) >= 0)
-		{
-			if (r == 0)
-			{
-				swap(pc, pd);
-				pd -= es;
-			}
-			pc -= es;
-		}
-		if (pb > pc)
-			break;
-		swap(pb, pc);
-		pb += es;
-		pc -= es;
-	}
-	pn = (char *) a + n * es;
-	d1 = Min(pa - (char *) a, pb - pa);
-	vecswap(a, pb - d1, d1);
-	d1 = Min(pd - pc, pn - pd - es);
-	vecswap(pb, pn - d1, d1);
-	d1 = pb - pa;
-	d2 = pd - pc;
-	if (d1 <= d2)
-	{
-		/* Recurse on left partition, then iterate on right partition */
-		if (d1 > es)
-			qsort_arg(a, d1 / es, es, cmp, arg);
-		if (d2 > es)
-		{
-			/* Iterate rather than recurse to save stack space */
-			/* qsort_arg(pn - d2, d2 / es, es, cmp, arg); */
-			a = pn - d2;
-			n = d2 / es;
-			goto loop;
-		}
-	}
-	else
-	{
-		/* Recurse on right partition, then iterate on left partition */
-		if (d2 > es)
-			qsort_arg(pn - d2, d2 / es, es, cmp, arg);
-		if (d1 > es)
-		{
-			/* Iterate rather than recurse to save stack space */
-			/* qsort_arg(a, d1 / es, es, cmp, arg); */
-			n = d1 / es;
-			goto loop;
-		}
-	}
-}
+#define ST_SORT qsort_arg
+#define ST_ELEMENT_TYPE_VOID
+#define ST_COMPARATOR_TYPE_NAME qsort_arg_comparator
+#define ST_COMPARE_RUNTIME_POINTER
+#define ST_COMPARE_ARG_TYPE void
+#define ST_SCOPE
+#define ST_DEFINE
+#include "lib/sort_template.h"
-- 
2.20.1

0004-Use-sort_template.h-for-qsort_tuple-and-qsort_ssup.patchtext/x-patch; charset=US-ASCII; name=0004-Use-sort_template.h-for-qsort_tuple-and-qsort_ssup.patchDownload
From 6a767c0e5ebf72aac2cf0883b1333a839a87567a Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Wed, 19 Aug 2020 20:25:12 +1200
Subject: [PATCH 4/4] Use sort_template.h for qsort_tuple() and qsort_ssup().

Replace the Perl code the previously generated specialized sort
functions with an instantiation of sort_template.h.
---
 src/backend/utils/sort/.gitignore         |   1 -
 src/backend/utils/sort/Makefile           |   3 -
 src/backend/utils/sort/gen_qsort_tuple.pl | 271 ----------------------
 src/backend/utils/sort/tuplesort.c        |  21 +-
 4 files changed, 20 insertions(+), 276 deletions(-)
 delete mode 100644 src/backend/utils/sort/.gitignore
 delete mode 100644 src/backend/utils/sort/gen_qsort_tuple.pl

diff --git a/src/backend/utils/sort/.gitignore b/src/backend/utils/sort/.gitignore
deleted file mode 100644
index f2958633e6..0000000000
--- a/src/backend/utils/sort/.gitignore
+++ /dev/null
@@ -1 +0,0 @@
-/qsort_tuple.c
diff --git a/src/backend/utils/sort/Makefile b/src/backend/utils/sort/Makefile
index 7ac3659261..39f132ef68 100644
--- a/src/backend/utils/sort/Makefile
+++ b/src/backend/utils/sort/Makefile
@@ -23,9 +23,6 @@ OBJS = \
 
 tuplesort.o: qsort_tuple.c
 
-qsort_tuple.c: gen_qsort_tuple.pl
-	$(PERL) $(srcdir)/gen_qsort_tuple.pl $< > $@
-
 include $(top_srcdir)/src/backend/common.mk
 
 maintainer-clean:
diff --git a/src/backend/utils/sort/gen_qsort_tuple.pl b/src/backend/utils/sort/gen_qsort_tuple.pl
deleted file mode 100644
index eb0f7c5814..0000000000
--- a/src/backend/utils/sort/gen_qsort_tuple.pl
+++ /dev/null
@@ -1,271 +0,0 @@
-#!/usr/bin/perl
-
-#
-# gen_qsort_tuple.pl
-#
-# This script generates specialized versions of the quicksort algorithm for
-# tuple sorting.  The quicksort code is derived from the NetBSD code.  The
-# code generated by this script runs significantly faster than vanilla qsort
-# when used to sort tuples.  This speedup comes from a number of places.
-# The major effects are (1) inlining simple tuple comparators is much faster
-# than jumping through a function pointer and (2) swap and vecswap operations
-# specialized to the particular data type of interest (in this case, SortTuple)
-# are faster than the generic routines.
-#
-#	Modifications from vanilla NetBSD source:
-#	  Add do ... while() macro fix
-#	  Remove __inline, _DIAGASSERTs, __P
-#	  Remove ill-considered "swap_cnt" switch to insertion sort,
-#	  in favor of a simple check for presorted input.
-#	  Take care to recurse on the smaller partition, to bound stack usage.
-#
-#     Instead of sorting arbitrary objects, we're always sorting SortTuples.
-#     Add CHECK_FOR_INTERRUPTS().
-#
-# CAUTION: if you change this file, see also qsort.c and qsort_arg.c
-#
-
-use strict;
-use warnings;
-
-my $SUFFIX;
-my $EXTRAARGS;
-my $EXTRAPARAMS;
-my $CMPPARAMS;
-
-emit_qsort_boilerplate();
-
-$SUFFIX      = 'tuple';
-$EXTRAARGS   = ', SortTupleComparator cmp_tuple, Tuplesortstate *state';
-$EXTRAPARAMS = ', cmp_tuple, state';
-$CMPPARAMS   = ', state';
-emit_qsort_implementation();
-
-$SUFFIX      = 'ssup';
-$EXTRAARGS   = ', SortSupport ssup';
-$EXTRAPARAMS = ', ssup';
-$CMPPARAMS   = ', ssup';
-print <<'EOM';
-
-#define cmp_ssup(a, b, ssup) \
-	ApplySortComparator((a)->datum1, (a)->isnull1, \
-						(b)->datum1, (b)->isnull1, ssup)
-
-EOM
-emit_qsort_implementation();
-
-sub emit_qsort_boilerplate
-{
-	print <<'EOM';
-/*
- * autogenerated by src/backend/utils/sort/gen_qsort_tuple.pl, do not edit!
- *
- * This file is included by tuplesort.c, rather than compiled separately.
- */
-
-/*	$NetBSD: qsort.c,v 1.13 2003/08/07 16:43:42 agc Exp $	*/
-
-/*-
- * Copyright (c) 1992, 1993
- *	The Regents of the University of California.  All rights reserved.
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *	  notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *	  notice, this list of conditions and the following disclaimer in the
- *	  documentation and/or other materials provided with the distribution.
- * 3. Neither the name of the University nor the names of its contributors
- *	  may be used to endorse or promote products derived from this software
- *	  without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- */
-
-/*
- * Qsort routine based on J. L. Bentley and M. D. McIlroy,
- * "Engineering a sort function",
- * Software--Practice and Experience 23 (1993) 1249-1265.
- *
- * We have modified their original by adding a check for already-sorted input,
- * which seems to be a win per discussions on pgsql-hackers around 2006-03-21.
- *
- * Also, we recurse on the smaller partition and iterate on the larger one,
- * which ensures we cannot recurse more than log(N) levels (since the
- * partition recursed to is surely no more than half of the input).  Bentley
- * and McIlroy explicitly rejected doing this on the grounds that it's "not
- * worth the effort", but we have seen crashes in the field due to stack
- * overrun, so that judgment seems wrong.
- */
-
-static void
-swapfunc(SortTuple *a, SortTuple *b, size_t n)
-{
-	do
-	{
-		SortTuple 	t = *a;
-		*a++ = *b;
-		*b++ = t;
-	} while (--n > 0);
-}
-
-#define swap(a, b)						\
-	do { 								\
-		SortTuple t = *(a);				\
-		*(a) = *(b);					\
-		*(b) = t;						\
-	} while (0)
-
-#define vecswap(a, b, n) if ((n) > 0) swapfunc(a, b, n)
-
-EOM
-
-	return;
-}
-
-sub emit_qsort_implementation
-{
-	print <<EOM;
-static SortTuple *
-med3_$SUFFIX(SortTuple *a, SortTuple *b, SortTuple *c$EXTRAARGS)
-{
-	return cmp_$SUFFIX(a, b$CMPPARAMS) < 0 ?
-		(cmp_$SUFFIX(b, c$CMPPARAMS) < 0 ? b :
-			(cmp_$SUFFIX(a, c$CMPPARAMS) < 0 ? c : a))
-		: (cmp_$SUFFIX(b, c$CMPPARAMS) > 0 ? b :
-			(cmp_$SUFFIX(a, c$CMPPARAMS) < 0 ? a : c));
-}
-
-static void
-qsort_$SUFFIX(SortTuple *a, size_t n$EXTRAARGS)
-{
-	SortTuple  *pa,
-			   *pb,
-			   *pc,
-			   *pd,
-			   *pl,
-			   *pm,
-			   *pn;
-	size_t		d1,
-				d2;
-	int			r,
-				presorted;
-
-loop:
-	CHECK_FOR_INTERRUPTS();
-	if (n < 7)
-	{
-		for (pm = a + 1; pm < a + n; pm++)
-			for (pl = pm; pl > a && cmp_$SUFFIX(pl - 1, pl$CMPPARAMS) > 0; pl--)
-				swap(pl, pl - 1);
-		return;
-	}
-	presorted = 1;
-	for (pm = a + 1; pm < a + n; pm++)
-	{
-		CHECK_FOR_INTERRUPTS();
-		if (cmp_$SUFFIX(pm - 1, pm$CMPPARAMS) > 0)
-		{
-			presorted = 0;
-			break;
-		}
-	}
-	if (presorted)
-		return;
-	pm = a + (n / 2);
-	if (n > 7)
-	{
-		pl = a;
-		pn = a + (n - 1);
-		if (n > 40)
-		{
-			size_t		d = (n / 8);
-
-			pl = med3_$SUFFIX(pl, pl + d, pl + 2 * d$EXTRAPARAMS);
-			pm = med3_$SUFFIX(pm - d, pm, pm + d$EXTRAPARAMS);
-			pn = med3_$SUFFIX(pn - 2 * d, pn - d, pn$EXTRAPARAMS);
-		}
-		pm = med3_$SUFFIX(pl, pm, pn$EXTRAPARAMS);
-	}
-	swap(a, pm);
-	pa = pb = a + 1;
-	pc = pd = a + (n - 1);
-	for (;;)
-	{
-		while (pb <= pc && (r = cmp_$SUFFIX(pb, a$CMPPARAMS)) <= 0)
-		{
-			if (r == 0)
-			{
-				swap(pa, pb);
-				pa++;
-			}
-			pb++;
-			CHECK_FOR_INTERRUPTS();
-		}
-		while (pb <= pc && (r = cmp_$SUFFIX(pc, a$CMPPARAMS)) >= 0)
-		{
-			if (r == 0)
-			{
-				swap(pc, pd);
-				pd--;
-			}
-			pc--;
-			CHECK_FOR_INTERRUPTS();
-		}
-		if (pb > pc)
-			break;
-		swap(pb, pc);
-		pb++;
-		pc--;
-	}
-	pn = a + n;
-	d1 = Min(pa - a, pb - pa);
-	vecswap(a, pb - d1, d1);
-	d1 = Min(pd - pc, pn - pd - 1);
-	vecswap(pb, pn - d1, d1);
-	d1 = pb - pa;
-	d2 = pd - pc;
-	if (d1 <= d2)
-	{
-		/* Recurse on left partition, then iterate on right partition */
-		if (d1 > 1)
-			qsort_$SUFFIX(a, d1$EXTRAPARAMS);
-		if (d2 > 1)
-		{
-			/* Iterate rather than recurse to save stack space */
-			/* qsort_$SUFFIX(pn - d2, d2$EXTRAPARAMS); */
-			a = pn - d2;
-			n = d2;
-			goto loop;
-		}
-	}
-	else
-	{
-		/* Recurse on right partition, then iterate on left partition */
-		if (d2 > 1)
-			qsort_$SUFFIX(pn - d2, d2$EXTRAPARAMS);
-		if (d1 > 1)
-		{
-			/* Iterate rather than recurse to save stack space */
-			/* qsort_$SUFFIX(a, d1$EXTRAPARAMS); */
-			n = d1;
-			goto loop;
-		}
-	}
-}
-EOM
-
-	return;
-}
diff --git a/src/backend/utils/sort/tuplesort.c b/src/backend/utils/sort/tuplesort.c
index 3c49476483..8119c41328 100644
--- a/src/backend/utils/sort/tuplesort.c
+++ b/src/backend/utils/sort/tuplesort.c
@@ -676,8 +676,27 @@ static void tuplesort_updatemax(Tuplesortstate *state);
  * reduces to ApplySortComparator(), that is single-key MinimalTuple sorts
  * and Datum sorts.
  */
-#include "qsort_tuple.c"
 
+#define ST_SORT qsort_tuple
+#define ST_ELEMENT_TYPE SortTuple
+#define ST_COMPARE_RUNTIME_POINTER
+#define ST_COMPARE_ARG_TYPE Tuplesortstate
+#define ST_CHECK_FOR_INTERRUPTS
+#define ST_SCOPE static
+#define ST_DECLARE
+#define ST_DEFINE
+#include "lib/sort_template.h"
+
+#define ST_SORT qsort_ssup
+#define ST_ELEMENT_TYPE SortTuple
+#define ST_COMPARE(a, b, ssup) \
+	ApplySortComparator((a)->datum1, (a)->isnull1, \
+						(b)->datum1, (b)->isnull1, (ssup))
+#define ST_COMPARE_ARG_TYPE SortSupportData
+#define ST_CHECK_FOR_INTERRUPTS
+#define ST_SCOPE static
+#define ST_DEFINE
+#include "lib/sort_template.h"
 
 /*
  *		tuplesort_begin_xxx
-- 
2.20.1

0001-Add-sort_template.h-for-making-fast-sort-functions.patchtext/x-patch; charset=US-ASCII; name=0001-Add-sort_template.h-for-making-fast-sort-functions.patchDownload
From 8d0b569fcf6141b622c63fc4bc102c762f01ca9e Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 17 Aug 2020 21:31:56 +1200
Subject: [PATCH 1/4] Add sort_template.h for making fast sort functions.

Move our qsort implementation into a header that can be used to
define specialized functions for better performance.

Discussion: https://postgr.es/m/CA%2BhUKGKMQFVpjr106gRhwk6R-nXv0qOcTreZuQzxgpHESAL6dw%40mail.gmail.com
---
 src/include/lib/sort_template.h | 428 ++++++++++++++++++++++++++++++++
 1 file changed, 428 insertions(+)
 create mode 100644 src/include/lib/sort_template.h

diff --git a/src/include/lib/sort_template.h b/src/include/lib/sort_template.h
new file mode 100644
index 0000000000..a279bcf959
--- /dev/null
+++ b/src/include/lib/sort_template.h
@@ -0,0 +1,428 @@
+/*-------------------------------------------------------------------------
+ *
+ * sort_template.h
+ *
+ *	  A template for a sort algorithm that supports varying degrees of
+ *	  specialization.
+ *
+ * Copyright (c) 2020, PostgreSQL Global Development Group
+ *
+ * Usage notes:
+ *
+ *	  To generate functions specialized for a type, the following parameter
+ *	  macros should be #define'd before this file is included.
+ *
+ *	  - ST_SORT - the name of a sort function to be generated
+ *	  - ST_ELEMENT_TYPE - type of the referenced elements
+ *	  - ST_DECLARE - if defined the functions and types are declared
+ *	  - ST_DEFINE - if defined the functions and types are defined
+ *	  - ST_SCOPE - scope (e.g. extern, static inline) for functions
+ *
+ *	  Instead of ST_ELEMENT_TYPE, ST_ELEMENT_TYPE_VOID can be defined.  Then
+ *	  the generated functions will automatically gain an "element_size"
+ *	  parameter.  This allows us to generate a traditional qsort function.
+ *
+ *	  One of the following macros must be defined, to show how to compare
+ *	  elements.  The first two options are arbitrary expressions depending
+ *	  on whether an extra pass-through argument is desired, and the third
+ *	  option should be defined if the sort function should receive a
+ *	  function pointer at runtime.
+ *
+ * 	  - ST_COMPARE(a, b) - a simple comparison expression
+ *	  - ST_COMPARE(a, b, arg) - variant that takes an extra argument
+ *	  - ST_COMPARE_RUNTIME_POINTER - sort function takes a function pointer
+ *
+ *	  To say that the comparator and therefore also sort function should
+ *	  receive an extra pass-through argument, specify the type of the
+ *	  argument.
+ *
+ *	  - ST_COMPARE_ARG_TYPE - type of extra argument
+ *
+ *	  The prototype of the generated sort function is:
+ *
+ *	  void ST_SORT(ST_ELEMENT_TYPE *data, size_t n,
+ *				   [size_t element_size,]
+ *				   [ST_SORT_compare_function compare,]
+ *				   [ST_COMPARE_ARG_TYPE *arg]);
+ *
+ *	  ST_SORT_compare_function is a function pointer of the following type:
+ *
+ *	  int (*)(const ST_ELEMENT_TYPE *a, const ST_ELEMENT_TYPE *b,
+ *			  [ST_COMPARE_ARG_TYPE *arg])
+ *
+ * HISTORY
+ *
+ *	  Modifications from vanilla NetBSD source:
+ *	  - Add do ... while() macro fix
+ *	  - Remove __inline, _DIAGASSERTs, __P
+ *	  - Remove ill-considered "swap_cnt" switch to insertion sort, in favor
+ *		of a simple check for presorted input.
+ *	  - Take care to recurse on the smaller partition, to bound stack usage
+ *	  - Convert into a header that can generate specialized functions
+ *
+ * IDENTIFICATION
+ *		src/include/lib/sort_template.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+/*	  $NetBSD: qsort.c,v 1.13 2003/08/07 16:43:42 agc Exp $   */
+
+/*-
+ * Copyright (c) 1992, 1993
+ *	  The Regents of the University of California.  All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *	  notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *	  notice, this list of conditions and the following disclaimer in the
+ *	  documentation and/or other materials provided with the distribution.
+ * 3. Neither the name of the University nor the names of its contributors
+ *	  may be used to endorse or promote products derived from this software
+ *	  without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+/*
+ * Qsort routine based on J. L. Bentley and M. D. McIlroy,
+ * "Engineering a sort function",
+ * Software--Practice and Experience 23 (1993) 1249-1265.
+ *
+ * We have modified their original by adding a check for already-sorted
+ * input, which seems to be a win per discussions on pgsql-hackers around
+ * 2006-03-21.
+ *
+ * Also, we recurse on the smaller partition and iterate on the larger one,
+ * which ensures we cannot recurse more than log(N) levels (since the
+ * partition recursed to is surely no more than half of the input).  Bentley
+ * and McIlroy explicitly rejected doing this on the grounds that it's "not
+ * worth the effort", but we have seen crashes in the field due to stack
+ * overrun, so that judgment seems wrong.
+ */
+
+#define ST_MAKE_PREFIX(a) CppConcat(a,_)
+#define ST_MAKE_NAME(a,b) ST_MAKE_NAME_(ST_MAKE_PREFIX(a),b)
+#define ST_MAKE_NAME_(a,b) CppConcat(a,b)
+
+/*
+ * If the element type is void, we'll also need an element_size argument
+ * because we don't know the size.
+ */
+#ifdef ST_ELEMENT_TYPE_VOID
+#define ST_ELEMENT_TYPE void
+#define ST_SORT_PROTO_SIZE , size_t element_size
+#define ST_SORT_INVOKE_SIZE , element_size
+#else
+#define ST_SORT_PROTO_SIZE
+#define ST_SORT_INVOKE_SIZE
+#endif
+
+/*
+ * If the user wants to be able to pass in compare functions at runtime,
+ * we'll need to make that an argument of the sort and med3 functions.
+ */
+#ifdef ST_COMPARE_RUNTIME_POINTER
+/*
+ * The type of the comparator function pointer that ST_SORT will take, unless
+ * you've already declared a type name manually and want to use that instead of
+ * having a new one defined.
+ */
+#ifndef ST_COMPARATOR_TYPE_NAME
+#define ST_COMPARATOR_TYPE_NAME ST_MAKE_NAME(ST_SORT, compare_function)
+#endif
+#define ST_COMPARE compare
+#ifndef ST_COMPARE_ARG_TYPE
+#define ST_SORT_PROTO_COMPARE , ST_COMPARATOR_TYPE_NAME compare
+#define ST_SORT_INVOKE_COMPARE , compare
+#else
+#define ST_SORT_PROTO_COMPARE , ST_COMPARATOR_TYPE_NAME compare
+#define ST_SORT_INVOKE_COMPARE , compare
+#endif
+#else
+#define ST_SORT_PROTO_COMPARE
+#define ST_SORT_INVOKE_COMPARE
+#endif
+
+/*
+ * If the user wants to use a compare function or expression that takes an
+ * extra argument, we'll need to make that an argument of the sort, compare and
+ * med3 functions.
+ */
+#ifdef ST_COMPARE_ARG_TYPE
+#define ST_SORT_PROTO_ARG , ST_COMPARE_ARG_TYPE *arg
+#define ST_SORT_INVOKE_ARG , arg
+#else
+#define ST_SORT_PROTO_ARG
+#define ST_SORT_INVOKE_ARG
+#endif
+
+#ifdef ST_DECLARE
+
+#ifdef ST_COMPARE_RUNTIME_POINTER
+typedef int (*ST_COMPARATOR_TYPE_NAME)(const ST_ELEMENT_TYPE *,
+									   const ST_ELEMENT_TYPE *
+									   ST_SORT_PROTO_ARG);
+#endif
+
+/* Declare the sort function.  Note optional arguments at end. */
+ST_SCOPE void ST_SORT(ST_ELEMENT_TYPE *first, size_t n
+					  ST_SORT_PROTO_SIZE
+					  ST_SORT_PROTO_COMPARE
+					  ST_SORT_PROTO_ARG);
+
+#endif
+
+#ifdef ST_DEFINE
+
+/* sort private helper functions */
+#define ST_MED3 ST_MAKE_NAME(ST_SORT, med3)
+#define ST_SWAP ST_MAKE_NAME(ST_SORT, swap)
+#define ST_SWAPN ST_MAKE_NAME(ST_SORT, swapn)
+
+/* Users expecting to run very large sorts may need them to be interruptible. */
+#ifdef ST_CHECK_FOR_INTERRUPTS
+#define DO_CHECK_FOR_INTERRUPTS() CHECK_FOR_INTERRUPTS()
+#else
+#define DO_CHECK_FOR_INTERRUPTS()
+#endif
+
+/*
+ * Create wrapper macros that know how to invoke compare, med3 and sort with
+ * the right arguments.
+ */
+#ifdef ST_COMPARE_RUNTIME_POINTER
+#define DO_COMPARE(a_, b_) ST_COMPARE((a_), (b_) ST_SORT_INVOKE_ARG)
+#elif defined(ST_COMPARE_ARG_TYPE)
+#define DO_COMPARE(a_, b_) ST_COMPARE((a_), (b_), arg)
+#else
+#define DO_COMPARE(a_, b_) ST_COMPARE((a_), (b_))
+#endif
+#define DO_MED3(a_, b_, c_)												\
+	ST_MED3((a_), (b_), (c_)											\
+			ST_SORT_INVOKE_COMPARE										\
+			ST_SORT_INVOKE_ARG)
+#define DO_SORT(a_, n_)													\
+	ST_SORT((a_), (n_)													\
+			ST_SORT_INVOKE_SIZE											\
+			ST_SORT_INVOKE_COMPARE										\
+			ST_SORT_INVOKE_ARG)
+
+/*
+ * If we're working with void pointers, we'll use pointer arithmetic based on
+ * uint8, and use the runtime element_size to step through the array and swap
+ * elements.  Otherwise we'll work with ST_ELEMENT_TYPE.
+ */
+#ifndef ST_ELEMENT_TYPE_VOID
+#define ST_POINTER_TYPE ST_ELEMENT_TYPE
+#define ST_POINTER_STEP 1
+#define DO_SWAPN(a_, b_, n_) ST_SWAPN((a_), (b_), (n_))
+#define DO_SWAP(a_, b_) ST_SWAP((a_), (b_))
+#else
+#define ST_POINTER_TYPE uint8
+#define ST_POINTER_STEP element_size
+#define DO_SWAPN(a_, b_, n_) ST_SWAPN((a_), (b_), (n_))
+#define DO_SWAP(a_, b_) DO_SWAPN((a_), (b_), element_size)
+#endif
+
+/*
+ * Find the median of three values.  Currently, performance seems to be best
+ * if the the comparator is inlined here, but the med3 function is not inlined
+ * in the qsort function.
+ */
+static pg_noinline ST_ELEMENT_TYPE *
+ST_MED3(ST_ELEMENT_TYPE *a,
+		ST_ELEMENT_TYPE *b,
+		ST_ELEMENT_TYPE *c
+		ST_SORT_PROTO_COMPARE
+		ST_SORT_PROTO_ARG)
+{
+		return DO_COMPARE(a, b) < 0 ?
+		(DO_COMPARE(b, c) < 0 ? b : (DO_COMPARE(a, c) < 0 ? c : a))
+		: (DO_COMPARE(b, c) > 0 ? b : (DO_COMPARE(a, c) < 0 ? a : c));
+}
+
+static inline void
+ST_SWAP(ST_POINTER_TYPE *a, ST_POINTER_TYPE *b)
+{
+	ST_POINTER_TYPE tmp = *a;
+
+	*a = *b;
+	*b = tmp;
+}
+
+static inline void
+ST_SWAPN(ST_POINTER_TYPE *a, ST_POINTER_TYPE *b, size_t n)
+{
+	for (size_t i = 0; i < n; ++i)
+		ST_SWAP(&a[i], &b[i]);
+}
+
+/*
+ * Sort an array.
+ */
+ST_SCOPE void
+ST_SORT(ST_ELEMENT_TYPE *data, size_t n
+		ST_SORT_PROTO_SIZE
+		ST_SORT_PROTO_COMPARE
+		ST_SORT_PROTO_ARG)
+{
+	ST_POINTER_TYPE *a = (ST_POINTER_TYPE *) data,
+			   *pa,
+			   *pb,
+			   *pc,
+			   *pd,
+			   *pl,
+			   *pm,
+			   *pn;
+	size_t		d1,
+				d2;
+	int			r,
+				presorted;
+
+loop:
+	DO_CHECK_FOR_INTERRUPTS();
+	if (n < 7)
+	{
+		for (pm = a + ST_POINTER_STEP; pm < a + n * ST_POINTER_STEP;
+			 pm += ST_POINTER_STEP)
+			for (pl = pm; pl > a && DO_COMPARE(pl - ST_POINTER_STEP, pl) > 0;
+				 pl -= ST_POINTER_STEP)
+				DO_SWAP(pl, pl - ST_POINTER_STEP);
+		return;
+	}
+	presorted = 1;
+	for (pm = a + ST_POINTER_STEP; pm < a + n * ST_POINTER_STEP;
+		 pm += ST_POINTER_STEP)
+	{
+		DO_CHECK_FOR_INTERRUPTS();
+		if (DO_COMPARE(pm - ST_POINTER_STEP, pm) > 0)
+		{
+			presorted = 0;
+			break;
+		}
+	}
+	if (presorted)
+		return;
+	pm = a + (n / 2) * ST_POINTER_STEP;
+	if (n > 7)
+	{
+		pl = a;
+		pn = a + (n - 1) * ST_POINTER_STEP;
+		if (n > 40)
+		{
+			size_t		d = (n / 8) * ST_POINTER_STEP;
+
+			pl = DO_MED3(pl, pl + d, pl + 2 * d);
+			pm = DO_MED3(pm - d, pm, pm + d);
+			pn = DO_MED3(pn - 2 * d, pn - d, pn);
+		}
+		pm = DO_MED3(pl, pm, pn);
+	}
+	DO_SWAP(a, pm);
+	pa = pb = a + ST_POINTER_STEP;
+	pc = pd = a + (n - 1) * ST_POINTER_STEP;
+	for (;;)
+	{
+		while (pb <= pc && (r = DO_COMPARE(pb, a)) <= 0)
+		{
+			if (r == 0)
+			{
+				DO_SWAP(pa, pb);
+				pa += ST_POINTER_STEP;
+			}
+			pb += ST_POINTER_STEP;
+			DO_CHECK_FOR_INTERRUPTS();
+		}
+		while (pb <= pc && (r = DO_COMPARE(pc, a)) >= 0)
+		{
+			if (r == 0)
+			{
+				DO_SWAP(pc, pd);
+				pd -= ST_POINTER_STEP;
+			}
+			pc -= ST_POINTER_STEP;
+			DO_CHECK_FOR_INTERRUPTS();
+		}
+		if (pb > pc)
+			break;
+		DO_SWAP(pb, pc);
+		pb += ST_POINTER_STEP;
+		pc -= ST_POINTER_STEP;
+	}
+	pn = a + n * ST_POINTER_STEP;
+	d1 = Min(pa - a, pb - pa);
+	DO_SWAPN(a, pb - d1, d1);
+	d1 = Min(pd - pc, pn - pd - ST_POINTER_STEP);
+	DO_SWAPN(pb, pn - d1, d1);
+	d1 = pb - pa;
+	d2 = pd - pc;
+	if (d1 <= d2)
+	{
+		/* Recurse on left partition, then iterate on right partition */
+		if (d1 > ST_POINTER_STEP)
+			DO_SORT(a, d1 / ST_POINTER_STEP);
+		if (d2 > ST_POINTER_STEP)
+		{
+			/* Iterate rather than recurse to save stack space */
+			/* DO_SORT(pn - d2, d2 / ST_POINTER_STEP) */
+			a = pn - d2;
+			n = d2 / ST_POINTER_STEP;
+			goto loop;
+		}
+	}
+	else
+	{
+		/* Recurse on right partition, then iterate on left partition */
+		if (d2 > ST_POINTER_STEP)
+			DO_SORT(pn - d2, d2 / ST_POINTER_STEP);
+		if (d1 > ST_POINTER_STEP)
+		{
+			/* Iterate rather than recurse to save stack space */
+			/* DO_SORT(a, d1 / ST_POINTER_STEP) */
+			n = d1 / ST_POINTER_STEP;
+			goto loop;
+		}
+	}
+}
+#endif
+
+#undef DO_COMPARE
+#undef DO_MED3
+#undef DO_SORT
+#undef DO_SWAP
+#undef DO_SWAPN
+#undef ST_COMPARATOR_TYPE_NAME
+#undef ST_COMPARE
+#undef ST_COMPARE_ARG_TYPE
+#undef ST_COMPARE_RUNTIME_POINTER
+#undef ST_ELEMENT_TYPE
+#undef ST_ELEMENT_TYPE_VOID
+#undef ST_MAKE_NAME
+#undef ST_MAKE_NAME_
+#undef ST_MAKE_PREFIX
+#undef ST_MED3
+#undef ST_POINTER_STEP
+#undef ST_POINTER_TYPE
+#undef ST_SORT
+#undef ST_SORT_INVOKE_ARG
+#undef ST_SORT_INVOKE_COMPARE
+#undef ST_SORT_INVOKE_SIZE
+#undef ST_SORT_PROTO_ARG
+#undef ST_SORT_PROTO_COMPARE
+#undef ST_SORT_PROTO_SIZE
+#undef ST_SWAP
+#undef ST_SWAPN
-- 
2.20.1

0002-Use-sort_template.h-for-compactify_tuples.patchtext/x-patch; charset=US-ASCII; name=0002-Use-sort_template.h-for-compactify_tuples.patchDownload
From fc58e1ad97a1eaad7bd1277a9d493af239cc77ef Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 17 Aug 2020 20:59:32 +1200
Subject: [PATCH 2/4] Use sort_template.h for compactify_tuples().

Since compactify_tuples() is called often, a specialized sort function
for sorting tuple offsets can give measurable speed-up.

Discussion: https://postgr.es/m/CA%2BhUKGKMQFVpjr106gRhwk6R-nXv0qOcTreZuQzxgpHESAL6dw%40mail.gmail.com
---
 src/backend/storage/page/bufpage.c | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)

diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index d708117a40..f81fa8cf3d 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -421,13 +421,13 @@ typedef struct itemIdSortData
 } itemIdSortData;
 typedef itemIdSortData *itemIdSort;
 
-static int
-itemoffcompare(const void *itemidp1, const void *itemidp2)
-{
-	/* Sort in decreasing itemoff order */
-	return ((itemIdSort) itemidp2)->itemoff -
-		((itemIdSort) itemidp1)->itemoff;
-}
+/* Create a specialized sort function for descending offset order. */
+#define ST_SORT qsort_itemoff
+#define ST_ELEMENT_TYPE itemIdSortData
+#define ST_COMPARE(a, b) ((b)->itemoff - (a)->itemoff)
+#define ST_SCOPE static
+#define ST_DEFINE
+#include "lib/sort_template.h"
 
 /*
  * After removing or marking some line pointers unused, move the tuples to
@@ -441,8 +441,7 @@ compactify_tuples(itemIdSort itemidbase, int nitems, Page page)
 	int			i;
 
 	/* sort itemIdSortData array into decreasing itemoff order */
-	qsort((char *) itemidbase, nitems, sizeof(itemIdSortData),
-		  itemoffcompare);
+	qsort_itemoff(itemidbase, nitems);
 
 	upper = phdr->pd_special;
 	for (i = 0; i < nitems; i++)
-- 
2.20.1

#4Thomas Munro
thomas.munro@gmail.com
In reply to: Thomas Munro (#3)
4 attachment(s)
Re: Optimising compactify_tuples()

On Wed, Aug 19, 2020 at 11:41 PM Thomas Munro <thomas.munro@gmail.com> wrote:

On Tue, Aug 18, 2020 at 6:53 AM Peter Geoghegan <pg@bowt.ie> wrote:

I definitely think that we should have something like this, though.
It's a relatively easy win. There are plenty of workloads that spend
lots of time on pruning.

Alright then, here's an attempt to flesh the idea out a bit more, and
replace the three other copies of qsort() while I'm at it.

I fixed up the copyright messages, and removed some stray bits of
build scripting relating to the Perl-generated file. Added to
commitfest.

Attachments:

v2-0001-Add-sort_template.h-for-making-fast-sort-function.patchtext/x-patch; charset=US-ASCII; name=v2-0001-Add-sort_template.h-for-making-fast-sort-function.patchDownload
From 940100ea1b2021e434976f6ce10aae75dd265d26 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 17 Aug 2020 21:31:56 +1200
Subject: [PATCH v2 1/5] Add sort_template.h for making fast sort functions.

Move our qsort implementation into a header that can be used to
define specialized functions for better performance.

Discussion: https://postgr.es/m/CA%2BhUKGKMQFVpjr106gRhwk6R-nXv0qOcTreZuQzxgpHESAL6dw%40mail.gmail.com
---
 src/include/lib/sort_template.h | 431 ++++++++++++++++++++++++++++++++
 1 file changed, 431 insertions(+)
 create mode 100644 src/include/lib/sort_template.h

diff --git a/src/include/lib/sort_template.h b/src/include/lib/sort_template.h
new file mode 100644
index 0000000000..0e163d4d8a
--- /dev/null
+++ b/src/include/lib/sort_template.h
@@ -0,0 +1,431 @@
+/*-------------------------------------------------------------------------
+ *
+ * sort_template.h
+ *
+ *	  A template for a sort algorithm that supports varying degrees of
+ *	  specialization.
+ *
+ * Copyright (c) 2020, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1992-1994, Regents of the University of California
+ *
+ * Usage notes:
+ *
+ *	  To generate functions specialized for a type, the following parameter
+ *	  macros should be #define'd before this file is included.
+ *
+ *	  - ST_SORT - the name of a sort function to be generated
+ *	  - ST_ELEMENT_TYPE - type of the referenced elements
+ *	  - ST_DECLARE - if defined the functions and types are declared
+ *	  - ST_DEFINE - if defined the functions and types are defined
+ *	  - ST_SCOPE - scope (e.g. extern, static inline) for functions
+ *
+ *	  Instead of ST_ELEMENT_TYPE, ST_ELEMENT_TYPE_VOID can be defined.  Then
+ *	  the generated functions will automatically gain an "element_size"
+ *	  parameter.  This allows us to generate a traditional qsort function.
+ *
+ *	  One of the following macros must be defined, to show how to compare
+ *	  elements.  The first two options are arbitrary expressions depending
+ *	  on whether an extra pass-through argument is desired, and the third
+ *	  option should be defined if the sort function should receive a
+ *	  function pointer at runtime.
+ *
+ * 	  - ST_COMPARE(a, b) - a simple comparison expression
+ *	  - ST_COMPARE(a, b, arg) - variant that takes an extra argument
+ *	  - ST_COMPARE_RUNTIME_POINTER - sort function takes a function pointer
+ *
+ *	  To say that the comparator and therefore also sort function should
+ *	  receive an extra pass-through argument, specify the type of the
+ *	  argument.
+ *
+ *	  - ST_COMPARE_ARG_TYPE - type of extra argument
+ *
+ *	  The prototype of the generated sort function is:
+ *
+ *	  void ST_SORT(ST_ELEMENT_TYPE *data, size_t n,
+ *				   [size_t element_size,]
+ *				   [ST_SORT_compare_function compare,]
+ *				   [ST_COMPARE_ARG_TYPE *arg]);
+ *
+ *	  ST_SORT_compare_function is a function pointer of the following type:
+ *
+ *	  int (*)(const ST_ELEMENT_TYPE *a, const ST_ELEMENT_TYPE *b,
+ *			  [ST_COMPARE_ARG_TYPE *arg])
+ *
+ * HISTORY
+ *
+ *	  Modifications from vanilla NetBSD source:
+ *	  - Add do ... while() macro fix
+ *	  - Remove __inline, _DIAGASSERTs, __P
+ *	  - Remove ill-considered "swap_cnt" switch to insertion sort, in favor
+ *		of a simple check for presorted input.
+ *	  - Take care to recurse on the smaller partition, to bound stack usage
+ *	  - Convert into a header that can generate specialized functions
+ *
+ * IDENTIFICATION
+ *		src/include/lib/sort_template.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+/*	  $NetBSD: qsort.c,v 1.13 2003/08/07 16:43:42 agc Exp $   */
+
+/*-
+ * Copyright (c) 1992, 1993
+ *	  The Regents of the University of California.  All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *	  notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *	  notice, this list of conditions and the following disclaimer in the
+ *	  documentation and/or other materials provided with the distribution.
+ * 3. Neither the name of the University nor the names of its contributors
+ *	  may be used to endorse or promote products derived from this software
+ *	  without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+/*
+ * Qsort routine based on J. L. Bentley and M. D. McIlroy,
+ * "Engineering a sort function",
+ * Software--Practice and Experience 23 (1993) 1249-1265.
+ *
+ * We have modified their original by adding a check for already-sorted
+ * input, which seems to be a win per discussions on pgsql-hackers around
+ * 2006-03-21.
+ *
+ * Also, we recurse on the smaller partition and iterate on the larger one,
+ * which ensures we cannot recurse more than log(N) levels (since the
+ * partition recursed to is surely no more than half of the input).  Bentley
+ * and McIlroy explicitly rejected doing this on the grounds that it's "not
+ * worth the effort", but we have seen crashes in the field due to stack
+ * overrun, so that judgment seems wrong.
+ */
+
+#define ST_MAKE_PREFIX(a) CppConcat(a,_)
+#define ST_MAKE_NAME(a,b) ST_MAKE_NAME_(ST_MAKE_PREFIX(a),b)
+#define ST_MAKE_NAME_(a,b) CppConcat(a,b)
+
+/*
+ * If the element type is void, we'll also need an element_size argument
+ * because we don't know the size.
+ */
+#ifdef ST_ELEMENT_TYPE_VOID
+#define ST_ELEMENT_TYPE void
+#define ST_SORT_PROTO_ELEMENT_SIZE , size_t element_size
+#define ST_SORT_INVOKE_ELEMENT_SIZE , element_size
+#else
+#define ST_SORT_PROTO_ELEMENT_SIZE
+#define ST_SORT_INVOKE_ELEMENT_SIZE
+#endif
+
+/*
+ * If the user wants to be able to pass in compare functions at runtime,
+ * we'll need to make that an argument of the sort and med3 functions.
+ */
+#ifdef ST_COMPARE_RUNTIME_POINTER
+/*
+ * The type of the comparator function pointer that ST_SORT will take, unless
+ * you've already declared a type name manually and want to use that instead of
+ * having a new one defined.
+ */
+#ifndef ST_COMPARATOR_TYPE_NAME
+#define ST_COMPARATOR_TYPE_NAME ST_MAKE_NAME(ST_SORT, compare_function)
+#endif
+#define ST_COMPARE compare
+#ifndef ST_COMPARE_ARG_TYPE
+#define ST_SORT_PROTO_COMPARE , ST_COMPARATOR_TYPE_NAME compare
+#define ST_SORT_INVOKE_COMPARE , compare
+#else
+#define ST_SORT_PROTO_COMPARE , ST_COMPARATOR_TYPE_NAME compare
+#define ST_SORT_INVOKE_COMPARE , compare
+#endif
+#else
+#define ST_SORT_PROTO_COMPARE
+#define ST_SORT_INVOKE_COMPARE
+#endif
+
+/*
+ * If the user wants to use a compare function or expression that takes an
+ * extra argument, we'll need to make that an argument of the sort, compare and
+ * med3 functions.
+ */
+#ifdef ST_COMPARE_ARG_TYPE
+#define ST_SORT_PROTO_ARG , ST_COMPARE_ARG_TYPE *arg
+#define ST_SORT_INVOKE_ARG , arg
+#else
+#define ST_SORT_PROTO_ARG
+#define ST_SORT_INVOKE_ARG
+#endif
+
+#ifdef ST_DECLARE
+
+#ifdef ST_COMPARE_RUNTIME_POINTER
+typedef int (*ST_COMPARATOR_TYPE_NAME)(const ST_ELEMENT_TYPE *,
+									   const ST_ELEMENT_TYPE *
+									   ST_SORT_PROTO_ARG);
+#endif
+
+/* Declare the sort function.  Note optional arguments at end. */
+ST_SCOPE void ST_SORT(ST_ELEMENT_TYPE *first, size_t n
+					  ST_SORT_PROTO_ELEMENT_SIZE
+					  ST_SORT_PROTO_COMPARE
+					  ST_SORT_PROTO_ARG);
+
+#endif
+
+#ifdef ST_DEFINE
+
+/* sort private helper functions */
+#define ST_MED3 ST_MAKE_NAME(ST_SORT, med3)
+#define ST_SWAP ST_MAKE_NAME(ST_SORT, swap)
+#define ST_SWAPN ST_MAKE_NAME(ST_SORT, swapn)
+
+/* Users expecting to run very large sorts may need them to be interruptible. */
+#ifdef ST_CHECK_FOR_INTERRUPTS
+#define DO_CHECK_FOR_INTERRUPTS() CHECK_FOR_INTERRUPTS()
+#else
+#define DO_CHECK_FOR_INTERRUPTS()
+#endif
+
+/*
+ * Create wrapper macros that know how to invoke compare, med3 and sort with
+ * the right arguments.
+ */
+#ifdef ST_COMPARE_RUNTIME_POINTER
+#define DO_COMPARE(a_, b_) ST_COMPARE((a_), (b_) ST_SORT_INVOKE_ARG)
+#elif defined(ST_COMPARE_ARG_TYPE)
+#define DO_COMPARE(a_, b_) ST_COMPARE((a_), (b_), arg)
+#else
+#define DO_COMPARE(a_, b_) ST_COMPARE((a_), (b_))
+#endif
+#define DO_MED3(a_, b_, c_)												\
+	ST_MED3((a_), (b_), (c_)											\
+			ST_SORT_INVOKE_COMPARE										\
+			ST_SORT_INVOKE_ARG)
+#define DO_SORT(a_, n_)													\
+	ST_SORT((a_), (n_)													\
+			ST_SORT_INVOKE_ELEMENT_SIZE									\
+			ST_SORT_INVOKE_COMPARE										\
+			ST_SORT_INVOKE_ARG)
+
+/*
+ * If we're working with void pointers, we'll use pointer arithmetic based on
+ * uint8, and use the runtime element_size to step through the array and swap
+ * elements.  Otherwise we'll work with ST_ELEMENT_TYPE.
+ */
+#ifndef ST_ELEMENT_TYPE_VOID
+#define ST_POINTER_TYPE ST_ELEMENT_TYPE
+#define ST_POINTER_STEP 1
+#define DO_SWAPN(a_, b_, n_) ST_SWAPN((a_), (b_), (n_))
+#define DO_SWAP(a_, b_) ST_SWAP((a_), (b_))
+#else
+#define ST_POINTER_TYPE uint8
+#define ST_POINTER_STEP element_size
+#define DO_SWAPN(a_, b_, n_) ST_SWAPN((a_), (b_), (n_))
+#define DO_SWAP(a_, b_) DO_SWAPN((a_), (b_), element_size)
+#endif
+
+/*
+ * Find the median of three values.  Currently, performance seems to be best
+ * if the the comparator is inlined here, but the med3 function is not inlined
+ * in the qsort function.
+ */
+static pg_noinline ST_ELEMENT_TYPE *
+ST_MED3(ST_ELEMENT_TYPE *a,
+		ST_ELEMENT_TYPE *b,
+		ST_ELEMENT_TYPE *c
+		ST_SORT_PROTO_COMPARE
+		ST_SORT_PROTO_ARG)
+{
+		return DO_COMPARE(a, b) < 0 ?
+		(DO_COMPARE(b, c) < 0 ? b : (DO_COMPARE(a, c) < 0 ? c : a))
+		: (DO_COMPARE(b, c) > 0 ? b : (DO_COMPARE(a, c) < 0 ? a : c));
+}
+
+static inline void
+ST_SWAP(ST_POINTER_TYPE *a, ST_POINTER_TYPE *b)
+{
+	ST_POINTER_TYPE tmp = *a;
+
+	*a = *b;
+	*b = tmp;
+}
+
+static inline void
+ST_SWAPN(ST_POINTER_TYPE *a, ST_POINTER_TYPE *b, size_t n)
+{
+	for (size_t i = 0; i < n; ++i)
+		ST_SWAP(&a[i], &b[i]);
+}
+
+/*
+ * Sort an array.
+ */
+ST_SCOPE void
+ST_SORT(ST_ELEMENT_TYPE *data, size_t n
+		ST_SORT_PROTO_ELEMENT_SIZE
+		ST_SORT_PROTO_COMPARE
+		ST_SORT_PROTO_ARG)
+{
+	ST_POINTER_TYPE *a = (ST_POINTER_TYPE *) data,
+			   *pa,
+			   *pb,
+			   *pc,
+			   *pd,
+			   *pl,
+			   *pm,
+			   *pn;
+	size_t		d1,
+				d2;
+	int			r,
+				presorted;
+
+loop:
+	DO_CHECK_FOR_INTERRUPTS();
+	if (n < 7)
+	{
+		for (pm = a + ST_POINTER_STEP; pm < a + n * ST_POINTER_STEP;
+			 pm += ST_POINTER_STEP)
+			for (pl = pm; pl > a && DO_COMPARE(pl - ST_POINTER_STEP, pl) > 0;
+				 pl -= ST_POINTER_STEP)
+				DO_SWAP(pl, pl - ST_POINTER_STEP);
+		return;
+	}
+	presorted = 1;
+	for (pm = a + ST_POINTER_STEP; pm < a + n * ST_POINTER_STEP;
+		 pm += ST_POINTER_STEP)
+	{
+		DO_CHECK_FOR_INTERRUPTS();
+		if (DO_COMPARE(pm - ST_POINTER_STEP, pm) > 0)
+		{
+			presorted = 0;
+			break;
+		}
+	}
+	if (presorted)
+		return;
+	pm = a + (n / 2) * ST_POINTER_STEP;
+	if (n > 7)
+	{
+		pl = a;
+		pn = a + (n - 1) * ST_POINTER_STEP;
+		if (n > 40)
+		{
+			size_t		d = (n / 8) * ST_POINTER_STEP;
+
+			pl = DO_MED3(pl, pl + d, pl + 2 * d);
+			pm = DO_MED3(pm - d, pm, pm + d);
+			pn = DO_MED3(pn - 2 * d, pn - d, pn);
+		}
+		pm = DO_MED3(pl, pm, pn);
+	}
+	DO_SWAP(a, pm);
+	pa = pb = a + ST_POINTER_STEP;
+	pc = pd = a + (n - 1) * ST_POINTER_STEP;
+	for (;;)
+	{
+		while (pb <= pc && (r = DO_COMPARE(pb, a)) <= 0)
+		{
+			if (r == 0)
+			{
+				DO_SWAP(pa, pb);
+				pa += ST_POINTER_STEP;
+			}
+			pb += ST_POINTER_STEP;
+			DO_CHECK_FOR_INTERRUPTS();
+		}
+		while (pb <= pc && (r = DO_COMPARE(pc, a)) >= 0)
+		{
+			if (r == 0)
+			{
+				DO_SWAP(pc, pd);
+				pd -= ST_POINTER_STEP;
+			}
+			pc -= ST_POINTER_STEP;
+			DO_CHECK_FOR_INTERRUPTS();
+		}
+		if (pb > pc)
+			break;
+		DO_SWAP(pb, pc);
+		pb += ST_POINTER_STEP;
+		pc -= ST_POINTER_STEP;
+	}
+	pn = a + n * ST_POINTER_STEP;
+	d1 = Min(pa - a, pb - pa);
+	DO_SWAPN(a, pb - d1, d1);
+	d1 = Min(pd - pc, pn - pd - ST_POINTER_STEP);
+	DO_SWAPN(pb, pn - d1, d1);
+	d1 = pb - pa;
+	d2 = pd - pc;
+	if (d1 <= d2)
+	{
+		/* Recurse on left partition, then iterate on right partition */
+		if (d1 > ST_POINTER_STEP)
+			DO_SORT(a, d1 / ST_POINTER_STEP);
+		if (d2 > ST_POINTER_STEP)
+		{
+			/* Iterate rather than recurse to save stack space */
+			/* DO_SORT(pn - d2, d2 / ST_POINTER_STEP) */
+			a = pn - d2;
+			n = d2 / ST_POINTER_STEP;
+			goto loop;
+		}
+	}
+	else
+	{
+		/* Recurse on right partition, then iterate on left partition */
+		if (d2 > ST_POINTER_STEP)
+			DO_SORT(pn - d2, d2 / ST_POINTER_STEP);
+		if (d1 > ST_POINTER_STEP)
+		{
+			/* Iterate rather than recurse to save stack space */
+			/* DO_SORT(a, d1 / ST_POINTER_STEP) */
+			n = d1 / ST_POINTER_STEP;
+			goto loop;
+		}
+	}
+}
+#endif
+
+#undef DO_CHECK_FOR_INTERRUPTS
+#undef DO_COMPARE
+#undef DO_MED3
+#undef DO_SORT
+#undef DO_SWAP
+#undef DO_SWAPN
+#undef ST_COMPARATOR_TYPE_NAME
+#undef ST_COMPARE
+#undef ST_COMPARE_ARG_TYPE
+#undef ST_COMPARE_RUNTIME_POINTER
+#undef ST_ELEMENT_TYPE
+#undef ST_ELEMENT_TYPE_VOID
+#undef ST_MAKE_NAME
+#undef ST_MAKE_NAME_
+#undef ST_MAKE_PREFIX
+#undef ST_MED3
+#undef ST_POINTER_STEP
+#undef ST_POINTER_TYPE
+#undef ST_SCOPE
+#undef ST_SORT
+#undef ST_SORT_INVOKE_ARG
+#undef ST_SORT_INVOKE_COMPARE
+#undef ST_SORT_INVOKE_ELEMENT_SIZE
+#undef ST_SORT_PROTO_ARG
+#undef ST_SORT_PROTO_COMPARE
+#undef ST_SORT_PROTO_ELEMENT_SIZE
+#undef ST_SWAP
+#undef ST_SWAPN
-- 
2.20.1

v2-0002-Use-sort_template.h-for-compactify_tuples.patchtext/x-patch; charset=US-ASCII; name=v2-0002-Use-sort_template.h-for-compactify_tuples.patchDownload
From 9290693b090ec69bf837f2a2f53e1f1663bdc8ef Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 17 Aug 2020 20:59:32 +1200
Subject: [PATCH v2 2/5] Use sort_template.h for compactify_tuples().

Since compactify_tuples() is called often, a specialized sort function
for sorting tuple offsets can give measurable speed-up.

Discussion: https://postgr.es/m/CA%2BhUKGKMQFVpjr106gRhwk6R-nXv0qOcTreZuQzxgpHESAL6dw%40mail.gmail.com
---
 src/backend/storage/page/bufpage.c | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)

diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index d708117a40..f81fa8cf3d 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -421,13 +421,13 @@ typedef struct itemIdSortData
 } itemIdSortData;
 typedef itemIdSortData *itemIdSort;
 
-static int
-itemoffcompare(const void *itemidp1, const void *itemidp2)
-{
-	/* Sort in decreasing itemoff order */
-	return ((itemIdSort) itemidp2)->itemoff -
-		((itemIdSort) itemidp1)->itemoff;
-}
+/* Create a specialized sort function for descending offset order. */
+#define ST_SORT qsort_itemoff
+#define ST_ELEMENT_TYPE itemIdSortData
+#define ST_COMPARE(a, b) ((b)->itemoff - (a)->itemoff)
+#define ST_SCOPE static
+#define ST_DEFINE
+#include "lib/sort_template.h"
 
 /*
  * After removing or marking some line pointers unused, move the tuples to
@@ -441,8 +441,7 @@ compactify_tuples(itemIdSort itemidbase, int nitems, Page page)
 	int			i;
 
 	/* sort itemIdSortData array into decreasing itemoff order */
-	qsort((char *) itemidbase, nitems, sizeof(itemIdSortData),
-		  itemoffcompare);
+	qsort_itemoff(itemidbase, nitems);
 
 	upper = phdr->pd_special;
 	for (i = 0; i < nitems; i++)
-- 
2.20.1

v2-0003-Use-sort_template.h-for-qsort-and-qsort_arg.patchtext/x-patch; charset=US-ASCII; name=v2-0003-Use-sort_template.h-for-qsort-and-qsort_arg.patchDownload
From c4f01775f54a9942d9cd7e90206438233d1f4e6a Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Wed, 19 Aug 2020 19:34:45 +1200
Subject: [PATCH v2 3/5] Use sort_template.h for qsort() and qsort_arg().

Reduce duplication by using the new template.
---
 src/port/qsort.c     | 227 ++----------------------------------------
 src/port/qsort_arg.c | 228 ++-----------------------------------------
 2 files changed, 15 insertions(+), 440 deletions(-)

diff --git a/src/port/qsort.c b/src/port/qsort.c
index fa992e2081..7879e6cd56 100644
--- a/src/port/qsort.c
+++ b/src/port/qsort.c
@@ -1,229 +1,16 @@
 /*
  *	qsort.c: standard quicksort algorithm
- *
- *	Modifications from vanilla NetBSD source:
- *	  Add do ... while() macro fix
- *	  Remove __inline, _DIAGASSERTs, __P
- *	  Remove ill-considered "swap_cnt" switch to insertion sort,
- *	  in favor of a simple check for presorted input.
- *	  Take care to recurse on the smaller partition, to bound stack usage.
- *
- *	CAUTION: if you change this file, see also qsort_arg.c, gen_qsort_tuple.pl
- *
- *	src/port/qsort.c
- */
-
-/*	$NetBSD: qsort.c,v 1.13 2003/08/07 16:43:42 agc Exp $	*/
-
-/*-
- * Copyright (c) 1992, 1993
- *	The Regents of the University of California.  All rights reserved.
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *	  notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *	  notice, this list of conditions and the following disclaimer in the
- *	  documentation and/or other materials provided with the distribution.
- * 3. Neither the name of the University nor the names of its contributors
- *	  may be used to endorse or promote products derived from this software
- *	  without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
  */
 
 #include "c.h"
 
-
-static char *med3(char *a, char *b, char *c,
-				  int (*cmp) (const void *, const void *));
-static void swapfunc(char *, char *, size_t, int);
-
-/*
- * Qsort routine based on J. L. Bentley and M. D. McIlroy,
- * "Engineering a sort function",
- * Software--Practice and Experience 23 (1993) 1249-1265.
- *
- * We have modified their original by adding a check for already-sorted input,
- * which seems to be a win per discussions on pgsql-hackers around 2006-03-21.
- *
- * Also, we recurse on the smaller partition and iterate on the larger one,
- * which ensures we cannot recurse more than log(N) levels (since the
- * partition recursed to is surely no more than half of the input).  Bentley
- * and McIlroy explicitly rejected doing this on the grounds that it's "not
- * worth the effort", but we have seen crashes in the field due to stack
- * overrun, so that judgment seems wrong.
- */
-
-#define swapcode(TYPE, parmi, parmj, n) \
-do {		\
-	size_t i = (n) / sizeof (TYPE);			\
-	TYPE *pi = (TYPE *)(void *)(parmi);			\
-	TYPE *pj = (TYPE *)(void *)(parmj);			\
-	do {						\
-		TYPE	t = *pi;			\
-		*pi++ = *pj;				\
-		*pj++ = t;				\
-		} while (--i > 0);				\
-} while (0)
-
-#define SWAPINIT(a, es) swaptype = ((char *)(a) - (char *)0) % sizeof(long) || \
-	(es) % sizeof(long) ? 2 : (es) == sizeof(long)? 0 : 1
-
-static void
-swapfunc(char *a, char *b, size_t n, int swaptype)
-{
-	if (swaptype <= 1)
-		swapcode(long, a, b, n);
-	else
-		swapcode(char, a, b, n);
-}
-
-#define swap(a, b)						\
-	if (swaptype == 0) {					\
-		long t = *(long *)(void *)(a);			\
-		*(long *)(void *)(a) = *(long *)(void *)(b);	\
-		*(long *)(void *)(b) = t;			\
-	} else							\
-		swapfunc(a, b, es, swaptype)
-
-#define vecswap(a, b, n) if ((n) > 0) swapfunc(a, b, n, swaptype)
-
-static char *
-med3(char *a, char *b, char *c, int (*cmp) (const void *, const void *))
-{
-	return cmp(a, b) < 0 ?
-		(cmp(b, c) < 0 ? b : (cmp(a, c) < 0 ? c : a))
-		: (cmp(b, c) > 0 ? b : (cmp(a, c) < 0 ? a : c));
-}
-
-void
-pg_qsort(void *a, size_t n, size_t es, int (*cmp) (const void *, const void *))
-{
-	char	   *pa,
-			   *pb,
-			   *pc,
-			   *pd,
-			   *pl,
-			   *pm,
-			   *pn;
-	size_t		d1,
-				d2;
-	int			r,
-				swaptype,
-				presorted;
-
-loop:SWAPINIT(a, es);
-	if (n < 7)
-	{
-		for (pm = (char *) a + es; pm < (char *) a + n * es; pm += es)
-			for (pl = pm; pl > (char *) a && cmp(pl - es, pl) > 0;
-				 pl -= es)
-				swap(pl, pl - es);
-		return;
-	}
-	presorted = 1;
-	for (pm = (char *) a + es; pm < (char *) a + n * es; pm += es)
-	{
-		if (cmp(pm - es, pm) > 0)
-		{
-			presorted = 0;
-			break;
-		}
-	}
-	if (presorted)
-		return;
-	pm = (char *) a + (n / 2) * es;
-	if (n > 7)
-	{
-		pl = (char *) a;
-		pn = (char *) a + (n - 1) * es;
-		if (n > 40)
-		{
-			size_t		d = (n / 8) * es;
-
-			pl = med3(pl, pl + d, pl + 2 * d, cmp);
-			pm = med3(pm - d, pm, pm + d, cmp);
-			pn = med3(pn - 2 * d, pn - d, pn, cmp);
-		}
-		pm = med3(pl, pm, pn, cmp);
-	}
-	swap(a, pm);
-	pa = pb = (char *) a + es;
-	pc = pd = (char *) a + (n - 1) * es;
-	for (;;)
-	{
-		while (pb <= pc && (r = cmp(pb, a)) <= 0)
-		{
-			if (r == 0)
-			{
-				swap(pa, pb);
-				pa += es;
-			}
-			pb += es;
-		}
-		while (pb <= pc && (r = cmp(pc, a)) >= 0)
-		{
-			if (r == 0)
-			{
-				swap(pc, pd);
-				pd -= es;
-			}
-			pc -= es;
-		}
-		if (pb > pc)
-			break;
-		swap(pb, pc);
-		pb += es;
-		pc -= es;
-	}
-	pn = (char *) a + n * es;
-	d1 = Min(pa - (char *) a, pb - pa);
-	vecswap(a, pb - d1, d1);
-	d1 = Min(pd - pc, pn - pd - es);
-	vecswap(pb, pn - d1, d1);
-	d1 = pb - pa;
-	d2 = pd - pc;
-	if (d1 <= d2)
-	{
-		/* Recurse on left partition, then iterate on right partition */
-		if (d1 > es)
-			pg_qsort(a, d1 / es, es, cmp);
-		if (d2 > es)
-		{
-			/* Iterate rather than recurse to save stack space */
-			/* pg_qsort(pn - d2, d2 / es, es, cmp); */
-			a = pn - d2;
-			n = d2 / es;
-			goto loop;
-		}
-	}
-	else
-	{
-		/* Recurse on right partition, then iterate on left partition */
-		if (d2 > es)
-			pg_qsort(pn - d2, d2 / es, es, cmp);
-		if (d1 > es)
-		{
-			/* Iterate rather than recurse to save stack space */
-			/* pg_qsort(a, d1 / es, es, cmp); */
-			n = d1 / es;
-			goto loop;
-		}
-	}
-}
+#define ST_SORT pg_qsort
+#define ST_ELEMENT_TYPE_VOID
+#define ST_COMPARE_RUNTIME_POINTER
+#define ST_SCOPE
+#define ST_DECLARE
+#define ST_DEFINE
+#include "lib/sort_template.h"
 
 /*
  * qsort comparator wrapper for strcmp.
diff --git a/src/port/qsort_arg.c b/src/port/qsort_arg.c
index 6d54fbc2b4..fa7e11a3b8 100644
--- a/src/port/qsort_arg.c
+++ b/src/port/qsort_arg.c
@@ -1,226 +1,14 @@
 /*
  *	qsort_arg.c: qsort with a passthrough "void *" argument
- *
- *	Modifications from vanilla NetBSD source:
- *	  Add do ... while() macro fix
- *	  Remove __inline, _DIAGASSERTs, __P
- *	  Remove ill-considered "swap_cnt" switch to insertion sort,
- *	  in favor of a simple check for presorted input.
- *	  Take care to recurse on the smaller partition, to bound stack usage.
- *
- *	CAUTION: if you change this file, see also qsort.c, gen_qsort_tuple.pl
- *
- *	src/port/qsort_arg.c
- */
-
-/*	$NetBSD: qsort.c,v 1.13 2003/08/07 16:43:42 agc Exp $	*/
-
-/*-
- * Copyright (c) 1992, 1993
- *	The Regents of the University of California.  All rights reserved.
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *	  notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *	  notice, this list of conditions and the following disclaimer in the
- *	  documentation and/or other materials provided with the distribution.
- * 3. Neither the name of the University nor the names of its contributors
- *	  may be used to endorse or promote products derived from this software
- *	  without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
  */
 
 #include "c.h"
 
-
-static char *med3(char *a, char *b, char *c,
-				  qsort_arg_comparator cmp, void *arg);
-static void swapfunc(char *, char *, size_t, int);
-
-/*
- * Qsort routine based on J. L. Bentley and M. D. McIlroy,
- * "Engineering a sort function",
- * Software--Practice and Experience 23 (1993) 1249-1265.
- *
- * We have modified their original by adding a check for already-sorted input,
- * which seems to be a win per discussions on pgsql-hackers around 2006-03-21.
- *
- * Also, we recurse on the smaller partition and iterate on the larger one,
- * which ensures we cannot recurse more than log(N) levels (since the
- * partition recursed to is surely no more than half of the input).  Bentley
- * and McIlroy explicitly rejected doing this on the grounds that it's "not
- * worth the effort", but we have seen crashes in the field due to stack
- * overrun, so that judgment seems wrong.
- */
-
-#define swapcode(TYPE, parmi, parmj, n) \
-do {		\
-	size_t i = (n) / sizeof (TYPE);			\
-	TYPE *pi = (TYPE *)(void *)(parmi);			\
-	TYPE *pj = (TYPE *)(void *)(parmj);			\
-	do {						\
-		TYPE	t = *pi;			\
-		*pi++ = *pj;				\
-		*pj++ = t;				\
-		} while (--i > 0);				\
-} while (0)
-
-#define SWAPINIT(a, es) swaptype = ((char *)(a) - (char *)0) % sizeof(long) || \
-	(es) % sizeof(long) ? 2 : (es) == sizeof(long)? 0 : 1
-
-static void
-swapfunc(char *a, char *b, size_t n, int swaptype)
-{
-	if (swaptype <= 1)
-		swapcode(long, a, b, n);
-	else
-		swapcode(char, a, b, n);
-}
-
-#define swap(a, b)						\
-	if (swaptype == 0) {					\
-		long t = *(long *)(void *)(a);			\
-		*(long *)(void *)(a) = *(long *)(void *)(b);	\
-		*(long *)(void *)(b) = t;			\
-	} else							\
-		swapfunc(a, b, es, swaptype)
-
-#define vecswap(a, b, n) if ((n) > 0) swapfunc(a, b, n, swaptype)
-
-static char *
-med3(char *a, char *b, char *c, qsort_arg_comparator cmp, void *arg)
-{
-	return cmp(a, b, arg) < 0 ?
-		(cmp(b, c, arg) < 0 ? b : (cmp(a, c, arg) < 0 ? c : a))
-		: (cmp(b, c, arg) > 0 ? b : (cmp(a, c, arg) < 0 ? a : c));
-}
-
-void
-qsort_arg(void *a, size_t n, size_t es, qsort_arg_comparator cmp, void *arg)
-{
-	char	   *pa,
-			   *pb,
-			   *pc,
-			   *pd,
-			   *pl,
-			   *pm,
-			   *pn;
-	size_t		d1,
-				d2;
-	int			r,
-				swaptype,
-				presorted;
-
-loop:SWAPINIT(a, es);
-	if (n < 7)
-	{
-		for (pm = (char *) a + es; pm < (char *) a + n * es; pm += es)
-			for (pl = pm; pl > (char *) a && cmp(pl - es, pl, arg) > 0;
-				 pl -= es)
-				swap(pl, pl - es);
-		return;
-	}
-	presorted = 1;
-	for (pm = (char *) a + es; pm < (char *) a + n * es; pm += es)
-	{
-		if (cmp(pm - es, pm, arg) > 0)
-		{
-			presorted = 0;
-			break;
-		}
-	}
-	if (presorted)
-		return;
-	pm = (char *) a + (n / 2) * es;
-	if (n > 7)
-	{
-		pl = (char *) a;
-		pn = (char *) a + (n - 1) * es;
-		if (n > 40)
-		{
-			size_t		d = (n / 8) * es;
-
-			pl = med3(pl, pl + d, pl + 2 * d, cmp, arg);
-			pm = med3(pm - d, pm, pm + d, cmp, arg);
-			pn = med3(pn - 2 * d, pn - d, pn, cmp, arg);
-		}
-		pm = med3(pl, pm, pn, cmp, arg);
-	}
-	swap(a, pm);
-	pa = pb = (char *) a + es;
-	pc = pd = (char *) a + (n - 1) * es;
-	for (;;)
-	{
-		while (pb <= pc && (r = cmp(pb, a, arg)) <= 0)
-		{
-			if (r == 0)
-			{
-				swap(pa, pb);
-				pa += es;
-			}
-			pb += es;
-		}
-		while (pb <= pc && (r = cmp(pc, a, arg)) >= 0)
-		{
-			if (r == 0)
-			{
-				swap(pc, pd);
-				pd -= es;
-			}
-			pc -= es;
-		}
-		if (pb > pc)
-			break;
-		swap(pb, pc);
-		pb += es;
-		pc -= es;
-	}
-	pn = (char *) a + n * es;
-	d1 = Min(pa - (char *) a, pb - pa);
-	vecswap(a, pb - d1, d1);
-	d1 = Min(pd - pc, pn - pd - es);
-	vecswap(pb, pn - d1, d1);
-	d1 = pb - pa;
-	d2 = pd - pc;
-	if (d1 <= d2)
-	{
-		/* Recurse on left partition, then iterate on right partition */
-		if (d1 > es)
-			qsort_arg(a, d1 / es, es, cmp, arg);
-		if (d2 > es)
-		{
-			/* Iterate rather than recurse to save stack space */
-			/* qsort_arg(pn - d2, d2 / es, es, cmp, arg); */
-			a = pn - d2;
-			n = d2 / es;
-			goto loop;
-		}
-	}
-	else
-	{
-		/* Recurse on right partition, then iterate on left partition */
-		if (d2 > es)
-			qsort_arg(pn - d2, d2 / es, es, cmp, arg);
-		if (d1 > es)
-		{
-			/* Iterate rather than recurse to save stack space */
-			/* qsort_arg(a, d1 / es, es, cmp, arg); */
-			n = d1 / es;
-			goto loop;
-		}
-	}
-}
+#define ST_SORT qsort_arg
+#define ST_ELEMENT_TYPE_VOID
+#define ST_COMPARATOR_TYPE_NAME qsort_arg_comparator
+#define ST_COMPARE_RUNTIME_POINTER
+#define ST_COMPARE_ARG_TYPE void
+#define ST_SCOPE
+#define ST_DEFINE
+#include "lib/sort_template.h"
-- 
2.20.1

v2-0004-Use-sort_template.h-for-qsort_tuple-and-qsort_ssu.patchtext/x-patch; charset=US-ASCII; name=v2-0004-Use-sort_template.h-for-qsort_tuple-and-qsort_ssu.patchDownload
From ad98b873695ce84dbf7c4ce1b23337db66eef76d Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Wed, 19 Aug 2020 20:25:12 +1200
Subject: [PATCH v2 4/5] Use sort_template.h for qsort_tuple() and
 qsort_ssup().

Replace the Perl code the previously generated specialized sort
functions with an instantiation of sort_template.h.
---
 src/backend/Makefile                      |   4 +-
 src/backend/utils/sort/.gitignore         |   1 -
 src/backend/utils/sort/Makefile           |   8 -
 src/backend/utils/sort/gen_qsort_tuple.pl | 271 ----------------------
 src/backend/utils/sort/tuplesort.c        |  21 +-
 src/tools/msvc/Solution.pm                |  10 -
 src/tools/msvc/clean.bat                  |   1 -
 7 files changed, 21 insertions(+), 295 deletions(-)
 delete mode 100644 src/backend/utils/sort/.gitignore
 delete mode 100644 src/backend/utils/sort/gen_qsort_tuple.pl

diff --git a/src/backend/Makefile b/src/backend/Makefile
index 9706a95848..5a9858e35e 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -190,7 +190,6 @@ distprep:
 	$(MAKE) -C utils	distprep
 	$(MAKE) -C utils/adt	jsonpath_gram.c jsonpath_scan.c
 	$(MAKE) -C utils/misc	guc-file.c
-	$(MAKE) -C utils/sort	qsort_tuple.c
 
 
 ##########################################################################
@@ -312,8 +311,7 @@ maintainer-clean: distclean
 	      storage/lmgr/lwlocknames.h \
 	      utils/adt/jsonpath_gram.c \
 	      utils/adt/jsonpath_scan.c \
-	      utils/misc/guc-file.c \
-	      utils/sort/qsort_tuple.c
+	      utils/misc/guc-file.c
 
 
 ##########################################################################
diff --git a/src/backend/utils/sort/.gitignore b/src/backend/utils/sort/.gitignore
deleted file mode 100644
index f2958633e6..0000000000
--- a/src/backend/utils/sort/.gitignore
+++ /dev/null
@@ -1 +0,0 @@
-/qsort_tuple.c
diff --git a/src/backend/utils/sort/Makefile b/src/backend/utils/sort/Makefile
index 7ac3659261..26f65fcaf7 100644
--- a/src/backend/utils/sort/Makefile
+++ b/src/backend/utils/sort/Makefile
@@ -21,12 +21,4 @@ OBJS = \
 	tuplesort.o \
 	tuplestore.o
 
-tuplesort.o: qsort_tuple.c
-
-qsort_tuple.c: gen_qsort_tuple.pl
-	$(PERL) $(srcdir)/gen_qsort_tuple.pl $< > $@
-
 include $(top_srcdir)/src/backend/common.mk
-
-maintainer-clean:
-	rm -f qsort_tuple.c
diff --git a/src/backend/utils/sort/gen_qsort_tuple.pl b/src/backend/utils/sort/gen_qsort_tuple.pl
deleted file mode 100644
index eb0f7c5814..0000000000
--- a/src/backend/utils/sort/gen_qsort_tuple.pl
+++ /dev/null
@@ -1,271 +0,0 @@
-#!/usr/bin/perl
-
-#
-# gen_qsort_tuple.pl
-#
-# This script generates specialized versions of the quicksort algorithm for
-# tuple sorting.  The quicksort code is derived from the NetBSD code.  The
-# code generated by this script runs significantly faster than vanilla qsort
-# when used to sort tuples.  This speedup comes from a number of places.
-# The major effects are (1) inlining simple tuple comparators is much faster
-# than jumping through a function pointer and (2) swap and vecswap operations
-# specialized to the particular data type of interest (in this case, SortTuple)
-# are faster than the generic routines.
-#
-#	Modifications from vanilla NetBSD source:
-#	  Add do ... while() macro fix
-#	  Remove __inline, _DIAGASSERTs, __P
-#	  Remove ill-considered "swap_cnt" switch to insertion sort,
-#	  in favor of a simple check for presorted input.
-#	  Take care to recurse on the smaller partition, to bound stack usage.
-#
-#     Instead of sorting arbitrary objects, we're always sorting SortTuples.
-#     Add CHECK_FOR_INTERRUPTS().
-#
-# CAUTION: if you change this file, see also qsort.c and qsort_arg.c
-#
-
-use strict;
-use warnings;
-
-my $SUFFIX;
-my $EXTRAARGS;
-my $EXTRAPARAMS;
-my $CMPPARAMS;
-
-emit_qsort_boilerplate();
-
-$SUFFIX      = 'tuple';
-$EXTRAARGS   = ', SortTupleComparator cmp_tuple, Tuplesortstate *state';
-$EXTRAPARAMS = ', cmp_tuple, state';
-$CMPPARAMS   = ', state';
-emit_qsort_implementation();
-
-$SUFFIX      = 'ssup';
-$EXTRAARGS   = ', SortSupport ssup';
-$EXTRAPARAMS = ', ssup';
-$CMPPARAMS   = ', ssup';
-print <<'EOM';
-
-#define cmp_ssup(a, b, ssup) \
-	ApplySortComparator((a)->datum1, (a)->isnull1, \
-						(b)->datum1, (b)->isnull1, ssup)
-
-EOM
-emit_qsort_implementation();
-
-sub emit_qsort_boilerplate
-{
-	print <<'EOM';
-/*
- * autogenerated by src/backend/utils/sort/gen_qsort_tuple.pl, do not edit!
- *
- * This file is included by tuplesort.c, rather than compiled separately.
- */
-
-/*	$NetBSD: qsort.c,v 1.13 2003/08/07 16:43:42 agc Exp $	*/
-
-/*-
- * Copyright (c) 1992, 1993
- *	The Regents of the University of California.  All rights reserved.
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *	  notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *	  notice, this list of conditions and the following disclaimer in the
- *	  documentation and/or other materials provided with the distribution.
- * 3. Neither the name of the University nor the names of its contributors
- *	  may be used to endorse or promote products derived from this software
- *	  without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- */
-
-/*
- * Qsort routine based on J. L. Bentley and M. D. McIlroy,
- * "Engineering a sort function",
- * Software--Practice and Experience 23 (1993) 1249-1265.
- *
- * We have modified their original by adding a check for already-sorted input,
- * which seems to be a win per discussions on pgsql-hackers around 2006-03-21.
- *
- * Also, we recurse on the smaller partition and iterate on the larger one,
- * which ensures we cannot recurse more than log(N) levels (since the
- * partition recursed to is surely no more than half of the input).  Bentley
- * and McIlroy explicitly rejected doing this on the grounds that it's "not
- * worth the effort", but we have seen crashes in the field due to stack
- * overrun, so that judgment seems wrong.
- */
-
-static void
-swapfunc(SortTuple *a, SortTuple *b, size_t n)
-{
-	do
-	{
-		SortTuple 	t = *a;
-		*a++ = *b;
-		*b++ = t;
-	} while (--n > 0);
-}
-
-#define swap(a, b)						\
-	do { 								\
-		SortTuple t = *(a);				\
-		*(a) = *(b);					\
-		*(b) = t;						\
-	} while (0)
-
-#define vecswap(a, b, n) if ((n) > 0) swapfunc(a, b, n)
-
-EOM
-
-	return;
-}
-
-sub emit_qsort_implementation
-{
-	print <<EOM;
-static SortTuple *
-med3_$SUFFIX(SortTuple *a, SortTuple *b, SortTuple *c$EXTRAARGS)
-{
-	return cmp_$SUFFIX(a, b$CMPPARAMS) < 0 ?
-		(cmp_$SUFFIX(b, c$CMPPARAMS) < 0 ? b :
-			(cmp_$SUFFIX(a, c$CMPPARAMS) < 0 ? c : a))
-		: (cmp_$SUFFIX(b, c$CMPPARAMS) > 0 ? b :
-			(cmp_$SUFFIX(a, c$CMPPARAMS) < 0 ? a : c));
-}
-
-static void
-qsort_$SUFFIX(SortTuple *a, size_t n$EXTRAARGS)
-{
-	SortTuple  *pa,
-			   *pb,
-			   *pc,
-			   *pd,
-			   *pl,
-			   *pm,
-			   *pn;
-	size_t		d1,
-				d2;
-	int			r,
-				presorted;
-
-loop:
-	CHECK_FOR_INTERRUPTS();
-	if (n < 7)
-	{
-		for (pm = a + 1; pm < a + n; pm++)
-			for (pl = pm; pl > a && cmp_$SUFFIX(pl - 1, pl$CMPPARAMS) > 0; pl--)
-				swap(pl, pl - 1);
-		return;
-	}
-	presorted = 1;
-	for (pm = a + 1; pm < a + n; pm++)
-	{
-		CHECK_FOR_INTERRUPTS();
-		if (cmp_$SUFFIX(pm - 1, pm$CMPPARAMS) > 0)
-		{
-			presorted = 0;
-			break;
-		}
-	}
-	if (presorted)
-		return;
-	pm = a + (n / 2);
-	if (n > 7)
-	{
-		pl = a;
-		pn = a + (n - 1);
-		if (n > 40)
-		{
-			size_t		d = (n / 8);
-
-			pl = med3_$SUFFIX(pl, pl + d, pl + 2 * d$EXTRAPARAMS);
-			pm = med3_$SUFFIX(pm - d, pm, pm + d$EXTRAPARAMS);
-			pn = med3_$SUFFIX(pn - 2 * d, pn - d, pn$EXTRAPARAMS);
-		}
-		pm = med3_$SUFFIX(pl, pm, pn$EXTRAPARAMS);
-	}
-	swap(a, pm);
-	pa = pb = a + 1;
-	pc = pd = a + (n - 1);
-	for (;;)
-	{
-		while (pb <= pc && (r = cmp_$SUFFIX(pb, a$CMPPARAMS)) <= 0)
-		{
-			if (r == 0)
-			{
-				swap(pa, pb);
-				pa++;
-			}
-			pb++;
-			CHECK_FOR_INTERRUPTS();
-		}
-		while (pb <= pc && (r = cmp_$SUFFIX(pc, a$CMPPARAMS)) >= 0)
-		{
-			if (r == 0)
-			{
-				swap(pc, pd);
-				pd--;
-			}
-			pc--;
-			CHECK_FOR_INTERRUPTS();
-		}
-		if (pb > pc)
-			break;
-		swap(pb, pc);
-		pb++;
-		pc--;
-	}
-	pn = a + n;
-	d1 = Min(pa - a, pb - pa);
-	vecswap(a, pb - d1, d1);
-	d1 = Min(pd - pc, pn - pd - 1);
-	vecswap(pb, pn - d1, d1);
-	d1 = pb - pa;
-	d2 = pd - pc;
-	if (d1 <= d2)
-	{
-		/* Recurse on left partition, then iterate on right partition */
-		if (d1 > 1)
-			qsort_$SUFFIX(a, d1$EXTRAPARAMS);
-		if (d2 > 1)
-		{
-			/* Iterate rather than recurse to save stack space */
-			/* qsort_$SUFFIX(pn - d2, d2$EXTRAPARAMS); */
-			a = pn - d2;
-			n = d2;
-			goto loop;
-		}
-	}
-	else
-	{
-		/* Recurse on right partition, then iterate on left partition */
-		if (d2 > 1)
-			qsort_$SUFFIX(pn - d2, d2$EXTRAPARAMS);
-		if (d1 > 1)
-		{
-			/* Iterate rather than recurse to save stack space */
-			/* qsort_$SUFFIX(a, d1$EXTRAPARAMS); */
-			n = d1;
-			goto loop;
-		}
-	}
-}
-EOM
-
-	return;
-}
diff --git a/src/backend/utils/sort/tuplesort.c b/src/backend/utils/sort/tuplesort.c
index 3c49476483..8119c41328 100644
--- a/src/backend/utils/sort/tuplesort.c
+++ b/src/backend/utils/sort/tuplesort.c
@@ -676,8 +676,27 @@ static void tuplesort_updatemax(Tuplesortstate *state);
  * reduces to ApplySortComparator(), that is single-key MinimalTuple sorts
  * and Datum sorts.
  */
-#include "qsort_tuple.c"
 
+#define ST_SORT qsort_tuple
+#define ST_ELEMENT_TYPE SortTuple
+#define ST_COMPARE_RUNTIME_POINTER
+#define ST_COMPARE_ARG_TYPE Tuplesortstate
+#define ST_CHECK_FOR_INTERRUPTS
+#define ST_SCOPE static
+#define ST_DECLARE
+#define ST_DEFINE
+#include "lib/sort_template.h"
+
+#define ST_SORT qsort_ssup
+#define ST_ELEMENT_TYPE SortTuple
+#define ST_COMPARE(a, b, ssup) \
+	ApplySortComparator((a)->datum1, (a)->isnull1, \
+						(b)->datum1, (b)->isnull1, (ssup))
+#define ST_COMPARE_ARG_TYPE SortSupportData
+#define ST_CHECK_FOR_INTERRUPTS
+#define ST_SCOPE static
+#define ST_DEFINE
+#include "lib/sort_template.h"
 
 /*
  *		tuplesort_begin_xxx
diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm
index bc8904732f..ca2e3f6329 100644
--- a/src/tools/msvc/Solution.pm
+++ b/src/tools/msvc/Solution.pm
@@ -662,16 +662,6 @@ sub GenerateFiles
 		);
 	}
 
-	if (IsNewer(
-			'src/backend/utils/sort/qsort_tuple.c',
-			'src/backend/utils/sort/gen_qsort_tuple.pl'))
-	{
-		print "Generating qsort_tuple.c...\n";
-		system(
-			'perl src/backend/utils/sort/gen_qsort_tuple.pl > src/backend/utils/sort/qsort_tuple.c'
-		);
-	}
-
 	if (IsNewer('src/bin/psql/sql_help.h', 'src/bin/psql/create_help.pl'))
 	{
 		print "Generating sql_help.h...\n";
diff --git a/src/tools/msvc/clean.bat b/src/tools/msvc/clean.bat
index 672bb2d650..5b6b796ff9 100755
--- a/src/tools/msvc/clean.bat
+++ b/src/tools/msvc/clean.bat
@@ -61,7 +61,6 @@ if %DIST%==1 if exist src\backend\storage\lmgr\lwlocknames.h del /q src\backend\
 if %DIST%==1 if exist src\pl\plpython\spiexceptions.h del /q src\pl\plpython\spiexceptions.h
 if %DIST%==1 if exist src\pl\plpgsql\src\plerrcodes.h del /q src\pl\plpgsql\src\plerrcodes.h
 if %DIST%==1 if exist src\pl\tcl\pltclerrcodes.h del /q src\pl\tcl\pltclerrcodes.h
-if %DIST%==1 if exist src\backend\utils\sort\qsort_tuple.c del /q src\backend\utils\sort\qsort_tuple.c
 if %DIST%==1 if exist src\bin\psql\sql_help.c del /q src\bin\psql\sql_help.c
 if %DIST%==1 if exist src\bin\psql\sql_help.h del /q src\bin\psql\sql_help.h
 if %DIST%==1 if exist src\common\kwlist_d.h del /q src\common\kwlist_d.h
-- 
2.20.1

#5David Rowley
dgrowleyml@gmail.com
In reply to: Thomas Munro (#4)
Re: Optimising compactify_tuples()

On Thu, 20 Aug 2020 at 11:28, Thomas Munro <thomas.munro@gmail.com> wrote:

I fixed up the copyright messages, and removed some stray bits of
build scripting relating to the Perl-generated file. Added to
commitfest.

I'm starting to look at this. So far I've only just done a quick
performance test on it. With the workload I ran, using 0001+0002.

The test replayed ~2.2 GB of WAL. master took 148.581 seconds and
master+0001+0002 took 115.588 seconds. That's about 28% faster. Pretty
nice!

I found running a lower heap fillfactor will cause quite a few more
heap cleanups to occur. Perhaps that's one of the reasons the speedup
I got was more than the 12% you reported.

More details of the test:

Setup:

drowley@amd3990x:~$ cat recoverbench.sh
#!/bin/bash

pg_ctl stop -D pgdata -m smart
pg_ctl start -D pgdata -l pg.log -w
psql -c "drop table if exists t1;" postgres > /dev/null
psql -c "create table t1 (a int primary key, b int not null) with
(fillfactor = 85);" postgres > /dev/null
psql -c "insert into t1 select x,0 from generate_series(1,10000000)
x;" postgres > /dev/null
psql -c "drop table if exists log_wal;" postgres > /dev/null
psql -c "create table log_wal (lsn pg_lsn not null);" postgres > /dev/null
psql -c "insert into log_wal values(pg_current_wal_lsn());" postgres > /dev/null
pgbench -n -f update.sql -t 60000 -c 200 -j 200 -M prepared postgres > /dev/null
psql -c "select 'Used ' ||
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), lsn)) || ' of
WAL' from log_wal limit 1;" postgres
pg_ctl stop -D pgdata -m immediate -w
echo Starting Postgres...
pg_ctl start -D pgdata -l pg.log

drowley@amd3990x:~$ cat update.sql
\set i random(1,10000000)
update t1 set b = b+1 where a = :i;

Results:

master

Recovery times are indicated in the postgresql log:

2020-09-06 22:38:58.992 NZST [6487] LOG: redo starts at 3/16E4A988
2020-09-06 22:41:27.570 NZST [6487] LOG: invalid record length at
3/F67F8B48: wanted 24, got 0
2020-09-06 22:41:27.573 NZST [6487] LOG: redo done at 3/F67F8B20

recovery duration = 00:02:28.581

drowley@amd3990x:~$ ./recoverbench.sh
waiting for server to shut down.... done
server stopped
waiting for server to start.... done
server started
?column?
---------------------
Used 2333 MB of WAL
(1 row)

waiting for server to shut down.... done
server stopped
Starting Postgres...

recovery profile:
28.79% postgres postgres [.] pg_qsort
13.58% postgres postgres [.] itemoffcompare
12.27% postgres postgres [.] PageRepairFragmentation
8.26% postgres libc-2.31.so [.] 0x000000000018e48f
5.90% postgres postgres [.] swapfunc
4.86% postgres postgres [.] hash_search_with_hash_value
2.95% postgres postgres [.] XLogReadBufferExtended
1.83% postgres postgres [.] PinBuffer
1.80% postgres postgres [.] compactify_tuples
1.71% postgres postgres [.] med3
0.99% postgres postgres [.] hash_bytes
0.90% postgres libc-2.31.so [.] 0x000000000018e470
0.89% postgres postgres [.] StartupXLOG
0.84% postgres postgres [.] XLogReadRecord
0.72% postgres postgres [.] LWLockRelease
0.71% postgres postgres [.] PageGetHeapFreeSpace
0.61% postgres libc-2.31.so [.] 0x000000000018e499
0.50% postgres postgres [.] heap_xlog_update
0.50% postgres postgres [.] DecodeXLogRecord
0.50% postgres postgres [.] pg_comp_crc32c_sse42
0.45% postgres postgres [.] LWLockAttemptLock
0.40% postgres postgres [.] ReadBuffer_common
0.40% postgres [kernel.kallsyms] [k] copy_user_generic_string
0.36% postgres libc-2.31.so [.] 0x000000000018e49f
0.33% postgres postgres [.] SlruSelectLRUPage
0.32% postgres postgres [.] PageAddItemExtended
0.31% postgres postgres [.] ReadPageInternal

Patched v2-0001 + v2-0002:

Recovery times are indicated in the postgresql log:

2020-09-06 22:54:25.532 NZST [13252] LOG: redo starts at 3/F67F8C70
2020-09-06 22:56:21.120 NZST [13252] LOG: invalid record length at
4/D633FCD0: wanted 24, got 0
2020-09-06 22:56:21.120 NZST [13252] LOG: redo done at 4/D633FCA8

recovery duration = 00:01:55.588

drowley@amd3990x:~$ ./recoverbench.sh
waiting for server to shut down.... done
server stopped
waiting for server to start.... done
server started
?column?
---------------------
Used 2335 MB of WAL
(1 row)

waiting for server to shut down.... done
server stopped
Starting Postgres...

recovery profile:
32.29% postgres postgres [.] qsort_itemoff
17.73% postgres postgres [.] PageRepairFragmentation
10.98% postgres libc-2.31.so [.] 0x000000000018e48f
5.54% postgres postgres [.] hash_search_with_hash_value
3.60% postgres postgres [.] XLogReadBufferExtended
2.32% postgres postgres [.] compactify_tuples
2.14% postgres postgres [.] PinBuffer
1.39% postgres postgres [.] PageGetHeapFreeSpace
1.38% postgres postgres [.] hash_bytes
1.36% postgres postgres [.] qsort_itemoff_med3
0.94% postgres libc-2.31.so [.] 0x000000000018e499
0.89% postgres postgres [.] XLogReadRecord
0.74% postgres postgres [.] LWLockRelease
0.74% postgres postgres [.] DecodeXLogRecord
0.73% postgres postgres [.] heap_xlog_update
0.66% postgres postgres [.] LWLockAttemptLock
0.65% postgres libc-2.31.so [.] 0x000000000018e470
0.64% postgres postgres [.] pg_comp_crc32c_sse42
0.63% postgres postgres [.] StartupXLOG
0.61% postgres [kernel.kallsyms] [k] copy_user_generic_string
0.60% postgres postgres [.] PageAddItemExtended
0.60% postgres libc-2.31.so [.] 0x000000000018e49f
0.56% postgres libc-2.31.so [.] 0x000000000018e495
0.54% postgres postgres [.] ReadBuffer_common

Settings:
shared_buffers = 10GB
checkpoint_timeout = 1 hour
max_wal_size = 100GB

Hardware:

AMD 3990x
Samsung 970 EVO SSD
64GB DDR4 3600MHz

I'll spend some time looking at the code soon.

David

#6David Rowley
dgrowleyml@gmail.com
In reply to: David Rowley (#5)
1 attachment(s)
Re: Optimising compactify_tuples()

On Sun, 6 Sep 2020 at 23:37, David Rowley <dgrowleyml@gmail.com> wrote:

The test replayed ~2.2 GB of WAL. master took 148.581 seconds and
master+0001+0002 took 115.588 seconds. That's about 28% faster. Pretty
nice!

I was wondering today if we could just get rid of the sort in
compactify_tuples() completely. It seems to me that the existing sort
is there just so that the memmove() is done in order of tuple at the
end of the page first. We seem to be just shunting all the tuples to
the end of the page so we need to sort the line items in reverse
offset so as not to overwrite memory for other tuples during the copy.

I wondered if we could get around that just by having another buffer
somewhere and memcpy the tuples into that first then copy the tuples
out that buffer back into the page. No need to worry about the order
we do that in as there's no chance to overwrite memory belonging to
other tuples.

Doing that gives me 79.166 seconds in the same recovery test. Or about
46% faster, instead of 22% (I mistakenly wrote 28% yesterday)

The top of perf report says:

24.19% postgres postgres [.] PageRepairFragmentation
8.37% postgres postgres [.] hash_search_with_hash_value
7.40% postgres libc-2.31.so [.] 0x000000000018e74b
5.59% postgres libc-2.31.so [.] 0x000000000018e741
5.49% postgres postgres [.] XLogReadBufferExtended
4.05% postgres postgres [.] compactify_tuples
3.27% postgres postgres [.] PinBuffer
2.88% postgres libc-2.31.so [.] 0x000000000018e470
2.02% postgres postgres [.] hash_bytes

(I'll need to figure out why libc's debug symbols are not working)

I was thinking that there might be a crossover point to where this
method becomes faster than the sort method. e.g sorting 1 tuple is
pretty cheap, but copying the memory for the entire tuple space might
be expensive as that includes the tuples we might be getting rid of.
So if we did go down that path we might need some heuristics to decide
which method is likely best. Maybe that's based on the number of
tuples, I'm not really sure. I've not made any attempt to try to give
it a worst-case workload to see if there is a crossover point that's
worth worrying about.

The attached patch is what I used to test this. It kinda goes and
sticks a page-sized variable on the stack, which is not exactly ideal.
I think we'd likely want to figure some other way to do that, but I
just don't know what that would look like yet. I just put the attached
together quickly to test out the idea.

(I don't want to derail the sort improvements here. I happen to think
those are quite important improvements, so I'll continue to review
that patch still. Longer term, we might just end up with something
slightly different for compactify_tuples)

David

Attachments:

compactify_tuples_dgr.patch.txttext/plain; charset=US-ASCII; name=compactify_tuples_dgr.patch.txtDownload
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index d708117a40..5bf35aff37 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -440,22 +440,53 @@ compactify_tuples(itemIdSort itemidbase, int nitems, Page page)
 	Offset		upper;
 	int			i;
 
-	/* sort itemIdSortData array into decreasing itemoff order */
-	qsort((char *) itemidbase, nitems, sizeof(itemIdSortData),
-		  itemoffcompare);
+	if (nitems > MaxHeapTuplesPerPage / 2)
+	{
+		char buffer[BLCKSZ + MAXIMUM_ALIGNOF];
+		char *bufp = (char *) MAXALIGN(&buffer);
+
+		/*
+		 * Make a temp copy of the tuple data so that we can rearange
+		 * the tuples freely without having to worry about stomping
+		 * on the memory of other tuples.
+		 */
+		memcpy(bufp + phdr->pd_upper,
+			   page + phdr->pd_upper,
+			   phdr->pd_special - phdr->pd_upper);
 
-	upper = phdr->pd_special;
-	for (i = 0; i < nitems; i++)
+		upper = phdr->pd_special;
+		for (i = 0; i < nitems; i++)
+		{
+			itemIdSort	itemidptr = &itemidbase[i];
+			ItemId		lp;
+
+			lp = PageGetItemId(page, itemidptr->offsetindex + 1);
+			upper -= itemidptr->alignedlen;
+			memcpy((char *) page + upper,
+				   bufp + itemidptr->itemoff,
+				   itemidptr->alignedlen);
+			lp->lp_off = upper;
+		}
+	}
+	else
 	{
-		itemIdSort	itemidptr = &itemidbase[i];
-		ItemId		lp;
-
-		lp = PageGetItemId(page, itemidptr->offsetindex + 1);
-		upper -= itemidptr->alignedlen;
-		memmove((char *) page + upper,
-				(char *) page + itemidptr->itemoff,
-				itemidptr->alignedlen);
-		lp->lp_off = upper;
+		/* sort itemIdSortData array into decreasing itemoff order */
+		qsort((char *) itemidbase, nitems, sizeof(itemIdSortData),
+			  itemoffcompare);
+
+		upper = phdr->pd_special;
+		for (i = 0; i < nitems; i++)
+		{
+			itemIdSort	itemidptr = &itemidbase[i];
+			ItemId		lp;
+
+			lp = PageGetItemId(page, itemidptr->offsetindex + 1);
+			upper -= itemidptr->alignedlen;
+			memmove((char *) page + upper,
+					(char *) page + itemidptr->itemoff,
+					itemidptr->alignedlen);
+			lp->lp_off = upper;
+		}
 	}
 
 	phdr->pd_upper = upper;
#7Thomas Munro
thomas.munro@gmail.com
In reply to: David Rowley (#6)
Re: Optimising compactify_tuples()

On Mon, Sep 7, 2020 at 7:48 PM David Rowley <dgrowleyml@gmail.com> wrote:

I was wondering today if we could just get rid of the sort in
compactify_tuples() completely. It seems to me that the existing sort
is there just so that the memmove() is done in order of tuple at the
end of the page first. We seem to be just shunting all the tuples to
the end of the page so we need to sort the line items in reverse
offset so as not to overwrite memory for other tuples during the copy.

I wondered if we could get around that just by having another buffer
somewhere and memcpy the tuples into that first then copy the tuples
out that buffer back into the page. No need to worry about the order
we do that in as there's no chance to overwrite memory belonging to
other tuples.

Doing that gives me 79.166 seconds in the same recovery test. Or about
46% faster, instead of 22% (I mistakenly wrote 28% yesterday)

Wow.

One thought is that if we're going to copy everything out and back in
again, we might want to consider doing it in a
memory-prefetcher-friendly order. Would it be a good idea to
rearrange the tuples to match line pointer order, so that the copying
work and also later sequential scans are in a forward direction? The
copying could also perhaps be done with single memcpy() for ranges of
adjacent tuples. Another thought is that it might be possible to
identify some easy cases that it can handle with an alternative
in-place shifting algorithm without having to go to the
copy-out-and-back-in path. For example, when the offset order already
matches line pointer order but some dead tuples just need to be
squeezed out by shifting ranges of adjacent tuples, and maybe some
slightly more complicated cases, but nothing requiring hard work like
sorting.

(I don't want to derail the sort improvements here. I happen to think
those are quite important improvements, so I'll continue to review
that patch still. Longer term, we might just end up with something
slightly different for compactify_tuples)

Yeah. Perhaps qsort specialisation needs to come back in a new thread
with a new use case.

#8David Rowley
dgrowleyml@gmail.com
In reply to: Thomas Munro (#7)
Re: Optimising compactify_tuples()

On Tue, 8 Sep 2020 at 12:08, Thomas Munro <thomas.munro@gmail.com> wrote:

One thought is that if we're going to copy everything out and back in
again, we might want to consider doing it in a
memory-prefetcher-friendly order. Would it be a good idea to
rearrange the tuples to match line pointer order, so that the copying
work and also later sequential scans are in a forward direction?

That's an interesting idea but wouldn't that require both the copy to
the separate buffer *and* a qsort? That's the worst of both
implementations. We'd need some other data structure too in order to
get the index of the sorted array by reverse lineitem point, which
might require an additional array and an additional sort.

The
copying could also perhaps be done with single memcpy() for ranges of
adjacent tuples.

I wonder if the additional code required to check for that would be
cheaper than the additional function call. If it was then it might be
worth trying, but since the tuples can be in any random order then
it's perhaps not likely to pay off that often. I'm not really sure
how often adjacent line items will also be neighbouring tuples for
pages we call compactify_tuples() for. It's certainly going to be
common with INSERT only tables, but if we're calling
compactify_tuples() then it's not read-only.

Another thought is that it might be possible to
identify some easy cases that it can handle with an alternative
in-place shifting algorithm without having to go to the
copy-out-and-back-in path. For example, when the offset order already
matches line pointer order but some dead tuples just need to be
squeezed out by shifting ranges of adjacent tuples, and maybe some
slightly more complicated cases, but nothing requiring hard work like
sorting.

It's likely worth experimenting. The only thing is that the workload
I'm using seems to end up with the tuples with line items not in the
same order as the tuple offset. So adding a precheck to check the
ordering will regress the test I'm doing. We'd need to see if there is
any other workload that would keep the tuples more in order then
determine how likely that is to occur in the real world.

(I don't want to derail the sort improvements here. I happen to think
those are quite important improvements, so I'll continue to review
that patch still. Longer term, we might just end up with something
slightly different for compactify_tuples)

Yeah. Perhaps qsort specialisation needs to come back in a new thread
with a new use case.

hmm, yeah, perhaps that's a better way given the subject here is about
compactify_tuples()

David

#9David Rowley
dgrowleyml@gmail.com
In reply to: David Rowley (#6)
3 attachment(s)
Re: Optimising compactify_tuples()

On Mon, 7 Sep 2020 at 19:47, David Rowley <dgrowleyml@gmail.com> wrote:

I wondered if we could get around that just by having another buffer
somewhere and memcpy the tuples into that first then copy the tuples
out that buffer back into the page. No need to worry about the order
we do that in as there's no chance to overwrite memory belonging to
other tuples.

Doing that gives me 79.166 seconds in the same recovery test. Or about
46% faster, instead of 22% (I mistakenly wrote 28% yesterday)

I did some more thinking about this and considered if there's a way to
just get rid of the sorting version of compactify_tuples() completely.
In the version from yesterday, I fell back on the sort version for
when more than half the tuples from the page were being pruned. I'd
thought that in this case copying out *all* of the page from pd_upper
up to the pd_special (the tuple portion of the page) would maybe be
more costly since that would include (needlessly) copying all the
pruned tuples too. The sort also becomes cheaper in that case since
the number of items to sort is less, hence I thought it was a good
idea to keep the old version for some cases. However, I now think we
can just fix this by conditionally copying all tuples when in 1 big
memcpy when not many tuples have been pruned and when more tuples are
pruned we can just do individual memcpys into the separate buffer.

I wrote a little .c program to try to figure out of there's some good
cut off point to where one method becomes better than the other and I
find that generally if we're pruning away about 75% of tuples then
doing a memcpy() per non-pruned tuple is faster, otherwise, it seems
better just to copy the entire tuple area of the page. See attached
compact_test.c

I ran this and charted the cut off at (nitems < totaltups / 4) and
(nitems < totaltups / 2), and nitems < 16)
./compact_test 32 192
./compact_test 64 96
./compact_test 128 48
./compact_test 256 24
./compact_test 512 12

The / 4 one gives me the graphs with the smallest step when the method
switches. See attached 48_tuples_per_page.png for comparison.

I've so far come up with the attached
compactify_tuples_dgr_v2.patch.txt. Thomas pointed out to me off-list
that using PGAlignedBlock is the general way to allocate a page-sized
data on the stack. I'm still classing this patch as PoC grade. I'll
need to look a bit harder at the correctness of it all.

I did spend quite a bit of time trying to find a case where this is
slower than master's version. I can't find a case where there's any
noticeable slowdown. Using the same script from [1]/messages/by-id/CAApHDvoKwqAzhiuxEt8jSquPJKDpH8DNUZDFUSX9P7DXrJdc3Q@mail.gmail.com I tried a few
variations of the t1 table by adding an additional column to pad out
the tuple to make it wider. Obviously a wider tuple means fewer
tuples on the page so less tuples for master's qsort to sort during
compactify_tuples(). I did manage to squeeze a bit more performance
out of the test cases. Yesterday I got 79.166 seconds. This version
gets me 76.623 seconds.

Here are the results of the various tuple widths:

narrow width row test: insert into t1 select x,0 from
generate_series(1,10000000) x; (32 byte tuples)

patched: 76.623
master: 137.593

medium width row test: insert into t1 select x,0,md5(x::text) ||
md5((x+1)::Text) from generate_series(1,10000000) x; (about 64 byte
tuples)

patched: 64.411
master: 95.576

wide row test: insert into t1 select x,0,(select
string_agg(md5(y::text),'') from generate_Series(x,x+30) y) from
generate_series(1,1000000)x; (1024 byte tuples)

patched: 88.653
master: 90.077

Changing the test so instead of having 10 million rows in the table
and updating a random row 12 million times. I put just 10 rows in the
table and updated them 12 million times. This results in
compactify_tuples() pruning all but 1 row (since autovac can't keep up
with this, each row does end up on a page by itself). I wanted to
ensure I didn't regress a workload that master's qsort() version would
have done very well at. qsorting 1 element is pretty fast.

10-row narrow test:

patched: 10.633 <--- small regression
master: 10.366

I could special case this and do a memmove without copying the tuple
to another buffer, but I don't think the slowdown is enough to warrant
having such a special case.

Another thing I tried was to instead of compacting the page in
compactify_tuples(), I just get rid of that function and did the
compacting in the existing loop in PageRepairFragmentation(). This
does require changing the ERROR check to a PANIC since we may have
already started shuffling tuples around when we find the corrupted
line pointer. However, I was just unable to make this faster than the
attached version. I'm still surprised at this as I can completely get
rid of the itemidbase array. The best run-time I got with this method
out the original test was 86 seconds, so 10 seconds slower than what
the attached can do. So I threw that idea away.

David

[1]: /messages/by-id/CAApHDvoKwqAzhiuxEt8jSquPJKDpH8DNUZDFUSX9P7DXrJdc3Q@mail.gmail.com

Attachments:

compact_test.ctext/plain; charset=US-ASCII; name=compact_test.cDownload
48_tuples_per_page.pngimage/png; name=48_tuples_per_page.pngDownload
�PNG


IHDR�GA3sRGB���gAMA���a	pHYs%%IR$���IDATx^��wpwz&~���r�|U�-��:����~%�]l�}.S"�W%��JZQZi����I� A0'0�$���3H0'0�	&�A� �������|��$����SS(LwOOw����@3�+1��K1��K1��K1��K1��K1��K1��K1��K1��K1��K1��K1��K�dT�r��o[]�v�j�Q$A7�f�����>����?�|��Q6l���V)��


��>TXX�!O2�$��'�����+Wb�uQ��
KG ��Pb�P����z�==�+��T���'��2L���B�E	+���cQ���a��$����t�R�St��~�p���]
n�������z��U)JMM��#F�}��=�Q������l��
=����
V ����\V�t�`�<_S���bdV�Q��o/jkkQ�?V#"�g�����N�U:X�~�� ��������V�I�������*��P(4o�<���o��(-q�+d�l����T���~���^y����1.+V���������X�-Z�E�jDDR�LTU@��z����D�/���6�M����t�������VBC��1��{`����/'N�����F=�Q5E�>�J�V��}k�������O[�R���srr�Y�'"����ZWW7z�h����h����Z1�_�>*��c=h�S*��1��K������9���CV�%9������/#�r��S	��Q����#F��{���(5�l/�={���)S��q1yIw�*~[/Z�+�i���]�6�����q"�u:(b��(1FUr��G�b�;v��^��OfT
���jOq��
�D���}kr��3g�4M�j����J��,hV#"��Q���CX��3M���+++���������<6�h���g;�FQ�������v[��9�X�n��#�&_}�����+**bOHHU1��avv�_|�n����Y�f]�r%�Y
Xe?~<//O�4�Jy������V�����n��<y��j�*�}0<��a����^�/�b�w���G�c��9|���E<iLX�p��LL=t�'??���,[9�?UUU�������q�����W�V[At����g�S^�x��1_�|y��8g6��c�rrrTi�]K�,��+j�R0J��>c�`��qG��-��Q�
���>��p��{�ba�[%T@��Xgee�3@�(����������!���dOj��O�~����~��u�,V�R�F�-��z�q�����]�iT�QT[������������0:���$V�X��w���
����vs�"�������
V:X��V�UV4�h����\U�A����tz�^;G�~TE�.**j��	�iu�*����3*�EAO��nL���J9�0}���O\�m���Wc���~�-�*���C��0���_��)��|?GMCSou�
}��y��<;!=�]�l��,7`����WE�(��;T��__�������[h�=w���u���JM�(��Z__?i�$�EG��w��m����U�Zu�`�1en���|�c4�z����q'  +���Y��!	3����PD-X�e���N�d,zX�}�a,1��w�+Lu�R\�����FuE�EuPJ�$��c��8vyQb��$��MU����w��/""�I?�b�6m�4�k����6��W=


��X)������H�����2��y�@\�k���o�����!9?~������:N�����Ok���=����K�6$��Y]�R����;b��
 �&��LVm���	��������F�j�����:�M����JB�c�s�����/�l�qyyy�d5_��G�c�:��jOT��Y%??_���Y������"����^dG�t1�b��|�����Yg����`L��3����+�����������_JNXl����+�6�;�����w��qW,6�6nTM}�?��ZMaA[�f
[�l�W�6l�:���V]�MUW�
.��A�V#"�'��jo�����t�������������~���X}[�Z��gls�n�\��V�v����@b�jjj����2
V�7o��>o���Tm��gXw_�tI��}>���[U�D��}����5

�U����B�&�n�:�Qb�tPf���vA�����c�s�ig;������D���8/fW��9��� �:a�J�C+�!�L����*�Zk�:�V�w���k������;8�m�,|d����3����\�b������*XH_���9H6{f�������YB�����b�+��@a���:����A:{�����9w9g����k�������T�&>g|��~�zU)�2�L�z�)v*����h��u6���e5)��3._���#�w�u2�����K��aP1��>�wW�^�K|`����s����b�CV�����P�zM!n�6�ED)U��.���NE�0��'�}�S�M1zbM��S�M�#�����`���+t���b��K����A#����$ ��'��������������j�
_���m��m�[ 5������M��?���!v����0���$g��3�=$�Q�K��*jT������MMM*R����>�=���8�h���.
mW',���Xq�����j{��������}��r���!,v\�]�����{�'�����;f5m�>�P��}c�+��@a�����H'V�6X���+���Y�E?�c�yC�)QS���3���R��b�O$*f����;w��'���H6������{�j�El��w�[�W�a���_G�'�b��|��Q�J���c�j�������U�
����T�����q���~_�@�DU�Wx�*2���>��E���j��������$cW�q��h���F�����������Rq�|�[ql���R
��(j��&;��W�M���;u���r�������C�N�Q,{:D��a����G�^�tI��u�gu���-Y��j�6����t*�	�����7��dG��s������f�[�]�&�.���M*�����nw76�h����JQ�1�!n���O���={V�x�s�N���U�v�A���l��%���&�Ye��5�Di�����=Le�O����Ij;FY��=�;����U���/��B��m��=�CW�r�&��������W?S�}il/��Lh�?�H�.GU�k�.]�u6�Q��D�#u�	�������d�����hb�m5j�/�W�6W���-���C�6;9�Q[;a{�;P����	���}p6���W�\I�Dd�M��M�(�'��;�8�%�4�4&l@]]���'W�\��"�g��dOv���m�����;�����Q�����q�s��Y�n]lC���n;�����E���>E
m���Z8p �����(��W������#{��k���s�^�B������s(>����D%��B��m���F�4Lq���"~*L�>����)����?��R<"���"J����u9��������M�u��1���.��j������
�_�\���+nP�d�@�"j���Mn_��]0�����-
��}��T��W'�hQ��F6�-w���(�	�^�����������=n���v����)l�c��(Q??���w�������j2F5T]��I&WW�* j��9
~NL�4)������\U�FI������R��������[���q')���;�vMS\�?�������(������{c�X�D���W��(�3H���<��QU�4Sa��n���n���4�$��X�:�W�q{wW�Z	���j$J�B5�G��TD���a@�1c��]�N��au��FQ�*5��H2�)Jc�b&��$&�?�0~�����# �L�<���=�qk
�����F1#���w<��J#�f�s����?_��tB��fu�|;�B]�f��������=U1'��������.^�����555������j�h���XA�T�w�]�&,��������;��h�xR�]��;�)�u�}&I�Q�^i�cT��[������q5���v���F�5�Z��\���:��+W�TgpB*;�����X�9�:N��9��:�T��5�)�����fC��/F�t�p��������_&��H�f�.�����|V��:?w:����P����D�<5�$�WzQ��1{���:u�}��,�'i�WU�;IS9@]]������fHu�c��E���Z�&T�5��+qk���?�������1:]:��s�%��+q�:�	����"��k��{I���������Dmw�}�Q�u�8�@j%q�P���qnz��(v�JtqC�`�����`;�3.{:$�NH]WagS{��R�D��7E]���!�����={Vu�d_�bo��p��eU���D3a��3[��i�)��qgE����q����Q
�OL���6�h��nFU���z�8���JEO�R�NR{�z�/��������[�F
m�5u�Sls�G�9a�����'WUU��m'��gS�_����DHE�q����o3"��.G�$���<��+���`_V���7�v7����>���R�G��������>�n~�T��M����}�L]�������{�v"�����W��VcO
{����/_V�9�q��=���S]�����Q�yS!��:��������`��G���k����'D��a*�:01�5*�V�b����
�Ng5Y�����C��@t`�&I��@����3c������-�@�KKK�G�{"q'�=<q�|��T�^��db�*�=�*���M����C�W`�1��V�uu���n����+�D�)*� ~"�����U�&��K���<�������
�|��E�{qce���h} 1{����<eoG+\l���\���=�kX��@Zpl��<y��O������W�n~�}�����P������={����Y�T���j����[����Q��5�h���`�t|�N|�����U;Z ��;8�>����
ws���NX������s�
��c�s0�Y�G�7�� ���,�}{�=>���SU�g��+W���>�~}a&�������ml4��s���,8G��	�lg,������(�:���?�W8��J��t)��������:r?�?�@UY�O�����A�`t��u�Qb�$]�h�$%�G �a��$��}.Z����_�l�83\�5{����=G[0�*��Y#���q�t�Gsu��������!�!��.��*���+��VOt�"����*�2X���@BB.��N����`�|�����bQ��u���-���5��6�=
�b�*�����X�` �����-P�����r�}{[����e����<�m�8Ler~u���TW',BU��5qa0����S��p�SoUvI���+K5�q�KbZ��-6���9lq��}S�`�������B�����z0�"���s���������c(��T�Le����z������t�9��.����7����U��US_�Q�S-���K�D�<_�;b��X�)�����H�>��-Z7(�#�O�v�cH�{��b�
�eu��Im�1H�����c����L]c�����X�P����t����B�p���i�$��eJ��g�(--����P��E�Di�����Y���;��Jc�:t(n�������v<�	}��c������ ���0�l��9�#
rI����s�s��Ul4���b��l�V��@1��~`�0���=U3s�������S\�{���<k :���dY�;&�����&B�y���jB���Q]�)bn�:D�vR��U�*�����T�Gr�i�Q�T���]Q��fL����BDB�]T�+�\�2k�,��k�Q�Fa���L���+W�u.��7n�Z����l�a��I�^�;���v�3++k���q�*��G��|�avv6�W
��q����g�u|�B�3L�ld���(�rn�0�kLR|o~~~���_m�=�0 ��4Q�mH2�I�1a���[*��=o���v�;���)���hbw�,�
��`����*��SH'G����"t�~����S�j�w��NggY���Y��K�����h������Vm�����MQ����fx��i,�=5�7o�LT�j�`��a����m�D�B�%K��K
ffLm�l�?Z��������FMQ;���+����Y���z�(�:�����f~|6�����U0A.�{�G�]6{����KD�'�j��t]�fI�G���	��Tbbwy� u*d*�a�,7/�=E]
�9��A�HFUF���V�g�&��g�PhQ��|{pP\��b�HG}LBT�b�t��&"�UU{^LX�������g�������.��-T�;�zk���n^A�S<U����p9�T#"�UU{^L�[��4����)����f�B2���#����X��X�r/"�<FUF���7���_�/��u���{�
��;���Wy;����:�y	�������l�F�u��.E?�zc�����M8&�pT�B�r���9*BDAtT%""""7cT%""""�bT%""""�bT%""""�bT%""""�bT%""""�bT%""""�bT%""""�bT%""""�bT%""""�bT%""""�bT%""""�bT%""""�bT%""""�bT%""""�bT%""""�bT%""""�bT%""""�bT%""""�bT%""""��������a������j��\^^�������[-Z%oKDDDD�.��jmm��A�T���Q��DgM������� ��*R)�&B������,P;�v�-�AG����A����Z��WP�}�&���KdpTE�6l��3h�����>��z���u]O��zODDDD�-S�*2%��
�6�|�����:��0��l���{""""�o�U�!{g�,,,DE����U�y��z�$�O�����F���7)3]�pK������G��7�u,w�aT�uv�Tg���Q�����Z��C��a�f�!�c�R?N�7�u,�4�Uc�����*n"�"�"}Z�x��]Da�bv���&��U���M�Wu]�����%�av����E�[�>���	�V�UG�����T���0���r��"
�-Mf���NPb��3�v�-y	��(,�@�.����x��*���]Da�bv����Q����E�[ fQXniU���]Da�bv����Q����E�[ fQXniU���]Da�bv����Q����E�[ fQXniU���]Da�bv����Q����E�[ fQX�^r�>b��2���}�.���1���r��=��,8zm��<m�h�;�*�*y��(,�@�.����d���U�f�0�����7��oZ��
�*y��(,�@�.�����t'����b���M�^�~7�WeT%�cv����E�;u�Zd[Y0{M���ZT*�_/O��6%B��sU��	��(,�@�.�����6Ev�
��k~�(����k�W�����x���J���"
�-��(,w��[���M�����=�������s7���2�*y��(,�@�.�����G\��i|��?xLt$u�^���o1J+B��d*���}�.���1��"���5�u�[n,��=*�F��-��l3��k2��~�U���]Da�bTET�k�"O�V��Mxd�(�/g�c��3/��2��~�U���]Da�bT�����H��P���i	��>���v�����^��SU���]Da�bT���>[���|A���?X�����1T���J�FU�>fQXn�UE�R�����3�[�l\����M�'om�(*��Eu�*y��(,�@���x��G*C7C����'r��U�u���M���&��J���"
�-��(�[���������O���7�g�MO��*y��(,�@���d\��|��R��Yq��������/�>��)FU�>fQXn�UE��r�Fd������n��t}�����bT%�cv���QU��;i>x)4bu �!R��i�w����0���1���r��*�k�]~+<y�����SQ��h���]	E�5-���}�.���1����r_�
�:����v����qNE�����l�Z�SzU���]Da�bT����DN\
-?h�]�uA������#f����=�Q����E�[ FUQ����H�����3��;�/���MJ�~���)Z���OE�a���}�.���1�����>[���6'�NS�5x���}������9���0���1���r��*Jo��He(wc �����/�h�]�/�c�\U�1��FU�>fQXn�UE��r���=��#�?�>6N{�?o���x�lU�w��/���}�.���1����r7���'��,�sg~��b�������(�-U���]Da�bT%�r�4FV6?*�?4*:��5d���:��lP7�P]�Q����E�[ FUQb��7��}�����;��U��WC���S��f��L��	Z����K!�G�J���}�.���1��Q|�|m��W�6�b���D�gc��g*�����wgn0�fFU�>fQXn�U%���#V�Bg�^�����Wjx�~�aT%�cv���Q�����_���^�C�i��i/O�������������ky��;��G��F����U���]Da�bT���������O��9�����z7|�!����
��J���"
�-��W�����vH8�W|�d��aT%�cv�������6F~]�~���)ZEu�i�,�4���}�.���1�x��k�'r������������Q����E�[ f/A ]���v�������;<�����Q����E�[ f�h�#�.n����������b��aT%�cv�����*���Mj?��e���s�?�-
�*y��(,�@�.P|�8�
�g���v8���rK��J���"
�-�KF���l�s���wb��aT%�cv����%s]�
�6���T_,�7vvW�[FU�>fQXn��]\�lUx�Nc�c��@���WE��
����F��k��O�}->�����-
�*y��(,�@�.n�`�=S�|�Vv=d��3,�4���}�.���1���������0����\mh���t��9z���}���g��aT%�cv���������R����J�������=���/����*��7��w#qo>�U,�4���}�.���1��3�����C����;�P�k�{�-
�*y��(,�@�.}�zm����W��|�v�j�'�v�-
�*y��(,�@�.}l����1VH�����.�l�M,�4���}�.���1��_ ����WP=2�W|4�;L� �[FU�>fQXn��]�F������+�^��_���SSc���0���1���r�������y{��FY!�����G��Fc��aT%�cv�����W�4D���~�����~J��XniU���]Da�bv�=%B��k?��q������������Q����E�[ f��p�.��*`�T�R|Foc���HT-//�����T�Z
6L�u�E��m�K�]Da�bv�Y���������T��V?\A�-��*R&�&�3�:���������0���r���#"������.h?-U�F�
\�;��rK���������3���%%%i�%�av�������DW1_��~N�z�7O?x�?������&��*���>X\\��vT���4h�z�&����0���r����z-2{��Dn��:`���~����rK��Q�>E��'��z� ������Z��C�]Da�bvI���p������	�G��&l
����k��c������NQU�B��gIII�}t9h������m���!�.���1�t�������8*^O�j{���!Ua������`j�KFUJ��E�[ ��e�����5�^���9K�`��s��@�fc�.c��8<x)Tv#t�&\�i�Gv�
�37�������G�@����1�J��Q5*n�ATmll��ADD��6�9��)w��fz�W'�l�`����QD+��I�FU$K��X*n��F���&jk�����ge�Z����:�[ Tu��x���uQq3�����;���z�i����1����Q�����d���U������H�8��x����W�u8v�������U��������&����O��_�s���)��'j?���q�k]zi�$���2�\U���WH��I����0���r����D�o8��zf���\��[��U��fT���<���-y��(,�@�.��L����:aS��j-��4�������I4y[�fQXn�<�]j#_/�� ��f�g�2��}�`T��#Q�(	fQXn�<�]"�-�9���3��1�E�� cj+FUiU���]Da��Rv�v7u����.|TcT��Q����E�[ od3�<{�10�=�>=A�V&���DU�aT%�cv��(��KMCd����S����5f]�����q0�J��J���"
�-P&f��\Z��b�:=�I�/O�O_Y�QFUiU���]Da�����4��
��.�;�����}��A���U�aT%�cv�������1��Xp����;�w���������E��cT��Q����E�[ f����bh�#�i���+�+Rl�=&�.`T��Q����E�[���.�M�C�B{���5�JM�i��Z����b�G;���K���E�E���7O���*
�*y��(,�@��]L����]}�5[���8Z2x��ncT��Q����E�[�^�.�n������}����Z����s�F����I���0���1���r�����L�d8s�������G���L�j�1�7W67��:,���*��v7\�a6�U���0���1���r��������g��+*5�����*
�*y��(,�@=�]j"�,��!/�EC�5���4���}�.���u?������w�|��<m�9^�F���0���1���r���r�f��t��|7Z�;S]�QUFU�>fQXn���.M���
�m!�_��/��O]�QUFU�>fQXn���.[��OO��S��QUFU�>fQXn���]n����Oe*FUiU���]Da��4���"�n�w�	.�o������?����2��4���}�.���9�K�������c����+o���I�8~^>��U�aT%�cv���f��������{���	#i�������|*#1�J��J���"
�-Bj*��������k��[N�n�j��'5�1�J��J���"
��y�p��c����!u���[M}��?y����YZ�z7l�4T�aT��Q����E���V2���!�>3Q�������V�u���0���1���r{�l	���u�C����f0��"�-
�*y��(,����KK��O�R����"�*�.����0���1���r{��l^|�|��@)�^���;u�v��E�[FU�>fQXn��������!u���DtHU�]Da��aT%�cv���hG*C��C�E���'�]���"
�-
�*y��(,w�	��{�����*^/M�����vS�.����0���1���rg�F1������t�����4}kY0�����]Da��aT%�cv��v���H�Q��E��lj�>[�GH��N
��(,�4���}�.����t�^���|�@�M���e�
�8��<���E�[FU�>fQXnW��"svo�����z==A�!p�R��:]�.����0���1���r�
��|�>r�*\r!��xp�>3o�1|e���W����\ �|��;|6�=��0���rK��J���"
��{V1�_�q�6xlt�L���\}�~�JM�;�v��(,�4���}�.������h���Sy=���6C�z��m��]�q0���rK��J���"
����`����3���^F�����1K�t������FQ���t���PEu��������E�[FU�>fQX��7��0�����������Kw�u�>��)`v����Q����E���4#�h��d����4U�t*��;J���E�[FU�>fQX������'�G���.�
��(,�4���}�.���ih�G��6��0�!Uav����Q����E��K�������!�������RfQXniU���]Da�ST�Ef�0��P�zq���T�����.����0���1���rw
!u�v#�6�jOj$s��:1���rK��J���"
��D��:-�C���"
�-
�*y��(,w\�S�����KS��e�w�?��(,�4���}�.���Q�=��I��}��������"
�-
�*y��(,�M�I�
�o��w�z%�Z�]Da��aT%�cv����!�����r/����"
�-
�*y��(����:%&��3W�1du�E�.����0���1��"��w"'��6�f�	8�
�%�RfQXniU��UE�p����;�C��y��/���L�����zo�~���C���"
�-
�*y��(�)��;�-�������������>M�z���)!Uav����Q���QU�L/wMCd�~����Kc_����6C�z��6��UY!Uav����Q���QU�-�/)>j~��C�^OO�����\(�cl9,�����O���E�[FU�>FUQ2��{��<~B�p�?o����y�b��N8��;M�fQXniU��UE��rG��O^��x||�3P�������M�]�*fQXniU��UEqy���
��i<�'��:]������:��1���rK��J���*�;�]��,-5���s���|m���m&��1���rK��Q�����6YYYV�6���>��j;l�0]�����%/aT�m�>y5��:00;:�>=A��l�]�0�1���rK��Q�TM�3q:���z[�FUQ\R�&d�a�����Q��F�
��xK����"
�-M�FU�5�{R�?�������$���1����{����]��S�>]��W�k�{��(,�4�9WA�������
BU���$y[�FUQ���~�y���[��w�>>^�����������E�[�DU]��
������j�kaa�j� ������Z��CUE��r_���l}Q?b��A��0���rK��QU�}����
�=�6�Q�M���{�FUQ���f�y[Y����w�8�7�8p��F�;�.����d|Tu^ eg�������X6(]�pA��H�4�]r�:C]�^�W�{,�)*�>3�q��������R_A�Qw�
y��{u���&�9WUeVu���*"��C������C��Cg�^F��M�3c��#Vo��2�j�������a�y�j���{��R�h�!�K��n�co&�������[y�T�C�Qw�
y�-M�FU�g4��(U�����w��QU��=s���,���kt���$/�rfQXni25��p���F���U*n��7��l����0���i��v��[y��#�2��(,�4|���N�����[-Z9��'oK��*J�rC��v�j���n�^u���3��(,�4�}��:
@����<����%���$*w���G������M�fQXni<uYQ\�����;j93��3��9��Z�L�fQXniU��UE�*w���A�}�pg��0���rK��J���*�]�`�y��;S5�;S���E�[FU�>FUQT��v���-,1C����"
�-
�*y��({OU�Xr��x�5[�R���e�.����0���1�z^U]�������!�4gH}8�W��2�z��(,�4���}�������:�!�����~�>S�t�)UfQXniU��U������]��s�=���)w�7���.����0���1�f�����C�K���F�R���4=o�Qr!t��-�[fQXniU��U3T0���d��3�@}z�6bu��4��(����E�[FU�>f�������OM�s�#c|�-������Ce�bv����Q����%�����m6�G��;s�Y;�cWBVw���1���rK��J�����n��[��x�^��k+���\"�r��"
�-
�*y���E"�{���}B����me�p���r��"
�-
�*y��;��UG�_L�>!��B��K��O����E�[FU�>f���Ef�2��!���2��b����1���rK��J�����n����>y5�����3��W������������E�[FU�>f�^�����gn����.�gN�d|�"����S�!9��z�^C�i�v���	�
�-��(,�4���}�.���)������d8M�:I[~�k�������E�[FU�>f���u��-�GE���'r��f���V&o56�
Z��,�@�.����t7����4���V�-,,t�%rf���b��U���^?���9K�l������FQ���t�he��:\����S�r��"
�-M��*r�<�������&YYYhh���;f��!f�]�n<�}�)�L�8s#��J��[ fQXni�����6��S�EC�BV#�~����u�9��'G'�A�}���\�{S���r��"
�-M�Q���v��A����{�@�����z_d�!��9�K
�{����/��������E�[��F���,��2��{0�t��1R|�����������L��3o��MO��-��(,�4�:WU���co�0n�%��.�\�^��|gn�N=1^���8w����c�bv����[QU�Xm����R%Wav��:s����8	u������V��b����1���rK�����W�+��Jn���H���P�f��Iq�D�6zm��RF/������1���rK�Q���$g�����<��&���8	u�$m������g0�
��"
�-
�*y���r�n����w�y�>{WF���
FU��]Da����
kkk����;��/i���V��qB��s�E�MDX�;�bT��E�[�nEUu��SAEUu��VE�!'���|q���?.��<l�4x�R��1���rK������"�����Z��U��H�./�~3?:�~��_Q��}��Ubv���&���������~U��.���=��o��p��#[�}ZyG\HUUbv����W�j�VEn����4o9|���Q��e�
x�l��Ubv�����O�_4��U�=<�]�P��c��t���C�|#�����
��@�.����t+����`���M�=<�]�����g�:����}c�2����QU fQXni�U�fU
�����6� z�k���#Wj��n��_	��q&��Dp�a������p���<h*wc�zG�Ubv�����*���0���zmF�G�����) ��S]��*��(,�4���}��.�@d����J�5y�Q������@�.����t7�fee��P���9�*��.;����� ������KS�_���_��|����Qk6�m7��5�J�������=���/���R;��*��(,�4����bUKJJTNU�W�;7d������H}�"P���Ubv���&���x���r��_]��lnuJ���7��#�K�����������!�5�4FU��]Da��I?�:���0U4�#�=�1���
�1������}3vF�jK��QU fQXni�U�i��L����9���{�Kv����M��Q�;S]��T�`T��E�[�n��j_Gj7�}�*�U%������\�y��'�k��y^j�`T��E�[�nEUuZ*���x*u&s*�J_f����F�	4���}�QU fQXni�U�2B�d�p�yi�����!�����k�|��1�
��"
�-
�*y_d������O�5�i�����*��(,�4���}}�]~1������Uu�|��0�
��"
�-M�����_]G�7�"����2o��B��1��e�U?cT��E�[����}MU\�����]n�G����.������QU fQXni����*e�^�.Z���:]���0�
��"
�-Mw����T�~��]��UN��l��+0�
��"
�-M��U-))���J�Z��]���'s����mXM��1�
��"
�-M��*8X��sU�=z)��n������j5����*��(,�4�����7����a�S�������QU fQXnixYy_�g�p�����
�?*�[M�Ubv�����U���xv)*5UN4�w������QU fQXni�u�jyy���yY�\�f�����1����x#U�aT��E�[���`������{�lv�b	o��j��1���rK��J����e_9o��v��1���rK���]aa��x �����|0���%=�]��g&Z7R��7Ru)FU��]Da���������sW�3�*�<��-yLOe����Sy#U7cT��E�[�L����g6U#��H��������<�G�KeM��Q�����x#U�bT��E�[�L����Q�R�W���������t?�D����c�H��y���j��1���rK����:��������Q�|EUG�������t?��9f�H��l��Z^M�j��1���rK���������c��:@�E�M��zO�����G����*�cXM��Ubv����#QU]be�(�������e�2����-=�,�W9uh^���VCr�n��2*��[o��X����c�7�����>��Y�j�kQ�I�t�9�S���VS"""�aT�yH�����S��F���&jk�'�/E,����0C�/N��g���T3C����������:�[���*q�:?�IEX�~Vg��m�{��.{��>>^k�y#����*��(,�4�U�q�K�L���}V�:{�Ko�{��.�k�g[��7���T3��@�.����djTU���B�D�Dj����������1�e����F��QU fQXni25�f�>_*V�<�D��%/I#�l-����fFU��]Da�������R�Fvy6���j�n�H5�0�
��"
�-
�*y_��������VN����I���QU fQXniU��R�.�YP�k�X_�����2��@�.����0�����]|���E~;��6C���)��QU fQXniU��:�.�o��o;��������g,FU��]Da��aT%�K�]�2��?��:bZ-(31�
��"
�-
�*y_���D�]�~�������<���Ubv����Q��/nv�:���B��G�z��@�.����0����f����F�f�4"��^��*��(,�4���}��u�H�v��H�)��1���rK��J�gg����_������T�aT��E�[FU�>�]V6��B*^�#�W1�
��"
�-
�*y_��[����C*k�Q����Ubv����Q�<N3"�Mk�s�s���j����Ubv����Q���6���"�O�t���q��1���rK��J^���b�;�)y��@�.����0��g�X�s�w�7���u��1���rK��J�4y�a�����cv��QU fQXniU���7��:|e����EFU��]Da��aT%��|*h����CafYXn��]Da��aT%O�s>8`��S����7�gv����E�[FU���WCg[9���z���/��(,�@�.����0��G��
k��_L��}��Oev����E�[FU��u���k*�>���4t��?��(,�@�.����0�R�C0E<U9��\
��j���E�[ fQXniU)�5��_L�r�����8��gv����E�[FU�`���,]��G����Y-:bv����E�[FU�TF��7��G�?��;R?���(,�@�.����0�RF
G�?_l���|{���@5fQXn��]Da��aT��in��8�r*^�O%����"
�-��(,�4���y&lj����V���]Da�bv����Q�2����Sg�4��I1���r��"
�-
�*e�H�9g}���1�V��0���r��"
�-
�*e3��e�u^7H���(,�@�.����0�R�"������W*��:1���r��"
�-
�*�]Mc���VN}h�o{Y'���bv����E�[FUr�+5�g&Z�M<�w,�}��`v����E�[FUr���BCr���D�VQ����`v����E�[FUr�������S_�����fNfQXn��]Da��aT%7Zv��R�zk�^��~��.���1���rK��J��L:q��B*^-��v��.���1���rK��J.7�vy��SX@��cv����E�[FUr��|���M�����"
�-��(,�4���
u���3����[v��G��]Da�bv����Q�������I�M�f�v���M��cv����E�[FU�g[N���r��9��+���?9fQXn��]Da��aT�~s�n��E�'�>=1���'��"
�-��(,�4���|������mw�����z��n�<5	fQXn��]Da��aT�>�4��D��	��f��m74��r*0���r��"
�-
�*��s7���e]��^�.�W���A'fQXn��]Da��aT��P��&��/M��_��+��bv����E�[FU�]�p����9�G�6��h�i�QLm��"
�-��(,�4����J+B/O�p�dq���OK���E�[ fQXniU�W���|��^������������"
�-��(,�4�����f���@�������;��]��.���1���rK��J=�����y����5e���������E�[ fQXniU���9t�����j���fQXn��]Da��aT�����i��Gs4�V��0���r��"
�-
�*uW0�<zm�=S{�Q�icv����E�[FU�_ ����+��������:��"
�-��(,�4����;
�W���6��"��Z�
��(,�@�.����x!���>l����B�}����|��V��Y-Z%oK�:w3��������0\�7�
��(,�@�.����x!�fee!nFEUgU�y4y[�������VH}8����EQ�bv����E�[����j���QQU�����4�Rr�%�����O�����������E�[ fQXni28��=�P\\����Z[[;h� P�}�&��RQ��<U�}�����1���r��"
�-MfGU�T�OTT�mH��(���{�u��G�Z r*0���r��"
�-��U���%%%��F
T[[�����:�U��b�������-�]Da�bv����Q��iTmll��!��g��B��Q�	++�DDD��:V�qFUK*Q?�Zs�f`��3�gc���c��.\�h�!�c�B�Qw�
y��{U{KlTUMb���\�Dm���t��d�i��j�$�������",~X�Yo��Xn����
y�-�7�j-��m����;��[���2�����]Da�bv�����Q����j���t�>����+���&`��L�.���1���rK���
�!��y|?y[�����Rs�����l�VZ��;�w��E�[ fQXni<U��Gc�h��UT����~;��|���{���.���1���rK���J��7��l3��3���VF^A��(,�@�.����0�Jw�24tR��S���03����`v����E�[FU��}��U������B��zo��V�.���1���rK��*��������L}j���,h��fQXn��]Da��aT��.��������6�����T��(,�@�.����0��2���/M�O\��{Qu��E�[ fQXniU���E~3�����}�w��cjfQXn��]Da��aT���������3W�v�;�����(,�@�.����0�z������eX-�`v����E�[FU/�"_�����$��G�.���1���rK���Y�������1K���{����E�[ fQXniU�i�����VH�+g}@�Tq1���r��"
�-
�����#���A�����g���"fQXn��]Da��aT������f�����)Z�AW�'��"
�-��(,�4�����bhHN����.�k���S�0���r��"
�-
�����S�vH}8����i�#faXn��]Da��aT�xu�����?3Q;sC�%T�0���r��"
�-
�jf�z�G���Gcv����E�[F�LU��d�i������=�*E�.���1���rK�����J��c�C����>�*E�.���1���rK���a�]	�bJ������i�A�d�]Da�bv����Q5c���|�����x=>^[~�����av����E�[F�0�g�2�J}h�o�&�Q����0���r��"
�-
����8|nR�#�-�_����.`v����E�[FU��R~~�#�/O��_��S]��"
�-��(,�4��n�D�6�j�����4����fQXn��]Da��aTu�H�y������G�X'l
�������E�[ fQXniU]�����3����G�Wj�+���]Da�bv����Q������GH}y�^r����fQXn��]Da��aT�e7B/:���NK��QO`v����E�[F��d��n3���W�*�����]Da�bv����Q��\�~yj���O��v�	Z��G1���r��"
�-
�j?�[�>�������|���[�]Da�bv����Q��]��p���m�	�L�]�.���1���rK���wB�������;S����i���^��"
�-��(,�4��}�zm�W��w��[y�dJ��.���1���rK����"��������;S���W�������(,�@�.����0�����#���L8��h���J}��E�[ fQXniU{���#����;S_���)����E�[ fQXniU{E��t���x�1�v���]Da�bv����Q����������4U?�;S���(,�@�.����0���E�M;��5y�a������E�[ fQXniU{L����������gJufQXn��]Da��aT��o����~���zU����(,�@�.����0����;<�j��@��S���E�[ fQXniU���|������1��g�R]��E�[ fQXniU�WQ~>�������kwy����]Da�bv����Q5M�G;<+udq�oZ��m�]Da�bv����Q��I���!��1�u�y����]Da�bv����Q�k*��_���L��hhb�#�bv����E�[F�.�x28x�R��2�����(,�@�.����0����/b�T�V�����E�[ fQXniUSU�U���*�y�?�0���r��"
�-
�j�"�����_�}��0�.���1���rK����:CjFbv����E�[FU�>fQXn��]Da��aT%�cv����E�[FU�>fQXn��]Da��aT%�cv����E�[FU�>fQXn��]Da��aT%�cv����E�[�Q�����|���a�t]�Z��0���r��"
�-������
���1���r��"
�-�������xZRR�-y��(,�@�.����H������
B<���kB^��"
�-��(,�4��:�_XXh�o���s���E�[ fQXni$F����������
�������0���r��"
�-
���Q���]Da�bv����Q��iT����DDDD����r�U�����UE�fy��[�DDDDAbT�u}��a<���	@�[""""r	�Q��*""""�U��V�S����\HnT%""""�cT%""""�bT%""""�bT%""""�bT%""""�bT%o*//8p �Z����>����9�J�����=����,��>��gee���HB�hX]�V��s!U����,�zM��-�9s��,�3�(L�
���0hs�QgNU�V�A���t��r0���������
.vX�zKG=g�Ye��������[����qPYg���n�[��o���2�ZWG���>�Q�<E�l�f���\��������2��"�����J'��c��&���
�9!j��mB��FP��U:���0����uY*�-�C��u(����v���������]<CE{�E������<�KF���Q+�������1��7���b�dq�e.����bJ�v+v��L�
���qw���(��
�%:j���\FU���P�U���}lv5������2���*�3�z�� 	D-�\����J�J�j�0���TU%Pu��FU��Zc3�J��J�J�$���[���2��w�����e4gEF�Uy�j&R�o�n�*��.�\�{�*yVXQQU vn�b�Pf��g�E�����2�*_T:qF��E>�	e�N�*���0��7�n��V�n��26ZI*����-�-e,��
�4c�*��Y6�-e��Uz�
<�-y�*yS�*jS��&V;�4j�d��N'�;f\2ZlA���C�q�>6��]�s}.�*yS�zMq���^�h�*��G���9��i���D�i�9�K����>�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\�Q�����\*�j(Z�l�o���K�Z�b���]�v���~��w���~(((�|�r$��Hz�7}���?��c�q$it]�3g����o��m5��$�U8nll��PR>�k!�M�t�Wjz��1� V����5Ms���>i��#G���iu���&,SX��FD�	2 �=z��������
�F]�z5;;�|����L���/���[��R��X�a�7f�����[����k����`'G����7c6��4~q)��+l��N�ZTTd����u��5���ohh����z���bn_�d	F����V����=������_�u^^���w��Z!�o��	���;�^�����=�b�;r�H����2�e�z���/�\�jU��1�S���f�Q_aT�(�/_������[__o5�����RNN6���:�t�R�C�
����(]=�+�����!����7o��%p��u���}<�(L�81jjci�2��?��J���������g��m���k���7�F�u�����F|���f���j_����o��au���m��=�*�;��3f���������	���8�b��~�qf�&&T�D���U�u��j�i�|��������,=g����>}:>���Y���{;v������m���>�
K�����[tl5%"wsoT�:e��]_~����~���k�M��VYYYUUU��S|�0��7�`{��'��J.�]��sss��9����Nm�OY�}R�D�P��}i��M�5k�$���������:���Ts|*>|Xm5&M�y�e���,{��)���{�*��?��V7H��(O�ZQu��)h�'��+Ws�}�%��Jn`��;w�L��N����)Y]]�l�����k�E�Y-��$��bT�3j�~��={�j�@YY�����uV#�0a�v��QQs~���2�X%�.����a=�5��������h-
�����V��Qg���}��8y���(���w��w�1#\��={��$�7��*/�����j	���h�Vx�
��Y���&�����uk��^I�*V�h�`����l����\p��������Y3v�Xt#F��>}����c��MDM
5��������J~_:�0a�Y|5�=���������1c��c�%�����U0����W�3|��_��~��D_�^a���F��JJJ�����^���;7s�L�������=z4�)��?�"�;�9g�����+W�����|n�[R�tFj��	�9
39f6���X��0e�i��X�|9�d�vp�*�j@^^>�YK�}�-�����U�D�
X�O�:��Go�1�U����Cok&M���(c��.hQ��V�cr��x�&�
3��={0:������QK�WX}m���^v`���q'{��"yM�f�(v[��j����I��~{R�/�e�����IrX�����:� �����E����������H,eX��FD�bn��X��Z���/��
�+�l��j��a���\�u�+#|
�;���������Vy�`#��w�]!:�D��1.-}l�Pah������XIU1K�,A+l���H�+Y����KVwm�EQDu��o����{��Y�%e�]���D�w�}C���!v���>6i��������
�j|�d������=Z��������G0�0��N���s��9.�o�M��q�����"pc:�����W���]g0a��@�=�!Fc���
|�sRG�W��bO��D�B�_���c����;g���g
����G|����;��2!��N@��H�qRrX �a��[��1������u���Q�dT�����v����;w����{�����8F-����K5M�bQ����Ni,�X�`����?���~���.��%�%BkFNNNSS���
�,,_h[TT��|ED}��Q�6l���?{�,�#��bu���QkI�7+w������NWF�U�O����\}+h���q�(�vl���R�����[�������f�0��2���/0�����mL\��
�q5�������S%�d�S�u5��.������k�&)V_��N4
�	&8�6lq{�����g����83�z��)�N@oG��O�=WO�l�"�������s����	E�	|����j���n�*,��U0���JJJ��EXH�^����~���L�]1�b����V�>|��q��7�8�/>e�?L��S�"_"�`�au����Q�d,w��qg�w��3G�= y��D�N��#�7��RM��Xlv��Q5���4����c�15����
�C�g�������Nu���/�GJKK�F��OV�����K���^|�/����\����\U�E?~<V"7nT��^�]���'O�T�\'����T~��j>�p���
ju��4�?���{'���]����M�VYY���������b�^\\l���nQ��Vol�bE�P!H]�x�L�[_4T����M�'��9c�������V�ma�jkkU�I�mz�o�P������=��y��Yh����'i���h0��@���w7l���v�{��1�=��E�0}�%���@���S<�4&,2++��Z0=1w�[��a�T�C�0�T���h���!z��G�;8��6{�>�M56��
|�������.Z�(��	��T�{�������5�oUs�F����+*���:�	CM
�=f05y1��	�T�>c��=��&���/_n�w���:�����A�U90.��j�c��FPM��}��&��A��h����Q���x�����s��Q�0����O���,*�%:�}u�jLL"��o���A1���c\:�QC[L@�	j�����uk���o$�,�S�H�����*����t��qW,6�mlTU5E�N;b����k�������:N���?���m���`�[�`����j�fQ3�O��V#"r+wEU���-[�U�����]v@����Z	�Q�R�)t���6$����:���;kp�Q+�b�+��3��h������kF�����w6�nQ���M��=q���v�B��V}/6'����/F��\T6l���$��`81�f�)|/z��S{��-���3w�\���F>��.//W���c���C��O1@�1a����A�T����?x�~���;Ts{�������0�V�Z��w����;�r�!����[0)�9��F���b!��c=�ND5�A��aOj�Z�>~�aPU���
���l�j!Z�z�s�bt��<�2���K��gL�[�n����^+���H�*�a�U
�����7�����P�.9L=�Qj,0O���.����t����r�:������5|�K��	��{@T��vR�fE�@��V
G[{BA�j�c��66��]85T(��������7�_�>j"���v�$J�0����o��i�cH��5;��|�2a���[�+���.6�XE"l�k�Dka�j�U�VB*�F������/_�^a��bCh} ���QG����V�(;F�nQ�ZM������U[w�������-`Rl��	c���j�%��
�i��H%3DO��`��x��� Em���z�jUUU��=��U��j��4&,�@��2��������M�=�W�\�����UG������n�����`E7��V����
�^��$'5�~DMjl���G+T����
���l��#��!F0��w�+�3�N`����-jx�i����������� a�����9s�hhh���:e~o`�1��E�5����5�8�$�45�G����+��?�.�v�rV*��B��mv[{��.-�h����	s/~`tT�]�%H}o�W0HX��T�����j���� �Z����\U�c���}����Ud��0��q�rss���ko�������X����R�h�v������m����-�=����MH����2����-��#G�����h2u�T��[u������?���(����CM�����a�����`07�6�������2������YJc�F�|�����kiiiAA�:{S�&;5�=�qu�	g�7l��&�N#��&Q�QTg�����\��q��I�l��+W�X��`��!f�gc�+�'��!��H$�D���������fL��Kg�����f�|qq���40���=21b�k������`��	�����c�	�������:�N�T3<8����X��XT� c����E��rV*��B��mv[{��.-�h�������V���RSS������>8����������]��=�#"rKT���XOa�<}���\�d�b���{n���� >��9�Xwc
n5j����\Qm�[ {���Y��D��B�nQ�]�X�c��u�p��q�_�6�����S6t��7���Y�fUWW��'��Ro��9G�7��0Hj��`Z���JJJTb�7iLX��q��������Gc`0����M�0���W�����DQwdt�{��Eh���U���J�|?7���@�q/�����q��=�
�}c��[�C��t+//W�R�J��eGUg-����~���j�V0�00���l,��'�������^0��xNN������k�;I�%�[��7�>b{eC��|�s~��S�6��P���/hq�U��?�������#������������m����sQ��)S���)��b�O}"r-�DU��|�k�����7����O-]�����je��-��c{?Pj���r�Z5�6�
��o���^	��ORb��[50�p~56	�����;w�
6�l�S���A�s[e�l*�-w�������'l(��a����4�x������8�2X��������n�:t���jj���[��-�I�e�Y�uQM�Ng{2:�]�����dru5��a7o�Di0�F�D���H$l=?����� *�����������"y��;I����'��=1���+��`&DDS��w����R\\�	*�����a�I�c"u�p������&L�+W��X�BU�a������dW'��=`���s�0<(7��������~�=��M���d "�rKT����^���&�k���J�Mi�D�o���W�X�+n�f#�!I���/�������o���'|Ia��-��c���uee%�!����oX���7���Sj:|��wq'>>k�,t�<��zM���k|u�p�&!v|S���=v�RF?o0�`0�0q0$�J�F�`o�:����>W�	_�p�Bt�,���*�0���D��.k�/
up�>�%����s��Ng�d���=��
�,Yb
nbv�H�|�U��A������RTTdG��DWs�R�@MM
��J{�ysu^^��b:����3~F5�r��I�B����(^���:e������	���D���.]R?8�*��B���"�Y�.-�V�V�{U�0/���bNFO�%`���ZuW'���V�����&f]��r��hWw{�a���[evTU;M�c��������*X`���������x�m�~�-��7n��`�:�N����-
Z����U�4`��M��7o�T�2vKM�����Nuuu������e�3f�-�~�.v|S��	�j��s+*&��}E������P� �rf������Y�={��I�������h�&���A�u[��lOFgC����c�0���X�H/��0��QA?�T�Jt
r"�	�O�"���+�W��Lr����{Fq�ud�W������T��0lH��G��������P����Sy1��b��Y�.-�V����a��D�����*�w�*�m|
_��z���T�Q�[H�
JqN �~����X��US<Xy�88o�<�����y.777v������2a���j�X���
���n����L����7��#_�x��������Z��+:�{�$��l��?�� ���9o�W�9-lZ����*`�5U�E��Q��\��E��+W�n*����Wc����'l��m��:a1�j���;`.\P;���T�#*�a3��_�/U������n���0�)���=��Q'���qO���B���T[������dt6DNR��0��v���MX��b,�4�k{�A�E�UO�8�>gee�����j���\W!9a
p��%��:+�9�I�NR�������]���>����+��f������Z�����k
<�P���C��Wp�H�R\��n�������B�K�IU��o�~�*,�j���<�E
0�b��}9���%�&�h-��`��V��L�8�4�H�R�&_m0����2*������;v���:(��~��a|�bE��m���9�@��2:Sk||[b�<<��}b7?h�v��c�&�6f�{$�#G���|��W�p��W�X��1<HfXq�I�����aP��r*���B�����6�G/^�v:ba_u�m��?��p~5>u��-����2���#q7�������R7I��ab�M�� ����;�e_����h�V�!z��_���%;���aL.�d�F�T@�P2{>���������=�-0�����T[��%�5Qo]�xQU
������bQSTV�3��H}D�*q��KQ?G�T�_�������a��ap����w����a�iRj�"�`
X�Z�y�����n5_as&��^a2�s���1[b�����=:/��RM�E_�e_}���E[�ih��QW����c�R��=����R��7�EZ����)�/�uc1�||/&iyyy����������*!����Q��m7~��kpl�������]�I�.����Dm��{����W`57e��&v>������3t�>�>�mC�P%���34������P�A��a8l�1�h�I��U�T4�_��D��
#x�[�SU�]�E|��
���?|�W#��h��$��Nuu�b��-[������ 6���qu���qw~�Yn@�S{w����Ej>D��mO_K���N�#f?��&Za�b�!���a�;�1�Hh���!;v����
��&#�����LmT�1����{*�b� �����:���3gN�#�]�B�?v�KN�����J@�JLe������������	;���get��Tu�
���
����U��������).����g�Fst5���)7����l��B<�.��+�J�>;f���=�D�B�Uk�S�Na��b!Gb�o��������\u���bQ���F:����`����g�Z�;!N!�:�*���3;�F�
�w�]g�I��/--�t�*������*�9�?�w�[4��D
	��V��N�1aQ}ug�X���]�T�w�xGY�s>�`j��9�
��������XH3�)�D��=%���B+L��h��s��Ng����
f��V�F�J��0�M;�@OEUhllT;�����7��`z�A&s�����v0VWWG�v��J�4u��XH��tj�2~����[���bj����~�\�|Y��Y�4����$�)g���0j��K%��tJwr�|��9�g"r������4Vv���Ggj+��)����*2j���s��Id�E;v���wU+�����FE���Y�|9V�X�a��_m���a��j���������J��:.++�6m�����{l���=�nT0
�j���*�a�``V�\����I��V]�z���{X����?,[�,Q�P}�]����Q��]�D��|�D�k�5���iLX�E���e6���j�@n@s�����I�<3e�w�����h��D�G�C�C�&>5b��� �}x
�����������b�!���aK4������b�m���j�z�����c�:���}���SQ�u{V���FUT
�M��(��,������050���
��}�;�j���!Q��w�������Sq����t���8v��j��{��
�0a����{7�����+0�j�W=�j��.��
G�%��0�+&SU�]����=����&�*���P566&Y����Ug��&6���l������0�$�Y�+z�5�P���[��;��h�B���s�qD[4W��[�NJ��I���T8�>�k���?q�Mb?�p|�|��������v���U��#�0vz:?�FHu�l���;������bQ��S���4�pl5j�f��C�-�����������j��9���H��}�i�:�L
�qtc5j�oW6���f����q{x00���'��b%���=5�U[5���mz5u~���e��Y�����C;c�R�J�������S�A�����#��XD��w�Qqn��F}������������v����j���YYY��-���^�~]��;wn�D�M�8p����\lh��D}�?�>�e
K��_~y��a���j�cT�%}0a����R�|�����w]8z����<���%MMM�)�]�#:Q7y>��w�_|��K������Q��1���>���pX���n}gV+��	�����'O�������N�R��r�J}��QK�:����3INf "WaT�y����o&l}}�:!5��w��4�������=�E9�!oGU,MX�V�Z���jDD�����U{I�LX�������O;����L�r��!����1�H�c���oTD��<U�������e
K����2�j�������UZ����/`�R�z�/���
��^_������dbu�vW�!@�6�.���1���e����2�*��*��*��*��*��*��*��*��*��*��*��*��*��*��*��*��*��*��*��j~���o�����=j�nn�|���!CTst������%"""�.aT����Y���JJJT����A������j������<��-u�j������-[�\t�u]����E-**R�?����#G����������Q���K�>��Cw����;���?����vr�|��?��?�7���DDDD�F����_��E��U+���ZUTT����}AA�z�P����z��w�~��V#""""�
F�v��H����������|�����}��G+++�����w~�wv���:V����RWW�����������Q���{��,���<x��UNN��	��������JA4hPmmm����>���G!�X�']���B���V�G5���:t��O>Y__�Q����e����=z�zC^�r������:��� �X�']���D"����N���g�����Ml}�����!y[�}�p8����4X�Yo��Xn�Pq��zC^�r�D+���Q���c�7��z�v����w��0`���~u�?�,m��V#���7ob�f�!�c�B�Qw�
y��f���n������t��A�:*�2e���a�����H���?��O��[�VH��~�����������%�av����E���U��B!d�����������V���W�^E[����_������A�:t��	k���=�.���1���r��j���/,,|��s�=w���*+�0�o��j��r��}|?y[�fQXn��]Da���Q5��ig���u+*k"�^�|9���%�.���1���r��*Q:�]Da�bv���
��z�����*Q:�]Da�bv���q�|�/��o�������D�`v����E��gU��7�?h���Q���y_U����"
�-��(,w*���,��l����=���V��1����E�[ fQX��r�j����C�Zr��'j+B����Q�(�.���1���r��]��/N��l��/M��n�L�n����J�fQXn��]Da��o�A��IB*^���+�������*Q:�]Da�bv���D��;��&X9��B�z]$�sw�gT%J��(,�@�.���i��������rj�����=�$FU�t0���r��"
����z���������#}S�w{�9���D�`v����E�;
�������3��l��R�^����3����E�[ fQX���r7��\}`��S�V7�4����S�Q�(�.���1���rwI���3uu��'�k��u��r*0����E�[ fQX���\�4U��S���G+C�����0����E�[ fQX�5���.�}�~�*����'��J�fQXn��]D�Y���p����������g��N�5�4�3f�2�n3&l4�mh�X��D+��7O�\��M��cT%J��(,�@���,����g�/M�_���0Y��=7I{&O{z��T���x��q���Q�kp���x}��_u/��wOM�Q�(�.���1��"��H�#V�������Bx}>_��t}�&��~_��V��D�`v���QU9��p"�����S�O
��.��X@���(�c�����[�����G+Cn���|'|�&|�n���Os*0����E�[ FUQ$��F]��U�;Sf��6H�7�E��Gj"���|��z������l6�-���w��D�`v���QUo�qs�q��Y�z�^�L�����|�hzU����"
�-��(.��3u@kH}8�7q�q�N������ FU�t0���r��*�'�
7�?a�jV����L�w��6����=�Q�(�.���1����r���ZxvR�����-;S{��R��Q�(�.���1����r�3S���qg�������jcT%J��(,�@���x��-;SW�s�L�!p�:#w��U����"
�-��(�(�����3�w��4U�~&�w��U����"
�-��(W�P��zm��"�������|�����I�=S����\��M�����J�fQXn�UEq��}������sv#��.�����bJ�S��=�m�I���)��������jcT%J��(,�@������7���/��4'n2>Y�6Gu��|��d�6xl{0u������\�����6FU�t0���r��*�K�}_��V�
��_�����b������q�C�������W���+3wNO\
WT�k<�/��*Q:�]Da�bT��0#���W2,�=Gi���m`�l����b��M��Cf��P�����������Fs�����J�fQXn�UE��r_��x2����uA���g�4��}�5l�>jM�`���T�lU���������H�?b���H��J�fQXn�UE��r�i�-��i|������O������_/M�,,?d�������~�^�x����bT%J��(,�@����j�������������r������>��}��?w�q�"t�:|�~�~S�\��m��D�`v���QU�^*w��p�~����|m����w��7�N��l9�_�E�����Q�(�.���1������p+<m�1l������_���X�r|�������Ou��%��J�fQXn�UE��r��6��L����!�W���%���PEu�5�<��%��D�`v���QU����oF��	~����4�gm��0��Q�����;��z$�:�.aT%J��(,�@��r���Z|m���.�\��Li�HM�Yu���Pqr�R�_�}.t��k���{��D�`v���QU��g������������o���-��X��>0o����y�r�I�>��ZU���|o���$��QVB}l|��MK+B7�E���%u�*Q:�]Da�bT�`�����w6�������&h��k�L�5[w^K~��X���|*��|0o�����'K
h��3y��m��k-7C
�X�aT%J��(,�@��������3m��?5���i
�M����I����:$�������KS��w����o�k�"~�iaT%J��(,�@���@9bu���+�?Y�_��L����������B�O�3&l2�]�g���$�N�Q��������`����[U����"
�-��WU��?]��v�~�����ag���f�h��Ej"7����������y.���9c�1jM��E�/��[N[Nc
0��.FU�t0���r���G*C�g�z����WB���n}�)��1n���T�m[�Z��G�n�������J�fQXn�U����P����7f��7���V`�x���]]5$G+>j����,��1����E�[ f�~Wv=�����Dm
���a
����/�]��l���\�y���v3FU�t0���r������#��=��;�����9�%�����)��)���'����P�OYn7cT%J��(,�@�.�����;s���O�����:�f�_�c�����u�����V��|��o�X��B��p�MOYn7cT%J��(,�@�.�����{�t�����i�F��5���-~�2`?�T�aMX+��-�?2���#}c��\��w�>��f��D�`v�����_���|��o���g��`{�T���B0ku@P;�N�	�������]�?�q�~�����v3FU�t0���r�������|���������7���`���^��bpdq`�X��~T`]s,��t�T���ix���,��1����E�[ f�>v�1���l�Gs�����T[(�|��"��6��j�q��f���<�o5���>��b��Xn7cT%J��(,�@�.}��)��R����P��6��F'�RA`�U>|9��>`_;���b��+����r��*Q:�]Da�bv�3���o����R��k���r�
��������Z=��d�����?.���U����"
�-�K�h�G�k��w�Q3�G��#-��:v%��9��!p�.���v3FU�t0���r����@d�Z�����}��M�4s�
��NC��z����U����"
�-�Ko�����?k��g������6���U����"
�-�K�
#���^���������[N���U�q���o��c��}+�p`����@ `�h��-y	��(,�@�.�'j��90�����F����~���r��j|H��F��������B�Qk}��W���V��?��	&�y4y[�fQXn��]zI(�<c��xkN0�7�y��s*��n�����5�8�Qu���H�����[!�����O���<��E�[ f�g���
��c<���S����H��>�r��jg��8p����_����jCC���C@���U�'N��������;mK���"
�-�K������*>��)��b����'X�Sg�0�|����r��j�7n���#F�8v���>hG�����������3�[0M���>����H��jD��"
�-�KzB�����}���sdq����7g���kCr4�D���:�%��6�%���f�� Vfgg�����e����Q������{��m�6�V���z��'���������0���r���:����
o<����|���y����?���P���c�ast��e�LW�T`���Q�dM���������ZRR����z� �4���6y[�=y��(,�@�.�����V���9�O��A1E�jj�����\����X{�<Z*���7���Sa���Q����g�M���g�&��AT�u��2SYYVm��:�[ Tu��PE����w�Q��f����]��?���L��I�����zdL��������J��+w�o�a����\�z�J,w�A��BO�U�M�2q���~��_��_�����H��~�mCC��������,6����+MMM��Z�c466VRf����7�u,�@�8�n��re���W��]�������oO��J~�����mxhT��M��
�k�p����{w��|�z��[��T��_�@/2��{u���.F�vH�������[�n����+���u�U��?���_
������~�L��_���zC^�r�����1�7�O_��m��������������g&6��M���Xm�l}Tq`Q���<x�Z���y�����z�id��o �X�']���.^���a���g����:t�0����W^y�����v�����$77�'?����?y[���(
�-*.��E3���T�����I�2M{j��pv{6���<��E�I[��������7�7�"��H(l�!�I+wfaTM(�\�P(�i�&������}��Gy��?��?<xpeee�m�{�]Da��].V����4U0�C<��}��@�s������K��;
�p�EQ����f��	EEUhjjZ�x1��:1y3�}|?y[�fQXn��d�?����n�>x�O����V���_������)R�������={������T�u�N�>�D��%/av�����%��|�r����&Y���0Y[~�,��Uir���z���1����E�[ og�����������'���.p�ZHZB�1���*Q:�]Da��jv	����?[��o�����d��^XhJm���f��D�`v��������p���+�����M�n��
�[�{#���1����E�[ �e-Yq���|�gm�O}����|��Q���v��n��J�fQXn���]�]	��20����g'i�����'n��#U��Q�(�.���y#��6Ef�2����R��e���j��3�FU7cT%J��(,�@��].�/9`~���d��3����c���p�15���1����E�[��.��E6�
�Yxo��l�R6V��l����������f��D�`v��(��K�?��<��������T����������3�����0���*Q:�]Da����b#���
��_�_��g�F�����o������o�M>9�3��n��J�fQXn���]"��SQ�2�����?=�=���`��`�q�R�JM�G�S���f��D�`v����Q���
�z�;�������SQ3�?4_{��N����=o���l��V���	�kU��Q�(������.�>*���eX��G�Av	�Z��_�Ej"U��:.��]�*��-�<t�f�2?.��<M4�=�>5A�:��X���P��H��\����f��D�`T�m���%���3u$���h<�7�jv�x;<u���������X��l���������@6Gs������/��Sx=2F�t�Q�q�R�Z-���]��n��J�FUQ\U����q��=y��Q�S��U{^/e�u��=&"�c�K���zk�>e���By�w��)��n��J�FUQ�S���C�,���qW����5���.w#E�-G��r\����q�3�'k��h���n���B��K��,�g�
d�
������������p"Xv=T�����U��Q�(������f�y�!���-�����~��s�������z0�4�#k��H����|[NK.����_ G+C'��N_��
���x;|�N���������p����=_�Z������f��D�`T���}�~d�&ch[�y*W[z���~~���X%�H!o���g5u)��Hv������w+�/�Uj�(�'���OQ� ����n��J�FUQ�������^�4��:o��w�6�[r��M�j��������,<6N{�@����t'�+���]�P�������W���-�����<d��
s���0���*Q:UE��r#��<l�=�=�d�
��
���S����3y���������vk���k,���k�N�)���.G+C9�o���=���iZa�Yv#�X&7bTu3FU�t0���/��i��o1���v�>���7��u�)�7�_�jup�'G[Rj>1��>���	�g���
P��>��e�37B�~]<������M�a����n���f��D�`T���]v=��2��A��g�q�����9c��������f�4�~��?�m�4~`
o:�%�����r�nx�.��y�}�����M��K����n���f��D�`T������������#����]�s��:���D�K�cWB�M���0rM��)0��U������e5Gl}y���*������(��]���w�;����������������V}�%���n��J�FUQ���uM�;�_L��c��{�k���
�_m�d�/�j�����������	�� ���m��vI�C��7f�c��v��B��r�nx��`�^s���G���������7��[O��a������f��D�`T�o�}�F����=so���x�k���;�sF���[S}Yd�q��3��7��{������[�����?h����-O�:�
�Q���^��B�p���6��"��3u���[�0���Z|4x��w��<��n��J�FUQz��f��������V��Z8q5�Q��.���@x~��/�6�&�3y��������:@���������]���aT�������Lt��b���Z���'f��KS��V���5�����Y���Q��U����*J���F]8os��7��1��Mv�?�j~����=����d?m���:�������T�nb�o?�r�~��RO�j��?�s����o�������������l5�G+C�w����&7cTu3FU�t0���{��q6��b��9V��,}��`}�s��=�����~��<��O�Z�G�����d�[No]w<�q�u�@�A��&PTj�\hy�����{���1���QU��(wuCd�v��3�������m0N^
��#$-u&d5��l?��A;i��F��@�2���v����&k�,���m�8Du���y�ObTu3FU�t0������{>�e��8+Z�6]/>���~2C�o��B��T���Bv��v�?��Q%�-�����v���]w]
UT���"����1���*Q:UE��r�6Ef�4��i��A�[��z�J����q����z��s�����gMO��K�]8�7��Zd[��k7�]�`Tu3FU�t0���#��D�V��[��oG��dm�a�z���������������>2�K�5E�Y��:=����t�7jw0���r��*Q:UE�~��k��=����e�g��V]
i)_���`��������D�3	S/g}``���O����	��]��]Da���Q�(���t�����L�j�8��|�������RTt�:��E�? n���x���T�F�V1��"-fQXn7cT%J��(i���)�o������C�|�-��Bs��UUQ7��Uw��"����8T��0�7g����3���r��*Q:UEI����������?��-�o^������H����;��=��sv����&P������E���U����*JW�����?���� }����b��wv�:�<T��Q�n5���o���"��^8��]Da���Q�(���t��G.��L}i�������=�����b�4D�i��Gs|��{=��s7C����cV4���4fQXn7cT%J��()��������;suuM:^�/��:����O]u�#�C-��M�f|��o?:��I������fQXn7cT%J��(���F��~���D�=S�
������>�����u�o�e�9��}.����u��hN�H��k3�
'���~fQXn7cT%J��(��}�fh������@5bu`��P���5����mI����z��}��)��dpdq��Y�}��������K���C�au�K�]Da���Q�(���$*�}=�����B?B��Ro������3���W��:��4���v7�����r��S5�����h�.�3�������0���r��*Q:UE�-w(���B����3y��'r��-���P��p�I�Y����wo����<w34���=t�5U�zn�6v]`[Y��:��7�O��E���U����*JT�����o7�j{J*�"]m/�ipK*�m�<6�%�
���q�9�P�m�n�p����]��w%B7��}����E���U����*�]n���;���?$�JT/M�
KZ/�rY �f�u�j�2�Z�lU��B0wc�US-��7�T��6�[�gv��v3FU�t0����}�J(g}�S�����^(�����a{�u��f��>Uu�C�B������W>\���
j�j��q��+h{�F����U�.���n��J�FUQ�+o�Yv��y��
��9t�������s����~�F=r�fx����c����K����_��=�v!�z=7I���q6x�oOHM��E���U����*��wE��iP����;�S�B.ITI_i�P��O��y/r�2��Xp�6#kU��B��3��&�_)�^���'l,9`�+U�sW�gv��v3FU�t0�
��|�������>��2��s��&7�����u�3�`���c���[�o���-�_��kM7���X�����+�wk�K.��T�o���^�Cjz�]Da���Q�(������3��Y�����4��a����{D�y�u_��l��[�;�g��3w�:��P��&i-���,,1���\]�F�w�Uh��]Da���Q�(���v�.��)����1���W_�����N#�jf���]��k�����g������e���������P����M��}�{��(,��1���Q��]
!i���|�������C���e���W���9��G�J�����h���c��U-WG]�����g��6fQXn7cT%J��'�����w��Z��#c��[��7�U[���s�������Z�E��vy�����}�����YY����
�.���n��J�FU�������oe��&i��5����rg���������me��/N���}h����:"~��p�\��7�]Da���Q�(��s�Zx���}��������z[���r�R#��)�L�'>[r!8v}�����'����}�~���P�����*fQXn7cT%J��g��O?X�W�8���!p�Z�y;���D�o�U�<y-��G���cWB��o����v�4U���8Z����R���E���U�����
�Zd����g�>5A[����}��;w�u���];1������{�t�:��&i�[��C�zVw^��"
��f��D�`T��Kw�c���G�z}��������_�^n�M5�/�k~3Y��DZ��=u-��dp�ns����t���'s����������m�.���n��J�F�L��\��%��m�����JtUP���o4�bJK"0�w�2�#�z7\ZZy���b_�`���i��qV���g�w��~&x��CWN%��"
��f��D�`T�\��H�Q����P=6N����T�,�y���[�s�mh9��/r�*��Lpa���>��R�;s���4;���'�k_��7+�H	�
��(,��1�F;t���m6l�`5mS]]=b��v��E�iZ-Z%oK^����_�Zxa�����i��#f����@�O_��xvRK4�:�q�����mw��_H�/M�?[���(*myj�MO��*E�.���n������h�����r&N$�O?����T�������RlK���qn���k�[`�L�il��P �e�������j������s��U��;��'���������p*���]Da���Q���SO=��c�Mm��XUU�:@lEE��"�^�p!���1��$`��������N�|&OC&C����7�=m�ag�����[w�N�dh�uz�2TY��y�z�1���r��j����e��97HK�.E�TM|>�����z������{��A�3g�O����Q5S��
�o5^���3�����g�]���7�}�f����-�No�G$�:M��E���U2M377q���o�^����S�NUm������������e��V#�FU���E�4?Y���w�2M_���x���]z��F��he��������S�]Da���Q5�������[o=��3EEE�h�����6m���Q��������������0��Y(�|�bp���3y������v�������[ fQXn7cT�C�"}����O�t��qH�hXRR�&���QF
T[[����>B0�,e���J���7�&��5���{������3
���]�gu�u,�@�8�n�!�c�{������q���M�<����Cf-((0M�7�jcc�9�L�N����zC�p�����n}���gc�TH2�����wT�*;ou��[ Tu�������Q�
=�bTM���W_U�O�<�'�'�a��^@�����@�C�����X�Yo��������M��B�C�|�.��=��~���Xn�Pq��zC^�r���_�������������������=�UUU����<w�\�E�)�����_|�������E������.�m�!_���8s#�������<yQ���U�UTT�����~��_�G�jr�������;�����_x��EC����y���������?y[�f�8Z�����O�.�8���
��s��'/rg�bv��v3F�v���~��������^�M��������u��6�-��������������:mK�������������W!��g���I����[ fQXn7cT������?|�
���y��������	& ��m�l�b�O��<������6��kg����[�C�au��Xn��]Da���Q5��[������3*k"�.^�8���%�.���Yq��p��3��Q����w�
�6��m�Yn��]Da���Q�(�.������M��m7�&O���(���G����1���r��*Q:�]��/)>����������l����|*�[ fQXn7cT%J�K_*���b<�o�L}j�6}G����}�({�[ fQXn7cT%J�K��������K�{���p����=__��V,�@�.���n��J�f�>PY���xq��3���������z����c�bv��v3FU�t0���@0��t����G���T���������`�bv��v3FU�t0���k���;�W��*�>6N��)p�2����6�[ fQXn7cT%J�Ko�D�V��_��Xkg��9���f�����jc�bv��v3FU�t0��8��
�z��3u�X-g}������'i��1���r��*Q:�]z�����;��������y�^?�L���1���r��*Q:�]z��k����G������,��V[7`�bv��v3FU�t0��3�t*��B��Q-!��1��
-����ew�����E���U�����}������KS����M��3o��,��b�bv��v3FU�t0�tSEu8wc��q�A��7�j���mXn��]Da���Q�(�.i����~���pvKH4Z�.p�J�=�?
,�@�.���n��J�f��4�#����e����<m���5W�'�r��"
��f��D�`vI����g��|�ur�{��O}�����1���r��*Q:�]�$i>|)������-!u�h��5-�Ju�A'�[ fQXn7cT%J�K�t3��X��y�A�g���������D��c�bv��v37F�+W���3G�4�������Dn����z_���|�����������������E���\U�^���K/��?���k�T�I�&�����|�V�	Q�cvI��K��?k}�����U���BV���r��"
��f����i����������SWW����|��'h�V�@5$�_�.��4F�4�m;���Dm�N�JM&�wb�bv��v3wE������{�7����S�EC�BV#�~���H(��3u�����-��jy���c�=��;��1���r����jmm��A�&M�d�w@C�B�{�~��������������H��[��W3�J�DXn��]Da���]QU����z�_��_N�8a5j��h�V��jD���]����c��o�L}���`�/�Sj+�[ fQXn7s�eU[�l�������k�EC��:"�o�.N5jgj���H�7����_�����E���\Uu]_�d�C=����!w��{0�(�;SG�~3����y��;Sm,�@�.���n��������G��Z���4D�J[�LU;S��nly�gv��Xn��]Da����Q����g�2���;S�-��8l�4zjg��QU fQXn7siT��k���N�<�i���K���k�#r��ytii�������� �zog��QU fQXn7s]TE$�<y����O��ZXXX[[��O���;��������r�.��,8���T����4��{sg��QU fQXn7s]T]�b�������_}��_��_ �������G�G��|Z����R���**5G��<�
�Cr���.���|Tj�0�
��"
��f�����!�>��3��_W�@TEs�����:�O�"���]������c����������$T�zk����y���;Sm��1���r����jT<�����|>������%i�V�q&8k�����y�nT��}��?y��_��T��@�.���n�������~����z���k��z�����E+�Z�\�����)r�"�h���*��t��QVB<V{g�>n}`�a���PF?�?=��1���r����U��i�_��_�������������������?��?Z�r��Q�Rv����������_���Z	��l�/g�?�,>`�V��4�K�6FU��]Da���uQU�������SwP�v���<�O����R��l-N�n|�������/N��Y(�c�>�^	���FU��]Da���uQU��{�p��Z���KmSd�6����	i��E�������n�������QU fQXn7siT%r�L�.���'m1��P���~3_��Xs�<}=�0�FcT��E����Uw���n�:uU{�*w���dtv������S�����5�V�j��PbT��E���\UO�>�o��o��
�u�����I�������[�f��w����n��OM�05�!���1���r����j05j������Z�
Qu���������~���h�V�����_ehv��g�
��P=��-=hL���1�
��"
��f���


C�9r�i��o�~���[4����=21���
�vy`@�T��k���F�95%��1���r�������~~>���s���M�29U���?��Q��#����+�/���x�<U[w<����nbT��E����U5M{��7�����3�����yyy������������P{[�N��Ufe�C�B.�r�/g��OCa���QU fQXn7s�eU��m�����x���sss^�?���|���'OZ���.�.�~]`��7f�;�yK��bT��E���\UM�\�f���~;y���w��	���Tr���.����5[W9��9���<;5��1���r����*QFpv	��7��6���,�\������*��(,��1������6�=|a��S?[�?V������*��(,���.����d���As���#�Wn�.~�y�a��<���_/������-��1���r����j0\�r�_��_�'TE4hPmm��)Q�rmv��E�������
��[����Ubv��v3wE������������g�}���@�4�S�~�����G�3���S��*PY���Ubv��v3�E���{����S��\�]�o��m3�������}��W�2��FU��]Da���]Q	5??�7��M]]�����\�]��������Z7O4���!p�sj�aT��E���\wY���W�{���?�xT���"^VE.���r�jh�V��|������&n2�4��T=�QU fQXn7s�eU�7o~�����:�eU�n�.��#k���=1��Q��������M��=�QU fQXn7s�eU����'�|c��9���\���l.���PWP���$m�����!>��70�
��"
��fn��j�����\������%���9�����}��_|���qgjoaT��E���\wY�������K^VE.�/��h�q68bu�gc����N��n7�V�����QU fQXn7sWT�u}��)�W�������l�2^VE.���I���p�������!9��W��5�1��1�
��"
��f�������
��������=�2���E6�~��?0�%����}�^�Z-�K�#��1���r������i�f��,�������=;����[�����U]]��y��kK^�7�%n>q54u��L������^�1�S����@�.���n������S_x�k/�����/^l�u�$�������v�
6�y4y[��>�.w�GV6?X�0�%�>���h��!�.�E��Ubv��v3F�v��_}���/�����S�q����������9�:X�f
���/�h�E�r�J*m�cz;�����xl��3���Z�f
����Ubv��v3F�v/^D0>|�}����k�\�L���d�
�j�O�v�����%�����4o+~\h���X��K�������oUbv��v3F�vH��~���5A]������o������T�� �������y0L��jD�K���������������;����|��1�
��"
��f���l��t��m�������v�Z�J���z���������xv	�Z@�cq���-9��Q�����^T���*��(,��1�&t���Gy��7���������xU[aT�B+y[�}�0�(3]�z�6�M�U��V6~5��g�c9M�����i�ZS��rSF@�Qw�
y��{u���.F�8����m�~��_��?���c�T�������)3���a�f���/�8r=����qM*��bR]����O^�� ��rSA�Qw�
y��{�l�Q5���/**�����C�n���j��������$6�>������O��z#���t��y���7��x����W~=�z��������+���z���\�G�M�G��7�u,w�A��BO�U;�w���3~����������V�V��]����%K�X�[C��o���'��������<��'/���,9`����������>���x��@�8O^���v3F�v�`p���H�YYYw����������>;x�`���[t�f������QW�'oK�����4��!0xL�T���k�u^A�V��1���r��j��W�"�"_�9r�����=M��>}�O~���^z	
���������;x� >��-yO�����t*�����O�j9�����:���>�d&v���In��L�I2�8�ss����Lg��m06x����6�!��
��%$v���B����!b 	-�]�U]��G�E��j��J���GG��}�������wUWU����m��!���r{��U���oI�����U���@yT�SMKK���G��8����Wf��q����M��f��>lL�:DU��]XA��Q�������d���<����.D��xr��E
�6���r�����p������S�����DU��]XA��Q�9n(�T�����O���zv��f�GP�DU��]XA��Q�9��]��L��v��;W^���*v��DU��]XA��Q�9��]��zF�|�uU�����r
pU�ATe�����U������o�<��}����{��V�R-6��$DU��]XA��Q�9�g��2c�6���N�>3�eTECN�U��!���r{�*@s4�]4�~���#���p��%rV����UBva��2DU����]j$su���Tk������+E�8�?�!�2���
��e����]�]2�o�����)R�^�N���x����+(��!�4�3��F}V�������O�?�/o?����Q�!dVPn/CTh;��H
���w�?:���*�[���qQ�!dVPn/CTh�]�_6f�P���v�?7^��]��5���*C�.���^��
���o8�������D�;��5�����#DU��]XA��Q��)Z����_[.6�>���4�})5n!�2���
��e��7FV����Y+rj�1�q���j���g��!���r{�*�
����'G�����d�����UBva��2DU���L
�]�gNm���=�,+;�Y@Te�����U��F2����k��}���^�����Q�!dVPn/CT��j�9'C}$�!�>���P�w���DU��]XA��Q�:*��;���rj�!�/�.��.���!���r{�*@4e5���J��������@qU�AT�.���!���r{�*@�����qe���}CV.�Z�#���r3���
��e���]�4F�
tHl������Q�����
����!dVPn/CT���ec��@���S;
��\�T��9y*�+(7C�.���^��
*��H\a��G���nTj����#���r3���
��e���8Qd|�(@!�����Y�Bs*Ava�f�����U�:Z��O�EN�6�?m�"�r*Ava�f�����U,����%�S����KQ��9� ���r3���
��e��
��}fZ9�G��U3����]XA�Bva��2DU�����4+�>3NZ�_�:���
���+(��!�w����Z9��	������+(7C�.���^��
|f��<��)VN�;[^{�I9� ���r3���
��e����n����{M�rj�������N��+(7C�.���^��
QN�8��8�����#��V_� ���r3���
��e����������SO\���J�]XA�Bva��2DU��r�����rj���T���
���+(��!�#���S��9����������!dVPn/CT.T�~��kr���f�T���
���+(��!��S��z�m��J�]XA�Bva��2DU����S����-�S	�+(7C�.���^��
qN��7�\�����@N%�.���!���r{�*�3%�S�}5�����y��@va�f�����U!nQN�pT{����Z,�dVPn��]XA��Q�S@m���FZ9�����* ���r3���
��e���(��;b�T��?O����9� ���r3���
��e��o�_6VP�sj�D�{���-�=U@va�f�����U!~��������	C6����[~{����
���+(��!�B<���3%����8�?�u�� M>�:9� ���r3���
��e���*|�������}�&Kc7(E�5F+@va�f�����U!�)Z��Bc�6�������#��,��������jAva�f�����U!VW���ho��6�v�{s�4}�RQ��Sm�.���!���r{�*�I1����U:�XS{������H�A�.���!���r{�*���?w�X���4Q!���?�����S ���r3���
��e��3j$sO������I��S/N�F������R#Ava�f�����U!hF��bcN���k�������M��q\���
���+(��!���U��m���������!���Ti�����=�!�]XA�Bva��2DU�������cW�E��h�gK�}g�hS��+(7C�.���^��
��9�
WI!�c���Y��=j���Sm�.���!���r{�*x�/`n?��>�:���a���N^l���6dVPn��]XA��Q<���\�O�:�����8��MJ��Sm�.���!���r{�*x�a���h�\�t�RL��5SZ����UR�����!dVPn/CT�L��C�]�p�����v�����h������j��<��l�B���Z(/��2Dva�f�����U#;|�p�����k
Q�5k�O��vA4Bff��G��Bt�u��l��Xk��3c�c7�t5��Ava�f�����UC���=z���U7o�LI�o��A��_�Ey����)��J�ye������
9�C�����^������
���+(��!�^���n���O=�����;���UE���/��'O�����;�-[v�^h���g��-�v�w��.)��M��.���!���r{��5N�:��G�w�}w��-��{�3����>|�5����3�0@����V\��on:��8�:#U�T��������wBva�f�����U�QTT�o�>�A�5$�=z��?����K�����?���uuu�{�ap(�0�f(��v��6UZ�Om��7�+(7C�.���^��YxT���h������(�&$$���G����h�V�Oye��s�����
SR�g����������Gm��;��!�8���x�r��:V�i.D�����������w���
��O-;��Y�y��]�O[��������f
@�C���S���w(w���c���BT�,<�fee�q��a�k����!z�5F��
N�)������Z�S_�"���+�Tiu�����x�r3D��[�P��CQ�
=���YxT������~�|�rk8x��~����K7��ZM�����W�"�>����\y��>�?^d�f�*�/��r{�jd�Q����k���<�Laa!
���k�����7��
�n/s�j��G�����#�?tu��*6��o�+(7C�.���^��YxTU%99���o��wW�X�h���z�'?���-[����n��*6�oRNi8���$����%Y�t�c�]XA�Bva��2D����*)**�����vA�DSSS����{�������fZ���2������X8�S ���r3���
��e���Q�|��w�m�f
_!�
7n\H����i�_(7����������x���J�?�w�;!���r3���
��e����d��Z�_,����&���-m<��S ���r3���
��e�����j�uG�g�YS���jy��e���#�.���!���r{�*�-xU�������!��k�4k�")��5�
dVPn��]XA��QZ@�df����a����P�4y�������!���r3���
��e��p�.\6��Qn������F)�����6dVPn��]XA��Q�O��s
�O[���M]�_����T� ���r3���
��e���L5������kcj�����������]XA�Bva��2DUh��
s���P+�>7�?~�RG�M�.dVPn��]XA��Qn�n��,6W\����x>�*"dVPn��]XA��Qn�?`f�j�M�z����S�\v�;!���r3���
��e���Te5���j���N��c���*U�v�;!���r3���
��e��p}�Y�[j��x0�!�vH��2EZ��j7�Z�]XA�Bva��2DU���j:o��-�������+e�gp>���]XA�Bva��2DU���on8��H�zM�!+e5Lw�;!���r3���
��e���i���v�����}/M��d�
�c�#Cva�f�����U!U�?Vh|v�2T	��wfI��R�Bva�f�����U!T�l�:��8������K��r����+(7C�.���^��
W)Z}�%ca��e���P��&mU���P�.���!���r{�*4������c�{��#�;���&Q�5\����!dVPn/CT��!�^26�>�o�T�{t��B�L	���^�+(7C�.���^������j�������H���di��@����� ���r3���
��e�����������?$�&�
8��������
���+(��!��"B��#�?�F��R�����!dVPn/CT���������&�����}!�� ���r3���
��e���OR��2c���l�jH���di��@���]XA�Bva��2D�x����e����;�$�P�O����z!�& ���r3���
��e��q�0w���v��NC��L	nI=�[#As!���r3���
��e�����g.?�un]t�s0�_8V�S��dVPn��]XA��Q5VS�)����^��k��R[�+(7C�.���^��W4��L��������p���l����oy�.���!���r{�j�����yz�I��S��^(����V���
���+(��!���j�����m���������Y�!������!dVPn/CT�%U���cRCHm��{i�����S��*dVPn��]XA��Q5�F}�%�����S;��1C:����:dVPn��]XA��Q5�)Z����3��v��0]�P�������!dVPn/CT�U�������+�v��:P#a��K�]XA�Bva��2D��t��\�O}$�:���q���T�S]���
���+(��!�����Pn�X��g�O��2E�z\���-�.���!���r{�j,Q���E�?��bcj�������5�
 ���r3���
��e��1�.`����g�8����/�"��
dVPn��]XA��Q56���K�T� ��c�q��6���
���+(��!�z�f���l������!���i�A���6���
���+(��!�z�_1���O�6�>��{k�t������]XA�Bva��2DU��\k�>�>>�:#U������j��TO@va�f�����U�H7�H5j��A��*���xi�NU��T�@va�f�����U=GR�����i��(���tio.R�� ���r3���
��e���RQg�;�ui��|���%��*���dVPn��]XA��Q�+�����Iy0�!��Ol�\��m��Qy�+(7C�.���^���	��X��o���?a����������]�.���!���r{�j����[�kO��v�?6���B�b%v�{�+(7C�.���^�����*�)���!�N��c�S��2T1����!dVPn/CTmK�V��<�����Ti�	�456 ���r3���
��e��m�r�)rj���������?f ���r3���
��e��mIR��G�{��'lV���
�����!dVPn/CTmc3w*�s��?� ���r3���
��e������
���+(��!�4�+(7C�.���^��
��.���!���r{�*@s ���r3���
��e������
���+(��!���@ p���CA�����-dVPn��]XA��Q��PN]�~��w��.��'����EZ�W�.���!���r{�j������J~�����jyy��
�����!dVPn/CTm��}���w�uWvv�h?~�m���~�z1q����!dVPn/CTm���<���!C�������n��}��'�
@\Bva�f�����U[��c������Z�A�~���}>�5q����!dVPn/CTm������pE����\5.!���r3���
��e��-�yQU��3�v���b�
k���U��n
@�C�[E+�4�j�������[�����>ZYYi
���q����%���2��={�}�9���u���>z������������E+�4�j����|��G���SUUE��aPm���~�9�� ���@ 0p��[o���iNN���z���}�{+W�������b��;��sgJ�����������G���!��$J���u�?A_|�r*��@T�BT�BT�BT�BTh*EQ.\�@���+���O\A��V�Y�$Y�<q���@�u�#����^�9��������/_�,z��_"��>��+��sBThZ������8u���D�e)))�e�n��-�Q�7o�U�v����?:t�^��������h�� �j���R}������\�f�3�RN}���E/���j|���=��{�u~�����U��^��|�QF7n�w���_����s�������L���r�-�������G?�����#\�p�^�Q�� 5�^�9TYzS���T��7�^z����������w�m��+�^'b�Qb�������s�BT������k���������B�jqq1%�g�}V\P���mj�v1���/��?��?����e���T>��SJ'd���4H��^jDv�Q�������r����;]��>Lo���L�4�6�B�����rss;w�����s�BT��������g���2eJHT��m�-���w�^k��>++���S�51�����%I���w�>}(��|�^�z����n�����K�@L������;�T�x����w��K/^L����!�7������G!��<�,DU����y���2w�����&+//OHH�vkb��K�(�6�n����o�>))It	���F���!6Q|��6}/��(����I_P;w�L�U�Ch�@v�]���3�c��������s�BTh���:x�`� ��3k��G�[��$I���������4(b
�D����XDqDIC���u�<�*�kb}���w����/��!o^|�{�*@S!��B��?�w����*ZU���q���o&�V)�"����/���k��1bQ5V �4Ux(IJJ�����b�i�eee}������>�����������o��*5:wBL1b�-���m�6�{�>}"F�X�;t]0`��������3A����{���aCEE���s�BTh���Jk,��sJ��k��E�����=z���?���h}��s�)S�����
�F����"��Q���k��RFy��kkkE�py�B�N�3�b�)>�=Q����*��n����K�*�S���r)���:t��O~"~��$6�Q��x��h�4�<'��Z�~�(�P�Ee�]�v��|���4H��{���e��o��=FUUU����w�u}\��}���4>�=Q����*��������l�������t�Z�5�UU?����6��"��=����6��N7h���!�Pqt����\���~�>p�����=j�v�MOOo��]��=�y7!��|�����U�*<��+V�y��b/��!{�!�9r�VNV-�_�QFy��(��v�A�.��j��K/�~������?����&�mJ�V_�����1.�#�����
�T��/���S�7lj��t�j��,��;��p�1J��A�����$���j�E����#����
��
��
��
��
��
��
��
��
��
��
��
��
��
��
��
��
��
��
��
��
��
��
��
��
��
��
��
��
��
��
��
A ����u�v=.=�� UU��+�~?��5ph������
�*@s�����{O�:e
��4�������O�v�����i��=VG�~��'4{��M��������04�
�� �D��QU����{�z�������������ZA���			��.��h+��K4M������������4Mg/��������w��u]����A�o��Q�������s��E��4Y�����.z8��Z�N3PRR�bPQQQ���4��9��L�8Q<j���'"8[��4�a��p�Lx��8K���@S��y���#n7V��sK�
Y��)�q����<_��#u!��DU�%"1,^���� ��


D	��J�			4�G}�k�
6(����_��W4H�<(&%�������th''�c�Etg/=�w��M4o����$�mQ�~��)�����]��S�tz.���OPp���L��~����i��Q�[�f����M����bA�pNSp�������s�T�S�-�vc���RoYY�}G���m�p>�Qb�����7�����}�{����z�-������a�:�������W_}��E�����)�R�����z��)�����c��bd�������
�]kkk_z�%_��o}�[v/=��������rw�O�@�077���3
���Z�	�IQ#������<���My��|���|�1�?#
jtG���^�!���-�vc�����,U��g����cE�%%%4'�"���B�	y,p�*��?(\�_p����O�{�����Tg%!�����S����{�R��"� M��w��0t��������]���;F�"������E�$99�7������Kq�ptw�P}��w��t�Z����oU���������6�)���+�Y8����L����s"����_��#Z�9M��"n7V����s�:�,8[���+���={6�R���XL�<�����KD�HKK�������/�IEp�P�8�{�n�%2�o�a����)�P�]����e(�������St;{�l�t�Z����q�dJ~��w��3g���z�M-�N�[6�:�2�������32dM��sVV���
�|t��"n7V�������)��������|E��?���A�
y,p�*�g�-�{)	����|"
!#S/�#��=���4f815�t"��B���fkDU�S&�gm�y�g��Y����-��]�u��)���5��.8[��6qnE���7��������@T�X��-�w���7��s�N�E���{�Q����PVV��}����J��7�x�����w���k�������DD�[O�M-bf"F%[HT��S��Y
���y�����|^�	t[4�v�m��3g�4����>?�#�.�li���.�����|E��������{^;ep�*��(��nS�0jG(M�&O�l��hT�����-&E�7o�L�h���4������/WUU5�-����F�t"�����9bO�nS��`S���'N����6���@��h|�,�����(���������O����v�v�}~��(8[�>���<V�
R/}-)**�8ep�*�������3���o)����VWWo����[o�{o4���?���?���O����]~����������������K�������C����=���nS��ilS�*=�/xr��>e��4~�g���(����[o�Ew!b:"X��>?�G�-M�[�f+zC
2ep�*�����BU��})pP��w�Cq�����J����E����}���^�`���_�~"�4Z��]��8��D~w��}4z��J��sg���K�.5�)w���~4��,���Qt�;Z�3@�v�m"X��>?�G�-M�[�f+��
���8�nBT�XB����ow����8�E4:{)�������1��������������L6���"4��,����yw��i:45�C���^�!�<e����ij��i4p�)4���Yp.�pQ��y[p�����f?�����x�}���qK�?{C������;U��*�:DU����
�Q n-����[s�@T�BT�BT�BT�BT�BT�BT�BT�BT�BT�BT�BT�BT�BT�BT�BT�BT�BT�BT�BT�BT�BT�BT�BT�BT�BT�BT�BT�BT�BT�BT����z�����
��k��I���9�����$h���6l�e��������\�`���(�:z�����b�
z�[M7(�����@��&��m-R�o
GB��x�bZZ��D���OG���~UU�1���g����]b���Q����;�����Oy�ju�O��K�4��?���>����>JJJ�����k����������[�.�;w.�vc��|�2����K����uEo�F���V4
-����"��i��F��5k��2�D8p����C�	����3�����L�[T5Ms���?b����_O�T�B������***������f����X�v��\��
Nw�%A�����������+�����F]�x1%%�>Uh�6U�X+W������
:o�<*�5vp��������s�-�=zt��A�S>�����"))I�,XPVV&I���;q���a����+����}uu��_MwAT�6G����1��AYM���* b��T_O1/�u��i�|9�4Ui4��EcR*-))��O�-[�l��R;�?y��5vP �4i}���%V@����ZYYIq�>�?��3����f=v��Xy,_�\Q;������811���N�U�G��
A�f��q����{��lRm��J��������-���b�
Z\vN%�*}1�5k������s~"i����)��,\�P4��9B]C�������T�DU]�,X@����^�l}�G\�=z�a���������+h%1{�l��J1�jm�*xDvv6���d��f5�(�Q5dee����?����6������7^1���m��F��Q���T��555����j����v���;v�X��Vk}�9r$}�m���j�S�U80`��O?�4//o�����Y��V<��v����O�b�_1����>|����VK�������[XXH�(�B�^��;�M-�N����`RR��jl��4�����DUZ��hM�����D���K��4Z��m��92q�Dz,�V��(�7���4������w��)��iRT�������yO��V�4���������c-+��������V1�-=�����'�8q��g�F�)..��"�����p���Q�w��������84o�-�����yM�:�^�U-d��/6���g�Di���,Y�MN,g��hP,@�'�^���h��Y#�2-Z8�����Q��H3f�qE�K���A�����z3�]��*���X�bEJJ
�Fh|*M'��o=M��/��'.��M���7o��|�������m���st_�B��JKIL��=�/Ej��,I�����[��iy��H\'Q��������}����]��@��?i�$���Jk#Zw��>��%T����p����8z�(�����S���v�s�e�p~~>���$�u�SvtZA��'O��U�s
��4�I1��0%��3g���G�F
&��%���Apn].((��cO�&E����E�q���?>�=�=�t����?{��5�����+�aqz Z�[cD��K�e�\,���i��|��)l�������&�m
m���^�4�Th����3-��K��>W�9�<=�ovv��t��b[�j�(t�cQ�N�>m���0��q��%@��7T�K�Z��DYY�Mp�T�%1d�{��](Sp�%s��EJ��.�M#S�5� z��,@"j4f��fl`��z�M�0�JI���f������P���E����~��Y�����!O�)h�h���=v�����!��Zs������6m��bv@	�q����i]+VB�N�hd��'�R�s�Z/����SBj����q��Gv�D
�f���s��D���K��k\����+E �tH����L�O4e1��h�+F��^'N�t���2b�gt��M�>��"!h|�`�y���5j�H1���/�o�l���/-���-CB��{�nk���{�R<���e)g����A�������
Gw�����b�Lqi���?���C^Wb���*�t������:�E�^g���
}-�R�/dz����4�*JKK�_����������4����W/-
z%S#��s����<-����������qcUU�N{s�u��JS����~��s���.zI���;����N�Kh�eddD�<,~@w��v_*�(��*}Fo���V���*))�v@q�8��V?3f��q�(U�����9��$I���f����F{�DkP1�S�U��aZ��^zF4��*��6e����Xf?A)H]]����P4<~�8��\Q4_�f�j�����b|Z;��:E|z,M�h��?���;&����k��Q���!����	���i�555�v����EOa����
���HN4?+V���A�Is{����<�"5���x�4}�#����R���ot����-[F��^!4�X��_lo��	�r�uBv��4M�K�s����+��e��I4e�>��f�b=G������g��qe��
�4��+������+**�%A-�N���e�!�uE����e �<}�4
�s�Eh�����)E=�,��s?t������<����B�/Nb~�%j�iR�K�����	M�^9�Qh�41��[�R#�J��J�X��Kw��3�LT^^.F����,\���>������k���4�(I��"�����@�E��'���j��48f���o �aqH���
�b>���@�����u�h���N�<����Dk���4�F�(��nH$�WN7U	��)
�W<Z��\�Zu9��d��="�L�>����4)Z��:uJ$'�F"fQ�-3g��Gh��P��g/BsE��7)���#%))�b5�jUl
�����M�f��H���Q��o�H(�P���Dw����6%%�)7P������3��D����������s���.�,���(;�B�F<33S����4�f�^�4��9s�G�_l�0��������?��������D,���N�7-
J��N_�(�����������R���a�T��S�����?�t15"���E��<y2�S�E�R=�O�������������5�P��4�Y����L���h���w�E���
�34����e��^�4�g�}FI�j�;�Ui�4~�xZ���7��"�Y{�.�����FZ�9rDlb{���j���b�
g<��8�L�i�D
�aq�Z��?>|%M#���h},Vl4�b�#��5�u�������W��CO�zi��8q�Z(��M�4?�o�hN4�"���o�|I�����KO�"�$"�a����/=kj���T��7o�����r�D��C-P"n����o�y�#f����_l#F��}O�.B3i5�HT�	��eK����w��az7�8;v���K��S"���l�L���u������`�5L������z����s~qjv	��w�����}���{7������=x@��]�re��:t��@�,��:�8�Q�R��e��c�V�!��"�Y	}�S�+���^����@�}��
�F������C�q(��}���y!�����G�����T�-�	!�	=����x�|�rj!�)"��xD�4����{9�LF��������ihp������~�@�O��F��!b0��f�)��5c������c�V�Z5r�H���/=k��;e{V�(��<�v�)����P�m,�_��FpnDl���g��u��l��-�c��rB����*	�)8?�D�M<���4�b�T����Fw�
;�?|�k��bM�gZ���_�W���,�zb���������C�R@l����.Ax���L���.^�h�}�Xt��B�J�za�zO�J .�pT��E��9r$d5q�Jjkk��SZS��9�%�����h�TT�Uoc	�a'$��~��.6�R����y$"����1Z+/Z���"�	��i��di�Ii�)[����b�
��Z!O��F�c���h��,
6����&#F��j��%���������if(.���b��C#;����9s�}C���
�v�)}��A��p�"�XLaE��1�.�1I�y�I3^���X�!�����FZ&�n���h��%6e��B�_*����m!��g��J����m����IM�XZZ*�������g���T��H�yg��E�M��E��,8�j�k7�{B��\�6;��/�q)����K��5�X��+QJ]��rEEE���.�gb"!��@���g�wCr��r��h��k������D)��i
�_�e/^�hNK��b�n�Z�)���:�>h��O�)�<�&j����1b�z��{h!��g����b��w��s���{{|��G�5�|�rHT��FS\wkY��j�_�!u".g�C��MA��b�%�����J(J�7�C�M�>]��mT;z=_w�4�����
zP����K=����9Y��@-���/B���Z�u���^�bOH��[�n���NG���)�5���X�x��H��&��F�%�Dh�#�s�%���m;QUu��)4�}���i�J-&L�G��5����b��1==]�+B+�3g�X�5�����S�N���b��s�N�M-����5�f��`��m�]�����@������Z��9������v��^q�X�m�~1��.))����x�Q�{�c����]�~��/��y#��X�!��}���_~~�	��(������*�W)�omm-�}KMM�7���9�)��x�����#�EE�C�?}c��=�����������m�7����O$,dee�9�x�T;�F��xEU�pb�)���f����b�����������f����y�s�%���B�w�;UTT��'N�SfH���I�K���@�f��1e#�\/h�/V����h/�a��E<H�����8�����J�b��y�Mt�V���9���X�L;N���>}:�P���z\��r�"�O2�ExC�[���"�#����Y,���;vP��~x(���Z#����`J������k���]Q�i
EEE�	 "/��:��cpC".dEL������e����KS���X���^�'O����NGUZ��5�>����S4H�K��#J��{�)�i���M�5��o�������v �n���b%J��.�&&���UMsN�+�>s����K-4��%K�7&�t�-[��_�JW���
<8b����������!�R��XC�O����+Q{�c��.�h�,X`�����nt�R.�Az"��T��������)S���X���z�]nJT���(��m�A�n�����3��G9K��fC�m��%';'�GC1EE\�b1�4Rx)y��Oe@/!ZP�D��7��!7U)MN�<��L��aGA�o�fDU�^��WT\��
y�Mq!����{!<
���H��J�1��ac�
��}�}����NG�("�Y���,���w�������%�s��M�4It%''��������k��n��0x&sZ��CPv�57���!D�q������8���Y�{�})����Q"qn�����b����������J��5�/�c������bEH�7o�L+u�2��fU����~���r ����Xz���).��ko7����U�N3�|hZt���w�g���e�u]7�`�+�����3gD�d����f6�h�V.���45��(7����W��Yn��xhZ\���:��h�Q����3�7��e�"eu�sRx4$ST��,#����E�����L���N�F�
Je���?��^V-U�!/^,��R��O��^��R����/(��G���w]*���S#n�����F!~ ���hi�'-
j'���^�6zm���6d���J�s�2&h=J+TB�C�
Z�Q^	_1���1�*���i:)))beF�]�re��.�4?��K�.�1p���S/��i�Dc���}�JW��s\x��^����\�����>�L�S���SZ��'��M[cg�f���M�&��Ga��������\��t(H9�o�]�y�����-qz�������Flaa!uQ;��\�tG;t���}��o������,7��5k��^�UG����C���'��H��b�������9)<��)*�r�E;=eA
)d��N��
��#�U��]���-U	}��9����:=4�����8�����������Q���D���������Q<���+��� �����>�g����6Z=dee���uQ���Ih�{��]���d��@%///|�e����y:�h�~�������1��:��9�\����6�(E������4)zt�h][VV�L�t�Z�5t��T���f�j�,��w���E��At���&h�'����[���Mg��9��LE�>~��u~~~��)b�}�\���n2����:_T#F���h���.�p���9:w8�`T���3��)�-}��M�-%�Bv�����s������w$ZV@<�U�s�V����0a�X��j����Z�d�E�Y�����*==]��)�P^�����o����L+l2r���G����.{���?_YYiO��/�X�p!�Uxzn,����n[�z5���q������]����<���|�����Z�4^b��DL��f/IDAT�)�S�������>|��5-Rz�Q�F�t�����C-�.Z,B�0m�4*
(�.�=����`�E\�_�i�6
)NQ���4)��w����mj!���'z1�4i6���Kzq/J����u���d4���y�}�~��GC��q9���.+�������'�����G���=#�}+�MBFUBs%^������������^�6q!������Bu�0�{GWU��	��A3@L���J�����>����;�h����V���C��*��H���V�4Y�E�P���;�n;���i��q�h���������M�6�4��B���mzd�tG���>������gD�<�(��`	�/�\����������.F�9�E��6��'��0g��@#��$�EK�k������B���sK3c59����bc��YV�!���zi��O��W�8���H��PM��h���4���1�_��x#���9��e��CP\�m����y��!>�j����(VD�T��\X�K�,<x��!C��b���ix����J6#�g��{��q�����yD @�BT�D�V���A1q����O��D)������7O]D]uW��L���o�G}4j��V} �z��_��]�6|�5@�AT�D�V���-((�|<}��(~@I�)G��^ZI�&N�Hi533�j�8u���A�
>�RU=Q����`u]���8�*�����_�w�7��p~~>��#FPE�V�;�@`����qNN~�
 �z�j+qg�j�VZZ�<��}(zQQ�9U���w�^
17nD��W�����GNq��U=!P������a5��0jkki��a��ls�-X���X����#���G��*�/���g'��A�|���b����*x�*x�*x�*x�*x�*x�*x�*x�*x�*x�*x�*x�*x�*x�*x�*x�*x�*x�*x�j�ey��A�y�X���yyy�;w�4&�ouE�`Q�TTTP�lw����������p�-���;����G��p������~���n�����$I�w���D���E{��})�����)��!���t������v���z���)���5//�����a�����jHJJ�����y�����AEWnn�}��7c�1Ht]0`@�>}dY��k5p��z�(SR����;���N�{��|�;�:u������{��r�-��m#����sEEE�^k�+D��UYYI��O�Sff����C�R9r$�fddPx��(�&$$���G�����|�cq���zUo����AtC�\�t�{���?�xUUUkD�����N�<y��k���U��n
@�C�[E+� ���4?��S7�?~�=����^�z�7����p�04�M���>���w(7CTq��5��n=u����z�<Hq��[�W~�*�fYYY����G����)�R���k5A�x�"}�Y�Pn���Twk���D��U\\��?�)!!AGE�����~w���7)�Rf��~�j�*��;h���|�;������]XA�Bva�v����u�����������Q���z��y����g��_���?�����nP�����8	k�^�?�.���!���r�Q�TWW��;��O?��������E��y3�P����s�����g�]XA�Bva�v�j������(..���G����������!dVPnw ������!dVPnw ������!dVPnw ������!dVPnw ������!dVPnw ������!dVPnw ������!dVPnw ������!dVPnw ������!dVPnw ������!dVPnw ������!dVPnw ������!dVPnw ������!dVPnw ������!dVPnw ������!dVP��P'���
k Q�=�.���!���r� U��wV�Ayu����0�v��
�dVPn��]XA�o�i���h����=S�9��y����}����W�*�*�{�]XA�Bva��%U��,�������G�;$�(����i��X�DU� ���r3���
�����)G�jY��I����X���������;�#zI���
dVPn��]XA��N��S���J���N��
��q�����\�����_RDU� ���r3���
���r#����W�$��,��q�G�?E�
��<�1G;w���35��W8DU� ���r3���
���h�g���'��;�����K/O�z��w��P�_�)�C�����e�����SV��G��
�dVPn��]Xa^n�b��hl<�M��|��!��8Q����:���c������{k�4g�z��(�2��_�*�{�]XA�BTe�g����&nV>^ �6�!�vO�?>��Prh6��n#�����W�f���~�Z#��
dT�*�{�]XA�BTe�[��_6�\�9��e�?�������O�-�'lVVR����O/���P�k�?ucU����
���*+|�-�f�^��d�>r��������1G�:��G�yR;q��,[ReV�L��@�fCTp�+(7C���0)��s�i��������L���l�Q�hZ+��f��U����
���*+q_��:s�&�����I
!�C�/qE��E�Nn������
�dVPn�UY��rS����5S�2�����x�Z�j\
��*�{�]XA�BTe%^�}�����@�T���	C|c6(g�����5 ������!DUV����j.�T_�r���W�J���:�7�: ������!DUV���������>�y��v��eCm�C��Q�=�.���!��7����7+/L�:^9[��i��|���1��*�{�]XA�BTe%>��9G{{�������/��V�M���e!������!DUVb���jR��>�o>���U�nT>_"��-QH�9�:|�}���%����sQYw�DU� ���r3����7�]^g�.6v��WT�oW��|8_�3CzuJ�%�)�>1��eX�U��7����c���ZqU����U����
���*+^+����������^����K������w�{0x��	C|�����V��~���� ������!DUV�S���Kd���qQ����2E�p�L�t�ve�Au��x��[b��xoS��*�{�]XA�BTe��>Yl��
�8Qz(���	C���=K�40v���W�xT;tN?y��+3
+�i���LM�&�q���Ava�fQ���-7��1�^��G�����p?e�#����e���,�3keSV�MOo6�DU� ���r3���J[��r�1c��{���0�S��SV_0(��r(�Q�=�.���!���~�k�����)ui����0�yV/�����-Q�=�.���!���f�e�\����#=9�/N2E����w��/V��gN��U����
���*+��[7����>J�{�Z����>3��G������w�*�{�]XA�BTe��rg���-��������$i�~5��!��8qQ�=�.���!����r���u�Y\i�.6���Oh+��v��(�V(��7O�����'J	WN��3U������@��(����Ava�fQ������ec�	m�6%eu��Er�9��3���K�L�^�(=7��=��z��
o��eS�%O����I[�S�F��WHU����
���*+Q��j����u�������������)q>���rA���n#�/O��\8|^���RDU� ���r3���JH��d��}�u���?���N�z��w�o�X(��������{M����-�������SY�_������(�/j8�YM�����U����
���*+T����}g�����U�wf�=���;��'e�=>��o�<j]����cZ�y�H�~���$���b�%U��Z��o�f@5��>���BTp�+(7C��|�;��=����5/N���$�:'�������3v([�k�
�s��K5
I����Z�*�{�]XA�BT���g�^xf����:g6�<��g�4lM`�>5+O?UlUU~d����
�dVPn�U���s��i�}���G�}8_�IYwX;Z��-3J��:���K[�*�{�]XA�BT�c���C}y�$���p����;�^���3SYE8m-���Ava�fQ5^T�����FZS���rL���0��DU� ���r3��L�~�Q���������k�K�
��f���@Tp�+(7C�.q��g�Zxv�$N>�e�������JPnw ������!d�xr�@��&w�r����=g���oRQnw ������!d�����{�W�H�GP=���6p���tk�v�*�{�]XA�Bv������'G[S��_���VG8��v�*�{�]XA�Bv�i�Y�)G{{�u��?O>x��+�OD�r�Q������[�l���Dc����G�#(z/�dVPn��]bW��p
���K�B����i���KF������@TmI�2o����s�ZM�$��/Pc�������F�i����g�]XA�Bv��:�,�6���������G��������-
��^���	��9Z�����r�Q�%��=��&%NgT�<y2%�#F��`J�=z�)�g�]XA�Bv���K��lm�^u�6e���������+�=Kzc����������O��w�lx����
^8^d�p�~����-���c;v������?����555��w�ZUU%Z��������c�^���+(7C�.m�Z2�e�o����j����NC�	C����p�^�����u=(�;U[Faaa��=������{��vT������~7y�d1HTU��?���o_EQ��ZMG�]XA�BviC��pP������1��u���	�7�K������
L����QWT���������r�Q�P�LJJ����dO�:���YYY��v��M���0x������*z�5q����!d���}N�zy����������y����i������lu�Im�~��~�H?]b�_2
+��j����x*)�z�9S��v�j��I����Ct;$�fdd�k����A��hBBByyy�^k��+(7C�.�+�2'lV^�r�S���/�R����f����MY5�����(�;Uo����)���9SUUt!�J���)''�>���w(7CTq��5�,�D����}g��
���wJ��ln��m��s�X#�&���P��B���7n��?�����/����)�4�������?������/�PWW��S[[������G�5��f�*Nu����=��p[�;S��
�!��D��s��d�e�(���ju(w���c�D��G���j8�e����o��Yk��Q�=z�8p����{��0����


����x�r3D��[�:�]����������������KU��Fr��z(�X�Q���9sf���I�~����������S�����^�����m�F#S��#~�������B���YA������������S�`�H����������b#��<����@Tma!�U�u}��u^��_w����������~8??����]XA�Bvi%�\sX{��m�R��6�}^����K�,����-,$�������4�����G��m����q����!d�fW=�-m��i�)m�Qm�>u�Ne�����K�����������'��*��Rj��D�F�s��EEE�p�h\t����$��	�+(7C�.Mq���������+��+_/\ �O�������^�*�<Iz~�����
���?�rS�,-�����V�(�;U����
���Kt�b.���5����;7����F��lU��@�nK�
�v�*�{�]XA�Bv���}��@���W=���M\�t��?M�b�<|M`�e�nu�u�1��9���+!U@����
�dVPn��]"����;���${�;��Q�S�*�������k�gu
�G�������r���(���z�����Pnw ������!d�f}������n#���]G���PO%�Fy�Y+��UO]�r�Q�=�.���!�8�V��7+�&K���o�������^���~�fC����
�dVPn��]�����p�.W���s�q���0bsjD(�;U����
���9��&��I�5a�/yU��9���#�[
��DU� ���r3�<�������o��HJCH���O����*�xK�A���@Tp�+(7C����B=qy��Xk������6)9�z�\�����@Tp�+(7C<�K�l��P��!=t�\T�������j�r[�U���@Tp�+(7C����sa��Q���(kc�����w)�K<q������DU� ���r3�'�\�m������\���g���yz�?�7��U���
�dVPn�8d��js��������sQ��Gg�jDUw ���������.�j�{�����F_
�/M�f�P_��sQ]��;U����
��P�f
��{�����k�4kgCH���O�(DUw ���������.C���H�v�G.�Ul~���;U����
��P<e�K�
��?�6��z%��H�C����DU� ���r3��BjzXH}m�4�z�!�*DUw ���������.��y��������kBj����2B�5U���
�dVPn�b.�hz��RcS�6a���B���������7gH�{�c�BjD���@Tp�+(7C1�](u�T����s2�/����_�(=:�jBm����Ly�>
�zBj�U���
�dVPn���]|��}�~u��@�4��)����xJ����c���AY��+4����z�U���
�dVPn�<�]*}fa�y��8xN���7���H~s��s�?!�j<M��=MJ^X���9���h�V�*����DUw ������!������fA�q���?O�~R[��.�Tg�PS7��^�@~���l����kS���������%��]��������`x������DU� ���r3�z���$e�I[�����It���4��d����������wJ�=�t5����8��i
��h�����j�'��y���@Tp�+(7C��](�N��P6
�8��_�a����O���-\ _��<�e�����kM��&7	Q����Ava�f�����)[CCj�a����W�H}g��,�WF�W�mW����i[�k��}yz�y�x��[jW��fMZ��;U����
��Pf�����~s���[!������Ki{��S�����`�+3.�%�f����I]���DU� ���r3�"������?�����RNt� ���DUw ���������.�.�v�����:B������jN!��DUw �������fg��rc�.��yr��VH����7G^��+���=
Q����Ava����E}��S��������-�~EZ^g�H��D��E�����ir�+!��(�.��/2$�
<Q����Ava�f�@��w��\j�+�j��)��w�������/�?]�bI���
��(��+6+S�5�}0_~r����?M^�_=y��Uk��Y���@Tp�+(7���7gHv����q���d_�����7���j�@H�%���@Tp�+(7Yg�7�75���QH�@^���.1p������DU� ���rs��{5��:E��2w��Ks�(���e��?w�:}�B�gJU�&1Q����Ava��{�S{O�rj����|�)�E3�Z�_1ke��o"��.DUw �����;��;�����Yy�i"���r�Q�=�.���ql�)-<�dVPnw �����;^m=��2%BN%�.����@Tp�+(w\��J�����9� ���r�Q�=�.����gcN��J�]XA����
�dVP�8�)G{i�#�����K�"���r�Q�=�.����d����T���
��DU� ���r�J��S_�p%�N�2���������Ava���H�d75�dVPnw �����;�����x+��|��J�]XA����
�dVP��F�t�����������Ava��Q��y0_���<;����Nm8��us*Ava�v�*�{�]XA�cK@3�3��������������@�n�t=�.����@Tp�+(wL���S����j���;���F[	U��9�r*Ava�v�*�{�]XA���0����5���u�{����$��c����S���@N%�.����@Tp�+(��f}A��1G�A�`���x�c����m�������J�i-����{5�+(�;U����
��)u����6q�2p���D)!�jB�2���<y�6�F8Sb�M8~��.����@Tp�+(�;T����+����co���������W��]M��_�����>Yz8�jB}8���,i�&eS�v����XS�	�.����@Tp�+(w��)4��Q�lSF���
|�,0h��?M�;[~}���$��q������wr5��4��u��G�����-�Pm�.����@Tp�+(wk(�2�f����O��w�����i��GR��\�>Y�\�t�z��^ReXSlQ�.����@Tp�+(w������O�3���������O���4Izs��?M�lq yU u�2m���W]yP[T�vB�wV/�0�V��dVPnw �������������9�����hH��i�j�Oj4�����"�l�A��R�Y#���p�T� ���r�Q�=�.���-�L�1m���t��+��_�*��QO�D�q>�V���
��D���o����X�f��zEii��_~)z���������	�+(�M*�3�U?H�;�B�����7+��t����.����@Tm�M��o��������3qR}��������������={v{!� ���r7���[�k_-P6!��d�K[�i�_��<	������7��f��]}���W���;�(�R�<j�R=}�tSz!� ���r7�i�g����WzM��C*�{��$K�+3<�R��]XA����z����.\�\!-X����h��|/��"����`g��;(�N�6�nG������
�}�
+��;�wfI	WN��#�?m�B�U��oR���
��D�����#(n�����������?�?^�Y��|��w�}����k5AAva�n�j�\yP�d��e�������!+�Oh5����^���
��D����RSS_{��'�|2==�B'5��������u��8����i�������0�dVP������'�Q��g�^=[��i2%����<j�@va�v�j��H��k�������
��J��B��8�����������p
�d!����G�5���.�b�����U�W�H�\�4�sckS���>VQ^Qe�S��Twk���z��>Q��TUU����;��O?��:c�UU[#�������t���h� �������7i]����������.Ck�.��!�����x1�*Nu� �������zU[e�_|Q��?|����������={R��Z�a(�Al:s�}�Y�P�%�e���XV�gr��������xi�Ze�����U��1�*Nu� �����y�������{�������� {�hQQ��������[������K~���i�{�&�#��"+(�MV��<}�V��y�cW����{}�D�;O��U�t�TTq�x�����7+77�������/����������~��7dY������'
R#u����3g�����8�?z/�dVPnR+��O�c���)?|��%�)���P�����8	��+(�;Uo��A�������{����:v��o��o�V��^M����i������~�����?''����]Xa^��s�a-%x��}���G�?["/�����b�O�]XA�����N�:��_�vWP��9sfee���#G��<j�n��������*+����+�������mn�:xY��	����G�?eu`��l�������+(�;U[Fqq��+�n��5)����5��	�*+qYn����3�]2_�w��WT��V'lV��|�D��&�9Czq�d� ��^�*���l=�U��yRo�+(�;U����J|��������mI�:c�2z���"0h��o���T�����F�������!���g�T3s�����Aj�.����@Tp�*+qP��i��~���&JO���'����u��F�;[� ;l��(S=R�����@va�v�*�{UY��rU������NCC#�#)�����$��jY`�e�Nuq�����qZ�>��)1���8�Aj�.����@Tp�*+1Z�J��������Wo��,)yU`�f����C��������B��%�R����c���������ATe%���W�-��!+=�^
�/O��lU���*J�<��6�+(�;U����J�������
�^�$;�R`M��8��J\��y�.����@Tp�*+1Qn���)��lS���z��'F�SV�����R�]XA����
�DUV�_��K����{��^���a�/��aq�����
��DU� ����r�����k��s�?���./;�Q~���fCva�v�*�{UY�`���[j�;�}�"�����N��#��UO^4tlK�9�.����@Tp�*+)�a�_�l�4��}��������[!�}���i���j�y]��[�+(�;U����J����g�����&oU/
��!=1��fT�{i�4q�������� dVPnw ��Q���-���c�qm�N5yU����3cC/��)���4i����SZ��Jma�.����@Tp�*+��[7�*�����}�������'K����c���	����M��*�|�!�U ���r�Q�=����v�+��'���$��t���k���������s��k���Oh�.U�����]XA����
�DUVZ���Q���4�a�+S�����t��������;��G�#��jg�r
�+(�;U����J����gf���lS?�/?9��6��!��'I���(+�Yy��rG��	dVPnw ��Q���*�a���+jCWzO�:��C��wf���*�������%F
.����]XA����
�DUVn��u��yV��S�x���q����
�?]��<��������
����������6m�����rx���h��<�`�f�}��X���\��9�����|o���nj�j�%\Y���]XA��'Q�����=�����.��1c���g?���/u��6�3��T3v(o�����V�W���������r��W7�>9�������zq��!���r�#�������?�����+**DcFF�{��G��E#�F���0��;��<�ihC��;[.�t:�&�[3��N��8K���!���%�z7��_Y{X;[jh�6L�dVPnw�CT���y����~�m;�
4H��E#XMm�UT���e���fJv����U��V���R�y���:[������o���9N��u��y�N�"��ZlF��.�������Z^^���0f�k����F�����z����!������4QJ�p��a�
G��E)��~�<y��|L��K�2��9r�^�����9]�:���v��P�5� ���r�#��$I����_�����l�)����h�	�Mq���fn��>�/�S�?���H�~BK^i��IRn)��D�5�a���3��}j���t���C�W�)�xj�xa�UO��W��m8i�5	�5�.��������6l���_��K�.h���	���}T-�2f�T��`m)�6�?a�r��!�^(7zO�~���@��b��on=T6~���[�/�6\����l@���/x��������b�gE�}�.�������*I����;t������T�;�;�f���+���g�����<\}�)���
q��_���<���������������Ww^%�����J������;�
���^�t��.�������*����+�:�!^�j�d�8x������Y)�Z�WhF��m�������q�KLU3�]2��h�*=�>3�.���������~l]�T��RP���-dVPnw�UT������K�q���"��+���VD�XX^g�O���}gI%U1���:�<V��;�M��|�X��d)��%O���0���k>�yq�^u�����u����]XA��WQu��m��>���,Xp��e���,����)G�@W�o��{o����&)��������^h���4��j��y�����
�i�3c��&��#����#�*K��uYew���.�������J�t�����_��T���[^^��c��������\<E��jc�.��+GP=:�?bm�hA���.;�=��p�N)�u��x���jV�Mz����Ez�y��y��9}n���*��,�"�x����������5�<x&�8�fM���
���8���/���;_y�����'EUI�F�}�]wQ�����#�&������zU�I��=j���+�)����4Q�p%���FN�NOg�)m�Qm�5}�:}�2q�2j] ie��%����~s�O�;���L��s����I[�&u�����~DU��]XA��Q���SH}��'

��(�R;����O���q�*��8�.�Q�����,k~�!�A�;N��
n-(7��nM���r����2�z���D?Y$�?O~{�������'G��5��GO����K��+��5����< �2���
���x��!���MRSSq�*��X�.�n�:�=;���M��>mj3d���n��I�y.��J��5�5���Y4��y�������K�M���n��J�4eU m�����_v���UBva�vG<DUI�^}���]�^�p�U�9������pjU����.�\����r!��g�[Cvy�����CS���?��V��N6d��\yO����&��#�.��!����L�����.���>�m>�Q�>tN?QdP<���H!�2���
���8����u���?����_������s���}�������,Yb���b7��JC���H5p���%��Z�3?�o����YRiu��h�R�9e�b��T����3
�L��h����D��f���BTe����qU%I�;w�< � �������#F��/`���^��~�z���2��"������1�[��U�/��Nix z��������?�ETe����qU�����@�V�7�bv���;�.��	/������_yP	�S�o��V����}�RY���$���9��{DU��]XA��WQ��b.�T����qP<����s�Z~���#��!���	��[2D�f��S�����<���>PX���4UBva�vG�D�����Z�J�B����
��)��]���	��v��L�
[(j��W\i�9����������h��#��W�b��������UBva�vG�D��G�>���{��$������*��.k$��C�����A��J��6���<�=�w��b����E��/�T�g������d���K9� �2���
���x����%&&�w�}K�.��:~��;���O�>�F���Q�T�d��*s������������_����~�.�r*=�����7�3��:s�v�����}fJ�j����
Q�!dVPnw�CT��������_��jIII� �A����U�w�Dv)(7�����i�o���������x�����f�����TX��I���������{�S-��!���r�#��8�jj*���u�]w�5n�8���$}�������.ye�W��H���|S�)-���&:yQ~��)t�:������M����B#q��;&5
v��OY�D��!���r�#������W��o?x��'�|��?���S�


�����bk�5*@��xv�H���H��#�3w*��^�_XuH�x8���\��>]�N�����Q�N�������?������\��E��+�&�DU��]XA��'�Um������m�v�����#FPx=t��~���>l�����](�}����O����V�J�T"���u���[�V�4c�L�(�&�L��,��n����g���*s�Q����AT���g�R��m�DU��]XA��'QUU�+V4h����/_��O��S�S<�]����>=��0S��h�������}���J���G���3���~�O���P���q�;�6���
<3����&K���S ��!���r�#N�*@L�fv9|A��+���_v@UZ���7D3��u��������f�RF�W�X�7Gzq�_\@+��z����I?UBva�v�*�{��]�~�)���VN}~��a���tCje�B�q������0S��E�2�?M�=M�>��0���"�z����*C�.�������*I�����������My*�����{�^����/M�6��3S�.�6�������9���emvU8DU��]XA��QU��%K����W�
���P^^n�
��<�]�>��>|m�����G}��#o9�����	Q�!dVPnw�CT�������o~�����3c���o�
����]j%s�!���*��G�5\����68C~�CTe����qU�~�����
�O�k���[jL��<1�:�K�����Z��V����+(�;�!�RBMMM}���+**�&Oj������>]d]�?a����
;�U9�� �2���
���89�����O?������&==�U�G�Uv)�6���/L���zr��&%��CG �%DU��]XA��'�U�_��������p?�f��s��5�G�Z;�����d�mr�TnUBva�vG�V��_������(��i�pXx����F2W���%���NC}_/�;��Hw �2���
����9���/��aU�qnf�����-J�Q�������oW.Vb��{UBva�vG�V5u���>��U����]j�����eq�|��?M^Dx�r�| �2���
���x���$�7����~�����,\��U�G��]���y����[GP=6�?|M�X��/��UBva�vG<D�������(�08�
������=eu��k��k�$��u���6UBva�vG<DU��?e�����.�:~�x�7n��k��^QQ1s����B<i��P������'���P�����]�t�Mm;��!���r�#�j�����gOk+n�v��������_P:t��~�#�w��5v��q���KE��0��N�n���������UBva�v����H:`������o�����{����������a���D�y������s���q�5���c�F��a�N���$��u8m� �2���
��D��u��
�����3��+WRr7n�� ��wo
�v��iii�������E3�3Ni�,�;$6���!�O2N�6�z�*C�.����@T�Y�2��};kq������vaa����'q[�D��W�>�@����V���.5�����k�����H����9���DU��]XA������6o�Lt��Mt;;;�G?����+E�0x���={���F���!��Tv)�0�mS(�����i�^�V�NoATe������-���=��+��r��%���h�����Qq
����pEQ� 6�?�>���f����wF�2�������]�fUM�5x���bU��n
@�C�[E+� �� M�6m��������<x��hl��Z[[{bSNN}�Y7.����m��O�S;���]�93�����rC,��S���w(w�q�[FTm�,��������������z��?�����SO=U]]��
!6�<y�>���t���a��w^-rj�I������b���f�
1�*Nu� �������zU[Dee��I����?�������Vk�����?��?����_����SU5z��q��?^<Qd�Zg]���$�G������c��
�Ue�*�/��r�Q�fi����F�s���eeeV����O=���?L_�h�F^�b�=��#�������E��w��>�/�����p����3%H�1Q�!dVPnw ������SN�|���_�t��y3EOUU'N��������������������I�����F�K��\}H}a�uF�'J3w��>�UBva�v����?~�Hz��-��ZVV6`������hZZ��?z/���.'��q���G[g�z��1GSu��Q�!dVPnw ��������d���<����.D��x�����o;�}�.����0�������?� �2���
��DU�4%�\�l��P�kmLm8��r�;�c�*C�.����@TpO�����{s��W��S��.����b�*C�.����@TpO��RQg�8��:�:�����Q���2���a��!���r�Q�=��i�+��oRf��s��0S����?�!�2���
��DU��g����i�����>���|��7W��
�Q�!dVPnw ��'$���d���tcmL}f��f��;���*C�.����@Tp��]�~���e��CB��I�~s��UY��������+(�;U�#���s�~��I�T]����)�q��Q�!dVPnw �������M'R7(��^9m�di���M�G��!���r�Q�%�Z�h��W'\j����[�k:~��UBva�v�*�*���Y���j������M�����UBva�v�*@�;�����N���I������Q�!dVPnw ��"��?rA�jy@����7&��9�Y��UBva�v�*@kQ��'����#�;
�}�^�r�1���*C�.����@Th5������8k������(��� ������+(�;UZ^q�1c��e��S_�"-��T����!dVPnw ���E���������'�8��k�#���r3���
��DU���
�K� MSI�}�,p�����]XA�Bva�v�*@����uG�'ZQ=9�?n�r���3R!���r3���
��DU�p����[�6��q���H�{UY
=s*�+(7C�.����@T�YgJ����!��S��'o;a�85�+(7C�.����@Th>������( .��H�o���)��SC ���r3���
��DU�fR��-��W�X?N}j������@va�f����������r���X?N�5����R��SC ���r3���
��DU�f
WL����SL��c���D�.����
���+(�;Un�Y_�7W~B�~����A���E��85�+(7C�.����@T�1�����S����.75�dVPn��]XA����
pNo��rj�9��JTM���
���+(�;U�*��������I9�7�S	�+(7C�.����@Th��
����S_�(��������
���+(�;U�����������6m�����]XA�Bva�v�*�uT��c6�'6���G�W��s���!���r3���
��DU�h�%s�v�C0�v���G�xq�&Bva�f�������������Cr�c��3w�Zs~�z�+(7C�.����@T�LV���WIi�$U���I[��{�-�.���!���r�Q U�_��u��SN��Z���������!dVPnw ��������n#rj�!��U�Z�r*Ava�f�������0��]���
9��$�K��������!dVPnw �\E��@�����K��O�}�P.�j��J�]XA�Bva�v�*�UG
���Z����&_�|��N�����!dVPnw �4�������fZ��{��[��9� ���r3���
��DU����o=��>�����T����;�j#�]XA�Bva�v�*p�_f���v
�O�L���i��J�]XA�Bva�v�*�%��Yy��K"�>�����]�Z+�dVPn��]XA����
L�T�+h���6�>5�?|M�By��>�	����!dVPnw �;�Q���^y(x����7�K�������������!dVPnw �/�~s�1���������X>p�w�;!���r3���
��DU��4���������W���'H�7)u��1����
���+(�;U�I����?]$�Ol�	C|���7����mj(dVPn��]XA����
����\qP{f��1��Q����K��� dVPn��]XA����
�L����#�)%[GP�6U�������wBva�f�����������cZ�+���4���9��KGPE���
���+(�;U!�#��f�]�Y;��/����U���T�+(7C�.����@T�xS�3w��?Y$?<mj�$�;��������S-�.���!���r�Q������hL��<1��T_.�*n�#�"Bva�f������'J���G������0���iN�Z�FGPE���
���+(�;U!��j�����J������c��+G
<���Z�.���!���r�Qb�Y_��XyP�5�:����?����R�_���]XA�Bva�v�*��Z�����^H�R�'�^�(�\8���o"dVPn��]XA����
�G7������=�\��������s��P��Bva�f������-I��C�]�p�����v�����h����!*��E5`�u5��|�N�&mi�s�6�+(7C�.����@TmI�������s�� J��f���O�.�F�����h�^pR��E��m��#���O���8����@J
Bva�f������-ClO��������y�fJ���������/�����M��8�[3�����i�n�F��������!dVPnw �������k�>��S�������gTU�o��@O�<)Z���;�\�l�u{A�������+��S��p.����������
���+(�;U[��S�z������n����{�uF�������������{��y������j��V2w���\���y��_�\TO�����]XA�Bva�v�j(**��o���U�=����t����p�������������0c�
��wimL�1�?r�r����G���
���+(�;U[RxT���h������(�&$$���G����h�V�.UT�S�Z��O�{i����.�]����Myyy��f
@�C���S���w(w���c�D���NT���=�2��OY��ct����eX�s*��>ku�����G�5��f�*Nu� �������zU[VxT������;��h��]�KC�^k8�$Iq�ryE���#WV<�\G!�}���$i�&aI�5F�����w(7CTq��5��n=u��������j~~��~�����[���������o_���jb����������zd���<y���9gjS������Q���E>Pnw �����ZUU��k�g�y����u]��k�o~��a��]����*c���1WOG5|�R��!�.���!���r�Q�%�GUEQ���o���w�}w���-z���~���l�����L�z��������Bj�!�>3%��Z<������!dVPnw �����J���z��Ey�]%���T{�~���W-��OhoL�����#��/	�)���QE���
���+(�;U[��w�yg��m������q���$�����4��_6��V�o���!x����O����N�.���!���r�Q�@@�?|^�l��>�1��P��������D-^!���r3���
��DUp[�����=?�:���q�Q����;�*"dVPn��]XA����
�:���C}8�!�v�{s���P�oL�!���r3���
��DUp������.�������8>�*"dVPn��]XA����
n���[�k/L�z��1����N'dVPn��]XA����
���hA�1{��i�u��i�����ks�����!dVPnw �B+R���B���q��#)
�J�)����7
�+(7C�.����@T��R�7���^�d����Q�����v;���]XA�Bva�v�*���
#m��������}/M�f�TF��G���
���+(�;U��5\����jy�*���!�����S�Sj�+(7C�.����@T��T+��Nk�L�v�?6����@A9���N�.���!���r�QZ��Js�^��������O����S�Bva�f�������4��T�1d�����$����uG41�Z�.���!���r�Qn�/`�=��6����i���t����
�.���!���r�QnJi��l���k���c��������+(7C�.����@T�f2���2c�������}�&K���6�6����!dVPnw �BsT��9��YlL}8���\i>��P5�+(7C�.����@T�V�37��c��r�?ie��;������!dVPnw ��
0���rc���dk�������*.C�D�.���!���r�Q�J����,
�;����2�b��
@va�f�������$u������Dk��Q����*���1�.���!���r�Q��"��}j�a
9�C�����������.���!���r�Q����3�F���P=���;[���������
���+(�;U�Q�b��{_�U�����d����.���!���r�Q"�\k���(�RHm��{1��?�!��dVPn��]XA����
�(����7)����o��v��OS[�+(7C�.����@T��4��������=����\T���-����!dVPnw �B��/�2v��>^h]+�}���	���j@ENm1�.���!���r�Q�;�l�Y��|}���������,)�4v��0dVPn��]XA�����Z���)�S7���J������1�r�a�-����!dVPnw �2U+��K����h+�>����di�z��!�� ���r3���
��DUv$��l��0S}~�u�T���sQ���������]XA�Bva�v�*#�V��X{X{m�RL�=;���b9+O��S.@va�f������,hF}Q������<�}0�vH�uO��.o;�����dVPn��]XA�����.��{s�O�������7[ZyP�q"*w!���r3���
��D�8w���^I�r��G�_�&����H�m����!dVPnw ����
���R;��2E��U)����6���
���+(�;U��Y_��1p�u��n#�#���R��+(7C�.����@T�C�Y�[j��m���h��]�Q�' ���r3���
��D�xc������[9��������^���
���+(�;U��n�+�{M�rj���%�U�<����!dVPnw ��M��>�?w�T���o8�Y}�
�.���!���r�Q5N�Z}V��=���/L�v�BN�dVPn��]XA����Z��3���R��{q��?O���K�]XA�Bva�v�j����m'�.#���k�t�r�G!���r3���
��D���W��9Z��
9�C������"�<���]XA�Bva�v�j��������M���{}�t�9���]XA�Bva�v�j����%Yj�����`���L9�r��!���r3���
��D��T�7��V:\��o���*�Sc�+(7C�.����@T�=�Q?}��>xR��C|��+�T!��dVPn��]XA����{�%�B*�%$���'�����1����!dVPnw ����u��I���}/T��Sc	�+(7C�.����@T�=�Q�w�<hQ�VBN�1�.���!���r�Q5&]�4�������
���+(�;U����
���+(�;U����
���+(�;U����
���+(�;U����
���+(�;U����
���+(�;U����
���+(�;U�^ 8y��������UR��+(7C�.����@Tmc�S��_��w�z��'rss�V��+(7C�.����@Tmc����S�o�~��RZ-//��!� ���r3���
��D���i��~x�]wegg������v�m����g�]XA�Bva�v�j[*--}���b
��WWWw����O>�o��+(7C�.����@TmK������.\h

<���_��|�0�dVPn��]XA������222��kG��� ��			��j\Bva�f������m�yQU��3�v���b�
k���U��n
@�C�[E+� ������[o�5<�>�������p�� @�r��BTmKg�������������G���[�@�j
s���Eq���zU�Vee���>��O���*4�js���;�	��j[
���[)����8p���������r�Jk�U���s�:w�Li5x]�v�S?���({��@Tm{�V�u�������TQ<
Q<
Q<
Q<
Q�u)�r���o
_Q]]}�
�m�B��$�*����[A4H���B�q4???�
~��e�K����XG����y�BThE�>��c�<p��)�)�>�RRR�����[L�,2o�<������O:t���2�����G����A���E������/�f�g�����o�^B�
���@�=z������H��ykCTh-��,�s�2��q�����������{�?��d�e�&%�[n��?��?���G?���������2����Q�B������.7���o��Q��������w���n��^!�:#@������t|��Q�U����]������]w�U���)�<��������nS�� �P�_x���������-+V�������R:!�'O�Ajt�R#�K�?~�c�=f�;//�����4x��az�0@�e����A�z�G�X�����s���t|��Q�U�����g?����)SB���m�n����{�Z���YYY�E���a�)�E(�8��,IR������Ca������K��t�_�~�H]�bZeee�������3f�����(��^�x�b�8~��51��i������>
�H���UZ�����F��s��D��5YyyyBB�[��.]�D�e��at����}��III�K�W5R�5������{��C�]�4M����sg��bB����K�Re��1;v�O���t|��Q�u�G����}�Y�W>����X&I������AS�e z��W�"�#�H����/6��qT_+�����{���~��y�������Q�*����������T������ww}3��J�Q5�\�x�W�^c������CTh]��$)))�G[�>b�-�i�������������O�_��������
�����i1m����r��m���}���U,X`
C��u}����������w�{��g��
4>�]��
����*�����y|(����b�����z������>�_��Y����L�b�O7h���b��NRTTDE����nJ��_���]D��y�
�
�;��<B����<w�*@�
�������[�.]�OO��v�1�[(�v���'?���5����F)������n����C�j����,BE-�Yw��Eo���� e�{���ZD����]�F�UUU��W�_8�u�]�qMo��c���<w�*@�
������_�J+��7�����mj��|[TU������(���2z��o���oD;��Ajw��CY�Y����Sr�����[�����������7==�]�v={����X����s �����JV�Xq��w��H�n��5�r��Z9Y�t��F��^�p#��
"��.��K/�t���������b�@�)�Z}�^��V����t|��6DU���|��N�:�����_A��V�A�R:Q����7���dY4h�((	O����V_�^�]?��y��U��U��U��U��U��U��U��U��U��U��U��U��U��U��U��U��U��U��U��U��U��U��U��U��U��U��U��U��U��U��U��U������5pTU�|�2����qXVqQ������'���;��	������04�
  ��������DU�aY�DU���i��3gx���'�����I-^��5Fp�%�h�a�|>j$�$��Hx����h$�7b�M����V���UU��G<1f����VWW�Gt9�,8[���K��|iL�����
UZ�LBBB����l�;w.�F���_���G}d�?��"�Q��m�I�o��5VG�v4���/GL��i
�q{�����P�.((�����{���6m��PWRR�s���E�n���?_�-++��H]��m�������
��233��_��������_���'����p�3��B��������;���{������A�t]������^���[h�|�[�����Fw����i���+�s���E���7�a�=�=�����~���;���E����l�{���������ZD����>`��������
�������3�	��n������A��]�v���{�;2������e�Irr�o~�1����)��;wN$����/�����l�����[���O�>4?�N�����m�.]���1e��"n7�X��������ZD�bJ�!�1Q���hT��7��{�n�EN�>M������4M���?�0z����+�6��1�B����~���%z��ur>��l����D��4��'O��������)��������|E��?lgbB�
y,�9����F���Qw<x0����$�w��t;�377�������-��]�u��)���5��.8[��6qnE���7������*	��DU�VUw����o|c���b��m;T�h�6K����>��W_}E���%��o���o~��;����RRR�	����v=���3��#��������8[�8���4���?_�������~���Nb�*@�QU�'�N����{���i�����PE��no��Y�����l���4hG7�������/WUU�m�&E�����(��������������)��?��E��#
�����p3����m�m��}QQQ�)@�BThu"�8�<���3g~���R�*--�����a����j�*
���7h���|�����~�������������o~�&K�	4���JOO�<��[o��D������V��q>��li��
7�XQ������ F!��������;Sl�w��%��}����E-�;��%H�!B��{���K�.��,X &��n4�~���@&�]�v�*-��Fw�:�g������18�kD��#
�����p3�����~���vNb�*�(��$�"�+�Xv��D�����Hg��F��Brr���]�����?���bKgDQ��y[p�����f?�����x�} � �x��c��,��b�*�� ��U�Q���
�-,���RO�������
��
��
��
��
��
��
��
��
��
��
��
��
�T_���]�.���dIEND�B`�
compactify_tuples_dgr_v2.patch.txttext/plain; charset=US-ASCII; name=compactify_tuples_dgr_v2.patch.txtDownload
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index d708117a40..b20559230c 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -411,50 +411,67 @@ PageRestoreTempPage(Page tempPage, Page oldPage)
 }
 
 /*
- * sorting support for PageRepairFragmentation and PageIndexMultiDelete
+ * Item pruning support for PageRepairFragmentation and PageIndexMultiDelete
  */
-typedef struct itemIdSortData
+typedef struct itemIdCompactData
 {
 	uint16		offsetindex;	/* linp array index */
 	int16		itemoff;		/* page offset of item data */
 	uint16		alignedlen;		/* MAXALIGN(item data len) */
-} itemIdSortData;
-typedef itemIdSortData *itemIdSort;
-
-static int
-itemoffcompare(const void *itemidp1, const void *itemidp2)
-{
-	/* Sort in decreasing itemoff order */
-	return ((itemIdSort) itemidp2)->itemoff -
-		((itemIdSort) itemidp1)->itemoff;
-}
+} itemIdCompactData;
+typedef itemIdCompactData *itemIdCompact;
 
 /*
  * After removing or marking some line pointers unused, move the tuples to
  * remove the gaps caused by the removed items.
  */
 static void
-compactify_tuples(itemIdSort itemidbase, int nitems, Page page)
+compactify_tuples(itemIdCompact itemidbase, int nitems, Page page)
 {
 	PageHeader	phdr = (PageHeader) page;
 	Offset		upper;
+	PGAlignedBlock scratch;
+	char	   *scratchptr = scratch.data;
 	int			i;
 
-	/* sort itemIdSortData array into decreasing itemoff order */
-	qsort((char *) itemidbase, nitems, sizeof(itemIdSortData),
-		  itemoffcompare);
+	/*
+	 * The tuples in the itemidbase array may be in any order so in order to
+	 * move these to the end of the page we must make a temp copy of each
+	 * tuple before we copy them back into the page at the new position.
+	 *
+	 * If a large number of the tuples have been pruned (>75%) then we'll copy
+	 * these into the temp buffer tuple-by-tuple, otherwise we'll just copy
+	 * the entire tuple section of the page in a single memcpy().  Doing this
+	 * saves us doing large copies when we're pruning most of the tuples.
+	 */
+	if (nitems < PageGetMaxOffsetNumber(page) / 4)
+	{
+		for (i = 0; i < nitems; i++)
+		{
+			itemIdCompact	itemidptr = &itemidbase[i];
+			memcpy(scratchptr + itemidptr->itemoff, page + itemidptr->itemoff,
+				   itemidptr->alignedlen);
+		}
+	}
+	else
+	{
+		/* Copy the entire tuple space */
+		memcpy(scratchptr + phdr->pd_upper,
+			   page + phdr->pd_upper,
+			   phdr->pd_special - phdr->pd_upper);
+	}
 
 	upper = phdr->pd_special;
 	for (i = 0; i < nitems; i++)
 	{
-		itemIdSort	itemidptr = &itemidbase[i];
+		itemIdCompact	itemidptr = &itemidbase[i];
 		ItemId		lp;
 
 		lp = PageGetItemId(page, itemidptr->offsetindex + 1);
 		upper -= itemidptr->alignedlen;
-		memmove((char *) page + upper,
-				(char *) page + itemidptr->itemoff,
-				itemidptr->alignedlen);
+		memcpy((char *) page + upper,
+			   scratchptr + itemidptr->itemoff,
+			   itemidptr->alignedlen);
 		lp->lp_off = upper;
 	}
 
@@ -477,8 +494,8 @@ PageRepairFragmentation(Page page)
 	Offset		pd_lower = ((PageHeader) page)->pd_lower;
 	Offset		pd_upper = ((PageHeader) page)->pd_upper;
 	Offset		pd_special = ((PageHeader) page)->pd_special;
-	itemIdSortData itemidbase[MaxHeapTuplesPerPage];
-	itemIdSort	itemidptr;
+	itemIdCompactData itemidbase[MaxHeapTuplesPerPage];
+	itemIdCompact	itemidptr;
 	ItemId		lp;
 	int			nline,
 				nstorage,
@@ -831,9 +848,9 @@ PageIndexMultiDelete(Page page, OffsetNumber *itemnos, int nitems)
 	Offset		pd_lower = phdr->pd_lower;
 	Offset		pd_upper = phdr->pd_upper;
 	Offset		pd_special = phdr->pd_special;
-	itemIdSortData itemidbase[MaxIndexTuplesPerPage];
+	itemIdCompactData itemidbase[MaxIndexTuplesPerPage];
 	ItemIdData	newitemids[MaxIndexTuplesPerPage];
-	itemIdSort	itemidptr;
+	itemIdCompact	itemidptr;
 	ItemId		lp;
 	int			nline,
 				nused;
#10Thomas Munro
thomas.munro@gmail.com
In reply to: David Rowley (#8)
Re: Optimising compactify_tuples()

On Wed, Sep 9, 2020 at 3:47 AM David Rowley <dgrowleyml@gmail.com> wrote:

On Tue, 8 Sep 2020 at 12:08, Thomas Munro <thomas.munro@gmail.com> wrote:

One thought is that if we're going to copy everything out and back in
again, we might want to consider doing it in a
memory-prefetcher-friendly order. Would it be a good idea to
rearrange the tuples to match line pointer order, so that the copying
work and also later sequential scans are in a forward direction?

That's an interesting idea but wouldn't that require both the copy to
the separate buffer *and* a qsort? That's the worst of both
implementations. We'd need some other data structure too in order to
get the index of the sorted array by reverse lineitem point, which
might require an additional array and an additional sort.

Well I may not have had enough coffee yet but I thought you'd just
have to spin though the item IDs twice. Once to compute sum(lp_len)
so you can compute the new pd_upper, and the second time to copy the
tuples from their random locations on the temporary page to new
sequential locations, so that afterwards item ID order matches offset
order.

#11David Rowley
dgrowleyml@gmail.com
In reply to: Thomas Munro (#10)
1 attachment(s)
Re: Optimising compactify_tuples()

On Wed, 9 Sep 2020 at 05:38, Thomas Munro <thomas.munro@gmail.com> wrote:

On Wed, Sep 9, 2020 at 3:47 AM David Rowley <dgrowleyml@gmail.com> wrote:

On Tue, 8 Sep 2020 at 12:08, Thomas Munro <thomas.munro@gmail.com> wrote:

One thought is that if we're going to copy everything out and back in
again, we might want to consider doing it in a
memory-prefetcher-friendly order. Would it be a good idea to
rearrange the tuples to match line pointer order, so that the copying
work and also later sequential scans are in a forward direction?

That's an interesting idea but wouldn't that require both the copy to
the separate buffer *and* a qsort? That's the worst of both
implementations. We'd need some other data structure too in order to
get the index of the sorted array by reverse lineitem point, which
might require an additional array and an additional sort.

Well I may not have had enough coffee yet but I thought you'd just
have to spin though the item IDs twice. Once to compute sum(lp_len)
so you can compute the new pd_upper, and the second time to copy the
tuples from their random locations on the temporary page to new
sequential locations, so that afterwards item ID order matches offset
order.

I think you were adequately caffeinated. You're right that this is
fairly simple to do, but it looks even more simple than looping twice
of the array. I think it's just a matter of looping over the
itemidbase backwards and putting the higher itemid tuples at the end
of the page. I've done it this way in the attached patch.

I also added a presorted path which falls back on doing memmoves
without the temp buffer when the itemidbase array indicates that
higher lineitems all have higher offsets. I'm doing the presort check
in the calling function since that loops over the lineitems already.
We can just memmove the tuples in reverse order without overwriting
any yet to be moved tuples when these are in order.

Also, I added code to collapse the memcpy and memmoves for adjacent
tuples so that we perform the minimal number of calls to those
functions. Once we've previously compacted a page it seems that the
code is able to reduce the number of calls significantly. I added
some logging and reviewed at after a run of the benchmark and saw that
for about 192 tuples we're mostly just doing 3-4 memcpys in the
non-presorted path and just 2 memmoves, for the presorted code path.
I also found that in my test the presorted path was only taken 12.39%
of the time. Trying with 120 million UPDATEs instead of 12 million in
the test ended up reducing this to just 10.89%. It seems that it'll
just be 1 or 2 tuples spoiling it since new tuples will still be added
earlier in the page after we free up space to add more.

I also experimented seeing what would happen if I also tried to
collapse the memcpys for copying to the temp buffer. The performance
got a little worse from doing that. So I left that code #ifdef'd out

With the attached v3, performance is better. The test now runs
recovery 65.6 seconds, vs master's 148.5 seconds. So about 2.2x
faster.

We should probably consider what else can be done to try to write
pages with tuples for earlier lineitems earlier in the page. VACUUM
FULLs and friends will switch back to the opposite order when
rewriting the heap.

Also fixed my missing libc debug symbols:

24.90% postgres postgres [.] PageRepairFragmentation
15.26% postgres libc-2.31.so [.] __memmove_avx_unaligned_erms
9.61% postgres postgres [.] hash_search_with_hash_value
8.03% postgres postgres [.] compactify_tuples
6.25% postgres postgres [.] XLogReadBufferExtended
3.74% postgres postgres [.] PinBuffer
2.25% postgres postgres [.] hash_bytes
1.79% postgres postgres [.] heap_xlog_update
1.47% postgres postgres [.] LWLockRelease
1.44% postgres postgres [.] XLogReadRecord
1.33% postgres postgres [.] PageGetHeapFreeSpace
1.16% postgres postgres [.] DecodeXLogRecord
1.13% postgres postgres [.] pg_comp_crc32c_sse42
1.12% postgres postgres [.] LWLockAttemptLock
1.09% postgres postgres [.] StartupXLOG
0.90% postgres postgres [.] ReadBuffer_common
0.84% postgres postgres [.] SlruSelectLRUPage
0.74% postgres libc-2.31.so [.] __memcmp_avx2_movbe
0.68% postgres [kernel.kallsyms] [k] copy_user_generic_string
0.66% postgres postgres [.] PageAddItemExtended
0.66% postgres postgres [.] PageIndexTupleOverwrite
0.62% postgres postgres [.] smgropen
0.60% postgres postgres [.] ReadPageInternal
0.57% postgres postgres [.] GetPrivateRefCountEntry
0.52% postgres postgres [.] heap_redo
0.51% postgres postgres [.] AdvanceNextFullTransactionIdPastXid

David

Attachments:

compactify_tuples_dgr_v3.patch.txttext/plain; charset=US-ASCII; name=compactify_tuples_dgr_v3.patch.txtDownload
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index d708117a40..339dfc0763 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -411,51 +411,187 @@ PageRestoreTempPage(Page tempPage, Page oldPage)
 }
 
 /*
- * sorting support for PageRepairFragmentation and PageIndexMultiDelete
+ * Item pruning support for PageRepairFragmentation and PageIndexMultiDelete
  */
-typedef struct itemIdSortData
+typedef struct itemIdCompactData
 {
 	uint16		offsetindex;	/* linp array index */
 	int16		itemoff;		/* page offset of item data */
 	uint16		alignedlen;		/* MAXALIGN(item data len) */
-} itemIdSortData;
-typedef itemIdSortData *itemIdSort;
-
-static int
-itemoffcompare(const void *itemidp1, const void *itemidp2)
-{
-	/* Sort in decreasing itemoff order */
-	return ((itemIdSort) itemidp2)->itemoff -
-		((itemIdSort) itemidp1)->itemoff;
-}
+} itemIdCompactData;
+typedef itemIdCompactData *itemIdCompact;
 
 /*
  * After removing or marking some line pointers unused, move the tuples to
  * remove the gaps caused by the removed items.
+ *
+ * Callers may pass 'presorted' as true if the itemidbase array is sorted in
+ * ascending order of itemoff.  This allows a slightly more optimal code path
+ * to be taken.
  */
 static void
-compactify_tuples(itemIdSort itemidbase, int nitems, Page page)
+compactify_tuples(itemIdCompact itemidbase, int nitems, Page page, bool presorted)
 {
 	PageHeader	phdr = (PageHeader) page;
 	Offset		upper;
 	int			i;
 
-	/* sort itemIdSortData array into decreasing itemoff order */
-	qsort((char *) itemidbase, nitems, sizeof(itemIdSortData),
-		  itemoffcompare);
-
-	upper = phdr->pd_special;
-	for (i = 0; i < nitems; i++)
+	if (presorted)
 	{
-		itemIdSort	itemidptr = &itemidbase[i];
-		ItemId		lp;
+		Offset	copy_tail;
+		Offset	copy_head;
+		itemIdCompact	itemidptr;
+
+		/*
+		 * The order of lineitem offsets is already in the optimal order, i.e,
+		 * lower item pointers have a lower offset.  This allows us to move
+		 * the tuples up to the end of the heap in reverse order without
+		 * having to worry about overwriting memory of other tuples during the
+		 * move operation.
+		 */
+
+		/*
+		 * Double check the caller didn't mess up while setting the presorted
+		 * flag to true.
+		 */
+#ifdef USE_ASSERT_CHECKING
+		{
+			Offset lastoff = 0;
+
+			for (i = 0; i < nitems; i++)
+			{
+				itemidptr = &itemidbase[i];
+				if (lastoff > itemidptr->itemoff)
+					Assert(false);
+				lastoff = itemidptr->itemoff;
+			}
+		}
+#endif /* USE_ASSERT_CHECKING */
 
-		lp = PageGetItemId(page, itemidptr->offsetindex + 1);
-		upper -= itemidptr->alignedlen;
+		/*
+		 * Do the tuple compactification.  Collapse memmove calls for adjacent
+		 * tuples.
+		 */
+		upper = phdr->pd_special;
+		copy_tail = copy_head = itemidbase[nitems - 1].itemoff + itemidbase[nitems - 1].alignedlen;
+		for (i = nitems - 1; i >= 0; i--)
+		{
+			ItemId		lp;
+
+			itemidptr = &itemidbase[i];
+			lp = PageGetItemId(page, itemidptr->offsetindex + 1);
+
+			if (copy_head != itemidptr->itemoff + itemidptr->alignedlen)
+			{
+				memmove((char *) page + upper,
+						page + copy_head,
+						copy_tail - copy_head);
+				copy_tail = itemidptr->itemoff + itemidptr->alignedlen;
+			}
+			upper -= itemidptr->alignedlen;
+			copy_head = itemidptr->itemoff;
+
+			lp->lp_off = upper;
+
+		}
+
+		/* move the remaining chunk */
 		memmove((char *) page + upper,
-				(char *) page + itemidptr->itemoff,
-				itemidptr->alignedlen);
-		lp->lp_off = upper;
+				page + copy_head,
+				copy_tail - copy_head);
+	}
+	else
+	{
+		Offset	copy_tail;
+		Offset	copy_head;
+		itemIdCompact	itemidptr;
+		PGAlignedBlock scratch;
+		char	   *scratchptr = scratch.data;
+
+		/*
+		 * The tuples in the itemidbase array may be in any order so in order to
+		 * move these to the end of the page we must make a temp copy of each
+		 * tuple before we copy them back into the page at the new position.
+		 *
+		 * If a large number of the tuples have been pruned (>75%) then we'll copy
+		 * these into the temp buffer tuple-by-tuple, otherwise, we'll just copy
+		 * the entire tuple section of the page in a single memcpy().  Doing this
+		 * saves us doing large copies when we're pruning most of the tuples.
+		 */
+		if (nitems < PageGetMaxOffsetNumber(page) / 4)
+		{
+			for (i = 0; i < nitems; i++)
+			{
+				itemIdCompact	itemidptr = &itemidbase[i];
+				memcpy(scratchptr + itemidptr->itemoff, page + itemidptr->itemoff,
+					   itemidptr->alignedlen);
+			}
+		}
+		else
+		{
+#ifdef NOTUSED
+			/*
+			 * Try to do the minimal amount of copying to get the required
+			 * tuples into the temp buffer.
+			 */
+			copy_tail = copy_head = itemidbase[0].itemoff + itemidbase[0].alignedlen;
+			for (i = 0; i < nitems; i++)
+			{
+				itemidptr = &itemidbase[i];
+
+				if (copy_head != itemidptr->itemoff + itemidptr->alignedlen)
+				{
+					memcpy((char *) scratchptr + copy_head,
+						   page + copy_head,
+						   copy_tail - copy_head);
+					copy_tail = itemidptr->itemoff + itemidptr->alignedlen;
+				}
+				copy_head = itemidptr->itemoff;
+			}
+
+			/* Copy the remaining chunk */
+			memcpy((char *) scratchptr + copy_head,
+				   page + copy_head,
+				   copy_tail - copy_head);
+#else
+			/* Copy the entire tuple space */
+			memcpy(scratchptr + phdr->pd_upper,
+				   page + phdr->pd_upper,
+				   phdr->pd_special - phdr->pd_upper);
+#endif
+		}
+
+		/*
+		 * Do the tuple compactification.  Collapse memmove calls for adjacent
+		 * tuples.
+		 */
+		upper = phdr->pd_special;
+		copy_tail = copy_head = itemidbase[nitems - 1].itemoff + itemidbase[nitems - 1].alignedlen;
+		for (i = nitems - 1; i >= 0; i--)
+		{
+			ItemId		lp;
+
+			itemidptr = &itemidbase[i];
+			lp = PageGetItemId(page, itemidptr->offsetindex + 1);
+
+			if (copy_head != itemidptr->itemoff + itemidptr->alignedlen)
+			{
+				memcpy((char *) page + upper,
+					   scratchptr + copy_head,
+					   copy_tail - copy_head);
+				copy_tail = itemidptr->itemoff + itemidptr->alignedlen;
+			}
+			upper -= itemidptr->alignedlen;
+			copy_head = itemidptr->itemoff;
+
+			lp->lp_off = upper;
+
+		}
+
+		/* Copy the remaining chunk */
+		memcpy((char *) page + upper,
+			   scratchptr + copy_head,
+			   copy_tail - copy_head);
 	}
 
 	phdr->pd_upper = upper;
@@ -477,14 +613,16 @@ PageRepairFragmentation(Page page)
 	Offset		pd_lower = ((PageHeader) page)->pd_lower;
 	Offset		pd_upper = ((PageHeader) page)->pd_upper;
 	Offset		pd_special = ((PageHeader) page)->pd_special;
-	itemIdSortData itemidbase[MaxHeapTuplesPerPage];
-	itemIdSort	itemidptr;
+	Offset		last_offset;
+	itemIdCompactData itemidbase[MaxHeapTuplesPerPage];
+	itemIdCompact	itemidptr;
 	ItemId		lp;
 	int			nline,
 				nstorage,
 				nunused;
 	int			i;
 	Size		totallen;
+	bool		presorted = true; /* For now */
 
 	/*
 	 * It's worth the trouble to be more paranoid here than in most places,
@@ -509,6 +647,7 @@ PageRepairFragmentation(Page page)
 	nline = PageGetMaxOffsetNumber(page);
 	itemidptr = itemidbase;
 	nunused = totallen = 0;
+	last_offset = 0;
 	for (i = FirstOffsetNumber; i <= nline; i++)
 	{
 		lp = PageGetItemId(page, i);
@@ -518,6 +657,12 @@ PageRepairFragmentation(Page page)
 			{
 				itemidptr->offsetindex = i - 1;
 				itemidptr->itemoff = ItemIdGetOffset(lp);
+
+				if (last_offset > itemidptr->itemoff)
+					presorted = false;
+				else
+					last_offset = itemidptr->itemoff;
+
 				if (unlikely(itemidptr->itemoff < (int) pd_upper ||
 							 itemidptr->itemoff >= (int) pd_special))
 					ereport(ERROR,
@@ -552,7 +697,7 @@ PageRepairFragmentation(Page page)
 					 errmsg("corrupted item lengths: total %u, available space %u",
 							(unsigned int) totallen, pd_special - pd_lower)));
 
-		compactify_tuples(itemidbase, nstorage, page);
+		compactify_tuples(itemidbase, nstorage, page, presorted);
 	}
 
 	/* Set hint bit for PageAddItem */
@@ -831,9 +976,9 @@ PageIndexMultiDelete(Page page, OffsetNumber *itemnos, int nitems)
 	Offset		pd_lower = phdr->pd_lower;
 	Offset		pd_upper = phdr->pd_upper;
 	Offset		pd_special = phdr->pd_special;
-	itemIdSortData itemidbase[MaxIndexTuplesPerPage];
+	itemIdCompactData itemidbase[MaxIndexTuplesPerPage];
 	ItemIdData	newitemids[MaxIndexTuplesPerPage];
-	itemIdSort	itemidptr;
+	itemIdCompact	itemidptr;
 	ItemId		lp;
 	int			nline,
 				nused;
@@ -932,7 +1077,7 @@ PageIndexMultiDelete(Page page, OffsetNumber *itemnos, int nitems)
 	phdr->pd_lower = SizeOfPageHeaderData + nused * sizeof(ItemIdData);
 
 	/* and compactify the tuple data */
-	compactify_tuples(itemidbase, nused, page);
+	compactify_tuples(itemidbase, nused, page, false);
 }
 
 
#12Thomas Munro
thomas.munro@gmail.com
In reply to: David Rowley (#11)
Re: Optimising compactify_tuples()

On Thu, Sep 10, 2020 at 2:34 AM David Rowley <dgrowleyml@gmail.com> wrote:

I think you were adequately caffeinated. You're right that this is
fairly simple to do, but it looks even more simple than looping twice
of the array. I think it's just a matter of looping over the
itemidbase backwards and putting the higher itemid tuples at the end
of the page. I've done it this way in the attached patch.

Yeah, I was wondering how to make as much of the algorithm work in a
memory-forwards direction as possible (even the item pointer access),
but it was just a hunch. Once you have the adjacent-tuple merging
thing so you're down to just a couple of big memcpy calls, it's
probably moot anyway.

I also added a presorted path which falls back on doing memmoves
without the temp buffer when the itemidbase array indicates that
higher lineitems all have higher offsets. I'm doing the presort check
in the calling function since that loops over the lineitems already.
We can just memmove the tuples in reverse order without overwriting
any yet to be moved tuples when these are in order.

Great.

I wonder if we could also identify a range at the high end that is
already correctly sorted and maximally compacted so it doesn't even
need to be copied out.

+        * Do the tuple compactification.  Collapse memmove calls for adjacent
+        * tuples.

s/memmove/memcpy/

With the attached v3, performance is better. The test now runs
recovery 65.6 seconds, vs master's 148.5 seconds. So about 2.2x
faster.

On my machine I'm seeing 57s, down from 86s on unpatched master, for
the simple pgbench workload from
https://github.com/macdice/redo-bench/. That's not quite what you're
reporting but it still blows the doors off the faster sorting patch,
which does it in 74s.

We should probably consider what else can be done to try to write
pages with tuples for earlier lineitems earlier in the page. VACUUM
FULLs and friends will switch back to the opposite order when
rewriting the heap.

Yeah, and also bulk inserts/COPY. Ultimately if we flipped our page
format on its head that'd come for free, but that'd be a bigger
project with more ramifications.

So one question is whether we want to do the order-reversing as part
of this patch, or wait for a more joined-up project to make lots of
code paths collude on making scan order match memory order
(corellation = 1). Most or all of the gain from your patch would
presumably still apply if did the exact opposite and forced offset
order to match reverse-item ID order (correlation = -1), which also
happens to be the initial state when you insert tuples today; you'd
still tend towards a state that allows nice big memmov/memcpy calls
during page compaction. IIUC currently we start with correlation -1
and then tend towards correlation = 0 after many random updates
because we can't change the order, so it gets scrambled over time.
I'm not sure what I think about that.

PS You might as well post future patches with .patch endings so that
the cfbot tests them; it seems pretty clear now that a patch to
optimise sorting (as useful as it may be for future work) can't beat a
patch to skip it completely. I took the liberty of switching the
author and review names in the commitfest entry to reflect this.

#13David Rowley
dgrowleyml@gmail.com
In reply to: Thomas Munro (#12)
5 attachment(s)
Re: Optimising compactify_tuples()

On Thu, 10 Sep 2020 at 10:40, Thomas Munro <thomas.munro@gmail.com> wrote:

I wonder if we could also identify a range at the high end that is
already correctly sorted and maximally compacted so it doesn't even
need to be copied out.

I've experimented quite a bit with this patch today. I think I've
tested every idea you've mentioned here, so there's quite a lot of
information to share.

I did write code to skip the copy to the separate buffer for tuples
that are already in the correct place and with a version of the patch
which keeps tuples in their traditional insert order (later lineitem's
tuple located earlier in the page) I see a generally a very large
number of tuples being skipped with this method. See attached
v4b_skipped_tuples.png. The vertical axis is the number of
compactify_tuple() calls during the benchmark where we were able to
skip that number of tuples. The average skipped tuples overall calls
during recovery was 81 tuples, so we get to skip about half the tuples
in the page doing this on this benchmark.

With the attached v3, performance is better. The test now runs
recovery 65.6 seconds, vs master's 148.5 seconds. So about 2.2x
faster.

On my machine I'm seeing 57s, down from 86s on unpatched master, for
the simple pgbench workload from
https://github.com/macdice/redo-bench/. That's not quite what you're
reporting but it still blows the doors off the faster sorting patch,
which does it in 74s.

Thanks for running the numbers on that. I might be seeing a bit more
gain as I dropped the fillfactor down to 85. That seems to cause more
calls to compactify_tuples().

So one question is whether we want to do the order-reversing as part
of this patch, or wait for a more joined-up project to make lots of
code paths collude on making scan order match memory order
(corellation = 1). Most or all of the gain from your patch would
presumably still apply if did the exact opposite and forced offset
order to match reverse-item ID order (correlation = -1), which also
happens to be the initial state when you insert tuples today; you'd
still tend towards a state that allows nice big memmov/memcpy calls
during page compaction. IIUC currently we start with correlation -1
and then tend towards correlation = 0 after many random updates
because we can't change the order, so it gets scrambled over time.
I'm not sure what I think about that.

So I did lots of benchmarking with both methods and my conclusion is
that I think we should stick to the traditional INSERT order with this
patch. But we should come back and revisit that more generally one
day. The main reason that I'm put off flipping the tuple order is that
it significantly reduces the number of times we hit the preordered
case. We go to all the trouble of reversing the order only to have it
broken again when we add 1 more tuple to the page. If we keep this
the traditional way, then it's much more likely that we'll maintain
that pre-order and hit the more optimal memmove code path.

To put that into numbers, using my benchmark, I see 13.25% of calls to
compactify_tuples() when the tuple order is reversed (i.e earlier
lineitems earlier in the page). However, if I keep the lineitems in
their proper order where earlier lineitems are at the end of the page
then I hit the preordered case 60.37% of the time. Hitting the
presorted case really that much more often is really speeding things
up even further.

I've attached some benchmark results as benchmark_table.txt, and
benchmark_chart.png.

The v4 patch implements your copy skipping idea for the prefix of
tuples which are already in the correct location. v4b is that patch
but changed to keep the tuples in the traditional order. v5 was me
experimenting further by adding a precalculated end of tuple Offset to
save having to calculate it each time by adding itemoff and alignedlen
together. It's not an improvement, so but I just wanted to mention
that I tried it.

If you look the benchmark results, you'll see that v4b is the winner.
The v4b + NOTUSED is me changing the #ifdef NOTUSED part so that we
use the smarter code to populate the backup buffer. Remember that I
got 60.37% of calls hitting the preordered case in v4b, so less than
40% had to do the backup buffer. So the slowness of that code is more
prominent when you compare v5 to v5 NOTUSED since the benchmark is
hitting the non-preordered code 86.75% of the time with that version.

PS You might as well post future patches with .patch endings so that
the cfbot tests them; it seems pretty clear now that a patch to
optimise sorting (as useful as it may be for future work) can't beat a
patch to skip it completely. I took the liberty of switching the
author and review names in the commitfest entry to reflect this.

Thank you.

I've attached v4b (b is for backwards since the traditional backwards
tuple order is maintained). v4b seems to be able to run my benchmark
in 63 seconds. I did 10 runs today of yesterday's v3 patch and got an
average of 72.8 seconds, so quite a big improvement from yesterday.

The profile indicates there's now bigger fish to fry:

25.25% postgres postgres [.] PageRepairFragmentation
13.57% postgres libc-2.31.so [.] __memmove_avx_unaligned_erms
10.87% postgres postgres [.] hash_search_with_hash_value
7.07% postgres postgres [.] XLogReadBufferExtended
5.57% postgres postgres [.] compactify_tuples
4.06% postgres postgres [.] PinBuffer
2.78% postgres postgres [.] heap_xlog_update
2.42% postgres postgres [.] hash_bytes
1.65% postgres postgres [.] XLogReadRecord
1.55% postgres postgres [.] LWLockRelease
1.42% postgres postgres [.] SlruSelectLRUPage
1.38% postgres postgres [.] PageGetHeapFreeSpace
1.20% postgres postgres [.] DecodeXLogRecord
1.16% postgres postgres [.] pg_comp_crc32c_sse42
1.15% postgres postgres [.] StartupXLOG
1.14% postgres postgres [.] LWLockAttemptLock
0.90% postgres postgres [.] ReadBuffer_common
0.81% postgres libc-2.31.so [.] __memcmp_avx2_movbe
0.71% postgres postgres [.] smgropen
0.65% postgres postgres [.] PageAddItemExtended
0.60% postgres postgres [.] PageIndexTupleOverwrite
0.57% postgres postgres [.] ReadPageInternal
0.54% postgres postgres [.] UnpinBuffer.constprop.0
0.53% postgres postgres [.] AdvanceNextFullTransactionIdPastXid

I'll still class v4b as POC grade. I've not thought too hard about
comments or done a huge amount of testing on it. We'd better decide on
all the exact logic first.

I've also attached another tiny patch that I think is pretty useful
separate from this. It basically changes:

LOG: redo done at 0/D518FFD0

into:

LOG: redo done at 0/D518FFD0 system usage: CPU: user: 58.93 s,
system: 0.74 s, elapsed: 62.31 s

(I was getting sick of having to calculate the time spent from the log
timestamps.)

David

Attachments:

benchmark_table.txttext/plain; charset=US-ASCII; name=benchmark_table.txtDownload
v4b_skipped_tuples.pngimage/png; name=v4b_skipped_tuples.pngDownload
compactify_tuples_dgr_v4b.patchapplication/octet-stream; name=compactify_tuples_dgr_v4b.patchDownload
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index d708117a40..5e834f58fb 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -411,51 +411,218 @@ PageRestoreTempPage(Page tempPage, Page oldPage)
 }
 
 /*
- * sorting support for PageRepairFragmentation and PageIndexMultiDelete
+ * Item pruning support for PageRepairFragmentation and PageIndexMultiDelete
  */
-typedef struct itemIdSortData
+typedef struct itemIdCompactData
 {
 	uint16		offsetindex;	/* linp array index */
 	int16		itemoff;		/* page offset of item data */
 	uint16		alignedlen;		/* MAXALIGN(item data len) */
-} itemIdSortData;
-typedef itemIdSortData *itemIdSort;
-
-static int
-itemoffcompare(const void *itemidp1, const void *itemidp2)
-{
-	/* Sort in decreasing itemoff order */
-	return ((itemIdSort) itemidp2)->itemoff -
-		((itemIdSort) itemidp1)->itemoff;
-}
+} itemIdCompactData;
+typedef itemIdCompactData *itemIdCompact;
 
 /*
  * After removing or marking some line pointers unused, move the tuples to
  * remove the gaps caused by the removed items.
+ *
+ * Callers may pass 'presorted' as true if the itemidbase array is sorted in
+ * ascending order of itemoff.  This allows a slightly more optimal code path
+ * to be taken.
  */
 static void
-compactify_tuples(itemIdSort itemidbase, int nitems, Page page)
+compactify_tuples(itemIdCompact itemidbase, int nitems, Page page, bool presorted)
 {
 	PageHeader	phdr = (PageHeader) page;
 	Offset		upper;
 	int			i;
 
-	/* sort itemIdSortData array into decreasing itemoff order */
-	qsort((char *) itemidbase, nitems, sizeof(itemIdSortData),
-		  itemoffcompare);
-
-	upper = phdr->pd_special;
-	for (i = 0; i < nitems; i++)
+	if (presorted)
 	{
-		itemIdSort	itemidptr = &itemidbase[i];
-		ItemId		lp;
+		Offset	copy_tail;
+		Offset	copy_head;
+		itemIdCompact	itemidptr;
+
+		/*
+		 * The order of lineitem offsets is already in the optimal order, i.e,
+		 * lower item pointers have a lower offset.  This allows us to move
+		 * the tuples up to the end of the heap in reverse order without
+		 * having to worry about overwriting memory of other tuples during the
+		 * move operation.
+		 */
+
+		/*
+		 * Double check the caller didn't mess up while setting the presorted
+		 * flag to true.
+		 */
+#ifdef USE_ASSERT_CHECKING
+		{
+			Offset lastoff = 0;
 
-		lp = PageGetItemId(page, itemidptr->offsetindex + 1);
-		upper -= itemidptr->alignedlen;
+			for (i = 0; i < nitems; i++)
+			{
+				itemidptr = &itemidbase[i];
+				if (lastoff > itemidptr->itemoff)
+					Assert(false);
+				lastoff = itemidptr->itemoff;
+			}
+		}
+#endif /* USE_ASSERT_CHECKING */
+
+		/*
+		 * Do the tuple compactification.  Collapse memmove calls for adjacent
+		 * tuples.
+		 */
+		upper = phdr->pd_special;
+		copy_tail = copy_head = itemidbase[0].itemoff + itemidbase[0].alignedlen;
+		for (i = 0; i < nitems; i++)
+		{
+			ItemId		lp;
+
+			itemidptr = &itemidbase[i];
+			lp = PageGetItemId(page, itemidptr->offsetindex + 1);
+
+			if (copy_head != itemidptr->itemoff + itemidptr->alignedlen)
+			{
+				/*
+				 * memmove is likely to have pre-checks when the destination
+				 * and source are the same address.
+				 */
+				memmove((char *) page + upper,
+						page + copy_head,
+						copy_tail - copy_head);
+				copy_tail = itemidptr->itemoff + itemidptr->alignedlen;
+			}
+			upper -= itemidptr->alignedlen;
+			copy_head = itemidptr->itemoff;
+
+			lp->lp_off = upper;
+
+		}
+
+		/* move the remaining chunk */
 		memmove((char *) page + upper,
-				(char *) page + itemidptr->itemoff,
-				itemidptr->alignedlen);
-		lp->lp_off = upper;
+				page + copy_head,
+				copy_tail - copy_head);
+	}
+	else
+	{
+		Offset	copy_tail;
+		Offset	copy_head;
+		itemIdCompact	itemidptr = NULL;
+		PGAlignedBlock scratch;
+		char	   *scratchptr = scratch.data;
+
+		/*
+		 * The tuples in the itemidbase array may be in any order so in order to
+		 * move these to the end of the page we must make a temp copy of each
+		 * tuple before we copy them back into the page at the new position.
+		 *
+		 * If a large number of the tuples have been pruned (>75%) then we'll copy
+		 * these into the temp buffer tuple-by-tuple, otherwise, we'll just copy
+		 * the entire tuple section of the page in a single memcpy().  Doing this
+		 * saves us doing large copies when we're pruning most of the tuples.
+		 */
+		if (nitems < PageGetMaxOffsetNumber(page) / 4)
+		{
+			for (i = 0; i < nitems; i++)
+			{
+				itemIdCompact	itemidptr = &itemidbase[i];
+				memcpy(scratchptr + itemidptr->itemoff, page + itemidptr->itemoff,
+					   itemidptr->alignedlen);
+			}
+		}
+		else
+		{
+#ifdef NOTUSED
+			upper = phdr->pd_special;
+
+			/* We can skip backing up tuples that we'll later not need to move. */
+			i = 0;
+			do {
+				itemidptr = &itemidbase[i];
+				if (upper != itemidptr->itemoff + itemidptr->alignedlen)
+					break;
+				upper -= itemidptr->alignedlen;
+
+				i++;
+			} while (i < nitems);
+
+			/*
+			 * Try to do the minimal amount of copying to get the required
+			 * tuples into the temp buffer.
+			 */
+			copy_tail = copy_head = itemidptr->itemoff + itemidptr->alignedlen;
+			for (; i < nitems; i++)
+			{
+				itemidptr = &itemidbase[i];
+
+				if (copy_head != itemidptr->itemoff + itemidptr->alignedlen)
+				{
+					memcpy((char *) scratchptr + copy_head,
+						   page + copy_head,
+						   copy_tail - copy_head);
+					copy_tail = itemidptr->itemoff + itemidptr->alignedlen;
+				}
+				copy_head = itemidptr->itemoff;
+			}
+
+			/* Copy the remaining chunk */
+			memcpy((char *) scratchptr + copy_head,
+				   page + copy_head,
+				   copy_tail - copy_head);
+#else
+			/* Copy the entire tuple space */
+			memcpy(scratchptr + phdr->pd_upper,
+				   page + phdr->pd_upper,
+				   phdr->pd_special - phdr->pd_upper);
+#endif
+		}
+
+		upper = phdr->pd_special;
+
+		/* Skip over initial tuples that are already in the correct place */
+		i = 0;
+		do {
+			itemidptr = &itemidbase[i];
+			if (upper != itemidptr->itemoff + itemidptr->alignedlen)
+				break;
+			upper -= itemidptr->alignedlen;
+
+			i++;
+		} while (i < nitems);
+
+		/*
+		 * Do the tuple compactification.  Collapse memcpy calls for adjacent
+		 * tuples.
+		 */
+		copy_tail = copy_head = itemidptr->itemoff + itemidptr->alignedlen;
+		for (; i < nitems; i++)
+		{
+			ItemId		lp;
+
+			itemidptr = &itemidbase[i];
+			lp = PageGetItemId(page, itemidptr->offsetindex + 1);
+
+			/* copy pending tuples when we detect a gap */
+			if (copy_head != itemidptr->itemoff + itemidptr->alignedlen)
+			{
+				memcpy((char *) page + upper,
+					   scratchptr + copy_head,
+					   copy_tail - copy_head);
+				/* reset where we've copied up to */
+				copy_tail = itemidptr->itemoff + itemidptr->alignedlen;
+			}
+			upper -= itemidptr->alignedlen;
+			copy_head = itemidptr->itemoff;
+
+			lp->lp_off = upper;
+
+		}
+
+		/* Copy the remaining chunk */
+		memcpy((char *) page + upper,
+			   scratchptr + copy_head,
+			   copy_tail - copy_head);
 	}
 
 	phdr->pd_upper = upper;
@@ -477,14 +644,16 @@ PageRepairFragmentation(Page page)
 	Offset		pd_lower = ((PageHeader) page)->pd_lower;
 	Offset		pd_upper = ((PageHeader) page)->pd_upper;
 	Offset		pd_special = ((PageHeader) page)->pd_special;
-	itemIdSortData itemidbase[MaxHeapTuplesPerPage];
-	itemIdSort	itemidptr;
+	Offset		last_offset;
+	itemIdCompactData itemidbase[MaxHeapTuplesPerPage];
+	itemIdCompact	itemidptr;
 	ItemId		lp;
 	int			nline,
 				nstorage,
 				nunused;
 	int			i;
 	Size		totallen;
+	bool		presorted = true; /* For now */
 
 	/*
 	 * It's worth the trouble to be more paranoid here than in most places,
@@ -509,6 +678,7 @@ PageRepairFragmentation(Page page)
 	nline = PageGetMaxOffsetNumber(page);
 	itemidptr = itemidbase;
 	nunused = totallen = 0;
+	last_offset = pd_special;
 	for (i = FirstOffsetNumber; i <= nline; i++)
 	{
 		lp = PageGetItemId(page, i);
@@ -518,6 +688,12 @@ PageRepairFragmentation(Page page)
 			{
 				itemidptr->offsetindex = i - 1;
 				itemidptr->itemoff = ItemIdGetOffset(lp);
+
+				if (last_offset > itemidptr->itemoff)
+					last_offset = itemidptr->itemoff;
+				else
+					presorted = false;
+
 				if (unlikely(itemidptr->itemoff < (int) pd_upper ||
 							 itemidptr->itemoff >= (int) pd_special))
 					ereport(ERROR,
@@ -552,7 +728,7 @@ PageRepairFragmentation(Page page)
 					 errmsg("corrupted item lengths: total %u, available space %u",
 							(unsigned int) totallen, pd_special - pd_lower)));
 
-		compactify_tuples(itemidbase, nstorage, page);
+		compactify_tuples(itemidbase, nstorage, page, presorted);
 	}
 
 	/* Set hint bit for PageAddItem */
@@ -831,9 +1007,9 @@ PageIndexMultiDelete(Page page, OffsetNumber *itemnos, int nitems)
 	Offset		pd_lower = phdr->pd_lower;
 	Offset		pd_upper = phdr->pd_upper;
 	Offset		pd_special = phdr->pd_special;
-	itemIdSortData itemidbase[MaxIndexTuplesPerPage];
+	itemIdCompactData itemidbase[MaxIndexTuplesPerPage];
 	ItemIdData	newitemids[MaxIndexTuplesPerPage];
-	itemIdSort	itemidptr;
+	itemIdCompact	itemidptr;
 	ItemId		lp;
 	int			nline,
 				nused;
@@ -932,7 +1108,7 @@ PageIndexMultiDelete(Page page, OffsetNumber *itemnos, int nitems)
 	phdr->pd_lower = SizeOfPageHeaderData + nused * sizeof(ItemIdData);
 
 	/* and compactify the tuple data */
-	compactify_tuples(itemidbase, nused, page);
+	compactify_tuples(itemidbase, nused, page, false);
 }
 
 
report_recovery_resouce_usage_at_end_of_recovery.patchapplication/octet-stream; name=report_recovery_resouce_usage_at_end_of_recovery.patchDownload
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 09c01ed4ae..046c99bfec 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -76,6 +76,7 @@
 #include "utils/memutils.h"
 #include "utils/ps_status.h"
 #include "utils/relmapper.h"
+#include "utils/pg_rusage.h"
 #include "utils/snapmgr.h"
 #include "utils/timestamp.h"
 
@@ -7169,6 +7170,9 @@ StartupXLOG(void)
 		{
 			ErrorContextCallback errcallback;
 			TimestampTz xtime;
+			PGRUsage	ru0;
+
+			pg_rusage_init(&ru0);
 
 			InRedo = true;
 
@@ -7435,8 +7439,9 @@ StartupXLOG(void)
 			}
 
 			ereport(LOG,
-					(errmsg("redo done at %X/%X",
-							(uint32) (ReadRecPtr >> 32), (uint32) ReadRecPtr)));
+					(errmsg("redo done at %X/%X system usage: %s",
+							(uint32) (ReadRecPtr >> 32), (uint32) ReadRecPtr,
+							pg_rusage_show(&ru0))));
 			xtime = GetLatestXTime();
 			if (xtime)
 				ereport(LOG,
benchmark_chart.pngimage/png; name=benchmark_chart.pngDownload
#14David Rowley
dgrowleyml@gmail.com
In reply to: David Rowley (#13)
2 attachment(s)
Re: Optimising compactify_tuples()

On Fri, 11 Sep 2020 at 01:45, David Rowley <dgrowleyml@gmail.com> wrote:

I've attached v4b (b is for backwards since the traditional backwards
tuple order is maintained). v4b seems to be able to run my benchmark
in 63 seconds. I did 10 runs today of yesterday's v3 patch and got an
average of 72.8 seconds, so quite a big improvement from yesterday.

After reading the patch back again I realised there are a few more
things that can be done to make it a bit faster.

1. When doing the backup buffer, use code to skip over tuples that
don't need to be moved at the end of the page and only memcpy() tuples
earlier than that.
2. The position that's determined in #1 can be used to start the
memcpy() loop at the first tuple that needs to be moved.
3. In the memmove() code for the preorder check, we can do a similar
skip of the tuples at the end of the page that don't need to be moved.

I also ditched the #ifdef'd out code as I'm pretty sure #1 and #2 are
a much better way of doing the backup buffer given how many tuples are
likely to be skipped due to maintaining the traditional tuple order.

That gets my benchmark down to 60.8 seconds, so 2.2 seconds better than v4b.

I've attached v6b and an updated chart showing the results of the 10
runs I did of it.

David

Attachments:

benchmark_chart_v6b.pngimage/png; name=benchmark_chart_v6b.pngDownload
�PNG


IHDR$�A�sRGB���gAMA���a	pHYs%%IR$��{IDATx^��y�lUy���?�c�;�������M�h�jbL��W���cTJ��� EEDAi�D@EQT�W@@T�C������Yo�������k�U���z�1��v����������j�%	`�� S�	�:,H��aA�0uX�����L$`�� S�	�:,H��aA�0uX�����L$`�� Sg�GuT�~������?�v�}�t�q��o�q������.}���l����pt�����;�n�����/~��i������x����3
n������~5}��h���>����������kt�~��{������0m?�X�c�8<�@:���������;��~����������-�9�}����o~�>��O�C=4����m:���������|���[��j����N�V��+y	0<�� q����m����m��6}�Cj�tj�Z6P�^{mSc��{�����aq���7������/,�/�_�&��e��g?����~p�1G���i����{���e��g>��f�y���u��1���>�������|�#M�u��g?���^k����B��#��[��g���^u�U�[��>����v�5*���_S(����o4�l�\T&/���� ����h���G[��`A�	�i/H|���~���<��Vj�M�k2�[p�-HhqL���E-�_z���{�9�zn��� �E��O>�Y=���������?Z��a���������%��/H��
:
��D��-����<��l��7�u��,�^�8��#��k����OoN+��~�}���P��]���&����������������t�q!|3p���X:��NG���;����w�[s~��_4�?�����r�{4��1�����7��$�>=�7����>5x��W�����,�\�����=��k���s���Od����.k������o!��D�g�EP��h��&]�c���k�����`t���	����D�&fN8�����>=u��,�\����~���axD&����7�+�1��8���������u]KB�����o�.VX��I��.H������IM�xh�B_��W�7�z���;4�=���~�|�p.��E����L;��S�
�ZA�<��sg=�|���z�������{�t�i�=�����S�����S9��\�<y�%��n}]0�[��V�m����h���sL����F�z$�7��A�<F��Z.�m��~�Oi�����'�O��D�	P�W�����{F?���	��+����wz.�~��"�:����^����nK�{�����C�h�������}���o~O����q���qv�&X�����\�lk�����N�}�^��W��l�E�C���>��Qu�����~o��,y����=��^z���z��~s����\s�������5$TW�W��^�Z�D�w�U�R�Q����j�����L�I�G���������y�����^��o_u�>�_[�p�
��_�rSo=mS������m����u�]��vx������t�mo���?�����B�������N�\ul��������������~�5�t[{\m��� �������R��mR?�k^���������=ou��}u�>�|�<7��~�����R��1}������SN9������$T�������r[�����+���[3����q��<6�h�����c���N'���nm=��t[����<��C��jw{����on^�����e�]f������~�����]����$��������7��=M���W���a�f��#�l���
������a����k�k&4���M�����n����}��w��>��������_��L��i:fBKzc��i;�������;=���?4�r����T�'_��^���o�
����Nj>��7��M�*3M`����$��'�8�DGKw"IN�4��.��@D����?��O�}�mG�C������������������x������\>����k��[��~�Y�6;�������c��k2M�+�}=�����u������o��f�����������^{��Bj����Z���6���������zR����?����d���9���+���Z��y���c�8m_K-������A��t�n�}�2�������{Z��>mS�Y}�[h�_�����QZd���7�������g�+��_���������}�Z��>U=�����<���n�Y���+f�]���]�}�fB������!J��y�����.u����^����E�ni�D�Au�rV�}5Soj|���}�����n��;����Z��k�d��nz��M�����~�����Mzz�Y��~��u��}[g�nm�d^��b�z3�I�3�8���h��jT���IM�u��zj�����X��[o��~����������4��z_�E�I?������q��Gu���W����
5��[������4���&��	����}}�S�"�M�R�&-�&��<��+�;��O�k��8�j?k���I�A������������V�����D��&_4����`�D�&Wf��&��O����������ImW*���L���`�iF��qjb}��'���1�s�~�v�I�V}���D�L�MF��G�r�W-F��~�������w���z��k����7���G����o_�I=�G���?����:��������
M�����3=������yJ'�/��g?G&�g�]����:>���V�L�������}
E�<7��^�����Fy<S
t���0>���!��.z��|��5�����A���:�Y�W��g�M[�'�/����g�]�J8��3�m�g���G7�����z�������{������~{���g�6k2/`q^��GMv���L��z�����y����7�z��N2��:��&�/_��������&/����/|��YM��G�?���7���&��iS��m��>���O�k����������;����oF�g�8��m���~W�S-�;��r�\���t���;4A�����i�V�h��N\��u?�bE�h�������^M��>�C���h�L�����p4��$=����G�����~����O&N��'Q>���6&�}�E���\����'�U��~_K�G�K�P?��iw�,�ZhbL�R����x�B���~���[K�M���+����G�+�G�A�ROh�i��������~��Em_?����~k����c{=����h������� �
�\�o����nC���~���.Fm[C���������^+���Z���>�����������]���6�s�����m�u���H_M�y�������h��ZP/���;�1�O��j_���N��HMfwv������z�����m�����>���/������T�j_��a��������~�.�u���n����Y�S�W��{�9������n���~_�W�����������.�������r���ObD��������j+U+}��=���!��K5_�����N�h>����]���`-�k�����1�����tQo�g:�t��8z��������l�g����5��z�zm�tX���7����n�G�zo�R����H��^�[�G�S����=����Z�y	����D_5	�7���,z��~�@o<���v�E���_����\oz[�������&[�w����u�����������q��MT�1��}��N��>N���N��
t;��I�q�=}*V?�����M��
�����������k1�}S�>���'`�}M4t��O�j�B��N�����B?�"�����N$��h?t'��_��vA�Y��~:w} ����z����mh����>��=�N{
.�L�x�}=�C���&Q�����}����T/ib�����q�����&.�_��	��>�~o���"F�<�i�=v�����&���O���j'��7��]�������/����KM������,Ft��Tu��E��vh�L������T����	3��k���m>�7�W�~:�h"��������L�a=n}�J��������������=��[�=m��M�����c��g����<�|�i�����m���v��B�����wM=v�����$�<�.��������7�g�[=����cP�S�����L"t����L��S��1wU�zF������1Z<h'���}�fA�8�>.=_=����o��Z�)�����^���K���������Y�2S�E��6�/��KX\<*�\��&��������������D�8��A;������6��#�7�R���:7�L_o�u����Z<h���O�jA@�"]4��O����S�hR��T�NW0���.z���
}J�;������Cw��������������&|�_���H�djMN�����E�Of�I��������/����vMhv'Y��t���z��/5���R�o�	-�����s;q<>���w�����q����G�&���'��.�%=��I�.��=��7A����}j�;���=��1��Di-�k��B�,M��t�Z�W3��\D�j��C�?~���S�O}�z������>��
�y���2�1@�K=������<uM~z�!��Ed�~��nNx�mj?������������������o���]�C���&�5q���!���I�Cwb[�}_��:^-z�w��:.��W�t3^��Q�������������^�&��P_�=>�s��j����[f��X�y	�����&���mMl��M{��z�Oj�
�LoT�����g�;��I�vQAo�K�1N���u:-��v:UB�M�&�u�&�O��=UD���Z�h�����&9�SCi�v?�������
M�i�`�q�������Y�oK�N$i�b6��jk��?�$�@���	Mdx�4����N��.�w�,�}��;�;>��R�f��xv'>g��j��x�L����Mbk�z���7A���3��L����h�x���o}2�����l��jk����&x�s����Z���}&"�D�j�C�z��(��������7���t��3�����>Y��k�C����}�����\BY��k��;��l���	��B�LD�[�����e�l��O���
-���i�D>h��T����o������~�L�{�X5��lt�Q�o���K����8��s����G�=G�=_�;�q�{n3�;�z���|�S������H�4��\}���.�qr���EMn��3�O��� �'��SiB����&G=��?�v��0����\�iD�	����t��w�'����N�h��{�]�	M�i��r�I�}j����>NMP�����G���D�N�2����;�>m�gRM�i��=��Lt'Xf�|��o|����l�\2�9�����vt�&�����~S���Wz���~�:h���<��cVZv>t'���G�Y[K}By�Zv?%�^l�i�������(Y�����������)�f�;�h[��W�@V_*O��t�l'O�C��+"��%����xz��6iqX�M����1)"���L����>�?�^��������$�A�<��o���U���~_6P��t�z���G�5�b�8m��/s]�|��-b��������I�W��}=u�y�L�^y	���-H����'\o���������k���5�2���k1�;�0�Z1�_��o�6��������K_��V�}8�d� ���vg{���o_��}.���lt'���;�>m��^�V��'��`)������=f���N���v�N�T��������o����K��,�)W���o���>?�z��A�g���\��sh���O�����R���o_M�y���l�����>����>���.��SM���W�];C���7+�p����"��e���l}_����D{�&��8��t:>}"|D�[�����x���3��}�Wf�����}�}��h�B�����]���������3=������9�	7��	�5�ZiaO�B���h- �[L�Y�.bv�y��m}�u�e�����{|�x�����DK{Nl�^lyM~�����&�����M�v��t�?�
���;�����Nw�}c����Ow�S�z�]�iO��	��\�-�V�&��sM�y�s&����N�t'�f����j& ���Z���dq�[�t'l���������}�^�r} ���mRMx}�z���}?�*&1���r��_2�9���s�i:4���'MXi�K�Q��������X��g��������W6[��t\(�;!�����r�gz��u��7|�}=^�R"}5��Q�3��o[���S�7Z�c��O�"�m�O�s�+��O_�7��z�����x���2��o��/y]
M��9*����fQX���=]��{<�$�����T�x����m��<t,��^��.�w{��z�T>��W�����>�2K~��(T�C=�9
��o@���A��E��^Sm�t���B�Q���}��z^����Z`�bL�z�7��t<nf�������zG_o�~aA`L}ABh��==�&@�@�7�]4!�NLh�~>�����i����_����EM�T��+��j��:��8z����~��%��o�e�]�T�!���~^����h'.d���&�����9�����������D��y?^�����&���h1�>��g�T3�q{�U�y�Z ]�;�U/i1I�k,sM����&@�E0M�u���7u���K���������3�I?��3��lt'���Qi-��o?��,��n_"}5��Q�3�	�x�2�1@j�y��������,�z��������e�]�����N#�������\����f���^�h|�w�G����w�B��s�x/�=�}�M*<��	��&����6��Nwr_���u{���Z4h�tz��^m��>�	�c� �������D���t����K��~�~�;���zk:/`q��,H�M��5�7��M4\q������f�\��6I��������f��D�.�S����no'�5!�7������iBH�+�$�&�S�����
�6u}2\�����	]�T��d���'
�f]���s�E]��*o'.d��������`t�i�5�Q�f����t'�f;�noO����$�@��9����������/M�z�$����&�u�����`�k�PD$��Uk���{h��+���Kt?k�o�=�\����y)���-Htk��L� 2�&��Ok;�Gw�z�V}�����GI�D$�sh��������c�&4���,�z����%�K���N�b��)�t,�m?�\�M�y�\��������i�w�G5���~g��E
��������-�Zl���n��P���jO)�]��~3D�m{v.�������ZF[/�Px�{=7-B�5i{��x�/f������P���q3��E����7,H�GeAB�
�G�������d��������j-(���z�Nf����sN3A��h�Co�[�������\}*Q�`zS���v����R����P���t_��ZG3������q���~2R�I��h���i�N�W�n'���u��8`���I���Ew���&"��t:�������s��H�����jk�j���l��@L�D��3M�����:��&}�3�o�w������=.}�[���lq&1�2���(����ot���~�y����n����H������~VM5����V�?/A�w��n-������h�o[K-��9�~:�zwQp�	��D�j����g�R�u=��x�����.�h?t�7��>O��:6��G�z�Z�j���5o"�Y��?��^���e��/y]���}=�vA\5�{�d�������j������v=�6�����<������NW��u��|�;�������D�A���#��z������^+���o�_:e]���}-(�v�X3���~O�?�C[��1k�������x�xu����oI���}��)�w;�����=��������Bz�-m������eM�%,.�	�	��
����rzS���������xM�h�J��5���q�c�BG��6t?���}�I�����Wo�[����ImW����E���N��:�C�}Mx�t�I��&C�����5�nK���D���E�����&,�w�m���G)��$MJ��������N�H�p|�c}�������a��~�}�������������{z:}H;���D'X�Z���#������t{���7���Y��o�o��gz�x�� ��R���������#1�i_"}5��Q�3�	}�Z����p�qj;�c�>����g���#�����m�O�G��������"���������,H�9�� 5�<���I3�s���&���t�
�oz��}�@?���-�w�N����z������������H>�tLz���x_���N�������1P�V_��J8��3���o����f������}�}�������n/k���������[�SG������<�#���t^���Q]��P}:P����j��=o���ZM�t'<��V�'����mh��]�I����Lt�����7�:�l���V{������
�NB���lW���bw�A��tAB��mM4�0��8J�N$�S�r���pz���D�6a�e�>V}�=����,�znZH�N�Lb����l�?� �}�O�����5������k>t��~��n��i}&e���g��j���^���L������qA�6A��������(����w��v���������%���L�_�t>�v��S�G��IR��.����Q�>��1[��Y��	�V�7��\�M�_����i�b�r[���������W������S����t�H>�������5^����U��?z��a��{j�R�Y�~nOq7���u����o��4�����5���^/Hc3=�.Z�����������x��_�k�GuAB����1����[�Q����d�����R�yo'%��1�7��f�&��'1Z4��m�T:�I;��IM�h�A�/����w�FZ���rJ��1>I��S��o���%�I�G���i!��Q���c�D�&��#D;q!�,Hh�����%�'4�0>����==�@�U�w��h�6a�e�>V�51��U�G}�nW���+�t���`�NB���#BH�_}���_}�U����&�4���XjwBmR�Y5�iV������+���lh��� !f��^[:E��T����&H��|��eR���g��z����~�^�S���^�:i�~�1X�M�{���1R�K�����s]�g|[������qK��o{�|����g��>���.[��Z����$���	�-zk"������c�i�O�3S�����h����}��w"��}����B��v��?���cZt��P�����Efz}�h��Q������}H�z��o_:^k�0��:��[�A��s��B�z���>@���>:-T���c�R�m�e>�����x_��e�z����6�<��J���&K�3M:�����6���9����6���	w�w�\�f����#����=�)�s=v=���z��h�B�R��������x�c��o��G���o���}K�7����K�9���[����������~�V����=~������z�������t[�������$�s���v�qt?=������Z���zn�>���|���h�^_u���(����w��z�:n��}�����>����k��&F���}>3Md�w?k?vG����}�����(y]�����+���UW]5��t�������c���}��z��{��oR�7�o�G���m����B�������iA�]$�
e��U�l�d}_�zm�q��E����
�����������h����:n����}^o�{���7�����������&2��jMxh�d���}�$�G�[w���	/F��IvM���Zh��"���o/*��;�t��_�X�� 10�	�v�K�F��7���t�����5a������S���Z����&�u
���/V�rj��N�j�s��Z<�M
]�Y���)��P,H-Fr�!i�wl�=���Z&7�t��^������u�����.��):��~G2��D���KG[�.CY��)�t--(�x���X����.��������J`Aj����IM�h��U�bg�u��'LZX����^����.�[�����D]l��`(B�����s����n5t�m}�������&%� ����9��S�E}rS�):U�\_(�������9������~�����W^9��Y�X�h����e�a���G���������������m��	�
$�N����HM�h�`hRV�ML�p��I�oI��_��i���o��y������^�b�t��ST;m�w��*����o�5�}��1W�,X�����L$`�� S�	�:,H��aA�0uX�����L$`�� S�	�:,H��aA�0uX�������Y�@JG�q:��������m�Rx -������z�f�{��}������j�����wM����p�����[0�|��'�=/�J��f�{?���7\8����	������7��<��g���{.O7���h��q��{�����t���i�.[��i�M7��X\����G���k�������mq������W�Iz����9�����nm`��������{�+w�;���{`���s����-��2��?�C��o������O��m��i��U/{�//mq�|���>��{4Xu�?������>x���-No=���c��>����u��w�%�-N��~���_�R���B����`A`�����,H\z�t$����F�����L�{����/�"��/H?���G��.����C_���������3�������%�z���[���s��]`�����d���2�>���>���7������C_v�t^��{o�u�]������~MJ�����B����`A`�,��8 ��U�J?�����W_����K_���o�����bZ�������L� �3�>bAbn.������?=���Z��.X���~��������^bA<X�0kjAB���l���-}�z���jN�r�m�����>������t�]w�-��"m��v������Z�(�G-�;��s�i�����!,HLz����\sM��������mZ����}��@-�}����~���V����fM-HhQ����#��p�w������S��TZ�j���}B��xF:��#��N�b�kjA��>�������_��Q�t��Z���>�m�y�k�f�m��&�y�{^s��G�5� Q�/����Z��.z,�}	$��Z��B���x:�&/O;���-�I�������&M5��h��$��#}�\����5?��I�#��k����W���K�UW]�~��6�x����L�O�5� Q�/��������5�y�[���&�!�2�
c!�KXx� 0`��5$.���������F��f���o|c���G�<��{���S��y�s�OY��kH��#M����g���������� !&��t�&��=�������O:�����>����6k�s�K�~I�z�"�_��f��}M����M��g?��4ja_���	��&$�S�|�ck&1����N����+������h�(�h��~M.H���t�.���L/[�l�[�Z���>�m����o.���fr��������&$���z�i��iO{Z:���W��������9��N�&�����}kpAB��>�����^�����^xa��i�tM���/y�K�7����-�cM.H���t�~�W�W���D����u?]D��SNi�;M����$��&���v��b����-�D���;�ez�������7���[o���/|!�G�=}���\�,H�5� !���iO
s�
74?�'�u
�|��'��<���krAB�������6����GkrABLji}�M6I��v[��m��O~r:��3F�L�5� !f����sO�j������3��\�,H�5� ��"f�vx����Ok��:��'�;������G�N�5� 1�>������_q����G��d}j��;�l�;M�������&��o���g�1����&��Mo�&����^�(yM>�Y���y��&����fM/H��:��&/���~b\�����&@_���5���/n����;6���fM/H���<4<�kH�I�#����6��i��o/z���O��|���'����^��k_��M�n�iz�s�;�kr!�KXx� 0`�����'�u��7��
����&-����ft�;�O�5� !f�GC[���G�m���n.���t�M�+z4X�b�}��o����I�5����>���kz_���	��$��j]@W���>5�N�^~���}4'4���\�hM��w,��I�#��^[�re��5�oaA�d_j_u_���[�,H��� ��Y��� QaA`Z� 0`�����C_�O�Rk����]`��w�5�b}<�����������8\�����C_���m�E������c�2~�����Y�m�F���t�qG���<o���:��h�r�������������Q�7mJ���;����N:��c���7^�x�}	$��{Lw����0�������w�9o��t-�5����������+G��+�Iw�w��]���
%,H��aA�0uX�����L$`�� S�	�:,H��aA�0uX�����L$`�� S�	�:,H��aA�^xaz�^��O}�S��{���a~��_�
6�`��$��b�_��_�%K��������������o�}t/[�X{�����d��y��Y���ZH��G>��������>:�}����n�i��?��t�������>���,F|��_���g�g��.�����fQa�-�L+V����O��f���c�i��oJ���:�}�����6q������Ei�=����<�@��G?�^��W4�dji�
q���6����+�������/}��w���+���n�6�|sN����1������������K/|����'�O|���;�l�s�Yge]v�q���7�!�z���[��1�������8�����g>3�GJ��~zs��;�$�.]��-[6�X���kC���|�+��'���<��������$t����������7��������������-����w^��?��t�QG5��h��|������z���_|qs���$��������{t���P]t""""""""""��U�].��s}���6�
7��|�b�=����7,���75����b��U��""""""""""�bu���Y��aA�C�����Z+]{���[Ss��'>��i�m�i�9��oA���4N����;��|����jn����sOs��?��?L�}�s�?��?6j��������8���~���~���6�L�y��^����4��]w]s?0X����nJ���s���������Nk,Z�����7�E��~Z����K�N�0$X�p���;��H�^r�%�E�[�(q����.��b�#aA�0uX�����L$`�� S�	�:,H��aA�0uX�����L$`�� S�	�E�-w=�~r��t�E�W���<Z� ���������H��wO�`���V���h{�,H�!4������"�����p���{�+w�{bj{�,H�t��wtOz�n��I�>j���������L\|���?_R�,H�N�x�;9Q����j���?�?�����
�]R���X��Ap��+�G��7m}��y���v&�����NG�6�~��G�/+/�E�k�����y_HmC����������F���7%�	X��������Bj0L�������W�����vX��Ip��'� �m�<p��t��?t�����v`a��>*�|����M7L7����W� ���mq�Y8}����"��	��	X�,?��t��7L�l�VHmC�����~�������D����>e|���J���M���b�}m�O-/,��G���z���Q������a�� ����<-{�R7�����m���~3b\m �4'z���#�{��������}T�������cFDm,H�4`A4��9�
���&o  �4'z&q����,��G�w�@��Q���0
X��
���G�2����PTX8x��
0���$`� �
&�7�
�	�/^����m��	�,H���`�K� ����!�
�@8*@$`�x��
0���$`� �
�9������'m���!�
m��7�
�	�/^����m��	�,H���`�.�V�:{,w5�Q������T��o��2}�����b�������i�����C�
m��	�/^��B�Lcl��M�� ��	����`:����{����}m��`�.G�u�;���mB�x��
u�7�o�gy��O��/U����(14�}���Io�=�x�mh[��1�0_��G������y��aA�0����:lEz������|�������`ta0���}T�����[����A�����n�#j�P/��x�������}T��i�m������4`A&�����=��WmOl������Q�n��W���9������	�2����^(*,��G������y���=8L$`"L�k�]�=A�
�K� }���!�
�C0_��G���j�����L�g��BQa���>*��W���y��al��	��&� ����3�o����~�Cj��
�!�/^��B�x5�
�CNA_��3�{���p�j���yT��M�0��i��L�Ulu�o5�i��}�F���u�`��W��-+yI�{����?�eHmC����W��P?���'�����cH��}m����PTX8x��
u��<�`��n�=��a��L�Ulus�������mB�0����>�x��[�����t�_�fN�
mK��5�W��P?��)-"����M�y�{)����vX�XL�g�����W��]V^������Cj�p�jU0oS7��)��>���4�R�~��P�$`"x���`��i�=���}Tq�Oq�&��	k��Q�~��S��}=��1_�=X�L�g��G���W��-��zH�!�[��NHmC����W���y���fN����t����n}���1�T��������	��&� ��f��u2��iN�����?����������W��b��YQ��5�W��-z�t��{���>j��ZXL3���������g�=��=(,��G�V��u_��>j[|Sba��<�`��n��Sw~�cn�����$`"x���`��i��4{f����[6|�dN���v�=X8x��*��z�jU������m;����W���)����������>�X�������m����yT�x�n��S�w��`q��L�Ulu3�`�:�f�p*���jU�S���<��4_u3���j�<����Qc�z�jU�3uCN�4`A&�w��*���f�A�Lu0��0���}TAN��W����|��Ts��}TX��30_��G�m���yTA��
9��	��&� ��f��u�`��W�������Q=S7��������>� ����yTA��
9��	��&� ��f��u�`��W�������Q=S7��������>� ����yTA��
9��	��&� ��f��&V|��t�NN�o��y���v`a�`��W�������Q=S7��������>� ����yTA��
9��	��&� ��f���E�[6Z+~]��~_�Yq�C0_��G�T�x5�*���!��/����Q9U/^��
z�n�)�,H�D�0Q�V7��[�c}���U��5�!�/^��
r�^��G�L��S�z��W�������Q=S7�L$`"x���`���[���>�=X�0����>� ����yTA��
9}�g`�x��*��z�jU�3uCN�4`A&�w��*��!��/����Q9U/^��
z�n�)�=���}TAN��W�����r
�0�LTA��
�}�g`�x��*��z�jU�3uCNA_��/^��
r�^��G�L��S0
X����`�
��n6�=���}TAN��W�����r
�B��|�jU�S���<��g����i��L�UluC�A_��/^���i��������������������jU0��r
�B��|�jU�S���<��g����i��L�UluC�A_��/^���i���w��V]������O��|����?�m��jU0��r
�B��|�jU�S���<��g����i��L�UluC�A_��/^���i����#����71�=(��yT���n�)�=���}TAN��W�����r
�0�LTA��
�}�g`�x��*��S+O{���0o����<�`lS7��������>� ����yTA��
9��	��&� ���`���30_��G��)wQ!(���<�`lS7��������>� ����yTA��
9��	��&� ���`���30_��G��)oA!*���<�`lS7��������>� ����yTA��
9��	��&� ���`���30_��G��)oA!*���<�`lS7��������>� ����yTA��
9��	��&� ���`���30_��G��)oA!*���<�`lS7��������>� ����yTA��
9��	��&� ���`���30_��G��)oA!*���<�`lS7��������>� ����yTA��
9��	��&� ��f1��w������m��`��W��b�9�-(D�r��G�m������30_��G�T�x5�*���!�`� �;�D[�,�`{����U��<�:��b>�
m�a0���}T1����B9^��
�6uCNA_��/^��
r�^��G�L��S0
X����`�
��nc��[
+�dw�o^>�-�)Q�!�/^���i��{�
�x5�*��
9}�g`�x��*��z�jU�3uCN�4`A&�w��*�l�wczp�����>����4�"�a1����N�E�6�C0_��G��)�x��jUL{lkr
�B��|�jU�S���<��g����i��L�UL3�����������_�V������h;��c1�W}�����mB�`�x��*��S��"*���<��f��������30_��G�T�x5�*���!�`� �;�D��.}�;Y3_�=��b6$�,�`�x��*��S��"*���<��f��������30_��G�T�x5�*���!�`� �;�D����=�����m�����5�!�/^������s��
�x5�*��3��!��/����Q9U/^��
z�n�)�,H�D�0Q�4�����
�X�����������Q9U/^���i��y�)�=���}TAN��W�����r
�0�LT1�`�&j�B?c�� �fY�������:<��x���h�n�lC���x��*��z�jUL�g`��Xs
��m����Q9U/^��
z�n��4`A&�w��*�l�DMT��b6$�,��g������W���=��/����mB?��G�T�x5�*��3B���}7=x���W��qf^,���5c�6���}TAN��W�������<�y��s�~�������	��&��f�������l,H�Ye��x�[���&���}TAN��W��b�=�O��h���g/L��z����?�>�����S�falc2���W�������Q=S7�ul��o�J��}��}<����`�� �p�]w�]v�%m�����m�]���~7����6��������t�}���&��f�y����E���5����k>��=��	��jU�S���<��f�<p�����Wm��sJh�F��Uo��Fg~n�l���~0�1����}TAN��W��������S�V��������|h[|Sb�� 1���_��]w����?>-Y�������6�l�lQB������~O{����8�E	�UL3���LP���|6��[��#�����;k������g�n�@^��
r�^��GS�����}�>�=��b�)��b�y��L�<�
m��b����dl���Q9U/^��
z�neN�fo���M�,Ht�B������������k�u�]�������/yt����#>������:���,J\z���{
�UL5��LT�o���
���^�����i�S�Mo���[������=c���?^��
r�^��G�L�,����;k�6&c��x��*��z�jU�3uCN���daA��u�]�^��W5W_��D�w��N<���������N���F5�����?�s�g�����m�%14�LTA��
�f����������u&��W���`�?^�����S��zM:�������3���m��W����M�,���f���M(������t�i�8���R���N�x��*�0���`lS7��INM$:|�;�i�	���~wt��5�\�^������t�q�����7�<m��V�;m�w��*s���������Q����l������]X���^m�3fw0��s�<��~���|�x��*cN������'r��6�����yT���607����)�z�l{F�3o:���EG���N��}m�����Qc�6}�yl�����S&95Y&� q�	'�M6���[���?<��e/K���o�����|$�~�9��������O��0;��c����o�}t�0�0Q�b
����v��;���������U+���RQk��1�����k���6������R�����W+^�����S?��,�xQ��
��Q�b�@�����
=c�=��C�/g���W+^��
�6&c�2�b�0oS9e�S�%� �or�!���s����w��������&�:�����o>���N��������t�W4�;�������q� �t���l���-��;�D�1��:����h���U��;�f����?jm�3f�3���W������W��b1��!���?��Y^�����3B�^|t����Bj5/~�Sf�S�����v~(_�~(U����=c�=�����^�x��*���m���g��)��2���2�	}c�C�Pz����\7a�}�m.�<�n��5����vR!��->l������Z�>���6���g�E�I-H�X�"�v�mU�`�j����=�D�6�m�U�=���i������O���}DmS��jU�]~�vn�#j���W�����Ez�l{�������|���}]�^��j��1��<��n�#j����E��Q����3�������0�c���mi����Ir�ls���NIo���Nk��v��v_�"=c�=��?j��k��}Tm������Lmw1��6��S&9Un	�Z���O��Zk���v��C=4}�{�k������s�o�M7M�}�k��_����~�����U�F��t��7���^{���.�����'=i�	-h�y���[fF����K��;�D�v��mg�Q����LTmw���`"j�������[���Mm��}Tm���7wkQ�����Gm�u-�3&=�_��Q�]r��1���G�vc�~�qn�#j����Ir�lsj�����j{���Ez�dl�_��Q�]�6&c�2�]z���g$9e�Se^uU�7�z/H�u�]����|���X�GZt-L|���v�&�N���V�����o���Zh_!k�s�x���o�t4f����o�W��&��{��{�������`�j�{��P�Q�l�uM������G�6�m��Q���.��[�������>j��k��1���z�����Sf�9��<���{f�ss�Q�l�uM�Sf�S/<j]���U�k�u-�3&c��z����2�1�����3f�=#�)��*��;��z�N�	}���/,�p���/�|�oT�I�`������>[0�7:������6��y��f�����o{�TNp@��?����C�;�D��\���(�Q���Eh��"�j�6�����W����2k�)��Q=c2�)����
z��g���>� �L�6e
z�dlS.9�;DB�n��7��|�A��������5��Zl�5/����7�y�sx�K_����'6��Z�9������3�g��6J�x�3�?��?�_�������w��*6�`+�`��
z��g���>� ��s��yTA���m�%�rk��1���x��*�)��M���1��KN����D��]��{�i.Z������-������O}�Ylh������t���7��h������������(�����w��*6�`+�`��
z��g���>� ��s��yTA���m�%�rk��1���x��*�)��M���1��KN���������}l�D�$v�u�f�^��:��~�����Xh���Nh�u!���������g�1���`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�C$� �	�u�Y'�����D�u�]�^��W���^;�|���m�����v��w��*6�`+�`��
z��g���>� ��s��yTA���m�%�rk��1���x��*�)��M���1��KN��������t�����������'�'��$��C��}�k.�AC�x���`3	�r	����gLz�?^��
r��1���G�����\r*�6�����W����2��)���M��T�	-H�4FZp�������w�#���/o��p�UW�_��W�B�.��n�
�TFC�;�D�I��K���=c�3��jU�Sf�9��<��gL�6��S��A���L��G�����LA���m�%�r�H���g�}v���:���v�*.������z����W^9�7��w��*6�`+�`��
z��g���>� ��s��yTA���m�%�rk��1���x��*�)��M���1��KN�������p�����:(��������,F�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���"�	�`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�C����.T��X����,V���A�x���`3	�r	����gLz�?^��
r��1���G�����\r*�6�����W����2��)���M��T��� �l���t��������=��Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���"�$V�X����o�<�q��vJOx��;�������6���Ci~��;�D�I��K���=c�3��jU�Sf�9��<��gL�6��S��A���L��G�����LA���m�%�r�H����g�}�������_�jt����m��������`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�C$� q����7��M�_����G��{4��}�^�LTA��[�[nm�3&=���Q9e��S^��
z�dlS.9�[��I����}TAN��m������\r*w��$�������W���������0����~s���;�D�I��K���=c�3��jU�Sf�9��<��gL�6��S��A���L��G�����LA���m�%�r�HhAb�����c�i��E�M7�4S��g�����`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�C$� !n���t��G�����i��%��M?�}�n�LTA��[�[nm�3&=���Q9e��S^��
z�dlS.9�[��I����}TAN��m������\r*w��$Z�=����~�e�6�&� �L��\�-�6�����W����2k�)��Q=c2�)����
z��g���>� �L�6e
z�dlS.9�;D&� ��;�D�I��K���=c�3��jU�Sf�9��<��gL�6��S��A���L��G�����LA���m�%�r��D$N:������wF��k�t��v��:�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!^��b�^��G\?����K��e�F�5�`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�C$� �o=���oM�|�3�������w_�O<1�X�b�[P#�&� �L��\�-�6�����W����2k�)��Q=c2�)����
z��g���>� �L�6e
z�dlS.9�;DB�����������0p�LTA��[�[nm�3&=���Q9e��S^��
z�dlS.9�[��I����}TAN��m������\r*w��$�������=��ct�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���"���+W���:*���������
C�;�D�I��K���=c�3��jU�Sf�9��<��gL�6��S��A���L��G�����LA���m�%�r�H����'������u�Y'm��f�������`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�Cd"��X�d���>�/��w��*6�`+�`��
z��g���>� ��s��yTA���m�%�rk��1���x��*�)��M���1��KN�����.d��o|#���^���p����0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!Z�h�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!2�����?}�k_K���3u�~��`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�C$� q��7��w�9=�Y�z��#t�~��@�x���`3	�r	����gLz�?^��
r��1���G�����\r*�6�����W����2��)���M��T�	/Hy���O��O��o���c�L����>P7�&� �L��\�-�6�����W����2k�)��Q=c2�)����
z��g���>� �L�6e
z�dlS.9�;DBw�}wz����,<\v�e�[F��g�����`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�C$� �l���t��t����ny$�����B�x���`3	�r	����gLz�?^��
r��1���G�����\r*�6�����W����2��)���M��T�	-H�X�"��=�I��/���=�������3�G��z�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!���7������?�Yx�r�-3u�~��@�x���`3	�r	����gLz�?^��
r��1���G�����\r*�6�����W����2��)���M��T�	/H�z�����N������,Y�����6v�a�}�n�LTA��[�[nm�3&=���Q9e��S^��
z�dlS.9�[��I����}TAN��m������\r*w��$���+��������}.�����j�;�D�I��K���=c�3��jU�Sf�9��<��gL�6��S��A���L��G�����LA���m�%�r��D$����t�������8���q��w��*6�`+�`��
z��g���>� ��s��yTA���m�%�rk��1���x��*�)��M���1��KN�����#�_���M�^{������s���'?9����m�u�`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�C$� q���7�p�n���n����OR:��3�?���g�����`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�C$� q����u�]7m���bD�n��t���;�D�I��K���=c�3��jU�Sf�9��<��gL�6��S��A���L��G�����LA���m�%�r�HhAb��ei����7 fB?�}t_��Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���"��{��'m�����/~qs�qt�~����P/�&� �L��\�-�6�����W����2k�)��Q=c2�)����
z��g���>� �L�6e
z�dlS.9�;D����������g����ui�m���m���u�`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�C$� �o>s�1�i��,Y����3�Q?�&� �L��\�-�6�����W����2k�)��Q=c2�)����
z��g���>� �L�6e
z�dlS.9�;D�-?��O�������`x���`3	�r	����gLz�?^��
r��1���G�����\r*�6�����W����2��)���M��T�����Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���"[��u"��v����??-_�<q����n�j�;�D�I��K���=c�3��jU�Sf�9��<��gL�6��S��A���L��G�����LA���m�%�r�HxAB{��Gz�K^����~xZ�lYz������w�k��fto��Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���"����>:����Yz�{��>�����?�����.d������=�qi���J����7�F�LTA��[�[nm�3&=���Q9e��S^��
z�dlS.9�[��I����}TAN��m������\r*w��$tZ�w��i���O�]w]����K�6��[nI�o�yZo���w���u�`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�C$� 1�1�o�oG�6���;�D�I��K���=c�3��jU�Sf�9��<��gL�6��S��A���L��G�����LA���m�%�r�HhAB�ez�������7�_����X�����K_���>�/��w��*6�`+�`��
z��g���>� ��s��yTA���m�%�rk��1���x��*�)��M���1��KN���5$N9�����=-m���i�-�L���,Pl��v�u
���?~to��Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���"�	}���#�L�~����%K2u�A����w��*6�`+�`��
z��g���>� ��s��yTA���m�%�rk��1���x��*�)��M���1��KN����D�O~�����;g�6�&� �L��\�-�6�����W����2k�)��Q=c2�)����
z��g���>� �L�6e
z�dlS.9�;D&� ��;�D�I��K���=c�3��jU�Sf�9��<��gL�6��S��A���L��G�����LA���m�%�r��D$~���N8!��������|�Q���`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�C$� q������ei�M7m�'��_���w{	���u�`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�C$� �oD�Z/x��7���fAb�}�I�}�c�[l�v�a��g�O��	��Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���"��;��#���z�S��TZ�re������_�5����[���M?�}t_��Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���"��e����K��������������������Y�X�|y�z��������D�x���`3	�r	����gLz�?^��
r��1���G�����\r*�6�����W����2��)���M��T�	-H�M�z����_���������Z+�����e�]�������| =��O]�
��Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���"��Z�[����X��{��g�Pq���7�^g�u�������V�LTA��[�[nm�3&=���Q9e��S^��
z�dlS.9�[��I����}TAN��m������\r*w��$���o}�[���D:����8	�W�f1bx���`3	�r	����gLz�?^��
r��1���G�����\r*�6�����W����2��)���M��T��� ��B��G?J�������{���v��LTA��[�[nm�3&=���Q9e��S^��
z�dlS.9�[��I����}TAN��m������\r*w��^����t����u"�p�E�v�!r�!�v��LTA��[�[nm�3&=���Q9e��S^��
z�dlS.9�[��I����}TAN��m������\r*w����MZX�l���v�m�>�����;l�o<�v�\�[{�����4�u�`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�Cd��������^{��������>��i�m�m�����+^����}�U��Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���"��Z�:'�xb����%K�<B}+B?�u'�^�LTA��[�[nm�3&=���Q9e��S^��
z�dlS.9�[��I����}TAN��m������\r*w��$�h�A����3
�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���"[��a�`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�C��9X�lY:�������-�����D������r���O��w��*6�`+�`��
z��g���>� ��s��yTA���m�%�rk��1���x��*�)��M���1��KN�$fA��zhz�c�?�����#v�y��g��2�����N8a���&� �L��\�-�6�����W����2k�)��Q=c2�)����
z��g���>� �L�6e
z�dlS.9�;DX���SO=5��/h�$�;��f1�mo{[�~��_���7�����G��&� �L��\�-�6�����W����2k�)��Q=c2�)����
z��g���>� �L�6e
z�dlS.9�;DX������7�-oyKZo���c��lA��{�I�z����k��vtkJ�����S���G,^�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H8�tL��G�{�������$=�����X���8���^�[�����w���i�m��i��LTA��[�[nm�3&=���Q9e��S^��
z�dlS.9�[��I����}TAN��m������\r*w�LlA��?�q�i����_4�"�i�4����B����������/�0]v�e�X�8�����5�x���[f�wl~��;��2�LTA��[�[nm�3&=���Q9e��S^��
z�dlS.9�[��I����}TAN��m������\r*w��$����/~1����^}qgM�����Z+m��������{�s��7������Nj��-H�~�����������K�b���������
�LTm��}?�`"j���w�����~���LDm���5���oqkQ�����G�v��h���Mm��}�v_�"=c�3��jU�%��s��yTm��1��KN���������W���.9e2�)S��gL�6��S�����[n�e4�=;����?����&�l���j��������N_����7�_�o���)��p����6�����\����;����w����^��O�������`�j�����LDm���5����i���W�����?{�[�������>j��k��1���z�����Sf�9��<��K���m�%�r�}]���I����}Tm��2�����3&c�r���v_��5�\3�������N���w�;��
oH�����IxM����7�|s�t�M�o,�Sq��B���;���E��=�q����<���9��M�g6����[�|�����~5�`�j����{���mj��&����y_q0��v_������}DmS��jU����qkQ�����Gm�u-�3&=�_��Q�]r��1���G�v���M��Tn��k��1���z�����S&c�2�]z�dlS.9����Z,!� 1�1�o��>�c`M�����S����E�����Y���.�l��������:�LT��M�EX.�"��
z��g���>� ��s��yTA���m�%�rk��1���x��*�)��M���1��KN�����Nk��w���F�Nw4� q��7���}�S -d~����������
�o�r�!���j��������)��=�����=-���^�mC�;�D�I��K���=c�3��jU�Sf�9��<��gL�6��S��A���L��G�����LA���m�%�r�H�������Fh����h3!���O}�S�5��1����������!�o?t�A�sk���]�����K�}�s�9��3��p�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!^��7>��������8�m{��g����F�^|xB�i�vh%����-��tM�;�D�I��K���=c�3��jU�Sf�9��<��gL�6��S��A���L��G�����LA���m�%�r�HxA���SOM���2u�bG{��G���F�<L����w�q�\��&� �L��\�-�6�����W����2k�)��Q=c2�)����
z��g���>� �L�6e
z�dlS.9�;D&� ��;�D�I��K���=c�3��jU�Sf�9��<��gL�6��S��A���L��G�����LA���m�%�r�0�LTA��[�[nm�3&=���Q9e��S^��
z�dlS.9�[��I����}TAN��m������\r*w��$����t�QG�]v�eF�s���;�D�I��K���=c�3��jU�Sf�9��<��gL�6��S��A���L��G�����LA���m�%�r�HhAb�������OO����Y��t���l���oA�x���`3	�r	����gLz�?^��
r��1���G�����\r*�6�����W����2��)���M��T�	-H�q���o~sz������j������t�Ai�������0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!^�Xo���N;��������0D�LTA��[�[nm�3&=���Q9e��S^��
z�dlS.9�[��I����}TAN��m������\r*w��$����{�-��"�r�-�[a�x���`3	�r	����gLz�?^��
r��1���G�����\r*�6�����W����2��)���M��T�	_���k�I���n�f�m�����z�1�pQ���0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!���)���\����u+���Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���"���N����[�x��\��r�LTA��[�[nm�3&=���Q9e��S^��
z�dlS.9�[��I����}TAN��m������\r*w��$n����tM;��#�8�&� �L��\�-�6�����W����2k�)��Q=c2�)����
z��g���>� �L�6e
z�dlS.9�;D����>��sQ���`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�C$� �b����~��g<��C�P�}��]�=����P/�&� �L��\�-�6�����W����2k�)��Q=c2�)����
z��g���>� �L�6e
z�dlS.9�;DB�P�\��\��~�LTA��[�[nm�3&=���Q9e��S^��
z�dlS.9�[��I����}TAN��m������\r*w��$������������jVu���;�D�I��K���=c�3��jU�Sf�9��<��gL�6��S��A���L��G�����LA���m�%�r�HhA��;�D�I��K���=c�3��jU�Sf�9��<��gL�6��S��A���L��G�����LA���m�%�r�H�	]����Kg�u������??������Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���"�$��F����\CV�`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�C�������~���O>y��o�����p
���0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!�5$`"x���`3	�r	����gLz�?^��
r��1���G�����\r*�6�����W����2��)���M��T�	-H���|��o;���\v�e�}��D�x���`3	�r	����gLz�?^��
r��1���G�����\r*�6�����W����2��)���M��T���E�O8�����{�O���������[n���s�M6��;�D�I��K���=c�3��jU�Sf�9��<��gL�6��S��A���L��G�����LA���m�%�r����!��Bx��I��kH��w��*6�`+�`��
z��g���>� ��s��yTA���m�%�rk��1���x��*�)��M���1��KN��y����s�I�o�yz�;�����'�u�Y���L��P7�&� �L��\�-�6�����W����2k�)��Q=c2�)����
z��g���>� �L�6e
z�dlS.9�;DB������w�t�UW�n���`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�C$� ��`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�C�	��&� �L��\�-�6�����W����2k�)��Q=c2�)����
z��g���>� �L�6e
z�dlS.9�;DX����`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�C�	��&� �L��\�-�6�����W����2k�)��Q=c2�)����
z��g���>� �L�6e
z�dlS.9�;DB+W�L+V����w��*6�`+�`��
z��g���>� ��s��yTA���m�%�rk��1���x��*�)��M���1��KN������w�������}������/���w��*6�`+�`��
z��g���>� ��s��yTA���m�%�rk��1���x��*�)��M���1��KN������w����{������i��%�mo{������G�7(`x���`3	�r	����gLz�?^��
r��1���G�����\r*�6�����W����2��)���M��T���5$�����������o{z���^����=��3}�_HG}t����F���0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!2��Z�[{��W�8������������;4�W\q���P�&� �L��\�-�6�����W����2k�)��Q=c2�)����
z��g���>� �L�6e
z�dlS.9�;D&� ����>���~�����3�����w�3��=�I�z���oP�{���{C-x���`3	�r	����gLz�?^��
r��1���G�����\r*�6�����W����2��)���M��T������+�����fb���n���Zk5�GqD�������}�������6���`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�C$� �E��N:)�����7!����:���H�n�is]	��?<-]�4-[�lt��w��*6�`+�`��
z��g���>� ��s��yTA���m�%�rk��1���x��*�)��M���1��KN���������7��Y��bC�m����$t�k]k��;�D�I��K���=c�3��jU�Sf�9��<��gL�6��S��A���L��G�����LA���m�%�r�HhA���oOoz����G=���w��*6�`+�`��
z��g���>� ��s��yTA���m�%�rk��1���x��*�)��M���1��KN������w��6�`���{�n���`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�C$� q�]w�}�Ci�m�m�-��;�D�I��K���=c�3��jU�Sf�9��<��gL�6��S��A���L��G�����LA���m�%�r�H���{��wz���v�i�t���~���n���`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�Cd"�^�d���>�/��w��*6�`+�`��
z��g���>� ��s��yTA���m�%�rk��1���x��*�)��M���1��KN���)�>����w��������P/�&� �L��\�-�6�����W����2k�)��Q=c2�)����
z��g���>� �L�6e
z�dlS.9�;DB-�&� �L��\�-�6�����W����2k�)��Q=c2�)����
z��g���>� �L�6e
z�dlS.9�;D&� q���:���+�L��{o��~����w��*6�`+�`��
z��g���>� ��s��yTA���m�%�rk��1���x��*�)��M���1��KN������?����Zk��f����\3������'?��t�
7��
��`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�C$� ���|'=��OL�{���l������Y��5#v�a��'�'��CM�V����w��*6�`+�`��
z��g���>� ��s��yTA���m�%�rk��1���x��*�)��M���1��KN�����N�����/���/O�����oE,]��Y������Fm�6�p�t��w6�A�x���`3	�r	����gLz�?^��
r��1���G�����\r*�6�����W����2��)���M��T�	-H�/@��[���~�m���w��*6�`+�`��
z��g���>� ��s��yTA���m�%�rk��1���x��*�)��M���1��KN�����N�����w�����o|���UW]�\GB��}�^�LTA��[�[nm�3&=���Q9e��S^��
z�dlS.9�[��I����}TAN��m������\r*w���!q��G7�����?���{���g=�Y�8��C���_��WF��Z�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!^��7v�m����>7-Y�$S�m�����$�n�LTA��[�[nm�3&=���Q9e��S^��
z�dlS.9�[��I����}TAN��m������\r*w��$ZN8������5S��0�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!Z����{�������0l�LTA��[�[nm�3&=���Q9e��S^��
z�dlS.9�[��I����}TAN��m������\r*w��$��X��M_�����'�������OaHx���`3	�r	����gLz�?^��
r��1���G�����\r*�6�����W����2��)���M��T�	-H�����g�&�l������'=�I�3��L�8q��g�U�V��	��`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�Cd��������������W�����j'.������V�LTA��[�[nm�3&=���Q9e��S^��
z�dlS.9�[��I����}TAN��m������\r*w�LlA�E���u�]��o��[o�fqb��7o�7��`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�Cd�Zx����������/|��o��o��[o�,T@�x���`3	�r	����gLz�?^��
r��1���G�����\r*�6�����W����2��)���M��T������q�Yg��;,���~�"�������F5^~���{B�x���`3	�r	����gLz�?^��
r��1���G�����\r*�6�����W����2��)���M��T�	-H����zj:����+_����L���n�q�����C�;�D�I��K���=c�3��jU�Sf�9��<��gL�6��S��A���L��G�����LA���m�%�r�HhAb��ei����5�yM�p�
�.������w��*6�`+�`��
z��g���>� ��s��yTA���m�%�rk��1���x��*�)��M���1��KN�������)��;�D�I��K���=c�3��jU�Sf�9��<��gL�6��S��A���L��G�����LA���m�%�r����!����,q���M?���0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!^����;�	'��^��W�%K�d�6�L����0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!^�8��S�_��_���Hoy�[2u�~��@�x���`3	�r	����gLz�?^��
r��1���G�����\r*�6�����W����2��)���M��T�	-H�{��i��7oN?�������3�G��z�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!Z�X�lYZ�ti:���G�<�L��}�^�LTA��[�[nm�3&=���Q9e��S^��
z�dlS.9�[��I����}TAN��m������\r*w��$���������6�,������0�M?�}t�����G��s�9��8��w�������y�!�`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�C$|
�C=4���q�������M?�}�"J����~����`tO;����>��FF�~g�q� %�LTA��[�[nm�3&=���Q9e��S^��
z�dlS.9�[��I����}TAN��m������\r*w��$~����|���O}j6�/���g�m�����b����K���'�c����Zk���_�Q����~v����G�L�{��^���W�*��%���$�&� �L��\�-�6�����W����2k�)��Q=c2�)����
z��g���>� �L�6e
z�dlS.9�;D�B�_��WO��~��_�cqp��W7[m�U�
���~���"������[�����q�e�5���~�w�����`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�Cd"7�|s:��������h�����[o�l�A���}����/��W��E��^�v�m7��6y���`3	�r	����gLz�?^��
r��1���G�����\r*�6�����W����2��)���M��T�	/Hh1b�wl&���*:���o|�K_j���9�����w�I'5�������5�����h_l��F��"���;�D�I��K���=c�3��jU�Sf�9��<��gL�6��S��A���L��G�����LA���m�%�r�HhAB�8��#�Wo������1����_������X�1p���6���7'��w���7��������M�oU	�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���"�	}@��N�w�m��b���P�;���Z�7�~���~2���+W�[o��
�LTmw�{�������`�j�{��e�Q�l�uM��n�#j���W������l��>���m{�����Z�gLz��^��j���YcNy5����3&c�r���v_�"=c�3��jU�%�L�6ej������\r*���5��&�l�$�k+��L3��-�o����zj�����O��O��=��s�9�����f\�����t�m��n����/�B�U��f���LDmS��0Q��O�`�Q�l�uM~�����G�6�m��Q������[�������>j��k��1���z�����Sf�9��<��K���m�%�r�}]���I����}Tm��2�����3&c�r���v_��UW]5�������&�7�`���F:��8���t���5��q��\3B)���/G?y�k��&���/l��E[l�E�r�-�.�}�=���I�Q�U��y�=�LDmS��0Q��=�9�=�D�6�}]��}w�+������>���J����1���z�����Sf�9��<��K���m�%�r�}]���I����}Tm��2�����3&c�r���v_�b	��Z~���1�yL3���=S��g��b@�i:������6��=
����-�7����b����vZz�s��v�}���!�`�
�Ehr.�r9anm�3&=���Q9e��S^��
z�dlS.9�[��I����}TAN��m������\r*w��$����v�)=���l���U��g��������k^��fQ��Cy������,<�����[s��v!��_�j��Oy�S������w��*6�`+�`��
z��g���>� ��s��yTA���m�%�rk��1���x��*�)��M���1��KN����D�����:�d�_{a�s���?bQ����n��fI���~���E���Z��w�}�N�T�&� �L��\�-�6�����W����2k�)��Q=c2�)����
z��g���>� �L�6e
z�dlS.9�;D&� QG}tZ{��gt���������_��_3�|Hx���`3	�r	����gLz�?^��
r��1���G�����\r*�6�����W����2��)���M��T�����.�|���6��7�i&��������kt��Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���"�	-8�z���-oy��S�"���-kN���>�4�u�`�
��$��%�rk��1���x��*�)����jU�3&c�r�����gLz�?^��
r�dlS��gL�6��S�C$� ���'?���e/{Yz��^������Y������f��Gyds1h��Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���"�	��i�-�L�{�����^�|b�������5���7��7��S7U�w��*6�`+�`��
z��g���>� ��s��yTA���m�%�rk��1���x��*�)��M���1��KN�������������i���;�D�I��K���=c�3��jU�Sf�9��<��gL�6��S��A���L��G�����LA���m�%�r�HhAB�e��#��j�t�-�<bA����Kl�As���;�D�I��K���=c�3��jU�Sf�9��<��gL�6��S��A���L��G�����LA���m�%�r�H��vXs��]v�����>����	'���W������4�7��w��*6�`+�`��
z��g���>� ��s��yTA���m�%�rk��1���x��*�)��M���1��KN���������;�����JK�,��m����kI@�x���`3	�r	����gLz�?^��
r��1���G�����\r*�6�����W����2��)���M��T�	/H�|�k_Kox�2u�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���"��U�V��.�(]~���[`�x���`3	�r	����gLz�?^��
r��1���G�����\r*�6�����W����2��)���M��T�	-H�u�]i��7N�����*�&� �L��\�-�6�����W����2k�)��Q=c2�)����
z��g���>� �L�6e
z�dlS.9�;DB�~���o�>��O4�0\�LTA��[�[nm�3&=���Q9e��S^��
z�dlS.9�[��I����}TAN��m������\r*w��$����t�!���<�)i���O'�x��Yg�����;�D�I��K���=c�3��jU�Sf�9��<��gL�6��S��A���L��G�����LA���m�%�r�HhAb��ei���i��%�����P/�&� �L��\�-�6�����W����2k�)��Q=c2�)����
z��g���>� �L�6e
z�dlS.9�;D��l�~����^��Y�}t_��Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���"�	��Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H�D�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!��L�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H�D�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!��L�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H�D�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!��L�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H�D�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!��L�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H�D�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!��L�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H�D�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!��L�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H�D�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!��L�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H�D�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!��L�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H�D�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!��L�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H�D�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!��L�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H�D�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!��L�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H�D�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!��L�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H�D�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!��L�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H�D�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!��L�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H�D�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!��L�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H�D�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!��L�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H�D�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!��L�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H�D�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!��L�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H�D�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!��L�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H�D�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!��L�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H�D�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!��L�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H�D�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!��L�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H�D�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!��L�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H�D�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!��L�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H�D�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!��L�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H�D�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!��L�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H�D�0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!��L�Ul&�V.��[��I����}TAN�5��W����1��KN��=c�3��jU�S&c�2=c2�)���",H���{�/��t���7��7�I<������;�D�I��K���=c�3��jU�Sf�9��<��gL�6��S��A���L��G�����LA���m�%�r��D�'�|rz�����,Y���:��_��W�\��0Q�fl�l��A���L��G��YcNy5�*���M��Tnm�3&=���Q9e2�)S�3&c�r���!���<9��������>��q�{\�(�l������w��*6�`+�`��
z��g���>� ��s��yTA���m�%�rk��1���x��*�)��M���1��KN�$����+�?�����>6]p��[S���������7'��w��*6�`+�`��
z��g���>� ��s��yTA���m�%�rk��1���x��*�)��M���1��KN�$��
7��^��W�]v�et�q������^;m����;m�w��*6�`+�`��
z��g���>� ��s��yTA���m�%�rk��1���x��*�)��M���1��KN�$���_���������0;��cz��������C�;�D�I��K���=c�3��jU�Sf�9��<��gL�6��S��A���L��G�����LA���m�%�r�����Oo.b������K��:�&� �L��\�-�6�����W����2k�)��Q=c2�)����
z��g���>� �L�6e
z�dlS.9�;DX���Z����{�W\Q�/������^��������������m�������v?��}���~�D��w�Y��kr��]��>��G����W�����?��[��W������Gm�u-�3&=�_��Q�]r��1���G�v���M��Tn��k��1���z�����S&c�2�]z�dlS.9��������G������<8�������}��������[o�23��{n���g|�}�k����}�;>�����������W����#=�Mr
�J��-9�}�gp�INa_���%��u�-��z����+�L�{����'�<��X�jU���>�6�l�t����n��K.�$s�1�������������X����������/�v�m�m<�@�����8���.�m`� 1������i�����_��9��������O�$}�������k�I��o��,J��R�����N�0$X��E���^;����M�'>�	#X�����L$`�� S�	,��w_���k�����U��o~��t������{���G=����������`n�OW_}u����G�<��O����R~x(��K��al31>���
���7���������8�y�	c���\y�p`A�@��vZz�+_�.������D�����<��%/IK�,Y�G�`fD������������
@P������?M�x�3���>���\r�%M�t�7r���i��e�{<�&u��\������q���g�2.��	Mn���Y�����3N����3u3J���N:���G���:������[������6���yX����7a��~s z��������E�^y���������/��/��o�-����<��z�����������}�R/��w��`f�����o�X���8��Sq���|�;�m��6����s�9M�������~y������������|0�y����W�jl���p�
9��#���m��&���X��v��>
\~��_5�����Q�p�	�
�F�Z�Ty�\N��\?�50(������o;��E/J�}�cY��Y��<��|����������H�z����7�8���rM���i���[S�����,d���?���O�
�����7=����r�c�Xz����ofB9���f�}��:sv��&���k�f�P��}��oo�����+:����z���v\3�Q�e���w��2:���������9�$f�����6��`�YX��A���~5=�)OiB��_�"0+��W���sM>�y�k>�P�>��c�&fB<�{l���t�0w�}w��������;�n���HN�3��}2P��,<�|�$��g�����(���I��g��?M$~�_�}��C9�n�i�����.>����MojN����~�9m���D;F�����-�l~�����	�����O�M�� ���D�Zk����aV����$���k_���� �q�	M~���\���	�	}�T���������1{����k��W:����:��rnn(��[om�9���.ca����o�6�d�t�E5����:��-�N��8�,���/7�t������:��b������j�_���D;F���?=��a�C|Kk���$`>�S=?����D�|����� �Z�������x�_�c���	�
]�Z������ws~e��s���Qz����d<���+����n����G�,R��&��w�"�����
�s��69�=����/�������2���/o����5����e����G1�0`A!��>9x��g��(F=�	fM>��Om�4	���`_��k�
6X}laA�B�m��E�o��?��5o�u�k�.:�h2�5�yM�K�B������gt*2
f��;�Lo}�[�v�I;����;�8>�.�6��^��f��]�R�9���� �xaA!���|}�k_k���o��iw�7z��s]��������4q�7\:�\q��:'�&�f����g���y������o�t�M�l��F�d��5��O�DF����z�5�`.4���?�����{�n�E���MF��<�uiK����B�S���$���L���Y�� ��	(��[nY������>�(�8�����^���8w`&4�n?�����]t���4���e���0ltn��w�2
�B�k����^���+��rt+���6����@<��CF�������h�iABcd}�T����Z�c���~=X�����.(A_5�b�-�c����s�7�z����^ ���:o��C���S`�������G���0��)/}�K��+�o��8�������	f�������QPB{m,]t���\hQ|<��_�[���0��L�o
w�G�6��"�`��$����#���oG����u9��aA00�;�i>�����7_���q�[o�u�7:�N�+�����7P��I9���8��s�����3��-oI���/�c�n�D��_��t�UW��	`(>��6�}����>��+��	'������o�����c�i>��}����M�EFA��$�N���m����{I�?~*9X3� ��	����}z�>���~���s����'=�I����
|����4?(�	��)��G?���O�$���<�1|�fDU������R�3�+�����]�s1SF�����e��2J������3����]��&�<��t�<_�� �E��x��^G����7�#O}�Sg�7�0�����������!��l����Y6�30�#�R�3����{�/������xF�>������ET�]�`��� ,t��Z,�������l,�-(�
DT��N&>�9�'/2�}0sro���gfh2��4���-�.AH'H�	 � �$�t��N��	@:AH'H�	 � �$�t��N��	@:AH'H�	 � �$�t��N��	@:AH'H�	 � �$�t��N��	@:AH'H�	�U^^^���1���������C5+G�~�W�}�����=e9�������Z� ����a������Y������xW��������W�}�������4���cll�:677���Vk��$�V���������������Q�(brr��?;;[�������$ � �� (lmm���s=�3<4�����;"�ymm-:�N��&`�	�UJP(�i����'}?	������u_O�a$�V���0�]���~`8	�U�����Q�n����z�wpp H�$�Vi
��!����I���k���0	�U�����ULLLT���U���~LMM	0	�U��A����������N�r^XX���mAF� ����I,..���m=����g����k��o������ �$�t��N��	@:AH'H�	 � �$�t��N��	@��_�US��IEND�B`�
compactify_tuples_dgr_v6b.patchapplication/octet-stream; name=compactify_tuples_dgr_v6b.patchDownload
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index d708117a40..637847dfd0 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -411,51 +411,201 @@ PageRestoreTempPage(Page tempPage, Page oldPage)
 }
 
 /*
- * sorting support for PageRepairFragmentation and PageIndexMultiDelete
+ * Item pruning support for PageRepairFragmentation and PageIndexMultiDelete
  */
-typedef struct itemIdSortData
+typedef struct itemIdCompactData
 {
 	uint16		offsetindex;	/* linp array index */
 	int16		itemoff;		/* page offset of item data */
 	uint16		alignedlen;		/* MAXALIGN(item data len) */
-} itemIdSortData;
-typedef itemIdSortData *itemIdSort;
-
-static int
-itemoffcompare(const void *itemidp1, const void *itemidp2)
-{
-	/* Sort in decreasing itemoff order */
-	return ((itemIdSort) itemidp2)->itemoff -
-		((itemIdSort) itemidp1)->itemoff;
-}
+} itemIdCompactData;
+typedef itemIdCompactData *itemIdCompact;
 
 /*
  * After removing or marking some line pointers unused, move the tuples to
  * remove the gaps caused by the removed items.
+ *
+ * Callers may pass 'presorted' as true if the itemidbase array is sorted in
+ * descending order of itemoff.  This allows a slightly more optimal code path
+ * to be taken.
+ *
+ * Callers must ensure that nitems is > 0
  */
 static void
-compactify_tuples(itemIdSort itemidbase, int nitems, Page page)
+compactify_tuples(itemIdCompact itemidbase, int nitems, Page page, bool presorted)
 {
 	PageHeader	phdr = (PageHeader) page;
 	Offset		upper;
 	int			i;
 
-	/* sort itemIdSortData array into decreasing itemoff order */
-	qsort((char *) itemidbase, nitems, sizeof(itemIdSortData),
-		  itemoffcompare);
+	/* Code within will not work correctly if nitems == 0 */
+	Assert(nitems > 0);
 
-	upper = phdr->pd_special;
-	for (i = 0; i < nitems; i++)
+	if (presorted)
 	{
-		itemIdSort	itemidptr = &itemidbase[i];
-		ItemId		lp;
+		Offset	copy_tail;
+		Offset	copy_head;
+		itemIdCompact	itemidptr;
+
+		/*
+		 * The order of lineitem offsets is already in the optimal order, i.e,
+		 * lower item pointers have a higher offset.  This allows us to move
+		 * the tuples up to the end of the heap without having to worry about
+		 * overwriting memory of other tuples during the move operation.
+		 */
+#ifdef USE_ASSERT_CHECKING
+		{
+			/* Check caller set the preordered flag correctly */
+			Offset lastoff = phdr->pd_special;
+
+			for (i = 0; i < nitems; i++)
+			{
+				itemidptr = &itemidbase[i];
+
+				Assert(lastoff > itemidptr->itemoff);
+
+				lastoff = itemidptr->itemoff;
+			}
+		}
+#endif /* USE_ASSERT_CHECKING */
+
+		upper = phdr->pd_special;
 
-		lp = PageGetItemId(page, itemidptr->offsetindex + 1);
-		upper -= itemidptr->alignedlen;
+		/* Skip over initial tuples that are already in the correct place */
+		i = 0;
+		do {
+			itemidptr = &itemidbase[i];
+			if (upper != itemidptr->itemoff + itemidptr->alignedlen)
+				break;
+			upper -= itemidptr->alignedlen;
+
+			i++;
+		} while (i < nitems);
+
+		/*
+		 * Do the tuple compactification.  Collapse memmove calls for adjacent
+		 * tuples.
+		 */
+		copy_tail = copy_head = itemidptr->itemoff + itemidptr->alignedlen;
+		for (; i < nitems; i++)
+		{
+			ItemId		lp;
+
+			itemidptr = &itemidbase[i];
+			lp = PageGetItemId(page, itemidptr->offsetindex + 1);
+
+			if (copy_head != itemidptr->itemoff + itemidptr->alignedlen)
+			{
+				/*
+				 * memmove is likely to have pre-checks when the destination
+				 * and source are the same address.
+				 */
+				memmove((char *) page + upper,
+						page + copy_head,
+						copy_tail - copy_head);
+				copy_tail = itemidptr->itemoff + itemidptr->alignedlen;
+			}
+			upper -= itemidptr->alignedlen;
+			copy_head = itemidptr->itemoff;
+
+			lp->lp_off = upper;
+
+		}
+
+		/* move the remaining chunk. Could be 0 bytes. */
 		memmove((char *) page + upper,
-				(char *) page + itemidptr->itemoff,
-				itemidptr->alignedlen);
-		lp->lp_off = upper;
+				page + copy_head,
+				copy_tail - copy_head);
+	}
+	else
+	{
+		Offset	copy_tail;
+		Offset	copy_head;
+		itemIdCompact	itemidptr = NULL;
+		PGAlignedBlock scratch;
+		char	   *scratchptr = scratch.data;
+
+		/*
+		 * The tuples in the itemidbase array may be in any order so in order to
+		 * move these to the end of the page we must make a temp copy of each
+		 * tuple before we copy them back into the page at the new position.
+		 *
+		 * If a large number of the tuples have been pruned (>75%) then we'll copy
+		 * these into the temp buffer tuple-by-tuple, otherwise we'll just do a
+		 * single memcpy for all tuples that we need to store in the temp buffer.
+		 */
+		if (nitems < PageGetMaxOffsetNumber(page) / 4)
+		{
+			for (i = 0; i < nitems; i++)
+			{
+				itemidptr = &itemidbase[i];
+				memcpy(scratchptr + itemidptr->itemoff, page + itemidptr->itemoff,
+					   itemidptr->alignedlen);
+			}
+
+			/* Set things up for the compactification code below */
+			i = 0;
+			itemidptr = &itemidbase[0];
+			upper = phdr->pd_special;
+		}
+		else
+		{
+			upper = phdr->pd_special;
+
+			/*
+			 * Detect which tuples at the end of the page don't need to be
+			 * moved.  It's quite common that many tuples won't need to be
+			 * touched so we don't need to copy these to our temp buffer.
+			 */
+			i = 0;
+			do {
+				itemidptr = &itemidbase[i];
+				if (upper != itemidptr->itemoff + itemidptr->alignedlen)
+					break;
+				upper -= itemidptr->alignedlen;
+
+				i++;
+			} while (i < nitems);
+
+			/* Copy all tuples that need to be moved into the temp buffer */
+			memcpy(scratchptr + phdr->pd_upper,
+				   page + phdr->pd_upper,
+				   upper - phdr->pd_upper);
+		}
+	
+		/*
+		 * Do the tuple compactification.  Depending on which path we took
+		 * above to do the backup, we may have already skipped the tuples
+		 * at the end of the page which don't need to be moved.
+		 */
+		copy_tail = copy_head = itemidptr->itemoff + itemidptr->alignedlen;
+		for (; i < nitems; i++)
+		{
+			ItemId		lp;
+
+			itemidptr = &itemidbase[i];
+			lp = PageGetItemId(page, itemidptr->offsetindex + 1);
+
+			/* copy pending tuples when we detect a gap */
+			if (copy_head != itemidptr->itemoff + itemidptr->alignedlen)
+			{
+				memcpy((char *) page + upper,
+					   scratchptr + copy_head,
+					   copy_tail - copy_head);
+				/* reset where we've copied up to */
+				copy_tail = itemidptr->itemoff + itemidptr->alignedlen;
+			}
+			upper -= itemidptr->alignedlen;
+			copy_head = itemidptr->itemoff;
+
+			lp->lp_off = upper;
+
+		}
+
+		/* Copy the remaining chunk */
+		memcpy((char *) page + upper,
+			   scratchptr + copy_head,
+			   copy_tail - copy_head);
 	}
 
 	phdr->pd_upper = upper;
@@ -477,14 +627,16 @@ PageRepairFragmentation(Page page)
 	Offset		pd_lower = ((PageHeader) page)->pd_lower;
 	Offset		pd_upper = ((PageHeader) page)->pd_upper;
 	Offset		pd_special = ((PageHeader) page)->pd_special;
-	itemIdSortData itemidbase[MaxHeapTuplesPerPage];
-	itemIdSort	itemidptr;
+	Offset		last_offset;
+	itemIdCompactData itemidbase[MaxHeapTuplesPerPage];
+	itemIdCompact	itemidptr;
 	ItemId		lp;
 	int			nline,
 				nstorage,
 				nunused;
 	int			i;
 	Size		totallen;
+	bool		presorted = true; /* For now */
 
 	/*
 	 * It's worth the trouble to be more paranoid here than in most places,
@@ -509,6 +661,7 @@ PageRepairFragmentation(Page page)
 	nline = PageGetMaxOffsetNumber(page);
 	itemidptr = itemidbase;
 	nunused = totallen = 0;
+	last_offset = pd_special;
 	for (i = FirstOffsetNumber; i <= nline; i++)
 	{
 		lp = PageGetItemId(page, i);
@@ -518,6 +671,12 @@ PageRepairFragmentation(Page page)
 			{
 				itemidptr->offsetindex = i - 1;
 				itemidptr->itemoff = ItemIdGetOffset(lp);
+
+				if (last_offset > itemidptr->itemoff)
+					last_offset = itemidptr->itemoff;
+				else
+					presorted = false;
+
 				if (unlikely(itemidptr->itemoff < (int) pd_upper ||
 							 itemidptr->itemoff >= (int) pd_special))
 					ereport(ERROR,
@@ -552,7 +711,7 @@ PageRepairFragmentation(Page page)
 					 errmsg("corrupted item lengths: total %u, available space %u",
 							(unsigned int) totallen, pd_special - pd_lower)));
 
-		compactify_tuples(itemidbase, nstorage, page);
+		compactify_tuples(itemidbase, nstorage, page, presorted);
 	}
 
 	/* Set hint bit for PageAddItem */
@@ -831,9 +990,9 @@ PageIndexMultiDelete(Page page, OffsetNumber *itemnos, int nitems)
 	Offset		pd_lower = phdr->pd_lower;
 	Offset		pd_upper = phdr->pd_upper;
 	Offset		pd_special = phdr->pd_special;
-	itemIdSortData itemidbase[MaxIndexTuplesPerPage];
+	itemIdCompactData itemidbase[MaxIndexTuplesPerPage];
 	ItemIdData	newitemids[MaxIndexTuplesPerPage];
-	itemIdSort	itemidptr;
+	itemIdCompact	itemidptr;
 	ItemId		lp;
 	int			nline,
 				nused;
@@ -932,7 +1091,7 @@ PageIndexMultiDelete(Page page, OffsetNumber *itemnos, int nitems)
 	phdr->pd_lower = SizeOfPageHeaderData + nused * sizeof(ItemIdData);
 
 	/* and compactify the tuple data */
-	compactify_tuples(itemidbase, nused, page);
+	compactify_tuples(itemidbase, nused, page, false);
 }
 
 
#15Thomas Munro
thomas.munro@gmail.com
In reply to: David Rowley (#13)
Re: Optimising compactify_tuples()

On Fri, Sep 11, 2020 at 1:45 AM David Rowley <dgrowleyml@gmail.com> wrote:

On Thu, 10 Sep 2020 at 10:40, Thomas Munro <thomas.munro@gmail.com> wrote:

I wonder if we could also identify a range at the high end that is
already correctly sorted and maximally compacted so it doesn't even
need to be copied out.

I've experimented quite a bit with this patch today. I think I've
tested every idea you've mentioned here, so there's quite a lot of
information to share.

I did write code to skip the copy to the separate buffer for tuples
that are already in the correct place and with a version of the patch
which keeps tuples in their traditional insert order (later lineitem's
tuple located earlier in the page) I see a generally a very large
number of tuples being skipped with this method. See attached
v4b_skipped_tuples.png. The vertical axis is the number of
compactify_tuple() calls during the benchmark where we were able to
skip that number of tuples. The average skipped tuples overall calls
during recovery was 81 tuples, so we get to skip about half the tuples
in the page doing this on this benchmark.

Excellent.

So one question is whether we want to do the order-reversing as part
of this patch, or wait for a more joined-up project to make lots of
code paths collude on making scan order match memory order
(corellation = 1). Most or all of the gain from your patch would
presumably still apply if did the exact opposite and forced offset
order to match reverse-item ID order (correlation = -1), which also
happens to be the initial state when you insert tuples today; you'd
still tend towards a state that allows nice big memmov/memcpy calls
during page compaction. IIUC currently we start with correlation -1
and then tend towards correlation = 0 after many random updates
because we can't change the order, so it gets scrambled over time.
I'm not sure what I think about that.

So I did lots of benchmarking with both methods and my conclusion is
that I think we should stick to the traditional INSERT order with this
patch. But we should come back and revisit that more generally one
day. The main reason that I'm put off flipping the tuple order is that
it significantly reduces the number of times we hit the preordered
case. We go to all the trouble of reversing the order only to have it
broken again when we add 1 more tuple to the page. If we keep this
the traditional way, then it's much more likely that we'll maintain
that pre-order and hit the more optimal memmove code path.

Right, that makes sense. Thanks for looking into it!

I've also attached another tiny patch that I think is pretty useful
separate from this. It basically changes:

LOG: redo done at 0/D518FFD0

into:

LOG: redo done at 0/D518FFD0 system usage: CPU: user: 58.93 s,
system: 0.74 s, elapsed: 62.31 s

+1

#16Thomas Munro
thomas.munro@gmail.com
In reply to: David Rowley (#14)
Re: Optimising compactify_tuples()

On Fri, Sep 11, 2020 at 3:53 AM David Rowley <dgrowleyml@gmail.com> wrote:

That gets my benchmark down to 60.8 seconds, so 2.2 seconds better than v4b.

. o O ( I wonder if there are opportunities to squeeze some more out
of this with __builtin_prefetch... )

I've attached v6b and an updated chart showing the results of the 10
runs I did of it.

One failure seen like this while running check word (cfbot):

#2 0x000000000091f93f in ExceptionalCondition
(conditionName=conditionName@entry=0x987284 "nitems > 0",
errorType=errorType@entry=0x97531d "FailedAssertion",
fileName=fileName@entry=0xa9df0d "bufpage.c",
lineNumber=lineNumber@entry=442) at assert.c:67

#17David Rowley
dgrowleyml@gmail.com
In reply to: Thomas Munro (#16)
2 attachment(s)
Re: Optimising compactify_tuples()

On Fri, 11 Sep 2020 at 17:48, Thomas Munro <thomas.munro@gmail.com> wrote:

On Fri, Sep 11, 2020 at 3:53 AM David Rowley <dgrowleyml@gmail.com> wrote:

That gets my benchmark down to 60.8 seconds, so 2.2 seconds better than v4b.

. o O ( I wonder if there are opportunities to squeeze some more out
of this with __builtin_prefetch... )

I'd be tempted to go down that route if we had macros already defined
for that, but it looks like we don't.

I've attached v6b and an updated chart showing the results of the 10
runs I did of it.

One failure seen like this while running check word (cfbot):

#2 0x000000000091f93f in ExceptionalCondition
(conditionName=conditionName@entry=0x987284 "nitems > 0",
errorType=errorType@entry=0x97531d "FailedAssertion",
fileName=fileName@entry=0xa9df0d "bufpage.c",
lineNumber=lineNumber@entry=442) at assert.c:67

Thanks. I neglected to check the other call site properly checked for
nitems > 0. Looks like PageIndexMultiDelete() relied on
compacify_tuples() to set pd_upper to pd_special when nitems == 0.
That's not what PageRepairFragmentation() did, so I've now aligned the
two so they work the same way.

I've attached patches in git format-patch format. I'm proposing to
commit these in about 48 hours time unless there's some sort of
objection before then.

Thanks for reviewing this.

David

Attachments:

v8-0001-Optimize-compactify_tuples-function.patchapplication/octet-stream; name=v8-0001-Optimize-compactify_tuples-function.patchDownload
From 020ee66b792e2a4398fed8ae37c0adf827e8dd19 Mon Sep 17 00:00:00 2001
From: "dgrowley@gmail.com" <dgrowley@gmail.com>
Date: Fri, 11 Sep 2020 23:06:31 +1200
Subject: [PATCH v8 1/2] Optimize compactify_tuples function

This function could often be seen in profiles of the recovery process. A
qsort was performed in order to sort the item pointers in reverse offset
order so that we could safely move tuples up to the end of the page
without overwriting the memory of yet-to-be-moved tuples. i.e. we used to
compact the page starting at the back of the page and move towards the
front.  The qsort that this required could be expensive for pages with a
large number of tuples.

In this commit, we take another approach to the tuple defragmentation.

Now, instead of sorting the remaining item pointers, we first check if the
remaining line pointers are presorted and only memmove the tuples that
need to be moved. This presorted check can be done very cheaply in the
calling functions when they're building the array of remaining line
pointers.  This presorted case is very fast.

When the item pointer array is not presorted we must copy tuples that need
to be moved into a temp buffer before copying them back into the page
again. This differs from what we used to do here as we're now copying the
tuples back into the page in reverse line pointer order. Previously we
left the existing order alone.  Reordering the tuples results in an
increased likelihood of hitting the pre-sorted case the next time around.
Any newly added tuple which consumes a new line pointer will also maintain
the correct sort order of tuples in the page which will result in the
presorted case being hit the next time.  Only consuming an unused line
pointer can cause the order of tuples to go out again, but that will be
corrected next time the function is called for the page.

Benchmarks have shown that the non-presorted case is at least equally as
fast as the original qsort method when the page just has a few tuples.
As the number of tuples becomes larger the new method maintains its
performance.  The original qsort method became much slower when the number
of tuples on the page became large.

Author: David Rowley
Reviewed-by: Thomas Munro
Discussion: https://postgr.es/m/CA+hUKGKMQFVpjr106gRhwk6R-nXv0qOcTreZuQzxgpHESAL6dw@mail.gmail.com
---
 src/backend/storage/page/bufpage.c | 279 +++++++++++++++++++++++++----
 1 file changed, 246 insertions(+), 33 deletions(-)

diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index d708117a40..6dab1c264d 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -411,51 +411,243 @@ PageRestoreTempPage(Page tempPage, Page oldPage)
 }
 
 /*
- * sorting support for PageRepairFragmentation and PageIndexMultiDelete
+ * Tuple defrag support for PageRepairFragmentation and PageIndexMultiDelete
  */
-typedef struct itemIdSortData
+typedef struct itemIdCompactData
 {
 	uint16		offsetindex;	/* linp array index */
 	int16		itemoff;		/* page offset of item data */
 	uint16		alignedlen;		/* MAXALIGN(item data len) */
-} itemIdSortData;
-typedef itemIdSortData *itemIdSort;
-
-static int
-itemoffcompare(const void *itemidp1, const void *itemidp2)
-{
-	/* Sort in decreasing itemoff order */
-	return ((itemIdSort) itemidp2)->itemoff -
-		((itemIdSort) itemidp1)->itemoff;
-}
+} itemIdCompactData;
+typedef itemIdCompactData *itemIdCompact;
 
 /*
- * After removing or marking some line pointers unused, move the tuples to
- * remove the gaps caused by the removed items.
+ * After removing or marking some line pointers unused, rearrange the tuples
+ * so that they're located in the page in reverse line pointer order and
+ * remove the gaps caused by the newly unused line pointers.
+ *
+ * This function can be fairly hot, especially during recovery, so it pays to
+ * take some measures to make it as optimal as possible.
+ *
+ * Callers may pass 'presorted' as true iif the 'itemidbase' array is sorted
+ * in descending order of itemoff.  When this is true we can just memmove
+ * tuples towards the end of the page.  This is quite a common case as it's
+ * the order that tuples are initially inserted into pages.  It's also the
+ * order that this function will reorder the tuples into, so hitting this case
+ * is still fairly common for tables that are heavily updated.
+ *
+ * When the 'itemidbase' array is not presorted then we're unable to just
+ * memmove tuples around freely.  Doing so could cause us to overwrite the
+ * memory belonging to a tuple we've not moved yet.  In this case, we copy all
+ * the tuples that need to be moved into a temporary buffer.  We can then
+ * simply memcpy() out of that temporary buffer back into the page at the
+ * correct offset.
+ *
+ * Callers must ensure that nitems is > 0
  */
 static void
-compactify_tuples(itemIdSort itemidbase, int nitems, Page page)
+compactify_tuples(itemIdCompact itemidbase, int nitems, Page page, bool presorted)
 {
 	PageHeader	phdr = (PageHeader) page;
 	Offset		upper;
+	Offset		copy_tail;
+	Offset		copy_head;
+	itemIdCompact itemidptr;
 	int			i;
 
-	/* sort itemIdSortData array into decreasing itemoff order */
-	qsort((char *) itemidbase, nitems, sizeof(itemIdSortData),
-		  itemoffcompare);
+	/* Code within will not work correctly if nitems == 0 */
+	Assert(nitems > 0);
 
-	upper = phdr->pd_special;
-	for (i = 0; i < nitems; i++)
+	if (presorted)
 	{
-		itemIdSort	itemidptr = &itemidbase[i];
-		ItemId		lp;
 
-		lp = PageGetItemId(page, itemidptr->offsetindex + 1);
-		upper -= itemidptr->alignedlen;
+#ifdef USE_ASSERT_CHECKING
+		{
+			/*
+			 * Verify we've not gotten any new callers that are incorrectly
+			 * passing a true presorted value.
+			 */
+			Offset		lastoff = phdr->pd_special;
+
+			for (i = 0; i < nitems; i++)
+			{
+				itemidptr = &itemidbase[i];
+
+				Assert(lastoff > itemidptr->itemoff);
+
+				lastoff = itemidptr->itemoff;
+			}
+		}
+#endif							/* USE_ASSERT_CHECKING */
+
+		/*
+		 * 'itemidbase' is already in the optimal order, i.e, lower item
+		 * pointers have a higher offset.  This allows us to memmove the
+		 * tuples up to the end of the page without having to worry about
+		 * overwriting other tuples that have not been moved yet.
+		 *
+		 * There's a good chance that there are tuples already right at the
+		 * end of the page that we can simply skip over because they're
+		 * already in the correct location within the page.  We'll do that
+		 * first...
+		 */
+		upper = phdr->pd_special;
+		i = 0;
+		do
+		{
+			itemidptr = &itemidbase[i];
+			if (upper != itemidptr->itemoff + itemidptr->alignedlen)
+				break;
+			upper -= itemidptr->alignedlen;
+
+			i++;
+		} while (i < nitems);
+
+		/*
+		 * Now that we've found the first tuple that needs to be moved we can
+		 * do the tuple compactification.  We try and make the least number of
+		 * memmove calls and only call memmove when there's a gap.  When we
+		 * see a gap we just move all tuples after the gap up until the point
+		 * of the last move operation.
+		 */
+		copy_tail = copy_head = itemidptr->itemoff + itemidptr->alignedlen;
+		for (; i < nitems; i++)
+		{
+			ItemId		lp;
+
+			itemidptr = &itemidbase[i];
+			lp = PageGetItemId(page, itemidptr->offsetindex + 1);
+
+			if (copy_head != itemidptr->itemoff + itemidptr->alignedlen)
+			{
+				memmove((char *) page + upper,
+						page + copy_head,
+						copy_tail - copy_head);
+
+				/*
+				 * We've now moved all tuples already seen, but not the
+				 * current tuple, so we set the copy_tail to the end of this
+				 * tuple so it can be moved in another iteration of the loop.
+				 */
+				copy_tail = itemidptr->itemoff + itemidptr->alignedlen;
+			}
+			/* shift the target offset down by the length of this tuple */
+			upper -= itemidptr->alignedlen;
+			/* point the copy_head to the start of this tuple */
+			copy_head = itemidptr->itemoff;
+
+			/* update the line pointer to reference the new offset */
+			lp->lp_off = upper;
+
+		}
+
+		/* move the remaining tuples. */
 		memmove((char *) page + upper,
-				(char *) page + itemidptr->itemoff,
-				itemidptr->alignedlen);
-		lp->lp_off = upper;
+				page + copy_head,
+				copy_tail - copy_head);
+	}
+	else
+	{
+		PGAlignedBlock scratch;
+		char	   *scratchptr = scratch.data;
+
+		/*
+		 * Non-presorted case:  The tuples in the itemidbase array may be in
+		 * any order. So, in order to move these to the end of the page we
+		 * must make a temp copy of each tuple that needs to be moved before
+		 * we copy them back into the page at the new offset.
+		 *
+		 * If a large percentage of tuples have been pruned (>75%) then we'll
+		 * copy these into the temp buffer tuple-by-tuple, otherwise, we'll
+		 * just do a single memcpy for all tuples that need to be moved.
+		 */
+		if (nitems < PageGetMaxOffsetNumber(page) / 4)
+		{
+			i = 0;
+			do
+			{
+				itemidptr = &itemidbase[i];
+				memcpy(scratchptr + itemidptr->itemoff, page + itemidptr->itemoff,
+					   itemidptr->alignedlen);
+				i++;
+			} while (i < nitems);
+
+			/* Set things up for the compactification code below */
+			i = 0;
+			itemidptr = &itemidbase[0];
+			upper = phdr->pd_special;
+		}
+		else
+		{
+			upper = phdr->pd_special;
+
+			/*
+			 * Many tuples are likely to already be in the correct location.
+			 * There's no need to copy these into the temp buffer.  Instead
+			 * we'll just skip forward in the itemidbase array to the position
+			 * that we do need to move tuples from so that the code below just
+			 * leaves these ones alone.
+			 */
+			i = 0;
+			do
+			{
+				itemidptr = &itemidbase[i];
+				if (upper != itemidptr->itemoff + itemidptr->alignedlen)
+					break;
+				upper -= itemidptr->alignedlen;
+
+				i++;
+			} while (i < nitems);
+
+			/* Copy all tuples that need to be moved into the temp buffer */
+			memcpy(scratchptr + phdr->pd_upper,
+				   page + phdr->pd_upper,
+				   upper - phdr->pd_upper);
+		}
+
+		/*
+		 * Do the tuple compactification.  itemidptr is already pointing to
+		 * the first tuple that we're going to move.  Here we collapse the
+		 * memcpy calls for adjacent tuples into a single call.  This is done
+		 * by delaying the memcpy call until we find a gap that needs to be
+		 * closed.
+		 */
+		copy_tail = copy_head = itemidptr->itemoff + itemidptr->alignedlen;
+		for (; i < nitems; i++)
+		{
+			ItemId		lp;
+
+			itemidptr = &itemidbase[i];
+			lp = PageGetItemId(page, itemidptr->offsetindex + 1);
+
+			/* copy pending tuples when we detect a gap */
+			if (copy_head != itemidptr->itemoff + itemidptr->alignedlen)
+			{
+				memcpy((char *) page + upper,
+					   scratchptr + copy_head,
+					   copy_tail - copy_head);
+
+				/*
+				 * We've now copied all tuples already seen, but not the
+				 * current tuple, so we set the copy_tail to the end of this
+				 * tuple.
+				 */
+				copy_tail = itemidptr->itemoff + itemidptr->alignedlen;
+			}
+			/* shift the target offset down by the length of this tuple */
+			upper -= itemidptr->alignedlen;
+			/* point the copy_head to the start of this tuple */
+			copy_head = itemidptr->itemoff;
+
+			/* update the line pointer to reference the new offset */
+			lp->lp_off = upper;
+
+		}
+
+		/* Copy the remaining chunk */
+		memcpy((char *) page + upper,
+			   scratchptr + copy_head,
+			   copy_tail - copy_head);
 	}
 
 	phdr->pd_upper = upper;
@@ -477,14 +669,16 @@ PageRepairFragmentation(Page page)
 	Offset		pd_lower = ((PageHeader) page)->pd_lower;
 	Offset		pd_upper = ((PageHeader) page)->pd_upper;
 	Offset		pd_special = ((PageHeader) page)->pd_special;
-	itemIdSortData itemidbase[MaxHeapTuplesPerPage];
-	itemIdSort	itemidptr;
+	Offset		last_offset;
+	itemIdCompactData itemidbase[MaxHeapTuplesPerPage];
+	itemIdCompact itemidptr;
 	ItemId		lp;
 	int			nline,
 				nstorage,
 				nunused;
 	int			i;
 	Size		totallen;
+	bool		presorted = true;	/* For now */
 
 	/*
 	 * It's worth the trouble to be more paranoid here than in most places,
@@ -509,6 +703,7 @@ PageRepairFragmentation(Page page)
 	nline = PageGetMaxOffsetNumber(page);
 	itemidptr = itemidbase;
 	nunused = totallen = 0;
+	last_offset = pd_special;
 	for (i = FirstOffsetNumber; i <= nline; i++)
 	{
 		lp = PageGetItemId(page, i);
@@ -518,6 +713,12 @@ PageRepairFragmentation(Page page)
 			{
 				itemidptr->offsetindex = i - 1;
 				itemidptr->itemoff = ItemIdGetOffset(lp);
+
+				if (last_offset > itemidptr->itemoff)
+					last_offset = itemidptr->itemoff;
+				else
+					presorted = false;
+
 				if (unlikely(itemidptr->itemoff < (int) pd_upper ||
 							 itemidptr->itemoff >= (int) pd_special))
 					ereport(ERROR,
@@ -552,7 +753,7 @@ PageRepairFragmentation(Page page)
 					 errmsg("corrupted item lengths: total %u, available space %u",
 							(unsigned int) totallen, pd_special - pd_lower)));
 
-		compactify_tuples(itemidbase, nstorage, page);
+		compactify_tuples(itemidbase, nstorage, page, presorted);
 	}
 
 	/* Set hint bit for PageAddItem */
@@ -831,9 +1032,10 @@ PageIndexMultiDelete(Page page, OffsetNumber *itemnos, int nitems)
 	Offset		pd_lower = phdr->pd_lower;
 	Offset		pd_upper = phdr->pd_upper;
 	Offset		pd_special = phdr->pd_special;
-	itemIdSortData itemidbase[MaxIndexTuplesPerPage];
+	Offset		last_offset;
+	itemIdCompactData itemidbase[MaxIndexTuplesPerPage];
 	ItemIdData	newitemids[MaxIndexTuplesPerPage];
-	itemIdSort	itemidptr;
+	itemIdCompact itemidptr;
 	ItemId		lp;
 	int			nline,
 				nused;
@@ -842,6 +1044,7 @@ PageIndexMultiDelete(Page page, OffsetNumber *itemnos, int nitems)
 	unsigned	offset;
 	int			nextitm;
 	OffsetNumber offnum;
+	bool		presorted = true;	/* For now */
 
 	Assert(nitems <= MaxIndexTuplesPerPage);
 
@@ -883,6 +1086,7 @@ PageIndexMultiDelete(Page page, OffsetNumber *itemnos, int nitems)
 	totallen = 0;
 	nused = 0;
 	nextitm = 0;
+	last_offset = pd_special;
 	for (offnum = FirstOffsetNumber; offnum <= nline; offnum = OffsetNumberNext(offnum))
 	{
 		lp = PageGetItemId(page, offnum);
@@ -906,6 +1110,12 @@ PageIndexMultiDelete(Page page, OffsetNumber *itemnos, int nitems)
 		{
 			itemidptr->offsetindex = nused; /* where it will go */
 			itemidptr->itemoff = offset;
+
+			if (last_offset > itemidptr->itemoff)
+				last_offset = itemidptr->itemoff;
+			else
+				presorted = false;
+
 			itemidptr->alignedlen = MAXALIGN(size);
 			totallen += itemidptr->alignedlen;
 			newitemids[nused] = *lp;
@@ -932,7 +1142,10 @@ PageIndexMultiDelete(Page page, OffsetNumber *itemnos, int nitems)
 	phdr->pd_lower = SizeOfPageHeaderData + nused * sizeof(ItemIdData);
 
 	/* and compactify the tuple data */
-	compactify_tuples(itemidbase, nused, page);
+	if (nused > 0)
+		compactify_tuples(itemidbase, nused, page, presorted);
+	else
+		phdr->pd_upper = pd_special;
 }
 
 
-- 
2.25.1

v8-0002-Report-resource-usage-at-the-end-of-recovery.patchapplication/octet-stream; name=v8-0002-Report-resource-usage-at-the-end-of-recovery.patchDownload
From dc1078865fc790d1306b9f4b0622fa7431a0bf43 Mon Sep 17 00:00:00 2001
From: "dgrowley@gmail.com" <dgrowley@gmail.com>
Date: Fri, 11 Sep 2020 23:27:48 +1200
Subject: [PATCH v8 2/2] Report resource usage at the end of recovery

Reporting this has been rather useful in some recent recovery speedup
work.  It also seems like something that will be useful to the average DBA
too.

Author: David Rowley
Reviewed-by: Thomas Munro
Discussion: https://postgr.es/m/CAApHDvqYVORiZxq2xPvP6_ndmmsTkvr6jSYv4UTNaFa5i1kd%3DQ%40mail.gmail.com
---
 src/backend/access/transam/xlog.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 09c01ed4ae..046c99bfec 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -76,6 +76,7 @@
 #include "utils/memutils.h"
 #include "utils/ps_status.h"
 #include "utils/relmapper.h"
+#include "utils/pg_rusage.h"
 #include "utils/snapmgr.h"
 #include "utils/timestamp.h"
 
@@ -7169,6 +7170,9 @@ StartupXLOG(void)
 		{
 			ErrorContextCallback errcallback;
 			TimestampTz xtime;
+			PGRUsage	ru0;
+
+			pg_rusage_init(&ru0);
 
 			InRedo = true;
 
@@ -7435,8 +7439,9 @@ StartupXLOG(void)
 			}
 
 			ereport(LOG,
-					(errmsg("redo done at %X/%X",
-							(uint32) (ReadRecPtr >> 32), (uint32) ReadRecPtr)));
+					(errmsg("redo done at %X/%X system usage: %s",
+							(uint32) (ReadRecPtr >> 32), (uint32) ReadRecPtr,
+							pg_rusage_show(&ru0))));
 			xtime = GetLatestXTime();
 			if (xtime)
 				ereport(LOG,
-- 
2.25.1

#18Jakub Wartak
Jakub.Wartak@tomtom.com
In reply to: David Rowley (#17)
Re: Optimising compactify_tuples()

David Rowley wrote:

I've attached patches in git format-patch format. I'm proposing to commit these in about 48 hours time unless there's some sort of objection before then.

Hi David, no objections at all, I've just got reaffirming results here, as per [1]/messages/by-id/VI1PR0701MB696023DA7815207237196DC8F6570@VI1PR0701MB6960.eurprd07.prod.outlook.com (SLRU thread but combined results with qsort testing) I've repeated crash-recovery tests here again:

TEST0a: check-world passes
TEST0b: brief check: DB after recovery returns correct data which was present only into the WAL stream - SELECT sum(c) from sometable

TEST1: workload profile test as per standard TPC-B [2]pgbench -i -s 100, pgbench -c8 -j8 -T 240, ~1.6GB DB with 2.3GB after crash in pg_wal to be replayed, with majority of records in WAL stream being Heap/HOT_UPDATE on same system with NVMe as described there.

results of master (62e221e1c01e3985d2b8e4b68c364f8486c327ab) @ 15/09/2020 as baseline:
15.487, 1.013
15.789, 1.033
15.942, 1.118

profile looks most of the similar:
17.14% postgres libc-2.17.so [.] __memmove_ssse3_back
---__memmove_ssse3_back
compactify_tuples
PageRepairFragmentation
heap2_redo
StartupXLOG
8.16% postgres postgres [.] hash_search_with_hash_value
---hash_search_with_hash_value
|--4.49%--BufTableLookup
[..]
--3.67%--smgropen

master with 2 patches by David (v8-0001-Optimize-compactify_tuples-function.patch + v8-0002-Report-resource-usage-at-the-end-of-recovery.patch):
14.236, 1.02
14.431, 1.083
14.256, 1.02

so 9-10% faster in this simple verification check. If I had pgbench running the result would be probably better. Profile is similar:

13.88% postgres libc-2.17.so [.] __memmove_ssse3_back
---__memmove_ssse3_back
--13.47%--compactify_tuples

10.61% postgres postgres [.] hash_search_with_hash_value
---hash_search_with_hash_value
|--5.31%--smgropen
[..]
--5.31%--BufTableLookup

TEST2: update-only test, just as you performed in [3]/messages/by-id/CAApHDvoKwqAzhiuxEt8jSquPJKDpH8DNUZDFUSX9P7DXrJdc3Q@mail.gmail.com , in my case: pgbench -c 16 -j 16 -T 240 -f update.sql , ~1GB DB with 4.3GB after crash in pg_wal to be replayed to trigger the hotspot, with table fillfactor=85 and update.sql (100% updates, ~40% Heap/HOT_UPDATE [N], ~40-50% [record sizes]) with slightly different amount of data.

results of master as baseline:
233.377, 0.727
233.233, 0.72
234.085, 0.729

with profile:
24.49% postgres postgres [.] pg_qsort
17.01% postgres postgres [.] PageRepairFragmentation
12.93% postgres postgres [.] itemoffcompare
(sometimes I saw also a ~13% swapfunc)

results of master with above 2 patches, 2.3x speedup:
101.6, 0.709
101.837, 0.71
102.243, 0.712

with profile (so yup the qsort is gone, hurray!):

32.65% postgres postgres [.] PageRepairFragmentation
---PageRepairFragmentation
heap2_redo
StartupXLOG
10.88% postgres postgres [.] compactify_tuples
---compactify_tuples
8.84% postgres postgres [.] hash_search_with_hash_value

BTW: this message "redo done at 0/9749FF70 system usage: CPU: user: 13.46 s, system: 0.78 s, elapsed: 14.25 s" is priceless addition :)

-J.

[1]: /messages/by-id/VI1PR0701MB696023DA7815207237196DC8F6570@VI1PR0701MB6960.eurprd07.prod.outlook.com
[2]: pgbench -i -s 100, pgbench -c8 -j8 -T 240, ~1.6GB DB with 2.3GB after crash in pg_wal to be replayed
[3]: /messages/by-id/CAApHDvoKwqAzhiuxEt8jSquPJKDpH8DNUZDFUSX9P7DXrJdc3Q@mail.gmail.com , in my case: pgbench -c 16 -j 16 -T 240 -f update.sql , ~1GB DB with 4.3GB after crash in pg_wal to be replayed

#19David Rowley
dgrowleyml@gmail.com
In reply to: Jakub Wartak (#18)
Re: Optimising compactify_tuples()

On Wed, 16 Sep 2020 at 02:10, Jakub Wartak <Jakub.Wartak@tomtom.com> wrote:

BTW: this message "redo done at 0/9749FF70 system usage: CPU: user: 13.46 s, system: 0.78 s, elapsed: 14.25 s" is priceless addition :)

Thanks a lot for the detailed benchmark results and profiles. That was
useful. I've pushed both patches now. I did a bit of a sweep of the
comments on the 0001 patch before pushing it.

I also did some further performance tests of something other than
recovery. I can also report a good performance improvement in VACUUM.
Something around the ~25% reduction mark

psql -c "drop table if exists t1;" postgres > /dev/null
psql -c "create table t1 (a int primary key, b int not null) with
(autovacuum_enabled = false, fillfactor = 85);" postgres > /dev/null
psql -c "insert into t1 select x,0 from generate_series(1,10000000)
x;" postgres > /dev/null
psql -c "drop table if exists log_wal;" postgres > /dev/null
psql -c "create table log_wal (lsn pg_lsn not null);" postgres > /dev/null
psql -c "insert into log_wal values(pg_current_wal_lsn());" postgres > /dev/null
pgbench -n -f update.sql -t 60000 -c 200 -j 200 -M prepared postgres
psql -c "select 'Used ' ||
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), lsn)) || ' of
WAL' from log_wal limit 1;" postgres
psql postgres

\timing on
VACUUM t1;

Fillfactor = 85

patched:

Time: 2917.515 ms (00:02.918)
Time: 2944.564 ms (00:02.945)
Time: 3004.136 ms (00:03.004)

master:
Time: 4050.355 ms (00:04.050)
Time: 4104.999 ms (00:04.105)
Time: 4158.285 ms (00:04.158)

Fillfactor = 100

Patched:

Time: 4245.676 ms (00:04.246)
Time: 4251.485 ms (00:04.251)
Time: 4247.802 ms (00:04.248)

Master:
Time: 5459.433 ms (00:05.459)
Time: 5917.356 ms (00:05.917)
Time: 5430.986 ms (00:05.431)

David

#20Thomas Munro
thomas.munro@gmail.com
In reply to: David Rowley (#19)
Re: Optimising compactify_tuples()

On Wed, Sep 16, 2020 at 1:30 PM David Rowley <dgrowleyml@gmail.com> wrote:

Thanks a lot for the detailed benchmark results and profiles. That was
useful. I've pushed both patches now. I did a bit of a sweep of the
comments on the 0001 patch before pushing it.

I also did some further performance tests of something other than
recovery. I can also report a good performance improvement in VACUUM.
Something around the ~25% reduction mark

Wonderful results. It must be rare for a such a localised patch to
have such a large effect on such common workloads.

In reply to: Thomas Munro (#20)
Re: Optimising compactify_tuples()

On Tue, Sep 15, 2020 at 7:01 PM Thomas Munro <thomas.munro@gmail.com> wrote:

On Wed, Sep 16, 2020 at 1:30 PM David Rowley <dgrowleyml@gmail.com> wrote:

I also did some further performance tests of something other than
recovery. I can also report a good performance improvement in VACUUM.
Something around the ~25% reduction mark

Wonderful results. It must be rare for a such a localised patch to
have such a large effect on such common workloads.

Yes, that's terrific.

--
Peter Geoghegan

#22Simon Riggs
simon@2ndquadrant.com
In reply to: David Rowley (#13)
Re: Optimising compactify_tuples()

On Thu, 10 Sep 2020 at 14:45, David Rowley <dgrowleyml@gmail.com> wrote:

I've also attached another tiny patch that I think is pretty useful
separate from this. It basically changes:

LOG: redo done at 0/D518FFD0

into:

LOG: redo done at 0/D518FFD0 system usage: CPU: user: 58.93 s,
system: 0.74 s, elapsed: 62.31 s

(I was getting sick of having to calculate the time spent from the log
timestamps.)

I really like this patch, thanks for proposing it.

Should pg_rusage_init(&ru0);
be at the start of the REDO loop, since you only use it if we take that path?

--
Simon Riggs http://www.2ndQuadrant.com/
Mission Critical Databases

#23Robert Haas
robertmhaas@gmail.com
In reply to: Simon Riggs (#22)
Re: Optimising compactify_tuples()

On Wed, Sep 16, 2020 at 2:54 PM Simon Riggs <simon@2ndquadrant.com> wrote:

I really like this patch, thanks for proposing it.

I'm pleased to be able to say that I agree completely with this
comment from Simon. :-)

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

#24Andres Freund
andres@anarazel.de
In reply to: Thomas Munro (#20)
Re: Optimising compactify_tuples()

On 2020-09-16 14:01:21 +1200, Thomas Munro wrote:

On Wed, Sep 16, 2020 at 1:30 PM David Rowley <dgrowleyml@gmail.com> wrote:

Thanks a lot for the detailed benchmark results and profiles. That was
useful. I've pushed both patches now. I did a bit of a sweep of the
comments on the 0001 patch before pushing it.

I also did some further performance tests of something other than
recovery. I can also report a good performance improvement in VACUUM.
Something around the ~25% reduction mark

Wonderful results. It must be rare for a such a localised patch to
have such a large effect on such common workloads.

Indeed!

#25David Rowley
dgrowleyml@gmail.com
In reply to: Simon Riggs (#22)
Re: Optimising compactify_tuples()

Hi Simon,

On Thu, 17 Sep 2020 at 06:54, Simon Riggs <simon@2ndquadrant.com> wrote:

Should pg_rusage_init(&ru0);
be at the start of the REDO loop, since you only use it if we take that path?

Thanks for commenting.

I may be misunderstanding your words, but as far as I see it the
pg_rusage_init() is only called if we're going to go into recovery.
The pg_rusage_init() and pg_rusage_show() seem to be in the same
scope, so I can't quite see how we could do the pg_rusage_init()
without the pg_rusage_show(). Oh wait, there's the possibility that
if recoveryTargetAction == RECOVERY_TARGET_ACTION_SHUTDOWN that we'll
exit before we report end of recovery. I'm pretty sure I'm
misunderstanding you though.

If it's easier to explain, please just post a small patch with what you mean.

David