Why is src/test/modules/committs/t/002_standby.pl flaky?

Started by Thomas Munroabout 4 years ago55 messages
#1Thomas Munro
thomas.munro@gmail.com

Hi,

There's a wait for replay that is open coded (instead of using the
wait_for_catchup() routine), and sometimes the second of two such
waits at line 51 (in master) times out after 3 minutes with "standby
never caught up". It's happening on three particular Windows boxes,
but once also happened on the AIX box "tern".

branch | animal | count
---------------+-----------+-------
HEAD | drongo | 1
HEAD | fairywren | 8
REL_10_STABLE | drongo | 3
REL_10_STABLE | fairywren | 10
REL_10_STABLE | jacana | 3
REL_11_STABLE | drongo | 1
REL_11_STABLE | fairywren | 4
REL_11_STABLE | jacana | 3
REL_12_STABLE | drongo | 2
REL_12_STABLE | fairywren | 5
REL_12_STABLE | jacana | 1
REL_12_STABLE | tern | 1
REL_13_STABLE | fairywren | 3
REL_14_STABLE | drongo | 2
REL_14_STABLE | fairywren | 6

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2021-12-30%2014:42:30
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2021-12-30%2013:13:22
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2021-12-30%2006:03:07
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2021-12-22%2011:37:37
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2021-12-22%2010:46:07
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2021-12-22%2009:03:06
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2021-12-17%2004:59:17
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2021-12-17%2003:59:51
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2021-12-16%2004:37:58
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2021-12-15%2009:57:14
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2021-12-15%2002:38:43
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2021-12-14%2020:42:15
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2021-12-14%2012:08:41
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2021-12-14%2000:35:32
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2021-12-13%2023:40:11
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2021-12-13%2022:47:25
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2021-12-09%2006:59:10
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2021-12-09%2006:04:04
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2021-12-09%2001:36:09
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2021-12-08%2019:20:35
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2021-12-08%2018:04:28
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2021-12-08%2014:12:32
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2021-12-08%2011:15:58
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2021-12-08%2004:04:22
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=jacana&dt=2021-12-03%2017:31:49
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2021-11-11%2015:58:55
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2021-10-02%2022:00:17
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2021-09-09%2005:16:43
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2021-08-24%2004:45:09
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=tern&dt=2021-07-17%2010:57:49
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=jacana&dt=2021-06-12%2016:05:32
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2021-02-07%2012:59:43
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=jacana&dt=2020-03-24%2012:49:50
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=jacana&dt=2020-02-01%2018:00:27
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=jacana&dt=2020-02-01%2017:26:27
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=jacana&dt=2020-01-30%2023:49:49
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=jacana&dt=2019-12-22%2014:19:02
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2019-12-13%2000:12:11
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2019-12-09%2006:02:05
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2019-12-06%2003:07:42
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2019-11-02%2014:41:04
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2019-10-25%2013:12:08
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2019-10-24%2013:12:41
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2019-10-23%2023:10:00
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2019-10-23%2018:00:39
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2019-10-22%2015:05:57
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2019-10-18%2013:29:49
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2019-10-16%2014:54:46
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2019-10-15%2014:21:11
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2019-10-14%2013:15:07
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2019-10-13%2014:19:41
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2019-10-12%2016:32:06
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2019-10-10%2013:12:09

#2Andrew Dunstan
andrew@dunslane.net
In reply to: Thomas Munro (#1)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

On 12/30/21 15:01, Thomas Munro wrote:

Hi,

There's a wait for replay that is open coded (instead of using the
wait_for_catchup() routine), and sometimes the second of two such
waits at line 51 (in master) times out after 3 minutes with "standby
never caught up". It's happening on three particular Windows boxes,
but once also happened on the AIX box "tern".

branch | animal | count
---------------+-----------+-------
HEAD | drongo | 1
HEAD | fairywren | 8
REL_10_STABLE | drongo | 3
REL_10_STABLE | fairywren | 10
REL_10_STABLE | jacana | 3
REL_11_STABLE | drongo | 1
REL_11_STABLE | fairywren | 4
REL_11_STABLE | jacana | 3
REL_12_STABLE | drongo | 2
REL_12_STABLE | fairywren | 5
REL_12_STABLE | jacana | 1
REL_12_STABLE | tern | 1
REL_13_STABLE | fairywren | 3
REL_14_STABLE | drongo | 2
REL_14_STABLE | fairywren | 6

FYI, drongo and fairywren are run on the same AWS/EC2 Windows Server
2019 instance. Nothing else runs on it.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

#3Tom Lane
tgl@sss.pgh.pa.us
In reply to: Andrew Dunstan (#2)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

Andrew Dunstan <andrew@dunslane.net> writes:

On 12/30/21 15:01, Thomas Munro wrote:

There's a wait for replay that is open coded (instead of using the
wait_for_catchup() routine), and sometimes the second of two such
waits at line 51 (in master) times out after 3 minutes with "standby
never caught up". It's happening on three particular Windows boxes,
but once also happened on the AIX box "tern".

FYI, drongo and fairywren are run on the same AWS/EC2 Windows Server
2019 instance. Nothing else runs on it.

I spent a little time looking into this just now. There are similar
failures in both 002_standby.pl and 003_standby_2.pl, which is
unsurprising because there are essentially-identical test sequences
in both. What I've realized is that the issue is triggered by
this sequence:

$standby->start;
...
$primary->restart;
$primary->safe_psql('postgres', 'checkpoint');
my $primary_lsn =
$primary->safe_psql('postgres', 'select pg_current_wal_lsn()');
$standby->poll_query_until('postgres',
qq{SELECT '$primary_lsn'::pg_lsn <= pg_last_wal_replay_lsn()})
or die "standby never caught up";

(the failing poll_query_until is at line 51 in 002_standby.pl, or
line 37 in 003_standby_2.pl). That is, we have forced a primary
restart since the standby first connected to the primary, and
now we have to wait for the standby to reconnect and catch up.

*These two tests seem to be the only TAP tests that do that*.
So I think there's not really anything specific to commit_ts testing
involved, it's just a dearth of primary restarts elsewhere.

Looking at the logs in the failing cases, there's no evidence
that the standby has even detected the primary's disconnection,
which explains why it hasn't attempted to reconnect. For
example, in the most recent HEAD failure,

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&amp;dt=2022-01-03%2018%3A04%3A41

the standby reports successful connection:

2022-01-03 18:58:04.920 UTC [179700:1] LOG: started streaming WAL from primary at 0/3000000 on timeline 1

(which we can also see in the primary's log), but after that
there's no log traffic at all except the test script's vain
checks of pg_last_wal_replay_lsn(). In the same animal's
immediately preceding successful run,

https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=fairywren&amp;dt=2022-01-03%2015%3A04%3A41&amp;stg=module-commit_ts-check

we see:

2022-01-03 15:59:24.186 UTC [176664:1] LOG: started streaming WAL from primary at 0/3000000 on timeline 1
2022-01-03 15:59:25.003 UTC [176664:2] LOG: replication terminated by primary server
2022-01-03 15:59:25.003 UTC [176664:3] DETAIL: End of WAL reached on timeline 1 at 0/3030CB8.
2022-01-03 15:59:25.003 UTC [176664:4] FATAL: could not send end-of-streaming message to primary: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
no COPY in progress
2022-01-03 15:59:25.005 UTC [177092:5] LOG: invalid record length at 0/3030CB8: wanted 24, got 0
...
2022-01-03 15:59:25.564 UTC [177580:1] LOG: started streaming WAL from primary at 0/3000000 on timeline 1

So for some reason, on these machines detection of walsender-initiated
connection close is unreliable ... or maybe, the walsender didn't close
the connection, but is somehow still hanging around? Don't have much idea
where to dig beyond that, but maybe someone else will. I wonder in
particular if this could be related to our recent discussions about
whether to use shutdown(2) on Windows --- could we need to do the
equivalent of 6051857fc/ed52c3707 on walsender connections?

regards, tom lane

#4Tom Lane
tgl@sss.pgh.pa.us
In reply to: Tom Lane (#3)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

I wrote:

So for some reason, on these machines detection of walsender-initiated
connection close is unreliable ... or maybe, the walsender didn't close
the connection, but is somehow still hanging around? Don't have much idea
where to dig beyond that, but maybe someone else will. I wonder in
particular if this could be related to our recent discussions about
whether to use shutdown(2) on Windows --- could we need to do the
equivalent of 6051857fc/ed52c3707 on walsender connections?

... wait a minute. After some more study of the buildfarm logs,
it was brought home to me that these failures started happening
just after 6051857fc went in:

https://buildfarm.postgresql.org/cgi-bin/show_failures.pl?max_days=90&amp;branch=&amp;member=&amp;stage=module-commit_tsCheck&amp;filter=Submit

The oldest matching failure is jacana's on 2021-12-03.
(The above sweep finds an unrelated-looking failure on 2021-11-11,
but no others before 6051857fc went in on 2021-12-02. Also, it
looks likely that ed52c3707 on 2021-12-07 made the failure more
probable, because jacana's is the only matching failure before 12-07.)

So I'm now thinking it's highly likely that those commits are
causing it somehow, but how?

regards, tom lane

#5Alexander Lakhin
exclusion@gmail.com
In reply to: Tom Lane (#4)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

Hello Tom,
09.01.2022 04:17, Tom Lane wrote:

So for some reason, on these machines detection of walsender-initiated
connection close is unreliable ... or maybe, the walsender didn't close
the connection, but is somehow still hanging around? Don't have much idea
where to dig beyond that, but maybe someone else will. I wonder in
particular if this could be related to our recent discussions about
whether to use shutdown(2) on Windows --- could we need to do the
equivalent of 6051857fc/ed52c3707 on walsender connections?

... wait a minute. After some more study of the buildfarm logs,
it was brought home to me that these failures started happening
just after 6051857fc went in:

https://buildfarm.postgresql.org/cgi-bin/show_failures.pl?max_days=90&amp;branch=&amp;member=&amp;stage=module-commit_tsCheck&amp;filter=Submit

The oldest matching failure is jacana's on 2021-12-03.
(The above sweep finds an unrelated-looking failure on 2021-11-11,
but no others before 6051857fc went in on 2021-12-02. Also, it
looks likely that ed52c3707 on 2021-12-07 made the failure more
probable, because jacana's is the only matching failure before 12-07.)

So I'm now thinking it's highly likely that those commits are
causing it somehow, but how?

I've managed to reproduce this failure too.
Removing "shutdown(MyProcPort->sock, SD_SEND);" doesn't help here, so
the culprit is exactly "closesocket(MyProcPort->sock);".
I've added `system("netstat -ano");` before die() in 002_standby.pl and see:
# Postmaster PID for node "primary" is 944
  Proto  Local Address          Foreign Address        State           PID
...
  TCP    127.0.0.1:58545        127.0.0.1:61995        FIN_WAIT_2      944
...
  TCP    127.0.0.1:61995        127.0.0.1:58545        CLOSE_WAIT      1352

(Replacing SD_SEND with SD_BOTH doesn't change the behaviour.)

Looking at the libpqwalreceiver.c:
        /* Now that we've consumed some input, try again */
        rawlen = PQgetCopyData(conn->streamConn, &conn->recvBuf, 1);
here we get -1 on the primary disconnection.
Then we get COMMAND_OK here:
        res = libpqrcv_PQgetResult(conn->streamConn);
        if (PQresultStatus(res) == PGRES_COMMAND_OK)
and finally just hang at:
            /* Verify that there are no more results. */
            res = libpqrcv_PQgetResult(conn->streamConn);
until the standby gets interrupted by the TAP test. (That call can also
return NULL and then the test completes successfully.)
Going down through the call chain, I see that at the end of it
WaitForMultipleObjects() hangs while waiting for the primary connection
socket event. So it looks like the socket, that is closed by the
primary, can get into a state unsuitable for WaitForMultipleObjects().
I tried to check the socket state with the WSAPoll() function and
discovered that it returns POLLHUP for the "problematic" socket.
The following draft addition in latch.c:
int
WaitLatchOrSocket(Latch *latch, int wakeEvents, pgsocket sock,
                  long timeout, uint32 wait_event_info)
{
    int            ret = 0;
    int            rc;
    WaitEvent    event;

#ifdef WIN32
    if (wakeEvents & WL_SOCKET_MASK) {
        WSAPOLLFD pollfd;
        pollfd.fd = sock;
        pollfd.events = POLLRDNORM | POLLWRNORM;
        pollfd.revents = 0;
        int rc = WSAPoll(&pollfd, 1, 0);
        if ((rc == 1) && (pollfd.revents & POLLHUP)) {
            elog(WARNING, "WaitLatchOrSocket: A stream-oriented
connection was either disconnected or aborted.");
            return WL_SOCKET_MASK;
        }
    }
#endif

makes the test 002_stanby.pl pass (100 of 100 iterations, while without
the fix I get failures roughly on each third). I'm not sure where to
place this check, maybe it's better to move it up to
libpqrcv_PQgetResult() to minimize it's footprint or to find less
Windows-specific approach, but I'd prefer a client-side fix anyway, as
graceful closing a socket by a server seems a legitimate action.

Best regards,
Alexander

#6Tom Lane
tgl@sss.pgh.pa.us
In reply to: Alexander Lakhin (#5)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

Alexander Lakhin <exclusion@gmail.com> writes:

09.01.2022 04:17, Tom Lane wrote:

... wait a minute. After some more study of the buildfarm logs,
it was brought home to me that these failures started happening
just after 6051857fc went in:

I've managed to reproduce this failure too.
Removing "shutdown(MyProcPort->sock, SD_SEND);" doesn't help here, so
the culprit is exactly "closesocket(MyProcPort->sock);".

Ugh. Did you try removing the closesocket and keeping shutdown?
I don't recall if we tried that combination before.

... I'm not sure where to
place this check, maybe it's better to move it up to
libpqrcv_PQgetResult() to minimize it's footprint or to find less
Windows-specific approach, but I'd prefer a client-side fix anyway, as
graceful closing a socket by a server seems a legitimate action.

What concerns me here is whether this implies that other clients
(libpq, jdbc, etc) are going to need changes as well. Maybe
libpq is okay, because we've not seen failures of the isolation
tests that use pg_cancel_backend(), but still it's worrisome.
I'm not entirely sure whether the isolationtester would notice
that a connection that should have died didn't.

regards, tom lane

#7Thomas Munro
thomas.munro@gmail.com
In reply to: Alexander Lakhin (#5)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

On Mon, Jan 10, 2022 at 12:00 AM Alexander Lakhin <exclusion@gmail.com> wrote:

Going down through the call chain, I see that at the end of it
WaitForMultipleObjects() hangs while waiting for the primary connection
socket event. So it looks like the socket, that is closed by the
primary, can get into a state unsuitable for WaitForMultipleObjects().

I wonder if FD_CLOSE is edge-triggered, and it's already told us once.
I think that's what these Python Twisted guys are saying:

https://stackoverflow.com/questions/7598936/how-can-a-disconnected-tcp-socket-be-reliably-detected-using-msgwaitformultipleo

I tried to check the socket state with the WSAPoll() function and
discovered that it returns POLLHUP for the "problematic" socket.

Good discovery. I guess if the above theory is right, there's a
memory somewhere that makes this level-triggered as expected by users
of poll().

#8Thomas Munro
thomas.munro@gmail.com
In reply to: Thomas Munro (#7)
1 attachment(s)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

On Mon, Jan 10, 2022 at 8:06 AM Thomas Munro <thomas.munro@gmail.com> wrote:

On Mon, Jan 10, 2022 at 12:00 AM Alexander Lakhin <exclusion@gmail.com> wrote:

Going down through the call chain, I see that at the end of it
WaitForMultipleObjects() hangs while waiting for the primary connection
socket event. So it looks like the socket, that is closed by the
primary, can get into a state unsuitable for WaitForMultipleObjects().

I wonder if FD_CLOSE is edge-triggered, and it's already told us once.

Can you reproduce it with this patch?

Attachments:

0001-Make-Windows-FD_CLOSE-reporting-sticky.patchtext/x-patch; charset=US-ASCII; name=0001-Make-Windows-FD_CLOSE-reporting-sticky.patchDownload
From d3cfff53911ce068d4a31fe7ac7933958936d1b0 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 10 Jan 2022 14:45:05 +1300
Subject: [PATCH] Make Windows FD_CLOSE reporting sticky.

XXX Just testing an idea...
---
 src/backend/storage/ipc/latch.c | 16 ++++++++++++++++
 src/include/storage/latch.h     |  1 +
 2 files changed, 17 insertions(+)

diff --git a/src/backend/storage/ipc/latch.c b/src/backend/storage/ipc/latch.c
index 1d893cf863..4a61b31cd5 100644
--- a/src/backend/storage/ipc/latch.c
+++ b/src/backend/storage/ipc/latch.c
@@ -899,6 +899,7 @@ AddWaitEventToSet(WaitEventSet *set, uint32 events, pgsocket fd, Latch *latch,
 	event->user_data = user_data;
 #ifdef WIN32
 	event->reset = false;
+	event->closed = false;
 #endif
 
 	if (events == WL_LATCH_SET)
@@ -1882,6 +1883,20 @@ WaitEventSetWaitBlock(WaitEventSet *set, int cur_timeout,
 				return 1;
 			}
 		}
+
+		/*
+		 * XXX: HYPOTHESIS TO TEST
+		 * Windows only reports FD_CLOSE once.  If we've seen that already,
+		 * continue to report it.
+		 */
+		if ((cur_event & WL_SOCKET_MASK) && cur_event->closed)
+		{
+			occurred_events->pos = cur_event->pos;
+			occurred_events->user_data = cur_event->user_data;
+			occurred_events->events = (cur_event->events & WL_SOCKET_MASK);
+			occurred_events->fd = cur_event->fd;
+			return 1;
+		}
 	}
 
 	/*
@@ -2002,6 +2017,7 @@ WaitEventSetWaitBlock(WaitEventSet *set, int cur_timeout,
 		{
 			/* EOF/error, so signal all caller-requested socket flags */
 			occurred_events->events |= (cur_event->events & WL_SOCKET_MASK);
+			cur_event->closed = true;
 		}
 
 		if (occurred_events->events != 0)
diff --git a/src/include/storage/latch.h b/src/include/storage/latch.h
index 44f9368c64..c24f46dc37 100644
--- a/src/include/storage/latch.h
+++ b/src/include/storage/latch.h
@@ -147,6 +147,7 @@ typedef struct WaitEvent
 	void	   *user_data;		/* pointer provided in AddWaitEventToSet */
 #ifdef WIN32
 	bool		reset;			/* Is reset of the event required? */
+	bool		closed;			/* Has FD_CLOSE event been reported? */
 #endif
 } WaitEvent;
 
-- 
2.33.1

#9Alexander Lakhin
exclusion@gmail.com
In reply to: Thomas Munro (#8)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

10.01.2022 05:00, Thomas Munro wrote:

On Mon, Jan 10, 2022 at 8:06 AM Thomas Munro <thomas.munro@gmail.com> wrote:

On Mon, Jan 10, 2022 at 12:00 AM Alexander Lakhin <exclusion@gmail.com> wrote:

Going down through the call chain, I see that at the end of it
WaitForMultipleObjects() hangs while waiting for the primary connection
socket event. So it looks like the socket, that is closed by the
primary, can get into a state unsuitable for WaitForMultipleObjects().

I wonder if FD_CLOSE is edge-triggered, and it's already told us once.

Can you reproduce it with this patch?

Unfortunately, this fix (with the correction "(cur_event &
WL_SOCKET_MASK)" -> "(cur_event->events & WL_SOCKET_MASK") doesn't work,
because we have two separate calls to libpqrcv_PQgetResult():

Then we get COMMAND_OK here:
        res = libpqrcv_PQgetResult(conn->streamConn);
        if (PQresultStatus(res) == PGRES_COMMAND_OK)
and finally just hang at:
            /* Verify that there are no more results. */
            res = libpqrcv_PQgetResult(conn->streamConn);

The libpqrcv_PQgetResult function, in turn, invokes WaitLatchOrSocket()
where WaitEvents are defined locally, and the closed flag set on the
first invocation but expected to be checked on second.

I've managed to reproduce this failure too.
Removing "shutdown(MyProcPort->sock, SD_SEND);" doesn't help here, so
the culprit is exactly "closesocket(MyProcPort->sock);".

Ugh. Did you try removing the closesocket and keeping shutdown?
I don't recall if we tried that combination before.

Even with shutdown() only I still observe WaitForMultipleObjects()
hanging (and WSAPoll() returns POLLHUP for the socket).

As to your concern regarding other clients, I suspect that this issue is
caused by libpqwalreceiver' specific call pattern and may be other
clients just don't do that. I need some more time to analyze this.

Best regards,
Alexander

#10Thomas Munro
thomas.munro@gmail.com
In reply to: Alexander Lakhin (#9)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

On Mon, Jan 10, 2022 at 8:00 PM Alexander Lakhin <exclusion@gmail.com> wrote:

The libpqrcv_PQgetResult function, in turn, invokes WaitLatchOrSocket()
where WaitEvents are defined locally, and the closed flag set on the
first invocation but expected to be checked on second.

D'oh, right. There's also a WaitLatchOrSocket call in walreceiver.c.
We'd need a long-lived WaitEventSet common across all of these sites,
which is hard here (because the socket might change under you, as
discussed in other threads that introduced long lived WaitEventSets to
other places but not here).

/me wonders if it's possible that graceful FD_CLOSE is reported only
once, but abortive/error FD_CLOSE is reported multiple times...

#11Thomas Munro
thomas.munro@gmail.com
In reply to: Thomas Munro (#10)
2 attachment(s)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

On Mon, Jan 10, 2022 at 10:20 PM Thomas Munro <thomas.munro@gmail.com> wrote:

On Mon, Jan 10, 2022 at 8:00 PM Alexander Lakhin <exclusion@gmail.com> wrote:

The libpqrcv_PQgetResult function, in turn, invokes WaitLatchOrSocket()
where WaitEvents are defined locally, and the closed flag set on the
first invocation but expected to be checked on second.

D'oh, right. There's also a WaitLatchOrSocket call in walreceiver.c.
We'd need a long-lived WaitEventSet common across all of these sites,
which is hard here (because the socket might change under you, as
discussed in other threads that introduced long lived WaitEventSets to
other places but not here).

This is super quick-and-dirty code (and doesn't handle some errors or
socket changes correctly), but does it detect the closed socket?

Attachments:

v2-0001-Make-Windows-FD_CLOSE-reporting-sticky.patchtext/x-patch; charset=US-ASCII; name=v2-0001-Make-Windows-FD_CLOSE-reporting-sticky.patchDownload
From 3590a8c9b3e8992a65dea7a9d6aecdca2f25582d Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 10 Jan 2022 14:45:05 +1300
Subject: [PATCH v2 1/2] Make Windows FD_CLOSE reporting sticky.

XXX Just testing an idea...
---
 src/backend/storage/ipc/latch.c | 16 ++++++++++++++++
 src/include/storage/latch.h     |  1 +
 2 files changed, 17 insertions(+)

diff --git a/src/backend/storage/ipc/latch.c b/src/backend/storage/ipc/latch.c
index 1d893cf863..8f3176a00f 100644
--- a/src/backend/storage/ipc/latch.c
+++ b/src/backend/storage/ipc/latch.c
@@ -899,6 +899,7 @@ AddWaitEventToSet(WaitEventSet *set, uint32 events, pgsocket fd, Latch *latch,
 	event->user_data = user_data;
 #ifdef WIN32
 	event->reset = false;
+	event->closed = false;
 #endif
 
 	if (events == WL_LATCH_SET)
@@ -1882,6 +1883,20 @@ WaitEventSetWaitBlock(WaitEventSet *set, int cur_timeout,
 				return 1;
 			}
 		}
+
+		/*
+		 * XXX: HYPOTHESIS TO TEST
+		 * Windows only reports FD_CLOSE once.  If we've seen that already,
+		 * continue to report it.
+		 */
+		if ((cur_event->events & WL_SOCKET_MASK) && cur_event->closed)
+		{
+			occurred_events->pos = cur_event->pos;
+			occurred_events->user_data = cur_event->user_data;
+			occurred_events->events = (cur_event->events & WL_SOCKET_MASK);
+			occurred_events->fd = cur_event->fd;
+			return 1;
+		}
 	}
 
 	/*
@@ -2002,6 +2017,7 @@ WaitEventSetWaitBlock(WaitEventSet *set, int cur_timeout,
 		{
 			/* EOF/error, so signal all caller-requested socket flags */
 			occurred_events->events |= (cur_event->events & WL_SOCKET_MASK);
+			cur_event->closed = true;
 		}
 
 		if (occurred_events->events != 0)
diff --git a/src/include/storage/latch.h b/src/include/storage/latch.h
index 44f9368c64..c24f46dc37 100644
--- a/src/include/storage/latch.h
+++ b/src/include/storage/latch.h
@@ -147,6 +147,7 @@ typedef struct WaitEvent
 	void	   *user_data;		/* pointer provided in AddWaitEventToSet */
 #ifdef WIN32
 	bool		reset;			/* Is reset of the event required? */
+	bool		closed;			/* Has FD_CLOSE event been reported? */
 #endif
 } WaitEvent;
 
-- 
2.33.1

v2-0002-XXX-Quick-hack-to-use-persistent-WaitEventSet-in-.patchtext/x-patch; charset=US-ASCII; name=v2-0002-XXX-Quick-hack-to-use-persistent-WaitEventSet-in-.patchDownload
From 755ee3da7934d33a706d59eb0abc0e4f1829acb0 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 10 Jan 2022 22:02:21 +1300
Subject: [PATCH v2 2/2] XXX Quick hack to use persistent WaitEventSet in
 walreceiver

XXX There are several things wrong with this, it's just experimental code.
---
 .../libpqwalreceiver/libpqwalreceiver.c       | 97 ++++++++++++-------
 src/backend/replication/walreceiver.c         | 12 +--
 src/include/replication/walreceiver.h         |  8 ++
 3 files changed, 75 insertions(+), 42 deletions(-)

diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index c08e599eef..f8dd76e2d0 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -40,6 +40,7 @@ void		_PG_init(void);
 
 struct WalReceiverConn
 {
+	WaitEventSet *wes;
 	/* Current connection to the primary, if any */
 	PGconn	   *streamConn;
 	/* Used to remember if the connection is logical or physical */
@@ -82,6 +83,7 @@ static WalRcvExecResult *libpqrcv_exec(WalReceiverConn *conn,
 									   const int nRetTypes,
 									   const Oid *retTypes);
 static void libpqrcv_disconnect(WalReceiverConn *conn);
+static int libpqrcv_wait(WalReceiverConn *conn, int timeout, int wait_event);
 
 static WalReceiverFunctionsType PQWalReceiverFunctions = {
 	libpqrcv_connect,
@@ -98,12 +100,13 @@ static WalReceiverFunctionsType PQWalReceiverFunctions = {
 	libpqrcv_create_slot,
 	libpqrcv_get_backend_pid,
 	libpqrcv_exec,
-	libpqrcv_disconnect
+	libpqrcv_disconnect,
+	libpqrcv_wait
 };
 
 /* Prototypes for private functions */
-static PGresult *libpqrcv_PQexec(PGconn *streamConn, const char *query);
-static PGresult *libpqrcv_PQgetResult(PGconn *streamConn);
+static PGresult *libpqrcv_PQexec(WalReceiverConn *conn, const char *query);
+static PGresult *libpqrcv_PQgetResult(WalReceiverConn *conn);
 static char *stringlist_to_identifierstr(PGconn *conn, List *strings);
 
 /*
@@ -182,6 +185,15 @@ libpqrcv_connect(const char *conninfo, bool logical, const char *appname,
 		return NULL;
 	}
 
+	conn->wes = CreateWaitEventSet(TopMemoryContext, 3);
+	AddWaitEventToSet(conn->wes, WL_SOCKET_WRITEABLE,
+					  PQsocket(conn->streamConn), NULL, NULL);
+	AddWaitEventToSet(conn->wes, WL_LATCH_SET, PGINVALID_SOCKET,
+					  MyLatch, NULL);
+	AddWaitEventToSet(conn->wes, WL_EXIT_ON_PM_DEATH, PGINVALID_SOCKET,
+					  NULL, NULL);
+
+
 	/*
 	 * Poll connection until we have OK or FAILED status.
 	 *
@@ -191,7 +203,7 @@ libpqrcv_connect(const char *conninfo, bool logical, const char *appname,
 	do
 	{
 		int			io_flag;
-		int			rc;
+		WaitEvent	event;
 
 		if (status == PGRES_POLLING_READING)
 			io_flag = WL_SOCKET_READABLE;
@@ -203,21 +215,20 @@ libpqrcv_connect(const char *conninfo, bool logical, const char *appname,
 		else
 			io_flag = WL_SOCKET_WRITEABLE;
 
-		rc = WaitLatchOrSocket(MyLatch,
-							   WL_EXIT_ON_PM_DEATH | WL_LATCH_SET | io_flag,
-							   PQsocket(conn->streamConn),
-							   0,
-							   WAIT_EVENT_LIBPQWALRECEIVER_CONNECT);
+		ModifyWaitEvent(conn->wes, 0, io_flag, NULL);
+		if (WaitEventSetWait(conn->wes, -1, &event, 1,
+							 WAIT_EVENT_LIBPQWALRECEIVER_CONNECT) < 1)
+			continue;
 
 		/* Interrupted? */
-		if (rc & WL_LATCH_SET)
+		if (event.events & WL_LATCH_SET)
 		{
 			ResetLatch(MyLatch);
 			ProcessWalRcvInterrupts();
 		}
 
 		/* If socket is ready, advance the libpq state machine */
-		if (rc & io_flag)
+		if (event.events & io_flag)
 			status = PQconnectPoll(conn->streamConn);
 	} while (status != PGRES_POLLING_OK && status != PGRES_POLLING_FAILED);
 
@@ -231,7 +242,7 @@ libpqrcv_connect(const char *conninfo, bool logical, const char *appname,
 	{
 		PGresult   *res;
 
-		res = libpqrcv_PQexec(conn->streamConn,
+		res = libpqrcv_PQexec(conn,
 							  ALWAYS_SECURE_SEARCH_PATH_SQL);
 		if (PQresultStatus(res) != PGRES_TUPLES_OK)
 		{
@@ -359,7 +370,7 @@ libpqrcv_identify_system(WalReceiverConn *conn, TimeLineID *primary_tli)
 	 * Get the system identifier and timeline ID as a DataRow message from the
 	 * primary server.
 	 */
-	res = libpqrcv_PQexec(conn->streamConn, "IDENTIFY_SYSTEM");
+	res = libpqrcv_PQexec(conn, "IDENTIFY_SYSTEM");
 	if (PQresultStatus(res) != PGRES_TUPLES_OK)
 	{
 		PQclear(res);
@@ -482,7 +493,7 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
 						 options->proto.physical.startpointTLI);
 
 	/* Start streaming. */
-	res = libpqrcv_PQexec(conn->streamConn, cmd.data);
+	res = libpqrcv_PQexec(conn, cmd.data);
 	pfree(cmd.data);
 
 	if (PQresultStatus(res) == PGRES_COMMAND_OK)
@@ -532,7 +543,7 @@ libpqrcv_endstreaming(WalReceiverConn *conn, TimeLineID *next_tli)
 	 * If we had not yet received CopyDone from the backend, PGRES_COPY_OUT is
 	 * also possible in case we aborted the copy in mid-stream.
 	 */
-	res = libpqrcv_PQgetResult(conn->streamConn);
+	res = libpqrcv_PQgetResult(conn);
 	if (PQresultStatus(res) == PGRES_TUPLES_OK)
 	{
 		/*
@@ -547,7 +558,7 @@ libpqrcv_endstreaming(WalReceiverConn *conn, TimeLineID *next_tli)
 		PQclear(res);
 
 		/* the result set should be followed by CommandComplete */
-		res = libpqrcv_PQgetResult(conn->streamConn);
+		res = libpqrcv_PQgetResult(conn);
 	}
 	else if (PQresultStatus(res) == PGRES_COPY_OUT)
 	{
@@ -561,7 +572,7 @@ libpqrcv_endstreaming(WalReceiverConn *conn, TimeLineID *next_tli)
 							pchomp(PQerrorMessage(conn->streamConn)))));
 
 		/* CommandComplete should follow */
-		res = libpqrcv_PQgetResult(conn->streamConn);
+		res = libpqrcv_PQgetResult(conn);
 	}
 
 	if (PQresultStatus(res) != PGRES_COMMAND_OK)
@@ -572,7 +583,7 @@ libpqrcv_endstreaming(WalReceiverConn *conn, TimeLineID *next_tli)
 	PQclear(res);
 
 	/* Verify that there are no more results */
-	res = libpqrcv_PQgetResult(conn->streamConn);
+	res = libpqrcv_PQgetResult(conn);
 	if (res != NULL)
 		ereport(ERROR,
 				(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -597,7 +608,7 @@ libpqrcv_readtimelinehistoryfile(WalReceiverConn *conn,
 	 * Request the primary to send over the history file for given timeline.
 	 */
 	snprintf(cmd, sizeof(cmd), "TIMELINE_HISTORY %u", tli);
-	res = libpqrcv_PQexec(conn->streamConn, cmd);
+	res = libpqrcv_PQexec(conn, cmd);
 	if (PQresultStatus(res) != PGRES_TUPLES_OK)
 	{
 		PQclear(res);
@@ -641,8 +652,9 @@ libpqrcv_readtimelinehistoryfile(WalReceiverConn *conn,
  * May return NULL, rather than an error result, on failure.
  */
 static PGresult *
-libpqrcv_PQexec(PGconn *streamConn, const char *query)
+libpqrcv_PQexec(WalReceiverConn *conn, const char *query)
 {
+	PGconn *streamConn = conn->streamConn;
 	PGresult   *lastResult = NULL;
 
 	/*
@@ -665,7 +677,7 @@ libpqrcv_PQexec(PGconn *streamConn, const char *query)
 		/* Wait for, and collect, the next PGresult. */
 		PGresult   *result;
 
-		result = libpqrcv_PQgetResult(streamConn);
+		result = libpqrcv_PQgetResult(conn);
 		if (result == NULL)
 			break;				/* query is complete, or failure */
 
@@ -690,30 +702,31 @@ libpqrcv_PQexec(PGconn *streamConn, const char *query)
  * Perform the equivalent of PQgetResult(), but watch for interrupts.
  */
 static PGresult *
-libpqrcv_PQgetResult(PGconn *streamConn)
+libpqrcv_PQgetResult(WalReceiverConn *conn)
 {
+	PGconn *streamConn = conn->streamConn;
+
 	/*
 	 * Collect data until PQgetResult is ready to get the result without
 	 * blocking.
 	 */
 	while (PQisBusy(streamConn))
 	{
-		int			rc;
+		WaitEvent	event;
 
 		/*
 		 * We don't need to break down the sleep into smaller increments,
 		 * since we'll get interrupted by signals and can handle any
 		 * interrupts here.
 		 */
-		rc = WaitLatchOrSocket(MyLatch,
-							   WL_EXIT_ON_PM_DEATH | WL_SOCKET_READABLE |
-							   WL_LATCH_SET,
-							   PQsocket(streamConn),
-							   0,
-							   WAIT_EVENT_LIBPQWALRECEIVER_RECEIVE);
+		ModifyWaitEvent(conn->wes, 0, WL_SOCKET_READABLE, NULL);
+
+		if (WaitEventSetWait(conn->wes, -1, &event, 1,
+							 WAIT_EVENT_LIBPQWALRECEIVER_RECEIVE) < 1)
+			continue;
 
 		/* Interrupted? */
-		if (rc & WL_LATCH_SET)
+		if (event.events & WL_LATCH_SET)
 		{
 			ResetLatch(MyLatch);
 			ProcessWalRcvInterrupts();
@@ -737,12 +750,28 @@ libpqrcv_PQgetResult(PGconn *streamConn)
 static void
 libpqrcv_disconnect(WalReceiverConn *conn)
 {
+	FreeWaitEventSet(conn->wes);
 	PQfinish(conn->streamConn);
 	if (conn->recvBuf != NULL)
 		PQfreemem(conn->recvBuf);
 	pfree(conn);
 }
 
+/*
+ * Wait for data, latch, or timeout.
+ */
+static int
+libpqrcv_wait(WalReceiverConn *conn, int timeout, int wait_event)
+{
+	WaitEvent event;
+
+	if (WaitEventSetWait(conn->wes, timeout, &event, 1, wait_event) == 1)
+		return event.events;
+
+	return 0;
+}
+
+
 /*
  * Receive a message available from XLOG stream.
  *
@@ -793,13 +822,13 @@ libpqrcv_receive(WalReceiverConn *conn, char **buffer,
 	{
 		PGresult   *res;
 
-		res = libpqrcv_PQgetResult(conn->streamConn);
+		res = libpqrcv_PQgetResult(conn);
 		if (PQresultStatus(res) == PGRES_COMMAND_OK)
 		{
 			PQclear(res);
 
 			/* Verify that there are no more results. */
-			res = libpqrcv_PQgetResult(conn->streamConn);
+			res = libpqrcv_PQgetResult(conn);
 			if (res != NULL)
 			{
 				PQclear(res);
@@ -941,7 +970,7 @@ libpqrcv_create_slot(WalReceiverConn *conn, const char *slotname,
 			appendStringInfoString(&cmd, " PHYSICAL RESERVE_WAL");
 	}
 
-	res = libpqrcv_PQexec(conn->streamConn, cmd.data);
+	res = libpqrcv_PQexec(conn, cmd.data);
 	pfree(cmd.data);
 
 	if (PQresultStatus(res) != PGRES_TUPLES_OK)
@@ -1068,7 +1097,7 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
 				(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
 				 errmsg("the query interface requires a database connection")));
 
-	pgres = libpqrcv_PQexec(conn->streamConn, query);
+	pgres = libpqrcv_PQexec(conn, query);
 
 	switch (PQresultStatus(pgres))
 	{
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 7a7eb3784e..f308ea9c24 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -507,13 +507,9 @@ WalReceiverMain(void)
 				 * could add and remove just the socket each time, potentially
 				 * avoiding some system calls.
 				 */
-				Assert(wait_fd != PGINVALID_SOCKET);
-				rc = WaitLatchOrSocket(MyLatch,
-									   WL_EXIT_ON_PM_DEATH | WL_SOCKET_READABLE |
-									   WL_TIMEOUT | WL_LATCH_SET,
-									   wait_fd,
-									   NAPTIME_PER_CYCLE,
-									   WAIT_EVENT_WAL_RECEIVER_MAIN);
+				//Assert(wait_fd != PGINVALID_SOCKET);
+				rc = walrcv_wait(wrconn, NAPTIME_PER_CYCLE,
+								 WAIT_EVENT_WAL_RECEIVER_MAIN);
 				if (rc & WL_LATCH_SET)
 				{
 					ResetLatch(MyLatch);
@@ -532,7 +528,7 @@ WalReceiverMain(void)
 						XLogWalRcvSendReply(true, false);
 					}
 				}
-				if (rc & WL_TIMEOUT)
+				if (rc == 0)
 				{
 					/*
 					 * We didn't receive anything new. If we haven't heard
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 0b607ed777..3e80ff63ce 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -380,6 +380,11 @@ typedef WalRcvExecResult *(*walrcv_exec_fn) (WalReceiverConn *conn,
  */
 typedef void (*walrcv_disconnect_fn) (WalReceiverConn *conn);
 
+/*
+ * XXX Wait for readable data, timeout, or latch.
+ */
+typedef int (*walrcv_wait_fn) (WalReceiverConn *conn, int timeout, int wait_event);
+
 typedef struct WalReceiverFunctionsType
 {
 	walrcv_connect_fn walrcv_connect;
@@ -397,6 +402,7 @@ typedef struct WalReceiverFunctionsType
 	walrcv_get_backend_pid_fn walrcv_get_backend_pid;
 	walrcv_exec_fn walrcv_exec;
 	walrcv_disconnect_fn walrcv_disconnect;
+	walrcv_wait_fn walrcv_wait;
 } WalReceiverFunctionsType;
 
 extern PGDLLIMPORT WalReceiverFunctionsType *WalReceiverFunctions;
@@ -431,6 +437,8 @@ extern PGDLLIMPORT WalReceiverFunctionsType *WalReceiverFunctions;
 	WalReceiverFunctions->walrcv_exec(conn, exec, nRetTypes, retTypes)
 #define walrcv_disconnect(conn) \
 	WalReceiverFunctions->walrcv_disconnect(conn)
+#define walrcv_wait(conn, timeout, wait_event) \
+	WalReceiverFunctions->walrcv_wait(conn, timeout, wait_event)
 
 static inline void
 walrcv_clear_result(WalRcvExecResult *walres)
-- 
2.33.1

#12Alexander Lakhin
exclusion@gmail.com
In reply to: Thomas Munro (#11)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

10.01.2022 12:40, Thomas Munro wrote:

This is super quick-and-dirty code (and doesn't handle some errors or
socket changes correctly), but does it detect the closed socket?

Yes, it fixes the behaviour and makes the 002_standby test pass (100 of
100 iterations). I'm yet to find out whether the other
WaitLatchOrSocket' users (e. g. postgres_fdw) can suffer from the
disconnected socket state, but this approach definitely works for
walreceiver.

Best regards,
Alexander

#13Thomas Munro
thomas.munro@gmail.com
In reply to: Alexander Lakhin (#12)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

On Tue, Jan 11, 2022 at 6:00 AM Alexander Lakhin <exclusion@gmail.com> wrote:

10.01.2022 12:40, Thomas Munro wrote:

This is super quick-and-dirty code (and doesn't handle some errors or
socket changes correctly), but does it detect the closed socket?

Yes, it fixes the behaviour and makes the 002_standby test pass (100 of
100 iterations).

Thanks for testing. That result does seem to confirm the hypothesis
that FD_CLOSE is reported only once for the socket on graceful
shutdown (that is, it's edge-triggered and incidentally you won't get
FD_READ), so you need to keep track of it carefully. Incidentally,
another observation is that your WSAPoll() test appears to be
returning POLLHUP where at least Linux, FreeBSD and Solaris would not:
a socket that is only half shut down (the primary shut down its end
gracefully, but walreceiver did not), so I suspect Windows' POLLHUP
might have POLLRDHUP semantics.

I'm yet to find out whether the other
WaitLatchOrSocket' users (e. g. postgres_fdw) can suffer from the
disconnected socket state, but this approach definitely works for
walreceiver.

I see where you're going: there might be safe call sequences and
unsafe call sequences, and maybe walreceiver is asking for trouble by
double-polling. I'm not sure about that; I got the impression
recently that it's possible to get FD_CLOSE while you still have
buffered data to read, so then the next recv() will return > 0 and
then we don't have any state left anywhere to remember that we saw
FD_CLOSE, even if you're careful to poll and read in the ideal
sequence. I could be wrong, and it would be nice if there is an easy
fix along those lines... The documentation around FD_CLOSE is
unclear.

I do plan to make a higher quality patch like the one I showed
(material from earlier unfinished work[1]/messages/by-id/CA+hUKGJPaygh-6WHEd0FnH89GrkTpVyN_ew9ckv3+nwjmLcSeg@mail.gmail.com that needs a bit more
infrastructure), but to me that's new feature/efficiency work, not
something we'd want to back-patch.

Hmm, one thing I'm still unclear on: did this problem really start
with 6051857fc/ed52c3707? My initial email in this thread lists
similar failures going back further, doesn't it? (And what's tern
doing mixed up in this mess?)

[1]: /messages/by-id/CA+hUKGJPaygh-6WHEd0FnH89GrkTpVyN_ew9ckv3+nwjmLcSeg@mail.gmail.com

#14Andrew Dunstan
andrew@dunslane.net
In reply to: Thomas Munro (#13)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

On 1/10/22 15:52, Thomas Munro wrote:

Hmm, one thing I'm still unclear on: did this problem really start
with 6051857fc/ed52c3707? My initial email in this thread lists
similar failures going back further, doesn't it? (And what's tern
doing mixed up in this mess?)

Your list contains at least some false positives. e.g.
<https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=jacana&amp;dt=2019-12-22%2014:19:02&gt;
which has a different script failing.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

#15Tom Lane
tgl@sss.pgh.pa.us
In reply to: Thomas Munro (#13)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

Thomas Munro <thomas.munro@gmail.com> writes:

Hmm, one thing I'm still unclear on: did this problem really start
with 6051857fc/ed52c3707? My initial email in this thread lists
similar failures going back further, doesn't it? (And what's tern
doing mixed up in this mess?)

Well, those earlier ones may be committs failures, but a lot of
them contain different-looking symptoms, eg pg_ctl failures.

tern's failure at
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=tern&amp;dt=2021-07-17+10%3A57%3A49
does look similar, but we can see in its log that the standby
*did* notice the primary disconnection and then reconnect:

2021-07-17 16:29:08.248 UTC [17498380:2] LOG: replication terminated by primary server
2021-07-17 16:29:08.248 UTC [17498380:3] DETAIL: End of WAL reached on timeline 1 at 0/30378F8.
2021-07-17 16:29:08.248 UTC [17498380:4] FATAL: could not send end-of-streaming message to primary: no COPY in progress
2021-07-17 16:29:08.248 UTC [25166230:5] LOG: invalid record length at 0/30378F8: wanted 24, got 0
2021-07-17 16:29:08.350 UTC [16318578:1] FATAL: could not connect to the primary server: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2021-07-17 16:29:36.369 UTC [7077918:1] FATAL: could not connect to the primary server: FATAL: the database system is starting up
2021-07-17 16:29:36.380 UTC [11338028:1] FATAL: could not connect to the primary server: FATAL: the database system is starting up
...
2021-07-17 16:29:36.881 UTC [17367092:1] LOG: started streaming WAL from primary at 0/3000000 on timeline 1

So I'm not sure what happened there, but it's not an instance
of this problem. One thing that looks a bit suspicious is
this in the primary's log:

2021-07-17 16:26:47.832 UTC [12386550:1] LOG: using stale statistics instead of current ones because stats collector is not responding

which makes me wonder if the timeout is down to out-of-date
pg_stats data. The loop in 002_standby.pl doesn't appear to
depend on the stats collector:

my $primary_lsn =
$primary->safe_psql('postgres', 'select pg_current_wal_lsn()');
$standby->poll_query_until('postgres',
qq{SELECT '$primary_lsn'::pg_lsn <= pg_last_wal_replay_lsn()})
or die "standby never caught up";

but maybe I'm missing the connection.

Apropos of that, it's worth noting that wait_for_catchup *is*
dependent on up-to-date stats, and here's a recent run where
it sure looks like the timeout cause is AWOL stats collector:

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=sungazer&amp;dt=2022-01-10%2004%3A51%3A34

I wonder if we should refactor wait_for_catchup to probe the
standby directly instead of relying on the upstream's view.

regards, tom lane

#16Alexander Lakhin
exclusion@gmail.com
In reply to: Thomas Munro (#13)
1 attachment(s)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

10.01.2022 23:52, Thomas Munro wrote:

I'm yet to find out whether the other
WaitLatchOrSocket' users (e. g. postgres_fdw) can suffer from the
disconnected socket state, but this approach definitely works for
walreceiver.

I see where you're going: there might be safe call sequences and
unsafe call sequences, and maybe walreceiver is asking for trouble by
double-polling. I'm not sure about that; I got the impression
recently that it's possible to get FD_CLOSE while you still have
buffered data to read, so then the next recv() will return > 0 and
then we don't have any state left anywhere to remember that we saw
FD_CLOSE, even if you're careful to poll and read in the ideal
sequence. I could be wrong, and it would be nice if there is an easy
fix along those lines... The documentation around FD_CLOSE is
unclear.

I had no strong opinion regarding unsafe sequence, though initially I
suspected that exactly the second libpqrcv_PQgetResult call could cause
the issue. But after digging into WaitLatchOrSocket I'd inclined to put
the fix deeper to satisfy all possible callers.
At the other hand, I've shared Tom's concerns regarding other clients,
that can stuck on WaitForMultipleObjects() just as walreceiver does, and
hoped that only walreceiver suffer from a graceful server socket closing.
So to get these doubts cleared, I've made a simple test for postgres_fdw
(please look at the attachment; you can put it into
contrib/postgres_fdw/t and run `vcregress taptest contrib\postgres_fdw`).
This test shows for me:
===
...
t/001_disconnection.pl .. # 12:13:39.481084 executing query...
# 12:13:43.245277 result:       0
# 0|0

# 12:13:43.246342 executing query...
# 12:13:46.525924 result:       0
# 0|0

# 12:13:46.527097 executing query...
# 12:13:47.745176 result:       3
#
# psql:<stdin>:1: WARNING:  no connection to the server
# psql:<stdin>:1: ERROR:  FATAL:  terminating connection due to
administrator co
mmand
# server closed the connection unexpectedly
#       This probably means the server terminated abnormally
#       before or while processing the request.
# CONTEXT:  remote SQL command: FETCH 100 FROM c1
# 12:13:47.794612 executing query...
# 12:13:51.073318 result:       0
# 0|0

# 12:13:51.074347 executing query...
===

With the simple logging added to connection.c:
                /* Sleep until there's something to do */
elog(LOG, "pgfdw_get_result before WaitLatchOrSocket");
                wc = WaitLatchOrSocket(MyLatch,
                                       WL_LATCH_SET | WL_SOCKET_READABLE |
                                       WL_EXIT_ON_PM_DEATH,
                                       PQsocket(conn),
                                       -1L, PG_WAIT_EXTENSION);
elog(LOG, "pgfdw_get_result after WaitLatchOrSocket");

I see in 001_disconnection_local.log:
...
2022-01-11 15:13:52.875 MSK|Administrator|postgres|61dd747f.5e4|LOG: 
pgfdw_get_result after WaitLatchOrSocket
2022-01-11 15:13:52.875
MSK|Administrator|postgres|61dd747f.5e4|STATEMENT:  SELECT * FROM large
WHERE a = fx2(a)
2022-01-11 15:13:52.875 MSK|Administrator|postgres|61dd747f.5e4|LOG: 
pgfdw_get_result before WaitLatchOrSocket
2022-01-11 15:13:52.875
MSK|Administrator|postgres|61dd747f.5e4|STATEMENT:  SELECT * FROM large
WHERE a = fx2(a)
2022-01-11 15:14:36.976 MSK|||61dd74ac.840|DEBUG:  autovacuum:
processing database "postgres"
2022-01-11 15:14:51.088 MSK|Administrator|postgres|61dd747f.5e4|LOG: 
pgfdw_get_result after WaitLatchOrSocket
2022-01-11 15:14:51.088
MSK|Administrator|postgres|61dd747f.5e4|STATEMENT:  SELECT * FROM large
WHERE a = fx2(a)
2022-01-11 15:14:51.089 MSK|Administrator|postgres|61dd747f.5e4|LOG: 
pgfdw_get_result before WaitLatchOrSocket
2022-01-11 15:14:51.089
MSK|Administrator|postgres|61dd747f.5e4|STATEMENT:  SELECT * FROM large
WHERE a = fx2(a)
2022-01-11 15:15:37.006 MSK|||61dd74e9.9e8|DEBUG:  autovacuum:
processing database "postgres"
2022-01-11 15:16:37.116 MSK|||61dd7525.ad0|DEBUG:  autovacuum:
processing database "postgres"
2022-01-11 15:17:37.225 MSK|||61dd7561.6a0|DEBUG:  autovacuum:
processing database "postgres"
2022-01-11 15:18:36.916 MSK|||61dd7470.704|LOG:  checkpoint starting: time
...
2022-01-11 15:36:38.225 MSK|||61dd79d6.2a0|DEBUG:  autovacuum:
processing database "postgres"
...

So here we get similar hanging on WaitLatchOrSocket().
Just to make sure that it's indeed the same issue, I've removed socket
shutdown&close and the test executed to the end (several times). Argh.

Best regards,
Alexander

Attachments:

001_disconnection.plapplication/x-perl; name=001_disconnection.plDownload
#17Thomas Munro
thomas.munro@gmail.com
In reply to: Alexander Lakhin (#16)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

On Wed, Jan 12, 2022 at 4:00 AM Alexander Lakhin <exclusion@gmail.com> wrote:

So here we get similar hanging on WaitLatchOrSocket().
Just to make sure that it's indeed the same issue, I've removed socket
shutdown&close and the test executed to the end (several times). Argh.

Ouch. I think our options at this point are:
1. Revert 6051857fc (and put it back when we have a working
long-lived WES as I showed). This is not very satisfying, now that we
understand the bug, because even without that change I guess you must
be able to reach the hanging condition by using Windows postgres_fdw
to talk to a non-Windows server (ie a normal TCP stack with graceful
shutdown/linger on process exit).
2. Put your poll() check into the READABLE side. There's some
precedent for that sort of kludge on the WRITEABLE side (and a
rejection of the fragile idea that clients of latch.c should only
perform "safe" sequences):

/*
* Windows does not guarantee to log an FD_WRITE network event
* indicating that more data can be sent unless the previous send()
* failed with WSAEWOULDBLOCK. While our caller might well have made
* such a call, we cannot assume that here. Therefore, if waiting for
* write-ready, force the issue by doing a dummy send(). If the dummy
* send() succeeds, assume that the socket is in fact write-ready, and
* return immediately. Also, if it fails with something other than
* WSAEWOULDBLOCK, return a write-ready indication to let our caller
* deal with the error condition.
*/

#18Tom Lane
tgl@sss.pgh.pa.us
In reply to: Thomas Munro (#17)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

Thomas Munro <thomas.munro@gmail.com> writes:

Ouch. I think our options at this point are:
1. Revert 6051857fc (and put it back when we have a working
long-lived WES as I showed). This is not very satisfying, now that we
understand the bug, because even without that change I guess you must
be able to reach the hanging condition by using Windows postgres_fdw
to talk to a non-Windows server (ie a normal TCP stack with graceful
shutdown/linger on process exit).

It'd be worth checking, perhaps. One thing I've been wondering all
along is how much of this behavior is specific to the local-loopback
case where Windows can see both ends of the connection. You'd think
that they couldn't long get away with such blatant violations of the
TCP specs when talking to external servers, because the failures
would be visible to everyone with a web browser.

regards, tom lane

#19Alexander Lakhin
exclusion@gmail.com
In reply to: Tom Lane (#18)
3 attachment(s)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

11.01.2022 23:16, Tom Lane wrote:

Thomas Munro <thomas.munro@gmail.com> writes:

Ouch. I think our options at this point are:
1. Revert 6051857fc (and put it back when we have a working
long-lived WES as I showed). This is not very satisfying, now that we
understand the bug, because even without that change I guess you must
be able to reach the hanging condition by using Windows postgres_fdw
to talk to a non-Windows server (ie a normal TCP stack with graceful
shutdown/linger on process exit).

It'd be worth checking, perhaps. One thing I've been wondering all
along is how much of this behavior is specific to the local-loopback
case where Windows can see both ends of the connection. You'd think
that they couldn't long get away with such blatant violations of the
TCP specs when talking to external servers, because the failures
would be visible to everyone with a web browser.

I've split my test (both parts attached) and run it on two virtual
machines with clean builds from master (ac7c8075) on both (just the
debugging output added to connection.c). I provide probably redundant
info (also see attached screenshot) just to make sure that I didn't make
a mistake.
The excerpt from 001_disconnection1_local.log:
...
2022-01-12 09:29:48.099 MSK|Administrator|postgres|61de755a.a54|LOG: 
pgfdw_get_result: before WaitLatchOrSocket
2022-01-12 09:29:48.099
MSK|Administrator|postgres|61de755a.a54|STATEMENT:  SELECT * FROM large
WHERE a = fx2(a)
2022-01-12 09:29:48.100 MSK|Administrator|postgres|61de755a.a54|LOG: 
pgfdw_get_result: after WaitLatchOrSocket
2022-01-12 09:29:48.100
MSK|Administrator|postgres|61de755a.a54|STATEMENT:  SELECT * FROM large
WHERE a = fx2(a)
2022-01-12 09:29:48.100 MSK|Administrator|postgres|61de755a.a54|LOG: 
pgfdw_get_result: before WaitLatchOrSocket
2022-01-12 09:29:48.100
MSK|Administrator|postgres|61de755a.a54|STATEMENT:  SELECT * FROM large
WHERE a = fx2(a)
2022-01-12 09:29:48.100 MSK|Administrator|postgres|61de755a.a54|LOG: 
pgfdw_get_result: after WaitLatchOrSocket
2022-01-12 09:29:48.100
MSK|Administrator|postgres|61de755a.a54|STATEMENT:  SELECT * FROM large
WHERE a = fx2(a)
2022-01-12 09:29:48.100 MSK|Administrator|postgres|61de755a.a54|ERROR: 
FATAL:  terminating connection due to administrator command
    server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
2022-01-12 09:29:48.100
MSK|Administrator|postgres|61de755a.a54|CONTEXT:  remote SQL command:
FETCH 100 FROM c1
2022-01-12 09:29:48.100
MSK|Administrator|postgres|61de755a.a54|WARNING:  no connection to the
server
2022-01-12 09:29:48.100
MSK|Administrator|postgres|61de755a.a54|CONTEXT:  remote SQL command:
ABORT TRANSACTION
2022-01-12 09:29:48.107 MSK|Administrator|postgres|61de755a.a54|LOG: 
disconnection: session time: 0:00:01.577 user=Administrator
database=postgres host=127.0.0.1 port=49752
2022-01-12 09:29:48.257 MSK|[unknown]|[unknown]|61de755c.a4c|LOG: 
connection received: host=127.0.0.1 port=49754
2022-01-12 09:29:48.261 MSK|Administrator|postgres|61de755c.a4c|LOG: 
connection authenticated: identity="WIN-FCPSOVMM1JC\Administrator"
method=sspi
(C:/src/postgrespro/contrib/postgres_fdw/tmp_check/t_001_disconnection1_local_data/pgdata/pg_hba.conf:2)
2022-01-12 09:29:48.261 MSK|Administrator|postgres|61de755c.a4c|LOG: 
connection authorized: user=Administrator database=postgres
application_name=001_disconnection1.pl
2022-01-12 09:29:48.263 MSK|Administrator|postgres|61de755c.a4c|LOG: 
statement: SELECT * FROM large WHERE a = fx2(a)
2022-01-12 09:29:48.285 MSK|Administrator|postgres|61de755c.a4c|LOG: 
pgfdw_get_result: before WaitLatchOrSocket
2022-01-12 09:29:48.285
MSK|Administrator|postgres|61de755c.a4c|STATEMENT:  SELECT * FROM large
WHERE a = fx2(a)
...

By the look of things, you are right and this is the localhost-only issue.
I've rechecked that the test 001_disconnection.pl (local-loopback
version) hangs on both machines while 001_disconnection1.pl performs
successfully in both directions. I'm not sure whether the Windows client
and non-Windows server or reverse combinations are of interest in light
of the above.

Best regards,
Alexander

Attachments:

001_disconnection1.plapplication/x-perl; name=001_disconnection1.plDownload
001_restart.plapplication/x-perl; name=001_restart.plDownload
Screenshot_20220112_093004.pngimage/png; name=Screenshot_20220112_093004.pngDownload
�PNG


IHDRj���-	pHYs���+ IDATx^��{PTW�(���n����% A���a4�$q������3������������n<�:�W��d���3�FO�T&S���9fLrbF�(�(����<������
��s��X�w��������O�����k����o�^!�B!�B!�B!�B!�B!�B!�B!�B!�B!�B!�Bh�P���]�x���D>����?��������/�I4M��|C���������3B!�B!�Z=���
�����~��������>�����I���1�(�����w��9N>B!�B!�B�	U��)~B!�B!V�E7>6<�O�~B!�w8�!�B!��$??U|b�2�h��Y���s�rKu�r�Tf�,���Ka��R��`�G��/e�C��
|���w�uQ���^������NP�T7�T�;���<�s������hn�	���B!�Byb�O��
h�=hGb���H��$I��J�"����X'��T��s�����y3��=6C�Z� r>���h�9�������E���#h��;�|!�b���J4P�.)��,�-���*@9+r;J��+���aY��\h��!��MH�M�0|�B!�B��Ln���t�����[�@���7ytyy�c,��+���a���k�=�-)�b���D�h$�q�;�\n�����r6!�+����"�H�3�`��k�Iv��n�q���J^w�x��.	���jF��=x�M���|��c�B!�B�����
�e(
����/7��@�d����[�t:���,lG���b��l2��;
h
��y�	#�LIN
�(IE��;�,�����M]�s�y�H��&������6$��7�|a�!�B!��$��MHRG�F���4161<08��0C�v8��i�)`c�4����b��4�F<E��-{*��H���9.W���
�Q��[�4M�4	37�s�J|�8�	 z���|�`1V{,I[��������/���w
y��c�B!�zR(C�K��bb�C���)kS'����7�f��.4��D����a_�����\��[16I����^����IS���ff�~
2�d���!s�)����t�{�5��-�����[lW���/�Y���������5�K�F����y���7�|����i!�B!�;Q11eOoW���3�����)����&���0k4P2
4M�@���5��6
4)D6h&�A�����b#dL�FWd��
���a%�,V���9O���6��69avih�v%	FWrdhh�x>�?KA�Ly�������q=F�"�D�Kf��r��\�SG�8��q���U���8�B!��B������,?!�BKJ�,{z���?��\.�OL�X,�����L��%P����D�ip�ESx\�W�n����8�$#�����	
�I��j1S2��h8�A��S��]������5�w�BQ�������I��d�?y�3��Or��"���r�������1`!������i�yn��!&F��{��]���ZtlL~���M�	I���#�C�����UNOMgE�t=4��?}��!��'���-d���n�8;�+29>a1[��ryxdxo?��q�@�d����6�2�l�[H��#I4�1c�a0����V��yl�������',�'���#GQ�����)k3��2�&����_�T*v���/8�e��Q2
4�	Q����_N&C0�6����r[��dqw�c>���e�h�q�;���Yx��}Lb��6����B!Dl���N7y���	��{�p��j��63=19���r����GK�����.8$d�d�h�?��!��
�E�Q�����u���0DR������;��g�`��@Q6���_�wY�������������V��$�0Wl�������n��z����^NE����	r���w����l��5q����#"(
��]C��BK���=]�}z�����h��S�XV�-&Nu���O�������%E]��M-l��J�����H��n����Np84M�f���=����B-ZL\,�k��bU��z�v��������I%��f�����m[�nTLL�*vL�eM���������)�6���
#�n/�iv����<�T�X���Qv�\�(6����hp���e�f���sT�*U�jzz�����h���8�v�����h6
I�@�[�3��o�12���u�y�s����yh��������l�{[J�@��I�����!�Qs���%y%3Xn���h�~_��4-�>�%�=P�{�!��Z�&A�x����?<"�,f3��T�WY����_\��;7oM��"����o����OMo��ybl�FMmxD��O�&t3C\��
���G�H�i�/|7c0�����]
�b���-;����7v��[������N�#�q�X�������&1�hsq����C7GLl���M��y��@_�:%����������������L�����Ml�����gy�DTz�����E���y'P��-��������BO����\.��eTL��n��M�3<���-+���5>F����vTLt�����o�d;.��9*&�sH�H�h�yMOL���gb�����y�/g�fj����>T�#��4/J�����{��I��G1	�-�S�)�������6����~%���4P�q���Q���ip��P��8�2����L&����_7I�
��e$��b�����n(�{>l�(f������t�'���s��DH����� �&y��8��V�,&��=�x�DFG�nB
a����6�z�B��F��7�+
����_����4�&!������hUL����\ec]��vl�s�M��d��Q�*ggf���M��*C��
#�"M����n��������hU����
��i�Y�[��iG5��;�x�����_��������^�����\uJ�B�0�M����:��z��M�=9>a��
���a���kv�nbrdp(ymJ��?�W������<��l����y��!��!.aM���e��f������+   4<�a�����`e������F���_��,FC�dE9�v��r��o��@`P`���U��j�l����=��ci�ym�9��QMS��M�%}�=v�}�s�uW�m������GE�u)(�� &N}zZ��f{�f
�
����� ��-+{Y��S3���mq84����*����)����$�o,.�S�840������o���� 2:�t{���D_���1��`�Q�*'��E/����
�E��z�b�(CC���k��x��Bhu�������_�)������l�\w�/�3rTmu
�(��f��XW\����1pC���Xf����$
@1��9����8�Y�O�����!p�+!��C?9|(��	��%�$�r�K�����v`PPv^�LF9$�2�����xg����,S�hp@�2xzj�v����g�01q*�njnv(`r�����&)������������R�\�K}H�������������b�g>����w1�3���+}��nBeX�L&�z�p8f���/EDGl�wo����)0v�B�3��sss��1���5	����a_����AWl����y���i.'}A�����@������W��l~���(��NOM���oerY���
�
o������b��w�d�U��l�����M��b�5��������mV[��3��B��)�ThX��hd���O��k���~�W'r�b���3��2MF���,g�w�S�Q���W><0HVZLIO���a�X7m��_����M�8�����.7[�XWOZV;g�+)�2��}���C�!O}��`0����� eh���7Q3<*��#����cP������0��i�O�K��m������7@Q�~n�*�v�9��<77�����;��@��L�����f�k� emZOop�J���������1�?�/�Nt���d2YtlLPP��f$.���H��HAQ��77o�hm���!�V���	���y����dQ�Q�*��E1o�HU7��Eo.�����!����;�5�� �4&�F`v�/�]��l�_�s^�g��0���+��s�/��3�2qKv�H��	'�'�C��PTZ�Z���6;3k�;��CyE���\��(H�X��t�w��f(�
�kh���������

��-3'k��?":"4444,419���wjr���/15%4,�a�i�5cZ��0
�����A�Fr;
��=�s����E���VD����n��_h�R<��D^]ocP��`}k���I��������-w[��>����Y�s�����om��p8�2��dc�!�������4Z�U{K��n�)����1q�q����ro�����i���


��]����3�tv���eHdt��+��v��nok���3wcm����b�b
z��N�O���}&�I��HLJ������Z'(�*���AG���n��������"����8r���r�V���D��27�]	A71i�3���d2���P[GB�]�;��x
�O_tqda�
�_DT�������7�'��������r4M[��:��p��K~~~���.]!�x�8'�
��������9m����kT-w\KUP�=�MIK��d�du�����s�,�����#������y9y�2��n�55�5��z��-w�����-���v;�b�B�vi6����Hx���y�����v8(R��)C(h��sf��vv�]F�r=�(p���I�v��;�<�!�
�U��p�NZ|X�`%�^�5�]�o��^�����0��|]+,"L��i����-��r�H%�^��h����M�(�BC�C�:��[B���!0(�v8��3@��:I=�7`���h��1��4��N������M�5{;�2����dj
������[<�*?���v�t-nw�C��
����D�&��������F;7)�/����_�/�W�i����{��������@DTdN^���{�Z#�=�TEjV���2�c�!��1�63'+*6�l6�p��a&v�*F;28�/
@b���;d2Z-V6*�~
/(8�j����F���(?����f����OL��ht���7�w�wDEG��k0�L�����^�.7{�����6���E��v�����-��yn*(~��Y�p��������)k���d%W(�f�Q2��y�,��
����vOXm`P EQ�<��)AQ#C"�CTF����l&�#Y�: 0�j��%)�����-d�;�[����@�qy������M�Q��Q�S�:�kr�����
yA���1��I�I�AN�s%��tu�{��3�a�>V����mQ�)f���k��z:��
7����W�B�n�����]��w[s��&2��L�P������e[@b�
�B@8h�n���L�O�(*:&Z�����F;�g_����9��T��<���4��~�Q��M�w��r���vy��]��!vC2(��+����`�@�t������QhD���hoi5L�A���"o�@PHp������Y�l��~gFVfPH0[,2*r�}n��?Ph�O;�n�j5c&�(��fw��A�mv��NV�
����A�i��41>�v���]��������|R�_n��>�����j�����0�N���g������Lg�����"/h]����{_W�M85���t/'?������##��r�;EjV��v6?�ebk�B��i4E��	�	����#���������[�����3������r��0;������ 8$��i2�uL������_S{��8kV�$�fgfI���7��y�������;7�����b%k�QO�>�5����VB��<��$<|^b�T�y���\����4�g��@��'����t��#Z�yn��p�������/��;��;x�s���!^���v�M\�O����������'W(�d��u����C	I��QQ�E'lV���PrZJtl�0d��\��K�p8(��l9?�,z��h�6L�5�#q�k�5cR7 0 3'���{���q���!�?$\;�G���v�T�x\3Vw����c��1t&!c��E��>�;l��y�9��b���@�V�

�����~rEl�*D���u���IX�.sL����
�3�#���8UHX��f�����ed������T��~&�qdhD�R���m��Q��^�Y��������Y�����������2:<b4 444F���w8���bc�Z�XY/C1]{BL*'b������zgf�de�219�A���y�����u�M85=��9��3�Bf�<�W��%&'������@_wOJz����\!�ulFo����+��:���h2���lV��2����n��3Y�����3M�kS��B��V3c@8bB��V4o�������d����yG�����f�8o)�T�$������?���j�6�edev�� 95�nC�~>�EW���5��*�o-7���������l�BB�E?5������<�fthd��m��9���W�333zCV^N{s���_ZF:I�1�&u����m��������~^��j3��Y���hG5Y��
��(��i��f3��~���2�l����v���
�������u������#��o\���%z������lv�������4??��l�}��7�����(��������#Z�Ao��M��X���b�������i�>jmj)*-�3��
�G�*��hc[s����x���3zCn�������C����z;��_�yS@@������D����Om�;���Q~��Jx,KDT�P��L&s83�����i�T@`��u����J	
]��0�����n,�4���ns�S[�o�oj���!�#��1���������!��2���`��T��.����e1J�m��a�7<0D6Xa��~~�`2Y)J��(
&'�C�#B�
�f��E
E@L\<���A!�3&��	(*2*�j��&�@P�2(H	�����P���8;Q�*�
�&

�)4,24,��7+�M&��d������(�C��l2���/hM��$"!��;�<��c����bY�iC�
������~<DD�L������]�K����rX�tT�
���4��Q�Q`�8�a���H����}*K�����B����^������t{p�WRG1Q�q���B�GZ�&u�Z�u��n�n�kG}]f������
��{~���hlk�WP��d��v��h�3�U�l������-$}L�-*-f�k�GK��M-��_�ySpp����h�Se����D����| �������w�om,�T��g-fs��f�\ZB�p�R�h�G�	����f�?��b���w6m�����lkb���:}�jo^��WX���J�<����E��G�F�k�2���P3���;�C��r�]��z�`��g��z��KFx�-����pW�`M�O��s#���#��+��j��XR��6�n�������;6V�����lW����� }�\d�c�(��<�?H�[�����688��s����=�������B������vlr|<*&�r97LL����n�"�3�����t��i��8_�lP���=�1�.
�|�� !�B+U��)~B!������%_>����zv���^'j!�B�e�r{�N�!�Vk��.��aMMN(��(
(�$�+�	����\&����p���#���B!��1k"�B�ew8�w�|���3V�����>  ���B=�f3uW�[-^>�b�Z�j�c�@��Y����0������
���B!�V0�}�B!�BO
e�rcIY�Bhr|�N}�D�R���C�6������p�1���cv����=�>���B=
���d~B!�B����b�����p8�
�B&�;�����p[���{m���������%k"} � IDAT��j�G�F��6�����s��c�_���wy0|�B=|���B!�B=hzL�����_���A�h+��������,�.�4;�X���y4�F�B!�"`�!�B!��w3V�����df����L��	;��4��{�6��B!�����B!�By7k�i+=eq�4M{x�OzVfzV�h�X�HB!�x!Jel����<���!�B!����8(3m�h��Nf�;�9�8}]fzV&@�}��w?�_B!������6O���bbT�E���:��x�E���geF��X-��n+8�R���l�k���3���!�B!����������l���C��
�v]FzV&������n�p"6���BO�-����&�����m�+���U�������I5U�k/���<r�����u�!!s&SG����>Q�l(.����X,�-����p�2����S��w�=�����?e�������o���������jki�MLrs����J���6����l���TTZ�� W(�~W=9>���]���.<"���`�y��5�?�/����]�������M)�����;��e��b�T>F!�B!����!��4�����4E���������t��i(��&'l,� K!�d���B��V�IP'��6��������1*�J�!�������[S������������S���l��QS�e�S�	����fCq��a��������d����AG�B�X�.c���.|���n�f�m,h��<28��X~�u
���=�1k��67�n�9t��p���l,�����_tI
���S����L.�OL��z�����(�r���B!h�s���B�������i�z7ac�26�v8��g
�
PV���^[TtT��f=d������;����=n�B���Q��
�
��f�����&u4M�I�����(*Z�u�8��~�������9�P�����[��|[e��`SaDT�i�������WX���Q�����;�6�uk�6����#T�usTL4Y�"*&z�����&;?W���P(LFc��;c�l�V�����d{r|�8;f���"�k/�8�v�����P�����-~��7�\�W'y8��Z@��f{�����l���d��
Q��m���:�B��v����V�L�O��o4�� 1D���bT1�LFQ��n�X,��VX-l*�Q���������n�c�=��[�����s��45��TZ���c��w?WYw�zA��������kxT�X���b�T�����n[���l�`�@nA^pH�q��	�����C���_�����F!�����}���jY!�V�[�����%�wji�4�����x~��p�w$�����u���'���y�w��/yc��0��r�d��~���h���?M�@
�}���~'Pbac�$AT!�C�~nn.:6F32�&!^;�q8�k��{z#"#(�j�9Y7�]�3����/(�X}�;(�Z:>6~��5����Oma��NOM���oerY���
�
o������b��w�d�U��l�����M��b�5��������mV[xd�qv���;����rJ7j2�!�S���X���xu"�({8?��kV```zV��h������nC�~j:"*r������� %=�����b��es~����7Ab�������lehhc]=iEX��i��l��N��_�Q�<��r�� z��������C#�D���L.���"�n.n��7L�i�����m[�&'/|�
P����*ao�����ssc�:9����)����i���A!�B!����+��t�w-y��4YS����dS����!���d�4�JB�	06���S�d2�������Hl�J.������c�����LFM�����PP������7�s8s&��NfJi�2$2:�����n�Z�mM��
��FCQTL\�Ao��j�� F����&��"1)I&�MM�,f�uEQ��7=���1�r���ZM�f�p�����
�y{�?�}�������5���i�M���I��98$D.��m���:��d����w�I��!�7b�*CC#�"���9��028��N�6��`���&�4m1[H<���OM�C��������{w[�v��f3�Ll=^{;���NM���5*vk�(|tB!�B!d�GQ��]��c��03@&�f
�]�p8��!��]�:��S�E&!#���eL�������1��$T:k��]��Q���K��L@�aw�d2
�Z����9g�0(8�j����F���(?����f����OL��ht���7�w�wDEG��k0�L�����^�.7{��������2�r�`dh�,�Kb�v�����-n�U����l:����<��Y����63=emZ`p����
�l6J&c{e�3��r�B!5D��VHQ�3��eJP����%%<k k^Z����(���g������.r���koG�F6l*���������������1B!�B!����6Z7�?�a��n����\A@dtT������P���P��}��9��K�!���eL�)*-NHLf�,V��E�D7���^V��i���O.������TI:YX�C�i�&3[�4������5��k���`eH�:avf�LV5�
�o�"�`dd��hk�jw}v�����b%k�QO�����AlC��a3z�)�����;<�?/1q�����W�MM�h����Y �	����N�����Vk��s8���5��</��&�Ls3C�:���>��&1�,*�) �%����O�P#�^{������#���+W�xB!�B!����7�gu>0��,[�/�Qy��n7��'&�'�4����<��� �B�	a1[�S�	�jv������$�yn�������������#���2�I���05����'W(���r����I�P;�U�$Y��������j2��i5		��(�f����
?�Y�2��x�f���u�&�4M���r���	I�����X�����W~~~6��8k�(*5c����l������~~~�~�9Y}}�q�xD�5�
����������� �!����������������������m����.�.M�g�����r�\&�)C�ly_z;����N�����e��c�B!�B�����F��J�������&s��r�[�6�&td��n|���+� �q{|8�!���j4�k�N�O��i���n��2�4�������
��{~���hlk�WP��d��v��h�3�U�l������-$}L�-*-&I��Hb������7o
�;�-w,$$����v�=��@������W����X�����Z�������������J3<��N����6��Ag�fd��77�-��q��{zh`���YCj�x���y�F^a���*)�l�[���
/���H}m]f����
`�0�r�yH,/�����6m|f���������N���fT�]=�����W	A!����+_��K��y�Bhi��e���/��S�<U��'q�y�o|������Tl��/d���J��������������v�_z�g���.7���$�n���l�b�:��l�\�/g�����a�G!�������/^��<��i��{n��g�$Xup�
�B!�ZxQ�E���%���^d��\S�f��H%���4���T��!�B��G����<�~�
VTZ�����3V\�!�B!���2�<+y�j^������o;��Q���`�u9?SY���(�Q�Bh�������������
a������^:p���������2�����.��?����\���g�Z�X�zD�������{5���<��\�/~�Gm[�OB!��6�L��3)�:8�4���U�i��3���1� �X�D2B!�V\������g8�����+��I�R����x��Ru���,U=j��������+�R��D=�;�.������s����?�l�����*~�k'����U�F�["V��?��W�r8�:U��������rk�9����TWjy=8�:w�G���@�o4�n���x�������S%�?d�O�
U����L�
@sW���cR}\��S�!�B��Z�����o��������{{��@�k�����\�zV�����g^��Q�aB��k?����?�,�� �=�	��Lw4��wE`{�;!����h���*����Yw�^{�^���P��C� ����:�I������]�AoEX|Ht�5,�@�ylb��h=-}����a��E����2x��	(�b^��hp����/�R�S�y���|�I1vS>y����A~��D����
������,#�B�!��1B�Z�g����O����*�����w���>s�G��wv�f����A�@��?�x������������^r��x�F���w3~��>WI����v�[��po���������p����<�����%g��~�si�T(�U�urz.6����&:��# �oY.��	fL�?�1�o��*7cGn�[9i��7���$�]rf�N�������q�
���>x���8jIW&YVa~W���]�m������\\�j��t`I�xy5x���`Z?yu�
W���(--\�bEE������z��p�{�Q�B!�����c�V���_��;8s�����)���d���W�;~�-Y�b/�������v]i}��YL�� ������-�+�?x�Jk����_�{8�O��U�����^�n���/C�;�[���[u�f����ir�/��v-� ���{��'��v�;ubMR]�S����hH�6����t<e��t��(�r�wp��Jk��#E_��q�Ok��V��J����	��k�i��CQ&�W;/S�&_���s�dq����H��m��k1��NN��\NW�*q-����F�����O����<p~3�����l�������N��5���N��O||��],�"$��{�����R�S����������U��8�:��'�|�L���O�s�wb,�']�;�Dj����r	����9J��D�<�VO��UsN��2	�G�))�I�F`��w��{O�pT���h��I��7/��;^�d�~!��y���1B!�V/?!�xH������l&���;~���r��.��U�����=�p>Z��������3S>��$]�����������si��>����������e�� Y��b��T_�s���uN�����/�y����O�����+���W��M�w�u��\l�Z�H�x���,~��}�u�K������3������E�Cr���A��������@L�6���MJ9u��"`���<-:��������R��q��:����a�U�;��\8j�:�F�g��O�=s"����$����JO?}�HER��9�=��U;�*`��!��O�{���m���dS�'�)i������\�
s"/���L�����_h���,>��J�f��i�Nt|<T�ng=���
�;�'[��n�)Hu������U�HH:�=T����2_�3�[_����v��u<s<��f���D��7���q�Dj�����D�S�6#�oA=�>�$�=Agf��P��k��Ta�N�0K'�������^�,r~;H5��7�r;T��'-�r�)E��<a�#����<������6�D��'$R!�B+�>F�	�����t��<���;��M+�HKu
�&5�����T�Nr�����teu��U�Zw���q?V�^�������{��8�V��6H6!�U��)�s��DCI}���%�1�F�������D�)�"Ad��#��Rr��W��-N��e�����U=g�������]� ��cq������v���>�t��H��
������8�nrf�s����O��<�IV��9�����T{�a��������g��b���9����?�a���u�a�9��18Qz��?cTt|<U��K�]��������#�>��.q�:�.y���K
 v��)>	;���|Z��s�`�ig��\����o�W'�����m����M���k��;?�/����G`��������V7��y��];������B!�h��c�V���<���J��+�x�����vfn���v������d�����>v��z�[�}]�S�$�����EXI#��x���Q2���t���RYe6��� 	����{Y
k�����tp�HbYU2���uz��K�i��Pxj1��'�[���3"��� $�w����PW��=�V�'~�?x8���������K>��P�C�3���N@>;cq-��o�xA�
�f;7�O@}x_�a�}����z]�NE���S��T])9��l���Y�H����'�@Y�$	�Y�1W���tP�R�/5�WM�2��{��h`v����q�k�mPgD�v8|������6�
��������	T������TaO��6�q'x��Q���h���6��1P�������(
(�����+
d��B����\f���]�@!�<�V~�Ba���,y����~����-�g�l�������3�'$���Zw~���?����r�z�J�i��u�uPs�+X�~2j����u;�[w��*h}����KP���	�������`�j$]�G���!K@0�]4/�MFFK�"m?4����2+���hQ]�1��&_8����N
?�Ls;��t���"m���~�����z��'������^P�����-!�k� �����C_d�{�7-�Vx�_9�j]*v�kw�]�����������
�?�0.A�K����b5�\5a�����h�?�Z�#~	����.�U=g��~���6�����kB�����b)(0"���dr�L��d2�����er9%�S2�L���2J��d2��Qr%��drJ.�(P@6(�(�� �`��c�1�e��b��l���C�/v��&cp!�V�[���������Y�:�o<w85���(��	��+~*@�/*�I� �' �V��?������?�x�M�y����������5
�_9���\���$��'/52�����?w���C��g����?|&��/JAP�}~�= YR���b>���sga�o�8������������C ��hWy�u�=g,�H�F@���$;&ht�H���k�{�F��)ha�S�5w��k�P�������]���I����i��������,2��|�=l�������s�kw@R��������d����+����}�I�iQx�/�<Io�����_Q��\�[sRe��K�#(�Fx��)^
v��`%��q_+7S����S{��]'�k��%Z�0�s�-	����M���y,��^;�a�.�����*n�^�y�c����T0�W97���io�R��B�G���v�o�V��;��������l�����p��� %��0�X��=Uo�.}������M*��
�Qj�z ���������\����w�N�`j�������m}�/~]x�����V���)���v�>���1�[?s+.��L�&���#�s�eI���Y.��]�%A�����m���I{3g����l���,��;����
��t�Y�����g��^x���jO�?�7_g�_������O�\rf\�9W���+����$ZL��c�;O�k����U����o��Y�(�]����
�����{s�'[��,KuI�;=_�R�H� �j���V
vO^b�� �pV[u��O���N
�w�]�Xt<�5so3D�%~�=���{\D{���X�;�+?I��7/Z�\^�;����+F�
1����.S�hn#!�z���������]/����o�?uO�����+5��ndA���v��@��]W������*�:$���+��E: 8���w�|��o�o$�*R�x�Y�2�����RY��v1��%G��QWy���[�]{h�Q�N�?��3�*��C��yO�L:RW�Z�����O��f��^6K�r��3%*����DE�~h�tv@�;���| 9S���{��A��y�N� IDAT��tD[	��%����7���Y.����N�[�?��S��T~�����v����+.��0����RZ��@�R��� �j���^/�/=_�	�?}����o�;C��?����p��{�t�G�So�����K�VC������3e���m&��	�j�b�Op�B7�)C��:��,���?I�~�.�[�\��dKXgN��d�?u�#�+�\;�%$x�_�f���������	!���E]i�G:B���?�t���+^J�������?���~]B5��8�z*�������$?c9����_�����,T����'y���G�$���8�jL�!0�p��^=�'.���N+�1P���<Q'�{?
��&+��
J����2����)�s�c�%�Q29%�Q2Y���(�(���d@��kY��|jIt�c��}�BO��]��X�:W�����^
�>F�	�������b���R�ng���;^rK5KU��,U�Fh�}������}JZ*U=g���x�c	���+%����?���mV2z���z����2�NRv���;&�<�z�@!��r��1Bh��W?X2��=,�2u!����*������@Q��z����f�������D�8��a��i��)v-r1_4o�1B!��N9u�^�OZ�����l���C����!�B�O���#���&��vq����2�����c�}]c�����,���w�B�qT��I�x�ga�I�[���!�B!�|D3+Q�h�W.�jLP���B=z�Y��g%/7�����Xy0|�B!�B�;U_S��R&���r��\&�
�9��K�4HL:F!�'�L��3W"�A�!�B!�����������l�6�l�6
N��l�d�c.����OE
W�:^�Oy����r���(yk������^^��a���T	�go� ���rk�9����TWjy=8�:w�G��V��+�uC._�-1#�B!���27k6�s3O�fYL3[r,F��m�"�c����c�9��x�������p�t'@��~WX�g���+�,���gl.��7��k�k_~j_xha����T��4i�P��2��yGW��m�y[p
>PDE������n�u�D���.-��G��>��~����z��^�����4�B!�B��o����$����?	���p������!I
������&�8)���cV���+�X�]��w ����/t7��s��t�0'!Hz�X�?]�8��Y���-�C�>s���>F���
��^\�OC!�B!�fP�=g�J�;=�(l�������J8p��jGY@�&�N5��qUcj��i=-��N��.����f(.���<n8�:��U�b	���gN40;�O:R�<0�UWJ���e����p�H��u��O���Wo��X�Qr�����p���28�v����D�G���r?#���{�����p8�}�?��(���I�]r����3#�s
��&L��{��(s�O�{���d�#�FJ6��������Y"5����?������� �;J��&7,e��5/��j�p�����.8��xK��2>:!��<�Mj�fV�g�B�'�I��Z��T������S	|n���������@L�6���MJ9u��"`���<-:��������R��q��:����:��\8H���3�����9�[Y�M�C�:~�����rWsP{���������
<y����/�yj���-L}�D���bZd#���0'��[���;t�O�B{H
�f��7T�7IT����x�v1��z�/6d���9vO��3��� �%����WM�"!���P�[�>�d������/���@����l�Rs����u���w5H�H
�uRx���0�
���%�*�e��kb�}��L�����=8��vH��Ju�^�OZ���S�0F,LY�0|�Bh�Q�����"�Bh�]�t����i�����_������}���N:^
a)�p�����$��9[��E�+���M���<uU�Y��P|Ra���#3�^=�B�4@T��-.��K*��r�'���UwN@�l�4-���u�p~�HERrf\�����yj��$��c��$Z�s�8�UR>7\
e�=�69p����`e,�Er,{"������~�D�y�z��6v��C�O�{���	�s"����j���ddI��E�4dH����l�9� �%�|�!�WmP����8�����`�+V���s��%�M���3e��I/�
�
����m��|��Zx��k^���4���!_`�!���]�|�~�iv�-����a�g�<���Eb[��p������ �w<��J�B�^��Y*�wj����g�}��+��g����q���VV�
zj?H�����^V��(��1t��"E��N����]:�M����S���$��Jn�I��,��;=��d?��������8�����y�lv��=_�I�s�	!���S' ��0��k��C�������p?q��}�����So���u}�_d|�=U�M�����6�.GE���5=y��
$I@��D��\}�:�A�K	���\5a�|p���clz\�Zv����;��Da��p����*��	�7��ri�H
8��G�#�Ze�v���E��v^����/_�����|�Ka�YL%Kk�t!���"�����|����<��2�ME�~h���C+dV�����zN��/�_^Z'�X��8?�������i������5n���?y��
�<����H��3���"��h=*��������r��Z�����D�����?��?���bC���'�K�R��.�X
"WM����+%����a��_��-��K�*�&�����Nn,�Z�A�6��[���d�b��	"�>?��y�0�_������l,��A#�����n����mKp�^x��X�+�	�V����;�n����G��o���Xm����X����/Kc��}x�
�g:��\�������q�X��c$v)��0O?���ydc����:��e��x��A!�Lx+W�����s��)��4
;^F�������s�Y���g���m���[",%&93Z�]	������,2��|�>l�������s�kw@R��������9����+����t�+����(<���IzRV���*�����^�n>�F�����TA��R�J�^5a�W�]:8X�.�������>p���og�e��o'=�P�C+���fy8
<6G,G����q�q�����,�����-��������U*��y����3x�����n��~��,'����q�4�Z�
�^���i.F{1\W
I��n������8�9/;�",�;n���i���r8��]�� �S�ar~��}~������(r����X�����W����_��%'������];���F�:V^K��TO/�����Y���r��l%�X$�r�0��zDwE�?dR���8H�������n�p�D��R��R��Dn:B���?�1����?C�Y���X~a����8��e�����l8�~����g������}��L<�~v�%S��A{{'�A�K��	(����%���c�R��
�����h�����V��$��$C=��b�#���*Sz8NH���\�l�d~���5k?��3�G��������Z{�����Y��~�Cz�W�7�������!��5���Iq����K{�����:x"��+�R��?�&�Z�;<��eo����lQ_w�o�\��y�����oI�V���l3�����l��}��������>�o��3��i�5R���t�
WM��jx*%f-���>b��0����i|�
��3-w�o�/��[���&ah���_,��B�;)UNO�Z����`[���k����M�2�S��\"�������;�u��d<[.f���6���]��If��Ea^��Xi)N���{�L�����I
�5S���
����Y�S5�`��/)=������s���b	��8���eG���
�K)o��U�+��"�yy3N�g�;�];���������T�����ts��M���`P�5+n���0fqB!#L2e,������&o<�[���Y�[����h ��/�?Vy������r�:~�G�����\�����2�O7Q�[���A�W�p6��	|\�����gy-��l���|y}�|���"{Y��cO����}�zvm8@[�f����:������4b���U�_�Wq&]�]��1��J�q��vZ�P��r���������~����/���7����~)��P��W��.y�������OqZ�v;���l��p:�~��o������'�y9�-��t��lu���g8xg�|9��<���B�3����+��q������s���!��];��`��CS�9���f��=���V�d[�W7�y��&L����.������,��������K%����ca�g�������c��Kj��W76��)=t�]���ng�������Y�&-�U��}X��Y��]���
�!�2_������c��)S����p�������&,���`RN(	TcG04+u&������+�>68Y�����V�t"������K�r2_��a�E�So��l�Kq�d���{�����(����dg���^c�8tNoq��'(yv2/	��4b�?[�V�;Dl[<��	!�\�n�y���c].���V\���v��n�+��n<d|�<�%�1������r$���]�|W�
��8a��T������[
�~Kx-���.�dz#i�:&�r��������[�f�v�bw2�|�����ns�.e��c��[��zH���gM���u�<������?w���V'}�$���s�
WM��z���	�Z���" �l�?6Z69�����3���uP�!e3}|������T#���C��6��|-0�����e�O���_����u;6�����;NL�oSm1z��p�V��m���}�3gJ���)�jIU��j�|��M��W7��.���'�������h����Z����{gJ��-��7�3��6�3���7mZ�����l�IA���u�Yn��L1��3Vq�����]]��u�$N��I��q|�x�/�F���	w��f(�:sy��J��F�&KH
o�G����t!��-�iK��3%���N6�7�����N}]�?/�%�g�;����@��dJSW7�g
~�l�Ww�K��9Dq�"��R�b.����`�����`}�=;��	!�����O>)GF����g��.p�����r�/�DB��1!�V���k������6V�}����LH�L����:S:�v?���v�����4��p����.��t|��y7�H�\��NKS��������q��������9��*F��dO��+g�����u�<�������\=�DSG�����*�
Y:'����ng���`J5K4���r���2�)}��8gt�r������?A�i���x5�ao7���������\��]	� �Z���4��p�������M�b]>Y�K������t�}��p�n�y����8���g��*M��O�%���{��^�@	�:����E����L��U.(���9�������X@�7Pys���b���B������g?-���#�L1!���o)9�d^��e�j�����LH�l��Q�������^hi�7�����������`���P>~2~��]�ufr`i9��g��*^L�~�4���,���Q��Ie�7n<�s���{�;�����
q'+��g�>��W���;��3O��r^�����
��a����'#��qf8�>s�3�bTL�J�Q���a2��0h�p��A����2��d���e���������UXD\ba�e!�8�r�O>�����?�dB�Ha0�����D����U�����L�K�����5��s�����gOuw�8S:y<�Q�C|�@x�Yw��3�,���*N+�����c���9������c)�_5a���(.�|&&��y����������>�����������\���ow|Ii��_���A}��u
N�mv���v���U{�~�|���J0�������K7��)'Kw���;���Y�����FW��2�������y����^������,��FqI��=rP���l����V����E��k���a0#�Z�f��U��.g��aI����K�0��E��B�x����2!�B���1��6q�_�l��=�J��������L������J�6n�����/^z����
OY�ii`N���t���c���j�um��
�Qq���CH~�>�_�=q�d:����O>s���)Qu��-�e�;����a�������g�PU$R���g���Y�� �L��,�o����	b�3�u5�]��������O�0e��,.���`L�K��9�(���'K��U���3�����:\^;��*XF�������y�~u����wq����_�lJ���r��X�K!���,fda%!�B.j5����l)�XL��t���U� ��'�l7���lu���H8�������Y��N/�C�B`---��_;I�����D�y9�I��[���"���UP��\��K���v�����*(P��v+.��v)���.��(n��~BQ�n(������(H.�E��@QE�O�(J�0RA���I�N{I!�\d�}g����J�f�%I	%g\�l7���lu����8�|4�8��B!�B!����+����p��Cg�rr��A��vs�;�FPWI~��'�B!�B��C�c")�U��x�����vs�;�FPWI~��S�����NK!�B!������B�a�b��JB!d��Qccg{���n���r�].���v�\.���r�[.P�-��>v+P�Kq�\.��\.6�qj�c��b��R�+P
���Oy��"�k���B��{���GQ��B!�BFG��\s�?<%�!��---r(�W�F��>��������n������M�a������@�r��[�C�B!������\�[n�E^�b��Q�}������yo�B�q��z �Nl����2�����!'���r(7�SJ\p�9G!�2p�?Q�?N������J 59X���AB!� �B,�0�!�B�M�-��������B!$_F��Hhq�-�;����Z��r�������z���&n��V9�?�x<�����3n�89���O>�����eee+�{�#Q���!��[bz<�����ckv��� ���"���.y�hZ���{��}�"����P^7��s2��!EL����"�7��Dc������<3����G:>	G��G�/1@�R�U�,/3k�B!�X[�r�e�]VPP�v�7o����=����z���o��r�\.Wmm-[�����\������={\.�w�`���n��8yE}}=�(�`��y,�����(s��ijjR���������|.��_ed�*�82���nFv�i3.2,�K?	!�C�~r��_����'�+pxk���|��K��W�1HkqmuUYg2S�V������'�-��^�����Y����?��Q�HT+�B��Q��f^k+��D**�S����f������3f�M�Z[[����}�xy�j$��m��/}��N�����|8p��1@��9�N}��k�p{���l�f-�@-�����{�V�gx}���w���G�;@�����p\���@��;>���I�Q5m��l�F��F$�(
��"��"��VT��<e�4��-l��m���)���q����Y##�U���?:��R��Y9b3������
B!���u�V6lx���y�>����?�m��5k������k��U����%K������?>�7�x�7����7��T�x���������xV��&�S�<��Z����0n�+��|-/iAj�5�*� +i�]�<+����|_E!dt���o������d����t�NDTC��>O[������|2�u>.������������~�w���#2�Z�h&R���x<���PGl$�5M���HD�4UU477{�^�����HU�X2�k�^���e
���R��c;7+ IDAT;3���I�#=���?�m
O,\���Yi~RHgT�
�Vj�s;sLo�X@�����<�����\�S��������z��,	�������X4��=�Q.g�H�I�[�j��8��U�����B9��$����"�G�����T;I���=M��jH��H4�]�
 �:�zq
q�U�L�&���i��i����qM�b�X<��b�������3�8���mB��j�X� /%��9b6S��"!�BF���,+*�J��i����o��a����a��E��$�2�\���_�����ry�1����&�j�
D5�W��������iZL��{B�8���=�Ie�4`�KU~��@UU�~���gFe���Vd� ���D��lk��!��E4��bE�g����]z�YT�c1���A��/oKE�D�Z��+a4�������2�qM�i�P4����Jo�������,�+���EU���zw.	�����~�T���PD���kjY��
w�b�L�L�:>����S�jZ2����UwF�5�B�����::f�����w>���Kr'V/H�e>�/����#����l^��CF�1�q������-�-R=@�F�������\������g�
F�f?�N!�2��AYZ�'gY�R_����H����z���������B�������'��Son���_����^z�����������_��O������������M�|�n�����=��1���T5�b�*��(G�T��
���s��������
ma�c�UUe�T�5�%�5M[�lG<������f
X��0d��/*T�-T�Jv
6�`5�lH��4�*X�|��H���:4��Z����z�3�����S>����~�?	7�����-r�L���m�����P�#�jo���������|4S��-��"��u�,��X-T���\���Qx�B�W�8����U���B�r#f���js$�_�����XT������A�l��i�I�����,��1l��%vR���U�H���x�B!�"���{^x��_|��t�&>vj���������lN��f����z��|��~;<1m�0!��|�������2�z�{+�������+�7_x&T��������>K,��������wW�?����Zy�?Ulx����q���w�&�MI>
�dhZ!T����rp`c�����"Q�vg����-RT�HP5UUUU]��c��33�A�]U��Y9����������!h�4���#M��!��U�aU����k�f�X�d	/�#�i_C�*��W��%U���Vlk���}r[M6�PM�7���K4h�|�����y���X�KM�����*�2��s���5��h�=�QT�}hS]���
}>_������M����Q�� �To�����=�-��|�H�h����/��i�2����e��S�O(nLld������i�h�4-����$wvvF��q�~����7�������<:�>UC$��|�qm8�Xb�Y�b�Z�	�[���}O�$�BIZ�n]AA6���=:�8�1{n��/��#����_p��7�$l�c��>�����g<�R�,����T�/���Y;,�v����}�R�k'.��#�[&�2�X���'�����s� ��^������.����-��~2�z��7W���]���7mY�Ulv�	��eU�bI4����d�����Xee��
��h�C����{�;���6���F���U5
�>i������������"�.X����'�@eG4�?^�b�k���6�WW/��.S���a�*������GS3D5m�_��y�B�C/��b������Mu�m�S��j�{<E��6�)�m^��;hio�����z@E�m�"����k�p(-ZQY���-��{��,*�zT
�CGvW��i�;��v��{^Z[�[�S��B������x�M����-���iq-�y=�OYCkHXo@�"1��@�k�4h����mbo(I8�B� _UU���7�0M�
�g����������#Z$�c���>�8{�fU�^vnU-���[^�JS��w|�n3!��Q��g��C��<�_|��e�x�=:O�pa�
��y�����<���2�G8q_��g��-K�3�������e�0�Y�%��w<w�~��A�r������y�3��Vq�����������+���/�_`�_���&����N@E�
5�-�,��

���rNb���Z-�!���-�`F/�=Q}�&��c(�i�UU���:z��a�x���UUU>��"������f���x�m�=��s�Mg�	��Y��db]�Y��S3|��������AMum]��p$������X�\��x|��e}�:�OQ���3�����V������q
���S
��x�9�����B���e���e/�6�G���S��#�8�qv���w �(l�h/*T�<E�p8�J�CR#����f�4D���X�yE:���^oEE��G����p8<q�D�tJ��vO�'���}�p�������sVL�:u��}���H/��|�p���3����x�IK�N�������C�%��v�S�5����A�
-���[�-H�`$ �Y��JB!����c����������w�\.6y���<��4yEuu�i�!��Q@L�����	K_�p��^������;�vlf#����c���}=��
R���q4j{���s������;���"�EBP�z��J������R����R�[�fTVJ�x�}�����%R4M��+�v�m>����P(������k�:�����3������:#��#����b�bN�����%��OO�J��5-n��H���>b_[��H8\�-�D�,�����|���^�+,,��������z����w�����P���*�Z�����|ik����"�9O5���#E~��3��5h���:O!j����GZ6����i_��S����j��N��#6����Hg4��4��}��1������(�`�����Lggg[[�i#�4����Z�)t����h>������G�'���P�E^���i��������V�.��3�B�h=���rPSc5��#N�B!#�~��>���w?�1�����������soZ���Ow�	+,����a�������ou�.)�9UUcr4�T��ZHSUO��8�G����1����
�i#.�o[g�s���PVV8�6L4�O�B�s;������W~�Zx��%�TF3{<����'WT�@jR�,�s�,�K>�1>p5�!���$a�ld�#Pwdk$�)f(����F�V�+�qM�b�"��9+��3�r���S;��H�3j�Y�p_C���
S+�j�G"��=W'W3w��5�^Z�a�
ttv�(/�����?��7s���w>i|�S�8Z;"�8�[C6g47k��3���y�n���=r{��N�-�h��t�?�J���x4vr�=�</��X�i��F���_2r�C}�-��u6x|U���=�0�b����B!�B!��pon���|�
���������SKY6�\4@��U=��Z�sZgZ!�S@�tIe�L�a�3]��5�A���NMK�b�����H���=e/=�f�����+T�_ko�6|��{������wn��_�-_�l�y��j5�
!��D"���
��.Q$ae�a�#E��~Y?�C6�0��p6w���G�"Z'T,{i14h�CCY���_�"U�j�H���`D���cG8�Z�(EM��#::��2���f����{^Z��[�7h�*�[�� T���[4���6Gk��*����qM�|���!B��C;�+}��p��i���u�::��,(l��*5-j����H��*6F�l�XdyY��HkZIyi����~��ggG���������{�j�����EBT����X�
��vo��U#4�1!�B!�22L�:<��o�Y���0����`�/�};9�����������E h4h��y<1X��Q�0s�,i'a���k�����j���z�Wr,*�'^�����=h���U,_�����Y-*���B��4L���=zFY]�y�oEZ#m�m�����m������if�e"�����[o����g�u�4qM�;����rVv�ka�����!�q���cj�X��yj��U��p��\�z?����H@g,�yXXb��?�Xz��=���q��bZ|�W�Z�i�������J��������6��������f,|4�yiD��%M��Eb�v���"�i(�"
{|~���@^a�;�YZ��~��fj�<���H8|�Wnw ��N�g�������YfY�V!�T4-z�������*|�O��wr�}L!�B!�g����dR������;mw����������u������q��j@�
�D��T��0��:�1� ��9�����A{�uEY�P;����;a���<��r=�����aA�)v����Y�q��>
���I#HmWU����x�=�+������O���O%I�P��d}��fE�d�s�k���o��G�X,��<����O�L������X�P��^��n��=/}��E�0����������x$��|�k�|�����[|���f��;��{#>U�y�9�"����[-������i�p�"���*����mG��7l�r������|r`!g�wj���G�
�=u�hq
gJ-���6�[P��!�g-z9��H�����$:!�B!�B
��� �P9o�����z�X��
�xJE����8�r$��m5��^]��������S=�@L�z�x�������HY������������2����0�2�����l�1�����{�^�@,��a��rz�
h��Ba�?�z|~h�K�ffm]��P�#t����@�4U���|U�J����N����^�M����Hd����sM���i�*��Zk��;���C�*/l��/OO�3k�j���y�N_e����Y"�]*��CMC����k������$�:5�����3�H�!�-H^[����B��^+���Yw���6��R�}�o[���5��"�Z�
����pcwL!�B!�����u�r������c�h�F�QuE�aS�Q���~L��I�3�\Cx_������������p��~�����S���h'���g�<��!�����a[#a�V�p>����-��b����K��oEEC�[[����-[���?�Cm���2l��o���m�����|t�o����w��z���M�Hx_�Ay�ml�r?8z-
z�����pLUT���b�'*������'��\8s�\'��yx�\^h(� ���E*4-�B��#�
�R^�����YS=N�����������}��>S�_��i�z�n��ym�P�����!�B!�r1*�Tox�mrX���gO��I�������L��e��y����v9
�W��HG������,g{�*��T����mC'P���Xa�HMMrV��/u��9�����15�(,�����;���~>����
Gc���iY~�3w��u7������e}XSM���{?~�!�Z��f�4���UP����;���a�RO���A��C�Qov��Gj��g�W�w�A����U/�Y�	!�B!�2�|��=r,+�'1WXX8q�I�m�F��Q��6����Yyf��)�~��xr��l����I*��K��������������d�k)/��Y���\���g���*��E�a���E{^!��b��un�{���b|��mn���r�w�}b��^p�\n�{���b|���\.�-��"�<�(
�y������FEQE��N��Tss3EQf��-��>�(n'�B�\r�B�+~[H���nhoc���Z^v$���\���B!�B�}r�B�03��?�CC�'��G�E^x��/������W9�	!�dm���r�v�Z9�����`��Er0�|9�;w�TUU�!��g���BF�_<��������n�I�F�7�������[K��PsV������\3O�Yd���4�>(�G,���u;���#�E��y���Z����m���Qt^!�B!�"���C�r�B!d`�A��` ��b��	M����H&J[��gl���/-�%����/�Ul�-�U�Z�0����������f�/9�U����6��/������B!�B�}L!d����C�f����^k6�Ys����,�rt0�!��a����G������=|���~���b|�[6�o!Gyo�B�hU��/~Q�]����?�!B!)�fC\��a��!��l�E��b���v�;@3N�[������9-/����?6�qt�K!D2{�l�3��C%`{r'V^�fb�Qi��������7H!d�*8}�������?��e�%� �n��m�o��������v��C��?f���y!�2����7��gM�^��2!�2"�����]}�����z�)y�po?��#�$��0����b�,�/����K� �o�f��iU����l98���*��/�U�A1�#R�f��b&��c���v�URSvZ&��/+W�����


�n����Y���s�\<�����].������e���y��u��w��g������;�����v'����g6���y�X@cc#EQ�������(JUU_�G�g
_�6��!�x�6��#m�E��3�,!��QO��������9Ws�����G�
20�p�/~��4~ ��$��M�^���r4���%r���n�!�d������"�2v�M9������r4�3f<��36l��r�y�����n���k����Z�
��cF�����.\��7�P��oD*w<w�\MMM���Sm���R&��0����l�_�4��~�������3+cgY|��U�B�%��?��+9�-zt���>�H��q�������O���>��tG1~'�7J�B!vd�T?y��ivX�`0h���?�
����1l�0h��
��EB!���	!���dxH!��%�A�r�����ce�V}3�k3�w^�)������*��BF��s7��#�B!~C"��B!������^x��_d3�3W�7w�������F6'r�9��������T����r�/f�����NaB!#�>&�B!�b��u�


���9��<l�c�M|���/��G�x����|��b	�������������#j�L.c���gH��������X��?����mW���@ `'.��#�[&�rQ��q�}t�����z�H_��$<*J.����?��$9��>���Cx���@��B�����gf��5r���/X�l�,Y��u
{h3o�<�,>:����g�!�9b�8'`��1�e��8Y����/��2+cg�3r�k	!��E������~��w�����p}}�D_�������4��K���7�[�v���	��O�k����G��Q����������?Q"�	(��������������%�D"������w��y����x�:)t���#<
���0y�7!�B.B�=�[��}��A��{���b���/_��NI�WTWW�%�������Z��
�_�BVl���x����_�m��_���oX�]���Dy��f�����}�\:VI�ip]H�^P�P\H@)�������O���g}.���;N�_)A��O\6�/:�tA��s�x�i��+����jY0j���s_���?^���������9s�"/6P�=]wG��.�L
���23�e�A�����J�#l������G�p���*�Z����s����O�/��S���7���B!������!@MM������yd���XE!��>v����/��jl���O�p����A�Y9n�(5lG��O]}}7�]`��v�������Z{��(\.���<��H�O��z>����'� �g�}�H$���*��o��WN�6
�����Z?�bP{wwW��@yq������;K5+��ei9g IDAT��G�6�G�f�e�O���c�����9��)06m=!�B���+W^v�en�{�����t|����r�����\.��y���w�\��r�����\.�O q����|�f�VpC��)�[�b�3!�2�2�I'?����W����:O������?O|g��~�
��1 ��Ne*�D�_�;�Y�����g���z�������������Aoo������_��_���D"a����s���1�3aZ3����7���}�=#������K�������$����]Ni�B!����[�l�������������.��W,\�p���r4���f��#��)�n��1!��a�'��a����i$N>}�*,*_��zK��L������ne����mi3�)��I�\��m�����m������~5��*a�1q��v�z���0��o���7��C9^x��dU������r����oy���T�x��+.�+a2x����������P�#�{{�D���x��
?�q)��>�X�������b�o��	c��L��@y~��#�~+��Z Y������9�|�+)N}E�1����������zwJ�0����}����'O��b������h��a���vRC�C(�4N�Ix?�)�v�g��1������1>�]N;�z�N5(�|����l �@��P��������bO�1�;�7�Gx`/������@yq����T�����v���&���KCg{&���}�������2��~1h @sB!%�G�Y[�l�/�K1�h�"�ev�������P<�w��H!��E��������\�|���kc�o����Xl;K�~��mKG*k�N�*,�kD(�����E�/j��v��Y�c���z7�z�����R�z�.�
�V�W�b��B��z/�B�8���o��wYl�����H�%������K.�������D"9`0B��3=���o�[���0��<(�2e�z�������1��PuOL�,3� ��s�������OM*����	�@y����g�v����[�[����N�l(|����b�������������}�k,x��Q�D2�e��a��<��T�zN��;0�����\*k|
zN���9u<��	e��,�5�Y�']2f�_���c�r���:�y�M��|]�)��|�)v�{N�������5�x���5Lo��K�����	���O���+��B!$?����g���w��� M^a�����(�����c����*a��������Y;�qd����6��B��+��-����]������r�� ^��oxSb���z����[S=��%���G����C���n��������}U.$.|w���x����?~�����>�>�"H$}������&�/�����%h���L�n�����@��d���l���~�v0��0�l��1R��l7|~��Bqy�=����&$#�2R;�0�%m�3���N�%/��
<���\.�0c���G��G�	eeN����S�1���v�� (I�������h����t���f�wwe\��������1c��v��M���d���v�W^��6I3���u�t]Z\�s������w8��!�2��[�����#�<"�3���/�d1�d�����fX��==������I� ��y��?Q��`�F��O�e��Y<����V��{�/,���	!���c��y��>����S�Xh\3U��K8g6����(8���e�^_�w7o��������W2�>NW���w��T*'�B����:��3<�f���mq�#�L
L8w���n>7��4���s�[X���z�����I�����S��1c�����Hi/�|��av�
OzVL&SG�p��V���	����`0X2i:Omn�)�'��K�
�I3��\�J&<!?�{J!���K�����[l�������A�������G^�Dcc�b�X����g�l,K��._��Q��B���Z��;6>����r��7�*��2'��-���zV���7x��7���Gx�������Gl
�XH�����/�m���3%���$�8+�^%!?[J����|�����&����7��z������K\H���gp��,�u�'5dr��l���;��F�vu�V�����d������\����Rc���1�%m�3,lH(�s�;�S�.�����w�y�������f/�����bN��g��-u�����	�.���X^:�c.����������TI���6���L������zP\>�7����KQ�]��5���a���6���/����ib�=%�B��e���!'���;g����jq�1���Ma!b�c�D�V!}x��c]e�u�B��P��][n����=o.N~(�M����&�2�]w����5V�m���`�}[���p����u�]��|�.k){��\4���/l�*�����,�U��z?�y���o$��D����O�����]����S�c]���������M ��e�x������.�w��L����<��>m��I����M�f4p�P%�.�'<�2�v�j�[���.�����0v��[�.\P���c<���w������q6���1@�|���G7��L���CH����h�	�']��;,����������#�]<}��;����6���g��1�������5����<P���m��'q��i9����J��w
�
X�����,���/������/����gB;R��K����0�_�8tA�8>������+syz^����H�����������EQ�@�h���~KgW���n�B������}�k<��)���o�?LO������ ��EI$�����K�^������)j��p���c���~��}g���+��_\�����S��������=�����?��%�">���H$�r�E��wc����������������c��=��S�O�����0��=�!t�����S�����G������5k���A9����~�3�w;��y�f��R�0�a����>��AG����S����0��E\Z+G���BF���G��`0���Ir4��w�'R_����+������.+((p���7oN/%��{�+��_����cSR�|�����yDQEQn���?:�c��2�U�����8�v���}p�����L>k!��>y�����_��le1�q������mi��a��9jsj��=�}!�����|__"��W���|UOo__�7Q�
���#�~z�'�9d���q���'G���.����������#�
�;�F�I��4��������'��OoXu5�7\e�N@7^lp�13f�6��
�D�U�!#��p4����F��G��-\�PH=7�O3f����f�����Z}��7+C!��,��s��b�3D������r,�����<���@��(�a�m}d���0-��9�#�
��]��$��0C��7*�qy\�����8��*1h����v���_%���1+o�N�<�l7��$���Y0�_��71$,�K'�����4��"�BF"� �BF��c�j�����Y;fq����/y��d��a���^@_���V��7��A�XQ,����gM�$��l�iy^L�b��qw����8�K9nQl�l��8!�\��oK���B5�}L!��Z�Y���!4�V@�����^������UeQE��.�J������f�k���8�5|��B���{��r��n���f:.((X�j/���/��������8��-��=�����y���W����_���B!�����f��%����5k��������+���
y5��Q`X�D~�#.�
��{3����@`�ttl��d��,��"tZ~t�G,�����/B���|����Y�@]]�\B��}��~y��C��F�������7�Xo�(!�2lP������z����>}Z^A!D���J���4�/�������1��Y<_��oV�������������w?�F!d@������K��u�]���466�^����o���	!��Up��9F!�K�V��4f�����J���|9X��H�v�Uf���0�?u����-�7$����oV^��e��
��l��4����2o����%��
�������BF���:>s���?����&�0S__�r�E0o�<g��9s������E�={v�J���o9F,l%���6��BH~�����P���~��r�B�����[����Myvz�"������BF���G�����+c'���c����;/G3y��w�����>������>f���/�}\__`����x�
EQn��F���������	 �;������@�A���(���l�J1+l�L!����������*x����XV.����������?���B���UL�Cd�cw�l��?;!����?�������UUU577777WUU����P�1M!��8y���0Sl�S�L!����/!�b�4�������!`����������`C����
'�VCz�I7!��,�-}�����kBi��� ���#w�F��y�����S���fM��	�,.Zi�������M��q�=�����A�MA�S��$d��&�0w��C�555�����+���ZZZZZZ�2{�8}�dW���n�B�����XV.��r}����\�B�p���i�s�����2��4, �SF��3���4�2�������F���v�e�����H������N��Q �S"�k���{�=��v��+8��y���\�;�����{�n7����o��������Hd����
��3b-;q�~	�����r��I!d��}>�>��{��iK�#�qH�#6�)�$�2��[q�����rz{&��#kg�E��I]��
[���]r7������ w��|y���P�=��#�K�,�CR�������B������Y����G;q�2�B��+��s�����a����\�B���������'�I�b�H��X^���8!��z�#���P��0n��92�.���WB��/��B.4�1!��Q�Q�QaC�V]������1�n;��,���	��m���/�������H����}�����9��BY�2l-_�H��2��R�A'�;��/���Y���b!d��{�B��&o�c�L��O�2!��3G�sNs.A]�*kl�R���YVX��Y��\�k��E%�$�;jG$Ua/m�f�3���F��l����.;�!Cb��
.���G��;v�p�\���R\�w�^�s3o����>nll�(��(���l�7m N��'��B!��"o�c}����\�B���oZ��Q��:R<�)��@q�b�8!���b��8���b����l�fq36�2h��_/���o�CF�,Y�2��-�����%t������C�
Kf�M����Yy���z+G	!������a����\�B���oZ������/����x<�2��8���<h��[���I~��oS1c�$c��Y��@��"Y�n�l��M�s��7Kdq�q���eF���c��H!�8����>Gl�S�I!���n���22�V�-HA^+�{6����,]?���evqB��_��
�_�|9�]�<���e�0b���Dby�,���d��N������f���Y���V2�V�\	`��-6l��?�8�5k����C��������+����tv�����lAQ���g�8���g�k�;N�Jb��a�w���VIA�qB!d�(?��O�����,������/�\�)6�)W&�2�|�b���8(GS��N��	!�������1�)
����$GS�}g~����d���<���>~���\.�<{s��w�8��8�1��>f:$�}�r�UUUZZZ�2������a�����e�U��Ya�eB!�C�?��~\�f����F�eE�#6�)�$�B!�2�[��� ��Gy���f���mca�v�Z�j[�`�c|rF���,�����v�X��B�6y�a����\�B!�B����`�W����>���+���9i��Sl��D�Kq��5��a�
B!�����/������LG�W������O�&!�����_
���8�oT2v��k�)�[J�����[����}g��}�&��8�����f�� SSR��v�����~Z����u�:�)}y=���1,o���Y<#V�Q2��+td~�����<bGv����	!�+���r�\kk�0.9�-�L��O�&!�b�}��� [f�Z�Hi�!��a�f���	��$��R7������A�vl����v�-r��$ke����������w�}u������aX���f������<��U�uy=��f����e��_�����������]���d���la����kP__�`�����o~����������F����$�z����^��cF:��%?�,��R��Y;fqN��G�k���8_f�!��E��	!�8��U��=��%��e$��#kg�E��I]��
[���]r7�k�����!^^����9�.#������+���o���S�����e�9X�f�2�d�9��p�B9�;w�`2s�H:�f��Y��Yy�8�_��0v�fe!����[��0Sl�S�L!d�sVd��b���f�������#�;�4N�@0������<T�1���p�������kN��]y���r���X=#���r��E�_$��!��!����>Gl�S�I!���=�so���!�;���q}���FJ�Y�-;��g��`�_����9������H�S�s 53�P�[��E<�q�����~���N,���X��7��~["�&�u
�qR�A���8�����<��!}S�|����0j�/�����b�X�K�B�H����a���g�������=�?0���#_�:�9B!�!G�g<���x�!wl�R���YVX��Y��\�k��E%�$�;jG$Ua/m�f�3���F��l���1��w�-���y;�����f������e�a�W��>��6����U67=:�[������vo��Y^�n�������(������;x�����\�[n��G���].��(�������g��,�[����*�B�@(��{7�q��,��G^��>Gl�S������o+w|p����t��~�u��g�R��rp������9�������{���o�~�k&���x9����f���U����~�KYs*��v��LnH,���f��y����q�]��#�&��ZW'�{oW�3�s-�/,Z���Y�������Ms���no(���������?)v���}����Wg�V�������������]��Y�%���w�fF\�76<� ��������7*0�"n��������,����n�����1���YI�����B@�73�o����8j��q1(�_,fXq[�r%��>����<������.{���K�����x�
y�%�G���K18B���z+G	!��A�������St|o��u�f�
����h����}�4$~�H�8���_'���e����\��_?�AU�M������;Z��dC����o(Y�2$�����z��'?n���+�C+���7�qn�7+���3�!��{S�30k��v��N}U������	��o��7?;+��)��k��o�L�o�������P������$~�h�\�}u"a/�*���~��GW�H��	��Z4qb{����}����o�����M|������
G���y�;~��_VX��]"9�����|G���Z)h��[���@tC�&{i����k�Ny�2fqG���$/��n�9DN��Yyq�fe�u�V9d��e����E�������;f?=G �saB!�
���/+��k��8d�)6�����L>�P��'
���f��H���G���?:����{&�w"<v�5$�wIG�wu"����2L�
�*���`�K��j��Z�g�5X*xq��Ei�jC��?�b�$3\��-&}�f����)|-�j�O
s,?���������9gf�>��e�k�� IDAT�|�>��s��	���=����	����?{�����M��?��<[bFd|�b�e|�a������5R���W��?��xS��������X���?��6\��������L����&���������=2�U1"�6��w���wlZ��3��/�`�=�DVq�*o9�7����M���[p�������������KR�k��q�������q��;��q��G�����'z�-i����b�=��R�/>���x��_u$9���g6�'������w}0�OA��,L�V����~�#o�|����{?:t��xX�V}c��������l�QWa�#�]���=�a�;����_-�x����Ue�q�E�>��7��b��&n��<�����0oH3�x�����^i?����m���g��)�g/���f~x���|�����Ca��6��l:��>��x<��_��f}�qc�jR;,���#���E�A��jPN�����?���J�j�������mq/��'������---999��+������l6������Y�����#�M����������� k\�H
��B!����O�P�>����>�����Yfb�X�����>R~��=��7
���>R~�#9��b�M��pQ�#��&��G�����vrS�~���b��G��zy���>:P.u`�/�0��|�p>:P���3��J:�Y{�=g�w���X�����&�y$q�b�X���T.%$����A��q�XL?nR����x7���xO��J��X,�s��I,�=�xb��&����.1��.DL��u;�4������������"'�I��}]����/���nI-����+�����T����[���X�����������?7����x�>����[�<��3z�K�)B!��Q����?�����������-3�[�r��U�V�^]__/�OKSS�����m��;v��hmm���-���O\�������tvvvvv�"���2�"������@@3��u�l�#z���	!�\��o�����oW�|���2����>"��B�����?��K�e3^�M^��B!#�����-+
oN�Y�&yxA��y9�JYY����
����n�?���r�dMMMv������,�_���<����W���1�}�O@6�}lp�RV�~1.�1(RG�K�zGu#�B.W��o//���fWW��o�)G�<���=G�h�r�_�K�B��=^s�"�B��z�cK������z�d)�l�{����q�
B!�[��B��?w���8IG��&;��IY�Ky�Ym_��d,5e�2/�^Szq2Y�Y��P��\S�Q�Fu���0]2��n��Q��>z4��G����z����&w������+���Y����o����6����x��8�4t���2o��'�B2a��B�����2|�Jcw����^�Ym_�R�A�-fg4��U�G�
j����^��7b�B����#�
7�Tz��J]��1C����I�u�q}gE�q}5��z�����V���K���1��+W466��m��%��������m���{���l���m����Y|��o�[$����wtt 9w����~���dy�@ �G�����S�K/���xD]��,��6+"�B��>&�b
�7��N�o<�a�VS��qr��1�&�t�,U��9B������K��sV�����=1#����wT�o\�*�YK�?@9��[�����Z9��x�b9����CtV�������������g���c&�W�B5�>&�b�GX�BJ�1C�;#�-�:��)h7���O0��5 ]jN3������>V�$��s@+G�-z���F���j��������O���d�~��_�����q	���P���O!�d��?��J��	!�Lxz
E��rl���k������yP�.�v��TY�t�)�U��]
�>�mKq"2M���|G���I�2��1�j���q��J�SA��f5N��:h�cx}�����(��M��)�g��C�q��:�^��1be�eJV�2��H&�2���'?�C�N���_�����/�~,T �B�)��H�����)=R���#������@�����)�#J
��guXe��zq�~/4/�����l������>.#��C��@ ����|'���LM��+�
u�x���H3�w1�>��k6����"�)�vX�O���+W^s�5���999������566�rrrl6��n_�t)����j��-Z�#mmmv��f��?>���>���P�Qg����f~B!d$����c�
_p��?}��v��\�B��qc�l������r�`f����c%��gMI_�^�+��	��T���`w��w���WW����z5���W?
���w\=V�c�}���������f;��'�������?��3���W[[���`�������*l��}����|>L���xKF�G�Xo�(!�2���G�����W}�����,����*�B��?q�7&���{6c��rtt�sd=�7K�nR�S����LV����]����f��;M3���5��/Q��V��Z�\jjj��w��]q�qz���K?5�����y�B!�Mn}}��������OO{������*�
!d��������;�j��������7�l���w���(LS������
Y;m�;��zd��_������&W��e�S?��1+�I�{J��{��*�G4����W��|/M�>oA/����
)��2>�����x��,U?���ezq"2Y��d���E��H�k��E�x�V}M&�Au8u�o�R��hWl*%���WZ������.<�>(���!W���f�rEKKKNNi�
=���l6��f����q�;�z�<���K=85G�&�v4�|��R�H
��B!c��v�����g8~���^p�}���r
B��tr�����)����~���b�l�e�jG���;^�����&�#��
�uC/n����g�r�Mx��)r�B!#����Lw�����o��	G^�t^��RVV��>�����Whjj�>�& ��>f��|�c�;������	�g�5��~��f��k�S�b��H�cf��H����M!���}����r9������7���	G^����r4]����'���������M�cB.�����sx�1p�p����������@�����5���8�!�B&�W_}��} e��[�|��e���E���rT[��'^��{�V;j�@�u�B9����n��c��s�����<w����Z���2Q�o��}��ZSVO�����v�n�7��&����d�%��`upr��i��X2����#�?m�/Qg�����Y;���O�\����[�c������Y;���o�g���jG�&v\�����%���d��q��NW��z�u��P��p��X'w.���|�.��'����huO���O��^B!�d�z�cK���*+++**��+��c6Y���z��q5��u���!���!������?/���,�bL.$�LP������\c-S����y��U���N�����������|}g�^���y�>>p_L��6��3���w��jpbC�s�;�Ne�h|o�����r8�sY�k�X;W}��k?>P��t����V}T�f��_p�������r?�-|���x���7����:��U�65{�u�.�A���U�����
8�v������xPuN"�K,�����F��+�;��o}$VV��]��&>�������|�
j�H�zqB�����@���r/���?R#H��'�d��U��_�^.���^������������5���������2o��'�BR��_�:u���?�\.��w���_c��B��rY����{�_�������������w��������8���yC�]�&����%����mM��;u�����l�y
���
x|_R��}��>l��wf=�!1�����9�I��P���K�:�~�u��P���s���PF-��z#;������|�Y".�������@�r���U��X%>_�A��N$i2Y
BM�]������[�#\�t�|�Y��������R�	!��r�J9�����ml��-�---���l{����w�f�mmm{���}��>��o����|�
1w��9�~�����W�O^S1������s�"Q��+�q���!��l6��O?mi�������"9
����iK~va�'r	!��w��?�o����g�y�7��W�~U�[w����J���,��?�Yj0�������b���?m��7�\�s�f�}��P�t���?�4�GWr?�1���qtOP���6�z������X��<.I��D�D2�AZ�3���~v��?����g9B�?>�SZ����w���i��2�^}�U9����V%,^�X�����D��z��}��������3�Ru�1��C!���b���q��i��PQQQ__����]�s�.^�(F	!��i����N�a��K���{;u������^����v����'?>!�,������nG�Cd W`e�{�1F�1��T�z��D6���[6��A����4������d������BI��P����B�����_�M�����t���n��!v��1J��n�7����M�����Uw����7<�ro���o�*�;��������q��������N�B|A	X;�g�t��j&QB��u�R��+����Z�����}�&�N5�:��coGjJ/�>"�)�g/�����	*��@��8!����{�}�B��L�<����b���=1�fry��d����Yq�2a������S�������}1�f5��OWH�X$��p_���'���z}�p}������o\����������z=X����*�,'�:qF�W��"Se����!4�*Q��{'\���� K�N��
�6�XzfsLf�MY�I��x_�>����y��	��1�G>���	!�������^z��M�6m�������W^+l��m��b��}�����_�Z�������=�����h�ztV��.�B������.^1y��H$���+sO�9	�h�1!����z;����{%��A+�s0�8��������#�_?{����iw���|Yk��k?��i�J��z|���h����[����;������@�[�0{��o�t��J����Ww���M���?�xi����p��k�.��	~F�m}s�OX�X��OL_��:����D��	���P)[��t�.�&3>K7�����~�����R6��Uv
f:���kJ:�f������H���qB9O=��/��G?������
[��{�Iz_����x��D��	��%�B2c����������X�+_�J$�����r�o���cB.SS���x�����{L���W������oWa��'G0{���J1b;���5�����9�}9��1@u
I/��]���7s��>t����v���+���������S��4{��|~q|���I�r���~9*�9�FR���?:��b��$^O��O�8��RV�
P}c�~�WAV�KXMu5�8!d����I���/_.�LO{{;�����O��1�G$�B�b������w�}������A�F�k����/�����x��n�K!W����g��s�����%�d�s��zg���T�wG��^��y�L�9&�iP3��GG�!���`����������T������	Q3?<������J��E�����z�������	!������Ge�MMMv��n����,[�,�����6��n����?��Y�����G���6l6��y���?R~&�O$�h�Y�5�Y$M�	!��q�v�W�^G!:�v��n2�2�EG�N�;^��7�C&F���\��~������+G^�m�"�B����)���@ �����h����.����;vL^x��������,���K�m���>nmm����-b/���TWW��o��f���;YKWUU����{�^~��:��S�R\$�1�m\���U��&�re��}{yy�����u���jG^���������_������9s�\�����	!$+�E�Hs9��PW��+��	!�2���}l`���l�qJ^�����|>���N�g��,AL!��&3kV�A�cB!�B!�!M=�j��r���joo������`S�}>����jJ�8�!��JB��?�����8s'�Ay��5�K�&i_���?�bi��� ����.��a�>!$�����::::;;�����+�^������b��g��U�����n�B	�>&�21�O\�@ ���p<�w�d.��$�~&�pwB,a�M���7���j�S���tB�d�?�<���i[��c_�������v���K������������}�����mmm|�
1w�TVVvvv�2_�����;��|��^�S���x<z��q��>
!�2�(}L!���#n\�O;�L-Io/B�����:�.��;B���SO����zH/^,�$�7��z���+���O����u�:�����!�B�����T�����B��kV���#O<����H4)��nJ����N4�7��4O��l��5��'�N��}�B�~�����t����i9F!��?�����J
��l_��:(�,��qy�Il/�/�
�}�q1.�b^cFs��������>!��'!��+M�����c�B�����@@+��<��9u5���w�N:����l�6�qc�}B���+�����������������	9996���}�������E��H[[��ngk��?��S���H^�X��Sb4Y=:�+�B!#�.!�C�aF�1�����5B�������[fA~�V_��clP�<��z|f�Do��l��q��
�y��,Ymm-�X�|��e���UWW/X�@����=/e��G*}�B'h�cB!��D���Tc��;/�x&x��z�fAIXk0�I�ol��7��������u?���kG/��~��:B1�������#���w���1�;f<��f���B�
JB��������bD�E�^�R�q#�F�����;.��6��U=����^���m_��"|�bFss3_����%''��x����v6��f�UTT�8�{�^a�����.����gA��f�4'�B��-��1B!W�������+G^�m�"�B����)���@ �����h����.�����������v���>+�H���$�}\SS#���>f���a�c�;������	�g�5��~��f��k�S�b��H�cf��H����M!���}����r9������7���	G^����r4]�
��Jz6��D�����5�S���[ ��"�sM�����5������)O�����A�R+���:H�2:}y���5'������4S0������2"_��*��80��7I�XVZ:p�7+�sM�1��������#�����W������>���UrB!�x��W�^�@��1�|�r���r�����+�R�7o^ �'m���KB!Yw��)9��������us�����J��Z�������u�����=Ry���g6�8�O#x����KP����U3q�31|��S�Q���(^^��u�� �w\��w�pG:��B��hJ�}�;�4�N�f_�@�6���$��?2��=}f���6j�����w@�D!��i��Ul������|>��'M@6������B.?_����P�&���g��N��m�M+�m�M)<�s0��1�"���8��x���e
�86s��c�I�F�{�Lt��{�o_������{�M)����@������3g�:*��k4���9&��2Y_�x�L�AV�"d$L,�X}KZ����!dt�Z�
������Q!-^�D����o���Wx��8f�EV?X�����n�B�Kk�F��������t'Z+��LL2d�8�����k:,�=}fs�<�O�g����k:1����[�V��,�����%3
;������pm����?O��Rg�:h�\�9�������Bu����u�T]|mu��.���d��]3w\���5���Bf���O�.l^�1s��	a�J IDATc��1��7���0	I���:0�S�������������L"}�q���%����iE����O$����?;4��y�����})(���{��-�=�)�OA�	�������4��j}=lG���R��?l���f��-��\�8��K��j�>c�?����r�J9�����ml����_�����{�nw�}7���6����|���f����]UU�����s�������<���`��G��Yd����zqN]�#|�����#����(�B�����V�!�s��O��+�;��������k:�D#�ssGa��:��*�k�����p��<��uO�o����'J4�,���~;���)O�[O�$v��;:g<�n]Ap��f��2��dF��3ph�=��W��`<x���B�&y����
���L��uK0|���)����a?�I��)F�r�],��k�W�����
�2�S�����q�d�0��B;�N3H���H1� �#��%����p���7K=t!��
�iJ��33���N:�w&����n}U��/S
�O�
���O�{��N�+��s��1�O5V��LV��_3XeK=��>!j|�xi6�oC�d���CQl�c�jkk�P��������Z�Y�B��3z�3zq=z�����Ta����B!�&W;2p����II`�t���9�����V���=\1#s������y���uK���s�Oe,L$_
g��@����I9)����-@�����Wr��p�'�L}IK�0~�D����;{���si���BVz��pP����~��`w�`n���g��S��S�b��]���R��R}�������s�y���c�
���Mo$���hw����|{��{��@Ga��1�.�,���w�a?M)�|�}?$_yF���X;��kNs�&Q�'���������KL�+H<����FR�Q'f��tF��x.f��7�7��K���7�E���K��<.�K�E=T��x�!���?��~OF!�Lk[���1S0�����O�x �y�������yx��q:I`��^7o�L9�?������z;^I�~�,4���7�I��3��T�Y��q���y�;����%B��������W6�mm��U�_�d[&���)��g�S���M��Z�:�����l�S���m��U��e�5����4�������5!����%�r����`J!�u����)�R��������'z7�{��p�u}�_h6�{�8WP2�0{�WP2����KT�CA���FI
Jf�����1+�'[0��N������c3����Si�b���q���nJa�K��'�Au���j� �����hv����{����L6�9��d�u�`nE��;E�f����D?M*�[Q����h����uD�[o,������dA����K�����x�':VY�?�������x�B!����y�9����������_��m��;����z�-�Bkk���k1����w��}�����O����>��*c������?���B!������%OT6lN�����l�����;����k��
++g�xre�������:o��5k�������������yM
g�,��+�[Wsf��lp�
Jf�rb4/iA�tg��Jf�w7�$��t��RF}��?�V>�L5�S���E"����q��-�[b������5���A�vj�Fc�8��v4huuxy��5��0���L�J���BL���j�,��M������=�����_R7Wc�,�q�y��et_����?��c>��|X�Y��z$�$f#��L��%c��q��BRZ�z��R[[����}o^ss�\C��}l5w����Z�X���n1C��!�2�l�HD�����
��TXH�fI&��d�Q6��J�k,n�\O��_���	��6�g$���(�<�����:hf�SyD]�I��Af�dX�G1���7ST?��Y_]Y��TS��:b�R��qu�2��~Ko�@ �����h����.����;vl��U���999v���g��k$kjj���>� {��[o-]�4���>�?�\�����^�\�o}�L��B�����������������S�&yx��G���t��>&��`��������h�����;�&PWIv���x�Iz��$f��lC�/>w�������������X�G���5��q5��Iy\��Y�'H.')�'�>ZxDM�,�������B��X�r%�
6����e�477��qKKKNN���h���zZ����l6��VQQ��z�c�O���.�����V�T*��&��B�����$C�6w��m��#(��f��(�@]%�5��>��c9D&2��+�	!��+�a�qYY����
������c�����c��jjj����V���4�������tvv�d�������l�Z�������:f������z��B.o4��\�
����^oyder�L�e��$������Hz.%�b[��m?���)�@V�>6���n�����y���@�O����d�5��1!�2>Q��B����0!����z�-^a&w�IS��b��%^�����|>i���]���'� �B��!��
��#nB!�����N�~��~��G�~�� ����^�x���3��B!����/����]w�%G/;��4����f���;#�����=�����Wz�`�B�j<�}|��1�W��������������v���_��_���������������o�����<�������^��X��Y�w�q;�qERZ�m�v<�o-B�b����BF���G9d���|$���Gq��j!��+��
�����Z9��x�b9����CtV���1�������zqF]��0f�zu!��Qc��B�!��� �B�2��N!�\r?��S���t^{��B�)-�1%���P� D��<���D�&z����������{r���|�;r({��h8��	f�������'���R����Po0�E����:�j���L�������2�����������'E���C��
�����yr����O�}
QY�	@��8�DF%����r�)�~��<4���r�%��t(�%Ez�B!���o�	!�\�rKK��x�`5��D���E}�DLQJ;�#[>]��������.����q���p8,^���C�k���WVV&FE���B�5��������#��y���_z�rk��������e�����m���C=yNgM�ze��#�����-O�����]�%���'��w�,�s����C�#�>�=����*��m���d�Ihyy��|���E��|'�Y�@|�!��oONg~q�W���kd�����O�&��|rV�B!�{����v{NN[��������
��m���Y�l��k/Z���Sb������hl��zt>[��.�B�h�U���s:�.�+
�z��/�P|" ��P�rv���z��� s���Df������	��K���.)	����5Q%
��aEQ��W\\�v�S�+r8B�x�x0��;��U�}���]�@���3	}$_[^�����b��N���o+$&�����fyG��-����>	@����p(p:�]��[2j�1��h+..
�Cu,~��328
_zn�\O���O��|��t��A��uE���UMz?�9��|�K����	WXI�f�5E~�*����`��8
�H���ED��J��a��u���j4U%�D��H$�(JOO�\;����n��HA��5r�RR8���o3i�cB!����SO���~��hhh�k����[[[�C0�;��x��D��	��%�B��\��gKKKKY��j�����*�+��.v��	shGKh�S�&�������,+/?lb*�����^�W*��!'��aEq9�����r��,��3�gQ-��8���P7%���D��`wW~I9����)/�,�U��	7v>]^��S���0<��3��2�p��������n�����%W�;�DGQ�V��ta��N�zOG�3��(���#�b���K{
B�K/����5T�;?<]q���V<�0i��s����-F���BrH���%��7�i�S�0iKw�8��"�`XYSa[��+�s�u�O��������#n$� �B�������W^#��/_�����r��f�G���B���_�9�eee===���qy��G(�D���0��� ;�NETQX�XQ���vM�y������i�<����� Pn*�5��E����I�U�JY��Yy���Tz�y�l<�p?�@��i�)�|�1���.v��!_�g&}�r���A~��un��(��B��==��y���?�*��v�Q��k��p���h;�iu�& ����
�D��*e�[\�+�%��E�W�������t���r����?���<��MCVE�H�N9�&�;F�)cB!�3��MMMl9i�
=mmmv��f��?>���qee%�0>��m�l�y��%Z� �����q��s^*IA�qB!d��e�lG$�(JQQ��>�������MX?��� CQ��0��I/'�|�<a�0� ��&�i�H��� �|�PgOO��,����O>�����z�b^'C���������{&��
�������/�"O6~�Rm:�PE��	���(P���.W�K�R\\��~�����ZiG�z e���VU#=�������5�\�J����V/vw�[���5���	�a�*��	���D�zN]r�L)����;B?V��"�E�h1�5��h�}W��Ml��y=�(�kp/)�� ��P=#�P���op0�$���f�_K��$������gg��P�"?k\k�>���,�'���c�B�Bl���n����z��m������6,�o��>���;��+	��{����~�_�A6����v �n��{���R��c/���x�6!�2�/^��+W�(�E����/y>� ���$]&s���cQ���� �/�4����Ngyy��B������|IYw������+v����l����e.U�W�f�(��X�X�N�0�(��r����z��O�������]cgCWw���Y
�Ng~O��`	d�cF�
��r�\]]��z�-�w*P�K�J8A,-/���z��q��O��������n(
m�����7������8Uy�Sy`��e"EA��~��`Y�K�*�h����t��

��� Q����E���U,7���$�P(P�}�����_���U�&&k6���1�����������4��j��>&�B	��>6�w�^6�8%�����|>���M/q����#���BF����'9����fs�q��8���}�]�����
�d�����cF3Qe���g/�^����5�����^���r������=��n�������[�f7}�^&T�g�Sd`��uae	Nl��P�r���gAmE]C��P8T\\��p�D������H���
����l���`WHQz��\n��U���*@(2^��p�1�������K������k�W�F#��'��\�<'������2����@T����'?�Q���B������$5�8��CD�� "�nV�����<������>
�B���7����nK�3����V��`�M,���5k����������Q�����nI��V��x�S���{���c�S�	!�B���������cIUUU{{{GG��~�����4�����q�
B!W�n�A�kx�c��L!^�@�=<�~���������Y��eK�y=�������se��R�<im��)�HP��.p����\������,--s:w�e���i_1MF����v&��������p�;�_@�|������)�;�(Q���Z�G1���p8�/.
�C,����o��c������h4�(p:���/= ��x������8���\��/-�~c���h���xOK���.g��U�+J�y���u�
��k�����yl�3]�a%�{���W_Q������A��(,�������;???�b�����������A��h��W{E���9<�`�����k`
�}�^�"��23�`��nc\?��Cg~P�Z�M�(37ele��B������f���WTUUutttvv�o���Wx�^����4Lo/�F�}B!d$���n���9�|���o\5\K�(�P�%�|��z�H�
m��pD���S�F��i#.��K}{�v=DQQ��{q��z|[�8���/�"W��O�x�\Md4C����T/��t:C�Pii�2�A���L�f�-ETA�K������n�@����H���lh��>\�.�*�q�k'����Z;g������78���]s*����[QW?�t��h8��k����{����7�Q������(���D�t��r����}�y�r:�����7���4���^#}}�{��5���k��1G��D�����@_�s��8��|����������DC
�b���0����r�f�	��hS�#�=�e��Y�|����#����B!c����g�6mb�Wp����vss��n_�t)��������{7�����������/^�^��������g����4��1���*������H�����������B!c.�>�s��'�������{z���y����@���)��Y�}�'1���J���;�^��J*�g�4s�#�/{�^:C>�kW�=+.���v�����Z��[S�my���Z�X����1�hLB���������D�p��	���X\O�d�2��B+������5��>8P��=P� 
EE���R�;�J0�[�Qn���
����.�(�P@oo�Y�����������5��hWO���
��m����g�]������fU�rm���}!(
�@g4�����.�:7.�X�=�V������bW���lD�j��;p������"�eE!�9�]I5
d�K�n9��bkwGu����qs�%��G�0x���j^��Q�_���gV��f~��B���SO����yj�u�	��:�}|�1��-5V(����3�Ru�1��C!���x��������.���m./-���V��1(J��w^E 
(
(��8�����y���z)H3	;VG�����/�:�����s�|��w���\E�][4��EEE��VVW�mq�Hj�{�K�Th|m��(,e�(���b^'�e�}�;�I����c�����y�D9-��BJ��rp���EF#�A�"a�������4O��US��U�#����Nv�H�/d5=���7N�CAD��*V���(����+n�2�7~�F����5�L�F��.�~.�YiD�>%EQ��#�����w)�w�!2r��l�Yi�i��gi�_��������t��p(\�*1;�Z'���U|rVc�
�df���J�����z������\�^g�v�Cs�����R�������R��f;V����wR,���}u�B�J��D!�JO����Q�������=�-�����?��*N�D�P�P�A(�kahZ�=�v���_������.,L\���h�vV��k���6<���tA���c��=��(����A�Pv���<T��$)j>������Q-�����>Jb�F�Hd0<q*�{KSLj0M���"�
�`�������D��u���;�P0�x�����m������\����<����F������>���T2o��Pj��E�H����P����p8Q"�J]���HQs�+�/�<)��k�=N[�8���y@�-\�@�O��a4
+Pj�P�*��#��uRR�d�x����@  =����)NO���}V����I��S����^�w��9^��7"���A)���>�A}�R��J�}�H/N!����C!�JO*�`�pO���w�s�e����LMId��)��<"@JM�KG�� K_X.*R��8v�`�E#@Q�XAMcr�K���dC6���z�v����R�H8\__�p��L�s#�!5�W��F�.7��P��S���1��}�P4�P��p��]��r����N�F}{���EUP} IDAT��aT������-M:�:?�4�{]�]]�M\`O��[����.K�O�sj_�lx���Wy����],�^*���EA��.u+P��T��"�*���F�`�=�p�!�mH�o�G\��`S�-r�����>�(Id��9��Xm�j}M���Acl_����k�M�����	!������n���������v����������
��m���Y�l�����v��E�x<���N���Pc?P�*�k������]!���O/+_���Q^Z�'�������e��5�����/�J�1K=�U4VYN�{��g����pD�
��,E�Q=
���'{��4�H����ruwwG"��xl�����s�@qi��pq]�2��USS�+��y�3�����q�Zx���h��*}�{�Sd�-��X��m](z�n�\fZ�~'�����+�[���"����8�t���U��.�{����97��d��._"��F
���8��|%���N�w0������������Z����U�����e}�1�7���'��{f��V�3�?+2����I����[�oUz����\B!�x���^x��{�R4[����U.0����c$��T.Xo�(!�2n����(w-������y����-|�8�C[���2���{JO
�L����YJb�����E���(
�v�@PT��p���'��i�(��Z�d�����\+V�5R[_U��7z�U�?���EL��}��� ��������7��[8�8���:83kgkZU���-�?[SJ�������C-���	G�()V�\
�h�Rx]Jvs��S�/�Htx
EAT��{�p0����4bI]���7����2� ��q#���o����O�x&x��~�<Kg��2k�B�RWW��+�������/����s�c&��z�2�G$�BF�M�[�������u�]r��3�O�z�9�a����3{����~�Q�������Ly���������"�!"���[�l��6�h����j���$�K����:(�5����B�?����/�
)(������:�I����$6E!W��o�0�9�f9�p���K��h*���C�^x!''��'����PW�1��������`Z[[��+����v��f0�|g����*a|>���l���K.L��0����q�'��H
���D?	!�\��o�^^^.G�uuu���r4���~�sT��+�x!�b�1���WM�k�if[������;��m�2���L\����A}�����f�B�8�i�&��.Fz�!��m�z���,X���}�>���;��}*F"w��z��~����ANIL1���e��qO��f��������R<^�B(}L!����H�Q�������J;_B!z��}l`���l�qJ^�����|>���^�Xm�~�Q��B��E����8�S�l��I=��0s���)r�����1o��<��+�|	!��+W���>�TUU���wttTVV�q6����i.^!N�s���BH�r�BI%�G2�A!��-l�
1�>����6�������G�6#�>!�2���!���z*���y�B���������}9^NN[���:�f�����K�hii�����(������W�q,}\QQ�^�}u����)]q/3qE��0+J����ME!W����y�>&��+�W��U9|��V��q�P���B!d������R��B�NJ���?
��q�=��cZ��B�\_��W?�"�K����B!�7>eX=��B�,�j�;#c�/���D2Ac�����
O�B!��[sB!�@�h<�rS�2#������7l����@���h,��� �!�BL�������>��\ hlld��>fZ[[�v��E�x$���z�����k�E'�]\Y�B��\�b4�~��'r�����g�^����_������L�X"�N��f3��
!�r�X�z��R[[���`�������*������'b_��2w��7�M�����QB!d���B!��7�jB!�2�j�*9dNMM�[o�%F�����/�c>w��ay��H!d�����(}L!W�s�����r�B!��X,&�l5	+W��a������&477��+ZZZrrrH�W�ioo`��l6[E���RX�����#N�U'm�g������ k\�H
��B�2���+�r�BI������A!��+����P*��m��c�Y�x����,w\UUUYY	���S���������v*����v����f�������B���}�����
�*7�t`�5�u��j�������_��-v�"�)t��%4���Wv�?�����^�!�B�V�����
���>��3�$�k0������v>	���y������\��Q�YcB!D4V�5�~�_��_�e��G:�����
���644���������.]��b�X���K�.]:����t:��>	��?N-�y�k'����r��;�R�qWK�@&�#��.��}����n��!�B^}�U���0�;�jjj��UUUr�z�>����I�W��y��_���'� �B$�}���]3�+��.���������))��$*��
�<�������	���j���s��������������/^����o���B������k'���������qc@��-Q-u$Q1�-@�3h��(�%�]PG5z+�|����I���T&�#����^���q5��5�`�������*�oAo��C����/f���(��!�erz�b+W������������m6����h�q���Gz{�7��B!���/�vC_^�p��ph���s��<��6���'��":y4Hd�L:y���!9:�x�����
����/�����n��7�q���/��RQ��3g����b1���g�<�����q�zh�#i�k�x��`������ao�Y�#`��8����*@Lwx�������z�Oz�}�K�z��		�����t�c��f.��@&���W��g�}6�$Icc#���m[r	ZZZZ[[�����w������������?�������H�;noo���@r��aS����X9�����QG����^zqN]�#�Rue1��Y!�2&r�_^���/�v���K������I0����8@�4i�����
{�4��[������g*�x�������7����G���8v��b������}y��$?��E�7�`Kbt��:���$�~��.����XCJ�O�#n���FN��F��V{h�>!&��e~�gk4�?bzF�����+L���a��R[[+�/^,����r���">�����������g���c&�W�B5��O|~�b,������rc����s�&O��m_=i��sCC�#��?�:y�w�/��d����Dx���[�N��������)):y4���C�3����H�^���Z�m���_�wGq�A���)T]C'�����3_��.��������1O�tuu�l6v���K����s�����n�w�h��8s��)7�6��t)���mn������������T���������`t����%��]�:�����o��M%_L�
����������rA2��f�H�1��t���4����NJ���K5��Z��Q�7���L}3�9[�[�KM�N;zq��"1(�	!������B�D�����]�p!�OU�@��@:]}���`�7p��N������;90u�����O����������q�����x�_���z<%,i{r���\w68Pp��V�*
������&�����'�	#���*��x�����<�oe��\�x��n~����[�b�������;C�����������n\�?C'��K������A��W'������z�4��,�zx����+`���X��+��T��\}R>�[��x<%��������*��.�]���<{*��0�9-�g6)�*3�AGr;zX���q)������^}1(�'�<�c��F>&�?�8��hy��������}�q�|<���^�c��.���	!����&�B.��.]�	�~����g�����f���'M.�z��3�����V�T�<�����S F&]W0y������R�R�����/~%��`���x}$���@&\7	`����L,�c��yB@YY������[	ue�X�R���)n�O�K��Wy��t�0t.����:�q�r%�QKYqe��a����%�J�D��_��)
�.���G��,+����s=�����v!$�#9+�0+���_��y
�9��0>5��5n���`����\sMnnnNN��nO���=!''���d����Bkk��n_�h������v������y<%������{��Wf�������B!#!����b�`��>;s�T��X�W6Eb�I�#b}��*��
Sn�-��$�h.��F���d/^:����f�PVV�����RFO-�T���kmHdn����P���x�<%I������D����Kt_F�@���)�rq�2�b����F'�3���?�d������^�����jV�g��$�i�wG������;lI���vQ7n�C:jkk���,_�@ss�\C��}�o�>��������1������%�BF�����8s�g�1r�f�]�]�
������+��#)M��!��:;�j`������C((��5��9W��q�:;���W��-M��`����������u����oJ�omsmC�/�b���Q%
F��(B�
�X�)q"D�v�����b �5"(�E3��7�U�S��� ��q%�W�T���>'B�X���3��A���L�|�9�|X����{m�M�z���@�|���~���������rw�5�@���a��e]n�������a�����.������G����=z�\�)��a,�_�n3+�sz��
���5Yy��-��W[O��F��/-k��������%��`�T�x</k��Yw�/J��Y;'�
]�X����W���t{=�
��M'�	�����?�|]�Jd�L�+��������H���8���z����Z����'N(��_�<��:�k����D�����+�{����I��/}�����(������M��'�{�~�5R�w~����?�r�����[��Z������=��g~z�;���������>��w�������P����w~���X
j�O}��/��s�/����=t�!���}��O�������������O�d���~*#�=������\���U�w}h4G���"�S!����[���c~���NS��eZ���dk��H�/6�6��������.�/���|[��/J,�}W�o���<���ib������F���o��/�6��C���q"��������X�2Y.�A�|[�"(�s���Oq�t���JS�|%Y,�{#�F�^�*g�khh�L^�355�0�0^z�%��P($#�|7��
������6.��s�&%�2NDD�������b�	��\�������������?��<���n-������&<���G1���~TLD����������{Epyy��O?�O��?������������������'���.�MS��<q���}�T����dS6��l��m�7�����/�x����}�{r������������y��?�}Z��?-�@DDD&��V,w��r������h�������/� �>>w���lq��E����d������B�},j����fff�
�m�8���!��,���f�7��M��.Y�LDD;��+W�~��������+j���_��?��F��89���n���~����������������T�yr��={����G�����,,.������B��JT�g~v���O������u���Z���l������y��#���������[�����o�w�#""���J�����g��}�[�I��>v055%n�WQGGG.��9�K��Z�c%
�DDD[Y���jl��z6|V
�h����Cj�������P�j������i�T�V�<Fj�Zr��</�r�u�����������DDDD���}��}��B�L&��(�������b������O������]f������������y���e��������[�C�/����.�o#U�"z��b19yS(�W�TA�f���yr������}JW��{�����h=X>&"�m�|qe� �[a6�<��8��[�P��,�k�������<9J���������������i�>���3gT��bddD,\�|Y�_!�[��q��W_}�������?����a�������SSSr�
s�X�B�lVV������)-?�e���������IF�r0��c��e�V���6L��/���h������N����������L�_KT�:���W5�:J�o"ylm�]�����6��c�Z�����n��q��f�/�-�WDT5�w�QC��i�q4UC%===jp��15@3s��z,�>�tq]�..X[��M\�CDD�a��������1���}�{���u!X>JT,�el�F�e\7����G�A�`m2uq���q�'XzR�MJ����._�NN_A��u�;)"��*����:7�)��'"zl�������h;j�GmS�z\.�S����s�~t���zn^Q��#X�����Qs�h2�.�s�|]�������o������m���x���RV����L�A�]� �6FM�e>vq J�5ND��� �T��~��#""z\���F^��4�#u9V�?.�]��4UT�*
�u%/�KO������U���u�='"����}>_CCCCC�����|���}}}2����


o�������}>����e�"���lm�G�����Y��m�������O<����{��������w����W�$"�:k��!5d"/r��������wXb'=��i[Y���,_�+������U}����e""�{������@?��>�H��s_�vMmp4==
�c�����O��$�V�m>5�Q��cQ)���5��hS�+1�B]mdK\s�Q��z�������S����d��������<X�3��N�����w�U"'O�4=���������|�V'n�/,"""[�j`�Xk��Rl��k��*h�@<����GW��q�,����C?�T�Gi��������,��l���e�.�����7���X	��Xm��&�,����ef]y�����}7x��h;������/^�Y(�W�LLL�|>�0=zT�E����#2"d2�`FGGGy��O-�)'��m���D�&%�2NDD�e���Wc�����V��que""�������j�d���X����'"����+~�_��������5Z��/��GU��|���j�����<��2�:��p��5����;�����a��Jq�E������&�R�8
�f��d7���7��M��.Y�LDD�����vK�V�7R2>�Z���-�d�K���qK�J��zF/�bj�ji6`���+��C�q#0��F�K��m�as5W�����l��L��N�{��$%Mkl��P �Rz��hw#[�N���o�*�������f�Oke��m:q�i^ ""�H������/��G�t����j��(�"rGG��v\�`0X�n.��4mS��h���x��~�p���#��~S���{��n����r����k�XT�uq��t�5�"�W���c����e��������q���jEr(��p~��j��pw!���[I��&Q���I���l��6�_O�����X�
�R[��w��}�w��6P�?9H���MH�]�,�Q����,�l��X�����������Fl!�j5'	�g��!O����75�W�}����n7�}Z+�s�h���.�'���=	�m~����������V���d�n��-5�w��Q�s/�����]���|���h�]&	���P����������e��_����F\����]�]c��i��O��|�<�3V����9��5���@Kf���Z2���� IDAT2�������9m:=nt}�`?�K�����H!�ZL[)����b�y�]a_"�Q���H.���}�����w������Q<�bd�����a�NF�5:,�j�oy��b�h�����F-�k�?%��/���p�[�s�,tO�����]
n��n)�)� ��8��79����+��@����Sk�������h����J��y�V�ji������#��Q�?��o:_L0�5����u+����%�j���y���c�>-�`G�]���y��>t��Gv�S�����N�������g��'\G^���0S����eS�NP��m'J\G����R
�Ut���k���#�b��?��2y���������y�<9yE(�f��l�\A��[�����w�DDD�����{���/c���ai�
uN�5u
�Xk���������b��}��>Sm�*�����Ba���}����������bH�|di60��ba�P(L���S�-������_B�"�I�����pw���">�&��)�~-
����_M��@!��W�J�T��'�������"Y�]��X����1��h��O'�_�!R(D�5����b�����:zMT����L������/���x�)��;x�8�]hnj����g��z��F�/Q�|���&}����J[�t���Q�)&���|������������^���pu�����ywC�b��5U����I��(�~���
���LK���t�q�	]?��S��������}a�������9�'��)=�^j�^�+%AF�&q�$�:�q]�.�L�����G�]�Ze�5_�������}���b!�H������������W?��c�<66v��5�|���7n�4�����\;������P=��W�L�])#�����..Y�d��jM6���h"""�R�.���L���>|���=�z����:?��o�VG"���v�j���i��`�M|�6�� 0��+��|Y�knj�]1��������8�G�����^���)�4�O��7�B_q�����+����ku����B ��o����V��_��]0��2��@��D�X�-T(B;x����.1�@���J�?U=_�]�Q���b�	K��J��ES<�F?���9tW�_��]�{x�<����u�[��o'�I�8�}����P���O���,����y��m<�>�
�On'z�V�_�0�mM�ud}�:��ew~`�������?�������(�V��u?u���1/X����4��nE��.�%k?�uq���e0O8����{������~[
zzz��R��z,�
����]�..X[��M\�CDD���W�{u���]�0>����������R�n�T
k�X7�X� [��R����M�N
�����1��8Pqj:��/Qy����cF#��&X��4����T�|� 4�F��E`mxfza��5��]��;s0�V�u��/�����Sj���;<��3�s�F��������%�,CA���%�����������[9����e\�����������N����C�Gy�M����WT:q��F�""�O~A��|m_>u�
`W{�!��.�G�q@T����VW���n�����F�j�q���?'^�]K��Pf�����+K���&�����?�6!@:6h�s��w�.-����$����o���?}��H������?�x8������vl%y*3W�l=^�����j��E��i6��M"�
��.��Q1i�$F�R�.����'�(����]�����0�?V��b%y*������s|���'��'�?�4<�V��Z���|~��[����V���M{��d��k�+Q��%K�NV���@���W�tq�/V��J?��*�H����i"Sn�����._�ibA�e���f��v����YO�|("�����LDD���!�8���v�Fe�
���Hvdd��#�btW��{##�J���pu����G{"o-�/���[��nq6�b��b��R�����s�k��;9P��%��U�%���;�5�FpA��O�z�3-��dY��9�]�;d�&0�2���fwI���/����D�?����t���*��N�~����9�?Xh��
<�x	5)���W{����������K���k�?U>_]����p�����M�U�����0w}!��a��U�G���/ �������9�r���H�M��t��F�����G�S�8�w8�����~��G |��s-���{{�����.Y��J���}��������nm?L������S����yi6��>q�
'kv9wCxl��k��n��������+��X���##������3���X7M��f������3��@D�����~��_�Z�Z����J���|o���L�v����;~���LLL�|>�0=zT�+�f�(�����W����u���~""�z0�����6�O<a���w�{��w�W;&Z?�/���%�:�K���[���k��q�zk�j)QEK�������~�{��0���5�xa#/��e8��9�������ff�j��d��C\Gw,n����m��~��sD+*m�e\���"8$�Y����YV��\�NDT[W�\��=�s���������I��"\x�W5Z���~�����F7�c/^��|o����W�V,��y����L�0���ca?���t��6U�/���������b��F�km���������LKK���F������V���hU#9d��1h�|q�!�����e��R�m[�Y�tqA�������#��������	�t���u�����8s�e���:�v�5ND�5�b15���'�����n�{��d�b�����Wo�`���ADD�]p�1��G�t���
��[��4�o�{�v�T�D�mW�f�����A��I�O[��"h��6�M<hyI����9()+*�H��B��vg���Mb�6����y2XE�������6~��� ���544p9�xjj
�a�a���K2.j��PHF�����p[�~l��O�g�m��8\�'�5������M+?������������2�S���"���y�
�����s""��_UnX�X�}�@v9y��wvv��� +����l6k���.n���-;7Y#�d�2�l5���a��Q
lk�X�Sa��k�c#h���xhnu W�����K�[x�"�����'\�zUm�bjjJ�R������k�A]�~��_+DDD[������cQ)�������T�E{ukUT�n��v<��������P(��d2��2Y|V����s[�[d7���lZ��Z#��c"��B^w��SF���eS�N�������+;)8�f���~�qe��I���n���?��{������+P� g�Yy�<9yE���+��P�����^��'""Z�}LDD��s��7W;J�����%���e]��n��t���q������m�f���\M'��
�<GDD�P�)�����^x��'|>�|�f�������}>���B2�:OLI��������q���_~�[�I�l�� �g����C���8,_f��b?A��DD�S�����s�����=�z��w��,gG����H�c�)�fV�Z#^��������|�U���4h���F
U�P���.�z�K����%��r�|�d�{e�g���;-�����l��[Y��m���HD���R)5��F�PIOO�;vL
��\a&����Y����������u9DDD�Q|����gL�����[��D�����8=�����h���cQ)������p�X^I���7�gT�|8���y5\c�����T�����V�4f#K���J��w���2M2j���\�5wk7���y���w��FDD�������GDD;�O�o���������{������E���-��Yk���>��5h�d��G
�����N����a�x�/����1hC�b�J20h��1[�����Z��l�4b��b��ok%4���*u[P[�~����!f��>�����Z�;of�d�����@L�^�P���j�gq����ZZ���v�+Q��%K�R,�v}?2���&l9o�=�����m7a�oeM+�����[�q��vEW��C���h��_m�v#"��V���~�[�Vb7�GF�9-j�Z;�r�qkj1t���H������v�xOb�O�������$w�3����|�f�f����$�F#��&@�/�P���Q�-L�(�����������+�b�hK��Da ��Xl�Vm��k���I���5��)�?���~D
��QLZf�n��v���)���	��Ok��|���A�v�h{�X�\.�n��,n���k���v��S�v���bA��~V�Q��X�4?DiOD�����h���b�����}5��<�qCC�a>���7��	��]��|������G�������u����!�[����W!""��F��������z��PO�}V��3����kk���F�j���i��`�M|�6�� 0��+�E%�)�&�F�����D����o"�R��7R�>77�a���zn���_L��D��7R,���Kv���u������V��R��o�?O�=7��Uz������}�T+�<�ZX������lME�b���0����sg��"�,�b��P��K&��]�����M�J�R���n'E���m'"X�V\����q�|q�=����F/^�@�7����j��������j��L&�c��������U�DDD�@y���Pq�����_m��dk�����@�/M�|��dup���+�;HL�M.Dn�=�f�5l���o�pW�����t�B�h�
�/{�_��v��]
nN~��}�!�D�Q���y�,C�e��y[��Z#:�3������Co�A��.9�������5v��sK�a�CDTo'N��������_���:�k�B
�����-�D�:s��������r{�~�q����c��2v5���x�5��pk��?]����T��s�����<�5��������P\�,�����p� �t������v+�+��r����������\[����c�:���4�	�����`�0z�$m>y�f�X��5�����d��8o����<^]\��..��q&3s�i1l�w���U&�eWV�9������dD.�H�BA,�R���^�*g�khh�L^�355�0�0^z�%��P($#�����$�6)t���EPtn��]����6]�2s���}$;22����.gG���_�����cRk������<�������f4��6z���`i6���D
)���a`8�����pw���ad�����}������f�(V<�n7\,�M������|�������������x�@��j�E�+����9�?Xh��
<�x	������y]�_�1����KDI^�T��ti��-�d�&[��&��mu��8�����u�
���kG�'g�����+���/�b�������k����wvv������1W���<���:���\��m�r�X����A��������xhMV��l""���������|�wD�J�&����������I���b���p�pXf�
e���+��68�k������pj����]�V���*�B-�_;����������y��3�j���Q=��W����
��>v055%n�WQGG����"����+���h�kT�Z;����������/��p��8������6��3g�;wNm����j�1�X
�2�L&�Q����5��7��
"""�V>���7h�1m+��BA�[��6���ry���Q���8��Fr9�w���0O^!*��lV�=/����/*���+��u�Q�Z���""������b��F�k��"�����ODDD[P�f:�|�ry������7n��q��X���H��b�����o�
�q�SSS���(�b������#�r9kM�q 	Q���%k��X[����\MDDD�������y
c^�^;Y�L���Q��?^
iD��{�[�[�Y;vL
��\a&���tq]�..X[��M\�CDD�a|j`�Xk�}LD�-���������DDD;G�&c������;�>CDD;������c��cV���v61H�]DDD[\,{��'��;�6���444�����x�
�p��5��w��q�����|�N}G��������V��+����Q��W!""��H���jY�r���].���5��Yk��}�4�-�A""���f��4���������
����W�����O�<���o�7�;v�X8V��d2���T��.���jLDD[�/�F���C{������im��������=�h4=����ZQ�*���n����b�tk^+��������jO^�X��L�����u��	%������W�}�X��
r0��6DD�'�X���'���������h�L�wD��|h�g�6����jG_����'N��f""�\�`0������������H�Rj���W���+���(�W�LMM0�0��^zI�E�8
����[��M�����cA��m�t'""�t��:o�3�2���1>2����n/�K�?�w�cm�jXk����*�������/#	��'oR������N��.�9��j����_�
�|�6h\�|Y������]3uD�������������l%�A�K�r���2�-Y��]<X^�V����xhMV��l""���`u��z����W��������xf���4yu~|���#�y��f��}<y�.�c_�M���+�%�}}LDDDDDTg��P��yf'O�p��U�����)qK��:::d5Va��V�X�@LDD[_#���1`�����]�0>��.������(,���v\��c ����x�Om""""""����b������}��}��B�L&��(�WX������b������b�)����n�����D�x9;2v�@4j��������]���DDDDDD�1��?���������Vb�
sD���B6�u��������Fk�����q�����]����rvd�+��D;�?0e��z�����������k\]��������~���-�)?����6��+W�~��������+j���_����F+y���|����������<������D�_!�o�'��x��WLLL���Q��/����$Q>�����Xs��k����8LM�����m�O�t#>��LDD�T����?��F�e��_�m�'�x�:��w���;����1��b����h���Ue���?�\
�	K�DDT�����|�S�Z;�bk\]�������h+�C��c������F5�Q��cQ)���5��������:&"��j������n�1+�DDDDDD[��~������>�������O&\�|�����7�����������}�l�?6��I$�n�<��9NDD�El���1Gm/����X�����/Y�h������_W��LOO�E�v7���X5&"�-n������n���&mI����9y��y�qu�����v,lb9�"LDD;��M^a�s�1����G��+.^�(��P&�������|�a8z�������#GdD�d2b�0�����F��\.���*s�Ufs�m\E��MJ�e���h���c""""""Z�D"�����y����z�-s�����p8,
�����r�v
�B��l6�f�#����+K���|h�$"f�~�8���)�_-kY��Z��������6�V�Z;^�����@lA
���
c�0
c(Yq����H�.���Vz���o5����
c0�\1e������$
��CM����
m��W:6hCI���y������7"""���~�c�t����j����"rGG�u�q�W���r�pLDD��4F�Q����?����[��D��M������C�h�H����������v�}\����N��[��J2�p,u��<�Lz���da `i6������9f�����X�
�r��pw!��v2�x����7S`M-��J��7��@\��Hr(v�/e>�� IDAT����E��";C}�+����]���y������@DDD�'������6��uvvNMMMOO+�W�
���rx��[d7����P��x����t�j����{��v�$���F��:Xk���>��5�0��O�v,�$�a��]��D���b���X����5h'+����9��/Q�+�Or������%�j����e%$�(m�ngDW���G$_��-�v5�)W��O�_�����8E���|���_��X��w��_v����x��V�j�v����Vz�����~��)`����%wf�����R���A���N<��._����\�5h��e{\v�/�����z#�"������?����a(��m^x�����;�=h�/}?���8������%��\^�)�m����b'p7u`��(;)8�f���f�������~nM�@���Q���y�0�W^yEF:;;���gff�����Wd��l6k� W�-S�Z���""�z��������������pm���9-�SZ��;������2���5������}.j����S����M
��@!�eK�H���.��������x_������|)
\w�D��|a���+�����s�����ssJ��n�����?�WH�����x�NGF/�����!�t
%!FS���_�C������_�b{��\�hiWM����N��P8���K�T~����^S<?PH���/��Z!0�)����x�]�n?u��/�q������t��<�I��:�����$��)
���Qw�c��q��^�_��K����Q���&��� #J��8P�u����s���&g�~�qs��|A�E�5nnA�c!"���?�D��/^��� ��^���������k����7n��q��c�`���\;������P=��W�L��*#DZ.�S���%k��X[����\MDDD[J#���{���������z��PO�}V��3���r$���h�X���s�
k��������u����m�/^A`,WF����{����.1�@��������ss���P[���T���Z���l>|�m��(N5�;�L���G��~?����FD6�/��~*9�wr�����\$W���/����P������H�_�vME�2��*�����q�z>%��0��������u���kz�6������d�����Tq�������R�w6��$G���WQS�RhT�
�z������������J�������S��Q��6��5���&V��(;�%�d�g���o���<���<��v��=��{�=5T���o�!@OO�P� ;�C�QA�������k�5"���r���6]#���1`��bixW�����+���G�~|������v�}�V�-�})h����&�S���	��H���M�4[�<�M��%�g�X�pj�P,�-���d��g��������>e����/�u:�����{��45z��M�w���������cQ�4g�s	�����z�_+U����+WW�;�C�|]�����\PVq��(���	�|��J'��c��CWDD���/����w_"""[>u�
`W{�!��.�G�q@����GJA,g���]��c��2v5����u��%�)~vo`����A#�����/Rk��<���kx~���\Wr(�4���4=p��v?�����������m��M��7�L�����N����vW��2m����U���x?�"��o�-�����ox��J�.��x-���Z_k���'*��������`To�����������t��x?�E�D�K�8,;Y��v}?2���&l9oW�y�V"_�	7]���"n~h�G�����4� �2G�C3�d�]�~��'_>�RJQ�C&"������_DDD;^�2s���}$;22����5�=�����bu����G{�.�2J�-o]��\�x�4�{����R���Z��x�[Q��j���a�Tv�,������?B��X�Cv;��j�����{GK���b��L�)`�>f"��=��Po����b���������?q:�������oXq���h{\��A��6��Uqg��������S��������ufE�NE��F���{{��7�P�;�y��_-��i6���������oi�;�|=h�/}?��G4�F�+�p���]�����������]���v�
s�m����dE0=�es�m��r������3��@D����bO>�dcccCC��s���r###�����0|>�o�!�]�����?.#>��0G������,��?�e���a�n]~��_����������
��OX+������y������j��z*���hZ�
��%���c����~�{������/l����g^0���\2Y��l[m����X#)�������J�y�e�.���)Za�����&���V��,����,+tM.W'"��+W������[������z�9��_����F+y��R���g��|�>�@�(w��E����[o�z�j��� n���|��d��X;6�s��MW�
mS5��������O
lk�X7�X]�6\r���z���a��_A��%�c�0��/#�{%�a��\��(5;�����u]7�Z���mW7��_�O�,O��K�}��u������zp�e��O�Z;�'"��D��:'N�P"�����v�U&���q����^��A�����h[hT�Z;�bk\]�6\����vZD�_�)h�!AF�q��Jw}e{�g��8o���z=^��^��e[�`P��&n���$�Q~���V�����y���y?�|��D�X�DD�QEW�^�C������>����`�a/������q(����l�.>��M
]?�q��6)A�q""�M�i�W<����J�������5��LDD��<y��O��LG^�+�nx�����Y��oZ������??s�����Q��B�}�@v9y��wvv��� +����l6k���.n���-;7Y#�d�2�l5�����}LDDD[WP3:�"�J���2�d���3DD;�����={�����8��p��IW�^U����������C|q��Z}P��+��
���i�ck�XT�uq"""z<Uw�^�Z��["��:��'b��"
e2�L&�@���2k����7��
"""�V>���9���h���]./���5�|�T��~��)������e��.�V����������������X,������2yJ�l6+��''���q,>��~tW��{�����h=����f�p)�Ym "";�>�9n�v��C7�K�u��d�����|�H=�7����,5��N9L��&"���O�X���/����O>�r����1�����WH�[��))^}�U2"n����/C�<)���d��,�?��~h;������L4U�'��w-""��\���*q������###bi�����]k-�����?<���4(1��5���>^�
�o��Z�8 }>�vv��c"����4�7W5J��������J������g����;��T@��l#�����q�i�q4UC%===jp��15@3s��,��g]\G����VkDp��������g���V�5�/�����)�:?>vk��ht�)(��O���=��W������Z�tk�E,�$c�R����f��]��k9
���������V���������.wf.�*����m��u�~l�N�;Pl#""�q�_|��#"���8�����?�.
=^���;���$i9;r�@O����u���5�jG_����'N��[9���hK������e�4��Z���eu��	
O�����u�����|s$�������P.���R6��
m_�j#"���V���~�[�Vb7�GF�9-V���q$���2�z��q���[S���-_FEO>�T����DD[��2�mERvbm��rY���x���m\��s���Y�e��T���;'"����~����L����}}}2����


o������>>~���Wd�u�-�_���u�����*DDD���ry�x�����9�m������|{w;���������]>W�G��}<88hN0?��i��`�M|�6�� 0����Gs�1Q����B�/xl/��^�������5s��T��v�E��Gg}^��h�z���~��������G�b��k���
������v h��P�m�{�F�����F��c��C������_mo������!����g��S;�f��;��cWW.�K@S<��/������1/�eh�L�1o��]k�+7�����V�]\������=�`���k��[�;"�Zy��w������'O�Vgjj
�j�B
�����-�U�:s��������r{�~�q�c]�b{���g�j���q�1�� ���k\���]z���l������Y&�p��o��e����a�������9�t��u���U&�eWV�9������dD.�G}$g��x����B��Bgbb��������2.j�G��!�)��0��1�i��m�@��n��k�UiR�.�DDD[�q�����]����rvdZLS��#���6
k�"W�����V�u?��]X�
����a����<k�����>�FK�Z��2���+W�~��������+j���_����F+������7���y��w�}��j��X�v��y����	��p��M�0^y����|���Y\@�v
�d�Y�
�,�*q3s��e�&kD��[&"��S6���E�����w�S�%��"<��-5Z��]���w,�L
m�G������+��DN�vLDDDDD���}� �N����B�L&��dB���p\O����1�v��6�m�X�S]��pj �������������^��6�q���SSS�����b�q&����bK
��"�ADDT�M+[k�����DD������3e���o��9����v6e�
������333��yr��P(��f�����l������r�����O
l�J��OuM""�V��� #:^����h}���b!�H������������W?��c�<66v��5�|���7n�4�q:��y�&s�XC�gff�C9�X�����_6��tq��$#�Vk�9.�E�����DD����1/X���\�I^����h����{j�����VC���5�TAv ���l���]�..X[��M\�CDD��|j`��V�m�k����7�"""������DDD;G�&c������������o�F
Q���""z�lZ���R����LDD;����1""�-.�=������


���S������J���|o���L�v����;~���LLL�|>�0=zT�+�f�(��������u�rt��U�����qddD,�9����k�e9;2���G�������zPl.�e��~��d���[[��X����&b�m���RGK�@K����jK���;/R�j��c���')&""�X�5�������=������j�I4�x�"��'O�z���a!�>��s�z�<�`0������-bo�(��k�F�V��3��������G���� �����J��'�x~��q��B*H�c�x3�1c4�X�7�X�E�L����9�/����R[�����ID�����BY���4�k>��c�M^q����?��y�������v,lby��HDDT��+����Iwi���|���h�]&���F����	8��E�s��Tq��(�
bH�?�x	�Z����IQz�����|i��Q�~?"�|\��5�1�k��Xc��1,��j�c�]��Qc����� "�~�u�9"������H�Rj���W���+���(�W�LMM0�0��^zI�E�8
����[��
������m����8��k���{�������������:��W�#bB�u�C���~�+�x��x����|!��]--�D��G:K#F�����d�@T������]�#���E���B��X���&l4��������������5k�D�=�+���]�s���!"""�Mh��_�
���f�/_�b�����k��AQ;���033333c� [�y����\.��LlK�����G�W��&��$Z��x1���h�4X].��>|���=�z����:?��o�n�]��rH�rv$�����F��iZi��4���X�7��IP6�wm�p8������k1��khKa�9~���
��rsmM����P�����������c"""""�h�&�8s�����3s?�����)qK��:::d5Va��V�X�@LDD[_#���1`�����]�0>��n������Z���R��YZ�2D����4���) �=���|�����&�0)���H���������{'R7���V��m�X,���������'��b��"
e2�L&��_a�k��o�-�DDD��+�]�����d��;�X~��`~|����L��������=�k�X�S]��9�H.��#.~���x��r�@�L�������%����t
��1_q
�cF�J�e����x~�]((MDDDDDD66`������8{����s?E����0O^!*��lV�=/���!��a�2���quk�W���������_������J|9;2-f96Or<?>r�`������ZO?���R��	�}�����o�9@�4��9����� `���H����������*k��&"��|�g�O�����:�����+W�~��������+j���_����F+y���|�������������_���B2�:OLI��������q���_~�[�Ib�cYA�Xa=�c�~��05)q�Mml�O�t#>��LDD�T6���i�����w�S�%��"<��-5Z-����jlC<����b�OuM""�3��������P>��������KDDU���c��(��b����DDDDDDD[�2lkLDD��5���b[)����IDDDDDD��x2FDD��lZ���R����LDDDDDD[��~������>�������O&\�|�����7�����������������Hx����~""���i�WXk�����DDDDDD�5���{b�������ey������_]�V2==
�c���n+c������M+�V�m�k������*��'O��Wgjj
�j��&V��� +�DD��l����b����DDDDDD�%}��Gr���/��,��+t&&&|>�a�=*��v|��2��X0������`0���r���\s���lA��m�t'""��8���������+�H����5G�~����z�q011 �����G5����B�P( �����r�(����..�6����%NDD�e�FJ��W�Z��������G��b����^�c�K�5�J�1����H.�m,��Ay2K��{[k
$�1�������[P���J���7T�m��+=np�DDDD���������7o�Q;�j,�����U�_17����1����F������n�>��7�U���~�+���<w;Q(��xQ8U�D����d v0�
������XJN�-�s)��0��,5���w������@ y,oF:f�F�fK�@K?gd��fh��U
{��i�OjxG��b�\���x��DDDDD&���=	�m.D:;;��������+d����B�U��b��Q��W,�����]���|���h�c�q�.^k�X��q��dR�%6e�#��	�-3�����~�XL��nM������b�&�o���������R�e(�J�H.��+�a��i�0�@ H�\����
c�+�����������N8���b�>o��x�P��LC��d`m�LV�lq�cr�
q���a����Q�4��Dl��8X�a�����f�;:(�
���Q�2�W�
�0d 3��R�������]Y��^��!��S��P�����Z9<_�y���@��:��_��_��o��Z�g�������=.��CiD�d�d��d%_�HY�k?��J��,8��M�AA���l�='"�z�y��2yEgg'����d2r�
X&�_1^����n�w�DDD������#�� IDAT{���/c����xq�
���xUl+��?��������B�P(,FF[��d��]������NN
�	�?!�(���t�e4�(B����/%������
����%��e@���)�a12Z�=5���B��[�����V<_(L������F��^O����d`��������b�/��Ba�W,kU����0�E�i`N���`�"��6@:f�N\a12�/vH���DW �����t�����C�A8u�t&h1XT�7ktb
(�t�����e���h���b���'��Jr-?���E�}������+�u���~<��nX��������<&�u?.�a���{�L��$�����B>��:��_��_���x�7@s<���N?:�|�9?�oC�J^��?�N�&q�$�:qU�dz��9_����k9������O%y���""�1>��C��H$�[p���?��b������X�v��X�q���7;v��+^y�� �����,���+d����KVYK��M2bm�&��rY4m)�V���~�[�VW>����C=��]X����w�;��c[)��)W��,3�����i�[��H�E�9~)1j���L�����Tql8���-Vh�Gn���
�H�������,i������k1��R��@����J#�J������6C6�Nj��X����{�gf��ux���f`Q6�cF�F���B
�o'
�W�@���m�k��������R\���|�
��Ftq�<
]L��c�}�s_@[�	�A��c��������(��lC�|J��\��DI����LM!���^��J�P\������,��y�h�'FeNiA��(Ek�8�����%�w��%���>'����/���NE����T@:69S:z���
�}wa�?�<>���y�zMe����u+��u	.����Z�{vyz� �S�M�������*y�������G
(U����K����l��+F����VkDp��m�F��c��C�b.�]�0>��^��B���R��	�?�����P��8�v��R��m��t�v�A5Com�����|i�b�9�N
�X:f����4r[hy��f�W�X���0��C7W&F���Ha��ZH�Z���o��R�6���Z���[��"�����(��+��o�wCIJ����"m�r�����a�Efd��F�4���B��NhJJ
ar\

��$����)��9����mI�`C���6�@�f|�����7�����X���Z{m�-K�����g={���d����<Hnd}��W��r��7������$WbM��8������h4*+��%����Q��a������t#l���y��s�>������N����OP����6����Z'��1��m{�"�e��KDD���\�h����3����C<S�{��c��m�-��4��������Zh����b����\�b���(K���r��^������z�^�7~^�ng=n��}����o�-��r�����.�J�;��d�0��8���
��o��4���~�6������[�r�o9�>)��
`����������
5�J���\�� �8��+Z2xR,O��n)��o�g��-R`��
<�?����c���Z[��2������W�����������Ol�w�b*�?�{��]`��-`�s�z\y�--��0
v��gv�~��M%4���8�����D���e�H��P����mq�&6�^jD>T�&������H�|("�����LDD���!?8�����Fm�
���#3::
x�#^2^�J������}�N����&�������qzk��d�d����.�	��������������r�h~���x��&4����V�������sw"r�Pg�*�K�uur��m{�K���H�o6W(��E�hhM$��#��8pw}M	��`�����w�3�E4Q�qF��P|}�o6�_$�h��Z~�l�J2�<�8��
Cy$���[����V�����G�*@������/���'>�g\iF����,<H��9�n`�� �@DZ�W�8pC���\~������C����]0��&|���y�U��������y������%�����K�`e��Kx�lY�&�f�
6��� (;1�l�R�z�P� L����f�Am�����g4G�j�g���[����g�'���Q��{�}��z�0::�4448�������.���\�=|��������+��<(�%���S�?����W3a�.?z��BDDT
�;Z��>c����a���5b��.��^,���X_�+��\�x=�Q�x����������W?��"�����z����	``@���������N<�<���@����*�����:*[��~l�X����>�d���v��E�c�z��#>���=;bE�%�]�Wk�.\���46��9H�G�x����~~�^�Bk�x�1n_��5����� of�����x�2����W?>��?l~�=t�L!l�V�s���'����r��i=d�����=z��k���X�X|u^p�W��D7���b�z�����\=P+��b��"?�xi��q"�?�/HF�y�����e���n��������>+�&u{������FR���t�,
z�'�-��8�����mI���g��I�q�����Sq�gk-O�������c�K9)�v,�-OA�%��lq�����Wx�E�����I�8Q}����C����Z���#�����>�z��F+��NDDT3�����XM<��cf������DDTe-/�����&��nq�
��,��4����v�~Jv�����?�a���aQn���I����_�A}��$5^j��P������d�k��=cN6�
����z,#NDd�����E�JZZZ��������~����z��o���������A�066����c�\�vMV�'&&h�Wb���x���1w��W^�qQ;noo���{������g\�i��{��#�8��h�d2������_~��-�����o��������SO=eV�m��w���J�cE��T�"�r�,�!���w��va�C�j�DD��[>�kk�����|,j�����
�g�8��8����a��l@�m�&3bK�m��V���F=P+f���_}���Z�����?��U��O<T[}�]�]�Rv�0�!"z�_������#��W[[����F��z��T?&�c����>mZ���Rl���3=2��i/o����-��g�}��}�iooO���tZ��,��Ee�|���7E���������������DD���}W�0mf��/�Jv��O[�$s������]���W�u���[|y��=�DDTm��(T�3����<�xE���W�7j��������v�DDD��k_�|Yo(v��IV���j�����&'�����>��d���]�H�el%�'��ZD>����A��7����F
}��=��v"�j����_~����������u���_�P������]��WH�W��%)��?�������/����~�W�I�L�� �7gA}���������J4��'�w-""��z����Q��������-�����v������_������$g���;z[�G}���WnF��K}����Eq""�yK�n�����o��'5���"<�z��\�����=/[���<I���z�.������=8t��`Y�B%������������jF� q[Q�4������][���zgG<�G	�:2��2��)��A[�"u��c"�MU&�T��D������8��\����j��K�:������'��:�|���~��#����������v���0�xm9}��x�E&	-���-�/K�u�����e�-��wi��hae��������6���><f��7���F���������m�xX�!4�A""���mDD�]5�����Ow���C��������Z~Y��}LDD�L�2�gERvb6��d�-�
>�5H�?s���m��D�8�T�P��{&"�G��\�mhh"�����x��	�p������7�|SF���>,�%�_��I|�m��X��������R#�������������=��X[�L/�t�D������?����F��&r���C]#O&�r�
"��$or�����x�J���W?��j3��v���`UGe>/DD�E�9s��?088���Kz�A�}<55�7����G��1�����1Z=JDDT7xT�����r���L.��}����:������*sl��,��.����DD�G���E�l�c��5#�%���!�>�����l����y�h����1�M��N���RN�:u��5r��Q�ay����v,T�S,���������(v�t��7�Ymi�`�!�iS�0��pU� 9DDT����1������K�G�
�~���-,��3�-_��MZ��j\��o����l�-��� q�-Ze��qm�c>�B����MDD�.]�+W�����,��+lfff\�u���e\��;::dDH��_��8N[��W�������g\~��V�I���F��(Z�;2����g;�j�g~��1��%�|J��	�4[��O�O�s/3"y6
�M�A��d�{�I�5���5>M[��<)"��K$��������z����������q��/�x��W�$������d2�L�� !��b;[��Q35���6�%��L���l""��������3vO[<��%?��u� 9DDDDDD����}�#�J���%������t:���^^��$����1m]�z�V��c�����*s�t{k�����?&"""""�3�N��Ca������������ym�
1�8�N{.^�N��tu2""�2lZ�X��O�<)�����/_�"������E""�9y?��L�q��������v��
�q��+:;;��������+�����+��.�(o����?Q5�z�V����q}O""�R��� #6a����h]�pAl$�����}���b���k�����������������: �J��q�����z��� �����\v�X����e�Kf����f���������l��c�Rl��=��hS�{u���2L���Q��9sF���3����!�
�9�XdS��l���`��!H��CDD��\=P+f�������y{cnmo�������7�>&"�M��3m�����5�V>6k����� mob�o��������{��>�����Q�����q�u_�u�055������edff�u]�q<xP�K�d2(^��������rvs�]�����-�\^+jY�Z2�~�2�5���}��8�Hr��)�L��-U�R8j��Wl%y����G�x����!W�zSn���T�����u���� ��T?�'��Ha��(��$�S�]������f��x����q��i=d������G�|���Fo���bz����lDv�����1�	7������O������yt��x��BP�:0���Y;��>����������l�fsy��o�rb����+�w{��F����i`1���M��
@2�_���]8����w/����b�w'�"e����:��%�O�9�K�H� 6��,�.bx}@�,r�E�&�i�V�&��vo��6#of�
����DDDTU���z(�7�xC�9rD�z\���ca+��h����V���[����-;������gd�X���{����O<\+���Y;�����O�
��P/���$��J,4�`��b.�$&�F�+���������8���Y��b�D"I�+�L��=|���$#�3�4�U����T���5���q�r"7W����T�T~{(���f�q���urwu��
�8H/a��4Y8����Wp���a9���x<)��X��R��Oe��b������g(�|��i���i�����'""���C�]�vM�\111155555������Y�����������������A(�E�_$4�+��~<����I��A""��������O�������p2�J���;Z~��|t��3���\�v\���(QA��������{Ho,�r�}K������\.7��)H�%'+�����
�J2"f�--�����p.7�'���s�D��,��JO��
��>������&`�0�N��N1���pnnF��^���\�f1�b�+�sxa�P&n���[����v�������|'o�n����2�Y��=��2����~,$#C�c�/r��O�w�[(Y�&����O�I��fDR����hk�z���������#G��sss:;;;::hd�XY�
QvV�h�c����M"��������6]#���{;�������?����{�-;��<�^n�n���#�k���5l��l���>�|�4�6�>)<�r��S�&�.6w-H�;��+�feu�5q7?96<`%9��������/��������7�+^��R��Y?�2���>9��#����w
��n���y������e%b����+8 w35�K��G/�M���/��f���m`�|�q��w�;]]�G��_���b�����5�x���,�a�,��8}VWN����=HDDDD����G\�vMocnn�q=����-���e���T?&Y5&""�[�����'��'�hy��k--;����;_�����{�k-���c����
�Q�����1�
�B���~h����T��i�������&=kw� ��D�9��M�`f����s��t�n������MKa/���s>b_��x�.N4��$���oLv����/��<�{7��+����C5����~$#����s���0{o���DDDDDd�}Jgg������t:����P����_[|S��0���|������������x��L���7��q�Fh�q��c/M����/G\,���A������������;+����b��H����v}�}R!���
��Xw.'��
�y�<�v�e�Mx��5n���z���Z	+�P&�
��le/{�����q�^V��j��/y��|K���<�~�KP��~��v
,�������c�u�����k}9q��y���;�R�p��FE��$�8�\������(���m�
Q8V�LF-k�{��g���N�v�DDD�������m�_����x�#��G����SO������s���q}��3j���S(��(+�H�R>�OPV��P���Zo�j��������������Z�E����.]e���{�S�0cW��c�����e�'|���9�=��/�����]��Pa��{?�8������{��GyI�����##K���<"������O%$#�w���0V�h�l._)N��pZ(���Q�5��������=���Eu���������?��*�-���>���U~KIiKKK7��U=Z���c��S�����/?������


������z�btt�-�WH


\�KR������_�������E��W^Q�D����M<TK�j�8l����8�&�4l��=��F�a�LDD[T&�	�n��f����h�W��-��-�s��}=V�=��9�����'O��z���j/5�����]z����DDD�����~��"K�DDT�z.�z�V�����q}O"��b���v%��s������F=P+f�XT����'Q=	5e���h����Y;��>f��$m��u^������h]�p�u���� ������'N��	W�^mhhx��7eD]�����2^���<���U���D������w!""��M[���s�1m+M�9O���q���188��?����[OO��#G�h)���P;����3V�����mZ����f�{Q]:u��9z��:��<sssV;6���FY&"��d��0k��}LDDDDD��]�tI�\166&��������q]�q�qQ;����!�N�
�q�����F��l6�-Y�U��j�g\E��MZ0`����nq�1mT"�����������;�F|�����b�p�4s�9 IDAT�_��D�������@&��3���`Q��%][\>�l��-NDDT�GGG���{�[v���fF�<�o�c����s�1�588���KzC�TJ�>.���=�N�����vs�qy�W��Uc""�-�1�X[����@<��zgG<�G	b53�p<�`53:�\\n���f��LDDDDD�%�k��y|�tgg��������~��w�N�=���{�����a�!��������.�������eb����g��H������U��9��5b�>&"�*��X�3m���/�Jv�`K�G2�|<�I��%k,�Xr��x�������	7n�p��W_������������y���L&�����>e��+�j�ODDT
.�-����[{��'G����|x���?��r>e�~���ja�2��c��c}O""�R��� #Z�����m��xIff��"���d����-���>m�qQ�.\� 6�Dq���>��S�}����?�\lOLLLMM�������i��J�n��@�b�����x(��:r��c�9��e�Kf����f���������4X[����nZ[{��??��'��k����������69:�5<���.e0k��}LD�%�{u���2iibw���s[B@�~�;4��g�D���gn:��n�RQ=8s��*8~������!�
�9�XdS��[�-.��fD��m�F��c�����hy��k-�G��5-V3�(�
�Y����� ����fE�J�AT�(��+K����g��������������=�mqsG�����T�(�A=��g������x���tEDD�������9�V�(V�P��;Z������������������/�z4�F���DD� �H��}��������~4f?��>�xl�����x ���{6���u���&6��|[�F�Vk�r�Tf��q�����q��_d=��Y@DD��Fm�
���#3::
x��P&^������~�#�'�Y�v���DD� �l�O-�I���F���>�5H�`�m<�������Ry���/m|<���|����@D�H9}�tcccCC������b���nACC��8������2ajj�u������������8x������dP���'����	{t��|""�jh�������-��^���5b�>&"�����Q��[���]��g�n���|���_�������|�W4�$=:���|��z��x<>66�����]��g�������3�:�&���������Q""��s�@���c��c}O""�T�����0�F���{?[k5�#�5��Oa�}�����B�%�qmq�����^��
I}����I�8Q��x��
��7��"G�Q��'x�X�����h4�~dm�z�V��1gmKQcmbq����*������gP��e�l6�F��c�����jWj�F6e��+T�����)��h4j�W�1�D>�:���h��W�����ib[�CDD%]�vM�\111���@[��fnn��8�����+2.j����2"���������l����������I�m:����z�&�z�)�R<44t��93��LDD�����_�)�GF~��e�y7�m6?��vNDd����Z[[�������U�|��X�_���������?�x�������kk�����x�\�X��;;;,,,�d��q&�q����W�>�l��Mf��l�&"��-���z��f���z��������Z���������~E-��K��d�/Nv�0�!"�6��g�����������^��}�cnnN|�^Immm��#�V���R����""���i�c�v,*��8=���i/o����-��g�}��}�iooO���tZ��,��Ee�|���7E����������1gm��+�
�6�F��M%;�}��-^�6��������Ym�6�j{�����~h�W�PA�d2�����Q�5��{x�w���
���m��j���f�{V���H���%"���;"AF�&��%�:����������W�l��H�T���:��DD�t�����{@��+FGG����W�[011155%�����������L*����/�����+
��������Q\;���L&#�:r6����2�������^��d6���j&�q�-����6E��PbOw����_���!���''D����=�I������}����w�\�Z���kqx����t��S�8���)�7d�f�9����2��P���pbHZ��.X�h�xH�u�<����[�@��[��3�ck����]z��H^q\���z������w~S�1W�:?��=��a�;��4��mG��-! [?[�NL�n� ��s�m�K��r&"����a=d���PAOO�:tH��\��S��{�-nc������A��""��i�k�����zm�(�����L<�
`53�Y�P���fW��<�6�>������o��y�������X��~���}h��������r3��\��T�����\��%$��?wb8D
�A��O���@������|��@l��,����U��W��
���JD�n����]�\�:$�D�{v!5���p7w�	��.�)��:|�s����U���/WV���_k
x [�-n6?,�P�=_��������j�x�����m;�?���h;q������������=�f5������[�����?��j!�f������>���z�O=.*�^�r��K#������{�����8CN�d�3�8C�3���Y�+�3�8������9��d��A2b�s[v����� r�*4���;xKM:�'�K?�{���
0~f?����%Q���E�/�9��������J���x?�
1fs�:�~���L$�`=�hT�z��o���,Ms��S�����\)�_;�q�^'���
/*�K����2��g����G��������s�J��5��l��hq�K���	���v�d��`c�5N��d?%�����]Lm5�a�����C�����#B~��������FX[����n�����0��qMg�PA.���O=�w�n�����&0��2�Y��5�����\X�y��J�>7y�'1����^��s����M~:�Hz<�vnq���k��S;��]�������s�fS��s��$#X,^�!ye}�(n�;W��-���������������S�z_!��?�};��r�
�,{�G�S��<z����$'���r3�\Aa�^���O��y�c=�]���G�?M��~�L3��=��~�N�m�������xs�����E������^�u�|����c��������Cx�?_����|=X���0������������;�_��z�-�(�������N�&�**-w�-�3N���U�q��4����+ ���*��mW.\p]�������u]�uO�8!�^�������o����DCC�u>,�%�_��I|�m��s��������R#��U�*�-^f�8��c�)�PTA�X��������=@d"9����x,�`h�z�e��]�����~v
����B��g�������;\�\�����E���;����
��
%��������8����$�.�[���o��������*}���������[�.\�����[|�����|K�c��������>q�C�R@�����'���^�������q���^�O���?����B��?�t
�����������M����|��\����{���w�|�V��|}_��S0�[��E0�c�n�~��79�F�ox<o��������F�����>q������'�>D���2�""����9���p��%=� �>�������LQ�O�z[����DDDu���Jl�~����v��r���@�������>��z81����^u���u�A�{$fw�'o������QD�]�����G�t���
�����R����]��{2��P�y�{�o����B�E�~<
���+�M�/qVL�-�������@�|�S��;���zP��I���T�������I���E�l�c��5#�%���!l��m���|���A�c�V���c�,��\������SW�\Q#G�U�gnn�j�B?�����������B�����������|������Y#4�����+���P�r��Ol����~������?�����M��=�s��������K���m��;���w�QT�3��m�@�O���Xa:mJ�
����&�D��Q������u��:�����o�b~�,������x�*s}6|�{mwW��!��K<����f8��������qK�c!& ��>�F�6�u+�r|_'��Mx���������y<_>�����e��l*�����~�jF���3��k�n��~$��`��^��B
>����3��&m<"b�K�Sff�o�m���m�*����k�������x����K���cccb9m�
����u�p��A��������s�����Q���!��Q�����k�g�g~�8Q�rn������|Y^����_���g��q�-���z��

�;w���;o�^;����[����+�I��;���������Hsz	@_on����0��A2��x����R�C��{q`|��M@k���n�����(��?
�+����cY�}��|2(1��(Zs��2{n8f����>_��)j����5Q�,a�"/��}72�d"wb�:/>������~�,����'�8�?�&�^|�o����������+�qe��OY�u+�����^O�����[���#��w��������������������)��s���������d�N��_U�m����|}��g���z�nii�����G��},����R���[=������O�:���kSSS���333:�������j~��|�����@�v��� ��0+�����UjN�m�&3bK�m��)���� �f���z��������Z���������{��}|����'OV�v���dI[��v2��U������|
������|LDDTW��|��;��t��F��r�q��1�+����em9>�]�&3"���N���QS���F=P+f�XT�m�*�
�+1��������}�����Sf�������MDDDD��vV,�q_���9777??��_!
��t�s�������0�������Z1k�?�x�c��v
,�\@��&���������GS6��dr�����Rn����_�1�xaaAF����~�T^�����T�""�j����Y;��>��$"""""��p����H$�[066������k��}���b{bbbjjJlOOOOOO���q*��q��r�$��
��z,�����P�c�|���-.�M2b���j\n�&""���i�W��c�>&"�����a�S�����3z�����z�����*�>^y���m�������mq�l5#B��-���h�mZ����f��LDTW����>#7�2����
������|NDD��M+�5b�>&"""""��o��o���\	�JB=DDD�������1gm-.\p]�������u]�uO�8!�^�������o����DCC�u>,�%����%,L�����]���BDDTK�h����-O�XQ�jF����o���d�QE�+M�d�_m��h��}V�T�+������R�n�;C�����LjDD����Mm����5r�����?�����zzz�9�GK���G��1
u����	��DDT��x<��{�?���������<:��*��3k������1��X��C�l.�w�-Q�M�;��wE�n�x�����5q7���������D�h��r��E��i`17�����@��C�ln8�v���r/�&c��}��|���CDTK�&�� ""�G��S�����G�������C������h4�_���h;�/^�zo�/�w����6���w�t���C�n�Y;�����O0�6�>���zq�.���[���&lX�
�i��#Fb��/�� 6�{��.7K�wa6WT;n_��5��D��N<�<���@��
+7#�i$�^��7U��K���cccb9m�
����u�p��A��������p��m�6V�F��l6���*���A��g�'""�[.�-���������v\���(T�-f��|3p�{���6,�������X�EW$������R�G�d���Zp��Z��u.`AD�'����ADDD��D"��������;vL�������D���/��3���q{{{{{;�L&�g#���WY����C�&Q����DDDu������Ow�-�xE���@����2�A�r��S�&�.6w-H�;��+��%Zlv�yQ�������� ���	��\~�sA��}��;3�7X���DDDDDDU688���KzC�TJ�>.���=�N�����vs�qy�W��Uc""�-���Jl�W�Y;��>.� ���(��~�Qn~��*��\nH�G��6����&�.4!��t]L
�U�&"""""�c�����b�I���sss�����b�q:��\�BN�����N�ADDT�F�B�G����Fh�q��c/M��"����j���
�MD��SA�07=����
b�V��=gL=V�W�w����E&"�y?V�3u�N��Vd���Z?�;���X���vP��2��>;��x&�
���l�S�%�W��9U��7�y��We���s~~~aaA~{��xE&��d2j��w���
���U�������_�����������������jkXf��6�X��"��q2�5��.�q�Hre=>����fG��Y�R�J2�8N������������f'��4����%�-;�������jle��KxQY<Y�}q������F���
u���55{�!z�9�)TrX�s�(�4��.9u5n#������Y��R�5�	��!�zv����H$�[066������k��}���b{bbbjjJlOOOOOO8t��T*u��
j�XS��C9�X�����#m/[\2�d�l5����MDDDu�qGK���|f|O[<��Z�v\������������-����K_�[���A�{:�d���R�(|�������q���D��ou��h�����a�S���68��n�v.D-�o��0f~X������i������������T�+���*���3g��������!@OO�P� ��S�QA6�}K������jF� q[��k��b��m���UA�"�~�78~=��_,u{/�x�\�X���{DDU���/
Z��V���x�����A�q�����j$��f���F����J�\����T���69�-�3n;�-K?���
�I
��>��T�q=��#�����=�3�C=�m[>���mc�?��GDD�m�z�V�qMgo%{�s���I�u���1m)��F��������1"_���k;��,�����<��l����&�o��6����f����&-�/x~��Iv"-�E����mq!x���-.�&�PF�C���`��d�b�����)[�o���o����C��!*�m��@�!m{�}�J2�<��G���:<""D�I���S����q� ��R��=1O�B	����M�p)|Tox>�<n�<Y������������O��'-T?D����g]�}��������[����8�������L���r]����2233����8<(�%e2/�I��������-%�.DDD��i�c�F�ef�X���h��73�Fy76b��wS�<o�D5�^�[��z��G�|�"��V/�I�W��%���M��z�m�9����>"Y���}�M>	*��P�mu����������=z��k���X�X|u^p�W��D�>����%""�9W��Y;��>��$"�M%����P����>gv��2���Co]a����\��U���`�?l��*u�����9>/� �����4A�!��N�>���y��7���#G����	^;|�%�-�z���9�>&"���j��(ui> IDAT)Z����J
�mu/����WkR{V�b[�������'��wn�W�z���.�����Qo��yi-y [�-��[\kR�2YnG�Q�~��<n�{�#{�����o��H�x��]����������>8{���V��k���


��+l���8��8�+��"��v���.#��������6�4�~<�"(:�l���DDD���������z���~�|���P����� �X�K����-�U��eyC��K*{G""�m&����@�f���z�����r���-���_F�������kk�����x�\�X��;;;,,,�d��q&�qG\+�/��J�	���dFl��m""��*�A����z�\����b���=�-;,O�~��z���Mj�<rf�;����|��G�����4���zs�{�&""*���a?�F�����JN�>����7���������v��>�177'�R�����l6+~y������$�!""�[��x���dz=�lG�m��W3�����w�V3�����\��Z�W����O� ����1Q��}�U��DDDT
b�1�xE���d�}��}�iooO���tZ��,~��Y��	[|S��0���|��n�zo�/����[���v@��i���������o
9�	X�S��Old""*I�w��L�����<��O���g���|�^4I>wZ^��������qf�0��9B�l5#BeG^c��UV���*�}��z�����{=�����-^�B9���o���WD��8.�UT�^�U�""��p`m�����7�8V���L�-N���z�RD����q+�GK	uK&�=�������P������S�qJ�k	2��_f��60I�E�{���+����z��8�m��VVu�aU�|+��v��{��
����2�W�^-n���������������333�T����������(�;������Gq�XS�3��x(���lV<w�����if?��d6���j&�q�-����6E#���{;���[l������wv��xH����>���(����>�x�W�����y2q�+W���0��y�Tcr0�HD<����=����	�������]P|%mga��8=�C�x<�`����|b/��";����P�����*��u�?���z����V��m��>�C^�2��zzz����Cz�e�
��z,�^*���-��V3"��r���j��g�^��������'�����g�|{���Ee��(^�"6|nC]op�c""���f��(�(�%�m�I��������y\����-�F��l�T�R��]����s[n����/3(^'��|%����j���h[P��~
6����������c����0���-�3mA����|(��_&�����=�g�*��j���K�ePn�&���6w�j�������g?>q�g���m'�u�
#�����L<����1���E��2q�c"��wP�VJn��jA-.�E���f���Wfz�?��U�AA6��Uf����f
� ��zF���lq��2�i#�����_��u%����S�	�z������K0��A-.{P[��|��M�����h�����>q!x���-.�&�PF�C���`���~J�?�Iv+v�����d���EP����x<�m����I}(6$m<"�d?%�+��`��c�S�O�|������F�*��+����_��~-�v������|���T�Y;&"�lQen5n���m�j7�l�4���JNZ��A��6a����#��������OB���gET����r��[m��U%�rP�3"~XB=�b�\*{^f�%���155���������_���V�4�)��0�md�Zp�����<�;Z����*��;Z��-j`��������c"�r��u������$�[�����
���������g���������r�ic�^w�6%�����_���h�I(�8���a���e��?�:|���	~A�������?�X[[����O��\g{����a!����wHDDT�F=P+r�����I���;���<��7�����,��,��*,�������lf�A�+s���U���<O�$�r��YlE��f�o���G�N��6r� l��������_���D��~^�~l��}:r���qa�^���DDD��s��}=V�=��� ���|����'���'UY�K����-��nyC��J�I2_����JZ��i;.�&��D�	J��<����m<;�"A��@���26[\�V3��o�:~�[�@�A�A3_M�*8~A�<-��q��QE��4�Il�A�����������<�m[>J�/�&[��N����xl������e�$�+3��mJ�G��	2G��<;��U���dA6yv��l;���gW�A�]<��L&`�B6���/^��_�>���?�h)/�����?������p��y�p�������n"�p]�u�S�N����1�u�;���k������&&&��W�����Xz����"`nn��8������V��$f��k�|���oc�W<���Q�~\""�C��_�&����mZ������j��w�:w���w&"�j*Y>�C�h73���#m
�g�?�D�U�����������=������/�~�aCC�;�����K���'N���|,hk����C���q�q�W_}��qgg'������J�oj�������~���������V�
u�3����P�?�+X>���+��X����{��"�d�^���G�;�f����\"�Dj�4���Pok�9������%�,0�
9�z���(�M+��cQ)�����/���o�G��h���c){G"�D��%��b{SeV[�&��e�cH@���~$��&����HDD�o���f�xsg�vLDTo��ox�����������W�^+�+W���977777'�D.�����f����%l~����G�x��h	2�x��DD[��j���f�{V��_~����hS��l""�zs�����{�\�XJ$�1���p����G��'&&&����������tq����������Co���d�F�@6�_-dP>�j������xT��#	�e��!������r��?�Sl=����e�����^��qk������������{�1�R|����'O�5�m#������>����J����c��pLo$5�ta6�]��D�=T�����"""����S�W�}���z@�W�����7�|So����#""�r*�A_��������'���g;�m{-����?����=��\�vl�}\�
�����D
Tx�c�H�k�9T�2���r/��}���>��O�&6�Ko""`�m�1yG����3��a4Q��p���H$���2>::��������X���c""�s�������B-���mO�Y��i�op�����u*���gEt}����@%k���?��]�9�/�G�V
�~�G�?���m~n_��5��z�r3�E���W|����K��N;$"P����c�x�@
�Mj\l�8��3g���������Z;���?��xEXr�
�����j�����|�[o���fF��X�������2��+OTxs�+<�z��H���g�;�CnR�hv�	�9����t�94'�|%�H����C�8�t������TwL/B�������H355����������d2�����I��6KQ��m��$���=����!�xLa�;$"�jk��jT�K|O[<��fF7�x�Y#4��xJo*���q�����0[(
���7��)�u�|�y��Xa�F�+w~v?�������z�x��}��;�P��HPoZ�;4"""�
jkk�
����w�l6�F���g������W��J���R*�!U[#�*1{����L�����,��q���F�tC���U��L}�](������hS9rDU�Y"l�T�]��j�DD�(h�Z�������o����vv�)�a�5�@��+���[��+���-�>���aH�c��w�����
�tg��	�+�+���gs�Z������}�q�1�`������u��/��Qm���?������


����.������&	�]y�N�Mccc��;v��k�\�}���LLL444P���������E����q:::�ioo���l�Ex}���2_
�q�����d�������C��
jA�*�|��������z���{;�����+W�Q#-��������e��m���=+���~������2��w��:��-y�9$n���kDY9��`��HRvSJlo~���"���A$���c[|e��Kx�5#��H�F��o���-.��CsC��EDk"""�Mt���>����G9s���>�hpp���+W�({x����M=��b�p��_������[XX�v��fm�JO2��_<|~i�MrH��:N9-��Y�K����{�9�XJ�U�xM�8eP�k���-
�KDDU�����s�W<��qE�����>6f�Vx����p���X��b��v����'���P�����=�������lM��zLDDDDD[W���Y��|j�e��j�f�r������3�s�A>�h�+�F=�DDD��j���fW��\,�����4������������u/z�=���#xqv�+W��g�0�r��U��/7�s��W�9���hf?�y�j�>;Q�lZ����t��a{��{�s��Xp���f=��E���������T�d�Q�;�w���a��/�������r���O?�����b�c��qX���ssssssbM�2d3U�����3����B�~OPv���
�����h	2XF���T���vl�}��Y^�����q��G+-�!���j�-�<_R�q}�Lm^�D���>}��������kM�D���?p��	<v����G��SSSb{zzzzz��]���)�>��d2��-�M�]N�C��f
Z~Ijf���L5�9H�64�{��D�V����'[�m�j\�?H\>����`�'"��q�����j���3+��/_>y��YS&"�Zjyi������o����G�w���<�-�����yUJ���sG� ��L&�g3����/����~����z��o��V>������w�yGo���o�������<���^y�-���h�����7_��rm��f��6��6�m��1Q�E�5+V�D��&{�eP���-
fP���g?>l���$����h�k;��-�3�
�d�~lq�f���}��I��4s[
x\���a�hA[\��'9Tm��H�p���H$���2>::��������X���c""�s�@��5b�N���������D�����O��L�W�����-���d����+[�oBm������B���[\<�M�����������v\���Q��.�~lq�P6���|-ND�qg��y��w�y��v �?~�s��������&�>�|���"b-5R%�}LDT%Q��M����j��.>	*��P��8�+�|+�<(�=�gW�?����<�K�$������)`'(������r]����zC��zC&""����Q��������'G��y=C�V3��?<�o��C�3�m+d����jS;F�f���4���zs�{�&"�-B��
�{0��y����h�IP�����"�;��������x*�R�Uo�m9%9!""���x<��{�?���Q=����oD���U��S������_~�E6��_H]d����<y3fnx�f�j!U�U2���]�M�� 6��6B�~6��*l?�����x���)�}^6���T<� ?zDDe8r�H�����E�Q�������dfGw�����fF�ggO����o<�Cx���v��=44�?����sz�Rf�F�'z����O�u��c"�w-/����Rz�`�W�e)`9I��S�=��A�����x��vQ��h
�*
Ie�Q������7��-
%�d�-_P�Fd�s<6�A�.>��<��P��A-.�>q�>������<����f*�����S�����/?������


����.������&	�]y�N�Mccc��;v��k�\�}���LLL444P�����yqsss����XXXp���g��C�����?����-}�?��n���iqY�Pv��5� 
6��q�lQ�����I�������������q&�a�����BM���?i�������g���9���w��$�<�g^���=wbf��<�in���L&#2H�y�|a�y���x���9�&l�����Q
D�F���Wz�\�������].��.�����T�n��0v��%�>�/Ww�c��[����
2��DTJ����'��C}F\G�5[�M)������###.^�����+WD���k�XL����q#��>}�����},���������8��C�P�������P���_��^�0�|Y������H�6A�{���.�6�F�o����VW��x�K�1��?����}�xw��'r8r�c���;����z���)D�
�|��j�!j��;�OY;&""�����h���Xcr�`���$�eP�/���X��e(#�yH�����h+��T�����T�	@aUr�c�}��Y�20�������j���\�����O������hk��;�z�'Q�x�FR8*O�N�,
�x�WM�9�����8V�����xD���E�1e*���`�
"�MC�W��
�yg��q�.g�\l�yj|��O�����<T������Z���`�}�������'o��=>>.v:Vw����unnnnnN��8��^��<Z�l��$�y��y]Ie�6R��M'""����l���p~���zX���M~��w�.��}a���}S������>������6y���*��mC�y��O&�>�uM�
J��J�����T�<+��Vxx5�~mq��288x��r�c)�N��'���Boo��'�&''���D{zzzzz��_����������wX������e�Y�����wM&1������)�x5�l����ymq""����.�#��_���2������F����OD�i�{�QG��7QAI�K����j�<T�{Q�yQ$�����!���8�c���Hww�Zw���<���M�������)HP}d����'��x��yd;��-{�f��*�jE�>���<)`��r�����}����	=9nLDu+i����D�.���^�-�I{�D[������<jD
J�xu��L�I�����kyJ����e�<D��P�a�W������3��������gP���ClA5�R��	h5��������=�de�[��U���CCC��N��
��������8~�����m^q��A}U��jE�����\�tI���M���>�ix1w��c"��,k���d2i�ed�Fl#�i��2��ov�H�����>�A3.���OOb�h��T���o������x��?ETy�x�J>,/��51���2��e�W0�k� �y����Z���G<Tiy��C�<Du�������;{����8�J�<y�s��������:�>�������>&"����yJ*_,�"U%��J���)ym"�=����||��C|�|���c�sx$�=����CL�$))��d�155��������*���k8yZ���Y#{Y�q���`���( Y�P[�L\uTw�"C�y�il���y9e�3���S��>���#�|�|���c�sx$�Me���4�;o>o5�e���
�""���j���V�GV�����T�����	16���� yE�VB�*�6�Py��7�x9�g>A�y���	���_/����[\�?BD��#G�����E#�Ln�_�����( ����z�&^y��R��?�Q�w\���DD$5���������x]V+��l��M����,�hm��$hyJ&����&,��^�!����<'�5����_M��D��z��S������3b�N��g��J�4��0��<��yj3T���`?Gm��%j�7�x��5D[6�����qm��V<��K�qI;m���� IDATb�L&�:����'?���u{�_�������{��m����X����"844����s�������3g���[�n��������;��=z���d,�n^q��=_����&����8����2??�8Nss���d2�A�~}��g�gP����qX��������'��������J��k����v�2W_�|���Kf\?�
���q��E�d�XQ0�n-�g�uT_�%���{��D~WY^�xdd���]��r���_�v-��={��7������a/����|���	������:t��qkk+���y%�@��WC����&�jC;D`�#i������C3�9����E�o����ccc�����������}�������I��fK�i��Y#�T�q����vm����������O�����}���*ZN�x��������=�Un�-���h��;4������<��I�����D5���	������6�d2���O�I��CDDd��R)�Kw��J������k������������W�<�����/>�����������9=Os�
��t�T������\?���=N��]����h���;�z�'�V��S����u��p,�J"{�=�<DDD&W�o���w�c���iw�[�G���t�q����X
DY��:��t���O������r�oC�b.�4�p�I//�����g������o��q�D"��c���}��e���p�q�v2������sy����p��N�K1�H�w���	�rb�eQ�h+�V�Fs;�����J^��+�9j����n�g��6��'O�����;W'�����{����WJT��j��U�����1j�Wc���e�3����e���h3q`u���7vw^����eV�xxf����X?2*�����/
����7���O���N�������z4�g�DO.�����Y�h��iNx�3!+���	�1���#@C����1�}$��9%�V
���\�����
����>�qaF��������y$�KD�C�l�M��]V�*�yH&�eE>�?N��6�������������t:��'����'N�X��������������~]kkkkkkKK��a��e���j�Sm�+�Lb��q5��xST�mq�K���'""����l�]z�����j��W2c��T�������q�����i�
�@�W�m~�h�3*Z�O�����9��S3�j��1q������G��q�G�7�
��9���L�mH���S�/G��k�=;}�^�6]��z�_��_!�J�����"�[�2�-u�>��<���-b�]/��$�7�f`@�X�p��1���^�._�'�������y�y�z	���Wyv�$���6����3(������h3��T��q��������0k����Ed��X	������n�(*{��O��8x��K|;h_���w�����gi8�f���p�{��{yo��S3�Fp��x2;���j7������K���,f����� �9^M�=?��}x�����s�!�c��B���\�����Ur>�W�j��`�����}Z����L�������y���g
"�wCCC��N��
��������8~�������:77777�8���#�����\�*#�tw��R�d���_���hOd�Fh�q��k��M�Z���������v���?�7���p�8�o�l>�v���������l���q���"�Z����v��+�����]/�����s7�}�������Z��15�5d[�����Jt��������-[X<���<��m�6r�v�yF���v\
���j�]�d�8��-������gB�_3��m��)H5.��.���6�����;w�������T*u��I��+���W�vLDDT{q}��3����������Ou�V{�2k��VG����WG���E�v�Uwh��YOB����L��^��O�[n�}�p�=��������
�=�#:����$�t����^�k+��m!�������\@�i��4�%Y�����i�l6+�U���Jx��!������)�4J
{]���p�S$W�����s2�u�$.M;��<��<��$�
hjj�u������d2b��$^��K��P�����l*5���DDD\|{cW�����q��qMW�KV�^\�������<{���x=��������U�wJ�K���>y����U&�J�-���4�B^�H�%�v���4"�.��E�����7�2���|����g�:z>�hSjjjR_�������4�.M�]_��`�\��3��DDDu���b��m���#�caaAE����v���_��D[��G2�������v~��ew�2R
��.���j\��{x[<,yx�v�M5�������W#���h;r�H�K�/8��!&�I3�g����~��@������>6D�:�P��DD��c��*q�'"Z�uF�n0��6����a�<&�K���<��y��"f�=o���r�2b����>�S�<�g����lY���MR���d�<2�u�q"�#}}}��m����X����"844����s�������3g���[�n��������;��=z���d,�n^q��=[O����8���9��������;����,{7���`/t���8��p�?��jb��]f������.]2���U����j�'"�P�;��'��C���/���m��zQ�~��>��*������f�����h��~����?�h)��������/��{���v�Z,;{�,�7n�����O�^>���E���������9t�
����V���J~��Z~����S�
��gK.�Riq�a�6mn���/}-��.���F���DDD�������A������f���L&�"�7���Wx����CR��)���*�n�W��cQ)�������������.l����qi��*�������VA.�#""��u+�5b�>&"��~&�����F/�2�����Ro�4����J����>�� ����G���b�����',q����*"f�����vj�k�{2��0���������t�����o��������+�kmm������{"GB��m�W�UQ�=')z�$""�*W��Y;��>������=�j�'"���!�P��A�NI�eR���<��� O�?�E�R���I*���<�����K��#�<VM����m���m���#O��.�3Qy"*���������t:��'����'N�X��������������~]kkkkkkKK���%���F6��yER�.�1��`y� m9+��O���h�����D����]��,�����2���{����T����p��1W�q3�5|�ml7?j<i��1s�?T�t������O���$�����Y��sO��<�f����lg�l?>[����
�u�����/�����B�{�l�`||��c�d���{��@|i����&��W������/���P��>�m]Q���EDDT%�T*`u�����Z]�8�����_�hI�Do%���m�qm*���������j&�����
>X�:$�����2R�G=����.[P��2���y�"���l�����[�������O�-nj���,UI���������KDA

�F:�V+�ccc��������2��y����DDDTM���W��x�+����k�ok�8V�>�i*�({+a�������4��D���Q�-)�HyRIT���d�Y�c��_��Z�do#�%�Zo�l���1+97�����`�������#����$�-J$I�e���y���Q�+&�?G"����?��b�R$�/=&""���`u���7��.#��#v�������EXf������H
R`���A�^���8�va/*�P�K������2��������%�$��M)i�!�W��2!��055�����������K�k���q��>�<��R������h�p��<���n�F|u������S�T����.��2�5b�>&"���l��f�/��J�L&�"��bz����AI#<��L1��d3����9��W�x6��(��mq���^{��e�#DD�.�W��7��I���|e�|�������Q=rX��^�W���oo�)�-���a��m���#�caaAE�����j�����#��\�q�6LeF�0���d��H&i�c�a����)�*�O���d�y�����P������������#�Xzl��(E�"�L&�W9� Q���;THF|{c��gc�����V~�O�*����s�1�f�T���x�-�!�l��E�$�<B�i��%�=	�9�x�K�}�{�/��Wm�A����dF-���m��(5��@�� �Qj\��v�:%�Pv��N�E-y�����\�I=�r�<e����������m��x<���^�*�CCC���;w.�N���;s����u���������������GLNN�b1������ymmm"`nn��8---�����477�^5~�P_����6�'""Z�������w5��;V��z�W2c����j�]���]��J����/]�d������{W;?Q�����?�����_��e���]��6�L&��8����'?���u{�_�������{###.^�����+WD���k�X����n����O�>
{�X��>����N���w���C(��������8x�`!�7�/�dPF��Z�?��^G�����MDD�[�o�K_�G����������Jy�C3k�\}LDD�q���BDD�q��k��SDY��v��L&#���3��"9Q�\=P+f�XT���~duT{o�j�'"""""�L&��w���Q��FN���J�x(��2HDD�^�z�V��1W�yW�����n~����d��L��K��6�m�-.<o�8,����'���6��'O��}{||\�t��\\kk���������lI� �q����D��V�)��M>�8����b������hSR���-��cQ�D��6��D�K�F��co�K�$�um@ynDDT����.�K�t��O> 6>z{{O�8�6099955%��������������������KSS�l6+�1e���)���A����j\D<�y������'NDD�^��8���������f����ro�
��)��m������O8��|�����n���I|o��������8v���tww�u�/����d[������b���x'�l�o��*{WTq"""�������m�qm*�U��V;?Q�$�=+d�hDuDxm�ZW���<oy��<���Sz�����$���T�(�����s�z���>�I}RmACCC��N��
��������8~�����m^q��A}DH|�&""
e���f������� �H��}n���]���Pe����D��x����q�a*m����S65�H(�2�h�c<���6ZZ��S�2�k��m�
���&��]�]DD�:���R)=T�PK����(Z�V>6k�\}LD��e;j�O�I�.�2(N-�J�##�lV�#������#��]a����J�Z��9^�p2���f�?DD���)�u>�wTL}���%	8�<��ko���u�����D����]��,�����%��1k�\}������Q���q��k��a����~THO��	�\i���3�h��@��b�h��9��hN�0��\}{m���pb���(}�����f�:��u�g����I~�W����U(�>��S2a2p���p����S������w>�+�|�F���Orp6��V�S������\/�&���O"�z��R�T*���U%�f�������V�GFb�c-����E*R����1sJ�FrE�c
���J����(~�a����8s
�fry=
/�v�|�i�����*I�GrA��
��O������hX������S����/����I~(7����v���H��� ���T=���
+�����x�i��S��d�G��m��W���jxmJ�B�$a�DD[��#G�����d���������x2����IDDe�o^��l��]����]M�;H�<f�����;>�����T"^\���H��#�t
r�1�~���#�[�*�>bq0���N�\��@��6�������\aoC��S$>^n�o�����k�8��E�.������S3����g5�G��Y���#eM��u������g�3(��
�9ujtTP"xk0�s��������DH���W�iq��v��yc�ly���f����|l�S����W�<�Z~�m�m>""�1�d�6>H\�cv��(�.��${�m�����<�����0��DD�!"�"����m���c����WEphh�u�s����i�]yg��]�n�r]�����;w\�=z�(����X,@�����{""�"���M����p���e~~�q���f����m����C�7�"��7����P��A[\;�:��3�O\<O�<�<�k�T�g�
�K
Qp������t���]M�����0v��eV�/_�|��%3���[���k�Nfr@:���`#���C�h�j�<�=��0[����H����Z�(=�J����N��������>��U��D�-�?�K|;XT�-�,'>��X#2��t<��v���X+�.'��]�?P���hN��������iG~����3�S(U��9W�V��OV��lqX�2���>s�[���������Pd�U{Y&2�[>q����;>��&'"2e2�P�9�l��?��-����s���-����p��E�u�\�"���]��bg��p���x<~��i�������(wvv����8�B�v,�=o~~���9��Wi��hK�s��%��m�P��-O����� m�x[[>�e���I�A�gN�f�6L*9��hE�F���Wz�\q�+�v�������b���>�iC�
l��[���q
�v��A<.�b��|��a/�O�V>h��:�V���H\���ZX����^�|�N>�����,�{\<�������=�X@�S3����k5��>�I���X����\���i0�7�3�X8�5���tn1��7�/��,O��Y;}������~85��f��}�6?�'������}�����=ND�)$-��K��d���M�E����%�hDC��'�_��� cjF��&�I�%{�=��.-�`a��ODD�8[58l<�v,*��x�Q����:�n�x
W��0|q�N�\"��V����
�=,�r����E�E�Gr����}NbX��=hy�N'6Y��_Dbx���a�*..�� "����_-���J�RZ""�;������]�a���N����?<��!l���������w58l<$�Fh�q���71j���x4�i�Fp ���XC'������,��*�eX��
{_^|��v��kC�E8���:���D������*���L�o��[0{}o?�?x,��<��EzP����{�-=��u���#1�YH$����y,J�b���{F��
�/��K�mq�j���xh�����d��L��=�w��H����Ur>����6��j�=�DD���'o��=>>.v:Vw����unnnnnN���)�w�����s%�e��1Z�&��j>1�mY����o����*v�Q{<,�vl[}����+����a�j���������at��H�k�:X�9�^���}%��j�b�
�8�� K���:��'�����D�(F;�q���Gr={�qg�DOa	�l��<z4 z�
����t@�� 6>�E���g��;:{ �3��>�h`O��y������Es�����;k� ��3%�t�����N��(5����������~������D�z�">��O�j��������?1^������z^�|l�����i���
������������t:��'����'N�X��������������~]kkkkkkKK��a��i��
�/���_�m�%�=Q���q����`�m�OX�<��xh�����:^���x[\��m5x�u IDATh�'"���������x��W�J�������7fM������=����J�7���H?�i���>��@O��"����x��x�_[el�U����������q������v��h(c��O�r�:>H[�����c��	����P��lk������It����m����6���Q�������o�����3������;�)�������K�o�u��0k������ ��&��j����uj`����v�:���GH?��7��9{���L��h[3�}NG���F��o� �uEe��(~>c��mq�jRKf���U�,���F-�j?/��mq�� ?w�D�@����6���!�H�����066������������c""�[���Y#�T�q����vm���7��Fr�����1'6�P��n��D���J{W.��{��~�j�>�F�v�,���6Ba�c;/���lylq!�)��+8�G�d�e��D��K;��3HDD�����z�J��P���c"""Z�V>6k�\}LD��-SzV$e���0�0	�ylq5�0�>��������(�}���OJDDd355���������d2�NYU��h�=���G������u+�5b�>&"��&DmD~C�y�#���������a��6����=g"�U�j��QpMMM�[�|��/I�/}�*O�
������z�V���m��~du,,,��HU;?Q���
��/�0�V0��j7Bj�?�X�d~-h�l��<'/Ryvy���Y��SQU>5����OD�59r$��������{�&"�H8��?�c5��+����?������o��2�R�;���Y=Z0���eyQ�3���F�x56	���L"x��'?,����������R���JNF2u>��h��q�/���z���������O=Z�{���m��x<���^�*�CCC���;w.�N���;s����u���������������GLNN�b1�������D[[������8NKK�����8�������Ii�V�w1-�vy��6�y��uj5�#Tm0,]A�KDDe���~����h���|�k��/^��s��Q�������\�������(B%��z�����,������###.^�����+WD���k�X����n����O�>
{�X��>����N���w���C(��������8x�`!�7��iR��Aty�eP z=�����'��M��Z�6L*9�����}�������h�����q;���c_��6B��d�|x�%��[�)�\Y�������
~�o�V��1��7q���-���������[z���VNUY��������(
��Ng6I��DD���S�������k�a5��t�ZR��V2c���
���)���Z�W�|��O�@���k�_����1Q�Y���q���#Q��D-X�	R;���6�������v��7v5�n5��7v�������
�u��.dL9:>�*�Q�v����jI�H�w>��
m���<���'O��}{||\�t��\\kk�����������������S�����/?�LDDu����g;��]���������0���rG�����z�dL�D�`aaA��Bm�mX�D��������f/\�@n|,���O>����X���=q��� �������hOOOOOO��Z[[[[[[ZZ�/�LF4�������!".����g�$�~���	�G��]����DD�E���<��F��c�W�Q���K��UTR@VW������(����>��`ZX��_\����]���~��+����=����>���'""����g300������;&#���k��K����6�V�:/����M��}���a����$nka;��6
a��&`��������|{�v<[Ym��~��}���y��n���^�{�m~�#�fp����~�}LD��T�����DDD�e

�F:�V+�ccc��������2��y����DDDTMq�C�G���o��T��q�.ldL���71j����~��LDDT���?���K�#|�1E.��PQ`��7�������7[R����>V�C3j����Q�y~"���%�DDDT��:�c���z��������y]r�'"�-������[����nJIFgH��A�����FY��:�'"�ya6�����A��\{��C�Z r��d�Uc"��,�jE�>���<)`��r�6T;��>�|`�����?&��I���q�aR��DDD�)�������PD�K������%x��h+[��������|Y�w�N=R��1"_l�v����s��A"�u%���
��mI��DDDT}}}��m����X����"844��n,��b��������o��%�=/��o�������9�����q�>�@t��w�u]�ymmm"(k�<��=������n^��d�0A�!?]�Z�FV��"�����<�<�j��e��6�'"�:���Z�j��Rl��#�caaAE�����j)�L����'�I��fDR����h]�~����=����wj����n��!"�{sz{{O�8���;"����?�`zzZ4:;;���E��������kii9x���hnn�-=�!d)S4D\�E�'�+QkU".�+���W>DP�%���q��&�@[� l������~���c��^q�������j�'"""""�����x<�]����]�����������1�Wo2fUje�u�/<3�M2X�< ��"6��d""�w!����Q;��>�M��{�{~"""""�����!_���z����#��+������2p3�,�FX��$I<�+�^�/����F�����~duT��Z����������/j�W���������������������`����2�yDCVW#�_K�u��ZD�D�,#NDD��n�c�vl[}�Y������
^�p��+W��b�T
�������;vL��W����*����N�������������?x����x������L&��dd��d��a�@�����Y����$fW3��7�/���TF�^�����8�/�?��?D����]��,�����2`%3��;��-���rL^y��R����/����)���o�t�'�M��������-�_����j-�����e��?��gz��o�n����z��o��F���)�DDDA�o�K_�G���R�T*���U%�f��j������x�]Z]�
��v\���3k��W�n���[#!j�_&�D�X�G���Q��*����<�������7v5�.t�����|�Q�w7��x�RU�v,*�f\?2�j����t�_����n���}���N�^���>�D!~W[�h��Xm�'�q���E��9��8��3��8Nbx8�^���_&�����J�������"�����������g;��]�����w|�_K������W
]e0k�5]}���]l�����R���k��8��EO.�� J�
8�C��|�R]�Ez���0�g��fr#��l���:����l����\.�����L<��Q>|Xr��M@��K���������Q����ot�=f|wS����������_�q�+<�v�������E�E^���B��e~M!�����-�k{�o�N9O-��_��
�=��f��m�@:�����=#
f�x��-���
��9m����s�������H777'�@�����Y��L&��z�q�7""�\��z������o_����|�,�c����
r�Rf8Jmqaa�W���	P;.�Hk���t5��w��o�g������������%""""�2����!Ess��H�kk x9���&;TxT�mq�J��M��0�q���*�-F�w06[���\�Y\�@O.�H9/��~j����}_\��q�����_�:���b!���\�o ��(�r����{^��y��ADDDD�����m��-��b��W���������X,�������V��u���������"899	 �9r�/���8|����w������hkkAY;~����:x���<|�P4�ijj�L&#�	j�S�5�J^�����E�`���d<����6�<�����c�e\m��'mp���e�c3��+x������<j�������0j������up����~`�[�^���w���k���}����p.;��v{�~<������{������#���=���+�O� ���c""""���_����w���?{�����7n��������'N�s���www��)���������vQ8��_�*�r��������1���f���d2�Y�q�<�I��C�C5�|��A-.�D�O![���R������J�2{��Z*9R��C""�_���]���-����J��������>6j����t��r�X���K�f��rkmh7"��?b�����q�H�� l�x�7��q�{D�K&""""�����x\����}�u]=
���:|����x����L&���Z6U����3(�T�c���M��<�>�!"��)��<j������ ��\5������U�R=""""��p��u=����W9r��s���v���Wn8��*
�Z�,���<k���<�r>��Q��Q�Z���Y#���cC�k�����H%�C��h7K�x�U2	��x&����;7yH$ylq��k���7�#�l7 q!�8s""�x����W�\�;,>��s�q�;�w�4??e��P����zG�{�l���{YT�q����<m��"�g|���6W��G����X?�:"�;�P�����&>W%dD��`[�'���][��_��]���:~#�4Q��p�����T*�������d�X|u�����X���0;;{��}������.����P\;~������L&��dd�����b�|G��e�����jC�����d�c������`��E�S�A""�_�����XM���+f����;_|��YS&"�Zj|g�/�4�G����������y�*[��
e������!���<*�<&�� q�v�������
e#�����d2�P���l��?��-����s���-��o��C����������(B���/}��-��m^�Q;��>�M��{�{~"")i)qV[��R��������y�� �~v�C����y��D6�C|�j��S�x�W�����<�O*""���������RX����U��h�Y���Y#~��{��������v�s����A�-���0����1���4�����O���m9y"9�s�-n�����<E*3�v^���p�<o�0IvQ���x��dm�V>��s�q�����J���?��'�$f�I`�7`�����+�)�0�P�h�����=�x������\�x�}Qy���\�=|���Q1�E{��g�����`�
���""���ccc�����������{����T�n��tw���x�1WW�����$y+�6"�'��;2��J��cSF�Hz�.�N5�H!��z�s���gm(����v6��O��^��^o�����y]��������B����]]����Z`u����R��d�2+�B�Jf��OS�.=^��1WW�����$y��6�FXf���<�����_
�c���O�g0Z����l6�������	;^%��wTA�yjSR�{��mG��CUV�w��|��U.�y����v��x�����}��?���������M"�������
v�����s�q%�����lZ�MD��v�c�����
f����������$�������-��h�J&���A�j�K�G��`��d�z��zR[\��6�<�&����1DD[D__��m���x,�z��

��{���t:�����g��]�n�r]�����;w\�=z�(����X,@�����{"�8���6077�q������y�q���e��\}a
�5���	�K��a�-��e*��B���ao���<��Kw������\����Ou5�wW�)�o��t�!�
]e��k�G��q��y�����w�=?m)������f�h��/j��L$����oh�������L�L&�kN6���O~�G��������GKy���FFF\�x�u�+W����k�b����g��q#��>}�����},���������8��C�P�������p���Bo��=c�/�>o4�]�1��=��m������9���6��m""��������J������l���b+�@KJ����K�k=2k�/^����GVG�k������h]������b�����I��Q-%�I�Z%a�g��9Kp�Xyu���e���Er
""�4�V��x�k-���w��oo��g+���%�?�����cm��2��cQ)������w�=?�z)����J�RZ""�J&�qG@E�t��t�il��hDPV�����h�cu��z��;��L\�����X)����W;�H�#�0k�/^p���U;?�J�W���\�"���IP��3���M�!����h<�A�+�����h����v�DDe;y���������N��������������=�#!^��{
���lV4DDS/����z��H1@�<ND�1�W������w166��fK�PH��������7�����|�<�c�>�@���M|>����l�pC�u�S���q��
�����%���]����"'��l��Ek>�DD�488�����%��y���������?�(_�`zz�|�����Q���_SS�x}�W�d����A5.�t�lyd$[���MF�}Na��v���W�+"��'""�oo�j����R)#�]���,��Y#���JT;?�$�+�F�{m�8�v`Tw/���Pc��O]�O�)��e���	EH|o��������8v���tww�u�/����d[����|^�=�<���+#Z�9���v�|��������U��hc���Z��s�q����HJ*�T��%���%8�|lq�`%x���j���Z�U�z��c���eu�>""�D���D#�N����1�u�z�����xX���GT��F��0""����|l�������ODT%����}�V�L�[K�-�����?mJ>q3����
>��&�_�f�2(�f�"(�����Sh<�uD\��B��*w��y=H�*��U���Q�+<����Z��[V�GV�����T��'"��d2)n�d=��gER&1�Lb�<D;�-��sJf�pN=�6,�|l'���*g�Y��".��!�����������_��G+��d����'��aE�'��:o���#��O3.{�'"��\=P+f�XT���~duT{�n��'"��'o�!�O��a���g���d��>���(�O&�����g���[U���������V�{M���J��Q6�C	�����DD[������1WW�����$���l�8�F�U���^���-.�^�%Ux�$������yd[��0�|j�z�����h}9r����z4R��e�d2�j����z�w�|*<�DDT	�}�W��������lIc��L��������7r�A��c;�-[��qA�K�#���W��������e[��C��aQ/J<T{��6����m�����X,v��Ur]���s�tZ|W��3gD��[�\����p���u�=
`rr2�p]WV����'"�+����D����qZZZ����inn����oX��g\��|����T�(5>i|����2�6���q5(�.9g9���=��!l�x[��h�r�?��jb��]�b�A.�������V`�=�j���G�l�����6����������f�h��/j��LDDD�L&����f����������=����R�{����/^t]���+"~���X,v��Y7n�����O���|,LMM�����N���w���C(��������8x�`!�7Y���f������38H[eK%j��k>��	2�sL��+y=���lc$s�-I����DD�����_��r�cw�Vet%S�fV�������5����������p���������'j���M���8��n_��a\�2!��/���������#�.�$�3\�����S��������s��"�)��>z����L&C��W([�w�^�ykc�����
 IDAT
�%S������
�_N,�E�d���y "���T*�J����*C�Kw�E�[�@����7��J��cQ)6������XD�w���/-y�q�`�|�dvm�����RE��/?{����]��`t�ox1w)����:xc��g[�����u�q]f���Ce(W��;��T��c�\"d��2�#�fN���J�>���DDDD�NJ(E�g�J���f���s|�yFB���YL���?oDD��+���l����e��7v�������_k���+���W���t�q��Z�4����.��t��������F'�����
Q�k���N��Q����X��-����t����/�������kG��U��w����we�B}��p��3��hG��r�g$��}���������;��rabE�`�_��^-��Z�f"�yH1���������D��<���<�??�?���/����?H~?P�����O�^m�p�<%~��������`MDDD6'O�0>>����Pw�Nl[177������)E���}���7u��-.������9�S�+�������� ND�/�K�v��Umo|ccc����S]ke���c����V�;>���;xs��������5������{��'�	t/����yhx����<�L�,ON�|�[|
U���x��O�p�x�o�R������}{/���Z6�����D-���������y��L|��r��
x��R��������
�/������������}8��8���2����t��I�3�g��t��"���=A�[�����q��������O7���t��Q{M���7��>��p�t���H|:��\}�f.��|���3����������ov�'�{K��8���o��}>��[<D5���������;���,d�����DDDD!
��qr�cI|u��� 6>�cI~u���i|��:@#*�(�}�/��������c�j&{�u��x���<����-b�x�v@����6��j\��l��f*��d2��9���MC��m1�3n;�-ND�������c`�hI�v��Rfi�����5�/^��/:�{Il
�|���5E����]��W���������/����j[b>�5���91�,_�^O|������Zh�,�n�N�jy����/G�y��F�ixm|W1�/�|������>���o��������Q�����|��f.���z�9<���Z����'�����v�G��?���/�-5OO��/�����	������?k���2��^o����\-��/76��m�8�x@�<�~��#?O�
~�&""������(�����;&#���k��K����6�V�:/���&=T`��b�����%���]�I[W���m|�y�����'�������w�{���_�Q=�q�;����x���X���G�8������P��h~�.���j������G�����E�4����/�������3�h��mq8�����M��y�v<��\���AV�����yF/`z���>�~��,��QE#���h��t:�V����\����?.�a���������9�������H����h�q�������1��q�5����zm4�������kj��Z�>u?���>�����1��������8�G�k�����}���0��_.������&?�����/�v�v^�	���y�v<�o����l��{�k�=/�*��/^���s�?���OH���`B�sx���`\/����^-�[�{x]^���x$Za��)������(�����;w�������T*u������'N��2����������v��%z����s���1�lI���d�|'���}%�En��
��Z}�6����.tK
�mJp ��"��~����������v��+���+����S�O8��g8��T�����|*�x �|j����~���<|8������F�o���}N��Im��������L��cy�yGE�����[���f���N�\����w�$��(N����������[�w$���g�������}�a��������u�v����<��O�H��e'����N}?���;\��~����"355���������d2�NYuL�*VF���� V[U����j3�y��O)�)����z�&^y��J���/�����c
m�����?���k��0�1���_&>�g~_�W7���m|g�/�d]�}����x�@DDTe�L&��m6���O~�G��������GK���o�P�T>������ ����1S��W���y
��O���B���~����h��z�V<j��\}l�x��������^v���o:��:;��l�K$�������cT��W���
����#zhc+��G|.�������)<�DD��[����x��u��~duT��
����:��1U������l����m���c����WEphh�u�s����i�]yg��]�n�r]�����;w\�=z�(����X,@]}|��=[O����8���9��������;����,{k#[XV�����w�2"�Z�T������g~���8mp�����e��~du,,,��H�{~""U2��e�����������������ND��?������o�[7o�T�����m[z���.
���_ED���W����X;�f��^Y�g�G�DD�/�">��r����<�>�eP�3�'nR'`K�m>�8m|���v,*�f\?�:��z����Q��JO+�f���_��f��F����"9�y��u]DDT�u�����V���LDDDDDD[A�b�Ta�3��Z����NxN{\���n�c�F���z�}LDD����5�P��B�4�y��ky�'>R���;����?����9P=�s�������f�(����Pw3'� N�<y�����q�����8���������9�'�:��g�YT�^�mq����^���]�W�A�8���j��vlY}�IDD[��������.%��S�>D�S�B
K&W�"/S>�%����m�O$`~�������f��f�h��p��+W�h]�t��O>p��i���=q��� �������hOOOOOO��Z[[���z��L&#�k�|��f��#�6^}�����F�>��o2��m|���9�s|(jf5."j��6[���6�����h�����q�h�d�|x�%��[,����xpy��qy����{8�~����w��I�3��K#��l��1Z������+��:�s��2>�p��3����7���L�}6�|�r�#�/NCG�8����7�����A�1H��4�.J��0F��q'����qDV�mr�L/!�j0�F��40s���(`1�>�C��4��O��������qor�"���"]�Y�z�R�������54�����p:q����+��
_�r���f�8���W*�
O_9'��	�wfn�<��
k�V6����
�����W���r�����kO���8?�a/������7���+��O��yj�G�;37���`�(�;#Xr��O�v���K�}����F�����9cH�������E\�������b�*����GK�V�����!�����
�l������)Y���{^y��\�t�Z]att�������y\�s��yY�s�/��KF]��	�-�SUX:�U&� �h'�'&&����T����e\��8`{uvu{��q����^���Wg�V{<�14�c��c{9�*�)�3��rM��3k9�^bY�x�u��dLM���\7���91�r!��[n)���)���4��SO��SG���z�6������3<=m����O'0�+�-�����+�D�>p����s��33'��0������:O��P��
�������oxy������wff���S��V�4��?>==�9?�;������sW�b�����+���v�����=��{g��SW����vx\���+����q	�|M���I�of�O�^�P����R0+�U�(�>�0aj�����q��}$]T��bQ��J�ZS��H��K��-U�aY��O��x�n�����c���j�(�t�x�4����U5�s�V�!�k*�K�*J>�X�����B:�3�����H$���w����H�W\�pA� � ��xgo?8��xu7���g��#O~���8~~�Sq���O~��D���}��C��k���f�0�4<�NLM�j�-o��z4��TgO_9ss#����h��t�IR�����t)����v3T�E��z
�[�{WV*�)���=���++��W�����*���v�N$��������k[g���_�_��
o/�i�(�
����K��-N�/4�c�����+����l�u���geGh��W�#���a���K���u�������G������bC���������wm�Y7Z����x������
Uamgx������+�?���<7�o^7����i�R������s6=��}]����9���>�7��BJ��|'������>%�UXV�(��b�E
u^�������2d��p�"C��j�*5�N�x��X��R[i�l���Wc0q��T��)��3��E��I2�?��j9&�IW1�k�W1�p��
�o�7�����AH^{�5Y��J���1AA�s��y�o��9�<�#-&��r��O|�|c��*hr������|So�n_�Xu0
I'�a��Xe�p9�����(l�p,=Z'�R^�#�����&o�/��hl������^���cK��2��++�������N��wf*�,w���^�������o�����������M���]��������o�2��~��q��]�G�]�>��ei����s��7��o�������M����]i���)��N��������:�8�<Z�c��H|��7�^`�]������;,>����6��c��o���?��Jt����Fo��������>��5��o�`�R����l�w������?wtw�s��$�F����]���=w������r�����G@H�k_�|�����������x<^��va��Vd[D���~�P����	��P~D�x^�	Ka�u��,Z�o���}}D�;�h�����&�KX � L,..F"��X=��q�q��}�~T����G�����A]�a{�b���}��
������������������$�9b���r&���>G8Lk��&�L~`���g�b�r*�Lio�p4�n:/yQ����h��b��d��dKW`�� ���]�@����s��g~�����|�ao���s��?W�b�assg�������:��=wjg~��37V0<}�����J��]C��w>���m/�����o��R���a�?z�����t�%G@��OO���D�:�t��U|����{��^���'6g����������y�h/v���s������������������jEH�����M,��|��l� ��� �w��o�h����c�9$u_
��D��wW~D��Jn+��U���Xh���
�.�F�a�ux!���~����?^\�kq)T����/�b�W������1�f�Z��BRC����X��D����������{����`������v///K���������[U��\� ���d��<��W������3��������p��Ds������z���`�d�n���/�v������h����������$���wj2PH��_��;���z���T�����n������l���U2Sf��=�{lX�?&6!�+��?t��m�a���)��o~��?�
s��D/KY����{G�p������+����g�?��M������g�������s/�^�����q����>�c�!����	YHm��'�-�i���j��_#���:�v��NV����e�XZ�������&�~�����1C�7���_��?>A0�y�Yj��X����k_� ��I��\q\�z���=���������NL�7�;�n�����V���b�r���h*�qS/,l$RQ �J ���Mo0'z���_��h�����,��r&6�����~e�����������y�2w�_l~���=w
�������y'����]��f�j����X����?�vs�T��k���������^!���j6�o��������L�,���^w��T���i���7n�G�y��k���
[Q���=wjgM���9%��P�_3�������8�
�}�����9V�h��\�mb��n
u�Ui^�:8z����{g&�3�!�9C����9������s��o�O'��j��E�����0pB�>(gfg��x�0xR�����J�����0T�+n(]2El"Fr]�����T6�L�dp�n���E�Y���ul���XU��%uZ�S�I�����U������'(�~����K����x��.�Gj+����$�L^�v��?�����E��7�x��7���[o��E~ �{�����������Usss���/��_��W��b�TX]]�"/��Y���+�h�EQ�-����(2D]�ML�H1���U�HA&��y������Eq{uv���������m<�S5�W4��O?��;\W��y*GBgD���������A$��9��I'��9��V�J�Q~��J'W�7F�p����;33��g�y�6������=�Kx�����#��&���I��wfn���m��X{���9��/�����U��<��q�@m��V�q�������o�����Y������U��9�P������x}Ll�3|��Q9��
T|y�DN'����~���^�3������dr:1=�_��M�,cy@��uh^g����������+G�9V�7�.���(gf��z�\��y�Ig��������y���jLA��Vy��o������I�
	� ������b�X������
���q�/��z����m6�p���H$���������D~��x���"�����}��;�D"�����������%���
��c������c�2>�<���j�K?e�����r�����K��V�A���/��O~'���<|�P���O<��>�=vl��#m���3�������L�����33�j�V������A-`��%�����)���C�$��1�3a�u���A��#�������x��C����?��6�����:�=(�������P	v:s���M��\#},"U�m�("A�B���CL��v��[�>��.���[�S����S���x<��qa37��+wo�"{�r�juW��a�4���_���Ka3��K�q|�{:h'�W�H�
	� �h'l�q:��D"?����Z��������9�������!X���[S�x� �qf���j�xgg�W��-�&Ur��������
�q�����+�r�X�^�:��_�x|��������E���u��eu�8y���s��"E�C��#.��v�]�������Q����!G�UF�#o3�������B����![�EK	:���4�h��������z������Wp~���;���+�Hz����L���_�SLA<Dd�]hr����rK�������������j� CCC
����F�]��O����J�`-|�l��KQ��P�.���j|(��i`���k�M������ng���u���aK���������^��@������{�=<w|��%��y����|���jqq�P������������?����>���?��cq�1{t;���AfoE���o�b��_P^��j| L>&��*�	W�Z5X�uK<AAhyw�8������'"l���T��{>i$�n��I |�3�������b����Q���@���i*��S���#u�/M�vX+(���&-��2��l��[8��`��~t=���5��|��c7o�uvx���cT2������Q^y��e	@%���3����.��S�V�D5���0�0TiE��� �xl����&wl�}������/� �����A�!��
^���E
fe�JE�g&Lm��r�>N����J�X,�K_�ZS�
S������_A^�u�t� IDAT�UT��Ox��NP��DQG�xq�bE�)����tS�&��E^P�D��[�cBj�`"k.��%��?���|`���_���o���s�X�x��j�(���VT�Mh},:�8 ����'� :��,�5G������������n�'�hv��n�x��W%��Y�/F������'�R���^��E�*orS	�X,j�u������70N���4?~�V��k�>5�g�j�&j+���EI�b���K^�}�y�2��"��K�����|L:��U��+��a�a�t��.�~U�-k�/���������D��)��h�M����J�d�4P��n�q
*&�N<&����� b�������`b����������hO���w��� "t��M���A�Mz+��C�������c]K�*��������'���gZ�e<l���i�g(�n��Sn]���>��/��`?#��%�����s	w^�]��y��7:t���H$�D������S:�>:������/�,*�������/���k� ���V�_����jE]���{��<Z���,������~(�=�c�~�n�������e��U�{���B�������r&f��v��u:21>�d�_�L���346���d�		�A�e_��h~���244�n]v�0�����!���S�V�2��(Z���"ZJ��P��	i�CCC���/
5��K��hG�mU3���n�����Y�^��a�	�s���'��$��O~����:����SSS?�����������}�����w�����/x�_�����N�U,���8�I� �@dbbbbb���'���>���L��e�@�^��{�U/���ts��*���i�������&)Mq����xvir0����T�g]�M����@�x���!S�����^'n��/���h����U@!��`�u]�u�y[�P����-��\N��K��Fd�>�rf-��cq��Vzw�ny��������R������?�$�>G-�����������~��1�#v�_465q�:a���}e���f���'
84��?&��z�t�A�;>�6|���c�������%%�NK
����/������D�m���a��|�j� � �D��m?8����>��qV�9���{^�������3��f�y����F�-eHC��{���}�����g���w�fb��l�;�{[21��u�KW�.��SId����H�I�����c\��|�p%��������h��40��Xh,S�LD3�x�u��luWrm�q<���Z	��\�u���[S����f~'����&�'���{c�W�����?��{cv�/��x.��~E)`�W8�P�d��D���/Yq�l������\�7!���^D-&����#�8M��J�&b����aX7.J�8V�P#�\,k�&V�2C��&�*�:���K���I�$��e�O Duv�_�Q���O?�u������	m�e�����X`X���1��[����I�(�_�]��.]�t��������?����?�U+VWW�
*�����R�Xey��j��3��H�4�@AZ��bo}������+�����������q�J����O�������\W��7�l���KW1r�[rv���/gb�N�����d���R*Z�E�����)�������L���]Y���V���E9��R�D2vHdK'���Y�9-
�8P&���^�y���]���82�d�*����F��i�2�fx� 
Ig$���[J��L"�:��o�X<1���x!��<��L�����nit1�75�ETDP����	��|=
���Nz��7�>[���v�^���{�f�m�*���_�����3����O�����,�;�nNnH�� ����/A����j��b���_9%������_�,�����O���?>t����7�����G"���G�tzjj����>|���+Wfgg#�������{�}���}���#�������V��
�g����/��r������������ge�q�Du?�L������H��j[Ui5����5Aa"�_����NV�0���G��j������������a_������������BEP���o��fm%��>�P�J������������c��n���F"PX�H�%/�M�\��6/�}4U�6d@��'p��������L�z5���Mgd�r5Y��`��u�\�2�l6~���x6[�k9sbpK)�3���;�xvi!�)�RQ�x��S}��`��j9��{�n�3U�(���@��������F�F�n���g7����J�'����u����w\��O$k�!A�KF�^��%�n�q���
��]�s��dyy�����c,�,r��y�A444��S�jiaV�� �8��d��<��W������3J�����������7 ���Yn�e���g\��b(��w�_D�?1�q�\����@>SHl��
�C������Br!70�M����
��|b�-E����U�T�Y���M��"R�ga!70��u���ON-�I��������U�'�%�!���g������UR�����h�&���Uh�������J�
	��G�'� ��uY�N2��O��OrE[�x�"�.�"�},�>�X����������>���X80�#� ZA{��<s\�z��K�S���g���-qD�)B��� Bn������wka#1������`|l`J8e������-�M��]W���;��glllL��M ���>(��L�.kr�Z6���*�������[�h����prld#-�PC!)�\��19M%6n
�6����<y-�.a��I��|=�'z��V�r�D��M�o
�|���[������1�����
� � ����c�n�ejJ�c6��~���6��������U���������P����M�
��� ��������9�<�g=���wf�"���<;q��UP��cYU����Qkkk�e�8����m$F���
�����Y7���<��?'��e��h*1XQ��wy������`�K�z�O���{w����EL]�4>�B�q��\%�q�X���o�1�C9�����-$YO,�g�[�~��< !Z��WM����uFp��A���9 7Ry0]m�Sg�8��I<�/����Y��c�D+���:?mC�n����u�&_���;��!� ��"[=Y��k����7�L��o���+W����������������|<77777����W����*Nz������E��;f��C��c^��T����P���,^��X^�h}������+R� � :
�������x����^8���y����JD�P���M�.�����@����b3������#j	��y
#�eA�cK�O�i��y-bqq@$q@eg����/���}�!� ��/����/�����W%�7��/}�q�<���-���	BK4���T~�0�;��J!���X1�<��)l�&_p)w���q����J+�:�*� � B��y�mcttT��[M��&� ��}K�����G^��������Q��;F+�����	�@���'���j��q_�������f����]P�Xo��:���y���:�LA���_{� ������#��b��jZ�{���	� ����A�������_�D"?������o�}��av�1����;t��w�������8���K/�u�A��"��7~�Z��>�����@��� ���,�5G������������n�'�h~SA�A�9���e���7�)K-�E���"+�	e	�;O�xbB�a�p��aA��������i����5����R��������AA�c�w��]�[���Vl�����bK����?� �����j�xgg�W��-[C�w���������A�t� �C�s����-��A�����;�H$���W���8t��H$��o��/..F"uO�G}��v�.0qee���0�\]]p��y�w�D3��j�#��nA�#����T�bY�����PRD�����A�OD��&wL���������V����!�?�4���.��T��L��� �WA�H:���W_}�{���$2^~���_~�|����E�x\:����_|���/�����Z�G����e�Ef���V;U�`���k"jQky/����Z+*��!��_rL:A��Df+���qu{���nWc�J�5w�2��.�l
��}I�v��N7��lC�����d�Z���a�*b��31!�R]V��2�L,Y�Jr�J��d2�@��I�c����$7eU&�u��AADw��!s�
���fP�I�>>�������]�
�HA�������Kg������2.2�"��D���2�
��������x��$������������g�&�[�[JE�x�u���`�Z���J7�*��Y�u����������e�K��\r]�Mzz!��`����J����2���������,kfV9�'���r9@4U�J��]6�J�i<a�:���t�0� � $�?<{��x�������SSSo���+W��A��%^�7�������y����_A'��o������p��g��B��'����������{�M������x���Ac����!;]��w��db|����hmL����S������?��J�TM7�P��K{a~��V�Y�-�j�B2�a��e<a��u������!���
�#��[�P`�l�d��������9��Q�0\E� ����z���2<<,|����>���hhsw�o�nO/AD����G�9^�z�<�e����_�����d~�$j�����!g���}Y��!;]��q(���@~(������2����8�/�u[�p�O+�F*�lFr��0F4UJ������N�9����&BZg����y$Fk�6��	��z��y����i`}�7L{��A<V���;?���X������/~QS]���(�B����������x��|��G������L��+�g��O6�)*~�jE;�/�k�h��/7���\r��e�_��0��U��`�RY���ADWs����#��W�������q�[_]���}�'|�<��5�R3�d5w������k��n�����
����%������]�_���}6�&��3��BhH�Sc.��F��T�����+���R��T�e[������<��*��|YpL�i���l���>�������L����{��!�h?���X�?��������%���K���o[��2���^z��YY�m@F%e�	��A���mUW4�%�U�,��V���k/Z�bNAt8�(��/�<��{z�`{�����OL�_�{����4�;�n��i�qM���������^x��k ;��Y�'��\�
�eG�|kcr����:Q�������r9�������]�N��"R�������ON-��C�K�a�OK��:��./���	��������-1X�sQ�7��%C%Z��f���`q�^]i��~M����4��:>&����V��I��G�gaj���_�R*�1�h������P�2A�	�_%�o� ���tr��#��z�q�Ps�;�<�5�3�u����!;��?r[���ollL��M ����������:�u]7��P�c���S�������50QH�C���oL�iS���?{��Y������yES������&1�d���^���>&L�M�u�a������U��5a�7�������K^�/��/&�I��c�W��1�.51��tv����)^�	� ����>�	� �������5g�.C|���������������r�j����Xn
��3�x�+��������?1p�����LV�B�q������)�}Sw�F
�����3��;�x�B�q���F�X��r&�8�H��h\����n�l_g�=H����M���8�PH���4���`��|�Z�|�
�O�4��g�����nmX��;S��X,���D1��a�0�q������v�����?,��i]��F;/6�������}�>AAAu9�sf���(@<�B�V5w����������������������g]�w���"�u�lE��)�����jg2jM�F2�U�~@�>5X�B!�u����Qb������,
Y�%@��'�oK��~Mt�x�"�yu�AAA�-Yh��q;w+������!;��O0xBJ-h��YO:�����&�����H#��4�/����A1��;
i��ly?�0�7,�'~~�� � �8�8>�����O����c�v=Rs�AD;9������x�F�k�<��3�$���`��9��M��SQg�Z?�����K�#%E�1�k������e~�����U�x����pQ;b�����n����EI��E�����C���h{�D� ����j��b���_9%������_�,�����Oe� � �0���'���F�����O?-��{{�8p]��z������}-�o���t��vr���\r�����NOX4:���cY""8��������$��JJAD��/���>���A��	���[}}Ml3�2N�x�����A�9���>��qD��&wLg7��!;��'� $���N�[�IAAD�8,�B������t�ekh�_������������{7�=�������u�t:d�u�)a��B�=`X��h����h�m�y+0�<�t� � �h������/�����v��4x
F��tK^&�8-�CCC���F��1�L�����40�&_�&�����I'� � ����0���������]��[���5����Y�X�������/������B�a�2e�*Y����:�31!�q��exr,T��]MU���f�j�:�(�e��R�E�YXaAlX�<o�d��&�2�����(Y���.<���)�?/CI�n������=����E{�IAADc�����>�R�����qqb�8���������+u�^.�B��v�$��t#�Eamm-�Z��C�v��?�]������@4���L,�kqD���-$c��N4U�B�2������eH�\��|b`����l�]�n�VZ�[�T��tXd&�UZ
IgK������R��������B*O���L�2J���X UZ���	�X,3Z�n�7���nR�8�������u�O����t�������[�r&v���r���~�������l@�]��0���C���P��H��[�dM�i�X��bU�~����"G�7!C���)��x���S7����4�@�0�kuqb��2�d���c���
%E+J�b0CmbE���H�(J�L>����?���5��]3nAAa"��o������p��g��B��'����J5G����#��tU����B�v��N������-��d;p�G��L!� ����Tii`�/Y@9��=�M��t�"�R���HWR���JKS|7n"��[sFla������}��au�g��7k�O���������
�����@�bQJ


�y����M�0������Z���!��E������J�->ZX+�V"���7��)���K{a��xf�/�g�j�C����s�������W�X+U��i<�RD�����>AA��D`o���g�W��3�c�;�b��W���2��T4}x��;6�>�[�Em-��_�����R���@���@�NZ�6�N�9�HN���������O�<oX��`[�+��&7�W����%�&o}����>������r����X��GO�������b~`,����r���H�@��� �c���"fx2����f�+���v<���:k������
��D����NM�j	���~��{��3� � �����yf��m��rv��W���%�{��O���H�ZF��������f�[&C`��+~o�%�K����?Rs�E=zb�QX�H���_�����Lo�R[��}��q�^�m�����0 IDAT������n 7�x����V)
��s|l`�od0�f��)�D�>������n��96&�*y��lf�g=���������C�+����9�)u:
�3�u���2����X����4�_���a��~����m<AA�(��/�<��{z�`{����q��g���C-4�c�������j�J���W�!;��Y�'��\�
�eG�|kcr���U�������B�/��r�]���2�����DG���T�����
�w����3'���Z�����������'�	��>��r�J�@!�PGuz�>��;[������K���y�4��A���g<~|xs~)�2�����c���xL�c���AAA4OD:�@�|�������{{����%�5G�����Q�;CXg�������/�#��������X��r���
���y�gb#X*�����N��������Lld#-%�������[�'�XN�pkj�������6���w�2��'\xA*K�9^f�����+!���_JUb��l���j�����/�G�u^fUZL>D[o�%��)���K������@>&�/�bPq�n"������xY-�����mY�4a���i<\�_044d6AAA4���/o}�3~��(n��.����8q�x�b}~�w<yvb�r�EC<����L���u5z�������
�o���t�!����t��9��V)E!��k���������%7{?�L���q4��f�r�Z+K�U�V�i���+���tn��o|�xCo$e0�UJ�_�G�,Ud�K['o����Q�.j��f	��W��3����O�����Z���x��>��������	��>V���z����?���V�����������O?�%� � � �_����NV�y������'�xB�}�{����G-�AV(}�
�=���it�8	�q�~NOB����1AAt���1}��v���>�[����5Y
��C�v��� T���!�u�2N�x���S� � �m��v���wvvzu���5���������v����\�j�;�k/��k��������&�~�Q����}]� � � �	�>�h��K��C�v��_C9s���8<����4"
d����B�jX����Z����^au�AAAt8������/�����v
�T��=}�U��o����;��yL��x�&k�u����R��AA��`������a�q{2�kkk-����������cH8��j"������[~&��m����[��[��h� 5��0���@��/q"\���g_tQuX�����V^]�D;���|�M��A}D��b/�@AAD'��v���wvv����5�:wF�v��N��Dg244$&�xjL�/s]�g�"��o�o�o�be������	�8M�0���#�M���EQE�}x�Ui����\�}],���5��U}D��Y�?>���X�$� � �������}I�v��N��D��]�!�4�b�����x�Z,^V��K��a�%�������y2s1���;���������-�O��'A����_�D"�:t�P$�D"��r��Wann��C�^~�e��������q�~D�:��A�����e�'�^?�������_��8q�8=luPs�;;t�q������&�L��^��J�(+��H�S�R���fb��]����L:y,Mg�rEK(�;#Xr���[����p`�4B���[��_�8�P��m�D��Lz���������O�'A���k��������233�k���}�{���.]��2��B�_Z���k��#���������2��Q�CAt�������Kg������2.2�"�W���{�������1���U]n
KW%ammMR���������O��&1��z$��3e�����[���$r��H����,���lc(g�r8=��ln������z������C�Xl�T��E�����X��{��9/S����|A�zPx���
o�>�lYO1^T��At#|��;��S[��t��L��������X�C� � :������W{�����R��[�_���+yo}~��gX�&y��'�Lq��c;��9���t#�e1D�����V�B�Ys�=��L���R���_5[��.�]�U��aypp�Rec���yr�b��]�i�+6��bx�4����M.��x9�����E�*\c3��7�WK��\2�+���ONN�r�A�Q���M�v��3����O�	���z���6�#[�4xV���m�VdX��h����d����>��T������S��T�N�1�[Ci�]�w��%E�N��`	6U�:C�3�evuu� DVWW������)Y���w��gY����~*K�7�x���C?���������������,�������+,~nn�^��_��q�o}�[L_\\�D"����7����>��^|���?������0��8������"Qd�]O������$)_<,���0�{2��V�=R_D�B�4�����G�D	� $���_��w��(���k��x>�<;����������O��z�E#<���r�������-B�����W��C�v���&}�R���d����s%
�"��Igd���fbN>Q9���tn�d���9���?�BH���7�(��|�����%kR�"�ygd'�u����i��Lu�M:����V�v\F=���e� � �d�;���`��� *��U��>�g��1�w�y'�H�W��>���/�����o-..���:���K/�x�A��c�q���Y�xxx���*�A^]]uG�������C�dh��T�X�%+�UZxC���	:�.�P}�
�KY�$�x<	����y[V�0���G��j������������o������OO��4q�{��+�g������wv|�}�$��}�T2h^���%����������s#�]���V)���Fm@�h�����x��M.������R5��.-�2�T*ZX�H�%/�M�Xx��S}��`��j9��{�.���P7=[�������
��	�K^8zb������`��.��-�q �w���u�!�a������� �6��>~�m�7��G�
��Xeyy���,�\�i�AM��[�X��gK��0/aih<>�� ��������F9@�y�9�����{g�6�'�z[�{�|��BMP4�c���5d9Q�����K�!;�?�� ��	���tln`,�g]��PH:�L3O�������U���i~A����`p0�v0fA�>��7F<n����/�rrE]��4c�N�����v,�gU�-A�%,���g��d��AD���zm�=G��=@���?�`vu�������W�+1�Qs��v���;�}"�o������)=T������e�C����F�����%`��Ol�2�xE/,l$�Q����Xf��@<R���kYw,�8��&�Kl�/#Ul���^`����4��(�n'4[�����)��it�I�S�#�G���]��)�%�bT�����8��
q)�E�U�!���:�@����0�b���2E-���l����&]�:~���������9������1~����g�E:��aL�6&}�0���7����?h�	��I'� ������#���3�cy�c����_�T-���O��y����-C�6w`mmM�l��C�v��?�!7�8����z<�&�}��8�#@�N��A�\���������/�8B��?����r#��k��I��|S���ks�������nU0��'{D^n�IX�9ww�/�)�t�Ey���8=��m�u�&_���q��v?�.E]-��	��Byv�>�����czQ�*Q?��hr��:�V�n��"�/AP�����o�M�V�e��}x�(V�{�C�A������
�t�����x��W���{����v�q�P���HX^^���`' �E���!~4��\�}�\P�����^0����$[b<wch�5��5��X%���!�AD�q>|(km��'�P3�����<z���	��vX��`z�]\�yj\�Q���M�.���tn+�L:����RJ�}��t�N�}����y���dI��/���Rv.K:�2��K���*�]U�`| L�#�[��u"�^wu����Z���KQW[�����(I)��LH1�����<X�%��j}hN]M�O�	��y���� � TB�E�����j���v��v�'���������v�3����rS�B��������lh@��&���q��{�R�����&w6s�/t�j��n���?��o���F�.�A�D�QZ^e�*UT����C�`�����M�`�����oX
�R�n"P��i�ZD�y5���m�|	� ��d��#� � Bd���j�xg��g���������t�?����������g�iYk�q��5�����v�(I�)	����4�;���Eq$�b��<������*�)�������1���3����x�V��R0��bm�<�!���j�0���$\LC���5^G�����A]}|A��O�xC�e�����1�>n��C�vZ�O
�'�$�3M\��}�=�N�Y�TP������L�T|��>�V����z B1	����7���
4!� � �������6��[�X����f ;�o�����������k�'>hZ���������A���a�O��v����R���&��m������;�_@�)������hGhzAAq��B����
�����ammM�B������n�'<C���Z5�'������A���`�6Y
�����A}�{D�O�b����-�����&��s����CAA����CYkO<���)�=vl��#5�LA��3����O�Z!��^����?b<��(y��Zm^I���!&,��*���a3LV�n��R7�4���������K���R� �U��M`�G'W8j�e�Z�~y�TUW���P��C�j�b����Z<���A"���������O?�%� � � �_����NVe���O?���;�����u5z�i����o���t�?�u�M�q�&�� ��"��JJAD��/�����
��{\�^����\\�������
����9�.�l
����������0����;�����
���dUN&���2�^�U�L!)�H�bk~�,��h�x$��h�vD2��X|���JO��.��e�:=��M��1��T%���XH:�@<�	�uAAA��Adbbbbb���'���>���L��e!������7��;f�bU�[���UI��W��C�v��_�NXBb�F��b���A\u1���@S��IL.���e��-$�|b��[��HNn(�.M��\�������������w��L/
��Dp<�Vj�\�u�q��K>��`��J4U�:0;@���1���0�����7������ ��
@4Ub�uK)�L���r|y�Y�R$� � � ���0���G������g��#O~���83~�H�
���wvvzu��2Fnc�*Fns!����o���t����l.�zkM������t)eb4UrS5a2}'��D31'��*����	�5������8��\,�RUG4U*�*�`�����������=�/y��v?6�%7���}$Ix�[%�u������PS� W*����&0�`��H���X�|Pie�)���.u���wl]:x� � � �����?8�����s�y,{'T�=���%M\���q;w��A���K��C�v��_Y��d�ul��6��{�/�{8�������8�/�-���T�������������OX2�"`4�����T)�����h<K���T�R��DJ��`��������d����he�;�NAAAQ����yf��m�'&���������j���>�q��6==-^�	�p����v_����9��#�]�~��L����j=pz������/#7�xY���V)
l��B�gT�*�����F��D�r�|�ZUHN
��IR&��Lo�R[��}��q�^����'~-q�r&�n���d�31'v�E]xBP,�<�)��A�"��M>��S7�IL�T��1-�I��S���AAA`�c%{�����rOl������=��2f���������0j�J�����/�@�5������,GS�i_�Gwb8ZB�,�CIW�`3d���,9��I,$rc��.�Z���(,���U��
����a���d[H��[�����X�gnW����M���	YDS�'�|&���3+x�&�5�������v���Q5�!"��QI��x��7�jC���&Q�Q� vaB/6�L��Q;�XAAA0"�I=G���p-��7��#f�T���2,��Zq�+��!;��?r[����D93�83�y��~����|\C=666&����&���d������G�����7
R�P����?�R)E4UZ�r<�/��M]�� ^H:�����oL����N���U�t(	;��SM`�1��E���	o��O+��`#�S/����M�	�
����K^��W��f��%S*!u�LAAq�8��D�s�y���zW�^���z��7��;f�b�2��3������!;���2��3V�a8���6����3�)l�����	�c�<2���sr��
�&�w�>g��M.�V�w��z@4���(gb}Sw���Oo�R��9�Fz�t�?����n��4bcR�E�R	�(��9Sw�������37sF,����������@<�4����NU��[���
I�?yP��9���]W�6!(&����j�A��_~�
�}Z
��4~�6��	3
�|�+h	 � B�P(8��q��^zI��$��@�����;u^�x�H�I��d+^�M��4���t(�"X� ���3��e����	E���7��#������r�PP�.��;#;�o���U�]������tV�l.����|�d����k����[)WZJ�95Uuk
��rm���V�A��������@<������*��z�����j��7�}]�"�!
uR�Q;BA���|Y�%� ����E���7������o~�_�����K+��t,�[M�X}������`V=����n�/��A�%"�B�;6�>�[����5Y
��C�v���
����l=&�����3�������W����e�t�����L�i
�������M����"�W���E��B0Q5	�A<���q�%O�h�����P��;���UZ�1������n5��iu��_�	� ��aYhj�����Z��������n�o��xe�,��	'$pE��/�������Z�����5�A1�p��Y�I�|��&���<���L�~t�G�b�0O���;5��8�x�/����� "t��+>��#��8�s��&�����q����Jvuu��?���������T%�&]�e��NQ��

59�n�o������0�&�A2E,7�v��g�]��y�:A�'���e�-<����L���u5z�i����o���t�Q=6X��	���_�6����~���'������e�`4s�4>-5'�PY]]
��S,��+�d����}����e��~��,��}<::*�,w���/�����2KX]]�2�,}���ZV/%,���C���/]��V��9Zs�C��_E�D��!����h����2�71��DR���2Hu}�2A�#�_����NV�;v?z$7����^hi/�o���t��W��?hO��k�_�D��n��}�k-�&E�n���2� �C?�U���n����2��4?/�/+HUui���]���jU���k�-�&A[ul�A�-}���>n��c� � :����kU��A�����	��y]�1�(�3���� ��3�X/#4�`�W����u�e� �}K�9b��c� ���_4�o��/�b|�o����D����7��7 IDAT�y�@>P���4����~�i�i5��� �h�������B������Or�<��
x_C����q����o�gS����O�X,��^*|���A�Nw�}���Y=ADg��7��},~Kf���Z���Os����l����4�4S/-1��v0��n�:��R�'�h���ig3
�{���8��y�G��?��_2��4��L��;Q��v<&�>��I��3�j������hRP���s<����Qu�%�Pk��2(�PE=��uq:�D��A>	�}�g������g/���a�����/�^�8����qc���s����w�����LF��F����y�?}kXI��+��>����B��5���8����TO5��W��t6�~�|�4y�W����3��}��;���{���\M<T����?���-������g����g�[O_I���������=��
7��u	���g�[p�o����_����3�K\��_)���o��#>���i=N�te<�	���P����f�������t��z���&*����o:�7w�y���QL?�+��
�������	���b���9)��M���|����7Q���_7�s��������4��AZ�[��s�$X����jY��6}n�t��"�(jT�aj��0�Z�a�2�ZL��.^���0�k=�Kh�$Q��&��1��:/kE� [OLL�[�_�H{����81q�������(6����7t�����&���}_�i�����������?�!�r-���W�9�`���������e�W�9|k��e��G��jMC�y�IG��9�<�?����Y���?����+��X��+	�we�e�{~�H��g��b������*���$�������6����$#����������oD2I�Rrq�FDy`����`2�]�A��L������.�vW�w`&�!8�=��/���_�2�R�aU�^��^��9u�����y@��^{�}NUW���*xa1�
;�AW�}�m��<���g�n������es]r������;~�>�a�����?��'0�-������+����c���������0��������sG���
��sG�F�~x����;���%���G�hTx����mE��n�/�v�����^��(E�	Zu�x���Z���&����S'H��	4� #����T�����/��q�����?���76l�����D�&+���F����K������U��W�Z��n�p�z�����h�w���;��k�l������m����l�4�wf�~��?�7<�����kG�q��.�<�X�/mw���t���Dr��'7��
_��/{T��~r�+���/`�K2���3��t������{p������Q�};\�W�z�y��u%��)]����A���+�,_����]{��t�����Vy�:\k�a������U��A��Kd\c�������]��CK#<�M��c�3|uvd��`�h}5�'��tG��~�����P,���������v�s����n����l��z���iO6�CD�@5{�������NDD������;�X;���K���}�����rV��b�{��vYi�in9C����ri�:vd�����O\�|W$I"A��$	y;������R�]}���[��Bn�A�������x>������s�8~�?5��L�~��e�3�WS��>3��s��G?m�=�\�voyQ��c���wtX����q��i�<�wKKD�����x~�������k�x�o����C���i��[K��l<�����i����4��d�n����?����@������v����@r��q�w�-����[lp������9�����t����������;�z�����������[�,�E[������������-�����W���>O�6Y�<w���������b�k��u|L��Kt�
�p���q�z��fCx�����y�B��:���?�>�j���o?��wl�����p��-;����oi�n��������.m��:>v9���W��]��������/qSd>��rs����7���6�q�m��Uk������L�e������|:A�+�y#�||V+�$��[��S�������!'��P����r���>��w/������1���#�.�g���RV``E^�N�<�K�Nzz�N�����[�:���P[ODW/��c�Ko��QAW_��z���D����k����/���_���_���\`{�hl�a�K���vby]x��O��s����_����m��'�{�������D�F6�G��k�������n�������;���m?������Fm��Z{�p��<���h����h�G7�~�'�����[�������o������ny���h����Bc�\��{7��B��^}i������?��sDD��-"w�,���U���X�
;>}���7���{�[Ca����g���q���|��S�G[���������������w�����=���sW?��Z�<#����f����[��D�uO���
7Q�����r�����/-]w`�"�-���y����wl�����W~��+����tz�B��w�|�6�K������6�����x�w&gQz���(��Q3��W��X���X�^�#���a�`���^��V����j�Jj���&&&xy���D���X�����}Y?��S)	��T�n+0���"I��|=���NJR���:���5�"u�)����[=���c��
��KWo�uC$�O`�8r������6�|�r�6�l�%7win��#g��}c�.��V��6y�n�U/�M>�v�1����K_��v�����a���|��|���+4�����{�����V�l��n���k���w.T��Z�UW��}�7]=�oog���d�Fzf���\w��G��__��x���yz��;��������>�p������||�z�7������6����W.tZ������/��r�����%'sz�-��2���e����>S��E�����|����9�m��oZ/���V����z���K;��|���z)��2&G�u���{�6�^e�s�f����\U�[�x>������q""���6����k����y���K��Z�p��f��G������9�`�G6]^���=���_��?p�]�����cc���KsG?���~�;�~�������K����M7n!�p���\��9���;���l���|����_�n�J���MW~}���m[n����������a���n��v���s|��/���9z��Ff�s���|�����{W�%"�����Mx;��ya��C������w�����r���B�������������{�}��-������oh���
���C}�����>�`�����vt�?�z�������b��xT�N�����m[�h��#���^?�|����
?���:�|�����Q���@�m$�?r.�����t��i�s)���<�$��������?���|����M\7Y�N����&.��$���Ns�Yi�����Cp��?j�Z��<����j\,�n�1��N�4�`�8@/���&�S�)9���x���#�"J�=���n��k�^�?u�������]�;����?X).�������C��Sx��5<u���8����_����Q{���v��O����.�������j�>p���,�j������O�������v�C�������F�;~��ut�bs.�+dD�^{��������������2���z�7����#�?�c����/n\�i>�1.�C�_![:�;f��w���O���Dz��'�����n�~:O���
[�����_�3�2�
�Z��5��;��Z�������[����/���Q��Bu��}�"[�<8N�^�'���u��;���N\����;��&�s�z��n���o�us����/"2��ri��_|g�O��:���xq�?>��m(����}����!��}�����/�z=�*F�$?{��n����M6��G7������P���j�T�7������{��1�����w������:4^���;�{[?�
���g����&�����[<��[���RV~09���k��5+��U�����O�������d�J�����X_\s�5����n���;��k�D�|\�Ks�� g�8�X���{�?JwtX��z������7+5.����?���3�Y��@� X>4�������	�/�_v����f[rD����uV���U��,����19R��k�-~M->�����3���N���`�A_���z7�/�������|�m3o?��vLD�~���s�Z�?6��t�v�����7�����3��tywV]������z���[�qV����G\#���P������r��+�:�p�+v@^'+5����L��5���54+�|��_�|yS(������������tc������d��w��e���Y�qV���o������X�R�d(x����|���1}�c��x��KE�J�O�������[9A��������;
�����di��+�Rm>�6���+r��VF��`�Twn���� ��7����4g3N��%����A�����y���|�\�;��+��DT*�v��-�L��B���x���F�$�#~��j5e�S��:nA~����'��+��mNv	��.��f�O���5N��4�Ft�/+.X�
��/��+�|�����O�~,�||�^wC�s9
J��9+r��V>�T���\>X�
��/p����%�vl�����/���*������o�j��}��O�
j���V���-�����������/�tv���nO�����4>>��a��J}�9�JYq��y�:M�P�.�:��!�:�����o��P�RN�x2�8:�G�5��|���5�������X����xrD�3����]�]���g^�*�K�V�~i9�����_�|9w{�F��^��~���5���	=�6�!�>�y-����{�P��CUj�ftv���
��v+8��3��3�����������o��1MLL�����y��./��L*�J�/��S��L�w!�Z���DL��:)9�~=�h���������E�~q��kZ�8�}#s�R��Cd>N�#���">k>V<�O�W�R�5)�{����{���]+���>s�vs|7�^^(�T?}�6������c��c�g!N|�	|��UV�����;g.6�F�qb��ZC-����:y��h�8�#�tZ
�:��#g.��t���f_����5I��@g�k��>����O�9��A[��On;H�yx�]7P���7�R�����z���E�^>XqI��%��.�u���{8�t��c���i�5�q-~_�}�Md�,e�X���4��_������c��������a��W�^%�@DW�^���8�y�����D�������v|�r�}���t�����(���UW_�����.�-��n�>�
;gj-uv��l��s'8�M�|	������N���C�}���3�i�L��r��33t��nv�7fG����a/<6�������O���\�&'�f��sD2ns"�H
W�������N�c�����\zitk�f�~���n@T*�;�!�5� �yll�ZK��v5��%��@F\�|&�N����B��\������O0�f����F�4��aps�:���^GVZ��$��(,����O�"��z"�����q��Vh��;�������vokYy����~��]�o%����������hs�WV��c����� ���U_��������.6��?K��k�K���P=T����X���3�#���n:�a�X�K�JgGK�nj��H����>:�hLRs��fZMs��g.6�������g�&�������03���z��]���������#"9r����o��]G���9~����1Q�?2u����s�����)@z�	������X�N�>]*���d'��s�Tp���W���$I$(���Lq_����dC::�7
�8���L���S?i��+��"��ok�'�P��N�$����a~N�����N�9S�
�$���i������~�W7��2���:�8������cp>z8V��I�'�����1���h���n&"��j���t+/ o�u��g�z�_������k�,w��_#Nu��w��n�VH�k���P+�g������jY^�\�=~���f�m�M�y`{I�h4�2��3�����}�Z�����s��4���E����'��������X�����6������&���o���3�c4���x�P���I"�:~��3�G@�?r�v���x�&�K�Gn?���
���w�!�|����q~����~}��d=C+�
yM�|Q�4yAr<S���SV��c�����`����GE���Tr=/�q��c�u�)����m��7r���e�g�tsX���[=���c��
��KWom���[����_���3����U�V�)t1l~������_}���B�Pi�c�u������i���5y��8���C�����#��T�fG��������mW1?�&����'�������'���>L��<�1h�W(����/����zP�`������J�'��)�����u���E���~����q}���J����:�2�%�|V�2��b�
m�HW���!���iG7�����/_���=���6Z��h��_6y���k����6y��3z�����7Q$���^{�[��J����-������4,�~�u/���3n��Fn�Dt�YM����h�����U����?���g�K��F��&��c������3,��E�V�� ~��^H�DV"d�Y�����	F ?n��{���k��4k|~��4�V��$�NQAcuI�a���)�<������k�?���������O��������w��h8�~�����R��@�p��h��^e��W�Z?7;Z�&�}VR���2�i���ZA��r���G��v<Hssg��u��}\����c����CvDD����?1�<����g��<tv�B{��o�_�=�uz�����y�^�����&��[?v���4���'z��Q�z=���$I~����h��_�l���n�������B�����e�����{9�����R����=��:4�0�,RH��wc?�j����u����a�OO*�W\�����Y]���4�+?k�ZMi��:��Wp\����z��q����5�������N�8o��#�}<���f>:^��?�_��q�y�~���������k���7�p��w����1@j�"lH����������gF���@@��1��)�[�.�Y���|\v�X;6�}����_|�
j���Y#G>}�~�����T=T*�J�}sg��v��DT/��G�O���1���
���v|���M����7���C���_��k<��<�����F��d�b����c����� |�`�������y"�{������O�J���q��]�^��'��'���/���~0_Y�Ri���L�JYq@�������S�RAV}�$IpP?�:���O�>��|�R�5+��O��v�:���KQ���+��~���/_���={��k��^�w���x#�v���]��P'�J���Y�K������Wa�8!��R0X�*b�a���A�#�����x�RAV}��qZ����"�q�YG�Y���q~Z�:�z��/�{\Xq�+�O������������S(k������d���[t~�$�w�  Y}��deJ�_!�|����If~M\�����������aQs�O�zUX���Z��}�����"*�J���ZMJ��u�q|�'N���
Y�=�w�����V��q	V*g>����bE�sZ#�).��b����E�7�Si��;��r�'����x���t,�w����6��O?��Dt��i��hll����z�^�W�y��Z���Z��7i����K���'@���L�A��v�W��4>N>y;S������lHG�8WZO��7�� IDAT�g�Yq����s�*�����4)��JB$N�qg\��k�|*�
?\�#�8������cp>z8=+��v�5V���_~����^x��;�p��9��X�pj�U����#����[~-pxD��_�s8Y���C>�\���H2o���Q�)O���F|�,��2k>V\�q���8��y#��,�_�|"�Iv�5���;w�1X�����!�>:u����WX�\^IO/��X��K9?�\�Au�!�T��'����?�k>V���q������D�q}�o��L���~\K�����5��/~��n������j'��)?������,G�	q�A�dHu�T��R�X����3��(\��+���Y&�(u�`��M��'���q/J}�=�X<z��S�Y2��#z�0_�fUT���0k>�R��pc����w����(�:�~��O=Yu�-s��$�i>����'
VGk�b�tYGZu�|~(	Z};�(��eo�g0h��z���$I~����h��_�l���n�����������Q�\.�J���<�y���<�����{���^�sZ�s3�s4�WGX�x%t>����d�B�����O��b���G����#��t�2���E���&N^�������j�xE��t\G:��d��x��g.��2�,��dO��������N�����A?��������������=x�
E?�����7T�29����f���h�:>�I��
n;�D$�,�Wo��A���)�[��-�e��?��>�F�Z����e?��3
����s�����|tY��<w�a0����"��t�u��+�!,�|��)��?h�� ���>�����1�$�H����8g��W��������u����Q|>>k�������x[�)IN0��[��N�/�
��3�l[���q+�j�4���~�
���cXI��]�^�R�o��=�t����	�4���-�(d>�^_pqH7�E73L���#I������������A��1�{����zt���d�G��Y�0������8'?{e�g�|�"����������n(/,@6r��7
?����	e���u�*�]+�����'������.���I��LMMU�URk�D411�+��w�&����Z�������S2��)��Al�V�W�sR�$I��Cd�g�|�����q;w�tCy���q�Pi�����4}y����*=���g����0�
���\pivt��Y"":x�qlR�q$y��Cv�D��7�����iv����f�z����y��bO4�M�{,����g���s��L����h�0�L����q��%}���(���Pt�`0�u�Y��$I��)��V��5_���
=�r���t~0�#]J���������0,��H���pd��H�����e_���P^+�|\=�oNV/�4�����d^2�.�~�})��M4���z���H�&Y~�Y��2�?I������X�������t�mc���zh�h4&�z�����3G9�S�������;'��ya�\3d~k\+nq�9�:("�L��$�T*�|:�qXu8�%�R�X�i�������~|�V����=���t���8��G$��V�_�\���h��]������J�����������9��
������Vj���v~d���y3��������������dQ}lz����h�����Cs��-�Z{Q}l��]���&��y���KG���eO�"�dR~����xP$9�d�[q2��A�g~�aV��w��a0���#eE���v���!��wk�`�`PN���5���O����w������`����@��o�:I	J>y;S��Z_�W@�w�N����fG�wJ��N���������]=T=th����*���o��No�*�rK�#0���z�4:;{�C�������C�����7��?������x�Xz����[�������wE$����r��h{{������{�Bk/��xM��Gn�~���~������a��C6
�3�:��������S�n��W��n
��]JB�~}���m�b��~��+���J��z"��D���'Tmm�fG�����Q���>r�����.6�,_�vH�C��K��*��'h����u�&�#,�����N��@�q���J��96��^�fG���oU�����h�,���;7���0�wy8��o ����f~�������Up����M��o7����&`������WG=*E�[XP���q�Uq)g����:��8���	��z"���9�oT�<���kt���?�>�����.��
;Lq���g>��v�u��m-M���wE����VJ����v6�s��� =��M����yz�����p�{��3O!�K�&�5������{x~�����"q�P���L��~-'-�������{J��w��T�������}/����#�gR~�q����Lg��N��r&C�K�u�����hI�EI~���/A����/����Z�_��466V������j<��D�-T*�J�/Xq��� ���
{~Q"��*4�����H�N��KO���=]��'-�����V����n�I��N;q���
I^�?���ME��y����~�^�V�Y�����,�I�i�H��l�x�K/���D�u��RG�"#�b�Ef��P�?�����y�8y��x����dr�����E�{�����[�
�e�`:_���*-n�G^Q��E�99>J�|Is�uy���&����5r��E��K4y�v�~�zlr�hi����#���y�u������r�?u`�����/�hb����I)wGX))�lGn�~�y�����$q�C�w�p,�><����D�j���y���v~�~7�H%-���<�jJ�=7�(���V<��a�{��#;��]���4��e�s� �wz#�	��?���VBJE����$%+?~���4H�KVC7a�!255���&&&�������:��b5�#�YMY�m
B~�����l
Y�)d~�V�K�T""��g�i��D�o�K�Ri�����M�8��>?p�/#9���9>G�����3'���R5/��<vq�Z�,W���+%������w�/���;��]�:/��u�����o<���>�����y�DD_?��Z���;J��om?���#G���}�^��tAN�����T*}(����:�^�[����t����NS���1���w�O�|���eK���R��p�����w����h>K��[_�����*=wW����������SO��0s���K��t�>������/���f*I�M�"gJ��#�7k~��������~����k�J�S�^����$���w�my�k��?���vr��y7E(�����7�J{��z7�������i����������~�g�d92I}��������	z\��)�C�q�O��2�Z{���A��ZG��SV^�r�h~~�\.�J%"��w���N~4/r��^��Ny�w�'�9�� /9�^�������>��>K��[������h�z2������<�p��z`�X#��8�Vr���Jo~�%�uef+X'����F���J�I:S
���=����n��y�#'O�<u���={�$ellLV����d����0�����>�K��y�[/~+�[M���3����:��D��$/�����������#g���i����H�q)�,�Y����M�{�����K�A��P�������j�`�����(����X LX���x����0#G8x���NMk�V�q�1�V�u3���J���<��_�������6a��5�"��w�F�N��ls�$�6��.��?"z���n�B�;W,,,���-..�����Q�V��]�������:?d��Z`I\�����G��X�]��Kk��@\��?�����o�7��<7=V�/��O����S.��+����z��v�����|������?���D����k�e�,�����hH�K�������?��?��4;{�������>�[/��n}����I�V�^������$�����<y����:/gQ����`q^��5)�y�a�������P#��a���4�-����g�h��������Lw]�?��~7�G��ySSS{��M��y��n	I/l���A��[B�'��4/�X�a���4���#G�4��Eh�X�}�0��2��7�����N`�x7�&�8%�e�H���Q�I"?n�����\����8��i���Xq�����a�V������|�Z�l+�OLL,,,,,,�J�������Z�666&7>v~:�R��T7�^<r��v�R���@nY_����z}Z������sD�����/�1X���{���O������N�z��_����X�
��o�t^�n J�_}gV��`����7���'}������<��e�����{���r�T���8���X
��������R������������#n(���v���X���""">p��5nH�SC���>��t��������N��_��������z��{�Oc��/��$�+H�4}��%�"�/f~�f�g��9�����
�UvQr"�ot�/=~�����]�r�GdZ�r����T����i^H)�|Y{���=X=��1�����Hiaa��J�R�Tw�=zYJ_)��E��F<���#��~�>�K\��VXS������@\O��`u���=�k�D�������+�|�
�p�},�'I�$)�}P�R)j�58�����>�x��?�|maa��G�cccr�c���Q& ���m�D����d��a���O����v|�q�z��X�-D�ue1yW�R:HUi���:�3����� 3��������R�i�B�i���/�>SK���2q�?�������K0�z������Ty%����N�<IDw�q�_\\$�4�?.V���J��$I��S���������.�P�eJ&�w����D��#Th>����>�����1
>����8?��jb���+���
�!|]�'���d7�hw�������A���*j\���x���y�������'&&���v��}��_�k�.�p��O^�v���i+	:M"q:?t�
�w�t7R�#���	XA_B���"��:��l���g���o���
� �d_���-��"`mZk7sXk��C�|�������{�����	_�S�A�������1!�`��A��[�����[9z�1�EG$���::�@�r5�U����^�g�KI��g�������������F;9������Y��0��Z4s��p�1dSQ+�i��8���������b�
�+��<�x'�#I�$�e�n��^Ag���G�����f\+?����d����
�I�xd>A9�
�����>�����������:uV\�] .xme�
��S2�y����-nDb*��l���O���:��k�+��J��N�2i������V5_�|2f�$�����D������ )"C��8q��o�Y��U���Ci���������O*sy���������g�o�[q~���(�@��[��q�Vv}�Waz��F����7
���_�$�!b����|�������$���w�4~|4����q��I�W�;��0Dp��U������/@&�z�7���8+>mm�dt��q����!8np����A�������:Ay�Q$�o��z�Ul��fn��'�K����M�-����)�~
����������$�/��/�f\�)�zxe�����������g��ZQ�f��&_r"/�4u����HS`�����'���+2����:��5�m�K�R0>���+� os+��+(��������qS�.rd��0�
��/g��Y�V<+��w�t\�e�R�D�d7G�`���Z����I�-w�
'����P���[��m��5�z��F[�>��
u������#�*S��3} &I�����F[^��d�Ot���?�
@���Rv��W��xG�;@!p��U������z=����j�;@!�|��=%�|s	}r�A�����I_<}��twOU}��<wN�B�zt� �����'o�����GX�3�3���������A^������?�����+��AU�Tr�dZn��_�
\!2�/Sr�L���~���('�$H0�_��p&&t�[��*��������V�8���������-�������&�]�8��z�_�K��<�2���a���[#HY�U*�$I�#��?>??�]��HZ{��f�A<4��GO�HZ�q�����H���f�,>�����z���R����E�oQu��P.c�`U�=+d�-�������	�x�!,V�����S��q�:�$I�i��[��s�8�����3(������4�����q���'�D������`}+n�k���c�%(~�Z��|,N�A����!�X_�e#_��#����:�V��q	�7����w	�::��-�:�8@����j��
0�{o�$�JE���N�����M�|�i�����V��C'_�4I}�o�|��/k>%I����jV���1O������/}�Rr�8"�&	B�5��]�^��/A'.t���CM����/��� �/#q�>n����P���D�!��0+�s�t�'e��<�Z��+H~|�t�X�d�'�o�_2�I?�
���:��8������Xq�1�W,�)����u� ��6��.��o�����x/�M#i-
8����xh'���[��g����'�k�w"��iDr�E�B����w�w��Z|�zdE�����dz����b����Q���Gy�\.�[�n��u�r�\.��p�z�V��F�T��kW{cf��
�����I�� �R��c�?�J�7�}7�^�|
~���+�T��9�
�?"k��GT0n�������3�,�>����������HB>�G��*S�a���|����? ��W����_���o}�{�#������'�p�{fll��"r�*�����+���,��	A�
�R��+��L�l��$IR�7v�_<���S8=�J��k�p
�^#�ue�WJ7���I�
�z�g��4Y��5��������:�^���o���a�.=@�y+�J����g!�Oc�&�*�n��70�pFC��-�����U7�2��M|_IwoS���H���"i�Y���3Ok\RM��T�w98?���,�D�����'o�������G�������z k�~�������3��'J<�;Z%���Os�xCE����]����?��-Y��L��|���	�c����#q�t�K�A'n�8�	����`A�����d&M�"�dkh�%X*�����z=eY�$���w�my�k��?���vr��y7����}o��u���y������W��U"��O~R*����"��/~AD���#�r����~�����9��I{��mU���j]<4>>.qV��R��"x`�8uzN9X1>1�4u8"59b	���7O��n��w*R������A������,�X�2�����n���CJ�;������(� |�,p�����&�_����R��;��������<y�T*����Zk�D���HD�
r��ca�YYo\N���I�����"�"~��t	��w�+"�V�H�Uo>�-Cy�c��&�Y?e#SGX)�J%I���_.eaa���%�N)e�������HB���V*�]7k���Y��>��P.���~��#� ��@/������K~�J���5?.���M����$�N��g��*_}����@��n`0$-n�I���|��g��X�nH��+�T�|�B�E`�a�V�������}��y0Hp�	�Px��j�o� IDATGy��������KD?����}�_ �������_w���p��ij��q�V��j�]�����4�du5i��=��7�:c�����jN�)����:�'��d[�|�M��M2�v��A���h(: 
ke�����0��!��:/7�}`-�_���t�����O�����A�������jaY4�
�_*�rV����A'�	`@�YJ�6~�w��/��n ���/���M:��s�5s����|��z�Y2����c�9�s`����e�n^X>���[�9q��8i��J�����#�yd\����`��4@�`�z�8Fe7���z��h�!"����f��JE6��W}�������o�.��5�",��������>��R��a5-�z��e�>t��-���_0�y��p��1d#U���Z;�����g��t7��l*��������[:9��S�����:u"�u"��n�I���f[�����+?w&�1NF+.A���t��G�����o�����
?�	Zq	����L�������8%�����z�^~������jW��X��.�Iw�{%I�A�l�k���8K��Xq~(M�P"��Yu����c��J��;]�:V�J�<���8@�>���T*I�$j�L�U-w�$h��Lu�?��r�oQ�A)���R����w��Zp��%��NkP��HY�:�(���D455�O�:U*��������Z���o~��]:���7H��vS`�9��:���IN_!�����u��W@��pCDD��y�^�����������#�)��E�2�)D������2h�)JQ�5hu��Z�NNN�<y�m������u7�5�9�$�s������5��u������g���+"����<�����v��5@Jr2�o%I�'C�<1�����9��I��:��u���e�������Y����'oG^k�����O�@���#*�J������f�H�?�3��2%G������~�u�J��[nqc�c�n�����z,j��5�z��F[�>����*]�$]0����.�`��E������d?���Dtk\�������!�X�I����p:"��|,zP���q\������`$N)�O0^	���	@����L�I���}7�����6����h'���wC���W��e���?�7�XXX�����t��j����������"����Y����P����:��	���t��������g��T���c��:d4Yu(����j>�-X>^X>��q��
@v�m!�9�
���J������SSS'O���w������'&&�hqq��x��������o^�����&M�u������`�
$��D��Bo����Y�Id�o���������i>�-���
X����������'k
C��L��}&�$|��~��w�"ZXX����~OKC�tR�Wg�O��t#�J.�uXHK%�����w/�k����dX>�~��1�������y���Ww�n�m-�"��W�����e���5�������������.���3)j�������d����F`�y����u�?�u�t�����8OG$_
�u����J�]t��v����G+�6��W����V��e}�B~����G_��;��U������[�����(���dr������.�<#��J%�:���\[�{����c�]>�]v������|��������I��]�k�Z�V�����(Iyc�m�S����9A:_:M:��1_�����
�Y�C�mn�d����s��N3�4���:V\7������#���O�@����r����(��y���u7��9�J�O����:�x~�uY{d�W�a�������u��G�YYqXS�uz��yk� |�[p�
��R��+��s��P1~�y5q��2��V<�9h�-q��q\���~�EO���O�:�'w�t��H�l�]���T���`\OC�km�C��:���H0������bu�:��	8t�iXu�3n��W�;n9��uS
�Rv=PjqO����J�����z��l�4��U�c}��#����u���9���I�#u�8�7����?�Y�[�V�j�!��#�\J����~���K��K�*	~�O�Y���nu���5~�9u����_``��c�Y8���a���AA��)��q�n�����#�)����s�H�)����]"	Z$-SK�{!��r����WH����49kY�Z���R��g��y�����Y0,����(����9�~R�)��.i���O��Zq�&<Dr��c���&r��7��9��n���
���i��+����,�8e*�O�7E�HZ�:�H�B��r<�T�4���C��������y"��w/�:u��^���;���J��I��G%IR��sY��X_�y��|����;98N�����shcM�o���e��vCJ��L#S��������|��Is���I��c�W�5+���y	^;���tVN���R���^d���`0��o�P���k�x0�R�S`e�����9�N�+M�to�7}������E&���X�9���I������d�?�
������<�U�e���Yu"�!8(Y�[�V\"�����7�`�:]J������l�9��os����[q�N���#������C����m�	�:��9RJ��q��	���s����~Kc||���ZMJ���]�x�^���������c\���7��w�Hu�%R�z�
���[�|+�Ob��R���$9��s��S��!VDd��8�M�[n�����V*_��'�n�e����3\�(�k�k�N��z����N��g�������������F;9��*_}<55��y�xbb����W�y�xll����:�
2/���#Y����Hr�mGE��-J7D0����I&#?R�����t�<�(�T��t��Bq"~��L�?>�6�� |�[p�1@?��7od:-�3�'�{����)@n�����?(k����	V�����qy�i�(G����H������o����.Y{
�9@���aX�N�<�2���Zp���R�T�O�
��O2�rd��AVi_A���x�iE8k�� G,���q�m_�
�4n�
|R�7��9����d��o�.�^�O��w
a����V��g�z���+���5O+^���9�_��#CA�N����#�e��a����L
�V08r�J���D�/��}����x����'O�$"����z���p��ij��8�����nr����#u2�X�i�����m�
����(E~��V�����Wo8�8Gt+o�������!$8��h��<H���X��5�����������E�.��5X���|��gI+>�V�~�� ��NO::�)�*6������_��1<yR���F�uY���#����A�98]�9,^�e�����H\7u����uP�)�>I���m-N��|,��Xu�]t��ut\�z��4�)�B���n �dp��G�R��b�4����lK����7Y�+���[��[E4
k�Is��#r�������|]G��)HZ��������d�y��::.������K5n������-�W,��.QI���K�D���~��`��w���+(�4:��_��;���:�����d����w����N�_g`�'��	��:�������8��d'���W!��z�z�X�7X��+��X5_K
�.��u��R�4
��`w:_�#��cz����NP��O���:���-�P�ceV�9Er(�������5@�����r���u}M�OV�^F�^?�K���q���g%�sT��+k��ud;r<u���P����74�B�R��	^�4��t�-��1r���u�(��T��SOV�h��'6���J��#{��<o��4Al
Y�)(��q�J�g���N�HO3%�N�"VS?��:��l#q�!�N�N�FD��&'cJi�c�����q�����?������G]�n��u����}��'����r�\.�[��_���_��u�>�������,�Jw�y'�������<y�T*�����,,,P��8���9X��xcll�^���]�8$�`�{N��K�I_�y��[tM�3U?'}���`�z��W!,@�:.�!�,V�WY�S����>��j�U��y��r���o}����r���o|���z��r����~��~����J�{����������Dt��wZ���v<11AD���D�� ��1�^A��� �9VZ>R���:���	2?�
��zK�
����-�s��N��:_���|��/���	������MD�����������c����G?����g�}�����5,,,�>}��Q�V��w))�6���B���|�8M�|n�A�B����V�7���k�c/���������D���|E?�>������#��8hll�:]w�
~cL��(��)�}�c�E��i�`�2�o)���8��q	:5��4���������|�}(�� M�����k���2^�d����Q����Q�]�-�V<��?e�/gP�a���J��WLOO?���O<��s2y7��������V���&����s�����Y��V��[9�(�������F�u0B���E:(C���4�+�zRt���������>����$H�� +��.��[q���l�RG��A�1��+@���333D$k���{/��W�����>��o|LD{��� _w,7��:O6RJ�D�S���r���-N����`�5�p&��x0������ �UD6��<����T*��F�����Z���V���T����I�B�yY�r?�|��(�c������]XO����/�����@��u�z�o^����s�}��nH���tC��+��9��~~������/���2�d����1dSQ���KE)�OvV�R�^<=Yl�Q*=�s|��
�`��w-��F�������q�����5nQ�Uj�-��5�(�H���F������j"o�|+X�n``dZ$J���w������
��}yV�_)��n���>7�C+�"i3���c���`����������f�P�%IgPf�3)�H��)I<����4�R���Cd�o)��)��cD�N�F��/3�v�����^d���2��#f�x5��/eZV�����z��3)�H��)U�y\�C�.�g�>���u���[W.�����������ry��u_��T�����r��?��o���@D�R�T*e��<~����~���%�>��:Y�,}���yR`���@�|��7�H�$e~���K��p$��<���q<�L@�B�J__v�3�]�#Z��d��4
>9v���[������^B�F<��#D��o~szz����#��G�r�~�"�����r���&''y����S��������w����r�(�}��G�w-?���5�[����kE
WTe707z���7��/?:.��T;#i�c
�$I
��U����?_�������.���G����w)��uVk��0�fff����z��������x��g�}��_���nCF�Z�V���u'���F�7�J���/k�>(j>�:Y�8��z�b������"!��<u�7:
�ai�g���gb��&�G��"��-�z[#��V\���u���R��HGW�
�K�%�D�.zJ�P�����6�q�w�[$31������p�)u��]�fjj�
�%��MC�R�op�_!e������*�7F]\�
�%�:�`�7w'N��`}+n���!��N���t�-��1��g]h�7��.���|���m���&7�Q�;��z����8I����f7����&�������n���G)������fff����x�����>|�����[!���1;y�$��>&������z�TJ��Y6��D?������$S��������|=��������X���~�z�����C]�7��zK�
���s� ��������|�K_����?u�B��*��������o���E�$����%���jB>7u}�a�w	��G$���$q�TOOO,��8uD����U����0\p�
J�J�uU2��m�W?���?��WnCq�}�Y"*�����g%�������j5"*�J�v���l��[	��"�`�����������)G�i��wED�+2�L���e�0\�|C	��+K���'�>�������E��N�����H�.�P����������`�����ozz�
e�k����Dt���R��g�i��W���8��t�����}�V-}���5+�����{1nV2=��P�~9����aV��q���{�������9�������.���������<��	����m�A?>P���7�)���������{���>�n��u��������op�?�A�{��q�\�d�������w�V=:�Jj�<����d��q����uPt�d
Jv��NS����H�TG���J���q�5�`<��]�A!�V��qX��A��>���&k�����4�sNw�:u�\T����7+��A�dQ��y��;�o�����_��}�"�����O���_������o�!""��W���lSSSn�En|���A�G���G�'�)��["�V�'�Ie;�)���z;������"��>(���q����c�
�����w�I�t�3J�n)����?n��s�;�iR�y���`��;:Ed��	�8�W�CX����S$>�H)�!��������}�x�x����g.4�/z�I�+������	�����o�/	������a����:�du�]���1���,:?I���N-��	HN0��[��N�C����qe��e��[�V��	`���?3x��������|���������7�y+������e�8��'�����	z\��)�C�q#������H\����<hp�`��8����/�H����fs������.�n="��o�����#���D���0����h~~^�}�w�^����K�R�_��������cK����,�?�5�_`|u��[}���a����4z���R��6�i�`��J���S�����; ir����.���I���O�l����y�"��7���x���G�>|��~������%�n������$��'O�:u�:������������W�)���?4�$I:����e[�+��x5\Qu��g�����������L&�����$�<	�R,��H��������h��Ul�����J�]km�n�Ze�u��b�y��(O
�H���������|~�������g��y����s���H��wN��(>&�tI���S]R��8���S;���z7e�{����N�,h������?��q�cn��m[��y��7U=���w������q��s|�b'���T��Vt�+���XX8 ��sy����W�\i�Y�n�������/���k�u�u��-[�1��/|���Z��=���}P�����y�E!�6B�Y�?���Uw��H��"���E%q�T����	IO��S���S)v��d��-�����1,H5�=u�<�]j<���{���Sq�i���q��'����6U��BH�vx����z��+__�|���C�_:����v����o
��]�8���v��2g1U���/UW���~;=����"�������<U�>T��y!d aS�Nu�!��h�����q�S���qO��.�c�xwN�!=:9!�$��uk���l������q�����E�]���������Zs��'��W<_�f��[9p���=����?��Z}����N���|�;k��Q[!��ce����z�ch{�
����'��9���=>��&� IDAT{A��vQI���?>�=��������x�(�x��&sI5I;������<T
g;q|����9�jBH�u�7z����Y���4���T�&�^Bz�z3���c�)��-�4{Z���!��>�u8|x��i�[�Okiw���b����=��M7���N��k�A���o�����s����Z����1~[�L3�x�h�mI�m�o��Gu���<�7��s��I���o�k��j��u���1�C�BH�B�1h(5&��&�M`{t��s��i	!��&�����y���m����{������e���5[��U�����4Mz��EW�C���a��Bz�k�_W�2�.��`���B!����{�9g��9���	*;^�d	l�����_w���+����n��4���m[���z���������-:+��&��
��v�������]��yR=3��#��q��7����������;Z'��N�����f��1��\���!]G{B!}�kKD?���I`[l��t�cx���5M�4M�t|��w�������}��c��z=�[�x����>���+g��L��OZ�:8�:4>iQI:O����M:>�Tw
��R�!�];��m7���V�����R<�m�]R=����E[�{LzJ����M5U;���O�n��X��������c��Q|LH����B�k�U��qbv�xW���1!���:��2P%~�o�9��KS���u:1��!�c���	�B!��/*/N�����1e��r����H�G�1!�B�tRK�);&E���E���+���d��?��/8!�B!}__��S�}�<�L;�>�W_}��������������s^=o����w�v
�����m�Oq�O5OW�������jbB�]��]h�
2�t�5%	!�B!���C������+W�u�]��O��?���p��7�3;�%K������_Wu�/Z�H]7��7�l9'��b�����n�r���h�����+����]w�]�B�4�cW���#h�1!�B!���Z�z���+��5k��-��bW�^~�e��_{�������e0����/�����q]�T�?�e��m�s;=O�uW���<��$��O�n\XI�1!�B!�������v�z�+���.�����v������:v��]tN��$-���������.RI:�z��S��C%����B�.B!�B!���g�y��g��]���+_��W��t����`����p��E�i����7;�).\�pa���mqv����3�K���t{�]W�T���8O*������p�&��yT�}��$�����,g�>t���>N��I���R�g�����y!�?����B!����XJm^�v����{���nrw'x�����u� � ������sF���m�_nl��a��f��o�fW���]@��t�����^�E�!��5��3�KI�����q���:�����y!����BHd�U�����*!�Bz��zw��2��@�`w5n���P������+K���|W�'$��X]=����{w=�S�'f���z<�G��A�b;?
����[��=/����xxz�m��<��T�/�}���B: �;��7��B�/�M[�r���?V<���0y���>�l���0�Yo?�u�:Ae�K�,��72���+`��M�����u�V�XGS<��9OL�S����u��iI��r]�Xp�q�;�+�����Bw}\��M:�=�(���h��)���}�+F/`S�Nu�!�|�1n\��k�*!�B��n[����+�?�p����/}l���xb�Rou��Z�=�����i��9���y�����?�QU4M[�|y���T|�9g��3>V�1c��+�p�������x�:�����O|���+�<��T9�<I�18UW�:�x���j�
�n�����9'���T�%�]R=����E[�{LzJ����M5U;���O��[�v�s��m���]�k�F�M(>&��~�%P������B��WO���\�|���������T�1!���&
s	!d����1w!�B!�|*+.�	�%�����'vpL�1!��w��1!�B!�|z�-���"�:����iE���G��cB!�B��:U�<#NU'�BHO���{�Y�5*>���!��a)��B!�t����x��pB�:!�B�g^|�c`�1!�B!����\8�o^�~�v�Y��N!�����G�xj�nU�����	!�$��mK�%B!�(�W��[�����7o��_�|}������B9?}��{����g�@�1!��D(Mw�$�j$�W�o��_��^0y�������a�O��e��K���|U]V����5��Z����+��-]}X����&ku���[�Y������dZ�}�����c����-W�B!��J��ovl����!�S��QR�,�B!������!�B����[,X����m��?g\����w��n\F��	!�PjL!�B!�B��	!m�������������e���!�B!�����cBHr�r(B!�B!��)��B!�B!�B!@�����,�������R�7~�x��w�����1c����B!���C��#�Bz��m����A���o�5���^~v��nqW��/<c?0W;�+7���������G�I���^�#|���f�}?�����|���z�	!�B�!�������g�pM�����4�k���qM�u�9�\��q�i�1����c�qM��1�8on0�s?�0�o��h����]�c���BH?���]9�L�:�;�]���1!�B!��sk��9���s]���4�k�
��]
�\��1�5��s
c��������1h���
��dP���sa;G�����:$�BH����B!�B�A�mH\L!��~��V�T�1??�]Ma{U�����JH����L����M���'7��8���m�����v��K�
]vu4;�=L��=/�n�gw!�B!}L���V��2'��xB!��]
K����Q������G���WN����3R�{������N=fLAii�����c
JJF�9x���=�S���&��g�*����O�����K!�BH���D
��%�BH�ce��L��w!�M��ww��1#�v������3���=s��]�������������D�T�@���.�d������B!��w����$�@�9�b
�I!��^����IC2|�~e����3��o����������*(�t��}C���?OB����w��r��N�4M�uu��UTT$�,**Rm�v��%�����U��c��o����/i�����J^��W��46f����)��1=!�BPJ������0g�����pvm[u��
�;���l[u,��1ui�{i��G���]Q�}���Y��+�&Lq�;�9�M�z+����^�L!����O����~�0tpv��A�������!�sO����^�V�D;���l��|��^�pp��n�9��Z��!�?��7_����__��c�Y���q��n��q�hF��0��u���p��uu�/�{���E�/�{xCE�Y��^z���GB!�������������G`���Q��o����M����=i�S]�j�}J���j�]������J�������>��%[Y�]=�BH�s��������O*��Jw��0���?vb���2����l\����l�q�CcK[~�����V�I�D��<��g���%HD!�e	K�H%JD��N\[5x��~����j�j>��ET�=p��o,�������H����|.�X��/�
5�@�)���{:>����wW��9/..~��G@J���6'�bx���������&�	�B!d�(����/���~��e���!���}i9������>��U���9��Y�<�g��i����K���k�pG#'��	!���Q|�C4x8�:c(���>���fLy)2���<k8S�

����s��`�X��g��wT��U�B��������D�������o"�h8��z�k�����>
y*K?wa>�@@�(��������^���~
j�
D�@a7T�Y�R���F��h(
��j�U>$�@����C4@����6������JAn���q�����e�nyz���B!�l�oo:����V?�{}�bE�����!�X�`E#�{����{`��r���.������b�2_v��|���������a��;��/kt]�����.����+���O<=v�8�s�\�Dj"B!��������
����e��;f��OBC�gd��.���A�A����c�l9~6���G�t���+���J�g��h^;�b��?'���F��p(	E��j��?��!,������*���D�3@fO5d�m�����������c���7�G����A�a�W���B!d�)����7�h��K+}{��y��#v��o���|��mn,-��1��������;h����{��[��e�� ~^W!
D���zS��`�
P��h7/�q�
���J�S�c���B�-=o�
��������y�h_�9��(�k�����&bE,�838�x�?�����&
	Bm��^� "�D����F�"���}�������g 6�gFR4	3�����p�� H,/�&����S�:���q���X���XYX�{��}�����:z6����@��1�bb���*�l��a���B!��Y�P��}����i~w_�J��Tuh_���IS�y��_9/�����k��ES��/�^1���9����b?]V4�q�Mc�-RW�K�T�k���U+��������p^.o�M�����F����sO\��oh���3IG��
��&��R}:/�wz*>���V�����B��������$��(PT�e��GL��q������YC�[�CR��	D;�E�,$b��>����L����1��h�,�S��Zw����E�q�k:
!����}�N�F�������(P��<�c6����
�7l�W]u�������9s\�>b����>��k���+V��O~�2(����������l�(��;�nB!��^g�Cg��������;M��������n�a��}v1}���Q�O�a��c����=O>�g��CW�l�<���,��
qw�aS�����m���)o��hd��o��������������>U�i���U�'�l��G������b��������1��_B!�f=�_�"@F"�4�K�u�����Y)
x�dz�:�l������%bP�h^�,�!�D���c�C��nt����1�wh~���yTn�`@�@l^
 E��x���A�(����(e��]D	R 
@�(��/���__TT$���q��W^	[�l�?�{\S��kj~9�f��u���u�]w��s��M�~=<�U������'��	!�Bz��{��O��Cg|�������v�*�B�b��'Jv>��;�y��bhg�N��������������J��,/1�r{��3�����q]�5��T��3v����V�������{�\2K
��������,��r���uc|�������B���
@��!$V����	��"����3���QR������,C�;<�����DSHa���d%�WB�/�DG{���e\s�_�0Y#>���l��B\<PFD���GN�]t������j���j�c�g���,C���$�2�%�������;w��=��H��z^8�,e_(/��4-?�e���*9��3�*��?�pZ���mw7!�BH�rf��>���y���^[���Q�u����1~��g�}���xq��Q��9Cd������E,���a�����}>T]�<�����/v�%��nx���������&-K5���W��[dWt�o��E�o�b��0b�������E�]���9sd��s������B����n�D!�����U�5DL�3�sC�tKl<U��0o�_�����X�1bJS�%�D&��^Y���u���+��d�q����D�`�2�!g_������&�@iZ�6P_]v��VS�:�4�������?�{�n�0,�r:���K���l})������~SU�m8���d��XnMf&YfB!��;���p[��������>n�������������3=���1�q��q��z����@�x��L��������q�-�`�����G��d�K�>?j����#
u�W]����B��c���vM�)#f��j�s���|��_���b�}l�S��lsns��������������U���k�{N���t}���!��>"��;�}���{��nj�`_��P�-z&i�"��qn�2�ki�����&�����t���2^���>�
M�Hc+1�%F�����t/���5z��=�vv��(� V�(�n���R
l��B�����j�=6n�8w�����'N��*�W��Xv���4�q2|G`��5���gU��W��~u��������Y�B!�B�b�������,^�jUbv�������wh����������}�]������iA���`8���U�����8�"OF��{��#"�P����:����M��k�;����K�zE����_X�pW������uOm�i��B!}�y��:"`��`STc���)�46E��f]��	YUM���|�����0L��^4(�hE���HiE��:�Pk��epv��1-��/��y�
Pk����v�D��[�hZ�J3����_�)e����p�@f���6l������������z5�)����@4�;���z�G��&�
�&�������{��M������3Yys�}�	F8O$�B��M7\��#b1w_��B��#�j���\Yux�u�+�O�/-v�;^^>��r���v��{���3���3���qv�������
;������[���o�kE#���������a;k��/����u�!I�z�1��E!���~�_�v�z�e����Z���&��*RL�7nOcj�
)E$���u�������Q��h����^<Ts��|)a���gvW�)�)dL�%u���ayca��F���`���;#d����b<>�������X�[-
)PD��7F���������p8�e=:��������������Gn�?���Gn�o��n�����80t;p�\�mq�L!��)*;�t�e�^|����S��?]�����c��^2�?��BuT��
�qm2^z����=#F���aC]���G�������/�C�4���g^����6��s�3/j�{P��?��]oz�i�������
�n����zU������T���ktV���C���Mz��Fi������>o����w�^wG�7c����9=���9!}���W��y��O�y��w�+ugN;���#��2-iYu5
��2*��Q���d1���=M�L����-��������!�`Jf	�>/J�����%�$���N���A2��V4.���{S��c����jU��w{N8��o_ee���?F���
��+V��{�v���C3�*���KrF����DB!���Q���^��{���0-�\� 8g��^^������������Pp���&��&���o��m}���5�_������k6�����'U4w�������qW���/������������(����!��bN�8���>�s�}��L{#�O-���|9��n0M�����u�q�q�5�����SQ��3��jk���S�1c�
��*OP�p����x���y�&����g�~�w����3�s�3�r3x~{�=?{�yt���O�u>�Gzo���C~�S�������U�a������������Uv9��+.{�d�U�(AJKb�)4qHfqu��!���������>-;�Xq�����@�cu��vW�V���eE�$�$�����>/p���n��F��	r�f���j����"��#5N�����uuu���^4��.���b��g IDAT2�9T����c�wX�-�����F���8�2H�{�B!�����]�b���
���L�;�����Y���i����Y����z�%e_�3h���9g�[�����}�[550c���U9�#3k������L���kC��������s���{b��U����j������X�l��"�����ba����]��7�������E[���B�E��:3��Ss
rw�/��k�$�h�Q��2��"�]��BH�4-�jj����Q������p��O�������{��
�
�T�f�Q1�)Q�,��!�������D�CHdC�g 2�! ���9l��z���g��Sc�zRF����+"��;��#�pw��h��N��@86"jB:�,� �v�B!���'5a���h�������t�����v��)t�K�5�����-<��g7���p�����g7i�����rz�%��R�����R\0��}-F������O��Y�e���������RX$��[��'��
����B!�O��L3����.�^��FMC���y�7|�2>��Fy���MP ���Y�X$1��Z��u��|���]Q�u�9�5���{�UEb!!�R���`B�� 0��R��O�5��x�o|@ 
�C�0j����8��x{�������l>�x������BB
)%��h�#$���zha��k�2��(������i���X����{�|x���~p���`��/:W��0X�>�B!���O�>����
�o+�������>n�g&O�|.x��q�uZ�t1@^^�q^^sq�{�t*��A<G�@�x'^��>�"�������v�:�e�^�������]��[� ;qMS�/S[I���xs4'���+eN���r�������W�<�^~��KoB	�s��>�%>� }�p������m��eZ�4��#g($gFW7��������7%x�P��)M'?N��Bu:��?��=���������f��*�~����h7������z)>^>-�������&wG�4_������`|���&E4���b&JC��^�n�3������"������cX~���e���HE��LJ�!7�8P�P��
���
h��B
�#7�����K�L�i|��A���W���jsZ<�D��9^�1�?�������4t;J��76����@]�R����vVA�Kv����	ua�4(�;(P���B!���y���[d���}��	��
W���5��p�'=����#�~��7W���p����PS7�8��;�����\�9��vr���?��]m{�c��r���V���	Sf���>x�u8u�9���b�:��rU�7>.}{�[?��+�.}{S���,�����V�ut��0g�*���S���i�=����1��k	��:l	j�I1c����#C	m,�qD�m��N������h�Z����WN���<,����c
E�M�UUUU'kjkj0����,��.��2#=~HK�P2�L��=W7��G?�����t��1c
JK�����.SPR2��������7�����<�5���>�T�����y���F�kBzM/����gv�v����~�eDj����M���_|t����E���\c8�5�t���Y<97�c	��P���
��3����@�@&�M����#����4?0.uC������P��(PB����.�\L��a
�!�$�D$F�%F��
�J�	4%X��D9w�w���	���@�=��x F��${'2����e�F��Jh�=&1��ez�	!�����3�&M���O=S�g���|W�m��F�2u���`����Z�.�P������6�=�5�+����X��������e�~�����
y�y8u��\u��E������hd����4���^�V4����w���MP{����oXp�c��:uA?[��$�����o�|T\c+�H�c����Zv�Ji�`"�5.R��g>�-�5���d�4��miB��s^^^u��Xnn������>���uC�RZ��D55���5�����!���'|������1��	x��EEE6lp����kK�,Q��������U{���CCU������s���@���.g��k��#y
!��z)>��,x{U�n��sp;���L��H$2rH�G��$����g��{�TcTJ^��3~�k�R�%!b�N,)d�=R��>�
��F�$��j��X�:�s�%!��D��������Y�q�B<�0���K�!���`@ �i��X:�$�����s�q�F���?YS��/���A�����	!�B�g��?e��+m�W:�4�������^zi�e�;�iT��+������;�u�&^�l���x�����s��~G���_i���wop�K�3��Q�k��������qIOq
�<�g���j���_�z���vn�I5m�0(%C!�c��!h��{�Y>On�op�??#mh�pF�O�t/N�1�kf6l���K��w_������*(�t��}C���?OB�������E���i��i�����1��H������@�������A�gf�w~�G��������'q��5M�u�4��G����G������HHYTT�����%K������i	�������_������*ix�__=����a�b2��z��������c��v(�����RJaY�`���o��D���Yw�y��i�����������\�@2!���R"h������>���=��14���V����F~�G�Gez$B j�4FDDD���1d<\�J��L����@&�4i��sj���)��A����|v�T�����B!��P	��C{�����;���+w>=�������}������p���3��3;��Kn�6tb�q�P�k�����������g�E� �)���]�����9���|%qL���9���`�������y�2�g����Y��4�\�u��j�W(�w�h�������>=���tBa����S����CBC��t�	U�,�Z� v�?����7�&������_���h4���3��;n��y�����������^�V�!�4d��a��A����D��������,��y<<���|�����6���&{*���J��K	4�%�b��[������������_i����7T���
1�����,�����hy]I�KX�����O��LT��G(�=���������6yt�y	�z;>V:�whp�1�=�&9�Fc�����"a�snq�f�����:cc@.%g��D�3���a�{*J2s=����1JD�PZ(�tmpt�V�0L�}6p�������B:��W7�K��O"V�B�W4 ���f�M�P"H�\"���T�8�{�����}2�t(�"$��!�B�yg��G�������C�]������~���S,�Pw��u_�����\}�����5�IOqH%qLb����������{�������i�Z��i�s�!1V	�c�qK^�,An�&�4����3'?�<[��O�����N,0|��x�9�iVa�������%�-?�zu�N+���2)1��]y�I�(��X���%�z7�%��}'��<�q?�/��j�j>P�_+�0��9S���������O�G�u�|�s���o~yn��"MA��~�c��\c�YR�%���f���W4d.�D�
g�Dc"�"Qa�Ve
�C�p��}$��1��������G)�w����T��X��b��>�����|��4��1��Id���D�� �D@ J��(�E���6���U-����'>���a����9dXU����uAC0�i�1`$�!p`����1� Cd��e��3MuQM���i\��o��Z}�B����Z���������5{xFYm�Hu@"J�
)��RJ�RJ�	y�������.���n,�����P�@��e�X��z��Zl�-���W�����B!�F�����
���[d�A�:%q@��1����=2o��l]�U��T��:,N,6s]]�T��!<�q�1�b�{��K3&�����1dj����pL����=G�&��}~��yPV���@ �G�,�%�N��pF��~	G�����?t��2VW�L(��,�h����%�Lk��{:>�x}���)�Z�2��n���"�4M3�F��P(����|H
���2AC�� �x���8a1�B���M0,@���0V���0`�t��������g�{*��Y�V�4C���o8+�Yo�~�����Be{��[���Kew	4��>�K����i�90`��5xHD)8�cAH������R�l�IIC(<cH^Iu�����(>NiX��7������Uz��,}��Of��K��}��DD���I`��P���2���O�.��g���~�P"Z(��b���wn�lt�)	`���O�O�����{a�hI�`4PQ���&3����1��M�h��L!��)c_��gE#[~|���=h�V�:%q�s�������ON�X�;f�8��1�IEK��'V�dg����������1.�`h9Zv���c��$d0`�q A�����B��|��������XS��������u�g��3GN4��G�����oEK#�X8��"�Y5e�������BQ"��U&�m�%f���j��rg�o�����?��s��AyV�g��HI�I�?��6����l��rb�p�di�lug 4�W��	�0%J��K�;����������19|o�j�Uww�@���B?����y�7����*M?�.0D��sT���������*{V%B�=�Hu����L��r�S��p�pw��1���?����y<=�r%�4��(%i0�Z��pf
	J�*5�"���VH�s��z��PVSc�w���N�}������/���qM��),!D�3�����u���
F��N~���}��� �B!S��\����'3�9�uJ����#+?|?�P������S�c+��S�Tk��������Y�+�)Zn[�W'�}.���������;�����  HDP^S��5+bI�������i��&6QH����@�hP\���@D������3���@lp��(�hf �g�3��G?@�X^bM�y����u�7B�2�i�H���<>�v��;���@z�U���Yc���~i]M]C�Q
���.#sxum��a��XCc8���@6U�
�,���yM�8�����C(>�R��?fHt�l|��)����X_����2Ye~2����������/���L�_�y����a�j�O�������������9YV����	���_�8��8��y���p��V����5#=C�x�ev��s��p���3��Q�w��%u
5<�5
��\����0
1��Ci"Z(M&M��$�@h~����"�N|��6i���c#MA):���W. ���5�������S�����r���>1��q?��B!�OI��K}����?��yu8g�*p��8`��;��}��WL��];h���]�'I<K��O��-	q�C���������J��{;n~~���m]k=�c����+��o�����RE�u�A�i�S"j���{Gez`�P��'����d��oA�U A"f����
i>�4���g*\����X�N��*S�lKL��j��DMG!���#�3�o����(�90������<�c6��������
����r��w��3g������`[�n����b�h,j������A�������56�}�(0-=�2��k�s�5��UVK�L3���>����-�O|����Y+V������eP
�eo9���C��O������{��*p�g��*ix���G�AL���|��~m"���KSr�<�2	h[O�UB�*����g�(	���w�s%A�7�?��W_����sMHsw���'�[��4#�h�7
��&b����Px��=�R�6��"H�PTZ1�b�/��"��v������V������~�n�8�tC���'"I�����m C!�2���s�{vD��?>0��o�S���v#
u;�w�)���y�n�]i&�X�(�����`;#�%�s[��p���S�u�YN�����* 2!d$K��^���
������wL��������,m�Z"&��:
�_���Qb�v��~�`]��uK�1�wh~���y����y��j�ZY)@�X}��uY��Q�#��I��k��]D	R 
@�(������������7n���+`��-����H���y��a�AVTrM�17�5�(�j�L!%2�B+
�"�;^�oDL�q
4���x�'g���X4�|�\�NU�����U�k��Y�n\w�u7�8w�����3YE�F���>���c�������;K�7~\���c��L�4#�~���O�"C@����p�
�����3�����eO��f5��w�5c��������A����:0����"���^8<5O���s�{�GC�f�L�R��2�:^8<�`�p��������w�+�����9�����O~�^���f��_?������O�:��JB!������������6u���B����\�4UT��%�cc�5��6s����E�p`�XF�(��B�U5��C�k�����s���#���� wd�'���O��1"�RX�t)�D������#��^/�x���3Y#>���l�A\<PFD���GN�]t������j���j�c�g���,C�p|lY���X}�p�B��s����g�"��W����e!����N�0��E~���_�W�B���k�'g���%��Y0���A�A�����B_Lh�q�i�s�0**��30��g��>�Uf����d�sUrR�g�=T�!$���<5�������d|��w�;��y�<�"R�d		��A�����%�v����C�P8U�[���Rv��*������Y}j��>B�Xz���m�R"
	����6�c(�}�+'-���l�������;���?��dg�����K��,w�B!���J�>��;��� 9�z-�c�1��m�T�J�B"YS�1$�:k��g:������x�nia�`�>������Dc���K 
D�L H�=��>������W0�}�����G;8�(�<e�C&����k��[P�4-u��.;�N�){�i��`p�������{�a}��Y�����!�4�������kg�C���$�(��**�,�����i�U[����PS�e�����`����
g�K���O�~����l����sX&+o�rkZ0�K�wb��*B�Ok�#3�,�!��a�����_|bq�?�~dYAD&�#���.pN��$���sU��)�j�2Z|L�^��]{Y��P82�pd����i����?�IR�W��.B!�2���8�;wLU���������[�2����2��{�:������m��=�4D�q�87t��4���X�u��|ZA���Q/}\W���h$������#����X��s�����q<S��Q���V�(�X	�P�U��k�7oa!���_�^�q���s�VTT�8q����C�>��fhg��iy,!��L!4������&�2�R
 �>�dCC��Y?p�q�s�stOmM����h�'M�`408���`����5k��������:=���|�Y��9loA)�����Cd1�����i�bk���	��iIDd ��}�X�(�l���D����D���}_��K�B!�B��8�����b�����.����('I�[W�JE����`� �!�����duCSDr`����I��^A�|���[.��k���
J{�dU��2#LLd������mLK34���n����bw����3Q� U6���q���*&v�����H!e�����5��n�����y999


������T�{�D�7-
�����H,;~���_x���Z���Ca��IDATdx�4G���!A��{�VY��5CmX�5��z�!d�M����@4�;�U���r���:�5��+V�K���t�O<1��W���LV��q_m:���arC�����W���!/���c1i	��K�l/�S[}#���L��������Z6tq�@HG�1!�B!��s{��3\���~�� ��q�X�V� 3�A�����Uf�%v�q��&w��F�A���w�D�WE��I���i�y���`8���E��&SjPc:FQZ`�a:O�{�P��S����c3��]	�d��1��s������~n=���2��e����j'��J��;�c���|z"t��X^^������{`������!�`����&Z�����~A$�L�:�����?^r�5W.\��G�������=���KNCY9��53���a����������B
t�}_�*
q��.zm����Ozj����?����j*b'���)W��C[�'�$T_0��?��w���f�o<\u��?8�����ex������J���q9��yZ}S���}��	?�!�_����]#}I���-�^pW��s�'��������g�b��z����A�~�m�G��sB!�Bz�ON����~��q�����u���BV�������g���������.w���9�����c����Ic���v^�c�BZ�%-���at^FeCSc8j1�,�:�u���i�i�������!"��=� $�L�,����E������d[�v��������o�\�,����_{�s��~O�{F-$����.0.�$9�`��bc�1��?����*.(��J�\`L��T��\N�H�[$��@�H3�y�����k����{o��Z���{�?�����>��s��������0I��0�����-&w#&���s���g�}v~~>���.�?�~����^��f��_��t��_�����������WN�������k����?��|��������=���w�i-,c�b�j�	X�+���������h������6���|���i�?���L�<<{��+�8",�$�$������/�2��<��tv-x��������/
b�p�����,�+^7�������fg{���}&M��^C��{����1�������	�^��?�a>����v��^~%��d||bm}=u8�����t�.OD=K�#�O�����J���-.-���X����DDD�������1)H���*�#V@{u�7�����_����H�X�VV�E����"u
"#��@D���������u>07�{_���<�����=_8Y����0�����l�M���X^�|~	��*��P���M������������}�7�$01"��j���KE(�W�G;P�@�������@El�1)���{?�o�������mbee����n���=�����w��of��j����w�u���/������PZGA`��[Iit��[n�������O������=�/G�W*T_=�<�����l��+;q��R��o���
��;�EY��m��z����}��KX?~���]a�����@ HX����G��dD !����[:v�!��y�}�������Mc'����
Z�����p8�]�
BWEPg���n�P2��
��Fw����Y�
�Bq`��b�RZ!��������6���<k�;^�.BQo��l�����bNI�p�2�a+����1�+�N��o����p��K�����<6����+�8��c8��$N��mozp���C{(T��������f&�f��kk�����q+1�Ad3����Rq�������y;��u`"`�)QC��@�0g��.i����c�����s
=�Pdg�WA����c�{�r�K����*�J�V*��1���~5���������(
��v+���68t�m�=��/M�������0(���O�{��w�6@��c����R���v���C���A��+���B85uA�\�|�#��a@� �~l���3}9�S���j;���l����>����J#������=����q8~��df���ML>�p8����slg���
�.
@�4&��v�coE��D������DD�< ���@Ji�dJK���^;l=�R�&wr�����l��t�lt;�n���*���#�G����k�4���.gl�����GMW��b+��M���8��0
� 6q+L��<�J%])�v��/�M(������Yxq!�Z��l"f#H�02X����U���6m|�a��QS!����{~�����_�������}���="""������E����a���������>�	y��q�{�h��~B
��,��o���9,�*@��*�z��Vo�����t�#_�����Ga����k�FCk�j5_|�������A�R���]�{�>����S���?�gO��Z��������@wm,���"l?���c���r�����W�_����,���H	����6�0���a1"&A�@"��������[8}�pl?&��>B�=����p8v)$[�!~�<�����+��NU2�T*j3k��W��ds��""R�u�qf��������b���m�.Q��}����U�zlomd�s�������$�I���u�������7�o����#_������wP�+t�	�p�\���X���X������}�TM�g/5>���a�?6�'����|q>��La�0R��?[������w���"{e<5��S�n:r��s��V�9<z���l�+	:;T�����"�V�yrn��/�x�������~�������.�-~�s�K��"*�
�b���$�)��C��3���F������f���,R-I�Y]];wn��x�u�7R�+Ab�������l�}������2:�����\���BD��� ������~m$XXz����?y��G��p��@@��U���&[�l���F�I��qM�������oD*����p8�c������
��y��4�;F��
S�/��'b����V�����
�u�)���q'��c�m��R����W"��t��:m���� �^w���~��~��[��-m����/�H&��RZ�a|`��������V����n)����G__�{�v��-�2F�1�/^*U��{��}_�L���0X��[��!�cC_1p]�?"~l�D,�A�i��C��%23�����s�]��?�yg���	Df���'������>xpv}m���83�'I�ZKg���P�7�ze`��j��t�n�p�
�Fc=6���74���l?���lR��r��=s���~�4������f.����,.}�����2�|��w<P+"�V`�!����ED�r�������������c�c�q�oD��|���p8�I���������+�	�{��vM�!M�K'�!IgG[���Hk�L�MN/�g�q�����:e;�8���[��zc���Nvw6����\(�����<��&�+�<���B�I��E��� f'^��L`��+���z���9sj=dVg�:_����	#���S���~0P)��TJJ���|l�)zC��-XE�C�#	�L`��4by~��H8J���,x��J�c��J�FP����VBU�0v���5�I8S	���knr�X�kohdx��������?�����G@�.\��O�8��\���������|q~!�96>5�����7����;U�3'����j����zX��%������	����%@����Er?��Z��kg)	�M�����s���3����.�
���]������H�E��-����p8���6t,r�f�����@��@`�S�	�����	@Yr��!�����e����f"gt<r�;&"BO&2�����a�b���z��s]�s�����g���}��3�$i5����2K���_?�m�5)�$`��]	�?���G0���96������j��Z{;U��>ND�I^��Q�\�����a����z`cD��@l�m!BA��ja|��<�2@���2���Y�A�1A������'�x������QGQ�����w�{7���z���x���#�<p����z���p~���;��.��5���}|#�4z���N}��,�Ul�q��#,bN@l�
���O�,�x���c������$�++'O�>u����r���A�@'��!"�Z��q-����������c���F��p8�c��#d3�{�L-o���aJ�����;u��w�Ii�
"�l�m�"��Y|��c��h�8V)��J������R���v��U=�a��v���=�l1�=������O��(��H�fEa
x�B����RI�������4Q��a��VV������HAvP�H"���r���^j$��.����-�YkyJ#�f�CY��b���Kk������HU�6�X$������8i����]�T��?�l5��f'�$Q%-�,+�{����Z�='����*,�-=��q�;�*����-g����J�
�}q`A��.����Q"������E�a�13tY8�������G��5��(+�������Q������DW�&�W��yI"a"a�'&�����9{����`�V��b������p8��+����zd��q�N������!bK[a+E$6i�dnWk�k;U,���lVryd���_n7�Z�*��@�0\*�U���D�L��c��8�Q�r�M<��L��e#;��?��C���*����F�&"�HH)h!"�����q�P�?�\	�=U��e���>����D��$i����x�Zx}5�$w���.�^Z��I���D�Y��E���}u9<sqy~q�hm�V���S���)
�l6�(j�N�(J�$�c�$ILb������N�D���$���$l�u����w�zT�|�r�|���cw+����@������7�V��eF}MO�����s��#G*�J}}=�(��G�����#���/5����^pp�:{���a�cO����������5H�eJ���p8������|���w��������
���q�Zm6[�h����[$9�#""��-�J��5�4����H��o�4�\h�4�t��i5X����?,F���jq�V�(�WK#�b�&�G�=����*�`�Jg����"{��C��x���b#��Hal��7U+y��U�;�����01(1���0�F��G������y�K���0CX$6��
bas���s`�@%����z�[Qx�W����VO�)PzyL���I�cLl��gg6Iw_;*�B@�i��K��|t3�w��4��:h�M���8v�����>$@��/|����9R*�8���qE�v��n
����~�����>8��[�S���?N;;A����?��r���p8�n���������5�Y��D������������[B +$ ���4�(��Hi�2q�5�`� w�Z�Hw� BM������z������j�,����Zi�V���&j��"�66������ghl�r��C����6���C���W���a](���mDX�c[�B�	�����������bYYklDL�`{k�x���G���UN���f�����'>��|_J)����lc��k��)���������j�Q_A}M��:
|�	�Yc�ZGI���� 1����-y�m���o�bCV�x�������R�,�A`��c�I���(����j�1l�a"f�}96�����������w	���F���p8v	W�!�����$�d��!���]��g�DX{l(s�YH�����VVSj�	6Q�l��22�u�Z��D�)%����������u�
)zZi���� {:����4~�l������3���)��~��ENb�zc(�������!���.��N���aT��J���ta��m1�H$��#�X$��c# LvrwGE�M���|��c3��f����65ZWzZ���jM{[����S(bt�1!��B���i�emY�.�I ���a�]�E4��0*�������~�b%���}����a��><#l���c�S��l�af���f�Y��G��}��x�q����~�:�w� ��/����p8��`�F�@���v�����������[c[��Y_�i�qf��M:����fkX�MF�k
"��e��@�Z����6���r�a�n<�z���'wM�t�������r�k�p�z�|�Qp���n�b���d���^����7�_��V:����EZ!'�$l�{�����E���/������P�R���n���|��U��R(�Q*��Q�pK���ee^��eu��� }��{��n���O>F#����U�X�U�:y���������c�I�n�������x��b���)����-�I������xk���q8����7W�V������I���U�Y�@B;J���@=(����d�nG[�Ld}qT����!g�8-v("�����;��a�.�������&���}���<?��H����fj�eOM�����[O���Z_������j�~�q�L��&^����^?�=���<�M�P��(R�
=8���YXZ�d��Y_��sfy�/ym�m���MN�
�~?_�;=���)[$�c������y��'�����z�(�V�TD��l�VVV���/^����t���n.��]q8�A�>v8��g���~3u8�����X���v t41{Y<�&(��Y���@�IN�A Y�
�M�@*�a��>&R��Rdm�~���u�D�k�>���a��Ir��"���vuy#3���������7�c�|�'��Z+~���f�`�����cG�
���O&=y����;�IA������
B����������!=6���R���':�k@����q������F�$���R�����������h�I�A�)�G�-7T�gJC#�j��9�G�q�����vP�����p8��F:I��+�ecjr���������D�rF��&g+K���D���c�T�����:���`��c�,n7�k�=Q:�6{���q������C=�,;������Z>zUn=>q��=XWdb_V�\)<�����}d��-�&����3>c}�/���]���������c{��H�Q���p8�]�l>Y!�b��i������gJ�����6��<"%")I]������`""eKvv��*���;��b���4���;�3��l�LD��u����_��p8G�q���p\;����p8�������u�]��V� (!������=`�&[k%2P�Nk�)m�.QZb4-Ra�������`;�W
w�q6���K�;N7�ve�{�7,nF~���p8;����pl?3��!G_8~�����Q���p8�]�l�F�yOvY����1�~���3��-�lZn�;n[�)zll�������F���t����l���1��p���|h�s��q�<�L�c�s���c/?������������pl?3��!G_������������p8���$���A����k���bd�	"D6�����"�D�E��b��u�h��K7(�n�{od/��yPi���4����> �����oG��voN��8�:m�O�M'������n m]��c?J�G��q
��EB���X��IEND�B`�
#20Thomas Munro
thomas.munro@gmail.com
In reply to: Alexander Lakhin (#19)
1 attachment(s)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

On Wed, Jan 12, 2022 at 8:00 PM Alexander Lakhin <exclusion@gmail.com> wrote:

By the look of things, you are right and this is the localhost-only issue.

But can't that be explained with timing races? You change some stuff
around and it becomes less likely that you get a FIN to arrive in a
super narrow window, which I'm guessing looks something like: recv ->
EWOULDBLOCK, [receive FIN], wait -> FD_CLOSE, wait [hangs]. Note that
it's not happening on several Windows BF animals, and the ones it is
happening on only do it only every few weeks.

Here's a draft attempt at a fix. First I tried to use recv(fd, &c, 1,
MSG_PEEK) == 0 to detect EOF, which seemed to me to be a reasonable
enough candidate, but somehow it corrupts the stream (!?), so I used
Alexander's POLLHUP idea, except I pushed it down to a more principled
place IMHO. Then I suppressed it after the initial check because then
the logic from my earlier patch takes over, so stuff like FeBeWaitSet
doesn't suffer from extra calls, only these two paths that haven't
been converted to long-lived WESes yet. Does this pass the test?

I wonder if this POLLHUP technique is reliable enough (I know that
wouldn't work on other systems[1]https://illumos.topicbox.com/groups/developer/T5576767e764aa26a-Maf8f3460c2866513b0ac51bf, which is why I was trying to make
MSG_PEEK work...).

What about environment variable PG_TEST_USE_UNIX_SOCKETS=1, does it
reproduce with that set, and does the patch fix it? I'm hoping that
explains some Windows CI failures from a nearby thread[2]/messages/by-id/CALT9ZEG=C=JSypzt2gz6NoNtx-ew2tYHbwiOfY_xNo+yBY_=jw@mail.gmail.com.

[1]: https://illumos.topicbox.com/groups/developer/T5576767e764aa26a-Maf8f3460c2866513b0ac51bf
[2]: /messages/by-id/CALT9ZEG=C=JSypzt2gz6NoNtx-ew2tYHbwiOfY_xNo+yBY_=jw@mail.gmail.com

Attachments:

0001-Fix-handling-of-socket-FD_CLOSE-events-on-Windows.patchtext/x-patch; charset=US-ASCII; name=0001-Fix-handling-of-socket-FD_CLOSE-events-on-Windows.patchDownload
From a2cf2b016afc03d384a1682cd02aab760273d6fb Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Thu, 13 Jan 2022 15:48:14 +1300
Subject: [PATCH] Fix handling of socket FD_CLOSE events on Windows.

In some situations we could hang waiting for an FD_CLOSE event that has
already been reported.  Fix this by adding some state to WaitEvent to
remember that we've already been told the socket is closed.  To handle
client code that creates and destroys multiple temporary WaitEventSet
objects and thus would lose that state, check for EOF explicitly the
first time through.
---
 src/backend/storage/ipc/latch.c | 41 +++++++++++++++++++++++++++++++++
 src/include/storage/latch.h     |  1 +
 2 files changed, 42 insertions(+)

diff --git a/src/backend/storage/ipc/latch.c b/src/backend/storage/ipc/latch.c
index 61c876beff..86b86e71c4 100644
--- a/src/backend/storage/ipc/latch.c
+++ b/src/backend/storage/ipc/latch.c
@@ -899,6 +899,7 @@ AddWaitEventToSet(WaitEventSet *set, uint32 events, pgsocket fd, Latch *latch,
 	event->user_data = user_data;
 #ifdef WIN32
 	event->reset = false;
+	event->eof = -1;			/* unknown */
 #endif
 
 	if (events == WL_LATCH_SET)
@@ -1851,6 +1852,45 @@ WaitEventSetWaitBlock(WaitEventSet *set, int cur_timeout,
 			cur_event->reset = false;
 		}
 
+		/*
+		 * Windows does not report FD_CLOSE repeatedly, and does not report
+		 * FD_READ on EOF.  Therefore we need to remember if we've seen
+		 * FD_CLOSE, to defend against callers that wait twice in a row
+		 * without a recv() that fails with WSAEWOULDBLOCK in between.
+		 */
+		if (cur_event->events & WL_SOCKET_READABLE)
+		{
+			/*
+			 * Check for EOF explicitly the first time through, to bridge the
+			 * gap between temporary WaitEventSet objects (such as those
+			 * created by WaitLatchOrSocket()).
+			 */
+			if (cur_event->eof == -1)
+			{
+				WSAPOLLFD	pollfd;
+				int			rc;
+
+				pollfd.fd = cur_event->fd;
+				pollfd.events = POLLRDNORM;
+				pollfd.revents = 0;
+				rc = WSAPoll(&pollfd, 1, 0);
+				if (rc == 1 && pollfd.revents & POLLHUP)
+					cur_event->eof = 1; /* EOF */
+				else
+					cur_event->eof = 0; /* data or error available */
+			}
+
+			/* If we've seen EOF or FD_CLOSE, keep reporting readiness. */
+			if (cur_event->eof == 1)
+			{
+				occurred_events->pos = cur_event->pos;
+				occurred_events->user_data = cur_event->user_data;
+				occurred_events->events = WL_SOCKET_READABLE;
+				occurred_events->fd = cur_event->fd;
+				return 1;
+			}
+		}
+
 		/*
 		 * Windows does not guarantee to log an FD_WRITE network event
 		 * indicating that more data can be sent unless the previous send()
@@ -2002,6 +2042,7 @@ WaitEventSetWaitBlock(WaitEventSet *set, int cur_timeout,
 		{
 			/* EOF/error, so signal all caller-requested socket flags */
 			occurred_events->events |= (cur_event->events & WL_SOCKET_MASK);
+			cur_event->eof = 1;
 		}
 
 		if (occurred_events->events != 0)
diff --git a/src/include/storage/latch.h b/src/include/storage/latch.h
index 3aa7b33834..41a857c0c3 100644
--- a/src/include/storage/latch.h
+++ b/src/include/storage/latch.h
@@ -147,6 +147,7 @@ typedef struct WaitEvent
 	void	   *user_data;		/* pointer provided in AddWaitEventToSet */
 #ifdef WIN32
 	bool		reset;			/* Is reset of the event required? */
+	int8		eof;			/* Has EOF been reached on a socket? */
 #endif
 } WaitEvent;
 
-- 
2.33.1

#21Alexander Lakhin
exclusion@gmail.com
In reply to: Thomas Munro (#20)
1 attachment(s)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

13.01.2022 09:36, Thomas Munro wrote:

On Wed, Jan 12, 2022 at 8:00 PM Alexander Lakhin <exclusion@gmail.com> wrote:

By the look of things, you are right and this is the localhost-only issue.

But can't that be explained with timing races? You change some stuff
around and it becomes less likely that you get a FIN to arrive in a
super narrow window, which I'm guessing looks something like: recv ->
EWOULDBLOCK, [receive FIN], wait -> FD_CLOSE, wait [hangs]. Note that
it's not happening on several Windows BF animals, and the ones it is
happening on only do it only every few weeks.

But I still see the issue when I run both test parts on a single
machine: first instance is `vcregress taptest src\test\restart` and the
second `set NO_TEMP_INSTALL=1 & vcregress taptest contrib/postgres_fdw`
(see attachment).

I'll try new tests and continue investigation later today/tomorrow. Thanks!

Best regards,
Alexander

Attachments:

Screenshot_20220113_095249.pngimage/png; name=Screenshot_20220113_095249.pngDownload
#22Thomas Munro
thomas.munro@gmail.com
In reply to: Thomas Munro (#20)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

On Thu, Jan 13, 2022 at 7:36 PM Thomas Munro <thomas.munro@gmail.com> wrote:

... First I tried to use recv(fd, &c, 1,
MSG_PEEK) == 0 to detect EOF, which seemed to me to be a reasonable
enough candidate, but somehow it corrupts the stream (!?),

Ahh, that'd be because recv() and friends are redirected to our
wrappers in socket.c, where we use the overlapped Winsock API (that
is, async network IO), which is documented as not supporting MSG_PEEK.
OK then.

Andres and I chatted about this stuff off list and he pointed out
something else about the wrappers in socket.c: there are more paths in
there that work with socket events, which means more ways to lose the
precious FD_CLOSE event. I don't know if any of those paths are
reachable in the relevant cases, but it does look a little bit like
the lack of graceful shutdown might have been hiding a whole class of
event tracking bug.

#23Alexander Lakhin
exclusion@gmail.com
In reply to: Thomas Munro (#20)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

13.01.2022 09:36, Thomas Munro wrote:

Here's a draft attempt at a fix. First I tried to use recv(fd, &c, 1,
MSG_PEEK) == 0 to detect EOF, which seemed to me to be a reasonable
enough candidate, but somehow it corrupts the stream (!?), so I used
Alexander's POLLHUP idea, except I pushed it down to a more principled
place IMHO. Then I suppressed it after the initial check because then
the logic from my earlier patch takes over, so stuff like FeBeWaitSet
doesn't suffer from extra calls, only these two paths that haven't
been converted to long-lived WESes yet. Does this pass the test?

Yes, this fix eliminates the flakiness for me. The test commit_ts (with
002_standby and 003_standby_2) passed 2x200 iterations.
It also makes my test postgres_fdw/001_disconnection pass reliably.

I wonder if this POLLHUP technique is reliable enough (I know that
wouldn't work on other systems[1], which is why I was trying to make
MSG_PEEK work...).

What about environment variable PG_TEST_USE_UNIX_SOCKETS=1, does it
reproduce with that set, and does the patch fix it? I'm hoping that
explains some Windows CI failures from a nearby thread[2].

With PG_TEST_USE_UNIX_SOCKETS=1 the test commit_ts/002_standby fails on
the unpatched HEAD:
t/002_standby.pl .... 1/4 # poll_query_until timed out executing this query:
# SELECT '0/303C628'::pg_lsn <= pg_last_wal_replay_lsn()
# expecting this output:
# t
# last actual query output:
# f
# with stderr:
# Looks like your test exited with 25 just after 1.
t/002_standby.pl .... Dubious, test returned 25 (wstat 6400, 0x1900)

002_standby_primary.log contains:
2022-01-13 18:57:32.925 PST [1448] LOG:  starting PostgreSQL 15devel,
compiled by Visual C++ build 1928, 64-bit
2022-01-13 18:57:32.926 PST [1448] LOG:  listening on Unix socket
"C:/Users/1/AppData/Local/Temp/yOKQPH1FoO/.s.PGSQL.62733"

The same with my postgres_fdw test:
# 03:41:12.533246 result:       0
# 0|0
# 03:41:12.534758 executing query...
# 03:41:14.267594 result:       3
#
# psql:<stdin>:1: WARNING:  no connection to the server
# psql:<stdin>:1: ERROR:  FATAL:  terminating connection due to
administrator command
# server closed the connection unexpectedly
#       This probably means the server terminated abnormally
#       before or while processing the request.
# CONTEXT:  remote SQL command: FETCH 100 FROM c1
# 03:41:14.270449 executing query...
# 03:41:14.334437 result:       3
#
# psql:<stdin>:1: ERROR:  could not connect to server "fpg"
# DETAIL:  connection to server on socket
"C:/Users/1/AppData/Local/Temp/hJWD9mzPHM/.s.PGSQL.57414" failed:
Connection refused (0x0000274D/10061)
#       Is the server running locally and accepting connections on that
socket?
# 03:41:14.336918 executing query...
# 03:41:14.422381 result:       3
#
# psql:<stdin>:1: ERROR:  could not connect to server "fpg"
# DETAIL:  connection to server on socket
"C:/Users/1/AppData/Local/Temp/hJWD9mzPHM/.s.PGSQL.57414" failed:
Connection refused (0x0000274D/10061)
#       Is the server running locally and accepting connections on that
socket?
# 03:41:14.425628 executing query...
...hang...

With the patch these tests pass successfully.

I can also confirm that on Windows 10 20H2 (previous tests were
performed on Windows Server 2012) the unpatched HEAD +
PG_TEST_USE_UNIX_SOCKETS=1 hangs on src\test\recovery\001_stream_rep (on
iterations 1, 1, 4 for me).
(v9-0001-Add-option-for-amcheck-and-pg_amcheck-to-check-un.patch [1]/messages/by-id/CALT9ZEHx2+9rqAeAANkUXJCsTueQqdx2Tt6ypaig9tozJkWvkQ@mail.gmail.com not
required to see that.)
001_stream_rep_primary.log contains:
...
2022-01-13 19:46:48.287 PST [1364] LOG:  listening on Unix socket
"C:/Users/1/AppData/Local/Temp/EWzapwiXjV/.s.PGSQL.58248"
2022-01-13 19:46:48.317 PST [6736] LOG:  database system was shut down
at 2022-01-13 19:46:37 PST
2022-01-13 19:46:48.331 PST [1364] LOG:  database system is ready to
accept connections
2022-01-13 19:46:49.536 PST [1088] 001_stream_rep.pl LOG:  statement:
CREATE TABLE tab_int AS SELECT generate_series(1,1002) AS a
2022-01-13 19:46:49.646 PST [3028] 001_stream_rep.pl LOG:  statement:
SELECT pg_current_wal_insert_lsn()
2022-01-13 19:46:49.745 PST [3360] 001_stream_rep.pl LOG:  statement:
SELECT '0/3023268' <= replay_lsn AND state = 'streaming' FROM
pg_catalog.pg_stat_replication WHERE application_name = 'standby_1';
...
2022-01-13 19:50:39.755 PST [4924] 001_stream_rep.pl LOG:  statement:
SELECT '0/3023268' <= replay_lsn AND state = 'streaming' FROM
pg_catalog.pg_stat_replication WHERE application_name = 'standby_1';
2022-01-13 19:50:39.928 PST [5924] 001_stream_rep.pl LOG:  statement:
SELECT '0/3023268' <= replay_lsn AND state = 'streaming' FROM
pg_catalog.pg_stat_replication WHERE application_name = 'standby_1';

Without PG_TEST_USE_UNIX_SOCKETS=1 and without the fix the
001_stream_rep hangs too (but on iterations 3, 8, 2). So it seems that
using unix sockets increases the fail rate.

With the fix 100 iterations with PG_TEST_USE_UNIX_SOCKETS=1 and 40 (and
still counting) iterations without PG_TEST_USE_UNIX_SOCKETS pass.

So the fix looks as absolutely working to me with the tests that we have
for now.

[1]: /messages/by-id/CALT9ZEHx2+9rqAeAANkUXJCsTueQqdx2Tt6ypaig9tozJkWvkQ@mail.gmail.com
/messages/by-id/CALT9ZEHx2+9rqAeAANkUXJCsTueQqdx2Tt6ypaig9tozJkWvkQ@mail.gmail.com

#24Andres Freund
andres@anarazel.de
In reply to: Thomas Munro (#22)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

Hi,

On 2022-01-14 20:31:22 +1300, Thomas Munro wrote:

Andres and I chatted about this stuff off list and he pointed out
something else about the wrappers in socket.c: there are more paths in
there that work with socket events, which means more ways to lose the
precious FD_CLOSE event.

I think it doesn't even need to touch socket.c to cause breakage. Using two
different WaitEventSets is enough.

https://docs.microsoft.com/en-us/windows/win32/api/winsock2/nf-winsock2-wsaeventselect
says:

The FD_CLOSE network event is recorded when a close indication is received
for the virtual circuit corresponding to the socket. In TCP terms, this
means that the FD_CLOSE is recorded when the connection goes into the TIME
WAIT or CLOSE WAIT states. This results from the remote end performing a
shutdown on the send side or a closesocket. FD_CLOSE being posted after all
data is read from a socket

So FD_CLOSE is *recorded* internally when the connection is closed. But only
posted to the visible event once all data is read. All good so far. But
combine that with:

Issuing a WSAEventSelect for a socket cancels any previous WSAAsyncSelect or
WSAEventSelect for the same socket and clears the internal network event
record.

Note the bit about clearing the internal network event record. Which seems to
pretty precisely explain why we're loosing FD_CLOSEs?

And it does also explain why this is more likely after the shutdown changes:
It's more likely the network stack knows it has readable data *and* that the
connection closed. Which is recorded in the "internal network event
record". But once all the data is read, walsender.c will do another
WaitLatchOrSocket(), which does WSAEventSelect(), clearing the "internal event
record" and loosing the FD_CLOSE.

My first inclination was that we ought to wrap the socket created for windows
in pgwin32_socket() in a custom type with some additional data - including
information about already received events, an EVENT, etc. I think that might
help to remove a lot of the messy workarounds we have in socket.c etc. But: It
wouldn't do much good here, because the socket is not a socket created by
socket.c but by libpq :(.

Greetings,

Andres Freund

#25Andres Freund
andres@anarazel.de
In reply to: Andres Freund (#24)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

Hi,

On 2022-01-14 12:28:48 -0800, Andres Freund wrote:

But once all the data is read, walsender.c will do another
WaitLatchOrSocket(), which does WSAEventSelect(), clearing the "internal event
record" and loosing the FD_CLOSE.

Walreceiver only started using WES in
2016-03-29 [314cbfc5d] Add new replication mode synchronous_commit = 'remote_ap

With that came the following comment:

/*
* Ideally we would reuse a WaitEventSet object repeatedly
* here to avoid the overheads of WaitLatchOrSocket on epoll
* systems, but we can't be sure that libpq (or any other
* walreceiver implementation) has the same socket (even if
* the fd is the same number, it may have been closed and
* reopened since the last time). In future, if there is a
* function for removing sockets from WaitEventSet, then we
* could add and remove just the socket each time, potentially
* avoiding some system calls.
*/
Assert(wait_fd != PGINVALID_SOCKET);
rc = WaitLatchOrSocket(MyLatch,
WL_EXIT_ON_PM_DEATH | WL_SOCKET_READABLE |
WL_TIMEOUT | WL_LATCH_SET,
wait_fd,
NAPTIME_PER_CYCLE,
WAIT_EVENT_WAL_RECEIVER_MAIN);

I don't really see how libpq could have changed the socket underneath us, as
long as we get it the first time after the connection successfully was
established? I mean, there's a running command that we're processing the
result of? Nor do I understand what "any other walreceiver implementation"
refers to?

Thomas, I think you wrote that?

Greetings,

Andres Freund

#26Thomas Munro
thomas.munro@gmail.com
In reply to: Andres Freund (#24)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

On Sat, Jan 15, 2022 at 9:28 AM Andres Freund <andres@anarazel.de> wrote:

I think it doesn't even need to touch socket.c to cause breakage. Using two
different WaitEventSets is enough.

Right. I was interested in your observation because so far we'd
*only* been considering the two-consecutive-WaitEventSets case, which
we could grok experimentally. The patch Alexander tested most
recently uses a tri-state eof flag, so (1) the WES event starts out in
"unknown" state and polls with WSAPoll() to figure out whether the
socket was already closed when it wasn't looking, and then (2) it
switches to believing that we'll definitely get an FD_CLOSE so we
don't need to make the extra call on every wait. That does seem to do
the job, but if there are *other* places that can eat FD_CLOSE event
once we've switched to believing that it will come, that might be
fatal to the second part of that idea, and we might have to assume
"unknown" all the time, which would be somewhat similar to the way we
do a dummy WSASend() every time when waiting for WRITEABLE.

(That patch is assuming that we're looking for something simple to
back-patch, in the event that we decide not to just revert the
graceful shutdown patch from back branches. Reverting might be a
better idea for now, and then we could fix all this stuff going
forward.)

Issuing a WSAEventSelect for a socket cancels any previous WSAAsyncSelect or
WSAEventSelect for the same socket and clears the internal network event
record.

Note the bit about clearing the internal network event record. Which seems to
pretty precisely explain why we're loosing FD_CLOSEs?

Indeed.

And it does also explain why this is more likely after the shutdown changes:
It's more likely the network stack knows it has readable data *and* that the
connection closed. Which is recorded in the "internal network event
record". But once all the data is read, walsender.c will do another
WaitLatchOrSocket(), which does WSAEventSelect(), clearing the "internal event
record" and loosing the FD_CLOSE.

Yeah.

#27Thomas Munro
thomas.munro@gmail.com
In reply to: Thomas Munro (#26)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

On Sat, Jan 15, 2022 at 10:59 AM Thomas Munro <thomas.munro@gmail.com> wrote:

(That patch is assuming that we're looking for something simple to
back-patch, in the event that we decide not to just revert the
graceful shutdown patch from back branches. Reverting might be a
better idea for now, and then we could fix all this stuff going
forward.)

(Though, as mentioned already, reverting isn't really enough either,
because other OSes that Windows might be talking to use lingering
sockets... and there may still be ways for this to break...)

#28Andres Freund
andres@anarazel.de
In reply to: Thomas Munro (#26)
1 attachment(s)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

Hi,

On 2022-01-15 10:59:00 +1300, Thomas Munro wrote:

On Sat, Jan 15, 2022 at 9:28 AM Andres Freund <andres@anarazel.de> wrote:

I think it doesn't even need to touch socket.c to cause breakage. Using two
different WaitEventSets is enough.

Right. I was interested in your observation because so far we'd
*only* been considering the two-consecutive-WaitEventSets case, which
we could grok experimentally.

There likely are further problems in other parts, but I think socket.c is
unlikely to be involved in walreceiver case - there shouldn't be any socket.c
style socket in walreceiver itself, nor do I think we are doing a
send/recv/select backed by socket.c.

The patch Alexander tested most recently uses a tri-state eof flag [...]

What about instead giving WalReceiverConn an internal WaitEventSet, and using
that consistently? I've attached a draft for that.

Alexander, could you test with that patch applied?

Greetings,

Andres Freund

Attachments:

0001-WIP-use-long-lived-WaitEventSet-for-libpqwalreceiver.patchtext/x-diff; charset=us-asciiDownload
From 8139d324687364164c9abbd3e2b2a54b5d5bbcad Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 14 Jan 2022 14:39:57 -0800
Subject: [PATCH] WIP: use long-lived WaitEventSet for libpqwalreceiver
 connections.

---
 src/include/replication/walreceiver.h         |  29 +++--
 .../libpqwalreceiver/libpqwalreceiver.c       | 112 ++++++++++++------
 src/backend/replication/logical/tablesync.c   |  11 +-
 src/backend/replication/logical/worker.c      |  13 +-
 src/backend/replication/walreceiver.c         |  26 +---
 5 files changed, 112 insertions(+), 79 deletions(-)

diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 92f73a55b8d..07a16470d48 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -253,6 +253,17 @@ typedef void (*walrcv_check_conninfo_fn) (const char *conninfo);
  */
 typedef char *(*walrcv_get_conninfo_fn) (WalReceiverConn *conn);
 
+/*
+ * walrcv_wait_fn
+ *
+ * Waits for socket to become readable, or a latch interrupt.
+ *
+ * timeout is passed on to WaitEventSetWait(), return value is
+ * like WaitLatchOrSocket's.
+ */
+typedef int (*walrcv_wait_fn) (WalReceiverConn *conn, long timeout,
+							   uint32 wait_event_info);
+
 /*
  * walrcv_get_senderinfo_fn
  *
@@ -317,14 +328,13 @@ typedef void (*walrcv_endstreaming_fn) (WalReceiverConn *conn,
 /*
  * walrcv_receive_fn
  *
- * Receive a message available from the WAL stream.  'buffer' is a pointer
- * to a buffer holding the message received.  Returns the length of the data,
- * 0 if no data is available yet ('wait_fd' is a socket descriptor which can
- * be waited on before a retry), and -1 if the cluster ended the COPY.
+ * Receive a message available from the WAL stream.  'buffer' is a pointer to
+ * a buffer holding the message received.  Returns the length of the data, 0
+ * if no data is available yet (see walrcv_wait()), and -1 if the cluster
+ * ended the COPY.
  */
 typedef int (*walrcv_receive_fn) (WalReceiverConn *conn,
-								  char **buffer,
-								  pgsocket *wait_fd);
+								  char **buffer);
 
 /*
  * walrcv_send_fn
@@ -385,6 +395,7 @@ typedef struct WalReceiverFunctionsType
 	walrcv_connect_fn walrcv_connect;
 	walrcv_check_conninfo_fn walrcv_check_conninfo;
 	walrcv_get_conninfo_fn walrcv_get_conninfo;
+	walrcv_wait_fn walrcv_wait;
 	walrcv_get_senderinfo_fn walrcv_get_senderinfo;
 	walrcv_identify_system_fn walrcv_identify_system;
 	walrcv_server_version_fn walrcv_server_version;
@@ -407,6 +418,8 @@ extern PGDLLIMPORT WalReceiverFunctionsType *WalReceiverFunctions;
 	WalReceiverFunctions->walrcv_check_conninfo(conninfo)
 #define walrcv_get_conninfo(conn) \
 	WalReceiverFunctions->walrcv_get_conninfo(conn)
+#define walrcv_wait(conn, timeout, wait_event_info) \
+	WalReceiverFunctions->walrcv_wait(conn, timeout, wait_event_info)
 #define walrcv_get_senderinfo(conn, sender_host, sender_port) \
 	WalReceiverFunctions->walrcv_get_senderinfo(conn, sender_host, sender_port)
 #define walrcv_identify_system(conn, primary_tli) \
@@ -419,8 +432,8 @@ extern PGDLLIMPORT WalReceiverFunctionsType *WalReceiverFunctions;
 	WalReceiverFunctions->walrcv_startstreaming(conn, options)
 #define walrcv_endstreaming(conn, next_tli) \
 	WalReceiverFunctions->walrcv_endstreaming(conn, next_tli)
-#define walrcv_receive(conn, buffer, wait_fd) \
-	WalReceiverFunctions->walrcv_receive(conn, buffer, wait_fd)
+#define walrcv_receive(conn, buffer) \
+	WalReceiverFunctions->walrcv_receive(conn, buffer)
 #define walrcv_send(conn, buffer, nbytes) \
 	WalReceiverFunctions->walrcv_send(conn, buffer, nbytes)
 #define walrcv_create_slot(conn, slotname, temporary, two_phase, snapshot_action, lsn) \
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 0d89db4e6a6..24f062f779c 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -46,6 +46,8 @@ struct WalReceiverConn
 	bool		logical;
 	/* Buffer for currently read records */
 	char	   *recvBuf;
+
+	WaitEventSet *wes;
 };
 
 /* Prototypes for interface functions */
@@ -54,6 +56,8 @@ static WalReceiverConn *libpqrcv_connect(const char *conninfo,
 										 char **err);
 static void libpqrcv_check_conninfo(const char *conninfo);
 static char *libpqrcv_get_conninfo(WalReceiverConn *conn);
+static int libpqrcv_wait(WalReceiverConn *conn, long timeout,
+						 uint32 wait_event_info);
 static void libpqrcv_get_senderinfo(WalReceiverConn *conn,
 									char **sender_host, int *sender_port);
 static char *libpqrcv_identify_system(WalReceiverConn *conn,
@@ -66,8 +70,7 @@ static bool libpqrcv_startstreaming(WalReceiverConn *conn,
 									const WalRcvStreamOptions *options);
 static void libpqrcv_endstreaming(WalReceiverConn *conn,
 								  TimeLineID *next_tli);
-static int	libpqrcv_receive(WalReceiverConn *conn, char **buffer,
-							 pgsocket *wait_fd);
+static int	libpqrcv_receive(WalReceiverConn *conn, char **buffer);
 static void libpqrcv_send(WalReceiverConn *conn, const char *buffer,
 						  int nbytes);
 static char *libpqrcv_create_slot(WalReceiverConn *conn,
@@ -87,6 +90,7 @@ static WalReceiverFunctionsType PQWalReceiverFunctions = {
 	libpqrcv_connect,
 	libpqrcv_check_conninfo,
 	libpqrcv_get_conninfo,
+	libpqrcv_wait,
 	libpqrcv_get_senderinfo,
 	libpqrcv_identify_system,
 	libpqrcv_server_version,
@@ -102,8 +106,8 @@ static WalReceiverFunctionsType PQWalReceiverFunctions = {
 };
 
 /* Prototypes for private functions */
-static PGresult *libpqrcv_PQexec(PGconn *streamConn, const char *query);
-static PGresult *libpqrcv_PQgetResult(PGconn *streamConn);
+static PGresult *libpqrcv_PQexec(WalReceiverConn *conn, const char *query);
+static PGresult *libpqrcv_PQgetResult(WalReceiverConn *conn);
 static char *stringlist_to_identifierstr(PGconn *conn, List *strings);
 
 /*
@@ -203,6 +207,11 @@ libpqrcv_connect(const char *conninfo, bool logical, const char *appname,
 		else
 			io_flag = WL_SOCKET_WRITEABLE;
 
+		/*
+		 * NB: cannot yet use/create conn->wes, during connection
+		 * establishment the socket can change.
+		 */
+
 		rc = WaitLatchOrSocket(MyLatch,
 							   WL_EXIT_ON_PM_DEATH | WL_LATCH_SET | io_flag,
 							   PQsocket(conn->streamConn),
@@ -227,11 +236,20 @@ libpqrcv_connect(const char *conninfo, bool logical, const char *appname,
 		return NULL;
 	}
 
+	conn->wes = CreateWaitEventSet(CurrentMemoryContext, 3);
+	AddWaitEventToSet(conn->wes, WL_LATCH_SET, PGINVALID_SOCKET,
+					  MyLatch, NULL);
+	/* XXX: This isn't necessarily nice to do unconditionally? */
+	AddWaitEventToSet(conn->wes, WL_EXIT_ON_PM_DEATH, PGINVALID_SOCKET,
+					  NULL, NULL);
+	AddWaitEventToSet(conn->wes, WL_SOCKET_READABLE, PQsocket(conn->streamConn),
+					  NULL, NULL);
+
 	if (logical)
 	{
 		PGresult   *res;
 
-		res = libpqrcv_PQexec(conn->streamConn,
+		res = libpqrcv_PQexec(conn,
 							  ALWAYS_SECURE_SEARCH_PATH_SQL);
 		if (PQresultStatus(res) != PGRES_TUPLES_OK)
 		{
@@ -359,7 +377,7 @@ libpqrcv_identify_system(WalReceiverConn *conn, TimeLineID *primary_tli)
 	 * Get the system identifier and timeline ID as a DataRow message from the
 	 * primary server.
 	 */
-	res = libpqrcv_PQexec(conn->streamConn, "IDENTIFY_SYSTEM");
+	res = libpqrcv_PQexec(conn, "IDENTIFY_SYSTEM");
 	if (PQresultStatus(res) != PGRES_TUPLES_OK)
 	{
 		PQclear(res);
@@ -482,7 +500,7 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
 						 options->proto.physical.startpointTLI);
 
 	/* Start streaming. */
-	res = libpqrcv_PQexec(conn->streamConn, cmd.data);
+	res = libpqrcv_PQexec(conn, cmd.data);
 	pfree(cmd.data);
 
 	if (PQresultStatus(res) == PGRES_COMMAND_OK)
@@ -532,7 +550,7 @@ libpqrcv_endstreaming(WalReceiverConn *conn, TimeLineID *next_tli)
 	 * If we had not yet received CopyDone from the backend, PGRES_COPY_OUT is
 	 * also possible in case we aborted the copy in mid-stream.
 	 */
-	res = libpqrcv_PQgetResult(conn->streamConn);
+	res = libpqrcv_PQgetResult(conn);
 	if (PQresultStatus(res) == PGRES_TUPLES_OK)
 	{
 		/*
@@ -547,7 +565,7 @@ libpqrcv_endstreaming(WalReceiverConn *conn, TimeLineID *next_tli)
 		PQclear(res);
 
 		/* the result set should be followed by CommandComplete */
-		res = libpqrcv_PQgetResult(conn->streamConn);
+		res = libpqrcv_PQgetResult(conn);
 	}
 	else if (PQresultStatus(res) == PGRES_COPY_OUT)
 	{
@@ -561,7 +579,7 @@ libpqrcv_endstreaming(WalReceiverConn *conn, TimeLineID *next_tli)
 							pchomp(PQerrorMessage(conn->streamConn)))));
 
 		/* CommandComplete should follow */
-		res = libpqrcv_PQgetResult(conn->streamConn);
+		res = libpqrcv_PQgetResult(conn);
 	}
 
 	if (PQresultStatus(res) != PGRES_COMMAND_OK)
@@ -572,7 +590,7 @@ libpqrcv_endstreaming(WalReceiverConn *conn, TimeLineID *next_tli)
 	PQclear(res);
 
 	/* Verify that there are no more results */
-	res = libpqrcv_PQgetResult(conn->streamConn);
+	res = libpqrcv_PQgetResult(conn);
 	if (res != NULL)
 		ereport(ERROR,
 				(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -597,7 +615,7 @@ libpqrcv_readtimelinehistoryfile(WalReceiverConn *conn,
 	 * Request the primary to send over the history file for given timeline.
 	 */
 	snprintf(cmd, sizeof(cmd), "TIMELINE_HISTORY %u", tli);
-	res = libpqrcv_PQexec(conn->streamConn, cmd);
+	res = libpqrcv_PQexec(conn, cmd);
 	if (PQresultStatus(res) != PGRES_TUPLES_OK)
 	{
 		PQclear(res);
@@ -641,7 +659,7 @@ libpqrcv_readtimelinehistoryfile(WalReceiverConn *conn,
  * May return NULL, rather than an error result, on failure.
  */
 static PGresult *
-libpqrcv_PQexec(PGconn *streamConn, const char *query)
+libpqrcv_PQexec(WalReceiverConn *conn, const char *query)
 {
 	PGresult   *lastResult = NULL;
 
@@ -657,7 +675,7 @@ libpqrcv_PQexec(PGconn *streamConn, const char *query)
 	 * theoretically block.  In practice, since we don't send very long query
 	 * strings, the risk seems negligible.
 	 */
-	if (!PQsendQuery(streamConn, query))
+	if (!PQsendQuery(conn->streamConn, query))
 		return NULL;
 
 	for (;;)
@@ -665,7 +683,7 @@ libpqrcv_PQexec(PGconn *streamConn, const char *query)
 		/* Wait for, and collect, the next PGresult. */
 		PGresult   *result;
 
-		result = libpqrcv_PQgetResult(streamConn);
+		result = libpqrcv_PQgetResult(conn);
 		if (result == NULL)
 			break;				/* query is complete, or failure */
 
@@ -679,24 +697,49 @@ libpqrcv_PQexec(PGconn *streamConn, const char *query)
 		if (PQresultStatus(lastResult) == PGRES_COPY_IN ||
 			PQresultStatus(lastResult) == PGRES_COPY_OUT ||
 			PQresultStatus(lastResult) == PGRES_COPY_BOTH ||
-			PQstatus(streamConn) == CONNECTION_BAD)
+			PQstatus(conn->streamConn) == CONNECTION_BAD)
 			break;
 	}
 
 	return lastResult;
 }
 
+/*
+ * XXX: Currently the WES is configured to only check if socket is readable,
+ * not writeable. That might need to be configurable in the future.
+ */
+static int
+libpqrcv_wait(WalReceiverConn *conn, long timeout, uint32 wait_event_info)
+{
+	int			nevents;
+	WaitEvent	event;
+	int			ret = 0;
+
+	nevents = WaitEventSetWait(conn->wes, timeout, &event, 1, wait_event_info);
+
+	if (nevents == 0)
+		ret |= WL_TIMEOUT;
+	else
+	{
+		ret |= event.events & (WL_LATCH_SET |
+							   WL_POSTMASTER_DEATH |
+							   WL_SOCKET_MASK);
+	}
+
+	return ret;
+}
+
 /*
  * Perform the equivalent of PQgetResult(), but watch for interrupts.
  */
 static PGresult *
-libpqrcv_PQgetResult(PGconn *streamConn)
+libpqrcv_PQgetResult(WalReceiverConn *conn)
 {
 	/*
 	 * Collect data until PQgetResult is ready to get the result without
 	 * blocking.
 	 */
-	while (PQisBusy(streamConn))
+	while (PQisBusy(conn->streamConn))
 	{
 		int			rc;
 
@@ -705,12 +748,7 @@ libpqrcv_PQgetResult(PGconn *streamConn)
 		 * since we'll get interrupted by signals and can handle any
 		 * interrupts here.
 		 */
-		rc = WaitLatchOrSocket(MyLatch,
-							   WL_EXIT_ON_PM_DEATH | WL_SOCKET_READABLE |
-							   WL_LATCH_SET,
-							   PQsocket(streamConn),
-							   0,
-							   WAIT_EVENT_LIBPQWALRECEIVER_RECEIVE);
+		rc = libpqrcv_wait(conn, -1, WAIT_EVENT_LIBPQWALRECEIVER_RECEIVE);
 
 		/* Interrupted? */
 		if (rc & WL_LATCH_SET)
@@ -720,7 +758,7 @@ libpqrcv_PQgetResult(PGconn *streamConn)
 		}
 
 		/* Consume whatever data is available from the socket */
-		if (PQconsumeInput(streamConn) == 0)
+		if (PQconsumeInput(conn->streamConn) == 0)
 		{
 			/* trouble; return NULL */
 			return NULL;
@@ -728,7 +766,7 @@ libpqrcv_PQgetResult(PGconn *streamConn)
 	}
 
 	/* Now we can collect and return the next PGresult */
-	return PQgetResult(streamConn);
+	return PQgetResult(conn->streamConn);
 }
 
 /*
@@ -737,6 +775,12 @@ libpqrcv_PQgetResult(PGconn *streamConn)
 static void
 libpqrcv_disconnect(WalReceiverConn *conn)
 {
+	if (conn->wes != NULL)
+	{
+		FreeWaitEventSet(conn->wes);
+		conn->wes = NULL;
+	}
+
 	PQfinish(conn->streamConn);
 	if (conn->recvBuf != NULL)
 		PQfreemem(conn->recvBuf);
@@ -752,16 +796,15 @@ libpqrcv_disconnect(WalReceiverConn *conn)
  *	 point to a buffer holding the received message. The buffer is only valid
  *	 until the next libpqrcv_* call.
  *
- *	 If no data was available immediately, returns 0, and *wait_fd is set to a
- *	 socket descriptor which can be waited on before trying again.
+ *	 If no data was available immediately, returns 0. In that case walrcv_wait()
+ *	 can be used to wait for socket readiness or interrupts before trying again.
  *
  *	 -1 if the server ended the COPY.
  *
  * ereports on error.
  */
 static int
-libpqrcv_receive(WalReceiverConn *conn, char **buffer,
-				 pgsocket *wait_fd)
+libpqrcv_receive(WalReceiverConn *conn, char **buffer)
 {
 	int			rawlen;
 
@@ -785,7 +828,6 @@ libpqrcv_receive(WalReceiverConn *conn, char **buffer,
 		if (rawlen == 0)
 		{
 			/* Tell caller to try again when our socket is ready. */
-			*wait_fd = PQsocket(conn->streamConn);
 			return 0;
 		}
 	}
@@ -793,13 +835,13 @@ libpqrcv_receive(WalReceiverConn *conn, char **buffer,
 	{
 		PGresult   *res;
 
-		res = libpqrcv_PQgetResult(conn->streamConn);
+		res = libpqrcv_PQgetResult(conn);
 		if (PQresultStatus(res) == PGRES_COMMAND_OK)
 		{
 			PQclear(res);
 
 			/* Verify that there are no more results. */
-			res = libpqrcv_PQgetResult(conn->streamConn);
+			res = libpqrcv_PQgetResult(conn);
 			if (res != NULL)
 			{
 				PQclear(res);
@@ -941,7 +983,7 @@ libpqrcv_create_slot(WalReceiverConn *conn, const char *slotname,
 			appendStringInfoString(&cmd, " PHYSICAL RESERVE_WAL");
 	}
 
-	res = libpqrcv_PQexec(conn->streamConn, cmd.data);
+	res = libpqrcv_PQexec(conn, cmd.data);
 	pfree(cmd.data);
 
 	if (PQresultStatus(res) != PGRES_TUPLES_OK)
@@ -1068,7 +1110,7 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
 				(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
 				 errmsg("the query interface requires a database connection")));
 
-	pgres = libpqrcv_PQexec(conn->streamConn, query);
+	pgres = libpqrcv_PQexec(conn, query);
 
 	switch (PQresultStatus(pgres))
 	{
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index e596b69d466..9eb78d1faab 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -637,14 +637,13 @@ copy_read_data(void *outbuf, int minread, int maxread)
 
 	while (maxread > 0 && bytesread < minread)
 	{
-		pgsocket	fd = PGINVALID_SOCKET;
 		int			len;
 		char	   *buf = NULL;
 
 		for (;;)
 		{
 			/* Try read the data. */
-			len = walrcv_receive(LogRepWorkerWalRcvConn, &buf, &fd);
+			len = walrcv_receive(LogRepWorkerWalRcvConn, &buf);
 
 			CHECK_FOR_INTERRUPTS();
 
@@ -676,11 +675,8 @@ copy_read_data(void *outbuf, int minread, int maxread)
 		/*
 		 * Wait for more data or latch.
 		 */
-		(void) WaitLatchOrSocket(MyLatch,
-								 WL_SOCKET_READABLE | WL_LATCH_SET |
-								 WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
-								 fd, 1000L, WAIT_EVENT_LOGICAL_SYNC_DATA);
-
+		(void) walrcv_wait(LogRepWorkerWalRcvConn, 1000L,
+						   WAIT_EVENT_LOGICAL_SYNC_DATA);
 		ResetLatch(MyLatch);
 	}
 
@@ -967,6 +963,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos)
 	 * application_name, so that it is different from the main apply worker,
 	 * so that synchronous replication can distinguish them.
 	 */
+	Assert(LogRepWorkerWalRcvConn == NULL);
 	LogRepWorkerWalRcvConn =
 		walrcv_connect(MySubscription->conninfo, true, slotname, &err);
 	if (LogRepWorkerWalRcvConn == NULL)
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index c9af775bc18..95d4d0c7d1f 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -2597,7 +2597,6 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
 	/* This outer loop iterates once per wait. */
 	for (;;)
 	{
-		pgsocket	fd = PGINVALID_SOCKET;
 		int			rc;
 		int			len;
 		char	   *buf = NULL;
@@ -2608,7 +2607,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
 
 		MemoryContextSwitchTo(ApplyMessageContext);
 
-		len = walrcv_receive(LogRepWorkerWalRcvConn, &buf, &fd);
+		len = walrcv_receive(LogRepWorkerWalRcvConn, &buf);
 
 		if (len != 0)
 		{
@@ -2688,7 +2687,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
 					MemoryContextReset(ApplyMessageContext);
 				}
 
-				len = walrcv_receive(LogRepWorkerWalRcvConn, &buf, &fd);
+				len = walrcv_receive(LogRepWorkerWalRcvConn, &buf);
 			}
 		}
 
@@ -2729,11 +2728,8 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
 		else
 			wait_time = NAPTIME_PER_CYCLE;
 
-		rc = WaitLatchOrSocket(MyLatch,
-							   WL_SOCKET_READABLE | WL_LATCH_SET |
-							   WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
-							   fd, wait_time,
-							   WAIT_EVENT_LOGICAL_APPLY_MAIN);
+		rc = walrcv_wait(LogRepWorkerWalRcvConn, wait_time,
+						 WAIT_EVENT_LOGICAL_APPLY_MAIN);
 
 		if (rc & WL_LATCH_SET)
 		{
@@ -3541,6 +3537,7 @@ ApplyWorkerMain(Datum main_arg)
 		origin_startpos = replorigin_session_get_progress(false);
 		CommitTransactionCommand();
 
+		Assert(LogRepWorkerWalRcvConn == NULL);
 		LogRepWorkerWalRcvConn = walrcv_connect(MySubscription->conninfo, true,
 												MySubscription->name, &err);
 		if (LogRepWorkerWalRcvConn == NULL)
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index b39fce8c23c..d70fc462bea 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -423,7 +423,6 @@ WalReceiverMain(void)
 				char	   *buf;
 				int			len;
 				bool		endofwal = false;
-				pgsocket	wait_fd = PGINVALID_SOCKET;
 				int			rc;
 
 				/*
@@ -446,7 +445,7 @@ WalReceiverMain(void)
 				}
 
 				/* See if we can read data immediately */
-				len = walrcv_receive(wrconn, &buf, &wait_fd);
+				len = walrcv_receive(wrconn, &buf);
 				if (len != 0)
 				{
 					/*
@@ -478,7 +477,7 @@ WalReceiverMain(void)
 							endofwal = true;
 							break;
 						}
-						len = walrcv_receive(wrconn, &buf, &wait_fd);
+						len = walrcv_receive(wrconn, &buf);
 					}
 
 					/* Let the primary know that we received some data. */
@@ -496,24 +495,9 @@ WalReceiverMain(void)
 				if (endofwal)
 					break;
 
-				/*
-				 * Ideally we would reuse a WaitEventSet object repeatedly
-				 * here to avoid the overheads of WaitLatchOrSocket on epoll
-				 * systems, but we can't be sure that libpq (or any other
-				 * walreceiver implementation) has the same socket (even if
-				 * the fd is the same number, it may have been closed and
-				 * reopened since the last time).  In future, if there is a
-				 * function for removing sockets from WaitEventSet, then we
-				 * could add and remove just the socket each time, potentially
-				 * avoiding some system calls.
-				 */
-				Assert(wait_fd != PGINVALID_SOCKET);
-				rc = WaitLatchOrSocket(MyLatch,
-									   WL_EXIT_ON_PM_DEATH | WL_SOCKET_READABLE |
-									   WL_TIMEOUT | WL_LATCH_SET,
-									   wait_fd,
-									   NAPTIME_PER_CYCLE,
-									   WAIT_EVENT_WAL_RECEIVER_MAIN);
+				rc = walrcv_wait(wrconn, NAPTIME_PER_CYCLE,
+								 WAIT_EVENT_WAL_RECEIVER_MAIN);
+
 				if (rc & WL_LATCH_SET)
 				{
 					ResetLatch(MyLatch);
-- 
2.34.0

#29Tom Lane
tgl@sss.pgh.pa.us
In reply to: Thomas Munro (#26)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

Thomas Munro <thomas.munro@gmail.com> writes:

(That patch is assuming that we're looking for something simple to
back-patch, in the event that we decide not to just revert the
graceful shutdown patch from back branches. Reverting might be a
better idea for now, and then we could fix all this stuff going
forward.)

FWIW, I'm just fine with reverting, particularly in the back branches.
It seems clear that this dank corner of Windows contains even more
creepy-crawlies than we thought.

regards, tom lane

#30Thomas Munro
thomas.munro@gmail.com
In reply to: Andres Freund (#28)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

On Sat, Jan 15, 2022 at 11:44 AM Andres Freund <andres@anarazel.de> wrote:

The patch Alexander tested most recently uses a tri-state eof flag [...]

What about instead giving WalReceiverConn an internal WaitEventSet, and using
that consistently? I've attached a draft for that.

Alexander, could you test with that patch applied?

Isn't your patch nearly identical to one that I already posted, that
Alexander tested and reported success with here?

/messages/by-id/5d507424-13ce-d19f-2f5d-ab4c6a987316@gmail.com

I can believe that fixes walreceiver (if we're sure that there isn't a
libpq-changes-the-socket problem), but AFAICS the same problem exists
for postgres_fdw and async append. That's why I moved to trying to
fix the multiple-WES thing (though of course I agree we should be
using long lived WESes wherever possible, I just didn't think that
seemed back-patchable, so it's more of a feature patch for the
future).

#31Thomas Munro
thomas.munro@gmail.com
In reply to: Andres Freund (#25)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

On Sat, Jan 15, 2022 at 9:47 AM Andres Freund <andres@anarazel.de> wrote:

Walreceiver only started using WES in
2016-03-29 [314cbfc5d] Add new replication mode synchronous_commit = 'remote_ap

With that came the following comment:

/*
* Ideally we would reuse a WaitEventSet object repeatedly
* here to avoid the overheads of WaitLatchOrSocket on epoll
* systems, but we can't be sure that libpq (or any other
* walreceiver implementation) has the same socket (even if
* the fd is the same number, it may have been closed and
* reopened since the last time). In future, if there is a
* function for removing sockets from WaitEventSet, then we
* could add and remove just the socket each time, potentially
* avoiding some system calls.
*/
Assert(wait_fd != PGINVALID_SOCKET);
rc = WaitLatchOrSocket(MyLatch,
WL_EXIT_ON_PM_DEATH | WL_SOCKET_READABLE |
WL_TIMEOUT | WL_LATCH_SET,
wait_fd,
NAPTIME_PER_CYCLE,
WAIT_EVENT_WAL_RECEIVER_MAIN);

I don't really see how libpq could have changed the socket underneath us, as
long as we get it the first time after the connection successfully was
established? I mean, there's a running command that we're processing the
result of?

Erm, I didn't analyse the situation much back then, I just knew that
libpq could reconnect in early phases. I can see that once you reach
that stage you can count on socket stability though, so yeah that
should work as long as you can handle it correctly in the earlier
connection phase (probably using the other patch I posted and
Alexander tested), it should all work nicely. You'd probably want to
formalise the interface/documentation on that point.

Nor do I understand what "any other walreceiver implementation"
refers to?

I think I meant that it goes via function pointers to talk to
libpqwalreceiver.c, but I know now that we don't actually support
using that to switch to different code, it's just a solution to a
backend/frontend linking problem. The comment was probably just
paranoia based on the way the interface works.

#32Andres Freund
andres@anarazel.de
In reply to: Thomas Munro (#30)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

Hi,

On 2022-01-15 13:19:42 +1300, Thomas Munro wrote:

On Sat, Jan 15, 2022 at 11:44 AM Andres Freund <andres@anarazel.de> wrote:

The patch Alexander tested most recently uses a tri-state eof flag [...]

What about instead giving WalReceiverConn an internal WaitEventSet, and using
that consistently? I've attached a draft for that.

Alexander, could you test with that patch applied?

Isn't your patch nearly identical to one that I already posted, that
Alexander tested and reported success with here?

Sorry, somehow I missed that across the many patches in the thread... And yes,
it does look remarkably similar.

/messages/by-id/5d507424-13ce-d19f-2f5d-ab4c6a987316@gmail.com

I can believe that fixes walreceiver

One thing that still bothers me around this is that we didn't detect the
problem of the dead walreceiver connection, even after missing the
FD_CLOSE. There's plenty other ways that a connection can get stalled for
prolonged was, that we'd not get notified about either. That's why there's
wal_receiver_timeout, after all.

But from what I can see wal_receiver_timeout doesn't work even halfway
reliably, because of all the calls down to libpqrcv_PQgetResult, where we just
block indefinitely?

This actually seems like a significant issue to me, and not just on windows.

(if we're sure that there isn't a libpq-changes-the-socket problem)

I just don't see what that problem could be, once connection is
established. The only way a the socket fd could change is a reconnect, which
doesn't happen automatically.

I actually was searching the archives for threads on it, but I didn't find
much besides the references around [1]/messages/by-id/CA+hUKG+zCNJZBXcURPdQvdY-tjyD0y7Li2wZEC6XChyUej1S5w@mail.gmail.com. And I didn't see a concrete risk
explained there?

but AFAICS the same problem exists for postgres_fdw and async append.

Perhaps - but I suspect it'll matter far less with them than with walreceiver.

That's why I moved to trying to
fix the multiple-WES thing (though of course I agree we should be
using long lived WESes wherever possible

That approach seems like a very very leaky bandaid, with a decent potential
for unintended consequences. Perhaps there's nothing better that we can do,
but I'd rather try to fix the problem closer to the root...

I just didn't think that seemed back-patchable, so it's more of a feature
patch for the future).

Hm, it doesn't seem crazy invasive to me. But I guess we might be looking at a
revert of the shutdown changes for now anyway? In that case we should be
fixing this anyway, but we might be able to afford doing it in master only?

Greetings,

Andres Freund

[1]: /messages/by-id/CA+hUKG+zCNJZBXcURPdQvdY-tjyD0y7Li2wZEC6XChyUej1S5w@mail.gmail.com

#33Alexander Lakhin
exclusion@gmail.com
In reply to: Andres Freund (#28)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

Hello Andres,
15.01.2022 01:44, Andres Freund wrote:

What about instead giving WalReceiverConn an internal WaitEventSet, and using
that consistently? I've attached a draft for that.

Alexander, could you test with that patch applied?

Unfortunately, this patch doesn't fix the issue. The test
commit_ts/002_standby still fails (on iterations 3, 1, 8 for me).
With the debugging logging added:
...
elog(LOG, "libpqrcv_wait() before WaitEventSetWait(%p)", conn->wes);
��� nevents = WaitEventSetWait(conn->wes, timeout, &event, 1,
wait_event_info);
elog(LOG, "libpqrcv_wait() after WaitEventSetWait");
...
elog(LOG, "WaitEventSetWaitBlock before WaitForMultipleObjects(%p)",
set->handles);
��� rc = WaitForMultipleObjects(set->nevents + 1, set->handles, FALSE,
��� ��� ��� ��� ��� ��� ��� ��� cur_timeout);
elog(LOG, "WaitEventSetWaitBlock aftet WaitForMultipleObjects");
...
and so on.

I see in 002_standby_standby.log
...
2022-01-16 13:31:36.244 MSK [1336] LOG:� libpqrcv_receive PQgetCopyData
returned: 145
2022-01-16 13:31:36.244 MSK [1336] LOG:� libpqrcv_receive PQgetCopyData
returned: 0
2022-01-16 13:31:36.244 MSK [1336] LOG:� libpqrcv_wait() before
WaitEventSetWait(000000000063ABA8)
2022-01-16 13:31:36.244 MSK [1336] LOG:� WaitEventSetWait before
WaitEventSetWaitBlock(000000000063ABA8)
2022-01-16 13:31:36.244 MSK [1336] LOG:� WaitEventSetWaitBlock before
WaitForMultipleObjects(000000000063AC30)
2022-01-16 13:31:36.244 MSK [2820] LOG:� WaitEventSetWaitBlock aftet
WaitForMultipleObjects
2022-01-16 13:31:36.244 MSK [2820] LOG:� WaitEventSetWait after
WaitEventSetWaitBlock
2022-01-16 13:31:36.244 MSK [1336] LOG:� WaitEventSetWaitBlock aftet
WaitForMultipleObjects
2022-01-16 13:31:36.244 MSK [1336] LOG:� WaitEventSetWait after
WaitEventSetWaitBlock
2022-01-16 13:31:36.244 MSK [1336] LOG:� libpqrcv_wait() after
WaitEventSetWait
2022-01-16 13:31:36.244 MSK [1336] LOG:� libpqrcv_receive PQgetCopyData
returned: 0
2022-01-16 13:31:36.244 MSK [1336] LOG:� libpqrcv_wait() before
WaitEventSetWait(000000000063ABA8)
2022-01-16 13:31:36.244 MSK [1336] LOG:� WaitEventSetWait before
WaitEventSetWaitBlock(000000000063ABA8)
2022-01-16 13:31:36.244 MSK [1336] LOG:� WaitEventSetWaitBlock before
WaitForMultipleObjects(000000000063AC30)
2022-01-16 13:31:36.244 MSK [2820] LOG:� WaitEventSetWait before
WaitEventSetWaitBlock(0000000000649FB8)
2022-01-16 13:31:36.244 MSK [2820] LOG:� WaitEventSetWaitBlock before
WaitForMultipleObjects(000000000064A020)
2022-01-16 13:31:36.247 MSK [1336] LOG:� WaitEventSetWaitBlock aftet
WaitForMultipleObjects
2022-01-16 13:31:36.247 MSK [1336] LOG:� WaitEventSetWait after
WaitEventSetWaitBlock
2022-01-16 13:31:36.247 MSK [1336] LOG:� libpqrcv_wait() after
WaitEventSetWait
2022-01-16 13:31:36.247 MSK [1336] LOG:� libpqrcv_receive PQgetCopyData
returned: -1
2022-01-16 13:31:36.247 MSK [1336] LOG:� libpqrcv_receive before
libpqrcv_PQgetResult(1)
2022-01-16 13:31:36.247 MSK [1336] LOG:� libpqrcv_receive
libpqrcv_PQgetResult(1) returned 0000000000692400
2022-01-16 13:31:36.247 MSK [1336] LOG:� libpqrcv_receive before
libpqrcv_PQgetResult(2)
2022-01-16 13:31:36.247 MSK [1336] LOG:� libpqrcv_wait() before
WaitEventSetWait(000000000063ABA8)
2022-01-16 13:31:36.247 MSK [1336] LOG:� WaitEventSetWait before
WaitEventSetWaitBlock(000000000063ABA8)
2022-01-16 13:31:36.247 MSK [1336] LOG:� WaitEventSetWaitBlock before
WaitForMultipleObjects(000000000063AC30)
2022-01-16 13:31:36.368 MSK [984] LOG:� WaitEventSetWaitBlock aftet
WaitForMultipleObjects
2022-01-16 13:31:36.368 MSK [984] LOG:� WaitEventSetWait after
WaitEventSetWaitBlock
...
After that, the process 1336 hangs till shutdown.

Best regards,
Alexander

#34Andres Freund
andres@anarazel.de
In reply to: Tom Lane (#29)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

Hi,

On 2022-01-14 17:51:52 -0500, Tom Lane wrote:

FWIW, I'm just fine with reverting, particularly in the back branches.
It seems clear that this dank corner of Windows contains even more
creepy-crawlies than we thought.

Seems we should revert now-ish? There's a minor release coming up and I think
it'd be bad to ship these changes to users.

Greetings,

Andres Freund

#35Tom Lane
tgl@sss.pgh.pa.us
In reply to: Andres Freund (#34)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

Andres Freund <andres@anarazel.de> writes:

On 2022-01-14 17:51:52 -0500, Tom Lane wrote:

FWIW, I'm just fine with reverting, particularly in the back branches.
It seems clear that this dank corner of Windows contains even more
creepy-crawlies than we thought.

Seems we should revert now-ish? There's a minor release coming up and I think
it'd be bad to ship these changes to users.

Sure. Do we want to revert in HEAD too?

regards, tom lane

#36Andres Freund
andres@anarazel.de
In reply to: Tom Lane (#35)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

Hi,

On 2022-01-24 15:35:25 -0500, Tom Lane wrote:

Andres Freund <andres@anarazel.de> writes:

On 2022-01-14 17:51:52 -0500, Tom Lane wrote:

FWIW, I'm just fine with reverting, particularly in the back branches.
It seems clear that this dank corner of Windows contains even more
creepy-crawlies than we thought.

Seems we should revert now-ish? There's a minor release coming up and I think
it'd be bad to ship these changes to users.

Sure. Do we want to revert in HEAD too?

Not sure. I'm also OK with trying to go with Thomas' patch to walreceiver and
try a bit longer to get all this working. Thomas?

Greetings,

Andres Freund

#37Thomas Munro
thomas.munro@gmail.com
In reply to: Andres Freund (#36)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

On Tue, Jan 25, 2022 at 10:28 AM Andres Freund <andres@anarazel.de> wrote:

On 2022-01-24 15:35:25 -0500, Tom Lane wrote:

Andres Freund <andres@anarazel.de> writes:

On 2022-01-14 17:51:52 -0500, Tom Lane wrote:

FWIW, I'm just fine with reverting, particularly in the back branches.
It seems clear that this dank corner of Windows contains even more
creepy-crawlies than we thought.

Seems we should revert now-ish? There's a minor release coming up and I think
it'd be bad to ship these changes to users.

Sure. Do we want to revert in HEAD too?

Not sure. I'm also OK with trying to go with Thomas' patch to walreceiver and
try a bit longer to get all this working. Thomas?

I vote for reverting in release branches only. I'll propose a better
WES patch set for master that hopefully also covers async append etc
(which I was already planning to do before we knew about this Windows
problem). More soon.

#38Tom Lane
tgl@sss.pgh.pa.us
In reply to: Thomas Munro (#37)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

Thomas Munro <thomas.munro@gmail.com> writes:

On Tue, Jan 25, 2022 at 10:28 AM Andres Freund <andres@anarazel.de> wrote:

On 2022-01-24 15:35:25 -0500, Tom Lane wrote:

Sure. Do we want to revert in HEAD too?

Not sure. I'm also OK with trying to go with Thomas' patch to walreceiver and
try a bit longer to get all this working. Thomas?

I vote for reverting in release branches only. I'll propose a better
WES patch set for master that hopefully also covers async append etc
(which I was already planning to do before we knew about this Windows
problem). More soon.

WFM, but we'll have to remember to revert this in v15 if we don't
have a solid fix by then.

It's kinda late here, so I'll push the reverts tomorrow.

regards, tom lane

#39Thomas Munro
thomas.munro@gmail.com
In reply to: Tom Lane (#38)
3 attachment(s)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

On Tue, Jan 25, 2022 at 3:50 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Thomas Munro <thomas.munro@gmail.com> writes:

I vote for reverting in release branches only. I'll propose a better
WES patch set for master that hopefully also covers async append etc
(which I was already planning to do before we knew about this Windows
problem). More soon.

WFM, but we'll have to remember to revert this in v15 if we don't
have a solid fix by then.

Phew, after a couple of days of very slow compile/test cycles on
Windows exploring a couple of different ideas, I finally have
something new. First let me recap the three main ideas in this
thread:

1. It sounds like no one really loves the WSAPoll() kludge, even
though it apparently works for simple cases. It's not totally clear
that it really works in enough cases, for one thing. It doesn't allow
for a socket to be in two WESes at the same time, and I'm not sure I
want to bank on Winsock's WSAPoll() being guaranteed to report POLLHUP
when half closed (as mentioned, no other OS does AFAIK).

2. The long-lived-WaitEventSets-everywhere concept was initially
appealling to me and solves the walreceiver problem (when combined
with a sticky seen_fd_close flag), and I've managed to get that
working correctly across libpq reconnects. As mentioned, I also have
some toy patches along those lines for the equivalent but more complex
problem in postgres_fdw, because I've been studying how to make
parallel append generate a tidy stream of epoll_wait()/kevent() calls,
instead of a quadratic explosion of setup/teardown spam. I'll write
some more about those patches and hopefully propose them soon anyway,
but on reflection I don't really want that Unix efficiency problem to
be tangled up with this Windows correctness problem. That'd require a
programming rule that I don't want to burden us with forever: you'd
*never* be able to use a socket in more than one WaitEventSet, and
WaitLatchOrSocket() would have to be removed.

3. The real solution to this problem is to recognise that we just
have the event objects in the wrong place. WaitEventSets shouldn't
own them: they need to be 1:1 with sockets, or Winsock will eat
events. Likewise for the flag you need for edge->level conversion, or
*we'll* eat events. Having now tried that, it's starting to feel like
the best way forward, even though my initial prototype (see attached)
is maybe a tad cumbersome with bookkeeping. I believe it means that
all existing coding patterns *should* now be safe (not yet confirmed
by testing), and we're free to put sockets in multiple WESes even at
the same time if the need arises.

The basic question is: how should a socket user find the associated
event handle and flags? Some answers:

1. "pgsocket" could become a pointer to a heap-allocated wrapper
object containing { socket, event, flags } on Windows, or something
like that, but that seems a bit invasive and tangled up with public
APIs like libpq, which put me off trying that. I'm willing to explore
it if people object to my other idea.

2. "pgsocket" could stay unchanged, but we could have a parallel
array with extra socket state, indexed by file descriptor. We could
use new socket()/close() libpq events so that libpq's sockets could be
registered in this scheme without libpq itself having to know anything
about that. That worked pretty nicely when I developed it on my
FreeBSD box, but on Windows I soon learned that SOCKET is really yet
another name for HANDLE, so it's not a dense number space anchored at
0 like Unix file descriptors. The array could be prohibitively big.

3. I tried the same as #2 but with a hash table, and ran into another
small problem when putting it all together: we probably don't want to
longjump out of libpq callbacks on allocation failure. So, I modified
simplehash to add a no-OOM behaviour. That's the POC patch set I'm
attaching for show-and-tell. Some notes and TODOs in the commit
messages and comments.

Thoughts?

Attachments:

0001-Add-low-level-socket-events-for-libpq.patchapplication/x-patch; name=0001-Add-low-level-socket-events-for-libpq.patchDownload
From bdd90aeb65d82ecae8fe58b441d25a1e1b129bf3 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Sat, 29 Jan 2022 02:15:10 +1300
Subject: [PATCH 1/3] Add low level socket events for libpq.

Provide a way to get a callback when a socket is created or closed.

XXX TODO handle callback failure
XXX TODO investigate overheads/other implications of having a callback
installed
---
 src/interfaces/libpq/fe-connect.c   | 24 ++++++++++++++++++++++++
 src/interfaces/libpq/libpq-events.c | 11 +++++++++++
 src/interfaces/libpq/libpq-events.h | 10 +++++++++-
 3 files changed, 44 insertions(+), 1 deletion(-)

diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index a6a1db3356..ddc3f38cf1 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -452,7 +452,19 @@ pqDropConnection(PGconn *conn, bool flushInput)
 
 	/* Close the socket itself */
 	if (conn->sock != PGINVALID_SOCKET)
+	{
+		/* Report that the socket is closing. */
+		for (int i = 0; i < conn->nEvents; i++)
+		{
+			PGEventSocket evt;
+
+			evt.conn = conn;
+			evt.socket = conn->sock;
+			(void) conn->events[i].proc(PGEVT_SOCKETCLOSE, &evt,
+										conn->events[i].passThrough);
+		}
 		closesocket(conn->sock);
+	}
 	conn->sock = PGINVALID_SOCKET;
 
 	/* Optionally discard any unread data */
@@ -2576,6 +2588,18 @@ keep_going:						/* We will come back to here until there is
 						goto error_return;
 					}
 
+					/* Report that a new socket has been created. */
+					for (int i = 0; i < conn->nEvents; i++)
+					{
+						PGEventSocket evt;
+
+						evt.conn = conn;
+						evt.socket = conn->sock;
+						(void) conn->events[i].proc(PGEVT_SOCKET, &evt,
+													conn->events[i].passThrough);
+						/* XXX check call result and fail */
+					}
+
 					/*
 					 * Once we've identified a target address, all errors
 					 * except the preceding socket()-failure case should be
diff --git a/src/interfaces/libpq/libpq-events.c b/src/interfaces/libpq/libpq-events.c
index 7754c37748..2c549ec457 100644
--- a/src/interfaces/libpq/libpq-events.c
+++ b/src/interfaces/libpq/libpq-events.c
@@ -87,6 +87,17 @@ PQregisterEventProc(PGconn *conn, PGEventProc proc,
 		return false;
 	}
 
+	/* If we already have a socket, report a socket event immediately. */
+	if (conn->sock != PGINVALID_SOCKET)
+	{
+		PGEventSocket evt;
+
+		evt.conn = conn;
+		evt.socket = conn->sock;
+		(void) proc(PGEVT_SOCKET, &evt, passThrough);
+		/* XXX check return and fail (callback out of memory) */
+	}
+
 	return true;
 }
 
diff --git a/src/interfaces/libpq/libpq-events.h b/src/interfaces/libpq/libpq-events.h
index d93b7c4d4e..da64dc866b 100644
--- a/src/interfaces/libpq/libpq-events.h
+++ b/src/interfaces/libpq/libpq-events.h
@@ -31,7 +31,9 @@ typedef enum
 	PGEVT_CONNDESTROY,
 	PGEVT_RESULTCREATE,
 	PGEVT_RESULTCOPY,
-	PGEVT_RESULTDESTROY
+	PGEVT_RESULTDESTROY,
+	PGEVT_SOCKET,
+	PGEVT_SOCKETCLOSE
 } PGEventId;
 
 typedef struct
@@ -66,6 +68,12 @@ typedef struct
 	PGresult   *result;
 } PGEventResultDestroy;
 
+typedef struct
+{
+	PGconn	   *conn;
+	pgsocket   socket;
+} PGEventSocket;
+
 typedef int (*PGEventProc) (PGEventId evtId, void *evtInfo, void *passThrough);
 
 /* Registers an event proc with the given PGconn. */
-- 
2.33.1

0002-Allow-no-throw-allocators-in-simplehash.h.patchapplication/x-patch; name=0002-Allow-no-throw-allocators-in-simplehash.h.patchDownload
From 9e03fb83dcd95b7b793728b21726a1494943d47e Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Tue, 1 Feb 2022 11:21:57 +1300
Subject: [PATCH 2/3] Allow no-throw allocators in simplehash.h.

Previously, elog-style error reporting was assumed.  Allow SH_NO_OOM to
be defined, to indicate that you'd like creation and insertion to return
NULL to indicate failure instead, for use in no-throw programming
contexts.  A custom allocation function should return NULL on failure,
and the built-in MemoryContext support will use MCXT_NO_OOM when this
option is used.

Also allow SH_RAW_FREE to be defined, and SH_RAW_ALLOCATOR_NOZERO to
state that the SH_RAW_ALLOCATOR doesn't zero memory.  These
configuration points allow standard malloc and free to be used, which
was mostly interesting to allow testing of the SH_NO_OOM with different
allocator options.

XXX Proof-of-concept
---
 src/include/lib/simplehash.h | 74 +++++++++++++++++++++++++++++++-----
 1 file changed, 64 insertions(+), 10 deletions(-)

diff --git a/src/include/lib/simplehash.h b/src/include/lib/simplehash.h
index 8192927010..758669e1ea 100644
--- a/src/include/lib/simplehash.h
+++ b/src/include/lib/simplehash.h
@@ -41,7 +41,11 @@
  *	  - SH_SCOPE - in which scope (e.g. extern, static inline) do function
  *		declarations reside
  *	  - SH_RAW_ALLOCATOR - if defined, memory contexts are not used; instead,
- *	    use this to allocate bytes. The allocator must zero the returned space.
+ *	    use this to allocate bytes. The allocator must zero the returned space,
+ *	    or SH_RAW_ALLOCATOR_NOZERO must also be defined.
+ *	  - SH_RAW_FREE - if defined, use this to free memory, instead of pfree()
+ *	  - SH_NO_OOM - if defined, the allocator can fail by returning NULL,
+ *	    otherwise allocator failure is assumed to exit nonlocally (ERROR)
  *	  - SH_USE_NONDEFAULT_ALLOCATOR - if defined no element allocator functions
  *		are defined, so you can supply your own
  *	  The following parameters are only relevant when SH_DEFINE is defined:
@@ -204,14 +208,14 @@ SH_SCOPE void SH_DESTROY(SH_TYPE * tb);
 SH_SCOPE void SH_RESET(SH_TYPE * tb);
 
 /* void <prefix>_grow(<prefix>_hash *tb, uint64 newsize) */
-SH_SCOPE void SH_GROW(SH_TYPE * tb, uint64 newsize);
+SH_SCOPE bool SH_GROW(SH_TYPE * tb, uint64 newsize);
 
 /* <element> *<prefix>_insert(<prefix>_hash *tb, <key> key, bool *found) */
 SH_SCOPE	SH_ELEMENT_TYPE *SH_INSERT(SH_TYPE * tb, SH_KEY_TYPE key, bool *found);
 
 /*
  * <element> *<prefix>_insert_hash(<prefix>_hash *tb, <key> key, uint32 hash,
- * 								  bool *found)
+ *								  bool *found)
  */
 SH_SCOPE	SH_ELEMENT_TYPE *SH_INSERT_HASH(SH_TYPE * tb, SH_KEY_TYPE key,
 											uint32 hash, bool *found);
@@ -282,6 +286,12 @@ SH_SCOPE void SH_STAT(SH_TYPE * tb);
 #define SH_COMPARE_KEYS(tb, ahash, akey, b) (SH_EQUAL(tb, b->SH_KEY, akey))
 #endif
 
+#ifdef SH_NO_OOM
+#define SH_MCXT_FLAGS (MCXT_ALLOC_ZERO | MCXT_ALLOC_NO_OOM)
+#else
+#define SH_MCXT_FLAGS MCXT_ALLOC_ZERO
+#endif
+
 /*
  * Wrap the following definitions in include guards, to avoid multiple
  * definition errors if this header is included more than once.  The rest of
@@ -400,10 +410,18 @@ static inline void *
 SH_ALLOCATE(SH_TYPE * type, Size size)
 {
 #ifdef SH_RAW_ALLOCATOR
-	return SH_RAW_ALLOCATOR(size);
+	void *result = SH_RAW_ALLOCATOR(size);
+#ifdef SH_RAW_ALLOCATOR_NOZERO
+#ifdef SH_NO_OOM
+	if (!result)
+		return NULL;
+#endif
+	memset(result, 0, size);
+#endif
+	return result;
 #else
 	return MemoryContextAllocExtended(type->ctx, size,
-									  MCXT_ALLOC_HUGE | MCXT_ALLOC_ZERO);
+									  SH_MCXT_FLAGS | MCXT_ALLOC_HUGE);
 #endif
 }
 
@@ -411,7 +429,11 @@ SH_ALLOCATE(SH_TYPE * type, Size size)
 static inline void
 SH_FREE(SH_TYPE * type, void *pointer)
 {
+#ifdef SH_RAW_FREE
+	SH_RAW_FREE(pointer);
+#else
 	pfree(pointer);
+#endif
 }
 
 #endif
@@ -439,7 +461,16 @@ SH_CREATE(MemoryContext ctx, uint32 nelements, void *private_data)
 #ifdef SH_RAW_ALLOCATOR
 	tb = SH_RAW_ALLOCATOR(sizeof(SH_TYPE));
 #else
-	tb = MemoryContextAllocZero(ctx, sizeof(SH_TYPE));
+	tb = MemoryContextAllocExtended(ctx, sizeof(SH_TYPE), SH_MCXT_FLAGS);
+#endif
+#ifdef SH_NO_OOM
+	if (!tb)
+		return NULL;
+#endif
+#ifdef SH_RAW_ALLOCATOR_NOZERO
+	memset(tb, 0, sizeof(SH_TYPE));
+#endif
+#ifndef SH_RAW_ALLOCATOR
 	tb->ctx = ctx;
 #endif
 	tb->private_data = private_data;
@@ -451,6 +482,14 @@ SH_CREATE(MemoryContext ctx, uint32 nelements, void *private_data)
 
 	tb->data = SH_ALLOCATE(tb, sizeof(SH_ELEMENT_TYPE) * tb->size);
 
+#ifdef SH_NO_OOM
+	if (!tb->data)
+	{
+		SH_FREE(tb, tb);
+		return NULL;
+	}
+#endif
+
 	return tb;
 }
 
@@ -476,8 +515,10 @@ SH_RESET(SH_TYPE * tb)
  * Usually this will automatically be called by insertions/deletions, when
  * necessary. But resizing to the exact input size can be advantageous
  * performance-wise, when known at some point.
+ *
+ * Return true on success.  Can only return false if SH_NO_OOM is defined.
  */
-SH_SCOPE void
+SH_SCOPE bool
 SH_GROW(SH_TYPE * tb, uint64 newsize)
 {
 	uint64		oldsize = tb->size;
@@ -494,9 +535,18 @@ SH_GROW(SH_TYPE * tb, uint64 newsize)
 	/* compute parameters for new table */
 	SH_COMPUTE_PARAMETERS(tb, newsize);
 
-	tb->data = SH_ALLOCATE(tb, sizeof(SH_ELEMENT_TYPE) * tb->size);
+	newdata = SH_ALLOCATE(tb, sizeof(SH_ELEMENT_TYPE) * newsize);
+
+#ifdef SH_NO_OOM
+	if (!newdata)
+		return false;
+#endif
+#ifdef SH_RAW_ALLOCATOR_NOZERO
+	memset(newdata, 0, sizeof(SH_ELEMENT_TYPE) * newsize);
+#endif
 
-	newdata = tb->data;
+	tb->data = newdata;
+	tb->size = newsize;
 
 	/*
 	 * Copy entries from the old data to newdata. We theoretically could use
@@ -581,6 +631,8 @@ SH_GROW(SH_TYPE * tb, uint64 newsize)
 	}
 
 	SH_FREE(tb, olddata);
+
+	return true;
 }
 
 /*
@@ -615,7 +667,8 @@ restart:
 		 * When optimizing, it can be very useful to print these out.
 		 */
 		/* SH_STAT(tb); */
-		SH_GROW(tb, tb->size * 2);
+		if (!SH_GROW(tb, tb->size * 2))
+			return NULL;
 		/* SH_STAT(tb); */
 	}
 
@@ -1147,6 +1200,7 @@ SH_STAT(SH_TYPE * tb)
 #undef SH_GROW_MAX_MOVE
 #undef SH_GROW_MIN_FILLFACTOR
 #undef SH_MAX_SIZE
+#undef SH_MCXT_FLAGS
 
 /* types */
 #undef SH_TYPE
-- 
2.33.1

0003-Switch-to-per-socket-Windows-event-handles.patchapplication/x-patch; name=0003-Switch-to-per-socket-Windows-event-handles.patchDownload
From 2222901c1947bb4c72f26b5706f5f3248cec96b8 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Sun, 30 Jan 2022 13:01:13 +1300
Subject: [PATCH 3/3] Switch to per-socket Windows event handles.

Previously, each WaitEventSet would create a new operating system
'event' object for each socket.  Create a dedicated event and some state
flags for each socket, independent of WaitEventSets.  This fixes three
problems:

1.  When a socket is used in a new WaitEventSet (for example, one after
another when using temporary WaitEventSets), queued but not yet reported
FD_CLOSE events could be lost by the operating system.  This is now
avoided with a single long-lived event for the socket.

2.  We need to report Winsock's FD_CLOSE event as WL_SOCKET_READABLE
with level-triggered semantics, but it's only reported once by Winsock
on graceful socket shutdown, so could be lost if a caller waits twice in
a row without reading the socket.  This is avoided by adding a flag to
remember that we've already received FD_CLOSE.

3.  It should also be possible to use a socket in two WaitEventSet
objects at the same time (though not from two threads without more
work), since the network event mask will be adjusted on each wait if
required when you switch between sets containing the same socket.
Though this isn't currently needed by existing PostgreSQL code AFAIK,
it's a nice restriction to lift, to match Unix.

Manage the new extra per-socket state in a hash table indexed by socket,
and require clients of WaitEventSet to register and deregister sockets
with a new API.  On Unix this is a no-op, other than correctness checks
in assertion builds.  For libwalreceiver and postgres_fdw, a new
libpq-events callback automatically manages socket registration.

XXX Proof of concept

XXX There is some more temporary event-switching going on inside
src/backend/port/win32/socket.c, but not in code paths that were
affected by the recent graceful shutdown brokenness.  They may need to
be integrated with this, potentially widening the set of sockets that
need to be registered with socket_table.c to 'all of them'?

Discussion: https://postgr.es/m/CA%2BhUKG%2BOeoETZQ%3DQw5Ub5h3tmwQhBmDA%3DnuNO3KG%3DzWfUypFAw%40mail.gmail.com
---
 contrib/postgres_fdw/connection.c             |  25 +++
 src/backend/libpq/pqcomm.c                    |   3 +
 src/backend/port/Makefile                     |   3 +-
 src/backend/port/socket_table.c               | 160 ++++++++++++++++++
 src/backend/postmaster/pgstat.c               |  38 +++--
 src/backend/postmaster/syslogger.c            |   3 +
 .../libpqwalreceiver/libpqwalreceiver.c       |  33 ++++
 src/backend/storage/ipc/latch.c               |  73 ++++----
 src/include/port.h                            |  18 ++
 src/include/storage/latch.h                   |   2 +-
 10 files changed, 302 insertions(+), 56 deletions(-)
 create mode 100644 src/backend/port/socket_table.c

diff --git a/contrib/postgres_fdw/connection.c b/contrib/postgres_fdw/connection.c
index 29fcb6a76e..783b9a3a74 100644
--- a/contrib/postgres_fdw/connection.c
+++ b/contrib/postgres_fdw/connection.c
@@ -17,6 +17,7 @@
 #include "catalog/pg_user_mapping.h"
 #include "commands/defrem.h"
 #include "funcapi.h"
+#include "libpq-events.h"
 #include "mb/pg_wchar.h"
 #include "miscadmin.h"
 #include "pgstat.h"
@@ -109,6 +110,7 @@ static void pgfdw_abort_cleanup(ConnCacheEntry *entry, const char *sql,
 								bool toplevel);
 static bool UserMappingPasswordRequired(UserMapping *user);
 static bool disconnect_cached_connections(Oid serverid);
+static int	pgfdw_eventproc(PGEventId evtId, void *evtInfo, void *passThrough);
 
 /*
  * Get a PGconn which can be used to execute queries on the remote PostgreSQL
@@ -469,6 +471,10 @@ connect_pg_server(ForeignServer *server, UserMapping *user)
 							server->servername),
 					 errdetail_internal("%s", pchomp(PQerrorMessage(conn)))));
 
+		/* Make sure that all sockets are registered for use by latch.c. */
+		if (!PQregisterEventProc(conn, pgfdw_eventproc, "postgres_fdw", NULL))
+			elog(ERROR, "postgres_fdw could not register for libpq events");
+
 		/*
 		 * Check that non-superuser has used password to establish connection;
 		 * otherwise, he's piggybacking on the postgres server's user
@@ -1709,3 +1715,22 @@ disconnect_cached_connections(Oid serverid)
 
 	return result;
 }
+
+/*
+ * A callback for libpq to allow its sockets to be used in WaitEventSet on Windows.
+ */
+static int
+pgfdw_eventproc(PGEventId evtId, void *evtInfo, void *passThrough)
+{
+	if (evtId == PGEVT_SOCKET)
+	{
+		PGEventSocket *evt = (PGEventSocket *) evtInfo;
+		return SocketTableAdd(evt->socket, true) ? 1 : 0;
+	}
+	else if (evtId == PGEVT_SOCKETCLOSE)
+	{
+		PGEventSocket *evt = (PGEventSocket *) evtInfo;
+		SocketTableDrop(evt->socket);
+	}
+	return 1;
+}
diff --git a/src/backend/libpq/pqcomm.c b/src/backend/libpq/pqcomm.c
index f05723dc92..bb35618af4 100644
--- a/src/backend/libpq/pqcomm.c
+++ b/src/backend/libpq/pqcomm.c
@@ -204,6 +204,9 @@ pq_init(void)
 				(errmsg("could not set socket to nonblocking mode: %m")));
 #endif
 
+	/* Make sure the socket can be used in a WaitEventSet on Windows. */
+	SocketTableAdd(MyProcPort->sock, false);
+
 	FeBeWaitSet = CreateWaitEventSet(TopMemoryContext, 3);
 	socket_pos = AddWaitEventToSet(FeBeWaitSet, WL_SOCKET_WRITEABLE,
 								   MyProcPort->sock, NULL, NULL);
diff --git a/src/backend/port/Makefile b/src/backend/port/Makefile
index 2d00b4f05a..03df581e06 100644
--- a/src/backend/port/Makefile
+++ b/src/backend/port/Makefile
@@ -25,7 +25,8 @@ OBJS = \
 	$(TAS) \
 	atomics.o \
 	pg_sema.o \
-	pg_shmem.o
+	pg_shmem.o\
+	socket_table.o
 
 ifeq ($(PORTNAME), win32)
 SUBDIRS += win32
diff --git a/src/backend/port/socket_table.c b/src/backend/port/socket_table.c
new file mode 100644
index 0000000000..c2bbc2e0cb
--- /dev/null
+++ b/src/backend/port/socket_table.c
@@ -0,0 +1,160 @@
+/*-------------------------------------------------------------------------
+ *
+ * socket_table.c
+ *	  Routines for dealing with extra per-socket state on Windows.
+ *
+ * Windows sockets can only safely be associate with one 'event', or they risk
+ * losing FD_CLOSE events.  FD_CLOSE is also edge-triggered and not
+ * resettable, so we need space for state to make it level-triggered.
+ *
+ * These functions are no-ops on Unix systems, except in assertion builds
+ * where we do the minimum book keeping required to report failure to register
+ * sockets appropriately.
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ *	  src/backend/port/socket_table.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "common/hashfn.h"
+
+typedef struct SocketTableEntry
+{
+	pgsocket	sock;
+	int			status;
+	ExtraSocketState *extra;
+} SocketTableEntry;
+
+#define SH_PREFIX		socktab
+#define SH_ELEMENT_TYPE SocketTableEntry
+#define SH_KEY_TYPE		pgsocket
+#define SH_KEY			sock
+#define SH_HASH_KEY(tb, key)	hash_bytes((uint8 *) &(key), sizeof(key))
+#define SH_EQUAL(tb, a, b)		((a) == (b))
+#define SH_SCOPE static inline
+#define SH_NO_OOM
+#define SH_DECLARE
+#define SH_DEFINE
+#include "lib/simplehash.h"
+
+#if defined(WIN32) || defined(USE_ASSERT_CHECKING)
+static socktab_hash *socket_table;
+#endif
+
+/*
+ * Must be called exactly once for each socket before including it in code
+ * that requires ExtraSocketState, such as WaitEventSet.  The socket must be
+ * not already registered, and must be dropped before closing it.  Return NULL
+ * on out-of-memory, if no_oom is true, otherwise raises errors.
+ */
+ExtraSocketState *
+SocketTableAdd(pgsocket sock, bool no_oom)
+{
+#if defined(WIN32) || defined(USE_ASSERT_CHECKING)
+	SocketTableEntry *ste;
+	ExtraSocketState *ess;
+	bool		found;
+
+	/*
+	 * Create a new SocketTableEntry object.  We can't use a pointer into the
+	 * hash table because it would move around, and we want WaitEventSet to be
+	 * able to hold pointers.
+	 */
+	if (!(ess = malloc(sizeof(*ess))))
+		goto out_of_memory;
+	memset(ess, 0, sizeof(*ess));
+
+#ifdef WIN32
+	/* Set up Windows kernel event for this socket. */
+	ess->event_handle = WSACreateEvent();
+	if (ess->event_handle == WSA_INVALID_EVENT)
+		goto out_of_memory;
+#endif
+
+	/* Create socket_table on demand. */
+	/* XXX dedicated memory context for debugging? */
+	if (!socket_table &&
+		(!(socket_table = socktab_create(TopMemoryContext, 128, NULL))))
+		goto out_of_memory;
+
+	/* Try to insert. */
+	ste = socktab_insert(socket_table, sock, &found);
+	if (!ste)
+		goto out_of_memory;
+	if (found)
+	{
+#ifdef WIN32
+		WSACloseEvent(ess->event_handle);
+#endif
+		free(ess);
+		elog(ERROR, "cannot register socket, already registered");
+	}
+
+	/* Success. */
+	ste->extra = ess;
+	return ess;
+
+ out_of_memory:
+	if (ess)
+	{
+#ifdef WIN32
+		if (ess->event_handle != WSA_INVALID_EVENT)
+			WSACloseEvent(ess->event_handle);
+#endif
+		free(ess);
+	}
+	if (!no_oom)
+		elog(ERROR, "out of memory");
+	return NULL;
+#else
+	return NULL;
+#endif
+}
+
+/*
+ * Unregister a socket that was registered.  This must be done before closing
+ * the socket.
+ */
+void
+SocketTableDrop(pgsocket sock)
+{
+#if defined(WIN32) || defined(USE_ASSERT_CHECKING)
+	SocketTableEntry *ste = NULL;
+	ExtraSocketState *ess;
+
+	if (socket_table)
+		ste = socktab_lookup(socket_table, sock);
+	if (!ste)
+		elog(ERROR, "cannot unregister socket that is not registered");
+	ess = ste->extra;
+
+#ifdef WIN32
+	WSACloseEvent(ess->event_handle);
+#endif
+
+	free(ess);
+	socktab_delete_item(socket_table, ste);
+#endif
+}
+
+ExtraSocketState *
+SocketTableGet(pgsocket sock)
+{
+#if defined(WIN32) || defined(USE_ASSERT_CHECKING)
+	SocketTableEntry *ste = NULL;
+
+	if (socket_table)
+		ste = socktab_lookup(socket_table, sock);
+	if (!ste)
+		elog(ERROR, "socket is not registered");
+	return ste->extra;
+#else
+	return NULL;
+#endif
+}
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 0646f53098..dd95353880 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -382,6 +382,7 @@ static void pgstat_recv_replslot(PgStat_MsgReplSlot *msg, int len);
 static void pgstat_recv_tempfile(PgStat_MsgTempFile *msg, int len);
 static void pgstat_recv_subscription_purge(PgStat_MsgSubscriptionPurge *msg, int len);
 static void pgstat_recv_subworker_error(PgStat_MsgSubWorkerError *msg, int len);
+static void pgstat_release_socket(void);
 
 /* ------------------------------------------------------------
  * Public functions called from postmaster follow
@@ -483,8 +484,7 @@ pgstat_init(void)
 			ereport(LOG,
 					(errcode_for_socket_access(),
 					 errmsg("could not bind socket for statistics collector: %m")));
-			closesocket(pgStatSock);
-			pgStatSock = PGINVALID_SOCKET;
+			pgstat_release_socket();
 			continue;
 		}
 
@@ -494,8 +494,7 @@ pgstat_init(void)
 			ereport(LOG,
 					(errcode_for_socket_access(),
 					 errmsg("could not get address of socket for statistics collector: %m")));
-			closesocket(pgStatSock);
-			pgStatSock = PGINVALID_SOCKET;
+			pgstat_release_socket();
 			continue;
 		}
 
@@ -510,8 +509,7 @@ pgstat_init(void)
 			ereport(LOG,
 					(errcode_for_socket_access(),
 					 errmsg("could not connect socket for statistics collector: %m")));
-			closesocket(pgStatSock);
-			pgStatSock = PGINVALID_SOCKET;
+			pgstat_release_socket();
 			continue;
 		}
 
@@ -531,8 +529,7 @@ retry1:
 			ereport(LOG,
 					(errcode_for_socket_access(),
 					 errmsg("could not send test message on socket for statistics collector: %m")));
-			closesocket(pgStatSock);
-			pgStatSock = PGINVALID_SOCKET;
+			pgstat_release_socket();
 			continue;
 		}
 
@@ -557,8 +554,7 @@ retry1:
 			ereport(LOG,
 					(errcode_for_socket_access(),
 					 errmsg("select() failed in statistics collector: %m")));
-			closesocket(pgStatSock);
-			pgStatSock = PGINVALID_SOCKET;
+			pgstat_release_socket();
 			continue;
 		}
 		if (sel_res == 0 || !FD_ISSET(pgStatSock, &rset))
@@ -572,8 +568,7 @@ retry1:
 			ereport(LOG,
 					(errcode(ERRCODE_CONNECTION_FAILURE),
 					 errmsg("test message did not get through on socket for statistics collector")));
-			closesocket(pgStatSock);
-			pgStatSock = PGINVALID_SOCKET;
+			pgstat_release_socket();
 			continue;
 		}
 
@@ -587,8 +582,7 @@ retry2:
 			ereport(LOG,
 					(errcode_for_socket_access(),
 					 errmsg("could not receive test message on socket for statistics collector: %m")));
-			closesocket(pgStatSock);
-			pgStatSock = PGINVALID_SOCKET;
+			pgstat_release_socket();
 			continue;
 		}
 
@@ -597,8 +591,7 @@ retry2:
 			ereport(LOG,
 					(errcode(ERRCODE_INTERNAL_ERROR),
 					 errmsg("incorrect test message transmission on socket for statistics collector")));
-			closesocket(pgStatSock);
-			pgStatSock = PGINVALID_SOCKET;
+			pgstat_release_socket();
 			continue;
 		}
 
@@ -669,7 +662,7 @@ startup_failed:
 		pg_freeaddrinfo_all(hints.ai_family, addrs);
 
 	if (pgStatSock != PGINVALID_SOCKET)
-		closesocket(pgStatSock);
+		pgstat_release_socket();
 	pgStatSock = PGINVALID_SOCKET;
 
 	/*
@@ -3527,6 +3520,9 @@ PgstatCollectorMain(int argc, char *argv[])
 	pgStatRunningInCollector = true;
 	pgStatDBHash = pgstat_read_statsfiles(InvalidOid, true, true);
 
+	/* Make sure pgStatSock can be used in a WaitEventSet on Windows. */
+	SocketTableAdd(pgStatSock, false);
+
 	/* Prepare to wait for our latch or data in our socket. */
 	wes = CreateWaitEventSet(CurrentMemoryContext, 3);
 	AddWaitEventToSet(wes, WL_LATCH_SET, PGINVALID_SOCKET, MyLatch, NULL);
@@ -6418,3 +6414,11 @@ pgstat_count_slru_truncate(int slru_idx)
 {
 	slru_entry(slru_idx)->m_truncate += 1;
 }
+
+static void
+pgstat_release_socket(void)
+{
+	SocketTableDrop(pgStatSock);
+	closesocket(pgStatSock);
+	pgStatSock = PGINVALID_SOCKET;
+}
diff --git a/src/backend/postmaster/syslogger.c b/src/backend/postmaster/syslogger.c
index 25e2131e31..aa551b2719 100644
--- a/src/backend/postmaster/syslogger.c
+++ b/src/backend/postmaster/syslogger.c
@@ -301,6 +301,9 @@ SysLoggerMain(int argc, char *argv[])
 	 */
 	whereToSendOutput = DestNone;
 
+	/* Make sure the pipe can be used in a WaitEventSet on Windows. */
+	SocketTableAdd((pgsocket) syslogPipe[0], false);
+
 	/*
 	 * Set up a reusable WaitEventSet object we'll use to wait for our latch,
 	 * and (except on Windows) our socket.
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 0d89db4e6a..8508f93aa6 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -23,6 +23,7 @@
 #include "catalog/pg_type.h"
 #include "common/connect.h"
 #include "funcapi.h"
+#include "libpq-events.h"
 #include "libpq-fe.h"
 #include "mb/pg_wchar.h"
 #include "miscadmin.h"
@@ -83,6 +84,10 @@ static WalRcvExecResult *libpqrcv_exec(WalReceiverConn *conn,
 									   const Oid *retTypes);
 static void libpqrcv_disconnect(WalReceiverConn *conn);
 
+static int libpqrcv_eventproc(PGEventId evtId,
+							  void *evtInfo,
+							  void *passThrough);
+
 static WalReceiverFunctionsType PQWalReceiverFunctions = {
 	libpqrcv_connect,
 	libpqrcv_check_conninfo,
@@ -182,6 +187,15 @@ libpqrcv_connect(const char *conninfo, bool logical, const char *appname,
 		return NULL;
 	}
 
+	if (PQregisterEventProc(conn->streamConn, libpqrcv_eventproc,
+							"libwalrcv", NULL) == 0)
+	{
+		PQfinish(conn->streamConn);
+		conn->streamConn = NULL;
+		*err = pstrdup("could not register for libpq events");
+		return NULL;
+	}
+
 	/*
 	 * Poll connection until we have OK or FAILED status.
 	 *
@@ -1165,3 +1179,22 @@ stringlist_to_identifierstr(PGconn *conn, List *strings)
 
 	return res.data;
 }
+
+/*
+ * A callback for libpq to allow its sockets to be used in WaitEventSet on Windows.
+ */
+static int
+libpqrcv_eventproc(PGEventId evtId, void *evtInfo, void *passThrough)
+{
+	if (evtId == PGEVT_SOCKET)
+	{
+		PGEventSocket *evt = (PGEventSocket *) evtInfo;
+		return SocketTableAdd(evt->socket, true) ? 1 : 0;
+	}
+	else if (evtId == PGEVT_SOCKETCLOSE)
+	{
+		PGEventSocket *evt = (PGEventSocket *) evtInfo;
+		SocketTableDrop(evt->socket);
+	}
+	return 1;
+}
diff --git a/src/backend/storage/ipc/latch.c b/src/backend/storage/ipc/latch.c
index 61c876beff..5a6fb36fa2 100644
--- a/src/backend/storage/ipc/latch.c
+++ b/src/backend/storage/ipc/latch.c
@@ -60,6 +60,7 @@
 #include "storage/shmem.h"
 #include "utils/memutils.h"
 
+
 /*
  * Select the fd readiness primitive to use. Normally the "most modern"
  * primitive supported by the OS will be used, but for testing it can be
@@ -803,28 +804,6 @@ FreeWaitEventSet(WaitEventSet *set)
 #elif defined(WAIT_USE_KQUEUE)
 	close(set->kqueue_fd);
 	ReleaseExternalFD();
-#elif defined(WAIT_USE_WIN32)
-	WaitEvent  *cur_event;
-
-	for (cur_event = set->events;
-		 cur_event < (set->events + set->nevents);
-		 cur_event++)
-	{
-		if (cur_event->events & WL_LATCH_SET)
-		{
-			/* uses the latch's HANDLE */
-		}
-		else if (cur_event->events & WL_POSTMASTER_DEATH)
-		{
-			/* uses PostmasterHandle */
-		}
-		else
-		{
-			/* Clean up the event object we created for the socket */
-			WSAEventSelect(cur_event->fd, NULL, 0);
-			WSACloseEvent(set->handles[cur_event->pos + 1]);
-		}
-	}
 #endif
 
 	pfree(set);
@@ -897,9 +876,17 @@ AddWaitEventToSet(WaitEventSet *set, uint32 events, pgsocket fd, Latch *latch,
 	event->fd = fd;
 	event->events = events;
 	event->user_data = user_data;
+
+	if (events & WL_SOCKET_MASK)
+	{
 #ifdef WIN32
-	event->reset = false;
+		/* Point to our Windows event tracking state for the socket. */
+		event->extra_socket_state = SocketTableGet(fd);
+#else
+		/* On Unix, check it's registered to avoid portability bugs. */
+		(void) SocketTableGet(fd);
 #endif
+	}
 
 	if (events == WL_LATCH_SET)
 	{
@@ -1265,6 +1252,7 @@ WaitEventAdjustWin32(WaitEventSet *set, WaitEvent *event)
 	}
 	else
 	{
+		ExtraSocketState *ess;
 		int			flags = FD_CLOSE;	/* always check for errors/EOF */
 
 		if (event->events & WL_SOCKET_READABLE)
@@ -1274,18 +1262,19 @@ WaitEventAdjustWin32(WaitEventSet *set, WaitEvent *event)
 		if (event->events & WL_SOCKET_CONNECTED)
 			flags |= FD_CONNECT;
 
-		if (*handle == WSA_INVALID_EVENT)
-		{
-			*handle = WSACreateEvent();
-			if (*handle == WSA_INVALID_EVENT)
-				elog(ERROR, "failed to create event for socket: error code %d",
-					 WSAGetLastError());
-		}
-		if (WSAEventSelect(event->fd, *handle, flags) != 0)
+		/* Find the event associated with this socket. */
+		ess = event->extra_socket_state;
+		*handle = ess->event_handle;
+
+		/*
+		 * Adjust the flags for this event, if they don't already match what we
+		 * want to wait for.
+		 */
+		if (ess->flags != flags &&
+			WSAEventSelect(event->fd, *handle, flags) != 0)
 			elog(ERROR, "failed to set up event for socket: error code %d",
 				 WSAGetLastError());
-
-		Assert(event->fd != PGINVALID_SOCKET);
+		ess->flags = flags;
 	}
 }
 #endif
@@ -1845,10 +1834,19 @@ WaitEventSetWaitBlock(WaitEventSet *set, int cur_timeout,
 		 cur_event < (set->events + set->nevents);
 		 cur_event++)
 	{
-		if (cur_event->reset)
-		{
+		/* Reset the event mask if necessary. */
+		if (cur_event->events & WL_SOCKET_MASK)
 			WaitEventAdjustWin32(set, cur_event);
-			cur_event->reset = false;
+
+		/* If we've already seen FD_CLOSE, keep reporting readiness. */
+		if ((cur_event->events & WL_SOCKET_MASK) &&
+			cur_event->extra_socket_state->seen_fd_close)
+		{
+			occurred_events->pos = cur_event->pos;
+			occurred_events->user_data = cur_event->user_data;
+			occurred_events->events = cur_event->events;
+			occurred_events->fd = cur_event->fd;
+			return 1;
 		}
 
 		/*
@@ -1984,7 +1982,7 @@ WaitEventSetWaitBlock(WaitEventSet *set, int cur_timeout,
 			 * for the behavior of socket events.
 			 *------
 			 */
-			cur_event->reset = true;
+			cur_event->extra_socket_state->flags = 0;
 		}
 		if ((cur_event->events & WL_SOCKET_WRITEABLE) &&
 			(resEvents.lNetworkEvents & FD_WRITE))
@@ -2002,6 +2000,7 @@ WaitEventSetWaitBlock(WaitEventSet *set, int cur_timeout,
 		{
 			/* EOF/error, so signal all caller-requested socket flags */
 			occurred_events->events |= (cur_event->events & WL_SOCKET_MASK);
+			cur_event->extra_socket_state->seen_fd_close = true;
 		}
 
 		if (occurred_events->events != 0)
diff --git a/src/include/port.h b/src/include/port.h
index 3d103a2b31..e780852bd1 100644
--- a/src/include/port.h
+++ b/src/include/port.h
@@ -543,4 +543,22 @@ extern char *wait_result_to_str(int exit_status);
 extern bool wait_result_is_signal(int exit_status, int signum);
 extern bool wait_result_is_any_signal(int exit_status, bool include_command_not_found);
 
+/* backend/port/socket_table.c */
+#if !defined(FRONTEND)
+struct ExtraSocketState
+{
+#ifdef WIN32
+	HANDLE		event_handle;		/* one event for the life of the socket */
+	int			flags;				/* most recent WSAEventSelect() flags */
+	bool		seen_fd_close;		/* has FD_CLOSE been received? */
+#else
+	int			dummy;				/* none of this is needed for Unix */
+#endif
+};
+typedef struct ExtraSocketState ExtraSocketState;
+extern ExtraSocketState *SocketTableAdd(pgsocket sock, bool no_oom);
+extern ExtraSocketState *SocketTableGet(pgsocket sock);
+extern void SocketTableDrop(pgsocket sock);
+#endif
+
 #endif							/* PG_PORT_H */
diff --git a/src/include/storage/latch.h b/src/include/storage/latch.h
index 3aa7b33834..576d339829 100644
--- a/src/include/storage/latch.h
+++ b/src/include/storage/latch.h
@@ -146,7 +146,7 @@ typedef struct WaitEvent
 	pgsocket	fd;				/* socket fd associated with event */
 	void	   *user_data;		/* pointer provided in AddWaitEventToSet */
 #ifdef WIN32
-	bool		reset;			/* Is reset of the event required? */
+	ExtraSocketState *extra_socket_state;
 #endif
 } WaitEvent;
 
-- 
2.33.1

#40Andres Freund
andres@anarazel.de
In reply to: Thomas Munro (#39)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

Hi,

On 2022-02-01 18:02:34 +1300, Thomas Munro wrote:

1. It sounds like no one really loves the WSAPoll() kludge, even
though it apparently works for simple cases.

Yea, at least I don't :)

2. The long-lived-WaitEventSets-everywhere concept was initially
appealling to me and solves the walreceiver problem (when combined
with a sticky seen_fd_close flag), and I've managed to get that
working correctly across libpq reconnects. As mentioned, I also have
some toy patches along those lines for the equivalent but more complex
problem in postgres_fdw, because I've been studying how to make
parallel append generate a tidy stream of epoll_wait()/kevent() calls,
instead of a quadratic explosion of setup/teardown spam. I'll write
some more about those patches and hopefully propose them soon anyway,
but on reflection I don't really want that Unix efficiency problem to
be tangled up with this Windows correctness problem. That'd require a
programming rule that I don't want to burden us with forever: you'd
*never* be able to use a socket in more than one WaitEventSet, and
WaitLatchOrSocket() would have to be removed.

Which seems like a bad direction to go in.

3. The real solution to this problem is to recognise that we just
have the event objects in the wrong place. WaitEventSets shouldn't
own them: they need to be 1:1 with sockets, or Winsock will eat
events. Likewise for the flag you need for edge->level conversion, or
*we'll* eat events. Having now tried that, it's starting to feel like
the best way forward, even though my initial prototype (see attached)
is maybe a tad cumbersome with bookkeeping. I believe it means that
all existing coding patterns *should* now be safe (not yet confirmed
by testing), and we're free to put sockets in multiple WESes even at
the same time if the need arises.

Agreed.

The basic question is: how should a socket user find the associated
event handle and flags? Some answers:

1. "pgsocket" could become a pointer to a heap-allocated wrapper
object containing { socket, event, flags } on Windows, or something
like that, but that seems a bit invasive and tangled up with public
APIs like libpq, which put me off trying that. I'm willing to explore
it if people object to my other idea.

I'm not sure if the libpq aspect really is a problem. We're not going to have
to do that conversion repeatedly, I think.

2. "pgsocket" could stay unchanged, but we could have a parallel
array with extra socket state, indexed by file descriptor. We could
use new socket()/close() libpq events so that libpq's sockets could be
registered in this scheme without libpq itself having to know anything
about that. That worked pretty nicely when I developed it on my
FreeBSD box, but on Windows I soon learned that SOCKET is really yet
another name for HANDLE, so it's not a dense number space anchored at
0 like Unix file descriptors. The array could be prohibitively big.

Yes, that seems like a no-go. It also doesn't seem like it'd gain much in the
robustness department over 1) - you'd not know if a socket had been closed and
a new one with the same integer value had been created.

3. I tried the same as #2 but with a hash table, and ran into another
small problem when putting it all together: we probably don't want to
longjump out of libpq callbacks on allocation failure. So, I modified
simplehash to add a no-OOM behaviour. That's the POC patch set I'm
attaching for show-and-tell. Some notes and TODOs in the commit
messages and comments.

1) seems more plausible to me, but I can see this working as well.

From bdd90aeb65d82ecae8fe58b441d25a1e1b129bf3 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Sat, 29 Jan 2022 02:15:10 +1300
Subject: [PATCH 1/3] Add low level socket events for libpq.

Provide a way to get a callback when a socket is created or closed.

XXX TODO handle callback failure
XXX TODO investigate overheads/other implications of having a callback
installed

What do we need this for? I still don't understand what kind of reconnects we
need to automatically need to detect.

+#ifdef SH_RAW_ALLOCATOR_NOZERO
+	memset(tb, 0, sizeof(SH_TYPE));
+#endif

...

+#ifdef SH_RAW_ALLOCATOR_NOZERO
+	memset(newdata, 0, sizeof(SH_ELEMENT_TYPE) * newsize);
+#endif

Seems like this should be handled in an allocator wrapper, rather than in
multiple places in the simplehash code?

+#if defined(WIN32) || defined(USE_ASSERT_CHECKING)
+static socktab_hash *socket_table;
+#endif

Perhaps a separate #define for this would be appropriate? So we don't have to
spell the exact condition out every time.

+ExtraSocketState *
+SocketTableAdd(pgsocket sock, bool no_oom)
+{
+#if defined(WIN32) || defined(USE_ASSERT_CHECKING)
+	SocketTableEntry *ste;
+	ExtraSocketState *ess;

Given there's goto targets that test for ess != NULL, it seems nicer to
initialize it to NULL. I don't think there's problematic paths right now, but
it seems unnecessary to "risk" that changing over time.

+#if !defined(FRONTEND)
+struct ExtraSocketState
+{
+#ifdef WIN32
+	HANDLE		event_handle;		/* one event for the life of the socket */
+	int			flags;				/* most recent WSAEventSelect() flags */
+	bool		seen_fd_close;		/* has FD_CLOSE been received? */
+#else
+	int			dummy;				/* none of this is needed for Unix */
+#endif
+};

Seems like we might want to track more readiness events than just close? If we
e.g. started tracking whether we've seen writes blocking / write readiness,
we could get rid of cruft like

/*
* Windows does not guarantee to log an FD_WRITE network event
* indicating that more data can be sent unless the previous send()
* failed with WSAEWOULDBLOCK. While our caller might well have made
* such a call, we cannot assume that here. Therefore, if waiting for
* write-ready, force the issue by doing a dummy send(). If the dummy
* send() succeeds, assume that the socket is in fact write-ready, and
* return immediately. Also, if it fails with something other than
* WSAEWOULDBLOCK, return a write-ready indication to let our caller
* deal with the error condition.
*/

that seems likely to just make bugs less likely, rather than actually fix them...

Greetings,

Andres Freund

#41Thomas Munro
thomas.munro@gmail.com
In reply to: Andres Freund (#40)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

On Wed, Feb 2, 2022 at 6:38 AM Andres Freund <andres@anarazel.de> wrote:

On 2022-02-01 18:02:34 +1300, Thomas Munro wrote:

1. "pgsocket" could become a pointer to a heap-allocated wrapper
object containing { socket, event, flags } on Windows, or something
like that, but that seems a bit invasive and tangled up with public
APIs like libpq, which put me off trying that. I'm willing to explore
it if people object to my other idea.

I'm not sure if the libpq aspect really is a problem. We're not going to have
to do that conversion repeatedly, I think.

Alright, I'm prototyping that variant today.

Provide a way to get a callback when a socket is created or closed.

XXX TODO handle callback failure
XXX TODO investigate overheads/other implications of having a callback
installed

What do we need this for? I still don't understand what kind of reconnects we
need to automatically need to detect.

libpq makes new sockets in various cases like when trying multiple
hosts/ports (the easiest test to set up) or in some SSL and GSSAPI
cases. In the model shown in the most recent patch where there is a
hash table holding ExtraSocketState objects that libpq doesn't even
know about, we have to do SocketTableDrop(old socket),
SocketTableAdd(new socket) at those times, which is why I introduced
that callback.

If we switch to the model where a socket is really a pointer to a
wrapper struct (which I'm about to prototype), the need for all that
bookkeeping goes away, no callbacks, no hash table, but now libpq has
to participate knowingly in a socket wrapping scheme to help the
backend while also somehow providing unwrapped SOCKET for client API
stability. Trying some ideas, more on that soon.

+#if !defined(FRONTEND)
+struct ExtraSocketState
+{
+#ifdef WIN32
+     HANDLE          event_handle;           /* one event for the life of the socket */
+     int                     flags;                          /* most recent WSAEventSelect() flags */
+     bool            seen_fd_close;          /* has FD_CLOSE been received? */
+#else
+     int                     dummy;                          /* none of this is needed for Unix */
+#endif
+};

Seems like we might want to track more readiness events than just close? If we
e.g. started tracking whether we've seen writes blocking / write readiness,
we could get rid of cruft like

/*
* Windows does not guarantee to log an FD_WRITE network event
* indicating that more data can be sent unless the previous send()
* failed with WSAEWOULDBLOCK. While our caller might well have made
* such a call, we cannot assume that here. Therefore, if waiting for
* write-ready, force the issue by doing a dummy send(). If the dummy
* send() succeeds, assume that the socket is in fact write-ready, and
* return immediately. Also, if it fails with something other than
* WSAEWOULDBLOCK, return a write-ready indication to let our caller
* deal with the error condition.
*/

that seems likely to just make bugs less likely, rather than actually fix them...

Yeah. Unlike FD_CLOSE, FD_WRITE is a non-terminal condition so would
also need to be cleared, which requires seeing all
send()/sendto()/write() calls with wrapper functions, but we already
do stuff like that. Looking into it...

#42Andres Freund
andres@anarazel.de
In reply to: Thomas Munro (#37)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

Hi,

Over in another thread I made some wild unsubstantiated guesses that the
windows issues could have been made much more likely by a somewhat odd bit of
code in PQisBusy():

/messages/by-id/1959196.1644544971@sss.pgh.pa.us

Alexander, any chance you'd try if that changes the likelihood of the problem
occurring, without any other fixes / reverts applied?

Greetings,

Andres Freund

#43Alexander Lakhin
exclusion@gmail.com
In reply to: Andres Freund (#42)
1 attachment(s)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

Hello Andres,
11.02.2022 05:22, Andres Freund wrote:

Over in another thread I made some wild unsubstantiated guesses that the
windows issues could have been made much more likely by a somewhat odd bit of
code in PQisBusy():

/messages/by-id/1959196.1644544971@sss.pgh.pa.us

Alexander, any chance you'd try if that changes the likelihood of the problem
occurring, without any other fixes / reverts applied?

Unfortunately I haven't seen an improvement for the test in question.
With the PQisBusy-fix.patch from [1]/messages/by-id/2187263.1644616494@sss.pgh.pa.us and without any other changes on
the master branch (52377bb8) it still fails (on iterations 13, 5, 2, 2
for me).
The diagnostic logging (in attachment) added:
2022-02-12 01:04:16.341 PST [4912] LOG:  libpqrcv_receive: PQgetCopyData
returned 0
2022-02-12 01:04:16.341 PST [4912] LOG:  libpqrcv_receive: PQgetCopyData
2 returned -1
2022-02-12 01:04:16.341 PST [4912] LOG:  libpqrcv_receive:
end-of-streaming or error: -1
2022-02-12 01:04:16.341 PST [4912] LOG:  libpqrcv_PQgetResult:
streamConn->asyncStatus: 1 && streamConn->status: 0
2022-02-12 01:04:16.341 PST [4912] LOG:  libpqrcv_receive
libpqrcv_PQgetResult returned 10551584, 1
2022-02-12 01:04:16.341 PST [4912] LOG:  libpqrcv_receive
libpqrcv_PQgetResult PGRES_COMMAND_OK
2022-02-12 01:04:16.341 PST [4912] LOG:  libpqrcv_PQgetResult:
streamConn->asyncStatus: 1 && streamConn->status: 0
2022-02-12 01:04:16.341 PST [4912] LOG:  libpqrcv_PQgetResult loop
before WaitLatchOrSocket
2022-02-12 01:04:16.341 PST [4912] LOG:  WSAEventSelect event->fd: 948,
flags: 21
2022-02-12 01:04:16.341 PST [4912] LOG:  WaitLatchOrSocket before
WaitEventSetWait
2022-02-12 01:04:16.341 PST [4912] LOG:  WaitEventSetWait before
WaitEventSetWaitBlock
2022-02-12 01:04:16.341 PST [4912] LOG:  WaitEventSetWaitBlock before
WaitForMultipleObjects: 3
...
shows that before the doomed WaitForMultipleObjects() call the field
conn->status is 0 (CONNECTION_OK).

[1]: /messages/by-id/2187263.1644616494@sss.pgh.pa.us

Best regards,
Alexander

Attachments:

libpqrcv-diagnostic.patchtext/x-patch; charset=UTF-8; name=libpqrcv-diagnostic.patchDownload
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 0d89db4e6a..0776885306 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -34,6 +34,9 @@
 #include "utils/pg_lsn.h"
 #include "utils/tuplestore.h"
 
+#undef ENABLE_THREAD_SAFETY
+#include "libpq-int.h"
+
 PG_MODULE_MAGIC;
 
 void		_PG_init(void);
@@ -696,10 +699,13 @@ libpqrcv_PQgetResult(PGconn *streamConn)
 	 * Collect data until PQgetResult is ready to get the result without
 	 * blocking.
 	 */
+elog(LOG, "libpqrcv_PQgetResult: streamConn->asyncStatus: %d && streamConn->status: %d", streamConn->asyncStatus, streamConn->status);
+
 	while (PQisBusy(streamConn))
 	{
 		int			rc;
 
+elog(LOG, "libpqrcv_PQgetResult loop before WaitLatchOrSocket");
 		/*
 		 * We don't need to break down the sleep into smaller increments,
 		 * since we'll get interrupted by signals and can handle any
@@ -711,6 +717,7 @@ libpqrcv_PQgetResult(PGconn *streamConn)
 							   PQsocket(streamConn),
 							   0,
 							   WAIT_EVENT_LIBPQWALRECEIVER_RECEIVE);
+elog(LOG, "libpqrcv_PQgetResult loop after WaitLatchOrSocket: %d");
 
 		/* Interrupted? */
 		if (rc & WL_LATCH_SET)
@@ -771,6 +778,7 @@ libpqrcv_receive(WalReceiverConn *conn, char **buffer,
 
 	/* Try to receive a CopyData message */
 	rawlen = PQgetCopyData(conn->streamConn, &conn->recvBuf, 1);
+elog(LOG, "libpqrcv_receive: PQgetCopyData returned %d", rawlen);
 	if (rawlen == 0)
 	{
 		/* Try consuming some data. */
@@ -782,6 +790,7 @@ libpqrcv_receive(WalReceiverConn *conn, char **buffer,
 
 		/* Now that we've consumed some input, try again */
 		rawlen = PQgetCopyData(conn->streamConn, &conn->recvBuf, 1);
+elog(LOG, "libpqrcv_receive: PQgetCopyData 2 returned %d", rawlen);
 		if (rawlen == 0)
 		{
 			/* Tell caller to try again when our socket is ready. */
@@ -791,15 +800,20 @@ libpqrcv_receive(WalReceiverConn *conn, char **buffer,
 	}
 	if (rawlen == -1)			/* end-of-streaming or error */
 	{
+elog(LOG, "libpqrcv_receive: end-of-streaming or error: %d", rawlen);
+
 		PGresult   *res;
 
 		res = libpqrcv_PQgetResult(conn->streamConn);
+elog(LOG, "libpqrcv_receive libpqrcv_PQgetResult returned %d, %d", res, PQresultStatus(res));
 		if (PQresultStatus(res) == PGRES_COMMAND_OK)
 		{
+elog(LOG, "libpqrcv_receive libpqrcv_PQgetResult PGRES_COMMAND_OK");
 			PQclear(res);
 
 			/* Verify that there are no more results. */
 			res = libpqrcv_PQgetResult(conn->streamConn);
+elog(LOG, "libpqrcv_receive libpqrcv_PQgetResult 2 returned %p", res);
 			if (res != NULL)
 			{
 				PQclear(res);
@@ -809,6 +823,7 @@ libpqrcv_receive(WalReceiverConn *conn, char **buffer,
 				 * we'd seen an error, or PGRES_COPY_IN) don't report an error
 				 * here, but let callers deal with it.
 				 */
+elog(LOG, "libpqrcv_receive PQstatus(conn->streamConn) returned %d", PQstatus(conn->streamConn));
 				if (PQstatus(conn->streamConn) == CONNECTION_BAD)
 					return -1;
 
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index b39fce8c23..6bbbf67344 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -425,6 +425,7 @@ WalReceiverMain(void)
 				bool		endofwal = false;
 				pgsocket	wait_fd = PGINVALID_SOCKET;
 				int			rc;
+elog(LOG, "!!!wal receiver loop");
 
 				/*
 				 * Exit walreceiver if we're not in recovery. This should not
@@ -447,6 +448,8 @@ WalReceiverMain(void)
 
 				/* See if we can read data immediately */
 				len = walrcv_receive(wrconn, &buf, &wait_fd);
+elog(LOG, "!!!walrcv_receive returned %d", len);
+
 				if (len != 0)
 				{
 					/*
@@ -514,6 +517,7 @@ WalReceiverMain(void)
 									   wait_fd,
 									   NAPTIME_PER_CYCLE,
 									   WAIT_EVENT_WAL_RECEIVER_MAIN);
+elog(LOG, "!!!WaitLatchOrSocket returned %d", rc);
 				if (rc & WL_LATCH_SET)
 				{
 					ResetLatch(MyLatch);
@@ -583,6 +587,7 @@ WalReceiverMain(void)
 					XLogWalRcvSendHSFeedback(false);
 				}
 			}
+elog(LOG, "!!!wal receiver loop finished");
 
 			/*
 			 * The backend finished streaming. Exit streaming COPY-mode from
diff --git a/src/backend/storage/ipc/latch.c b/src/backend/storage/ipc/latch.c
index 5bb609b368..11afc03072 100644
--- a/src/backend/storage/ipc/latch.c
+++ b/src/backend/storage/ipc/latch.c
@@ -529,12 +529,12 @@ WaitLatchOrSocket(Latch *latch, int wakeEvents, pgsocket sock,
 	if (wakeEvents & WL_SOCKET_MASK)
 	{
 		int			ev;
-
 		ev = wakeEvents & WL_SOCKET_MASK;
 		AddWaitEventToSet(set, ev, sock, NULL, NULL);
 	}
-
+elog(LOG, "WaitLatchOrSocket before WaitEventSetWait");
 	rc = WaitEventSetWait(set, timeout, &event, 1, wait_event_info);
+elog(LOG, "WaitLatchOrSocket after WaitEventSetWait");
 
 	if (rc == 0)
 		ret |= WL_TIMEOUT;
@@ -1280,6 +1280,7 @@ WaitEventAdjustWin32(WaitEventSet *set, WaitEvent *event)
 				elog(ERROR, "failed to create event for socket: error code %d",
 					 WSAGetLastError());
 		}
+elog(LOG, "WSAEventSelect event->fd: %d, flags: %x", event->fd, flags);
 		if (WSAEventSelect(event->fd, *handle, flags) != 0)
 			elog(ERROR, "failed to set up event for socket: error code %d",
 				 WSAGetLastError());
@@ -1392,8 +1393,10 @@ WaitEventSetWait(WaitEventSet *set, long timeout,
 		 * this file. If -1 is returned, a timeout has occurred, if 0 we have
 		 * to retry, everything >= 1 is the number of returned events.
 		 */
+elog(LOG, "WaitEventSetWait before WaitEventSetWaitBlock");
 		rc = WaitEventSetWaitBlock(set, cur_timeout,
 								   occurred_events, nevents);
+elog(LOG, "WaitEventSetWait after WaitEventSetWaitBlock");
 
 		if (set->latch)
 		{
@@ -1888,9 +1891,11 @@ WaitEventSetWaitBlock(WaitEventSet *set, int cur_timeout,
 	 *
 	 * Need to wait for ->nevents + 1, because signal handle is in [0].
 	 */
+elog(LOG, "WaitEventSetWaitBlock before WaitForMultipleObjects: %d", set->nevents);
 	rc = WaitForMultipleObjects(set->nevents + 1, set->handles, FALSE,
 								cur_timeout);
 
+elog(LOG, "WaitEventSetWaitBlock after WaitForMultipleObjects");
 	/* Check return code */
 	if (rc == WAIT_FAILED)
 		elog(ERROR, "WaitForMultipleObjects() failed: error code %lu",
#44Tom Lane
tgl@sss.pgh.pa.us
In reply to: Alexander Lakhin (#43)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

Alexander Lakhin <exclusion@gmail.com> writes:

11.02.2022 05:22, Andres Freund wrote:

Over in another thread I made some wild unsubstantiated guesses that the
windows issues could have been made much more likely by a somewhat odd bit of
code in PQisBusy():
/messages/by-id/1959196.1644544971@sss.pgh.pa.us
Alexander, any chance you'd try if that changes the likelihood of the problem
occurring, without any other fixes / reverts applied?

Unfortunately I haven't seen an improvement for the test in question.

Yeah, that's what I expected, sadly. While I think this PQisBusy behavior
is definitely a bug, it will not lead to an infinite loop, just to write
failures being reported in a less convenient fashion than intended.

I wonder whether it would help to put a PQconsumeInput call *before*
the PQisBusy loop, so that any pre-existing EOF condition will be
detected. If you don't like duplicating code, we could restructure
the loop as

for (;;)
{
int rc;

/* Consume whatever data is available from the socket */
if (PQconsumeInput(streamConn) == 0)
{
/* trouble; return NULL */
return NULL;
}

/* Done? */
if (!PQisBusy(streamConn))
break;

/* Wait for more data */
rc = WaitLatchOrSocket(MyLatch,
WL_EXIT_ON_PM_DEATH | WL_SOCKET_READABLE |
WL_LATCH_SET,
PQsocket(streamConn),
0,
WAIT_EVENT_LIBPQWALRECEIVER_RECEIVE);

/* Interrupted? */
if (rc & WL_LATCH_SET)
{
ResetLatch(MyLatch);
ProcessWalRcvInterrupts();
}
}

/* Now we can collect and return the next PGresult */
return PQgetResult(streamConn);

In combination with the PQisBusy fix, this might actually help ...

regards, tom lane

#45Andres Freund
andres@anarazel.de
In reply to: Tom Lane (#44)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

Hi,

On 2022-02-12 11:47:20 -0500, Tom Lane wrote:

Alexander Lakhin <exclusion@gmail.com> writes:

11.02.2022 05:22, Andres Freund wrote:

Over in another thread I made some wild unsubstantiated guesses that the
windows issues could have been made much more likely by a somewhat odd bit of
code in PQisBusy():
/messages/by-id/1959196.1644544971@sss.pgh.pa.us
Alexander, any chance you'd try if that changes the likelihood of the problem
occurring, without any other fixes / reverts applied?

Unfortunately I haven't seen an improvement for the test in question.

Thanks for testing!

Yeah, that's what I expected, sadly. While I think this PQisBusy behavior
is definitely a bug, it will not lead to an infinite loop, just to write
failures being reported in a less convenient fashion than intended.

FWIW, I didn't think it'd end up looping indefinitely, but that there's a
chance it could end up waiting indefinitely. The WaitLatchOrSocket() doesn't
have a timeout, and if I understand the windows FD_CLOSE stuff correctly,
you're not guaranteed to get an event if you do WaitForMultipleObjects if
FD_CLOSE was already consumed and if there isn't any data to read.

ISTM that it's not a great idea for libpqrcv_receive() to do blocking IO at
all. The caller expects it to not block...

I wonder whether it would help to put a PQconsumeInput call *before*
the PQisBusy loop, so that any pre-existing EOF condition will be
detected. If you don't like duplicating code, we could restructure
the loop as

That does look a bit saner. Even leaving EOF and windows issues aside, it
seems weird to do a WaitLatchOrSocket() without having tried to read more
data.

Greetings,

Andres Freund

#46Noah Misch
noah@leadboat.com
In reply to: Tom Lane (#15)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

On Mon, Jan 10, 2022 at 04:25:27PM -0500, Tom Lane wrote:

Apropos of that, it's worth noting that wait_for_catchup *is*
dependent on up-to-date stats, and here's a recent run where
it sure looks like the timeout cause is AWOL stats collector:

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=sungazer&amp;dt=2022-01-10%2004%3A51%3A34

I wonder if we should refactor wait_for_catchup to probe the
standby directly instead of relying on the upstream's view.

It would be nice. For logical replication tests, do we have a monitoring API
independent of the stats collector? If not and we don't want to add one, a
hacky alternative might be for wait_for_catchup to run a WAL-writing command
every ~20s. That way, if the stats collector misses the datagram about the
standby reaching a certain LSN, the stats collector would have more chances.

#47Thomas Munro
thomas.munro@gmail.com
In reply to: Noah Misch (#46)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

I have a new socket abstraction patch that should address the known
Windows socket/event bugs, but it's a little bigger than I thought it
would be, not quite ready, and now too late to expect people to review
for 15, so I think it should go into the next cycle. I've bounced
https://commitfest.postgresql.org/37/3523/ into the next CF. We'll
need to do something like 75674c7e for master.

#48Tom Lane
tgl@sss.pgh.pa.us
In reply to: Thomas Munro (#47)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

Thomas Munro <thomas.munro@gmail.com> writes:

I have a new socket abstraction patch that should address the known
Windows socket/event bugs, but it's a little bigger than I thought it
would be, not quite ready, and now too late to expect people to review
for 15, so I think it should go into the next cycle. I've bounced
https://commitfest.postgresql.org/37/3523/ into the next CF. We'll
need to do something like 75674c7e for master.

OK. You want me to push 75674c7e to HEAD?

regards, tom lane

#49Thomas Munro
thomas.munro@gmail.com
In reply to: Tom Lane (#48)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

On Tue, Mar 22, 2022 at 4:13 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Thomas Munro <thomas.munro@gmail.com> writes:

I have a new socket abstraction patch that should address the known
Windows socket/event bugs, but it's a little bigger than I thought it
would be, not quite ready, and now too late to expect people to review
for 15, so I think it should go into the next cycle. I've bounced
https://commitfest.postgresql.org/37/3523/ into the next CF. We'll
need to do something like 75674c7e for master.

OK. You want me to push 75674c7e to HEAD?

Thanks, yes, please do.

#50Tom Lane
tgl@sss.pgh.pa.us
In reply to: Thomas Munro (#49)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

Thomas Munro <thomas.munro@gmail.com> writes:

On Tue, Mar 22, 2022 at 4:13 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

OK. You want me to push 75674c7e to HEAD?

Thanks, yes, please do.

Done.

regards, tom lane

#51Thomas Munro
thomas.munro@gmail.com
In reply to: Tom Lane (#50)
6 attachment(s)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

Here is a new attempt to fix this mess. Disclaimer: this based
entirely on reading the manual and vicariously hacking a computer I
don't have via CI.

The two basic ideas are:

* keep per-socket event handles in a hash table
* add our own level-triggered event memory

The socket table entries are reference counted, and exist as long as
the socket is currently in at least one WaitEventSet. When creating a
new entry, extra polling logic re-checks the initial level-triggered
state (an overhead that we had in an ad-hoc way already, and that can
be avoided by more widespread use of long lived WaitEventSet). You
are not allowed to close a socket while it's in a WaitEventSet,
because then a new socket could be allocated with the same number and
chaos would ensue. For example, if we revive the idea of hooking
libpq connections up to long-lived WaitEventSets, we'll probably need
to invent a libpq event callback that says 'I am going to close socket
X!', so you have a chance to remove the socket from any WaitEventSet
*before* it's closed, to maintain that invariant. Other lazier ideas
are possible, but probably become impossible in a hypothetical
multi-threaded future.

With these changes, AFAIK it should be safe to reinstate graceful
socket shutdowns, to fix the field complaints about FATAL error
messages being eaten by a grue and the annoying random CI/BF failures.

Here are some other ideas that I considered but rejected for now:

1. We could throw the WAIT_USE_WIN32 code away, and hack
WAIT_USE_POLL to use WSAPoll() on Windows; we could create a
'self-pipe' using a pair of connected AF_UNIX sockets to implement
latches and fake signals. It seems like a lot of work, and makes
latches a bit worse (instead of "everything is an event!" we have
"everything is a socket!" with a helper thread, and we don't even have
socketpair() on this OS). Blah.

2. We could figure out how to do fancy asynchronous sockets and IOCP.
That's how NT really wants to talk to the world, it doesn't really
want to pretend to be Unix. I expect that is where we'll get to
eventually but it's a much bigger cross-platform R&D job.

3. Maybe there is a kind of partial step towards idea 2 that Andres
mentioned on another thread somewhere: one could use an IOCP, and then
use event callbacks that run on system threads to post IOCP messages
(a bit like we do for our fake waitpid()).

What I have here is the simplest way I could see to patch up what we
already have, with the idea that in the fullness of time we'll
eventually get around to idea 2, once someone is ready to do the
press-ups.

Review/poking-with-a-stick/trying-to-break-it most welcome.

Attachments:

v2-0001-simplehash-Allow-raw-memory-to-be-freed.patchtext/x-patch; charset=US-ASCII; name=v2-0001-simplehash-Allow-raw-memory-to-be-freed.patchDownload
From 4614df55bd0e98b70b942543482e6a9eb767f718 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Wed, 1 Nov 2023 05:53:12 +1300
Subject: [PATCH v2 1/6] simplehash: Allow raw memory to be freed.

Commit 48995040d5e introduced SH_RAW_ALLOCATOR, but assumed that memory
allocated that way could be freed with pfree().  Allow SH_RAW_FREE to be
defined too, for cases where that isn't true.
---
 src/include/lib/simplehash.h | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/src/include/lib/simplehash.h b/src/include/lib/simplehash.h
index b7adc16b80..cd354e2f11 100644
--- a/src/include/lib/simplehash.h
+++ b/src/include/lib/simplehash.h
@@ -42,6 +42,7 @@
  *		declarations reside
  *	  - SH_RAW_ALLOCATOR - if defined, memory contexts are not used; instead,
  *	    use this to allocate bytes. The allocator must zero the returned space.
+ *	  - SH_RAW_FREE - free operation corresponding to SH_RAW_ALLOCATOR
  *	  - SH_USE_NONDEFAULT_ALLOCATOR - if defined no element allocator functions
  *		are defined, so you can supply your own
  *	  The following parameters are only relevant when SH_DEFINE is defined:
@@ -410,7 +411,11 @@ SH_ALLOCATE(SH_TYPE * type, Size size)
 static inline void
 SH_FREE(SH_TYPE * type, void *pointer)
 {
+#ifdef SH_RAW_FREE
+	SH_RAW_FREE(pointer);
+#else
 	pfree(pointer);
+#endif
 }
 
 #endif
@@ -458,7 +463,11 @@ SH_SCOPE void
 SH_DESTROY(SH_TYPE * tb)
 {
 	SH_FREE(tb, tb->data);
+#ifdef SH_RAW_FREE
+	SH_RAW_FREE(tb);
+#else
 	pfree(tb);
+#endif
 }
 
 /* reset the contents of a previously created hash table */
-- 
2.42.0

v2-0002-simplehash-Allow-raw-allocation-to-fail.patchtext/x-patch; charset=US-ASCII; name=v2-0002-simplehash-Allow-raw-allocation-to-fail.patchDownload
From 3ee479b182ac53e4839da1156ef30cd9f2523de7 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Wed, 1 Nov 2023 06:51:38 +1300
Subject: [PATCH v2 2/6] simplehash: Allow raw allocation to fail.

Commit 48995040d5e allowed for raw allocators to be used instead of the
MemoryContext API, but didn't contemplate allocation failure.  Teach the
grow and insert operations to report failure to the caller.
---
 src/include/lib/simplehash.h | 33 ++++++++++++++++++++++++++++-----
 1 file changed, 28 insertions(+), 5 deletions(-)

diff --git a/src/include/lib/simplehash.h b/src/include/lib/simplehash.h
index cd354e2f11..d1034baf67 100644
--- a/src/include/lib/simplehash.h
+++ b/src/include/lib/simplehash.h
@@ -205,7 +205,7 @@ SH_SCOPE void SH_DESTROY(SH_TYPE * tb);
 SH_SCOPE void SH_RESET(SH_TYPE * tb);
 
 /* void <prefix>_grow(<prefix>_hash *tb, uint64 newsize) */
-SH_SCOPE void SH_GROW(SH_TYPE * tb, uint64 newsize);
+SH_SCOPE bool SH_GROW(SH_TYPE * tb, uint64 newsize);
 
 /* <element> *<prefix>_insert(<prefix>_hash *tb, <key> key, bool *found) */
 SH_SCOPE	SH_ELEMENT_TYPE *SH_INSERT(SH_TYPE * tb, SH_KEY_TYPE key, bool *found);
@@ -442,6 +442,8 @@ SH_CREATE(MemoryContext ctx, uint32 nelements, void *private_data)
 
 #ifdef SH_RAW_ALLOCATOR
 	tb = (SH_TYPE *) SH_RAW_ALLOCATOR(sizeof(SH_TYPE));
+	if (!tb)
+		return NULL;
 #else
 	tb = (SH_TYPE *) MemoryContextAllocZero(ctx, sizeof(SH_TYPE));
 	tb->ctx = ctx;
@@ -454,6 +456,17 @@ SH_CREATE(MemoryContext ctx, uint32 nelements, void *private_data)
 	SH_COMPUTE_PARAMETERS(tb, size);
 
 	tb->data = (SH_ELEMENT_TYPE *) SH_ALLOCATE(tb, sizeof(SH_ELEMENT_TYPE) * tb->size);
+#ifdef SH_RAW_ALLOCATOR
+	if (!tb->data)
+	{
+#ifdef SH_RAW_FREE
+		SH_RAW_FREE(tb);
+#else
+		pfree(tb);
+#endif
+		return NULL;
+	}
+#endif
 
 	return tb;
 }
@@ -485,7 +498,7 @@ SH_RESET(SH_TYPE * tb)
  * necessary. But resizing to the exact input size can be advantageous
  * performance-wise, when known at some point.
  */
-SH_SCOPE void
+SH_SCOPE bool
 SH_GROW(SH_TYPE * tb, uint64 newsize)
 {
 	uint64		oldsize = tb->size;
@@ -502,9 +515,13 @@ SH_GROW(SH_TYPE * tb, uint64 newsize)
 	/* compute parameters for new table */
 	SH_COMPUTE_PARAMETERS(tb, newsize);
 
-	tb->data = (SH_ELEMENT_TYPE *) SH_ALLOCATE(tb, sizeof(SH_ELEMENT_TYPE) * tb->size);
+	newdata = (SH_ELEMENT_TYPE *) SH_ALLOCATE(tb, sizeof(SH_ELEMENT_TYPE) * tb->size);
+#ifdef SH_RAW_ALLOCATOR
+	if (!newdata)
+		return false;
+#endif
 
-	newdata = tb->data;
+	tb->data = newdata;
 
 	/*
 	 * Copy entries from the old data to newdata. We theoretically could use
@@ -589,6 +606,8 @@ SH_GROW(SH_TYPE * tb, uint64 newsize)
 	}
 
 	SH_FREE(tb, olddata);
+
+	return true;
 }
 
 /*
@@ -623,7 +642,11 @@ restart:
 		 * When optimizing, it can be very useful to print these out.
 		 */
 		/* SH_STAT(tb); */
-		SH_GROW(tb, tb->size * 2);
+		if (!SH_GROW(tb, tb->size * 2))
+		{
+			*found = false;
+			return NULL;
+		}
 		/* SH_STAT(tb); */
 	}
 
-- 
2.42.0

v2-0003-Redesign-Windows-socket-event-management.patchtext/x-patch; charset=US-ASCII; name=v2-0003-Redesign-Windows-socket-event-management.patchDownload
From 03a5bd46e5cdf1e551a64b16bb8915aead67ad85 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Sun, 19 Mar 2023 16:07:20 +1300
Subject: [PATCH v2 3/6] Redesign Windows socket event management.

Previously, we created a Winsock event handle for each socket in each
WaitEventSet, and then we translated an FD_CLOSE event directly to
WL_SOCKET_READABLE.  Since FD_CLOSE is reported only once when the
remote end shuts down gracefully, we could hang in rare scenarios where
backend code relies on WL_SOCKET_READABLE being level-triggered.

We got away with this in the past when the thing on the other end of the
socket was another PostgreSQL server (ie via postgres_fdw, replication
etc), because the remote server would exit without shutting down or
closing its socket, and that produces a repeating 'abortive' FD_CLOSE.
We'd like to change that as it also eats error messages, producing user
complaints and random CI failures, but that's a sepaarate issue and
we'll need to fix this first.

New design:

* for each socket, we now create just one event handle to be used by
  all WaitEventSet objects that are interested in the socket

* for each socket, we now track a set of sticky events that are reported
  as poll() would until they are cleared by either the send()/recv()
  wrappers, or failing that by an explicit re-check

The lifetime management of event handles and associated state is done
by reference counting.
---
 src/backend/port/win32/socket.c  | 364 +++++++++++++++++++++++++++++++
 src/backend/storage/ipc/latch.c  | 212 ++++++------------
 src/include/port/win32_port.h    |   6 +
 src/include/storage/latch.h      |   3 -
 src/tools/pgindent/typedefs.list |   1 +
 5 files changed, 441 insertions(+), 145 deletions(-)

diff --git a/src/backend/port/win32/socket.c b/src/backend/port/win32/socket.c
index 9c339397d1..a7fa98cb1d 100644
--- a/src/backend/port/win32/socket.c
+++ b/src/backend/port/win32/socket.c
@@ -13,6 +13,8 @@
 
 #include "postgres.h"
 
+#include "common/hashfn.h"
+
 /*
  * Indicate if pgwin32_recv() and pgwin32_send() should operate
  * in non-blocking mode.
@@ -37,6 +39,77 @@ int			pgwin32_noblock = 0;
 #undef recv
 #undef send
 
+/*
+ * An entry in our socket table.
+ */
+typedef struct SocketTableEntry
+{
+	SOCKET		sock;
+	char		status;
+
+	/*
+	 * The reference count for the event handle.  Client code that wants to
+	 * use the event functions must acquire a reference and release it when
+	 * finished.
+	 */
+	int			reference_count;
+
+	/*
+	 * The FD_XXX events that were most recently selected for this socket
+	 * number with WSAEventSelect().
+	 */
+	int			selected_events;
+
+	/*
+	 * The FD_XXX events already reported by Winsock, that we'll continue to
+	 * report as long as they are true.  They are cleared by our send/recv
+	 * wrappers, because those are 're-enabling' functions that will cause
+	 * Winsock to report them again.  The are also cleared by an explicit
+	 * check we perform for the benefit of hypothetical code that might be
+	 * reach Winsock send/recv wrappers without going via our wrappers.
+	 */
+	int			level_triggered_events;
+
+	/*
+	 * Windows kernel event most recently associated with the socket number.
+	 */
+	HANDLE		event_handle;
+} SocketTableEntry;
+
+static inline void *
+malloc0(size_t size)
+{
+	void	   *result;
+
+	result = malloc(size);
+	if (result)
+		memset(result, 0, size);
+
+	return result;
+}
+
+/*
+ * It almost seems feasible to use an array to store our per-socket state,
+ * based on the observation that Windows socket descriptors seem to be small
+ * integers as on Unix, but the manual warns against making that assumption.
+ * So we use a hash table.
+ */
+
+#define SH_PREFIX socket_table
+#define SH_ELEMENT_TYPE SocketTableEntry
+#define SH_RAW_ALLOCATOR malloc0
+#define SH_RAW_FREE free
+#define SH_SCOPE static inline
+#define SH_KEY_TYPE SOCKET
+#define SH_KEY sock
+#define SH_HASH_KEY(tb, key) murmurhash32(key)
+#define SH_EQUAL(tb, a, b) (a) == (b)
+#define SH_DECLARE
+#define SH_DEFINE
+#include "lib/simplehash.h"
+
+static socket_table_hash * socket_table;
+
 /*
  * Blocking socket functions implemented so they listen on both
  * the socket and the signal event, required for signal handling.
@@ -310,6 +383,265 @@ pgwin32_socket(int af, int type, int protocol)
 	return s;
 }
 
+/*
+ * Check if any of FD_READ, FD_WRITE or FD_CLOSE is still true.  Used to
+ * re-check level-triggered events.
+ */
+static int
+pgwin32_socket_poll(SOCKET s, int events)
+{
+	int			revents = 0;
+
+	if (events & (FD_READ | FD_CLOSE))
+	{
+		ssize_t		rc;
+		char		c;
+
+		rc = recv(s, &c, 1, MSG_PEEK);
+		if (rc == 1)
+		{
+			/* At least one byte to read. */
+			if (events & FD_READ)
+				revents |= FD_READ;
+		}
+		else if (rc == 0 || WSAGetLastError() != WSAEWOULDBLOCK)
+		{
+			/* EOF due to graceful shutdown, or error. */
+			if (events & FD_CLOSE)
+				revents |= FD_CLOSE;
+		}
+	}
+
+	if (events & FD_WRITE)
+	{
+		char		c;
+
+		/* If it looks like we could write or get an error, report that. */
+		if (send(s, &c, 0, 0) == 0 || WSAGetLastError() != WSAEWOULDBLOCK)
+			revents |= FD_WRITE;
+	}
+
+	return revents;
+}
+
+/*
+ * Adjust the set of FD_XXX events this socket's event handle should wake up
+ * for.  Returns 0 on success, otherwise -1 and sets errno.
+ */
+int
+pgwin32_socket_select_events(SOCKET s, int selected_events)
+{
+	SocketTableEntry *entry;
+
+	Assert(socket_table);
+	entry = socket_table_lookup(socket_table, s);
+
+	Assert(entry);
+	Assert(entry->reference_count > 0);
+	Assert(entry->event_handle != WSA_INVALID_EVENT);
+
+	/* Do nothing if no change. */
+	if (selected_events == entry->selected_events)
+		return 0;
+
+	/*
+	 * Tell Winsock to link the socket to the event handle, and which events
+	 * we're interested in.
+	 */
+	if (WSAEventSelect(s, entry->event_handle, selected_events) == SOCKET_ERROR)
+	{
+		TranslateSocketError();
+		return -1;
+	}
+
+	entry->selected_events = selected_events;
+
+	/*
+	 * The manual tells us: "Issuing a WSAEventSelect for a socket cancels any
+	 * previous WSAAsyncSelect or WSAEventSelect for the same socket and
+	 * clears the internal network event record."  If that is true, we might
+	 * have wiped an internal flag we're interested in.  Close that race by
+	 * triggering an explicit poll before we sleep, by pretending we have seen
+	 * all of these events.
+	 */
+	if (selected_events & (FD_READ | FD_WRITE))
+		entry->level_triggered_events = selected_events & (FD_READ | FD_WRITE | FD_CLOSE);
+	else
+		entry->level_triggered_events = 0;
+
+	return 0;
+}
+
+/*
+ * Before waiting on the event handle, check if we have pending
+ * level-triggered events that are still true, and if so take measures to
+ * prevent the sleep.
+ */
+void
+pgwin32_socket_prepare_to_wait(SOCKET s)
+{
+	SocketTableEntry *entry;
+
+	Assert(socket_table);
+	entry = socket_table_lookup(socket_table, s);
+
+	Assert(entry);
+	Assert(entry->reference_count > 0);
+	Assert(entry->event_handle != WSA_INVALID_EVENT);
+
+	/*
+	 * If we're not waiting for FD_READ or FD_WRITE, don't try to poll the
+	 * socket.  Server sockets and client sockets that haven't connected yet
+	 * can't be polled by that technique.
+	 */
+	if ((entry->selected_events & (FD_READ | FD_WRITE)) &&
+		entry->level_triggered_events != 0)
+	{
+		/*
+		 * Re-check the level-triggered events we have recorded.  This is
+		 * necessary because someone might access WSASend()/WSARecv() directly
+		 * without going via our wrapper functions, so they might never be
+		 * cleared otherwise.
+		 */
+		entry->level_triggered_events =
+			pgwin32_socket_poll(s,
+								entry->level_triggered_events & entry->selected_events);
+		if (entry->level_triggered_events)
+		{
+			/*
+			 * At least one readiness condition is still true.  Prevent
+			 * sleeping, and let pgwin32_socket_enumerate_events() report
+			 * these level-triggered events.
+			 */
+			WSASetEvent(entry->event_handle);
+		}
+	}
+}
+
+/*
+ * After the Windows event handle has been signaled, this function can be
+ * called to find out which socket events occurred, and atomically reset the
+ * event handle for the next sleep.
+ *
+ * The events returned are also remembered in our level-triggered event mask,
+ * so they'll prevent sleeping and be reported again as long as they remain
+ * true.
+ */
+int
+pgwin32_socket_enumerate_events(SOCKET s)
+{
+	WSANETWORKEVENTS new_events = {0};
+	SocketTableEntry *entry;
+	int			result;
+
+	Assert(socket_table);
+	entry = socket_table_lookup(socket_table, s);
+
+	Assert(entry);
+	Assert(entry->reference_count > 0);
+	Assert(entry->event_handle != WSA_INVALID_EVENT);
+
+	/*
+	 * Atomically consume the internal network event record and reset the
+	 * associated event handle.  This guarantees that we can't miss future
+	 * wakeups.
+	 */
+	if (WSAEnumNetworkEvents(s, entry->event_handle, &new_events) != 0)
+	{
+		TranslateSocketError();
+		return -1;
+	}
+
+	/* Add any events pgwin32_socket_prepare_to_wait() decided to feed us. */
+	result = entry->level_triggered_events | new_events.lNetworkEvents;
+
+	/* Remember certain events for next time around. */
+	if (entry->selected_events & (FD_READ | FD_WRITE))
+		entry->level_triggered_events = result & (FD_READ | FD_WRITE | FD_CLOSE);
+	else
+		entry->level_triggered_events = 0;
+
+	return result;
+}
+
+/*
+ * Acquire a reference-counted Windows event handle for this socket.  This can
+ * be used for waiting for socket events.  Returns NULL and sets errno on
+ * failure.
+ */
+HANDLE
+pgwin32_socket_acquire_event_handle(SOCKET s)
+{
+	SocketTableEntry *entry;
+	bool		found;
+
+	/* First-time initialization. */
+	if (unlikely(socket_table == NULL))
+	{
+		socket_table = socket_table_create(16, NULL);
+		if (socket_table == NULL)
+		{
+			errno = ENOMEM;
+			return NULL;
+		}
+	}
+
+	/* If we already have it, just bump the count. */
+	entry = socket_table_insert(socket_table, s, &found);
+	if (likely(found))
+	{
+		Assert(entry->event_handle != WSA_INVALID_EVENT);
+		entry->reference_count++;
+		return entry->event_handle;
+	}
+
+	/* Did we run out of memory? */
+	if (entry == NULL)
+	{
+		errno = ENOMEM;
+		return NULL;
+	}
+
+	/* Allocate a new event handle. */
+	entry->event_handle = WSACreateEvent();
+	if (entry->event_handle == WSA_INVALID_EVENT)
+	{
+		socket_table_delete_item(socket_table, entry);
+		errno = ENOMEM;
+		return NULL;
+	}
+
+	entry->selected_events = 0;
+	entry->level_triggered_events = 0;
+	entry->reference_count = 1;
+
+	return entry->event_handle;
+}
+
+/*
+ * Release a reference-counted event handle.
+ */
+void
+pgwin32_socket_release_event_handle(SOCKET s)
+{
+	SocketTableEntry *entry;
+
+	Assert(socket_table);
+	entry = socket_table_lookup(socket_table, s);
+
+	Assert(entry);
+	Assert(entry->reference_count > 0);
+	Assert(entry->event_handle != WSA_INVALID_EVENT);
+
+	if (--entry->reference_count == 0)
+	{
+		WSACloseEvent(entry->event_handle);
+		socket_table_delete_item(socket_table, entry);
+
+		/* XXX Free socket_table if it is empty? */
+	}
+}
+
 int
 pgwin32_bind(SOCKET s, struct sockaddr *addr, int addrlen)
 {
@@ -402,6 +734,22 @@ pgwin32_recv(SOCKET s, char *buf, int len, int f)
 		return -1;
 	}
 
+	/*
+	 * WSARecv() is a re-enabling function for Winsock's FD_READ event, so it
+	 * is now safe to clear our level-triggered flag.  This is only an
+	 * optimization for a common case, and not required for correctness.  If
+	 * someone calls WSARecv() directly instead of going through this wrapper,
+	 * pgwin32_socket_prepare_to_wait() will figure that out and clear it
+	 * anyway.
+	 */
+	if (socket_table)
+	{
+		SocketTableEntry *entry = socket_table_lookup(socket_table, s);
+
+		if (entry)
+			entry->level_triggered_events &= ~FD_READ;
+	}
+
 	if (pgwin32_noblock)
 	{
 		/*
@@ -485,6 +833,22 @@ pgwin32_send(SOCKET s, const void *buf, int len, int flags)
 			return -1;
 		}
 
+		/*
+		 * WSASend() is a re-enabling function for Winsock's FD_WRITE event,
+		 * so it is now safe to clear our level-triggered flag.  This is only
+		 * an optimization for a common case, and not required for
+		 * correctness.  If someone calls WSASend() directly instead of going
+		 * through this wrapper, pgwin32_socket_prepare_to_wait() will figure
+		 * that out and clear it anyway.
+		 */
+		if (socket_table)
+		{
+			SocketTableEntry *entry = socket_table_lookup(socket_table, s);
+
+			if (entry)
+				entry->level_triggered_events &= ~FD_WRITE;
+		}
+
 		if (pgwin32_noblock)
 		{
 			/*
diff --git a/src/backend/storage/ipc/latch.c b/src/backend/storage/ipc/latch.c
index 2fd386a4ed..5bf03a3cd9 100644
--- a/src/backend/storage/ipc/latch.c
+++ b/src/backend/storage/ipc/latch.c
@@ -847,20 +847,9 @@ FreeWaitEventSet(WaitEventSet *set)
 		 cur_event < (set->events + set->nevents);
 		 cur_event++)
 	{
-		if (cur_event->events & WL_LATCH_SET)
-		{
-			/* uses the latch's HANDLE */
-		}
-		else if (cur_event->events & WL_POSTMASTER_DEATH)
-		{
-			/* uses PostmasterHandle */
-		}
-		else
-		{
-			/* Clean up the event object we created for the socket */
-			WSAEventSelect(cur_event->fd, NULL, 0);
-			WSACloseEvent(set->handles[cur_event->pos + 1]);
-		}
+		/* Release reference to socket's event handle. */
+		if (cur_event->events & WL_SOCKET_MASK)
+			pgwin32_socket_release_event_handle(cur_event->fd);
 	}
 #endif
 
@@ -955,9 +944,6 @@ AddWaitEventToSet(WaitEventSet *set, uint32 events, pgsocket fd, Latch *latch,
 	event->fd = fd;
 	event->events = events;
 	event->user_data = user_data;
-#ifdef WIN32
-	event->reset = false;
-#endif
 
 	if (events == WL_LATCH_SET)
 	{
@@ -976,10 +962,21 @@ AddWaitEventToSet(WaitEventSet *set, uint32 events, pgsocket fd, Latch *latch,
 	}
 	else if (events == WL_POSTMASTER_DEATH)
 	{
-#ifndef WIN32
+#if defined(WAIT_USE_WIN32)
+		set->handles[event->pos + 1] = PostmasterHandle;
+		event->fd = PGINVALID_SOCKET;
+#else
 		event->fd = postmaster_alive_fds[POSTMASTER_FD_WATCH];
 #endif
 	}
+	else if (events & WL_SOCKET_MASK)
+	{
+#if defined(WAIT_USE_WIN32)
+		set->handles[event->pos + 1] = pgwin32_socket_acquire_event_handle(fd);
+		if (!set->handles[event->pos + 1])
+			elog(ERROR, "could not acquire socket event handle: %m");
+#endif
+	}
 
 	/* perform wait primitive specific initialization, if needed */
 #if defined(WAIT_USE_EPOLL)
@@ -1322,45 +1319,52 @@ WaitEventAdjustKqueue(WaitEventSet *set, WaitEvent *event, int old_events)
 #endif
 
 #if defined(WAIT_USE_WIN32)
+static int
+ToWinsockEvents(int pg_events)
+{
+	int			winsock_events = 0;
+
+	if (pg_events & WL_SOCKET_READABLE)
+		winsock_events |= FD_CLOSE | FD_READ;
+	if (pg_events & WL_SOCKET_WRITEABLE)
+		winsock_events |= FD_CLOSE | FD_WRITE;
+	if (pg_events & WL_SOCKET_CONNECTED)
+		winsock_events |= FD_CLOSE | FD_CONNECT;
+	if (pg_events & WL_SOCKET_ACCEPT)
+		winsock_events |= FD_CLOSE | FD_ACCEPT;
+
+	return winsock_events;
+}
+
+static int
+FromWinsockEvents(int winsock_events)
+{
+	int			pg_events = 0;
+
+	if (winsock_events & (FD_CLOSE | FD_READ))
+		pg_events |= WL_SOCKET_READABLE;
+	if (winsock_events & (FD_CLOSE | FD_WRITE))
+		pg_events |= WL_SOCKET_WRITEABLE;
+	if (winsock_events & (FD_CLOSE | FD_CONNECT))
+		pg_events |= WL_SOCKET_CONNECTED;
+	if (winsock_events & (FD_CLOSE | FD_ACCEPT))
+		pg_events |= WL_SOCKET_ACCEPT;
+
+	return pg_events;
+}
+
 static void
 WaitEventAdjustWin32(WaitEventSet *set, WaitEvent *event)
 {
-	HANDLE	   *handle = &set->handles[event->pos + 1];
-
-	if (event->events == WL_LATCH_SET)
+	if (event->events & WL_LATCH_SET)
 	{
-		Assert(set->latch != NULL);
-		*handle = set->latch->event;
+		set->handles[event->pos + 1] = set->latch->event;
 	}
-	else if (event->events == WL_POSTMASTER_DEATH)
-	{
-		*handle = PostmasterHandle;
-	}
-	else
+	else if (event->events & WL_SOCKET_MASK)
 	{
-		int			flags = FD_CLOSE;	/* always check for errors/EOF */
-
-		if (event->events & WL_SOCKET_READABLE)
-			flags |= FD_READ;
-		if (event->events & WL_SOCKET_WRITEABLE)
-			flags |= FD_WRITE;
-		if (event->events & WL_SOCKET_CONNECTED)
-			flags |= FD_CONNECT;
-		if (event->events & WL_SOCKET_ACCEPT)
-			flags |= FD_ACCEPT;
-
-		if (*handle == WSA_INVALID_EVENT)
-		{
-			*handle = WSACreateEvent();
-			if (*handle == WSA_INVALID_EVENT)
-				elog(ERROR, "failed to create event for socket: error code %d",
-					 WSAGetLastError());
-		}
-		if (WSAEventSelect(event->fd, *handle, flags) != 0)
-			elog(ERROR, "failed to set up event for socket: error code %d",
-				 WSAGetLastError());
-
-		Assert(event->fd != PGINVALID_SOCKET);
+		if (pgwin32_socket_select_events(event->fd,
+										 ToWinsockEvents(event->events)) < 0)
+			elog(ERROR, "failed to set up event for socket: %m");
 	}
 }
 #endif
@@ -1945,48 +1949,16 @@ WaitEventSetWaitBlock(WaitEventSet *set, int cur_timeout,
 	DWORD		rc;
 	WaitEvent  *cur_event;
 
-	/* Reset any wait events that need it */
+	/*
+	 * Allow level-triggered events to be signaled, causing
+	 * WaitForMultipleObjects() to return immediately.
+	 */
 	for (cur_event = set->events;
 		 cur_event < (set->events + set->nevents);
 		 cur_event++)
 	{
-		if (cur_event->reset)
-		{
-			WaitEventAdjustWin32(set, cur_event);
-			cur_event->reset = false;
-		}
-
-		/*
-		 * Windows does not guarantee to log an FD_WRITE network event
-		 * indicating that more data can be sent unless the previous send()
-		 * failed with WSAEWOULDBLOCK.  While our caller might well have made
-		 * such a call, we cannot assume that here.  Therefore, if waiting for
-		 * write-ready, force the issue by doing a dummy send().  If the dummy
-		 * send() succeeds, assume that the socket is in fact write-ready, and
-		 * return immediately.  Also, if it fails with something other than
-		 * WSAEWOULDBLOCK, return a write-ready indication to let our caller
-		 * deal with the error condition.
-		 */
-		if (cur_event->events & WL_SOCKET_WRITEABLE)
-		{
-			char		c;
-			WSABUF		buf;
-			DWORD		sent;
-			int			r;
-
-			buf.buf = &c;
-			buf.len = 0;
-
-			r = WSASend(cur_event->fd, &buf, 1, &sent, 0, NULL, NULL);
-			if (r == 0 || WSAGetLastError() != WSAEWOULDBLOCK)
-			{
-				occurred_events->pos = cur_event->pos;
-				occurred_events->user_data = cur_event->user_data;
-				occurred_events->events = WL_SOCKET_WRITEABLE;
-				occurred_events->fd = cur_event->fd;
-				return 1;
-			}
-		}
+		if (cur_event->events & WL_SOCKET_MASK)
+			pgwin32_socket_prepare_to_wait(cur_event->fd);
 	}
 
 	/*
@@ -2067,64 +2039,20 @@ WaitEventSetWaitBlock(WaitEventSet *set, int cur_timeout,
 		}
 		else if (cur_event->events & WL_SOCKET_MASK)
 		{
-			WSANETWORKEVENTS resEvents;
-			HANDLE		handle = set->handles[cur_event->pos + 1];
+			int			winsock_events;
+			int			pg_events;
 
 			Assert(cur_event->fd);
 
-			occurred_events->fd = cur_event->fd;
+			winsock_events = pgwin32_socket_enumerate_events(cur_event->fd);
+			if (winsock_events < 0)
+				elog(ERROR, "could not enumerate socket events: %m");
 
-			ZeroMemory(&resEvents, sizeof(resEvents));
-			if (WSAEnumNetworkEvents(cur_event->fd, handle, &resEvents) != 0)
-				elog(ERROR, "failed to enumerate network events: error code %d",
-					 WSAGetLastError());
-			if ((cur_event->events & WL_SOCKET_READABLE) &&
-				(resEvents.lNetworkEvents & FD_READ))
-			{
-				/* data available in socket */
-				occurred_events->events |= WL_SOCKET_READABLE;
-
-				/*------
-				 * WaitForMultipleObjects doesn't guarantee that a read event
-				 * will be returned if the latch is set at the same time.  Even
-				 * if it did, the caller might drop that event expecting it to
-				 * reoccur on next call.  So, we must force the event to be
-				 * reset if this WaitEventSet is used again in order to avoid
-				 * an indefinite hang.
-				 *
-				 * Refer
-				 * https://msdn.microsoft.com/en-us/library/windows/desktop/ms741576(v=vs.85).aspx
-				 * for the behavior of socket events.
-				 *------
-				 */
-				cur_event->reset = true;
-			}
-			if ((cur_event->events & WL_SOCKET_WRITEABLE) &&
-				(resEvents.lNetworkEvents & FD_WRITE))
-			{
-				/* writeable */
-				occurred_events->events |= WL_SOCKET_WRITEABLE;
-			}
-			if ((cur_event->events & WL_SOCKET_CONNECTED) &&
-				(resEvents.lNetworkEvents & FD_CONNECT))
-			{
-				/* connected */
-				occurred_events->events |= WL_SOCKET_CONNECTED;
-			}
-			if ((cur_event->events & WL_SOCKET_ACCEPT) &&
-				(resEvents.lNetworkEvents & FD_ACCEPT))
-			{
-				/* incoming connection could be accepted */
-				occurred_events->events |= WL_SOCKET_ACCEPT;
-			}
-			if (resEvents.lNetworkEvents & FD_CLOSE)
-			{
-				/* EOF/error, so signal all caller-requested socket flags */
-				occurred_events->events |= (cur_event->events & WL_SOCKET_MASK);
-			}
-
-			if (occurred_events->events != 0)
+			pg_events = FromWinsockEvents(winsock_events) & cur_event->events;
+			if (pg_events)
 			{
+				occurred_events->fd = cur_event->fd;
+				occurred_events->events = pg_events;
 				occurred_events++;
 				returned_events++;
 			}
diff --git a/src/include/port/win32_port.h b/src/include/port/win32_port.h
index 27a11c7868..a0ed6aaeaa 100644
--- a/src/include/port/win32_port.h
+++ b/src/include/port/win32_port.h
@@ -506,6 +506,12 @@ extern int	pgwin32_recv(SOCKET s, char *buf, int len, int flags);
 extern int	pgwin32_send(SOCKET s, const void *buf, int len, int flags);
 extern int	pgwin32_waitforsinglesocket(SOCKET s, int what, int timeout);
 
+extern HANDLE pgwin32_socket_acquire_event_handle(SOCKET s);
+extern void pgwin32_socket_release_event_handle(SOCKET s);
+extern int	pgwin32_socket_select_events(SOCKET s, int events);
+extern void pgwin32_socket_prepare_to_wait(SOCKET s);
+extern int	pgwin32_socket_enumerate_events(SOCKET s);
+
 extern PGDLLIMPORT int pgwin32_noblock;
 
 #endif							/* FRONTEND */
diff --git a/src/include/storage/latch.h b/src/include/storage/latch.h
index 99cc47874a..cbcc5ef23f 100644
--- a/src/include/storage/latch.h
+++ b/src/include/storage/latch.h
@@ -153,9 +153,6 @@ typedef struct WaitEvent
 	uint32		events;			/* triggered events */
 	pgsocket	fd;				/* socket fd associated with event */
 	void	   *user_data;		/* pointer provided in AddWaitEventToSet */
-#ifdef WIN32
-	bool		reset;			/* Is reset of the event required? */
-#endif
 } WaitEvent;
 
 /* forward declaration to avoid exposing latch.c implementation details */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index bf50a32119..15dd7fa2b8 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2585,6 +2585,7 @@ Snapshot
 SnapshotData
 SnapshotType
 SockAddr
+SocketTableEntry
 Sort
 SortBy
 SortByDir
-- 
2.42.0

v2-0004-Remove-pgwin32_select.patchtext/x-patch; charset=US-ASCII; name=v2-0004-Remove-pgwin32_select.patchDownload
From c561d1f7091368c5e09735ab099eb4a656587dfb Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Fri, 10 Nov 2023 08:41:28 +1300
Subject: [PATCH v2 4/6] Remove pgwin32_select().

pgwin32_select(), used to replace select() in backend code on Windows,
would need to be updated to use the new per-socket event handles.  Since
the last remaining user of select() in the backend is scheduled for
replacement with the WaitEventSet API, it seems better to demolish it
instead.

Any extension code that is relying on select() with fake signals will
still compile, but will no longer respond to signals.  Hypothetical code
like that is probably buggy anyway, because backend code should also be
handling interrupts, and should switch to the various WaitEventSet APIs.
---
 src/backend/port/win32/socket.c | 200 --------------------------------
 src/include/port/win32_port.h   |   2 -
 2 files changed, 202 deletions(-)

diff --git a/src/backend/port/win32/socket.c b/src/backend/port/win32/socket.c
index a7fa98cb1d..d0cf08392f 100644
--- a/src/backend/port/win32/socket.c
+++ b/src/backend/port/win32/socket.c
@@ -867,203 +867,3 @@ pgwin32_send(SOCKET s, const void *buf, int len, int flags)
 
 	return -1;
 }
-
-
-/*
- * Wait for activity on one or more sockets.
- * While waiting, allow signals to run
- *
- * NOTE! Currently does not implement exceptfds check,
- * since it is not used in postgresql!
- */
-int
-pgwin32_select(int nfds, fd_set *readfds, fd_set *writefds, fd_set *exceptfds, const struct timeval *timeout)
-{
-	WSAEVENT	events[FD_SETSIZE * 2]; /* worst case is readfds totally
-										 * different from writefds, so
-										 * 2*FD_SETSIZE sockets */
-	SOCKET		sockets[FD_SETSIZE * 2];
-	int			numevents = 0;
-	int			i;
-	int			r;
-	DWORD		timeoutval = WSA_INFINITE;
-	FD_SET		outreadfds;
-	FD_SET		outwritefds;
-	int			nummatches = 0;
-
-	Assert(exceptfds == NULL);
-
-	if (pgwin32_poll_signals())
-		return -1;
-
-	FD_ZERO(&outreadfds);
-	FD_ZERO(&outwritefds);
-
-	/*
-	 * Windows does not guarantee to log an FD_WRITE network event indicating
-	 * that more data can be sent unless the previous send() failed with
-	 * WSAEWOULDBLOCK.  While our caller might well have made such a call, we
-	 * cannot assume that here.  Therefore, if waiting for write-ready, force
-	 * the issue by doing a dummy send().  If the dummy send() succeeds,
-	 * assume that the socket is in fact write-ready, and return immediately.
-	 * Also, if it fails with something other than WSAEWOULDBLOCK, return a
-	 * write-ready indication to let our caller deal with the error condition.
-	 */
-	if (writefds != NULL)
-	{
-		for (i = 0; i < writefds->fd_count; i++)
-		{
-			char		c;
-			WSABUF		buf;
-			DWORD		sent;
-
-			buf.buf = &c;
-			buf.len = 0;
-
-			r = WSASend(writefds->fd_array[i], &buf, 1, &sent, 0, NULL, NULL);
-			if (r == 0 || WSAGetLastError() != WSAEWOULDBLOCK)
-				FD_SET(writefds->fd_array[i], &outwritefds);
-		}
-
-		/* If we found any write-ready sockets, just return them immediately */
-		if (outwritefds.fd_count > 0)
-		{
-			memcpy(writefds, &outwritefds, sizeof(fd_set));
-			if (readfds)
-				FD_ZERO(readfds);
-			return outwritefds.fd_count;
-		}
-	}
-
-
-	/* Now set up for an actual select */
-
-	if (timeout != NULL)
-	{
-		/* timeoutval is in milliseconds */
-		timeoutval = timeout->tv_sec * 1000 + timeout->tv_usec / 1000;
-	}
-
-	if (readfds != NULL)
-	{
-		for (i = 0; i < readfds->fd_count; i++)
-		{
-			events[numevents] = WSACreateEvent();
-			sockets[numevents] = readfds->fd_array[i];
-			numevents++;
-		}
-	}
-	if (writefds != NULL)
-	{
-		for (i = 0; i < writefds->fd_count; i++)
-		{
-			if (!readfds ||
-				!FD_ISSET(writefds->fd_array[i], readfds))
-			{
-				/* If the socket is not in the read list */
-				events[numevents] = WSACreateEvent();
-				sockets[numevents] = writefds->fd_array[i];
-				numevents++;
-			}
-		}
-	}
-
-	for (i = 0; i < numevents; i++)
-	{
-		int			flags = 0;
-
-		if (readfds && FD_ISSET(sockets[i], readfds))
-			flags |= FD_READ | FD_ACCEPT | FD_CLOSE;
-
-		if (writefds && FD_ISSET(sockets[i], writefds))
-			flags |= FD_WRITE | FD_CLOSE;
-
-		if (WSAEventSelect(sockets[i], events[i], flags) != 0)
-		{
-			TranslateSocketError();
-			/* release already-assigned event objects */
-			while (--i >= 0)
-				WSAEventSelect(sockets[i], NULL, 0);
-			for (i = 0; i < numevents; i++)
-				WSACloseEvent(events[i]);
-			return -1;
-		}
-	}
-
-	events[numevents] = pgwin32_signal_event;
-	r = WaitForMultipleObjectsEx(numevents + 1, events, FALSE, timeoutval, TRUE);
-	if (r != WAIT_TIMEOUT && r != WAIT_IO_COMPLETION && r != (WAIT_OBJECT_0 + numevents))
-	{
-		/*
-		 * We scan all events, even those not signaled, in case more than one
-		 * event has been tagged but Wait.. can only return one.
-		 */
-		WSANETWORKEVENTS resEvents;
-
-		for (i = 0; i < numevents; i++)
-		{
-			ZeroMemory(&resEvents, sizeof(resEvents));
-			if (WSAEnumNetworkEvents(sockets[i], events[i], &resEvents) != 0)
-				elog(ERROR, "failed to enumerate network events: error code %d",
-					 WSAGetLastError());
-			/* Read activity? */
-			if (readfds && FD_ISSET(sockets[i], readfds))
-			{
-				if ((resEvents.lNetworkEvents & FD_READ) ||
-					(resEvents.lNetworkEvents & FD_ACCEPT) ||
-					(resEvents.lNetworkEvents & FD_CLOSE))
-				{
-					FD_SET(sockets[i], &outreadfds);
-
-					nummatches++;
-				}
-			}
-			/* Write activity? */
-			if (writefds && FD_ISSET(sockets[i], writefds))
-			{
-				if ((resEvents.lNetworkEvents & FD_WRITE) ||
-					(resEvents.lNetworkEvents & FD_CLOSE))
-				{
-					FD_SET(sockets[i], &outwritefds);
-
-					nummatches++;
-				}
-			}
-		}
-	}
-
-	/* Clean up all the event objects */
-	for (i = 0; i < numevents; i++)
-	{
-		WSAEventSelect(sockets[i], NULL, 0);
-		WSACloseEvent(events[i]);
-	}
-
-	if (r == WSA_WAIT_TIMEOUT)
-	{
-		if (readfds)
-			FD_ZERO(readfds);
-		if (writefds)
-			FD_ZERO(writefds);
-		return 0;
-	}
-
-	/* Signal-like events. */
-	if (r == WAIT_OBJECT_0 + numevents || r == WAIT_IO_COMPLETION)
-	{
-		pgwin32_dispatch_queued_signals();
-		errno = EINTR;
-		if (readfds)
-			FD_ZERO(readfds);
-		if (writefds)
-			FD_ZERO(writefds);
-		return -1;
-	}
-
-	/* Overwrite socket sets with our resulting values */
-	if (readfds)
-		memcpy(readfds, &outreadfds, sizeof(fd_set));
-	if (writefds)
-		memcpy(writefds, &outwritefds, sizeof(fd_set));
-	return nummatches;
-}
diff --git a/src/include/port/win32_port.h b/src/include/port/win32_port.h
index a0ed6aaeaa..1b605d9403 100644
--- a/src/include/port/win32_port.h
+++ b/src/include/port/win32_port.h
@@ -492,7 +492,6 @@ extern int	pgkill(int pid, int sig);
 #define listen(s, backlog) pgwin32_listen(s, backlog)
 #define accept(s, addr, addrlen) pgwin32_accept(s, addr, addrlen)
 #define connect(s, name, namelen) pgwin32_connect(s, name, namelen)
-#define select(n, r, w, e, timeout) pgwin32_select(n, r, w, e, timeout)
 #define recv(s, buf, len, flags) pgwin32_recv(s, buf, len, flags)
 #define send(s, buf, len, flags) pgwin32_send(s, buf, len, flags)
 
@@ -501,7 +500,6 @@ extern int	pgwin32_bind(SOCKET s, struct sockaddr *addr, int addrlen);
 extern int	pgwin32_listen(SOCKET s, int backlog);
 extern SOCKET pgwin32_accept(SOCKET s, struct sockaddr *addr, int *addrlen);
 extern int	pgwin32_connect(SOCKET s, const struct sockaddr *name, int namelen);
-extern int	pgwin32_select(int nfds, fd_set *readfds, fd_set *writefds, fd_set *exceptfds, const struct timeval *timeout);
 extern int	pgwin32_recv(SOCKET s, char *buf, int len, int flags);
 extern int	pgwin32_send(SOCKET s, const void *buf, int len, int flags);
 extern int	pgwin32_waitforsinglesocket(SOCKET s, int what, int timeout);
-- 
2.42.0

v2-0005-Refactor-pgwin32_waitforsinglesocket-to-share-eve.patchtext/x-patch; charset=US-ASCII; name=v2-0005-Refactor-pgwin32_waitforsinglesocket-to-share-eve.patchDownload
From 27144fba3c17e3d15e973e250f3ce3151079aea0 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Fri, 10 Nov 2023 08:59:45 +1300
Subject: [PATCH v2 5/6] Refactor pgwin32_waitforsinglesocket() to share
 events.

This function is hardly used, since sockets in the backend are almost
always in non-blocking mode.  Ideally they would *always* be
non-blocking, and if we ever get to that situation by project policy we
could just delete this and related code, but in the meantime, we have to
adjust it to use the new per-socket event handle or it could lose
network events.

While here, delete the code paths for UDP which are probably dead code
since we retired the UDP-powered stats collector.  There is another user
of UDP in auth.c, but it's using sendto() and thus not reaching this
code.  Any other user of UDP that I might have missed probably isn't
generating 'high load' like the old stats system that presumably
motivated that sleeping logic.
---
 src/backend/port/win32/socket.c | 103 ++++++--------------------------
 1 file changed, 17 insertions(+), 86 deletions(-)

diff --git a/src/backend/port/win32/socket.c b/src/backend/port/win32/socket.c
index d0cf08392f..797a1f503e 100644
--- a/src/backend/port/win32/socket.c
+++ b/src/backend/port/win32/socket.c
@@ -238,120 +238,51 @@ pgwin32_poll_signals(void)
 	return 0;
 }
 
-static int
-isDataGram(SOCKET s)
-{
-	int			type;
-	int			typelen = sizeof(type);
-
-	if (getsockopt(s, SOL_SOCKET, SO_TYPE, (char *) &type, &typelen))
-		return 1;
-
-	return (type == SOCK_DGRAM) ? 1 : 0;
-}
-
 int
 pgwin32_waitforsinglesocket(SOCKET s, int what, int timeout)
 {
-	static HANDLE waitevent = INVALID_HANDLE_VALUE;
-	static SOCKET current_socket = INVALID_SOCKET;
-	static int	isUDP = 0;
 	HANDLE		events[2];
 	int			r;
 
-	/* Create an event object just once and use it on all future calls */
-	if (waitevent == INVALID_HANDLE_VALUE)
-	{
-		waitevent = CreateEvent(NULL, TRUE, FALSE, NULL);
-
-		if (waitevent == INVALID_HANDLE_VALUE)
-			ereport(ERROR,
-					(errmsg_internal("could not create socket waiting event: error code %lu", GetLastError())));
-	}
-	else if (!ResetEvent(waitevent))
-		ereport(ERROR,
-				(errmsg_internal("could not reset socket waiting event: error code %lu", GetLastError())));
-
-	/*
-	 * Track whether socket is UDP or not.  (NB: most likely, this is both
-	 * useless and wrong; there is no reason to think that the behavior of
-	 * WSAEventSelect is different for TCP and UDP.)
-	 */
-	if (current_socket != s)
-		isUDP = isDataGram(s);
-	current_socket = s;
+	events[0] = pgwin32_signal_event;
+	events[1] = pgwin32_socket_acquire_event_handle(s);
 
-	/*
-	 * Attach event to socket.  NOTE: we must detach it again before
-	 * returning, since other bits of code may try to attach other events to
-	 * the socket.
-	 */
-	if (WSAEventSelect(s, waitevent, what) != 0)
+	if (events[1] == NULL)
 	{
-		TranslateSocketError();
+		/* errno is set */
 		return 0;
 	}
 
-	events[0] = pgwin32_signal_event;
-	events[1] = waitevent;
-
-	/*
-	 * Just a workaround of unknown locking problem with writing in UDP socket
-	 * under high load: Client's pgsql backend sleeps infinitely in
-	 * WaitForMultipleObjectsEx, pgstat process sleeps in pgwin32_select().
-	 * So, we will wait with small timeout(0.1 sec) and if socket is still
-	 * blocked, try WSASend (see comments in pgwin32_select) and wait again.
-	 */
-	if ((what & FD_WRITE) && isUDP)
+	if (pgwin32_socket_select_events(s, what) < 0)
 	{
-		for (;;)
-		{
-			r = WaitForMultipleObjectsEx(2, events, FALSE, 100, TRUE);
-
-			if (r == WAIT_TIMEOUT)
-			{
-				char		c;
-				WSABUF		buf;
-				DWORD		sent;
-
-				buf.buf = &c;
-				buf.len = 0;
-
-				r = WSASend(s, &buf, 1, &sent, 0, NULL, NULL);
-				if (r == 0)		/* Completed - means things are fine! */
-				{
-					WSAEventSelect(s, NULL, 0);
-					return 1;
-				}
-				else if (WSAGetLastError() != WSAEWOULDBLOCK)
-				{
-					TranslateSocketError();
-					WSAEventSelect(s, NULL, 0);
-					return 0;
-				}
-			}
-			else
-				break;
-		}
+		pgwin32_socket_release_event_handle(s);
+		return 0;
 	}
-	else
-		r = WaitForMultipleObjectsEx(2, events, FALSE, timeout, TRUE);
 
-	WSAEventSelect(s, NULL, 0);
+	pgwin32_socket_prepare_to_wait(s);
+
+	r = WaitForMultipleObjectsEx(2, events, FALSE, timeout, TRUE);
 
 	if (r == WAIT_OBJECT_0 || r == WAIT_IO_COMPLETION)
 	{
+		pgwin32_socket_release_event_handle(s);
 		pgwin32_dispatch_queued_signals();
 		errno = EINTR;
 		return 0;
 	}
 	if (r == WAIT_OBJECT_0 + 1)
+	{
+		pgwin32_socket_enumerate_events(s);
+		pgwin32_socket_release_event_handle(s);
 		return 1;
+	}
 	if (r == WAIT_TIMEOUT)
 	{
+		pgwin32_socket_release_event_handle(s);
 		errno = EWOULDBLOCK;
 		return 0;
 	}
+	pgwin32_socket_release_event_handle(s);
 	ereport(ERROR,
 			(errmsg_internal("unrecognized return value from WaitForMultipleObjects: %d (error code %lu)", r, GetLastError())));
 	return 0;
-- 
2.42.0

v2-0006-Reinstate-graceful-shutdown-changes-for-Windows.patchtext/x-patch; charset=US-ASCII; name=v2-0006-Reinstate-graceful-shutdown-changes-for-Windows.patchDownload
From 86d3a318af19d8fa476e36aae1c6b754912d7c4c Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Fri, 10 Nov 2023 10:24:32 +1300
Subject: [PATCH v2 6/6] Reinstate "graceful shutdown" changes for Windows.

This reverts commit 29992a6a509b256efc4ac560a1586b51a64b2637.

See the commit messages for 6051857fc and ed52c3707.
---
 src/backend/libpq/pqcomm.c | 29 ++++++++++++++++++++++-------
 1 file changed, 22 insertions(+), 7 deletions(-)

diff --git a/src/backend/libpq/pqcomm.c b/src/backend/libpq/pqcomm.c
index 522584e597..0ca93fefc8 100644
--- a/src/backend/libpq/pqcomm.c
+++ b/src/backend/libpq/pqcomm.c
@@ -280,15 +280,30 @@ socket_close(int code, Datum arg)
 		secure_close(MyProcPort);
 
 		/*
-		 * Formerly we did an explicit close() here, but it seems better to
-		 * leave the socket open until the process dies.  This allows clients
-		 * to perform a "synchronous close" if they care --- wait till the
-		 * transport layer reports connection closure, and you can be sure the
-		 * backend has exited.
+		 * On most platforms, we leave the socket open until the process dies.
+		 * This allows clients to perform a "synchronous close" if they care
+		 * --- wait till the transport layer reports connection closure, and
+		 * you can be sure the backend has exited.  Saves a kernel call, too.
 		 *
-		 * We do set sock to PGINVALID_SOCKET to prevent any further I/O,
-		 * though.
+		 * However, that does not work on Windows: if the kernel closes the
+		 * socket it will invoke an "abortive shutdown" that discards any data
+		 * not yet sent to the client.  (This is a flat-out violation of the
+		 * TCP RFCs, but count on Microsoft not to care about that.)  To get
+		 * the spec-compliant "graceful shutdown" behavior, we must invoke
+		 * closesocket() explicitly.  When using OpenSSL, it seems that clean
+		 * shutdown also requires an explicit shutdown() call.
+		 *
+		 * This code runs late enough during process shutdown that we should
+		 * have finished all externally-visible shutdown activities, so that
+		 * in principle it's good enough to act as a synchronous close on
+		 * Windows too.  But it's a lot more fragile than the other way.
 		 */
+#ifdef WIN32
+		shutdown(MyProcPort->sock, SD_SEND);
+		closesocket(MyProcPort->sock);
+#endif
+
+		/* In any case, set sock to PGINVALID_SOCKET to prevent further I/O */
 		MyProcPort->sock = PGINVALID_SOCKET;
 	}
 }
-- 
2.42.0

#52Alexander Lakhin
exclusion@gmail.com
In reply to: Thomas Munro (#51)
5 attachment(s)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

Hello Thomas,

10.11.2023 06:31, Thomas Munro wrote:

Here is a new attempt to fix this mess. Disclaimer: this based
entirely on reading the manual and vicariously hacking a computer I
don't have via CI.

As it also might (and I would like it to) be the final attempt, I decided
to gather information and all the cases that we had on this topic.
At least, for the last 5 years we've seen:

[1]: /messages/by-id/87k1iy44fd.fsf@news-spur.riddles.org.uk
idle-in-transaction timeout
    test 099_case_15598 made (in attachment)
no commit

[2]: /messages/by-id/CAEepm=2n6Nv+5tFfe8YnkUm1fXgvxR0Mm1FoD+QKG-vLNGLyKg@mail.gmail.com
    references [1]/messages/by-id/87k1iy44fd.fsf@news-spur.riddles.org.uk
    test 099_rare_ssl_failures made (in attachment)
commit 2019-03-19 1f39a1c06: Restructure libpq's hqandling of send failures.

[3]: /messages/by-id/E1iaD8h-0004us-K9@gemulon.postgresql.org
    test 051_dropdb_force was proposed (included in the attachment)
commit 2019-11-30 98a9b37ba: Revert commits 290acac92b and 8a7e9e9dad.

[4]: /messages/by-id/CALDaNm2tEvr_Kum7SyvFn0=6H3P0P-Zkhnd=dkkX+Q=wKutZ=A@mail.gmail.com
    references [1]/messages/by-id/87k1iy44fd.fsf@news-spur.riddles.org.uk, [2]/messages/by-id/CAEepm=2n6Nv+5tFfe8YnkUm1fXgvxR0Mm1FoD+QKG-vLNGLyKg@mail.gmail.com, [3]/messages/by-id/E1iaD8h-0004us-K9@gemulon.postgresql.org; a documentation change proposed
no commit

[5]: /messages/by-id/20200603201242.ofvm4jztpqytwfye@alap3.anarazel.de
    test 099_pgbench_with_server_off made (in attachment)
commit 2020-06-07 7247e243a: Try to read data from the socket in pqSendSome's write_failed paths. (a fix for 1f39a1c06)

[6]: /messages/by-id/16678-253e48d34dc0c376@postgresql.org
no commit

[7]: /messages/by-id/90b34057-4176-7bb0-0dbb-9822a5f6425b@greiz-reinsdorf.de
   references [6]/messages/by-id/16678-253e48d34dc0c376@postgresql.org
commit: 2021-12-02 6051857fc: On Windows, close the client socket explicitly during backend shutdown.

[8]: /messages/by-id/af5e0bf3-6a61-bb97-6cba-061ddf22ff6b@dunslane.net
commit: 2021-12-07 ed52c3707: On Windows, also call shutdown() while closing the client socket.

[9]: /messages/by-id/CA+hUKG+OeoETZQ=Qw5Ub5h3tmwQhBmDA=nuNO3KG=zWfUypFAw@mail.gmail.com
   additional test 099_postgres_fdw_disconnect made (in attachment)
commit 2022-01-26 75674c7ec: Revert "graceful shutdown" changes for Windows, in back branches only. (REL_14_STABLE)
commit 2022-03-22 29992a6a5: Revert "graceful shutdown" changes for Windows. (master)

[10]: /messages/by-id/17391-304f81bcf724b58b@postgresql.org
OpenBSD 7.0
commit 2022-02-12 335fa5a26: Fix thinko in PQisBusy(). (a fix for 1f39a1c06)
commit 2022-02-12 faa189c93: Move libpq's write_failed mechanism down to pqsecure_raw_write(). (a fix for 1f39a1c06)

As it becomes difficult to test/check all those cases scattered around, I
decided to retell the whole story by means of tests. Please look at the
script win-sock-tests-01.cmd attached, which can be executed both on
Windows (as regular cmd) and on Linux (bash win-sock-tests-01.cmd).

At the end of the script we can see several things.
First, the last patchset you posted, applied to b282fa88d~1, fixes the
issue discussed in this thread (it eliminates failures of
commit_ts_002_standby (and also 099_postgres_fdw_disconnect)).

Second, with the sleep added (see [6]/messages/by-id/16678-253e48d34dc0c376@postgresql.org), I had the same results of
`meson test` on Windows and on Linux.
Namely, there are some tests failing (see win-sock-tests-01.cmd) due to
walsender preventing server stop.
I describe this issue separately (see details in walsender-cannot-exit.txt;
maybe it's worth to discuss it in a separate thread) as it's kind of
off-topic. With the supplementary sleep() added to WalSndLoop(), the
complete `meson test` passes successfully both on Windows and on Linux.

Third, cases [1]/messages/by-id/87k1iy44fd.fsf@news-spur.riddles.org.uk and [3]/messages/by-id/E1iaD8h-0004us-K9@gemulon.postgresql.org are still broken, due to a Windows peculiarity.
Please see server.c and client.c attached, which demonstrate:
Case "no shutdown/closesocket" on Windows:
C:\src>server
Listening for incoming connections...
                        C:\src>client
Client connected: 127.0.0.1:64395
                        Connection to server established. Enter message: msg
Client message: msg
Sending message...
                        Sleeping...
Exiting...
C:\src>
                        Calling recv()...
                        recv() failed

Case "no shutdown/closesocket" on Linux:
$ server
Listening for incoming connections...
                        $ client
Client connected: 127.0.0.1:33044
                        Connection to server established. Enter message: msg
Client message: msg
Sending message...
                        Sleeping...
Exiting...
$
                        Calling recv()...
                        Server message: MESSAGE

Case "shutdown/closesocket" on Windows:
C:\src>server shutdown closesocket
Listening for incoming connections...
                        C:\src>client
Client connected: 127.0.0.1:64396
                        Connection to server established. Enter message: msg
Client message: msg
Sending message...
                        Sleeping...
Calling shutdown()...
Calling closesocket()...
Exiting...
C:\src>
                        Calling recv()...
                        Server message: MESSAGE

That's okay so far, but what makes cases [1]/messages/by-id/87k1iy44fd.fsf@news-spur.riddles.org.uk/[3]/messages/by-id/E1iaD8h-0004us-K9@gemulon.postgresql.org different from all cases
in the whole existing test suite, which now performed successfully, is
that psql calls send() before recv() on a socket closed and abandoned by
the server.
Those programs show on Windows:
C:\src>server shutdown closesocket
Listening for incoming connections...
                        C:\src>client send_before_recv
Client connected: 127.0.0.1:64397
                        Connection to server established. Enter message: msg
Client message: msg
Sending message...
                        Sleeping...
Calling shutdown()...
Calling closesocket()...
Exiting...
C:\src>
                        send() returned 4
                        Calling recv()...
                        recv() failed

As known, on Linux the same scenario works just fine.

Fourth, tests 099_rare_ssl_failures (and 001_ssl_tests, though more rarely)
fail for me with the latest patches (only on Windows again):
...
 8/10 postgresql:ssl_1 / ssl_1/099_rare_ssl_failures ERROR           141.34s   exit status 3
...
 9/10 postgresql:ssl_7 / ssl_7/099_rare_ssl_failures OK              142.52s   2000 subtests passed
10/10 postgresql:ssl_6 / ssl_6/099_rare_ssl_failures OK              143.00s   2000 subtests passed

Ok:                 2
Expected Fail:      0
Fail:               8

ssl_1\099_rare_ssl_failures\log\regress_log_099_rare_ssl_failures.txt:
...
iteration 354
[20:57:06.984](0.106s) ok 707 - certificate authorization fails with revoked client cert with server-side CRL directory
[20:57:06.984](0.000s) ok 708 - certificate authorization fails with revoked client cert with server-side CRL directory:
matches
iteration 355
[20:57:07.156](0.172s) ok 709 - certificate authorization fails with revoked client cert with server-side CRL directory
[20:57:07.156](0.001s) not ok 710 - certificate authorization fails with revoked client cert with server-side CRL
directory: matches
[20:57:07.159](0.003s) #   Failed test 'certificate authorization fails with revoked client cert with server-side CRL
directory: matches'
#   at .../src/test/ssl_1/t/099_rare_ssl_failures.pl line 88.
[20:57:07.159](0.000s) #                   'psql: error: connection to server at "127.0.0.1", port 59843 failed: could
not receive data from server: Software caused connection abort (0x00002745/10053)
# SSL SYSCALL error: Software caused connection abort (0x00002745/10053)'
#     doesn't match '(?^:SSL error: sslv3 alert certificate revoked)'
...

It seems to me that it can have the same explanation (if openssl can call
send() before recv() under the hood), but maybe it should be investigated
further.

Review/poking-with-a-stick/trying-to-break-it most welcome.

I could not find anything suspicious in the code, except for maybe a typo
"The are ...".

[1]: /messages/by-id/87k1iy44fd.fsf@news-spur.riddles.org.uk
[2]: /messages/by-id/CAEepm=2n6Nv+5tFfe8YnkUm1fXgvxR0Mm1FoD+QKG-vLNGLyKg@mail.gmail.com
[3]: /messages/by-id/E1iaD8h-0004us-K9@gemulon.postgresql.org
[4]: /messages/by-id/CALDaNm2tEvr_Kum7SyvFn0=6H3P0P-Zkhnd=dkkX+Q=wKutZ=A@mail.gmail.com
/messages/by-id/CALDaNm2tEvr_Kum7SyvFn0=6H3P0P-Zkhnd=dkkX+Q=wKutZ=A@mail.gmail.com
[5]: /messages/by-id/20200603201242.ofvm4jztpqytwfye@alap3.anarazel.de
[6]: /messages/by-id/16678-253e48d34dc0c376@postgresql.org
[7]: /messages/by-id/90b34057-4176-7bb0-0dbb-9822a5f6425b@greiz-reinsdorf.de
[8]: /messages/by-id/af5e0bf3-6a61-bb97-6cba-061ddf22ff6b@dunslane.net
[9]: /messages/by-id/CA+hUKG+OeoETZQ=Qw5Ub5h3tmwQhBmDA=nuNO3KG=zWfUypFAw@mail.gmail.com
/messages/by-id/CA+hUKG+OeoETZQ=Qw5Ub5h3tmwQhBmDA=nuNO3KG=zWfUypFAw@mail.gmail.com
[10]: /messages/by-id/17391-304f81bcf724b58b@postgresql.org

Best regards,
Alexander

Attachments:

walsender-cannot-exit.txttext/plain; charset=UTF-8; name=walsender-cannot-exit.txtDownload
server.ctext/x-csrc; charset=UTF-8; name=server.cDownload
client.ctext/x-csrc; charset=UTF-8; name=client.cDownload
win-sock-tests-01.patchtext/x-patch; charset=UTF-8; name=win-sock-tests-01.patchDownload
diff --git a/contrib/postgres_fdw/meson.build b/contrib/postgres_fdw/meson.build
index 2b451f165e..674ab0a7b9 100644
--- a/contrib/postgres_fdw/meson.build
+++ b/contrib/postgres_fdw/meson.build
@@ -39,4 +39,9 @@ tests += {
     ],
     'regress_args': ['--dlpath', meson.build_root() / 'src/test/regress'],
   },
+  'tap': {
+    'tests': [
+      't/099_postgres_fdw_disconnect.pl',
+    ],
+  },
 }
diff --git a/contrib/postgres_fdw/t/099_postgres_fdw_disconnect.pl b/contrib/postgres_fdw/t/099_postgres_fdw_disconnect.pl
new file mode 100644
index 0000000000..f1b51fb448
--- /dev/null
+++ b/contrib/postgres_fdw/t/099_postgres_fdw_disconnect.pl
@@ -0,0 +1,87 @@
+# Test postgres_fdw reaction on target server disconnection
+
+use strict;
+use warnings;
+
+use PostgreSQL::Test::Utils;
+use Test::More tests => 1;
+use PostgreSQL::Test::Cluster;
+
+use Time::HiRes qw(time usleep);
+use DateTime;
+
+use threads;
+use threads::shared;
+
+sub restart_thread {
+	my @args = @_;
+	my ($server) = @args;
+	$SIG{'KILL'} = sub { threads->exit(); };
+	sleep(3);
+	for (my $i = 0; $i < 100; $i++) {
+		sleep(5);
+		$server->restart();
+	}
+}
+
+my $foreign = PostgreSQL::Test::Cluster->new('foreign');
+$foreign->init();
+
+$foreign->append_conf(
+	'postgresql.conf', qq{
+log_min_messages = DEBUG1
+log_min_error_statement = log
+log_connections = on
+log_disconnections = on
+log_line_prefix = '%m|%u|%d|%c|'
+log_statement = 'all'
+	});
+$foreign->start;
+my $foreign_port = $foreign->port;
+
+my $local = PostgreSQL::Test::Cluster->new('local');
+$local->init();
+
+$local->append_conf(
+	'postgresql.conf', qq{
+log_min_messages = DEBUG1
+log_min_error_statement = log
+log_connections = on
+log_disconnections = on
+log_line_prefix = '%m|%u|%d|%c|'
+log_statement = 'all'
+	});
+$local->start();
+
+$foreign->safe_psql('postgres', qq{
+CREATE TABLE large(a int, t text);
+INSERT INTO large SELECT x, rpad(x::text, 100) FROM generate_series(0, 999999) x;
+});
+
+
+$local->safe_psql('postgres', qq{
+CREATE EXTENSION postgres_fdw;
+CREATE SERVER fpg FOREIGN DATA WRAPPER postgres_fdw OPTIONS (dbname 'postgres', port '$foreign_port');
+CREATE USER MAPPING FOR CURRENT_USER SERVER fpg;
+CREATE FUNCTION fx2(i integer) RETURNS int AS 'begin return i * 2; end;' LANGUAGE plpgsql;
+});
+
+my $outputdir = $PostgreSQL::Test::Utils::tmp_check;
+
+$local->safe_psql('postgres', 'IMPORT FOREIGN SCHEMA public FROM SERVER fpg INTO public;');
+print($local->psql('postgres', 'EXPLAIN (VERBOSE) SELECT * FROM large WHERE a = fx2(a)') . "\n");
+my $thread = threads->create('restart_thread', $foreign);
+$ENV{PGHOST} = $local->host;
+$ENV{PGPORT} = $local->port;
+$ENV{PGCTLTIMEOUT} = 180;
+for (my $i = 1; $i <= 50; $i++) {
+	diag(" executing query ($i)...");
+# Avoid using IPC::Run due to it's own quirks
+#	my ($ret, $stdout, $stderr) = $local->psql('postgres', 'SELECT * FROM large WHERE a = fx2(a)');
+	my $ret = system("psql postgres -c \"SELECT $i i, * FROM large WHERE a = fx2(a)\" >>\"$outputdir/psql.log\" 2>&1");
+	diag(" result: \t$ret");
+}
+$thread->kill('KILL')->detach;
+sleep(2);
+$foreign->_update_pid(1);
+ok(1);
diff --git a/src/test/modules/test_misc/meson.build b/src/test/modules/test_misc/meson.build
index 911084ac0f..bee227eabe 100644
--- a/src/test/modules/test_misc/meson.build
+++ b/src/test/modules/test_misc/meson.build
@@ -10,6 +10,11 @@ tests += {
       't/002_tablespace.pl',
       't/003_check_guc.pl',
       't/004_io_direct.pl',
+      't/099_case_15598.pl',
+      't/051_dropdb_force.pl',
+      't/099_pgbench_with_server_off.pl',
+      't/099_ecpg_16678.pl',
+      't/099_commit_ts_002_standby.pl',
     ],
   },
 }
diff --git a/src/test/modules/test_misc/t/051_dropdb_force.pl b/src/test/modules/test_misc/t/051_dropdb_force.pl
new file mode 100644
index 0000000000..94d170f081
--- /dev/null
+++ b/src/test/modules/test_misc/t/051_dropdb_force.pl
@@ -0,0 +1,106 @@
+#
+# Tests the force option of drop database command.
+#
+use strict;
+use warnings;
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# To avoid hanging while expecting some specific input from a psql
+# instance being driven by us, add a timeout high enough that it
+# should never trigger even on very slow machines, unless something
+# is really wrong.
+my $psql_timeout = IPC::Run::timer(60);
+
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init;
+$node->start;
+
+# Create a database that will be dropped.  This will test that the force
+# option works when no other backend is connected to the database being
+# dropped.
+$node->safe_psql('postgres', 'CREATE DATABASE foobar');
+$node->issues_sql_like(
+        [ 'dropdb', '--force', 'foobar' ],
+        qr/statement: DROP DATABASE foobar WITH \(FORCE\);/,
+        'SQL DROP DATABASE (FORCE) run');
+
+# database foobar must not exist.
+is( $node->safe_psql(
+		'postgres',
+		qq[SELECT EXISTS(SELECT * FROM pg_database WHERE datname='foobar');]
+	),
+	'f',
+	'database foobar was removed');
+
+# Create a database that will be dropped.  This will test that the force
+# option works when one other backend is connected to the database being
+# dropped.
+$node->safe_psql('postgres', 'CREATE DATABASE foobar1');
+
+# Run psql, keeping session alive, so we have an alive backend to kill.
+my ($killme_stdin, $killme_stdout, $killme_stderr) = ('', '', '');
+my $killme = IPC::Run::start(
+	[
+		'psql', '-X', '-qAt', '-v', 'ON_ERROR_STOP=1', '-f', '-', '-d',
+		$node->connstr('foobar1')
+	],
+	'<',
+	\$killme_stdin,
+	'>',
+	\$killme_stdout,
+	'2>',
+	\$killme_stderr,
+	$psql_timeout);
+
+# Ensure killme process is active.
+$killme_stdin .= q[
+SELECT pg_backend_pid();
+];
+ok( pump_until(
+		$killme, $psql_timeout, \$killme_stdout, qr/[[:digit:]]+[\r\n]$/m),
+	'acquired pid for SIGTERM');
+my $pid = $killme_stdout;
+chomp($pid);
+$killme_stdout = '';
+$killme_stderr = '';
+
+# Check the connections on foobar1 database.
+is( $node->safe_psql(
+		'postgres',
+		qq[SELECT pid FROM pg_stat_activity WHERE datname='foobar1' AND pid = $pid;]
+	),
+	$pid,
+	'database foobar1 is used');
+
+# Now drop database with dropdb --force command.
+$node->issues_sql_like(
+	[ 'dropdb', '--force', 'foobar1' ],
+	qr/statement: DROP DATABASE foobar1 WITH \(FORCE\);/,
+	'SQL DROP DATABASE (FORCE) run');
+
+# Check that psql sees the killed backend as having been terminated.
+$killme_stdin .= q[
+SELECT 1;
+];
+ok( pump_until(
+		$killme, $psql_timeout, \$killme_stderr,
+		qr/FATAL:  terminating connection due to administrator command/m),
+	"psql query died successfully after SIGTERM");
+$killme_stderr = '';
+$killme_stdout = '';
+$killme->finish;
+
+# database foobar1 must not exist.
+is( $node->safe_psql(
+		'postgres',
+		qq[SELECT EXISTS(SELECT * FROM pg_database WHERE datname='foobar1');]
+	),
+	'f',
+	'database foobar1 was removed');
+
+$node->stop();
+
+done_testing();
diff --git a/src/test/modules/test_misc/t/099_case_15598.pl b/src/test/modules/test_misc/t/099_case_15598.pl
new file mode 100644
index 0000000000..57243b04d7
--- /dev/null
+++ b/src/test/modules/test_misc/t/099_case_15598.pl
@@ -0,0 +1,61 @@
+# Run ecpg regression tests in a loop
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use File::Basename;
+
+# Initialize node
+my $node = PostgreSQL::Test::Cluster->new('p1');
+$node->init(
+);
+
+$node->append_conf(
+	'postgresql.conf',
+	qq{
+});
+
+$node->start;
+
+$node->safe_psql('postgres', 'select 1');
+
+my $psql_timeout = IPC::Run::timer(10);
+# Run psql, ...
+my ($psql_stdin, $psql_stdout, $psql_stderr) = ('', '', '');
+my $psql = IPC::Run::start(
+    [
+        'psql', '-X', '-At', '-v', 'ON_ERROR_STOP=1', '-f', '-', '-d',
+        $node->connstr('postgres')
+    ],
+    '<',
+    \$psql_stdin,
+    '>',
+    \$psql_stdout,
+    '2>',
+    \$psql_stderr,
+    $psql_timeout);
+
+$psql_stdin .= q[
+SET idle_in_transaction_session_timeout=500;
+BEGIN;
+SELECT * FROM pg_class;
+];
+
+$psql->pump_nb();
+sleep(1);
+
+$psql_stdin .= q[
+SELECT 'OK';
+];
+
+ok(pump_until(
+        $psql, $psql_timeout, \$psql_stderr, qr/FATAL:  terminating connection due to idle-in-transaction timeout/m), "expected FATAL message received");
+
+print($psql_stdin . "\n");
+print("psql stdout: ". $psql_stdout . "\npsql stderr: " . $psql_stderr. "\n");
+
+ok(1);
+$node->stop;
+
+done_testing();
diff --git a/src/test/modules/test_misc/t/099_commit_ts_002_standby.pl b/src/test/modules/test_misc/t/099_commit_ts_002_standby.pl
new file mode 100644
index 0000000000..59cc2b1244
--- /dev/null
+++ b/src/test/modules/test_misc/t/099_commit_ts_002_standby.pl
@@ -0,0 +1,68 @@
+
+# Copyright (c) 2021-2023, PostgreSQL Global Development Group
+
+# Test simple scenario involving a standby
+
+use strict;
+use warnings;
+
+use PostgreSQL::Test::Utils;
+use Test::More;
+use PostgreSQL::Test::Cluster;
+
+my $bkplabel = 'backup';
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(allows_streaming => 1);
+
+$primary->append_conf(
+	'postgresql.conf', qq{
+	track_commit_timestamp = on
+	max_wal_senders = 5
+	});
+$primary->start;
+$primary->backup($bkplabel);
+
+my $standby = PostgreSQL::Test::Cluster->new('standby');
+$standby->init_from_backup($primary, $bkplabel, has_streaming => 1);
+$standby->start;
+
+for my $i (1 .. 10)
+{
+	$primary->safe_psql('postgres', "create table t$i()");
+}
+my $primary_ts = $primary->safe_psql('postgres',
+	qq{SELECT ts.* FROM pg_class, pg_xact_commit_timestamp(xmin) AS ts WHERE relname = 't10'}
+);
+my $primary_lsn =
+  $primary->safe_psql('postgres', 'select pg_current_wal_lsn()');
+$standby->poll_query_until('postgres',
+	qq{SELECT '$primary_lsn'::pg_lsn <= pg_last_wal_replay_lsn()})
+  or die "standby never caught up";
+
+my $standby_ts = $standby->safe_psql('postgres',
+	qq{select ts.* from pg_class, pg_xact_commit_timestamp(xmin) ts where relname = 't10'}
+);
+is($primary_ts, $standby_ts, "standby gives same value as primary");
+
+$primary->append_conf('postgresql.conf', 'track_commit_timestamp = off');
+$primary->restart;
+$primary->safe_psql('postgres', 'checkpoint');
+$primary_lsn = $primary->safe_psql('postgres', 'select pg_current_wal_lsn()');
+$standby->poll_query_until('postgres',
+	qq{SELECT '$primary_lsn'::pg_lsn <= pg_last_wal_replay_lsn()})
+  or die "standby never caught up";
+$standby->safe_psql('postgres', 'checkpoint');
+
+# This one should raise an error now
+my ($ret, $standby_ts_stdout, $standby_ts_stderr) = $standby->psql('postgres',
+	'select ts.* from pg_class, pg_xact_commit_timestamp(xmin) ts where relname = \'t10\''
+);
+is($ret, 3, 'standby errors when primary turned feature off');
+is($standby_ts_stdout, '',
+	"standby gives no value when primary turned feature off");
+like(
+	$standby_ts_stderr,
+	qr/could not get commit timestamp data/,
+	'expected error when primary turned feature off');
+
+done_testing();
diff --git a/src/test/modules/test_misc/t/099_ecpg_16678.pl b/src/test/modules/test_misc/t/099_ecpg_16678.pl
new file mode 100755
index 0000000000..53d1af7fe2
--- /dev/null
+++ b/src/test/modules/test_misc/t/099_ecpg_16678.pl
@@ -0,0 +1,56 @@
+# Run ecpg regression tests in a loop
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use File::Basename;
+
+# Initialize primary node
+my $node_primary = PostgreSQL::Test::Cluster->new('p1');
+$node_primary->init(
+	'auth_extra' => [ '--create-role', 'regress_ecpg_user1,regress_ecpg_user2' ]
+);
+
+$node_primary->start;
+
+my $outputdir = $PostgreSQL::Test::Utils::tmp_check;
+my $NUM_ITERATIONS = 2;
+
+for (my $i = 0; $i < $NUM_ITERATIONS; $i++)
+{
+my $extra_opts = $ENV{EXTRA_REGRESS_OPTS} || "";
+
+my $rc =
+  system("\"$outputdir/../../../../src/interfaces/ecpg/test/pg_regress_ecpg\" "
+	  . " $extra_opts "
+	  . "--dbname=ecpg1_regression,ecpg2_regression --create-role=regress_ecpg_user1,regress_ecpg_user2 "
+	  . "--bindir= "
+	  . "--host="
+	  . $node_primary->host . " "
+	  . "--port="
+	  . $node_primary->port . " "
+	  . "--outputdir=$outputdir "
+	  . "--inputdir=$outputdir/../../../../src/interfaces/ecpg/test/ "
+	  . "--expecteddir=../../../../src/interfaces/ecpg/test/ "
+	  . ( "connect/test5 " x 50)
+);
+
+if ($rc != 0)
+{
+	# Dump out the regression diffs file, if there is one
+	my $diffs = "$outputdir/regression.diffs";
+	if (-e $diffs)
+	{
+		print "=== dumping $diffs ===\n";
+		print slurp_file($diffs);
+		print "=== EOF ===\n";
+	}
+}
+is($rc, 0, 'ecpg regression tests pass');
+($rc == 0) || last;
+}
+
+$node_primary->stop;
+
+done_testing();
diff --git a/src/test/modules/test_misc/t/099_pgbench_with_server_off.pl b/src/test/modules/test_misc/t/099_pgbench_with_server_off.pl
new file mode 100644
index 0000000000..409ef1621f
--- /dev/null
+++ b/src/test/modules/test_misc/t/099_pgbench_with_server_off.pl
@@ -0,0 +1,48 @@
+
+# Copyright (c) 2021-2023, PostgreSQL Global Development Group
+
+#
+# test pgbench with server stopped abruptly during the pgbench run
+#
+
+use strict;
+use warnings;
+
+use PostgreSQL::Test::Utils;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+
+my $node = PostgreSQL::Test::Cluster->new('main');
+
+$node->init();
+$node->start;
+
+my $pb_timeout =
+  IPC::Run::timer($PostgreSQL::Test::Utils::timeout_default);
+my ($pb_stdin, $pb_stdout, $pb_stderr) = ('', '', '');
+
+$ENV{PGDATABASE} = 'postgres';
+$ENV{PGHOST} = $node->host;
+$ENV{PGPORT} = $node->port;
+
+my $pb = IPC::Run::start(
+    [
+        'pgbench',
+        '-i', '-s', '1000'
+    ],
+    '<',
+    \$pb_stdin,
+    '>',
+    \$pb_stdout,
+    '2>',
+    \$pb_stderr,
+    $pb_timeout);
+sleep(5);
+$node->stop('immediate');
+ok( pump_until(
+        $pb, $pb_timeout,
+        \$pb_stderr, qr/PQputline failed/),
+    'pgbench reacted to server shutdown during copy');
+$pb->finish();
+
+done_testing();
diff --git a/src/test/ssl/meson.build b/src/test/ssl/meson.build
index abb30ab214..9228acf6fb 100644
--- a/src/test/ssl/meson.build
+++ b/src/test/ssl/meson.build
@@ -13,6 +13,7 @@ tests += {
       't/001_ssltests.pl',
       't/002_scram.pl',
       't/003_sslinfo.pl',
+      't/099_rare_ssl_failures.pl',
     ],
   },
 }
diff --git a/src/test/ssl/t/099_rare_ssl_failures-v12.pl b/src/test/ssl/t/099_rare_ssl_failures-v12.pl
new file mode 100644
index 0000000000..1c43f00301
--- /dev/null
+++ b/src/test/ssl/t/099_rare_ssl_failures-v12.pl
@@ -0,0 +1,84 @@
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 100;
+
+use File::Copy;
+
+use FindBin;
+use lib $FindBin::RealBin;
+
+use SSLServer;
+
+#### Some configuration
+
+# This is the hostname used to connect to the server. This cannot be a
+# hostname, because the server certificate is always for the domain
+# postgresql-ssl-regression.test.
+my $SERVERHOSTADDR = '127.0.0.1';
+
+my $common_connstr;
+
+sub connect_fails
+{
+	my ($node, $connstr, $test_name, %params) = @_;
+
+	my $cmd = [
+		'psql', '-X', '-A', '-t', '-c', "SELECT 'connected with $connstr'",
+		'-d', "$common_connstr $connstr" ];
+
+	my ($stdout, $stderr);
+
+	print("# Running: " . join(" ", @{$cmd}) . "\n");
+	my $result = IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr;
+
+	if (defined($params{expected_stderr}))
+	{
+		like($stderr, $params{expected_stderr}, "$test_name: matches");
+	}
+}
+
+# The client's private key must not be world-readable, so take a copy
+# of the key stored in the code tree and update its permissions.
+copy("ssl/client.key", "ssl/client_tmp.key");
+chmod 0600, "ssl/client_tmp.key";
+
+#### Part 0. Set up the server.
+
+note "setting up data directory";
+my $node = get_new_node('master');
+$node->init;
+
+# PGHOST is enforced here to set up the node, subsequent connections
+# will use a dedicated connection string.
+$ENV{PGHOST} = $node->host;
+$ENV{PGPORT} = $node->port;
+$node->start;
+configure_test_server_for_ssl($node, $SERVERHOSTADDR, 'trust');
+switch_server_cert($node, 'server-cn-only');
+
+### Part 1. Run client-side tests.
+###
+### Test that libpq accepts/rejects the connection correctly, depending
+### on sslmode and whether the server's certificate looks correct. No
+### client certificate is used in these tests.
+
+note "running client tests";
+
+$common_connstr =
+"user=ssltestuser dbname=trustdb sslcert=invalid hostaddr=$SERVERHOSTADDR host=common-name.pg-ssltest.test";
+
+
+for (my $i = 1; $i <= 100; $i++) {
+print("iteration $i\n");
+connect_fails(
+	$node,
+	"user=ssltestuser sslcert=ssl/client-revoked.crt sslkey=ssl/client-revoked.key",
+	"certificate authorization fails with revoked client cert",
+	expected_stderr => qr/SSL error: sslv3 alert certificate revoked/,
+);
+}
+
+# clean up
+unlink "ssl/client_tmp.key";
diff --git a/src/test/ssl/t/099_rare_ssl_failures.pl b/src/test/ssl/t/099_rare_ssl_failures.pl
new file mode 100644
index 0000000000..0ca70c0ff6
--- /dev/null
+++ b/src/test/ssl/t/099_rare_ssl_failures.pl
@@ -0,0 +1,101 @@
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+use File::Copy;
+
+use FindBin;
+use lib $FindBin::RealBin;
+
+use SSL::Server;
+
+#### Some configuration
+
+# This is the hostname used to connect to the server. This cannot be a
+# hostname, because the server certificate is always for the domain
+# postgresql-ssl-regression.test.
+my $SERVERHOSTADDR = '127.0.0.1';
+# This is the pattern to use in pg_hba.conf to match incoming connections.
+my $SERVERHOSTCIDR = '127.0.0.1/32';
+
+my $common_connstr;
+
+sub connect_fails
+{
+	my ($node, $connstr, $test_name, %params) = @_;
+
+	my $cmd = [
+		'psql', '-X', '-A', '-t', '-c', "SELECT 'connected with $connstr'",
+		'-d', "$common_connstr $connstr" ];
+
+	my ($stdout, $stderr);
+
+	print("# Running: " . join(" ", @{$cmd}) . "\n");
+	my $result = IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr;
+
+	if (defined($params{expected_stderr}))
+	{
+		like($stderr, $params{expected_stderr}, "$test_name: matches");
+	}
+}
+
+my $ssl_server = SSL::Server->new();
+
+# The client's private key must not be world-readable, so take a copy
+# of the key stored in the code tree and update its permissions.
+copy("ssl/client.key", "ssl/client_tmp.key");
+chmod 0600, "ssl/client_tmp.key";
+
+#### Part 0. Set up the server.
+
+note "setting up data directory";
+my $node = PostgreSQL::Test::Cluster->new('primary');
+$node->init;
+
+# PGHOST is enforced here to set up the node, subsequent connections
+# will use a dedicated connection string.
+$ENV{PGHOST} = $node->host;
+$ENV{PGPORT} = $node->port;
+$node->start;
+
+$ssl_server->configure_test_server_for_ssl($node, $SERVERHOSTADDR,
+    $SERVERHOSTCIDR, 'trust');
+
+$ssl_server->switch_server_cert(
+    $node,
+    certfile => 'server-cn-only',
+    crldir => 'root+client-crldir');
+
+### Part 1. Run client-side tests.
+###
+### Test that libpq accepts/rejects the connection correctly, depending
+### on sslmode and whether the server's certificate looks correct. No
+### client certificate is used in these tests.
+
+note "running client tests";
+
+$common_connstr =
+"user=ssltestuser dbname=trustdb sslcert=invalid hostaddr=$SERVERHOSTADDR host=common-name.pg-ssltest.test";
+
+
+for (my $i = 1; $i <= 1000; $i++) {
+print("iteration $i\n");
+$node->connect_fails(
+	"$common_connstr user=ssltestuser sslcert=ssl/client-revoked.crt "
+	. $ssl_server->sslkey('client-revoked.key'),
+	"certificate authorization fails with revoked client cert with server-side CRL directory",
+	expected_stderr => qr/SSL error: sslv3 alert certificate revoked/,
+	# temporarily(?) skip this check due to timing issue
+	#   log_like => [
+	#       qr{Client certificate verification failed at depth 0: certificate revoked},
+	#       qr{Failed certificate data \(unverified\): subject "/CN=ssltestuser", serial number 2315134995201656577, issuer "/CN=Test CA for PostgreSQL SSL regression test client certs"},
+	#   ]
+);
+}
+
+# clean up
+unlink "ssl/client_tmp.key";
+
+done_testing();
win-sock-tests-01.cmd.txttext/plain; charset=UTF-8; name=win-sock-tests-01.cmd.txtDownload
#53Robert Haas
robertmhaas@gmail.com
In reply to: Thomas Munro (#51)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

On Thu, Nov 9, 2023 at 10:32 PM Thomas Munro <thomas.munro@gmail.com> wrote:

Here is a new attempt to fix this mess. Disclaimer: this based
entirely on reading the manual and vicariously hacking a computer I
don't have via CI.

I'd first like to congratulate this thread on reaching its second
birthday. The CommitFest entry hasn't quite made it to the two year
mark yet - expect that in another month or so - but thread itself is
over the line.

Regarding 0001, I don't know if we really need SH_RAW_FREE. You can
just define your own SH_FREE implementation in userspace. That doesn't
work for SH_RAW_ALLOCATOR because there's code in simplehash.h that
knows about memory contexts apart from the actual definition of
SH_ALLOCATE - e.g. we include a MemoryContext pointer in SH_TYPE, and
in the signature of SH_CREATE. But SH_FREE doesn't seem to have any
similar issues. Maybe it's still worth doing for convenience -- I
haven't thought about that very hard -- but it doesn't seem to be
required in the same way that SH_RAW_ALLOCATOR was.

I wonder whether we really want 0002. It seems like a pretty
significant behavior change -- now everybody using simplehash has to
worry about whether failure cases are possible. And maybe there's some
performance overhead. And most of the changes are restricted to the
SH_RAW_ALLOCATOR case, but the changes to SH_GROW are not. And making
this contingent on SH_RAW_ALLOCATOR doesn't seem principled.

I kind of wonder whether trying to handle OOM here is the wrong
direction to go. What if we just bail out hard if we can't insert into
the hash table? I think that we don't expect the hash table to ever be
very large (right?) and we don't install these kinds of defenses
everywhere that OOM on a small memory allocation is a possibility (or
at least I don't think we do). I'm actually sort of unclear about why
you're trying to force this to use raw malloc/free instead of
palloc/pfree. Do we need to use this on the frontend side? Do we need
it on the backend side prior to the memory context infrastructure
being up?

--
Robert Haas
EDB: http://www.enterprisedb.com

#54Peter Smith
smithpb2250@gmail.com
In reply to: Alexander Lakhin (#52)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

2024-01 Commitfest.

Hi, This patch has a CF status of "Needs Review" [1]https://cirrus-ci.com/github/postgresql-cfbot/postgresql/commitfest/46/3523, but it seems
like there were CFbot test failures last time it was run [1]https://cirrus-ci.com/github/postgresql-cfbot/postgresql/commitfest/46/3523. Please
have a look and post an updated version if necessary.

======
1[] https://commitfest.postgresql.org/46/3523/
[1]: https://cirrus-ci.com/github/postgresql-cfbot/postgresql/commitfest/46/3523

#55vignesh C
vignesh21@gmail.com
In reply to: Peter Smith (#54)
Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

On Mon, 22 Jan 2024 at 09:56, Peter Smith <smithpb2250@gmail.com> wrote:

2024-01 Commitfest.

Hi, This patch has a CF status of "Needs Review" [1], but it seems
like there were CFbot test failures last time it was run [1]. Please
have a look and post an updated version if necessary.

The patch which you submitted has been awaiting your attention for
quite some time now. As such, we have moved it to "Returned with
Feedback" and removed it from the reviewing queue. Depending on
timing, this may be reversible. Kindly address the feedback you have
received, and resubmit the patch to the next CommitFest.

Regards,
Vignesh