HACKERS[PROPOSAL] split ProcArrayLock into multiple parts

Started by Jim Van Fleetover 8 years ago11 messages
#1Jim Van Fleet
vanfleet@us.ibm.com
2 attachment(s)

Hi,

I have been experimenting with splitting the ProcArrayLock into parts.
That is, to Acquire the ProcArrayLock in shared mode, it is only necessary
to acquire one of the parts in shared mode; to acquire the lock in
exclusive mode, all of the parts must be acquired in exclusive mode. For
those interested, I have attached a design description of the change.

This approach has been quite successful on large systems with the hammerdb
benchmark. With a prototype based on 10 master source and running on
power8 (model 8335-GCA with 2sockets, 20 core)
hammerdb improved by 16%; On intel (Intel(R) Xeon(R) CPU E5-2699 v4 @
2.20GHz, 2 socket, 44 core) with 9.6 base and prototype hammerdb improved
by 4%. (attached is a set of spreadsheets for power8.

The down side is that on smaller configurations (single socket) where
there is less "lock thrashing" in the storage subsystem and there are
multiple Lwlocks to take for an exclusive acquire, there is a decided
downturn in performance. On hammerdb, the prototype was 6% worse than the
base on a single socket power configuration.

If there is interest in this approach, I will submit a patch.

Jim Van Fleet

Attachments:

compare_10base_toPrototype.odsapplication/octet-stream; name=compare_10base_toPrototype.odsDownload
PK��J�l9�..mimetypeapplication/vnd.oasis.opendocument.spreadsheetPK��JConfigurations2/images/Bitmaps/PK��J'Configurations2/accelerator/current.xmlPKPK��JConfigurations2/floater/PK��JConfigurations2/menubar/PK��JConfigurations2/progressbar/PK��JConfigurations2/popupmenu/PK��JConfigurations2/statusbar/PK��JConfigurations2/toolbar/PK��JConfigurations2/toolpanel/PK��JObject 1/content.xml��r�����Pi*G.�N��9����9d����H�B�$��|}�)J�=f<S�Jn����/����S��^�)+����+R&,�e����������?<���&d���P�R	+�]uYo�q}����������d�*RvT{��R;�8���d�Z���K,q�x;�s��S��K\0�N�cs�Oun�X����#)N9-<��BT�:���5�,���@{���:���J��D2�-d"��-��s����H���>�4X�+��/���x�n�&�c>;6��{�t�{�T�-����Id}`����K,�b./�;0U�i5[M���3�zQ%�J�F\��=K�5��]�#��p
=����<�-��)��� /2L�����o8���uz��~�������L_G6hY\^,��nj�[�T���0�������E��Nw	�P3���� �kA�C�/�����[
� ��R �j��P�$����m
@N�T��s:FQ��!�X���U���Z�r�O]��� 	ib4��~zPE��Mjw���������!��UA�F��R�QA�����5q�F��X�$���*�	i��S���y��#��4?�-���$�����*�9���z%{��8�d%i�j��s��kV:3U������T4���0U4���$;5����X"�=�n��,��-��99��7(����C���^�CoS��x�f�
z��������]+������QI g��g�I����������L�CJ�=MSR	�sH��kP���u�RU��`a����D�����f{a�2�Ij���E�+��L��n6���������Hi
u�l�xK��>9�0���&#�p"���I�����������5~�mz��fS�n����`�o��jW�����������7�m��Q��-�
q���a�G����m�QRL��4�Q0U�/���4#g�R�e��-:��+��v�cO6��wc�_9�,����*��~e�~S���(i~�%�n��Z����~�������l�h#]|��3���h��6{N�fl��Z�qM�������[�7u�E���.�t���N
A�4&0�`����g��j�y�l:R�Te[R&��_NW��Vz�4>����u��n����{a��Q����w%�>d���:xd�e}%B���"�Kc�����?a�����8t5]3v���Xhi���Ku�������d)@�3%��O"�T�^V�jp9!����b~|nmn~A�F[~s�J[~E���N,UD��'0C�Z�(�hv����0��v�M}
l�62���m����VS��&a�C��J ��H�t�h��wG<3B���o�X
��S�?��U���N_e�����q��N��q�������}�e���(�(y�/��:s�O��~68&��y�����+A�u4_���2�4� ��^�i��t��Pi�������S���-E?,Wpc#)�f4e�f���[���*�f����<�o���{�y>��f�]	9��h'�5
�n����u���Y�a�j^?X����������d�6���N���"[�����Z�PG��?wm��h����2��)��/��OJ�/Z�����C���OV�����a��y�D���]3�d03A�MI�<��|�:��#]J������lG��N��[W��
��0�8s�G�: ��X ����:��x]�?���bz����c�^�A^�C�����-�����D��bM|mwn�;����C"�Y&2=[������c��:�0�����C��f��!�]���� w�/���L���Hsl��a��3�����B��g���B4�?�|�c���45��xDQ���{���!K�w��������p�����wa,4����\�v�a��!K94b�w����B>��eQ�M}�B��1�O2q%o���R����[�=�����g8t!e#-S���;��3���cg�S������3M�G���3�� k=5^��"?���s�p<�� KM���(�_���7�s�S����������R}M��������PK]]��
.PK��JObject 1/styles.xml��]n� ���
c�����y�
�0�)���*V����Q���p���e�@���a'a@8�M~~�Go��|:�����J��%\GJ3��I�\���
����DA���_E���
������-��5��xfwZt��laW]Id|�3;�����W<*�"�������_Ex���cb��B6 ����-a�q]/��*#s0�8+���J�wyH��QC�=��yP�!�=_�G�U�#��]�X��;�B�z!���Y��6��$/`�vhs�I5��q��� �{��LDD�y�+-����
$���["�����s�l{�-{l��tEYUw�)�L��
�����>�,����"M����w�9[����\6#(����,PK$)W�}sPK��JObject 1/meta.xml��AO�0���
R�
C�4�&��E��4�,[����������������N���m�#�^]�$@-�T�.���S�@��Ma�{%�I#��[t<���gK�$����^�L�{�3�U�.i�����Q��$�:0�c4���5�<��wWT�3�
����
�=�����	�jf/#c�F3���vIg��W��R6�.0�)L	���Q�x�*��[��	��C$�z��%�Z�1�
�F��;c��4l���EqD7�J�I�]�=�:k�Q8�b��m��0��Y�����}��PK:-[�HPK��JObjectReplacements/Object 1�][lU������mw�R��V �4PX����n�����Z��k[$��E@A���	Ay1AMM�D�������`�_�E������t��=�lg6�8���g�?�������93s6�*�di|�����|TA��gK��q�	c���D��R�����Q���.�?��QC{q���4i�U�8�����i��|]/��8��L��o�~�m���^�.��T�x3����y3�~kU}�]o�d����l�����{J�`J��� JlKt���t��9f�GGqRB�Ms���������c+���.Z&8`��>�}&K�m����OV��|:��������q��������H��~v�M���GEl����q�M�&o�fs���
�������''���J�V�X��jD%��e�/����Q	���K��[��������k��FT�@���5�V#*��"O���5�V#*��9 �b�x�g�n"f��[�����.�=3�Q%V������_,]o7��W���K����x�!$�b�x����K�_,]o7��W|�H�K����x��R���5�v#x�j�_,]o7��W<��`���������+�x��*x�U�O�N�.��"�]��g7��W���6�F<���w�����^������+��/�v#xE�����n��x��r��x�o<e�
�n�Y��y=��{���������g���������� Qn�;��>,&j�\=�hd.�����7{4�����9&H>�����5�8e����b4ru�:����Q���$O����	

�z��j.����8���?�@O�P���e�\-�Rj��Z�mMs-�77�m��M��!�o��k?���������������k��z��xY'�G/�^���&������02��!-�4�e��H�>>�����&��[���_�����K5YU�R��?G��}k����<U�2��>����0��tPp_S�*�g�&��r?}�rO!�}wXU�}!������
S��%��/KU���������S��)�W�����e&�
	�o�����o�*��T���H���a�}S���r�����6��������������v��~2d�h�=�
�k5�~���	?jl�h�T�����+OaH�,�Ul�����J4O-��m�10�������y�G����y*+Uk6�G�oq���z��m�,�����t%�Q��t�a+�~�E�rU�\�!WeJU���#D���$EC�V��iI��4/T����E�,/%EC�hAI"T"^�8*��D�t�I�P)�P��D�L�B��P�����
�H���$B���B�v��7�I�PY�/B�>�P��B��P��P9�C����*
I�J�*��J5�~f��������P���P9��~��m���i-m�5��\��vzG-�I��s��y� ���
�\��;*>K/��,D��l�2t�L���[��u)��Npm���`�`4K�j����rU@����r��<��rU4_�
�UrU�\U$U���_�C����O'��"'z�>�zx&��(!��Xd���zS'�z��L���F���9_�C��^���i=��m8�"������m�o����3��
�����G���}$��4QW���'F�����HW�@i
��.�wj�Bn/f�C��A\AZ�\R9��$5�DC�
��hX�*��J��R�j�\U&W�������rU�\U)W���R9��G�
Qu�1�k�����o,N"��0�"��^�5�kdI�^��h��(��Z�Ih�Xd9-�'Y������'XP�]����@XNW��&]V#���w���xb6&�H�gN�f���"1����nn�Shn�kp���c��w!�j�q'4�Hm��M��wc����4�y2?�.��#����N��D�E�r�m������kH��g�)��H���r����� ��H�5����C����@ZR89�f C�f����:a���s4�,L6�I��_��*x�|
<�m��.��g�x�����\���14�������!ch���*x]b�C�������S80;7��'�I�-�Q���PK�(�69gPK��JObjectReplacements/Object 2�]]lU>�m-��v��J+�4m��``Yh�O+])�1uQ�%�B)�Q@@�}0bBT�1�� ��M0D��1"��D��&F���;sg��lo[�3�k�!w�����|�9���w�VWD�,�-��L����z�l��=N4m��4�}�A���q�g��
��:����v����i���|�]��up��������Y�g��^����hG'���������4@[�����6���6%���56g�@yw�LJ�4?Q56M&�Xe�������A|)��M�b����������'Z�G5,>`���g����X�f,�40Y����6e������>������Ne	v�C�\QLpEt��~�q�������l�>�lC�U���v�9�~��D~Q����5�V#*���\,]#o5���EK��[��D�����k��FT"|����5�V#*��*G�b�y���_��X�F�j��C��]�2��w�t������*�r_��:W�b�y����K.������+@�������k��F<�
���L�b�y����s*.������+@�W)���a��v#y�k
W����B�����m7��W�������x�!��k���"�]�c7��W���/��F<���w�����G^�������+B����v#y��LW���8K>0�������K���6 �(�����D�\���Ot��UDsbsMR|n�(��x� �59���Qr�y�t����N�}P�]=�i�����p�y��%9��>����]n�E1��EW�Z��5,�5��k��.�c/����k[�3h^W{|�P�}�����]k�������s}������@�&s���0� �V����:�e���i�.�����"K�m������IUK����*�Q�0�G���Or���T�}5dO��=W�	/�jV��?�&p�:�H�T�����ExfU
Oz�5<u���.�HxbE"<;���'�����=���{�����X�jx��Z�#�=���{����\�jxnW��	�{�#��,�(�c���z���h
����Gb�y/��)������+U��@���Z\��I0�����������2���U�
��������Y���]���_Q��N:��5��J��~#~�*]����2��QRU����8�M�JI�l��DdS(�lJ��e�K��a����bR4�G��p���P��&���l���� )�MeA�M'�M	5�lr)��D6��#E�)�'��.�lJ��e�K�ts���wrI�lZ�+�iV���P��&����[2�M��I�lj��4;�lJ��e����ss����D���D6}�\6���1������*z�:�Yz8��<N����������?H��sVN�J���W�Qp�$a	zQ&������r��b��r#�-
��}��-rUd�\�%We�Uc��9B���e�T�j�\�'U������$������"'��N��o&��(BgB�1�8����YE�����Z]��W�P�Q;:�&\�+]8����-p?�@�q����{?�>P��;���(#7�zG����U�D��rY1�����G��F��(wP�/ ��\�����&\&Z�AR9L��U�|��@��\	�U�rU�\U,W���J��R9/�*R&W��U���
�*����)D+�U'#�J�v��&�d���y����}����E�4�n����n@>l7`�_#�n�P?����vk�]����I��������E���w����I��~�����|\�O��9�����Cw#�5��wb������q�Zj�G�8����X[�i�������jx!�D��']0�`�X���O8������7���R�XU��S�
��P��+��x���/����(o���<�M/ZR8�hI����yz���C8>QH�l�������s�:������<F8!(y��b����(���
/�������E����ls.���}�g��{P�B�(���7
{��7
����~}�������j��PKD[d���aPK��JObjectReplacements/Object 3�]]lU>�---m����?�_D��0��eK��R�Z���RJ@^DM4� ���?��F���!!�1*����'	&��;sg��t����lna��93s���|������������@��*I.* ������hf��8�xB��*���Y��Q�QL���Wq��8�)\�u�
��m��<�7O������ ��F;�Q�N�.�	��0���Y�`��������h;|��D��IDkQ��p$-��Dih�}�W��J'O�d���d����@ck���e��V������M+�
�v��9�.g���A
����tC���5�n���n&7��J�x�"�8DH�SA�CL�_�$�f�bKx�5k2����ehz����}�T����B��6�7Q	|�d�S�����>1K���6�f!*�/]���6�f!*���
����,D%�?&	����,$:�G��ry�����/���BT��{��IB/����
q�+>�#�bjx���wd
����*������^Lmo��W��<�S��[�8����S��[�8��H���6�V!x���	��xmx��������
Q|b��X�8�o�.��"�m��g��W���6�B�������U�^����
q�+��/�V!x�k
[�[�D|���?<�w4�G��z�S������6���3U|K���9���~�����a=�~$h�T����
��w���~0^Oo�~	,
��-�����(m8�%*����18�����4�����~����8�=g�G�<�8�x��t���.p>1��.w�)�s�T::�~�1W3!��$�Ws�Lrhu��:��R��"��X�������~.�[�?��{;;������y8�#-x�5�����k5����Qp��VT��&4�]LZ�Z�`��3(�.������.I�����#s(�%e-��j1���P�PEd�����=)��������B��n��~m���E���<-h��!m?w���?C�������fm?/����S�e������������������r��W��h�9��r���a%�\�Iu��?;N�QE����X�gO�p��SUu���{*�����s��B�o���������0�������T���CM���XjvOu�T�}�9�@�����E����
O��x�������;biX����a��f���+����I�B���g�����6���3wPm�>��/gM��bGrD,�U5��,��*�8�r� �8���8���j�qPQT;qQ\��8���j��qPQ�8qQ�^���4U��|������������ 5I���dU�`c������������_�D��2�H����@�V��D\?E��6P/=I�����?P�@l
�D��c�$�}#�k
i9��� g%JY�����D��-i6����h������,4�
;��Ym�>����.�>���}��b���^-nzp��O~�1�A���W���5��?C��p���i4:�2�R
�lE��S�n.���P�L":�R��9!U|n�D�v���!3:������-Wd���oQ>�*�k<wkT����7Q�J����,_��5I�J��R��4��r��#g��YrYr�/S���YYrV���#�P��M��r��<9+_����+����YS��B9�H�*��J��XN17��B�xZ�HZt� ������L�E,�����!��C�o�jZ�����m��)���UBB4�`���L!��
��qT�:���t��
�=1h.��X��w�����q�r/V��5��f��6L��Zj�S~�{�f�wQ;,�i@��(�#9������sO���D��
>C��^=����g��0%�a�M�${^Ov.��H��&�zQ^��7�'��6���0�e�����L�<������0�iS>
�����E=�Q$��e�2��$���m�xi��h��}\B�Bh�h��z����o������X����}Dc�=��sMc0��n����(�Q>�`�c����Q��c8
�YFo�>�'�������PKH��Z*�`PK��JObjectReplacements/Object 4�]klU>��k��n���J_���.�����%!�
-�`hx��D���?�"`R#B5$�?<4�HT��DM��c=g�3��m;���.�4w���{����o�����]\�t�<�^<P��\	@C�o"@������c�8����
�e���VS&�-��U\�kq��8e+-����n}�������>��k	��~��V�(K��Q"OT"U��5h���z�=�]
��:$���=��n�V`��#��df`�m�����������oraYKc����%�C���g����,f��Z���p#�}S,Z��d�Mk������wB�p#x�uTA�C� ?���@*P�DEh�S#���&�Y��@y
�%��j���'�*����,���v���L��
9.�ot"�7�8.�ot"����j������l��j�����n�E�c��Nd�>����1�F'2������v���=�G�)����b�t���q�c���,Z�_�ON���v���^���s\T;���/��,��j������%?)�qQ�x�������v���^�8.�ov���|C1�E�c��N\���[�qQ�x���?,���v���^�t7����}t���^��=EG�����bR��7;q�K���f'.xI�;>����/	x�'�����%��,��$���dv��������N\���w|����^����3;q�K^���(x�{�g��g����S��]y
�k�2;lE����3G�<��'��[<���4]������.���a�jk�A��)�Ad��%(��[���1�{%�q��1�S������8N�3��*��Yc W	yr��=�?�����2�L?�F��*�������4.X��:��%�Z`�Z����	������M+=������K�W���d{`�U?=!��g7c��X�w�����h���c���;ek'�w�9���`�GhZ$��b4�����Y��9���:UV�[Rt�k�2����������!+���F�#��&Fu.����-+��e���F�}m��~����9 +��~#��t_��/�e����}y��}]D�������3�M��r�]���H���Q��+b������"�������Q�?V���-���9%F�#��>Fu����r���O��psOg��+�^���q�6�?'4E��4M����x�|�g�����z��c�)"�����Y��a9^�{�d;<������~0�
f)��C��6�A�
�ql�'������Wl����h��j�`�8Yu0u���*K:�ru`AM��?
e���BM�-�`��:���:X�/���5T[�A��:�`���U�9�j,�������b,����_�A�%��:����������`G���:K:�su`A����L�U�i:����zWt@��K:�4EV�Ka���G���]���x�0a_0?�|k�.Qz �K�Bl��;a
F��`5l�����yX	�o
��s�J�@}&b8�X^B���X>@'��k?�;(��,8�p��D���X�c�\�y����8.% ]'������[�&uL��bS0IlJ�|bS���*6��M�b\bS0Cl�����l��/6���)Wh�f���a�G-��l�~��mv��m~��S�m7�Bk�mx.wA7���X�y]��t9��tvw�}��[�)�~���c�J���\�8&��,!�Ka"��P2��*5�<�����=J�"Q���#j�y
�
��L?yE|W�=�������JP��Ba9TXl
��M�bS���dxo2�R<fg�I��)�(
L�;+��p���{�=�?�{A�>������]��6{{Vv��&��~��
DxRaTt�����Q����.��h�u�]@=>�z\�
�]��F��������6C�V�������f���vdt%�oU>��+s���?�a=���M,�0�����.z��X~�r�0�6Y8�\�R��5,�pT8�^(\�r�V�r��Z�����^4�%c����;�q=,��?�������zua�,����b�i���O��.�1;2`S$�����##��g�n�v)�'�8r��;�1��<�V�����K7j�\`��t:�c��t���������w�����,�,,��K�������ZW��PKc��X�}rPK��Jw�{A��Thumbnails/thumbnail.png�PNG


IHDR���[{�IDATx���=r�J������bx2	�I��g�H ���)��{�^���4���<U��}n?U�w�43�1m	�����v�A|�_�O�����i������8���;��s(�����~�/�|�����1[|�n�������������ao}�7J<��|����~�=_���z
�nn��
���.a=������l9�����`���[��+<�)����9o����h��M��.A��}�|����9_�������W��nr��A
U	�bU�U3k#g��Zdu^B�y9@������_���|�aQ1eW�����U^�n������t�����e�_���vx���s����-R������19+�����~�=M&���:;��	���|�b����)+x,W��jZh1���� ���X�$�b���7~q��m������UV�?���_=��P��P��P��P��P��P��P��P��P��P��P��P����S�0��1���w����&�1�����=����o
9��;]�TWX����,6����K�m����*v��������m��n����a?L���7���>��|~j�bUN�!
�6��b����*����az&��]�W����`xX��������q�@&�������tb���Q]S8y��S��i�����������V%7�V97r�����]����=����F=��+�l�n�>��y�aQ}V��;��p5N�����=������X��`���>�
�F�u�����u�������pl��-����gD\g��w��m������a������F�7>�r.�$I�_z7������Re$��&����%t�tn]�
m�>�a���\����#^?��C	����?������������5j�]�^eP�r'{�=0�(�;$	\&���~��u�S��3������}��Cdi�cqXg(��3Q�u�}:�7,e�@�gs<N��ag���j��0��n����{��_��]����d��7��q��0��� �(��)>�������rkx�,�Y�(c��,�����I�������`�������+<�����������zx(�������&K����\����7��@�
��_\0+�[a@��H���?���v��'����~�)h3��h��x���PV��'J:A��i�g�	�9+H���9�O/�e�~����1��\�)�������J ����{
����K���P�K�1�������S�:I�L�����z�d�p�<�������v2\j��gh�����_��
���;�f:SO���|�O���9^�}�a����	w����v���`[��,S*����V������/�B>@U���wX���������~�
S���8"�&�
}�e
w��t��]g;!��)Y]>2��}�*�\a�+��W�����,��{���������N�H�K�����l���>��axJ�� �H$������\Bz��=z8'��!��!��!�:CT,��=��{8��H�fTX�����hQ�]z�1%����-tz������[;���>lj2��{���e��������Ajr��K�x�^��)t��� E�~� "/��O�&���++0�w����=�c_i�5X%l<�w�Q7�$A���]y�������&#���]���YQ�<�H�	��C%|�����e�U���^!��� �duEA>�$��C�S���Q�����u	������	��~
o(
�MAFW�?�
��&;$lM��
������@(t�@(t�@~�$(���B�3�;��������B<�����/	b?h�n�3d$E���(��
��#i����g)���O|IP��v;�� �,����X_���I��xJ������$�ItE\��|��^.�HZ��"�
������H�v�2� ����y���x�5�H����S��7�3�a����f+�����k��������s"
"
"�h�N�8\������9r�i{.:=_��:(���/z����}`ZY������S������ �D�C�#�$A?�t�L^��4o����l����-�)#��(�D�CV�;�p7��U!5����6�P]RE��y��������_�����/��?��}��4o�����	wN���d�  ���kzcG�����+���
r�
��X�;��/�s�u�q��G.:����@(t�@(t�@(t�@�������aHM:�������X�U655�e���'t�F2��,��:��,5b\s4K��G��M��F=�+��>H�I��i�XA���CL��r\��Se�c��yC+�*�^��N��f&���M�������_�?:(��B�\�;k�S�O�.�d�0�j�m[E�������������D`"}L6���y�������������t��(�J���9��H$]��GH�r���?���n5�K�#h+q�#}���)������4����U8�`��W%�3��bK�LhO�����dj�.r�x7,������^u�(�=UQ�@����vlumWNFO�'\DE�SU8�sn���E��h(~�j��^]{�55rR	�X�#u<S��z�a.�C<K�l�d@����v��������R	�X�#u<#�\5�nl�P%�\�
m�a�mw��^����B��B��B�$���y��j�dU���F)�5�m}���@8�UKP�Z�Lm�
�����jj�j$�Sl6U��o���4�~be"���
��r��+
�z�E)��N3�k��9"�)��X_J��t���\�n��L�_,9 �lli�/��D����I��L��
2��N��k���7�cK`97>�����>&�m��*�W����	�75��W-��s��\�H����B��B�$R�X�����Fg��c	uI|���z�n"=����BZSaO���X��;���b?u�}8���[��b=�;��L�-�K�Q>��+���j?u�}h5��RN�p����/�V0��	��I�ZS#�~j��T��WN�p����_���?�n�j��y�FS#d?u�}8��_N�p�
$Ao�sG�QG����r_�jdD%L�B�D :D :D :D :D :D :D :D :D :D :D :D :D }�T�c��S���^�j�|������e]s�
�`|�������Nu�ez���b���������x���/�vL��Kg1�S�^�a%�}8����6b�����������k1U��9�O���L^;�M{�W�m���M�Oi��]�:U�}��N�_Z�������L��Ul��g�����*�}C�*/���e0�v�����C;[��p>��+���������*������u�L��������	����u�r������Dib�F�u&����v_��:����K���96���d��B���V���{�������;�LX�;f�m�����P�$U~Y�n��F��g�22�-z��<D�cn	<��!G��n\W����M���W**�����������d��uj���R��D���p���~N���b!�4pf�y/����n�!����8,�3����(G���>���q���H�A���K��a��2m#vI�*/�����c�Fqx����?�45a��x1>4����9�*�
��,;����IWyq���D�
���2���n"}�a�����v@�W�*OO��|�~���sW�*��{=+�'~q���&|�!]��B�$z���8/�K/����#-�|�������%RJT)��Z�+L9�����,"���C��t�<r6/B�*������G�%��������O���\6/��#qR��U+�mydo^�h�D��%�g�SH�PW"Yg������
G�Z�k^d�%�D�a�/�L�.]��g;[��c��s���EJ�t�K�������N}�D���������l��/�pNB�CB�CB�C1tWop�C�a���#*�b=���z���h�#U��V�����=r�6WYJ�t�")�����Ke�����jd�� Eb����=r�& ���tO�*���FZ'%��L���JM62z�K�J����d�#!:c-��=�S������LX�)	v����*K�r����0EtFZp��+���F��������I�',�6l���Z��
���")nj�$d��L8��s"
"
"��-	�>�"�X�mnZB]�U0;)QS�p�K�Z�sjj��hf�m�n	u�W���������J��/	������X,>?�y���|iZB]������WH���U	'�A�i*��������D�����*�Y����R
)$n|vU��7��RS��$�^��
w�6WD?CT��uR�B���Jx�����aIPin�m��~�$��^��*����@(t�@(t�@(t�@��:��
����������RM�(�x��rx�B%\D
��h������)H��C��5�T�{[�CYf�P{���Ity@uo{����ziR�235YQ����,��S�M��a���V���������d��Rdk���O��_��"~��*���E�U+�Be6�d�@5���g e������
�
���nl���?c{��&����\��e�����@(t�@(t�@��$�TU��S�([.��`j���������V(TS�����d&m���q��i���8#.���x�`:�(��
M��{�~!�92���e@��^;*W�
�_ J�(�B��������xm
�O��
w6K� NeF�r�8h=�nW�`:�87,\(4�2M��{��������fS-�\�}Yc?h�u��s�	�H��*��	�FY,%��s�A]	��z8'��!��!���X*?~�%h�*m����'��&�
�u5�B�',��#njt�%���jA"�h�����V��
�c��a%�
KV~���w	"�/[I�_�V�K|�A+������ Z�`k���h�K�����&�+:(Z
��g�j��X�#���Fw]��(�~}S#��%��P��g�x%^A�&��
�@n~��ty���g�����N�T�r,�pNB�CB�CB�C�`�H��55��r��c����G��S��'�j	*UK�4o[S#������
�Yq������}{s4��b����o#��������	�b�b0M�-U��:�SS�����S�z=�����!@kb4�� �Z����U��S%\D���J=�����%@��#�Q�L�A]�I�y���H�
�6SJ���9h�F�a0�w�*8��<z8'��!��!�:�*�6�i��u���TjZB]z#�5��F����\��,C������Lr�%��7��n�#I�#��p.|�_������;�������F��jIK��s��\NY
WJ�U�:�0��P�����
���|<q?�t����%��*�F�%�3���F�<�p�R���|I�N��{�]fLz��$���$����9������������H:���n��S���e�}�eg~�F-��;n���;�����.t�+,�c~\`���%����f���<�X�c2����[�8���w�^�J.�p���)M�|o!J��>1�2�X��S���c�4L���u����
��`�����S"�w������6������K��TS8y��S��i�����������V%�ohV���7�e�/�����
��m<����J,{�� �����*����"��a��3�PLg���L`�G��������$�H�v����������_g�?�i�m
��=����X�4����wo[}9��5
b��������m�����P�$U~Y�n��F��gj1�t���[����%t���b�����u�}��*�gSx(��<_>~��z�g	�Om�N���Q�7������,���t���.�Y|�^��)�X,���l�!k�k��~o�!����8,�3����(G���>���q�����������s��p�L��]����/v����h�Q����1F#!���8E�����Gs��`Y��$Q��Yn?DW�.����VSV�F~��y���?���j��LU��f5�6���?�;
�z�7�rM���Ya���u�@(t�@��a�Z��"�C�J[cr�?�����k��Ea!RJ�@�u$U��*q�H����WQb��[[$V"��D	�Ifw"QW��}��k@����u����y���h$�>��J��p'���)*���q
O�IG�955K�����P�"�B�u1�=n~u�*1��3��F��U7YQ��Y��0���H}��#��^��+���P����@(t�@(t�@(t�@"��������F������^�����`��	�Bu�?�n�������U���ZTK��^)��R������M��RC��d��~������hs4
� }Y*���D#=UK5m�p���b���sM���G�0�
���H)~~QG���z�iS]�e�z���������2j�&��H)��r��j)��	/1��\���M��G5e���*����U��%���9������F�tl�m�R�k�nW��_�z1���^
z�WTR��67���$��n�
I���/	&Fkq�$�>3���[����QL��)�A�E���$A���U|�=�$(��`�J�s�����$	��H���Z�*	f��>�I��'!&��]!��������$8�o 	=������I���'v:]�8��$����9���H4]�.�jr��p����.��2]/?�m����������"R� ���/�,����d6xOEA�	O*\��X��-5K�j���	��H��v��.K��m���A���'��UfC���R��xU�����|%|�Mfl
98[���6���"�WxR�+CM���J�U5�27}��'~��6�L����p��S� ����1���X)���X���~?�oP���x����l1���X)�Tc\A��p@�D :D :D �5Ag|#-yIEND�B`�PK��Jcontent.xml�]�s�8r�����T�ru%�����K�n'��%�������h����DE���� �h����DE��h�@��F�h6������s��g��r@\��I6�-o.?��OC=����!���M��4��-�e1�d���^����������q��g��2Y��q1g�t��j[���<e]<�k_^6�W������mpmrU��1�z�'�u/vm-�������_���ufQ_��bv����l�����(V������3��7#b���;�'�v��|^��NF�<u��G����"-����������*�kC��������mbo�	f���5�|}��-]�o��yr�����l�
��6��.������Gme�����\���c��T�|��}��5�>����?�Kv)�|�����W����"�A��W�O��d�x����##�b�~v"��D������5^O���>~�ir�.�}��������H�{dr���S1��U�;`��+_������b1Zu��m��|:�lj�a#�F�$~�����������v�;K��Y�k�����*�g�N����bmA������j/��+��Fo�����hG�����:���i:��������|�������YbA�zb�`1�?���RW3�I��q+����z�X������{��a���?�Mf����d���y9��tZ��?%�l�/m}��Y|X��?&��J|v��>���:�..�7y���D��]d���fN+<biWq�����yCO�
;P1��~v�^�
x�dd7���r���q~�X�WB�pe�b��t}q����4�mx�Z�i;tCo{�4��M����6T���
~����E���9�9JuC�x<����s"sXr��9�9��
'��8!m�[��&����,r���6���NUU
��9�[��lU��|//���	�Oh��1�%��x�T#�%.�����Ur����^'w�����y�w:[������Mo�@�n�pa
���<W
7}j�I�;Sa��n��,0�iQ����?Q�����7��qo�du;��0��!
������o��e������|�&�e:�f�����X��|v�tw�,����g7�d�j1���������g��W�����>�O�fR���Nw����;`����'-�M�U6}���?O���4-���e����/��m�X����Yg��R��}-|���{�*�N���:���A��F�U\^Z7�1���M���zUiR��1�S/k�HV
������>����jUP����!m;8?|v��x$��������|<�aw��kL�w��xX����YR�����nH���z#�e����z<b�5�r?�/7���1��1s��Z�s[>)�� ���.=v���7�[��yf����f���'�����G#��hv�(�O�S��C���h�+0�U[S�0E������N��K`e4�,�V���T�iN�
�O���$�g�|�a#4�"����z���^r+�����[��t����ck�D���Rn�Uz�{��#cM���b�>�-��k>�:R�iQ�c;�(�MK���x�� 5Z�^q� 3$x��TRK�'jK������uv����c.�bv�0�����u%�~=���MZu��.����4U����
�Ut����w���������}E��+:|O���<������������7��l�*�J�����VL��dR���s9���d�}���#�<��"�I+�K��'��y��x��PN���w.���R��(�`*JW��jq.V_��$��Xe���O��f�
� �e�K"|&V_��&�qU�{+�����������J��;$�����������W&�jm���j��;$������c���
q����w��/F��R�����[���u�n����)���rW��Gq��w�[<q�����^]�0������
�(��Vs�0�bRm�=���Cw��r�]�/����w���/�=���nT��o#,����
W���rP��gn��B�"x�]/���t�N�����U����5��I��haT�B�m��b�j��[#>�/i2���r�,���PW�S/�q��b�Y[=S��e��gA�.��3�����)��.w�����n�I����`�=�SC<u[xJlM�)#
�'$�O
�1��9&���"�0pBj?�d�P�����0�8���C���<
p�LK1��0M8�\
u'$w���R�-%&�VF�1	�r���c����0�����R{�f;I����4��MY��T+:~������Gq��ZO&����k�%�a\��!�0.�����"%��#�����^�����=�
{~^��JRu:�P�����Co:9t�=I�A\	�Or�����KF{����o`���bki��5���Ar��d���?�2��'?:
�W����_��*~����?�Sf�(����'�*�;7�����P,�B)0\@���:��k(��(��
(#�i&���R�m�MM��S]r�d��yC�95L ���>,���tz�=3�8N�r�>F�E�y$
��}X� F$�PGn�������kw�T�K�E������N_���j��-�R4����W��6mW~�4��
���U�����.�M�g�D��J�'����V�L��X��BTq�@��-s�g�PB�&~=����J����E�l�mM���������.� ��j(��mC�c��RQ���M��������u��DC����E�64����8G�(Ah�zmk��_���~��!��&��	w$*_s���o6����H�p��M$��o��8��/m������
��@�����3�6,Q�XA$�Ig: Gp`��d?Zy��;�*_
p�F�`����2mf	CA�i���>o.�$F��e ���n�����B�N{GJ}��X�R�q8^@��xv�?\����@��{�g��VZ��s����g� l�6��2n�����3�1H���^�H
��~��X��2D%���@r��|���*}������G�LZ��:QA�������{��������a�����r��o����F�_����������
��d5���J!��	=�E+����	G�C���l�f���l�������f�fR#��Q�9���om��fG,L����2a`��-�'����
c&���Gjr����&�"J���������S����)��d$"�� ��'�[�D�I��S'�B� ��������D��AL��2�Q�����m���V��=?�d	���S����/����h������e�����-T�L��W�`ZY�3�����o��8+>o-�QQ����~@	������T#e�4A��=�P/t�e8I��Mm=;%��� r�a8c������D�9�'�H�M���C$C�� �R{%��I�/����Z1�!A4*���8�*Za$���^�NC��p�����A"��o�Kd���XI���8Y���X1�y���h��,V� ����5u�T�qXk�k��z�o����=�
�}:��P�cy��5�n���s<���K�k-��?C�)�\!�����9���c�K}9 ���W���fS�?
�J��R�(�����MM��3��\�~DS��QM�X�zZ'�[M�(�4�j��$�����iu M����,+e�in%M("3z�L���<��6!O�zS��.p�R����5��Zqv�����a������u M��&���i�HG0��]���:OW�r���SN���+��7�]`_�{Ys��o5����LbL� ��vhjH�������f��Z��������;��6�O9��N�@���,����bZ�����p���GdR;4�������6	7�&��K����D���v���f�)�UC��5_
r�b%>T	eM/�V�3���K�M�������u M������%j�XFX?�hk@�X��e?k�������K��s���������r���<	.n7���S;4����l�M�R�C�&�.#�z�d��vy�$��G6�[�
aA(9�b���F}����l� ��
R�j��V��1�M�R�C�f�&�h���p���;�<������sL��B4�������g���{����������S�=��Q*�:
2J"�%�����zR(��}�.�K�Ht����6J�H97v���
7������Y_O6��B�`�
P{1�b�dK��D�(�����2��%���FT�{����(�U�k�������(��P��R���P�qH���������P>3\'���B{���(��s[Z�e�[���iL��F�����r�sv�7@�PF	5��+$�������(��0�f��|N�T����2��.�k�!BO�S{e�k%0b�zyK��1�s�9V.U$���NC��p����5��]_������Cj���S�]�-hZQ�����2���)e�0�%�^@�@W�~�����{j/��c�+a�2�K����2��.��cD���G{j/��b�+�]"�)
R{e#���a��I��NC��p����5��1H���=��N�e��t����rj?��a�KC4����x@��Q�t��!#Rq
�P{e�0�^u��C{j/��b�[Q!����9�Q{e�0���s���S{e$#���"�1�����m%���b����O�}��&���7J�N��s��c�v�+
/pPx\����5��K[�������������vA��Yk����W"��])_~��+ ��EI�4��[��d	_�����^���Ks�>^����?���$���*x�l�L� ���� �+��y6�
��q���
�{Q����l[�����q�����\|�����V����e�=���Yb�U�`}Tm���1��_>�������-���	�6��I��F�qB#M���6�rg$�l��.	��X���2Li8����r��iK�1�!/��s�<���+�����W,�@�L[J����e�KB~&6[}M�4��&���r��K�~��6�T�������Xm��t�`0���_��(�m=a�iD���r���\���bN��0r�,�b1'P����040�r���L,��F�QP�}�5�9��m)n'���,wH���v�o�	�(_~��N��m=a�t���N	��Xo�C`5�����W�����Rq�����V_�S��4�����<��H�g����Xo��"arY~�b�d����#��ll��bng|��,�b1�q6��y�y�{�����{~^���7���{f����lg�:�����\����9�����]t/c�k.�Pp��U-|]�\*�~a�=�����v���Q:3����D��87�cx�<�3�=���|
��wZ���M�����j�����&-<i�2Q�}Z�\���-�#!cP�H����I�,����6M�i~��:�����ak����t�o�|�h������I1���������	�����A�W�er}����x�M��i6�s���}��7�PKi�i�y#PK��JObject 2/content.xml�ZK��8���0<�������sX �a���[��h�J�Rt��_�EQ�)Yv�{��	0i���X��UE!?�x!���|\"�].H�������_��d��������[��u��}AJi�������z�����(���^�� �Z�k^���Z���F������o�MiI�r������|�
�)�	|�+�x!�����>���r�zQaIGV-�=.wRVk�9������$��P{������pe�CQ�j���x"�\��iR�/6D�
��"��K>/����;,fc�a�������L�������W 6���r��(��R��P��V�����<��7U	�����n����}��~Ta��7�S��>���
�!8,��`�_��"�9��3���������;R�33}���e-qy��PI��i�Rq!��l�L�������~��c�E�M��9�W.��B���A=����i�W��r����:%��W=or����w�t���0�x�6�5z[I�{y���U���p	ij5��~z�E��Mzw���������1-�UA����2�S	�CK����8s�"�V�Te$�fV�pJ5�4B+�T���H���e'������W;[1'BRR/T������$�ZC�-u�mu��T���J�U���&�R��ZFuqSd��&�TR���J�W���m[p�py���#�������9t��I��9�6���wn����0�q$~���g���������YI�~f���I�9����+��L�>#��f)��0W�Y
�_��I����R,-RT���H����A�|'-\��d>�Z��r������f�|�ZF�����PO����0�c�;vMG�D55S�T���7�����h|��Hk#��M_G����7�E��?2s���on�7���6�x@��b��%O�12���@3�rrm7N��V��0��*���1���|_�o���	G���6����{��0���sp��>nV�m�+���j��Ei���"����%lxv��MO�����a(���#j�x\&j��T��	/c����5Ui�:@�p]�������C�4��z�tVH*U0A)�%�]�uF��}m���U�t����7�Lw���-�V����RCM���ojCPch��*���?k��'qN�������K����0���Y���Xfsa�OX-;�;Y�����x:W
ZYx��R?�w��i�!M��I�$#����C-'��,DeX��6��'���-��g�(���5��E
#4����}��v���p��4�%��ll��?�-�(V0j=���S��=,���������JB�#�/U�6J�m}4���O]iq���a�-�8Y���D�w�>�4}���d���$K<���M�9���"�n68&��y�lOj��|b�����T�@�O�������Y�Ai������C��4��8-e?*W�^#`������1�-q�rq�f����kx;n�����������}������5�I^��M��`<�0U5�>\����������R�6�������e����C��������3Ry������6��?�*��3(�0���_�3����1;�Z8M��*����\����������'��	�lF������9��{p:�{S����� R��Y|y��Z�\�~
��0��9@�p��\(0�DgI4/����#3?if��0��(H��2�sa;nt?�������Tz�A%�b�^|-���S?@�E+#5#���^��C'����#�M�wJ�m��Om��
�;�"$������3���9B�J�X;�������������t��Jg���c�s�����(�����(Z���Ts�{$5���K���wJj���(��Bq4(�c������^x�1
��GRo+}���g$wJ��FA`�h"opS��;�We���0D~<������w��V�>n��v�GK���[������PKm�l�e*PK��JObject 2/styles.xml��]n� ���
c�����y�
�0�)���*V����Q���p���e�@���a'a@8�M~~�Go��|:�����J��%\GJ3��I�\���
����DA���_E���
������-��5��xfwZt��laW]Id|�3;�����W<*�"�������_Ex���cb��B6 ����-a�q]/��*#s0�8+���J�wyH��QC�=��yP�!�=_�G�U�#��]�X��;�B�z!���Y��6��$/`�vhs�I5��q��� �{��LDD�y�+-����
$���["�����s�l{�-{l��tEYUw�)�L��
�����>�,����"M����w�9[����\6#(����,PK$)W�}sPK��JObject 2/meta.xml��AO�0���
R�
C�4�&��E��4�,[����������������N���m�#�^]�$@-�T�.���S�@��Ma�{%�I#��[t<���gK�$����^�L�{�3�U�.i�����Q��$�:0�c4���5�<��wWT�3�
����
�=�����	�jf/#c�F3���vIg��W��R6�.0�)L	���Q�x�*��[��	��C$�z��%�Z�1�
�F��;c��4l���EqD7�J�I�]�=�:k�Q8�b��m��0��Y�����}��PK:-[�HPK��JObject 3/content.xml�[Ks����W���G�o�T��C�r�����*7DBv@� K��O )H�4���v�U�i���������P������i����W9+H�y����s��o�Y����xQ�|W�J:9�$�=�J,���t��C��E�J,2_�W����^h]fE�#,��mi�r���=�E���5�-]p�*�x�����
>��x���$V(��?M�R������}8c|��Y��z�38�������]L�R&\��-o�%j���M�v�
���A���x�F����k�-����������l���JLR�l�o�^NX��P]���U9'��kn[�1���L�js��\C[����{N$�{~�=G4�<��>������L;�+G�+�k�;fQ\=���^��oq�N������D��3\��Mc���q�9f=�`B�����,��tW�-��E/+�����x���_���m<d�f:K����j���6�:���PcN��U�qJn8�zaI�6�V�
��m�B;� 	I���"���(5���NL�Q�����L��*I����\A6D�����t{�|^�C�+����YX����ZR�iE6�[����b.	�6�(�����Vki��.��NS�2S����	"�W%�i�FF5P[d���@ �A�����.;pT]�T���C�}�������[���<]���r��_n�&�K�0�2����B�55�kBi��-t�x��;�������C�������������S
��FT������2W��%�H:����=�l*�;N6[��jCq���P������������U�nz� ����P����>�m��mI��p�j3���qn����9&S����/}z ���w����arw�y��j����8�+Fy���4�{R����yi^���	�t�W]N;0k�b��v�y����������^G��[�O;����
Z�QR|^3l������\�yo�{ufn6V�8v���yi��k+���_[����4S�c���cx��f-mp,�
s[�s��hn�U$�c�!�dQ�tk�$R9�X��4�@~T��Q}g��nF���%��j�6K�:�U%v�3m�,	lm�I�k���6���P&'{�����pT��+'%Fb��;;Z��)$U1�	�k5m��fYZF��,0q�0hd���*���B���)����w��^F���fj�q8�3��E���n��W�g�����}�/zV_�d���5�e�[�Mie;�����[@/g��)�,I,��.�
��Y</�<��b�x�.r�8�C$�4�
x����08S�:��RG<�3e\�k5u���M��}��kx:#~t������c�q�k���m8)zY���.�����s2�K,x�����1i{��;,��������~g�F�9��9�`���X��
�����T��w���d7k����T������g��������/?H��>��3�������������\�urW
���H��M��e9��Mvv��l1*0W]zWV�|�,*E�I�.���O�F��,����a�v���r���v�wG�����C=��f���f�S������A~���NpW���7�?��Dp����^z���
eP�,��;jL^���m�|�%{
^9��x����q���;\��BC@5��O������k��b�������3a�kue�(=Yn�;1}���o}�����^�w"�Z��8�?��H0�-�����b?�|;,�IW&�
`M��5R��q.�Xu%�Fr"d}�W�`$FI�xv�4���E#���d���0��)z�K�(�3L���H�L������������qZ�l�Gl>�3�$fc��0������iY��#��0���X�Qd����$f�0C?"`���/�{V�1��$��.�����?��������'B�!+��0�����t�i`�����?R���fci���M��+��8	�4S�Cb6������� 4Dv�iz��y�?��a���l�7��y��g^��=�4=V�s;f��3f�+�|��)F�����.H�K-u�_A��PKk��*K2PK��JObject 3/styles.xml��]n� ���
c�����y�
�0�)���*V����Q���p���e�@���a'a@8�M~~�Go��|:�����J��%\GJ3��I�\���
����DA���_E���
������-��5��xfwZt��laW]Id|�3;�����W<*�"�������_Ex���cb��B6 ����-a�q]/��*#s0�8+���J�wyH��QC�=��yP�!�=_�G�U�#��]�X��;�B�z!���Y��6��$/`�vhs�I5��q��� �{��LDD�y�+-����
$���["�����s�l{�-{l��tEYUw�)�L��
�����>�,����"M����w�9[����\6#(����,PK$)W�}sPK��JObject 3/meta.xml��AO�0���
R�
C�4�&��E��4�,[����������������N���m�#�^]�$@-�T�.���S�@��Ma�{%�I#��[t<���gK�$����^�L�{�3�U�.i�����Q��$�:0�c4���5�<��wWT�3�
����
�=�����	�jf/#c�F3���vIg��W��R6�.0�)L	���Q�x�*��[��	��C$�z��%�Z�1�
�F��;c��4l���EqD7�J�I�]�=�:k�Q8�b��m��0��Y�����}��PK:-[�HPK��JObject 4/content.xml�Z�o�8��"�����YX`fv�e��{+[It#[^Ii��������N�����)��)�?R$����SA'/�����?��\f,'��i����$���?��vK2��Yv(p)���~N@�K��4=�r�� bY�������e#����Z�Y�L�kf[Z��*�x/d�f�f�lK��
+^�����'A�-��
Ire������t/e�t���8;�3�w�����w[����:p�����+e��g���X���)^���Pl0
��UT��npF��n@�����2�a><�an�H�o�$q�������.x1T����*�������c��J�\Pmn�ys����.�������e��Z�Y���.p8�E�i��
qC p�v�,��G���/�f{\���|��!������*7=�\�+�e�vx��h�m{Y���]�6�;����`N�������|��������LW�������Mu�is�Z�|�0'�wDU�8��!�X���M���V�
���U��dp	I���"�+S���dV'��(�O�z�+�IM��V�@.';"!����^�=g�W���
��?��� ���0����jW�.��`1Qm���P�k���{����4e+3P!�TL�Q�����@m�-�r� ��J?�d��Yv �QykSn�O�����w��{��nv���97�%�zi��S=��M����{K(mx�����k0��=@{7;@����()3z���'y�K'�����"*�����[�U{Yr����J����d�e�d��*w�:a��r��q�������W�*}w����(:;m0�������/Hv�
��6c��g�z_��0�bgh|���D���w7�og��0
7���
U��w:�s�a����jKC|8�\
���K�����L%q_u�v`���;�J�T���gw��}��^oU�v�*�����$�f8�����!&�;����"���\olX~n	m�zeR�V���M~���cO�T-��!�{���3�5��cAT����Q$D�9<V������.���+$�
LP
���y�qV9���P/�����Du��`�7,'{T����[o�\KW�X6W�c��t��d��v���W�29Y��j�{���|����X@�UR���Z'P���t�.B��p�&X:jYx�������~��*��FF;�x/#�]�S35�8f��y��
q��o0�}��������S/��g0�!af8��c��7[,,�}[��MfQ^'i�1���-xz.3�89$��� ��f~\(��+1`�_��f~��k�SsN;_����~j�5 V�F����C��s�q�[�5��8�{YW7�@�e\����~�k�k^�$�1/��pe������e��s��3z��s|��4��PI�PJ������N���R�Cs��C�Z� ����/=V~y�/}�w �o���(�o^����%������"Y�V��l$(�^���.g�Q��j����{fQ)zL�w���|�T3:V�I���j�P5�6`�J-D�iQ��6{W+�gr*��1���]���q������`� ����V;�}.��4{��g/����h3���l=�B���H�����L�O����c�����
���������M�!�4
��\z$�;���NE�=�������kq�$� ��6�Cy��-��<�ibY������!�=��%5��R3�R3G$O���H��4J�v�}W�H�qpa���7�i��+�IW�4���%o�u���0/I}��)z$]���������fb,-`�(���)z����'�]���q0��L�1���.S��$��]������`j��t��0�� *z,]Q�C��?"`�8��u������0�;���*��yp1'*���]�����#�����:�H�����o���PKM��(�'+PK��JObject 4/styles.xml��]n� ���
c�����y�
�0�)���*V����Q���p���e�@���a'a@8�M~~�Go��|:�����J��%\GJ3��I�\���
����DA���_E���
������-��5��xfwZt��laW]Id|�3;�����W<*�"�������_Ex���cb��B6 ����-a�q]/��*#s0�8+���J�wyH��QC�=��yP�!�=_�G�U�#��]�X��;�B�z!���Y��6��$/`�vhs�I5��q��� �{��LDD�y�+-����
$���["�����s�l{�-{l��tEYUw�)�L��
�����>�,����"M����w�9[����\6#(����,PK$)W�}sPK��JObject 4/meta.xml��AO�0���
R�
C�4�&��E��4�,[����������������N���m�#�^]�$@-�T�.���S�@��Ma�{%�I#��[t<���gK�$����^�L�{�3�U�.i�����Q��$�:0�c4���5�<��wWT�3�
����
�=�����	�jf/#c�F3���vIg��W��R6�.0�)L	���Q�x�*��[��	��C$�z��%�Z�1�
�F��;c��4l���EqD7�J�I�]�=�:k�Q8�b��m��0��Y�����}��PK:-[�HPK��Jsettings.xml���s�8��������G��abw�r�5�fl'���#���DhI���	cw6�����	���W�G��=�X��y����p]�}�nz�|o}�q��9u�v�	<��)A)��l��\����n#�T��x m����
��>mo��W�}k��o�Za��G(����Yksw���|NyM�O?5���Ebg6���v��o5�N>	M�����������O�*���4��#���6i�(���f��{>��J:c0@��[��j����+k�>o�y��
�����U�4�^�{������2������Y^��G�&�.<���a�,m���%�y<���M�)��K�D��*O#���S�#r���C���@�����kq��%Uz+|K�{���S�7M�SP�
��\6�U����X�(��A(��ROx��P���������+���	_d��[L|�o������RR�Z��T/P)�*���M�J�+:�',H����]4df���f��\����LU!VI��A�A�j�TC��j
��F�*�����btY
LO�����iZ���iM��1M����'5Mk��4�i��4���i�����lF�7�{-�
G c3"2���w����$�O�F.��[Q�>�?Cd@�5�&���� p���2�DPB�3*S�	�|�Q�r�:B�"!�0�L���q�{R0y6��rE�v=���+ .3F&K��c@�Z~	�&ko�LN yDVbd��?�1�
�]���������1c�z�3��Na'�������3����t�"�t��.�X~�H�d�+���� *�l_�? A��: �M�k>2�9���7�F�;�L��O �PR�o������vR�+�K3���GK"��"��}!�����,M�����|� �Tx�=:�I���o4�+�����K4���`Sa���&���}�b� \|���$��D���2�sFT��\./5�������R�:	<��ox�,�Z~�~47���>?K�H���e���l�n��3Y�n��?��i��x��H��!Mv��������F��d��+��lgR�&HvBgD�i��r���z��k"g����Y�}���>b(M����,��e�(}������x�*��k��8������?PK_�5�mx/PK��J
styles.xml�Ymo�6��_!(C�c$9iR���a��mP���>2%�D�����~GR�������K�$�;��������ywDH������G��'����������+��4&���UN
���3"=.��.��J+�%���D�T��%)����^����Q6W��������������w6��t"�n����m�������</��=+�->���R�*v�������,���e`V��WV�T�����(p��(<�>�m�TT�-���
�Z
"��q9OQ[�_w�����&h��X��3���Y2?T���l��v��/�7�h�{��W"����v��-g�������T-`��������[��Q�NPED����
�<#
pQD�t�;������<��B5���/;`g���V�l:U���f"IF�`�YiI��(��t����/���n���.���������Ln�lr6���c��_�`4������mQ��2RA��rG�� J�b��;���������&�{��},(-2b�\r�rk���������
.�Q~��o�;��<U�_�����p�
3���x��f���S�\���Y��W��l���$�5(&��^b�3��-*$�P�!�h��K�P�p�;���9-��6d(g��dU�W������L��*��>�����#\Le��:}�y�Vj�n����z]�������oi_k�������s<a(�q��z�&8��,���
����t��[����D�{��(z�w�����`���cq��,��;M�P��mCXY��L�bl������PZt+��N*y�VtA�6����_�/�"!���(�v�V>�L��6�����S��J=���ls���S�"&RG.�J�wCv�;��b�E#4�h�7[I�8����9�(�/�'"
��������2bq^�����������&�m����D�n�	���G9- ��@.�U�4��lt���d��m��oCw�;� ��{��<�`��r�����a[>����S1q>��i�*$'�m�?�[���C|���}.%�T^� 5It��!���j��A��f�3h	�q��x�P{a�aV����d���	\���u��aoLgB]N�B���'����p��m���7�j<�-��f��MU���<� ���������E�muOy$���h?Q�T�v5���	[�,FrF�	��&���+����C�:������������*v���']?��2����;�?��Fh��a4+�����MJ��rx����(��4�wK@�A���A����q��_�~,r������(u��C���3�T��/���A��Rup� ��F7���=�T�X��?�m.�@����-����s)����:L_���	��b����6,�"�5FR���5�;{���_s��p�������U�9��Rh�G���|1�������-��N�g����������@J���E&��� �?���H��~��_CRu<#�������.m�x0y��9��
y����t��jg���g��\����o�%��7���WA��){$�@���p*�a��������>K=�f��q��Zs���GM<>D���k�ff]7����
�B�����bm�?���3e���;4Wd�����!����M]���p��"�������/�3�����bu~�=>]��[LyAh����
�s�(���l�����PK)f�c��PK��Jmanifest.rdf���n�0��<�e��@/r(��j��5�X/������VQ�������F3�����a�����T4c)%�Hh��+:�.���:���+��j���*�wn*9_��-7l���(x��<O�"��8qH���	�Bi��|9��	fWQt���y� =��:���
a�R��� ��@�	L��t��NK�3��Q9�����`����<`�+�������^����\��|�hz�czu����#�`�2�O��;y���.�����vDl@��g�����UG�PK��h��PK��JBv��AAmeta.xml<?xml version="1.0" encoding="UTF-8"?>
<office:document-meta xmlns:office="urn:oasis:names:tc:opendocument:xmlns:office:1.0" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:meta="urn:oasis:names:tc:opendocument:xmlns:meta:1.0" xmlns:ooo="http://openoffice.org/2004/office" xmlns:grddl="http://www.w3.org/2003/g/data-view#" office:version="1.2"><office:meta><meta:creation-date>2017-05-30T13:22:39.89</meta:creation-date><dc:date>2017-06-02T11:32:46.55</dc:date><meta:editing-duration>P2DT1H16M24S</meta:editing-duration><meta:editing-cycles>35</meta:editing-cycles><meta:generator>OpenOffice/4.0.1$Win32 OpenOffice.org_project/401m5$Build-9714</meta:generator><meta:document-statistic meta:table-count="4" meta:cell-count="377" meta:object-count="4"/></office:meta></office:document-meta>PK��JMETA-INF/manifest.xml��Ao� ����]*���ar����U���;O?;L\x���dJ�n���FQn��{���o��].{�<��B���gg$�u#TW���W�Gr9?��L�,V�A���}���Uif����V�+=�j4w=(���Wk��� '��d��
	��7�mt�`)��	)8C�MT����rf��|Ir��:)����&���PvW��U+:g��6��s����P��	�~o����m�T��9�������GX"
�;���	�s�A�m��R��J��W�dx����f�?��P]�AM����v�����G���V��a8���{������t��~/���q�X��=.���p���-��=����-t�p��bBZ��a��<��o�<�u�G���^�q�QElS���5��bJ��2�)�AM)�5��L�Yo�`�K�iO_T��e>jR���3���h�PK<0���U
PK��J�l9�..mimetypePK��JTConfigurations2/images/Bitmaps/PK��J'�Configurations2/accelerator/current.xmlPK��J�Configurations2/floater/PK��JConfigurations2/menubar/PK��JTConfigurations2/progressbar/PK��J�Configurations2/popupmenu/PK��J�Configurations2/statusbar/PK��J�Configurations2/toolbar/PK��J4Configurations2/toolpanel/PK��J]]��
.lObject 1/content.xmlPK��J$)W�}s�
Object 1/styles.xmlPK��J:-[�HvObject 1/meta.xmlPK��J�(�69g�
ObjectReplacements/Object 1PK��JD[d���aQObjectReplacements/Object 2PK��JH��Z*�`yObjectReplacements/Object 3PK��Jc��X�}r�&ObjectReplacements/Object 4PK��Jw�{A���/Thumbnails/thumbnail.pngPK��Ji�i�y#Lcontent.xmlPK��Jm�l�e*�aObject 2/content.xmlPK��J$)W�}s�iObject 2/styles.xmlPK��J:-[�HukObject 2/meta.xmlPK��Jk��*K2�lObject 3/content.xmlPK��J$)W�}suObject 3/styles.xmlPK��J:-[�H�vObject 3/meta.xmlPK��JM��(�'+9xObject 4/content.xmlPK��J$)W�}s�Object 4/styles.xmlPK��J:-[�H��Object 4/meta.xmlPK��J_�5�mx/%�settings.xmlPK��J)f�c��
��styles.xmlPK��J��h����manifest.rdfPK��JBv��AA	�meta.xmlPK��J<0���U
p�META-INF/manifest.xmlPK!!�s�
patchDescription.odtapplication/octet-stream; name=patchDescription.odtDownload
#2Sokolov Yura
y.sokolov@postgrespro.ru
In reply to: Jim Van Fleet (#1)
Re: HACKERS[PROPOSAL] split ProcArrayLock into multiple parts

<div dir='auto'><div dir="auto">Hi, Jim.</div><div dir="auto"><br></div><div>How do you ensure of transaction order?<div dir="auto"><br></div><div dir="auto">Example:</div><div dir="auto">- you lock shard A and gather info. You find transaction T1 in-progress.</div><div dir="auto">- Then you unlock shard A.</div><div dir="auto">- T1 completes. T2, that depends on T1, also completes. But T2 was on shard B.</div><div dir="auto">- you lock shard B, and gather info from.</div><div dir="auto">- You didn't saw T2 as in progress, so you will lookup into clog then and will find it as commited.</div><div dir="auto"><br></div><div dir="auto">Now you see T2 as commited, but T1 as in-progress - clear violation of transaction order.</div><div dir="auto"><br></div><div dir="auto">Probably you've already solved this issue. If so it would be great to learn the solution.</div><br><div class="gmail_extra"><br><div class="gmail_quote">5 июня 2017 г. 10:30 PM пользователь Jim Van Fleet &lt;vanfleet@us.ibm.com&gt; написал:<br type="attribution"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><font size="2" face="sans-serif">Hi,</font><br><br><font size="2" face="sans-serif">I have been experimenting with splitting
&nbsp;the ProcArrayLock into parts. &nbsp;That is, to Acquire the ProcArrayLock
in shared mode, it is only necessary to acquire one of the parts in shared
mode; to acquire the lock in exclusive mode, all of the parts must be acquired
in exclusive mode. For those interested, I have attached a design description
of the change.</font><br><br><font size="2" face="sans-serif">This approach has been quite successful
on large systems with the hammerdb benchmark.</font><font size="3" face="Times New Roman">With a prototype based on 10 master source and running on power8 (model
8335-GCA with 2sockets, 20 core)</font><br><font size="3" face="Times New Roman">&nbsp;hammerdb &nbsp;improved by
16%; On intel (Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz, 2 socket, 44
core) with 9.6 base and prototype hammerdb improved by 4%. (attached is
a set of spreadsheets for power8.</font><br><br><font size="3" face="Times New Roman">The down side is that on smaller
configurations (single socket) where there is less "lock thrashing"
in the storage subsystem and there are multiple Lwlocks to take for an
exclusive acquire, there is a decided downturn in performance. On &nbsp;hammerdb,
the prototype was 6% worse than the base on a single socket power configuration.</font><br><br><font size="3" face="Times New Roman">If there is interest in this approach,
I will submit a patch.</font><br><br><font size="3" face="Times New Roman">Jim Van Fleet</font><br><br><br></blockquote></div><br></div></div></div>

#3Sokolov Yura
y.sokolov@postgrespro.ru
In reply to: Sokolov Yura (#2)
Re: HACKERS[PROPOSAL] split ProcArrayLock into multiple parts

<div dir='auto'><span style="font-family: sans-serif;">Excuse me, Jim.</span><div dir="auto" style="font-family: sans-serif;"><br></div><div dir="auto" style="font-family: sans-serif;">I was tired and misunderstand proposal: I thought of ProcArray sharding, but proposal is about ProcArrayLock sharding.</div><div dir="auto" style="font-family: sans-serif;"><br></div><div dir="auto" style="font-family: sans-serif;">BTW, I just posted improvement to LWLock:</div><div dir="auto" style="font-family: sans-serif;"><br></div><div dir="auto"><font face="sans-serif">/messages/by-id/2968c0be065baab8865c4c95de3f435c@postgrespro.ru&lt;/font&gt;&lt;br&gt;&lt;/div&gt;&lt;div dir="auto"><font face="sans-serif"><br></font></div><div dir="auto"><font face="sans-serif">Would you mind to test against that and together with that?</font></div></div><div class="gmail_extra"><br><div class="gmail_quote">5 июня 2017 г. 11:11 PM пользователь Sokolov Yura &lt;y.sokolov@postgrespro.ru&gt; написал:<br type="attribution"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><div dir="auto">Hi, Jim.</div><div dir="auto"><br /></div><div>How do you ensure of transaction order?<div dir="auto"><br /></div><div dir="auto">Example:</div><div dir="auto">- you lock shard A and gather info. You find transaction T1 in-progress.</div><div dir="auto">- Then you unlock shard A.</div><div dir="auto">- T1 completes. T2, that depends on T1, also completes. But T2 was on shard B.</div><div dir="auto">- you lock shard B, and gather info from.</div><div dir="auto">- You didn&#39;t saw T2 as in progress, so you will lookup into clog then and will find it as commited.</div><div dir="auto"><br /></div><div dir="auto">Now you see T2 as commited, but T1 as in-progress - clear violation of transaction order.</div><div dir="auto"><br /></div><div dir="auto">Probably you&#39;ve already solved this issue. If so it would be great to learn the solution.</div><br /><div><br /><div class="elided-text">5 июня 2017 г. 10:30 PM пользователь Jim Van Fleet &lt;vanfleet&#64;us.ibm.com&gt; написал:<br /><blockquote style="margin:0 0 0 0.8ex;border-left:1px #ccc solid;padding-left:1ex"><font size="2" face="sans-serif">Hi,</font><br /><br /><font size="2" face="sans-serif">I have been experimenting with splitting
 the ProcArrayLock into parts.  That is, to Acquire the ProcArrayLock
in shared mode, it is only necessary to acquire one of the parts in shared
mode; to acquire the lock in exclusive mode, all of the parts must be acquired
in exclusive mode. For those interested, I have attached a design description
of the change.</font><br /><br /><font size="2" face="sans-serif">This approach has been quite successful
on large systems with the hammerdb benchmark.</font><font size="3" face="Times New Roman">With a prototype based on 10 master source and running on power8 (model
8335-GCA with 2sockets, 20 core)</font><br /><font size="3" face="Times New Roman"> hammerdb  improved by
16%; On intel (Intel(R) Xeon(R) CPU E5-2699 v4 &#64; 2.20GHz, 2 socket, 44
core) with 9.6 base and prototype hammerdb improved by 4%. (attached is
a set of spreadsheets for power8.</font><br /><br /><font size="3" face="Times New Roman">The down side is that on smaller
configurations (single socket) where there is less &#34;lock thrashing&#34;
in the storage subsystem and there are multiple Lwlocks to take for an
exclusive acquire, there is a decided downturn in performance. On  hammerdb,
the prototype was 6% worse than the base on a single socket power configuration.</font><br /><br /><font size="3" face="Times New Roman">If there is interest in this approach,
I will submit a patch.</font><br /><br /><font size="3" face="Times New Roman">Jim Van Fleet</font><br /><br /><br /></blockquote></div><br /></div></div></div></blockquote></div><br></div>

#4Jim Van Fleet
vanfleet@us.ibm.com
In reply to: Sokolov Yura (#3)
Re: HACKERS[PROPOSAL] split ProcArrayLock into multiple parts

NP, Sokolov --

pgsql-hackers-owner@postgresql.org wrote on 06/05/2017 03:26:46 PM:

From: Sokolov Yura <y.sokolov@postgrespro.ru>
To: Jim Van Fleet <vanfleet@us.ibm.com>
Cc: pgsql-hackers@postgresql.org
Date: 06/05/2017 03:28 PM
Subject: Re: [HACKERS] HACKERS[PROPOSAL] split ProcArrayLock into
multiple parts
Sent by: pgsql-hackers-owner@postgresql.org

Excuse me, Jim.

I was tired and misunderstand proposal: I thought of ProcArray
sharding, but proposal is about ProcArrayLock sharding.

BTW, I just posted improvement to LWLock:

https://www.postgresql.org/message-id/
2968c0be065baab8865c4c95de3f435c%40postgrespro.ru

Would you mind to test against that and together with that?

I will give them a try ..

Jim

#5Jim Van Fleet
vanfleet@us.ibm.com
In reply to: Sokolov Yura (#3)
Re: HACKERS[PROPOSAL] split ProcArrayLock into multiple parts

Hi Sokolov --

I tried your patch. I only had time for doing a few points on power8.
pgbench rw on two sockets is awesome! Keeps getting more throughput as
threads are added -- in contrast to base and my prototype. I did not run
single socket pgbench.

Hammerdb, 1 socket was in the same ballpark as the base, but slightly
lower. 2 socket was also in the same ballpark as the base, again slightly
lower. I did not do a series of points (just one at the previous "sweet
spot"), so the "final" results may be better, The ProcArrayLock multiple
parts was lower except in two socket case. The performance data I
collected for your patch on hammerdb showed the same sort of issues as
the base.

I don't see much point in combining the two because of the ProcArrayLock
down side -- that is, single socket. poor performance. Unless we could
come up with some heuristic to use one part on light loads and two on
heavy (and still stay correct), then I don't see it ... With the
combination, what I think we would see is awesome pgbench rw, awesome
hammerdb 2 socket performance, and degraded single socket hammerdb.

Jim

From: Sokolov Yura <y.sokolov@postgrespro.ru>
To: Jim Van Fleet <vanfleet@us.ibm.com>
Cc: pgsql-hackers@postgresql.org
Date: 06/05/2017 03:28 PM
Subject: Re: [HACKERS] HACKERS[PROPOSAL] split ProcArrayLock into
multiple parts
Sent by: pgsql-hackers-owner@postgresql.org

Excuse me, Jim.

I was tired and misunderstand proposal: I thought of ProcArray sharding,
but proposal is about ProcArrayLock sharding.

BTW, I just posted improvement to LWLock:

/messages/by-id/2968c0be065baab8865c4c95de3f435c@postgrespro.ru

Would you mind to test against that and together with that?

5 июня 2017 г. 11:11 PM пользователь Sokolov Yura
<y.sokolov@postgrespro.ru> написал:
Hi, Jim.

How do you ensure of transaction order?

Example:
- you lock shard A and gather info. You find transaction T1 in-progress.
- Then you unlock shard A.
- T1 completes. T2, that depends on T1, also completes. But T2 was on
shard B.
- you lock shard B, and gather info from.
- You didn't saw T2 as in progress, so you will lookup into clog then and
will find it as commited.

Now you see T2 as commited, but T1 as in-progress - clear violation of
transaction order.

Probably you've already solved this issue. If so it would be great to
learn the solution.

5 июня 2017 г. 10:30 PM пользователь Jim Van Fleet <vanfleet@us.ibm.com>
написал:
Hi,

I have been experimenting with splitting the ProcArrayLock into parts.
That is, to Acquire the ProcArrayLock in shared mode, it is only
necessary to acquire one of the parts in shared mode; to acquire the lock
in exclusive mode, all of the parts must be acquired in exclusive mode.
For those interested, I have attached a design description of the change.

This approach has been quite successful on large systems with the hammerdb
benchmark.With a prototype based on 10 master source and running on power8
(model 8335-GCA with 2sockets, 20 core)
hammerdb improved by 16%; On intel (Intel(R) Xeon(R) CPU E5-2699 v4 @
2.20GHz, 2 socket, 44 core) with 9.6 base and prototype hammerdb improved
by 4%. (attached is a set of spreadsheets for power8.

The down side is that on smaller configurations (single socket) where
there is less "lock thrashing" in the storage subsystem and there are
multiple Lwlocks to take for an exclusive acquire, there is a decided
downturn in performance. On hammerdb, the prototype was 6% worse than the
base on a single socket power configuration.

If there is interest in this approach, I will submit a patch.

Jim Van Fleet

#6Amit Kapila
amit.kapila16@gmail.com
In reply to: Jim Van Fleet (#1)
Re: HACKERS[PROPOSAL] split ProcArrayLock into multiple parts

On Tue, Jun 6, 2017 at 1:00 AM, Jim Van Fleet <vanfleet@us.ibm.com> wrote:

Hi,

I have been experimenting with splitting the ProcArrayLock into parts.
That is, to Acquire the ProcArrayLock in shared mode, it is only necessary
to acquire one of the parts in shared mode; to acquire the lock in exclusive
mode, all of the parts must be acquired in exclusive mode. For those
interested, I have attached a design description of the change.

This approach has been quite successful on large systems with the hammerdb
benchmark.With a prototype based on 10 master source and running on power8
(model 8335-GCA with 2sockets, 20 core)
hammerdb improved by 16%; On intel (Intel(R) Xeon(R) CPU E5-2699 v4 @
2.20GHz, 2 socket, 44 core) with 9.6 base and prototype hammerdb improved by
4%. (attached is a set of spreadsheets for power8.

The down side is that on smaller configurations (single socket) where there
is less "lock thrashing" in the storage subsystem and there are multiple
Lwlocks to take for an exclusive acquire, there is a decided downturn in
performance. On hammerdb, the prototype was 6% worse than the base on a
single socket power configuration.

I think any patch having 6% regression on one machine configuration
and 16% improvement on another machine configuration is not a net win.
However, if there is a way to address the regression, then it will
look much attractive.

If there is interest in this approach, I will submit a patch.

The basic idea is clear from your description, but it will be better
if you share the patch as well. It will not only help people to
review and provide you feedback but also allow them to test and see if
they can reproduce the numbers you have mentioned in the mail.

There is some related work which was previously proposed in this area
("Cache the snapshot") [1]/messages/by-id/CAD__OuiwEi5sHe2wwQCK36Ac9QMhvJuqG3CfPN+OFCMb7rdruQ@mail.gmail.com and it claims to reduce contention around
ProcArrayLock. I am not sure if that patch still applies, however, if
you find it relevant and you are interested in evaluating the same,
then we can request the author to post a rebased version if it doesn't
apply.

[1]: /messages/by-id/CAD__OuiwEi5sHe2wwQCK36Ac9QMhvJuqG3CfPN+OFCMb7rdruQ@mail.gmail.com

--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#7Jim Van Fleet
vanfleet@us.ibm.com
In reply to: Amit Kapila (#6)
Re: HACKERS[PROPOSAL] split ProcArrayLock into multiple parts

Amit Kapila <amit.kapila16@gmail.com> wrote on 06/07/2017 07:34:06 AM:

...

The down side is that on smaller configurations (single socket) where

there

is less "lock thrashing" in the storage subsystem and there are

multiple

Lwlocks to take for an exclusive acquire, there is a decided downturn

in

performance. On hammerdb, the prototype was 6% worse than the base on

a

single socket power configuration.

I think any patch having 6% regression on one machine configuration
and 16% improvement on another machine configuration is not a net win.
However, if there is a way to address the regression, then it will
look much attractive.

I have to agree.

If there is interest in this approach, I will submit a patch.

The basic idea is clear from your description, but it will be better
if you share the patch as well. It will not only help people to
review and provide you feedback but also allow them to test and see if
they can reproduce the numbers you have mentioned in the mail.

OK -- would love the feedback and any suggestions on how to mitigate the
low end problems.

There is some related work which was previously proposed in this area
("Cache the snapshot") [1] and it claims to reduce contention around
ProcArrayLock. I am not sure if that patch still applies, however, if
you find it relevant and you are interested in evaluating the same,
then we can request the author to post a rebased version if it doesn't
apply.

Sokolov Yura has a patch which, to me, looks good for pgbench rw
performance. Does not do so well with hammerdb (about the same as base)
on single socket and two socket.

Show quoted text

[1] - https://www.postgresql.org/message-id/
CAD__OuiwEi5sHe2wwQCK36Ac9QMhvJuqG3CfPN%2BOFCMb7rdruQ%40mail.gmail.com

--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

#8Robert Haas
robertmhaas@gmail.com
In reply to: Jim Van Fleet (#7)
Re: HACKERS[PROPOSAL] split ProcArrayLock into multiple parts

On Wed, Jun 7, 2017 at 12:29 PM, Jim Van Fleet <vanfleet@us.ibm.com> wrote:

The basic idea is clear from your description, but it will be better
if you share the patch as well. It will not only help people to
review and provide you feedback but also allow them to test and see if
they can reproduce the numbers you have mentioned in the mail.

OK -- would love the feedback and any suggestions on how to mitigate the low
end problems.

Did you intend to attach a patch?

Sokolov Yura has a patch which, to me, looks good for pgbench rw
performance. Does not do so well with hammerdb (about the same as base) on
single socket and two socket.

Any idea why? I think we will have to understand *why* certain things
help in some situations and not others, not just *that* they do, in
order to come up with a good solution to this problem.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#9Jim Van Fleet
vanfleet@us.ibm.com
In reply to: Robert Haas (#8)
Re: HACKERS[PROPOSAL] split ProcArrayLock into multiple parts

Robert Haas <robertmhaas@gmail.com> wrote on 06/07/2017 12:12:02 PM:

OK -- would love the feedback and any suggestions on how to mitigate

the low

end problems.

Did you intend to attach a patch?

Yes I do -- tomorrow or Thursday -- needs a little cleaning up ...

Sokolov Yura has a patch which, to me, looks good for pgbench rw
performance. Does not do so well with hammerdb (about the same as

base) on

single socket and two socket.

Any idea why? I think we will have to understand *why* certain things
help in some situations and not others, not just *that* they do, in
order to come up with a good solution to this problem.

Looking at the data now -- LWLockAquire philosophy is different -- at
first glance I would have guessed "about the same" as the base, but I can
not yet explain why we have super pgbench rw performance and "the same"
hammerdb performance.

Show quoted text

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

#10Sokolov Yura
funny.falcon@postgrespro.ru
In reply to: Jim Van Fleet (#9)
Re: HACKERS[PROPOSAL] split ProcArrayLock into multiple parts

Good day Robert, Jim, and everyone.

On 2017-06-08 00:06, Jim Van Fleet wrote:

Robert Haas <robertmhaas@gmail.com> wrote on 06/07/2017 12:12:02 PM:

OK -- would love the feedback and any suggestions on how to

mitigate the low

end problems.

Did you intend to attach a patch?

Yes I do -- tomorrow or Thursday -- needs a little cleaning up ...

Sokolov Yura has a patch which, to me, looks good for pgbench rw
performance. Does not do so well with hammerdb (about the same as

base) on

single socket and two socket.

Any idea why? I think we will have to understand *why* certain

things

help in some situations and not others, not just *that* they do, in
order to come up with a good solution to this problem.

My patch improves acquiring contended/blocking LWLock on NUMA cause:
a. patched procedure generates much lesser writes, especially because
taking WaitListLock is unified with acquiring the lock itself.
Access to modified memory is very expensive on NUMA, so less writes
leads to less wasted time.
b. it spins several time on lock->state in attempts to acquire lock
before starting attempts to queue self to wait list. It is really the
cause of some speedup. Without spinning patch just removes
degradation on contention.
I don't know why spinning doesn't improves single socket performance
though :-) Probably still because all algorithmic overhead (system
calls, sleeping and awakening process) is not too expensive until
NUMA is involved.

Looking at the data now -- LWLockAquire philosophy is different -- at
first glance I would have guessed "about the same" as the base, but I
can not yet explain why we have super pgbench rw performance and "the
same" hammerdb performance

My patch improves only blocking contention, ie when a lot of EXCLUSIVE
locks are involved. pgbench rw generates a lot of write traffic, so
there is a lot of contention and waiting on WALInsertLocks (in
XLogInsertRecord, and waiting in XLogFlush), WalWriteLock (in
XLogFlush), CLogControlLock (in TransactionIdSetTreeStatus).

The case when SHARED lock is much more common than EXCLUSIVE is not
affected by patch, because SHARED is acquired then on the fast path
in both original and patched version.

So, looks like hammerdb doesn't produce much EXCLUSIVE contention on
LWLocks, so it is not improved with the patch.

Splitting ProcArrayLock helps with acquiring SHARED lock on NUMA in
absence of EXCLUSIVE lock because of the same reason why my patch
improves acquiring of blocking lock: less writes to same memory.
Since every process writes to some one part of ProcArrayLock, there
is a lot less writes to each part of ProcArrayLock, so acquiring
SHARED lock pays lesser for accessing to modified memory on NUMA.

Probably I'm mistaken somewhere.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sokolov Yura aka funny_falcon
Postgres Professional: https://postgrespro.ru
The Russian Postgres Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#11Jim Van Fleet
vanfleet@us.ibm.com
In reply to: Jim Van Fleet (#1)
Re: HACKERS[PROPOSAL] split ProcArrayLock into multiple parts

pgsql-hackers-owner@postgresql.org wrote on 06/07/2017 04:06:57 PM:

...

Did you intend to attach a patch?

Yes I do -- tomorrow or Thursday -- needs a little cleaning up ...

meant Friday

Sokolov Yura has a patch which, to me, looks good for pgbench rw
performance. Does not do so well with hammerdb (about the same

as base) on

single socket and two socket.

Any idea why? I think we will have to understand *why* certain things
help in some situations and not others, not just *that* they do, in
order to come up with a good solution to this problem.

Looking at the data now -- LWLockAquire philosophy is different --
at first glance I would have guessed "about the same" as the base,
but I can not yet explain why we have super pgbench rw performance
and "the same" hammerdb performance.

(data taken from perf cycles when I invoked the performance data gathering
script, generally in the middle of the run)
In hammerdb two socket, the ProcArrayLock is the bottle neck in
LWLockAcquire (called from GetSnapshotData about 75% of the calls to
LWLockAquire). With Sokolov's patch, LWLockAcquire (with LWLockAttemptLock
included) is a little over 9%; pgbench, on the other hand, has
LWLockAquire at 1.3% with GetSnapshotData calling only 11% of the calls to
LWLockAcquire.

What I think that means is that there is no ProcArrayLock bottleneck in
pgbench. GetSnapshotData runs the entire proc chain of PGXACT's so is held
a rather long time. Guessing that the other locks are held a much shorter
time; Sukolov's patch handles the other locks better because of spinning.
We see much more time in LWLockAcquire with hammerdb because of the
spinning -- with the ProcArrayLock, spinning does not help much because of
the longer hold time.

The spin count is relatively high (100/2), so I made it much smaller
(20/2) in the hopes that the spin would still handle the shorter hold time
locks but not be a bother with long hold times.

Running pgbench with 96 users, the thruput was slightly less at 70K tsp vs
75K tps (vs base of 40K tps at 96 threads and peak of 58K at 64 threads);
hammerdb two socket was slightly better (about 3%) than the peak base.

What all this tells me is that LWLockAcquire would (probably) benefit from
some spinning.

Show quoted text

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company