Init connection time grows quadratically
Hello!
I ran some experiments with pgbench to measure the initialization time and found that the time increases quadratically with the number of clients. It was surprising to me and I would like to understand a reason of such behavior.
Some details on how it was done:
1) I used the branch REL_16_STABLE (commit 2caa85f4).
2) The default system configuration was modified (CPU speed control, memory control, network, ram disk). Briefly:
sudo cpupower frequency-set -g performance
sudo cpupower idle-set -D0
sudo swapoff -a
sudo sh -c 'echo 16384 >/proc/sys/net/core/somaxconn'
sudo sh -c 'echo 16384 >/proc/sys/net/core/netdev_max_backlog'
sudo sh -c ‘echo 16384 >/proc/sys/net/ipv4/tcp_max_syn_backlog’
numactl --membind=0 bash
sudo mount -t tmpfs -o rw,size=512G tmpfs /mnt/ramdisk
exit
Hyperthreading and cpu boost were disabled:
echo 0 | sudo tee /sys/devices/system/cpu/cpufreq/boost
Please note: When testing on a fast multi-core server with a large number of clients, when the speed of creation of new connections becomes very high, even with such kernel parameters an error may occur:
pgbench:pgbench: error: connection to server on socket "/tmp/.s.PGSQL.5114" failed: Resource temporarily unavailable
In such case you need to apply the patch 0001-Fix-fast-connection-rate-issue.patch (attached).
3) The server was configured as:
./configure --enable-debug --with-perl --with-icu --enable-depend --enable-tap-tests
4) Build and install on ramdrive:
make -j$(nproc) -s && make install
5) DB initialization:
/mnt/ramdisk/bin/initdb -k -D /mnt/ramdisk/data -U postgres
Add to the postgresql.conf:
huge_pages = off #for the sake of test stability and reproducibility
shared_buffers = 1GB
max_connections = 16384
6) Start command:
a) Start server (e.g on the first numa socket)
/mnt/ramdisk/bin/pg_ctl -w -D /mnt/ramdisk/data start
b) create test database and stop the server
/mnt/ramdisk/bin/psql -U postgres -c 'create database bench'
/mnt/ramdisk/bin/pg_ctl -w -D /mnt/ramdisk/data stop
7) pgbench commands:
Perform the single test sequence (I've got a dual socket server, so the server was running on the first socket while the clients were running on the second one):
export PATH=/mnt/ramdisk/bin:$PATH
export NUMACTL_CLIENT="--physcpubind=96-191 --membind=1"
export NUMACTL_SERVER="--physcpubind=0-95 --membind=0"
export CLIENTS=1024
numactl $NUMACTL_SERVER pg_ctl -w -D /mnt/ramdisk/data start
numactl $NUMACTL_CLIENT pgbench -U postgres -i -s100 bench
numactl $NUMACTL_CLIENT psql -U postgres -d bench -c "checkpoint"
numactl $NUMACTL_CLIENT pgbench -U postgres -c$CLIENTS -j$CLIENTS -t100 -S bench
numactl $NUMACTL_SERVER pg_ctl -m smart -w -D /mnt/ramdisk/data stop
8) Measurements & Results
Before the measurements I rebooted host machine and configured the host as described above. After that I ran a script that did 30 measurements of init connection time per a given number of clients, average time and standard deviation were also calculated.
The measurements results are presented as graph an in table form:
Number of clientsAverage init time, ms1024~435 +-202048~1062 +-204096~3284 +-408192~11617 +-12016384~43391 +-230
9) The Question
It turned out that the results correspond to a quadratic dependence like y ~ 0.0002x^2 where x is a number of clients and y is init time (ms).
Here there is a question: is it expected behavior or a bug? What do you think? I appreciate any comments and opinions.
--
Best regards,
Alexander Potapov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
Attachments:
3BDBA-68359800-F-249DA480image/png; name=3BDBA-68359800-F-249DA480Download
�PNG
IHDR [ 5 ���/ pHYs � ��+ IDATx�����+y���O�NPTMSi�����kNN����3��==3�Rg���>��O����0�4r{W:X#��}b��J%����i�t����OuAAA&Pu�As��~9�y�� l���_�/��JG����?��/V���k L��$mmm] ,����] ,"� ��� > l �[ �� ��� > l �[ �� ��� > l �[ �� ��� > l �[ �� ��� > l �[ �� ��� :8*�,��N��LG%'�����T ��.���r�W����. 0�*9Y�R��f,�t���d*�I+j4��^�<z1�YY�Q"W�c��D^��+��SJ��F����~N�uQ���*9K�R����+�;������K�@�Tl�u�����i����}�:W��b�������r���}q�J%�u��f|���#*d�U�
_�� � � ��������M��v~i��~}�P�h�q�$����e��}��,�*y�������W�)�,(�s��$�4e�U��{�>�7���5����r����\�Ty'�|�����YY�~�r����E6�������;��J�������l��i��|���J*9�����A7B �d��JD$UO5��g���[�D^���w�I�F@R�v��p��VURD��vd��C�=������]>�����`��}8�oy�zT:�r�;GGe[2w��VNL�l��&|t*97d���l�#�fd��$qg 0!�N�S�\8�|>�n���*7!i�M�)sT�w��
C�fA�����
m����C�d*�=l+�6C�:o�tv������+�FZ��Su��mz?:��������:�j����e5�T���I�OKWEI=�;JF��Je9Q�s�;[ ����`K����a��z�����Mo�]���MR��l7����1�����C���`��6 �iURH�3K������]Z�D^���_BY5Y%g��������T��M�1|w�������{��)*Uw5���p<�|&&Sj��z��=����l&�?��������QL%;&�0�����-`npg pM�
IK���%����s�=��Q��:U1�{b�p<�X5�B)+k���*:*�VD��o3�wd
}��t7�1����F;��{��7[�f�������p���h���]���
UG���e���gBJ%JZ��>�����t�_Qw�"�t��'�?W�������TNtX��y����T=u��l�I�`�- �5u�(X�~qW��/��M��I�,e���M4�U*4j�kL�Q�����Ue���5W76�T*��F;�v��������C�v��%4�������l3�Ls��z��������Qx�6;���z0����C�hZ�hk���Z�t��I1b����������@�q�_+`�� ����U�,�Uq{��j����tLj����3�xQ]�16��Em�=��u'HjtO,y�V<S5Y�j���K�S�g��;4��a�8�&s�u�f�L��FX4[cX�DD��h+�����:���gC�IYu�3��9A� L��.q��2 �5*R��o�FT�#eKU�tp���B(���j��qk���f������j*��������s�Z���_e����d��:f��w������q�������D%��:������������ �alk�,�q�h����u=��>�+���~m]�g�k~��r��wA�dH�N1�*�F��j����������u��q���.��~v�S��D����@��v���f# LH}v=����>x�/��K%����u�6e��JNV��p�gxy
��#�{�)���>NJ�g��w���N;f�kL_n�2}f�k7fJ���%�NX��4��]m=���K<��-�pt����0+�u�������d��I�Ocv�����1K��T��q3"ks�;[ ��q��*�=;�ww���#�g�����k<���U����%��*��R���{#�H��+K)iY���a���|���2��c2��y9����d�������zB�5��q��hZ���?�����,�F}�Y}6���D$�l�v]��:���Im���}2���:_�g,�./�zYG-�+Z���{|�Su��}���r||\���
� 0c���]L�;�b��]��899�! h�xhpYG<<wvTU��nks�n� �%��-�����t�<C�������,�1\=����� ���� |B� � ��- �a |@� >�(gYJ�?����dY9U�P L�� ���R�`w/0c����~�Q��]%�q��[S1�T[s�������Z�Y�v>