PATCH: pgbench - option to build using ppoll() for larger connection counts
Hackorum builds and tests every patch posted to the lists, not only commitfest submissions. This is Hackorum's own CI rather than the PostgreSQL project's, and it is still under testing - please report anything that looks wrong.
This patch enables building pgbench to use ppoll() instead of select()
to allow for more than (FD_SETSIZE - 10) connections. As implemented,
when using ppoll(), the only connection limitation is system resources.
The patch has been implemented to introduce a minimal of #ifdef/#ifndef
clutter in the code.
Two patches attached.
One based on REL9_6_STABLE.
One based on 'master' which can also apply to REL_10_STABLE.
doug
--
Doug Rady
Amazon Aurora PostgreSQL
radydoug@amazon.com
On 2017-09-25 18:01:40 +0000, Rady, Doug wrote:
This patch enables building pgbench to use ppoll() instead of select()
to allow for more than (FD_SETSIZE - 10) connections. As implemented,
when using ppoll(), the only connection limitation is system resources.
Hm, is there any need of using ppoll over poll? IIRC it's a good bit
more common and there's, also iirc, a number of platforms with buggy
ppoll implementations.
Greetings,
Andres Freund
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Hello Again,
Two patches attached.
One based on REL9_6_STABLE.
I'd be surprise that there would be a backport unless there is a bug, so
this one might not be useful.
One based on 'master' which can also apply to REL_10_STABLE.
Could you add your patches to the next CF?
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 9/25/17, 11:07, "Andres Freund" <andres@anarazel.de> wrote:
On 2017-09-25 18:01:40 +0000, Rady, Doug wrote:
This patch enables building pgbench to use ppoll() instead of select()
to allow for more than (FD_SETSIZE - 10) connections. As implemented,
when using ppoll(), the only connection limitation is system resources.
Hm, is there any need of using ppoll over poll? IIRC it's a good bit
more common and there's, also iirc, a number of platforms with buggy
ppoll implementations.
Greetings,
Andres Freund
I used ppoll() as it can support the microseconds duration for \SLEEP meta command.
thanks!
doug
--
Doug Rady
Amazon Aurora PostgreSQL
radydoug@amazon.com
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
This patch enables building pgbench to use ppoll() instead of select()
to allow for more than (FD_SETSIZE - 10) connections. As implemented,
when using ppoll(), the only connection limitation is system resources.One based on 'master' which can also apply to REL_10_STABLE.
/home/fabien/pgbench-ppoll.patch:137: trailing whitespace.
#define PFD_THREAD_INIT(t,s,n) { do ...
error: patch failed: configure:13024
error: configure: patch does not apply
error: patch failed: configure.in:1430
error: configure.in: patch does not apply
error: patch failed: src/bin/pgbench/pgbench.c:4588
error: src/bin/pgbench/pgbench.c: patch does not apply
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Fixed the trailing garbage … sigh.
Fixed the init of unused pollfds.
Re-based for both 10 & master.
--
Thank you!
doug
On 10/3/17, 23:21, "Fabien COELHO" <coelho@cri.ensmp.fr> wrote:
This patch enables building pgbench to use ppoll() instead of select()
to allow for more than (FD_SETSIZE - 10) connections. As implemented,
when using ppoll(), the only connection limitation is system resources.One based on 'master' which can also apply to REL_10_STABLE.
/home/fabien/pgbench-ppoll.patch:137: trailing whitespace.
#define PFD_THREAD_INIT(t,s,n) { do ...
error: patch failed: configure:13024
error: configure: patch does not apply
error: patch failed: configure.in:1430
error: configure.in: patch does not apply
error: patch failed: src/bin/pgbench/pgbench.c:4588
error: src/bin/pgbench/pgbench.c: patch does not apply
--
Fabien.
On Mon, Sep 25, 2017 at 8:01 PM, Rady, Doug <radydoug@amazon.com> wrote:
This patch enables building pgbench to use ppoll() instead of select()
to allow for more than (FD_SETSIZE - 10) connections. As implemented,
when using ppoll(), the only connection limitation is system resources.
So what's an example of something that fails without this patch but
works with the patch?
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Without this patch, one is limited to '(FD_SETSIZE - 10)’ number of connections.
Example of something that fails without this patch but works with the patch:
Without the patch:
$ pgbench -j 3000 -c 1500
invalid number of clients: "1500"
With the patch:
$ pgbench -j 3000 -c 1500
starting vacuum...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 2000
query mode: simple
number of clients: 1500
number of threads: 1500
number of transactions per client: 10
number of transactions actually processed: 15000/15000
latency average = 631.730 ms
tps = 2374.430587 (including connections establishing)
tps = 4206.524986 (excluding connections establishing)
--
doug
On 10/26/17, 04:46, "Robert Haas" <robertmhaas@gmail.com> wrote:
On Mon, Sep 25, 2017 at 8:01 PM, Rady, Doug <radydoug@amazon.com> wrote:
This patch enables building pgbench to use ppoll() instead of select()
to allow for more than (FD_SETSIZE - 10) connections. As implemented,
when using ppoll(), the only connection limitation is system resources.
So what's an example of something that fails without this patch but
works with the patch?
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Hello,
Could you rebase the v11 patch?
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Fri, Nov 3, 2017 at 2:29 PM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:
Could you rebase the v11 patch?
This patch has been waiting for a rebase for more than three weeks as
of today, I am marking it as returned with feedback. It would be a
good idea to reply to Robert's input in
/messages/by-id/CA+TgmoYybNv-DdhVPMxLB2nx2SqeNJirtWHmdEAZUCGoTB2VBg@mail.gmail.com.
--
Michael
Hello Michaï¿œl,
Could you rebase the v11 patch?
This patch has been waiting for a rebase for more than three weeks as
of today, I am marking it as returned with feedback. It would be a
good idea to reply to Robert's input in
/messages/by-id/CA+TgmoYybNv-DdhVPMxLB2nx2SqeNJirtWHmdEAZUCGoTB2VBg@mail.gmail.com.
ISTM that this was done: If -c is high enough, pgbench fails without the
patch.
--
Fabien.