PATCH: pgbench - option to build using ppoll() for larger connection counts

On 2017-09-25 18:01:40 +0000, Rady, Doug wrote:

This patch enables building pgbench to use ppoll() instead of select()
to allow for more than (FD_SETSIZE - 10) connections. As implemented,
when using ppoll(), the only connection limitation is system resources.

Hm, is there any need of using ppoll over poll? IIRC it's a good bit
more common and there's, also iirc, a number of platforms with buggy
ppoll implementations.

Greetings,

Andres Freund

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Hello Again,

Two patches attached.
One based on REL9_6_STABLE.

I'd be surprise that there would be a backport unless there is a bug, so
this one might not be useful.

One based on 'master' which can also apply to REL_10_STABLE.

Could you add your patches to the next CF?

--
Fabien.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 9/25/17, 11:07, "Andres Freund" <andres@anarazel.de> wrote:

On 2017-09-25 18:01:40 +0000, Rady, Doug wrote:

This patch enables building pgbench to use ppoll() instead of select()
to allow for more than (FD_SETSIZE - 10) connections. As implemented,
when using ppoll(), the only connection limitation is system resources.

Hm, is there any need of using ppoll over poll? IIRC it's a good bit
more common and there's, also iirc, a number of platforms with buggy
ppoll implementations.

Greetings,

Andres Freund

I used ppoll() as it can support the microseconds duration for \SLEEP meta command.

thanks!
doug
--
Doug Rady
Amazon Aurora PostgreSQL
radydoug@amazon.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

This patch enables building pgbench to use ppoll() instead of select()
to allow for more than (FD_SETSIZE - 10) connections. As implemented,
when using ppoll(), the only connection limitation is system resources.

One based on 'master' which can also apply to REL_10_STABLE.

/home/fabien/pgbench-ppoll.patch:137: trailing whitespace.
#define PFD_THREAD_INIT(t,s,n) { do ...
error: patch failed: configure:13024
error: configure: patch does not apply
error: patch failed: configure.in:1430
error: configure.in: patch does not apply
error: patch failed: src/bin/pgbench/pgbench.c:4588
error: src/bin/pgbench/pgbench.c: patch does not apply

--
Fabien.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Fixed the trailing garbage … sigh.
Fixed the init of unused pollfds.

Re-based for both 10 & master.

--
Thank you!
doug

On 10/3/17, 23:21, "Fabien COELHO" <coelho@cri.ensmp.fr> wrote:

This patch enables building pgbench to use ppoll() instead of select()
to allow for more than (FD_SETSIZE - 10) connections. As implemented,
when using ppoll(), the only connection limitation is system resources.

One based on 'master' which can also apply to REL_10_STABLE.

/home/fabien/pgbench-ppoll.patch:137: trailing whitespace.
#define PFD_THREAD_INIT(t,s,n) { do ...
error: patch failed: configure:13024
error: configure: patch does not apply
error: patch failed: configure.in:1430
error: configure.in: patch does not apply
error: patch failed: src/bin/pgbench/pgbench.c:4588
error: src/bin/pgbench/pgbench.c: patch does not apply

--
Fabien.

On Mon, Sep 25, 2017 at 8:01 PM, Rady, Doug <radydoug@amazon.com> wrote:

This patch enables building pgbench to use ppoll() instead of select()

to allow for more than (FD_SETSIZE - 10) connections. As implemented,

when using ppoll(), the only connection limitation is system resources.

So what's an example of something that fails without this patch but
works with the patch?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Without this patch, one is limited to '(FD_SETSIZE - 10)’ number of connections.
Example of something that fails without this patch but works with the patch:

Without the patch:

$ pgbench -j 3000 -c 1500
invalid number of clients: "1500"

With the patch:

$ pgbench -j 3000 -c 1500
starting vacuum...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 2000
query mode: simple
number of clients: 1500
number of threads: 1500
number of transactions per client: 10
number of transactions actually processed: 15000/15000
latency average = 631.730 ms
tps = 2374.430587 (including connections establishing)
tps = 4206.524986 (excluding connections establishing)

--
doug

On 10/26/17, 04:46, "Robert Haas" <robertmhaas@gmail.com> wrote:

On Mon, Sep 25, 2017 at 8:01 PM, Rady, Doug <radydoug@amazon.com> wrote:

This patch enables building pgbench to use ppoll() instead of select()

to allow for more than (FD_SETSIZE - 10) connections. As implemented,

when using ppoll(), the only connection limitation is system resources.

So what's an example of something that fails without this patch but
works with the patch?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Hello,

Could you rebase the v11 patch?

--
Fabien.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Fri, Nov 3, 2017 at 2:29 PM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:

Could you rebase the v11 patch?

This patch has been waiting for a rebase for more than three weeks as
of today, I am marking it as returned with feedback. It would be a
good idea to reply to Robert's input in
/messages/by-id/CA+TgmoYybNv-DdhVPMxLB2nx2SqeNJirtWHmdEAZUCGoTB2VBg@mail.gmail.com.
--
Michael

Hello Michaï¿œl,

Could you rebase the v11 patch?

This patch has been waiting for a rebase for more than three weeks as
of today, I am marking it as returned with feedback. It would be a
good idea to reply to Robert's input in
/messages/by-id/CA+TgmoYybNv-DdhVPMxLB2nx2SqeNJirtWHmdEAZUCGoTB2VBg@mail.gmail.com.

ISTM that this was done: If -c is high enough, pgbench fails without the
patch.

--
Fabien.