parallel-processing multiple similar query tasks - any example?
multiple similar query tasks are as follows:
select * from a_table where country ='UK'
select * from a_table where country='France'
and so on
How best to parallel-processing such types of multiple similar query tasks?
Any example available?
Regards,
David
Import Notes
Reply to msg id not found: CA+i5JwaC9Z-rjWiyo7LpncdK=CuQcKzuVNMnAhPW1oi=8c9Sug@mail.gmail.comReference msg id not found: CA+i5JwaC9Z-rjWiyo7LpncdK=CuQcKzuVNMnAhPW1oi=8c9Sug@mail.gmail.com
On Wed, Apr 27, 2022 at 4:34 PM Shaozhong SHI <shishaozhong@gmail.com>
wrote:
multiple similar query tasks are as follows:
select * from a_table where country ='UK'
select * from a_table where country='France'
and so onHow best to parallel-processing such types of multiple similar query tasks?
Any example available?
You should search for how to run processes/commands in parallel in whatever
client-side execution environment you want to use. There isn't anything
specific to PostgreSQL here.
David J.
On Wed, Apr 27, 2022 at 4:44 PM David G. Johnston <
david.g.johnston@gmail.com> wrote:
On Wed, Apr 27, 2022 at 4:34 PM Shaozhong SHI <shishaozhong@gmail.com>
wrote:multiple similar query tasks are as follows:
select * from a_table where country ='UK'
select * from a_table where country='France'
and so onHow best to parallel-processing such types of multiple similar query
tasks?Any example available?
You should search for how to run processes/commands in parallel in
whatever client-side execution environment you want to use. There isn't
anything specific to PostgreSQL here.
You should also read:
https://www.postgresql.org/docs/current/parallel-query.html
If you want to see when a single query can be executed using parallel
workers.
David J.
On 2022-Apr-28, Shaozhong SHI wrote:
multiple similar query tasks are as follows:
select * from a_table where country ='UK'
select * from a_table where country='France'
and so onHow best to parallel-processing such types of multiple similar query tasks?
Any example available?
for country in UK France Germany Ireland; do
echo "select pg_sleep(1); select * from a_table where country = '${country//\'/''/}'"
done | \
xargs -d"\n" -P10 -n1 psql -X -c
Note the ${country/} stuff is a bash-ism.
--
Álvaro Herrera
No, No.
Why sleep(1)?
It should be all active - doing work concurrently.
Regards,
David
On Thu, 28 Apr 2022 at 16:17, Alvaro Herrera <alvherre@alvh.no-ip.org>
wrote:
Show quoted text
On 2022-Apr-28, Shaozhong SHI wrote:
multiple similar query tasks are as follows:
select * from a_table where country ='UK'
select * from a_table where country='France'
and so onHow best to parallel-processing such types of multiple similar query
tasks?
Any example available?
for country in UK France Germany Ireland; do
echo "select pg_sleep(1); select * from a_table where country =
'${country//\'/''/}'"
done | \
xargs -d"\n" -P10 -n1 psql -X -cNote the ${country/} stuff is a bash-ism.
--
Álvaro Herrera
On 2022-Apr-28, Shaozhong SHI wrote:
Why sleep(1)?
It is sleeping to show that they are running concurrently. If it runs
five sleeps of one second each and the whole command lasts one second,
then all sleeps ran in parallel. Had the whole command taken five
seconds, you would know that the queries ran serially.
It should be all active - doing work concurrently.
They are all active simultaneously. You just need to supply your own
query, without any sleeps.
--
Álvaro Herrera
Expand and explain please.
Regards,
David
On Thursday, 28 April 2022, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
Show quoted text
On 2022-Apr-28, Shaozhong SHI wrote:
Why sleep(1)?
It is sleeping to show that they are running concurrently. If it runs
five sleeps of one second each and the whole command lasts one second,
then all sleeps ran in parallel. Had the whole command taken five
seconds, you would know that the queries ran serially.It should be all active - doing work concurrently.
They are all active simultaneously. You just need to supply your own
query, without any sleeps.--
Álvaro Herrera
On 2022-Apr-28, Shaozhong SHI wrote:
Expand and explain please.
No, thanks.
--
Álvaro Herrera
Well, I guess that does not work.
Never mind.
Regards,
David
On Thursday, 28 April 2022, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
Show quoted text
On 2022-Apr-28, Shaozhong SHI wrote:
Expand and explain please.
No, thanks.
--
Álvaro Herrera