pg_background contrib module proposal

Started by Amul Sulover 9 years ago42 messageshackers
Jump to latest
#1Amul Sul
sulamul@gmail.com

Hi All,

I would like to take over pg_background patch and repost for
discussion and review.

Initially Robert Haas has share this for parallelism demonstration[1]. /messages/by-id/CA+Tgmoam66dTzCP8N2cRcS6S6dBMFX+JMba+mDf68H=KAkNjPQ@mail.gmail.com
and abandoned later with
summary of open issue[2]. /messages/by-id/CA+TgmobPiT_3Qgjeh3_v+8Cq2nMczkPyAYernF_7_W9a-6T1PA@mail.gmail.com with this pg_background patch need to be
fixed, most of them seems to be
addressed in core except handling of type exists without binary
send/recv functions and documentation.
I have added handling for types that don't have binary send/recv
functions in the attach patch and will
work on documentation at the end.

One concern with this patch is code duplication with
exec_simple_query(), we could
consider Jim Nasby’s patch[3]. /messages/by-id/54541779.1010906@BlueTreble.com to overcome this, but certainly we
will end up by complicating
exec_simple_query() to make pg_background happy.

As discussed previously[1]. /messages/by-id/CA+Tgmoam66dTzCP8N2cRcS6S6dBMFX+JMba+mDf68H=KAkNjPQ@mail.gmail.com pg_background is a contrib module that lets
you launch
arbitrary command in a background worker.

• VACUUM in background
• Autonomous transaction implementation better than dblink way (i.e.
no separate authentication required).
• Allows to perform task like CREATE INDEX CONCURRENTLY from a
procedural language.

This module comes with following SQL APIs:

• pg_background_launch : This API takes SQL command, which user wants
to execute, and size of queue buffer.
This function returns the process id of background worker.
• pg_background_result : This API takes the process id as input
parameter and returns the result of command
executed thought the background worker.
• pg_background_detach : This API takes the process id and detach the
background process which is waiting for
user to read its results.

Here's an example of running vacuum and then fetching the results.
Notice that the
notices from the original session are propagated to our session; if an
error had occurred,
it would be re-thrown locally when we try to read the results.

postgres=# create table foo (a int);
CREATE TABLE
postgres=# insert into foo values(generate_series(1,5));
INSERT 0 5

postgres=# select pg_background_launch('vacuum verbose foo');
pg_background_launch
----------------------
65427
(1 row)

postgres=# select * from pg_background_result(65427) as (x text);
INFO: vacuuming "public.foo"
INFO: "foo": found 0 removable, 5 nonremovable row versions in 1 out of 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
There were 0 unused item pointers.
Skipped 0 pages due to buffer pins.
0 pages are entirely empty.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
x
--------
VACUUM
(1 row)

Thanks to Vibhor kumar, Rushabh Lathia and Robert Haas for feedback.

Please let me know your thoughts, and thanks for reading.

[1]: . /messages/by-id/CA+Tgmoam66dTzCP8N2cRcS6S6dBMFX+JMba+mDf68H=KAkNjPQ@mail.gmail.com
[2]: . /messages/by-id/CA+TgmobPiT_3Qgjeh3_v+8Cq2nMczkPyAYernF_7_W9a-6T1PA@mail.gmail.com
[3]: . /messages/by-id/54541779.1010906@BlueTreble.com

Regards,
Amul

Attachments:

0001-pg_background_v7.patchapplication/octet-stream; name=0001-pg_background_v7.patchDownload+1095-6
#2Andrey Borodin
amborodin@acm.org
In reply to: Amul Sul (#1)
Re: pg_background contrib module proposal

This is simply wonderful!
Finaly, I can implement my favorite sleepsort in postgres:

create table input as select round(random()*20) x from generate_series(1,5,1);
create table output(place int,value int);
create sequence s start 1;
create table handles as select pg_background_launch('select
pg_sleep('||x||'); insert into output values
(nextval(''s''),'||x||');') h from input;
select (select * from pg_background_result(h) as (x text) limit 1) from handles;
select * from output;

Works like a charm. It has perfrect running time O(1) where n is
number of items to sort, but it cannot sort more than
max_worker_processes-1.
Next step of user cuncurrency mechanics is future indexes (indexes
under cunstruction are taken into plans, query is executed when index
is actualy constructed) and promised tables (query waits untial there
is actually data in the table).

I like the idea and implementation and I'm planning to post review by
the end of december.
Right now I have just one useful idea: I do not see comfortable way to
check up state of task (finished\failed) without possibility of haning
for long or catching fire.

Best regards, Andrey Borodin.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#3Andrey Borodin
amborodin@acm.org
In reply to: Andrey Borodin (#2)
Re: pg_background contrib module proposal

I've read the code and now I have more suggestions.

1. Easy one. The case of different natts of expected and acutal result
throws same errmsg as the case of wrong types.
I think we should assist the user more here

if (natts != tupdesc->natts)
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("remote query result rowtype does not match the
specified FROM clause rowtype")));

and here

if (type_id != tupdesc->attrs[i]->atttypid)
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("remote query result rowtype does not match the
specified FROM clause rowtype")));

2. This code
errhint("You may need to increase max_worker_processes.")
Is technically hinting absolutley right.
But this extension requires something like pg_bouncer or what is
called "thread pool".
You just cannot safely fire a background task because you do not know
how many workes can be spawned. Maybe it'd be better to create 'pool'
of workers and form a queue of queries for them?
This will make possible start a millions of subtasks.

3. About getting it to the core, not as an extension. IMO this is too
sharp thing to place it into core, I think that it's best place is in
contribs.

Thanks for the work on such a cool and fun extension.

Best regards, Andrey Borodin.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#4Pavel Stehule
pavel.stehule@gmail.com
In reply to: Andrey Borodin (#3)
Re: pg_background contrib module proposal

2016-12-09 13:48 GMT+01:00 Andrew Borodin <borodin@octonica.com>:

I've read the code and now I have more suggestions.

1. Easy one. The case of different natts of expected and acutal result
throws same errmsg as the case of wrong types.
I think we should assist the user more here

if (natts != tupdesc->natts)
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("remote query result rowtype does not match the
specified FROM clause rowtype")));

and here

if (type_id != tupdesc->attrs[i]->atttypid)
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("remote query result rowtype does not match the
specified FROM clause rowtype")));

2. This code
errhint("You may need to increase max_worker_processes.")
Is technically hinting absolutley right.
But this extension requires something like pg_bouncer or what is
called "thread pool".
You just cannot safely fire a background task because you do not know
how many workes can be spawned. Maybe it'd be better to create 'pool'
of workers and form a queue of queries for them?
This will make possible start a millions of subtasks.

This is not easy, because pgworker is related to database, not to server.
There are servers where you can have thousands databases.

Regards

Pavel

Show quoted text

3. About getting it to the core, not as an extension. IMO this is too
sharp thing to place it into core, I think that it's best place is in
contribs.

Thanks for the work on such a cool and fun extension.

Best regards, Andrey Borodin.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#5Robert Haas
robertmhaas@gmail.com
In reply to: Amul Sul (#1)
Re: pg_background contrib module proposal

On Wed, Nov 23, 2016 at 10:46 PM, amul sul <sulamul@gmail.com> wrote:

I would like to take over pg_background patch and repost for
discussion and review.

Initially Robert Haas has share this for parallelism demonstration[1]
and abandoned later with
summary of open issue[2] with this pg_background patch need to be
fixed, most of them seems to be
addressed in core except handling of type exists without binary
send/recv functions and documentation.
I have added handling for types that don't have binary send/recv
functions in the attach patch and will
work on documentation at the end.

One concern with this patch is code duplication with
exec_simple_query(), we could
consider Jim Nasby’s patch[3] to overcome this, but certainly we
will end up by complicating
exec_simple_query() to make pg_background happy.

As discussed previously[1] pg_background is a contrib module that lets
you launch
arbitrary command in a background worker.

It looks like this could be reworked as a client of Peter Eisentraut's
background sessions code, which I think is also derived from
pg_background:

http://archives.postgresql.org/message-id/e1c2d331-ee6a-432d-e9f5-dcf85cffaf29@2ndquadrant.com

That might be good, because then we wouldn't have to maintain two
copies of the code.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#6Andrey Borodin
amborodin@acm.org
In reply to: Robert Haas (#5)
Re: pg_background contrib module proposal

2016-12-09 18:00 GMT+05:00 Robert Haas <robertmhaas@gmail.com>:

It looks like this could be reworked as a client of Peter Eisentraut's
background sessions code, which I think is also derived from
pg_background:

http://archives.postgresql.org/message-id/e1c2d331-ee6a-432d-e9f5-dcf85cffaf29@2ndquadrant.com

That might be good, because then we wouldn't have to maintain two
copies of the code.

Code looks quite different. I mean bgsession.c code and pg_background.c code.
Definitly, there is possibility to refactor both patches to have
common subset of base routines, they operate with similar concepts.
But to start, it's better to choose which patch goes first, or merge
them.

There is no possibility to make one on base of other since they both
require some work.
Personally, I like C code from pg_background more. It is far better
commented and has more exceptions info for user. But interface of
bgsessions is crispier. Finally, they solve different problems.

I signed up for review there too (in background sessions patch). I
hope I'll have enough resources to provide decent review for both in
december, before commitfest.

Best regards, Andrey Borodin.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#7Andrey Borodin
amborodin@acm.org
In reply to: Andrey Borodin (#6)
Re: pg_background contrib module proposal

Hi!

Just in case you'd like to include sleepsort as a test, here it is
wrapped as a regression test(see attachment). But it has serious
downside: it runs no less than 5 seconds.

Also I'll list here every parallelism feature I managed to imagine. It
is not to say that I suggest having some of these features at v1. We
certainly have to start somewhere, this list is just for information
purposes.
1. Poolable workers. Just to be sure you can reliably queue your task
somewhere without having "increase max_connections" hint.
2. Inside one pool, you can have task chaining. After competition of
task A (query A) start task B, in case of failure start task C. Task C
is followed by task D.
3. Reliably read task status: running\awaiting\completed\errored\in a
lock. Get a query plan of a task? (I know, there are reasons why it is
not possible now)
4. Query as a future (actually it is implemented already by
pg_background_result)
5. Promised result. Declare that you are waiting for data of specific
format, execute a query, someone (from another backend) will
eventually place a data to promised result and your query continues
execution.
6. Cancelation: a way to signal to background query that it's time to
quit gracefully.

Best regards, Andrey Borodin.

Attachments:

pgb_test.difftext/plain; charset=US-ASCII; name=pgb_test.diffDownload+37-0
#8Craig Ringer
craig@2ndquadrant.com
In reply to: Andrey Borodin (#7)
Re: pg_background contrib module proposal

On 13 December 2016 at 01:17, Andrew Borodin <borodin@octonica.com> wrote:

6. Cancelation: a way to signal to background query that it's time to
quit gracefully.

That at least should be fuss-free. SIGTERM it, and make sure the
worker does CHECK_FOR_INTERRUPTS() in regularly-hit places and loops.
Ensure the worker waits on latches rather than pg_usleep()ing.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#9Amul Sul
sulamul@gmail.com
In reply to: Andrey Borodin (#7)
Re: pg_background contrib module proposal

On Mon, Dec 12, 2016 at 10:47 PM, Andrew Borodin <borodin@octonica.com> wrote:

Hi!

Thanks a lot for your review.

Just in case you'd like to include sleepsort as a test, here it is
wrapped as a regression test(see attachment). But it has serious
downside: it runs no less than 5 seconds.

Also I'll list here every parallelism feature I managed to imagine. It
is not to say that I suggest having some of these features at v1. We
certainly have to start somewhere, this list is just for information
purposes.
1. Poolable workers. Just to be sure you can reliably queue your task
somewhere without having "increase max_connections" hint.
2. Inside one pool, you can have task chaining. After competition of
task A (query A) start task B, in case of failure start task C. Task C
is followed by task D.

I think background-session code is not that much deviated from
pg_background code, IIUC background-session patch we can manage to
reuse it, as suggested by Robert. This will allow us to maintain
session in long run, we could reuse this session to run subsequent
queries instead of maintaining separate worker pool. Thoughts?

3. Reliably read task status: running\awaiting\completed\errored\in a
lock. Get a query plan of a task? (I know, there are reasons why it is
not possible now)

+1, Let me check this.

4. Query as a future (actually it is implemented already by
pg_background_result)
5. Promised result. Declare that you are waiting for data of specific
format, execute a query, someone (from another backend) will
eventually place a data to promised result and your query continues
execution.

Could you please elaborate more?
Do you mean two way communication between foreground & background process?

6. Cancelation: a way to signal to background query that it's time to
quit gracefully.

Let me check this too.

Thanks & Regards,
Amul

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#10Andrey Borodin
amborodin@acm.org
In reply to: Amul Sul (#9)
Re: pg_background contrib module proposal

2016-12-13 12:55 GMT+05:00 amul sul <sulamul@gmail.com>:

I think background-session code is not that much deviated from
pg_background code,

It is not derived, though it is not much deviated. background sessions
code do not have detailed exceptions and code comments, but it is
doing somewhat similar things.

IIUC background-session patch we can manage to
reuse it, as suggested by Robert. This will allow us to maintain
session in long run, we could reuse this session to run subsequent
queries instead of maintaining separate worker pool. Thoughts?

One API to deal with both features would be much better, sure.
"Object" like sessions pool are much easier to implement on top of
session "object" then on top of worker process, PIDs etc.

4. Query as a future (actually it is implemented already by
pg_background_result)
5. Promised result. Declare that you are waiting for data of specific
format, execute a query, someone (from another backend) will
eventually place a data to promised result and your query continues
execution.

Could you please elaborate more?
Do you mean two way communication between foreground & background process?

It is from C++11 threading: future, promise and async - these are
related but different kinds of aquiring result between threads.
Feature - is when caller Cl start thread T(or dequeue thread from
pool) and Cl can wait until T finishes and provides result.
Here waiting the result is just like selecting from pg_background_result().
Promise - is when you declare a variable and caller do not know which
thread will put the data to this variable. But any thread reading
promise will wait until other thread put a data to promise.
There are more parallelism patterns there, like async, deffered, lazy,
but futures and promises from my point of view are most used.

6. Cancelation: a way to signal to background query that it's time to
quit gracefully.

Let me check this too.

I think Craig is right: any background query must be ready to be shut
down. That's what databases are about, you can safely pull the plug at
any moment.
I've remembered one more parallelism pattern: critical region. It's
when you ask the system "plz no TERM now, and, if you can, postpone
the vacuums, OOMKs and that kind of stuff" from user code. But I do
not think it's usable pattern here.

Thank you for your work.

Best regards, Andrey Borodin.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#11Amul Sul
sulamul@gmail.com
In reply to: Andrey Borodin (#10)
Re: pg_background contrib module proposal

On Tue, Dec 13, 2016 at 2:05 PM, Andrew Borodin <borodin@octonica.com> wrote:

2016-12-13 12:55 GMT+05:00 amul sul <sulamul@gmail.com>:

I think background-session code is not that much deviated from
pg_background code,

It is not derived, though it is not much deviated. background sessions
code do not have detailed exceptions and code comments, but it is
doing somewhat similar things.

IIUC background-session patch we can manage to
reuse it, as suggested by Robert. This will allow us to maintain
session in long run, we could reuse this session to run subsequent
queries instead of maintaining separate worker pool. Thoughts?

One API to deal with both features would be much better, sure.
"Object" like sessions pool are much easier to implement on top of
session "object" then on top of worker process, PIDs etc.

4. Query as a future (actually it is implemented already by
pg_background_result)
5. Promised result. Declare that you are waiting for data of specific
format, execute a query, someone (from another backend) will
eventually place a data to promised result and your query continues
execution.

Could you please elaborate more?
Do you mean two way communication between foreground & background process?

It is from C++11 threading: future, promise and async - these are
related but different kinds of aquiring result between threads.
Feature - is when caller Cl start thread T(or dequeue thread from
pool) and Cl can wait until T finishes and provides result.
Here waiting the result is just like selecting from pg_background_result().
Promise - is when you declare a variable and caller do not know which
thread will put the data to this variable. But any thread reading
promise will wait until other thread put a data to promise.
There are more parallelism patterns there, like async, deffered, lazy,
but futures and promises from my point of view are most used.

Nice, thanks for detailed explanation.

We can use shm_mq infrastructure to share any kind of message between
two processes,
but perhaps we might end up with overestimating what originally pg_background
could used for - the user backend will launch workers and give them an
initial set
of instruction and then results will stream back from the workers to
the user backend.

6. Cancelation: a way to signal to background query that it's time to
quit gracefully.

Let me check this too.

I think Craig is right: any background query must be ready to be shut
down. That's what databases are about, you can safely pull the plug at
any moment.

SIGTERM is handled in current pg_background patch, user can terminate
backend execution using pg_cancel_backend() or pg_terminate_backend()
as shown below:

postgres=> select pg_background_launch('insert into foo
values(generate_series(1,100000000))');
pg_background_launch
----------------------
67069
(1 row)

postgres=> select pg_terminate_backend(67069);
pg_terminate_backend
----------------------
t
(1 row)

postgres=> select * from pg_background_result(67069) as (x text);
ERROR: terminating connection due to administrator command
CONTEXT: background worker, pid 67069
postgres=>

Thanks & Regards,
Amul

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#12Craig Ringer
craig@2ndquadrant.com
In reply to: Amul Sul (#11)
Re: pg_background contrib module proposal

On 13 Dec. 2016 20:54, "amul sul" <sulamul@gmail.com> wrote:

postgres=> select * from pg_background_result(67069) as (x text);
ERROR: terminating connection due to administrator command
CONTEXT: background worker, pid 67069
postgres=>

It'll also want to handle cancellation due to conflict with recovery if you
intend it to be used on a standby, probably by making use
of procsignal_sigusr1_handler. The rest of the work is done by
CHECK_FOR_INTERRUPTS() .

This only matters if it's meant to work on standbys of course. I haven't
checked if you write to catalogs or otherwise do non-standby-friendly
things.

#13David Fetter
david@fetter.org
In reply to: Andrey Borodin (#7)
Re: pg_background contrib module proposal

On Mon, Dec 12, 2016 at 10:17:24PM +0500, Andrew Borodin wrote:

Hi!

Just in case you'd like to include sleepsort as a test, here it is
wrapped as a regression test(see attachment). But it has serious
downside: it runs no less than 5 seconds.

Couldn't it sleep in increments smaller than a second? Like maybe
milliseconds? Also, it's probably cleaner (or at least more
comprehensible) to write something using format() and dollar quoting
than the line with the double 's.

Best,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#14Andrey Borodin
amborodin@acm.org
In reply to: David Fetter (#13)
Re: pg_background contrib module proposal

2016-12-19 4:21 GMT+05:00 David Fetter <david@fetter.org>:

Couldn't it sleep in increments smaller than a second? Like maybe
milliseconds? Also, it's probably cleaner (or at least more
comprehensible) to write something using format() and dollar quoting
than the line with the double 's.

Right. Here's version which waits for half a second. I do not see how
to path doubles via dollar sign, pg_background_launch() gets a string
as a parameter, it's not EXCEUTE USING.

Best regards, Andrey Borodin.

Attachments:

pgb_sleepsort.difftext/plain; charset=US-ASCII; name=pgb_sleepsort.diffDownload+45-0
#15David Fetter
david@fetter.org
In reply to: Amul Sul (#1)
Re: pg_background contrib module proposal

On Thu, Nov 24, 2016 at 09:16:53AM +0530, amul sul wrote:

Hi All,

I would like to take over pg_background patch and repost for
discussion and review.

This looks great.

Sadly, it now breaks initdb when applied atop
dd728826c538f000220af98de025c00114ad0631 with:

performing post-bootstrap initialization ... TRAP: FailedAssertion("!(((((const Node*)(rinfo))->type) == T_RestrictInfo))", File: "costsize.c", Line: 660)
sh: line 1: 2840 Aborted (core dumped) "/home/shackle/pggit/postgresql/tmp_install/home/shackle/10/bin/postgres" --single -F -O -j -c search_path=pg_catalog -c exit_on_error=true template1 > /dev/null

Best,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#16Amul Sul
sulamul@gmail.com
In reply to: David Fetter (#15)
Re: pg_background contrib module proposal

On Tue, Dec 20, 2016 at 12:21 AM, David Fetter <david@fetter.org> wrote:

On Thu, Nov 24, 2016 at 09:16:53AM +0530, amul sul wrote:

Hi All,

I would like to take over pg_background patch and repost for
discussion and review.

This looks great.

Sadly, it now breaks initdb when applied atop
dd728826c538f000220af98de025c00114ad0631 with:

performing post-bootstrap initialization ... TRAP: FailedAssertion("!(((((const Node*)(rinfo))->type) == T_RestrictInfo))", File: "costsize.c", Line: 660)
sh: line 1: 2840 Aborted (core dumped) "/home/shackle/pggit/postgresql/tmp_install/home/shackle/10/bin/postgres" --single -F -O -j -c search_path=pg_catalog -c exit_on_error=true template1 > /dev/null

I've complied binary with --enable-cassert flag and pg_background
patch to the top of described commit as well as on latest HEAD, but I
am not able to reproduce this crash, see this:

[VM postgresql]$ /home/amul/Public/pg_inst/pg-master/bin/postgres
--single -F -O -j -c search_path=pg_catalog -c exit_on_error=true
template1

PostgreSQL stand-alone backend 10devel
backend> CREATE EXTENSION pg_background;

backend> select * from pg_extension;

1: extname (typeid = 19, len = 64, typmod = -1, byval = f)
2: extowner (typeid = 26, len = 4, typmod = -1, byval = t)
3: extnamespace (typeid = 26, len = 4, typmod = -1, byval = t)
4: extrelocatable (typeid = 16, len = 1, typmod = -1, byval = t)
5: extversion (typeid = 25, len = -1, typmod = -1, byval = f)
6: extconfig (typeid = 1028, len = -1, typmod = -1, byval = f)
7: extcondition (typeid = 1009, len = -1, typmod = -1, byval = f)
----
1: extname = "plpgsql" (typeid = 19, len = 64, typmod = -1, byval = f)
2: extowner = "10" (typeid = 26, len = 4, typmod = -1, byval = t)
3: extnamespace = "11" (typeid = 26, len = 4, typmod = -1, byval = t)
4: extrelocatable = "f" (typeid = 16, len = 1, typmod = -1, byval = t)
5: extversion = "1.0" (typeid = 25, len = -1, typmod = -1, byval = f)
----
1: extname = "pg_background" (typeid = 19, len = 64, typmod = -1, byval = f)
2: extowner = "10" (typeid = 26, len = 4, typmod = -1, byval = t)
3: extnamespace = "11" (typeid = 26, len = 4, typmod = -1, byval = t)
4: extrelocatable = "t" (typeid = 16, len = 1, typmod = -1, byval = t)
5: extversion = "1.0" (typeid = 25, len = -1, typmod = -1, byval = f)
----
backend>

I hope I am missing anything. Thanks !

Regards,
Amul

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#17David Fetter
david@fetter.org
In reply to: Amul Sul (#16)
Re: pg_background contrib module proposal

On Tue, Dec 20, 2016 at 11:11:36AM +0530, amul sul wrote:

On Tue, Dec 20, 2016 at 12:21 AM, David Fetter <david@fetter.org> wrote:

On Thu, Nov 24, 2016 at 09:16:53AM +0530, amul sul wrote:

Hi All,

I would like to take over pg_background patch and repost for
discussion and review.

This looks great.

Sadly, it now breaks initdb when applied atop
dd728826c538f000220af98de025c00114ad0631 with:

performing post-bootstrap initialization ... TRAP: FailedAssertion("!(((((const Node*)(rinfo))->type) == T_RestrictInfo))", File: "costsize.c", Line: 660)
sh: line 1: 2840 Aborted (core dumped) "/home/shackle/pggit/postgresql/tmp_install/home/shackle/10/bin/postgres" --single -F -O -j -c search_path=pg_catalog -c exit_on_error=true template1 > /dev/null

I've complied binary with --enable-cassert flag and pg_background
patch to the top of described commit as well as on latest HEAD, but I
am not able to reproduce this crash, see this:

[VM postgresql]$ /home/amul/Public/pg_inst/pg-master/bin/postgres
--single -F -O -j -c search_path=pg_catalog -c exit_on_error=true
template1

I haven't managed to reproduce it either. Sorry about the noise.

Best,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#18David Fetter
david@fetter.org
In reply to: Andrey Borodin (#14)
Re: pg_background contrib module proposal

On Mon, Dec 19, 2016 at 09:30:32PM +0500, Andrew Borodin wrote:

2016-12-19 4:21 GMT+05:00 David Fetter <david@fetter.org>:

Couldn't it sleep in increments smaller than a second? Like maybe
milliseconds? Also, it's probably cleaner (or at least more
comprehensible) to write something using format() and dollar quoting
than the line with the double 's.

Right. Here's version which waits for half a second. I do not see how
to path doubles via dollar sign, pg_background_launch() gets a string
as a parameter, it's not EXCEUTE USING.

I see.

I find the following a little easier to follow, and the sleeps still
work even when very short.

Best,
David.

CREATE TABLE input AS
SELECT x, row_number() OVER (ORDER BY x) n
FROM
generate_series(0,.000005,0.000001) x
ORDER BY x DESC;

CREATE TABLE output(place int,value float);

CREATE TABLE handles AS
SELECT pg_background_launch(
format($$
SELECT pg_sleep(%s);
INSERT INTO output VALUES (%s, %s)
$$,
x, n, x
)
) h
FROM input;

--wait until everyone finishes
SELECT * FROM handles JOIN LATERAL pg_background_result(handles.h) AS (x TEXT) ON (true);

--output results
SELECT * FROM output ORDER BY place;

--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#19Simon Riggs
simon@2ndQuadrant.com
In reply to: Robert Haas (#5)
Re: pg_background contrib module proposal

On 9 December 2016 at 13:00, Robert Haas <robertmhaas@gmail.com> wrote:

That might be good, because then we wouldn't have to maintain two
copies of the code.

So why are there two things at all? Why is this being worked on as
well as Peter's patch? What will that give us?

--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#20Robert Haas
robertmhaas@gmail.com
In reply to: Simon Riggs (#19)
Re: pg_background contrib module proposal

On Tue, Dec 20, 2016 at 7:32 PM, Simon Riggs <simon@2ndquadrant.com> wrote:

On 9 December 2016 at 13:00, Robert Haas <robertmhaas@gmail.com> wrote:

That might be good, because then we wouldn't have to maintain two
copies of the code.

So why are there two things at all? Why is this being worked on as
well as Peter's patch? What will that give us?

A feature that can be accessed without writing C code.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#21Andrey Borodin
amborodin@acm.org
In reply to: David Fetter (#18)
#22Craig Ringer
craig@2ndquadrant.com
In reply to: Andrey Borodin (#21)
#23David Fetter
david@fetter.org
In reply to: Craig Ringer (#22)
#24Robert Haas
robertmhaas@gmail.com
In reply to: David Fetter (#23)
#25David Fetter
david@fetter.org
In reply to: Robert Haas (#24)
#26Andrey Borodin
amborodin@acm.org
In reply to: Robert Haas (#24)
#27Amul Sul
sulamul@gmail.com
In reply to: Andrey Borodin (#26)
#28Jim Nasby
Jim.Nasby@BlueTreble.com
In reply to: Amul Sul (#27)
#29Robert Haas
robertmhaas@gmail.com
In reply to: Jim Nasby (#28)
#30Amul Sul
sulamul@gmail.com
In reply to: Robert Haas (#29)
#31Andrey Borodin
amborodin@acm.org
In reply to: Amul Sul (#30)
#32Amul Sul
sulamul@gmail.com
In reply to: Andrey Borodin (#31)
#33Jim Nasby
Jim.Nasby@BlueTreble.com
In reply to: Robert Haas (#29)
#34Amul Sul
sulamul@gmail.com
In reply to: Jim Nasby (#33)
#35Jim Nasby
Jim.Nasby@BlueTreble.com
In reply to: Amul Sul (#34)
#36Amul Sul
sulamul@gmail.com
In reply to: Jim Nasby (#35)
#37Andrey Borodin
amborodin@acm.org
In reply to: Amul Sul (#1)
#38Peter Eisentraut
peter_e@gmx.net
In reply to: Andrey Borodin (#37)
#39Robert Haas
robertmhaas@gmail.com
In reply to: Peter Eisentraut (#38)
#40Andrey Borodin
amborodin@acm.org
In reply to: Robert Haas (#39)
#41Michael Paquier
michael@paquier.xyz
In reply to: Robert Haas (#39)
#42Peter Eisentraut
peter_e@gmx.net
In reply to: Michael Paquier (#41)