Review: Extra Daemons / bgworker
Hello Álvaro,
first of all, thank you for bringing this up again and providing a
patch. My first attempt on that was more than two years ago [1]That was during commit fest 2010-09. Back then, neither extensions, latches nor the walsender existed. The patches had a dependency on imessages, which was not accepted. Other than that, it's a working, pooled bgworker implementation on top of which autovacuum runs just fine. It lacks extensibility and the extra daemons feature, though. For those who missed it:. As the
author of a former bgworker patch, I'd like to provide an additional
review - KaiGai was simply faster to sing up as a reviewer on the
commitfest app...
I started my review is based on rev 6d7e51.. from your bgworker branch
on github, only to figure out later that it wasn't quite up to date. I
upgraded to bgworker-7.patch. I hope that's the most recent version.
# Concept
I appreciate the ability to run daemons that are not connected to
Postgres shared memory. It satisfies a user request that came up several
times when I talked about the bgworkers in Postgres-R.
Another good point is the flexible interface via extensions, even
allowing different starting points for such background workers.
One thing I'd miss (for use of that framework in Postgres-R) is the
ability to start a registered bgworker only upon request, way after the
system started. So that the bgworker process doesn't even exist until it
is really needed. I realize that RegisterBackgroundWorker() is still
necessary in advance to reserve the slots in BackgroundWorkerList and
shared memory.
As a use case independent of Postgres-R, think of something akin to a
worker_spi, but wanting that to perform a task every 24h on hundreds of
databases. You don't want to keep hundreds of processes occupying PGPROC
slots just perform a measly task every 24h.
(We've discussed the ability to let bgworkers re-connect to another
database back then. For one, you'd still have currently unneeded worker
processes around all the time. And second, I think that's hard to get
right - after all, a terminated process is guaranteed to not leak any
stale data into a newly started one, no matter what.)
From my point of view, autovacuum is the very first example of a
background worker process. And I'm a bit puzzled about it not being
properly integrated into this framework. Once you think of autovacuum as
a background job which needs access to Postgres shared memory and a
database, but no client connection, it looks like a bit of code
duplication (and not using code we already have). I realize this kind of
needs the above feature being able to request the (re)start of bgworkers
at arbitrary points in time. However, it would also be a nice test case
for the bgworker infrastructure.
I'd be happy to help with extending the current patch into that
direction, if you agree it's generally useful. Or adapt my bgworker code
accordingly.
# Minor technical issues or questions
In assign_maxconnections, et al, GetNumRegisteredBackgroundWorkers() is
used in relation to MAX_BACKENDS or to calculate MaxBackends. That seems
to "leak" the bgworkers that registered with neither
BGWORKER_SHMEM_ACCESS nor BGWORKER_BACKEND_DATABASE_CONNECTION set. Or
is there any reason to discount such extra daemon processes?
The additional contrib modules auth_counter and worker_spi are missing
from the contrib/Makefile. If that's intentional, they should probably
be listed under "Missing".
The auth_counter module leaves worker.bgw_restart_time uninitialized.
Being an example, it might make sense for auth_counter to provide a
signal that just calls SetLatch() to interrupt WaitLatch() and make
auth_counter emit its log line upon request, i.e. show how to use SIGUSR1.
The bgw_restart_time doesn't always work (certainly not the way I'm
expecting it to). For example, if you forget to pass the
BGWORKER_SHMEM_ACCESS flag, trying to LWLockAcquire() leads to the
worker being restarted immediately and repeatedly - independent of the
bgw_restart_time setting. The same holds true if the bgworker exits with
status 0 or in case it segfaults. Not when exiting with code 1, though.
Why is that? Especially in case of a segfault or equally "hard" errors
that can be expected to occur repeatedly, I don't want the worker to be
restarted that frequently.
# Documentation
There are two working examples in contrib. The auth_counter could use a
header comment similar to worker_spi, quickly describing what it does.
There's no example of a plain extra daemon, without shmem access.
Coding guidelines for bgworker / extra daemon writers are missing. I
read these must not use sleep(), with an explanation in both examples.
Other questions that come to mind: what about signal handlers? fork()?
threads? Can/should it use PG_TRY/PG_CATCH or setjmp()/longjmp(). How to
best load other 3rd party libraries, i.e. for OpenGL processing?
Especially for extra daemons (i.e. bgw_flags = 0), differences to
running an external daemon should be documented. I myself am unclear
about all of the implications that running as a child of the postmaster
has (OOM and cgroups come to mind - there certainly are other aspects).
(I myself still have a hard time finding a proper use case for extra
daemons. I don't want the postmaster to turn into a general purpose
watchdog for everything and the kitchen sink.)
# Tests
No additional automated tests are included - hard to see how that could
be tested automatically, given the lack of a proper test harness.
I hope this review provides useful input.
Regards
Markus Wanner
[1]: That was during commit fest 2010-09. Back then, neither extensions, latches nor the walsender existed. The patches had a dependency on imessages, which was not accepted. Other than that, it's a working, pooled bgworker implementation on top of which autovacuum runs just fine. It lacks extensibility and the extra daemons feature, though. For those who missed it:
latches nor the walsender existed. The patches had a dependency on
imessages, which was not accepted. Other than that, it's a working,
pooled bgworker implementation on top of which autovacuum runs just
fine. It lacks extensibility and the extra daemons feature, though. For
those who missed it:
https://commitfest.postgresql.org/action/patch_view?id=339
http://archives.postgresql.org/message-id/4C3C789C.1040409@bluegap.ch
http://git.postgres-r.org/?p=bgworker;a=summary
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Markus Wanner wrote:
Hi Markus,
Many thanks for your review.
first of all, thank you for bringing this up again and providing a
patch. My first attempt on that was more than two years ago [1]. As the
author of a former bgworker patch, I'd like to provide an additional
review - KaiGai was simply faster to sing up as a reviewer on the
commitfest app...
I remember your patchset. I didn't look at it for this round, for no
particular reason. I did look at KaiGai's submission from two
commitfests ago, and also at a patch from Simon which AFAIK was never
published openly. Simon's patch merged the autovacuum code to make
autovac workers behave like bgworkers as handled by his patch, just like
you suggest. To me it didn't look all that good; I didn't have the guts
for that, hence the separation. If later bgworkers are found to work
well enough, we can "port" autovac workers to use this framework, but
for now it seems to me that the conservative thing is to leave autovac
untouched.
(As an example, we introduced "ilist" some commits back and changed some
uses to it; but there are still many places where we're using SHM_QUEUE,
or List, or open-coded lists, which we could port to the new
infrastructure, but there's no pressing need to do it.)
I started my review is based on rev 6d7e51.. from your bgworker branch
on github, only to figure out later that it wasn't quite up to date. I
upgraded to bgworker-7.patch. I hope that's the most recent version.
Sorry about that -- forgot to push to github. bgworker-7 corresponds to
commit 0a49a540b which I have just pushed to github.
# Concept
I appreciate the ability to run daemons that are not connected to
Postgres shared memory. It satisfies a user request that came up several
times when I talked about the bgworkers in Postgres-R.Another good point is the flexible interface via extensions, even
allowing different starting points for such background workers.
Great.
One thing I'd miss (for use of that framework in Postgres-R) is the
ability to start a registered bgworker only upon request, way after the
system started. So that the bgworker process doesn't even exist until it
is really needed. I realize that RegisterBackgroundWorker() is still
necessary in advance to reserve the slots in BackgroundWorkerList and
shared memory.As a use case independent of Postgres-R, think of something akin to a
worker_spi, but wanting that to perform a task every 24h on hundreds of
databases. You don't want to keep hundreds of processes occupying PGPROC
slots just perform a measly task every 24h.
Yeah, this is something I specifically kept out initially to keep things
simple.
Maybe one thing to do in this area would be to ensure that there is a
certain number of PGPROC elements reserved specifically for bgworkers;
kind of like autovacuum workers have. That would avoid having regular
clients exhausting slots for bgworkers, and vice versa.
How are you envisioning that the requests would occur?
# Minor technical issues or questions
In assign_maxconnections, et al, GetNumRegisteredBackgroundWorkers() is
used in relation to MAX_BACKENDS or to calculate MaxBackends. That seems
to "leak" the bgworkers that registered with neither
BGWORKER_SHMEM_ACCESS nor BGWORKER_BACKEND_DATABASE_CONNECTION set. Or
is there any reason to discount such extra daemon processes?
No, I purposefully let those out, because those don't need a PGPROC. In
fact that seems to me to be the whole point of non-shmem-connected
workers: you can have as many as you like and they won't cause a
backend-side impact. You can use such a worker to connect via libpq to
the server, for example.
The additional contrib modules auth_counter and worker_spi are missing
from the contrib/Makefile. If that's intentional, they should probably
be listed under "Missing".The auth_counter module leaves worker.bgw_restart_time uninitialized.
Being an example, it might make sense for auth_counter to provide a
signal that just calls SetLatch() to interrupt WaitLatch() and make
auth_counter emit its log line upon request, i.e. show how to use SIGUSR1.
KaiGai proposed that we remove auth_counter. I don't see why not; I
mean, worker_spi is already doing most of what auth_counter is doing, so
why not? However, as you say, maybe we need more coding examples.
The bgw_restart_time doesn't always work (certainly not the way I'm
expecting it to). For example, if you forget to pass the
BGWORKER_SHMEM_ACCESS flag, trying to LWLockAcquire() leads to the
worker being restarted immediately and repeatedly - independent of the
bgw_restart_time setting. The same holds true if the bgworker exits with
status 0 or in case it segfaults. Not when exiting with code 1, though.
Why is that? Especially in case of a segfault or equally "hard" errors
that can be expected to occur repeatedly, I don't want the worker to be
restarted that frequently.
Ah, that's just a bug, of course.
--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 11/28/2012 03:51 PM, Alvaro Herrera wrote:
I remember your patchset. I didn't look at it for this round, for no
particular reason. I did look at KaiGai's submission from two
commitfests ago, and also at a patch from Simon which AFAIK was never
published openly. Simon's patch merged the autovacuum code to make
autovac workers behave like bgworkers as handled by his patch, just like
you suggest. To me it didn't look all that good; I didn't have the guts
for that, hence the separation. If later bgworkers are found to work
well enough, we can "port" autovac workers to use this framework, but
for now it seems to me that the conservative thing is to leave autovac
untouched.
Hm.. interesting to see how the same idea grows into different patches
and gets refined until we end up with something considered committable.
Do you remember anything in particular that didn't look good? Would you
help reviewing a patch on top of bgworker-7 that changed autovacuum into
a bgworker?
(As an example, we introduced "ilist" some commits back and changed some
uses to it; but there are still many places where we're using SHM_QUEUE,
or List, or open-coded lists, which we could port to the new
infrastructure, but there's no pressing need to do it.)
Well, I usually like cleaning things up earlier rather than later (my
desk clearly being an exception to that rule, though). But yeah, new
code usually brings new bugs with it.
Sorry about that -- forgot to push to github. bgworker-7 corresponds to
commit 0a49a540b which I have just pushed to github.
Thanks.
Maybe one thing to do in this area would be to ensure that there is a
certain number of PGPROC elements reserved specifically for bgworkers;
kind of like autovacuum workers have. That would avoid having regular
clients exhausting slots for bgworkers, and vice versa.
Yeah, I think that's mandatory, anyways, see below.
How are you envisioning that the requests would occur?
Just like av_launcher does it now: set a flag in shared memory and
signal the postmaster (PMSIGNAL_START_AUTOVAC_WORKER).
(That's why I'm so puzzled: it looks like it's pretty much all there,
already. I even remember a discussion about that mechanism probably not
being fast enough to spawn bgworkers. And a proposal to add multiple
such flags, so an avlauncher-like daemon could ask for multiple
bgworkers to be started in parallel. I've even measured the serial
bgworker fork rate back then, IIRC it was in the hundreds of forks per
second...)
# Minor technical issues or questions
In assign_maxconnections, et al, GetNumRegisteredBackgroundWorkers() is
used in relation to MAX_BACKENDS or to calculate MaxBackends. That seems
to "leak" the bgworkers that registered with neither
BGWORKER_SHMEM_ACCESS nor BGWORKER_BACKEND_DATABASE_CONNECTION set. Or
is there any reason to discount such extra daemon processes?No, I purposefully let those out, because those don't need a PGPROC. In
fact that seems to me to be the whole point of non-shmem-connected
workers: you can have as many as you like and they won't cause a
backend-side impact. You can use such a worker to connect via libpq to
the server, for example.
Hm.. well, in that case, the shmem-connected ones are *not* counted. If
I create and run an extensions that registers 100 shmem-connected
bgworkers, I cannot connect to a system with max_connections=100
anymore, because bgworkers then occupy all of the connections, already.
Please add the registered shmem-connected bgworkers to the
max_connections limit. I think it's counter intuitive to have autovacuum
workers reserved separately, but extension's bgworkers eat (client)
connections. Or put another way: max_connections should always be the
max number of *client* connections the DBA wants to allow.
(Or, if that's in some way complicated, please give the DBA an
additional GUC akin to max_background_workers. That can be merged with
the current max_autovacuum_workers, once/if we rebase autovaccum onto
bgworkers).
The additional contrib modules auth_counter and worker_spi are missing
from the contrib/Makefile. If that's intentional, they should probably
be listed under "Missing".The auth_counter module leaves worker.bgw_restart_time uninitialized.
Being an example, it might make sense for auth_counter to provide a
signal that just calls SetLatch() to interrupt WaitLatch() and make
auth_counter emit its log line upon request, i.e. show how to use SIGUSR1.KaiGai proposed that we remove auth_counter. I don't see why not; I
mean, worker_spi is already doing most of what auth_counter is doing, so
why not?
Agreed.
However, as you say, maybe we need more coding examples.
Maybe a minimally usable extra daemon example? Showing how to avoid
common pitfalls? Use cases, anybody? :-)
Ah, that's just a bug, of course.
I see. Glad my review found it.
Regards
Markus Wanner
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Markus Wanner <markus@bluegap.ch> writes:
However, as you say, maybe we need more coding examples.
Maybe a minimally usable extra daemon example? Showing how to avoid
common pitfalls? Use cases, anybody? :-)
What about the PGQ ticker, pgqd?
https://github.com/markokr/skytools/tree/master/sql/ticker
https://github.com/markokr/skytools/blob/master/sql/ticker/pgqd.c
Or maybe pgAgent, which seems to live there, but is in C++ so might need
a rewrite to the specs:
Maybe it would be easier to have a version of GNU mcron as an extension,
with the abitity to fire PostgreSQL stored procedures directly? (That
way the cron specific parts of the logic are already implemented)
http://www.gnu.org/software/mcron/
Another idea would be to have a pgbouncer extension. We would still need
of course to have pgbouncer as a separate component so that client
connection can outlive a postmaster crash, but that would still be very
useful as a first step into admission control. Let's not talk about the
feedback loop and per-cluster resource usage monitoring yet, but I guess
that you can see the drift.
Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
2012/11/30 Dimitri Fontaine <dimitri@2ndquadrant.fr>:
Markus Wanner <markus@bluegap.ch> writes:
However, as you say, maybe we need more coding examples.
Maybe a minimally usable extra daemon example? Showing how to avoid
common pitfalls? Use cases, anybody? :-)What about the PGQ ticker, pgqd?
https://github.com/markokr/skytools/tree/master/sql/ticker
https://github.com/markokr/skytools/blob/master/sql/ticker/pgqd.cOr maybe pgAgent, which seems to live there, but is in C++ so might need
a rewrite to the specs:Maybe it would be easier to have a version of GNU mcron as an extension,
with the abitity to fire PostgreSQL stored procedures directly? (That
way the cron specific parts of the logic are already implemented)http://www.gnu.org/software/mcron/
Another idea would be to have a pgbouncer extension. We would still need
of course to have pgbouncer as a separate component so that client
connection can outlive a postmaster crash, but that would still be very
useful as a first step into admission control. Let's not talk about the
feedback loop and per-cluster resource usage monitoring yet, but I guess
that you can see the drift.
both will be nice
Pavel
Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 11/30/2012 11:09 AM, Dimitri Fontaine wrote:
Markus Wanner <markus@bluegap.ch> writes:
However, as you say, maybe we need more coding examples.
Maybe a minimally usable extra daemon example? Showing how to avoid
common pitfalls? Use cases, anybody? :-)What about the PGQ ticker, pgqd?
https://github.com/markokr/skytools/tree/master/sql/ticker
https://github.com/markokr/skytools/blob/master/sql/ticker/pgqd.c
AFAICS pgqd currently uses libpq, so I think it would rather turn into
what I call a background worker, with a connection to Postgres shared
memory. I perfectly well see use cases (plural!) for those.
What I'm questioning is the use for what I rather call "extra daemons",
that is, additional processes started by the postmaster without a
connection to Postgres shared memory (and thus without a database
connection).
I was asking for a minimal example of such an extra daemon, similar to
worker_spi, showing how to properly write such a thing. Ideally showing
how to avoid common pitfalls.
Maybe it would be easier to have a version of GNU mcron as an extension,
with the abitity to fire PostgreSQL stored procedures directly? (That
way the cron specific parts of the logic are already implemented)
Again, that's something that would eventually require a database
connection. Or at least access to Postgres shared memory to eventually
start the required background jobs. (Something Alvaro's patch doesn't
implement, yet. But which exactly matches what the coordinator and
bgworkers in Postgres-R do.)
For ordinary extra daemons, I'm worried about things like an OOM killer
deciding to kill the postmaster, being its parent. Or (io)niceness
settings. Or Linux cgroups. IMO all of these things just get harder to
administrate for extra daemons, when they move under the hat of the
postmaster.
Without access to shared memory, the only thing an extra daemon would
gain by moving under postmaster is the "knowledge" of the start, stop
and restart (crash) events of the database. And that in a very
inflexible way: the extra process is forced to start, stop and restart
together with the database to "learn" about these events.
Using a normal client connection arguably is a better way to learn about
crash events - it doesn't have the above limitation. And the start and
stop events, well, the DBA or sysadmin is under control of these,
already. We can possibly improve on that, yes. But extra daemons are not
the way to go, IMO.
Another idea would be to have a pgbouncer extension. We would still need
of course to have pgbouncer as a separate component so that client
connection can outlive a postmaster crash, but that would still be very
useful as a first step into admission control. Let's not talk about the
feedback loop and per-cluster resource usage monitoring yet, but I guess
that you can see the drift.
Sorry, I don't. Especially not with something like pgbouncer, because I
definitely *want* to control and administrate that separately.
So I guess this is a vote to disallow `worker.bgw_flags = 0` on the
basis that everything a process, which doesn't need to access Postgres
shared memory, better does whatever it does outside of Postgres. For the
benefit of the stability of Postgres and for ease of administration of
the two.
Or, maybe, rather drop the BGWORKER_SHMEM_ACCESS flag and imply that
every registered process wants to have access to Postgres shared memory.
Then document the gotchas and requirements of living under the
Postmaster, as I've requested before. (If you want a foot gun, you can
still write an extension that doesn't need to access Postgres shared
memory, but hey.. I we can't help those who desperately try to shoot
their foot).
Regards
Markus Wanner
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Markus Wanner <markus@bluegap.ch> writes:
AFAICS pgqd currently uses libpq, so I think it would rather turn into
what I call a background worker, with a connection to Postgres shared
memory. I perfectly well see use cases (plural!) for those.What I'm questioning is the use for what I rather call "extra daemons",
that is, additional processes started by the postmaster without a
connection to Postgres shared memory (and thus without a database
connection).
I totally missed the need to connect to shared memory to be able to
connect to a database and query it. Can't we just link the bgworkder
code to the client libpq library, just as plproxy is doing I believe?
Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Dimitri Fontaine wrote:
Markus Wanner <markus@bluegap.ch> writes:
AFAICS pgqd currently uses libpq, so I think it would rather turn into
what I call a background worker, with a connection to Postgres shared
memory. I perfectly well see use cases (plural!) for those.What I'm questioning is the use for what I rather call "extra daemons",
that is, additional processes started by the postmaster without a
connection to Postgres shared memory (and thus without a database
connection).I totally missed the need to connect to shared memory to be able to
connect to a database and query it. Can't we just link the bgworkder
code to the client libpq library, just as plproxy is doing I believe?
One of the uses for bgworkers that don't have shmem connection is to
have them use libpq connections instead. I don't really see the point
of forcing everyone to use backend connections when libpq connections
are enough. In particular, they are easier to port from existing code;
and they make it easier to share code with systems that still have to
support older PG versions.
--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2012-11-30 09:57:20 -0300, Alvaro Herrera wrote:
Dimitri Fontaine wrote:
Markus Wanner <markus@bluegap.ch> writes:
AFAICS pgqd currently uses libpq, so I think it would rather turn into
what I call a background worker, with a connection to Postgres shared
memory. I perfectly well see use cases (plural!) for those.What I'm questioning is the use for what I rather call "extra daemons",
that is, additional processes started by the postmaster without a
connection to Postgres shared memory (and thus without a database
connection).I totally missed the need to connect to shared memory to be able to
connect to a database and query it. Can't we just link the bgworkder
code to the client libpq library, just as plproxy is doing I believe?One of the uses for bgworkers that don't have shmem connection is to
have them use libpq connections instead. I don't really see the point
of forcing everyone to use backend connections when libpq connections
are enough. In particular, they are easier to port from existing code;
and they make it easier to share code with systems that still have to
support older PG versions.
They also can get away with a lot more crazy stuff without corrupting
the database. You better know something about what youre doing before
doing something with direct shared memory access.
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Andres Freund <andres@2ndquadrant.com> writes:
One of the uses for bgworkers that don't have shmem connection is to
have them use libpq connections instead. I don't really see the point
of forcing everyone to use backend connections when libpq connections
are enough. In particular, they are easier to port from existing code;
and they make it easier to share code with systems that still have to
support older PG versions.
Exactly, I think most bgworker would just use libpq if that's available,
using a backend's infrastructure is not that good a fit here. I mean,
connect from your worker to a database using libpq and call a backend's
function (provided by the same extension I guess) in there.
That's how I think pgqd would get integrated into the worker
infrastructure, right?
They also can get away with a lot more crazy stuff without corrupting
the database. You better know something about what youre doing before
doing something with direct shared memory access.
And there's a whole lot you can already do just with a C coded stored
procedure already.
Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
2012/11/30 Dimitri Fontaine <dimitri@2ndquadrant.fr>:
Andres Freund <andres@2ndquadrant.com> writes:
One of the uses for bgworkers that don't have shmem connection is to
have them use libpq connections instead. I don't really see the point
of forcing everyone to use backend connections when libpq connections
are enough. In particular, they are easier to port from existing code;
and they make it easier to share code with systems that still have to
support older PG versions.Exactly, I think most bgworker would just use libpq if that's available,
using a backend's infrastructure is not that good a fit here. I mean,
connect from your worker to a database using libpq and call a backend's
function (provided by the same extension I guess) in there.That's how I think pgqd would get integrated into the worker
infrastructure, right?
One thing we have to pay attention is, the backend code cannot distinguish
connection from pgworker via libpq from other regular connections, from
perspective of access control.
Even if we implement major portion with C-function, do we have a reliable way
to prohibit C-function being invoked with user's query?
I also plan to use bgworker to load data chunk from shared_buffer to GPU
device in parallel, it shall be performed under the regular access control
stuff.
I think, using libpq is a good "option" for 95% of development, however, it
also should be possible to use SPI interface for corner case usage.
Thanks,
--
KaiGai Kohei <kaigai@kaigai.gr.jp>
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 11/30/2012 01:40 PM, Dimitri Fontaine wrote:
I totally missed the need to connect to shared memory to be able to
connect to a database and query it. Can't we just link the bgworkder
code to the client libpq library, just as plproxy is doing I believe?
Well, sure you can use libpq. But so can external processes. So that's
no benefit of running under the postmaster.
Regards
Markus Wanner
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 11/30/2012 01:59 PM, Andres Freund wrote:
On 2012-11-30 09:57:20 -0300, Alvaro Herrera wrote:
One of the uses for bgworkers that don't have shmem connection is to
have them use libpq connections instead. I don't really see the point
of forcing everyone to use backend connections when libpq connections
are enough.
Requiring a libpq connection is a good indication for *not* wanting the
process to run under the postmaster, IMO.
In particular, they are easier to port from existing code;
and they make it easier to share code with systems that still have to
support older PG versions.They also can get away with a lot more crazy stuff without corrupting
the database.
Exactly. That's a good reason to *not* tie that to the postmaster, then.
Please keep as much of the potentially dangerous stuff separate (and
advice developers to do so as well, instead of offering them a foot
gun). So that our postmaster can do its job. And do it reliably, without
trying to be a general purpose start/stop daemon. There are better and
well established tools for that.
Regards
Markus Wanner
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Kohei KaiGai <kaigai@kaigai.gr.jp> writes:
One thing we have to pay attention is, the backend code cannot distinguish
connection from pgworker via libpq from other regular connections, from
perspective of access control.
Even if we implement major portion with C-function, do we have a reliable way
to prohibit C-function being invoked with user's query?
Why would you want to do that? And maybe the way to enforce that would
be by having your extension do its connecting using SPI to be able to
place things in known pieces of memory for the function to check?
I also plan to use bgworker to load data chunk from shared_buffer to GPU
device in parallel, it shall be performed under the regular access control
stuff.
That sounds like a job where you need shared memory access but maybe not
a database connection?
I think, using libpq is a good "option" for 95% of development, however, it
also should be possible to use SPI interface for corner case usage.
+1, totally agreed.
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Markus Wanner wrote:
On 11/30/2012 01:40 PM, Dimitri Fontaine wrote:
I totally missed the need to connect to shared memory to be able to
connect to a database and query it. Can't we just link the bgworkder
code to the client libpq library, just as plproxy is doing I believe?Well, sure you can use libpq. But so can external processes. So that's
no benefit of running under the postmaster.
No, it's not a benefit of that; but such a process would get started up
when postmaster is started, and shut down when postmaster stops. So it
makes easier to have processes that need to run alongside postmaster.
--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
2012/11/30 Dimitri Fontaine <dimitri@2ndquadrant.fr>:
Kohei KaiGai <kaigai@kaigai.gr.jp> writes:
One thing we have to pay attention is, the backend code cannot distinguish
connection from pgworker via libpq from other regular connections, from
perspective of access control.
Even if we implement major portion with C-function, do we have a reliable way
to prohibit C-function being invoked with user's query?Why would you want to do that? And maybe the way to enforce that would
be by having your extension do its connecting using SPI to be able to
place things in known pieces of memory for the function to check?
As long as SPI is an option of bgworker, I have nothing to argue here.
If the framework enforced extension performing as background worker using
libpq for database connection, a certain kind of works being tied with internal
stuff has to be implemented as C-functions. Thus, I worried about it.
I also plan to use bgworker to load data chunk from shared_buffer to GPU
device in parallel, it shall be performed under the regular access control
stuff.That sounds like a job where you need shared memory access but maybe not
a database connection?
Both of them are needed in this case. This "io-worker" will perform according
to the request from regular backend process, and fetch tuples from the heap
to the DMA buffer being on shared memory.
I think, using libpq is a good "option" for 95% of development, however, it
also should be possible to use SPI interface for corner case usage.+1, totally agreed.
Thanks,
--
KaiGai Kohei <kaigai@kaigai.gr.jp>
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Alvaro,
On 11/30/2012 02:44 PM, Alvaro Herrera wrote:
So it
makes easier to have processes that need to run alongside postmaster.
That's where we are in respectful disagreement, then. As I don't think
that's easier, overall, but in my eye, this looks like a foot gun.
As long as things like pgbouncer, pgqd, etc.. keep to be available as
separate daemons, I'm happy, though.
Regards
Markus Wanner
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
2012/11/30 Markus Wanner <markus@bluegap.ch>:
Alvaro,
On 11/30/2012 02:44 PM, Alvaro Herrera wrote:
So it
makes easier to have processes that need to run alongside postmaster.That's where we are in respectful disagreement, then. As I don't think
that's easier, overall, but in my eye, this looks like a foot gun.As long as things like pgbouncer, pgqd, etc.. keep to be available as
separate daemons, I'm happy, though.
This feature does not enforce them to implement with this new framework.
If they can perform as separate daemons, it is fine enough.
But it is not all the cases where we want background workers being tied
with postmaster's duration.
--
KaiGai Kohei <kaigai@kaigai.gr.jp>
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 11/30/2012 03:16 PM, Kohei KaiGai wrote:
This feature does not enforce them to implement with this new framework.
If they can perform as separate daemons, it is fine enough.
I'm not clear on what exactly you envision, but if a process needs
access to shared buffers, it sounds like it should be a bgworker. I
don't quite understand why that process also wants a libpq connection,
but that's certainly doable.
But it is not all the cases where we want background workers being tied
with postmaster's duration.
Not wanting a process to be tied to postmaster's duration is a strong
indication that it better not be a bgworker, I think.
Regards
Markus Wanner
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
2012/11/30 Markus Wanner <markus@bluegap.ch>:
On 11/30/2012 03:16 PM, Kohei KaiGai wrote:
This feature does not enforce them to implement with this new framework.
If they can perform as separate daemons, it is fine enough.I'm not clear on what exactly you envision, but if a process needs
access to shared buffers, it sounds like it should be a bgworker. I
don't quite understand why that process also wants a libpq connection,
but that's certainly doable.
It seemed to me you are advocating that any use case of background-
worker can be implemented with existing separate daemon approach.
What I wanted to say is, we have some cases that background-worker
framework allows to implement such kind of extensions with more
reasonable design.
Yes, one reason I want to use background-worker is access to shared-
memory segment. Also, it want to connect databases simultaneously
out of access controls; like as autovacuum. It is a reason why I'm saying
SPI interface should be only an option, not only libpq.
(If extension choose libpq, it does not take anything special, does it?)
But it is not all the cases where we want background workers being tied
with postmaster's duration.Not wanting a process to be tied to postmaster's duration is a strong
indication that it better not be a bgworker, I think.
It also probably means, in case when a process whose duration wants to
be tied with duration of postmaster, its author can consider to implement
it as background worker.
Thanks,
--
KaiGai Kohei <kaigai@kaigai.gr.jp>
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 11/30/2012 03:58 PM, Kohei KaiGai wrote:
It seemed to me you are advocating that any use case of background-
worker can be implemented with existing separate daemon approach.
That sounds like a misunderstanding. All I'm advocating is that only
3rd-party processes with a real need (like accessing shared memory)
should run under the postmaster.
What I wanted to say is, we have some cases that background-worker
framework allows to implement such kind of extensions with more
reasonable design.
I absolutely agree to that. And I think I can safely claim to be the
first person to publish a patch that provides some kind of background
worker infrastructure for Postgres.
Yes, one reason I want to use background-worker is access to shared-
memory segment. Also, it want to connect databases simultaneously
out of access controls; like as autovacuum.
Yeah, that's the entire reason for background workers. For clarity and
differentiation, I'd add: .. without having a client connection. That's
what makes them *background* workers. (Not to be confused with the
frontend vs backend differentiation. They are background backends, if
you want).
It is a reason why I'm saying
SPI interface should be only an option, not only libpq.
I'm extending that to say extensions should better *not* use libpq.
After all, they have a more direct access, already.
It also probably means, in case when a process whose duration wants to
be tied with duration of postmaster, its author can consider to implement
it as background worker.
I personally don't count that as a real need. There are better tools for
this job; while there clearly are dangers in (ab)using the postmaster to
do it.
Regards
Markus Wanner
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Markus Wanner wrote:
On 11/28/2012 03:51 PM, Alvaro Herrera wrote:
I remember your patchset. I didn't look at it for this round, for no
particular reason. I did look at KaiGai's submission from two
commitfests ago, and also at a patch from Simon which AFAIK was never
published openly. Simon's patch merged the autovacuum code to make
autovac workers behave like bgworkers as handled by his patch, just like
you suggest. To me it didn't look all that good; I didn't have the guts
for that, hence the separation. If later bgworkers are found to work
well enough, we can "port" autovac workers to use this framework, but
for now it seems to me that the conservative thing is to leave autovac
untouched.Hm.. interesting to see how the same idea grows into different patches
and gets refined until we end up with something considered committable.Do you remember anything in particular that didn't look good? Would you
help reviewing a patch on top of bgworker-7 that changed autovacuum into
a bgworker?
I'm not really that interested in it currently ... and there are enough
other patches to review. I would like bgworkers to mature a bit more
and get some heavy real world testing before we change autovacuum.
How are you envisioning that the requests would occur?
Just like av_launcher does it now: set a flag in shared memory and
signal the postmaster (PMSIGNAL_START_AUTOVAC_WORKER).
I'm not sure how this works. What process is in charge of setting such
a flag?
In assign_maxconnections, et al, GetNumRegisteredBackgroundWorkers() is
used in relation to MAX_BACKENDS or to calculate MaxBackends. That seems
to "leak" the bgworkers that registered with neither
BGWORKER_SHMEM_ACCESS nor BGWORKER_BACKEND_DATABASE_CONNECTION set. Or
is there any reason to discount such extra daemon processes?No, I purposefully let those out, because those don't need a PGPROC. In
fact that seems to me to be the whole point of non-shmem-connected
workers: you can have as many as you like and they won't cause a
backend-side impact. You can use such a worker to connect via libpq to
the server, for example.Hm.. well, in that case, the shmem-connected ones are *not* counted. If
I create and run an extensions that registers 100 shmem-connected
bgworkers, I cannot connect to a system with max_connections=100
anymore, because bgworkers then occupy all of the connections, already.
This is actually a very silly bug: it turns out that guc.c obtains the
count of workers before workers have actually registered. So this
necessitates some reshuffling.
Or put another way: max_connections should always be the
max number of *client* connections the DBA wants to allow.
Completely agreed.
--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 12/03/2012 03:28 PM, Alvaro Herrera wrote:
I'm not really that interested in it currently ... and there are enough
other patches to review. I would like bgworkers to mature a bit more
and get some heavy real world testing before we change autovacuum.
Understood.
Just like av_launcher does it now: set a flag in shared memory and
signal the postmaster (PMSIGNAL_START_AUTOVAC_WORKER).I'm not sure how this works. What process is in charge of setting such
a flag?
The only process that currently starts background workers ... ehm ...
autovacuum workers is the autovacuum launcher. It uses the above
Postmaster Signal in autovacuum.c:do_start_autovacuum_worker() to have
the postmaster launch bg/autovac workers on demand.
(And yes, this kind of is an exception to the rule that the postmaster
must not rely on shared memory. However, these are just flags we write
atomically, see pmsignal.[ch])
I'd like to extend that, so that other processes can request to start
(pre-registered) background workers more dynamically. I'll wait for an
update of your patch, though.
This is actually a very silly bug: it turns out that guc.c obtains the
count of workers before workers have actually registered. So this
necessitates some reshuffling.
Okay, thanks.
Regards
Markus Wanner
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 3 December 2012 15:17, Markus Wanner <markus@bluegap.ch> wrote:
Just like av_launcher does it now: set a flag in shared memory and
signal the postmaster (PMSIGNAL_START_AUTOVAC_WORKER).I'm not sure how this works. What process is in charge of setting such
a flag?The only process that currently starts background workers ... ehm ...
autovacuum workers is the autovacuum launcher. It uses the above
Postmaster Signal in autovacuum.c:do_start_autovacuum_worker() to have
the postmaster launch bg/autovac workers on demand.
My understanding was that the patch keep autovac workers and
background workers separate at this point.
Is there anything to be gained *now* from merging those two concepts?
I saw that as refactoring that can occur once we are happy it should
take place, but isn't necessary.
--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Markus Wanner wrote:
On 12/03/2012 03:28 PM, Alvaro Herrera wrote:
Just like av_launcher does it now: set a flag in shared memory and
signal the postmaster (PMSIGNAL_START_AUTOVAC_WORKER).I'm not sure how this works. What process is in charge of setting such
a flag?The only process that currently starts background workers ... ehm ...
autovacuum workers is the autovacuum launcher. It uses the above
Postmaster Signal in autovacuum.c:do_start_autovacuum_worker() to have
the postmaster launch bg/autovac workers on demand.
Oh, I understand that. My question was more about what mechanism are
you envisioning for new bgworkers. What process would be in charge of
sending such signals? Would it be a persistent bgworker which would
decide to start up other bgworkers based on external conditions such as
timing? And how would postmaster know which bgworker to start -- would
it use the bgworker's name to find it in the registered workers list?
The trouble with the above rough sketch is how does the coordinating
bgworker communicate with postmaster. Autovacuum has a very, um,
peculiar mechanism to make this work: avlauncher sends a signal (which
has a hardcoded-in-core signal number) and postmaster knows to start a
generic avworker; previously avlauncher has set things up in shared
memory, and the generic avworker knows where to look to morph into the
specific bgworker required. So postmaster never really looks at shared
memory other than the signal number (which is the only use of shmem in
postmaster that's acceptable, because it's been carefully coded to be
robust). This doesn't work for generic modules because we don't have a
hardcoded signal number; if two modules wanted to start up generic
bgworkers, how would postmaster know which worker-main function to call?
Now, maybe we can make this work in some robust fashion ("robust"
meaning we don't put postmaster at risk, which is of utmost importance;
and this in turn means don't trust anything in shared memory.) I don't
say it's impossible; only that it needs some more thought and careful
design.
As posted, the feature is already useful and it'd be good to have it
committed soon so that others can experiment with whatever sample
bgworkers they see fit. That will give us more insight on other API
refinements we might need.
I'm going to disappear on paternity leave, most likely later this week,
or early next week; I would like to commit this patch before that. When
I'm back we can discuss other improvements. That will give us several
weeks before the end of the 9.3 development period to get these in. Of
course, other committers are welcome to improve the code in my absence.
--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Simon Riggs wrote:
On 3 December 2012 15:17, Markus Wanner <markus@bluegap.ch> wrote:
The only process that currently starts background workers ... ehm ...
autovacuum workers is the autovacuum launcher. It uses the above
Postmaster Signal in autovacuum.c:do_start_autovacuum_worker() to have
the postmaster launch bg/autovac workers on demand.My understanding was that the patch keep autovac workers and
background workers separate at this point.
That is correct.
Is there anything to be gained *now* from merging those two concepts?
I saw that as refactoring that can occur once we are happy it should
take place, but isn't necessary.
IMO it's a net loss in robustness of the autovac implementation.
--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 12/03/2012 04:27 PM, Simon Riggs wrote:
My understanding was that the patch keep autovac workers and
background workers separate at this point.
I agree to that, for this patch. However, that might not keep me from
trying to (re-)sumbit some of the bgworker patches in my queue. :-)
Regards
Markus Wanner
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 12/03/2012 04:44 PM, Alvaro Herrera wrote:
Simon Riggs wrote:
Is there anything to be gained *now* from merging those two concepts?
I saw that as refactoring that can occur once we are happy it should
take place, but isn't necessary.IMO it's a net loss in robustness of the autovac implementation.
Agreed.
That's only one aspect of it, though. From the other point of view, it
would be a proof of stability for the bgworker implementation if
autovacuum worked on top of it.
Regards
Markus Wanner
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Alvaro,
in general, please keep in mind that there are two aspects that I tend
to mix and confuse: one is what's implemented and working for
Postgres-R. The other this is how I envision (parts of it) to be merged
back into Postgres, itself. Sorry if that causes confusion.
On 12/03/2012 04:43 PM, Alvaro Herrera wrote:
Oh, I understand that. My question was more about what mechanism are
you envisioning for new bgworkers. What process would be in charge of
sending such signals? Would it be a persistent bgworker which would
decide to start up other bgworkers based on external conditions such as
timing? And how would postmaster know which bgworker to start -- would
it use the bgworker's name to find it in the registered workers list?
Well, in the Postgres-R case, I've extended the autovacuum launcher to a
more general purpose job scheduler, named coordinator. Lacking your
bgworker patch, it is the only process that is able to launch background
workers.
Your work now allows extensions to register background workers. If I
merge the two concepts, I can easily imagine allowing other (bgworker)
processes to launch bgworkers.
Another thing I can imagine is turning that coordinator into something
that can schedule and trigger jobs registered by extensions (or even
user defined ones). Think: cron daemon for SQL jobs in the background.
(After all, that's pretty close to what the autovacuum launcher does,
already.)
The trouble with the above rough sketch is how does the coordinating
bgworker communicate with postmaster.
I know. I've gone through that pain.
In Postgres-R, I've solved this with imessages (which was the primary
reason for rejection of the bgworker patches back in 2010).
The postmaster only needs to starts *a* background worker process. That
process itself then has to figure out (by checking its imessage queue)
what job it needs to perform. Or if you think in terms of bgworker
types: what type of bgworker it needs to be.
Autovacuum has a very, um,
peculiar mechanism to make this work: avlauncher sends a signal (which
has a hardcoded-in-core signal number) and postmaster knows to start a
generic avworker; previously avlauncher has set things up in shared
memory, and the generic avworker knows where to look to morph into the
specific bgworker required. So postmaster never really looks at shared
memory other than the signal number (which is the only use of shmem in
postmaster that's acceptable, because it's been carefully coded to be
robust).
In Postgres-R, I've extended exactly that mechanism to work for other
jobs that autovacuum.
This doesn't work for generic modules because we don't have a
hardcoded signal number; if two modules wanted to start up generic
bgworkers, how would postmaster know which worker-main function to call?
You've just described above how this works for autovacuum: the
postmaster doesn't *need* to know. Leave it to the bgworker to determine
what kind of task it needs to perform.
As posted, the feature is already useful and it'd be good to have it
committed soon so that others can experiment with whatever sample
bgworkers they see fit. That will give us more insight on other API
refinements we might need.
I completely agree. I didn't ever intend to provide an alternative patch
or hold you back. (Except for the extra daemon issue, where we disagree,
but that's not a reason to keep this feature back). So please, go ahead
and commit this feature (once the issues I've mentioned in the review
are fixed).
Please consider all of these plans or ideas in here (or in Postgres-R)
as extending on your work, rather than competing against.
I'm going to disappear on paternity leave, most likely later this week,
or early next week; I would like to commit this patch before that. When
I'm back we can discuss other improvements. That will give us several
weeks before the end of the 9.3 development period to get these in. Of
course, other committers are welcome to improve the code in my absence.
Okay, thanks for sharing that. I'd certainly appreciate your inputs on
further refinements for bgworkers and/or autovacuum.
Regards
Markus Wanner
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
So here's version 8. This fixes a couple of bugs and most notably
creates a separate PGPROC list for bgworkers, so that they don't
interfere with client-connected backends.
I tested starting 1000 non-shmem attached bgworkers -- postmaster
doesn't break a sweat. Also running 200 backend-connected bgworkers
works fine (assuming they do nothing). This is on my laptop, which is a
dual-core Intel i5 processor.
One notable thing is that I had to introduce this in the postmaster
startup sequence:
/*
* process any libraries that should be preloaded at postmaster start
*/
process_shared_preload_libraries();
/*
* If loadable modules have added background workers, MaxBackends needs to
* be updated. Do so now.
*/
// RerunAssignHook("max_connections");
if (GetNumShmemAttachedBgworkers() > 0)
SetConfigOption("max_connections",
GetConfigOption("max_connections", false, false),
PGC_POSTMASTER, PGC_S_OVERRIDE);
Note the intention here is to re-run the GUC assign hook for
max_connections (hence the commented out hypothetical call to do so).
Obviously, having to go through GetConfigOption and SetConfigOption is
not a nice thing to do; we'll have to add some new entry point to guc.c
for this to have a nicer interface.
(I also observed that it's probably a good idea to have something like
FunctionSetConfigOption for the places that are currently calling
set_config_option directly).
--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Attachments:
bgworker-8.patchtext/x-diff; charset=us-asciiDownload
*** a/contrib/Makefile
--- b/contrib/Makefile
***************
*** 50,56 **** SUBDIRS = \
test_parser \
tsearch2 \
unaccent \
! vacuumlo
ifeq ($(with_openssl),yes)
SUBDIRS += sslinfo
--- 50,57 ----
test_parser \
tsearch2 \
unaccent \
! vacuumlo \
! worker_spi
ifeq ($(with_openssl),yes)
SUBDIRS += sslinfo
*** /dev/null
--- b/contrib/worker_spi/Makefile
***************
*** 0 ****
--- 1,14 ----
+ # contrib/worker_spi/Makefile
+
+ MODULES = worker_spi
+
+ ifdef USE_PGXS
+ PG_CONFIG = pg_config
+ PGXS := $(shell $(PG_CONFIG) --pgxs)
+ include $(PGXS)
+ else
+ subdir = contrib/worker_spi
+ top_builddir = ../..
+ include $(top_builddir)/src/Makefile.global
+ include $(top_srcdir)/contrib/contrib-global.mk
+ endif
*** /dev/null
--- b/contrib/worker_spi/worker_spi.c
***************
*** 0 ****
--- 1,263 ----
+ /* -------------------------------------------------------------------------
+ *
+ * worker_spi.c
+ * Sample background worker code that demonstrates usage of a database
+ * connection.
+ *
+ * This code connects to a database, create a schema and table, and summarizes
+ * the numbers contained therein. To see it working, insert an initial value
+ * with "total" type and some initial value; then insert some other rows with
+ * "delta" type. Delta rows will be deleted by this worker and their values
+ * aggregated into the total.
+ *
+ * Copyright (C) 2012, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * contrib/worker_spi/worker_spi.c
+ *
+ * -------------------------------------------------------------------------
+ */
+ #include "postgres.h"
+
+ /* These are always necessary for a bgworker */
+ #include "miscadmin.h"
+ #include "postmaster/bgworker.h"
+ #include "storage/ipc.h"
+ #include "storage/latch.h"
+ #include "storage/lwlock.h"
+ #include "storage/proc.h"
+ #include "storage/shmem.h"
+
+ /* these headers are used by this particular worker's code */
+ #include "access/xact.h"
+ #include "executor/spi.h"
+ #include "fmgr.h"
+ #include "lib/stringinfo.h"
+ #include "utils/builtins.h"
+ #include "utils/snapmgr.h"
+
+ PG_MODULE_MAGIC;
+
+ void _PG_init(void);
+
+ static bool got_sigterm = false;
+
+
+ typedef struct worktable
+ {
+ const char *schema;
+ const char *name;
+ } worktable;
+
+ static void
+ worker_spi_sigterm(SIGNAL_ARGS)
+ {
+ int save_errno = errno;
+
+ got_sigterm = true;
+ if (MyProc)
+ SetLatch(&MyProc->procLatch);
+
+ errno = save_errno;
+ }
+
+ static void
+ worker_spi_sighup(SIGNAL_ARGS)
+ {
+ elog(LOG, "got sighup!");
+ if (MyProc)
+ SetLatch(&MyProc->procLatch);
+ }
+
+ static void
+ initialize_worker_spi(worktable *table)
+ {
+ int ret;
+ int ntup;
+ bool isnull;
+ StringInfoData buf;
+
+ StartTransactionCommand();
+ SPI_connect();
+ PushActiveSnapshot(GetTransactionSnapshot());
+
+ initStringInfo(&buf);
+ appendStringInfo(&buf, "select count(*) from pg_namespace where nspname = '%s'",
+ table->schema);
+
+ ret = SPI_execute(buf.data, true, 0);
+ if (ret != SPI_OK_SELECT)
+ elog(FATAL, "SPI_execute failed: error code %d", ret);
+
+ if (SPI_processed != 1)
+ elog(FATAL, "not a singleton result");
+
+ ntup = DatumGetInt32(SPI_getbinval(SPI_tuptable->vals[0],
+ SPI_tuptable->tupdesc,
+ 1, &isnull));
+ if (isnull)
+ elog(FATAL, "null result");
+
+ if (ntup == 0)
+ {
+ resetStringInfo(&buf);
+ appendStringInfo(&buf,
+ "CREATE SCHEMA \"%s\" "
+ "CREATE TABLE \"%s\" ("
+ " type text CHECK (type IN ('total', 'delta')), "
+ " value integer)"
+ "CREATE UNIQUE INDEX \"%s_unique_total\" ON \"%s\" (type) "
+ "WHERE type = 'total'",
+ table->schema, table->name, table->name, table->name);
+
+ ret = SPI_execute(buf.data, false, 0);
+
+ if (ret != SPI_OK_UTILITY)
+ elog(FATAL, "failed to create my schema");
+ }
+
+ SPI_finish();
+ PopActiveSnapshot();
+ CommitTransactionCommand();
+ }
+
+ static void
+ worker_spi_main(void *main_arg)
+ {
+ worktable *table = (worktable *) main_arg;
+ StringInfoData buf;
+
+ /* We're now ready to receive signals */
+ BackgroundWorkerUnblockSignals();
+
+ /* Connect to our database */
+ BackgroundWorkerInitializeConnection("postgres", NULL);
+
+ elog(LOG, "%s initialized with %s.%s",
+ MyBgworkerEntry->bgw_name, table->schema, table->name);
+ initialize_worker_spi(table);
+
+ /*
+ * Quote identifiers passed to us. Note that this must be done after
+ * initialize_worker_spi, because that routine assumes the names are not
+ * quoted.
+ *
+ * Note some memory might be leaked here.
+ */
+ table->schema = quote_identifier(table->schema);
+ table->name = quote_identifier(table->name);
+
+ initStringInfo(&buf);
+ appendStringInfo(&buf,
+ "WITH deleted AS (DELETE "
+ "FROM %s.%s "
+ "WHERE type = 'delta' RETURNING value), "
+ "total AS (SELECT coalesce(sum(value), 0) as sum "
+ "FROM deleted) "
+ "UPDATE %s.%s "
+ "SET value = %s.value + total.sum "
+ "FROM total WHERE type = 'total' "
+ "RETURNING %s.value",
+ table->schema, table->name,
+ table->schema, table->name,
+ table->name,
+ table->name);
+
+ while (!got_sigterm)
+ {
+ int ret;
+ int rc;
+
+ /*
+ * Background workers mustn't call usleep() or any direct equivalent:
+ * instead, they may wait on their process latch, which sleeps as
+ * necessary, but is awakened if postmaster dies. That way the
+ * background process goes away immediately in an emergency.
+ */
+ rc = WaitLatch(&MyProc->procLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ 1000L);
+ ResetLatch(&MyProc->procLatch);
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+
+ StartTransactionCommand();
+ SPI_connect();
+ PushActiveSnapshot(GetTransactionSnapshot());
+
+ ret = SPI_execute(buf.data, false, 0);
+
+ if (ret != SPI_OK_UPDATE_RETURNING)
+ elog(FATAL, "cannot select from table %s.%s: error code %d",
+ table->schema, table->name, ret);
+
+ if (SPI_processed > 0)
+ {
+ bool isnull;
+ int32 val;
+
+ val = DatumGetInt32(SPI_getbinval(SPI_tuptable->vals[0],
+ SPI_tuptable->tupdesc,
+ 1, &isnull));
+ if (!isnull)
+ elog(LOG, "%s: count in %s.%s is now %d",
+ MyBgworkerEntry->bgw_name,
+ table->schema, table->name, val);
+ }
+
+ SPI_finish();
+ PopActiveSnapshot();
+ CommitTransactionCommand();
+ }
+
+ proc_exit(0);
+ }
+
+ /*
+ * Entrypoint of this module.
+ *
+ * We register two worker processes here, to demonstrate how that can be done.
+ */
+ void
+ _PG_init(void)
+ {
+ BackgroundWorker worker;
+ worktable *table;
+
+ /* register the worker processes. These values are common for both */
+ worker.bgw_flags = BGWORKER_SHMEM_ACCESS |
+ BGWORKER_BACKEND_DATABASE_CONNECTION;
+ worker.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ worker.bgw_main = worker_spi_main;
+ worker.bgw_sighup = worker_spi_sighup;
+ worker.bgw_sigterm = worker_spi_sigterm;
+
+ /*
+ * These values are used for the first worker.
+ *
+ * Note these are palloc'd. The reason this works after starting a new
+ * worker process is that if we only fork, they point to valid allocated
+ * memory in the child process; and if we fork and then exec, the exec'd
+ * process will run this code again, and so the memory is also valid there.
+ */
+ table = palloc(sizeof(worktable));
+ table->schema = pstrdup("schema1");
+ table->name = pstrdup("counted");
+
+ worker.bgw_name = "SPI worker 1";
+ worker.bgw_restart_time = BGW_NEVER_RESTART;
+ worker.bgw_main_arg = (void *) table;
+ RegisterBackgroundWorker(&worker);
+
+ /* Values for the second worker */
+ table = palloc(sizeof(worktable));
+ table->schema = pstrdup("our schema2");
+ table->name = pstrdup("counted rows");
+
+ worker.bgw_name = "SPI worker 2";
+ worker.bgw_restart_time = 2;
+ worker.bgw_main_arg = (void *) table;
+ RegisterBackgroundWorker(&worker);
+ }
*** a/src/backend/postmaster/postmaster.c
--- b/src/backend/postmaster/postmaster.c
***************
*** 103,108 ****
--- 103,109 ----
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
+ #include "postmaster/bgworker.h"
#include "postmaster/fork_process.h"
#include "postmaster/pgarch.h"
#include "postmaster/postmaster.h"
***************
*** 132,138 ****
* children we have and send them appropriate signals when necessary.
*
* "Special" children such as the startup, bgwriter and autovacuum launcher
! * tasks are not in this list. Autovacuum worker and walsender processes are
* in it. Also, "dead_end" children are in it: these are children launched just
* for the purpose of sending a friendly rejection message to a would-be
* client. We must track them because they are attached to shared memory,
--- 133,140 ----
* children we have and send them appropriate signals when necessary.
*
* "Special" children such as the startup, bgwriter and autovacuum launcher
! * tasks are not in this list. Autovacuum worker, walsender and general
! * background worker processes are
* in it. Also, "dead_end" children are in it: these are children launched just
* for the purpose of sending a friendly rejection message to a would-be
* client. We must track them because they are attached to shared memory,
***************
*** 144,150 **** typedef struct bkend
pid_t pid; /* process id of backend */
long cancel_key; /* cancel key for cancels for this backend */
int child_slot; /* PMChildSlot for this backend, if any */
! bool is_autovacuum; /* is it an autovacuum process? */
bool dead_end; /* is it going to send an error and quit? */
dlist_node elem; /* list link in BackendList */
} Backend;
--- 146,157 ----
pid_t pid; /* process id of backend */
long cancel_key; /* cancel key for cancels for this backend */
int child_slot; /* PMChildSlot for this backend, if any */
! int bkend_type; /* flavor of backend or auxiliary process
! Note that BACKEND_TYPE_WALSND backends
! initially announce themselves as
! BACKEND_TYPE_NORMAL, so if bkend_type is
! normal then you should check for a recent
! transition. */
bool dead_end; /* is it going to send an error and quit? */
dlist_node elem; /* list link in BackendList */
} Backend;
***************
*** 155,160 **** static dlist_head BackendList = DLIST_STATIC_INIT(BackendList);
--- 162,190 ----
static Backend *ShmemBackendArray;
#endif
+
+ /*
+ * List of background workers.
+ */
+ typedef struct RegisteredBgWorker
+ {
+ BackgroundWorker *worker; /* its registry entry */
+ Backend *backend; /* its BackendList entry, or NULL */
+ pid_t pid; /* 0 if not running */
+ int child_slot;
+ TimestampTz crashed_at; /* if not 0, time it last crashed */
+ #ifdef EXEC_BACKEND
+ int cookie;
+ #endif
+ slist_node lnode; /* list link */
+ } RegisteredBgWorker;
+
+ static slist_head BackgroundWorkerList = SLIST_STATIC_INIT(BackgroundWorkerList);
+
+ BackgroundWorker *MyBgworkerEntry = NULL;
+
+
+
/* The socket number we are listening for connections on */
int PostPortNumber;
/* The directory names for Unix socket(s) */
***************
*** 306,311 **** static volatile sig_atomic_t start_autovac_launcher = false;
--- 336,345 ----
/* the launcher needs to be signalled to communicate some condition */
static volatile bool avlauncher_needs_signal = false;
+ /* set when there's a worker that needs to be started up */
+ static volatile bool StartWorkerNeeded = true;
+ static volatile bool HaveCrashedWorker = false;
+
/*
* State for assigning random salts and cancel keys.
* Also, the global MyCancelKey passes the cancel key assigned to a given
***************
*** 341,348 **** static void reaper(SIGNAL_ARGS);
--- 375,385 ----
static void sigusr1_handler(SIGNAL_ARGS);
static void startup_die(SIGNAL_ARGS);
static void dummy_handler(SIGNAL_ARGS);
+ static int GetNumRegisteredBackgroundWorkers(int flags);
static void StartupPacketTimeoutHandler(void);
static void CleanupBackend(int pid, int exitstatus);
+ static bool CleanupBackgroundWorker(int pid, int exitstatus);
+ static void do_start_bgworker(void);
static void HandleChildCrash(int pid, int exitstatus, const char *procname);
static void LogChildExit(int lev, const char *procname,
int pid, int exitstatus);
***************
*** 361,366 **** static long PostmasterRandom(void);
--- 398,404 ----
static void RandomSalt(char *md5Salt);
static void signal_child(pid_t pid, int signal);
static bool SignalSomeChildren(int signal, int targets);
+ static bool SignalUnconnectedWorkers(int signal);
#define SignalChildren(sig) SignalSomeChildren(sig, BACKEND_TYPE_ALL)
***************
*** 371,379 **** static bool SignalSomeChildren(int signal, int targets);
#define BACKEND_TYPE_NORMAL 0x0001 /* normal backend */
#define BACKEND_TYPE_AUTOVAC 0x0002 /* autovacuum worker process */
#define BACKEND_TYPE_WALSND 0x0004 /* walsender process */
! #define BACKEND_TYPE_ALL 0x0007 /* OR of all the above */
static int CountChildren(int target);
static bool CreateOptsFile(int argc, char *argv[], char *fullprogname);
static pid_t StartChildProcess(AuxProcType type);
static void StartAutovacuumWorker(void);
--- 409,422 ----
#define BACKEND_TYPE_NORMAL 0x0001 /* normal backend */
#define BACKEND_TYPE_AUTOVAC 0x0002 /* autovacuum worker process */
#define BACKEND_TYPE_WALSND 0x0004 /* walsender process */
! #define BACKEND_TYPE_BGWORKER 0x0008 /* bgworker process */
! #define BACKEND_TYPE_ALL 0x000F /* OR of all the above */
!
! #define BACKEND_TYPE_WORKER (BACKEND_TYPE_AUTOVAC | BACKEND_TYPE_BGWORKER)
static int CountChildren(int target);
+ static int CountUnconnectedWorkers(void);
+ static void StartOneBackgroundWorker(void);
static bool CreateOptsFile(int argc, char *argv[], char *fullprogname);
static pid_t StartChildProcess(AuxProcType type);
static void StartAutovacuumWorker(void);
***************
*** 473,478 **** static bool save_backend_variables(BackendParameters *param, Port *port,
--- 516,523 ----
static void ShmemBackendArrayAdd(Backend *bn);
static void ShmemBackendArrayRemove(Backend *bn);
+
+ static BackgroundWorker *find_bgworker_entry(int cookie);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
***************
*** 844,849 **** PostmasterMain(int argc, char *argv[])
--- 889,904 ----
process_shared_preload_libraries();
/*
+ * If loadable modules have added background workers, MaxBackends needs to
+ * be updated. Do so now.
+ */
+ // RerunAssignHook("max_connections");
+ if (GetNumShmemAttachedBgworkers() > 0)
+ SetConfigOption("max_connections",
+ GetConfigOption("max_connections", false, false),
+ PGC_POSTMASTER, PGC_S_OVERRIDE);
+
+ /*
* Establish input sockets.
*/
for (i = 0; i < MAXLISTEN; i++)
***************
*** 1087,1093 **** PostmasterMain(int argc, char *argv[])
* handling setup of child processes. See tcop/postgres.c,
* bootstrap/bootstrap.c, postmaster/bgwriter.c, postmaster/walwriter.c,
* postmaster/autovacuum.c, postmaster/pgarch.c, postmaster/pgstat.c,
! * postmaster/syslogger.c and postmaster/checkpointer.c.
*/
pqinitmask();
PG_SETMASK(&BlockSig);
--- 1142,1149 ----
* handling setup of child processes. See tcop/postgres.c,
* bootstrap/bootstrap.c, postmaster/bgwriter.c, postmaster/walwriter.c,
* postmaster/autovacuum.c, postmaster/pgarch.c, postmaster/pgstat.c,
! * postmaster/syslogger.c, postmaster/bgworker.c and
! * postmaster/checkpointer.c.
*/
pqinitmask();
PG_SETMASK(&BlockSig);
***************
*** 1177,1182 **** PostmasterMain(int argc, char *argv[])
--- 1233,1241 ----
Assert(StartupPID != 0);
pmState = PM_STARTUP;
+ /* Some workers may be scheduled to start now */
+ StartOneBackgroundWorker();
+
status = ServerLoop();
/*
***************
*** 1342,1347 **** checkDataDir(void)
--- 1401,1490 ----
}
/*
+ * Determine how long should we let ServerLoop sleep.
+ *
+ * In normal conditions we wait at most one minute, to ensure that the other
+ * background tasks handled by ServerLoop get done even when no requests are
+ * arriving. However, if there are background workers waiting to be started,
+ * we don't actually sleep so that they are quickly serviced.
+ */
+ static void
+ DetermineSleepTime(struct timeval *timeout)
+ {
+ TimestampTz next_wakeup = 0;
+
+ /*
+ * Normal case: either there are no background workers at all, or we're in
+ * a shutdown sequence (during which we ignore bgworkers altogether).
+ */
+ if (Shutdown > NoShutdown ||
+ (!StartWorkerNeeded && !HaveCrashedWorker))
+ {
+ timeout->tv_sec = 60;
+ timeout->tv_usec = 0;
+ return;
+ }
+
+ if (StartWorkerNeeded)
+ {
+ timeout->tv_sec = 0;
+ timeout->tv_usec = 0;
+ return;
+ }
+
+ if (HaveCrashedWorker)
+ {
+ slist_iter siter;
+
+ /*
+ * When there are crashed bgworkers, we sleep just long enough that
+ * they are restarted when they request to be. Scan the list to
+ * determine the minimum of all wakeup times according to most
+ * recent crash time and requested restart interval.
+ */
+ slist_foreach(siter, &BackgroundWorkerList)
+ {
+ RegisteredBgWorker *rw;
+ TimestampTz this_wakeup;
+
+ rw = slist_container(RegisteredBgWorker, lnode, siter.cur);
+
+ if (rw->crashed_at == 0)
+ continue;
+
+ if (rw->worker->bgw_restart_time == BGW_NEVER_RESTART)
+ continue;
+
+ this_wakeup = TimestampTzPlusMilliseconds(rw->crashed_at,
+ 1000L * rw->worker->bgw_restart_time);
+ if (next_wakeup == 0 || this_wakeup < next_wakeup)
+ next_wakeup = this_wakeup;
+ }
+ }
+
+ if (next_wakeup != 0)
+ {
+ int microsecs;
+
+ TimestampDifference(GetCurrentTimestamp(), next_wakeup,
+ &timeout->tv_sec, µsecs);
+ timeout->tv_usec = microsecs;
+
+ /* Ensure we don't exceed one minute */
+ if (timeout->tv_sec > 60)
+ {
+ timeout->tv_sec = 60;
+ timeout->tv_usec = 0;
+ }
+ }
+ else
+ {
+ timeout->tv_sec = 60;
+ timeout->tv_usec = 0;
+ }
+ }
+
+ /*
* Main idle loop of postmaster
*/
static int
***************
*** 1364,1372 **** ServerLoop(void)
/*
* Wait for a connection request to arrive.
*
- * We wait at most one minute, to ensure that the other background
- * tasks handled below get done even when no requests are arriving.
- *
* If we are in PM_WAIT_DEAD_END state, then we don't want to accept
* any new connections, so we don't call select() at all; just sleep
* for a little bit with signals unblocked.
--- 1507,1512 ----
***************
*** 1385,1392 **** ServerLoop(void)
/* must set timeout each time; some OSes change it! */
struct timeval timeout;
! timeout.tv_sec = 60;
! timeout.tv_usec = 0;
selres = select(nSockets, &rmask, NULL, NULL, &timeout);
}
--- 1525,1531 ----
/* must set timeout each time; some OSes change it! */
struct timeval timeout;
! DetermineSleepTime(&timeout);
selres = select(nSockets, &rmask, NULL, NULL, &timeout);
}
***************
*** 1498,1503 **** ServerLoop(void)
--- 1637,1646 ----
kill(AutoVacPID, SIGUSR2);
}
+ /* Get other worker processes running, if needed */
+ if (StartWorkerNeeded || HaveCrashedWorker)
+ StartOneBackgroundWorker();
+
/*
* Touch Unix socket and lock files every 58 minutes, to ensure that
* they are not removed by overzealous /tmp-cleaning tasks. We assume
***************
*** 1513,1519 **** ServerLoop(void)
}
}
-
/*
* Initialise the masks for select() for the ports we are listening on.
* Return the number of sockets to listen on.
--- 1656,1661 ----
***************
*** 2205,2212 **** pmdie(SIGNAL_ARGS)
if (pmState == PM_RUN || pmState == PM_RECOVERY ||
pmState == PM_HOT_STANDBY || pmState == PM_STARTUP)
{
! /* autovacuum workers are told to shut down immediately */
! SignalSomeChildren(SIGTERM, BACKEND_TYPE_AUTOVAC);
/* and the autovac launcher too */
if (AutoVacPID != 0)
signal_child(AutoVacPID, SIGTERM);
--- 2347,2357 ----
if (pmState == PM_RUN || pmState == PM_RECOVERY ||
pmState == PM_HOT_STANDBY || pmState == PM_STARTUP)
{
! /* autovac workers are told to shut down immediately */
! /* and bgworkers too; does this need tweaking? */
! SignalSomeChildren(SIGTERM,
! BACKEND_TYPE_AUTOVAC | BACKEND_TYPE_BGWORKER);
! SignalUnconnectedWorkers(SIGTERM);
/* and the autovac launcher too */
if (AutoVacPID != 0)
signal_child(AutoVacPID, SIGTERM);
***************
*** 2258,2269 **** pmdie(SIGNAL_ARGS)
signal_child(BgWriterPID, SIGTERM);
if (WalReceiverPID != 0)
signal_child(WalReceiverPID, SIGTERM);
if (pmState == PM_RECOVERY)
{
/*
! * Only startup, bgwriter, walreceiver, and/or checkpointer
! * should be active in this state; we just signaled the first
! * three, and we don't want to kill checkpointer yet.
*/
pmState = PM_WAIT_BACKENDS;
}
--- 2403,2416 ----
signal_child(BgWriterPID, SIGTERM);
if (WalReceiverPID != 0)
signal_child(WalReceiverPID, SIGTERM);
+ SignalUnconnectedWorkers(SIGTERM);
if (pmState == PM_RECOVERY)
{
/*
! * Only startup, bgwriter, walreceiver, unconnected bgworkers,
! * and/or checkpointer should be active in this state; we just
! * signaled the first four, and we don't want to kill
! * checkpointer yet.
*/
pmState = PM_WAIT_BACKENDS;
}
***************
*** 2275,2283 **** pmdie(SIGNAL_ARGS)
{
ereport(LOG,
(errmsg("aborting any active transactions")));
! /* shut down all backends and autovac workers */
SignalSomeChildren(SIGTERM,
! BACKEND_TYPE_NORMAL | BACKEND_TYPE_AUTOVAC);
/* and the autovac launcher too */
if (AutoVacPID != 0)
signal_child(AutoVacPID, SIGTERM);
--- 2422,2431 ----
{
ereport(LOG,
(errmsg("aborting any active transactions")));
! /* shut down all backends and workers */
SignalSomeChildren(SIGTERM,
! BACKEND_TYPE_NORMAL | BACKEND_TYPE_AUTOVAC |
! BACKEND_TYPE_BGWORKER);
/* and the autovac launcher too */
if (AutoVacPID != 0)
signal_child(AutoVacPID, SIGTERM);
***************
*** 2321,2326 **** pmdie(SIGNAL_ARGS)
--- 2469,2475 ----
signal_child(PgArchPID, SIGQUIT);
if (PgStatPID != 0)
signal_child(PgStatPID, SIGQUIT);
+ SignalUnconnectedWorkers(SIGQUIT);
ExitPostmaster(0);
break;
}
***************
*** 2449,2454 **** reaper(SIGNAL_ARGS)
--- 2598,2606 ----
if (PgStatPID == 0)
PgStatPID = pgstat_start();
+ /* some workers may be scheduled to start now */
+ StartOneBackgroundWorker();
+
/* at this point we are really open for business */
ereport(LOG,
(errmsg("database system is ready to accept connections")));
***************
*** 2615,2620 **** reaper(SIGNAL_ARGS)
--- 2767,2780 ----
continue;
}
+ /* Was it one of our background workers? */
+ if (CleanupBackgroundWorker(pid, exitstatus))
+ {
+ /* have it be restarted */
+ HaveCrashedWorker = true;
+ continue;
+ }
+
/*
* Else do standard backend child cleanup.
*/
***************
*** 2633,2643 **** reaper(SIGNAL_ARGS)
--- 2793,2892 ----
errno = save_errno;
}
+ /*
+ * Scan the bgworkers list and see if the given PID (which has just stopped
+ * or crashed) is in it. Handle its shutdown if so, and return true. If not a
+ * bgworker, return false.
+ *
+ * This is heavily based on CleanupBackend. One important difference is that
+ * we don't know yet that the dying process is a bgworker, so we must be silent
+ * until we're sure it is.
+ */
+ static bool
+ CleanupBackgroundWorker(int pid,
+ int exitstatus) /* child's exit status */
+ {
+ char namebuf[MAXPGPATH];
+ slist_iter iter;
+
+ slist_foreach(iter, &BackgroundWorkerList)
+ {
+ RegisteredBgWorker *rw;
+
+ rw = slist_container(RegisteredBgWorker, lnode, iter.cur);
+
+ if (rw->pid != pid)
+ continue;
+
+ #ifdef WIN32
+ /* see CleanupBackend */
+ if (exitstatus == ERROR_WAIT_NO_CHILDREN)
+ exitstatus = 0;
+ #endif
+
+ snprintf(namebuf, MAXPGPATH, "%s: %s", _("worker process"),
+ rw->worker->bgw_name);
+
+ /* Delay restarting any bgworker that exits with a nonzero status. */
+ if (!EXIT_STATUS_0(exitstatus))
+ rw->crashed_at = GetCurrentTimestamp();
+ else
+ rw->crashed_at = 0;
+
+ /*
+ * Additionally, for shared-memory-connected workers, just like a
+ * backend, any exit status other than 0 or 1 is considered a crash
+ * and causes a system-wide restart.
+ */
+ if (rw->worker->bgw_flags & BGWORKER_SHMEM_ACCESS)
+ {
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ {
+ rw->crashed_at = GetCurrentTimestamp();
+ HandleChildCrash(pid, exitstatus, namebuf);
+ return true;
+ }
+ }
+
+ if (!ReleasePostmasterChildSlot(rw->child_slot))
+ {
+ /*
+ * Uh-oh, the child failed to clean itself up. Treat as a
+ * crash after all.
+ */
+ rw->crashed_at = GetCurrentTimestamp();
+ HandleChildCrash(pid, exitstatus, namebuf);
+ return true;
+ }
+
+ /* Get it out of the BackendList and clear out remaining data */
+ if (rw->backend)
+ {
+ Assert(rw->worker->bgw_flags & BGWORKER_BACKEND_DATABASE_CONNECTION);
+ dlist_delete(&rw->backend->elem);
+ #ifdef EXEC_BACKEND
+ ShmemBackendArrayRemove(rw->backend);
+ #endif
+ free(rw->backend);
+ rw->backend = NULL;
+ }
+ rw->pid = 0;
+ rw->child_slot = 0;
+
+ LogChildExit(LOG, namebuf, pid, exitstatus);
+
+ return true;
+ }
+
+ return false;
+ }
/*
* CleanupBackend -- cleanup after terminated backend.
*
* Remove all local state associated with backend.
+ *
+ * If you change this, see also CleanupBackgroundWorker.
*/
static void
CleanupBackend(int pid,
***************
*** 2705,2711 **** CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
! * walwriter or autovacuum.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
--- 2954,2960 ----
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
! * walwriter, autovacuum, or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
***************
*** 2714,2719 **** static void
--- 2963,2969 ----
HandleChildCrash(int pid, int exitstatus, const char *procname)
{
dlist_mutable_iter iter;
+ slist_iter siter;
Backend *bp;
/*
***************
*** 2727,2732 **** HandleChildCrash(int pid, int exitstatus, const char *procname)
--- 2977,3030 ----
(errmsg("terminating any other active server processes")));
}
+ /* Process unconnected background workers */
+ slist_foreach(siter, &BackgroundWorkerList)
+ {
+ RegisteredBgWorker *rw;
+
+ rw = slist_container(RegisteredBgWorker, lnode, siter.cur);
+ if (rw->pid == pid)
+ {
+ /*
+ * Found entry for freshly-dead worker, so remove it.
+ */
+ (void) ReleasePostmasterChildSlot(rw->child_slot);
+ if (rw->backend)
+ {
+ dlist_delete(&rw->backend->elem);
+ #ifdef EXEC_BACKEND
+ ShmemBackendArrayRemove(rw->backend);
+ #endif
+ free(rw->backend);
+ rw->backend = NULL;
+ }
+ rw->pid = 0;
+ rw->child_slot = 0;
+ /* don't reset crashed_at */
+ /* Keep looping so we can signal remaining workers */
+ }
+ else
+ {
+ /*
+ * This worker is still alive. Unless we did so already, tell it
+ * to commit hara-kiri.
+ *
+ * SIGQUIT is the special signal that says exit without proc_exit
+ * and let the user know what's going on. But if SendStop is set
+ * (-s on command line), then we send SIGSTOP instead, so that we
+ * can get core dumps from all backends by hand.
+ */
+ if (!FatalError)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) rw->pid)));
+ signal_child(rw->pid, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+ }
+ }
+
/* Process regular backends */
dlist_foreach_modify(iter, &BackendList)
{
***************
*** 2761,2767 **** HandleChildCrash(int pid, int exitstatus, const char *procname)
--- 3059,3071 ----
*
* We could exclude dead_end children here, but at least in the
* SIGSTOP case it seems better to include them.
+ *
+ * Background workers were already processed above; ignore them
+ * here.
*/
+ if (bp->bkend_type == BACKEND_TYPE_BGWORKER)
+ continue;
+
if (!FatalError)
{
ereport(DEBUG2,
***************
*** 3005,3011 **** PostmasterStateMachine(void)
{
/*
* PM_WAIT_BACKENDS state ends when we have no regular backends
! * (including autovac workers) and no walwriter, autovac launcher or
* bgwriter. If we are doing crash recovery then we expect the
* checkpointer to exit as well, otherwise not. The archiver, stats,
* and syslogger processes are disregarded since they are not
--- 3309,3316 ----
{
/*
* PM_WAIT_BACKENDS state ends when we have no regular backends
! * (including autovac workers), no bgworkers (including unconnected
! * ones), and no walwriter, autovac launcher or
* bgwriter. If we are doing crash recovery then we expect the
* checkpointer to exit as well, otherwise not. The archiver, stats,
* and syslogger processes are disregarded since they are not
***************
*** 3014,3020 **** PostmasterStateMachine(void)
* later after writing the checkpoint record, like the archiver
* process.
*/
! if (CountChildren(BACKEND_TYPE_NORMAL | BACKEND_TYPE_AUTOVAC) == 0 &&
StartupPID == 0 &&
WalReceiverPID == 0 &&
BgWriterPID == 0 &&
--- 3319,3326 ----
* later after writing the checkpoint record, like the archiver
* process.
*/
! if (CountChildren(BACKEND_TYPE_NORMAL | BACKEND_TYPE_WORKER) == 0 &&
! CountUnconnectedWorkers() == 0 &&
StartupPID == 0 &&
WalReceiverPID == 0 &&
BgWriterPID == 0 &&
***************
*** 3227,3232 **** signal_child(pid_t pid, int signal)
--- 3533,3569 ----
}
/*
+ * Send a signal to bgworkers that did not request backend connections
+ */
+ static bool
+ SignalUnconnectedWorkers(int signal)
+ {
+ bool signaled = false;
+ slist_iter iter;
+
+ slist_foreach(iter, &BackgroundWorkerList)
+ {
+ RegisteredBgWorker *rw;
+
+ rw = slist_container(RegisteredBgWorker, lnode, iter.cur);
+
+ if (rw->pid == 0)
+ continue;
+ /* ignore connected workers */
+ if (rw->backend != NULL)
+ continue;
+
+ ereport(DEBUG4,
+ (errmsg_internal("sending signal %d to process %d",
+ signal, (int) rw->pid)));
+ signal_child(rw->pid, signal);
+ signaled = true;
+ }
+
+ return signaled;
+ }
+
+ /*
* Send a signal to the targeted children (but NOT special children;
* dead_end children are never signaled, either).
*/
***************
*** 3249,3263 **** SignalSomeChildren(int signal, int target)
*/
if (target != BACKEND_TYPE_ALL)
{
! int child;
! if (bp->is_autovacuum)
! child = BACKEND_TYPE_AUTOVAC;
! else if (IsPostmasterChildWalSender(bp->child_slot))
! child = BACKEND_TYPE_WALSND;
! else
! child = BACKEND_TYPE_NORMAL;
! if (!(target & child))
continue;
}
--- 3586,3600 ----
*/
if (target != BACKEND_TYPE_ALL)
{
! /*
! * Assign bkend_type for any recently announced
! * WAL Sender processes.
! */
! if (bp->bkend_type == BACKEND_TYPE_NORMAL &&
! IsPostmasterChildWalSender(bp->child_slot))
! bp->bkend_type = BACKEND_TYPE_WALSND;
! if (!(target & bp->bkend_type))
continue;
}
***************
*** 3375,3381 **** BackendStartup(Port *port)
* of backends.
*/
bn->pid = pid;
! bn->is_autovacuum = false;
dlist_push_head(&BackendList, &bn->elem);
#ifdef EXEC_BACKEND
--- 3712,3718 ----
* of backends.
*/
bn->pid = pid;
! bn->bkend_type = BACKEND_TYPE_NORMAL; /* Can change later to WALSND */
dlist_push_head(&BackendList, &bn->elem);
#ifdef EXEC_BACKEND
***************
*** 3744,3750 **** internal_forkexec(int argc, char *argv[], Port *port)
fp = AllocateFile(tmpfilename, PG_BINARY_W);
if (!fp)
{
! /* As in OpenTemporaryFile, try to make the temp-file directory */
mkdir(PG_TEMP_FILES_DIR, S_IRWXU);
fp = AllocateFile(tmpfilename, PG_BINARY_W);
--- 4081,4090 ----
fp = AllocateFile(tmpfilename, PG_BINARY_W);
if (!fp)
{
! /*
! * As in OpenTemporaryFileInTablespace, try to make the temp-file
! * directory
! */
mkdir(PG_TEMP_FILES_DIR, S_IRWXU);
fp = AllocateFile(tmpfilename, PG_BINARY_W);
***************
*** 4078,4084 **** SubPostmasterMain(int argc, char *argv[])
if (strcmp(argv[1], "--forkbackend") == 0 ||
strcmp(argv[1], "--forkavlauncher") == 0 ||
strcmp(argv[1], "--forkavworker") == 0 ||
! strcmp(argv[1], "--forkboot") == 0)
PGSharedMemoryReAttach();
/* autovacuum needs this set before calling InitProcess */
--- 4418,4425 ----
if (strcmp(argv[1], "--forkbackend") == 0 ||
strcmp(argv[1], "--forkavlauncher") == 0 ||
strcmp(argv[1], "--forkavworker") == 0 ||
! strcmp(argv[1], "--forkboot") == 0 ||
! strncmp(argv[1], "--forkbgworker=", 15) == 0)
PGSharedMemoryReAttach();
/* autovacuum needs this set before calling InitProcess */
***************
*** 4213,4218 **** SubPostmasterMain(int argc, char *argv[])
--- 4554,4579 ----
AutoVacWorkerMain(argc - 2, argv + 2); /* does not return */
}
+ if (strncmp(argv[1], "--forkbgworker=", 15) == 0)
+ {
+ int cookie;
+
+ /* Close the postmaster's sockets */
+ ClosePostmasterPorts(false);
+
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores(false, 0);
+
+ cookie = atoi(argv[1] + 15);
+ MyBgworkerEntry = find_bgworker_entry(cookie);
+ do_start_bgworker();
+ }
if (strcmp(argv[1], "--forkarch") == 0)
{
/* Close the postmaster's sockets */
***************
*** 4312,4317 **** sigusr1_handler(SIGNAL_ARGS)
--- 4673,4681 ----
(errmsg("database system is ready to accept read only connections")));
pmState = PM_HOT_STANDBY;
+
+ /* Some workers may be scheduled to start now */
+ StartOneBackgroundWorker();
}
if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
***************
*** 4482,4487 **** PostmasterRandom(void)
--- 4846,4877 ----
}
/*
+ * Count up number of worker processes that did not request backend connections
+ */
+ static int
+ CountUnconnectedWorkers(void)
+ {
+ slist_iter iter;
+ int cnt = 0;
+
+ slist_foreach(iter, &BackgroundWorkerList)
+ {
+ RegisteredBgWorker *rw;
+
+ rw = slist_container(RegisteredBgWorker, lnode, iter.cur);
+
+ if (rw->pid == 0)
+ continue;
+ /* ignore connected workers */
+ if (rw->backend != NULL)
+ continue;
+
+ cnt++;
+ }
+ return cnt;
+ }
+
+ /*
* Count up number of child processes of specified types (dead_end chidren
* are always excluded).
*/
***************
*** 4504,4518 **** CountChildren(int target)
*/
if (target != BACKEND_TYPE_ALL)
{
! int child;
! if (bp->is_autovacuum)
! child = BACKEND_TYPE_AUTOVAC;
! else if (IsPostmasterChildWalSender(bp->child_slot))
! child = BACKEND_TYPE_WALSND;
! else
! child = BACKEND_TYPE_NORMAL;
! if (!(target & child))
continue;
}
--- 4894,4908 ----
*/
if (target != BACKEND_TYPE_ALL)
{
! /*
! * Assign bkend_type for any recently announced
! * WAL Sender processes.
! */
! if (bp->bkend_type == BACKEND_TYPE_NORMAL &&
! IsPostmasterChildWalSender(bp->child_slot))
! bp->bkend_type = BACKEND_TYPE_WALSND;
! if (!(target & bp->bkend_type))
continue;
}
***************
*** 4671,4677 **** StartAutovacuumWorker(void)
bn->pid = StartAutoVacWorker();
if (bn->pid > 0)
{
! bn->is_autovacuum = true;
dlist_push_head(&BackendList, &bn->elem);
#ifdef EXEC_BACKEND
ShmemBackendArrayAdd(bn);
--- 5061,5067 ----
bn->pid = StartAutoVacWorker();
if (bn->pid > 0)
{
! bn->bkend_type = BACKEND_TYPE_AUTOVAC;
dlist_push_head(&BackendList, &bn->elem);
#ifdef EXEC_BACKEND
ShmemBackendArrayAdd(bn);
***************
*** 4746,4763 **** CreateOptsFile(int argc, char *argv[], char *fullprogname)
*
* This reports the number of entries needed in per-child-process arrays
* (the PMChildFlags array, and if EXEC_BACKEND the ShmemBackendArray).
! * These arrays include regular backends, autovac workers and walsenders,
! * but not special children nor dead_end children. This allows the arrays
! * to have a fixed maximum size, to wit the same too-many-children limit
! * enforced by canAcceptConnections(). The exact value isn't too critical
! * as long as it's more than MaxBackends.
*/
int
MaxLivePostmasterChildren(void)
{
! return 2 * MaxBackends;
}
#ifdef EXEC_BACKEND
--- 5136,5758 ----
*
* This reports the number of entries needed in per-child-process arrays
* (the PMChildFlags array, and if EXEC_BACKEND the ShmemBackendArray).
! * These arrays include regular backends, autovac workers, walsenders
! * and background workers, but not special children nor dead_end children.
! * This allows the arrays to have a fixed maximum size, to wit the same
! * too-many-children limit enforced by canAcceptConnections(). The exact value
! * isn't too critical as long as it's more than MaxBackends.
*/
int
MaxLivePostmasterChildren(void)
{
! return 2 * (MaxConnections + autovacuum_max_workers + 1 +
! GetNumRegisteredBackgroundWorkers(0));
! }
!
! /*
! * Register a new background worker.
! *
! * This can only be called in the _PG_init function of a module library
! * that's loaded by shared_preload_libraries; otherwise it has no effect.
! */
! void
! RegisterBackgroundWorker(BackgroundWorker *worker)
! {
! RegisteredBgWorker *rw;
! int namelen = strlen(worker->bgw_name);
! #ifdef EXEC_BACKEND
! /*
! * Use 1 here, not 0, to avoid confusing a possible bogus cookie read
! * by atoi() in SubPostmasterMain.
! */
! static int BackgroundWorkerCookie = 1;
! #endif
!
! if (!IsUnderPostmaster)
! ereport(LOG,
! (errmsg("registering background worker: %s", worker->bgw_name)));
!
! if (!process_shared_preload_libraries_in_progress)
! {
! if (!IsUnderPostmaster)
! ereport(LOG,
! (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("background worker \"%s\": must be registered in shared_preload_libraries",
! worker->bgw_name)));
! return;
! }
!
! /* sanity check for flags */
! if (worker->bgw_flags & BGWORKER_BACKEND_DATABASE_CONNECTION)
! {
! if (!(worker->bgw_flags & BGWORKER_SHMEM_ACCESS))
! {
! if (!IsUnderPostmaster)
! ereport(LOG,
! (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
! errmsg("background worker \"%s\": must attach to shared memory in order to request a database connection",
! worker->bgw_name)));
! return;
! }
!
! if (worker->bgw_start_time == BgWorkerStart_PostmasterStart)
! {
! if (!IsUnderPostmaster)
! ereport(LOG,
! (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
! errmsg("background worker \"%s\": cannot request database access if starting at postmaster start",
! worker->bgw_name)));
! return;
! }
!
! /* XXX other checks? */
! }
!
! if ((worker->bgw_restart_time < 0 &&
! worker->bgw_restart_time != BGW_NEVER_RESTART) ||
! (worker->bgw_restart_time > USECS_PER_DAY / 1000))
! {
! if (!IsUnderPostmaster)
! ereport(LOG,
! (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
! errmsg("background worker \"%s\": invalid restart interval",
! worker->bgw_name)));
! return;
! }
!
! /*
! * Copy the registration data into the registered workers list.
! */
! rw = malloc(MAXALIGN(sizeof(RegisteredBgWorker)) +
! MAXALIGN(sizeof(BackgroundWorker)) + namelen + 1);
! if (rw == NULL)
! {
! ereport(LOG,
! (errcode(ERRCODE_OUT_OF_MEMORY),
! errmsg("out of memory")));
! return;
! }
!
! rw->worker = (BackgroundWorker *)
! MAXALIGN((char *) rw + sizeof(RegisteredBgWorker));
! memcpy(rw->worker, worker, sizeof(BackgroundWorker));
! rw->worker->bgw_name = (char *)
! MAXALIGN((char *) rw->worker + sizeof(BackgroundWorker));
! strlcpy(rw->worker->bgw_name, worker->bgw_name, namelen + 1);
!
! rw->backend = NULL;
! rw->pid = 0;
! rw->child_slot = 0;
! rw->crashed_at = 0;
! #ifdef EXEC_BACKEND
! rw->cookie = BackgroundWorkerCookie++;
! #endif
!
! slist_push_head(&BackgroundWorkerList, &rw->lnode);
! }
!
! /*
! * Connect background worker to a database.
! */
! void
! BackgroundWorkerInitializeConnection(char *dbname, char *username)
! {
! BackgroundWorker *worker = MyBgworkerEntry;
!
! /* XXX is this the right errcode? */
! if (!(worker->bgw_flags & BGWORKER_BACKEND_DATABASE_CONNECTION))
! ereport(FATAL,
! (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
! errmsg("database connection requirement not indicated during registration")));
!
! InitPostgres(dbname, InvalidOid, username, NULL);
!
! /* it had better not gotten out of "init" mode yet */
! if (!IsInitProcessingMode())
! ereport(ERROR,
! (errmsg("invalid processing mode in bgworker")));
! SetProcessingMode(NormalProcessing);
! }
!
! /*
! * Block/unblock signals in a background worker
! */
! void
! BackgroundWorkerBlockSignals(void)
! {
! PG_SETMASK(&BlockSig);
! }
!
! void
! BackgroundWorkerUnblockSignals(void)
! {
! PG_SETMASK(&UnBlockSig);
! }
!
! #ifdef EXEC_BACKEND
! static BackgroundWorker *
! find_bgworker_entry(int cookie)
! {
! slist_iter iter;
!
! slist_foreach(iter, &BackgroundWorkerList)
! {
! RegisteredBgWorker *rw;
!
! rw = slist_container(RegisteredBgWorker, lnode, iter.cur);
! if (rw->cookie == cookie)
! return rw->worker;
! }
!
! return NULL;
! }
! #endif
!
! static void
! bgworker_quickdie(SIGNAL_ARGS)
! {
! sigaddset(&BlockSig, SIGQUIT); /* prevent nested calls */
! PG_SETMASK(&BlockSig);
!
! /*
! * We DO NOT want to run proc_exit() callbacks -- we're here because
! * shared memory may be corrupted, so we don't want to try to clean up our
! * transaction. Just nail the windows shut and get out of town. Now that
! * there's an atexit callback to prevent third-party code from breaking
! * things by calling exit() directly, we have to reset the callbacks
! * explicitly to make this work as intended.
! */
! on_exit_reset();
!
! /*
! * Note we do exit(0) here, not exit(2) like quickdie. The reason is that
! * we don't want to be seen this worker as independently crashed, because
! * then postmaster would delay restarting it again afterwards. If some
! * idiot DBA manually sends SIGQUIT to a random bgworker, the "dead man
! * switch" will ensure that postmaster sees this as a crash.
! */
! exit(0);
! }
!
! static void
! do_start_bgworker(void)
! {
! sigjmp_buf local_sigjmp_buf;
! char buf[MAXPGPATH];
! BackgroundWorker *worker = MyBgworkerEntry;
!
! if (worker == NULL)
! elog(FATAL, "unable to find bgworker entry");
!
! /* we are a postmaster subprocess now */
! IsUnderPostmaster = true;
! IsBackgroundWorker = true;
!
! /* reset MyProcPid */
! MyProcPid = getpid();
!
! /* record Start Time for logging */
! MyStartTime = time(NULL);
!
! /* Identify myself via ps */
! snprintf(buf, MAXPGPATH, "bgworker: %s", worker->bgw_name);
! init_ps_display(buf, "", "", "");
!
! SetProcessingMode(InitProcessing);
!
! /* Apply PostAuthDelay */
! if (PostAuthDelay > 0)
! pg_usleep(PostAuthDelay * 1000000L);
!
! /*
! * If possible, make this process a group leader, so that the postmaster
! * can signal any child processes too.
! */
! #ifdef HAVE_SETSID
! if (setsid() < 0)
! elog(FATAL, "setsid() failed: %m");
! #endif
!
! /*
! * Set up signal handlers.
! */
! if (worker->bgw_flags & BGWORKER_BACKEND_DATABASE_CONNECTION)
! {
! /*
! * SIGINT is used to signal canceling the current action
! */
! pqsignal(SIGINT, StatementCancelHandler);
! pqsignal(SIGUSR1, procsignal_sigusr1_handler);
! pqsignal(SIGFPE, FloatExceptionHandler);
!
! /* XXX Any other handlers needed here? */
! }
! else
! {
! pqsignal(SIGINT, SIG_IGN);
! pqsignal(SIGUSR1, SIG_IGN);
! pqsignal(SIGFPE, SIG_IGN);
! }
!
! /* These are configurable */
! pqsignal(SIGTERM, worker->bgw_sigterm);
! pqsignal(SIGHUP, worker->bgw_sighup);
!
! pqsignal(SIGQUIT, bgworker_quickdie);
! InitializeTimeouts(); /* establishes SIGALRM handler */
!
! pqsignal(SIGPIPE, SIG_IGN);
! pqsignal(SIGUSR2, SIG_IGN);
! pqsignal(SIGCHLD, SIG_DFL);
!
! /*
! * If an exception is encountered, processing resumes here.
! *
! * See notes in postgres.c about the design of this coding.
! */
! if (sigsetjmp(local_sigjmp_buf, 1) != 0)
! {
! /* Since not using PG_TRY, must reset error stack by hand */
! error_context_stack = NULL;
!
! /* Prevent interrupts while cleaning up */
! HOLD_INTERRUPTS();
!
! /* Report the error to the server log */
! EmitErrorReport();
!
! /*
! * we might need more cleanup here ... LWLockReleaseAll, etc.
! * Note that in some cases we will call InitPocess which will register
! * ProcKill as exit callback.
! */
!
! /* and go away */
! proc_exit(1);
! }
!
! /* We can now handle ereport(ERROR) */
! PG_exception_stack = &local_sigjmp_buf;
!
! /* Early initialization */
! BaseInit();
!
! /*
! * If necessary, create a per-backend PGPROC struct in shared memory,
! * except in the EXEC_BACKEND case where this was done in
! * SubPostmasterMain. We must do this before we can use LWLocks (and in the
! * EXEC_BACKEND case we already had to do some stuff with LWLocks).
! */
! #ifndef EXEC_BACKEND
! if (worker->bgw_flags & BGWORKER_SHMEM_ACCESS)
! InitProcess();
! #endif
!
! /*
! * Note that in normal processes, we would call InitPostgres here. For
! * a worker, however, we don't know what database to connect to, yet; so we
! * need to wait until the user code does it via
! * BackgroundWorkerInitializeConnection().
! */
!
! /*
! * Now invoke the user-defined worker code
! */
! worker->bgw_main(worker->bgw_main_arg);
!
! /* ... and if it returns, we're done */
! proc_exit(0);
! }
!
! /*
! * Return the number of background workers registered that have at least
! * one of the passed flag bits set.
! */
! static int
! GetNumRegisteredBackgroundWorkers(int flags)
! {
! slist_iter iter;
! int count = 0;
!
! slist_foreach(iter, &BackgroundWorkerList)
! {
! RegisteredBgWorker *rw;
!
! rw = slist_container(RegisteredBgWorker, lnode, iter.cur);
!
! if (flags != 0 &&
! !(rw->worker->bgw_flags & flags))
! continue;
!
! count++;
! }
!
! return count;
! }
!
! /*
! * Return the number of bgworkers that need to have PGPROC entries.
! */
! int
! GetNumShmemAttachedBgworkers(void)
! {
! return GetNumRegisteredBackgroundWorkers(BGWORKER_SHMEM_ACCESS);
! }
!
! #ifdef EXEC_BACKEND
! static pid_t
! bgworker_forkexec(int cookie)
! {
! char *av[10];
! int ac = 0;
! char forkav[MAXPGPATH];
!
! snprintf(forkav, MAXPGPATH, "--forkbgworker=%d", cookie);
!
! av[ac++] = "postgres";
! av[ac++] = forkav;
! av[ac++] = NULL; /* filled in by postmaster_forkexec */
! av[ac] = NULL;
!
! Assert(ac < lengthof(av));
!
! return postmaster_forkexec(ac, av);
! }
! #endif
!
! /*
! * Start a new bgworker.
! * Starting time conditions must have been checked already.
! *
! * This code is heavily based on autovacuum.c, q.v.
! */
! static void
! start_bgworker(RegisteredBgWorker *rw)
! {
! pid_t worker_pid;
!
! ereport(LOG,
! (errmsg("starting background worker process \"%s\"",
! rw->worker->bgw_name)));
!
! #ifdef EXEC_BACKEND
! switch ((worker_pid = bgworker_forkexec(rw->cookie)))
! #else
! switch ((worker_pid = fork_process()))
! #endif
! {
! case -1:
! ereport(LOG,
! (errmsg("could not fork worker process: %m")));
! return;
!
! #ifndef EXEC_BACKEND
! case 0:
! /* in postmaster child ... */
! /* Close the postmaster's sockets */
! ClosePostmasterPorts(false);
!
! /* Lose the postmaster's on-exit routines */
! on_exit_reset();
!
! /* Do NOT release postmaster's working memory context */
!
! MyBgworkerEntry = rw->worker;
! do_start_bgworker();
! break;
! #endif
! default:
! rw->pid = worker_pid;
! if (rw->backend)
! rw->backend->pid = rw->pid;
! }
! }
!
! /*
! * Does the current postmaster state require starting a worker with the
! * specified start_time?
! */
! static bool
! bgworker_should_start_now(BgWorkerStartTime start_time)
! {
! /* XXX maybe this'd be better as a table */
! switch (pmState)
! {
! case PM_NO_CHILDREN:
! case PM_WAIT_DEAD_END:
! case PM_SHUTDOWN_2:
! case PM_SHUTDOWN:
! case PM_WAIT_BACKENDS:
! case PM_WAIT_READONLY:
! case PM_WAIT_BACKUP:
! break;
!
! case PM_RUN:
! if (start_time == BgWorkerStart_RecoveryFinished)
! return true;
! /* fall through */
!
! case PM_HOT_STANDBY:
! if (start_time == BgWorkerStart_ConsistentState)
! return true;
! /* fall through */
!
! case PM_RECOVERY:
! case PM_STARTUP:
! case PM_INIT:
! if (start_time == BgWorkerStart_PostmasterStart)
! return true;
! /* fall through */
! }
!
! return false;
! }
!
! /*
! * Allocate the Backend struct for a connected background worker, but don't
! * add it to the list of backends just yet.
! *
! * Some info from the Backend is copied into the passed rw.
! */
! static bool
! assign_backendlist_entry(RegisteredBgWorker *rw)
! {
! Backend *bn = malloc(sizeof(Backend));
!
! if (bn == NULL)
! {
! ereport(LOG,
! (errcode(ERRCODE_OUT_OF_MEMORY),
! errmsg("out of memory")));
!
! /*
! * The worker didn't really crash, but setting this nonzero makes
! * postmaster wait a bit before attempting to start it again; if it
! * tried again right away, most likely it'd find itself under the same
! * memory pressure.
! */
! rw->crashed_at = GetCurrentTimestamp();
! return false;
! }
!
! /*
! * Compute the cancel key that will be assigned to this session.
! * We probably don't need cancel keys for background workers, but
! * we'd better have something random in the field to prevent
! * unfriendly people from sending cancels to them.
! */
! MyCancelKey = PostmasterRandom();
! bn->cancel_key = MyCancelKey;
!
! bn->child_slot = MyPMChildSlot = AssignPostmasterChildSlot();
! bn->bkend_type = BACKEND_TYPE_BGWORKER;
! bn->dead_end = false;
!
! rw->backend = bn;
! rw->child_slot = bn->child_slot;
!
! return true;
}
+ /*
+ * If the time is right, start one background worker.
+ *
+ * As a side effect, the bgworker control variables are set or reset whenever
+ * there are more workers to start after this one, and whenever the overall
+ * system state requires it.
+ */
+ static void
+ StartOneBackgroundWorker(void)
+ {
+ slist_iter iter;
+ TimestampTz now = 0;
+
+ if (FatalError)
+ {
+ StartWorkerNeeded = false;
+ HaveCrashedWorker = false;
+ return; /* not yet */
+ }
+
+ HaveCrashedWorker = false;
+
+ slist_foreach(iter, &BackgroundWorkerList)
+ {
+ RegisteredBgWorker *rw;
+
+ rw = slist_container(RegisteredBgWorker, lnode, iter.cur);
+
+ /* already running? */
+ if (rw->pid != 0)
+ continue;
+
+ /*
+ * If this worker has crashed previously, maybe it needs to be
+ * restarted (unless on registration it specified it doesn't want to be
+ * restarted at all). Check how long ago did a crash last happen. If
+ * the last crash is too recent, don't start it right away; let it be
+ * restarted once enough time has passed.
+ */
+ if (rw->crashed_at != 0)
+ {
+ if (rw->worker->bgw_restart_time == BGW_NEVER_RESTART)
+ continue;
+
+ if (now == 0)
+ now = GetCurrentTimestamp();
+
+ if (!TimestampDifferenceExceeds(rw->crashed_at, now,
+ rw->worker->bgw_restart_time * 1000))
+ {
+ HaveCrashedWorker = true;
+ continue;
+ }
+ }
+
+ if (bgworker_should_start_now(rw->worker->bgw_start_time))
+ {
+ /* reset crash time before calling assign_backendlist_entry */
+ rw->crashed_at = 0;
+
+ /*
+ * If necessary, allocate and assign the Backend element. Note we
+ * must do this before forking, so that we can handle out of memory
+ * properly.
+ *
+ * If not connected, we don't need a Backend element, but we still
+ * need a PMChildSlot.
+ */
+ if (rw->worker->bgw_flags & BGWORKER_BACKEND_DATABASE_CONNECTION)
+ {
+ if (!assign_backendlist_entry(rw))
+ return;
+ }
+ else
+ rw->child_slot = MyPMChildSlot = AssignPostmasterChildSlot();
+
+ start_bgworker(rw); /* sets rw->pid */
+
+ if (rw->backend)
+ {
+ dlist_push_head(&BackendList, &rw->backend->elem);
+ #ifdef EXEC_BACKEND
+ ShmemBackendArrayAdd(rw->backend);
+ #endif
+ }
+
+ /*
+ * Have ServerLoop call us again. Note that there might not
+ * actually *be* another runnable worker, but we don't care all
+ * that much; we will find out the next time we run.
+ */
+ StartWorkerNeeded = true;
+ return;
+ }
+ }
+
+ /* no runnable worker found */
+ StartWorkerNeeded = false;
+ }
#ifdef EXEC_BACKEND
*** a/src/backend/storage/lmgr/proc.c
--- b/src/backend/storage/lmgr/proc.c
***************
*** 40,45 ****
--- 40,46 ----
#include "access/xact.h"
#include "miscadmin.h"
#include "postmaster/autovacuum.h"
+ #include "postmaster/bgworker.h"
#include "replication/syncrep.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
***************
*** 140,146 **** ProcGlobalSemas(void)
* So, now we grab enough semaphores to support the desired max number
* of backends immediately at initialization --- if the sysadmin has set
* MaxConnections or autovacuum_max_workers higher than his kernel will
! * support, he'll find out sooner rather than later.
*
* Another reason for creating semaphores here is that the semaphore
* implementation typically requires us to create semaphores in the
--- 141,149 ----
* So, now we grab enough semaphores to support the desired max number
* of backends immediately at initialization --- if the sysadmin has set
* MaxConnections or autovacuum_max_workers higher than his kernel will
! * support, he'll find out sooner rather than later. (The number of
! * background worker processes registered by loadable modules is also taken
! * into consideration.)
*
* Another reason for creating semaphores here is that the semaphore
* implementation typically requires us to create semaphores in the
***************
*** 171,176 **** InitProcGlobal(void)
--- 174,180 ----
ProcGlobal->spins_per_delay = DEFAULT_SPINS_PER_DELAY;
ProcGlobal->freeProcs = NULL;
ProcGlobal->autovacFreeProcs = NULL;
+ ProcGlobal->bgworkerFreeProcs = NULL;
ProcGlobal->startupProc = NULL;
ProcGlobal->startupProcPid = 0;
ProcGlobal->startupBufferPinWaitBufId = -1;
***************
*** 179,188 **** InitProcGlobal(void)
/*
* Create and initialize all the PGPROC structures we'll need. There are
! * four separate consumers: (1) normal backends, (2) autovacuum workers
! * and the autovacuum launcher, (3) auxiliary processes, and (4) prepared
! * transactions. Each PGPROC structure is dedicated to exactly one of
! * these purposes, and they do not move between groups.
*/
procs = (PGPROC *) ShmemAlloc(TotalProcs * sizeof(PGPROC));
ProcGlobal->allProcs = procs;
--- 183,193 ----
/*
* Create and initialize all the PGPROC structures we'll need. There are
! * five separate consumers: (1) normal backends, (2) autovacuum workers
! * and the autovacuum launcher, (3) background workers, (4) auxiliary
! * processes, and (5) prepared transactions. Each PGPROC structure is
! * dedicated to exactly one of these purposes, and they do not move between
! * groups.
*/
procs = (PGPROC *) ShmemAlloc(TotalProcs * sizeof(PGPROC));
ProcGlobal->allProcs = procs;
***************
*** 191,197 **** InitProcGlobal(void)
ereport(FATAL,
(errcode(ERRCODE_OUT_OF_MEMORY),
errmsg("out of shared memory")));
- MemSet(procs, 0, TotalProcs * sizeof(PGPROC));
/*
* Also allocate a separate array of PGXACT structures. This is separate
--- 196,201 ----
***************
*** 223,234 **** InitProcGlobal(void)
procs[i].pgprocno = i;
/*
! * Newly created PGPROCs for normal backends or for autovacuum must be
! * queued up on the appropriate free list. Because there can only
! * ever be a small, fixed number of auxiliary processes, no free list
! * is used in that case; InitAuxiliaryProcess() instead uses a linear
! * search. PGPROCs for prepared transactions are added to a free list
! * by TwoPhaseShmemInit().
*/
if (i < MaxConnections)
{
--- 227,238 ----
procs[i].pgprocno = i;
/*
! * Newly created PGPROCs for normal backends, autovacuum and bgworkers
! * must be queued up on the appropriate free list. Because there can
! * only ever be a small, fixed number of auxiliary processes, no free
! * list is used in that case; InitAuxiliaryProcess() instead uses a
! * linear search. PGPROCs for prepared transactions are added to a
! * free list by TwoPhaseShmemInit().
*/
if (i < MaxConnections)
{
***************
*** 236,247 **** InitProcGlobal(void)
procs[i].links.next = (SHM_QUEUE *) ProcGlobal->freeProcs;
ProcGlobal->freeProcs = &procs[i];
}
! else if (i < MaxBackends)
{
/* PGPROC for AV launcher/worker, add to autovacFreeProcs list */
procs[i].links.next = (SHM_QUEUE *) ProcGlobal->autovacFreeProcs;
ProcGlobal->autovacFreeProcs = &procs[i];
}
/* Initialize myProcLocks[] shared memory queues. */
for (j = 0; j < NUM_LOCK_PARTITIONS; j++)
--- 240,257 ----
procs[i].links.next = (SHM_QUEUE *) ProcGlobal->freeProcs;
ProcGlobal->freeProcs = &procs[i];
}
! else if (i < MaxConnections + autovacuum_max_workers + 1)
{
/* PGPROC for AV launcher/worker, add to autovacFreeProcs list */
procs[i].links.next = (SHM_QUEUE *) ProcGlobal->autovacFreeProcs;
ProcGlobal->autovacFreeProcs = &procs[i];
}
+ else if (i < MaxBackends)
+ {
+ /* PGPROC for bgworker, add to bgworkerFreeProcs list */
+ procs[i].links.next = (SHM_QUEUE *) ProcGlobal->bgworkerFreeProcs;
+ ProcGlobal->bgworkerFreeProcs = &procs[i];
+ }
/* Initialize myProcLocks[] shared memory queues. */
for (j = 0; j < NUM_LOCK_PARTITIONS; j++)
***************
*** 299,304 **** InitProcess(void)
--- 309,316 ----
if (IsAnyAutoVacuumProcess())
MyProc = procglobal->autovacFreeProcs;
+ else if (IsBackgroundWorker)
+ MyProc = procglobal->bgworkerFreeProcs;
else
MyProc = procglobal->freeProcs;
***************
*** 306,311 **** InitProcess(void)
--- 318,325 ----
{
if (IsAnyAutoVacuumProcess())
procglobal->autovacFreeProcs = (PGPROC *) MyProc->links.next;
+ else if (IsBackgroundWorker)
+ procglobal->bgworkerFreeProcs = (PGPROC *) MyProc->links.next;
else
procglobal->freeProcs = (PGPROC *) MyProc->links.next;
SpinLockRelease(ProcStructLock);
***************
*** 782,787 **** ProcKill(int code, Datum arg)
--- 796,806 ----
MyProc->links.next = (SHM_QUEUE *) procglobal->autovacFreeProcs;
procglobal->autovacFreeProcs = MyProc;
}
+ else if (IsBackgroundWorker)
+ {
+ MyProc->links.next = (SHM_QUEUE *) procglobal->bgworkerFreeProcs;
+ procglobal->bgworkerFreeProcs = MyProc;
+ }
else
{
MyProc->links.next = (SHM_QUEUE *) procglobal->freeProcs;
*** a/src/backend/utils/init/globals.c
--- b/src/backend/utils/init/globals.c
***************
*** 87,92 **** pid_t PostmasterPid = 0;
--- 87,93 ----
bool IsPostmasterEnvironment = false;
bool IsUnderPostmaster = false;
bool IsBinaryUpgrade = false;
+ bool IsBackgroundWorker = false;
bool ExitOnAnyError = false;
*** a/src/backend/utils/init/miscinit.c
--- b/src/backend/utils/init/miscinit.c
***************
*** 498,507 **** void
InitializeSessionUserIdStandalone(void)
{
/*
! * This function should only be called in single-user mode and in
! * autovacuum workers.
*/
! AssertState(!IsUnderPostmaster || IsAutoVacuumWorkerProcess());
/* call only once */
AssertState(!OidIsValid(AuthenticatedUserId));
--- 498,507 ----
InitializeSessionUserIdStandalone(void)
{
/*
! * This function should only be called in single-user mode, in
! * autovacuum workers, and in background workers.
*/
! AssertState(!IsUnderPostmaster || IsAutoVacuumWorkerProcess() || IsBackgroundWorker);
/* call only once */
AssertState(!OidIsValid(AuthenticatedUserId));
*** a/src/backend/utils/init/postinit.c
--- b/src/backend/utils/init/postinit.c
***************
*** 627,632 **** InitPostgres(const char *in_dbname, Oid dboid, const char *username,
--- 627,645 ----
errhint("You should immediately run CREATE USER \"%s\" SUPERUSER;.",
username)));
}
+ else if (IsBackgroundWorker)
+ {
+ if (username == NULL)
+ {
+ InitializeSessionUserIdStandalone();
+ am_superuser = true;
+ }
+ else
+ {
+ InitializeSessionUserId(username);
+ am_superuser = superuser();
+ }
+ }
else
{
/* normal multiuser case */
*** a/src/backend/utils/misc/guc.c
--- b/src/backend/utils/misc/guc.c
***************
*** 52,57 ****
--- 52,58 ----
#include "parser/scansup.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
+ #include "postmaster/bgworker.h"
#include "postmaster/bgwriter.h"
#include "postmaster/postmaster.h"
#include "postmaster/syslogger.h"
***************
*** 108,114 ****
* removed, we still could not exceed INT_MAX/4 because some places compute
* 4*MaxBackends without any overflow check. This is rechecked in
* check_maxconnections, since MaxBackends is computed as MaxConnections
! * plus autovacuum_max_workers plus one (for the autovacuum launcher).
*/
#define MAX_BACKENDS 0x7fffff
--- 109,116 ----
* removed, we still could not exceed INT_MAX/4 because some places compute
* 4*MaxBackends without any overflow check. This is rechecked in
* check_maxconnections, since MaxBackends is computed as MaxConnections
! * plus the number of bgworkers plus autovacuum_max_workers plus one (for the
! * autovacuum launcher).
*/
#define MAX_BACKENDS 0x7fffff
***************
*** 8628,8634 **** show_tcp_keepalives_count(void)
static bool
check_maxconnections(int *newval, void **extra, GucSource source)
{
! if (*newval + autovacuum_max_workers + 1 > MAX_BACKENDS)
return false;
return true;
}
--- 8630,8637 ----
static bool
check_maxconnections(int *newval, void **extra, GucSource source)
{
! if (*newval + GetNumShmemAttachedBgworkers() + autovacuum_max_workers + 1 >
! MAX_BACKENDS)
return false;
return true;
}
***************
*** 8636,8648 **** check_maxconnections(int *newval, void **extra, GucSource source)
static void
assign_maxconnections(int newval, void *extra)
{
! MaxBackends = newval + autovacuum_max_workers + 1;
}
static bool
check_autovacuum_max_workers(int *newval, void **extra, GucSource source)
{
! if (MaxConnections + *newval + 1 > MAX_BACKENDS)
return false;
return true;
}
--- 8639,8653 ----
static void
assign_maxconnections(int newval, void *extra)
{
! MaxBackends = newval + autovacuum_max_workers + 1 +
! GetNumShmemAttachedBgworkers();
}
static bool
check_autovacuum_max_workers(int *newval, void **extra, GucSource source)
{
! if (MaxConnections + *newval + 1 + GetNumShmemAttachedBgworkers() >
! MAX_BACKENDS)
return false;
return true;
}
***************
*** 8650,8656 **** check_autovacuum_max_workers(int *newval, void **extra, GucSource source)
static void
assign_autovacuum_max_workers(int newval, void *extra)
{
! MaxBackends = MaxConnections + newval + 1;
}
static bool
--- 8655,8661 ----
static void
assign_autovacuum_max_workers(int newval, void *extra)
{
! MaxBackends = MaxConnections + newval + 1 + GetNumShmemAttachedBgworkers();
}
static bool
*** a/src/include/miscadmin.h
--- b/src/include/miscadmin.h
***************
*** 131,136 **** do { \
--- 131,137 ----
extern pid_t PostmasterPid;
extern bool IsPostmasterEnvironment;
extern PGDLLIMPORT bool IsUnderPostmaster;
+ extern bool IsBackgroundWorker;
extern bool IsBinaryUpgrade;
extern bool ExitOnAnyError;
*** /dev/null
--- b/src/include/postmaster/bgworker.h
***************
*** 0 ****
--- 1,104 ----
+ /*--------------------------------------------------------------------
+ * bgworker.h
+ * POSTGRES pluggable background workers interface
+ *
+ * A background worker is a process able to run arbitrary, user-supplied code,
+ * including normal transactions.
+ *
+ * Any external module loaded via shared_preload_libraries can register a
+ * worker. Then, at the appropriate time, the worker process is forked from
+ * the postmaster and runs the user-supplied "main" function. This code may
+ * connect to a database and run transactions. Once started, it stays active
+ * until shutdown or crash. The process should sleep during periods of
+ * inactivity.
+ *
+ * If the fork() call fails in the postmaster, it will try again later. Note
+ * that the failure can only be transient (fork failure due to high load,
+ * memory pressure, too many processes, etc); more permanent problems, like
+ * failure to connect to a database, are detected later in the worker and dealt
+ * with just by having the worker exit normally. Postmaster will launch a new
+ * worker again later.
+ *
+ * Note that there might be more than one worker in a database concurrently,
+ * and the same module may request more than one worker running the same (or
+ * different) code.
+ *
+ *
+ * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/include/postmaster/bgworker.h
+ *--------------------------------------------------------------------
+ */
+ #ifndef BGWORKER_H
+ #define BGWORKER_H
+
+ /*---------------------------------------------------------------------
+ * External module API.
+ *---------------------------------------------------------------------
+ */
+
+ /*
+ * Pass this flag to have your worker be able to connect to shared memory.
+ */
+ #define BGWORKER_SHMEM_ACCESS 0x0001
+
+ /*
+ * This flag means the bgworker requires a database connection. The connection
+ * is not established automatically; the worker must establish it later.
+ * It requires that BGWORKER_SHMEM_ACCESS was passed too.
+ */
+ #define BGWORKER_BACKEND_DATABASE_CONNECTION 0x0002
+
+
+ typedef void (*bgworker_main_type)(void *main_arg);
+ typedef void (*bgworker_sighdlr_type)(SIGNAL_ARGS);
+
+ /*
+ * Points in time at which a bgworker can request to be started
+ */
+ typedef enum
+ {
+ BgWorkerStart_PostmasterStart,
+ BgWorkerStart_ConsistentState,
+ BgWorkerStart_RecoveryFinished
+ } BgWorkerStartTime;
+
+ #define BGW_DEFAULT_RESTART_INTERVAL 60
+ #define BGW_NEVER_RESTART -1
+
+ typedef struct BackgroundWorker
+ {
+ char *bgw_name;
+ int bgw_flags;
+ BgWorkerStartTime bgw_start_time;
+ int bgw_restart_time; /* in seconds, or BGW_NEVER_RESTART */
+ bgworker_main_type bgw_main;
+ void *bgw_main_arg;
+ bgworker_sighdlr_type bgw_sighup;
+ bgworker_sighdlr_type bgw_sigterm;
+ } BackgroundWorker;
+
+ /* Register a new bgworker */
+ extern void RegisterBackgroundWorker(BackgroundWorker *worker);
+
+ /* This is valid in a running worker */
+ extern BackgroundWorker *MyBgworkerEntry;
+
+ /*
+ * Connect to the specified database, as the specified user. Only a worker
+ * that passed BGWORKER_BACKEND_DATABASE_CONNECTION during registration may
+ * call this.
+ *
+ * If username is NULL, bootstrapping superuser is used.
+ * If dbname is NULL, connection is made to no specific database;
+ * only shared catalogs can be accessed.
+ */
+ extern void BackgroundWorkerInitializeConnection(char *dbname, char *username);
+
+ /* Block/unblock signals in a background worker process */
+ extern void BackgroundWorkerBlockSignals(void);
+ extern void BackgroundWorkerUnblockSignals(void);
+
+ #endif /* BGWORKER_H */
*** a/src/include/postmaster/postmaster.h
--- b/src/include/postmaster/postmaster.h
***************
*** 51,56 **** extern void ClosePostmasterPorts(bool am_syslogger);
--- 51,58 ----
extern int MaxLivePostmasterChildren(void);
+ extern int GetNumShmemAttachedBgworkers(void);
+
#ifdef EXEC_BACKEND
extern pid_t postmaster_forkexec(int argc, char *argv[]);
extern void SubPostmasterMain(int argc, char *argv[]) __attribute__((noreturn));
*** a/src/include/storage/proc.h
--- b/src/include/storage/proc.h
***************
*** 189,194 **** typedef struct PROC_HDR
--- 189,196 ----
PGPROC *freeProcs;
/* Head of list of autovacuum's free PGPROC structures */
PGPROC *autovacFreeProcs;
+ /* Head of list of bgworker free PGPROC structures */
+ PGPROC *bgworkerFreeProcs;
/* WALWriter process's latch */
Latch *walwriterLatch;
/* Checkpointer process's latch */
On 12/03/2012 10:35 PM, Alvaro Herrera wrote:
So here's version 8. This fixes a couple of bugs and most notably
creates a separate PGPROC list for bgworkers, so that they don't
interfere with client-connected backends.
Nice, thanks. I'll try to review this again, shortly.
Regards
Markus Wanner
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Alvaro Herrera wrote:
One notable thing is that I had to introduce this in the postmaster
startup sequence:/*
* process any libraries that should be preloaded at postmaster start
*/
process_shared_preload_libraries();/*
* If loadable modules have added background workers, MaxBackends needs to
* be updated. Do so now.
*/
// RerunAssignHook("max_connections");
if (GetNumShmemAttachedBgworkers() > 0)
SetConfigOption("max_connections",
GetConfigOption("max_connections", false, false),
PGC_POSTMASTER, PGC_S_OVERRIDE);Note the intention here is to re-run the GUC assign hook for
max_connections (hence the commented out hypothetical call to do so).
Obviously, having to go through GetConfigOption and SetConfigOption is
not a nice thing to do; we'll have to add some new entry point to guc.c
for this to have a nicer interface.
After fooling with guc.c to create such a new entry point, I decided
that it looks too ugly. guc.c is already complex enough with the API we
have today that I don't want to be seen creating a partially-duplicate
interface, even if it is going to result in simplified processing of
this one place. If we ever get around to needing another place to
require rerunning a variable's assign_hook we can discuss it; for now it
doesn't seem worth it.
This is only called at postmaster start time, so it's not too
performance-sensitive, hence SetConfigOption( .., GetConfigOption(), ..)
seems acceptable from that POV.
--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Here's a first attempt at a new documentation chapter. This goes in
part "Server Programming", just after the SPI chapter.
I just noticed that worker_spi could use some more sample code, for
example auth_counter was getting its own LWLock and also its own shmem
area, which would be helpful to demonstrate I think.
--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Attachments:
On 2012-12-05 12:09:58 -0300, Alvaro Herrera wrote:
Here's a first attempt at a new documentation chapter. This goes in
part "Server Programming", just after the SPI chapter.I just noticed that worker_spi could use some more sample code, for
example auth_counter was getting its own LWLock and also its own shmem
area, which would be helpful to demonstrate I think.
I am not exactly renowned for my english skills, but I have made a pass
over the file made some slight changes and extended it in two places.
I've also added a comment with a question that came to my mind when
reading the docs...
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Attachments:
bgworker.patchtext/x-patch; charset=us-asciiDownload
--- /tmp/bgworker.old.sgml 2012-12-05 22:12:00.220609454 +0100
+++ /tmp/bgworker.sgml 2012-12-05 22:26:11.604712943 +0100
@@ -11,16 +11,16 @@
<para>
PostgreSQL can be extended to run user-supplied code in separate processes.
Such processes are started, stopped and monitored by <command>postgres</command>,
- which permits them have a lifetime closely linked to the server's status.
+ which permits them to have a lifetime closely linked to the server's status.
These processes have the option to attach to <productname>PostgreSQL</>'s
- shared memory area and connect to databases internally.
+ shared memory area and to connect to databases internally.
</para>
<warning>
<para>
There are considerable robustness and security risks in using background
- worker processes, because them being written in the <literal>C</> language
- gives them unrestricted access to data. Administrators wishing to enable
+ worker processes because, being written in the <literal>C</> language,
+ they have unrestricted access to data. Administrators wishing to enable
modules that include background worker process should exercise extreme
caution. Only carefully audited modules should be permitted to run
background worker processes.
@@ -31,7 +31,8 @@
Only modules listed in <varname>shared_preload_libraries</> can run
background workers. A module wishing to register a background worker needs
to register it by calling
- <function>RegisterBackgroundWorker(<type>BackgroundWorker *worker</type>)</function>.
+ <function>RegisterBackgroundWorker(<type>BackgroundWorker *worker</type>)</function>
+ from it's <function>_PG_Init()</>.
The structure <structname>BackgroundWorker</structname> is defined thus:
<programlisting>
typedef struct BackgroundWorker
@@ -50,12 +51,12 @@
<para>
<structfield>bgw_name</> is a string to be used in log messages, process
- lists and similar contexts.
+ listings and similar contexts.
</para>
<para>
<structfield>bgw_flags</> is a bitwise-or'd bitmask indicating the
- capabilities that the module would like to have. Possible values are
+ capabilities that the module wants. Possible values are
<literal>BGWORKER_SHMEM_ACCESS</literal> (requesting shared memory access)
and <literal>BGWORKER_BACKEND_DATABASE_CONNECTION</literal> (requesting the
ability to establish a database connection, through which it can later run
@@ -68,33 +69,35 @@
<literal>BgWorkerStart_PostmasterStart</> (start as soon as
<command>postgres</> itself has finished its own initialization; processes
requesting this are not eligible for database connections),
- <literal>BgWorkerStart_ConsistentState</> (start as soon as consistent state
+ <literal>BgWorkerStart_ConsistentState</> (start as soon as a consistent state
has been reached in a HOT standby, allowing processes to connect to
databases and run read-only queries), and
<literal>BgWorkerStart_RecoveryFinished</> (start as soon as the system has
entered normal read-write state). Note the last two values are equivalent
in a server that's not a HOT standby.
</para>
-
+
<para>
<structfield>bgw_restart_time</structfield> is the interval, in seconds, that
- <command>postgres</command> should wait before restarting the process,
- in case it crashes. It can be any positive value, or <literal>BGW_NEVER_RESTART</literal>, indicating not to restart the process in case of a crash.
+ <command>postgres</command> should wait before restarting the process, in
+ case it crashes. It can be any positive value,
+ or <literal>BGW_NEVER_RESTART</literal>, indicating not to restart the
+ process in case of a crash.
</para>
<para>
<structfield>bgw_main</structfield> is a pointer to the function to run once
the process is started. <structfield>bgw_main_arg</structfield> will be
- passed to it as its only argument. Note that
- <literal>MyBgworkerEntry</literal> is a pointer to a copy of the
- <structname>BackgroundWorker</structname> structure passed
- at registration time.
+ passed to it as its only argument. Note that the global variable
+ <literal>MyBgworkerEntry</literal> points to a copy of the
+ <structname>BackgroundWorker</structname> structure passed at registration
+ time.
</para>
<para>
<structfield>bgw_sighup</structfield> and <structfield>bgw_sigterm</> are
pointers to functions that will be installed as signal handlers for the new
- process.
+ process. XXX: Can they be NULL?
</para>
<para>Once running, the process can connect to a database by calling
@@ -104,6 +107,8 @@
the session is not connected to any particular database, but shared catalogs
can be accessed. If <varname>username</> is NULL, the process will run as
the superuser created during <command>initdb</>.
+ BackgroundWorkerInitializeConnection can only be called once per background
+ process, it is not possible to switch databases.
</para>
<para>
Andres Freund wrote:
On 2012-12-05 12:09:58 -0300, Alvaro Herrera wrote:
Here's a first attempt at a new documentation chapter. This goes in
part "Server Programming", just after the SPI chapter.I just noticed that worker_spi could use some more sample code, for
example auth_counter was getting its own LWLock and also its own shmem
area, which would be helpful to demonstrate I think.I am not exactly renowned for my english skills, but I have made a pass
over the file made some slight changes and extended it in two places.
Thanks, I have applied it.
I've also added a comment with a question that came to my mind when
reading the docs...
<para>
<structfield>bgw_sighup</structfield> and <structfield>bgw_sigterm</> are
pointers to functions that will be installed as signal handlers for the new
- process.
+ process. XXX: Can they be NULL?
</para>
Hm. The code doesn't check, so what happens is probably a bug anyhow.
I don't know whether sigaction crashes in this case; its manpage doesn't
say. I guess the right thing to do is have RegisterBackgroundWorker
check for a NULL sighandler, and set it to something standard if so (say
SIG_IGN for SIGHUP and maybe quickdie() or similar for SIGTERM).
--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 5 December 2012 15:09, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:
Here's a first attempt at a new documentation chapter. This goes in
part "Server Programming", just after the SPI chapter.I just noticed that worker_spi could use some more sample code, for
example auth_counter was getting its own LWLock and also its own shmem
area, which would be helpful to demonstrate I think.
"to run once" -> "to run when"
Prefer
BgWorkerStart_ConsistentState to be renamed to BgWorkerRun_InHotStandby
BgWorkerStart_RecoveryFinished to be renamed to BgWorkerRun_InNormalMode
presumably a process will shutdown if (BgWorkerRun_InHotStandby &&
!BgWorkerRun_InNormalMode)
--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Simon Riggs wrote:
On 5 December 2012 15:09, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:
Here's a first attempt at a new documentation chapter. This goes in
part "Server Programming", just after the SPI chapter.I just noticed that worker_spi could use some more sample code, for
example auth_counter was getting its own LWLock and also its own shmem
area, which would be helpful to demonstrate I think."to run once" -> "to run when"
Thanks.
Prefer
BgWorkerStart_ConsistentState to be renamed to BgWorkerRun_InHotStandby
BgWorkerStart_RecoveryFinished to be renamed to BgWorkerRun_InNormalModepresumably a process will shutdown if (BgWorkerRun_InHotStandby &&
!BgWorkerRun_InNormalMode)
Hmm, no, I haven't considered that. You mean that a bgworker that
specifies to start at BgWorkerStart_ConsistentState will stop once
normal mode is reached? Currently they don't do that. And since we
don't have the notion that workers stop working, it wouldn't work --
postmaster would start them back up immediately.
--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2012-12-05 19:03:44 -0300, Alvaro Herrera wrote:
Simon Riggs wrote:
Prefer
BgWorkerStart_ConsistentState to be renamed to BgWorkerRun_InHotStandby
BgWorkerStart_RecoveryFinished to be renamed to BgWorkerRun_InNormalModepresumably a process will shutdown if (BgWorkerRun_InHotStandby &&
!BgWorkerRun_InNormalMode)Hmm, no, I haven't considered that. You mean that a bgworker that
specifies to start at BgWorkerStart_ConsistentState will stop once
normal mode is reached? Currently they don't do that. And since we
don't have the notion that workers stop working, it wouldn't work --
postmaster would start them back up immediately.
I personally don't see too much demand for this from a use-case
perspective. Simon, did you have anything in mind that made you ask
this?
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2012-12-05 18:42:42 -0300, Alvaro Herrera wrote:
<para>
<structfield>bgw_sighup</structfield> and <structfield>bgw_sigterm</> are
pointers to functions that will be installed as signal handlers for the new
- process.
+ process. XXX: Can they be NULL?
</para>Hm. The code doesn't check, so what happens is probably a bug anyhow.
I don't know whether sigaction crashes in this case; its manpage doesn't
say. I guess the right thing to do is have RegisterBackgroundWorker
check for a NULL sighandler, and set it to something standard if so (say
SIG_IGN for SIGHUP and maybe quickdie() or similar for SIGTERM).
Afair a NULL sigaction is used to query the current handler. Which
indirectly might lead to problems due to the wrong handler being called.
Setting up SIG_IGN and quickdie in that case seems to be sensible.
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 5 December 2012 22:22, Andres Freund <andres@2ndquadrant.com> wrote:
On 2012-12-05 19:03:44 -0300, Alvaro Herrera wrote:
Simon Riggs wrote:
Prefer
BgWorkerStart_ConsistentState to be renamed to BgWorkerRun_InHotStandby
BgWorkerStart_RecoveryFinished to be renamed to BgWorkerRun_InNormalModepresumably a process will shutdown if (BgWorkerRun_InHotStandby &&
!BgWorkerRun_InNormalMode)Hmm, no, I haven't considered that. You mean that a bgworker that
specifies to start at BgWorkerStart_ConsistentState will stop once
normal mode is reached? Currently they don't do that. And since we
don't have the notion that workers stop working, it wouldn't work --
postmaster would start them back up immediately.I personally don't see too much demand for this from a use-case
perspective. Simon, did you have anything in mind that made you ask
this?
Just clarifying how it worked, for the docs; happy with the way it is.
--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Andres Freund wrote:
On 2012-12-05 18:42:42 -0300, Alvaro Herrera wrote:
<para>
<structfield>bgw_sighup</structfield> and <structfield>bgw_sigterm</> are
pointers to functions that will be installed as signal handlers for the new
- process.
+ process. XXX: Can they be NULL?
</para>Hm. The code doesn't check, so what happens is probably a bug anyhow.
I don't know whether sigaction crashes in this case; its manpage doesn't
say. I guess the right thing to do is have RegisterBackgroundWorker
check for a NULL sighandler, and set it to something standard if so (say
SIG_IGN for SIGHUP and maybe quickdie() or similar for SIGTERM).Afair a NULL sigaction is used to query the current handler. Which
indirectly might lead to problems due to the wrong handler being called.Setting up SIG_IGN and quickdie in that case seems to be sensible.
Here's v9. It adds that change, the tweaked docs, a bug fix (postmaster
would kill itself if there's a process crash and a bgworker is stopped),
and some pgindent and code reordering.
github.com/alvherre/postgres/tree/bgworker contains this submission as
commit fa4731c.
I think we can get this committed as a useful starting point for this
feature.
--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Attachments:
bgworker-9.patchtext/x-diff; charset=us-asciiDownload
*** a/contrib/Makefile
--- b/contrib/Makefile
***************
*** 50,56 **** SUBDIRS = \
test_parser \
tsearch2 \
unaccent \
! vacuumlo
ifeq ($(with_openssl),yes)
SUBDIRS += sslinfo
--- 50,57 ----
test_parser \
tsearch2 \
unaccent \
! vacuumlo \
! worker_spi
ifeq ($(with_openssl),yes)
SUBDIRS += sslinfo
*** /dev/null
--- b/contrib/worker_spi/Makefile
***************
*** 0 ****
--- 1,14 ----
+ # contrib/worker_spi/Makefile
+
+ MODULES = worker_spi
+
+ ifdef USE_PGXS
+ PG_CONFIG = pg_config
+ PGXS := $(shell $(PG_CONFIG) --pgxs)
+ include $(PGXS)
+ else
+ subdir = contrib/worker_spi
+ top_builddir = ../..
+ include $(top_builddir)/src/Makefile.global
+ include $(top_srcdir)/contrib/contrib-global.mk
+ endif
*** /dev/null
--- b/contrib/worker_spi/worker_spi.c
***************
*** 0 ****
--- 1,263 ----
+ /* -------------------------------------------------------------------------
+ *
+ * worker_spi.c
+ * Sample background worker code that demonstrates usage of a database
+ * connection.
+ *
+ * This code connects to a database, create a schema and table, and summarizes
+ * the numbers contained therein. To see it working, insert an initial value
+ * with "total" type and some initial value; then insert some other rows with
+ * "delta" type. Delta rows will be deleted by this worker and their values
+ * aggregated into the total.
+ *
+ * Copyright (C) 2012, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * contrib/worker_spi/worker_spi.c
+ *
+ * -------------------------------------------------------------------------
+ */
+ #include "postgres.h"
+
+ /* These are always necessary for a bgworker */
+ #include "miscadmin.h"
+ #include "postmaster/bgworker.h"
+ #include "storage/ipc.h"
+ #include "storage/latch.h"
+ #include "storage/lwlock.h"
+ #include "storage/proc.h"
+ #include "storage/shmem.h"
+
+ /* these headers are used by this particular worker's code */
+ #include "access/xact.h"
+ #include "executor/spi.h"
+ #include "fmgr.h"
+ #include "lib/stringinfo.h"
+ #include "utils/builtins.h"
+ #include "utils/snapmgr.h"
+
+ PG_MODULE_MAGIC;
+
+ void _PG_init(void);
+
+ static bool got_sigterm = false;
+
+
+ typedef struct worktable
+ {
+ const char *schema;
+ const char *name;
+ } worktable;
+
+ static void
+ worker_spi_sigterm(SIGNAL_ARGS)
+ {
+ int save_errno = errno;
+
+ got_sigterm = true;
+ if (MyProc)
+ SetLatch(&MyProc->procLatch);
+
+ errno = save_errno;
+ }
+
+ static void
+ worker_spi_sighup(SIGNAL_ARGS)
+ {
+ elog(LOG, "got sighup!");
+ if (MyProc)
+ SetLatch(&MyProc->procLatch);
+ }
+
+ static void
+ initialize_worker_spi(worktable *table)
+ {
+ int ret;
+ int ntup;
+ bool isnull;
+ StringInfoData buf;
+
+ StartTransactionCommand();
+ SPI_connect();
+ PushActiveSnapshot(GetTransactionSnapshot());
+
+ initStringInfo(&buf);
+ appendStringInfo(&buf, "select count(*) from pg_namespace where nspname = '%s'",
+ table->schema);
+
+ ret = SPI_execute(buf.data, true, 0);
+ if (ret != SPI_OK_SELECT)
+ elog(FATAL, "SPI_execute failed: error code %d", ret);
+
+ if (SPI_processed != 1)
+ elog(FATAL, "not a singleton result");
+
+ ntup = DatumGetInt32(SPI_getbinval(SPI_tuptable->vals[0],
+ SPI_tuptable->tupdesc,
+ 1, &isnull));
+ if (isnull)
+ elog(FATAL, "null result");
+
+ if (ntup == 0)
+ {
+ resetStringInfo(&buf);
+ appendStringInfo(&buf,
+ "CREATE SCHEMA \"%s\" "
+ "CREATE TABLE \"%s\" ("
+ " type text CHECK (type IN ('total', 'delta')), "
+ " value integer)"
+ "CREATE UNIQUE INDEX \"%s_unique_total\" ON \"%s\" (type) "
+ "WHERE type = 'total'",
+ table->schema, table->name, table->name, table->name);
+
+ ret = SPI_execute(buf.data, false, 0);
+
+ if (ret != SPI_OK_UTILITY)
+ elog(FATAL, "failed to create my schema");
+ }
+
+ SPI_finish();
+ PopActiveSnapshot();
+ CommitTransactionCommand();
+ }
+
+ static void
+ worker_spi_main(void *main_arg)
+ {
+ worktable *table = (worktable *) main_arg;
+ StringInfoData buf;
+
+ /* We're now ready to receive signals */
+ BackgroundWorkerUnblockSignals();
+
+ /* Connect to our database */
+ BackgroundWorkerInitializeConnection("postgres", NULL);
+
+ elog(LOG, "%s initialized with %s.%s",
+ MyBgworkerEntry->bgw_name, table->schema, table->name);
+ initialize_worker_spi(table);
+
+ /*
+ * Quote identifiers passed to us. Note that this must be done after
+ * initialize_worker_spi, because that routine assumes the names are not
+ * quoted.
+ *
+ * Note some memory might be leaked here.
+ */
+ table->schema = quote_identifier(table->schema);
+ table->name = quote_identifier(table->name);
+
+ initStringInfo(&buf);
+ appendStringInfo(&buf,
+ "WITH deleted AS (DELETE "
+ "FROM %s.%s "
+ "WHERE type = 'delta' RETURNING value), "
+ "total AS (SELECT coalesce(sum(value), 0) as sum "
+ "FROM deleted) "
+ "UPDATE %s.%s "
+ "SET value = %s.value + total.sum "
+ "FROM total WHERE type = 'total' "
+ "RETURNING %s.value",
+ table->schema, table->name,
+ table->schema, table->name,
+ table->name,
+ table->name);
+
+ while (!got_sigterm)
+ {
+ int ret;
+ int rc;
+
+ /*
+ * Background workers mustn't call usleep() or any direct equivalent:
+ * instead, they may wait on their process latch, which sleeps as
+ * necessary, but is awakened if postmaster dies. That way the
+ * background process goes away immediately in an emergency.
+ */
+ rc = WaitLatch(&MyProc->procLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ 1000L);
+ ResetLatch(&MyProc->procLatch);
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+
+ StartTransactionCommand();
+ SPI_connect();
+ PushActiveSnapshot(GetTransactionSnapshot());
+
+ ret = SPI_execute(buf.data, false, 0);
+
+ if (ret != SPI_OK_UPDATE_RETURNING)
+ elog(FATAL, "cannot select from table %s.%s: error code %d",
+ table->schema, table->name, ret);
+
+ if (SPI_processed > 0)
+ {
+ bool isnull;
+ int32 val;
+
+ val = DatumGetInt32(SPI_getbinval(SPI_tuptable->vals[0],
+ SPI_tuptable->tupdesc,
+ 1, &isnull));
+ if (!isnull)
+ elog(LOG, "%s: count in %s.%s is now %d",
+ MyBgworkerEntry->bgw_name,
+ table->schema, table->name, val);
+ }
+
+ SPI_finish();
+ PopActiveSnapshot();
+ CommitTransactionCommand();
+ }
+
+ proc_exit(0);
+ }
+
+ /*
+ * Entrypoint of this module.
+ *
+ * We register two worker processes here, to demonstrate how that can be done.
+ */
+ void
+ _PG_init(void)
+ {
+ BackgroundWorker worker;
+ worktable *table;
+
+ /* register the worker processes. These values are common for both */
+ worker.bgw_flags = BGWORKER_SHMEM_ACCESS |
+ BGWORKER_BACKEND_DATABASE_CONNECTION;
+ worker.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ worker.bgw_main = worker_spi_main;
+ worker.bgw_sighup = worker_spi_sighup;
+ worker.bgw_sigterm = worker_spi_sigterm;
+
+ /*
+ * These values are used for the first worker.
+ *
+ * Note these are palloc'd. The reason this works after starting a new
+ * worker process is that if we only fork, they point to valid allocated
+ * memory in the child process; and if we fork and then exec, the exec'd
+ * process will run this code again, and so the memory is also valid there.
+ */
+ table = palloc(sizeof(worktable));
+ table->schema = pstrdup("schema1");
+ table->name = pstrdup("counted");
+
+ worker.bgw_name = "SPI worker 1";
+ worker.bgw_restart_time = BGW_NEVER_RESTART;
+ worker.bgw_main_arg = (void *) table;
+ RegisterBackgroundWorker(&worker);
+
+ /* Values for the second worker */
+ table = palloc(sizeof(worktable));
+ table->schema = pstrdup("our schema2");
+ table->name = pstrdup("counted rows");
+
+ worker.bgw_name = "SPI worker 2";
+ worker.bgw_restart_time = 2;
+ worker.bgw_main_arg = (void *) table;
+ RegisterBackgroundWorker(&worker);
+ }
*** /dev/null
--- b/doc/src/sgml/bgworker.sgml
***************
*** 0 ****
--- 1,146 ----
+ <!-- doc/src/sgml/bgworker.sgml -->
+
+ <chapter id="bgworker">
+ <title>Background Worker Processes</title>
+
+ <indexterm zone="bgworker">
+ <primary>Background workers</primary>
+ </indexterm>
+
+ <para>
+ PostgreSQL can be extended to run user-supplied code in separate processes.
+ Such processes are started, stopped and monitored by <command>postgres</command>,
+ which permits them to have a lifetime closely linked to the server's status.
+ These processes have the option to attach to <productname>PostgreSQL</>'s
+ shared memory area and to connect to databases internally; they can also run
+ multiple transactions serially, just like a regular client-connected server
+ process. Also, by linking to <application>libpq</> they can connect to the
+ server and behave like a regular client application.
+ </para>
+
+ <warning>
+ <para>
+ There are considerable robustness and security risks in using background
+ worker processes because, being written in the <literal>C</> language,
+ they have unrestricted access to data. Administrators wishing to enable
+ modules that include background worker process should exercise extreme
+ caution. Only carefully audited modules should be permitted to run
+ background worker processes.
+ </para>
+ </warning>
+
+ <para>
+ Only modules listed in <varname>shared_preload_libraries</> can run
+ background workers. A module wishing to run a background worker needs
+ to register it by calling
+ <function>RegisterBackgroundWorker(<type>BackgroundWorker *worker</type>)</function>
+ from its <function>_PG_init()</>.
+ The structure <structname>BackgroundWorker</structname> is defined thus:
+ <programlisting>
+ typedef void (*bgworker_main_type)(void *main_arg);
+ typedef void (*bgworker_sighdlr_type)(SIGNAL_ARGS);
+ typedef struct BackgroundWorker
+ {
+ char *bgw_name;
+ int bgw_flags;
+ BgWorkerStartTime bgw_start_time;
+ int bgw_restart_time; /* in seconds, or BGW_NEVER_RESTART */
+ bgworker_main_type bgw_main;
+ void *bgw_main_arg;
+ bgworker_sighdlr_type bgw_sighup;
+ bgworker_sighdlr_type bgw_sigterm;
+ } BackgroundWorker;
+ </programlisting>
+ </para>
+
+ <para>
+ <structfield>bgw_name</> is a string to be used in log messages, process
+ listings and similar contexts.
+ </para>
+
+ <para>
+ <structfield>bgw_flags</> is a bitwise-or'd bitmask indicating the
+ capabilities that the module wants. Possible values are
+ <literal>BGWORKER_SHMEM_ACCESS</literal> (requesting shared memory access)
+ and <literal>BGWORKER_BACKEND_DATABASE_CONNECTION</literal> (requesting the
+ ability to establish a database connection, through which it can later run
+ transactions and queries).
+ </para>
+
+ <para>
+ <structfield>bgw_start_time</structfield> is the server state during which
+ <command>postgres</> should start the process; it can be one of
+ <literal>BgWorkerStart_PostmasterStart</> (start as soon as
+ <command>postgres</> itself has finished its own initialization; processes
+ requesting this are not eligible for database connections),
+ <literal>BgWorkerStart_ConsistentState</> (start as soon as a consistent state
+ has been reached in a HOT standby, allowing processes to connect to
+ databases and run read-only queries), and
+ <literal>BgWorkerStart_RecoveryFinished</> (start as soon as the system has
+ entered normal read-write state). Note the last two values are equivalent
+ in a server that's not a HOT standby. Note that this setting only indicates
+ when the processes are to be started; they do not stop when a different state
+ is reached.
+ </para>
+
+ <para>
+ <structfield>bgw_restart_time</structfield> is the interval, in seconds, that
+ <command>postgres</command> should wait before restarting the process, in
+ case it crashes. It can be any positive value,
+ or <literal>BGW_NEVER_RESTART</literal>, indicating not to restart the
+ process in case of a crash.
+ </para>
+
+ <para>
+ <structfield>bgw_main</structfield> is a pointer to the function to run when
+ the process is started. This function must take a single argument of type
+ <type>void *</> and return <type>void</>.
+ <structfield>bgw_main_arg</structfield> will be passed to it as its only
+ argument. Note that the global variable <literal>MyBgworkerEntry</literal>
+ points to a copy of the <structname>BackgroundWorker</structname> structure
+ passed at registration time.
+ </para>
+
+ <para>
+ <structfield>bgw_sighup</structfield> and <structfield>bgw_sigterm</> are
+ pointers to functions that will be installed as signal handlers for the new
+ process. If <structfield>bgw_sighup</> is NULL, then <literal>SIG_IGN</>
+ is used; if <structfield>bgw_sigterm</> is NULL, a handler is installed that
+ will terminate the process after logging a suitable message.
+ </para>
+
+ <para>Once running, the process can connect to a database by calling
+ <function>BackgroundWorkerInitializeConnection(<parameter>char *dbname</parameter>, <parameter>char *username</parameter>)</function>.
+ This allows the process to run transactions and queries using the
+ <literal>SPI</literal> interface. If <varname>dbname</> is NULL,
+ the session is not connected to any particular database, but shared catalogs
+ can be accessed. If <varname>username</> is NULL, the process will run as
+ the superuser created during <command>initdb</>.
+ BackgroundWorkerInitializeConnection can only be called once per background
+ process, it is not possible to switch databases.
+ </para>
+
+ <para>
+ Signals are initially blocked when control reaches the
+ <structfield>bgw_main</> function, and must be unblocked by it; this is to
+ allow the process to further customize its signal handlers, if necessary.
+ Signals can be unblocked in the new process by calling
+ <function>BackgroundWorkerUnblockSignals</> and blocked by calling
+ <function>BackgroundWorkerBlockSignals</>.
+ </para>
+
+ <para>
+ Background workers are expected to be continuously running; if they exit
+ cleanly, <command>postgres</> will restart them immediately. Consider doing
+ interruptible sleep when they have nothing to do; this can be achieved by
+ calling <function>WaitLatch()</function>. Make sure the
+ <literal>WL_POSTMASTER_DEATH</> flag is set when calling that function, and
+ verify the return code for a prompt exit in the emergency case that
+ <command>postgres</> itself has terminated.
+ </para>
+
+ <para>
+ The <filename>worker_spi</> contrib module contains a working example,
+ which demonstrates some useful techniques.
+ </para>
+ </chapter>
*** a/doc/src/sgml/filelist.sgml
--- b/doc/src/sgml/filelist.sgml
***************
*** 50,55 ****
--- 50,56 ----
<!ENTITY wal SYSTEM "wal.sgml">
<!-- programmer's guide -->
+ <!ENTITY bgworker SYSTEM "bgworker.sgml">
<!ENTITY dfunc SYSTEM "dfunc.sgml">
<!ENTITY ecpg SYSTEM "ecpg.sgml">
<!ENTITY extend SYSTEM "extend.sgml">
*** a/doc/src/sgml/postgres.sgml
--- b/doc/src/sgml/postgres.sgml
***************
*** 218,223 ****
--- 218,224 ----
&plpython;
&spi;
+ &bgworker;
</part>
*** a/src/backend/postmaster/postmaster.c
--- b/src/backend/postmaster/postmaster.c
***************
*** 103,108 ****
--- 103,109 ----
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
+ #include "postmaster/bgworker.h"
#include "postmaster/fork_process.h"
#include "postmaster/pgarch.h"
#include "postmaster/postmaster.h"
***************
*** 126,150 ****
/*
* List of active backends (or child processes anyway; we don't actually
* know whether a given child has become a backend or is still in the
* authorization phase). This is used mainly to keep track of how many
* children we have and send them appropriate signals when necessary.
*
* "Special" children such as the startup, bgwriter and autovacuum launcher
! * tasks are not in this list. Autovacuum worker and walsender processes are
! * in it. Also, "dead_end" children are in it: these are children launched just
! * for the purpose of sending a friendly rejection message to a would-be
! * client. We must track them because they are attached to shared memory,
! * but we know they will never become live backends. dead_end children are
! * not assigned a PMChildSlot.
*/
typedef struct bkend
{
pid_t pid; /* process id of backend */
long cancel_key; /* cancel key for cancels for this backend */
int child_slot; /* PMChildSlot for this backend, if any */
! bool is_autovacuum; /* is it an autovacuum process? */
bool dead_end; /* is it going to send an error and quit? */
dlist_node elem; /* list link in BackendList */
} Backend;
--- 127,170 ----
/*
+ * Possible types of a backend. Beyond being the possible bkend_type values in
+ * struct bkend, these are OR-able request flag bits for SignalSomeChildren()
+ * and CountChildren().
+ */
+ #define BACKEND_TYPE_NORMAL 0x0001 /* normal backend */
+ #define BACKEND_TYPE_AUTOVAC 0x0002 /* autovacuum worker process */
+ #define BACKEND_TYPE_WALSND 0x0004 /* walsender process */
+ #define BACKEND_TYPE_BGWORKER 0x0008 /* bgworker process */
+ #define BACKEND_TYPE_ALL 0x000F /* OR of all the above */
+
+ #define BACKEND_TYPE_WORKER (BACKEND_TYPE_AUTOVAC | BACKEND_TYPE_BGWORKER)
+
+ /*
* List of active backends (or child processes anyway; we don't actually
* know whether a given child has become a backend or is still in the
* authorization phase). This is used mainly to keep track of how many
* children we have and send them appropriate signals when necessary.
*
* "Special" children such as the startup, bgwriter and autovacuum launcher
! * tasks are not in this list. Autovacuum worker and walsender are in it.
! * Also, "dead_end" children are in it: these are children launched just for
! * the purpose of sending a friendly rejection message to a would-be client.
! * We must track them because they are attached to shared memory, but we know
! * they will never become live backends. dead_end children are not assigned a
! * PMChildSlot.
*/
typedef struct bkend
{
pid_t pid; /* process id of backend */
long cancel_key; /* cancel key for cancels for this backend */
int child_slot; /* PMChildSlot for this backend, if any */
!
! /*
! * Flavor of backend or auxiliary process. Note that BACKEND_TYPE_WALSND
! * backends initially announce themselves as BACKEND_TYPE_NORMAL, so if
! * bkend_type is normal, you should check for a recent transition.
! */
! int bkend_type;
bool dead_end; /* is it going to send an error and quit? */
dlist_node elem; /* list link in BackendList */
} Backend;
***************
*** 155,160 **** static dlist_head BackendList = DLIST_STATIC_INIT(BackendList);
--- 175,203 ----
static Backend *ShmemBackendArray;
#endif
+
+ /*
+ * List of background workers.
+ */
+ typedef struct RegisteredBgWorker
+ {
+ BackgroundWorker rw_worker; /* its registry entry */
+ Backend *rw_backend; /* its BackendList entry, or NULL */
+ pid_t rw_pid; /* 0 if not running */
+ int rw_child_slot;
+ TimestampTz rw_crashed_at; /* if not 0, time it last crashed */
+ #ifdef EXEC_BACKEND
+ int rw_cookie;
+ #endif
+ slist_node rw_lnode; /* list link */
+ } RegisteredBgWorker;
+
+ static slist_head BackgroundWorkerList = SLIST_STATIC_INIT(BackgroundWorkerList);
+
+ BackgroundWorker *MyBgworkerEntry = NULL;
+
+
+
/* The socket number we are listening for connections on */
int PostPortNumber;
/* The directory names for Unix socket(s) */
***************
*** 306,311 **** static volatile sig_atomic_t start_autovac_launcher = false;
--- 349,358 ----
/* the launcher needs to be signalled to communicate some condition */
static volatile bool avlauncher_needs_signal = false;
+ /* set when there's a worker that needs to be started up */
+ static volatile bool StartWorkerNeeded = true;
+ static volatile bool HaveCrashedWorker = false;
+
/*
* State for assigning random salts and cancel keys.
* Also, the global MyCancelKey passes the cancel key assigned to a given
***************
*** 341,348 **** static void reaper(SIGNAL_ARGS);
--- 388,398 ----
static void sigusr1_handler(SIGNAL_ARGS);
static void startup_die(SIGNAL_ARGS);
static void dummy_handler(SIGNAL_ARGS);
+ static int GetNumRegisteredBackgroundWorkers(int flags);
static void StartupPacketTimeoutHandler(void);
static void CleanupBackend(int pid, int exitstatus);
+ static bool CleanupBackgroundWorker(int pid, int exitstatus);
+ static void do_start_bgworker(void);
static void HandleChildCrash(int pid, int exitstatus, const char *procname);
static void LogChildExit(int lev, const char *procname,
int pid, int exitstatus);
***************
*** 361,379 **** static long PostmasterRandom(void);
static void RandomSalt(char *md5Salt);
static void signal_child(pid_t pid, int signal);
static bool SignalSomeChildren(int signal, int targets);
#define SignalChildren(sig) SignalSomeChildren(sig, BACKEND_TYPE_ALL)
- /*
- * Possible types of a backend. These are OR-able request flag bits
- * for SignalSomeChildren() and CountChildren().
- */
- #define BACKEND_TYPE_NORMAL 0x0001 /* normal backend */
- #define BACKEND_TYPE_AUTOVAC 0x0002 /* autovacuum worker process */
- #define BACKEND_TYPE_WALSND 0x0004 /* walsender process */
- #define BACKEND_TYPE_ALL 0x0007 /* OR of all the above */
-
static int CountChildren(int target);
static bool CreateOptsFile(int argc, char *argv[], char *fullprogname);
static pid_t StartChildProcess(AuxProcType type);
static void StartAutovacuumWorker(void);
--- 411,423 ----
static void RandomSalt(char *md5Salt);
static void signal_child(pid_t pid, int signal);
static bool SignalSomeChildren(int signal, int targets);
+ static bool SignalUnconnectedWorkers(int signal);
#define SignalChildren(sig) SignalSomeChildren(sig, BACKEND_TYPE_ALL)
static int CountChildren(int target);
+ static int CountUnconnectedWorkers(void);
+ static void StartOneBackgroundWorker(void);
static bool CreateOptsFile(int argc, char *argv[], char *fullprogname);
static pid_t StartChildProcess(AuxProcType type);
static void StartAutovacuumWorker(void);
***************
*** 473,478 **** static bool save_backend_variables(BackendParameters *param, Port *port,
--- 517,524 ----
static void ShmemBackendArrayAdd(Backend *bn);
static void ShmemBackendArrayRemove(Backend *bn);
+
+ static BackgroundWorker *find_bgworker_entry(int cookie);
#endif /* EXEC_BACKEND */
#define StartupDataBase() StartChildProcess(StartupProcess)
***************
*** 844,849 **** PostmasterMain(int argc, char *argv[])
--- 890,906 ----
process_shared_preload_libraries();
/*
+ * If loadable modules have added background workers, MaxBackends needs to
+ * be updated. Do so now by forcing a no-op update of max_connections.
+ * XXX This is a pretty ugly way to do it, but it doesn't seem worth
+ * introducing a new entry point in guc.c to do it in a cleaner fashion.
+ */
+ if (GetNumShmemAttachedBgworkers() > 0)
+ SetConfigOption("max_connections",
+ GetConfigOption("max_connections", false, false),
+ PGC_POSTMASTER, PGC_S_OVERRIDE);
+
+ /*
* Establish input sockets.
*/
for (i = 0; i < MAXLISTEN; i++)
***************
*** 1087,1093 **** PostmasterMain(int argc, char *argv[])
* handling setup of child processes. See tcop/postgres.c,
* bootstrap/bootstrap.c, postmaster/bgwriter.c, postmaster/walwriter.c,
* postmaster/autovacuum.c, postmaster/pgarch.c, postmaster/pgstat.c,
! * postmaster/syslogger.c and postmaster/checkpointer.c.
*/
pqinitmask();
PG_SETMASK(&BlockSig);
--- 1144,1151 ----
* handling setup of child processes. See tcop/postgres.c,
* bootstrap/bootstrap.c, postmaster/bgwriter.c, postmaster/walwriter.c,
* postmaster/autovacuum.c, postmaster/pgarch.c, postmaster/pgstat.c,
! * postmaster/syslogger.c, postmaster/bgworker.c and
! * postmaster/checkpointer.c.
*/
pqinitmask();
PG_SETMASK(&BlockSig);
***************
*** 1177,1182 **** PostmasterMain(int argc, char *argv[])
--- 1235,1243 ----
Assert(StartupPID != 0);
pmState = PM_STARTUP;
+ /* Some workers may be scheduled to start now */
+ StartOneBackgroundWorker();
+
status = ServerLoop();
/*
***************
*** 1342,1347 **** checkDataDir(void)
--- 1403,1492 ----
}
/*
+ * Determine how long should we let ServerLoop sleep.
+ *
+ * In normal conditions we wait at most one minute, to ensure that the other
+ * background tasks handled by ServerLoop get done even when no requests are
+ * arriving. However, if there are background workers waiting to be started,
+ * we don't actually sleep so that they are quickly serviced.
+ */
+ static void
+ DetermineSleepTime(struct timeval *timeout)
+ {
+ TimestampTz next_wakeup = 0;
+
+ /*
+ * Normal case: either there are no background workers at all, or we're in
+ * a shutdown sequence (during which we ignore bgworkers altogether).
+ */
+ if (Shutdown > NoShutdown ||
+ (!StartWorkerNeeded && !HaveCrashedWorker))
+ {
+ timeout->tv_sec = 60;
+ timeout->tv_usec = 0;
+ return;
+ }
+
+ if (StartWorkerNeeded)
+ {
+ timeout->tv_sec = 0;
+ timeout->tv_usec = 0;
+ return;
+ }
+
+ if (HaveCrashedWorker)
+ {
+ slist_iter siter;
+
+ /*
+ * When there are crashed bgworkers, we sleep just long enough that
+ * they are restarted when they request to be. Scan the list to
+ * determine the minimum of all wakeup times according to most recent
+ * crash time and requested restart interval.
+ */
+ slist_foreach(siter, &BackgroundWorkerList)
+ {
+ RegisteredBgWorker *rw;
+ TimestampTz this_wakeup;
+
+ rw = slist_container(RegisteredBgWorker, rw_lnode, siter.cur);
+
+ if (rw->rw_crashed_at == 0)
+ continue;
+
+ if (rw->rw_worker.bgw_restart_time == BGW_NEVER_RESTART)
+ continue;
+
+ this_wakeup = TimestampTzPlusMilliseconds(rw->rw_crashed_at,
+ 1000L * rw->rw_worker.bgw_restart_time);
+ if (next_wakeup == 0 || this_wakeup < next_wakeup)
+ next_wakeup = this_wakeup;
+ }
+ }
+
+ if (next_wakeup != 0)
+ {
+ int microsecs;
+
+ TimestampDifference(GetCurrentTimestamp(), next_wakeup,
+ &timeout->tv_sec, µsecs);
+ timeout->tv_usec = microsecs;
+
+ /* Ensure we don't exceed one minute */
+ if (timeout->tv_sec > 60)
+ {
+ timeout->tv_sec = 60;
+ timeout->tv_usec = 0;
+ }
+ }
+ else
+ {
+ timeout->tv_sec = 60;
+ timeout->tv_usec = 0;
+ }
+ }
+
+ /*
* Main idle loop of postmaster
*/
static int
***************
*** 1364,1372 **** ServerLoop(void)
/*
* Wait for a connection request to arrive.
*
- * We wait at most one minute, to ensure that the other background
- * tasks handled below get done even when no requests are arriving.
- *
* If we are in PM_WAIT_DEAD_END state, then we don't want to accept
* any new connections, so we don't call select() at all; just sleep
* for a little bit with signals unblocked.
--- 1509,1514 ----
***************
*** 1385,1392 **** ServerLoop(void)
/* must set timeout each time; some OSes change it! */
struct timeval timeout;
! timeout.tv_sec = 60;
! timeout.tv_usec = 0;
selres = select(nSockets, &rmask, NULL, NULL, &timeout);
}
--- 1527,1533 ----
/* must set timeout each time; some OSes change it! */
struct timeval timeout;
! DetermineSleepTime(&timeout);
selres = select(nSockets, &rmask, NULL, NULL, &timeout);
}
***************
*** 1498,1503 **** ServerLoop(void)
--- 1639,1648 ----
kill(AutoVacPID, SIGUSR2);
}
+ /* Get other worker processes running, if needed */
+ if (StartWorkerNeeded || HaveCrashedWorker)
+ StartOneBackgroundWorker();
+
/*
* Touch Unix socket and lock files every 58 minutes, to ensure that
* they are not removed by overzealous /tmp-cleaning tasks. We assume
***************
*** 1513,1519 **** ServerLoop(void)
}
}
-
/*
* Initialise the masks for select() for the ports we are listening on.
* Return the number of sockets to listen on.
--- 1658,1663 ----
***************
*** 1867,1873 **** processCancelRequest(Port *port, void *pkt)
Backend *bp;
#ifndef EXEC_BACKEND
! dlist_iter iter;
#else
int i;
#endif
--- 2011,2017 ----
Backend *bp;
#ifndef EXEC_BACKEND
! dlist_iter iter;
#else
int i;
#endif
***************
*** 2205,2212 **** pmdie(SIGNAL_ARGS)
if (pmState == PM_RUN || pmState == PM_RECOVERY ||
pmState == PM_HOT_STANDBY || pmState == PM_STARTUP)
{
! /* autovacuum workers are told to shut down immediately */
! SignalSomeChildren(SIGTERM, BACKEND_TYPE_AUTOVAC);
/* and the autovac launcher too */
if (AutoVacPID != 0)
signal_child(AutoVacPID, SIGTERM);
--- 2349,2359 ----
if (pmState == PM_RUN || pmState == PM_RECOVERY ||
pmState == PM_HOT_STANDBY || pmState == PM_STARTUP)
{
! /* autovac workers are told to shut down immediately */
! /* and bgworkers too; does this need tweaking? */
! SignalSomeChildren(SIGTERM,
! BACKEND_TYPE_AUTOVAC | BACKEND_TYPE_BGWORKER);
! SignalUnconnectedWorkers(SIGTERM);
/* and the autovac launcher too */
if (AutoVacPID != 0)
signal_child(AutoVacPID, SIGTERM);
***************
*** 2258,2269 **** pmdie(SIGNAL_ARGS)
signal_child(BgWriterPID, SIGTERM);
if (WalReceiverPID != 0)
signal_child(WalReceiverPID, SIGTERM);
if (pmState == PM_RECOVERY)
{
/*
! * Only startup, bgwriter, walreceiver, and/or checkpointer
! * should be active in this state; we just signaled the first
! * three, and we don't want to kill checkpointer yet.
*/
pmState = PM_WAIT_BACKENDS;
}
--- 2405,2418 ----
signal_child(BgWriterPID, SIGTERM);
if (WalReceiverPID != 0)
signal_child(WalReceiverPID, SIGTERM);
+ SignalUnconnectedWorkers(SIGTERM);
if (pmState == PM_RECOVERY)
{
/*
! * Only startup, bgwriter, walreceiver, unconnected bgworkers,
! * and/or checkpointer should be active in this state; we just
! * signaled the first four, and we don't want to kill
! * checkpointer yet.
*/
pmState = PM_WAIT_BACKENDS;
}
***************
*** 2275,2283 **** pmdie(SIGNAL_ARGS)
{
ereport(LOG,
(errmsg("aborting any active transactions")));
! /* shut down all backends and autovac workers */
SignalSomeChildren(SIGTERM,
! BACKEND_TYPE_NORMAL | BACKEND_TYPE_AUTOVAC);
/* and the autovac launcher too */
if (AutoVacPID != 0)
signal_child(AutoVacPID, SIGTERM);
--- 2424,2433 ----
{
ereport(LOG,
(errmsg("aborting any active transactions")));
! /* shut down all backends and workers */
SignalSomeChildren(SIGTERM,
! BACKEND_TYPE_NORMAL | BACKEND_TYPE_AUTOVAC |
! BACKEND_TYPE_BGWORKER);
/* and the autovac launcher too */
if (AutoVacPID != 0)
signal_child(AutoVacPID, SIGTERM);
***************
*** 2321,2326 **** pmdie(SIGNAL_ARGS)
--- 2471,2477 ----
signal_child(PgArchPID, SIGQUIT);
if (PgStatPID != 0)
signal_child(PgStatPID, SIGQUIT);
+ SignalUnconnectedWorkers(SIGQUIT);
ExitPostmaster(0);
break;
}
***************
*** 2449,2454 **** reaper(SIGNAL_ARGS)
--- 2600,2608 ----
if (PgStatPID == 0)
PgStatPID = pgstat_start();
+ /* some workers may be scheduled to start now */
+ StartOneBackgroundWorker();
+
/* at this point we are really open for business */
ereport(LOG,
(errmsg("database system is ready to accept connections")));
***************
*** 2615,2620 **** reaper(SIGNAL_ARGS)
--- 2769,2782 ----
continue;
}
+ /* Was it one of our background workers? */
+ if (CleanupBackgroundWorker(pid, exitstatus))
+ {
+ /* have it be restarted */
+ HaveCrashedWorker = true;
+ continue;
+ }
+
/*
* Else do standard backend child cleanup.
*/
***************
*** 2633,2643 **** reaper(SIGNAL_ARGS)
--- 2795,2894 ----
errno = save_errno;
}
+ /*
+ * Scan the bgworkers list and see if the given PID (which has just stopped
+ * or crashed) is in it. Handle its shutdown if so, and return true. If not a
+ * bgworker, return false.
+ *
+ * This is heavily based on CleanupBackend. One important difference is that
+ * we don't know yet that the dying process is a bgworker, so we must be silent
+ * until we're sure it is.
+ */
+ static bool
+ CleanupBackgroundWorker(int pid,
+ int exitstatus) /* child's exit status */
+ {
+ char namebuf[MAXPGPATH];
+ slist_iter iter;
+
+ slist_foreach(iter, &BackgroundWorkerList)
+ {
+ RegisteredBgWorker *rw;
+
+ rw = slist_container(RegisteredBgWorker, rw_lnode, iter.cur);
+
+ if (rw->rw_pid != pid)
+ continue;
+
+ #ifdef WIN32
+ /* see CleanupBackend */
+ if (exitstatus == ERROR_WAIT_NO_CHILDREN)
+ exitstatus = 0;
+ #endif
+
+ snprintf(namebuf, MAXPGPATH, "%s: %s", _("worker process"),
+ rw->rw_worker.bgw_name);
+
+ /* Delay restarting any bgworker that exits with a nonzero status. */
+ if (!EXIT_STATUS_0(exitstatus))
+ rw->rw_crashed_at = GetCurrentTimestamp();
+ else
+ rw->rw_crashed_at = 0;
+
+ /*
+ * Additionally, for shared-memory-connected workers, just like a
+ * backend, any exit status other than 0 or 1 is considered a crash
+ * and causes a system-wide restart.
+ */
+ if (rw->rw_worker.bgw_flags & BGWORKER_SHMEM_ACCESS)
+ {
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ {
+ rw->rw_crashed_at = GetCurrentTimestamp();
+ HandleChildCrash(pid, exitstatus, namebuf);
+ return true;
+ }
+ }
+
+ if (!ReleasePostmasterChildSlot(rw->rw_child_slot))
+ {
+ /*
+ * Uh-oh, the child failed to clean itself up. Treat as a crash
+ * after all.
+ */
+ rw->rw_crashed_at = GetCurrentTimestamp();
+ HandleChildCrash(pid, exitstatus, namebuf);
+ return true;
+ }
+
+ /* Get it out of the BackendList and clear out remaining data */
+ if (rw->rw_backend)
+ {
+ Assert(rw->rw_worker.bgw_flags & BGWORKER_BACKEND_DATABASE_CONNECTION);
+ dlist_delete(&rw->rw_backend->elem);
+ #ifdef EXEC_BACKEND
+ ShmemBackendArrayRemove(rw->rw_backend);
+ #endif
+ free(rw->rw_backend);
+ rw->rw_backend = NULL;
+ }
+ rw->rw_pid = 0;
+ rw->rw_child_slot = 0;
+
+ LogChildExit(LOG, namebuf, pid, exitstatus);
+
+ return true;
+ }
+
+ return false;
+ }
/*
* CleanupBackend -- cleanup after terminated backend.
*
* Remove all local state associated with backend.
+ *
+ * If you change this, see also CleanupBackgroundWorker.
*/
static void
CleanupBackend(int pid,
***************
*** 2705,2711 **** CleanupBackend(int pid,
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
! * walwriter or autovacuum.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
--- 2956,2962 ----
/*
* HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
! * walwriter, autovacuum, or background worker.
*
* The objectives here are to clean up our local state about the child
* process, and to signal all other remaining children to quickdie.
***************
*** 2714,2719 **** static void
--- 2965,2971 ----
HandleChildCrash(int pid, int exitstatus, const char *procname)
{
dlist_mutable_iter iter;
+ slist_iter siter;
Backend *bp;
/*
***************
*** 2727,2732 **** HandleChildCrash(int pid, int exitstatus, const char *procname)
--- 2979,3034 ----
(errmsg("terminating any other active server processes")));
}
+ /* Process unconnected background workers */
+ slist_foreach(siter, &BackgroundWorkerList)
+ {
+ RegisteredBgWorker *rw;
+
+ rw = slist_container(RegisteredBgWorker, rw_lnode, siter.cur);
+ if (rw->rw_pid == 0)
+ continue; /* not running */
+ if (rw->rw_pid == pid)
+ {
+ /*
+ * Found entry for freshly-dead worker, so remove it.
+ */
+ (void) ReleasePostmasterChildSlot(rw->rw_child_slot);
+ if (rw->rw_backend)
+ {
+ dlist_delete(&rw->rw_backend->elem);
+ #ifdef EXEC_BACKEND
+ ShmemBackendArrayRemove(rw->rw_backend);
+ #endif
+ free(rw->rw_backend);
+ rw->rw_backend = NULL;
+ }
+ rw->rw_pid = 0;
+ rw->rw_child_slot = 0;
+ /* don't reset crashed_at */
+ /* Keep looping so we can signal remaining workers */
+ }
+ else
+ {
+ /*
+ * This worker is still alive. Unless we did so already, tell it
+ * to commit hara-kiri.
+ *
+ * SIGQUIT is the special signal that says exit without proc_exit
+ * and let the user know what's going on. But if SendStop is set
+ * (-s on command line), then we send SIGSTOP instead, so that we
+ * can get core dumps from all backends by hand.
+ */
+ if (!FatalError)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) rw->rw_pid)));
+ signal_child(rw->rw_pid, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+ }
+ }
+
/* Process regular backends */
dlist_foreach_modify(iter, &BackendList)
{
***************
*** 2761,2767 **** HandleChildCrash(int pid, int exitstatus, const char *procname)
--- 3063,3075 ----
*
* We could exclude dead_end children here, but at least in the
* SIGSTOP case it seems better to include them.
+ *
+ * Background workers were already processed above; ignore them
+ * here.
*/
+ if (bp->bkend_type == BACKEND_TYPE_BGWORKER)
+ continue;
+
if (!FatalError)
{
ereport(DEBUG2,
***************
*** 3005,3020 **** PostmasterStateMachine(void)
{
/*
* PM_WAIT_BACKENDS state ends when we have no regular backends
! * (including autovac workers) and no walwriter, autovac launcher or
! * bgwriter. If we are doing crash recovery then we expect the
! * checkpointer to exit as well, otherwise not. The archiver, stats,
! * and syslogger processes are disregarded since they are not
! * connected to shared memory; we also disregard dead_end children
! * here. Walsenders are also disregarded, they will be terminated
! * later after writing the checkpoint record, like the archiver
! * process.
*/
! if (CountChildren(BACKEND_TYPE_NORMAL | BACKEND_TYPE_AUTOVAC) == 0 &&
StartupPID == 0 &&
WalReceiverPID == 0 &&
BgWriterPID == 0 &&
--- 3313,3329 ----
{
/*
* PM_WAIT_BACKENDS state ends when we have no regular backends
! * (including autovac workers), no bgworkers (including unconnected
! * ones), and no walwriter, autovac launcher or bgwriter. If we are
! * doing crash recovery then we expect the checkpointer to exit as
! * well, otherwise not. The archiver, stats, and syslogger processes
! * are disregarded since they are not connected to shared memory; we
! * also disregard dead_end children here. Walsenders are also
! * disregarded, they will be terminated later after writing the
! * checkpoint record, like the archiver process.
*/
! if (CountChildren(BACKEND_TYPE_NORMAL | BACKEND_TYPE_WORKER) == 0 &&
! CountUnconnectedWorkers() == 0 &&
StartupPID == 0 &&
WalReceiverPID == 0 &&
BgWriterPID == 0 &&
***************
*** 3227,3239 **** signal_child(pid_t pid, int signal)
}
/*
* Send a signal to the targeted children (but NOT special children;
* dead_end children are never signaled, either).
*/
static bool
SignalSomeChildren(int signal, int target)
{
! dlist_iter iter;
bool signaled = false;
dlist_foreach(iter, &BackendList)
--- 3536,3582 ----
}
/*
+ * Send a signal to bgworkers that did not request backend connections
+ *
+ * The reason this is interesting is that workers that did request connections
+ * are considered by SignalChildren; this function complements that one.
+ */
+ static bool
+ SignalUnconnectedWorkers(int signal)
+ {
+ bool signaled = false;
+ slist_iter iter;
+
+ slist_foreach(iter, &BackgroundWorkerList)
+ {
+ RegisteredBgWorker *rw;
+
+ rw = slist_container(RegisteredBgWorker, rw_lnode, iter.cur);
+
+ if (rw->rw_pid == 0)
+ continue;
+ /* ignore connected workers */
+ if (rw->rw_backend != NULL)
+ continue;
+
+ ereport(DEBUG4,
+ (errmsg_internal("sending signal %d to process %d",
+ signal, (int) rw->rw_pid)));
+ signal_child(rw->rw_pid, signal);
+ signaled = true;
+ }
+
+ return signaled;
+ }
+
+ /*
* Send a signal to the targeted children (but NOT special children;
* dead_end children are never signaled, either).
*/
static bool
SignalSomeChildren(int signal, int target)
{
! dlist_iter iter;
bool signaled = false;
dlist_foreach(iter, &BackendList)
***************
*** 3249,3263 **** SignalSomeChildren(int signal, int target)
*/
if (target != BACKEND_TYPE_ALL)
{
! int child;
! if (bp->is_autovacuum)
! child = BACKEND_TYPE_AUTOVAC;
! else if (IsPostmasterChildWalSender(bp->child_slot))
! child = BACKEND_TYPE_WALSND;
! else
! child = BACKEND_TYPE_NORMAL;
! if (!(target & child))
continue;
}
--- 3592,3606 ----
*/
if (target != BACKEND_TYPE_ALL)
{
! /*
! * Assign bkend_type for any recently announced WAL Sender
! * processes.
! */
! if (bp->bkend_type == BACKEND_TYPE_NORMAL &&
! IsPostmasterChildWalSender(bp->child_slot))
! bp->bkend_type = BACKEND_TYPE_WALSND;
! if (!(target & bp->bkend_type))
continue;
}
***************
*** 3375,3381 **** BackendStartup(Port *port)
* of backends.
*/
bn->pid = pid;
! bn->is_autovacuum = false;
dlist_push_head(&BackendList, &bn->elem);
#ifdef EXEC_BACKEND
--- 3718,3724 ----
* of backends.
*/
bn->pid = pid;
! bn->bkend_type = BACKEND_TYPE_NORMAL; /* Can change later to WALSND */
dlist_push_head(&BackendList, &bn->elem);
#ifdef EXEC_BACKEND
***************
*** 3744,3750 **** internal_forkexec(int argc, char *argv[], Port *port)
fp = AllocateFile(tmpfilename, PG_BINARY_W);
if (!fp)
{
! /* As in OpenTemporaryFile, try to make the temp-file directory */
mkdir(PG_TEMP_FILES_DIR, S_IRWXU);
fp = AllocateFile(tmpfilename, PG_BINARY_W);
--- 4087,4096 ----
fp = AllocateFile(tmpfilename, PG_BINARY_W);
if (!fp)
{
! /*
! * As in OpenTemporaryFileInTablespace, try to make the temp-file
! * directory
! */
mkdir(PG_TEMP_FILES_DIR, S_IRWXU);
fp = AllocateFile(tmpfilename, PG_BINARY_W);
***************
*** 4078,4084 **** SubPostmasterMain(int argc, char *argv[])
if (strcmp(argv[1], "--forkbackend") == 0 ||
strcmp(argv[1], "--forkavlauncher") == 0 ||
strcmp(argv[1], "--forkavworker") == 0 ||
! strcmp(argv[1], "--forkboot") == 0)
PGSharedMemoryReAttach();
/* autovacuum needs this set before calling InitProcess */
--- 4424,4431 ----
if (strcmp(argv[1], "--forkbackend") == 0 ||
strcmp(argv[1], "--forkavlauncher") == 0 ||
strcmp(argv[1], "--forkavworker") == 0 ||
! strcmp(argv[1], "--forkboot") == 0 ||
! strncmp(argv[1], "--forkbgworker=", 15) == 0)
PGSharedMemoryReAttach();
/* autovacuum needs this set before calling InitProcess */
***************
*** 4213,4218 **** SubPostmasterMain(int argc, char *argv[])
--- 4560,4585 ----
AutoVacWorkerMain(argc - 2, argv + 2); /* does not return */
}
+ if (strncmp(argv[1], "--forkbgworker=", 15) == 0)
+ {
+ int cookie;
+
+ /* Close the postmaster's sockets */
+ ClosePostmasterPorts(false);
+
+ /* Restore basic shared memory pointers */
+ InitShmemAccess(UsedShmemSegAddr);
+
+ /* Need a PGPROC to run CreateSharedMemoryAndSemaphores */
+ InitProcess();
+
+ /* Attach process to shared data structures */
+ CreateSharedMemoryAndSemaphores(false, 0);
+
+ cookie = atoi(argv[1] + 15);
+ MyBgworkerEntry = find_bgworker_entry(cookie);
+ do_start_bgworker();
+ }
if (strcmp(argv[1], "--forkarch") == 0)
{
/* Close the postmaster's sockets */
***************
*** 4312,4317 **** sigusr1_handler(SIGNAL_ARGS)
--- 4679,4687 ----
(errmsg("database system is ready to accept read only connections")));
pmState = PM_HOT_STANDBY;
+
+ /* Some workers may be scheduled to start now */
+ StartOneBackgroundWorker();
}
if (CheckPostmasterSignal(PMSIGNAL_WAKEN_ARCHIVER) &&
***************
*** 4482,4494 **** PostmasterRandom(void)
}
/*
* Count up number of child processes of specified types (dead_end chidren
* are always excluded).
*/
static int
CountChildren(int target)
{
! dlist_iter iter;
int cnt = 0;
dlist_foreach(iter, &BackendList)
--- 4852,4891 ----
}
/*
+ * Count up number of worker processes that did not request backend connections
+ * See SignalUnconnectedWorkers for why this is interesting.
+ */
+ static int
+ CountUnconnectedWorkers(void)
+ {
+ slist_iter iter;
+ int cnt = 0;
+
+ slist_foreach(iter, &BackgroundWorkerList)
+ {
+ RegisteredBgWorker *rw;
+
+ rw = slist_container(RegisteredBgWorker, rw_lnode, iter.cur);
+
+ if (rw->rw_pid == 0)
+ continue;
+ /* ignore connected workers */
+ if (rw->rw_backend != NULL)
+ continue;
+
+ cnt++;
+ }
+ return cnt;
+ }
+
+ /*
* Count up number of child processes of specified types (dead_end chidren
* are always excluded).
*/
static int
CountChildren(int target)
{
! dlist_iter iter;
int cnt = 0;
dlist_foreach(iter, &BackendList)
***************
*** 4504,4518 **** CountChildren(int target)
*/
if (target != BACKEND_TYPE_ALL)
{
! int child;
! if (bp->is_autovacuum)
! child = BACKEND_TYPE_AUTOVAC;
! else if (IsPostmasterChildWalSender(bp->child_slot))
! child = BACKEND_TYPE_WALSND;
! else
! child = BACKEND_TYPE_NORMAL;
! if (!(target & child))
continue;
}
--- 4901,4915 ----
*/
if (target != BACKEND_TYPE_ALL)
{
! /*
! * Assign bkend_type for any recently announced WAL Sender
! * processes.
! */
! if (bp->bkend_type == BACKEND_TYPE_NORMAL &&
! IsPostmasterChildWalSender(bp->child_slot))
! bp->bkend_type = BACKEND_TYPE_WALSND;
! if (!(target & bp->bkend_type))
continue;
}
***************
*** 4671,4677 **** StartAutovacuumWorker(void)
bn->pid = StartAutoVacWorker();
if (bn->pid > 0)
{
! bn->is_autovacuum = true;
dlist_push_head(&BackendList, &bn->elem);
#ifdef EXEC_BACKEND
ShmemBackendArrayAdd(bn);
--- 5068,5074 ----
bn->pid = StartAutoVacWorker();
if (bn->pid > 0)
{
! bn->bkend_type = BACKEND_TYPE_AUTOVAC;
dlist_push_head(&BackendList, &bn->elem);
#ifdef EXEC_BACKEND
ShmemBackendArrayAdd(bn);
***************
*** 4746,4763 **** CreateOptsFile(int argc, char *argv[], char *fullprogname)
*
* This reports the number of entries needed in per-child-process arrays
* (the PMChildFlags array, and if EXEC_BACKEND the ShmemBackendArray).
! * These arrays include regular backends, autovac workers and walsenders,
! * but not special children nor dead_end children. This allows the arrays
! * to have a fixed maximum size, to wit the same too-many-children limit
! * enforced by canAcceptConnections(). The exact value isn't too critical
! * as long as it's more than MaxBackends.
*/
int
MaxLivePostmasterChildren(void)
{
! return 2 * MaxBackends;
}
#ifdef EXEC_BACKEND
--- 5143,5784 ----
*
* This reports the number of entries needed in per-child-process arrays
* (the PMChildFlags array, and if EXEC_BACKEND the ShmemBackendArray).
! * These arrays include regular backends, autovac workers, walsenders
! * and background workers, but not special children nor dead_end children.
! * This allows the arrays to have a fixed maximum size, to wit the same
! * too-many-children limit enforced by canAcceptConnections(). The exact value
! * isn't too critical as long as it's more than MaxBackends.
*/
int
MaxLivePostmasterChildren(void)
{
! return 2 * (MaxConnections + autovacuum_max_workers + 1 +
! GetNumRegisteredBackgroundWorkers(0));
}
+ /*
+ * Register a new background worker.
+ *
+ * This can only be called in the _PG_init function of a module library
+ * that's loaded by shared_preload_libraries; otherwise it has no effect.
+ */
+ void
+ RegisterBackgroundWorker(BackgroundWorker *worker)
+ {
+ RegisteredBgWorker *rw;
+ int namelen = strlen(worker->bgw_name);
+
+ #ifdef EXEC_BACKEND
+
+ /*
+ * Use 1 here, not 0, to avoid confusing a possible bogus cookie read by
+ * atoi() in SubPostmasterMain.
+ */
+ static int BackgroundWorkerCookie = 1;
+ #endif
+
+ if (!IsUnderPostmaster)
+ ereport(LOG,
+ (errmsg("registering background worker: %s", worker->bgw_name)));
+
+ if (!process_shared_preload_libraries_in_progress)
+ {
+ if (!IsUnderPostmaster)
+ ereport(LOG,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("background worker \"%s\": must be registered in shared_preload_libraries",
+ worker->bgw_name)));
+ return;
+ }
+
+ /* sanity check for flags */
+ if (worker->bgw_flags & BGWORKER_BACKEND_DATABASE_CONNECTION)
+ {
+ if (!(worker->bgw_flags & BGWORKER_SHMEM_ACCESS))
+ {
+ if (!IsUnderPostmaster)
+ ereport(LOG,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("background worker \"%s\": must attach to shared memory in order to request a database connection",
+ worker->bgw_name)));
+ return;
+ }
+
+ if (worker->bgw_start_time == BgWorkerStart_PostmasterStart)
+ {
+ if (!IsUnderPostmaster)
+ ereport(LOG,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("background worker \"%s\": cannot request database access if starting at postmaster start",
+ worker->bgw_name)));
+ return;
+ }
+
+ /* XXX other checks? */
+ }
+
+ if ((worker->bgw_restart_time < 0 &&
+ worker->bgw_restart_time != BGW_NEVER_RESTART) ||
+ (worker->bgw_restart_time > USECS_PER_DAY / 1000))
+ {
+ if (!IsUnderPostmaster)
+ ereport(LOG,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("background worker \"%s\": invalid restart interval",
+ worker->bgw_name)));
+ return;
+ }
+
+ /*
+ * Copy the registration data into the registered workers list.
+ */
+ rw = malloc(sizeof(RegisteredBgWorker) + namelen + 1);
+ if (rw == NULL)
+ {
+ ereport(LOG,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+ errmsg("out of memory")));
+ return;
+ }
+
+ rw->rw_worker = *worker;
+ rw->rw_worker.bgw_name = ((char *) rw) + sizeof(RegisteredBgWorker);
+ strlcpy(rw->rw_worker.bgw_name, worker->bgw_name, namelen + 1);
+
+ rw->rw_backend = NULL;
+ rw->rw_pid = 0;
+ rw->rw_child_slot = 0;
+ rw->rw_crashed_at = 0;
+ #ifdef EXEC_BACKEND
+ rw->rw_cookie = BackgroundWorkerCookie++;
+ #endif
+
+ slist_push_head(&BackgroundWorkerList, &rw->rw_lnode);
+ }
+
+ /*
+ * Connect background worker to a database.
+ */
+ void
+ BackgroundWorkerInitializeConnection(char *dbname, char *username)
+ {
+ BackgroundWorker *worker = MyBgworkerEntry;
+
+ /* XXX is this the right errcode? */
+ if (!(worker->bgw_flags & BGWORKER_BACKEND_DATABASE_CONNECTION))
+ ereport(FATAL,
+ (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("database connection requirement not indicated during registration")));
+
+ InitPostgres(dbname, InvalidOid, username, NULL);
+
+ /* it had better not gotten out of "init" mode yet */
+ if (!IsInitProcessingMode())
+ ereport(ERROR,
+ (errmsg("invalid processing mode in bgworker")));
+ SetProcessingMode(NormalProcessing);
+ }
+
+ /*
+ * Block/unblock signals in a background worker
+ */
+ void
+ BackgroundWorkerBlockSignals(void)
+ {
+ PG_SETMASK(&BlockSig);
+ }
+
+ void
+ BackgroundWorkerUnblockSignals(void)
+ {
+ PG_SETMASK(&UnBlockSig);
+ }
+
+ #ifdef EXEC_BACKEND
+ static BackgroundWorker *
+ find_bgworker_entry(int cookie)
+ {
+ slist_iter iter;
+
+ slist_foreach(iter, &BackgroundWorkerList)
+ {
+ RegisteredBgWorker *rw;
+
+ rw = slist_container(RegisteredBgWorker, rw_lnode, iter.cur);
+ if (rw->rw_cookie == cookie)
+ return &rw->rw_worker;
+ }
+
+ return NULL;
+ }
+ #endif
+
+ static void
+ bgworker_quickdie(SIGNAL_ARGS)
+ {
+ sigaddset(&BlockSig, SIGQUIT); /* prevent nested calls */
+ PG_SETMASK(&BlockSig);
+
+ /*
+ * We DO NOT want to run proc_exit() callbacks -- we're here because
+ * shared memory may be corrupted, so we don't want to try to clean up our
+ * transaction. Just nail the windows shut and get out of town. Now that
+ * there's an atexit callback to prevent third-party code from breaking
+ * things by calling exit() directly, we have to reset the callbacks
+ * explicitly to make this work as intended.
+ */
+ on_exit_reset();
+
+ /*
+ * Note we do exit(0) here, not exit(2) like quickdie. The reason is that
+ * we don't want to be seen this worker as independently crashed, because
+ * then postmaster would delay restarting it again afterwards. If some
+ * idiot DBA manually sends SIGQUIT to a random bgworker, the "dead man
+ * switch" will ensure that postmaster sees this as a crash.
+ */
+ exit(0);
+ }
+
+ /*
+ * Standard SIGTERM handler for background workers
+ */
+ static void
+ bgworker_die(SIGNAL_ARGS)
+ {
+ PG_SETMASK(&BlockSig);
+
+ ereport(FATAL,
+ (errcode(ERRCODE_ADMIN_SHUTDOWN),
+ errmsg("terminating background worker \"%s\" due to administrator command",
+ MyBgworkerEntry->bgw_name)));
+ }
+
+ static void
+ do_start_bgworker(void)
+ {
+ sigjmp_buf local_sigjmp_buf;
+ char buf[MAXPGPATH];
+ BackgroundWorker *worker = MyBgworkerEntry;
+
+ if (worker == NULL)
+ elog(FATAL, "unable to find bgworker entry");
+
+ /* we are a postmaster subprocess now */
+ IsUnderPostmaster = true;
+ IsBackgroundWorker = true;
+
+ /* reset MyProcPid */
+ MyProcPid = getpid();
+
+ /* record Start Time for logging */
+ MyStartTime = time(NULL);
+
+ /* Identify myself via ps */
+ snprintf(buf, MAXPGPATH, "bgworker: %s", worker->bgw_name);
+ init_ps_display(buf, "", "", "");
+
+ SetProcessingMode(InitProcessing);
+
+ /* Apply PostAuthDelay */
+ if (PostAuthDelay > 0)
+ pg_usleep(PostAuthDelay * 1000000L);
+
+ /*
+ * If possible, make this process a group leader, so that the postmaster
+ * can signal any child processes too.
+ */
+ #ifdef HAVE_SETSID
+ if (setsid() < 0)
+ elog(FATAL, "setsid() failed: %m");
+ #endif
+
+ /*
+ * Set up signal handlers.
+ */
+ if (worker->bgw_flags & BGWORKER_BACKEND_DATABASE_CONNECTION)
+ {
+ /*
+ * SIGINT is used to signal canceling the current action
+ */
+ pqsignal(SIGINT, StatementCancelHandler);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
+ pqsignal(SIGFPE, FloatExceptionHandler);
+
+ /* XXX Any other handlers needed here? */
+ }
+ else
+ {
+ pqsignal(SIGINT, SIG_IGN);
+ pqsignal(SIGUSR1, SIG_IGN);
+ pqsignal(SIGFPE, SIG_IGN);
+ }
+
+ /* SIGTERM and SIGHUP are configurable */
+ if (worker->bgw_sigterm)
+ pqsignal(SIGTERM, worker->bgw_sigterm);
+ else
+ pqsignal(SIGTERM, bgworker_die);
+
+ if (worker->bgw_sighup)
+ pqsignal(SIGHUP, worker->bgw_sighup);
+ else
+ pqsignal(SIGHUP, SIG_IGN);
+
+ pqsignal(SIGQUIT, bgworker_quickdie);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
+
+ pqsignal(SIGPIPE, SIG_IGN);
+ pqsignal(SIGUSR2, SIG_IGN);
+ pqsignal(SIGCHLD, SIG_DFL);
+
+ /*
+ * If an exception is encountered, processing resumes here.
+ *
+ * See notes in postgres.c about the design of this coding.
+ */
+ if (sigsetjmp(local_sigjmp_buf, 1) != 0)
+ {
+ /* Since not using PG_TRY, must reset error stack by hand */
+ error_context_stack = NULL;
+
+ /* Prevent interrupts while cleaning up */
+ HOLD_INTERRUPTS();
+
+ /* Report the error to the server log */
+ EmitErrorReport();
+
+ /*
+ * Do we need more cleanup here? For shmem-connected bgworkers, we
+ * will call InitProcess below, which will install ProcKill as exit
+ * callback. That will take care of releasing locks, etc.
+ */
+
+ /* and go away */
+ proc_exit(1);
+ }
+
+ /* We can now handle ereport(ERROR) */
+ PG_exception_stack = &local_sigjmp_buf;
+
+ /* Early initialization */
+ BaseInit();
+
+ /*
+ * If necessary, create a per-backend PGPROC struct in shared memory,
+ * except in the EXEC_BACKEND case where this was done in
+ * SubPostmasterMain. We must do this before we can use LWLocks (and in
+ * the EXEC_BACKEND case we already had to do some stuff with LWLocks).
+ */
+ #ifndef EXEC_BACKEND
+ if (worker->bgw_flags & BGWORKER_SHMEM_ACCESS)
+ InitProcess();
+ #endif
+
+ /*
+ * Note that in normal processes, we would call InitPostgres here. For a
+ * worker, however, we don't know what database to connect to, yet; so we
+ * need to wait until the user code does it via
+ * BackgroundWorkerInitializeConnection().
+ */
+
+ /*
+ * Now invoke the user-defined worker code
+ */
+ worker->bgw_main(worker->bgw_main_arg);
+
+ /* ... and if it returns, we're done */
+ proc_exit(0);
+ }
+
+ /*
+ * Return the number of background workers registered that have at least
+ * one of the passed flag bits set.
+ */
+ static int
+ GetNumRegisteredBackgroundWorkers(int flags)
+ {
+ slist_iter iter;
+ int count = 0;
+
+ slist_foreach(iter, &BackgroundWorkerList)
+ {
+ RegisteredBgWorker *rw;
+
+ rw = slist_container(RegisteredBgWorker, rw_lnode, iter.cur);
+
+ if (flags != 0 &&
+ !(rw->rw_worker.bgw_flags & flags))
+ continue;
+
+ count++;
+ }
+
+ return count;
+ }
+
+ /*
+ * Return the number of bgworkers that need to have PGPROC entries.
+ */
+ int
+ GetNumShmemAttachedBgworkers(void)
+ {
+ return GetNumRegisteredBackgroundWorkers(BGWORKER_SHMEM_ACCESS);
+ }
+
+ #ifdef EXEC_BACKEND
+ static pid_t
+ bgworker_forkexec(int cookie)
+ {
+ char *av[10];
+ int ac = 0;
+ char forkav[MAXPGPATH];
+
+ snprintf(forkav, MAXPGPATH, "--forkbgworker=%d", cookie);
+
+ av[ac++] = "postgres";
+ av[ac++] = forkav;
+ av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ av[ac] = NULL;
+
+ Assert(ac < lengthof(av));
+
+ return postmaster_forkexec(ac, av);
+ }
+ #endif
+
+ /*
+ * Start a new bgworker.
+ * Starting time conditions must have been checked already.
+ *
+ * This code is heavily based on autovacuum.c, q.v.
+ */
+ static void
+ start_bgworker(RegisteredBgWorker *rw)
+ {
+ pid_t worker_pid;
+
+ ereport(LOG,
+ (errmsg("starting background worker process \"%s\"",
+ rw->rw_worker.bgw_name)));
+
+ #ifdef EXEC_BACKEND
+ switch ((worker_pid = bgworker_forkexec(rw->rw_cookie)))
+ #else
+ switch ((worker_pid = fork_process()))
+ #endif
+ {
+ case -1:
+ ereport(LOG,
+ (errmsg("could not fork worker process: %m")));
+ return;
+
+ #ifndef EXEC_BACKEND
+ case 0:
+ /* in postmaster child ... */
+ /* Close the postmaster's sockets */
+ ClosePostmasterPorts(false);
+
+ /* Lose the postmaster's on-exit routines */
+ on_exit_reset();
+
+ /* Do NOT release postmaster's working memory context */
+
+ MyBgworkerEntry = &rw->rw_worker;
+ do_start_bgworker();
+ break;
+ #endif
+ default:
+ rw->rw_pid = worker_pid;
+ if (rw->rw_backend)
+ rw->rw_backend->pid = rw->rw_pid;
+ }
+ }
+
+ /*
+ * Does the current postmaster state require starting a worker with the
+ * specified start_time?
+ */
+ static bool
+ bgworker_should_start_now(BgWorkerStartTime start_time)
+ {
+ /* XXX maybe this'd be better as a table */
+ switch (pmState)
+ {
+ case PM_RUN:
+ if (start_time == BgWorkerStart_RecoveryFinished)
+ return true;
+ /* fall through */
+
+ case PM_HOT_STANDBY:
+ if (start_time == BgWorkerStart_ConsistentState)
+ return true;
+ /* fall through */
+
+ case PM_RECOVERY:
+ case PM_STARTUP:
+ case PM_INIT:
+ if (start_time == BgWorkerStart_PostmasterStart)
+ return true;
+ /* fall through */
+
+ case PM_NO_CHILDREN:
+ case PM_WAIT_DEAD_END:
+ case PM_SHUTDOWN_2:
+ case PM_SHUTDOWN:
+ case PM_WAIT_BACKENDS:
+ case PM_WAIT_READONLY:
+ case PM_WAIT_BACKUP:
+ /* fall through */
+ }
+
+ return false;
+ }
+
+ /*
+ * Allocate the Backend struct for a connected background worker, but don't
+ * add it to the list of backends just yet.
+ *
+ * Some info from the Backend is copied into the passed rw.
+ */
+ static bool
+ assign_backendlist_entry(RegisteredBgWorker *rw)
+ {
+ Backend *bn = malloc(sizeof(Backend));
+
+ if (bn == NULL)
+ {
+ ereport(LOG,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+ errmsg("out of memory")));
+
+ /*
+ * The worker didn't really crash, but setting this nonzero makes
+ * postmaster wait a bit before attempting to start it again; if it
+ * tried again right away, most likely it'd find itself under the same
+ * memory pressure.
+ */
+ rw->rw_crashed_at = GetCurrentTimestamp();
+ return false;
+ }
+
+ /*
+ * Compute the cancel key that will be assigned to this session. We
+ * probably don't need cancel keys for background workers, but we'd better
+ * have something random in the field to prevent unfriendly people from
+ * sending cancels to them.
+ */
+ MyCancelKey = PostmasterRandom();
+ bn->cancel_key = MyCancelKey;
+
+ bn->child_slot = MyPMChildSlot = AssignPostmasterChildSlot();
+ bn->bkend_type = BACKEND_TYPE_BGWORKER;
+ bn->dead_end = false;
+
+ rw->rw_backend = bn;
+ rw->rw_child_slot = bn->child_slot;
+
+ return true;
+ }
+
+ /*
+ * If the time is right, start one background worker.
+ *
+ * As a side effect, the bgworker control variables are set or reset whenever
+ * there are more workers to start after this one, and whenever the overall
+ * system state requires it.
+ */
+ static void
+ StartOneBackgroundWorker(void)
+ {
+ slist_iter iter;
+ TimestampTz now = 0;
+
+ if (FatalError)
+ {
+ StartWorkerNeeded = false;
+ HaveCrashedWorker = false;
+ return; /* not yet */
+ }
+
+ HaveCrashedWorker = false;
+
+ slist_foreach(iter, &BackgroundWorkerList)
+ {
+ RegisteredBgWorker *rw;
+
+ rw = slist_container(RegisteredBgWorker, rw_lnode, iter.cur);
+
+ /* already running? */
+ if (rw->rw_pid != 0)
+ continue;
+
+ /*
+ * If this worker has crashed previously, maybe it needs to be
+ * restarted (unless on registration it specified it doesn't want to
+ * be restarted at all). Check how long ago did a crash last happen.
+ * If the last crash is too recent, don't start it right away; let it
+ * be restarted once enough time has passed.
+ */
+ if (rw->rw_crashed_at != 0)
+ {
+ if (rw->rw_worker.bgw_restart_time == BGW_NEVER_RESTART)
+ continue;
+
+ if (now == 0)
+ now = GetCurrentTimestamp();
+
+ if (!TimestampDifferenceExceeds(rw->rw_crashed_at, now,
+ rw->rw_worker.bgw_restart_time * 1000))
+ {
+ HaveCrashedWorker = true;
+ continue;
+ }
+ }
+
+ if (bgworker_should_start_now(rw->rw_worker.bgw_start_time))
+ {
+ /* reset crash time before calling assign_backendlist_entry */
+ rw->rw_crashed_at = 0;
+
+ /*
+ * If necessary, allocate and assign the Backend element. Note we
+ * must do this before forking, so that we can handle out of
+ * memory properly.
+ *
+ * If not connected, we don't need a Backend element, but we still
+ * need a PMChildSlot.
+ */
+ if (rw->rw_worker.bgw_flags & BGWORKER_BACKEND_DATABASE_CONNECTION)
+ {
+ if (!assign_backendlist_entry(rw))
+ return;
+ }
+ else
+ rw->rw_child_slot = MyPMChildSlot = AssignPostmasterChildSlot();
+
+ start_bgworker(rw); /* sets rw->rw_pid */
+
+ if (rw->rw_backend)
+ {
+ dlist_push_head(&BackendList, &rw->rw_backend->elem);
+ #ifdef EXEC_BACKEND
+ ShmemBackendArrayAdd(rw->rw_backend);
+ #endif
+ }
+
+ /*
+ * Have ServerLoop call us again. Note that there might not
+ * actually *be* another runnable worker, but we don't care all
+ * that much; we will find out the next time we run.
+ */
+ StartWorkerNeeded = true;
+ return;
+ }
+ }
+
+ /* no runnable worker found */
+ StartWorkerNeeded = false;
+ }
#ifdef EXEC_BACKEND
*** a/src/backend/storage/lmgr/proc.c
--- b/src/backend/storage/lmgr/proc.c
***************
*** 140,146 **** ProcGlobalSemas(void)
* So, now we grab enough semaphores to support the desired max number
* of backends immediately at initialization --- if the sysadmin has set
* MaxConnections or autovacuum_max_workers higher than his kernel will
! * support, he'll find out sooner rather than later.
*
* Another reason for creating semaphores here is that the semaphore
* implementation typically requires us to create semaphores in the
--- 140,148 ----
* So, now we grab enough semaphores to support the desired max number
* of backends immediately at initialization --- if the sysadmin has set
* MaxConnections or autovacuum_max_workers higher than his kernel will
! * support, he'll find out sooner rather than later. (The number of
! * background worker processes registered by loadable modules is also taken
! * into consideration.)
*
* Another reason for creating semaphores here is that the semaphore
* implementation typically requires us to create semaphores in the
***************
*** 171,176 **** InitProcGlobal(void)
--- 173,179 ----
ProcGlobal->spins_per_delay = DEFAULT_SPINS_PER_DELAY;
ProcGlobal->freeProcs = NULL;
ProcGlobal->autovacFreeProcs = NULL;
+ ProcGlobal->bgworkerFreeProcs = NULL;
ProcGlobal->startupProc = NULL;
ProcGlobal->startupProcPid = 0;
ProcGlobal->startupBufferPinWaitBufId = -1;
***************
*** 179,188 **** InitProcGlobal(void)
/*
* Create and initialize all the PGPROC structures we'll need. There are
! * four separate consumers: (1) normal backends, (2) autovacuum workers
! * and the autovacuum launcher, (3) auxiliary processes, and (4) prepared
! * transactions. Each PGPROC structure is dedicated to exactly one of
! * these purposes, and they do not move between groups.
*/
procs = (PGPROC *) ShmemAlloc(TotalProcs * sizeof(PGPROC));
ProcGlobal->allProcs = procs;
--- 182,192 ----
/*
* Create and initialize all the PGPROC structures we'll need. There are
! * five separate consumers: (1) normal backends, (2) autovacuum workers
! * and the autovacuum launcher, (3) background workers, (4) auxiliary
! * processes, and (5) prepared transactions. Each PGPROC structure is
! * dedicated to exactly one of these purposes, and they do not move between
! * groups.
*/
procs = (PGPROC *) ShmemAlloc(TotalProcs * sizeof(PGPROC));
ProcGlobal->allProcs = procs;
***************
*** 223,234 **** InitProcGlobal(void)
procs[i].pgprocno = i;
/*
! * Newly created PGPROCs for normal backends or for autovacuum must be
! * queued up on the appropriate free list. Because there can only
! * ever be a small, fixed number of auxiliary processes, no free list
! * is used in that case; InitAuxiliaryProcess() instead uses a linear
! * search. PGPROCs for prepared transactions are added to a free list
! * by TwoPhaseShmemInit().
*/
if (i < MaxConnections)
{
--- 227,238 ----
procs[i].pgprocno = i;
/*
! * Newly created PGPROCs for normal backends, autovacuum and bgworkers
! * must be queued up on the appropriate free list. Because there can
! * only ever be a small, fixed number of auxiliary processes, no free
! * list is used in that case; InitAuxiliaryProcess() instead uses a
! * linear search. PGPROCs for prepared transactions are added to a
! * free list by TwoPhaseShmemInit().
*/
if (i < MaxConnections)
{
***************
*** 236,247 **** InitProcGlobal(void)
procs[i].links.next = (SHM_QUEUE *) ProcGlobal->freeProcs;
ProcGlobal->freeProcs = &procs[i];
}
! else if (i < MaxBackends)
{
/* PGPROC for AV launcher/worker, add to autovacFreeProcs list */
procs[i].links.next = (SHM_QUEUE *) ProcGlobal->autovacFreeProcs;
ProcGlobal->autovacFreeProcs = &procs[i];
}
/* Initialize myProcLocks[] shared memory queues. */
for (j = 0; j < NUM_LOCK_PARTITIONS; j++)
--- 240,257 ----
procs[i].links.next = (SHM_QUEUE *) ProcGlobal->freeProcs;
ProcGlobal->freeProcs = &procs[i];
}
! else if (i < MaxConnections + autovacuum_max_workers + 1)
{
/* PGPROC for AV launcher/worker, add to autovacFreeProcs list */
procs[i].links.next = (SHM_QUEUE *) ProcGlobal->autovacFreeProcs;
ProcGlobal->autovacFreeProcs = &procs[i];
}
+ else if (i < MaxBackends)
+ {
+ /* PGPROC for bgworker, add to bgworkerFreeProcs list */
+ procs[i].links.next = (SHM_QUEUE *) ProcGlobal->bgworkerFreeProcs;
+ ProcGlobal->bgworkerFreeProcs = &procs[i];
+ }
/* Initialize myProcLocks[] shared memory queues. */
for (j = 0; j < NUM_LOCK_PARTITIONS; j++)
***************
*** 299,304 **** InitProcess(void)
--- 309,316 ----
if (IsAnyAutoVacuumProcess())
MyProc = procglobal->autovacFreeProcs;
+ else if (IsBackgroundWorker)
+ MyProc = procglobal->bgworkerFreeProcs;
else
MyProc = procglobal->freeProcs;
***************
*** 306,311 **** InitProcess(void)
--- 318,325 ----
{
if (IsAnyAutoVacuumProcess())
procglobal->autovacFreeProcs = (PGPROC *) MyProc->links.next;
+ else if (IsBackgroundWorker)
+ procglobal->bgworkerFreeProcs = (PGPROC *) MyProc->links.next;
else
procglobal->freeProcs = (PGPROC *) MyProc->links.next;
SpinLockRelease(ProcStructLock);
***************
*** 782,787 **** ProcKill(int code, Datum arg)
--- 796,806 ----
MyProc->links.next = (SHM_QUEUE *) procglobal->autovacFreeProcs;
procglobal->autovacFreeProcs = MyProc;
}
+ else if (IsBackgroundWorker)
+ {
+ MyProc->links.next = (SHM_QUEUE *) procglobal->bgworkerFreeProcs;
+ procglobal->bgworkerFreeProcs = MyProc;
+ }
else
{
MyProc->links.next = (SHM_QUEUE *) procglobal->freeProcs;
*** a/src/backend/utils/init/globals.c
--- b/src/backend/utils/init/globals.c
***************
*** 87,92 **** pid_t PostmasterPid = 0;
--- 87,93 ----
bool IsPostmasterEnvironment = false;
bool IsUnderPostmaster = false;
bool IsBinaryUpgrade = false;
+ bool IsBackgroundWorker = false;
bool ExitOnAnyError = false;
*** a/src/backend/utils/init/miscinit.c
--- b/src/backend/utils/init/miscinit.c
***************
*** 498,507 **** void
InitializeSessionUserIdStandalone(void)
{
/*
! * This function should only be called in single-user mode and in
! * autovacuum workers.
*/
! AssertState(!IsUnderPostmaster || IsAutoVacuumWorkerProcess());
/* call only once */
AssertState(!OidIsValid(AuthenticatedUserId));
--- 498,507 ----
InitializeSessionUserIdStandalone(void)
{
/*
! * This function should only be called in single-user mode, in
! * autovacuum workers, and in background workers.
*/
! AssertState(!IsUnderPostmaster || IsAutoVacuumWorkerProcess() || IsBackgroundWorker);
/* call only once */
AssertState(!OidIsValid(AuthenticatedUserId));
*** a/src/backend/utils/init/postinit.c
--- b/src/backend/utils/init/postinit.c
***************
*** 627,632 **** InitPostgres(const char *in_dbname, Oid dboid, const char *username,
--- 627,645 ----
errhint("You should immediately run CREATE USER \"%s\" SUPERUSER;.",
username)));
}
+ else if (IsBackgroundWorker)
+ {
+ if (username == NULL)
+ {
+ InitializeSessionUserIdStandalone();
+ am_superuser = true;
+ }
+ else
+ {
+ InitializeSessionUserId(username);
+ am_superuser = superuser();
+ }
+ }
else
{
/* normal multiuser case */
*** a/src/backend/utils/misc/guc.c
--- b/src/backend/utils/misc/guc.c
***************
*** 52,57 ****
--- 52,58 ----
#include "parser/scansup.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
+ #include "postmaster/bgworker.h"
#include "postmaster/bgwriter.h"
#include "postmaster/postmaster.h"
#include "postmaster/syslogger.h"
***************
*** 108,114 ****
* removed, we still could not exceed INT_MAX/4 because some places compute
* 4*MaxBackends without any overflow check. This is rechecked in
* check_maxconnections, since MaxBackends is computed as MaxConnections
! * plus autovacuum_max_workers plus one (for the autovacuum launcher).
*/
#define MAX_BACKENDS 0x7fffff
--- 109,116 ----
* removed, we still could not exceed INT_MAX/4 because some places compute
* 4*MaxBackends without any overflow check. This is rechecked in
* check_maxconnections, since MaxBackends is computed as MaxConnections
! * plus the number of bgworkers plus autovacuum_max_workers plus one (for the
! * autovacuum launcher).
*/
#define MAX_BACKENDS 0x7fffff
***************
*** 8628,8634 **** show_tcp_keepalives_count(void)
static bool
check_maxconnections(int *newval, void **extra, GucSource source)
{
! if (*newval + autovacuum_max_workers + 1 > MAX_BACKENDS)
return false;
return true;
}
--- 8630,8637 ----
static bool
check_maxconnections(int *newval, void **extra, GucSource source)
{
! if (*newval + GetNumShmemAttachedBgworkers() + autovacuum_max_workers + 1 >
! MAX_BACKENDS)
return false;
return true;
}
***************
*** 8636,8648 **** check_maxconnections(int *newval, void **extra, GucSource source)
static void
assign_maxconnections(int newval, void *extra)
{
! MaxBackends = newval + autovacuum_max_workers + 1;
}
static bool
check_autovacuum_max_workers(int *newval, void **extra, GucSource source)
{
! if (MaxConnections + *newval + 1 > MAX_BACKENDS)
return false;
return true;
}
--- 8639,8653 ----
static void
assign_maxconnections(int newval, void *extra)
{
! MaxBackends = newval + autovacuum_max_workers + 1 +
! GetNumShmemAttachedBgworkers();
}
static bool
check_autovacuum_max_workers(int *newval, void **extra, GucSource source)
{
! if (MaxConnections + *newval + 1 + GetNumShmemAttachedBgworkers() >
! MAX_BACKENDS)
return false;
return true;
}
***************
*** 8650,8656 **** check_autovacuum_max_workers(int *newval, void **extra, GucSource source)
static void
assign_autovacuum_max_workers(int newval, void *extra)
{
! MaxBackends = MaxConnections + newval + 1;
}
static bool
--- 8655,8661 ----
static void
assign_autovacuum_max_workers(int newval, void *extra)
{
! MaxBackends = MaxConnections + newval + 1 + GetNumShmemAttachedBgworkers();
}
static bool
*** a/src/include/miscadmin.h
--- b/src/include/miscadmin.h
***************
*** 131,136 **** do { \
--- 131,137 ----
extern pid_t PostmasterPid;
extern bool IsPostmasterEnvironment;
extern PGDLLIMPORT bool IsUnderPostmaster;
+ extern bool IsBackgroundWorker;
extern bool IsBinaryUpgrade;
extern bool ExitOnAnyError;
*** /dev/null
--- b/src/include/postmaster/bgworker.h
***************
*** 0 ****
--- 1,104 ----
+ /*--------------------------------------------------------------------
+ * bgworker.h
+ * POSTGRES pluggable background workers interface
+ *
+ * A background worker is a process able to run arbitrary, user-supplied code,
+ * including normal transactions.
+ *
+ * Any external module loaded via shared_preload_libraries can register a
+ * worker. Then, at the appropriate time, the worker process is forked from
+ * the postmaster and runs the user-supplied "main" function. This code may
+ * connect to a database and run transactions. Once started, it stays active
+ * until shutdown or crash. The process should sleep during periods of
+ * inactivity.
+ *
+ * If the fork() call fails in the postmaster, it will try again later. Note
+ * that the failure can only be transient (fork failure due to high load,
+ * memory pressure, too many processes, etc); more permanent problems, like
+ * failure to connect to a database, are detected later in the worker and dealt
+ * with just by having the worker exit normally. Postmaster will launch a new
+ * worker again later.
+ *
+ * Note that there might be more than one worker in a database concurrently,
+ * and the same module may request more than one worker running the same (or
+ * different) code.
+ *
+ *
+ * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/include/postmaster/bgworker.h
+ *--------------------------------------------------------------------
+ */
+ #ifndef BGWORKER_H
+ #define BGWORKER_H
+
+ /*---------------------------------------------------------------------
+ * External module API.
+ *---------------------------------------------------------------------
+ */
+
+ /*
+ * Pass this flag to have your worker be able to connect to shared memory.
+ */
+ #define BGWORKER_SHMEM_ACCESS 0x0001
+
+ /*
+ * This flag means the bgworker requires a database connection. The connection
+ * is not established automatically; the worker must establish it later.
+ * It requires that BGWORKER_SHMEM_ACCESS was passed too.
+ */
+ #define BGWORKER_BACKEND_DATABASE_CONNECTION 0x0002
+
+
+ typedef void (*bgworker_main_type)(void *main_arg);
+ typedef void (*bgworker_sighdlr_type)(SIGNAL_ARGS);
+
+ /*
+ * Points in time at which a bgworker can request to be started
+ */
+ typedef enum
+ {
+ BgWorkerStart_PostmasterStart,
+ BgWorkerStart_ConsistentState,
+ BgWorkerStart_RecoveryFinished
+ } BgWorkerStartTime;
+
+ #define BGW_DEFAULT_RESTART_INTERVAL 60
+ #define BGW_NEVER_RESTART -1
+
+ typedef struct BackgroundWorker
+ {
+ char *bgw_name;
+ int bgw_flags;
+ BgWorkerStartTime bgw_start_time;
+ int bgw_restart_time; /* in seconds, or BGW_NEVER_RESTART */
+ bgworker_main_type bgw_main;
+ void *bgw_main_arg;
+ bgworker_sighdlr_type bgw_sighup;
+ bgworker_sighdlr_type bgw_sigterm;
+ } BackgroundWorker;
+
+ /* Register a new bgworker */
+ extern void RegisterBackgroundWorker(BackgroundWorker *worker);
+
+ /* This is valid in a running worker */
+ extern BackgroundWorker *MyBgworkerEntry;
+
+ /*
+ * Connect to the specified database, as the specified user. Only a worker
+ * that passed BGWORKER_BACKEND_DATABASE_CONNECTION during registration may
+ * call this.
+ *
+ * If username is NULL, bootstrapping superuser is used.
+ * If dbname is NULL, connection is made to no specific database;
+ * only shared catalogs can be accessed.
+ */
+ extern void BackgroundWorkerInitializeConnection(char *dbname, char *username);
+
+ /* Block/unblock signals in a background worker process */
+ extern void BackgroundWorkerBlockSignals(void);
+ extern void BackgroundWorkerUnblockSignals(void);
+
+ #endif /* BGWORKER_H */
*** a/src/include/postmaster/postmaster.h
--- b/src/include/postmaster/postmaster.h
***************
*** 51,56 **** extern void ClosePostmasterPorts(bool am_syslogger);
--- 51,58 ----
extern int MaxLivePostmasterChildren(void);
+ extern int GetNumShmemAttachedBgworkers(void);
+
#ifdef EXEC_BACKEND
extern pid_t postmaster_forkexec(int argc, char *argv[]);
extern void SubPostmasterMain(int argc, char *argv[]) __attribute__((noreturn));
*** a/src/include/storage/proc.h
--- b/src/include/storage/proc.h
***************
*** 188,193 **** typedef struct PROC_HDR
--- 188,195 ----
PGPROC *freeProcs;
/* Head of list of autovacuum's free PGPROC structures */
PGPROC *autovacFreeProcs;
+ /* Head of list of bgworker free PGPROC structures */
+ PGPROC *bgworkerFreeProcs;
/* WALWriter process's latch */
Latch *walwriterLatch;
/* Checkpointer process's latch */