Database viewpoint of subject - Sending e-mails from database table with Cronjob

I generate e-mail messages to a database table and then with a CronJob I
sent the e-mails.

My doubt is... The CronJob runs every 10 minutes, but If I have 100.000
e-mails to send the script will not be able to send all the 100.000 e-mails
in 10 minutes.

How can I deal with this problem? There is no problem to have multiple
CronJobs runing in background?

If N is the number of mails you can send per cronjob in 10 minutes,
then run 100 000/N cron jobs.
You will be very happy if you also add som kind of "sleep" (for a
growing number of seconds) to the cronjobs when there are currently no
more mails to send. Otherwise you will end up with a very annoying CPU
load.

The other way is to let the cron job spawn new processes (up to a
limited number of child proceses) as long as there are mails to send.
These child processes runs as long as there are mails to send, then
they die. The cron job will then mostly do process controll/start new
processes.

That ought to do the trick.

Hi,

Thanks for the reply.

[quote]
The other way is to let the cron job spawn new processes (up to a
limited number of child proceses) as long as there are mails to send.
These child processes runs as long as there are mails to send, then
they die. The cron job will then mostly do process controll/start new
processes.
[/quote]

How can I do this. I'am thinking to use a PHP Script?

Best Regards,

On Wed, Apr 21, 2010 at 12:35 PM, A B <gentosaker@gmail.com> wrote:

On Wed, Apr 21, 2010 at 3:41 PM, Andre Lopes <lopes80andre@gmail.com> wrote:

Hi,

Thanks for the reply.

[quote]
The other way is to let the cron job spawn new processes (up to a
limited number of child proceses) as long as there are mails to send.
These child processes runs as long as there are mails to send, then
they die. The cron job will then mostly do process controll/start new
processes.
[/quote]

How can I do this. I'am thinking to use a PHP Script?

Best Regards,

On Wed, Apr 21, 2010 at 12:35 PM, A B <gentosaker@gmail.com> wrote:

I generate e-mail messages to a database table and then with a CronJob I
sent the e-mails.

My doubt is... The CronJob runs every 10 minutes, but If I have 100.000
e-mails to send the script will not be able to send all the 100.000
e-mails
in 10 minutes.

How can I deal with this problem? There is no problem to have multiple
CronJobs runing in background?

If N is the number of mails you can send per cronjob in 10 minutes,
then run 100 000/N cron jobs.
You will be very happy if you also add som kind of "sleep" (for a
growing number of seconds) to the cronjobs when there are currently no
more mails to send. Otherwise you will end up with a very annoying CPU
load.

The other way is to let the cron job spawn new processes (up to a
limited number of child proceses) as long as there are mails to send.
These child processes runs as long as there are mails to send, then
they die. The cron job will then mostly do process controll/start new
processes.

That ought to do the trick.

I really hope this is not for SPAM mail.

Allan.

It is not for SPAM, the 100.000 e-mails is only for example purposes. But I
must to draw the code to be possible to send this volume, just in case if it
happens.

There are examples on how can I do this?

Best Regards,

On Wed, Apr 21, 2010 at 4:01 PM, Allan Kamau <kamauallan@gmail.com> wrote:

[quote]
The other way is to let the cron job spawn new processes (up to a
limited number of child proceses) as long as there are mails to send.
These child processes runs as long as there are mails to send, then
they die. The cron job will then mostly do process controll/start new
processes.
[/quote]

How can I do this. I'am thinking to use a PHP Script?

This is getting a little of topic, but I guess something like
exec('php some_script.php') will work?

In response to A B <gentosaker@gmail.com>:

[quote]
The other way is to let the cron job spawn new processes (up to a
limited number of child proceses) as long as there are mails to send.
These child processes runs as long as there are mails to send, then
they die. The cron job will then mostly do process controll/start new
processes.
[/quote]

How can I do this. I'am thinking to use a PHP Script?

This is getting a little of topic, but I guess something like
exec('php some_script.php') will work?

There's a php/pgsql list available on this listserve that will get
you a lot more help in this area.

However, I would suggest PHP's pcntl_fork() and friends for creating
multiple processes. Of course, that's only needed if you want to
run them in parallel, which was unclear to me.

--
Bill Moran
http://www.potentialtech.com
http://people.collaborativefusion.com/~wmoran/

Thanks for the reply.

I will move the topic to the php/pgsql if I don't get how to do it.

Thanks for the help.

Best Regards,

On Wed, Apr 21, 2010 at 4:49 PM, Bill Moran <wmoran@potentialtech.com>wrote:

Andre Lopes <lopes80andre@gmail.com> wrote:

My doubt is... The CronJob runs every 10 minutes, but If I have 100.000
e-mails to send the script will not be able to send all the 100.000 e-mails
in 10 minutes.

More generally, your question is how to deal with a cron job that is
intended to run every ten minutes, but which you don't want to run more
than one copy of (i.e. this has little or nothing to do with PostgreSQL,
really).

My usual solution is twofold:

1. have the cron job take a lock file as the first thing it does: this
ensures only one job can run at a time

Something like FreeBSD's lockf(1) is good for this, and isn't hard to
port or write a similar utility for any OS I've had to use:

http://www.freebsd.org/cgi/man.cgi?query=lockf&amp;apropos=0&amp;sektion=1&amp;manpath=FreeBSD+8.0-RELEASE&amp;format=html

2. include in the job a 'dead man switch' that terminates the job
if it runs for "too long", whatever "too long" is in your context.

Obviously, 10 minutes is possible; 10 hours is probably not.

Making sure client applications terminate (be they cron jobs, CGI
scripts, or anything else) stops all your database connections
being tied up by hung scripts, to bring the discussion slightly
back to PostgreSQL).

How can I deal with this problem? There is no problem to have multiple
CronJobs runing in background?

Multiple cron jobs in general, no. Multiple instances of a single
cron job depends on the cron job.

As another poster wrote, I do hope this isn't for spam, as you say it
isn't, but the basics I'd follow are #1 and #2 above, and let the mail
system handle both the queue and the allowable amount of parallel
outbound SMTP connections. That's what mail software is for.

Of course, I'd _definitely_ not use any such thing for spamming....

Regards,

Giles