keeping track of function execution
Hi,
I have a question on transactions/isolation levels/etc...
In my PL/pgSQL function main loop goes through inventory list of active
devices, for each one executing processing applicable for given device,
like:
FOR i in --i is %rowtype
select device_id as device_id,
type as type
from devices_list
where active = 1
LOOP
(...)
-- here is CASE statement, checking value of 'type' parameter
(....)
END LOOP; --simple enough, right?
This processing is pretty heavy and takes lot of time... so, I'd like to
be able to monitor as processing progresses and I need to be able to say:
-which devices've been processed already
-which ones've not been processed yet
My first idea was to create table, updated by my function each time next
device is processed, like:
device_id;status
1;0--done
2;0--done
3;1--processing is running
4;2--to be processed
But... Postgress treats function as single transaction, of course.
Hence, I'm not able to see any changes in my progress monitoring table
until my main function is finished and all the statuses are set to 0.
Which is not really the intent (again, the intent is to be able to
monitor which devices are yet to be processed while function is still
running!)
My ideas so far (none is perfect, unfortunately)
- move my loop to php/other external piece of code... so it will log-in
progress in my function using separate transactions (well, I don't want
to use external code, would prefer to stay in PL/pgSQL)
- log to text file (slow and not easy to report later on)
Can I ask for any other suggestions/comments? Is there a way I can have
this functionality, please?
Regards,
foo
Wojtek wrote:
But... Postgress treats function as single transaction, of course.
Hence, I'm not able to see any changes in my progress monitoring table
until my main function is finished and all the statuses are set to 0.
You could use dblink() to insert into your logging table.
David.
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Wojtek wrote:
<blockquote cite="mid:49FDBA8D.2030301@twine.pl" type="cite">Hi, <br>
<br>
I have a question on transactions/isolation levels/etc... <br>
In my PL/pgSQL function main loop goes through inventory list of active
devices, for each one executing processing applicable for given device,
like: <br>
FOR i in --i is %rowtype <br>
select device_id as device_id, <br>
type as type <br>
from devices_list <br>
where active = 1 <br>
LOOP <br>
(...) <br>
-- here is CASE statement, checking value of 'type' parameter <br>
(....) <br>
END LOOP; --simple enough, right? <br>
<br>
This processing is pretty heavy and takes lot of time... so, I'd like
to be able to monitor as processing progresses and I need to be able to
say: <br>
-which devices've been processed already <br>
-which ones've not been processed yet <br>
<br>
My first idea was to create table, updated by my function each time
next device is processed, like: <br>
device_id;status <br>
1;0--done <br>
2;0--done <br>
3;1--processing is running <br>
4;2--to be processed <br>
<br>
But... Postgress treats function as single transaction, of course.
Hence, I'm not able to see any changes in my progress monitoring table
until my main function is finished and all the statuses are set to 0.
Which is not really the intent (again, the intent is to be able to
monitor which devices are yet to be processed while function is still
running!) <br>
<br>
My ideas so far (none is perfect, unfortunately) <br>
- move my loop to php/other external piece of code... so it will log-in
progress in my function using separate transactions (well, I don't want
to use external code, would prefer to stay in PL/pgSQL) <br>
- log to text file (slow and not easy to report later on) <br>
<br>
Can I ask for any other suggestions/comments? Is there a way I can have
this functionality, please? <br>
<br>
Regards, <br>
foo <br>
<br>
</blockquote>
<font face="Arial">What about using Triggers when a device is being
processed throw a flag in anther table or in the same table. <br>
<br>
I'm guessing some other kind of process set the status 0, 1 or 2 before
this slow process gets going. if that is the case just add a trigger
on update to set the status. Then a simple query will get you what
you want.<br>
<br>
If I'm understanding what your after.<br>
</font>
<br>
</body>
</html>
On Sun, May 3, 2009 at 9:38 AM, Wojtek <foo@twine.pl> wrote:
Hi,
I have a question on transactions/isolation levels/etc...
In my PL/pgSQL function main loop goes through inventory list of active
devices, for each one executing processing applicable for given device,
like:
FOR i in --i is %rowtype
select device_id as device_id,
type as type
from devices_list
where active = 1
LOOP
(...)
-- here is CASE statement, checking value of 'type' parameter
(....)
END LOOP; --simple enough, right?This processing is pretty heavy and takes lot of time... so, I'd like to be
able to monitor as processing progresses and I need to be able to say:
-which devices've been processed already
-which ones've not been processed yet
You could raise notice and then tail the logs as it runs. Or the
other option mentioned of using dblink() works well too.
On 2009-05-03, Scott Marlowe <scott.marlowe@gmail.com> wrote:
On Sun, May 3, 2009 at 9:38 AM, Wojtek <foo@twine.pl> wrote:
Hi,
I have a question on transactions/isolation levels/etc...
In my PL/pgSQL function main loop goes through inventory list of active
devices, for each one executing processing applicable for given device,
like:
FOR i in --i is %rowtype
select device_id as device_id,
type as type
from devices_list
where active = 1
LOOP
(...)
-- here is CASE statement, checking value of 'type' parameter
(....)
END LOOP; --simple enough, right?This processing is pretty heavy and takes lot of time... so, I'd like to be
able to monitor as processing progresses and I need to be able to say:
-which devices've been processed already
-which ones've not been processed yetYou could raise notice and then tail the logs as it runs. Or the
other option mentioned of using dblink() works well too.
you can retrieve notices through libpq. we're experimenting with using them,
and notifies to control the business logic in applications.
Scott Marlowe wrote:
On Sun, May 3, 2009 at 9:38 AM, Wojtek <foo@twine.pl> wrote:
(...)
You could raise notice and then tail the logs as it runs. Or the
other option mentioned of using dblink() works well too.
dblink did the trick, thanks guys!
foo
Hi,
I'm looking for solution to speed up my data load process. Currently, I
have up to 100 insert/second loaded to partitioned table and application
is inserting data directly to partitions (not to master table).
As a solution, I wanted to explore pg_bulkloader but as I understand it
needs to have destination table specified in config file, so:
- I can't make it to write directly to partitions (as I need to write
data to several partitions every day)
- I doubt pg_bulkload will consider rules/constraints to put data to
right partitions automatically
Anyone can share good practices on loading data to partitioned table,
please? not necessarily using pg_bulkload...
Regards,
foo