realtime data inserts
I have realtime data flowing at a rate of 500, 512 byte packets per second.
I want to log the info in a database table with two other columns, one for a
timestamp and one for a name of the packet. The max rate I can achieve is
350 inserts per second on a sun blade 2000. The inserts are grouped in a
transaction and I commit every 1200 records. I am storing the binary data
in a bytea. I am using the libpq conversion function. Not sure if that is
slowing me down. But I think it is the insert not the conversion.
Any thoughts on how to achive this goal?
"Adam Siegel" <adam@sycamorehq.com> writes:
I have realtime data flowing at a rate of 500, 512 byte packets per second.
I want to log the info in a database table with two other columns, one for a
timestamp and one for a name of the packet. The max rate I can achieve is
350 inserts per second on a sun blade 2000. The inserts are grouped in a
transaction and I commit every 1200 records.
Have you thought about using COPY?
regards, tom lane
Had the same problem recently...
Format your data like a pg text dump into a file and then...
COPY <tablename> (a,b,c) FROM stdin;
1 2 3
4 5 6
\.
psql <yourdatabase < dumpfile.sql
I've achieved thousands of rows per seconds with this method.
- Ericson Smith
eric@did-it.com
http://www.did-it.com
Adam Siegel wrote:
Show quoted text
I have realtime data flowing at a rate of 500, 512 byte packets per second.
I want to log the info in a database table with two other columns, one for a
timestamp and one for a name of the packet. The max rate I can achieve is
350 inserts per second on a sun blade 2000. The inserts are grouped in a
transaction and I commit every 1200 records. I am storing the binary data
in a bytea. I am using the libpq conversion function. Not sure if that is
slowing me down. But I think it is the insert not the conversion.Any thoughts on how to achive this goal?
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
Are you binding your insert? IE:
prepare statement INSERT INTO blah VALUES (?, ?, ?);
execute statement (a, b, c)
Instead of just "INSERT INTO blah VALUES(a, b, c)"
On Sat, May 10, 2003 at 11:25:16AM -0400, Adam Siegel wrote:
I have realtime data flowing at a rate of 500, 512 byte packets per second.
I want to log the info in a database table with two other columns, one for a
timestamp and one for a name of the packet. The max rate I can achieve is
350 inserts per second on a sun blade 2000. The inserts are grouped in a
transaction and I commit every 1200 records. I am storing the binary data
in a bytea. I am using the libpq conversion function. Not sure if that is
slowing me down. But I think it is the insert not the conversion.Any thoughts on how to achive this goal?
--
Jim C. Nasby (aka Decibel!) jim@nasby.net
Member: Triangle Fraternity, Sports Car Club of America
Give your computer some brain candy! www.distributed.net Team #1828
Windows: "Where do you want to go today?"
Linux: "Where do you want to go tomorrow?"
FreeBSD: "Are you guys coming, or what?"
On Sat, 2003-05-10 at 11:00, Tom Lane wrote:
"Adam Siegel" <adam@sycamorehq.com> writes:
I have realtime data flowing at a rate of 500, 512 byte packets per second.
I want to log the info in a database table with two other columns, one for a
timestamp and one for a name of the packet. The max rate I can achieve is
350 inserts per second on a sun blade 2000. The inserts are grouped in a
transaction and I commit every 1200 records.Have you thought about using COPY?
Generate a temporary file, and then system("COPY /tmp/foobar ...") ?
--
+---------------------------------------------------------------+
| Ron Johnson, Jr. mailto:ron.l.johnson@cox.net |
| Jefferson, LA USA http://members.cox.net/ron.l.johnson |
| |
| The purpose of the military isn't to pay your college tuition |
| or give you a little extra income; it's to "kill people and |
| break things". Surprisingly, not everyone understands that. |
+---------------------------------------------------------------+Ron Johnson <ron.l.johnson@cox.net> writes:
On Sat, 2003-05-10 at 11:00, Tom Lane wrote:
Have you thought about using COPY?
Generate a temporary file, and then system("COPY /tmp/foobar ...") ?
No, copy from stdin. No need for a temp file.
regards, tom lane
On Sat, 2003-05-10 at 21:46, Tom Lane wrote:
Ron Johnson <ron.l.johnson@cox.net> writes:
On Sat, 2003-05-10 at 11:00, Tom Lane wrote:
Have you thought about using COPY?
Generate a temporary file, and then system("COPY /tmp/foobar ...") ?
No, copy from stdin. No need for a temp file.
But wouldn't that only work if the input stream is acceptable to
COPY ?
--
+---------------------------------------------------------------+
| Ron Johnson, Jr. mailto:ron.l.johnson@cox.net |
| Jefferson, LA USA http://members.cox.net/ron.l.johnson |
| |
| The purpose of the military isn't to pay your college tuition |
| or give you a little extra income; it's to "kill people and |
| break things". Surprisingly, not everyone understands that. |
+---------------------------------------------------------------+
Ron Johnson <ron.l.johnson@cox.net> writes:
On Sat, 2003-05-10 at 21:46, Tom Lane wrote:
Ron Johnson <ron.l.johnson@cox.net> writes:
On Sat, 2003-05-10 at 11:00, Tom Lane wrote:
Have you thought about using COPY?
Generate a temporary file, and then system("COPY /tmp/foobar ...") ?
No, copy from stdin. No need for a temp file.
But wouldn't that only work if the input stream is acceptable to
COPY ?
Yes, but you could always pipe it through a script or C program to
make it so...
-Doug
Import Notes
Reply to msg id not found: RonJohnsonsmessageof11May2003115450-0500
The copy from method (PQputline) allows me to achieve around 1000 inserts
per second.
On Sat, 10 May 2003, Jim C. Nasby wrote:
Show quoted text
Are you binding your insert? IE:
prepare statement INSERT INTO blah VALUES (?, ?, ?);
execute statement (a, b, c)
Instead of just "INSERT INTO blah VALUES(a, b, c)"
On Sat, May 10, 2003 at 11:25:16AM -0400, Adam Siegel wrote:
I have realtime data flowing at a rate of 500, 512 byte packets per second.
I want to log the info in a database table with two other columns, one for a
timestamp and one for a name of the packet. The max rate I can achieve is
350 inserts per second on a sun blade 2000. The inserts are grouped in a
transaction and I commit every 1200 records. I am storing the binary data
in a bytea. I am using the libpq conversion function. Not sure if that is
slowing me down. But I think it is the insert not the conversion.Any thoughts on how to achive this goal?
Depends - we don't know enough about your needs. Some questions:
Is this constant data or just capturing a burst?
Are you feeding it through one connection or several in parallel?
Did you tune your memory configs in postgresql.conf or are they still at the
minimalized defaults?
How soon does the data need to be available for query? (Obviously there will
be up to a 1200 record delay just due to the transaction.)
What generates the timestamp? Ie. is it an insert into foo values (now(),
packetname, data) or is the app providing the timestamp?
More info about the app will help.
Cheers,
Steve
Show quoted text
On Saturday 10 May 2003 8:25 am, Adam Siegel wrote:
I have realtime data flowing at a rate of 500, 512 byte packets per second.
I want to log the info in a database table with two other columns, one for
a timestamp and one for a name of the packet. The max rate I can achieve
is 350 inserts per second on a sun blade 2000. The inserts are grouped in
a transaction and I commit every 1200 records. I am storing the binary
data in a bytea. I am using the libpq conversion function. Not sure if
that is slowing me down. But I think it is the insert not the conversion.Any thoughts on how to achive this goal?
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
Doug McNaught wrote:
Ron Johnson <ron.l.johnson@cox.net> writes:
On Sat, 2003-05-10 at 21:46, Tom Lane wrote:
Ron Johnson <ron.l.johnson@cox.net> writes:
On Sat, 2003-05-10 at 11:00, Tom Lane wrote:
Have you thought about using COPY?
Generate a temporary file, and then system("COPY /tmp/foobar ...") ?
No, copy from stdin. No need for a temp file.
But wouldn't that only work if the input stream is acceptable to
COPY ?Yes, but you could always pipe it through a script or C program to
make it so...
lets say I have an about 1kb/s continuus datastream comming in for many
hours and I'd like to store this data in my db using COPY table FROM stdin.
At what time should I COMMIT or close the stream to feed the database
and COPY FROM again?
You probably want to have a process that constantly stores the data in a
text file. Every "n" minutes, you will cause the logger to rotate the
text file, then process that batch.
Over here, we are able to dump around 5,000 records per second in one of
our tables using that methodology.
- Ericson Smith
eric@did-it.com
On Fri, 2003-05-16 at 16:27, alex b. wrote:
Doug McNaught wrote:
Ron Johnson <ron.l.johnson@cox.net> writes:
On Sat, 2003-05-10 at 21:46, Tom Lane wrote:
Ron Johnson <ron.l.johnson@cox.net> writes:
On Sat, 2003-05-10 at 11:00, Tom Lane wrote:
Have you thought about using COPY?
Generate a temporary file, and then system("COPY /tmp/foobar ...") ?
No, copy from stdin. No need for a temp file.
But wouldn't that only work if the input stream is acceptable to
COPY ?Yes, but you could always pipe it through a script or C program to
make it so...lets say I have an about 1kb/s continuus datastream comming in for many
hours and I'd like to store this data in my db using COPY table FROM stdin.At what time should I COMMIT or close the stream to feed the database
and COPY FROM again?---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?
--
Ericson Smith <eric@did-it.com>
On Fri, 2003-05-16 at 15:33, Ericson Smith wrote:
You probably want to have a process that constantly stores the data in a
text file. Every "n" minutes, you will cause the logger to rotate the
text file, then process that batch.
Does the logger spawn the DB writer?
Over here, we are able to dump around 5,000 records per second in one of
our tables using that methodology.- Ericson Smith
eric@did-it.comOn Fri, 2003-05-16 at 16:27, alex b. wrote:
Doug McNaught wrote:
Ron Johnson <ron.l.johnson@cox.net> writes:
On Sat, 2003-05-10 at 21:46, Tom Lane wrote:
Ron Johnson <ron.l.johnson@cox.net> writes:
On Sat, 2003-05-10 at 11:00, Tom Lane wrote:
Have you thought about using COPY?
Generate a temporary file, and then system("COPY /tmp/foobar ...") ?
No, copy from stdin. No need for a temp file.
But wouldn't that only work if the input stream is acceptable to
COPY ?Yes, but you could always pipe it through a script or C program to
make it so...lets say I have an about 1kb/s continuus datastream comming in for many
hours and I'd like to store this data in my db using COPY table FROM stdin.At what time should I COMMIT or close the stream to feed the database
and COPY FROM again?---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?
--
+---------------------------------------------------------------+
| Ron Johnson, Jr. mailto:ron.l.johnson@cox.net |
| Jefferson, LA USA http://members.cox.net/ron.l.johnson |
| |
| The purpose of the military isn't to pay your college tuition |
| or give you a little extra income; it's to "kill people and |
| break things". Surprisingly, not everyone understands that. |
+---------------------------------------------------------------+