looping over the rows in a table
Hi, this is slightly offtopic, but is based on Postgres:
I have a table with 10M rows and I have a Python script using psycopg
that needs to look at each row of the table. My current strategy is
to do in the Python script
cursor.execute("select acol from atable")
while True:
ret = cursor.fetchone()
if not ret: break
However if I understand correctly Postgres will basically try and
return *all* the rows of the table as the result set, thus taking a
long time and probably running out of memory.
Is there a way I can modify the SQL or do something on the Postgres
side, so that I can loop over all the rows in the table?
Thanks,
-------------------------------------------------------------------
Rajarshi Guha <rguha@indiana.edu>
GPG Fingerprint: 0CCA 8EE2 2EEB 25E2 AB04 06F7 1BB9 E634 9B87 56EE
-------------------------------------------------------------------
A bug in the hand is better than one as yet undetected.
On Nov 9, 2007 6:12 PM, Rajarshi Guha <rguha@indiana.edu> wrote:
Hi, this is slightly offtopic, but is based on Postgres:
I have a table with 10M rows and I have a Python script using psycopg
that needs to look at each row of the table. My current strategy is
to do in the Python scriptcursor.execute("select acol from atable")
while True:
ret = cursor.fetchone()
if not ret: breakHowever if I understand correctly Postgres will basically try and
return *all* the rows of the table as the result set, thus taking a
long time and probably running out of memory.Is there a way I can modify the SQL or do something on the Postgres
side, so that I can loop over all the rows in the table?
Assuming you can't do the work you need in SQL or a stored procedure
or something, yes.
Look up Declare Cursor. I think 8.3 introduces updateable cursors.
don't know if you need that or not with what you're doing.