support for hyper-long transactions ...

Hans-Juergen Schoenig wrote:

at the moment i am working on an application which is supposed to run
extremely large transactions (a lot of server side stored procedure
stuff which can hardly be split into small transactions for visibility
reasons).
so, from time to time it happens that i exceed my CommandCounter (>
2.000.000.000 statements inside the same transaction).

Interesting. What kind of a stored procedure is that?

One optimization that might be possible is to not increment it for
statements that do on-disk changes.

my idea is: how about adding a configure option to compile postgres with
a 64 bit command counter. this would allow larger transactions for
special purpose applications while it would not have an impact on normal
applications.

One objection to expanding TransactionId to 64-bits has been that we
depend on assignment of TransactionId to be atomic. That might not be an
issue with CommandIds; I don't think we store commandids in shared
memory structures. It still doesn't feel like a good idea to me, though.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Hans-Juergen Schoenig wrote:

at the moment i am working on an application which is supposed to run
extremely large transactions (a lot of server side stored procedure
stuff which can hardly be split into small transactions for visibility
reasons).
so, from time to time it happens that i exceed my CommandCounter (>
2.000.000.000 statements inside the same transaction).

AFAICS, maximum number of command ids is actually 2^32-1, or over 4
billion. Are you sure you bumped into that limit and not something else?
What's the error message you're getting?

What version of Postgres is this? PG 8.3 will have another related limit
on the number of combocids you can have.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Heikki Linnakangas írta:

Hans-Juergen Schoenig wrote:

at the moment i am working on an application which is supposed to run
extremely large transactions (a lot of server side stored procedure
stuff which can hardly be split into small transactions for visibility
reasons).
so, from time to time it happens that i exceed my CommandCounter (>
2.000.000.000 statements inside the same transaction).

Interesting. What kind of a stored procedure is that?

One optimization that might be possible is to not increment it for
statements that do on-disk changes.

my idea is: how about adding a configure option to compile postgres with
a 64 bit command counter. this would allow larger transactions for
special purpose applications while it would not have an impact on normal
applications.

One objection to expanding TransactionId to 64-bits has been that we
depend on assignment of TransactionId to be atomic. That might not be an
issue with CommandIds; I don't think we store commandids in shared
memory structures. It still doesn't feel like a good idea to me, though.

Isn't 64-bit assignment atomic on 64-bit machines?
With a little autoconf magic, the conditional can be
disabled for 32-bit platforms. So that's not a real obstacle
for going to 64-bit TransactionIds.

--
----------------------------------
Zoltán Böszörményi
Cybertec Geschwinde & Schönig GmbH
http://www.postgresql.at/

AFAICS, maximum number of command ids is actually 2^32-1, or over 4
billion. Are you sure you bumped into that limit and not something
else?
What's the error message you're getting?

What version of Postgres is this? PG 8.3 will have another related
limit
on the number of combocids you can have.

it is clearly caused by the CommandCounter - it is indicated by the
error message. i don't have the message on my notebook here but this
is exactly what is going on if we run out of statements.

i would not see atomic as a problem here as we can support it for 64
bit boxes only.
i would vote for some autoconf flag which is off by default to make
sure that other applications don't waste space here.

to answer the question you had before:
it is an application going through some enormous amount of raw data
and trying to do some filtering, analysis and preaggregation (which
is not an issue here). the thing is that filtering and analysis are
quite complex and have to be done on a per entry level (a lot of
conditional lookups, if statements, custom aggregated, status changed
and so on). if you are forced to do this fancy logic for 1xx mio
records you can easily run out of commands.

many thanks,

hans

--
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26, 2700 Wiener Neustadt
Tel: +43/1/205 10 35 / 340
www.postgresql.at, www.cybertec.at