xact.c state machine redesign

After looking over the state machine in xact.c, I'm thinking of removing
the TBLOCK_SUBENDABORT_ALL and TBLOCK_SUBENDABORT_RELEASE states in
favor of having the ROLLBACK command mark the whole transaction state
stack similarly to what is now done for COMMIT. In detail this would
require adding a TBLOCK_ABORT_PENDING state to use at the top level,
and ROLLBACK would act thus:

* For each subtransaction level: if it's in SUBABORT state (ie, already
aborted) then shift it to SUBENDABORT state (giving permission to pop it
from the stack); otherwise mark it SUBABORT_PENDING.

* At the outer level: if it's in ABORT state then shift to ENDABORT,
otherwise mark it ABORT_PENDING.

In CommitTransactionCommand we would have the behaviors:

ABORT:
SUBABORT:
do nothing (same as now)

SUBENDABORT:
cleanup & pop
recursively examine parent

SUBABORT_PENDING:
abort subtransaction
cleanup & pop
recursively examine parent

ENDABORT:
cleanup
go to DEFAULT state

ABORT_PENDING:
abort transaction
cleanup
go to DEFAULT state

I'm also toying with handling ROLLBACK TO by marking all the levels
above the target as SUBENDABORT or SUBABORT_PENDING, and then marking
the target level with one of two new states, TBLOCK_SUBRESTART or
TBLOCK_SUBABORT_RESTART (the latter if it was already SUBABORT).
These would have the behaviors

TBLOCK_SUBRESTART:
abort subtransaction
cleanup & pop
start new subtransaction with same name

TBLOCK_SUBABORT_RESTART:
cleanup & pop
start new subtransaction with same name

This isn't any fewer states than we have now, but the states seem much
more clearly organized to me --- in particular, other than the RESTART
states there's full symmetry between outer-level and subtransaction
states. Also, this ensures that the planned state transitions are fully
marked out on the state stack before we start to do anything, which
I think is going to be more robust. AbortOutOfAnyTransaction is a bit of
a kluge and I don't really want to depend on it to implement ROLLBACK.

Comments?

regards, tom lane