RE: TOAST-table vacuuming (was Re: Idea for reducing pl anning time)

We'd still want XID keys for the locks that are used to wait for a
particular transaction to complete (eg when waiting to update
a tuple). I think that's OK since VACUUM doesn't need to hold any
such locks, but it'd probably mean making separate lmgr API entry
points for those locks as opposed to normal table-level locks.

In this case XID is used as key in LOCKTAG, ie in lock identifier,
but we are going to change XIDTAG, ie just holder identifier.
No new entry will be required.

Vadim

I can work on this if you don't have time to...

I have no -:)

Vadim

"Mikheev, Vadim" <vmikheev@SECTORBASE.COM> writes:

Use BackendID instead of XID in XIDTAG?
Add internal (ie per backend) hash of locks that should not be
released at commit time?
And couple additional funcs in lmgr API?

I think that would work. I'm inclined to use backend PID instead of
BackendID in XIDTAG, because that way you could tell if a lock entry
is stale even after the BackendID gets recycled to a new process.
(Of course PIDs get recycled too, but not as fast as BackendIDs.)
Besides, PID is already being used by the USERLOCK case, and we could
eliminate some #ifdefs by making the two cases the same.

We'd still want XID keys for the locks that are used to wait for a
particular transaction to complete (eg when waiting to update a tuple).
I think that's OK since VACUUM doesn't need to hold any such locks,
but it'd probably mean making separate lmgr API entry points for those
locks as opposed to normal table-level locks.

Error cleanup should release all locks including those marked as
surviving the current xact. (We'd need to improve that when we
start to work on nested xacts, but it'll do for now.)

Other than that, it seems like it'd work, and it'd allow us to do a
normal transaction commit internally in VACUUM, which is a lot cleaner
than what VACUUM does now.

Comments, better ideas, anyone?

I can work on this if you don't have time to...

regards, tom lane

"Mikheev, Vadim" <vmikheev@SECTORBASE.COM> writes:

We'd still want XID keys for the locks that are used to wait for a
particular transaction to complete (eg when waiting to update
a tuple). I think that's OK since VACUUM doesn't need to hold any
such locks, but it'd probably mean making separate lmgr API entry
points for those locks as opposed to normal table-level locks.

In this case XID is used as key in LOCKTAG, ie in lock identifier,
but we are going to change XIDTAG, ie just holder identifier.
No new entry will be required.

Oh, OK. What say I rename the data structure to HOLDERTAG or something
like that, so it's more clear what it's for? Any suggestions for a
name?

regards, tom lane

"Mikheev, Vadim" wrote:

Also, is TOAST-table vacuuming fixed now?

Still broken. Hiroshi had muttered something about fixing
the internal commit of VACUUM so that it's more like a real
commit --- including advancing the transaction ID --- but
still doesn't release the exclusive lock held by VACUUM.
Basically we need to propagate the locks forward to the new
xact instead of releasing them. I think that would be a nice
clean solution if we could do it. Do you have any ideas about how?

Yes, it would be nice for cursors too - they should be able to cross
transaction boundaries...

Cursors outside transactions is a nice feature I've
long wanted.
It would be nice if one of the requirement is solved.

How many factors should be solved to achieve it ?
For example,not release the buffers kept by such
cursors ...

Regards.
Hiroshi Inoue

I said:

Other than that, it seems like it'd work, and it'd allow us to do a
normal transaction commit internally in VACUUM, which is a lot cleaner
than what VACUUM does now.

I punted on actually changing repair_frag's RecordTransactionCommit()
call into CommitTransactionCommand()/StartTransactionCommand(). That
would now work as far as holding an exclusive lock on the table goes,
but there's more code that would have to be added to close/reopen the
relation and its indexes, ensure that all of VACUUM's internal data
structures survive into the new transaction, etc etc. I didn't have time
for that right now, so I left it as-is. Perhaps someone will feel like
cleaning it up for 7.2 or later.

However, the important thing is fixed: a TOAST table is now vacuumed
in a separate transaction from its master, while still holding a lock
that prevents anyone else from touching the master.

regards, tom lane