Per tuple overhead, cmin, cmax

Manfred Koizar <mkoi-pg@aon.at> writes:

(3d) t_xmin == t_xmax == current transaction. The tuple has been
inserted and then deleted by the current transaction. Then I claim
(but I'm not absolutely sure), that insert and delete cannot have
happened in the same command,
so t_cmin < t_cmax,
so t_cmin < CurrentCommandId,
so the exact value of t_cmin is irrelevant.

The hole in this logic is that there can be multiple active scans with
different values of CurrentCommandId (eg, within a function
CurrentCommandId may be different than it is outside). If you overwrite
cmin with cmax then you are destroying the information needed by a scan
with smaller CurrentCommandId than yours.

(Q4) Is it really easy to change the size of HeapTupleHeaderData? Are
the data of this struct only accessed by field names or are there
dirty tricks using memcpy() and pointer arithmetic?

AFAIK there are no dirty tricks there. I am hesitant to change the
header layout without darn good reason, because it breaks any chance
of having a working pg_upgrade process. But that's strictly a
production-system concern, and need not discourage you from
experimenting.

(Q5) Are these thoughts obsolete as soon as nested transactions are
considered?

Possibly. We haven't worked out exactly how nested transactions would
work, but to the extent that they are handled as different CommandIds
we'd have the same issue already mentioned: we should not assume that
execution of different CommandIds can't overlap in time.

regards, tom lane

Tom,
thanks for answering.

On Thu, 02 May 2002 17:16:38 -0400, you wrote:

The hole in this logic is that there can be multiple active scans with
different values of CurrentCommandId (eg, within a function
CurrentCommandId may be different than it is outside). If you overwrite
cmin with cmax then you are destroying the information needed by a scan
with smaller CurrentCommandId than yours.

Oh, I see :-(
Let me throw in one of my infamous wild ideas in an attempt to rescue
my proposal: We have 4 32-bit-numbers: xmin, cmin, xmax, and cmax.
The only case, where we need cmin *and* cmax, is, when xmin == xmax.
So if we find a single bit to flag this case, we only need 3
32-bit-numbers to store this information on disk.

To keep the code readable we probably would need some accessor
functions or macros to access these fields.

As of 7.2 there are three unused bits in t_infomask.

(Q4) Is it really easy to change the size of HeapTupleHeaderData? Are
the data of this struct only accessed by field names or are there
dirty tricks using memcpy() and pointer arithmetic?

AFAIK there are no dirty tricks there. I am hesitant to change the
header layout without darn good reason, because it breaks any chance
of having a working pg_upgrade process. But that's strictly a
production-system concern, and need not discourage you from
experimenting.

Is saving 4 bytes per tuple a "darn good reason"? Is a change
acceptable for 7.3? Do you think it's worth the effort?

We haven't worked out exactly how nested transactions would
work, but to the extent that they are handled as different CommandIds
we'd have the same issue already mentioned: we should not assume that
execution of different CommandIds can't overlap in time.

Assuming that a subtransaction is completely contained in the outer
transaction and there is no activity by the outer transaction while
the subtransaction is active, I believe, this problem can be solved
...
It's late now, I'll try to think clearer tomorrow.

Good night
Manfred

Manfred Koizar <mkoi-pg@aon.at> writes:

Let me throw in one of my infamous wild ideas in an attempt to rescue
my proposal: We have 4 32-bit-numbers: xmin, cmin, xmax, and cmax.
The only case, where we need cmin *and* cmax, is, when xmin == xmax.
So if we find a single bit to flag this case, we only need 3
32-bit-numbers to store this information on disk.

Hmm ... that might work. Actually, we are trying to stuff *five*
numbers into these fields: xmin, xmax, cmin, cmax, and a VACUUM FULL
transaction id (let's call it xvac just to have a name). The code
currently assumes that cmin is not interesting simultaneously with xvac.
I think it might be true that cmax is not interesting simultaneously
with xvac either, in which case this could be made to work. (Vadim,
your thoughts?)

To keep the code readable we probably would need some accessor
functions or macros to access these fields.

Amen. But that would be cleaner than now, at least for VACUUM;
it's just using cmin where it means xvac.

Is saving 4 bytes per tuple a "darn good reason"? Is a change
acceptable for 7.3? Do you think it's worth the effort?

I'm on the fence about it. My thoughts are probably colored by the
fact that I prefer platforms that have MAXALIGN=8, so half the time
(including all null-free rows) there'd be no savings at all. Now if
we could get rid of 8 bytes in the header, I'd get excited ;-)

Any other opinions out there?

regards, tom lane

PS: I did like your point about BITMAPLEN; I think that might be
a free savings. I was waiting for you to bring it up on hackers
before commenting though...

On Thu, 02 May 2002 21:10:40 -0400, Tom Lane wrote:

Hmm ... that might work. Actually, we are trying to stuff *five*
numbers into these fields: xmin, xmax, cmin, cmax, and a VACUUM FULL
transaction id (let's call it xvac just to have a name). The code
currently assumes that cmin is not interesting simultaneously with xvac.
I think it might be true that cmax is not interesting simultaneously
with xvac either, in which case this could be made to work. (Vadim,
your thoughts?)

Having read the sources recently I'm pretty sure you're right.

I'm on the fence about it. My thoughts are probably colored by the
fact that I prefer platforms that have MAXALIGN=8, so half the time
(including all null-free rows) there'd be no savings at all.

But the other half of the time we'd save 8 bytes. So on average we
get savings of 4 bytes per tuple, don't we?

Now if
we could get rid of 8 bytes in the header, I'd get excited ;-)

I keep trying :-)

Servus
Manfred

On Thu, 02 May 2002 21:10:40 -0400, Tom Lane <tgl@sss.pgh.pa.us>
wrote:

Manfred Koizar <mkoi-pg@aon.at> writes:

Is saving 4 bytes per tuple a "darn good reason"?

[...] Now if
we could get rid of 8 bytes in the header, I'd get excited ;-)

Tom,

what about WITHOUT OIDS? I know dropping the OID from some tables and
keeping it for others is not trivial, because t_oid is the _first_
field of HeapTupleHeaderData. I'm vaguely considering a few possible
implementations and will invest more work in a detailed proposal, if
it's wanted.

Servus
Manfred

Manfred Koizar <mkoi-pg@aon.at> writes:

what about WITHOUT OIDS? I know dropping the OID from some tables and
keeping it for others is not trivial, because t_oid is the _first_
field of HeapTupleHeaderData. I'm vaguely considering a few possible
implementations and will invest more work in a detailed proposal, if
it's wanted.

Yeah, I had been toying with the notion of treating OID like a user
field --- ie, it'd be present in the variable-length part of the record
if at all. It'd be a bit tricky to find all the places that would need
to change, but I think there are not all that many.

As usual, the major objection to any such change is losing the chance
of doing pg_upgrade. But we didn't have pg_upgrade during the 7.2
cycle either. If we put together several such changes and did them
all at once, the benefit might be enough to overcome that complaint.

regards, tom lane

On Tue, 21 May 2002 09:57:32 -0400, Tom Lane <tgl@sss.pgh.pa.us>
wrote:

Manfred Koizar <mkoi-pg@aon.at> writes:

what about WITHOUT OIDS? I know dropping the OID from some tables and
keeping it for others is not trivial, because t_oid is the _first_
field of HeapTupleHeaderData. I'm vaguely considering a few possible
implementations and will invest more work in a detailed proposal, if
it's wanted.

Yeah, I had been toying with the notion of treating OID like a user
field --- ie, it'd be present in the variable-length part of the record
if at all. It'd be a bit tricky to find all the places that would need
to change, but I think there are not all that many.

That was one of the possible solutions I thought of, unfortunately the
one I'm most afraid of. Not because I think it's not the cleanest
way, but I don't (yet) feel comfortable enough with the code to rip
out oids from system tables. However, if you tell me it's feasible
and if you give me some hints where to start, I'll give it a try...

Other possible implementations would leave the oid in the tuple
header:

. typedef two structs HeapTupleHeaderDataWithOid and
HeapTupleHeaderDataWithoutOid, wrap access to *all* HeapTupleHeader
fields in accessor macros/functions, give these accessors enough
information to know which variant to use.

. Decouple on-disk format from in-memory structures, use
HeapTupleHeaderPack() and HeapTupleHeaderUnpack() to store/extract
header data to/from disk buffers. Concurrency?

As usual, the major objection to any such change is losing the chance
of doing pg_upgrade. But we didn't have pg_upgrade during the 7.2
cycle either.

I thought, it is quite common to need pg_dump/restore when upgrading
between releases. Or are you talking about those hackers, testers and
users(?), who are using a cvs version now?

Anyway, as long as our changes don't make heap tuples larger, it
should be possible to write a tool that converts version x data files
to version y data files. I've done that before (not for PG though)
and I know it's a lot of work, but wouldn't it be great for the PG
marketing department ;-)

If we put together several such changes [...]

I can't guarantee that; my ideas come in drop by drop :-)
BTW, is there a 7.3 schedule?

Servus
Manfred

Manfred Koizar <mkoi-pg@aon.at> writes:

That was one of the possible solutions I thought of, unfortunately the
one I'm most afraid of. Not because I think it's not the cleanest
way, but I don't (yet) feel comfortable enough with the code to rip
out oids from system tables.

The system tables that have OIDs will certainly continue to have OIDs.

I suppose the messiest aspect of that solution would be changing all
the places that currently do "tuple->t_data->t_oid". If OID is not at
a fixed offset in the tuple then it'll be necessary to change *all*
those places. Ugh. While certainly we should have been using accessor
macros for that, I'm not sure I want to try to change it.

Other possible implementations would leave the oid in the tuple
header:

. typedef two structs HeapTupleHeaderDataWithOid and
HeapTupleHeaderDataWithoutOid, wrap access to *all* HeapTupleHeader
fields in accessor macros/functions, give these accessors enough
information to know which variant to use.

If OID is made to be the last fixed-offset field, instead of the first,
then this approach would be fairly workable. Actually I'd still use
just one struct definition, but do offsetof() calculations to decide
where the null-bitmap starts.

Decouple on-disk format from in-memory structures, use
HeapTupleHeaderPack() and HeapTupleHeaderUnpack() to store/extract
header data to/from disk buffers. Concurrency?

Inefficient, and you'd have problems still with the changeable fields
(t_infomask etc).

As usual, the major objection to any such change is losing the chance
of doing pg_upgrade. But we didn't have pg_upgrade during the 7.2
cycle either.

I thought, it is quite common to need pg_dump/restore when upgrading
between releases.

Yes, and we get loud complaints every time we require it...

Anyway, as long as our changes don't make heap tuples larger, it
should be possible to write a tool that converts version x data files
to version y data files. I've done that before (not for PG though)
and I know it's a lot of work, but wouldn't it be great for the PG
marketing department ;-)

I'd be afraid to use a conversion-in-place tool for this sort of thing.
If it crashes halfway through, what then?

regards, tom lane

On Tue, 21 May 2002 11:53:04 -0400, Tom Lane <tgl@sss.pgh.pa.us>
wrote:

The system tables that have OIDs will certainly continue to have OIDs.

That's clear. I should have written: "... rip out oids from tuple
headers of system tables."

Ugh. While certainly we should have been using accessor
macros for that, I'm not sure I want to try to change it.

I already did this for xmin, xmax, cmin, cmax, and xvac (see my patch
posted 2002-05-12).

If OID is made to be the last fixed-offset field, instead of the first,

That would introduce some padding.

then this approach would be fairly workable. Actually I'd still use
just one struct definition, but do offsetof() calculations to decide
where the null-bitmap starts.

... and for calculating the tuple header size.

Decouple on-disk format from in-memory structures, use
HeapTupleHeaderPack() and HeapTupleHeaderUnpack() to store/extract
header data to/from disk buffers. Concurrency?

Inefficient,

Just to be sure: You mean the CPU cycles wasted by Pack() and
Unpack()?

I'd be afraid to use a conversion-in-place tool for this sort of thing.

Me too. No, not in place! I thought of a filter reading an old
format data file, one page at a time, and writing a new format data
file. This would work as long as the conversions don't cause page
overflow.

No comment on a planned 7.3 timeframe? :-(

Servus
Manfred

Manfred Koizar <mkoi-pg@aon.at> writes:

No comment on a planned 7.3 timeframe? :-(

I think we are planning to go beta in late summer (end of August, say).
Probably in July we'll start pressing people to finish up any major
development items, or admit that they won't happen for 7.3. So we've
still got a couple months of full-tilt development mode before we
start to worry about tightening up for release.

regards, tom lane

Tom Lane wrote:

Manfred Koizar <mkoi-pg@aon.at> writes:

No comment on a planned 7.3 timeframe? :-(

I think we are planning to go beta in late summer (end of August, say).
Probably in July we'll start pressing people to finish up any major
development items, or admit that they won't happen for 7.3. So we've
still got a couple months of full-tilt development mode before we
start to worry about tightening up for release.

I am concerned about slowing down too early. We did that in previous
releases and didn't get the beta focus we needed, and it was too
paralyzing on people to know what is to be slowed and what to keep
going. I think a slowdown two weeks before beta would be fine.

Basically, I think the slowdown lengthened the time we were not doing
something productive.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tom Lane wrote:

Manfred Koizar <mkoi-pg@aon.at> writes:

what about WITHOUT OIDS? I know dropping the OID from some tables and
keeping it for others is not trivial, because t_oid is the _first_
field of HeapTupleHeaderData. I'm vaguely considering a few possible
implementations and will invest more work in a detailed proposal, if
it's wanted.

Yeah, I had been toying with the notion of treating OID like a user
field --- ie, it'd be present in the variable-length part of the record
if at all. It'd be a bit tricky to find all the places that would need
to change, but I think there are not all that many.

As usual, the major objection to any such change is losing the chance
of doing pg_upgrade. But we didn't have pg_upgrade during the 7.2
cycle either. If we put together several such changes and did them
all at once, the benefit might be enough to overcome that complaint.

I think it is inevitable that there be enough binary file changes the
pg_upgrade will not work for 7.3 --- it just seems it is only a matter
of time.

One idea is to allow alternate page layouts using the heap page version
number, but that will be difficult/confusing in the code.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

On Fri, 7 Jun 2002, Bruce Momjian wrote:

Tom Lane wrote:

Manfred Koizar <mkoi-pg@aon.at> writes:

No comment on a planned 7.3 timeframe? :-(

I think we are planning to go beta in late summer (end of August, say).
Probably in July we'll start pressing people to finish up any major
development items, or admit that they won't happen for 7.3. So we've
still got a couple months of full-tilt development mode before we
start to worry about tightening up for release.

I am concerned about slowing down too early. We did that in previous
releases and didn't get the beta focus we needed, and it was too
paralyzing on people to know what is to be slowed and what to keep
going. I think a slowdown two weeks before beta would be fine.

Other then personal slow downs, there is no reason for anything to "slow
down" until beta itself starts ...

Marc G. Fournier wrote:

On Fri, 7 Jun 2002, Bruce Momjian wrote:

Tom Lane wrote:

Manfred Koizar <mkoi-pg@aon.at> writes:

No comment on a planned 7.3 timeframe? :-(

I think we are planning to go beta in late summer (end of August, say).
Probably in July we'll start pressing people to finish up any major
development items, or admit that they won't happen for 7.3. So we've
still got a couple months of full-tilt development mode before we
start to worry about tightening up for release.

I am concerned about slowing down too early. We did that in previous
releases and didn't get the beta focus we needed, and it was too
paralyzing on people to know what is to be slowed and what to keep
going. I think a slowdown two weeks before beta would be fine.

Other then personal slow downs, there is no reason for anything to "slow
down" until beta itself starts ...

I assume Tom doesn't want a huge patch applied the day before beta, and
I can understand that, but patch problems usually appear within two
weeks of application, so I think we only have to worry about those last
two weeks. Of course, another option is to continue in full development
until the end of August, then start beta in September as soon as the
code is stable.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

On Fri, 7 Jun 2002, Bruce Momjian wrote:

Marc G. Fournier wrote:

On Fri, 7 Jun 2002, Bruce Momjian wrote:

Tom Lane wrote:

Manfred Koizar <mkoi-pg@aon.at> writes:

No comment on a planned 7.3 timeframe? :-(

I think we are planning to go beta in late summer (end of August, say).
Probably in July we'll start pressing people to finish up any major
development items, or admit that they won't happen for 7.3. So we've
still got a couple months of full-tilt development mode before we
start to worry about tightening up for release.

I am concerned about slowing down too early. We did that in previous
releases and didn't get the beta focus we needed, and it was too
paralyzing on people to know what is to be slowed and what to keep
going. I think a slowdown two weeks before beta would be fine.

Other then personal slow downs, there is no reason for anything to "slow
down" until beta itself starts ...

I assume Tom doesn't want a huge patch applied the day before beta, and

No offence to Tom, but who cares whether Tom wants a huge patch or not?

I can understand that, but patch problems usually appear within two
weeks of application, so I think we only have to worry about those last
two weeks. Of course, another option is to continue in full development
until the end of August, then start beta in September as soon as the
code is stable.

That kinda was the plan ... code will be frozen around the 1st of
September, with a release packaged then that will be label'd beta1 ...
regardless of how stable it is ... beta is "we're not taking any more
changes except fixes, so pound on this and let us know what needs to be
fixed" ... there shouldn't be any 'lead up' to beta, the lead up is the
development period itself ... *shrug*

Marc G. Fournier wrote:

On Fri, 7 Jun 2002, Bruce Momjian wrote:

I am concerned about slowing down too early. We did that in previous
releases and didn't get the beta focus we needed, and it was too
paralyzing on people to know what is to be slowed and what to keep
going. I think a slowdown two weeks before beta would be fine.

Other then personal slow downs, there is no reason for anything to "slow
down" until beta itself starts ...

I assume Tom doesn't want a huge patch applied the day before beta, and

No offence to Tom, but who cares whether Tom wants a huge patch or not?

Well, he does clean up lots of breakage, so I was willing to settle on a
compromise of 2 weeks. However, if other feel we should just go
full-bore until beta starts, I am fine with that.

I will say that I was disapointed by previous release delays and will be
more vocal about moving things forward than I have in the past.

I can understand that, but patch problems usually appear within two
weeks of application, so I think we only have to worry about those last
two weeks. Of course, another option is to continue in full development
until the end of August, then start beta in September as soon as the
code is stable.

That kinda was the plan ... code will be frozen around the 1st of
September, with a release packaged then that will be label'd beta1 ...
regardless of how stable it is ... beta is "we're not taking any more
changes except fixes, so pound on this and let us know what needs to be
fixed" ... there shouldn't be any 'lead up' to beta, the lead up is the
development period itself ... *shrug*

Agreed.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

On Fri, 7 Jun 2002, Bruce Momjian wrote:

I will say that I was disapointed by previous release delays and will be
more vocal about moving things forward than I have in the past.

I don't know ... I kinda like being able to confidently say to clients
that "the latest release is always the most stable/reliable" and not being
burned by it :) Hell, last release, I started moving to v7.2 on some of
our production servers around beta4 or so, since a) I was confident and b)
I needed some of the features ...

Not many software packages you can say that about anymore, eh?

Marc G. Fournier wrote:

On Fri, 7 Jun 2002, Bruce Momjian wrote:

I will say that I was disappointed by previous release delays and will be
more vocal about moving things forward than I have in the past.

I don't know ... I kinda like being able to confidently say to clients
that "the latest release is always the most stable/reliable" and not being
burned by it :) Hell, last release, I started moving to v7.2 on some of
our production servers around beta4 or so, since a) I was confident and b)
I needed some of the features ...

Not many software packages you can say that about anymore, eh?

Yes, true, I am not going to push the schedule. What I want to try to
do is keep people more informed of where we are in the release process,
and more informed about the open issues involved.

That squishy freeze is one of those items that is very hard to organize
because it is so arbitrary about what should be added. I think we need
to communicate CODE-CODE-CODE until September 1, then communicate
TEST-TEST-TEST after that. We will not release before it is ready, but
we will marshall our forces better.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

On Fri, 7 Jun 2002 02:01:40 -0400 (EDT), Bruce Momjian
<pgman@candle.pha.pa.us> wrote:

I think it is inevitable that there be enough binary file changes the
pg_upgrade will not work for 7.3 --- it just seems it is only a matter
of time.

As far as it concerns changes proposed by me, I'll (try to) provide a
conversion program, if that's necessary for my patches to be accepted.
Then move_objfiles() in pg_update would have to call pg_convert, or
whatever we call it, instead of mv. And yes, users would need twice
the disk space during pg_upgrade.

One idea is to allow alternate page layouts using the heap page version
number, but that will be difficult/confusing in the code.

This is a good idea, IMHO. By saying "*the* heap page version number"
do you mean, that there already is such a number by now? I could not
find one in bufpage.h. Should I have looked somewhere else?

While we're at it, does each file start with a magic number
identifying its type? AFAICS nbtree does; but I can't tell for heap
and have not yet checked for rtree, gist, ... This is the reason for
the "try to" in the first paragraph.

Servus
Manfred