Re: memory management suggestion

Started by Bruce Momjianover 25 years ago8 messages

pgman@candle.pha.pa.us

over 25 years ago

Can anyone comment on this work? Karel?

I start detail study of PG's memory management (because, I want remove
prepared query-cache to shmem (more is in my old discussion with Jan)).

I see current code in the aset.c and I found small non-effective memory
usage.

Description:

The postgresql use blocks for allocation. These blocks are inward
organized/split via chunks.

If a palloc() wants memory:
1) try use some chunk in a freelist of chunks
2) try use free space in an actual block
3) if wanted space is large - allocate specific one-block-for-one-chunk
4) if previous options are not possible allocate new block

A problem:

- if use option 4) and already exist (old) block (but space in this block
is less than wanted space) current algorithm _skip_ and not _use_ this small
space in old block. For a detail see the 'else' on line 327 in aset.c.

I test it and average is 8-10b per a block (but max is 1000b) - large is
this space if a palloc() wants bigger space.

A solution:

Create from this non-used residual space chunk and remove it into free
chunk list.

Comments?

Karel

-----> A tested patch (hmm, we are freeze, possible for 7.0.?):
*** aset.orig.c	Thu Apr 13 18:33:45 2000
--- aset.c	Thu Apr 20 18:45:50 2000
***************
*** 323,326 ****
--- 323,346 ----
else
{
+ 			int 	oldfree = set->blocks->endptr - set->blocks->freeptr;
+ 		
+ 			/*
+ 			 * Try create from residual space in block free chunk
+ 			 */
+ 			if (oldfree > MAXALIGN(1) + ALLOC_CHUNKHDRSZ) {
+ 				
+ 				int x_fidx = AllocSetFreeIndex(oldfree - ALLOC_CHUNKHDRSZ );
+ 				
+ 				chunk = (AllocChunk) (set->blocks->freeptr);
+ 				chunk->size = oldfree - ALLOC_CHUNKHDRSZ;
+ 				
+ 				/* put chunk into freelist */
+ 				chunk->aset = (void *) set->freelist[x_fidx];
+ 				set->freelist[x_fidx] = chunk;
+ 				
+ 				/* unset free space in block */                
+ 				set->blocks->freeptr = set->blocks->endptr;
+ 			}
+ 		
/* Get size of prior block */
blksize = set->blocks->endptr - ((char *) set->blocks);

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Import Notes

Reply to msg id not found: Pine.LNX.3.96.1000420164240.9973E-100000@ara.zf.jcu.cz

Bruce Momjian

pgman@candle.pha.pa.us

over 25 years ago

In reply to: Bruce Momjian (#1)

Here is a followup to it.

Karel Zak <zakkr@zf.jcu.cz> writes:

Create from this non-used residual space chunk and remove it into free
chunk list.

There's at least one bug in that code: the minimum acceptable chunk size
is not MAXALIGN(1), but 1 << ALLOC_MINBITS, and after that it goes up by
powers of 2. You are putting the chunk into a freelist based on which
freelist would be used to allocate a request for X amount of space...
but it had better go into a freelist based on being large enough for the
largest request that would be directed to that freelist, instead. As is,
the chunk could be handed out to someone who would scribble past the
allocated end of the block.

-----> A tested patch (hmm, we are freeze, possible for 7.0.?):

It's getting pretty late in the cycle for this sort of thing.
We should consider it for 7.1, after you get it right...

regards, tom lane

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Import Notes

Reply to msg id not found: 8223.956253496@sss.pgh.pa.us | Resolved by subject fallback

Tom Lane

tgl@sss.pgh.pa.us

over 25 years ago

In reply to: Bruce Momjian (#1)

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Can anyone comment on this work? Karel?

I thought it was a good idea, except for the one possible bug about
whether small wasted chunks go into the right freelist or not.

I think Karel submitted an updated version of the patch later, no?

regards, tom lane

Bruce Momjian

pgman@candle.pha.pa.us

over 25 years ago

In reply to: Tom Lane (#3)

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Can anyone comment on this work? Karel?

I thought it was a good idea, except for the one possible bug about
whether small wasted chunks go into the right freelist or not.

I think Karel submitted an updated version of the patch later, no?

I see a patch, and your comment on it, but nothing after that.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Karel Zak

zakkr@zf.jcu.cz

over 25 years ago

In reply to: Bruce Momjian (#1)

Can anyone comment on this work? Karel?

Yes. Tom & Jan already comment it. I prepare it for 7.1 with
Tom's advices.

Karel

Bruce Momjian

pgman@candle.pha.pa.us

over 25 years ago

In reply to: Karel Zak (#5)

Can anyone comment on this work? Karel?

Yes. Tom & Jan already comment it. I prepare it for 7.1 with
Tom's advices.

Is it already applied to 7.1?

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Karel Zak

zakkr@zf.jcu.cz

over 25 years ago

In reply to: Bruce Momjian (#6)

On Tue, 13 Jun 2000, Bruce Momjian wrote:

Can anyone comment on this work? Karel?

Yes. Tom & Jan already comment it. I prepare it for 7.1 with
Tom's advices.

Is it already applied to 7.1?

No. I said it bad, I want say "I *will* prepare..." :-)

Well, I write it next week. Now I work on contrib/pg_lodump and the
others things...

Bruce, (it is probably already discussed, but refresh me..) for what time
is planned 7.1?

Karel

Bruce Momjian

pgman@candle.pha.pa.us

over 25 years ago

In reply to: Karel Zak (#7)

On Tue, 13 Jun 2000, Bruce Momjian wrote:

Can anyone comment on this work? Karel?

Yes. Tom & Jan already comment it. I prepare it for 7.1 with
Tom's advices.

Is it already applied to 7.1?

No. I said it bad, I want say "I *will* prepare..." :-)

Well, I write it next week. Now I work on contrib/pg_lodump and the
others things...

Oh, OK. Just checking.

Bruce, (it is probably already discussed, but refresh me..) for what time
is planned 7.1?

August.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026