Number of buckets/partitions of dshash

Hello.

At Fri, 12 Oct 2018 06:19:12 +0000, "Ideriha, Takeshi" <ideriha.takeshi@jp.fujitsu.com> wrote in <4E72940DA2BF16479384A86D54D0988A6F1C1779@G01JPEXMBKW04>

Hi, let me clarify my understanding about the $title.

It seems that the number of hash partitions is fixed at 128 in dshash and
right now we cannot change it unless dshash.c code itself is changed, right?

According to the comment of dshash.c, DSHASH_NUM_PARTITIONS could be runtime parameter in future.
Are there someone working on it?
(I want to do it, but TBH I cannot promise it because of some other work.)

In my current development for allocating catalog cache on the shared memory[1]
I'm trying to put hash table of each CatCache to the shared memory using dshash.
The number of buckets for CatCache is pre-defined by cacheinfo and most of them is under 128 like 8 or 16.
This would cause some waste of memory on DSA because some partitions (buckets) is allocated but not used.

So I'm thinking that current dshash design is still ok but flexible size of partition is appropriate
for some use cases like mine.

Do you have any thoughts?

We could do that easily, but shouldn't we find a way to reduce or
eliminate the impact of locking first? dshash needs to hold
partition lock while the caller is examining a returned entry.

[1] /messages/by-id/4E72940DA2BF16479384A86D54D0988A567B9245@G01JPEXMBKW04

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center

From: Kyotaro HORIGUCHI [mailto:horiguchi.kyotaro@lab.ntt.co.jp]

This would cause some waste of memory on DSA because some partitions (buckets)

is allocated but not used.

So I'm thinking that current dshash design is still ok but flexible
size of partition is appropriate for some use cases like mine.

Do you have any thoughts?

We could do that easily, but shouldn't we find a way to reduce or eliminate the impact
of locking first? dshash needs to hold partition lock while the caller is examining a
returned entry.

Thanks for the comment.
I agreed.
It would take a long time to achieve it but as you've pointed out
finding way to minimize the locking time seems benefit for everyone and first priority.

Regards,
Takeshi Ideriha