Parallel GiST build on Cube

Started by Shyam Saladiover 5 years ago2 messages
#1Shyam Saladi
saladi@caltech.edu

Hello --

I regularly build GiST indexes on large databases. In recent times, as the
size of the database has ballooned (30 million rows) along with the build
time on a column of points in 3- and 8-dimensional space (0-volume cube).

Is anyone working on (or already implemented) a parallel GiST build on
Cube? If not, I'd like to try contributing this--any pointers from folks
familiar with the backend of Cube? Any input would be great.

Thanks,
Shyam

--
Shyam Saladi <http://shyam.saladi.org&gt;
NSF Graduate Research Fellow - Clemons Lab
Biochemistry and Molecular Biophysics
California Institute of Technology

In reply to: Shyam Saladi (#1)
Re: Parallel GiST build on Cube

Hello,

These things for GIST I know that can help:
- Fast sorting GIST build commitfest entry by Andrey Borodin, not parallel
but faster -
/messages/by-id/1A36620E-CAD8-4267-9067-FB31385E7C0D@yandex-team.ru

- Fast sorting GIST build by Nikita Glukhov, reuses btree code so also
parallel -
https://github.com/postgres/postgres/compare/master...glukhovn:gist_btree_build

- "Choose Subtree" routine is needed, as current "penalty" is very
inefficient -
/messages/by-id/CAPpHfdssv2i7CXTBfiyR6=_A3tp19+iLo-pkkfD6Guzs2-tvEw@mail.gmail.com

These are very wanted for PostGIS which also indexes everything by 2-4
dimensional cubes and require improvements in core infrastructure and
opclass.

On Mon, Apr 27, 2020 at 8:57 PM Shyam Saladi <saladi@caltech.edu> wrote:

Hello --

I regularly build GiST indexes on large databases. In recent times, as the
size of the database has ballooned (30 million rows) along with the build
time on a column of points in 3- and 8-dimensional space (0-volume cube).

Is anyone working on (or already implemented) a parallel GiST build on
Cube? If not, I'd like to try contributing this--any pointers from folks
familiar with the backend of Cube? Any input would be great.

Thanks,
Shyam

--
Shyam Saladi <http://shyam.saladi.org&gt;
NSF Graduate Research Fellow - Clemons Lab
Biochemistry and Molecular Biophysics
California Institute of Technology

--
Darafei Praliaskouski
Support me: http://patreon.com/komzpa