Inserting millions of record in a partitioned Table
Hi guys,
with Postgresql 9.6.1 we need to insert, from a certain select query, some millions of record in a partitioned table.
The table is partitioned by day.
The datas we will import can, often, be mixed between two different days.
We noticed that if we import directly into the global table it is really, really slow.
Importing directly in the single partition is faster.
Any suggestions here please?
Thank you, best!
F
--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
On Wed, Sep 20, 2017 at 10:10 AM, Job <Job@colliniconsulting.it> wrote:
We noticed that if we import directly into the global table it is really,
really slow.
Importing directly in the single partition is faster.
Do you have a rule or trigger on the main table to redirect to the
partitions? You should expect that to take some extra time *per row*. Your
best bet is to just import into the proper partition and make sure your
application produces batch files that align with your partitions.
Either that or write a program that reads the data, determines the
partition, and then inserts directly to it. It might be faster.
On 09/20/2017 02:46 PM, Vick Khera wrote:
On Wed, Sep 20, 2017 at 10:10 AM, Job <Job@colliniconsulting.it
<mailto:Job@colliniconsulting.it>> wrote:We noticed that if we import directly into the global table it is
really, really slow.
Importing directly in the single partition is faster.Do you have a rule or trigger on the main table to redirect to the
partitions? You should expect that to take some extra time *per row*.
Your best bet is to just import into the proper partition and make
sure your application produces batch files that align with your
partitions.Either that or write a program that reads the data, determines the
partition, and then inserts directly to it. It might be faster.
I wonder if this is a case of hurry up and wait. A script which could
load say 10 records, and assuming that takes much less than one second,
run once per second (waiting 1000 - runtime ms) would by now have done
about a million records since the question was asked.