[ERROR] Copy from CSV fails due to memory error.
Hey.
I am facing an issue when I try to run the following command
COPY <table_name> FROM <file> WITH DELIMITER E',’;
This file, is rather large, it's around 178GBs.
When I try to run this COPY command I get the following error:
ERROR: out of memory
DETAIL: Failed on request of size 2048 in memory context "AfterTriggerEvents".
CONTEXT: COPY ssbm300_lineorder, line 50796791
Clearly a memory allocation function is failing but I have no clue how to fix it.
I have tried experimenting with shared_buffers value in postgresql.conf file but after searching a bit I quickly realized that I do not know what I am doing there so I left it with default value. Same with work_mem value.
Did you face this issue before? Can you help me resolve it?
Thanks in advance!
On 1/19/22 14:01, Kostas Chasialis wrote:
Hey.
I am facing an issue when I try to run the following command
COPY <table_name> FROM <file> WITH DELIMITER E',’;
This file, is rather large, it's around 178GBs.
When I try to run this COPY command I get the following error:
ERROR: out of memory
DETAIL: Failed on request of size 2048 in memory context "AfterTriggerEvents".
CONTEXT: COPY ssbm300_lineorder, line 50796791Clearly a memory allocation function is failing but I have no clue how to fix it.
I have tried experimenting with shared_buffers value in postgresql.conf file but after searching a bit I quickly realized that I do not know what I am doing there so I left it with default value. Same with work_mem value.
Did you face this issue before? Can you help me resolve it?
Well, it's clearly related to "after" triggers - do you have anything
such triggers on the table? AFAIK it might be related to deferred
constraints (like unique / foreign keys). Do you have anything like that?
If yes, I guess the only solution is to make the constraints not
deferred or split the copy into smaller chunks.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company