psql freezes loading large file
I have a test ".sql" file of the form:
insert into testtable values ('1');
insert into testtable values ('2');
insert into testtable values ('3');
....
100 Million
Right before I call the file with "\i" I do a begin transaction.
At some point during the load the process stops.
After some 5+ minutes of not seeing the inserts go by I pressed CTRL+C and
got the errors:
psql in free(): warning: recursive call
xmalloc: out of virtual memory
xmalloc: out of virtual memory
Tried without calling begin transaction and got the same problem.
Did psql ran out of memory?
When I did it without begin I did a select count(*) from the table and only
472 records got merged. Can't imagine psql running out of memory after
loading such small amount of records.
This is postgresql 8.1.3 Running on FreeBSD 6.1
Francisco Reyes <lists@stringsutils.com> writes:
I have a test ".sql" file of the form:
insert into testtable values ('1');
insert into testtable values ('2');
insert into testtable values ('3');
....
100 Million
I don't have the patience to run this for 10^8 rows, but the test case
I did run showed absolutely zero memory growth in either psql or the
backend. Are you using any nonstandard psql options?
regards, tom lane
Tom Lane writes:
I don't have the patience to run this for 10^8 rows, but the test case
I got suspicious of my 'test' file so I took 1000 rows. That had problems
and pointed out problems with the file.
It seems I had mismatched single quotes.. my guess is that psql got confused
and went into into an infinite loop somewhow. With the 1000 rows sample it
reported the error. I am re-doing the input file.