Invalid memory alloc
Hello, I'm processing a 100Million row table.
I get error message about memory and I'ld like to know what can cause this issue.
...
psql:/home/ubuntu/create_topo.sql:12: NOTICE: 104855000 edges processed
psql:/home/ubuntu/create_topo.sql:12: NOTICE: 104856000 edges processed
psql:/home/ubuntu/create_topo.sql:12: NOTICE: 104857000 edges processed
psql:/home/ubuntu/create_topo.sql:12: NOTICE: invalid memory alloc request size 1677721600
psql:/home/ubuntu/create_topo.sql:12: NOTICE: UPDATE public.way_noded SET source = 88374866,target = 88362922 WHERE id = 142645362
pgr_createtopology
--------------------
FAIL
(1 row)
The server has a 10Gb of shared_buffer.
Do you thing this quantity of memory allowed should normaly be enough to process the data?
Thanks
Marc
--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
On 23-04-2015 16:55, Marc-André Goderre wrote:
Hello, I'm processing a 100Million row table.
I get error message about memory and I'ld like to know what can cause this issue....
psql:/home/ubuntu/create_topo.sql:12: NOTICE: 104855000 edges processed
psql:/home/ubuntu/create_topo.sql:12: NOTICE: 104856000 edges processed
psql:/home/ubuntu/create_topo.sql:12: NOTICE: 104857000 edges processed
psql:/home/ubuntu/create_topo.sql:12: NOTICE: invalid memory alloc request size 1677721600
psql:/home/ubuntu/create_topo.sql:12: NOTICE: UPDATE public.way_noded SET source = 88374866,target = 88362922 WHERE id = 142645362
pgr_createtopology
--------------------
FAIL
(1 row)The server has a 10Gb of shared_buffer.
Do you thing this quantity of memory allowed should normaly be enough to process the data?Thanks
Marc
My question would sound stupid... you have 10Gb shared buffer, but how
much physical memory on this server?
How have you configured the kernel swappines, overcommit_memoryt,
overcommit_ratio?
Have you set anything different in shmmax or shmall?
Edson
--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
Hello, I'm processing a 100Million row table.
I get error message about memory and I'ld like to know what can cause this issue....
psql:/home/ubuntu/create_topo.sql:12: NOTICE: 104855000 edges processed
psql:/home/ubuntu/create_topo.sql:12: NOTICE: 104856000 edges processed
psql:/home/ubuntu/create_topo.sql:12: NOTICE: 104857000 edges processed
psql:/home/ubuntu/create_topo.sql:12: NOTICE: invalid memory alloc request size 1677721600
psql:/home/ubuntu/create_topo.sql:12: NOTICE: UPDATE public.way_noded SET source = 88374866,target = 88362922 WHERE id = 142645362
pgr_createtopology
--------------------
FAIL
(1 row)The server has a 10Gb of shared_buffer.
Do you thing this quantity of memory allowed should normaly be enough to process the data?Thanks
Marc
Hi,
what version of Postgres are you using? Any extensions? I guess you're using PostGIS, right?
This error indicates something is trying to allocate 1600 MB of memory in the backend - that
should never happen, as data chunks that are larger than 1 GB are broken down in smaller pieces.
I hope you're not suffering data corruption, what happens if you do select * from public.way_noded
where id = 142645362 ?
Any other hints? Log messages (from Linux or the postgres backend)?
Bye,
Chris.
--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
On 4/23/15 3:15 PM, Edson Richter wrote:
My question would sound stupid... you have 10Gb shared buffer, but how
much physical memory on this server?
How have you configured the kernel swappines, overcommit_memoryt,
overcommit_ratio?
Have you set anything different in shmmax or shmall?
I don't think this is an OS thing at all. That error messages is pretty
associated with this code:
#define MaxAllocSize ((Size) 0x3fffffff) /* 1 gigabyte - 1 */
#define AllocSizeIsValid(size) ((Size) (size) <= MaxAllocSize)
If malloc failed you'd get an actual out of memory error, not a notice.
We need more information from the OP about what they're doing.
--
Jim Nasby, Data Architect, Blue Treble Consulting
Data in Trouble? Get it in Treble! http://BlueTreble.com
--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
Jim Nasby <Jim.Nasby@BlueTreble.com> writes:
We need more information from the OP about what they're doing.
Yeah. Those NOTICEs about "nnn edges processed" are not coming out of
anything in core Postgres; I'll bet whatever is producing those is at
fault (by trying to palloc indefinitely-large amounts of memory as a
single chunk).
regards, tom lane
--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
Postgresql is running on a 30 Gb of memory server.
I should have access to the server in few minutes and I'll give you more information about the other spec.
Thanks
Marc
-----Message d'origine-----
De : pgsql-general-owner@postgresql.org [mailto:pgsql-general-owner@postgresql.org] De la part de Edson Richter
Envoyé : 23 avril 2015 16:16
À : pgsql-general@postgresql.org
Objet : Re: [GENERAL] Invalid memory alloc
On 23-04-2015 16:55, Marc-André Goderre wrote:
Hello, I'm processing a 100Million row table.
I get error message about memory and I'ld like to know what can cause this issue....
psql:/home/ubuntu/create_topo.sql:12: NOTICE: 104855000 edges
processed
psql:/home/ubuntu/create_topo.sql:12: NOTICE: 104856000 edges
processed
psql:/home/ubuntu/create_topo.sql:12: NOTICE: 104857000 edges
processed
psql:/home/ubuntu/create_topo.sql:12: NOTICE: invalid memory alloc
request size 1677721600
psql:/home/ubuntu/create_topo.sql:12: NOTICE: UPDATE public.way_noded SET source = 88374866,target = 88362922 WHERE id = 142645362
pgr_createtopology
--------------------
FAIL
(1 row)The server has a 10Gb of shared_buffer.
Do you thing this quantity of memory allowed should normaly be enough to process the data?Thanks
Marc
My question would sound stupid... you have 10Gb shared buffer, but how much physical memory on this server?
How have you configured the kernel swappines, overcommit_memoryt, overcommit_ratio?
Have you set anything different in shmmax or shmall?
Edson
--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
Server setting
kernel.shmmax = 13322600448
kernel.shmall = 4194304
vm.swappiness = 60
vm.overcommit_memory = 0
vm.overcommit_ratio = 50
Postgresql setting
effective_cache_size | 128MB
checkpoint_segments | 64
maintenance_work_mem | 5GB
segment_size | 1GB
shared_buffers | 10GB
work_mem | 256MB
I use Postgis and PGrouting extension.
The error come when I use a pgrouting function pgr_createtopology()
Marc
Marc-André Goderre
Analyste en informatique
Courriel: magoderre@cgq.qc.ca
Tel.: (418) 698-5995 poste 1628
Téléc.: (418) 698-4108
Cégep de Chicoutimi, 534 rue Jacques-Cartier Est, Chicoutimi (Québec) G7H 1Z6
Visitez notre site web au : www.cgq.qc.ca
Suivez-nous sur Twitter : @lecgq
AVIS DE CONFIDENTIALITÉ :
Le présent courriel peut contenir des renseignements confidentiels s’adressant qu’au destinataire ou à celui ont le nom figure ci-dessus. Si ce courriel vous est parvenu par mégarde, veuillez le supprimer et nous en aviser aussitôt par téléphone. Merci.
Pensez à l'environnement avant d'imprimer ce message
-----Message d'origine-----
De : pgsql-general-owner@postgresql.org [mailto:pgsql-general-owner@postgresql.org] De la part de Edson Richter
Envoyé : 23 avril 2015 16:16
À : pgsql-general@postgresql.org
Objet : Re: [GENERAL] Invalid memory alloc
On 23-04-2015 16:55, Marc-André Goderre wrote:
Hello, I'm processing a 100Million row table.
I get error message about memory and I'ld like to know what can cause this issue....
psql:/home/ubuntu/create_topo.sql:12: NOTICE: 104855000 edges
processed
psql:/home/ubuntu/create_topo.sql:12: NOTICE: 104856000 edges
processed
psql:/home/ubuntu/create_topo.sql:12: NOTICE: 104857000 edges
processed
psql:/home/ubuntu/create_topo.sql:12: NOTICE: invalid memory alloc
request size 1677721600
psql:/home/ubuntu/create_topo.sql:12: NOTICE: UPDATE public.way_noded SET source = 88374866,target = 88362922 WHERE id = 142645362
pgr_createtopology
--------------------
FAIL
(1 row)The server has a 10Gb of shared_buffer.
Do you thing this quantity of memory allowed should normaly be enough to process the data?Thanks
Marc
My question would sound stupid... you have 10Gb shared buffer, but how much physical memory on this server?
How have you configured the kernel swappines, overcommit_memoryt, overcommit_ratio?
Have you set anything different in shmmax or shmall?
Edson
--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
I use Postgis and PGrouting extension.
The error come when I use a pgrouting function pgr_createtopology()
It appears pgrouting violates the 1GB per chunk limit in the postgres backend
when processing large datasets:
https://github.com/pgRouting/pgrouting/issues/291
Bye,
Chris.
--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
Can I change the segment size to allow more memory?
Is it a good idea?
The concerned function work only on the entire table then I can't process a part of it.
Should I split the table in multiple table and merge them after the process?
Thanks
Marc
-----Message d'origine-----
De : Chris Mair [mailto:chris@1006.org]
Envoyé : 24 avril 2015 15:41
À : Marc-André Goderre; pgsql-general@postgresql.org
Objet : Re: [GENERAL] Invalid memory alloc
I use Postgis and PGrouting extension.
The error come when I use a pgrouting function pgr_createtopology()
It appears pgrouting violates the 1GB per chunk limit in the postgres backend when processing large datasets:
https://github.com/pgRouting/pgrouting/issues/291
Bye,
Chris.
--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
On 4/27/15 8:45 AM, Marc-André Goderre wrote:
Can I change the segment size to allow more memory?
Is it a good idea?
The concerned function work only on the entire table then I can't process a part of it.
Should I split the table in multiple table and merge them after the process?
Please don't top-post.
I use Postgis and PGrouting extension.
The error come when I use a pgrouting function pgr_createtopology()It appears pgrouting violates the 1GB per chunk limit in the postgres backend when processing large datasets:
Changing the segment size would just push the problem down the road. At
some point the same error will happen.
That issue URL has a comment about "Don't try and process all of Europe
at once, give it a bounding box", so that's one possible solution.
Really the function should be changed so it doesn't trying and palloc
more than 1G in a single go... but OTOH there's only so far you can
probably go there too. I imagine the complexity of what the function is
trying to do grows geometrically with the size of the data set, so you
probably need to find some way to break your data into smaller pieces
and process each piece individually.
--
Jim Nasby, Data Architect, Blue Treble Consulting
Data in Trouble? Get it in Treble! http://BlueTreble.com
--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general