BuildTupleFromCStrings Memory Documentation?

Started by Jason Petersenover 10 years ago2 messages
#1Jason Petersen
jason@citusdata.com
1 attachment(s)

Within the core codebase, BuildTupleFromCStrings is often called within a temporary memory context cleared after the call. In dblink.c, this is justified as being needed to “[clean up] not only the data we have direct access to, but any cruft the I/O functions might leak”.

I wrote a pretty minimal case to call BuildTupleFromCStrings in a loop (attached) and found that I was using 40GB of RAM in a few minutes, though I was not allocating any memory myself and immediately freed the tuple it returned.

Is the need to wrap this call in a protective context documented anywhere? Portions of the documentation use BuildTupleFromCStrings in examples without mentioning this precaution. Is it just well-known, or did I miss a README or comment somewhere?

--
Jason Petersen
Software Engineer | Citus Data
303.736.9255
jason@citusdata.com

Attachments:

tup_memleak.capplication/octet-stream; name=tup_memleak.cDownload
#2Tom Lane
tgl@sss.pgh.pa.us
In reply to: Jason Petersen (#1)
Re: BuildTupleFromCStrings Memory Documentation?

Jason Petersen <jason@citusdata.com> writes:

Within the core codebase, BuildTupleFromCStrings is often called within a temporary memory context cleared after the call. In dblink.c, this is justified as being needed to “[clean up] not only the data we have direct access to, but any cruft the I/O functions might leak”.
I wrote a pretty minimal case to call BuildTupleFromCStrings in a loop (attached) and found that I was using 40GB of RAM in a few minutes, though I was not allocating any memory myself and immediately freed the tuple it returned.

Is the need to wrap this call in a protective context documented anywhere? Portions of the documentation use BuildTupleFromCStrings in examples without mentioning this precaution. Is it just well-known, or did I miss a README or comment somewhere?

Most uses of BuildTupleFromCStrings only do it once per invocation of the
calling function, so that an outer-level reset of the calling function's
evaluation context is what takes care of any memory leaked by the I/O
functions BuildTupleFromCStrings invokes. If you intend to call it many
times within the same function call then you should use a context you can
reset between calls. This risk is hardly unique to BuildTupleFromCStrings,
which is why the documentation doesn't make a big point of it.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers