Detoasting and memory usage
Hi,
I discovered a slightly misleading (or at least non-intuitive to
me) behaviour of the pg_detoast_datum call while processing variable length
objects. I was reading a large number of text fields on fetched
results and noticed some strange memory behavior. I was seeing that in
some situations, memory usage would shoot up dramatically, while in
others it would not. Puzzled, I tracked it down my usage of
DatumGetTextP(), which eventually reduces to pg_detoast_datum. As it turns
out, most of the text fields I was processing were small enough not to
be toasted and everything was OK. For the larger ones, pg_detoast_datum
would give me a palloced copy, and since I was making all these calls in
the same memory context and not pfreeing the results, I had a memory
leak. Just pfreeing the result would obviously not work since
pg_detoast_datum doesn't return a palloced result in most cases (i.e.
smaller varlength objects). I ended up using DatumGetTextPCopy (to
ensure that the result was palloced) with a pfree and all problems were
solved.
I guess my situation is an odd fringe case that probably doesn't happen
too much, but I thought that knowledge of it might be useful to someone
somewhere.
-Aaron
apb18@cornell.edu wrote:
I guess my situation is an odd fringe case that probably doesn't happen
too much, but I thought that knowledge of it might be useful to someone
somewhere.
See PG_FREE_IF_COPY in fmgr.h
/*
* Support for cleaning up detoasted copies of inputs. This must only
* be used for pass-by-ref datatypes, and normally would only be used
* for toastable types. If the given pointer is different from the
* original argument, assume it's a palloc'd detoasted copy, and pfree
* it.
*
* NOTE: most functions on toastable types do not have to worry about
* this, but we currently require that support functions for indexes
* not leak memory.
*/
#define PG_FREE_IF_COPY(ptr,n) \
do { \
if ((Pointer) (ptr) != PG_GETARG_POINTER(n)) \
pfree(ptr); \
} while (0)
Maybe you can use that or do something similar?
HTH,
Joe
Hmm.. I can't believe I missed that. While that exact macro wouldn't
apply in my situation (because I fetch the value from a field in a tuple and
not the argument of a function), the underlying concept is the same as
what I would need. Now I do not need to feel the wrath of unnecessary
memory allocation/deallocation for the non-toasted case. Excellent.
Thanks very much!
-Aaron
Show quoted text
#define PG_FREE_IF_COPY(ptr,n) \
do { \
if ((Pointer) (ptr) != PG_GETARG_POINTER(n)) \
pfree(ptr); \
} while (0)Maybe you can use that or do something similar?
HTH,
Joe