Memory leakage associated with plperl spi_prepare/spi_freeplan

Started by Tom Lanealmost 13 years ago4 messages
#1Tom Lane
tgl@sss.pgh.pa.us

I looked into the problem described here:
/messages/by-id/5125087D.8090105@deriva.de

The core of the problem is that plperl's plperl_spi_prepare() sets up
input conversion functions for the parameters of a prepared query
using perm_fmgr_info(), which allocates FmgrInfo structs for the
I/O functions in TopMemoryContext and makes their fn_mcxt values point
to TopMemoryContext too. So when domains.c allocates assorted stuff
in fn_mcxt, that stuff lives forever ... but guess what, the prepared
query doesn't. So we have a memory leak on every use of spi_freeplan().

The leak is particularly egregious for this specific test case, where
an 8K-or-so ExprContext is made as a consequence of domain_check_input's
call to CreateStandaloneExprContext(); but it would add up to something
noticeable eventually even with input functions as innocuous as int4in.
Proof is to try this:

create or replace function perlleak(n int) returns void as $$
my ($n) = @_;
while ($n--) {
my $stmt = spi_prepare('select $1', 'int');
spi_freeplan($stmt);
}
$$ language plperl volatile strict;

with repeat count of a million or so.

I'm surprised we've not seen this reported before --- maybe people
don't tend to use spi_freeplan() much in plperl.

I'm inclined to think the right fix is to make a small memory context
for each prepared plan made by plperl_spi_prepare(). The qdesc for it
could be made right in the context (getting rid of the unchecked
malloc's near the top of the function), the FmgrInfos and their
subsidiary data could live there too, and plperl_spi_freeplan could
replace its retail free's with a single MemoryContextDelete.

Not being particularly a plperl user, I don't really want to code and
test this. Any takers?

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#2Alex Hunsaker
badalex@gmail.com
In reply to: Tom Lane (#1)
1 attachment(s)
Re: Memory leakage associated with plperl spi_prepare/spi_freeplan

On Tue, Feb 26, 2013 at 3:56 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

I'm inclined to think the right fix is to make a small memory context
for each prepared plan made by plperl_spi_prepare(). The qdesc for it
could be made right in the context (getting rid of the unchecked
malloc's near the top of the function), the FmgrInfos and their
subsidiary data could live there too, and plperl_spi_freeplan could
replace its retail free's with a single MemoryContextDelete.

Seemed fairly trivial, find the above approach in the attached. I added a
CHECK_FOR_INTERRUPTS() at the top of plperl_spi_prepare(), it was fairly
annoying that I couldn't ctrl+c my way out of test function.

One annonce is it still leaks :-(. I tracked it down and it seemed to stem
from parseTypeString(). I chased down the rabbit hole for a bit, but
nothing jumped out... raw_parser() is a bit of a black box to me. Adding
the seemingly obvious list_free(raw_parsetree_list); or setting the memory
context before parseTypeString() didn't seem to do much.

It would be nice to squish the other leaks due to perm_fmgr_info()... but
this is a start.

Attachments:

plperl_spi_leak.patchapplication/octet-stream; name=plperl_spi_leak.patchDownload
*** a/src/pl/plperl/plperl.c
--- b/src/pl/plperl/plperl.c
***************
*** 189,194 **** typedef struct plperl_query_desc
--- 189,195 ----
  	Oid		   *argtypes;
  	FmgrInfo   *arginfuncs;
  	Oid		   *argtypioparams;
+ 	MemoryContext plan_ctx;
  } plperl_query_desc;
  
  /* hash table entry for query desc	*/
***************
*** 3211,3216 **** plperl_spi_prepare(char *query, int argc, SV **argv)
--- 3212,3218 ----
  {
  	plperl_query_desc *qdesc;
  	plperl_query_entry *hash_entry;
+ 	MemoryContext       plan_ctx;
  	bool		found;
  	SPIPlanPtr	plan;
  	int			i;
***************
*** 3220,3238 **** plperl_spi_prepare(char *query, int argc, SV **argv)
  
  	check_spi_usage_allowed();
  
! 	BeginInternalSubTransaction(NULL);
! 	MemoryContextSwitchTo(oldcontext);
  
  	/************************************************************
  	 * Allocate the new querydesc structure
  	 ************************************************************/
! 	qdesc = (plperl_query_desc *) malloc(sizeof(plperl_query_desc));
! 	MemSet(qdesc, 0, sizeof(plperl_query_desc));
  	snprintf(qdesc->qname, sizeof(qdesc->qname), "%p", qdesc);
  	qdesc->nargs = argc;
! 	qdesc->argtypes = (Oid *) malloc(argc * sizeof(Oid));
! 	qdesc->arginfuncs = (FmgrInfo *) malloc(argc * sizeof(FmgrInfo));
! 	qdesc->argtypioparams = (Oid *) malloc(argc * sizeof(Oid));
  
  	PG_TRY();
  	{
--- 3222,3248 ----
  
  	check_spi_usage_allowed();
  
! 	CHECK_FOR_INTERRUPTS();
  
  	/************************************************************
  	 * Allocate the new querydesc structure
  	 ************************************************************/
! 	plan_ctx = AllocSetContextCreate(TopMemoryContext,
! 								  "PL/Perl spi_prepare cxt",
! 								  ALLOCSET_DEFAULT_MINSIZE,
! 								  ALLOCSET_DEFAULT_INITSIZE,
! 								  ALLOCSET_DEFAULT_MAXSIZE);
! 	MemoryContextSwitchTo(plan_ctx);
! 	qdesc = (plperl_query_desc *) palloc0(sizeof(plperl_query_desc));
  	snprintf(qdesc->qname, sizeof(qdesc->qname), "%p", qdesc);
  	qdesc->nargs = argc;
! 	qdesc->argtypes = (Oid *) palloc(argc * sizeof(Oid));
! 	qdesc->arginfuncs = (FmgrInfo *) palloc(argc * sizeof(FmgrInfo));
! 	qdesc->argtypioparams = (Oid *) palloc(argc * sizeof(Oid));
! 	qdesc->plan_ctx = plan_ctx;
! 
! 	BeginInternalSubTransaction(NULL);
! 	MemoryContextSwitchTo(oldcontext);
  
  	PG_TRY();
  	{
***************
*** 3256,3262 **** plperl_spi_prepare(char *query, int argc, SV **argv)
  			getTypeInputInfo(typId, &typInput, &typIOParam);
  
  			qdesc->argtypes[i] = typId;
! 			perm_fmgr_info(typInput, &(qdesc->arginfuncs[i]));
  			qdesc->argtypioparams[i] = typIOParam;
  		}
  
--- 3266,3272 ----
  			getTypeInputInfo(typId, &typInput, &typIOParam);
  
  			qdesc->argtypes[i] = typId;
! 			fmgr_info_cxt(typInput, &(qdesc->arginfuncs[i]), plan_ctx);
  			qdesc->argtypioparams[i] = typIOParam;
  		}
  
***************
*** 3295,3304 **** plperl_spi_prepare(char *query, int argc, SV **argv)
  	{
  		ErrorData  *edata;
  
! 		free(qdesc->argtypes);
! 		free(qdesc->arginfuncs);
! 		free(qdesc->argtypioparams);
! 		free(qdesc);
  
  		/* Save error info */
  		MemoryContextSwitchTo(oldcontext);
--- 3305,3311 ----
  	{
  		ErrorData  *edata;
  
! 		MemoryContextDelete(plan_ctx);
  
  		/* Save error info */
  		MemoryContextSwitchTo(oldcontext);
***************
*** 3329,3335 **** plperl_spi_prepare(char *query, int argc, SV **argv)
  	 * Insert a hashtable entry for the plan and return
  	 * the key to the caller.
  	 ************************************************************/
- 
  	hash_entry = hash_search(plperl_active_interp->query_hash, qdesc->qname,
  							 HASH_ENTER, &found);
  	hash_entry->query_data = qdesc;
--- 3336,3341 ----
***************
*** 3619,3630 **** plperl_spi_freeplan(char *query)
  	hash_entry = hash_search(plperl_active_interp->query_hash, query,
  							 HASH_FIND, NULL);
  	if (hash_entry == NULL)
! 		elog(ERROR, "spi_exec_prepared: Invalid prepared query passed");
  
  	qdesc = hash_entry->query_data;
  
  	if (qdesc == NULL)
! 		elog(ERROR, "spi_exec_freeplan: panic - plperl query_hash value vanished");
  
  	/*
  	 * free all memory before SPI_freeplan, so if it dies, nothing will be
--- 3625,3636 ----
  	hash_entry = hash_search(plperl_active_interp->query_hash, query,
  							 HASH_FIND, NULL);
  	if (hash_entry == NULL)
! 		elog(ERROR, "spi_freeplan: Invalid prepared query passed");
  
  	qdesc = hash_entry->query_data;
  
  	if (qdesc == NULL)
! 		elog(ERROR, "spi_freeplan: panic - plperl query_hash value vanished");
  
  	/*
  	 * free all memory before SPI_freeplan, so if it dies, nothing will be
***************
*** 3634,3643 **** plperl_spi_freeplan(char *query)
  				HASH_REMOVE, NULL);
  
  	plan = qdesc->plan;
! 	free(qdesc->argtypes);
! 	free(qdesc->arginfuncs);
! 	free(qdesc->argtypioparams);
! 	free(qdesc);
  
  	SPI_freeplan(plan);
  }
--- 3640,3646 ----
  				HASH_REMOVE, NULL);
  
  	plan = qdesc->plan;
! 	MemoryContextDelete(qdesc->plan_ctx);
  
  	SPI_freeplan(plan);
  }
#3Tom Lane
tgl@sss.pgh.pa.us
In reply to: Alex Hunsaker (#2)
Re: Memory leakage associated with plperl spi_prepare/spi_freeplan

Alex Hunsaker <badalex@gmail.com> writes:

On Tue, Feb 26, 2013 at 3:56 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

I'm inclined to think the right fix is to make a small memory context
for each prepared plan made by plperl_spi_prepare(). The qdesc for it
could be made right in the context (getting rid of the unchecked
malloc's near the top of the function), the FmgrInfos and their
subsidiary data could live there too, and plperl_spi_freeplan could
replace its retail free's with a single MemoryContextDelete.

Seemed fairly trivial, find the above approach in the attached.

Applied with some fixes.

I added a
CHECK_FOR_INTERRUPTS() at the top of plperl_spi_prepare(), it was fairly
annoying that I couldn't ctrl+c my way out of test function.

Good idea, but it wasn't safe where it was --- needs to be inside the
PG_TRY(), so as to convert from postgres to perl error handling.

One annonce is it still leaks :-(.

I fixed that, at least for the function-lifespan leakage from
spi_prepare() --- is that what you meant?

It would be nice to squish the other leaks due to perm_fmgr_info()... but
this is a start.

Yeah, I'm sure there's more left to do in the area --- but at least you
can create and free plans without it leaking.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#4Alex Hunsaker
badalex@gmail.com
In reply to: Tom Lane (#3)
Re: Memory leakage associated with plperl spi_prepare/spi_freeplan

On Fri, Mar 1, 2013 at 7:38 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Alex Hunsaker <badalex@gmail.com> writes:

Applied with some fixes.

Thanks! Your version looks much better than mine.

One annonce is it still leaks :-(.

I fixed that, at least for the function-lifespan leakage from
spi_prepare() --- is that what you meant?

Yep, I don't see the leak with your version.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers