Memory management with C aggregate

Ian Burrell <imb@rentrak.com> writes:

We have user-defined aggregates written in C running inside the server.
We are running into some memory management issues and wondering what
is the best way to solve the problem.

The state of the aggregates is a structure with a pointer to allocated
memory. The structure and memory are allocated in the
PortalMemoryContext and freed in the ffunc. We just discovered that the
ffunc function can be called multiple times with the same state on
certain queries. The double free causes a crash.

AFAICS the ffunc should be called only once per aggregated group.
Perhaps your code is confused about the handling of groups? If so,
the double free is hardly your only worry --- you'll be computing wrong
answers anyway till you fix that.

From what I could find, the proper way to handle this is let the memory
be freed when the memory context is freed. What is the right memory
context to be used to allocate the aggregate state?

I'd use the memory context identified by fcinfo->flinfo->fn_mcxt.

regards, tom lane

Tom Lane wrote:

Ian Burrell <imb@rentrak.com> writes:

We have user-defined aggregates written in C running inside the server.
We are running into some memory management issues and wondering what
is the best way to solve the problem.

The state of the aggregates is a structure with a pointer to allocated
memory. The structure and memory are allocated in the
PortalMemoryContext and freed in the ffunc. We just discovered that the
ffunc function can be called multiple times with the same state on
certain queries. The double free causes a crash.

AFAICS the ffunc should be called only once per aggregated group.
Perhaps your code is confused about the handling of groups? If so,
the double free is hardly your only worry --- you'll be computing wrong
answers anyway till you fix that.

The aggregate is in a subquery which might make a difference. The ffunc
is only called multiple times when a nested loop is used in the query
plan. With other query plans, the ffunc is only called once. The ffunc
seems to be called once for every combination.

I can't get the following query to use a nested loop, but our query is
similar, but with a compound key and a custom aggregate. If x has N
rows, y has M, with N groups, then the ffunc can be called N*M times, M
times for each N group.

SELECT x.key_no, s.agg
FROM x
INNER JOIN (
SELECT y.key_no, custom_agg(num) AS agg
FROM y
GROUP BY key_no
) s
USING (key_no)
ORDER BY key_no

I'll try to come up with a test case that illustrates the problem.

- Ian