Retrieve memory size allocated by libpq
Hello,
I would like to be able to retrieve the size of memory internally
allocated by libpq for a result. The reason is that we have a Ruby
wrapper that exposes libpq in Ruby. The problem is that Ruby's GC
doesn't know how much memory has been allocated by libpq, so no pressure
is applied to the GC when it should be. With this function we could
instruct the GC about the memory usage associated to each result object.
This issue has already been discussed in the following thread, with the
request to use custom malloc/realloc/free functions:
/messages/by-id/20170828172834.GA71455@TC.local
Retrieving the allocated memory size is another approach to solve the
same base issue. However since the relation between memory consumption
and the particular result object is maintained, it can additionally be
used to provide diagnostic information to each object.
What do you think about adding such a function?
--
Kind Regards,
Lars
Attachments:
0001-libpq-Add-function-PQresultMemsize.patchtext/x-patch; name=0001-libpq-Add-function-PQresultMemsize.patchDownload
From d3ac8089a1b8c26d29d8d8e93c48a892cec75e53 Mon Sep 17 00:00:00 2001
From: Lars Kanis <lars@greiz-reinsdorf.de>
Date: Sat, 23 Jun 2018 19:34:11 +0200
Subject: [PATCH] libpq: Add function PQresultMemsize()
This function retrieves the number of bytes allocated for a given result.
That can be used to instruct the GC about the memory consumed behind a
wrapping object and for diagnosing memory consumtion.
This is an alternative approach to customizable malloc/realloc/free
functions as discussed here:
https://www.postgresql.org/message-id/flat/20170828172834.GA71455%40TC.local#20170828172834.GA71455@TC.local
---
src/interfaces/libpq/exports.txt | 1 +
src/interfaces/libpq/fe-exec.c | 14 +++++++++++++-
src/interfaces/libpq/libpq-fe.h | 1 +
src/interfaces/libpq/libpq-int.h | 2 ++
4 files changed, 17 insertions(+), 1 deletion(-)
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index d6a38d0df8..0b50dddbb7 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -172,3 +172,4 @@ PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
PQencryptPasswordConn 172
+PQresultMemsize 173
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 4c0114c514..064c7a693c 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -166,6 +166,7 @@ PQmakeEmptyPGresult(PGconn *conn, ExecStatusType status)
result->curBlock = NULL;
result->curOffset = 0;
result->spaceLeft = 0;
+ result->memsize = sizeof(PGresult);
if (conn)
{
@@ -215,6 +216,12 @@ PQmakeEmptyPGresult(PGconn *conn, ExecStatusType status)
return result;
}
+size_t
+PQresultMemsize(const PGresult *res)
+{
+ return res->memsize;
+}
+
/*
* PQsetResultAttrs
*
@@ -567,9 +574,11 @@ pqResultAlloc(PGresult *res, size_t nBytes, bool isBinary)
*/
if (nBytes >= PGRESULT_SEP_ALLOC_THRESHOLD)
{
- block = (PGresult_data *) malloc(nBytes + PGRESULT_BLOCK_OVERHEAD);
+ size_t alloc_size = nBytes + PGRESULT_BLOCK_OVERHEAD;
+ block = (PGresult_data *) malloc(alloc_size);
if (!block)
return NULL;
+ res->memsize += alloc_size;
space = block->space + PGRESULT_BLOCK_OVERHEAD;
if (res->curBlock)
{
@@ -594,6 +603,7 @@ pqResultAlloc(PGresult *res, size_t nBytes, bool isBinary)
block = (PGresult_data *) malloc(PGRESULT_DATA_BLOCKSIZE);
if (!block)
return NULL;
+ res->memsize += PGRESULT_DATA_BLOCKSIZE;
block->next = res->curBlock;
res->curBlock = block;
if (isBinary)
@@ -711,6 +721,7 @@ PQclear(PGresult *res)
res->errFields = NULL;
res->events = NULL;
res->nEvents = 0;
+ res->memsize = 0;
/* res->curBlock was zeroed out earlier */
/* Free the PGresult structure itself */
@@ -927,6 +938,7 @@ pqAddTuple(PGresult *res, PGresAttValue *tup, const char **errmsgp)
realloc(res->tuples, newSize * sizeof(PGresAttValue *));
if (!newTuples)
return false; /* malloc or realloc failed */
+ res->memsize += (newSize - res->tupArrSize) * sizeof(PGresAttValue *);
res->tupArrSize = newSize;
res->tuples = newTuples;
}
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index ed9c806861..4fd9a4fcda 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -491,6 +491,7 @@ extern int PQgetlength(const PGresult *res, int tup_num, int field_num);
extern int PQgetisnull(const PGresult *res, int tup_num, int field_num);
extern int PQnparams(const PGresult *res);
extern Oid PQparamtype(const PGresult *res, int param_num);
+extern size_t PQresultMemsize(const PGresult *res);
/* Describe prepared statements and portals */
extern PGresult *PQdescribePrepared(PGconn *conn, const char *stmt);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 9a586ff25a..37c9c3853d 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -208,6 +208,8 @@ struct pg_result
PGresult_data *curBlock; /* most recently allocated block */
int curOffset; /* start offset of free space in block */
int spaceLeft; /* number of free bytes remaining in block */
+
+ size_t memsize; /* Number of bytes of all memory allocated for this result */
};
/* PGAsyncStatusType defines the state of the query-execution state machine */
--
2.17.1
Attached is an updated patch of PQresultMemsize(). The implementation is
unchanged, but I added SGML documentation about this new function.
I'd be pleased about comments to adding this to libpq.
--
Kind Regards,
Lars
Attachments:
0001-libpq-Add-function-PQresultMemsize-v2.patchtext/x-patch; name=0001-libpq-Add-function-PQresultMemsize-v2.patchDownload
From 766a7f2381ae6c9d442a7359cabc58515186f8c4 Mon Sep 17 00:00:00 2001
From: Lars Kanis <lars@greiz-reinsdorf.de>
Date: Sat, 23 Jun 2018 19:34:11 +0200
Subject: [PATCH] libpq: Add function PQresultMemsize()
This function retrieves the number of bytes allocated for a given result.
That can be used to instruct the GC about the memory consumed behind a
wrapping object and for diagnosing memory consumption.
This is an alternative approach to customizable malloc/realloc/free
functions as discussed here:
https://www.postgresql.org/message-id/flat/20170828172834.GA71455%40TC.local#20170828172834.GA71455@TC.local
---
doc/src/sgml/libpq.sgml | 28 ++++++++++++++++++++++++++++
src/interfaces/libpq/exports.txt | 1 +
src/interfaces/libpq/fe-exec.c | 14 +++++++++++++-
src/interfaces/libpq/libpq-fe.h | 1 +
src/interfaces/libpq/libpq-int.h | 2 ++
5 files changed, 45 insertions(+), 1 deletion(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index d67212b831..c573c79ae3 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -6384,6 +6384,34 @@ void *PQresultAlloc(PGresult *res, size_t nBytes);
</listitem>
</varlistentry>
+ <varlistentry id="libpq-pqresultmemsize">
+ <term>
+ <function>PQresultMemsize</function>
+ <indexterm>
+ <primary>PQresultMemsize</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Retrieves the number of bytes allocated for a <structname>PGresult</structname> object.
+<synopsis>
+size_t PQresultMemsize(const PGresult *res);
+</synopsis>
+ </para>
+
+ <para>
+ The number of bytes includes the memory allocated for the PGresult itself,
+ memory to store data from the server, required internal metadata of a
+ PGresult object and data allocated by <function>PQresultAlloc</function>.
+ That is to say all memory which gets freed by <function>PQclear</function>.
+
+ This information can be used for diagnosing memory consumption and to instruct
+ a garbage collector about the memory consumed behind a wrapping object.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="libpq-pqlibversion">
<term>
<function>PQlibVersion</function>
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index d6a38d0df8..0b50dddbb7 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -172,3 +172,4 @@ PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
PQencryptPasswordConn 172
+PQresultMemsize 173
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 4c0114c514..064c7a693c 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -166,6 +166,7 @@ PQmakeEmptyPGresult(PGconn *conn, ExecStatusType status)
result->curBlock = NULL;
result->curOffset = 0;
result->spaceLeft = 0;
+ result->memsize = sizeof(PGresult);
if (conn)
{
@@ -215,6 +216,12 @@ PQmakeEmptyPGresult(PGconn *conn, ExecStatusType status)
return result;
}
+size_t
+PQresultMemsize(const PGresult *res)
+{
+ return res->memsize;
+}
+
/*
* PQsetResultAttrs
*
@@ -567,9 +574,11 @@ pqResultAlloc(PGresult *res, size_t nBytes, bool isBinary)
*/
if (nBytes >= PGRESULT_SEP_ALLOC_THRESHOLD)
{
- block = (PGresult_data *) malloc(nBytes + PGRESULT_BLOCK_OVERHEAD);
+ size_t alloc_size = nBytes + PGRESULT_BLOCK_OVERHEAD;
+ block = (PGresult_data *) malloc(alloc_size);
if (!block)
return NULL;
+ res->memsize += alloc_size;
space = block->space + PGRESULT_BLOCK_OVERHEAD;
if (res->curBlock)
{
@@ -594,6 +603,7 @@ pqResultAlloc(PGresult *res, size_t nBytes, bool isBinary)
block = (PGresult_data *) malloc(PGRESULT_DATA_BLOCKSIZE);
if (!block)
return NULL;
+ res->memsize += PGRESULT_DATA_BLOCKSIZE;
block->next = res->curBlock;
res->curBlock = block;
if (isBinary)
@@ -711,6 +721,7 @@ PQclear(PGresult *res)
res->errFields = NULL;
res->events = NULL;
res->nEvents = 0;
+ res->memsize = 0;
/* res->curBlock was zeroed out earlier */
/* Free the PGresult structure itself */
@@ -927,6 +938,7 @@ pqAddTuple(PGresult *res, PGresAttValue *tup, const char **errmsgp)
realloc(res->tuples, newSize * sizeof(PGresAttValue *));
if (!newTuples)
return false; /* malloc or realloc failed */
+ res->memsize += (newSize - res->tupArrSize) * sizeof(PGresAttValue *);
res->tupArrSize = newSize;
res->tuples = newTuples;
}
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index ed9c806861..4fd9a4fcda 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -491,6 +491,7 @@ extern int PQgetlength(const PGresult *res, int tup_num, int field_num);
extern int PQgetisnull(const PGresult *res, int tup_num, int field_num);
extern int PQnparams(const PGresult *res);
extern Oid PQparamtype(const PGresult *res, int param_num);
+extern size_t PQresultMemsize(const PGresult *res);
/* Describe prepared statements and portals */
extern PGresult *PQdescribePrepared(PGconn *conn, const char *stmt);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 9a586ff25a..37c9c3853d 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -208,6 +208,8 @@ struct pg_result
PGresult_data *curBlock; /* most recently allocated block */
int curOffset; /* start offset of free space in block */
int spaceLeft; /* number of free bytes remaining in block */
+
+ size_t memsize; /* Number of bytes of all memory allocated for this result */
};
/* PGAsyncStatusType defines the state of the query-execution state machine */
--
2.17.1
The following review has been posted through the commitfest application:
make installcheck-world: tested, passed
Implements feature: not tested
Spec compliant: not tested
Documentation: tested, passed
The implementation of PQresultMemsize is simple and correct. There are no warnings, and all tests passed.
I'll mark this patch as ready for committer
The new status of this patch is: Ready for Committer
Pavel Stehule <pavel.stehule@gmail.com> writes:
The implementation of PQresultMemsize is simple and correct.
Hm, well, only for small values of "correct". A quick look at PQclear
to see just what it clears shows that PGEvent objects associated with the
PGresult were overlooked. While that's not hard to fix (and I did so),
there's a problem hiding behind that: we have no idea what sort of
"instance data" the external event procedures may have associated with
the PGresult. In principle, any space occupied by such data needs to
be accounted for in this number.
This isn't an issue if the event procedure takes the lazy man's approach
of using PQresultAlloc to allocate such data. That's recommendable anyway
since it saves having to write cleanup logic; and even if you don't do it
like that, the space involved may be negligible. Still, I can foresee an
eventual need to let event procedures adjust the result-size number,
along the lines of PQincrementMemorySize(PGresult *res, size_t delta).
But that can probably wait till somebody actually requests it.
I didn't like the name "PQresultMemsize" much either, and changed it to
"PQresultMemorySize", which seems more in keeping with naming and
capitalization conventions elsewhere in libpq.
Pushed with those and some cosmetic changes.
regards, tom lane
2018-09-12 0:59 GMT+02:00 Tom Lane <tgl@sss.pgh.pa.us>:
Pavel Stehule <pavel.stehule@gmail.com> writes:
The implementation of PQresultMemsize is simple and correct.
Hm, well, only for small values of "correct". A quick look at PQclear
to see just what it clears shows that PGEvent objects associated with the
PGresult were overlooked.
My mistake. I though so event data are independent.
While that's not hard to fix (and I did so),
there's a problem hiding behind that: we have no idea what sort of
"instance data" the external event procedures may have associated with
the PGresult. In principle, any space occupied by such data needs to
be accounted for in this number.
This isn't an issue if the event procedure takes the lazy man's approach
of using PQresultAlloc to allocate such data. That's recommendable anyway
since it saves having to write cleanup logic; and even if you don't do it
like that, the space involved may be negligible. Still, I can foresee an
eventual need to let event procedures adjust the result-size number,
along the lines of PQincrementMemorySize(PGresult *res, size_t delta).
But that can probably wait till somebody actually requests it.I didn't like the name "PQresultMemsize" much either, and changed it to
"PQresultMemorySize", which seems more in keeping with naming and
capitalization conventions elsewhere in libpq.Pushed with those and some cosmetic changes.
Thank you
Pavel
Show quoted text
regards, tom lane