Stack overflow issue
Hello, I recently got a server crash (bug #17583 [1]/messages/by-id/CAMbWs499ytQiH4mLMhRxRWP-iEUz3-DSinpAD-cUCtVo_23Wtg@mail.gmail.com) caused by a stack overflow.
Tom Lane and Richard Guo, in a discussion of this bug, suggested that there could be more such places.
Therefore, Alexander Lakhin and I decided to deal with this issue and Alexander developed a methodology. We processed src/backend/*/*.c with "clang -emit-llvm ... | opt -analyze -print-calgraph" to find all the functions that call themselves directly. I checked each of them for features that protect against stack overflows.
We analyzed 4 catalogs: regex, tsearch, snowball and adt.
Firstly, we decided to test the regex catalog functions and found 6 of them that lack the check_stach_depth() call.
zaptreesubs
markst
next
nfatree
numst
repeat
We have tried to exploit the recursion in the function zaptreesubs():
select regexp_matches('a' || repeat(' a', 11000), '(.)(' || repeat(' \1', 11000) || ')?');
ERROR: invalid regular expression: regular expression is too complex
repeat():
select regexp_match('abc01234xyz',repeat('a{0,2}',100001));
ERROR: invalid regular expression: regular expression is too complex
numst():
select regexp_match('abc01234xyz',repeat('(.)\1e',100001));
ERROR: invalid regular expression: regular expression is too complex
markst():
markst is called in the code after v->tree = parse(...);
it is necessary that the tree be successfully parsed, but with a nesting level of about 100,000 this will not work - stack protection will work during parsing and v->ntree = numst(...); is also there.
next():
we were able to crash the server with the following query:
(printf "SELECT regexp_match('abc', 'a"; for ((i=1;i<1000000;i++)); do printf "(?#)"; done; printf "b')" ) | psql
Secondly, we have tried to exploit the recursion in the adt catalog functions and Alexander was able to crash the server with the following query:
regex_selectivity_sub():
SELECT * FROM pg_proc WHERE proname ~ ('(a' || repeat('|', 200000) || 'b)');
And this query:
(n=100000;
printf "SELECT polygon '((0,0),(0,1000000))' <@ polygon '((-200000,1000000),";
for ((i=1;i<$n;i++)); do printf "(100000,$(( 300000 + $i))),(-100000,$((800000 + $i))),"; done;
printf "(200000,900000),(200000,0))';"
) | psql
Thirdly, the snowball catalog, Alexander has tried to exploit the recursion in the r_stem_suffix_chain_before_ki function and crashed a server using this query:
r_stem_suffix_chain_before_ki():
SELECT ts_lexize('turkish_stem', repeat('lerdeki', 1000000));
The last one is the tsearch catalog. We have found 4 functions that didn't have check_stach_depth() function:
SplitToVariants
mkANode
mkSPNode
LexizeExec
We have tried to exploit the recursion in the SplitToVariants function and Alexander crashed a server using this:
SplitToVariants():
CREATE TEXT SEARCH DICTIONARY ispell (Template=ispell, DictFile=ispell_sample,AffFile=ispell_sample);
SELECT ts_lexize('ispell', repeat('bally', 10000));
After trying to exploit the recursion in the LexizeExec function Alexander made this conlusion:
LexizeExec has two branches "ld->curDictId == InvalidOid" (usual mode) and "ld->curDictId != InvalidOid" (multiword mode) - we start with the first one, then make recursive call to switch to the multiword mode, but then we return to the usual mode again.
mkANode and mkSPNode deal with the dictionary structs, not with user-supplied data, so we believe these functions are not vulnerable.
[1]: /messages/by-id/CAMbWs499ytQiH4mLMhRxRWP-iEUz3-DSinpAD-cUCtVo_23Wtg@mail.gmail.com
Hi,
Can we have a parameter to control the recursion depth in these cases to
avoid crashes?
Just a thought.
Thanks,
Mahendrakar.
On Wed, 24 Aug, 2022, 3:21 pm Егор Чиндяскин, <kyzevan23@mail.ru> wrote:
Show quoted text
Hello, I recently got a server crash (bug #17583 [1]) caused by a stack
overflow.Tom Lane and Richard Guo, in a discussion of this bug, suggested that
there could be more such places.
Therefore, Alexander Lakhin and I decided to deal with this issue and
Alexander developed a methodology. We processed src/backend/*/*.c with
"clang -emit-llvm ... | opt -analyze -print-calgraph" to find all the
functions that call themselves directly. I checked each of them for
features that protect against stack overflows.
We analyzed 4 catalogs: regex, tsearch, snowball and adt.
Firstly, we decided to test the regex catalog functions and found 6 of
them that lack the check_stach_depth() call.zaptreesubs
markst
next
nfatree
numst
repeatWe have tried to exploit the recursion in the function zaptreesubs():
select regexp_matches('a' || repeat(' a', 11000), '(.)(' || repeat(' \1',
11000) || ')?');ERROR: invalid regular expression: regular expression is too complex
repeat():
select regexp_match('abc01234xyz',repeat('a{0,2}',100001));ERROR: invalid regular expression: regular expression is too complex
numst():
select regexp_match('abc01234xyz',repeat('(.)\1e',100001));ERROR: invalid regular expression: regular expression is too complex
markst():
markst is called in the code after v->tree = parse(...);
it is necessary that the tree be successfully parsed, but with a nesting
level of about 100,000 this will not work - stack protection will work
during parsing and v->ntree = numst(...); is also there.next():
we were able to crash the server with the following query:
(printf "SELECT regexp_match('abc', 'a"; for ((i=1;i<1000000;i++)); do
printf "(?#)"; done; printf "b')" ) | psqlSecondly, we have tried to exploit the recursion in the adt catalog
functions and Alexander was able to crash the server with the following
query:regex_selectivity_sub():
SELECT * FROM pg_proc WHERE proname ~ ('(a' || repeat('|', 200000) ||
'b)');And this query:
(n=100000;
printf "SELECT polygon '((0,0),(0,1000000))' <@ polygon
'((-200000,1000000),";
for ((i=1;i<$n;i++)); do printf "(100000,$(( 300000 +
$i))),(-100000,$((800000 + $i))),"; done;
printf "(200000,900000),(200000,0))';"
) | psqlThirdly, the snowball catalog, Alexander has tried to exploit the
recursion in the r_stem_suffix_chain_before_ki function and crashed a
server using this query:r_stem_suffix_chain_before_ki():
SELECT ts_lexize('turkish_stem', repeat('lerdeki', 1000000));The last one is the tsearch catalog. We have found 4 functions that didn't
have check_stach_depth() function:SplitToVariants
mkANode
mkSPNode
LexizeExecWe have tried to exploit the recursion in the SplitToVariants function and
Alexander crashed a server using this:SplitToVariants():
CREATE TEXT SEARCH DICTIONARY ispell (Template=ispell,
DictFile=ispell_sample,AffFile=ispell_sample);
SELECT ts_lexize('ispell', repeat('bally', 10000));After trying to exploit the recursion in the LexizeExec function Alexander
made this conlusion:LexizeExec has two branches "ld->curDictId == InvalidOid" (usual mode) and
"ld->curDictId != InvalidOid" (multiword mode) - we start with the first
one, then make recursive call to switch to the multiword mode, but then we
return to the usual mode again.mkANode and mkSPNode deal with the dictionary structs, not with
user-supplied data, so we believe these functions are not vulnerable.[1]
/messages/by-id/CAMbWs499ytQiH4mLMhRxRWP-iEUz3-DSinpAD-cUCtVo_23Wtg@mail.gmail.com
On 2022-Aug-24, mahendrakar s wrote:
Hi,
Can we have a parameter to control the recursion depth in these cases to
avoid crashes?
We already have one (max_stack_depth). The problem is lack of calling
the control function in a few places.
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
Thanks.
On Wed, 24 Aug, 2022, 4:19 pm Alvaro Herrera, <alvherre@alvh.no-ip.org>
wrote:
Show quoted text
On 2022-Aug-24, mahendrakar s wrote:
Hi,
Can we have a parameter to control the recursion depth in these cases to
avoid crashes?We already have one (max_stack_depth). The problem is lack of calling
the control function in a few places.--
Álvaro Herrera 48°01'N 7°57'E —
https://www.EnterpriseDB.com/
On Wed, Aug 24, 2022 at 6:49 PM Alvaro Herrera <alvherre@alvh.no-ip.org>
wrote:
On 2022-Aug-24, mahendrakar s wrote:
Hi,
Can we have a parameter to control the recursion depth in these cases to
avoid crashes?We already have one (max_stack_depth). The problem is lack of calling
the control function in a few places.
Thanks Egor and Alexander for the work! I think we can just add
check_stack_depth checks in these cases.
Thanks
Richard
On Wed, Aug 24, 2022 at 7:12 PM Richard Guo <guofenglinux@gmail.com> wrote:
On Wed, Aug 24, 2022 at 6:49 PM Alvaro Herrera <alvherre@alvh.no-ip.org>
wrote:On 2022-Aug-24, mahendrakar s wrote:
Hi,
Can we have a parameter to control the recursion depth in these cases to
avoid crashes?We already have one (max_stack_depth). The problem is lack of calling
the control function in a few places.Thanks Egor and Alexander for the work! I think we can just add
check_stack_depth checks in these cases.
Attached adds the checks in these places. But I'm not sure about the
snowball case. Can we edit src/backend/snowball/libstemmer/*.c directly?
Thanks
Richard
Attachments:
v1-0001-add-check_stack_depth-in-more-places.patchapplication/octet-stream; name=v1-0001-add-check_stack_depth-in-more-places.patchDownload
From 5c23f9cd9c16c605acdc36393e1a31e8c9b97a93 Mon Sep 17 00:00:00 2001
From: pgsql-guo <richard.guo@openpie.com>
Date: Wed, 24 Aug 2022 11:45:35 +0000
Subject: [PATCH v1] add check_stack_depth in more places
---
src/backend/regex/regc_lex.c | 3 +++
src/backend/snowball/libstemmer/stem_UTF_8_turkish.c | 3 +++
src/backend/tsearch/spell.c | 4 ++++
src/backend/utils/adt/geo_ops.c | 3 +++
src/backend/utils/adt/like_support.c | 4 ++++
5 files changed, 17 insertions(+)
diff --git a/src/backend/regex/regc_lex.c b/src/backend/regex/regc_lex.c
index 45727ffa01..1169e731a1 100644
--- a/src/backend/regex/regc_lex.c
+++ b/src/backend/regex/regc_lex.c
@@ -201,6 +201,9 @@ next(struct vars *v)
{
chr c;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
/* errors yield an infinite sequence of failures */
if (ISERR())
return 0; /* the error has set nexttype to EOS */
diff --git a/src/backend/snowball/libstemmer/stem_UTF_8_turkish.c b/src/backend/snowball/libstemmer/stem_UTF_8_turkish.c
index 3d04032787..9254574525 100644
--- a/src/backend/snowball/libstemmer/stem_UTF_8_turkish.c
+++ b/src/backend/snowball/libstemmer/stem_UTF_8_turkish.c
@@ -1,6 +1,7 @@
/* Generated by Snowball 2.2.0 - https://snowballstem.org/ */
#include "header.h"
+#include "miscadmin.h"
#ifdef __cplusplus
extern "C" {
@@ -1156,6 +1157,8 @@ lab0:
}
static int r_stem_suffix_chain_before_ki(struct SN_env * z) {
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
z->ket = z->c;
{ int ret = r_mark_ki(z);
if (ret <= 0) return ret;
diff --git a/src/backend/tsearch/spell.c b/src/backend/tsearch/spell.c
index f07321b61d..edd2fbbd3a 100644
--- a/src/backend/tsearch/spell.c
+++ b/src/backend/tsearch/spell.c
@@ -63,6 +63,7 @@
#include "postgres.h"
#include "catalog/pg_collation.h"
+#include "miscadmin.h"
#include "tsearch/dicts/spell.h"
#include "tsearch/ts_locale.h"
#include "utils/memutils.h"
@@ -2400,6 +2401,9 @@ SplitToVariants(IspellDict *Conf, SPNode *snode, SplitVar *orig, char *word, int
char *notprobed;
int compoundflag = 0;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
notprobed = (char *) palloc(wordlen);
memset(notprobed, 1, wordlen);
var = CopyVar(orig, 1);
diff --git a/src/backend/utils/adt/geo_ops.c b/src/backend/utils/adt/geo_ops.c
index b79705f8b3..535301a218 100644
--- a/src/backend/utils/adt/geo_ops.c
+++ b/src/backend/utils/adt/geo_ops.c
@@ -3833,6 +3833,9 @@ lseg_inside_poly(Point *a, Point *b, POLYGON *poly, int start)
bool res = true,
intersection = false;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
t.p[0] = *a;
t.p[1] = *b;
s.p[0] = poly->p[(start == 0) ? (poly->npts - 1) : (start - 1)];
diff --git a/src/backend/utils/adt/like_support.c b/src/backend/utils/adt/like_support.c
index 65a57fc3c4..2d3aaaaf6b 100644
--- a/src/backend/utils/adt/like_support.c
+++ b/src/backend/utils/adt/like_support.c
@@ -44,6 +44,7 @@
#include "catalog/pg_statistic.h"
#include "catalog/pg_type.h"
#include "mb/pg_wchar.h"
+#include "miscadmin.h"
#include "nodes/makefuncs.h"
#include "nodes/nodeFuncs.h"
#include "nodes/supportnodes.h"
@@ -1364,6 +1365,9 @@ regex_selectivity_sub(const char *patt, int pattlen, bool case_insensitive)
int paren_pos = 0; /* dummy init to keep compiler quiet */
int pos;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
for (pos = 0; pos < pattlen; pos++)
{
if (patt[pos] == '(')
--
2.25.1
Hi Richard,
Patch is looking good to me. Would request others to take a look at it as
well.
Thanks,
Mahendrakar.
On Wed, 24 Aug 2022 at 17:24, Richard Guo <guofenglinux@gmail.com> wrote:
Show quoted text
On Wed, Aug 24, 2022 at 7:12 PM Richard Guo <guofenglinux@gmail.com>
wrote:On Wed, Aug 24, 2022 at 6:49 PM Alvaro Herrera <alvherre@alvh.no-ip.org>
wrote:On 2022-Aug-24, mahendrakar s wrote:
Hi,
Can we have a parameter to control the recursion depth in these casesto
avoid crashes?
We already have one (max_stack_depth). The problem is lack of calling
the control function in a few places.Thanks Egor and Alexander for the work! I think we can just add
check_stack_depth checks in these cases.Attached adds the checks in these places. But I'm not sure about the
snowball case. Can we edit src/backend/snowball/libstemmer/*.c directly?Thanks
Richard
=?UTF-8?B?0JXQs9C+0YAg0KfQuNC90LTRj9GB0LrQuNC9?= <kyzevan23@mail.ru> writes:
Therefore, Alexander Lakhin and I decided to deal with this issue and Alexander developed a methodology. We processed src/backend/*/*.c with "clang -emit-llvm ... | opt -analyze -print-calgraph" to find all the functions that call themselves directly. I checked each of them for features that protect against stack overflows.
We analyzed 4 catalogs: regex, tsearch, snowball and adt.
Firstly, we decided to test the regex catalog functions and found 6 of them that lack the check_stach_depth() call.
Nice work! I wonder if you can make the regex crashes reachable by
reducing the value of max_stack_depth enough that it's hit before
reaching the "regular expression is too complex" limit.
regards, tom lane
Richard Guo <guofenglinux@gmail.com> writes:
Attached adds the checks in these places. But I'm not sure about the
snowball case. Can we edit src/backend/snowball/libstemmer/*.c directly?
No, that file is generated code, as it says right at the top.
I think most likely we should report this to Snowball upstream
and see what they think is an appropriate fix.
regards, tom lane
=?UTF-8?B?0JXQs9C+0YAg0KfQuNC90LTRj9GB0LrQuNC9?= <kyzevan23@mail.ru> writes:
Firstly, we decided to test the regex catalog functions and found 6 of them that lack the check_stach_depth() call.
zaptreesubs
markst
next
nfatree
numst
repeat
I took a closer look at these. I think the markst, numst, and nfatree
cases are non-issues. They are recursing on a subre tree that was just
built by parse(), so parse() must have successfully recursed the same
number of levels. parse() surely has a larger stack frame, and it
does have a stack overflow guard (in subre()), so it would have failed
cleanly before making a data structure that could be hazardous here.
Also, having markst error out would be problematic for the reasons
discussed in its comment, so I'm disinclined to try to add checks
that have no use.
BTW, I wonder why your test didn't notice freesubre()? But the
same analysis applies there, as does the concern that we can't
just error out.
Likewise, zaptreesubs() can't recurse more levels than cdissect() did,
and that has a stack check, so I'm not very excited about adding
another one there.
I believe that repeat() is a non-issue because (a) the number of
recursion levels in it is limited by DUPMAX, which is generally going
to be 255, or at least not enormous, and (b) it will recurse at most
once before calling dupnfa(), which contains stack checks.
I think next() is a legit issue, although your example doesn't crash
for me. I suppose that's because my compiler turned the tail recursion
into a loop, and I suggest that we fix it by doing that manually.
(Richard's proposed fix is wrong anyway: we can't just throw elog(ERROR)
in the regex code without creating memory leaks.)
regards, tom lane
I wrote:
I think most likely we should report this to Snowball upstream
and see what they think is an appropriate fix.
Done at [1]https://lists.tartarus.org/pipermail/snowball-discuss/2022-August/001734.html, and I pushed the other fixes. Thanks again for the report!
regards, tom lane
[1]: https://lists.tartarus.org/pipermail/snowball-discuss/2022-August/001734.html
I wrote:
I think most likely we should report this to Snowball upstream
and see what they think is an appropriate fix.
Done at [1], and I pushed the other fixes. Thanks again for the report!
The upstream recommendation, which seems pretty sane to me, is to
simply reject any string exceeding some threshold length as not
possibly being a word. Apparently it's common to use thresholds
as small as 64 bytes, but in the attached I used 1000 bytes.
regards, tom lane
Attachments:
limit-length-of-strings-passed-to-snowball.patchtext/x-diff; charset=us-ascii; name=limit-length-of-strings-passed-to-snowball.patchDownload
diff --git a/src/backend/snowball/dict_snowball.c b/src/backend/snowball/dict_snowball.c
index 68c9213f69..aaf4ff72b6 100644
--- a/src/backend/snowball/dict_snowball.c
+++ b/src/backend/snowball/dict_snowball.c
@@ -272,11 +272,25 @@ dsnowball_lexize(PG_FUNCTION_ARGS)
DictSnowball *d = (DictSnowball *) PG_GETARG_POINTER(0);
char *in = (char *) PG_GETARG_POINTER(1);
int32 len = PG_GETARG_INT32(2);
- char *txt = lowerstr_with_len(in, len);
TSLexeme *res = palloc0(sizeof(TSLexeme) * 2);
+ char *txt;
+ /*
+ * Reject strings exceeding 1000 bytes, as they're surely not words in any
+ * human language. This restriction avoids wasting cycles on stuff like
+ * base64-encoded data, and it protects us against possible inefficiency
+ * or misbehavior in the stemmers (for example, the Turkish stemmer has an
+ * indefinite recursion so it can crash on long-enough strings).
+ */
+ if (len <= 0 || len > 1000)
+ PG_RETURN_POINTER(res);
+
+ txt = lowerstr_with_len(in, len);
+
+ /* txt is probably not zero-length now, but we'll check anyway */
if (*txt == '\0' || searchstoplist(&(d->stoplist), txt))
{
+ /* empty or stopword, so reject */
pfree(txt);
}
else
I wrote:
The upstream recommendation, which seems pretty sane to me, is to
simply reject any string exceeding some threshold length as not
possibly being a word. Apparently it's common to use thresholds
as small as 64 bytes, but in the attached I used 1000 bytes.
On further thought: that coding treats anything longer than 1000
bytes as a stopword, but maybe we should just accept it unmodified.
The manual says "A Snowball dictionary recognizes everything, whether
or not it is able to simplify the word". While "recognizes" formally
includes the case of "recognizes as a stopword", people might find
this behavior surprising. We could alternatively do it as attached,
which accepts overlength words but does nothing to them except
case-fold. This is closer to the pre-patch behavior, but gives up
the opportunity to avoid useless downstream processing of long words.
regards, tom lane
Attachments:
limit-length-of-strings-passed-to-snowball-2.patchtext/x-diff; charset=us-ascii; name=limit-length-of-strings-passed-to-snowball-2.patchDownload
diff --git a/src/backend/snowball/dict_snowball.c b/src/backend/snowball/dict_snowball.c
index 68c9213f69..1d5dfff5a0 100644
--- a/src/backend/snowball/dict_snowball.c
+++ b/src/backend/snowball/dict_snowball.c
@@ -275,8 +275,24 @@ dsnowball_lexize(PG_FUNCTION_ARGS)
char *txt = lowerstr_with_len(in, len);
TSLexeme *res = palloc0(sizeof(TSLexeme) * 2);
- if (*txt == '\0' || searchstoplist(&(d->stoplist), txt))
+ /*
+ * Do not pass strings exceeding 1000 bytes to the stemmer, as they're
+ * surely not words in any human language. This restriction avoids
+ * wasting cycles on stuff like base64-encoded data, and it protects us
+ * against possible inefficiency or misbehavior in the stemmer. (For
+ * example, the Turkish stemmer has an indefinite recursion, so it can
+ * crash on long-enough strings.) However, Snowball dictionaries are
+ * defined to recognize all strings, so we can't reject the string as an
+ * unknown word.
+ */
+ if (len > 1000)
+ {
+ /* return the lexeme lowercased, but otherwise unmodified */
+ res->lexeme = txt;
+ }
+ else if (*txt == '\0' || searchstoplist(&(d->stoplist), txt))
{
+ /* empty or stopword, so report as stopword */
pfree(txt);
}
else
On Wed, Aug 31, 2022 at 6:57 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
I wrote:
The upstream recommendation, which seems pretty sane to me, is to
simply reject any string exceeding some threshold length as not
possibly being a word. Apparently it's common to use thresholds
as small as 64 bytes, but in the attached I used 1000 bytes.On further thought: that coding treats anything longer than 1000
bytes as a stopword, but maybe we should just accept it unmodified.
The manual says "A Snowball dictionary recognizes everything, whether
or not it is able to simplify the word". While "recognizes" formally
includes the case of "recognizes as a stopword", people might find
this behavior surprising. We could alternatively do it as attached,
which accepts overlength words but does nothing to them except
case-fold. This is closer to the pre-patch behavior, but gives up
the opportunity to avoid useless downstream processing of long words.
This patch looks good to me. It avoids overly-long words (> 1000 bytes)
going through the stemmer so the stack overflow issue in Turkish stemmer
should not exist any more.
Thanks
Richard
24.08.2022 20:58, Tom Lane writes:
Nice work! I wonder if you can make the regex crashes reachable by
reducing the value of max_stack_depth enough that it's hit before
reaching the "regular expression is too complex" limit.regards, tom lane
Hello everyone! It's been a while since me and Alexander Lakhin have
published a list of functions that have a stack overflow illness. We are
back to tell you more about such places.
During our analyze we made a conclusion that some functions can be
crashed without changing any of the parameters and some can be crashed
only if we change some stuff.
The first function crashes without any changes:
# CheckAttributeType
(n=60000; printf "create domain dint as int; create domain dinta0 as
dint[];"; for ((i=1;i<=$n;i++)); do printf "create domain dinta$i as
dinta$(( $i - 1 ))[]; "; done; ) | psql
psql -c "create table t(f1 dinta60000[]);"
Some of the others crash if we change "max_locks_per_transaction"
parameter:
# findDependentObjects
max_locks_per_transaction = 200
(n=10000; printf "create table t (i int); create view v0 as select *
from t;"; for ((i=1;i<$n;i++)); do printf "create view v$i as select *
from v$(( $i - 1 )); "; done; ) | psql
psql -c "drop table t"
# ATExecDropColumn
max_locks_per_transaction = 300
(n=50000; printf "create table t0 (a int, b int); "; for
((i=1;i<=$n;i++)); do printf "create table t$i() inherits(t$(( $i - 1
))); "; done; printf "alter table t0 drop b;" ) | psql
# ATExecDropConstraint
max_locks_per_transaction = 300
(n=50000; printf "create table t0 (a int, b int, constraint bc check (b
0));"; for ((i=1;i<=$n;i++)); do printf "create table t$i()
inherits(t$(( $i - 1 ))); "; done; printf "alter table t0 drop
constraint bc;" ) | psql
# ATExecAddColumn
max_locks_per_transaction = 200
(n=50000; printf "create table t0 (a int, b int);"; for
((i=1;i<=$n;i++)); do printf "create table t$i() inherits(t$(( $i - 1
))); "; done; printf "alter table t0 add column c int;" ) | psql
# ATExecAlterConstrRecurse
max_locks_per_transaction = 300
(n=50000;
printf "create table t(a int primary key); create table pt (a int
primary key, foreign key(a) references t) partition by range (a);";
printf "create table pt0 partition of pt for values from (0) to (100000)
partition by range (a);";
for ((i=1;i<=$n;i++)); do printf "create table pt$i partition of pt$((
$i - 1 )) for values from ($i) to (100000) partition by range (a); "; done;
printf "alter table pt alter constraint pt_a_fkey deferrable initially
deferred;"
) | psql
This is where the fun begins. According to Tom Lane, a decrease in
max_stack_depth could lead to new crashes, but it turned out that
Alexander was able to find new crashes precisely due to the increase in
this parameter. Also, we had ulimit -s set to 8MB as the default value.
# eval_const_expressions_mutator
max_stack_depth = '7000kB'
(n=10000; printf "select 'a' "; for ((i=1;i<$n;i++)); do printf "
collate \"C\" "; done; ) | psql
If you didn’t have a crash, like me, when Alexander shared his find,
then probably you configured your cluster with an optimization flag -Og.
In the process of trying to break this function, we came to the
conclusion that the maximum stack depth depends on the optimization flag
(-O0/-Og). As it turned out, when optimizing, the function frame on the
stack becomes smaller and because of this, the limit is reached more
slowly, therefore, the system can withstand more torment. Therefore,
this query will fail if you have a cluster configured with the -O0
optimization flag.
The crash of the next function not only depends on the optimization
flag, but also on a number of other things. While researching, we
noticed that postgres enforces a distance ~400kB from max_stack_depth to
ulimit -s. We thought we could hit the max_stack_depth limit and then
hit the OS limit as well. Therefore, Alexander wrote a recursive SQL
function, that eats up a stack within max_stack_depth, including a query
that eats up the remaining ~400kB. And this causes a crash.
# executeBoolItem
max_stack_depth = '7600kB'
create function infinite_recurse(i int) returns int as $$
begin
raise notice 'Level %', i;
begin
perform jsonb_path_query('{"a":[1]}'::jsonb, ('$.a[*] ? (' ||
repeat('!(', 4800) || '@ == @' || repeat(')', 4800) || ')')::jsonpath);
exception
when others then raise notice 'jsonb_path_query error at level %,
%', i, sqlerrm;
end;
begin
select infinite_recurse(i + 1) into i;
exception
when others then raise notice 'Max stack depth reached at level %,
%', i, sqlerrm;
end;
return i;
end;
$$ language plpgsql;
select infinite_recurse(1);
To sum it all up, we have not yet decided on a general approach to such
functions. Some functions are definitely subject to stack overflow. Some
are definitely not. This can be seen from the code where the recurse
flag is passed, or a function that checks the stack is called before a
recursive call. Some require special conditions - for example, you need
to parse the query and build a plan, and at that stage the stack is
eaten faster (and checked) than by the function that we are interested in.
We keep researching and hope to come up with a good solution sooner or
later.
Среда, 26 октября 2022, 21:47 +07:00 от Egor Chindyaskin <kyzevan23@mail.ru>:
24.08.2022 20:58, Tom Lane writes:Nice work! I wonder if you can make the regex crashes reachable by
reducing the value of max_stack_depth enough that it's hit before
reaching the "regular expression is too complex" limit.regards, tom lane Hello everyone! It's been a while since me and Alexander Lakhin have
published a list of functions that have a stack overflow illness. We are
back to tell you more about such places.
During our analyze we made a conclusion that some functions can be
crashed without changing any of the parameters and some can be crashed
only if we change some stuff.The first function crashes without any changes:
# CheckAttributeType
(n=60000; printf "create domain dint as int; create domain dinta0 as
dint[];"; for ((i=1;i<=$n;i++)); do printf "create domain dinta$i as
dinta$(( $i - 1 ))[]; "; done; ) | psql
psql -c "create table t(f1 dinta60000[]);"Some of the others crash if we change "max_locks_per_transaction"
parameter:# findDependentObjects
max_locks_per_transaction = 200
(n=10000; printf "create table t (i int); create view v0 as select *
from t;"; for ((i=1;i<$n;i++)); do printf "create view v$i as select *
from v$(( $i - 1 )); "; done; ) | psql
psql -c "drop table t"# ATExecDropColumn
max_locks_per_transaction = 300
(n=50000; printf "create table t0 (a int, b int); "; for
((i=1;i<=$n;i++)); do printf "create table t$i() inherits(t$(( $i - 1
))); "; done; printf "alter table t0 drop b;" ) | psql# ATExecDropConstraint
max_locks_per_transaction = 300
(n=50000; printf "create table t0 (a int, b int, constraint bc check (b
> 0));"; for ((i=1;i<=$n;i++)); do printf "create table t$i()
inherits(t$(( $i - 1 ))); "; done; printf "alter table t0 drop
constraint bc;" ) | psql# ATExecAddColumn
max_locks_per_transaction = 200
(n=50000; printf "create table t0 (a int, b int);"; for
((i=1;i<=$n;i++)); do printf "create table t$i() inherits(t$(( $i - 1
))); "; done; printf "alter table t0 add column c int;" ) | psql# ATExecAlterConstrRecurse
max_locks_per_transaction = 300
(n=50000;
printf "create table t(a int primary key); create table pt (a int
primary key, foreign key(a) references t) partition by range (a);";
printf "create table pt0 partition of pt for values from (0) to (100000)
partition by range (a);";
for ((i=1;i<=$n;i++)); do printf "create table pt$i partition of pt$((
$i - 1 )) for values from ($i) to (100000) partition by range (a); "; done;
printf "alter table pt alter constraint pt_a_fkey deferrable initially
deferred;"
) | psqlThis is where the fun begins. According to Tom Lane, a decrease in
max_stack_depth could lead to new crashes, but it turned out that
Alexander was able to find new crashes precisely due to the increase in
this parameter. Also, we had ulimit -s set to 8MB as the default value.# eval_const_expressions_mutator
max_stack_depth = '7000kB'
(n=10000; printf "select 'a' "; for ((i=1;i<$n;i++)); do printf "
collate \"C\" "; done; ) | psqlIf you didn’t have a crash, like me, when Alexander shared his find,
then probably you configured your cluster with an optimization flag -Og.
In the process of trying to break this function, we came to the
conclusion that the maximum stack depth depends on the optimization flag
(-O0/-Og). As it turned out, when optimizing, the function frame on the
stack becomes smaller and because of this, the limit is reached more
slowly, therefore, the system can withstand more torment. Therefore,
this query will fail if you have a cluster configured with the -O0
optimization flag.The crash of the next function not only depends on the optimization
flag, but also on a number of other things. While researching, we
noticed that postgres enforces a distance ~400kB from max_stack_depth to
ulimit -s. We thought we could hit the max_stack_depth limit and then
hit the OS limit as well. Therefore, Alexander wrote a recursive SQL
function, that eats up a stack within max_stack_depth, including a query
that eats up the remaining ~400kB. And this causes a crash.# executeBoolItem
max_stack_depth = '7600kB'
create function infinite_recurse(i int) returns int as $$
begin
raise notice 'Level %', i;
begin
perform jsonb_path_query('{"a":[1]}'::jsonb, ('$.a[*] ? (' ||
repeat('!(', 4800) || '@ == @' || repeat(')', 4800) || ')')::jsonpath);
exception
when others then raise notice 'jsonb_path_query error at level %,
%', i, sqlerrm;
end;
begin
select infinite_recurse(i + 1) into i;
exception
when others then raise notice 'Max stack depth reached at level %,
%', i, sqlerrm;
end;
return i;
end;
$$ language plpgsql;select infinite_recurse(1);
To sum it all up, we have not yet decided on a general approach to such
functions. Some functions are definitely subject to stack overflow. Some
are definitely not. This can be seen from the code where the recurse
flag is passed, or a function that checks the stack is called before a
recursive call. Some require special conditions - for example, you need
to parse the query and build a plan, and at that stage the stack is
eaten faster (and checked) than by the function that we are interested in.We keep researching and hope to come up with a good solution sooner or
later.
Hello, in continuation of the topic of the stack overflow problem, Alexander Lakhin was able to find a few more similar places.
An important point for the first function is that the server must be built with asserts enabled, otherwise the crash will not happen.
Also, the result in the form of a server crash will be achieved only after 2-3 hours.
#MemoryContextCheck
(n=1000000; printf "begin;"; for ((i=1;i<=$n;i++)); do printf "savepoint s$i;"; done; printf "release s1;" ) | psql >/dev/null
Other functions could be crashed without asserts enabled.
#CommitTransactionCommand
(n=1000000; printf "BEGIN;"; for ((i=1;i<=$n;i++)); do printf "SAVEPOINT s$i;"; done; printf "ERROR; COMMIT;") | psql >/dev/null
#MemoryContextStatsInternal
(n=1000000; printf "BEGIN;"; for ((i=1;i<=$n;i++)); do printf "SAVEPOINT s$i;"; done; printf "SELECT pg_log_backend_memory_contexts(pg_backend_pid())") | psql >/dev/null
#ShowTransactionStateRec
(n=1000000; printf "BEGIN;"; for ((i=1;i<=$n;i++)); do printf "SAVEPOINT s$i;"; done; printf "SET log_min_messages = 'DEBUG5'; SAVEPOINT sp;") | psql >/dev/null
The following next two functions call each other; the following way was found to overflow the stack (with modified server configuration):
#MemoryContextDeleteChildren with MemoryContextDelete
max_connections = 1000
max_stack_depth = '7600kB'
create table idxpart (a int) partition by range (a);
select 'create index on idxpart (a)' from generate_series(1, 40000);
\gexec
create table idxpart (a int) partition by range (a);
select 'create index on idxpart (a)' from generate_series(1, 40000);
\gexec
create function infinite_recurse(level int) returns int as $$
declare l int;
begin
begin
select infinite_recurse(level + 1) into level;
exception
when others then raise notice 'Max stack depth reached at level %, %', level, sqlerrm;
create table idxpart1 partition of idxpart for values from (1) to (2) partition by range (a);
end;
return level;
end;
$$ language plpgsql;
select infinite_recurse(1);
Finally, there are yet two recursive functions in mcxt.c:
#MemoryContextResetChildren - could be vulnerable but not used at all after eaa5808e.
#MemoryContextMemAllocated - at present called only with local contexts.
Great work. Max Stack depth is memory dependent? Processor dependent?
Егор Чиндяскин <kyzevan23@mail.ru> schrieb am Mi., 24. Aug. 2022, 11:51:
Show quoted text
Hello, I recently got a server crash (bug #17583 [1]) caused by a stack
overflow.Tom Lane and Richard Guo, in a discussion of this bug, suggested that
there could be more such places.
Therefore, Alexander Lakhin and I decided to deal with this issue and
Alexander developed a methodology. We processed src/backend/*/*.c with
"clang -emit-llvm ... | opt -analyze -print-calgraph" to find all the
functions that call themselves directly. I checked each of them for
features that protect against stack overflows.
We analyzed 4 catalogs: regex, tsearch, snowball and adt.
Firstly, we decided to test the regex catalog functions and found 6 of
them that lack the check_stach_depth() call.zaptreesubs
markst
next
nfatree
numst
repeatWe have tried to exploit the recursion in the function zaptreesubs():
select regexp_matches('a' || repeat(' a', 11000), '(.)(' || repeat(' \1',
11000) || ')?');ERROR: invalid regular expression: regular expression is too complex
repeat():
select regexp_match('abc01234xyz',repeat('a{0,2}',100001));ERROR: invalid regular expression: regular expression is too complex
numst():
select regexp_match('abc01234xyz',repeat('(.)\1e',100001));ERROR: invalid regular expression: regular expression is too complex
markst():
markst is called in the code after v->tree = parse(...);
it is necessary that the tree be successfully parsed, but with a nesting
level of about 100,000 this will not work - stack protection will work
during parsing and v->ntree = numst(...); is also there.next():
we were able to crash the server with the following query:
(printf "SELECT regexp_match('abc', 'a"; for ((i=1;i<1000000;i++)); do
printf "(?#)"; done; printf "b')" ) | psqlSecondly, we have tried to exploit the recursion in the adt catalog
functions and Alexander was able to crash the server with the following
query:regex_selectivity_sub():
SELECT * FROM pg_proc WHERE proname ~ ('(a' || repeat('|', 200000) ||
'b)');And this query:
(n=100000;
printf "SELECT polygon '((0,0),(0,1000000))' <@ polygon
'((-200000,1000000),";
for ((i=1;i<$n;i++)); do printf "(100000,$(( 300000 +
$i))),(-100000,$((800000 + $i))),"; done;
printf "(200000,900000),(200000,0))';"
) | psqlThirdly, the snowball catalog, Alexander has tried to exploit the
recursion in the r_stem_suffix_chain_before_ki function and crashed a
server using this query:r_stem_suffix_chain_before_ki():
SELECT ts_lexize('turkish_stem', repeat('lerdeki', 1000000));The last one is the tsearch catalog. We have found 4 functions that didn't
have check_stach_depth() function:SplitToVariants
mkANode
mkSPNode
LexizeExecWe have tried to exploit the recursion in the SplitToVariants function and
Alexander crashed a server using this:SplitToVariants():
CREATE TEXT SEARCH DICTIONARY ispell (Template=ispell,
DictFile=ispell_sample,AffFile=ispell_sample);
SELECT ts_lexize('ispell', repeat('bally', 10000));After trying to exploit the recursion in the LexizeExec function Alexander
made this conlusion:LexizeExec has two branches "ld->curDictId == InvalidOid" (usual mode) and
"ld->curDictId != InvalidOid" (multiword mode) - we start with the first
one, then make recursive call to switch to the multiword mode, but then we
return to the usual mode again.mkANode and mkSPNode deal with the dictionary structs, not with
user-supplied data, so we believe these functions are not vulnerable.[1]
/messages/by-id/CAMbWs499ytQiH4mLMhRxRWP-iEUz3-DSinpAD-cUCtVo_23Wtg@mail.gmail.com
Hello! In continuation of the topic, I, under the leadership of
Alexander Lakhin, prepared patches that fix these problems.
We decided that these checks would be enough and put them in the places
we saw fit.
Attachments:
stack_overflow_fix_15.patchtext/x-patch; charset=UTF-8; name=stack_overflow_fix_15.patchDownload
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index d0e5bc26a7..8c1691fdd9 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -3024,6 +3024,9 @@ CommitTransactionCommand(void)
TransactionState s = CurrentTransactionState;
SavedTransactionCharacteristics savetc;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
SaveTransactionCharacteristics(&savetc);
switch (s->blockState)
@@ -5471,6 +5474,9 @@ ShowTransactionStateRec(const char *str, TransactionState s)
{
StringInfoData buf;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
initStringInfo(&buf);
if (s->nChildXids > 0)
diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c
index 85e2a902a2..7a48ad63ef 100644
--- a/src/backend/catalog/dependency.c
+++ b/src/backend/catalog/dependency.c
@@ -75,6 +75,7 @@
#include "commands/trigger.h"
#include "commands/typecmds.h"
#include "funcapi.h"
+#include "miscadmin.h"
#include "nodes/nodeFuncs.h"
#include "parser/parsetree.h"
#include "rewrite/rewriteRemove.h"
@@ -516,6 +517,11 @@ findDependentObjects(const ObjectAddress *object,
if (stack_address_present_add_flags(object, objflags, stack))
return;
+ /* since this function recurses, it could be driven to stack overflow,
+ * because of the deep dependency tree, not only due to dependency loops.
+ */
+ check_stack_depth();
+
/*
* It's also possible that the target object has already been completely
* processed and put into targetObjects. If so, again we just add the
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index a21de89874..e46002c932 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -552,6 +552,9 @@ CheckAttributeType(const char *attname,
char att_typtype = get_typtype(atttypid);
Oid att_typelem;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
if (att_typtype == TYPTYPE_PSEUDO)
{
/*
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index fa4faff3c5..e194aff40e 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -6705,6 +6705,9 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
TupleDesc tupdesc;
FormData_pg_attribute *aattr[] = {&attribute};
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
/* At top level, permission check was done in ATPrepCmd, else do it */
if (recursing)
ATSimplePermissions((*cmd)->subtype, rel, ATT_TABLE | ATT_FOREIGN_TABLE);
@@ -8433,6 +8436,10 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/* Initialize addrs on the first invocation */
Assert(!recursing || addrs != NULL);
+
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
if (!recursing)
addrs = new_object_addresses();
@@ -10895,6 +10902,9 @@ ATExecAlterConstrRecurse(Constraint *cmdcon, Relation conrel, Relation tgrel,
Oid refrelid;
bool changed = false;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
currcon = (Form_pg_constraint) GETSTRUCT(contuple);
conoid = currcon->oid;
refrelid = currcon->confrelid;
@@ -11896,6 +11906,9 @@ ATExecDropConstraint(Relation rel, const char *constrName,
bool is_no_inherit_constraint = false;
char contype;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
/* At top level, permission check was done in ATPrepCmd, else do it */
if (recursing)
ATSimplePermissions(AT_DropConstraint, rel, ATT_TABLE | ATT_FOREIGN_TABLE);
diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c
index bf3a7cae60..e92d0316aa 100644
--- a/src/backend/optimizer/util/clauses.c
+++ b/src/backend/optimizer/util/clauses.c
@@ -2281,6 +2281,10 @@ static Node *
eval_const_expressions_mutator(Node *node,
eval_const_expressions_context *context)
{
+
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
if (node == NULL)
return NULL;
switch (nodeTag(node))
diff --git a/src/backend/utils/adt/jsonpath_exec.c b/src/backend/utils/adt/jsonpath_exec.c
index ae27baacce..8095f957c9 100644
--- a/src/backend/utils/adt/jsonpath_exec.c
+++ b/src/backend/utils/adt/jsonpath_exec.c
@@ -1231,6 +1231,9 @@ executeBoolItem(JsonPathExecContext *cxt, JsonPathItem *jsp,
JsonPathBool res;
JsonPathBool res2;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
if (!canHaveNext && jspHasNext(jsp))
elog(ERROR, "boolean jsonpath item cannot have next item");
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index e12be1b9bd..d754b51113 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -223,6 +223,9 @@ MemoryContextDelete(MemoryContext context)
/* And not CurrentMemoryContext, either */
Assert(context != CurrentMemoryContext);
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
/* save a function call in common case where there are no children */
if (context->firstchild != NULL)
MemoryContextDeleteChildren(context);
@@ -572,6 +575,9 @@ MemoryContextStatsInternal(MemoryContext context, int level,
AssertArg(MemoryContextIsValid(context));
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
/* Examine the context itself */
context->methods->stats(context,
print ? MemoryContextStatsPrint : NULL,
@@ -737,6 +743,9 @@ MemoryContextCheck(MemoryContext context)
AssertArg(MemoryContextIsValid(context));
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
context->methods->check(context);
for (child = context->firstchild; child != NULL; child = child->nextchild)
MemoryContextCheck(child);
stack_overflow_fix_master.patchtext/x-patch; charset=UTF-8; name=stack_overflow_fix_master.patchDownload
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index d85e313908..102d0e1574 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -3043,6 +3043,9 @@ CommitTransactionCommand(void)
TransactionState s = CurrentTransactionState;
SavedTransactionCharacteristics savetc;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
SaveTransactionCharacteristics(&savetc);
switch (s->blockState)
@@ -5500,6 +5503,9 @@ ShowTransactionStateRec(const char *str, TransactionState s)
{
StringInfoData buf;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
initStringInfo(&buf);
if (s->nChildXids > 0)
diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c
index 7acf654bf8..58a1d70b8a 100644
--- a/src/backend/catalog/dependency.c
+++ b/src/backend/catalog/dependency.c
@@ -76,6 +76,7 @@
#include "commands/trigger.h"
#include "commands/typecmds.h"
#include "funcapi.h"
+#include "miscadmin.h"
#include "nodes/nodeFuncs.h"
#include "parser/parsetree.h"
#include "rewrite/rewriteRemove.h"
@@ -524,6 +525,11 @@ findDependentObjects(const ObjectAddress *object,
if (stack_address_present_add_flags(object, objflags, stack))
return;
+ /* since this function recurses, it could be driven to stack overflow,
+ * because of the deep dependency tree, not only due to dependency loops.
+ */
+ check_stack_depth();
+
/*
* It's also possible that the target object has already been completely
* processed and put into targetObjects. If so, again we just add the
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index 4f006820b8..a88550913d 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -552,6 +552,9 @@ CheckAttributeType(const char *attname,
char att_typtype = get_typtype(atttypid);
Oid att_typelem;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
if (att_typtype == TYPTYPE_PSEUDO)
{
/*
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 7c697a285b..f22da79c65 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -6684,6 +6684,9 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
TupleDesc tupdesc;
FormData_pg_attribute *aattr[] = {&attribute};
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
/* At top level, permission check was done in ATPrepCmd, else do it */
if (recursing)
ATSimplePermissions((*cmd)->subtype, rel, ATT_TABLE | ATT_FOREIGN_TABLE);
@@ -8383,6 +8386,10 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/* Initialize addrs on the first invocation */
Assert(!recursing || addrs != NULL);
+
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
if (!recursing)
addrs = new_object_addresses();
@@ -10839,6 +10846,9 @@ ATExecAlterConstrRecurse(Constraint *cmdcon, Relation conrel, Relation tgrel,
Oid refrelid;
bool changed = false;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
currcon = (Form_pg_constraint) GETSTRUCT(contuple);
conoid = currcon->oid;
refrelid = currcon->confrelid;
@@ -11839,6 +11849,9 @@ ATExecDropConstraint(Relation rel, const char *constrName,
bool is_no_inherit_constraint = false;
char contype;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
/* At top level, permission check was done in ATPrepCmd, else do it */
if (recursing)
ATSimplePermissions(AT_DropConstraint, rel, ATT_TABLE | ATT_FOREIGN_TABLE);
diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c
index aa584848cf..77a5eb526c 100644
--- a/src/backend/optimizer/util/clauses.c
+++ b/src/backend/optimizer/util/clauses.c
@@ -2318,6 +2318,10 @@ static Node *
eval_const_expressions_mutator(Node *node,
eval_const_expressions_context *context)
{
+
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
if (node == NULL)
return NULL;
switch (nodeTag(node))
diff --git a/src/backend/utils/adt/jsonpath_exec.c b/src/backend/utils/adt/jsonpath_exec.c
index b561f0e7e8..dc7ab387ea 100644
--- a/src/backend/utils/adt/jsonpath_exec.c
+++ b/src/backend/utils/adt/jsonpath_exec.c
@@ -1232,6 +1232,9 @@ executeBoolItem(JsonPathExecContext *cxt, JsonPathItem *jsp,
JsonPathBool res;
JsonPathBool res2;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
if (!canHaveNext && jspHasNext(jsp))
elog(ERROR, "boolean jsonpath item cannot have next item");
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 0b00802df7..cc06c00a49 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -392,6 +392,9 @@ MemoryContextDelete(MemoryContext context)
/* And not CurrentMemoryContext, either */
Assert(context != CurrentMemoryContext);
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
/* save a function call in common case where there are no children */
if (context->firstchild != NULL)
MemoryContextDeleteChildren(context);
@@ -750,6 +753,9 @@ MemoryContextStatsInternal(MemoryContext context, int level,
Assert(MemoryContextIsValid(context));
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
/* Examine the context itself */
context->methods->stats(context,
print ? MemoryContextStatsPrint : NULL,
@@ -915,6 +921,9 @@ MemoryContextCheck(MemoryContext context)
Assert(MemoryContextIsValid(context));
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
context->methods->check(context);
for (child = context->firstchild; child != NULL; child = child->nextchild)
MemoryContextCheck(child);
stack_overflow_fix_11.patchtext/x-patch; charset=UTF-8; name=stack_overflow_fix_11.patchDownload
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index fd1a7ac5d5..318f620ed5 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -2812,6 +2812,9 @@ CommitTransactionCommand(void)
{
TransactionState s = CurrentTransactionState;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
switch (s->blockState)
{
/*
@@ -5194,6 +5197,9 @@ ShowTransactionStateRec(const char *str, TransactionState s)
{
StringInfoData buf;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
initStringInfo(&buf);
if (s->nChildXids > 0)
diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c
index f8510a1483..d1bbc11d3a 100644
--- a/src/backend/catalog/dependency.c
+++ b/src/backend/catalog/dependency.c
@@ -74,6 +74,7 @@
#include "nodes/nodeFuncs.h"
#include "parser/parsetree.h"
#include "rewrite/rewriteRemove.h"
+#include "miscadmin.h"
#include "storage/lmgr.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -489,6 +490,11 @@ findDependentObjects(const ObjectAddress *object,
if (stack_address_present_add_flags(object, objflags, stack))
return;
+ /* since this function recurses, it could be driven to stack overflow,
+ * because of the deep dependency tree, not only due to dependency loops.
+ */
+ check_stack_depth();
+
/*
* It's also possible that the target object has already been completely
* processed and put into targetObjects. If so, again we just add the
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index 846c530154..2f243bcf5e 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -513,6 +513,9 @@ CheckAttributeType(const char *attname,
char att_typtype = get_typtype(atttypid);
Oid att_typelem;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
if (att_typtype == TYPTYPE_PSEUDO)
{
/*
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index f56219973e..4cf08a1e04 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -5578,6 +5578,9 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
AclResult aclresult;
ObjectAddress address;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
/* At top level, permission check was done in ATPrepCmd, else do it */
if (recursing)
ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
@@ -7004,6 +7007,9 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
ObjectAddress object;
bool is_expr;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
/* At top level, permission check was done in ATPrepCmd, else do it */
if (recursing)
ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
@@ -8563,6 +8569,9 @@ ATExecAlterConstrRecurse(Constraint *cmdcon, Relation conrel, Relation tgrel,
Oid conoid;
bool changed = false;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
currcon = (Form_pg_constraint) GETSTRUCT(contuple);
conoid = HeapTupleGetOid(contuple);
@@ -9552,6 +9561,9 @@ ATExecDropConstraint(Relation rel, const char *constrName,
bool is_no_inherit_constraint = false;
char contype;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
/* At top level, permission check was done in ATPrepCmd, else do it */
if (recursing)
ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c
index edb56ab3a4..67b3a1f4bf 100644
--- a/src/backend/optimizer/util/clauses.c
+++ b/src/backend/optimizer/util/clauses.c
@@ -2650,6 +2650,10 @@ static Node *
eval_const_expressions_mutator(Node *node,
eval_const_expressions_context *context)
{
+
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
if (node == NULL)
return NULL;
switch (nodeTag(node))
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 22da98c19d..800c1ffac0 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -216,6 +216,9 @@ MemoryContextDelete(MemoryContext context)
/* And not CurrentMemoryContext, either */
Assert(context != CurrentMemoryContext);
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
/* save a function call in common case where there are no children */
if (context->firstchild != NULL)
MemoryContextDeleteChildren(context);
@@ -516,6 +519,9 @@ MemoryContextStatsInternal(MemoryContext context, int level,
AssertArg(MemoryContextIsValid(context));
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
/* Examine the context itself */
context->methods->stats(context,
print ? MemoryContextStatsPrint : NULL,
@@ -646,6 +652,9 @@ MemoryContextCheck(MemoryContext context)
AssertArg(MemoryContextIsValid(context));
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
context->methods->check(context);
for (child = context->firstchild; child != NULL; child = child->nextchild)
MemoryContextCheck(child);
stack_overflow_fix_12_13_14.patchtext/x-patch; charset=UTF-8; name=stack_overflow_fix_12_13_14.patchDownload
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index b304a85d08..387e8acd27 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -2913,6 +2913,9 @@ CommitTransactionCommand(void)
{
TransactionState s = CurrentTransactionState;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
if (s->chain)
SaveTransactionCharacteristics();
@@ -5361,6 +5364,9 @@ ShowTransactionStateRec(const char *str, TransactionState s)
{
StringInfoData buf;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
initStringInfo(&buf);
if (s->nChildXids > 0)
diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c
index 46ced9bb60..3115476632 100644
--- a/src/backend/catalog/dependency.c
+++ b/src/backend/catalog/dependency.c
@@ -73,6 +73,7 @@
#include "commands/sequence.h"
#include "commands/trigger.h"
#include "commands/typecmds.h"
+#include "miscadmin.h"
#include "nodes/nodeFuncs.h"
#include "parser/parsetree.h"
#include "rewrite/rewriteRemove.h"
@@ -509,6 +510,11 @@ findDependentObjects(const ObjectAddress *object,
if (stack_address_present_add_flags(object, objflags, stack))
return;
+ /* since this function recurses, it could be driven to stack overflow,
+ * because of the deep dependency tree, not only due to dependency loops.
+ */
+ check_stack_depth();
+
/*
* It's also possible that the target object has already been completely
* processed and put into targetObjects. If so, again we just add the
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index ceccbdb8fe..395e5edd71 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -595,6 +595,9 @@ CheckAttributeType(const char *attname,
char att_typtype = get_typtype(atttypid);
Oid att_typelem;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
if (att_typtype == TYPTYPE_PSEUDO)
{
/*
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index a57fe61187..1dc23f23db 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -5900,4 +5900,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
/* At top level, permission check was done in ATPrepCmd, else do it */
if (recursing)
ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
@@ -7460,6 +7463,10 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/* Initialize addrs on the first invocation */
Assert(!recursing || addrs != NULL);
+
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
if (!recursing)
addrs = new_object_addresses();
@@ -9587,6 +9594,9 @@ ATExecAlterConstrRecurse(Constraint *cmdcon, Relation conrel, Relation tgrel,
Oid refrelid;
bool changed = false;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
currcon = (Form_pg_constraint) GETSTRUCT(contuple);
conoid = currcon->oid;
refrelid = currcon->confrelid;
@@ -10556,6 +10566,9 @@ ATExecDropConstraint(Relation rel, const char *constrName,
bool is_no_inherit_constraint = false;
char contype;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
/* At top level, permission check was done in ATPrepCmd, else do it */
if (recursing)
ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c
index 946e232cca..726b30a4e2 100644
--- a/src/backend/optimizer/util/clauses.c
+++ b/src/backend/optimizer/util/clauses.c
@@ -2392,6 +2392,10 @@ static Node *
eval_const_expressions_mutator(Node *node,
eval_const_expressions_context *context)
{
+
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
if (node == NULL)
return NULL;
switch (nodeTag(node))
diff --git a/src/backend/utils/adt/jsonpath_exec.c b/src/backend/utils/adt/jsonpath_exec.c
index 37b08f1742..e115b03a3d 100644
--- a/src/backend/utils/adt/jsonpath_exec.c
+++ b/src/backend/utils/adt/jsonpath_exec.c
@@ -1160,6 +1160,9 @@ executeBoolItem(JsonPathExecContext *cxt, JsonPathItem *jsp,
JsonPathBool res;
JsonPathBool res2;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
if (!canHaveNext && jspHasNext(jsp))
elog(ERROR, "boolean jsonpath item cannot have next item");
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index b07be12236..16ecf0538c 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -216,6 +216,9 @@ MemoryContextDelete(MemoryContext context)
/* And not CurrentMemoryContext, either */
Assert(context != CurrentMemoryContext);
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
/* save a function call in common case where there are no children */
if (context->firstchild != NULL)
MemoryContextDeleteChildren(context);
@@ -516,6 +519,9 @@ MemoryContextStatsInternal(MemoryContext context, int level,
AssertArg(MemoryContextIsValid(context));
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
/* Examine the context itself */
context->methods->stats(context,
print ? MemoryContextStatsPrint : NULL,
@@ -646,6 +652,9 @@ MemoryContextCheck(MemoryContext context)
AssertArg(MemoryContextIsValid(context));
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
context->methods->check(context);
for (child = context->firstchild; child != NULL; child = child->nextchild)
MemoryContextCheck(child);
03.01.2023 22:45, Sascha Kuhl writes:
Great work. Max Stack depth is memory dependent? Processor dependent?
Hello! These situations are not specific to the x86_64 architecture, but
also manifest themselves, for example, on aarch64 architecture.
For example this query, ran on aarch64, (n=1000000;printf "begin;"; for
((i=1;i<=$n;i++)); do printf "savepoint s$i;"; done; printf "release
s1;" ) | psql > /dev/null
crashed the server on the savepoint174617 with the following stacktrace:
Core was generated by `postgres: test test [local]
SAVEPOINT '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 AllocSetCheck (context=<error reading variable: Cannot access memory
at address 0xffffe2397fe8>) at aset.c:1409
1409 {
(gdb) bt
#0 AllocSetCheck (context=<error reading variable: Cannot access memory
at address 0xffffe2397fe8>) at aset.c:1409
#1 0x0000aaaad78c38c4 in MemoryContextCheck (context=0xaaab39ee16a0) at
mcxt.c:740
#2 0x0000aaaad78c38dc in MemoryContextCheck (context=0xaaab39edf690) at
mcxt.c:742
#3 0x0000aaaad78c38dc in MemoryContextCheck (context=0xaaab39edd680) at
mcxt.c:742
#4 0x0000aaaad78c38dc in MemoryContextCheck (context=0xaaab39edb670) at
mcxt.c:742
#5 0x0000aaaad78c38dc in MemoryContextCheck (context=0xaaab39ed9660) at
mcxt.c:742
#6 0x0000aaaad78c38dc in MemoryContextCheck (context=0xaaab39ed7650) at
mcxt.c:742
#7 0x0000aaaad78c38dc in MemoryContextCheck (context=0xaaab39ed5640) at
mcxt.c:742
#8 0x0000aaaad78c38dc in MemoryContextCheck (context=0xaaab39ed3630) at
mcxt.c:742
#9 0x0000aaaad78c38dc in MemoryContextCheck (context=0xaaab39ed1620) at
mcxt.c:742
#10 0x0000aaaad78c38dc in MemoryContextCheck (context=0xaaab39ecf610) at
mcxt.c:742
...
#174617 0x0000aaaad78c38dc in MemoryContextCheck
(context=0xaaaae47994b0) at mcxt.c:742
#174618 0x0000aaaad78c38dc in MemoryContextCheck
(context=0xaaaae476dcd0) at mcxt.c:742
#174619 0x0000aaaad78c38dc in MemoryContextCheck
(context=0xaaaae46ead50) at mcxt.c:742
#174620 0x0000aaaad76c7e24 in finish_xact_command () at postgres.c:2739
#174621 0x0000aaaad76c55b8 in exec_simple_query
(query_string=0xaaaae46f0540 "savepoint s174617;") at postgres.c:1238
#174622 0x0000aaaad76ca7a4 in PostgresMain (argc=1, argv=0xffffe2b96898,
dbname=0xaaaae471c098 "test", username=0xaaaae471c078 "test") at
postgres.c:4508
#174623 0x0000aaaad75e263c in BackendRun (port=0xaaaae4711470) at
postmaster.c:4530
#174624 0x0000aaaad75e1f70 in BackendStartup (port=0xaaaae4711470) at
postmaster.c:4252
#174625 0x0000aaaad75dd4c0 in ServerLoop () at postmaster.c:1745
#174626 0x0000aaaad75dcd3c in PostmasterMain (argc=3,
argv=0xaaaae46eacb0) at postmaster.c:1417
#174627 0x0000aaaad74d462c in main (argc=3, argv=0xaaaae46eacb0) at
main.c:209
Hello! In continuation of the topic I would like to suggest solution.
This patch adds several checks to the vulnerable functions above.
Attachments:
v1-0001-Add-some-checks-to-avoid-stack-overflow.patchtext/x-patch; charset=UTF-8; name=v1-0001-Add-some-checks-to-avoid-stack-overflow.patchDownload
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index d85e313908..102d0e1574 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -3043,6 +3043,9 @@ CommitTransactionCommand(void)
TransactionState s = CurrentTransactionState;
SavedTransactionCharacteristics savetc;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
SaveTransactionCharacteristics(&savetc);
switch (s->blockState)
@@ -5500,6 +5503,9 @@ ShowTransactionStateRec(const char *str, TransactionState s)
{
StringInfoData buf;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
initStringInfo(&buf);
if (s->nChildXids > 0)
diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c
index 7acf654bf8..58a1d70b8a 100644
--- a/src/backend/catalog/dependency.c
+++ b/src/backend/catalog/dependency.c
@@ -76,6 +76,7 @@
#include "commands/trigger.h"
#include "commands/typecmds.h"
#include "funcapi.h"
+#include "miscadmin.h"
#include "nodes/nodeFuncs.h"
#include "parser/parsetree.h"
#include "rewrite/rewriteRemove.h"
@@ -524,6 +525,11 @@ findDependentObjects(const ObjectAddress *object,
if (stack_address_present_add_flags(object, objflags, stack))
return;
+ /* since this function recurses, it could be driven to stack overflow,
+ * because of the deep dependency tree, not only due to dependency loops.
+ */
+ check_stack_depth();
+
/*
* It's also possible that the target object has already been completely
* processed and put into targetObjects. If so, again we just add the
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index 4f006820b8..a88550913d 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -552,6 +552,9 @@ CheckAttributeType(const char *attname,
char att_typtype = get_typtype(atttypid);
Oid att_typelem;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
if (att_typtype == TYPTYPE_PSEUDO)
{
/*
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 7c697a285b..f22da79c65 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -6684,6 +6684,9 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
TupleDesc tupdesc;
FormData_pg_attribute *aattr[] = {&attribute};
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
/* At top level, permission check was done in ATPrepCmd, else do it */
if (recursing)
ATSimplePermissions((*cmd)->subtype, rel, ATT_TABLE | ATT_FOREIGN_TABLE);
@@ -8383,6 +8386,10 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/* Initialize addrs on the first invocation */
Assert(!recursing || addrs != NULL);
+
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
if (!recursing)
addrs = new_object_addresses();
@@ -10839,6 +10846,9 @@ ATExecAlterConstrRecurse(Constraint *cmdcon, Relation conrel, Relation tgrel,
Oid refrelid;
bool changed = false;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
currcon = (Form_pg_constraint) GETSTRUCT(contuple);
conoid = currcon->oid;
refrelid = currcon->confrelid;
@@ -11839,6 +11849,9 @@ ATExecDropConstraint(Relation rel, const char *constrName,
bool is_no_inherit_constraint = false;
char contype;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
/* At top level, permission check was done in ATPrepCmd, else do it */
if (recursing)
ATSimplePermissions(AT_DropConstraint, rel, ATT_TABLE | ATT_FOREIGN_TABLE);
diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c
index aa584848cf..77a5eb526c 100644
--- a/src/backend/optimizer/util/clauses.c
+++ b/src/backend/optimizer/util/clauses.c
@@ -2318,6 +2318,10 @@ static Node *
eval_const_expressions_mutator(Node *node,
eval_const_expressions_context *context)
{
+
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
if (node == NULL)
return NULL;
switch (nodeTag(node))
diff --git a/src/backend/utils/adt/jsonpath_exec.c b/src/backend/utils/adt/jsonpath_exec.c
index b561f0e7e8..dc7ab387ea 100644
--- a/src/backend/utils/adt/jsonpath_exec.c
+++ b/src/backend/utils/adt/jsonpath_exec.c
@@ -1232,6 +1232,9 @@ executeBoolItem(JsonPathExecContext *cxt, JsonPathItem *jsp,
JsonPathBool res;
JsonPathBool res2;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
if (!canHaveNext && jspHasNext(jsp))
elog(ERROR, "boolean jsonpath item cannot have next item");
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 0b00802df7..cc06c00a49 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -392,6 +392,9 @@ MemoryContextDelete(MemoryContext context)
/* And not CurrentMemoryContext, either */
Assert(context != CurrentMemoryContext);
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
/* save a function call in common case where there are no children */
if (context->firstchild != NULL)
MemoryContextDeleteChildren(context);
@@ -750,6 +753,9 @@ MemoryContextStatsInternal(MemoryContext context, int level,
Assert(MemoryContextIsValid(context));
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
/* Examine the context itself */
context->methods->stats(context,
print ? MemoryContextStatsPrint : NULL,
@@ -915,6 +921,9 @@ MemoryContextCheck(MemoryContext context)
Assert(MemoryContextIsValid(context));
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
context->methods->check(context);
for (child = context->firstchild; child != NULL; child = child->nextchild)
MemoryContextCheck(child);
On 21/06/2023 16:45, Egor Chindyaskin wrote:
Hello! In continuation of the topic I would like to suggest solution.
This patch adds several checks to the vulnerable functions above.
I looked at this last patch. The depth checks are clearly better than
segfaulting, but I think we can also avoid the recursions and having to
error out. That seems nice especially for MemoryContextDelete(), which
is called at transaction cleanup.
1. CommitTransactionCommand
This is just tail recursion. The compiler will almost certainly optimize
it away unless you're using -O0. We can easily turn it into iteration
ourselves to avoid that hazard, per attached
0001-Turn-tail-recursion-into-iteration-in-CommitTransact.patch.
2. ShowTransactionStateRec
Since this is just a debugging aid, I think we can just stop recursing
if we're about to run out of stack space. Seems nicer than erroring out,
although it can still error if you run out of memory. See
0002-Avoid-stack-overflow-in-ShowTransactionStateRec.patch.
3. All the MemoryContext functions
I'm reluctant to add stack checks to these, because they are called in
places like cleaning up after transaction abort. MemoryContextDelete()
in particular. If you ereport an error, it's not clear that you can
recover cleanly; you'll leak memory if nothing else.
Fortunately MemoryContext contains pointers to parent and siblings, so
we can traverse a tree of MemoryContexts iteratively, without using stack.
MemoryContextStats() is a bit tricky, but we can put a limit on how much
it recurses, and just print a summary line if the limit is reached.
That's what we already do if a memory context has a lot of children.
(Actually, if we didn't try keep track of the # of children at each
level, to trigger the summarization, we could traverse the tree without
using stack. But a limit seems useful.)
What do you think?
--
Heikki Linnakangas
Neon (https://neon.tech)
Attachments:
0001-Turn-tail-recursion-into-iteration-in-CommitTransact.patchtext/x-patch; charset=UTF-8; name=0001-Turn-tail-recursion-into-iteration-in-CommitTransact.patchDownload
From 543e6569a3f4730f3105379cc946cc6b54525179 Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <heikki.linnakangas@iki.fi>
Date: Fri, 24 Nov 2023 14:13:10 +0200
Subject: [PATCH 1/4] Turn tail recursion into iteration in
CommitTransactionCommand()
Usually the compiler will optimize away the tail recursion anyway, but
if it doesn't, you can drive the function into stack overflow. For
example:
(n=1000000; printf "BEGIN;"; for ((i=1;i<=$n;i++)); do printf "SAVEPOINT s$i;"; done; printf "ERROR; COMMIT;") | psql >/dev/null
Report by Egor Chindyaskin and Alexander Lakhin.
Discussion: https://www.postgresql.org/message-id/1672760457.940462079%40f306.i.mail.ru
---
src/backend/access/transam/xact.c | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 536edb3792..e16b73a9f1 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -3033,12 +3033,15 @@ RestoreTransactionCharacteristics(const SavedTransactionCharacteristics *s)
void
CommitTransactionCommand(void)
{
- TransactionState s = CurrentTransactionState;
+ TransactionState s;
SavedTransactionCharacteristics savetc;
+recurse:
+ s = CurrentTransactionState;
/* Must save in case we need to restore below */
SaveTransactionCharacteristics(&savetc);
+ s = CurrentTransactionState;
switch (s->blockState)
{
/*
@@ -3226,8 +3229,7 @@ CommitTransactionCommand(void)
*/
case TBLOCK_SUBABORT_END:
CleanupSubTransaction();
- CommitTransactionCommand();
- break;
+ goto recurse;
/*
* As above, but it's not dead yet, so abort first.
@@ -3235,8 +3237,7 @@ CommitTransactionCommand(void)
case TBLOCK_SUBABORT_PENDING:
AbortSubTransaction();
CleanupSubTransaction();
- CommitTransactionCommand();
- break;
+ goto recurse;
/*
* The current subtransaction is the target of a ROLLBACK TO
--
2.39.2
0002-Avoid-stack-overflow-in-ShowTransactionStateRec.patchtext/x-patch; charset=UTF-8; name=0002-Avoid-stack-overflow-in-ShowTransactionStateRec.patchDownload
From 1da20313b6252346fbc3f46cdfde612532e98c84 Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <heikki.linnakangas@iki.fi>
Date: Fri, 24 Nov 2023 15:06:53 +0200
Subject: [PATCH 2/4] Avoid stack overflow in ShowTransactionStateRec()
The function recurses, but didn't perform stack-depth checks. It's
just a debugging aid, so instead of the usual check_stack_depth()
call, stop the printing if we'd risk stack overflow.
Here's an example of how to test this:
(n=1000000; printf "BEGIN;"; for ((i=1;i<=$n;i++)); do printf "SAVEPOINT s$i;"; done; printf "SET log_min_messages = 'DEBUG5'; SAVEPOINT sp;") | psql >/dev/null
In the passing, swap building the list of child XIDs and recursing to
parent. That saves memory while recursing, reducing the risk of out of
memory errors with lots of subtransactions. The saving is not very
significant in practice, but this order seems more logical anyway.
Report by Egor Chindyaskin and Alexander Lakhin.
Discussion: https://www.postgresql.org/message-id/1672760457.940462079%40f306.i.mail.ru
---
src/backend/access/transam/xact.c | 21 +++++++++++++++------
1 file changed, 15 insertions(+), 6 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index e16b73a9f1..53584f7dc6 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -5507,8 +5507,22 @@ ShowTransactionStateRec(const char *str, TransactionState s)
{
StringInfoData buf;
- initStringInfo(&buf);
+ if (s->parent)
+ {
+ /*
+ * Since this function recurses, it could be driven to stack overflow.
+ * This is just a debugging aid, so we can leave out some details
+ * instead of erroring out with check_stack_depth().
+ */
+ if (stack_is_too_deep())
+ ereport(DEBUG5,
+ (errmsg_internal("%s(%d): parent omitted to avoid stack overflow",
+ str, s->nestingLevel)));
+ else
+ ShowTransactionStateRec(str, s->parent);
+ }
+ initStringInfo(&buf);
if (s->nChildXids > 0)
{
int i;
@@ -5517,10 +5531,6 @@ ShowTransactionStateRec(const char *str, TransactionState s)
for (i = 1; i < s->nChildXids; i++)
appendStringInfo(&buf, " %u", s->childXids[i]);
}
-
- if (s->parent)
- ShowTransactionStateRec(str, s->parent);
-
ereport(DEBUG5,
(errmsg_internal("%s(%d) name: %s; blockState: %s; state: %s, xid/subid/cid: %u/%u/%u%s%s",
str, s->nestingLevel,
@@ -5532,7 +5542,6 @@ ShowTransactionStateRec(const char *str, TransactionState s)
(unsigned int) currentCommandId,
currentCommandIdUsed ? " (used)" : "",
buf.data)));
-
pfree(buf.data);
}
--
2.39.2
0003-Avoid-recursion-in-MemoryContext-functions.patchtext/x-patch; charset=UTF-8; name=0003-Avoid-recursion-in-MemoryContext-functions.patchDownload
From be7a1be658479de72ee694e41a8ce0f07b8b40ad Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <heikki.linnakangas@iki.fi>
Date: Fri, 24 Nov 2023 17:03:28 +0200
Subject: [PATCH 3/4] Avoid recursion in MemoryContext functions.
You might run out of stack space with recursion, which is not nice in
functions that might be used e.g. at cleanup after transaction
abort. MemoryContext contains pointer to parent and siblings, so we
can traverse a tree of contexts iteratively, without using
stack. Refactor the functions to do that.
MemoryContextStats() still recurses, but it now has a limit to how
deep it recurses. Once the limit is reached, it prints just a summary
of the rest of the hierarchy, similar to how it summarizes contexts
with lots of children. That seems good anyway, because a context dump
with hundreds of nested contexts isn't very readable.
Report by Egor Chindyaskin and Alexander Lakhin.
Discussion: https://www.postgresql.org/message-id/1672760457.940462079%40f306.i.mail.ru
---
src/backend/utils/mmgr/mcxt.c | 244 +++++++++++++++++++++++-----------
src/include/utils/memutils.h | 3 +-
2 files changed, 167 insertions(+), 80 deletions(-)
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 9fc83f11f6..2e9563d8c1 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -149,9 +149,10 @@ MemoryContext CurTransactionContext = NULL;
/* This is a transient link to the active portal's memory context: */
MemoryContext PortalContext = NULL;
+static void MemoryContextDeleteOnly(MemoryContext context);
static void MemoryContextCallResetCallbacks(MemoryContext context);
static void MemoryContextStatsInternal(MemoryContext context, int level,
- bool print, int max_children,
+ int max_level, int max_children,
MemoryContextCounters *totals,
bool print_to_stderr);
static void MemoryContextStatsPrint(MemoryContext context, void *passthru,
@@ -260,6 +261,41 @@ BogusGetChunkSpace(void *pointer)
return 0; /* keep compiler quiet */
}
+/*
+ * MemoryContextTraverse
+ * Helper function to traverse all descendants of a memory context
+ * without recursion.
+ *
+ * Recursion could lead to out-of-stack errors with deep context hierarchies,
+ * which would be unpleasant in error cleanup code paths.
+ *
+ * To process 'context' and all its descendants, use a loop like this:
+ *
+ * <process 'context'>
+ * for (MemoryContext curr = context->firstchild;
+ * curr != NULL;
+ * curr = MemoryContextTraverse(curr, context))
+ * {
+ * <process 'curr'>
+ * }
+ *
+ * This visits all the contexts in preorder.
+ */
+static MemoryContext
+MemoryContextTraverse(MemoryContext curr, MemoryContext top)
+{
+ if (curr->firstchild != NULL)
+ return curr->firstchild;
+
+ while (curr->nextchild == NULL)
+ {
+ curr = curr->parent;
+ if (curr == top)
+ return NULL;
+ }
+ return curr->nextchild;
+}
+
/*****************************************************************************
* EXPORTED ROUTINES *
@@ -379,14 +415,13 @@ MemoryContextResetOnly(MemoryContext context)
void
MemoryContextResetChildren(MemoryContext context)
{
- MemoryContext child;
-
Assert(MemoryContextIsValid(context));
- for (child = context->firstchild; child != NULL; child = child->nextchild)
+ for (MemoryContext curr = context->firstchild;
+ curr != NULL;
+ curr = MemoryContextTraverse(curr, context))
{
- MemoryContextResetChildren(child);
- MemoryContextResetOnly(child);
+ MemoryContextResetOnly(curr);
}
}
@@ -401,16 +436,53 @@ MemoryContextResetChildren(MemoryContext context)
*/
void
MemoryContextDelete(MemoryContext context)
+{
+ MemoryContext curr;
+
+ /*
+ * Delete subcontexts from the bottom up.
+ *
+ * Note: Do not use recursion here. A "stack depth limit exceeded" error
+ * would be unpleasant if we're already in the process of cleaning up from
+ * transaction abort. We also cannot use MemoryContextTraverse() here
+ * because we modify the tree as we go.
+ */
+ curr = context;
+ for (;;)
+ {
+ MemoryContext parent;
+
+ /* Descend down until we find a leaf context with no children */
+ while (curr->firstchild != NULL)
+ curr = curr->firstchild;
+
+ /*
+ * We're now at a leaf with no children. Free it and continue from the
+ * parent. Or if this was the original node, we're all done.
+ */
+ parent = curr->parent;
+ MemoryContextDeleteOnly(curr);
+
+ if (curr == context)
+ break;
+ curr = parent;
+ }
+}
+
+/*
+ * Subroutine of MemoryContextDelete, to delete a context that has no
+ * children.
+ */
+static void
+MemoryContextDeleteOnly(MemoryContext context)
{
Assert(MemoryContextIsValid(context));
/* We had better not be deleting TopMemoryContext ... */
Assert(context != TopMemoryContext);
/* And not CurrentMemoryContext, either */
Assert(context != CurrentMemoryContext);
-
- /* save a function call in common case where there are no children */
- if (context->firstchild != NULL)
- MemoryContextDeleteChildren(context);
+ /* All the children should've been deleted already */
+ Assert(context->firstchild == NULL);
/*
* It's not entirely clear whether 'tis better to do this before or after
@@ -676,12 +748,12 @@ MemoryContextMemAllocated(MemoryContext context, bool recurse)
if (recurse)
{
- MemoryContext child;
-
- for (child = context->firstchild;
- child != NULL;
- child = child->nextchild)
- total += MemoryContextMemAllocated(child, true);
+ for (MemoryContext curr = context->firstchild;
+ curr != NULL;
+ curr = MemoryContextTraverse(curr, context))
+ {
+ total += curr->mem_allocated;
+ }
}
return total;
@@ -698,8 +770,8 @@ MemoryContextMemAllocated(MemoryContext context, bool recurse)
void
MemoryContextStats(MemoryContext context)
{
- /* A hard-wired limit on the number of children is usually good enough */
- MemoryContextStatsDetail(context, 100, true);
+ /* Hard-wired limits are usually good enough */
+ MemoryContextStatsDetail(context, 100, 100, true);
}
/*
@@ -711,14 +783,15 @@ MemoryContextStats(MemoryContext context)
* with fprintf(stderr), otherwise use ereport().
*/
void
-MemoryContextStatsDetail(MemoryContext context, int max_children,
+MemoryContextStatsDetail(MemoryContext context,
+ int max_level, int max_children,
bool print_to_stderr)
{
MemoryContextCounters grand_totals;
memset(&grand_totals, 0, sizeof(grand_totals));
- MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals, print_to_stderr);
+ MemoryContextStatsInternal(context, 0, max_level, max_children, &grand_totals, print_to_stderr);
if (print_to_stderr)
fprintf(stderr,
@@ -756,78 +829,89 @@ MemoryContextStatsDetail(MemoryContext context, int max_children,
*/
static void
MemoryContextStatsInternal(MemoryContext context, int level,
- bool print, int max_children,
+ int max_level, int max_children,
MemoryContextCounters *totals,
bool print_to_stderr)
{
- MemoryContextCounters local_totals;
MemoryContext child;
int ichild;
Assert(MemoryContextIsValid(context));
/* Examine the context itself */
- context->methods->stats(context,
- print ? MemoryContextStatsPrint : NULL,
- (void *) &level,
- totals, print_to_stderr);
+ {
+ context->methods->stats(context,
+ MemoryContextStatsPrint,
+ (void *) &level,
+ totals,
+ print_to_stderr);
+ }
/*
- * Examine children. If there are more than max_children of them, we do
- * not print the rest explicitly, but just summarize them.
+ * Examine children.
+ *
+ * If we are past the recursion depth limit or already running low on
+ * stack, do not print them explicitly but just summarize them. Similarly,
+ * if there are more than max_children of them, we do not print the rest
+ * explicitly, but just summarize them.
*/
- memset(&local_totals, 0, sizeof(local_totals));
-
- for (child = context->firstchild, ichild = 0;
- child != NULL;
- child = child->nextchild, ichild++)
+ ichild = 0;
+ child = context->firstchild;
+ if (level < max_level && !stack_is_too_deep())
{
- if (ichild < max_children)
+ for (; child != NULL && ichild < max_children;
+ child = child->nextchild)
+ {
MemoryContextStatsInternal(child, level + 1,
- print, max_children,
+ max_level, max_children,
totals,
print_to_stderr);
- else
- MemoryContextStatsInternal(child, level + 1,
- false, max_children,
- &local_totals,
- print_to_stderr);
+ ichild++;
+ }
}
- /* Deal with excess children */
- if (ichild > max_children)
+ if (child != NULL)
{
- if (print)
+ /* Summarize the rest of the children, avoiding recursion. */
+ MemoryContextCounters local_totals;
+
+ memset(&local_totals, 0, sizeof(local_totals));
+
+ while (child != NULL)
{
- if (print_to_stderr)
- {
- int i;
-
- for (i = 0; i <= level; i++)
- fprintf(stderr, " ");
- fprintf(stderr,
- "%d more child contexts containing %zu total in %zu blocks; %zu free (%zu chunks); %zu used\n",
- ichild - max_children,
- local_totals.totalspace,
- local_totals.nblocks,
- local_totals.freespace,
- local_totals.freechunks,
- local_totals.totalspace - local_totals.freespace);
- }
- else
- ereport(LOG_SERVER_ONLY,
- (errhidestmt(true),
- errhidecontext(true),
- errmsg_internal("level: %d; %d more child contexts containing %zu total in %zu blocks; %zu free (%zu chunks); %zu used",
- level,
- ichild - max_children,
- local_totals.totalspace,
- local_totals.nblocks,
- local_totals.freespace,
- local_totals.freechunks,
- local_totals.totalspace - local_totals.freespace)));
+ child->methods->stats(child, NULL, NULL, &local_totals, print_to_stderr);
+ child = MemoryContextTraverse(child, context);
+ ichild++;
}
+ if (print_to_stderr)
+ {
+ int i;
+
+ for (i = 0; i <= level; i++)
+ fprintf(stderr, " ");
+ fprintf(stderr,
+ "%d more child contexts containing %zu total in %zu blocks; %zu free (%zu chunks); %zu used\n",
+ ichild - max_children,
+ local_totals.totalspace,
+ local_totals.nblocks,
+ local_totals.freespace,
+ local_totals.freechunks,
+ local_totals.totalspace - local_totals.freespace);
+ }
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("level: %d; %d more child contexts containing %zu total in %zu blocks; %zu free (%zu chunks); %zu used",
+ level,
+ ichild - max_children,
+ local_totals.totalspace,
+ local_totals.nblocks,
+ local_totals.freespace,
+ local_totals.freechunks,
+ local_totals.totalspace - local_totals.freespace)));
+
if (totals)
{
totals->nblocks += local_totals.nblocks;
@@ -847,8 +931,7 @@ MemoryContextStatsInternal(MemoryContext context, int level,
*/
static void
MemoryContextStatsPrint(MemoryContext context, void *passthru,
- const char *stats_string,
- bool print_to_stderr)
+ const char *stats_string, bool print_to_stderr)
{
int level = *(int *) passthru;
const char *name = context->name;
@@ -927,13 +1010,15 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
void
MemoryContextCheck(MemoryContext context)
{
- MemoryContext child;
-
Assert(MemoryContextIsValid(context));
-
context->methods->check(context);
- for (child = context->firstchild; child != NULL; child = child->nextchild)
- MemoryContextCheck(child);
+
+ for (MemoryContext curr = context->firstchild;
+ curr != NULL;
+ curr = MemoryContextTraverse(curr, context))
+ {
+ curr->methods->check(curr);
+ }
}
#endif
@@ -1212,14 +1297,15 @@ ProcessLogMemoryContextInterrupt(void)
/*
* When a backend process is consuming huge memory, logging all its memory
* contexts might overrun available disk space. To prevent this, we limit
- * the number of child contexts to log per parent to 100.
+ * the depth of the hierarchy, as well as the number of child contexts to
+ * log per parent to 100.
*
* As with MemoryContextStats(), we suppose that practical cases where the
* dump gets long will typically be huge numbers of siblings under the
* same parent context; while the additional debugging value from seeing
* details about individual siblings beyond 100 will not be large.
*/
- MemoryContextStatsDetail(TopMemoryContext, 100, false);
+ MemoryContextStatsDetail(TopMemoryContext, 100, 100, false);
}
void *
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index d14e8546a6..c25f982dc6 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -85,7 +85,8 @@ extern MemoryContext MemoryContextGetParent(MemoryContext context);
extern bool MemoryContextIsEmpty(MemoryContext context);
extern Size MemoryContextMemAllocated(MemoryContext context, bool recurse);
extern void MemoryContextStats(MemoryContext context);
-extern void MemoryContextStatsDetail(MemoryContext context, int max_children,
+extern void MemoryContextStatsDetail(MemoryContext context,
+ int max_level, int max_children,
bool print_to_stderr);
extern void MemoryContextAllowInCriticalSection(MemoryContext context,
bool allow);
--
2.39.2
On 24/11/2023 21:14, Heikki Linnakangas wrote:
What do you think?
Hello! Thank you for researching the problem! I'm more of a tester than
a developer, so I was able to check the patches from that side.
I've configured the server with CFLAGS=" -O0" and cassert enabled and
checked the following queries:
#CommitTransactionCommand
(n=1000000; printf "BEGIN;"; for ((i=1;i<=$n;i++)); do printf "SAVEPOINT
s$i;"; done; printf "ERROR; COMMIT;") | psql >/dev/null
#ShowTransactionStateRec
(n=1000000; printf "BEGIN;"; for ((i=1;i<=$n;i++)); do printf "SAVEPOINT
s$i;"; done; printf "SET log_min_messages = 'DEBUG5'; SAVEPOINT sp;") |
psql >/dev/null
#MemoryContextCheck
(n=1000000; printf "begin;"; for ((i=1;i<=$n;i++)); do printf "savepoint
s$i;"; done; printf "release s1;" ) | psql >/dev/null
#MemoryContextStatsInternal
(n=1000000; printf "BEGIN;"; for ((i=1;i<=$n;i++)); do printf "SAVEPOINT
s$i;"; done; printf "SELECT
pg_log_backend_memory_contexts(pg_backend_pid())") | psql >/dev/null
On my system, every of that queries led to a server crash at a number of
savepoints in the range from 174,400 to 174,700.
With your patches applied, the savepoint counter goes well beyond these
values, I settled on an amount of approximately 300,000 savepoints.
Your patches look good to me.
Best regards,
Egor Chindyaskin
Postgres Professional: http://postgrespro.com/
On Fri, Nov 24, 2023 at 10:47 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
What do you think?
At least for 0001 and 0002, I think we should just add the stack depth checks.
With regard to 0001, CommitTransactionCommand() and friends are hard
enough to understand as it is; they need "goto" like I need an extra
hole in my head.
With regard to 0002, this function isn't sufficiently important to
justify adding special-case code for an extremely rare event. We
should just handle it the way we do in general.
I agree that in the memory-context case it might be worth expending
some more code to be more clever. But I probably wouldn't do that for
MemoryContextStats(); check_stack_depth() seems fine for that one.
In general, I think we should try to keep the number of places that
handle stack overflow in "special" ways as small as possible.
--
Robert Haas
EDB: http://www.enterprisedb.com
Hi,
On 2024-01-05 12:23:25 -0500, Robert Haas wrote:
I agree that in the memory-context case it might be worth expending
some more code to be more clever. But I probably wouldn't do that for
MemoryContextStats(); check_stack_depth() seems fine for that one.
We run MemoryContextStats() when we fail to allocate memory, including during
abort processing after a previous error. So I think it qualifies for being
somewhat special. Thus I suspect check_stack_depth() wouldn't be a good idea -
but we could make the stack_is_too_deep() path simpler and just return in the
existing MemoryContextStatsInternal() when that's the case.
Greetings,
Andres Freund
On Fri, Jan 5, 2024 at 3:16 PM Andres Freund <andres@anarazel.de> wrote:
On 2024-01-05 12:23:25 -0500, Robert Haas wrote:
I agree that in the memory-context case it might be worth expending
some more code to be more clever. But I probably wouldn't do that for
MemoryContextStats(); check_stack_depth() seems fine for that one.We run MemoryContextStats() when we fail to allocate memory, including during
abort processing after a previous error. So I think it qualifies for being
somewhat special.
OK.
Thus I suspect check_stack_depth() wouldn't be a good idea -
but we could make the stack_is_too_deep() path simpler and just return in the
existing MemoryContextStatsInternal() when that's the case.
Since this kind of code will be exercised so rarely, it's highly
vulnerable to bugs, so I favor keeping it as simple as we can.
--
Robert Haas
EDB: http://www.enterprisedb.com
On 05/01/2024 19:23, Robert Haas wrote:
On Fri, Nov 24, 2023 at 10:47 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
What do you think?
At least for 0001 and 0002, I think we should just add the stack depth checks.
With regard to 0001, CommitTransactionCommand() and friends are hard
enough to understand as it is; they need "goto" like I need an extra
hole in my head.With regard to 0002, this function isn't sufficiently important to
justify adding special-case code for an extremely rare event. We
should just handle it the way we do in general.I agree that in the memory-context case it might be worth expending
some more code to be more clever. But I probably wouldn't do that for
MemoryContextStats(); check_stack_depth() seems fine for that one.In general, I think we should try to keep the number of places that
handle stack overflow in "special" ways as small as possible.
The problem with CommitTransactionCommand (or rather
AbortCurrentTransaction() which has the same problem)
and ShowTransactionStateRec is that they get called in a state where
aborting can lead to a panic. If you add a "check_stack_depth()" to them
and try to reproducer scripts that Egor posted, you still get a panic.
I'm not sure if MemoryContextStats() could safely elog(ERROR). But at
least it would mask the "out of memory" that caused the stats to be
printed in the first place.
--
Heikki Linnakangas
Neon (https://neon.tech)
On Wed, Jan 10, 2024 at 4:25 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
The problem with CommitTransactionCommand (or rather
AbortCurrentTransaction() which has the same problem)
and ShowTransactionStateRec is that they get called in a state where
aborting can lead to a panic. If you add a "check_stack_depth()" to them
and try to reproducer scripts that Egor posted, you still get a panic.
Hmm, that's unfortunate. I'm not sure what to do about that. But I'd
still suggest looking for a goto-free approach.
--
Robert Haas
EDB: http://www.enterprisedb.com
On 11/01/2024 19:37, Robert Haas wrote:
On Wed, Jan 10, 2024 at 4:25 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
The problem with CommitTransactionCommand (or rather
AbortCurrentTransaction() which has the same problem)
and ShowTransactionStateRec is that they get called in a state where
aborting can lead to a panic. If you add a "check_stack_depth()" to them
and try to reproducer scripts that Egor posted, you still get a panic.Hmm, that's unfortunate. I'm not sure what to do about that. But I'd
still suggest looking for a goto-free approach.
Here's one goto-free attempt. It adds a local loop to where the
recursion was, so that if you have a chain of subtransactions that need
to be aborted in CommitTransactionCommand, they are aborted iteratively.
The TBLOCK_SUBCOMMIT case already had such a loop.
I added a couple of comments in the patch marked with "REVIEWER NOTE",
to explain why I changed some things. They are to be removed before
committing.
I'm not sure if this is better than a goto. In fact, even if we commit
this, I think I'd still prefer to replace the remaining recursive calls
with a goto. Recursion feels a weird to me, when we're unwinding the
states from the stack as we go.
Of course we could use a "for (;;) { ... continue }" construct around
the whole function, instead of a goto, but I don't think that's better
than a goto in this case.
--
Heikki Linnakangas
Neon (https://neon.tech)
Attachments:
v2-0001-Turn-tail-recursion-into-iteration-in-AbortCurren.patchtext/x-patch; charset=UTF-8; name=v2-0001-Turn-tail-recursion-into-iteration-in-AbortCurren.patchDownload
From 44568e6feef79d604bbe168c0839ab2e03c99f8a Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <heikki.linnakangas@iki.fi>
Date: Fri, 12 Jan 2024 17:07:25 +0200
Subject: [PATCH v2 1/1] Turn tail recursion into iteration in
AbortCurrentTransaction()
Usually the compiler will optimize away the tail recursion anyway, but
if it doesn't, you can drive the function into stack overflow. For
example:
(n=1000000; printf "BEGIN;"; for ((i=1;i<=$n;i++)); do printf "SAVEPOINT s$i;"; done; printf "ERROR; COMMIT;") | psql >/dev/null
The usual fix would be to add a check_stack_depth() to the function,
but that's not good in AbortCurrentTransaction(), as you are already
trying to clean up after a failure. ereporting there leads to a panic.
Fix CurrentTransactionCommand() in a similar fashion, although
ereporting would be less bad there.
Report by Egor Chindyaskin and Alexander Lakhin.
Discussion: https://www.postgresql.org/message-id/1672760457.940462079%40f306.i.mail.ru
---
src/backend/access/transam/xact.c | 141 ++++++++++++++++--------------
1 file changed, 75 insertions(+), 66 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 464858117e0..329a139c991 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -3194,47 +3194,62 @@ CommitTransactionCommand(void)
CommitSubTransaction();
s = CurrentTransactionState; /* changed by pop */
} while (s->blockState == TBLOCK_SUBCOMMIT);
+ /*
+ * REVIEWER NOTE: We used to have duplicated code here, to handle
+ * the TBLOCK_END and TBLOCK_PREPARE cases exactly the same way
+ * that we do above. I replaced that with a recursive call to
+ * CommitTransactionCommand(), to make this consistent with the
+ * TBLOCK_SUBABORT_END and TBLOCK_SUBABORT_PENDING handling. This
+ * can only recurse once, so there's no risk of stack overflow.
+ */
/* If we had a COMMIT command, finish off the main xact too */
- if (s->blockState == TBLOCK_END)
- {
- Assert(s->parent == NULL);
- CommitTransaction();
- s->blockState = TBLOCK_DEFAULT;
- if (s->chain)
- {
- StartTransaction();
- s->blockState = TBLOCK_INPROGRESS;
- s->chain = false;
- RestoreTransactionCharacteristics(&savetc);
- }
- }
- else if (s->blockState == TBLOCK_PREPARE)
- {
- Assert(s->parent == NULL);
- PrepareTransaction();
- s->blockState = TBLOCK_DEFAULT;
- }
- else
+ if (s->blockState != TBLOCK_END && s->blockState != TBLOCK_PREPARE)
elog(ERROR, "CommitTransactionCommand: unexpected state %s",
BlockStateAsString(s->blockState));
+ CommitTransactionCommand();
break;
/*
- * The current already-failed subtransaction is ending due to a
- * ROLLBACK or ROLLBACK TO command, so pop it and recursively
- * examine the parent (which could be in any of several states).
+ * The current subtransaction is ending due to a ROLLBACK or
+ * ROLLBACK TO command. Pop it, as well as any of it parents that
+ * are also being rolled back.
*/
case TBLOCK_SUBABORT_END:
- CleanupSubTransaction();
- CommitTransactionCommand();
- break;
+ case TBLOCK_SUBABORT_PENDING:
+ do {
+ /*
+ * The difference between SUBABORT_END and SUBABORT_PENDING is
+ * whether the subtransaction has already been aborted and we
+ * just need to clean it up, or if we need to also abort it
+ * first.
+ */
+ if (s->blockState == TBLOCK_SUBABORT_PENDING)
+ AbortSubTransaction();
+ CleanupSubTransaction();
+ s = CurrentTransactionState; /* changed by pop */
+ } while (s->blockState == TBLOCK_SUBABORT_END ||
+ s->blockState == TBLOCK_SUBABORT_PENDING);
/*
- * As above, but it's not dead yet, so abort first.
+ * REVIEWER NOTE: the old comment said that the parent can be "in
+ * any of several states". I looked at the code that sets the
+ * state to TBLOCK_SUBABORT_END or TBLOCK_SUBABORT_PENDING, and
+ * came to the conclusion that these are the only possible states
+ * above the chain of TBLOCK_SUBABORT_END or
+ * TBLOCK_SUBABORT_PENDING subtransactions.
*/
- case TBLOCK_SUBABORT_PENDING:
- AbortSubTransaction();
- CleanupSubTransaction();
+ /*
+ * Recurse to roll back the top-level transaction too, or to
+ * restart at a savepoint in case of ROLLBACK TO.
+ */
+ if (s->blockState != TBLOCK_ABORT_PENDING &&
+ s->blockState != TBLOCK_ABORT &&
+ s->blockState != TBLOCK_SUBRESTART &&
+ s->blockState != TBLOCK_SUBABORT_RESTART)
+ {
+ elog(ERROR, "CommitTransactionCommand: unexpected state %s",
+ BlockStateAsString(s->blockState));
+ }
CommitTransactionCommand();
break;
@@ -3243,35 +3258,13 @@ CommitTransactionCommand(void)
* command. Abort and pop it, then start a new subtransaction
* with the same name.
*/
- case TBLOCK_SUBRESTART:
- {
- char *name;
- int savepointLevel;
-
- /* save name and keep Cleanup from freeing it */
- name = s->name;
- s->name = NULL;
- savepointLevel = s->savepointLevel;
-
- AbortSubTransaction();
- CleanupSubTransaction();
-
- DefineSavepoint(NULL);
- s = CurrentTransactionState; /* changed by push */
- s->name = name;
- s->savepointLevel = savepointLevel;
-
- /* This is the same as TBLOCK_SUBBEGIN case */
- Assert(s->blockState == TBLOCK_SUBBEGIN);
- StartSubTransaction();
- s->blockState = TBLOCK_SUBINPROGRESS;
- }
- break;
-
/*
- * Same as above, but the subtransaction had already failed, so we
- * don't need AbortSubTransaction.
+ * REVIEWER NOTE: I merged the TBLOCK_SUBRESTART and
+ * TBLOCK_SUBABORT_RESTART cases. Not strictly necessary, but it
+ * makes this more similar to how the TBLOCK_SUBABORT_END and
+ * TBLOCK_SUBABORT_PENDING cases are handled now.
*/
+ case TBLOCK_SUBRESTART:
case TBLOCK_SUBABORT_RESTART:
{
char *name;
@@ -3282,6 +3275,12 @@ CommitTransactionCommand(void)
s->name = NULL;
savepointLevel = s->savepointLevel;
+ /*
+ * In SUBABORT_RESTART state, the subtransaction had already
+ * failed, so we don't need AbortSubTransaction.
+ */
+ if (s->blockState == TBLOCK_SUBRESTART)
+ AbortSubTransaction();
CleanupSubTransaction();
DefineSavepoint(NULL);
@@ -3437,17 +3436,27 @@ AbortCurrentTransaction(void)
case TBLOCK_SUBCOMMIT:
case TBLOCK_SUBABORT_PENDING:
case TBLOCK_SUBRESTART:
- AbortSubTransaction();
- CleanupSubTransaction();
- AbortCurrentTransaction();
- break;
-
- /*
- * Same as above, except the Abort() was already done.
- */
case TBLOCK_SUBABORT_END:
case TBLOCK_SUBABORT_RESTART:
- CleanupSubTransaction();
+ /*
+ * If multiple subtransactions need to be cleaned up, iterate
+ * through them here. The recursion would handle them too, but
+ * this avoids running out of stack if there is a deep stack of
+ * aborted subtransactions, and the compiler isn't smart enough
+ * to optimize away the tail recursion.
+ */
+ do {
+ if (s->blockState == TBLOCK_SUBABORT_END || s->blockState == TBLOCK_SUBABORT_RESTART)
+ AbortSubTransaction();
+ CleanupSubTransaction();
+ } while(s->blockState == TBLOCK_SUBBEGIN ||
+ s->blockState == TBLOCK_SUBRELEASE ||
+ s->blockState == TBLOCK_SUBCOMMIT ||
+ s->blockState == TBLOCK_SUBABORT_PENDING ||
+ s->blockState == TBLOCK_SUBRESTART ||
+ s->blockState == TBLOCK_SUBABORT_END ||
+ s->blockState == TBLOCK_SUBABORT_RESTART);
+ /* Abort the top-level transaction, too */
AbortCurrentTransaction();
break;
}
--
2.39.2
On Fri, Jan 12, 2024 at 10:12 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
Here's one goto-free attempt. It adds a local loop to where the
recursion was, so that if you have a chain of subtransactions that need
to be aborted in CommitTransactionCommand, they are aborted iteratively.
The TBLOCK_SUBCOMMIT case already had such a loop.I added a couple of comments in the patch marked with "REVIEWER NOTE",
to explain why I changed some things. They are to be removed before
committing.I'm not sure if this is better than a goto. In fact, even if we commit
this, I think I'd still prefer to replace the remaining recursive calls
with a goto. Recursion feels a weird to me, when we're unwinding the
states from the stack as we go.
I'm not able to quickly verify whether this version is correct, but I
do think the code looks nicer this way.
I understand that's a question of opinion rather than fact, though.
--
Robert Haas
EDB: http://www.enterprisedb.com
Hi!
On Fri, Jan 12, 2024 at 11:00 PM Robert Haas <robertmhaas@gmail.com> wrote:
On Fri, Jan 12, 2024 at 10:12 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
Here's one goto-free attempt. It adds a local loop to where the
recursion was, so that if you have a chain of subtransactions that need
to be aborted in CommitTransactionCommand, they are aborted iteratively.
The TBLOCK_SUBCOMMIT case already had such a loop.I added a couple of comments in the patch marked with "REVIEWER NOTE",
to explain why I changed some things. They are to be removed before
committing.I'm not sure if this is better than a goto. In fact, even if we commit
this, I think I'd still prefer to replace the remaining recursive calls
with a goto. Recursion feels a weird to me, when we're unwinding the
states from the stack as we go.I'm not able to quickly verify whether this version is correct, but I
do think the code looks nicer this way.I understand that's a question of opinion rather than fact, though.
I'd like to revive this thread. The attached 0001 patch represents my
attempt to remove recursion in
CommitTransactionCommand()/AbortCurrentTransaction() by adding a
wrapper function. This method doesn't use goto, doesn't require much
code changes and subjectively provides good readability.
Regarding ShowTransactionStateRec() and memory context function, as I
get from this thread they are called in states where abortion can lead
to a panic. So, it's preferable to change them into loops too rather
than just adding check_stack_depth(). The 0002 and 0003 patches by
Heikki posted in [1] look good to me. Can we accept them?
Also there are a number of recursive functions, which seem to be not
used in critical states where abortion can lead to a panic. I've
extracted them from [2] into an attached 0002 patch. I'd like to push
it if there is no objection.
Links.
1. /messages/by-id/6b48c746-9704-46dc-b9be-01fe4137c824@iki.fi
2. /messages/by-id/4530546a-3216-eaa9-4c92-92d33290a211@mail.ru
------
Regards,
Alexander Korotkov
Attachments:
0002-Add-missing-check_stack_depth-to-some-recursive-f-v3.patchapplication/octet-stream; name=0002-Add-missing-check_stack_depth-to-some-recursive-f-v3.patchDownload
From 6a4d963540000877dc1cdaf586ca01121eb9db57 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <akorotkov@postgresql.org>
Date: Wed, 14 Feb 2024 13:38:44 +0200
Subject: [PATCH 2/2] Add missing check_stack_depth() to some recursive
functions
Report by Egor Chindyaskin and Alexander Lakhin.
Discussion: https://postgr.es/m/1672760457.940462079%40f306.i.mail.ru
---
src/backend/catalog/dependency.c | 7 +++++++
src/backend/catalog/heap.c | 3 +++
src/backend/commands/tablecmds.c | 13 +++++++++++++
src/backend/optimizer/util/clauses.c | 4 ++++
src/backend/utils/adt/jsonpath_exec.c | 3 +++
5 files changed, 30 insertions(+)
diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c
index e742c78ea35..df9886efc95 100644
--- a/src/backend/catalog/dependency.c
+++ b/src/backend/catalog/dependency.c
@@ -76,6 +76,7 @@
#include "commands/trigger.h"
#include "commands/typecmds.h"
#include "funcapi.h"
+#include "miscadmin.h"
#include "nodes/nodeFuncs.h"
#include "parser/parsetree.h"
#include "rewrite/rewriteRemove.h"
@@ -524,6 +525,12 @@ findDependentObjects(const ObjectAddress *object,
if (stack_address_present_add_flags(object, objflags, stack))
return;
+ /*
+ * since this function recurses, it could be driven to stack overflow,
+ * because of the deep dependency tree, not only due to dependency loops.
+ */
+ check_stack_depth();
+
/*
* It's also possible that the target object has already been completely
* processed and put into targetObjects. If so, again we just add the
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index c73f7bcd011..252e106cada 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -552,6 +552,9 @@ CheckAttributeType(const char *attname,
char att_typtype = get_typtype(atttypid);
Oid att_typelem;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
if (att_typtype == TYPTYPE_PSEUDO)
{
/*
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 86ea3a228b6..679dee10da4 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -7047,6 +7047,9 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
ObjectAddress address;
TupleDesc tupdesc;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
/* At top level, permission check was done in ATPrepCmd, else do it */
if (recursing)
ATSimplePermissions((*cmd)->subtype, rel, ATT_TABLE | ATT_FOREIGN_TABLE);
@@ -9095,6 +9098,10 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/* Initialize addrs on the first invocation */
Assert(!recursing || addrs != NULL);
+
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
if (!recursing)
addrs = new_object_addresses();
@@ -11648,6 +11655,9 @@ ATExecAlterConstrRecurse(Constraint *cmdcon, Relation conrel, Relation tgrel,
Oid refrelid;
bool changed = false;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
currcon = (Form_pg_constraint) GETSTRUCT(contuple);
conoid = currcon->oid;
refrelid = currcon->confrelid;
@@ -12728,6 +12738,9 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
/* Guard against stack overflow due to overly deep inheritance tree. */
check_stack_depth();
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
/* At top level, permission check was done in ATPrepCmd, else do it */
if (recursing)
ATSimplePermissions(AT_DropConstraint, rel, ATT_TABLE | ATT_FOREIGN_TABLE);
diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c
index 94eb56a1e79..edc25d712e9 100644
--- a/src/backend/optimizer/util/clauses.c
+++ b/src/backend/optimizer/util/clauses.c
@@ -2423,6 +2423,10 @@ static Node *
eval_const_expressions_mutator(Node *node,
eval_const_expressions_context *context)
{
+
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
if (node == NULL)
return NULL;
switch (nodeTag(node))
diff --git a/src/backend/utils/adt/jsonpath_exec.c b/src/backend/utils/adt/jsonpath_exec.c
index 8372863de74..50fa724b728 100644
--- a/src/backend/utils/adt/jsonpath_exec.c
+++ b/src/backend/utils/adt/jsonpath_exec.c
@@ -1674,6 +1674,9 @@ executeBoolItem(JsonPathExecContext *cxt, JsonPathItem *jsp,
JsonPathBool res;
JsonPathBool res2;
+ /* since this function recurses, it could be driven to stack overflow */
+ check_stack_depth();
+
if (!canHaveNext && jspHasNext(jsp))
elog(ERROR, "boolean jsonpath item cannot have next item");
--
2.39.3 (Apple Git-145)
0001-Turn-tail-recursion-into-iteration-in-CommitTrans-v3.patchapplication/octet-stream; name=0001-Turn-tail-recursion-into-iteration-in-CommitTrans-v3.patchDownload
From 0d474daa36aa6dd83c64fb6ea4b0fc62a87030a6 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <akorotkov@postgresql.org>
Date: Wed, 14 Feb 2024 13:16:41 +0200
Subject: [PATCH 1/2] Turn tail recursion into iteration in
CommitTransactionCommand()
Usually the compiler will optimize away the tail recursion anyway, but
if it doesn't, you can drive the function into stack overflow. For
example:
(n=1000000; printf "BEGIN;"; for ((i=1;i<=$n;i++)); do printf "SAVEPOINT s$i;"; done; printf "ERROR; COMMIT;") | psql >/dev/null
In order to get better readability and less changes to the existing code the
recursion-replacing loop is implemented as a wrapper function.
Report by Egor Chindyaskin and Alexander Lakhin.
Discussion: https://postgr.es/m/1672760457.940462079%40f306.i.mail.ru
---
src/backend/access/transam/xact.c | 147 ++++++++++++++++++++----------
1 file changed, 101 insertions(+), 46 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 464858117e0..fad1157f068 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -3028,14 +3028,18 @@ RestoreTransactionCharacteristics(const SavedTransactionCharacteristics *s)
/*
- * CommitTransactionCommand
+ * CommitTransactionCommandInternal
*/
-void
-CommitTransactionCommand(void)
+static void
+CommitTransactionCommandInternal(void)
{
TransactionState s = CurrentTransactionState;
SavedTransactionCharacteristics savetc;
+ /* This states are handled in CommitTransactionCommand() */
+ Assert(s->blockState != TBLOCK_SUBABORT_END &&
+ s->blockState != TBLOCK_SUBABORT_PENDING);
+
/* Must save in case we need to restore below */
SaveTransactionCharacteristics(&savetc);
@@ -3219,25 +3223,6 @@ CommitTransactionCommand(void)
BlockStateAsString(s->blockState));
break;
- /*
- * The current already-failed subtransaction is ending due to a
- * ROLLBACK or ROLLBACK TO command, so pop it and recursively
- * examine the parent (which could be in any of several states).
- */
- case TBLOCK_SUBABORT_END:
- CleanupSubTransaction();
- CommitTransactionCommand();
- break;
-
- /*
- * As above, but it's not dead yet, so abort first.
- */
- case TBLOCK_SUBABORT_PENDING:
- AbortSubTransaction();
- CleanupSubTransaction();
- CommitTransactionCommand();
- break;
-
/*
* The current subtransaction is the target of a ROLLBACK TO
* command. Abort and pop it, then start a new subtransaction
@@ -3295,17 +3280,66 @@ CommitTransactionCommand(void)
s->blockState = TBLOCK_SUBINPROGRESS;
}
break;
+ default:
+ /* Keep compiler quiet */
+ break;
}
}
/*
- * AbortCurrentTransaction
+ * CommitTransactionCommand -- the wrapper function handling the
+ * loop over subtransactions to avoid potentially dangerous recursion in
+ * CommitTransactionCommandInternal();
*/
void
-AbortCurrentTransaction(void)
+CommitTransactionCommand(void)
+{
+ while (true)
+ {
+ switch (CurrentTransactionState->blockState)
+ {
+ /*
+ * The current already-failed subtransaction is ending due to
+ * a ROLLBACK or ROLLBACK TO command, so pop it and
+ * recursively examine the parent (which could be in any of
+ * several states).
+ */
+ case TBLOCK_SUBABORT_END:
+ CleanupSubTransaction();
+ continue;
+
+ /*
+ * As above, but it's not dead yet, so abort first.
+ */
+ case TBLOCK_SUBABORT_PENDING:
+ AbortSubTransaction();
+ CleanupSubTransaction();
+ continue;
+ default:
+ break;
+ }
+ CommitTransactionCommandInternal();
+ break;
+ }
+}
+
+/*
+ * AbortCurrentTransactionInternal
+ */
+static void
+AbortCurrentTransactionInternal(void)
{
TransactionState s = CurrentTransactionState;
+ /* This states are handled in AbortCurrentTransaction() */
+ Assert(s->blockState != TBLOCK_SUBBEGIN &&
+ s->blockState != TBLOCK_SUBRELEASE &&
+ s->blockState != TBLOCK_SUBCOMMIT &&
+ s->blockState != TBLOCK_SUBABORT_PENDING &&
+ s->blockState != TBLOCK_SUBRESTART &&
+ s->blockState != TBLOCK_SUBABORT_END &&
+ s->blockState != TBLOCK_SUBABORT_RESTART);
+
switch (s->blockState)
{
case TBLOCK_DEFAULT:
@@ -3426,30 +3460,51 @@ AbortCurrentTransaction(void)
AbortSubTransaction();
s->blockState = TBLOCK_SUBABORT;
break;
-
- /*
- * If we failed while trying to create a subtransaction, clean up
- * the broken subtransaction and abort the parent. The same
- * applies if we get a failure while ending a subtransaction.
- */
- case TBLOCK_SUBBEGIN:
- case TBLOCK_SUBRELEASE:
- case TBLOCK_SUBCOMMIT:
- case TBLOCK_SUBABORT_PENDING:
- case TBLOCK_SUBRESTART:
- AbortSubTransaction();
- CleanupSubTransaction();
- AbortCurrentTransaction();
+ default:
+ /* Keep compiler quiet */
break;
+ }
+}
- /*
- * Same as above, except the Abort() was already done.
- */
- case TBLOCK_SUBABORT_END:
- case TBLOCK_SUBABORT_RESTART:
- CleanupSubTransaction();
- AbortCurrentTransaction();
- break;
+/*
+ * AbortCurrentTransaction -- the wrapper function handling the
+ * loop over subtransactions to avoid potentially dangerous recursion in
+ * AbortCurrentTransactionInternal();
+ */
+void
+AbortCurrentTransaction(void)
+{
+ while (true)
+ {
+ switch (CurrentTransactionState->blockState)
+ {
+ /*
+ * If we failed while trying to create a subtransaction, clean
+ * up the broken subtransaction and abort the parent. The
+ * same applies if we get a failure while ending a
+ * subtransaction.
+ */
+ case TBLOCK_SUBBEGIN:
+ case TBLOCK_SUBRELEASE:
+ case TBLOCK_SUBCOMMIT:
+ case TBLOCK_SUBABORT_PENDING:
+ case TBLOCK_SUBRESTART:
+ AbortSubTransaction();
+ CleanupSubTransaction();
+ continue;
+
+ /*
+ * Same as above, except the Abort() was already done.
+ */
+ case TBLOCK_SUBABORT_END:
+ case TBLOCK_SUBABORT_RESTART:
+ CleanupSubTransaction();
+ continue;
+ default:
+ break;
+ }
+ AbortCurrentTransactionInternal();
+ break;
}
}
--
2.39.3 (Apple Git-145)
On Wed, Feb 14, 2024 at 2:00 PM Alexander Korotkov <aekorotkov@gmail.com> wrote:
On Fri, Jan 12, 2024 at 11:00 PM Robert Haas <robertmhaas@gmail.com> wrote:
On Fri, Jan 12, 2024 at 10:12 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
Here's one goto-free attempt. It adds a local loop to where the
recursion was, so that if you have a chain of subtransactions that need
to be aborted in CommitTransactionCommand, they are aborted iteratively.
The TBLOCK_SUBCOMMIT case already had such a loop.I added a couple of comments in the patch marked with "REVIEWER NOTE",
to explain why I changed some things. They are to be removed before
committing.I'm not sure if this is better than a goto. In fact, even if we commit
this, I think I'd still prefer to replace the remaining recursive calls
with a goto. Recursion feels a weird to me, when we're unwinding the
states from the stack as we go.I'm not able to quickly verify whether this version is correct, but I
do think the code looks nicer this way.I understand that's a question of opinion rather than fact, though.
I'd like to revive this thread. The attached 0001 patch represents my
attempt to remove recursion in
CommitTransactionCommand()/AbortCurrentTransaction() by adding a
wrapper function. This method doesn't use goto, doesn't require much
code changes and subjectively provides good readability.Regarding ShowTransactionStateRec() and memory context function, as I
get from this thread they are called in states where abortion can lead
to a panic. So, it's preferable to change them into loops too rather
than just adding check_stack_depth(). The 0002 and 0003 patches by
Heikki posted in [1] look good to me. Can we accept them?Also there are a number of recursive functions, which seem to be not
used in critical states where abortion can lead to a panic. I've
extracted them from [2] into an attached 0002 patch. I'd like to push
it if there is no objection.
The revised set of remaining patches is attached.
0001 Turn tail recursion into iteration in CommitTransactionCommand()
I did minor revision of comments and code blocks order to improve the
readability.
0002 Avoid stack overflow in ShowTransactionStateRec()
I didn't notice any issues, leave this piece as is.
0003 Avoid recursion in MemoryContext functions
I've renamed MemoryContextTraverse() => MemoryContextTraverseNext(),
which I think is a bit more intuitive. Also I fixed
MemoryContextMemConsumed(), which was still trying to use the removed
argument "print" of MemoryContextStatsInternal() function.
Generally, I think this patchset fixes important stack overflow holes.
It is quite straightforward, clear and the code has a good shape. I'm
going to push this if no objections.
------
Regards,
Alexander Korotkov
Attachments:
v4-0002-Avoid-stack-overflow-in-ShowTransactionStateRec.patchapplication/x-patch; name=v4-0002-Avoid-stack-overflow-in-ShowTransactionStateRec.patchDownload
From d9c2a7e8eae2706ff1431ed090146e3a26ce07dc Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <akorotkov@postgresql.org>
Date: Wed, 6 Mar 2024 13:59:41 +0200
Subject: [PATCH v4 2/3] Avoid stack overflow in ShowTransactionStateRec()
The function recurses, but didn't perform stack-depth checks. It's
just a debugging aid, so instead of the usual check_stack_depth()
call, stop the printing if we'd risk stack overflow.
Here's an example of how to test this:
(n=1000000; printf "BEGIN;"; for ((i=1;i<=$n;i++)); do printf "SAVEPOINT s$i;"; done; printf "SET log_min_messages = 'DEBUG5'; SAVEPOINT sp;") | psql >/dev/null
In the passing, swap building the list of child XIDs and recursing to
parent. That saves memory while recursing, reducing the risk of out of
memory errors with lots of subtransactions. The saving is not very
significant in practice, but this order seems more logical anyway.
Report by Egor Chindyaskin and Alexander Lakhin.
Discussion: https://www.postgresql.org/message-id/1672760457.940462079%40f306.i.mail.ru
Author: Heikki Linnakangas
Reviewed-by: Robert Haas, Andres Freund, Alexander Korotkov
---
src/backend/access/transam/xact.c | 21 +++++++++++++++------
1 file changed, 15 insertions(+), 6 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 12f8bec1aeb..2f80111e18d 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -5583,8 +5583,22 @@ ShowTransactionStateRec(const char *str, TransactionState s)
{
StringInfoData buf;
- initStringInfo(&buf);
+ if (s->parent)
+ {
+ /*
+ * Since this function recurses, it could be driven to stack overflow.
+ * This is just a debugging aid, so we can leave out some details
+ * instead of erroring out with check_stack_depth().
+ */
+ if (stack_is_too_deep())
+ ereport(DEBUG5,
+ (errmsg_internal("%s(%d): parent omitted to avoid stack overflow",
+ str, s->nestingLevel)));
+ else
+ ShowTransactionStateRec(str, s->parent);
+ }
+ initStringInfo(&buf);
if (s->nChildXids > 0)
{
int i;
@@ -5593,10 +5607,6 @@ ShowTransactionStateRec(const char *str, TransactionState s)
for (i = 1; i < s->nChildXids; i++)
appendStringInfo(&buf, " %u", s->childXids[i]);
}
-
- if (s->parent)
- ShowTransactionStateRec(str, s->parent);
-
ereport(DEBUG5,
(errmsg_internal("%s(%d) name: %s; blockState: %s; state: %s, xid/subid/cid: %u/%u/%u%s%s",
str, s->nestingLevel,
@@ -5608,7 +5618,6 @@ ShowTransactionStateRec(const char *str, TransactionState s)
(unsigned int) currentCommandId,
currentCommandIdUsed ? " (used)" : "",
buf.data)));
-
pfree(buf.data);
}
--
2.39.3 (Apple Git-145)
v4-0003-Avoid-recursion-in-MemoryContext-functions.patchapplication/x-patch; name=v4-0003-Avoid-recursion-in-MemoryContext-functions.patchDownload
From 693e9707be2a006c70a06052564c549c37acc07b Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <akorotkov@postgresql.org>
Date: Wed, 6 Mar 2024 13:59:55 +0200
Subject: [PATCH v4 3/3] Avoid recursion in MemoryContext functions.
You might run out of stack space with recursion, which is not nice in
functions that might be used e.g. at cleanup after transaction
abort. MemoryContext contains pointer to parent and siblings, so we
can traverse a tree of contexts iteratively, without using
stack. Refactor the functions to do that.
MemoryContextStats() still recurses, but it now has a limit to how
deep it recurses. Once the limit is reached, it prints just a summary
of the rest of the hierarchy, similar to how it summarizes contexts
with lots of children. That seems good anyway, because a context dump
with hundreds of nested contexts isn't very readable.
Report by Egor Chindyaskin and Alexander Lakhin.
Discussion: https://www.postgresql.org/message-id/1672760457.940462079%40f306.i.mail.ru
Author: Heikki Linnakangas
Reviewed-by: Robert Haas, Andres Freund, Alexander Korotkov
---
src/backend/utils/mmgr/mcxt.c | 246 +++++++++++++++++++++++-----------
src/include/utils/memutils.h | 3 +-
2 files changed, 170 insertions(+), 79 deletions(-)
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 1a615becae4..04b540a578e 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -145,9 +145,10 @@ MemoryContext CurTransactionContext = NULL;
/* This is a transient link to the active portal's memory context: */
MemoryContext PortalContext = NULL;
+static void MemoryContextDeleteOnly(MemoryContext context);
static void MemoryContextCallResetCallbacks(MemoryContext context);
static void MemoryContextStatsInternal(MemoryContext context, int level,
- bool print, int max_children,
+ int max_level, int max_children,
MemoryContextCounters *totals,
bool print_to_stderr);
static void MemoryContextStatsPrint(MemoryContext context, void *passthru,
@@ -256,6 +257,41 @@ BogusGetChunkSpace(void *pointer)
return 0; /* keep compiler quiet */
}
+/*
+ * MemoryContextTraverseNext
+ * Helper function to traverse all descendants of a memory context
+ * without recursion.
+ *
+ * Recursion could lead to out-of-stack errors with deep context hierarchies,
+ * which would be unpleasant in error cleanup code paths.
+ *
+ * To process 'context' and all its descendants, use a loop like this:
+ *
+ * <process 'context'>
+ * for (MemoryContext curr = context->firstchild;
+ * curr != NULL;
+ * curr = MemoryContextTraverseNext(curr, context))
+ * {
+ * <process 'curr'>
+ * }
+ *
+ * This visits all the contexts in the depth-first order.
+ */
+static MemoryContext
+MemoryContextTraverseNext(MemoryContext curr, MemoryContext top)
+{
+ if (curr->firstchild != NULL)
+ return curr->firstchild;
+
+ while (curr->nextchild == NULL)
+ {
+ curr = curr->parent;
+ if (curr == top)
+ return NULL;
+ }
+ return curr->nextchild;
+}
+
/*****************************************************************************
* EXPORTED ROUTINES *
@@ -375,14 +411,13 @@ MemoryContextResetOnly(MemoryContext context)
void
MemoryContextResetChildren(MemoryContext context)
{
- MemoryContext child;
-
Assert(MemoryContextIsValid(context));
- for (child = context->firstchild; child != NULL; child = child->nextchild)
+ for (MemoryContext curr = context->firstchild;
+ curr != NULL;
+ curr = MemoryContextTraverseNext(curr, context))
{
- MemoryContextResetChildren(child);
- MemoryContextResetOnly(child);
+ MemoryContextResetOnly(curr);
}
}
@@ -397,16 +432,53 @@ MemoryContextResetChildren(MemoryContext context)
*/
void
MemoryContextDelete(MemoryContext context)
+{
+ MemoryContext curr;
+
+ /*
+ * Delete subcontexts from the bottom up.
+ *
+ * Note: Do not use recursion here. A "stack depth limit exceeded" error
+ * would be unpleasant if we're already in the process of cleaning up from
+ * transaction abort. We also cannot use MemoryContextTraverseNext() here
+ * because we modify the tree as we go.
+ */
+ curr = context;
+ for (;;)
+ {
+ MemoryContext parent;
+
+ /* Descend down until we find a leaf context with no children */
+ while (curr->firstchild != NULL)
+ curr = curr->firstchild;
+
+ /*
+ * We're now at a leaf with no children. Free it and continue from the
+ * parent. Or if this was the original node, we're all done.
+ */
+ parent = curr->parent;
+ MemoryContextDeleteOnly(curr);
+
+ if (curr == context)
+ break;
+ curr = parent;
+ }
+}
+
+/*
+ * Subroutine of MemoryContextDelete, to delete a context that has no
+ * children.
+ */
+static void
+MemoryContextDeleteOnly(MemoryContext context)
{
Assert(MemoryContextIsValid(context));
/* We had better not be deleting TopMemoryContext ... */
Assert(context != TopMemoryContext);
/* And not CurrentMemoryContext, either */
Assert(context != CurrentMemoryContext);
-
- /* save a function call in common case where there are no children */
- if (context->firstchild != NULL)
- MemoryContextDeleteChildren(context);
+ /* All the children should've been deleted already */
+ Assert(context->firstchild == NULL);
/*
* It's not entirely clear whether 'tis better to do this before or after
@@ -672,12 +744,12 @@ MemoryContextMemAllocated(MemoryContext context, bool recurse)
if (recurse)
{
- MemoryContext child;
-
- for (child = context->firstchild;
- child != NULL;
- child = child->nextchild)
- total += MemoryContextMemAllocated(child, true);
+ for (MemoryContext curr = context->firstchild;
+ curr != NULL;
+ curr = MemoryContextTraverseNext(curr, context))
+ {
+ total += curr->mem_allocated;
+ }
}
return total;
@@ -691,9 +763,15 @@ void
MemoryContextMemConsumed(MemoryContext context,
MemoryContextCounters *consumed)
{
+
memset(consumed, 0, sizeof(*consumed));
- MemoryContextStatsInternal(context, 0, false, 0, consumed, false);
+ for (MemoryContext curr = context->firstchild;
+ curr != NULL;
+ curr = MemoryContextTraverseNext(curr, context))
+ {
+ child->methods->stats(child, NULL, NULL, consumed, print_to_stderr);
+ }
}
/*
@@ -707,8 +785,8 @@ MemoryContextMemConsumed(MemoryContext context,
void
MemoryContextStats(MemoryContext context)
{
- /* A hard-wired limit on the number of children is usually good enough */
- MemoryContextStatsDetail(context, 100, true);
+ /* Hard-wired limits are usually good enough */
+ MemoryContextStatsDetail(context, 100, 100, true);
}
/*
@@ -720,14 +798,15 @@ MemoryContextStats(MemoryContext context)
* with fprintf(stderr), otherwise use ereport().
*/
void
-MemoryContextStatsDetail(MemoryContext context, int max_children,
+MemoryContextStatsDetail(MemoryContext context,
+ int max_level, int max_children,
bool print_to_stderr)
{
MemoryContextCounters grand_totals;
memset(&grand_totals, 0, sizeof(grand_totals));
- MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals, print_to_stderr);
+ MemoryContextStatsInternal(context, 0, max_level, max_children, &grand_totals, print_to_stderr);
if (print_to_stderr)
fprintf(stderr,
@@ -765,11 +844,10 @@ MemoryContextStatsDetail(MemoryContext context, int max_children,
*/
static void
MemoryContextStatsInternal(MemoryContext context, int level,
- bool print, int max_children,
+ int max_level, int max_children,
MemoryContextCounters *totals,
bool print_to_stderr)
{
- MemoryContextCounters local_totals;
MemoryContext child;
int ichild;
@@ -777,65 +855,75 @@ MemoryContextStatsInternal(MemoryContext context, int level,
/* Examine the context itself */
context->methods->stats(context,
- print ? MemoryContextStatsPrint : NULL,
+ MemoryContextStatsPrint,
(void *) &level,
- totals, print_to_stderr);
+ totals,
+ print_to_stderr);
/*
- * Examine children. If there are more than max_children of them, we do
- * not print the rest explicitly, but just summarize them.
+ * Examine children.
+ *
+ * If we are past the recursion depth limit or already running low on
+ * stack, do not print them explicitly but just summarize them. Similarly,
+ * if there are more than max_children of them, we do not print the rest
+ * explicitly, but just summarize them.
*/
- memset(&local_totals, 0, sizeof(local_totals));
-
- for (child = context->firstchild, ichild = 0;
- child != NULL;
- child = child->nextchild, ichild++)
+ ichild = 0;
+ child = context->firstchild;
+ if (level < max_level && !stack_is_too_deep())
{
- if (ichild < max_children)
+ for (; child != NULL && ichild < max_children;
+ child = child->nextchild)
+ {
MemoryContextStatsInternal(child, level + 1,
- print, max_children,
+ max_level, max_children,
totals,
print_to_stderr);
- else
- MemoryContextStatsInternal(child, level + 1,
- false, max_children,
- &local_totals,
- print_to_stderr);
+ ichild++;
+ }
}
- /* Deal with excess children */
- if (ichild > max_children)
+ if (child != NULL)
{
- if (print)
+ /* Summarize the rest of the children, avoiding recursion. */
+ MemoryContextCounters local_totals;
+
+ memset(&local_totals, 0, sizeof(local_totals));
+
+ while (child != NULL)
+ {
+ child->methods->stats(child, NULL, NULL, &local_totals, print_to_stderr);
+ child = MemoryContextTraverseNext(child, context);
+ ichild++;
+ }
+
+ if (print_to_stderr)
{
- if (print_to_stderr)
- {
- int i;
-
- for (i = 0; i <= level; i++)
- fprintf(stderr, " ");
- fprintf(stderr,
- "%d more child contexts containing %zu total in %zu blocks; %zu free (%zu chunks); %zu used\n",
- ichild - max_children,
- local_totals.totalspace,
- local_totals.nblocks,
- local_totals.freespace,
- local_totals.freechunks,
- local_totals.totalspace - local_totals.freespace);
- }
- else
- ereport(LOG_SERVER_ONLY,
- (errhidestmt(true),
- errhidecontext(true),
- errmsg_internal("level: %d; %d more child contexts containing %zu total in %zu blocks; %zu free (%zu chunks); %zu used",
- level,
- ichild - max_children,
- local_totals.totalspace,
- local_totals.nblocks,
- local_totals.freespace,
- local_totals.freechunks,
- local_totals.totalspace - local_totals.freespace)));
+ int i;
+
+ for (i = 0; i <= level; i++)
+ fprintf(stderr, " ");
+ fprintf(stderr,
+ "%d more child contexts containing %zu total in %zu blocks; %zu free (%zu chunks); %zu used\n",
+ ichild - max_children,
+ local_totals.totalspace,
+ local_totals.nblocks,
+ local_totals.freespace,
+ local_totals.freechunks,
+ local_totals.totalspace - local_totals.freespace);
}
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("level: %d; %d more child contexts containing %zu total in %zu blocks; %zu free (%zu chunks); %zu used",
+ level,
+ ichild - max_children,
+ local_totals.totalspace,
+ local_totals.nblocks,
+ local_totals.freespace,
+ local_totals.freechunks,
+ local_totals.totalspace - local_totals.freespace)));
if (totals)
{
@@ -856,8 +944,7 @@ MemoryContextStatsInternal(MemoryContext context, int level,
*/
static void
MemoryContextStatsPrint(MemoryContext context, void *passthru,
- const char *stats_string,
- bool print_to_stderr)
+ const char *stats_string, bool print_to_stderr)
{
int level = *(int *) passthru;
const char *name = context->name;
@@ -936,13 +1023,15 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
void
MemoryContextCheck(MemoryContext context)
{
- MemoryContext child;
-
Assert(MemoryContextIsValid(context));
-
context->methods->check(context);
- for (child = context->firstchild; child != NULL; child = child->nextchild)
- MemoryContextCheck(child);
+
+ for (MemoryContext curr = context->firstchild;
+ curr != NULL;
+ curr = MemoryContextTraverseNext(curr, context))
+ {
+ curr->methods->check(curr);
+ }
}
#endif
@@ -1183,14 +1272,15 @@ ProcessLogMemoryContextInterrupt(void)
/*
* When a backend process is consuming huge memory, logging all its memory
* contexts might overrun available disk space. To prevent this, we limit
- * the number of child contexts to log per parent to 100.
+ * the depth of the hierarchy, as well as the number of child contexts to
+ * log per parent to 100.
*
* As with MemoryContextStats(), we suppose that practical cases where the
* dump gets long will typically be huge numbers of siblings under the
* same parent context; while the additional debugging value from seeing
* details about individual siblings beyond 100 will not be large.
*/
- MemoryContextStatsDetail(TopMemoryContext, 100, false);
+ MemoryContextStatsDetail(TopMemoryContext, 100, 100, false);
}
void *
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 7fd41d20caf..6e5fa72b0e1 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -87,7 +87,8 @@ extern Size MemoryContextMemAllocated(MemoryContext context, bool recurse);
extern void MemoryContextMemConsumed(MemoryContext context,
MemoryContextCounters *consumed);
extern void MemoryContextStats(MemoryContext context);
-extern void MemoryContextStatsDetail(MemoryContext context, int max_children,
+extern void MemoryContextStatsDetail(MemoryContext context,
+ int max_level, int max_children,
bool print_to_stderr);
extern void MemoryContextAllowInCriticalSection(MemoryContext context,
bool allow);
--
2.39.3 (Apple Git-145)
v4-0001-Turn-tail-recursion-into-iteration-in-CommitTrans.patchapplication/x-patch; name=v4-0001-Turn-tail-recursion-into-iteration-in-CommitTrans.patchDownload
From cc905afa528ab5a98814167dc28ecf976dc7f52f Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <akorotkov@postgresql.org>
Date: Wed, 14 Feb 2024 13:16:41 +0200
Subject: [PATCH v4 1/3] Turn tail recursion into iteration in
CommitTransactionCommand()
Usually the compiler will optimize away the tail recursion anyway, but
if it doesn't, you can drive the function into stack overflow. For
example:
(n=1000000; printf "BEGIN;"; for ((i=1;i<=$n;i++)); do printf "SAVEPOINT s$i;"; done; printf "ERROR; COMMIT;") | psql >/dev/null
In order to get better readability and less changes to the existing code the
recursion-replacing loop is implemented as a wrapper function.
Report by Egor Chindyaskin and Alexander Lakhin.
Discussion: https://postgr.es/m/1672760457.940462079%40f306.i.mail.ru
Author: Alexander Korotkov, Heikki Linnakangas
---
src/backend/access/transam/xact.c | 151 +++++++++++++++++++++---------
1 file changed, 106 insertions(+), 45 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index ccd3f4fc550..12f8bec1aeb 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -341,6 +341,9 @@ static void CommitTransaction(void);
static TransactionId RecordTransactionAbort(bool isSubXact);
static void StartTransaction(void);
+static void CommitTransactionCommandInternal(void);
+static void AbortCurrentTransactionInternal(void);
+
static void StartSubTransaction(void);
static void CommitSubTransaction(void);
static void AbortSubTransaction(void);
@@ -3041,16 +3044,58 @@ RestoreTransactionCharacteristics(const SavedTransactionCharacteristics *s)
XactDeferrable = s->save_XactDeferrable;
}
-
/*
- * CommitTransactionCommand
+ * CommitTransactionCommand -- a wrapper function handling the
+ * loop over subtransactions to avoid a potentially dangerous recursion in
+ * CommitTransactionCommandInternal().
*/
void
CommitTransactionCommand(void)
+{
+ while (true)
+ {
+ switch (CurrentTransactionState->blockState)
+ {
+ /*
+ * The current already-failed subtransaction is ending due to
+ * a ROLLBACK or ROLLBACK TO command, so pop it and
+ * recursively examine the parent (which could be in any of
+ * several states).
+ */
+ case TBLOCK_SUBABORT_END:
+ CleanupSubTransaction();
+ continue;
+
+ /*
+ * As above, but it's not dead yet, so abort first.
+ */
+ case TBLOCK_SUBABORT_PENDING:
+ AbortSubTransaction();
+ CleanupSubTransaction();
+ continue;
+ default:
+ break;
+ }
+ CommitTransactionCommandInternal();
+ break;
+ }
+}
+
+/*
+ * CommitTransactionCommandInternal - a function doing all the material work
+ * regarding handling the commit transaction command except for loop over
+ * subtransactions.
+ */
+static void
+CommitTransactionCommandInternal(void)
{
TransactionState s = CurrentTransactionState;
SavedTransactionCharacteristics savetc;
+ /* This states are handled in CommitTransactionCommand() */
+ Assert(s->blockState != TBLOCK_SUBABORT_END &&
+ s->blockState != TBLOCK_SUBABORT_PENDING);
+
/* Must save in case we need to restore below */
SaveTransactionCharacteristics(&savetc);
@@ -3234,25 +3279,6 @@ CommitTransactionCommand(void)
BlockStateAsString(s->blockState));
break;
- /*
- * The current already-failed subtransaction is ending due to a
- * ROLLBACK or ROLLBACK TO command, so pop it and recursively
- * examine the parent (which could be in any of several states).
- */
- case TBLOCK_SUBABORT_END:
- CleanupSubTransaction();
- CommitTransactionCommand();
- break;
-
- /*
- * As above, but it's not dead yet, so abort first.
- */
- case TBLOCK_SUBABORT_PENDING:
- AbortSubTransaction();
- CleanupSubTransaction();
- CommitTransactionCommand();
- break;
-
/*
* The current subtransaction is the target of a ROLLBACK TO
* command. Abort and pop it, then start a new subtransaction
@@ -3310,17 +3336,73 @@ CommitTransactionCommand(void)
s->blockState = TBLOCK_SUBINPROGRESS;
}
break;
+ default:
+ /* Keep compiler quiet */
+ break;
}
}
/*
- * AbortCurrentTransaction
+ * AbortCurrentTransaction -- a wrapper function handling the
+ * loop over subtransactions to avoid potentially dangerous recursion in
+ * AbortCurrentTransactionInternal().
*/
void
AbortCurrentTransaction(void)
+{
+ while (true)
+ {
+ switch (CurrentTransactionState->blockState)
+ {
+ /*
+ * If we failed while trying to create a subtransaction, clean
+ * up the broken subtransaction and abort the parent. The
+ * same applies if we get a failure while ending a
+ * subtransaction.
+ */
+ case TBLOCK_SUBBEGIN:
+ case TBLOCK_SUBRELEASE:
+ case TBLOCK_SUBCOMMIT:
+ case TBLOCK_SUBABORT_PENDING:
+ case TBLOCK_SUBRESTART:
+ AbortSubTransaction();
+ CleanupSubTransaction();
+ continue;
+
+ /*
+ * Same as above, except the Abort() was already done.
+ */
+ case TBLOCK_SUBABORT_END:
+ case TBLOCK_SUBABORT_RESTART:
+ CleanupSubTransaction();
+ continue;
+ default:
+ break;
+ }
+ AbortCurrentTransactionInternal();
+ break;
+ }
+}
+
+/*
+ * AbortCurrentTransactionInternal - a function doing all the material work
+ * regarding handling the abort transaction command except for loop over
+ * subtransactions.
+ */
+static void
+AbortCurrentTransactionInternal(void)
{
TransactionState s = CurrentTransactionState;
+ /* This states are handled in AbortCurrentTransaction() */
+ Assert(s->blockState != TBLOCK_SUBBEGIN &&
+ s->blockState != TBLOCK_SUBRELEASE &&
+ s->blockState != TBLOCK_SUBCOMMIT &&
+ s->blockState != TBLOCK_SUBABORT_PENDING &&
+ s->blockState != TBLOCK_SUBRESTART &&
+ s->blockState != TBLOCK_SUBABORT_END &&
+ s->blockState != TBLOCK_SUBABORT_RESTART);
+
switch (s->blockState)
{
case TBLOCK_DEFAULT:
@@ -3441,29 +3523,8 @@ AbortCurrentTransaction(void)
AbortSubTransaction();
s->blockState = TBLOCK_SUBABORT;
break;
-
- /*
- * If we failed while trying to create a subtransaction, clean up
- * the broken subtransaction and abort the parent. The same
- * applies if we get a failure while ending a subtransaction.
- */
- case TBLOCK_SUBBEGIN:
- case TBLOCK_SUBRELEASE:
- case TBLOCK_SUBCOMMIT:
- case TBLOCK_SUBABORT_PENDING:
- case TBLOCK_SUBRESTART:
- AbortSubTransaction();
- CleanupSubTransaction();
- AbortCurrentTransaction();
- break;
-
- /*
- * Same as above, except the Abort() was already done.
- */
- case TBLOCK_SUBABORT_END:
- case TBLOCK_SUBABORT_RESTART:
- CleanupSubTransaction();
- AbortCurrentTransaction();
+ default:
+ /* Keep compiler quiet */
break;
}
}
--
2.39.3 (Apple Git-145)
Alexander Korotkov <aekorotkov@gmail.com> writes:
The revised set of remaining patches is attached.
...
0003 Avoid recursion in MemoryContext functions
I've renamed MemoryContextTraverse() => MemoryContextTraverseNext(),
which I think is a bit more intuitive. Also I fixed
MemoryContextMemConsumed(), which was still trying to use the removed
argument "print" of MemoryContextStatsInternal() function.
This patch still doesn't compile for me --- MemoryContextMemConsumed
got modified some more by commit 743112a2e, and needs minor fixes.
I initially didn't like the definition of MemoryContextTraverseNext
because it requires two copies of the "process node" logic. However,
that seems fine for most of the callers, and even where we are
duplicating logic it's just a line or so, so I guess it's ok.
However, MemoryContextTraverseNext seems undercommented to me, plus
the claim that it traverses in depth-first order is just wrong.
I found some bugs in MemoryContextStatsInternal too: the old
logic assumed that ichild exceeding max_children was the only
way to get into the summarization logic, but now ichild minus
max_children could very well be negative. Fortunately we can
just reset ichild to zero and not worry about having any
connection between the first loop and the second.
Here's a v5 of 0003 with those issues and some more-cosmetic ones
cleaned up. I didn't look at 0001 or 0002.
regards, tom lane
Attachments:
v5-0003-Avoid-recursion-in-MemoryContext-functions.patchtext/x-diff; charset=us-ascii; name=v5-0003-Avoid-recursion-in-MemoryContext-functions.patchDownload
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 1a615becae..5d426795d9 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -145,9 +145,10 @@ MemoryContext CurTransactionContext = NULL;
/* This is a transient link to the active portal's memory context: */
MemoryContext PortalContext = NULL;
+static void MemoryContextDeleteOnly(MemoryContext context);
static void MemoryContextCallResetCallbacks(MemoryContext context);
static void MemoryContextStatsInternal(MemoryContext context, int level,
- bool print, int max_children,
+ int max_level, int max_children,
MemoryContextCounters *totals,
bool print_to_stderr);
static void MemoryContextStatsPrint(MemoryContext context, void *passthru,
@@ -219,6 +220,50 @@ GetMemoryChunkHeader(const void *pointer)
return header;
}
+/*
+ * MemoryContextTraverseNext
+ * Helper function to traverse all descendants of a memory context
+ * without recursion.
+ *
+ * Recursion could lead to out-of-stack errors with deep context hierarchies,
+ * which would be unpleasant in error cleanup code paths.
+ *
+ * To process 'context' and all its descendants, use a loop like this:
+ *
+ * <process 'context'>
+ * for (MemoryContext curr = context->firstchild;
+ * curr != NULL;
+ * curr = MemoryContextTraverseNext(curr, context))
+ * {
+ * <process 'curr'>
+ * }
+ *
+ * This visits all the contexts in pre-order, that is a node is visited
+ * before its children.
+ */
+static MemoryContext
+MemoryContextTraverseNext(MemoryContext curr, MemoryContext top)
+{
+ /* After processing a node, traverse to its first child if any */
+ if (curr->firstchild != NULL)
+ return curr->firstchild;
+
+ /*
+ * After processing a childless node, traverse to its next sibling if
+ * there is one. If there isn't, traverse back up to the parent (which
+ * has already been visited, and now so have all its descendants). We're
+ * done if that is "top", otherwise traverse to its next sibling if any,
+ * otherwise repeat moving up.
+ */
+ while (curr->nextchild == NULL)
+ {
+ curr = curr->parent;
+ if (curr == top)
+ return NULL;
+ }
+ return curr->nextchild;
+}
+
/*
* Support routines to trap use of invalid memory context method IDs
* (from calling pfree or the like on a bogus pointer). As a possible
@@ -375,14 +420,13 @@ MemoryContextResetOnly(MemoryContext context)
void
MemoryContextResetChildren(MemoryContext context)
{
- MemoryContext child;
-
Assert(MemoryContextIsValid(context));
- for (child = context->firstchild; child != NULL; child = child->nextchild)
+ for (MemoryContext curr = context->firstchild;
+ curr != NULL;
+ curr = MemoryContextTraverseNext(curr, context))
{
- MemoryContextResetChildren(child);
- MemoryContextResetOnly(child);
+ MemoryContextResetOnly(curr);
}
}
@@ -392,21 +436,60 @@ MemoryContextResetChildren(MemoryContext context)
* allocated therein.
*
* The type-specific delete routine removes all storage for the context,
- * but we have to recurse to handle the children.
- * We must also delink the context from its parent, if it has one.
+ * but we have to deal with descendant nodes here.
*/
void
MemoryContextDelete(MemoryContext context)
+{
+ MemoryContext curr;
+
+ Assert(MemoryContextIsValid(context));
+
+ /*
+ * Delete subcontexts from the bottom up.
+ *
+ * Note: Do not use recursion here. A "stack depth limit exceeded" error
+ * would be unpleasant if we're already in the process of cleaning up from
+ * transaction abort. We also cannot use MemoryContextTraverseNext() here
+ * because we modify the tree as we go.
+ */
+ curr = context;
+ for (;;)
+ {
+ MemoryContext parent;
+
+ /* Descend down until we find a leaf context with no children */
+ while (curr->firstchild != NULL)
+ curr = curr->firstchild;
+
+ /*
+ * We're now at a leaf with no children. Free it and continue from the
+ * parent. Or if this was the original node, we're all done.
+ */
+ parent = curr->parent;
+ MemoryContextDeleteOnly(curr);
+
+ if (curr == context)
+ break;
+ curr = parent;
+ }
+}
+
+/*
+ * Subroutine of MemoryContextDelete,
+ * to delete a context that has no children.
+ * We must also delink the context from its parent, if it has one.
+ */
+static void
+MemoryContextDeleteOnly(MemoryContext context)
{
Assert(MemoryContextIsValid(context));
/* We had better not be deleting TopMemoryContext ... */
Assert(context != TopMemoryContext);
/* And not CurrentMemoryContext, either */
Assert(context != CurrentMemoryContext);
-
- /* save a function call in common case where there are no children */
- if (context->firstchild != NULL)
- MemoryContextDeleteChildren(context);
+ /* All the children should've been deleted already */
+ Assert(context->firstchild == NULL);
/*
* It's not entirely clear whether 'tis better to do this before or after
@@ -672,12 +755,12 @@ MemoryContextMemAllocated(MemoryContext context, bool recurse)
if (recurse)
{
- MemoryContext child;
-
- for (child = context->firstchild;
- child != NULL;
- child = child->nextchild)
- total += MemoryContextMemAllocated(child, true);
+ for (MemoryContext curr = context->firstchild;
+ curr != NULL;
+ curr = MemoryContextTraverseNext(curr, context))
+ {
+ total += curr->mem_allocated;
+ }
}
return total;
@@ -691,9 +774,20 @@ void
MemoryContextMemConsumed(MemoryContext context,
MemoryContextCounters *consumed)
{
+ Assert(MemoryContextIsValid(context));
+
memset(consumed, 0, sizeof(*consumed));
- MemoryContextStatsInternal(context, 0, false, 0, consumed, false);
+ /* Examine the context itself */
+ context->methods->stats(context, NULL, NULL, consumed, false);
+
+ /* Examine children, using iteration not recursion */
+ for (MemoryContext curr = context->firstchild;
+ curr != NULL;
+ curr = MemoryContextTraverseNext(curr, context))
+ {
+ curr->methods->stats(curr, NULL, NULL, consumed, false);
+ }
}
/*
@@ -707,8 +801,8 @@ MemoryContextMemConsumed(MemoryContext context,
void
MemoryContextStats(MemoryContext context)
{
- /* A hard-wired limit on the number of children is usually good enough */
- MemoryContextStatsDetail(context, 100, true);
+ /* Hard-wired limits are usually good enough */
+ MemoryContextStatsDetail(context, 100, 100, true);
}
/*
@@ -720,14 +814,16 @@ MemoryContextStats(MemoryContext context)
* with fprintf(stderr), otherwise use ereport().
*/
void
-MemoryContextStatsDetail(MemoryContext context, int max_children,
+MemoryContextStatsDetail(MemoryContext context,
+ int max_level, int max_children,
bool print_to_stderr)
{
MemoryContextCounters grand_totals;
memset(&grand_totals, 0, sizeof(grand_totals));
- MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals, print_to_stderr);
+ MemoryContextStatsInternal(context, 0, max_level, max_children,
+ &grand_totals, print_to_stderr);
if (print_to_stderr)
fprintf(stderr,
@@ -736,7 +832,7 @@ MemoryContextStatsDetail(MemoryContext context, int max_children,
grand_totals.freespace, grand_totals.freechunks,
grand_totals.totalspace - grand_totals.freespace);
else
-
+ {
/*
* Use LOG_SERVER_ONLY to prevent the memory contexts from being sent
* to the connected client.
@@ -754,22 +850,22 @@ MemoryContextStatsDetail(MemoryContext context, int max_children,
grand_totals.totalspace, grand_totals.nblocks,
grand_totals.freespace, grand_totals.freechunks,
grand_totals.totalspace - grand_totals.freespace)));
+ }
}
/*
* MemoryContextStatsInternal
* One recursion level for MemoryContextStats
*
- * Print this context if print is true, but in any case accumulate counts into
- * *totals (if given).
+ * Print stats for this context if possible, but in any case accumulate counts
+ * into *totals (if not NULL).
*/
static void
MemoryContextStatsInternal(MemoryContext context, int level,
- bool print, int max_children,
+ int max_level, int max_children,
MemoryContextCounters *totals,
bool print_to_stderr)
{
- MemoryContextCounters local_totals;
MemoryContext child;
int ichild;
@@ -777,65 +873,72 @@ MemoryContextStatsInternal(MemoryContext context, int level,
/* Examine the context itself */
context->methods->stats(context,
- print ? MemoryContextStatsPrint : NULL,
+ MemoryContextStatsPrint,
(void *) &level,
totals, print_to_stderr);
/*
- * Examine children. If there are more than max_children of them, we do
- * not print the rest explicitly, but just summarize them.
+ * Examine children.
+ *
+ * If we are past the recursion depth limit or already running low on
+ * stack, do not print them explicitly but just summarize them. Similarly,
+ * if there are more than max_children of them, we do not print the rest
+ * explicitly, but just summarize them.
*/
- memset(&local_totals, 0, sizeof(local_totals));
-
- for (child = context->firstchild, ichild = 0;
- child != NULL;
- child = child->nextchild, ichild++)
+ child = context->firstchild;
+ ichild = 0;
+ if (level < max_level && !stack_is_too_deep())
{
- if (ichild < max_children)
+ for (; child != NULL && ichild < max_children;
+ child = child->nextchild, ichild++)
+ {
MemoryContextStatsInternal(child, level + 1,
- print, max_children,
+ max_level, max_children,
totals,
print_to_stderr);
- else
- MemoryContextStatsInternal(child, level + 1,
- false, max_children,
- &local_totals,
- print_to_stderr);
+ }
}
- /* Deal with excess children */
- if (ichild > max_children)
+ if (child != NULL)
{
- if (print)
+ /* Summarize the rest of the children, avoiding recursion. */
+ MemoryContextCounters local_totals;
+
+ memset(&local_totals, 0, sizeof(local_totals));
+
+ ichild = 0;
+ while (child != NULL)
+ {
+ child->methods->stats(child, NULL, NULL, &local_totals, false);
+ ichild++;
+ child = MemoryContextTraverseNext(child, context);
+ }
+
+ if (print_to_stderr)
{
- if (print_to_stderr)
- {
- int i;
-
- for (i = 0; i <= level; i++)
- fprintf(stderr, " ");
- fprintf(stderr,
- "%d more child contexts containing %zu total in %zu blocks; %zu free (%zu chunks); %zu used\n",
- ichild - max_children,
- local_totals.totalspace,
- local_totals.nblocks,
- local_totals.freespace,
- local_totals.freechunks,
- local_totals.totalspace - local_totals.freespace);
- }
- else
- ereport(LOG_SERVER_ONLY,
- (errhidestmt(true),
- errhidecontext(true),
- errmsg_internal("level: %d; %d more child contexts containing %zu total in %zu blocks; %zu free (%zu chunks); %zu used",
- level,
- ichild - max_children,
- local_totals.totalspace,
- local_totals.nblocks,
- local_totals.freespace,
- local_totals.freechunks,
- local_totals.totalspace - local_totals.freespace)));
+ for (int i = 0; i <= level; i++)
+ fprintf(stderr, " ");
+ fprintf(stderr,
+ "%d more child contexts containing %zu total in %zu blocks; %zu free (%zu chunks); %zu used\n",
+ ichild,
+ local_totals.totalspace,
+ local_totals.nblocks,
+ local_totals.freespace,
+ local_totals.freechunks,
+ local_totals.totalspace - local_totals.freespace);
}
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("level: %d; %d more child contexts containing %zu total in %zu blocks; %zu free (%zu chunks); %zu used",
+ level,
+ ichild,
+ local_totals.totalspace,
+ local_totals.nblocks,
+ local_totals.freespace,
+ local_totals.freechunks,
+ local_totals.totalspace - local_totals.freespace)));
if (totals)
{
@@ -928,7 +1031,7 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
/*
* MemoryContextCheck
- * Check all chunks in the named context.
+ * Check all chunks in the named context and its children.
*
* This is just a debugging utility, so it's not fancy.
*/
@@ -936,13 +1039,16 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
void
MemoryContextCheck(MemoryContext context)
{
- MemoryContext child;
-
Assert(MemoryContextIsValid(context));
-
context->methods->check(context);
- for (child = context->firstchild; child != NULL; child = child->nextchild)
- MemoryContextCheck(child);
+
+ for (MemoryContext curr = context->firstchild;
+ curr != NULL;
+ curr = MemoryContextTraverseNext(curr, context))
+ {
+ Assert(MemoryContextIsValid(curr));
+ curr->methods->check(curr);
+ }
}
#endif
@@ -1183,14 +1289,15 @@ ProcessLogMemoryContextInterrupt(void)
/*
* When a backend process is consuming huge memory, logging all its memory
* contexts might overrun available disk space. To prevent this, we limit
- * the number of child contexts to log per parent to 100.
+ * the depth of the hierarchy, as well as the number of child contexts to
+ * log per parent to 100.
*
* As with MemoryContextStats(), we suppose that practical cases where the
* dump gets long will typically be huge numbers of siblings under the
* same parent context; while the additional debugging value from seeing
* details about individual siblings beyond 100 will not be large.
*/
- MemoryContextStatsDetail(TopMemoryContext, 100, false);
+ MemoryContextStatsDetail(TopMemoryContext, 100, 100, false);
}
void *
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 7fd41d20ca..6e5fa72b0e 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -87,7 +87,8 @@ extern Size MemoryContextMemAllocated(MemoryContext context, bool recurse);
extern void MemoryContextMemConsumed(MemoryContext context,
MemoryContextCounters *consumed);
extern void MemoryContextStats(MemoryContext context);
-extern void MemoryContextStatsDetail(MemoryContext context, int max_children,
+extern void MemoryContextStatsDetail(MemoryContext context,
+ int max_level, int max_children,
bool print_to_stderr);
extern void MemoryContextAllowInCriticalSection(MemoryContext context,
bool allow);
On Thu, Mar 7, 2024 at 12:52 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Alexander Korotkov <aekorotkov@gmail.com> writes:
The revised set of remaining patches is attached.
...
0003 Avoid recursion in MemoryContext functions
I've renamed MemoryContextTraverse() => MemoryContextTraverseNext(),
which I think is a bit more intuitive. Also I fixed
MemoryContextMemConsumed(), which was still trying to use the removed
argument "print" of MemoryContextStatsInternal() function.This patch still doesn't compile for me --- MemoryContextMemConsumed
got modified some more by commit 743112a2e, and needs minor fixes.I initially didn't like the definition of MemoryContextTraverseNext
because it requires two copies of the "process node" logic. However,
that seems fine for most of the callers, and even where we are
duplicating logic it's just a line or so, so I guess it's ok.
However, MemoryContextTraverseNext seems undercommented to me, plus
the claim that it traverses in depth-first order is just wrong.I found some bugs in MemoryContextStatsInternal too: the old
logic assumed that ichild exceeding max_children was the only
way to get into the summarization logic, but now ichild minus
max_children could very well be negative. Fortunately we can
just reset ichild to zero and not worry about having any
connection between the first loop and the second.Here's a v5 of 0003 with those issues and some more-cosmetic ones
cleaned up. I didn't look at 0001 or 0002.
Tom, thank you for your revision of this patch!
Sorry for tediousness, but isn't pre-order a variation of depth-first order
[1]: ?
Links.
1. https://en.wikipedia.org/wiki/Tree_traversal#Depth-first_search
------
Regards,
Alexander Korotkov
Alexander Korotkov <aekorotkov@gmail.com> writes:
Sorry for tediousness, but isn't pre-order a variation of depth-first order
[1]?
To me, depth-first implies visiting children before parents.
Do I have the terminology wrong?
regards, tom lane
Hi, Egor!
On Thu, Mar 7, 2024 at 9:53 AM Egor Chindyaskin <kyzevan23@mail.ru> wrote:
6 march 2024 г., at 19:17, Alexander Korotkov <aekorotkov@gmail.com> wrote:
The revised set of remaining patches is attached.
0001 Turn tail recursion into iteration in CommitTransactionCommand()
I did minor revision of comments and code blocks order to improve the
readability.0002 Avoid stack overflow in ShowTransactionStateRec()
I didn't notice any issues, leave this piece as is.0003 Avoid recursion in MemoryContext functions
I've renamed MemoryContextTraverse() => MemoryContextTraverseNext(),
which I think is a bit more intuitive. Also I fixed
MemoryContextMemConsumed(), which was still trying to use the removed
argument "print" of MemoryContextStatsInternal() function.Generally, I think this patchset fixes important stack overflow holes.
It is quite straightforward, clear and the code has a good shape. I'm
going to push this if no objections.I have tested the scripts from message [1]. After applying these patches and Tom Lane’s patch from message [2], all of the above mentioned functions no longer caused the server to crash. I also tried increasing the values in the presented scripts, which also did not lead to server crashes. Thank you!
Also, I would like to clarify something. Will fixes from message [3] and others be backported to all other branches, not just the master branch? As far as I remember, Tom Lane made corrections to all branches. For example [4].Links:
1. /messages/by-id/343ff14f-3060-4f88-9cc6-efdb390185df@mail.ru
2. /messages/by-id/386032.1709765547@sss.pgh.pa.us
3. /messages/by-id/CAPpHfduZqAjF+7rDRP-RGNHjOXy7nvFROQ0MGS436f8FPY5DpQ@mail.gmail.com
4. https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=e07ebd4b
Thank you for your feedback!
Initially I didn't intend to backpatch any of these. But on second
thought with the references you provided, I think we should backpatch
simple check_stack_depth() checks from d57b7cc333 to all supported
branches, but apply refactoring of memory contextes and transaction
commit/abort just to master. Opinions?
------
Regards,
Alexander Korotkov
Import Notes
Reply to msg id not found: DE5FD776-A8CD-4378-BCFA-3BF30F1F6D60@mail.ru
On Thu, Mar 7, 2024 at 1:49 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Alexander Korotkov <aekorotkov@gmail.com> writes:
Sorry for tediousness, but isn't pre-order a variation of depth-first order
[1]?To me, depth-first implies visiting children before parents.
Do I have the terminology wrong?
According to Wikipedia, depth-first is a general term describing the
tree traversal algorithm, which goes as deep as possible in one branch
before visiting other branches. The order of between parents and
children, and between siblings specifies the variation of depth-first
search, and pre-order is one of them. But "pre-order" is the most
accurate term for MemoryContextTraverseNext() anyway.
------
Regards,
Alexander Korotkov
On Thu, Mar 7, 2024 at 11:07 AM Alexander Korotkov <aekorotkov@gmail.com> wrote:
On Thu, Mar 7, 2024 at 9:53 AM Egor Chindyaskin <kyzevan23@mail.ru> wrote:
6 march 2024 г., at 19:17, Alexander Korotkov <aekorotkov@gmail.com> wrote:
The revised set of remaining patches is attached.
0001 Turn tail recursion into iteration in CommitTransactionCommand()
I did minor revision of comments and code blocks order to improve the
readability.0002 Avoid stack overflow in ShowTransactionStateRec()
I didn't notice any issues, leave this piece as is.0003 Avoid recursion in MemoryContext functions
I've renamed MemoryContextTraverse() => MemoryContextTraverseNext(),
which I think is a bit more intuitive. Also I fixed
MemoryContextMemConsumed(), which was still trying to use the removed
argument "print" of MemoryContextStatsInternal() function.Generally, I think this patchset fixes important stack overflow holes.
It is quite straightforward, clear and the code has a good shape. I'm
going to push this if no objections.I have tested the scripts from message [1]. After applying these patches and Tom Lane’s patch from message [2], all of the above mentioned functions no longer caused the server to crash. I also tried increasing the values in the presented scripts, which also did not lead to server crashes. Thank you!
Also, I would like to clarify something. Will fixes from message [3] and others be backported to all other branches, not just the master branch? As far as I remember, Tom Lane made corrections to all branches. For example [4].Links:
1. /messages/by-id/343ff14f-3060-4f88-9cc6-efdb390185df@mail.ru
2. /messages/by-id/386032.1709765547@sss.pgh.pa.us
3. /messages/by-id/CAPpHfduZqAjF+7rDRP-RGNHjOXy7nvFROQ0MGS436f8FPY5DpQ@mail.gmail.com
4. https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=e07ebd4bThank you for your feedback!
Initially I didn't intend to backpatch any of these. But on second
thought with the references you provided, I think we should backpatch
simple check_stack_depth() checks from d57b7cc333 to all supported
branches, but apply refactoring of memory contextes and transaction
commit/abort just to master. Opinions?
I've just backpatched check_stack_depth() checks to all supported branches.
------
Regards,
Alexander Korotkov
Hi,
On 2024-03-06 14:17:23 +0200, Alexander Korotkov wrote:
0001 Turn tail recursion into iteration in CommitTransactionCommand()
I did minor revision of comments and code blocks order to improve the
readability.
After sending
/messages/by-id/20240414223305.m3i5eju6zylabvln@awork3.anarazel.de
I looked some more at important areas where changes didn't have code
coverage. One thing I noticed was that the "non-internal" part of
AbortCurrentTransaction() is uncovered:
https://anarazel.de/postgres/cov/16-vs-HEAD-2024-04-14/src/backend/access/transam/xact.c.gcov.html#L3403
Which made me try to understand fefd9a3fed2. I'm a bit confused about why
some parts are handled in CommitCurrentTransaction()/AbortCurrentTransaction()
and others are in the *Internal functions.
I understand that fefd9a3fed2 needed to remove the recursion in
CommitTransactionCommand()/AbortCurrentTransaction(). But I don't understand
why that means having some code in in the non-internal and some in the
internal functions? Wouldn't it be easier to just have all the state handling
code in the Internal() function and just break after the
CleanupSubTransaction() calls?
That's of course largely unrelated to the coverage aspects. I just got
curious.
Greetings,
Andres Freund
On Tue, Apr 16, 2024 at 1:48 AM Andres Freund <andres@anarazel.de> wrote:
On 2024-03-06 14:17:23 +0200, Alexander Korotkov wrote:
0001 Turn tail recursion into iteration in CommitTransactionCommand()
I did minor revision of comments and code blocks order to improve the
readability.After sending
/messages/by-id/20240414223305.m3i5eju6zylabvln@awork3.anarazel.de
I looked some more at important areas where changes didn't have code
coverage. One thing I noticed was that the "non-internal" part of
AbortCurrentTransaction() is uncovered:
https://anarazel.de/postgres/cov/16-vs-HEAD-2024-04-14/src/backend/access/transam/xact.c.gcov.html#L3403Which made me try to understand fefd9a3fed2. I'm a bit confused about why
some parts are handled in CommitCurrentTransaction()/AbortCurrentTransaction()
and others are in the *Internal functions.I understand that fefd9a3fed2 needed to remove the recursion in
CommitTransactionCommand()/AbortCurrentTransaction(). But I don't understand
why that means having some code in in the non-internal and some in the
internal functions? Wouldn't it be easier to just have all the state handling
code in the Internal() function and just break after the
CleanupSubTransaction() calls?
I'm not sure I correctly get what you mean. Do you think the attached
patch matches the direction you're pointing? The patch itself is not
final, it requires cleanup and comments revision, just to check the
direction.
------
Regards,
Alexander Korotkov
Attachments:
abort_commit_transaction.patchapplication/octet-stream; name=abort_commit_transaction.patchDownload
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index df5a67e4c31..8870d1f961e 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -346,8 +346,8 @@ static void CommitTransaction(void);
static TransactionId RecordTransactionAbort(bool isSubXact);
static void StartTransaction(void);
-static void CommitTransactionCommandInternal(void);
-static void AbortCurrentTransactionInternal(void);
+static bool CommitTransactionCommandInternal(void);
+static bool AbortCurrentTransactionInternal(void);
static void StartSubTransaction(void);
static void CommitSubTransaction(void);
@@ -3092,33 +3092,7 @@ RestoreTransactionCharacteristics(const SavedTransactionCharacteristics *s)
void
CommitTransactionCommand(void)
{
- while (true)
- {
- switch (CurrentTransactionState->blockState)
- {
- /*
- * The current already-failed subtransaction is ending due to
- * a ROLLBACK or ROLLBACK TO command, so pop it and
- * recursively examine the parent (which could be in any of
- * several states).
- */
- case TBLOCK_SUBABORT_END:
- CleanupSubTransaction();
- continue;
-
- /*
- * As above, but it's not dead yet, so abort first.
- */
- case TBLOCK_SUBABORT_PENDING:
- AbortSubTransaction();
- CleanupSubTransaction();
- continue;
- default:
- break;
- }
- CommitTransactionCommandInternal();
- break;
- }
+ while (!CommitTransactionCommandInternal());
}
/*
@@ -3126,16 +3100,12 @@ CommitTransactionCommand(void)
* regarding handling the commit transaction command except for loop over
* subtransactions.
*/
-static void
+static bool
CommitTransactionCommandInternal(void)
{
TransactionState s = CurrentTransactionState;
SavedTransactionCharacteristics savetc;
- /* This states are handled in CommitTransactionCommand() */
- Assert(s->blockState != TBLOCK_SUBABORT_END &&
- s->blockState != TBLOCK_SUBABORT_PENDING);
-
/* Must save in case we need to restore below */
SaveTransactionCharacteristics(&savetc);
@@ -3376,10 +3346,31 @@ CommitTransactionCommandInternal(void)
s->blockState = TBLOCK_SUBINPROGRESS;
}
break;
+
+ /*
+ * The current already-failed subtransaction is ending due to
+ * a ROLLBACK or ROLLBACK TO command, so pop it and
+ * recursively examine the parent (which could be in any of
+ * several states).
+ */
+ case TBLOCK_SUBABORT_END:
+ CleanupSubTransaction();
+ return false;
+
+ /*
+ * As above, but it's not dead yet, so abort first.
+ */
+ case TBLOCK_SUBABORT_PENDING:
+ AbortSubTransaction();
+ CleanupSubTransaction();
+ return false;
+
default:
/* Keep compiler quiet */
break;
}
+
+ return true;
}
/*
@@ -3390,38 +3381,7 @@ CommitTransactionCommandInternal(void)
void
AbortCurrentTransaction(void)
{
- while (true)
- {
- switch (CurrentTransactionState->blockState)
- {
- /*
- * If we failed while trying to create a subtransaction, clean
- * up the broken subtransaction and abort the parent. The
- * same applies if we get a failure while ending a
- * subtransaction.
- */
- case TBLOCK_SUBBEGIN:
- case TBLOCK_SUBRELEASE:
- case TBLOCK_SUBCOMMIT:
- case TBLOCK_SUBABORT_PENDING:
- case TBLOCK_SUBRESTART:
- AbortSubTransaction();
- CleanupSubTransaction();
- continue;
-
- /*
- * Same as above, except the Abort() was already done.
- */
- case TBLOCK_SUBABORT_END:
- case TBLOCK_SUBABORT_RESTART:
- CleanupSubTransaction();
- continue;
- default:
- break;
- }
- AbortCurrentTransactionInternal();
- break;
- }
+ while (!AbortCurrentTransactionInternal());
}
/*
@@ -3429,20 +3389,11 @@ AbortCurrentTransaction(void)
* regarding handling the abort transaction command except for loop over
* subtransactions.
*/
-static void
+static bool
AbortCurrentTransactionInternal(void)
{
TransactionState s = CurrentTransactionState;
- /* This states are handled in AbortCurrentTransaction() */
- Assert(s->blockState != TBLOCK_SUBBEGIN &&
- s->blockState != TBLOCK_SUBRELEASE &&
- s->blockState != TBLOCK_SUBCOMMIT &&
- s->blockState != TBLOCK_SUBABORT_PENDING &&
- s->blockState != TBLOCK_SUBRESTART &&
- s->blockState != TBLOCK_SUBABORT_END &&
- s->blockState != TBLOCK_SUBABORT_RESTART);
-
switch (s->blockState)
{
case TBLOCK_DEFAULT:
@@ -3563,10 +3514,36 @@ AbortCurrentTransactionInternal(void)
AbortSubTransaction();
s->blockState = TBLOCK_SUBABORT;
break;
+
+ /*
+ * If we failed while trying to create a subtransaction, clean
+ * up the broken subtransaction and abort the parent. The
+ * same applies if we get a failure while ending a
+ * subtransaction.
+ */
+ case TBLOCK_SUBBEGIN:
+ case TBLOCK_SUBRELEASE:
+ case TBLOCK_SUBCOMMIT:
+ case TBLOCK_SUBABORT_PENDING:
+ case TBLOCK_SUBRESTART:
+ AbortSubTransaction();
+ CleanupSubTransaction();
+ return false;
+
+ /*
+ * Same as above, except the Abort() was already done.
+ */
+ case TBLOCK_SUBABORT_END:
+ case TBLOCK_SUBABORT_RESTART:
+ CleanupSubTransaction();
+ return false;
+
default:
/* Keep compiler quiet */
break;
}
+
+ return true;
}
/*
Hi,
On 2024-04-16 15:45:42 +0300, Alexander Korotkov wrote:
On Tue, Apr 16, 2024 at 1:48 AM Andres Freund <andres@anarazel.de> wrote:
On 2024-03-06 14:17:23 +0200, Alexander Korotkov wrote:
0001 Turn tail recursion into iteration in CommitTransactionCommand()
I did minor revision of comments and code blocks order to improve the
readability.After sending
/messages/by-id/20240414223305.m3i5eju6zylabvln@awork3.anarazel.de
I looked some more at important areas where changes didn't have code
coverage. One thing I noticed was that the "non-internal" part of
AbortCurrentTransaction() is uncovered:
https://anarazel.de/postgres/cov/16-vs-HEAD-2024-04-14/src/backend/access/transam/xact.c.gcov.html#L3403Which made me try to understand fefd9a3fed2. I'm a bit confused about why
some parts are handled in CommitCurrentTransaction()/AbortCurrentTransaction()
and others are in the *Internal functions.I understand that fefd9a3fed2 needed to remove the recursion in
CommitTransactionCommand()/AbortCurrentTransaction(). But I don't understand
why that means having some code in in the non-internal and some in the
internal functions? Wouldn't it be easier to just have all the state handling
code in the Internal() function and just break after the
CleanupSubTransaction() calls?I'm not sure I correctly get what you mean. Do you think the attached
patch matches the direction you're pointing? The patch itself is not
final, it requires cleanup and comments revision, just to check the
direction.
Something like that, yea. The split does seem less confusing that way to me,
but also not 100% certain.
Greetings,
Andres Freund
On Tue, Apr 16, 2024 at 6:35 PM Andres Freund <andres@anarazel.de> wrote:
On 2024-04-16 15:45:42 +0300, Alexander Korotkov wrote:
On Tue, Apr 16, 2024 at 1:48 AM Andres Freund <andres@anarazel.de> wrote:
On 2024-03-06 14:17:23 +0200, Alexander Korotkov wrote:
0001 Turn tail recursion into iteration in CommitTransactionCommand()
I did minor revision of comments and code blocks order to improve the
readability.After sending
/messages/by-id/20240414223305.m3i5eju6zylabvln@awork3.anarazel.de
I looked some more at important areas where changes didn't have code
coverage. One thing I noticed was that the "non-internal" part of
AbortCurrentTransaction() is uncovered:
https://anarazel.de/postgres/cov/16-vs-HEAD-2024-04-14/src/backend/access/transam/xact.c.gcov.html#L3403Which made me try to understand fefd9a3fed2. I'm a bit confused about why
some parts are handled in CommitCurrentTransaction()/AbortCurrentTransaction()
and others are in the *Internal functions.I understand that fefd9a3fed2 needed to remove the recursion in
CommitTransactionCommand()/AbortCurrentTransaction(). But I don't understand
why that means having some code in in the non-internal and some in the
internal functions? Wouldn't it be easier to just have all the state handling
code in the Internal() function and just break after the
CleanupSubTransaction() calls?I'm not sure I correctly get what you mean. Do you think the attached
patch matches the direction you're pointing? The patch itself is not
final, it requires cleanup and comments revision, just to check the
direction.Something like that, yea. The split does seem less confusing that way to me,
but also not 100% certain.
Thank you for your feedback. I'm going to go ahead and polish this patch.
------
Regards,
Alexander Korotkov
On Tue, Apr 16, 2024 at 7:42 PM Alexander Korotkov <aekorotkov@gmail.com> wrote:
On Tue, Apr 16, 2024 at 6:35 PM Andres Freund <andres@anarazel.de> wrote:
On 2024-04-16 15:45:42 +0300, Alexander Korotkov wrote:
On Tue, Apr 16, 2024 at 1:48 AM Andres Freund <andres@anarazel.de> wrote:
On 2024-03-06 14:17:23 +0200, Alexander Korotkov wrote:
0001 Turn tail recursion into iteration in CommitTransactionCommand()
I did minor revision of comments and code blocks order to improve the
readability.After sending
/messages/by-id/20240414223305.m3i5eju6zylabvln@awork3.anarazel.de
I looked some more at important areas where changes didn't have code
coverage. One thing I noticed was that the "non-internal" part of
AbortCurrentTransaction() is uncovered:
https://anarazel.de/postgres/cov/16-vs-HEAD-2024-04-14/src/backend/access/transam/xact.c.gcov.html#L3403Which made me try to understand fefd9a3fed2. I'm a bit confused about why
some parts are handled in CommitCurrentTransaction()/AbortCurrentTransaction()
and others are in the *Internal functions.I understand that fefd9a3fed2 needed to remove the recursion in
CommitTransactionCommand()/AbortCurrentTransaction(). But I don't understand
why that means having some code in in the non-internal and some in the
internal functions? Wouldn't it be easier to just have all the state handling
code in the Internal() function and just break after the
CleanupSubTransaction() calls?I'm not sure I correctly get what you mean. Do you think the attached
patch matches the direction you're pointing? The patch itself is not
final, it requires cleanup and comments revision, just to check the
direction.Something like that, yea. The split does seem less confusing that way to me,
but also not 100% certain.Thank you for your feedback. I'm going to go ahead and polish this patch.
I've invested more time into polishing this. I'm intended to push
this. Could you, please, take a look before?
------
Regards,
Alexander Korotkov
Attachments:
v2-0001-Refactoring-for-CommitTransactionCommand-AbortCur.patchapplication/octet-stream; name=v2-0001-Refactoring-for-CommitTransactionCommand-AbortCur.patchDownload
From a3d8d3e2cff9eaf7ec7bbfe6b84f1171d1b9a811 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <akorotkov@postgresql.org>
Date: Wed, 17 Apr 2024 14:27:33 +0300
Subject: [PATCH v2] Refactoring for
CommitTransactionCommand()/AbortCurrentTransaction()
fefd9a3fed turned tail recursion of CommitTransactionCommand() and
AbortCurrentTransaction() into iteration. However, it splits the handling of
cases between different functions.
This commit puts the handling of all the cases into
AbortCurrentTransactionInternal() and CommitTransactionCommandInternal().
Now CommitTransactionCommand() and AbortCurrentTransaction() are just doing
the repeated calls of internal functions.
Reported-by: Andres Freund
Discussion: https://postgr.es/m/20240415224834.w6piwtefskoh32mv%40awork3.anarazel.de
Author: Andres Freund
---
src/backend/access/transam/xact.c | 159 +++++++++++++-----------------
1 file changed, 71 insertions(+), 88 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index df5a67e4c31..d82d083cf43 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -346,8 +346,8 @@ static void CommitTransaction(void);
static TransactionId RecordTransactionAbort(bool isSubXact);
static void StartTransaction(void);
-static void CommitTransactionCommandInternal(void);
-static void AbortCurrentTransactionInternal(void);
+static bool CommitTransactionCommandInternal(void);
+static bool AbortCurrentTransactionInternal(void);
static void StartSubTransaction(void);
static void CommitSubTransaction(void);
@@ -3092,50 +3092,25 @@ RestoreTransactionCharacteristics(const SavedTransactionCharacteristics *s)
void
CommitTransactionCommand(void)
{
- while (true)
- {
- switch (CurrentTransactionState->blockState)
- {
- /*
- * The current already-failed subtransaction is ending due to
- * a ROLLBACK or ROLLBACK TO command, so pop it and
- * recursively examine the parent (which could be in any of
- * several states).
- */
- case TBLOCK_SUBABORT_END:
- CleanupSubTransaction();
- continue;
-
- /*
- * As above, but it's not dead yet, so abort first.
- */
- case TBLOCK_SUBABORT_PENDING:
- AbortSubTransaction();
- CleanupSubTransaction();
- continue;
- default:
- break;
- }
- CommitTransactionCommandInternal();
- break;
- }
+ /*
+ * Repeatedly call CommitTransactionCommandInternal() untill all the work
+ * is done.
+ */
+ while (!CommitTransactionCommandInternal());
}
/*
- * CommitTransactionCommandInternal - a function doing all the material work
- * regarding handling the commit transaction command except for loop over
- * subtransactions.
+ * CommitTransactionCommandInternal - a function doing an iteration of work
+ * regarding handling the commit transaction command. In the case of
+ * subtransactions more than one iterations could be required. Returns
+ * true when no more iterations required, false otherwise.
*/
-static void
+static bool
CommitTransactionCommandInternal(void)
{
TransactionState s = CurrentTransactionState;
SavedTransactionCharacteristics savetc;
- /* This states are handled in CommitTransactionCommand() */
- Assert(s->blockState != TBLOCK_SUBABORT_END &&
- s->blockState != TBLOCK_SUBABORT_PENDING);
-
/* Must save in case we need to restore below */
SaveTransactionCharacteristics(&savetc);
@@ -3319,6 +3294,25 @@ CommitTransactionCommandInternal(void)
BlockStateAsString(s->blockState));
break;
+ /*
+ * The current already-failed subtransaction is ending due to a
+ * ROLLBACK or ROLLBACK TO command, so pop it and recursively
+ * examine the parent (which could be in any of several states).
+ * As we need to examine the parent, return false to request the
+ * caller to do the next iteration.
+ */
+ case TBLOCK_SUBABORT_END:
+ CleanupSubTransaction();
+ return false;
+
+ /*
+ * As above, but it's not dead yet, so abort first.
+ */
+ case TBLOCK_SUBABORT_PENDING:
+ AbortSubTransaction();
+ CleanupSubTransaction();
+ return false;
+
/*
* The current subtransaction is the target of a ROLLBACK TO
* command. Abort and pop it, then start a new subtransaction
@@ -3376,10 +3370,10 @@ CommitTransactionCommandInternal(void)
s->blockState = TBLOCK_SUBINPROGRESS;
}
break;
- default:
- /* Keep compiler quiet */
- break;
}
+
+ /* Done, no more iterations required */
+ return true;
}
/*
@@ -3390,59 +3384,24 @@ CommitTransactionCommandInternal(void)
void
AbortCurrentTransaction(void)
{
- while (true)
- {
- switch (CurrentTransactionState->blockState)
- {
- /*
- * If we failed while trying to create a subtransaction, clean
- * up the broken subtransaction and abort the parent. The
- * same applies if we get a failure while ending a
- * subtransaction.
- */
- case TBLOCK_SUBBEGIN:
- case TBLOCK_SUBRELEASE:
- case TBLOCK_SUBCOMMIT:
- case TBLOCK_SUBABORT_PENDING:
- case TBLOCK_SUBRESTART:
- AbortSubTransaction();
- CleanupSubTransaction();
- continue;
-
- /*
- * Same as above, except the Abort() was already done.
- */
- case TBLOCK_SUBABORT_END:
- case TBLOCK_SUBABORT_RESTART:
- CleanupSubTransaction();
- continue;
- default:
- break;
- }
- AbortCurrentTransactionInternal();
- break;
- }
+ /*
+ * Repeatedly call AbortCurrentTransactionInternal() untill all the work
+ * is done.
+ */
+ while (!AbortCurrentTransactionInternal());
}
/*
- * AbortCurrentTransactionInternal - a function doing all the material work
- * regarding handling the abort transaction command except for loop over
- * subtransactions.
+ * AbortCurrentTransactionInternal - a function doing an iteration of work
+ * regarding handling the current transaction abort. In the case of
+ * subtransactions more than one iterations could be required. Returns
+ * true when no more iterations required, false otherwise.
*/
-static void
+static bool
AbortCurrentTransactionInternal(void)
{
TransactionState s = CurrentTransactionState;
- /* This states are handled in AbortCurrentTransaction() */
- Assert(s->blockState != TBLOCK_SUBBEGIN &&
- s->blockState != TBLOCK_SUBRELEASE &&
- s->blockState != TBLOCK_SUBCOMMIT &&
- s->blockState != TBLOCK_SUBABORT_PENDING &&
- s->blockState != TBLOCK_SUBRESTART &&
- s->blockState != TBLOCK_SUBABORT_END &&
- s->blockState != TBLOCK_SUBABORT_RESTART);
-
switch (s->blockState)
{
case TBLOCK_DEFAULT:
@@ -3563,10 +3522,34 @@ AbortCurrentTransactionInternal(void)
AbortSubTransaction();
s->blockState = TBLOCK_SUBABORT;
break;
- default:
- /* Keep compiler quiet */
- break;
+
+ /*
+ * If we failed while trying to create a subtransaction, clean up
+ * the broken subtransaction and abort the parent. The same
+ * applies if we get a failure while ending a subtransaction. As
+ * we need to abort the parent, return false to request the caller
+ * to do the next iteration.
+ */
+ case TBLOCK_SUBBEGIN:
+ case TBLOCK_SUBRELEASE:
+ case TBLOCK_SUBCOMMIT:
+ case TBLOCK_SUBABORT_PENDING:
+ case TBLOCK_SUBRESTART:
+ AbortSubTransaction();
+ CleanupSubTransaction();
+ return false;
+
+ /*
+ * Same as above, except the Abort() was already done.
+ */
+ case TBLOCK_SUBABORT_END:
+ case TBLOCK_SUBABORT_RESTART:
+ CleanupSubTransaction();
+ return false;
}
+
+ /* Done, no more iterations required */
+ return true;
}
/*
--
2.39.3 (Apple Git-145)
On Wed, Apr 17, 2024 at 2:37 PM Alexander Korotkov <aekorotkov@gmail.com> wrote:
On Tue, Apr 16, 2024 at 7:42 PM Alexander Korotkov <aekorotkov@gmail.com> wrote:
On Tue, Apr 16, 2024 at 6:35 PM Andres Freund <andres@anarazel.de> wrote:
On 2024-04-16 15:45:42 +0300, Alexander Korotkov wrote:
On Tue, Apr 16, 2024 at 1:48 AM Andres Freund <andres@anarazel.de> wrote:
On 2024-03-06 14:17:23 +0200, Alexander Korotkov wrote:
0001 Turn tail recursion into iteration in CommitTransactionCommand()
I did minor revision of comments and code blocks order to improve the
readability.After sending
/messages/by-id/20240414223305.m3i5eju6zylabvln@awork3.anarazel.de
I looked some more at important areas where changes didn't have code
coverage. One thing I noticed was that the "non-internal" part of
AbortCurrentTransaction() is uncovered:
https://anarazel.de/postgres/cov/16-vs-HEAD-2024-04-14/src/backend/access/transam/xact.c.gcov.html#L3403Which made me try to understand fefd9a3fed2. I'm a bit confused about why
some parts are handled in CommitCurrentTransaction()/AbortCurrentTransaction()
and others are in the *Internal functions.I understand that fefd9a3fed2 needed to remove the recursion in
CommitTransactionCommand()/AbortCurrentTransaction(). But I don't understand
why that means having some code in in the non-internal and some in the
internal functions? Wouldn't it be easier to just have all the state handling
code in the Internal() function and just break after the
CleanupSubTransaction() calls?I'm not sure I correctly get what you mean. Do you think the attached
patch matches the direction you're pointing? The patch itself is not
final, it requires cleanup and comments revision, just to check the
direction.Something like that, yea. The split does seem less confusing that way to me,
but also not 100% certain.Thank you for your feedback. I'm going to go ahead and polish this patch.
I've invested more time into polishing this. I'm intended to push
this. Could you, please, take a look before?
Just after sending this I spotted a typo s/untill/until/. The updated
version is attached.
------
Regards,
Alexander Korotkov
Attachments:
v3-0001-Refactoring-for-CommitTransactionCommand-AbortCur.patchapplication/octet-stream; name=v3-0001-Refactoring-for-CommitTransactionCommand-AbortCur.patchDownload
From 846c833033f06766f725a63e8121579430de1dd2 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <akorotkov@postgresql.org>
Date: Wed, 17 Apr 2024 14:27:33 +0300
Subject: [PATCH v3] Refactoring for
CommitTransactionCommand()/AbortCurrentTransaction()
fefd9a3fed turned tail recursion of CommitTransactionCommand() and
AbortCurrentTransaction() into iteration. However, it splits the handling of
cases between different functions.
This commit puts the handling of all the cases into
AbortCurrentTransactionInternal() and CommitTransactionCommandInternal().
Now CommitTransactionCommand() and AbortCurrentTransaction() are just doing
the repeated calls of internal functions.
Reported-by: Andres Freund
Discussion: https://postgr.es/m/20240415224834.w6piwtefskoh32mv%40awork3.anarazel.de
Author: Andres Freund
---
src/backend/access/transam/xact.c | 159 +++++++++++++-----------------
1 file changed, 71 insertions(+), 88 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index df5a67e4c31..62a42f29d44 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -346,8 +346,8 @@ static void CommitTransaction(void);
static TransactionId RecordTransactionAbort(bool isSubXact);
static void StartTransaction(void);
-static void CommitTransactionCommandInternal(void);
-static void AbortCurrentTransactionInternal(void);
+static bool CommitTransactionCommandInternal(void);
+static bool AbortCurrentTransactionInternal(void);
static void StartSubTransaction(void);
static void CommitSubTransaction(void);
@@ -3092,50 +3092,25 @@ RestoreTransactionCharacteristics(const SavedTransactionCharacteristics *s)
void
CommitTransactionCommand(void)
{
- while (true)
- {
- switch (CurrentTransactionState->blockState)
- {
- /*
- * The current already-failed subtransaction is ending due to
- * a ROLLBACK or ROLLBACK TO command, so pop it and
- * recursively examine the parent (which could be in any of
- * several states).
- */
- case TBLOCK_SUBABORT_END:
- CleanupSubTransaction();
- continue;
-
- /*
- * As above, but it's not dead yet, so abort first.
- */
- case TBLOCK_SUBABORT_PENDING:
- AbortSubTransaction();
- CleanupSubTransaction();
- continue;
- default:
- break;
- }
- CommitTransactionCommandInternal();
- break;
- }
+ /*
+ * Repeatedly call CommitTransactionCommandInternal() until all the work
+ * is done.
+ */
+ while (!CommitTransactionCommandInternal());
}
/*
- * CommitTransactionCommandInternal - a function doing all the material work
- * regarding handling the commit transaction command except for loop over
- * subtransactions.
+ * CommitTransactionCommandInternal - a function doing an iteration of work
+ * regarding handling the commit transaction command. In the case of
+ * subtransactions more than one iterations could be required. Returns
+ * true when no more iterations required, false otherwise.
*/
-static void
+static bool
CommitTransactionCommandInternal(void)
{
TransactionState s = CurrentTransactionState;
SavedTransactionCharacteristics savetc;
- /* This states are handled in CommitTransactionCommand() */
- Assert(s->blockState != TBLOCK_SUBABORT_END &&
- s->blockState != TBLOCK_SUBABORT_PENDING);
-
/* Must save in case we need to restore below */
SaveTransactionCharacteristics(&savetc);
@@ -3319,6 +3294,25 @@ CommitTransactionCommandInternal(void)
BlockStateAsString(s->blockState));
break;
+ /*
+ * The current already-failed subtransaction is ending due to a
+ * ROLLBACK or ROLLBACK TO command, so pop it and recursively
+ * examine the parent (which could be in any of several states).
+ * As we need to examine the parent, return false to request the
+ * caller to do the next iteration.
+ */
+ case TBLOCK_SUBABORT_END:
+ CleanupSubTransaction();
+ return false;
+
+ /*
+ * As above, but it's not dead yet, so abort first.
+ */
+ case TBLOCK_SUBABORT_PENDING:
+ AbortSubTransaction();
+ CleanupSubTransaction();
+ return false;
+
/*
* The current subtransaction is the target of a ROLLBACK TO
* command. Abort and pop it, then start a new subtransaction
@@ -3376,10 +3370,10 @@ CommitTransactionCommandInternal(void)
s->blockState = TBLOCK_SUBINPROGRESS;
}
break;
- default:
- /* Keep compiler quiet */
- break;
}
+
+ /* Done, no more iterations required */
+ return true;
}
/*
@@ -3390,59 +3384,24 @@ CommitTransactionCommandInternal(void)
void
AbortCurrentTransaction(void)
{
- while (true)
- {
- switch (CurrentTransactionState->blockState)
- {
- /*
- * If we failed while trying to create a subtransaction, clean
- * up the broken subtransaction and abort the parent. The
- * same applies if we get a failure while ending a
- * subtransaction.
- */
- case TBLOCK_SUBBEGIN:
- case TBLOCK_SUBRELEASE:
- case TBLOCK_SUBCOMMIT:
- case TBLOCK_SUBABORT_PENDING:
- case TBLOCK_SUBRESTART:
- AbortSubTransaction();
- CleanupSubTransaction();
- continue;
-
- /*
- * Same as above, except the Abort() was already done.
- */
- case TBLOCK_SUBABORT_END:
- case TBLOCK_SUBABORT_RESTART:
- CleanupSubTransaction();
- continue;
- default:
- break;
- }
- AbortCurrentTransactionInternal();
- break;
- }
+ /*
+ * Repeatedly call AbortCurrentTransactionInternal() until all the work
+ * is done.
+ */
+ while (!AbortCurrentTransactionInternal());
}
/*
- * AbortCurrentTransactionInternal - a function doing all the material work
- * regarding handling the abort transaction command except for loop over
- * subtransactions.
+ * AbortCurrentTransactionInternal - a function doing an iteration of work
+ * regarding handling the current transaction abort. In the case of
+ * subtransactions more than one iterations could be required. Returns
+ * true when no more iterations required, false otherwise.
*/
-static void
+static bool
AbortCurrentTransactionInternal(void)
{
TransactionState s = CurrentTransactionState;
- /* This states are handled in AbortCurrentTransaction() */
- Assert(s->blockState != TBLOCK_SUBBEGIN &&
- s->blockState != TBLOCK_SUBRELEASE &&
- s->blockState != TBLOCK_SUBCOMMIT &&
- s->blockState != TBLOCK_SUBABORT_PENDING &&
- s->blockState != TBLOCK_SUBRESTART &&
- s->blockState != TBLOCK_SUBABORT_END &&
- s->blockState != TBLOCK_SUBABORT_RESTART);
-
switch (s->blockState)
{
case TBLOCK_DEFAULT:
@@ -3563,10 +3522,34 @@ AbortCurrentTransactionInternal(void)
AbortSubTransaction();
s->blockState = TBLOCK_SUBABORT;
break;
- default:
- /* Keep compiler quiet */
- break;
+
+ /*
+ * If we failed while trying to create a subtransaction, clean up
+ * the broken subtransaction and abort the parent. The same
+ * applies if we get a failure while ending a subtransaction. As
+ * we need to abort the parent, return false to request the caller
+ * to do the next iteration.
+ */
+ case TBLOCK_SUBBEGIN:
+ case TBLOCK_SUBRELEASE:
+ case TBLOCK_SUBCOMMIT:
+ case TBLOCK_SUBABORT_PENDING:
+ case TBLOCK_SUBRESTART:
+ AbortSubTransaction();
+ CleanupSubTransaction();
+ return false;
+
+ /*
+ * Same as above, except the Abort() was already done.
+ */
+ case TBLOCK_SUBABORT_END:
+ case TBLOCK_SUBABORT_RESTART:
+ CleanupSubTransaction();
+ return false;
}
+
+ /* Done, no more iterations required */
+ return true;
}
/*
--
2.39.3 (Apple Git-145)
Hi,
On 2024-04-17 14:39:14 +0300, Alexander Korotkov wrote:
On Wed, Apr 17, 2024 at 2:37 PM Alexander Korotkov <aekorotkov@gmail.com> wrote:
I've invested more time into polishing this. I'm intended to push
this. Could you, please, take a look before?Just after sending this I spotted a typo s/untill/until/. The updated
version is attached.
Nice, I see you moved the code back to "where it was", the diff to 16 got
smaller this way.
+ /* + * Repeatedly call CommitTransactionCommandInternal() until all the work + * is done. + */ + while (!CommitTransactionCommandInternal());
Personally I'd use
{
}
instead of just ; here. The above scans weirdly for me. But it's also not
important.
Greetings,
Andres Freund