Return INT_MAX or INT_MIN from strtoint() on overflow

Started by Chao Li25 days ago10 messageshackers
Jump to latest
#1Chao Li
li.evan.chao@gmail.com

Hi,

I recently reviewed a patch [1]/messages/by-id/852FF0DD-8C49-4A7A-8CE1-C7421F1E97B7@gmail.com that switches strtoint() to strtoi64() to handle an int32 overflow problem. I initially thought we could keep using strtoint(), but then realized that its current return value cannot reliably indicate the direction of overflow, which the caller needs to clamp the value correctly.

Looking at the current code:
```
/*
* strtoint --- just like strtol, but returns int not long
*/
int
strtoint(const char *pg_restrict str, char **pg_restrict endptr, int base)
{
long val;

val = strtol(str, endptr, base);
if (val != (int) val)
errno = ERANGE;
return (int) val;
}
```

When overflow occurs, it returns “(int) val”, and when the value is outside the range of int, the cast produces an implementation-defined result, which cannot reliably indicate the direction of overflow. See the C11 draft [2]https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf#page=69, section 6.3.1.3, paragraph 3.

The header comment says “just like strtol”, but strtol() returns LONG_MAX or LONG_MIN when overflow happens. See [3]https://man7.org/linux/man-pages/man3/strtol.3p.html:
```
If the correct value is outside the range of representable values,
{LONG_MIN}, {LONG_MAX}, {LLONG_MIN}, or {LLONG_MAX} shall be
returned (according to the sign of the value), and errno set to
[ERANGE].
```

So, the current implementation doesn’t fully match the intention described in the header comment.

Attached is a simple fix to change strtoint() to return INT_MAX or INT_MIN upon overflow, which makes its behavior consistent with strtol(). I have run “make check-world”, and all tests passed.

If this patch is accepted, I will go back to patch [1]/messages/by-id/852FF0DD-8C49-4A7A-8CE1-C7421F1E97B7@gmail.com and comment my original thoughts on that patch.

[1]: /messages/by-id/852FF0DD-8C49-4A7A-8CE1-C7421F1E97B7@gmail.com
[2]: https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf#page=69
[3]: https://man7.org/linux/man-pages/man3/strtol.3p.html

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

Attachments:

t254419_1
v1-0001-Return-INT_MAX-or-INT_MIN-from-strtoint-on-overfl.patchapplication/octet-stream; name=v1-0001-Return-INT_MAX-or-INT_MIN-from-strtoint-on-overfl.patch; x-unix-mode=0644Download+13-2
#2surya poondla
suryapoondla4@gmail.com
In reply to: Chao Li (#1)
Re: Return INT_MAX or INT_MIN from strtoint() on overflow

Hi Chao,

I'm not sure if the change is needed.

strtoint() job has always been to detect overflow through errno. All the
core callers in that style (datetime.c, read.c, tsearchcmds.c, jsonfuncs.c,
psql, option_utils.c) check errno == ERANGE and never
look at the return value after an overflow. For them v1 changes nothing.
On Windows, where long is 32 bits, strtol() already returns
INT_MAX/INT_MIN, so nothing changes there either.

The only callers that see a difference are the ones that ignore errno:
pgcrypto's px_crypt_shacrypt() and a few ecpg pgtypeslib/dt_common.c paths.
For those, the real bug is that they don't check errno.

Regards,
Surya Poondla

#3Takeshi Ideriha
iderihatakeshi@gmail.com
In reply to: Chao Li (#1)
Re: Return INT_MAX or INT_MIN from strtoint() on overflow

Hi Chao

On 2026/10/28 10:28, Chao Li wrote:

Hi,

I recently reviewed a patch [1] that switches strtoint() to strtoi64() to handle an int32 overflow problem. I initially thought we could keep using strtoint(), but then realized that its current return value cannot reliably indicate the direction of overflow, which the caller needs to clamp the value correctly.

Thank you for the suggestion.
I suggested using strtoi64 instead of strtoint in the original thread
and this is exactly why I suggested.

Looking at the current code:
```
/*
* strtoint --- just like strtol, but returns int not long
*/
int
strtoint(const char *pg_restrict str, char **pg_restrict endptr, int base)
{
long val;

val = strtol(str, endptr, base);
if (val != (int) val)
errno = ERANGE;
return (int) val;
}
```

When overflow occurs, it returns “(int) val”, and when the value is outside the range of int, the cast produces an implementation-defined result, which cannot reliably indicate the direction of overflow. See the C11 draft [2], section 6.3.1.3, paragraph 3.

The header comment says “just like strtol”, but strtol() returns LONG_MAX or LONG_MIN when overflow happens. See [3]:
```
If the correct value is outside the range of representable values,
{LONG_MIN}, {LONG_MAX}, {LLONG_MIN}, or {LLONG_MAX} shall be
returned (according to the sign of the value), and errno set to
[ERANGE].
```

So, the current implementation doesn’t fully match the intention described in the header comment.

Attached is a simple fix to change strtoint() to return INT_MAX or INT_MIN upon overflow, which makes its behavior consistent with strtol(). I have run “make check-world”, and all tests passed.

I think it's still caller's responsibility to check if overflow occurs
but providing overflow direction as an additional information would be
no harm and sometimes helpful.

If this patch is accepted, I will go back to patch [1] and comment my original thoughts on that patch.

+1

[1] /messages/by-id/852FF0DD-8C49-4A7A-8CE1-C7421F1E97B7@gmail.com
[2] https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf#page=69
[3] https://man7.org/linux/man-pages/man3/strtol.3p.html

Regards,
Takeshi Ideriha

#4Chao Li
li.evan.chao@gmail.com
In reply to: surya poondla (#2)
Re: Return INT_MAX or INT_MIN from strtoint() on overflow

On Oct 30, 2026, at 04:44, surya poondla <suryapoondla4@gmail.com> wrote:

Hi Chao,

I'm not sure if the change is needed.

Thanks a lot for the review.

strtoint() job has always been to detect overflow through errno. All the core callers in that style (datetime.c, read.c, tsearchcmds.c, jsonfuncs.c, psql, option_utils.c) check errno == ERANGE and never
look at the return value after an overflow. For them v1 changes nothing.

That’s true, but it doesn’t mean the current return-value behavior matches the header comment. As I mentioned in the patch email, the comment explicitly says “strtoint --- just like strtol, but returns int not long”. However, unlike strtol(), it doesn’t reliably return the corresponding bound on overflow. When val is outside the range of int, “(int) val” produces an implementation-defined result.

On Windows, where long is 32 bits, strtol() already returns INT_MAX/INT_MIN, so nothing changes there either.

Agreed, but the problem here is that strtoint() doesn’t reliably return INT_MAX or INT_MIN on all platforms.

The only callers that see a difference are the ones that ignore errno: pgcrypto's px_crypt_shacrypt() and a few ecpg pgtypeslib/dt_common.c paths. For those, the real bug is that they don't check errno.

Not checking errno is not necessarily a bug, it depends on how the caller uses the result.

In the px_crypt_shacrypt() case, an overflowing value needs to be clamped to PX_SHACRYPT_ROUNDS_MAX or PX_SHACRYPT_ROUNDS_MIN depending on the direction of overflow. Checking ERANGE alone doesn’t tell us which bound to use.

If strtoint() reliably returned INT_MAX or INT_MIN on overflow, the existing range checks would clamp correctly, since both values are outside the supported rounds range. For that purpose, px_crypt_shacrypt() wouldn’t need to check ERANGE, and patch [1]/messages/by-id/852FF0DD-8C49-4A7A-8CE1-C7421F1E97B7@gmail.com wouldn’t need to switch from strtoint() to strtoi64().

So, I think the underlying problem is that strtoint()’s overflow behavior doesn't fully match its “just like strtol” description. Patch [1]/messages/by-id/852FF0DD-8C49-4A7A-8CE1-C7421F1E97B7@gmail.com works around this by switching to strtoi64(). If strtoint() reliably returned INT_MAX or INT_MIN on overflow, the existing call and range checks would already handle clamping correctly, and that switch would be unnecessary. From this perspective, I consider this is a bug of strtoint().

* Note: I’m not saying patch [1]/messages/by-id/852FF0DD-8C49-4A7A-8CE1-C7421F1E97B7@gmail.com is no longer needed, because it also changes the NOTICE messages to report the original input, which addresses a separate issue.

[1]: /messages/by-id/852FF0DD-8C49-4A7A-8CE1-C7421F1E97B7@gmail.com

--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

#5Tom Lane
tgl@sss.pgh.pa.us
In reply to: Chao Li (#4)
Re: Return INT_MAX or INT_MIN from strtoint() on overflow

Chao Li <li.evan.chao@gmail.com> writes:

On Oct 30, 2026, at 04:44, surya poondla <suryapoondla4@gmail.com> wrote:
strtoint() job has always been to detect overflow through errno. All the core callers in that style (datetime.c, read.c, tsearchcmds.c, jsonfuncs.c, psql, option_utils.c) check errno == ERANGE and never
look at the return value after an overflow. For them v1 changes nothing.

That’s true, but it doesn’t mean the current return-value behavior matches the header comment.

Yeah. POSIX is 100% clear about what strtol is required to do:

If the correct value is outside the range of representable values,
{LONG_MIN}, {LONG_MAX}, {LLONG_MIN}, or {LLONG_MAX} shall be
returned (according to the sign of the value), and errno set to
[ERANGE].

So a bitwise-truncated value is not OK; we must preserve the original
sign in case the caller wants to know that, and we must ensure that
callers' range checks deliver sensible results. Even if none of our
in-core callers cared at the moment, it makes sense to me that we
should match this behavior. All the more reason if there are places
that could immediately make use of the tighter API spec.

regards, tom lane

#6Prashant Chinnam
crprashant@hotmail.com
In reply to: Tom Lane (#5)
Re: Return INT_MAX or INT_MIN from strtoint() on overflow

Hi Chao,

+1 -- I ran into exactly this while writing patch [1].

My first version used strtoint() with an errno check, but
the returned value could not be trusted to show the
direction of the overflow, so the clamp direction had to
be taken from the input string instead. A saturating
return value, as strtol() provides, is what that code
actually wanted.

While looking at this I audited the in-tree strtoint()
callers: 33 call sites, of which 21 check errno and
discard the value on overflow, so they are unaffected.
The other 12 are all in ecpg's dt_common.c, which does
not check errno; for those, saturation seems strictly
safer than a truncated value, and I found no caller that
relies on the truncated result. Two paths there might be
worth a look either way: the DecodeTimezone colon form
has no range check on the parsed hour and minute, and
the DecodeTime hour bound is similar. Neither is made
worse by this change.

On patch [1]: I intend to keep strtoi64() there even if
this lands, because the pgcrypto fix is meant to be
backpatched to REL_18_STABLE, where strtoint() keeps its
current behavior, while strtoi64() behaves the same on
both branches. Happy to follow a different direction if
committers prefer otherwise.

The caller audit above was done with AI assistance.

Regards,
Prashant Chinnam

#7Tom Lane
tgl@sss.pgh.pa.us
In reply to: Prashant Chinnam (#6)
Re: Return INT_MAX or INT_MIN from strtoint() on overflow

Prashant Chinnam <crprashant@hotmail.com> writes:

On patch [1]: I intend to keep strtoi64() there even if
this lands, because the pgcrypto fix is meant to be
backpatched to REL_18_STABLE, where strtoint() keeps its
current behavior, while strtoi64() behaves the same on
both branches. Happy to follow a different direction if
committers prefer otherwise.

My thought here is to back-patch the strtoint() change.
I cannot envision any case that that would make worse.
And failing to do so leaves a land-mine for potential
future back-patches of code using strtoint().

regards, tom lane

#8Prashant Chinnam
crprashant@hotmail.com
In reply to: Tom Lane (#7)
Re: Return INT_MAX or INT_MIN from strtoint() on overflow

Hi Tom,

My thought here is to back-patch the strtoint() change.
I cannot envision any case that that would make worse.
And failing to do so leaves a land-mine for potential
future back-patches of code using strtoint().

Thanks -- back-patching the strtoint() change removes my
only reason for preferring strtoi64() in patch [1]. Once
it has landed, I will switch that patch back to
strtoint(); that was the shape of my original v1, minus
the errno workaround that the saturating return value
makes unnecessary.

Regards,
Prashant Chinnam

#9Tom Lane
tgl@sss.pgh.pa.us
In reply to: Prashant Chinnam (#8)
Re: Return INT_MAX or INT_MIN from strtoint() on overflow

Prashant Chinnam <crprashant@hotmail.com> writes:

Hi Tom,

My thought here is to back-patch the strtoint() change.
I cannot envision any case that that would make worse.
And failing to do so leaves a land-mine for potential
future back-patches of code using strtoint().

Thanks -- back-patching the strtoint() change removes my
only reason for preferring strtoi64() in patch [1]. Once
it has landed, I will switch that patch back to
strtoint(); that was the shape of my original v1, minus
the errno workaround that the saturating return value
makes unnecessary.

Pushed. I made the code look like this:

val = strtol(str, endptr, base);
if (val != (int) val)
+ {
+ val = (val < 0) ? INT_MIN : INT_MAX;
errno = ERANGE;
+ }
return (int) val;

I was concerned that Chao's original formulation could provoke
compiler warnings on machines with 32-bit long: it would not
be surprising to get "condition is always false" warnings
from "val > INT_MAX" etc. (In principle the existing test
could provoke such warnings too, but it doesn't. I very
vaguely recall that we settled on the cast-and-compare
method specifically to avoid warnings, way back when.)

regards, tom lane

#10Chao Li
li.evan.chao@gmail.com
In reply to: Tom Lane (#9)
Re: Return INT_MAX or INT_MIN from strtoint() on overflow

On Oct 30, 2026, at 23:07, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Prashant Chinnam <crprashant@hotmail.com> writes:

Hi Tom,

My thought here is to back-patch the strtoint() change.
I cannot envision any case that that would make worse.
And failing to do so leaves a land-mine for potential
future back-patches of code using strtoint().

Thanks -- back-patching the strtoint() change removes my
only reason for preferring strtoi64() in patch [1]. Once
it has landed, I will switch that patch back to
strtoint(); that was the shape of my original v1, minus
the errno workaround that the saturating return value
makes unnecessary.

Pushed. I made the code look like this:

val = strtol(str, endptr, base);
if (val != (int) val)
+ {
+ val = (val < 0) ? INT_MIN : INT_MAX;
errno = ERANGE;
+ }
return (int) val;

I was concerned that Chao's original formulation could provoke
compiler warnings on machines with 32-bit long: it would not
be surprising to get "condition is always false" warnings
from "val > INT_MAX" etc. (In principle the existing test
could provoke such warnings too, but it doesn't. I very
vaguely recall that we settled on the cast-and-compare
method specifically to avoid warnings, way back when.)

Happy to learn that.

Thanks for pushing.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/