pgsql: Permit super-MaxAllocSize allocations with MemoryContextAllocHug

Hi,

On 2013-06-27 19:05:22 +0000, Noah Misch wrote:

Permit super-MaxAllocSize allocations with MemoryContextAllocHuge().

The MaxAllocSize guard is convenient for most callers, because it
reduces the need for careful attention to overflow, data type selection,
and the SET_VARSIZE() limit. A handful of callers are happy to navigate
those hazards in exchange for the ability to allocate a larger chunk.
Introduce MemoryContextAllocHuge() and repalloc_huge(). Use this in
tuplesort.c and tuplestore.c, enabling internal sorts of up to INT_MAX
tuples, a factor-of-48 increase. In particular, B-tree index builds can
now benefit from much-larger maintenance_work_mem settings.

This commit causes a bunch of warnings like:

src/backend/utils/sort/tuplesort.c: In function ‘tuplesort_begin_common’:
src/backend/utils/sort/tuplesort.c:399:33: warning: comparison of unsigned expression < 0 is always false
[-Wtype-limits]
#define LACKMEM(state) ((state)->availMem < 0)

to be thrown during compilation. And I think it is spot on. Unless you
overhaul a good bit of the respective logic making availMem unsigned
isn't going to fly.
As is, that code will probably be optimized away which isn't good...

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

On Tue, Jul 02, 2013 at 12:08:42PM +0200, Andres Freund wrote:

On 2013-06-27 19:05:22 +0000, Noah Misch wrote:

Permit super-MaxAllocSize allocations with MemoryContextAllocHuge().

The MaxAllocSize guard is convenient for most callers, because it
reduces the need for careful attention to overflow, data type selection,
and the SET_VARSIZE() limit. A handful of callers are happy to navigate
those hazards in exchange for the ability to allocate a larger chunk.
Introduce MemoryContextAllocHuge() and repalloc_huge(). Use this in
tuplesort.c and tuplestore.c, enabling internal sorts of up to INT_MAX
tuples, a factor-of-48 increase. In particular, B-tree index builds can
now benefit from much-larger maintenance_work_mem settings.

This commit causes a bunch of warnings like:

src/backend/utils/sort/tuplesort.c: In function ???tuplesort_begin_common???:
src/backend/utils/sort/tuplesort.c:399:33: warning: comparison of unsigned expression < 0 is always false
[-Wtype-limits]
#define LACKMEM(state) ((state)->availMem < 0)

to be thrown during compilation. And I think it is spot on. Unless you
overhaul a good bit of the respective logic making availMem unsigned
isn't going to fly.

True. Will look into it; thanks.

--
Noah Misch
EnterpriseDB http://www.enterprisedb.com

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

On Tue, Jul 02, 2013 at 10:33:38AM -0400, Noah Misch wrote:

On Tue, Jul 02, 2013 at 12:08:42PM +0200, Andres Freund wrote:

On 2013-06-27 19:05:22 +0000, Noah Misch wrote:

Permit super-MaxAllocSize allocations with MemoryContextAllocHuge().

The MaxAllocSize guard is convenient for most callers, because it
reduces the need for careful attention to overflow, data type selection,
and the SET_VARSIZE() limit. A handful of callers are happy to navigate
those hazards in exchange for the ability to allocate a larger chunk.
Introduce MemoryContextAllocHuge() and repalloc_huge(). Use this in
tuplesort.c and tuplestore.c, enabling internal sorts of up to INT_MAX
tuples, a factor-of-48 increase. In particular, B-tree index builds can
now benefit from much-larger maintenance_work_mem settings.

This commit causes a bunch of warnings like:

src/backend/utils/sort/tuplesort.c: In function ???tuplesort_begin_common???:
src/backend/utils/sort/tuplesort.c:399:33: warning: comparison of unsigned expression < 0 is always false
[-Wtype-limits]
#define LACKMEM(state) ((state)->availMem < 0)

to be thrown during compilation. And I think it is spot on. Unless you
overhaul a good bit of the respective logic making availMem unsigned
isn't going to fly.

True. Will look into it; thanks.

In retrospect, I did not need the "long" -> "Size" changes at all. I thought
they were necessary to prevent new overflow possibilities on 64-bit Windows,
but we already limited work_mem to 2 GiB on that platform to account for its
32-bit "long". That's a limit to be removed rather than perpetuated.
Therefore, rather than change back to "long", I've changed to "int64". That's
simple to reason about, optimal on 64-bit, and no big loss on 32-bit.

Thanks,
nm

--
Noah Misch
EnterpriseDB http://www.enterprisedb.com