2GB limit for temp_file_limit on 32bit platform

On 07/19/2012 01:04 PM, Pavel Stehule wrote:

I did a backport of temp_file_limit feature to 9.1, but when we tested
this patch, we found very restristrictive limit to 2GB.

2GB is nonsense, because this is session limit of temp files, and
these files should be longer than 2GB.

I haven't read the patch but... don't all 32bit platforms have a 2GB
limit (by default)?

Sincerely,

jD

--
Command Prompt, Inc. - http://www.commandprompt.com/
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, Postgres-XC
@cmdpromptinc - 509-416-6579

On Thu, Jul 19, 2012 at 4:29 PM, Joshua D. Drake <jd@commandprompt.com> wrote:

On 07/19/2012 01:04 PM, Pavel Stehule wrote:

I did a backport of temp_file_limit feature to 9.1, but when we tested
this patch, we found very restristrictive limit to 2GB.

2GB is nonsense, because this is session limit of temp files, and
these files should be longer than 2GB.

I haven't read the patch but... don't all 32bit platforms have a 2GB limit
(by default)?

I don't think so.

LFS got done in the mid-90s, which is long enough ago for people to
start forgetting about it. Are there any supported platforms that
didn't adopt LFS?

http://en.wikipedia.org/wiki/Large_file_support
--
When confronted by a difficult problem, solve it by reducing it to the
question, "How would the Lone Ranger handle this?"

On 07/19/2012 01:48 PM, Christopher Browne wrote:

On Thu, Jul 19, 2012 at 4:29 PM, Joshua D. Drake <jd@commandprompt.com> wrote:

On 07/19/2012 01:04 PM, Pavel Stehule wrote:

I did a backport of temp_file_limit feature to 9.1, but when we tested
this patch, we found very restristrictive limit to 2GB.

2GB is nonsense, because this is session limit of temp files, and
these files should be longer than 2GB.

I haven't read the patch but... don't all 32bit platforms have a 2GB limit
(by default)?

I don't think so.

LFS got done in the mid-90s, which is long enough ago for people to
start forgetting about it. Are there any supported platforms that
didn't adopt LFS?

http://en.wikipedia.org/wiki/Large_file_support

Note: "by default" :). I know they could support LFS but as I recall you
had to compile specifically for it (at least on linux and old versions
of pg).

So I was curious if it was that specific limitation or a limitation
within the Pg code itself.

Sincerely,

Joshua D. Drake

--
Command Prompt, Inc. - http://www.commandprompt.com/
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, Postgres-XC
@cmdpromptinc - 509-416-6579

On 20/07/12 09:08, Joshua D. Drake wrote:

On 07/19/2012 01:48 PM, Christopher Browne wrote:

On Thu, Jul 19, 2012 at 4:29 PM, Joshua D. Drake
<jd@commandprompt.com> wrote:

On 07/19/2012 01:04 PM, Pavel Stehule wrote:

I did a backport of temp_file_limit feature to 9.1, but when we tested
this patch, we found very restristrictive limit to 2GB.

2GB is nonsense, because this is session limit of temp files, and
these files should be longer than 2GB.

I haven't read the patch but... don't all 32bit platforms have a 2GB
limit
(by default)?

I don't think so.

LFS got done in the mid-90s, which is long enough ago for people to
start forgetting about it. Are there any supported platforms that
didn't adopt LFS?

http://en.wikipedia.org/wiki/Large_file_support

Note: "by default" :). I know they could support LFS but as I recall
you had to compile specifically for it (at least on linux and old
versions of pg).

So I was curious if it was that specific limitation or a limitation
within the Pg code itself.

It is to do with the datatype of the GUC used for the setting - I
haven't got the patch in from of me to look at, but recall that going
larger meant using a float type which meant you couldn't get nice units
displayed (MB, GB etc).

I'll take a proper look later.

Cheers

Mark

On 20/07/12 09:58, Mark Kirkwood wrote:

On 20/07/12 09:08, Joshua D. Drake wrote:

On 07/19/2012 01:48 PM, Christopher Browne wrote:

On Thu, Jul 19, 2012 at 4:29 PM, Joshua D. Drake
<jd@commandprompt.com> wrote:

On 07/19/2012 01:04 PM, Pavel Stehule wrote:

I did a backport of temp_file_limit feature to 9.1, but when we
tested
this patch, we found very restristrictive limit to 2GB.

2GB is nonsense, because this is session limit of temp files, and
these files should be longer than 2GB.

I haven't read the patch but... don't all 32bit platforms have a
2GB limit
(by default)?

I don't think so.

LFS got done in the mid-90s, which is long enough ago for people to
start forgetting about it. Are there any supported platforms that
didn't adopt LFS?

http://en.wikipedia.org/wiki/Large_file_support

Note: "by default" :). I know they could support LFS but as I recall
you had to compile specifically for it (at least on linux and old
versions of pg).

So I was curious if it was that specific limitation or a limitation
within the Pg code itself.

It is to do with the datatype of the GUC used for the setting - I
haven't got the patch in from of me to look at, but recall that going
larger meant using a float type which meant you couldn't get nice
units displayed (MB, GB etc).

I'll take a proper look later.

From src/backend/utils/misc/guc.c

{"temp_file_limit", PGC_SUSET, RESOURCES_DISK,
gettext_noop("Limits the total size of all temp files used
by each session."),
gettext_noop("-1 means no limit."),
GUC_UNIT_KB
},
&temp_file_limit,
-1, -1, INT_MAX,
NULL, NULL, NULL
},

Pavel Stehule <pavel.stehule@gmail.com> writes:

I did a backport of temp_file_limit feature to 9.1, but when we tested
this patch, we found very restristrictive limit to 2GB.

2GB is nonsense, because this is session limit of temp files, and
these files should be longer than 2GB.

This claim is nonsense. The variable's value is measured in KB, so the
effective limit is actually 2TB not 2GB.

regards, tom lane

On 20/07/12 12:02, Tom Lane wrote:

Pavel Stehule <pavel.stehule@gmail.com> writes:

I did a backport of temp_file_limit feature to 9.1, but when we tested
this patch, we found very restristrictive limit to 2GB.
2GB is nonsense, because this is session limit of temp files, and
these files should be longer than 2GB.

This claim is nonsense. The variable's value is measured in KB, so the
effective limit is actually 2TB not 2GB.

Did you guys perchance pick up one of the earlier patches that had
MAX_KILOBYTES instead of INT_MAX as the limit? i.e:

{"temp_file_limit", PGC_SUSET, RESOURCES_DISK,
gettext_noop("Limits the total size of all temp files used
by each session."),
gettext_noop("-1 means no limit."),
GUC_UNIT_KB
},
&temp_file_limit,
-1, -1, **MAX_KILOBYTES**, <====
NULL, NULL, NULL
},

2012/7/20 Tom Lane <tgl@sss.pgh.pa.us>:

Pavel Stehule <pavel.stehule@gmail.com> writes:

I did a backport of temp_file_limit feature to 9.1, but when we tested
this patch, we found very restristrictive limit to 2GB.

2GB is nonsense, because this is session limit of temp files, and
these files should be longer than 2GB.

This claim is nonsense. The variable's value is measured in KB, so the
effective limit is actually 2TB not 2GB.

you have true - it works on 9.2. The problem will be somewhere in
backport on 9.1

sorry for false alarm

Regards

Pavel Stěhule

2012/7/20 Pavel Stehule <pavel.stehule@gmail.com>:

2012/7/20 Tom Lane <tgl@sss.pgh.pa.us>:

Pavel Stehule <pavel.stehule@gmail.com> writes:

I did a backport of temp_file_limit feature to 9.1, but when we tested
this patch, we found very restristrictive limit to 2GB.

2GB is nonsense, because this is session limit of temp files, and
these files should be longer than 2GB.

This claim is nonsense. The variable's value is measured in KB, so the
effective limit is actually 2TB not 2GB.

you have true - it works on 9.2. The problem will be somewhere in
backport on 9.1

it works well with 9.1.4, but not with 9.1.3