Questions about guc units
Hi hackers,
I have some questions about guc units, new feature in 8.2.
#shared_buffers = 32000kB # min 128kB or max_connections*16kB
#temp_buffers = 8000kB # min 800kB
#effective_cache_size = 8000kB
Are there any reasons to continue to use 1000-unit numbers? Megabyte-unit
(32MB and 8MB) seems to be more friendly for users. It increases some amount
of values (4000 vs. 4096), but there is little in it.
#max_fsm_pages = 1600000 # min max_fsm_relations*16, 6 bytes each
#wal_buffers = 8 # min 4, 8kB each
They don't have units now, but should they have GUC_UNIT_BLOCKS and
GUC_UNIT_XLOG_BLCKSZ unit? I feel inconsistency in them.
Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center
Am Montag, 25. September 2006 04:04 schrieb ITAGAKI Takahiro:
#shared_buffers = 32000kB # min 128kB or max_connections*16kB
#temp_buffers = 8000kB # min 800kB
#effective_cache_size = 8000kBAre there any reasons to continue to use 1000-unit numbers? Megabyte-unit
(32MB and 8MB) seems to be more friendly for users. It increases some
amount of values (4000 vs. 4096), but there is little in it.
The reason with the shared_buffers is that the detection code in initdb has
400kB as minimum value, and it would be pretty complicated to code the
detection code to handle both kB and MB units. If someone wants to try it,
though, please go ahead.
We could probably change the others.
#max_fsm_pages = 1600000 # min max_fsm_relations*16, 6 bytes each
#wal_buffers = 8 # min 4, 8kB eachThey don't have units now, but should they have GUC_UNIT_BLOCKS and
GUC_UNIT_XLOG_BLCKSZ unit? I feel inconsistency in them.
max_fsm_pages doesn't have a discernible unit, but wal_buffers probably
should.
--
Peter Eisentraut
http://developer.postgresql.org/~petere/
On Mon, Sep 25, 2006 at 10:03:50AM +0200, Peter Eisentraut wrote:
Am Montag, 25. September 2006 04:04 schrieb ITAGAKI Takahiro:
#shared_buffers = 32000kB # min 128kB or max_connections*16kB
#temp_buffers = 8000kB # min 800kB
#effective_cache_size = 8000kBAre there any reasons to continue to use 1000-unit numbers? Megabyte-unit
(32MB and 8MB) seems to be more friendly for users. It increases some
amount of values (4000 vs. 4096), but there is little in it.The reason with the shared_buffers is that the detection code in initdb has
400kB as minimum value, and it would be pretty complicated to code the
detection code to handle both kB and MB units. If someone wants to try it,
though, please go ahead.
What about 0.4MB? Granted, it's uglier than 400kB, but anyone running on
a machine that can't handle at least 1MB is already in the "pretty ugly"
realm...
--
Jim Nasby jim@nasby.net
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
Jim C. Nasby wrote:
The reason with the shared_buffers is that the detection code in
initdb has 400kB as minimum value, and it would be pretty
complicated to code the detection code to handle both kB and MB
units. If someone wants to try it, though, please go ahead.What about 0.4MB?
That isn't valid code, so I don't know how that helps.
--
Peter Eisentraut
http://developer.postgresql.org/~petere/
On Sep 25, 2006, at 1:03 AM, Peter Eisentraut wrote:
Am Montag, 25. September 2006 04:04 schrieb ITAGAKI Takahiro:
#shared_buffers = 32000kB # min 128kB or max_connections*16kB
#temp_buffers = 8000kB # min 800kB
#effective_cache_size = 8000kBAre there any reasons to continue to use 1000-unit numbers?
Megabyte-unit
(32MB and 8MB) seems to be more friendly for users. It increases some
amount of values (4000 vs. 4096), but there is little in it.The reason with the shared_buffers is that the detection code in
initdb has
400kB as minimum value, and it would be pretty complicated to code the
detection code to handle both kB and MB units. If someone wants to
try it,
though, please go ahead.
Seems like the unit used for shared_buffers (and others) should be
megabytes then with a minimum of 1 (or more). Is less than 1MB
granularity really useful here? On modern hardware 1MB of RAM is in
the noise.
-Casey
Peter Eisentraut <peter_e@gmx.net> wrote:
#max_fsm_pages = 1600000 # min max_fsm_relations*16, 6 bytes each
max_fsm_pages doesn't have a discernible unit
Yes, max_fsm_*pages* doesn't have a unit, but can we treat the value as
"the amount of trackable database size by fsm" or "estimated database size" ?
(the latter is a bit too radical interpretation, though.)
So I think it is not so odd to give a unit to max_fsm_pages.
Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center
The attached patch changes units of the some default values in postgresql.conf.
- shared_buffers = 32000kB => 32MB
- temp_buffers = 8000kB => 8MB
- wal_buffers = 8 => 64kB
The code of initdb was a bit modified to write MB-unit values.
Values greater than 8000kB are rounded out to MB.
GUC_UNIT_XBLOCKS is added for wal_buffers. It is like GUC_UNIT_BLOCKS,
but uses XLOG_BLCKSZ instead of BLCKSZ.
Also, I cleaned up the test of GUC_UNIT_* flags in preparation to
add more unit flags in less bits.
Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center
Attachments:
guc_units.patchapplication/octet-stream; name=guc_units.patchDownload
diff -cpr pgsql-orig/src/backend/utils/misc/guc.c pgsql/src/backend/utils/misc/guc.c
*** pgsql-orig/src/backend/utils/misc/guc.c Tue Sep 26 10:25:37 2006
--- pgsql/src/backend/utils/misc/guc.c Tue Sep 26 11:31:47 2006
*************** static struct config_int ConfigureNamesI
*** 1149,1155 ****
GUC_UNIT_BLOCKS
},
&NBuffers,
! 1000, 16, INT_MAX / 2, NULL, NULL
},
{
--- 1149,1155 ----
GUC_UNIT_BLOCKS
},
&NBuffers,
! 1024, 16, INT_MAX / 2, NULL, NULL
},
{
*************** static struct config_int ConfigureNamesI
*** 1159,1165 ****
GUC_UNIT_BLOCKS
},
&num_temp_buffers,
! 1000, 100, INT_MAX / 2, NULL, show_num_temp_buffers
},
{
--- 1159,1165 ----
GUC_UNIT_BLOCKS
},
&num_temp_buffers,
! 1024, 100, INT_MAX / 2, NULL, show_num_temp_buffers
},
{
*************** static struct config_int ConfigureNamesI
*** 1414,1420 ****
{
{"wal_buffers", PGC_POSTMASTER, WAL_SETTINGS,
gettext_noop("Sets the number of disk-page buffers in shared memory for WAL."),
! NULL
},
&XLOGbuffers,
8, 4, INT_MAX, NULL, NULL
--- 1414,1421 ----
{
{"wal_buffers", PGC_POSTMASTER, WAL_SETTINGS,
gettext_noop("Sets the number of disk-page buffers in shared memory for WAL."),
! NULL,
! GUC_UNIT_XBLOCKS
},
&XLOGbuffers,
8, 4, INT_MAX, NULL, NULL
*************** parse_int(const char *value, int *result
*** 3606,3613 ****
endptr += 2;
}
! if (used && (flags & GUC_UNIT_BLOCKS))
! val /= (BLCKSZ/1024);
}
if ((flags & GUC_UNIT_TIME) && endptr != value)
--- 3607,3624 ----
endptr += 2;
}
! if (used)
! {
! switch (flags & GUC_UNIT_MEMORY)
! {
! case GUC_UNIT_BLOCKS:
! val /= (BLCKSZ/1024);
! break;
! case GUC_UNIT_XBLOCKS:
! val /= (XLOG_BLCKSZ/1024);
! break;
! }
! }
}
if ((flags & GUC_UNIT_TIME) && endptr != value)
*************** parse_int(const char *value, int *result
*** 3647,3656 ****
endptr += 1;
}
! if (used && (flags & GUC_UNIT_S))
! val /= MS_PER_S;
! else if (used && (flags & GUC_UNIT_MIN))
! val /= MS_PER_MIN;
}
if (endptr == value || *endptr != '\0' || errno == ERANGE
--- 3658,3675 ----
endptr += 1;
}
! if (used)
! {
! switch (flags & GUC_UNIT_TIME)
! {
! case GUC_UNIT_S:
! val /= MS_PER_S;
! break;
! case GUC_UNIT_MIN:
! val /= MS_PER_MIN;
! break;
! }
! }
}
if (endptr == value || *endptr != '\0' || errno == ERANGE
*************** GetConfigOptionByNum(int varnum, const c
*** 4961,4983 ****
/* unit */
if (conf->vartype == PGC_INT)
{
! if (conf->flags & GUC_UNIT_KB)
! values[2] = "kB";
! else if (conf->flags & GUC_UNIT_BLOCKS)
! {
! static char buf[8];
! snprintf(buf, sizeof(buf), "%dkB", BLCKSZ/1024);
! values[2] = buf;
}
- else if (conf->flags & GUC_UNIT_MS)
- values[2] = "ms";
- else if (conf->flags & GUC_UNIT_S)
- values[2] = "s";
- else if (conf->flags & GUC_UNIT_MIN)
- values[2] = "min";
- else
- values[2] = "";
}
else
values[2] = NULL;
--- 4980,5013 ----
/* unit */
if (conf->vartype == PGC_INT)
{
! static char buf[8];
! switch (conf->flags & (GUC_UNIT_MEMORY | GUC_UNIT_TIME))
! {
! case GUC_UNIT_KB:
! values[2] = "kB";
! break;
! case GUC_UNIT_BLOCKS:
! snprintf(buf, sizeof(buf), "%dkB", BLCKSZ/1024);
! values[2] = buf;
! break;
! case GUC_UNIT_XBLOCKS:
! snprintf(buf, sizeof(buf), "%dkB", XLOG_BLCKSZ/1024);
! values[2] = buf;
! break;
! case GUC_UNIT_MS:
! values[2] = "ms";
! break;
! case GUC_UNIT_S:
! values[2] = "s";
! break;
! case GUC_UNIT_MIN:
! values[2] = "min";
! break;
! default:
! values[2] = "";
! break;
}
}
else
values[2] = NULL;
*************** _ShowOption(struct config_generic * reco
*** 5246,5253 ****
if (use_units && result > 0 && (record->flags & GUC_UNIT_MEMORY))
{
! if (record->flags & GUC_UNIT_BLOCKS)
! result *= BLCKSZ/1024;
if (result % KB_PER_GB == 0)
{
--- 5276,5290 ----
if (use_units && result > 0 && (record->flags & GUC_UNIT_MEMORY))
{
! switch (record->flags & GUC_UNIT_MEMORY)
! {
! case GUC_UNIT_BLOCKS:
! result *= BLCKSZ/1024;
! break;
! case GUC_UNIT_XBLOCKS:
! result *= XLOG_BLCKSZ/1024;
! break;
! }
if (result % KB_PER_GB == 0)
{
*************** _ShowOption(struct config_generic * reco
*** 5266,5275 ****
}
else if (use_units && result > 0 && (record->flags & GUC_UNIT_TIME))
{
! if (record->flags & GUC_UNIT_S)
! result = result * MS_PER_S;
! else if (record->flags & GUC_UNIT_MIN)
! result = result * MS_PER_MIN;
if (result % MS_PER_D == 0)
{
--- 5303,5317 ----
}
else if (use_units && result > 0 && (record->flags & GUC_UNIT_TIME))
{
! switch (record->flags & GUC_UNIT_TIME)
! {
! case GUC_UNIT_S:
! result *= MS_PER_S;
! break;
! case GUC_UNIT_MIN:
! result *= MS_PER_MIN;
! break;
! }
if (result % MS_PER_D == 0)
{
diff -cpr pgsql-orig/src/backend/utils/misc/postgresql.conf.sample pgsql/src/backend/utils/misc/postgresql.conf.sample
*** pgsql-orig/src/backend/utils/misc/postgresql.conf.sample Tue Sep 26 10:25:37 2006
--- pgsql/src/backend/utils/misc/postgresql.conf.sample Tue Sep 26 11:25:31 2006
***************
*** 98,106 ****
# - Memory -
! #shared_buffers = 32000kB # min 128kB or max_connections*16kB
# (change requires restart)
! #temp_buffers = 8000kB # min 800kB
#max_prepared_transactions = 5 # can be 0 or more
# (change requires restart)
# Note: increasing max_prepared_transactions costs ~600 bytes of shared memory
--- 98,106 ----
# - Memory -
! #shared_buffers = 32MB # min 128kB or max_connections*16kB
# (change requires restart)
! #temp_buffers = 8MB # min 800kB
#max_prepared_transactions = 5 # can be 0 or more
# (change requires restart)
# Note: increasing max_prepared_transactions costs ~600 bytes of shared memory
***************
*** 111,117 ****
# - Free Space Map -
! #max_fsm_pages = 1600000 # min max_fsm_relations*16, 6 bytes each
# (change requires restart)
#max_fsm_relations = 1000 # min 100, ~70 bytes each
# (change requires restart)
--- 111,117 ----
# - Free Space Map -
! #max_fsm_pages = 1638400 # min max_fsm_relations*16, 6 bytes each
# (change requires restart)
#max_fsm_relations = 1000 # min 100, ~70 bytes each
# (change requires restart)
***************
*** 154,160 ****
# fsync_writethrough
# open_sync
#full_page_writes = on # recover from partial page writes
! #wal_buffers = 8 # min 4, 8kB each
# (change requires restart)
#commit_delay = 0 # range 0-100000, in microseconds
#commit_siblings = 5 # range 1-1000
--- 154,160 ----
# fsync_writethrough
# open_sync
#full_page_writes = on # recover from partial page writes
! #wal_buffers = 64kB # min 4, 8kB each
# (change requires restart)
#commit_delay = 0 # range 0-100000, in microseconds
#commit_siblings = 5 # range 1-1000
diff -cpr pgsql-orig/src/bin/initdb/initdb.c pgsql/src/bin/initdb/initdb.c
*** pgsql-orig/src/bin/initdb/initdb.c Tue Sep 26 10:25:37 2006
--- pgsql/src/bin/initdb/initdb.c Tue Sep 26 12:14:27 2006
*************** test_config_settings(void)
*** 1113,1122 ****
static const int trial_conns[] = {
100, 50, 40, 30, 20, 10
};
static const int trial_bufs[] = {
! 32000, 28000, 24000, 20000, 16000, 12000,
! 8000, 7200, 6400, 5600, 4800, 4000,
! 3200, 2400, 1600, 800, 400
};
char cmd[MAXPGPATH];
--- 1113,1127 ----
static const int trial_conns[] = {
100, 50, 40, 30, 20, 10
};
+
+ /*
+ * Candidate values for shared_buffers in kB. When the value is
+ * divisible by 1024, we write it in MB-unit to configuration files.
+ */
static const int trial_bufs[] = {
! 32768, 28672, 24576, 20480, 16384, 12288,
! 8192, 7200, 6400, 5600, 4800, 4000,
3200, 2400, 1600, 800, 400
};
char cmd[MAXPGPATH];
*************** test_config_settings(void)
*** 1190,1196 ****
n_buffers = test_buffs;
n_fsm_pages = FSM_FOR_BUFS(n_buffers);
! printf("%dkB/%d\n", n_buffers, n_fsm_pages);
}
/*
--- 1195,1204 ----
n_buffers = test_buffs;
n_fsm_pages = FSM_FOR_BUFS(n_buffers);
! if (n_buffers % 1024 == 0)
! printf("%dMB/%d\n", n_buffers/1024, n_fsm_pages);
! else
! printf("%dkB/%d\n", n_buffers, n_fsm_pages);
}
/*
*************** setup_config(void)
*** 1213,1223 ****
snprintf(repltok, sizeof(repltok), "max_connections = %d", n_connections);
conflines = replace_token(conflines, "#max_connections = 100", repltok);
! snprintf(repltok, sizeof(repltok), "shared_buffers = %dkB", n_buffers);
! conflines = replace_token(conflines, "#shared_buffers = 32000kB", repltok);
snprintf(repltok, sizeof(repltok), "max_fsm_pages = %d", n_fsm_pages);
! conflines = replace_token(conflines, "#max_fsm_pages = 1600000", repltok);
#if DEF_PGPORT != 5432
snprintf(repltok, sizeof(repltok), "#port = %d", DEF_PGPORT);
--- 1221,1234 ----
snprintf(repltok, sizeof(repltok), "max_connections = %d", n_connections);
conflines = replace_token(conflines, "#max_connections = 100", repltok);
! if (n_buffers % 1024 == 0)
! snprintf(repltok, sizeof(repltok), "shared_buffers = %dMB", n_buffers/1024);
! else
! snprintf(repltok, sizeof(repltok), "shared_buffers = %dkB", n_buffers);
! conflines = replace_token(conflines, "#shared_buffers = 32MB", repltok);
snprintf(repltok, sizeof(repltok), "max_fsm_pages = %d", n_fsm_pages);
! conflines = replace_token(conflines, "#max_fsm_pages = 1638400", repltok);
#if DEF_PGPORT != 5432
snprintf(repltok, sizeof(repltok), "#port = %d", DEF_PGPORT);
diff -cpr pgsql-orig/src/include/utils/guc_tables.h pgsql/src/include/utils/guc_tables.h
*** pgsql-orig/src/include/utils/guc_tables.h Tue Sep 26 10:25:40 2006
--- pgsql/src/include/utils/guc_tables.h Tue Sep 26 11:31:06 2006
*************** struct config_generic
*** 132,143 ****
#define GUC_UNIT_KB 0x0400 /* value is in 1 kB */
#define GUC_UNIT_BLOCKS 0x0800 /* value is in blocks */
! #define GUC_UNIT_MEMORY (GUC_UNIT_KB|GUC_UNIT_BLOCKS)
#define GUC_UNIT_MS 0x1000 /* value is in milliseconds */
#define GUC_UNIT_S 0x2000 /* value is in seconds */
#define GUC_UNIT_MIN 0x4000 /* value is in minutes */
! #define GUC_UNIT_TIME (GUC_UNIT_MS|GUC_UNIT_S|GUC_UNIT_MIN)
/* bit values in status field */
#define GUC_HAVE_TENTATIVE 0x0001 /* tentative value is defined */
--- 132,144 ----
#define GUC_UNIT_KB 0x0400 /* value is in 1 kB */
#define GUC_UNIT_BLOCKS 0x0800 /* value is in blocks */
! #define GUC_UNIT_XBLOCKS 0x0C00 /* value is in xlog blocks */
! #define GUC_UNIT_MEMORY 0x0C00 /* mask for KB, BLOCKS, XBLOCKS */
#define GUC_UNIT_MS 0x1000 /* value is in milliseconds */
#define GUC_UNIT_S 0x2000 /* value is in seconds */
#define GUC_UNIT_MIN 0x4000 /* value is in minutes */
! #define GUC_UNIT_TIME 0x7000 /* mask for MS, S, MIN */
/* bit values in status field */
#define GUC_HAVE_TENTATIVE 0x0001 /* tentative value is defined */
Casey Duncan wrote:
Seems like the unit used for shared_buffers (and others) should be
megabytes then with a minimum of 1 (or more). Is less than 1MB
granularity really useful here?
Yes, there are platforms that allow as little as 512 kB of shared memory
by default.
--
Peter Eisentraut
http://developer.postgresql.org/~petere/
Patch applied. Thanks.
---------------------------------------------------------------------------
ITAGAKI Takahiro wrote:
The attached patch changes units of the some default values in postgresql.conf.
- shared_buffers = 32000kB => 32MB
- temp_buffers = 8000kB => 8MB
- wal_buffers = 8 => 64kBThe code of initdb was a bit modified to write MB-unit values.
Values greater than 8000kB are rounded out to MB.GUC_UNIT_XBLOCKS is added for wal_buffers. It is like GUC_UNIT_BLOCKS,
but uses XLOG_BLCKSZ instead of BLCKSZ.Also, I cleaned up the test of GUC_UNIT_* flags in preparation to
add more unit flags in less bits.Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center
[ Attachment, skipping... ]
---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings
--
Bruce Momjian bruce@momjian.us
EnterpriseDB http://www.enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +