fix stats_fetch_consistency value in postgresql.conf.sample
In postgresql.conf.sample, stats_fetch_consistency is set to "none," but
the default appears to be "cache." Should these be consistent? I've
attached a patch to change the entry in the sample.
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
Attachments:
fix_sample.patchtext/x-diff; charset=us-asciiDownload
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index a5a6d14cd4..48ad80cf2e 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -619,7 +619,7 @@
#track_io_timing = off
#track_wal_io_timing = off
#track_functions = none # none, pl, all
-#stats_fetch_consistency = none
+#stats_fetch_consistency = cache
# - Monitoring -
On Tue, May 24, 2022 at 03:01:47PM -0700, Nathan Bossart wrote:
In postgresql.conf.sample, stats_fetch_consistency is set to "none," but
the default appears to be "cache." Should these be consistent? I've
attached a patch to change the entry in the sample.
Yes, postgresql.conf.sample should reflect the default, and that's
PGSTAT_FETCH_CONSISTENCY_CACHE in guc.c. Andres, shouldn't
pgstat_fetch_consistency be initialized to the same in pgstat.c?
--
Michael
At Tue, 24 May 2022 15:01:47 -0700, Nathan Bossart <nathandbossart@gmail.com> wrote in
In postgresql.conf.sample, stats_fetch_consistency is set to "none," but
the default appears to be "cache." Should these be consistent? I've
attached a patch to change the entry in the sample.
Good catch:)
The base C variable is inirtialized with none.
The same GUC is intialized with "cache".
The default valur for the GUC is "none" in the sample file.
I think we set the same value to C variable. However, I wonder if it
would be possible to reduce the burden of unifying the three inital
values.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
Hi,
On 2022-05-25 13:11:40 +0900, Kyotaro Horiguchi wrote:
At Tue, 24 May 2022 15:01:47 -0700, Nathan Bossart <nathandbossart@gmail.com> wrote in
In postgresql.conf.sample, stats_fetch_consistency is set to "none," but
the default appears to be "cache." Should these be consistent? I've
attached a patch to change the entry in the sample.Good catch:)
The base C variable is inirtialized with none.
The same GUC is intialized with "cache".
The default valur for the GUC is "none" in the sample file.I think we set the same value to C variable. However, I wonder if it
would be possible to reduce the burden of unifying the three inital
values.
Yes, they should be the same. I think we ended up switching the default at
some point, and evidently I missed a step when doing so.
Will apply.
I wonder if we should make src/test/modules/test_misc/t/003_check_guc.pl
detect this kind of thing?
Greetings,
Andres Freund
On 2022-05-24 21:23:32 -0700, Andres Freund wrote:
Will apply.
And done. Thanks Nathan!
On Tue, May 24, 2022 at 09:28:49PM -0700, Andres Freund wrote:
And done. Thanks Nathan!
Shouldn't you also refresh pgstat_fetch_consistency in pgstat.c for
consistency?
I wonder if we should make src/test/modules/test_misc/t/003_check_guc.pl
detect this kind of thing?
That sounds like a good idea to me.
--
Michael
At Wed, 25 May 2022 14:00:23 +0900, Michael Paquier <michael@paquier.xyz> wrote in
On Tue, May 24, 2022 at 09:28:49PM -0700, Andres Freund wrote:
And done. Thanks Nathan!
Shouldn't you also refresh pgstat_fetch_consistency in pgstat.c for
consistency?I wonder if we should make src/test/modules/test_misc/t/003_check_guc.pl
detect this kind of thing?That sounds like a good idea to me.
I think the work as assumed not to use detailed knowledge of each
variable. Due to lack of knowlege about the detail of each variable,
for example, type, unit, internal exansion, we cannot check for the
following values.
- Numbers with unit (MB, kB, s, min ...)
- internally expanded string (FILE, ConfigDir)
So it's hard to automate to check consistency of all variables, but I
found the following inconsistencies between the sample config file and
GUC default value. C default value cannot be revealed so it is
ignored.
The following results are not deeply confirmed yet.
13 apparent inconsistencies are found. These should be fixed.
archive_command = "(disabled)" != ""
bgwriter_flush_after = "64" != "0"
checkpoint_flush_after = "32" != "0"
cluster_name = "main" != ""
default_text_search_config = "pg_catalog.english" != "pg_catalog.simple"
fsync = "off" != "on"
log_replication_commands = "on" != "off"
log_statement = "all" != "none"
max_wal_senders = "0" != "10"
restart_after_crash = "off" != "on"
stats_fetch_consistency = "cache" != "none"
wal_sync_method = "fdatasync" != "fsync"
11 has letter-case inconsistencies. Are these need to be fixed?
event_source = "postgresql" != "PostgreSQL"
lc_messages = "c" != "C"
lc_monetary = "en_us.utf-8" != "C"
lc_numeric = "en_us.utf-8" != "C"
lc_time = "en_us.utf-8" != "C"
log_filename = "postgresql-%y-%m-%d_%h%m%s.log" != "postgresql-%Y-%m-%d_%H%M%S.log"
log_line_prefix = "%m [%p] %q%a " != "%m [%p] "
ssl_ciphers = "high:medium:+3des:!anull" != "HIGH:MEDIUM:+3DES:!aNULL"
ssl_min_protocol_version = "tlsv1.2" != "TLSv1.2"
syslog_facility = "local0" != "LOCAL0"
timezone_abbreviations = "default" != "Default"
The followings are the result of automatic configuration?
client_encoding = "utf8" != "sql_ascii"
data_directory = "/home/horiguti/work/postgresql/src/test/modules/test_misc/tmp_check/t_003_check_guc_main_data/pgdata" != "ConfigDir"
hba_file = "/home/horiguti/work/postgresql/src/test/modules/test_misc/tmp_check/t_003_check_guc_main_data/pgdata/pg_hba.conf" != "ConfigDir/pg_hba.conf"
ident_file = "/home/horiguti/work/postgresql/src/test/modules/test_misc/tmp_check/t_003_check_guc_main_data/pgdata/pg_ident.conf" != "ConfigDir/pg_ident.conf"
krb_server_keyfile = "file:/home/horiguti/bin/pgsql_work/etc/krb5.keytab" != "FILE:${sysconfdir}/krb5.keytab"
log_timezone = "asia/tokyo" != "GMT"
timezone = "asia/tokyo" != "GMT"
unix_socket_directories = "/tmp/g3fpspvjuy" != "/tmp"
wal_buffers = "512" != "-1"
The followings are the result of TAP harness?
listen_addresses = "" != "localhost"
port = "60866" != "5432"
wal_level = "minimal" != "replica"
The following is inconsistent, but I'm not sure where the "500" came
from. In guc.c it is defined as 5000 and normal (out of TAP test)
server returns 5000.
wal_retrieve_retry_interval = "500" , "5s"
The followings cannot be automaticaly compared due to hidden unit
conversion, but looks consistent.
authentication_timeout = "60" , "1min"
autovacuum_naptime = "60" , "1min"
autovacuum_vacuum_cost_delay = "2" , "2ms"
bgwriter_delay = "200" , "200ms"
checkpoint_timeout = "300" , "5min"
checkpoint_warning = "30" , "30s"
deadlock_timeout = "1000" , "1s"
effective_cache_size = "524288" , "4GB"
gin_pending_list_limit = "4096" , "4MB"
log_autovacuum_min_duration = "600000" , "10min"
log_rotation_age = "1440" , "1d"
log_rotation_size = "10240" , "10MB"
log_startup_progress_interval = "10000" , "10s"
logical_decoding_work_mem = "65536" , "64MB"
maintenance_work_mem = "65536" , "64MB"
max_stack_depth = "2048" , "2MB"
max_standby_archive_delay = "30000" , "30s"
max_standby_streaming_delay = "30000" , "30s"
max_wal_size = "1024" , "1GB"
min_dynamic_shared_memory = "0" , "0MB"
min_parallel_index_scan_size = "64" , "512kB"
min_parallel_table_scan_size = "1024" , "8MB"
min_wal_size = "80" , "80MB"
shared_buffers = "16384" , "128MB"
temp_buffers = "1024" , "8MB"
wal_decode_buffer_size = "524288" , "512kB"
wal_receiver_status_interval = "10" , "10s"
wal_receiver_timeout = "60000" , "60s"
wal_sender_timeout = "60000" , "60s"
wal_skip_threshold = "2048" , "2MB"
wal_writer_delay = "200" , "200ms"
wal_writer_flush_after = "128" , "1MB"
work_mem = "4096" , "4MB"
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
At Wed, 25 May 2022 15:56:23 +0900 (JST), Kyotaro Horiguchi <horikyota.ntt@gmail.com> wrote in
The following results are not deeply confirmed yet.
13 apparent inconsistencies are found. These should be fixed.
archive_command = "(disabled)" != ""
The "(disabled)" is the representation of "".
bgwriter_flush_after = "64" != "0"
checkpoint_flush_after = "32" != "0"
They vary according to the existence of sync_file_range().
cluster_name = "main" != ""
This is named by 003_check_guc.pl.
default_text_search_config = "pg_catalog.english" != "pg_catalog.simple"
initdb decided this.
fsync = "off" != "on"
log_line_prefix = "%m [%p] %q%a " != "%m [%p] "
log_replication_commands = "on" != "off"
log_statement = "all" != "none"
max_wal_senders = "0" != "10"
restart_after_crash = "off" != "on"
These are set by Cluster.pm.
wal_sync_method = "fdatasync" != "fsync"
This is platform dependent.
stats_fetch_consistency = "cache" != "none"
This has been fixed recently.
11 has letter-case inconsistencies. Are these need to be fixed?
event_source = "postgresql" != "PostgreSQL"
lc_messages = "c" != "C"
lc_monetary = "en_us.utf-8" != "C"
lc_numeric = "en_us.utf-8" != "C"
lc_time = "en_us.utf-8" != "C"
log_filename = "postgresql-%y-%m-%d_%h%m%s.log" != "postgresql-%Y-%m-%d_%H%M%S.log"
ssl_ciphers = "high:medium:+3des:!anull" != "HIGH:MEDIUM:+3DES:!aNULL"
ssl_min_protocol_version = "tlsv1.2" != "TLSv1.2"
syslog_facility = "local0" != "LOCAL0"
timezone_abbreviations = "default" != "Default"
These are harmless. Since no significant inconsistency is found,
there's no need to fix these either.
(sigh..) As the result, no need to fix in this area for now, and I
don't think there's any generic and reliable way to detect
inconsistencies of guc variable definitions.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
On Wed, May 25, 2022 at 04:12:07PM +0900, Kyotaro Horiguchi wrote:
(sigh..) As the result, no need to fix in this area for now, and I
don't think there's any generic and reliable way to detect
inconsistencies of guc variable definitions.
Hmm. Making the automation test painless in terms of maintenance
consists in making it require zero manual filtering in the list of
GUCs involved, while still being useful in what it can detect. The
units involved in a GUC make the checks between postgresql.conf.sample
and pg_settings.boot_value annoying because they would require extra
calculations depending on the unit with a logic maintained in the
test.
I may be missing something obvious, of course, but it seems to me that
as long as you fetch the values from postgresql.conf.sample and
cross-check them with pg_settings.boot_value for GUCs that do not have
units, the maintenance would be painless, while still being useful (it
would cover the case of enums, for one). The values need to be
lower-cased for consistency, similarly to the GUC names.
--
Michael
At Thu, 26 May 2022 08:53:55 +0900, Michael Paquier <michael@paquier.xyz> wrote in
On Wed, May 25, 2022 at 04:12:07PM +0900, Kyotaro Horiguchi wrote:
(sigh..) As the result, no need to fix in this area for now, and I
don't think there's any generic and reliable way to detect
inconsistencies of guc variable definitions.Hmm. Making the automation test painless in terms of maintenance
consists in making it require zero manual filtering in the list of
GUCs involved, while still being useful in what it can detect. The
units involved in a GUC make the checks between postgresql.conf.sample
and pg_settings.boot_value annoying because they would require extra
calculations depending on the unit with a logic maintained in the
test.I may be missing something obvious, of course, but it seems to me that
as long as you fetch the values from postgresql.conf.sample and
cross-check them with pg_settings.boot_value for GUCs that do not have
units, the maintenance would be painless, while still being useful (it
would cover the case of enums, for one). The values need to be
lower-cased for consistency, similarly to the GUC names.
Yeah, "boot_val" is appropreate here. And I noticed that pg_settings
has the "unit" field. I'll try using them.
Thanks for the suggestion!
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
On Thu, May 26, 2022 at 11:10:18AM +0900, Kyotaro Horiguchi wrote:
Yeah, "boot_val" is appropreate here. And I noticed that pg_settings
has the "unit" field. I'll try using them.
I wrote this in guc.sql, which seems promising, but it needs to be rewritten in
check_guc.pl to access postgresql.conf from the source tree. Do you want to
handle that ?
+\getenv abs_srcdir PG_ABS_SRCDIR
+\set filename :abs_srcdir '../../../../src/backend/utils/misc/postgresql.conf.sample'
+
+begin;
+CREATE TEMP TABLE sample_conf AS
+-- SELECT m[1] AS name, trim(BOTH '''' FROM m[3]) AS sample_value
+SELECT m[1] AS name, COALESCE(m[3], m[5]) AS sample_value
+FROM (SELECT regexp_split_to_table(pg_read_file(:'filename'), '\n') AS ln) conf,
+-- regexp_match(ln, '^#?([_[:alpha:]]+) (= ([^[:space:]]*)|[^ ]*$).*') AS m
+regexp_match(ln, '^#?([_[:alpha:]]+) (= ''([^'']*)''|(= ([^[:space:]]*))|[^ ]*$).*') AS m
+WHERE ln ~ '^#?[[:alpha:]]';
+
+-- test that GUCs in postgresql.conf have correct default values
+SELECT name, tsf.cooked_value, sc.sample_value
+FROM tab_settings_flags tsf JOIN sample_conf sc USING(name)
+WHERE NOT not_in_sample AND tsf.cooked_value != sc.sample_value AND tsf.cooked_value||'.0' != sc.sample_value
+ORDER BY 1;
+rollback;
It detects the original problem:
stats_fetch_consistency | cache | none
And I think these should be updated it postgresql.conf to use the same unit as
in current_setting().
track_activity_query_size | 1kB | 1024
wal_buffers | 4MB | -1
wal_receiver_timeout | 1min | 60s
wal_sender_timeout | 1min | 60s
--
Justin
At Wed, 25 May 2022 21:25:53 -0500, Justin Pryzby <pryzby@telsasoft.com> wrote in
On Thu, May 26, 2022 at 11:10:18AM +0900, Kyotaro Horiguchi wrote:
Yeah, "boot_val" is appropreate here. And I noticed that pg_settings
has the "unit" field. I'll try using them.I wrote this in guc.sql, which seems promising, but it needs to be rewritten in
check_guc.pl to access postgresql.conf from the source tree. Do you want to
handle that ?
Yes.
+\getenv abs_srcdir PG_ABS_SRCDIR +\set filename :abs_srcdir '../../../../src/backend/utils/misc/postgresql.conf.sample' + +begin; +CREATE TEMP TABLE sample_conf AS +-- SELECT m[1] AS name, trim(BOTH '''' FROM m[3]) AS sample_value +SELECT m[1] AS name, COALESCE(m[3], m[5]) AS sample_value +FROM (SELECT regexp_split_to_table(pg_read_file(:'filename'), '\n') AS ln) conf, +-- regexp_match(ln, '^#?([_[:alpha:]]+) (= ([^[:space:]]*)|[^ ]*$).*') AS m +regexp_match(ln, '^#?([_[:alpha:]]+) (= ''([^'']*)''|(= ([^[:space:]]*))|[^ ]*$).*') AS m +WHERE ln ~ '^#?[[:alpha:]]'; + +-- test that GUCs in postgresql.conf have correct default values +SELECT name, tsf.cooked_value, sc.sample_value +FROM tab_settings_flags tsf JOIN sample_conf sc USING(name) +WHERE NOT not_in_sample AND tsf.cooked_value != sc.sample_value AND tsf.cooked_value||'.0' != sc.sample_value +ORDER BY 1; +rollback;It detects the original problem:
stats_fetch_consistency | cache | none
Yeah, it is a straight forward outcome.
And I think these should be updated it postgresql.conf to use the same unit as
in current_setting().track_activity_query_size | 1kB | 1024
wal_buffers | 4MB | -1
wal_receiver_timeout | 1min | 60s
wal_sender_timeout | 1min | 60s
I'm not sure we should do so. Rather I'd prefer 60s than 1min here.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
At Thu, 26 May 2022 13:00:45 +0900 (JST), Kyotaro Horiguchi <horikyota.ntt@gmail.com> wrote in
At Wed, 25 May 2022 21:25:53 -0500, Justin Pryzby <pryzby@telsasoft.com> wrote in
And I think these should be updated it postgresql.conf to use the same unit as
in current_setting().track_activity_query_size | 1kB | 1024
wal_buffers | 4MB | -1
wal_receiver_timeout | 1min | 60s
wal_sender_timeout | 1min | 60sI'm not sure we should do so. Rather I'd prefer 60s than 1min here.
It could be in SQL, but *I* prefer to use perl for this, since it
allows me to write a bit complex things (than simple string
comparison) simpler.
So the attached is a wip version of that
Numeric values are compared considering units. But does not require
the units of the both values to match. Some variables are ignored by
an explicit instruction (ignored_parameters). Some variables are
compared case-insensitively by an explicit instruction
(case_insensitive_params). bool and enum are compared
case-insensitively automatically.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
Attachments:
0001-Add-fileval-bootval-consistency-check-of-GUC-paramet.patchtext/x-patch; charset=us-asciiDownload
From d06e248ced6a45aa10badae9b223e016605e61c5 Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Date: Thu, 26 May 2022 15:55:34 +0900
Subject: [PATCH] Add fileval-bootval consistency check of GUC parameters
We should keep GUC variables consistent between the default values
written in postgresql.conf.sample and defined in guc.c. Add an
automated way to check for the consistency to the TAP test suite. Some
variables are still excluded since they cannot be checked simple way.
---
src/test/modules/test_misc/t/003_check_guc.pl | 151 +++++++++++++++++-
1 file changed, 146 insertions(+), 5 deletions(-)
diff --git a/src/test/modules/test_misc/t/003_check_guc.pl b/src/test/modules/test_misc/t/003_check_guc.pl
index 60459ef759..efaa7ec182 100644
--- a/src/test/modules/test_misc/t/003_check_guc.pl
+++ b/src/test/modules/test_misc/t/003_check_guc.pl
@@ -11,18 +11,65 @@ my $node = PostgreSQL::Test::Cluster->new('main');
$node->init;
$node->start;
+# unit conversion factors
+my %unit_conv_factor =
+ ('b', 1,
+ 'kb', 1024,
+ 'mb', 1024 * 1024,
+ 'gb', 1024 * 1024 * 1024,
+ 'tb', 1024 * 1024 * 1024 * 1024,
+ '8kb', 8 * 1024,
+ 'us', 1.0 / (1000 / 1000),
+ 'ms', 1.0 / 1000,
+ 's', 1,
+ 'min', 60,
+ 'h', 3600,
+ 'd', 24 * 3600);
+
+# parameter names that cannot get consistency check performed
+my @ignored_parameters =
+ ('data_directory',
+ 'hba_file',
+ 'ident_file',
+ 'krb_server_keyfile',
+ 'max_stack_depth',
+ 'bgwriter_flush_after',
+ 'wal_sync_method',
+ 'checkpoint_flush_after',
+ 'timezone_abbreviations',
+ 'lc_messages');
+
+# parameter names that requires case-insensitive check
+my @case_insensitive_params =
+ ('ssl_ciphers',
+ 'log_filename',
+ 'event_source',
+ 'log_timezone',
+ 'timezone',
+ 'lc_monetary',
+ 'lc_numeric',
+ 'lc_time');
+
# Grab the names of all the parameters that can be listed in the
# configuration sample file. config_file is an exception, it is not
# in postgresql.conf.sample but is part of the lists from guc.c.
my $all_params = $node->safe_psql(
'postgres',
- "SELECT name
+ "SELECT name, vartype, unit, boot_val, '!'
FROM pg_settings
WHERE NOT 'NOT_IN_SAMPLE' = ANY (pg_settings_get_flags(name)) AND
name <> 'config_file'
ORDER BY 1");
# Note the lower-case conversion, for consistency.
-my @all_params_array = split("\n", lc($all_params));
+my %all_params_hash;
+foreach my $line (split("\n", lc($all_params)))
+{
+ my @f = split('\|', $line);
+ fail("query returned wrong number of columns: $#f : $line") if ($#f != 4);
+ $all_params_hash{$f[0]}->{type} = $f[1];
+ $all_params_hash{$f[0]}->{unit} = $f[2];
+ $all_params_hash{$f[0]}->{bootval} = $f[3];
+}
# Grab the names of all parameters marked as NOT_IN_SAMPLE.
my $not_in_sample = $node->safe_psql(
@@ -44,6 +91,7 @@ my @gucs_in_file;
# Read the sample file line-by-line, checking its contents to build a list
# of everything known as a GUC.
my $num_tests = 0;
+my $failed = 0;
open(my $contents, '<', $sample_file)
|| die "Could not open $sample_file: $!";
while (my $line = <$contents>)
@@ -53,11 +101,16 @@ while (my $line = <$contents>)
# file.
# - Valid configuration options are followed immediately by " = ",
# with one space before and after the equal sign.
- if ($line =~ m/^#?([_[:alpha:]]+) = .*/)
+ if ($line =~ m/^#?([_[:alpha:]]+) = (.*)$/)
{
# Lower-case conversion matters for some of the GUCs.
my $param_name = lc($1);
+ # extract value
+ my $file_value = $2;
+ $file_value =~ s/\s*#.*$//; # strip trailing comment
+ $file_value =~ s/^'(.*)'$/$1/; # strip quotes
+
# Ignore some exceptions.
next if $param_name eq "include";
next if $param_name eq "include_dir";
@@ -66,19 +119,28 @@ while (my $line = <$contents>)
# Update the list of GUCs found in the sample file, for the
# follow-up tests.
push @gucs_in_file, $param_name;
+
+ # Check for consistency between bootval and file value.
+ $failed++
+ if (!check_val($param_name,
+ $all_params_hash{$param_name}->{type},
+ $all_params_hash{$param_name}->{bootval},
+ $all_params_hash{$param_name}->{unit},
+ $file_value));
}
}
close $contents;
+pass("fileval-bootval consistency is fine") if ($failed == 0);
+
# Cross-check that all the GUCs found in the sample file match the ones
# fetched above. This maps the arrays to a hash, making the creation of
# each exclude and intersection list easier.
my %gucs_in_file_hash = map { $_ => 1 } @gucs_in_file;
-my %all_params_hash = map { $_ => 1 } @all_params_array;
my %not_in_sample_hash = map { $_ => 1 } @not_in_sample_array;
-my @missing_from_file = grep(!$gucs_in_file_hash{$_}, @all_params_array);
+my @missing_from_file = grep(!$gucs_in_file_hash{$_}, keys %all_params_hash);
is(scalar(@missing_from_file),
0, "no parameters missing from postgresql.conf.sample");
@@ -108,3 +170,82 @@ foreach my $param (@sample_intersect)
}
done_testing();
+
+
+# parse $val according to $type.
+# $val may be suffixed by a unit if $optunit is not given.
+sub parse_val
+{
+ my ($type, $val, $optunit) = @_;
+
+ if ($val =~ /^(\d+)([a-zA-Z]*)$/)
+ {
+ $val = $1;
+ my $unit = $2;
+
+ # integer allows octal representaion
+ $val = oct($val) if ($type eq "integer" && $val =~ /^0./);
+
+ # accept $optunit only if $val was not suffixed by a unit
+ if (length $optunit)
+ {
+ fail("check if unit \"$unit\" is known to this script")
+ if (length $unit);
+
+ $unit = $optunit;
+ }
+
+
+ if (length($unit))
+ {
+ $unit = lc($unit);
+
+ fail("check if unit \"$unit\" is known to this script")
+ if (!defined $unit_conv_factor{$unit});
+
+ $val *= $unit_conv_factor{$unit};
+ }
+ }
+
+ return $val;
+}
+
+# check if $bootval and $fileval match according to $type and $unit
+sub check_val
+{
+ my ($param_name, $type, $bootval, $unit, $fileval) = @_;
+
+ my $match = 0;
+
+ if (grep { $_ eq $param_name } @ignored_parameters)
+ {
+ # this parameter cannot be checked, skip.
+ $match = 1;
+ }
+ elsif ($type eq "integer" || $type eq "real")
+ {
+ # this parameter is numeric, parse with unit conversion
+ my $bootval = parse_val($type, $bootval, $unit);
+ my $fileval = parse_val($type, $fileval);
+
+ $match = 1 if ($bootval == $fileval);
+ }
+ elsif (($type eq "bool" || $type eq "enum") ||
+ grep { $_ eq $param_name} @case_insensitive_params)
+ {
+ # this parameter is compared case-insensitively
+ $match = 1 if (lc($bootval) eq lc($fileval));
+ }
+ else
+ {
+ # exact match
+ $match = 1 if ($bootval eq $fileval);
+ }
+
+ if (!$match)
+ {
+ fail("$param_name inconsistent (\"$bootval\" vs \"$fileval\")");
+ }
+
+ return $match;
+}
--
2.31.1
Hi,
On 2022-05-25 14:00:23 +0900, Michael Paquier wrote:
On Tue, May 24, 2022 at 09:28:49PM -0700, Andres Freund wrote:
And done. Thanks Nathan!
Shouldn't you also refresh pgstat_fetch_consistency in pgstat.c for
consistency?
Yes. Now that the visible sheen of embarassment in my face has subsided a bit
(and pgcon has ended), I pushed this bit too.
- Andres
Hi,
On 2022-05-26 16:27:53 +0900, Kyotaro Horiguchi wrote:
It could be in SQL, but *I* prefer to use perl for this, since it
allows me to write a bit complex things (than simple string
comparison) simpler.
I wonder if we shouldn't just expose a C function to do this, rather than
having a separate implementation in a tap test.
+# parameter names that cannot get consistency check performed +my @ignored_parameters =
I think most of these we could ignore by relying on source <> 'override'
instead of listing them?
+# parameter names that requires case-insensitive check +my @case_insensitive_params = + ('ssl_ciphers', + 'log_filename', + 'event_source', + 'log_timezone', + 'timezone', + 'lc_monetary', + 'lc_numeric', + 'lc_time');
Why do these differ by case?
Greetings,
Andres Freund
On Sat, May 28, 2022 at 01:14:09PM -0700, Andres Freund wrote:
I pushed this bit too.
Thanks for taking care of that!
--
Michael
At Sat, 28 May 2022 13:22:45 -0700, Andres Freund <andres@anarazel.de> wrote in
Hi,
On 2022-05-26 16:27:53 +0900, Kyotaro Horiguchi wrote:
It could be in SQL, but *I* prefer to use perl for this, since it
allows me to write a bit complex things (than simple string
comparison) simpler.I wonder if we shouldn't just expose a C function to do this, rather than
having a separate implementation in a tap test.
It was annoying that I needed to copy the unit-conversion stuff. I
did that in the attached. parse_val() and check_val() and the duped
data is removed.
+# parameter names that cannot get consistency check performed +my @ignored_parameters =I think most of these we could ignore by relying on source <> 'override'
instead of listing them?+# parameter names that requires case-insensitive check +my @case_insensitive_params = + ('ssl_ciphers', + 'log_filename', + 'event_source', + 'log_timezone', + 'timezone', + 'lc_monetary', + 'lc_numeric', + 'lc_time');Why do these differ by case?
Mmm. It just came out of a thinko. I somehow believed that the script
down-cases only the parameter names among the values from
pg_settings. I felt that something's strange while on it,
though.. After fixing it, there are only the following values that
differ only in letter cases. In passing I changed "bool" and "enum"
are case-sensitive, too.
name conf bootval
client_encoding: "sql_ascii" "SQL_ASCII"
datestyle : "iso, mdy" "ISO, MDY"
syslog_facility: "LOCAL0" "local0"
It seems to me that the bootval is right for all variables.
I added a testing-aid function pg_normalize_config_option(name,value)
so the consistency check can be performed like this.
SELECT f.n, f.v, s.boot_val
FROM (VALUES ('work_mem','4MB'),...) f(n,v)
JOIN pg_settings s ON s.name = f.n '.
WHERE pg_normalize_config_value(f.n, f.v) <> '.
pg_normalize_config_value(f.n, s.boot_val)';
There're some concerns on the function.
- _ShowConfig() returns archive_command as "(disabled)" regardless of
its value. The test passes accidentally for the variable...
- _ShowConfig() errors out for "timezone_abbreviations" and "" since
the check function tries to open the timezone file. (It is excluded
from the test.)
I don't want to create a copy of the function only for this purpose.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
Attachments:
v2-0001-Add-fileval-bootval-consistency-check-of-GUC-para.patchtext/x-patch; charset=us-asciiDownload
From 1e52b23111b5e8860092dcfea0ac9459f1725f0d Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Date: Mon, 30 May 2022 16:11:34 +0900
Subject: [PATCH v2] Add fileval-bootval consistency check of GUC parameters
We should keep GUC variables consistent between the default values
written in postgresql.conf.sample and defined in guc.c. Add an
automated way to check for the consistency to the TAP test suite. Some
variables are still excluded since they cannot be checked simple way.
---
src/backend/utils/misc/guc.c | 70 +++++++++++++++++++
src/backend/utils/misc/postgresql.conf.sample | 6 +-
src/include/catalog/pg_proc.dat | 5 ++
src/test/modules/test_misc/t/003_check_guc.pl | 55 +++++++++++++--
4 files changed, 127 insertions(+), 9 deletions(-)
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 55d41ae7d6..cd47dede1a 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -7514,6 +7514,76 @@ parse_and_validate_value(struct config_generic *record,
return true;
}
+/*
+ * Helper function for pg_normalize_config_value().
+ * Makes a palloced copy of src then link val to it.
+ * DO NOT destroy val while dst is in use.
+ */
+static struct config_generic *
+copy_config_and_set_value(struct config_generic *src, union config_var_val *val)
+{
+ struct config_generic *dst;
+
+#define CREATE_CONFIG_COPY(dst, src, t) \
+ dst = palloc(sizeof(struct t)); \
+ *(struct t *) dst = *(struct t *) src; \
+
+ switch (src->vartype)
+ {
+ case PGC_BOOL:
+ CREATE_CONFIG_COPY(dst, src, config_bool);
+ ((struct config_bool *)dst)->variable = &val->boolval;
+ break;
+ case PGC_INT:
+ CREATE_CONFIG_COPY(dst, src, config_int);
+ ((struct config_int *)dst)->variable = &val->intval;
+ break;
+ case PGC_REAL:
+ CREATE_CONFIG_COPY(dst, src, config_real);
+ ((struct config_real *)dst)->variable = &val->realval;
+ break;
+ case PGC_STRING:
+ CREATE_CONFIG_COPY(dst, src, config_string);
+ ((struct config_string *)dst)->variable = &val->stringval;
+ break;
+ case PGC_ENUM:
+ CREATE_CONFIG_COPY(dst, src, config_enum);
+ ((struct config_enum *)dst)->variable = &val->enumval;
+ break;
+ }
+
+ return dst;
+}
+
+
+/*
+ * Normalize given value according to the specified GUC variable
+ */
+Datum
+pg_normalize_config_value(PG_FUNCTION_ARGS)
+{
+ char *name = "";
+ char *value = "";
+ struct config_generic *record;
+ char *result;
+ void *extra;
+ union config_var_val altval;
+
+ if (!PG_ARGISNULL(0))
+ name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ if (!PG_ARGISNULL(1))
+ value = text_to_cstring(PG_GETARG_TEXT_PP(1));
+
+ record = find_option(name, true, false, ERROR);
+
+ parse_and_validate_value(record, name, value, PGC_S_TEST, WARNING,
+ &altval, &extra);
+ record = copy_config_and_set_value(record, &altval);
+
+ result = _ShowOption(record, true);
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
/*
* Sets option `name' to given value.
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 48ad80cf2e..a6d5e401a9 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -476,7 +476,7 @@
# in all cases.
# These are relevant when logging to syslog:
-#syslog_facility = 'LOCAL0'
+#syslog_facility = 'local0'
#syslog_ident = 'postgres'
#syslog_sequence_numbers = on
#syslog_split_messages = on
@@ -709,7 +709,7 @@
# - Locale and Formatting -
-#datestyle = 'iso, mdy'
+#datestyle = 'ISO, MDY'
#intervalstyle = 'postgres'
#timezone = 'GMT'
#timezone_abbreviations = 'Default' # Select the set of available time zone
@@ -721,7 +721,7 @@
# share/timezonesets/.
#extra_float_digits = 1 # min -15, max 3; any value >0 actually
# selects precise output mode
-#client_encoding = sql_ascii # actually, defaults to database
+#client_encoding = SQL_ASCII # actually, defaults to database
# encoding
# These settings are initialized by initdb, but they can be changed.
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 87aa571a33..c089b351e9 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6118,6 +6118,11 @@
proname => 'pg_settings_get_flags', provolatile => 's', prorettype => '_text',
proargtypes => 'text', prosrc => 'pg_settings_get_flags' },
+{ oid => '9956', descr => 'normalize value to the unit of specified GUC',
+ proname => 'pg_normalize_config_value', proisstrict => 'f',
+ provolatile => 's', prorettype => 'text', proargtypes => 'text text',
+ proargnames => '{varname,value}', prosrc => 'pg_normalize_config_value' },
+
{ oid => '3329', descr => 'show config file settings',
proname => 'pg_show_all_file_settings', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
diff --git a/src/test/modules/test_misc/t/003_check_guc.pl b/src/test/modules/test_misc/t/003_check_guc.pl
index 60459ef759..a92206942f 100644
--- a/src/test/modules/test_misc/t/003_check_guc.pl
+++ b/src/test/modules/test_misc/t/003_check_guc.pl
@@ -11,18 +11,40 @@ my $node = PostgreSQL::Test::Cluster->new('main');
$node->init;
$node->start;
+# parameter names that cannot get consistency check performed
+my @ignored_parameters = (
+ 'data_directory',
+ 'hba_file',
+ 'ident_file',
+ 'krb_server_keyfile',
+ 'max_stack_depth',
+ 'bgwriter_flush_after',
+ 'wal_sync_method',
+ 'checkpoint_flush_after',
+ 'timezone_abbreviations',
+ 'lc_messages'
+ );
+
# Grab the names of all the parameters that can be listed in the
# configuration sample file. config_file is an exception, it is not
# in postgresql.conf.sample but is part of the lists from guc.c.
my $all_params = $node->safe_psql(
'postgres',
- "SELECT name
+ "SELECT lower(name), vartype, unit, boot_val, '!'
FROM pg_settings
WHERE NOT 'NOT_IN_SAMPLE' = ANY (pg_settings_get_flags(name)) AND
name <> 'config_file'
ORDER BY 1");
# Note the lower-case conversion, for consistency.
-my @all_params_array = split("\n", lc($all_params));
+my %all_params_hash;
+foreach my $line (split("\n", $all_params))
+{
+ my @f = split('\|', $line);
+ fail("query returned wrong number of columns: $#f : $line") if ($#f != 4);
+ $all_params_hash{$f[0]}->{type} = $f[1];
+ $all_params_hash{$f[0]}->{unit} = $f[2];
+ $all_params_hash{$f[0]}->{bootval} = $f[3];
+}
# Grab the names of all parameters marked as NOT_IN_SAMPLE.
my $not_in_sample = $node->safe_psql(
@@ -43,7 +65,7 @@ my @gucs_in_file;
# Read the sample file line-by-line, checking its contents to build a list
# of everything known as a GUC.
-my $num_tests = 0;
+my @check_elems = ();
open(my $contents, '<', $sample_file)
|| die "Could not open $sample_file: $!";
while (my $line = <$contents>)
@@ -53,11 +75,16 @@ while (my $line = <$contents>)
# file.
# - Valid configuration options are followed immediately by " = ",
# with one space before and after the equal sign.
- if ($line =~ m/^#?([_[:alpha:]]+) = .*/)
+ if ($line =~ m/^#?([_[:alpha:]]+) = (.*)$/)
{
# Lower-case conversion matters for some of the GUCs.
my $param_name = lc($1);
+ # extract value
+ my $file_value = $2;
+ $file_value =~ s/\s*#.*$//; # strip trailing comment
+ $file_value =~ s/^'(.*)'$/$1/; # strip quotes
+
# Ignore some exceptions.
next if $param_name eq "include";
next if $param_name eq "include_dir";
@@ -66,19 +93,35 @@ while (my $line = <$contents>)
# Update the list of GUCs found in the sample file, for the
# follow-up tests.
push @gucs_in_file, $param_name;
+
+ # Check for consistency between bootval and file value.
+ if (!grep { $_ eq $param_name } @ignored_parameters)
+ {
+ push (@check_elems, "('$param_name','$file_value')");
+ }
}
}
close $contents;
+# run consistency check between config-file's default value and boot values.
+my $check_query =
+ 'SELECT f.n, f.v, s.boot_val FROM (VALUES '.
+ join(',', @check_elems).
+ ') f(n,v) JOIN pg_settings s ON s.name = f.n '.
+ 'WHERE pg_normalize_config_value(f.n, f.v) <> '.
+ 'pg_normalize_config_value(f.n, s.boot_val)';
+
+is ($node->safe_psql('postgres', $check_query), '',
+ 'check if fileval-bootval consistency is fine');
+
# Cross-check that all the GUCs found in the sample file match the ones
# fetched above. This maps the arrays to a hash, making the creation of
# each exclude and intersection list easier.
my %gucs_in_file_hash = map { $_ => 1 } @gucs_in_file;
-my %all_params_hash = map { $_ => 1 } @all_params_array;
my %not_in_sample_hash = map { $_ => 1 } @not_in_sample_array;
-my @missing_from_file = grep(!$gucs_in_file_hash{$_}, @all_params_array);
+my @missing_from_file = grep(!$gucs_in_file_hash{$_}, keys %all_params_hash);
is(scalar(@missing_from_file),
0, "no parameters missing from postgresql.conf.sample");
--
2.31.1
On Mon, May 30, 2022 at 05:27:19PM +0900, Kyotaro Horiguchi wrote:
At Sat, 28 May 2022 13:22:45 -0700, Andres Freund <andres@anarazel.de> wrote in
Hi,
On 2022-05-26 16:27:53 +0900, Kyotaro Horiguchi wrote:
It could be in SQL, but *I* prefer to use perl for this, since it
allows me to write a bit complex things (than simple string
comparison) simpler.I wonder if we shouldn't just expose a C function to do this, rather than
having a separate implementation in a tap test.It was annoying that I needed to copy the unit-conversion stuff. I
did that in the attached. parse_val() and check_val() and the duped
data is removed.
Note that this gives:
guc.c:7573:9: warning: ‘dst’ may be used uninitialized in this function [-Wmaybe-uninitialized]
with gcc version 9.2.1 20191008 (Ubuntu 9.2.1-9ubuntu2)
I wonder whether you'd consider renaming pg_normalize_config_value() to
pg_pretty_config_value() or similar.
--
Justin
On Sat, Jun 11, 2022 at 09:41:37AM -0500, Justin Pryzby wrote:
Note that this gives:
guc.c:7573:9: warning: ‘dst’ may be used uninitialized in this function [-Wmaybe-uninitialized]
with gcc version 9.2.1 20191008 (Ubuntu 9.2.1-9ubuntu2)
I wonder whether you'd consider renaming pg_normalize_config_value() to
pg_pretty_config_value() or similar.
I have looked at the patch, and I am not convinced that we need a
function that does a integer -> integer-with-unit conversion for the
purpose of this test. One thing is that it can be unstable with the
unit in the conversion where values are close to a given threshold
(aka for cases like 2048kB/2MB). On top of that, this overlaps with
the existing system function in charge of converting values with bytes
as size unit, while this stuff handles more unit types and all GUC
types. I think that there could be some room in doing the opposite
conversion, feeding the value from postgresql.conf.sample to something
and compare it directly with boot_val. That's solvable at SQL level,
still a system function may be more user-friendly.
Extending the tests to check after the values is something worth
doing, but I think that I would limit the checks to the parameters
that do not have units for now, until we figure out which interface
would be more adapted for doing the normalization of the parameter
values.
-#syslog_facility = 'LOCAL0'
+#syslog_facility = 'local0'
Those changes should not be necessary in postgresql.conf.sample. The
test should be in charge of applying the lower() conversion, in the
same way as guc.c does internally, and that's a mode supported by the
parameter parsing. Using an upper-case value in the sample file is
actually meaningful sometimes (for example, syslog can use upper-case
strings to refer to LOCAL0~7).
--
Michael
At Thu, 16 Jun 2022 12:07:03 +0900, Michael Paquier <michael@paquier.xyz> wrote in
On Sat, Jun 11, 2022 at 09:41:37AM -0500, Justin Pryzby wrote:
Note that this gives:
guc.c:7573:9: warning: ‘dst’ may be used uninitialized in this function [-Wmaybe-uninitialized]
with gcc version 9.2.1 20191008 (Ubuntu 9.2.1-9ubuntu2)
I wonder whether you'd consider renaming pg_normalize_config_value() to
pg_pretty_config_value() or similar.I have looked at the patch, and I am not convinced that we need a
function that does a integer -> integer-with-unit conversion for the
purpose of this test. One thing is that it can be unstable with the
unit in the conversion where values are close to a given threshold
(aka for cases like 2048kB/2MB). On top of that, this overlaps with
I agree that needing to-with-unit conversion is a bit crumsy. One of
the reason is I didn't want to add a function that has no use of other
than testing,
the existing system function in charge of converting values with bytes
as size unit, while this stuff handles more unit types and all GUC
types. I think that there could be some room in doing the opposite
conversion, feeding the value from postgresql.conf.sample to something
and compare it directly with boot_val. That's solvable at SQL level,
still a system function may be more user-friendly.
The output value must be the same with what pg_settings shows, so it
needs to take in some code from GetConfigOptionByNum() (and needs to
keep in-sync with it), which is what I didn't wnat to do. Anyway done
in the attached.
This method has a problem for wal_buffers. parse_and_validate_value()
returns 512 for -1 input since check_wal_buffers() converts it to 512.
It is added to the exclusion list. (Conversely the previous method
regarded "-1" and "512" as identical.)
Extending the tests to check after the values is something worth
doing, but I think that I would limit the checks to the parameters
that do not have units for now, until we figure out which interface
would be more adapted for doing the normalization of the parameter
values.
The attached second is that. FWIW, I'd like to support integer/real
values since I think they need more support of this kind of check.
-#syslog_facility = 'LOCAL0'
+#syslog_facility = 'local0'
Those changes should not be necessary in postgresql.conf.sample. The
test should be in charge of applying the lower() conversion, in the
same way as guc.c does internally, and that's a mode supported by the
parameter parsing. Using an upper-case value in the sample file is
actually meaningful sometimes (for example, syslog can use upper-case
strings to refer to LOCAL0~7).
I didn't notice, but now know parse_and_validate_value() convers
values the same way with bootval so finally case-unification is not
needed.
=# select pg_config_unitless_value('datestyle', 'iso, mdy');
pg_config_unitless_value
--------------------------
ISO, MDY
However, the "datestyle" variable is shown as "DateStyle" in the
pg_settings view. So the name in the view needs to be lower-cased
instead. The same can be said to "TimeZone" and "IntervalStyle". The
old query missed the case where there's no variable with the names
appear in the config file. Fixed it.
At Sat, 11 Jun 2022 09:41:37 -0500, Justin Pryzby <pryzby@telsasoft.com> wrote in
Note that this gives:
guc.c:7573:9: warning: ‘dst’ may be used uninitialized in this function [-Wmaybe-uninitialized]
Mmm. I don't have an idea where the 'dst' came from...
I wonder whether you'd consider renaming pg_normalize_config_value() to
pg_pretty_config_value() or similar.
Yeah, that's sensible, the function is now changed (not renamed) to
pg_config_unitless_value(). This name also doesn't satisfies me at
all..:(
So, the attached are:
v2-0001-Add-fileval-bootval-consistency-check-of-GUC-para.patch:
New version of the previous patch. It is changed after Michael's
suggestions.
0001-Add-fileval-bootval-consistency-check-of-GUC-paramet-simple.patch
Another version that doesn't need new C function. It ignores
variables that have units but I didn't count how many variables are
ignored by this chnage.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
Attachments:
v2-0001-Add-fileval-bootval-consistency-check-of-GUC-para.patchtext/x-patch; charset=us-asciiDownload
From 7ffa4f5f59564880107ffc3fa0bbd631e020054e Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Date: Thu, 16 Jun 2022 17:08:02 +0900
Subject: [PATCH v2] Add fileval-bootval consistency check of GUC parameters
We should keep GUC variables consistent between the default values
written in postgresql.conf.sample and defined in guc.c. Add an
automated way to check for the consistency to the TAP test suite. Some
variables are still excluded since they cannot be checked simple way.
---
src/backend/utils/misc/guc.c | 53 ++++++++++++++++
src/include/catalog/pg_proc.dat | 5 ++
src/test/modules/test_misc/t/003_check_guc.pl | 60 +++++++++++++++++--
3 files changed, 112 insertions(+), 6 deletions(-)
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index a7cc49898b..7b8b3a4417 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -7520,6 +7520,59 @@ parse_and_validate_value(struct config_generic *record,
return true;
}
+/*
+ * Convert value to unitless value according to the specified GUC variable
+ */
+Datum
+pg_config_unitless_value(PG_FUNCTION_ARGS)
+{
+ char *name = "";
+ char *value = "";
+ struct config_generic *record;
+ char *result;
+ void *extra;
+ union config_var_val val;
+ const char *p;
+ char buffer[256];
+
+ if (!PG_ARGISNULL(0))
+ name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ if (!PG_ARGISNULL(1))
+ value = text_to_cstring(PG_GETARG_TEXT_PP(1));
+
+ record = find_option(name, true, false, ERROR);
+
+ parse_and_validate_value(record, name, value, PGC_S_TEST, WARNING,
+ &val, &extra);
+
+ switch (record->vartype)
+ {
+ case PGC_BOOL:
+ result = (val.boolval ? "on" : "off");
+ break;
+ case PGC_INT:
+ snprintf(buffer, sizeof(buffer), "%d", val.intval);
+ result = pstrdup(buffer);
+ break;
+ case PGC_REAL:
+ snprintf(buffer, sizeof(buffer), "%g", val.realval);
+ result = pstrdup(buffer);
+ break;
+ case PGC_STRING:
+ p = val.stringval;
+ if (p == NULL)
+ p = "";
+ result = pstrdup(p);
+ break;
+ case PGC_ENUM:
+ p = config_enum_lookup_by_value((struct config_enum *)record,
+ val.boolval);
+ result = pstrdup(p);
+ break;
+ }
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
/*
* Sets option `name' to given value.
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 87aa571a33..9eb2a584e1 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6118,6 +6118,11 @@
proname => 'pg_settings_get_flags', provolatile => 's', prorettype => '_text',
proargtypes => 'text', prosrc => 'pg_settings_get_flags' },
+{ oid => '9956', descr => 'normalize value to the unit of specified GUC',
+ proname => 'pg_config_unitless_value', proisstrict => 'f',
+ provolatile => 's', prorettype => 'text', proargtypes => 'text text',
+ proargnames => '{varname,value}', prosrc => 'pg_config_unitless_value' },
+
{ oid => '3329', descr => 'show config file settings',
proname => 'pg_show_all_file_settings', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
diff --git a/src/test/modules/test_misc/t/003_check_guc.pl b/src/test/modules/test_misc/t/003_check_guc.pl
index 60459ef759..9de6a386de 100644
--- a/src/test/modules/test_misc/t/003_check_guc.pl
+++ b/src/test/modules/test_misc/t/003_check_guc.pl
@@ -11,18 +11,41 @@ my $node = PostgreSQL::Test::Cluster->new('main');
$node->init;
$node->start;
+# parameter names that cannot get consistency check performed
+my @ignored_parameters = (
+ 'data_directory',
+ 'hba_file',
+ 'ident_file',
+ 'krb_server_keyfile',
+ 'max_stack_depth',
+ 'bgwriter_flush_after',
+ 'wal_sync_method',
+ 'checkpoint_flush_after',
+ 'timezone_abbreviations',
+ 'lc_messages',
+ 'wal_buffers'
+ );
+
# Grab the names of all the parameters that can be listed in the
# configuration sample file. config_file is an exception, it is not
# in postgresql.conf.sample but is part of the lists from guc.c.
my $all_params = $node->safe_psql(
'postgres',
- "SELECT name
+ "SELECT lower(name), vartype, unit, boot_val, '!'
FROM pg_settings
WHERE NOT 'NOT_IN_SAMPLE' = ANY (pg_settings_get_flags(name)) AND
name <> 'config_file'
ORDER BY 1");
# Note the lower-case conversion, for consistency.
-my @all_params_array = split("\n", lc($all_params));
+my %all_params_hash;
+foreach my $line (split("\n", $all_params))
+{
+ my @f = split('\|', $line);
+ fail("query returned wrong number of columns: $#f : $line") if ($#f != 4);
+ $all_params_hash{$f[0]}->{type} = $f[1];
+ $all_params_hash{$f[0]}->{unit} = $f[2];
+ $all_params_hash{$f[0]}->{bootval} = $f[3];
+}
# Grab the names of all parameters marked as NOT_IN_SAMPLE.
my $not_in_sample = $node->safe_psql(
@@ -43,7 +66,7 @@ my @gucs_in_file;
# Read the sample file line-by-line, checking its contents to build a list
# of everything known as a GUC.
-my $num_tests = 0;
+my @check_elems = ();
open(my $contents, '<', $sample_file)
|| die "Could not open $sample_file: $!";
while (my $line = <$contents>)
@@ -53,11 +76,16 @@ while (my $line = <$contents>)
# file.
# - Valid configuration options are followed immediately by " = ",
# with one space before and after the equal sign.
- if ($line =~ m/^#?([_[:alpha:]]+) = .*/)
+ if ($line =~ m/^#?([_[:alpha:]]+) = (.*)$/)
{
# Lower-case conversion matters for some of the GUCs.
my $param_name = lc($1);
+ # extract value
+ my $file_value = $2;
+ $file_value =~ s/\s*#.*$//; # strip trailing comment
+ $file_value =~ s/^'(.*)'$/$1/; # strip quotes
+
# Ignore some exceptions.
next if $param_name eq "include";
next if $param_name eq "include_dir";
@@ -66,19 +94,39 @@ while (my $line = <$contents>)
# Update the list of GUCs found in the sample file, for the
# follow-up tests.
push @gucs_in_file, $param_name;
+
+ # Check for consistency between bootval and file value.
+ if (!grep { $_ eq $param_name } @ignored_parameters)
+ {
+ push (@check_elems, "('$param_name','$file_value')");
+ }
}
}
close $contents;
+# Run consistency check between config-file's default value and boot
+# values. To show sample setting that is not found in the view, use
+# LEFT JOIN and make sure pg_settings.name is not NULL.
+my $check_query =
+ 'SELECT f.n, f.v, s.boot_val FROM (VALUES '.
+ join(',', @check_elems).
+ ') f(n,v) LEFT JOIN pg_settings s ON lower(s.name) = f.n '.
+ "WHERE pg_config_unitless_value(f.n, f.v) <> COALESCE(s.boot_val, '') ".
+ 'OR s.name IS NULL';
+
+print $check_query;
+
+is ($node->safe_psql('postgres', $check_query), '',
+ 'check if fileval-bootval consistency is fine');
+
# Cross-check that all the GUCs found in the sample file match the ones
# fetched above. This maps the arrays to a hash, making the creation of
# each exclude and intersection list easier.
my %gucs_in_file_hash = map { $_ => 1 } @gucs_in_file;
-my %all_params_hash = map { $_ => 1 } @all_params_array;
my %not_in_sample_hash = map { $_ => 1 } @not_in_sample_array;
-my @missing_from_file = grep(!$gucs_in_file_hash{$_}, @all_params_array);
+my @missing_from_file = grep(!$gucs_in_file_hash{$_}, keys %all_params_hash);
is(scalar(@missing_from_file),
0, "no parameters missing from postgresql.conf.sample");
--
2.31.1
0001-Add-fileval-bootval-consistency-check-of-GUC-paramet-simple.patchtext/x-patch; charset=us-asciiDownload
From 4385545514099eb94c1a624d2a082e544ca9c6bc Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Date: Thu, 16 Jun 2022 17:00:14 +0900
Subject: [PATCH] Add fileval-bootval consistency check of GUC parameters
We should keep GUC variables consistent between the default values
written in postgresql.conf.sample and defined in guc.c. Add an
automated way to check for the consistency to the TAP test suite.
For simplicity we don't check values that require unit conversion.
Some variables are still excluded since they cannot be checked simple
way.
---
src/test/modules/test_misc/t/003_check_guc.pl | 76 +++++++++++++++++--
1 file changed, 70 insertions(+), 6 deletions(-)
diff --git a/src/test/modules/test_misc/t/003_check_guc.pl b/src/test/modules/test_misc/t/003_check_guc.pl
index 60459ef759..a3334fad76 100644
--- a/src/test/modules/test_misc/t/003_check_guc.pl
+++ b/src/test/modules/test_misc/t/003_check_guc.pl
@@ -11,18 +11,39 @@ my $node = PostgreSQL::Test::Cluster->new('main');
$node->init;
$node->start;
+# parameter names that cannot get consistency check performed
+my @ignored_parameters = (
+ 'data_directory',
+ 'hba_file',
+ 'ident_file',
+ 'krb_server_keyfile',
+ 'timezone_abbreviations',
+ 'lc_messages',
+ 'log_file_mode',
+ 'unix_socket_permissions',
+ 'wal_sync_method'
+ );
+
# Grab the names of all the parameters that can be listed in the
# configuration sample file. config_file is an exception, it is not
# in postgresql.conf.sample but is part of the lists from guc.c.
my $all_params = $node->safe_psql(
'postgres',
- "SELECT name
+ "SELECT lower(name), vartype, unit, boot_val, '!'
FROM pg_settings
WHERE NOT 'NOT_IN_SAMPLE' = ANY (pg_settings_get_flags(name)) AND
name <> 'config_file'
ORDER BY 1");
# Note the lower-case conversion, for consistency.
-my @all_params_array = split("\n", lc($all_params));
+my %all_params_hash;
+foreach my $line (split("\n", $all_params))
+{
+ my @f = split('\|', $line);
+ fail("query returned wrong number of columns: $#f : $line") if ($#f != 4);
+ $all_params_hash{$f[0]}->{type} = $f[1];
+ $all_params_hash{$f[0]}->{unit} = $f[2];
+ $all_params_hash{$f[0]}->{bootval} = $f[3];
+}
# Grab the names of all parameters marked as NOT_IN_SAMPLE.
my $not_in_sample = $node->safe_psql(
@@ -43,7 +64,7 @@ my @gucs_in_file;
# Read the sample file line-by-line, checking its contents to build a list
# of everything known as a GUC.
-my $num_tests = 0;
+my @check_elems = ();
open(my $contents, '<', $sample_file)
|| die "Could not open $sample_file: $!";
while (my $line = <$contents>)
@@ -53,11 +74,16 @@ while (my $line = <$contents>)
# file.
# - Valid configuration options are followed immediately by " = ",
# with one space before and after the equal sign.
- if ($line =~ m/^#?([_[:alpha:]]+) = .*/)
+ if ($line =~ m/^#?([_[:alpha:]]+) = (.*)$/)
{
# Lower-case conversion matters for some of the GUCs.
my $param_name = lc($1);
+ # extract value
+ my $file_value = $2;
+ $file_value =~ s/\s*#.*$//; # strip trailing comment
+ $file_value =~ s/^'(.*)'$/$1/; # strip quotes
+
# Ignore some exceptions.
next if $param_name eq "include";
next if $param_name eq "include_dir";
@@ -66,19 +92,57 @@ while (my $line = <$contents>)
# Update the list of GUCs found in the sample file, for the
# follow-up tests.
push @gucs_in_file, $param_name;
+
+ # Check for consistency between bootval and file value.
+ if (!grep { $_ eq $param_name } @ignored_parameters)
+ {
+ push (@check_elems, "('$param_name','$file_value')");
+ }
}
}
close $contents;
+# Run consistency check between config-file's default value and boot
+# values. For now we check only variables in string and integers
+# without unit.
+my $check_query =
+ 'SELECT f.n, f.v, s.boot_val FROM (VALUES '.
+ join(',', @check_elems).
+ ') f(n,v) LEFT JOIN pg_settings s ON lower(s.name) = f.n '.
+ "WHERE (lower(f.v) <> COALESCE(lower(s.boot_val), '') ".
+ " AND (s.vartype = 'string' OR s.vartype = 'enum'))".
+ 'OR s.name IS NULL';
+
+is ($node->safe_psql('postgres', $check_query), '',
+ 'check if fileval-bootval consistency is fine for string variables');
+
+$check_query =
+ 'SELECT f.n, f.v, s.boot_val FROM (VALUES '.
+ join(',', @check_elems).
+ ') f(n,v) LEFT JOIN pg_settings s ON lower(s.name) = f.n '.
+ "WHERE ((s.vartype = 'integer' AND s.unit IS NULL) AND f.v <> s.boot_val) ".
+ 'OR s.name IS NULL';
+is ($node->safe_psql('postgres', $check_query), '',
+ 'check if fileval-bootval consistency is fine for integer variables');
+
+$check_query =
+ 'SELECT f.n, f.v, s.boot_val FROM (VALUES '.
+ join(',', @check_elems).
+ ') f(n,v) LEFT JOIN pg_settings s ON lower(s.name) = f.n '.
+ "WHERE ((s.vartype = 'real' AND s.unit IS NULL) ".
+ " AND (f.v::real <> s.boot_val::real)) ".
+ 'OR s.name IS NULL';
+is ($node->safe_psql('postgres', $check_query), '',
+ 'check if fileval-bootval consistency is fine for real variables');
+
# Cross-check that all the GUCs found in the sample file match the ones
# fetched above. This maps the arrays to a hash, making the creation of
# each exclude and intersection list easier.
my %gucs_in_file_hash = map { $_ => 1 } @gucs_in_file;
-my %all_params_hash = map { $_ => 1 } @all_params_array;
my %not_in_sample_hash = map { $_ => 1 } @not_in_sample_array;
-my @missing_from_file = grep(!$gucs_in_file_hash{$_}, @all_params_array);
+my @missing_from_file = grep(!$gucs_in_file_hash{$_}, keys %all_params_hash);
is(scalar(@missing_from_file),
0, "no parameters missing from postgresql.conf.sample");
--
2.31.1
Import Notes
Reply to msg id not found: YqqeVyiwttFhJgA2@paquier.xyz20220611144137.GC29853@telsasoft.com
On Thu, Jun 16, 2022 at 05:19:46PM +0900, Kyotaro Horiguchi wrote:
At Sat, 11 Jun 2022 09:41:37 -0500, Justin Pryzby <pryzby@telsasoft.com> wrote in
Note that this gives:
guc.c:7573:9: warning: ‘dst’ may be used uninitialized in this function [-Wmaybe-uninitialized]
Mmm. I don't have an idea where the 'dst' came from...
Well, in your latest patch, you've renamed it.
guc.c:7586:19: warning: ‘result’ may be used uninitialized in this function [-Wmaybe-uninitialized]
7586 | PG_RETURN_TEXT_P(cstring_to_text(result));
--
Justin
At Thu, 16 Jun 2022 08:23:07 -0500, Justin Pryzby <pryzby@telsasoft.com> wrote in
On Thu, Jun 16, 2022 at 05:19:46PM +0900, Kyotaro Horiguchi wrote:
At Sat, 11 Jun 2022 09:41:37 -0500, Justin Pryzby <pryzby@telsasoft.com> wrote in
Note that this gives:
guc.c:7573:9: warning: ‘dst’ may be used uninitialized in this function [-Wmaybe-uninitialized]
Mmm. I don't have an idea where the 'dst' came from...
Well, in your latest patch, you've renamed it.
guc.c:7586:19: warning: ‘result’ may be used uninitialized in this function [-Wmaybe-uninitialized]
7586 | PG_RETURN_TEXT_P(cstring_to_text(result));
Ooo. I find that the patch on my hand was different from that on this
list by some reason uncertain to me. I now understand what's
happening.
At Sat, 11 Jun 2022 09:41:37 -0500, Justin Pryzby <pryzby@telsasoft.com> wrote in
with gcc version 9.2.1 20191008 (Ubuntu 9.2.1-9ubuntu2)
My compiler (gcc 8.5.0) (with -Wswitch) is satisfied by finding that
the switch() covers all enum values. I don't know why the new
compiler complains with this, but compilers in such environment should
shut up by the following change.
- char *result;
+ char *result = "";
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
Attachments:
v3-0001-Add-fileval-bootval-consistency-check-of-GUC-para.patchtext/x-patch; charset=us-asciiDownload
From d67c13712ab1d47956c1c311d7ed80342fa51e2f Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Date: Thu, 16 Jun 2022 17:08:02 +0900
Subject: [PATCH v3] Add fileval-bootval consistency check of GUC parameters
We should keep GUC variables consistent between the default values
written in postgresql.conf.sample and defined in guc.c. Add an
automated way to check for the consistency to the TAP test suite. Some
variables are still excluded since they cannot be checked simple way.
---
src/backend/utils/misc/guc.c | 53 ++++++++++++++++
src/include/catalog/pg_proc.dat | 5 ++
src/test/modules/test_misc/t/003_check_guc.pl | 60 +++++++++++++++++--
3 files changed, 112 insertions(+), 6 deletions(-)
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index a7cc49898b..136a21ba88 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -7520,6 +7520,59 @@ parse_and_validate_value(struct config_generic *record,
return true;
}
+/*
+ * Convert value to unitless value according to the specified GUC variable
+ */
+Datum
+pg_config_unitless_value(PG_FUNCTION_ARGS)
+{
+ char *name = "";
+ char *value = "";
+ struct config_generic *record;
+ char *result = "";
+ void *extra;
+ union config_var_val val;
+ const char *p;
+ char buffer[256];
+
+ if (!PG_ARGISNULL(0))
+ name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ if (!PG_ARGISNULL(1))
+ value = text_to_cstring(PG_GETARG_TEXT_PP(1));
+
+ record = find_option(name, true, false, ERROR);
+
+ parse_and_validate_value(record, name, value, PGC_S_TEST, WARNING,
+ &val, &extra);
+
+ switch (record->vartype)
+ {
+ case PGC_BOOL:
+ result = (val.boolval ? "on" : "off");
+ break;
+ case PGC_INT:
+ snprintf(buffer, sizeof(buffer), "%d", val.intval);
+ result = pstrdup(buffer);
+ break;
+ case PGC_REAL:
+ snprintf(buffer, sizeof(buffer), "%g", val.realval);
+ result = pstrdup(buffer);
+ break;
+ case PGC_STRING:
+ p = val.stringval;
+ if (p == NULL)
+ p = "";
+ result = pstrdup(p);
+ break;
+ case PGC_ENUM:
+ p = config_enum_lookup_by_value((struct config_enum *)record,
+ val.boolval);
+ result = pstrdup(p);
+ break;
+ }
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
/*
* Sets option `name' to given value.
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 87aa571a33..9eb2a584e1 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6118,6 +6118,11 @@
proname => 'pg_settings_get_flags', provolatile => 's', prorettype => '_text',
proargtypes => 'text', prosrc => 'pg_settings_get_flags' },
+{ oid => '9956', descr => 'normalize value to the unit of specified GUC',
+ proname => 'pg_config_unitless_value', proisstrict => 'f',
+ provolatile => 's', prorettype => 'text', proargtypes => 'text text',
+ proargnames => '{varname,value}', prosrc => 'pg_config_unitless_value' },
+
{ oid => '3329', descr => 'show config file settings',
proname => 'pg_show_all_file_settings', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
diff --git a/src/test/modules/test_misc/t/003_check_guc.pl b/src/test/modules/test_misc/t/003_check_guc.pl
index 60459ef759..9de6a386de 100644
--- a/src/test/modules/test_misc/t/003_check_guc.pl
+++ b/src/test/modules/test_misc/t/003_check_guc.pl
@@ -11,18 +11,41 @@ my $node = PostgreSQL::Test::Cluster->new('main');
$node->init;
$node->start;
+# parameter names that cannot get consistency check performed
+my @ignored_parameters = (
+ 'data_directory',
+ 'hba_file',
+ 'ident_file',
+ 'krb_server_keyfile',
+ 'max_stack_depth',
+ 'bgwriter_flush_after',
+ 'wal_sync_method',
+ 'checkpoint_flush_after',
+ 'timezone_abbreviations',
+ 'lc_messages',
+ 'wal_buffers'
+ );
+
# Grab the names of all the parameters that can be listed in the
# configuration sample file. config_file is an exception, it is not
# in postgresql.conf.sample but is part of the lists from guc.c.
my $all_params = $node->safe_psql(
'postgres',
- "SELECT name
+ "SELECT lower(name), vartype, unit, boot_val, '!'
FROM pg_settings
WHERE NOT 'NOT_IN_SAMPLE' = ANY (pg_settings_get_flags(name)) AND
name <> 'config_file'
ORDER BY 1");
# Note the lower-case conversion, for consistency.
-my @all_params_array = split("\n", lc($all_params));
+my %all_params_hash;
+foreach my $line (split("\n", $all_params))
+{
+ my @f = split('\|', $line);
+ fail("query returned wrong number of columns: $#f : $line") if ($#f != 4);
+ $all_params_hash{$f[0]}->{type} = $f[1];
+ $all_params_hash{$f[0]}->{unit} = $f[2];
+ $all_params_hash{$f[0]}->{bootval} = $f[3];
+}
# Grab the names of all parameters marked as NOT_IN_SAMPLE.
my $not_in_sample = $node->safe_psql(
@@ -43,7 +66,7 @@ my @gucs_in_file;
# Read the sample file line-by-line, checking its contents to build a list
# of everything known as a GUC.
-my $num_tests = 0;
+my @check_elems = ();
open(my $contents, '<', $sample_file)
|| die "Could not open $sample_file: $!";
while (my $line = <$contents>)
@@ -53,11 +76,16 @@ while (my $line = <$contents>)
# file.
# - Valid configuration options are followed immediately by " = ",
# with one space before and after the equal sign.
- if ($line =~ m/^#?([_[:alpha:]]+) = .*/)
+ if ($line =~ m/^#?([_[:alpha:]]+) = (.*)$/)
{
# Lower-case conversion matters for some of the GUCs.
my $param_name = lc($1);
+ # extract value
+ my $file_value = $2;
+ $file_value =~ s/\s*#.*$//; # strip trailing comment
+ $file_value =~ s/^'(.*)'$/$1/; # strip quotes
+
# Ignore some exceptions.
next if $param_name eq "include";
next if $param_name eq "include_dir";
@@ -66,19 +94,39 @@ while (my $line = <$contents>)
# Update the list of GUCs found in the sample file, for the
# follow-up tests.
push @gucs_in_file, $param_name;
+
+ # Check for consistency between bootval and file value.
+ if (!grep { $_ eq $param_name } @ignored_parameters)
+ {
+ push (@check_elems, "('$param_name','$file_value')");
+ }
}
}
close $contents;
+# Run consistency check between config-file's default value and boot
+# values. To show sample setting that is not found in the view, use
+# LEFT JOIN and make sure pg_settings.name is not NULL.
+my $check_query =
+ 'SELECT f.n, f.v, s.boot_val FROM (VALUES '.
+ join(',', @check_elems).
+ ') f(n,v) LEFT JOIN pg_settings s ON lower(s.name) = f.n '.
+ "WHERE pg_config_unitless_value(f.n, f.v) <> COALESCE(s.boot_val, '') ".
+ 'OR s.name IS NULL';
+
+print $check_query;
+
+is ($node->safe_psql('postgres', $check_query), '',
+ 'check if fileval-bootval consistency is fine');
+
# Cross-check that all the GUCs found in the sample file match the ones
# fetched above. This maps the arrays to a hash, making the creation of
# each exclude and intersection list easier.
my %gucs_in_file_hash = map { $_ => 1 } @gucs_in_file;
-my %all_params_hash = map { $_ => 1 } @all_params_array;
my %not_in_sample_hash = map { $_ => 1 } @not_in_sample_array;
-my @missing_from_file = grep(!$gucs_in_file_hash{$_}, @all_params_array);
+my @missing_from_file = grep(!$gucs_in_file_hash{$_}, keys %all_params_hash);
is(scalar(@missing_from_file),
0, "no parameters missing from postgresql.conf.sample");
--
2.31.1
Import Notes
Reply to msg id not found: 20220616132306.GD29853@telsasoft.com20220611144137.GC29853@telsasoft.com
Hi,
On 2022-06-17 09:43:58 +0900, Kyotaro Horiguchi wrote:
+/* + * Convert value to unitless value according to the specified GUC variable + */ +Datum +pg_config_unitless_value(PG_FUNCTION_ARGS) +{ + char *name = ""; + char *value = ""; + struct config_generic *record; + char *result = ""; + void *extra; + union config_var_val val; + const char *p; + char buffer[256]; + + if (!PG_ARGISNULL(0)) + name = text_to_cstring(PG_GETARG_TEXT_PP(0)); + if (!PG_ARGISNULL(1)) + value = text_to_cstring(PG_GETARG_TEXT_PP(1)); + + record = find_option(name, true, false, ERROR); + + parse_and_validate_value(record, name, value, PGC_S_TEST, WARNING, + &val, &extra); +
Hm. I think this should error out for options that the user doesn't have the
permissions for - good. I suggest adding a test for that.
+ switch (record->vartype) + { + case PGC_BOOL: + result = (val.boolval ? "on" : "off"); + break; + case PGC_INT: + snprintf(buffer, sizeof(buffer), "%d", val.intval); + result = pstrdup(buffer); + break; + case PGC_REAL: + snprintf(buffer, sizeof(buffer), "%g", val.realval); + result = pstrdup(buffer); + break; + case PGC_STRING: + p = val.stringval; + if (p == NULL) + p = ""; + result = pstrdup(p); + break;
Is this a good idea? I wonder if we shouldn't instead return NULL, rather than
making NULL and "" undistinguishable.
Not that it matters for efficiency here, but why are you pstrdup'ing the
buffers? cstring_to_text() will already copy the string, no?
+# parameter names that cannot get consistency check performed +my @ignored_parameters = (
Perhaps worth adding comments explaining why these can't get checked?
+foreach my $line (split("\n", $all_params)) +{ + my @f = split('\|', $line); + fail("query returned wrong number of columns: $#f : $line") if ($#f != 4); + $all_params_hash{$f[0]}->{type} = $f[1]; + $all_params_hash{$f[0]}->{unit} = $f[2]; + $all_params_hash{$f[0]}->{bootval} = $f[3]; +}
Might look a bit nicer to generate the hash in a local variable and then
assign to $all_params_hash{$f[0]} once, rather than repeating that part
multiple times.
- if ($line =~ m/^#?([_[:alpha:]]+) = .*/) + if ($line =~ m/^#?([_[:alpha:]]+) = (.*)$/) { # Lower-case conversion matters for some of the GUCs. my $param_name = lc($1);+ # extract value + my $file_value = $2; + $file_value =~ s/\s*#.*$//; # strip trailing comment + $file_value =~ s/^'(.*)'$/$1/; # strip quotes + # Ignore some exceptions. next if $param_name eq "include"; next if $param_name eq "include_dir";
So there's now two ignore mechanisms? Why not just handle include[_dir] via
@ignored_parameters?
@@ -66,19 +94,39 @@ while (my $line = <$contents>) # Update the list of GUCs found in the sample file, for the # follow-up tests. push @gucs_in_file, $param_name; + + # Check for consistency between bootval and file value.
You're not checking the consistency here though?
+ if (!grep { $_ eq $param_name } @ignored_parameters) + { + push (@check_elems, "('$param_name','$file_value')"); + } } }
close $contents;
+# Run consistency check between config-file's default value and boot +# values. To show sample setting that is not found in the view, use +# LEFT JOIN and make sure pg_settings.name is not NULL. +my $check_query = + 'SELECT f.n, f.v, s.boot_val FROM (VALUES '. + join(',', @check_elems). + ') f(n,v) LEFT JOIN pg_settings s ON lower(s.name) = f.n '. + "WHERE pg_config_unitless_value(f.n, f.v) <> COALESCE(s.boot_val, '') ". + 'OR s.name IS NULL'; + +print $check_query; + +is ($node->safe_psql('postgres', $check_query), '', + 'check if fileval-bootval consistency is fine');
"fileval-bootval" isn't that easy to understand, "is fine" doesn't quite sound
right. Maybe something like "GUC values in .sample and boot value match"?
I wonder if it'd not result in easier to understand output if the query just
called pg_config_unitless_value() for all the .sample values, but then did the
comparison of the results in perl.
Greetings,
Andres Freund
Thanks!
At Wed, 22 Jun 2022 16:07:10 -0700, Andres Freund <andres@anarazel.de> wrote in
Hi,
On 2022-06-17 09:43:58 +0900, Kyotaro Horiguchi wrote:
+/* + * Convert value to unitless value according to the specified GUC variable + */ +Datum +pg_config_unitless_value(PG_FUNCTION_ARGS) +{
...
+ record = find_option(name, true, false, ERROR); + + parse_and_validate_value(record, name, value, PGC_S_TEST, WARNING, + &val, &extra); +Hm. I think this should error out for options that the user doesn't have the
permissions for - good. I suggest adding a test for that.
Generally sounds reasonable but it doesn't reveal its setting. It
just translates (or decodes) given string to internal value. And
currently most of all values are string and only two are enum (TLS
version), that are returned almost as-is. That being said, the
suggested behavior seems better. So I did that in the attached.
And I added the test for this to rolenames in modules/unsafe_tests.
+ switch (record->vartype) + { + case PGC_BOOL: + result = (val.boolval ? "on" : "off"); + break; + case PGC_INT: + snprintf(buffer, sizeof(buffer), "%d", val.intval); + result = pstrdup(buffer); + break; + case PGC_REAL: + snprintf(buffer, sizeof(buffer), "%g", val.realval); + result = pstrdup(buffer); + break; + case PGC_STRING: + p = val.stringval; + if (p == NULL) + p = ""; + result = pstrdup(p); + break;Is this a good idea? I wonder if we shouldn't instead return NULL, rather than
making NULL and "" undistinguishable.
Anyway NULL cannot be seen there and I don't recall the reason I made
the function non-strict. I changed the SQL function back to 'strict',
which makes things cleaner and simpler.
Not that it matters for efficiency here, but why are you pstrdup'ing the
buffers? cstring_to_text() will already copy the string, no?
Right. That's a silly thinko, omitting that behavior..
+# parameter names that cannot get consistency check performed +my @ignored_parameters = (Perhaps worth adding comments explaining why these can't get checked?
Mmm. I agree. I rewrote it as the follows.
# The following parameters are defaultly set with
# environment-dependent values which may not match the default values
# written in the sample config file.
+foreach my $line (split("\n", $all_params)) +{ + my @f = split('\|', $line); + fail("query returned wrong number of columns: $#f : $line") if ($#f != 4); + $all_params_hash{$f[0]}->{type} = $f[1]; + $all_params_hash{$f[0]}->{unit} = $f[2]; + $all_params_hash{$f[0]}->{bootval} = $f[3]; +}Might look a bit nicer to generate the hash in a local variable and then
assign to $all_params_hash{$f[0]} once, rather than repeating that part
multiple times.
Yeah, but I noticed that that hash is no longer needed..
- if ($line =~ m/^#?([_[:alpha:]]+) = .*/) + if ($line =~ m/^#?([_[:alpha:]]+) = (.*)$/) { # Lower-case conversion matters for some of the GUCs. my $param_name = lc($1);+ # extract value + my $file_value = $2; + $file_value =~ s/\s*#.*$//; # strip trailing comment + $file_value =~ s/^'(.*)'$/$1/; # strip quotes + # Ignore some exceptions. next if $param_name eq "include"; next if $param_name eq "include_dir";So there's now two ignore mechanisms? Why not just handle include[_dir] via
@ignored_parameters?
The two ignore mechanisms work for different arrays. So we need to
distinct between the two uses. I tried that but it looks like
reseparating particles that were uselessly mixed. Finally I changed
the variable to hash and apply the same mechanism to "include" and
friends but by using different hash.
@@ -66,19 +94,39 @@ while (my $line = <$contents>) # Update the list of GUCs found in the sample file, for the # follow-up tests. push @gucs_in_file, $param_name; + + # Check for consistency between bootval and file value.You're not checking the consistency here though?
Mmm. Right. I reworded it following the comment just above.
+ if (!grep { $_ eq $param_name } @ignored_parameters) + { + push (@check_elems, "('$param_name','$file_value')"); + } } }close $contents;
+# Run consistency check between config-file's default value and boot +# values. To show sample setting that is not found in the view, use +# LEFT JOIN and make sure pg_settings.name is not NULL. +my $check_query = + 'SELECT f.n, f.v, s.boot_val FROM (VALUES '. + join(',', @check_elems). + ') f(n,v) LEFT JOIN pg_settings s ON lower(s.name) = f.n '. + "WHERE pg_config_unitless_value(f.n, f.v) <> COALESCE(s.boot_val, '') ". + 'OR s.name IS NULL'; + +print $check_query; + +is ($node->safe_psql('postgres', $check_query), '', + 'check if fileval-bootval consistency is fine');"fileval-bootval" isn't that easy to understand, "is fine" doesn't quite sound
right. Maybe something like "GUC values in .sample and boot value match"?
No objection. Changed.
I wonder if it'd not result in easier to understand output if the query just
called pg_config_unitless_value() for all the .sample values, but then did the
comparison of the results in perl.
It is a fair alternative. I said exactly the same thing (perl is
easier to understand than the same (procedual) logic in SQL)
upthread:p So done that in the attached.
I tempted to find extra filevals by the code added here but it is
cleaner to leave it to the existing checking code.
- Change the behavior of pg_config_unitless_value according to the comment.
- and added the test for the function's behavior about privileges.
- Skip "include" and friends by using a hash similar to ignore_parameters.
- Removed %all_params_hash. (Currently it is @file_vals)
- A comment reworded (but it donesn't look fine..).
- Moved value-check logic from SQL to perl.
And I'll add this to the coming CF.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
Attachments:
v4-0001-Add-fileval-bootval-consistency-check-of-GUC-para.patchtext/x-patch; charset=us-asciiDownload
From 8ab61286529f819a08e0e70d0b5522cabf4592b1 Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Date: Thu, 16 Jun 2022 17:08:02 +0900
Subject: [PATCH v4] Add fileval-bootval consistency check of GUC parameters
We should keep GUC variables consistent between the default values
written in postgresql.conf.sample and defined in guc.c. Add an
automated way to check for the consistency to the TAP test suite. Some
variables are still excluded since they cannot be checked simple way.
---
src/backend/utils/misc/guc.c | 57 ++++++++++++++++
src/include/catalog/pg_proc.dat | 5 ++
src/test/modules/test_misc/t/003_check_guc.pl | 67 +++++++++++++++++--
.../unsafe_tests/expected/rolenames.out | 13 ++++
.../modules/unsafe_tests/sql/rolenames.sql | 7 ++
5 files changed, 143 insertions(+), 6 deletions(-)
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index a7cc49898b..1e7a0a2edc 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -7520,6 +7520,63 @@ parse_and_validate_value(struct config_generic *record,
return true;
}
+/*
+ * Convert value to unitless value according to the specified GUC variable
+ */
+Datum
+pg_config_unitless_value(PG_FUNCTION_ARGS)
+{
+ char *name;
+ char *value;
+ struct config_generic *record;
+ const char *result = "";
+ void *extra;
+ union config_var_val val;
+ char buffer[256];
+
+ name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ value = text_to_cstring(PG_GETARG_TEXT_PP(1));
+
+ record = find_option(name, true, false, ERROR);
+
+ /*
+ * This function doesn't reveal values of the variables, but be consistent
+ * with similar functions.
+ */
+ if ((record->flags & GUC_SUPERUSER_ONLY) &&
+ !has_privs_of_role(GetUserId(), ROLE_PG_READ_ALL_SETTINGS))
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+ errmsg("must be superuser or have privileges of pg_read_all_settings to convert value for \"%s\"",
+ name)));
+
+ parse_and_validate_value(record, name, value, PGC_S_TEST, WARNING,
+ &val, &extra);
+
+ switch (record->vartype)
+ {
+ case PGC_BOOL:
+ result = (val.boolval ? "on" : "off");
+ break;
+ case PGC_INT:
+ snprintf(buffer, sizeof(buffer), "%d", val.intval);
+ result = buffer;
+ break;
+ case PGC_REAL:
+ snprintf(buffer, sizeof(buffer), "%g", val.realval);
+ result = buffer;
+ break;
+ case PGC_STRING:
+ result = val.stringval;
+ break;
+ case PGC_ENUM:
+ result = config_enum_lookup_by_value((struct config_enum *)record,
+ val.boolval);
+ break;
+ }
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
/*
* Sets option `name' to given value.
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 87aa571a33..794ba12dae 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6118,6 +6118,11 @@
proname => 'pg_settings_get_flags', provolatile => 's', prorettype => '_text',
proargtypes => 'text', prosrc => 'pg_settings_get_flags' },
+{ oid => '9956', descr => 'normalize value to the unit of specified GUC',
+ proname => 'pg_config_unitless_value', provolatile => 's',
+ prorettype => 'text', proargtypes => 'text text',
+ proargnames => '{varname,value}', prosrc => 'pg_config_unitless_value' },
+
{ oid => '3329', descr => 'show config file settings',
proname => 'pg_show_all_file_settings', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
diff --git a/src/test/modules/test_misc/t/003_check_guc.pl b/src/test/modules/test_misc/t/003_check_guc.pl
index 60459ef759..d213b3b2bc 100644
--- a/src/test/modules/test_misc/t/003_check_guc.pl
+++ b/src/test/modules/test_misc/t/003_check_guc.pl
@@ -11,6 +11,28 @@ my $node = PostgreSQL::Test::Cluster->new('main');
$node->init;
$node->start;
+# These are non-variables but that are mistakenly parsed as variable
+# settings in the loop below.
+my %skip_names =
+ map { $_ => 1 } ('include', 'include_dir', 'include_if_exists');
+
+# The following parameters are defaultly set with
+# environment-dependent values at run-time which may not match the
+# default values written in the sample config file.
+my %ignore_parameters =
+ map { $_ => 1 } (
+ 'data_directory',
+ 'hba_file',
+ 'ident_file',
+ 'krb_server_keyfile',
+ 'max_stack_depth',
+ 'bgwriter_flush_after',
+ 'wal_sync_method',
+ 'checkpoint_flush_after',
+ 'timezone_abbreviations',
+ 'lc_messages',
+ 'wal_buffers');
+
# Grab the names of all the parameters that can be listed in the
# configuration sample file. config_file is an exception, it is not
# in postgresql.conf.sample but is part of the lists from guc.c.
@@ -43,7 +65,7 @@ my @gucs_in_file;
# Read the sample file line-by-line, checking its contents to build a list
# of everything known as a GUC.
-my $num_tests = 0;
+my @file_vals = ();
open(my $contents, '<', $sample_file)
|| die "Could not open $sample_file: $!";
while (my $line = <$contents>)
@@ -53,19 +75,29 @@ while (my $line = <$contents>)
# file.
# - Valid configuration options are followed immediately by " = ",
# with one space before and after the equal sign.
- if ($line =~ m/^#?([_[:alpha:]]+) = .*/)
+ if ($line =~ m/^#?([_[:alpha:]]+) = (.*)$/)
{
# Lower-case conversion matters for some of the GUCs.
my $param_name = lc($1);
- # Ignore some exceptions.
- next if $param_name eq "include";
- next if $param_name eq "include_dir";
- next if $param_name eq "include_if_exists";
+ # extract value
+ my $file_value = $2;
+ $file_value =~ s/\s*#.*$//; # strip trailing comment
+ $file_value =~ s/^'(.*)'$/$1/; # strip quotes
+
+ next if (defined $skip_names{$param_name});
# Update the list of GUCs found in the sample file, for the
# follow-up tests.
push @gucs_in_file, $param_name;
+
+ # Update the list of GUCs that value check between the sample
+ # file and pg_setting.boot_val will be performed.
+ if (!defined $ignore_parameters{$param_name})
+ {
+ push(@file_vals, [$param_name, $file_value]);
+ }
+
}
}
@@ -107,4 +139,27 @@ foreach my $param (@sample_intersect)
);
}
+# Check if GUC values in config-file and boot value match
+my $values = $node->safe_psql(
+ 'postgres',
+ 'SELECT f.n, pg_config_unitless_value(f.n, f.v), s.boot_val, \'!\' '.
+ 'FROM (VALUES '.
+ join(',', map { "('${$_}[0]','${$_}[1]')" } @file_vals).
+ ') f(n,v) '.
+ 'JOIN pg_settings s ON (s.name = f.n)');
+
+my $fails = "";
+foreach my $l (split("\n", $values))
+{
+ # $l: <varname>|<fileval>|<boot_val>|!
+ my @t = split("\\|", $l);
+ if ($t[1] ne $t[2])
+ {
+ $fails .= "\n" if ($fails ne "");
+ $fails .= "$t[0]: file \"$t[1]\" != boot_val \"$t[2]\"";
+ }
+}
+
+is($fails, "", "check if GUC values in .sample and boot value match");
+
done_testing();
diff --git a/src/test/modules/unsafe_tests/expected/rolenames.out b/src/test/modules/unsafe_tests/expected/rolenames.out
index 88b1ff843b..17983005d8 100644
--- a/src/test/modules/unsafe_tests/expected/rolenames.out
+++ b/src/test/modules/unsafe_tests/expected/rolenames.out
@@ -1081,6 +1081,19 @@ ERROR: must be superuser or have privileges of pg_read_all_settings to examine
RESET SESSION AUTHORIZATION;
ERROR: current transaction is aborted, commands ignored until end of transaction block
ROLLBACK;
+BEGIN;
+SET SESSION AUTHORIZATION regress_role_haspriv;
+-- passes with role member of pg_read_all_settings
+SELECT pg_config_unitless_value('session_preload_libraries', 'val');
+ pg_config_unitless_value
+--------------------------
+ val
+(1 row)
+
+SET SESSION AUTHORIZATION regress_role_nopriv;
+SELECT pg_config_unitless_value('session_preload_libraries', 'val');
+ERROR: must be superuser or have privileges of pg_read_all_settings to convert value for "session_preload_libraries"
+ROLLBACK;
REVOKE pg_read_all_settings FROM regress_role_haspriv;
-- clean up
\c
diff --git a/src/test/modules/unsafe_tests/sql/rolenames.sql b/src/test/modules/unsafe_tests/sql/rolenames.sql
index adac36536d..355aa32c2a 100644
--- a/src/test/modules/unsafe_tests/sql/rolenames.sql
+++ b/src/test/modules/unsafe_tests/sql/rolenames.sql
@@ -492,6 +492,13 @@ SET SESSION AUTHORIZATION regress_role_nopriv;
SHOW session_preload_libraries;
RESET SESSION AUTHORIZATION;
ROLLBACK;
+BEGIN;
+SET SESSION AUTHORIZATION regress_role_haspriv;
+-- passes with role member of pg_read_all_settings
+SELECT pg_config_unitless_value('session_preload_libraries', 'val');
+SET SESSION AUTHORIZATION regress_role_nopriv;
+SELECT pg_config_unitless_value('session_preload_libraries', 'val');
+ROLLBACK;
REVOKE pg_read_all_settings FROM regress_role_haspriv;
-- clean up
--
2.31.1
+# The following parameters are defaultly set with +# environment-dependent values at run-time which may not match the +# default values written in the sample config file. +my %ignore_parameters = + map { $_ => 1 } ( + 'data_directory', + 'hba_file', + 'ident_file', + 'krb_server_keyfile', + 'max_stack_depth', + 'bgwriter_flush_after', + 'wal_sync_method', + 'checkpoint_flush_after', + 'timezone_abbreviations', + 'lc_messages', + 'wal_buffers');
How did you make this list ? Was it by excluding things that failed for you ?
cfbot is currently failing due to io_concurrency on windows.
I think there are more GUC which should be included here.
http://cfbot.cputube.org/kyotaro-horiguchi.html
--
Justin
On Wed, Jul 13, 2022 at 12:30:00PM -0500, Justin Pryzby wrote:
How did you make this list ? Was it by excluding things that failed for you ?
cfbot is currently failing due to io_concurrency on windows.
I think there are more GUC which should be included here.
FWIW, I am not really a fan of making this test depend on a hardcoded
list of GUCs. The design strength of the existing test is that we
don't have such a dependency now, making less to think about in terms
of maintenance in the long-term, even if this is now run
automatically.
--
Michael
Hi,
On 2022-07-14 08:46:02 +0900, Michael Paquier wrote:
On Wed, Jul 13, 2022 at 12:30:00PM -0500, Justin Pryzby wrote:
How did you make this list ? Was it by excluding things that failed for you ?
cfbot is currently failing due to io_concurrency on windows.
I think there are more GUC which should be included here.FWIW, I am not really a fan of making this test depend on a hardcoded
list of GUCs.
I wonder if we should add flags indicating platform dependency etc to guc.c?
That should allow to remove most of them?
The design strength of the existing test is that we
don't have such a dependency now, making less to think about in terms
of maintenance in the long-term, even if this is now run
automatically.
There's no existing test for things covered by these exceptions, unless I am
missing something?
Greetings,
Andres Freund
On Thu, Jul 14, 2022 at 08:46:02AM +0900, Michael Paquier wrote:
On Wed, Jul 13, 2022 at 12:30:00PM -0500, Justin Pryzby wrote:
How did you make this list ? Was it by excluding things that failed for you ?
cfbot is currently failing due to io_concurrency on windows.
I think there are more GUC which should be included here.FWIW, I am not really a fan of making this test depend on a hardcoded
list of GUCs. The design strength of the existing test is that we
don't have such a dependency now, making less to think about in terms
of maintenance in the long-term, even if this is now run
automatically.
It doesn't really need to be stated that an inclusive list wouldn't be useful.
That's a list of GUCs to be excluded.
Which is hardly different from the pre-existing list of exceptions.
# Ignore some exceptions.
next if $param_name eq "include";
next if $param_name eq "include_dir";
next if $param_name eq "include_if_exists";
-- Exceptions are transaction_*.
SELECT name FROM tab_settings_flags
WHERE NOT no_show_all AND no_reset_all
ORDER BY 1;
name
------------------------
transaction_deferrable
transaction_isolation
transaction_read_only
(3 rows)
How else do you propose to make this work for guc whose defaults vary by
platform in guc.c or in initdb ?
--
Justin
At Wed, 13 Jul 2022 18:54:45 -0500, Justin Pryzby <pryzby@telsasoft.com> wrote in
On Thu, Jul 14, 2022 at 08:46:02AM +0900, Michael Paquier wrote:
On Wed, Jul 13, 2022 at 12:30:00PM -0500, Justin Pryzby wrote:
How did you make this list ? Was it by excluding things that failed for you ?
Yes. I didn't confirm each variable. They are the variables differ on
RHEL-family OSes. io_concurrency differs according to
USE_PREFETCH. Regarding to effects of macro definitions, I searched
guc.c for non-GUC_NOT_IN_SAMPLE variables with macro-affected defaults.
WIN32 affects update_process_title
USE_PREFETCH affects effective_io_concurrency and maintenance_io_concurrency
HAVE_UNIX_SOCKETS affects unix_socket_directories and unix_socket_directories
USE_SSL affects ssl_ecdh_curve
USE_OPENSSL affects ssl_ciphers
HAVE_SYSLOG affects syslog_facility
Different from most of the variables already in the exclusion list,
these could be changed at build time, but I haven't found a sensible
way to do that. Otherwise we need to add them to the exclusion list...
cfbot is currently failing due to io_concurrency on windows.
I think there are more GUC which should be included here.FWIW, I am not really a fan of making this test depend on a hardcoded
list of GUCs. The design strength of the existing test is that we
don't have such a dependency now, making less to think about in terms
of maintenance in the long-term, even if this is now run
automatically.It doesn't really need to be stated that an inclusive list wouldn't be useful.
+1
That's a list of GUCs to be excluded.
Which is hardly different from the pre-existing list of exceptions.# Ignore some exceptions.
next if $param_name eq "include";
next if $param_name eq "include_dir";
next if $param_name eq "include_if_exists";-- Exceptions are transaction_*.
SELECT name FROM tab_settings_flags
WHERE NOT no_show_all AND no_reset_all
ORDER BY 1;
name
------------------------
transaction_deferrable
transaction_isolation
transaction_read_only
(3 rows)How else do you propose to make this work for guc whose defaults vary by
platform in guc.c or in initdb ?
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
On Tue, Jul 19, 2022 at 03:04:27PM +0900, Kyotaro Horiguchi wrote:
At Wed, 13 Jul 2022 18:54:45 -0500, Justin Pryzby <pryzby@telsasoft.com> wrote in
On Thu, Jul 14, 2022 at 08:46:02AM +0900, Michael Paquier wrote:
On Wed, Jul 13, 2022 at 12:30:00PM -0500, Justin Pryzby wrote:
How did you make this list ? Was it by excluding things that failed for you ?
Yes. I didn't confirm each variable. They are the variables differ on
RHEL-family OSes. io_concurrency differs according to
USE_PREFETCH. Regarding to effects of macro definitions, I searched
guc.c for non-GUC_NOT_IN_SAMPLE variables with macro-affected defaults.
I think you'd also need to handle the ones which are changed by initdb.c.
This patch takes Andres' suggestion.
The list of GUCs I flagged is probably incomplete, maybe inaccurate, and at
least up for discussion.
BTW I still think it might have been better to leave pg_settings_get_flags()
deliberately undocumented.
--
Justin
Attachments:
0001-wip-add-DYNAMIC_DEFAULT-for-settings-which-are-vary-.patchtext/x-diff; charset=us-asciiDownload
From 5f9a56a9c0d73a221e55b2c02e006d3104f4937b Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Tue, 19 Jul 2022 08:31:56 -0500
Subject: [PATCH 1/2] wip: add DYNAMIC_DEFAULT for settings which are vary by
configure flags or platform or initdb
---
doc/src/sgml/func.sgml | 4 ++++
src/backend/utils/misc/guc.c | 36 +++++++++++++++++++++++-------------
src/include/utils/guc.h | 6 ++++--
3 files changed, 31 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index e3e24e185a4..0fd6234eaf2 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -24977,6 +24977,10 @@ SELECT currval(pg_get_serial_sequence('sometable', 'id'));
an empty array if the GUC exists but there are no flags to show.
Only the most useful flags are exposed, as of the following:
<simplelist>
+ <member>
+ <literal>DYNAMIC_DEFAULT</literal>: parameters whose default varies by
+ platform, or compile-time options, or initdb.
+ </member>
<member>
<literal>EXPLAIN</literal>: parameters included in
<command>EXPLAIN (SETTINGS)</command> commands.
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 0328029d430..dbbadc5a475 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -1435,7 +1435,7 @@ static struct config_bool ConfigureNamesBool[] =
{"debug_assertions", PGC_INTERNAL, PRESET_OPTIONS,
gettext_noop("Shows whether the running server has assertion checks enabled."),
NULL,
- GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE
+ GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE | GUC_DYNAMIC_DEFAULT
},
&assert_enabled,
#ifdef USE_ASSERT_CHECKING
@@ -1611,7 +1611,8 @@ static struct config_bool ConfigureNamesBool[] =
{
{"update_process_title", PGC_SUSET, PROCESS_TITLE,
gettext_noop("Updates the process title to show the active SQL command."),
- gettext_noop("Enables updating of the process title every time a new SQL command is received by the server.")
+ gettext_noop("Enables updating of the process title every time a new SQL command is received by the server."),
+ GUC_DYNAMIC_DEFAULT
},
&update_process_title,
#ifdef WIN32
@@ -2362,6 +2363,7 @@ static struct config_int ConfigureNamesInt[] =
gettext_noop("Sets the maximum number of concurrent connections."),
NULL
},
+
&MaxConnections,
100, 1, MAX_BACKENDS,
check_maxconnections, NULL, NULL
@@ -2397,7 +2399,7 @@ static struct config_int ConfigureNamesInt[] =
{"shared_buffers", PGC_POSTMASTER, RESOURCES_MEM,
gettext_noop("Sets the number of shared memory buffers used by the server."),
NULL,
- GUC_UNIT_BLOCKS
+ GUC_UNIT_BLOCKS | GUC_DYNAMIC_DEFAULT
},
&NBuffers,
16384, 16, INT_MAX / 2,
@@ -3142,7 +3144,7 @@ static struct config_int ConfigureNamesInt[] =
RESOURCES_ASYNCHRONOUS,
gettext_noop("Number of simultaneous requests that can be handled efficiently by the disk subsystem."),
NULL,
- GUC_EXPLAIN
+ GUC_EXPLAIN|GUC_DYNAMIC_DEFAULT
},
&effective_io_concurrency,
#ifdef USE_PREFETCH
@@ -3160,7 +3162,7 @@ static struct config_int ConfigureNamesInt[] =
RESOURCES_ASYNCHRONOUS,
gettext_noop("A variant of effective_io_concurrency that is used for maintenance work."),
NULL,
- GUC_EXPLAIN
+ GUC_EXPLAIN|GUC_DYNAMIC_DEFAULT
},
&maintenance_io_concurrency,
#ifdef USE_PREFETCH
@@ -3327,9 +3329,11 @@ static struct config_int ConfigureNamesInt[] =
gettext_noop("Shows the size of write ahead log segments."),
NULL,
GUC_UNIT_BYTE | GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE | GUC_RUNTIME_COMPUTED
+
},
&wal_segment_size,
DEFAULT_XLOG_SEG_SIZE,
+
WalSegMinSize,
WalSegMaxSize,
NULL, NULL, NULL
@@ -3523,6 +3527,7 @@ static struct config_int ConfigureNamesInt[] =
GUC_UNIT_BLOCKS | GUC_EXPLAIN,
},
&effective_cache_size,
+
DEFAULT_EFFECTIVE_CACHE_SIZE, 1, INT_MAX,
NULL, NULL, NULL
},
@@ -3620,7 +3625,7 @@ static struct config_int ConfigureNamesInt[] =
{"debug_discard_caches", PGC_SUSET, DEVELOPER_OPTIONS,
gettext_noop("Aggressively flush system caches for debugging purposes."),
NULL,
- GUC_NOT_IN_SAMPLE
+ GUC_NOT_IN_SAMPLE | GUC_DYNAMIC_DEFAULT
},
&debug_discard_caches,
#ifdef DISCARD_CACHES_ENABLED
@@ -4447,7 +4452,7 @@ static struct config_string ConfigureNamesString[] =
{"unix_socket_directories", PGC_POSTMASTER, CONN_AUTH_SETTINGS,
gettext_noop("Sets the directories where Unix-domain sockets will be created."),
NULL,
- GUC_LIST_INPUT | GUC_LIST_QUOTE | GUC_SUPERUSER_ONLY
+ GUC_LIST_INPUT | GUC_LIST_QUOTE | GUC_SUPERUSER_ONLY | GUC_DYNAMIC_DEFAULT
},
&Unix_socket_directories,
#ifdef HAVE_UNIX_SOCKETS
@@ -4532,7 +4537,7 @@ static struct config_string ConfigureNamesString[] =
{"ssl_library", PGC_INTERNAL, PRESET_OPTIONS,
gettext_noop("Shows the name of the SSL library."),
NULL,
- GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE
+ GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE | GUC_DYNAMIC_DEFAULT
},
&ssl_library,
#ifdef USE_SSL
@@ -4618,7 +4623,7 @@ static struct config_string ConfigureNamesString[] =
{"ssl_ciphers", PGC_SIGHUP, CONN_AUTH_SSL,
gettext_noop("Sets the list of allowed SSL ciphers."),
NULL,
- GUC_SUPERUSER_ONLY
+ GUC_SUPERUSER_ONLY | GUC_DYNAMIC_DEFAULT
},
&SSLCipherSuites,
#ifdef USE_OPENSSL
@@ -4633,7 +4638,7 @@ static struct config_string ConfigureNamesString[] =
{"ssl_ecdh_curve", PGC_SIGHUP, CONN_AUTH_SSL,
gettext_noop("Sets the curve to use for ECDH."),
NULL,
- GUC_SUPERUSER_ONLY
+ GUC_SUPERUSER_ONLY | GUC_DYNAMIC_DEFAULT
},
&SSLECDHCurve,
#ifdef USE_SSL
@@ -4871,7 +4876,8 @@ static struct config_enum ConfigureNamesEnum[] =
{
{"syslog_facility", PGC_SIGHUP, LOGGING_WHERE,
gettext_noop("Sets the syslog \"facility\" to be used when syslog enabled."),
- NULL
+ NULL,
+ GUC_DYNAMIC_DEFAULT
},
&syslog_facility,
#ifdef HAVE_SYSLOG
@@ -4984,7 +4990,8 @@ static struct config_enum ConfigureNamesEnum[] =
{
{"dynamic_shared_memory_type", PGC_POSTMASTER, RESOURCES_MEM,
gettext_noop("Selects the dynamic shared memory implementation used."),
- NULL
+ NULL,
+ GUC_DYNAMIC_DEFAULT
},
&dynamic_shared_memory_type,
DEFAULT_DYNAMIC_SHARED_MEMORY_TYPE, dynamic_shared_memory_options,
@@ -5008,6 +5015,7 @@ static struct config_enum ConfigureNamesEnum[] =
},
&sync_method,
DEFAULT_SYNC_METHOD, sync_method_options,
+
NULL, assign_xlog_sync_method, NULL
},
@@ -9943,7 +9951,7 @@ GetConfigOptionByName(const char *name, const char **varname, bool missing_ok)
Datum
pg_settings_get_flags(PG_FUNCTION_ARGS)
{
-#define MAX_GUC_FLAGS 5
+#define MAX_GUC_FLAGS 6
char *varname = TextDatumGetCString(PG_GETARG_DATUM(0));
struct config_generic *record;
int cnt = 0;
@@ -9956,6 +9964,8 @@ pg_settings_get_flags(PG_FUNCTION_ARGS)
if (record == NULL)
PG_RETURN_NULL();
+ if (record->flags & GUC_DYNAMIC_DEFAULT)
+ flags[cnt++] = CStringGetTextDatum("DYNAMIC_DEFAULT");
if (record->flags & GUC_EXPLAIN)
flags[cnt++] = CStringGetTextDatum("EXPLAIN");
if (record->flags & GUC_NO_RESET_ALL)
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 4d0920c42e2..bb09c507f81 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -221,14 +221,14 @@ typedef enum
#define GUC_UNIT_KB 0x1000 /* value is in kilobytes */
#define GUC_UNIT_BLOCKS 0x2000 /* value is in blocks */
-#define GUC_UNIT_XBLOCKS 0x3000 /* value is in xlog blocks */
+#define GUC_UNIT_XBLOCKS 0x3000 /* value is in xlog blocks */ //
#define GUC_UNIT_MB 0x4000 /* value is in megabytes */
#define GUC_UNIT_BYTE 0x8000 /* value is in bytes */
#define GUC_UNIT_MEMORY 0xF000 /* mask for size-related units */
#define GUC_UNIT_MS 0x10000 /* value is in milliseconds */
#define GUC_UNIT_S 0x20000 /* value is in seconds */
-#define GUC_UNIT_MIN 0x30000 /* value is in minutes */
+#define GUC_UNIT_MIN 0x30000 /* value is in minutes */ //
#define GUC_UNIT_TIME 0xF0000 /* mask for time-related units */
#define GUC_EXPLAIN 0x100000 /* include in explain */
@@ -239,6 +239,8 @@ typedef enum
*/
#define GUC_RUNTIME_COMPUTED 0x200000
+#define GUC_DYNAMIC_DEFAULT 0x400000 /* default value is dynamic */
+
#define GUC_UNIT (GUC_UNIT_MEMORY | GUC_UNIT_TIME)
--
2.17.1
0002-Add-fileval-bootval-consistency-check-of-GUC-paramet.patchtext/x-diff; charset=us-asciiDownload
From fabe483dc806294eab85fb1dd0690344c5bd1e06 Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Date: Mon, 30 May 2022 16:11:34 +0900
Subject: [PATCH 2/2] Add fileval-bootval consistency check of GUC parameters
We should keep GUC variables consistent between the default values
written in postgresql.conf.sample and defined in guc.c. Add an
automated way to check for the consistency to the TAP test suite. Some
variables are still excluded since they cannot be checked simple way.
---
src/backend/utils/misc/guc.c | 70 +++++++++++++++++++
src/backend/utils/misc/postgresql.conf.sample | 6 +-
src/include/catalog/pg_proc.dat | 5 ++
src/test/modules/test_misc/t/003_check_guc.pl | 57 +++++++++++++--
4 files changed, 129 insertions(+), 9 deletions(-)
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index dbbadc5a475..a700cbbf4bc 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -7526,6 +7526,76 @@ parse_and_validate_value(struct config_generic *record,
return true;
}
+/*
+ * Helper function for pg_normalize_config_value().
+ * Makes a palloced copy of src then link val to it.
+ * DO NOT destroy val while dst is in use.
+ */
+static struct config_generic *
+copy_config_and_set_value(struct config_generic *src, union config_var_val *val)
+{
+ struct config_generic *dst;
+
+#define CREATE_CONFIG_COPY(dst, src, t) \
+ dst = palloc(sizeof(struct t)); \
+ *(struct t *) dst = *(struct t *) src; \
+
+ switch (src->vartype)
+ {
+ case PGC_BOOL:
+ CREATE_CONFIG_COPY(dst, src, config_bool);
+ ((struct config_bool *)dst)->variable = &val->boolval;
+ break;
+ case PGC_INT:
+ CREATE_CONFIG_COPY(dst, src, config_int);
+ ((struct config_int *)dst)->variable = &val->intval;
+ break;
+ case PGC_REAL:
+ CREATE_CONFIG_COPY(dst, src, config_real);
+ ((struct config_real *)dst)->variable = &val->realval;
+ break;
+ case PGC_STRING:
+ CREATE_CONFIG_COPY(dst, src, config_string);
+ ((struct config_string *)dst)->variable = &val->stringval;
+ break;
+ case PGC_ENUM:
+ CREATE_CONFIG_COPY(dst, src, config_enum);
+ ((struct config_enum *)dst)->variable = &val->enumval;
+ break;
+ }
+
+ return dst;
+}
+
+
+/*
+ * Normalize given value according to the specified GUC variable
+ */
+Datum
+pg_normalize_config_value(PG_FUNCTION_ARGS)
+{
+ char *name = "";
+ char *value = "";
+ struct config_generic *record;
+ char *result;
+ void *extra;
+ union config_var_val altval;
+
+ if (!PG_ARGISNULL(0))
+ name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ if (!PG_ARGISNULL(1))
+ value = text_to_cstring(PG_GETARG_TEXT_PP(1));
+
+ record = find_option(name, true, false, ERROR);
+
+ parse_and_validate_value(record, name, value, PGC_S_TEST, WARNING,
+ &altval, &extra);
+ record = copy_config_and_set_value(record, &altval);
+
+ result = _ShowOption(record, true);
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
/*
* Sets option `name' to given value.
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index b4bc06e5f5a..b6403a14496 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -476,7 +476,7 @@
# in all cases.
# These are relevant when logging to syslog:
-#syslog_facility = 'LOCAL0'
+#syslog_facility = 'local0'
#syslog_ident = 'postgres'
#syslog_sequence_numbers = on
#syslog_split_messages = on
@@ -709,7 +709,7 @@
# - Locale and Formatting -
-#datestyle = 'iso, mdy'
+#datestyle = 'ISO, MDY'
#intervalstyle = 'postgres'
#timezone = 'GMT'
#timezone_abbreviations = 'Default' # Select the set of available time zone
@@ -721,7 +721,7 @@
# share/timezonesets/.
#extra_float_digits = 1 # min -15, max 3; any value >0 actually
# selects precise output mode
-#client_encoding = sql_ascii # actually, defaults to database
+#client_encoding = SQL_ASCII # actually, defaults to database
# encoding
# These settings are initialized by initdb, but they can be changed.
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 8873078c160..a92630d768b 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6118,6 +6118,11 @@
proname => 'pg_settings_get_flags', provolatile => 's', prorettype => '_text',
proargtypes => 'text', prosrc => 'pg_settings_get_flags' },
+{ oid => '9956', descr => 'normalize value to the unit of specified GUC',
+ proname => 'pg_normalize_config_value', proisstrict => 'f',
+ provolatile => 's', prorettype => 'text', proargtypes => 'text text',
+ proargnames => '{varname,value}', prosrc => 'pg_normalize_config_value' },
+
{ oid => '3329', descr => 'show config file settings',
proname => 'pg_show_all_file_settings', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
diff --git a/src/test/modules/test_misc/t/003_check_guc.pl b/src/test/modules/test_misc/t/003_check_guc.pl
index 60459ef759e..dfb2d203247 100644
--- a/src/test/modules/test_misc/t/003_check_guc.pl
+++ b/src/test/modules/test_misc/t/003_check_guc.pl
@@ -11,18 +11,40 @@ my $node = PostgreSQL::Test::Cluster->new('main');
$node->init;
$node->start;
+# parameter names that cannot get consistency check performed
+my @ignored_parameters = (
+ 'data_directory',
+ 'hba_file',
+ 'ident_file',
+ 'krb_server_keyfile',
+ 'max_stack_depth',
+ 'bgwriter_flush_after',
+ 'wal_sync_method',
+ 'checkpoint_flush_after',
+ 'timezone_abbreviations',
+ 'lc_messages'
+ );
+
# Grab the names of all the parameters that can be listed in the
# configuration sample file. config_file is an exception, it is not
# in postgresql.conf.sample but is part of the lists from guc.c.
my $all_params = $node->safe_psql(
'postgres',
- "SELECT name
+ "SELECT lower(name), vartype, unit, boot_val, '!'
FROM pg_settings
WHERE NOT 'NOT_IN_SAMPLE' = ANY (pg_settings_get_flags(name)) AND
name <> 'config_file'
ORDER BY 1");
# Note the lower-case conversion, for consistency.
-my @all_params_array = split("\n", lc($all_params));
+my %all_params_hash;
+foreach my $line (split("\n", $all_params))
+{
+ my @f = split('\|', $line);
+ fail("query returned wrong number of columns: $#f : $line") if ($#f != 4);
+ $all_params_hash{$f[0]}->{type} = $f[1];
+ $all_params_hash{$f[0]}->{unit} = $f[2];
+ $all_params_hash{$f[0]}->{bootval} = $f[3];
+}
# Grab the names of all parameters marked as NOT_IN_SAMPLE.
my $not_in_sample = $node->safe_psql(
@@ -43,7 +65,7 @@ my @gucs_in_file;
# Read the sample file line-by-line, checking its contents to build a list
# of everything known as a GUC.
-my $num_tests = 0;
+my @check_elems = ();
open(my $contents, '<', $sample_file)
|| die "Could not open $sample_file: $!";
while (my $line = <$contents>)
@@ -53,11 +75,16 @@ while (my $line = <$contents>)
# file.
# - Valid configuration options are followed immediately by " = ",
# with one space before and after the equal sign.
- if ($line =~ m/^#?([_[:alpha:]]+) = .*/)
+ if ($line =~ m/^#?([_[:alpha:]]+) = (.*)$/)
{
# Lower-case conversion matters for some of the GUCs.
my $param_name = lc($1);
+ # extract value
+ my $file_value = $2;
+ $file_value =~ s/\s*#.*$//; # strip trailing comment
+ $file_value =~ s/^'(.*)'$/$1/; # strip quotes
+
# Ignore some exceptions.
next if $param_name eq "include";
next if $param_name eq "include_dir";
@@ -66,19 +93,37 @@ while (my $line = <$contents>)
# Update the list of GUCs found in the sample file, for the
# follow-up tests.
push @gucs_in_file, $param_name;
+
+ # Check for consistency between bootval and file value.
+ if (!grep { $_ eq $param_name } @ignored_parameters)
+ {
+ push (@check_elems, "('$param_name','$file_value')");
+ }
}
}
close $contents;
+# run consistency check between config-file's default value and boot values.
+my $check_query =
+ 'SELECT f.n, f.v, s.boot_val FROM (VALUES '.
+ join(',', @check_elems).
+ ') f(n,v) JOIN pg_settings s ON s.name = f.n '.
+ 'JOIN pg_settings_get_flags(s.name) flags ON true '.
+ "WHERE 'DYNAMIC_DEFAULT' <> ALL(flags) AND " .
+ 'pg_normalize_config_value(f.n, f.v) <> '.
+ 'pg_normalize_config_value(f.n, s.boot_val)';
+
+is ($node->safe_psql('postgres', $check_query), '',
+ 'check if fileval-bootval consistency is fine');
+
# Cross-check that all the GUCs found in the sample file match the ones
# fetched above. This maps the arrays to a hash, making the creation of
# each exclude and intersection list easier.
my %gucs_in_file_hash = map { $_ => 1 } @gucs_in_file;
-my %all_params_hash = map { $_ => 1 } @all_params_array;
my %not_in_sample_hash = map { $_ => 1 } @not_in_sample_array;
-my @missing_from_file = grep(!$gucs_in_file_hash{$_}, @all_params_array);
+my @missing_from_file = grep(!$gucs_in_file_hash{$_}, keys %all_params_hash);
is(scalar(@missing_from_file),
0, "no parameters missing from postgresql.conf.sample");
--
2.17.1
Note that this can currently exposes internal elog() errors to users:
postgres=# select pg_normalize_config_value('log_min_messages','abc');
WARNING: invalid value for parameter "log_min_messages": "abc"
HINT: Available values: debug5, debug4, debug3, debug2, debug1, info, notice, warning, error, log, fatal, panic.
ERROR: could not find enum option 0 for log_min_messages
postgres=# \errverbose
ERROR: XX000: could not find enum option 0 for log_min_messages
LOCATION: config_enum_lookup_by_value, guc.c:7284
Hi,
Checking if you're planning to work on this patch still ?
Show quoted text
On Thu, Jul 28, 2022 at 05:27:34PM -0500, Justin Pryzby wrote:
Note that this can currently exposes internal elog() errors to users:
postgres=# select pg_normalize_config_value('log_min_messages','abc');
WARNING: invalid value for parameter "log_min_messages": "abc"
HINT: Available values: debug5, debug4, debug3, debug2, debug1, info, notice, warning, error, log, fatal, panic.
ERROR: could not find enum option 0 for log_min_messagespostgres=# \errverbose
ERROR: XX000: could not find enum option 0 for log_min_messages
LOCATION: config_enum_lookup_by_value, guc.c:7284
This is an alternative implementation, which still relies on adding the
GUC_DYNAMIC, flag but doesn't require adding a new, sql-accessible
function to convert the GUC to a pretty/human display value.
Attachments:
0001-add-DYNAMIC_DEFAULT-for-settings-which-vary-by-.-con.patchtext/x-diff; charset=us-asciiDownload
From 25ee6d6ed23ff273e622551fd033c8d086953fe5 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Tue, 19 Jul 2022 08:31:56 -0500
Subject: [PATCH 1/2] add DYNAMIC_DEFAULT for settings which vary by
./configure or platform or initdb
---
doc/src/sgml/func.sgml | 8 ++++++
src/backend/utils/misc/guc_funcs.c | 4 ++-
src/backend/utils/misc/guc_tables.c | 39 ++++++++++++++++-------------
src/include/utils/guc.h | 2 ++
4 files changed, 35 insertions(+), 18 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 67eb3806326..63063d054c5 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -24341,6 +24341,14 @@ SELECT collation for ('foo' COLLATE "de_DE");
<row><entry>Flag</entry><entry>Description</entry></row>
</thead>
<tbody>
+
+ <row>
+ <entry><literal>DYNAMIC_DEFAULT</literal></entry>
+ <entry>Parameters with this flag have default values which vary by
+ platform, or compile-time options, or are set dynamically during initdb.
+ </entry>
+ </row>
+
<row>
<entry><literal>EXPLAIN</literal></entry>
<entry>Parameters with this flag are included in
diff --git a/src/backend/utils/misc/guc_funcs.c b/src/backend/utils/misc/guc_funcs.c
index 3d2df18659b..3376f23e076 100644
--- a/src/backend/utils/misc/guc_funcs.c
+++ b/src/backend/utils/misc/guc_funcs.c
@@ -539,7 +539,7 @@ ShowAllGUCConfig(DestReceiver *dest)
Datum
pg_settings_get_flags(PG_FUNCTION_ARGS)
{
-#define MAX_GUC_FLAGS 5
+#define MAX_GUC_FLAGS 6
char *varname = TextDatumGetCString(PG_GETARG_DATUM(0));
struct config_generic *record;
int cnt = 0;
@@ -552,6 +552,8 @@ pg_settings_get_flags(PG_FUNCTION_ARGS)
if (record == NULL)
PG_RETURN_NULL();
+ if (record->flags & GUC_DYNAMIC_DEFAULT)
+ flags[cnt++] = CStringGetTextDatum("DYNAMIC_DEFAULT");
if (record->flags & GUC_EXPLAIN)
flags[cnt++] = CStringGetTextDatum("EXPLAIN");
if (record->flags & GUC_NO_RESET_ALL)
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 7bff45e10d6..bd3c84bc7b8 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -1201,7 +1201,7 @@ struct config_bool ConfigureNamesBool[] =
{"debug_assertions", PGC_INTERNAL, PRESET_OPTIONS,
gettext_noop("Shows whether the running server has assertion checks enabled."),
NULL,
- GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE
+ GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE | GUC_DYNAMIC_DEFAULT
},
&assert_enabled,
#ifdef USE_ASSERT_CHECKING
@@ -1377,7 +1377,8 @@ struct config_bool ConfigureNamesBool[] =
{
{"update_process_title", PGC_SUSET, PROCESS_TITLE,
gettext_noop("Updates the process title to show the active SQL command."),
- gettext_noop("Enables updating of the process title every time a new SQL command is received by the server.")
+ gettext_noop("Enables updating of the process title every time a new SQL command is received by the server."),
+ GUC_DYNAMIC_DEFAULT
},
&update_process_title,
#ifdef WIN32
@@ -2126,7 +2127,8 @@ struct config_int ConfigureNamesInt[] =
{
{"max_connections", PGC_POSTMASTER, CONN_AUTH_SETTINGS,
gettext_noop("Sets the maximum number of concurrent connections."),
- NULL
+ NULL,
+ GUC_DYNAMIC_DEFAULT
},
&MaxConnections,
100, 1, MAX_BACKENDS,
@@ -2163,7 +2165,7 @@ struct config_int ConfigureNamesInt[] =
{"shared_buffers", PGC_POSTMASTER, RESOURCES_MEM,
gettext_noop("Sets the number of shared memory buffers used by the server."),
NULL,
- GUC_UNIT_BLOCKS
+ GUC_UNIT_BLOCKS | GUC_DYNAMIC_DEFAULT
},
&NBuffers,
16384, 16, INT_MAX / 2,
@@ -2677,7 +2679,7 @@ struct config_int ConfigureNamesInt[] =
{"checkpoint_flush_after", PGC_SIGHUP, WAL_CHECKPOINTS,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
NULL,
- GUC_UNIT_BLOCKS
+ GUC_UNIT_BLOCKS | GUC_DYNAMIC_DEFAULT
},
&checkpoint_flush_after,
DEFAULT_CHECKPOINT_FLUSH_AFTER, 0, WRITEBACK_MAX_PENDING_FLUSHES,
@@ -2895,7 +2897,7 @@ struct config_int ConfigureNamesInt[] =
{"bgwriter_flush_after", PGC_SIGHUP, RESOURCES_BGWRITER,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
NULL,
- GUC_UNIT_BLOCKS
+ GUC_UNIT_BLOCKS | GUC_DYNAMIC_DEFAULT
},
&bgwriter_flush_after,
DEFAULT_BGWRITER_FLUSH_AFTER, 0, WRITEBACK_MAX_PENDING_FLUSHES,
@@ -2908,7 +2910,7 @@ struct config_int ConfigureNamesInt[] =
RESOURCES_ASYNCHRONOUS,
gettext_noop("Number of simultaneous requests that can be handled efficiently by the disk subsystem."),
NULL,
- GUC_EXPLAIN
+ GUC_EXPLAIN | GUC_DYNAMIC_DEFAULT
},
&effective_io_concurrency,
#ifdef USE_PREFETCH
@@ -2926,7 +2928,7 @@ struct config_int ConfigureNamesInt[] =
RESOURCES_ASYNCHRONOUS,
gettext_noop("A variant of effective_io_concurrency that is used for maintenance work."),
NULL,
- GUC_EXPLAIN
+ GUC_EXPLAIN | GUC_DYNAMIC_DEFAULT
},
&maintenance_io_concurrency,
#ifdef USE_PREFETCH
@@ -2943,7 +2945,7 @@ struct config_int ConfigureNamesInt[] =
{"backend_flush_after", PGC_USERSET, RESOURCES_ASYNCHRONOUS,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
NULL,
- GUC_UNIT_BLOCKS
+ GUC_UNIT_BLOCKS | GUC_DYNAMIC_DEFAULT
},
&backend_flush_after,
DEFAULT_BACKEND_FLUSH_AFTER, 0, WRITEBACK_MAX_PENDING_FLUSHES,
@@ -3397,7 +3399,7 @@ struct config_int ConfigureNamesInt[] =
{"debug_discard_caches", PGC_SUSET, DEVELOPER_OPTIONS,
gettext_noop("Aggressively flush system caches for debugging purposes."),
NULL,
- GUC_NOT_IN_SAMPLE
+ GUC_NOT_IN_SAMPLE | GUC_DYNAMIC_DEFAULT
},
&debug_discard_caches,
#ifdef DISCARD_CACHES_ENABLED
@@ -4224,7 +4226,7 @@ struct config_string ConfigureNamesString[] =
{"unix_socket_directories", PGC_POSTMASTER, CONN_AUTH_SETTINGS,
gettext_noop("Sets the directories where Unix-domain sockets will be created."),
NULL,
- GUC_LIST_INPUT | GUC_LIST_QUOTE | GUC_SUPERUSER_ONLY
+ GUC_LIST_INPUT | GUC_LIST_QUOTE | GUC_SUPERUSER_ONLY | GUC_DYNAMIC_DEFAULT
},
&Unix_socket_directories,
DEFAULT_PGSOCKET_DIR,
@@ -4305,7 +4307,7 @@ struct config_string ConfigureNamesString[] =
{"ssl_library", PGC_INTERNAL, PRESET_OPTIONS,
gettext_noop("Shows the name of the SSL library."),
NULL,
- GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE
+ GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE | GUC_DYNAMIC_DEFAULT
},
&ssl_library,
#ifdef USE_SSL
@@ -4391,7 +4393,7 @@ struct config_string ConfigureNamesString[] =
{"ssl_ciphers", PGC_SIGHUP, CONN_AUTH_SSL,
gettext_noop("Sets the list of allowed SSL ciphers."),
NULL,
- GUC_SUPERUSER_ONLY
+ GUC_SUPERUSER_ONLY | GUC_DYNAMIC_DEFAULT
},
&SSLCipherSuites,
#ifdef USE_OPENSSL
@@ -4406,7 +4408,7 @@ struct config_string ConfigureNamesString[] =
{"ssl_ecdh_curve", PGC_SIGHUP, CONN_AUTH_SSL,
gettext_noop("Sets the curve to use for ECDH."),
NULL,
- GUC_SUPERUSER_ONLY
+ GUC_SUPERUSER_ONLY | GUC_DYNAMIC_DEFAULT
},
&SSLECDHCurve,
#ifdef USE_SSL
@@ -4644,7 +4646,8 @@ struct config_enum ConfigureNamesEnum[] =
{
{"syslog_facility", PGC_SIGHUP, LOGGING_WHERE,
gettext_noop("Sets the syslog \"facility\" to be used when syslog enabled."),
- NULL
+ NULL,
+ GUC_DYNAMIC_DEFAULT
},
&syslog_facility,
#ifdef HAVE_SYSLOG
@@ -4757,7 +4760,8 @@ struct config_enum ConfigureNamesEnum[] =
{
{"dynamic_shared_memory_type", PGC_POSTMASTER, RESOURCES_MEM,
gettext_noop("Selects the dynamic shared memory implementation used."),
- NULL
+ NULL,
+ GUC_DYNAMIC_DEFAULT
},
&dynamic_shared_memory_type,
DEFAULT_DYNAMIC_SHARED_MEMORY_TYPE, dynamic_shared_memory_options,
@@ -4777,7 +4781,8 @@ struct config_enum ConfigureNamesEnum[] =
{
{"wal_sync_method", PGC_SIGHUP, WAL_SETTINGS,
gettext_noop("Selects the method used for forcing WAL updates to disk."),
- NULL
+ NULL,
+ GUC_DYNAMIC_DEFAULT
},
&sync_method,
DEFAULT_SYNC_METHOD, sync_method_options,
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index e426dd757d9..0447d3eb56a 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -238,6 +238,8 @@ typedef enum
*/
#define GUC_RUNTIME_COMPUTED 0x200000
+#define GUC_DYNAMIC_DEFAULT 0x400000 /* default value is dynamic */
+
#define GUC_UNIT (GUC_UNIT_MEMORY | GUC_UNIT_TIME)
--
2.25.1
0002-WIP-test-guc-default-values.patchtext/x-diff; charset=us-asciiDownload
From d7940be9ac7f1e4d29b6651a315c8dbfc54f84e2 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Wed, 25 May 2022 05:14:43 -0500
Subject: [PATCH 2/2] WIP: test guc default values
---
src/backend/utils/misc/postgresql.conf.sample | 2 +-
src/test/modules/test_misc/t/003_check_guc.pl | 32 +++++++++++++++++++
2 files changed, 33 insertions(+), 1 deletion(-)
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 8a0b383eb71..88a033103d1 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -721,7 +721,7 @@
# share/timezonesets/.
#extra_float_digits = 1 # min -15, max 3; any value >0 actually
# selects precise output mode
-#client_encoding = sql_ascii # actually, defaults to database
+#client_encoding = SQL_ASCII # actually, defaults to database
# encoding
# These settings are initialized by initdb, but they can be changed.
diff --git a/src/test/modules/test_misc/t/003_check_guc.pl b/src/test/modules/test_misc/t/003_check_guc.pl
index 60459ef759e..dd5ac2c8bc4 100644
--- a/src/test/modules/test_misc/t/003_check_guc.pl
+++ b/src/test/modules/test_misc/t/003_check_guc.pl
@@ -107,4 +107,36 @@ foreach my $param (@sample_intersect)
);
}
+# Test that GUCs in postgresql.conf.sample show the correct default values
+my $check_defaults = $node->safe_psql(
+ 'postgres',
+ "
+ CREATE TABLE sample_conf AS
+ SELECT m[1] AS name, COALESCE(m[3], m[5]) AS sample_value
+ FROM (SELECT regexp_split_to_table(pg_read_file('../../../$sample_file'), '\n') AS ln) conf,
+ regexp_match(ln, '^#?([_[:alpha:]]+) (= ''([^'']*)''|(= ([^[:space:]]*))).*') AS m
+ WHERE ln ~ '^#?[[:alpha:]]'
+ ");
+
+$check_defaults = $node->safe_psql(
+ 'postgres',
+ "
+ SELECT name, current_setting(name), sc.sample_value, boot_val
+ FROM pg_show_all_settings() psas
+ JOIN sample_conf sc USING(name)
+ WHERE sc.sample_value != boot_val -- same value
+ AND sc.sample_value != current_setting(name) -- same value, with human units
+ AND sc.sample_value != current_setting(name)||'.0' -- same value with .0 suffix
+ AND 'DYNAMIC_DEFAULT' != ALL(pg_settings_get_flags(psas.name)) -- dynamically-set defaults
+ AND NOT (sc.sample_value ~ '^0' AND current_setting(name) ~ '^0') -- zeros may be written differently
+ AND NOT (sc.sample_value='60s' AND current_setting(name) = '1min') -- two ways to write 1min
+ AND name NOT IN ('krb_server_keyfile', 'wal_retrieve_retry_interval', 'log_autovacuum_min_duration'); -- exceptions
+ ");
+
+my @check_defaults_array = split("\n", lc($check_defaults));
+print(STDERR "$check_defaults\n");
+
+is (@check_defaults_array, 0,
+ 'check for consistency of defaults in postgresql.conf.sample');
+
done_testing();
--
2.25.1
At Sat, 17 Sep 2022 23:53:07 -0500, Justin Pryzby <pryzby@telsasoft.com> wrote in
This is an alternative implementation, which still relies on adding the
GUC_DYNAMIC, flag but doesn't require adding a new, sql-accessible
function to convert the GUC to a pretty/human display value.
Thanks!
I'm not sure shared_buffer is GUC_DYNAMIC_DEFAULT, and we need to read
postgresql.conf.sample using SQL, but +1 for the direction.
+ AND NOT (sc.sample_value ~ '^0' AND current_setting(name) ~ '^0') -- zeros may be written differently
+ AND NOT (sc.sample_value='60s' AND current_setting(name) = '1min') -- two ways to write 1min
Mmm. Couldn't we get away from that explicit exceptions?
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
Hi, I was hoping to use this patch in my other thread [1]GUC C var sanity check - /messages/by-id/CAHut+Ps91wgaE9P7JORnK_dGq7zPB56WLDJwLNCLgGXxqrh9=Q@mail.gmail.com, but your
latest attachment is reported broken in cfbot [2]cfbot fail - http://cfbot.cputube.org/patch_40_3736.log. Please rebase it.
------
[1]: GUC C var sanity check - /messages/by-id/CAHut+Ps91wgaE9P7JORnK_dGq7zPB56WLDJwLNCLgGXxqrh9=Q@mail.gmail.com
/messages/by-id/CAHut+Ps91wgaE9P7JORnK_dGq7zPB56WLDJwLNCLgGXxqrh9=Q@mail.gmail.com
[2]: cfbot fail - http://cfbot.cputube.org/patch_40_3736.log
Kind Regards,
Peter Smith.
Fujitsu Australia
At Fri, 21 Oct 2022 11:58:15 +1100, Peter Smith <smithpb2250@gmail.com> wrote in
Hi, I was hoping to use this patch in my other thread [1], but your
latest attachment is reported broken in cfbot [2]. Please rebase it.
Ouch. I haven't reach here. I'll do that next Monday.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
On Mon, Sep 26, 2022 at 05:29:58PM +0900, Kyotaro Horiguchi wrote:
At Sat, 17 Sep 2022 23:53:07 -0500, Justin Pryzby <pryzby@telsasoft.com> wrote in
This is an alternative implementation, which still relies on adding the
GUC_DYNAMIC, flag but doesn't require adding a new, sql-accessible
function to convert the GUC to a pretty/human display value.Thanks!
I'm not sure shared_buffer is GUC_DYNAMIC_DEFAULT, and we need to read
It's set during initdb.
postgresql.conf.sample using SQL, but +1 for the direction.
+ AND NOT (sc.sample_value ~ '^0' AND current_setting(name) ~ '^0') -- zeros may be written differently + AND NOT (sc.sample_value='60s' AND current_setting(name) = '1min') -- two ways to write 1minMmm. Couldn't we get away from that explicit exceptions?
Suggestions are welcomed.
Rebased the patch.
I also split the flag into DEFAULTS_COMPILE and DEFAULTS_INITDB, since
that makes it easier to understand what the flags mean and the intent of
the patch. And maybe allows fewer exclusions in patches like Peter's,
which I think would only want to exclude compile-time defaults.
--
Justin
Attachments:
0001-add-DYNAMIC_DEFAULT-for-settings-which-vary-by-.-con.patchtext/x-diff; charset=us-asciiDownload
From c3ac7bd40d26eb692f939d58935415ceb1cf308e Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Tue, 19 Jul 2022 08:31:56 -0500
Subject: [PATCH 1/2] add DYNAMIC_DEFAULT for settings which vary by
./configure or platform or initdb
---
doc/src/sgml/func.sgml | 15 ++++++
src/backend/utils/misc/guc_funcs.c | 6 ++-
src/backend/utils/misc/guc_tables.c | 78 ++++++++++++++++++-----------
src/include/utils/guc.h | 2 +
4 files changed, 70 insertions(+), 31 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 6e0425cb3dc..b8e05073d1a 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -24374,6 +24374,21 @@ SELECT collation for ('foo' COLLATE "de_DE");
<row><entry>Flag</entry><entry>Description</entry></row>
</thead>
<tbody>
+
+ <row>
+ <entry><literal>DEFAULT_COMPILE</literal></entry>
+ <entry>Parameters with this flag have default values which vary by
+ platform, or compile-time options.
+ </entry>
+ </row>
+
+ <row>
+ <entry><literal>DEFAULT_INITDB</literal></entry>
+ <entry>Parameters with this flag have default values which are
+ determined dynamically during initdb.
+ </entry>
+ </row>
+
<row>
<entry><literal>EXPLAIN</literal></entry>
<entry>Parameters with this flag are included in
diff --git a/src/backend/utils/misc/guc_funcs.c b/src/backend/utils/misc/guc_funcs.c
index 108b3bd1290..2b666e8d014 100644
--- a/src/backend/utils/misc/guc_funcs.c
+++ b/src/backend/utils/misc/guc_funcs.c
@@ -538,7 +538,7 @@ ShowAllGUCConfig(DestReceiver *dest)
Datum
pg_settings_get_flags(PG_FUNCTION_ARGS)
{
-#define MAX_GUC_FLAGS 6
+#define MAX_GUC_FLAGS 8
char *varname = TextDatumGetCString(PG_GETARG_DATUM(0));
struct config_generic *record;
int cnt = 0;
@@ -551,6 +551,10 @@ pg_settings_get_flags(PG_FUNCTION_ARGS)
if (record == NULL)
PG_RETURN_NULL();
+ if (record->flags & GUC_DEFAULT_COMPILE)
+ flags[cnt++] = CStringGetTextDatum("DEFAULT_COMPILE");
+ if (record->flags & GUC_DEFAULT_INITDB)
+ flags[cnt++] = CStringGetTextDatum("DEFAULT_INITDB");
if (record->flags & GUC_EXPLAIN)
flags[cnt++] = CStringGetTextDatum("EXPLAIN");
if (record->flags & GUC_NO_RESET)
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 6454d034e7e..885d3dd8b81 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -601,7 +601,7 @@ StaticAssertDecl(lengthof(GucContext_Names) == (PGC_USERSET + 1),
const char *const GucSource_Names[] =
{
/* PGC_S_DEFAULT */ "default",
- /* PGC_S_DYNAMIC_DEFAULT */ "default",
+ /* PGC_S_DYNAMIC_DEFAULT */ "default", //
/* PGC_S_ENV_VAR */ "environment variable",
/* PGC_S_FILE */ "configuration file",
/* PGC_S_ARGV */ "command line",
@@ -1191,7 +1191,7 @@ struct config_bool ConfigureNamesBool[] =
{"debug_assertions", PGC_INTERNAL, PRESET_OPTIONS,
gettext_noop("Shows whether the running server has assertion checks enabled."),
NULL,
- GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE
+ GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE | GUC_DEFAULT_COMPILE
},
&assert_enabled,
#ifdef USE_ASSERT_CHECKING
@@ -1367,7 +1367,8 @@ struct config_bool ConfigureNamesBool[] =
{
{"update_process_title", PGC_SUSET, PROCESS_TITLE,
gettext_noop("Updates the process title to show the active SQL command."),
- gettext_noop("Enables updating of the process title every time a new SQL command is received by the server.")
+ gettext_noop("Enables updating of the process title every time a new SQL command is received by the server."),
+ GUC_DEFAULT_COMPILE
},
&update_process_title,
#ifdef WIN32
@@ -2116,7 +2117,8 @@ struct config_int ConfigureNamesInt[] =
{
{"max_connections", PGC_POSTMASTER, CONN_AUTH_SETTINGS,
gettext_noop("Sets the maximum number of concurrent connections."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
},
&MaxConnections,
100, 1, MAX_BACKENDS,
@@ -2153,7 +2155,7 @@ struct config_int ConfigureNamesInt[] =
{"shared_buffers", PGC_POSTMASTER, RESOURCES_MEM,
gettext_noop("Sets the number of shared memory buffers used by the server."),
NULL,
- GUC_UNIT_BLOCKS
+ GUC_UNIT_BLOCKS | GUC_DEFAULT_INITDB
},
&NBuffers,
16384, 16, INT_MAX / 2,
@@ -2196,7 +2198,8 @@ struct config_int ConfigureNamesInt[] =
{
{"port", PGC_POSTMASTER, CONN_AUTH_SETTINGS,
gettext_noop("Sets the TCP port the server listens on."),
- NULL
+ NULL,
+ GUC_DEFAULT_COMPILE
},
&PostPortNumber,
DEF_PGPORT, 1, 65535,
@@ -2225,7 +2228,8 @@ struct config_int ConfigureNamesInt[] =
"to be a numeric mode specification in the form "
"accepted by the chmod and umask system calls. "
"(To use the customary octal format the number must "
- "start with a 0 (zero).)")
+ "start with a 0 (zero).)"),
+ GUC_DEFAULT_INITDB
},
&Log_file_mode,
0600, 0000, 0777,
@@ -2667,7 +2671,7 @@ struct config_int ConfigureNamesInt[] =
{"checkpoint_flush_after", PGC_SIGHUP, WAL_CHECKPOINTS,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
NULL,
- GUC_UNIT_BLOCKS
+ GUC_UNIT_BLOCKS | GUC_DEFAULT_COMPILE
},
&checkpoint_flush_after,
DEFAULT_CHECKPOINT_FLUSH_AFTER, 0, WRITEBACK_MAX_PENDING_FLUSHES,
@@ -2885,7 +2889,7 @@ struct config_int ConfigureNamesInt[] =
{"bgwriter_flush_after", PGC_SIGHUP, RESOURCES_BGWRITER,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
NULL,
- GUC_UNIT_BLOCKS
+ GUC_UNIT_BLOCKS | GUC_DEFAULT_COMPILE
},
&bgwriter_flush_after,
DEFAULT_BGWRITER_FLUSH_AFTER, 0, WRITEBACK_MAX_PENDING_FLUSHES,
@@ -2898,7 +2902,7 @@ struct config_int ConfigureNamesInt[] =
RESOURCES_ASYNCHRONOUS,
gettext_noop("Number of simultaneous requests that can be handled efficiently by the disk subsystem."),
NULL,
- GUC_EXPLAIN
+ GUC_EXPLAIN | GUC_DEFAULT_COMPILE
},
&effective_io_concurrency,
#ifdef USE_PREFETCH
@@ -2916,7 +2920,7 @@ struct config_int ConfigureNamesInt[] =
RESOURCES_ASYNCHRONOUS,
gettext_noop("A variant of effective_io_concurrency that is used for maintenance work."),
NULL,
- GUC_EXPLAIN
+ GUC_EXPLAIN | GUC_DEFAULT_COMPILE
},
&maintenance_io_concurrency,
#ifdef USE_PREFETCH
@@ -2933,7 +2937,7 @@ struct config_int ConfigureNamesInt[] =
{"backend_flush_after", PGC_USERSET, RESOURCES_ASYNCHRONOUS,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
NULL,
- GUC_UNIT_BLOCKS
+ GUC_UNIT_BLOCKS | GUC_DEFAULT_COMPILE
},
&backend_flush_after,
DEFAULT_BACKEND_FLUSH_AFTER, 0, WRITEBACK_MAX_PENDING_FLUSHES,
@@ -3387,7 +3391,7 @@ struct config_int ConfigureNamesInt[] =
{"debug_discard_caches", PGC_SUSET, DEVELOPER_OPTIONS,
gettext_noop("Aggressively flush system caches for debugging purposes."),
NULL,
- GUC_NOT_IN_SAMPLE
+ GUC_NOT_IN_SAMPLE | GUC_DEFAULT_COMPILE
},
&debug_discard_caches,
#ifdef DISCARD_CACHES_ENABLED
@@ -3880,7 +3884,8 @@ struct config_string ConfigureNamesString[] =
{
{"log_timezone", PGC_SIGHUP, LOGGING_WHAT,
gettext_noop("Sets the time zone to use in log messages."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
},
&log_timezone_string,
"GMT",
@@ -3892,7 +3897,7 @@ struct config_string ConfigureNamesString[] =
gettext_noop("Sets the display format for date and time values."),
gettext_noop("Also controls interpretation of ambiguous "
"date inputs."),
- GUC_LIST_INPUT | GUC_REPORT
+ GUC_LIST_INPUT | GUC_REPORT | GUC_DEFAULT_INITDB
},
&datestyle_string,
"ISO, MDY",
@@ -3994,7 +3999,8 @@ struct config_string ConfigureNamesString[] =
{
{"lc_messages", PGC_SUSET, CLIENT_CONN_LOCALE,
gettext_noop("Sets the language in which messages are displayed."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
},
&locale_messages,
"",
@@ -4004,7 +4010,8 @@ struct config_string ConfigureNamesString[] =
{
{"lc_monetary", PGC_USERSET, CLIENT_CONN_LOCALE,
gettext_noop("Sets the locale for formatting monetary amounts."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
},
&locale_monetary,
"C",
@@ -4014,7 +4021,8 @@ struct config_string ConfigureNamesString[] =
{
{"lc_numeric", PGC_USERSET, CLIENT_CONN_LOCALE,
gettext_noop("Sets the locale for formatting numbers."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
},
&locale_numeric,
"C",
@@ -4024,7 +4032,8 @@ struct config_string ConfigureNamesString[] =
{
{"lc_time", PGC_USERSET, CLIENT_CONN_LOCALE,
gettext_noop("Sets the locale for formatting date and time values."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
},
&locale_time,
"C",
@@ -4183,7 +4192,8 @@ struct config_string ConfigureNamesString[] =
{"TimeZone", PGC_USERSET, CLIENT_CONN_LOCALE,
gettext_noop("Sets the time zone for displaying and interpreting time stamps."),
NULL,
- GUC_REPORT
+ GUC_REPORT | GUC_DEFAULT_INITDB
+
},
&timezone_string,
"GMT",
@@ -4214,7 +4224,7 @@ struct config_string ConfigureNamesString[] =
{"unix_socket_directories", PGC_POSTMASTER, CONN_AUTH_SETTINGS,
gettext_noop("Sets the directories where Unix-domain sockets will be created."),
NULL,
- GUC_LIST_INPUT | GUC_LIST_QUOTE | GUC_SUPERUSER_ONLY
+ GUC_LIST_INPUT | GUC_LIST_QUOTE | GUC_SUPERUSER_ONLY | GUC_DEFAULT_COMPILE
},
&Unix_socket_directories,
DEFAULT_PGSOCKET_DIR,
@@ -4295,7 +4305,7 @@ struct config_string ConfigureNamesString[] =
{"ssl_library", PGC_INTERNAL, PRESET_OPTIONS,
gettext_noop("Shows the name of the SSL library."),
NULL,
- GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE
+ GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE | GUC_DEFAULT_COMPILE
},
&ssl_library,
#ifdef USE_SSL
@@ -4370,7 +4380,9 @@ struct config_string ConfigureNamesString[] =
{
{"default_text_search_config", PGC_USERSET, CLIENT_CONN_LOCALE,
gettext_noop("Sets default text search configuration."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
+
},
&TSCurrentConfig,
"pg_catalog.simple",
@@ -4381,7 +4393,7 @@ struct config_string ConfigureNamesString[] =
{"ssl_ciphers", PGC_SIGHUP, CONN_AUTH_SSL,
gettext_noop("Sets the list of allowed SSL ciphers."),
NULL,
- GUC_SUPERUSER_ONLY
+ GUC_SUPERUSER_ONLY | GUC_DEFAULT_COMPILE
},
&SSLCipherSuites,
#ifdef USE_OPENSSL
@@ -4396,7 +4408,7 @@ struct config_string ConfigureNamesString[] =
{"ssl_ecdh_curve", PGC_SIGHUP, CONN_AUTH_SSL,
gettext_noop("Sets the curve to use for ECDH."),
NULL,
- GUC_SUPERUSER_ONLY
+ GUC_SUPERUSER_ONLY | GUC_DEFAULT_COMPILE
},
&SSLECDHCurve,
#ifdef USE_SSL
@@ -4634,7 +4646,8 @@ struct config_enum ConfigureNamesEnum[] =
{
{"syslog_facility", PGC_SIGHUP, LOGGING_WHERE,
gettext_noop("Sets the syslog \"facility\" to be used when syslog enabled."),
- NULL
+ NULL,
+ GUC_DEFAULT_COMPILE
},
&syslog_facility,
#ifdef HAVE_SYSLOG
@@ -4747,7 +4760,9 @@ struct config_enum ConfigureNamesEnum[] =
{
{"dynamic_shared_memory_type", PGC_POSTMASTER, RESOURCES_MEM,
gettext_noop("Selects the dynamic shared memory implementation used."),
- NULL
+ NULL,
+ /* platform-specific default, which is also overriden during initdb */
+ GUC_DEFAULT_COMPILE | GUC_DEFAULT_INITDB
},
&dynamic_shared_memory_type,
DEFAULT_DYNAMIC_SHARED_MEMORY_TYPE, dynamic_shared_memory_options,
@@ -4757,7 +4772,8 @@ struct config_enum ConfigureNamesEnum[] =
{
{"shared_memory_type", PGC_POSTMASTER, RESOURCES_MEM,
gettext_noop("Selects the shared memory implementation used for the main shared memory region."),
- NULL
+ NULL,
+ GUC_DEFAULT_COMPILE
},
&shared_memory_type,
DEFAULT_SHARED_MEMORY_TYPE, shared_memory_options,
@@ -4767,7 +4783,8 @@ struct config_enum ConfigureNamesEnum[] =
{
{"wal_sync_method", PGC_SIGHUP, WAL_SETTINGS,
gettext_noop("Selects the method used for forcing WAL updates to disk."),
- NULL
+ NULL,
+ GUC_DEFAULT_COMPILE
},
&sync_method,
DEFAULT_SYNC_METHOD, sync_method_options,
@@ -4829,7 +4846,8 @@ struct config_enum ConfigureNamesEnum[] =
{
{"password_encryption", PGC_USERSET, CONN_AUTH_AUTH,
gettext_noop("Chooses the algorithm for encrypting passwords."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
},
&Password_encryption,
PASSWORD_TYPE_SCRAM_SHA_256, password_encryption_options,
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index b3aaff9665b..79053583a03 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -221,6 +221,8 @@ typedef enum
#define GUC_DISALLOW_IN_AUTO_FILE \
0x002000 /* can't set in PG_AUTOCONF_FILENAME */
#define GUC_RUNTIME_COMPUTED 0x004000 /* delay processing in 'postgres -C' */
+#define GUC_DEFAULT_COMPILE 0x008000 /* default is determined at compile time */
+#define GUC_DEFAULT_INITDB 0x010000 /* default is determined at during initdb */
#define GUC_UNIT_KB 0x01000000 /* value is in kilobytes */
#define GUC_UNIT_BLOCKS 0x02000000 /* value is in blocks */
--
2.25.1
0002-WIP-test-guc-default-values.patchtext/x-diff; charset=us-asciiDownload
From 862586ef48b35071393c858c01419144d09c8400 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Wed, 25 May 2022 05:14:43 -0500
Subject: [PATCH 2/2] WIP: test guc default values
---
src/backend/utils/misc/postgresql.conf.sample | 2 +-
src/test/modules/test_misc/t/003_check_guc.pl | 33 +++++++++++++++++++
2 files changed, 34 insertions(+), 1 deletion(-)
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 1ce0dee6d04..21bfbef3291 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -720,7 +720,7 @@
# share/timezonesets/.
#extra_float_digits = 1 # min -15, max 3; any value >0 actually
# selects precise output mode
-#client_encoding = sql_ascii # actually, defaults to database
+#client_encoding = SQL_ASCII # actually, defaults to database
# encoding
# These settings are initialized by initdb, but they can be changed.
diff --git a/src/test/modules/test_misc/t/003_check_guc.pl b/src/test/modules/test_misc/t/003_check_guc.pl
index 60459ef759e..a660e85db58 100644
--- a/src/test/modules/test_misc/t/003_check_guc.pl
+++ b/src/test/modules/test_misc/t/003_check_guc.pl
@@ -107,4 +107,37 @@ foreach my $param (@sample_intersect)
);
}
+# Test that GUCs in postgresql.conf.sample show the correct default values
+my $check_defaults = $node->safe_psql(
+ 'postgres',
+ "
+ CREATE TABLE sample_conf AS
+ SELECT m[1] AS name, COALESCE(m[3], m[5]) AS sample_value
+ FROM (SELECT regexp_split_to_table(pg_read_file('../../../$sample_file'), '\n') AS ln) conf,
+ regexp_match(ln, '^#?([_[:alpha:]]+) (= ''([^'']*)''|(= ([^[:space:]]*))).*') AS m
+ WHERE ln ~ '^#?[[:alpha:]]'
+ ");
+
+$check_defaults = $node->safe_psql(
+ 'postgres',
+ "
+ SELECT name, current_setting(name), sc.sample_value, boot_val
+ FROM pg_show_all_settings() psas
+ JOIN sample_conf sc USING(name)
+ WHERE sc.sample_value != boot_val -- same value (specified by Cluster.pm)
+ AND sc.sample_value != current_setting(name) -- same value, with human units
+ AND sc.sample_value != current_setting(name)||'.0' -- same value with .0 suffix
+ AND 'DEFAULT_COMPILE' != ALL(pg_settings_get_flags(psas.name)) -- dynamically-set defaults
+ AND 'DEFAULT_INITDB' != ALL(pg_settings_get_flags(psas.name)) -- dynamically-set defaults
+ AND NOT (sc.sample_value ~ '^0' AND current_setting(name) ~ '^0') -- zeros may be written differently
+ AND NOT (sc.sample_value='60s' AND current_setting(name) = '1min') -- two ways to write 1min
+ AND name NOT IN ('krb_server_keyfile', 'wal_retrieve_retry_interval', 'log_autovacuum_min_duration'); -- exceptions
+ ");
+
+my @check_defaults_array = split("\n", lc($check_defaults));
+print(STDERR "$check_defaults\n");
+
+is (@check_defaults_array, 0,
+ 'check for consistency of defaults in postgresql.conf.sample');
+
done_testing();
--
2.25.1
On Tue, Oct 25, 2022 at 9:05 AM Justin Pryzby <pryzby@telsasoft.com> wrote:
On Mon, Sep 26, 2022 at 05:29:58PM +0900, Kyotaro Horiguchi wrote:
At Sat, 17 Sep 2022 23:53:07 -0500, Justin Pryzby <pryzby@telsasoft.com> wrote in
...
Rebased the patch.
I also split the flag into DEFAULTS_COMPILE and DEFAULTS_INITDB, since
that makes it easier to understand what the flags mean and the intent of
the patch. And maybe allows fewer exclusions in patches like Peter's,
which I think would only want to exclude compile-time defaults.
Thanks!
FYI, I'm making use of this patch now as a prerequisite for my GUC C
var sanity-checker [1]/messages/by-id/CAHut+Pss16YBiYYKyrZBvSp_4uSQfCy7aYfDXU0N8w5VZ5dd_g@mail.gmail.com.
------
[1]: /messages/by-id/CAHut+Pss16YBiYYKyrZBvSp_4uSQfCy7aYfDXU0N8w5VZ5dd_g@mail.gmail.com
Kind Regards,
Peter Smith.
Fujitsu Australia
@cfbot: re-rebased again
Attachments:
0001-pg_settings_get_flags-add-DEFAULT_COMPILE-and-DEFAUL.patchtext/x-diff; charset=us-asciiDownload
From e1f0940e7682052b2ec9d58c361f56630aa1742e Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Tue, 19 Jul 2022 08:31:56 -0500
Subject: [PATCH 1/2] pg_settings_get_flags(): add DEFAULT_COMPILE and
DEFAULT_INITDB ..
for settings which vary by ./configure or platform or initdb
Note that this is independent from PGC_S_DYNAMIC_DEFAULT.
---
doc/src/sgml/func.sgml | 15 ++++++
src/backend/utils/misc/guc_funcs.c | 6 ++-
src/backend/utils/misc/guc_tables.c | 76 ++++++++++++++++++-----------
src/include/utils/guc.h | 2 +
4 files changed, 69 insertions(+), 30 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 82fba48d5f7..67c68fc9292 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -24377,6 +24377,21 @@ SELECT collation for ('foo' COLLATE "de_DE");
<row><entry>Flag</entry><entry>Description</entry></row>
</thead>
<tbody>
+
+ <row>
+ <entry><literal>DEFAULT_COMPILE</literal></entry>
+ <entry>Parameters with this flag have default values which vary by
+ platform, or compile-time options.
+ </entry>
+ </row>
+
+ <row>
+ <entry><literal>DEFAULT_INITDB</literal></entry>
+ <entry>Parameters with this flag have default values which are
+ determined dynamically during initdb.
+ </entry>
+ </row>
+
<row>
<entry><literal>EXPLAIN</literal></entry>
<entry>Parameters with this flag are included in
diff --git a/src/backend/utils/misc/guc_funcs.c b/src/backend/utils/misc/guc_funcs.c
index 108b3bd1290..2b666e8d014 100644
--- a/src/backend/utils/misc/guc_funcs.c
+++ b/src/backend/utils/misc/guc_funcs.c
@@ -538,7 +538,7 @@ ShowAllGUCConfig(DestReceiver *dest)
Datum
pg_settings_get_flags(PG_FUNCTION_ARGS)
{
-#define MAX_GUC_FLAGS 6
+#define MAX_GUC_FLAGS 8
char *varname = TextDatumGetCString(PG_GETARG_DATUM(0));
struct config_generic *record;
int cnt = 0;
@@ -551,6 +551,10 @@ pg_settings_get_flags(PG_FUNCTION_ARGS)
if (record == NULL)
PG_RETURN_NULL();
+ if (record->flags & GUC_DEFAULT_COMPILE)
+ flags[cnt++] = CStringGetTextDatum("DEFAULT_COMPILE");
+ if (record->flags & GUC_DEFAULT_INITDB)
+ flags[cnt++] = CStringGetTextDatum("DEFAULT_INITDB");
if (record->flags & GUC_EXPLAIN)
flags[cnt++] = CStringGetTextDatum("EXPLAIN");
if (record->flags & GUC_NO_RESET)
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index e5cc4e3205a..6bd206c164a 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -1205,7 +1205,7 @@ struct config_bool ConfigureNamesBool[] =
{"debug_assertions", PGC_INTERNAL, PRESET_OPTIONS,
gettext_noop("Shows whether the running server has assertion checks enabled."),
NULL,
- GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE
+ GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE | GUC_DEFAULT_COMPILE
},
&assert_enabled,
DEFAULT_ASSERT_ENABLED,
@@ -1377,7 +1377,8 @@ struct config_bool ConfigureNamesBool[] =
{
{"update_process_title", PGC_SUSET, PROCESS_TITLE,
gettext_noop("Updates the process title to show the active SQL command."),
- gettext_noop("Enables updating of the process title every time a new SQL command is received by the server.")
+ gettext_noop("Enables updating of the process title every time a new SQL command is received by the server."),
+ GUC_DEFAULT_COMPILE
},
&update_process_title,
DEFAULT_UPDATE_PROCESS_TITLE,
@@ -2122,7 +2123,8 @@ struct config_int ConfigureNamesInt[] =
{
{"max_connections", PGC_POSTMASTER, CONN_AUTH_SETTINGS,
gettext_noop("Sets the maximum number of concurrent connections."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
},
&MaxConnections,
100, 1, MAX_BACKENDS,
@@ -2159,7 +2161,7 @@ struct config_int ConfigureNamesInt[] =
{"shared_buffers", PGC_POSTMASTER, RESOURCES_MEM,
gettext_noop("Sets the number of shared memory buffers used by the server."),
NULL,
- GUC_UNIT_BLOCKS
+ GUC_UNIT_BLOCKS | GUC_DEFAULT_INITDB
},
&NBuffers,
16384, 16, INT_MAX / 2,
@@ -2202,7 +2204,8 @@ struct config_int ConfigureNamesInt[] =
{
{"port", PGC_POSTMASTER, CONN_AUTH_SETTINGS,
gettext_noop("Sets the TCP port the server listens on."),
- NULL
+ NULL,
+ GUC_DEFAULT_COMPILE
},
&PostPortNumber,
DEF_PGPORT, 1, 65535,
@@ -2231,7 +2234,8 @@ struct config_int ConfigureNamesInt[] =
"to be a numeric mode specification in the form "
"accepted by the chmod and umask system calls. "
"(To use the customary octal format the number must "
- "start with a 0 (zero).)")
+ "start with a 0 (zero).)"),
+ GUC_DEFAULT_INITDB
},
&Log_file_mode,
0600, 0000, 0777,
@@ -2673,7 +2677,7 @@ struct config_int ConfigureNamesInt[] =
{"checkpoint_flush_after", PGC_SIGHUP, WAL_CHECKPOINTS,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
NULL,
- GUC_UNIT_BLOCKS
+ GUC_UNIT_BLOCKS | GUC_DEFAULT_COMPILE
},
&checkpoint_flush_after,
DEFAULT_CHECKPOINT_FLUSH_AFTER, 0, WRITEBACK_MAX_PENDING_FLUSHES,
@@ -2891,7 +2895,7 @@ struct config_int ConfigureNamesInt[] =
{"bgwriter_flush_after", PGC_SIGHUP, RESOURCES_BGWRITER,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
NULL,
- GUC_UNIT_BLOCKS
+ GUC_UNIT_BLOCKS | GUC_DEFAULT_COMPILE
},
&bgwriter_flush_after,
DEFAULT_BGWRITER_FLUSH_AFTER, 0, WRITEBACK_MAX_PENDING_FLUSHES,
@@ -2904,7 +2908,7 @@ struct config_int ConfigureNamesInt[] =
RESOURCES_ASYNCHRONOUS,
gettext_noop("Number of simultaneous requests that can be handled efficiently by the disk subsystem."),
NULL,
- GUC_EXPLAIN
+ GUC_EXPLAIN | GUC_DEFAULT_COMPILE
},
&effective_io_concurrency,
DEFAULT_EFFECTIVE_IO_CONCURRENCY,
@@ -2918,7 +2922,7 @@ struct config_int ConfigureNamesInt[] =
RESOURCES_ASYNCHRONOUS,
gettext_noop("A variant of effective_io_concurrency that is used for maintenance work."),
NULL,
- GUC_EXPLAIN
+ GUC_EXPLAIN | GUC_DEFAULT_COMPILE
},
&maintenance_io_concurrency,
DEFAULT_MAINTENANCE_IO_CONCURRENCY,
@@ -2931,7 +2935,7 @@ struct config_int ConfigureNamesInt[] =
{"backend_flush_after", PGC_USERSET, RESOURCES_ASYNCHRONOUS,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
NULL,
- GUC_UNIT_BLOCKS
+ GUC_UNIT_BLOCKS | GUC_DEFAULT_COMPILE
},
&backend_flush_after,
DEFAULT_BACKEND_FLUSH_AFTER, 0, WRITEBACK_MAX_PENDING_FLUSHES,
@@ -3385,7 +3389,7 @@ struct config_int ConfigureNamesInt[] =
{"debug_discard_caches", PGC_SUSET, DEVELOPER_OPTIONS,
gettext_noop("Aggressively flush system caches for debugging purposes."),
NULL,
- GUC_NOT_IN_SAMPLE
+ GUC_NOT_IN_SAMPLE | GUC_DEFAULT_COMPILE
},
&debug_discard_caches,
#ifdef DISCARD_CACHES_ENABLED
@@ -3878,7 +3882,8 @@ struct config_string ConfigureNamesString[] =
{
{"log_timezone", PGC_SIGHUP, LOGGING_WHAT,
gettext_noop("Sets the time zone to use in log messages."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
},
&log_timezone_string,
"GMT",
@@ -3890,7 +3895,7 @@ struct config_string ConfigureNamesString[] =
gettext_noop("Sets the display format for date and time values."),
gettext_noop("Also controls interpretation of ambiguous "
"date inputs."),
- GUC_LIST_INPUT | GUC_REPORT
+ GUC_LIST_INPUT | GUC_REPORT | GUC_DEFAULT_INITDB
},
&datestyle_string,
"ISO, MDY",
@@ -3992,7 +3997,8 @@ struct config_string ConfigureNamesString[] =
{
{"lc_messages", PGC_SUSET, CLIENT_CONN_LOCALE,
gettext_noop("Sets the language in which messages are displayed."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
},
&locale_messages,
"",
@@ -4002,7 +4008,8 @@ struct config_string ConfigureNamesString[] =
{
{"lc_monetary", PGC_USERSET, CLIENT_CONN_LOCALE,
gettext_noop("Sets the locale for formatting monetary amounts."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
},
&locale_monetary,
"C",
@@ -4012,7 +4019,8 @@ struct config_string ConfigureNamesString[] =
{
{"lc_numeric", PGC_USERSET, CLIENT_CONN_LOCALE,
gettext_noop("Sets the locale for formatting numbers."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
},
&locale_numeric,
"C",
@@ -4022,7 +4030,8 @@ struct config_string ConfigureNamesString[] =
{
{"lc_time", PGC_USERSET, CLIENT_CONN_LOCALE,
gettext_noop("Sets the locale for formatting date and time values."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
},
&locale_time,
"C",
@@ -4181,7 +4190,8 @@ struct config_string ConfigureNamesString[] =
{"TimeZone", PGC_USERSET, CLIENT_CONN_LOCALE,
gettext_noop("Sets the time zone for displaying and interpreting time stamps."),
NULL,
- GUC_REPORT
+ GUC_REPORT | GUC_DEFAULT_INITDB
+
},
&timezone_string,
"GMT",
@@ -4212,7 +4222,7 @@ struct config_string ConfigureNamesString[] =
{"unix_socket_directories", PGC_POSTMASTER, CONN_AUTH_SETTINGS,
gettext_noop("Sets the directories where Unix-domain sockets will be created."),
NULL,
- GUC_LIST_INPUT | GUC_LIST_QUOTE | GUC_SUPERUSER_ONLY
+ GUC_LIST_INPUT | GUC_LIST_QUOTE | GUC_SUPERUSER_ONLY | GUC_DEFAULT_COMPILE
},
&Unix_socket_directories,
DEFAULT_PGSOCKET_DIR,
@@ -4293,7 +4303,7 @@ struct config_string ConfigureNamesString[] =
{"ssl_library", PGC_INTERNAL, PRESET_OPTIONS,
gettext_noop("Shows the name of the SSL library."),
NULL,
- GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE
+ GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE | GUC_DEFAULT_COMPILE
},
&ssl_library,
#ifdef USE_SSL
@@ -4368,7 +4378,9 @@ struct config_string ConfigureNamesString[] =
{
{"default_text_search_config", PGC_USERSET, CLIENT_CONN_LOCALE,
gettext_noop("Sets default text search configuration."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
+
},
&TSCurrentConfig,
"pg_catalog.simple",
@@ -4379,7 +4391,7 @@ struct config_string ConfigureNamesString[] =
{"ssl_ciphers", PGC_SIGHUP, CONN_AUTH_SSL,
gettext_noop("Sets the list of allowed SSL ciphers."),
NULL,
- GUC_SUPERUSER_ONLY
+ GUC_SUPERUSER_ONLY | GUC_DEFAULT_COMPILE
},
&SSLCipherSuites,
#ifdef USE_OPENSSL
@@ -4394,7 +4406,7 @@ struct config_string ConfigureNamesString[] =
{"ssl_ecdh_curve", PGC_SIGHUP, CONN_AUTH_SSL,
gettext_noop("Sets the curve to use for ECDH."),
NULL,
- GUC_SUPERUSER_ONLY
+ GUC_SUPERUSER_ONLY | GUC_DEFAULT_COMPILE
},
&SSLECDHCurve,
#ifdef USE_SSL
@@ -4632,7 +4644,8 @@ struct config_enum ConfigureNamesEnum[] =
{
{"syslog_facility", PGC_SIGHUP, LOGGING_WHERE,
gettext_noop("Sets the syslog \"facility\" to be used when syslog enabled."),
- NULL
+ NULL,
+ GUC_DEFAULT_COMPILE
},
&syslog_facility,
DEFAULT_SYSLOG_FACILITY,
@@ -4741,7 +4754,9 @@ struct config_enum ConfigureNamesEnum[] =
{
{"dynamic_shared_memory_type", PGC_POSTMASTER, RESOURCES_MEM,
gettext_noop("Selects the dynamic shared memory implementation used."),
- NULL
+ NULL,
+ /* platform-specific default, which is also overriden during initdb */
+ GUC_DEFAULT_COMPILE | GUC_DEFAULT_INITDB
},
&dynamic_shared_memory_type,
DEFAULT_DYNAMIC_SHARED_MEMORY_TYPE, dynamic_shared_memory_options,
@@ -4751,7 +4766,8 @@ struct config_enum ConfigureNamesEnum[] =
{
{"shared_memory_type", PGC_POSTMASTER, RESOURCES_MEM,
gettext_noop("Selects the shared memory implementation used for the main shared memory region."),
- NULL
+ NULL,
+ GUC_DEFAULT_COMPILE
},
&shared_memory_type,
DEFAULT_SHARED_MEMORY_TYPE, shared_memory_options,
@@ -4761,7 +4777,8 @@ struct config_enum ConfigureNamesEnum[] =
{
{"wal_sync_method", PGC_SIGHUP, WAL_SETTINGS,
gettext_noop("Selects the method used for forcing WAL updates to disk."),
- NULL
+ NULL,
+ GUC_DEFAULT_COMPILE
},
&sync_method,
DEFAULT_SYNC_METHOD, sync_method_options,
@@ -4823,7 +4840,8 @@ struct config_enum ConfigureNamesEnum[] =
{
{"password_encryption", PGC_USERSET, CONN_AUTH_AUTH,
gettext_noop("Chooses the algorithm for encrypting passwords."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
},
&Password_encryption,
PASSWORD_TYPE_SCRAM_SHA_256, password_encryption_options,
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index b3aaff9665b..79053583a03 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -221,6 +221,8 @@ typedef enum
#define GUC_DISALLOW_IN_AUTO_FILE \
0x002000 /* can't set in PG_AUTOCONF_FILENAME */
#define GUC_RUNTIME_COMPUTED 0x004000 /* delay processing in 'postgres -C' */
+#define GUC_DEFAULT_COMPILE 0x008000 /* default is determined at compile time */
+#define GUC_DEFAULT_INITDB 0x010000 /* default is determined at during initdb */
#define GUC_UNIT_KB 0x01000000 /* value is in kilobytes */
#define GUC_UNIT_BLOCKS 0x02000000 /* value is in blocks */
--
2.25.1
0002-WIP-test-GUC-default-values.patchtext/x-diff; charset=us-asciiDownload
From 8ba199da40fc44c851224d6b088a15e4464c2419 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Wed, 25 May 2022 05:14:43 -0500
Subject: [PATCH 2/2] WIP: test GUC default values
---
src/backend/utils/misc/postgresql.conf.sample | 2 +-
src/test/modules/test_misc/t/003_check_guc.pl | 33 +++++++++++++++++++
2 files changed, 34 insertions(+), 1 deletion(-)
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 1ce0dee6d04..21bfbef3291 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -720,7 +720,7 @@
# share/timezonesets/.
#extra_float_digits = 1 # min -15, max 3; any value >0 actually
# selects precise output mode
-#client_encoding = sql_ascii # actually, defaults to database
+#client_encoding = SQL_ASCII # actually, defaults to database
# encoding
# These settings are initialized by initdb, but they can be changed.
diff --git a/src/test/modules/test_misc/t/003_check_guc.pl b/src/test/modules/test_misc/t/003_check_guc.pl
index 1786cd19299..22e8612350c 100644
--- a/src/test/modules/test_misc/t/003_check_guc.pl
+++ b/src/test/modules/test_misc/t/003_check_guc.pl
@@ -106,4 +106,37 @@ foreach my $param (@sample_intersect)
);
}
+# Test that GUCs in postgresql.conf.sample show the correct default values
+my $check_defaults = $node->safe_psql(
+ 'postgres',
+ "
+ CREATE TABLE sample_conf AS
+ SELECT m[1] AS name, COALESCE(m[3], m[5]) AS sample_value
+ FROM (SELECT regexp_split_to_table(pg_read_file('$sample_file'), '\n') AS ln) conf,
+ regexp_match(ln, '^#?([_[:alpha:]]+) (= ''([^'']*)''|(= ([^[:space:]]*))).*') AS m
+ WHERE ln ~ '^#?[[:alpha:]]'
+ ");
+
+$check_defaults = $node->safe_psql(
+ 'postgres',
+ "
+ SELECT name, current_setting(name), sc.sample_value, boot_val
+ FROM pg_show_all_settings() psas
+ JOIN sample_conf sc USING(name)
+ WHERE sc.sample_value != boot_val -- same value (specified by Cluster.pm)
+ AND sc.sample_value != current_setting(name) -- same value, with human units
+ AND sc.sample_value != current_setting(name)||'.0' -- same value with .0 suffix
+ AND 'DEFAULT_COMPILE' != ALL(pg_settings_get_flags(psas.name)) -- dynamically-set defaults
+ AND 'DEFAULT_INITDB' != ALL(pg_settings_get_flags(psas.name)) -- dynamically-set defaults
+ AND NOT (sc.sample_value ~ '^0' AND current_setting(name) ~ '^0') -- zeros may be written differently
+ AND NOT (sc.sample_value='60s' AND current_setting(name) = '1min') -- two ways to write 1min
+ AND name NOT IN ('krb_server_keyfile', 'wal_retrieve_retry_interval', 'log_autovacuum_min_duration'); -- exceptions
+ ");
+
+my @check_defaults_array = split("\n", lc($check_defaults));
+print(STDERR "$check_defaults\n");
+
+is (@check_defaults_array, 0,
+ 'check for consistency of defaults in postgresql.conf.sample');
+
done_testing();
--
2.25.1
On Wed, Jul 13, 2022 at 04:49:00PM -0700, Andres Freund wrote:
On 2022-07-14 08:46:02 +0900, Michael Paquier wrote:
On Wed, Jul 13, 2022 at 12:30:00PM -0500, Justin Pryzby wrote:
How did you make this list ? Was it by excluding things that failed for you ?
cfbot is currently failing due to io_concurrency on windows.
I think there are more GUC which should be included here.FWIW, I am not really a fan of making this test depend on a hardcoded
list of GUCs.I wonder if we should add flags indicating platform dependency etc to guc.c?
That should allow to remove most of them?
Michael commented on this, but on another thread, so I'm copying and
pasting it here.
On Thu, Mar 23, 2023 at 08:59:57PM -0500, Justin Pryzby wrote:
On Fri, Mar 24, 2023 at 10:24:43AM +0900, Michael Paquier wrote:
* Check consistency of GUC defaults between .sample.conf and pg_settings.boot_val
- It looks like this was pretty active until last October and might
have been ready to apply at least partially? But no further work or
review has happened since.FWIW, I don't find much appealing the addition of two GUC flags for
only the sole purpose of that,The flags seem independently interesting - adding them here follows
a suggestion Andres made in response to your complaint.
20220713234900.z4rniuaerkq34s4v@awork3.anarazel.departicularly as we get a stronger
dependency between GUCs that can be switched dynamically at
initialization and at compile-time.What do you mean by "stronger dependency between GUCs" ?
I'm still not clear what that means ?
I updated the patch to handle the GUC added at 1671f990d.
--
Justin
Attachments:
0001-pg_settings_get_flags-add-DEFAULT_COMPILE-and-DEFAUL.patchtext/x-diff; charset=us-asciiDownload
From b38dce442fa20439c6b75b9f4bac20d52d3dd94b Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Tue, 19 Jul 2022 08:31:56 -0500
Subject: [PATCH 1/2] pg_settings_get_flags(): add DEFAULT_COMPILE and
DEFAULT_INITDB ..
for settings which vary by ./configure or platform or initdb
Note that this is independent from PGC_S_DYNAMIC_DEFAULT.
---
doc/src/sgml/func.sgml | 15 ++++++
src/backend/utils/misc/guc_funcs.c | 6 ++-
src/backend/utils/misc/guc_tables.c | 76 ++++++++++++++++++-----------
src/include/utils/guc.h | 2 +
4 files changed, 69 insertions(+), 30 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index d599b243b10..66451a6c759 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -24825,6 +24825,21 @@ SELECT collation for ('foo' COLLATE "de_DE");
<row><entry>Flag</entry><entry>Description</entry></row>
</thead>
<tbody>
+
+ <row>
+ <entry><literal>DEFAULT_COMPILE</literal></entry>
+ <entry>Parameters with this flag have default values which vary by
+ platform, or compile-time options.
+ </entry>
+ </row>
+
+ <row>
+ <entry><literal>DEFAULT_INITDB</literal></entry>
+ <entry>Parameters with this flag have default values which are
+ determined dynamically during initdb.
+ </entry>
+ </row>
+
<row>
<entry><literal>EXPLAIN</literal></entry>
<entry>Parameters with this flag are included in
diff --git a/src/backend/utils/misc/guc_funcs.c b/src/backend/utils/misc/guc_funcs.c
index 2ce1e53ec54..70fdce3868d 100644
--- a/src/backend/utils/misc/guc_funcs.c
+++ b/src/backend/utils/misc/guc_funcs.c
@@ -577,7 +577,7 @@ ShowAllGUCConfig(DestReceiver *dest)
Datum
pg_settings_get_flags(PG_FUNCTION_ARGS)
{
-#define MAX_GUC_FLAGS 6
+#define MAX_GUC_FLAGS 8
char *varname = TextDatumGetCString(PG_GETARG_DATUM(0));
struct config_generic *record;
int cnt = 0;
@@ -590,6 +590,10 @@ pg_settings_get_flags(PG_FUNCTION_ARGS)
if (record == NULL)
PG_RETURN_NULL();
+ if (record->flags & GUC_DEFAULT_COMPILE)
+ flags[cnt++] = CStringGetTextDatum("DEFAULT_COMPILE");
+ if (record->flags & GUC_DEFAULT_INITDB)
+ flags[cnt++] = CStringGetTextDatum("DEFAULT_INITDB");
if (record->flags & GUC_EXPLAIN)
flags[cnt++] = CStringGetTextDatum("EXPLAIN");
if (record->flags & GUC_NO_RESET)
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 2c3061b7359..0d6c782a1df 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -1233,7 +1233,7 @@ struct config_bool ConfigureNamesBool[] =
{"debug_assertions", PGC_INTERNAL, PRESET_OPTIONS,
gettext_noop("Shows whether the running server has assertion checks enabled."),
NULL,
- GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE
+ GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE | GUC_DEFAULT_COMPILE
},
&assert_enabled,
DEFAULT_ASSERT_ENABLED,
@@ -1425,7 +1425,8 @@ struct config_bool ConfigureNamesBool[] =
{
{"update_process_title", PGC_SUSET, PROCESS_TITLE,
gettext_noop("Updates the process title to show the active SQL command."),
- gettext_noop("Enables updating of the process title every time a new SQL command is received by the server.")
+ gettext_noop("Enables updating of the process title every time a new SQL command is received by the server."),
+ GUC_DEFAULT_COMPILE
},
&update_process_title,
DEFAULT_UPDATE_PROCESS_TITLE,
@@ -2170,7 +2171,8 @@ struct config_int ConfigureNamesInt[] =
{
{"max_connections", PGC_POSTMASTER, CONN_AUTH_SETTINGS,
gettext_noop("Sets the maximum number of concurrent connections."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
},
&MaxConnections,
100, 1, MAX_BACKENDS,
@@ -2218,7 +2220,7 @@ struct config_int ConfigureNamesInt[] =
{"shared_buffers", PGC_POSTMASTER, RESOURCES_MEM,
gettext_noop("Sets the number of shared memory buffers used by the server."),
NULL,
- GUC_UNIT_BLOCKS
+ GUC_UNIT_BLOCKS | GUC_DEFAULT_INITDB
},
&NBuffers,
16384, 16, INT_MAX / 2,
@@ -2261,7 +2263,8 @@ struct config_int ConfigureNamesInt[] =
{
{"port", PGC_POSTMASTER, CONN_AUTH_SETTINGS,
gettext_noop("Sets the TCP port the server listens on."),
- NULL
+ NULL,
+ GUC_DEFAULT_COMPILE
},
&PostPortNumber,
DEF_PGPORT, 1, 65535,
@@ -2290,7 +2293,8 @@ struct config_int ConfigureNamesInt[] =
"to be a numeric mode specification in the form "
"accepted by the chmod and umask system calls. "
"(To use the customary octal format the number must "
- "start with a 0 (zero).)")
+ "start with a 0 (zero).)"),
+ GUC_DEFAULT_INITDB
},
&Log_file_mode,
0600, 0000, 0777,
@@ -2723,7 +2727,7 @@ struct config_int ConfigureNamesInt[] =
{"checkpoint_flush_after", PGC_SIGHUP, WAL_CHECKPOINTS,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
NULL,
- GUC_UNIT_BLOCKS
+ GUC_UNIT_BLOCKS | GUC_DEFAULT_COMPILE
},
&checkpoint_flush_after,
DEFAULT_CHECKPOINT_FLUSH_AFTER, 0, WRITEBACK_MAX_PENDING_FLUSHES,
@@ -2941,7 +2945,7 @@ struct config_int ConfigureNamesInt[] =
{"bgwriter_flush_after", PGC_SIGHUP, RESOURCES_BGWRITER,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
NULL,
- GUC_UNIT_BLOCKS
+ GUC_UNIT_BLOCKS | GUC_DEFAULT_COMPILE
},
&bgwriter_flush_after,
DEFAULT_BGWRITER_FLUSH_AFTER, 0, WRITEBACK_MAX_PENDING_FLUSHES,
@@ -2954,7 +2958,7 @@ struct config_int ConfigureNamesInt[] =
RESOURCES_ASYNCHRONOUS,
gettext_noop("Number of simultaneous requests that can be handled efficiently by the disk subsystem."),
NULL,
- GUC_EXPLAIN
+ GUC_EXPLAIN | GUC_DEFAULT_COMPILE
},
&effective_io_concurrency,
DEFAULT_EFFECTIVE_IO_CONCURRENCY,
@@ -2968,7 +2972,7 @@ struct config_int ConfigureNamesInt[] =
RESOURCES_ASYNCHRONOUS,
gettext_noop("A variant of effective_io_concurrency that is used for maintenance work."),
NULL,
- GUC_EXPLAIN
+ GUC_EXPLAIN | GUC_DEFAULT_COMPILE
},
&maintenance_io_concurrency,
DEFAULT_MAINTENANCE_IO_CONCURRENCY,
@@ -2981,7 +2985,7 @@ struct config_int ConfigureNamesInt[] =
{"backend_flush_after", PGC_USERSET, RESOURCES_ASYNCHRONOUS,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
NULL,
- GUC_UNIT_BLOCKS
+ GUC_UNIT_BLOCKS | GUC_DEFAULT_COMPILE
},
&backend_flush_after,
DEFAULT_BACKEND_FLUSH_AFTER, 0, WRITEBACK_MAX_PENDING_FLUSHES,
@@ -3436,7 +3440,7 @@ struct config_int ConfigureNamesInt[] =
{"debug_discard_caches", PGC_SUSET, DEVELOPER_OPTIONS,
gettext_noop("Aggressively flush system caches for debugging purposes."),
NULL,
- GUC_NOT_IN_SAMPLE
+ GUC_NOT_IN_SAMPLE | GUC_DEFAULT_COMPILE
},
&debug_discard_caches,
#ifdef DISCARD_CACHES_ENABLED
@@ -3930,7 +3934,8 @@ struct config_string ConfigureNamesString[] =
{
{"log_timezone", PGC_SIGHUP, LOGGING_WHAT,
gettext_noop("Sets the time zone to use in log messages."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
},
&log_timezone_string,
"GMT",
@@ -3942,7 +3947,7 @@ struct config_string ConfigureNamesString[] =
gettext_noop("Sets the display format for date and time values."),
gettext_noop("Also controls interpretation of ambiguous "
"date inputs."),
- GUC_LIST_INPUT | GUC_REPORT
+ GUC_LIST_INPUT | GUC_REPORT | GUC_DEFAULT_INITDB
},
&datestyle_string,
"ISO, MDY",
@@ -4056,7 +4061,8 @@ struct config_string ConfigureNamesString[] =
{
{"lc_messages", PGC_SUSET, CLIENT_CONN_LOCALE,
gettext_noop("Sets the language in which messages are displayed."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
},
&locale_messages,
"",
@@ -4066,7 +4072,8 @@ struct config_string ConfigureNamesString[] =
{
{"lc_monetary", PGC_USERSET, CLIENT_CONN_LOCALE,
gettext_noop("Sets the locale for formatting monetary amounts."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
},
&locale_monetary,
"C",
@@ -4076,7 +4083,8 @@ struct config_string ConfigureNamesString[] =
{
{"lc_numeric", PGC_USERSET, CLIENT_CONN_LOCALE,
gettext_noop("Sets the locale for formatting numbers."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
},
&locale_numeric,
"C",
@@ -4086,7 +4094,8 @@ struct config_string ConfigureNamesString[] =
{
{"lc_time", PGC_USERSET, CLIENT_CONN_LOCALE,
gettext_noop("Sets the locale for formatting date and time values."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
},
&locale_time,
"C",
@@ -4245,7 +4254,8 @@ struct config_string ConfigureNamesString[] =
{"TimeZone", PGC_USERSET, CLIENT_CONN_LOCALE,
gettext_noop("Sets the time zone for displaying and interpreting time stamps."),
NULL,
- GUC_REPORT
+ GUC_REPORT | GUC_DEFAULT_INITDB
+
},
&timezone_string,
"GMT",
@@ -4276,7 +4286,7 @@ struct config_string ConfigureNamesString[] =
{"unix_socket_directories", PGC_POSTMASTER, CONN_AUTH_SETTINGS,
gettext_noop("Sets the directories where Unix-domain sockets will be created."),
NULL,
- GUC_LIST_INPUT | GUC_LIST_QUOTE | GUC_SUPERUSER_ONLY
+ GUC_LIST_INPUT | GUC_LIST_QUOTE | GUC_SUPERUSER_ONLY | GUC_DEFAULT_COMPILE
},
&Unix_socket_directories,
DEFAULT_PGSOCKET_DIR,
@@ -4357,7 +4367,7 @@ struct config_string ConfigureNamesString[] =
{"ssl_library", PGC_INTERNAL, PRESET_OPTIONS,
gettext_noop("Shows the name of the SSL library."),
NULL,
- GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE
+ GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE | GUC_DEFAULT_COMPILE
},
&ssl_library,
#ifdef USE_SSL
@@ -4432,7 +4442,9 @@ struct config_string ConfigureNamesString[] =
{
{"default_text_search_config", PGC_USERSET, CLIENT_CONN_LOCALE,
gettext_noop("Sets default text search configuration."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
+
},
&TSCurrentConfig,
"pg_catalog.simple",
@@ -4443,7 +4455,7 @@ struct config_string ConfigureNamesString[] =
{"ssl_ciphers", PGC_SIGHUP, CONN_AUTH_SSL,
gettext_noop("Sets the list of allowed SSL ciphers."),
NULL,
- GUC_SUPERUSER_ONLY
+ GUC_SUPERUSER_ONLY | GUC_DEFAULT_COMPILE
},
&SSLCipherSuites,
#ifdef USE_OPENSSL
@@ -4458,7 +4470,7 @@ struct config_string ConfigureNamesString[] =
{"ssl_ecdh_curve", PGC_SIGHUP, CONN_AUTH_SSL,
gettext_noop("Sets the curve to use for ECDH."),
NULL,
- GUC_SUPERUSER_ONLY
+ GUC_SUPERUSER_ONLY | GUC_DEFAULT_COMPILE
},
&SSLECDHCurve,
#ifdef USE_SSL
@@ -4717,7 +4729,8 @@ struct config_enum ConfigureNamesEnum[] =
{
{"syslog_facility", PGC_SIGHUP, LOGGING_WHERE,
gettext_noop("Sets the syslog \"facility\" to be used when syslog enabled."),
- NULL
+ NULL,
+ GUC_DEFAULT_COMPILE
},
&syslog_facility,
DEFAULT_SYSLOG_FACILITY,
@@ -4826,7 +4839,9 @@ struct config_enum ConfigureNamesEnum[] =
{
{"dynamic_shared_memory_type", PGC_POSTMASTER, RESOURCES_MEM,
gettext_noop("Selects the dynamic shared memory implementation used."),
- NULL
+ NULL,
+ /* platform-specific default, which is also overriden during initdb */
+ GUC_DEFAULT_COMPILE | GUC_DEFAULT_INITDB
},
&dynamic_shared_memory_type,
DEFAULT_DYNAMIC_SHARED_MEMORY_TYPE, dynamic_shared_memory_options,
@@ -4836,7 +4851,8 @@ struct config_enum ConfigureNamesEnum[] =
{
{"shared_memory_type", PGC_POSTMASTER, RESOURCES_MEM,
gettext_noop("Selects the shared memory implementation used for the main shared memory region."),
- NULL
+ NULL,
+ GUC_DEFAULT_COMPILE
},
&shared_memory_type,
DEFAULT_SHARED_MEMORY_TYPE, shared_memory_options,
@@ -4846,7 +4862,8 @@ struct config_enum ConfigureNamesEnum[] =
{
{"wal_sync_method", PGC_SIGHUP, WAL_SETTINGS,
gettext_noop("Selects the method used for forcing WAL updates to disk."),
- NULL
+ NULL,
+ GUC_DEFAULT_COMPILE
},
&sync_method,
DEFAULT_SYNC_METHOD, sync_method_options,
@@ -4910,7 +4927,8 @@ struct config_enum ConfigureNamesEnum[] =
{
{"password_encryption", PGC_USERSET, CONN_AUTH_AUTH,
gettext_noop("Chooses the algorithm for encrypting passwords."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
},
&Password_encryption,
PASSWORD_TYPE_SCRAM_SHA_256, password_encryption_options,
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 15c61992f06..2b17cd08fd9 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -222,6 +222,8 @@ typedef enum
#define GUC_DISALLOW_IN_AUTO_FILE \
0x002000 /* can't set in PG_AUTOCONF_FILENAME */
#define GUC_RUNTIME_COMPUTED 0x004000 /* delay processing in 'postgres -C' */
+#define GUC_DEFAULT_COMPILE 0x008000 /* default is determined at compile time */
+#define GUC_DEFAULT_INITDB 0x010000 /* default is determined at during initdb */
#define GUC_UNIT_KB 0x01000000 /* value is in kilobytes */
#define GUC_UNIT_BLOCKS 0x02000000 /* value is in blocks */
--
2.34.1
0002-test-GUC-default-values-in-postgresql.conf.sample.patchtext/x-diff; charset=us-asciiDownload
From 0cc261a37af071554a0e2954b3a49ed6b3be2b60 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Wed, 25 May 2022 05:14:43 -0500
Subject: [PATCH 2/2] test GUC default values in postgresql.conf.sample
---
src/test/modules/test_misc/t/003_check_guc.pl | 33 +++++++++++++++++++
1 file changed, 33 insertions(+)
diff --git a/src/test/modules/test_misc/t/003_check_guc.pl b/src/test/modules/test_misc/t/003_check_guc.pl
index e9f33f3c775..ab6f16dad7b 100644
--- a/src/test/modules/test_misc/t/003_check_guc.pl
+++ b/src/test/modules/test_misc/t/003_check_guc.pl
@@ -106,4 +106,37 @@ foreach my $param (@sample_intersect)
);
}
+# Test that GUCs in postgresql.conf.sample show the correct default values
+my $check_defaults = $node->safe_psql(
+ 'postgres',
+ "
+ CREATE TABLE sample_conf AS
+ SELECT m[1] AS name, COALESCE(m[3], m[5]) AS sample_value
+ FROM (SELECT regexp_split_to_table(pg_read_file('$sample_file'), '\n') AS ln) conf,
+ regexp_match(ln, '^#?([_[:alpha:]]+) (= ''([^'']*)''|(= ([^[:space:]]*))).*') AS m
+ WHERE ln ~ '^#?[[:alpha:]]'
+ ");
+
+$check_defaults = $node->safe_psql(
+ 'postgres',
+ "
+ SELECT name, current_setting(name), sc.sample_value, boot_val
+ FROM pg_show_all_settings() psas
+ JOIN sample_conf sc USING(name)
+ WHERE lower(sc.sample_value) != lower(boot_val) -- same value (specified by Cluster.pm)
+ AND sc.sample_value != current_setting(name) -- same value, with human units
+ AND sc.sample_value != current_setting(name)||'.0' -- same value with .0 suffix
+ AND 'DEFAULT_COMPILE' != ALL(pg_settings_get_flags(psas.name)) -- dynamically-set defaults
+ AND 'DEFAULT_INITDB' != ALL(pg_settings_get_flags(psas.name)) -- dynamically-set defaults
+ AND NOT (sc.sample_value ~ '^0' AND current_setting(name) ~ '^0') -- zeros may be written differently
+ AND NOT (sc.sample_value='60s' AND current_setting(name) = '1min') -- two ways to write 1min
+ AND name NOT IN ('krb_server_keyfile', 'wal_retrieve_retry_interval'); -- exceptions
+ ");
+
+my @check_defaults_array = split("\n", lc($check_defaults));
+print(STDERR "$check_defaults\n");
+
+is (@check_defaults_array, 0,
+ 'check for consistency of defaults in postgresql.conf.sample');
+
done_testing();
--
2.34.1
On Wed, Mar 29, 2023 at 11:03:59PM -0500, Justin Pryzby wrote:
On Wed, Jul 13, 2022 at 04:49:00PM -0700, Andres Freund wrote:
On 2022-07-14 08:46:02 +0900, Michael Paquier wrote:
On Wed, Jul 13, 2022 at 12:30:00PM -0500, Justin Pryzby wrote:
How did you make this list ? Was it by excluding things that failed for you ?
cfbot is currently failing due to io_concurrency on windows.
I think there are more GUC which should be included here.FWIW, I am not really a fan of making this test depend on a hardcoded
list of GUCs.I wonder if we should add flags indicating platform dependency etc to guc.c?
That should allow to remove most of them?Michael commented on this, but on another thread, so I'm copying and
pasting it here.On Thu, Mar 23, 2023 at 08:59:57PM -0500, Justin Pryzby wrote:
On Fri, Mar 24, 2023 at 10:24:43AM +0900, Michael Paquier wrote:
* Check consistency of GUC defaults between .sample.conf and pg_settings.boot_val
- It looks like this was pretty active until last October and might
have been ready to apply at least partially? But no further work or
review has happened since.FWIW, I don't find much appealing the addition of two GUC flags for
only the sole purpose of that,The flags seem independently interesting - adding them here follows
a suggestion Andres made in response to your complaint.
20220713234900.z4rniuaerkq34s4v@awork3.anarazel.departicularly as we get a stronger
dependency between GUCs that can be switched dynamically at
initialization and at compile-time.What do you mean by "stronger dependency between GUCs" ?
I'm still not clear what that means ?
Michael ?
This fixes an issue with the last version that failed with
log_autovacuum_min_duration in cirrusci's pg_ci_base.conf.
And now includes both a perl and a sql-based versions of the test - both
of which rely on the flags.
Attachments:
0001-pg_settings_get_flags-add-DEFAULT_COMPILE-and-DEFAUL.patchtext/x-diff; charset=us-asciiDownload
From 963b56636b9f7fd4a78e502c6acba07919518910 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Tue, 19 Jul 2022 08:31:56 -0500
Subject: [PATCH 1/2] pg_settings_get_flags(): add DEFAULT_COMPILE and
DEFAULT_INITDB ..
for settings which vary by ./configure or platform or initdb
Note that this is independent from PGC_S_DYNAMIC_DEFAULT.
---
doc/src/sgml/func.sgml | 15 ++++++
src/backend/utils/misc/guc_funcs.c | 6 ++-
src/backend/utils/misc/guc_tables.c | 76 ++++++++++++++++++-----------
src/include/utils/guc.h | 2 +
4 files changed, 69 insertions(+), 30 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index e13fce6f6b1..17069d2249e 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -24945,6 +24945,21 @@ SELECT collation for ('foo' COLLATE "de_DE");
<row><entry>Flag</entry><entry>Description</entry></row>
</thead>
<tbody>
+
+ <row>
+ <entry><literal>DEFAULT_COMPILE</literal></entry>
+ <entry>Parameters with this flag have default values which vary by
+ platform, or compile-time options.
+ </entry>
+ </row>
+
+ <row>
+ <entry><literal>DEFAULT_INITDB</literal></entry>
+ <entry>Parameters with this flag have default values which are
+ determined dynamically during initdb.
+ </entry>
+ </row>
+
<row>
<entry><literal>EXPLAIN</literal></entry>
<entry>Parameters with this flag are included in
diff --git a/src/backend/utils/misc/guc_funcs.c b/src/backend/utils/misc/guc_funcs.c
index 52c361e9755..183943c1973 100644
--- a/src/backend/utils/misc/guc_funcs.c
+++ b/src/backend/utils/misc/guc_funcs.c
@@ -551,7 +551,7 @@ ShowAllGUCConfig(DestReceiver *dest)
Datum
pg_settings_get_flags(PG_FUNCTION_ARGS)
{
-#define MAX_GUC_FLAGS 6
+#define MAX_GUC_FLAGS 8
char *varname = TextDatumGetCString(PG_GETARG_DATUM(0));
struct config_generic *record;
int cnt = 0;
@@ -564,6 +564,10 @@ pg_settings_get_flags(PG_FUNCTION_ARGS)
if (record == NULL)
PG_RETURN_NULL();
+ if (record->flags & GUC_DEFAULT_COMPILE)
+ flags[cnt++] = CStringGetTextDatum("DEFAULT_COMPILE");
+ if (record->flags & GUC_DEFAULT_INITDB)
+ flags[cnt++] = CStringGetTextDatum("DEFAULT_INITDB");
if (record->flags & GUC_EXPLAIN)
flags[cnt++] = CStringGetTextDatum("EXPLAIN");
if (record->flags & GUC_NO_RESET)
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 2f42cebaf62..94b0aa24a98 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -1233,7 +1233,7 @@ struct config_bool ConfigureNamesBool[] =
{"debug_assertions", PGC_INTERNAL, PRESET_OPTIONS,
gettext_noop("Shows whether the running server has assertion checks enabled."),
NULL,
- GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE
+ GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE | GUC_DEFAULT_COMPILE
},
&assert_enabled,
DEFAULT_ASSERT_ENABLED,
@@ -1425,7 +1425,8 @@ struct config_bool ConfigureNamesBool[] =
{
{"update_process_title", PGC_SUSET, PROCESS_TITLE,
gettext_noop("Updates the process title to show the active SQL command."),
- gettext_noop("Enables updating of the process title every time a new SQL command is received by the server.")
+ gettext_noop("Enables updating of the process title every time a new SQL command is received by the server."),
+ GUC_DEFAULT_COMPILE
},
&update_process_title,
DEFAULT_UPDATE_PROCESS_TITLE,
@@ -2180,7 +2181,8 @@ struct config_int ConfigureNamesInt[] =
{
{"max_connections", PGC_POSTMASTER, CONN_AUTH_SETTINGS,
gettext_noop("Sets the maximum number of concurrent connections."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
},
&MaxConnections,
100, 1, MAX_BACKENDS,
@@ -2228,7 +2230,7 @@ struct config_int ConfigureNamesInt[] =
{"shared_buffers", PGC_POSTMASTER, RESOURCES_MEM,
gettext_noop("Sets the number of shared memory buffers used by the server."),
NULL,
- GUC_UNIT_BLOCKS
+ GUC_UNIT_BLOCKS | GUC_DEFAULT_INITDB
},
&NBuffers,
16384, 16, INT_MAX / 2,
@@ -2282,7 +2284,8 @@ struct config_int ConfigureNamesInt[] =
{
{"port", PGC_POSTMASTER, CONN_AUTH_SETTINGS,
gettext_noop("Sets the TCP port the server listens on."),
- NULL
+ NULL,
+ GUC_DEFAULT_COMPILE
},
&PostPortNumber,
DEF_PGPORT, 1, 65535,
@@ -2311,7 +2314,8 @@ struct config_int ConfigureNamesInt[] =
"to be a numeric mode specification in the form "
"accepted by the chmod and umask system calls. "
"(To use the customary octal format the number must "
- "start with a 0 (zero).)")
+ "start with a 0 (zero).)"),
+ GUC_DEFAULT_INITDB
},
&Log_file_mode,
0600, 0000, 0777,
@@ -2744,7 +2748,7 @@ struct config_int ConfigureNamesInt[] =
{"checkpoint_flush_after", PGC_SIGHUP, WAL_CHECKPOINTS,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
NULL,
- GUC_UNIT_BLOCKS
+ GUC_UNIT_BLOCKS | GUC_DEFAULT_COMPILE
},
&checkpoint_flush_after,
DEFAULT_CHECKPOINT_FLUSH_AFTER, 0, WRITEBACK_MAX_PENDING_FLUSHES,
@@ -2962,7 +2966,7 @@ struct config_int ConfigureNamesInt[] =
{"bgwriter_flush_after", PGC_SIGHUP, RESOURCES_BGWRITER,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
NULL,
- GUC_UNIT_BLOCKS
+ GUC_UNIT_BLOCKS | GUC_DEFAULT_COMPILE
},
&bgwriter_flush_after,
DEFAULT_BGWRITER_FLUSH_AFTER, 0, WRITEBACK_MAX_PENDING_FLUSHES,
@@ -2975,7 +2979,7 @@ struct config_int ConfigureNamesInt[] =
RESOURCES_ASYNCHRONOUS,
gettext_noop("Number of simultaneous requests that can be handled efficiently by the disk subsystem."),
NULL,
- GUC_EXPLAIN
+ GUC_EXPLAIN | GUC_DEFAULT_COMPILE
},
&effective_io_concurrency,
DEFAULT_EFFECTIVE_IO_CONCURRENCY,
@@ -2989,7 +2993,7 @@ struct config_int ConfigureNamesInt[] =
RESOURCES_ASYNCHRONOUS,
gettext_noop("A variant of effective_io_concurrency that is used for maintenance work."),
NULL,
- GUC_EXPLAIN
+ GUC_EXPLAIN | GUC_DEFAULT_COMPILE
},
&maintenance_io_concurrency,
DEFAULT_MAINTENANCE_IO_CONCURRENCY,
@@ -3002,7 +3006,7 @@ struct config_int ConfigureNamesInt[] =
{"backend_flush_after", PGC_USERSET, RESOURCES_ASYNCHRONOUS,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
NULL,
- GUC_UNIT_BLOCKS
+ GUC_UNIT_BLOCKS | GUC_DEFAULT_COMPILE
},
&backend_flush_after,
DEFAULT_BACKEND_FLUSH_AFTER, 0, WRITEBACK_MAX_PENDING_FLUSHES,
@@ -3457,7 +3461,7 @@ struct config_int ConfigureNamesInt[] =
{"debug_discard_caches", PGC_SUSET, DEVELOPER_OPTIONS,
gettext_noop("Aggressively flush system caches for debugging purposes."),
NULL,
- GUC_NOT_IN_SAMPLE
+ GUC_NOT_IN_SAMPLE | GUC_DEFAULT_COMPILE
},
&debug_discard_caches,
#ifdef DISCARD_CACHES_ENABLED
@@ -3951,7 +3955,8 @@ struct config_string ConfigureNamesString[] =
{
{"log_timezone", PGC_SIGHUP, LOGGING_WHAT,
gettext_noop("Sets the time zone to use in log messages."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
},
&log_timezone_string,
"GMT",
@@ -3963,7 +3968,7 @@ struct config_string ConfigureNamesString[] =
gettext_noop("Sets the display format for date and time values."),
gettext_noop("Also controls interpretation of ambiguous "
"date inputs."),
- GUC_LIST_INPUT | GUC_REPORT
+ GUC_LIST_INPUT | GUC_REPORT | GUC_DEFAULT_INITDB
},
&datestyle_string,
"ISO, MDY",
@@ -4077,7 +4082,8 @@ struct config_string ConfigureNamesString[] =
{
{"lc_messages", PGC_SUSET, CLIENT_CONN_LOCALE,
gettext_noop("Sets the language in which messages are displayed."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
},
&locale_messages,
"",
@@ -4087,7 +4093,8 @@ struct config_string ConfigureNamesString[] =
{
{"lc_monetary", PGC_USERSET, CLIENT_CONN_LOCALE,
gettext_noop("Sets the locale for formatting monetary amounts."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
},
&locale_monetary,
"C",
@@ -4097,7 +4104,8 @@ struct config_string ConfigureNamesString[] =
{
{"lc_numeric", PGC_USERSET, CLIENT_CONN_LOCALE,
gettext_noop("Sets the locale for formatting numbers."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
},
&locale_numeric,
"C",
@@ -4107,7 +4115,8 @@ struct config_string ConfigureNamesString[] =
{
{"lc_time", PGC_USERSET, CLIENT_CONN_LOCALE,
gettext_noop("Sets the locale for formatting date and time values."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
},
&locale_time,
"C",
@@ -4266,7 +4275,8 @@ struct config_string ConfigureNamesString[] =
{"TimeZone", PGC_USERSET, CLIENT_CONN_LOCALE,
gettext_noop("Sets the time zone for displaying and interpreting time stamps."),
NULL,
- GUC_REPORT
+ GUC_REPORT | GUC_DEFAULT_INITDB
+
},
&timezone_string,
"GMT",
@@ -4297,7 +4307,7 @@ struct config_string ConfigureNamesString[] =
{"unix_socket_directories", PGC_POSTMASTER, CONN_AUTH_SETTINGS,
gettext_noop("Sets the directories where Unix-domain sockets will be created."),
NULL,
- GUC_LIST_INPUT | GUC_LIST_QUOTE | GUC_SUPERUSER_ONLY
+ GUC_LIST_INPUT | GUC_LIST_QUOTE | GUC_SUPERUSER_ONLY | GUC_DEFAULT_COMPILE
},
&Unix_socket_directories,
DEFAULT_PGSOCKET_DIR,
@@ -4378,7 +4388,7 @@ struct config_string ConfigureNamesString[] =
{"ssl_library", PGC_INTERNAL, PRESET_OPTIONS,
gettext_noop("Shows the name of the SSL library."),
NULL,
- GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE
+ GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE | GUC_DEFAULT_COMPILE
},
&ssl_library,
#ifdef USE_SSL
@@ -4453,7 +4463,9 @@ struct config_string ConfigureNamesString[] =
{
{"default_text_search_config", PGC_USERSET, CLIENT_CONN_LOCALE,
gettext_noop("Sets default text search configuration."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
+
},
&TSCurrentConfig,
"pg_catalog.simple",
@@ -4464,7 +4476,7 @@ struct config_string ConfigureNamesString[] =
{"ssl_ciphers", PGC_SIGHUP, CONN_AUTH_SSL,
gettext_noop("Sets the list of allowed SSL ciphers."),
NULL,
- GUC_SUPERUSER_ONLY
+ GUC_SUPERUSER_ONLY | GUC_DEFAULT_COMPILE
},
&SSLCipherSuites,
#ifdef USE_OPENSSL
@@ -4479,7 +4491,7 @@ struct config_string ConfigureNamesString[] =
{"ssl_ecdh_curve", PGC_SIGHUP, CONN_AUTH_SSL,
gettext_noop("Sets the curve to use for ECDH."),
NULL,
- GUC_SUPERUSER_ONLY
+ GUC_SUPERUSER_ONLY | GUC_DEFAULT_COMPILE
},
&SSLECDHCurve,
#ifdef USE_SSL
@@ -4738,7 +4750,8 @@ struct config_enum ConfigureNamesEnum[] =
{
{"syslog_facility", PGC_SIGHUP, LOGGING_WHERE,
gettext_noop("Sets the syslog \"facility\" to be used when syslog enabled."),
- NULL
+ NULL,
+ GUC_DEFAULT_COMPILE
},
&syslog_facility,
DEFAULT_SYSLOG_FACILITY,
@@ -4847,7 +4860,9 @@ struct config_enum ConfigureNamesEnum[] =
{
{"dynamic_shared_memory_type", PGC_POSTMASTER, RESOURCES_MEM,
gettext_noop("Selects the dynamic shared memory implementation used."),
- NULL
+ NULL,
+ /* platform-specific default, which is also overriden during initdb */
+ GUC_DEFAULT_COMPILE | GUC_DEFAULT_INITDB
},
&dynamic_shared_memory_type,
DEFAULT_DYNAMIC_SHARED_MEMORY_TYPE, dynamic_shared_memory_options,
@@ -4857,7 +4872,8 @@ struct config_enum ConfigureNamesEnum[] =
{
{"shared_memory_type", PGC_POSTMASTER, RESOURCES_MEM,
gettext_noop("Selects the shared memory implementation used for the main shared memory region."),
- NULL
+ NULL,
+ GUC_DEFAULT_COMPILE
},
&shared_memory_type,
DEFAULT_SHARED_MEMORY_TYPE, shared_memory_options,
@@ -4867,7 +4883,8 @@ struct config_enum ConfigureNamesEnum[] =
{
{"wal_sync_method", PGC_SIGHUP, WAL_SETTINGS,
gettext_noop("Selects the method used for forcing WAL updates to disk."),
- NULL
+ NULL,
+ GUC_DEFAULT_COMPILE
},
&sync_method,
DEFAULT_SYNC_METHOD, sync_method_options,
@@ -4931,7 +4948,8 @@ struct config_enum ConfigureNamesEnum[] =
{
{"password_encryption", PGC_USERSET, CONN_AUTH_AUTH,
gettext_noop("Chooses the algorithm for encrypting passwords."),
- NULL
+ NULL,
+ GUC_DEFAULT_INITDB
},
&Password_encryption,
PASSWORD_TYPE_SCRAM_SHA_256, password_encryption_options,
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index ba89d013e64..2b98595b3ea 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -221,6 +221,8 @@ typedef enum
#define GUC_DISALLOW_IN_AUTO_FILE \
0x002000 /* can't set in PG_AUTOCONF_FILENAME */
#define GUC_RUNTIME_COMPUTED 0x004000 /* delay processing in 'postgres -C' */
+#define GUC_DEFAULT_COMPILE 0x008000 /* default is determined at compile time */
+#define GUC_DEFAULT_INITDB 0x010000 /* default is determined at during initdb */
#define GUC_UNIT_KB 0x01000000 /* value is in kilobytes */
#define GUC_UNIT_BLOCKS 0x02000000 /* value is in blocks */
--
2.34.1
0002-WIP-test-GUC-default-values-in-postgresql.conf.sampl.patchtext/x-diff; charset=us-asciiDownload
From 107e96e12dd25c8360470bd4b65f975925300d86 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Wed, 25 May 2022 05:14:43 -0500
Subject: [PATCH 2/2] WIP: test GUC default values in postgresql.conf.sample
This tests for consistency between the default values written in
postgresql.conf.sample and defined in guc.c.
Kyotaro Horiguchi and Justin Pryzby
//-os-only: freebsd, macos
---
src/backend/utils/misc/guc.c | 80 +++++++++++++++
src/include/catalog/pg_proc.dat | 5 +
src/test/modules/test_misc/t/003_check_guc.pl | 99 +++++++++++++++++--
.../unsafe_tests/expected/rolenames.out | 14 +++
.../modules/unsafe_tests/sql/rolenames.sql | 7 ++
5 files changed, 199 insertions(+), 6 deletions(-)
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 53d1d9a06a7..9eef85a4fa4 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -3239,6 +3239,86 @@ parse_and_validate_value(struct config_generic *record,
return true;
}
+/*
+ * Helper for pg_config_unitless_value
+ *
+ * Return the the value of the enum at the given index, or NULL if not found.
+ */
+static const char *
+config_enum_lookup_by_value_soft(struct config_enum *record, int val)
+{
+ const struct config_enum_entry *entry;
+
+ for (entry = record->options; entry && entry->name; entry++)
+ {
+ if (entry->val == val)
+ return entry->name;
+ }
+
+ return NULL;
+}
+
+/*
+ * Convert value to unitless value according to the specified GUC variable
+ */
+Datum
+pg_config_unitless_value(PG_FUNCTION_ARGS)
+{
+ char *name;
+ char *value;
+ struct config_generic *record;
+ const char *result = "";
+ void *extra;
+ union config_var_val val;
+ char buffer[256];
+
+ name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ value = text_to_cstring(PG_GETARG_TEXT_PP(1));
+
+ record = find_option(name, true, false, ERROR);
+
+ /*
+ * This function doesn't reveal values of the variables, but be consistent
+ * with similar functions.
+ */
+ if ((record->flags & GUC_SUPERUSER_ONLY) &&
+ !ConfigOptionIsVisible(record))
+ ereport(ERROR,
+ errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+ errmsg("permission denied to examine \"%s\"", name),
+ errdetail("Only roles with privileges of the \"%s\" role may examine this parameter.",
+ "pg_read_all_settings"));
+
+ parse_and_validate_value(record, name, value, PGC_S_TEST, WARNING,
+ &val, &extra);
+
+ switch (record->vartype)
+ {
+ case PGC_BOOL:
+ result = (val.boolval ? "on" : "off");
+ break;
+ case PGC_INT:
+ snprintf(buffer, sizeof(buffer), "%d", val.intval);
+ result = buffer;
+ break;
+ case PGC_REAL:
+ snprintf(buffer, sizeof(buffer), "%g", val.realval);
+ result = buffer;
+ break;
+ case PGC_STRING:
+ result = val.stringval;
+ break;
+ case PGC_ENUM:
+ result = config_enum_lookup_by_value_soft((struct config_enum *) record,
+ val.intval);
+ break;
+ }
+
+ if (result == NULL)
+ PG_RETURN_NULL();
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
/*
* set_config_option: sets option `name' to given value.
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index aa293c43b6a..fe82629b1f6 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6242,6 +6242,11 @@
proname => 'pg_settings_get_flags', provolatile => 's', prorettype => '_text',
proargtypes => 'text', prosrc => 'pg_settings_get_flags' },
+{ oid => '9956', descr => 'normalize value to the unit of specified GUC',
+ proname => 'pg_config_unitless_value', provolatile => 's',
+ prorettype => 'text', proargtypes => 'text text',
+ proargnames => '{varname,value}', prosrc => 'pg_config_unitless_value' },
+
{ oid => '3329', descr => 'show config file settings',
proname => 'pg_show_all_file_settings', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
diff --git a/src/test/modules/test_misc/t/003_check_guc.pl b/src/test/modules/test_misc/t/003_check_guc.pl
index e9f33f3c775..ce2ec1bf9ba 100644
--- a/src/test/modules/test_misc/t/003_check_guc.pl
+++ b/src/test/modules/test_misc/t/003_check_guc.pl
@@ -11,6 +11,26 @@ my $node = PostgreSQL::Test::Cluster->new('main');
$node->init;
$node->start;
+# These are non-variables but that are mistakenly parsed as variable
+# settings in the loop below.
+my %skip_names =
+ map { $_ => 1 } ('include', 'include_dir', 'include_if_exists');
+
+# The following parameters have defaults which are
+# environment-dependent and may not match the default
+# values written in the sample config file.
+my %ignore_parameters =
+ map { $_ => 1 } (
+ 'data_directory',
+ 'hba_file',
+ 'ident_file',
+ 'krb_server_keyfile',
+ 'timezone_abbreviations',
+ 'lc_messages',
+ 'max_stack_depth', # XXX
+ 'wal_buffers', # XXX
+ );
+
# Grab the names of all the parameters that can be listed in the
# configuration sample file. config_file is an exception, it is not
# in postgresql.conf.sample but is part of the lists from guc_tables.c.
@@ -42,7 +62,7 @@ my @gucs_in_file;
# Read the sample file line-by-line, checking its contents to build a list
# of everything known as a GUC.
-my $num_tests = 0;
+my @file_vals = ();
open(my $contents, '<', $sample_file)
|| die "Could not open $sample_file: $!";
while (my $line = <$contents>)
@@ -52,19 +72,28 @@ while (my $line = <$contents>)
# file.
# - Valid configuration options are followed immediately by " = ",
# with one space before and after the equal sign.
- if ($line =~ m/^#?([_[:alpha:]]+) = .*/)
+ if ($line =~ m/^#?([_[:alpha:]]+) = (.*)$/)
{
# Lower-case conversion matters for some of the GUCs.
my $param_name = lc($1);
- # Ignore some exceptions.
- next if $param_name eq "include";
- next if $param_name eq "include_dir";
- next if $param_name eq "include_if_exists";
+ # extract value
+ my $file_value = $2;
+ $file_value =~ s/\s*#.*$//; # strip trailing comment
+ $file_value =~ s/^'(.*)'$/$1/; # strip quotes
+
+ next if (defined $skip_names{$param_name});
# Update the list of GUCs found in the sample file, for the
# follow-up tests.
push @gucs_in_file, $param_name;
+
+ # Update the list of GUCs whose value is checked for consistency
+ # between the sample file and pg_setting.boot_val
+ if (!defined $ignore_parameters{$param_name})
+ {
+ push(@file_vals, [$param_name, $file_value]);
+ }
}
}
@@ -106,4 +135,62 @@ foreach my $param (@sample_intersect)
);
}
+# Test that GUCs in postgresql.conf.sample show the correct default values
+my $check_defaults = $node->safe_psql(
+ 'postgres',
+ "
+ CREATE TABLE sample_conf AS
+ SELECT m[1] AS name, COALESCE(m[3], m[5]) AS sample_value
+ FROM (SELECT regexp_split_to_table(pg_read_file('$sample_file'), '\n') AS ln) conf,
+ regexp_match(ln, '^#?([_[:alpha:]]+) (= ''([^'']*)''|(= ([^[:space:]]*))).*') AS m
+ WHERE ln ~ '^#?[[:alpha:]]'
+ ");
+
+$check_defaults = $node->safe_psql(
+ 'postgres',
+ "
+ SELECT name, sc.sample_value, boot_val
+ FROM pg_settings ps
+ JOIN sample_conf sc USING(name)
+ JOIN pg_settings_get_flags(ps.name) AS flags ON true
+ WHERE boot_val != sc.sample_value -- same value
+ AND boot_val != pg_config_unitless_value(name, sc.sample_value) -- same value with different units
+ AND 'DEFAULT_COMPILE' != ALL(flags) -- dynamically-set defaults
+ AND 'DEFAULT_INITDB' != ALL(flags) -- dynamically-set defaults
+ AND name NOT IN ('max_stack_depth', 'krb_server_keyfile'); -- exceptions
+ ");
+
+my @check_defaults_array = split("\n", lc($check_defaults));
+
+is (@check_defaults_array, 0,
+ "check for consistency of defaults in postgresql.conf.sample: $check_defaults");
+
+# XXX An alternative, perl implementation of the same thing:
+
+# Check if GUC values in config-file and boot value match
+my $values = $node->safe_psql(
+ 'postgres',
+ 'SELECT f.n, pg_config_unitless_value(f.n, f.v), s.boot_val, \'!\' '.
+ 'FROM (VALUES '.
+ join(',', map { "('${$_}[0]','${$_}[1]')" } @file_vals).
+ ') f(n,v) '.
+ "JOIN pg_settings s ON (s.name = f.n)".
+ "JOIN pg_settings_get_flags(s.name) AS flags ON true ".
+ "AND 'DEFAULT_COMPILE' != ALL(flags) -- dynamically-set defaults ".
+ "AND 'DEFAULT_INITDB' != ALL(flags) -- dynamically-set defaults");
+
+my $fails = "";
+foreach my $l (split("\n", $values))
+{
+ # $l: <varname>|<fileval>|<boot_val>|!
+ my @t = split("\\|", $l);
+ if ($t[1] ne $t[2])
+ {
+ $fails .= "\n" if ($fails ne "");
+ $fails .= "$t[0]: file \"$t[1]\" != boot_val \"$t[2]\"";
+ }
+}
+
+is($fails, "", "check if GUC values in .sample and boot value match");
+
done_testing();
diff --git a/src/test/modules/unsafe_tests/expected/rolenames.out b/src/test/modules/unsafe_tests/expected/rolenames.out
index 61396b2a805..1bba9c87239 100644
--- a/src/test/modules/unsafe_tests/expected/rolenames.out
+++ b/src/test/modules/unsafe_tests/expected/rolenames.out
@@ -1082,6 +1082,20 @@ DETAIL: Only roles with privileges of the "pg_read_all_settings" role may exami
RESET SESSION AUTHORIZATION;
ERROR: current transaction is aborted, commands ignored until end of transaction block
ROLLBACK;
+BEGIN;
+SET SESSION AUTHORIZATION regress_role_haspriv;
+-- passes with role member of pg_read_all_settings
+SELECT pg_config_unitless_value('session_preload_libraries', 'val');
+ pg_config_unitless_value
+--------------------------
+ val
+(1 row)
+
+SET SESSION AUTHORIZATION regress_role_nopriv;
+SELECT pg_config_unitless_value('session_preload_libraries', 'val');
+ERROR: permission denied to examine "session_preload_libraries"
+DETAIL: Only roles with privileges of the "pg_read_all_settings" role may examine this parameter.
+ROLLBACK;
REVOKE pg_read_all_settings FROM regress_role_haspriv;
-- clean up
\c
diff --git a/src/test/modules/unsafe_tests/sql/rolenames.sql b/src/test/modules/unsafe_tests/sql/rolenames.sql
index adac36536db..355aa32c2ac 100644
--- a/src/test/modules/unsafe_tests/sql/rolenames.sql
+++ b/src/test/modules/unsafe_tests/sql/rolenames.sql
@@ -492,6 +492,13 @@ SET SESSION AUTHORIZATION regress_role_nopriv;
SHOW session_preload_libraries;
RESET SESSION AUTHORIZATION;
ROLLBACK;
+BEGIN;
+SET SESSION AUTHORIZATION regress_role_haspriv;
+-- passes with role member of pg_read_all_settings
+SELECT pg_config_unitless_value('session_preload_libraries', 'val');
+SET SESSION AUTHORIZATION regress_role_nopriv;
+SELECT pg_config_unitless_value('session_preload_libraries', 'val');
+ROLLBACK;
REVOKE pg_read_all_settings FROM regress_role_haspriv;
-- clean up
--
2.34.1
On Wed, 10 May 2023 at 06:07, Justin Pryzby <pryzby@telsasoft.com> wrote:
On Wed, Mar 29, 2023 at 11:03:59PM -0500, Justin Pryzby wrote:
On Wed, Jul 13, 2022 at 04:49:00PM -0700, Andres Freund wrote:
On 2022-07-14 08:46:02 +0900, Michael Paquier wrote:
On Wed, Jul 13, 2022 at 12:30:00PM -0500, Justin Pryzby wrote:
How did you make this list ? Was it by excluding things that failed for you ?
cfbot is currently failing due to io_concurrency on windows.
I think there are more GUC which should be included here.FWIW, I am not really a fan of making this test depend on a hardcoded
list of GUCs.I wonder if we should add flags indicating platform dependency etc to guc.c?
That should allow to remove most of them?Michael commented on this, but on another thread, so I'm copying and
pasting it here.On Thu, Mar 23, 2023 at 08:59:57PM -0500, Justin Pryzby wrote:
On Fri, Mar 24, 2023 at 10:24:43AM +0900, Michael Paquier wrote:
* Check consistency of GUC defaults between .sample.conf and pg_settings.boot_val
- It looks like this was pretty active until last October and might
have been ready to apply at least partially? But no further work or
review has happened since.FWIW, I don't find much appealing the addition of two GUC flags for
only the sole purpose of that,The flags seem independently interesting - adding them here follows
a suggestion Andres made in response to your complaint.
20220713234900.z4rniuaerkq34s4v@awork3.anarazel.departicularly as we get a stronger
dependency between GUCs that can be switched dynamically at
initialization and at compile-time.What do you mean by "stronger dependency between GUCs" ?
I'm still not clear what that means ?
Michael ?
This fixes an issue with the last version that failed with
log_autovacuum_min_duration in cirrusci's pg_ci_base.conf.And now includes both a perl and a sql-based versions of the test - both
of which rely on the flags.
I'm seeing that there has been no activity in this thread for more
than 8 months, I'm planning to close this in the current commitfest
unless someone is planning to take it forward.
Regards,
Vignesh
On Sat, Jan 20, 2024 at 07:59:22AM +0530, vignesh C wrote:
I'm seeing that there has been no activity in this thread for more
than 8 months, I'm planning to close this in the current commitfest
unless someone is planning to take it forward.
Thanks, that seems right to me.
I have been looking again at the patch after seeing your reply (spent
some time looking at it but I could not decide what to do), and I am
not really excited with the amount of new facilities this requires in
the TAP test (especially the list of hardcoded parameters that may
change) and the backend-side changes for the GUC flags as well as the
requirements to make the checks flexible enough to work across initdb
and platform-dependent default values. In short, I'm happy to let
003_check_guc.pl be what check_guc was able to do (script gone in
cf29a11ef646) for the parameter names.
--
Michael
On Sat, 20 Jan 2024 at 08:39, Michael Paquier <michael@paquier.xyz> wrote:
On Sat, Jan 20, 2024 at 07:59:22AM +0530, vignesh C wrote:
I'm seeing that there has been no activity in this thread for more
than 8 months, I'm planning to close this in the current commitfest
unless someone is planning to take it forward.Thanks, that seems right to me.
Thanks, I have updated the commitfest entry to "returned with
feedback". Feel free to start a new entry when someone wants to pursue
it further more actively.
Regards,
Vignesh