WIP: a way forward on bootstrap data
Hi,
I was looking through the archives one day for a few topics that
interest me, and saw there was continued interest in making bootstrap
data less painful [1]/messages/by-id/20150220234142.GH12653@awork2.anarazel.de [2]/messages/by-id/CAGoxFiFeW64k4t95Ez2udXZmKA+tazUFAaSTtYQLM4Jhzw+-pg@mail.gmail.com. There were quite a few good ideas thrown
around in those threads, but not much in the way of concrete results.
I took a few of them as a starting point and threw together the
attached WIP patchset.
==
An overview (warning: long):
1 through 3 are small tweaks worth doing even without a data format
change. 4 through 7 are separated for readability and flexibility, but
should be understood as one big patch. I tried to credit as many
people's ideas as possible. Some things are left undone until basic
agreement is reached.
--
Patch 1 - Minor corrections
--
Patch 2 - Minor cleanup
Be more consistent style-wise, change a confusing variable name, fix
perltidy junk.
--
Patch 3 - Remove hard-coded schema information about pg_attribute from
genbki.pl.
This means slightly less code maintenance, but more importantly it's a
proving ground for mechanisms used in later patches.
1. Label a column's default value in the catalog header [3]/messages/by-id/20161113171017.7sgaqdeq6jslmsr3@alap3.anarazel.de.
Note: I implemented it in the simplest way possible for now, which
happens to prevents a column from having both FORCE_(NOT_)NULL and a
default at the same time, but I think in practice that would almost
never matter. If more column options are added in the future, this
will have to be rewritten.
2. Add a new function to Catalog.pm to fill in a tuple with default
values. It will complain loudly if it can't find either a default or a
given value, so change the signature of emit_pgattr_row() so we can
pass a partially built tuple to it.
3. Format the schema macro entries according to their types.
4. Commit 8137f2c32322c624e0431fac1621e8e9315202f9 labeled which
columns are variable length. Expose that label so we can exclude those
columns from schema macros in a general fashion.
--
Patch 4 - Infrastructure for the data conversion
1. Copy a modified version of Catalogs() from Catalog.pm to a new
script that turns DATA()/(SH)DESCR() statements into serialized Perl
data structures in pg_*.dat files, preserving comments along the way.
Toast and index statements are unaffected. Although it's a one-off as
far as the core project is concerned, I imagine third-parties might
want access to this tool, so it's in the patch and not separate.
2. Remove data parsing from the original Catalogs() function and
rename it to reflect its new, limited role of extracting the schema
info from a header. The data files are handled by a new function.
3. Introduce a script to rewrite pg_*.dat files in a standard format
[4]: /messages/by-id/D8F1D509-6498-43AC-BEFC-052DFE16847A@2ndquadrant.com
lines. It can also change default values on the fly. It is intended to
be run when adding new data.
4. Add default values to a few catalog headers for demonstration
purposes. There might be some questionable choices here. Review is
appreciated.
Note: At this point, the build is broken.
TODO: See what pgindent does to the the modified headers.
--
Patch 5 - Mechanical data conversion
This is the result of:
cd src/include/catalog
perl convert_header2dat.pl list-of-catalog-headers-from-the-Makefile
perl -I ../../backend/catalog rewrite_dat.pl *.dat
rm *.bak
Note: The data has been stripped of all double-quotes for readability,
since the Perl hash literals have single quotes everywhere. Patches 6
and 7 restore them where needed.
--
Patch 6 - Hand edits
Re-doublequote values that are macros expanded by initdb.c, remove
stray comments, fix up whitespace, and do a minimum of comment editing
to reflect the new data format.
At this point the files are ready to take a look at. Format is the
central issue, of course. I tried to structure things so it wouldn't
be a huge lift to bikeshed on the format. See pg_authid.dat for a
conveniently small example. Each entry is 1 or 2 lines long, depending
on whether oid or (sh)descr is present.
Regarding pg_proc.dat, I think readability is improved, and to some
extent line length, although it's not great:
pg_proc.h: avg=175, stdev=25
pg_proc.dat: avg=159, stdev=43
(grep -E '^DATA' pg_proc.h | awk '{print length}'
grep prosrc pg_proc.dat | awk '{print length}')
Many lines now fit in an editor window, but the longest line has
ballooned from 576 chars to 692. I made proparallel default to 'u' for
safety, but the vast majority are 's'. If we risked making 's' the
default, most entries would shrink by 20 chars. On the other hand,
human-readable types would inflate that again, but maybe patch 8 below
can help with editing. There was some discussion about abbreviated
labels that were mapped to column names - I haven't thought about that
yet.
--
Patch 7 - Distprep scripts
1. Teach genbki.pl and Gen_fmgrtab.pl to read the data files, and
arrange for the former to double-quote certain values so bootscanner.l
can read them.
2. Introduce Makefile dependencies on the data files.
The build works now.
Note: Since the original DATA() entries contained some (as far as I
can tell) useless quotes, the postgres.bki diff won't be zero, but it
will be close.
Performance note: On my laptop, running Make in the catalog dir with
no compilation goes from ~700ms on master to ~1000ms with the new data
files.
--
Patch 8 - SQL output
1. Write out postgres.sql, which allows you to insert all the source
BKI data into a development catalog schema for viewing and possibly
bulk-editing [5]/messages/by-id/20150304150712.GV29780@tamriel.snowman.net. It retains oids, and (sh)descr fields in their own
columns, and implements default values.
2. Make it a distclean target.
TODO: Find a way to dump dev catalog tuples into the canonical format.
--
Not implemented yet:
-Gut the header files of DATA() statements. I'm thinking we should
keep the #defines in the headers, but see below:
-Update README and documentation
--
Future work:
-More lookups for human-readable types, operators, etc.
-Automate pg_type #defines a la pg_proc [6]/messages/by-id/15697.1479161432@sss.pgh.pa.us, which could also be used
to maintain ecpg's copy of pg_type #defines.
--
[1]: /messages/by-id/20150220234142.GH12653@awork2.anarazel.de
[2]: /messages/by-id/CAGoxFiFeW64k4t95Ez2udXZmKA+tazUFAaSTtYQLM4Jhzw+-pg@mail.gmail.com
[3]: /messages/by-id/20161113171017.7sgaqdeq6jslmsr3@alap3.anarazel.de
[4]: /messages/by-id/D8F1D509-6498-43AC-BEFC-052DFE16847A@2ndquadrant.com
[5]: /messages/by-id/20150304150712.GV29780@tamriel.snowman.net
[6]: /messages/by-id/15697.1479161432@sss.pgh.pa.us
==
I'll add this to the next commitfest soon.
-John Naylor
Attachments:
0001_corrections_v1.patchtext/x-patch; charset=US-ASCII; name=0001_corrections_v1.patchDownload
src/backend/catalog/Catalog.pm | 3 ++-
src/backend/catalog/Makefile | 2 +-
src/backend/utils/Gen_fmgrtab.pl | 3 ++-
src/include/catalog/pg_partitioned_table.h | 2 +-
src/include/catalog/pg_sequence.h | 10 ++++++++++
src/include/catalog/pg_subscription_rel.h | 5 +----
6 files changed, 17 insertions(+), 8 deletions(-)
diff --git a/src/backend/catalog/Catalog.pm b/src/backend/catalog/Catalog.pm
new file mode 100644
index 54f8353..0b260ec
*** a/src/backend/catalog/Catalog.pm
--- b/src/backend/catalog/Catalog.pm
*************** sub Catalogs
*** 36,42 ****
'int64' => 'int8',
'Oid' => 'oid',
'NameData' => 'name',
! 'TransactionId' => 'xid');
foreach my $input_file (@_)
{
--- 36,43 ----
'int64' => 'int8',
'Oid' => 'oid',
'NameData' => 'name',
! 'TransactionId' => 'xid',
! 'XLogRecPtr' => 'pg_lsn');
foreach my $input_file (@_)
{
diff --git a/src/backend/catalog/Makefile b/src/backend/catalog/Makefile
new file mode 100644
index fd33426..30ca509
*** a/src/backend/catalog/Makefile
--- b/src/backend/catalog/Makefile
*************** POSTGRES_BKI_SRCS = $(addprefix $(top_sr
*** 45,51 ****
pg_default_acl.h pg_init_privs.h pg_seclabel.h pg_shseclabel.h \
pg_collation.h pg_partitioned_table.h pg_range.h pg_transform.h \
pg_sequence.h pg_publication.h pg_publication_rel.h pg_subscription.h \
! pg_subscription_rel.h toasting.h indexing.h \
toasting.h indexing.h \
)
--- 45,51 ----
pg_default_acl.h pg_init_privs.h pg_seclabel.h pg_shseclabel.h \
pg_collation.h pg_partitioned_table.h pg_range.h pg_transform.h \
pg_sequence.h pg_publication.h pg_publication_rel.h pg_subscription.h \
! pg_subscription_rel.h \
toasting.h indexing.h \
)
diff --git a/src/backend/utils/Gen_fmgrtab.pl b/src/backend/utils/Gen_fmgrtab.pl
new file mode 100644
index ee89d50..c0efc07
*** a/src/backend/utils/Gen_fmgrtab.pl
--- b/src/backend/utils/Gen_fmgrtab.pl
***************
*** 2,8 ****
#-------------------------------------------------------------------------
#
# Gen_fmgrtab.pl
! # Perl script that generates fmgroids.h and fmgrtab.c from pg_proc.h
#
# Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
# Portions Copyright (c) 1994, Regents of the University of California
--- 2,9 ----
#-------------------------------------------------------------------------
#
# Gen_fmgrtab.pl
! # Perl script that generates fmgroids.h, fmgrprotos.h, and fmgrtab.c
! # from pg_proc.h
#
# Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
# Portions Copyright (c) 1994, Regents of the University of California
diff --git a/src/include/catalog/pg_partitioned_table.h b/src/include/catalog/pg_partitioned_table.h
new file mode 100644
index 525e541..731147e
*** a/src/include/catalog/pg_partitioned_table.h
--- b/src/include/catalog/pg_partitioned_table.h
***************
*** 10,16 ****
* src/include/catalog/pg_partitioned_table.h
*
* NOTES
! * the genbki.sh script reads this file and generates .bki
* information from the DATA() statements.
*
*-------------------------------------------------------------------------
--- 10,16 ----
* src/include/catalog/pg_partitioned_table.h
*
* NOTES
! * the genbki.pl script reads this file and generates .bki
* information from the DATA() statements.
*
*-------------------------------------------------------------------------
diff --git a/src/include/catalog/pg_sequence.h b/src/include/catalog/pg_sequence.h
new file mode 100644
index 8ae6b71..6de54bb
*** a/src/include/catalog/pg_sequence.h
--- b/src/include/catalog/pg_sequence.h
***************
*** 1,3 ****
--- 1,13 ----
+ /* -------------------------------------------------------------------------
+ *
+ * pg_sequence.h
+ * definition of the system "sequence" relation (pg_sequence)
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * -------------------------------------------------------------------------
+ */
#ifndef PG_SEQUENCE_H
#define PG_SEQUENCE_H
diff --git a/src/include/catalog/pg_subscription_rel.h b/src/include/catalog/pg_subscription_rel.h
new file mode 100644
index 991ca9d..5748297
*** a/src/include/catalog/pg_subscription_rel.h
--- b/src/include/catalog/pg_subscription_rel.h
***************
*** 23,37 ****
*/
#define SubscriptionRelRelationId 6102
- /* Workaround for genbki not knowing about XLogRecPtr */
- #define pg_lsn XLogRecPtr
-
CATALOG(pg_subscription_rel,6102) BKI_WITHOUT_OIDS
{
Oid srsubid; /* Oid of subscription */
Oid srrelid; /* Oid of relation */
char srsubstate; /* state of the relation in subscription */
! pg_lsn srsublsn; /* remote lsn of the state change used for
* synchronization coordination */
} FormData_pg_subscription_rel;
--- 23,34 ----
*/
#define SubscriptionRelRelationId 6102
CATALOG(pg_subscription_rel,6102) BKI_WITHOUT_OIDS
{
Oid srsubid; /* Oid of subscription */
Oid srrelid; /* Oid of relation */
char srsubstate; /* state of the relation in subscription */
! XLogRecPtr srsublsn; /* remote lsn of the state change used for
* synchronization coordination */
} FormData_pg_subscription_rel;
0002_cleanup_v1.patchtext/x-patch; charset=US-ASCII; name=0002_cleanup_v1.patchDownload
src/backend/catalog/Catalog.pm | 14 +++++++-------
src/backend/catalog/genbki.pl | 38 +++++++++++++++++++++-----------------
src/backend/utils/Gen_fmgrtab.pl | 5 +++--
3 files changed, 31 insertions(+), 26 deletions(-)
diff --git a/src/backend/catalog/Catalog.pm b/src/backend/catalog/Catalog.pm
new file mode 100644
index 0b260ec..6bc14d2
*** a/src/backend/catalog/Catalog.pm
--- b/src/backend/catalog/Catalog.pm
*************** sub Catalogs
*** 163,169 ****
/BKI_WITHOUT_OIDS/ ? ' without_oids' : '';
$catalog{rowtype_oid} =
/BKI_ROWTYPE_OID\((\d+)\)/ ? " rowtype_oid $1" : '';
! $catalog{schema_macro} = /BKI_SCHEMA_MACRO/ ? 'True' : '';
$declaring_attributes = 1;
}
elsif ($declaring_attributes)
--- 163,169 ----
/BKI_WITHOUT_OIDS/ ? ' without_oids' : '';
$catalog{rowtype_oid} =
/BKI_ROWTYPE_OID\((\d+)\)/ ? " rowtype_oid $1" : '';
! $catalog{schema_macro} = /BKI_SCHEMA_MACRO/ ? 1 : 0;
$declaring_attributes = 1;
}
elsif ($declaring_attributes)
*************** sub Catalogs
*** 176,182 ****
}
else
{
! my %row;
my ($atttype, $attname, $attopt) = split /\s+/, $_;
die "parse error ($input_file)" unless $attname;
if (exists $RENAME_ATTTYPE{$atttype})
--- 176,182 ----
}
else
{
! my %column;
my ($atttype, $attname, $attopt) = split /\s+/, $_;
die "parse error ($input_file)" unless $attname;
if (exists $RENAME_ATTTYPE{$atttype})
*************** sub Catalogs
*** 189,206 ****
$atttype .= '[]'; # variable-length only
}
! $row{'type'} = $atttype;
! $row{'name'} = $attname;
if (defined $attopt)
{
if ($attopt eq 'BKI_FORCE_NULL')
{
! $row{'forcenull'} = 1;
}
elsif ($attopt eq 'BKI_FORCE_NOT_NULL')
{
! $row{'forcenotnull'} = 1;
}
else
{
--- 189,206 ----
$atttype .= '[]'; # variable-length only
}
! $column{type} = $atttype;
! $column{name} = $attname;
if (defined $attopt)
{
if ($attopt eq 'BKI_FORCE_NULL')
{
! $column{forcenull} = 1;
}
elsif ($attopt eq 'BKI_FORCE_NOT_NULL')
{
! $column{forcenotnull} = 1;
}
else
{
*************** sub Catalogs
*** 208,214 ****
"unknown column option $attopt on column $attname";
}
}
! push @{ $catalog{columns} }, \%row;
}
}
}
--- 208,214 ----
"unknown column option $attopt on column $attname";
}
}
! push @{ $catalog{columns} }, \%column;
}
}
}
diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl
new file mode 100644
index 256a9c9..4bd614f
*** a/src/backend/catalog/genbki.pl
--- b/src/backend/catalog/genbki.pl
*************** use strict;
*** 20,26 ****
use warnings;
my @input_files;
! our @include_path;
my $output_path = '';
my $major_version;
--- 20,26 ----
use warnings;
my @input_files;
! my @include_path;
my $output_path = '';
my $major_version;
*************** print $bki "# PostgreSQL $major_version\
*** 105,111 ****
my %schemapg_entries;
my @tables_needing_macros;
my %regprocoids;
! our @types;
# produce output, one catalog at a time
foreach my $catname (@{ $catalogs->{names} })
--- 105,111 ----
my %schemapg_entries;
my @tables_needing_macros;
my %regprocoids;
! my @types;
# produce output, one catalog at a time
foreach my $catname (@{ $catalogs->{names} })
*************** foreach my $catname (@{ $catalogs->{name
*** 124,130 ****
my $first = 1;
print $bki " (\n";
! foreach my $column (@{ $catalog->{columns} })
{
my $attname = $column->{name};
my $atttype = $column->{type};
--- 124,131 ----
my $first = 1;
print $bki " (\n";
! my $schema = $catalog->{columns};
! foreach my $column (@$schema)
{
my $attname = $column->{name};
my $atttype = $column->{type};
*************** foreach my $catname (@{ $catalogs->{name
*** 150,157 ****
}
print $bki "\n )\n";
! # open it, unless bootstrap case (create bootstrap does this automatically)
! if ($catalog->{bootstrap} eq '')
{
print $bki "open $catname\n";
}
--- 151,159 ----
}
print $bki "\n )\n";
! # Open it, unless bootstrap case (create bootstrap does this
! # automatically)
! if (!$catalog->{bootstrap})
{
print $bki "open $catname\n";
}
*************** foreach my $catname (@{ $catalogs->{name
*** 169,189 ****
Catalog::SplitDataLine($row->{bki_values});
# Perform required substitutions on fields
! foreach my $att (keys %bki_values)
{
# Substitute constant values we acquired above.
# (It's intentional that this can apply to parts of a field).
! $bki_values{$att} =~ s/\bPGUID\b/$BOOTSTRAP_SUPERUSERID/g;
! $bki_values{$att} =~ s/\bPGNSP\b/$PG_CATALOG_NAMESPACE/g;
# Replace regproc columns' values with OIDs.
# If we don't have a unique value to substitute,
# just do nothing (regprocin will complain).
! if ($bki_attr{$att}->{type} eq 'regproc')
{
! my $procoid = $regprocoids{ $bki_values{$att} };
! $bki_values{$att} = $procoid
if defined($procoid) && $procoid ne 'MULTIPLE';
}
}
--- 171,193 ----
Catalog::SplitDataLine($row->{bki_values});
# Perform required substitutions on fields
! foreach my $column (@$schema)
{
+ my $attname = $column->{name};
+ my $atttype = $column->{type};
# Substitute constant values we acquired above.
# (It's intentional that this can apply to parts of a field).
! $bki_values{$attname} =~ s/\bPGUID\b/$BOOTSTRAP_SUPERUSERID/g;
! $bki_values{$attname} =~ s/\bPGNSP\b/$PG_CATALOG_NAMESPACE/g;
# Replace regproc columns' values with OIDs.
# If we don't have a unique value to substitute,
# just do nothing (regprocin will complain).
! if ($atttype eq 'regproc')
{
! my $procoid = $regprocoids{ $bki_values{$attname} };
! $bki_values{$attname} = $procoid
if defined($procoid) && $procoid ne 'MULTIPLE';
}
}
*************** foreach my $catname (@{ $catalogs->{name
*** 215,221 ****
printf $bki "insert %s( %s )\n", $oid,
join(' ', @bki_values{@attnames});
! # Write comments to postgres.description and postgres.shdescription
if (defined $row->{descr})
{
printf $descr "%s\t%s\t0\t%s\n", $row->{oid}, $catname,
--- 219,226 ----
printf $bki "insert %s( %s )\n", $oid,
join(' ', @bki_values{@attnames});
! # Write comments to postgres.description and
! # postgres.shdescription
if (defined $row->{descr})
{
printf $descr "%s\t%s\t0\t%s\n", $row->{oid}, $catname,
*************** foreach my $catname (@{ $catalogs->{name
*** 240,250 ****
# Currently, all bootstrapped relations also need schemapg.h
# entries, so skip if the relation isn't to be in schemapg.h.
! next if $table->{schema_macro} ne 'True';
$schemapg_entries{$table_name} = [];
push @tables_needing_macros, $table_name;
- my $is_bootstrap = $table->{bootstrap};
# Generate entries for user attributes.
my $attnum = 0;
--- 245,254 ----
# Currently, all bootstrapped relations also need schemapg.h
# entries, so skip if the relation isn't to be in schemapg.h.
! next if !$table->{schema_macro};
$schemapg_entries{$table_name} = [];
push @tables_needing_macros, $table_name;
# Generate entries for user attributes.
my $attnum = 0;
*************** foreach my $catname (@{ $catalogs->{name
*** 259,265 ****
$priornotnull &= ($row->{attnotnull} eq 't');
# If it's bootstrapped, put an entry in postgres.bki.
! if ($is_bootstrap eq ' bootstrap')
{
bki_insert($row, @attnames);
}
--- 263,269 ----
$priornotnull &= ($row->{attnotnull} eq 't');
# If it's bootstrapped, put an entry in postgres.bki.
! if ($table->{bootstrap})
{
bki_insert($row, @attnames);
}
*************** foreach my $catname (@{ $catalogs->{name
*** 276,282 ****
# Generate entries for system attributes.
# We only need postgres.bki entries, not schemapg.h entries.
! if ($is_bootstrap eq ' bootstrap')
{
$attnum = 0;
my @SYS_ATTRS = (
--- 280,286 ----
# Generate entries for system attributes.
# We only need postgres.bki entries, not schemapg.h entries.
! if ($table->{bootstrap})
{
$attnum = 0;
my @SYS_ATTRS = (
*************** foreach my $catname (@{ $catalogs->{name
*** 296,302 ****
# some catalogs don't have oids
next
! if $table->{without_oids} eq ' without_oids'
&& $row->{attname} eq 'oid';
bki_insert($row, @attnames);
--- 300,306 ----
# some catalogs don't have oids
next
! if $table->{without_oids}
&& $row->{attname} eq 'oid';
bki_insert($row, @attnames);
diff --git a/src/backend/utils/Gen_fmgrtab.pl b/src/backend/utils/Gen_fmgrtab.pl
new file mode 100644
index c0efc07..a51a755
*** a/src/backend/utils/Gen_fmgrtab.pl
--- b/src/backend/utils/Gen_fmgrtab.pl
*************** die "No include path; you must specify -
*** 57,62 ****
--- 57,64 ----
my $FirstBootstrapObjectId =
Catalog::FindDefinedSymbol('access/transam.h', \@include_path, 'FirstBootstrapObjectId');
+ my $INTERNALlanguageId =
+ Catalog::FindDefinedSymbol('catalog/pg_language.h', \@include_path, 'INTERNALlanguageId');
# Read all the data from the include/catalog files.
my $catalogs = Catalog::Catalogs($infile);
*************** foreach my $row (@$data)
*** 78,85 ****
@bki_values{@attnames} = Catalog::SplitDataLine($row->{bki_values});
# Select out just the rows for internal-language procedures.
! # Note assumption here that INTERNALlanguageId is 12.
! next if $bki_values{prolang} ne '12';
push @fmgr,
{ oid => $row->{oid},
--- 80,86 ----
@bki_values{@attnames} = Catalog::SplitDataLine($row->{bki_values});
# Select out just the rows for internal-language procedures.
! next if $bki_values{prolang} ne $INTERNALlanguageId;
push @fmgr,
{ oid => $row->{oid},
0003_pgattr_schema_isolation_v1.patchtext/x-patch; charset=US-ASCII; name=0003_pgattr_schema_isolation_v1.patchDownload
src/backend/catalog/Catalog.pm | 49 +++++++++++-
src/backend/catalog/genbki.pl | 158 ++++++++++++++++++-------------------
src/include/catalog/genbki.h | 3 +
src/include/catalog/pg_attribute.h | 22 +++---
4 files changed, 140 insertions(+), 92 deletions(-)
diff --git a/src/backend/catalog/Catalog.pm b/src/backend/catalog/Catalog.pm
new file mode 100644
index 6bc14d2..246aa36
*** a/src/backend/catalog/Catalog.pm
--- b/src/backend/catalog/Catalog.pm
*************** sub Catalogs
*** 44,49 ****
--- 44,50 ----
my %catalog;
$catalog{columns} = [];
$catalog{data} = [];
+ my $is_varlen = 0;
open(my $ifh, '<', $input_file) || die "$input_file: $!";
*************** sub Catalogs
*** 169,175 ****
elsif ($declaring_attributes)
{
next if (/^{|^$/);
! next if (/^#/);
if (/^}/)
{
undef $declaring_attributes;
--- 170,183 ----
elsif ($declaring_attributes)
{
next if (/^{|^$/);
! if (/^#/)
! {
! if (/^#ifdef\s+CATALOG_VARLEN/)
! {
! $is_varlen = 1;
! }
! next;
! }
if (/^}/)
{
undef $declaring_attributes;
*************** sub Catalogs
*** 177,182 ****
--- 185,194 ----
else
{
my %column;
+ if ($is_varlen)
+ {
+ $column{is_varlen} = 1;
+ }
my ($atttype, $attname, $attopt) = split /\s+/, $_;
die "parse error ($input_file)" unless $attname;
if (exists $RENAME_ATTTYPE{$atttype})
*************** sub Catalogs
*** 186,192 ****
if ($attname =~ /(.*)\[.*\]/) # array attribute
{
$attname = $1;
! $atttype .= '[]'; # variable-length only
}
$column{type} = $atttype;
--- 198,204 ----
if ($attname =~ /(.*)\[.*\]/) # array attribute
{
$attname = $1;
! $atttype .= '[]';
}
$column{type} = $atttype;
*************** sub Catalogs
*** 202,207 ****
--- 214,223 ----
{
$column{forcenotnull} = 1;
}
+ elsif ($attopt =~ /BKI_DEFAULT\((\S+)\)/)
+ {
+ $column{default} = $1;
+ }
else
{
die
*************** sub SplitDataLine
*** 240,245 ****
--- 256,290 ----
return @result;
}
+ # Fill in default values of a record using the given schema. It's the
+ # caller's responsibility to specify other values beforehand.
+ sub AddDefaultValues
+ {
+ my ($row, $schema, $catname) = @_;
+
+ die "Schema undefined for $catname\n"
+ if !defined $schema;
+
+ foreach my $column (@$schema)
+ {
+ my $attname = $column->{name};
+ my $atttype = $column->{type};
+
+ if (defined $row->{$attname})
+ {
+ ;
+ }
+ elsif (defined $column->{default})
+ {
+ $row->{$attname} = $column->{default};
+ }
+ else
+ {
+ die "Unspecified value in $catname.$attname\n";
+ }
+ }
+ }
+
# Rename temporary files to final names.
# Call this function with the final file name and the .tmp extension
# Note: recommended extension is ".tmp$$", so that parallel make steps
diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl
new file mode 100644
index 4bd614f..1876399
*** a/src/backend/catalog/genbki.pl
--- b/src/backend/catalog/genbki.pl
*************** foreach my $catname (@{ $catalogs->{name
*** 119,125 ****
. $catalog->{without_oids}
. $catalog->{rowtype_oid} . "\n";
- my %bki_attr;
my @attnames;
my $first = 1;
--- 119,124 ----
*************** foreach my $catname (@{ $catalogs->{name
*** 129,135 ****
{
my $attname = $column->{name};
my $atttype = $column->{type};
- $bki_attr{$attname} = $column;
push @attnames, $attname;
if (!$first)
--- 128,133 ----
*************** foreach my $catname (@{ $catalogs->{name
*** 257,281 ****
foreach my $attr (@user_attrs)
{
$attnum++;
! my $row = emit_pgattr_row($table_name, $attr, $priornotnull);
! $row->{attnum} = $attnum;
! $row->{attstattarget} = '-1';
! $priornotnull &= ($row->{attnotnull} eq 't');
# If it's bootstrapped, put an entry in postgres.bki.
if ($table->{bootstrap})
{
! bki_insert($row, @attnames);
}
# Store schemapg entries for later.
! $row =
! emit_schemapg_row($row,
! grep { $bki_attr{$_}{type} eq 'bool' } @attnames);
! push @{ $schemapg_entries{$table_name} }, '{ '
! . join(
! ', ', grep { defined $_ }
! map $row->{$_}, @attnames) . ' }';
}
# Generate entries for system attributes.
--- 255,279 ----
foreach my $attr (@user_attrs)
{
$attnum++;
! my %row;
! $row{attnum} = $attnum;
! $row{attrelid} = $table->{relation_oid};
!
! emit_pgattr_row(\%row, $attr, $priornotnull, $schema);
! $priornotnull &= ($row{attnotnull} eq 't');
# If it's bootstrapped, put an entry in postgres.bki.
if ($table->{bootstrap})
{
! bki_insert(\%row, @attnames);
}
# Store schemapg entries for later.
! emit_schemapg_row(\%row, $schema);
! push @{ $schemapg_entries{$table_name} },
! '{ '
! . join(', ', grep { defined $_ } @row{@attnames})
! . ' }';
}
# Generate entries for system attributes.
*************** foreach my $catname (@{ $catalogs->{name
*** 294,309 ****
foreach my $attr (@SYS_ATTRS)
{
$attnum--;
! my $row = emit_pgattr_row($table_name, $attr, 1);
! $row->{attnum} = $attnum;
! $row->{attstattarget} = '0';
# some catalogs don't have oids
next
if $table->{without_oids}
! && $row->{attname} eq 'oid';
! bki_insert($row, @attnames);
}
}
}
--- 292,310 ----
foreach my $attr (@SYS_ATTRS)
{
$attnum--;
! my %row;
! $row{attnum} = $attnum;
! $row{attrelid} = $table->{relation_oid};
! $row{attstattarget} = '0';
!
! emit_pgattr_row(\%row, $attr, 1, $schema);
# some catalogs don't have oids
next
if $table->{without_oids}
! && $row{attname} eq 'oid';
! bki_insert(\%row, @attnames);
}
}
}
*************** exit 0;
*** 380,398 ****
#################### Subroutines ########################
! # Given a system catalog name and a reference to a key-value pair corresponding
! # to the name and type of a column, generate a reference to a hash that
! # represents a pg_attribute entry. We must also be told whether preceding
# columns were all not-null.
sub emit_pgattr_row
{
! my ($table_name, $attr, $priornotnull) = @_;
my $attname = $attr->{name};
my $atttype = $attr->{type};
- my %row;
! $row{attrelid} = $catalogs->{$table_name}->{relation_oid};
! $row{attname} = $attname;
# Adjust type name for arrays: foo[] becomes _foo
# so we can look it up in pg_type
--- 381,397 ----
#################### Subroutines ########################
! # Given the schema of pg_attribute, generate an entry for it using information
! # about the attribute it describes. Any value that is not handled here
! # must be supplied by the caller. We must also be told whether preceding
# columns were all not-null.
sub emit_pgattr_row
{
! my ($row, $attr, $priornotnull, $pgattr_schema) = @_;
my $attname = $attr->{name};
my $atttype = $attr->{type};
! $row->{attname} = $attname;
# Adjust type name for arrays: foo[] becomes _foo
# so we can look it up in pg_type
*************** sub emit_pgattr_row
*** 406,428 ****
{
if (defined $type->{typname} && $type->{typname} eq $atttype)
{
! $row{atttypid} = $type->{oid};
! $row{attlen} = $type->{typlen};
! $row{attbyval} = $type->{typbyval};
! $row{attstorage} = $type->{typstorage};
! $row{attalign} = $type->{typalign};
# set attndims if it's an array type
! $row{attndims} = $type->{typcategory} eq 'A' ? '1' : '0';
! $row{attcollation} = $type->{typcollation};
if (defined $attr->{forcenotnull})
{
! $row{attnotnull} = 't';
}
elsif (defined $attr->{forcenull})
{
! $row{attnotnull} = 'f';
}
elsif ($priornotnull)
{
--- 405,427 ----
{
if (defined $type->{typname} && $type->{typname} eq $atttype)
{
! $row->{atttypid} = $type->{oid};
! $row->{attlen} = $type->{typlen};
! $row->{attbyval} = $type->{typbyval};
! $row->{attstorage} = $type->{typstorage};
! $row->{attalign} = $type->{typalign};
# set attndims if it's an array type
! $row->{attndims} = $type->{typcategory} eq 'A' ? '1' : '0';
! $row->{attcollation} = $type->{typcollation};
if (defined $attr->{forcenotnull})
{
! $row->{attnotnull} = 't';
}
elsif (defined $attr->{forcenull})
{
! $row->{attnotnull} = 'f';
}
elsif ($priornotnull)
{
*************** sub emit_pgattr_row
*** 431,437 ****
# fixed-width and prior columns are all NOT NULL ---
# compare DefineAttr in bootstrap.c. oidvector and
# int2vector are also treated as not-nullable.
! $row{attnotnull} =
$type->{typname} eq 'oidvector' ? 't'
: $type->{typname} eq 'int2vector' ? 't'
: $type->{typlen} eq 'NAMEDATALEN' ? 't'
--- 430,436 ----
# fixed-width and prior columns are all NOT NULL ---
# compare DefineAttr in bootstrap.c. oidvector and
# int2vector are also treated as not-nullable.
! $row->{attnotnull} =
$type->{typname} eq 'oidvector' ? 't'
: $type->{typname} eq 'int2vector' ? 't'
: $type->{typlen} eq 'NAMEDATALEN' ? 't'
*************** sub emit_pgattr_row
*** 440,464 ****
}
else
{
! $row{attnotnull} = 'f';
}
last;
}
}
!
! # Add in default values for pg_attribute
! my %PGATTR_DEFAULTS = (
! attcacheoff => '-1',
! atttypmod => '-1',
! atthasdef => 'f',
! attidentity => '',
! attisdropped => 'f',
! attislocal => 't',
! attinhcount => '0',
! attacl => '_null_',
! attoptions => '_null_',
! attfdwoptions => '_null_');
! return { %PGATTR_DEFAULTS, %row };
}
# Write a pg_attribute entry to postgres.bki
--- 439,450 ----
}
else
{
! $row->{attnotnull} = 'f';
}
last;
}
}
! Catalog::AddDefaultValues($row, $pgattr_schema, 'pg_attribute');
}
# Write a pg_attribute entry to postgres.bki
*************** sub bki_insert
*** 467,474 ****
my $row = shift;
my @attnames = @_;
my $oid = $row->{oid} ? "OID = $row->{oid} " : '';
! my $bki_values = join ' ', map { $_ eq '' ? '""' : $_ } map $row->{$_},
! @attnames;
printf $bki "insert %s( %s )\n", $oid, $bki_values;
}
--- 453,459 ----
my $row = shift;
my @attnames = @_;
my $oid = $row->{oid} ? "OID = $row->{oid} " : '';
! my $bki_values = join ' ', map $row->{$_}, @attnames;
printf $bki "insert %s( %s )\n", $oid, $bki_values;
}
*************** sub bki_insert
*** 476,509 ****
# quite identical, to the corresponding values in postgres.bki.
sub emit_schemapg_row
{
! my $row = shift;
! my @bool_attrs = @_;
! # Replace empty string by zero char constant
! $row->{attidentity} ||= '\0';
! # Supply appropriate quoting for these fields.
! $row->{attname} = q|{"| . $row->{attname} . q|"}|;
! $row->{attstorage} = q|'| . $row->{attstorage} . q|'|;
! $row->{attalign} = q|'| . $row->{attalign} . q|'|;
! $row->{attidentity} = q|'| . $row->{attidentity} . q|'|;
! # We don't emit initializers for the variable length fields at all.
! # Only the fixed-size portions of the descriptors are ever used.
! delete $row->{attacl};
! delete $row->{attoptions};
! delete $row->{attfdwoptions};
! # Expand booleans from 'f'/'t' to 'false'/'true'.
! # Some values might be other macros (eg FLOAT4PASSBYVAL), don't change.
! foreach my $attr (@bool_attrs)
! {
! $row->{$attr} =
! $row->{$attr} eq 't' ? 'true'
! : $row->{$attr} eq 'f' ? 'false'
! : $row->{$attr};
}
- return $row;
}
sub usage
--- 461,509 ----
# quite identical, to the corresponding values in postgres.bki.
sub emit_schemapg_row
{
! my $row = shift;
! my $pgattr_schema = shift;
! foreach my $column (@$pgattr_schema)
! {
! my $pgattr_name = $column->{name};
! my $pgattr_type = $column->{type};
! # Supply appropriate quoting for these fields.
! if ($pgattr_type eq 'name')
! {
! $row->{$pgattr_name} = q|{"| . $row->{$pgattr_name} . q|"}|;
! }
! elsif ($pgattr_type eq 'char')
! {
! # Replace empty string by zero char constant
! if ($row->{$pgattr_name} eq q|""|)
! {
! $row->{$pgattr_name} = '\0';
! }
! $row->{$pgattr_name} = q|'| . $row->{$pgattr_name} . q|'|;
! }
!
! # Expand booleans from 'f'/'t' to 'false'/'true'.
! # Some values might be other macros (eg FLOAT4PASSBYVAL),
! # don't change.
! elsif ($pgattr_type eq 'bool')
! {
! $row->{$pgattr_name} =
! $row->{$pgattr_name} eq 't' ? 'true'
! : $row->{$pgattr_name} eq 'f' ? 'false'
! : $row->{$pgattr_name};
! }
!
! # We don't emit initializers for the variable length fields at all.
! # Only the fixed-size portions of the descriptors are ever used.
! if ($column->{is_varlen})
! {
! delete $row->{$pgattr_name};
! }
}
}
sub usage
diff --git a/src/include/catalog/genbki.h b/src/include/catalog/genbki.h
new file mode 100644
index a2cb313..71fc579
*** a/src/include/catalog/genbki.h
--- b/src/include/catalog/genbki.h
***************
*** 31,36 ****
--- 31,39 ----
#define BKI_FORCE_NULL
#define BKI_FORCE_NOT_NULL
+ /* Specifies a default value for a catalog field */
+ #define BKI_DEFAULT(value)
+
/*
* This is never defined; it's here only for documentation.
*
diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h
new file mode 100644
index bcf28e8..5436a90
*** a/src/include/catalog/pg_attribute.h
--- b/src/include/catalog/pg_attribute.h
*************** CATALOG(pg_attribute,1249) BKI_BOOTSTRAP
*** 54,60 ****
* that no value has been explicitly set for this column, so ANALYZE
* should use the default setting.
*/
! int32 attstattarget;
/*
* attlen is a copy of the typlen field from pg_type for this attribute.
--- 54,60 ----
* that no value has been explicitly set for this column, so ANALYZE
* should use the default setting.
*/
! int32 attstattarget BKI_DEFAULT(-1);
/*
* attlen is a copy of the typlen field from pg_type for this attribute.
*************** CATALOG(pg_attribute,1249) BKI_BOOTSTRAP
*** 90,96 ****
* descriptor, we may then update attcacheoff in the copies. This speeds
* up the attribute walking process.
*/
! int32 attcacheoff;
/*
* atttypmod records type-specific data supplied at table creation time
--- 90,96 ----
* descriptor, we may then update attcacheoff in the copies. This speeds
* up the attribute walking process.
*/
! int32 attcacheoff BKI_DEFAULT(-1);
/*
* atttypmod records type-specific data supplied at table creation time
*************** CATALOG(pg_attribute,1249) BKI_BOOTSTRAP
*** 98,104 ****
* type-specific input and output functions as the third argument. The
* value will generally be -1 for types that do not need typmod.
*/
! int32 atttypmod;
/*
* attbyval is a copy of the typbyval field from pg_type for this
--- 98,104 ----
* type-specific input and output functions as the third argument. The
* value will generally be -1 for types that do not need typmod.
*/
! int32 atttypmod BKI_DEFAULT(-1);
/*
* attbyval is a copy of the typbyval field from pg_type for this
*************** CATALOG(pg_attribute,1249) BKI_BOOTSTRAP
*** 131,143 ****
bool attnotnull;
/* Has DEFAULT value or not */
! bool atthasdef;
/* One of the ATTRIBUTE_IDENTITY_* constants below, or '\0' */
! char attidentity;
/* Is dropped (ie, logically invisible) or not */
! bool attisdropped;
/*
* This flag specifies whether this column has ever had a local
--- 131,143 ----
bool attnotnull;
/* Has DEFAULT value or not */
! bool atthasdef BKI_DEFAULT(f);
/* One of the ATTRIBUTE_IDENTITY_* constants below, or '\0' */
! char attidentity BKI_DEFAULT("");
/* Is dropped (ie, logically invisible) or not */
! bool attisdropped BKI_DEFAULT(f);
/*
* This flag specifies whether this column has ever had a local
*************** CATALOG(pg_attribute,1249) BKI_BOOTSTRAP
*** 148,157 ****
* not dropped by a parent's DROP COLUMN even if this causes the column's
* attinhcount to become zero.
*/
! bool attislocal;
/* Number of times inherited from direct parent relation(s) */
! int32 attinhcount;
/* attribute's collation */
Oid attcollation;
--- 148,157 ----
* not dropped by a parent's DROP COLUMN even if this causes the column's
* attinhcount to become zero.
*/
! bool attislocal BKI_DEFAULT(t);
/* Number of times inherited from direct parent relation(s) */
! int32 attinhcount BKI_DEFAULT(0);
/* attribute's collation */
Oid attcollation;
*************** CATALOG(pg_attribute,1249) BKI_BOOTSTRAP
*** 160,172 ****
/* NOTE: The following fields are not present in tuple descriptors. */
/* Column-level access permissions */
! aclitem attacl[1];
/* Column-level options */
! text attoptions[1];
/* Column-level FDW options */
! text attfdwoptions[1];
#endif
} FormData_pg_attribute;
--- 160,172 ----
/* NOTE: The following fields are not present in tuple descriptors. */
/* Column-level access permissions */
! aclitem attacl[1] BKI_DEFAULT(_null_);
/* Column-level options */
! text attoptions[1] BKI_DEFAULT(_null_);
/* Column-level FDW options */
! text attfdwoptions[1] BKI_DEFAULT(_null_);
#endif
} FormData_pg_attribute;
0004_conversion_scripts_and_headers_v1.patchtext/x-patch; charset=US-ASCII; name=0004_conversion_scripts_and_headers_v1.patchDownload
src/backend/catalog/Catalog.pm | 156 +++++--------
src/include/catalog/convert_header2dat.pl | 370 ++++++++++++++++++++++++++++++
src/include/catalog/pg_aggregate.h | 38 +--
src/include/catalog/pg_authid.h | 22 +-
src/include/catalog/pg_class.h | 54 ++---
src/include/catalog/pg_opclass.h | 16 +-
src/include/catalog/pg_operator.h | 28 +--
src/include/catalog/pg_opfamily.h | 8 +-
src/include/catalog/pg_proc.h | 54 ++---
src/include/catalog/pg_type.h | 40 ++--
src/include/catalog/rewrite_dat.pl | 346 ++++++++++++++++++++++++++++
11 files changed, 899 insertions(+), 233 deletions(-)
diff --git a/src/backend/catalog/Catalog.pm b/src/backend/catalog/Catalog.pm
new file mode 100644
index 246aa36..ebed363
*** a/src/backend/catalog/Catalog.pm
--- b/src/backend/catalog/Catalog.pm
***************
*** 1,7 ****
#----------------------------------------------------------------------
#
# Catalog.pm
! # Perl module that extracts info from catalog headers into Perl
# data structures
#
# Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
--- 1,7 ----
#----------------------------------------------------------------------
#
# Catalog.pm
! # Perl module that extracts info from catalog files into Perl
# data structures
#
# Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
*************** package Catalog;
*** 16,32 ****
use strict;
use warnings;
- require Exporter;
- our @ISA = qw(Exporter);
- our @EXPORT = ();
- our @EXPORT_OK = qw(Catalogs SplitDataLine RenameTempFile FindDefinedSymbol);
-
# Call this function with an array of names of header files to parse.
! # Returns a nested data structure describing the data in the headers.
! sub Catalogs
{
! my (%catalogs, $catname, $declaring_attributes, $most_recent);
$catalogs{names} = [];
# There are a few types which are given one name in the C source, but a
# different name at the SQL level. These are enumerated here.
--- 16,29 ----
use strict;
use warnings;
# Call this function with an array of names of header files to parse.
! # Returns a data structure describing the schemas of the system catalogs.
! sub ParseHeader
{
! my (%catalogs, $catname, $declaring_attributes);
$catalogs{names} = [];
+ $catalogs{toasting} = [];
+ $catalogs{indexing} = [];
# There are a few types which are given one name in the C source, but a
# different name at the SQL level. These are enumerated here.
*************** sub Catalogs
*** 43,55 ****
{
my %catalog;
$catalog{columns} = [];
- $catalog{data} = [];
my $is_varlen = 0;
open(my $ifh, '<', $input_file) || die "$input_file: $!";
my ($filename) = ($input_file =~ m/(\w+)\.h$/);
- my $natts_pat = "Natts_$filename";
# Scan the input file.
while (<$ifh>)
--- 40,50 ----
*************** sub Catalogs
*** 78,145 ****
s/\s+/ /g;
# Push the data into the appropriate data structure.
! if (/$natts_pat\s+(\d+)/)
! {
! $catalog{natts} = $1;
! }
! elsif (
! /^DATA\(insert(\s+OID\s+=\s+(\d+))?\s+\(\s*(.*)\s*\)\s*\)$/)
! {
! check_natts($filename, $catalog{natts}, $3, $input_file,
! $input_line_number);
!
! push @{ $catalog{data} }, { oid => $2, bki_values => $3 };
! }
! elsif (/^DESCR\(\"(.*)\"\)$/)
! {
! $most_recent = $catalog{data}->[-1];
!
! # this tests if most recent line is not a DATA() statement
! if (ref $most_recent ne 'HASH')
! {
! die "DESCR() does not apply to any catalog ($input_file)";
! }
! if (!defined $most_recent->{oid})
! {
! die "DESCR() does not apply to any oid ($input_file)";
! }
! elsif ($1 ne '')
! {
! $most_recent->{descr} = $1;
! }
! }
! elsif (/^SHDESCR\(\"(.*)\"\)$/)
{
! $most_recent = $catalog{data}->[-1];
- # this tests if most recent line is not a DATA() statement
- if (ref $most_recent ne 'HASH')
- {
- die
- "SHDESCR() does not apply to any catalog ($input_file)";
- }
- if (!defined $most_recent->{oid})
- {
- die "SHDESCR() does not apply to any oid ($input_file)";
- }
- elsif ($1 ne '')
- {
- $most_recent->{shdescr} = $1;
- }
- }
- elsif (/^DECLARE_TOAST\(\s*(\w+),\s*(\d+),\s*(\d+)\)/)
- {
- $catname = 'toasting';
my ($toast_name, $toast_oid, $index_oid) = ($1, $2, $3);
! push @{ $catalog{data} },
"declare toast $toast_oid $index_oid on $toast_name\n";
}
elsif (/^DECLARE_(UNIQUE_)?INDEX\(\s*(\w+),\s*(\d+),\s*(.+)\)/)
{
- $catname = 'indexing';
my ($is_unique, $index_name, $index_oid, $using) =
($1, $2, $3, $4);
! push @{ $catalog{data} },
sprintf(
"declare %sindex %s %s %s\n",
$is_unique ? 'unique ' : '',
--- 73,93 ----
s/\s+/ /g;
# Push the data into the appropriate data structure.
! if (/^DECLARE_TOAST\(\s*(\w+),\s*(\d+),\s*(\d+)\)/)
{
! # Clear last catname so we don't overwrite that
! # catalog's schema with a blank one.
! undef $catname;
my ($toast_name, $toast_oid, $index_oid) = ($1, $2, $3);
! push @{ $catalogs{toasting} },
"declare toast $toast_oid $index_oid on $toast_name\n";
}
elsif (/^DECLARE_(UNIQUE_)?INDEX\(\s*(\w+),\s*(\d+),\s*(.+)\)/)
{
my ($is_unique, $index_name, $index_oid, $using) =
($1, $2, $3, $4);
! push @{ $catalogs{indexing} },
sprintf(
"declare %sindex %s %s %s\n",
$is_unique ? 'unique ' : '',
*************** sub Catalogs
*** 147,153 ****
}
elsif (/^BUILD_INDICES/)
{
! push @{ $catalog{data} }, "build indices\n";
}
elsif (/^CATALOG\(([^,]*),(\d+)\)/)
{
--- 95,101 ----
}
elsif (/^BUILD_INDICES/)
{
! push @{ $catalogs{indexing} }, "build indices\n";
}
elsif (/^CATALOG\(([^,]*),(\d+)\)/)
{
*************** sub Catalogs
*** 228,259 ****
}
}
}
! $catalogs{$catname} = \%catalog;
close $ifh;
}
return \%catalogs;
}
! # Split a DATA line into fields.
! # Call this on the bki_values element of a DATA item returned by Catalogs();
! # it returns a list of field values. We don't strip quoting from the fields.
! # Note: it should be safe to assign the result to a list of length equal to
! # the nominal number of catalog fields, because check_natts already checked
! # the number of fields.
! sub SplitDataLine
{
! my $bki_values = shift;
! # This handling of quoted strings might look too simplistic, but it
! # matches what bootscanner.l does: that has no provision for quote marks
! # inside quoted strings, either. If we don't have a quoted string, just
! # snarf everything till next whitespace. That will accept some things
! # that bootscanner.l will see as erroneous tokens; but it seems wiser
! # to do that and let bootscanner.l complain than to silently drop
! # non-whitespace characters.
! my @result = $bki_values =~ /"[^"]*"|\S+/g;
! return @result;
}
# Fill in default values of a record using the given schema. It's the
--- 176,217 ----
}
}
}
! # Prevent toasting and indexing blank %catalog vars from
! # over-writing the last real catalog.
! if (defined $catname)
! {
! $catalogs{$catname} = \%catalog;
! }
close $ifh;
}
return \%catalogs;
}
! # Takes an array of names of data files containing Perl data structure
! # literals. This function simply calls eval on the whole string.
! # XXX This is much slower than parsing DATA() statements was.
! # Is there a better way?
! sub ParseData
{
! my %catalog_data;
! foreach my $input_file (@_)
! {
! my ($filename) = ($input_file =~ m/(\w+)\.dat$/);
! my $raw_data = do
! {
! local $/ = undef;
! open my $ifh, "<", $input_file || die "$input_file: $!";
! <$ifh>;
! };
! my $parsed_data;
! eval '$parsed_data = ' . $raw_data;
! # XXX: Is this enough error reporting?
! print "Error : $@\n" if $@;
! $catalog_data{$filename} = $parsed_data;
! }
! return \%catalog_data;
}
# Fill in default values of a record using the given schema. It's the
*************** sub AddDefaultValues
*** 280,286 ****
}
else
{
! die "Unspecified value in $catname.$attname\n";
}
}
}
--- 238,255 ----
}
else
{
! printf "Unspecified value in $catname.$attname:\n";
!
! # Give user a clue where the problem was.
! # We could call this function after reading each hash
! # retail, and give a line number, but that would be more
! # complicated. Just print out the given values.
! my $msg;
! foreach (keys %{$row})
! {
! $msg .= "$_ => $row->{$_}, ";
! }
! die "$msg\n";
}
}
}
*************** sub RenameTempFile
*** 298,304 ****
rename($temp_name, $final_name) || die "rename: $temp_name: $!";
}
-
# Find a symbol defined in a particular header file and extract the value.
#
# The include path has to be passed as a reference to an array.
--- 267,272 ----
*************** sub FindDefinedSymbol
*** 330,351 ****
die "$catalog_header: not found in any include directory\n";
}
-
- # verify the number of fields in the passed-in DATA line
- sub check_natts
- {
- my ($catname, $natts, $bki_val, $file, $line) = @_;
-
- die
- "Could not find definition for Natts_${catname} before start of DATA() in $file\n"
- unless defined $natts;
-
- my $nfields = scalar(SplitDataLine($bki_val));
-
- die sprintf
- "Wrong number of attributes in DATA() entry at %s:%d (expected %d but got %d)\n",
- $file, $line, $natts, $nfields
- unless $natts == $nfields;
- }
-
1;
--- 298,301 ----
diff --git a/src/include/catalog/convert_header2dat.pl b/src/include/catalog/convert_header2dat.pl
new file mode 100644
index ...d61cdc8
*** a/src/include/catalog/convert_header2dat.pl
--- b/src/include/catalog/convert_header2dat.pl
***************
*** 0 ****
--- 1,370 ----
+ #!/usr/bin/perl -w
+ #----------------------------------------------------------------------
+ #
+ # convert_header2dat.pl
+ # Perl script that reads BKI data from the catalog header files
+ # and writes them out as native perl data structures. Commments and
+ # white space are preserved. Some functions are loosely copied from
+ # src/backend/catalog/Catalog.pm, whose equivalents will be removed.
+ #
+ # Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ # Portions Copyright (c) 1994, Regents of the University of California
+ #
+ # /src/include/catalog/convert_header2dat.pl
+ #
+ #----------------------------------------------------------------------
+
+ use strict;
+ use warnings;
+
+ use Data::Dumper;
+ # No $VARs - we add our own later.
+ $Data::Dumper::Terse = 1;
+
+ my @input_files;
+ my $output_path = '';
+ my $major_version;
+
+ # Process command line switches.
+ while (@ARGV)
+ {
+ my $arg = shift @ARGV;
+ if ($arg !~ /^-/)
+ {
+ push @input_files, $arg;
+ }
+ elsif ($arg =~ /^-o/)
+ {
+ $output_path = length($arg) > 2 ? substr($arg, 2) : shift @ARGV;
+ }
+ else
+ {
+ usage();
+ }
+ }
+
+ # Sanity check arguments.
+ die "No input files.\n" if !@input_files;
+ foreach my $input_file (@input_files)
+ {
+ if ($input_file !~ /\.h$/)
+ {
+ die "Input files need to be header files.\n";
+ }
+ }
+
+ # Make sure output_path ends in a slash.
+ if ($output_path ne '' && substr($output_path, -1) ne '/')
+ {
+ $output_path .= '/';
+ }
+
+ # Read all the input header files into internal data structures
+ # XXX This script is not robust against non-catalog headers. It's best
+ # to pass it the same list found in backend/catalog/Makefile.
+ my $catalogs = catalogs(@input_files);
+
+ # produce output, one catalog at a time
+ foreach my $catname (@{ $catalogs->{names} })
+ {
+ my $catalog = $catalogs->{$catname};
+ my $schema = $catalog->{columns};
+
+ # First, see if the header has any data. This is necessary
+ # not only because of catalogs with no data, but also because some
+ # values coming down the pike are comments or newlines.
+ my $found_one = 0;
+ foreach my $data (@{ $catalog->{data} })
+ {
+ if (ref $data eq 'HASH')
+ {
+ $found_one = 1;
+ }
+ }
+ next if !$found_one;
+
+ my @attnames;
+ foreach my $column (@$schema)
+ {
+ my $attname = $column->{name};
+ my $atttype = $column->{type};
+ push @attnames, $attname;
+ }
+
+ my $datfile = "$output_path$catname.dat";
+ open my $dat, '>', $datfile
+ or die "can't open $datfile: $!";
+
+ # Write out data file.
+ print $dat "# $catname.dat\n";
+
+ # Note: Put extra newlines after brackets because otherwise
+ # some catalogs have newlines and some don't, because of whitespace
+ # around DATA() comments.
+ print $dat "[\n\n";
+
+ foreach my $data (@{ $catalog->{data} })
+ {
+
+ # Either a newline or comment - just write it out.
+ if (! ref $data)
+ {
+ print $dat "$data\n";
+ }
+ # Hash ref representing a data entry.
+ elsif (ref $data eq 'HASH')
+ {
+ # Split line into tokens without interpreting their meaning.
+ my %bki_values;
+ @bki_values{@attnames} = split_data_line($data->{bki_values});
+
+ # Flatten data hierarchy.
+ delete $data->{bki_values};
+ my %flat_data = (%$data, %bki_values);
+
+ # Strip double quotes for readability. Most will be put
+ # back in when writing postgres.bki
+ foreach (values %flat_data)
+ {
+ s/"//g;
+ }
+
+ print $dat Dumper(\%flat_data);
+ print $dat ",\n";
+ }
+ }
+ print $dat "\n]\n";
+ }
+
+
+ # This function is a heavily modified version of its former namesake
+ # in Catalog.pm. There's probably some dead code here. It's not worth removing.
+ sub catalogs
+ {
+ my (%catalogs, $catname, $declaring_attributes, $most_recent);
+ $catalogs{names} = [];
+
+ # There are a few types which are given one name in the C source, but a
+ # different name at the SQL level. These are enumerated here.
+ my %RENAME_ATTTYPE = (
+ 'int16' => 'int2',
+ 'int32' => 'int4',
+ 'int64' => 'int8',
+ 'Oid' => 'oid',
+ 'NameData' => 'name',
+ 'TransactionId' => 'xid',
+ 'XLogRecPtr' => 'pg_lsn');
+
+ foreach my $input_file (@_)
+ {
+ my %catalog;
+ $catalog{columns} = [];
+ $catalog{data} = [];
+ my $is_varlen = 0;
+ my $saving_comments = 0;
+
+ open(my $ifh, '<', $input_file) || die "$input_file: $!";
+
+ # Scan the input file.
+ while (<$ifh>)
+ {
+ # Determine that we're in the DATA section and should
+ # Start saving DATA comments.
+ if (/(\/|\s)\*\s+initial contents of pg_/)
+ {
+ $saving_comments = 1;
+ }
+
+ if ($saving_comments)
+ {
+ if ( m;^(/|\s+)\*\s+(.+); )
+ {
+ my $comment = $2;
+
+ # Ugly way to strip */ off the end
+ if ($comment =~ m;\*/$;)
+ {
+ $comment =~ s/.{2}$//;
+ }
+
+ # Turn C-style comment into Perl-style.
+
+ # Filter out comments we know we don't want.
+ if ($comment !~ /^-+$/
+ and $comment !~ /initial contents of pg/
+ and $comment !~ /PG_\w+_H/)
+ {
+ # Trim whitespace.
+ $comment =~ s/^\s+//;
+ $comment =~ s/\s+$//;
+ push @{ $catalog{data} }, "# $comment";
+ }
+ }
+ elsif (/^$/)
+ {
+ # Preserve blank lines
+ # Newline gets added by caller.
+ push @{ $catalog{data} }, '';
+ }
+ }
+ else
+ {
+ # Strip C-style comments.
+ s;/\*(.|\n)*\*/;;g;
+ if (m;/\*;)
+ {
+ # handle multi-line comments properly.
+ my $next_line = <$ifh>;
+ die "$input_file: ends within C-style comment\n"
+ if !defined $next_line;
+ $_ .= $next_line;
+ redo;
+ }
+ }
+ # Remember input line number for later.
+ my $input_line_number = $.;
+
+ # Strip useless whitespace and trailing semicolons.
+ chomp;
+ s/^\s+//;
+ s/;\s*$//;
+ s/\s+/ /g;
+
+ # Push the data into the appropriate data structure.
+ if (
+ /^DATA\(insert(\s+OID\s+=\s+(\d+))?\s+\(\s*(.*)\s*\)\s*\)$/)
+ {
+ if ($2)
+ {
+ push @{ $catalog{data} }, { oid => $2, bki_values => $3 };
+ }
+ else
+ {
+ push @{ $catalog{data} }, { bki_values => $3 };
+ }
+ }
+ elsif (/^DESCR\(\"(.*)\"\)$/)
+ {
+ $most_recent = $catalog{data}->[-1];
+
+ # this tests if most recent line is not a DATA() statement
+ if (ref $most_recent ne 'HASH')
+ {
+ die "DESCR() does not apply to any catalog ($input_file)";
+ }
+ if (!defined $most_recent->{oid})
+ {
+ die "DESCR() does not apply to any oid ($input_file)";
+ }
+ elsif ($1 ne '')
+ {
+ $most_recent->{descr} = $1;
+ }
+ }
+ elsif (/^SHDESCR\(\"(.*)\"\)$/)
+ {
+ $most_recent = $catalog{data}->[-1];
+
+ # this tests if most recent line is not a DATA() statement
+ if (ref $most_recent ne 'HASH')
+ {
+ die
+ "SHDESCR() does not apply to any catalog ($input_file)";
+ }
+ if (!defined $most_recent->{oid})
+ {
+ die "SHDESCR() does not apply to any oid ($input_file)";
+ }
+ elsif ($1 ne '')
+ {
+ $most_recent->{shdescr} = $1;
+ }
+ }
+ elsif (/^CATALOG\(([^,]*),(\d+)\)/)
+ {
+ $catname = $1;
+ $catalog{relation_oid} = $2;
+
+ # Store pg_* catalog names in the same order we receive them
+ push @{ $catalogs{names} }, $catname;
+
+ $declaring_attributes = 1;
+ }
+ elsif ($declaring_attributes)
+ {
+ next if (/^{|^$/);
+ next if (/^#/);
+ if (/^}/)
+ {
+ undef $declaring_attributes;
+ }
+ else
+ {
+ my %column;
+ if ($is_varlen)
+ {
+ $column{is_varlen} = 1;
+ }
+ my ($atttype, $attname, $attopt) = split /\s+/, $_;
+ die "parse error ($input_file)" unless $attname;
+ if (exists $RENAME_ATTTYPE{$atttype})
+ {
+ $atttype = $RENAME_ATTTYPE{$atttype};
+ }
+ if ($attname =~ /(.*)\[.*\]/) # array attribute
+ {
+ $attname = $1;
+ $atttype .= '[]';
+ }
+
+ $column{type} = $atttype;
+ $column{name} = $attname;
+
+ push @{ $catalog{columns} }, \%column;
+ }
+ }
+ }
+ if (defined $catname)
+ {
+ $catalogs{$catname} = \%catalog;
+ }
+ close $ifh;
+ }
+ return \%catalogs;
+ }
+
+ # Split a DATA line into fields.
+ # Call this on the bki_values element of a DATA item returned by catalogs();
+ # it returns a list of field values. We don't strip quoting from the fields.
+ # Note: it should be safe to assign the result to a list of length equal to
+ # the nominal number of catalog fields, because the number of fields were
+ # checked in the original Catalog module.
+ sub split_data_line
+ {
+ my $bki_values = shift;
+
+ # This handling of quoted strings might look too simplistic, but it
+ # matches what bootscanner.l does: that has no provision for quote marks
+ # inside quoted strings, either. If we don't have a quoted string, just
+ # snarf everything till next whitespace. That will accept some things
+ # that bootscanner.l will see as erroneous tokens; but it seems wiser
+ # to do that and let bootscanner.l complain than to silently drop
+ # non-whitespace characters.
+ my @result = $bki_values =~ /"[^"]*"|\S+/g;
+
+ return @result;
+ }
+
+ sub usage
+ {
+ die <<EOM;
+ Usage: convert_macro2dat.pl [options] header...
+
+ Options:
+ -o output path
+
+ convert_macro2dat.pl generates data files from the same header files
+ currently parsed by Catalag.pm.
+
+ EOM
+ }
diff --git a/src/include/catalog/pg_aggregate.h b/src/include/catalog/pg_aggregate.h
new file mode 100644
index 13f1bce..f7be2c0
*** a/src/include/catalog/pg_aggregate.h
--- b/src/include/catalog/pg_aggregate.h
***************
*** 55,83 ****
CATALOG(pg_aggregate,2600) BKI_WITHOUT_OIDS
{
regproc aggfnoid;
! char aggkind;
! int16 aggnumdirectargs;
regproc aggtransfn;
! regproc aggfinalfn;
! regproc aggcombinefn;
! regproc aggserialfn;
! regproc aggdeserialfn;
! regproc aggmtransfn;
! regproc aggminvtransfn;
! regproc aggmfinalfn;
! bool aggfinalextra;
! bool aggmfinalextra;
! char aggfinalmodify;
! char aggmfinalmodify;
! Oid aggsortop;
Oid aggtranstype;
! int32 aggtransspace;
! Oid aggmtranstype;
! int32 aggmtransspace;
#ifdef CATALOG_VARLEN /* variable-length fields start here */
! text agginitval;
! text aggminitval;
#endif
} FormData_pg_aggregate;
--- 55,83 ----
CATALOG(pg_aggregate,2600) BKI_WITHOUT_OIDS
{
regproc aggfnoid;
! char aggkind BKI_DEFAULT(n);
! int16 aggnumdirectargs BKI_DEFAULT(0);
regproc aggtransfn;
! regproc aggfinalfn BKI_DEFAULT(-);
! regproc aggcombinefn BKI_DEFAULT(-);
! regproc aggserialfn BKI_DEFAULT(-);
! regproc aggdeserialfn BKI_DEFAULT(-);
! regproc aggmtransfn BKI_DEFAULT(-);
! regproc aggminvtransfn BKI_DEFAULT(-);
! regproc aggmfinalfn BKI_DEFAULT(-);
! bool aggfinalextra BKI_DEFAULT(f);
! bool aggmfinalextra BKI_DEFAULT(f);
! char aggfinalmodify BKI_DEFAULT(r);
! char aggmfinalmodify BKI_DEFAULT(r);
! Oid aggsortop BKI_DEFAULT(0);
Oid aggtranstype;
! int32 aggtransspace BKI_DEFAULT(0);
! Oid aggmtranstype BKI_DEFAULT(0);
! int32 aggmtransspace BKI_DEFAULT(0);
#ifdef CATALOG_VARLEN /* variable-length fields start here */
! text agginitval BKI_DEFAULT(_null_);
! text aggminitval BKI_DEFAULT(_null_);
#endif
} FormData_pg_aggregate;
diff --git a/src/include/catalog/pg_authid.h b/src/include/catalog/pg_authid.h
new file mode 100644
index 9b6b52c..984f661
*** a/src/include/catalog/pg_authid.h
--- b/src/include/catalog/pg_authid.h
***************
*** 44,63 ****
CATALOG(pg_authid,1260) BKI_SHARED_RELATION BKI_ROWTYPE_OID(2842) BKI_SCHEMA_MACRO
{
! NameData rolname; /* name of role */
! bool rolsuper; /* read this field via superuser() only! */
! bool rolinherit; /* inherit privileges from other roles? */
! bool rolcreaterole; /* allowed to create more roles? */
! bool rolcreatedb; /* allowed to create databases? */
! bool rolcanlogin; /* allowed to log in as session user? */
! bool rolreplication; /* role used for streaming replication */
! bool rolbypassrls; /* bypasses row level security? */
! int32 rolconnlimit; /* max connections allowed (-1=no limit) */
/* remaining fields may be null; use heap_getattr to read them! */
#ifdef CATALOG_VARLEN /* variable-length fields start here */
! text rolpassword; /* password, if any */
! timestamptz rolvaliduntil; /* password expiration time, if any */
#endif
} FormData_pg_authid;
--- 44,63 ----
CATALOG(pg_authid,1260) BKI_SHARED_RELATION BKI_ROWTYPE_OID(2842) BKI_SCHEMA_MACRO
{
! NameData rolname; /* name of role */
! bool rolsuper BKI_DEFAULT(f); /* read this field via superuser() only! */
! bool rolinherit BKI_DEFAULT(t); /* inherit privileges from other roles? */
! bool rolcreaterole BKI_DEFAULT(f); /* allowed to create more roles? */
! bool rolcreatedb BKI_DEFAULT(f); /* allowed to create databases? */
! bool rolcanlogin BKI_DEFAULT(f); /* allowed to log in as session user? */
! bool rolreplication BKI_DEFAULT(f); /* role used for streaming replication */
! bool rolbypassrls BKI_DEFAULT(f); /* bypasses row level security? */
! int32 rolconnlimit BKI_DEFAULT(-1); /* max connections allowed (-1=no limit) */
/* remaining fields may be null; use heap_getattr to read them! */
#ifdef CATALOG_VARLEN /* variable-length fields start here */
! text rolpassword BKI_DEFAULT(_null_); /* password, if any */
! timestamptz rolvaliduntil BKI_DEFAULT(_null_); /* password expiration time, if any */
#endif
} FormData_pg_authid;
diff --git a/src/include/catalog/pg_class.h b/src/include/catalog/pg_class.h
new file mode 100644
index b256657..435d82a
*** a/src/include/catalog/pg_class.h
--- b/src/include/catalog/pg_class.h
***************
*** 32,57 ****
CATALOG(pg_class,1259) BKI_BOOTSTRAP BKI_ROWTYPE_OID(83) BKI_SCHEMA_MACRO
{
NameData relname; /* class name */
! Oid relnamespace; /* OID of namespace containing this class */
Oid reltype; /* OID of entry in pg_type for table's
* implicit row type */
! Oid reloftype; /* OID of entry in pg_type for underlying
* composite type */
! Oid relowner; /* class owner */
! Oid relam; /* index access method; 0 if not an index */
! Oid relfilenode; /* identifier of physical storage file */
/* relfilenode == 0 means it is a "mapped" relation, see relmapper.c */
! Oid reltablespace; /* identifier of table space for relation */
! int32 relpages; /* # of blocks (not always up-to-date) */
! float4 reltuples; /* # of tuples (not always up-to-date) */
! int32 relallvisible; /* # of all-visible blocks (not always
* up-to-date) */
! Oid reltoastrelid; /* OID of toast table; 0 if none */
! bool relhasindex; /* T if has (or has had) any indexes */
! bool relisshared; /* T if shared across databases */
! char relpersistence; /* see RELPERSISTENCE_xxx constants below */
! char relkind; /* see RELKIND_xxx constants below */
int16 relnatts; /* number of user attributes */
/*
--- 32,57 ----
CATALOG(pg_class,1259) BKI_BOOTSTRAP BKI_ROWTYPE_OID(83) BKI_SCHEMA_MACRO
{
NameData relname; /* class name */
! Oid relnamespace BKI_DEFAULT(PGNSP); /* OID of namespace containing this class */
Oid reltype; /* OID of entry in pg_type for table's
* implicit row type */
! Oid reloftype BKI_DEFAULT(0); /* OID of entry in pg_type for underlying
* composite type */
! Oid relowner BKI_DEFAULT(PGUID); /* class owner */
! Oid relam BKI_DEFAULT(0); /* index access method; 0 if not an index */
! Oid relfilenode BKI_DEFAULT(0); /* identifier of physical storage file */
/* relfilenode == 0 means it is a "mapped" relation, see relmapper.c */
! Oid reltablespace BKI_DEFAULT(0); /* identifier of table space for relation */
! int32 relpages BKI_DEFAULT(0); /* # of blocks (not always up-to-date) */
! float4 reltuples BKI_DEFAULT(0); /* # of tuples (not always up-to-date) */
! int32 relallvisible BKI_DEFAULT(0); /* # of all-visible blocks (not always
* up-to-date) */
! Oid reltoastrelid BKI_DEFAULT(0); /* OID of toast table; 0 if none */
! bool relhasindex BKI_DEFAULT(f); /* T if has (or has had) any indexes */
! bool relisshared BKI_DEFAULT(f); /* T if shared across databases */
! char relpersistence BKI_DEFAULT(p); /* see RELPERSISTENCE_xxx constants below */
! char relkind BKI_DEFAULT(r); /* see RELKIND_xxx constants below */
int16 relnatts; /* number of user attributes */
/*
*************** CATALOG(pg_class,1259) BKI_BOOTSTRAP BKI
*** 59,85 ****
* (with attnums ranging from 1 to relnatts) for this class. It may also
* contain entries with negative attnums for system attributes.
*/
! int16 relchecks; /* # of CHECK constraints for class */
bool relhasoids; /* T if we generate OIDs for rows of rel */
! bool relhaspkey; /* has (or has had) PRIMARY KEY index */
! bool relhasrules; /* has (or has had) any rules */
! bool relhastriggers; /* has (or has had) any TRIGGERs */
! bool relhassubclass; /* has (or has had) derived classes */
! bool relrowsecurity; /* row security is enabled or not */
! bool relforcerowsecurity; /* row security forced for owners or
* not */
! bool relispopulated; /* matview currently holds query results */
! char relreplident; /* see REPLICA_IDENTITY_xxx constants */
! bool relispartition; /* is relation a partition? */
TransactionId relfrozenxid; /* all Xids < this are frozen in this rel */
TransactionId relminmxid; /* all multixacts in this rel are >= this.
* this is really a MultiXactId */
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* NOTE: These fields are not present in a relcache entry's rd_rel field. */
! aclitem relacl[1]; /* access permissions */
! text reloptions[1]; /* access-method-specific options */
! pg_node_tree relpartbound; /* partition bound node tree */
#endif
} FormData_pg_class;
--- 59,85 ----
* (with attnums ranging from 1 to relnatts) for this class. It may also
* contain entries with negative attnums for system attributes.
*/
! int16 relchecks BKI_DEFAULT(0); /* # of CHECK constraints for class */
bool relhasoids; /* T if we generate OIDs for rows of rel */
! bool relhaspkey BKI_DEFAULT(f); /* has (or has had) PRIMARY KEY index */
! bool relhasrules BKI_DEFAULT(f); /* has (or has had) any rules */
! bool relhastriggers BKI_DEFAULT(f); /* has (or has had) any TRIGGERs */
! bool relhassubclass BKI_DEFAULT(f); /* has (or has had) derived classes */
! bool relrowsecurity BKI_DEFAULT(f); /* row security is enabled or not */
! bool relforcerowsecurity BKI_DEFAULT(f); /* row security forced for owners or
* not */
! bool relispopulated BKI_DEFAULT(t); /* matview currently holds query results */
! char relreplident BKI_DEFAULT(n); /* see REPLICA_IDENTITY_xxx constants */
! bool relispartition BKI_DEFAULT(f); /* is relation a partition? */
TransactionId relfrozenxid; /* all Xids < this are frozen in this rel */
TransactionId relminmxid; /* all multixacts in this rel are >= this.
* this is really a MultiXactId */
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* NOTE: These fields are not present in a relcache entry's rd_rel field. */
! aclitem relacl[1] BKI_DEFAULT(_null_); /* access permissions */
! text reloptions[1] BKI_DEFAULT(_null_); /* access-method-specific options */
! pg_node_tree relpartbound BKI_DEFAULT(_null_); /* partition bound node tree */
#endif
} FormData_pg_class;
diff --git a/src/include/catalog/pg_opclass.h b/src/include/catalog/pg_opclass.h
new file mode 100644
index 28dbc74..d441e1f
*** a/src/include/catalog/pg_opclass.h
--- b/src/include/catalog/pg_opclass.h
***************
*** 50,63 ****
CATALOG(pg_opclass,2616)
{
! Oid opcmethod; /* index access method opclass is for */
! NameData opcname; /* name of this opclass */
! Oid opcnamespace; /* namespace of this opclass */
! Oid opcowner; /* opclass owner */
! Oid opcfamily; /* containing operator family */
! Oid opcintype; /* type of data indexed by opclass */
! bool opcdefault; /* T if opclass is default for opcintype */
! Oid opckeytype; /* type of data in index, or InvalidOid */
} FormData_pg_opclass;
/* ----------------
--- 50,63 ----
CATALOG(pg_opclass,2616)
{
! Oid opcmethod; /* index access method opclass is for */
! NameData opcname; /* name of this opclass */
! Oid opcnamespace BKI_DEFAULT(PGNSP); /* namespace of this opclass */
! Oid opcowner BKI_DEFAULT(PGUID); /* opclass owner */
! Oid opcfamily; /* containing operator family */
! Oid opcintype; /* type of data indexed by opclass */
! bool opcdefault BKI_DEFAULT(t); /* T if opclass is default for opcintype */
! Oid opckeytype BKI_DEFAULT(0); /* type of data in index, or InvalidOid */
} FormData_pg_opclass;
/* ----------------
diff --git a/src/include/catalog/pg_operator.h b/src/include/catalog/pg_operator.h
new file mode 100644
index ff9b470..dd4c7e1
*** a/src/include/catalog/pg_operator.h
--- b/src/include/catalog/pg_operator.h
***************
*** 33,52 ****
CATALOG(pg_operator,2617)
{
! NameData oprname; /* name of operator */
! Oid oprnamespace; /* OID of namespace containing this oper */
! Oid oprowner; /* operator owner */
! char oprkind; /* 'l', 'r', or 'b' */
! bool oprcanmerge; /* can be used in merge join? */
! bool oprcanhash; /* can be used in hash join? */
! Oid oprleft; /* left arg type, or 0 if 'l' oprkind */
! Oid oprright; /* right arg type, or 0 if 'r' oprkind */
! Oid oprresult; /* result datatype */
! Oid oprcom; /* OID of commutator oper, or 0 if none */
! Oid oprnegate; /* OID of negator oper, or 0 if none */
! regproc oprcode; /* OID of underlying function */
! regproc oprrest; /* OID of restriction estimator, or 0 */
! regproc oprjoin; /* OID of join estimator, or 0 */
} FormData_pg_operator;
/* ----------------
--- 33,52 ----
CATALOG(pg_operator,2617)
{
! NameData oprname; /* name of operator */
! Oid oprnamespace BKI_DEFAULT(PGNSP); /* OID of namespace containing this oper */
! Oid oprowner BKI_DEFAULT(PGUID); /* operator owner */
! char oprkind BKI_DEFAULT(b); /* 'l', 'r', or 'b' */
! bool oprcanmerge BKI_DEFAULT(f); /* can be used in merge join? */
! bool oprcanhash BKI_DEFAULT(f); /* can be used in hash join? */
! Oid oprleft; /* left arg type, or 0 if 'l' oprkind */
! Oid oprright; /* right arg type, or 0 if 'r' oprkind */
! Oid oprresult; /* result datatype */
! Oid oprcom BKI_DEFAULT(0); /* OID of commutator oper, or 0 if none */
! Oid oprnegate BKI_DEFAULT(0); /* OID of negator oper, or 0 if none */
! regproc oprcode; /* OID of underlying function */
! regproc oprrest BKI_DEFAULT(-); /* OID of restriction estimator, or 0 */
! regproc oprjoin BKI_DEFAULT(-); /* OID of join estimator, or 0 */
} FormData_pg_operator;
/* ----------------
diff --git a/src/include/catalog/pg_opfamily.h b/src/include/catalog/pg_opfamily.h
new file mode 100644
index 0d0ba7c..ac7df78
*** a/src/include/catalog/pg_opfamily.h
--- b/src/include/catalog/pg_opfamily.h
***************
*** 30,39 ****
CATALOG(pg_opfamily,2753)
{
! Oid opfmethod; /* index access method opfamily is for */
! NameData opfname; /* name of this opfamily */
! Oid opfnamespace; /* namespace of this opfamily */
! Oid opfowner; /* opfamily owner */
} FormData_pg_opfamily;
/* ----------------
--- 30,39 ----
CATALOG(pg_opfamily,2753)
{
! Oid opfmethod; /* index access method opfamily is for */
! NameData opfname; /* name of this opfamily */
! Oid opfnamespace BKI_DEFAULT(PGNSP); /* namespace of this opfamily */
! Oid opfowner BKI_DEFAULT(PGUID); /* opfamily owner */
} FormData_pg_opfamily;
/* ----------------
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
new file mode 100644
index c969375..18a5761
*** a/src/include/catalog/pg_proc.h
--- b/src/include/catalog/pg_proc.h
***************
*** 35,59 ****
CATALOG(pg_proc,1255) BKI_BOOTSTRAP BKI_ROWTYPE_OID(81) BKI_SCHEMA_MACRO
{
! NameData proname; /* procedure name */
! Oid pronamespace; /* OID of namespace containing this proc */
! Oid proowner; /* procedure owner */
! Oid prolang; /* OID of pg_language entry */
! float4 procost; /* estimated execution cost */
! float4 prorows; /* estimated # of rows out (if proretset) */
! Oid provariadic; /* element type of variadic array, or 0 */
! regproc protransform; /* transforms calls to it during planning */
! bool proisagg; /* is it an aggregate? */
! bool proiswindow; /* is it a window function? */
! bool prosecdef; /* security definer */
! bool proleakproof; /* is it a leak-proof function? */
! bool proisstrict; /* strict with respect to NULLs? */
! bool proretset; /* returns a set? */
! char provolatile; /* see PROVOLATILE_ categories below */
! char proparallel; /* see PROPARALLEL_ categories below */
! int16 pronargs; /* number of arguments */
! int16 pronargdefaults; /* number of arguments with defaults */
! Oid prorettype; /* OID of result type */
/*
* variable-length fields start here, but we allow direct access to
--- 35,59 ----
CATALOG(pg_proc,1255) BKI_BOOTSTRAP BKI_ROWTYPE_OID(81) BKI_SCHEMA_MACRO
{
! NameData proname; /* procedure name */
! Oid pronamespace BKI_DEFAULT(PGNSP); /* OID of namespace containing this proc */
! Oid proowner BKI_DEFAULT(PGUID); /* procedure owner */
! Oid prolang BKI_DEFAULT(12); /* OID of pg_language entry */
! float4 procost BKI_DEFAULT(1); /* estimated execution cost */
! float4 prorows BKI_DEFAULT(0); /* estimated # of rows out (if proretset) */
! Oid provariadic BKI_DEFAULT(0); /* element type of variadic array, or 0 */
! regproc protransform BKI_DEFAULT(0); /* transforms calls to it during planning */
! bool proisagg BKI_DEFAULT(f); /* is it an aggregate? */
! bool proiswindow BKI_DEFAULT(f); /* is it a window function? */
! bool prosecdef BKI_DEFAULT(f); /* security definer */
! bool proleakproof BKI_DEFAULT(f); /* is it a leak-proof function? */
! bool proisstrict BKI_DEFAULT(t); /* strict with respect to NULLs? */
! bool proretset BKI_DEFAULT(f); /* returns a set? */
! char provolatile BKI_DEFAULT(i); /* see PROVOLATILE_ categories below */
! char proparallel BKI_DEFAULT(u); /* see PROPARALLEL_ categories below */
! int16 pronargs; /* number of arguments */
! int16 pronargdefaults BKI_DEFAULT(0); /* number of arguments with defaults */
! Oid prorettype; /* OID of result type */
/*
* variable-length fields start here, but we allow direct access to
*************** CATALOG(pg_proc,1255) BKI_BOOTSTRAP BKI_
*** 62,77 ****
oidvector proargtypes; /* parameter types (excludes OUT params) */
#ifdef CATALOG_VARLEN
! Oid proallargtypes[1]; /* all param types (NULL if IN only) */
! char proargmodes[1]; /* parameter modes (NULL if IN only) */
! text proargnames[1]; /* parameter names (NULL if no names) */
! pg_node_tree proargdefaults; /* list of expression trees for argument
* defaults (NULL if none) */
! Oid protrftypes[1]; /* types for which to apply transforms */
text prosrc BKI_FORCE_NOT_NULL; /* procedure source text */
! text probin; /* secondary procedure info (can be NULL) */
! text proconfig[1]; /* procedure-local GUC settings */
! aclitem proacl[1]; /* access permissions */
#endif
} FormData_pg_proc;
--- 62,77 ----
oidvector proargtypes; /* parameter types (excludes OUT params) */
#ifdef CATALOG_VARLEN
! Oid proallargtypes[1] BKI_DEFAULT(_null_); /* all param types (NULL if IN only) */
! char proargmodes[1] BKI_DEFAULT(_null_); /* parameter modes (NULL if IN only) */
! text proargnames[1] BKI_DEFAULT(_null_); /* parameter names (NULL if no names) */
! pg_node_tree proargdefaults BKI_DEFAULT(_null_); /* list of expression trees for argument
* defaults (NULL if none) */
! Oid protrftypes[1] BKI_DEFAULT(_null_); /* types for which to apply transforms */
text prosrc BKI_FORCE_NOT_NULL; /* procedure source text */
! text probin BKI_DEFAULT(_null_); /* secondary procedure info (can be NULL) */
! text proconfig[1] BKI_DEFAULT(_null_); /* procedure-local GUC settings */
! aclitem proacl[1] BKI_DEFAULT(_null_); /* access permissions */
#endif
} FormData_pg_proc;
diff --git a/src/include/catalog/pg_type.h b/src/include/catalog/pg_type.h
new file mode 100644
index e355144..c7b8b62
*** a/src/include/catalog/pg_type.h
--- b/src/include/catalog/pg_type.h
***************
*** 37,44 ****
CATALOG(pg_type,1247) BKI_BOOTSTRAP BKI_ROWTYPE_OID(71) BKI_SCHEMA_MACRO
{
NameData typname; /* type name */
! Oid typnamespace; /* OID of namespace containing this type */
! Oid typowner; /* type owner */
/*
* For a fixed-size type, typlen is the number of bytes we use to
--- 37,44 ----
CATALOG(pg_type,1247) BKI_BOOTSTRAP BKI_ROWTYPE_OID(71) BKI_SCHEMA_MACRO
{
NameData typname; /* type name */
! Oid typnamespace BKI_DEFAULT(PGNSP); /* OID of namespace containing this type */
! Oid typowner BKI_DEFAULT(PGUID); /* type owner */
/*
* For a fixed-size type, typlen is the number of bytes we use to
*************** CATALOG(pg_type,1247) BKI_BOOTSTRAP BKI_
*** 66,72 ****
*
* If typtype is 'c', typrelid is the OID of the class' entry in pg_class.
*/
! char typtype;
/*
* typcategory and typispreferred help the parser distinguish preferred
--- 66,72 ----
*
* If typtype is 'c', typrelid is the OID of the class' entry in pg_class.
*/
! char typtype BKI_DEFAULT(b);
/*
* typcategory and typispreferred help the parser distinguish preferred
*************** CATALOG(pg_type,1247) BKI_BOOTSTRAP BKI_
*** 76,92 ****
*/
char typcategory; /* arbitrary type classification */
! bool typispreferred; /* is type "preferred" within its category? */
/*
* If typisdefined is false, the entry is only a placeholder (forward
* reference). We know the type name, but not yet anything else about it.
*/
! bool typisdefined;
! char typdelim; /* delimiter for arrays of this type */
! Oid typrelid; /* 0 if not a composite type */
/*
* If typelem is not 0 then it identifies another row in pg_type. The
--- 76,92 ----
*/
char typcategory; /* arbitrary type classification */
! bool typispreferred BKI_DEFAULT(f); /* is type "preferred" within its category? */
/*
* If typisdefined is false, the entry is only a placeholder (forward
* reference). We know the type name, but not yet anything else about it.
*/
! bool typisdefined BKI_DEFAULT(t);
! char typdelim BKI_DEFAULT(\054); /* delimiter for arrays of this type */
! Oid typrelid BKI_DEFAULT(0); /* 0 if not a composite type */
/*
* If typelem is not 0 then it identifies another row in pg_type. The
*************** CATALOG(pg_type,1247) BKI_BOOTSTRAP BKI_
*** 99,105 ****
*
* typelem != 0 and typlen == -1.
*/
! Oid typelem;
/*
* If there is a "true" array type having this type as element type,
--- 99,105 ----
*
* typelem != 0 and typlen == -1.
*/
! Oid typelem BKI_DEFAULT(0);
/*
* If there is a "true" array type having this type as element type,
*************** CATALOG(pg_type,1247) BKI_BOOTSTRAP BKI_
*** 118,130 ****
/*
* I/O functions for optional type modifiers.
*/
! regproc typmodin;
! regproc typmodout;
/*
* Custom ANALYZE procedure for the datatype (0 selects the default).
*/
! regproc typanalyze;
/* ----------------
* typalign is the alignment required when storing a value of this
--- 118,130 ----
/*
* I/O functions for optional type modifiers.
*/
! regproc typmodin BKI_DEFAULT(-);
! regproc typmodout BKI_DEFAULT(-);
/*
* Custom ANALYZE procedure for the datatype (0 selects the default).
*/
! regproc typanalyze BKI_DEFAULT(-);
/* ----------------
* typalign is the alignment required when storing a value of this
*************** CATALOG(pg_type,1247) BKI_BOOTSTRAP BKI_
*** 162,168 ****
* 'm' MAIN like 'x' but try to keep in main tuple
* ----------------
*/
! char typstorage;
/*
* This flag represents a "NOT NULL" constraint against this datatype.
--- 162,168 ----
* 'm' MAIN like 'x' but try to keep in main tuple
* ----------------
*/
! char typstorage BKI_DEFAULT(p);
/*
* This flag represents a "NOT NULL" constraint against this datatype.
*************** CATALOG(pg_type,1247) BKI_BOOTSTRAP BKI_
*** 172,203 ****
*
* Used primarily for domain types.
*/
! bool typnotnull;
/*
* Domains use typbasetype to show the base (or domain) type that the
* domain is based on. Zero if the type is not a domain.
*/
! Oid typbasetype;
/*
* Domains use typtypmod to record the typmod to be applied to their base
* type (-1 if base type does not use a typmod). -1 if this type is not a
* domain.
*/
! int32 typtypmod;
/*
* typndims is the declared number of dimensions for an array domain type
* (i.e., typbasetype is an array type). Otherwise zero.
*/
! int32 typndims;
/*
* Collation: 0 if type cannot use collations, DEFAULT_COLLATION_OID for
* collatable base types, possibly other OID for domains
*/
! Oid typcollation;
#ifdef CATALOG_VARLEN /* variable-length fields start here */
--- 172,203 ----
*
* Used primarily for domain types.
*/
! bool typnotnull BKI_DEFAULT(f);
/*
* Domains use typbasetype to show the base (or domain) type that the
* domain is based on. Zero if the type is not a domain.
*/
! Oid typbasetype BKI_DEFAULT(0);
/*
* Domains use typtypmod to record the typmod to be applied to their base
* type (-1 if base type does not use a typmod). -1 if this type is not a
* domain.
*/
! int32 typtypmod BKI_DEFAULT(-1);
/*
* typndims is the declared number of dimensions for an array domain type
* (i.e., typbasetype is an array type). Otherwise zero.
*/
! int32 typndims BKI_DEFAULT(0);
/*
* Collation: 0 if type cannot use collations, DEFAULT_COLLATION_OID for
* collatable base types, possibly other OID for domains
*/
! Oid typcollation BKI_DEFAULT(0);
#ifdef CATALOG_VARLEN /* variable-length fields start here */
*************** CATALOG(pg_type,1247) BKI_BOOTSTRAP BKI_
*** 206,212 ****
* a default expression for the type. Currently this is only used for
* domains.
*/
! pg_node_tree typdefaultbin;
/*
* typdefault is NULL if the type has no associated default value. If
--- 206,212 ----
* a default expression for the type. Currently this is only used for
* domains.
*/
! pg_node_tree typdefaultbin BKI_DEFAULT(_null_);
/*
* typdefault is NULL if the type has no associated default value. If
*************** CATALOG(pg_type,1247) BKI_BOOTSTRAP BKI_
*** 216,227 ****
* external representation of the type's default value, which may be fed
* to the type's input converter to produce a constant.
*/
! text typdefault;
/*
* Access permissions
*/
! aclitem typacl[1];
#endif
} FormData_pg_type;
--- 216,227 ----
* external representation of the type's default value, which may be fed
* to the type's input converter to produce a constant.
*/
! text typdefault BKI_DEFAULT(_null_);
/*
* Access permissions
*/
! aclitem typacl[1] BKI_DEFAULT(_null_);
#endif
} FormData_pg_type;
diff --git a/src/include/catalog/rewrite_dat.pl b/src/include/catalog/rewrite_dat.pl
new file mode 100644
index ...f6b01cf
*** a/src/include/catalog/rewrite_dat.pl
--- b/src/include/catalog/rewrite_dat.pl
***************
*** 0 ****
--- 1,346 ----
+ #!/usr/bin/perl -w
+ #----------------------------------------------------------------------
+ #
+ # rewrite_dat.pl
+ # Perl script that reads in a catalog data file and writes out
+ # a functionally equivalent file in a standard format.
+ #
+ # -Metadata fields are on their own line
+ # -Fields are in the same order they would be in the catalog table
+ # -Default values for the catalog are left out.
+ #
+ # Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ # Portions Copyright (c) 1994, Regents of the University of California
+ #
+ # /src/include/catalog/rewrite_dat.pl
+ #
+ #----------------------------------------------------------------------
+
+ use Catalog;
+
+ use strict;
+ use warnings;
+
+ my @input_files;
+ my $output_path = '';
+ my $major_version;
+
+ # Process command line switches.
+ while (@ARGV)
+ {
+ my $arg = shift @ARGV;
+ if ($arg !~ /^-/)
+ {
+ push @input_files, $arg;
+ }
+ elsif ($arg =~ /^-o/)
+ {
+ $output_path = length($arg) > 2 ? substr($arg, 2) : shift @ARGV;
+ }
+ elsif ($arg eq '--revert')
+ {
+ revert();
+ }
+ else
+ {
+ usage();
+ }
+ }
+
+ # Sanity check arguments.
+ die "No input files.\n" if !@input_files;
+ foreach my $input_file (@input_files)
+ {
+ if ($input_file !~ /\.dat$/)
+ {
+ die "Input files need to be data (.dat) files.\n";
+ }
+ }
+
+ # Make sure output_path ends in a slash.
+ if ($output_path ne '' && substr($output_path, -1) ne '/')
+ {
+ $output_path .= '/';
+ }
+
+ # We pass data file names as arguments and then look for matching
+ # headers to parse the schema from.
+ my @header_files;
+ foreach my $datfile (@input_files)
+ {
+ $datfile =~ /(.+)\.dat$/;
+ my $header = "$1.h";
+ if (-e $header)
+ {
+ push @header_files, $header;
+ }
+ else
+ {
+ die "There in no header file corresponding to $datfile";
+ }
+ }
+
+ # Metadata of a catalog entry
+ my @metafields = ('oid', 'descr', 'shdescr');
+
+ # Read all the input files into internal data structures
+ my $catalogs = Catalog::ParseHeader(@header_files);
+ my $catalog_data = parse_data_preserve_comments(@input_files);
+
+ # produce output, one catalog at a time
+ foreach my $catname (@{ $catalogs->{names} })
+ {
+ my @attnames;
+ my $catalog = $catalogs->{$catname};
+ my $schema = $catalog->{columns};
+ my @newschema;
+
+ foreach my $column (@$schema)
+ {
+ my $attname = $column->{name};
+ push @attnames, $attname;
+
+ # Everything past here is a no-op unless you are changing a
+ # default value.
+ my %newcolumn = %$column;
+
+ # Change default value here
+ # NB: After rewriting the data file, you must also change the
+ # default in the corresponding header file as well.
+ #~ if ($attname eq 'proparallel')
+ #~ {
+ #~ $newcolumn{default} = 'u';
+ #~ }
+
+ push @newschema, \%newcolumn;
+ }
+
+ if (defined $catalog_data->{$catname})
+ {
+
+ # Back up old data file rather than overwrite it.
+ # We don't assume the input path and output path are the same,
+ # but they can be.
+ my $newdatfile = "$output_path$catname.dat";
+ if (-e $newdatfile)
+ {
+ rename($newdatfile, $newdatfile . '.bak') or die "rename: $newdatfile: $!";
+ }
+ open my $dat, '>', $newdatfile
+ or die "can't open $newdatfile: $!";
+
+ foreach my $data (@{ $catalog_data->{$catname} })
+ {
+ # Either a newline or comment - just write it out.
+ if (! ref $data)
+ {
+ print $dat "$data\n";
+ }
+ # Hash ref representing a data entry.
+ elsif (ref $data eq 'HASH')
+ {
+ my %values = %$data;
+ print $dat "{ ";
+
+ # Strip default values. We always fill in the defaults
+ # first, in case we are changing the defaults on the fly.
+ Catalog::AddDefaultValues(\%values, $schema, $catname);
+ strip_default_values(\%values, \@newschema, $catname);
+
+ # We separate out a couple metadata fields for readability.
+ my $metadata_line = format_line(\%values, @metafields);
+ if ($metadata_line)
+ {
+ print $dat $metadata_line;
+ print $dat ",\n";
+ }
+ my $data_line = format_line(\%values, @attnames);
+
+ # Line up with metadata line, if there is one.
+ if ($metadata_line)
+ {
+ print $dat ' ';
+ }
+ print $dat $data_line;
+ print $dat " },\n";
+ }
+ else
+ {
+ die "Unexpected data type";
+ }
+ }
+ }
+ }
+
+ # When the build scripts consume the data, they just slurp the whole
+ # file and ignore non-data. That won't work if we want to transform
+ # the data into a standard format, because we want to preserve comments
+ # and blank lines. We have to separate data from comments, and
+ # parse data entries retail.
+ sub parse_data_preserve_comments
+ {
+ my %catalog_data;
+ foreach my $input_file (@_)
+ {
+ open(my $ifh, '<', $input_file) || die "$input_file: $!";
+ my ($filename) = ($input_file =~ m/(\w+)\.dat$/);
+ my $data = [];
+ my $prev_blank = 0;
+
+ # Scan the input file.
+ while (<$ifh>)
+ {
+ if (/^$/)
+ {
+ # Preserve non-consecutive blank lines.
+ # Newline gets added by caller.
+ if (!$prev_blank)
+ {
+ push @$data, '';
+ $prev_blank = 1;
+ }
+ }
+ else
+ {
+ $prev_blank = 0;
+ if (/{/)
+ {
+ # Capture the hash ref
+ # NB: Assumes that the next hash ref can't start on the
+ # same line where the present one ended.
+ # Not foolproof, but we shouldn't need a full lexer,
+ # since we expect relatively well-behaved input.
+
+ # Quick hack to detect when we have a full hash ref to
+ # parse. We can't just use a regex because of values in
+ # pg_aggregate and pg_proc like '{0,0}'.
+ my $lcnt = tr/{//;
+ my $rcnt = tr/}//;
+
+ if ($lcnt == $rcnt)
+ {
+ my $entry;
+ eval '$entry = ' . $_;
+ if (!ref $entry)
+ {
+ die "Error parsing $_\n";
+ }
+ push @$data, $entry;
+ }
+ else
+ {
+ my $next_line = <$ifh>;
+ $_ .= $next_line;
+ redo;
+ }
+ }
+ # Capture comments that are on their own line.
+ elsif (/^\s*#\s*(.+)\s*/)
+ {
+ my $comment = $1;
+ push @$data, "# $comment";
+ }
+ # Assume bracket is the only token in the line.
+ elsif (/^\s*(\[|\])\s*$/)
+ {
+ push @$data, $1;
+ }
+ }
+ }
+ $catalog_data{$filename} = $data;
+ }
+ return \%catalog_data;
+ }
+
+ sub strip_default_values
+ {
+ my $row = shift;
+ my $schema = shift;
+ my $catname = shift;
+
+ foreach my $column (@$schema)
+ {
+ my $attname = $column->{name};
+
+ # Delete values that match defaults.
+ if (defined $row->{$attname})
+ {
+ if (defined $column->{default}
+ and ($row->{$attname} eq $column->{default}))
+ {
+ delete $row->{$attname};
+ }
+ }
+ else
+ {
+ if (!defined $column->{default})
+ {
+ die "Missing value in $catname.$attname\n";
+ }
+ }
+ }
+ }
+
+
+ sub format_line
+ {
+ my $data = shift;
+ my @atts = @_;
+
+ my $first = 1;
+ my $value;
+ my $line = '';
+
+ foreach my $field (@atts)
+ {
+ next if !defined $data->{$field};
+ $value = $data->{$field};
+
+ # Re-escape single quotes.
+ $value =~ s/'/\\'/g;
+
+ if (!$first)
+ {
+ $line .= ', ';
+ }
+ $first = 0;
+
+ $line .= "$field => '$value'";
+ }
+ return $line;
+ }
+
+
+ # Rename .bak files back to .dat
+ # Pass the data files to the script as normal.
+ sub revert
+ {
+ foreach my $datfile (@input_files)
+ {
+ my $bakfile = "$datfile.bak";
+ if (-e $bakfile)
+ {
+ rename($bakfile, $datfile) or die "rename: $bakfile: $!";
+ }
+ }
+ exit 0;
+ }
+
+
+ sub usage
+ {
+ die <<EOM;
+ Usage: rewrite_dat.pl [options] datafile...
+
+ Options:
+ -o output path
+ --revert rename .bak files back to .dat
+
+ Expects a list of .dat files as arguments.
+
+ Rewrites data files and strips defaults values.
+ Make sure location of Catalog.pm is passed to the perl interpreter:
+ perl -I /path/to/Catalog.pm/ ...
+
+ EOM
+ }
0005_data_files_mechanical_v1.patch.tar.gzapplication/x-gzip; name=0005_data_files_mechanical_v1.patch.tar.gzDownload
� <%Z ��{����'8������]U�Y%�������Sg�e��v�����aJ�L�$Q��Q���� II$ @P���=��{)����@ �Z���M�����l�w����W�b{����P������g�Z%I����i����ZE~'�������$��c�I����3e����-��_����{s���?����A�x�q�e�_o_6��k6����������X>m����7�������Ov���O<?���>L�~�����G����^��~e����K�\m�y����.�V�d��!��M3�QG&���������x�=_��k�C�\���������?�O�O�T���l���5?�� �y��R��>���i��@;��c���'<a��a����v4�g�g�I�'�!�G��#E)�-&��T��s�O6�����Dz�9r��������:�g?�T�v*�Y*����G#`���G�����7
��H0�:��qw�Tn�z�*}��3\\�O���7������{�Z��e*P_{�/�w �]�)�UEo�����}���h�Q����?G�}sww�����OGe�u�o��r;��v���'���_x������x����o�����6�?���y����[��e�y�wO�}�������������������<V��C�����c]��})�cx�� �:�����i����M �������]����=�t��<w�k�<��-����N���l
��N��������A�D:��TH$�%"���z`��1����c,��A���w��{�o����u�">W�] �]���I��.x�c�cw� �QP���
j�0���*���7
��l�_C�S���f��B%�H����.��E�}��������+����F�f��`&��(���K1|����Zk!� ���;jk!`��R�b�?�����,���=�������B��eu�j4�o�]�g�njd�PC3����D0E p��q��.W�
��E �b������X����)���%Td�X�7�2����$�}��y���L>�)$���pW��rX4��
3n��:�[�v-*��/�<0���1�*.��� hT���U�hj�]���
���������mq|*�*�7����m}�}�d��-�j���V��)���R���$�R��W u�y�,�TM\�pD;�a�j�\o������P�'�|�f����!�A)�@�S��}�b%y�?m���W��>`�K
!`���n�j�� }<�0��cJ% _�i�?���0���T��D�`�kWc?�W�\P�� 0����-v�����3��x,��V������\�o��
@��"�����_�@ UP����m���
����h% ���F�0���B
��x ������*��c�QC�|b�R8z�T"� ��pX?G
� �L�U�2�i*����JWd+��+� �!�Q2�kF@�x �'��@��B'P�_�����I |c,�`�4<��|�V{PO`��5�b�U���'�-����m�9Dz���>����m�#=�@I��:�
� �>�P?��8!��m�J6�F��$����oC@-��u�r&��Z�
�v���2�������(���(7�|6c5 ���q��G�����C;C��4��nM��������X��X7L�%���K�hM�)�@�8H'�6�x
k���^w��L#^�|�i��F�|��Q�:�� Sz���A�_�F�p���z���Z
����-tq/ �!�����1g�x�"_���ZG/~:��4�=�:��[�j ��O�_p��/ �Sl����W��/���u��oA�Ql�V��}�����U���A��r�����x�ju&0~O��N�����mA�:A�c�y���A��~KY��2�Rg�7Rp�(�=���q�����+l��������6�M}�<�����@I����x��Mzl������Rhc��h��-�`���?��T��s5k�,S�B{4���@�Uj�� ^�M�k{��o���Q��������"���n��(a���$�w����D�*(9c��_�����Med[����2N2�'h~m������G���8������������������i��e�5�m��{��:��G���!����G�����W�~|�U��������e)]��.k�kcu[���w���v�)��2�P�*��_�z{1��u��[���:^�c�I����[J�2���9��-'B��
��"��^FK���5���-�A:���?���
����+��)t�����������t�*�����F�w�7\&���`So�Z%E��o*)e�<��^w��B�M�������~d����lZ_��[�u���*4�c�4�ku8W��$^����B?vL\;����6\�d���#/��w���jzl���G�X�����=�m�T���$�j����a����D5>� ��(�E(��d������}���A3���Xj�[�d��0x���-W��llf0������b/�=�j� � s��yU��1�tu9z�o�Cy#���}w�v]�q�
�P���S
K�Whh�3A����|��>j@��b���������o<���G��JR�_�_�p��=�/��J��v��t���sE`�a�m�/�Pv��>�w�������1�qD�q~h��,12�����'�)G������^���}���O%�h�"�g�h(��P�0��LP+{ �>ASP��Vs��P5O �.�EJ."J
P{�Ss���P-���/%��5 �gQ���a����k�7�L�>PR����n��kO�W��~]�p+�s����]�
��!�$�o���Oe�}���t����Y��V�� 5ZA��R������ ���o�:]@����j����w&��(>�e� ���/�c�H�4�������~
���#9�@�
���Y?��\���/�� (H���<8D���)v$,��>n�c��k�����?���l�Vb�P�,�S���}������m�#�S�����������_&�w�+�}[m��M��{�t��{��7���^}���-�����Z�27�����������s���p��`��^�mu�i��n7ma�����h:y�����������������zA���/�`��{���9��r�S������Y��������-���r�n�}S^Y���w'�n�2���� LV�z��������tz��nz���y��._�n�f}�cw���I2��X����ve�\o~���������.�*��3�����v�W��y���D-�'����G�S��")�����W�}�����R��")i����[,�=@x�������w��s�@��c$�0��������/� |��?��g�S��b���aN��o�(|�~�q<����fT~������/h����=��zw9�����>���3�����)�=��t;|o��e{�-�qF�z�����
3������`/�C���p�=co���7g��0�W���c;B5J�. 7@������
1���jx_��we��o}��>d��bD�Mq���8����xQ����2x1��71 ^�6F�/J���Z3
nV7m���&���#l3�� ��#|� �($����d�m�;E�j�C�k��/;�&pq�j�j��"�em� \\�����/.R�-�*}cd�jTpc�j�������1��e��"���uc�z��"Ug��F�HU��u�j��2p1��-�j4���Rob��c�l���.FS�%#��i������.6�&x1j��.oM�b�0���&x1�-l8L�b41�m|��'ff�������L�������Q�,�^
`\�&]6J5��������,��`�:��7�/���jsi�8�l
�/�A��g|q2��;����� �#�A>����������N����O��!����dE�'�����j�B�����B`���V�T�G
��B|`��t����� �(b��H4�0F��DI�b4�
0�*� ��"����'�+�,B�z���� aZ�p����-!^�g��<
^�g��"�(��X��<cd�icw�J_��7?T�+��t�yyn��de��Q�JVYI���d�_
�(u%� �@�RX�"8��+�j��#���+�������M���+�������N���o��
�i��p1��n&d;�Q^�,cO��5���(��f��^!Fy���^k��rc��#�Zs��hd��$;�B�5�v( ��&Y�%d���*^�B���.`r���8�����]�$�N
�(_IcB(�b�15hQN���Q�EyHW����{����y�T`$�A�`Bd� ���0���t���B��� .J
�����V��[���V�3A�����FO1��C�`� /N�vr�����j��.N�6������qc��0��eu>�-������UL2]�0�����d��
3j�����>����&���Z1�����zv��u�������uI���$��/J+s*K����yL��)���1�Y�P�Eibne�G�X<4m�];F��x!��s�S�4�%h ��)�D�4�1�SMo
����4�����V+��Bn��FB���[ �!���Lda-M3:*�����of\XK�����kiV�T�qa�U�x��]�q-n��u����������!M�,fT�zhU�f��2�4� "�����f�
3��=��K�W��G�2U%Z�a��*Q��3NU�"a"�8U%Ja�����
y��������:�3JGW4;qT�Q:j�f\3JGWDV"�(]Y�1���tm���am�V�����4��T�q~����
3��Z��.��G��~�C^��qy���SaF�*QCM*�(U�k��f���5~\3JU�h)C^C-����i��QaF%{C���T�Q�����fT�7�9*F���
���*��������0����6s�)�������qA/Mo*���7����2j���A"���
3.����N�����C����f���y#�3���� �y�(o�B����;�:������G���*��h��j��h�!O������?����V��U?H�����u4�i:�Q�F�h��2�(�ckH��1M�1h���V�F:���=�+�������)8�|`�4�j�@�5����4.mD�*�4.oD�,�4.qd�]1��]�.6��ZA��=�������P��"��R����>�(��z�@F)f@���2J-�|����B/�KE��A
��/���2i��A��%�L2c� c�2�+M[
2F/d�$.d�^2��G��e��>��t�)lb��I������@����%!��s�4M�� �����&
�8w����c�|���j��b'`m?G�+fJs��4J5S�h�r�4i72�(�Li���F)h��@����H7�7�X-[�����Eid���
�e�i�c ���0!:-L�u�����C�*�/9��x�|`��JU�0�<�U��b�q�����3�/fD-�h0���U�H/�G�j��I��&�x�����,�@������'����fL�����d�qW�Z]��f�A�;m[�J-=��v�B�;�I ��ii.1'��a�� \\�OsC!`\ ��%�o�N���B�,�N3"�I��������D
5�@�t4 ��D�qzjU�1�����Gmy������O���)�^t�D��� �hgN� F9N��\�B����j�a)������F:y,�3��Ap.6:�07���5ib-����4ws!�hel���b�V�4}2��:l:ff[D�,�L3�+E��
����eFs��1F/3�E&
b�^f4�c�c�2��, b�^fV���^�c��`C8�n�=����-M/*����4}m�0�V�4>�
3*1�����
3���U��I,�����T]Dp����P����T�Q9���=2�
r��X2��PN�2�4*�[Ut�:Z��'��3���������*g�i4�4FSC�FS�@�J�}M%�*j�]5������,5��r�>���4q#h�/MiHT�q��fUA�KS��+*�8_j��4��t�m�:/�O�O� /�����>0��v�������F-h�%:K�E��e2�(/jw������y�������/J�\� *^�HA�u�����2�����#Gm��0��@TC^���
�)gx��E�����&�?4%�����}b�1���C�ck���A�/���"��y=���lP9�9C��fY�{vO��2�� u\�B��
|D�b��p����cs�` FW�[����G�W1�k*��FK��� ��`��k�n1��ii��F�K��������[+\o������:�.|�j�f���5&`?�H�('2���+���[����r��d /2a���\����3��lv�X��>#����\��Frp������
��r�~�I�:���37��+��W��/��>���`�NE�X]���*vFt��:���j�k�C'5h������
�eC��%��l�
x7r���,4���.N��K�r��,�jW\R�S[
gJ�Li�����j��}�A�}��ye{���|A�i�tgz/�-�4�)��\��S�����LHg25l�����f�,�M�W�kS��U[�4L(�?BU|Z�)E9�,��h��K��Y���=�� �-�ao�����i��6��cb\�/����a�:~�1�T5-��>�������H�=E-���*
[�C�S�A{%7C�����C�7C�)����j7C�����-��!NP��d���(�@Dv3�
1����8 O���y�t�k2p>*���K����+
`o����,yU��*M��g1_��C]�� ��~N�v��Qy ����SAF!&��D�Q��m���1�`�P��A�K1T�s�A��E-A6��~���W��)�Z\c����`K��!�p!t��lc3�f��A�O����}��=���3uNLS%��
��r
�����@��)@��o�WEP1���r
P���!F����7C��)P-)�r
TKG
������f��9��d��9�4/ fdN�5�8lN�5�>lN�5�9lN�F��~F�n���D�����j[U�p}b�8~��qEe4�"����i�!D�1���P���AF��"��A���N��!�� N���Z�P@�D�>Q:�2&<��ReD�q�)YHM�J0�)f���3��A�4o���-I0#��W57���M��[|j��������,H#��{��� �N<0���P��3��5JA��[�F6���!o.�������8��c%a�� }`q�����@�O��������u6�4���@�O�x��f�Q��b<�����O�h'X���v������
�4R�(N��>0g'!jcFel���BR�F����,2�(c��4��
4��$p����}���
���F���Q/��Mf����o1���Wd�"�$C��7��Qt�6})���T���PF�)�@tkh��������4-��@�l~J�]�@�l~j`6_��)|��13��z���<#�I�3�6Y���<�'� �0�[l��9Bx�|`qN?���T�Q2���4�Bf�M��h�������FY�*24X���'l�RO�C�c?������Bd��@�,
�X2�HKCR�@gih�����Y�c�d�14I5hT���� ��F�M}�1� Pt����B:��k�S�����t�9����{���h�������V�7"�@��n����
G�����0[��
d�Z���<�(?��p�������'��lOHF��������!���]�g�
����%c�����;bT�Z��p��5<���6xQ��!���V�1��!~Ec��� T�{3��K���!�L�$yE�,��[�u����4��A�V��b�e���������& h�U��hT�8���f�qW�F�=�hT�8������9��/���oy��'p��)'�|r��g��-=�����k`�W���������F�u`U�,��L�C�>kf<#��?Y����Ta����l�����O�%j�
�[���Q�%j�-�[���S�%j�M�[���U����o9����i��E7��MN-����<X���5���q��=��l}��� [��p;��W=���]��l}��� ���p;��=���}7m�2���j_*w����N���lSN.�-�@
7,
i ��??X�y;�{�0��q0��a4�v�q���h���0�ZI+ ��f��`�i�a���9�M�v�1!u��dLH����h`B�<�>;z�&h�y
w��;�I��`v��7m��o������K�!h{�~C��N������
1[��b���7�l���������o����MS6��]�����mY�� ��������(���@F-�3��H�({�Ck�
1��1`�o���4�E�
1�W~~��e����?]?��*x ��(���]���tw���o���
0F���[�E��C^�V�Q���k7��}�+�C}����j� ��s�;�Af4��t�j�,�wd�����rg���\y�y�Aw~NE�G�#��A'