logical changeset generation v5
Hi!
I am rather pleased to announce the next version of the changeset
extraction patchset. Thanks to help from a large number of people I
think we are slowly getting to the point where it is getting
committable.
Since the last submitted version
(20121115002746.GA7692@awork2.anarazel.de) a large number of fixes and
the result of good amount of review has been added to the tree. All
bugs known to me have been fixed.
Fixes include:
* synchronous replication support
* don't peg the xmin for user tables, do it only for catalog ones.
* arbitrarily large transaction support by spilling large transactions
to disk
* spill snapshots to disk, so we can restart without waiting for a new
snapshot to be built
* Don't read all WAL from the establishment of a logical slot
* tests via SQL interface to changeset extraction
The todo list includes:
* morph the "logical slot" interface into being "replication slots" that
can also be used by streaming replication
* move some more code from snapbuild.c to decode.c to remove a largely
duplicated switch
* do some more header/comment cleanup & clarification
* move pg_receivellog into its own directory in src/bin or contrib/.
* user/developer level documentation
The patch series currently has two interfaces to logical decoding. One -
which is primarily useful for pg_regress style tests and playing around
- is SQL based, the other one uses a walsender replication connection.
A quick demonstration of the SQL interface (server needs to be started
with wal_level = logical and max_logical_slots > 0):
=# CREATE EXTENSION test_logical_decoding;
=# SELECT * FROM init_logical_replication('regression_slot', 'test_decoding');
slotname | xlog_position
-----------------+---------------
regression_slot | 0/17D5908
(1 row)
=# CREATE TABLE foo(id serial primary key, data text);
=# INSERT INTO foo(data) VALUES(1);
=# UPDATE foo SET id = -id, data = ':'||data;
=# DELETE FROM foo;
=# DROP TABLE foo;
=# SELECT * FROM start_logical_replication('regression_slot', 'now', 'hide-xids', '0');
location | xid | data
-----------+-----+--------------------------------------------------------------------------------
0/17D59B8 | 695 | BEGIN
0/17D59B8 | 695 | COMMIT
0/17E8B58 | 696 | BEGIN
0/17E8B58 | 696 | table "foo": INSERT: id[int4]:1 data[text]:1
0/17E8B58 | 696 | COMMIT
0/17E8CA8 | 697 | BEGIN
0/17E8CA8 | 697 | table "foo": UPDATE: old-pkey: id[int4]:1 new-tuple: id[int4]:-1 data[text]::1
0/17E8CA8 | 697 | COMMIT
0/17E8E50 | 698 | BEGIN
0/17E8E50 | 698 | table "foo": DELETE: id[int4]:-1
0/17E8E50 | 698 | COMMIT
0/17E9058 | 699 | BEGIN
0/17E9058 | 699 | COMMIT
(13 rows)
=# SELECT * FROM pg_stat_logical_decoding ;
slot_name | plugin | database | active | xmin | restart_decoding_lsn
-----------------+---------------+----------+--------+------+----------------------
regression_slot | test_decoding | 12042 | f | 695 | 0/17D58D0
(1 row)
=# SELECT * FROM stop_logical_replication('regression_slot');
stop_logical_replication
--------------------------
0
The walsender interface has the same calls
INIT_LOGICAL_REPLICATION 'slot' 'plugin';
START_LOGICAL_REPLICATION 'slot' restart_lsn [(option value)*];
STOP_LOGICAL_REPLICATION 'slot';
The only difference is that START_LOGICAL_REPLICATION can stream changes
and it can support synchronous replication.
The output seen in the 'data' column is produced by a so called 'output
plugin' which users of the facility can write to suit their needs. They
can be written by implementing 5 functions in the shared object that's
passed to init_logical_replication() above:
* pg_decode_init (optional)
* pg_decode_begin_txn
* pg_decode_change
* pg_decode_commit_txn
* pg_decode_cleanup (optional)
The most interesting function pg_decode_change get's passed a structure
containing old/new versions of the row, the 'struct Relation' belonging
to it and metainformation about the transaction.
The output plugin can rely on syscache lookups et al. to decode the
changed tuple in whatever fashion it wants.
I'd like to invite reviewers to first look at:
* the output plugin interface
* the walsender/SRF interface
* patch 12 which contains most of the code
When reading the code, the information flow during decoding might be
interesting:
---------------
+---------------+
| XLogReader |
+---------------+
|
XLOG Records
|
v
+---------------+
| decode.c |
+---------------+
| |
| |
v |
+---------------+ |
| snapbuild.c | HeapTupleData
+---------------+ |
| |
catalog snapshots |
| |
v v
+---------------+
|reorderbuffer.c|
+---------------+
|
HeapTuple & Metadata
|
v
+---------------+
| Output Plugin |
+---------------+
|
Whatever you want
|
v
+---------------+
| Output Handler|
| |
|WalSnd or SRF |
+---------------+
---------------
Overview of the attached patches:
0001: indirect toast tuples; required but submitted independently
0002: functions for testing; not required,
0003: (tablespace, filenode) syscache; required
0004: RelationMapFilenodeToOid: required, simple
0005: pg_relation_by_filenode() function; not required but useful
0006: Introduce InvalidCommandId: required, simple
0007: Adjust Satisfies* interface: required, mechanical,
0008: Allow walsender to attach to a database: required, needs review
0009: New GetOldestXmin() parameter; required, pretty boring
0010: Log xl_running_xact regularly in the bgwriter: required
0011: make fsync_fname() public; required, needs to be in a different file
0012: Relcache support for an Relation's primary key: required
0013: Actual changeset extraction; required
0014: Output plugin demo; not required (except for testing) but useful
0015: Add pg_receivellog program: not required but useful
0016: Add test_logical_decoding extension; not required, but contains
the tests for the feature. Uses 0014
0017: Snapshot building docs; not required
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
The git tree is at:
git://git.postgresql.org/git/users/andresfreund/postgres.git branch xlog-decoding-rebasing-cf4
http://git.postgresql.org/gitweb/?p=users/andresfreund/postgres.git;a=shortlog;h=refs/heads/xlog-decoding-rebasing-cf4
On 2013-06-15 00:48:17 +0200, Andres Freund wrote:
Overview of the attached patches:
0001: indirect toast tuples; required but submitted independently
0002: functions for testing; not required,
0003: (tablespace, filenode) syscache; required
0004: RelationMapFilenodeToOid: required, simple
0005: pg_relation_by_filenode() function; not required but useful
0006: Introduce InvalidCommandId: required, simple
0007: Adjust Satisfies* interface: required, mechanical,
0008: Allow walsender to attach to a database: required, needs review
0009: New GetOldestXmin() parameter; required, pretty boring
0010: Log xl_running_xact regularly in the bgwriter: required
0011: make fsync_fname() public; required, needs to be in a different file
0012: Relcache support for an Relation's primary key: required
0013: Actual changeset extraction; required
0014: Output plugin demo; not required (except for testing) but useful
0015: Add pg_receivellog program: not required but useful
0016: Add test_logical_decoding extension; not required, but contains
the tests for the feature. Uses 0014
0017: Snapshot building docs; not required
Version v5-01 attached
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Attachments:
0010-wal_decoding-Log-xl_running_xact-s-at-a-higher-frequ.patchtext/x-patch; charset=us-asciiDownload+67-5
0011-wal_decoding-copydir-make-fsync_fname-public.patchtext/x-patch; charset=us-asciiDownload+2-5
0012-wal_decoding-Add-information-about-a-tables-primary-.patchtext/x-patch; charset=us-asciiDownload+61-4
0013-wal_decoding-Introduce-wal-decoding-via-catalog-time.patchtext/x-patch; charset=us-asciiDownload+9101-204
0014-wal_decoding-test_decoding-Add-a-simple-decoding-mod.patchtext/x-patch; charset=us-asciiDownload+342-1
0015-wal_decoding-pg_receivellog-Introduce-pg_receivexlog.patchtext/x-patch; charset=us-asciiDownload+880-4
0016-wal_decoding-test_logical_decoding-Add-extension-for.patchtext/x-patch; charset=us-asciiDownload+1166-1
0017-wal_decoding-design-document-v2.4-and-snapshot-build.patchtext/x-patch; charset=us-asciiDownload+840-1
0001-Add-support-for-multiple-kinds-of-external-toast-dat.patchtext/x-patch; charset=us-asciiDownload+153-33
0002-wal_decoding-Add-pg_xlog_wait_remote_-apply-receive-.patchtext/x-patch; charset=us-asciiDownload+80-1
0003-wal_decoding-Add-a-new-RELFILENODE-syscache-to-fetch.patchtext/x-patch; charset=us-asciiDownload+14-1
0004-wal_decoding-Add-RelationMapFilenodeToOid-function-t.patchtext/x-patch; charset=us-asciiDownload+55-1
0005-wal_decoding-Add-pg_relation_by_filenode-to-lookup-u.patchtext/x-patch; charset=us-asciiDownload+89-2
0006-wal_decoding-Introduce-InvalidCommandId-and-declare-.patchtext/x-patch; charset=us-asciiDownload+3-3
0007-wal_decoding-Adjust-all-Satisfies-routines-to-take-a.patchtext/x-patch; charset=us-asciiDownload+90-38
0008-wal_decoding-Allow-walsender-s-to-connect-to-a-speci.patchtext/x-patch; charset=us-asciiDownload+41-15
0009-wal_decoding-Add-alreadyLocked-parameter-to-GetOldes.patchtext/x-patch; charset=us-asciiDownload+17-17
Andres Freund <andres@2ndquadrant.com> wrote:
0007: Adjust Satisfies* interface: required, mechanical,
Version v5-01 attached
I'm still working on a review and hope to post something more
substantive by this weekend, but when applying patches in numeric
order, this one did not compile cleanly.
gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -I../../../../src/include -D_GNU_SOURCE -I/usr/include/libxml2 -c -o allpaths.o allpaths.c -MMD -MP -MF .deps/allpaths.Po
vacuumlazy.c: In function ‘heap_page_is_all_visible’:
vacuumlazy.c:1725:3: warning: passing argument 1 of ‘HeapTupleSatisfiesVacuum’ from incompatible pointer type [enabled by default]
In file included from vacuumlazy.c:61:0:
../../../src/include/utils/tqual.h:84:20: note: expected ‘HeapTuple’ but argument is of type ‘HeapTupleHeader’
Could you post a new version of that?
--
Kevin Grittner
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Hi Kevin!
On 2013-06-20 15:57:07 -0700, Kevin Grittner wrote:
Andres Freund <andres@2ndquadrant.com> wrote:
0007: Adjust Satisfies* interface: required, mechanical,
Version v5-01 attached
I'm still working on a review and hope to post something more
substantive by this weekend
Cool!
, but when applying patches in numeric
order, this one did not compile cleanly.gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -I../../../../src/include -D_GNU_SOURCE -I/usr/include/libxml2 -c -o allpaths.o allpaths.c -MMD -MP -MF .deps/allpaths.Po
vacuumlazy.c: In function ‘heap_page_is_all_visible’:
vacuumlazy.c:1725:3: warning: passing argument 1 of ‘HeapTupleSatisfiesVacuum’ from incompatible pointer type [enabled by default]
In file included from vacuumlazy.c:61:0:
../../../src/include/utils/tqual.h:84:20: note: expected ‘HeapTuple’ but argument is of type ‘HeapTupleHeader’Could you post a new version of that?
Hrmpf. There was one hunk in 0013 instead of 0007.
I made sure that every commit again applies and compiles cleanly. git
rebase -i --exec to the rescue.
Found two other issues:
* recptr not assigned in 0010
* unsafe use of non-volatile variable across longjmp() 0013
Pushed and attached.
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Attachments:
0013-wal_decoding-Introduce-wal-decoding-via-catalog-time.patch.gzapplication/x-patch-gzipDownload+2-0
0016-wal_decoding-test_logical_decoding-Add-extension-for.patch.gzapplication/x-patch-gzipDownload
0017-wal_decoding-design-document-v2.4-and-snapshot-build.patch.gzapplication/x-patch-gzipDownload+0-1
Andres Freund <andres@2ndquadrant.com> wrote:
The git tree is at:
git://git.postgresql.org/git/users/andresfreund/postgres.git branch
xlog-decoding-rebasing-cf4
http://git.postgresql.org/gitweb/?p=users/andresfreund/postgres.git;a=shortlog;h=refs/heads/xlog-decoding-rebasing-cf4On 2013-06-15 00:48:17 +0200, Andres Freund wrote:
Overview of the attached patches:
0001: indirect toast tuples; required but submitted independently
0002: functions for testing; not required,
0003: (tablespace, filenode) syscache; required
0004: RelationMapFilenodeToOid: required, simple
0005: pg_relation_by_filenode() function; not required but useful
0006: Introduce InvalidCommandId: required, simple
0007: Adjust Satisfies* interface: required, mechanical,
0008: Allow walsender to attach to a database: required, needs review
0009: New GetOldestXmin() parameter; required, pretty boring
0010: Log xl_running_xact regularly in the bgwriter: required
0011: make fsync_fname() public; required, needs to be in a different file
0012: Relcache support for an Relation's primary key: required
0013: Actual changeset extraction; required
0014: Output plugin demo; not required (except for testing) but useful
0015: Add pg_receivellog program: not required but useful
0016: Add test_logical_decoding extension; not required, but contains
the tests for the feature. Uses 0014
0017: Snapshot building docs; not requiredVersion v5-01 attached
Confirmed that all 17 patch files now apply cleanly, and that `make
check-world` builds cleanly after each patch in turn.
Reviewing and testing the final result now. If that all looks
good, will submit a separate review of each patch.
Simon, do you want to do the final review and commit after I do
each piece?
--
Kevin Grittner
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Kevin Grittner <kgrittn@ymail.com> wrote:
Confirmed that all 17 patch files now apply cleanly, and that `make
check-world` builds cleanly after each patch in turn.
Just to be paranoid, I did one last build with all 17 patch files
applied to 7dfd5cd21c0091e467b16b31a10e20bbedd0a836 using this
line:
make maintainer-clean ; ./configure --prefix=$PWD/Debug --enable-debug --enable-cassert --enable-depend --with-libxml --with-libxslt --with-openssl --with-perl --with-python && make -j4 world
and it died with this:
gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -I../../../src/interfaces/libpq -I../../../src/include -D_GNU_SOURCE -I/usr/include/libxml2 -c -o pg_receivexlog.o pg_receivexlog.c -MMD -MP -MF .deps/pg_receivexlog.Po
gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -I. -I. -I../../../src/interfaces/libpq -I../../../src/bin/pg_dump -I../../../src/include -D_GNU_SOURCE -I/usr/include/libxml2 -c -o mainloop.o mainloop.c -MMD -MP -MF .deps/mainloop.Po
gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g pg_receivellog.o receivelog.o streamutil.o -L../../../src/port -lpgport -L../../../src/common -lpgcommon -L../../../src/interfaces/libpq -lpq -L../../../src/port -L../../../src/common -L/usr/lib -Wl,--as-needed -Wl,-rpath,'/home/kgrittn/pg/master/Debug/lib',--enable-new-dtags -lpgport -lpgcommon -lxslt -lxml2 -lssl -lcrypto -lz -lreadline -lcrypt -ldl -lm -o pg_receivellog
gcc: error: pg_receivellog.o: No such file or directory
make[3]: *** [pg_receivellog] Error 1
make[3]: Leaving directory `/home/kgrittn/pg/master/src/bin/pg_basebackup'
make[2]: *** [all-pg_basebackup-recurse] Error 2
make[2]: *** Waiting for unfinished jobs....
It works with this patch-on-patch:
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index a41b73c..18d02f3 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -42,6 +42,7 @@ installdirs:
uninstall:
rm -f '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_receivexlog$(X)'
+ rm -f '$(DESTDIR)$(bindir)/pg_receivellog$(X)'
clean distclean maintainer-clean:
- rm -f pg_basebackup$(X) pg_receivexlog$(X) $(OBJS) pg_basebackup.o pg_receivexlog.o pg_receivellog.o
+ rm -f pg_basebackup$(X) pg_receivexlog$(X) pg_receivellog$(X) $(OBJS) pg_basebackup.o pg_receivexlog.o pg_receivellog.oIt appears to be an omission from file 0015.
--
Kevin Grittner
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Kevin Grittner <kgrittn@ymail.com> wrote:
uninstall:
rm -f '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_receivexlog$(X)'
+ rm -f '$(DESTDIR)$(bindir)/pg_receivellog$(X)'
Oops. That part is not needed.
clean distclean maintainer-clean: - rm -f pg_basebackup$(X) pg_receivexlog$(X) $(OBJS) pg_basebackup.o pg_receivexlog.o pg_receivellog.o + rm -f pg_basebackup$(X) pg_receivexlog$(X) pg_receivellog$(X) $(OBJS) pg_basebackup.o pg_receivexlog.o pg_receivellog.o
Just that part.
--
Kevin Grittner
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Andres Freund <andres@2ndquadrant.com> wrote:
Pushed and attached.
The contrib/test_logical_decoding/sql/ddl.sql script is generating
unexpected results. For both table_with_pkey and
table_with_unique_not_null, updates of the primary key column are
showing:
old-pkey: id[int4]:0
... instead of the expected value of 2 or -2.
See attached.
--
Kevin Grittner
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
regression.diffsapplication/octet-stream; name=regression.diffsDownload+8-8
On 2013-06-23 08:27:32 -0700, Kevin Grittner wrote:
Kevin Grittner <kgrittn@ymail.com> wrote:
Confirmed that all 17 patch files now apply cleanly, and that `make
check-world` builds cleanly after each patch in turn.Just to be paranoid, I did one last build with all 17 patch files
applied to 7dfd5cd21c0091e467b16b31a10e20bbedd0a836 using this
line:make maintainer-clean ; ./configure --prefix=$PWD/Debug --enable-debug --enable-cassert --enable-depend --with-libxml --with-libxslt --with-openssl --with-perl --with-python && make -j4 world
and it died with this:
gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -I../../../src/interfaces/libpq -I../../../src/include -D_GNU_SOURCE -I/usr/include/libxml2 -c -o pg_receivexlog.o pg_receivexlog.c -MMD -MP -MF .deps/pg_receivexlog.Po
gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -I. -I. -I../../../src/interfaces/libpq -I../../../src/bin/pg_dump -I../../../src/include -D_GNU_SOURCE -I/usr/include/libxml2 -c -o mainloop.o mainloop.c -MMD -MP -MF .deps/mainloop.Po
gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g pg_receivellog.o receivelog.o streamutil.o -L../../../src/port -lpgport -L../../../src/common -lpgcommon -L../../../src/interfaces/libpq -lpq -L../../../src/port -L../../../src/common -L/usr/lib -Wl,--as-needed -Wl,-rpath,'/home/kgrittn/pg/master/Debug/lib',--enable-new-dtags -lpgport -lpgcommon -lxslt -lxml2 -lssl -lcrypto -lz -lreadline -lcrypt -ldl -lm -o pg_receivellog
gcc: error: pg_receivellog.o: No such file or directory
make[3]: *** [pg_receivellog] Error 1
make[3]: Leaving directory `/home/kgrittn/pg/master/src/bin/pg_basebackup'
make[2]: *** [all-pg_basebackup-recurse] Error 2
make[2]: *** Waiting for unfinished jobs....
I have seen that once as well. It's really rather strange since
pg_receivellog.o is a clear prerequisite for pg_receivellog. I couldn't
reproduce it reliably though, even after doing some dozen rebuilds or so.
It works with this patch-on-patch:
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile index a41b73c..18d02f3 100644 --- a/src/bin/pg_basebackup/Makefile +++ b/src/bin/pg_basebackup/Makefile @@ -42,6 +42,7 @@ installdirs: uninstall: rm -f '$(DESTDIR)$(bindir)/pg_basebackup$(X)' rm -f '$(DESTDIR)$(bindir)/pg_receivexlog$(X)' + rm -f '$(DESTDIR)$(bindir)/pg_receivellog$(X)' clean distclean maintainer-clean: - rm -f pg_basebackup$(X) pg_receivexlog$(X) $(OBJS) pg_basebackup.o pg_receivexlog.o pg_receivellog.o + rm -f pg_basebackup$(X) pg_receivexlog$(X) pg_receivellog$(X) $(OBJS) pg_basebackup.o pg_receivexlog.o pg_receivellog.oIt appears to be an omission from file 0015.
Yes, both are missing.
+ rm -f '$(DESTDIR)$(bindir)/pg_receivellog$(X)'
Oops. That part is not needed.
Hm. Why not?
I don't think either hunk has anything to do with that buildfailure
though - can you reproduce the error without?
Thanks,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2013-06-23 10:32:05 -0700, Kevin Grittner wrote:
Andres Freund <andres@2ndquadrant.com> wrote:
Pushed and attached.
The contrib/test_logical_decoding/sql/ddl.sql script is generating
unexpected results. For both table_with_pkey and
table_with_unique_not_null, updates of the primary key column are
showing:old-pkey: id[int4]:0
... instead of the expected value of 2 or -2.
See attached.
Hm. Any chance this was an incomplete rebuild? I seem to remember having
seen that once because some header dependency wasn't recognized
correctly after applying some patch.
Otherwise, could you give me:
* the version you aplied the patch on
* os/compiler
Because I can't reproduce it, despite some playing around...
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Andres Freund <andres@2ndquadrant.com> writes:
On 2013-06-23 08:27:32 -0700, Kevin Grittner wrote:
gcc: error: pg_receivellog.o: No such file or directory
make[3]: *** [pg_receivellog] Error 1
I have seen that once as well. It's really rather strange since
pg_receivellog.o is a clear prerequisite for pg_receivellog. I couldn't
reproduce it reliably though, even after doing some dozen rebuilds or so.
What versions of gmake are you guys using? It wouldn't be the first
time we've tripped over bugs in parallel make. See for instance
http://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=1fc698cf14d17a3a8ad018cf9ec100198a339447
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2013-06-23 16:48:41 -0400, Tom Lane wrote:
Andres Freund <andres@2ndquadrant.com> writes:
On 2013-06-23 08:27:32 -0700, Kevin Grittner wrote:
gcc: error: pg_receivellog.o: No such file or directory
make[3]: *** [pg_receivellog] Error 1I have seen that once as well. It's really rather strange since
pg_receivellog.o is a clear prerequisite for pg_receivellog. I couldn't
reproduce it reliably though, even after doing some dozen rebuilds or so.What versions of gmake are you guys using? It wouldn't be the first
time we've tripped over bugs in parallel make. See for instance
http://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=1fc698cf14d17a3a8ad018cf9ec100198a339447
3.81 here. That was supposed to be the "safe" one, right? At least to
the bugs seen/fixed recently.
Kevin, any chance you still have more log than in the upthread mail
available?
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Andres Freund <andres@2ndquadrant.com> wrote:
On 2013-06-23 08:27:32 -0700, Kevin Grittner wrote:
make maintainer-clean ; ./configure --prefix=$PWD/Debug --enable-debug
--enable-cassert --enable-depend --with-libxml --with-libxslt --with-openssl
--with-perl --with-python && make -j4 world
[ build failure referencing pg_receivellog.o ]
I have seen that once as well. It's really rather strange since
pg_receivellog.o is a clear prerequisite for pg_receivellog. I couldn't
reproduce it reliably though, even after doing some dozen rebuilds or so.It works with this patch-on-patch:
clean distclean maintainer-clean: - rm -f pg_basebackup$(X) pg_receivexlog$(X) $(OBJS) pg_basebackup.o pg_receivexlog.o pg_receivellog.o + rm -f pg_basebackup$(X) pg_receivexlog$(X) pg_receivellog$(X) $(OBJS) pg_basebackup.o pg_receivexlog.o pg_receivellog.o
+ rm -f '$(DESTDIR)$(bindir)/pg_receivellog$(X)'
Oops. That part is not needed.
Hm. Why not?
Well, I could easily be wrong on just about anything to do with
make files, but on a second look that appeared to be dealing with
eliminating an installed pg_receivellog binary, which is not
created.
I don't think either hunk has anything to do with that buildfailure
though - can you reproduce the error without?
I tried that scenario three times and it failed three times. Then
I made the above changes and it worked. Then I eliminated the one
on the uninstall target and tried a couple more times and it worked
on both attempts. The scenario is to have a `make world` build in
the source tree, and run the above line starting with `make
maintainer-clean` and going to `make -j4 world`.
I did notice that without that change to the maintainer-clean
target I did not get a pg_receivellog.Po file in
src/bin/pg_basebackup/.deps/ -- and with it I do. I admit to being
at about a 1.5 on a 10 point scale of make file competence -- I
just look for patterns used for things similar to what I want to do
and copy without much understanding of what it all means. :-( So
when I got an error on pg_receivellog which didn't happen on
pg_receivexlog, I looked for differences -- my suggestion has no
more basis than that and the fact that empirical testing seemed to
show that it worked.
--
Kevin Grittner
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Andres Freund <andres@2ndquadrant.com> wrote:
On 2013-06-23 16:48:41 -0400, Tom Lane wrote:
Andres Freund <andres@2ndquadrant.com> writes:
On 2013-06-23 08:27:32 -0700, Kevin Grittner wrote:
gcc: error: pg_receivellog.o: No such file or directory
make[3]: *** [pg_receivellog] Error 1I have seen that once as well. It's really rather strange since
pg_receivellog.o is a clear prerequisite for pg_receivellog. I
couldn't reproduce it reliably though, even after doing some
dozen rebuilds or so.What versions of gmake are you guys using? It wouldn't be the
first time we've tripped over bugs in parallel make. See for
instance
http://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=1fc698cf14d17a3a8ad018cf9ec100198a3394473.81 here. That was supposed to be the "safe" one, right? At
least to the bugs seen/fixed recently.
There is no executable named gmake in my distro, but...
kgrittn@Kevin-Desktop:~/pg/master$ make --version
GNU Make 3.81
Which is what I'm using.
Kevin, any chance you still have more log than in the upthread
mail available?
Well, I just copied from the console, and that was gone; but
reverting my change I get the same thing. All console output
attached. Let me know if you need something else.
Note that the dependency file disappeared:
kgrittn@Kevin-Desktop:~/pg/master$ ll src/bin/pg_basebackup/.deps/
total 24
drwxrwxr-x 2 kgrittn kgrittn 4096 Jun 24 08:57 ./
drwxrwxr-x 4 kgrittn kgrittn 4096 Jun 24 08:57 ../
-rw-rw-r-- 1 kgrittn kgrittn 1298 Jun 24 08:57 pg_basebackup.Po
-rw-rw-r-- 1 kgrittn kgrittn 1729 Jun 24 08:57 pg_receivexlog.Po
-rw-rw-r-- 1 kgrittn kgrittn 1646 Jun 24 08:57 receivelog.Po
-rw-rw-r-- 1 kgrittn kgrittn 953 Jun 24 08:57 streamutil.Po
It was there from the build with the change I made to the
maintainer-clean target, and went away when I built without it.
--
Kevin Grittner
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
Andres Freund <andres@2ndquadrant.com> wrote:
On 2013-06-23 10:32:05 -0700, Kevin Grittner wrote:
The contrib/test_logical_decoding/sql/ddl.sql script is generating
unexpected results. For both table_with_pkey and
table_with_unique_not_null, updates of the primary key column are
showing:old-pkey: id[int4]:0
... instead of the expected value of 2 or -2.
See attached.
Hm. Any chance this was an incomplete rebuild?
With my hack on the pg_basebackup Makefile, `make -j4 world` is
finishing with no errors and:
PostgreSQL, contrib, and documentation successfully made. Ready to install.
I seem to remember having seen that once because some header
dependency wasn't recognized correctly after applying some patch.
I wonder whether this is related to the build problems we've been
discussing on the other fork of this thread. I was surprised to
see this error when I got past the maintainer-clean full build
problems, because I thought I had seen clean `make check-world`
regression tests after applying each incremental patch file. Until
I read this I had been assuming that somehow I missed the error on
the 16th and 17th iterations; but now I'm suspecting that I didn't
miss anything after all -- it may just be another symptom of the
build problems.
Otherwise, could you give me:
* the version you aplied the patch on
7dfd5cd21c0091e467b16b31a10e20bbedd0a836
* os/compiler
Linux Kevin-Desktop 3.5.0-34-generic #55-Ubuntu SMP Thu Jun 6 20:18:19 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
gcc (Ubuntu/Linaro 4.7.2-2ubuntu1) 4.7.2
Because I can't reproduce it, despite some playing around...
Maybe if you can reproduce the build problems I'm seeing....
--
Kevin Grittner
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2013-06-24 06:44:53 -0700, Kevin Grittner wrote:
Andres Freund <andres@2ndquadrant.com> wrote:
On 2013-06-23 08:27:32 -0700, Kevin Grittner wrote:
make maintainer-clean ; ./configure --prefix=$PWD/Debug --enable-debug
--enable-cassert --enable-depend --with-libxml --with-libxslt --with-openssl
--with-perl --with-python && make -j4 world[ build failure referencing pg_receivellog.o ]
I have seen that once as well. It's really rather strange since
pg_receivellog.o is a clear prerequisite for pg_receivellog. I couldn't
reproduce it reliably though, even after doing some dozen rebuilds or so.It works with this patch-on-patch:
clean distclean maintainer-clean: - rm -f pg_basebackup$(X) pg_receivexlog$(X) $(OBJS) pg_basebackup.o pg_receivexlog.o pg_receivellog.o + rm -f pg_basebackup$(X) pg_receivexlog$(X) pg_receivellog$(X) $(OBJS) pg_basebackup.o pg_receivexlog.o pg_receivellog.o+ rm -f '$(DESTDIR)$(bindir)/pg_receivellog$(X)'
Oops. That part is not needed.
Hm. Why not?
Well, I could easily be wrong on just about anything to do with
make files, but on a second look that appeared to be dealing with
eliminating an installed pg_receivellog binary, which is not
created.
I think it actually is?
install: all installdirs
$(INSTALL_PROGRAM) pg_basebackup$(X) '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
$(INSTALL_PROGRAM) pg_receivexlog$(X) '$(DESTDIR)$(bindir)/pg_receivexlog$(X)'
$(INSTALL_PROGRAM) pg_receivellog$(X) '$(DESTDIR)$(bindir)/pg_receivellog$(X)'
I don't think either hunk has anything to do with that buildfailure
though - can you reproduce the error without?I tried that scenario three times and it failed three times. Then
I made the above changes and it worked. Then I eliminated the one
on the uninstall target and tried a couple more times and it worked
on both attempts. The scenario is to have a `make world` build in
the source tree, and run the above line starting with `make
maintainer-clean` and going to `make -j4 world`.
Hm. I think it might be something in makes intermediate target logic
biting us. Anyway, if the patch fixes that: Great ;). Merged it logally
since it's obviously missing.
I did notice that without that change to the maintainer-clean
target I did not get a pg_receivellog.Po file in
src/bin/pg_basebackup/.deps/ -- and with it I do.
Yea, according to your log it's not even built before pg_receivellog is
linked.
Thanks,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Andres Freund <andres@2ndquadrant.com> wrote:
On 2013-06-24 06:44:53 -0700, Kevin Grittner wrote:
Andres Freund <andres@2ndquadrant.com> wrote:
On 2013-06-23 08:27:32 -0700, Kevin Grittner wrote:
+ rm -f '$(DESTDIR)$(bindir)/pg_receivellog$(X)'
Oops. That part is not needed.
Hm. Why not?
Well, I could easily be wrong on just about anything to do with
make files, but on a second look that appeared to be dealing with
eliminating an installed pg_receivellog binary, which is not
created.I think it actually is?
Oh, yeah.... I see it now. I warned you I could be wrong. :-/
I just had a thought thought -- perhaps the dependency information
is being calculated incorrectly. Attached is the dependency file
from the successful build (with the adjusted Makefile), which still
fails the test_logical_decoding regression test, with the same diff.
--
Kevin Grittner
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
pg_receivellog.Potext/x-gettext-translation; name=pg_receivellog.PoDownload
On 2013-06-24 07:29:43 -0700, Kevin Grittner wrote:
Andres Freund <andres@2ndquadrant.com> wrote:
On 2013-06-23 10:32:05 -0700, Kevin Grittner wrote:
The contrib/test_logical_decoding/sql/ddl.sql script is generating
unexpected results. For both table_with_pkey and
table_with_unique_not_null, updates of the primary key column are
showing:old-pkey: id[int4]:0
... instead of the expected value of 2 or -2.
See attached.
Hm. Any chance this was an incomplete rebuild?
With my hack on the pg_basebackup Makefile, `make -j4 world` is
finishing with no errors and:
Hm. There were some issues with the test_logical_decoding Makefile not
cleaning up the regression installation properly. Which might have
caused the issue.
Could you try after applying the patches and executing a clean and then
rebuild?
Otherwise, could you try applying my git tree so we are sure we test the
same thing?
$ git remote add af git://git.postgresql.org/git/users/andresfreund/postgres.git
$ git fetch af
$ git checkout -b xlog-decoding af/xlog-decoding-rebasing-cf4
$ ./configure ...
$ make
Because I can't reproduce it, despite some playing around...
Maybe if you can reproduce the build problems I'm seeing....
Tried your recipe but still couldn't...
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
I'm looking at the combined patches 0003-0005, which are essentially all
about adding a function to obtain relation OID from (tablespace,
filenode). It takes care to look through the relation mapper, and uses
a new syscache underneath for performance.
One question about this patch, originally, was about the usage of
that relfilenode syscache. It is questionable because it would be the
only syscache to apply on top of a non-unique index. It is said that
this doesn't matter because the only non-unique values that can exist
would reference entries that have relfilenode = 0; and in turn this
doesn't matter because those values would be queried through the
relation mapper anyway, not from the syscache. (This is implemented in
the higher-level function.)
This means that there would be one syscache that is damn easy to misuse
.. and we've setup things so that syscaches are very easy to use in the
first place. From that perspective, this doesn't look good. However,
it's an easy mistake to notice and fix, so perhaps this is not a serious
problem. (I would much prefer for there to be a way to define partial
indexes in BKI.)
I'm not sure about the placing of the new SQL-callable function in
dbsize.c either. It is certainly not a function that has anything to do
with object sizes. The insides of it would belong more in lsyscache.c,
I think, except then that file does not otherwise concern itself with
the relation mapper so its scope would have to expand a bit. But this
is no place for the SQL-callable portion, so that would have to find a
different home as well.
The other option, of course, it to provide a separate caching layer for
these objects altogether, but given how concise this implementation is,
it doesn't sound too palatable.
Thoughts?
--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Hi,
On 2013-06-27 17:33:04 -0400, Alvaro Herrera wrote:
One question about this patch, originally, was about the usage of
that relfilenode syscache. It is questionable because it would be the
only syscache to apply on top of a non-unique index. It is said that
this doesn't matter because the only non-unique values that can exist
would reference entries that have relfilenode = 0; and in turn this
doesn't matter because those values would be queried through the
relation mapper anyway, not from the syscache. (This is implemented in
the higher-level function.)
Well, you can even query the syscache without hurt for mapped relations,
you just won't get an answer. The only thing you may not do because it
would yield multiple results is to query the syscache with
(tablespace, InvalidOid/0). Which is still not nice although it doesn't
make much sense to query with InvalidOid.
I'm not sure about the placing of the new SQL-callable function in
dbsize.c either. It is certainly not a function that has anything to do
with object sizes.
Not happy with that myself. I only placed the function there because
pg_relation_filenode() already was in it. Happy to change if somebody
has a good idea.
(I would much prefer for there to be a way to define partial
indexes in BKI.)
I don't think that's the hard part, it's that we don't use the full
machinery for updating indexes but rather the relatively simplistic
CatalogUpdateIndexes(). I am not sure we can guarantee that the required
infrastructure is available in all the cases to support doing generic
predicate evaluation.
Should bki really be the problem we probably could create the index
after bki based bootstrapping finished.
Thanks,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
