pgsql: Allow read only connections during recovery, known as Hot

Started by Nonameabout 16 years ago4 messages

sriggs@postgresql.org

about 16 years ago

Log Message:
-----------
Allow read only connections during recovery, known as Hot Standby.

Enabled by recovery_connections = on (default) and forcing archive recovery using a recovery.conf. Recovery processing now emulates the original transactions as they are replayed, providing full locking and MVCC behaviour for read only queries. Recovery must enter consistent state before connections are allowed, so there is a delay, typically short, before connections succeed. Replay of recovering transactions can conflict and in some cases deadlock with queries during recovery; these result in query cancellation after max_standby_delay seconds have expired. Infrastructure changes have minor effects on normal running, though introduce four new types of WAL record.

New test mode "make standbycheck" allows regression tests of static command behaviour on a standby server while in recovery. Typical and extreme dynamic behaviours have been checked via code inspection and manual testing. Few port specific behaviours have been utilised, though primary testing has been on Linux only so far.

This commit is the basic patch. Additional changes will follow in this release to enhance some aspects of behaviour, notably improved handling of conflicts, deadlock detection and query cancellation. Changes to VACUUM FULL are also required.

Simon Riggs, with significant and lengthy review by Heikki Linnakangas, including streamlined redesign of snapshot creation and two-phase commit.

Important contributions from Florian Pflug, Mark Kirkwood, Merlin Moncure, Greg Stark, Gianni Ciolli, Gabriele Bartolini, Hannu Krosing, Robert Haas, Tatsuo Ishii, Hiroyuki Yamada plus support and feedback from many other community members.

Robert Haas

robertmhaas@gmail.com

about 16 years ago

In reply to: Noname (#1)

Re: [COMMITTERS] pgsql: Allow read only connections during recovery, known as Hot

On Fri, Dec 18, 2009 at 8:32 PM, Simon Riggs <sriggs@postgresql.org> wrote:

Log Message:
-----------
Allow read only connections during recovery, known as Hot Standby.

Enabled by recovery_connections = on (default) and forcing archive recovery using a recovery.conf. Recovery processing now emulates the original transactions as they are replayed, providing full locking and MVCC behaviour for read only queries. Recovery must enter consistent state before connections are allowed, so there is a delay, typically short, before connections succeed. Replay of recovering transactions can conflict and in some cases deadlock with queries during recovery; these result in query cancellation after max_standby_delay seconds have expired. Infrastructure changes have minor effects on normal running, though introduce four new types of WAL record.

New test mode "make standbycheck" allows regression tests of static command behaviour on a standby server while in recovery. Typical and extreme dynamic behaviours have been checked via code inspection and manual testing. Few port specific behaviours have been utilised, though primary testing has been on Linux only so far.

This commit is the basic patch. Additional changes will follow in this release to enhance some aspects of behaviour, notably improved handling of conflicts, deadlock detection and query cancellation. Changes to VACUUM FULL are also required.

Simon Riggs, with significant and lengthy review by Heikki Linnakangas, including streamlined redesign of snapshot creation and two-phase commit.

Important contributions from Florian Pflug, Mark Kirkwood, Merlin Moncure, Greg Stark, Gianni Ciolli, Gabriele Bartolini, Hannu Krosing, Robert Haas, Tatsuo Ishii, Hiroyuki Yamada plus support and feedback from many other community members.

Modified Files:
--------------
pgsql/doc/src/sgml:
backup.sgml (r2.130 -> r2.131)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/doc/src/sgml/backup.sgml?r1=2.130&r2=2.131)
config.sgml (r1.238 -> r1.239)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/doc/src/sgml/config.sgml?r1=1.238&r2=1.239)
func.sgml (r1.493 -> r1.494)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/doc/src/sgml/func.sgml?r1=1.493&r2=1.494)
pgsql/doc/src/sgml/ref:
checkpoint.sgml (r1.16 -> r1.17)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/doc/src/sgml/ref/checkpoint.sgml?r1=1.16&r2=1.17)
pgsql/src/backend/access/gin:
ginxlog.c (r1.19 -> r1.20)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/access/gin/ginxlog.c?r1=1.19&r2=1.20)
pgsql/src/backend/access/gist:
gistxlog.c (r1.32 -> r1.33)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/access/gist/gistxlog.c?r1=1.32&r2=1.33)
pgsql/src/backend/access/heap:
heapam.c (r1.278 -> r1.279)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/access/heap/heapam.c?r1=1.278&r2=1.279)
pruneheap.c (r1.18 -> r1.19)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/access/heap/pruneheap.c?r1=1.18&r2=1.19)
pgsql/src/backend/access/index:
genam.c (r1.77 -> r1.78)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/access/index/genam.c?r1=1.77&r2=1.78)
indexam.c (r1.115 -> r1.116)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/access/index/indexam.c?r1=1.115&r2=1.116)
pgsql/src/backend/access/nbtree:
README (r1.20 -> r1.21)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/access/nbtree/README?r1=1.20&r2=1.21)
nbtinsert.c (r1.174 -> r1.175)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/access/nbtree/nbtinsert.c?r1=1.174&r2=1.175)
nbtpage.c (r1.113 -> r1.114)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/access/nbtree/nbtpage.c?r1=1.113&r2=1.114)
nbtree.c (r1.172 -> r1.173)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/access/nbtree/nbtree.c?r1=1.172&r2=1.173)
nbtxlog.c (r1.55 -> r1.56)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/access/nbtree/nbtxlog.c?r1=1.55&r2=1.56)
pgsql/src/backend/access/transam:
README (r1.12 -> r1.13)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/access/transam/README?r1=1.12&r2=1.13)
clog.c (r1.53 -> r1.54)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/access/transam/clog.c?r1=1.53&r2=1.54)
multixact.c (r1.32 -> r1.33)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/access/transam/multixact.c?r1=1.32&r2=1.33)
recovery.conf.sample (r1.4 -> r1.5)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/access/transam/recovery.conf.sample?r1=1.4&r2=1.5)
rmgr.c (r1.27 -> r1.28)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/access/transam/rmgr.c?r1=1.27&r2=1.28)
subtrans.c (r1.24 -> r1.25)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/access/transam/subtrans.c?r1=1.24&r2=1.25)
twophase.c (r1.56 -> r1.57)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/access/transam/twophase.c?r1=1.56&r2=1.57)
twophase_rmgr.c (r1.10 -> r1.11)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/access/transam/twophase_rmgr.c?r1=1.10&r2=1.11)
xact.c (r1.277 -> r1.278)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/access/transam/xact.c?r1=1.277&r2=1.278)
xlog.c (r1.353 -> r1.354)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/access/transam/xlog.c?r1=1.353&r2=1.354)
pgsql/src/backend/commands:
dbcommands.c (r1.228 -> r1.229)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/commands/dbcommands.c?r1=1.228&r2=1.229)
lockcmds.c (r1.25 -> r1.26)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/commands/lockcmds.c?r1=1.25&r2=1.26)
sequence.c (r1.162 -> r1.163)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/commands/sequence.c?r1=1.162&r2=1.163)
tablespace.c (r1.63 -> r1.64)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/commands/tablespace.c?r1=1.63&r2=1.64)
vacuum.c (r1.398 -> r1.399)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/commands/vacuum.c?r1=1.398&r2=1.399)
vacuumlazy.c (r1.124 -> r1.125)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/commands/vacuumlazy.c?r1=1.124&r2=1.125)
pgsql/src/backend/postmaster:
postmaster.c (r1.596 -> r1.597)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/postmaster/postmaster.c?r1=1.596&r2=1.597)
pgsql/src/backend/storage/ipc:
Makefile (r1.22 -> r1.23)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/storage/ipc/Makefile?r1=1.22&r2=1.23)
procarray.c (r1.51 -> r1.52)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/storage/ipc/procarray.c?r1=1.51&r2=1.52)
sinvaladt.c (r1.79 -> r1.80)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/storage/ipc/sinvaladt.c?r1=1.79&r2=1.80)
pgsql/src/backend/storage/lmgr:
README (r1.24 -> r1.25)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/storage/lmgr/README?r1=1.24&r2=1.25)
lock.c (r1.188 -> r1.189)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/storage/lmgr/lock.c?r1=1.188&r2=1.189)
proc.c (r1.209 -> r1.210)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/storage/lmgr/proc.c?r1=1.209&r2=1.210)
pgsql/src/backend/tcop:
postgres.c (r1.578 -> r1.579)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/tcop/postgres.c?r1=1.578&r2=1.579)
utility.c (r1.324 -> r1.325)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/tcop/utility.c?r1=1.324&r2=1.325)
pgsql/src/backend/utils/adt:
txid.c (r1.8 -> r1.9)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/utils/adt/txid.c?r1=1.8&r2=1.9)
xid.c (r1.12 -> r1.13)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/utils/adt/xid.c?r1=1.12&r2=1.13)
pgsql/src/backend/utils/cache:
inval.c (r1.89 -> r1.90)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/utils/cache/inval.c?r1=1.89&r2=1.90)
pgsql/src/backend/utils/error:
elog.c (r1.219 -> r1.220)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/utils/error/elog.c?r1=1.219&r2=1.220)
pgsql/src/backend/utils/init:
postinit.c (r1.198 -> r1.199)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/utils/init/postinit.c?r1=1.198&r2=1.199)
pgsql/src/backend/utils/misc:
guc.c (r1.527 -> r1.528)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/utils/misc/guc.c?r1=1.527&r2=1.528)
postgresql.conf.sample (r1.271 -> r1.272)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/utils/misc/postgresql.conf.sample?r1=1.271&r2=1.272)
pgsql/src/backend/utils/time:
snapmgr.c (r1.12 -> r1.13)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/utils/time/snapmgr.c?r1=1.12&r2=1.13)
tqual.c (r1.113 -> r1.114)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/utils/time/tqual.c?r1=1.113&r2=1.114)
pgsql/src/bin/pg_controldata:
pg_controldata.c (r1.44 -> r1.45)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/bin/pg_controldata/pg_controldata.c?r1=1.44&r2=1.45)
pgsql/src/include:
miscadmin.h (r1.215 -> r1.216)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/include/miscadmin.h?r1=1.215&r2=1.216)
pgsql/src/include/access:
heapam.h (r1.144 -> r1.145)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/include/access/heapam.h?r1=1.144&r2=1.145)
htup.h (r1.107 -> r1.108)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/include/access/htup.h?r1=1.107&r2=1.108)
nbtree.h (r1.125 -> r1.126)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/include/access/nbtree.h?r1=1.125&r2=1.126)
relscan.h (r1.67 -> r1.68)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/include/access/relscan.h?r1=1.67&r2=1.68)
rmgr.h (r1.19 -> r1.20)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/include/access/rmgr.h?r1=1.19&r2=1.20)
subtrans.h (r1.12 -> r1.13)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/include/access/subtrans.h?r1=1.12&r2=1.13)
transam.h (r1.70 -> r1.71)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/include/access/transam.h?r1=1.70&r2=1.71)
twophase.h (r1.12 -> r1.13)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/include/access/twophase.h?r1=1.12&r2=1.13)
twophase_rmgr.h (r1.9 -> r1.10)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/include/access/twophase_rmgr.h?r1=1.9&r2=1.10)
xact.h (r1.98 -> r1.99)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/include/access/xact.h?r1=1.98&r2=1.99)
xlog.h (r1.93 -> r1.94)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/include/access/xlog.h?r1=1.93&r2=1.94)
xlog_internal.h (r1.25 -> r1.26)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/include/access/xlog_internal.h?r1=1.25&r2=1.26)
pgsql/src/include/catalog:
pg_control.h (r1.44 -> r1.45)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/include/catalog/pg_control.h?r1=1.44&r2=1.45)
pg_proc.h (r1.556 -> r1.557)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/include/catalog/pg_proc.h?r1=1.556&r2=1.557)
pgsql/src/include/storage:
lock.h (r1.116 -> r1.117)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/include/storage/lock.h?r1=1.116&r2=1.117)
proc.h (r1.114 -> r1.115)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/include/storage/proc.h?r1=1.114&r2=1.115)
procarray.h (r1.26 -> r1.27)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/include/storage/procarray.h?r1=1.26&r2=1.27)
sinval.h (r1.53 -> r1.54)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/include/storage/sinval.h?r1=1.53&r2=1.54)
sinvaladt.h (r1.51 -> r1.52)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/include/storage/sinvaladt.h?r1=1.51&r2=1.52)
pgsql/src/include/utils:
builtins.h (r1.341 -> r1.342)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/include/utils/builtins.h?r1=1.341&r2=1.342)
snapshot.h (r1.5 -> r1.6)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/include/utils/snapshot.h?r1=1.5&r2=1.6)
pgsql/src/test/regress:
GNUmakefile (r1.80 -> r1.81)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/test/regress/GNUmakefile?r1=1.80&r2=1.81)
pg_regress.c (r1.67 -> r1.68)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/test/regress/pg_regress.c?r1=1.67&r2=1.68)

Added Files:
-----------
pgsql/src/backend/storage/ipc:
standby.c (r1.1)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/storage/ipc/standby.c?rev=1.1&content-type=text/x-cvsweb-markup)
pgsql/src/include/storage:
standby.h (r1.1)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/include/storage/standby.h?rev=1.1&content-type=text/x-cvsweb-markup)
pgsql/src/test/regress:
standby_schedule (r1.1)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/test/regress/standby_schedule?rev=1.1&content-type=text/x-cvsweb-markup)
pgsql/src/test/regress/expected:
hs_standby_allowed.out (r1.1)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/test/regress/expected/hs_standby_allowed.out?rev=1.1&content-type=text/x-cvsweb-markup)
hs_standby_check.out (r1.1)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/test/regress/expected/hs_standby_check.out?rev=1.1&content-type=text/x-cvsweb-markup)
hs_standby_disallowed.out (r1.1)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/test/regress/expected/hs_standby_disallowed.out?rev=1.1&content-type=text/x-cvsweb-markup)
hs_standby_functions.out (r1.1)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/test/regress/expected/hs_standby_functions.out?rev=1.1&content-type=text/x-cvsweb-markup)
pgsql/src/test/regress/sql:
hs_primary_extremes.sql (r1.1)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/test/regress/sql/hs_primary_extremes.sql?rev=1.1&content-type=text/x-cvsweb-markup)
hs_primary_setup.sql (r1.1)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/test/regress/sql/hs_primary_setup.sql?rev=1.1&content-type=text/x-cvsweb-markup)
hs_standby_allowed.sql (r1.1)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/test/regress/sql/hs_standby_allowed.sql?rev=1.1&content-type=text/x-cvsweb-markup)
hs_standby_check.sql (r1.1)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/test/regress/sql/hs_standby_check.sql?rev=1.1&content-type=text/x-cvsweb-markup)
hs_standby_disallowed.sql (r1.1)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/test/regress/sql/hs_standby_disallowed.sql?rev=1.1&content-type=text/x-cvsweb-markup)
hs_standby_functions.sql (r1.1)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/test/regress/sql/hs_standby_functions.sql?rev=1.1&content-type=text/x-cvsweb-markup)

Congratulations! And, may I be the first to say - woo hoo!

...Robert

Dimitri Fontaine

dfontaine@hi-media.com

about 16 years ago

In reply to: Robert Haas (#2)

Re: Re: [COMMITTERS] pgsql: Allow read only connections during recovery, known as Hot

Le 19 déc. 2009 à 03:01, Robert Haas a écrit :

On Fri, Dec 18, 2009 at 8:32 PM, Simon Riggs <sriggs@postgresql.org> wrote:

Log Message:
-----------
Allow read only connections during recovery, known as Hot Standby.

Congratulations! And, may I be the first to say - woo hoo!

+1!

--
dim

Jaime Casanova

jcasanov@systemguards.com.ec

about 16 years ago

In reply to: Dimitri Fontaine (#3)

Re: Re: [COMMITTERS] pgsql: Allow read only connections during recovery, known as Hot

On Sat, Dec 19, 2009 at 8:44 AM, Dimitri Fontaine
<dfontaine@hi-media.com> wrote:

Le 19 déc. 2009 à 03:01, Robert Haas a écrit :

On Fri, Dec 18, 2009 at 8:32 PM, Simon Riggs <sriggs@postgresql.org> wrote:

Log Message:
-----------
Allow read only connections during recovery, known as Hot Standby.

Congratulations! And, may I be the first to say - woo hoo!

+1!

+1
i will start to play with HS in windows

--
Atentamente,
Jaime Casanova
Soporte y capacitación de PostgreSQL
Asesoría y desarrollo de sistemas
Guayaquil - Ecuador
Cel. +59387171157