Latest version of Hot Standby patch

On Wed, 2008-12-17 at 15:21 +0000, Simon Riggs wrote:

http://wiki.postgresql.org/wiki/Hot_Standby

now contains a link to latest version of this patch. This is still at
"v5", just brought forward to CVS HEAD.

I will be doing further work on the patch from here and will reply to
this post each time I submit a new version.

New version (already!) v5 - this time slightly broken down to aid review

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Can't we use the existing backendid in place of the slot id?

(sorry if this has been discussed already; can't remember)

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

On Thu, 2008-12-18 at 15:13 +0200, Heikki Linnakangas wrote:

Can't we use the existing backendid in place of the slot id?

(sorry if this has been discussed already; can't remember)

Exactly the sort of question we need, but unfortunately I'm a little
hazy, but I just woke up some maybe some coffee will change that answer
later.

They're certainly related and I did try it initially. SlotId was not
present in early versions of the patch up to 13 Oct, though backendId
was.

IIRC there were a couple of reasons

* corner case behaviour of backendids - bgwriter writes checkpoint WAL
records. Has no backendid, but needs a slotid (possibly others)

* slotids are assigned once and never changed, so allowing them to be
used as array lookups directly

I think it would be possible to use slotids as backendids, but not the
other way around. Anyway, seemed like a great way to induce bugs into
the sinval code, so I didn't try too hard to make them identical.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

On Wed, 2008-12-17 at 15:21 +0000, Simon Riggs wrote:

http://wiki.postgresql.org/wiki/Hot_Standby

now contains a link to latest version of this patch. This is still at
"v5", just brought forward to CVS HEAD.

I will be doing further work on the patch from here and will reply to
this post each time I submit a new version.

New version: patch apply fixed, doc typos corrected

First half of Wiki docs checked and updated to explain User and Admin
Overview exactly as currently implemented.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Simon Riggs wrote:

* corner case behaviour of backendids - bgwriter writes checkpoint WAL
records. Has no backendid, but needs a slotid (possibly others)

Why does bgwriter need a slotid? It doesn't run any transactions.

* slotids are assigned once and never changed, so allowing them to be
used as array lookups directly

So are backend ids.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

On Fri, 2008-12-19 at 10:59 +0200, Heikki Linnakangas wrote:

Simon Riggs wrote:

* corner case behaviour of backendids - bgwriter writes checkpoint WAL
records. Has no backendid, but needs a slotid (possibly others)

Why does bgwriter need a slotid? It doesn't run any transactions.

* slotids are assigned once and never changed, so allowing them to be
used as array lookups directly

So are backend ids.

I'm a little hazy, to be sure. I'm pretty sure there was a blocker, but
if I cannot recall it we should assume it doesn't exist.

Where are you going with the thought? Remove slotId from each proc and
then use backendId to identify the recovery proc?

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Simon Riggs wrote:

On Fri, 2008-12-19 at 10:59 +0200, Heikki Linnakangas wrote:

Simon Riggs wrote:

* corner case behaviour of backendids - bgwriter writes checkpoint WAL
records. Has no backendid, but needs a slotid (possibly others)

Why does bgwriter need a slotid? It doesn't run any transactions.

* slotids are assigned once and never changed, so allowing them to be
used as array lookups directly

So are backend ids.

I'm a little hazy, to be sure. I'm pretty sure there was a blocker, but
if I cannot recall it we should assume it doesn't exist.

Where are you going with the thought? Remove slotId from each proc and
then use backendId to identify the recovery proc?

Yep.

Well, to be honest, I don't much like the whole notion of tracking the
slots. I think we should just rely on the XLOG_RECOVERY_END records to
purge stale PGPROC entries, belonging to backends that died without
writing an abort record.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Heikki Linnakangas wrote:

Well, to be honest, I don't much like the whole notion of tracking the
slots. I think we should just rely on the XLOG_RECOVERY_END records to
purge stale PGPROC entries, belonging to backends that died without
writing an abort record.

Sorry, I meant XLOG_XACT_RUNNING_XACTS, not XLOG_RECOVERY_END.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

On Fri, 2008-12-19 at 14:00 +0200, Heikki Linnakangas wrote:

Heikki Linnakangas wrote:

Well, to be honest, I don't much like the whole notion of tracking the
slots. I think we should just rely on the XLOG_RECOVERY_END records to
purge stale PGPROC entries, belonging to backends that died without
writing an abort record.

Sorry, I meant XLOG_XACT_RUNNING_XACTS, not XLOG_RECOVERY_END.

OK, previous response aborted, but the problem is fairly similar.

If we just assign a recovery proc to each new transaction started then

* we would need an expandable number of recovery procs to cope with the
fact that we have more potentially emulated transactions than procs.
This info needs to be in shared memory, so however many you allocate, it
can still run out. Then what do you do? PANIC recovery and permanently
fail? Kick off all queries until a XLOG_XACT_RUNNING_XACTS arrives?

* there would be no direct mapping between the commit record and the
proc, so each commit would need to scan the whole proc array to get the
correct proc (which is potentially getting bigger and bigger because of
the first point)

The slotid (or backendid) is essential to keeping the number of emulated
transactions bounded at all times. I don't see that it costs much, if
anything and the resulting code is at least as simple as the
alternatives.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

On Wed, 2008-12-17 at 15:21 +0000, Simon Riggs wrote:

http://wiki.postgresql.org/wiki/Hot_Standby

now contains a link to latest version of this patch.

v6 of Hot Standby now uploaded to Wiki (link above), with these changes:

* Must ignore_killed_tuples and never kill_prior_tuple during index
scans in recovery (v6)
* XLOG_BTREE_DELETE records handled correctly (v6)
* btree VACUUM code - must scan every block of index (v6)
* BEGIN TRANSACTION READ WRITE should throw error (v6)

New test cycle starting with this patch over next few days.

Work continues on other items.

Happy New Year everyone,

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Simon Riggs a écrit :

On Wed, 2008-12-17 at 15:21 +0000, Simon Riggs wrote:

http://wiki.postgresql.org/wiki/Hot_Standby

now contains a link to latest version of this patch.

v6 of Hot Standby now uploaded to Wiki (link above), with these changes:

* Must ignore_killed_tuples and never kill_prior_tuple during index
scans in recovery (v6)
* XLOG_BTREE_DELETE records handled correctly (v6)
* btree VACUUM code - must scan every block of index (v6)
* BEGIN TRANSACTION READ WRITE should throw error (v6)

New test cycle starting with this patch over next few days.

I use latest CVS version. I tried to apply the patches and I have the
following error :

./hs.apply.sh: line 4: hs.prevent.v5.patch

I think you forgot to update the script. hs.prevent.v5.patch doesn't
exist in the tar file but hs.prevent.v6.patch does. Not sure we really
need this new file because I have a compilation error if I use the new
one. I don't have this error when I don't use the hs.prevent.v6.patch
file. Compilation error is :

utility.c: In function ‘ProcessUtility’:

utility.c:292: erreur: expected ‘)’ before ‘{’ token

utility.c:306: erreur: expected expression before ‘}’ token

And one file didn't want to get patched :

patching file src/include/catalog/pg_proc.h
Hunk #1 FAILED at 3223.
1 out of 1 hunk FAILED -- saving rejects to file
src/include/catalog/pg_proc.h.rej

Not sure why. I did patch it manually, but I still have my compilation
error.

Regards.

--
Guillaume.
http://www.postgresqlfr.org
http://dalibo.com

On Fri, 2009-01-02 at 11:02 +0100, Guillaume Lelarge wrote:

Simon Riggs a écrit :

On Wed, 2008-12-17 at 15:21 +0000, Simon Riggs wrote:

http://wiki.postgresql.org/wiki/Hot_Standby

now contains a link to latest version of this patch.

v6 of Hot Standby now uploaded to Wiki (link above), with these changes:

* Must ignore_killed_tuples and never kill_prior_tuple during index
scans in recovery (v6)
* XLOG_BTREE_DELETE records handled correctly (v6)
* btree VACUUM code - must scan every block of index (v6)
* BEGIN TRANSACTION READ WRITE should throw error (v6)

New test cycle starting with this patch over next few days.

I use latest CVS version. I tried to apply the patches and I have the
following error :

Thanks, will fix.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

On Fri, 2009-01-02 at 17:35 +0000, Simon Riggs wrote:

I use latest CVS version. I tried to apply the patches and I have the
following error :

Thanks, will fix.

Fixed various bit rots and re-packaged. v6a now up, v6 unlinked.

Thanks,

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Simon Riggs a �crit :

On Fri, 2009-01-02 at 17:35 +0000, Simon Riggs wrote:

I use latest CVS version. I tried to apply the patches and I have the
following error :

Thanks, will fix.

Fixed various bit rots and re-packaged. v6a now up, v6 unlinked.

Thanks. I only did a few checks and it worked great for me. I will try
to put some more time on it.

--
Guillaume.
http://www.postgresqlfr.org
http://dalibo.com

Simon Riggs wrote:

On Wed, 2008-12-17 at 15:21 +0000, Simon Riggs wrote:

http://wiki.postgresql.org/wiki/Hot_Standby

now contains a link to latest version of this patch.

v6 of Hot Standby now uploaded to Wiki (link above), with these changes:

* Must ignore_killed_tuples and never kill_prior_tuple during index
scans in recovery (v6)
* XLOG_BTREE_DELETE records handled correctly (v6)
* btree VACUUM code - must scan every block of index (v6)
* BEGIN TRANSACTION READ WRITE should throw error (v6)

New test cycle starting with this patch over next few days.

Work continues on other items.

Happy New Year everyone,

I'm running some tests on v6a. The setup is:

- install master, setup standby as usual, start standby
- create database bench on master
- initialize pgbench dataset size 100 on master
- start 4 clients doing 500000 transactions each.

After about 100000 transactions have been processed on the master, query
the standby:

bench=# \d history
Table "public.history"
Column | Type | Modifiers
--------+-----------------------------+-----------
tid | integer |
bid | integer |
aid | integer |
delta | integer |
mtime | timestamp without time zone |
filler | character(22) |

bench=# select now(),count(*) from history;
ERROR: could not open relation base/16384/16394: No such file or directory

regards

Mark

On Sun, 2009-01-04 at 21:03 +1300, Mark Kirkwood wrote:

bench=# select now(),count(*) from history;
ERROR: could not open relation base/16384/16394: No such file or
directory

Thanks for the report.

I'm attempting to recreate now.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

On Sun, 2009-01-04 at 21:03 +1300, Mark Kirkwood wrote:

bench=# \d history
Table "public.history"
Column | Type | Modifiers
--------+-----------------------------+-----------
tid | integer |
bid | integer |
aid | integer |
delta | integer |
mtime | timestamp without time zone |
filler | character(22) |

bench=# select now(),count(*) from history;
ERROR: could not open relation base/16384/16394: No such file or
directory

From my recreating your test case, the oids are consistent with the

History table. So the cache looks good.

md.c should be cacheing the file descriptor so the second use of the
file should not be reopening it. I've not touched smgr/md so a missing
file error is a surprise.

I wonder if this is an error associated with large file handling and
file forks? Smells like an FSM or VM error.

Is the file actually missing? i.e. ls -l mydatadir/base/16384/16394*

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Simon Riggs wrote:

On Sun, 2009-01-04 at 21:03 +1300, Mark Kirkwood wrote:

bench=# \d history
Table "public.history"
Column | Type | Modifiers
--------+-----------------------------+-----------
tid | integer |
bid | integer |
aid | integer |
delta | integer |
mtime | timestamp without time zone |
filler | character(22) |

bench=# select now(),count(*) from history;
ERROR: could not open relation base/16384/16394: No such file or
directory

From my recreating your test case, the oids are consistent with the

History table. So the cache looks good.

md.c should be cacheing the file descriptor so the second use of the
file should not be reopening it. I've not touched smgr/md so a missing
file error is a surprise.

I wonder if this is an error associated with large file handling and
file forks? Smells like an FSM or VM error.

Is the file actually missing? i.e. ls -l mydatadir/base/16384/16394*

Yeah -
$ ls -l $PGDATA/base/16384/16394*
ls: /data0/pgslave/8.4/base/16384/16394*: No such file or directory

regards

Mark

Mark Kirkwood wrote:

Simon Riggs wrote:

On Sun, 2009-01-04 at 21:03 +1300, Mark Kirkwood wrote:

bench=# \d history
Table "public.history"
Column | Type | Modifiers
--------+-----------------------------+-----------
tid | integer |
bid | integer |
aid | integer |
delta | integer |
mtime | timestamp without time zone |
filler | character(22) |

bench=# select now(),count(*) from history;
ERROR: could not open relation base/16384/16394: No such file or
directory

From my recreating your test case, the oids are consistent with the

History table. So the cache looks good.

md.c should be cacheing the file descriptor so the second use of the
file should not be reopening it. I've not touched smgr/md so a missing
file error is a surprise.

I wonder if this is an error associated with large file handling and
file forks? Smells like an FSM or VM error.

Is the file actually missing? i.e. ls -l mydatadir/base/16384/16394*

Yeah -
$ ls -l $PGDATA/base/16384/16394*
ls: /data0/pgslave/8.4/base/16384/16394*: No such file or directory

This might be useful:

the other tables in the dataset (accounts, branches, tellers) all
behave as expected:

bench=# select now(),count(*) from branches;
now | count
-------------------------------+-------
2009-01-04 22:17:00.298597+13 | 100
(1 row)

I'm guessing something tied up with the fact that history has no rows to
start with...