DBmirror replication - replacement for DBMirror.pl
Not sure whether this is any use to anyone, or whether this is the right
list to post to but...
I've just released a C++ implementation of the DBMirror.pl script as
part of Whitebeam (http://www.whitebeam.org). We had *real* performance
issues with the Perl implementation replicating large fields. It's an
almost drop in replacement - except configuration file format is
difffernt. It's also more fault tolerant than the Perl implementation
and has some parameters to help distribute the replication load over
time (if you delete 10,000 rows from a table, you don't really want
you're main application to grind to a halt as replication soak up most
of your resource!)
I needed to do this quickly - so it links to the utility classes in
Whitebeam, things like a string class etc. It wouldn't be too difficult
to decouple these and if there is any interest I'll do that when I get a
little spare time. Once it's built though it's entirely generic and
doesn't use anything else from Whitebeam.
If anyone has contact information for the original DBMirror author then
I'd like to get in touch.
Would like to get feedback from anyone using DBmirror or thinks this
would be useful..
Background
========
Our Whitebeam application server uses PostgreSQL for back-end data
storage. We originally used IBM DB2, but ported to Postgres about 3
years ago, but we never sorted out replication (partly because we were
using Large Objects). I recently sorted that out and looked around for a
replication scheme to use - first choice being Slony-I. I couldn't get
it to work - the configuration scheme just kept claiming half our schema
tables didn't have primary keys (they did!). Also the documentation was
virtually non-existent.
Dropping back to DBmirror (which comes with the PostgreSQL distribution
in the 'contrib' directory) worked fine-ish. Unfortunately the
replication Perl script was un-usably slow, taking minutes to replicate
a 100K BYTEA field, which we used to store images. The replication Perl
script seemed to be rather inefficient, using a lot of regular
expressions to decode field values etc. Perl isn't something I felt too
confident in - and I needed a solution quickly and hence the C++
implementation
Pete
--
Peter Wilson
YellowHawk : http://www.yellowhawk.co.uk
Whitebeam : http:/www.whitebeam.org
-----
Just to add - the replacement for DBMirror.pl automatically works out
the schema in use and will replicate using DBmirror from 7.x or from 8.x
(table names all changed!). This is done transparently.
I also wrote a short article at
http://www.whitebeam.org/library/guide/TechNotes/replicate.rhtm
Pete
Peter Wilson wrote:
Not sure whether this is any use to anyone, or whether this is the
right list to post to but...I've just released a C++ implementation of the DBMirror.pl script as
part of Whitebeam (http://www.whitebeam.org). We had *real*
performance issues with the Perl implementation replicating large
fields. It's an almost drop in replacement - except configuration file
format is difffernt. It's also more fault tolerant than the Perl
implementation and has some parameters to help distribute the
replication load over time (if you delete 10,000 rows from a table,
you don't really want you're main application to grind to a halt as
replication soak up most of your resource!)I needed to do this quickly - so it links to the utility classes in
Whitebeam, things like a string class etc. It wouldn't be too
difficult to decouple these and if there is any interest I'll do that
when I get a little spare time. Once it's built though it's entirely
generic and doesn't use anything else from Whitebeam.If anyone has contact information for the original DBMirror author
then I'd like to get in touch.Would like to get feedback from anyone using DBmirror or thinks this
would be useful..Background
========Our Whitebeam application server uses PostgreSQL for back-end data
storage. We originally used IBM DB2, but ported to Postgres about 3
years ago, but we never sorted out replication (partly because we were
using Large Objects). I recently sorted that out and looked around for
a replication scheme to use - first choice being Slony-I. I couldn't
get it to work - the configuration scheme just kept claiming half our
schema tables didn't have primary keys (they did!). Also the
documentation was virtually non-existent.Dropping back to DBmirror (which comes with the PostgreSQL
distribution in the 'contrib' directory) worked fine-ish.
Unfortunately the replication Perl script was un-usably slow, taking
minutes to replicate a 100K BYTEA field, which we used to store
images. The replication Perl script seemed to be rather inefficient,
using a lot of regular expressions to decode field values etc. Perl
isn't something I felt too confident in - and I needed a solution
quickly and hence the C++ implementationPete
--
Peter Wilson
YellowHawk : http://www.yellowhawk.co.uk
Whitebeam : http:/www.whitebeam.org
--------------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend
--
Peter Wilson
T: 01707 891840
M: 07796 656566
http://www.yellowhawk.co.uk
<http://www.yellowhawk.co.uk>
------------------------------------------------------------------------