xlogdump

Gregory Stark wrote:

There's an xlogdump project on pgfoundry. However it suffers from perennial
bitrot as it has to maintain its own table of xlog record types and code to
decode each xlog record type.

...

I think this module should be rewritten to depend more closely on the Postgres
source files. What I'm doing now is making an SRF in the style of the
pageinspect module which will read an arbitrary wal file and generate records
directly.

This has a big disadvantage compared to the original approach, namely that you
need a functioning Postgres instance of the same version to dissect wal
records.

But it also has a big advantage, namely that it will always be in sync. It
will just use the same RmgrTable to find the rm_name and call the rm_desc
method to decode the record. The result might not be quite as or dense as the
rm_desc function is meant for debugging messages. We could address that
sometime with a new method if we wanted to.

Would it still be possible to compile it as a stand-alone program, using
the backend source files? It would be a hack, we just went through some
effort to clean up references to server private header files from ecpg
and initdb, but it feels a lot nicer to use as a standalone program than
requiring a running postgres instance.

How much infrastructure would you need to call rm_name and rm_desc from
a standalone program? palloc and friends, I presume, What else?

I'm thinking of actually dropping it directly into the pageinspect contrib
module. It's not quite an exact fit but it doesn't seem to deserve it's own
contrib module and it's likely to suffer the same bitrot problem if it lives
in pgfoundry.

I'd vote for pgfoundry or a new contrib module. It shouldn't suffer from
bitrot as easily as what's there now. That was the whole point of
switching over to the new approach, right?

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Heikki Linnakangas wrote:

Gregory Stark wrote:

There's an xlogdump project on pgfoundry. However it suffers from
perennial
bitrot as it has to maintain its own table of xlog record types and
code to
decode each xlog record type.

...

I think this module should be rewritten to depend more closely on the
Postgres
source files. What I'm doing now is making an SRF in the style of the
pageinspect module which will read an arbitrary wal file and generate
records
directly.

This has a big disadvantage compared to the original approach, namely
that you
need a functioning Postgres instance of the same version to dissect wal
records.

But it also has a big advantage, namely that it will always be in
sync. It
will just use the same RmgrTable to find the rm_name and call the rm_desc
method to decode the record. The result might not be quite as or dense
as the
rm_desc function is meant for debugging messages. We could address that
sometime with a new method if we wanted to.

Would it still be possible to compile it as a stand-alone program, using
the backend source files? It would be a hack, we just went through some
effort to clean up references to server private header files from ecpg
and initdb, but it feels a lot nicer to use as a standalone program than
requiring a running postgres instance.

How much infrastructure would you need to call rm_name and rm_desc from
a standalone program? palloc and friends, I presume, What else?

Do you actually need palloc and friends, or just "something named
palloc"? We already have some stuff in src/port that deals with using
palloc calls in routines used in frontend programs...

//Magnus

--On Freitag, November 02, 2007 10:54:45 +0000 Gregory Stark
<stark@enterprisedb.com> wrote:

I think this module should be rewritten to depend more closely on the
Postgres source files. What I'm doing now is making an SRF in the style
of the pageinspect module which will read an arbitrary wal file and
generate records directly.

Isn't that already partially implemented in xlogviewer, hosted on the same
project:

<http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/xlogviewer/xlogviewer/

The last time i've used it it had some problems with exhausted memory, but
it implements a SRF returning WAL records from a given log segment file.

--
Thanks

Bernd

On Fri, 2007-11-02 at 10:54 +0000, Gregory Stark wrote:

Incidentally I would like to call xlog.c:RecordIsValid() which is currently a
static function. Any objection to exporting it? It doesn't depend on any
external xlog.c state.

You'll have some fun with that because most of the stuff in xlog.c
depends upon module-level state.

We should change that, but I'd like to see Florian's code go in first
then refactor things.

If you change those programs too much we'll need completely separate
programs for each release, which might be a bad thing.

--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com

Added to TODO:

* Move pgfoundry's xlogdump to /contrib and have it rely more closely
on the WAL backend code

http://archives.postgresql.org/pgsql-hackers/2007-11/msg00035.php

---------------------------------------------------------------------------

Gregory Stark wrote:

There's an xlogdump project on pgfoundry. However it suffers from perennial
bitrot as it has to maintain its own table of xlog record types and code to
decode each xlog record type.

Earlier I modified xlogdump to generate a CSV loadable data set so I could do
some basic analysis and see what types of operations are generating the most
wal traffic. But I found it had bitrotted and needed some attention to bring
it up to date.

Again now I wanted to repeat that analysis to measure the effect HOT has had
on WAL traffic. And again now I find it has bitrotted, not least because of
HOT of course...

I think this module should be rewritten to depend more closely on the Postgres
source files. What I'm doing now is making an SRF in the style of the
pageinspect module which will read an arbitrary wal file and generate records
directly.

This has a big disadvantage compared to the original approach, namely that you
need a functioning Postgres instance of the same version to dissect wal
records.

But it also has a big advantage, namely that it will always be in sync. It
will just use the same RmgrTable to find the rm_name and call the rm_desc
method to decode the record. The result might not be quite as or dense as the
rm_desc function is meant for debugging messages. We could address that
sometime with a new method if we wanted to.

I'm thinking of actually dropping it directly into the pageinspect contrib
module. It's not quite an exact fit but it doesn't seem to deserve it's own
contrib module and it's likely to suffer the same bitrot problem if it lives
in pgfoundry.

Incidentally I would like to call xlog.c:RecordIsValid() which is currently a
static function. Any objection to exporting it? It doesn't depend on any
external xlog.c state.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's On-Demand Production Tuning

---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +