Page layout footprint
Hi Heikki,
I'm sorry for lack of explanation. It is my fault.
Heikki says (on commit fest wiki):
------------
I believe I debunked this patch enough already. Apparently there's some
compatibility issue between 32-bit and 64-bit Sparcs, but this patch
didn't catch that. It doesn't seem like this provides any extra safeness
or better error messages. If I'm missing something, please provide more
details on what scenario we currently have a problem, and how this helps
with it.
------------
The original proposal
(http://archives.postgresql.org/message-id/489FC8E1.9090307@sun.com)
contains two parts. First part is implementation of --footprint cmd line
switch which shows you page layout structures footprint. It is useful
for development (mostly for in-place upgrade) and also for manual data
recovery when you need to know exact structures. Second part was add
this information also into pg_control.file, but how you correctly
mentioned there is not real use case to do it at this moment.
However, there is still --footprint switch which is useful and it is
reason why I put it on wiki for review and feedback. The switch could
also use in build farm for collecting footprints from build farm members.
32/64 bit issue is little bit different story and it is general (not
only SPARC but on SPARC has bigger impact). Problem is that CRC32 gives
probably different result when it is compiled 32bit or 64bit. I'm going
to examine it more.
Zdenek
Zdenek Kotala wrote:
Hi Heikki,
I'm sorry for lack of explanation. It is my fault.
Heikki says (on commit fest wiki):
------------
I believe I debunked this patch enough already. Apparently there's some
compatibility issue between 32-bit and 64-bit Sparcs, but this patch
didn't catch that. It doesn't seem like this provides any extra safeness
or better error messages. If I'm missing something, please provide more
details on what scenario we currently have a problem, and how this helps
with it.
------------The original proposal
(http://archives.postgresql.org/message-id/489FC8E1.9090307@sun.com)
contains two parts. First part is implementation of --footprint cmd line
switch which shows you page layout structures footprint. It is useful
for development (mostly for in-place upgrade) and also for manual data
recovery when you need to know exact structures. Second part was add
this information also into pg_control.file, but how you correctly
mentioned there is not real use case to do it at this moment.However, there is still --footprint switch which is useful and it is
reason why I put it on wiki for review and feedback. The switch could
also use in build farm for collecting footprints from build farm members.
Ok, understood. I'll take another look from that point of view.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
Zdenek Kotala napsal(a):
32/64 bit issue is little bit different story and it is general (not
only SPARC but on SPARC has bigger impact). Problem is that CRC32 gives
probably different result when it is compiled 32bit or 64bit. I'm going
to examine it more.
I'm sorry about noise. Everything works as expected. I tested 8.3 and
looked into 8.4 code :(. The problem with 8.3 is that control file uses
time_t which is defined as long and it is not portable between 32bits
and 64bits server version. Version 8.4 is OK, because it uses pg_time_t
which is alway 64bit long.
Zdenek
Zdenek Kotala wrote:
The original proposal
(http://archives.postgresql.org/message-id/489FC8E1.9090307@sun.com)
contains two parts. First part is implementation of --footprint cmd line
switch which shows you page layout structures footprint. It is useful
for development (mostly for in-place upgrade) and also for manual data
recovery when you need to know exact structures.
I'm afraid I still fail to see the usefulness of this. gdb knows how to
deal with structs, and for manual data recovery you need so much more
than the page header structure. And if you're working at such a low
level, it's not that hard to calculate the offsets within the struct
manually.
BTW, this makes me wonder if it would be possible to use the
upgrade-in-place machinery to convert a data directory from one
architecture to another? Just a thought..
However, there is still --footprint switch which is useful and it is
reason why I put it on wiki for review and feedback. The switch could
also use in build farm for collecting footprints from build farm members.
If we needed more information about the architectures, we could just
collect the output of pg_controldata. But I think the configure logs
already contains all the useful information.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
Heikki Linnakangas napsal(a):
Zdenek Kotala wrote:
The original proposal
(http://archives.postgresql.org/message-id/489FC8E1.9090307@sun.com)
contains two parts. First part is implementation of --footprint cmd
line switch which shows you page layout structures footprint. It is
useful for development (mostly for in-place upgrade) and also for
manual data recovery when you need to know exact structures.I'm afraid I still fail to see the usefulness of this. gdb knows how to
deal with structs,
If I correct that GDB knows structure only if you have debug version. But
usually you don't have debug version on production system. And another small
advantage is that --footprint switch is easy to use. It is easier that work with
gdb and you can easy get info from users who are not familiar with gdb.
and for manual data recovery you need so much more
than the page header structure.
Yeah, I know, but I didn't want to spend several days with full coding without
idea approval. There are special data, meta pages and so on which have to be added.
And if you're working at such a low
level, it's not that hard to calculate the offsets within the struct
manually.
I'm not sure if it is so easy. Are you able do it for SPARC, PPC or other non
x86 CPUs?
BTW, this makes me wonder if it would be possible to use the
upgrade-in-place machinery to convert a data directory from one
architecture to another? Just a thought..
Hmm, good question. For example ZFS is platform independent, you can take disk
from SPARC machine and plug it into x86 and ZFS works perfectly. ZFS converts
its own data during a read and any new block is written in a new format. You are
able read all binary platform independent data like MP3, JPEG and so on. But
PostgreSQL will not work, because PostgreSQL fails during pg_control file
reading, because endianes are different.
Convert data structures like PageHeader and so on could be possible but you
don't have control over user data types.
I think in this case is better to develop platform independent replication and
use this mechanism for data transfer.
However, there is still --footprint switch which is useful and it is
reason why I put it on wiki for review and feedback. The switch could
also use in build farm for collecting footprints from build farm members.If we needed more information about the architectures, we could just
collect the output of pg_controldata. But I think the configure logs
already contains all the useful information.
It seems to be good idea. Only what we need is extend buildfarm to parse
config.log and shows this data for each build machine. It could also report
changes in alignment.
Zdenek
--
Zdenek Kotala Sun Microsystems
Prague, Czech Republic http://sun.com/postgresql
Zdenek Kotala <Zdenek.Kotala@Sun.COM> writes:
Hmm, good question. For example ZFS is platform independent, you can take disk
from SPARC machine and plug it into x86 and ZFS works perfectly.
FWIW as far as I know *all* filesystems are platform independent. (Of course
now someone is surely going to find some counter-example) Doesn't really
change the argument though.
--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's Slony Replication support!
Gregory Stark napsal(a):
Zdenek Kotala <Zdenek.Kotala@Sun.COM> writes:
Hmm, good question. For example ZFS is platform independent, you can take disk
from SPARC machine and plug it into x86 and ZFS works perfectly.FWIW as far as I know *all* filesystems are platform independent. (Of course
now someone is surely going to find some counter-example) Doesn't really
change the argument though.
Yeah, of course. I selected bad word. ZFS write data structures in native endian
format. It does not need convert data to correct byte order. Only if you switch
harddisk from SPARC to x86 you need convert bytes order. Other systems has a
penalty for endian conversion on oposit platform or conversion is not supported.
However, I'm not sure if ext3 filesystem which is created on x86 machine is
readable under linux on SPARC machine? FAT32 works fine :-) everywhere.
Maybe optimized platform independence is better term.
Zdenek
--
Zdenek Kotala Sun Microsystems
Prague, Czech Republic http://sun.com/postgresql
Zdenek Kotala wrote:
Heikki Linnakangas napsal(a):
I'm afraid I still fail to see the usefulness of this. gdb knows how
to deal with structs,If I correct that GDB knows structure only if you have debug version.
But usually you don't have debug version on production system.
Using gdb without debug systems is pretty much a lost cause anyway.
And
another small advantage is that --footprint switch is easy to use. It is
easier that work with gdb and you can easy get info from users who are
not familiar with gdb.
AFAICS you can get all the same information from pg_controldata. We have
a pretty good idea of the alignments of all the usual platforms anyway.
If someone says in a bug report that they're running on x86_64 or 32-bit
Sparc, we know what the alignments on those platforms are.
And if you're working at such a low level, it's not that hard to
calculate the offsets within the struct manually.I'm not sure if it is so easy. Are you able do it for SPARC, PPC or
other non x86 CPUs?
Not off the top of my head. But I am able to do it on x86 and x86_64
which are the platforms I work and debug on.
If we needed more information about the architectures, we could just
collect the output of pg_controldata. But I think the configure logs
already contains all the useful information.It seems to be good idea. Only what we need is extend buildfarm to parse
config.log and shows this data for each build machine.
Well, the information is already there. I'm not convinced it's such an
important issue that it's worth the effort to add special handling to
extract that information from the log. Of course, if someone feels
otherwise and does it, I won't object.
It could also report changes in alignment.
Huh? I would be pretty surprised if the alignment changed randomly on
some platform.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
Heikki Linnakangas napsal(a):
Zdenek Kotala wrote:
Heikki Linnakangas napsal(a):
<snip>
AFAICS you can get all the same information from pg_controldata. We have
a pretty good idea of the alignments of all the usual platforms anyway.
If someone says in a bug report that they're running on x86_64 or 32-bit
Sparc, we know what the alignments on those platforms are.And if you're working at such a low level, it's not that hard to
calculate the offsets within the struct manually.I'm not sure if it is so easy. Are you able do it for SPARC, PPC or
other non x86 CPUs?Not off the top of my head. But I am able to do it on x86 and x86_64
which are the platforms I work and debug on.
OK. You convinced me that information could be collected from other sources.
<snip>
It could also report changes in alignment.
Huh? I would be pretty surprised if the alignment changed randomly on
some platform.
I thought if somebody change compiler switches - for example 32/64 compilation.
But it is probably rare case.
Zdenek
--
Zdenek Kotala Sun Microsystems
Prague, Czech Republic http://sun.com/postgresql
Zdenek Kotala wrote:
OK. You convinced me that information could be collected from other
sources.
Great, I'm glad we're in agreement.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com