automatic system info tool?

Andrew,

Someone at the conference mentioned a tool that would portably and
reliably report system info such as architecture. If someone has details
I would like to have them, as it would help solve the buildfarm
personality problem properly.

There's potentially two, actually. A SF start-up is going to be
open-sourcing a tool, soon, and Jim Nasby said that Distributed.net has code
for this.

--
Josh Berkus
PostgreSQL @ Sun
San Francisco

Andrew Dunstan wrote:

Someone at the conference mentioned a tool that would portably and
reliably report system info such as architecture.

What's wrong with config.guess?

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut wrote:

Andrew Dunstan wrote:

Someone at the conference mentioned a tool that would portably and
reliably report system info such as architecture.

What's wrong with config.guess?

That will probably be OK for architecture.

We also classify buildfarm machines by <os, os_version, compiler,
compiler_version> and config.guess doesn't give us that, unfortunately.

However, I will look at canonicalising the architecture to the
config.guess values reported.

cheers

andrew

-----Original Message-----
From: "Andrew Dunstan" <andrew@dunslane.net>
To: "Peter Eisentraut" <peter_e@gmx.net>
Cc: "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Sent: 16/07/06 23:50
Subject: Re: [HACKERS] automatic system info tool?

We also classify buildfarm machines by <os, os_version, compiler,
compiler_version> and config.guess doesn't give us that, unfortunately.

It also won't work on Win32 without msys (or SFU/Interix).

/D

On Sun, Jul 16, 2006 at 06:49:26PM -0400, Andrew Dunstan wrote:

We also classify buildfarm machines by <os, os_version, compiler,
compiler_version> and config.guess doesn't give us that, unfortunately.

It would seem to be a lot easier to use the values from perl itself,
given you're already using it:

# perl -MConfig -e 'print "os=$Config{osname}, osvers=$Config{osvers}, archname=$Config{archname}\n"'
os=linux, osvers=2.6.15.4, archname=i486-linux-gnu-thread-multi

If you look at perl -V it give you a subset of useful values, probably
a lot nicer than munging config.guess. It'll even tell you the size of
of the C datatypes if you're interested :)

Have a nice day,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/

I'm fairly familiar with it :-)

The trouble is that it gives info set at the time perl was compiled,
which doesn't help with the problem where a machine has been upgraded.
For example, on this FC3 machine it reports a different kernel version
from the one I have upgraded to, not surprisingly.

So what I need if possible is a runtime tool to detect the info we need.

cheers

andrew

Martijn van Oosterhout wrote:

On Mon, Jul 17, 2006 at 09:06:34AM -0400, Andrew Dunstan wrote:

I'm fairly familiar with it :-)

The trouble is that it gives info set at the time perl was compiled,
which doesn't help with the problem where a machine has been upgraded.
For example, on this FC3 machine it reports a different kernel version
from the one I have upgraded to, not surprisingly.

Hmm. The osname and archname might still be useful, but the rest could
be out of date.

So what I need if possible is a runtime tool to detect the info we need.

On UNIX systems uname may work pretty well. But I guess each system may
have slightly different options. What'll probably happen is that you
end up with a big if() statement testing $Config{osname} wtih each case
having specific code to determine the specifics. But for that you need
information. How do you get the currently running release of windows
for example?

Have a nice day,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/

On UNIX systems uname may work pretty well. But I guess each
system may
have slightly different options. What'll probably happen is that you
end up with a big if() statement testing $Config{osname} wtih
each case
having specific code to determine the specifics. But for that you need
information. How do you get the currently running release of windows
for example?

If you can open a command shell you can get the OS version with the
'ver' command under Windows:

C:\>ver

Microsoft Windows XP [Version 5.1.2600]

C:\>

This should work on 2000 or later. (Windows 2000 is 5.0.something.)

Regards,
Paul

On Mon, Jul 17, 2006 at 11:06:50AM -0400, Bort, Paul wrote:

If you can open a command shell you can get the OS version with the
'ver' command under Windows:

C:\>ver

Microsoft Windows XP [Version 5.1.2600]

How do you do this from a program though. Under UNIX uname() is a
function call as well as a program. It returns the os name, version,
hostname and system type.

Mind you, maybe perl provides emulation for uname?

Have a nice day,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/

On Jul 17, 2006, at 12:57 PM, Martijn van Oosterhout wrote:

On Mon, Jul 17, 2006 at 11:06:50AM -0400, Bort, Paul wrote:

If you can open a command shell you can get the OS version with the
'ver' command under Windows:

C:\>ver

Microsoft Windows XP [Version 5.1.2600]

How do you do this from a program though. Under UNIX uname() is a
function call as well as a program. It returns the os name, version,
hostname and system type.

GetVersionEx() will get you the windows version, service pack, etc IIRC.

Cheers,
Steve

How do you do this from a program though. Under UNIX uname() is a
function call as well as a program. It returns the os name, version,
hostname and system type.

Multiple methods (TIMTOWTDI) depending on what you want:

my $verstring = `cmd.exe /c ver`;

# or

use Win32;
my ($string, $major, $minor, $build, $id ) = Win32::GetOSVersion;

The environment variables PROCESSOR_ARCHITECTURE and
PROCESSOR_IDENTIFIER should provide the basic hardware information.

Mind you, maybe perl provides emulation for uname?

Not that I know of.

Hi,

I have posted a patch to the CVS head for on-disk bitmap index to
pgsql-patches. If this can get in 8.2, that would be great. Any comments and
suggestions are welcome.

I still need to add several items:

(1) README file in src/backend/access/bitmap.
(2) Bitmap index documentation.
(3) Hiding the internal btree.

Also, I have disabled the multi-column index support because there is a
known problem. Assume that there is a bitmap index on a and b. When a query
predicate has only a, the current code may generate a wrong result. That's
because the current code assumes that b is null. The ultimate problem is
because the search code only handles one bitmap vector now. I need a fix to
support manipulating multiple bitmap vectors.

If you find any other problems, please let me know.

Thanks,
Jie

On 7/18/06, Bort, Paul <pbort@tmwsystems.com> wrote:

Mind you, maybe perl provides emulation for uname?

Not that I know of.

Wouldn't $^0 and $Config{archname} cover quite a few, though?

If you can open a command shell you can get the OS version with the

'ver' command under Windows:

C:\>ver

Microsoft Windows XP [Version 5.1.2600]

How do you do this from a program though. Under UNIX uname() is a
function call as well as a program. It returns the os name, version,

hostname and system type.

GetVersionEx() will get you the windows version, service
pack, etc IIRC.

in perl:

use POSIX;
print join(',',POSIX::uname()),"\n";

prints:
Windows NT,hostname.domain.com,5.0,Build 2195 (Service Pack 4),x86

Works on all Platforms.

(more detail on Win with: use Win32; join(' ', Win32::GetOSVersion()),
"\n";)

Andreas

Andrej Ricnik-Bay wrote:

On 7/18/06, Bort, Paul <pbort@tmwsystems.com> wrote:

Mind you, maybe perl provides emulation for uname?

Not that I know of.

Wouldn't $^0 and $Config{archname} cover quite a few, though?

No. As previously explained, these values reflect what was true when and
where perl was compiled. In general, I need what is true at the time
buildfarm is run.

Anyway, I think we have some good possibilities.

cheers

andrew

On Mon, 17 Jul 2006, Jie Zhang wrote:

Hi,

I have posted a patch to the CVS head for on-disk bitmap index to
pgsql-patches. If this can get in 8.2, that would be great. Any comments and
suggestions are welcome.

I still need to add several items:

(1) README file in src/backend/access/bitmap.
(2) Bitmap index documentation.
(3) Hiding the internal btree.

Also, I have disabled the multi-column index support because there is a
known problem. Assume that there is a bitmap index on a and b. When a query
predicate has only a, the current code may generate a wrong result. That's
because the current code assumes that b is null. The ultimate problem is
because the search code only handles one bitmap vector now. I need a fix to
support manipulating multiple bitmap vectors.

If you find any other problems, please let me know.

Is anyone else looking at this patch? I am reviewing it with Jie, tidying
it up and trying to solve some of the problems mentioned above. If anyone
wants to take a look at us, please ask for an updated patch.

Thanks,

Gavin

Gavin Sherry <swm@linuxworld.com.au> writes:

Is anyone else looking at this patch?

It's on my list of things to look at, but so are a lot of other patches ;-)

A couple of comments after a *very* fast scan through the patch:

* The xlog routines need help; they seem to not be updated for recent
changes in the API for xlog recovery code.

* The hacks on vacuum.c (where it tests the AM name) are utterly
unacceptable. If you need the main code to do something special for a
particular index AM, define a bool flag column for it in pg_am.

* The interface to the existing executor bitmap scan code is mighty
messy --- seems like there's a lot of almost duplicate code, a lot
of rather invasive hacking, etc. This needs to be rethought and
refactored.

* Why in the world is it introducing duplicate copies of a lot
of btree comparison functions? Use the ones that are there.

* The patch itself is a mess: it introduces .orig and .rej files,
changes around $PostgreSQL$ lines, etc.

However, the main problem I've got with this is that a new index AM is a
pretty large burden, and no one's made the slightest effort to sell
pghackers on taking this on. What's the use-case, what's the
performance like, where is it really an improvement over what we've got?

regards, tom lane

On Sun, 23 Jul 2006, Tom Lane wrote:

Gavin Sherry <swm@linuxworld.com.au> writes:

Is anyone else looking at this patch?

It's on my list of things to look at, but so are a lot of other patches ;-)

A couple of comments after a *very* fast scan through the patch:

* The xlog routines need help; they seem to not be updated for recent
changes in the API for xlog recovery code.

Yep. The patch was actually against 8.1 and was hastily brought up to 8.2.
I think Jie's intention was to simply let everyone know that this was
going on.

* The hacks on vacuum.c (where it tests the AM name) are utterly
unacceptable. If you need the main code to do something special for a
particular index AM, define a bool flag column for it in pg_am.

Yes.

* The interface to the existing executor bitmap scan code is mighty
messy --- seems like there's a lot of almost duplicate code, a lot
of rather invasive hacking, etc. This needs to be rethought and
refactored.

I agree.

* Why in the world is it introducing duplicate copies of a lot
of btree comparison functions? Use the ones that are there.

Yes, I raised this with Jie and she has fixed it. One thought is, we may
want to rename those comparison functions prefixed with 'bm' to make their
naming less confusing. They'll be used by btree, gin and bitmap index
methods. Anyway, a seperate patch.

* The patch itself is a mess: it introduces .orig and .rej files,
changes around $PostgreSQL$ lines, etc.

Right, not to mention patches to configure and a lot of style which needs
to be knocked into shape.

However, the main problem I've got with this is that a new index AM is a
pretty large burden, and no one's made the slightest effort to sell
pghackers on taking this on. What's the use-case, what's the
performance like, where is it really an improvement over what we've got?

For low cardinality sets, bitmaps greatly out perform btree. Jie and
others at Greenplum tested this implementation (and others) against very
large, low cardinality sets. I wasn't involved in it. I urge Jie and/or
Luke to present those results.

Thanks,

Gavin

Gavin Sherry <swm@linuxworld.com.au> writes:

On Sun, 23 Jul 2006, Tom Lane wrote:

However, the main problem I've got with this is that a new index AM is a
pretty large burden, and no one's made the slightest effort to sell
pghackers on taking this on.

For low cardinality sets, bitmaps greatly out perform btree.

If the column is sufficiently low cardinality, you might as well just do
a seqscan --- you'll be hitting most of the heap's pages anyway. I'm
still waiting to be convinced that there's a sweet spot wide enough to
justify supporting another index AM. (I'm also wondering whether this
doesn't overlap the use-case for GIN.)

regards, tom lane