TAP test breakage on MacOS X

On 10/2/14 3:19 PM, Robert Haas wrote:

1..2
ok 1 - initdb with invalid option nonzero exit code
ok 2 - initdb with invalid option prints error message
# Looks like your test exited with 256 just after 2.
not ok 3 - initdb options handling

# Failed test 'initdb options handling'
# at /opt/local/lib/perl5/5.12.5/Test/Builder.pm line 229.

Is this repeatable? Bisectable? Has it ever worked? Have you tried a
clean build? Did you upgrade something in your operating system?

It appears to work "everywhere else".

If none of this gets us closer to an answer, I can try to produce a
patch that produces more details for such failures.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2014-10-02 17:09:43 -0400, Peter Eisentraut wrote:

On 10/2/14 3:19 PM, Robert Haas wrote:

1..2
ok 1 - initdb with invalid option nonzero exit code
ok 2 - initdb with invalid option prints error message
# Looks like your test exited with 256 just after 2.
not ok 3 - initdb options handling

# Failed test 'initdb options handling'
# at /opt/local/lib/perl5/5.12.5/Test/Builder.pm line 229.

Is this repeatable? Bisectable? Has it ever worked? Have you tried a
clean build? Did you upgrade something in your operating system?

It appears to work "everywhere else".

If none of this gets us closer to an answer, I can try to produce a
patch that produces more details for such failures.

FWIW, the current amount of details on errors is clearly insufficient.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Thu, Oct 2, 2014 at 5:09 PM, Peter Eisentraut <peter_e@gmx.net> wrote:

On 10/2/14 3:19 PM, Robert Haas wrote:

1..2
ok 1 - initdb with invalid option nonzero exit code
ok 2 - initdb with invalid option prints error message
# Looks like your test exited with 256 just after 2.
not ok 3 - initdb options handling

# Failed test 'initdb options handling'
# at /opt/local/lib/perl5/5.12.5/Test/Builder.pm line 229.

Is this repeatable? Bisectable? Has it ever worked? Have you tried a
clean build? Did you upgrade something in your operating system?

I don't think it's ever worked. I just didn't get around to reporting
it before now.

If none of this gets us closer to an answer, I can try to produce a
patch that produces more details for such failures.

A test that fails for no reason that can be gleaned from the output is
not an improvement over not having a test at all.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Robert Haas <robertmhaas@gmail.com> writes:

make check-world dies ingloriously for me, like this:

FWIW, it works fine for me on my Mac laptop, using the Perl 5.16.2 that
comes standard with OSX 10.9.5. I did have to install IPC::Run from
CPAN though.

# Failed test 'initdb options handling'
# at /opt/local/lib/perl5/5.12.5/Test/Builder.pm line 229.

This output seems to be pretty clear proof that you're not using
Apple's Perl. What is it exactly (where did you get it from)?

Also, noticing that what you're using is evidently Perl 5.12, I'm
wondering whether our TAP test scripts require a fairly new Perl version.
I recall some of my Salesforce colleagues griping that the TAP scripts
didn't work with older Perls.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

I wrote:

Also, noticing that what you're using is evidently Perl 5.12, I'm
wondering whether our TAP test scripts require a fairly new Perl version.
I recall some of my Salesforce colleagues griping that the TAP scripts
didn't work with older Perls.

Well, they definitely don't work with Perl 5.10.0: on an older Mac
box I get

/bin/sh ../../../config/install-sh -c -d tmp_check/log
make -C ../../.. DESTDIR='/Users/tgl/pgsql/src/bin/initdb'/tmp_check/install install >'/Users/tgl/pgsql/src/bin/initdb'/tmp_check/log/install.log 2>&1
cd . && TESTDIR='/Users/tgl/pgsql/src/bin/initdb' PATH="/Users/tgl/pgsql/src/bin/initdb/tmp_check/install/Users/tgl/testversion/bin:$PATH" DYLD_LIBRARY_PATH='/Users/tgl/pgsql/src/bin/initdb/tmp_check/install/Users/tgl/testversion/lib' PGPORT='65440' prove --ext='.pl' -I ../../../src/test/perl/ --verbose t/
Unknown option: ext
make: *** [check] Error 1

How badly do we need --ext? It looks to me from a quick perusal of the
prove man page that we could replace both --ext and the t/ argument with
an explicit list of test file names, which'd be easy enough to generate
with $(wildcard).

On RHEL6, which has perl 5.10.1, we seem to dodge that bullet
but nonetheless most of the tests get skipped because:

1..0 # SKIP version of Test::More is too old to support subplans
skipped: version of Test::More is too old to support subplans

So I remain of the opinion that the TAP tests need a lot of portability
work before they're going to be good for much.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Thu, Oct 2, 2014 at 10:08 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Robert Haas <robertmhaas@gmail.com> writes:

make check-world dies ingloriously for me, like this:

FWIW, it works fine for me on my Mac laptop, using the Perl 5.16.2 that
comes standard with OSX 10.9.5. I did have to install IPC::Run from
CPAN though.

# Failed test 'initdb options handling'
# at /opt/local/lib/perl5/5.12.5/Test/Builder.pm line 229.

This output seems to be pretty clear proof that you're not using
Apple's Perl. What is it exactly (where did you get it from)?

Also, noticing that what you're using is evidently Perl 5.12, I'm
wondering whether our TAP test scripts require a fairly new Perl version.
I recall some of my Salesforce colleagues griping that the TAP scripts
didn't work with older Perls.

I installed it via MacPorts.

[rhaas pgsql]$ which perl
/opt/local/bin/perl
[rhaas pgsql]$ perl -v

This is perl 5, version 12, subversion 5 (v5.12.5) built for
darwin-thread-multi-2level

Copyright 1987-2012, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using "man perl" or "perldoc perl". If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Thu, 2014-10-02 at 21:18 -0400, Robert Haas wrote:

If none of this gets us closer to an answer, I can try to produce a
patch that produces more details for such failures.

A test that fails for no reason that can be gleaned from the output is
not an improvement over not having a test at all.

I understand that this isn't great, and it's certainly something I'm
looking into. But it's like pg_regress saying that psql crashed and
leaving you to find out why. I don't think saying that the entire
regression test suite is useless because of that is fair. The TAP tests
are arguably already much easier to debug than pg_regress ever was.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Mon, Oct 6, 2014 at 8:15 PM, Peter Eisentraut <peter_e@gmx.net> wrote:

On Thu, 2014-10-02 at 21:18 -0400, Robert Haas wrote:

If none of this gets us closer to an answer, I can try to produce a
patch that produces more details for such failures.

A test that fails for no reason that can be gleaned from the output is
not an improvement over not having a test at all.

I understand that this isn't great, and it's certainly something I'm
looking into. But it's like pg_regress saying that psql crashed and
leaving you to find out why. I don't think saying that the entire
regression test suite is useless because of that is fair. The TAP tests
are arguably already much easier to debug than pg_regress ever was.

Well, maybe. I wasn't able, after about 5 minutes of searching, to
locate either a log file with details of the failure or the code that
revealed what the test, the expected result, and the actual result
were. It's possible that all that information is there and I just
don't know where to look; it took me a while to learn where the
various logs (postmaster.log, initdb.log, results) left behind by
pg_regress were, too. If that information is not there, then I'd say
it's not easier to debug. If it is and I don't know where to look ...
well then I just need to get educated.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Robert Haas <robertmhaas@gmail.com> writes:

On Mon, Oct 6, 2014 at 8:15 PM, Peter Eisentraut <peter_e@gmx.net> wrote:

The TAP tests
are arguably already much easier to debug than pg_regress ever was.

Well, maybe. I wasn't able, after about 5 minutes of searching, to
locate either a log file with details of the failure or the code that
revealed what the test, the expected result, and the actual result
were. It's possible that all that information is there and I just
don't know where to look; it took me a while to learn where the
various logs (postmaster.log, initdb.log, results) left behind by
pg_regress were, too. If that information is not there, then I'd say
it's not easier to debug. If it is and I don't know where to look ...
well then I just need to get educated.

The given case seemed pretty opaque to me too. Could we maybe
have some documentation about how to debug TAP failures? Or in
other words, if they're "arguably" easier to debug, how about
presenting that argument?

Also to the point: does the buildfarm script know how to collect
the information needed to debug a TAP failure?

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 10/07/2014 12:15 AM, Tom Lane wrote:

Robert Haas <robertmhaas@gmail.com> writes:

On Mon, Oct 6, 2014 at 8:15 PM, Peter Eisentraut <peter_e@gmx.net> wrote:

The TAP tests
are arguably already much easier to debug than pg_regress ever was.

Well, maybe. I wasn't able, after about 5 minutes of searching, to
locate either a log file with details of the failure or the code that
revealed what the test, the expected result, and the actual result
were. It's possible that all that information is there and I just
don't know where to look; it took me a while to learn where the
various logs (postmaster.log, initdb.log, results) left behind by
pg_regress were, too. If that information is not there, then I'd say
it's not easier to debug. If it is and I don't know where to look ...
well then I just need to get educated.

The given case seemed pretty opaque to me too. Could we maybe
have some documentation about how to debug TAP failures? Or in
other words, if they're "arguably" easier to debug, how about
presenting that argument?

Also to the point: does the buildfarm script know how to collect
the information needed to debug a TAP failure?

No. In fact, it doesn't yet know how to run those tests. That's on my
TODO list.

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 10/07/2014 09:53 AM, Andrew Dunstan wrote:

On 10/07/2014 12:15 AM, Tom Lane wrote:

Robert Haas <robertmhaas@gmail.com> writes:

On Mon, Oct 6, 2014 at 8:15 PM, Peter Eisentraut <peter_e@gmx.net>
wrote:

The TAP tests
are arguably already much easier to debug than pg_regress ever was.

Well, maybe. I wasn't able, after about 5 minutes of searching, to
locate either a log file with details of the failure or the code that
revealed what the test, the expected result, and the actual result
were. It's possible that all that information is there and I just
don't know where to look; it took me a while to learn where the
various logs (postmaster.log, initdb.log, results) left behind by
pg_regress were, too. If that information is not there, then I'd say
it's not easier to debug. If it is and I don't know where to look ...
well then I just need to get educated.

The given case seemed pretty opaque to me too. Could we maybe
have some documentation about how to debug TAP failures? Or in
other words, if they're "arguably" easier to debug, how about
presenting that argument?

Also to the point: does the buildfarm script know how to collect
the information needed to debug a TAP failure?

No. In fact, it doesn't yet know how to run those tests. That's on my
TODO list.

OK, I have a preliminary cut at adding these tests to the client. See
<http://www.pgbuildfarm.org/cgi-bin/show_stage_log.pl?nm=crake&amp;dt=2014-10-07%2015%3A38%3A04&amp;stg=bin-check&gt;
for an example run. The patch is at
<https://github.com/PGBuildFarm/client-code/commit/6f644b779c90b16f96e4454b807e804bde48b563&gt;

I don't much like the idea of doing an install/initdb/start for every
directory in src/bin, though. Can't we at least manage a single
installation directory for all these?

Also I notice that the tests remove their data directories. That could
make collecting any diagnosis data more difficult. Right now, I have no
idea what I'm looking for anyway.

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Andrew Dunstan <andrew@dunslane.net> writes:

I don't much like the idea of doing an install/initdb/start for every
directory in src/bin, though. Can't we at least manage a single
installation directory for all these?

Peter had a patch to eliminate the overhead of multiple subinstalls;
not sure where that stands, but presumably it would address your issue.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Mon, Oct 6, 2014 at 8:15 PM, Peter Eisentraut <peter_e@gmx.net> wrote:

On Thu, 2014-10-02 at 21:18 -0400, Robert Haas wrote:

If none of this gets us closer to an answer, I can try to produce a
patch that produces more details for such failures.

A test that fails for no reason that can be gleaned from the output is
not an improvement over not having a test at all.

I understand that this isn't great, and it's certainly something I'm
looking into. But it's like pg_regress saying that psql crashed and
leaving you to find out why. I don't think saying that the entire
regression test suite is useless because of that is fair. The TAP tests
are arguably already much easier to debug than pg_regress ever was.

I looked into this a bit more. This fixes it:

diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 545b2f3..2ffb7eb 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -205,6 +205,7 @@ sub program_options_handling_ok
                  '2>', \$stderr;
                ok(!$result, "$cmd with invalid option nonzero exit code");
                isnt($stderr, '', "$cmd with invalid option prints
error message");
+               run [ '/usr/bin/true' ];
        };
 }

Not that I'm recommending that fix, of course.

The problem is that running initdb --not-a-valid-option leaves $? set
to 256, as seems entirely unsurprising. After running the anonymous
block passed to it, Test::Builder::subtest calls Test::Build::finalize
which calls Test::Build::_ending, which sets $real_exit_code to $? -
i.e. 256. That function later throws up if $real_exit_code isn't 0,
hence the failure.

Off-hand, I'm not quite sure why this works for anyone.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 10/9/14 3:38 PM, Robert Haas wrote:

The problem is that running initdb --not-a-valid-option leaves $? set
to 256, as seems entirely unsurprising. After running the anonymous
block passed to it, Test::Builder::subtest calls Test::Build::finalize
which calls Test::Build::_ending, which sets $real_exit_code to $? -
i.e. 256. That function later throws up if $real_exit_code isn't 0,
hence the failure.

Off-hand, I'm not quite sure why this works for anyone.

Thank you for this analysis. The problem is that the version of
Test::Simple that ships with Perl 5.12 is broken in this particular way.
I have committed a fix to skip the tests in that case.

You might want to review whether this is really the Perl installation
that you want to use. It looks like it is coming from MacPorts, and
they have stopped updating their perl package.

I have run the PostgreSQL build against all Perl major versions between
5.8 and 5.20, and there have been no more unaddressed version-related
issues.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 10/07/2014 01:57 PM, Tom Lane wrote:

Andrew Dunstan <andrew@dunslane.net> writes:

I don't much like the idea of doing an install/initdb/start for every
directory in src/bin, though. Can't we at least manage a single
installation directory for all these?

Peter had a patch to eliminate the overhead of multiple subinstalls;
not sure where that stands, but presumably it would address your issue.

Is there any progress on this. I'm reluctant to add this to the
buildfarm client until it's solved. These tests currently take a heck of
a lot longer than any other test suite.

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Andrew Dunstan <andrew@dunslane.net> writes:

On 10/07/2014 01:57 PM, Tom Lane wrote:

Peter had a patch to eliminate the overhead of multiple subinstalls;
not sure where that stands, but presumably it would address your issue.

Is there any progress on this. I'm reluctant to add this to the
buildfarm client until it's solved. These tests currently take a heck of
a lot longer than any other test suite.

While I'd like to see that patch committed to cut the runtime of "make
check" in contrib, it's hardly the only stumbling block between us and
enabling TAP tests in the buildfarm.

The pathname length problem I noted in
/messages/by-id/16477.1413831571@sss.pgh.pa.us
seems like a show-stopper as well, since undoubtedly a number of
buildfarm critters are using buildroots with paths long enough to
trigger it.

The larger issue though is that even with both the above things fixed,
the TAP tests would still be an expensive no-op on the majority of
buildfarm members. AFAICT, I do not own a single machine on which the
current TAP tests will consent to run, and in most cases that's after
going out of my way to fetch CPAN modules that aren't in the vendor Perl
installs. Peter's mostly been fixing the portability issues by disabling
tests, which I guess is better than no fix at all, but it leaves darn
little useful functionality.

I think we need a serious discussion about choosing a baseline Perl
version on which we need the TAP tests to work, and then some effort
to make the tests actually work (not just skip tests) on all versions
beyond that.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 10/26/2014 12:29 PM, Tom Lane wrote:

Andrew Dunstan <andrew@dunslane.net> writes:

On 10/07/2014 01:57 PM, Tom Lane wrote:

Peter had a patch to eliminate the overhead of multiple subinstalls;
not sure where that stands, but presumably it would address your issue.

Is there any progress on this. I'm reluctant to add this to the
buildfarm client until it's solved. These tests currently take a heck of
a lot longer than any other test suite.

While I'd like to see that patch committed to cut the runtime of "make
check" in contrib, it's hardly the only stumbling block between us and
enabling TAP tests in the buildfarm.

The pathname length problem I noted in
/messages/by-id/16477.1413831571@sss.pgh.pa.us
seems like a show-stopper as well, since undoubtedly a number of
buildfarm critters are using buildroots with paths long enough to
trigger it.

+1 for fixing that, although it seems like a problem in what's being
tested rather than in the test suite.

The larger issue though is that even with both the above things fixed,
the TAP tests would still be an expensive no-op on the majority of
buildfarm members. AFAICT, I do not own a single machine on which the
current TAP tests will consent to run, and in most cases that's after
going out of my way to fetch CPAN modules that aren't in the vendor Perl
installs. Peter's mostly been fixing the portability issues by disabling
tests, which I guess is better than no fix at all, but it leaves darn
little useful functionality.

I think we need a serious discussion about choosing a baseline Perl
version on which we need the TAP tests to work, and then some effort
to make the tests actually work (not just skip tests) on all versions
beyond that.

As far as the buildfarm goes, we could make it a cheap noop by checking
for the presence of the required modules (AFAIK that's Test::More,
IPC::CMD and IPC::Run).

I agree that just having it not run tests on most platforms is hardly a
solution.

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Andrew Dunstan <andrew@dunslane.net> writes:

On 10/26/2014 12:29 PM, Tom Lane wrote:

The pathname length problem I noted in
/messages/by-id/16477.1413831571@sss.pgh.pa.us
seems like a show-stopper as well, since undoubtedly a number of
buildfarm critters are using buildroots with paths long enough to
trigger it.

+1 for fixing that, although it seems like a problem in what's being
tested rather than in the test suite.

I agree, but nonetheless we don't want the buildfarm turning mostly
red because we enable TAP before fixing this.

The larger issue though is that even with both the above things fixed,
the TAP tests would still be an expensive no-op on the majority of
buildfarm members.

As far as the buildfarm goes, we could make it a cheap noop by checking
for the presence of the required modules (AFAIK that's Test::More,
IPC::CMD and IPC::Run).

You'd probably have to check not just presence but version; but yeah,
that is a potential solution to the cycle-wastage problem.

I agree that just having it not run tests on most platforms is hardly a
solution.

It doesn't do much to make the tests actually useful, for sure ...

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Sun, Oct 26, 2014 at 12:29 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

The larger issue though is that even with both the above things fixed,
the TAP tests would still be an expensive no-op on the majority of
buildfarm members. AFAICT, I do not own a single machine on which the
current TAP tests will consent to run, and in most cases that's after
going out of my way to fetch CPAN modules that aren't in the vendor Perl
installs. Peter's mostly been fixing the portability issues by disabling
tests, which I guess is better than no fix at all, but it leaves darn
little useful functionality.

I agree, emphatically. Honestly, if we can't get these tests running
everywhere with reasonable effort, we should just rip them out. We've
gone to a lot of trouble in general to make sure that our source code
can be ported even to systems that arguably nobody uses any more, and
instrumental to that effort is keeping the system requirements to
install and test PostgreSQL minimal. At this point, I wouldn't mind
moving the goalposts from C89 to C89 + a bunch of C99 features that
are available on all the platforms we have buildfarm coverage for, and
I wouldn't mind require perl to compile and install, full stop. But
this patch has gone much further than that: you need a new-enough
version of perl, and a new-enough version of a bunch of modules that
aren't installed by default, and maybe not even in the vendor install,
and the documentation on how to make it work is an embarrassment:

http://www.postgresql.org/docs/devel/static/regress-tap.html

Beyond all that, I have serious doubts about whether, even if we
eventually get these tests mostly working in most places, whether they
will actually catch any bugs. For example, consider dropdb. The TAP
tests cover the following points:

- When run with --help or --version, it should exit with code 0, print
something on stderr, and print nothing on stdout.
- When run with --not-a-valid-option, it should exit with a non-zero
exit code, print something on stderr, and print nothing on stdout.
- "dropdb foobar1" should cause "statement: DROP DATABASE foobar1" to
show up in the postgresql log file.
- "dropdb nonexistent" should exit non-zero.

These are certainly good things to test, but I'd argue that once
you've verified that they are working, they're unlikely to get broken
again in the future. I'm generally supportive of the idea that we
need more automated tests, but these tests seem pretty low-value to
me, even if they were running everywhere, and even moreso if they
aren't.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers