mb and ecpg regression tests

Andrew Dunstan wrote:

We've had ecpg regression tests being doing by the buildfarm for a
while, but they are not enabled for MSVC. Also, buildfarm has never
supported doing MBCS regression tests. In both cases the regression
tests are driven by Unix shell scripts, and in the MBCS case at least,
the script would need plenty of love before it would be suitable for the
buildfarm even on Unix.

Back when we were thinking of supporting MSVC builds, we thought that
everything would need to be a C program, and so we rewrote the main
regression script in C, for example. However, now we have a build system
for MSVC we can see that its main infrastructure is in fact perl, so I
think what we need to do is to convert the MBCS and ECPG regression
drivers to perl rather than C - which should be far, far simpler task.

I'm going to try to get these done and supported by the buildfarm by the
time we get to Beta.

I don't think it's essential, but one of the other reasons Magnus and I
discussed for rewriting the regression tests in C was that we could
include it in the future with the installer as a platform/installation
confidence test.

Regards, Dave.

Dave Page wrote:

Andrew Dunstan wrote:

We've had ecpg regression tests being doing by the buildfarm for a
while, but they are not enabled for MSVC. Also, buildfarm has never
supported doing MBCS regression tests. In both cases the regression
tests are driven by Unix shell scripts, and in the MBCS case at least,
the script would need plenty of love before it would be suitable for the
buildfarm even on Unix.

Back when we were thinking of supporting MSVC builds, we thought that
everything would need to be a C program, and so we rewrote the main
regression script in C, for example. However, now we have a build system
for MSVC we can see that its main infrastructure is in fact perl, so I
think what we need to do is to convert the MBCS and ECPG regression
drivers to perl rather than C - which should be far, far simpler task.

I'm going to try to get these done and supported by the buildfarm by the
time we get to Beta.

I don't think it's essential, but one of the other reasons Magnus and I
discussed for rewriting the regression tests in C was that we could
include it in the future with the installer as a platform/installation
confidence test.

That's a good point. So, do we want/need the same for MBCS and ECPG tests?

If so, we should work on creating a regression shared lib so we don't
have to replicate lots of code.

cheers

andrew

Andrew Dunstan <andrew@dunslane.net> writes:

Back when we were thinking of supporting MSVC builds, we thought that
everything would need to be a C program, and so we rewrote the main
regression script in C, for example. However, now we have a build system
for MSVC we can see that its main infrastructure is in fact perl, so I
think what we need to do is to convert the MBCS and ECPG regression
drivers to perl rather than C - which should be far, far simpler task.

I disagree with this for ecpg because (a) it would be a capability
regression from the point of view of anyone building a non-perl-enabled
build on Unix, and (b) we already have the C regression driver, why
can't we re-use or adapt it for ecpg instead of writing and debugging
and maintaining an entirely new set of test infrastructure?

For the MBCS stuff I don't care, since no one runs that on a regular
basis anyway (or should need to...)

regards, tom lane

Tom Lane wrote:

Andrew Dunstan <andrew@dunslane.net> writes:

Back when we were thinking of supporting MSVC builds, we thought that
everything would need to be a C program, and so we rewrote the main
regression script in C, for example. However, now we have a build system
for MSVC we can see that its main infrastructure is in fact perl, so I
think what we need to do is to convert the MBCS and ECPG regression
drivers to perl rather than C - which should be far, far simpler task.

I disagree with this for ecpg because (a) it would be a capability
regression from the point of view of anyone building a non-perl-enabled
build on Unix, and (b) we already have the C regression driver, why
can't we re-use or adapt it for ecpg instead of writing and debugging
and maintaining an entirely new set of test infrastructure?

For the MBCS stuff I don't care, since no one runs that on a regular
basis anyway (or should need to...)

Well, you need perl to build from CVS, IIRC, but I take the point.

I don't think ECPG can use pg_regress as is, because it does more than
just run psql. We probably need to look at factoring out the common bits
into a shared lib. That's not a bad idea anyway.

cheers

andrew

Andrew Dunstan <andrew@dunslane.net> writes:

I don't think ECPG can use pg_regress as is, because it does more than
just run psql. We probably need to look at factoring out the common bits
into a shared lib. That's not a bad idea anyway.

A shared library is probably overkill (it has a lot more installation
overhead than it's worth), but at least factor the source
code so we don't have two copies of the common bits.

It wouldn't be a bad goal to try to use the same "pg_regress 2.0" for
both uses, either. I'm tempted to suggest it should become a separate
subdirectory under src/bin/ to remind people it's decoupled from the
regression tests per se...

regards, tom lane

Andrew Dunstan wrote:

That's a good point. So, do we want/need the same for MBCS and ECPG tests?

It would be nice for ECPG. I wasn't even aware of the MBCS stuff - if it
will exercise the unicode sorting which is a little different on Windows
iirc (due to the lack of full native UTF-8 support) it might be a useful
addition, but I wouldn't say it was worth busting a gut over.

Regards, Dave.

Andrew Dunstan wrote:

We've had ecpg regression tests being doing by the buildfarm for a
while, but they are not enabled for MSVC. Also, buildfarm has never
supported doing MBCS regression tests. In both cases the regression
tests are driven by Unix shell scripts, and in the MBCS case at least,
the script would need plenty of love before it would be suitable for the
buildfarm even on Unix.

Back when we were thinking of supporting MSVC builds, we thought that
everything would need to be a C program, and so we rewrote the main
regression script in C, for example. However, now we have a build system
for MSVC we can see that its main infrastructure is in fact perl, so I
think what we need to do is to convert the MBCS and ECPG regression
drivers to perl rather than C - which should be far, far simpler task.

I'm going to try to get these done and supported by the buildfarm by the
time we get to Beta.

IIRC, Joachim had started working on the ecpg part. (Which is one of the
reasons I haven't looked at it myself yet)

Joachim, did you ever get anywhere with that?

//Magnus

On Thu, May 17, 2007 at 10:14:45PM +0200, Magnus Hagander wrote:

IIRC, Joachim had started working on the ecpg part. (Which is one of the
reasons I haven't looked at it myself yet)

Joachim, did you ever get anywhere with that?

I have ecpg tests somehow running under msvc but there are still some issues.
Here are the problems that I encounter(ed):

a) ecpg uses a program for each test that runs and generates output. Those
programs need to be compiled: in MSVC we would either compile them on the
fly while testing or generate a new project for every single executable.
Since the second option is overkill, Magnus suggested to try compiling them
on the fly with the command line compiler. This is ugly but seems to work.

b) ecpg tests use several directories. This causes several problems, one of
them being the resultmap format to be changed, it currently uses:

float4/i.86-pc-mingw32=...

we might want to change the / to some other character here.

c) ecpg uses 3 different files per test. stderr, stdout and generated
source. There is not only just one file to diff against an expected file.

My approach was to have separate init and test routines for each "module"
(backend regression tests, ecpg or whatever) and share the rest.
The init routine sets up stuff and the test routine knows about the specific
test logic. It generates lists of files to diff. The diffing and reporting
is kept common of course. A problem here are alternative comparison files.
We have the resultmap entries and we have "test_n.out". This has to be
combined with ecpg having 3 files. We might want to use

test-platform.stdout instead of test.stdout and
test_1.stderr instead of test.stderr

I have not yet sorted out this last item. Proposals welcome. I hope that I
can fix it on the weekend or at least by the end of next week and then send
my version to whoever is interested.

Joachim

Joachim Wieland wrote:

On Thu, May 17, 2007 at 10:14:45PM +0200, Magnus Hagander wrote:

IIRC, Joachim had started working on the ecpg part. (Which is one of the
reasons I haven't looked at it myself yet)

Joachim, did you ever get anywhere with that?

I have ecpg tests somehow running under msvc but there are still some issues.
Here are the problems that I encounter(ed):

a) ecpg uses a program for each test that runs and generates output. Those
programs need to be compiled: in MSVC we would either compile them on the
fly while testing or generate a new project for every single executable.
Since the second option is overkill, Magnus suggested to try compiling them
on the fly with the command line compiler. This is ugly but seems to work.

It might also be possible to generate a wildcard msbuild file for them,
perhaps? (If you have Visual Studio 2005, you will have msbuild) Worth
investigating, but this is probably the easiest thing to fix.

b) ecpg tests use several directories. This causes several problems, one of
them being the resultmap format to be changed, it currently uses:

float4/i.86-pc-mingw32=...

we might want to change the / to some other character here.

Hmm. I guess doable if necessary.

c) ecpg uses 3 different files per test. stderr, stdout and generated
source. There is not only just one file to diff against an expected file.

My approach was to have separate init and test routines for each "module"
(backend regression tests, ecpg or whatever) and share the rest.
The init routine sets up stuff and the test routine knows about the specific
test logic. It generates lists of files to diff. The diffing and reporting
is kept common of course. A problem here are alternative comparison files.
We have the resultmap entries and we have "test_n.out". This has to be
combined with ecpg having 3 files. We might want to use

test-platform.stdout instead of test.stdout and
test_1.stderr instead of test.stderr

I have not yet sorted out this last item. Proposals welcome. I hope that I
can fix it on the weekend or at least by the end of next week and then send
my version to whoever is interested.

Hmm. Probably easier for someone who already knows how the ecpg tests
actually work. I just keep breaking them ;-)

However - do go ahead and send what you have to -patches for comments as
a WIP patch, even if it's not ready. I'd certainly be interested at
looking at what you have so far.

//Magnus