"make check" improvement for cygwin

"Andrew Dunstan" <andrew@dunslane.net> writes:

The attached patch limits parallelism for "make check" on cygwin to 10 - me=
aning that at least on my installation "make check" actually succeeds at la=
st.

If we're going to do something like that, I'd rather see it exposed as a
more general "at most N connections" knob, where the user could choose
N. There are plenty of other scenarios where such a restriction could
be useful --- for instance, "make check" runs up against
max-processes-per-user limits on a number of platforms.

I would also rather that the user be forced to supply that number, to
make certain he realizes that he's got a very low number-of-connections
resource limit. Sweeping such problems under the rug is exactly what
"make check" should not do.

regards, tom lane

----- Original Message -----
From: "Tom Lane" <tgl@sss.pgh.pa.us>
To: "Andrew Dunstan" <andrew@dunslane.net>
Cc: "PG Patches" <pgsql-patches@postgresql.org>
Sent: Monday, October 27, 2003 7:41 PM
Subject: Re: [PATCHES] "make check" improvement for cygwin

"Andrew Dunstan" <andrew@dunslane.net> writes:

The attached patch limits parallelism for "make check" on cygwin to 10 -

me=

aning that at least on my installation "make check" actually succeeds at

la=

st.

If we're going to do something like that, I'd rather see it exposed as a
more general "at most N connections" knob, where the user could choose
N. There are plenty of other scenarios where such a restriction could
be useful --- for instance, "make check" runs up against
max-processes-per-user limits on a number of platforms.

I would also rather that the user be forced to supply that number, to
make certain he realizes that he's got a very low number-of-connections
resource limit. Sweeping such problems under the rug is exactly what
"make check" should not do.

Sure - I was just sharing what I did to get things working, as I have seen
reports on this from a number of people.

I should be able to generalise it fairly easily.

Something like this?

make MAX_CONNECTIONS=10 check

Maybe in the case of cygwin, where it is almost bound to fail without such
restrictions, we could put out a warning if it isn't used.

cheers

andrew

"Andrew Dunstan" <andrew@dunslane.net> writes:

Something like this?
make MAX_CONNECTIONS=10 check
Maybe in the case of cygwin, where it is almost bound to fail without such
restrictions, we could put out a warning if it isn't used.

That works for me. Possibly you should pop this proposal up on
pghackers, to make sure other people buy into it before you go to
the trouble of coding it...

regards, tom lane

----- Original Message -----
From: "Tom Lane" <tgl@sss.pgh.pa.us>
To: "Andrew Dunstan" <andrew@dunslane.net>
Cc: "PG Patches" <pgsql-patches@postgresql.org>
Sent: Monday, October 27, 2003 11:12 PM
Subject: Re: [PATCHES] "make check" improvement for cygwin

"Andrew Dunstan" <andrew@dunslane.net> writes:

Something like this?
make MAX_CONNECTIONS=10 check
Maybe in the case of cygwin, where it is almost bound to fail without

such

restrictions, we could put out a warning if it isn't used.

That works for me. Possibly you should pop this proposal up on
pghackers, to make sure other people buy into it before you go to
the trouble of coding it...

Well, I posted the note last night but I finished the patch today, because
it was very simple and took so little time.

I think this does what you suggested.

If it's accepted I'll do a doc patch to match.

cheers

andrew

Andrew,

Something like this?
make MAX_CONNECTIONS=10 check

AFAICT, non-server Windows versions have a listen backlog of 5:

http://support.microsoft.com/support/kb/articles/Q127/1/44.asp
http://216.239.57.104/search?q=cache:Ll4KYYj_BB8J:www.microsoft.com/windows2000/docs/tcpip2000.doc+site:microsoft.com+listen+backlog+winsock&amp;hl=en&amp;ie=UTF-8

So, I would suggest using "5" as the magic number instead of "10".

Anyway, I really appreciate you contributing this patch.

Thanks,
Jason

--
PGP/GPG Key: http://www.tishler.net/jason/pubkey.asc or key servers
Fingerprint: 7A73 1405 7F2B E669 C19D 8784 1AFD E4CC ECF4 8EF6

The number isn't hardcoded at all (except for the warning on gygwin if you
choose some high value) - you can choose whatever value you like at run time
(even 1). This will be documented.

10 works for me on my XP-HE/cygwin installation, though. YMMV I guess.

cheers

andrew

----- Original Message -----
From: "Jason Tishler" <jason@tishler.net>
To: "Andrew Dunstan" <andrew@dunslane.net>
Cc: "PG Patches" <pgsql-patches@postgresql.org>
Sent: Wednesday, October 29, 2003 8:40 AM
Subject: Re: [PATCHES] "make check" improvement for cygwin

Andrew,

Something like this?
make MAX_CONNECTIONS=10 check

AFAICT, non-server Windows versions have a listen backlog of 5:

http://support.microsoft.com/support/kb/articles/Q127/1/44.asp

http://216.239.57.104/search?q=cache:Ll4KYYj_BB8J:www.microsoft.com/windows2000/docs/tcpip2000.doc+site:microsoft.com+listen+backlog+winsock&amp;hl=en&amp;ie=UTF-8

Andrew,

On Wed, Oct 29, 2003 at 08:45:30AM -0500, Andrew Dunstan wrote:

The number isn't hardcoded at all

Understood.

(except for the warning on gygwin if you choose some high value)

The above is my concern -- sorry, for being unclear.

- you can choose whatever value you like at run time (even 1).

Understood.

This will be documented.

Great.

10 works for me on my XP-HE/cygwin installation, though. YMMV I guess.

^^^^

Bingo!

Jason

--
PGP/GPG Key: http://www.tishler.net/jason/pubkey.asc or key servers
Fingerprint: 7A73 1405 7F2B E669 C19D 8784 1AFD E4CC ECF4 8EF6

Jason Tishler wrote:

Andrew,

On Wed, Oct 29, 2003 at 08:45:30AM -0500, Andrew Dunstan wrote:

The number isn't hardcoded at all

Understood.

(except for the warning on gygwin if you choose some high value)

The above is my concern -- sorry, for being unclear.

I don't want to issue a warning on a setting that is likely to succeed
in some cases. How about this instead of what I had - it deals with the
most likely problem case?:

# ----------
# warn of cygwin likely failure
# if maxconnections = 0
# and we are running parallel tests
# ----------

case $host_platform in *-*-cygwin*)
case "$schedule" in *parallel*)
if [ $maxconnections -eq 0 ] ; then
echo using unlimited parallel connections is likely to fail or
hang on cygwin
echo try \"$me --max-connections=n\" or \"gmake MAX_CONNECTIONS=n
check\"
echo with n = 5 or 10 if this happens
echo
fi
;;
esac
;;
esac

Andrew,

On Wed, Oct 29, 2003 at 11:34:34AM -0500, Andrew Dunstan wrote:

I don't want to issue a warning on a setting that is likely to succeed
in some cases. How about this instead of what I had - it deals with
the most likely problem case?:

# ----------
# warn of cygwin likely failure
# if maxconnections = 0
# and we are running parallel tests
# ----------

[snip]

The above is fine. Sorry, to belabor this point.

Thanks,
Jason

--
PGP/GPG Key: http://www.tishler.net/jason/pubkey.asc or key servers
Fingerprint: 7A73 1405 7F2B E669 C19D 8784 1AFD E4CC ECF4 8EF6

"Andrew Dunstan" <andrew@dunslane.net> writes:

Well, I posted the note last night but I finished the patch today, because
it was very simple and took so little time.
I think this does what you suggested.
If it's accepted I'll do a doc patch to match.

It looks okay to me, modulo the later suggestion from Jason. Please add
that and incorporate a docs update.

regards, tom lane

"Andrew Dunstan" <andrew@dunslane.net> writes:

Well, I posted the note last night but I finished the patch today, because
it was very simple and took so little time.

I've committed this with the later revision about the warning message,
and some other minor cleanups. I added documentation in the "Regression
Tests" chapter, but if there is anyplace else you want to mention it,
send in a docs patch.

regards, tom lane

Tom Lane wrote:

"Andrew Dunstan" <andrew@dunslane.net> writes:

Well, I posted the note last night but I finished the patch today, because
it was very simple and took so little time.

I've committed this with the later revision about the warning message,
and some other minor cleanups. I added documentation in the "Regression
Tests" chapter, but if there is anyplace else you want to mention it,
send in a docs patch.

Thanks, Tom! I just sat down to do this work and saw your note.

This looks fine - the only other place it needs to be mentioned is in
FAQ_MSWIN I think.

patch for that attached.

cheers

andrew

I have attached a more comprehensive MSWIN FAQ patch. Basically, the
instructions were way out of date and incorrect. Installing Postgres
from Cygwin is easier these days than the FAQ would imply.

This patch already includes Andrew's previous patch.

Chris

Andrew Dunstan wrote:

Patch applied to 7.4 and HEAD. Thanks.

---------------------------------------------------------------------------

Christopher Kings-Lynne wrote:

I have attached a more comprehensive MSWIN FAQ patch. Basically, the
instructions were way out of date and incorrect. Installing Postgres
from Cygwin is easier these days than the FAQ would imply.

This patch already includes Andrew's previous patch.

Chris

Andrew Dunstan wrote:

Tom Lane wrote:

"Andrew Dunstan" <andrew@dunslane.net> writes:

Well, I posted the note last night but I finished the patch today,
because
it was very simple and took so little time.

I've committed this with the later revision about the warning message,
and some other minor cleanups. I added documentation in the "Regression
Tests" chapter, but if there is anyplace else you want to mention it,
send in a docs patch.

Index: doc/FAQ_MSWIN
===================================================================
RCS file: /projects/cvsroot/pgsql-server/doc/FAQ_MSWIN,v
retrieving revision 1.15
diff -c -r1.15 FAQ_MSWIN
*** doc/FAQ_MSWIN	11 Nov 2002 20:04:05 -0000	1.15
--- doc/FAQ_MSWIN	3 Nov 2003 08:08:13 -0000
***************
*** 2,37 ****
====================================
$Date: 2002/11/11 20:04:05 $

! 1. Install the latest Cygwin package, available at http://cygwin.com/.
! The Cygwin package provides a UNIX-like API on top of the Win32
! API.

! A pre-built PostgreSQL is part of the standard Cygwin distribution
! and is installed by Cygwin's setup.exe. You are encouraged to use
! this version unless it does not meet your needs. Please read the
! README file, /usr/doc/Cygwin/postgresql-${version}.README, where
! "${version}" is the version (e.g., 7.2).

! 2. Install the latest cygipc package, available at
! http://www.neuro.gatech.edu/users/cwilson/cygutils/V1.1/cygipc/.
! Do not use versions prior to 1.04, they will not work.

! Use the following command to install the cygipc package:

! $ tar -C / -xjf cygipc-${version}.tar.bz2

! where "${version}" is the version (e.g., 1.11-1).

! 3. The Cygwin bin directory has to be placed in the path before the
! Windows program directories, because the sort.exe has to be taken
! from Cygwin, not Windows.

! 4. Start ipc-daemon from the cygipc package. Use "net start ipc-daemon",
! if ipc-daemon is installed as a service; otherwise, use "ipc-daemon &".
! This program needs to be running anytime you start the PostgreSQL
! server (postmaster) or initialize a database (initdb).

! 5. Proceed according to the INSTALL file (i.e., ./configure; make; etc.)
noting the following Cygwin specific differences:

o The GNU make command is called "make" not "gmake".
--- 2,67 ----
====================================
$Date: 2002/11/11 20:04:05 $

! PostgreSQL requires the Cygwin set of libraries to be installed in
! order that it functions under Windows.

! This document assumes that you do not have Cygwin already installed
! on your system. If that is not the case, then you will need to
! adjust these instructions accordingly.

! 1. Download and run the Cygwin installer. Visit http://cygwin.com/ and
! click on the "Install Cygwin now' link. This will prompt you
! to download a "setup.exe". Save this file somewhere on your
! system and then execute it.

! 2. Proceed through the Cygwin install wizard. Choose 'Install from
! Internet', specify a Local Package Directory and choose a mirror
! site that's close to you. Leave everything else as-is.

! When you come to the point of choosing which packages to install,
! expand the 'Database' section and click 'Skip' next to PostgreSQL
! to change it to the latest version of PostgreSQL available for
! Cygwin.

! 3. Once the download and install process is complete, open a Cygwin
! shell and do the following:

! 3a. Start ipc-daemon2 for shared memory support. Use
! "net start ipc-daemon2", if you want ipc-daemon2 installed
! as a service; otherwise, use "ipc-daemon2 &". This program
! needs to be running anytime you start the PostgreSQL server
! (postmaster) or initialize a database (initdb).

! 3b. Use the initdb command to create a new database cluster. An
! example command would be:

! initdb -D /usr/local/pgsql/data -W -E LATIN1
!
! Which will create a cluster in the /usr/local/pgsql/data
! directory, will prompt for a superuser password and will
! set the default database encoding to LATIN1.
!
! 3c. Start up the postmaster. Use a command similar to the
! following:
!
! postmaster -D /usr/local/pgsql/data
!
! This will start the postmaster, and if successful you will
! see some initial log entries, and an entry "LOG: database
! system is ready".
!
! 4. You are now running a PostgreSQL server on your Windows machine.
!
! Building from source
! --------------------
!
! There are some points that are only relevant if you are building Cygwin
! PostgreSQL from source:
!
! 1. Set your path to use the Cygwin bin directory before the Windows
! utilities. Cygwin sort must be used in preference to Windows sort.exe.
!
! 2. Proceed according to the INSTALL file (i.e., ./configure; make; etc.)
noting the following Cygwin specific differences:

o The GNU make command is called "make" not "gmake".
***************
*** 44,56 ****
Alternatively, proceed according to the README file supplied with
the Cygwin PostgreSQL package.

! NOTE: The following are known issues with PostgreSQL on Windows:

1. Cygwin's AF_UNIX sockets are really implemented as AF_INET sockets
so they are inherently insecure.

! 2. make check can generate spurious regression test failures due to
overflowing the listen() backlog queue which causes connection
! refused errors.

Problem reports can be sent to pgsql-cygwin@postgresql.org.
--- 74,92 ----
Alternatively, proceed according to the README file supplied with
the Cygwin PostgreSQL package.

! Known issues
! ------------

1. Cygwin's AF_UNIX sockets are really implemented as AF_INET sockets
so they are inherently insecure.

! 2. "make check" can generate spurious regression test failures due to
overflowing the listen() backlog queue which causes connection
! refused errors or hangs. You can limit the number of connections
! using the MAX_CONNECTIONS option thus:
!
! make MAX_CONNECTIONS=5 check
!
! (On some systems you can have up to about 10 simultaneous connections).

Problem reports can be sent to pgsql-cygwin@postgresql.org.

---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Christopher,

On Sat, Nov 08, 2003 at 10:47:45PM -0500, Bruce Momjian wrote:

! PostgreSQL requires the Cygwin set of libraries to be installed in

^^^^^^^^^^^^^^^^^^^^^^^

! order that it functions under Windows.

The phrase "Cygwin set of DLLs" is more accurate. Actually, the phrase
"appropriate subset of Cygwin DLLs" is even better.

! 2. Proceed through the Cygwin install wizard. Choose 'Install from
! Internet', specify a Local Package Directory and choose a mirror
! site that's close to you. Leave everything else as-is.

^^^^^^^^^^^^^^^^^^^^^^^^^^^

The default will not work for all users.

! 3a. Start ipc-daemon2 for shared memory support. Use
! "net start ipc-daemon2", if you want ipc-daemon2 installed
! as a service; otherwise, use "ipc-daemon2 &". This program
! needs to be running anytime you start the PostgreSQL server
! (postmaster) or initialize a database (initdb).

The above is missing the "ipc-daemon2 --install-as-service" step.

You may want to add installing postmaster as a service too.

! There are some points that are only relevant if you are building Cygwin
! PostgreSQL from source:
!
! 1. Set your path to use the Cygwin bin directory before the Windows
! utilities. Cygwin sort must be used in preference to Windows sort.exe.

Is the above needed during runtime too?

Sorry for being picky and not supplying a patch. Nevertheless, I
appreciate you updating the README.

Thanks,
Jason

--
PGP/GPG Key: http://www.tishler.net/jason/pubkey.asc or key servers
Fingerprint: 7A73 1405 7F2B E669 C19D 8784 1AFD E4CC ECF4 8EF6

The phrase "Cygwin set of DLLs" is more accurate. Actually, the phrase
"appropriate subset of Cygwin DLLs" is even better.

Ok.

! 2. Proceed through the Cygwin install wizard. Choose 'Install from
! Internet', specify a Local Package Directory and choose a mirror
! site that's close to you. Leave everything else as-is.

^^^^^^^^^^^^^^^^^^^^^^^^^^^

The default will not work for all users.

Why not? I tried it on a couple of machines... I particularly noted
that it now installs cygipc by default..

! 3a. Start ipc-daemon2 for shared memory support. Use
! "net start ipc-daemon2", if you want ipc-daemon2 installed
! as a service; otherwise, use "ipc-daemon2 &". This program
! needs to be running anytime you start the PostgreSQL server
! (postmaster) or initialize a database (initdb).

The above is missing the "ipc-daemon2 --install-as-service" step.

You may want to add installing postmaster as a service too.

Can you please specify the exact set of steps to make this work
perfectly - I've never managed it myself... I managed to find your docs
on it on the Net once, but I never got it working 100%

! There are some points that are only relevant if you are building Cygwin
! PostgreSQL from source:
!
! 1. Set your path to use the Cygwin bin directory before the Windows
! utilities. Cygwin sort must be used in preference to Windows sort.exe.

Is the above needed during runtime too?

Not on any machine I've installed on. I just left those steps in
because they were in the old docs.

Sorry for being picky and not supplying a patch. Nevertheless, I
appreciate you updating the README.

That's cool. I would appreciate your tips on installing as a service,
however.

Chris

Jason Tishler writes:

! 1. Set your path to use the Cygwin bin directory before the Windows
! utilities. Cygwin sort must be used in preference to Windows sort.exe.

Is the above needed during runtime too?

Can anyone see *any* use of "sort" that would warrant the above step?

--
Peter Eisentraut peter_e@gmx.net

! 1. Set your path to use the Cygwin bin directory before the Windows
! utilities. Cygwin sort must be used in preference to Windows sort.exe.

Is the above needed during runtime too?

Can anyone see *any* use of "sort" that would warrant the above step?

I have no idea. I just left that in because it was in the original version.

tsort steps?

Chris

Christopher,

On Mon, Nov 10, 2003 at 10:18:14PM +0800, Christopher Kings-Lynne wrote:

The default will not work for all users.

Why not? I tried it on a couple of machines... I particularly noted
that it now installs cygipc by default..

The user may be behind a firewall, so the default for "Select Your
Internet Connection" may not be appropriate.

You may want to add installing postmaster as a service too.

Can you please specify the exact set of steps to make this work
perfectly - I've never managed it myself... I managed to find your
docs on it on the Net once,

I can only recommend culling to relevant information from my README:

http://www.tishler.net/jason/software/postgresql/postgresql-7.3.4.README

but I never got it working 100%

Really? They always seem to work for me... :,)

Jason

--
PGP/GPG Key: http://www.tishler.net/jason/pubkey.asc or key servers
Fingerprint: 7A73 1405 7F2B E669 C19D 8784 1AFD E4CC ECF4 8EF6