APR 1.0 released

Gaetano Mendola wrote:

Hi all,
now that Apache Portable Runtime was release why don't
use it on Postgres?

Now that we have discovered the formula for green cheese, why don't we
remake the moon out of it?

cheers

andrew

On Sat, 4 Sep 2004, Gaetano Mendola wrote:

Hi all,
now that Apache Portable Runtime was release why don't
use it on Postgres?

Short question: why? what does it give us, other then potential reliance
on another project to build ... ?

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664

Gaetano Mendola <mendola@bigfoot.com> writes:

now that Apache Portable Runtime was release why don't
use it on Postgres?

The sense of the question is backwards. Why *should* we use it?

regards, tom lane

Centuries ago, Nostradamus foresaw when scrappy@postgresql.org ("Marc G. Fournier") would write:

On Sat, 4 Sep 2004, Gaetano Mendola wrote:

Hi all,
now that Apache Portable Runtime was release why don't
use it on Postgres?

Short question: why? what does it give us, other then potential
reliance on another project to build ... ?

It would allow reopening all the threads about why PostgreSQL doesn't
use threading...

That being said, there are places where there would be merit to using
threading in PostgreSQL, though NOT where the usual futile discussions
ask for it. Notably, on an SMP system, it would be a neat idea for
complex queries involving joins to split themselves so that different
parts run in separate threads.

The other Way, Way Cool part would be for queries that are scanning
big tables to split the scans into unions of partial scans, so that on
an 8 CPU box you'd take the "Big 4GB Table" and have 8 threads
simultaneously scanning different parts of it. (And making ARC all
the more important :-).)

But that would, however it happened, involve BIG, SCARY changes...

... And since APR isn't BSD licensed, that would probably cause a
problem.
--
(format nil "~S@~S" "cbbrowne" "acm.org")
http://www3.sympatico.ca/cbbrowne/wp.html
Of course, unless one has a theory, one cannot expect much help from a
computer unless _it_ has a theory)... -- Marvin Minsky

Tom Lane wrote:

Gaetano Mendola <mendola@bigfoot.com> writes:

now that Apache Portable Runtime was release why don't
use it on Postgres?

The sense of the question is backwards. Why *should* we use it?

In order to avoid all the annoyance that someone else had in
write code portable. I mean, how much time ( I'm not a postgres
developer, I like to think, for lack of time ) was spent in order
to port postgres to win32 ? Don't you think that use of APR could
save time ?

Andrew: about the green cheese, why not remake the moon with it
if this have some benefit ?

Marc: you are not obliged to change APR version each eye blink.
Don't you think that use a portable library could save time ?

One example for all: actually postgres is multi process, some time
I see my server with 3 CPU in holiday and one overloaded to sort
thousand rows. Don't you think in some cases spawn a couple of
thread could improve it ? Let me dream that you agree on this and
may be in years someone start to do it ( I'm using postgres since
when "create or replace function" or "table functions" was a blasphemy
so I'm sure that will happen). What are you going to do? Reinvent
the hell and create a sort of framework to work with thread dealing
with Win32 world ? I don't know if APR provide a spin lock mechanism,
tell me how many times did you see discussion here on hackers about
on how make the spin lock effective?
In my experience I'm a C++ developer and each time I have to do
something I full rely on STL, BOOST, XALAN, XERCES and may be I'll
use the APR now that seem stable enough and I swear each time my
colleagues are reinventing the list, queue, thread interactions....

Regards
Gaetano Mendola

Gaetano Mendola <mendola@bigfoot.com> writes:

Don't you think that use of APR could save time ?

No, because we've already *done* the work it would purport to save.
It would cost us work to adapt our code to sit on top of APR, and
it's not clear to me that we'd be getting anything for it.

IIRC, this was proposed before and we looked at APR in some detail,
and came to the conclusion that it wouldn't be worth changing. See
the archives.

Don't you think in some cases spawn a couple of
thread could improve it ?

The fact that we were on top of APR would not automagically mean that
we could thread-ize the backend, nor that we would want to.

I don't know if APR provide a spin lock mechanism,

You don't even know that, but you're confident that we can throw away
our spinlock work and use APR anyway. You're wasting our time. Get
some evidence if you want to propose this.

regards, tom lane

Christopher Browne wrote:

... And since APR isn't BSD licensed, that would probably cause a
problem.

They are changin license for APR and I'll be not surprised if they
adopth the BSD one.

Regards
Gaetano Mendola

Gaetano Mendola wrote:

Tom Lane wrote:

Gaetano Mendola <mendola@bigfoot.com> writes:

now that Apache Portable Runtime was release why don't
use it on Postgres?

The sense of the question is backwards. Why *should* we use it?

In order to avoid all the annoyance that someone else had in
write code portable. I mean, how much time ( I'm not a postgres
developer, I like to think, for lack of time ) was spent in order
to port postgres to win32 ? Don't you think that use of APR could
save time ?

Andrew: about the green cheese, why not remake the moon with it
if this have some benefit ?

Go and study the history of how long it took the Apache people to get
APR done. Look at the history of the various MPMs. By contrast, we got
our Windows port done in rather less than a year, partly by *not* going
down ratholes like APR. Now it's true that they had a different (and
harder) set of problems to deal with - in particular scaling to huge
numbers of very short-lived connections. Even so, it took them a very
long time (years and years) to get right, and they still use a different
MPM by default on Windows from what they use on Unix - and you have to
choose it at configure time. I am not crtiticizing the Apache people - I
am just saying there is no evidence that using APR would have any
benefit at all for PostgreSQL - and it would be massively invasive and
require huge effort to do so.

cheers

andrew

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tom Lane wrote:

| Gaetano Mendola <mendola@bigfoot.com> writes:
|>I don't know if APR provide a spin lock mechanism,
|
|
| You don't even know that, but you're confident that we can throw away
| our spinlock work and use APR anyway. You're wasting our time. Get
| some evidence if you want to propose this.

I'm sorry to have wasted your time.

With the experience you have I don't think you need my evidences.

Regards
Gaetano Mendola

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (MingW32)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBOlKM7UpzwH2SGd4RAshqAJ9L1MeBsWlU/G2I1DRoA1/GifMj9ACg/RWX
IInlsOU+Z78LSyyL3maBC3Q=
=BcID
-----END PGP SIGNATURE-----

On Sun, 5 Sep 2004, Gaetano Mendola wrote:

Christopher Browne wrote:

... And since APR isn't BSD licensed, that would probably cause a
problem.

They are changin license for APR and I'll be not surprised if they
adopth the BSD one.

Since Apache has developed their own license, and I've seen at least one
non-Apache project (Spamassassin) switch over to it so far, I would be
very surprised is APR switched a non-Apache license ...

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664

Gaetano Mendola wrote:
[ PGP not available, raw data follows ]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tom Lane wrote:

| Gaetano Mendola <mendola@bigfoot.com> writes:
|>I don't know if APR provide a spin lock mechanism,
|
|
| You don't even know that, but you're confident that we can throw away
| our spinlock work and use APR anyway. You're wasting our time. Get
| some evidence if you want to propose this.

I'm sorry to have wasted your time.

With the experience you have I don't think you need my evidences.

I looked at the APR code to get some ideas for the Win32 port. Some of
the ideas were good, but in other places like rename they didn't do very
well we were better off doing it ourselves and getting it right.

I remember looking at their code to fix the rename/unlink while the file
is open problem, and they didn't seem to have a fix for that so we
developed our own method that works like Unix.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian schrieb:

I looked at the APR code to get some ideas for the Win32 port. Some of
the ideas were good, but in other places like rename they didn't do very
well we were better off doing it ourselves and getting it right.

I remember looking at their code to fix the rename/unlink while the file
is open problem, and they didn't seem to have a fix for that so we
developed our own method that works like Unix.

sorry, but your rename doesn't work on cygwin. maybe it works with mingw.

cygwin has it's own and working way of doing rename's.
maybe you should have looked at the cygwin sources instead.
(src/winsup/cygwin/syscalls.cc)

first doing a WinAPI MoveFileEx and then after a failure trying the
cygwin version, which will also try its own MoveFile loop, will not
work. they are conflicting.

same with unlink, but at least the mingw and cygwin unlink versions
don't conflict here. here you don't stack two conflicting loops together.
nevertheless cygwin's unlink is much better than pgunlink in case of
locking problems. it does its own sort of delayed removal then.

IMHO port/dirmod.c is a dirty and half-baked hack, which works for mingw
only.
--
Reini Urban
http://xarch.tu-graz.ac.at/home/rurban/

Reini Urban said:

Bruce Momjian schrieb:

I looked at the APR code to get some ideas for the Win32 port. Some
of the ideas were good, but in other places like rename they didn't do
very well we were better off doing it ourselves and getting it right.

I remember looking at their code to fix the rename/unlink while the
file is open problem, and they didn't seem to have a fix for that so
we developed our own method that works like Unix.

sorry, but your rename doesn't work on cygwin. maybe it works with
mingw.

cygwin has it's own and working way of doing rename's.
maybe you should have looked at the cygwin sources instead.
(src/winsup/cygwin/syscalls.cc)

first doing a WinAPI MoveFileEx and then after a failure trying the
cygwin version, which will also try its own MoveFile loop, will not
work. they are conflicting.

same with unlink, but at least the mingw and cygwin unlink versions
don't conflict here. here you don't stack two conflicting loops
together. nevertheless cygwin's unlink is much better than pgunlink in
case of locking problems. it does its own sort of delayed removal
then.

IMHO port/dirmod.c is a dirty and half-baked hack, which works for
mingw only.

Are you sure you are reading this code correctly? Your reading would only be
correct if WIN32 is defined on Cygwin - it isn't IIRC (don't have a
convenient way to test ATM). The relevant code is this:

#ifdef WIN32
while (!MoveFileEx(from, to, MOVEFILE_REPLACE_EXISTING))
#endif
#ifdef __CYGWIN__
while (rename(from, to) < 0)
#endif

If the code doesn't work, please submit empirical proof, rather than make
assertions of "half-baked hack". If it's broken we'll fix it. Bruce's point
about the usefulness of APR remains, nonetheless.

cheers

andrew

Andrew Dunstan schrieb:

Reini Urban said:

Bruce Momjian schrieb:

I looked at the APR code to get some ideas for the Win32 port. Some
of the ideas were good, but in other places like rename they didn't do
very well we were better off doing it ourselves and getting it right.

I remember looking at their code to fix the rename/unlink while the
file is open problem, and they didn't seem to have a fix for that so
we developed our own method that works like Unix.

sorry, but your rename doesn't work on cygwin. maybe it works with
mingw.

cygwin has it's own and working way of doing rename's.
maybe you should have looked at the cygwin sources instead.
(src/winsup/cygwin/syscalls.cc)

first doing a WinAPI MoveFileEx and then after a failure trying the
cygwin version, which will also try its own MoveFile loop, will not
work. they are conflicting.

same with unlink, but at least the mingw and cygwin unlink versions
don't conflict here. here you don't stack two conflicting loops
together. nevertheless cygwin's unlink is much better than pgunlink in
case of locking problems. it does its own sort of delayed removal
then.

IMHO port/dirmod.c is a dirty and half-baked hack, which works for
mingw only.

Are you sure you are reading this code correctly? Your reading would only be
correct if WIN32 is defined on Cygwin - it isn't IIRC (don't have a
convenient way to test ATM). The relevant code is this:

#ifdef WIN32
while (!MoveFileEx(from, to, MOVEFILE_REPLACE_EXISTING))
#endif
#ifdef __CYGWIN__
while (rename(from, to) < 0)
#endif

If the code doesn't work, please submit empirical proof, rather than make
assertions of "half-baked hack". If it's broken we'll fix it. Bruce's point
about the usefulness of APR remains, nonetheless.

I already posted my needed patches to make beta2 work on cygwin.
But on the pqsql-cygwin mailinglist:
http://xarch.tu-graz.ac.at/home/rurban/software/cygwin/postgresql/postgresql-8.0.0beta2-1
Only a plperl problem is pending. BTW: plperl never worked on cygwin before.

FYI: WIN32 is also defined because <windows.h> is included.
(/usr/incluse/w32api/windef.h)
If you want this or that, do proper nesting, and use #else.

#ifdef __CYGWIN__
while (rename(from, to) < 0)
#else
#ifdef WIN32
while (!MoveFileEx(from, to, MOVEFILE_REPLACE_EXISTING))
#endif
#endif

You cannot safely assume WIN32 is only defined on mingw, but not on
__CYGWIN__. And you need <windows.h> because of some winapi calls below.
The same false assumption was also in src/include/utils/palloc.h

But the whole pgrename #ifdef is fragile and a mess.
cygwin rename works good enough, and I just #ifdef'ed it away.
The two #undef need to be inserted before #include <unistd.h>, otherwise
pgrename will be declared, but rename not.

gcc -E -I../include dirmod-orig.c:
int
pgrename(const char *from, const char *to)
{
int loops = 0;
while (!MoveFileExA(from, to, 1))
while (rename(from, to) < 0)
{
if (GetLastError() != 5L)
if ((*__errno()) != 13)
return -1;
pg_usleep(100000);
if (loops == 30)
errstart(0, "dirmod-orig.c", 87,
__func__), elog_finish(15, "could not rename \"%s\" to \"%s\",
continuing to try",
from, to);
loops++;
}
if (loops > 30)
errstart(0, "dirmod-orig.c", 98, __func__),
elog_finish(15, "completed rename of \"%s\" to \"%s\"", from, to);
return 0;
}

--
Reini Urban
http://xarch.tu-graz.ac.at/home/rurban/

Reini Urban wrote:

FYI: WIN32 is also defined because <windows.h> is included.
(/usr/incluse/w32api/windef.h)
If you want this or that, do proper nesting, and use #else.

Ugh, yes. A little experimentation shows that __WIN32__ is defined for
MinGW only, but WIN32 is for both. I wonder how we missed that in
various places. Maybe we need a little audit of the use of WIN32.

cheers

andrew

OK, care to submit a patch. As I remember the fix for rename/unlink
also includes how the file is opened with flags. Anyway, we spent a lot
of time on this so you will have to go back in the archvies to find it
and determine how it can be improved.

Your track record for Cygwin diagnosis isn't 100%. I am going to need
complete research before changing anything at this point in beta.

---------------------------------------------------------------------------

Reini Urban wrote:

Bruce Momjian schrieb:

I looked at the APR code to get some ideas for the Win32 port. Some of
the ideas were good, but in other places like rename they didn't do very
well we were better off doing it ourselves and getting it right.

I remember looking at their code to fix the rename/unlink while the file
is open problem, and they didn't seem to have a fix for that so we
developed our own method that works like Unix.

sorry, but your rename doesn't work on cygwin. maybe it works with mingw.

cygwin has it's own and working way of doing rename's.
maybe you should have looked at the cygwin sources instead.
(src/winsup/cygwin/syscalls.cc)

first doing a WinAPI MoveFileEx and then after a failure trying the
cygwin version, which will also try its own MoveFile loop, will not
work. they are conflicting.

same with unlink, but at least the mingw and cygwin unlink versions
don't conflict here. here you don't stack two conflicting loops together.
nevertheless cygwin's unlink is much better than pgunlink in case of
locking problems. it does its own sort of delayed removal then.

IMHO port/dirmod.c is a dirty and half-baked hack, which works for mingw
only.
--
Reini Urban
http://xarch.tu-graz.ac.at/home/rurban/

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian schrieb:

OK, care to submit a patch. As I remember the fix for rename/unlink
also includes how the file is opened with flags. Anyway, we spent a lot
of time on this so you will have to go back in the archvies to find it
and determine how it can be improved.

Your track record for Cygwin diagnosis isn't 100%. I am going to need
complete research before changing anything at this point in beta.

Ok, I'll do an analysis and patch which will have a chance to be accepted.
Keeping pgrename in CYGWIN is probably a good idea.
At least for consistent error reporting (which helped me in finding the
problem)
Personally I don't think that any rename()-usleep loop is necessary.
I'll check the archives.

---------------------------------------------------------------------------
Reini Urban wrote:

Bruce Momjian schrieb:

I looked at the APR code to get some ideas for the Win32 port. Some of
the ideas were good, but in other places like rename they didn't do very
well we were better off doing it ourselves and getting it right.

I remember looking at their code to fix the rename/unlink while the file
is open problem, and they didn't seem to have a fix for that so we
developed our own method that works like Unix.

sorry, but your rename doesn't work on cygwin. maybe it works with mingw.

cygwin has it's own and working way of doing rename's.
maybe you should have looked at the cygwin sources instead.
(src/winsup/cygwin/syscalls.cc)

first doing a WinAPI MoveFileEx and then after a failure trying the
cygwin version, which will also try its own MoveFile loop, will not
work. they are conflicting.

same with unlink, but at least the mingw and cygwin unlink versions
don't conflict here. here you don't stack two conflicting loops together.
nevertheless cygwin's unlink is much better than pgunlink in case of
locking problems. it does its own sort of delayed removal then.

IMHO port/dirmod.c is a dirty and half-baked hack, which works for mingw
only.

--
Reini Urban
http://xarch.tu-graz.ac.at/home/rurban/

Andrew Dunstan wrote:

Reini Urban wrote:

FYI: WIN32 is also defined because <windows.h> is included.
(/usr/incluse/w32api/windef.h)
If you want this or that, do proper nesting, and use #else.

Ugh, yes. A little experimentation shows that __WIN32__ is defined for
MinGW only, but WIN32 is for both. I wonder how we missed that in
various places. Maybe we need a little audit of the use of WIN32.

Done, and patch attached and applied. Hopefully at least Cygwin will
compile dirmod.c now. (Most of the patch is consistency
reorganization.) We still need a review of that file.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian said:

Andrew Dunstan wrote:

Reini Urban wrote:

FYI: WIN32 is also defined because <windows.h> is included.
(/usr/incluse/w32api/windef.h)
If you want this or that, do proper nesting, and use #else.

Ugh, yes. A little experimentation shows that __WIN32__ is defined for
MinGW only, but WIN32 is for both. I wonder how we missed that in
various places. Maybe we need a little audit of the use of WIN32.

Done, and patch attached and applied. Hopefully at least Cygwin will
compile dirmod.c now. (Most of the patch is consistency
reorganization.) We still need a review of that file.

I don't understand most of this patch. What difference does changing the
preprocessor test order make?

cheers

andrew