single task postgresql

Oleg Bartunov wrote:

Having frustrated with performance on Windows box I'm wondering if it's
possible to get postgresql optimized for working without shared memory,
say in single-task mode. It looks like it's shared memory emulation on disk
(by cygipc daemon) is responsible for performance degradation.
In our project we have to use Windows for desktop application and it's
single task, so we don't need shared memory. In principle, it's possible
to hack cygipc, so it wouldn't emulate shared memory and address calls
to normal memory, but I'm wondering if it's possible from postgres side.

Regards,

How does cygwin do shared memory on Windows? Windows support real shared
memory, surely the cygwin guys are using "real" shared memory. Are you sure
that this is the problem? If so, might we not be able to use a few
macros/#ifdefs to do it right?

I'll be up for that if you guys don't mind some "ifdef/endif" stuff here and
there.

Oleg Bartunov <oleg@sai.msu.su> writes:

Having frustrated with performance on Windows box I'm wondering if it's
possible to get postgresql optimized for working without shared memory,
say in single-task mode. It looks like it's shared memory emulation on disk
(by cygipc daemon) is responsible for performance degradation.
In our project we have to use Windows for desktop application and it's
single task, so we don't need shared memory. In principle, it's possible
to hack cygipc, so it wouldn't emulate shared memory and address calls
to normal memory, but I'm wondering if it's possible from postgres side.

As mlw comments, that is probably not really the source of the
performance issue. However, you should be able to hack it if you
want to check. A standalone backend just malloc()s what would otherwise
be the shared memory area. As a first approximation you could just fire
up a standalone backend and see if it seems any faster.

regards, tom lane

On Tue, 2002-02-26 at 17:52, Oleg Bartunov wrote:

Having frustrated with performance on Windows box I'm wondering if it's
possible to get postgresql optimized for working without shared memory,
say in single-task mode. It looks like it's shared memory emulation on disk
(by cygipc daemon) is responsible for performance degradation.
In our project we have to use Windows for desktop application and it's
single task, so we don't need shared memory. In principle, it's possible
to hack cygipc, so it wouldn't emulate shared memory and address calls
to normal memory, but I'm wondering if it's possible from postgres side.

You might also be interested in hacking the threaded verion to run on
native windows without cygwin.

On Sun, 2002-01-27 at 02:19, mkscott@sacadia.com wrote:

For anyone interested, I have posted a new version of multi-threaded
Postgres 7.0.2 here:

http://sourceforge.net/projects/mtpgsql

---------------
Hannu

On 27 Feb 2002, Hannu Krosing wrote:

On Tue, 2002-02-26 at 17:52, Oleg Bartunov wrote:

Having frustrated with performance on Windows box I'm wondering if it's
possible to get postgresql optimized for working without shared memory,
say in single-task mode. It looks like it's shared memory emulation on disk
(by cygipc daemon) is responsible for performance degradation.
In our project we have to use Windows for desktop application and it's
single task, so we don't need shared memory. In principle, it's possible
to hack cygipc, so it wouldn't emulate shared memory and address calls
to normal memory, but I'm wondering if it's possible from postgres side.

You might also be interested in hacking the threaded verion to run on
native windows without cygwin.

unfortunately it's based on 7.0.2 and doesn't supports dynamic function call.

On Sun, 2002-01-27 at 02:19, mkscott@sacadia.com wrote:

For anyone interested, I have posted a new version of multi-threaded
Postgres 7.0.2 here:

http://sourceforge.net/projects/mtpgsql

---------------
Hannu

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

On Tue, 26 Feb 2002, Tom Lane wrote:

Oleg Bartunov <oleg@sai.msu.su> writes:

Having frustrated with performance on Windows box I'm wondering if it's
possible to get postgresql optimized for working without shared memory,
say in single-task mode. It looks like it's shared memory emulation on disk
(by cygipc daemon) is responsible for performance degradation.
In our project we have to use Windows for desktop application and it's
single task, so we don't need shared memory. In principle, it's possible
to hack cygipc, so it wouldn't emulate shared memory and address calls
to normal memory, but I'm wondering if it's possible from postgres side.

As mlw comments, that is probably not really the source of the
performance issue. However, you should be able to hack it if you

earlier versions of windows doesn't have shared memory and I did see
a file on disk when run postgresql on w98. That's why I suggest it's
a source of performance drop.

want to check. A standalone backend just malloc()s what would otherwise
be the shared memory area. As a first approximation you could just fire
up a standalone backend and see if it seems any faster.

thanks, it works and doesn't need shared memory, only 2 semaphors.
will investigate further.

regards, tom lane

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

Oleg Bartunov wrote:

we have to support earlier windows which have no shared memory support.

I'm pretty sure it could a configuration option then. Using a file is just
plain stupid. "Every" version of Windows has supported shared memory in one
form or another.

What version are you wishing to support which the cygwin guys claim does not
have shared memory?

I'm prety sure I can write a shared memory interface for Windows which should
work across all "supported" versions.

(BTW I have been a Windows developer since version 1.03.)

On Wed, 27 Feb 2002, mlw wrote:

Oleg Bartunov wrote:

we have to support earlier windows which have no shared memory support.

I'm pretty sure it could a configuration option then. Using a file is just
plain stupid. "Every" version of Windows has supported shared memory in one
form or another.

What version are you wishing to support which the cygwin guys claim does not
have shared memory?

We've tried windows 98 and would like to have support for ALL versions
beginning from W95.

I'm prety sure I can write a shared memory interface for Windows which should
work across all "supported" versions.

(BTW I have been a Windows developer since version 1.03.)

great ! We have had success to run our OpenFTS under Windows but need
postgresql runs a bit faster. It'd be great if you write such interface,
so postgresql could run under older windows without shared memory.
btw, BeoS seems is supported in similar way. I think people would appreciate
this. There are still many old hardware with Windows95 installed which
could be used for desktop application based on postgresql.

On Tue, 26 Feb 2002, mlw wrote:

Oleg Bartunov wrote:

Having frustrated with performance on Windows box I'm wondering if it's
possible to get postgresql optimized for working without shared memory,
say in single-task mode. It looks like it's shared memory emulation on disk
(by cygipc daemon) is responsible for performance degradation.
In our project we have to use Windows for desktop application and it's
single task, so we don't need shared memory. In principle, it's possible
to hack cygipc, so it wouldn't emulate shared memory and address calls
to normal memory, but I'm wondering if it's possible from postgres side.

Regards,

How does cygwin do shared memory on Windows? Windows support real shared
memory, surely the cygwin guys are using "real" shared memory. Are you sure
that this is the problem? If so, might we not be able to use a few
macros/#ifdefs to do it right?

I'll be up for that if you guys don't mind some "ifdef/endif" stuff here and
there.

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

Windows does not really have shared memory support. This has been a
beef with the Win32 API for a long time now. Because it has been a long
time complaint, it was finally added in Win2000 and later. Likewise,
I'd like to point out that thinks like sims, shared memory, pipes, etc,
and other entities commonly used for concurrent programming strategies
are slower in XP. So, because shared memory really isn't well
supported, they elected to have what is, in essense, memory mapped
files. Multiple processes then map the same file and read/write to it
as needed, more or less as you would shared memory. Unless you plan on
only targetting on Win 2000 and XP, it sounds like a waste of time.

As for your shared memory interface, I suspect you'll find up with
pretty much what the cygwin guys have. Your implementation may prove to
be slightly lighter as they may of had some other issues to address but
I doubt yours would prove to be much of a difference from a performance
perspective as it's still going to be pretty much the same thing under
the covers for most of the Win32 platforms. Also, expect to be taking a
performance hit with XP as is so I can't say targetting that platform
makes much sense, IMOHO.

Here's from MSDN:

CreateSharedMemory
The CreateSharedMemory function creates a section of memory that is
shared by client processes and the security package.

PVOID NTAPI CreateSharedMemory (
ULONG MaxSize,
ULONG InitialSize
);
Parameters
[in] MaxSize
Specifies the maximum size of the shared memory.
[in] InitialSize Specifies the initial size of the shared memory.
Return Values
The function returns a pointer to the block of shared memory, or NULL if
the block was not reserved.

Remarks
Creating a shared section for each client is not advisable because it is
a very expensive operation and may exhaust system resources.

The package's clients can write to shared memory which makes it
susceptible to attack. Data in the shared segment should not be trusted.

The pointer returned by the CreateSharedMemory function is required by
the AllocateSharedMemory, DeleteSharedMemory, and FreeSharedMemory
functions.

Use the DeleteSharedMemory function to release memory reserved by the
CreateSharedMemory function.

Pointers to these functions are available in the
LSA_SECPKG_FUNCTION_TABLE structure received by the SpInitialize function.

Requirements
Windows NT/2000 or later: Requires Windows 2000 or later.
Header: Declared in Ntsecpkg.h.

Here's an except from Python's Win32 API:

Memory Mapped Files

Access to Win32 Memory Mapped files. Memory mapped files allow efficient
sharing of data between separate processes. It allows a block of
(possibly shared) memory to behave like a file.

This was just thrown in to show that memory mapped files are the
standard for shared memory on Win32 systems. Likewise, I'd like to
point out that I've not seen anything published that would suggest that
shared memory versus memory mapped files via the Win32 API should yield
a higher performance.

Greg

mlw wrote:

Oleg Bartunov wrote:

we have to support earlier windows which have no shared memory support.

I'm pretty sure it could a configuration option then. Using a file is just
plain stupid. "Every" version of Windows has supported shared memory in one
form or another.

What version are you wishing to support which the cygwin guys claim does not
have shared memory?

I'm prety sure I can write a shared memory interface for Windows which should
work across all "supported" versions.

(BTW I have been a Windows developer since version 1.03.)

On Tue, 26 Feb 2002, mlw wrote:

Oleg Bartunov wrote:

Having frustrated with performance on Windows box I'm wondering if it's
possible to get postgresql optimized for working without shared memory,
say in single-task mode. It looks like it's shared memory emulation on disk
(by cygipc daemon) is responsible for performance degradation.
In our project we have to use Windows for desktop application and it's
single task, so we don't need shared memory. In principle, it's possible
to hack cygipc, so it wouldn't emulate shared memory and address calls
to normal memory, but I'm wondering if it's possible from postgres side.

Regards,

How does cygwin do shared memory on Windows? Windows support real shared
memory, surely the cygwin guys are using "real" shared memory. Are you sure
that this is the problem? If so, might we not be able to use a few
macros/#ifdefs to do it right?

I'll be up for that if you guys don't mind some "ifdef/endif" stuff here and
there.

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

--
Greg Copeland, Principal Consultant
Copeland Computer Consulting
--------------------------------------------------
PGP/GPG Key at http://www.keyserver.net
5A66 1470 38F5 5E1B CABD 19AF E25A F56E 96DC 2FA9
--------------------------------------------------

Oh ya, sorry for not including these in my previous posting, however,
here's the links that may prove to be of value for you:

Memory mapped files as shared memory:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/filemap_64jd.asp

Shared memory interface, added August 2001:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/security/customsecfunctions_13eh.asp

Enjoy,
Greg

mlw wrote:

Oleg Bartunov wrote:

we have to support earlier windows which have no shared memory support.

I'm pretty sure it could a configuration option then. Using a file is just
plain stupid. "Every" version of Windows has supported shared memory in one
form or another.

What version are you wishing to support which the cygwin guys claim does not
have shared memory?

I'm prety sure I can write a shared memory interface for Windows which should
work across all "supported" versions.

(BTW I have been a Windows developer since version 1.03.)

On Tue, 26 Feb 2002, mlw wrote:

Oleg Bartunov wrote:

Having frustrated with performance on Windows box I'm wondering if it's
possible to get postgresql optimized for working without shared memory,
say in single-task mode. It looks like it's shared memory emulation on disk
(by cygipc daemon) is responsible for performance degradation.
In our project we have to use Windows for desktop application and it's
single task, so we don't need shared memory. In principle, it's possible
to hack cygipc, so it wouldn't emulate shared memory and address calls
to normal memory, but I'm wondering if it's possible from postgres side.

Regards,

How does cygwin do shared memory on Windows? Windows support real shared
memory, surely the cygwin guys are using "real" shared memory. Are you sure
that this is the problem? If so, might we not be able to use a few
macros/#ifdefs to do it right?

I'll be up for that if you guys don't mind some "ifdef/endif" stuff here and
there.

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

--
Greg Copeland, Principal Consultant
Copeland Computer Consulting
--------------------------------------------------
PGP/GPG Key at http://www.keyserver.net
5A66 1470 38F5 5E1B CABD 19AF E25A F56E 96DC 2FA9
--------------------------------------------------

Greg Copeland wrote:

Windows does not really have shared memory support. This has been a
beef with the Win32 API for a long time now. Because it has been a long
time complaint, it was finally added in Win2000 and later. Likewise,
I'd like to point out that thinks like sims, shared memory, pipes, etc,
and other entities commonly used for concurrent programming strategies
are slower in XP. So, because shared memory really isn't well
supported, they elected to have what is, in essense, memory mapped
files. Multiple processes then map the same file and read/write to it
as needed, more or less as you would shared memory. Unless you plan on
only targetting on Win 2000 and XP, it sounds like a waste of time.

This is not really true. Under DOS windows, i.e. 95,98, etc. Shared memory can
be done in 16 bit land with a touch of assembly and a DLL. Allocate, with
globalalloc, a shared memory segment. The base selector is a valid 32 bit
selector, and the memory is mapped in the above 2G space shared and mapped to
all 32bit processes.

Under NT through 2K, yes using a memory mapped files is the way to do it, but
you do not actually need to create a file, you can use (HANDLE)0xFFFFFFFF,
which is the NT equivilent of the system memory file. The handle returned is a
system global object which can be shared across processes.

On Wed, 27 Feb 2002, mlw wrote:

Greg Copeland wrote:

Windows does not really have shared memory support. This has been a
beef with the Win32 API for a long time now. Because it has been a long
time complaint, it was finally added in Win2000 and later. Likewise,
I'd like to point out that thinks like sims, shared memory, pipes, etc,
and other entities commonly used for concurrent programming strategies
are slower in XP. So, because shared memory really isn't well
supported, they elected to have what is, in essense, memory mapped
files. Multiple processes then map the same file and read/write to it
as needed, more or less as you would shared memory. Unless you plan on
only targetting on Win 2000 and XP, it sounds like a waste of time.

This is not really true. Under DOS windows, i.e. 95,98, etc. Shared memory can
be done in 16 bit land with a touch of assembly and a DLL. Allocate, with
globalalloc, a shared memory segment. The base selector is a valid 32 bit
selector, and the memory is mapped in the above 2G space shared and mapped to
all 32bit processes.

Under NT through 2K, yes using a memory mapped files is the way to do it, but
you do not actually need to create a file, you can use (HANDLE)0xFFFFFFFF,
which is the NT equivilent of the system memory file. The handle returned is a
system global object which can be shared across processes.

Mark,

do you consider to work on this issue ?

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

mlw wrote:

Greg Copeland wrote:

Windows does not really have shared memory support. This has been a
beef with the Win32 API for a long time now. Because it has been a long
time complaint, it was finally added in Win2000 and later. Likewise,
I'd like to point out that thinks like sims, shared memory, pipes, etc,
and other entities commonly used for concurrent programming strategies
are slower in XP. So, because shared memory really isn't well
supported, they elected to have what is, in essense, memory mapped
files. Multiple processes then map the same file and read/write to it
as needed, more or less as you would shared memory. Unless you plan on
only targetting on Win 2000 and XP, it sounds like a waste of time.

This is not really true. Under DOS windows, i.e. 95,98, etc. Shared memory can
be done in 16 bit land with a touch of assembly and a DLL. Allocate, with
globalalloc, a shared memory segment. The base selector is a valid 32 bit
selector, and the memory is mapped in the above 2G space shared and mapped to
all 32bit processes.

That's fine ad all, however, we're talking about the Win32 API. None of
that qualifies. If you wish to implement back door magic for your own
use, I guess that's fine but I'd wonder how well it would be received by
the rest of the developers at large. I'm betting it wouldn't give me
the warm fuzzies.

Under NT through 2K, yes using a memory mapped files is the way to do it, but
you do not actually need to create a file, you can use (HANDLE)0xFFFFFFFF,
which is the NT equivilent of the system memory file. The handle returned is a
system global object which can be shared across processes.

Yes, that's fine, however, this pretty much puts us back to a memory
mapped file/shared memory implementation.

I guess you could have a feature set which is perhaps optimial (again,
I've not seen anything which indicates you'll see perf improvement) for
Win2000/XP platforms. Perhaps you'd be willing to perform some
benchmarks and provide source of random read/writes to these segments
and present to us for comparsion? I'm personally very curious as to see
if there is any addition speed to be gleaned even if I am doubtful.

On a side note, if you did create such a benchmark, I'm fairly sure
there are a couple of guys at IBM that may find them of interest and
might be willing to contrast these results with other platforms (various
win32 and unix/linux) and publish the results.

After all, if this path truely provides an optimized solution I'm sure
many Win32 coders would like to hear about it.

Greg

Greg Copeland wrote:

mlw wrote:

Greg Copeland wrote:

Windows does not really have shared memory support. This has been a
beef with the Win32 API for a long time now. Because it has been a long
time complaint, it was finally added in Win2000 and later. Likewise,
I'd like to point out that thinks like sims, shared memory, pipes, etc,
and other entities commonly used for concurrent programming strategies
are slower in XP. So, because shared memory really isn't well
supported, they elected to have what is, in essense, memory mapped
files. Multiple processes then map the same file and read/write to it
as needed, more or less as you would shared memory. Unless you plan on
only targetting on Win 2000 and XP, it sounds like a waste of time.

This is not really true. Under DOS windows, i.e. 95,98, etc. Shared memory can
be done in 16 bit land with a touch of assembly and a DLL. Allocate, with
globalalloc, a shared memory segment. The base selector is a valid 32 bit
selector, and the memory is mapped in the above 2G space shared and mapped to
all 32bit processes.

That's fine ad all, however, we're talking about the Win32 API. None of
that qualifies. If you wish to implement back door magic for your own
use, I guess that's fine but I'd wonder how well it would be received by
the rest of the developers at large. I'm betting it wouldn't give me
the warm fuzzies.

It cerainly wont give UNIX developers the warm and fuzzies, thats for sure, but
I have developed Windows software since there has been a Windows, and believe
me, there is no "non-trivial" Windows application which is not forced, at some
point, to do these sorts of things.

Windows, as a platform, is pure and simple, a disaster.

Under NT through 2K, yes using a memory mapped files is the way to do it, but
you do not actually need to create a file, you can use (HANDLE)0xFFFFFFFF,
which is the NT equivilent of the system memory file. The handle returned is a
system global object which can be shared across processes.

Yes, that's fine, however, this pretty much puts us back to a memory
mapped file/shared memory implementation.

Not nessisarily true. The system memory map should not have the same criteria
for disk writes. A memory mapped file has some notion hard disk "persistence."
NT will maintain the disk image. When a view of the system memory is mapped,
there is no requirement that the disk image is written. It will follow the OS
virtual memory strategy for generic memory. (It may even make it more
persistent than generic RAM, I will have to double check.)

I guess you could have a feature set which is perhaps optimial (again,
I've not seen anything which indicates you'll see perf improvement) for
Win2000/XP platforms. Perhaps you'd be willing to perform some
benchmarks and provide source of random read/writes to these segments
and present to us for comparsion? I'm personally very curious as to see
if there is any addition speed to be gleaned even if I am doubtful.

I am not sure if there will be much benefit, I too am skeptical, with one
caveat, if NT or Windows is using a shared file handle, there may be contention
at the disk level for the disk, and that could impact performance.

On NT, puting the shared memory file on one disk, and the database on another,
one should be able to see how often the disk which contains the shared image is
written.

Another posibility is to create a RAM disk and put the shared memory file on
it.

On a side note, if you did create such a benchmark, I'm fairly sure
there are a couple of guys at IBM that may find them of interest and
might be willing to contrast these results with other platforms (various
win32 and unix/linux) and publish the results.

After all, if this path truely provides an optimized solution I'm sure
many Win32 coders would like to hear about it.

Sigh, it looks like I have to reinstall NT and startup my Windows development
stuff again. I was having so much fun the last few years developing for Linux.
Again, sigh. :-|

Oleg Bartunov wrote:

On Wed, 27 Feb 2002, mlw wrote:

Greg Copeland wrote:

Windows does not really have shared memory support. This has been a
beef with the Win32 API for a long time now. Because it has been a long
time complaint, it was finally added in Win2000 and later. Likewise,
I'd like to point out that thinks like sims, shared memory, pipes, etc,
and other entities commonly used for concurrent programming strategies
are slower in XP. So, because shared memory really isn't well
supported, they elected to have what is, in essense, memory mapped
files. Multiple processes then map the same file and read/write to it
as needed, more or less as you would shared memory. Unless you plan on
only targetting on Win 2000 and XP, it sounds like a waste of time.

This is not really true. Under DOS windows, i.e. 95,98, etc. Shared memory can
be done in 16 bit land with a touch of assembly and a DLL. Allocate, with
globalalloc, a shared memory segment. The base selector is a valid 32 bit
selector, and the memory is mapped in the above 2G space shared and mapped to
all 32bit processes.

Under NT through 2K, yes using a memory mapped files is the way to do it, but
you do not actually need to create a file, you can use (HANDLE)0xFFFFFFFF,
which is the NT equivilent of the system memory file. The handle returned is a
system global object which can be shared across processes.

Mark,

do you consider to work on this issue ?

Yea, let me think about it. What is your time frame? When I offered to work on
it, I thought it could be a leasurely thing. I have to get a machine running
some form of Windows on which to develop and test.

I want to say yes, and if no one else does it, I will, but I'm not sure what
your timeframe is. If it is the mystical 7.3, then sure I can do it easily. If
you need something quickly, I can help, but I don't think I could shoulder the
whole thing.

I have a couple things I have promised people. Let me get those done. I will
try to write an equivilent set of functions for shget, shmat, etc. as soon as I
can. Anyone wanting to run with them can hack and test PostgreSQL on Windows.

How does that sound?

On Wed, 27 Feb 2002, mlw wrote:

Oleg Bartunov wrote:

On Wed, 27 Feb 2002, mlw wrote:

Greg Copeland wrote:

Windows does not really have shared memory support. This has been a
beef with the Win32 API for a long time now. Because it has been a long
time complaint, it was finally added in Win2000 and later. Likewise,
I'd like to point out that thinks like sims, shared memory, pipes, etc,
and other entities commonly used for concurrent programming strategies
are slower in XP. So, because shared memory really isn't well
supported, they elected to have what is, in essense, memory mapped
files. Multiple processes then map the same file and read/write to it
as needed, more or less as you would shared memory. Unless you plan on
only targetting on Win 2000 and XP, it sounds like a waste of time.

This is not really true. Under DOS windows, i.e. 95,98, etc. Shared memory can
be done in 16 bit land with a touch of assembly and a DLL. Allocate, with
globalalloc, a shared memory segment. The base selector is a valid 32 bit
selector, and the memory is mapped in the above 2G space shared and mapped to
all 32bit processes.

Under NT through 2K, yes using a memory mapped files is the way to do it, but
you do not actually need to create a file, you can use (HANDLE)0xFFFFFFFF,
which is the NT equivilent of the system memory file. The handle returned is a
system global object which can be shared across processes.

Mark,

do you consider to work on this issue ?

Yea, let me think about it. What is your time frame? When I offered to work on

yesterday :-) I think, at first we need to be sure this way could provide
performance win and compatibility.

it, I thought it could be a leasurely thing. I have to get a machine running
some form of Windows on which to develop and test.

I want to say yes, and if no one else does it, I will, but I'm not sure what
your timeframe is. If it is the mystical 7.3, then sure I can do it easily. If
you need something quickly, I can help, but I don't think I could shoulder the
whole thing.

looks like people will appreciate your work. Currently we're investigating
another possibility Tom suggested - standalone backend. But things are
still dim, so we'll also track your development.

I have a couple things I have promised people. Let me get those done. I will
try to write an equivilent set of functions for shget, shmat, etc. as soon as I
can. Anyone wanting to run with them can hack and test PostgreSQL on Windows.

How does that sound?

Loosk great, we could run and test. I'm not sure if we have some funding
for this work but I think I could talk with project manager once I'd be
sure this way is promising.

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

mlw wrote:

Oleg Bartunov wrote:

On Wed, 27 Feb 2002, mlw wrote:

[Greg's ramblings removed]

do you consider to work on this issue ?

Yea, let me think about it. What is your time frame? When I offered to work on
it, I thought it could be a leasurely thing. I have to get a machine running
some form of Windows on which to develop and test.

I want to say yes, and if no one else does it, I will, but I'm not sure what
your timeframe is. If it is the mystical 7.3, then sure I can do it easily. If
you need something quickly, I can help, but I don't think I could shoulder the
whole thing.

It appears that cygipc's shared memory implementation boils down to
pretty much this section of code. As extracted from shm_connect in shm.c:

GFdShm = open(CYGWIN_IPCNT_FILESHM, O_RDWR, 00666 ) ;
shareadrshm = (CYGWIN_IPCNT_SHMSTR *)
mmap(0, sizeof(CYGWIN_IPCNT_SHMSTR), PROT_WRITE|PROT_READ,
MAP_SHARED, GFdShm, 0) ;
if( shareadrshm == (CYGWIN_IPCNT_SHMSTR *) -1 )
{
close (GFdShm) ;
return (0) ;
}

So, I guess the question of the day is, how is Cygwin handling mmap()
calls??

Perhaps looking to see if Cygwin can support page file access via
INVALID_HANDLE_VALUE to the *assumed* CreateFileMapping call?? I get
the impression that you already understand this, however, I thought I'd
go ahead and toss this out there. By using the INVALID_HANDLE_VALUE
(0xFFFFFFFF) value during's initial mapped file's creation, it is
actually being backed by the VM's page file rather than a distinct file
layered on top of the FS.

As you said earlier, this may provide for less disk contension, however,
I'm doubtful you're going to find a significant speed improvement. Just
the same, I guess a couple percent here and there can add up.

In case you can't tell, I'm somewhat interested in this thread. When
you get ready to do the implementation, I wouldn't mind kicking the code
around a little. Of course, being that pesky voice that says, "what
about this..." can just be plain fun too. ;)

Greg

-----Original Message-----
From: Hannu Krosing [mailto:hannu@krosing.net]
Sent: Tuesday, February 26, 2002 9:54 PM
To: Oleg Bartunov
Cc: Pgsql Hackers
Subject: Re: [HACKERS] single task postgresql

On Tue, 2002-02-26 at 17:52, Oleg Bartunov wrote:

Having frustrated with performance on Windows box I'm wondering if

it's

possible to get postgresql optimized for working without shared

memory,

say in single-task mode. It looks like it's shared memory emulation on

disk

(by cygipc daemon) is responsible for performance degradation.
In our project we have to use Windows for desktop application and it's
single task, so we don't need shared memory. In principle, it's

possible

to hack cygipc, so it wouldn't emulate shared memory and address calls
to normal memory, but I'm wondering if it's possible from postgres

side.

You might also be interested in hacking the threaded verion to run on
native windows without cygwin.

On Sun, 2002-01-27 at 02:19, mkscott@sacadia.com wrote:

For anyone interested, I have posted a new version of multi-threaded
Postgres 7.0.2 here:

http://sourceforge.net/projects/mtpgsql

I did a modification of a port of POSIX threads to NT so that I can run
UNIX threaded chess programs on NT. I don't know if it will work with
the threaded PostgreSQL server, but you are welcome to try.
Here is the stuff I did, you are free to use it however you like:
ftp://cap.connx.com/pub/amy/pthreads.ZIP
<<

Mark,

I've found
"Fast synchronized access to shared memory for Windows and for i86 Unix-es"
http://www.ispras.ru/~knizhnik/shmem/Readme.htm
Would't be useful ?

Regards,

Oleg

On Wed, 27 Feb 2002, mlw wrote:

Oleg Bartunov wrote:

On Wed, 27 Feb 2002, mlw wrote:

Greg Copeland wrote:

Windows does not really have shared memory support. This has been a
beef with the Win32 API for a long time now. Because it has been a long
time complaint, it was finally added in Win2000 and later. Likewise,
I'd like to point out that thinks like sims, shared memory, pipes, etc,
and other entities commonly used for concurrent programming strategies
are slower in XP. So, because shared memory really isn't well
supported, they elected to have what is, in essense, memory mapped
files. Multiple processes then map the same file and read/write to it
as needed, more or less as you would shared memory. Unless you plan on
only targetting on Win 2000 and XP, it sounds like a waste of time.

This is not really true. Under DOS windows, i.e. 95,98, etc. Shared memory can
be done in 16 bit land with a touch of assembly and a DLL. Allocate, with
globalalloc, a shared memory segment. The base selector is a valid 32 bit
selector, and the memory is mapped in the above 2G space shared and mapped to
all 32bit processes.

Under NT through 2K, yes using a memory mapped files is the way to do it, but
you do not actually need to create a file, you can use (HANDLE)0xFFFFFFFF,
which is the NT equivilent of the system memory file. The handle returned is a
system global object which can be shared across processes.

Mark,

do you consider to work on this issue ?

Yea, let me think about it. What is your time frame? When I offered to work on
it, I thought it could be a leasurely thing. I have to get a machine running
some form of Windows on which to develop and test.

I want to say yes, and if no one else does it, I will, but I'm not sure what
your timeframe is. If it is the mystical 7.3, then sure I can do it easily. If
you need something quickly, I can help, but I don't think I could shoulder the
whole thing.

I have a couple things I have promised people. Let me get those done. I will
try to write an equivilent set of functions for shget, shmat, etc. as soon as I
can. Anyone wanting to run with them can hack and test PostgreSQL on Windows.

How does that sound?

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

Sorry for taking so long to get back to everyone. I wanted to post a
follow up to the profiling data that has been submitted as well as
comment on the provided link (thank you btw).

The profiling data provided had some minor issues with it. It seems
that everything was able to run in exactly zero % of overall time.
While this doesn't have to mean the results are invalid, it does raise
that question. It is certainly possible that the overall % duration for
the reported functions was so negligible that it should show up as 0%,
however, somewhere in the back of my head I seem to recall that cygwin's
profiler is broken in this regard. So, while there are some minor
differences, there are no timings to indicate which path in which
profiling is consuming the greatest amount of time. Is it possible to
run a longer benchmark to see if we can get anything to register with a
percent value higher than 0%?

At any rate, I'm still wading through the two files to determine if
there is yet any value to found within the profiling results.

Greg

On 6 Mar 2002, Greg Copeland wrote:

Sorry for taking so long to get back to everyone. I wanted to post a
follow up to the profiling data that has been submitted as well as
comment on the provided link (thank you btw).

The profiling data provided had some minor issues with it. It seems
that everything was able to run in exactly zero % of overall time.
While this doesn't have to mean the results are invalid, it does raise
that question. It is certainly possible that the overall % duration for
the reported functions was so negligible that it should show up as 0%,
however, somewhere in the back of my head I seem to recall that cygwin's
profiler is broken in this regard. So, while there are some minor
differences, there are no timings to indicate which path in which
profiling is consuming the greatest amount of time. Is it possible to
run a longer benchmark to see if we can get anything to register with a
percent value higher than 0%?

timings doesnt' correct !
gettimeofday doesn't works properly under Cygwin. It's claimed that
1.3.10 (we have used 1.3.9)
"- Use QueryPerformance* functions for gettimeofday calls. (cgf)".
We'll rerun tests and submit results.

btw, Konstatntin Knizhnik suggested that poor performance of postgres
under cygwin+cygipc could be caused by locking. We need to look at
task manager to see how cpu is occupied by postgres.

At any rate, I'm still wading through the two files to determine if
there is yet any value to found within the profiling results.

Greg

On Wed, 2002-03-06 at 09:36, Oleg Bartunov wrote:

Mark,

I've found
"Fast synchronized access to shared memory for Windows and for i86 Unix-es"
http://www.ispras.ru/~knizhnik/shmem/Readme.htm
Would't be useful ?

Regards,

Oleg

On Wed, 27 Feb 2002, mlw wrote:

Oleg Bartunov wrote:

On Wed, 27 Feb 2002, mlw wrote:

Greg Copeland wrote:

Windows does not really have shared memory support. This has been a
beef with the Win32 API for a long time now. Because it has been a long
time complaint, it was finally added in Win2000 and later. Likewise,
I'd like to point out that thinks like sims, shared memory, pipes, etc,
and other entities commonly used for concurrent programming strategies
are slower in XP. So, because shared memory really isn't well
supported, they elected to have what is, in essense, memory mapped
files. Multiple processes then map the same file and read/write to it
as needed, more or less as you would shared memory. Unless you plan on
only targetting on Win 2000 and XP, it sounds like a waste of time.

This is not really true. Under DOS windows, i.e. 95,98, etc. Shared memory can
be done in 16 bit land with a touch of assembly and a DLL. Allocate, with
globalalloc, a shared memory segment. The base selector is a valid 32 bit
selector, and the memory is mapped in the above 2G space shared and mapped to
all 32bit processes.

Under NT through 2K, yes using a memory mapped files is the way to do it, but
you do not actually need to create a file, you can use (HANDLE)0xFFFFFFFF,
which is the NT equivilent of the system memory file. The handle returned is a
system global object which can be shared across processes.

Mark,

do you consider to work on this issue ?

Yea, let me think about it. What is your time frame? When I offered to work on
it, I thought it could be a leasurely thing. I have to get a machine running
some form of Windows on which to develop and test.

I want to say yes, and if no one else does it, I will, but I'm not sure what
your timeframe is. If it is the mystical 7.3, then sure I can do it easily. If
you need something quickly, I can help, but I don't think I could shoulder the
whole thing.

I have a couple things I have promised people. Let me get those done. I will
try to write an equivilent set of functions for shget, shmat, etc. as soon as I
can. Anyone wanting to run with them can hack and test PostgreSQL on Windows.

How does that sound?

Regards,
Oleg

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

Seems I'm replying rather quickly but I thought I'd point out that I
went back and started looking at the cygipc code again. I'm now
starting to suspect that that the majority of the performance impact we
are seeing has to do with the semaphore implementation versus the shared
memory implementation.

Basically, the version which is using local memory is not having to
contend with a significant amount of semaphore negotiations while the
shared memory version must contend with this issue each and every time
memory is accessed.

It's worth noting that based on what I've seen so far (pointing out that
lots still need to be reviewed), the semaphore implementation on via the
cygipc library is going to yield an absolute worse case semaphore
performance on Win32 platforms. That is, of all the natve
synchronization mechanisms available for Win32, the use of a generic
Semaphore is going to deliver the worst performance whereby one would
expect it to be perhaps an order of magnitude or slower than an ideal
Win32 semaphore implementation.

So, if you have the time, perhaps you can write some quick benchmarks on
Win32. One which simply allocates a shared memory region and randomly
reads and writes to it. Same thing for the shared/memory mapped file
implementation. Next, create a multi-process implementation based on
each of your previous two tests. In these test, try using the cygwin
library, native Win32 Mutex implementation (warning, this has some
issues because on Win32 Mutexs are optimized for threading and not
multi-process implementations), a Win32 Critical section (warning SMP
scaling issues -- absolute fastest for uni-processor systems on Win32)
and a native Win32 Semaphore implementation (horrible performance). Of
course, timing each.

If my *guess* is correct, this will tell you that the significant
performance issues are directly associated with the use of Semaphores.

As I am starting to understand things, it looks like performance suffers
because the Win32 platforms are significantly biased toward threaded IPC
implementations and significantly suffer when forced into a
multi-process architecture (very Unix like).

Greg

Hehe.

Thank, I just submitted another email about this very issue.

Greg