PGXLOG variable worthwhile?
Hi everyone,
Am just wondering if we've ever considered adding a PGXLOG environment
variable that would point to the pg_xlog directory?
In a Unix environment it's not real necessary as filesystem links can be
created, but in other environments (i.e. the Native windows port) it's
looking like it might be useful.
:-)
Regards and best wishes,
Justin Clift
--
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi
We dealt this this (painfully) during 7.3 development. Some wanted a -X
flag to initdb/postgres/postmaster that would identify the pg_xlog
directory while others wanted the flag only on initdb and have initdb
create a symlink.
Finally, we decided to do nothing. and continue to recommend manually
moving pg_xlog using symlinks.
Also, I have heard symlinks are available in native Windows but the
interface to them isn't clearly visible. Can someone clarify that?
---------------------------------------------------------------------------
Justin Clift wrote:
Hi everyone,
Am just wondering if we've ever considered adding a PGXLOG environment
variable that would point to the pg_xlog directory?In a Unix environment it's not real necessary as filesystem links can be
created, but in other environments (i.e. the Native windows port) it's
looking like it might be useful.:-)
Regards and best wishes,
Justin Clift
--
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
--
Bruce Momjian | http://candle.pha.pa.us
pgman@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073
-----Original Message-----
From: Bruce Momjian [mailto:pgman@candle.pha.pa.us]
Sent: 12 September 2002 06:27
To: Justin Clift
Cc: PostgreSQL Hackers Mailing List
Subject: Re: [HACKERS] PGXLOG variable worthwhile?Also, I have heard symlinks are available in native Windows
but the interface to them isn't clearly visible. Can someone
clarify that?
Well there are 'shortcuts' but I wouldn't want to trust my xlog
directory to one.
Even if I did, iirc, unless you are using the shell api, they just
appear to be regular files anyway (for example, in Cygwin vi, I can edit
a shortcut to a directory).
Regards, Dave.
Import Notes
Resolved by subject fallback
Dave Page wrote:
-----Original Message-----
From: Bruce Momjian [mailto:pgman@candle.pha.pa.us]Also, I have heard symlinks are available in native Windows
but the interface to them isn't clearly visible. Can someone
clarify that?Well there are 'shortcuts' but I wouldn't want to trust my xlog
directory to one.
These are Shell OLE links. As Dave points out, it requires the
shell to interpret the shortcut.
Even if I did, iirc, unless you are using the shell api, they just
appear to be regular files anyway (for example, in Cygwin vi, I can edit
a shortcut to a directory).Regards, Dave.
In Windows 2000 and Windows XP with an NTFS filesystem,
Microsoft has added Reparse Points, which allow for the
implementation of symbolic links for directories. Microsoft
calls them "Junctions". I *believe* the function used for
creating reparse points is DeviceIoControl() with the
FSCTL_SET_REPARSE_POINT I/O control code. I don't have quick
access to 2K or XP, but it is clearly not supported by Win32 on
95/98/ME.
Here's a link discussing the features of NTFS5 and Reparse Points:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnw2kmag00/html/NTFSPart1.asp
Mike Mascari
mascarm@mascari.com
Mike Mascari wrote:
<snip>
In Windows 2000 and Windows XP with an NTFS filesystem,
Microsoft has added Reparse Points, which allow for the
implementation of symbolic links for directories. Microsoft
calls them "Junctions". I *believe* the function used for
creating reparse points is DeviceIoControl() with the
FSCTL_SET_REPARSE_POINT I/O control code. I don't have quick
access to 2K or XP, but it is clearly not supported by Win32 on
95/98/ME.Here's a link discussing the features of NTFS5 and Reparse Points:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnw2kmag00/html/NTFSPart1.asp
That's really useful info. Reparse points under Win2k (mount points to
the rest of us) are definitely something to try out in the future then.
:)
Seems like the NT4 users are left out in the cold though until we add
some kind of ability for PostgreSQL to not look at the filesystem for
info about where to put the xlog files.
Regards and best wishes,
Justin Clift
Mike Mascari
mascarm@mascari.com
--
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi
On Thu, 12 Sep 2002, Justin Clift wrote:
Mike Mascari wrote:
<snip>In Windows 2000 and Windows XP with an NTFS filesystem,
Microsoft has added Reparse Points, which allow for the
implementation of symbolic links for directories. Microsoft
calls them "Junctions". I *believe* the function used for
creating reparse points is DeviceIoControl() with the
FSCTL_SET_REPARSE_POINT I/O control code. I don't have quick
access to 2K or XP, but it is clearly not supported by Win32 on
95/98/ME.Here's a link discussing the features of NTFS5 and Reparse Points:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnw2kmag00/html/NTFSPart1.asp
That's really useful info. Reparse points under Win2k (mount points to
the rest of us) are definitely something to try out in the future then.
:)Seems like the NT4 users are left out in the cold though until we add
some kind of ability for PostgreSQL to not look at the filesystem for
info about where to put the xlog files.
This isn't true. With the resource kit, you get the gnu utils, and ln
works a charm under NT4 with ntfs. And not just for directories, but
files as well. Unless Microsoft somehow removed that functionality in the
intervening years since I've used NT. (wouldn't put it past them, but I
doubt they have.)
"scott.marlowe" wrote:
<snip>
Seems like the NT4 users are left out in the cold though until we add
some kind of ability for PostgreSQL to not look at the filesystem for
info about where to put the xlog files.This isn't true. With the resource kit, you get the gnu utils, and ln
works a charm under NT4 with ntfs. And not just for directories, but
files as well. Unless Microsoft somehow removed that functionality in the
intervening years since I've used NT. (wouldn't put it past them, but I
doubt they have.)
The reference point that I'm working from is this:
- Am testing out the third beta of the Native PostgreSQL port for
Windows, on NT4 SP6 at present.
- Have an internal RAID array of Seagate Cheetah 10kRPM drives. When
installing the PGDATA directory on one drive it gives a certain kind of
performance, and I'm interested in testing the performance of the Native
PostgreSQL port for Windows with the xlog directory being located on
another drive.
- Have tried doing normal shortcuts, and have also tried using the
cygwin "ln" command to create the appropriate soft link. Both
approaches create a shortcut object of the correct name pointing to the
correct place on the new drive, but the only thing that appears to
follow this shortcut is when I click on them using Windows Explorer.
The Native PostgreSQL port for Windows doesn't, and neither do a few
other applications I tested.
Would it be correct to say that the 'ln' command in the MS Resource Kit
creates this kind of shortcut too, as the Reparse Points feature doesn't
seem to be possible under NT4?
Can only think of two real solutions at present, one being for us to add
a PGXLOG environment variable or similar ability (GUC parameter
perhaps?), and the other would be for the Native PostgreSQL for Windows
port to follow these shortcuts.
Not if any of these is all that easy, or maybe there is another solution
that would work (apart from ignoring the problem).
:-)
Regards and best wishes,
Justin Clift
--
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi
On Fri, 13 Sep 2002, Justin Clift wrote:
"scott.marlowe" wrote:
<snip>Seems like the NT4 users are left out in the cold though until we add
some kind of ability for PostgreSQL to not look at the filesystem for
info about where to put the xlog files.This isn't true. With the resource kit, you get the gnu utils, and ln
works a charm under NT4 with ntfs. And not just for directories, but
files as well. Unless Microsoft somehow removed that functionality in the
intervening years since I've used NT. (wouldn't put it past them, but I
doubt they have.)The reference point that I'm working from is this:
- Am testing out the third beta of the Native PostgreSQL port for
Windows, on NT4 SP6 at present.
- Have an internal RAID array of Seagate Cheetah 10kRPM drives. When
installing the PGDATA directory on one drive it gives a certain kind of
performance, and I'm interested in testing the performance of the Native
PostgreSQL port for Windows with the xlog directory being located on
another drive.
- Have tried doing normal shortcuts, and have also tried using the
cygwin "ln" command to create the appropriate soft link. Both
approaches create a shortcut object of the correct name pointing to the
correct place on the new drive, but the only thing that appears to
follow this shortcut is when I click on them using Windows Explorer.
The Native PostgreSQL port for Windows doesn't, and neither do a few
other applications I tested.Would it be correct to say that the 'ln' command in the MS Resource Kit
creates this kind of shortcut too, as the Reparse Points feature doesn't
seem to be possible under NT4?
I wouldn't assume that. It's been years since I tested it, but back then,
the command line and all program I used could see the link created by ln
that came with the resource kit. They were distinctly different from the
shortcut type of links, in that they seems transparent like short cuts in
unix generally are.
Do you have the resource kit or the gnu utils from it?
Looking at this url:
http://unxutils.sourceforge.net/
the part for ln.exe says it makes real hard links on ntfs (which means
they would be on the same drive.) So I'm not sure if ntfs supports soft
links across volumes transparently or not now.
scott.marlowe wrote:
Seems like the NT4 users are left out in the cold though until we add
some kind of ability for PostgreSQL to not look at the filesystem for
info about where to put the xlog files.This isn't true. With the resource kit, you get the gnu utils, and ln
works a charm under NT4 with ntfs. And not just for directories, but
files as well. Unless Microsoft somehow removed that functionality in the
intervening years since I've used NT. (wouldn't put it past them, but I
doubt they have.)
Yes, this is what I remember, that Cygwin had symlinks, and at that time
that was the only Win32 OS we supported. Now, with native Win32 port
coming, we have to figure out what is available.
--
Bruce Momjian | http://candle.pha.pa.us
pgman@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073
On Thu, 12 Sep 2002, Justin Clift wrote:
Am just wondering if we've ever considered adding a PGXLOG environment
variable that would point to the pg_xlog directory?
IMHO, a much better way to support this is to put this information into
the config file. That way it can't easily change when you happen to, say,
start postgres in the wrong window.
cjs
--
Curt Sampson <cjs@cynic.net> +81 90 7737 2974 http://www.netbsd.org
Don't you know, in this new Dark Age, we're all light. --XTC
scott.marlowe wrote:
On Fri, 13 Sep 2002, Justin Clift wrote:
Would it be correct to say that the 'ln' command in the MS Resource Kit
creates this kind of shortcut too, as the Reparse Points feature doesn't
seem to be possible under NT4?I wouldn't assume that. It's been years since I tested it, but back then,
the command line and all program I used could see the link created by ln
that came with the resource kit. They were distinctly different from the
shortcut type of links, in that they seems transparent like short cuts in
unix generally are.Do you have the resource kit or the gnu utils from it?
The situation appears to be this:
1. Soft links are available on NTFS 5 (2K/XP) as Reparse Points
via the DeviceIoControl() function for any application using the
standard C library routines.
2. Soft links are available on any filesystem under
95/98/ME/NT4/2K/XP as OLE streams (.lnk files) for Shell-aware
applications.
3. Hard links are available on NTFS 5 (2K/XP) via the
CreateHardLink() API.
See:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/createhardlink.asp
4. Hard links are available on NTFS (NT3.1/NT4) via the
BackupWrite() API by writing a special stream to the NTFS.
Example:
http://www.mvps.org/win32/ntfs/lnw.cpp
The cygwin implementation of link():
1. Will use CreateHardLink() if on 2K/XP
2. Will try to use the BackupWrite() method
3. Failing #2 will just copy the file
See how fun Microsoft makes things?
Mike Mascari
mascarm@mascari.com
I wrote:
scott.marlowe wrote:
I wouldn't assume that. It's been years since I tested it, but back
then, the command line and all program I used could see the link
created by ln that came with the resource kit. They were distinctly
different from the shortcut type of links, in that they seems
transparent like short cuts in unix generally are.Do you have the resource kit or the gnu utils from it?
The situation appears to be this:
1. Soft links are available on NTFS 5 (2K/XP) as Reparse Points via the
DeviceIoControl() function for any application using the standard C
library routines.2. Soft links are available on any filesystem under 95/98/ME/NT4/2K/XP
as OLE streams (.lnk files) for Shell-aware applications.3. Hard links are available on NTFS 5 (2K/XP) via the CreateHardLink() API.
<snip>
4. Hard links are available on NTFS (NT3.1/NT4) via the BackupWrite()
API by writing a special stream to the NTFS.
I also believe (I could be wrong) that for directories, the only
two methods of links are the Soft link methods above. So PGXLOG
cannot use soft links on a non-XP/2K machine unless it is
"Shell-Aware". For example, in a cygwin bash command window:
mkdir dir1
ln dir1 dir2 <- Error using Cygwin implementation
ln -s dir1 dir2 <- Creates a Shell short-cut (NT4)
echo "Hello" > dir1/test.txt
cat dir2/test.txt
"Hello" <- Cygwin's cat(bash?) is shell short-cut aware
Now, in a Windows NT command prompt:
notepad dir2\test.txt <- Notepad can't find file
notepad dir2.lnk <- Displays link contents
That means for a native port with a different PGXLOG directory
running on NT4, the only choice *using links* is to make the
native port shell short-cut aware.
I could be wrong but I don't think so.
Mike Mascari
mascarm@mascari.com
Curt Sampson <cjs@cynic.net> writes:
On Thu, 12 Sep 2002, Justin Clift wrote:
Am just wondering if we've ever considered adding a PGXLOG environment
variable that would point to the pg_xlog directory?
IMHO, a much better way to support this is to put this information into
the config file. That way it can't easily change when you happen to, say,
start postgres in the wrong window.
Yes. We rejected environment-variable-based xlog location for reasons
that apply equally well to Windows. The xlog location *must* be stored
in a physical file in the data directory; anything else is too unsafe.
The current technology for that is a symlink.
While it doesn't have to be a symlink as opposed to some sort of config
file, I don't have the slightest problem with saying that we don't
support relocation of xlog on older Windoid platforms.
regards, tom lane
Tom Lane wrote:
Curt Sampson <cjs@cynic.net> writes:
On Thu, 12 Sep 2002, Justin Clift wrote:
Am just wondering if we've ever considered adding a PGXLOG environment
variable that would point to the pg_xlog directory?IMHO, a much better way to support this is to put this information into
the config file. That way it can't easily change when you happen to, say,
start postgres in the wrong window.Yes. We rejected environment-variable-based xlog location for reasons
that apply equally well to Windows. The xlog location *must* be stored
in a physical file in the data directory; anything else is too unsafe.
The current technology for that is a symlink.While it doesn't have to be a symlink as opposed to some sort of config
file, I don't have the slightest problem with saying that we don't
support relocation of xlog on older Windoid platforms.
Agreed. Win 4.X is pretty dead. I added this thread to TODO.detail.
--
Bruce Momjian | http://candle.pha.pa.us
pgman@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073Bruce Momjian wrote:
Tom Lane wrote:
<snip>
While it doesn't have to be a symlink as opposed to some sort of config
file, I don't have the slightest problem with saying that we don't
support relocation of xlog on older Windoid platforms.Agreed. Win 4.X is pretty dead. I added this thread to TODO.detail.
Huh? You've got to be joking.
Many of the *really large* enterprises around (i.e. with 40k+ PC's, etc)
are still running WinNT 4, due to the migration issues with upgrading.
Aka, Too Many Things Break when they move to Win2k, etc.
Although MS no longer considers WinNT 4.0 to be a supported platform,
there are *lots* of big places still running it.
That's part of the reason some of the bigger corporates are looking for
MS alternatives.
Regards and best wishes,
Justin Clift
-- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania 19073
--
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi
Justin Clift wrote:
Bruce Momjian wrote:
Tom Lane wrote:
<snip>
While it doesn't have to be a symlink as opposed to some sort of config
file, I don't have the slightest problem with saying that we don't
support relocation of xlog on older Windoid platforms.Agreed. Win 4.X is pretty dead. I added this thread to TODO.detail.
Huh? You've got to be joking.
Many of the *really large* enterprises around (i.e. with 40k+ PC's, etc)
are still running WinNT 4, due to the migration issues with upgrading.
Aka, Too Many Things Break when they move to Win2k, etc.Although MS no longer considers WinNT 4.0 to be a supported platform,
there are *lots* of big places still running it.That's part of the reason some of the bigger corporates are looking for
MS alternatives.
Oh, that is bad news. Well, can we accept they will not be moving XLOG
around?
The problem with the non-symlink solution is that it is error-prone/ugly
on all the platforms, not just NT4.X.
--
Bruce Momjian | http://candle.pha.pa.us
pgman@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073Bruce Momjian wrote:
<snip>
Oh, that is bad news. Well, can we accept they will not be moving XLOG
around?The problem with the non-symlink solution is that it is error-prone/ugly
on all the platforms, not just NT4.X.
What you guys are saying isn't necessarily wrong, in that it may not
definitely be very pretty.
However, moving the WAL files to another disk has a significant
performance gain attached to it for loaded servers, so we how about we
take the viewpoint that if WinNT/2k/XP are to be supported then we might
as well let it do things properly instead of handicapping it?
Does anyone care to estimate what the coding time+issues involved would
be, for adding a parameter to the postgresql.conf file that allows
PostgreSQL to directly use a different directory path for the WAL files?
'wal_path'
or
'wal_directory'
or similar. In the postgresql.conf it would probably be placed in the
'Write-ahead log (WAL)' or 'Misc' sections.
No guarantees just yet but if it's not an extremely expensive thing to
add, then there might be people willing to pay for it (have a group in
mind already).
:-)
Regards and best wishes,
Justin Clift
-- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania 19073
--
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi
Justin Clift <justin@postgresql.org> writes:
However, moving the WAL files to another disk has a significant
performance gain attached to it for loaded servers, so we how about we
take the viewpoint that if WinNT/2k/XP are to be supported then we might
as well let it do things properly instead of handicapping it?
Considering that we do not yet have support for WinAnything except via
cygwin, this thread strikes me as mighty premature.
And, to be blunt, I'm not likely to go out of my way to improve support
for WinAnything even when we do have a native port. In words of one
syllable: WinAnything is not, and never will be, a preferred platform
for Postgres. Accordingly, performance improvements for it are just a
distraction from our real business; a distraction which plays into the
hands of Gates & Co. No thank you. I'm okay with providing minimal
support for those who really want to run toy databases on a toy
platform. I will *not* buy into trying to make it a non-toy platform.
regards, tom lane
On Sun, 15 Sep 2002, Bruce Momjian wrote:
The problem with the non-symlink solution is that it is error-prone/ugly
on all the platforms, not just NT4.X.
Actually, it's really just the environment variable solution that's
error prone, I think. Putting it in the config file is fine. It's
just a matter of someone coding it.
cjs
--
Curt Sampson <cjs@cynic.net> +81 90 7737 2974 http://www.netbsd.org
Don't you know, in this new Dark Age, we're all light. --XTC
Tom Lane wrote:
<snip>
And, to be blunt, I'm not likely to go out of my way to improve support
for WinAnything even when we do have a native port. In words of one
syllable: WinAnything is not, and never will be, a preferred platform
for Postgres. Accordingly, performance improvements for it are just a
distraction from our real business; a distraction which plays into the
hands of Gates & Co. No thank you. I'm okay with providing minimal
support for those who really want to run toy databases on a toy
platform. I will *not* buy into trying to make it a non-toy platform.
Understood, and that's ok.
Allowing PostgreSQL to be productively used as best it can be, whereever
it can be, makes sense doesn't it? Especially when the real target here
would be to give existing MS places a lower cost of entry to the
PostgreSQL world.
Financial example :
WinNT/2k/XP costs a few hundred dollars.
MS SQL Server costs a few thousand dollars.
Whenever we displace MS SQL Server, we divert more revenue away from MS
than if we just say "Sorry but we're not happy with making the Windows
port perform in ways that let it compete adequately with MS SQL Server".
We both know the arguments for and against. You're in the "against"
camp, and I'm in the "for" camp.
Personally I'm hoping there are some other PostgreSQL coders around in
the "for" camp too that can assist with this as we're beginning to gain
some good public enterprise level of interest.
:-)
Regards and best wishes,
Justin Clift
regards, tom lane
--
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi
