pg_upgrade --link on Windows

On Fri, Jun 9, 2017 at 12:00:56PM +0200, Arnaud L. wrote:

Hi

The pg_upgrade documentation for PostgreSQL 9.6 states that --link will use
junction points on Windows.
Shouldn't it rather user hard-links ?
If I'm not mistaken, with junction points (i.e. soft-links to directories),
the old data dir cannot be removed.
With hard-links to file, we can get rid of the old data dir once we are sure
that the upgrade is fine.

I was told junction points on Windows were hard links and no one has
ever complained about not being able to remove them.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On 06/09/2017 07:07 AM, Bruce Momjian wrote:

On Fri, Jun 9, 2017 at 12:00:56PM +0200, Arnaud L. wrote:

Hi

The pg_upgrade documentation for PostgreSQL 9.6 states that --link will use
junction points on Windows.
Shouldn't it rather user hard-links ?
If I'm not mistaken, with junction points (i.e. soft-links to directories),
the old data dir cannot be removed.
With hard-links to file, we can get rid of the old data dir once we are sure
that the upgrade is fine.

I was told junction points on Windows were hard links and no one has
ever complained about not being able to remove them.

https://msdn.microsoft.com/en-us/library/windows/desktop/aa365006(v=vs.85).aspx

Seems to me the difference is hard links point to file, junctions to
directories.

So if I am following:

https://en.wikipedia.org/wiki/NTFS_junction_point#Creating_or_deleting_a_junction_point

You remove the junction and then the directory it points to.

--
Adrian Klaver
adrian.klaver@aklaver.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Le 9/06/2017 à 16:07, Bruce Momjian a écrit :

I was told junction points on Windows were hard links and no one has
ever complained about not being able to remove them.

Sorry, I think my explanation was not very clear.
You can remove the link, but the point is to remove the target (i.e. the
old-data-dir).
You can do this with a hard link (there still exists a hardlink pointing
to the inode so it remains), but with a soft link you end up with a link
to nothing.
Deleting a junction target in Windows will work, but you'll have an
error trying to access the junction directory (directory not found).

See this page for more details :
http://cects.com/overview-to-understanding-hard-links-junction-points-and-symbolic-links-in-windows/

Under "Hard Link (Linking for individual files)" :
"If the target is deleted, its content is still available through the
hard link"

Junction Point (Directory Hard Link):
"If the target is moved, renamed or deleted, the Junction Point still
exists, but points to a non-existing directory"

BUT, when I try to "pg_upgrade --link --check" with old-data-dir and
new-data-dir on different volumes, I get an error saying that both
directories must be on the same volume if --link is used.
So maybe pg_upgrade uses hard-links (i.e. to files), and only the
documentation is wrong by calling them junctions (i.e. soft links to
files) ?

Regards
--
Arnaud

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On 06/09/2017 07:39 AM, Arnaud L. wrote:

Le 9/06/2017 à 16:07, Bruce Momjian a écrit :

I was told junction points on Windows were hard links and no one has
ever complained about not being able to remove them.

Sorry, I think my explanation was not very clear.
You can remove the link, but the point is to remove the target (i.e. the
old-data-dir).
You can do this with a hard link (there still exists a hardlink pointing
to the inode so it remains), but with a soft link you end up with a link
to nothing.
Deleting a junction target in Windows will work, but you'll have an
error trying to access the junction directory (directory not found).

See this page for more details :
http://cects.com/overview-to-understanding-hard-links-junction-points-and-symbolic-links-in-windows/

Under "Hard Link (Linking for individual files)" :
"If the target is deleted, its content is still available through the
hard link"

Junction Point (Directory Hard Link):
"If the target is moved, renamed or deleted, the Junction Point still
exists, but points to a non-existing directory"

BUT, when I try to "pg_upgrade --link --check" with old-data-dir and
new-data-dir on different volumes, I get an error saying that both
directories must be on the same volume if --link is used.
So maybe pg_upgrade uses hard-links (i.e. to files), and only the
documentation is wrong by calling them junctions (i.e. soft links to
files) ?

Looks that way. In file.c in ~/src/bin/pg_upgrade I see:

#ifdef WIN32
300 /* implementation of pg_link_file() on Windows */
301 static int
302 win32_pghardlink(const char *src, const char *dst)
303 {
304 /*
305 * CreateHardLinkA returns zero for failure
306 * http://msdn.microsoft.com/en-us/library/aa363860(VS.85).aspx
307 */
308 if (CreateHardLinkA(dst, src, NULL) == 0)
309 {
310 _dosmaperr(GetLastError());
311 return -1;
312 }
313 else
314 return 0;
315 }
316 #endif

Regards
--
Arnaud

--
Adrian Klaver
adrian.klaver@aklaver.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Le 9/06/2017 à 16:55, Adrian Klaver a écrit :

On 06/09/2017 07:39 AM, Arnaud L. wrote:

So maybe pg_upgrade uses hard-links (i.e. to files), and only the
documentation is wrong by calling them junctions (i.e. soft links to
files) ?

Looks that way. In file.c in ~/src/bin/pg_upgrade I see:

#ifdef WIN32
300 /* implementation of pg_link_file() on Windows */
301 static int
302 win32_pghardlink(const char *src, const char *dst)
303 {
304 /*
305 * CreateHardLinkA returns zero for failure
306 * http://msdn.microsoft.com/en-us/library/aa363860(VS.85).aspx
307 */
308 if (CreateHardLinkA(dst, src, NULL) == 0)
309 {
310 _dosmaperr(GetLastError());
311 return -1;
312 }
313 else
314 return 0;
315 }
316 #endif

Great !
So I did a full upgrade for nothing (just for safety), but that's good
to know for next time !
Should this be submitted to postgresql-bugs, or is there something more
specific to the documentation ?

Regards
--
Arnaud

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Le 9/06/2017 à 17:02, Arnaud L. a écrit :

Le 9/06/2017 à 16:55, Adrian Klaver a écrit :

On 06/09/2017 07:39 AM, Arnaud L. wrote:

So maybe pg_upgrade uses hard-links (i.e. to files), and only the
documentation is wrong by calling them junctions (i.e. soft links to
files) ?

Looks that way. In file.c in ~/src/bin/pg_upgrade I see:

#ifdef WIN32
300 /* implementation of pg_link_file() on Windows */
301 static int
302 win32_pghardlink(const char *src, const char *dst)
303 {
304 /*
305 * CreateHardLinkA returns zero for failure
306 * http://msdn.microsoft.com/en-us/library/aa363860(VS.85).aspx
307 */
308 if (CreateHardLinkA(dst, src, NULL) == 0)
309 {
310 _dosmaperr(GetLastError());
311 return -1;
312 }
313 else
314 return 0;
315 }
316 #endif

Great !
So I did a full upgrade for nothing (just for safety), but that's good
to know for next time !
Should this be submitted to postgresql-bugs, or is there something more
specific to the documentation ?

I just found the pgsql-docs@postgresql.org list, for for the spam.

--
Arnaud

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On Fri, 9 Jun 2017 10:07:24 -0400, Bruce Momjian <bruce@momjian.us>
wrote:

On Fri, Jun 9, 2017 at 12:00:56PM +0200, Arnaud L. wrote:

Hi

The pg_upgrade documentation for PostgreSQL 9.6 states that --link will use
junction points on Windows.
Shouldn't it rather user hard-links ?
If I'm not mistaken, with junction points (i.e. soft-links to directories),
the old data dir cannot be removed.
With hard-links to file, we can get rid of the old data dir once we are sure
that the upgrade is fine.

I was told junction points on Windows were hard links and no one has
ever complained about not being able to remove them.

NTFS junctions are a distinct type of symbolic link which is meant for
filesystem mount points. In NTFS "normal" symlinks are restricted to
targets within the same filesystem.

You can use a junction anywhere you want a symlink, but not the
reverse. The downside is that pathname parsing is slower with
junctions than with symlinks because of the possibility that the path
may cross into a different filesystem.

The documentation is not very clear, IMO.

https://msdn.microsoft.com/en-us/library/windows/desktop/aa365006(v=vs.85).aspx
https://msdn.microsoft.com/en-us/library/windows/desktop/aa363878(v=vs.85).aspx
https://msdn.microsoft.com/en-us/library/windows/desktop/aa365503(v=vs.85).aspx

The mklink utility can create any of these types of links. Its
documentation does not describe the differences, but is shows that
hard links, symlinks, and junctions all are distinct concepts in
Windows.

https://technet.microsoft.com/en-us/library/cc753194(v=ws.11).aspx

George

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On Fri, 9 Jun 2017 07:24:03 -0700, Adrian Klaver
<adrian.klaver@aklaver.com> wrote:

https://msdn.microsoft.com/en-us/library/windows/desktop/aa365006(v=vs.85).aspx

Seems to me the difference is hard links point to file, junctions to
directories.

You can make either hard links or symlinks to files. Junctions are
distinct from normal symlinks in that junctions can cross filesystems.
Microsoft's cmdline tools complain if you try to make a junction to a
file, because Microsoft intended junctions for mount points ... but
you can do it programmatically, or trick the tool by creating the link
and then replacing the target, and in most cases it will work the same
as a normal symlink.

I have seen cases where a junction to a file didn't work, but they
seemed to be application related rather than an OS issue. Prior to
Vista, the mklink utility was not available, so people wanting to
create symlinks were forced to use the sysinternals junction utility.
https://technet.microsoft.com/en-us/sysinternals/bb545021.aspx

George

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On Fri, Jun 9, 2017 at 07:55:55AM -0700, Adrian Klaver wrote:

On 06/09/2017 07:39 AM, Arnaud L. wrote:

See this page for more details :
http://cects.com/overview-to-understanding-hard-links-junction-points-and-symbolic-links-in-windows/

Under "Hard Link (Linking for individual files)" :
"If the target is deleted, its content is still available through the hard
link"

Junction Point (Directory Hard Link):
"If the target is moved, renamed or deleted, the Junction Point still
exists, but points to a non-existing directory"

BUT, when I try to "pg_upgrade --link --check" with old-data-dir and
new-data-dir on different volumes, I get an error saying that both
directories must be on the same volume if --link is used.
So maybe pg_upgrade uses hard-links (i.e. to files), and only the
documentation is wrong by calling them junctions (i.e. soft links to
files) ?

Looks that way. In file.c in ~/src/bin/pg_upgrade I see:

#ifdef WIN32
300 /* implementation of pg_link_file() on Windows */
301 static int
302 win32_pghardlink(const char *src, const char *dst)
303 {
304 /*
305 * CreateHardLinkA returns zero for failure
306 * http://msdn.microsoft.com/en-us/library/aa363860(VS.85).aspx
307 */
308 if (CreateHardLinkA(dst, src, NULL) == 0)
309 {
310 _dosmaperr(GetLastError());
311 return -1;
312 }
313 else
314 return 0;
315 }
316 #endif

[docs list added]

I apologize for not being smarter on this thread. When I helped with
the Windows port, I was told Windows didn't have hard links for use by
tablespace directories, so I got it into my head that Windows didn't
have hard links. Therefore, when I was writing the docs, I called them
junction points.

Looking back to Postgres 9.0 where pg_upgrade was added to the tree, I
see that the code even at that time used hard links on Windows. I have
created the attached patch which I will apply to all current Postgres
versions to fix this error.

Thanks for the report and the research. :-)

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

-----Ursprüngliche Nachricht-----

I apologize for not being smarter on this thread. When I helped with the
Windows port, I was told Windows didn't have hard links for use by

tablespace

directories, so I got it into my head that Windows didn't have hard links.
Therefore, when I was writing the docs, I called them junction points.

It's actually not "Windows" providing hard links, it is the file system
NTFS. FAT and its modern cousins don't provide hard links - but this will
rarely be used for databases these days.
However, ReFS (introduced with server 2012 and providing some new features
like automatic integrity checks, clustering etc.) does no longer provide
hard links.
Are hard links used anywhere else but in pg_upgrade?

Klaus

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On Wed, Jun 14, 2017 at 09:59:04AM +0200, Klaus P. Pieper wrote:

-----Urspr�ngliche Nachricht-----

I apologize for not being smarter on this thread. When I helped with the
Windows port, I was told Windows didn't have hard links for use by

tablespace

directories, so I got it into my head that Windows didn't have hard links.
Therefore, when I was writing the docs, I called them junction points.

It's actually not "Windows" providing hard links, it is the file system
NTFS. FAT and its modern cousins don't provide hard links - but this will
rarely be used for databases these days.
However, ReFS (introduced with server 2012 and providing some new features
like automatic integrity checks, clustering etc.) does no longer provide
hard links.
Are hard links used anywhere else but in pg_upgrade?

Nope, it is used only by transfer_relfile() calling linkFile().

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On Tue, Jun 13, 2017 at 04:07:48PM -0400, Bruce Momjian wrote:

On Fri, Jun 9, 2017 at 07:55:55AM -0700, Adrian Klaver wrote:
I apologize for not being smarter on this thread. When I helped with
the Windows port, I was told Windows didn't have hard links for use by
tablespace directories, so I got it into my head that Windows didn't
have hard links. Therefore, when I was writing the docs, I called them
junction points.

Looking back to Postgres 9.0 where pg_upgrade was added to the tree, I
see that the code even at that time used hard links on Windows. I have
created the attached patch which I will apply to all current Postgres
versions to fix this error.

Thanks for the report and the research. :-)

Patch applied all the way back to 9.3, where the junction point mention
first appeared.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

--
Sent via pgsql-docs mailing list (pgsql-docs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-docs