Why isn't stats_temp_directory automatically created?

Fujii Masao escreveu:

Is it worth making the patch which creates stats_temp_directory
if not present?

+1.

--
Euler Taveira de Oliveira
http://www.timbira.com/

Euler Taveira de Oliveira <euler@timbira.com> wrote:

Fujii Masao escreveu:

Is it worth making the patch which creates stats_temp_directory
if not present?

+1.

+1, but AFAIK stats_temp_directory was designed to symlink to a RAM drive.
Even if stats_temp_directory exists as a symbolic link, the destination
directory might be lost in such a situation. If you try to make servers
more robust, you might also need to consider broken symlinks.

Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center

Hi,

On Tue, Apr 14, 2009 at 10:26 PM, Euler Taveira de Oliveira
<euler@timbira.com> wrote:

Fujii Masao escreveu:

Is it worth making the patch which creates stats_temp_directory
if not present?

+1.

Here is the patch.

This patch should be added to CommitFest-2009-First?,
or committed before 8.4 release? The patch is very small,
so I don't think that it'll block 8.4 release.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Fujii Masao wrote:

Hi,

On Tue, Apr 14, 2009 at 10:26 PM, Euler Taveira de Oliveira
<euler@timbira.com> wrote:

Fujii Masao escreveu:

Is it worth making the patch which creates stats_temp_directory
if not present?

+1.

Here is the patch.

This patch should be added to CommitFest-2009-First?,
or committed before 8.4 release? The patch is very small,
so I don't think that it'll block 8.4 release.

I think the fix should go into 8.4 - this is a fix for a new feature.
However, a couple of comments:

This does not take into account the effect of symlinks as mentioned by
Itakagi Takahiro. I haven't looked at the details, but I don't think it
would be that much more work to deal with it - and as he mentions, this
is a very common usecase.

Also, wouldn't it be better to isolate this to the first time when we
try to create the file - then we don't have to export the symbol?

//Magnus

Hi,

On Wed, Apr 15, 2009 at 5:37 PM, Magnus Hagander <magnus@hagander.net> wrote:

This does not take into account the effect of symlinks as mentioned by
Itakagi Takahiro. I haven't looked at the details, but I don't think it
would be that much more work to deal with it - and as he mentions, this
is a very common usecase.

Okey, I'll revise the patch; create also the directory which is
referenced by symlink if not present.

Also, wouldn't it be better to isolate this to the first time when we
try to create the file - then we don't have to export the symbol?

You mean having assign_pgstat_temp_directory() create the
directory instead of pgstat_start()? In this case, the directory is
created automatically not only at the beginning but also when
a configuration file is reloaded. This seems to be better behavior.

One problem of this is that some directories may be created
unexpectedly if stats_temp_directory is specified at multiple
locations. It's because assign_hook is called for each setting.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Fujii Masao wrote:

Hi,

Hi!

Sorry about the very late response - I've been out of the country and
generally busy.

On Wed, Apr 15, 2009 at 5:37 PM, Magnus Hagander <magnus@hagander.net> wrote:

This does not take into account the effect of symlinks as mentioned by
Itakagi Takahiro. I haven't looked at the details, but I don't think it
would be that much more work to deal with it - and as he mentions, this
is a very common usecase.

Okey, I'll revise the patch; create also the directory which is
referenced by symlink if not present.

Great.

Also, wouldn't it be better to isolate this to the first time when we
try to create the file - then we don't have to export the symbol?

You mean having assign_pgstat_temp_directory() create the
directory instead of pgstat_start()? In this case, the directory is
created automatically not only at the beginning but also when
a configuration file is reloaded. This seems to be better behavior.

No, I meant creating it when we open the file - in pgstat_write_statsfile().

//Magnus

Hi,

On Mon, Apr 20, 2009 at 1:29 AM, Magnus Hagander <magnus@hagander.net> wrote:

Sorry about the very late response - I've been out of the country and
generally busy.

Thanks for taking the time to comment!

On Wed, Apr 15, 2009 at 5:37 PM, Magnus Hagander <magnus@hagander.net> wrote:

This does not take into account the effect of symlinks as mentioned by
Itakagi Takahiro. I haven't looked at the details, but I don't think it
would be that much more work to deal with it - and as he mentions, this
is a very common usecase.

Okey, I'll revise the patch; create also the directory which is
referenced by symlink if not present.

Great.

Here is the revised patch; If stats_temp_directory indicates the symlink,
we pursue the chain of symlinks and create the referenced directory.

Also, wouldn't it be better to isolate this to the first time when we
try to create the file - then we don't have to export the symbol?

You mean having assign_pgstat_temp_directory() create the
directory instead of pgstat_start()? In this case, the directory is
created automatically not only at the beginning but also when
a configuration file is reloaded. This seems to be better behavior.

No, I meant creating it when we open the file - in pgstat_write_statsfile().

OK, I changed the patch so. Thanks.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Hi,

On Tue, Apr 21, 2009 at 4:33 PM, Fujii Masao <masao.fujii@gmail.com> wrote:

Here is the revised patch; If stats_temp_directory indicates the symlink,
we pursue the chain of symlinks and create the referenced directory.

BTW, this patch is useful also as the foundation for improving
creation of log_directory. Attached patch fixes the following
problem of log_directory by using that fundamental patch.

- log_directory is not created when a configuration file is reloaded
- creation of log_directory fails if the parent directory of it doesn't exist
- if log_directory indicates the symlink, it's not resolved

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Fujii Masao <masao.fujii@gmail.com> writes:

Here is the revised patch; If stats_temp_directory indicates the symlink,
we pursue the chain of symlinks and create the referenced directory.

I looked at this patch a bit. I'm still entirely unconvinced that we
should be doing this at all --- if the directory is not there, there's
a significant probability that there's something wrong that is beyond
the backend's ability to understand or correct. However, even ignoring
that objection, the patch is not ready to commit for a number of
reasons:

* Repeating the operation every time the stats file is written doesn't
seem like a particularly good idea; it eats cycles, and if the directory
disappears during live operation then there is *definitely* something
fishy going on. Can't we fix it so that the work is only done when the
path setting changes? (In principle you could do it in
assign_pgstat_temp_directory(), but I think something would be needed to
ensure that only the stats collector process actually tries to create
the directory. Or maybe it would be simplest to try to run the code only
when we get a failure from trying to create the stats temp file.)

* I don't think the mkdir_p code belongs in fd.c. It looks like
you copied-and-pasted it from initdb.c, which isn't any good either;
we don't want to maintain multiple copies of this. Maybe a new
src/port/ file is indicated.

* elog(LOG) is not exactly an adequate response if the final chdir fails
--- you have just broken the process beyond recovery.  That alone may be
sufficient reason to reject the attempt to deal with symlinks.  As far
as pgstat_temp_directory is concerned, I'm not sure of the point of
making the GUC point to a symlink anyway --- if you have a GUC why not
just point it where you want the directory to be?

regards, tom lane

Tom Lane wrote:

Fujii Masao <masao.fujii@gmail.com> writes:

Here is the revised patch; If stats_temp_directory indicates the symlink,
we pursue the chain of symlinks and create the referenced directory.

I looked at this patch a bit. I'm still entirely unconvinced that we
should be doing this at all --- if the directory is not there, there's
a significant probability that there's something wrong that is beyond
the backend's ability to understand or correct. However, even ignoring
that objection, the patch is not ready to commit for a number of
reasons:

* Repeating the operation every time the stats file is written doesn't
seem like a particularly good idea; it eats cycles, and if the directory
disappears during live operation then there is *definitely* something
fishy going on. Can't we fix it so that the work is only done when the
path setting changes? (In principle you could do it in
assign_pgstat_temp_directory(), but I think something would be needed to
ensure that only the stats collector process actually tries to create
the directory. Or maybe it would be simplest to try to run the code only
when we get a failure from trying to create the stats temp file.)

My idea was to have it run when it tries to create the temp file and
fails. Seems simpler than in the assign hook.

* I don't think the mkdir_p code belongs in fd.c. It looks like
you copied-and-pasted it from initdb.c, which isn't any good either;
we don't want to maintain multiple copies of this. Maybe a new
src/port/ file is indicated.

Yes, that's what I thought as well. How about src/port/dirmod.c though -
it has rename/unlink/symlink now which seem at least remotely related.

//Magnus

I assume we decided we didn't want this.

---------------------------------------------------------------------------

Tom Lane wrote:

Fujii Masao <masao.fujii@gmail.com> writes:

Here is the revised patch; If stats_temp_directory indicates the symlink,
we pursue the chain of symlinks and create the referenced directory.

I looked at this patch a bit. I'm still entirely unconvinced that we
should be doing this at all --- if the directory is not there, there's
a significant probability that there's something wrong that is beyond
the backend's ability to understand or correct. However, even ignoring
that objection, the patch is not ready to commit for a number of
reasons:

* Repeating the operation every time the stats file is written doesn't
seem like a particularly good idea; it eats cycles, and if the directory
disappears during live operation then there is *definitely* something
fishy going on. Can't we fix it so that the work is only done when the
path setting changes? (In principle you could do it in
assign_pgstat_temp_directory(), but I think something would be needed to
ensure that only the stats collector process actually tries to create
the directory. Or maybe it would be simplest to try to run the code only
when we get a failure from trying to create the stats temp file.)

* I don't think the mkdir_p code belongs in fd.c. It looks like
you copied-and-pasted it from initdb.c, which isn't any good either;
we don't want to maintain multiple copies of this. Maybe a new
src/port/ file is indicated.

* elog(LOG) is not exactly an adequate response if the final chdir fails
--- you have just broken the process beyond recovery.  That alone may be
sufficient reason to reject the attempt to deal with symlinks.  As far
as pgstat_temp_directory is concerned, I'm not sure of the point of
making the GUC point to a symlink anyway --- if you have a GUC why not
just point it where you want the directory to be?

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com
  PG East:  http://www.enterprisedb.com/community/nav-pg-east-2010.do
  + If your life is a hard drive, Christ can be your backup. +

Bruce Momjian <bruce@momjian.us> writes:

I assume we decided we didn't want this.

I thought the risk/reward ratio was pretty bad.

regards, tom lane

Tom Lane wrote:

Bruce Momjian <bruce@momjian.us> writes:

I assume we decided we didn't want this.

I thought the risk/reward ratio was pretty bad.

Yea, the symlink issue killed it for me.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com
  PG East:  http://www.enterprisedb.com/community/nav-pg-east-2010.do
  + If your life is a hard drive, Christ can be your backup. +