log_filename_prefix --> log_filename + strftime()

The patch is intended for 8.0.0 or later, and was generated and tested with
the cvs trunk as of 26-Aug-2004.

"Ed L." <pgsql@bluepolka.net> writes:

Attached is a patch which replaces the 'log_filename_prefix' configuration
directive with a similar 'log_filename' directive.
+ changes the default log filename to exclude the PID;

This would be better stated as "makes it impossible to use the PID
in the file name". While I'm prepared to grant that it may not be
necessary to do so in many scenarios, I'm not very happy with
arbitrarily removing the ability ... especially without giving any
justification.

regards, tom lane

On Friday August 27 2004 12:08, Tom Lane wrote:

"Ed L." <pgsql@bluepolka.net> writes:

Attached is a patch which replaces the 'log_filename_prefix'
configuration directive with a similar 'log_filename' directive.
+ changes the default log filename to exclude the PID;

This would be better stated as "makes it impossible to use the PID
in the file name". While I'm prepared to grant that it may not be
necessary to do so in many scenarios, I'm not very happy with
arbitrarily removing the ability ... especially without giving any
justification.

Yes, should have said more on that item. First, I didn't see how to easily
make it configurable in combination with strftime() without doing more
work, and it didn't appear to be worth the effort. By its addition,
hard-coding the PID into the filename deviates from what I would argue is
the de facto standard of Apache's rotatelogs and forces a naming convention
where none existed before. That creates work for us as we have a
considerable infrastructure setup to deal with logs; I suspect that may be
the case with others. I looked, but did not find, justification for why it
was introduced; I would assume it was added to allow for multiple
postmasters sharing the same log directory. I had difficulty fathoming the
usefulness of this being hard-coded, as it seems one could compensate
easily through the configurable 'log_filename' if one chose to share a log
directory among postmasters. Not by including the PID, but by some other
postmaster-unique naming approach. Given its a new 'feature', I'm hoping
it can be altered to return the freedom of filenaming to the administrator.

"Ed L." <pgsql@bluepolka.net> writes:

On Friday August 27 2004 12:08, Tom Lane wrote:

[ justification please ]

Yes, should have said more on that item. First, I didn't see how to easily
make it configurable in combination with strftime() without doing more
work, and it didn't appear to be worth the effort. By its addition,
hard-coding the PID into the filename deviates from what I would argue is
the de facto standard of Apache's rotatelogs and forces a naming convention
where none existed before. That creates work for us as we have a
considerable infrastructure setup to deal with logs; I suspect that may be
the case with others. I looked, but did not find, justification for why it
was introduced; I would assume it was added to allow for multiple
postmasters sharing the same log directory. I had difficulty fathoming the
usefulness of this being hard-coded, as it seems one could compensate
easily through the configurable 'log_filename' if one chose to share a log
directory among postmasters. Not by including the PID, but by some other
postmaster-unique naming approach. Given its a new 'feature', I'm hoping
it can be altered to return the freedom of filenaming to the administrator.

Or you could use different log_directory settings for different PMs.
Fair enough.

Anyone else have an opinion pro or con about this change? IMHO it's in
the gray area between bug fix and feature addition. If we want to do
it, though, doing it now is certainly better than holding it for 8.1,
since by then people would have gotten used to the present behavior.

BTW, as long as we are taking Apache as the de facto standard --- does
the default of "postgresql-%Y-%m-%d_%H%M%S.log" actually make sense, or
would something different be closer to the common practice with Apache?

regards, tom lane

On Friday August 27 2004 12:41, Tom Lane wrote:

BTW, as long as we are taking Apache as the de facto standard --- does
the default of "postgresql-%Y-%m-%d_%H%M%S.log" actually make sense, or
would something different be closer to the common practice with Apache?

Apache defaults to access_log.N where N is the epoch of the logfile start
time.

Ed

Tom Lane wrote:

"Ed L." <pgsql@bluepolka.net> writes:

On Friday August 27 2004 12:08, Tom Lane wrote:

[ justification please ]

Yes, should have said more on that item. First, I didn't see how to easily
make it configurable in combination with strftime() without doing more
work, and it didn't appear to be worth the effort. By its addition,
hard-coding the PID into the filename deviates from what I would argue is
the de facto standard of Apache's rotatelogs and forces a naming convention
where none existed before. That creates work for us as we have a
considerable infrastructure setup to deal with logs; I suspect that may be
the case with others. I looked, but did not find, justification for why it
was introduced; I would assume it was added to allow for multiple
postmasters sharing the same log directory. I had difficulty fathoming the
usefulness of this being hard-coded, as it seems one could compensate
easily through the configurable 'log_filename' if one chose to share a log
directory among postmasters. Not by including the PID, but by some other
postmaster-unique naming approach. Given its a new 'feature', I'm hoping
it can be altered to return the freedom of filenaming to the administrator.

Or you could use different log_directory settings for different PMs.
Fair enough.

Anyone else have an opinion pro or con about this change? IMHO it's in
the gray area between bug fix and feature addition. If we want to do
it, though, doing it now is certainly better than holding it for 8.1,
since by then people would have gotten used to the present behavior.

BTW, as long as we are taking Apache as the de facto standard --- does
the default of "postgresql-%Y-%m-%d_%H%M%S.log" actually make sense, or
would something different be closer to the common practice with Apache?

If the PID isn't there is there a danger of different postmasters
clobbering each other's logs? ISTM having the PID there gives some sort
of guarantee that that won't happen. I don't have any strong opinion one
way or another, but I wondered if a configurable strings with % escapes
like we use for log_line_prefix might be better. It could be argued to
be overkill, I guess. Alternatively, we could have a different boolean
option 'log_filename_use_pid' or some such.

cheers

andrew

On Friday August 27 2004 12:51, Ed L. wrote:

On Friday August 27 2004 12:41, Tom Lane wrote:

BTW, as long as we are taking Apache as the de facto standard --- does
the default of "postgresql-%Y-%m-%d_%H%M%S.log" actually make sense, or
would something different be closer to the common practice with Apache?

Apache defaults to access_log.N where N is the epoch of the logfile start
time.

I should say, Apache rotatelogs takes a configurable filename and then
appends ".N" where N is the logfile start time epoch. In one case, its
access_log.N, in another its error_log.N.

Ed

Andrew Dunstan <andrew@dunslane.net> writes:

If the PID isn't there is there a danger of different postmasters
clobbering each other's logs?

Only if they're logging into the same directory with the same filename
pattern. This isn't the default (the default log_directory is under
$PGDATA), and so I'm not that worried now that I think about it carefully.
I'm coming around to agree with Ed's position that it's not worth the
trouble to provide more than the strftime escapes.

regards, tom lane

"Ed L." <pgsql@bluepolka.net> writes:

On Friday August 27 2004 12:41, Tom Lane wrote:

BTW, as long as we are taking Apache as the de facto standard --- does
the default of "postgresql-%Y-%m-%d_%H%M%S.log" actually make sense, or
would something different be closer to the common practice with Apache?

I should say, Apache rotatelogs takes a configurable filename and then
appends ".N" where N is the logfile start time epoch. In one case, its
access_log.N, in another its error_log.N.

Hmm ... there isn't any way to emulate that with strftime escapes,
unless I missed the right one.

regards, tom lane

On Friday August 27 2004 1:03, Tom Lane wrote:

"Ed L." <pgsql@bluepolka.net> writes:

On Friday August 27 2004 12:41, Tom Lane wrote:

BTW, as long as we are taking Apache as the de facto standard --- does
the default of "postgresql-%Y-%m-%d_%H%M%S.log" actually make sense,
or would something different be closer to the common practice with
Apache?

I should say, Apache rotatelogs takes a configurable filename and then
appends ".N" where N is the logfile start time epoch. In one case, its
access_log.N, in another its error_log.N.

Hmm ... there isn't any way to emulate that with strftime escapes,
unless I missed the right one.

If you supply an escape, Apache will override that default epoch. So I
could see setting the default to "server_log" or "postgresql_log" or
whatever, and making the default (with no escapes supplied) be the epoch.
That would be easy tweak, and be much closer to Apache style.

Ed

Apache 1.3.31:

if (use_strftime) {
struct tm *tm_now;
tm_now = gmtime(&tLogStart);
strftime(buf2, sizeof(buf2), szLogRoot, tm_now);
}
else {
sprintf(buf2, "%s.%010d", szLogRoot, (int) tLogStart);
}

Tom Lane wrote:

"Ed L." <pgsql@bluepolka.net> writes:

Attached is a patch which replaces the 'log_filename_prefix' configuration
directive with a similar 'log_filename' directive.
+ changes the default log filename to exclude the PID;

This would be better stated as "makes it impossible to use the PID
in the file name". While I'm prepared to grant that it may not be
necessary to do so in many scenarios, I'm not very happy with
arbitrarily removing the ability ... especially without giving any
justification.

I don't have the time now to review the impact, but this might make
interpreting the log filename difficult or impossible, effectively
corrupting pg_logdir_ls.
I don't object against adjusting the timestamp format in a reasonable
way, but it should stay fixed; same about PID.

Regards,
Andreas

"Ed L." <pgsql@bluepolka.net> writes:

On Friday August 27 2004 1:03, Tom Lane wrote:

Hmm ... there isn't any way to emulate that with strftime escapes,
unless I missed the right one.

If you supply an escape, Apache will override that default epoch. So I
could see setting the default to "server_log" or "postgresql_log" or
whatever, and making the default (with no escapes supplied) be the epoch.
That would be easy tweak, and be much closer to Apache style.

Yeah, and it would also prevent a risk I now see with your initial
patch: if no %, it'll write the same filename each time, which
is almost certainly not desired. Works for me.

regards, tom lane

Tom Lane wrote:

"Ed L." <pgsql@bluepolka.net> writes:

On Friday August 27 2004 12:41, Tom Lane wrote:

BTW, as long as we are taking Apache as the de facto standard --- does
the default of "postgresql-%Y-%m-%d_%H%M%S.log" actually make sense, or
would something different be closer to the common practice with Apache?

I should say, Apache rotatelogs takes a configurable filename and then
appends ".N" where N is the logfile start time epoch. In one case, its
access_log.N, in another its error_log.N.

Hmm ... there isn't any way to emulate that with strftime escapes,
unless I missed the right one.

According to my Linux man page, the Olsen library has %s for that. I
don't see it in src/timezone/strftime.c, though

cheers

andrew

Andreas Pflug <pgadmin@pse-consulting.de> writes:

I don't have the time now to review the impact, but this might make
interpreting the log filename difficult or impossible, effectively
corrupting pg_logdir_ls.

So if you want to use that, you use a format that it can cope with.
I don't see a problem.

(This is probably an OK reason to keep the default log_filename in
Y/M/D/H/M/S style, though, rather than ".epoch" style.)

regards, tom lane

Tom Lane wrote:

Andreas Pflug <pgadmin@pse-consulting.de> writes:

I don't have the time now to review the impact, but this might make
interpreting the log filename difficult or impossible, effectively
corrupting pg_logdir_ls.

So if you want to use that, you use a format that it can cope with.
I don't see a problem.

(This is probably an OK reason to keep the default log_filename in
Y/M/D/H/M/S style, though, rather than ".epoch" style.)

Ah, so we keep the existing format but drop the pid, and just make it
changable by the user, and we rename it. Doesn't sound as drastic as it
first did.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Ah, so we keep the existing format but drop the pid, and just make it
changable by the user, and we rename it. Doesn't sound as drastic as it
first did.

Yeah, the only change in default behavior would be to drop the PID part
of the log filename, which doesn't seem too bad, since people aren't yet
depending on that.

regards, tom lane

On Friday August 27 2004 1:39, Tom Lane wrote:

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Ah, so we keep the existing format but drop the pid, and just make it
changable by the user, and we rename it. Doesn't sound as drastic as
it first did.

Yeah, the only change in default behavior would be to drop the PID part
of the log filename, which doesn't seem too bad, since people aren't yet
depending on that.

regards, tom lane

OK, if I read you correctly...

Default remains "postgresql-%Y-%m-%d_%H%M%S.log"

(Apache style: access_log.%s)

If log_filename = 'xxx', rotate with strftime() to 'xxx-%Y-%m-%d_%H%M%S'

(Apache style: xxx.%s)

If log_filename = 'xxx.%a', rotate with strftime() to 'xxx.%a'

(Apache style: xxx.%a)

Not a big fan of the verbose 32-character default name, 'server_log.%s'
would be my pick, but easy enough to override it.

Ed

"Ed L." <pgsql@bluepolka.net> writes:

If log_filename = 'xxx', rotate with strftime() to 'xxx-%Y-%m-%d_%H%M%S'

No, I was thinking that if no %'s in the log_filename, then use xxx.EPOCH
to provide Apache compatibility.

regards, tom lane

On Friday August 27 2004 2:15, Tom Lane wrote:

"Ed L." <pgsql@bluepolka.net> writes:

If log_filename = 'xxx', rotate with strftime() to
'xxx-%Y-%m-%d_%H%M%S'

No, I was thinking that if no %'s in the log_filename, then use xxx.EPOCH
to provide Apache compatibility.

OK, that works for me.

One addition I'd like to include with the revised patch: a boolean
postgresql.conf option ('log_truncate_on_rotation', default false) to
truncate any existing log file by the same name. Default behavior here and
with Apache is to always append, but it's a useful feature for us because
it largely eliminates the issue of logs filling up the disk. You don't
want the log clobbered on restarts, so the idea is to only truncate during
time/size-based rotation, not on the initial open. Thoughts?

Ed