more contrib: log rotator

Andrew Sullivan <andrew@libertyrms.info> writes:

Is anyone interested in having pglog-rotator?

FWIW, I saw an early version of pglog-rotator about a year and a half
ago (while consulting for LibertyRMS), and thought at the time that
it was pretty cool. So I'm for including it ... maybe even as
mainstream instead of contrib.

regards, tom lane

Would the plan be to add it to pg_ctl?

"Jim Buttafuoco" <jim@contactbda.com> writes:

Would the plan be to add it to pg_ctl?

You would not actually have to: you could just pipe pg_ctl's output to
pglog-rotator. But I think it'd be cool if pg_ctl had an option to use
pglog-rotator, or maybe even adopt it as standard behavior.

I think we would have to make the rotator script be mainstream rather
than contrib if we wanted pg_ctl to use it directly. That was why I was
thinking maybe mainstream ...

Andrew, could you toss up the script on pgsql-patches just so people can
take a look? Then we could think more about where to go with it.

regards, tom lane

Does this log rotator do something that apache's doesn't?

Dave
On Thu, 2003-04-03 at 13:41, Tom Lane wrote:

"Jim Buttafuoco" <jim@contactbda.com> writes:

Would the plan be to add it to pg_ctl?

You would not actually have to: you could just pipe pg_ctl's output to
pglog-rotator. But I think it'd be cool if pg_ctl had an option to use
pglog-rotator, or maybe even adopt it as standard behavior.

I think we would have to make the rotator script be mainstream rather
than contrib if we wanted pg_ctl to use it directly. That was why I was
thinking maybe mainstream ...

Andrew, could you toss up the script on pgsql-patches just so people can
take a look? Then we could think more about where to go with it.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org

--
Dave Cramer <dave@fastcrypt.com>
Cramer Consulting

On Thu, Apr 03, 2003 at 01:41:08PM -0500, Tom Lane wrote:

You would not actually have to: you could just pipe pg_ctl's output to
pglog-rotator. But I think it'd be cool if pg_ctl had an option to use
pglog-rotator, or maybe even adopt it as standard behavior.

It's currently built to call a program, and read its stdout and
stderr, rather than acting as a pipe. I guess it shouldn't be too
hard to modify, though. We actually call the postmaster directly
with it, so we use it as a replacement for pg_ctl at startup.

Andrew, could you toss up the script on pgsql-patches just so people can
take a look? Then we could think more about where to go with it.

Ok, I sent it.

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110

On Thu, Apr 03, 2003 at 02:12:03PM -0500, Dave Cramer wrote:

Does this log rotator do something that apache's doesn't?

Probably not. This was just easier for us.

A little information might be handy here: we run postgres nder a
hosted environment, and we do not have root on the relevant boxes.
So installing anything even a little complicated means building
everything ourselves. As a result, we end up re-creating plenty of
functionality just to make it easy to install.

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110

On Thu, Apr 03, 2003 at 01:41:08PM -0500, Tom Lane wrote:

Andrew, could you toss up the script on pgsql-patches just so people can
take a look? Then we could think more about where to go with it.

Ok, the first try failed (of course) because I wasn't subscribed.
Should be there now, though.

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110

Andrew Sullivan writes:

Is anyone interested in having pglog-rotator?

What would get me a whole lot more excited is if the server could write
directly to a file and do its own rotating (or at least reopening of
files).

Considering that your rotator is tailored to a rather specific setup, it
doesn't do anything better compared to established ones, it prevents the
use of pg_ctl, it's written in Perl, and it doesn't do anything for
Windows users, I think it's not suitable for a general audience.

--
Peter Eisentraut peter_e@gmx.net

On Fri, 4 Apr 2003, Peter Eisentraut wrote:

Andrew Sullivan writes:

Is anyone interested in having pglog-rotator?

What would get me a whole lot more excited is if the server could write
directly to a file and do its own rotating (or at least reopening of
files).

Considering that your rotator is tailored to a rather specific setup, it
doesn't do anything better compared to established ones, it prevents the
use of pg_ctl, it's written in Perl, and it doesn't do anything for
Windows users, I think it's not suitable for a general audience.

That said, a log rotation capability built right into pg_ctl or
thereabouts would be a very nice feature. I.e. 'pg_ctl -r 86400 -l
$PGDATA/logs/pgsql start'

where -r is the rotation period in seconds. If it's an external program
that pg_ctl calls that's fine, and it could even just be a carbon copy of
apache's log rotater if their license is compatible (isn't it?)

On Fri, Apr 04, 2003 at 09:16:39AM -0700, scott.marlowe wrote:

where -r is the rotation period in seconds. If it's an external program

Ours rotates based on size rather than time. I can see some
advantages to the time-based approach, but if you have wide
variations in traffic, you run the risk of rotating over useful files
with more or less empty ones if you use it.

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110

On Fri, Apr 04, 2003 at 05:13:13PM +0200, Peter Eisentraut wrote:

use of pg_ctl, it's written in Perl, and it doesn't do anything for
Windows users, I think it's not suitable for a general audience.

It doesn't prevent the use of pg_ctl, although it does indeed prevent
the use of pg_ctl for startup.

I'm not sufficiently familiar with Windows to know how this does or
does not help them. Could you elaborate? And what's wrong with
Perl?

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110

On Friday April 4 2003 9:16, scott.marlowe wrote:

That said, a log rotation capability built right into pg_ctl or
thereabouts would be a very nice feature. I.e. 'pg_ctl -r 86400 -l
$PGDATA/logs/pgsql start'

where -r is the rotation period in seconds. If it's an external program
that pg_ctl calls that's fine, and it could even just be a carbon copy of
apache's log rotater if their license is compatible (isn't it?)

By way of feature ideas, one very convenient but not widely used feature of
Apache's log rotator is the ability to specify a strftime() format string
for the file extension. For example, if I want to have my logs rollover
every 24 hours and be named log.Mon, log.Tue, log.Wed, I say something like

pg_ctl start | rotatelogs 86400 "%a"

This causes the logs to overwrite themselves every seven days, taking log
maintenance time to very near zero. We also customized our use of it to
allow us to automatically move existing logs out of the way to "log.1",
"log.2", or to simply overwrite existing logs.

Ed

On Friday April 4 2003 10:04, Ed L. wrote:

By way of feature ideas, one very convenient but not widely used feature
of Apache's log rotator is the ability to specify a strftime() format
string for the file extension. For example, if I want to have my logs
rollover every 24 hours and be named log.Mon, log.Tue, log.Wed, I say
something like

pg_ctl start | rotatelogs 86400 "%a"

More accurately, something like this:

pg_ctl start | rotatelogs 86400 "log.%a"

Ed

On Fri, 4 Apr 2003, Andrew Sullivan wrote:

On Fri, Apr 04, 2003 at 09:16:39AM -0700, scott.marlowe wrote:

where -r is the rotation period in seconds. If it's an external program

Ours rotates based on size rather than time. I can see some
advantages to the time-based approach, but if you have wide
variations in traffic, you run the risk of rotating over useful files
with more or less empty ones if you use it.

I would want time based for sure, and I can see the use for size based
splitting as well. I wouldn't be hard to have it do both would it?

I just like the idea of it being one of the dozens or so options for
pg_ctl so it's painless to use for joe six pack.

pg_ctl -r 86400 -l $PGDATA/logs/pgsql

where -r is the rotation period

OR

pg_ctl -f 10M -l $PGDATA/logs/pgsql

where -f is the max file size of a log

I'd recommend that the nameing convnention should probably
be:

filenamespec.timestamp, like: $PGDATA/logs/pgsql.1049414400

for time rotated logs, and

filename.incnumber like: $PGDATa/logs/pgsql.0000000001

Peter Eisentraut wrote:

Andrew Sullivan writes:

Is anyone interested in having pglog-rotator?

What would get me a whole lot more excited is if the server could write
directly to a file and do its own rotating (or at least reopening of
files).

From a technical point of view I don't think that is desirable. The
entire log traffic would have to be routed through the postmaster, as it
is in LibertyRMS's log rotator now through the perl script. And we
really try to keep everything outside the postmaster that does not
absolutely have to be in there for stability reasons.

We can discuss if the log rotator should be a child process of the
postmaster or the other way round, but that will not change the flow of
bytes between the processes in any way.

I would say it's better the way it is, because it does not pollute the
postmasters wait logic with another exception.

My ideal solution would be to integrate the log rotators functionality
into a C version of pg_ctl that forks and detaches from the control
terminal in the way, daemons should.

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

Peter Eisentraut <peter_e@gmx.net> writes:

What would get me a whole lot more excited is if the server could write
directly to a file and do its own rotating (or at least reopening of
files).

AFAICS, the only practical way to do this is to have a single process
collecting the stdout/stderr from the postmaster and all its children.
pglog-rotator is one implementation of that approach.

I too would rather this functionality were integrated into the server,
but I haven't noticed anyone stepping up to the plate to do it.

Considering that your rotator is tailored to a rather specific setup, it
doesn't do anything better compared to established ones, it prevents the
use of pg_ctl, it's written in Perl, and it doesn't do anything for
Windows users, I think it's not suitable for a general audience.

These might be good arguments for not putting it into the mainstream,
but I don't think they have any force if we consider it for contrib.

I feel we really ought to have *some* rotator included in the standard
distro, just so that the Admin Guide can point to a concrete solution
instead of having to arm-wave about what you can get off the net.
If someone can offer a better alternative than Andrew's, great, let's
see it.

regards, tom lane

On Friday April 4 2003 10:19, Tom Lane wrote:

I feel we really ought to have *some* rotator included in the standard
distro, just so that the Admin Guide can point to a concrete solution
instead of having to arm-wave about what you can get off the net.
If someone can offer a better alternative than Andrew's, great, let's
see it.

Out of curiosity, are there issues preventing inclusion of Apache's log
rotation code? It seems you'd be hard-pressed to find a more
battle-hardened log rotator.

Obviously some people also wish to rotate based on log file size, so adding
both to contrib at least seems sensible.

Ed

On Fri, 4 Apr 2003, Ed L. wrote:

On Friday April 4 2003 10:19, Tom Lane wrote:

I feel we really ought to have *some* rotator included in the standard
distro, just so that the Admin Guide can point to a concrete solution
instead of having to arm-wave about what you can get off the net.
If someone can offer a better alternative than Andrew's, great, let's
see it.

Out of curiosity, are there issues preventing inclusion of Apache's log
rotation code? It seems you'd be hard-pressed to find a more
battle-hardened log rotator.

Obviously some people also wish to rotate based on log file size, so adding
both to contrib at least seems sensible.

OK, I'm playing with the pg_ctl script that comes with 7.3, and trying to
make it startup with apaches rotatelog script, but this line won't pipe
output. I'm a total noob at bash shell scripting, so please feel free to
snicker when you answer.

rotatelogs is in my path and all, it just never sees it.

"$po_path" ${1+"$@"} </dev/null | $PGPATH/rotatelogs $logfile $DURATION 2>&1 &

"scott.marlowe" <scott.marlowe@ihs.com> writes:

rotatelogs is in my path and all, it just never sees it.

You mean the command fails? Or just that it doesn't capture output?

"$po_path" ${1+"$@"} </dev/null | $PGPATH/rotatelogs $logfile $DURATION 2>&1 &

Most if not all of the postmaster's log output goes to stderr, so you'd need

"$po_path" ${1+"$@"} </dev/null 2>&1 | $PGPATH/rotatelogs ...

to have any hope of useful results.

regards, tom lane