Log rotation?

So, is log rotation a concern? Is this a reasonable solution? Other
ideas?

(No Grand Unified Logging Solutions please. And no, "use syslog" doesn't
count.)

What's the problem with using newsyslog or logrotate at the moment? (ie.
use the system log rotator)

Chris

Peter Eisentraut <peter_e@gmx.net> writes:

With all the great work put into allowing true 24/7 operation, as
distributed we're still unable to rotate the log file. While the log file
tends to be smaller than the database system as a whole, this is still
going to annoy people because they can't control disk usage without taking
the server down.

I've been playing with a little program I wrote whose sole purpose is to
write its stdin to a file and close and reopen that file when it receives
a signal. I figured this could work well when integrated transparently
into pg_ctl.

Aren't there log-rotation utilities out there already? (I seem to
recall mention that Apache has one, for instance.) Seems like this
is a wheel we shouldn't have to reinvent.

Also, I kinda thought the long-range solution was to encourage everyone
to migrate to syslog logging ...

And no, "use syslog" doesn't count.

Why not?

regards, tom lane

Tom Lane <tgl@sss.pgh.pa.us> writes:

And no, "use syslog" doesn't count.

Why not?

The standard implementations of syslog lose log entries under heavy
load, because they rely on a daemon which reads from a named pipe with
a limited buffer space. This is not acceptable in a production
system, since heavy load is often just the time you need to see the
log entries.

It would be possible to implement the syslog(3) interface in a
different way, of course, which did not use syslogd. I don't know of
any such implementation.

(My personal preference these days is an approach like DJB's
daemontools, which separates the handling of log entries from the
program doing the logging.)

Ian

Ian Lance Taylor <ian@airs.com> writes:

Tom Lane <tgl@sss.pgh.pa.us> writes:

And no, "use syslog" doesn't count.

Why not?

The standard implementations of syslog lose log entries under heavy
load,

Okay, that's a sufficient answer for that point.

(My personal preference these days is an approach like DJB's
daemontools, which separates the handling of log entries from the
program doing the logging.)

That still leads back to my first question, which is whether we can't
rely on someone else's logrotation code.

regards, tom lane

Tom Lane writes:

Aren't there log-rotation utilities out there already? (I seem to
recall mention that Apache has one, for instance.) Seems like this
is a wheel we shouldn't have to reinvent.

I'm aware of the Apache rotatelogs utility, but I'm not completely
satisfied with it.

1. It tries to do the rotating itself. I'd rather rely on the OS'
rotating and archiving facilities.

2. Only offers a time-based rotate, no manual intervention possible (via
signal).

3. We don't want to have to tell people to install Apache and patch their
pg_ctl.

4. We don't want to include it in our distribution because the license
contains an advertisement clause.

It's not like what I wrote is going to look wildly different than theirs.
There's only so much variation you can put into 100 lines of code.

And no, "use syslog" doesn't count.

Why not?

1. Might not be available (?)

2. Might not be reliable

3. Might not have root access

4. Not all messages will go through elog. This is a bug, but not trivial
to fix.

--
Peter Eisentraut peter_e@gmx.net http://funkturm.homeip.net/~peter

Christopher Kings-Lynne writes:

What's the problem with using newsyslog or logrotate at the moment? (ie.
use the system log rotator)

The postmaster will never close the output file, so you can rotate all you
want, the original file will never be abandoned.

--
Peter Eisentraut peter_e@gmx.net http://funkturm.homeip.net/~peter

Thus spake Tom Lane

Also, I kinda thought the long-range solution was to encourage everyone
to migrate to syslog logging ...

And no, "use syslog" doesn't count.

Why not?

Well, one "why not" might be that syslog is not a guaranteed delivery
logging system. It might be good enough for some applications but I
don't think that it should be forced on everyone.

-- 
D'Arcy J.M. Cain <darcy@{druid|vex}.net>   |  Democracy is three wolves
http://www.druid.net/darcy/                |  and a sheep voting on
+1 416 425 1212     (DoD#0082)    (eNTP)   |  what's for dinner.

Peter Eisentraut <peter_e@gmx.net> writes:

Tom Lane writes:

Aren't there log-rotation utilities out there already? (I seem to
recall mention that Apache has one, for instance.) Seems like this
is a wheel we shouldn't have to reinvent.

I'm aware of the Apache rotatelogs utility, but I'm not completely
satisfied with it.

Okay, those are reasonable points. Given that it's only ~100 lines of
code, I'll withdraw my objection to rolling our own. Let's just do it.

regards, tom lane

At 08:54 PM 9/5/2001 -0700, Ian Lance Taylor wrote:

Tom Lane <tgl@sss.pgh.pa.us> writes:

And no, "use syslog" doesn't count.

Why not?

The standard implementations of syslog lose log entries under heavy
load, because they rely on a daemon which reads from a named pipe with
a limited buffer space. This is not acceptable in a production
system, since heavy load is often just the time you need to see the
log entries.

It would be possible to implement the syslog(3) interface in a
different way, of course, which did not use syslogd. I don't know of
any such implementation.

(My personal preference these days is an approach like DJB's
daemontools, which separates the handling of log entries from the
program doing the logging.)

Ian

Greetings,

Kind of ironic, I have been working on a similar logging system for Apache
that works with PostgreSQL, and I just released 2.0-beta last night. My
post to announcements was delayed, but you can check it out here:
http://www.digitalstratum.com/pglogd/

If pgLOGd looks like something similar to what you are looking for, I could
probably modify it to log for PostgreSQL. Two of its requirements during
development were fast and robust, and similar to what you described above
it does not "process" the entries, that is done later. You also got me
thinking that maybe syslogd needs an overhaul too...

Matthew

You may be interested in
http://www.ietf.org/internet-drafts/draft-ietf-syslog-reliable-12.txt which
builds a reliable syslog protocol on top of BEEP. There are free
implementations of BEEP in C and Java at http://beepcore.org

----- Original Message -----
From: "Matthew Hagerty" <mhagerty@voyager.net>
To: "Ian Lance Taylor" <ian@airs.com>; "Tom Lane" <tgl@sss.pgh.pa.us>
Cc: "Peter Eisentraut" <peter_e@gmx.net>; "PostgreSQL Development"
<pgsql-hackers@postgresql.org>
Sent: Thursday, September 06, 2001 10:08 AM
Subject: Re: [HACKERS] Log rotation?

At 08:54 PM 9/5/2001 -0700, Ian Lance Taylor wrote:

Tom Lane <tgl@sss.pgh.pa.us> writes:

And no, "use syslog" doesn't count.

Why not?

The standard implementations of syslog lose log entries under heavy
load, because they rely on a daemon which reads from a named pipe with
a limited buffer space. This is not acceptable in a production
system, since heavy load is often just the time you need to see the
log entries.

It would be possible to implement the syslog(3) interface in a
different way, of course, which did not use syslogd. I don't know of
any such implementation.

(My personal preference these days is an approach like DJB's
daemontools, which separates the handling of log entries from the
program doing the logging.)

Ian

Greetings,

Kind of ironic, I have been working on a similar logging system for Apache
that works with PostgreSQL, and I just released 2.0-beta last night. My
post to announcements was delayed, but you can check it out here:
http://www.digitalstratum.com/pglogd/

If pgLOGd looks like something similar to what you are looking for, I

could

Yeah, I use FreeBSD's wonderful newsyslog utility, and I do my logging like
this:

su -l pgsql -c '[ -d ${PGDATA} ] && exec /usr/local/bin/pg_ctl
start -s -w -o "-i" -l /var/log/pgsql.log'

And my /etc/newsyslog.conf entry:

/var/log/pgsql.log pgsql:pgsql 600 3 4096 * Z

Chris

Christopher Kings-Lynne writes:

Yeah, I use FreeBSD's wonderful newsyslog utility, and I do my logging like
this:

su -l pgsql -c '[ -d ${PGDATA} ] && exec /usr/local/bin/pg_ctl
start -s -w -o "-i" -l /var/log/pgsql.log'

And my /etc/newsyslog.conf entry:

/var/log/pgsql.log pgsql:pgsql 600 3 4096 * Z

Sorry, this does not convey any information to me. Does this newsyslog
thing do anything that's so smart that we should know about it?

--
Peter Eisentraut peter_e@gmx.net http://funkturm.homeip.net/~peter

Hi Peter,

I've been playing with a little program I wrote whose sole purpose is to
write its stdin to a file and close and reopen that file when it receives
a signal. I figured this could work well when integrated transparently
into pg_ctl.

So, is log rotation a concern? Is this a reasonable solution? Other
ideas?

There was a discussion of this over a year ago. After I contributed
to the discussion Tom Lane suggested I write something, and in the
tradition of software development I did so but never quite finished it
... I got to the point that it works for me, then got distracted
before completing a test suite.

You may well prefer your own code, but this one supports rotation on
size and/or time basis as well as on receipt of SIGHUP, and places a
timestamp on each line written. It's also pretty careful about errors,
which is one of the things that was disliked about the Apache program
last time it was discussed.

I am happy to contribute the code using the standard PostgreSQL
license if it's wanted. (If anyone wants it under a BSD license or
GPL for another purpose that's fine too.)

I use the code on HP-UX, and developed it on NetBSD. There shouldn't
be too many portability problems lurking, other than the usual hassles
of what % escape to use in printf() for off_t. I doubt anyone wants
log files larger than a couple of GB anyway? :-)

ftp://ftp.nemeton.com.au/pub/src/logwrite-1.0alpha.tar.gz

(No Grand Unified Logging Solutions please. And no, "use syslog" doesn't
count.)

<grin>

One improvement I suggest is that the postmaster be taught to start
(and restart if necessary) the log program. This avoids fragile
startup scripts and also avoids taking down PostgreSQL if someone
sends the wrong signal to the log program.

Cheers,

Giles