pg_ctl patch to integrate apache's rotatelogs

Am Freitag, 14. Mai 2004 17:35 schrieb Andrew Hammond:

Please find attached a patch to integrate rotatelogs into pg_ctl. I've
noticed a couple of questions about how to do this and it seems easier
(and more elegant) to solve it in code rather than in documentation. The
patch is pretty simple and I've made it reasonably self documenting. It
test for apache rotatelogs only when unless it is specifically used.
This code only adds functionality and causes no changes to current usage.

How is this better than just piping the output of pg_ctl into rotatelogs?

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Peter Eisentraut wrote:
| Am Freitag, 14. Mai 2004 17:35 schrieb Andrew Hammond:
|
|>Please find attached a patch to integrate rotatelogs into pg_ctl. I've
|>noticed a couple of questions about how to do this and it seems easier
|>(and more elegant) to solve it in code rather than in documentation. The
|>patch is pretty simple and I've made it reasonably self documenting. It
|>test for apache rotatelogs only when unless it is specifically used.
|>This code only adds functionality and causes no changes to current usage.
|
|
| How is this better than just piping the output of pg_ctl into rotatelogs?

That's essentially what the patch does. It's better because it does it
correctly instead of requiring an admin to learn how to do it correctly.
pg_ctl already has a -l option. I think it also makes a certain amount
of sense to keep as much of the logging stuff in one place as possible:

pg_ctl -l foo.log start

Oh... I need log rotation...

pg_ctl -l foo.log -r 10M start

Drew
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFApP5ygfzn5SevSpoRAqpwAKC23sHnxEBcg5y9umOxlIyhph6NhACeLmbF
c3Sne4Z7BFk0c7tkj7jaZEo=
=cSWp
-----END PGP SIGNATURE-----

Andrew Hammond wrote:

That's essentially what the patch does. It's better because it does
it correctly instead of requiring an admin to learn how to do it
correctly.

That is entirely unconvincing handwaving. We already have a method to
do what you propose, and it's no less correct or harder to learn about
than what you propose. Moreover, it isn't specific to any one
logrotation method and doesn't create any broken dependencies (or do
you propose that all PostgreSQL packages will in the future have to
depend on Apache)?

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Peter Eisentraut wrote:
| Andrew Hammond wrote:
|
|>That's essentially what the patch does. It's better because it does
|>it correctly instead of requiring an admin to learn how to do it
|>correctly.
|
| That is entirely unconvincing handwaving.

There was recently a huge thread in hackers about the value of making
postgres easier to install and use. I think this script makes postgres
easier to use for those of us who can't use syslog. Your perspective as
an experienced postgres developer may make it difficult to see the value
here, but I think it smooths out one more entry level barrier for Postgres.

| We already have a method to
| do what you propose, and it's no less correct or harder to learn about
| than what you propose. Moreover, it isn't specific to any one
| logrotation method

~From what I've seen, I have to disagree. The documentation says to "pipe
the stderr of the postmaster to some type of log rotation program."
which is pretty vague. It then includes an _incorrect_ example of how to
use logrotate (logrotate rotates existing log files, if you pipe output
into it, your postmaster will block). There are no explicit, working
examples of how to configure the various log rotation options.

Of the log rotation tools available _none_ of them are particularly well
supported for use with postgres:

- - DJB's multilog requires a whole mess of configuration, and is
difficult to run unless you're root. In which case you'd be far better
off to use syslog anyway. It is rsync friendly

- - logrotator does things to files which, as far as I could tell, would
require restarting postgres.

- - pg_logs.pl is very configurable, but ugly and expensive to run (I work
for Afilias, and this patch is because we're less than thrilled with
pg_logs and looking for alternatives. It may turn out that logging is
just expensive no matter what, but we'd like to find a better solution)

- - rotatelogs is the easiest of the above, rsync friendly, but an admin
still has to actually figure out the correct way of using it.

| and doesn't create any broken dependencies (or do
| you propose that all PostgreSQL packages will in the future have to
| depend on Apache)?

Of course not! :)

The idea here is to make it as easy as possible to use rotatelogs. The
script doesn't even test for existance of rotatelogs unless you
explicitly try to use it. And then, if it can't find rotatelogs, it says
so explicitly. I think it's an extreemly low risk patch,

Drew
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFApR6Hgfzn5SevSpoRAnmDAJwO7zHdaiMwPx7vjRLnzxWrrT+8ogCgpOEp
eIpMlyfR3G1MBcwRst6AQlQ=
=4b/R
-----END PGP SIGNATURE-----

On Fri, May 14, 2004 at 15:31:21 -0400,
Andrew Hammond <ahammond@ca.afilias.info> wrote:

- - DJB's multilog requires a whole mess of configuration, and is
difficult to run unless you're root. In which case you'd be far better
off to use syslog anyway. It is rsync friendly

I use multilog and it works just fine. If you are already using it for other
things it is very easy to use it for postgres too.

You aren't ever supposed to run multilog as root. Normally you just tell
daemontools what user you want to run it as. But you should also be able
to run it without daemontools as a nonprivileged user without doing anything
tricky.

Andrew Hammond wrote:

~From what I've seen, I have to disagree. The documentation says to "pipe
the stderr of the postmaster to some type of log rotation program."
which is pretty vague. It then includes an _incorrect_ example of how to
use logrotate (logrotate rotates existing log files, if you pipe output
into it, your postmaster will block). There are no explicit, working
examples of how to configure the various log rotation options.

Ouch. How long has that documentation been wrong? We have pointed
folks to that section of the docs tons of times, and no one mentioned
that "logrotate" is really "rotatelogs", and that it is missing
parameters?

I have applied the following patch to document rotatelogs properly.

Also, the pg_ctl man page should mention rotatelogs too, and the patch
includes that.

The idea here is to make it as easy as possible to use rotatelogs. The
script doesn't even test for existance of rotatelogs unless you
explicitly try to use it. And then, if it can't find rotatelogs, it says
so explicitly. I think it's an extreemly low risk patch,

This is a powerful argument. I think the big reason we have avoided
rotatelogs integration is the desire not to favor a particular log
rotate solution. However, we have syslog hooks, so maybe rotatelogs
hooks are OK.

Comments?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Bruno Wolff III wrote:
| On Fri, May 14, 2004 at 15:31:21 -0400,
| Andrew Hammond <ahammond@ca.afilias.info> wrote:
|
|>- - DJB's multilog requires a whole mess of configuration, and is
|>difficult to run unless you're root. In which case you'd be far better
|>off to use syslog anyway. It is rsync friendly
|
| I use multilog and it works just fine. If you are already using it for
other
| things it is very easy to use it for postgres too.
|
| You aren't ever supposed to run multilog as root. Normally you just tell
| daemontools what user you want to run it as. But you should also be able
| to run it without daemontools as a nonprivileged user without doing
anything
| tricky.

Yes, if you're running daemontools, then multilog is hands down the way
to go. Is it daemontools that really wants to be root? That might be
what I'm confusing it with. Anyway, my goal was to provide a solution
that fits nicely with pg_ctl's -l option, multilog didn't look like a
good way to do that.

Drew
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFApSXhgfzn5SevSpoRAvIlAJ98RdxrjPXlwxqXLwdKjOTgSf0LjACfYZks
iMmO3vDJQgS5NjKeaOrNhfc=
=sFcr
-----END PGP SIGNATURE-----

On Fri, May 14, 2004 at 16:02:43 -0400,
Andrew Hammond <ahammond@ca.afilias.info> wrote:

Yes, if you're running daemontools, then multilog is hands down the way
to go. Is it daemontools that really wants to be root? That might be
what I'm confusing it with. Anyway, my goal was to provide a solution
that fits nicely with pg_ctl's -l option, multilog didn't look like a
good way to do that.

The scanning process (svscan) and listening processes (tcpserver) run
as root. The actual services and logging processes are supposed to
run (relatively) unprivileged users.

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Ouch. How long has that documentation been wrong? We have pointed
folks to that section of the docs tons of times, and no one mentioned
that "logrotate" is really "rotatelogs", and that it is missing
parameters?

I have applied the following patch to document rotatelogs properly.

I think you just broke it. logrotate and rotatelogs are two different
programs, and AFAIK the existing documentation correctly described the
one that is easier to use with Postgres.

regards, tom lane

Tom Lane wrote:

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Ouch. How long has that documentation been wrong? We have pointed
folks to that section of the docs tons of times, and no one mentioned
that "logrotate" is really "rotatelogs", and that it is missing
parameters?

I have applied the following patch to document rotatelogs properly.

I think you just broke it. logrotate and rotatelogs are two different
programs, and AFAIK the existing documentation correctly described the
one that is easier to use with Postgres.

Oh, why don't I have logrotate. My system has:

rotatelogs - rotate Apache logs without having to kill the
server

rotatelogs logfile rotationtime [offset]

and it says it is Apache's. I must be wrong but can someone explain
this?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Andrew Hammond wrote:

There was recently a huge thread in hackers about the value of making
postgres easier to install and use. I think this script makes
postgres easier to use for those of us who can't use syslog.

So you're replacing

pg_ctl | rotatelogs

which everyone learns in his first hour of using Unix, by some arbitary
command-line option that people are going to have to learn. Next week,
we'll see four or five additional patches to add support for all the
other log rotation programs. Soon, we'll run out of logical option
letters. How does that improve anything?

All that you're doing here is adding syntactical complexity for no
technical or usability gain.

If the documentation is insufficient, please be invited to improve it.

Bruce Momjian wrote:

Tom Lane wrote:

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Ouch. How long has that documentation been wrong? We have
pointed folks to that section of the docs tons of times, and no
one mentioned that "logrotate" is really "rotatelogs", and that
it is missing parameters?

I have applied the following patch to document rotatelogs
properly.

I think you just broke it. logrotate and rotatelogs are two
different programs, and AFAIK the existing documentation correctly
described the one that is easier to use with Postgres.

Oh, why don't I have logrotate. My system has:

rotatelogs - rotate Apache logs without having to kill the
server

rotatelogs logfile rotationtime [offset]

and it says it is Apache's. I must be wrong but can someone explain
this?

logrotate rotates log files and hups the servers according to a
configuration file. rotatelogs reads the log data on stdin and reopens
a new output file once in a while. The first hunk of your patch was
wrong, the rest seemed OK.

Peter Eisentraut wrote:

Bruce Momjian wrote:

Tom Lane wrote:

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Ouch. How long has that documentation been wrong? We have
pointed folks to that section of the docs tons of times, and no
one mentioned that "logrotate" is really "rotatelogs", and that
it is missing parameters?

I have applied the following patch to document rotatelogs
properly.

I think you just broke it. logrotate and rotatelogs are two
different programs, and AFAIK the existing documentation correctly
described the one that is easier to use with Postgres.

Oh, why don't I have logrotate. My system has:

rotatelogs - rotate Apache logs without having to kill the
server

rotatelogs logfile rotationtime [offset]

and it says it is Apache's. I must be wrong but can someone explain
this?

logrotate rotates log files and hups the servers according to a
configuration file. rotatelogs reads the log data on stdin and reopens
a new output file once in a while. The first hunk of your patch was
wrong, the rest seemed OK.

OK, just to confirm, I backed out the first attached patch, but kept the
rest, which is my second attached patch.

I am still confused how these errors remained in the documentation for
so long.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Peter Eisentraut wrote:
| Andrew Hammond wrote:
|
|>There was recently a huge thread in hackers about the value of making
|>postgres easier to install and use. I think this script makes
|>postgres easier to use for those of us who can't use syslog.
|
|
| So you're replacing
|
| pg_ctl | rotatelogs
|
| which everyone learns in his first hour of using Unix, by some arbitary
| command-line option that people are going to have to learn.

No, pg_ctl | rotatelogs will still work as well as it did before.
However, for those people who don't _know_ about rotatelogs and are
reading the manual page for pg_ctl, the -r option will point them in the
correct dirrection. On a related note,

pg_ctl > $log_file

has already been replaced by the the arbitrary -l command-line option.
Clearly somebody thought there was some value to be had there and I see
the same sort of value for the -r option.

| Next week, we'll see four or five additional patches to add support
| for all the other log rotation programs.

Personally, I'd see this as a good thing since it helps people try
different tools and pick the one that's right for them. However I doubt
it will happen since the other log rotation tools work differently.
pg_ctl simply doesn't make sense as a place to integrate them.

| Soon, we'll run out of logical option letters. How does that improve
anything?

I thought that was the definition of "feature complete". :)

Drew
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFAqMovgfzn5SevSpoRAiYeAJ98OpXEPzlFDdIp0LcVBY2nJVsIPwCfbubX
AqSMtU7nxeh+SaNoSDqg1PI=
=aWxW
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

|>logrotate rotates log files and hups the servers according to a
|>configuration file. rotatelogs reads the log data on stdin and reopens
|>a new output file once in a while. The first hunk of your patch was
|>wrong, the rest seemed OK.
|
|
| OK, just to confirm, I backed out the first attached patch, but kept the
| rest, which is my second attached patch.

This looks correct to me.

| I am still confused how these errors remained in the documentation for
| so long.

Probably because non-syslog logging in a production environment is a
very small niche. Which might, unfortunately, be a good argument against
my proposed patch.

Drew
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFAqhIGgfzn5SevSpoRAvizAJ0fZRI5o1FuERzUen2ufWwCsz/J2ACghQ7B
agaY+YaC0LIVF/jNCk+QnmA=
=Rsjl
-----END PGP SIGNATURE-----

We documented using rotatelogs in the pg_ctl manual page, and have the
main docs correct on how to use it. Hopefully that will be enough.

---------------------------------------------------------------------------

Andrew Hammond wrote:
[ PGP not available, raw data follows ]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

|>logrotate rotates log files and hups the servers according to a
|>configuration file. rotatelogs reads the log data on stdin and reopens
|>a new output file once in a while. The first hunk of your patch was
|>wrong, the rest seemed OK.
|
|
| OK, just to confirm, I backed out the first attached patch, but kept the
| rest, which is my second attached patch.

This looks correct to me.

| I am still confused how these errors remained in the documentation for
| so long.

Probably because non-syslog logging in a production environment is a
very small niche. Which might, unfortunately, be a good argument against
my proposed patch.

Drew
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFAqhIGgfzn5SevSpoRAvizAJ0fZRI5o1FuERzUen2ufWwCsz/J2ACghQ7B
agaY+YaC0LIVF/jNCk+QnmA=
=Rsjl
-----END PGP SIGNATURE-----

[ End of raw data]

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Bruno, would you be willing to write something for docs about how to use
multilog for postgres?

Drew

Bruce Momjian wrote:
| We documented using rotatelogs in the pg_ctl manual page, and have the
| main docs correct on how to use it. Hopefully that will be enough.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFArgVKgfzn5SevSpoRAsQiAKCe+XAOFi1YuGxh8GLY8Qc/XjgWpgCggNwn
KH5dE0KAMa4P/XVfBNwpujk=
=yRWQ
-----END PGP SIGNATURE-----

On Fri, May 21, 2004 at 09:34:04 -0400,
Andrew Hammond <ahammond@ca.afilias.info> wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Bruno, would you be willing to write something for docs about how to use
multilog for postgres?

What are you looking for. Do you need to know about how to install
daemontools or do you just need to know how to set up the run file
for the postgres service? I think for someone that is already using
daemontools setting up postgres is going to be pretty simple. Output
goes to stdio and stderr and it responds to daemontool's signals in a
reasonable way so you can copy over the setup from one of DJB's tool's
services and it will work.
It is also possible to use multilog without daemontools, but I am guessing
the daemontools enviroment would be what you would want documentation for.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Bruno Wolff III wrote:
| On Fri, May 21, 2004 at 09:34:04 -0400,
| Andrew Hammond <ahammond@ca.afilias.info> wrote:
|
|>-----BEGIN PGP SIGNED MESSAGE-----
|>Hash: SHA1
|>
|>Bruno, would you be willing to write something for docs about how to use
|>multilog for postgres?
|
|
| What are you looking for. Do you need to know about how to install
| daemontools or do you just need to know how to set up the run file
| for the postgres service? I think for someone that is already using
| daemontools setting up postgres is going to be pretty simple. Output
| goes to stdio and stderr and it responds to daemontool's signals in a
| reasonable way so you can copy over the setup from one of DJB's tool's
| services and it will work.
| It is also possible to use multilog without daemontools, but I am guessing
| the daemontools enviroment would be what you would want documentation for.

I would be insterested in seeing how to set up multilog by it'self. I
figure that once you get that far, going to daemontools should be pretty
straighforward. It's also within the scope of Postgres' logging
documentation.

Drew
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFArm2sgfzn5SevSpoRAqmjAJ4we5Xr3P8g443axAEi4xfvtE1eHgCfQcw5
hq739C75nmix7NA5a/ADt1k=
=yHwd
-----END PGP SIGNATURE-----