BUG #6528: pglesslog still referenced in docs, but no 9.1 support
The following bug has been logged on the website:
Bug reference: 6528
Logged by: Stuart Bishop
Email address: stuart.bishop@canonical.com
PostgreSQL version: 9.1.3
Operating system: Ubuntu
Description:
In section 24.3.5.1 of the docs on setting up continuous archiving, there
are a few paragraphs and examples for using pglesslog (pg_compresslog and
pg_decompresslog). This tool did not get out of beta for 9.0 and the only
reports I find for 9.1 are for not being able to compile it. Suggest
removing it from the docs unless someone wants to pull this tool into the
core.
On Tue, Mar 13, 2012 at 5:00 AM, <stuart.bishop@canonical.com> wrote:
The following bug has been logged on the website:
Bug reference: 6528
Logged by: Stuart Bishop
Email address: stuart.bishop@canonical.com
PostgreSQL version: 9.1.3
Operating system: Ubuntu
Description:In section 24.3.5.1 of the docs on setting up continuous archiving, there
are a few paragraphs and examples for using pglesslog (pg_compresslog and
pg_decompresslog). This tool did not get out of beta for 9.0 and the only
reports I find for 9.1 are for not being able to compile it. Suggest
removing it from the docs unless someone wants to pull this tool into the
core.
This complaint appears to be accurate. I think we should go ahead and
remove that mention.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Mon, Apr 09, 2012 at 10:10:34AM -0400, Robert Haas wrote:
On Tue, Mar 13, 2012 at 5:00 AM, <stuart.bishop@canonical.com> wrote:
The following bug has been logged on the website:
Bug reference: � � �6528
Logged by: � � � � �Stuart Bishop
Email address: � � �stuart.bishop@canonical.com
PostgreSQL version: 9.1.3
Operating system: � Ubuntu
Description:In section 24.3.5.1 of the docs on setting up continuous archiving, there
are a few paragraphs and examples for using pglesslog (pg_compresslog and
pg_decompresslog). This tool did not get out of beta for 9.0 and the only
reports I find for 9.1 are for not being able to compile it. Suggest
removing it from the docs unless someone wants to pull this tool into the
core.This complaint appears to be accurate. I think we should go ahead and
remove that mention.
Agreed; removed with the attached patch. I didn't bother keeping the
gzip mention because I assume there is little value to using without
pglesslog.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ It's impossible for everything to be true. +
Attachments:
pglesslogtext/plain; charset=us-asciiDownload+0-16
Bruce Momjian <bruce@momjian.us> wrote:
Robert Haas wrote:
<stuart.bishop@canonical.com> wrote:
In section 24.3.5.1 of the docs on setting up continuous
archiving, there are a few paragraphs and examples for using
pglesslog (pg_compresslog and pg_decompresslog). This tool did
not get out of beta for 9.0 and the only reports I find for 9.1
are for not being able to compile it. Suggest removing it from
the docs unless someone wants to pull this tool into the core.This complaint appears to be accurate. I think we should go
ahead and remove that mention.Agreed; removed with the attached patch. I didn't bother keeping
the gzip mention because I assume there is little value to using
without pglesslog.
We found that gzip without doing something about the unused portion
of the WAL file generally reduced file size by about 50%, so it
still might be worth mentioning that.
-Kevin
Excerpts from Bruce Momjian's message of lun abr 09 15:13:10 -0300 2012:
On Mon, Apr 09, 2012 at 10:10:34AM -0400, Robert Haas wrote:
This complaint appears to be accurate. I think we should go ahead and
remove that mention.Agreed; removed with the attached patch. I didn't bother keeping the
gzip mention because I assume there is little value to using without
pglesslog.
I'm not sure that assumption holds. I think you're thinking of the hack
that zeroes out the "empty" holes in the middle of data pages; those
didn't compress well unless zeroed out before compression. This tool is
not about that, but rather about removing redundant info from WAL files.
It seems to me that WAL files would be as gzip-compressible regardless
of pglesslog being applied. (Another related tool is clearxlogtail
which zeroes areas from WAL files when they are empty because of an
early switch due to archive timeout).
The funny thing is, apparently pg_lesslog was intentionally broken by
changing XLR_BKP_REMOVABLE to XLP_BKP_REMOVABLE (different semantics?)
and the fixes to make it compile again look simple. It's a bit strange
that NTT stopped maintaining the tool .. Maybe it wasn't useful anymore?
--
Álvaro Herrera <alvherre@commandprompt.com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support
On Mon, Apr 09, 2012 at 03:37:09PM -0300, Alvaro Herrera wrote:
Excerpts from Bruce Momjian's message of lun abr 09 15:13:10 -0300 2012:
On Mon, Apr 09, 2012 at 10:10:34AM -0400, Robert Haas wrote:
This complaint appears to be accurate. I think we should go ahead and
remove that mention.Agreed; removed with the attached patch. I didn't bother keeping the
gzip mention because I assume there is little value to using without
pglesslog.I'm not sure that assumption holds. I think you're thinking of the hack
that zeroes out the "empty" holes in the middle of data pages; those
didn't compress well unless zeroed out before compression. This tool is
not about that, but rather about removing redundant info from WAL files.
It seems to me that WAL files would be as gzip-compressible regardless
of pglesslog being applied. (Another related tool is clearxlogtail
OK, docs updated with the attached patch.
which zeroes areas from WAL files when they are empty because of an
early switch due to archive timeout).
Should we document that?
The funny thing is, apparently pg_lesslog was intentionally broken by
changing XLR_BKP_REMOVABLE to XLP_BKP_REMOVABLE (different semantics?)
and the fixes to make it compile again look simple. It's a bit strange
that NTT stopped maintaining the tool .. Maybe it wasn't useful anymore?
No idea.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ It's impossible for everything to be true. +
Attachments:
gunziptext/plain; charset=us-asciiDownload+11-0
Bruce Momjian <bruce@momjian.us> wrote:
On Mon, Apr 09, 2012 at 03:37:09PM -0300, Alvaro Herrera wrote:
(Another related tool is clearxlogtail which zeroes areas from
WAL files when they are empty because of an early switch due to
archive timeout).Should we document that?
Our shop has been using that since before pglesslog existed, and it
has continued to work across many major releases with no change to
source code because it doesn't get down to the level of looking at
the xlog records themselves, just the segment and page-level
structures. I should probably put packaging that up as a proper
extension and posting it to PGXN as another thing on the list of
things I'll do when I stumble over that pot of round tuits at the
end of the rainbow. Right now the source is on pgfoundry. If
anybody wants to do anything with it before I find time, feel free.
-Kevin
On Mon, Apr 9, 2012 at 02:07:43PM -0500, Kevin Grittner wrote:
Bruce Momjian <bruce@momjian.us> wrote:
On Mon, Apr 09, 2012 at 03:37:09PM -0300, Alvaro Herrera wrote:
(Another related tool is clearxlogtail which zeroes areas from
WAL files when they are empty because of an early switch due to
archive timeout).Should we document that?
Our shop has been using that since before pglesslog existed, and it
has continued to work across many major releases with no change to
source code because it doesn't get down to the level of looking at
the xlog records themselves, just the segment and page-level
structures. I should probably put packaging that up as a proper
extension and posting it to PGXN as another thing on the list of
things I'll do when I stumble over that pot of round tuits at the
end of the rainbow. Right now the source is on pgfoundry. If
anybody wants to do anything with it before I find time, feel free.
I was going to add a mention of this to our docs, but it seems there are
no released files for the project:
http://pgfoundry.org/frs/?group_id=1000308
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ It's impossible for everything to be true. +
Bruce Momjian <bruce@momjian.us> wrote:
On Mon, Apr 9, 2012 at 02:07:43PM -0500, Kevin Grittner wrote:
Bruce Momjian <bruce@momjian.us> wrote:
On Mon, Apr 09, 2012 at 03:37:09PM -0300, Alvaro Herrera wrote:
(Another related tool is clearxlogtail which zeroes areas from
WAL files when they are empty because of an early switch due to
archive timeout).Should we document that?
Our shop has been using that since before pglesslog existed, and
it has continued to work across many major releases with no
change to source code because it doesn't get down to the level of
looking at the xlog records themselves, just the segment and
page-level structures. I should probably put packaging that up
as a proper extension and posting it to PGXN as another thing on
the list of things I'll do when I stumble over that pot of round
tuits at the end of the rainbow. Right now the source is on
pgfoundry. If anybody wants to do anything with it before I find
time, feel free.I was going to add a mention of this to our docs, but it seems
there are no released files for the project:
There is a single 175 line C source file to generate a stand-alone
executable filter. The source has not needed any changes since it
was finished over 4 years ago. Besides that C file, there is only a
make file. I wasn't sure what form of packaging would be easier
than just checking out the source from the SCM.
What do you think would be the right thing to do with it at this
point?
-Kevin
On Tue, Sep 4, 2012 at 12:11:53PM -0500, Kevin Grittner wrote:
Bruce Momjian <bruce@momjian.us> wrote:
On Mon, Apr 9, 2012 at 02:07:43PM -0500, Kevin Grittner wrote:
Bruce Momjian <bruce@momjian.us> wrote:
On Mon, Apr 09, 2012 at 03:37:09PM -0300, Alvaro Herrera wrote:
(Another related tool is clearxlogtail which zeroes areas from
WAL files when they are empty because of an early switch due to
archive timeout).Should we document that?
Our shop has been using that since before pglesslog existed, and
it has continued to work across many major releases with no
change to source code because it doesn't get down to the level of
looking at the xlog records themselves, just the segment and
page-level structures. I should probably put packaging that up
as a proper extension and posting it to PGXN as another thing on
the list of things I'll do when I stumble over that pot of round
tuits at the end of the rainbow. Right now the source is on
pgfoundry. If anybody wants to do anything with it before I find
time, feel free.I was going to add a mention of this to our docs, but it seems
there are no released files for the project:There is a single 175 line C source file to generate a stand-alone
executable filter. The source has not needed any changes since it
was finished over 4 years ago. Besides that C file, there is only a
make file. I wasn't sure what form of packaging would be easier
than just checking out the source from the SCM.What do you think would be the right thing to do with it at this
point?
Well, there should probably be a tar files with a README. What about
adding this to /contrib?
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ It's impossible for everything to be true. +
Bruce Momjian <bruce@momjian.us> wrote:
On Tue, Sep 4, 2012 at 12:11:53PM -0500, Kevin Grittner wrote:
What do you think would be the right thing to do with it at this
point?Well, there should probably be a tar files with a README. What
about adding this to /contrib?
I would be happy to add it to contrib if people want it there.
There was some discussion of that when I wrote it, but then there
was a feeling that it would be uninteresting once people had
streaming replication. I'm skeptical that people will ever lose
interest in archiving WAL files, because of the not-uncommon need to
keep archival backups or restore to a "point in time" to recover
from a mangled table. As long as those needs are there, I suspect
that people will want to compress WAL files, including making
partially-filled files smaller.
-Kevin
On Tue, Sep 4, 2012 at 12:49:40PM -0500, Kevin Grittner wrote:
Bruce Momjian <bruce@momjian.us> wrote:
On Tue, Sep 4, 2012 at 12:11:53PM -0500, Kevin Grittner wrote:
What do you think would be the right thing to do with it at this
point?Well, there should probably be a tar files with a README. What
about adding this to /contrib?I would be happy to add it to contrib if people want it there.
There was some discussion of that when I wrote it, but then there
was a feeling that it would be uninteresting once people had
streaming replication. I'm skeptical that people will ever lose
interest in archiving WAL files, because of the not-uncommon need to
keep archival backups or restore to a "point in time" to recover
from a mangled table. As long as those needs are there, I suspect
that people will want to compress WAL files, including making
partially-filled files smaller.
Agreed. Please move forward on the contrib idea.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ It's impossible for everything to be true. +
On Tue, Sep 4, 2012 at 02:01:54PM -0400, Bruce Momjian wrote:
On Tue, Sep 4, 2012 at 12:49:40PM -0500, Kevin Grittner wrote:
Bruce Momjian <bruce@momjian.us> wrote:
On Tue, Sep 4, 2012 at 12:11:53PM -0500, Kevin Grittner wrote:
What do you think would be the right thing to do with it at this
point?Well, there should probably be a tar files with a README. What
about adding this to /contrib?I would be happy to add it to contrib if people want it there.
There was some discussion of that when I wrote it, but then there
was a feeling that it would be uninteresting once people had
streaming replication. I'm skeptical that people will ever lose
interest in archiving WAL files, because of the not-uncommon need to
keep archival backups or restore to a "point in time" to recover
from a mangled table. As long as those needs are there, I suspect
that people will want to compress WAL files, including making
partially-filled files smaller.Agreed. Please move forward on the contrib idea.
Kevin, did this get accomplished?
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ It's impossible for everything to be true. +
--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs
Bruce Momjian escribió:
On Tue, Sep 4, 2012 at 02:01:54PM -0400, Bruce Momjian wrote:
On Tue, Sep 4, 2012 at 12:49:40PM -0500, Kevin Grittner wrote:
Bruce Momjian <bruce@momjian.us> wrote:
On Tue, Sep 4, 2012 at 12:11:53PM -0500, Kevin Grittner wrote:
What do you think would be the right thing to do with it at this
point?Well, there should probably be a tar files with a README. What
about adding this to /contrib?I would be happy to add it to contrib if people want it there.
There was some discussion of that when I wrote it, but then there
was a feeling that it would be uninteresting once people had
streaming replication. I'm skeptical that people will ever lose
interest in archiving WAL files, because of the not-uncommon need to
keep archival backups or restore to a "point in time" to recover
from a mangled table. As long as those needs are there, I suspect
that people will want to compress WAL files, including making
partially-filled files smaller.Agreed. Please move forward on the contrib idea.
Kevin, did this get accomplished?
Just to be clear, what we're proposing adding to contrib is
clearxlogtail, not pglesslog, right?
--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs
On Thu, Jan 24, 2013 at 08:43:37PM -0300, Alvaro Herrera wrote:
Bruce Momjian escribi�:
On Tue, Sep 4, 2012 at 02:01:54PM -0400, Bruce Momjian wrote:
On Tue, Sep 4, 2012 at 12:49:40PM -0500, Kevin Grittner wrote:
Bruce Momjian <bruce@momjian.us> wrote:
On Tue, Sep 4, 2012 at 12:11:53PM -0500, Kevin Grittner wrote:
What do you think would be the right thing to do with it at this
point?Well, there should probably be a tar files with a README. What
about adding this to /contrib?I would be happy to add it to contrib if people want it there.
There was some discussion of that when I wrote it, but then there
was a feeling that it would be uninteresting once people had
streaming replication. I'm skeptical that people will ever lose
interest in archiving WAL files, because of the not-uncommon need to
keep archival backups or restore to a "point in time" to recover
from a mangled table. As long as those needs are there, I suspect
that people will want to compress WAL files, including making
partially-filled files smaller.Agreed. Please move forward on the contrib idea.
Kevin, did this get accomplished?
Just to be clear, what we're proposing adding to contrib is
clearxlogtail, not pglesslog, right?
I was not clear on exactly what Kevin was going to add. Kevin?
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ It's impossible for everything to be true. +
--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs
On 2012-04-09 15:37:09 -0300, Alvaro Herrera wrote:
Excerpts from Bruce Momjian's message of lun abr 09 15:13:10 -0300 2012:
On Mon, Apr 09, 2012 at 10:10:34AM -0400, Robert Haas wrote:
This complaint appears to be accurate. I think we should go ahead and
remove that mention.Agreed; removed with the attached patch. I didn't bother keeping the
gzip mention because I assume there is little value to using without
pglesslog.I'm not sure that assumption holds. I think you're thinking of the hack
that zeroes out the "empty" holes in the middle of data pages; those
didn't compress well unless zeroed out before compression. This tool is
not about that, but rather about removing redundant info from WAL files.
It seems to me that WAL files would be as gzip-compressible regardless
of pglesslog being applied. (Another related tool is clearxlogtail
which zeroes areas from WAL files when they are empty because of an
early switch due to archive timeout).The funny thing is, apparently pg_lesslog was intentionally broken by
changing XLR_BKP_REMOVABLE to XLP_BKP_REMOVABLE (different semantics?)
and the fixes to make it compile again look simple. It's a bit strange
that NTT stopped maintaining the tool .. Maybe it wasn't useful anymore?
One problem is that it was hard to trust the tool, it had relatively bad
bugs several times and it wasn't maintained by pgdg...
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs
Bruce Momjian wrote:
On Thu, Jan 24, 2013 at 08:43:37PM -0300, Alvaro Herrera wrote:
Bruce Momjian escribió:
On Tue, Sep 4, 2012 at 02:01:54PM -0400, Bruce Momjian wrote:
On Tue, Sep 4, 2012 at 12:49:40PM -0500, Kevin Grittner wrote:
Bruce Momjian <bruce@momjian.us> wrote:
On Tue, Sep 4, 2012 at 12:11:53PM -0500, Kevin Grittner wrote:
What do you think would be the right thing to do with it at this
point?Well, there should probably be a tar files with a README. What
about adding this to /contrib?I would be happy to add it to contrib if people want it there.
There was some discussion of that when I wrote it, but then there
was a feeling that it would be uninteresting once people had
streaming replication. I'm skeptical that people will ever lose
interest in archiving WAL files, because of the not-uncommon need to
keep archival backups or restore to a "point in time" to recover
from a mangled table. As long as those needs are there, I suspect
that people will want to compress WAL files, including making
partially-filled files smaller.Agreed. Please move forward on the contrib idea.
Kevin, did this get accomplished?
Just to be clear, what we're proposing adding to contrib is
clearxlogtail, not pglesslog, right?I was not clear on exactly what Kevin was going to add. Kevin?
I was talking about pg_clearxlogtail. It is less fragile than
pglesslog because it doesn't look below the level of the page
structure -- it doesn't care about WAL record types at all. It
never needed one line of source code change until 9.3. Peter
Eisentraut was recently asking me questions about it and pointed
out that "It fails to build with 9.3devel, because of the change to
uint64 for WAL addressing."
Peter, do you have a version that works with 9.3?
Given that we're past the start of the last CF, do we still want to
consider including it? If you tilt your head at just the right
angle, I guess it could look like a bug fix -- but it is a stretch.
-Kevin
--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs
Import Notes
Resolved by subject fallback
On 1/25/13 1:35 AM, Kevin Grittner wrote:
Peter, do you have a version that works with 9.3?
I don't, but it shouldn't be hard for someone more up to date with the
internal WAL addressing changes.
Given that we're past the start of the last CF, do we still want to
consider including it? If you tilt your head at just the right
angle, I guess it could look like a bug fix -- but it is a stretch.
Given that clearxlog doesn't actually work in 9.3 right now (see above)
and doesn't have any documentation, I think the stretch is a bit too big.
I do like the idea behind it, but the first step would be to address
these two points and let people try it out. Note also that in our
discussion we had some questions about how exactly to write the archive
command to use it.
--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs
On Fri, Jan 25, 2013 at 2:01 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
On 1/25/13 1:35 AM, Kevin Grittner wrote:
Peter, do you have a version that works with 9.3?
I don't, but it shouldn't be hard for someone more up to date with the
internal WAL addressing changes.
I've been thinking about that. Should pg_clearxlogtail for 9.3 only
work for logs from 9.3, or should it be arranged to work on logs from
older versions as well (which probably means parts of the 9.2 headers
would need to be copied into the body of pg_clearxlogtail)?
Given that we're past the start of the last CF, do we still want to
consider including it? If you tilt your head at just the right
angle, I guess it could look like a bug fix -- but it is a stretch.Given that clearxlog doesn't actually work in 9.3 right now (see above)
and doesn't have any documentation, I think the stretch is a bit too big.I do like the idea behind it, but the first step would be to address
these two points and let people try it out. Note also that in our
discussion we had some questions about how exactly to write the archive
command to use it.
This is what I use:
archive_command = 'set -C -o pipefail;
(/usr/local/pgsql9_2/bin/pg_clearxlogtail < %p || cat %p) | gzip >
/mnt/foo/archivelog9_2/%f.gz'
The "|| cat %p" business is because pg_clearxlogtail will fail on
.history files and so needs an alternative path.
I think it would be better to have pg_clearxlogtail instead just
silently pass any unrecognized files through without manipulation,
rather than erroring out.
Most people would probably want to use a more explicit shell script
with notifications and such, like one Kevin posted to general a few
months ago under "checkpoint_timeout and archive_timeout".
Cheers,
Jeff
--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs
On 25-01-2013 20:23, Jeff Janes wrote:
I've been thinking about that. Should pg_clearxlogtail for 9.3 only
work for logs from 9.3, or should it be arranged to work on logs from
older versions as well (which probably means parts of the 9.2 headers
would need to be copied into the body of pg_clearxlogtail)?
As it is a new tool for contrib, I don't see the obligation to maintain
compatibility with prior versions unless someone want to do that legwork. If
the compatibility won't be included, the docs could point out the source for
prior versions.
This is what I use:
archive_command = 'set -C -o pipefail;
(/usr/local/pgsql9_2/bin/pg_clearxlogtail < %p || cat %p) | gzip >
/mnt/foo/archivelog9_2/%f.gz'The "|| cat %p" business is because pg_clearxlogtail will fail on
.history files and so needs an alternative path.
That's something that should be addressed, no?
I think it would be better to have pg_clearxlogtail instead just
silently pass any unrecognized files through without manipulation,
rather than erroring out.
... or let people choose one or another.
--
Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/
PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento
--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs