Renaming of pg_xlog and pg_clog

On Fri, Aug 26, 2016 at 11:39:29AM +0900, Michael Paquier wrote:

Now, one of the things discussed as well was that we may want to still
keep pg_xlog, and soft-link to pg_journal or whatever-the-new-name is
to not break the existing tools. Personally, I'd prefer a hard break.
That would not be complicated to fix for any tool maintainers.

I agree on a hard break, unless we get pushback from users, and even
then, they can create the symlinks themselves.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+                     Ancient Roman grave inscription +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Fri, Aug 26, 2016 at 11:55 AM, Bruce Momjian <bruce@momjian.us> wrote:

On Fri, Aug 26, 2016 at 11:39:29AM +0900, Michael Paquier wrote:

Now, one of the things discussed as well was that we may want to still
keep pg_xlog, and soft-link to pg_journal or whatever-the-new-name is
to not break the existing tools. Personally, I'd prefer a hard break.
That would not be complicated to fix for any tool maintainers.

I agree on a hard break, unless we get pushback from users, and even
then, they can create the symlinks themselves.

I strongly prefer symlink approach not to break many existing tools
and scripts.

Regards,

--
Fujii Masao

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

* Bruce Momjian (bruce@momjian.us) wrote:

On Fri, Aug 26, 2016 at 11:39:29AM +0900, Michael Paquier wrote:

Now, one of the things discussed as well was that we may want to still
keep pg_xlog, and soft-link to pg_journal or whatever-the-new-name is
to not break the existing tools. Personally, I'd prefer a hard break.
That would not be complicated to fix for any tool maintainers.

I agree on a hard break, unless we get pushback from users, and even
then, they can create the symlinks themselves.

I'm in favor of the hard break generally, but I'd like David's input on
it, to be sure we're not missing something.

Thanks!

Stephen

Re: Fujii Masao 2016-08-26 <CAHGQGwHK2yimfLvG_WQ1Vrq2h+CMzgv5u6OEmxr-cbJRO+WKWQ@mail.gmail.com>

I agree on a hard break, unless we get pushback from users, and even
then, they can create the symlinks themselves.

I strongly prefer symlink approach not to break many existing tools
and scripts.

Symlinks might actually be worse than removing the directories
altogether. If your backup tool fails because the pg_xlog directory is
gone, you'll hopefully notice, but if you end up with a backup that
consists merely of a copy of a symlink named pg_xlog, you might not
notice.

Christoph

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Fri, Aug 26, 2016 at 12:44 PM, Christoph Berg <myon@debian.org> wrote:

Re: Fujii Masao 2016-08-26 <CAHGQGwHK2yimfLvG_WQ1Vrq2h+
CMzgv5u6OEmxr-cbJRO+WKWQ@mail.gmail.com>

I agree on a hard break, unless we get pushback from users, and even
then, they can create the symlinks themselves.

I strongly prefer symlink approach not to break many existing tools
and scripts.

Symlinks might actually be worse than removing the directories
altogether. If your backup tool fails because the pg_xlog directory is
gone, you'll hopefully notice, but if you end up with a backup that
consists merely of a copy of a symlink named pg_xlog, you might not
notice.

+1. It's *much* better to cause a clean break. That way people will notice
it, and can fix it (and it will be an easy fix).

An unclean break might leave people with things that look like they work,
but don't. That's a lot more dangerous.

Same reason I'm also +1 for Stephens suggestion to put all things that
should not be in a base backup into the same directory. That may break
things now, but it will simplify things down the road. And doing it at the
same time as renaming these things makes a lot of sense, because it causes
breakage that tool-builders *have* to look at, and then they will hopefully
also notice the other change.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

On 26 August 2016 at 04:39, Michael Paquier <michael.paquier@gmail.com> wrote:

Hi all,

I am relaunching $subject as 10 development will begin soon. As far as
I know, there is agreement that we can do something here. Among the
different proposals I have found:
- pg_clog renamed to pg_commit_status, pg_xact or pg_commit
- pg_xlog renamed to pg_xjournal, pg_wal or pg_journal

Don't mean to be a party pooper, but what discussion and agreement are
we referring to here?

If we are going to suggest doing something we really should summarize
the reason for doing it rather than assume it is self evident, cos it
certainly isn't.

--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Fri, Aug 26, 2016 at 8:31 PM, Simon Riggs <simon@2ndquadrant.com> wrote:

On 26 August 2016 at 04:39, Michael Paquier <michael.paquier@gmail.com> wrote:

Hi all,

I am relaunching $subject as 10 development will begin soon. As far as
I know, there is agreement that we can do something here. Among the
different proposals I have found:
- pg_clog renamed to pg_commit_status, pg_xact or pg_commit
- pg_xlog renamed to pg_xjournal, pg_wal or pg_journal

Don't mean to be a party pooper, but what discussion and agreement are
we referring to here?

If we are going to suggest doing something we really should summarize
the reason for doing it rather than assume it is self evident, cos it
certainly isn't.

This thread was the previous one on the matter:
/messages/by-id/CAASwCXcVGma9KgEu-ESC6u928mW67noZvnawbPUSW7R7AN9UVg@mail.gmail.com

In short, with the current names, sometimes users think that pg_xlog
and pg_clog are just logs. And so it is fine to delete them to free up
space, corrupting their cluster, because they are just *logs*.
Personally I have seen that, and based on the thread I am not the only
one.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Hi,

On Fri, 2016-08-26 at 21:12 +0900, Michael Paquier wrote:

In short, with the current names, sometimes users think that pg_xlog
and pg_clog are just logs. And so it is fine to delete them to free up
space, corrupting their cluster, because they are just *logs*.

...and we also have "pg_logical", that includes a "log" keyword already...

Regards,

--
Devrim GÜNDÜZ
EnterpriseDB: http://www.enterprisedb.com
PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer
Twitter: @DevrimGunduz , @DevrimGunduzTR

On 26-08-2016 09:25, Devrim Gündüz wrote:

On Fri, 2016-08-26 at 21:12 +0900, Michael Paquier wrote:

In short, with the current names, sometimes users think that pg_xlog
and pg_clog are just logs. And so it is fine to delete them to free up
space, corrupting their cluster, because they are just *logs*.

...and we also have "pg_logical", that includes a "log" keyword already...

"clog" and "xlog" is almost "log"; "logical" is not. I don't imagine
people confusing "logical" meaning "log".

--
Euler Taveira Timbira - http://www.timbira.com.br/
PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Euler Taveira <euler@timbira.com.br> writes:

On 26-08-2016 09:25, Devrim Gündüz wrote:

...and we also have "pg_logical", that includes a "log" keyword already...

"clog" and "xlog" is almost "log"; "logical" is not. I don't imagine
people confusing "logical" meaning "log".

Well, I dunno; people with a weak grasp of English might have an issue
there. But I never liked that directory name anyway; my problem with
it is I read "pg_logical" and think "logical what?". Naming things using
a disconnected adjective is not good, especially one with as many
potential applications as that one has. If we're up for renaming things
under PGDATA, that one is high on my hit list.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

* Tom Lane (tgl@sss.pgh.pa.us) wrote:

Euler Taveira <euler@timbira.com.br> writes:

On 26-08-2016 09:25, Devrim Gündüz wrote:

...and we also have "pg_logical", that includes a "log" keyword already...

"clog" and "xlog" is almost "log"; "logical" is not. I don't imagine
people confusing "logical" meaning "log".

Well, I dunno; people with a weak grasp of English might have an issue
there. But I never liked that directory name anyway; my problem with
it is I read "pg_logical" and think "logical what?". Naming things using
a disconnected adjective is not good, especially one with as many
potential applications as that one has. If we're up for renaming things
under PGDATA, that one is high on my hit list.

iirc, pg_logical also has both temporary and non-temporary data in it
too. That complicates things for backup utilities that are trying to
exclude all temporary files. If we actually move all the temp files
into their own directory (or tree), then we're changing what's in
pg_logical anyway, so renaming it seems like a good idea.

Thanks!

Stephen

On 08/26/2016 03:48 AM, Magnus Hagander wrote:

Same reason I'm also +1 for Stephens suggestion to put all things that
should not be in a base backup into the same directory. That may break
things now, but it will simplify things down the road. And doing it at
the same time as renaming these things makes a lot of sense, because it
causes breakage that tool-builders *have* to look at, and then they will
hopefully also notice the other change.

If this is done this fall, developers will have at least a year to fix
their utilities.

JD

--
Command Prompt, Inc. http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.
Unless otherwise stated, opinions are my own.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 08/25/2016 07:39 PM, Michael Paquier wrote:

Hi all,

I am relaunching $subject as 10 development will begin soon. As far as
I know, there is agreement that we can do something here. Among the
different proposals I have found:
- pg_clog renamed to pg_commit_status, pg_xact or pg_commit
- pg_xlog renamed to pg_xjournal, pg_wal or pg_journal

I think the use of the pg_ prefix is redundant. Just a directory called:
wal will do (for example).

In reference to pg_xlog specifically, it should be wal. It is the Write
Ahead Log, not the Journal (although I recognize they can be
synonymous). All the documentation talks about Write Ahead Log.

Sincerely,

JD

--
Command Prompt, Inc. http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.
Unless otherwise stated, opinions are my own.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Aug 26, 2016 5:13 PM, "Joshua D. Drake" <jd@commandprompt.com> wrote:

On 08/25/2016 07:39 PM, Michael Paquier wrote:

Hi all,

I am relaunching $subject as 10 development will begin soon. As far as
I know, there is agreement that we can do something here. Among the
different proposals I have found:
- pg_clog renamed to pg_commit_status, pg_xact or pg_commit
- pg_xlog renamed to pg_xjournal, pg_wal or pg_journal

I think the use of the pg_ prefix is redundant. Just a directory called:

wal will do (for example).

In reference to pg_xlog specifically, it should be wal. It is the Write

Ahead Log, not the Journal (although I recognize they can be synonymous).
All the documentation talks about Write Ahead Log.

Yes, let's avoid inventing a *third* name for it, please. It's already bad
enough that we have both wal and xlog.

/Magnus

On 8/25/16 10:39 PM, Michael Paquier wrote:

I am relaunching $subject as 10 development will begin soon. As far as
I know, there is agreement that we can do something here. Among the
different proposals I have found:
- pg_clog renamed to pg_commit_status, pg_xact or pg_commit
- pg_xlog renamed to pg_xjournal, pg_wal or pg_journal

If we're going to do some renaming, then I suggest we do a
mini-file-system structure under $PGDATA, like

$PGDATA/etc
$PGDATA/log
$PGDATA/run (lock files etc.)
$PGDATA/tmp
$PGDATA/var

The names of all the things under "var" could still be refined, but it's
much less likely that users will confuse data with configuration or
plain logs under that scheme.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Aug 26, 2016 5:54 PM, "Peter Eisentraut" <
peter.eisentraut@2ndquadrant.com> wrote:

On 8/25/16 10:39 PM, Michael Paquier wrote:

I am relaunching $subject as 10 development will begin soon. As far as
I know, there is agreement that we can do something here. Among the
different proposals I have found:
- pg_clog renamed to pg_commit_status, pg_xact or pg_commit
- pg_xlog renamed to pg_xjournal, pg_wal or pg_journal

If we're going to do some renaming, then I suggest we do a
mini-file-system structure under $PGDATA, like

$PGDATA/etc
$PGDATA/log
$PGDATA/run (lock files etc.)
$PGDATA/tmp
$PGDATA/var

The names of all the things under "var" could still be refined, but it's
much less likely that users will confuse data with configuration or
plain logs under that scheme

Interesting idea. I worry a bit that this might encourage distributions to
split it up into different places though, and I'm not sure we want to
encourage that..

/Magnus

Magnus Hagander <magnus@hagander.net> writes:

On Aug 26, 2016 5:54 PM, "Peter Eisentraut" <
peter.eisentraut@2ndquadrant.com> wrote:

If we're going to do some renaming, then I suggest we do a
mini-file-system structure under $PGDATA, like

$PGDATA/etc
$PGDATA/log
$PGDATA/run (lock files etc.)
$PGDATA/tmp
$PGDATA/var

The names of all the things under "var" could still be refined, but it's
much less likely that users will confuse data with configuration or
plain logs under that scheme

Interesting idea. I worry a bit that this might encourage distributions to
split it up into different places though, and I'm not sure we want to
encourage that..

Yeah, I'm afraid that these names are not as well standardized as Peter
probably wishes they were. Also, I'd just as soon not move/rename things
that don't really need it. If, for example, we decide to move
postgresql.conf to etc/postgresql.conf, that is going to break a metric
ton of stuff that doesn't need to get broken AFAICS.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 08/26/2016 09:28 AM, Tom Lane wrote:

Magnus Hagander <magnus@hagander.net> writes:

On Aug 26, 2016 5:54 PM, "Peter Eisentraut" <
peter.eisentraut@2ndquadrant.com> wrote:

If we're going to do some renaming, then I suggest we do a
mini-file-system structure under $PGDATA, like

$PGDATA/etc
$PGDATA/log
$PGDATA/run (lock files etc.)
$PGDATA/tmp
$PGDATA/var

The names of all the things under "var" could still be refined, but it's
much less likely that users will confuse data with configuration or
plain logs under that scheme

Interesting idea. I worry a bit that this might encourage distributions to
split it up into different places though, and I'm not sure we want to
encourage that..

Yeah, I'm afraid that these names are not as well standardized as Peter
probably wishes they were. Also, I'd just as soon not move/rename things
that don't really need it. If, for example, we decide to move
postgresql.conf to etc/postgresql.conf, that is going to break a metric
ton of stuff that doesn't need to get broken AFAICS.

I am not so sure that is accurate. Yes, Windows is an outlying but any
Unix person is going to easily understand etc log run. Further as Linux
is by far our most run platform (outside of possibly Windows) it will
feel right at home for the largest growing user base. Personally I
really like the idea of:

$PGDATA/etc/postgresql.conf
$PGDATA/log/Thu.log
$PGDATA/run/postmaster.pid
$PGDATA/var/base/16758

Although... wouldn't run be under var?

JD

regards, tom lane

--
Command Prompt, Inc. http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 26 August 2016 at 18:28, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Also, I'd just as soon not move/rename things that don't really need it.

+1

Let's leave everything exactly as it is now... but put a small README
in each directory to explain why files in it shouldn't be deleted to
make space.

That helps the few people who made such mistakes, but doesn't cause
massive change as a result.

--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers