Prepping to break every past release...

Joshua D. Drake wrote:

Hello,

Something that continues to grind my teeth about our software is that we
are horribly inconsistent with our system catalogs. Now I am fully and
100% aware that changing this will break things in user land but I want
to do it anyway. In order to do that I believe we need to come up with a
very loud, extremely verbose method of communicating to people that 8.5
*will* break things.

It seems to me that the best method would be to follow the
information_schema naming conventions as information_schema is standard
compliant (right?).

Thoughts?

What would be the benefit? Apart from satisfying a passion for consistency?

cheers

andrew

On Wed, Mar 4, 2009 at 8:32 PM, Joshua D. Drake <jd@commandprompt.com> wrote:

Hello,

Something that continues to grind my teeth about our software is that we
are horribly inconsistent with our system catalogs. Now I am fully and
100% aware that changing this will break things in user land but I want
to do it anyway. In order to do that I believe we need to come up with a
very loud, extremely verbose method of communicating to people that 8.5
*will* break things.

I assume you'll be putting in the weeks/months of work required to fix
pgAdmin & phpPgAdmin which would be far better spent on new features
than uglifying the code in far nastier ways than the current state of
the catalogs?

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com

On Wed, 2009-03-04 at 15:50 -0500, Andrew Dunstan wrote:

It seems to me that the best method would be to follow the
information_schema naming conventions as information_schema is standard
compliant (right?).

Thoughts?

What would be the benefit? Apart from satisfying a passion for consistency?

It would lower the cost (intellectually as well as dollars) of
development and administration for every single user of the database.

I can't count how many times I accidentally type tablename versus
table_name or worse relname, etc...

Sincerely,

Joshua D. Drake

--
PostgreSQL - XMPP: jdrake@jabber.postgresql.org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

On Wed, 2009-03-04 at 21:01 +0000, Dave Page wrote:

On Wed, Mar 4, 2009 at 8:32 PM, Joshua D. Drake <jd@commandprompt.com> wrote:

Hello,

Something that continues to grind my teeth about our software is that we
are horribly inconsistent with our system catalogs. Now I am fully and
100% aware that changing this will break things in user land but I want
to do it anyway. In order to do that I believe we need to come up with a
very loud, extremely verbose method of communicating to people that 8.5
*will* break things.

I assume you'll be putting in the weeks/months of work required to fix
pgAdmin & phpPgAdmin which would be far better spent on new features
than uglifying the code in far nastier ways than the current state of
the catalogs?

*Shrug* long term consistency is the better choice.

Sincerely,

Joshua D. Drake

--
PostgreSQL - XMPP: jdrake@jabber.postgresql.org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

On Wed, Mar 4, 2009 at 9:09 PM, Joshua D. Drake <jd@commandprompt.com> wrote:

I assume you'll be putting in the weeks/months of work required to fix
pgAdmin & phpPgAdmin which would be far better spent on new features
than uglifying the code in far nastier ways than the current state of
the catalogs?

*Shrug* long term consistency is the better choice.

Easy to say if you're not one of the people for whom such a change
would mean weeks of recoding, the need to start QA'ing everything from
scratch again and a hideous mess of code to cope with after adding
support for a new version with a different catalog schema.

Besides - what percentage of users ever go anywhere near the
catalogs? I'd guess a fraction of a percent of users, and maybe 1 - 5%
of developers.

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com

On Wed, 2009-03-04 at 21:14 +0000, Dave Page wrote:

On Wed, Mar 4, 2009 at 9:09 PM, Joshua D. Drake <jd@commandprompt.com> wrote:

*Shrug* long term consistency is the better choice.

Easy to say if you're not one of the people for whom such a change
would mean weeks of recoding, the need to start QA'ing everything from
scratch again and a hideous mess of code to cope with after adding
support for a new version with a different catalog schema.

It is not easy to say. It is correct to say. I am under no illusion that
this will not be painful. As far as "man weeks of recoding". Sorry, I
know that will be tough.

Besides - what percentage of users ever go anywhere near the
catalogs? I'd guess a fraction of a percent of users, and maybe 1 - 5%
of developers.

Any DBA worth his salt uses system catalogs. Lowering the barrier on
uses these catalogs will lead to better and more useful tools as well.

Sincerely,

Joshua D. Drake

--
PostgreSQL - XMPP: jdrake@jabber.postgresql.org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

In response to Dave Page <dpage@pgadmin.org>:

On Wed, Mar 4, 2009 at 9:09 PM, Joshua D. Drake <jd@commandprompt.com> wrote:

I assume you'll be putting in the weeks/months of work required to fix
pgAdmin & phpPgAdmin which would be far better spent on new features
than uglifying the code in far nastier ways than the current state of
the catalogs?

*Shrug* long term consistency is the better choice.

Easy to say if you're not one of the people for whom such a change
would mean weeks of recoding

Don't those folks have to tweak their code with each new release anyway?
Because those tables are constantly changing? I know we hit problems
with the way triggers are stored in 8.3 compared to earlier versions.

Granted, a sweeping change will necessitate a much larger tweak than
a few changed columns, but the long-term benefit should be cleaner
client code.

Besides - what percentage of users ever go anywhere near the
catalogs? I'd guess a fraction of a percent of users, and maybe 1 - 5%
of developers.

You could always take a survey ... bosses love surveys ...

I, for one, know of lots of code that I've written that accesses those
catalogs. I can't speak for other people.

--
Bill Moran
http://www.potentialtech.com
http://people.collaborativefusion.com/~wmoran/

On Wed, Mar 4, 2009 at 9:23 PM, Joshua D. Drake <jd@commandprompt.com> wrote:

It is not easy to say. It is correct to say. I am under no illusion that
this will not be painful. As far as "man weeks of recoding". Sorry, I
know that will be tough.

It'll be a complete and utter waste of time, and make a horrible mess
of any client code that has to support multiple versions.

Besides - what percentage of users ever go anywhere near the
catalogs? I'd guess a fraction of a percent of users, and maybe 1 - 5%
of developers.

Any DBA worth his salt uses system catalogs. Lowering the barrier on
uses these catalogs will lead to better and more useful tools as well.

Then psql and pgAdmin aren't doing their jobs properly. Tell us what you need.

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com

"Joshua D. Drake" <jd@commandprompt.com> writes:

Something that continues to grind my teeth about our software is that we
are horribly inconsistent with our system catalogs. Now I am fully and
100% aware that changing this will break things in user land but I want
to do it anyway. In order to do that I believe we need to come up with a
very loud, extremely verbose method of communicating to people that 8.5
*will* break things.

It seems to me that the best method would be to follow the
information_schema naming conventions as information_schema is standard
compliant (right?).

This isn't happening. Don't waste our time suggesting it.

regards, tom lane

On Wed, Mar 4, 2009 at 9:24 PM, Bill Moran <wmoran@potentialtech.com> wrote:

In response to Dave Page <dpage@pgadmin.org>:

Don't those folks have to tweak their code with each new release anyway?
Because those tables are constantly changing?  I know we hit problems
with the way triggers are stored in 8.3 compared to earlier versions.

pg_trigger.tgenabled changed type in 8.3. That's an extremely rare
kind of change however - for example, triggers havent really changed
since at least 7.3 aside from that.

The sort of change JD seems to be suggesting /could/ involve rewriting
virtually every query in pgAdmin, as well as surrounding code.

Granted, a sweeping change will necessitate a much larger tweak than
a few changed columns, but the long-term benefit should be cleaner
client code.

Only if your app only targets one version of the server.

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com

On Wed, 2009-03-04 at 21:27 +0000, Dave Page wrote:

On Wed, Mar 4, 2009 at 9:23 PM, Joshua D. Drake <jd@commandprompt.com> wrote:

It is not easy to say. It is correct to say. I am under no illusion that
this will not be painful. As far as "man weeks of recoding". Sorry, I
know that will be tough.

It'll be a complete and utter waste of time, and make a horrible mess
of any client code that has to support multiple versions.

No actually it won't. It will make it easier for every product that uses
8.5 and above. Sometimes you have to cut the cord.

Any DBA worth his salt uses system catalogs. Lowering the barrier on
uses these catalogs will lead to better and more useful tools as well.

Then psql and pgAdmin aren't doing their jobs properly. Tell us what you need.

That is an interesting argument but honestly off topic for this thread
because that would be a series of new views and functions etc...

Sincerely,

Joshua D. Drake

--
PostgreSQL - XMPP: jdrake@jabber.postgresql.org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

On Wed, 2009-03-04 at 16:28 -0500, Tom Lane wrote:

"Joshua D. Drake" <jd@commandprompt.com> writes:

Something that continues to grind my teeth about our software is that we
are horribly inconsistent with our system catalogs. Now I am fully and
100% aware that changing this will break things in user land but I want
to do it anyway. In order to do that I believe we need to come up with a
very loud, extremely verbose method of communicating to people that 8.5
*will* break things.

It seems to me that the best method would be to follow the
information_schema naming conventions as information_schema is standard
compliant (right?).

This isn't happening. Don't waste our time suggesting it.

I wasn't try to waste anyone's time. I thought doing things correctly
was a goal of this project. Sorry for the noise.

Sincerely,

Joshua D. Drake

regards, tom lane

--
PostgreSQL - XMPP: jdrake@jabber.postgresql.org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

Joshua D. Drake wrote:

On Wed, 2009-03-04 at 21:27 +0000, Dave Page wrote:

On Wed, Mar 4, 2009 at 9:23 PM, Joshua D. Drake <jd@commandprompt.com> wrote:

It is not easy to say. It is correct to say. I am under no illusion that
this will not be painful. As far as "man weeks of recoding". Sorry, I
know that will be tough.

It'll be a complete and utter waste of time, and make a horrible mess
of any client code that has to support multiple versions.

No actually it won't. It will make it easier for every product that uses
8.5 and above. Sometimes you have to cut the cord.

I think this sounds a lot like another request for a set of system views
with nicer names. That way, there'd be no break with backwards
compatibility, and you could use the easier names if you were on the
newer versions.

//Magnus

"Joshua D. Drake" <jd@commandprompt.com> wrote:

It would lower the cost (intellectually as well as dollars) of
development and administration for every single user of the
database.

-1

Any savings couldn't possibly pay for the cost and pain of this, at
least in our shop.

I can't count how many times I accidentally type tablename versus
table_name or worse relname, etc...

I can count the times I have done so on one finger. I don't find it
that hard to keep track of what column names are used in what table.

-Kevin

Magnus Hagander <magnus@hagander.net> writes:

I think this sounds a lot like another request for a set of system views
with nicer names.

What's the state of the newsysviews project, anyway? I don't recall
hearing much about it lately.

regards, tom lane

Joshua D. Drake wrote:

Hello,

Something that continues to grind my teeth about our software is that we
are horribly inconsistent with our system catalogs. Now I am fully and
100% aware that changing this will break things in user land but I want
to do it anyway. In order to do that I believe we need to come up with a
very loud, extremely verbose method of communicating to people that 8.5
*will* break things.

It seems to me that the best method would be to follow the
information_schema naming conventions as information_schema is standard
compliant (right?).

Well, honestly, this was one of the reasons why AndrewSN, David and I
pushed NewSysViews. Non-hackers really shouldn't be accessing the
system catalogs directly, and if information_schema doesn't give them
enough data, we should start adding new views to I_S.

--Josh

Tom Lane wrote:

Magnus Hagander <magnus@hagander.net> writes:

I think this sounds a lot like another request for a set of system views
with nicer names.

What's the state of the newsysviews project, anyway? I don't recall
hearing much about it lately.

Final verdict was that we need to make it integrate better with
information_schema. At that point, our crew kinda ran out of energy and
it's been on hold ever since. But we've been talking about reviving it
again.

However, given the previous experience, I think we'd like some
reassurance that if NewSysviews was made conistent with
information_schema (and had good code, of course) that Hackers would be
interested in taking it. What nobody has any enthusiasm for is spending
a few dozen hours refactoring it, and then having to justify why we want
it *again*.

--Josh

Well, honestly, this was one of the reasons why AndrewSN, David and I

... and Elein ...

Bill,

You could always take a survey ... bosses love surveys ...

Done: http://wwwmaster.postgresql.org/community/

--Josh

Josh Berkus wrote:

Well, honestly, this was one of the reasons why AndrewSN, David and I

... and Elein ...

pushed NewSysViews. Non-hackers really shouldn't be accessing the
system catalogs directly, and if information_schema doesn't give them
enough data, we should start adding new views to I_S.

This is really a long way from JD's original suggestion, though.

Back on that track, I'd like to see a facility whereby we could provide
an alias (or synonym, to use a nearby subject) columns and other
objects. That would help to overcome naming glitches without breaking
things quite so much.

cheers

andrew

Andrew Dunstan <andrew@dunslane.net> wrote:

Back on that track, I'd like to see a facility whereby we could

provide

an alias (or synonym, to use a nearby subject) columns and other
objects. That would help to overcome naming glitches without breaking

things quite so much.

How is that different from what you can do with a view?

-Kevin

Andrew,

Back on that track, I'd like to see a facility whereby we could provide
an alias (or synonym, to use a nearby subject) columns and other
objects. That would help to overcome naming glitches without breaking
things quite so much.

Believe it or not, a large PostgreSQL user in LA just buttonholed me
about that particular feature idea at SCALE. So it might be generally
useful as well -- not just for the system catalogs, bug to allow
businesses with long-use databases to manage change over time.

--Josh

Josh Berkus <josh@agliodbs.com> wrote:

You could always take a survey ... bosses love surveys ...

Done: http://wwwmaster.postgresql.org/community/

Is there some assumed relationship between those options and whether
they'd like to see the names changed?

Also, it's hard to pick what option to choose there -- I typically
have cause about once or twice a month to poke around in those tables,
and we have development tools which reference these tables
directly....

I guess the closest option would be "access them all the time." I
hope that wouldn't be interpreted as supporting the proposed change.

-Kevin

Kevin Grittner wrote:

Josh Berkus <josh@agliodbs.com> wrote:

You could always take a survey ... bosses love surveys ...

Done: http://wwwmaster.postgresql.org/community/

Is there some assumed relationship between those options and whether
they'd like to see the names changed?

Also, it's hard to pick what option to choose there -- I typically
have cause about once or twice a month to poke around in those tables,
and we have development tools which reference these tables
directly....

I guess the closest option would be "access them all the time." I
hope that wouldn't be interpreted as supporting the proposed change.

I'm not taking a survey on how/whether we want to change it. Just
whether people are accessing them directly.

--Josh

On Mar 4, 2009, at 6:07 PM, Josh Berkus wrote:

Andrew,

Back on that track, I'd like to see a facility whereby we could
provide an alias (or synonym, to use a nearby subject) columns and
other objects. That would help to overcome naming glitches without
breaking things quite so much.

Believe it or not, a large PostgreSQL user in LA just buttonholed me
about that particular feature idea at SCALE. So it might be
generally useful as well -- not just for the system catalogs, bug to
allow businesses with long-use databases to manage change over time.

Schema change is a strong motivator for applications to access the
database through views and functions only. A column with multiple
names would likely make it *more* painful to migrate schemata.

Cheers,
M

"Tom" == Tom Lane <tgl@sss.pgh.pa.us> writes:

Magnus Hagander <magnus@hagander.net> writes:

I think this sounds a lot like another request for a set of system
views with nicer names.

Tom> What's the state of the newsysviews project, anyway? I don't
Tom> recall hearing much about it lately.

At the time it was proposed for inclusion (pre 8.1, when it was mostly
but not entirely complete), you personally, as I recall, expressed the
opinion that its objective was impossible; that there was no way to
produce a sufficiently complete set of views that was more stable and
compatible between releases than the system catalogs themselves were.
I believe these sum up your response:

http://archives.postgresql.org/pgsql-hackers/2005-05/msg00351.php
http://archives.postgresql.org/pgsql-hackers/2005-05/msg00891.php
http://archives.postgresql.org/pgsql-hackers/2005-05/msg00940.php

Another common response at the time was "just use information_schema".
(Which is of course inadequate for a huge number of reasons - speed,
hiding of "implementation details", mismatches between pg's behaviour
and the SQL spec, and so on.)

Given the extent to which this criticism of the project was based on
speculation ("we might make changes in future releases that would
break the views"), there was no better answer at the time than "we
think the design is flexible enough to handle that", and very little
of the pushback we got actually showed any signs of having reviewed
the design and (admittedly incomplete) implementation. Accordingly I
stopped spending any time on it and diverted my attention elsewhere.

Now, of course, counting the upcoming 8.4 there have been three (and a
bit - the original design predates 8.1, though it did anticipate some
8.1 features) new releases against which the original concept can be
tested. And, guess what, nothing in those releases has even come close
to invalidating the original design concept (as we knew all along).

If you're still not convinced of that fact, it would be possible to
take the original design and update it to 8.4 following the original
plan. But I'm not prepared to spend any time on this if the only result
is going to be more argument.

--
Andrew (irc:RhodiumToad)

On Wed, Mar 4, 2009 at 3:32 PM, Joshua D. Drake <jd@commandprompt.com> wrote:

Something that continues to grind my teeth about our software is that we
are horribly inconsistent with our system catalogs. Now I am fully and
100% aware that changing this will break things in user land but I want
to do it anyway. In order to do that I believe we need to come up with a
very loud, extremely verbose method of communicating to people that 8.5
*will* break things.

It seems to me that the best method would be to follow the
information_schema naming conventions as information_schema is standard
compliant (right?).

Thoughts?

Like everyone else who has responded to this thread, I think this is a
pretty terrible idea. It's possible that there are some specific
columns in some specific tables that could stand to be renamed for
consistency, and perhaps if you come up with some specific proposals
with careful justifications someone might support the idea of doing
some limited renaming. But too much renaming is not likely to be
popular with anyone for reasons that are somewhat summed up by your
subject line.

And, really, how much better would the new names be than the old ones
anyway? The idea that a casual user will be able to query the system
catalogs and gain some sort of useful information without reading the
documentation or at least cracking out a couple of \d commands strikes
me as a pipe dream. I'll admit that I'm a little mystified by why we
use pg_class to store relations (why not pg_relation?), relnamespace
to store the schema oid (why not relschema?), and so on, so some
improvement is probably possible. But I'm not sure you're going to be
able to come up with a name that's substantially clearer than
proargmodes. Sure, you could call it argument_modes, but that's not
really any clearer, it's just longer. In fact, it's my experience
that exercises of this type almost always end up replacing shorter
names with longer names without really making anything any better. In
the end you still have to RTFM.

...Robert

On Wed, 4 Mar 2009, Joshua D. Drake wrote:

Something that continues to grind my teeth about our software is that we
are horribly inconsistent with our system catalogs.

I'd suggest staring at the gyrations required to do an in-place upgrade,
then considering how the changes you're suggesting will make that even
more difficult, until this inconsistency no longer seems very important.

It seems to me that the best method would be to follow the
information_schema naming conventions as information_schema is standard
compliant (right?).

You could make a case for changing pg_tables.tablename to
pg_tables.table_name, so that it better matched the information_schema.
But it's not like that general approach makes this problem go away.
You'll still have pg_class.rel_name or relname, because pg_class contains
several types of relations: tables, indexes, etc. Since that particular
mismatch is impossible to resolve, you can't completely simplify this area
no matter how hard you try. That makes it hard to get excited about just
reducing the number of inconsistencies here.

Any DBA worth his salt uses system catalogs. Lowering the barrier on
uses these catalogs will lead to better and more useful tools as well.

I would wager that putting 1% of the total effort needed to actually
change the catalog schema names into a documentation/sample code push in
this area would give a bigger payback. I never write catalog stuff from
scratch anymore; I go back to the same couple of template pieces I always
end up needing for the various types of joins that always pop up and
customize from there.

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD

* Robert Haas (robertmhaas@gmail.com) wrote:

Like everyone else who has responded to this thread, I think this is a
pretty terrible idea.

[...]

+1 on Robert's assessment from me. I'm generally a huge fan of doing
the right thing, but as my boss often likes to point out, you need to
pick your battles wisely.

Thanks,

Stephen

On Wed, Mar 04, 2009 at 02:35:52PM -0800, Josh Berkus wrote:

Tom Lane wrote:

Magnus Hagander <magnus@hagander.net> writes:

I think this sounds a lot like another request for a set of system views
with nicer names.

What's the state of the newsysviews project, anyway? I don't recall
hearing much about it lately.

Final verdict was that we need to make it integrate better with
information_schema. At that point, our crew kinda ran out of energy and
it's been on hold ever since. But we've been talking about reviving it
again.

The information schema has things mandated by the SQL standard, and so
we really need to stay inside the lines with it. By its nature, it
must hide "implementation details" which newsysviews can expose, so
now that there's a track record of 5 versions of compatible
newsysviews, I think we should make it a schema that ships with every
database, starting with 8.5.

Cheers,
David.
--
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

If this is the worst inconsistency you can find in our system tables
after +20 years of development, I feel pretty good.

---------------------------------------------------------------------------

Joshua D. Drake wrote:

Hello,

Something that continues to grind my teeth about our software is that we
are horribly inconsistent with our system catalogs. Now I am fully and
100% aware that changing this will break things in user land but I want
to do it anyway. In order to do that I believe we need to come up with a
very loud, extremely verbose method of communicating to people that 8.5
*will* break things.

It seems to me that the best method would be to follow the
information_schema naming conventions as information_schema is standard
compliant (right?).

Thoughts?

Examples:

postgres=# \d pg_class
Table "pg_catalog.pg_class"
Column | Type | Modifiers
----------------+-----------+-----------
relname | name | not null
relnamespace | oid | not null
[...]

postgres=# \d pg_tables
View "pg_catalog.pg_tables"
Column | Type | Modifiers
-------------+---------+-----------
schemaname | name |
tablename | name |

postgres=# \d pg_stat_user_tables
View "pg_catalog.pg_stat_user_tables"
Column | Type | Modifiers
------------------+--------------------------+-----------
relid | oid |
schemaname | name |
relname | name |

postgres=# \d information_schema.tables
View "information_schema.tables"
Column | Type |
Modifiers
------------------------------+-----------------------------------+-----------
table_catalog | information_schema.sql_identifier |
table_schema | information_schema.sql_identifier |
table_name | information_schema.sql_identifier |

--
PostgreSQL - XMPP: jdrake@jabber.postgresql.org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Mon, 2009-03-09 at 13:59 -0400, Bruce Momjian wrote:

If this is the worst inconsistency you can find in our system tables
after +20 years of development, I feel pretty good.

I was using a single example. This would be a large project I am sure
and of course we should feel good. In all I would say we are likely one
of the more consistent pieces of software in terms of our age. That
doesn't mean we shouldn't try to continue to improve.

Sincerely,

Joshua D. Drake

--
PostgreSQL - XMPP: jdrake@jabber.postgresql.org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

On Wed, 2009-03-04 at 12:32 -0800, Joshua D. Drake wrote:

Something that continues to grind my teeth about our software is that we
are horribly inconsistent with our system catalogs. Now I am fully and
100% aware that changing this will break things in user land but I want
to do it anyway. In order to do that I believe we need to come up with a
very loud, extremely verbose method of communicating to people that 8.5
*will* break things.

I agree strongly with your general point.

The most consistent negative feedback I receive about Postgres is that
we make minor changes from release to release that make it extremely
difficult to upgrade without re-testing the applications. So we write
great software, then make it difficult for people to upgrade to it.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

On Thu, 2009-03-05 at 01:27 +0000, Andrew Gierth wrote:

Now, of course, counting the upcoming 8.4 there have been three (and a
bit - the original design predates 8.1, though it did anticipate some
8.1 features) new releases against which the original concept can be
tested. And, guess what, nothing in those releases has even come close
to invalidating the original design concept (as we knew all along).

If you're still not convinced of that fact, it would be possible to
take the original design and update it to 8.4 following the original
plan. But I'm not prepared to spend any time on this if the only
result is going to be more argument.

I see the use for some more stable views.

Would it be better to publish them as an external project? That way we
can still use them for both old and new releases. Once the project takes
hold it might then be included in core - but that's not hugely important
if you can persuade people to include the project with the Windows
installer.

The problem with anything included in core is that we don't/can't
quickly fix design flaws, so even if we did get something in now it
might not do everything we want (and then we'd have to change it...).

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

"Simon" == Simon Riggs <simon@2ndQuadrant.com> writes:

Now, of course, counting the upcoming 8.4 there have been three
(and a bit - the original design predates 8.1, though it did
anticipate some 8.1 features) new releases against which the
original concept can be tested. And, guess what, nothing in those
releases has even come close to invalidating the original design
concept (as we knew all along).

If you're still not convinced of that fact, it would be possible
to take the original design and update it to 8.4 following the
original plan. But I'm not prepared to spend any time on this if
the only result is going to be more argument.

Simon> I see the use for some more stable views.

Simon> Would it be better to publish them as an external project?

They already are, though they are not complete and have not been
maintained much for 8.1 and later;
http://pgfoundry.org/projects/newsysviews/

Simon> That way we can still use them for both old and new releases.

It was always expected that they would be available on pgfoundry for
use on releases prior to their inclusion in core.

Simon> Once the project takes hold it might then be included in core

Speaking purely for myself, I'm not prepared to spend any time on it
without an assurance that it will go into core if the project goals
are reasonably met.

As for Tom's opinion that this is impossible, there's an old saying:
"The one who says it cannot be done should not interrupt the one who
is doing it."

Simon> The problem with anything included in core is that we
Simon> don't/can't quickly fix design flaws, so even if we did get
Simon> something in now it might not do everything we want (and then
Simon> we'd have to change it...).

I'm not proposing that it go into core quickly; and certainly not
before the design is properly reviewed, criticised, whatever.

--
Andrew.

On Tue, Mar 10, 2009 at 08:46:28AM +0000, Simon Riggs wrote:

On Thu, 2009-03-05 at 01:27 +0000, Andrew Gierth wrote:

Now, of course, counting the upcoming 8.4 there have been three (and a
bit - the original design predates 8.1, though it did anticipate some
8.1 features) new releases against which the original concept can be
tested. And, guess what, nothing in those releases has even come close
to invalidating the original design concept (as we knew all along).

If you're still not convinced of that fact, it would be possible to
take the original design and update it to 8.4 following the original
plan. But I'm not prepared to spend any time on this if the only
result is going to be more argument.

I see the use for some more stable views.

Would it be better to publish them as an external project?

It's been an external project, newsysviews, since before 8.1 came out.
I think it's time to bring it in from the cold. Call the new schema
pg_sysviews, plop it in there, and call it done :)

Cheers,
David.
--
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

On Tue, 2009-03-10 at 07:28 -0700, David Fetter wrote:

Would it be better to publish them as an external project?

It's been an external project, newsysviews, since before 8.1 came out.
I think it's time to bring it in from the cold. Call the new schema
pg_sysviews, plop it in there, and call it done :)

Yeh Andrew said. That I never noticed in the last 3+ years makes me
think there's not many people using it...

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

On Tue, 2009-03-10 at 15:02 +0000, Simon Riggs wrote:

On Tue, 2009-03-10 at 07:28 -0700, David Fetter wrote:

Would it be better to publish them as an external project?

It's been an external project, newsysviews, since before 8.1 came out.
I think it's time to bring it in from the cold. Call the new schema
pg_sysviews, plop it in there, and call it done :)

Yeh Andrew said. That I never noticed in the last 3+ years makes me
think there's not many people using it...

Well I know of it and have never used it. Mainly because I didn't (and
still don't) really know what it does. From an outsider looking in, the
project is dead. The home page isn't updated (it talks about 8.1) and
the CVS repo appears to not have had a commit in 2 years.

How is anyone in the general community supposed to have any idea if this
is a good idea or not?

Sincerely,

Joshua D. Drake

--
PostgreSQL - XMPP: jdrake@jabber.postgresql.org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

"Joshua" == Joshua D Drake <jd@commandprompt.com> writes:

On Tue, 2009-03-10 at 15:02 +0000, Simon Riggs wrote:

Yeh Andrew said. That I never noticed in the last 3+ years makes
me think there's not many people using it...

The fact that it never got beyond an early incomplete alpha version is
a big factor in that.

Joshua> Well I know of it and have never used it. Mainly because I
Joshua> didn't (and still don't) really know what it does. From an
Joshua> outsider looking in, the project is dead. The home page isn't
Joshua> updated (it talks about 8.1) and the CVS repo appears to not
Joshua> have had a commit in 2 years.

Other than some experiments in getting it to load on 8.2, there hasn't
been any serious work done on it since May 2005, which is when it was
presented (and shot down) on -hackers.

The lack of useful feedback from -hackers also means that the design
hasn't had much criticism, and therefore I don't regard the current
definitions, the naming conventions, etc., as being cast in stone;
which is another reason for people not to use it as it stands.

(The plan we had when we started on it was to produce an alpha version
as a proof-of-concept, present it on -hackers, get feedback, use that
to sort out the naming conventions and a definitive set of
definitions, and produce a beta version intended to be in the final
form.)

--
Andrew.

On Tue, Mar 10, 2009 at 5:23 PM, Andrew Gierth
<andrew@tao11.riddles.org.uk> wrote:

Other than some experiments in getting it to load on 8.2, there hasn't
been any serious work done on it since May 2005, which is when it was
presented (and shot down) on -hackers.

If memory serves (and it may not - I'm practically brain dead from
reviewing a large pgAdmin patch all day) - much of the 'shooting down'
was at the suggestion that pgAdmin (and the like) should stop using
the catalogues directly and should use newsysviews instead. I still
maintain that'll never happen, but that doesn't mean that newsysviews
wouldn't be useful for other classes of user. Perhaps pgsql-general
would be a better place to poll.

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com

Simon Riggs wrote:

The most consistent negative feedback I receive about Postgres is that
we make minor changes from release to release that make it extremely
difficult to upgrade without re-testing the applications. So we write
great software, then make it difficult for people to upgrade to it.

Then I would maintain that part of that makes the software great is that
we have the ability to make incompatible changes once in a while,
avoiding the accumulation of cruft. We do maintain old releases for 5
years as compensation.

I did propose a deprecation policy that would address your concern to
some degree by issuing warnings in release N-1, so the testing after
upgrade can be taken care of for the most part by hunting down these
warnings while running the previous release. That didn't receive
universal support, but I think we should still look for a compromise in
that area.

The argument against was that this would slow down PostgreSQL
development too much. And note that the one-year major release cycle of
PostgreSQL is already pretty much the shortest one of any software of
this complexity.

So everyone has different expectations, it seems.

On Wed, 2009-03-11 at 08:33 +0200, Peter Eisentraut wrote:

Simon Riggs wrote:

The most consistent negative feedback I receive about Postgres is that
we make minor changes from release to release that make it extremely
difficult to upgrade without re-testing the applications. So we write
great software, then make it difficult for people to upgrade to it.

Then I would maintain that part of that makes the software great is that
we have the ability to make incompatible changes once in a while,
avoiding the accumulation of cruft. We do maintain old releases for 5
years as compensation.

Please remember I'm just the messenger, passing on client feedback. It
hasn't ever been my way to act this way, but the reality is that
difficult upgrades make for more consulting income. The cost to the
client is much higher because of re-test costs, difficulty in supporting
applications across different sites running different PG releases and
general delay.

We're getting very good at doing upgrades now...

I did propose a deprecation policy that would address your concern to
some degree by issuing warnings in release N-1, so the testing after
upgrade can be taken care of for the most part by hunting down these
warnings while running the previous release. That didn't receive
universal support, but I think we should still look for a compromise in
that area.

I agree with the need for a deprecation policy or approach to this
issue.

I think that particular deprecation policy was too strong, but where
possible, it would be good to have a way to avoid niggly changes of
behaviour. We have done that sometimes, e.g. sort_mem is now a synonym
for work_mem, just not consistently. An example solution might be a
parameter that allowed us to act like the previous release in some
aspects. A parameter for every behaviour change would be bad because
that's just another minefield to cross.

The first step is to record incompatibilities as they occur and record
them somewhere, so that people can say "that'll break my app". Often the
first people hear about these things is when we compile the release
notes, which is far too late either to complain or to fix.

The argument against was that this would slow down PostgreSQL
development too much. And note that the one-year major release cycle of
PostgreSQL is already pretty much the shortest one of any software of
this complexity.

You know I would not agree to that.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

On Wed, 2009-03-11 at 08:41 +0000, Simon Riggs wrote:

On Wed, 2009-03-11 at 08:33 +0200, Peter Eisentraut wrote:

The first step is to record incompatibilities as they occur and record
them somewhere, so that people can say "that'll break my app". Often the
first people hear about these things is when we compile the release
notes, which is far too late either to complain or to fix.

That is a simple modification of the release notes and something that
really should be done regardless of a deprecation policy.

Joshua D. Drake

--
PostgreSQL - XMPP: jdrake@jabber.postgresql.org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

On Mar 4, 2009, at 5:07 PM, Josh Berkus wrote:

Back on that track, I'd like to see a facility whereby we could
provide an alias (or synonym, to use a nearby subject) columns and
other objects. That would help to overcome naming glitches without
breaking things quite so much.

Believe it or not, a large PostgreSQL user in LA just buttonholed
me about that particular feature idea at SCALE. So it might be
generally useful as well -- not just for the system catalogs, bug
to allow businesses with long-use databases to manage change over
time.

Yes, I think aliasing (especially at the table level) would be handy.

And +1 on reviving newsysviews, but of course I'm biased... ;P
--
Decibel!, aka Jim C. Nasby, Database Architect decibel@decibel.org
Give your computer some brain candy! www.distributed.net Team #1828

Jim,

Yes, I think aliasing (especially at the table level) would be handy.

We already *have* table aliases. They're called "views". What we don't
have is column aliases.

However, for column aliases to be really useful for more than just
application refactoring, we'd have to support calculated column aliases,
which is much more complicated.

And +1 on reviving newsysviews, but of course I'm biased... ;P

Oh, right, forgot to credit you as well. Sorry.

--Josh

All,

For anyone who cares, we have some unscientific results on the system
views survey:

http://www.postgresql.org/community/survey.60

--Josh

On Mar 14, 2009, at 1:26 PM, Josh Berkus wrote:

Yes, I think aliasing (especially at the table level) would be handy.

We already *have* table aliases. They're called "views". What we
don't have is column aliases.

A view is not the same as a table alias. Even if you take into
account the new updatable views, you're still hosed if you add a
column to the table. I see that being a lot more useful than a simple
column alias (you're correct that we'd need to support calculated
ones, which is indeed a lot harder).
--
Decibel!, aka Jim C. Nasby, Database Architect decibel@decibel.org
Give your computer some brain candy! www.distributed.net Team #1828