Contributing with code

On 12/27/17 15:18, Antonio Belloni wrote:

I checked the code of pg_tablespace_databases, defined in
src/backend/utils/adt/misc.c, and see that it uses a much simpler
approach : It just reads the tablespaces directories and return the name
of the directories that represents databases OIDs. 

Although the function works as expected, I  can see some issues not
addressed in the code :

- It does not check for permissions. Any user can execute it;
- It does not check if the platform supports symlinks, which can cause
an error because the function is trying to follow the links defined in
base/pg_tblspc.

I could use the same approach and write a function that goes down one
more level in the directory structure and find the objects' OIDs inside
each database directory, but I don't know if this is the better way to
do that.

The information of what object is in what tablespace already exists in
the system catalogs, so I don't think we need new ways to discover that
information. What the todo item referred to, I think, was having a way
to discover that across database boundaries. But I think that problem
is unsolvable by design, because you can't look into other databases.
Looking into the file system to discover the OIDs of objects might work,
but then you can't do anything with those OIDs without having access to
the respective database to resolve them.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 28 December 2017 at 23:05, Peter Eisentraut <
peter.eisentraut@2ndquadrant.com> wrote:

On 12/27/17 15:18, Antonio Belloni wrote:

I checked the code of pg_tablespace_databases, defined in
src/backend/utils/adt/misc.c, and see that it uses a much simpler
approach : It just reads the tablespaces directories and return the name
of the directories that represents databases OIDs.

Although the function works as expected, I can see some issues not
addressed in the code :

- It does not check for permissions. Any user can execute it;
- It does not check if the platform supports symlinks, which can cause
an error because the function is trying to follow the links defined in
base/pg_tblspc.

I could use the same approach and write a function that goes down one
more level in the directory structure and find the objects' OIDs inside
each database directory, but I don't know if this is the better way to
do that.

The information of what object is in what tablespace already exists in
the system catalogs, so I don't think we need new ways to discover that
information. What the todo item referred to, I think, was having a way
to discover that across database boundaries. But I think that problem
is unsolvable by design, because you can't look into other databases.
Looking into the file system to discover the OIDs of objects might work,
but then you can't do anything with those OIDs without having access to
the respective database to resolve them.

Well, it's arguably solveable, if the solution is to allow at least limited
cross-database access to catalog relations.

I don't see any compelling reason we can't do it for catalogs, because:

- Catalog oids are compiled in and the same across all DBs;

- Catalog structure is also compiled in and the same across all DBs, using
Form_ structs and constants usable with fastgetattr

- We already have two relmapper instances (shared and per-db), so we should
be able to instantiate one for the other DB's per-db relmapper and read its
relation mapping to get oid-to-relfilenode mappings for nailed relations.

- Non-shared non-nailed relfilenodes could be discovered by reading the
other DB's pg_class. You couldn't use relcache lookups, but that shouldn't
matter.

The same is true for catalog indexes.

We can safely assume that all type oids, etc, that appear in catalog
relations are the same in our connected DB as the other DB, so it's safe to
use the syscache when reading "foreign" catalog relations.

It's nowhere near as practical to read *user* tables from another DB, since
we'd need a way to switch the active syscache and relcache, among numerous
other things. But all the restrictions we place on catalogs mean I think
cross-db catalog reads should actually be practical, at least as far as
reading the other db's pg_class and relmapper goes. Most of the juicy bits
are hardcoded.

AFAICS it'd be necessary to extend the relmapper interface to let you
instantiate new maps by dboid and pass them explicitly to the various
lookup functions. Relmapper invalidations would be a problem, I don't see
how it'd be practical to handle them, you'd have to be able to register as
interested in another DB's invals somehow. And you'd have no locking. So
you could probably only use it for things like this - mapping relfilenodes
to oids/relnames for other DBs.

Worth the effort? Probably not. But potentially fun.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Thu, Dec 28, 2017 at 5:42 PM, Craig Ringer <craig@2ndquadrant.com> wrote:

Well, it's arguably solveable, if the solution is to allow at least limited
cross-database access to catalog relations.

I think this is a bad idea. It's bound to add complexity and
fragility to the system and I don't think it's worth it.

Also, let's delete the TODO list. People keep using it as a source of
project ideas, and that's bad.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas <robertmhaas@gmail.com> writes:

Also, let's delete the TODO list. People keep using it as a source of
project ideas, and that's bad.

If we're not going to maintain/curate it properly, I agree it's not
worth keeping it around. But I'd rather see somebody put some effort
into it ...

regards, tom lane

Tom, Robert, all,

* Tom Lane (tgl@sss.pgh.pa.us) wrote:

Robert Haas <robertmhaas@gmail.com> writes:

Also, let's delete the TODO list. People keep using it as a source of
project ideas, and that's bad.

If we're not going to maintain/curate it properly, I agree it's not
worth keeping it around. But I'd rather see somebody put some effort
into it ...

I don't see anything particularly wrong with the specific item chosen
off of the todo list- it'd be nice to be able to tell what objects exist
in a tablespace even if they're in other databases.

The todo entry even talks about why it's difficult to do and what the
expected way to go about doing it is (that is, connect to each database
that has objects in the tablespace and query it to find out what's in
the tablespace). Craig's suggestion is an interesting alternative way
though and I'm not sure that it'd be all that bad, but it would be
limited to catalog tables.

If we'd extend the system to allow transparent usage of postgres_fdw to
access other databases which are part of the same cluster, then this
could end up being much simpler (eg: select * from
otherdatabase.pg_catalog.pg_class ...).

Thanks!

Stephen

On Sun, Dec 31, 2017 at 11:42 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Robert Haas <robertmhaas@gmail.com> writes:

Also, let's delete the TODO list. People keep using it as a source of
project ideas, and that's bad.

If we're not going to maintain/curate it properly, I agree it's not
worth keeping it around. But I'd rather see somebody put some effort
into it ...

​It probably needs three sub-sections. Fist the raw ideas put forth by
people not capable of implementation but needing capabilities; these get
moved to one of two sections: ideas that have gotten some attention by core
that have merit but don't hav​e development interest presently; and one
like this that have gotten the some attention and that core doesn't feel
would be worth maintaining even if someone was willing to develop it. We
already have this in practice but maybe a bit more formality would help.

I'm not seeing that having it, even if incorrect, does harm. Those who
would develop from it should be using it as a conversation starter and even
if it is discovered that the feature is no longer desirable or applicable
that conversation will have been worthwhile to that person and probably
many others browsing these mailing lists. I do think that all of the
commit-fest managers (and release note writers/reviewers) for the release
year could be asked/reminded to at least skim over the ToDo list at the end
of their session and see whether anything was addressed by one of the
commits that went in during their tenure and remove (or update) them.

A thread started by "hey, lets talk about this ToDo list item" is one that
reinforces the value of having such a list - and at worse should result in
that particular entry being updated in some fashion.

David J.

On Sun, Dec 31, 2017 at 6:35 PM, Robert Haas <robertmhaas@gmail.com> wrote:

Also, let's delete the TODO list. People keep using it as a source of
project ideas, and that's bad.

It would be great if the TODO list was deleted IMV.

Choosing the right project is one of the hardest and most important
parts of successfully contributing to PostgreSQL. It's usually
essential to understand in detail why the thing that you're thinking
of working on doesn't already exist. The TODO list seems to suggest
almost the opposite, and as such is a trap for inexperienced hackers.

--
Peter Geoghegan

On Sun, Dec 31, 2017 at 10:42 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

If we're not going to maintain/curate it properly, I agree it's not
worth keeping it around. But I'd rather see somebody put some effort
into it ...

If somebody was going to resolve to put some effort into maintaining
it to a high standard then it probably would have happened already.
The fact that it hasn't happened tells us plenty.

--
Peter Geoghegan

On 1 January 2018 at 03:27, Peter Geoghegan <pg@bowt.ie> wrote:

On Sun, Dec 31, 2017 at 6:35 PM, Robert Haas <robertmhaas@gmail.com>
wrote:

Also, let's delete the TODO list. People keep using it as a source of
project ideas, and that's bad.

It would be great if the TODO list was deleted IMV.

Choosing the right project is one of the hardest and most important
parts of successfully contributing to PostgreSQL. It's usually
essential to understand in detail why the thing that you're thinking
of working on doesn't already exist. The TODO list seems to suggest
almost the opposite, and as such is a trap for inexperienced hackers.

I don't entirely agree. It's a useful place to look for "are there other
things related to the thing I'm contemplating doing" and "has anyone tried
this? where did they get stuck?"

I'd rather rename it the "stuck, hard and abandoned projects list" ;)

For example I try to keep track of protocol-related stuff there, so that
if/when we do a real protocol revision we don't fail to consider things
that have come up and since been forgotten.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Sun, Dec 31, 2017 at 2:31 PM, Peter Geoghegan <pg@bowt.ie> wrote:

On Sun, Dec 31, 2017 at 10:42 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

If we're not going to maintain/curate it properly, I agree it's not
worth keeping it around. But I'd rather see somebody put some effort
into it ...

If somebody was going to resolve to put some effort into maintaining
it to a high standard then it probably would have happened already.
The fact that it hasn't happened tells us plenty.

+1, and well said.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Sun, Dec 31, 2017 at 1:51 PM, Stephen Frost <sfrost@snowman.net> wrote:

The todo entry even talks about why it's difficult to do and what the
expected way to go about doing it is (that is, connect to each database
that has objects in the tablespace and query it to find out what's in
the tablespace). Craig's suggestion is an interesting alternative way
though and I'm not sure that it'd be all that bad, but it would be
limited to catalog tables.

I think it'd be pretty bad. There's nothing in the system that
actually guarantees that the system catalog structure matches across
every database. Of course, if you change structural properties, then
the system will probably crash, but attstorage and attacl values could
be different, as could relfilenode, relpages, reltuples,
relallvisible, relfroxenxid, relminmxid, and relacl. I don't think
it's wise to let this work on the theory that none of that stuff
matters. Even if that's true, or can be made true with a crowbar,
it's a fragile assumption that might turn false in the future.

If we'd extend the system to allow transparent usage of postgres_fdw to
access other databases which are part of the same cluster, then this
could end up being much simpler (eg: select * from
otherdatabase.pg_catalog.pg_class ...).

It would probably be better to use background workers for this than
postgres_fdw to avoid for example making sure you can authenticate,
but even then this is a pretty significant body of work for what I'd
consider a fairly marginal benefit.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Sun, Dec 31, 2017 at 2:02 PM, David G. Johnston
<david.g.johnston@gmail.com> wrote:

It probably needs three sub-sections. Fist the raw ideas put forth by
people not capable of implementation but needing capabilities; these get
moved to one of two sections: ideas that have gotten some attention by core
that have merit but don't have development interest presently; and one like
this that have gotten the some attention and that core doesn't feel would be
worth maintaining even if someone was willing to develop it. We already
have this in practice but maybe a bit more formality would help.

I'm not seeing that having it, even if incorrect, does harm.

It causes people to waste time developing features we don't want.

It also has a note at the top saying we think it's complete, but we
don't think that, or I don't think it anyway.

It's basically disinformation.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert,

* Robert Haas (robertmhaas@gmail.com) wrote:

On Sun, Dec 31, 2017 at 1:51 PM, Stephen Frost <sfrost@snowman.net> wrote:

The todo entry even talks about why it's difficult to do and what the
expected way to go about doing it is (that is, connect to each database
that has objects in the tablespace and query it to find out what's in
the tablespace). Craig's suggestion is an interesting alternative way
though and I'm not sure that it'd be all that bad, but it would be
limited to catalog tables.

I think it'd be pretty bad. There's nothing in the system that
actually guarantees that the system catalog structure matches across
every database. Of course, if you change structural properties, then
the system will probably crash, but attstorage and attacl values could
be different, as could relfilenode, relpages, reltuples,
relallvisible, relfroxenxid, relminmxid, and relacl. I don't think
it's wise to let this work on the theory that none of that stuff
matters. Even if that's true, or can be made true with a crowbar,
it's a fragile assumption that might turn false in the future.

I'm surprised to hear it described as fragile when I would certainly
expect a huge amount of push-back from the developer community if
someone suggested making template1 have a different catalog structure
from the other databases for some reason. This seems more like an
unwritten rule to me than a 'it happens to work that way' kind of thing.

If we'd extend the system to allow transparent usage of postgres_fdw to
access other databases which are part of the same cluster, then this
could end up being much simpler (eg: select * from
otherdatabase.pg_catalog.pg_class ...).

It would probably be better to use background workers for this than
postgres_fdw to avoid for example making sure you can authenticate,
but even then this is a pretty significant body of work for what I'd
consider a fairly marginal benefit.

We could address the authentication issue, I believe, internally such
that users wouldn't have to actually worry about it (perhaps a special
entry in pg_hba.conf and something done through shared memory), which
would definitely be part of the point- what I was trying to get at above
is that we could possibly solve this in a much more general way by
supporting, more-or-less transparently, cross-database queries, which
would be a terribly useful feature, imv.

I agree that it'd be a significant body of work, but we would gain a
great deal more than just the ability to query the catalog of other
databases and that seems quite worthwhile to me.

Thanks!

Stephen

On 12/31/17 22:43, Craig Ringer wrote:

I'd rather rename it the "stuck, hard and abandoned projects list" ;) 

That might actually be useful.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 01/02/2018 11:17 AM, Robert Haas wrote:

On Sun, Dec 31, 2017 at 2:31 PM, Peter Geoghegan <pg@bowt.ie> wrote:

On Sun, Dec 31, 2017 at 10:42 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

If we're not going to maintain/curate it properly, I agree it's not
worth keeping it around. But I'd rather see somebody put some effort
into it ...

If somebody was going to resolve to put some effort into maintaining
it to a high standard then it probably would have happened already.
The fact that it hasn't happened tells us plenty.

+1, and well said.

O.k. what does it tell us though? Is it a resource issue? Is it a
barrier of entry issue? What does deleting it solve? What problems (and
there is a very large obvious one) are caused by deleting it?

Right now, the TODO list is the "only" portal to "potential" things we
"might" want. If we delete it we are just creating yet another barrier
of entry to potential contribution. I think we need to consider an
alternative solution because of that.

Thanks,

JD

--
Command Prompt, Inc. || http://the.postgres.company/ || @cmdpromptinc

PostgreSQL centered full stack support, consulting and development.
Advocate: @amplifypostgres || Learn: https://postgresconf.org
***** Unless otherwise stated, opinions are my own. *****

On 2 January 2018 at 17:52, Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:

On 12/31/17 22:43, Craig Ringer wrote:

I'd rather rename it the "stuck, hard and abandoned projects list" ;)

That might actually be useful.

Yep, agreed. Though it might do better to describe it in *slightly*
more positive terms, and/or to explain the matter in a bit more
detail...

"TODO notes... This page has collected together a variety of
proposed projects to enhance PostgreSQL. Newcomers should beware; the
fact that these proposals have not been implemented should be taken to
indicate that they represent approaches that are difficult, have
gotten stuck, or even that have been abandoned due to having drawbacks
exceeding the imagined advantages. Be sure to discuss proposals on
the pgsql-hackers list before getting too far into implementation,
otherwise there is considerable risk of wasting effort on a patch that
is likely to be rejected."

--
When confronted by a difficult problem, solve it by reducing it to the
question, "How would the Lone Ranger handle this?"

On Tue, Jan 2, 2018 at 3:42 PM, Joshua D. Drake <jd@commandprompt.com> wrote:

On 01/02/2018 11:17 AM, Robert Haas wrote:

On Sun, Dec 31, 2017 at 2:31 PM, Peter Geoghegan <pg@bowt.ie> wrote:

On Sun, Dec 31, 2017 at 10:42 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

If we're not going to maintain/curate it properly, I agree it's not
worth keeping it around. But I'd rather see somebody put some effort
into it ...

If somebody was going to resolve to put some effort into maintaining
it to a high standard then it probably would have happened already.
The fact that it hasn't happened tells us plenty.

+1, and well said.

O.k. what does it tell us though? Is it a resource issue? Is it a barrier of
entry issue? What does deleting it solve? What problems (and there is a very
large obvious one) are caused by deleting it?

Right now, the TODO list is the "only" portal to "potential" things we
"might" want. If we delete it we are just creating yet another barrier of
entry to potential contribution. I think we need to consider an alternative
solution because of that.

Thanks,

JD

--
Command Prompt, Inc. || http://the.postgres.company/ || @cmdpromptinc

PostgreSQL centered full stack support, consulting and development.
Advocate: @amplifypostgres || Learn: https://postgresconf.org
***** Unless otherwise stated, opinions are my own. *****

As a person looking to become a postgres contributor, perhaps I can
offer some perspective on this. I think there is value in providing
*some* starting point for new contributors in the form of concrete
problems to solve. The value I hope to extract from the time spent on
my first feature comes mostly from the learning experience and not
from the acceptance of the feature itself. I would not be upset if my
work was never accepted as long as I understand why. I expect most
people picking features at random from a TODO list would have a
similar outlook on their first contribution.

On Tue, Jan 02, 2018 at 05:52:37PM -0500, Peter Eisentraut wrote:

On 12/31/17 22:43, Craig Ringer wrote:

I'd rather rename it the "stuck, hard and abandoned projects list" ;)�

That might actually be useful.

+1. When I do refer to a TODO entry, it's usually because the entry bears a
list of threads illustrating the difficulty of a problem. I like the project
having such lists, but "TODO" is a bad heading for them.

Greetings,

* Patrick Krecker (pkrecker@gmail.com) wrote:

As a person looking to become a postgres contributor, perhaps I can
offer some perspective on this. I think there is value in providing
*some* starting point for new contributors in the form of concrete
problems to solve. The value I hope to extract from the time spent on
my first feature comes mostly from the learning experience and not
from the acceptance of the feature itself. I would not be upset if my
work was never accepted as long as I understand why. I expect most
people picking features at random from a TODO list would have a
similar outlook on their first contribution.

I really appreciate this view-point on it and think that it's definitely
a good one to have, though I don't expect everyone to share it.

While I tend to agree that the TODO list is valuable, what I don't think
it has that is really key is any notion around 'level of difficulty' or
'who you can talk to about trying to address this item'.

We had a rather successful Google Summer of Code in 2017 and I don't
think that any of the projects accepted were ever on the TODO list.
In running GSoC last year and reading through all of the documentation
provided by Google about how to run GSoC successfully and what they're
looking for in terms of a project list made me realize that the areas I
mention above are really key to success.

The projects proposed through GSoC are, necessairly, constrained to what
can be done in a single summer (or, at least, serious progress being
made), but that's actually a good thing because it means that the
projects are sized well- large enough to be interesting and serious
features while small enough for someone relatively new to the community.
The "easy" things tend to be done by regular hackers, the really hard
stuff is too big for a beginner to start with.

To that end, let me just say that I strongly encourage anyone who is
interested in hacking on PG to please feel free to review what's on the
GSoC 2018 page and feel free to reach out to me if you're interested in
working on it, even if you aren't planning to participate in GSoC 2018.
We can come up with other things for GSoC students to work on, I
believe, even if half of the items currently listed there get picked up
on by non-GSoC individuals.

For the regular hackers, when you see something that's really hard on
the TODO list, please mark it as such instead of just deciding that the
list is useless. If we could quantify the level of effort required,
that'd be even better. Ultimately, I think it'd be fantastic to have a
wiki page for each item on the TODO list that describes the project
itself, what's been discussed (with links to those discussions) and what
areas could use more exploration.

The way forward, at least from my perspective, is to try and improve the
list, not to throw it away. I'd love to, one day, be able to go through
the TODO list and find the links to GSoC-sized projects to build the
yearly GSoC page with instead of having to make it an independent
effort.

Lastly, as a reminder, we will be submitting for GSoC 2018 soon, so
please put your GSoC-sized ideas on the GSoC 2018 page:

https://wiki.postgresql.org/wiki/GSoC_2018

and feel free to put them on the TODO list too...

Thanks!

Stephen