pgsql: Provide much better wait information in pg_stat_activity.

Hi

2016-03-10 19:55 GMT+01:00 Robert Haas <rhaas@postgresql.org>:

Provide much better wait information in pg_stat_activity.

When a process is waiting for a heavyweight lock, we will now indicate
the type of heavyweight lock for which it is waiting. Also, you can
now see when a process is waiting for a lightweight lock - in which
case we will indicate the individual lock name or the tranche, as
appropriate - or for a buffer pin.

Amit Kapila, Ildus Kurbangaliev, reviewed by me. Lots of helpful
discussion and suggestions by many others, including Alexander
Korotkov, Vladimir Borodin, and many others.

Branch
------
master

I am trying to test this feature, and there I see not actual data. Maybe
this behave is not related to this patch:

create table foo(a int);
insert into foo values(10);

session one:

begin; select * from foo for update;

session two:

begin; select * from foo for update;
session two is waiting

session one:
select * from pg_stat_activity -- I don't see correct information about
session two

session two:
rollback; begin; select * from foo where a = 10 for update;
session two is waiting again

session one:
select * from pg_stat_activity; -- The content is not changed

So content of pg_stat_activity is not correct in holder lock session.
Independent third session see valid content of pg_stat_activity.

Hypothesis: the pg_stat_activity is not refreshed under opened transaction?

Regards

Pavel

On Thu, Mar 10, 2016 at 3:44 PM, Pavel Stehule <pavel.stehule@gmail.com> wrote:

I am trying to test this feature, and there I see not actual data. Maybe
this behave is not related to this patch:

create table foo(a int);
insert into foo values(10);

session one:

begin; select * from foo for update;

session two:

begin; select * from foo for update;
session two is waiting

session one:
select * from pg_stat_activity -- I don't see correct information about
session two

At this point, I get:

rhaas=# select query, state, wait_event, wait_event_type from pg_stat_activity;
query
| state | wait_event | wait_event_type
-------------------------------------------------------------------------+--------+---------------+-----------------
select query, state, wait_event, wait_event_type from
pg_stat_activity; | active | |
select * from foo for update;
| active | transactionid | Lock
(2 rows)

...which looks right to me.

session two:
rollback; begin; select * from foo where a = 10 for update;
session two is waiting again

I don't see how you can do this here - the session is blocked.

There could well be a bug here, but I need a little more help to find it.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

2016-03-10 22:24 GMT+01:00 Robert Haas <robertmhaas@gmail.com>:

rhaas=# select query, state, wait_event, wait_event_type from
pg_stat_activity;
query
| state | wait_event | wait_event_type

-------------------------------------------------------------------------+--------+---------------+-----------------
select query, state, wait_event, wait_event_type from
pg_stat_activity; | active | |
select * from foo for update;
| active | transactionid | Lock
(2 rows)

...which looks right to me.

session two:
rollback; begin; select * from foo where a = 10 for update;
session two is waiting again

I don't see how you can do this here - the session is blocked.

There could well be a bug here, but I need a little more help to find it.

Maybe it be clear from attached text file

Regards

On Thu, Mar 10, 2016 at 4:51 PM, Pavel Stehule <pavel.stehule@gmail.com> wrote:

Maybe it be clear from attached text file

Uh, yikes, that looks messed up, but I wouldn't have thought this
commit would have changed anything there one way or the other. The
current query is reported by pgstat_report_activity(), which I didn't
touch. I think.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Thu, Mar 10, 2016 at 5:05 PM, Robert Haas <robertmhaas@gmail.com> wrote:

On Thu, Mar 10, 2016 at 4:51 PM, Pavel Stehule <pavel.stehule@gmail.com> wrote:

Maybe it be clear from attached text file

Uh, yikes, that looks messed up, but I wouldn't have thought this
commit would have changed anything there one way or the other. The
current query is reported by pgstat_report_activity(), which I didn't
touch. I think.

I just tried this on 9.5 - changing the query only to "select pid,
state, query from pg_stat_activity" and doing everything else the
same - and I see the same behavior there. So it looks like this is a
preexisting bug.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Thu, Mar 10, 2016 at 5:07 PM, Robert Haas <robertmhaas@gmail.com> wrote:

On Thu, Mar 10, 2016 at 5:05 PM, Robert Haas <robertmhaas@gmail.com> wrote:

On Thu, Mar 10, 2016 at 4:51 PM, Pavel Stehule <pavel.stehule@gmail.com> wrote:

Maybe it be clear from attached text file

Uh, yikes, that looks messed up, but I wouldn't have thought this
commit would have changed anything there one way or the other. The
current query is reported by pgstat_report_activity(), which I didn't
touch. I think.

I just tried this on 9.5 - changing the query only to "select pid,
state, query from pg_stat_activity" and doing everything else the
same - and I see the same behavior there. So it looks like this is a
preexisting bug.

Or ... maybe this is intentional behavior? Now that I think about it,
doesn't each backend cache this info the first time its transaction
reads the data?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Robert Haas <robertmhaas@gmail.com> writes:

Or ... maybe this is intentional behavior? Now that I think about it,
doesn't each backend cache this info the first time its transaction
reads the data?

Your view of pg_stat_activity is supposed to hold still within a
transaction, yes. Otherwise it'd be really painful to do any complicated
joins. I think there may be a function to explicitly flush the cache,
if you really need to see intratransaction changes.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2016-03-10 18:55:47 +0000, Robert Haas wrote:

Provide much better wait information in pg_stat_activity.

When a process is waiting for a heavyweight lock, we will now indicate
the type of heavyweight lock for which it is waiting. Also, you can
now see when a process is waiting for a lightweight lock - in which
case we will indicate the individual lock name or the tranche, as
appropriate - or for a buffer pin.

My compiler quite validly complains:

/home/andres/src/postgresql/src/backend/storage/lmgr/lmgr.c: In function ‘GetLockNameFromTagType’:
/home/andres/src/postgresql/src/backend/storage/lmgr/lmgr.c:1018:9: warning: function may return address of local variable [-Wreturn-local-addr]
return locktypename;
^
/home/andres/src/postgresql/src/backend/storage/lmgr/lmgr.c:1007:8: note: declared here
char tnbuf[32];
^
In file included from /home/andres/src/postgresql/src/backend/commands/dbcommands.c:20:0:

Andres

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

This is an excellent feature, thanks!
But can we please keep the old boolean waiting column?
I see no reason to break backward-compatibility. Or maybe I'm missing something.

I just had to commit this to make our system run locally on 9.6:

commit 2e189f85fa56724bec5c5cab2fcf0d2f3a4ce22a
Author: Joel Jacobson <joel@trustly.com>
Date: Fri Mar 11 08:19:52 2016 +0700

Make Have_Queries_Waiting() work with both <9.6 and >=9.6.

Apparently pg_stat_activity.waiting was removed by this commit:
commit 53be0b1add7064ca5db3cd884302dfc3268d884e
Author: Robert Haas <rhaas@postgresql.org>
Date: Thu Mar 10 12:44:09 2016 -0500

Provide much better wait information in pg_stat_activity.

This forces us to do some ugly version checking to know which column to use.
I for one can think it would have been better to keep the old
boolean column,
which is not entirely useless as sometimes you just want to know
if something is
waiting and don't care about the details, then it's convenient to
have a boolean column
instead of having to write "wait_event IS NOT NULL".

Let's hope they will add back our dear waiting column so we can avoid this
ugly hack before upgrading to 9.6.

diff --git a/public/FUNCTIONS/have_queries_waiting.sql
b/public/FUNCTIONS/have_queries_waiting.sql
index d83e7c8..b54caf5 100644
--- a/public/FUNCTIONS/have_queries_waiting.sql
+++ b/public/FUNCTIONS/have_queries_waiting.sql
@@ -3,9 +3,16 @@ SET search_path TO 'public', pg_catalog;
 CREATE OR REPLACE FUNCTION have_queries_waiting() RETURNS boolean
     SECURITY DEFINER
     SET search_path TO public, pg_temp
-    LANGUAGE sql
+    LANGUAGE plpgsql
     AS $$
-    SELECT EXISTS (SELECT 1 FROM pg_stat_activity WHERE waiting)
+DECLARE
+BEGIN
+IF version() ~ '^PostgreSQL 9\.[1-5]' THEN
+    RETURN EXISTS (SELECT 1 FROM pg_stat_activity WHERE waiting);
+ELSE
+    RETURN EXISTS (SELECT 1 FROM pg_stat_activity WHERE wait_event IS
NOT NULL);
+END IF;
+END;
 $$;

On Fri, Mar 11, 2016 at 6:17 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Robert Haas <robertmhaas@gmail.com> writes:

Or ... maybe this is intentional behavior? Now that I think about it,
doesn't each backend cache this info the first time its transaction
reads the data?

Your view of pg_stat_activity is supposed to hold still within a
transaction, yes. Otherwise it'd be really painful to do any complicated
joins. I think there may be a function to explicitly flush the cache,
if you really need to see intratransaction changes.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

--
Joel Jacobson

Mobile: +46703603801
Trustly.com | Newsroom | LinkedIn | Twitter

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 3/10/16 7:32 PM, Andres Freund wrote:

On 2016-03-10 18:55:47 +0000, Robert Haas wrote:

Provide much better wait information in pg_stat_activity.

When a process is waiting for a heavyweight lock, we will now indicate
the type of heavyweight lock for which it is waiting. Also, you can
now see when a process is waiting for a lightweight lock - in which
case we will indicate the individual lock name or the tranche, as
appropriate - or for a buffer pin.

My compiler quite validly complains:

/home/andres/src/postgresql/src/backend/storage/lmgr/lmgr.c: In function ‘GetLockNameFromTagType’:
/home/andres/src/postgresql/src/backend/storage/lmgr/lmgr.c:1018:9: warning: function may return address of local variable [-Wreturn-local-addr]
return locktypename;
^
/home/andres/src/postgresql/src/backend/storage/lmgr/lmgr.c:1007:8: note: declared here
char tnbuf[32];
^
In file included from /home/andres/src/postgresql/src/backend/commands/dbcommands.c:20:0:

Needs a "static", it seems.

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

On Thu, Mar 10, 2016 at 8:31 PM, Joel Jacobson <joel@trustly.com> wrote:

This is an excellent feature, thanks!
But can we please keep the old boolean waiting column?
I see no reason to break backward-compatibility. Or maybe I'm missing something.

Well, this was discussed. If we keep the Boolean "waiting" column, then either:

1. We make it true only for heavyweight lock waits, and false for
other kinds of waits. That's pretty strange.
2. We make it true for all kinds of waits that we now know how to
report. That still breaks compatibility.

I do understand that changing this is backward-incompatible and a lot
of people are going to have to update their monitoring tools. But I
think that's the best alternative. If we choose option #1, we're
going to be saddled with a weird backward-compatibility column
forever, and ten years from now we'll be explaining that even if
waiting = false you might still be waiting depending on the value of
some other column. If we choose option #2, it won't be
backward-compatible and some people's queries will still break, just
less obviously. Neither of those things seems very appealing.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Fri, Mar 11, 2016 at 9:36 AM, Robert Haas <robertmhaas@gmail.com> wrote:

Well, this was discussed. If we keep the Boolean "waiting" column, then either:

Oh, sorry for missing out on that discussion.

1. We make it true only for heavyweight lock waits, and false for
other kinds of waits. That's pretty strange.
2. We make it true for all kinds of waits that we now know how to
report. That still breaks compatibility.

Why not 3: We make it true for exactly the same type of situations as
in previous versions. Or is it not possible to do so for some reason?

I do understand that changing this is backward-incompatible and a lot
of people are going to have to update their monitoring tools. But I
think that's the best alternative. If we choose option #1, we're
going to be saddled with a weird backward-compatibility column
forever, and ten years from now we'll be explaining that even if
waiting = false you might still be waiting depending on the value of
some other column. If we choose option #2, it won't be
backward-compatible and some people's queries will still break, just
less obviously. Neither of those things seems very appealing.

I understand it's necessary to break backward-compatibility if the
it's not possible to return the same boolean value for the "waiting"
column in exactly the same situations.
Actually, even if it would be possible, I agree with you it's better
to force people to learn how to improve their tools by using the new
features.

Off topic, but related to the backward-compatibility subject:

Is there any written policy/wiki/thread/document on the topic "When
breaking backward-compatibility is acceptable"?

It would be helpful to get a better understand of this, as some ideas
on how to improve things can quickly be ruled out or ruled in
depending on what is acceptable or not.
For instance, there are some minor but annoying flaws in PL/pgSQL that
I would love to get fixed,
but the main arguments against doing so have been that it might break
some users' code somewhere,
even though doing so would probably be a good thing as the user could
have a bug in the code.
See: https://wiki.postgresql.org/wiki/Improving_PL/PgSQL_(September_2014)

I think one general rule should be "Breaking backward-compatibility is
acceptable if the new major pg-version throws an error in a situation
where the old major pg-version would conceal a bug or allow misuse of
a feature".
Trying to select the now removed "waiting" column throws an error.
Good! That lead me as a user here to figure out why I can't and
shouldn't use it. :)
Trying to e.g. select a different number of columns into a different
number of variables in a PL/pgSQL function doesn't throw an error.
Bad. :(
Here I would argue it's better to throw an error, just like when
trying to select from "waiting". It will hopefully save the day for
some users out there who can't find the bug in their complicated
PL/pgSQL application with millions of lines of code.

Sorry if this was completely off-topic, maybe I should start a new
thread or read some old thread in the archives on
backward-compatibility instead.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Fri, Mar 11, 2016 at 9:19 AM, Joel Jacobson <joel@trustly.com> wrote:

On Fri, Mar 11, 2016 at 9:36 AM, Robert Haas <robertmhaas@gmail.com>

wrote:

Well, this was discussed. If we keep the Boolean "waiting" column,

then either:

Oh, sorry for missing out on that discussion.

1. We make it true only for heavyweight lock waits, and false for
other kinds of waits. That's pretty strange.
2. We make it true for all kinds of waits that we now know how to
report. That still breaks compatibility.

Why not 3: We make it true for exactly the same type of situations as
in previous versions. Or is it not possible to do so for some reason?

Thats exactly the first point (1) of Robert. One thing that will be
strange according to me is that in some cases where waiting will be false,
but still wait_event and wait_event_type contain some wait information and
I think that will look odd to anybody new looking at the view.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

On Thu, Mar 10, 2016 at 10:49 PM, Joel Jacobson <joel@trustly.com> wrote:

On Fri, Mar 11, 2016 at 9:36 AM, Robert Haas <robertmhaas@gmail.com> wrote:

Well, this was discussed. If we keep the Boolean "waiting" column, then either:

Oh, sorry for missing out on that discussion.

1. We make it true only for heavyweight lock waits, and false for
other kinds of waits. That's pretty strange.
2. We make it true for all kinds of waits that we now know how to
report. That still breaks compatibility.

Why not 3: We make it true for exactly the same type of situations as
in previous versions. Or is it not possible to do so for some reason?

3 = 1.

Off topic, but related to the backward-compatibility subject:

Is there any written policy/wiki/thread/document on the topic "When
breaking backward-compatibility is acceptable"?

Not to my knowledge. We end up hashing it out on a case-by-case basis.

It would be helpful to get a better understand of this, as some ideas
on how to improve things can quickly be ruled out or ruled in
depending on what is acceptable or not.
For instance, there are some minor but annoying flaws in PL/pgSQL that
I would love to get fixed,
but the main arguments against doing so have been that it might break
some users' code somewhere,
even though doing so would probably be a good thing as the user could
have a bug in the code.
See: https://wiki.postgresql.org/wiki/Improving_PL/PgSQL_(September_2014)

I think that with respect to this particular set of improvements, the
problem is basically that there are just a lot of things that you
could hypothetically change, and it's not altogether clear which ones
of those individually would please more people than they displeased,
and it's not clear how much change we want to allow in total for the
sake of preserving backward compatibility, and then, too, the designs
for a lot of the individual features are fertile ground for
bikeshedding. I'm not direly opposed to most of what's on that page,
but I'm not excited about most of it, either. I bet if we canvassed
10 different companies that made heavy use of PL/pgsql they'd all have
a list of proposed changes like that, and I bet some of them would
conflict with each other, and I bet if we did all that stuff the
average PL/pgsql user's life would not be much better, but the manual
would be much longer.

(Also, I bet the variable assignments thing would break large amounts
of code that is working as designed.)

I think one general rule should be "Breaking backward-compatibility is
acceptable if the new major pg-version throws an error in a situation
where the old major pg-version would conceal a bug or allow misuse of
a feature".
Trying to select the now removed "waiting" column throws an error.
Good! That lead me as a user here to figure out why I can't and
shouldn't use it. :)

Yes, I think we use this rubric quite often, and I agree it's a good one.

Trying to e.g. select a different number of columns into a different
number of variables in a PL/pgSQL function doesn't throw an error.
Bad. :(

Yeah, I'm sympathetic to that request. That seems like poor error
checking and nothing else.

(But note that I do not rule here.)

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

2016-03-11 0:17 GMT+01:00 Tom Lane <tgl@sss.pgh.pa.us>:

Robert Haas <robertmhaas@gmail.com> writes:

Or ... maybe this is intentional behavior? Now that I think about it,
doesn't each backend cache this info the first time its transaction
reads the data?

Your view of pg_stat_activity is supposed to hold still within a
transaction, yes. Otherwise it'd be really painful to do any complicated
joins. I think there may be a function to explicitly flush the cache,
if you really need to see intratransaction changes.

I understand.

This behave has impact on PL functions that try to repeated check of
pg_stat_activity. But this use case is not frequent.

Thank you.

Regards

Pavel

Yes, I think we use this rubric quite often, and I agree it's a good one.

Trying to e.g. select a different number of columns into a different
number of variables in a PL/pgSQL function doesn't throw an error.
Bad. :(

Yeah, I'm sympathetic to that request. That seems like poor error
checking and nothing else.

(But note that I do not rule here.)

I am not sure, but maybe this issue is covered by plpgsql_check. But not
possible to check it when dynamic SQL is used.

Pavel

On Fri, Mar 11, 2016 at 11:14 AM, Robert Haas <robertmhaas@gmail.com> wrote:

I'm not direly opposed to most of what's on that page,
but I'm not excited about most of it, either.

May I ask, what improvements of PL/pgSQL would you personally be most
excited about,
if you or someone else would have unlimited resources to hack on it?

I bet if we canvassed 10 different companies that made heavy use of PL/pgsql they'd all have
a list of proposed changes like that, and I bet some of them would
conflict with each other, and I bet if we did all that stuff the
average PL/pgsql user's life would not be much better, but the manual
would be much longer.

You as a professional PostgreSQL consultant obviously have a lot of more
contact than me with other companies who make heavy use of PL/pgSQL.

I'm assuming your bet on these proposed changes in conflict you talk about
are based on things you've picked up IRL from companies you've been
working with.

What would you say are the top most commonly proposed changes
from companies that make heavy use of PL/pgSQL, and which of those are
in conflict?

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Fri, Mar 11, 2016 at 3:44 PM, Joel Jacobson <joel@trustly.com> wrote:

On Fri, Mar 11, 2016 at 11:14 AM, Robert Haas <robertmhaas@gmail.com> wrote:

I'm not direly opposed to most of what's on that page,
but I'm not excited about most of it, either.

May I ask, what improvements of PL/pgSQL would you personally be most
excited about,
if you or someone else would have unlimited resources to hack on it?

I bet if we canvassed 10 different companies that made heavy use of PL/pgsql they'd all have
a list of proposed changes like that, and I bet some of them would
conflict with each other, and I bet if we did all that stuff the
average PL/pgsql user's life would not be much better, but the manual
would be much longer.

You as a professional PostgreSQL consultant obviously have a lot of more
contact than me with other companies who make heavy use of PL/pgSQL.

I'm assuming your bet on these proposed changes in conflict you talk about
are based on things you've picked up IRL from companies you've been
working with.

What would you say are the top most commonly proposed changes
from companies that make heavy use of PL/pgSQL, and which of those are
in conflict?

I don't think my experience in this area is as deep as you seem to
think. I can tell you that most of the requests EnterpriseDB gets for
PL/pgsql enhancements involve wanting it to be more like Oracle's
PL/SQL, which of course has very little overlap with the stuff that
you're interested in. But I'm not really commenting here based on
that. I'm just giving you my impression based on the discussion I've
seen on the mailing list and my own personal feelings. If there is an
outcry for STRICT as you have proposed it, I'm not especially opposed
to that. I just think it needs a consensus that I haven't seen
emerge.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Sat, Mar 12, 2016 at 4:08 AM, Robert Haas <robertmhaas@gmail.com> wrote:

I don't think my experience in this area is as deep as you seem to
think. I can tell you that most of the requests EnterpriseDB gets for
PL/pgsql enhancements involve wanting it to be more like Oracle's
PL/SQL, which of course has very little overlap with the stuff that
you're interested in.

Do you know who could possibly be more experienced
with companies who are heavy users of PL/pgSQL in the community?

and/or,

Do you know of any companies who officially are heavy users of PL/pgSQL?

The only other company I can think of is Zalado, but of course there
are many more,
I just wish I knew their names, because I want to compile a wish list with
proposed changes from as many companies who are heavy users of
PL/pgSQL as possible.

That's the only way to push this forward. As you say, we need a
consensus and input
from a broad range of heavy users, not just from people on this list
with feelings
and opinions who might not actually be heavy users themselves.

Of course almost everybody on this list uses PL/pgSQL from time to
time or even daily,
but it's a completely different thing to write an entire backend
system in the language,
it's first then when you start to become really excited of e.g. not
having to type
at least 30 characters of text every time you do an UPDATE/INSERT
to be sure you modified exactly one row.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers