factorial function/phase out postfix operators?

On 5/18/20 4:42 PM, Peter Eisentraut wrote:

There have been occasional discussions about deprecating or phasing out
postfix operators, to make various things easier in the parser.

The first step would in any case be to provide alternatives for the
existing postfix operators.  There is currently one, namely the numeric
factorial operator "!".  A sensible alternative for that would be
providing a function factorial(numeric) -- and that already exists but
is not documented.  (Note that the operator is mapped to proname
"numeric_fac".  The function "factorial" maps to the same prosrc but is
otherwise independent of the operator.)

So I suggest that we add that function to the documentation.

I think this should be done regardless.

(Some adjacent cleanup work might also be in order.  The test cases for
factorial are currently in int4.sql, but all the factorial functionality
was moved to numeric a long time ago.)

What are the thoughts about then marking the postfix operator deprecated
and eventually removing it?

I am greatly in favor of removing postfix operators as soon as possible.
--
Vik Fearing

On Mon, May 18, 2020 at 05:02:34PM +0200, Vik Fearing wrote:

On 5/18/20 4:42 PM, Peter Eisentraut wrote:

There have been occasional discussions about deprecating or phasing out
postfix operators, to make various things easier in the parser.

The first step would in any case be to provide alternatives for the
existing postfix operators.� There is currently one, namely the numeric
factorial operator "!".� A sensible alternative for that would be
providing a function factorial(numeric) -- and that already exists but
is not documented.� (Note that the operator is mapped to proname
"numeric_fac".� The function "factorial" maps to the same prosrc but is
otherwise independent of the operator.)

So I suggest that we add that function to the documentation.

I think this should be done regardless.

(Some adjacent cleanup work might also be in order.� The test cases for
factorial are currently in int4.sql, but all the factorial functionality
was moved to numeric a long time ago.)

What are the thoughts about then marking the postfix operator deprecated
and eventually removing it?

I am greatly in favor of removing postfix operators as soon as possible.

Agreed.

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:

What are the thoughts about then marking the postfix operator deprecated
and eventually removing it?

If we do this it'd require a plan. We'd have to also warn about the
feature deprecation in (at least) the CREATE OPERATOR man page, and
we'd have to decide how many release cycles the deprecation notices
need to stand for.

If that's the intention, though, it'd be good to get those deprecation
notices published in v13 not v14.

regards, tom lane

On Mon, May 18, 2020 at 10:03:13PM -0400, Tom Lane wrote:

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:

What are the thoughts about then marking the postfix operator deprecated
and eventually removing it?

If we do this it'd require a plan. We'd have to also warn about the
feature deprecation in (at least) the CREATE OPERATOR man page, and
we'd have to decide how many release cycles the deprecation notices
need to stand for.

If that's the intention, though, it'd be good to get those deprecation
notices published in v13 not v14.

+1 for deprecating in v13.

Best,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

On 5/19/20 4:03 AM, Tom Lane wrote:

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:

What are the thoughts about then marking the postfix operator deprecated
and eventually removing it?

If we do this it'd require a plan. We'd have to also warn about the
feature deprecation in (at least) the CREATE OPERATOR man page, and
we'd have to decide how many release cycles the deprecation notices
need to stand for.

I have never come across any custom postfix operators in the wild, and
I've never even seen ! used in practice.

So I would suggest a very short deprecation period. Deprecate now in
13, let 14 go by, and rip it all out for 15. That should give us enough
time to extend the deprecation period if we need to, or go back on it
entirely (like I seem to remember we did with VACUUM FREEZE).

If that's the intention, though, it'd be good to get those deprecation
notices published in v13 not v14.

+1
-- 
Vik Fearing

On Mon, May 18, 2020 at 10:42 AM Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:

What are the thoughts about then marking the postfix operator deprecated
and eventually removing it?

I wrote a little bit about this last year:

/messages/by-id/CA+TgmoarLfSQcLCh7jx0737SZ28qwbuy+rUWT6rSHAO=B-6xdw@mail.gmail.com

I think it's generally a good idea, though perhaps we should consider
continuing to allow '!' as a postfix operator and just removing
support for any other. That would probably allow us to have a very
short deprecation period, since real-world use of user-defined postfix
operators seems to be nil -- and it would also make this into a change
that only affects the lexer and parser, which might make it simpler.

I won't lose a lot of sleep if we decide to rip out '!' as well, but I
don't think that continuing to support it would cost us much.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

út 19. 5. 2020 v 14:27 odesílatel Robert Haas <robertmhaas@gmail.com>
napsal:

On Mon, May 18, 2020 at 10:42 AM Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:

What are the thoughts about then marking the postfix operator deprecated
and eventually removing it?

I wrote a little bit about this last year:

/messages/by-id/CA+TgmoarLfSQcLCh7jx0737SZ28qwbuy+rUWT6rSHAO=B-6xdw@mail.gmail.com

I think it's generally a good idea, though perhaps we should consider
continuing to allow '!' as a postfix operator and just removing
support for any other. That would probably allow us to have a very
short deprecation period, since real-world use of user-defined postfix
operators seems to be nil -- and it would also make this into a change
that only affects the lexer and parser, which might make it simpler.

I won't lose a lot of sleep if we decide to rip out '!' as well, but I
don't think that continuing to support it would cost us much.

This is little bit obscure feature. It can be removed and relative quickly.
Maybe some warning if somebody use it can be good (for Postgres 13)

Pavel

I won't lose a lot of sleep if we decide to rip out '!' as well, but I
don't think that continuing to support it would cost us much.

+1 for keeping ! and nuking the rest, if possible.

Regards,
Ken

Robert Haas <robertmhaas@gmail.com> writes:

I think it's generally a good idea, though perhaps we should consider
continuing to allow '!' as a postfix operator and just removing
support for any other.

Uh ... what exactly would be the point of that? The real reason to do
this at all is not that we have it in for '!', but that we want to
drop the possibility of postfix operators from the grammar altogether,
which will remove a boatload of ambiguity.

I won't lose a lot of sleep if we decide to rip out '!' as well, but I
don't think that continuing to support it would cost us much.

AFAICS, it would cost us the entire point of this change.

In my non-caffeinated state, I don't recall exactly which things are
blocked by the existence of postfix ops; but I think for instance it might
become possible to remove the restriction of requiring AS before column
aliases that happen to be unreserved keywords.

If we lobotomize CREATE OPERATOR but don't remove built-in postfix
ops, then none of those improvements will be available. That seems
like the worst possible choice.

I would also argue that having a feature that is available to
built-in operators but not user-defined ones is pretty antithetical
to Postgres philosophy.

regards, tom lane

On Tue, May 19, 2020 at 9:51 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Uh ... what exactly would be the point of that? The real reason to do
this at all is not that we have it in for '!', but that we want to
drop the possibility of postfix operators from the grammar altogether,
which will remove a boatload of ambiguity.

The ambiguity doesn't come from the mere existence of postfix
operators. It comes from the fact that, when we lex the input, we
can't tell whether a particular operator that we happen to encounter
is prefix, infix, or postfix. So hard-coding, for example, a rule that
'!' is always a postfix operator and anything else is never a postfix
operator is sufficient to solve the key problems. Then "SELECT a ! b"
can only be a postfix operator application followed by a column
labeling, a "SELECT a + b" can only be the application of an infix
operator.

The parser ambiguities could also be removed if the source of the
information where a GUC or a catalog lookup; there are good reasons
not to go that way, but my point is that the problem is not that
postfix operators are per se evil, but that the information we need is
not available at the right phase of the process. We can only make use
of the information in pg_operator after we start assigning type
information, which has to happen after we parse, but to avoid the
ambiguity here, we need the information before we parse - i.e. at the
lexing stage.

In my non-caffeinated state, I don't recall exactly which things are
blocked by the existence of postfix ops; but I think for instance it might
become possible to remove the restriction of requiring AS before column
aliases that happen to be unreserved keywords.

Right - which would be a huge win.

I would also argue that having a feature that is available to
built-in operators but not user-defined ones is pretty antithetical
to Postgres philosophy.

That I think is the policy question before us. I believe that any rule
that tells us which operators are postfix and which are not at the
lexing stage is good enough. I think here you are arguing for the
empty set, which will work, but I believe any other fixed set also
works, such as { '!' }. I don't think we're going to break a ton of
user code no matter which one we pick, but I do think that it's
possible to pick either one and still achieve our goals here, so
that's the issue that I wanted to raise.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 5/19/20 4:22 PM, Robert Haas wrote:

On Tue, May 19, 2020 at 9:51 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Uh ... what exactly would be the point of that? The real reason to do
this at all is not that we have it in for '!', but that we want to
drop the possibility of postfix operators from the grammar altogether,
which will remove a boatload of ambiguity.

The ambiguity doesn't come from the mere existence of postfix
operators. It comes from the fact that, when we lex the input, we
can't tell whether a particular operator that we happen to encounter
is prefix, infix, or postfix. So hard-coding, for example, a rule that
'!' is always a postfix operator and anything else is never a postfix
operator is sufficient to solve the key problems. Then "SELECT a ! b"
can only be a postfix operator application followed by a column
labeling, a "SELECT a + b" can only be the application of an infix
operator.

So if I make a complex UDT where a NOT operator makes a lot of sense[*],
why wouldn't I be allowed to make a prefix operator ! for it? All for
what? That one person in the corner over there who doesn't want to
rewrite their query to use factorial() instead?

I'm -1 on keeping ! around as a hard-coded postfix operator.

[*] I don't have a concrete example in mind, just this abstract one.
--
Vik Fearing

On Tue, May 19, 2020 at 10:36 AM Vik Fearing <vik@postgresfriends.org> wrote:

So if I make a complex UDT where a NOT operator makes a lot of sense[*],
why wouldn't I be allowed to make a prefix operator ! for it? All for
what? That one person in the corner over there who doesn't want to
rewrite their query to use factorial() instead?

I'm -1 on keeping ! around as a hard-coded postfix operator.

Fair enough. I think you may be in the majority on that one, too. I
just wanted to raise the issue, and we'll see if anyone else agrees.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas <robertmhaas@gmail.com> writes:

The ambiguity doesn't come from the mere existence of postfix
operators. It comes from the fact that, when we lex the input, we
can't tell whether a particular operator that we happen to encounter
is prefix, infix, or postfix. So hard-coding, for example, a rule that
'!' is always a postfix operator and anything else is never a postfix
operator is sufficient to solve the key problems.

If we were willing to say that '!' could *only* be a postfix operator,
then maybe the ambiguity would go away. Or maybe it wouldn't; if
you're seriously proposing this, I think it'd be incumbent on you
to demonstrate that we could still simplify the grammar to the same
extent. But that will incur its own set of compatibility problems,
because there's no reason to assume that nobody has made prefix or
infix '!' operators.

In any case, it's hard to decide that that's a less klugy solution
than getting rid of postfix ops altogether. There's a reason why
few programming languages have those.

In general, I put this on about the same level as when we decided
to remove ';' and ':' as operators (cf 259489bab, 766fb7f70).
Somebody thought it was cute that it was possible to have that,
which maybe it was, but it wasn't really sane in the big picture.
And as I recall, the amount of pushback we got was nil.

regards, tom lane

Greetings,

* Vik Fearing (vik@postgresfriends.org) wrote:

On 5/19/20 4:03 AM, Tom Lane wrote:

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:

What are the thoughts about then marking the postfix operator deprecated
and eventually removing it?

If we do this it'd require a plan. We'd have to also warn about the
feature deprecation in (at least) the CREATE OPERATOR man page, and
we'd have to decide how many release cycles the deprecation notices
need to stand for.

I have never come across any custom postfix operators in the wild, and
I've never even seen ! used in practice.

So I would suggest a very short deprecation period. Deprecate now in
13, let 14 go by, and rip it all out for 15. That should give us enough
time to extend the deprecation period if we need to, or go back on it
entirely (like I seem to remember we did with VACUUM FREEZE).

If that's the intention, though, it'd be good to get those deprecation
notices published in v13 not v14.

+1

I agree with putting notices into v13 saying they're deprecated, but
then actually removing them in v14. For that matter, I'd vote that we
generally accept a system whereby when we commit something that removes
a feature in the next major version, we put out some kind of notice that
it's been deprecated and won't be in v14. We don't want to run the risk
of saying XYZ has been deprecated and then it staying around for a few
years, nor trying to say "it'll be removed in v14" before we actually
know that it's been committed for v14.

In other words, wait to deprecate until the commit has happened for v14
(and maybe wait a couple days in case someone wasn't watching and argues
to revert, but not longer than any normal commit), and then go back and
mark it as "deprecated and removed in v14" for all back-branches. Users
will continue to have 5 years (by upgrading to v13, or whatever the last
release was before their favorite feature was removed, if they really
need to) to update their systems to deal with the change.

We do not do ourselves nor our users a real service by carrying forward
deprecated code/interfaces/views/etc, across major versions; instead
they tend to live on in infamy, with some users actually updating and
some not, ever, and then complaining when we suggest actually removing
it (we have lots of good examples of that too) and then we have to have
the debate again about removing it and, in some cases, we end up
un-deprecating it, which is confusing for users and a bit ridiculous.

Let's not do that.

Thanks,

Stephen

Vik Fearing <vik@postgresfriends.org> writes:

I'm -1 on keeping ! around as a hard-coded postfix operator.

Before we go much further on this, we should have some proof
that there's actually material benefit to be gained. I spent some
time just now trying to relax the AS restriction by ripping out
postfix ops, and the results were not too promising. Indeed the
postfix-ops problem goes away, but then you find out that SQL's
random syntax choices for type names become the stumbling block.
An example here is that given

SELECT 'foo'::character varying

it's not clear if "varying" is supposed to be part of the type name or a
column label. It looks to me like we'd have to increase the reserved-ness
of VARYING, PRECISION, and about half a dozen currently-unreserved
keywords involved in INTERVAL syntax, including such popular column names
as "month", "day", and "year".

Plus I got conflicts on WITHIN, GROUP, and FILTER from ordered-set
aggregate syntax; those are currently unreserved keywords, but they
can't be allowed as AS-less column labels.

We could possibly minimize the damage by inventing another keyword
classification besides the four we have now. Or maybe we should
think harder about using more lookahead between the lexer and grammar.
But this is going to be a lot more ticklish than I would've hoped,
and possibly not cost-free, so we might well end up never pulling
the trigger on such a change.

So right at the moment I'm agreeing with Stephen's nearby opinion:
let's not deprecate these until we've got a patch that gets some
concrete benefit from removing them.

regards, tom lane

On Tue, May 19, 2020 at 11:32 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Before we go much further on this, we should have some proof
that there's actually material benefit to be gained. I spent some
time just now trying to relax the AS restriction by ripping out
postfix ops, and the results were not too promising. Indeed the
postfix-ops problem goes away, but then you find out that SQL's
random syntax choices for type names become the stumbling block.
An example here is that given

SELECT 'foo'::character varying

it's not clear if "varying" is supposed to be part of the type name or a
column label. It looks to me like we'd have to increase the reserved-ness
of VARYING, PRECISION, and about half a dozen currently-unreserved
keywords involved in INTERVAL syntax, including such popular column names
as "month", "day", and "year".

Plus I got conflicts on WITHIN, GROUP, and FILTER from ordered-set
aggregate syntax; those are currently unreserved keywords, but they
can't be allowed as AS-less column labels.

I came to similar conclusions a couple of years ago:

/messages/by-id/CA+TgmoYzPvT7uiHjWgKtyTivHHLNCp0yLavCoipE-LyG3w2wOQ@mail.gmail.com

What I proposed at the time was creating a new category of keywords.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas <robertmhaas@gmail.com> writes:

On Tue, May 19, 2020 at 11:32 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Before we go much further on this, we should have some proof
that there's actually material benefit to be gained. I spent some
time just now trying to relax the AS restriction by ripping out
postfix ops, and the results were not too promising.

I came to similar conclusions a couple of years ago:
/messages/by-id/CA+TgmoYzPvT7uiHjWgKtyTivHHLNCp0yLavCoipE-LyG3w2wOQ@mail.gmail.com

Ah, right.

What I proposed at the time was creating a new category of keywords.

Might work. My main concern would be if we have to forbid those keywords
as column names --- for words like "year", in particular, that'd be a
disaster. If the net effect is only that they can't be AS-less col labels,
it won't break any cases that worked before.

Our existing four-way keyword classification is not something that was
handed down on stone tablets. I wonder whether postfix-ectomy changes
the situation enough that a complete rethinking would be helpful.

I also continue to think that more lookahead and token-merging would
be interesting to pursue. It'd hardly surprise anybody if the
token pair "character varying" were always treated as a type name,
for instance.

Anyway, the bottom-line conclusion remains the same: let's make sure
we know what we'd do after getting rid of postfix ops, before we do
that.

regards, tom lane

On Tue, May 19, 2020 at 2:30 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Might work. My main concern would be if we have to forbid those keywords
as column names --- for words like "year", in particular, that'd be a
disaster. If the net effect is only that they can't be AS-less col labels,
it won't break any cases that worked before.

ISTM that all we have to do to avoid that is switch from a four-way
classification to a five-way classification: just split
unreserved_keyword into totally_unreserved_keyword and
very_slightly_reserved_keyword.

Our existing four-way keyword classification is not something that was
handed down on stone tablets. I wonder whether postfix-ectomy changes
the situation enough that a complete rethinking would be helpful.

I don't see that they do, but I might be missing something. I think
there's an excellent argument for adding one new category, but it's
not clear to me why it should reshape the landscape any more than
that.

I also continue to think that more lookahead and token-merging would
be interesting to pursue. It'd hardly surprise anybody if the
token pair "character varying" were always treated as a type name,
for instance.

I think that line of attack will not buy very much. The ability to
avoid unexpected consequences is entirely contingent on the
unlikeliness of the keywords appearing adjacent to each other in some
other context, and the only argument for that here is that neither of
those words is a terribly likely column name. I think that when you
try to solve interesting problems with this, though, you very quickly
run into problems where that's not the case, and you'll need a
technique that has some knowledge of the parser state to actually do
something that works well. I read a paper some years ago that proposed
a solution to this problem: if the parser generator sees a
shift/reduce conflict, it checks whether the conflict can be resolve
by looking ahead one or more additional tokens. If so, it can build a
little DFA that gets run when you enter that state, with edges labeled
with lookahead tokens, and it runs that DFA whenever you reach the
problematic state. Since, hopefully, such states are relatively rarely
encountered, the overhead is low, yet it still gives you a way out of
conflicts in many practical cases. Unfortunately, the chances of bison
implementing such a thing do not seem very good.

Anyway, the bottom-line conclusion remains the same: let's make sure
we know what we'd do after getting rid of postfix ops, before we do
that.

Well, I don't think we really need to get too conservative here. I've
studied this issue enough over the years to be pretty darn sure that
this is a necessary prerequisite to doing something about the "AS
unreserved_keyword" issue, and that it is by far the most significant
issue in doing something about that problem. Sure, there are other
issues, but I think they are basically matters of politics or policy.
For example, if some key people DID think that the four-way keyword
classification was handed down on stone tablets, that could be quite a
problem, but if we're willing to take the view that solving the "AS
unreserved_keyword" problem is pretty important and we need to find a
way to get it done, then I think we an do that. It seems to me that
the first thing that we need to do here is get a deprecation notice
out, so that people know that we're planning to break this. I think we
should go ahead and make that happen now, or at least pretty soon.

I'm still interested in hearing what people think about hard-coding !
as a postfix operator vs. removing postfix operators altogether. I
think Vik and Tom are against keeping just !, Kenneth Marshall are for
it, and I'm not sure I understand Pavel's position. I'm about +0.3 for
keeping just ! myself. Maybe we'll get some other votes. If you're
willing to be persuaded that keeping only ! is a sensible thing to
consider, I could probably draft a very rough patch showing that it
would still be sufficient to get us out from under the "AS
unreserved_keyword" problem, but if you and/or enough other people
hate that direction with a fiery passion, I won't bother. I'm pretty
sure it's technically possible, but the issue is more about what
people actually want.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas <robertmhaas@gmail.com> writes:

On Tue, May 19, 2020 at 2:30 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Anyway, the bottom-line conclusion remains the same: let's make sure
we know what we'd do after getting rid of postfix ops, before we do
that.

Well, I don't think we really need to get too conservative here.
... It seems to me that
the first thing that we need to do here is get a deprecation notice
out, so that people know that we're planning to break this.

No, I disagree with that, because from what I've seen so far it's
not really clear to me that we have a full solution to the AS
problem excepting only postfix ops. I don't want to deprecate
postfix ops before it's completely clear that we can get something
out of it. Otherwise, we'll either end up un-deprecating them,
which makes us look silly, or removing a feature for zero benefit.

Stephen's nearby proposal to deprecate only after a patch has been
committed doesn't seem all that unreasonable, if you're only intending
to allow one cycle's worth of notice. In particular, I could easily
see us committing a fix sometime this summer and then sticking
deprecation notices into the back branches before v13 goes gold.
But let's have the complete fix in hand first.

I'm still interested in hearing what people think about hard-coding !
as a postfix operator vs. removing postfix operators altogether. I
think Vik and Tom are against keeping just !, Kenneth Marshall are for
it, and I'm not sure I understand Pavel's position.

Yes, I'm VERY strongly against keeping just !. I think it'd be a
ridiculous, and probably very messy, backwards-compatibility hack; and the
fact that it will break valid use-cases that we don't need to break seems
to me to well outweigh the possibility that someone would rather not
change their queries to use factorial() or !!.

However, we do have to have a benefit to show those people whose
queries we break. Hence my insistence on having a working AS fix
(or some other benefit) before not after.

regards, tom lane