ANY_VALUE aggregate

On Mon, Dec 5, 2022 at 7:57 AM Vik Fearing <vik@postgresfriends.org> wrote:

The SQL:2023 Standard defines a new aggregate named ANY_VALUE. It
returns an implementation-dependent (i.e. non-deterministic) value from
the rows in its group.

PFA an implementation of this aggregate.

Can we please add "first_value" and "last_value" if we are going to add
"some_random_value" to our library of aggregates?

Also, maybe we should have any_value do something like compute a 50/50
chance that any new value seen replaces the existing chosen value, instead
of simply returning the first value all the time. Maybe even prohibit the
first value from being chosen so long as a second value appears.

David J.

"David G. Johnston" <david.g.johnston@gmail.com> writes:

Can we please add "first_value" and "last_value" if we are going to add
"some_random_value" to our library of aggregates?

First and last according to what ordering? We have those in the
window-aggregate case, and I don't think we want to encourage people
to believe that "first" and "last" are meaningful otherwise.

ANY_VALUE at least makes it clear that you're getting an unspecified
one of the inputs.

regards, tom lane

On Mon, Dec 5, 2022 at 1:04 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

"David G. Johnston" <david.g.johnston@gmail.com> writes:

Can we please add "first_value" and "last_value" if we are going to add
"some_random_value" to our library of aggregates?

First and last according to what ordering? We have those in the
window-aggregate case, and I don't think we want to encourage people
to believe that "first" and "last" are meaningful otherwise.

ANY_VALUE at least makes it clear that you're getting an unspecified
one of the inputs.

I have personally implemented first_value() and last_value() in the
past in cases where I had guaranteed the ordering myself, or didn't
care what ordering was used. I think they're perfectly sensible. But
if we don't add them to core, at least they're easy to add in
user-space.

--
Robert Haas
EDB: http://www.enterprisedb.com

On Mon, Dec 5, 2022 at 12:57 PM David G. Johnston <
david.g.johnston@gmail.com> wrote:

On Mon, Dec 5, 2022 at 7:57 AM Vik Fearing <vik@postgresfriends.org>
wrote:

The SQL:2023 Standard defines a new aggregate named ANY_VALUE. It
returns an implementation-dependent (i.e. non-deterministic) value from
the rows in its group.

PFA an implementation of this aggregate.

Can we please add "first_value" and "last_value" if we are going to add
"some_random_value" to our library of aggregates?

Also, maybe we should have any_value do something like compute a 50/50
chance that any new value seen replaces the existing chosen value, instead
of simply returning the first value all the time. Maybe even prohibit the
first value from being chosen so long as a second value appears.

David J.

Adding to the pile of wanted aggregates: in the past I've lobbied for
only_value() which is like first_value() but it raises an error on
encountering a second value.

On Mon, Dec 5, 2022 at 2:31 PM Corey Huinker <corey.huinker@gmail.com> wrote:

Adding to the pile of wanted aggregates: in the past I've lobbied for only_value() which is like first_value() but it raises an error on encountering a second value.

Yeah, that's another that I have hand-rolled in the past.

--
Robert Haas
EDB: http://www.enterprisedb.com

On 12/5/22 15:57, Vik Fearing wrote:

The SQL:2023 Standard defines a new aggregate named ANY_VALUE.  It
returns an implementation-dependent (i.e. non-deterministic) value from
the rows in its group.

PFA an implementation of this aggregate.

Here is v2 of this patch. I had forgotten to update sql_features.txt.
--
Vik Fearing

On 12/5/22 18:56, David G. Johnston wrote:

Also, maybe we should have any_value do something like compute a 50/50
chance that any new value seen replaces the existing chosen value, instead
of simply returning the first value all the time. Maybe even prohibit the
first value from being chosen so long as a second value appears.

The spec says the result is implementation-dependent meaning we don't
even need to document how it is obtained, but surely behavior like this
would preclude future optimizations like the ones I mentioned?

I once wrote a random_agg() for a training course that used reservoir
sampling to get an evenly distributed value from the inputs. Something
like that seems to be what you are looking for here. I don't see the
use case for adding it to core, though.

The use case for ANY_VALUE is compliance with the standard.
--
Vik Fearing

On 12/5/22 20:31, Corey Huinker wrote:

Adding to the pile of wanted aggregates: in the past I've lobbied for
only_value() which is like first_value() but it raises an error on
encountering a second value.

I have had use for this in the past, but I can't remember why. What is
your use case for it? I will happily write a patch for it, and also
submit it to the SQL Committee for inclusion in the standard. I need to
justify why it's a good idea, though, and we would need to consider what
to do with nulls now that there is <unique null treatment>.
--
Vik Fearing

On Mon, 5 Dec 2022 at 22:52, Vik Fearing <vik@postgresfriends.org> wrote:

On 12/5/22 20:31, Corey Huinker wrote:

Adding to the pile of wanted aggregates: in the past I've lobbied for
only_value() which is like first_value() but it raises an error on
encountering a second value.

I have had use for this in the past, but I can't remember why. What is
your use case for it? I will happily write a patch for it, and also
submit it to the SQL Committee for inclusion in the standard. I need to
justify why it's a good idea, though, and we would need to consider what
to do with nulls now that there is <unique null treatment>.

I have this in my local library of "stuff that I really wish came with
Postgres", although I call it same_agg and it just goes to NULL if there
are more than one distinct value.

I sometimes use it when normalizing non-normalized data, but more commonly
I use it when the query planner isn't capable of figuring out that a column
I want to use in the output depends only on the grouping columns. For
example, something like:

SELECT group_id, group_name, count(*) from group_group as gg natural join
group_member as gm group by group_id

I think that exact example actually does or is supposed to work now, since
it realizes that I'm grouping on the primary key of group_group so the
group_name field in the same table can't differ between rows of a group,
but most of the time when I expect that feature to allow me to use a field
it actually doesn't.

I have a vague notion that part of the issue may be the distinction between
gg.group_id, gm.group_id, and group_id; maybe the above doesn't work but it
does work if I group by gg.group_id instead of by group_id. But obviously
there should be no difference because in this query those 3 values cannot
differ (outer joins are another story).

For reference, here is my definition:

CREATE OR REPLACE FUNCTION same_sfunc (
a anyelement,
b anyelement
) RETURNS anyelement
LANGUAGE SQL IMMUTABLE STRICT
SET search_path FROM CURRENT
AS $$
SELECT CASE WHEN $1 = $2 THEN $1 ELSE NULL END
$$;
COMMENT ON FUNCTION same_sfunc (anyelement, anyelement) IS 'SFUNC for
same_agg aggregate; returns common value of parameters, or NULL if they
differ';

DROP AGGREGATE IF EXISTS same_agg (anyelement);
CREATE AGGREGATE same_agg (anyelement) (
SFUNC = same_sfunc,
STYPE = anyelement
);
COMMENT ON AGGREGATE same_agg (anyelement) IS 'Return the common non-NULL
value of all non-NULL aggregated values, or NULL if some values differ';

You can tell I've had this for a while - there are several newer Postgres
features that could be used to clean this up noticeably.

I also have a repeat_agg which returns the last value (not so interesting)
but which is sometimes useful as a window function (more interesting:
replace NULLs with the previous non-NULL value in the column).

On Mon, Dec 5, 2022 at 8:46 PM Vik Fearing <vik@postgresfriends.org> wrote:

On 12/5/22 18:56, David G. Johnston wrote:

Also, maybe we should have any_value do something like compute a 50/50
chance that any new value seen replaces the existing chosen value,

instead

of simply returning the first value all the time. Maybe even prohibit

the

first value from being chosen so long as a second value appears.

The spec says the result is implementation-dependent meaning we don't
even need to document how it is obtained, but surely behavior like this
would preclude future optimizations like the ones I mentioned?

So, given the fact that we don't actually want to name a function
first_value (because some users are readily confused as to when the concept
of first is actually valid or not) but some users do actually wish for this
functionality - and you are proposing to implement it here anyway - how
about we actually do document that we promise to return the first non-null
value encountered by the aggregate. We can then direct people to this
function and just let them know to pretend the function is really named
first_value in the case where they specify an order by. (last_value comes
for basically free with descending sorting).

I once wrote a random_agg() for a training course that used reservoir
sampling to get an evenly distributed value from the inputs. Something
like that seems to be what you are looking for here. I don't see the
use case for adding it to core, though.

The use case was basically what Tom was saying - I don't want our users
that don't understand the necessity of order by, and don't read the
documentation, to observe that we consistently return the first non-null
value and assume that this is what the function promises when we are not
making any such promise to them. As noted above, my preference at this
point would be to just make that promise.

David J.

On 12/6/22 05:22, David G. Johnston wrote:

On Mon, Dec 5, 2022 at 8:46 PM Vik Fearing <vik@postgresfriends.org> wrote:

On 12/5/22 18:56, David G. Johnston wrote:

Also, maybe we should have any_value do something like compute a 50/50
chance that any new value seen replaces the existing chosen value,

instead

of simply returning the first value all the time. Maybe even prohibit

the

first value from being chosen so long as a second value appears.

The spec says the result is implementation-dependent meaning we don't
even need to document how it is obtained, but surely behavior like this
would preclude future optimizations like the ones I mentioned?

So, given the fact that we don't actually want to name a function
first_value (because some users are readily confused as to when the concept
of first is actually valid or not) but some users do actually wish for this
functionality - and you are proposing to implement it here anyway - how
about we actually do document that we promise to return the first non-null
value encountered by the aggregate. We can then direct people to this
function and just let them know to pretend the function is really named
first_value in the case where they specify an order by. (last_value comes
for basically free with descending sorting).

I can imagine an optimization that would remove an ORDER BY clause
because it isn't needed for any other aggregate. There is no reason to
cause an extra sort when the user has requested *any value*.

I once wrote a random_agg() for a training course that used reservoir
sampling to get an evenly distributed value from the inputs. Something
like that seems to be what you are looking for here. I don't see the
use case for adding it to core, though.

The use case was basically what Tom was saying - I don't want our users
that don't understand the necessity of order by, and don't read the
documentation, to observe that we consistently return the first non-null
value and assume that this is what the function promises when we are not
making any such promise to them.

Documenting something for the benefit of those who do not read the
documentation is a ridiculous proposal.

As noted above, my preference at this point would be to just make that promise.

I see no reason to paint ourselves into a corner here.
--
Vik Fearing

On Mon, Dec 5, 2022 at 9:48 PM Vik Fearing <vik@postgresfriends.org> wrote:

On 12/6/22 05:22, David G. Johnston wrote:

On Mon, Dec 5, 2022 at 8:46 PM Vik Fearing <vik@postgresfriends.org>

wrote:

On 12/5/22 18:56, David G. Johnston wrote:

Also, maybe we should have any_value do something like compute a 50/50
chance that any new value seen replaces the existing chosen value,

instead

of simply returning the first value all the time. Maybe even prohibit

the

first value from being chosen so long as a second value appears.

The spec says the result is implementation-dependent meaning we don't
even need to document how it is obtained, but surely behavior like this
would preclude future optimizations like the ones I mentioned?

So, given the fact that we don't actually want to name a function
first_value (because some users are readily confused as to when the

concept

of first is actually valid or not) but some users do actually wish for

this

functionality - and you are proposing to implement it here anyway - how
about we actually do document that we promise to return the first

non-null

value encountered by the aggregate. We can then direct people to this
function and just let them know to pretend the function is really named
first_value in the case where they specify an order by. (last_value comes
for basically free with descending sorting).

I can imagine an optimization that would remove an ORDER BY clause
because it isn't needed for any other aggregate.

I'm referring to the query:

select any_value(v order by v) from (values (2),(1),(3)) as vals (v);
// produces 1, per the documented implementation-defined behavior.

Someone writing:

select any_value(v) from (values (2),(1),(3)) as vals (v) order by v;

Is not presently, nor am I saying, promised the value 1.

I'm assuming you are thinking of the second query form, while the guarantee
only needs to apply to the first.

David J.

On 12/6/22 05:57, David G. Johnston wrote:

On Mon, Dec 5, 2022 at 9:48 PM Vik Fearing <vik@postgresfriends.org> wrote:

I can imagine an optimization that would remove an ORDER BY clause
because it isn't needed for any other aggregate.

I'm referring to the query:

select any_value(v order by v) from (values (2),(1),(3)) as vals (v);
// produces 1, per the documented implementation-defined behavior.

Implementation-dependent. It is NOT implementation-defined, per spec.

We often loosen the spec rules when they don't make technical sense to
us, but I don't know of any example of when we have tightened them.

Someone writing:

select any_value(v) from (values (2),(1),(3)) as vals (v) order by v;

Is not presently, nor am I saying, promised the value 1.

I'm assuming you are thinking of the second query form, while the guarantee
only needs to apply to the first.

I am saying that a theoretical pg_aggregate.aggorderdoesnotmatter could
bestow upon ANY_VALUE the ability to make those two queries equivalent.

If you care about which value you get back, use something else.
--
Vik Fearing

On Mon, Dec 5, 2022 at 10:40 PM Vik Fearing <vik@postgresfriends.org> wrote:

On 12/6/22 05:57, David G. Johnston wrote:

On Mon, Dec 5, 2022 at 9:48 PM Vik Fearing <vik@postgresfriends.org>

wrote:

I can imagine an optimization that would remove an ORDER BY clause
because it isn't needed for any other aggregate.

I'm referring to the query:

select any_value(v order by v) from (values (2),(1),(3)) as vals (v);
// produces 1, per the documented implementation-defined behavior.

Implementation-dependent. It is NOT implementation-defined, per spec.

I really don't care all that much about the spec here given that ORDER BY

in an aggregate call is non-spec.

We often loosen the spec rules when they don't make technical sense to

us, but I don't know of any example of when we have tightened them.

The function has to choose some row from among its inputs, and the system
has to obey an order by specification added to the function call. You are
de-facto creating a first_value aggregate (which is by definition
non-standard) whether you like it or not. I'm just saying to be upfront
and honest about it - our users do want such a capability so maybe accept
that there is a first time for everything. Not that picking an
advantageous "implementation-dependent" implementation should be considered
deviating from the spec.

Someone writing:

select any_value(v) from (values (2),(1),(3)) as vals (v) order by v;

Is not presently, nor am I saying, promised the value 1.

I'm assuming you are thinking of the second query form, while the

guarantee

only needs to apply to the first.

I am saying that a theoretical pg_aggregate.aggorderdoesnotmatter could
bestow upon ANY_VALUE the ability to make those two queries equivalent.

That theoretical idea should not be entertained. Removing a user's
explicitly added ORDER BY should be off-limits. Any approach at
optimization here should simply look at whether an ORDER BY is specified
and pass that information to the function. If the function itself really
believes that ordering matters it can emit its own runtime exception
stating that fact and the user can fix their query.

If you care about which value you get back, use something else.

There isn't a "something else" to use so that isn't presently an option.

I suppose it comes down to what level of belief and care you have that
people will simply mis-use this function if it is added in its current form
to get the desired first_value effect that it produces.

David J.

On Tue, Dec 6, 2022 at 4:57 AM David G. Johnston
<david.g.johnston@gmail.com> wrote:
...

I'm referring to the query:

select any_value(v order by v) from (values (2),(1),(3)) as vals (v);
// produces 1, per the documented implementation-defined behavior.

Someone writing:

select any_value(v) from (values (2),(1),(3)) as vals (v) order by v;

Is not presently, nor am I saying, promised the value 1.

Shouldn't the 2nd query be producing an error, as it has an implied
GROUP BY () - so column v cannot appear (unless aggregated) in SELECT
and ORDER BY?

On Wed, Dec 7, 2022 at 1:58 AM Pantelis Theodosiou <ypercube@gmail.com>
wrote:

On Tue, Dec 6, 2022 at 4:57 AM David G. Johnston
<david.g.johnston@gmail.com> wrote:
...

I'm referring to the query:

select any_value(v order by v) from (values (2),(1),(3)) as vals (v);
// produces 1, per the documented implementation-defined behavior.

Someone writing:

select any_value(v) from (values (2),(1),(3)) as vals (v) order by v;

Is not presently, nor am I saying, promised the value 1.

Shouldn't the 2nd query be producing an error, as it has an implied
GROUP BY () - so column v cannot appear (unless aggregated) in SELECT
and ORDER BY?

Right, that should be written as:

select any_value(v) from (values (2),(1),(3) order by 1) as vals (v);

(you said SELECT; the discussion here is that any_value is going to be
added as a new aggregate function)

David J.

On 12/7/22 04:22, David G. Johnston wrote:

On Mon, Dec 5, 2022 at 10:40 PM Vik Fearing <vik@postgresfriends.org> wrote:

On 12/6/22 05:57, David G. Johnston wrote:

On Mon, Dec 5, 2022 at 9:48 PM Vik Fearing <vik@postgresfriends.org>

wrote:

I can imagine an optimization that would remove an ORDER BY clause
because it isn't needed for any other aggregate.

I'm referring to the query:

select any_value(v order by v) from (values (2),(1),(3)) as vals (v);
// produces 1, per the documented implementation-defined behavior.

Implementation-dependent. It is NOT implementation-defined, per spec.

I really don't care all that much about the spec here given that ORDER BY
in an aggregate call is non-spec.

Well, this is demonstrably wrong.

<array aggregate function> ::=
ARRAY_AGG <left paren>
<value expression>
[ ORDER BY <sort specification list> ]
<right paren>

We often loosen the spec rules when they don't make technical sense to
us, but I don't know of any example of when we have tightened them.

The function has to choose some row from among its inputs,

True.

and the system has to obey an order by specification added to the function call.

False.

You are de-facto creating a first_value aggregate (which is by definition
non-standard) whether you like it or not.

I am de jure creating an any_value aggregate (which is by definition
standard) whether you like it or not.

I'm just saying to be upfront
and honest about it - our users do want such a capability so maybe accept
that there is a first time for everything. Not that picking an
advantageous "implementation-dependent" implementation should be considered
deviating from the spec.

Someone writing:

select any_value(v) from (values (2),(1),(3)) as vals (v) order by v;

Is not presently, nor am I saying, promised the value 1.

I'm assuming you are thinking of the second query form, while the

guarantee

only needs to apply to the first.

I am saying that a theoretical pg_aggregate.aggorderdoesnotmatter could
bestow upon ANY_VALUE the ability to make those two queries equivalent.

That theoretical idea should not be entertained. Removing a user's
explicitly added ORDER BY should be off-limits. Any approach at
optimization here should simply look at whether an ORDER BY is specified
and pass that information to the function. If the function itself really
believes that ordering matters it can emit its own runtime exception
stating that fact and the user can fix their query.

It absolutely should be entertained, and I plan on doing so in an
upcoming thread. Whether it errors or ignores is something that should
be discussed on that thread.

If you care about which value you get back, use something else.

There isn't a "something else" to use so that isn't presently an option.

The query

SELECT proposed_first_value(x ORDER BY y) FROM ...

is equivalent to

SELECT (ARRAY_AGG(x ORDER BY y))[1] FROM ...

so I am not very sympathetic to your claim of "no other option".

I suppose it comes down to what level of belief and care you have that
people will simply mis-use this function if it is added in its current form
to get the desired first_value effect that it produces.

People who rely on explicitly undefined behavior get what they deserve
when the implementation changes.
--
Vik Fearing

On Wed, Dec 7, 2022 at 10:00 PM Vik Fearing <vik@postgresfriends.org> wrote:

On 12/7/22 04:22, David G. Johnston wrote:

On Mon, Dec 5, 2022 at 10:40 PM Vik Fearing <vik@postgresfriends.org>

wrote:

On 12/6/22 05:57, David G. Johnston wrote:

On Mon, Dec 5, 2022 at 9:48 PM Vik Fearing <vik@postgresfriends.org>

wrote:

I can imagine an optimization that would remove an ORDER BY clause
because it isn't needed for any other aggregate.

I'm referring to the query:

select any_value(v order by v) from (values (2),(1),(3)) as vals (v);
// produces 1, per the documented implementation-defined behavior.

Implementation-dependent. It is NOT implementation-defined, per spec.

I really don't care all that much about the spec here given that ORDER BY
in an aggregate call is non-spec.

Well, this is demonstrably wrong.

<array aggregate function> ::=
ARRAY_AGG <left paren>
<value expression>
[ ORDER BY <sort specification list> ]
<right paren>

Demoable only by you and a few others...

We should update our documentation - the source of SQL Standard knowledge
for mere mortals.

https://www.postgresql.org/docs/current/sql-expressions.html#SYNTAX-AGGREGATES

"Note: The ability to specify both DISTINCT and ORDER BY in an aggregate
function is a PostgreSQL extension."

Apparently only DISTINCT remains as our extension.

You are de-facto creating a first_value aggregate (which is by definition
non-standard) whether you like it or not.

I am de jure creating an any_value aggregate (which is by definition
standard) whether you like it or not.

Yes, both statements seem true. At least until we decide to start ignoring
a user's explicit order by clause.

If you care about which value you get back, use something else.

There isn't a "something else" to use so that isn't presently an option.

The query

SELECT proposed_first_value(x ORDER BY y) FROM ...

is equivalent to

SELECT (ARRAY_AGG(x ORDER BY y))[1] FROM ...

so I am not very sympathetic to your claim of "no other option".

Semantically, yes, in terms of performance, not so much, for any
non-trivial sized group.

I'm done, and apologize for getting too emotionally invested in this. I
hope to get others to voice enough +1s to get a first_value function into
core along-side this one (which makes the above discussion either moot or
deferred until there is a concrete use case for ignoring an explicit ORDER
BY). If that doesn't happen, well, it isn't going to make or break us
either way.

David J.

On 12/8/22 06:48, David G. Johnston wrote:

On Wed, Dec 7, 2022 at 10:00 PM Vik Fearing <vik@postgresfriends.org> wrote:

On 12/7/22 04:22, David G. Johnston wrote:

On Mon, Dec 5, 2022 at 10:40 PM Vik Fearing <vik@postgresfriends.org>

wrote:

On 12/6/22 05:57, David G. Johnston wrote:

On Mon, Dec 5, 2022 at 9:48 PM Vik Fearing <vik@postgresfriends.org>

wrote:

I can imagine an optimization that would remove an ORDER BY clause
because it isn't needed for any other aggregate.

I'm referring to the query:

select any_value(v order by v) from (values (2),(1),(3)) as vals (v);
// produces 1, per the documented implementation-defined behavior.

Implementation-dependent. It is NOT implementation-defined, per spec.

I really don't care all that much about the spec here given that ORDER BY
in an aggregate call is non-spec.

Well, this is demonstrably wrong.

<array aggregate function> ::=
ARRAY_AGG <left paren>
<value expression>
[ ORDER BY <sort specification list> ]
<right paren>

Demoable only by you and a few others...

The standard is publicly available. It is strange that we, being so
open, hold ourselves to such a closed standard; but that is what we do.

We should update our documentation - the source of SQL Standard knowledge
for mere mortals.

https://www.postgresql.org/docs/current/sql-expressions.html#SYNTAX-AGGREGATES

"Note: The ability to specify both DISTINCT and ORDER BY in an aggregate
function is a PostgreSQL extension."

Apparently only DISTINCT remains as our extension.

Using DISTINCT in an aggregate is also standard. What that note is
saying is that the standard does not allow *both* to be used at the same
time.

The standard defines these things for specific aggregates whereas we are
much more generic about it and therefore have to deal with the combinations.

I have submitted a doc patch to clarify that.

You are de-facto creating a first_value aggregate (which is by definition
non-standard) whether you like it or not.

I am de jure creating an any_value aggregate (which is by definition
standard) whether you like it or not.

Yes, both statements seem true. At least until we decide to start ignoring
a user's explicit order by clause.

I ran some tests and including an ORDER BY in an aggregate that doesn't
care (like COUNT) is devastating for performance. I will be proposing a
solution to that soon and I invite you to participate in that
conversation when I do.
--
Vik Fearing

On 05.12.22 21:18, Vik Fearing wrote:

On 12/5/22 15:57, Vik Fearing wrote:

The SQL:2023 Standard defines a new aggregate named ANY_VALUE.  It
returns an implementation-dependent (i.e. non-deterministic) value
from the rows in its group.

PFA an implementation of this aggregate.

Here is v2 of this patch.  I had forgotten to update sql_features.txt.

In your patch, the documentation says the definition is any_value("any")
but the catalog definitions are any_value(anyelement). Please sort that
out.

Since the transition function is declared strict, null values don't need
to be checked. I think the whole function could be reduced to

Datum
any_value_trans(PG_FUNCTION_ARGS)
{
PG_RETURN_DATUM(PG_GETARG_DATUM(0));
}

Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:

On 05.12.22 21:18, Vik Fearing wrote:

On 12/5/22 15:57, Vik Fearing wrote:

The SQL:2023 Standard defines a new aggregate named ANY_VALUE.  It
returns an implementation-dependent (i.e. non-deterministic) value
from the rows in its group.

Since the transition function is declared strict, null values don't need
to be checked.

Hmm, but should it be strict? That means that what it's returning
is *not* "any value" but "any non-null value". What does the draft
spec have to say about that?

regards, tom lane

On 1/18/23 16:55, Tom Lane wrote:

Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:

On 05.12.22 21:18, Vik Fearing wrote:

On 12/5/22 15:57, Vik Fearing wrote:

The SQL:2023 Standard defines a new aggregate named ANY_VALUE.  It
returns an implementation-dependent (i.e. non-deterministic) value
from the rows in its group.

Since the transition function is declared strict, null values don't need
to be checked.

Hmm, but should it be strict? That means that what it's returning
is *not* "any value" but "any non-null value". What does the draft
spec have to say about that?

It falls into the same category as AVG() etc. That is, nulls are
removed before calculation.
--
Vik Fearing

On 1/18/23 16:06, Peter Eisentraut wrote:

On 05.12.22 21:18, Vik Fearing wrote:

On 12/5/22 15:57, Vik Fearing wrote:

The SQL:2023 Standard defines a new aggregate named ANY_VALUE.  It
returns an implementation-dependent (i.e. non-deterministic) value
from the rows in its group.

PFA an implementation of this aggregate.

Here is v2 of this patch.  I had forgotten to update sql_features.txt.

In your patch, the documentation says the definition is any_value("any")
but the catalog definitions are any_value(anyelement).  Please sort that
out.

Since the transition function is declared strict, null values don't need
to be checked.

Thank you for the review. Attached is a new version rebased to d540a02a72.
--
Vik Fearing

On 18.01.23 18:01, Vik Fearing wrote:

On 1/18/23 16:06, Peter Eisentraut wrote:

On 05.12.22 21:18, Vik Fearing wrote:

On 12/5/22 15:57, Vik Fearing wrote:

The SQL:2023 Standard defines a new aggregate named ANY_VALUE.  It
returns an implementation-dependent (i.e. non-deterministic) value
from the rows in its group.

PFA an implementation of this aggregate.

Here is v2 of this patch.  I had forgotten to update sql_features.txt.

In your patch, the documentation says the definition is
any_value("any") but the catalog definitions are
any_value(anyelement).  Please sort that out.

Since the transition function is declared strict, null values don't
need to be checked.

Thank you for the review.  Attached is a new version rebased to d540a02a72.

This looks good to me now.

On Thu, 19 Jan 2023 at 06:01, Vik Fearing <vik@postgresfriends.org> wrote:

Thank you for the review. Attached is a new version rebased to d540a02a72.

I've only a bunch of nit-picks, personal preferences and random
thoughts to offer as a review:

1. I'd be inclined *not* to mention the possible future optimisation in:

+ * Currently this just returns the first value, but in the future it might be
+ * able to signal to the aggregate that it does not need to be called anymore.

I think it's unlikely that the transfn would "signal" such a thing. It
seems more likely if we did anything about it that nodeAgg.c would
maybe have some additional knowledge not to call that function if the
agg state already has a value. Just so we're not preempting how we
might do such a thing in the future, it seems best just to remove the
mention of it. I don't really think it serves as a good reminder that
we might want to do this one day anyway.

2. +any_value_trans(PG_FUNCTION_ARGS)

Many of transition function names end in "transfn", not "trans". I
think it's better to follow the existing (loosely followed) naming
pattern that a few aggregates seem to follow rather than invent a new
one.

3. I tend to try to copy the capitalisation of keywords from the
surrounding regression tests. I see the following breaks that.

+SELECT any_value(v) FILTER (WHERE v > 2) FROM (VALUES (1), (2), (3)) AS v (v);

(obviously, ideally, we'd always just follow the same capitalisation
of keywords everywhere in each .sql file, but we've long broken that
and the best way can do is be consistent with surrounding tests)

4. I think I'd use the word "Returns" instead of "Chooses" in:

+ Chooses a non-deterministic value from the non-null input values.

5. I've not managed to find a copy of the 2023 draft, so I'm assuming
you've got the ignoring of NULLs correct. I tried to see what other
databases do using https://www.db-fiddle.com/ . I was surprised to see
MySQL 8.0 returning NULL with:

create table a (a int, b int);
insert into a values(1,null),(1,2),(1,null);

select any_value(b) from a group by a;

I'd have expected "2" to be returned. (It gets even weirder without
the GROUP BY clause, so I'm not too hopeful any useful information can
be obtained from looking here)

I know MySQL doesn't follow the spec quite as closely as we do, so I
might not be that surprised if they didn't pay attention to the
wording when implementing this, however, I've not seen the spec, so I
can only speculate what value should be returned. Certainly not doing
any aggregation for any_value() when there is no GROUP BY seems
strange. I see they don't do the same with sum(). Perhaps this is just
a side effect of their loose standards when it came to columns in the
SELECT clause that are not in the GROUP BY clause.

6. Is it worth adding a WindowFunc test somewhere in window.sql with
an any_value(...) over (...)? Is what any_value() returns as a
WindowFunc equally as non-deterministic as when it's used as an
Aggref? Can we assume there's no guarantee that it'll return the same
value for each partition in each row? Does the spec mention anything
about that?

7. I wondered if it's worth adding a
SupportRequestOptimizeWindowClause support function for this
aggregate. I'm thinking that it might not be as likely people would
use something more specific like first_value/nth_value/last_value
instead of using any_value as a WindowFunc. Also, I'm currently
thinking that a SupportRequestWFuncMonotonic for any_value() is not
worth the dozen or so lines of code it would take to write it. I'm
assuming it would always be a MONOTONICFUNC_BOTH function. It seems
unlikely that someone would have a subquery with a WHERE clause in the
upper-level query referencing the any_value() aggregate. Thought I'd
mention both of these things anyway as someone else might think of
some good reason we should add them that I didn't think of.

David

On 1/23/23 08:50, David Rowley wrote:

On Thu, 19 Jan 2023 at 06:01, Vik Fearing <vik@postgresfriends.org> wrote:

Thank you for the review. Attached is a new version rebased to d540a02a72.

I've only a bunch of nit-picks, personal preferences and random
thoughts to offer as a review:

1. I'd be inclined *not* to mention the possible future optimisation in:

+ * Currently this just returns the first value, but in the future it might be
+ * able to signal to the aggregate that it does not need to be called anymore.

I think it's unlikely that the transfn would "signal" such a thing. It
seems more likely if we did anything about it that nodeAgg.c would
maybe have some additional knowledge not to call that function if the
agg state already has a value. Just so we're not preempting how we
might do such a thing in the future, it seems best just to remove the
mention of it. I don't really think it serves as a good reminder that
we might want to do this one day anyway.

Modified. My logic in having the transition function signal that it is
finished is to one day allow something like:

array_agg(x order by y limit z)

2. +any_value_trans(PG_FUNCTION_ARGS)

Many of transition function names end in "transfn", not "trans". I
think it's better to follow the existing (loosely followed) naming
pattern that a few aggregates seem to follow rather than invent a new
one.

Renamed.

3. I tend to try to copy the capitalisation of keywords from the
surrounding regression tests. I see the following breaks that.

+SELECT any_value(v) FILTER (WHERE v > 2) FROM (VALUES (1), (2), (3)) AS v (v);

(obviously, ideally, we'd always just follow the same capitalisation
of keywords everywhere in each .sql file, but we've long broken that
and the best way can do is be consistent with surrounding tests)

Downcased.

4. I think I'd use the word "Returns" instead of "Chooses" in:

+ Chooses a non-deterministic value from the non-null input values.

Done.

5. I've not managed to find a copy of the 2023 draft, so I'm assuming
you've got the ignoring of NULLs correct.

Yes, I do. This is part of <computational operation>, so SQL:2016 10.9
GR 7.a applies.

6. Is it worth adding a WindowFunc test somewhere in window.sql with
an any_value(...) over (...)? Is what any_value() returns as a
WindowFunc equally as non-deterministic as when it's used as an
Aggref? Can we assume there's no guarantee that it'll return the same
value for each partition in each row? Does the spec mention anything
about that?

This is governed by SQL:2016 10.9 GR 1.d and 1.e which defines the
source rows for the aggregate: either a group or a window frame. There
is no difference in behavior. I don't think a windowed test is useful
here unless I were to implement moving transitions. I think that might
be overkill for this function.

7. I wondered if it's worth adding a
SupportRequestOptimizeWindowClause support function for this
aggregate. I'm thinking that it might not be as likely people would
use something more specific like first_value/nth_value/last_value
instead of using any_value as a WindowFunc. Also, I'm currently
thinking that a SupportRequestWFuncMonotonic for any_value() is not
worth the dozen or so lines of code it would take to write it. I'm
assuming it would always be a MONOTONICFUNC_BOTH function. It seems
unlikely that someone would have a subquery with a WHERE clause in the
upper-level query referencing the any_value() aggregate. Thought I'd
mention both of these things anyway as someone else might think of
some good reason we should add them that I didn't think of.

I thought about this for a while and decided that it was not worthwhile.

v4 attached. I am putting this back to Needs Review in the commitfest
app, but these changes were editorial so it is probably RfC like Peter
had set it. I will let you be the judge of that.
--
Vik Fearing

I could have used such an aggregate in the past, so +1.

This is maybe getting into nit-picking, but perhaps it should be
documented as returning an "arbitrary" value instead of a
"non-deterministic" one? Technically the value is deterministic:
there's a concrete algorithm specifying how it's selected. However,
the algorithm is reserved as an implementation detail, since the
function is designed for cases in which the caller should not care
which value is returned.

Thanks,
Maciek

On 09.02.23 10:42, Vik Fearing wrote:

v4 attached.  I am putting this back to Needs Review in the commitfest
app, but these changes were editorial so it is probably RfC like Peter
had set it.  I will let you be the judge of that.

I have committed this.

I made a few small last-minute tweaks:

- Changed "non-deterministic" to "arbitrary", as suggested by Maciek
Sakrejda nearby. This seemed like a handier and less jargony term.

- Removed trailing whitespace in misc.c.

- Changed the function description in pg_proc.dat. Apparently, we are
using 'aggregate transition function' there for all aggregate functions
(instead of 'any_value transition function' etc.).

- Made the tests a bit more interested by feeding in more rows and a mix
of null and nonnull values.