Implementing Incremental View Maintenance
Hi,
I would like to implement Incremental View Maintenance (IVM) on PostgreSQL.
IVM is a technique to maintain materialized views which computes and applies
only the incremental changes to the materialized views rather than
recomputate the contents as the current REFRESH command does.
I had a presentation on our PoC implementation of IVM at PGConf.eu 2018 [1]https://www.postgresql.eu/events/pgconfeu2018/schedule/session/2195-implementing-incremental-view-maintenance-on-postgresql/.
Our implementation uses row OIDs to compute deltas for materialized views.
The basic idea is that if we have information about which rows in base tables
are contributing to generate a certain row in a matview then we can identify
the affected rows when a base table is updated. This is based on an idea of
Dr. Masunaga [2]https://ipsj.ixsq.nii.ac.jp/ej/index.php?active_action=repository_view_main_item_detail&page_id=13&block_id=8&item_id=191254&item_no=1 (Japanese only) who is a member of our group and inspired from ID-based
approach[3]https://dl.acm.org/citation.cfm?id=2750546.
In our implementation, the mapping of the row OIDs of the materialized view
and the base tables are stored in "OID map". When a base relation is modified,
AFTER trigger is executed and the delta is recorded in delta tables using
the transition table feature. The accual udpate of the matview is triggerd
by REFRESH command with INCREMENTALLY option.
However, we realize problems of our implementation. First, WITH OIDS will
be removed since PG12, so OIDs are no longer available. Besides this, it would
be hard to implement this since it needs many changes of executor nodes to
collect base tables's OIDs during execuing a query. Also, the cost of maintaining
OID map would be high.
For these reasons, we started to think to implement IVM without relying on OIDs
and made a bit more surveys.
We also looked at Kevin Grittner's discussion [4]/messages/by-id/1368561126.64093.YahooMailNeo@web162904.mail.bf1.yahoo.com on incremental matview
maintenance. In this discussion, Kevin proposed to use counting algorithm [5]https://dl.acm.org/citation.cfm?id=170066
to handle projection views (using DISTNICT) properly. This algorithm need an
additional system column, count_t, in materialized views and delta tables of
base tables.
However, the discussion about IVM is now stoped, so we would like to restart and
progress this.
Through our PoC inplementation and surveys, I think we need to think at least
the followings for implementing IVM.
1. How to extract changes on base tables
I think there would be at least two approaches for it.
- Using transition table in AFTER triggers
- Extracting changes from WAL using logical decoding
In our PoC implementation, we used AFTER trigger and transition tables, but using
logical decoding might be better from the point of performance of base table
modification.
If we can represent a change of UPDATE on a base table as query-like rather than
OLD and NEW, it may be possible to update the materialized view directly instead
of performing delete & insert.
2. How to compute the delta to be applied to materialized views
Essentially, IVM is based on relational algebra. Theorically, changes on base
tables are represented as deltas on this, like "R <- R + dR", and the delta on
the materialized view is computed using base table deltas based on "change
propagation equations". For implementation, we have to derive the equation from
the view definition query (Query tree, or Plan tree?) and describe this as SQL
query to compulte delta to be applied to the materialized view.
There could be several operations for view definition: selection, projection,
join, aggregation, union, difference, intersection, etc. If we can prepare a
module for each operation, it makes IVM extensable, so we can start a simple
view definition, and then support more complex views.
3. How to identify rows to be modifed in materialized views
When applying the delta to the materialized view, we have to identify which row
in the matview is corresponding to a row in the delta. A naive method is matching
by using all columns in a tuple, but clearly this is unefficient. If thematerialized
view has unique index, we can use this. Maybe, we have to force materialized views
to have all primary key colums in their base tables. In our PoC implementation, we
used OID to identify rows, but this will be no longer available as said above.
4. When to maintain materialized views
There are two candidates of the timing of maintenance, immediate (eager) or deferred.
In eager maintenance, the materialized view is updated in the same transaction
where the base table is updated. In deferred maintenance, this is done after the
transaction is commited, for example, when view is accessed, as a response to user
request, etc.
In the previous discussion[4]/messages/by-id/1368561126.64093.YahooMailNeo@web162904.mail.bf1.yahoo.com, it is planned to start from "eager" approach. In our PoC
implementaion, we used the other aproach, that is, using REFRESH command to perform IVM.
I am not sure which is better as a start point, but I begin to think that the eager
approach may be more simple since we don't have to maintain base table changes in other
past transactions.
In the eager maintenance approache, we have to consider a race condition where two
different transactions change base tables simultaneously as discussed in [4]/messages/by-id/1368561126.64093.YahooMailNeo@web162904.mail.bf1.yahoo.com.
[1]: https://www.postgresql.eu/events/pgconfeu2018/schedule/session/2195-implementing-incremental-view-maintenance-on-postgresql/
[2]: https://ipsj.ixsq.nii.ac.jp/ej/index.php?active_action=repository_view_main_item_detail&page_id=13&block_id=8&item_id=191254&item_no=1 (Japanese only)
[3]: https://dl.acm.org/citation.cfm?id=2750546
[4]: /messages/by-id/1368561126.64093.YahooMailNeo@web162904.mail.bf1.yahoo.com
[5]: https://dl.acm.org/citation.cfm?id=170066
Regards,
--
Yugo Nagata <nagata@sraoss.co.jp>
Hi Yugo.
I would like to implement Incremental View Maintenance (IVM) on
PostgreSQL.
Great. :-)
I think it would address an important gap in PostgreSQL’s feature set.
2. How to compute the delta to be applied to materialized views
Essentially, IVM is based on relational algebra. Theorically, changes on
base
tables are represented as deltas on this, like "R <- R + dR", and the
delta on
the materialized view is computed using base table deltas based on "change
propagation equations". For implementation, we have to derive the
equation from
the view definition query (Query tree, or Plan tree?) and describe this as
SQL
query to compulte delta to be applied to the materialized view.
We had a similar discussion in this thread
/messages/by-id/FC784A9F-F599-4DCC-A45D-DBF6FA582D30@QQdd.eu,
and I’m very much in agreement that the "change propagation equations”
approach can solve for a very substantial subset of common MV use cases.
There could be several operations for view definition: selection,
projection,
join, aggregation, union, difference, intersection, etc. If we can
prepare a
module for each operation, it makes IVM extensable, so we can start a
simple
view definition, and then support more complex views.
Such a decomposition also allows ’stacking’, allowing complex MV definitions
to be attacked even with only a small handful of modules.
I did a bit of an experiment to see if "change propagation equations” could
be computed directly from the MV’s pg_node_tree representation in the
catalog in PlPgSQL. I found that pg_node_trees are not particularly friendly
to manipulation in PlPgSQL. Even with a more friendly-to-PlPgSQL
representation (I played with JSONB), then the next problem is making sense
of the structures, and unfortunately amongst the many plan/path/tree utility
functions in the code base, I figured only a very few could be sensibly
exposed to PlPgSQL. Ultimately, although I’m still attracted to the idea,
and I think it could be made to work, native code is the way to go at least
for now.
4. When to maintain materialized views
[...]
In the previous discussion[4], it is planned to start from "eager"
approach. In our PoC
implementaion, we used the other aproach, that is, using REFRESH command
to perform IVM.
I am not sure which is better as a start point, but I begin to think that
the eager
approach may be more simple since we don't have to maintain base table
changes in other
past transactions.
Certainly the eager approach allows progress to be made with less
infrastructure.
I am concerned that the eager approach only addresses a subset of the MV use
case space, though. For example, if we presume that an MV is present because
the underlying direct query would be non-performant, then we have to at
least question whether applying the delta-update would also be detrimental
to some use cases.
In the eager maintenance approache, we have to consider a race condition
where two
different transactions change base tables simultaneously as discussed in
[4]: .
I wonder if that nudges towards a logged approach. If the race is due to
fact of JOIN-worthy tuples been made visible after a COMMIT, but not before,
then does it not follow that the eager approach has to fire some kind of
reconciliation work at COMMIT time? That seems to imply a persistent queue
of some kind, since we can’t assume transactions to be so small to be able
to hold the queue in memory.
Hmm. I hadn’t really thought about that particular corner case. I guess a
‘catch' could be simply be to detect such a concurrent update and demote the
refresh approach by marking the MV stale awaiting a full refresh.
denty.
--
Sent from: http://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html
Hi all, just wanted to say I am very happy to see progress made on this,
my codebase has multiple "materialized tables" which are maintained with
statement triggers (transition tables) and custom functions. They are ugly
and a pain to maintain, but they work because I have no other
solution...for now at least.
I am concerned that the eager approach only addresses a subset of the MV use
case space, though. For example, if we presume that an MV is present
because
the underlying direct query would be non-performant, then we have to at
least question whether applying the delta-update would also be detrimental
to some use cases.
I will say that in my case, as long as my reads of the materialized view
are always consistent with the underlying data, that's what's important. I
don't mind if it's eager, or lazy (as long as lazy still means it will
refresh prior to reading).
Dear all,
We have some result on incremental update for MVs. We generate triggers on
C to do the incremental maintenance. We posted the code to github about 1
year ago, but unfortunately i posted a not-right ctrigger.h header. The
mistake was exposed to me when a person could not compile the generated
triggers and reported to me. And now i re-posted with the right ctrigger.h
file.
You can find the codes of the generator here:
https://github.com/ntqvinh/PgMvIncrementalUpdate/commits/master. You can
find how did we do here:
https://link.springer.com/article/10.1134/S0361768816050066. The paper is
about generating of codes in pl/pgsql. Anyway i see it is useful for
reading the codes. I don't know if i can share the paper or not so that i
don't publish anywhere else. The text about how to generate triggers in C
was published with open-access but unfortunately, it is in Vietnamese.
We are happy if the codes are useful for someone.
Thank you and best regards,
NTQ Vinh
TS. Nguyễn Trần Quốc Vinh
-----------------------------------------------
Chủ nhiệm khoa Tin học
Trường ĐH Sư phạm - ĐH Đà Nẵng
------------------------------------------------
Nguyen Tran Quoc Vinh, PhD
Dean
Faculty of Information Technology
Danang University of Education
Website: http://it.ued.udn.vn; http://www.ued.vn <http://www.ued.udn.vn/>;
http://www.ued.udn.vn
SCV: http://scv.ued.vn/~ntquocvinh <http://scv.ued.udn.vn/~ntquocvinh>
Phone: (+84) 511.6-512-586
Mobile: (+84) 914.78-08-98
On Mon, Dec 31, 2018 at 11:20 PM Adam Brusselback <adambrusselback@gmail.com>
wrote:
Show quoted text
Hi all, just wanted to say I am very happy to see progress made on this,
my codebase has multiple "materialized tables" which are maintained with
statement triggers (transition tables) and custom functions. They are ugly
and a pain to maintain, but they work because I have no other
solution...for now at least.I am concerned that the eager approach only addresses a subset of the MV
use
case space, though. For example, if we presume that an MV is present
because
the underlying direct query would be non-performant, then we have to at
least question whether applying the delta-update would also be detrimental
to some use cases.I will say that in my case, as long as my reads of the materialized view
are always consistent with the underlying data, that's what's important. I
don't mind if it's eager, or lazy (as long as lazy still means it will
refresh prior to reading).
Hi all, just wanted to say I am very happy to see progress made on this,
my codebase has multiple "materialized tables" which are maintained with
statement triggers (transition tables) and custom functions. They are ugly
and a pain to maintain, but they work because I have no other
solution...for now at least.I am concerned that the eager approach only addresses a subset of the MV use
case space, though. For example, if we presume that an MV is present
because
the underlying direct query would be non-performant, then we have to at
least question whether applying the delta-update would also be detrimental
to some use cases.I will say that in my case, as long as my reads of the materialized view
are always consistent with the underlying data, that's what's important. I
don't mind if it's eager, or lazy (as long as lazy still means it will
refresh prior to reading).
Assuming that we want to implement IVM incrementally (that means, for
example, we implement DELETE for IVM in PostgreSQL XX, then INSERT for
IVM for PostgreSQL XX+1... etc.), I think it's hard to do it with an
eager approach if want to MV is always consistent with base tables.
On the other hand, a lazy approach allows to implement IVM
incrementally because we could always let full MV build from scratch
if operations on MV include queries we do not support.
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
Dear All,
The tool analyzes the input query and then generates triggers (trigger
functions and pl/pgsql scripts as well) on all manipulating events
(insert/updates/delete) for all underlying base tables. The triggers do
incremental updates to the table that contains the query result (MV). You
can build the tool, then see the provided example and try the tool. It is
for synchronous maintenance. It was hard tested but you can use it with
your own risk.
For Asynchronous maintenance, we generate 1) triggers on all manipulating
events on base tables to collect all the data changes and save to the
'special' tables; then 2) the tool to do incremental updates of MVs.
Best regards,
Vinh
TS. Nguyễn Trần Quốc Vinh
-----------------------------------------------
Chủ nhiệm khoa Tin học
Trường ĐH Sư phạm - ĐH Đà Nẵng
------------------------------------------------
Nguyen Tran Quoc Vinh, PhD
Dean
Faculty of Information Technology
Danang University of Education
Website: http://it.ued.udn.vn; http://www.ued.vn <http://www.ued.udn.vn/>;
http://www.ued.udn.vn
SCV: http://scv.ued.vn/~ntquocvinh <http://scv.ued.udn.vn/~ntquocvinh>
Phone: (+84) 511.6-512-586
Mobile: (+84) 914.78-08-98
On Mon, Jan 7, 2019 at 9:00 AM Tatsuo Ishii <ishii@sraoss.co.jp> wrote:
Show quoted text
Hi all, just wanted to say I am very happy to see progress made on this,
my codebase has multiple "materialized tables" which are maintained with
statement triggers (transition tables) and custom functions. They areugly
and a pain to maintain, but they work because I have no other
solution...for now at least.I am concerned that the eager approach only addresses a subset of the MV
use
case space, though. For example, if we presume that an MV is present
because
the underlying direct query would be non-performant, then we have to at
least question whether applying the delta-update would also bedetrimental
to some use cases.
I will say that in my case, as long as my reads of the materialized view
are always consistent with the underlying data, that's what'simportant. I
don't mind if it's eager, or lazy (as long as lazy still means it will
refresh prior to reading).Assuming that we want to implement IVM incrementally (that means, for
example, we implement DELETE for IVM in PostgreSQL XX, then INSERT for
IVM for PostgreSQL XX+1... etc.), I think it's hard to do it with an
eager approach if want to MV is always consistent with base tables.On the other hand, a lazy approach allows to implement IVM
incrementally because we could always let full MV build from scratch
if operations on MV include queries we do not support.Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
Hi!
On Thu, Dec 27, 2018 at 4:57 AM Yugo Nagata <nagata@sraoss.co.jp> wrote:
I would like to implement Incremental View Maintenance (IVM) on PostgreSQL.
IVM is a technique to maintain materialized views which computes and applies
only the incremental changes to the materialized views rather than
recomputate the contents as the current REFRESH command does.
That sounds great! I am interested in this topic because I am
interested in reactive/live queries and support for them in
PostgreSQL. [1]/messages/by-id/CAKLmikP+PPB49z8rEEvRjFOD0D2DV72KdqYN7s9fjh9sM_32ZA@mail.gmail.com
In that context, the problem is very similar: based on some state of
query results and updated source tables, determine what should be new
updates to send to the client describing changes to the query results.
So after computing those incremental changes, instead of applying them
to materialized view I would send them to the client. One could see
materialized views only type of consumers of such information about
incremental change.
So I would like to ask if whatever is done in this setting is done in
a way that one could also outside of the context of materialized view.
Not sure what would API be thought.
From the perspective of reactive/live queries, this package [2]https://github.com/nothingisdead/pg-live-query is
interesting. To my understanding, it adds to all base tables two
columns, one for unique ID and one for revision of the row. And then
rewrites queries so that this information is passed all the way to
query results. In this way it can then determine mapping between
inputs and outputs. I am not sure if it then does incremental update
or just uses that to determine if view is invalidated. Not sure if
there is anything about such approach in literature. Or why both index
and revision columns are needed.
For these reasons, we started to think to implement IVM without relying on OIDs
and made a bit more surveys.
I also do not see much difference between asking users to have primary
key on base tables or asking them to have OIDs. Why do you think that
a requirement for primary keys is a hard one? I think we should first
focus on having IVM with base tables with primary keys. Maybe then
later on we could improve on that and make it also work without.
To me personally, having unique index on source tables and also on
materialized view is a reasonable restriction for this feature.
Especially for initial versions of it.
However, the discussion about IVM is now stoped, so we would like to restart and
progress this.
What would be next steps in your view to move this further?
If we can represent a change of UPDATE on a base table as query-like rather than
OLD and NEW, it may be possible to update the materialized view directly instead
of performing delete & insert.
Why do you need OLD and NEW? Don't you need just NEW and a list of
columns which changed from those in NEW? I use such diffing query [4]https://github.com/tozd/node-reactive-postgres/blob/eeda4f28d096b6e552d04c5ea138c258cb5b9389/index.js#L329-L340
to represent changes: first column has a flag telling if the row is
representing insert, update, and remove, the second column tells which
column are being changed in the case of the update, and then the NEW
columns follow.
I think that maybe standardizing structure for representing those
changes would be a good step towards making this modular and reusable.
Because then we can have three parts:
* Recording and storing changes in a standard format.
* A function which given original data, stored changes, computes
updates needed, also in some standard format.
* A function which given original data and updates needed, applies them.
In the previous discussion[4], it is planned to start from "eager" approach. In our PoC
implementaion, we used the other aproach, that is, using REFRESH command to perform IVM.
I am not sure which is better as a start point, but I begin to think that the eager
approach may be more simple since we don't have to maintain base table changes in other
past transactions.
I think if we split things into three parts as I described above, then
this is just a question of configuration. Or you call all three inside
one trigger to update in "eager" fashion. Or you store computed
updates somewhere and then on demand apply those in "lazy" fashion.
In the eager maintenance approache, we have to consider a race condition where two
different transactions change base tables simultaneously as discussed in [4].
But in the case of "lazy" maintenance there is a mirror problem: what
if later changes to base tables invalidate some previous change to the
materialized view. Imagine that one cell in a base table is first
updated too "foo" and we compute an update for the materialized view
to set it to "foo". And then the same cell is updated to "bar" and we
compute an update for the materialized view again. If we have not
applied any of those updates (because we are "lazy") now the
previously computed update can be discarded. We could still apply
both, but it would not be efficient.
[1]: /messages/by-id/CAKLmikP+PPB49z8rEEvRjFOD0D2DV72KdqYN7s9fjh9sM_32ZA@mail.gmail.com
[2]: https://github.com/nothingisdead/pg-live-query
[3]: https://www.postgresql.org/docs/devel/sql-createtable.html
[4]: https://github.com/tozd/node-reactive-postgres/blob/eeda4f28d096b6e552d04c5ea138c258cb5b9389/index.js#L329-L340
Mitar
On Tue, 1 Jan 2019 14:46:25 +0700
Nguyễn Trần Quốc Vinh <ntquocvinh@gmail.com> wrote:
We have some result on incremental update for MVs. We generate triggers on
C to do the incremental maintenance. We posted the code to github about 1
year ago, but unfortunately i posted a not-right ctrigger.h header. The
mistake was exposed to me when a person could not compile the generated
triggers and reported to me. And now i re-posted with the right ctrigger.h
file.You can find the codes of the generator here:
https://github.com/ntqvinh/PgMvIncrementalUpdate/commits/master. You can
find how did we do here:
https://link.springer.com/article/10.1134/S0361768816050066. The paper is
about generating of codes in pl/pgsql. Anyway i see it is useful for
reading the codes. I don't know if i can share the paper or not so that i
don't publish anywhere else. The text about how to generate triggers in C
was published with open-access but unfortunately, it is in Vietnamese.We are happy if the codes are useful for someone.
I have read your paper. It is interesting and great so that the algorithm
is described concretely.
After reading this, I have a few questions about your implementation.
Although I may be able to understand by reading your paper and code carefully,
I would appreciate it if you could answer these.
- It is said there are many limitations on the view definition query.
How does this work when the query is not supported?
- Is it possible to support materialized views that have DISTINCT,
OUTER JOIN, or sub-query in your approach?
- It is said that AVG is splitted to SUM and COUNT. Are these new additional
columns in MV visible for users?
- Does this can handle the race condition discussed in [1]/messages/by-id/1368561126.64093.YahooMailNeo@web162904.mail.bf1.yahoo.com, that is,
if concurrent transactions update different two tables in the join
view definition, is MV updated sucessfully?
[1]: /messages/by-id/1368561126.64093.YahooMailNeo@web162904.mail.bf1.yahoo.com
Regards,
--
Yugo Nagata <nagata@sraoss.co.jp>
On Mon, 7 Jan 2019 00:39:00 -0800
Mitar <mmitar@gmail.com> wrote:
That sounds great! I am interested in this topic because I am
interested in reactive/live queries and support for them in
PostgreSQL. [1]In that context, the problem is very similar: based on some state of
query results and updated source tables, determine what should be new
updates to send to the client describing changes to the query results.
So after computing those incremental changes, instead of applying them
to materialized view I would send them to the client. One could see
materialized views only type of consumers of such information about
incremental change.So I would like to ask if whatever is done in this setting is done in
a way that one could also outside of the context of materialized view.
Not sure what would API be thought.
I didn't know about reactive/live queries but this seems share a part of
problem with IVM, so we might have common API.
BTW, what is uecase of reactive/live queries? (just curious)
For these reasons, we started to think to implement IVM without relying on OIDs
and made a bit more surveys.I also do not see much difference between asking users to have primary
key on base tables or asking them to have OIDs. Why do you think that
a requirement for primary keys is a hard one? I think we should first
focus on having IVM with base tables with primary keys. Maybe then
later on we could improve on that and make it also work without.To me personally, having unique index on source tables and also on
materialized view is a reasonable restriction for this feature.
Especially for initial versions of it.
Initially, I chose to use OIDs for theoretical reason, that is, to handle
"bag-semantics" which allows duplicate rows in tables. However, I agree
that we start from the restriction of having unique index on base tables.
If we can represent a change of UPDATE on a base table as query-like rather than
OLD and NEW, it may be possible to update the materialized view directly instead
of performing delete & insert.Why do you need OLD and NEW? Don't you need just NEW and a list of
columns which changed from those in NEW? I use such diffing query [4]
to represent changes: first column has a flag telling if the row is
representing insert, update, and remove, the second column tells which
column are being changed in the case of the update, and then the NEW
columns follow.
According the change propagation equation approach, OLD is necessary
to calculate tuples in MV to be deleted or modified. However, if tables
has unique keys, such tuples can be identifeid using the keys, so
OLD may not be needed, at least in eager approach.
In lazy approach, OLD contents of table is useful. For example, with
a join view MV = R * S, when dR is inserted into R and dS is inserted
into S, the delta to be inserted into MV will be
dMV = (R_old * dS) + (dR * S_new)
= (R_old * dS) + (dR * S_old) + (dR * dS)
, hence the old contents of tables R and S are needed.
I think that maybe standardizing structure for representing those
changes would be a good step towards making this modular and reusable.
Because then we can have three parts:* Recording and storing changes in a standard format.
* A function which given original data, stored changes, computes
updates needed, also in some standard format.
* A function which given original data and updates needed, applies them.
I think if we split things into three parts as I described above, then
this is just a question of configuration. Or you call all three inside
one trigger to update in "eager" fashion. Or you store computed
updates somewhere and then on demand apply those in "lazy" fashion.
I agree that defining the format to represent changes is important. However,
I am not sure both of eager and lazy can be handled in the same manner. I'll
consider about this more.
In the eager maintenance approache, we have to consider a race condition where two
different transactions change base tables simultaneously as discussed in [4].But in the case of "lazy" maintenance there is a mirror problem: what
if later changes to base tables invalidate some previous change to the
materialized view. Imagine that one cell in a base table is first
updated too "foo" and we compute an update for the materialized view
to set it to "foo". And then the same cell is updated to "bar" and we
compute an update for the materialized view again. If we have not
applied any of those updates (because we are "lazy") now the
previously computed update can be discarded. We could still apply
both, but it would not be efficient.
In our PoC implementation, I handled this situation by removing
old contents from NEW delata table. In your example, when the base
table is updated from "foo" to "bar", the "foo" tuple is removed
from and the "bar" tuple is inserted in NEW delta and the delta
of MV is computed using the final NEW delta.
Regards,
--
Yugo Nagata <nagata@sraoss.co.jp>
Hi!
On Thu, Jan 31, 2019 at 6:20 AM Yugo Nagata <nagata@sraoss.co.jp> wrote:
BTW, what is uecase of reactive/live queries? (just curious)
It allows syncing the state between client and server. Client can then
have a subset of data and server can push changes as they are
happening to the client. Client can in a reactive manner render that
in the UI to the user. So you can easily create a reactive UI which
always shows up-to-date data without having to poll or something
similar.
How are things progressing? Any news on this topic?
Mitar
On Thu, 27 Dec 2018 21:57:26 +0900
Yugo Nagata <nagata@sraoss.co.jp> wrote:
Hi,
I would like to implement Incremental View Maintenance (IVM) on PostgreSQL.
I am now working on an initial patch for implementing IVM on PostgreSQL.
This enables materialized views to be updated incrementally after one
of their base tables is modified.
At the first patch, I want to start from very simple features.
Firstly, this will handle simple definition views which includes only
selection, projection, and join. Standard aggregations (count, sum, avg,
min, max) are not planned to be implemented in the first patch, but these
are commonly used in materialized views, so I'll implement them later on.
Views which include sub-query, outer-join, CTE, and window functions are also
out of scope of the first patch. Also, views including self-join or views
including other views in their definition is not considered well, either.
I need more investigation on these type of views although I found some papers
explaining how to handle sub-quries and outer-joins.
Next, this will handle materialized views with no duplicates in their
tuples. I am thinking of implementing an algorithm to handle duplicates
called "counting-algorithm" afterward, but I'll start from this
no-duplicates assumption in the first patch for simplicity.
In the first patch, I will implement only "immediate maintenance", that is, materialized views are updated immediately in a transaction where a base
table is modified. On other hand, in "deferred maintenance", materialized
views are updated after the transaction, for example, by the user command
like REFRESH. Although I plan to implement both eventually, I'll start from "immediate" because this seems to need smaller code than "deferred". For
implementing "deferred", it is need to implement a mechanism to maintain logs
for recording changes and an algorithm to compute the delta to be applied to
materialized views are necessary.
I plan to implement the immediate maintenance using AFTER triggers created
automatically on a materialized view's base tables. In AFTER trigger using
transition table features, changes occurs on base tables is recorded ephemeral relations. We can compute the delta to be applied to materialized views by
using these ephemeral relations and the view definition query, then update
the view by applying this delta.
--
Yugo Nagata <nagata@sraoss.co.jp>
On Sun, 31 Mar 2019 at 23:22, Yugo Nagata <nagata@sraoss.co.jp> wrote:
Firstly, this will handle simple definition views which includes only
selection, projection, and join. Standard aggregations (count, sum, avg,
min, max) are not planned to be implemented in the first patch, but these
are commonly used in materialized views, so I'll implement them later on.
It's fine to not have all the features from day 1 of course. But I
just picked up this comment and the followup talking about splitting
AVG into SUM and COUNT and I had a comment. When you do look at
tackling aggregates I don't think you should restrict yourself to
these specific standard aggregations. We have all the necessary
abstractions to handle all aggregations that are feasible, see
https://www.postgresql.org/docs/devel/xaggr.html#XAGGR-MOVING-AGGREGATES
What you need to do -- I think -- is store the "moving aggregate
state" before the final function. Then whenever a row is inserted or
deleted or updated (or whenever another column is updated which causes
the value to row to enter or leave the aggregation) apply either
aggtransfn or aggminvtransfn to the state. I'm not sure if you want to
apply the final function on every update or only lazily either may be
better in some usage.
On Mon, 1 Apr 2019 12:11:22 +0900
Yugo Nagata <nagata@sraoss.co.jp> wrote:
On Thu, 27 Dec 2018 21:57:26 +0900
Yugo Nagata <nagata@sraoss.co.jp> wrote:Hi,
I would like to implement Incremental View Maintenance (IVM) on PostgreSQL.
I am now working on an initial patch for implementing IVM on PostgreSQL.
This enables materialized views to be updated incrementally after one
of their base tables is modified.
Attached is a WIP patch of Incremental View Maintenance (IVM).
Major part is written by me, and changes in syntax and pg_class
are Hoshiai-san's work.
Although this is sill a draft patch in work-in-progress, any
suggestions or thoughts would be appreciated.
* What it is
This allows a kind of Immediate Maintenance of materialized views. if a
materialized view is created by CRATE INCREMENTAL MATERIALIZED VIEW command,
the contents of the mateview is updated automatically and incrementally
after base tables are updated. Noted this syntax is just tentative, so it
may be changed.
====== Example 1 ======
postgres=# CREATE INCREMENTAL MATERIALIZED VIEW m AS SELECT * FROM t0;
SELECT 3
postgres=# SELECT * FROM m;
i
---
3
2
1
(3 rows)
postgres=# INSERT INTO t0 VALUES (4);
INSERT 0 1
postgres=# SELECt * FROM m; -- automatically updated
i
---
3
2
1
4
(4 rows)
=============================
This implementation also supports matviews including duplicate tuples or
DISTINCT clause in its view definition query. For example, even if a matview
is defined with DISTINCT to remove duplication of tuples in a base table, this
can perform incremental update of the matview properly. That is, the contents
of the matview doesn't change when exiting tuples are inserted into the base
tables, and a tuple in the matview is deleted only when duplicity of the
corresponding tuple in the base table becomes zero.
This is due to "colunting alogorithm" in which the number of each tuple is
stored in matviews as a special column value.
====== Example 2 ======
postgres=# SELECT * FROM t1;
id | t
----+---
1 | A
2 | B
3 | C
4 | A
(4 rows)
postgres=# CREATE INCREMENTAL MATERIALIZED VIEW m1 AS SELECT t FROM t1;
SELECT 3
postgres=# CREATE INCREMENTAL MATERIALIZED VIEW m2 AS SELECT DISTINCT t FROM t1;
SELECT 3
postgres=# SELECT * FROM m1; -- with duplicity
t
---
A
A
C
B
(4 rows)
postgres=# SELECT * FROM m2;
t
---
A
B
C
(3 rows)
postgres=# INSERT INTO t1 VALUES (5, 'B');
INSERT 0 1
postgres=# DELETE FROM t1 WHERE id IN (1,3); -- delete (1,A),(3,C)
DELETE 2
postgres=# SELECT * FROM m1; -- one A left and one more B
t
---
B
B
A
(3 rows)
postgres=# SELECT * FROM m2; -- only C is removed
t
---
B
A
(2 rows)
=============================
* How it works
1. Creating matview
When a matview is created, AFTER triggers are internally created
on its base tables. When the base tables is modified (INSERT, DELETE,
UPDATE), the matview is updated incrementally in the trigger function.
When populating the matview, GROUP BY and count(*) are added to the
view definition query before this is executed for counting duplicity
of tuples in the matview. The result of count is stored in the matview
as a special column named "__ivm_count__".
2. Maintenance of matview
When base tables are modified, the change set of the table can be
referred as Ephemeral Named Relations (ENRs) thanks to Transition Table
(a feature of trigger implemented since PG10). We can calculate the diff
set of the matview by replacing the base table in the view definition
query with the ENR (at least if it is Selection-Projection -Join view).
As well as view definition time, GROUP BY and count(*) is added in order
to count the duplicity of tuples in the diff set. As a result, two diff
sets (to be deleted from and to be inserted into the matview) are
calculated, and the results are stored into temporary tables respectively.
The matiview is updated by merging these change sets. Instead of executing
DELETE or INSERT simply, the values of __ivm_count__ column in the matview
is decreased or increased. When the values becomes zero, the corresponding
tuple is deleted from the matview.
3. Access to matview
When SELECT is issued for IVM matviews defined with DISTINCT, all columns
except to __ivm_count__ of each tuple in the matview are returned. This is
correct because duplicity of tuples are eliminated by GROUP BY.
When DISTINCT is not used, SELECT for the IVM matviews returns each tuple
__ivm_count__ times. Currently, this is implemented by rewriting the SELECT
query to replace the matview RTE with a subquery which joins the matview
and generate_series function as bellow.
SELECT mv.* FROM mv, generate_series(1, mv.__ivm_count__);
__ivm_count__ column is invisible for users when "SELECT * FROM ..." is
issued, but users can see the value by specifying in target list explicitly.
====== Example 3 ======
postgres=# \d+ m1
Materialized view "public.m1"
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
---------------+--------+-----------+----------+---------+----------+--------------+-------------
t | text | | | | extended | |
__ivm_count__ | bigint | | | | plain | |
View definition:
SELECT t1.t
FROM t1;
Access method: heap
postgres=# \d+ m2
Materialized view "public.m2"
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
---------------+--------+-----------+----------+---------+----------+--------------+-------------
t | text | | | | extended | |
__ivm_count__ | bigint | | | | plain | |
View definition:
SELECT DISTINCT t1.t
FROM t1;
Access method: heap
postgres=# SELECT *, __ivm_count__ FROM m1;
t | __ivm_count__
---+---------------
B | 2
B | 2
A | 1
(3 rows)
postgres=# SELECT *, __ivm_count__ FROM m2;
t | __ivm_count__
---+---------------
B | 2
A | 1
(2 rows)
postgres=# EXPLAIN SELECT * FROM m1;
QUERY PLAN
------------------------------------------------------------------------------
Nested Loop (cost=0.00..61.03 rows=3000 width=2)
-> Seq Scan on m1 mv (cost=0.00..1.03 rows=3 width=10)
-> Function Scan on generate_series (cost=0.00..10.00 rows=1000 width=0)
(3 rows)
=============================
* Simple Performance Evaluation
I confirmed that "incremental" update of matviews is more effective
than the standard REFRESH by using simple exapmle. I used tables
of pgbench (SF=100) here.
Create two matviews, that is, without and with IVM.
test=# CREATE MATERIALIZED VIEW bench1 AS
SELECT aid, bid, abalance, bbalance
FROM pgbench_accounts JOIN pgbench_branches USING (bid)
WHERE abalance > 0 OR bbalance > 0;
SELECT 5001054
test=# CREATE INCREMENTAL MATERIALIZED VIEW bench2 AS
SELECT aid, bid, abalance, bbalance
FROM pgbench_accounts JOIN pgbench_branches USING (bid)
WHERE abalance > 0 OR bbalance > 0;
SELECT 5001054
The standard REFRESH of bench1 took more than 10 seconds.
test=# \timing
Timing is on.
test=# REFRESH MATERIALIZED VIEW bench1 ;
REFRESH MATERIALIZED VIEW
Time: 11210.563 ms (00:11.211)
Create an index on the IVM matview (bench2).
test=# CREATE INDEX on bench2(aid,bid);
CREATE INDEX
Updating a tuple in pgbench_accounts took 18ms. After this, bench2
was updated automatically and correctly.
test=# SELECT * FROM bench2 WHERE aid = 1;
aid | bid | abalance | bbalance
-----+-----+----------+----------
1 | 1 | 10 | 10
(1 row)
Time: 2.498 ms
test=# UPDATE pgbench_accounts SET abalance = 1000 WHERE aid = 1;
UPDATE 1
Time: 18.634 ms
test=# SELECT * FROM bench2 WHERE aid = 1;
aid | bid | abalance | bbalance
-----+-----+----------+----------
1 | 1 | 1000 | 10
(1 row)
However, if there is not the index on bench2, it took 4 sec, so
appropriate indexes are needed on IVM matviews.
test=# DROP INDEX bench2_aid_bid_idx ;
DROP INDEX
Time: 10.613 ms
test=# UPDATE pgbench_accounts SET abalance = 2000 WHERE aid = 1;
UPDATE 1
Time: 3931.274 ms (00:03.931)
* Restrictions on view definition
This patch is still in Work-in-Progress and there are many restrictions
on the view definition query of matviews.
The current implementation supports views including selection, projection,
and inner join with or without DISTINCT. Aggregation and GROUP BY are not
supported yet, but I plan to deal with these by the first release.
Self-join, subqueries, OUTER JOIN, CTE, window functions are not
considered well, either. I need more investigation on these type of views
although I found some papers explaining how to handle sub-queries and
outer-joins.
These unsupported views should be checked when a matview is created, but
this is not implemented yet. Hoshiai-san are working on this.
* Timing of view maintenance
This patch implements a kind of Immediate Maintenance, that is, a matview
is updated immediately when a base table is modified. On other hand, in
"Deferred Maintenance", matviews are updated after the transaction, for
example, by the user command like REFRESH.
For implementing "deferred", it is need to implement a mechanism to maintain
logs for recording changes of base tables and an algorithm to compute the
delta to be applied to matviews.
In addition, there could be another implementation of Immediate Maintenance
in which matview is updated at the end of a transaction that modified base
table, rather than in AFTER trigger. Oracle supports this type of IVM. To
implement this, we will need a mechanism to maintain change logs on base
tables as well as Deferred maintenance.
* Counting algorithm implementation
There will be also discussions on counting-algorithm implementation.
Firstly, the current patch treats "__ivm_count__" as a special column name
in a somewhat ad hoc way. This is used when maintaining and accessing matviews,
and when "SELECT * FROM ..." is issued, __ivm_count__ column is invisible for
users. Maybe this name has to be inhibited in user tables. Is it acceptable
to use such columns for IVM, and is there better way, if not?
Secondly, a matview with duplicate tuples is replaces with a subquery which
uses generate_series function. It does not have to be generate_series, and we
can make a new set returning function for this. Anyway, this internal behaviour
is visible in EXPLAIN results as shown in Example 3. Also, there is a
performance impact because estimated rows number is wrong, and what is worse,
the cost of join is not small when the size of matview is large. Therefore, we
might have to add a new plan node for selecting from matviews rather than using
such a special set returning function.
Ragards,
--
Yugo Nagata <nagata@sraoss.co.jp>
Attachments:
WIP_immediate_IVM.patchtext/x-diff; name=WIP_immediate_IVM.patchDownload
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 4c7e93892a..858b05b427 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -1944,6 +1944,13 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<entry>True if table or index is a partition</entry>
</row>
+ <row>
+ <entry><structfield>relisivm</structfield></entry>
+ <entry><type>bool</type></entry>
+ <entry></entry>
+ <entry>True if materialized view enables incremental view maintenance</entry>
+ </row>
+
<row>
<entry><structfield>relrewrite</structfield></entry>
<entry><type>oid</type></entry>
diff --git a/doc/src/sgml/ref/create_materialized_view.sgml b/doc/src/sgml/ref/create_materialized_view.sgml
index ec8847ed40..a23366a342 100644
--- a/doc/src/sgml/ref/create_materialized_view.sgml
+++ b/doc/src/sgml/ref/create_materialized_view.sgml
@@ -21,7 +21,7 @@ PostgreSQL documentation
<refsynopsisdiv>
<synopsis>
-CREATE MATERIALIZED VIEW [ IF NOT EXISTS ] <replaceable>table_name</replaceable>
+CREATE [ INCREMENTAL ] MATERIALIZED VIEW [ IF NOT EXISTS ] <replaceable>table_name</replaceable>
[ (<replaceable>column_name</replaceable> [, ...] ) ]
[ USING <replaceable class="parameter">method</replaceable> ]
[ WITH ( <replaceable class="parameter">storage_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] ) ]
@@ -54,6 +54,16 @@ CREATE MATERIALIZED VIEW [ IF NOT EXISTS ] <replaceable>table_name</replaceable>
<title>Parameters</title>
<variablelist>
+ <varlistentry>
+ <term><literal>INCREMENTAL</literal></term>
+ <listitem>
+ <para>
+ If specified, a materialized view enables incremental view maintenance.
+ You can replace only the contents of a materialized view, which based rows are changed.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>IF NOT EXISTS</literal></term>
<listitem>
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index ee6b72e550..823486c968 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -891,6 +891,7 @@ InsertPgClassTuple(Relation pg_class_desc,
values[Anum_pg_class_relrewrite - 1] = ObjectIdGetDatum(rd_rel->relrewrite);
values[Anum_pg_class_relfrozenxid - 1] = TransactionIdGetDatum(rd_rel->relfrozenxid);
values[Anum_pg_class_relminmxid - 1] = MultiXactIdGetDatum(rd_rel->relminmxid);
+ values[Anum_pg_class_relisivm - 1] = BoolGetDatum(rd_rel->relisivm);
if (relacl != (Datum) 0)
values[Anum_pg_class_relacl - 1] = relacl;
else
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index c8d22e1b65..57cd30718b 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -910,6 +910,7 @@ index_create(Relation heapRelation,
indexRelation->rd_rel->relowner = heapRelation->rd_rel->relowner;
indexRelation->rd_rel->relam = accessMethodObjectId;
indexRelation->rd_rel->relispartition = OidIsValid(parentIndexRelid);
+ indexRelation->rd_rel->relisivm = false;
/*
* store index's pg_class entry
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index 43c2fa9124..0098531dfd 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -51,6 +51,14 @@
#include "utils/rls.h"
#include "utils/snapmgr.h"
+#include "catalog/pg_trigger.h"
+#include "catalog/pg_type.h"
+#include "commands/trigger.h"
+#include "parser/parser.h"
+#include "parser/parsetree.h"
+#include "parser/parse_func.h"
+#include "nodes/print.h"
+
typedef struct
{
@@ -74,6 +82,8 @@ static bool intorel_receive(TupleTableSlot *slot, DestReceiver *self);
static void intorel_shutdown(DestReceiver *self);
static void intorel_destroy(DestReceiver *self);
+static void CreateIvmTrigger(Oid relOid, Oid viewOid, char *matviewname, int16 type);
+static void CreateIvmTriggersOnBaseTables(Query *qry, Node *jtnode, Oid matviewOid, char* matviewname);
/*
* create_ctas_internal
@@ -109,6 +119,8 @@ create_ctas_internal(List *attrList, IntoClause *into)
create->oncommit = into->onCommit;
create->tablespacename = into->tableSpaceName;
create->if_not_exists = false;
+ /* Using Materialized view only */
+ create->ivm = into->ivm;
create->accessMethod = into->accessMethod;
/*
@@ -239,6 +251,7 @@ ExecCreateTableAs(CreateTableAsStmt *stmt, const char *queryString,
List *rewritten;
PlannedStmt *plan;
QueryDesc *queryDesc;
+ Query *copied_query;
if (stmt->if_not_exists)
{
@@ -319,7 +332,29 @@ ExecCreateTableAs(CreateTableAsStmt *stmt, const char *queryString,
* and is executed repeatedly. (See also the same hack in EXPLAIN and
* PREPARE.)
*/
- rewritten = QueryRewrite(copyObject(query));
+
+ copied_query = copyObject(query);
+ if (is_matview && into->ivm)
+ {
+ TargetEntry *tle;
+ Node *node;
+ ParseState *pstate = make_parsestate(NULL);
+
+ FuncCall *fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+ fn->agg_star = true;
+
+ copied_query->groupClause = transformDistinctClause(NULL, &copied_query->targetList, copied_query->sortClause, false);
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(copied_query->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ copied_query->targetList = lappend(copied_query->targetList, tle);
+ copied_query->hasAggs = true;
+ }
+
+ rewritten = QueryRewrite(copied_query);
/* SELECT should never rewrite to more or less than one SELECT query */
if (list_length(rewritten) != 1)
@@ -378,11 +413,65 @@ ExecCreateTableAs(CreateTableAsStmt *stmt, const char *queryString,
/* Restore userid and security context */
SetUserIdAndSecContext(save_userid, save_sec_context);
+
+
+ if (into->ivm)
+ {
+ char *matviewname;
+ Oid matviewOid = address.objectId;
+ Relation matviewRel = heap_open(matviewOid, NoLock);
+ matviewname = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(matviewRel)),
+ RelationGetRelationName(matviewRel));
+ copied_query = copyObject(query);
+ AcquireRewriteLocks(copied_query, true, false);
+
+ CreateIvmTriggersOnBaseTables(copied_query, (Node *)copied_query->jointree, matviewOid, matviewname);
+
+ heap_close(matviewRel, NoLock);
+ }
}
return address;
}
+static void CreateIvmTriggersOnBaseTables(Query *qry, Node *jtnode, Oid matviewOid, char* matviewname)
+{
+
+ if (jtnode == NULL)
+ return;
+ if (IsA(jtnode, RangeTblRef))
+ {
+ int rti = ((RangeTblRef *) jtnode)->rtindex;
+ RangeTblEntry *rte = rt_fetch(rti, qry->rtable);
+
+ if (rte->rtekind == RTE_RELATION)
+ {
+ CreateIvmTrigger(rte->relid, matviewOid, matviewname, TRIGGER_TYPE_INSERT);
+ CreateIvmTrigger(rte->relid, matviewOid, matviewname, TRIGGER_TYPE_DELETE);
+ CreateIvmTrigger(rte->relid, matviewOid, matviewname, TRIGGER_TYPE_UPDATE);
+ }
+ else
+ elog(ERROR, "unsupported RTE kind: %d", (int) rte->rtekind);
+ }
+ else if (IsA(jtnode, FromExpr))
+ {
+ FromExpr *f = (FromExpr *) jtnode;
+ ListCell *l;
+
+ foreach(l, f->fromlist)
+ CreateIvmTriggersOnBaseTables(qry, lfirst(l), matviewOid, matviewname);
+ }
+ else if (IsA(jtnode, JoinExpr))
+ {
+ JoinExpr *j = (JoinExpr *) jtnode;
+
+ CreateIvmTriggersOnBaseTables(qry, j->larg, matviewOid, matviewname);
+ CreateIvmTriggersOnBaseTables(qry, j->rarg, matviewOid, matviewname);
+ }
+ else
+ elog(ERROR, "unrecognized node type: %d", (int) nodeTag(jtnode));
+}
+
/*
* GetIntoRelEFlags --- compute executor flags needed for CREATE TABLE AS
*
@@ -547,6 +636,11 @@ intorel_startup(DestReceiver *self, int operation, TupleDesc typeinfo)
if (is_matview && !into->skipData)
SetMatViewPopulatedState(intoRelationDesc, true);
+ /*
+ * Mark relisivm field, if it's a matview and into->ivm is true.
+ */
+ if (is_matview && into->ivm)
+ SetMatViewIVMState(intoRelationDesc, true);
/*
* Fill private fields of myState for use by later routines
*/
@@ -619,3 +713,74 @@ intorel_destroy(DestReceiver *self)
{
pfree(self);
}
+
+static void
+CreateIvmTrigger(Oid relOid, Oid viewOid, char *matviewname, int16 type)
+{
+ CreateTrigStmt *ivm_trigger;
+ List *transitionRels = NIL;
+ ObjectAddress address, refaddr;
+
+ refaddr.classId = RelationRelationId;
+ refaddr.objectId = viewOid;
+ refaddr.objectSubId = 0;
+
+
+ ivm_trigger = makeNode(CreateTrigStmt);
+ ivm_trigger->relation = NULL;
+ ivm_trigger->row = false;
+ ivm_trigger->timing = TRIGGER_TYPE_AFTER;
+
+ ivm_trigger->events = type;
+
+ switch (type)
+ {
+ case TRIGGER_TYPE_INSERT:
+ ivm_trigger->trigname = "IVM_trigger_ins";
+ break;
+ case TRIGGER_TYPE_DELETE:
+ ivm_trigger->trigname = "IVM_trigger_del";
+ break;
+ case TRIGGER_TYPE_UPDATE:
+ ivm_trigger->trigname = "IVM_trigger_upd";
+ break;
+ }
+
+ if (type == TRIGGER_TYPE_INSERT || type == TRIGGER_TYPE_UPDATE)
+ {
+ TriggerTransition *n = makeNode(TriggerTransition);
+ n->name = "ivm_newtable";
+ n->isNew = true;
+ n->isTable = true;
+
+ transitionRels = lappend(transitionRels, n);
+ }
+ if (type == TRIGGER_TYPE_DELETE || type == TRIGGER_TYPE_UPDATE)
+ {
+ TriggerTransition *n = makeNode(TriggerTransition);
+ n->name = "ivm_oldtable";
+ n->isNew = false;
+ n->isTable = true;
+
+ transitionRels = lappend(transitionRels, n);
+ }
+
+ ivm_trigger->funcname = SystemFuncName("IVM_immediate_maintenance");
+
+ ivm_trigger->columns = NIL;
+ ivm_trigger->transitionRels = transitionRels;
+ ivm_trigger->whenClause = NULL;
+ ivm_trigger->isconstraint = false;
+ ivm_trigger->deferrable = false;
+ ivm_trigger->initdeferred = false;
+ ivm_trigger->constrrel = NULL;
+ ivm_trigger->args = list_make1(makeString(matviewname));
+
+ address = CreateTrigger(ivm_trigger, NULL, relOid, InvalidOid, InvalidOid,
+ InvalidOid, InvalidOid, InvalidOid, NULL, true, false);
+
+ recordDependencyOn(&address, &refaddr, DEPENDENCY_AUTO);
+
+ /* Make changes-so-far visible */
+ CommandCounterIncrement();
+}
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index 99bf3c29f2..a403b93b4c 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -46,6 +46,11 @@
#include "utils/snapmgr.h"
#include "utils/syscache.h"
+#include "utils/regproc.h"
+#include "nodes/makefuncs.h"
+#include "parser/parse_clause.h"
+#include "parser/parse_func.h"
+
typedef struct
{
@@ -65,6 +70,7 @@ static bool transientrel_receive(TupleTableSlot *slot, DestReceiver *self);
static void transientrel_shutdown(DestReceiver *self);
static void transientrel_destroy(DestReceiver *self);
static uint64 refresh_matview_datafill(DestReceiver *dest, Query *query,
+ QueryEnvironment *queryEnv,
const char *queryString);
static char *make_temptable_name_n(char *tempname, int n);
static void refresh_by_match_merge(Oid matviewOid, Oid tempOid, Oid relowner,
@@ -74,6 +80,9 @@ static bool is_usable_unique_index(Relation indexRel);
static void OpenMatViewIncrementalMaintenance(void);
static void CloseMatViewIncrementalMaintenance(void);
+static void apply_delta(Oid matviewOid, Oid tempOid_new, Oid tempOid_old,
+ Oid relowner, int save_sec_context);
+
/*
* SetMatViewPopulatedState
* Mark a materialized view as populated, or not.
@@ -114,6 +123,46 @@ SetMatViewPopulatedState(Relation relation, bool newstate)
CommandCounterIncrement();
}
+/*
+ * SetMatViewIVMState
+ * Mark a materialized view as IVM, or not.
+ *
+ * NOTE: caller must be holding an appropriate lock on the relation.
+ */
+void
+SetMatViewIVMState(Relation relation, bool newstate)
+{
+ Relation pgrel;
+ HeapTuple tuple;
+
+ Assert(relation->rd_rel->relkind == RELKIND_MATVIEW);
+
+ /*
+ * Update relation's pg_class entry. Crucial side-effect: other backends
+ * (and this one too!) are sent SI message to make them rebuild relcache
+ * entries.
+ */
+ pgrel = table_open(RelationRelationId, RowExclusiveLock);
+ tuple = SearchSysCacheCopy1(RELOID,
+ ObjectIdGetDatum(RelationGetRelid(relation)));
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for relation %u",
+ RelationGetRelid(relation));
+
+ ((Form_pg_class) GETSTRUCT(tuple))->relisivm = newstate;
+
+ CatalogTupleUpdate(pgrel, &tuple->t_self, tuple);
+
+ heap_freetuple(tuple);
+ table_close(pgrel, RowExclusiveLock);
+
+ /*
+ * Advance command counter to make the updated pg_class row locally
+ * visible.
+ */
+ CommandCounterIncrement();
+}
+
/*
* ExecRefreshMatView -- execute a REFRESH MATERIALIZED VIEW command
*
@@ -311,7 +360,7 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
/* Generate the data, if wanted. */
if (!stmt->skipData)
- processed = refresh_matview_datafill(dest, dataQuery, queryString);
+ processed = refresh_matview_datafill(dest, dataQuery, NULL, queryString);
/* Make the matview match the newly generated data. */
if (concurrent)
@@ -369,6 +418,7 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
*/
static uint64
refresh_matview_datafill(DestReceiver *dest, Query *query,
+ QueryEnvironment *queryEnv,
const char *queryString)
{
List *rewritten;
@@ -405,7 +455,7 @@ refresh_matview_datafill(DestReceiver *dest, Query *query,
/* Create a QueryDesc, redirecting output to our tuple receiver */
queryDesc = CreateQueryDesc(plan, queryString,
GetActiveSnapshot(), InvalidSnapshot,
- dest, NULL, NULL, 0);
+ dest, NULL, queryEnv ? queryEnv: NULL, 0);
/* call ExecutorStart to prepare the plan for execution */
ExecutorStart(queryDesc, 0);
@@ -926,3 +976,366 @@ CloseMatViewIncrementalMaintenance(void)
matview_maintenance_depth--;
Assert(matview_maintenance_depth >= 0);
}
+
+/*
+ * IVM trigger function
+ */
+
+Datum
+IVM_immediate_maintenance(PG_FUNCTION_ARGS)
+{
+ TriggerData *trigdata = (TriggerData *) fcinfo->context;
+ Relation rel;
+ Oid relid;
+ Oid matviewOid;
+ Query *query, *old_delta_qry, *new_delta_qry;
+ char* matviewname = trigdata->tg_trigger->tgargs[0];
+ List *names;
+ Relation matviewRel;
+ int old_depth = matview_maintenance_depth;
+
+ Oid tableSpace;
+ Oid relowner;
+ Oid OIDDelta_new = InvalidOid;
+ Oid OIDDelta_old = InvalidOid;
+ DestReceiver *dest_new = NULL, *dest_old = NULL;
+ char relpersistence;
+ Oid save_userid;
+ int save_sec_context;
+ int save_nestlevel;
+
+ ParseState *pstate;
+ QueryEnvironment *queryEnv = create_queryEnv();
+
+ /* Create a dummy ParseState for addRangeTableEntryForENR */
+ pstate = make_parsestate(NULL);
+ pstate->p_queryEnv = queryEnv;
+
+ names = stringToQualifiedNameList(matviewname);
+ matviewOid = RangeVarGetRelid(makeRangeVarFromNameList(names), AccessShareLock, true);
+ matviewRel = heap_open(matviewOid, NoLock);
+
+ /* Make sure it is a materialized view. */
+ if (matviewRel->rd_rel->relkind != RELKIND_MATVIEW)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("\"%s\" is not a materialized view",
+ RelationGetRelationName(matviewRel))));
+
+ rel = trigdata->tg_relation;
+ relid = rel->rd_id;
+
+ query = get_view_query(matviewRel);
+
+ new_delta_qry = copyObject(query);
+ old_delta_qry = copyObject(query);
+
+ if (trigdata->tg_newtable)
+ {
+ RangeTblEntry *rte;
+ ListCell *lc;
+
+ TargetEntry *tle;
+ Node *node;
+ FuncCall *fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+
+ EphemeralNamedRelation enr =
+ palloc(sizeof(EphemeralNamedRelationData));
+
+ enr->md.name = trigdata->tg_trigger->tgnewtable;
+ enr->md.reliddesc = trigdata->tg_relation->rd_id;
+ enr->md.tupdesc = NULL;
+ enr->md.enrtype = ENR_NAMED_TUPLESTORE;
+ enr->md.enrtuples = tuplestore_tuple_count(trigdata->tg_newtable);
+ enr->reldata = trigdata->tg_newtable;
+ register_ENR(queryEnv, enr);
+
+ rte = addRangeTableEntryForENR(pstate, makeRangeVar(NULL, enr->md.name, -1), true);
+ new_delta_qry->rtable = lappend(new_delta_qry->rtable, rte);
+
+ foreach(lc, new_delta_qry->rtable)
+ {
+ RangeTblEntry *r = (RangeTblEntry*) lfirst(lc);
+ if (r->relid == relid)
+ {
+ lfirst(lc) = rte;
+ break;
+ }
+ }
+
+ fn->agg_star = true;
+ new_delta_qry->groupClause = transformDistinctClause(NULL, &new_delta_qry->targetList, new_delta_qry->sortClause, false);
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(new_delta_qry->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ new_delta_qry->targetList = lappend(new_delta_qry->targetList, tle);
+ new_delta_qry->hasAggs = true;
+ }
+
+ if (trigdata->tg_oldtable)
+ {
+ RangeTblEntry *rte;
+ ListCell *lc;
+
+ TargetEntry *tle;
+ Node *node;
+ FuncCall *fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+
+ EphemeralNamedRelation enr =
+ palloc(sizeof(EphemeralNamedRelationData));
+
+ enr->md.name = trigdata->tg_trigger->tgoldtable;
+ enr->md.reliddesc = trigdata->tg_relation->rd_id;
+ enr->md.tupdesc = NULL;
+ enr->md.enrtype = ENR_NAMED_TUPLESTORE;
+ enr->md.enrtuples = tuplestore_tuple_count(trigdata->tg_oldtable);
+ enr->reldata = trigdata->tg_oldtable;
+ register_ENR(queryEnv, enr);
+
+ rte = addRangeTableEntryForENR(pstate, makeRangeVar(NULL, enr->md.name, -1), true);
+ old_delta_qry->rtable = lappend(old_delta_qry->rtable, rte);
+
+ foreach(lc, old_delta_qry->rtable)
+ {
+ RangeTblEntry *r = (RangeTblEntry*) lfirst(lc);
+ if (r->relid == relid)
+ {
+ lfirst(lc) = rte;
+ break;
+ }
+ }
+
+ fn->agg_star = true;
+ old_delta_qry->groupClause = transformDistinctClause(NULL, &old_delta_qry->targetList, old_delta_qry->sortClause, false);
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(old_delta_qry->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ old_delta_qry->targetList = lappend(old_delta_qry->targetList, tle);
+ old_delta_qry->hasAggs = true;
+ }
+
+
+ /*
+ * Check for active uses of the relation in the current transaction, such
+ * as open scans.
+ *
+ * NB: We count on this to protect us against problems with refreshing the
+ * data using TABLE_INSERT_FROZEN.
+ */
+ CheckTableNotInUse(matviewRel, "REFRESH MATERIALIZED VIEW");
+
+ relowner = matviewRel->rd_rel->relowner;
+
+ /*
+ * Switch to the owner's userid, so that any functions are run as that
+ * user. Also arrange to make GUC variable changes local to this command.
+ * Don't lock it down too tight to create a temporary table just yet. We
+ * will switch modes when we are about to execute user code.
+ */
+ GetUserIdAndSecContext(&save_userid, &save_sec_context);
+ SetUserIdAndSecContext(relowner,
+ save_sec_context | SECURITY_LOCAL_USERID_CHANGE);
+ save_nestlevel = NewGUCNestLevel();
+
+ tableSpace = GetDefaultTablespace(RELPERSISTENCE_TEMP, false);
+ relpersistence = RELPERSISTENCE_TEMP;
+
+ /*
+ * Create the transient table that will receive the regenerated data. Lock
+ * it against access by any other process until commit (by which time it
+ * will be gone).
+ */
+ if (trigdata->tg_newtable)
+ {
+ OIDDelta_new = make_new_heap(matviewOid, tableSpace, relpersistence,
+ ExclusiveLock);
+ LockRelationOid(OIDDelta_new, AccessExclusiveLock);
+ dest_new = CreateTransientRelDestReceiver(OIDDelta_new);
+ }
+ if (trigdata->tg_oldtable)
+ {
+ if (trigdata->tg_newtable)
+ OIDDelta_old = make_new_heap(OIDDelta_new, tableSpace, relpersistence,
+ ExclusiveLock);
+ else
+ OIDDelta_old = make_new_heap(matviewOid, tableSpace, relpersistence,
+ ExclusiveLock);
+ LockRelationOid(OIDDelta_old, AccessExclusiveLock);
+ dest_old = CreateTransientRelDestReceiver(OIDDelta_old);
+ }
+
+ /*
+ * Now lock down security-restricted operations.
+ */
+ SetUserIdAndSecContext(relowner,
+ save_sec_context | SECURITY_RESTRICTED_OPERATION);
+
+ /* Generate the data. */
+ if (trigdata->tg_newtable)
+ refresh_matview_datafill(dest_new, new_delta_qry, queryEnv, NULL);
+ if (trigdata->tg_oldtable)
+ refresh_matview_datafill(dest_old, old_delta_qry, queryEnv, NULL);
+
+ PG_TRY();
+ {
+ apply_delta(matviewOid, OIDDelta_new, OIDDelta_old, relowner,
+ save_sec_context);
+ }
+ PG_CATCH();
+ {
+ matview_maintenance_depth = old_depth;
+ PG_RE_THROW();
+ }
+ PG_END_TRY();
+
+ heap_close(matviewRel, NoLock);
+
+ /* Roll back any GUC changes */
+ AtEOXact_GUC(false, save_nestlevel);
+
+ /* Restore userid and security context */
+ SetUserIdAndSecContext(save_userid, save_sec_context);
+
+ return PointerGetDatum(NULL);
+}
+
+static void
+apply_delta(Oid matviewOid, Oid tempOid_new, Oid tempOid_old,
+ Oid relowner, int save_sec_context)
+{
+ StringInfoData querybuf;
+ StringInfoData mvatts_buf, diffatts_buf;
+ Relation matviewRel;
+ Relation tempRel_new = NULL, tempRel_old = NULL;
+ char *matviewname;
+ char *tempname_new = NULL, *tempname_old = NULL;
+ int i;
+
+
+ initStringInfo(&querybuf);
+ matviewRel = heap_open(matviewOid, NoLock);
+ matviewname = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(matviewRel)),
+ RelationGetRelationName(matviewRel));
+
+ if (OidIsValid(tempOid_new))
+ {
+ tempRel_new = heap_open(tempOid_new, NoLock);
+ tempname_new = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(tempRel_new)),
+ RelationGetRelationName(tempRel_new));
+ }
+ if (OidIsValid(tempOid_old))
+ {
+ tempRel_old = heap_open(tempOid_old, NoLock);
+ tempname_old = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(tempRel_old)),
+ RelationGetRelationName(tempRel_old));
+ }
+
+ initStringInfo(&mvatts_buf);
+ initStringInfo(&diffatts_buf);
+ for (i = 0; i < matviewRel->rd_att->natts; i++)
+ {
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, i);
+
+ if (!strcmp(NameStr(attr->attname), "__ivm_count__"))
+ continue;
+
+ if (i > 0)
+ {
+ appendStringInfo(&mvatts_buf, ", ");
+ appendStringInfo(&diffatts_buf, ", ");
+ }
+ appendStringInfo(&mvatts_buf, "mv.%s", NameStr(attr->attname));
+ appendStringInfo(&diffatts_buf, "diff.%s", NameStr(attr->attname));
+ }
+
+ /* Open SPI context. */
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+
+ /* Analyze the temp table with the new contents. */
+ if (tempname_new)
+ {
+ appendStringInfo(&querybuf, "ANALYZE %s", tempname_new);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ if (tempname_old)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf, "ANALYZE %s", tempname_old);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+
+ SetUserIdAndSecContext(relowner,
+ save_sec_context | SECURITY_LOCAL_USERID_CHANGE);
+
+ OpenMatViewIncrementalMaintenance();
+
+ if (tempname_old)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH t AS ("
+ " SELECT diff.__ivm_count__, (diff.__ivm_count__ = mv.__ivm_count__) AS for_dlt, mv.ctid"
+ " FROM %s AS mv, %s AS diff WHERE (%s) = (%s)"
+ "), updt AS ("
+ " UPDATE %s AS mv SET __ivm_count__ = mv.__ivm_count__ - t.__ivm_count__"
+ " FROM t WHERE mv.ctid = t.ctid AND NOT for_dlt"
+ ") DELETE FROM %s AS mv USING t WHERE mv.ctid = t.ctid AND for_dlt;",
+ matviewname, tempname_old, mvatts_buf.data, diffatts_buf.data, matviewname, matviewname);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_DELETE)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+
+ }
+ if (tempname_new)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH updt AS ("
+ " UPDATE %s AS mv SET __ivm_count__ = mv.__ivm_count__ + diff.__ivm_count__"
+ " FROM %s AS diff WHERE (%s) = (%s)"
+ " RETURNING %s"
+ ") INSERT INTO %s (SELECT * FROM %s AS diff WHERE (%s) NOT IN (SELECT * FROM updt));",
+ matviewname, tempname_new, mvatts_buf.data, diffatts_buf.data, diffatts_buf.data, matviewname, tempname_new, diffatts_buf.data);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+
+ }
+
+ /* We're done maintaining the materialized view. */
+ CloseMatViewIncrementalMaintenance();
+
+ if (OidIsValid(tempOid_new))
+ heap_close(tempRel_new, NoLock);
+ if (OidIsValid(tempOid_old))
+ heap_close(tempRel_old, NoLock);
+
+ heap_close(matviewRel, NoLock);
+
+ /* Clean up temp tables. */
+ if (OidIsValid(tempOid_new))
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf, "DROP TABLE %s", tempname_new);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ if (OidIsValid(tempOid_old))
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf, "DROP TABLE %s", tempname_old);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+
+ /* Close SPI context. */
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+}
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 780d7ab00b..7a43e57186 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2364,6 +2364,7 @@ _copyRangeTblEntry(const RangeTblEntry *from)
COPY_SCALAR_FIELD(relkind);
COPY_SCALAR_FIELD(rellockmode);
COPY_NODE_FIELD(tablesample);
+ COPY_SCALAR_FIELD(relisivm);
COPY_NODE_FIELD(subquery);
COPY_SCALAR_FIELD(security_barrier);
COPY_SCALAR_FIELD(jointype);
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 4f2ebe5118..608d477bd5 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2641,6 +2641,7 @@ _equalRangeTblEntry(const RangeTblEntry *a, const RangeTblEntry *b)
COMPARE_SCALAR_FIELD(relkind);
COMPARE_SCALAR_FIELD(rellockmode);
COMPARE_NODE_FIELD(tablesample);
+ COMPARE_SCALAR_FIELD(relisivm);
COMPARE_NODE_FIELD(subquery);
COMPARE_SCALAR_FIELD(security_barrier);
COMPARE_SCALAR_FIELD(jointype);
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 387e4b9b71..2321d3f685 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -3045,6 +3045,7 @@ _outRangeTblEntry(StringInfo str, const RangeTblEntry *node)
WRITE_CHAR_FIELD(relkind);
WRITE_INT_FIELD(rellockmode);
WRITE_NODE_FIELD(tablesample);
+ WRITE_BOOL_FIELD(relisivm);
break;
case RTE_SUBQUERY:
WRITE_NODE_FIELD(subquery);
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 3b96492b36..a7bb7dce81 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1366,6 +1366,7 @@ _readRangeTblEntry(void)
READ_CHAR_FIELD(relkind);
READ_INT_FIELD(rellockmode);
READ_NODE_FIELD(tablesample);
+ READ_BOOL_FIELD(relisivm);
break;
case RTE_SUBQUERY:
READ_NODE_FIELD(subquery);
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 3dc0e8a4fb..61704c5c90 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -418,6 +418,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -645,7 +646,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -4054,30 +4055,32 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->relkind = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->relkind = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -4094,9 +4097,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -15127,6 +15135,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDEX
| INDEXES
| INHERIT
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
index 0640d11fac..f65c507f49 100644
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -37,6 +37,7 @@
#include "utils/syscache.h"
#include "utils/varlena.h"
+#include "commands/matview.h"
#define MAX_FUZZY_DISTANCE 3
@@ -1238,6 +1239,7 @@ addRangeTableEntry(ParseState *pstate,
rte->relid = RelationGetRelid(rel);
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -1317,6 +1319,7 @@ addRangeTableEntryForRelation(ParseState *pstate,
rte->relid = RelationGetRelid(rel);
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -2605,6 +2608,9 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
{
Form_pg_attribute attr = TupleDescAttr(tupdesc, varattno);
+ if (!strcmp("__ivm_count__", NameStr(attr->attname)) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
if (attr->attisdropped)
{
if (include_dropped)
diff --git a/src/backend/rewrite/rewriteDefine.c b/src/backend/rewrite/rewriteDefine.c
index 6bd889461e..fbab6b41f7 100644
--- a/src/backend/rewrite/rewriteDefine.c
+++ b/src/backend/rewrite/rewriteDefine.c
@@ -765,7 +765,8 @@ checkRuleResultList(List *targetList, TupleDesc resultDesc, bool isSelect,
attr->atttypmod))));
}
- if (i != resultDesc->natts)
+ /* No check for materialized views since this could have special columns for IVM */
+ if ((!isSelect || requireColumnNameMatch) && i != resultDesc->natts)
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
isSelect ?
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index 39080776b0..38305003d5 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -41,6 +41,8 @@
#include "utils/lsyscache.h"
#include "utils/rel.h"
+#include "parser/parser.h"
+#include "commands/matview.h"
/* We use a list of these to detect recursion in RewriteQuery */
typedef struct rewrite_event
@@ -1597,6 +1599,50 @@ ApplyRetrieveRule(Query *parsetree,
if (rule->qual != NULL)
elog(ERROR, "cannot handle qualified ON SELECT rule");
+ if (RelationIsIVM(relation))
+ {
+ rule_action = copyObject(linitial(rule->actions));
+
+ if (!rule_action->distinctClause)
+ {
+ StringInfoData str;
+ RawStmt *raw;
+ Query *sub;
+
+ if (rule_action->hasDistinctOn)
+ elog(ERROR, "DISTINCT ON is not supported in IVM");
+
+ initStringInfo(&str);
+ appendStringInfo(&str, "SELECT mv.*, __ivm_count__ FROM %s mv, generate_series(1, mv.__ivm_count__)",
+ quote_qualified_identifier(get_namespace_name(RelationGetNamespace(relation)),
+ RelationGetRelationName(relation)));
+
+ raw = (RawStmt*)linitial(raw_parser(str.data));
+ sub = transformStmt(make_parsestate(NULL),raw->stmt);
+
+ rte = rt_fetch(rt_index, parsetree->rtable);
+
+ rte->rtekind = RTE_SUBQUERY;
+ rte->subquery = sub;
+ rte->security_barrier = RelationIsSecurityView(relation);
+ /* Clear fields that should not be set in a subquery RTE */
+ rte->relid = InvalidOid;
+ rte->relkind = 0;
+ rte->rellockmode = 0;
+ rte->tablesample = NULL;
+ rte->inh = false; /* must not be set for a subquery */
+
+ rte->requiredPerms = 0; /* no permission check on subquery itself */
+ rte->checkAsUser = InvalidOid;
+ rte->selectedCols = NULL;
+ rte->insertedCols = NULL;
+ rte->updatedCols = NULL;
+ rte->extraUpdatedCols = NULL;
+ }
+
+ return parsetree;
+ }
+
if (rt_index == parsetree->resultRelation)
{
/*
@@ -1906,7 +1952,8 @@ fireRIRrules(Query *parsetree, List *activeRIRs)
* In that case this test would need to be postponed till after we've
* opened the rel, so that we could check its state.
*/
- if (rte->relkind == RELKIND_MATVIEW)
+ if (rte->relkind == RELKIND_MATVIEW &&
+ (!rte->relisivm || MatViewIncrementalMaintenanceIsEnabled() || parsetree->commandType != CMD_SELECT))
continue;
/*
diff --git a/src/backend/utils/cache/lsyscache.c b/src/backend/utils/cache/lsyscache.c
index b4f2d0f35a..82a2fd99bd 100644
--- a/src/backend/utils/cache/lsyscache.c
+++ b/src/backend/utils/cache/lsyscache.c
@@ -1844,6 +1844,30 @@ get_rel_relispartition(Oid relid)
return false;
}
+/*
+ * get_rel_relisivm
+ *
+ * Returns the relisivm flag associated with a given relation.
+ */
+bool
+get_rel_relisivm(Oid relid)
+{
+ HeapTuple tp;
+
+ tp = SearchSysCache1(RELOID, ObjectIdGetDatum(relid));
+ if (HeapTupleIsValid(tp))
+ {
+ Form_pg_class reltup = (Form_pg_class) GETSTRUCT(tp);
+ bool result;
+
+ result = reltup->relisivm;
+ ReleaseSysCache(tp);
+ return result;
+ }
+ else
+ return false;
+}
+
/*
* get_rel_tablespace
*
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index eaf2b18820..b3586cb273 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -1861,6 +1861,8 @@ formrdesc(const char *relationName, Oid relationReltype,
/* ... and they're always populated, too */
relation->rd_rel->relispopulated = true;
+ /* ... and they're always no ivm, too */
+ relation->rd_rel->relisivm = false;
relation->rd_rel->relreplident = REPLICA_IDENTITY_NOTHING;
relation->rd_rel->relpages = 0;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index e4c03de221..7bead138d0 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -985,6 +985,7 @@ static const pgsql_thing_t words_after_create[] = {
{"FOREIGN TABLE", NULL, NULL, NULL},
{"FUNCTION", NULL, NULL, Query_for_list_of_functions},
{"GROUP", Query_for_list_of_roles},
+ {"INCREMENTAL MATERIALIZED VIEW", NULL, NULL, &Query_for_list_of_matviews},
{"INDEX", NULL, NULL, &Query_for_list_of_indexes},
{"LANGUAGE", Query_for_list_of_languages},
{"LARGE OBJECT", NULL, NULL, NULL, THING_NO_CREATE | THING_NO_DROP},
@@ -2467,7 +2468,7 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("SEQUENCE", "TABLE", "VIEW");
/* Complete "CREATE UNLOGGED" with TABLE or MATVIEW */
else if (TailMatches("CREATE", "UNLOGGED"))
- COMPLETE_WITH("TABLE", "MATERIALIZED VIEW");
+ COMPLETE_WITH("TABLE", "MATERIALIZED VIEW", "INCREMENTAL MATERIALIZED VIEW");
/* Complete PARTITION BY with RANGE ( or LIST ( or ... */
else if (TailMatches("PARTITION", "BY"))
COMPLETE_WITH("RANGE (", "LIST (", "HASH (");
@@ -2672,13 +2673,16 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("SELECT");
/* CREATE MATERIALIZED VIEW */
- else if (Matches("CREATE", "MATERIALIZED"))
+ else if (Matches("CREATE", "MATERIALIZED") ||
+ Matches("CREATE", "INCREMENTAL", "MATERIALIZED"))
COMPLETE_WITH("VIEW");
- /* Complete CREATE MATERIALIZED VIEW <name> with AS */
- else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny))
+ /* Complete CREATE MATERIALIZED VIEW <name> with AS */
+ else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny) ||
+ Matches("CREATE", "INCREMENTAL", "MATERIALIZED", "VIEW", MatchAny))
COMPLETE_WITH("AS");
/* Complete "CREATE MATERIALIZED VIEW <sth> AS with "SELECT" */
- else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny, "AS"))
+ else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny, "AS") ||
+ Matches("CREATE", "INCREMENTAL", "MATERIALIZED", "VIEW", MatchAny, "AS"))
COMPLETE_WITH("SELECT");
/* CREATE EVENT TRIGGER */
diff --git a/src/include/catalog/pg_class.dat b/src/include/catalog/pg_class.dat
index 9bcf28676d..7deda405af 100644
--- a/src/include/catalog/pg_class.dat
+++ b/src/include/catalog/pg_class.dat
@@ -27,7 +27,7 @@
relpersistence => 'p', relkind => 'r', relnatts => '31', relchecks => '0',
relhasrules => 'f', relhastriggers => 'f', relhassubclass => 'f',
relrowsecurity => 'f', relforcerowsecurity => 'f', relispopulated => 't',
- relreplident => 'n', relispartition => 'f', relfrozenxid => '3',
+ relreplident => 'n', relispartition => 'f', relisivm => 'f', relfrozenxid => '3',
relminmxid => '1', relacl => '_null_', reloptions => '_null_',
relpartbound => '_null_' },
{ oid => '1249',
@@ -37,7 +37,7 @@
relpersistence => 'p', relkind => 'r', relnatts => '25', relchecks => '0',
relhasrules => 'f', relhastriggers => 'f', relhassubclass => 'f',
relrowsecurity => 'f', relforcerowsecurity => 'f', relispopulated => 't',
- relreplident => 'n', relispartition => 'f', relfrozenxid => '3',
+ relreplident => 'n', relispartition => 'f', relisivm => 'f', relfrozenxid => '3',
relminmxid => '1', relacl => '_null_', reloptions => '_null_',
relpartbound => '_null_' },
{ oid => '1255',
@@ -47,17 +47,17 @@
relpersistence => 'p', relkind => 'r', relnatts => '29', relchecks => '0',
relhasrules => 'f', relhastriggers => 'f', relhassubclass => 'f',
relrowsecurity => 'f', relforcerowsecurity => 'f', relispopulated => 't',
- relreplident => 'n', relispartition => 'f', relfrozenxid => '3',
+ relreplident => 'n', relispartition => 'f', relisivm => 'f', relfrozenxid => '3',
relminmxid => '1', relacl => '_null_', reloptions => '_null_',
relpartbound => '_null_' },
{ oid => '1259',
relname => 'pg_class', reltype => 'pg_class', relam => 'heap',
relfilenode => '0', relpages => '0', reltuples => '0', relallvisible => '0',
reltoastrelid => '0', relhasindex => 'f', relisshared => 'f',
- relpersistence => 'p', relkind => 'r', relnatts => '33', relchecks => '0',
+ relpersistence => 'p', relkind => 'r', relnatts => '34', relchecks => '0',
relhasrules => 'f', relhastriggers => 'f', relhassubclass => 'f',
relrowsecurity => 'f', relforcerowsecurity => 'f', relispopulated => 't',
- relreplident => 'n', relispartition => 'f', relfrozenxid => '3',
+ relreplident => 'n', relispartition => 'f', relisivm => 'f', relfrozenxid => '3',
relminmxid => '1', relacl => '_null_', reloptions => '_null_',
relpartbound => '_null_' },
diff --git a/src/include/catalog/pg_class.h b/src/include/catalog/pg_class.h
index 090b6ba907..ff535f5504 100644
--- a/src/include/catalog/pg_class.h
+++ b/src/include/catalog/pg_class.h
@@ -116,6 +116,9 @@ CATALOG(pg_class,1259,RelationRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(83,Relat
/* is relation a partition? */
bool relispartition;
+ /* is relation a matview with ivm? */
+ bool relisivm;
+
/* heap for rewrite during DDL, link to original rel */
Oid relrewrite BKI_DEFAULT(0);
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 87335248a0..5de996a72c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10677,4 +10677,9 @@
proname => 'pg_partition_root', prorettype => 'regclass',
proargtypes => 'regclass', prosrc => 'pg_partition_root' },
+# IVM
+{ oid => '784', descr => 'ivm trigger',
+ proname => 'IVM_immediate_maintenance', provolatile => 'v', prorettype => 'trigger',
+ proargtypes => '', prosrc => 'IVM_immediate_maintenance' },
+
]
diff --git a/src/include/commands/matview.h b/src/include/commands/matview.h
index d4461dd802..779037e3c3 100644
--- a/src/include/commands/matview.h
+++ b/src/include/commands/matview.h
@@ -23,6 +23,8 @@
extern void SetMatViewPopulatedState(Relation relation, bool newstate);
+extern void SetMatViewIVMState(Relation relation, bool newstate);
+
extern ObjectAddress ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
ParamListInfo params, char *completionTag);
@@ -30,4 +32,6 @@ extern DestReceiver *CreateTransientRelDestReceiver(Oid oid);
extern bool MatViewIncrementalMaintenanceIsEnabled(void);
+extern Datum IVM_immediate_maintenance(PG_FUNCTION_ARGS);
+
#endif /* MATVIEW_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 12e9730dd0..745d048d37 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1002,6 +1002,7 @@ typedef struct RangeTblEntry
char relkind; /* relation kind (see pg_class.relkind) */
int rellockmode; /* lock level that query requires on the rel */
struct TableSampleClause *tablesample; /* sampling info, or NULL */
+ bool relisivm;
/*
* Fields valid for a subquery RTE (else NULL):
@@ -2059,6 +2060,7 @@ typedef struct CreateStmt
char *tablespacename; /* table space to use, or NULL */
char *accessMethod; /* table access method */
bool if_not_exists; /* just do nothing if it already exists? */
+ bool ivm; /* incremental view maintenance is used by materialized view */
} CreateStmt;
/* ----------
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index f9b1cf2df7..105c2af4ce 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -117,6 +117,7 @@ typedef struct IntoClause
char *tableSpaceName; /* table space to use, or NULL */
Node *viewQuery; /* materialized view's SELECT query */
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 00ace8425e..d682ee11cc 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -198,6 +198,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD)
PG_KEYWORD("inherit", INHERIT, UNRESERVED_KEYWORD)
diff --git a/src/include/utils/lsyscache.h b/src/include/utils/lsyscache.h
index 9606d021b1..c66cc81141 100644
--- a/src/include/utils/lsyscache.h
+++ b/src/include/utils/lsyscache.h
@@ -129,6 +129,7 @@ extern Oid get_rel_namespace(Oid relid);
extern Oid get_rel_type_id(Oid relid);
extern char get_rel_relkind(Oid relid);
extern bool get_rel_relispartition(Oid relid);
+extern bool get_rel_relisivm(Oid relid);
extern Oid get_rel_tablespace(Oid relid);
extern char get_rel_persistence(Oid relid);
extern Oid get_transform_fromsql(Oid typid, Oid langid, List *trftypes);
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index d7f33abce3..057f38d5de 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -562,6 +562,8 @@ typedef struct ViewOptions
*/
#define RelationIsPopulated(relation) ((relation)->rd_rel->relispopulated)
+#define RelationIsIVM(relation) ((relation)->rd_rel->relisivm)
+
/*
* RelationIsAccessibleInLogicalDecoding
* True if we need to log enough information to have access via
Hi hackers,
Thank you for your many questions and feedbacks at PGCon 2019.
Attached is the patch rebased for the current master branch.
Regards,
Yugo Nagata
On Tue, 14 May 2019 15:46:48 +0900
Yugo Nagata <nagata@sraoss.co.jp> wrote:
On Mon, 1 Apr 2019 12:11:22 +0900
Yugo Nagata <nagata@sraoss.co.jp> wrote:On Thu, 27 Dec 2018 21:57:26 +0900
Yugo Nagata <nagata@sraoss.co.jp> wrote:Hi,
I would like to implement Incremental View Maintenance (IVM) on PostgreSQL.
I am now working on an initial patch for implementing IVM on PostgreSQL.
This enables materialized views to be updated incrementally after one
of their base tables is modified.Attached is a WIP patch of Incremental View Maintenance (IVM).
Major part is written by me, and changes in syntax and pg_class
are Hoshiai-san's work.Although this is sill a draft patch in work-in-progress, any
suggestions or thoughts would be appreciated.* What it is
This allows a kind of Immediate Maintenance of materialized views. if a
materialized view is created by CRATE INCREMENTAL MATERIALIZED VIEW command,
the contents of the mateview is updated automatically and incrementally
after base tables are updated. Noted this syntax is just tentative, so it
may be changed.====== Example 1 ======
postgres=# CREATE INCREMENTAL MATERIALIZED VIEW m AS SELECT * FROM t0;
SELECT 3
postgres=# SELECT * FROM m;
i
---
3
2
1
(3 rows)postgres=# INSERT INTO t0 VALUES (4);
INSERT 0 1
postgres=# SELECt * FROM m; -- automatically updated
i
---
3
2
1
4
(4 rows)
=============================This implementation also supports matviews including duplicate tuples or
DISTINCT clause in its view definition query. For example, even if a matview
is defined with DISTINCT to remove duplication of tuples in a base table, this
can perform incremental update of the matview properly. That is, the contents
of the matview doesn't change when exiting tuples are inserted into the base
tables, and a tuple in the matview is deleted only when duplicity of the
corresponding tuple in the base table becomes zero.This is due to "colunting alogorithm" in which the number of each tuple is
stored in matviews as a special column value.====== Example 2 ======
postgres=# SELECT * FROM t1;
id | t
----+---
1 | A
2 | B
3 | C
4 | A
(4 rows)postgres=# CREATE INCREMENTAL MATERIALIZED VIEW m1 AS SELECT t FROM t1;
SELECT 3
postgres=# CREATE INCREMENTAL MATERIALIZED VIEW m2 AS SELECT DISTINCT t FROM t1;
SELECT 3
postgres=# SELECT * FROM m1; -- with duplicity
t
---
A
A
C
B
(4 rows)postgres=# SELECT * FROM m2;
t
---
A
B
C
(3 rows)postgres=# INSERT INTO t1 VALUES (5, 'B');
INSERT 0 1
postgres=# DELETE FROM t1 WHERE id IN (1,3); -- delete (1,A),(3,C)
DELETE 2
postgres=# SELECT * FROM m1; -- one A left and one more B
t
---
B
B
A
(3 rows)postgres=# SELECT * FROM m2; -- only C is removed
t
---
B
A
(2 rows)
=============================* How it works
1. Creating matview
When a matview is created, AFTER triggers are internally created
on its base tables. When the base tables is modified (INSERT, DELETE,
UPDATE), the matview is updated incrementally in the trigger function.When populating the matview, GROUP BY and count(*) are added to the
view definition query before this is executed for counting duplicity
of tuples in the matview. The result of count is stored in the matview
as a special column named "__ivm_count__".2. Maintenance of matview
When base tables are modified, the change set of the table can be
referred as Ephemeral Named Relations (ENRs) thanks to Transition Table
(a feature of trigger implemented since PG10). We can calculate the diff
set of the matview by replacing the base table in the view definition
query with the ENR (at least if it is Selection-Projection -Join view).
As well as view definition time, GROUP BY and count(*) is added in order
to count the duplicity of tuples in the diff set. As a result, two diff
sets (to be deleted from and to be inserted into the matview) are
calculated, and the results are stored into temporary tables respectively.The matiview is updated by merging these change sets. Instead of executing
DELETE or INSERT simply, the values of __ivm_count__ column in the matview
is decreased or increased. When the values becomes zero, the corresponding
tuple is deleted from the matview.3. Access to matview
When SELECT is issued for IVM matviews defined with DISTINCT, all columns
except to __ivm_count__ of each tuple in the matview are returned. This is
correct because duplicity of tuples are eliminated by GROUP BY.When DISTINCT is not used, SELECT for the IVM matviews returns each tuple
__ivm_count__ times. Currently, this is implemented by rewriting the SELECT
query to replace the matview RTE with a subquery which joins the matview
and generate_series function as bellow.SELECT mv.* FROM mv, generate_series(1, mv.__ivm_count__);
__ivm_count__ column is invisible for users when "SELECT * FROM ..." is
issued, but users can see the value by specifying in target list explicitly.====== Example 3 ======
postgres=# \d+ m1
Materialized view "public.m1"
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
---------------+--------+-----------+----------+---------+----------+--------------+-------------
t | text | | | | extended | |
__ivm_count__ | bigint | | | | plain | |
View definition:
SELECT t1.t
FROM t1;
Access method: heappostgres=# \d+ m2
Materialized view "public.m2"
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
---------------+--------+-----------+----------+---------+----------+--------------+-------------
t | text | | | | extended | |
__ivm_count__ | bigint | | | | plain | |
View definition:
SELECT DISTINCT t1.t
FROM t1;
Access method: heappostgres=# SELECT *, __ivm_count__ FROM m1;
t | __ivm_count__
---+---------------
B | 2
B | 2
A | 1
(3 rows)postgres=# SELECT *, __ivm_count__ FROM m2;
t | __ivm_count__
---+---------------
B | 2
A | 1
(2 rows)postgres=# EXPLAIN SELECT * FROM m1;
QUERY PLAN
------------------------------------------------------------------------------
Nested Loop (cost=0.00..61.03 rows=3000 width=2)
-> Seq Scan on m1 mv (cost=0.00..1.03 rows=3 width=10)
-> Function Scan on generate_series (cost=0.00..10.00 rows=1000 width=0)
(3 rows)
=============================* Simple Performance Evaluation
I confirmed that "incremental" update of matviews is more effective
than the standard REFRESH by using simple exapmle. I used tables
of pgbench (SF=100) here.Create two matviews, that is, without and with IVM.
test=# CREATE MATERIALIZED VIEW bench1 AS
SELECT aid, bid, abalance, bbalance
FROM pgbench_accounts JOIN pgbench_branches USING (bid)
WHERE abalance > 0 OR bbalance > 0;
SELECT 5001054
test=# CREATE INCREMENTAL MATERIALIZED VIEW bench2 AS
SELECT aid, bid, abalance, bbalance
FROM pgbench_accounts JOIN pgbench_branches USING (bid)
WHERE abalance > 0 OR bbalance > 0;
SELECT 5001054The standard REFRESH of bench1 took more than 10 seconds.
test=# \timing
Timing is on.
test=# REFRESH MATERIALIZED VIEW bench1 ;
REFRESH MATERIALIZED VIEW
Time: 11210.563 ms (00:11.211)Create an index on the IVM matview (bench2).
test=# CREATE INDEX on bench2(aid,bid);
CREATE INDEXUpdating a tuple in pgbench_accounts took 18ms. After this, bench2
was updated automatically and correctly.test=# SELECT * FROM bench2 WHERE aid = 1;
aid | bid | abalance | bbalance
-----+-----+----------+----------
1 | 1 | 10 | 10
(1 row)Time: 2.498 ms
test=# UPDATE pgbench_accounts SET abalance = 1000 WHERE aid = 1;
UPDATE 1
Time: 18.634 ms
test=# SELECT * FROM bench2 WHERE aid = 1;
aid | bid | abalance | bbalance
-----+-----+----------+----------
1 | 1 | 1000 | 10
(1 row)However, if there is not the index on bench2, it took 4 sec, so
appropriate indexes are needed on IVM matviews.test=# DROP INDEX bench2_aid_bid_idx ;
DROP INDEX
Time: 10.613 ms
test=# UPDATE pgbench_accounts SET abalance = 2000 WHERE aid = 1;
UPDATE 1
Time: 3931.274 ms (00:03.931)* Restrictions on view definition
This patch is still in Work-in-Progress and there are many restrictions
on the view definition query of matviews.The current implementation supports views including selection, projection,
and inner join with or without DISTINCT. Aggregation and GROUP BY are not
supported yet, but I plan to deal with these by the first release.
Self-join, subqueries, OUTER JOIN, CTE, window functions are not
considered well, either. I need more investigation on these type of views
although I found some papers explaining how to handle sub-queries and
outer-joins.These unsupported views should be checked when a matview is created, but
this is not implemented yet. Hoshiai-san are working on this.* Timing of view maintenance
This patch implements a kind of Immediate Maintenance, that is, a matview
is updated immediately when a base table is modified. On other hand, in
"Deferred Maintenance", matviews are updated after the transaction, for
example, by the user command like REFRESH.For implementing "deferred", it is need to implement a mechanism to maintain
logs for recording changes of base tables and an algorithm to compute the
delta to be applied to matviews.In addition, there could be another implementation of Immediate Maintenance
in which matview is updated at the end of a transaction that modified base
table, rather than in AFTER trigger. Oracle supports this type of IVM. To
implement this, we will need a mechanism to maintain change logs on base
tables as well as Deferred maintenance.* Counting algorithm implementation
There will be also discussions on counting-algorithm implementation.
Firstly, the current patch treats "__ivm_count__" as a special column name
in a somewhat ad hoc way. This is used when maintaining and accessing matviews,
and when "SELECT * FROM ..." is issued, __ivm_count__ column is invisible for
users. Maybe this name has to be inhibited in user tables. Is it acceptable
to use such columns for IVM, and is there better way, if not?Secondly, a matview with duplicate tuples is replaces with a subquery which
uses generate_series function. It does not have to be generate_series, and we
can make a new set returning function for this. Anyway, this internal behaviour
is visible in EXPLAIN results as shown in Example 3. Also, there is a
performance impact because estimated rows number is wrong, and what is worse,
the cost of join is not small when the size of matview is large. Therefore, we
might have to add a new plan node for selecting from matviews rather than using
such a special set returning function.Ragards,
--
Yugo Nagata <nagata@sraoss.co.jp>
--
Yugo Nagata <nagata@sraoss.co.jp>
Attachments:
WIP_immediate_IVM_v2.patchtext/x-diff; name=WIP_immediate_IVM_v2.patchDownload
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 1300c7bbaa..ddac6032ba 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -1949,6 +1949,13 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<entry>True if table or index is a partition</entry>
</row>
+ <row>
+ <entry><structfield>relisivm</structfield></entry>
+ <entry><type>bool</type></entry>
+ <entry></entry>
+ <entry>True if materialized view enables incremental view maintenance</entry>
+ </row>
+
<row>
<entry><structfield>relrewrite</structfield></entry>
<entry><type>oid</type></entry>
diff --git a/doc/src/sgml/ref/create_materialized_view.sgml b/doc/src/sgml/ref/create_materialized_view.sgml
index ec8847ed40..a23366a342 100644
--- a/doc/src/sgml/ref/create_materialized_view.sgml
+++ b/doc/src/sgml/ref/create_materialized_view.sgml
@@ -21,7 +21,7 @@ PostgreSQL documentation
<refsynopsisdiv>
<synopsis>
-CREATE MATERIALIZED VIEW [ IF NOT EXISTS ] <replaceable>table_name</replaceable>
+CREATE [ INCREMENTAL ] MATERIALIZED VIEW [ IF NOT EXISTS ] <replaceable>table_name</replaceable>
[ (<replaceable>column_name</replaceable> [, ...] ) ]
[ USING <replaceable class="parameter">method</replaceable> ]
[ WITH ( <replaceable class="parameter">storage_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] ) ]
@@ -54,6 +54,16 @@ CREATE MATERIALIZED VIEW [ IF NOT EXISTS ] <replaceable>table_name</replaceable>
<title>Parameters</title>
<variablelist>
+ <varlistentry>
+ <term><literal>INCREMENTAL</literal></term>
+ <listitem>
+ <para>
+ If specified, a materialized view enables incremental view maintenance.
+ You can replace only the contents of a materialized view, which based rows are changed.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>IF NOT EXISTS</literal></term>
<listitem>
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index 86820eecfc..f0f0e3bb84 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -891,6 +891,7 @@ InsertPgClassTuple(Relation pg_class_desc,
values[Anum_pg_class_relrewrite - 1] = ObjectIdGetDatum(rd_rel->relrewrite);
values[Anum_pg_class_relfrozenxid - 1] = TransactionIdGetDatum(rd_rel->relfrozenxid);
values[Anum_pg_class_relminmxid - 1] = MultiXactIdGetDatum(rd_rel->relminmxid);
+ values[Anum_pg_class_relisivm - 1] = BoolGetDatum(rd_rel->relisivm);
if (relacl != (Datum) 0)
values[Anum_pg_class_relacl - 1] = relacl;
else
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 587b717242..d26cdfc057 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -910,6 +910,7 @@ index_create(Relation heapRelation,
indexRelation->rd_rel->relowner = heapRelation->rd_rel->relowner;
indexRelation->rd_rel->relam = accessMethodObjectId;
indexRelation->rd_rel->relispartition = OidIsValid(parentIndexRelid);
+ indexRelation->rd_rel->relisivm = false;
/*
* store index's pg_class entry
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index 4c1d909d38..34678666a3 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -51,6 +51,14 @@
#include "utils/rls.h"
#include "utils/snapmgr.h"
+#include "catalog/pg_trigger.h"
+#include "catalog/pg_type.h"
+#include "commands/trigger.h"
+#include "parser/parser.h"
+#include "parser/parsetree.h"
+#include "parser/parse_func.h"
+#include "nodes/print.h"
+
typedef struct
{
@@ -74,6 +82,8 @@ static bool intorel_receive(TupleTableSlot *slot, DestReceiver *self);
static void intorel_shutdown(DestReceiver *self);
static void intorel_destroy(DestReceiver *self);
+static void CreateIvmTrigger(Oid relOid, Oid viewOid, char *matviewname, int16 type);
+static void CreateIvmTriggersOnBaseTables(Query *qry, Node *jtnode, Oid matviewOid, char* matviewname);
/*
* create_ctas_internal
@@ -109,6 +119,8 @@ create_ctas_internal(List *attrList, IntoClause *into)
create->oncommit = into->onCommit;
create->tablespacename = into->tableSpaceName;
create->if_not_exists = false;
+ /* Using Materialized view only */
+ create->ivm = into->ivm;
create->accessMethod = into->accessMethod;
/*
@@ -239,6 +251,7 @@ ExecCreateTableAs(CreateTableAsStmt *stmt, const char *queryString,
List *rewritten;
PlannedStmt *plan;
QueryDesc *queryDesc;
+ Query *copied_query;
if (stmt->if_not_exists)
{
@@ -319,7 +332,29 @@ ExecCreateTableAs(CreateTableAsStmt *stmt, const char *queryString,
* and is executed repeatedly. (See also the same hack in EXPLAIN and
* PREPARE.)
*/
- rewritten = QueryRewrite(copyObject(query));
+
+ copied_query = copyObject(query);
+ if (is_matview && into->ivm)
+ {
+ TargetEntry *tle;
+ Node *node;
+ ParseState *pstate = make_parsestate(NULL);
+
+ FuncCall *fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+ fn->agg_star = true;
+
+ copied_query->groupClause = transformDistinctClause(NULL, &copied_query->targetList, copied_query->sortClause, false);
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(copied_query->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ copied_query->targetList = lappend(copied_query->targetList, tle);
+ copied_query->hasAggs = true;
+ }
+
+ rewritten = QueryRewrite(copied_query);
/* SELECT should never rewrite to more or less than one SELECT query */
if (list_length(rewritten) != 1)
@@ -378,11 +413,65 @@ ExecCreateTableAs(CreateTableAsStmt *stmt, const char *queryString,
/* Restore userid and security context */
SetUserIdAndSecContext(save_userid, save_sec_context);
+
+
+ if (into->ivm)
+ {
+ char *matviewname;
+ Oid matviewOid = address.objectId;
+ Relation matviewRel = table_open(matviewOid, NoLock);
+ matviewname = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(matviewRel)),
+ RelationGetRelationName(matviewRel));
+ copied_query = copyObject(query);
+ AcquireRewriteLocks(copied_query, true, false);
+
+ CreateIvmTriggersOnBaseTables(copied_query, (Node *)copied_query->jointree, matviewOid, matviewname);
+
+ table_close(matviewRel, NoLock);
+ }
}
return address;
}
+static void CreateIvmTriggersOnBaseTables(Query *qry, Node *jtnode, Oid matviewOid, char* matviewname)
+{
+
+ if (jtnode == NULL)
+ return;
+ if (IsA(jtnode, RangeTblRef))
+ {
+ int rti = ((RangeTblRef *) jtnode)->rtindex;
+ RangeTblEntry *rte = rt_fetch(rti, qry->rtable);
+
+ if (rte->rtekind == RTE_RELATION)
+ {
+ CreateIvmTrigger(rte->relid, matviewOid, matviewname, TRIGGER_TYPE_INSERT);
+ CreateIvmTrigger(rte->relid, matviewOid, matviewname, TRIGGER_TYPE_DELETE);
+ CreateIvmTrigger(rte->relid, matviewOid, matviewname, TRIGGER_TYPE_UPDATE);
+ }
+ else
+ elog(ERROR, "unsupported RTE kind: %d", (int) rte->rtekind);
+ }
+ else if (IsA(jtnode, FromExpr))
+ {
+ FromExpr *f = (FromExpr *) jtnode;
+ ListCell *l;
+
+ foreach(l, f->fromlist)
+ CreateIvmTriggersOnBaseTables(qry, lfirst(l), matviewOid, matviewname);
+ }
+ else if (IsA(jtnode, JoinExpr))
+ {
+ JoinExpr *j = (JoinExpr *) jtnode;
+
+ CreateIvmTriggersOnBaseTables(qry, j->larg, matviewOid, matviewname);
+ CreateIvmTriggersOnBaseTables(qry, j->rarg, matviewOid, matviewname);
+ }
+ else
+ elog(ERROR, "unrecognized node type: %d", (int) nodeTag(jtnode));
+}
+
/*
* GetIntoRelEFlags --- compute executor flags needed for CREATE TABLE AS
*
@@ -547,6 +636,11 @@ intorel_startup(DestReceiver *self, int operation, TupleDesc typeinfo)
if (is_matview && !into->skipData)
SetMatViewPopulatedState(intoRelationDesc, true);
+ /*
+ * Mark relisivm field, if it's a matview and into->ivm is true.
+ */
+ if (is_matview && into->ivm)
+ SetMatViewIVMState(intoRelationDesc, true);
/*
* Fill private fields of myState for use by later routines
*/
@@ -619,3 +713,74 @@ intorel_destroy(DestReceiver *self)
{
pfree(self);
}
+
+static void
+CreateIvmTrigger(Oid relOid, Oid viewOid, char *matviewname, int16 type)
+{
+ CreateTrigStmt *ivm_trigger;
+ List *transitionRels = NIL;
+ ObjectAddress address, refaddr;
+
+ refaddr.classId = RelationRelationId;
+ refaddr.objectId = viewOid;
+ refaddr.objectSubId = 0;
+
+
+ ivm_trigger = makeNode(CreateTrigStmt);
+ ivm_trigger->relation = NULL;
+ ivm_trigger->row = false;
+ ivm_trigger->timing = TRIGGER_TYPE_AFTER;
+
+ ivm_trigger->events = type;
+
+ switch (type)
+ {
+ case TRIGGER_TYPE_INSERT:
+ ivm_trigger->trigname = "IVM_trigger_ins";
+ break;
+ case TRIGGER_TYPE_DELETE:
+ ivm_trigger->trigname = "IVM_trigger_del";
+ break;
+ case TRIGGER_TYPE_UPDATE:
+ ivm_trigger->trigname = "IVM_trigger_upd";
+ break;
+ }
+
+ if (type == TRIGGER_TYPE_INSERT || type == TRIGGER_TYPE_UPDATE)
+ {
+ TriggerTransition *n = makeNode(TriggerTransition);
+ n->name = "ivm_newtable";
+ n->isNew = true;
+ n->isTable = true;
+
+ transitionRels = lappend(transitionRels, n);
+ }
+ if (type == TRIGGER_TYPE_DELETE || type == TRIGGER_TYPE_UPDATE)
+ {
+ TriggerTransition *n = makeNode(TriggerTransition);
+ n->name = "ivm_oldtable";
+ n->isNew = false;
+ n->isTable = true;
+
+ transitionRels = lappend(transitionRels, n);
+ }
+
+ ivm_trigger->funcname = SystemFuncName("IVM_immediate_maintenance");
+
+ ivm_trigger->columns = NIL;
+ ivm_trigger->transitionRels = transitionRels;
+ ivm_trigger->whenClause = NULL;
+ ivm_trigger->isconstraint = false;
+ ivm_trigger->deferrable = false;
+ ivm_trigger->initdeferred = false;
+ ivm_trigger->constrrel = NULL;
+ ivm_trigger->args = list_make1(makeString(matviewname));
+
+ address = CreateTrigger(ivm_trigger, NULL, relOid, InvalidOid, InvalidOid,
+ InvalidOid, InvalidOid, InvalidOid, NULL, true, false);
+
+ recordDependencyOn(&address, &refaddr, DEPENDENCY_AUTO);
+
+ /* Make changes-so-far visible */
+ CommandCounterIncrement();
+}
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index 537d0e8cef..05ab22715d 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -46,6 +46,11 @@
#include "utils/snapmgr.h"
#include "utils/syscache.h"
+#include "utils/regproc.h"
+#include "nodes/makefuncs.h"
+#include "parser/parse_clause.h"
+#include "parser/parse_func.h"
+
typedef struct
{
@@ -65,7 +70,8 @@ static bool transientrel_receive(TupleTableSlot *slot, DestReceiver *self);
static void transientrel_shutdown(DestReceiver *self);
static void transientrel_destroy(DestReceiver *self);
static uint64 refresh_matview_datafill(DestReceiver *dest, Query *query,
- const char *queryString);
+ QueryEnvironment *queryEnv,
+ const char *queryString);
static char *make_temptable_name_n(char *tempname, int n);
static void refresh_by_match_merge(Oid matviewOid, Oid tempOid, Oid relowner,
int save_sec_context);
@@ -74,6 +80,9 @@ static bool is_usable_unique_index(Relation indexRel);
static void OpenMatViewIncrementalMaintenance(void);
static void CloseMatViewIncrementalMaintenance(void);
+static void apply_delta(Oid matviewOid, Oid tempOid_new, Oid tempOid_old,
+ Oid relowner, int save_sec_context);
+
/*
* SetMatViewPopulatedState
* Mark a materialized view as populated, or not.
@@ -114,6 +123,46 @@ SetMatViewPopulatedState(Relation relation, bool newstate)
CommandCounterIncrement();
}
+/*
+ * SetMatViewIVMState
+ * Mark a materialized view as IVM, or not.
+ *
+ * NOTE: caller must be holding an appropriate lock on the relation.
+ */
+void
+SetMatViewIVMState(Relation relation, bool newstate)
+{
+ Relation pgrel;
+ HeapTuple tuple;
+
+ Assert(relation->rd_rel->relkind == RELKIND_MATVIEW);
+
+ /*
+ * Update relation's pg_class entry. Crucial side-effect: other backends
+ * (and this one too!) are sent SI message to make them rebuild relcache
+ * entries.
+ */
+ pgrel = table_open(RelationRelationId, RowExclusiveLock);
+ tuple = SearchSysCacheCopy1(RELOID,
+ ObjectIdGetDatum(RelationGetRelid(relation)));
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for relation %u",
+ RelationGetRelid(relation));
+
+ ((Form_pg_class) GETSTRUCT(tuple))->relisivm = newstate;
+
+ CatalogTupleUpdate(pgrel, &tuple->t_self, tuple);
+
+ heap_freetuple(tuple);
+ table_close(pgrel, RowExclusiveLock);
+
+ /*
+ * Advance command counter to make the updated pg_class row locally
+ * visible.
+ */
+ CommandCounterIncrement();
+}
+
/*
* ExecRefreshMatView -- execute a REFRESH MATERIALIZED VIEW command
*
@@ -311,7 +360,7 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
/* Generate the data, if wanted. */
if (!stmt->skipData)
- processed = refresh_matview_datafill(dest, dataQuery, queryString);
+ processed = refresh_matview_datafill(dest, dataQuery, NULL, queryString);
/* Make the matview match the newly generated data. */
if (concurrent)
@@ -369,6 +418,7 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
*/
static uint64
refresh_matview_datafill(DestReceiver *dest, Query *query,
+ QueryEnvironment *queryEnv,
const char *queryString)
{
List *rewritten;
@@ -405,7 +455,7 @@ refresh_matview_datafill(DestReceiver *dest, Query *query,
/* Create a QueryDesc, redirecting output to our tuple receiver */
queryDesc = CreateQueryDesc(plan, queryString,
GetActiveSnapshot(), InvalidSnapshot,
- dest, NULL, NULL, 0);
+ dest, NULL, queryEnv ? queryEnv: NULL, 0);
/* call ExecutorStart to prepare the plan for execution */
ExecutorStart(queryDesc, 0);
@@ -926,3 +976,366 @@ CloseMatViewIncrementalMaintenance(void)
matview_maintenance_depth--;
Assert(matview_maintenance_depth >= 0);
}
+
+/*
+ * IVM trigger function
+ */
+
+Datum
+IVM_immediate_maintenance(PG_FUNCTION_ARGS)
+{
+ TriggerData *trigdata = (TriggerData *) fcinfo->context;
+ Relation rel;
+ Oid relid;
+ Oid matviewOid;
+ Query *query, *old_delta_qry, *new_delta_qry;
+ char* matviewname = trigdata->tg_trigger->tgargs[0];
+ List *names;
+ Relation matviewRel;
+ int old_depth = matview_maintenance_depth;
+
+ Oid tableSpace;
+ Oid relowner;
+ Oid OIDDelta_new = InvalidOid;
+ Oid OIDDelta_old = InvalidOid;
+ DestReceiver *dest_new = NULL, *dest_old = NULL;
+ char relpersistence;
+ Oid save_userid;
+ int save_sec_context;
+ int save_nestlevel;
+
+ ParseState *pstate;
+ QueryEnvironment *queryEnv = create_queryEnv();
+
+ /* Create a dummy ParseState for addRangeTableEntryForENR */
+ pstate = make_parsestate(NULL);
+ pstate->p_queryEnv = queryEnv;
+
+ names = stringToQualifiedNameList(matviewname);
+ matviewOid = RangeVarGetRelid(makeRangeVarFromNameList(names), AccessShareLock, true);
+ matviewRel = table_open(matviewOid, NoLock);
+
+ /* Make sure it is a materialized view. */
+ if (matviewRel->rd_rel->relkind != RELKIND_MATVIEW)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("\"%s\" is not a materialized view",
+ RelationGetRelationName(matviewRel))));
+
+ rel = trigdata->tg_relation;
+ relid = rel->rd_id;
+
+ query = get_view_query(matviewRel);
+
+ new_delta_qry = copyObject(query);
+ old_delta_qry = copyObject(query);
+
+ if (trigdata->tg_newtable)
+ {
+ RangeTblEntry *rte;
+ ListCell *lc;
+
+ TargetEntry *tle;
+ Node *node;
+ FuncCall *fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+
+ EphemeralNamedRelation enr =
+ palloc(sizeof(EphemeralNamedRelationData));
+
+ enr->md.name = trigdata->tg_trigger->tgnewtable;
+ enr->md.reliddesc = trigdata->tg_relation->rd_id;
+ enr->md.tupdesc = NULL;
+ enr->md.enrtype = ENR_NAMED_TUPLESTORE;
+ enr->md.enrtuples = tuplestore_tuple_count(trigdata->tg_newtable);
+ enr->reldata = trigdata->tg_newtable;
+ register_ENR(queryEnv, enr);
+
+ rte = addRangeTableEntryForENR(pstate, makeRangeVar(NULL, enr->md.name, -1), true);
+ new_delta_qry->rtable = lappend(new_delta_qry->rtable, rte);
+
+ foreach(lc, new_delta_qry->rtable)
+ {
+ RangeTblEntry *r = (RangeTblEntry*) lfirst(lc);
+ if (r->relid == relid)
+ {
+ lfirst(lc) = rte;
+ break;
+ }
+ }
+
+ fn->agg_star = true;
+ new_delta_qry->groupClause = transformDistinctClause(NULL, &new_delta_qry->targetList, new_delta_qry->sortClause, false);
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(new_delta_qry->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ new_delta_qry->targetList = lappend(new_delta_qry->targetList, tle);
+ new_delta_qry->hasAggs = true;
+ }
+
+ if (trigdata->tg_oldtable)
+ {
+ RangeTblEntry *rte;
+ ListCell *lc;
+
+ TargetEntry *tle;
+ Node *node;
+ FuncCall *fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+
+ EphemeralNamedRelation enr =
+ palloc(sizeof(EphemeralNamedRelationData));
+
+ enr->md.name = trigdata->tg_trigger->tgoldtable;
+ enr->md.reliddesc = trigdata->tg_relation->rd_id;
+ enr->md.tupdesc = NULL;
+ enr->md.enrtype = ENR_NAMED_TUPLESTORE;
+ enr->md.enrtuples = tuplestore_tuple_count(trigdata->tg_oldtable);
+ enr->reldata = trigdata->tg_oldtable;
+ register_ENR(queryEnv, enr);
+
+ rte = addRangeTableEntryForENR(pstate, makeRangeVar(NULL, enr->md.name, -1), true);
+ old_delta_qry->rtable = lappend(old_delta_qry->rtable, rte);
+
+ foreach(lc, old_delta_qry->rtable)
+ {
+ RangeTblEntry *r = (RangeTblEntry*) lfirst(lc);
+ if (r->relid == relid)
+ {
+ lfirst(lc) = rte;
+ break;
+ }
+ }
+
+ fn->agg_star = true;
+ old_delta_qry->groupClause = transformDistinctClause(NULL, &old_delta_qry->targetList, old_delta_qry->sortClause, false);
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(old_delta_qry->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ old_delta_qry->targetList = lappend(old_delta_qry->targetList, tle);
+ old_delta_qry->hasAggs = true;
+ }
+
+
+ /*
+ * Check for active uses of the relation in the current transaction, such
+ * as open scans.
+ *
+ * NB: We count on this to protect us against problems with refreshing the
+ * data using TABLE_INSERT_FROZEN.
+ */
+ CheckTableNotInUse(matviewRel, "REFRESH MATERIALIZED VIEW");
+
+ relowner = matviewRel->rd_rel->relowner;
+
+ /*
+ * Switch to the owner's userid, so that any functions are run as that
+ * user. Also arrange to make GUC variable changes local to this command.
+ * Don't lock it down too tight to create a temporary table just yet. We
+ * will switch modes when we are about to execute user code.
+ */
+ GetUserIdAndSecContext(&save_userid, &save_sec_context);
+ SetUserIdAndSecContext(relowner,
+ save_sec_context | SECURITY_LOCAL_USERID_CHANGE);
+ save_nestlevel = NewGUCNestLevel();
+
+ tableSpace = GetDefaultTablespace(RELPERSISTENCE_TEMP, false);
+ relpersistence = RELPERSISTENCE_TEMP;
+
+ /*
+ * Create the transient table that will receive the regenerated data. Lock
+ * it against access by any other process until commit (by which time it
+ * will be gone).
+ */
+ if (trigdata->tg_newtable)
+ {
+ OIDDelta_new = make_new_heap(matviewOid, tableSpace, relpersistence,
+ ExclusiveLock);
+ LockRelationOid(OIDDelta_new, AccessExclusiveLock);
+ dest_new = CreateTransientRelDestReceiver(OIDDelta_new);
+ }
+ if (trigdata->tg_oldtable)
+ {
+ if (trigdata->tg_newtable)
+ OIDDelta_old = make_new_heap(OIDDelta_new, tableSpace, relpersistence,
+ ExclusiveLock);
+ else
+ OIDDelta_old = make_new_heap(matviewOid, tableSpace, relpersistence,
+ ExclusiveLock);
+ LockRelationOid(OIDDelta_old, AccessExclusiveLock);
+ dest_old = CreateTransientRelDestReceiver(OIDDelta_old);
+ }
+
+ /*
+ * Now lock down security-restricted operations.
+ */
+ SetUserIdAndSecContext(relowner,
+ save_sec_context | SECURITY_RESTRICTED_OPERATION);
+
+ /* Generate the data. */
+ if (trigdata->tg_newtable)
+ refresh_matview_datafill(dest_new, new_delta_qry, queryEnv, NULL);
+ if (trigdata->tg_oldtable)
+ refresh_matview_datafill(dest_old, old_delta_qry, queryEnv, NULL);
+
+ PG_TRY();
+ {
+ apply_delta(matviewOid, OIDDelta_new, OIDDelta_old, relowner,
+ save_sec_context);
+ }
+ PG_CATCH();
+ {
+ matview_maintenance_depth = old_depth;
+ PG_RE_THROW();
+ }
+ PG_END_TRY();
+
+ table_close(matviewRel, NoLock);
+
+ /* Roll back any GUC changes */
+ AtEOXact_GUC(false, save_nestlevel);
+
+ /* Restore userid and security context */
+ SetUserIdAndSecContext(save_userid, save_sec_context);
+
+ return PointerGetDatum(NULL);
+}
+
+static void
+apply_delta(Oid matviewOid, Oid tempOid_new, Oid tempOid_old,
+ Oid relowner, int save_sec_context)
+{
+ StringInfoData querybuf;
+ StringInfoData mvatts_buf, diffatts_buf;
+ Relation matviewRel;
+ Relation tempRel_new = NULL, tempRel_old = NULL;
+ char *matviewname;
+ char *tempname_new = NULL, *tempname_old = NULL;
+ int i;
+
+
+ initStringInfo(&querybuf);
+ matviewRel = table_open(matviewOid, NoLock);
+ matviewname = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(matviewRel)),
+ RelationGetRelationName(matviewRel));
+
+ if (OidIsValid(tempOid_new))
+ {
+ tempRel_new = table_open(tempOid_new, NoLock);
+ tempname_new = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(tempRel_new)),
+ RelationGetRelationName(tempRel_new));
+ }
+ if (OidIsValid(tempOid_old))
+ {
+ tempRel_old = table_open(tempOid_old, NoLock);
+ tempname_old = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(tempRel_old)),
+ RelationGetRelationName(tempRel_old));
+ }
+
+ initStringInfo(&mvatts_buf);
+ initStringInfo(&diffatts_buf);
+ for (i = 0; i < matviewRel->rd_att->natts; i++)
+ {
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, i);
+
+ if (!strcmp(NameStr(attr->attname), "__ivm_count__"))
+ continue;
+
+ if (i > 0)
+ {
+ appendStringInfo(&mvatts_buf, ", ");
+ appendStringInfo(&diffatts_buf, ", ");
+ }
+ appendStringInfo(&mvatts_buf, "mv.%s", NameStr(attr->attname));
+ appendStringInfo(&diffatts_buf, "diff.%s", NameStr(attr->attname));
+ }
+
+ /* Open SPI context. */
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+
+ /* Analyze the temp table with the new contents. */
+ if (tempname_new)
+ {
+ appendStringInfo(&querybuf, "ANALYZE %s", tempname_new);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ if (tempname_old)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf, "ANALYZE %s", tempname_old);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+
+ SetUserIdAndSecContext(relowner,
+ save_sec_context | SECURITY_LOCAL_USERID_CHANGE);
+
+ OpenMatViewIncrementalMaintenance();
+
+ if (tempname_old)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH t AS ("
+ " SELECT diff.__ivm_count__, (diff.__ivm_count__ = mv.__ivm_count__) AS for_dlt, mv.ctid"
+ " FROM %s AS mv, %s AS diff WHERE (%s) = (%s)"
+ "), updt AS ("
+ " UPDATE %s AS mv SET __ivm_count__ = mv.__ivm_count__ - t.__ivm_count__"
+ " FROM t WHERE mv.ctid = t.ctid AND NOT for_dlt"
+ ") DELETE FROM %s AS mv USING t WHERE mv.ctid = t.ctid AND for_dlt;",
+ matviewname, tempname_old, mvatts_buf.data, diffatts_buf.data, matviewname, matviewname);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_DELETE)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+
+ }
+ if (tempname_new)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH updt AS ("
+ " UPDATE %s AS mv SET __ivm_count__ = mv.__ivm_count__ + diff.__ivm_count__"
+ " FROM %s AS diff WHERE (%s) = (%s)"
+ " RETURNING %s"
+ ") INSERT INTO %s (SELECT * FROM %s AS diff WHERE (%s) NOT IN (SELECT * FROM updt));",
+ matviewname, tempname_new, mvatts_buf.data, diffatts_buf.data, diffatts_buf.data, matviewname, tempname_new, diffatts_buf.data);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+
+ }
+
+ /* We're done maintaining the materialized view. */
+ CloseMatViewIncrementalMaintenance();
+
+ if (OidIsValid(tempOid_new))
+ table_close(tempRel_new, NoLock);
+ if (OidIsValid(tempOid_old))
+ table_close(tempRel_old, NoLock);
+
+ table_close(matviewRel, NoLock);
+
+ /* Clean up temp tables. */
+ if (OidIsValid(tempOid_new))
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf, "DROP TABLE %s", tempname_new);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ if (OidIsValid(tempOid_old))
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf, "DROP TABLE %s", tempname_old);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+
+ /* Close SPI context. */
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+}
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 78deade89b..8262ac039b 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2361,6 +2361,7 @@ _copyRangeTblEntry(const RangeTblEntry *from)
COPY_SCALAR_FIELD(relkind);
COPY_SCALAR_FIELD(rellockmode);
COPY_NODE_FIELD(tablesample);
+ COPY_SCALAR_FIELD(relisivm);
COPY_NODE_FIELD(subquery);
COPY_SCALAR_FIELD(security_barrier);
COPY_SCALAR_FIELD(jointype);
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 4f2ebe5118..608d477bd5 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2641,6 +2641,7 @@ _equalRangeTblEntry(const RangeTblEntry *a, const RangeTblEntry *b)
COMPARE_SCALAR_FIELD(relkind);
COMPARE_SCALAR_FIELD(rellockmode);
COMPARE_NODE_FIELD(tablesample);
+ COMPARE_SCALAR_FIELD(relisivm);
COMPARE_NODE_FIELD(subquery);
COMPARE_SCALAR_FIELD(security_barrier);
COMPARE_SCALAR_FIELD(jointype);
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 8400dd319e..7897dd9c57 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -3042,6 +3042,7 @@ _outRangeTblEntry(StringInfo str, const RangeTblEntry *node)
WRITE_CHAR_FIELD(relkind);
WRITE_INT_FIELD(rellockmode);
WRITE_NODE_FIELD(tablesample);
+ WRITE_BOOL_FIELD(relisivm);
break;
case RTE_SUBQUERY:
WRITE_NODE_FIELD(subquery);
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 6c2626ee62..aa01e205c3 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1366,6 +1366,7 @@ _readRangeTblEntry(void)
READ_CHAR_FIELD(relkind);
READ_INT_FIELD(rellockmode);
READ_NODE_FIELD(tablesample);
+ READ_BOOL_FIELD(relisivm);
break;
case RTE_SUBQUERY:
READ_NODE_FIELD(subquery);
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 8311b1dd46..7cae68218b 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -418,6 +418,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -645,7 +646,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -4054,30 +4055,32 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->relkind = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->relkind = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -4094,9 +4097,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -15128,6 +15136,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDEX
| INDEXES
| INHERIT
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
index 77a48b039d..740ce89c34 100644
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -37,6 +37,7 @@
#include "utils/syscache.h"
#include "utils/varlena.h"
+#include "commands/matview.h"
#define MAX_FUZZY_DISTANCE 3
@@ -1238,6 +1239,7 @@ addRangeTableEntry(ParseState *pstate,
rte->relid = RelationGetRelid(rel);
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -1317,6 +1319,7 @@ addRangeTableEntryForRelation(ParseState *pstate,
rte->relid = RelationGetRelid(rel);
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -2605,6 +2608,9 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
{
Form_pg_attribute attr = TupleDescAttr(tupdesc, varattno);
+ if (!strcmp("__ivm_count__", NameStr(attr->attname)) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
if (attr->attisdropped)
{
if (include_dropped)
diff --git a/src/backend/rewrite/rewriteDefine.c b/src/backend/rewrite/rewriteDefine.c
index 7df2b6154c..5385a038b1 100644
--- a/src/backend/rewrite/rewriteDefine.c
+++ b/src/backend/rewrite/rewriteDefine.c
@@ -765,7 +765,8 @@ checkRuleResultList(List *targetList, TupleDesc resultDesc, bool isSelect,
attr->atttypmod))));
}
- if (i != resultDesc->natts)
+ /* No check for materialized views since this could have special columns for IVM */
+ if ((!isSelect || requireColumnNameMatch) && i != resultDesc->natts)
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
isSelect ?
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index ea40c28733..5b63c16616 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -41,6 +41,8 @@
#include "utils/lsyscache.h"
#include "utils/rel.h"
+#include "parser/parser.h"
+#include "commands/matview.h"
/* We use a list of these to detect recursion in RewriteQuery */
typedef struct rewrite_event
@@ -1597,6 +1599,50 @@ ApplyRetrieveRule(Query *parsetree,
if (rule->qual != NULL)
elog(ERROR, "cannot handle qualified ON SELECT rule");
+ if (RelationIsIVM(relation))
+ {
+ rule_action = copyObject(linitial(rule->actions));
+
+ if (!rule_action->distinctClause)
+ {
+ StringInfoData str;
+ RawStmt *raw;
+ Query *sub;
+
+ if (rule_action->hasDistinctOn)
+ elog(ERROR, "DISTINCT ON is not supported in IVM");
+
+ initStringInfo(&str);
+ appendStringInfo(&str, "SELECT mv.*, __ivm_count__ FROM %s mv, generate_series(1, mv.__ivm_count__)",
+ quote_qualified_identifier(get_namespace_name(RelationGetNamespace(relation)),
+ RelationGetRelationName(relation)));
+
+ raw = (RawStmt*)linitial(raw_parser(str.data));
+ sub = transformStmt(make_parsestate(NULL),raw->stmt);
+
+ rte = rt_fetch(rt_index, parsetree->rtable);
+
+ rte->rtekind = RTE_SUBQUERY;
+ rte->subquery = sub;
+ rte->security_barrier = RelationIsSecurityView(relation);
+ /* Clear fields that should not be set in a subquery RTE */
+ rte->relid = InvalidOid;
+ rte->relkind = 0;
+ rte->rellockmode = 0;
+ rte->tablesample = NULL;
+ rte->inh = false; /* must not be set for a subquery */
+
+ rte->requiredPerms = 0; /* no permission check on subquery itself */
+ rte->checkAsUser = InvalidOid;
+ rte->selectedCols = NULL;
+ rte->insertedCols = NULL;
+ rte->updatedCols = NULL;
+ rte->extraUpdatedCols = NULL;
+ }
+
+ return parsetree;
+ }
+
if (rt_index == parsetree->resultRelation)
{
/*
@@ -1906,7 +1952,8 @@ fireRIRrules(Query *parsetree, List *activeRIRs)
* In that case this test would need to be postponed till after we've
* opened the rel, so that we could check its state.
*/
- if (rte->relkind == RELKIND_MATVIEW)
+ if (rte->relkind == RELKIND_MATVIEW &&
+ (!rte->relisivm || MatViewIncrementalMaintenanceIsEnabled() || parsetree->commandType != CMD_SELECT))
continue;
/*
diff --git a/src/backend/utils/cache/lsyscache.c b/src/backend/utils/cache/lsyscache.c
index c13c08a97b..296cc53ffd 100644
--- a/src/backend/utils/cache/lsyscache.c
+++ b/src/backend/utils/cache/lsyscache.c
@@ -1844,6 +1844,30 @@ get_rel_relispartition(Oid relid)
return false;
}
+/*
+ * get_rel_relisivm
+ *
+ * Returns the relisivm flag associated with a given relation.
+ */
+bool
+get_rel_relisivm(Oid relid)
+{
+ HeapTuple tp;
+
+ tp = SearchSysCache1(RELOID, ObjectIdGetDatum(relid));
+ if (HeapTupleIsValid(tp))
+ {
+ Form_pg_class reltup = (Form_pg_class) GETSTRUCT(tp);
+ bool result;
+
+ result = reltup->relisivm;
+ ReleaseSysCache(tp);
+ return result;
+ }
+ else
+ return false;
+}
+
/*
* get_rel_tablespace
*
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 2b992d7832..d80bb30696 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -1861,6 +1861,8 @@ formrdesc(const char *relationName, Oid relationReltype,
/* ... and they're always populated, too */
relation->rd_rel->relispopulated = true;
+ /* ... and they're always no ivm, too */
+ relation->rd_rel->relisivm = false;
relation->rd_rel->relreplident = REPLICA_IDENTITY_NOTHING;
relation->rd_rel->relpages = 0;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 5e38f46399..d285ab6740 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -998,6 +998,7 @@ static const pgsql_thing_t words_after_create[] = {
{"FOREIGN TABLE", NULL, NULL, NULL},
{"FUNCTION", NULL, NULL, Query_for_list_of_functions},
{"GROUP", Query_for_list_of_roles},
+ {"INCREMENTAL MATERIALIZED VIEW", NULL, NULL, &Query_for_list_of_matviews},
{"INDEX", NULL, NULL, &Query_for_list_of_indexes},
{"LANGUAGE", Query_for_list_of_languages},
{"LARGE OBJECT", NULL, NULL, NULL, THING_NO_CREATE | THING_NO_DROP},
@@ -2482,7 +2483,7 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("SEQUENCE", "TABLE", "VIEW");
/* Complete "CREATE UNLOGGED" with TABLE or MATVIEW */
else if (TailMatches("CREATE", "UNLOGGED"))
- COMPLETE_WITH("TABLE", "MATERIALIZED VIEW");
+ COMPLETE_WITH("TABLE", "MATERIALIZED VIEW", "INCREMENTAL MATERIALIZED VIEW");
/* Complete PARTITION BY with RANGE ( or LIST ( or ... */
else if (TailMatches("PARTITION", "BY"))
COMPLETE_WITH("RANGE (", "LIST (", "HASH (");
@@ -2691,13 +2692,16 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("SELECT");
/* CREATE MATERIALIZED VIEW */
- else if (Matches("CREATE", "MATERIALIZED"))
+ else if (Matches("CREATE", "MATERIALIZED") ||
+ Matches("CREATE", "INCREMENTAL", "MATERIALIZED"))
COMPLETE_WITH("VIEW");
- /* Complete CREATE MATERIALIZED VIEW <name> with AS */
- else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny))
+ /* Complete CREATE MATERIALIZED VIEW <name> with AS */
+ else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny) ||
+ Matches("CREATE", "INCREMENTAL", "MATERIALIZED", "VIEW", MatchAny))
COMPLETE_WITH("AS");
/* Complete "CREATE MATERIALIZED VIEW <sth> AS with "SELECT" */
- else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny, "AS"))
+ else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny, "AS") ||
+ Matches("CREATE", "INCREMENTAL", "MATERIALIZED", "VIEW", MatchAny, "AS"))
COMPLETE_WITH("SELECT");
/* CREATE EVENT TRIGGER */
diff --git a/src/include/catalog/pg_class.dat b/src/include/catalog/pg_class.dat
index 9bcf28676d..7deda405af 100644
--- a/src/include/catalog/pg_class.dat
+++ b/src/include/catalog/pg_class.dat
@@ -27,7 +27,7 @@
relpersistence => 'p', relkind => 'r', relnatts => '31', relchecks => '0',
relhasrules => 'f', relhastriggers => 'f', relhassubclass => 'f',
relrowsecurity => 'f', relforcerowsecurity => 'f', relispopulated => 't',
- relreplident => 'n', relispartition => 'f', relfrozenxid => '3',
+ relreplident => 'n', relispartition => 'f', relisivm => 'f', relfrozenxid => '3',
relminmxid => '1', relacl => '_null_', reloptions => '_null_',
relpartbound => '_null_' },
{ oid => '1249',
@@ -37,7 +37,7 @@
relpersistence => 'p', relkind => 'r', relnatts => '25', relchecks => '0',
relhasrules => 'f', relhastriggers => 'f', relhassubclass => 'f',
relrowsecurity => 'f', relforcerowsecurity => 'f', relispopulated => 't',
- relreplident => 'n', relispartition => 'f', relfrozenxid => '3',
+ relreplident => 'n', relispartition => 'f', relisivm => 'f', relfrozenxid => '3',
relminmxid => '1', relacl => '_null_', reloptions => '_null_',
relpartbound => '_null_' },
{ oid => '1255',
@@ -47,17 +47,17 @@
relpersistence => 'p', relkind => 'r', relnatts => '29', relchecks => '0',
relhasrules => 'f', relhastriggers => 'f', relhassubclass => 'f',
relrowsecurity => 'f', relforcerowsecurity => 'f', relispopulated => 't',
- relreplident => 'n', relispartition => 'f', relfrozenxid => '3',
+ relreplident => 'n', relispartition => 'f', relisivm => 'f', relfrozenxid => '3',
relminmxid => '1', relacl => '_null_', reloptions => '_null_',
relpartbound => '_null_' },
{ oid => '1259',
relname => 'pg_class', reltype => 'pg_class', relam => 'heap',
relfilenode => '0', relpages => '0', reltuples => '0', relallvisible => '0',
reltoastrelid => '0', relhasindex => 'f', relisshared => 'f',
- relpersistence => 'p', relkind => 'r', relnatts => '33', relchecks => '0',
+ relpersistence => 'p', relkind => 'r', relnatts => '34', relchecks => '0',
relhasrules => 'f', relhastriggers => 'f', relhassubclass => 'f',
relrowsecurity => 'f', relforcerowsecurity => 'f', relispopulated => 't',
- relreplident => 'n', relispartition => 'f', relfrozenxid => '3',
+ relreplident => 'n', relispartition => 'f', relisivm => 'f', relfrozenxid => '3',
relminmxid => '1', relacl => '_null_', reloptions => '_null_',
relpartbound => '_null_' },
diff --git a/src/include/catalog/pg_class.h b/src/include/catalog/pg_class.h
index 090b6ba907..ff535f5504 100644
--- a/src/include/catalog/pg_class.h
+++ b/src/include/catalog/pg_class.h
@@ -116,6 +116,9 @@ CATALOG(pg_class,1259,RelationRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(83,Relat
/* is relation a partition? */
bool relispartition;
+ /* is relation a matview with ivm? */
+ bool relisivm;
+
/* heap for rewrite during DDL, link to original rel */
Oid relrewrite BKI_DEFAULT(0);
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 87335248a0..5de996a72c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10677,4 +10677,9 @@
proname => 'pg_partition_root', prorettype => 'regclass',
proargtypes => 'regclass', prosrc => 'pg_partition_root' },
+# IVM
+{ oid => '784', descr => 'ivm trigger',
+ proname => 'IVM_immediate_maintenance', provolatile => 'v', prorettype => 'trigger',
+ proargtypes => '', prosrc => 'IVM_immediate_maintenance' },
+
]
diff --git a/src/include/commands/matview.h b/src/include/commands/matview.h
index edf04bf415..8dd8193d9c 100644
--- a/src/include/commands/matview.h
+++ b/src/include/commands/matview.h
@@ -23,6 +23,8 @@
extern void SetMatViewPopulatedState(Relation relation, bool newstate);
+extern void SetMatViewIVMState(Relation relation, bool newstate);
+
extern ObjectAddress ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
ParamListInfo params, char *completionTag);
@@ -30,4 +32,6 @@ extern DestReceiver *CreateTransientRelDestReceiver(Oid oid);
extern bool MatViewIncrementalMaintenanceIsEnabled(void);
+extern Datum IVM_immediate_maintenance(PG_FUNCTION_ARGS);
+
#endif /* MATVIEW_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 94ded3c135..5c8a5ae3ca 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1002,6 +1002,7 @@ typedef struct RangeTblEntry
char relkind; /* relation kind (see pg_class.relkind) */
int rellockmode; /* lock level that query requires on the rel */
struct TableSampleClause *tablesample; /* sampling info, or NULL */
+ bool relisivm;
/*
* Fields valid for a subquery RTE (else NULL):
@@ -2059,6 +2060,7 @@ typedef struct CreateStmt
char *tablespacename; /* table space to use, or NULL */
char *accessMethod; /* table access method */
bool if_not_exists; /* just do nothing if it already exists? */
+ bool ivm; /* incremental view maintenance is used by materialized view */
} CreateStmt;
/* ----------
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 7c278c0e56..0aaef1ef15 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -117,6 +117,7 @@ typedef struct IntoClause
char *tableSpaceName; /* table space to use, or NULL */
Node *viewQuery; /* materialized view's SELECT query */
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 00ace8425e..d682ee11cc 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -198,6 +198,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD)
PG_KEYWORD("inherit", INHERIT, UNRESERVED_KEYWORD)
diff --git a/src/include/utils/lsyscache.h b/src/include/utils/lsyscache.h
index c8df5bff9f..8fd919349e 100644
--- a/src/include/utils/lsyscache.h
+++ b/src/include/utils/lsyscache.h
@@ -129,6 +129,7 @@ extern Oid get_rel_namespace(Oid relid);
extern Oid get_rel_type_id(Oid relid);
extern char get_rel_relkind(Oid relid);
extern bool get_rel_relispartition(Oid relid);
+extern bool get_rel_relisivm(Oid relid);
extern Oid get_rel_tablespace(Oid relid);
extern char get_rel_persistence(Oid relid);
extern Oid get_transform_fromsql(Oid typid, Oid langid, List *trftypes);
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index d7f33abce3..057f38d5de 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -562,6 +562,8 @@ typedef struct ViewOptions
*/
#define RelationIsPopulated(relation) ((relation)->rd_rel->relispopulated)
+#define RelationIsIVM(relation) ((relation)->rd_rel->relisivm)
+
/*
* RelationIsAccessibleInLogicalDecoding
* True if we need to log enough information to have access via
Hi Yugo,
I'd like to compare the performance of your MV refresh algorithm versus
an approach that logs changes into an mv log table, and can then apply the
changes at some later point in time. I'd like to handle the materialized
join view (mjv) case first, specifically a 2-way left outer join, with a UDF
in the SELECT list of the mjv.
Does your refresh algorithm handle mjv's with connected join graphs that
consist entirely of inner and left outer joins?
If so, I'd like to measure the overhead of your refresh algorithm on
pgbench, modified to include an mjv, versus a (hand coded) incremental
maintenance algorithm that uses mv log tables populated by ordinary
triggers. We may also want to look at capturing the deltas using logical
replication, which ought to be faster than a trigger-based solution.
I have someone available to do the performance testing for another 2
months, so if you can connect with me off-list to coordinate, we can set up
the performance experiments and run them on our AWS clusters.
best regards,
/Jim F
-----
Jim Finnerty, AWS, Amazon Aurora PostgreSQL
--
Sent from: http://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html
Hi Jim,
On Fri, 21 Jun 2019 08:41:11 -0700 (MST)
Jim Finnerty <jfinnert@amazon.com> wrote:
Hi Yugo,
I'd like to compare the performance of your MV refresh algorithm versus
an approach that logs changes into an mv log table, and can then apply the
changes at some later point in time. I'd like to handle the materialized
join view (mjv) case first, specifically a 2-way left outer join, with a UDF
in the SELECT list of the mjv.
Do you mean you have your implementation of IVM that using log tables?
I'm so interested in this, and I would appreciate it if you explain the
detail.
Does your refresh algorithm handle mjv's with connected join graphs that
consist entirely of inner and left outer joins?
If so, I'd like to measure the overhead of your refresh algorithm on
pgbench, modified to include an mjv, versus a (hand coded) incremental
maintenance algorithm that uses mv log tables populated by ordinary
triggers. We may also want to look at capturing the deltas using logical
replication, which ought to be faster than a trigger-based solution.
In the current our implementation, outer joins is not yet supported though
we plan to handle this in future. So,we would not be able to compare these
directly in the same workload in the current status.
However, the current our implementation supports only the way to update
materialized views in a trigger, and the performance of modifying base tables
will be lower than the approach which uses log tables. This is because queries
to update materialized views are issued in the trigger. This is not only a
overhead itself, but also takes a lock on a materialized view, which has an ]
impact on concurrent execution performance.
In the previous our PoC, we implemented IVM using log tables, in which logs are
captured by triggers and materialized views are update incrementally by a user
command[1]https://www.postgresql.eu/events/pgconfeu2018/schedule/session/2195-implementing-incremental-view-maintenance-on-postgresql/. However, to implement log table approach, we need a infrastructure
to maintain these logs. For example, which logs are necessary and which logs
can be discarded, etc. We thought this is not trivial work, so we decided to
start from the current approach which doesn't use log tables. We are now
preparing to implement this in the next step because this is also needed to
support deferred maintenance of views.
I agree that capturing the deltas using logical decoding will be faster than
using a trigger although we haven't yet consider this well.
Best regadrds,
Yugo Nagata
--
Yugo Nagata <nagata@sraoss.co.jp>
Hi,
Attached is a WIP patch of IVM which supports some aggregate functions.
Currently, only count and sum are supported. Avg, min, or max is not supported
although I think supporting this would not be so hard.
As a restriction, expressions specified in GROUP BY must appear in the target
list of views because tuples to be updated in MV are identified by using this
group keys.
In the case of views without aggregate functions, only the number of tuple
duplicates (__ivm_count__) are updated at incremental maintenance. On the other
hand, in the case of vies with aggregations, the aggregated values are also
updated. The way of update depends the kind of aggregate function.
In the case of sum (or agg functions except to count), NULL in input values is
ignored, and this returns a null value when no rows are selected. To support
this specification, the number of not-NULL input values is counted and stored
in MV as a hidden column whose name is like __ivm_count_sum__, for example.
In the case of count, this returns zero when no rows are selected, and count(*)
doesn't ignore NULL input. These specification are also supported.
Tuples to be updated in MV are identified by using keys specified by GROUP BY
clause. However, in the case of aggregation without GROUP BY, there is only one
tuple in the view, so keys are not uses to identify tuples.
In addition, a race condition which occurred in the previous version is
prevented in this patch. In the previous version, when two translocations
change a base tables concurrently, an anormal update of MV was possible because
a change in one transaction was not visible for another transaction even in
READ COMMITTED level.
To prevent this, I fix this to take a lock in early stage of view maintenance
to wait for concurrent transactions which are updating the same MV end. Also,
we have to get the latest snapshot before computting delta tables because any
changes which occurs in other transaction during lock waiting is not visible
even in READ COMMITTED level.
In REPEATABLE READ or SERIALIZABLE level, don't wait a lock, and raise an error
immediately to prevent anormal update. These solutions might be ugly, but
something to prevent anormal update is anyway necessary. There may be better
way.
Moreover, some regression test are added for aggregate functions support.
This is Hoshiai-san's work.
Although the code is not refined yet and I will need a deal of refactoring
and reorganizing, I submitted this to share the current status.
* Exapmle (from regression test)
=======================================================================
(1) creating tables
CREATE TABLE mv_base_a (i int, j int);
INSERT INTO mv_base_a VALUES
(1,10),
(2,20),
(3,30),
(4,40),
(5,50);
(2) Views sith SUM() and COUNT() aggregation function
BEGIN;
CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_agg AS SELECT i, SUM(j), COUNT(i) FROM mv_base_a GROUP BY i;
SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
i | sum | count
---+-----+-------
1 | 10 | 1
2 | 20 | 1
3 | 30 | 1
4 | 40 | 1
5 | 50 | 1
(5 rows)
INSERT INTO mv_base_a VALUES(2,100);
SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
i | sum | count
---+-----+-------
1 | 10 | 1
2 | 120 | 2
3 | 30 | 1
4 | 40 | 1
5 | 50 | 1
(5 rows)
UPDATE mv_base_a SET j = 200 WHERE (i,j) = (2,100);
SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
i | sum | count
---+-----+-------
1 | 10 | 1
2 | 220 | 2
3 | 30 | 1
4 | 40 | 1
5 | 50 | 1
(5 rows)
DELETE FROM mv_base_a WHERE (i,j) = (2,200);
SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
i | sum | count
---+-----+-------
1 | 10 | 1
2 | 20 | 1
3 | 30 | 1
4 | 40 | 1
5 | 50 | 1
(5 rows)
ROLLBACK;
(3) Views with COUNT(*) aggregation function
BEGIN;
CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_agg AS SELECT i, SUM(j),COUNT(*) FROM mv_base_a GROUP BY i;
SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
i | sum | count
---+-----+-------
1 | 10 | 1
2 | 20 | 1
3 | 30 | 1
4 | 40 | 1
5 | 50 | 1
(5 rows)
INSERT INTO mv_base_a VALUES(2,100);
SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
i | sum | count
---+-----+-------
1 | 10 | 1
2 | 120 | 2
3 | 30 | 1
4 | 40 | 1
5 | 50 | 1
(5 rows)
ROLLBACK;
(4) Views with aggregation function without GROUP clause
BEGIN;
CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_group AS SELECT SUM(j)FROM mv_base_a;
SELECT * FROM mv_ivm_group ORDER BY 1;
sum
-----
150
(1 row)
INSERT INTO mv_base_a VALUES(6,20);
SELECT * FROM mv_ivm_group ORDER BY 1;
sum
-----
170
(1 row)
=======================================================================
On Thu, 20 Jun 2019 16:44:10 +0900
Yugo Nagata <nagata@sraoss.co.jp> wrote:
Hi hackers,
Thank you for your many questions and feedbacks at PGCon 2019.
Attached is the patch rebased for the current master branch.Regards,
Yugo NagataOn Tue, 14 May 2019 15:46:48 +0900
Yugo Nagata <nagata@sraoss.co.jp> wrote:On Mon, 1 Apr 2019 12:11:22 +0900
Yugo Nagata <nagata@sraoss.co.jp> wrote:On Thu, 27 Dec 2018 21:57:26 +0900
Yugo Nagata <nagata@sraoss.co.jp> wrote:Hi,
I would like to implement Incremental View Maintenance (IVM) on PostgreSQL.
I am now working on an initial patch for implementing IVM on PostgreSQL.
This enables materialized views to be updated incrementally after one
of their base tables is modified.Attached is a WIP patch of Incremental View Maintenance (IVM).
Major part is written by me, and changes in syntax and pg_class
are Hoshiai-san's work.Although this is sill a draft patch in work-in-progress, any
suggestions or thoughts would be appreciated.* What it is
This allows a kind of Immediate Maintenance of materialized views. if a
materialized view is created by CRATE INCREMENTAL MATERIALIZED VIEW command,
the contents of the mateview is updated automatically and incrementally
after base tables are updated. Noted this syntax is just tentative, so it
may be changed.====== Example 1 ======
postgres=# CREATE INCREMENTAL MATERIALIZED VIEW m AS SELECT * FROM t0;
SELECT 3
postgres=# SELECT * FROM m;
i
---
3
2
1
(3 rows)postgres=# INSERT INTO t0 VALUES (4);
INSERT 0 1
postgres=# SELECt * FROM m; -- automatically updated
i
---
3
2
1
4
(4 rows)
=============================This implementation also supports matviews including duplicate tuples or
DISTINCT clause in its view definition query. For example, even if a matview
is defined with DISTINCT to remove duplication of tuples in a base table, this
can perform incremental update of the matview properly. That is, the contents
of the matview doesn't change when exiting tuples are inserted into the base
tables, and a tuple in the matview is deleted only when duplicity of the
corresponding tuple in the base table becomes zero.This is due to "colunting alogorithm" in which the number of each tuple is
stored in matviews as a special column value.====== Example 2 ======
postgres=# SELECT * FROM t1;
id | t
----+---
1 | A
2 | B
3 | C
4 | A
(4 rows)postgres=# CREATE INCREMENTAL MATERIALIZED VIEW m1 AS SELECT t FROM t1;
SELECT 3
postgres=# CREATE INCREMENTAL MATERIALIZED VIEW m2 AS SELECT DISTINCT t FROM t1;
SELECT 3
postgres=# SELECT * FROM m1; -- with duplicity
t
---
A
A
C
B
(4 rows)postgres=# SELECT * FROM m2;
t
---
A
B
C
(3 rows)postgres=# INSERT INTO t1 VALUES (5, 'B');
INSERT 0 1
postgres=# DELETE FROM t1 WHERE id IN (1,3); -- delete (1,A),(3,C)
DELETE 2
postgres=# SELECT * FROM m1; -- one A left and one more B
t
---
B
B
A
(3 rows)postgres=# SELECT * FROM m2; -- only C is removed
t
---
B
A
(2 rows)
=============================* How it works
1. Creating matview
When a matview is created, AFTER triggers are internally created
on its base tables. When the base tables is modified (INSERT, DELETE,
UPDATE), the matview is updated incrementally in the trigger function.When populating the matview, GROUP BY and count(*) are added to the
view definition query before this is executed for counting duplicity
of tuples in the matview. The result of count is stored in the matview
as a special column named "__ivm_count__".2. Maintenance of matview
When base tables are modified, the change set of the table can be
referred as Ephemeral Named Relations (ENRs) thanks to Transition Table
(a feature of trigger implemented since PG10). We can calculate the diff
set of the matview by replacing the base table in the view definition
query with the ENR (at least if it is Selection-Projection -Join view).
As well as view definition time, GROUP BY and count(*) is added in order
to count the duplicity of tuples in the diff set. As a result, two diff
sets (to be deleted from and to be inserted into the matview) are
calculated, and the results are stored into temporary tables respectively.The matiview is updated by merging these change sets. Instead of executing
DELETE or INSERT simply, the values of __ivm_count__ column in the matview
is decreased or increased. When the values becomes zero, the corresponding
tuple is deleted from the matview.3. Access to matview
When SELECT is issued for IVM matviews defined with DISTINCT, all columns
except to __ivm_count__ of each tuple in the matview are returned. This is
correct because duplicity of tuples are eliminated by GROUP BY.When DISTINCT is not used, SELECT for the IVM matviews returns each tuple
__ivm_count__ times. Currently, this is implemented by rewriting the SELECT
query to replace the matview RTE with a subquery which joins the matview
and generate_series function as bellow.SELECT mv.* FROM mv, generate_series(1, mv.__ivm_count__);
__ivm_count__ column is invisible for users when "SELECT * FROM ..." is
issued, but users can see the value by specifying in target list explicitly.====== Example 3 ======
postgres=# \d+ m1
Materialized view "public.m1"
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
---------------+--------+-----------+----------+---------+----------+--------------+-------------
t | text | | | | extended | |
__ivm_count__ | bigint | | | | plain | |
View definition:
SELECT t1.t
FROM t1;
Access method: heappostgres=# \d+ m2
Materialized view "public.m2"
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
---------------+--------+-----------+----------+---------+----------+--------------+-------------
t | text | | | | extended | |
__ivm_count__ | bigint | | | | plain | |
View definition:
SELECT DISTINCT t1.t
FROM t1;
Access method: heappostgres=# SELECT *, __ivm_count__ FROM m1;
t | __ivm_count__
---+---------------
B | 2
B | 2
A | 1
(3 rows)postgres=# SELECT *, __ivm_count__ FROM m2;
t | __ivm_count__
---+---------------
B | 2
A | 1
(2 rows)postgres=# EXPLAIN SELECT * FROM m1;
QUERY PLAN
------------------------------------------------------------------------------
Nested Loop (cost=0.00..61.03 rows=3000 width=2)
-> Seq Scan on m1 mv (cost=0.00..1.03 rows=3 width=10)
-> Function Scan on generate_series (cost=0.00..10.00 rows=1000 width=0)
(3 rows)
=============================* Simple Performance Evaluation
I confirmed that "incremental" update of matviews is more effective
than the standard REFRESH by using simple exapmle. I used tables
of pgbench (SF=100) here.Create two matviews, that is, without and with IVM.
test=# CREATE MATERIALIZED VIEW bench1 AS
SELECT aid, bid, abalance, bbalance
FROM pgbench_accounts JOIN pgbench_branches USING (bid)
WHERE abalance > 0 OR bbalance > 0;
SELECT 5001054
test=# CREATE INCREMENTAL MATERIALIZED VIEW bench2 AS
SELECT aid, bid, abalance, bbalance
FROM pgbench_accounts JOIN pgbench_branches USING (bid)
WHERE abalance > 0 OR bbalance > 0;
SELECT 5001054The standard REFRESH of bench1 took more than 10 seconds.
test=# \timing
Timing is on.
test=# REFRESH MATERIALIZED VIEW bench1 ;
REFRESH MATERIALIZED VIEW
Time: 11210.563 ms (00:11.211)Create an index on the IVM matview (bench2).
test=# CREATE INDEX on bench2(aid,bid);
CREATE INDEXUpdating a tuple in pgbench_accounts took 18ms. After this, bench2
was updated automatically and correctly.test=# SELECT * FROM bench2 WHERE aid = 1;
aid | bid | abalance | bbalance
-----+-----+----------+----------
1 | 1 | 10 | 10
(1 row)Time: 2.498 ms
test=# UPDATE pgbench_accounts SET abalance = 1000 WHERE aid = 1;
UPDATE 1
Time: 18.634 ms
test=# SELECT * FROM bench2 WHERE aid = 1;
aid | bid | abalance | bbalance
-----+-----+----------+----------
1 | 1 | 1000 | 10
(1 row)However, if there is not the index on bench2, it took 4 sec, so
appropriate indexes are needed on IVM matviews.test=# DROP INDEX bench2_aid_bid_idx ;
DROP INDEX
Time: 10.613 ms
test=# UPDATE pgbench_accounts SET abalance = 2000 WHERE aid = 1;
UPDATE 1
Time: 3931.274 ms (00:03.931)* Restrictions on view definition
This patch is still in Work-in-Progress and there are many restrictions
on the view definition query of matviews.The current implementation supports views including selection, projection,
and inner join with or without DISTINCT. Aggregation and GROUP BY are not
supported yet, but I plan to deal with these by the first release.
Self-join, subqueries, OUTER JOIN, CTE, window functions are not
considered well, either. I need more investigation on these type of views
although I found some papers explaining how to handle sub-queries and
outer-joins.These unsupported views should be checked when a matview is created, but
this is not implemented yet. Hoshiai-san are working on this.* Timing of view maintenance
This patch implements a kind of Immediate Maintenance, that is, a matview
is updated immediately when a base table is modified. On other hand, in
"Deferred Maintenance", matviews are updated after the transaction, for
example, by the user command like REFRESH.For implementing "deferred", it is need to implement a mechanism to maintain
logs for recording changes of base tables and an algorithm to compute the
delta to be applied to matviews.In addition, there could be another implementation of Immediate Maintenance
in which matview is updated at the end of a transaction that modified base
table, rather than in AFTER trigger. Oracle supports this type of IVM. To
implement this, we will need a mechanism to maintain change logs on base
tables as well as Deferred maintenance.* Counting algorithm implementation
There will be also discussions on counting-algorithm implementation.
Firstly, the current patch treats "__ivm_count__" as a special column name
in a somewhat ad hoc way. This is used when maintaining and accessing matviews,
and when "SELECT * FROM ..." is issued, __ivm_count__ column is invisible for
users. Maybe this name has to be inhibited in user tables. Is it acceptable
to use such columns for IVM, and is there better way, if not?Secondly, a matview with duplicate tuples is replaces with a subquery which
uses generate_series function. It does not have to be generate_series, and we
can make a new set returning function for this. Anyway, this internal behaviour
is visible in EXPLAIN results as shown in Example 3. Also, there is a
performance impact because estimated rows number is wrong, and what is worse,
the cost of join is not small when the size of matview is large. Therefore, we
might have to add a new plan node for selecting from matviews rather than using
such a special set returning function.Ragards,
--
Yugo Nagata <nagata@sraoss.co.jp>--
Yugo Nagata <nagata@sraoss.co.jp>
--
Yugo Nagata <nagata@sraoss.co.jp>
Attachments:
WIP_immediate_IVM_v3.patchtext/x-diff; name=WIP_immediate_IVM_v3.patchDownload
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index f2b9d404cb..e28b07698d 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -1952,6 +1952,13 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<entry>True if table or index is a partition</entry>
</row>
+ <row>
+ <entry><structfield>relisivm</structfield></entry>
+ <entry><type>bool</type></entry>
+ <entry></entry>
+ <entry>True if materialized view enables incremental view maintenance</entry>
+ </row>
+
<row>
<entry><structfield>relrewrite</structfield></entry>
<entry><type>oid</type></entry>
diff --git a/doc/src/sgml/ref/create_materialized_view.sgml b/doc/src/sgml/ref/create_materialized_view.sgml
index ec8847ed40..a23366a342 100644
--- a/doc/src/sgml/ref/create_materialized_view.sgml
+++ b/doc/src/sgml/ref/create_materialized_view.sgml
@@ -21,7 +21,7 @@ PostgreSQL documentation
<refsynopsisdiv>
<synopsis>
-CREATE MATERIALIZED VIEW [ IF NOT EXISTS ] <replaceable>table_name</replaceable>
+CREATE [ INCREMENTAL ] MATERIALIZED VIEW [ IF NOT EXISTS ] <replaceable>table_name</replaceable>
[ (<replaceable>column_name</replaceable> [, ...] ) ]
[ USING <replaceable class="parameter">method</replaceable> ]
[ WITH ( <replaceable class="parameter">storage_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] ) ]
@@ -54,6 +54,16 @@ CREATE MATERIALIZED VIEW [ IF NOT EXISTS ] <replaceable>table_name</replaceable>
<title>Parameters</title>
<variablelist>
+ <varlistentry>
+ <term><literal>INCREMENTAL</literal></term>
+ <listitem>
+ <para>
+ If specified, a materialized view enables incremental view maintenance.
+ You can replace only the contents of a materialized view, which based rows are changed.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>IF NOT EXISTS</literal></term>
<listitem>
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index 86820eecfc..f0f0e3bb84 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -891,6 +891,7 @@ InsertPgClassTuple(Relation pg_class_desc,
values[Anum_pg_class_relrewrite - 1] = ObjectIdGetDatum(rd_rel->relrewrite);
values[Anum_pg_class_relfrozenxid - 1] = TransactionIdGetDatum(rd_rel->relfrozenxid);
values[Anum_pg_class_relminmxid - 1] = MultiXactIdGetDatum(rd_rel->relminmxid);
+ values[Anum_pg_class_relisivm - 1] = BoolGetDatum(rd_rel->relisivm);
if (relacl != (Datum) 0)
values[Anum_pg_class_relacl - 1] = relacl;
else
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index bb60b23093..0e22a643d7 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -910,6 +910,7 @@ index_create(Relation heapRelation,
indexRelation->rd_rel->relowner = heapRelation->rd_rel->relowner;
indexRelation->rd_rel->relam = accessMethodObjectId;
indexRelation->rd_rel->relispartition = OidIsValid(parentIndexRelid);
+ indexRelation->rd_rel->relisivm = false;
/*
* store index's pg_class entry
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index 4c1d909d38..829c2b7bcd 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -51,6 +51,16 @@
#include "utils/rls.h"
#include "utils/snapmgr.h"
+#include "catalog/pg_trigger.h"
+#include "catalog/pg_type.h"
+#include "commands/trigger.h"
+#include "parser/parser.h"
+#include "parser/parsetree.h"
+#include "parser/parse_func.h"
+#include "nodes/print.h"
+#include "optimizer/optimizer.h"
+#include "commands/defrem.h"
+
typedef struct
{
@@ -74,6 +84,8 @@ static bool intorel_receive(TupleTableSlot *slot, DestReceiver *self);
static void intorel_shutdown(DestReceiver *self);
static void intorel_destroy(DestReceiver *self);
+static void CreateIvmTrigger(Oid relOid, Oid viewOid, char *matviewname, int16 type);
+static void CreateIvmTriggersOnBaseTables(Query *qry, Node *jtnode, Oid matviewOid, char* matviewname);
/*
* create_ctas_internal
@@ -109,6 +121,8 @@ create_ctas_internal(List *attrList, IntoClause *into)
create->oncommit = into->onCommit;
create->tablespacename = into->tableSpaceName;
create->if_not_exists = false;
+ /* Using Materialized view only */
+ create->ivm = into->ivm;
create->accessMethod = into->accessMethod;
/*
@@ -239,6 +253,7 @@ ExecCreateTableAs(CreateTableAsStmt *stmt, const char *queryString,
List *rewritten;
PlannedStmt *plan;
QueryDesc *queryDesc;
+ Query *copied_query;
if (stmt->if_not_exists)
{
@@ -319,7 +334,97 @@ ExecCreateTableAs(CreateTableAsStmt *stmt, const char *queryString,
* and is executed repeatedly. (See also the same hack in EXPLAIN and
* PREPARE.)
*/
- rewritten = QueryRewrite(copyObject(query));
+
+ copied_query = copyObject(query);
+ if (is_matview && into->ivm)
+ {
+ TargetEntry *tle;
+ Node *node;
+ ParseState *pstate = make_parsestate(NULL);
+ FuncCall *fn;
+
+ /* group keys must be in targetlist */
+ if (copied_query->groupClause)
+ {
+ ListCell *lc;
+ foreach(lc, copied_query->groupClause)
+ {
+ SortGroupClause *scl = (SortGroupClause *) lfirst(lc);
+ TargetEntry *tle = get_sortgroupclause_tle(scl, copied_query->targetList);
+
+ if (tle->resjunk)
+ elog(ERROR, "GROUP BY expression must appear in select list for incremental materialized views");
+ }
+ }
+ else if (!copied_query->hasAggs)
+ copied_query->groupClause = transformDistinctClause(NULL, &copied_query->targetList, copied_query->sortClause, false);
+
+ if (copied_query->hasAggs)
+ {
+ ListCell *lc;
+ List *agg_counts = NIL;
+ AttrNumber next_resno = list_length(copied_query->targetList) + 1;
+ Const *dmy_arg = makeConst(INT4OID,
+ -1,
+ InvalidOid,
+ sizeof(int32),
+ Int32GetDatum(1),
+ false,
+ true); /* pass by value */
+
+ foreach(lc, copied_query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ TargetEntry *tle_count;
+
+
+ if (IsA(tle->expr, Aggref))
+ {
+ Aggref *aggref = (Aggref *) tle->expr;
+ const char *aggname = get_func_name(aggref->aggfnoid);
+
+ /* XXX: need some generalization */
+ if (strcmp(aggname, "sum") !=0 && strcmp(aggname, "count") != 0)
+ elog(ERROR, "Aggrege function %s is not supported", aggname);
+
+ /* For aggregate functions except to count, add count func with the same arg parameters. */
+ if (strcmp(aggname, "count") != 0)
+ {
+ fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+
+ /* Make a Func with a dummy arg, and then override this by the original agg's args. */
+ node = ParseFuncOrColumn(pstate, fn->funcname, list_make1(dmy_arg), NULL, fn, false, -1);
+ ((Aggref *)node)->args = aggref->args;
+
+ tle_count = makeTargetEntry((Expr *) node,
+ next_resno,
+ pstrdup(makeObjectName("__ivm_count",tle->resname, "_")),
+ false);
+ agg_counts = lappend(agg_counts, tle_count);
+ next_resno++;
+ }
+
+ }
+ }
+ copied_query->targetList = list_concat(copied_query->targetList, agg_counts);
+
+ }
+
+ /* Add count(*) for counting algorithm */
+ fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+ fn->agg_star = true;
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(copied_query->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ copied_query->targetList = lappend(copied_query->targetList, tle);
+ copied_query->hasAggs = true;
+ }
+
+ rewritten = QueryRewrite(copied_query);
/* SELECT should never rewrite to more or less than one SELECT query */
if (list_length(rewritten) != 1)
@@ -378,11 +483,65 @@ ExecCreateTableAs(CreateTableAsStmt *stmt, const char *queryString,
/* Restore userid and security context */
SetUserIdAndSecContext(save_userid, save_sec_context);
+
+
+ if (into->ivm)
+ {
+ char *matviewname;
+ Oid matviewOid = address.objectId;
+ Relation matviewRel = table_open(matviewOid, NoLock);
+ matviewname = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(matviewRel)),
+ RelationGetRelationName(matviewRel));
+ copied_query = copyObject(query);
+ AcquireRewriteLocks(copied_query, true, false);
+
+ CreateIvmTriggersOnBaseTables(copied_query, (Node *)copied_query->jointree, matviewOid, matviewname);
+
+ table_close(matviewRel, NoLock);
+ }
}
return address;
}
+static void CreateIvmTriggersOnBaseTables(Query *qry, Node *jtnode, Oid matviewOid, char* matviewname)
+{
+
+ if (jtnode == NULL)
+ return;
+ if (IsA(jtnode, RangeTblRef))
+ {
+ int rti = ((RangeTblRef *) jtnode)->rtindex;
+ RangeTblEntry *rte = rt_fetch(rti, qry->rtable);
+
+ if (rte->rtekind == RTE_RELATION)
+ {
+ CreateIvmTrigger(rte->relid, matviewOid, matviewname, TRIGGER_TYPE_INSERT);
+ CreateIvmTrigger(rte->relid, matviewOid, matviewname, TRIGGER_TYPE_DELETE);
+ CreateIvmTrigger(rte->relid, matviewOid, matviewname, TRIGGER_TYPE_UPDATE);
+ }
+ else
+ elog(ERROR, "unsupported RTE kind: %d", (int) rte->rtekind);
+ }
+ else if (IsA(jtnode, FromExpr))
+ {
+ FromExpr *f = (FromExpr *) jtnode;
+ ListCell *l;
+
+ foreach(l, f->fromlist)
+ CreateIvmTriggersOnBaseTables(qry, lfirst(l), matviewOid, matviewname);
+ }
+ else if (IsA(jtnode, JoinExpr))
+ {
+ JoinExpr *j = (JoinExpr *) jtnode;
+
+ CreateIvmTriggersOnBaseTables(qry, j->larg, matviewOid, matviewname);
+ CreateIvmTriggersOnBaseTables(qry, j->rarg, matviewOid, matviewname);
+ }
+ else
+ elog(ERROR, "unrecognized node type: %d", (int) nodeTag(jtnode));
+}
+
/*
* GetIntoRelEFlags --- compute executor flags needed for CREATE TABLE AS
*
@@ -547,6 +706,11 @@ intorel_startup(DestReceiver *self, int operation, TupleDesc typeinfo)
if (is_matview && !into->skipData)
SetMatViewPopulatedState(intoRelationDesc, true);
+ /*
+ * Mark relisivm field, if it's a matview and into->ivm is true.
+ */
+ if (is_matview && into->ivm)
+ SetMatViewIVMState(intoRelationDesc, true);
/*
* Fill private fields of myState for use by later routines
*/
@@ -619,3 +783,74 @@ intorel_destroy(DestReceiver *self)
{
pfree(self);
}
+
+static void
+CreateIvmTrigger(Oid relOid, Oid viewOid, char *matviewname, int16 type)
+{
+ CreateTrigStmt *ivm_trigger;
+ List *transitionRels = NIL;
+ ObjectAddress address, refaddr;
+
+ refaddr.classId = RelationRelationId;
+ refaddr.objectId = viewOid;
+ refaddr.objectSubId = 0;
+
+
+ ivm_trigger = makeNode(CreateTrigStmt);
+ ivm_trigger->relation = NULL;
+ ivm_trigger->row = false;
+ ivm_trigger->timing = TRIGGER_TYPE_AFTER;
+
+ ivm_trigger->events = type;
+
+ switch (type)
+ {
+ case TRIGGER_TYPE_INSERT:
+ ivm_trigger->trigname = "IVM_trigger_ins";
+ break;
+ case TRIGGER_TYPE_DELETE:
+ ivm_trigger->trigname = "IVM_trigger_del";
+ break;
+ case TRIGGER_TYPE_UPDATE:
+ ivm_trigger->trigname = "IVM_trigger_upd";
+ break;
+ }
+
+ if (type == TRIGGER_TYPE_INSERT || type == TRIGGER_TYPE_UPDATE)
+ {
+ TriggerTransition *n = makeNode(TriggerTransition);
+ n->name = "ivm_newtable";
+ n->isNew = true;
+ n->isTable = true;
+
+ transitionRels = lappend(transitionRels, n);
+ }
+ if (type == TRIGGER_TYPE_DELETE || type == TRIGGER_TYPE_UPDATE)
+ {
+ TriggerTransition *n = makeNode(TriggerTransition);
+ n->name = "ivm_oldtable";
+ n->isNew = false;
+ n->isTable = true;
+
+ transitionRels = lappend(transitionRels, n);
+ }
+
+ ivm_trigger->funcname = SystemFuncName("IVM_immediate_maintenance");
+
+ ivm_trigger->columns = NIL;
+ ivm_trigger->transitionRels = transitionRels;
+ ivm_trigger->whenClause = NULL;
+ ivm_trigger->isconstraint = false;
+ ivm_trigger->deferrable = false;
+ ivm_trigger->initdeferred = false;
+ ivm_trigger->constrrel = NULL;
+ ivm_trigger->args = list_make1(makeString(matviewname));
+
+ address = CreateTrigger(ivm_trigger, NULL, relOid, InvalidOid, InvalidOid,
+ InvalidOid, InvalidOid, InvalidOid, NULL, true, false);
+
+ recordDependencyOn(&address, &refaddr, DEPENDENCY_AUTO);
+
+ /* Make changes-so-far visible */
+ CommandCounterIncrement();
+}
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index 537d0e8cef..6b023e6832 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -46,6 +46,15 @@
#include "utils/snapmgr.h"
#include "utils/syscache.h"
+#include "utils/regproc.h"
+#include "nodes/makefuncs.h"
+#include "parser/parse_clause.h"
+#include "parser/parse_func.h"
+#include "nodes/print.h"
+#include "catalog/pg_type_d.h"
+#include "optimizer/optimizer.h"
+#include "commands/defrem.h"
+
typedef struct
{
@@ -65,7 +74,8 @@ static bool transientrel_receive(TupleTableSlot *slot, DestReceiver *self);
static void transientrel_shutdown(DestReceiver *self);
static void transientrel_destroy(DestReceiver *self);
static uint64 refresh_matview_datafill(DestReceiver *dest, Query *query,
- const char *queryString);
+ QueryEnvironment *queryEnv,
+ const char *queryString);
static char *make_temptable_name_n(char *tempname, int n);
static void refresh_by_match_merge(Oid matviewOid, Oid tempOid, Oid relowner,
int save_sec_context);
@@ -74,6 +84,9 @@ static bool is_usable_unique_index(Relation indexRel);
static void OpenMatViewIncrementalMaintenance(void);
static void CloseMatViewIncrementalMaintenance(void);
+static void apply_delta(Oid matviewOid, Oid tempOid_new, Oid tempOid_old,
+ Query *query, Oid relowner, int save_sec_context);
+
/*
* SetMatViewPopulatedState
* Mark a materialized view as populated, or not.
@@ -114,6 +127,46 @@ SetMatViewPopulatedState(Relation relation, bool newstate)
CommandCounterIncrement();
}
+/*
+ * SetMatViewIVMState
+ * Mark a materialized view as IVM, or not.
+ *
+ * NOTE: caller must be holding an appropriate lock on the relation.
+ */
+void
+SetMatViewIVMState(Relation relation, bool newstate)
+{
+ Relation pgrel;
+ HeapTuple tuple;
+
+ Assert(relation->rd_rel->relkind == RELKIND_MATVIEW);
+
+ /*
+ * Update relation's pg_class entry. Crucial side-effect: other backends
+ * (and this one too!) are sent SI message to make them rebuild relcache
+ * entries.
+ */
+ pgrel = table_open(RelationRelationId, RowExclusiveLock);
+ tuple = SearchSysCacheCopy1(RELOID,
+ ObjectIdGetDatum(RelationGetRelid(relation)));
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for relation %u",
+ RelationGetRelid(relation));
+
+ ((Form_pg_class) GETSTRUCT(tuple))->relisivm = newstate;
+
+ CatalogTupleUpdate(pgrel, &tuple->t_self, tuple);
+
+ heap_freetuple(tuple);
+ table_close(pgrel, RowExclusiveLock);
+
+ /*
+ * Advance command counter to make the updated pg_class row locally
+ * visible.
+ */
+ CommandCounterIncrement();
+}
+
/*
* ExecRefreshMatView -- execute a REFRESH MATERIALIZED VIEW command
*
@@ -311,7 +364,7 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
/* Generate the data, if wanted. */
if (!stmt->skipData)
- processed = refresh_matview_datafill(dest, dataQuery, queryString);
+ processed = refresh_matview_datafill(dest, dataQuery, NULL, queryString);
/* Make the matview match the newly generated data. */
if (concurrent)
@@ -369,6 +422,7 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
*/
static uint64
refresh_matview_datafill(DestReceiver *dest, Query *query,
+ QueryEnvironment *queryEnv,
const char *queryString)
{
List *rewritten;
@@ -405,7 +459,7 @@ refresh_matview_datafill(DestReceiver *dest, Query *query,
/* Create a QueryDesc, redirecting output to our tuple receiver */
queryDesc = CreateQueryDesc(plan, queryString,
GetActiveSnapshot(), InvalidSnapshot,
- dest, NULL, NULL, 0);
+ dest, NULL, queryEnv ? queryEnv: NULL, 0);
/* call ExecutorStart to prepare the plan for execution */
ExecutorStart(queryDesc, 0);
@@ -926,3 +980,622 @@ CloseMatViewIncrementalMaintenance(void)
matview_maintenance_depth--;
Assert(matview_maintenance_depth >= 0);
}
+
+/*
+ * IVM trigger function
+ */
+
+Datum
+IVM_immediate_maintenance(PG_FUNCTION_ARGS)
+{
+ TriggerData *trigdata = (TriggerData *) fcinfo->context;
+ Relation rel;
+ Oid relid;
+ Oid matviewOid;
+ Query *query, *old_delta_qry, *new_delta_qry;
+ char* matviewname = trigdata->tg_trigger->tgargs[0];
+ List *names;
+ Relation matviewRel;
+ int old_depth = matview_maintenance_depth;
+
+ Oid tableSpace;
+ Oid relowner;
+ Oid OIDDelta_new = InvalidOid;
+ Oid OIDDelta_old = InvalidOid;
+ DestReceiver *dest_new = NULL, *dest_old = NULL;
+ char relpersistence;
+ Oid save_userid;
+ int save_sec_context;
+ int save_nestlevel;
+
+ ParseState *pstate;
+ QueryEnvironment *queryEnv = create_queryEnv();
+
+ Const *dmy_arg = makeConst(INT4OID,
+ -1,
+ InvalidOid,
+ sizeof(int32),
+ Int32GetDatum(1),
+ false,
+ true); /* pass by value */
+
+ /* Create a dummy ParseState for addRangeTableEntryForENR */
+ pstate = make_parsestate(NULL);
+ pstate->p_queryEnv = queryEnv;
+
+ names = stringToQualifiedNameList(matviewname);
+
+ /*
+ * Wait for concurrent transactions which update this materialized view at READ COMMITED.
+ * This is needed to see changes commited in othre transactions. No wait and raise an error
+ * at REPEATABLE READ or SERIALIZABLE to prevent anormal update of matviews.
+ * XXX: dead-lock is possible here.
+ */
+ if (!IsolationUsesXactSnapshot())
+ matviewOid = RangeVarGetRelid(makeRangeVarFromNameList(names), ExclusiveLock, true);
+ else
+ matviewOid = RangeVarGetRelidExtended(makeRangeVarFromNameList(names), ExclusiveLock, RVR_MISSING_OK | RVR_NOWAIT, NULL, NULL);
+
+ matviewRel = table_open(matviewOid, NoLock);
+
+ /*
+ * Get and push the latast snapshot to see any changes which is commited during waiting in
+ * other transactions at READ COMMITTED level.
+ * XXX: Is this safe?
+ */
+ PushActiveSnapshot(GetTransactionSnapshot());
+
+ /* Make sure it is a materialized view. */
+ if (matviewRel->rd_rel->relkind != RELKIND_MATVIEW)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("\"%s\" is not a materialized view",
+ RelationGetRelationName(matviewRel))));
+
+ rel = trigdata->tg_relation;
+ relid = rel->rd_id;
+
+ query = get_view_query(matviewRel);
+
+ new_delta_qry = copyObject(query);
+ old_delta_qry = copyObject(query);
+
+ if (trigdata->tg_newtable)
+ {
+ RangeTblEntry *rte;
+ ListCell *lc;
+
+ TargetEntry *tle;
+ Node *node;
+ FuncCall *fn;
+
+ EphemeralNamedRelation enr =
+ palloc(sizeof(EphemeralNamedRelationData));
+
+ enr->md.name = trigdata->tg_trigger->tgnewtable;
+ enr->md.reliddesc = trigdata->tg_relation->rd_id;
+ enr->md.tupdesc = NULL;
+ enr->md.enrtype = ENR_NAMED_TUPLESTORE;
+ enr->md.enrtuples = tuplestore_tuple_count(trigdata->tg_newtable);
+ enr->reldata = trigdata->tg_newtable;
+ register_ENR(queryEnv, enr);
+
+ rte = addRangeTableEntryForENR(pstate, makeRangeVar(NULL, enr->md.name, -1), true);
+ new_delta_qry->rtable = lappend(new_delta_qry->rtable, rte);
+
+ foreach(lc, new_delta_qry->rtable)
+ {
+ RangeTblEntry *r = (RangeTblEntry*) lfirst(lc);
+ if (r->relid == relid)
+ {
+ lfirst(lc) = rte;
+ break;
+ }
+ }
+
+ if (query->hasAggs)
+ {
+ ListCell *lc;
+ List *agg_counts = NIL;
+ AttrNumber next_resno = list_length(query->targetList) + 1;
+ Node *node;
+
+ foreach(lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ TargetEntry *tle_count;
+
+ if (IsA(tle->expr, Aggref))
+ {
+ Aggref *aggref = (Aggref *) tle->expr;
+ const char *aggname = get_func_name(aggref->aggfnoid);
+
+ if (strcmp(aggname, "count") != 0)
+ {
+ fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, list_make1(dmy_arg), NULL, fn, false, -1);
+ ((Aggref *)node)->args = aggref->args;
+
+ tle_count = makeTargetEntry((Expr *) node,
+ next_resno,
+ pstrdup(makeObjectName("__ivm_count",tle->resname, "_")),
+ false);
+ agg_counts = lappend(agg_counts, tle_count);
+ next_resno++;
+ }
+ }
+
+ }
+ new_delta_qry->targetList = list_concat(new_delta_qry->targetList, agg_counts);
+ }
+
+ fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+ fn->agg_star = true;
+ if (!new_delta_qry->groupClause && !new_delta_qry->hasAggs)
+ new_delta_qry->groupClause = transformDistinctClause(NULL, &new_delta_qry->targetList, new_delta_qry->sortClause, false);
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(new_delta_qry->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ new_delta_qry->targetList = lappend(new_delta_qry->targetList, tle);
+ new_delta_qry->hasAggs = true;
+ }
+
+ if (trigdata->tg_oldtable)
+ {
+ RangeTblEntry *rte;
+ ListCell *lc;
+
+ TargetEntry *tle;
+ Node *node;
+ FuncCall *fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+
+ EphemeralNamedRelation enr =
+ palloc(sizeof(EphemeralNamedRelationData));
+
+ enr->md.name = trigdata->tg_trigger->tgoldtable;
+ enr->md.reliddesc = trigdata->tg_relation->rd_id;
+ enr->md.tupdesc = NULL;
+ enr->md.enrtype = ENR_NAMED_TUPLESTORE;
+ enr->md.enrtuples = tuplestore_tuple_count(trigdata->tg_oldtable);
+ enr->reldata = trigdata->tg_oldtable;
+ register_ENR(queryEnv, enr);
+
+ rte = addRangeTableEntryForENR(pstate, makeRangeVar(NULL, enr->md.name, -1), true);
+ old_delta_qry->rtable = lappend(old_delta_qry->rtable, rte);
+
+ foreach(lc, old_delta_qry->rtable)
+ {
+ RangeTblEntry *r = (RangeTblEntry*) lfirst(lc);
+ if (r->relid == relid)
+ {
+ lfirst(lc) = rte;
+ break;
+ }
+ }
+
+ if (query->hasAggs)
+ {
+ ListCell *lc;
+ List *agg_counts = NIL;
+ AttrNumber next_resno = list_length(query->targetList) + 1;
+ Node *node;
+
+ foreach(lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ TargetEntry *tle_count;
+
+ if (IsA(tle->expr, Aggref))
+ {
+ Aggref *aggref = (Aggref *) tle->expr;
+ const char *aggname = get_func_name(aggref->aggfnoid);
+
+ if (strcmp(aggname, "count") != 0)
+ {
+ fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, list_make1(dmy_arg), NULL, fn, false, -1);
+ ((Aggref *)node)->args = aggref->args;
+
+ tle_count = makeTargetEntry((Expr *) node,
+ next_resno,
+ pstrdup(makeObjectName("__ivm_count",tle->resname, "_")),
+ false);
+ agg_counts = lappend(agg_counts, tle_count);
+ next_resno++;
+ }
+ }
+
+ }
+ old_delta_qry->targetList = list_concat(old_delta_qry->targetList, agg_counts);
+ }
+
+ fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+ fn->agg_star = true;
+
+ if (!old_delta_qry->groupClause && !old_delta_qry->hasAggs)
+ old_delta_qry->groupClause = transformDistinctClause(NULL, &old_delta_qry->targetList, old_delta_qry->sortClause, false);
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+ tle = makeTargetEntry((Expr *) node,
+ list_length(old_delta_qry->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ old_delta_qry->targetList = lappend(old_delta_qry->targetList, tle);
+ old_delta_qry->hasAggs = true;
+ }
+
+
+ /*
+ * Check for active uses of the relation in the current transaction, such
+ * as open scans.
+ *
+ * NB: We count on this to protect us against problems with refreshing the
+ * data using TABLE_INSERT_FROZEN.
+ */
+ CheckTableNotInUse(matviewRel, "REFRESH MATERIALIZED VIEW");
+
+ relowner = matviewRel->rd_rel->relowner;
+
+ /*
+ * Switch to the owner's userid, so that any functions are run as that
+ * user. Also arrange to make GUC variable changes local to this command.
+ * Don't lock it down too tight to create a temporary table just yet. We
+ * will switch modes when we are about to execute user code.
+ */
+ GetUserIdAndSecContext(&save_userid, &save_sec_context);
+ SetUserIdAndSecContext(relowner,
+ save_sec_context | SECURITY_LOCAL_USERID_CHANGE);
+ save_nestlevel = NewGUCNestLevel();
+
+ tableSpace = GetDefaultTablespace(RELPERSISTENCE_TEMP, false);
+ relpersistence = RELPERSISTENCE_TEMP;
+
+ /*
+ * Create the transient table that will receive the regenerated data. Lock
+ * it against access by any other process until commit (by which time it
+ * will be gone).
+ */
+ if (trigdata->tg_newtable)
+ {
+ OIDDelta_new = make_new_heap(matviewOid, tableSpace, relpersistence,
+ ExclusiveLock);
+ LockRelationOid(OIDDelta_new, AccessExclusiveLock);
+ dest_new = CreateTransientRelDestReceiver(OIDDelta_new);
+ }
+ if (trigdata->tg_oldtable)
+ {
+ if (trigdata->tg_newtable)
+ OIDDelta_old = make_new_heap(OIDDelta_new, tableSpace, relpersistence,
+ ExclusiveLock);
+ else
+ OIDDelta_old = make_new_heap(matviewOid, tableSpace, relpersistence,
+ ExclusiveLock);
+ LockRelationOid(OIDDelta_old, AccessExclusiveLock);
+ dest_old = CreateTransientRelDestReceiver(OIDDelta_old);
+ }
+
+ /*
+ * Now lock down security-restricted operations.
+ */
+ SetUserIdAndSecContext(relowner,
+ save_sec_context | SECURITY_RESTRICTED_OPERATION);
+
+ /* Generate the data. */
+ if (trigdata->tg_newtable)
+ refresh_matview_datafill(dest_new, new_delta_qry, queryEnv, NULL);
+ if (trigdata->tg_oldtable)
+ refresh_matview_datafill(dest_old, old_delta_qry, queryEnv, NULL);
+
+ PG_TRY();
+ {
+ apply_delta(matviewOid, OIDDelta_new, OIDDelta_old,
+ query, relowner, save_sec_context);
+ }
+ PG_CATCH();
+ {
+ matview_maintenance_depth = old_depth;
+ PG_RE_THROW();
+ }
+ PG_END_TRY();
+
+ /* Pop the original snapshot. */
+ PopActiveSnapshot();
+
+ table_close(matviewRel, NoLock);
+
+ /* Roll back any GUC changes */
+ AtEOXact_GUC(false, save_nestlevel);
+
+ /* Restore userid and security context */
+ SetUserIdAndSecContext(save_userid, save_sec_context);
+
+ return PointerGetDatum(NULL);
+}
+
+static void
+apply_delta(Oid matviewOid, Oid tempOid_new, Oid tempOid_old,
+ Query *query, Oid relowner, int save_sec_context)
+{
+ StringInfoData querybuf;
+ StringInfoData mvatts_buf, diffatts_buf;
+ StringInfoData mv_gkeys_buf, diff_gkeys_buf, updt_gkeys_buf;
+ StringInfoData diff_aggs_buf, update_aggs_old, update_aggs_new;
+ Relation matviewRel;
+ Relation tempRel_new = NULL, tempRel_old = NULL;
+ char *matviewname;
+ char *tempname_new = NULL, *tempname_old = NULL;
+ ListCell *lc;
+ char *sep, *sep_agg;
+ bool with_group = query->groupClause != NULL;
+
+
+ initStringInfo(&querybuf);
+ matviewRel = table_open(matviewOid, NoLock);
+ matviewname = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(matviewRel)),
+ RelationGetRelationName(matviewRel));
+
+ if (OidIsValid(tempOid_new))
+ {
+ tempRel_new = table_open(tempOid_new, NoLock);
+ tempname_new = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(tempRel_new)),
+ RelationGetRelationName(tempRel_new));
+ }
+ if (OidIsValid(tempOid_old))
+ {
+ tempRel_old = table_open(tempOid_old, NoLock);
+ tempname_old = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(tempRel_old)),
+ RelationGetRelationName(tempRel_old));
+ }
+
+ initStringInfo(&mvatts_buf);
+ initStringInfo(&diffatts_buf);
+ initStringInfo(&diff_aggs_buf);
+ initStringInfo(&update_aggs_old);
+ initStringInfo(&update_aggs_new);
+
+ sep = "";
+ sep_agg= "";
+ foreach (lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);;
+
+ if (tle->resjunk)
+ continue;
+
+ appendStringInfo(&mvatts_buf, "%s", sep);
+ appendStringInfo(&diffatts_buf, "%s", sep);
+ sep = ", ";
+
+ appendStringInfo(&mvatts_buf, "%s", quote_qualified_identifier("mv", tle->resname));
+ appendStringInfo(&diffatts_buf, "%s", quote_qualified_identifier("diff", tle->resname));
+ if (query->hasAggs && IsA(tle->expr, Aggref))
+ {
+ Aggref *aggref = (Aggref *) tle->expr;
+ const char *aggname = get_func_name(aggref->aggfnoid);
+
+ appendStringInfo(&update_aggs_old, "%s", sep_agg);
+ appendStringInfo(&update_aggs_new, "%s", sep_agg);
+ appendStringInfo(&diff_aggs_buf, "%s", sep_agg);
+
+ sep_agg = ", ";
+
+ if (!strcmp(aggname, "count"))
+ {
+ appendStringInfo(&update_aggs_old,
+ "%s = %s - %s",
+ quote_qualified_identifier(NULL, tle->resname),
+ quote_qualified_identifier("mv", tle->resname),
+ quote_qualified_identifier("t", tle->resname)
+ );
+ appendStringInfo(&update_aggs_new,
+ "%s = %s + %s",
+ quote_qualified_identifier(NULL, tle->resname),
+ quote_qualified_identifier("mv", tle->resname),
+ quote_qualified_identifier("diff", tle->resname)
+ );
+
+ appendStringInfo(&diff_aggs_buf, "%s",
+ quote_qualified_identifier("diff", tle->resname)
+ );
+ }
+ else if (!strcmp(aggname, "sum"))
+ {
+ appendStringInfo(&update_aggs_old,
+ "%s = CASE WHEN %s = %s THEN NULL ELSE COALESCE(%s,0) - COALESCE(%s, 0) END, "
+ "%s = %s - %s",
+ quote_qualified_identifier(NULL, tle->resname),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("t", makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("mv", tle->resname),
+ quote_qualified_identifier("t", tle->resname),
+ quote_qualified_identifier(NULL, makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("t", makeObjectName("__ivm_count",tle->resname,"_"))
+ );
+ appendStringInfo(&update_aggs_new,
+ "%s = CASE WHEN %s = 0 AND NULL = 0 THEN %s ELSE COALESCE(%s,0) + COALESCE(%s, 0) END, "
+ "%s = %s + %s",
+ quote_qualified_identifier(NULL, tle->resname),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("diff", makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("mv", tle->resname),
+ quote_qualified_identifier("diff", tle->resname),
+ quote_qualified_identifier(NULL, makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("diff", makeObjectName("__ivm_count",tle->resname,"_"))
+ );
+
+ appendStringInfo(&diff_aggs_buf, "%s, %s",
+ quote_qualified_identifier("diff", tle->resname),
+ quote_qualified_identifier("diff", makeObjectName("__ivm_count",tle->resname,"_"))
+ );
+ }
+ else
+ elog(ERROR, "unsupported aggregate function: %s", aggname);
+
+ }
+ }
+
+ if (query->hasAggs)
+ {
+ initStringInfo(&mv_gkeys_buf);
+ initStringInfo(&diff_gkeys_buf);
+ initStringInfo(&updt_gkeys_buf);
+
+ if (with_group)
+ {
+ sep_agg= "";
+ foreach (lc, query->groupClause)
+ {
+ SortGroupClause *sgcl = (SortGroupClause *) lfirst(lc);
+ TargetEntry *tle = get_sortgroupclause_tle(sgcl, query->targetList);
+
+ appendStringInfo(&mv_gkeys_buf, "%s", sep_agg);
+ appendStringInfo(&diff_gkeys_buf, "%s", sep_agg);
+ appendStringInfo(&updt_gkeys_buf, "%s", sep_agg);
+
+ sep_agg = ", ";
+
+ appendStringInfo(&mv_gkeys_buf, "%s", quote_qualified_identifier("mv", tle->resname));
+ appendStringInfo(&diff_gkeys_buf, "%s", quote_qualified_identifier("diff", tle->resname));
+ appendStringInfo(&updt_gkeys_buf, "%s", quote_qualified_identifier("updt", tle->resname));
+ }
+ }
+ else
+ {
+ appendStringInfo(&mv_gkeys_buf, "1");
+ appendStringInfo(&diff_gkeys_buf, "1");
+ appendStringInfo(&updt_gkeys_buf, "1");
+ }
+ }
+
+ /* Open SPI context. */
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+
+ /* Analyze the temp table with the new contents. */
+ if (tempname_new)
+ {
+ appendStringInfo(&querybuf, "ANALYZE %s", tempname_new);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ if (tempname_old)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf, "ANALYZE %s", tempname_old);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+
+ SetUserIdAndSecContext(relowner,
+ save_sec_context | SECURITY_LOCAL_USERID_CHANGE);
+
+ OpenMatViewIncrementalMaintenance();
+
+ if (query->hasAggs)
+ {
+ if (tempname_old)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH t AS ("
+ " SELECT diff.__ivm_count__, (diff.__ivm_count__ = mv.__ivm_count__) AS for_dlt, mv.ctid"
+ ", %s"
+ " FROM %s AS mv, %s AS diff WHERE (%s) = (%s)"
+ "), updt AS ("
+ " UPDATE %s AS mv SET __ivm_count__ = mv.__ivm_count__ - t.__ivm_count__"
+ ", %s "
+ " FROM t WHERE mv.ctid = t.ctid AND NOT for_dlt"
+ ") DELETE FROM %s AS mv USING t WHERE mv.ctid = t.ctid AND for_dlt;",
+ diff_aggs_buf.data,
+ matviewname, tempname_old, mv_gkeys_buf.data, diff_gkeys_buf.data,
+ matviewname, update_aggs_old.data, matviewname);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_DELETE)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ if (tempname_new)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH updt AS ("
+ " UPDATE %s AS mv SET __ivm_count__ = mv.__ivm_count__ + diff.__ivm_count__"
+ ", %s "
+ " FROM %s AS diff WHERE (%s) = (%s)"
+ " RETURNING %s"
+ ") INSERT INTO %s (SELECT * FROM %s AS diff WHERE (%s) NOT IN (SELECT %s FROM updt));",
+ matviewname, update_aggs_new.data, tempname_new,
+ mv_gkeys_buf.data, diff_gkeys_buf.data, diff_gkeys_buf.data,
+ matviewname, tempname_new, diff_gkeys_buf.data, updt_gkeys_buf.data);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ }
+ else
+ {
+ if (tempname_old)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH t AS ("
+ " SELECT diff.__ivm_count__, (diff.__ivm_count__ = mv.__ivm_count__) AS for_dlt, mv.ctid"
+ " FROM %s AS mv, %s AS diff WHERE (%s) = (%s)"
+ "), updt AS ("
+ " UPDATE %s AS mv SET __ivm_count__ = mv.__ivm_count__ - t.__ivm_count__"
+ " FROM t WHERE mv.ctid = t.ctid AND NOT for_dlt"
+ ") DELETE FROM %s AS mv USING t WHERE mv.ctid = t.ctid AND for_dlt;",
+ matviewname, tempname_old, mvatts_buf.data, diffatts_buf.data, matviewname, matviewname);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_DELETE)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ if (tempname_new)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH updt AS ("
+ " UPDATE %s AS mv SET __ivm_count__ = mv.__ivm_count__ + diff.__ivm_count__"
+ " FROM %s AS diff WHERE (%s) = (%s)"
+ " RETURNING %s"
+ ") INSERT INTO %s (SELECT * FROM %s AS diff WHERE (%s) NOT IN (SELECT * FROM updt));",
+ matviewname, tempname_new, mvatts_buf.data, diffatts_buf.data, diffatts_buf.data, matviewname, tempname_new, diffatts_buf.data);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ }
+
+ /* We're done maintaining the materialized view. */
+ CloseMatViewIncrementalMaintenance();
+
+ if (OidIsValid(tempOid_new))
+ table_close(tempRel_new, NoLock);
+ if (OidIsValid(tempOid_old))
+ table_close(tempRel_old, NoLock);
+
+ table_close(matviewRel, NoLock);
+
+ /* Clean up temp tables. */
+ if (OidIsValid(tempOid_new))
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf, "DROP TABLE %s", tempname_new);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ if (OidIsValid(tempOid_old))
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf, "DROP TABLE %s", tempname_old);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+
+ /* Close SPI context. */
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+}
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 78deade89b..8262ac039b 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2361,6 +2361,7 @@ _copyRangeTblEntry(const RangeTblEntry *from)
COPY_SCALAR_FIELD(relkind);
COPY_SCALAR_FIELD(rellockmode);
COPY_NODE_FIELD(tablesample);
+ COPY_SCALAR_FIELD(relisivm);
COPY_NODE_FIELD(subquery);
COPY_SCALAR_FIELD(security_barrier);
COPY_SCALAR_FIELD(jointype);
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 4f2ebe5118..608d477bd5 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2641,6 +2641,7 @@ _equalRangeTblEntry(const RangeTblEntry *a, const RangeTblEntry *b)
COMPARE_SCALAR_FIELD(relkind);
COMPARE_SCALAR_FIELD(rellockmode);
COMPARE_NODE_FIELD(tablesample);
+ COMPARE_SCALAR_FIELD(relisivm);
COMPARE_NODE_FIELD(subquery);
COMPARE_SCALAR_FIELD(security_barrier);
COMPARE_SCALAR_FIELD(jointype);
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 8400dd319e..7897dd9c57 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -3042,6 +3042,7 @@ _outRangeTblEntry(StringInfo str, const RangeTblEntry *node)
WRITE_CHAR_FIELD(relkind);
WRITE_INT_FIELD(rellockmode);
WRITE_NODE_FIELD(tablesample);
+ WRITE_BOOL_FIELD(relisivm);
break;
case RTE_SUBQUERY:
WRITE_NODE_FIELD(subquery);
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 6c2626ee62..aa01e205c3 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1366,6 +1366,7 @@ _readRangeTblEntry(void)
READ_CHAR_FIELD(relkind);
READ_INT_FIELD(rellockmode);
READ_NODE_FIELD(tablesample);
+ READ_BOOL_FIELD(relisivm);
break;
case RTE_SUBQUERY:
READ_NODE_FIELD(subquery);
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 8311b1dd46..7cae68218b 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -418,6 +418,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -645,7 +646,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -4054,30 +4055,32 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->relkind = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->relkind = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -4094,9 +4097,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -15128,6 +15136,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDEX
| INDEXES
| INHERIT
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
index 77a48b039d..3903b8ffd5 100644
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -37,6 +37,7 @@
#include "utils/syscache.h"
#include "utils/varlena.h"
+#include "commands/matview.h"
#define MAX_FUZZY_DISTANCE 3
@@ -56,9 +57,10 @@ static void expandTupleDesc(TupleDesc tupdesc, Alias *eref,
int count, int offset,
int rtindex, int sublevels_up,
int location, bool include_dropped,
- List **colnames, List **colvars);
+ List **colnames, List **colvars, bool is_ivm);
static int specialAttNum(const char *attname);
static bool isQueryUsingTempRelation_walker(Node *node, void *context);
+static bool isIvmColumn(const char *s);
/*
@@ -1238,6 +1240,7 @@ addRangeTableEntry(ParseState *pstate,
rte->relid = RelationGetRelid(rel);
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -1317,6 +1320,7 @@ addRangeTableEntryForRelation(ParseState *pstate,
rte->relid = RelationGetRelid(rel);
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -2315,7 +2319,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
expandTupleDesc(tupdesc, rte->eref,
rtfunc->funccolcount, atts_done,
rtindex, sublevels_up, location,
- include_dropped, colnames, colvars);
+ include_dropped, colnames, colvars, false);
}
else if (functypclass == TYPEFUNC_SCALAR)
{
@@ -2567,10 +2571,19 @@ expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up,
expandTupleDesc(rel->rd_att, eref, rel->rd_att->natts, 0,
rtindex, sublevels_up,
location, include_dropped,
- colnames, colvars);
+ colnames, colvars, RelationIsIVM(rel));
relation_close(rel, AccessShareLock);
}
+static bool
+isIvmColumn(const char *s)
+{
+ char pre[7];
+
+ strlcpy(pre, s, sizeof(pre));
+ return (strcmp(pre, "__ivm_") == 0);
+}
+
/*
* expandTupleDesc -- expandRTE subroutine
*
@@ -2584,7 +2597,7 @@ static void
expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
int rtindex, int sublevels_up,
int location, bool include_dropped,
- List **colnames, List **colvars)
+ List **colnames, List **colvars, bool is_ivm)
{
ListCell *aliascell = list_head(eref->colnames);
int varattno;
@@ -2605,6 +2618,9 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
{
Form_pg_attribute attr = TupleDescAttr(tupdesc, varattno);
+ if (is_ivm && isIvmColumn(NameStr(attr->attname)) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
if (attr->attisdropped)
{
if (include_dropped)
diff --git a/src/backend/rewrite/rewriteDefine.c b/src/backend/rewrite/rewriteDefine.c
index 7df2b6154c..5385a038b1 100644
--- a/src/backend/rewrite/rewriteDefine.c
+++ b/src/backend/rewrite/rewriteDefine.c
@@ -765,7 +765,8 @@ checkRuleResultList(List *targetList, TupleDesc resultDesc, bool isSelect,
attr->atttypmod))));
}
- if (i != resultDesc->natts)
+ /* No check for materialized views since this could have special columns for IVM */
+ if ((!isSelect || requireColumnNameMatch) && i != resultDesc->natts)
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
isSelect ?
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index ea40c28733..6e8dc1f1cd 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -41,6 +41,8 @@
#include "utils/lsyscache.h"
#include "utils/rel.h"
+#include "parser/parser.h"
+#include "commands/matview.h"
/* We use a list of these to detect recursion in RewriteQuery */
typedef struct rewrite_event
@@ -1597,6 +1599,50 @@ ApplyRetrieveRule(Query *parsetree,
if (rule->qual != NULL)
elog(ERROR, "cannot handle qualified ON SELECT rule");
+ if (RelationIsIVM(relation))
+ {
+ rule_action = copyObject(linitial(rule->actions));
+
+ if (!rule_action->distinctClause && !rule_action->groupClause && !rule_action->hasAggs)
+ {
+ StringInfoData str;
+ RawStmt *raw;
+ Query *sub;
+
+ if (rule_action->hasDistinctOn)
+ elog(ERROR, "DISTINCT ON is not supported in IVM");
+
+ initStringInfo(&str);
+ appendStringInfo(&str, "SELECT mv.*, __ivm_count__ FROM %s mv, generate_series(1, mv.__ivm_count__)",
+ quote_qualified_identifier(get_namespace_name(RelationGetNamespace(relation)),
+ RelationGetRelationName(relation)));
+
+ raw = (RawStmt*)linitial(raw_parser(str.data));
+ sub = transformStmt(make_parsestate(NULL),raw->stmt);
+
+ rte = rt_fetch(rt_index, parsetree->rtable);
+
+ rte->rtekind = RTE_SUBQUERY;
+ rte->subquery = sub;
+ rte->security_barrier = RelationIsSecurityView(relation);
+ /* Clear fields that should not be set in a subquery RTE */
+ rte->relid = InvalidOid;
+ rte->relkind = 0;
+ rte->rellockmode = 0;
+ rte->tablesample = NULL;
+ rte->inh = false; /* must not be set for a subquery */
+
+ rte->requiredPerms = 0; /* no permission check on subquery itself */
+ rte->checkAsUser = InvalidOid;
+ rte->selectedCols = NULL;
+ rte->insertedCols = NULL;
+ rte->updatedCols = NULL;
+ rte->extraUpdatedCols = NULL;
+ }
+
+ return parsetree;
+ }
+
if (rt_index == parsetree->resultRelation)
{
/*
@@ -1906,7 +1952,8 @@ fireRIRrules(Query *parsetree, List *activeRIRs)
* In that case this test would need to be postponed till after we've
* opened the rel, so that we could check its state.
*/
- if (rte->relkind == RELKIND_MATVIEW)
+ if (rte->relkind == RELKIND_MATVIEW &&
+ (!rte->relisivm || MatViewIncrementalMaintenanceIsEnabled() || parsetree->commandType != CMD_SELECT))
continue;
/*
diff --git a/src/backend/utils/cache/lsyscache.c b/src/backend/utils/cache/lsyscache.c
index c13c08a97b..296cc53ffd 100644
--- a/src/backend/utils/cache/lsyscache.c
+++ b/src/backend/utils/cache/lsyscache.c
@@ -1844,6 +1844,30 @@ get_rel_relispartition(Oid relid)
return false;
}
+/*
+ * get_rel_relisivm
+ *
+ * Returns the relisivm flag associated with a given relation.
+ */
+bool
+get_rel_relisivm(Oid relid)
+{
+ HeapTuple tp;
+
+ tp = SearchSysCache1(RELOID, ObjectIdGetDatum(relid));
+ if (HeapTupleIsValid(tp))
+ {
+ Form_pg_class reltup = (Form_pg_class) GETSTRUCT(tp);
+ bool result;
+
+ result = reltup->relisivm;
+ ReleaseSysCache(tp);
+ return result;
+ }
+ else
+ return false;
+}
+
/*
* get_rel_tablespace
*
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 2b992d7832..d80bb30696 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -1861,6 +1861,8 @@ formrdesc(const char *relationName, Oid relationReltype,
/* ... and they're always populated, too */
relation->rd_rel->relispopulated = true;
+ /* ... and they're always no ivm, too */
+ relation->rd_rel->relisivm = false;
relation->rd_rel->relreplident = REPLICA_IDENTITY_NOTHING;
relation->rd_rel->relpages = 0;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 97167d2c4b..9654e62a49 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -1491,6 +1491,7 @@ describeOneTableDetails(const char *schemaname,
char relpersistence;
char relreplident;
char *relam;
+ bool isivm;
} tableinfo;
bool show_column_details = false;
@@ -1511,6 +1512,7 @@ describeOneTableDetails(const char *schemaname,
"false AS relhasoids, c.relispartition, %s, c.reltablespace, "
"CASE WHEN c.reloftype = 0 THEN '' ELSE c.reloftype::pg_catalog.regtype::pg_catalog.text END, "
"c.relpersistence, c.relreplident, am.amname\n"
+ ",c.relisivm\n"
"FROM pg_catalog.pg_class c\n "
"LEFT JOIN pg_catalog.pg_class tc ON (c.reltoastrelid = tc.oid)\n"
"LEFT JOIN pg_catalog.pg_am am ON (c.relam = am.oid)\n"
@@ -1687,6 +1689,9 @@ describeOneTableDetails(const char *schemaname,
(char *) NULL : pg_strdup(PQgetvalue(res, 0, 14));
else
tableinfo.relam = NULL;
+ /* TODO: This will supported when sversion >= 130000 (or later). */
+ if (pset.sversion >= 120000)
+ tableinfo.isivm = strcmp(PQgetvalue(res, 0, 15), "t") == 0;
PQclear(res);
res = NULL;
@@ -3260,6 +3265,12 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf, _("Access method: %s"), tableinfo.relam);
printTableAddFooter(&cont, buf.data);
}
+
+ /* Incremental view maintance info */
+ if (verbose && tableinfo.relkind == RELKIND_MATVIEW && tableinfo.isivm)
+ {
+ printTableAddFooter(&cont, _("Incremental view maintenance: yes"));
+ }
}
/* reloptions, if verbose */
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 7dcf342413..c14f0e1f98 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -998,6 +998,7 @@ static const pgsql_thing_t words_after_create[] = {
{"FOREIGN TABLE", NULL, NULL, NULL},
{"FUNCTION", NULL, NULL, Query_for_list_of_functions},
{"GROUP", Query_for_list_of_roles},
+ {"INCREMENTAL MATERIALIZED VIEW", NULL, NULL, &Query_for_list_of_matviews},
{"INDEX", NULL, NULL, &Query_for_list_of_indexes},
{"LANGUAGE", Query_for_list_of_languages},
{"LARGE OBJECT", NULL, NULL, NULL, THING_NO_CREATE | THING_NO_DROP},
@@ -2483,7 +2484,7 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("SEQUENCE", "TABLE", "VIEW");
/* Complete "CREATE UNLOGGED" with TABLE or MATVIEW */
else if (TailMatches("CREATE", "UNLOGGED"))
- COMPLETE_WITH("TABLE", "MATERIALIZED VIEW");
+ COMPLETE_WITH("TABLE", "MATERIALIZED VIEW", "INCREMENTAL MATERIALIZED VIEW");
/* Complete PARTITION BY with RANGE ( or LIST ( or ... */
else if (TailMatches("PARTITION", "BY"))
COMPLETE_WITH("RANGE (", "LIST (", "HASH (");
@@ -2692,13 +2693,16 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("SELECT");
/* CREATE MATERIALIZED VIEW */
- else if (Matches("CREATE", "MATERIALIZED"))
+ else if (Matches("CREATE", "MATERIALIZED") ||
+ Matches("CREATE", "INCREMENTAL", "MATERIALIZED"))
COMPLETE_WITH("VIEW");
- /* Complete CREATE MATERIALIZED VIEW <name> with AS */
- else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny))
+ /* Complete CREATE MATERIALIZED VIEW <name> with AS */
+ else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny) ||
+ Matches("CREATE", "INCREMENTAL", "MATERIALIZED", "VIEW", MatchAny))
COMPLETE_WITH("AS");
/* Complete "CREATE MATERIALIZED VIEW <sth> AS with "SELECT" */
- else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny, "AS"))
+ else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny, "AS") ||
+ Matches("CREATE", "INCREMENTAL", "MATERIALIZED", "VIEW", MatchAny, "AS"))
COMPLETE_WITH("SELECT");
/* CREATE EVENT TRIGGER */
diff --git a/src/include/catalog/pg_class.dat b/src/include/catalog/pg_class.dat
index 9bcf28676d..7deda405af 100644
--- a/src/include/catalog/pg_class.dat
+++ b/src/include/catalog/pg_class.dat
@@ -27,7 +27,7 @@
relpersistence => 'p', relkind => 'r', relnatts => '31', relchecks => '0',
relhasrules => 'f', relhastriggers => 'f', relhassubclass => 'f',
relrowsecurity => 'f', relforcerowsecurity => 'f', relispopulated => 't',
- relreplident => 'n', relispartition => 'f', relfrozenxid => '3',
+ relreplident => 'n', relispartition => 'f', relisivm => 'f', relfrozenxid => '3',
relminmxid => '1', relacl => '_null_', reloptions => '_null_',
relpartbound => '_null_' },
{ oid => '1249',
@@ -37,7 +37,7 @@
relpersistence => 'p', relkind => 'r', relnatts => '25', relchecks => '0',
relhasrules => 'f', relhastriggers => 'f', relhassubclass => 'f',
relrowsecurity => 'f', relforcerowsecurity => 'f', relispopulated => 't',
- relreplident => 'n', relispartition => 'f', relfrozenxid => '3',
+ relreplident => 'n', relispartition => 'f', relisivm => 'f', relfrozenxid => '3',
relminmxid => '1', relacl => '_null_', reloptions => '_null_',
relpartbound => '_null_' },
{ oid => '1255',
@@ -47,17 +47,17 @@
relpersistence => 'p', relkind => 'r', relnatts => '29', relchecks => '0',
relhasrules => 'f', relhastriggers => 'f', relhassubclass => 'f',
relrowsecurity => 'f', relforcerowsecurity => 'f', relispopulated => 't',
- relreplident => 'n', relispartition => 'f', relfrozenxid => '3',
+ relreplident => 'n', relispartition => 'f', relisivm => 'f', relfrozenxid => '3',
relminmxid => '1', relacl => '_null_', reloptions => '_null_',
relpartbound => '_null_' },
{ oid => '1259',
relname => 'pg_class', reltype => 'pg_class', relam => 'heap',
relfilenode => '0', relpages => '0', reltuples => '0', relallvisible => '0',
reltoastrelid => '0', relhasindex => 'f', relisshared => 'f',
- relpersistence => 'p', relkind => 'r', relnatts => '33', relchecks => '0',
+ relpersistence => 'p', relkind => 'r', relnatts => '34', relchecks => '0',
relhasrules => 'f', relhastriggers => 'f', relhassubclass => 'f',
relrowsecurity => 'f', relforcerowsecurity => 'f', relispopulated => 't',
- relreplident => 'n', relispartition => 'f', relfrozenxid => '3',
+ relreplident => 'n', relispartition => 'f', relisivm => 'f', relfrozenxid => '3',
relminmxid => '1', relacl => '_null_', reloptions => '_null_',
relpartbound => '_null_' },
diff --git a/src/include/catalog/pg_class.h b/src/include/catalog/pg_class.h
index 090b6ba907..ff535f5504 100644
--- a/src/include/catalog/pg_class.h
+++ b/src/include/catalog/pg_class.h
@@ -116,6 +116,9 @@ CATALOG(pg_class,1259,RelationRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(83,Relat
/* is relation a partition? */
bool relispartition;
+ /* is relation a matview with ivm? */
+ bool relisivm;
+
/* heap for rewrite during DDL, link to original rel */
Oid relrewrite BKI_DEFAULT(0);
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 87335248a0..5de996a72c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10677,4 +10677,9 @@
proname => 'pg_partition_root', prorettype => 'regclass',
proargtypes => 'regclass', prosrc => 'pg_partition_root' },
+# IVM
+{ oid => '784', descr => 'ivm trigger',
+ proname => 'IVM_immediate_maintenance', provolatile => 'v', prorettype => 'trigger',
+ proargtypes => '', prosrc => 'IVM_immediate_maintenance' },
+
]
diff --git a/src/include/commands/matview.h b/src/include/commands/matview.h
index edf04bf415..8dd8193d9c 100644
--- a/src/include/commands/matview.h
+++ b/src/include/commands/matview.h
@@ -23,6 +23,8 @@
extern void SetMatViewPopulatedState(Relation relation, bool newstate);
+extern void SetMatViewIVMState(Relation relation, bool newstate);
+
extern ObjectAddress ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
ParamListInfo params, char *completionTag);
@@ -30,4 +32,6 @@ extern DestReceiver *CreateTransientRelDestReceiver(Oid oid);
extern bool MatViewIncrementalMaintenanceIsEnabled(void);
+extern Datum IVM_immediate_maintenance(PG_FUNCTION_ARGS);
+
#endif /* MATVIEW_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 94ded3c135..5c8a5ae3ca 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1002,6 +1002,7 @@ typedef struct RangeTblEntry
char relkind; /* relation kind (see pg_class.relkind) */
int rellockmode; /* lock level that query requires on the rel */
struct TableSampleClause *tablesample; /* sampling info, or NULL */
+ bool relisivm;
/*
* Fields valid for a subquery RTE (else NULL):
@@ -2059,6 +2060,7 @@ typedef struct CreateStmt
char *tablespacename; /* table space to use, or NULL */
char *accessMethod; /* table access method */
bool if_not_exists; /* just do nothing if it already exists? */
+ bool ivm; /* incremental view maintenance is used by materialized view */
} CreateStmt;
/* ----------
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 7c278c0e56..0aaef1ef15 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -117,6 +117,7 @@ typedef struct IntoClause
char *tableSpaceName; /* table space to use, or NULL */
Node *viewQuery; /* materialized view's SELECT query */
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 00ace8425e..d682ee11cc 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -198,6 +198,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD)
PG_KEYWORD("inherit", INHERIT, UNRESERVED_KEYWORD)
diff --git a/src/include/utils/lsyscache.h b/src/include/utils/lsyscache.h
index c8df5bff9f..8fd919349e 100644
--- a/src/include/utils/lsyscache.h
+++ b/src/include/utils/lsyscache.h
@@ -129,6 +129,7 @@ extern Oid get_rel_namespace(Oid relid);
extern Oid get_rel_type_id(Oid relid);
extern char get_rel_relkind(Oid relid);
extern bool get_rel_relispartition(Oid relid);
+extern bool get_rel_relisivm(Oid relid);
extern Oid get_rel_tablespace(Oid relid);
extern char get_rel_persistence(Oid relid);
extern Oid get_transform_fromsql(Oid typid, Oid langid, List *trftypes);
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index d7f33abce3..057f38d5de 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -562,6 +562,8 @@ typedef struct ViewOptions
*/
#define RelationIsPopulated(relation) ((relation)->rd_rel->relispopulated)
+#define RelationIsIVM(relation) ((relation)->rd_rel->relisivm)
+
/*
* RelationIsAccessibleInLogicalDecoding
* True if we need to log enough information to have access via
diff --git a/src/test/regress/expected/incremental_matview.out b/src/test/regress/expected/incremental_matview.out
new file mode 100644
index 0000000000..e58b9ad6d0
--- /dev/null
+++ b/src/test/regress/expected/incremental_matview.out
@@ -0,0 +1,215 @@
+-- create a table to use as a basis for views and materialized views in various combinations
+CREATE TABLE mv_base_a (i int, j int);
+INSERT INTO mv_base_a VALUES
+ (1,10),
+ (2,20),
+ (3,30),
+ (4,40),
+ (5,50);
+CREATE TABLE mv_base_b (i int, k int);
+INSERT INTO mv_base_b VALUES
+ (1,101),
+ (2,102),
+ (3,103),
+ (4,104);
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_1 AS SELECT i,j,k FROM mv_base_a a INNER JOIN mv_base_b b USING(i);
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+ i | j | k
+---+----+-----
+ 1 | 10 | 101
+ 2 | 20 | 102
+ 3 | 30 | 103
+ 4 | 40 | 104
+(4 rows)
+
+-- immediaite maintenance
+BEGIN;
+INSERT INTO mv_base_b VALUES(5,105);
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+ i | j | k
+---+----+-----
+ 1 | 10 | 101
+ 2 | 20 | 102
+ 3 | 30 | 103
+ 4 | 40 | 104
+ 5 | 50 | 105
+(5 rows)
+
+UPDATE mv_base_a SET j = 0 WHERE i = 1;
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+ i | j | k
+---+----+-----
+ 1 | 0 | 101
+ 2 | 20 | 102
+ 3 | 30 | 103
+ 4 | 40 | 104
+ 5 | 50 | 105
+(5 rows)
+
+DELETE FROM mv_base_b WHERE (i,k) = (5,105);
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+ i | j | k
+---+----+-----
+ 1 | 0 | 101
+ 2 | 20 | 102
+ 3 | 30 | 103
+ 4 | 40 | 104
+(4 rows)
+
+ROLLBACK;
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+ i | j | k
+---+----+-----
+ 1 | 10 | 101
+ 2 | 20 | 102
+ 3 | 30 | 103
+ 4 | 40 | 104
+(4 rows)
+
+-- result of materliazied view have DISTINCT clause or the duplicate result.
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_duplicate AS SELECT j FROM mv_base_a;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_distinct AS SELECT DISTINCT j FROM mv_base_a;
+INSERT INTO mv_base_a VALUES(6,20);
+SELECT * FROM mv_ivm_duplicate ORDER BY 1;
+ j
+----
+ 10
+ 20
+ 20
+ 30
+ 40
+ 50
+(6 rows)
+
+SELECT * FROM mv_ivm_distinct ORDER BY 1;
+ j
+----
+ 10
+ 20
+ 30
+ 40
+ 50
+(5 rows)
+
+DELETE FROM mv_base_a WHERE (i,j) = (2,20);
+SELECT * FROM mv_ivm_duplicate ORDER BY 1;
+ j
+----
+ 10
+ 20
+ 30
+ 40
+ 50
+(5 rows)
+
+SELECT * FROM mv_ivm_distinct ORDER BY 1;
+ j
+----
+ 10
+ 20
+ 30
+ 40
+ 50
+(5 rows)
+
+ROLLBACK;
+-- support SUM() and COUNT() aggregation function
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_agg AS SELECT i, SUM(j), COUNT(i) FROM mv_base_a GROUP BY i;
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
+ i | sum | count
+---+-----+-------
+ 1 | 10 | 1
+ 2 | 20 | 1
+ 3 | 30 | 1
+ 4 | 40 | 1
+ 5 | 50 | 1
+(5 rows)
+
+INSERT INTO mv_base_a VALUES(2,100);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
+ i | sum | count
+---+-----+-------
+ 1 | 10 | 1
+ 2 | 120 | 2
+ 3 | 30 | 1
+ 4 | 40 | 1
+ 5 | 50 | 1
+(5 rows)
+
+UPDATE mv_base_a SET j = 200 WHERE (i,j) = (2,100);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
+ i | sum | count
+---+-----+-------
+ 1 | 10 | 1
+ 2 | 220 | 2
+ 3 | 30 | 1
+ 4 | 40 | 1
+ 5 | 50 | 1
+(5 rows)
+
+DELETE FROM mv_base_a WHERE (i,j) = (2,200);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
+ i | sum | count
+---+-----+-------
+ 1 | 10 | 1
+ 2 | 20 | 1
+ 3 | 30 | 1
+ 4 | 40 | 1
+ 5 | 50 | 1
+(5 rows)
+
+ROLLBACK;
+-- support COUNT(*) aggregation function
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_agg AS SELECT i, SUM(j),COUNT(*) FROM mv_base_a GROUP BY i;
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
+ i | sum | count
+---+-----+-------
+ 1 | 10 | 1
+ 2 | 20 | 1
+ 3 | 30 | 1
+ 4 | 40 | 1
+ 5 | 50 | 1
+(5 rows)
+
+INSERT INTO mv_base_a VALUES(2,100);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
+ i | sum | count
+---+-----+-------
+ 1 | 10 | 1
+ 2 | 120 | 2
+ 3 | 30 | 1
+ 4 | 40 | 1
+ 5 | 50 | 1
+(5 rows)
+
+ROLLBACK;
+-- support having only aggregation function without GROUP clause
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_group AS SELECT SUM(j)FROM mv_base_a;
+SELECT * FROM mv_ivm_group ORDER BY 1;
+ sum
+-----
+ 150
+(1 row)
+
+INSERT INTO mv_base_a VALUES(6,20);
+SELECT * FROM mv_ivm_group ORDER BY 1;
+ sum
+-----
+ 170
+(1 row)
+
+ROLLBACK;
+-- unsupport aggregation function except for SUM(),COUNT()
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_min AS SELECT i, MIN(j) FROM mv_base_a GROUP BY i;
+ERROR: Aggrege function min is not supported
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_max AS SELECT i, MAX(j) FROM mv_base_a GROUP BY i;
+ERROR: Aggrege function max is not supported
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_avg AS SELECT i, AVG(j) FROM mv_base_a GROUP BY i;
+ERROR: Aggrege function avg is not supported
+DROP TABLE mv_base_b CASCADE;
+NOTICE: drop cascades to materialized view mv_ivm_1
+DROP TABLE mv_base_a CASCADE;
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index f23fe8d870..63e743dddc 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -78,7 +78,7 @@ test: brin gin gist spgist privileges init_privs security_label collate matview
# ----------
# Another group of parallel tests
# ----------
-test: create_table_like alter_generic alter_operator misc async dbsize misc_functions sysviews tsrf tidscan
+test: create_table_like alter_generic alter_operator misc async dbsize misc_functions sysviews tsrf tidscan incremental_matview
# rules cannot run concurrently with any test that creates
# a view or rule in the public schema
diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule
index ca200eb599..555c91d771 100644
--- a/src/test/regress/serial_schedule
+++ b/src/test/regress/serial_schedule
@@ -113,6 +113,7 @@ test: init_privs
test: security_label
test: collate
test: matview
+test: incremental_matview
test: lock
test: replica_identity
test: rowsecurity
diff --git a/src/test/regress/sql/incremental_matview.sql b/src/test/regress/sql/incremental_matview.sql
new file mode 100644
index 0000000000..94417093ce
--- /dev/null
+++ b/src/test/regress/sql/incremental_matview.sql
@@ -0,0 +1,77 @@
+
+-- create a table to use as a basis for views and materialized views in various combinations
+CREATE TABLE mv_base_a (i int, j int);
+INSERT INTO mv_base_a VALUES
+ (1,10),
+ (2,20),
+ (3,30),
+ (4,40),
+ (5,50);
+CREATE TABLE mv_base_b (i int, k int);
+INSERT INTO mv_base_b VALUES
+ (1,101),
+ (2,102),
+ (3,103),
+ (4,104);
+
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_1 AS SELECT i,j,k FROM mv_base_a a INNER JOIN mv_base_b b USING(i);
+
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+-- immediaite maintenance
+BEGIN;
+INSERT INTO mv_base_b VALUES(5,105);
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+UPDATE mv_base_a SET j = 0 WHERE i = 1;
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+DELETE FROM mv_base_b WHERE (i,k) = (5,105);
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+ROLLBACK;
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+
+-- result of materliazied view have DISTINCT clause or the duplicate result.
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_duplicate AS SELECT j FROM mv_base_a;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_distinct AS SELECT DISTINCT j FROM mv_base_a;
+INSERT INTO mv_base_a VALUES(6,20);
+SELECT * FROM mv_ivm_duplicate ORDER BY 1;
+SELECT * FROM mv_ivm_distinct ORDER BY 1;
+DELETE FROM mv_base_a WHERE (i,j) = (2,20);
+SELECT * FROM mv_ivm_duplicate ORDER BY 1;
+SELECT * FROM mv_ivm_distinct ORDER BY 1;
+ROLLBACK;
+
+-- support SUM() and COUNT() aggregation function
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_agg AS SELECT i, SUM(j), COUNT(i) FROM mv_base_a GROUP BY i;
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
+INSERT INTO mv_base_a VALUES(2,100);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
+UPDATE mv_base_a SET j = 200 WHERE (i,j) = (2,100);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
+DELETE FROM mv_base_a WHERE (i,j) = (2,200);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
+ROLLBACK;
+
+-- support COUNT(*) aggregation function
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_agg AS SELECT i, SUM(j),COUNT(*) FROM mv_base_a GROUP BY i;
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
+INSERT INTO mv_base_a VALUES(2,100);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
+ROLLBACK;
+
+-- support having only aggregation function without GROUP clause
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_group AS SELECT SUM(j)FROM mv_base_a;
+SELECT * FROM mv_ivm_group ORDER BY 1;
+INSERT INTO mv_base_a VALUES(6,20);
+SELECT * FROM mv_ivm_group ORDER BY 1;
+ROLLBACK;
+
+-- unsupport aggregation function except for SUM(),COUNT()
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_min AS SELECT i, MIN(j) FROM mv_base_a GROUP BY i;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_max AS SELECT i, MAX(j) FROM mv_base_a GROUP BY i;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_avg AS SELECT i, AVG(j) FROM mv_base_a GROUP BY i;
+
+DROP TABLE mv_base_b CASCADE;
+DROP TABLE mv_base_a CASCADE;
Hi Greg,
On Wed, 3 Apr 2019 17:41:36 -0400
Greg Stark <stark@mit.edu> wrote:
On Sun, 31 Mar 2019 at 23:22, Yugo Nagata <nagata@sraoss.co.jp> wrote:
Firstly, this will handle simple definition views which includes only
selection, projection, and join. Standard aggregations (count, sum, avg,
min, max) are not planned to be implemented in the first patch, but these
are commonly used in materialized views, so I'll implement them later on.It's fine to not have all the features from day 1 of course. But I
just picked up this comment and the followup talking about splitting
AVG into SUM and COUNT and I had a comment. When you do look at
tackling aggregates I don't think you should restrict yourself to
these specific standard aggregations. We have all the necessary
abstractions to handle all aggregations that are feasible, see
https://www.postgresql.org/docs/devel/xaggr.html#XAGGR-MOVING-AGGREGATESWhat you need to do -- I think -- is store the "moving aggregate
state" before the final function. Then whenever a row is inserted or
deleted or updated (or whenever another column is updated which causes
the value to row to enter or leave the aggregation) apply either
aggtransfn or aggminvtransfn to the state. I'm not sure if you want to
apply the final function on every update or only lazily either may be
better in some usage.
Thank you for your suggestion! I submitted the latest patch just now supporting
some aggregate functions, but this supports only sum and count, and lacking a
kind of generalization. However, I would like to refine this to support more
general aggregate functions. I think your suggestions is helpful for me to do
this. Thank you!
Best regards,
Yugo Nagata
--
Yugo Nagata <nagata@sraoss.co.jp>
On Fri, Jun 28, 2019 at 10:56 PM Yugo Nagata <nagata@sraoss.co.jp> wrote:
Attached is a WIP patch of IVM which supports some aggregate functions.
Hi Nagata-san and Hoshiai-san,
Thank you for working on this. I enjoyed your talk at PGCon. I've
added Kevin Grittner just in case he missed this thread; he has talked
often about implementing the counting algorithm, and he wrote the
"trigger transition tables" feature to support exactly this. While
integrating trigger transition tables with the new partition features,
we had to make a number of decisions about how that should work, and
we tried to come up with answers that would work for IMV, and I hope
we made the right choices!
I am quite interested to learn how IVM interacts with SERIALIZABLE.
A couple of superficial review comments:
+ const char *aggname = get_func_name(aggref->aggfnoid);
...
+ else if (!strcmp(aggname, "sum"))
I guess you need a more robust way to detect the supported aggregates
than their name, or I guess some way for aggregates themselves to
specify that they support this and somehow supply the extra logic.
Perhaps I just waid what Greg Stark already said, except not as well.
+ elog(ERROR, "Aggrege function %s is not
supported", aggname);
s/Aggrege/aggregate/
Of course it is not helpful to comment on typos at this early stage,
it's just that this one appears many times in the test output :-)
+static bool
+isIvmColumn(const char *s)
+{
+ char pre[7];
+
+ strlcpy(pre, s, sizeof(pre));
+ return (strcmp(pre, "__ivm_") == 0);
+}
What about strncmp(s, "__ivm_", 6) == 0)? As for the question of how
to reserve a namespace for system columns that won't clash with user
columns, according to our manual the SQL standard doesn't allow $ in
identifier names, and according to my copy SQL92 "intermediate SQL"
doesn't allow identifiers that end in an underscore. I don't know
what the best answer is but we should probably decide on a something
based the standard.
As for how to make internal columns invisible to SELECT *, previously
there have been discussions about doing that using a new flag in
pg_attribute:
/messages/by-id/CAEepm=3ZHh=p0nEEnVbs1Dig_UShPzHUcMNAqvDQUgYgcDo-pA@mail.gmail.com
+ "WITH t AS ("
+ " SELECT diff.__ivm_count__,
(diff.__ivm_count__ = mv.__ivm_count__) AS for_dlt, mv.ctid"
+ ", %s"
+ " FROM %s AS mv, %s AS diff WHERE (%s) = (%s)"
+ "), updt AS ("
+ " UPDATE %s AS mv SET __ivm_count__ =
mv.__ivm_count__ - t.__ivm_count__"
+ ", %s "
+ " FROM t WHERE mv.ctid = t.ctid AND NOT for_dlt"
+ ") DELETE FROM %s AS mv USING t WHERE
mv.ctid = t.ctid AND for_dlt;",
I fully understand that this is POC code, but I am curious about one
thing. These queries that are executed by apply_delta() would need to
be converted to C, or at least used reusable plans, right? Hmm,
creating and dropping temporary tables every time is a clue that the
ultimate form of this should be tuplestores and C code, I think,
right?
Moreover, some regression test are added for aggregate functions support.
This is Hoshiai-san's work.
Great. Next time you post a WIP patch, could you please fix this
small compiler warning?
describe.c: In function ‘describeOneTableDetails’:
describe.c:3270:55: error: ‘*((void *)&tableinfo+48)’ may be used
uninitialized in this function [-Werror=maybe-uninitialized]
if (verbose && tableinfo.relkind == RELKIND_MATVIEW && tableinfo.isivm)
^
describe.c:1495:4: note: ‘*((void *)&tableinfo+48)’ was declared here
} tableinfo;
^
Then our unofficial automatic CI system[1]cfbot.cputube.org will run these tests every
day, which sometimes finds problems.
[1]: cfbot.cputube.org
--
Thomas Munro
https://enterprisedb.com
As for how to make internal columns invisible to SELECT *, previously
there have been discussions about doing that using a new flag in
pg_attribute:/messages/by-id/CAEepm=3ZHh=p0nEEnVbs1Dig_UShPzHUcMNAqvDQUgYgcDo-pA@mail.gmail.com
Now that I realized that there are several use cases for invisible
columns, I think this is the way what we shoud go for.
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
Hi Thomas,
2019年7月8日(月) 15:32 Thomas Munro <thomas.munro@gmail.com>:
On Fri, Jun 28, 2019 at 10:56 PM Yugo Nagata <nagata@sraoss.co.jp> wrote:
Attached is a WIP patch of IVM which supports some aggregate functions.
Hi Nagata-san and Hoshiai-san,
Thank you for working on this. I enjoyed your talk at PGCon. I've
added Kevin Grittner just in case he missed this thread; he has talked
often about implementing the counting algorithm, and he wrote the
"trigger transition tables" feature to support exactly this. While
integrating trigger transition tables with the new partition features,
we had to make a number of decisions about how that should work, and
we tried to come up with answers that would work for IMV, and I hope
we made the right choices!I am quite interested to learn how IVM interacts with SERIALIZABLE.
Nagata-san has been studying this. Nagata-san, any comment?
A couple of superficial review comments:
Thank you for your review comments.
Please find attached patches. The some of your review is reflected in patch
too.
We manage and update IVM on following github repository.
https://github.com/sraoss/pgsql-ivm
you also can found latest WIP patch here.
+ const char *aggname = get_func_name(aggref->aggfnoid); ... + else if (!strcmp(aggname, "sum"))I guess you need a more robust way to detect the supported aggregates
than their name, or I guess some way for aggregates themselves to
specify that they support this and somehow supply the extra logic.
Perhaps I just waid what Greg Stark already said, except not as well.
We have recognized the issue and are welcome any input.
+ elog(ERROR, "Aggrege function %s is not
supported", aggname);
s/Aggrege/aggregate/
I fixed this typo.
Of course it is not helpful to comment on typos at this early stage,
it's just that this one appears many times in the test output :-)
+static bool +isIvmColumn(const char *s) +{ + char pre[7]; + + strlcpy(pre, s, sizeof(pre)); + return (strcmp(pre, "__ivm_") == 0); +}What about strncmp(s, "__ivm_", 6) == 0)?
I agree with you, I fixed it.
As for the question of how
to reserve a namespace for system columns that won't clash with user
columns, according to our manual the SQL standard doesn't allow $ in
identifier names, and according to my copy SQL92 "intermediate SQL"
doesn't allow identifiers that end in an underscore. I don't know
what the best answer is but we should probably decide on a something
based the standard.As for how to make internal columns invisible to SELECT *, previously
there have been discussions about doing that using a new flag in
pg_attribute:/messages/by-id/CAEepm=3ZHh=p0nEEnVbs1Dig_UShPzHUcMNAqvDQUgYgcDo-pA@mail.gmail.com
+ "WITH t AS (" + " SELECT diff.__ivm_count__, (diff.__ivm_count__ = mv.__ivm_count__) AS for_dlt, mv.ctid" + ", %s" + " FROM %s AS mv, %s AS diff WHERE (%s) = (%s)" + "), updt AS (" + " UPDATE %s AS mv SET __ivm_count__ = mv.__ivm_count__ - t.__ivm_count__" + ", %s " + " FROM t WHERE mv.ctid = t.ctid AND NOT for_dlt" + ") DELETE FROM %s AS mv USING t WHERE mv.ctid = t.ctid AND for_dlt;",I fully understand that this is POC code, but I am curious about one
thing. These queries that are executed by apply_delta() would need to
be converted to C, or at least used reusable plans, right? Hmm,
creating and dropping temporary tables every time is a clue that the
ultimate form of this should be tuplestores and C code, I think,
right?
Nagata-san is investing the issue.
Moreover, some regression test are added for aggregate functions support.
This is Hoshiai-san's work.Great. Next time you post a WIP patch, could you please fix this
small compiler warning?describe.c: In function ‘describeOneTableDetails’:
describe.c:3270:55: error: ‘*((void *)&tableinfo+48)’ may be used
uninitialized in this function [-Werror=maybe-uninitialized]
if (verbose && tableinfo.relkind == RELKIND_MATVIEW && tableinfo.isivm)
^
describe.c:1495:4: note: ‘*((void *)&tableinfo+48)’ was declared here
} tableinfo;
^
It is fixed too.
Then our unofficial automatic CI system[1] will run these tests every
day, which sometimes finds problems.
[1] cfbot.cputube.org
--
Thomas Munro
https://enterprisedb.com
Best regards,
Takuma Hoshiai
On Wed, 10 Jul 2019 11:07:15 +0900
Takuma Hoshiai <takuma.hoshiai@gmail.com> wrote:
Hi Thomas,
2019年7月8日(月) 15:32 Thomas Munro <thomas.munro@gmail.com>:
On Fri, Jun 28, 2019 at 10:56 PM Yugo Nagata <nagata@sraoss.co.jp> wrote:
Attached is a WIP patch of IVM which supports some aggregate functions.
Hi Nagata-san and Hoshiai-san,
Thank you for working on this. I enjoyed your talk at PGCon. I've
added Kevin Grittner just in case he missed this thread; he has talked
often about implementing the counting algorithm, and he wrote the
"trigger transition tables" feature to support exactly this. While
integrating trigger transition tables with the new partition features,
we had to make a number of decisions about how that should work, and
we tried to come up with answers that would work for IMV, and I hope
we made the right choices!I am quite interested to learn how IVM interacts with SERIALIZABLE.
Nagata-san has been studying this. Nagata-san, any comment?
A couple of superficial review comments:
Thank you for your review comments.
Please find attached patches. The some of your review is reflected in patch
too.
Sorry, I forgot attaching patch.
In addition, avg() function is supported newly. We found a issue
when use avg() with IVM, added its reproduction case in
regressio test. We are being to fix now.
We manage and update IVM on following github repository.
https://github.com/sraoss/pgsql-ivm
you also can found latest WIP patch here.+ const char *aggname = get_func_name(aggref->aggfnoid); ... + else if (!strcmp(aggname, "sum"))I guess you need a more robust way to detect the supported aggregates
than their name, or I guess some way for aggregates themselves to
specify that they support this and somehow supply the extra logic.
Perhaps I just waid what Greg Stark already said, except not as well.We have recognized the issue and are welcome any input.
+ elog(ERROR, "Aggrege function %s is not
supported", aggname);s/Aggrege/aggregate/
I fixed this typo.
Of course it is not helpful to comment on typos at this early stage,
it's just that this one appears many times in the test output :-)+static bool +isIvmColumn(const char *s) +{ + char pre[7]; + + strlcpy(pre, s, sizeof(pre)); + return (strcmp(pre, "__ivm_") == 0); +}What about strncmp(s, "__ivm_", 6) == 0)?
I agree with you, I fixed it.
As for the question of how
to reserve a namespace for system columns that won't clash with user
columns, according to our manual the SQL standard doesn't allow $ in
identifier names, and according to my copy SQL92 "intermediate SQL"
doesn't allow identifiers that end in an underscore. I don't know
what the best answer is but we should probably decide on a something
based the standard.As for how to make internal columns invisible to SELECT *, previously
there have been discussions about doing that using a new flag in
pg_attribute:/messages/by-id/CAEepm=3ZHh=p0nEEnVbs1Dig_UShPzHUcMNAqvDQUgYgcDo-pA@mail.gmail.com
+ "WITH t AS (" + " SELECT diff.__ivm_count__, (diff.__ivm_count__ = mv.__ivm_count__) AS for_dlt, mv.ctid" + ", %s" + " FROM %s AS mv, %s AS diff WHERE (%s) = (%s)" + "), updt AS (" + " UPDATE %s AS mv SET __ivm_count__ = mv.__ivm_count__ - t.__ivm_count__" + ", %s " + " FROM t WHERE mv.ctid = t.ctid AND NOT for_dlt" + ") DELETE FROM %s AS mv USING t WHERE mv.ctid = t.ctid AND for_dlt;",I fully understand that this is POC code, but I am curious about one
thing. These queries that are executed by apply_delta() would need to
be converted to C, or at least used reusable plans, right? Hmm,
creating and dropping temporary tables every time is a clue that the
ultimate form of this should be tuplestores and C code, I think,
right?Nagata-san is investing the issue.
Moreover, some regression test are added for aggregate functions support.
This is Hoshiai-san's work.Great. Next time you post a WIP patch, could you please fix this
small compiler warning?describe.c: In function ‘describeOneTableDetails’:
describe.c:3270:55: error: ‘*((void *)&tableinfo+48)’ may be used
uninitialized in this function [-Werror=maybe-uninitialized]
if (verbose && tableinfo.relkind == RELKIND_MATVIEW && tableinfo.isivm)
^
describe.c:1495:4: note: ‘*((void *)&tableinfo+48)’ was declared here
} tableinfo;
^It is fixed too.
Then our unofficial automatic CI system[1] will run these tests every
day, which sometimes finds problems.[1] cfbot.cputube.org
--
Thomas Munro
https://enterprisedb.comBest regards,
Takuma Hoshiai
--
Takuma Hoshiai <hoshiai@sraoss.co.jp>
Attachments:
WIP_immediate_IVM_v4.patchapplication/octet-stream; name=WIP_immediate_IVM_v4.patchDownload
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 3428a7c..05cde06 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -1953,6 +1953,13 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</row>
<row>
+ <entry><structfield>relisivm</structfield></entry>
+ <entry><type>bool</type></entry>
+ <entry></entry>
+ <entry>True if materialized view enables incremental view maintenance</entry>
+ </row>
+
+ <row>
<entry><structfield>relrewrite</structfield></entry>
<entry><type>oid</type></entry>
<entry><literal><link linkend="catalog-pg-class"><structname>pg_class</structname></link>.oid</literal></entry>
diff --git a/doc/src/sgml/ref/create_materialized_view.sgml b/doc/src/sgml/ref/create_materialized_view.sgml
index ec8847e..a23366a 100644
--- a/doc/src/sgml/ref/create_materialized_view.sgml
+++ b/doc/src/sgml/ref/create_materialized_view.sgml
@@ -21,7 +21,7 @@ PostgreSQL documentation
<refsynopsisdiv>
<synopsis>
-CREATE MATERIALIZED VIEW [ IF NOT EXISTS ] <replaceable>table_name</replaceable>
+CREATE [ INCREMENTAL ] MATERIALIZED VIEW [ IF NOT EXISTS ] <replaceable>table_name</replaceable>
[ (<replaceable>column_name</replaceable> [, ...] ) ]
[ USING <replaceable class="parameter">method</replaceable> ]
[ WITH ( <replaceable class="parameter">storage_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] ) ]
@@ -55,6 +55,16 @@ CREATE MATERIALIZED VIEW [ IF NOT EXISTS ] <replaceable>table_name</replaceable>
<variablelist>
<varlistentry>
+ <term><literal>INCREMENTAL</literal></term>
+ <listitem>
+ <para>
+ If specified, a materialized view enables incremental view maintenance.
+ You can replace only the contents of a materialized view, which based rows are changed.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><literal>IF NOT EXISTS</literal></term>
<listitem>
<para>
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index 3b8c8b1..a69a183 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -891,6 +891,7 @@ InsertPgClassTuple(Relation pg_class_desc,
values[Anum_pg_class_relrewrite - 1] = ObjectIdGetDatum(rd_rel->relrewrite);
values[Anum_pg_class_relfrozenxid - 1] = TransactionIdGetDatum(rd_rel->relfrozenxid);
values[Anum_pg_class_relminmxid - 1] = MultiXactIdGetDatum(rd_rel->relminmxid);
+ values[Anum_pg_class_relisivm - 1] = BoolGetDatum(rd_rel->relisivm);
if (relacl != (Datum) 0)
values[Anum_pg_class_relacl - 1] = relacl;
else
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index bb60b23..0e22a64 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -910,6 +910,7 @@ index_create(Relation heapRelation,
indexRelation->rd_rel->relowner = heapRelation->rd_rel->relowner;
indexRelation->rd_rel->relam = accessMethodObjectId;
indexRelation->rd_rel->relispartition = OidIsValid(parentIndexRelid);
+ indexRelation->rd_rel->relisivm = false;
/*
* store index's pg_class entry
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index 4c1d909..06073d0 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -51,6 +51,16 @@
#include "utils/rls.h"
#include "utils/snapmgr.h"
+#include "catalog/pg_trigger.h"
+#include "catalog/pg_type.h"
+#include "commands/trigger.h"
+#include "parser/parser.h"
+#include "parser/parsetree.h"
+#include "parser/parse_func.h"
+#include "nodes/print.h"
+#include "optimizer/optimizer.h"
+#include "commands/defrem.h"
+
typedef struct
{
@@ -74,6 +84,8 @@ static bool intorel_receive(TupleTableSlot *slot, DestReceiver *self);
static void intorel_shutdown(DestReceiver *self);
static void intorel_destroy(DestReceiver *self);
+static void CreateIvmTrigger(Oid relOid, Oid viewOid, char *matviewname, int16 type);
+static void CreateIvmTriggersOnBaseTables(Query *qry, Node *jtnode, Oid matviewOid, char* matviewname);
/*
* create_ctas_internal
@@ -109,6 +121,8 @@ create_ctas_internal(List *attrList, IntoClause *into)
create->oncommit = into->onCommit;
create->tablespacename = into->tableSpaceName;
create->if_not_exists = false;
+ /* Using Materialized view only */
+ create->ivm = into->ivm;
create->accessMethod = into->accessMethod;
/*
@@ -239,6 +253,7 @@ ExecCreateTableAs(CreateTableAsStmt *stmt, const char *queryString,
List *rewritten;
PlannedStmt *plan;
QueryDesc *queryDesc;
+ Query *copied_query;
if (stmt->if_not_exists)
{
@@ -319,7 +334,100 @@ ExecCreateTableAs(CreateTableAsStmt *stmt, const char *queryString,
* and is executed repeatedly. (See also the same hack in EXPLAIN and
* PREPARE.)
*/
- rewritten = QueryRewrite(copyObject(query));
+
+ copied_query = copyObject(query);
+ if (is_matview && into->ivm)
+ {
+ TargetEntry *tle;
+ Node *node;
+ ParseState *pstate = make_parsestate(NULL);
+ FuncCall *fn;
+
+ /* group keys must be in targetlist */
+ if (copied_query->groupClause)
+ {
+ ListCell *lc;
+ foreach(lc, copied_query->groupClause)
+ {
+ SortGroupClause *scl = (SortGroupClause *) lfirst(lc);
+ TargetEntry *tle = get_sortgroupclause_tle(scl, copied_query->targetList);
+
+ if (tle->resjunk)
+ elog(ERROR, "GROUP BY expression must appear in select list for incremental materialized views");
+ }
+ }
+ else if (!copied_query->hasAggs)
+ copied_query->groupClause = transformDistinctClause(NULL, &copied_query->targetList, copied_query->sortClause, false);
+
+ if (copied_query->hasAggs)
+ {
+ ListCell *lc;
+ List *agg_counts = NIL;
+ AttrNumber next_resno = list_length(copied_query->targetList) + 1;
+ Const *dmy_arg = makeConst(INT4OID,
+ -1,
+ InvalidOid,
+ sizeof(int32),
+ Int32GetDatum(1),
+ false,
+ true); /* pass by value */
+
+ foreach(lc, copied_query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ TargetEntry *tle_count;
+
+
+ if (IsA(tle->expr, Aggref))
+ {
+ Aggref *aggref = (Aggref *) tle->expr;
+ const char *aggname = get_func_name(aggref->aggfnoid);
+
+ /* XXX: need some generalization */
+ if (strcmp(aggname, "sum") !=0
+ && strcmp(aggname, "count") != 0
+ && strcmp(aggname, "avg") != 0
+ )
+ elog(ERROR, "aggregate function %s is not supported", aggname);
+
+ /* For aggregate functions except to count, add count func with the same arg parameters. */
+ if (strcmp(aggname, "count") != 0)
+ {
+ fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+
+ /* Make a Func with a dummy arg, and then override this by the original agg's args. */
+ node = ParseFuncOrColumn(pstate, fn->funcname, list_make1(dmy_arg), NULL, fn, false, -1);
+ ((Aggref *)node)->args = aggref->args;
+
+ tle_count = makeTargetEntry((Expr *) node,
+ next_resno,
+ pstrdup(makeObjectName("__ivm_count",tle->resname, "_")),
+ false);
+ agg_counts = lappend(agg_counts, tle_count);
+ next_resno++;
+ }
+
+ }
+ }
+ copied_query->targetList = list_concat(copied_query->targetList, agg_counts);
+
+ }
+
+ /* Add count(*) for counting algorithm */
+ fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+ fn->agg_star = true;
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(copied_query->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ copied_query->targetList = lappend(copied_query->targetList, tle);
+ copied_query->hasAggs = true;
+ }
+
+ rewritten = QueryRewrite(copied_query);
/* SELECT should never rewrite to more or less than one SELECT query */
if (list_length(rewritten) != 1)
@@ -378,11 +486,65 @@ ExecCreateTableAs(CreateTableAsStmt *stmt, const char *queryString,
/* Restore userid and security context */
SetUserIdAndSecContext(save_userid, save_sec_context);
+
+
+ if (into->ivm)
+ {
+ char *matviewname;
+ Oid matviewOid = address.objectId;
+ Relation matviewRel = table_open(matviewOid, NoLock);
+ matviewname = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(matviewRel)),
+ RelationGetRelationName(matviewRel));
+ copied_query = copyObject(query);
+ AcquireRewriteLocks(copied_query, true, false);
+
+ CreateIvmTriggersOnBaseTables(copied_query, (Node *)copied_query->jointree, matviewOid, matviewname);
+
+ table_close(matviewRel, NoLock);
+ }
}
return address;
}
+static void CreateIvmTriggersOnBaseTables(Query *qry, Node *jtnode, Oid matviewOid, char* matviewname)
+{
+
+ if (jtnode == NULL)
+ return;
+ if (IsA(jtnode, RangeTblRef))
+ {
+ int rti = ((RangeTblRef *) jtnode)->rtindex;
+ RangeTblEntry *rte = rt_fetch(rti, qry->rtable);
+
+ if (rte->rtekind == RTE_RELATION)
+ {
+ CreateIvmTrigger(rte->relid, matviewOid, matviewname, TRIGGER_TYPE_INSERT);
+ CreateIvmTrigger(rte->relid, matviewOid, matviewname, TRIGGER_TYPE_DELETE);
+ CreateIvmTrigger(rte->relid, matviewOid, matviewname, TRIGGER_TYPE_UPDATE);
+ }
+ else
+ elog(ERROR, "unsupported RTE kind: %d", (int) rte->rtekind);
+ }
+ else if (IsA(jtnode, FromExpr))
+ {
+ FromExpr *f = (FromExpr *) jtnode;
+ ListCell *l;
+
+ foreach(l, f->fromlist)
+ CreateIvmTriggersOnBaseTables(qry, lfirst(l), matviewOid, matviewname);
+ }
+ else if (IsA(jtnode, JoinExpr))
+ {
+ JoinExpr *j = (JoinExpr *) jtnode;
+
+ CreateIvmTriggersOnBaseTables(qry, j->larg, matviewOid, matviewname);
+ CreateIvmTriggersOnBaseTables(qry, j->rarg, matviewOid, matviewname);
+ }
+ else
+ elog(ERROR, "unrecognized node type: %d", (int) nodeTag(jtnode));
+}
+
/*
* GetIntoRelEFlags --- compute executor flags needed for CREATE TABLE AS
*
@@ -548,6 +710,11 @@ intorel_startup(DestReceiver *self, int operation, TupleDesc typeinfo)
SetMatViewPopulatedState(intoRelationDesc, true);
/*
+ * Mark relisivm field, if it's a matview and into->ivm is true.
+ */
+ if (is_matview && into->ivm)
+ SetMatViewIVMState(intoRelationDesc, true);
+ /*
* Fill private fields of myState for use by later routines
*/
myState->rel = intoRelationDesc;
@@ -619,3 +786,74 @@ intorel_destroy(DestReceiver *self)
{
pfree(self);
}
+
+static void
+CreateIvmTrigger(Oid relOid, Oid viewOid, char *matviewname, int16 type)
+{
+ CreateTrigStmt *ivm_trigger;
+ List *transitionRels = NIL;
+ ObjectAddress address, refaddr;
+
+ refaddr.classId = RelationRelationId;
+ refaddr.objectId = viewOid;
+ refaddr.objectSubId = 0;
+
+
+ ivm_trigger = makeNode(CreateTrigStmt);
+ ivm_trigger->relation = NULL;
+ ivm_trigger->row = false;
+ ivm_trigger->timing = TRIGGER_TYPE_AFTER;
+
+ ivm_trigger->events = type;
+
+ switch (type)
+ {
+ case TRIGGER_TYPE_INSERT:
+ ivm_trigger->trigname = "IVM_trigger_ins";
+ break;
+ case TRIGGER_TYPE_DELETE:
+ ivm_trigger->trigname = "IVM_trigger_del";
+ break;
+ case TRIGGER_TYPE_UPDATE:
+ ivm_trigger->trigname = "IVM_trigger_upd";
+ break;
+ }
+
+ if (type == TRIGGER_TYPE_INSERT || type == TRIGGER_TYPE_UPDATE)
+ {
+ TriggerTransition *n = makeNode(TriggerTransition);
+ n->name = "ivm_newtable";
+ n->isNew = true;
+ n->isTable = true;
+
+ transitionRels = lappend(transitionRels, n);
+ }
+ if (type == TRIGGER_TYPE_DELETE || type == TRIGGER_TYPE_UPDATE)
+ {
+ TriggerTransition *n = makeNode(TriggerTransition);
+ n->name = "ivm_oldtable";
+ n->isNew = false;
+ n->isTable = true;
+
+ transitionRels = lappend(transitionRels, n);
+ }
+
+ ivm_trigger->funcname = SystemFuncName("IVM_immediate_maintenance");
+
+ ivm_trigger->columns = NIL;
+ ivm_trigger->transitionRels = transitionRels;
+ ivm_trigger->whenClause = NULL;
+ ivm_trigger->isconstraint = false;
+ ivm_trigger->deferrable = false;
+ ivm_trigger->initdeferred = false;
+ ivm_trigger->constrrel = NULL;
+ ivm_trigger->args = list_make1(makeString(matviewname));
+
+ address = CreateTrigger(ivm_trigger, NULL, relOid, InvalidOid, InvalidOid,
+ InvalidOid, InvalidOid, InvalidOid, NULL, true, false);
+
+ recordDependencyOn(&address, &refaddr, DEPENDENCY_AUTO);
+
+ /* Make changes-so-far visible */
+ CommandCounterIncrement();
+}
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index 537d0e8..edede5a 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -46,6 +46,15 @@
#include "utils/snapmgr.h"
#include "utils/syscache.h"
+#include "utils/regproc.h"
+#include "nodes/makefuncs.h"
+#include "parser/parse_clause.h"
+#include "parser/parse_func.h"
+#include "nodes/print.h"
+#include "catalog/pg_type_d.h"
+#include "optimizer/optimizer.h"
+#include "commands/defrem.h"
+
typedef struct
{
@@ -65,7 +74,8 @@ static bool transientrel_receive(TupleTableSlot *slot, DestReceiver *self);
static void transientrel_shutdown(DestReceiver *self);
static void transientrel_destroy(DestReceiver *self);
static uint64 refresh_matview_datafill(DestReceiver *dest, Query *query,
- const char *queryString);
+ QueryEnvironment *queryEnv,
+ const char *queryString);
static char *make_temptable_name_n(char *tempname, int n);
static void refresh_by_match_merge(Oid matviewOid, Oid tempOid, Oid relowner,
int save_sec_context);
@@ -74,6 +84,9 @@ static bool is_usable_unique_index(Relation indexRel);
static void OpenMatViewIncrementalMaintenance(void);
static void CloseMatViewIncrementalMaintenance(void);
+static void apply_delta(Oid matviewOid, Oid tempOid_new, Oid tempOid_old,
+ Query *query, Oid relowner, int save_sec_context);
+
/*
* SetMatViewPopulatedState
* Mark a materialized view as populated, or not.
@@ -115,6 +128,46 @@ SetMatViewPopulatedState(Relation relation, bool newstate)
}
/*
+ * SetMatViewIVMState
+ * Mark a materialized view as IVM, or not.
+ *
+ * NOTE: caller must be holding an appropriate lock on the relation.
+ */
+void
+SetMatViewIVMState(Relation relation, bool newstate)
+{
+ Relation pgrel;
+ HeapTuple tuple;
+
+ Assert(relation->rd_rel->relkind == RELKIND_MATVIEW);
+
+ /*
+ * Update relation's pg_class entry. Crucial side-effect: other backends
+ * (and this one too!) are sent SI message to make them rebuild relcache
+ * entries.
+ */
+ pgrel = table_open(RelationRelationId, RowExclusiveLock);
+ tuple = SearchSysCacheCopy1(RELOID,
+ ObjectIdGetDatum(RelationGetRelid(relation)));
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for relation %u",
+ RelationGetRelid(relation));
+
+ ((Form_pg_class) GETSTRUCT(tuple))->relisivm = newstate;
+
+ CatalogTupleUpdate(pgrel, &tuple->t_self, tuple);
+
+ heap_freetuple(tuple);
+ table_close(pgrel, RowExclusiveLock);
+
+ /*
+ * Advance command counter to make the updated pg_class row locally
+ * visible.
+ */
+ CommandCounterIncrement();
+}
+
+/*
* ExecRefreshMatView -- execute a REFRESH MATERIALIZED VIEW command
*
* This refreshes the materialized view by creating a new table and swapping
@@ -311,7 +364,7 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
/* Generate the data, if wanted. */
if (!stmt->skipData)
- processed = refresh_matview_datafill(dest, dataQuery, queryString);
+ processed = refresh_matview_datafill(dest, dataQuery, NULL, queryString);
/* Make the matview match the newly generated data. */
if (concurrent)
@@ -369,6 +422,7 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
*/
static uint64
refresh_matview_datafill(DestReceiver *dest, Query *query,
+ QueryEnvironment *queryEnv,
const char *queryString)
{
List *rewritten;
@@ -405,7 +459,7 @@ refresh_matview_datafill(DestReceiver *dest, Query *query,
/* Create a QueryDesc, redirecting output to our tuple receiver */
queryDesc = CreateQueryDesc(plan, queryString,
GetActiveSnapshot(), InvalidSnapshot,
- dest, NULL, NULL, 0);
+ dest, NULL, queryEnv ? queryEnv: NULL, 0);
/* call ExecutorStart to prepare the plan for execution */
ExecutorStart(queryDesc, 0);
@@ -926,3 +980,664 @@ CloseMatViewIncrementalMaintenance(void)
matview_maintenance_depth--;
Assert(matview_maintenance_depth >= 0);
}
+
+/*
+ * IVM trigger function
+ */
+
+Datum
+IVM_immediate_maintenance(PG_FUNCTION_ARGS)
+{
+ TriggerData *trigdata = (TriggerData *) fcinfo->context;
+ Relation rel;
+ Oid relid;
+ Oid matviewOid;
+ Query *query, *old_delta_qry, *new_delta_qry;
+ char* matviewname = trigdata->tg_trigger->tgargs[0];
+ List *names;
+ Relation matviewRel;
+ int old_depth = matview_maintenance_depth;
+
+ Oid tableSpace;
+ Oid relowner;
+ Oid OIDDelta_new = InvalidOid;
+ Oid OIDDelta_old = InvalidOid;
+ DestReceiver *dest_new = NULL, *dest_old = NULL;
+ char relpersistence;
+ Oid save_userid;
+ int save_sec_context;
+ int save_nestlevel;
+
+ ParseState *pstate;
+ QueryEnvironment *queryEnv = create_queryEnv();
+
+ Const *dmy_arg = makeConst(INT4OID,
+ -1,
+ InvalidOid,
+ sizeof(int32),
+ Int32GetDatum(1),
+ false,
+ true); /* pass by value */
+
+ /* Create a dummy ParseState for addRangeTableEntryForENR */
+ pstate = make_parsestate(NULL);
+ pstate->p_queryEnv = queryEnv;
+
+ names = stringToQualifiedNameList(matviewname);
+
+ /*
+ * Wait for concurrent transactions which update this materialized view at READ COMMITED.
+ * This is needed to see changes commited in othre transactions. No wait and raise an error
+ * at REPEATABLE READ or SERIALIZABLE to prevent anormal update of matviews.
+ * XXX: dead-lock is possible here.
+ */
+ if (!IsolationUsesXactSnapshot())
+ matviewOid = RangeVarGetRelid(makeRangeVarFromNameList(names), ExclusiveLock, true);
+ else
+ matviewOid = RangeVarGetRelidExtended(makeRangeVarFromNameList(names), ExclusiveLock, RVR_MISSING_OK | RVR_NOWAIT, NULL, NULL);
+
+ matviewRel = table_open(matviewOid, NoLock);
+
+ /*
+ * Get and push the latast snapshot to see any changes which is commited during waiting in
+ * other transactions at READ COMMITTED level.
+ * XXX: Is this safe?
+ */
+ PushActiveSnapshot(GetTransactionSnapshot());
+
+ /* Make sure it is a materialized view. */
+ if (matviewRel->rd_rel->relkind != RELKIND_MATVIEW)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("\"%s\" is not a materialized view",
+ RelationGetRelationName(matviewRel))));
+
+ rel = trigdata->tg_relation;
+ relid = rel->rd_id;
+
+ query = get_view_query(matviewRel);
+
+ new_delta_qry = copyObject(query);
+ old_delta_qry = copyObject(query);
+
+ if (trigdata->tg_newtable)
+ {
+ RangeTblEntry *rte;
+ ListCell *lc;
+
+ TargetEntry *tle;
+ Node *node;
+ FuncCall *fn;
+
+ EphemeralNamedRelation enr =
+ palloc(sizeof(EphemeralNamedRelationData));
+
+ enr->md.name = trigdata->tg_trigger->tgnewtable;
+ enr->md.reliddesc = trigdata->tg_relation->rd_id;
+ enr->md.tupdesc = NULL;
+ enr->md.enrtype = ENR_NAMED_TUPLESTORE;
+ enr->md.enrtuples = tuplestore_tuple_count(trigdata->tg_newtable);
+ enr->reldata = trigdata->tg_newtable;
+ register_ENR(queryEnv, enr);
+
+ rte = addRangeTableEntryForENR(pstate, makeRangeVar(NULL, enr->md.name, -1), true);
+ new_delta_qry->rtable = lappend(new_delta_qry->rtable, rte);
+
+ foreach(lc, new_delta_qry->rtable)
+ {
+ RangeTblEntry *r = (RangeTblEntry*) lfirst(lc);
+ if (r->relid == relid)
+ {
+ lfirst(lc) = rte;
+ break;
+ }
+ }
+
+ if (query->hasAggs)
+ {
+ ListCell *lc;
+ List *agg_counts = NIL;
+ AttrNumber next_resno = list_length(query->targetList) + 1;
+ Node *node;
+
+ foreach(lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ TargetEntry *tle_count;
+
+ if (IsA(tle->expr, Aggref))
+ {
+ Aggref *aggref = (Aggref *) tle->expr;
+ const char *aggname = get_func_name(aggref->aggfnoid);
+
+ if (strcmp(aggname, "count") != 0)
+ {
+ fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, list_make1(dmy_arg), NULL, fn, false, -1);
+ ((Aggref *)node)->args = aggref->args;
+
+ tle_count = makeTargetEntry((Expr *) node,
+ next_resno,
+ pstrdup(makeObjectName("__ivm_count",tle->resname, "_")),
+ false);
+ agg_counts = lappend(agg_counts, tle_count);
+ next_resno++;
+ }
+ }
+
+ }
+ new_delta_qry->targetList = list_concat(new_delta_qry->targetList, agg_counts);
+ }
+
+ fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+ fn->agg_star = true;
+ if (!new_delta_qry->groupClause && !new_delta_qry->hasAggs)
+ new_delta_qry->groupClause = transformDistinctClause(NULL, &new_delta_qry->targetList, new_delta_qry->sortClause, false);
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(new_delta_qry->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ new_delta_qry->targetList = lappend(new_delta_qry->targetList, tle);
+ new_delta_qry->hasAggs = true;
+ }
+
+ if (trigdata->tg_oldtable)
+ {
+ RangeTblEntry *rte;
+ ListCell *lc;
+
+ TargetEntry *tle;
+ Node *node;
+ FuncCall *fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+
+ EphemeralNamedRelation enr =
+ palloc(sizeof(EphemeralNamedRelationData));
+
+ enr->md.name = trigdata->tg_trigger->tgoldtable;
+ enr->md.reliddesc = trigdata->tg_relation->rd_id;
+ enr->md.tupdesc = NULL;
+ enr->md.enrtype = ENR_NAMED_TUPLESTORE;
+ enr->md.enrtuples = tuplestore_tuple_count(trigdata->tg_oldtable);
+ enr->reldata = trigdata->tg_oldtable;
+ register_ENR(queryEnv, enr);
+
+ rte = addRangeTableEntryForENR(pstate, makeRangeVar(NULL, enr->md.name, -1), true);
+ old_delta_qry->rtable = lappend(old_delta_qry->rtable, rte);
+
+ foreach(lc, old_delta_qry->rtable)
+ {
+ RangeTblEntry *r = (RangeTblEntry*) lfirst(lc);
+ if (r->relid == relid)
+ {
+ lfirst(lc) = rte;
+ break;
+ }
+ }
+
+ if (query->hasAggs)
+ {
+ ListCell *lc;
+ List *agg_counts = NIL;
+ AttrNumber next_resno = list_length(query->targetList) + 1;
+ Node *node;
+
+ foreach(lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ TargetEntry *tle_count;
+
+ if (IsA(tle->expr, Aggref))
+ {
+ Aggref *aggref = (Aggref *) tle->expr;
+ const char *aggname = get_func_name(aggref->aggfnoid);
+
+ if (strcmp(aggname, "count") != 0)
+ {
+ fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, list_make1(dmy_arg), NULL, fn, false, -1);
+ ((Aggref *)node)->args = aggref->args;
+
+ tle_count = makeTargetEntry((Expr *) node,
+ next_resno,
+ pstrdup(makeObjectName("__ivm_count",tle->resname, "_")),
+ false);
+ agg_counts = lappend(agg_counts, tle_count);
+ next_resno++;
+ }
+ }
+
+ }
+ old_delta_qry->targetList = list_concat(old_delta_qry->targetList, agg_counts);
+ }
+
+ fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+ fn->agg_star = true;
+
+ if (!old_delta_qry->groupClause && !old_delta_qry->hasAggs)
+ old_delta_qry->groupClause = transformDistinctClause(NULL, &old_delta_qry->targetList, old_delta_qry->sortClause, false);
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+ tle = makeTargetEntry((Expr *) node,
+ list_length(old_delta_qry->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ old_delta_qry->targetList = lappend(old_delta_qry->targetList, tle);
+ old_delta_qry->hasAggs = true;
+ }
+
+
+ /*
+ * Check for active uses of the relation in the current transaction, such
+ * as open scans.
+ *
+ * NB: We count on this to protect us against problems with refreshing the
+ * data using TABLE_INSERT_FROZEN.
+ */
+ CheckTableNotInUse(matviewRel, "REFRESH MATERIALIZED VIEW");
+
+ relowner = matviewRel->rd_rel->relowner;
+
+ /*
+ * Switch to the owner's userid, so that any functions are run as that
+ * user. Also arrange to make GUC variable changes local to this command.
+ * Don't lock it down too tight to create a temporary table just yet. We
+ * will switch modes when we are about to execute user code.
+ */
+ GetUserIdAndSecContext(&save_userid, &save_sec_context);
+ SetUserIdAndSecContext(relowner,
+ save_sec_context | SECURITY_LOCAL_USERID_CHANGE);
+ save_nestlevel = NewGUCNestLevel();
+
+ tableSpace = GetDefaultTablespace(RELPERSISTENCE_TEMP, false);
+ relpersistence = RELPERSISTENCE_TEMP;
+
+ /*
+ * Create the transient table that will receive the regenerated data. Lock
+ * it against access by any other process until commit (by which time it
+ * will be gone).
+ */
+ if (trigdata->tg_newtable)
+ {
+ OIDDelta_new = make_new_heap(matviewOid, tableSpace, relpersistence,
+ ExclusiveLock);
+ LockRelationOid(OIDDelta_new, AccessExclusiveLock);
+ dest_new = CreateTransientRelDestReceiver(OIDDelta_new);
+ }
+ if (trigdata->tg_oldtable)
+ {
+ if (trigdata->tg_newtable)
+ OIDDelta_old = make_new_heap(OIDDelta_new, tableSpace, relpersistence,
+ ExclusiveLock);
+ else
+ OIDDelta_old = make_new_heap(matviewOid, tableSpace, relpersistence,
+ ExclusiveLock);
+ LockRelationOid(OIDDelta_old, AccessExclusiveLock);
+ dest_old = CreateTransientRelDestReceiver(OIDDelta_old);
+ }
+
+ /*
+ * Now lock down security-restricted operations.
+ */
+ SetUserIdAndSecContext(relowner,
+ save_sec_context | SECURITY_RESTRICTED_OPERATION);
+
+ /* Generate the data. */
+ if (trigdata->tg_newtable)
+ refresh_matview_datafill(dest_new, new_delta_qry, queryEnv, NULL);
+ if (trigdata->tg_oldtable)
+ refresh_matview_datafill(dest_old, old_delta_qry, queryEnv, NULL);
+
+ PG_TRY();
+ {
+ apply_delta(matviewOid, OIDDelta_new, OIDDelta_old,
+ query, relowner, save_sec_context);
+ }
+ PG_CATCH();
+ {
+ matview_maintenance_depth = old_depth;
+ PG_RE_THROW();
+ }
+ PG_END_TRY();
+
+ /* Pop the original snapshot. */
+ PopActiveSnapshot();
+
+ table_close(matviewRel, NoLock);
+
+ /* Roll back any GUC changes */
+ AtEOXact_GUC(false, save_nestlevel);
+
+ /* Restore userid and security context */
+ SetUserIdAndSecContext(save_userid, save_sec_context);
+
+ return PointerGetDatum(NULL);
+}
+
+static void
+apply_delta(Oid matviewOid, Oid tempOid_new, Oid tempOid_old,
+ Query *query, Oid relowner, int save_sec_context)
+{
+ StringInfoData querybuf;
+ StringInfoData mvatts_buf, diffatts_buf;
+ StringInfoData mv_gkeys_buf, diff_gkeys_buf, updt_gkeys_buf;
+ StringInfoData diff_aggs_buf, update_aggs_old, update_aggs_new;
+ Relation matviewRel;
+ Relation tempRel_new = NULL, tempRel_old = NULL;
+ char *matviewname;
+ char *tempname_new = NULL, *tempname_old = NULL;
+ ListCell *lc;
+ char *sep, *sep_agg;
+ bool with_group = query->groupClause != NULL;
+
+
+ initStringInfo(&querybuf);
+ matviewRel = table_open(matviewOid, NoLock);
+ matviewname = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(matviewRel)),
+ RelationGetRelationName(matviewRel));
+
+ if (OidIsValid(tempOid_new))
+ {
+ tempRel_new = table_open(tempOid_new, NoLock);
+ tempname_new = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(tempRel_new)),
+ RelationGetRelationName(tempRel_new));
+ }
+ if (OidIsValid(tempOid_old))
+ {
+ tempRel_old = table_open(tempOid_old, NoLock);
+ tempname_old = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(tempRel_old)),
+ RelationGetRelationName(tempRel_old));
+ }
+
+ initStringInfo(&mvatts_buf);
+ initStringInfo(&diffatts_buf);
+ initStringInfo(&diff_aggs_buf);
+ initStringInfo(&update_aggs_old);
+ initStringInfo(&update_aggs_new);
+
+ sep = "";
+ sep_agg= "";
+ foreach (lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);;
+
+ if (tle->resjunk)
+ continue;
+
+ appendStringInfo(&mvatts_buf, "%s", sep);
+ appendStringInfo(&diffatts_buf, "%s", sep);
+ sep = ", ";
+
+ appendStringInfo(&mvatts_buf, "%s", quote_qualified_identifier("mv", tle->resname));
+ appendStringInfo(&diffatts_buf, "%s", quote_qualified_identifier("diff", tle->resname));
+ if (query->hasAggs && IsA(tle->expr, Aggref))
+ {
+ Aggref *aggref = (Aggref *) tle->expr;
+ const char *aggname = get_func_name(aggref->aggfnoid);
+
+ appendStringInfo(&update_aggs_old, "%s", sep_agg);
+ appendStringInfo(&update_aggs_new, "%s", sep_agg);
+ appendStringInfo(&diff_aggs_buf, "%s", sep_agg);
+
+ sep_agg = ", ";
+
+ if (!strcmp(aggname, "count"))
+ {
+ appendStringInfo(&update_aggs_old,
+ "%s = %s - %s",
+ quote_qualified_identifier(NULL, tle->resname),
+ quote_qualified_identifier("mv", tle->resname),
+ quote_qualified_identifier("t", tle->resname)
+ );
+ appendStringInfo(&update_aggs_new,
+ "%s = %s + %s",
+ quote_qualified_identifier(NULL, tle->resname),
+ quote_qualified_identifier("mv", tle->resname),
+ quote_qualified_identifier("diff", tle->resname)
+ );
+
+ appendStringInfo(&diff_aggs_buf, "%s",
+ quote_qualified_identifier("diff", tle->resname)
+ );
+ }
+ else if (!strcmp(aggname, "sum"))
+ {
+ appendStringInfo(&update_aggs_old,
+ "%s = CASE WHEN %s = %s THEN NULL ELSE COALESCE(%s,0) - COALESCE(%s, 0) END, "
+ "%s = %s - %s",
+ quote_qualified_identifier(NULL, tle->resname),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("t", makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("mv", tle->resname),
+ quote_qualified_identifier("t", tle->resname),
+ quote_qualified_identifier(NULL, makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("t", makeObjectName("__ivm_count",tle->resname,"_"))
+ );
+ appendStringInfo(&update_aggs_new,
+ "%s = CASE WHEN %s = 0 AND %s = 0 THEN NULL ELSE COALESCE(%s,0) + COALESCE(%s, 0) END, "
+ "%s = %s + %s",
+ quote_qualified_identifier(NULL, tle->resname),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("diff", makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("mv", tle->resname),
+ quote_qualified_identifier("diff", tle->resname),
+ quote_qualified_identifier(NULL, makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("diff", makeObjectName("__ivm_count",tle->resname,"_"))
+ );
+
+ appendStringInfo(&diff_aggs_buf, "%s, %s",
+ quote_qualified_identifier("diff", tle->resname),
+ quote_qualified_identifier("diff", makeObjectName("__ivm_count",tle->resname,"_"))
+ );
+ }
+ else if (!strcmp(aggname, "avg"))
+ {
+ appendStringInfo(&update_aggs_old,
+ "%s = CASE WHEN %s = %s THEN NULL ELSE"
+ "(COALESCE(%s,0) * %s - COALESCE(%s, 0) * %s) / (%s - %s) END, "
+ "%s = %s - %s",
+ quote_qualified_identifier(NULL, tle->resname),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("t", makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("mv", tle->resname),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("t", tle->resname),
+ quote_qualified_identifier("t", makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("t", makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier(NULL, makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("t", makeObjectName("__ivm_count",tle->resname,"_"))
+ );
+ appendStringInfo(&update_aggs_new,
+ "%s = CASE WHEN %s = 0 AND %s = 0 THEN NULL ELSE"
+ " (COALESCE(%s,0) * %s + COALESCE(%s, 0) * %s) / (%s + %s) END, "
+ "%s = %s + %s",
+ quote_qualified_identifier(NULL, tle->resname),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("diff", makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("mv", tle->resname),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("diff", tle->resname),
+ quote_qualified_identifier("diff", makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("diff", makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier(NULL, makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",tle->resname,"_")),
+ quote_qualified_identifier("diff", makeObjectName("__ivm_count",tle->resname,"_"))
+ );
+
+ appendStringInfo(&diff_aggs_buf, "%s, %s",
+ quote_qualified_identifier("diff", tle->resname),
+ quote_qualified_identifier("diff", makeObjectName("__ivm_count",tle->resname,"_"))
+ );
+ }
+ else
+ elog(ERROR, "unsupported aggregate function: %s", aggname);
+
+ }
+ }
+
+ if (query->hasAggs)
+ {
+ initStringInfo(&mv_gkeys_buf);
+ initStringInfo(&diff_gkeys_buf);
+ initStringInfo(&updt_gkeys_buf);
+
+ if (with_group)
+ {
+ sep_agg= "";
+ foreach (lc, query->groupClause)
+ {
+ SortGroupClause *sgcl = (SortGroupClause *) lfirst(lc);
+ TargetEntry *tle = get_sortgroupclause_tle(sgcl, query->targetList);
+
+ appendStringInfo(&mv_gkeys_buf, "%s", sep_agg);
+ appendStringInfo(&diff_gkeys_buf, "%s", sep_agg);
+ appendStringInfo(&updt_gkeys_buf, "%s", sep_agg);
+
+ sep_agg = ", ";
+
+ appendStringInfo(&mv_gkeys_buf, "%s", quote_qualified_identifier("mv", tle->resname));
+ appendStringInfo(&diff_gkeys_buf, "%s", quote_qualified_identifier("diff", tle->resname));
+ appendStringInfo(&updt_gkeys_buf, "%s", quote_qualified_identifier("updt", tle->resname));
+ }
+ }
+ else
+ {
+ appendStringInfo(&mv_gkeys_buf, "1");
+ appendStringInfo(&diff_gkeys_buf, "1");
+ appendStringInfo(&updt_gkeys_buf, "1");
+ }
+ }
+
+ /* Open SPI context. */
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+
+ /* Analyze the temp table with the new contents. */
+ if (tempname_new)
+ {
+ appendStringInfo(&querybuf, "ANALYZE %s", tempname_new);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ if (tempname_old)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf, "ANALYZE %s", tempname_old);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+
+ SetUserIdAndSecContext(relowner,
+ save_sec_context | SECURITY_LOCAL_USERID_CHANGE);
+
+ OpenMatViewIncrementalMaintenance();
+
+ if (query->hasAggs)
+ {
+ if (tempname_old)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH t AS ("
+ " SELECT diff.__ivm_count__, (diff.__ivm_count__ = mv.__ivm_count__) AS for_dlt, mv.ctid"
+ ", %s"
+ " FROM %s AS mv, %s AS diff WHERE (%s) = (%s)"
+ "), updt AS ("
+ " UPDATE %s AS mv SET __ivm_count__ = mv.__ivm_count__ - t.__ivm_count__"
+ ", %s "
+ " FROM t WHERE mv.ctid = t.ctid AND NOT for_dlt"
+ ") DELETE FROM %s AS mv USING t WHERE mv.ctid = t.ctid AND for_dlt;",
+ diff_aggs_buf.data,
+ matviewname, tempname_old, mv_gkeys_buf.data, diff_gkeys_buf.data,
+ matviewname, update_aggs_old.data, matviewname);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_DELETE)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ if (tempname_new)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH updt AS ("
+ " UPDATE %s AS mv SET __ivm_count__ = mv.__ivm_count__ + diff.__ivm_count__"
+ ", %s "
+ " FROM %s AS diff WHERE (%s) = (%s)"
+ " RETURNING %s"
+ ") INSERT INTO %s (SELECT * FROM %s AS diff WHERE (%s) NOT IN (SELECT %s FROM updt));",
+ matviewname, update_aggs_new.data, tempname_new,
+ mv_gkeys_buf.data, diff_gkeys_buf.data, diff_gkeys_buf.data,
+ matviewname, tempname_new, diff_gkeys_buf.data, updt_gkeys_buf.data);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ }
+ else
+ {
+ if (tempname_old)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH t AS ("
+ " SELECT diff.__ivm_count__, (diff.__ivm_count__ = mv.__ivm_count__) AS for_dlt, mv.ctid"
+ " FROM %s AS mv, %s AS diff WHERE (%s) = (%s)"
+ "), updt AS ("
+ " UPDATE %s AS mv SET __ivm_count__ = mv.__ivm_count__ - t.__ivm_count__"
+ " FROM t WHERE mv.ctid = t.ctid AND NOT for_dlt"
+ ") DELETE FROM %s AS mv USING t WHERE mv.ctid = t.ctid AND for_dlt;",
+ matviewname, tempname_old, mvatts_buf.data, diffatts_buf.data, matviewname, matviewname);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_DELETE)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ if (tempname_new)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH updt AS ("
+ " UPDATE %s AS mv SET __ivm_count__ = mv.__ivm_count__ + diff.__ivm_count__"
+ " FROM %s AS diff WHERE (%s) = (%s)"
+ " RETURNING %s"
+ ") INSERT INTO %s (SELECT * FROM %s AS diff WHERE (%s) NOT IN (SELECT * FROM updt));",
+ matviewname, tempname_new, mvatts_buf.data, diffatts_buf.data, diffatts_buf.data, matviewname, tempname_new, diffatts_buf.data);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ }
+
+ /* We're done maintaining the materialized view. */
+ CloseMatViewIncrementalMaintenance();
+
+ if (OidIsValid(tempOid_new))
+ table_close(tempRel_new, NoLock);
+ if (OidIsValid(tempOid_old))
+ table_close(tempRel_old, NoLock);
+
+ table_close(matviewRel, NoLock);
+
+ /* Clean up temp tables. */
+ if (OidIsValid(tempOid_new))
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf, "DROP TABLE %s", tempname_new);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ if (OidIsValid(tempOid_old))
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf, "DROP TABLE %s", tempname_old);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+
+ /* Close SPI context. */
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+}
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 78deade..8262ac0 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2361,6 +2361,7 @@ _copyRangeTblEntry(const RangeTblEntry *from)
COPY_SCALAR_FIELD(relkind);
COPY_SCALAR_FIELD(rellockmode);
COPY_NODE_FIELD(tablesample);
+ COPY_SCALAR_FIELD(relisivm);
COPY_NODE_FIELD(subquery);
COPY_SCALAR_FIELD(security_barrier);
COPY_SCALAR_FIELD(jointype);
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 4f2ebe5..608d477 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2641,6 +2641,7 @@ _equalRangeTblEntry(const RangeTblEntry *a, const RangeTblEntry *b)
COMPARE_SCALAR_FIELD(relkind);
COMPARE_SCALAR_FIELD(rellockmode);
COMPARE_NODE_FIELD(tablesample);
+ COMPARE_SCALAR_FIELD(relisivm);
COMPARE_NODE_FIELD(subquery);
COMPARE_SCALAR_FIELD(security_barrier);
COMPARE_SCALAR_FIELD(jointype);
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 8400dd3..7897dd9 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -3042,6 +3042,7 @@ _outRangeTblEntry(StringInfo str, const RangeTblEntry *node)
WRITE_CHAR_FIELD(relkind);
WRITE_INT_FIELD(rellockmode);
WRITE_NODE_FIELD(tablesample);
+ WRITE_BOOL_FIELD(relisivm);
break;
case RTE_SUBQUERY:
WRITE_NODE_FIELD(subquery);
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 6c2626e..aa01e20 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1366,6 +1366,7 @@ _readRangeTblEntry(void)
READ_CHAR_FIELD(relkind);
READ_INT_FIELD(rellockmode);
READ_NODE_FIELD(tablesample);
+ READ_BOOL_FIELD(relisivm);
break;
case RTE_SUBQUERY:
READ_NODE_FIELD(subquery);
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 208b4a1..fc71280 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -418,6 +418,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -645,7 +646,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -4054,30 +4055,32 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->relkind = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->relkind = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -4094,9 +4097,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -15128,6 +15136,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDEX
| INDEXES
| INHERIT
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
index 77a48b0..f62013d 100644
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -37,6 +37,7 @@
#include "utils/syscache.h"
#include "utils/varlena.h"
+#include "commands/matview.h"
#define MAX_FUZZY_DISTANCE 3
@@ -56,9 +57,10 @@ static void expandTupleDesc(TupleDesc tupdesc, Alias *eref,
int count, int offset,
int rtindex, int sublevels_up,
int location, bool include_dropped,
- List **colnames, List **colvars);
+ List **colnames, List **colvars, bool is_ivm);
static int specialAttNum(const char *attname);
static bool isQueryUsingTempRelation_walker(Node *node, void *context);
+static bool isIvmColumn(const char *s);
/*
@@ -1238,6 +1240,7 @@ addRangeTableEntry(ParseState *pstate,
rte->relid = RelationGetRelid(rel);
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -1317,6 +1320,7 @@ addRangeTableEntryForRelation(ParseState *pstate,
rte->relid = RelationGetRelid(rel);
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -2315,7 +2319,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
expandTupleDesc(tupdesc, rte->eref,
rtfunc->funccolcount, atts_done,
rtindex, sublevels_up, location,
- include_dropped, colnames, colvars);
+ include_dropped, colnames, colvars, false);
}
else if (functypclass == TYPEFUNC_SCALAR)
{
@@ -2567,10 +2571,16 @@ expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up,
expandTupleDesc(rel->rd_att, eref, rel->rd_att->natts, 0,
rtindex, sublevels_up,
location, include_dropped,
- colnames, colvars);
+ colnames, colvars, RelationIsIVM(rel));
relation_close(rel, AccessShareLock);
}
+static bool
+isIvmColumn(const char *s)
+{
+ return (strncmp(s, "__ivm_", 6) == 0);
+}
+
/*
* expandTupleDesc -- expandRTE subroutine
*
@@ -2584,7 +2594,7 @@ static void
expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
int rtindex, int sublevels_up,
int location, bool include_dropped,
- List **colnames, List **colvars)
+ List **colnames, List **colvars, bool is_ivm)
{
ListCell *aliascell = list_head(eref->colnames);
int varattno;
@@ -2605,6 +2615,9 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
{
Form_pg_attribute attr = TupleDescAttr(tupdesc, varattno);
+ if (is_ivm && isIvmColumn(NameStr(attr->attname)) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
if (attr->attisdropped)
{
if (include_dropped)
diff --git a/src/backend/rewrite/rewriteDefine.c b/src/backend/rewrite/rewriteDefine.c
index 7df2b61..5385a03 100644
--- a/src/backend/rewrite/rewriteDefine.c
+++ b/src/backend/rewrite/rewriteDefine.c
@@ -765,7 +765,8 @@ checkRuleResultList(List *targetList, TupleDesc resultDesc, bool isSelect,
attr->atttypmod))));
}
- if (i != resultDesc->natts)
+ /* No check for materialized views since this could have special columns for IVM */
+ if ((!isSelect || requireColumnNameMatch) && i != resultDesc->natts)
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
isSelect ?
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index 5b047d1..0bba2a6 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -41,6 +41,8 @@
#include "utils/lsyscache.h"
#include "utils/rel.h"
+#include "parser/parser.h"
+#include "commands/matview.h"
/* We use a list of these to detect recursion in RewriteQuery */
typedef struct rewrite_event
@@ -1597,6 +1599,50 @@ ApplyRetrieveRule(Query *parsetree,
if (rule->qual != NULL)
elog(ERROR, "cannot handle qualified ON SELECT rule");
+ if (RelationIsIVM(relation))
+ {
+ rule_action = copyObject(linitial(rule->actions));
+
+ if (!rule_action->distinctClause && !rule_action->groupClause && !rule_action->hasAggs)
+ {
+ StringInfoData str;
+ RawStmt *raw;
+ Query *sub;
+
+ if (rule_action->hasDistinctOn)
+ elog(ERROR, "DISTINCT ON is not supported in IVM");
+
+ initStringInfo(&str);
+ appendStringInfo(&str, "SELECT mv.*, __ivm_count__ FROM %s mv, generate_series(1, mv.__ivm_count__)",
+ quote_qualified_identifier(get_namespace_name(RelationGetNamespace(relation)),
+ RelationGetRelationName(relation)));
+
+ raw = (RawStmt*)linitial(raw_parser(str.data));
+ sub = transformStmt(make_parsestate(NULL),raw->stmt);
+
+ rte = rt_fetch(rt_index, parsetree->rtable);
+
+ rte->rtekind = RTE_SUBQUERY;
+ rte->subquery = sub;
+ rte->security_barrier = RelationIsSecurityView(relation);
+ /* Clear fields that should not be set in a subquery RTE */
+ rte->relid = InvalidOid;
+ rte->relkind = 0;
+ rte->rellockmode = 0;
+ rte->tablesample = NULL;
+ rte->inh = false; /* must not be set for a subquery */
+
+ rte->requiredPerms = 0; /* no permission check on subquery itself */
+ rte->checkAsUser = InvalidOid;
+ rte->selectedCols = NULL;
+ rte->insertedCols = NULL;
+ rte->updatedCols = NULL;
+ rte->extraUpdatedCols = NULL;
+ }
+
+ return parsetree;
+ }
+
if (rt_index == parsetree->resultRelation)
{
/*
@@ -1906,7 +1952,8 @@ fireRIRrules(Query *parsetree, List *activeRIRs)
* In that case this test would need to be postponed till after we've
* opened the rel, so that we could check its state.
*/
- if (rte->relkind == RELKIND_MATVIEW)
+ if (rte->relkind == RELKIND_MATVIEW &&
+ (!rte->relisivm || MatViewIncrementalMaintenanceIsEnabled() || parsetree->commandType != CMD_SELECT))
continue;
/*
diff --git a/src/backend/utils/cache/lsyscache.c b/src/backend/utils/cache/lsyscache.c
index c13c08a..296cc53 100644
--- a/src/backend/utils/cache/lsyscache.c
+++ b/src/backend/utils/cache/lsyscache.c
@@ -1845,6 +1845,30 @@ get_rel_relispartition(Oid relid)
}
/*
+ * get_rel_relisivm
+ *
+ * Returns the relisivm flag associated with a given relation.
+ */
+bool
+get_rel_relisivm(Oid relid)
+{
+ HeapTuple tp;
+
+ tp = SearchSysCache1(RELOID, ObjectIdGetDatum(relid));
+ if (HeapTupleIsValid(tp))
+ {
+ Form_pg_class reltup = (Form_pg_class) GETSTRUCT(tp);
+ bool result;
+
+ result = reltup->relisivm;
+ ReleaseSysCache(tp);
+ return result;
+ }
+ else
+ return false;
+}
+
+/*
* get_rel_tablespace
*
* Returns the pg_tablespace OID associated with a given relation.
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 2b992d7..d80bb30 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -1861,6 +1861,8 @@ formrdesc(const char *relationName, Oid relationReltype,
/* ... and they're always populated, too */
relation->rd_rel->relispopulated = true;
+ /* ... and they're always no ivm, too */
+ relation->rd_rel->relisivm = false;
relation->rd_rel->relreplident = REPLICA_IDENTITY_NOTHING;
relation->rd_rel->relpages = 0;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 8b4cd53..926ef24 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -1491,6 +1491,7 @@ describeOneTableDetails(const char *schemaname,
char relpersistence;
char relreplident;
char *relam;
+ bool isivm;
} tableinfo;
bool show_column_details = false;
@@ -1511,6 +1512,7 @@ describeOneTableDetails(const char *schemaname,
"false AS relhasoids, c.relispartition, %s, c.reltablespace, "
"CASE WHEN c.reloftype = 0 THEN '' ELSE c.reloftype::pg_catalog.regtype::pg_catalog.text END, "
"c.relpersistence, c.relreplident, am.amname\n"
+ ",c.relisivm\n"
"FROM pg_catalog.pg_class c\n "
"LEFT JOIN pg_catalog.pg_class tc ON (c.reltoastrelid = tc.oid)\n"
"LEFT JOIN pg_catalog.pg_am am ON (c.relam = am.oid)\n"
@@ -1687,6 +1689,11 @@ describeOneTableDetails(const char *schemaname,
(char *) NULL : pg_strdup(PQgetvalue(res, 0, 14));
else
tableinfo.relam = NULL;
+ /* TODO: This will supported when sversion >= 130000 (or later). */
+ if (pset.sversion >= 120000)
+ tableinfo.isivm = strcmp(PQgetvalue(res, 0, 15), "t") == 0;
+ else
+ tableinfo.isivm = false;
PQclear(res);
res = NULL;
@@ -3260,6 +3267,12 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf, _("Access method: %s"), tableinfo.relam);
printTableAddFooter(&cont, buf.data);
}
+
+ /* Incremental view maintance info */
+ if (verbose && tableinfo.relkind == RELKIND_MATVIEW && tableinfo.isivm)
+ {
+ printTableAddFooter(&cont, _("Incremental view maintenance: yes"));
+ }
}
/* reloptions, if verbose */
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9009b05..7e21dca 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1001,6 +1001,7 @@ static const pgsql_thing_t words_after_create[] = {
{"FOREIGN TABLE", NULL, NULL, NULL},
{"FUNCTION", NULL, NULL, Query_for_list_of_functions},
{"GROUP", Query_for_list_of_roles},
+ {"INCREMENTAL MATERIALIZED VIEW", NULL, NULL, &Query_for_list_of_matviews},
{"INDEX", NULL, NULL, &Query_for_list_of_indexes},
{"LANGUAGE", Query_for_list_of_languages},
{"LARGE OBJECT", NULL, NULL, NULL, THING_NO_CREATE | THING_NO_DROP},
@@ -2486,7 +2487,7 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("SEQUENCE", "TABLE", "VIEW");
/* Complete "CREATE UNLOGGED" with TABLE or MATVIEW */
else if (TailMatches("CREATE", "UNLOGGED"))
- COMPLETE_WITH("TABLE", "MATERIALIZED VIEW");
+ COMPLETE_WITH("TABLE", "MATERIALIZED VIEW", "INCREMENTAL MATERIALIZED VIEW");
/* Complete PARTITION BY with RANGE ( or LIST ( or ... */
else if (TailMatches("PARTITION", "BY"))
COMPLETE_WITH("RANGE (", "LIST (", "HASH (");
@@ -2695,13 +2696,16 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("SELECT");
/* CREATE MATERIALIZED VIEW */
- else if (Matches("CREATE", "MATERIALIZED"))
+ else if (Matches("CREATE", "MATERIALIZED") ||
+ Matches("CREATE", "INCREMENTAL", "MATERIALIZED"))
COMPLETE_WITH("VIEW");
- /* Complete CREATE MATERIALIZED VIEW <name> with AS */
- else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny))
+ /* Complete CREATE MATERIALIZED VIEW <name> with AS */
+ else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny) ||
+ Matches("CREATE", "INCREMENTAL", "MATERIALIZED", "VIEW", MatchAny))
COMPLETE_WITH("AS");
/* Complete "CREATE MATERIALIZED VIEW <sth> AS with "SELECT" */
- else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny, "AS"))
+ else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny, "AS") ||
+ Matches("CREATE", "INCREMENTAL", "MATERIALIZED", "VIEW", MatchAny, "AS"))
COMPLETE_WITH("SELECT");
/* CREATE EVENT TRIGGER */
diff --git a/src/include/catalog/pg_class.dat b/src/include/catalog/pg_class.dat
index 9bcf286..7deda40 100644
--- a/src/include/catalog/pg_class.dat
+++ b/src/include/catalog/pg_class.dat
@@ -27,7 +27,7 @@
relpersistence => 'p', relkind => 'r', relnatts => '31', relchecks => '0',
relhasrules => 'f', relhastriggers => 'f', relhassubclass => 'f',
relrowsecurity => 'f', relforcerowsecurity => 'f', relispopulated => 't',
- relreplident => 'n', relispartition => 'f', relfrozenxid => '3',
+ relreplident => 'n', relispartition => 'f', relisivm => 'f', relfrozenxid => '3',
relminmxid => '1', relacl => '_null_', reloptions => '_null_',
relpartbound => '_null_' },
{ oid => '1249',
@@ -37,7 +37,7 @@
relpersistence => 'p', relkind => 'r', relnatts => '25', relchecks => '0',
relhasrules => 'f', relhastriggers => 'f', relhassubclass => 'f',
relrowsecurity => 'f', relforcerowsecurity => 'f', relispopulated => 't',
- relreplident => 'n', relispartition => 'f', relfrozenxid => '3',
+ relreplident => 'n', relispartition => 'f', relisivm => 'f', relfrozenxid => '3',
relminmxid => '1', relacl => '_null_', reloptions => '_null_',
relpartbound => '_null_' },
{ oid => '1255',
@@ -47,17 +47,17 @@
relpersistence => 'p', relkind => 'r', relnatts => '29', relchecks => '0',
relhasrules => 'f', relhastriggers => 'f', relhassubclass => 'f',
relrowsecurity => 'f', relforcerowsecurity => 'f', relispopulated => 't',
- relreplident => 'n', relispartition => 'f', relfrozenxid => '3',
+ relreplident => 'n', relispartition => 'f', relisivm => 'f', relfrozenxid => '3',
relminmxid => '1', relacl => '_null_', reloptions => '_null_',
relpartbound => '_null_' },
{ oid => '1259',
relname => 'pg_class', reltype => 'pg_class', relam => 'heap',
relfilenode => '0', relpages => '0', reltuples => '0', relallvisible => '0',
reltoastrelid => '0', relhasindex => 'f', relisshared => 'f',
- relpersistence => 'p', relkind => 'r', relnatts => '33', relchecks => '0',
+ relpersistence => 'p', relkind => 'r', relnatts => '34', relchecks => '0',
relhasrules => 'f', relhastriggers => 'f', relhassubclass => 'f',
relrowsecurity => 'f', relforcerowsecurity => 'f', relispopulated => 't',
- relreplident => 'n', relispartition => 'f', relfrozenxid => '3',
+ relreplident => 'n', relispartition => 'f', relisivm => 'f', relfrozenxid => '3',
relminmxid => '1', relacl => '_null_', reloptions => '_null_',
relpartbound => '_null_' },
diff --git a/src/include/catalog/pg_class.h b/src/include/catalog/pg_class.h
index 090b6ba..ff535f5 100644
--- a/src/include/catalog/pg_class.h
+++ b/src/include/catalog/pg_class.h
@@ -116,6 +116,9 @@ CATALOG(pg_class,1259,RelationRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(83,Relat
/* is relation a partition? */
bool relispartition;
+ /* is relation a matview with ivm? */
+ bool relisivm;
+
/* heap for rewrite during DDL, link to original rel */
Oid relrewrite BKI_DEFAULT(0);
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 604470c..141966c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10669,4 +10669,9 @@
proname => 'pg_partition_root', prorettype => 'regclass',
proargtypes => 'regclass', prosrc => 'pg_partition_root' },
+# IVM
+{ oid => '784', descr => 'ivm trigger',
+ proname => 'IVM_immediate_maintenance', provolatile => 'v', prorettype => 'trigger',
+ proargtypes => '', prosrc => 'IVM_immediate_maintenance' },
+
]
diff --git a/src/include/commands/matview.h b/src/include/commands/matview.h
index edf04bf..8dd8193 100644
--- a/src/include/commands/matview.h
+++ b/src/include/commands/matview.h
@@ -23,6 +23,8 @@
extern void SetMatViewPopulatedState(Relation relation, bool newstate);
+extern void SetMatViewIVMState(Relation relation, bool newstate);
+
extern ObjectAddress ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
ParamListInfo params, char *completionTag);
@@ -30,4 +32,6 @@ extern DestReceiver *CreateTransientRelDestReceiver(Oid oid);
extern bool MatViewIncrementalMaintenanceIsEnabled(void);
+extern Datum IVM_immediate_maintenance(PG_FUNCTION_ARGS);
+
#endif /* MATVIEW_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 94ded3c..5c8a5ae 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1002,6 +1002,7 @@ typedef struct RangeTblEntry
char relkind; /* relation kind (see pg_class.relkind) */
int rellockmode; /* lock level that query requires on the rel */
struct TableSampleClause *tablesample; /* sampling info, or NULL */
+ bool relisivm;
/*
* Fields valid for a subquery RTE (else NULL):
@@ -2059,6 +2060,7 @@ typedef struct CreateStmt
char *tablespacename; /* table space to use, or NULL */
char *accessMethod; /* table access method */
bool if_not_exists; /* just do nothing if it already exists? */
+ bool ivm; /* incremental view maintenance is used by materialized view */
} CreateStmt;
/* ----------
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 860a84d..6df58e2 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -117,6 +117,7 @@ typedef struct IntoClause
char *tableSpaceName; /* table space to use, or NULL */
Node *viewQuery; /* materialized view's SELECT query */
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 00ace84..d682ee1 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -198,6 +198,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD)
PG_KEYWORD("inherit", INHERIT, UNRESERVED_KEYWORD)
diff --git a/src/include/utils/lsyscache.h b/src/include/utils/lsyscache.h
index c8df5bf..8fd9193 100644
--- a/src/include/utils/lsyscache.h
+++ b/src/include/utils/lsyscache.h
@@ -129,6 +129,7 @@ extern Oid get_rel_namespace(Oid relid);
extern Oid get_rel_type_id(Oid relid);
extern char get_rel_relkind(Oid relid);
extern bool get_rel_relispartition(Oid relid);
+extern bool get_rel_relisivm(Oid relid);
extern Oid get_rel_tablespace(Oid relid);
extern char get_rel_persistence(Oid relid);
extern Oid get_transform_fromsql(Oid typid, Oid langid, List *trftypes);
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index d35b4a5..5e5856b 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -562,6 +562,8 @@ typedef struct ViewOptions
*/
#define RelationIsPopulated(relation) ((relation)->rd_rel->relispopulated)
+#define RelationIsIVM(relation) ((relation)->rd_rel->relisivm)
+
/*
* RelationIsAccessibleInLogicalDecoding
* True if we need to log enough information to have access via
diff --git a/src/test/regress/expected/incremental_matview.out b/src/test/regress/expected/incremental_matview.out
new file mode 100644
index 0000000..c4c1e8f
--- /dev/null
+++ b/src/test/regress/expected/incremental_matview.out
@@ -0,0 +1,254 @@
+-- create a table to use as a basis for views and materialized views in various combinations
+CREATE TABLE mv_base_a (i int, j int);
+INSERT INTO mv_base_a VALUES
+ (1,10),
+ (2,20),
+ (3,30),
+ (4,40),
+ (5,50);
+CREATE TABLE mv_base_b (i int, k int);
+INSERT INTO mv_base_b VALUES
+ (1,101),
+ (2,102),
+ (3,103),
+ (4,104);
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_1 AS SELECT i,j,k FROM mv_base_a a INNER JOIN mv_base_b b USING(i);
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+ i | j | k
+---+----+-----
+ 1 | 10 | 101
+ 2 | 20 | 102
+ 3 | 30 | 103
+ 4 | 40 | 104
+(4 rows)
+
+-- immediaite maintenance
+BEGIN;
+INSERT INTO mv_base_b VALUES(5,105);
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+ i | j | k
+---+----+-----
+ 1 | 10 | 101
+ 2 | 20 | 102
+ 3 | 30 | 103
+ 4 | 40 | 104
+ 5 | 50 | 105
+(5 rows)
+
+UPDATE mv_base_a SET j = 0 WHERE i = 1;
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+ i | j | k
+---+----+-----
+ 1 | 0 | 101
+ 2 | 20 | 102
+ 3 | 30 | 103
+ 4 | 40 | 104
+ 5 | 50 | 105
+(5 rows)
+
+DELETE FROM mv_base_b WHERE (i,k) = (5,105);
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+ i | j | k
+---+----+-----
+ 1 | 0 | 101
+ 2 | 20 | 102
+ 3 | 30 | 103
+ 4 | 40 | 104
+(4 rows)
+
+ROLLBACK;
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+ i | j | k
+---+----+-----
+ 1 | 10 | 101
+ 2 | 20 | 102
+ 3 | 30 | 103
+ 4 | 40 | 104
+(4 rows)
+
+-- result of materliazied view have DISTINCT clause or the duplicate result.
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_duplicate AS SELECT j FROM mv_base_a;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_distinct AS SELECT DISTINCT j FROM mv_base_a;
+INSERT INTO mv_base_a VALUES(6,20);
+SELECT * FROM mv_ivm_duplicate ORDER BY 1;
+ j
+----
+ 10
+ 20
+ 20
+ 30
+ 40
+ 50
+(6 rows)
+
+SELECT * FROM mv_ivm_distinct ORDER BY 1;
+ j
+----
+ 10
+ 20
+ 30
+ 40
+ 50
+(5 rows)
+
+DELETE FROM mv_base_a WHERE (i,j) = (2,20);
+SELECT * FROM mv_ivm_duplicate ORDER BY 1;
+ j
+----
+ 10
+ 20
+ 30
+ 40
+ 50
+(5 rows)
+
+SELECT * FROM mv_ivm_distinct ORDER BY 1;
+ j
+----
+ 10
+ 20
+ 30
+ 40
+ 50
+(5 rows)
+
+ROLLBACK;
+-- support SUM(), COUNT() and AVG() aggregation function
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_agg AS SELECT i, SUM(j), COUNT(i),AVG(j) FROM mv_base_a GROUP BY i;
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4;
+ i | sum | count | avg
+---+-----+-------+---------------------
+ 1 | 10 | 1 | 10.0000000000000000
+ 2 | 20 | 1 | 20.0000000000000000
+ 3 | 30 | 1 | 30.0000000000000000
+ 4 | 40 | 1 | 40.0000000000000000
+ 5 | 50 | 1 | 50.0000000000000000
+(5 rows)
+
+INSERT INTO mv_base_a VALUES(2,100);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4;
+ i | sum | count | avg
+---+-----+-------+---------------------
+ 1 | 10 | 1 | 10.0000000000000000
+ 2 | 120 | 2 | 60.0000000000000000
+ 3 | 30 | 1 | 30.0000000000000000
+ 4 | 40 | 1 | 40.0000000000000000
+ 5 | 50 | 1 | 50.0000000000000000
+(5 rows)
+
+UPDATE mv_base_a SET j = 200 WHERE (i,j) = (2,100);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4;
+ i | sum | count | avg
+---+-----+-------+----------------------
+ 1 | 10 | 1 | 10.0000000000000000
+ 2 | 220 | 2 | 110.0000000000000000
+ 3 | 30 | 1 | 30.0000000000000000
+ 4 | 40 | 1 | 40.0000000000000000
+ 5 | 50 | 1 | 50.0000000000000000
+(5 rows)
+
+DELETE FROM mv_base_a WHERE (i,j) = (2,200);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4;
+ i | sum | count | avg
+---+-----+-------+---------------------
+ 1 | 10 | 1 | 10.0000000000000000
+ 2 | 20 | 1 | 20.0000000000000000
+ 3 | 30 | 1 | 30.0000000000000000
+ 4 | 40 | 1 | 40.0000000000000000
+ 5 | 50 | 1 | 50.0000000000000000
+(5 rows)
+
+ROLLBACK;
+-- support COUNT(*) aggregation function
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_agg AS SELECT i, SUM(j),COUNT(*) FROM mv_base_a GROUP BY i;
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
+ i | sum | count
+---+-----+-------
+ 1 | 10 | 1
+ 2 | 20 | 1
+ 3 | 30 | 1
+ 4 | 40 | 1
+ 5 | 50 | 1
+(5 rows)
+
+INSERT INTO mv_base_a VALUES(2,100);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
+ i | sum | count
+---+-----+-------
+ 1 | 10 | 1
+ 2 | 120 | 2
+ 3 | 30 | 1
+ 4 | 40 | 1
+ 5 | 50 | 1
+(5 rows)
+
+ROLLBACK;
+-- support having only aggregation function without GROUP clause
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_group AS SELECT SUM(j)FROM mv_base_a;
+SELECT * FROM mv_ivm_group ORDER BY 1;
+ sum
+-----
+ 150
+(1 row)
+
+INSERT INTO mv_base_a VALUES(6,20);
+SELECT * FROM mv_ivm_group ORDER BY 1;
+ sum
+-----
+ 170
+(1 row)
+
+ROLLBACK;
+-- unsupport aggregation function except for SUM(),COUNT(),AVG()
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_min AS SELECT i, MIN(j) FROM mv_base_a GROUP BY i;
+ERROR: aggregate function min is not supported
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_max AS SELECT i, MAX(j) FROM mv_base_a GROUP BY i;
+ERROR: aggregate function max is not supported
+-- known issues: When use AVG() function and values is indivisible, result of AVG() is incorrect.
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_avg_bug AS SELECT i, SUM(j), COUNT(j), AVG(j) FROM mv_base_A GROUP BY i;
+SELECT * FROM mv_ivm_avg_bug ORDER BY 1,2,3;
+ i | sum | count | avg
+---+-----+-------+---------------------
+ 1 | 10 | 1 | 10.0000000000000000
+ 2 | 20 | 1 | 20.0000000000000000
+ 3 | 30 | 1 | 30.0000000000000000
+ 4 | 40 | 1 | 40.0000000000000000
+ 5 | 50 | 1 | 50.0000000000000000
+(5 rows)
+
+INSERT INTO mv_base_a VALUES
+ (1,0),
+ (1,0),
+ (2,30),
+ (2,30);
+SELECT * FROM mv_ivm_avg_bug ORDER BY 1,2,3;
+ i | sum | count | avg
+---+-----+-------+------------------------
+ 1 | 10 | 3 | 3.33333333333333333333
+ 2 | 80 | 3 | 26.6666666666666667
+ 3 | 30 | 1 | 30.0000000000000000
+ 4 | 40 | 1 | 40.0000000000000000
+ 5 | 50 | 1 | 50.0000000000000000
+(5 rows)
+
+DELETE FROM mv_base_a WHERE (i,j) = (1,0);
+DELETE FROM mv_base_a WHERE (i,j) = (2,30);
+SELECT * FROM mv_ivm_avg_bug ORDER BY 1,2,3;
+ i | sum | count | avg
+---+-----+-------+------------------------
+ 1 | 10 | 1 | 9.99999999999999999999
+ 2 | 20 | 1 | 20.0000000000000001
+ 3 | 30 | 1 | 30.0000000000000000
+ 4 | 40 | 1 | 40.0000000000000000
+ 5 | 50 | 1 | 50.0000000000000000
+(5 rows)
+
+ROLLBACK;
+DROP TABLE mv_base_b CASCADE;
+NOTICE: drop cascades to materialized view mv_ivm_1
+DROP TABLE mv_base_a CASCADE;
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 8fb55f0..39e79d8 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -78,7 +78,7 @@ test: brin gin gist spgist privileges init_privs security_label collate matview
# ----------
# Another group of parallel tests
# ----------
-test: create_table_like alter_generic alter_operator misc async dbsize misc_functions sysviews tsrf tidscan
+test: create_table_like alter_generic alter_operator misc async dbsize misc_functions sysviews tsrf tidscan incremental_matview
# rules cannot run concurrently with any test that creates
# a view or rule in the public schema
diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule
index a39ca10..552c80c8 100644
--- a/src/test/regress/serial_schedule
+++ b/src/test/regress/serial_schedule
@@ -112,6 +112,7 @@ test: init_privs
test: security_label
test: collate
test: matview
+test: incremental_matview
test: lock
test: replica_identity
test: rowsecurity
diff --git a/src/test/regress/sql/incremental_matview.sql b/src/test/regress/sql/incremental_matview.sql
new file mode 100644
index 0000000..a05f607
--- /dev/null
+++ b/src/test/regress/sql/incremental_matview.sql
@@ -0,0 +1,91 @@
+
+-- create a table to use as a basis for views and materialized views in various combinations
+CREATE TABLE mv_base_a (i int, j int);
+INSERT INTO mv_base_a VALUES
+ (1,10),
+ (2,20),
+ (3,30),
+ (4,40),
+ (5,50);
+CREATE TABLE mv_base_b (i int, k int);
+INSERT INTO mv_base_b VALUES
+ (1,101),
+ (2,102),
+ (3,103),
+ (4,104);
+
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_1 AS SELECT i,j,k FROM mv_base_a a INNER JOIN mv_base_b b USING(i);
+
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+-- immediaite maintenance
+BEGIN;
+INSERT INTO mv_base_b VALUES(5,105);
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+UPDATE mv_base_a SET j = 0 WHERE i = 1;
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+DELETE FROM mv_base_b WHERE (i,k) = (5,105);
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+ROLLBACK;
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+
+-- result of materliazied view have DISTINCT clause or the duplicate result.
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_duplicate AS SELECT j FROM mv_base_a;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_distinct AS SELECT DISTINCT j FROM mv_base_a;
+INSERT INTO mv_base_a VALUES(6,20);
+SELECT * FROM mv_ivm_duplicate ORDER BY 1;
+SELECT * FROM mv_ivm_distinct ORDER BY 1;
+DELETE FROM mv_base_a WHERE (i,j) = (2,20);
+SELECT * FROM mv_ivm_duplicate ORDER BY 1;
+SELECT * FROM mv_ivm_distinct ORDER BY 1;
+ROLLBACK;
+
+-- support SUM(), COUNT() and AVG() aggregation function
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_agg AS SELECT i, SUM(j), COUNT(i),AVG(j) FROM mv_base_a GROUP BY i;
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4;
+INSERT INTO mv_base_a VALUES(2,100);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4;
+UPDATE mv_base_a SET j = 200 WHERE (i,j) = (2,100);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4;
+DELETE FROM mv_base_a WHERE (i,j) = (2,200);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4;
+ROLLBACK;
+
+-- support COUNT(*) aggregation function
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_agg AS SELECT i, SUM(j),COUNT(*) FROM mv_base_a GROUP BY i;
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
+INSERT INTO mv_base_a VALUES(2,100);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
+ROLLBACK;
+
+-- support having only aggregation function without GROUP clause
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_group AS SELECT SUM(j)FROM mv_base_a;
+SELECT * FROM mv_ivm_group ORDER BY 1;
+INSERT INTO mv_base_a VALUES(6,20);
+SELECT * FROM mv_ivm_group ORDER BY 1;
+ROLLBACK;
+
+-- unsupport aggregation function except for SUM(),COUNT(),AVG()
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_min AS SELECT i, MIN(j) FROM mv_base_a GROUP BY i;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_max AS SELECT i, MAX(j) FROM mv_base_a GROUP BY i;
+
+-- known issues: When use AVG() function and values is indivisible, result of AVG() is incorrect.
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_avg_bug AS SELECT i, SUM(j), COUNT(j), AVG(j) FROM mv_base_A GROUP BY i;
+SELECT * FROM mv_ivm_avg_bug ORDER BY 1,2,3;
+INSERT INTO mv_base_a VALUES
+ (1,0),
+ (1,0),
+ (2,30),
+ (2,30);
+SELECT * FROM mv_ivm_avg_bug ORDER BY 1,2,3;
+DELETE FROM mv_base_a WHERE (i,j) = (1,0);
+DELETE FROM mv_base_a WHERE (i,j) = (2,30);
+SELECT * FROM mv_ivm_avg_bug ORDER BY 1,2,3;
+ROLLBACK;
+
+DROP TABLE mv_base_b CASCADE;
+DROP TABLE mv_base_a CASCADE;
I am quite interested to learn how IVM interacts with SERIALIZABLE.
Just for a fun, I have added:
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
right after every BEGIN; in incremental_matview.sql in regression test
and it seems it works.
A couple of superficial review comments:
+ const char *aggname = get_func_name(aggref->aggfnoid); ... + else if (!strcmp(aggname, "sum"))I guess you need a more robust way to detect the supported aggregates
than their name, or I guess some way for aggregates themselves to
specify that they support this and somehow supply the extra logic.
Perhaps I just waid what Greg Stark already said, except not as well.
I guess we could use moving aggregate (or partial aggregate?)
functions for this purpose, but then we need to run executor directly
rather using SPI. It needs more codes...
+ "WITH t AS (" + " SELECT diff.__ivm_count__, (diff.__ivm_count__ = mv.__ivm_count__) AS for_dlt, mv.ctid" + ", %s" + " FROM %s AS mv, %s AS diff WHERE (%s) = (%s)" + "), updt AS (" + " UPDATE %s AS mv SET __ivm_count__ = mv.__ivm_count__ - t.__ivm_count__" + ", %s " + " FROM t WHERE mv.ctid = t.ctid AND NOT for_dlt" + ") DELETE FROM %s AS mv USING t WHERE mv.ctid = t.ctid AND for_dlt;",I fully understand that this is POC code, but I am curious about one
thing. These queries that are executed by apply_delta() would need to
be converted to C, or at least used reusable plans, right? Hmm,
creating and dropping temporary tables every time is a clue that the
ultimate form of this should be tuplestores and C code, I think,
right?
Yes, we could reuse the temp tables and plans.
Then our unofficial automatic CI system[1] will run these tests every
day, which sometimes finds problems.[1] cfbot.cputube.org
I appreciate that you provide the system.
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
Hi Thomas,
Thank you for your review and discussion on this patch!
2019年7月8日(月) 15:32 Thomas Munro <thomas.munro@gmail.com>:
On Fri, Jun 28, 2019 at 10:56 PM Yugo Nagata <nagata@sraoss.co.jp> wrote:
Attached is a WIP patch of IVM which supports some aggregate functions.
Hi Nagata-san and Hoshiai-san,
Thank you for working on this. I enjoyed your talk at PGCon. I've
added Kevin Grittner just in case he missed this thread; he has talked
often about implementing the counting algorithm, and he wrote the
"trigger transition tables" feature to support exactly this. While
integrating trigger transition tables with the new partition features,
we had to make a number of decisions about how that should work, and
we tried to come up with answers that would work for IMV, and I hope
we made the right choices!
Transition tables is a great feature. I am now using this in my implementation
of IVM, but the first time I used this feature was when I implemented a PoC
for extending view updatability of PostgreSQL[1]https://www.pgcon.org/2017/schedule/events/1074.en.html. At that time, I didn't know
that this feature is made originally aiming to support IVM.
[1]: https://www.pgcon.org/2017/schedule/events/1074.en.html
I think transition tables is a good choice to implement a statement level
immediate view maintenance where materialized views are refreshed in a statement
level after trigger. However, when implementing a transaction level immediate
view maintenance where views are refreshed per transaction, or deferred view
maintenance, we can't update views in a after trigger, and we will need an
infrastructure to manage change logs of base tables. Transition tables can be
used to collect these logs, but using logical decoding of WAL is another candidate.
In any way, if these logs can be collected in a tuplestore, we might able to
convert this to "ephemeral named relation (ENR)" and use this to calculate diff
sets for views.
I am quite interested to learn how IVM interacts with SERIALIZABLE.
Nagata-san has been studying this. Nagata-san, any comment?
In SERIALIZABLE or REPEATABLE READ level, table changes occurred in other
ransactions are not visible, so views can not be maintained correctly in AFTER
triggers. If a view is defined on two tables and each table is modified in
different concurrent transactions respectively, the result of view maintenance
done in trigger functions can be incorrect due to the race condition. This is the
reason why such transactions are aborted immediately in that case in my current
implementation.
One idea to resolve this is performing view maintenance in two phases. Firstly,
views are updated using only table changes visible in this transaction. Then,
just after this transaction is committed, views have to be updated additionally
using changes happened in other transactions to keep consistency. This is a just
idea, but to implement this idea, I think, we will need keep to keep and
maintain change logs.
A couple of superficial review comments:
+ const char *aggname = get_func_name(aggref->aggfnoid); ... + else if (!strcmp(aggname, "sum"))I guess you need a more robust way to detect the supported aggregates
than their name, or I guess some way for aggregates themselves to
specify that they support this and somehow supply the extra logic.
Perhaps I just waid what Greg Stark already said, except not as well.
Yes. Using name is not robust because users can make same name aggregates like
sum(text) (although I am not sure this makes sense). We can use oids instead
of names, but it would be nice to extend pg_aggregate and add new attributes
for informing that this supports IVM and for providing functions for IVM logic.
As for the question of how
to reserve a namespace for system columns that won't clash with user
columns, according to our manual the SQL standard doesn't allow $ in
identifier names, and according to my copy SQL92 "intermediate SQL"
doesn't allow identifiers that end in an underscore. I don't know
what the best answer is but we should probably decide on a something
based the standard.
Ok, so we should use "__ivm_count__" since this ends in "_" at least.
Another idea is that users specify the name of this special column when
defining materialized views with IVM support. This way can avoid the conflict
because users will specify a name which does not appear in the target list.
As for aggregates supports, it may be also possible to make it a restriction
that count(expr) must be in the target list explicitly when sum(expr) or
avg(expr) is included, instead of making hidden column like __ivm_count_sum__,
like Oracle does.
As for how to make internal columns invisible to SELECT *, previously
there have been discussions about doing that using a new flag in
pg_attribute:/messages/by-id/CAEepm=3ZHh=p0nEEnVbs1Dig_UShPzHUcMNAqvDQUgYgcDo-pA@mail.gmail.com
I agree implementing this feature in PostgreSQL since there are at least a few
use cases, IVM and temporal database.
+ "WITH t AS (" + " SELECT diff.__ivm_count__, (diff.__ivm_count__ = mv.__ivm_count__) AS for_dlt, mv.ctid" + ", %s" + " FROM %s AS mv, %s AS diff WHERE (%s) = (%s)" + "), updt AS (" + " UPDATE %s AS mv SET __ivm_count__ = mv.__ivm_count__ - t.__ivm_count__" + ", %s " + " FROM t WHERE mv.ctid = t.ctid AND NOT for_dlt" + ") DELETE FROM %s AS mv USING t WHERE mv.ctid = t.ctid AND for_dlt;",I fully understand that this is POC code, but I am curious about one
thing. These queries that are executed by apply_delta() would need to
be converted to C, or at least used reusable plans, right? Hmm,
creating and dropping temporary tables every time is a clue that the
ultimate form of this should be tuplestores and C code, I think,
right?
I used SPI just because REFRESH CONCURRENTLY uses this, but, as you said,
it is inefficient to create/drop temp tables and perform parse/plan every times.
It seems to be enough to perform this once when creating materialized views or
at the first maintenance time.
Best regards,
Yugo Nagata
--
Yugo Nagata <nagata@sraoss.co.jp>
Hi,
I've updated the wiki page of Incremental View Maintenance.
https://wiki.postgresql.org/wiki/Incremental_View_Maintenance
On Thu, 11 Jul 2019 13:28:04 +0900
Yugo Nagata <nagata@sraoss.co.jp> wrote:
Hi Thomas,
Thank you for your review and discussion on this patch!
2019年7月8日(月) 15:32 Thomas Munro <thomas.munro@gmail.com>:
On Fri, Jun 28, 2019 at 10:56 PM Yugo Nagata <nagata@sraoss.co.jp> wrote:
Attached is a WIP patch of IVM which supports some aggregate functions.
Hi Nagata-san and Hoshiai-san,
Thank you for working on this. I enjoyed your talk at PGCon. I've
added Kevin Grittner just in case he missed this thread; he has talked
often about implementing the counting algorithm, and he wrote the
"trigger transition tables" feature to support exactly this. While
integrating trigger transition tables with the new partition features,
we had to make a number of decisions about how that should work, and
we tried to come up with answers that would work for IMV, and I hope
we made the right choices!Transition tables is a great feature. I am now using this in my implementation
of IVM, but the first time I used this feature was when I implemented a PoC
for extending view updatability of PostgreSQL[1]. At that time, I didn't know
that this feature is made originally aiming to support IVM.[1] https://www.pgcon.org/2017/schedule/events/1074.en.html
I think transition tables is a good choice to implement a statement level
immediate view maintenance where materialized views are refreshed in a statement
level after trigger. However, when implementing a transaction level immediate
view maintenance where views are refreshed per transaction, or deferred view
maintenance, we can't update views in a after trigger, and we will need an
infrastructure to manage change logs of base tables. Transition tables can be
used to collect these logs, but using logical decoding of WAL is another candidate.
In any way, if these logs can be collected in a tuplestore, we might able to
convert this to "ephemeral named relation (ENR)" and use this to calculate diff
sets for views.I am quite interested to learn how IVM interacts with SERIALIZABLE.
Nagata-san has been studying this. Nagata-san, any comment?
In SERIALIZABLE or REPEATABLE READ level, table changes occurred in other
ransactions are not visible, so views can not be maintained correctly in AFTER
triggers. If a view is defined on two tables and each table is modified in
different concurrent transactions respectively, the result of view maintenance
done in trigger functions can be incorrect due to the race condition. This is the
reason why such transactions are aborted immediately in that case in my current
implementation.One idea to resolve this is performing view maintenance in two phases. Firstly,
views are updated using only table changes visible in this transaction. Then,
just after this transaction is committed, views have to be updated additionally
using changes happened in other transactions to keep consistency. This is a just
idea, but to implement this idea, I think, we will need keep to keep and
maintain change logs.A couple of superficial review comments:
+ const char *aggname = get_func_name(aggref->aggfnoid); ... + else if (!strcmp(aggname, "sum"))I guess you need a more robust way to detect the supported aggregates
than their name, or I guess some way for aggregates themselves to
specify that they support this and somehow supply the extra logic.
Perhaps I just waid what Greg Stark already said, except not as well.Yes. Using name is not robust because users can make same name aggregates like
sum(text) (although I am not sure this makes sense). We can use oids instead
of names, but it would be nice to extend pg_aggregate and add new attributes
for informing that this supports IVM and for providing functions for IVM logic.As for the question of how
to reserve a namespace for system columns that won't clash with user
columns, according to our manual the SQL standard doesn't allow $ in
identifier names, and according to my copy SQL92 "intermediate SQL"
doesn't allow identifiers that end in an underscore. I don't know
what the best answer is but we should probably decide on a something
based the standard.Ok, so we should use "__ivm_count__" since this ends in "_" at least.
Another idea is that users specify the name of this special column when
defining materialized views with IVM support. This way can avoid the conflict
because users will specify a name which does not appear in the target list.As for aggregates supports, it may be also possible to make it a restriction
that count(expr) must be in the target list explicitly when sum(expr) or
avg(expr) is included, instead of making hidden column like __ivm_count_sum__,
like Oracle does.As for how to make internal columns invisible to SELECT *, previously
there have been discussions about doing that using a new flag in
pg_attribute:/messages/by-id/CAEepm=3ZHh=p0nEEnVbs1Dig_UShPzHUcMNAqvDQUgYgcDo-pA@mail.gmail.com
I agree implementing this feature in PostgreSQL since there are at least a few
use cases, IVM and temporal database.+ "WITH t AS (" + " SELECT diff.__ivm_count__, (diff.__ivm_count__ = mv.__ivm_count__) AS for_dlt, mv.ctid" + ", %s" + " FROM %s AS mv, %s AS diff WHERE (%s) = (%s)" + "), updt AS (" + " UPDATE %s AS mv SET __ivm_count__ = mv.__ivm_count__ - t.__ivm_count__" + ", %s " + " FROM t WHERE mv.ctid = t.ctid AND NOT for_dlt" + ") DELETE FROM %s AS mv USING t WHERE mv.ctid = t.ctid AND for_dlt;",I fully understand that this is POC code, but I am curious about one
thing. These queries that are executed by apply_delta() would need to
be converted to C, or at least used reusable plans, right? Hmm,
creating and dropping temporary tables every time is a clue that the
ultimate form of this should be tuplestores and C code, I think,
right?I used SPI just because REFRESH CONCURRENTLY uses this, but, as you said,
it is inefficient to create/drop temp tables and perform parse/plan every times.
It seems to be enough to perform this once when creating materialized views or
at the first maintenance time.Best regards,
Yugo Nagata--
Yugo Nagata <nagata@sraoss.co.jp>
--
Yugo Nagata <nagata@sraoss.co.jp>
Hi,
Attached is the latest patch for supporting min and max aggregate functions.
When new tuples are inserted into base tables, if new values are small
(for min) or large (for max), matview just have to be updated with these
new values. Otherwise, old values just remains.
However, in the case of deletion, this is more complicated. If deleted
values exists in matview as current min or max, we have to recomputate
new min or max values from base tables for affected groups, and matview
should be updated with these recomputated values.
Also, regression tests for min/max are also added.
In addition, incremental update algorithm of avg aggregate values is a bit
improved. If an avg result in materialized views is updated incrementally
y using the old avg value, numerical errors in avg values are accumulated
and the values get wrong eventually. To prevent this, both of sum and count
values are contained in views as hidden columns and use them to calculate
new avg value instead of using old avg values.
Regards,
On Fri, 26 Jul 2019 11:35:53 +0900
Yugo Nagata <nagata@sraoss.co.jp> wrote:
Hi,
I've updated the wiki page of Incremental View Maintenance.
https://wiki.postgresql.org/wiki/Incremental_View_Maintenance
On Thu, 11 Jul 2019 13:28:04 +0900
Yugo Nagata <nagata@sraoss.co.jp> wrote:Hi Thomas,
Thank you for your review and discussion on this patch!
2019年7月8日(月) 15:32 Thomas Munro <thomas.munro@gmail.com>:
On Fri, Jun 28, 2019 at 10:56 PM Yugo Nagata <nagata@sraoss.co.jp> wrote:
Attached is a WIP patch of IVM which supports some aggregate functions.
Hi Nagata-san and Hoshiai-san,
Thank you for working on this. I enjoyed your talk at PGCon. I've
added Kevin Grittner just in case he missed this thread; he has talked
often about implementing the counting algorithm, and he wrote the
"trigger transition tables" feature to support exactly this. While
integrating trigger transition tables with the new partition features,
we had to make a number of decisions about how that should work, and
we tried to come up with answers that would work for IMV, and I hope
we made the right choices!Transition tables is a great feature. I am now using this in my implementation
of IVM, but the first time I used this feature was when I implemented a PoC
for extending view updatability of PostgreSQL[1]. At that time, I didn't know
that this feature is made originally aiming to support IVM.[1] https://www.pgcon.org/2017/schedule/events/1074.en.html
I think transition tables is a good choice to implement a statement level
immediate view maintenance where materialized views are refreshed in a statement
level after trigger. However, when implementing a transaction level immediate
view maintenance where views are refreshed per transaction, or deferred view
maintenance, we can't update views in a after trigger, and we will need an
infrastructure to manage change logs of base tables. Transition tables can be
used to collect these logs, but using logical decoding of WAL is another candidate.
In any way, if these logs can be collected in a tuplestore, we might able to
convert this to "ephemeral named relation (ENR)" and use this to calculate diff
sets for views.I am quite interested to learn how IVM interacts with SERIALIZABLE.
Nagata-san has been studying this. Nagata-san, any comment?
In SERIALIZABLE or REPEATABLE READ level, table changes occurred in other
ransactions are not visible, so views can not be maintained correctly in AFTER
triggers. If a view is defined on two tables and each table is modified in
different concurrent transactions respectively, the result of view maintenance
done in trigger functions can be incorrect due to the race condition. This is the
reason why such transactions are aborted immediately in that case in my current
implementation.One idea to resolve this is performing view maintenance in two phases. Firstly,
views are updated using only table changes visible in this transaction. Then,
just after this transaction is committed, views have to be updated additionally
using changes happened in other transactions to keep consistency. This is a just
idea, but to implement this idea, I think, we will need keep to keep and
maintain change logs.A couple of superficial review comments:
+ const char *aggname = get_func_name(aggref->aggfnoid); ... + else if (!strcmp(aggname, "sum"))I guess you need a more robust way to detect the supported aggregates
than their name, or I guess some way for aggregates themselves to
specify that they support this and somehow supply the extra logic.
Perhaps I just waid what Greg Stark already said, except not as well.Yes. Using name is not robust because users can make same name aggregates like
sum(text) (although I am not sure this makes sense). We can use oids instead
of names, but it would be nice to extend pg_aggregate and add new attributes
for informing that this supports IVM and for providing functions for IVM logic.As for the question of how
to reserve a namespace for system columns that won't clash with user
columns, according to our manual the SQL standard doesn't allow $ in
identifier names, and according to my copy SQL92 "intermediate SQL"
doesn't allow identifiers that end in an underscore. I don't know
what the best answer is but we should probably decide on a something
based the standard.Ok, so we should use "__ivm_count__" since this ends in "_" at least.
Another idea is that users specify the name of this special column when
defining materialized views with IVM support. This way can avoid the conflict
because users will specify a name which does not appear in the target list.As for aggregates supports, it may be also possible to make it a restriction
that count(expr) must be in the target list explicitly when sum(expr) or
avg(expr) is included, instead of making hidden column like __ivm_count_sum__,
like Oracle does.As for how to make internal columns invisible to SELECT *, previously
there have been discussions about doing that using a new flag in
pg_attribute:/messages/by-id/CAEepm=3ZHh=p0nEEnVbs1Dig_UShPzHUcMNAqvDQUgYgcDo-pA@mail.gmail.com
I agree implementing this feature in PostgreSQL since there are at least a few
use cases, IVM and temporal database.+ "WITH t AS (" + " SELECT diff.__ivm_count__, (diff.__ivm_count__ = mv.__ivm_count__) AS for_dlt, mv.ctid" + ", %s" + " FROM %s AS mv, %s AS diff WHERE (%s) = (%s)" + "), updt AS (" + " UPDATE %s AS mv SET __ivm_count__ = mv.__ivm_count__ - t.__ivm_count__" + ", %s " + " FROM t WHERE mv.ctid = t.ctid AND NOT for_dlt" + ") DELETE FROM %s AS mv USING t WHERE mv.ctid = t.ctid AND for_dlt;",I fully understand that this is POC code, but I am curious about one
thing. These queries that are executed by apply_delta() would need to
be converted to C, or at least used reusable plans, right? Hmm,
creating and dropping temporary tables every time is a clue that the
ultimate form of this should be tuplestores and C code, I think,
right?I used SPI just because REFRESH CONCURRENTLY uses this, but, as you said,
it is inefficient to create/drop temp tables and perform parse/plan every times.
It seems to be enough to perform this once when creating materialized views or
at the first maintenance time.Best regards,
Yugo Nagata--
Yugo Nagata <nagata@sraoss.co.jp>--
Yugo Nagata <nagata@sraoss.co.jp>
--
Yugo Nagata <nagata@sraoss.co.jp>
Attachments:
WIP_immediate_IVM_v5.patchtext/x-diff; name=WIP_immediate_IVM_v5.patchDownload
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 68ad5071ca..cd1fff8409 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -1952,6 +1952,13 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<entry>True if table or index is a partition</entry>
</row>
+ <row>
+ <entry><structfield>relisivm</structfield></entry>
+ <entry><type>bool</type></entry>
+ <entry></entry>
+ <entry>True if materialized view enables incremental view maintenance</entry>
+ </row>
+
<row>
<entry><structfield>relrewrite</structfield></entry>
<entry><type>oid</type></entry>
diff --git a/doc/src/sgml/ref/create_materialized_view.sgml b/doc/src/sgml/ref/create_materialized_view.sgml
index ec8847ed40..a23366a342 100644
--- a/doc/src/sgml/ref/create_materialized_view.sgml
+++ b/doc/src/sgml/ref/create_materialized_view.sgml
@@ -21,7 +21,7 @@ PostgreSQL documentation
<refsynopsisdiv>
<synopsis>
-CREATE MATERIALIZED VIEW [ IF NOT EXISTS ] <replaceable>table_name</replaceable>
+CREATE [ INCREMENTAL ] MATERIALIZED VIEW [ IF NOT EXISTS ] <replaceable>table_name</replaceable>
[ (<replaceable>column_name</replaceable> [, ...] ) ]
[ USING <replaceable class="parameter">method</replaceable> ]
[ WITH ( <replaceable class="parameter">storage_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] ) ]
@@ -54,6 +54,16 @@ CREATE MATERIALIZED VIEW [ IF NOT EXISTS ] <replaceable>table_name</replaceable>
<title>Parameters</title>
<variablelist>
+ <varlistentry>
+ <term><literal>INCREMENTAL</literal></term>
+ <listitem>
+ <para>
+ If specified, a materialized view enables incremental view maintenance.
+ You can replace only the contents of a materialized view, which based rows are changed.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>IF NOT EXISTS</literal></term>
<listitem>
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index b7bcdd9d0f..cb7e8be84a 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -890,6 +890,7 @@ InsertPgClassTuple(Relation pg_class_desc,
values[Anum_pg_class_relrewrite - 1] = ObjectIdGetDatum(rd_rel->relrewrite);
values[Anum_pg_class_relfrozenxid - 1] = TransactionIdGetDatum(rd_rel->relfrozenxid);
values[Anum_pg_class_relminmxid - 1] = MultiXactIdGetDatum(rd_rel->relminmxid);
+ values[Anum_pg_class_relisivm - 1] = BoolGetDatum(rd_rel->relisivm);
if (relacl != (Datum) 0)
values[Anum_pg_class_relacl - 1] = relacl;
else
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 99ae159f98..c3cbe32ac7 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -910,6 +910,7 @@ index_create(Relation heapRelation,
indexRelation->rd_rel->relowner = heapRelation->rd_rel->relowner;
indexRelation->rd_rel->relam = accessMethodObjectId;
indexRelation->rd_rel->relispartition = OidIsValid(parentIndexRelid);
+ indexRelation->rd_rel->relisivm = false;
/*
* store index's pg_class entry
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index b7d220699f..1ff5d03d59 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -51,6 +51,17 @@
#include "utils/rls.h"
#include "utils/snapmgr.h"
+#include "catalog/pg_trigger.h"
+#include "catalog/pg_type.h"
+#include "commands/trigger.h"
+#include "parser/parser.h"
+#include "parser/parsetree.h"
+#include "parser/parse_func.h"
+#include "nodes/print.h"
+#include "nodes/primnodes.h"
+#include "optimizer/optimizer.h"
+#include "commands/defrem.h"
+
typedef struct
{
@@ -74,6 +85,9 @@ static bool intorel_receive(TupleTableSlot *slot, DestReceiver *self);
static void intorel_shutdown(DestReceiver *self);
static void intorel_destroy(DestReceiver *self);
+static void CreateIvmTrigger(Oid relOid, Oid viewOid, char *matviewname, int16 type);
+static void CreateIvmTriggersOnBaseTables(Query *qry, Node *jtnode, Oid matviewOid, char* matviewname);
+static void check_ivm_restriction_walker(Node *node);
/*
* create_ctas_internal
@@ -109,6 +123,8 @@ create_ctas_internal(List *attrList, IntoClause *into)
create->oncommit = into->onCommit;
create->tablespacename = into->tableSpaceName;
create->if_not_exists = false;
+ /* Using Materialized view only */
+ create->ivm = into->ivm;
create->accessMethod = into->accessMethod;
/*
@@ -239,6 +255,7 @@ ExecCreateTableAs(CreateTableAsStmt *stmt, const char *queryString,
List *rewritten;
PlannedStmt *plan;
QueryDesc *queryDesc;
+ Query *copied_query;
if (stmt->if_not_exists)
{
@@ -256,6 +273,9 @@ ExecCreateTableAs(CreateTableAsStmt *stmt, const char *queryString,
}
}
+ if (is_matview && into->ivm)
+ check_ivm_restriction_walker((Node *) query);
+
/*
* Create the tuple receiver object and insert info it will need
*/
@@ -319,7 +339,130 @@ ExecCreateTableAs(CreateTableAsStmt *stmt, const char *queryString,
* and is executed repeatedly. (See also the same hack in EXPLAIN and
* PREPARE.)
*/
- rewritten = QueryRewrite(copyObject(query));
+
+ copied_query = copyObject(query);
+ if (is_matview && into->ivm)
+ {
+ TargetEntry *tle;
+ Node *node;
+ ParseState *pstate = make_parsestate(NULL);
+ FuncCall *fn;
+
+ pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET;
+
+ /* group keys must be in targetlist */
+ if (copied_query->groupClause)
+ {
+ ListCell *lc;
+ foreach(lc, copied_query->groupClause)
+ {
+ SortGroupClause *scl = (SortGroupClause *) lfirst(lc);
+ TargetEntry *tle = get_sortgroupclause_tle(scl, copied_query->targetList);
+
+ if (tle->resjunk)
+ elog(ERROR, "GROUP BY expression must appear in select list for incremental materialized views");
+ }
+ }
+ else if (!copied_query->hasAggs)
+ copied_query->groupClause = transformDistinctClause(NULL, &copied_query->targetList, copied_query->sortClause, false);
+
+ if (copied_query->hasAggs)
+ {
+ ListCell *lc;
+ List *agg_counts = NIL;
+ AttrNumber next_resno = list_length(copied_query->targetList) + 1;
+ Const *dmy_arg = makeConst(INT4OID,
+ -1,
+ InvalidOid,
+ sizeof(int32),
+ Int32GetDatum(1),
+ false,
+ true); /* pass by value */
+
+ foreach(lc, copied_query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ TargetEntry *tle_count;
+ char *resname = (into->colNames == NIL ? tle->resname : strVal(list_nth(into->colNames, tle->resno-1)));
+
+
+ if (IsA(tle->expr, Aggref))
+ {
+ Aggref *aggref = (Aggref *) tle->expr;
+ const char *aggname = get_func_name(aggref->aggfnoid);
+
+ /* XXX: need some generalization
+ *
+ * Specifically, Using func names is not robust. We can use oids instead
+ * of names, but it would be nice to add some information to pg_aggregate.
+ */
+ if (strcmp(aggname, "sum") !=0
+ && strcmp(aggname, "count") != 0
+ && strcmp(aggname, "avg") != 0
+ && strcmp(aggname, "min") != 0
+ && strcmp(aggname, "max") != 0
+ )
+ elog(ERROR, "aggregate function %s is not supported", aggname);
+
+ /*
+ * For aggregate functions except to count, add count func with the same arg parameters.
+ * Also, add sum func for agv.
+ *
+ * XXX: If there are same expressions explicitly in the target list, we can use this instead
+ * of adding new duplicated one.
+ */
+ if (strcmp(aggname, "count") != 0)
+ {
+ fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+
+ /* Make a Func with a dummy arg, and then override this by the original agg's args. */
+ node = ParseFuncOrColumn(pstate, fn->funcname, list_make1(dmy_arg), NULL, fn, false, -1);
+ ((Aggref *)node)->args = aggref->args;
+
+ tle_count = makeTargetEntry((Expr *) node,
+ next_resno,
+ pstrdup(makeObjectName("__ivm_count",resname, "_")),
+ false);
+ agg_counts = lappend(agg_counts, tle_count);
+ next_resno++;
+ }
+ if (strcmp(aggname, "avg") == 0)
+ {
+ fn = makeFuncCall(list_make1(makeString("sum")), NIL, -1);
+
+ /* Make a Func with a dummy arg, and then override this by the original agg's args. */
+ node = ParseFuncOrColumn(pstate, fn->funcname, list_make1(dmy_arg), NULL, fn, false, -1);
+ ((Aggref *)node)->args = aggref->args;
+
+ tle_count = makeTargetEntry((Expr *) node,
+ next_resno,
+ pstrdup(makeObjectName("__ivm_sum",resname, "_")),
+ false);
+ agg_counts = lappend(agg_counts, tle_count);
+ next_resno++;
+ }
+
+ }
+ }
+ copied_query->targetList = list_concat(copied_query->targetList, agg_counts);
+
+ }
+
+ /* Add count(*) for counting algorithm */
+ fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+ fn->agg_star = true;
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(copied_query->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ copied_query->targetList = lappend(copied_query->targetList, tle);
+ copied_query->hasAggs = true;
+ }
+
+ rewritten = QueryRewrite(copied_query);
/* SELECT should never rewrite to more or less than one SELECT query */
if (list_length(rewritten) != 1)
@@ -378,11 +521,65 @@ ExecCreateTableAs(CreateTableAsStmt *stmt, const char *queryString,
/* Restore userid and security context */
SetUserIdAndSecContext(save_userid, save_sec_context);
+
+
+ if (into->ivm)
+ {
+ char *matviewname;
+ Oid matviewOid = address.objectId;
+ Relation matviewRel = table_open(matviewOid, NoLock);
+ matviewname = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(matviewRel)),
+ RelationGetRelationName(matviewRel));
+ copied_query = copyObject(query);
+ AcquireRewriteLocks(copied_query, true, false);
+
+ CreateIvmTriggersOnBaseTables(copied_query, (Node *)copied_query->jointree, matviewOid, matviewname);
+
+ table_close(matviewRel, NoLock);
+ }
}
return address;
}
+static void CreateIvmTriggersOnBaseTables(Query *qry, Node *jtnode, Oid matviewOid, char* matviewname)
+{
+
+ if (jtnode == NULL)
+ return;
+ if (IsA(jtnode, RangeTblRef))
+ {
+ int rti = ((RangeTblRef *) jtnode)->rtindex;
+ RangeTblEntry *rte = rt_fetch(rti, qry->rtable);
+
+ if (rte->rtekind == RTE_RELATION)
+ {
+ CreateIvmTrigger(rte->relid, matviewOid, matviewname, TRIGGER_TYPE_INSERT);
+ CreateIvmTrigger(rte->relid, matviewOid, matviewname, TRIGGER_TYPE_DELETE);
+ CreateIvmTrigger(rte->relid, matviewOid, matviewname, TRIGGER_TYPE_UPDATE);
+ }
+ else
+ elog(ERROR, "unsupported RTE kind: %d", (int) rte->rtekind);
+ }
+ else if (IsA(jtnode, FromExpr))
+ {
+ FromExpr *f = (FromExpr *) jtnode;
+ ListCell *l;
+
+ foreach(l, f->fromlist)
+ CreateIvmTriggersOnBaseTables(qry, lfirst(l), matviewOid, matviewname);
+ }
+ else if (IsA(jtnode, JoinExpr))
+ {
+ JoinExpr *j = (JoinExpr *) jtnode;
+
+ CreateIvmTriggersOnBaseTables(qry, j->larg, matviewOid, matviewname);
+ CreateIvmTriggersOnBaseTables(qry, j->rarg, matviewOid, matviewname);
+ }
+ else
+ elog(ERROR, "unrecognized node type: %d", (int) nodeTag(jtnode));
+}
+
/*
* GetIntoRelEFlags --- compute executor flags needed for CREATE TABLE AS
*
@@ -547,6 +744,11 @@ intorel_startup(DestReceiver *self, int operation, TupleDesc typeinfo)
if (is_matview && !into->skipData)
SetMatViewPopulatedState(intoRelationDesc, true);
+ /*
+ * Mark relisivm field, if it's a matview and into->ivm is true.
+ */
+ if (is_matview && into->ivm)
+ SetMatViewIVMState(intoRelationDesc, true);
/*
* Fill private fields of myState for use by later routines
*/
@@ -619,3 +821,229 @@ intorel_destroy(DestReceiver *self)
{
pfree(self);
}
+
+static void
+CreateIvmTrigger(Oid relOid, Oid viewOid, char *matviewname, int16 type)
+{
+ CreateTrigStmt *ivm_trigger;
+ List *transitionRels = NIL;
+ ObjectAddress address, refaddr;
+
+ refaddr.classId = RelationRelationId;
+ refaddr.objectId = viewOid;
+ refaddr.objectSubId = 0;
+
+
+ ivm_trigger = makeNode(CreateTrigStmt);
+ ivm_trigger->relation = NULL;
+ ivm_trigger->row = false;
+ ivm_trigger->timing = TRIGGER_TYPE_AFTER;
+
+ ivm_trigger->events = type;
+
+ switch (type)
+ {
+ case TRIGGER_TYPE_INSERT:
+ ivm_trigger->trigname = "IVM_trigger_ins";
+ break;
+ case TRIGGER_TYPE_DELETE:
+ ivm_trigger->trigname = "IVM_trigger_del";
+ break;
+ case TRIGGER_TYPE_UPDATE:
+ ivm_trigger->trigname = "IVM_trigger_upd";
+ break;
+ }
+
+ if (type == TRIGGER_TYPE_INSERT || type == TRIGGER_TYPE_UPDATE)
+ {
+ TriggerTransition *n = makeNode(TriggerTransition);
+ n->name = "ivm_newtable";
+ n->isNew = true;
+ n->isTable = true;
+
+ transitionRels = lappend(transitionRels, n);
+ }
+ if (type == TRIGGER_TYPE_DELETE || type == TRIGGER_TYPE_UPDATE)
+ {
+ TriggerTransition *n = makeNode(TriggerTransition);
+ n->name = "ivm_oldtable";
+ n->isNew = false;
+ n->isTable = true;
+
+ transitionRels = lappend(transitionRels, n);
+ }
+
+ ivm_trigger->funcname = SystemFuncName("IVM_immediate_maintenance");
+
+ ivm_trigger->columns = NIL;
+ ivm_trigger->transitionRels = transitionRels;
+ ivm_trigger->whenClause = NULL;
+ ivm_trigger->isconstraint = false;
+ ivm_trigger->deferrable = false;
+ ivm_trigger->initdeferred = false;
+ ivm_trigger->constrrel = NULL;
+ ivm_trigger->args = list_make1(makeString(matviewname));
+
+ address = CreateTrigger(ivm_trigger, NULL, relOid, InvalidOid, InvalidOid,
+ InvalidOid, InvalidOid, InvalidOid, NULL, true, false);
+
+ recordDependencyOn(&address, &refaddr, DEPENDENCY_AUTO);
+
+ /* Make changes-so-far visible */
+ CommandCounterIncrement();
+}
+
+/*
+ * check_ivm_restriction_walker --- look for specify nodes in the query tree
+ */
+static void
+check_ivm_restriction_walker(Node *node)
+{
+ /* This can recurse, so check for excessive recursion */
+ check_stack_depth();
+
+ if (node == NULL)
+ return;
+ /*
+ * We currently don't support Sub-Query.
+ */
+ if (IsA(node, SubPlan) || IsA(node, SubLink))
+ ereport(ERROR, (errmsg("subquery is not supported with IVM")));
+
+ switch (nodeTag(node))
+ {
+ case T_Query:
+ {
+ Query *qry = (Query *)node;
+ ListCell *lc;
+ /* if contained CTE, return error */
+ if (qry->cteList != NIL)
+ ereport(ERROR, (errmsg("CTE is not supported with IVM")));
+
+ /* if contained VIEW or subquery into RTE, return error */
+ foreach(lc, qry->rtable)
+ {
+ RangeTblEntry *rte = (RangeTblEntry *) lfirst(lc);
+ if (rte->relkind == RELKIND_VIEW ||
+ rte->relkind == RELKIND_MATVIEW)
+ ereport(ERROR, (errmsg("VIEW or MATERIALIZED VIEW is not supported with IVM")));
+ if (rte->rtekind == RTE_SUBQUERY)
+ ereport(ERROR, (errmsg("subquery is not supported with IVM")));
+ }
+
+ /* search in jointree */
+ check_ivm_restriction_walker((Node *) qry->jointree);
+
+ /* search in target lists */
+ foreach(lc, qry->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ check_ivm_restriction_walker((Node *) tle->expr);
+ }
+
+ break;
+ }
+ case T_JoinExpr:
+ {
+ JoinExpr *joinexpr = (JoinExpr *)node;
+ if (joinexpr->jointype > JOIN_INNER)
+ ereport(ERROR, (errmsg("OUTER JOIN is not supported with IVM")));
+ /* left side */
+ check_ivm_restriction_walker((Node *) joinexpr->larg);
+ /* right side */
+ check_ivm_restriction_walker((Node *) joinexpr->rarg);
+ check_ivm_restriction_walker((Node *) joinexpr->quals);
+ }
+ break;
+ case T_FromExpr:
+ {
+ ListCell *lc;
+ FromExpr *fromexpr = (FromExpr *)node;
+ foreach(lc, fromexpr->fromlist)
+ {
+ check_ivm_restriction_walker((Node *) lfirst(lc));
+ }
+ check_ivm_restriction_walker((Node *) fromexpr->quals);
+ }
+ break;
+ case T_Var:
+ {
+ /* if system column, return error */
+ Var *variable = (Var *) node;
+ if (variable->varattno < 0)
+ ereport(ERROR, (errmsg("system column is not supported with IVM")));
+ }
+ break;
+ case T_BoolExpr:
+ {
+ BoolExpr *boolexpr = (BoolExpr *) node;
+ ListCell *lc;
+
+ foreach(lc, boolexpr->args)
+ {
+ Node *arg = (Node *) lfirst(lc);
+ check_ivm_restriction_walker(arg);
+ }
+ break;
+ }
+ case T_NullIfExpr: /* same as OpExpr */
+ case T_DistinctExpr: /* same as OpExpr */
+ case T_OpExpr:
+ {
+ OpExpr *op = (OpExpr *) node;
+ ListCell *lc;
+ foreach(lc, op->args)
+ {
+ Node *arg = (Node *) lfirst(lc);
+ check_ivm_restriction_walker(arg);
+ }
+ break;
+ }
+ case T_CaseExpr:
+ {
+ CaseExpr *caseexpr = (CaseExpr *) node;
+ ListCell *lc;
+ /* result for ELSE clause */
+ check_ivm_restriction_walker((Node *) caseexpr->defresult);
+ /* expr for WHEN clauses */
+ foreach(lc, caseexpr->args)
+ {
+ CaseWhen *when = (CaseWhen *) lfirst(lc);
+ Node *w_expr = (Node *) when->expr;
+ /* result for WHEN clause */
+ check_ivm_restriction_walker((Node *) when->result);
+ /* expr clause*/
+ check_ivm_restriction_walker((Node *) w_expr);
+ }
+ break;
+ }
+ case T_SubLink:
+ {
+ /* Now, not supported */
+/*
+ SubLink *sublink = (SubLink *) node;
+ Query *qry =(Query *) sublink->subselect;
+ if (qry != NULL && qry->jointree != NULL)
+ check_ivm_restriction_walker((Node *) qry->jointree->quals);
+*/
+ break;
+ }
+ case T_SubPlan:
+ {
+ /* Now, not supported */
+ break;
+ }
+ case T_Aggref:
+ case T_GroupingFunc:
+ case T_WindowFunc:
+ case T_FuncExpr:
+ case T_SQLValueFunction:
+ case T_Const:
+ case T_Param:
+ default:
+ /* do nothing */
+ break;
+ }
+
+ return;
+}
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index 537d0e8cef..be235d28ca 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -46,6 +46,15 @@
#include "utils/snapmgr.h"
#include "utils/syscache.h"
+#include "utils/regproc.h"
+#include "nodes/makefuncs.h"
+#include "parser/parse_clause.h"
+#include "parser/parse_func.h"
+#include "nodes/print.h"
+#include "catalog/pg_type_d.h"
+#include "optimizer/optimizer.h"
+#include "commands/defrem.h"
+
typedef struct
{
@@ -60,12 +69,16 @@ typedef struct
static int matview_maintenance_depth = 0;
+static SPIPlanPtr plan_for_recalc_min_max = NULL;
+static SPIPlanPtr plan_for_set_min_max = NULL;
+
static void transientrel_startup(DestReceiver *self, int operation, TupleDesc typeinfo);
static bool transientrel_receive(TupleTableSlot *slot, DestReceiver *self);
static void transientrel_shutdown(DestReceiver *self);
static void transientrel_destroy(DestReceiver *self);
static uint64 refresh_matview_datafill(DestReceiver *dest, Query *query,
- const char *queryString);
+ QueryEnvironment *queryEnv,
+ const char *queryString);
static char *make_temptable_name_n(char *tempname, int n);
static void refresh_by_match_merge(Oid matviewOid, Oid tempOid, Oid relowner,
int save_sec_context);
@@ -74,6 +87,16 @@ static bool is_usable_unique_index(Relation indexRel);
static void OpenMatViewIncrementalMaintenance(void);
static void CloseMatViewIncrementalMaintenance(void);
+static void apply_delta(Oid matviewOid, Oid tempOid_new, Oid tempOid_old,
+ Query *query, Oid relowner, int save_sec_context);
+static SPIPlanPtr get_plan_for_recalc_min_max(Oid matviewOid, const char *min_max_list,
+ const char *group_keys, int nkeys, Oid *keyTypes);
+static SPIPlanPtr get_plan_for_set_min_max(char *matviewname, const char *min_max_list,
+ int num_min_max, Oid *valTypes);
+
+static Query *get_matview_query(Relation matviewRel);
+
+
/*
* SetMatViewPopulatedState
* Mark a materialized view as populated, or not.
@@ -114,6 +137,46 @@ SetMatViewPopulatedState(Relation relation, bool newstate)
CommandCounterIncrement();
}
+/*
+ * SetMatViewIVMState
+ * Mark a materialized view as IVM, or not.
+ *
+ * NOTE: caller must be holding an appropriate lock on the relation.
+ */
+void
+SetMatViewIVMState(Relation relation, bool newstate)
+{
+ Relation pgrel;
+ HeapTuple tuple;
+
+ Assert(relation->rd_rel->relkind == RELKIND_MATVIEW);
+
+ /*
+ * Update relation's pg_class entry. Crucial side-effect: other backends
+ * (and this one too!) are sent SI message to make them rebuild relcache
+ * entries.
+ */
+ pgrel = table_open(RelationRelationId, RowExclusiveLock);
+ tuple = SearchSysCacheCopy1(RELOID,
+ ObjectIdGetDatum(RelationGetRelid(relation)));
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for relation %u",
+ RelationGetRelid(relation));
+
+ ((Form_pg_class) GETSTRUCT(tuple))->relisivm = newstate;
+
+ CatalogTupleUpdate(pgrel, &tuple->t_self, tuple);
+
+ heap_freetuple(tuple);
+ table_close(pgrel, RowExclusiveLock);
+
+ /*
+ * Advance command counter to make the updated pg_class row locally
+ * visible.
+ */
+ CommandCounterIncrement();
+}
+
/*
* ExecRefreshMatView -- execute a REFRESH MATERIALIZED VIEW command
*
@@ -140,8 +203,6 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
{
Oid matviewOid;
Relation matviewRel;
- RewriteRule *rule;
- List *actions;
Query *dataQuery;
Oid tableSpace;
Oid relowner;
@@ -187,32 +248,8 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("CONCURRENTLY and WITH NO DATA options cannot be used together")));
- /*
- * Check that everything is correct for a refresh. Problems at this point
- * are internal errors, so elog is sufficient.
- */
- if (matviewRel->rd_rel->relhasrules == false ||
- matviewRel->rd_rules->numLocks < 1)
- elog(ERROR,
- "materialized view \"%s\" is missing rewrite information",
- RelationGetRelationName(matviewRel));
-
- if (matviewRel->rd_rules->numLocks > 1)
- elog(ERROR,
- "materialized view \"%s\" has too many rules",
- RelationGetRelationName(matviewRel));
-
- rule = matviewRel->rd_rules->rules[0];
- if (rule->event != CMD_SELECT || !(rule->isInstead))
- elog(ERROR,
- "the rule for materialized view \"%s\" is not a SELECT INSTEAD OF rule",
- RelationGetRelationName(matviewRel));
- actions = rule->actions;
- if (list_length(actions) != 1)
- elog(ERROR,
- "the rule for materialized view \"%s\" is not a single action",
- RelationGetRelationName(matviewRel));
+ dataQuery = get_matview_query(matviewRel);
/*
* Check that there is a unique index with no WHERE clause on one or more
@@ -247,12 +284,6 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
errhint("Create a unique index with no WHERE clause on one or more columns of the materialized view.")));
}
- /*
- * The stored query was rewritten at the time of the MV definition, but
- * has not been scribbled on by the planner.
- */
- dataQuery = linitial_node(Query, actions);
-
/*
* Check for active uses of the relation in the current transaction, such
* as open scans.
@@ -311,7 +342,7 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
/* Generate the data, if wanted. */
if (!stmt->skipData)
- processed = refresh_matview_datafill(dest, dataQuery, queryString);
+ processed = refresh_matview_datafill(dest, dataQuery, NULL, queryString);
/* Make the matview match the newly generated data. */
if (concurrent)
@@ -369,6 +400,7 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
*/
static uint64
refresh_matview_datafill(DestReceiver *dest, Query *query,
+ QueryEnvironment *queryEnv,
const char *queryString)
{
List *rewritten;
@@ -405,7 +437,7 @@ refresh_matview_datafill(DestReceiver *dest, Query *query,
/* Create a QueryDesc, redirecting output to our tuple receiver */
queryDesc = CreateQueryDesc(plan, queryString,
GetActiveSnapshot(), InvalidSnapshot,
- dest, NULL, NULL, 0);
+ dest, NULL, queryEnv ? queryEnv: NULL, 0);
/* call ExecutorStart to prepare the plan for execution */
ExecutorStart(queryDesc, 0);
@@ -926,3 +958,1024 @@ CloseMatViewIncrementalMaintenance(void)
matview_maintenance_depth--;
Assert(matview_maintenance_depth >= 0);
}
+
+/*
+ * IVM trigger function
+ */
+
+Datum
+IVM_immediate_maintenance(PG_FUNCTION_ARGS)
+{
+ TriggerData *trigdata = (TriggerData *) fcinfo->context;
+ Relation rel;
+ Oid relid;
+ Oid matviewOid;
+ Query *query, *old_delta_qry, *new_delta_qry;
+ char* matviewname = trigdata->tg_trigger->tgargs[0];
+ List *names;
+ Relation matviewRel;
+ int old_depth = matview_maintenance_depth;
+
+ Oid tableSpace;
+ Oid relowner;
+ Oid OIDDelta_new = InvalidOid;
+ Oid OIDDelta_old = InvalidOid;
+ DestReceiver *dest_new = NULL, *dest_old = NULL;
+ char relpersistence;
+ Oid save_userid;
+ int save_sec_context;
+ int save_nestlevel;
+
+ ParseState *pstate;
+ QueryEnvironment *queryEnv = create_queryEnv();
+
+ Const *dmy_arg = makeConst(INT4OID,
+ -1,
+ InvalidOid,
+ sizeof(int32),
+ Int32GetDatum(1),
+ false,
+ true); /* pass by value */
+
+ /* Create a dummy ParseState for addRangeTableEntryForENR */
+ pstate = make_parsestate(NULL);
+ pstate->p_queryEnv = queryEnv;
+ pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET;
+
+ names = stringToQualifiedNameList(matviewname);
+
+ /*
+ * Wait for concurrent transactions which update this materialized view at READ COMMITED.
+ * This is needed to see changes commited in othre transactions. No wait and raise an error
+ * at REPEATABLE READ or SERIALIZABLE to prevent anormal update of matviews.
+ * XXX: dead-lock is possible here.
+ */
+ if (!IsolationUsesXactSnapshot())
+ matviewOid = RangeVarGetRelid(makeRangeVarFromNameList(names), ExclusiveLock, true);
+ else
+ matviewOid = RangeVarGetRelidExtended(makeRangeVarFromNameList(names), ExclusiveLock, RVR_MISSING_OK | RVR_NOWAIT, NULL, NULL);
+
+ matviewRel = table_open(matviewOid, NoLock);
+
+ /*
+ * Get and push the latast snapshot to see any changes which is commited during waiting in
+ * other transactions at READ COMMITTED level.
+ * XXX: Is this safe?
+ */
+ PushActiveSnapshot(GetTransactionSnapshot());
+
+ /* Make sure it is a materialized view. */
+ if (matviewRel->rd_rel->relkind != RELKIND_MATVIEW)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("\"%s\" is not a materialized view",
+ RelationGetRelationName(matviewRel))));
+
+ rel = trigdata->tg_relation;
+ relid = rel->rd_id;
+
+ /* get view query*/
+ query = get_matview_query(matviewRel);
+
+ new_delta_qry = copyObject(query);
+ old_delta_qry = copyObject(query);
+
+ if (trigdata->tg_newtable)
+ {
+ RangeTblEntry *rte;
+ ListCell *lc;
+
+ TargetEntry *tle;
+ Node *node;
+ FuncCall *fn;
+
+ EphemeralNamedRelation enr =
+ palloc(sizeof(EphemeralNamedRelationData));
+
+ enr->md.name = trigdata->tg_trigger->tgnewtable;
+ enr->md.reliddesc = trigdata->tg_relation->rd_id;
+ enr->md.tupdesc = NULL;
+ enr->md.enrtype = ENR_NAMED_TUPLESTORE;
+ enr->md.enrtuples = tuplestore_tuple_count(trigdata->tg_newtable);
+ enr->reldata = trigdata->tg_newtable;
+ register_ENR(queryEnv, enr);
+
+ rte = addRangeTableEntryForENR(pstate, makeRangeVar(NULL, enr->md.name, -1), true);
+ new_delta_qry->rtable = lappend(new_delta_qry->rtable, rte);
+
+ foreach(lc, new_delta_qry->rtable)
+ {
+ RangeTblEntry *r = (RangeTblEntry*) lfirst(lc);
+ if (r->relid == relid)
+ {
+ lfirst(lc) = rte;
+ break;
+ }
+ }
+
+ if (query->hasAggs)
+ {
+ ListCell *lc;
+ List *agg_counts = NIL;
+ AttrNumber next_resno = list_length(query->targetList) + 1;
+ Node *node;
+
+ foreach(lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ TargetEntry *tle_count;
+
+ if (IsA(tle->expr, Aggref))
+ {
+ Aggref *aggref = (Aggref *) tle->expr;
+ const char *aggname = get_func_name(aggref->aggfnoid);
+
+ /*
+ * For aggregate functions except to count, add count func with the same arg parameters.
+ * Also, add sum func for agv.
+ *
+ * XXX: need some generalization
+ * XXX: If there are same expressions explicitly in the target list, we can use this instead
+ * of adding new duplicated one.
+ */
+ if (strcmp(aggname, "count") != 0)
+ {
+ fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+
+ /* Make a Func with a dummy arg, and then override this by the original agg's args. */
+ node = ParseFuncOrColumn(pstate, fn->funcname, list_make1(dmy_arg), NULL, fn, false, -1);
+ ((Aggref *)node)->args = aggref->args;
+
+ tle_count = makeTargetEntry((Expr *) node,
+ next_resno,
+ NULL,
+ false);
+ agg_counts = lappend(agg_counts, tle_count);
+ next_resno++;
+ }
+ if (strcmp(aggname, "avg") == 0)
+ {
+ fn = makeFuncCall(list_make1(makeString("sum")), NIL, -1);
+
+ /* Make a Func with a dummy arg, and then override this by the original agg's args. */
+ node = ParseFuncOrColumn(pstate, fn->funcname, list_make1(dmy_arg), NULL, fn, false, -1);
+ ((Aggref *)node)->args = aggref->args;
+
+ tle_count = makeTargetEntry((Expr *) node,
+ next_resno,
+ NULL,
+ false);
+ agg_counts = lappend(agg_counts, tle_count);
+ next_resno++;
+ }
+ }
+
+ }
+ new_delta_qry->targetList = list_concat(new_delta_qry->targetList, agg_counts);
+ }
+
+ fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+ fn->agg_star = true;
+ if (!new_delta_qry->groupClause && !new_delta_qry->hasAggs)
+ new_delta_qry->groupClause = transformDistinctClause(NULL, &new_delta_qry->targetList, new_delta_qry->sortClause, false);
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(new_delta_qry->targetList) + 1,
+ NULL,
+ false);
+ new_delta_qry->targetList = lappend(new_delta_qry->targetList, tle);
+ new_delta_qry->hasAggs = true;
+ }
+
+ if (trigdata->tg_oldtable)
+ {
+ RangeTblEntry *rte;
+ ListCell *lc;
+
+ TargetEntry *tle;
+ Node *node;
+ FuncCall *fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+
+ EphemeralNamedRelation enr =
+ palloc(sizeof(EphemeralNamedRelationData));
+
+ enr->md.name = trigdata->tg_trigger->tgoldtable;
+ enr->md.reliddesc = trigdata->tg_relation->rd_id;
+ enr->md.tupdesc = NULL;
+ enr->md.enrtype = ENR_NAMED_TUPLESTORE;
+ enr->md.enrtuples = tuplestore_tuple_count(trigdata->tg_oldtable);
+ enr->reldata = trigdata->tg_oldtable;
+ register_ENR(queryEnv, enr);
+
+ rte = addRangeTableEntryForENR(pstate, makeRangeVar(NULL, enr->md.name, -1), true);
+ old_delta_qry->rtable = lappend(old_delta_qry->rtable, rte);
+
+ foreach(lc, old_delta_qry->rtable)
+ {
+ RangeTblEntry *r = (RangeTblEntry*) lfirst(lc);
+ if (r->relid == relid)
+ {
+ lfirst(lc) = rte;
+ break;
+ }
+ }
+
+ if (query->hasAggs)
+ {
+ ListCell *lc;
+ List *agg_counts = NIL;
+ AttrNumber next_resno = list_length(query->targetList) + 1;
+ Node *node;
+
+ foreach(lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ TargetEntry *tle_count;
+
+ if (IsA(tle->expr, Aggref))
+ {
+ Aggref *aggref = (Aggref *) tle->expr;
+ const char *aggname = get_func_name(aggref->aggfnoid);
+
+ /*
+ * For aggregate functions except to count, add count func with the same arg parameters.
+ * Also, add sum func for agv.
+ *
+ * XXX: need some generalization
+ * XXX: If there are same expressions explicitly in the target list, we can use this instead
+ * of adding new duplicated one.
+ */
+
+ if (strcmp(aggname, "count") != 0)
+ {
+ fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+
+ /* Make a Func with a dummy arg, and then override this by the original agg's args. */
+ node = ParseFuncOrColumn(pstate, fn->funcname, list_make1(dmy_arg), NULL, fn, false, -1);
+ ((Aggref *)node)->args = aggref->args;
+
+ tle_count = makeTargetEntry((Expr *) node,
+ next_resno,
+ NULL,
+ false);
+ agg_counts = lappend(agg_counts, tle_count);
+ next_resno++;
+ }
+ if (strcmp(aggname, "avg") == 0)
+ {
+ fn = makeFuncCall(list_make1(makeString("sum")), NIL, -1);
+
+ /* Make a Func with a dummy arg, and then override this by the original agg's args. */
+ node = ParseFuncOrColumn(pstate, fn->funcname, list_make1(dmy_arg), NULL, fn, false, -1);
+ ((Aggref *)node)->args = aggref->args;
+
+ tle_count = makeTargetEntry((Expr *) node,
+ next_resno,
+ NULL,
+ false);
+ agg_counts = lappend(agg_counts, tle_count);
+ next_resno++;
+ }
+ }
+
+ }
+ old_delta_qry->targetList = list_concat(old_delta_qry->targetList, agg_counts);
+ }
+
+ fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+ fn->agg_star = true;
+
+ if (!old_delta_qry->groupClause && !old_delta_qry->hasAggs)
+ old_delta_qry->groupClause = transformDistinctClause(NULL, &old_delta_qry->targetList, old_delta_qry->sortClause, false);
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+ tle = makeTargetEntry((Expr *) node,
+ list_length(old_delta_qry->targetList) + 1,
+ NULL,
+ false);
+ old_delta_qry->targetList = lappend(old_delta_qry->targetList, tle);
+ old_delta_qry->hasAggs = true;
+ }
+
+
+ /*
+ * Check for active uses of the relation in the current transaction, such
+ * as open scans.
+ *
+ * NB: We count on this to protect us against problems with refreshing the
+ * data using TABLE_INSERT_FROZEN.
+ */
+ CheckTableNotInUse(matviewRel, "REFRESH MATERIALIZED VIEW");
+
+ relowner = matviewRel->rd_rel->relowner;
+
+ /*
+ * Switch to the owner's userid, so that any functions are run as that
+ * user. Also arrange to make GUC variable changes local to this command.
+ * Don't lock it down too tight to create a temporary table just yet. We
+ * will switch modes when we are about to execute user code.
+ */
+ GetUserIdAndSecContext(&save_userid, &save_sec_context);
+ SetUserIdAndSecContext(relowner,
+ save_sec_context | SECURITY_LOCAL_USERID_CHANGE);
+ save_nestlevel = NewGUCNestLevel();
+
+ tableSpace = GetDefaultTablespace(RELPERSISTENCE_TEMP, false);
+ relpersistence = RELPERSISTENCE_TEMP;
+
+ /*
+ * Create the transient table that will receive the regenerated data. Lock
+ * it against access by any other process until commit (by which time it
+ * will be gone).
+ */
+ if (trigdata->tg_newtable)
+ {
+ OIDDelta_new = make_new_heap(matviewOid, tableSpace, relpersistence,
+ ExclusiveLock);
+ LockRelationOid(OIDDelta_new, AccessExclusiveLock);
+ dest_new = CreateTransientRelDestReceiver(OIDDelta_new);
+ }
+ if (trigdata->tg_oldtable)
+ {
+ if (trigdata->tg_newtable)
+ OIDDelta_old = make_new_heap(OIDDelta_new, tableSpace, relpersistence,
+ ExclusiveLock);
+ else
+ OIDDelta_old = make_new_heap(matviewOid, tableSpace, relpersistence,
+ ExclusiveLock);
+ LockRelationOid(OIDDelta_old, AccessExclusiveLock);
+ dest_old = CreateTransientRelDestReceiver(OIDDelta_old);
+ }
+
+ /*
+ * Now lock down security-restricted operations.
+ */
+ SetUserIdAndSecContext(relowner,
+ save_sec_context | SECURITY_RESTRICTED_OPERATION);
+
+ /* Generate the data. */
+ if (trigdata->tg_newtable)
+ refresh_matview_datafill(dest_new, new_delta_qry, queryEnv, NULL);
+ if (trigdata->tg_oldtable)
+ refresh_matview_datafill(dest_old, old_delta_qry, queryEnv, NULL);
+
+ PG_TRY();
+ {
+ apply_delta(matviewOid, OIDDelta_new, OIDDelta_old,
+ query, relowner, save_sec_context);
+ }
+ PG_CATCH();
+ {
+ matview_maintenance_depth = old_depth;
+ PG_RE_THROW();
+ }
+ PG_END_TRY();
+
+ /* Pop the original snapshot. */
+ PopActiveSnapshot();
+
+ table_close(matviewRel, NoLock);
+
+ /* Roll back any GUC changes */
+ AtEOXact_GUC(false, save_nestlevel);
+
+ /* Restore userid and security context */
+ SetUserIdAndSecContext(save_userid, save_sec_context);
+
+ return PointerGetDatum(NULL);
+}
+
+static void
+apply_delta(Oid matviewOid, Oid tempOid_new, Oid tempOid_old,
+ Query *query, Oid relowner, int save_sec_context)
+{
+ StringInfoData querybuf;
+ StringInfoData mvatts_buf, diffatts_buf;
+ StringInfoData mv_gkeys_buf, diff_gkeys_buf, updt_gkeys_buf;
+ StringInfoData diff_aggs_buf, update_aggs_old, update_aggs_new;
+ StringInfoData returning_buf, result_buf;
+ StringInfoData min_or_max_buf;
+ Relation matviewRel;
+ Relation tempRel_new = NULL, tempRel_old = NULL;
+ char *matviewname;
+ char *tempname_new = NULL, *tempname_old = NULL;
+ ListCell *lc;
+ char *sep, *sep_agg;
+ bool with_group = query->groupClause != NULL;
+ int i;
+ bool has_min_or_max = false;
+ int num_group_keys = 0;
+ int num_min_or_max = 0;
+
+
+ initStringInfo(&querybuf);
+ matviewRel = table_open(matviewOid, NoLock);
+ matviewname = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(matviewRel)),
+ RelationGetRelationName(matviewRel));
+
+ if (OidIsValid(tempOid_new))
+ {
+ tempRel_new = table_open(tempOid_new, NoLock);
+ tempname_new = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(tempRel_new)),
+ RelationGetRelationName(tempRel_new));
+ }
+ if (OidIsValid(tempOid_old))
+ {
+ tempRel_old = table_open(tempOid_old, NoLock);
+ tempname_old = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(tempRel_old)),
+ RelationGetRelationName(tempRel_old));
+ }
+
+ initStringInfo(&mvatts_buf);
+ initStringInfo(&diffatts_buf);
+ initStringInfo(&diff_aggs_buf);
+ initStringInfo(&update_aggs_old);
+ initStringInfo(&update_aggs_new);
+ initStringInfo(&returning_buf);
+ initStringInfo(&result_buf);
+ initStringInfo(&min_or_max_buf);
+
+ sep = "";
+ sep_agg= "";
+ i = 0;
+ foreach (lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, i);
+ char *resname = NameStr(attr->attname);
+
+ i++;
+
+
+ if (tle->resjunk)
+ continue;
+
+ appendStringInfo(&mvatts_buf, "%s", sep);
+ appendStringInfo(&diffatts_buf, "%s", sep);
+ sep = ", ";
+
+ appendStringInfo(&mvatts_buf, "%s", quote_qualified_identifier("mv", resname));
+ appendStringInfo(&diffatts_buf, "%s", quote_qualified_identifier("diff", resname));
+ if (query->hasAggs && IsA(tle->expr, Aggref))
+ {
+ Aggref *aggref = (Aggref *) tle->expr;
+ const char *aggname = get_func_name(aggref->aggfnoid);
+ const char *aggtype = format_type_be(aggref->aggtype); /* XXX: should be add_cast_to ? */
+
+ appendStringInfo(&update_aggs_old, "%s", sep_agg);
+ appendStringInfo(&update_aggs_new, "%s", sep_agg);
+ appendStringInfo(&diff_aggs_buf, "%s", sep_agg);
+
+ sep_agg = ", ";
+
+ /* XXX: need some generalization
+ *
+ * Specifically, Using func names is not robust. We can use oids instead
+ * of names, but it would be nice to add some information to pg_aggregate
+ * and handler functions.
+ */
+
+ if (!strcmp(aggname, "count"))
+ {
+ appendStringInfo(&update_aggs_old,
+ "%s = %s - %s",
+ quote_qualified_identifier(NULL, resname),
+ quote_qualified_identifier("mv", resname),
+ quote_qualified_identifier("t", resname)
+ );
+ appendStringInfo(&update_aggs_new,
+ "%s = %s + %s",
+ quote_qualified_identifier(NULL, resname),
+ quote_qualified_identifier("mv", resname),
+ quote_qualified_identifier("diff", resname)
+ );
+
+ appendStringInfo(&diff_aggs_buf, "%s",
+ quote_qualified_identifier("diff", resname)
+ );
+ }
+ else if (!strcmp(aggname, "sum"))
+ {
+ appendStringInfo(&update_aggs_old,
+ "%s = CASE WHEN %s = %s THEN NULL ELSE "
+ "COALESCE(%s,0) - COALESCE(%s, 0) END, "
+ "%s = %s - %s",
+ quote_qualified_identifier(NULL, resname),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",resname,"_")),
+ quote_qualified_identifier("t", makeObjectName("__ivm_count",resname,"_")),
+
+ quote_qualified_identifier("mv", resname),
+ quote_qualified_identifier("t", resname),
+
+ quote_qualified_identifier(NULL, makeObjectName("__ivm_count",resname,"_")),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",resname,"_")),
+ quote_qualified_identifier("t", makeObjectName("__ivm_count",resname,"_"))
+ );
+ appendStringInfo(&update_aggs_new,
+ "%s = CASE WHEN %s = 0 AND %s = 0 THEN NULL ELSE "
+ "COALESCE(%s,0) + COALESCE(%s, 0) END, "
+ "%s = %s + %s",
+ quote_qualified_identifier(NULL, resname),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",resname,"_")),
+ quote_qualified_identifier("diff", makeObjectName("__ivm_count",resname,"_")),
+
+ quote_qualified_identifier("mv", resname),
+ quote_qualified_identifier("diff", resname),
+
+ quote_qualified_identifier(NULL, makeObjectName("__ivm_count",resname,"_")),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",resname,"_")),
+ quote_qualified_identifier("diff", makeObjectName("__ivm_count",resname,"_"))
+ );
+
+ appendStringInfo(&diff_aggs_buf, "%s, %s",
+ quote_qualified_identifier("diff", resname),
+ quote_qualified_identifier("diff", makeObjectName("__ivm_count",resname,"_"))
+ );
+ }
+ else if (!strcmp(aggname, "avg"))
+ {
+ appendStringInfo(&update_aggs_old,
+ "%s = CASE WHEN %s = %s THEN NULL ELSE "
+ "(COALESCE(%s,0) - COALESCE(%s, 0))::%s / (%s - %s) END, "
+ "%s = COALESCE(%s,0) - COALESCE(%s, 0), "
+ "%s = %s - %s",
+
+ quote_qualified_identifier(NULL, resname),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",resname,"_")),
+ quote_qualified_identifier("t", makeObjectName("__ivm_count",resname,"_")),
+
+ quote_qualified_identifier("mv", makeObjectName("__ivm_sum",resname,"_")),
+ quote_qualified_identifier("t", makeObjectName("__ivm_sum",resname,"_")),
+ aggtype,
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",resname,"_")),
+ quote_qualified_identifier("t", makeObjectName("__ivm_count",resname,"_")),
+
+ quote_qualified_identifier(NULL, makeObjectName("__ivm_sum",resname,"_")),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_sum",resname,"_")),
+ quote_qualified_identifier("t", makeObjectName("__ivm_sum",resname,"_")),
+
+ quote_qualified_identifier(NULL, makeObjectName("__ivm_count",resname,"_")),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",resname,"_")),
+ quote_qualified_identifier("t", makeObjectName("__ivm_count",resname,"_"))
+ );
+ appendStringInfo(&update_aggs_new,
+ "%s = CASE WHEN %s = 0 AND %s = 0 THEN NULL ELSE "
+ "(COALESCE(%s,0)+ COALESCE(%s, 0))::%s / (%s + %s) END, "
+ "%s = COALESCE(%s,0) + COALESCE(%s, 0), "
+ "%s = %s + %s",
+
+ quote_qualified_identifier(NULL, resname),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",resname,"_")),
+ quote_qualified_identifier("diff", makeObjectName("__ivm_count",resname,"_")),
+
+ quote_qualified_identifier("mv", makeObjectName("__ivm_sum",resname,"_")),
+ quote_qualified_identifier("diff", makeObjectName("__ivm_sum",resname,"_")),
+ aggtype,
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",resname,"_")),
+ quote_qualified_identifier("diff", makeObjectName("__ivm_count",resname,"_")),
+
+ quote_qualified_identifier(NULL, makeObjectName("__ivm_sum",resname,"_")),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_sum",resname,"_")),
+ quote_qualified_identifier("diff", makeObjectName("__ivm_sum",resname,"_")),
+
+ quote_qualified_identifier(NULL, makeObjectName("__ivm_count",resname,"_")),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",resname,"_")),
+ quote_qualified_identifier("diff", makeObjectName("__ivm_count",resname,"_"))
+ );
+
+ appendStringInfo(&diff_aggs_buf, "%s, %s, %s",
+ quote_qualified_identifier("diff", resname),
+ quote_qualified_identifier("diff", makeObjectName("__ivm_sum",resname,"_")),
+ quote_qualified_identifier("diff", makeObjectName("__ivm_count",resname,"_"))
+ );
+ }
+ else if (!strcmp(aggname, "min") || !strcmp(aggname, "max"))
+ {
+ bool is_min = !strcmp(aggname, "min");
+
+ if (!has_min_or_max)
+ {
+ has_min_or_max = true;
+ appendStringInfo(&returning_buf, "RETURNING mv.ctid, (");
+ }
+ else
+ {
+ appendStringInfo(&returning_buf, " OR ");
+ appendStringInfo(&min_or_max_buf, ",");
+ }
+
+ appendStringInfo(&returning_buf, "%s %s %s",
+ quote_qualified_identifier("mv", resname),
+ is_min ? ">=" : "<=",
+ quote_qualified_identifier("t", resname)
+ );
+ appendStringInfo(&min_or_max_buf, "%s", quote_qualified_identifier(NULL, resname));
+
+ appendStringInfo(&update_aggs_old,
+ "%s = CASE WHEN %s = %s THEN NULL ELSE "
+ "%s END, "
+ "%s = %s - %s",
+ quote_qualified_identifier(NULL, resname),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",resname,"_")),
+ quote_qualified_identifier("t", makeObjectName("__ivm_count",resname,"_")),
+
+ quote_qualified_identifier("mv", resname),
+
+ quote_qualified_identifier(NULL, makeObjectName("__ivm_count",resname,"_")),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",resname,"_")),
+ quote_qualified_identifier("t", makeObjectName("__ivm_count",resname,"_"))
+ );
+ appendStringInfo(&update_aggs_new,
+ "%s = CASE WHEN %s = 0 AND %s = 0 THEN NULL ELSE "
+ "%s(%s,%s) END, "
+ "%s = %s + %s",
+ quote_qualified_identifier(NULL, resname),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",resname,"_")),
+ quote_qualified_identifier("diff", makeObjectName("__ivm_count",resname,"_")),
+
+ is_min ? "least" : "greatest",
+ quote_qualified_identifier("mv", resname),
+ quote_qualified_identifier("diff", resname),
+
+ quote_qualified_identifier(NULL, makeObjectName("__ivm_count",resname,"_")),
+ quote_qualified_identifier("mv", makeObjectName("__ivm_count",resname,"_")),
+ quote_qualified_identifier("diff", makeObjectName("__ivm_count",resname,"_"))
+ );
+
+ appendStringInfo(&diff_aggs_buf, "%s, %s",
+ quote_qualified_identifier("diff", resname),
+ quote_qualified_identifier("diff", makeObjectName("__ivm_count",resname,"_"))
+ );
+
+ num_min_or_max++;
+ }
+ else
+ elog(ERROR, "unsupported aggregate function: %s", aggname);
+
+ }
+ }
+ if (has_min_or_max)
+ appendStringInfo(&returning_buf, ") AS recalc");
+
+ if (query->hasAggs)
+ {
+ initStringInfo(&mv_gkeys_buf);
+ initStringInfo(&diff_gkeys_buf);
+ initStringInfo(&updt_gkeys_buf);
+
+ if (with_group)
+ {
+ sep_agg= "";
+ foreach (lc, query->groupClause)
+ {
+ SortGroupClause *sgcl = (SortGroupClause *) lfirst(lc);
+ TargetEntry *tle = get_sortgroupclause_tle(sgcl, query->targetList);
+
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno-1);
+ char *resname = NameStr(attr->attname);
+
+ appendStringInfo(&mv_gkeys_buf, "%s", sep_agg);
+ appendStringInfo(&diff_gkeys_buf, "%s", sep_agg);
+ appendStringInfo(&updt_gkeys_buf, "%s", sep_agg);
+
+ sep_agg = ", ";
+
+ appendStringInfo(&mv_gkeys_buf, "%s", quote_qualified_identifier("mv", resname));
+ appendStringInfo(&diff_gkeys_buf, "%s", quote_qualified_identifier("diff", resname));
+ appendStringInfo(&updt_gkeys_buf, "%s", quote_qualified_identifier("updt", resname));
+
+ num_group_keys++;
+ }
+
+ if (has_min_or_max)
+ {
+ appendStringInfo(&returning_buf, ", %s", mv_gkeys_buf.data);
+ appendStringInfo(&result_buf, "SELECT ctid AS tid, %s FROM updt WHERE recalc", updt_gkeys_buf.data);
+ }
+
+ }
+ else
+ {
+ appendStringInfo(&mv_gkeys_buf, "1");
+ appendStringInfo(&diff_gkeys_buf, "1");
+ appendStringInfo(&updt_gkeys_buf, "1");
+ }
+ }
+
+
+ /* Open SPI context. */
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+
+ /* Analyze the temp table with the new contents. */
+ if (tempname_new)
+ {
+ appendStringInfo(&querybuf, "ANALYZE %s", tempname_new);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ if (tempname_old)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf, "ANALYZE %s", tempname_old);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+
+ SetUserIdAndSecContext(relowner,
+ save_sec_context | SECURITY_LOCAL_USERID_CHANGE);
+
+ OpenMatViewIncrementalMaintenance();
+
+ if (query->hasAggs)
+ {
+ if (tempname_old)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH t AS ("
+ " SELECT diff.__ivm_count__, "
+ " %s,"
+ " (diff.__ivm_count__ = mv.__ivm_count__) AS for_dlt, "
+ " mv.ctid"
+ " FROM %s AS mv, %s AS diff WHERE (%s) = (%s)"
+ "), updt AS ("
+ " UPDATE %s AS mv SET __ivm_count__ = mv.__ivm_count__ - t.__ivm_count__,"
+ " %s"
+ " FROM t WHERE mv.ctid = t.ctid AND NOT for_dlt"
+ " %s"
+ "), dlt AS ("
+ " DELETE FROM %s AS mv USING t WHERE mv.ctid = t.ctid AND for_dlt "
+ ") %s",
+ diff_aggs_buf.data,
+ matviewname, tempname_old, mv_gkeys_buf.data, diff_gkeys_buf.data,
+ matviewname, update_aggs_old.data,
+ returning_buf.data,
+ matviewname,
+ has_min_or_max ? result_buf.data : "SELECT 1"
+ );
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_SELECT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+
+ if (has_min_or_max && SPI_processed > 0)
+ {
+ SPITupleTable *tuptable_recalc = SPI_tuptable;
+ TupleDesc tupdesc_recalc = tuptable_recalc->tupdesc;
+ int64 processed = SPI_processed;
+ uint64 i;
+ Oid *keyTypes, *minmaxTypes;
+ char *keyNulls, *minmaxNulls;
+ Datum *keyVals, *minmaxVals;
+
+ keyTypes = palloc(sizeof(Oid) * num_group_keys);
+ keyNulls = palloc(sizeof(char) * num_group_keys);
+ keyVals = palloc(sizeof(Datum) * num_group_keys);
+
+ minmaxTypes = palloc(sizeof(Oid) * (num_min_or_max + 1));
+ minmaxNulls = palloc(sizeof(char) * (num_min_or_max + 1));
+ minmaxVals = palloc(sizeof(Datum) * (num_min_or_max + 1));
+
+ Assert(tupdesc_recalc->natts == num_group_keys + 1);
+
+ for (i = 0; i < num_group_keys; i++)
+ keyTypes[i] = TupleDescAttr(tupdesc_recalc, i+1)->atttypid;
+
+ for (i=0; i< processed; i++)
+ {
+ int j;
+ bool isnull;
+ SPIPlanPtr plan;
+ SPITupleTable *tuptable_minmax;
+ TupleDesc tupdesc_minmax;
+
+
+ for (j = 0; j < num_group_keys; j++)
+ {
+ keyVals[j] = SPI_getbinval(tuptable_recalc->vals[i], tupdesc_recalc, j+2, &isnull);
+ if (isnull)
+ keyNulls[j] = 'n';
+ else
+ keyNulls[j] = ' ';
+ }
+
+ plan = get_plan_for_recalc_min_max(matviewOid, min_or_max_buf.data,
+ mv_gkeys_buf.data, num_group_keys, keyTypes);
+
+ if (SPI_execute_plan(plan, keyVals, keyNulls, false, 0) != SPI_OK_SELECT)
+ elog(ERROR, "SPI_execcute_plan");
+ if (SPI_processed != 1)
+ elog(ERROR, "SPI_execcute_plan returned zere or more than one rows");
+
+ tuptable_minmax = SPI_tuptable;
+ tupdesc_minmax = tuptable_minmax->tupdesc;
+
+ Assert(tupdesc_minmax->natts == num_min_or_max);
+
+ for (j = 0; j < tupdesc_minmax->natts; j++)
+ {
+ if (i == 0)
+ minmaxTypes[j] = TupleDescAttr(tupdesc_minmax, j)->atttypid;
+
+ minmaxVals[j] = SPI_getbinval(tuptable_minmax->vals[0], tupdesc_minmax, j+1, &isnull);
+ if (isnull)
+ minmaxNulls[j] = 'n';
+ else
+ minmaxNulls[j] = ' ';
+ }
+ minmaxTypes[j] = TIDOID;
+ minmaxVals[j] = SPI_getbinval(tuptable_recalc->vals[i], tupdesc_recalc, 1, &isnull);
+ minmaxNulls[j] = ' ';
+
+ plan = get_plan_for_set_min_max(matviewname, min_or_max_buf.data,
+ num_min_or_max, minmaxTypes);
+
+ if (SPI_execute_plan(plan, minmaxVals, minmaxNulls, false, 0) != SPI_OK_UPDATE)
+ elog(ERROR, "SPI_execcute_plan");
+
+ }
+ }
+
+ }
+ if (tempname_new)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH updt AS ("
+ " UPDATE %s AS mv SET __ivm_count__ = mv.__ivm_count__ + diff.__ivm_count__"
+ ", %s "
+ " FROM %s AS diff WHERE (%s) = (%s)"
+ " RETURNING %s"
+ ") INSERT INTO %s (SELECT * FROM %s AS diff WHERE (%s) NOT IN (SELECT %s FROM updt));",
+ matviewname, update_aggs_new.data, tempname_new,
+ mv_gkeys_buf.data, diff_gkeys_buf.data, diff_gkeys_buf.data,
+ matviewname, tempname_new, diff_gkeys_buf.data, updt_gkeys_buf.data);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ }
+ else
+ {
+ if (tempname_old)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH t AS ("
+ " SELECT diff.__ivm_count__, (diff.__ivm_count__ = mv.__ivm_count__) AS for_dlt, mv.ctid"
+ " FROM %s AS mv, %s AS diff WHERE (%s) = (%s)"
+ "), updt AS ("
+ " UPDATE %s AS mv SET __ivm_count__ = mv.__ivm_count__ - t.__ivm_count__"
+ " FROM t WHERE mv.ctid = t.ctid AND NOT for_dlt"
+ ") DELETE FROM %s AS mv USING t WHERE mv.ctid = t.ctid AND for_dlt;",
+ matviewname, tempname_old, mvatts_buf.data, diffatts_buf.data, matviewname, matviewname);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_DELETE)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ if (tempname_new)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH updt AS ("
+ " UPDATE %s AS mv SET __ivm_count__ = mv.__ivm_count__ + diff.__ivm_count__"
+ " FROM %s AS diff WHERE (%s) = (%s)"
+ " RETURNING %s"
+ ") INSERT INTO %s (SELECT * FROM %s AS diff WHERE (%s) NOT IN (SELECT * FROM updt));",
+ matviewname, tempname_new, mvatts_buf.data, diffatts_buf.data, diffatts_buf.data, matviewname, tempname_new, diffatts_buf.data);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ }
+
+ /* We're done maintaining the materialized view. */
+ CloseMatViewIncrementalMaintenance();
+
+ if (OidIsValid(tempOid_new))
+ table_close(tempRel_new, NoLock);
+ if (OidIsValid(tempOid_old))
+ table_close(tempRel_old, NoLock);
+
+ table_close(matviewRel, NoLock);
+
+ /* Clean up temp tables. */
+ if (OidIsValid(tempOid_new))
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf, "DROP TABLE %s", tempname_new);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ if (OidIsValid(tempOid_old))
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf, "DROP TABLE %s", tempname_old);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+
+ /* Close SPI context. */
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+}
+
+
+static SPIPlanPtr
+get_plan_for_recalc_min_max(Oid matviewOid, const char *min_max_list,
+ const char *group_keys, int nkeys, Oid *keyTypes)
+{
+ StringInfoData str;
+ char *viewdef;
+ int i;
+
+ if (plan_for_recalc_min_max)
+ return plan_for_recalc_min_max;
+
+ /* get view definition of matview */
+ viewdef = text_to_cstring((text *) DatumGetPointer(
+ DirectFunctionCall1(pg_get_viewdef, ObjectIdGetDatum(matviewOid))));
+ /* get rid of tailling semi-collon */
+ viewdef[strlen(viewdef)-1] = '\0';
+
+ initStringInfo(&str);
+ appendStringInfo(&str, "SELECT %s FROM (%s) mv WHERE (%s) = (",
+ min_max_list, viewdef, group_keys);
+
+ for (i = 1; i <= nkeys; i++)
+ appendStringInfo(&str, "%s$%d", (i==1 ? "" : ", "),i );
+
+ appendStringInfo(&str, ")");
+
+ plan_for_recalc_min_max = SPI_prepare(str.data, nkeys, keyTypes);
+ SPI_keepplan(plan_for_recalc_min_max);
+
+ return plan_for_recalc_min_max;
+}
+
+static SPIPlanPtr
+get_plan_for_set_min_max(char *matviewname, const char *min_max_list,
+ int num_min_max, Oid *valTypes)
+{
+ StringInfoData str;
+ int i;
+
+ if (plan_for_set_min_max)
+ return plan_for_set_min_max;
+
+ initStringInfo(&str);
+ appendStringInfo(&str, "UPDATE %s AS mv SET (%s) = (",
+ matviewname, min_max_list);
+
+ for (i = 1; i <= num_min_max; i++)
+ appendStringInfo(&str, "%s$%d", (i==1 ? "" : ", "), i);
+
+ appendStringInfo(&str, ") WHERE ctid = $%d", i);
+
+ plan_for_set_min_max = SPI_prepare(str.data, num_min_max + 1, valTypes);
+ SPI_keepplan(plan_for_set_min_max);
+
+ return plan_for_set_min_max;
+}
+
+
+/*
+ * get_matview_query - get the Query from a matview's _RETURN rule.
+ */
+static Query *
+get_matview_query(Relation matviewRel)
+{
+ RewriteRule *rule;
+ List * actions;
+
+ /*
+ * Check that everything is correct for a refresh. Problems at this point
+ * are internal errors, so elog is sufficient.
+ */
+ if (matviewRel->rd_rel->relhasrules == false ||
+ matviewRel->rd_rules->numLocks < 1)
+ elog(ERROR,
+ "materialized view \"%s\" is missing rewrite information",
+ RelationGetRelationName(matviewRel));
+
+ if (matviewRel->rd_rules->numLocks > 1)
+ elog(ERROR,
+ "materialized view \"%s\" has too many rules",
+ RelationGetRelationName(matviewRel));
+
+ rule = matviewRel->rd_rules->rules[0];
+ if (rule->event != CMD_SELECT || !(rule->isInstead))
+ elog(ERROR,
+ "the rule for materialized view \"%s\" is not a SELECT INSTEAD OF rule",
+ RelationGetRelationName(matviewRel));
+
+ actions = rule->actions;
+ if (list_length(actions) != 1)
+ elog(ERROR,
+ "the rule for materialized view \"%s\" is not a single action",
+ RelationGetRelationName(matviewRel));
+
+ /*
+ * The stored query was rewritten at the time of the MV definition, but
+ * has not been scribbled on by the planner.
+ */
+ return linitial_node(Query, actions);
+}
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 6414aded0e..4f54258005 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2361,6 +2361,7 @@ _copyRangeTblEntry(const RangeTblEntry *from)
COPY_SCALAR_FIELD(relkind);
COPY_SCALAR_FIELD(rellockmode);
COPY_NODE_FIELD(tablesample);
+ COPY_SCALAR_FIELD(relisivm);
COPY_NODE_FIELD(subquery);
COPY_SCALAR_FIELD(security_barrier);
COPY_SCALAR_FIELD(jointype);
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 4f2ebe5118..608d477bd5 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2641,6 +2641,7 @@ _equalRangeTblEntry(const RangeTblEntry *a, const RangeTblEntry *b)
COMPARE_SCALAR_FIELD(relkind);
COMPARE_SCALAR_FIELD(rellockmode);
COMPARE_NODE_FIELD(tablesample);
+ COMPARE_SCALAR_FIELD(relisivm);
COMPARE_NODE_FIELD(subquery);
COMPARE_SCALAR_FIELD(security_barrier);
COMPARE_SCALAR_FIELD(jointype);
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 86c31a48c9..be07641e82 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -3044,6 +3044,7 @@ _outRangeTblEntry(StringInfo str, const RangeTblEntry *node)
WRITE_CHAR_FIELD(relkind);
WRITE_INT_FIELD(rellockmode);
WRITE_NODE_FIELD(tablesample);
+ WRITE_BOOL_FIELD(relisivm);
break;
case RTE_SUBQUERY:
WRITE_NODE_FIELD(subquery);
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 6c2626ee62..aa01e205c3 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1366,6 +1366,7 @@ _readRangeTblEntry(void)
READ_CHAR_FIELD(relkind);
READ_INT_FIELD(rellockmode);
READ_NODE_FIELD(tablesample);
+ READ_BOOL_FIELD(relisivm);
break;
case RTE_SUBQUERY:
READ_NODE_FIELD(subquery);
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index c97bb367f8..29040126e0 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -418,6 +418,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -645,7 +646,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -4054,30 +4055,32 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->relkind = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->relkind = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -4094,9 +4097,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -15128,6 +15136,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDEX
| INDEXES
| INHERIT
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
index 4dd81507a7..a6c8c3fd4d 100644
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -37,6 +37,7 @@
#include "utils/syscache.h"
#include "utils/varlena.h"
+#include "commands/matview.h"
#define MAX_FUZZY_DISTANCE 3
@@ -56,9 +57,10 @@ static void expandTupleDesc(TupleDesc tupdesc, Alias *eref,
int count, int offset,
int rtindex, int sublevels_up,
int location, bool include_dropped,
- List **colnames, List **colvars);
+ List **colnames, List **colvars, bool is_ivm);
static int specialAttNum(const char *attname);
static bool isQueryUsingTempRelation_walker(Node *node, void *context);
+static bool isIvmColumn(const char *s);
/*
@@ -1241,6 +1243,7 @@ addRangeTableEntry(ParseState *pstate,
rte->relid = RelationGetRelid(rel);
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -1320,6 +1323,7 @@ addRangeTableEntryForRelation(ParseState *pstate,
rte->relid = RelationGetRelid(rel);
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -2318,7 +2322,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
expandTupleDesc(tupdesc, rte->eref,
rtfunc->funccolcount, atts_done,
rtindex, sublevels_up, location,
- include_dropped, colnames, colvars);
+ include_dropped, colnames, colvars, false);
}
else if (functypclass == TYPEFUNC_SCALAR)
{
@@ -2570,10 +2574,16 @@ expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up,
expandTupleDesc(rel->rd_att, eref, rel->rd_att->natts, 0,
rtindex, sublevels_up,
location, include_dropped,
- colnames, colvars);
+ colnames, colvars, RelationIsIVM(rel));
relation_close(rel, AccessShareLock);
}
+static bool
+isIvmColumn(const char *s)
+{
+ return (strncmp(s, "__ivm_", 6) == 0);
+}
+
/*
* expandTupleDesc -- expandRTE subroutine
*
@@ -2587,7 +2597,7 @@ static void
expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
int rtindex, int sublevels_up,
int location, bool include_dropped,
- List **colnames, List **colvars)
+ List **colnames, List **colvars, bool is_ivm)
{
ListCell *aliascell;
int varattno;
@@ -2600,6 +2610,9 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
{
Form_pg_attribute attr = TupleDescAttr(tupdesc, varattno);
+ if (is_ivm && isIvmColumn(NameStr(attr->attname)) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
if (attr->attisdropped)
{
if (include_dropped)
diff --git a/src/backend/rewrite/rewriteDefine.c b/src/backend/rewrite/rewriteDefine.c
index 7df2b6154c..5385a038b1 100644
--- a/src/backend/rewrite/rewriteDefine.c
+++ b/src/backend/rewrite/rewriteDefine.c
@@ -765,7 +765,8 @@ checkRuleResultList(List *targetList, TupleDesc resultDesc, bool isSelect,
attr->atttypmod))));
}
- if (i != resultDesc->natts)
+ /* No check for materialized views since this could have special columns for IVM */
+ if ((!isSelect || requireColumnNameMatch) && i != resultDesc->natts)
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
isSelect ?
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index c96ef8595a..5f2cb21b3f 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -41,6 +41,8 @@
#include "utils/lsyscache.h"
#include "utils/rel.h"
+#include "parser/parser.h"
+#include "commands/matview.h"
/* We use a list of these to detect recursion in RewriteQuery */
typedef struct rewrite_event
@@ -1597,6 +1599,50 @@ ApplyRetrieveRule(Query *parsetree,
if (rule->qual != NULL)
elog(ERROR, "cannot handle qualified ON SELECT rule");
+ if (RelationIsIVM(relation))
+ {
+ rule_action = copyObject(linitial(rule->actions));
+
+ if (!rule_action->distinctClause && !rule_action->groupClause && !rule_action->hasAggs)
+ {
+ StringInfoData str;
+ RawStmt *raw;
+ Query *sub;
+
+ if (rule_action->hasDistinctOn)
+ elog(ERROR, "DISTINCT ON is not supported in IVM");
+
+ initStringInfo(&str);
+ appendStringInfo(&str, "SELECT mv.*, __ivm_count__ FROM %s mv, generate_series(1, mv.__ivm_count__)",
+ quote_qualified_identifier(get_namespace_name(RelationGetNamespace(relation)),
+ RelationGetRelationName(relation)));
+
+ raw = (RawStmt*)linitial(raw_parser(str.data));
+ sub = transformStmt(make_parsestate(NULL),raw->stmt);
+
+ rte = rt_fetch(rt_index, parsetree->rtable);
+
+ rte->rtekind = RTE_SUBQUERY;
+ rte->subquery = sub;
+ rte->security_barrier = RelationIsSecurityView(relation);
+ /* Clear fields that should not be set in a subquery RTE */
+ rte->relid = InvalidOid;
+ rte->relkind = 0;
+ rte->rellockmode = 0;
+ rte->tablesample = NULL;
+ rte->inh = false; /* must not be set for a subquery */
+
+ rte->requiredPerms = 0; /* no permission check on subquery itself */
+ rte->checkAsUser = InvalidOid;
+ rte->selectedCols = NULL;
+ rte->insertedCols = NULL;
+ rte->updatedCols = NULL;
+ rte->extraUpdatedCols = NULL;
+ }
+
+ return parsetree;
+ }
+
if (rt_index == parsetree->resultRelation)
{
/*
@@ -1906,7 +1952,8 @@ fireRIRrules(Query *parsetree, List *activeRIRs)
* In that case this test would need to be postponed till after we've
* opened the rel, so that we could check its state.
*/
- if (rte->relkind == RELKIND_MATVIEW)
+ if (rte->relkind == RELKIND_MATVIEW &&
+ (!rte->relisivm || MatViewIncrementalMaintenanceIsEnabled() || parsetree->commandType != CMD_SELECT))
continue;
/*
diff --git a/src/backend/utils/cache/lsyscache.c b/src/backend/utils/cache/lsyscache.c
index c13c08a97b..296cc53ffd 100644
--- a/src/backend/utils/cache/lsyscache.c
+++ b/src/backend/utils/cache/lsyscache.c
@@ -1844,6 +1844,30 @@ get_rel_relispartition(Oid relid)
return false;
}
+/*
+ * get_rel_relisivm
+ *
+ * Returns the relisivm flag associated with a given relation.
+ */
+bool
+get_rel_relisivm(Oid relid)
+{
+ HeapTuple tp;
+
+ tp = SearchSysCache1(RELOID, ObjectIdGetDatum(relid));
+ if (HeapTupleIsValid(tp))
+ {
+ Form_pg_class reltup = (Form_pg_class) GETSTRUCT(tp);
+ bool result;
+
+ result = reltup->relisivm;
+ ReleaseSysCache(tp);
+ return result;
+ }
+ else
+ return false;
+}
+
/*
* get_rel_tablespace
*
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 248860758c..531da6f879 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -1860,6 +1860,8 @@ formrdesc(const char *relationName, Oid relationReltype,
/* ... and they're always populated, too */
relation->rd_rel->relispopulated = true;
+ /* ... and they're always no ivm, too */
+ relation->rd_rel->relisivm = false;
relation->rd_rel->relreplident = REPLICA_IDENTITY_NOTHING;
relation->rd_rel->relpages = 0;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 774cc764ff..cc7388d412 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -1491,6 +1491,7 @@ describeOneTableDetails(const char *schemaname,
char relpersistence;
char relreplident;
char *relam;
+ bool isivm;
} tableinfo;
bool show_column_details = false;
@@ -1511,6 +1512,7 @@ describeOneTableDetails(const char *schemaname,
"false AS relhasoids, c.relispartition, %s, c.reltablespace, "
"CASE WHEN c.reloftype = 0 THEN '' ELSE c.reloftype::pg_catalog.regtype::pg_catalog.text END, "
"c.relpersistence, c.relreplident, am.amname\n"
+ ",c.relisivm\n"
"FROM pg_catalog.pg_class c\n "
"LEFT JOIN pg_catalog.pg_class tc ON (c.reltoastrelid = tc.oid)\n"
"LEFT JOIN pg_catalog.pg_am am ON (c.relam = am.oid)\n"
@@ -1687,6 +1689,10 @@ describeOneTableDetails(const char *schemaname,
(char *) NULL : pg_strdup(PQgetvalue(res, 0, 14));
else
tableinfo.relam = NULL;
+ if (pset.sversion >= 130000)
+ tableinfo.isivm = strcmp(PQgetvalue(res, 0, 15), "t") == 0;
+ else
+ tableinfo.isivm = false;
PQclear(res);
res = NULL;
@@ -3291,6 +3297,12 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf, _("Access method: %s"), tableinfo.relam);
printTableAddFooter(&cont, buf.data);
}
+
+ /* Incremental view maintance info */
+ if (verbose && tableinfo.relkind == RELKIND_MATVIEW && tableinfo.isivm)
+ {
+ printTableAddFooter(&cont, _("Incremental view maintenance: yes"));
+ }
}
/* reloptions, if verbose */
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 3f7001fb69..c481b3a2e6 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1001,6 +1001,7 @@ static const pgsql_thing_t words_after_create[] = {
{"FOREIGN TABLE", NULL, NULL, NULL},
{"FUNCTION", NULL, NULL, Query_for_list_of_functions},
{"GROUP", Query_for_list_of_roles},
+ {"INCREMENTAL MATERIALIZED VIEW", NULL, NULL, &Query_for_list_of_matviews},
{"INDEX", NULL, NULL, &Query_for_list_of_indexes},
{"LANGUAGE", Query_for_list_of_languages},
{"LARGE OBJECT", NULL, NULL, NULL, THING_NO_CREATE | THING_NO_DROP},
@@ -2486,7 +2487,7 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("SEQUENCE", "TABLE", "VIEW");
/* Complete "CREATE UNLOGGED" with TABLE or MATVIEW */
else if (TailMatches("CREATE", "UNLOGGED"))
- COMPLETE_WITH("TABLE", "MATERIALIZED VIEW");
+ COMPLETE_WITH("TABLE", "MATERIALIZED VIEW", "INCREMENTAL MATERIALIZED VIEW");
/* Complete PARTITION BY with RANGE ( or LIST ( or ... */
else if (TailMatches("PARTITION", "BY"))
COMPLETE_WITH("RANGE (", "LIST (", "HASH (");
@@ -2734,13 +2735,16 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("SELECT");
/* CREATE MATERIALIZED VIEW */
- else if (Matches("CREATE", "MATERIALIZED"))
+ else if (Matches("CREATE", "MATERIALIZED") ||
+ Matches("CREATE", "INCREMENTAL", "MATERIALIZED"))
COMPLETE_WITH("VIEW");
- /* Complete CREATE MATERIALIZED VIEW <name> with AS */
- else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny))
+ /* Complete CREATE MATERIALIZED VIEW <name> with AS */
+ else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny) ||
+ Matches("CREATE", "INCREMENTAL", "MATERIALIZED", "VIEW", MatchAny))
COMPLETE_WITH("AS");
/* Complete "CREATE MATERIALIZED VIEW <sth> AS with "SELECT" */
- else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny, "AS"))
+ else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny, "AS") ||
+ Matches("CREATE", "INCREMENTAL", "MATERIALIZED", "VIEW", MatchAny, "AS"))
COMPLETE_WITH("SELECT");
/* CREATE EVENT TRIGGER */
diff --git a/src/include/catalog/pg_class.dat b/src/include/catalog/pg_class.dat
index 9bcf28676d..7deda405af 100644
--- a/src/include/catalog/pg_class.dat
+++ b/src/include/catalog/pg_class.dat
@@ -27,7 +27,7 @@
relpersistence => 'p', relkind => 'r', relnatts => '31', relchecks => '0',
relhasrules => 'f', relhastriggers => 'f', relhassubclass => 'f',
relrowsecurity => 'f', relforcerowsecurity => 'f', relispopulated => 't',
- relreplident => 'n', relispartition => 'f', relfrozenxid => '3',
+ relreplident => 'n', relispartition => 'f', relisivm => 'f', relfrozenxid => '3',
relminmxid => '1', relacl => '_null_', reloptions => '_null_',
relpartbound => '_null_' },
{ oid => '1249',
@@ -37,7 +37,7 @@
relpersistence => 'p', relkind => 'r', relnatts => '25', relchecks => '0',
relhasrules => 'f', relhastriggers => 'f', relhassubclass => 'f',
relrowsecurity => 'f', relforcerowsecurity => 'f', relispopulated => 't',
- relreplident => 'n', relispartition => 'f', relfrozenxid => '3',
+ relreplident => 'n', relispartition => 'f', relisivm => 'f', relfrozenxid => '3',
relminmxid => '1', relacl => '_null_', reloptions => '_null_',
relpartbound => '_null_' },
{ oid => '1255',
@@ -47,17 +47,17 @@
relpersistence => 'p', relkind => 'r', relnatts => '29', relchecks => '0',
relhasrules => 'f', relhastriggers => 'f', relhassubclass => 'f',
relrowsecurity => 'f', relforcerowsecurity => 'f', relispopulated => 't',
- relreplident => 'n', relispartition => 'f', relfrozenxid => '3',
+ relreplident => 'n', relispartition => 'f', relisivm => 'f', relfrozenxid => '3',
relminmxid => '1', relacl => '_null_', reloptions => '_null_',
relpartbound => '_null_' },
{ oid => '1259',
relname => 'pg_class', reltype => 'pg_class', relam => 'heap',
relfilenode => '0', relpages => '0', reltuples => '0', relallvisible => '0',
reltoastrelid => '0', relhasindex => 'f', relisshared => 'f',
- relpersistence => 'p', relkind => 'r', relnatts => '33', relchecks => '0',
+ relpersistence => 'p', relkind => 'r', relnatts => '34', relchecks => '0',
relhasrules => 'f', relhastriggers => 'f', relhassubclass => 'f',
relrowsecurity => 'f', relforcerowsecurity => 'f', relispopulated => 't',
- relreplident => 'n', relispartition => 'f', relfrozenxid => '3',
+ relreplident => 'n', relispartition => 'f', relisivm => 'f', relfrozenxid => '3',
relminmxid => '1', relacl => '_null_', reloptions => '_null_',
relpartbound => '_null_' },
diff --git a/src/include/catalog/pg_class.h b/src/include/catalog/pg_class.h
index 090b6ba907..ff535f5504 100644
--- a/src/include/catalog/pg_class.h
+++ b/src/include/catalog/pg_class.h
@@ -116,6 +116,9 @@ CATALOG(pg_class,1259,RelationRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(83,Relat
/* is relation a partition? */
bool relispartition;
+ /* is relation a matview with ivm? */
+ bool relisivm;
+
/* heap for rewrite during DDL, link to original rel */
Oid relrewrite BKI_DEFAULT(0);
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 0902dce5f1..6e1057c5b6 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10701,4 +10701,9 @@
proname => 'pg_partition_root', prorettype => 'regclass',
proargtypes => 'regclass', prosrc => 'pg_partition_root' },
+# IVM
+{ oid => '786', descr => 'ivm trigger',
+ proname => 'IVM_immediate_maintenance', provolatile => 'v', prorettype => 'trigger',
+ proargtypes => '', prosrc => 'IVM_immediate_maintenance' },
+
]
diff --git a/src/include/commands/matview.h b/src/include/commands/matview.h
index edf04bf415..8dd8193d9c 100644
--- a/src/include/commands/matview.h
+++ b/src/include/commands/matview.h
@@ -23,6 +23,8 @@
extern void SetMatViewPopulatedState(Relation relation, bool newstate);
+extern void SetMatViewIVMState(Relation relation, bool newstate);
+
extern ObjectAddress ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
ParamListInfo params, char *completionTag);
@@ -30,4 +32,6 @@ extern DestReceiver *CreateTransientRelDestReceiver(Oid oid);
extern bool MatViewIncrementalMaintenanceIsEnabled(void);
+extern Datum IVM_immediate_maintenance(PG_FUNCTION_ARGS);
+
#endif /* MATVIEW_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 94ded3c135..5c8a5ae3ca 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1002,6 +1002,7 @@ typedef struct RangeTblEntry
char relkind; /* relation kind (see pg_class.relkind) */
int rellockmode; /* lock level that query requires on the rel */
struct TableSampleClause *tablesample; /* sampling info, or NULL */
+ bool relisivm;
/*
* Fields valid for a subquery RTE (else NULL):
@@ -2059,6 +2060,7 @@ typedef struct CreateStmt
char *tablespacename; /* table space to use, or NULL */
char *accessMethod; /* table access method */
bool if_not_exists; /* just do nothing if it already exists? */
+ bool ivm; /* incremental view maintenance is used by materialized view */
} CreateStmt;
/* ----------
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 860a84de7c..6df58e2ff9 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -117,6 +117,7 @@ typedef struct IntoClause
char *tableSpaceName; /* table space to use, or NULL */
Node *viewQuery; /* materialized view's SELECT query */
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 00ace8425e..d682ee11cc 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -198,6 +198,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD)
PG_KEYWORD("inherit", INHERIT, UNRESERVED_KEYWORD)
diff --git a/src/include/utils/lsyscache.h b/src/include/utils/lsyscache.h
index c8df5bff9f..8fd919349e 100644
--- a/src/include/utils/lsyscache.h
+++ b/src/include/utils/lsyscache.h
@@ -129,6 +129,7 @@ extern Oid get_rel_namespace(Oid relid);
extern Oid get_rel_type_id(Oid relid);
extern char get_rel_relkind(Oid relid);
extern bool get_rel_relispartition(Oid relid);
+extern bool get_rel_relisivm(Oid relid);
extern Oid get_rel_tablespace(Oid relid);
extern char get_rel_persistence(Oid relid);
extern Oid get_transform_fromsql(Oid typid, Oid langid, List *trftypes);
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index b0fe19ebc5..9da0a4a9f9 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -565,6 +565,8 @@ typedef struct ViewOptions
*/
#define RelationIsPopulated(relation) ((relation)->rd_rel->relispopulated)
+#define RelationIsIVM(relation) ((relation)->rd_rel->relisivm)
+
/*
* RelationIsAccessibleInLogicalDecoding
* True if we need to log enough information to have access via
diff --git a/src/test/regress/expected/incremental_matview.out b/src/test/regress/expected/incremental_matview.out
new file mode 100644
index 0000000000..68060cc179
--- /dev/null
+++ b/src/test/regress/expected/incremental_matview.out
@@ -0,0 +1,306 @@
+-- create a table to use as a basis for views and materialized views in various combinations
+CREATE TABLE mv_base_a (i int, j int);
+INSERT INTO mv_base_a VALUES
+ (1,10),
+ (2,20),
+ (3,30),
+ (4,40),
+ (5,50);
+CREATE TABLE mv_base_b (i int, k int);
+INSERT INTO mv_base_b VALUES
+ (1,101),
+ (2,102),
+ (3,103),
+ (4,104);
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_1 AS SELECT i,j,k FROM mv_base_a a INNER JOIN mv_base_b b USING(i);
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+ i | j | k
+---+----+-----
+ 1 | 10 | 101
+ 2 | 20 | 102
+ 3 | 30 | 103
+ 4 | 40 | 104
+(4 rows)
+
+-- immediaite maintenance
+BEGIN;
+INSERT INTO mv_base_b VALUES(5,105);
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+ i | j | k
+---+----+-----
+ 1 | 10 | 101
+ 2 | 20 | 102
+ 3 | 30 | 103
+ 4 | 40 | 104
+ 5 | 50 | 105
+(5 rows)
+
+UPDATE mv_base_a SET j = 0 WHERE i = 1;
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+ i | j | k
+---+----+-----
+ 1 | 0 | 101
+ 2 | 20 | 102
+ 3 | 30 | 103
+ 4 | 40 | 104
+ 5 | 50 | 105
+(5 rows)
+
+DELETE FROM mv_base_b WHERE (i,k) = (5,105);
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+ i | j | k
+---+----+-----
+ 1 | 0 | 101
+ 2 | 20 | 102
+ 3 | 30 | 103
+ 4 | 40 | 104
+(4 rows)
+
+ROLLBACK;
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+ i | j | k
+---+----+-----
+ 1 | 10 | 101
+ 2 | 20 | 102
+ 3 | 30 | 103
+ 4 | 40 | 104
+(4 rows)
+
+-- result of materliazied view have DISTINCT clause or the duplicate result.
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_duplicate AS SELECT j FROM mv_base_a;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_distinct AS SELECT DISTINCT j FROM mv_base_a;
+INSERT INTO mv_base_a VALUES(6,20);
+SELECT * FROM mv_ivm_duplicate ORDER BY 1;
+ j
+----
+ 10
+ 20
+ 20
+ 30
+ 40
+ 50
+(6 rows)
+
+SELECT * FROM mv_ivm_distinct ORDER BY 1;
+ j
+----
+ 10
+ 20
+ 30
+ 40
+ 50
+(5 rows)
+
+DELETE FROM mv_base_a WHERE (i,j) = (2,20);
+SELECT * FROM mv_ivm_duplicate ORDER BY 1;
+ j
+----
+ 10
+ 20
+ 30
+ 40
+ 50
+(5 rows)
+
+SELECT * FROM mv_ivm_distinct ORDER BY 1;
+ j
+----
+ 10
+ 20
+ 30
+ 40
+ 50
+(5 rows)
+
+ROLLBACK;
+-- support SUM(), COUNT() and AVG() aggregation function
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_agg AS SELECT i, SUM(j), COUNT(i),AVG(j) FROM mv_base_a GROUP BY i;
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4;
+ i | sum | count | avg
+---+-----+-------+---------------------
+ 1 | 10 | 1 | 10.0000000000000000
+ 2 | 20 | 1 | 20.0000000000000000
+ 3 | 30 | 1 | 30.0000000000000000
+ 4 | 40 | 1 | 40.0000000000000000
+ 5 | 50 | 1 | 50.0000000000000000
+(5 rows)
+
+INSERT INTO mv_base_a VALUES(2,100);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4;
+ i | sum | count | avg
+---+-----+-------+---------------------
+ 1 | 10 | 1 | 10.0000000000000000
+ 2 | 120 | 2 | 60.0000000000000000
+ 3 | 30 | 1 | 30.0000000000000000
+ 4 | 40 | 1 | 40.0000000000000000
+ 5 | 50 | 1 | 50.0000000000000000
+(5 rows)
+
+UPDATE mv_base_a SET j = 200 WHERE (i,j) = (2,100);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4;
+ i | sum | count | avg
+---+-----+-------+----------------------
+ 1 | 10 | 1 | 10.0000000000000000
+ 2 | 220 | 2 | 110.0000000000000000
+ 3 | 30 | 1 | 30.0000000000000000
+ 4 | 40 | 1 | 40.0000000000000000
+ 5 | 50 | 1 | 50.0000000000000000
+(5 rows)
+
+DELETE FROM mv_base_a WHERE (i,j) = (2,200);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4;
+ i | sum | count | avg
+---+-----+-------+---------------------
+ 1 | 10 | 1 | 10.0000000000000000
+ 2 | 20 | 1 | 20.0000000000000000
+ 3 | 30 | 1 | 30.0000000000000000
+ 4 | 40 | 1 | 40.0000000000000000
+ 5 | 50 | 1 | 50.0000000000000000
+(5 rows)
+
+ROLLBACK;
+-- support COUNT(*) aggregation function
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_agg AS SELECT i, SUM(j),COUNT(*) FROM mv_base_a GROUP BY i;
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
+ i | sum | count
+---+-----+-------
+ 1 | 10 | 1
+ 2 | 20 | 1
+ 3 | 30 | 1
+ 4 | 40 | 1
+ 5 | 50 | 1
+(5 rows)
+
+INSERT INTO mv_base_a VALUES(2,100);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
+ i | sum | count
+---+-----+-------
+ 1 | 10 | 1
+ 2 | 120 | 2
+ 3 | 30 | 1
+ 4 | 40 | 1
+ 5 | 50 | 1
+(5 rows)
+
+ROLLBACK;
+-- support having only aggregation function without GROUP clause
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_group AS SELECT SUM(j)FROM mv_base_a;
+SELECT * FROM mv_ivm_group ORDER BY 1;
+ sum
+-----
+ 150
+(1 row)
+
+INSERT INTO mv_base_a VALUES(6,20);
+SELECT * FROM mv_ivm_group ORDER BY 1;
+ sum
+-----
+ 170
+(1 row)
+
+ROLLBACK;
+-- resolved issue: When use AVG() function and values is indivisible, result of AVG() is incorrect.
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_avg_bug AS SELECT i, SUM(j), COUNT(j), AVG(j) FROM mv_base_A GROUP BY i;
+SELECT * FROM mv_ivm_avg_bug ORDER BY 1,2,3;
+ i | sum | count | avg
+---+-----+-------+---------------------
+ 1 | 10 | 1 | 10.0000000000000000
+ 2 | 20 | 1 | 20.0000000000000000
+ 3 | 30 | 1 | 30.0000000000000000
+ 4 | 40 | 1 | 40.0000000000000000
+ 5 | 50 | 1 | 50.0000000000000000
+(5 rows)
+
+INSERT INTO mv_base_a VALUES
+ (1,0),
+ (1,0),
+ (2,30),
+ (2,30);
+SELECT * FROM mv_ivm_avg_bug ORDER BY 1,2,3;
+ i | sum | count | avg
+---+-----+-------+---------------------
+ 1 | 10 | 3 | 3.3333333333333333
+ 2 | 80 | 3 | 26.6666666666666667
+ 3 | 30 | 1 | 30.0000000000000000
+ 4 | 40 | 1 | 40.0000000000000000
+ 5 | 50 | 1 | 50.0000000000000000
+(5 rows)
+
+DELETE FROM mv_base_a WHERE (i,j) = (1,0);
+DELETE FROM mv_base_a WHERE (i,j) = (2,30);
+SELECT * FROM mv_ivm_avg_bug ORDER BY 1,2,3;
+ i | sum | count | avg
+---+-----+-------+---------------------
+ 1 | 10 | 1 | 10.0000000000000000
+ 2 | 20 | 1 | 20.0000000000000000
+ 3 | 30 | 1 | 30.0000000000000000
+ 4 | 40 | 1 | 40.0000000000000000
+ 5 | 50 | 1 | 50.0000000000000000
+(5 rows)
+
+ROLLBACK;
+-- support MIN(), MAX() aggregation functions
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_min_max AS SELECT i, MIN(j), MAX(j) FROM mv_base_a GROUP BY i;
+INSERT INTO mv_base_a VALUES
+ (1,11), (1,12),
+ (2,21), (2,22),
+ (3,31), (3,32),
+ (4,41), (4,42),
+ (5,51), (5,52);
+SELECT * FROM mv_ivm_min_max ORDER BY 1,2,3;
+ i | min | max
+---+-----+-----
+ 1 | 10 | 12
+ 2 | 20 | 22
+ 3 | 30 | 32
+ 4 | 40 | 42
+ 5 | 50 | 52
+(5 rows)
+
+DELETE FROM mv_base_a WHERE (i,j) IN ((1,10), (2,21), (3,32));
+SELECT * FROM mv_ivm_min_max ORDER BY 1,2,3;
+ i | min | max
+---+-----+-----
+ 1 | 11 | 12
+ 2 | 20 | 22
+ 3 | 30 | 31
+ 4 | 40 | 42
+ 5 | 50 | 52
+(5 rows)
+
+ROLLBACK;
+-- restriction of incremental view maintenance
+-- contain system column
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm01 AS SELECT i,j,xmin FROM mv_base_a;
+ERROR: system column is not supported with IVM
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm02 AS SELECT i,j FROM mv_base_a WHERE xmin = '610';
+ERROR: system column is not supported with IVM
+-- contain subquery
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm03 AS SELECT i,j FROM mv_base_a WHERE i IN (SELECT i FROM mv_base_b WHERE k < 103 );
+ERROR: subquery is not supported with IVM
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm04 AS SELECT a.i,a.j FROM mv_base_a a,( SELECT * FROM mv_base_b) b WHERE a.i = b.i;
+ERROR: subquery is not supported with IVM
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm05 AS SELECT i,j, (SELECT k FROM mv_base_b b WHERE a.i = b.i) FROM mv_base_a a;
+ERROR: subquery is not supported with IVM
+-- contain CTE
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm06 AS WITH b AS (SELECT i,k FROM mv_base_b WHERE k < 103) SELECT a.i,a.j FROM mv_base_a a,b WHERE a.i = b.i;
+ERROR: CTE is not supported with IVM
+-- contain view or materialized view
+CREATE VIEW b_view AS SELECT i,k FROM mv_base_b;
+CREATE MATERIALIZED VIEW b_mview AS SELECT i,k FROM mv_base_b;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm07 AS SELECT a.i,a.j FROM mv_base_a a,b_view b WHERE a.i = b.i;
+ERROR: VIEW or MATERIALIZED VIEW is not supported with IVM
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm08 AS SELECT a.i,a.j FROM mv_base_a a,b_mview b WHERE a.i = b.i;
+ERROR: VIEW or MATERIALIZED VIEW is not supported with IVM
+DROP TABLE mv_base_b CASCADE;
+NOTICE: drop cascades to 3 other objects
+DETAIL: drop cascades to materialized view mv_ivm_1
+drop cascades to view b_view
+drop cascades to materialized view b_mview
+DROP TABLE mv_base_a CASCADE;
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 8fb55f045e..39e79d8d10 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -78,7 +78,7 @@ test: brin gin gist spgist privileges init_privs security_label collate matview
# ----------
# Another group of parallel tests
# ----------
-test: create_table_like alter_generic alter_operator misc async dbsize misc_functions sysviews tsrf tidscan
+test: create_table_like alter_generic alter_operator misc async dbsize misc_functions sysviews tsrf tidscan incremental_matview
# rules cannot run concurrently with any test that creates
# a view or rule in the public schema
diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule
index a39ca1012a..552c80c851 100644
--- a/src/test/regress/serial_schedule
+++ b/src/test/regress/serial_schedule
@@ -112,6 +112,7 @@ test: init_privs
test: security_label
test: collate
test: matview
+test: incremental_matview
test: lock
test: replica_identity
test: rowsecurity
diff --git a/src/test/regress/sql/incremental_matview.sql b/src/test/regress/sql/incremental_matview.sql
new file mode 100644
index 0000000000..2dd968f709
--- /dev/null
+++ b/src/test/regress/sql/incremental_matview.sql
@@ -0,0 +1,117 @@
+-- create a table to use as a basis for views and materialized views in various combinations
+CREATE TABLE mv_base_a (i int, j int);
+INSERT INTO mv_base_a VALUES
+ (1,10),
+ (2,20),
+ (3,30),
+ (4,40),
+ (5,50);
+CREATE TABLE mv_base_b (i int, k int);
+INSERT INTO mv_base_b VALUES
+ (1,101),
+ (2,102),
+ (3,103),
+ (4,104);
+
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_1 AS SELECT i,j,k FROM mv_base_a a INNER JOIN mv_base_b b USING(i);
+
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+-- immediaite maintenance
+BEGIN;
+INSERT INTO mv_base_b VALUES(5,105);
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+UPDATE mv_base_a SET j = 0 WHERE i = 1;
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+DELETE FROM mv_base_b WHERE (i,k) = (5,105);
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+ROLLBACK;
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+
+-- result of materliazied view have DISTINCT clause or the duplicate result.
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_duplicate AS SELECT j FROM mv_base_a;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_distinct AS SELECT DISTINCT j FROM mv_base_a;
+INSERT INTO mv_base_a VALUES(6,20);
+SELECT * FROM mv_ivm_duplicate ORDER BY 1;
+SELECT * FROM mv_ivm_distinct ORDER BY 1;
+DELETE FROM mv_base_a WHERE (i,j) = (2,20);
+SELECT * FROM mv_ivm_duplicate ORDER BY 1;
+SELECT * FROM mv_ivm_distinct ORDER BY 1;
+ROLLBACK;
+
+-- support SUM(), COUNT() and AVG() aggregation function
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_agg AS SELECT i, SUM(j), COUNT(i),AVG(j) FROM mv_base_a GROUP BY i;
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4;
+INSERT INTO mv_base_a VALUES(2,100);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4;
+UPDATE mv_base_a SET j = 200 WHERE (i,j) = (2,100);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4;
+DELETE FROM mv_base_a WHERE (i,j) = (2,200);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4;
+ROLLBACK;
+
+-- support COUNT(*) aggregation function
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_agg AS SELECT i, SUM(j),COUNT(*) FROM mv_base_a GROUP BY i;
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
+INSERT INTO mv_base_a VALUES(2,100);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
+ROLLBACK;
+
+-- support having only aggregation function without GROUP clause
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_group AS SELECT SUM(j)FROM mv_base_a;
+SELECT * FROM mv_ivm_group ORDER BY 1;
+INSERT INTO mv_base_a VALUES(6,20);
+SELECT * FROM mv_ivm_group ORDER BY 1;
+ROLLBACK;
+
+-- resolved issue: When use AVG() function and values is indivisible, result of AVG() is incorrect.
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_avg_bug AS SELECT i, SUM(j), COUNT(j), AVG(j) FROM mv_base_A GROUP BY i;
+SELECT * FROM mv_ivm_avg_bug ORDER BY 1,2,3;
+INSERT INTO mv_base_a VALUES
+ (1,0),
+ (1,0),
+ (2,30),
+ (2,30);
+SELECT * FROM mv_ivm_avg_bug ORDER BY 1,2,3;
+DELETE FROM mv_base_a WHERE (i,j) = (1,0);
+DELETE FROM mv_base_a WHERE (i,j) = (2,30);
+SELECT * FROM mv_ivm_avg_bug ORDER BY 1,2,3;
+ROLLBACK;
+
+-- support MIN(), MAX() aggregation functions
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_min_max AS SELECT i, MIN(j), MAX(j) FROM mv_base_a GROUP BY i;
+INSERT INTO mv_base_a VALUES
+ (1,11), (1,12),
+ (2,21), (2,22),
+ (3,31), (3,32),
+ (4,41), (4,42),
+ (5,51), (5,52);
+SELECT * FROM mv_ivm_min_max ORDER BY 1,2,3;
+DELETE FROM mv_base_a WHERE (i,j) IN ((1,10), (2,21), (3,32));
+SELECT * FROM mv_ivm_min_max ORDER BY 1,2,3;
+ROLLBACK;
+
+-- restriction of incremental view maintenance
+
+-- contain system column
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm01 AS SELECT i,j,xmin FROM mv_base_a;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm02 AS SELECT i,j FROM mv_base_a WHERE xmin = '610';
+-- contain subquery
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm03 AS SELECT i,j FROM mv_base_a WHERE i IN (SELECT i FROM mv_base_b WHERE k < 103 );
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm04 AS SELECT a.i,a.j FROM mv_base_a a,( SELECT * FROM mv_base_b) b WHERE a.i = b.i;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm05 AS SELECT i,j, (SELECT k FROM mv_base_b b WHERE a.i = b.i) FROM mv_base_a a;
+-- contain CTE
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm06 AS WITH b AS (SELECT i,k FROM mv_base_b WHERE k < 103) SELECT a.i,a.j FROM mv_base_a a,b WHERE a.i = b.i;
+-- contain view or materialized view
+CREATE VIEW b_view AS SELECT i,k FROM mv_base_b;
+CREATE MATERIALIZED VIEW b_mview AS SELECT i,k FROM mv_base_b;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm07 AS SELECT a.i,a.j FROM mv_base_a a,b_view b WHERE a.i = b.i;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm08 AS SELECT a.i,a.j FROM mv_base_a a,b_mview b WHERE a.i = b.i;
+
+DROP TABLE mv_base_b CASCADE;
+DROP TABLE mv_base_a CASCADE;
It's not mentioned below but some bugs including seg fault when
--enable-casser is enabled was also fixed in this patch.
BTW, I found a bug with min/max support in this patch and I believe
Yugo is working on it. Details:
https://github.com/sraoss/pgsql-ivm/issues/20
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
From: Yugo Nagata <nagata@sraoss.co.jp>
Subject: Re: Implementing Incremental View Maintenance
Date: Wed, 31 Jul 2019 18:08:51 +0900
Message-ID: <20190731180851.73856441d8abb494bf5e68e7@sraoss.co.jp>
Show quoted text
Hi,
Attached is the latest patch for supporting min and max aggregate functions.
When new tuples are inserted into base tables, if new values are small
(for min) or large (for max), matview just have to be updated with these
new values. Otherwise, old values just remains.However, in the case of deletion, this is more complicated. If deleted
values exists in matview as current min or max, we have to recomputate
new min or max values from base tables for affected groups, and matview
should be updated with these recomputated values.Also, regression tests for min/max are also added.
In addition, incremental update algorithm of avg aggregate values is a bit
improved. If an avg result in materialized views is updated incrementally
y using the old avg value, numerical errors in avg values are accumulated
and the values get wrong eventually. To prevent this, both of sum and count
values are contained in views as hidden columns and use them to calculate
new avg value instead of using old avg values.Regards,
On Fri, 26 Jul 2019 11:35:53 +0900
Yugo Nagata <nagata@sraoss.co.jp> wrote:Hi,
I've updated the wiki page of Incremental View Maintenance.
https://wiki.postgresql.org/wiki/Incremental_View_Maintenance
On Thu, 11 Jul 2019 13:28:04 +0900
Yugo Nagata <nagata@sraoss.co.jp> wrote:Hi Thomas,
Thank you for your review and discussion on this patch!
2019年7月8日(月) 15:32 Thomas Munro <thomas.munro@gmail.com>:
On Fri, Jun 28, 2019 at 10:56 PM Yugo Nagata <nagata@sraoss.co.jp> wrote:
Attached is a WIP patch of IVM which supports some aggregate functions.
Hi Nagata-san and Hoshiai-san,
Thank you for working on this. I enjoyed your talk at PGCon. I've
added Kevin Grittner just in case he missed this thread; he has talked
often about implementing the counting algorithm, and he wrote the
"trigger transition tables" feature to support exactly this. While
integrating trigger transition tables with the new partition features,
we had to make a number of decisions about how that should work, and
we tried to come up with answers that would work for IMV, and I hope
we made the right choices!Transition tables is a great feature. I am now using this in my implementation
of IVM, but the first time I used this feature was when I implemented a PoC
for extending view updatability of PostgreSQL[1]. At that time, I didn't know
that this feature is made originally aiming to support IVM.[1] https://www.pgcon.org/2017/schedule/events/1074.en.html
I think transition tables is a good choice to implement a statement level
immediate view maintenance where materialized views are refreshed in a statement
level after trigger. However, when implementing a transaction level immediate
view maintenance where views are refreshed per transaction, or deferred view
maintenance, we can't update views in a after trigger, and we will need an
infrastructure to manage change logs of base tables. Transition tables can be
used to collect these logs, but using logical decoding of WAL is another candidate.
In any way, if these logs can be collected in a tuplestore, we might able to
convert this to "ephemeral named relation (ENR)" and use this to calculate diff
sets for views.I am quite interested to learn how IVM interacts with SERIALIZABLE.
Nagata-san has been studying this. Nagata-san, any comment?
In SERIALIZABLE or REPEATABLE READ level, table changes occurred in other
ransactions are not visible, so views can not be maintained correctly in AFTER
triggers. If a view is defined on two tables and each table is modified in
different concurrent transactions respectively, the result of view maintenance
done in trigger functions can be incorrect due to the race condition. This is the
reason why such transactions are aborted immediately in that case in my current
implementation.One idea to resolve this is performing view maintenance in two phases. Firstly,
views are updated using only table changes visible in this transaction. Then,
just after this transaction is committed, views have to be updated additionally
using changes happened in other transactions to keep consistency. This is a just
idea, but to implement this idea, I think, we will need keep to keep and
maintain change logs.A couple of superficial review comments:
+ const char *aggname = get_func_name(aggref->aggfnoid); ... + else if (!strcmp(aggname, "sum"))I guess you need a more robust way to detect the supported aggregates
than their name, or I guess some way for aggregates themselves to
specify that they support this and somehow supply the extra logic.
Perhaps I just waid what Greg Stark already said, except not as well.Yes. Using name is not robust because users can make same name aggregates like
sum(text) (although I am not sure this makes sense). We can use oids instead
of names, but it would be nice to extend pg_aggregate and add new attributes
for informing that this supports IVM and for providing functions for IVM logic.As for the question of how
to reserve a namespace for system columns that won't clash with user
columns, according to our manual the SQL standard doesn't allow $ in
identifier names, and according to my copy SQL92 "intermediate SQL"
doesn't allow identifiers that end in an underscore. I don't know
what the best answer is but we should probably decide on a something
based the standard.Ok, so we should use "__ivm_count__" since this ends in "_" at least.
Another idea is that users specify the name of this special column when
defining materialized views with IVM support. This way can avoid the conflict
because users will specify a name which does not appear in the target list.As for aggregates supports, it may be also possible to make it a restriction
that count(expr) must be in the target list explicitly when sum(expr) or
avg(expr) is included, instead of making hidden column like __ivm_count_sum__,
like Oracle does.As for how to make internal columns invisible to SELECT *, previously
there have been discussions about doing that using a new flag in
pg_attribute:/messages/by-id/CAEepm=3ZHh=p0nEEnVbs1Dig_UShPzHUcMNAqvDQUgYgcDo-pA@mail.gmail.com
I agree implementing this feature in PostgreSQL since there are at least a few
use cases, IVM and temporal database.+ "WITH t AS (" + " SELECT diff.__ivm_count__, (diff.__ivm_count__ = mv.__ivm_count__) AS for_dlt, mv.ctid" + ", %s" + " FROM %s AS mv, %s AS diff WHERE (%s) = (%s)" + "), updt AS (" + " UPDATE %s AS mv SET __ivm_count__ = mv.__ivm_count__ - t.__ivm_count__" + ", %s " + " FROM t WHERE mv.ctid = t.ctid AND NOT for_dlt" + ") DELETE FROM %s AS mv USING t WHERE mv.ctid = t.ctid AND for_dlt;",I fully understand that this is POC code, but I am curious about one
thing. These queries that are executed by apply_delta() would need to
be converted to C, or at least used reusable plans, right? Hmm,
creating and dropping temporary tables every time is a clue that the
ultimate form of this should be tuplestores and C code, I think,
right?I used SPI just because REFRESH CONCURRENTLY uses this, but, as you said,
it is inefficient to create/drop temp tables and perform parse/plan every times.
It seems to be enough to perform this once when creating materialized views or
at the first maintenance time.Best regards,
Yugo Nagata--
Yugo Nagata <nagata@sraoss.co.jp>--
Yugo Nagata <nagata@sraoss.co.jp>--
Yugo Nagata <nagata@sraoss.co.jp>
On 2019-Aug-06, Tatsuo Ishii wrote:
It's not mentioned below but some bugs including seg fault when
--enable-casser is enabled was also fixed in this patch.BTW, I found a bug with min/max support in this patch and I believe
Yugo is working on it. Details:
https://github.com/sraoss/pgsql-ivm/issues/20
So is he posting an updated patch soon?
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
As I understand it, the current patch performs immediate IVM using AFTER
STATEMENT trigger transition tables.
However, multiple tables can be modified *before* AFTER STATEMENT triggers
are fired.
CREATE TABLE example1 (a int);
CREATE TABLE example2 (a int);
CREATE INCREMENTAL MATERIALIZED VIEW mv AS
SELECT example1.a, example2.a
FROM example1 JOIN example2 ON a;
WITH
insert1 AS (INSERT INTO example1 VALUES (1)),
insert2 AS (INSERT INTO example2 VALUES (1))
SELECT NULL;
Changes to example1 are visible in an AFTER STATEMENT trigger on example2,
and vice versa. Would this not result in the (1, 1) tuple being
"double-counted"?
IVM needs to either:
(1) Evaluate deltas "serially' (e.g. EACH ROW triggers)
(2) Have simultaneous access to multiple deltas:
delta_mv = example1 x delta_example2 + example2 x delta_example1 -
delta_example1 x delta_example2
This latter method is the "logged" approach that has been discussed for
deferred evaluation.
tl;dr It seems that AFTER STATEMENT triggers required a deferred-like
implementation anyway.
Import Notes
Resolved by subject fallback
On 2019-Aug-06, Tatsuo Ishii wrote:
It's not mentioned below but some bugs including seg fault when
--enable-casser is enabled was also fixed in this patch.BTW, I found a bug with min/max support in this patch and I believe
Yugo is working on it. Details:
https://github.com/sraoss/pgsql-ivm/issues/20So is he posting an updated patch soon?
I think he is going to post an updated patch by the end of this month.
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
Hi Paul,
Thank you for your suggestion.
On Sun, 15 Sep 2019 11:52:22 -0600
Paul Draper <paulddraper@gmail.com> wrote:
As I understand it, the current patch performs immediate IVM using AFTER
STATEMENT trigger transition tables.However, multiple tables can be modified *before* AFTER STATEMENT triggers
are fired.CREATE TABLE example1 (a int);
CREATE TABLE example2 (a int);CREATE INCREMENTAL MATERIALIZED VIEW mv AS
SELECT example1.a, example2.a
FROM example1 JOIN example2 ON a;WITH
insert1 AS (INSERT INTO example1 VALUES (1)),
insert2 AS (INSERT INTO example2 VALUES (1))
SELECT NULL;Changes to example1 are visible in an AFTER STATEMENT trigger on example2,
and vice versa. Would this not result in the (1, 1) tuple being
"double-counted"?IVM needs to either:
(1) Evaluate deltas "serially' (e.g. EACH ROW triggers)
(2) Have simultaneous access to multiple deltas:
delta_mv = example1 x delta_example2 + example2 x delta_example1 -
delta_example1 x delta_example2This latter method is the "logged" approach that has been discussed for
deferred evaluation.tl;dr It seems that AFTER STATEMENT triggers required a deferred-like
implementation anyway.
You are right, the latest patch doesn't support the situation where
multiple tables are modified in a query. I noticed this when working
on self-join, which also virtually need to handle multiple table
modification.
I am now working on this issue and the next patch will enable to handle
this situation. I plan to submit the patch during this month. Roughly
speaking, in the new implementation, AFTER STATEMENT triggers are used to
collect information of modified table and its changes (= transition tables),
and then the only last trigger updates the view. This will avoid the
double-counting. I think this implementation also would be a base of
deferred approach implementation in future where "logs" are used instead
of transition tables.
Regards,
Yugo Nagata
--
Yugo Nagata <nagata@sraoss.co.jp>
Have you had any thoughts for more than two joined tables?
Either there needs to be an quadratic number of joins, or intermediate join
results need to be stored and reused.
On Tue, Sep 17, 2019 at 8:50 AM Yugo Nagata <nagata@sraoss.co.jp> wrote:
Show quoted text
Hi Paul,
Thank you for your suggestion.
On Sun, 15 Sep 2019 11:52:22 -0600
Paul Draper <paulddraper@gmail.com> wrote:As I understand it, the current patch performs immediate IVM using AFTER
STATEMENT trigger transition tables.However, multiple tables can be modified *before* AFTER STATEMENT
triggers
are fired.
CREATE TABLE example1 (a int);
CREATE TABLE example2 (a int);CREATE INCREMENTAL MATERIALIZED VIEW mv AS
SELECT example1.a, example2.a
FROM example1 JOIN example2 ON a;WITH
insert1 AS (INSERT INTO example1 VALUES (1)),
insert2 AS (INSERT INTO example2 VALUES (1))
SELECT NULL;Changes to example1 are visible in an AFTER STATEMENT trigger on
example2,
and vice versa. Would this not result in the (1, 1) tuple being
"double-counted"?IVM needs to either:
(1) Evaluate deltas "serially' (e.g. EACH ROW triggers)
(2) Have simultaneous access to multiple deltas:
delta_mv = example1 x delta_example2 + example2 x delta_example1 -
delta_example1 x delta_example2This latter method is the "logged" approach that has been discussed for
deferred evaluation.tl;dr It seems that AFTER STATEMENT triggers required a deferred-like
implementation anyway.You are right, the latest patch doesn't support the situation where
multiple tables are modified in a query. I noticed this when working
on self-join, which also virtually need to handle multiple table
modification.I am now working on this issue and the next patch will enable to handle
this situation. I plan to submit the patch during this month. Roughly
speaking, in the new implementation, AFTER STATEMENT triggers are used to
collect information of modified table and its changes (= transition
tables),
and then the only last trigger updates the view. This will avoid the
double-counting. I think this implementation also would be a base of
deferred approach implementation in future where "logs" are used instead
of transition tables.Regards,
Yugo Nagata--
Yugo Nagata <nagata@sraoss.co.jp>
Have you had any thoughts for more than two joined tables?
I am not sure what you are asking here but if you are asking if IVM
supports two or more tables involved in a join, we already support it:
DROP MATERIALIZED VIEW mv1;
DROP MATERIALIZED VIEW
DROP TABLE t1;
DROP TABLE
DROP TABLE t2;
DROP TABLE
DROP TABLE t3;
DROP TABLE
CREATE TABLE t1(i int, j int);
CREATE TABLE
CREATE TABLE t2(k int, l int);
CREATE TABLE
CREATE TABLE t3(m int, n int);
CREATE TABLE
INSERT INTO t1 VALUES(1,10),(2,11);
INSERT 0 2
INSERT INTO t2 VALUES(1,20),(2,21);
INSERT 0 2
INSERT INTO t3 VALUES(1,30),(2,31);
INSERT 0 2
CREATE INCREMENTAL MATERIALIZED VIEW mv1 AS SELECT * FROM t1 INNER JOIN t2 ON t1.i = t2.k INNER JOIN t3 ON t1.i = t3.m;
SELECT 2
SELECT * FROM mv1;
i | j | k | l | m | n
---+----+---+----+---+----
1 | 10 | 1 | 20 | 1 | 30
2 | 11 | 2 | 21 | 2 | 31
(2 rows)
UPDATE t1 SET j = 15 WHERE i = 1;
UPDATE 1
SELECT * FROM mv1;
i | j | k | l | m | n
---+----+---+----+---+----
2 | 11 | 2 | 21 | 2 | 31
1 | 15 | 1 | 20 | 1 | 30
(2 rows)
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
Show quoted text
Either there needs to be an quadratic number of joins, or intermediate join
results need to be stored and reused.On Tue, Sep 17, 2019 at 8:50 AM Yugo Nagata <nagata@sraoss.co.jp> wrote:
Hi Paul,
Thank you for your suggestion.
On Sun, 15 Sep 2019 11:52:22 -0600
Paul Draper <paulddraper@gmail.com> wrote:As I understand it, the current patch performs immediate IVM using AFTER
STATEMENT trigger transition tables.However, multiple tables can be modified *before* AFTER STATEMENT
triggers
are fired.
CREATE TABLE example1 (a int);
CREATE TABLE example2 (a int);CREATE INCREMENTAL MATERIALIZED VIEW mv AS
SELECT example1.a, example2.a
FROM example1 JOIN example2 ON a;WITH
insert1 AS (INSERT INTO example1 VALUES (1)),
insert2 AS (INSERT INTO example2 VALUES (1))
SELECT NULL;Changes to example1 are visible in an AFTER STATEMENT trigger on
example2,
and vice versa. Would this not result in the (1, 1) tuple being
"double-counted"?IVM needs to either:
(1) Evaluate deltas "serially' (e.g. EACH ROW triggers)
(2) Have simultaneous access to multiple deltas:
delta_mv = example1 x delta_example2 + example2 x delta_example1 -
delta_example1 x delta_example2This latter method is the "logged" approach that has been discussed for
deferred evaluation.tl;dr It seems that AFTER STATEMENT triggers required a deferred-like
implementation anyway.You are right, the latest patch doesn't support the situation where
multiple tables are modified in a query. I noticed this when working
on self-join, which also virtually need to handle multiple table
modification.I am now working on this issue and the next patch will enable to handle
this situation. I plan to submit the patch during this month. Roughly
speaking, in the new implementation, AFTER STATEMENT triggers are used to
collect information of modified table and its changes (= transition
tables),
and then the only last trigger updates the view. This will avoid the
double-counting. I think this implementation also would be a base of
deferred approach implementation in future where "logs" are used instead
of transition tables.Regards,
Yugo Nagata--
Yugo Nagata <nagata@sraoss.co.jp>
On Tue, 17 Sep 2019 12:03:20 -0600
Paul Draper <paulddraper@gmail.com> wrote:
Have you had any thoughts for more than two joined tables?
Either there needs to be an quadratic number of joins, or intermediate join
results need to be stored and reused.
I don't think that we need to store intermediate join results.
Suppose that we have a view V joining table R,S, and new tuples are inserted
to each table, dR,dS, and dT respectively.
V = R*S*T
R_new = R + dR
S_new = S + dS
T_new = T + dT
In this situation, we can calculate the new view state as bellow.
V_new
= R_new * S_new * T_new
= (R + dR) * (S + dS) * (T + dT)
= R*S*T + dR*(S + dS)*(T + dT) + R*dS*(T + dT) + R*S*dT
= V + dR*(S + dS)*(T + dT) + R*dS*(T + dT) + R*S*dT
Although the number of terms is 2^3(=8), if we can use both of pre-update
state (eg. R) and post-update state (eg. R+dR), we only need only three joins.
Actually, post-update state is available in AFTER trigger, and pre-update state
can be calculated by using delta tables (transition tables) and cmin/xmin system
columns (or snapshot). This is the approach my implementation adopts.
On Tue, Sep 17, 2019 at 8:50 AM Yugo Nagata <nagata@sraoss.co.jp> wrote:
Hi Paul,
Thank you for your suggestion.
On Sun, 15 Sep 2019 11:52:22 -0600
Paul Draper <paulddraper@gmail.com> wrote:As I understand it, the current patch performs immediate IVM using AFTER
STATEMENT trigger transition tables.However, multiple tables can be modified *before* AFTER STATEMENT
triggers
are fired.
CREATE TABLE example1 (a int);
CREATE TABLE example2 (a int);CREATE INCREMENTAL MATERIALIZED VIEW mv AS
SELECT example1.a, example2.a
FROM example1 JOIN example2 ON a;WITH
insert1 AS (INSERT INTO example1 VALUES (1)),
insert2 AS (INSERT INTO example2 VALUES (1))
SELECT NULL;Changes to example1 are visible in an AFTER STATEMENT trigger on
example2,
and vice versa. Would this not result in the (1, 1) tuple being
"double-counted"?IVM needs to either:
(1) Evaluate deltas "serially' (e.g. EACH ROW triggers)
(2) Have simultaneous access to multiple deltas:
delta_mv = example1 x delta_example2 + example2 x delta_example1 -
delta_example1 x delta_example2This latter method is the "logged" approach that has been discussed for
deferred evaluation.tl;dr It seems that AFTER STATEMENT triggers required a deferred-like
implementation anyway.You are right, the latest patch doesn't support the situation where
multiple tables are modified in a query. I noticed this when working
on self-join, which also virtually need to handle multiple table
modification.I am now working on this issue and the next patch will enable to handle
this situation. I plan to submit the patch during this month. Roughly
speaking, in the new implementation, AFTER STATEMENT triggers are used to
collect information of modified table and its changes (= transition
tables),
and then the only last trigger updates the view. This will avoid the
double-counting. I think this implementation also would be a base of
deferred approach implementation in future where "logs" are used instead
of transition tables.Regards,
Yugo Nagata--
Yugo Nagata <nagata@sraoss.co.jp>
--
Yugo Nagata <nagata@sraoss.co.jp>
Hi,
Attached is the latest patch for supporting self-join views. This also
including the following fix mentioned by Tatsuo Ishii.
On 2019-Aug-06, Tatsuo Ishii wrote:
It's not mentioned below but some bugs including seg fault when
--enable-casser is enabled was also fixed in this patch.BTW, I found a bug with min/max support in this patch and I believe
Yugo is working on it. Details:
https://github.com/sraoss/pgsql-ivm/issues/20
This patch allows to support self-join views, simultaneous updates of more
than one base tables, and also multiple updates of the same base table.
I first tried to support just self-join, but I found that this is essentially
same as to support simultaneous table updates, so I decided to support them in
the same commit. I think this will be a base for implementing
Deferred-maintenance in future.
In the new implementation, AFTER triggers are used to collecting tuplestores
containing transition table contents. When multiple tables are changed,
multiple AFTER triggers are invoked, then the final AFTER trigger performs
actual update of the matview. In addition AFTER trigger, also BEFORE trigger
is used to handle global information for view maintenance.
For example, suppose that we have a view V joining table R,S, and new tuples are
inserted to each table, dR,dS, and dT respectively.
V = R*S*T
R_new = R + dR
S_new = S + dS
T_new = T + dT
In this situation, we can calculate the new view state as bellow.
V_new
= R_new * S_new * T_new
= (R + dR) * (S + dS) * (T + dT)
= R*S*T + dR*(S + dS)*(T + dT) + R*dS*(T + dT) + R*S*dT
= V + dR*(S + dS)*(T + dT) + R*dS*(T + dT) + R*S*dT
= V + (dR *S_new*T_new) + (R*dS*T_new) + (R*S*dT)
To calculate view deltas, we need both pre-state (R,S, and T) and post-state
(R_new, S_new, and T_new) of base tables.
Post-update states are available in AFTER trigger, and we calculate pre-update
states by filtering inserted tuples using cmin/xmin system columns, and appendding
deleted tuples which are contained in a old transition table.
In the original core implementation, tuplestores of transition tables were
freed for each query depth. However, we want to prolong their life plan because
we have to preserve these for a whole query assuming some base tables are changed
in other trigger functions, so I added a hack to trigger.c.
Regression tests are also added for self join view, multiple change on the same
table, simultaneous two table changes, and foreign reference constrains.
Here are behavior examples:
1. Table definition
- t: for self-join
- r,s: for 2-ways join
CREATE TABLE r (i int, v int);
CREATE TABLE
CREATE TABLE s (i int, v int);
CREATE TABLE
CREATE TABLE t (i int, v int);
CREATE TABLE
2. Initial data
INSERT INTO r VALUES (1, 10), (2, 20), (3, 30);
INSERT 0 3
INSERT INTO s VALUES (1, 100), (2, 200), (3, 300);
INSERT 0 3
INSERT INTO t VALUES (1, 10), (2, 20), (3, 30);
INSERT 0 3
3. View definition
3.1. self-join(mv_self, v_slef)
CREATE INCREMENTAL MATERIALIZED VIEW mv_self(v1, v2) AS
SELECT t1.v, t2.v FROM t t1 JOIN t t2 ON t1.i = t2.i;
SELECT 3
CREATE VIEW v_self(v1, v2) AS
SELECT t1.v, t2.v FROM t t1 JOIN t t2 ON t1.i = t2.i;
CREATE VIEW
3.2. 2-ways join (mv, v)
CREATE INCREMENTAL MATERIALIZED VIEW mv(v1, v2) AS
SELECT r.v, s.v FROM r JOIN s USING(i);
SELECT 3
CREATE VIEW v(v1, v2) AS
SELECT r.v, s.v FROM r JOIN s USING(i);
CREATE VIEW
3.3 Initial contents
SELECT * FROM mv_self ORDER BY v1;
v1 | v2
----+----
10 | 10
20 | 20
30 | 30
(3 rows)
SELECT * FROM mv ORDER BY v1;
v1 | v2
----+-----
10 | 100
20 | 200
30 | 300
(3 rows)
4. Update a base table for the self-join view
INSERT INTO t VALUES (4,40);
INSERT 0 1
DELETE FROM t WHERE i = 1;
DELETE 1
UPDATE t SET v = v*10 WHERE i=2;
UPDATE 1
4.1. Results
- Comparison with the normal view
SELECT * FROM mv_self ORDER BY v1;
v1 | v2
-----+-----
30 | 30
40 | 40
200 | 200
(3 rows)
SELECT * FROM v_self ORDER BY v1;
v1 | v2
-----+-----
30 | 30
40 | 40
200 | 200
(3 rows)
5. pdate a base table for the 2-way join view
WITH
ins_r AS (INSERT INTO r VALUES (1,11) RETURNING 1),
ins_r2 AS (INSERT INTO r VALUES (3,33) RETURNING 1),
ins_s AS (INSERT INTO s VALUES (2,222) RETURNING 1),
upd_r AS (UPDATE r SET v = v + 1000 WHERE i = 2 RETURNING 1),
dlt_s AS (DELETE FROM s WHERE i = 3 RETURNING 1)
SELECT NULL;
?column?
----------
(1 row)
5.1. Results
- Comparison with the normal view
SELECT * FROM mv ORDER BY v1;
v1 | v2
------+-----
10 | 100
11 | 100
1020 | 200
1020 | 222
(4 rows)
SELECT * FROM v ORDER BY v1;
v1 | v2
------+-----
10 | 100
11 | 100
1020 | 200
1020 | 222
(4 rows)
========
Best Regards,
Yugo Nagata
--
Yugo Nagata <nagata@sraoss.co.jp>
Attachments:
WIP_immediate_IVM_v6.patchtext/x-diff; name=WIP_immediate_IVM_v6.patchDownload
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 5e71a2e865..a288bddcda 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -1952,6 +1952,13 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<entry>True if table or index is a partition</entry>
</row>
+ <row>
+ <entry><structfield>relisivm</structfield></entry>
+ <entry><type>bool</type></entry>
+ <entry></entry>
+ <entry>True if materialized view enables incremental view maintenance</entry>
+ </row>
+
<row>
<entry><structfield>relrewrite</structfield></entry>
<entry><type>oid</type></entry>
diff --git a/doc/src/sgml/ref/create_materialized_view.sgml b/doc/src/sgml/ref/create_materialized_view.sgml
index ec8847ed40..a23366a342 100644
--- a/doc/src/sgml/ref/create_materialized_view.sgml
+++ b/doc/src/sgml/ref/create_materialized_view.sgml
@@ -21,7 +21,7 @@ PostgreSQL documentation
<refsynopsisdiv>
<synopsis>
-CREATE MATERIALIZED VIEW [ IF NOT EXISTS ] <replaceable>table_name</replaceable>
+CREATE [ INCREMENTAL ] MATERIALIZED VIEW [ IF NOT EXISTS ] <replaceable>table_name</replaceable>
[ (<replaceable>column_name</replaceable> [, ...] ) ]
[ USING <replaceable class="parameter">method</replaceable> ]
[ WITH ( <replaceable class="parameter">storage_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] ) ]
@@ -54,6 +54,16 @@ CREATE MATERIALIZED VIEW [ IF NOT EXISTS ] <replaceable>table_name</replaceable>
<title>Parameters</title>
<variablelist>
+ <varlistentry>
+ <term><literal>INCREMENTAL</literal></term>
+ <listitem>
+ <para>
+ If specified, a materialized view enables incremental view maintenance.
+ You can replace only the contents of a materialized view, which based rows are changed.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>IF NOT EXISTS</literal></term>
<listitem>
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 9162286c98..c5551f59e2 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -34,6 +34,7 @@
#include "catalog/pg_enum.h"
#include "catalog/storage.h"
#include "commands/async.h"
+#include "commands/matview.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "executor/spi.h"
@@ -2661,6 +2662,7 @@ AbortTransaction(void)
AtAbort_Notify();
AtEOXact_RelationMap(false, is_parallel_worker);
AtAbort_Twophase();
+ AtAbort_IVM();
/*
* Advertise the fact that we aborted in pg_xact (assuming that we got as
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index b7bcdd9d0f..cb7e8be84a 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -890,6 +890,7 @@ InsertPgClassTuple(Relation pg_class_desc,
values[Anum_pg_class_relrewrite - 1] = ObjectIdGetDatum(rd_rel->relrewrite);
values[Anum_pg_class_relfrozenxid - 1] = TransactionIdGetDatum(rd_rel->relfrozenxid);
values[Anum_pg_class_relminmxid - 1] = MultiXactIdGetDatum(rd_rel->relminmxid);
+ values[Anum_pg_class_relisivm - 1] = BoolGetDatum(rd_rel->relisivm);
if (relacl != (Datum) 0)
values[Anum_pg_class_relacl - 1] = relacl;
else
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 098732cc4a..5aadbf96a8 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -955,6 +955,7 @@ index_create(Relation heapRelation,
indexRelation->rd_rel->relowner = heapRelation->rd_rel->relowner;
indexRelation->rd_rel->relam = accessMethodObjectId;
indexRelation->rd_rel->relispartition = OidIsValid(parentIndexRelid);
+ indexRelation->rd_rel->relisivm = false;
/*
* store index's pg_class entry
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index b7d220699f..6f2639130c 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -51,6 +51,19 @@
#include "utils/rls.h"
#include "utils/snapmgr.h"
+#include "catalog/pg_trigger.h"
+#include "catalog/pg_type.h"
+#include "commands/trigger.h"
+#include "parser/parser.h"
+#include "parser/parsetree.h"
+#include "parser/parse_func.h"
+#include "parser/parse_type.h"
+#include "nodes/print.h"
+#include "nodes/primnodes.h"
+#include "optimizer/optimizer.h"
+#include "commands/defrem.h"
+#include "utils/syscache.h"
+
typedef struct
{
@@ -74,6 +87,9 @@ static bool intorel_receive(TupleTableSlot *slot, DestReceiver *self);
static void intorel_shutdown(DestReceiver *self);
static void intorel_destroy(DestReceiver *self);
+static void CreateIvmTrigger(Oid relOid, Oid viewOid, char *matviewname, int16 type, int16 timing);
+static void CreateIvmTriggersOnBaseTables(Query *qry, Node *jtnode, Oid matviewOid, char* matviewname, Bitmapset **relid_map);
+static void check_ivm_restriction_walker(Node *node);
/*
* create_ctas_internal
@@ -109,6 +125,8 @@ create_ctas_internal(List *attrList, IntoClause *into)
create->oncommit = into->onCommit;
create->tablespacename = into->tableSpaceName;
create->if_not_exists = false;
+ /* Using Materialized view only */
+ create->ivm = into->ivm;
create->accessMethod = into->accessMethod;
/*
@@ -239,6 +257,7 @@ ExecCreateTableAs(CreateTableAsStmt *stmt, const char *queryString,
List *rewritten;
PlannedStmt *plan;
QueryDesc *queryDesc;
+ Query *copied_query;
if (stmt->if_not_exists)
{
@@ -256,6 +275,9 @@ ExecCreateTableAs(CreateTableAsStmt *stmt, const char *queryString,
}
}
+ if (is_matview && into->ivm)
+ check_ivm_restriction_walker((Node *) query);
+
/*
* Create the tuple receiver object and insert info it will need
*/
@@ -319,7 +341,148 @@ ExecCreateTableAs(CreateTableAsStmt *stmt, const char *queryString,
* and is executed repeatedly. (See also the same hack in EXPLAIN and
* PREPARE.)
*/
- rewritten = QueryRewrite(copyObject(query));
+
+ copied_query = copyObject(query);
+ if (is_matview && into->ivm)
+ {
+ TargetEntry *tle;
+ Node *node;
+ ParseState *pstate = make_parsestate(NULL);
+ FuncCall *fn;
+
+ pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET;
+
+ /* group keys must be in targetlist */
+ if (copied_query->groupClause)
+ {
+ ListCell *lc;
+ foreach(lc, copied_query->groupClause)
+ {
+ SortGroupClause *scl = (SortGroupClause *) lfirst(lc);
+ TargetEntry *tle = get_sortgroupclause_tle(scl, copied_query->targetList);
+
+ if (tle->resjunk)
+ elog(ERROR, "GROUP BY expression must appear in select list for incremental materialized views");
+ }
+ }
+ else if (!copied_query->hasAggs)
+ copied_query->groupClause = transformDistinctClause(NULL, &copied_query->targetList, copied_query->sortClause, false);
+
+ if (copied_query->hasAggs)
+ {
+ ListCell *lc;
+ List *agg_counts = NIL;
+ AttrNumber next_resno = list_length(copied_query->targetList) + 1;
+ Const *dmy_arg = makeConst(INT4OID,
+ -1,
+ InvalidOid,
+ sizeof(int32),
+ Int32GetDatum(1),
+ false,
+ true); /* pass by value */
+
+ foreach(lc, copied_query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ TargetEntry *tle_count;
+ char *resname = (into->colNames == NIL ? tle->resname : strVal(list_nth(into->colNames, tle->resno-1)));
+
+
+ if (IsA(tle->expr, Aggref))
+ {
+ Aggref *aggref = (Aggref *) tle->expr;
+ const char *aggname = get_func_name(aggref->aggfnoid);
+
+ /* XXX: need some generalization
+ *
+ * Specifically, Using func names is not robust. We can use oids instead
+ * of names, but it would be nice to add some information to pg_aggregate.
+ */
+ if (strcmp(aggname, "sum") !=0
+ && strcmp(aggname, "count") != 0
+ && strcmp(aggname, "avg") != 0
+ && strcmp(aggname, "min") != 0
+ && strcmp(aggname, "max") != 0
+ )
+ elog(ERROR, "aggregate function %s is not supported", aggname);
+
+ /*
+ * For aggregate functions except to count, add count func with the same arg parameters.
+ * Also, add sum func for agv.
+ *
+ * XXX: If there are same expressions explicitly in the target list, we can use this instead
+ * of adding new duplicated one.
+ */
+ if (strcmp(aggname, "count") != 0)
+ {
+ fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+
+ /* Make a Func with a dummy arg, and then override this by the original agg's args. */
+ node = ParseFuncOrColumn(pstate, fn->funcname, list_make1(dmy_arg), NULL, fn, false, -1);
+ ((Aggref *)node)->args = aggref->args;
+
+ tle_count = makeTargetEntry((Expr *) node,
+ next_resno,
+ pstrdup(makeObjectName("__ivm_count",resname, "_")),
+ false);
+ agg_counts = lappend(agg_counts, tle_count);
+ next_resno++;
+ }
+ if (strcmp(aggname, "avg") == 0)
+ {
+ List *dmy_args = NIL;
+ ListCell *lc;
+ foreach(lc, aggref->aggargtypes)
+ {
+ Oid typeid = lfirst_oid(lc);
+ Type type = typeidType(typeid);
+
+ Const *con = makeConst(typeid,
+ -1,
+ typeTypeCollation(type),
+ typeLen(type),
+ (Datum) 0,
+ true,
+ typeByVal(type));
+ dmy_args = lappend(dmy_args, con);
+ ReleaseSysCache(type);
+
+ }
+ fn = makeFuncCall(list_make1(makeString("sum")), NIL, -1);
+
+ /* Make a Func with a dummy arg, and then override this by the original agg's args. */
+ node = ParseFuncOrColumn(pstate, fn->funcname, dmy_args, NULL, fn, false, -1);
+ ((Aggref *)node)->args = aggref->args;
+
+ tle_count = makeTargetEntry((Expr *) node,
+ next_resno,
+ pstrdup(makeObjectName("__ivm_sum",resname, "_")),
+ false);
+ agg_counts = lappend(agg_counts, tle_count);
+ next_resno++;
+ }
+
+ }
+ }
+ copied_query->targetList = list_concat(copied_query->targetList, agg_counts);
+
+ }
+
+ /* Add count(*) for counting algorithm */
+ fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+ fn->agg_star = true;
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(copied_query->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ copied_query->targetList = lappend(copied_query->targetList, tle);
+ copied_query->hasAggs = true;
+ }
+
+ rewritten = QueryRewrite(copied_query);
/* SELECT should never rewrite to more or less than one SELECT query */
if (list_length(rewritten) != 1)
@@ -378,11 +541,77 @@ ExecCreateTableAs(CreateTableAsStmt *stmt, const char *queryString,
/* Restore userid and security context */
SetUserIdAndSecContext(save_userid, save_sec_context);
+
+
+ if (into->ivm)
+ {
+
+ Oid matviewOid = address.objectId;
+ Relation matviewRel = table_open(matviewOid, NoLock);
+ char *matviewname = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(matviewRel)),
+ RelationGetRelationName(matviewRel));
+ Bitmapset *relid_map = NULL;
+
+ copied_query = copyObject(query);
+ AcquireRewriteLocks(copied_query, true, false);
+
+ CreateIvmTriggersOnBaseTables(copied_query, (Node *)copied_query->jointree, matviewOid, matviewname, &relid_map);
+
+ table_close(matviewRel, NoLock);
+
+ bms_free(relid_map);
+ }
}
return address;
}
+static void CreateIvmTriggersOnBaseTables(Query *qry, Node *jtnode, Oid matviewOid, char* matviewname, Bitmapset **relid_map)
+{
+
+ if (jtnode == NULL)
+ return;
+ if (IsA(jtnode, RangeTblRef))
+ {
+ int rti = ((RangeTblRef *) jtnode)->rtindex;
+ RangeTblEntry *rte = rt_fetch(rti, qry->rtable);
+
+ if (rte->rtekind == RTE_RELATION)
+ {
+ if (!bms_is_member(rte->relid, *relid_map))
+ {
+ CreateIvmTrigger(rte->relid, matviewOid, matviewname, TRIGGER_TYPE_INSERT, TRIGGER_TYPE_BEFORE);
+ CreateIvmTrigger(rte->relid, matviewOid, matviewname, TRIGGER_TYPE_DELETE, TRIGGER_TYPE_BEFORE);
+ CreateIvmTrigger(rte->relid, matviewOid, matviewname, TRIGGER_TYPE_UPDATE, TRIGGER_TYPE_BEFORE);
+ CreateIvmTrigger(rte->relid, matviewOid, matviewname, TRIGGER_TYPE_INSERT, TRIGGER_TYPE_AFTER);
+ CreateIvmTrigger(rte->relid, matviewOid, matviewname, TRIGGER_TYPE_DELETE, TRIGGER_TYPE_AFTER);
+ CreateIvmTrigger(rte->relid, matviewOid, matviewname, TRIGGER_TYPE_UPDATE, TRIGGER_TYPE_AFTER);
+
+ *relid_map = bms_add_member(*relid_map, rte->relid);
+ }
+ }
+ else
+ elog(ERROR, "unsupported RTE kind: %d", (int) rte->rtekind);
+ }
+ else if (IsA(jtnode, FromExpr))
+ {
+ FromExpr *f = (FromExpr *) jtnode;
+ ListCell *l;
+
+ foreach(l, f->fromlist)
+ CreateIvmTriggersOnBaseTables(qry, lfirst(l), matviewOid, matviewname, relid_map);
+ }
+ else if (IsA(jtnode, JoinExpr))
+ {
+ JoinExpr *j = (JoinExpr *) jtnode;
+
+ CreateIvmTriggersOnBaseTables(qry, j->larg, matviewOid, matviewname, relid_map);
+ CreateIvmTriggersOnBaseTables(qry, j->rarg, matviewOid, matviewname, relid_map);
+ }
+ else
+ elog(ERROR, "unrecognized node type: %d", (int) nodeTag(jtnode));
+}
+
/*
* GetIntoRelEFlags --- compute executor flags needed for CREATE TABLE AS
*
@@ -547,6 +776,11 @@ intorel_startup(DestReceiver *self, int operation, TupleDesc typeinfo)
if (is_matview && !into->skipData)
SetMatViewPopulatedState(intoRelationDesc, true);
+ /*
+ * Mark relisivm field, if it's a matview and into->ivm is true.
+ */
+ if (is_matview && into->ivm)
+ SetMatViewIVMState(intoRelationDesc, true);
/*
* Fill private fields of myState for use by later routines
*/
@@ -619,3 +853,237 @@ intorel_destroy(DestReceiver *self)
{
pfree(self);
}
+
+
+static void
+CreateIvmTrigger(Oid relOid, Oid viewOid, char *matviewname, int16 type, int16 timing)
+{
+ CreateTrigStmt *ivm_trigger;
+ List *transitionRels = NIL;
+ ObjectAddress address, refaddr;
+
+ Assert(timing == TRIGGER_TYPE_BEFORE || timing == TRIGGER_TYPE_AFTER);
+
+ refaddr.classId = RelationRelationId;
+ refaddr.objectId = viewOid;
+ refaddr.objectSubId = 0;
+
+ ivm_trigger = makeNode(CreateTrigStmt);
+ ivm_trigger->relation = NULL;
+ ivm_trigger->row = false;
+
+ ivm_trigger->timing = timing;
+ ivm_trigger->events = type;
+
+ switch (type)
+ {
+ case TRIGGER_TYPE_INSERT:
+ ivm_trigger->trigname = (timing == TRIGGER_TYPE_BEFORE ? "IVM_trigger_ins_before" : "IVM_trigger_ins_after");
+ break;
+ case TRIGGER_TYPE_DELETE:
+ ivm_trigger->trigname = (timing == TRIGGER_TYPE_BEFORE ? "IVM_trigger_del_before" : "IVM_trigger_del_after");
+ break;
+ case TRIGGER_TYPE_UPDATE:
+ ivm_trigger->trigname = (timing == TRIGGER_TYPE_BEFORE ? "IVM_trigger_upd_before" : "IVM_trigger_upd_after");
+ break;
+ default:
+ elog(ERROR, "unsupported trigger type");
+ }
+
+ if (timing == TRIGGER_TYPE_AFTER)
+ {
+ if (type == TRIGGER_TYPE_INSERT || type == TRIGGER_TYPE_UPDATE)
+ {
+ TriggerTransition *n = makeNode(TriggerTransition);
+ n->name = "ivm_newtable";
+ n->isNew = true;
+ n->isTable = true;
+
+ transitionRels = lappend(transitionRels, n);
+ }
+ if (type == TRIGGER_TYPE_DELETE || type == TRIGGER_TYPE_UPDATE)
+ {
+ TriggerTransition *n = makeNode(TriggerTransition);
+ n->name = "ivm_oldtable";
+ n->isNew = false;
+ n->isTable = true;
+
+ transitionRels = lappend(transitionRels, n);
+ }
+ }
+
+ ivm_trigger->funcname =
+ (timing == TRIGGER_TYPE_BEFORE ? SystemFuncName("IVM_immediate_before") : SystemFuncName("IVM_immediate_maintenance"));
+
+ ivm_trigger->columns = NIL;
+ ivm_trigger->transitionRels = transitionRels;
+ ivm_trigger->whenClause = NULL;
+ ivm_trigger->isconstraint = false;
+ ivm_trigger->deferrable = false;
+ ivm_trigger->initdeferred = false;
+ ivm_trigger->constrrel = NULL;
+ ivm_trigger->args = list_make1(makeString(matviewname));
+
+ address = CreateTrigger(ivm_trigger, NULL, relOid, InvalidOid, InvalidOid,
+ InvalidOid, InvalidOid, InvalidOid, NULL, true, false);
+
+ recordDependencyOn(&address, &refaddr, DEPENDENCY_AUTO);
+
+ /* Make changes-so-far visible */
+ CommandCounterIncrement();
+}
+
+/*
+ * check_ivm_restriction_walker --- look for specify nodes in the query tree
+ */
+static void
+check_ivm_restriction_walker(Node *node)
+{
+ /* This can recurse, so check for excessive recursion */
+ check_stack_depth();
+
+ if (node == NULL)
+ return;
+ /*
+ * We currently don't support Sub-Query.
+ */
+ if (IsA(node, SubPlan) || IsA(node, SubLink))
+ ereport(ERROR, (errmsg("subquery is not supported with IVM")));
+
+ switch (nodeTag(node))
+ {
+ case T_Query:
+ {
+ Query *qry = (Query *)node;
+ ListCell *lc;
+ /* if contained CTE, return error */
+ if (qry->cteList != NIL)
+ ereport(ERROR, (errmsg("CTE is not supported with IVM")));
+
+ /* if contained VIEW or subquery into RTE, return error */
+ foreach(lc, qry->rtable)
+ {
+ RangeTblEntry *rte = (RangeTblEntry *) lfirst(lc);
+ if (rte->relkind == RELKIND_VIEW ||
+ rte->relkind == RELKIND_MATVIEW)
+ ereport(ERROR, (errmsg("VIEW or MATERIALIZED VIEW is not supported with IVM")));
+ if (rte->rtekind == RTE_SUBQUERY)
+ ereport(ERROR, (errmsg("subquery is not supported with IVM")));
+ }
+
+ /* search in jointree */
+ check_ivm_restriction_walker((Node *) qry->jointree);
+
+ /* search in target lists */
+ foreach(lc, qry->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ check_ivm_restriction_walker((Node *) tle->expr);
+ }
+
+ break;
+ }
+ case T_JoinExpr:
+ {
+ JoinExpr *joinexpr = (JoinExpr *)node;
+ if (joinexpr->jointype > JOIN_INNER)
+ ereport(ERROR, (errmsg("OUTER JOIN is not supported with IVM")));
+ /* left side */
+ check_ivm_restriction_walker((Node *) joinexpr->larg);
+ /* right side */
+ check_ivm_restriction_walker((Node *) joinexpr->rarg);
+ check_ivm_restriction_walker((Node *) joinexpr->quals);
+ }
+ break;
+ case T_FromExpr:
+ {
+ ListCell *lc;
+ FromExpr *fromexpr = (FromExpr *)node;
+ foreach(lc, fromexpr->fromlist)
+ {
+ check_ivm_restriction_walker((Node *) lfirst(lc));
+ }
+ check_ivm_restriction_walker((Node *) fromexpr->quals);
+ }
+ break;
+ case T_Var:
+ {
+ /* if system column, return error */
+ Var *variable = (Var *) node;
+ if (variable->varattno < 0)
+ ereport(ERROR, (errmsg("system column is not supported with IVM")));
+ }
+ break;
+ case T_BoolExpr:
+ {
+ BoolExpr *boolexpr = (BoolExpr *) node;
+ ListCell *lc;
+
+ foreach(lc, boolexpr->args)
+ {
+ Node *arg = (Node *) lfirst(lc);
+ check_ivm_restriction_walker(arg);
+ }
+ break;
+ }
+ case T_NullIfExpr: /* same as OpExpr */
+ case T_DistinctExpr: /* same as OpExpr */
+ case T_OpExpr:
+ {
+ OpExpr *op = (OpExpr *) node;
+ ListCell *lc;
+ foreach(lc, op->args)
+ {
+ Node *arg = (Node *) lfirst(lc);
+ check_ivm_restriction_walker(arg);
+ }
+ break;
+ }
+ case T_CaseExpr:
+ {
+ CaseExpr *caseexpr = (CaseExpr *) node;
+ ListCell *lc;
+ /* result for ELSE clause */
+ check_ivm_restriction_walker((Node *) caseexpr->defresult);
+ /* expr for WHEN clauses */
+ foreach(lc, caseexpr->args)
+ {
+ CaseWhen *when = (CaseWhen *) lfirst(lc);
+ Node *w_expr = (Node *) when->expr;
+ /* result for WHEN clause */
+ check_ivm_restriction_walker((Node *) when->result);
+ /* expr clause*/
+ check_ivm_restriction_walker((Node *) w_expr);
+ }
+ break;
+ }
+ case T_SubLink:
+ {
+ /* Now, not supported */
+/*
+ SubLink *sublink = (SubLink *) node;
+ Query *qry =(Query *) sublink->subselect;
+ if (qry != NULL && qry->jointree != NULL)
+ check_ivm_restriction_walker((Node *) qry->jointree->quals);
+*/
+ break;
+ }
+ case T_SubPlan:
+ {
+ /* Now, not supported */
+ break;
+ }
+ case T_Aggref:
+ case T_GroupingFunc:
+ case T_WindowFunc:
+ case T_FuncExpr:
+ case T_SQLValueFunction:
+ case T_Const:
+ case T_Param:
+ default:
+ /* do nothing */
+ break;
+ }
+
+ return;
+}
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index 537d0e8cef..c2831590b8 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -46,6 +46,79 @@
#include "utils/snapmgr.h"
#include "utils/syscache.h"
+#include "utils/regproc.h"
+#include "nodes/makefuncs.h"
+#include "parser/analyze.h"
+#include "parser/parser.h"
+#include "parser/parse_clause.h"
+#include "parser/parse_func.h"
+#include "parser/parse_type.h"
+#include "nodes/print.h"
+#include "catalog/pg_type_d.h"
+#include "optimizer/optimizer.h"
+#include "commands/defrem.h"
+
+/*
+ * Local definitions
+ */
+
+#define MV_INIT_QUERYHASHSIZE 16
+
+/* MV query type codes */
+#define MV_PLAN_RECALC_MINMAX 1
+#define MV_PLAN_SET_MINMAX 2
+
+/*
+ * MI_QueryKey
+ *
+ * The key identifying a prepared SPI plan in our query hashtable
+ */
+typedef struct MV_QueryKey
+{
+ Oid matview_id; /* OID of materialized view */
+ int32 query_type; /* query type ID, see MV_PLAN_XXX above */
+} MV_QueryKey;
+
+/*
+ * MV_QueryHashEntry
+ */
+typedef struct MV_QueryHashEntry
+{
+ MV_QueryKey key;
+ SPIPlanPtr plan;
+} MV_QueryHashEntry;
+
+/*
+ * MV_TriggerHashEntry
+ */
+typedef struct MV_TriggerHashEntry
+{
+ Oid matview_id;
+ int before_trig_count;
+ int after_trig_count;
+ TransactionId xid;
+ CommandId cid;
+ List *tables;
+ bool has_old;
+ bool has_new;
+} MV_TriggerHashEntry;
+
+/*
+ * MV_TiggerTable
+ */
+typedef struct MV_TriggerTable
+{
+ Oid table_id;
+ List *old_tuplestores;
+ List *new_tuplestores;
+ RangeTblEntry *original_rte;
+ List *old_rtes;
+ List *new_rtes;
+ List *rte_indexes;
+} MV_TriggerTable;
+
+static HTAB *mv_query_cache = NULL;
+static HTAB *mv_trigger_info = NULL;
typedef struct
{
@@ -58,6 +131,12 @@ typedef struct
BulkInsertState bistate; /* bulk insert state */
} DR_transientrel;
+typedef enum
+{
+ IVM_ADD,
+ IVM_SUB
+} IvmOp;
+
static int matview_maintenance_depth = 0;
static void transientrel_startup(DestReceiver *self, int operation, TupleDesc typeinfo);
@@ -65,7 +144,8 @@ static bool transientrel_receive(TupleTableSlot *slot, DestReceiver *self);
static void transientrel_shutdown(DestReceiver *self);
static void transientrel_destroy(DestReceiver *self);
static uint64 refresh_matview_datafill(DestReceiver *dest, Query *query,
- const char *queryString);
+ QueryEnvironment *queryEnv,
+ const char *queryString);
static char *make_temptable_name_n(char *tempname, int n);
static void refresh_by_match_merge(Oid matviewOid, Oid tempOid, Oid relowner,
int save_sec_context);
@@ -74,6 +154,33 @@ static bool is_usable_unique_index(Relation indexRel);
static void OpenMatViewIncrementalMaintenance(void);
static void CloseMatViewIncrementalMaintenance(void);
+static char *get_null_condition_string(IvmOp op, char *arg1, char *arg2, char* count_col);
+static char *get_operation_string(IvmOp op, char *col, char *arg1, char *arg2,
+ char* count_col, const char *castType);
+static Query* rewrite_query_for_preupdate_state(Query *query, List *tables, TransactionId xid, CommandId cid, ParseState *pstate);
+static Query *rewrite_query_for_counting_and_aggregation(Query *query, ParseState *pstate);
+static void calc_delta(MV_TriggerTable *table, int index, Query *query,
+ DestReceiver *dest_old, DestReceiver *dest_new, QueryEnvironment *queryEnv);
+static RangeTblEntry* union_ENRs(RangeTblEntry *rte, Oid relid, List *enr_rtes, const char *prefix, QueryEnvironment *queryEnv);
+static char *make_delta_enr_name(const char *prefix, Oid relid, int count);
+static void register_delta_ENRs(ParseState *pstate, Query *query, List *tables);
+static void apply_delta(Oid matviewOid, Oid tempOid_new, Oid tempOid_old, Query *query);
+static void truncate_view_delta(Oid delta_oid);
+static void clean_up_IVM_hash_entry(MV_TriggerHashEntry *entry);
+static void clean_up_IVM_temptable(Oid tempOid_old, Oid tempOid_new);
+
+static void mv_InitHashTables(void);
+static SPIPlanPtr mv_FetchPreparedPlan(MV_QueryKey *key);
+static void mv_HashPreparedPlan(MV_QueryKey *key, SPIPlanPtr plan);
+static void mv_BuildQueryKey(MV_QueryKey *key, Oid matview_id, int32 query_type);
+static SPIPlanPtr get_plan_for_recalc_min_max(Oid matviewOid, const char *min_max_list,
+ const char *group_keys, int nkeys, Oid *keyTypes, bool with_group);
+static SPIPlanPtr get_plan_for_set_min_max(Oid matviewOid, char *matviewname, const char *min_max_list,
+ int num_min_max, Oid *valTypes, bool with_group);
+
+static Query *get_matview_query(Relation matviewRel);
+
+
/*
* SetMatViewPopulatedState
* Mark a materialized view as populated, or not.
@@ -114,6 +221,46 @@ SetMatViewPopulatedState(Relation relation, bool newstate)
CommandCounterIncrement();
}
+/*
+ * SetMatViewIVMState
+ * Mark a materialized view as IVM, or not.
+ *
+ * NOTE: caller must be holding an appropriate lock on the relation.
+ */
+void
+SetMatViewIVMState(Relation relation, bool newstate)
+{
+ Relation pgrel;
+ HeapTuple tuple;
+
+ Assert(relation->rd_rel->relkind == RELKIND_MATVIEW);
+
+ /*
+ * Update relation's pg_class entry. Crucial side-effect: other backends
+ * (and this one too!) are sent SI message to make them rebuild relcache
+ * entries.
+ */
+ pgrel = table_open(RelationRelationId, RowExclusiveLock);
+ tuple = SearchSysCacheCopy1(RELOID,
+ ObjectIdGetDatum(RelationGetRelid(relation)));
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for relation %u",
+ RelationGetRelid(relation));
+
+ ((Form_pg_class) GETSTRUCT(tuple))->relisivm = newstate;
+
+ CatalogTupleUpdate(pgrel, &tuple->t_self, tuple);
+
+ heap_freetuple(tuple);
+ table_close(pgrel, RowExclusiveLock);
+
+ /*
+ * Advance command counter to make the updated pg_class row locally
+ * visible.
+ */
+ CommandCounterIncrement();
+}
+
/*
* ExecRefreshMatView -- execute a REFRESH MATERIALIZED VIEW command
*
@@ -140,8 +287,6 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
{
Oid matviewOid;
Relation matviewRel;
- RewriteRule *rule;
- List *actions;
Query *dataQuery;
Oid tableSpace;
Oid relowner;
@@ -187,32 +332,8 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("CONCURRENTLY and WITH NO DATA options cannot be used together")));
- /*
- * Check that everything is correct for a refresh. Problems at this point
- * are internal errors, so elog is sufficient.
- */
- if (matviewRel->rd_rel->relhasrules == false ||
- matviewRel->rd_rules->numLocks < 1)
- elog(ERROR,
- "materialized view \"%s\" is missing rewrite information",
- RelationGetRelationName(matviewRel));
-
- if (matviewRel->rd_rules->numLocks > 1)
- elog(ERROR,
- "materialized view \"%s\" has too many rules",
- RelationGetRelationName(matviewRel));
-
- rule = matviewRel->rd_rules->rules[0];
- if (rule->event != CMD_SELECT || !(rule->isInstead))
- elog(ERROR,
- "the rule for materialized view \"%s\" is not a SELECT INSTEAD OF rule",
- RelationGetRelationName(matviewRel));
- actions = rule->actions;
- if (list_length(actions) != 1)
- elog(ERROR,
- "the rule for materialized view \"%s\" is not a single action",
- RelationGetRelationName(matviewRel));
+ dataQuery = get_matview_query(matviewRel);
/*
* Check that there is a unique index with no WHERE clause on one or more
@@ -247,12 +368,6 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
errhint("Create a unique index with no WHERE clause on one or more columns of the materialized view.")));
}
- /*
- * The stored query was rewritten at the time of the MV definition, but
- * has not been scribbled on by the planner.
- */
- dataQuery = linitial_node(Query, actions);
-
/*
* Check for active uses of the relation in the current transaction, such
* as open scans.
@@ -311,7 +426,7 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
/* Generate the data, if wanted. */
if (!stmt->skipData)
- processed = refresh_matview_datafill(dest, dataQuery, queryString);
+ processed = refresh_matview_datafill(dest, dataQuery, NULL, queryString);
/* Make the matview match the newly generated data. */
if (concurrent)
@@ -369,6 +484,7 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
*/
static uint64
refresh_matview_datafill(DestReceiver *dest, Query *query,
+ QueryEnvironment *queryEnv,
const char *queryString)
{
List *rewritten;
@@ -405,7 +521,7 @@ refresh_matview_datafill(DestReceiver *dest, Query *query,
/* Create a QueryDesc, redirecting output to our tuple receiver */
queryDesc = CreateQueryDesc(plan, queryString,
GetActiveSnapshot(), InvalidSnapshot,
- dest, NULL, NULL, 0);
+ dest, NULL, queryEnv ? queryEnv: NULL, 0);
/* call ExecutorStart to prepare the plan for execution */
ExecutorStart(queryDesc, 0);
@@ -926,3 +1042,1580 @@ CloseMatViewIncrementalMaintenance(void)
matview_maintenance_depth--;
Assert(matview_maintenance_depth >= 0);
}
+
+/*
+ * IVM trigger function
+ */
+
+Datum
+IVM_immediate_before(PG_FUNCTION_ARGS)
+{
+ TriggerData *trigdata = (TriggerData *) fcinfo->context;
+ char *matviewname = trigdata->tg_trigger->tgargs[0];
+ List *names = stringToQualifiedNameList(matviewname);
+ Oid matviewOid;
+
+ MV_TriggerHashEntry *entry;
+ bool found;
+
+ /*
+ * Wait for concurrent transactions which update this materialized view at READ COMMITED.
+ * This is needed to see changes commited in othre transactions. No wait and raise an error
+ * at REPEATABLE READ or SERIALIZABLE to prevent anormal update of matviews.
+ * XXX: dead-lock is possible here.
+ */
+ if (!IsolationUsesXactSnapshot())
+ matviewOid = RangeVarGetRelid(makeRangeVarFromNameList(names), ExclusiveLock, true);
+ else
+ matviewOid = RangeVarGetRelidExtended(makeRangeVarFromNameList(names), ExclusiveLock, RVR_MISSING_OK | RVR_NOWAIT, NULL, NULL);
+
+ /*
+ * On the first call initialize the hashtable
+ */
+ if (!mv_query_cache)
+ mv_InitHashTables();
+
+ entry = (MV_TriggerHashEntry *) hash_search(mv_trigger_info,
+ (void *) &matviewOid,
+ HASH_ENTER, &found);
+
+ /* On the first BEFORE to update the view, initialize trigger data */
+ if (!found)
+ {
+ Snapshot snapshot = GetActiveSnapshot();
+
+ entry->matview_id = matviewOid;
+ entry->before_trig_count = 0;
+ entry->after_trig_count = 0;
+ entry->xid = GetCurrentTransactionId();
+ entry->cid = snapshot->curcid;
+ entry->tables = NIL;
+ entry->has_old = false;
+ entry->has_new = false;
+
+ }
+
+ entry->before_trig_count++;
+
+ return PointerGetDatum(NULL);
+}
+
+Datum
+IVM_immediate_maintenance(PG_FUNCTION_ARGS)
+{
+ TriggerData *trigdata = (TriggerData *) fcinfo->context;
+ Relation rel;
+ Oid relid;
+ Oid matviewOid;
+ Query *query, *rewritten;
+ char* matviewname = trigdata->tg_trigger->tgargs[0];
+ List *names;
+ Relation matviewRel;
+ int old_depth = matview_maintenance_depth;
+
+ Oid tableSpace;
+ Oid relowner;
+ Oid OIDDelta_new = InvalidOid;
+ Oid OIDDelta_old = InvalidOid;
+ DestReceiver *dest_new = NULL, *dest_old = NULL;
+ Oid save_userid;
+ int save_sec_context;
+ int save_nestlevel;
+
+ MV_TriggerHashEntry *entry;
+ MV_TriggerTable *table;
+ bool found;
+ ListCell *lc;
+ MemoryContext oldcxt;
+
+
+ QueryEnvironment *queryEnv = create_queryEnv();
+ ParseState *pstate;
+
+ /* Create a ParseState for rewriting the view definition query */
+ pstate = make_parsestate(NULL);
+ pstate->p_queryEnv = queryEnv;
+ pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET;
+
+ rel = trigdata->tg_relation;
+ relid = rel->rd_id;
+
+ names = stringToQualifiedNameList(matviewname);
+ matviewOid = RangeVarGetRelid(makeRangeVarFromNameList(names), NoLock, true);
+
+
+ /*
+ * On the first call initialize the hashtable
+ */
+ if (!mv_query_cache)
+ mv_InitHashTables();
+
+ entry = (MV_TriggerHashEntry *) hash_search(mv_trigger_info,
+ (void *) &matviewOid,
+ HASH_FIND, &found);
+
+ Assert (entry != NULL);
+
+ entry->after_trig_count++;
+
+ found = false;
+ foreach(lc, entry->tables)
+ {
+ table = (MV_TriggerTable *) lfirst(lc);
+ if (table->table_id == relid)
+ {
+ found = true;
+ break;
+ }
+ }
+ if (!found)
+ {
+ oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+
+ table = (MV_TriggerTable *) palloc0(sizeof(MV_TriggerTable));
+ table->table_id = relid;
+ table->old_tuplestores = NIL;
+ table->new_tuplestores = NIL;
+ table->old_rtes = NIL;
+ table->new_rtes = NIL;
+ table->rte_indexes = NIL;
+ entry->tables = lappend(entry->tables, table);
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ if (trigdata->tg_oldtable)
+ {
+ oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ table->old_tuplestores = lappend(table->old_tuplestores, trigdata->tg_oldtable);
+ entry->has_old = true;
+ MemoryContextSwitchTo(oldcxt);
+ }
+ if (trigdata->tg_newtable)
+ {
+ oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ table->new_tuplestores = lappend(table->new_tuplestores, trigdata->tg_newtable);
+ entry->has_new = true;
+ MemoryContextSwitchTo(oldcxt);
+ }
+ if (entry->has_new || entry->has_old)
+ {
+ CmdType cmd;
+
+ if (TRIGGER_FIRED_BY_INSERT(trigdata->tg_event))
+ cmd = CMD_INSERT;
+ else if (TRIGGER_FIRED_BY_DELETE(trigdata->tg_event))
+ cmd = CMD_DELETE;
+ else if (TRIGGER_FIRED_BY_UPDATE(trigdata->tg_event))
+ cmd = CMD_UPDATE;
+ else
+ elog(ERROR,"unsupported trigger type");
+
+ SetTransitionTablePreserved(relid, cmd);
+ }
+
+
+ /* If this is not the last AFTER trigger call, immediately exit. */
+ Assert (entry->before_trig_count >= entry->after_trig_count);
+ if (entry->before_trig_count != entry->after_trig_count)
+ return PointerGetDatum(NULL);
+
+
+ /* If this is the last AFTER trigger call, update the view. */
+
+ matviewRel = table_open(matviewOid, NoLock);
+
+ /* get view query*/
+ query = get_matview_query(matviewRel);
+
+ /* Make sure it is a materialized view. */
+ Assert(matviewRel->rd_rel->relkind == RELKIND_MATVIEW);
+
+ /*
+ * Get and push the latast snapshot to see any changes which is commited during waiting in
+ * other transactions at READ COMMITTED level.
+ * XXX: Is this safe?
+ */
+ PushActiveSnapshot(GetTransactionSnapshot());
+
+ /*
+ * Check for active uses of the relation in the current transaction, such
+ * as open scans.
+ *
+ * NB: We count on this to protect us against problems with refreshing the
+ * data using TABLE_INSERT_FROZEN.
+ */
+ // XXX: necesarry?
+ CheckTableNotInUse(matviewRel, "REFRESH MATERIALIZED VIEW");
+
+ /* rewrite query */
+ rewritten = rewrite_query_for_preupdate_state(query, entry->tables, entry->xid, entry->cid, pstate);
+ rewritten = rewrite_query_for_counting_and_aggregation(rewritten, pstate);
+
+ relowner = matviewRel->rd_rel->relowner;
+
+ /*
+ * Switch to the owner's userid, so that any functions are run as that
+ * user. Also arrange to make GUC variable changes local to this command.
+ * Don't lock it down too tight to create a temporary table just yet. We
+ * will switch modes when we are about to execute user code.
+ */
+ GetUserIdAndSecContext(&save_userid, &save_sec_context);
+ SetUserIdAndSecContext(relowner,
+ save_sec_context | SECURITY_LOCAL_USERID_CHANGE);
+ save_nestlevel = NewGUCNestLevel();
+
+ /* Create temporary tables to store view deltas */
+ tableSpace = GetDefaultTablespace(RELPERSISTENCE_TEMP, false);
+ if (entry->has_old)
+ {
+ OIDDelta_old = make_new_heap(matviewOid, tableSpace, RELPERSISTENCE_TEMP,
+ ExclusiveLock);
+ LockRelationOid(OIDDelta_old, AccessExclusiveLock);
+ dest_old = CreateTransientRelDestReceiver(OIDDelta_old);
+ }
+ if (entry->has_new)
+ {
+ if (entry->has_old)
+ OIDDelta_new = make_new_heap(OIDDelta_old, tableSpace, RELPERSISTENCE_TEMP,
+ ExclusiveLock);
+ else
+ OIDDelta_new = make_new_heap(matviewOid, tableSpace, RELPERSISTENCE_TEMP,
+ ExclusiveLock);
+ LockRelationOid(OIDDelta_new, AccessExclusiveLock);
+ dest_new = CreateTransientRelDestReceiver(OIDDelta_new);
+ }
+
+ /*
+ * Now lock down security-restricted operations.
+ */
+ SetUserIdAndSecContext(relowner,
+ save_sec_context | SECURITY_RESTRICTED_OPERATION);
+
+ /* for all modified tables */
+ foreach(lc, entry->tables)
+ {
+ ListCell *lc2;
+
+ table = (MV_TriggerTable *) lfirst(lc);
+
+ /* loop for self-join */
+ foreach(lc2, table->rte_indexes)
+ {
+ int index = lfirst_int(lc2);
+
+ calc_delta(table, index, rewritten, dest_old, dest_new, queryEnv);
+
+ PG_TRY();
+ {
+ apply_delta(matviewOid, OIDDelta_new, OIDDelta_old, query);
+ }
+ PG_CATCH();
+ {
+ matview_maintenance_depth = old_depth;
+ PG_RE_THROW();
+ }
+ PG_END_TRY();
+
+ /* truncate view delta tables */
+ truncate_view_delta(OIDDelta_old);
+ truncate_view_delta(OIDDelta_new);
+ }
+ }
+
+ /* Pop the original snapshot. */
+ PopActiveSnapshot();
+
+ table_close(matviewRel, NoLock);
+
+ /* Roll back any GUC changes */
+ AtEOXact_GUC(false, save_nestlevel);
+
+ /* Restore userid and security context */
+ SetUserIdAndSecContext(save_userid, save_sec_context);
+
+ /* Clean up hash entry and drop temporary tables */
+ clean_up_IVM_hash_entry(entry);
+ clean_up_IVM_temptable(OIDDelta_old, OIDDelta_new);
+
+ return PointerGetDatum(NULL);
+}
+
+#define IVM_colname(type, col) makeObjectName("__ivm_" type, col, "_")
+
+static char *
+get_null_condition_string(IvmOp op, char *arg1, char *arg2, char* count_col)
+{
+ StringInfoData null_cond;
+ initStringInfo(&null_cond);
+
+ switch (op)
+ {
+ case IVM_ADD:
+ appendStringInfo(&null_cond,
+ "%s = 0 AND %s = 0",
+ quote_qualified_identifier(arg1, count_col),
+ quote_qualified_identifier(arg2, count_col)
+ );
+ break;
+ case IVM_SUB:
+ appendStringInfo(&null_cond,
+ "%s = %s",
+ quote_qualified_identifier(arg1, count_col),
+ quote_qualified_identifier(arg2, count_col)
+ );
+ break;
+ default:
+ elog(ERROR,"unkwon opration");
+ }
+
+ return null_cond.data;
+}
+
+static char *
+get_operation_string(IvmOp op, char *col, char *arg1, char *arg2, char* count_col, const char *castType)
+{
+ StringInfoData buf, castString;
+ char *col1 = quote_qualified_identifier(arg1, col);
+ char *col2 = quote_qualified_identifier(arg2, col);
+
+ char op_char = (op == IVM_SUB ? '-' : '+');
+ char *sign = (op == IVM_SUB ? "-" : "");
+
+ initStringInfo(&buf);
+ initStringInfo(&castString);
+
+ if (castType)
+ appendStringInfo(&castString, "::%s", castType);
+
+
+ if (!count_col)
+ {
+ appendStringInfo(&buf, "(%s %c %s)%s",
+ col1, op_char, col2, castString.data);
+ }
+ else
+ {
+ char *null_cond = get_null_condition_string(op, arg1, arg2, count_col);
+
+ appendStringInfo(&buf,
+ "(CASE WHEN %s THEN NULL "
+ " WHEN %s IS NULL THEN %s%s "
+ " WHEN %s IS NULL THEN (%s) "
+ " ELSE (%s %c %s)%s END)",
+ null_cond,
+ col1, sign, col2,
+ col2, col1,
+ col1, op_char, col2, castString.data
+ );
+ }
+
+ return buf.data;
+}
+
+
+static void
+calc_delta(MV_TriggerTable *table, int index, Query *query,
+ DestReceiver *dest_old, DestReceiver *dest_new, QueryEnvironment *queryEnv)
+{
+ ListCell *lc;
+ RangeTblEntry *rte;
+
+ lc = list_nth_cell(query->rtable, index);
+ rte = (RangeTblEntry *) lfirst(lc);
+
+ /* Generate old delta */
+ if (list_length(table->old_rtes) > 0)
+ {
+ lfirst(lc) = union_ENRs(rte, table->table_id, table->old_rtes, "old", queryEnv);
+ refresh_matview_datafill(dest_old, query, queryEnv, NULL);
+ }
+
+ /* Generate new delta */
+ if (list_length(table->new_rtes) > 0)
+ {
+ lfirst(lc) = union_ENRs(rte, table->table_id, table->new_rtes, "new", queryEnv);
+ refresh_matview_datafill(dest_new, query, queryEnv, NULL);
+ }
+
+ /* Retore the original RTE */
+ lfirst(lc) = table->original_rte;
+}
+
+
+static RangeTblEntry*
+union_ENRs(RangeTblEntry *rte, Oid relid, List *enr_rtes, const char *prefix, QueryEnvironment *queryEnv)
+{
+ StringInfoData str;
+ ParseState *pstate;
+ RawStmt *raw;
+ Query *sub;
+ int i;
+
+ /* Create a ParseState for rewriting the view definition query */
+ pstate = make_parsestate(NULL);
+ pstate->p_queryEnv = queryEnv;
+ pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET;
+
+ initStringInfo(&str);
+
+ for (i = 0; i < list_length(enr_rtes); i++)
+ {
+ if (i > 0)
+ appendStringInfo(&str, " UNION ALL ");
+ appendStringInfo(&str," SELECT * FROM %s", make_delta_enr_name(prefix, relid, i));
+ }
+
+ raw = (RawStmt*)linitial(raw_parser(str.data));
+ sub = transformStmt(pstate, raw->stmt);
+
+ rte->rtekind = RTE_SUBQUERY;
+ rte->subquery = sub;
+ rte->security_barrier = false;
+ /* Clear fields that should not be set in a subquery RTE */
+ rte->relid = InvalidOid;
+ rte->relkind = 0;
+ rte->rellockmode = 0;
+ rte->tablesample = NULL;
+ rte->inh = false; /* must not be set for a subquery */
+
+ rte->requiredPerms = 0; /* no permission check on subquery itself */
+ rte->checkAsUser = InvalidOid;
+ rte->selectedCols = NULL;
+ rte->insertedCols = NULL;
+ rte->updatedCols = NULL;
+ rte->extraUpdatedCols = NULL;
+
+ return rte;
+}
+
+static char*
+make_delta_enr_name(const char *prefix, Oid relid, int count)
+{
+ char buf[NAMEDATALEN];
+ char *name;
+
+ snprintf(buf, NAMEDATALEN, "%s_%u_%u", prefix, relid, count);
+ name = pstrdup(buf);
+
+ return name;
+}
+
+static void
+register_delta_ENRs(ParseState *pstate, Query *query, List *tables)
+{
+ QueryEnvironment *queryEnv = pstate->p_queryEnv;
+ ListCell *lc;
+ RangeTblEntry *rte;
+
+ foreach(lc, tables)
+ {
+ MV_TriggerTable *table = (MV_TriggerTable *) lfirst(lc);
+ ListCell *lc2;
+ int count;
+
+ count = 0;
+ foreach(lc2, table->old_tuplestores)
+ {
+ Tuplestorestate *oldtable = (Tuplestorestate *) lfirst(lc2);
+ EphemeralNamedRelation enr =
+ palloc(sizeof(EphemeralNamedRelationData));
+
+ enr->md.name = make_delta_enr_name("old", table->table_id, count);
+ enr->md.reliddesc = table->table_id;
+ enr->md.tupdesc = NULL;
+ enr->md.enrtype = ENR_NAMED_TUPLESTORE;
+ enr->md.enrtuples = tuplestore_tuple_count(oldtable);
+ enr->reldata = oldtable;
+ register_ENR(queryEnv, enr);
+
+ rte = addRangeTableEntryForENR(pstate, makeRangeVar(NULL, enr->md.name, -1), true);
+ query->rtable = lappend(query->rtable, rte);
+ table->old_rtes = lappend(table->old_rtes, rte);
+
+ count++;
+ }
+
+ count = 0;
+ foreach(lc2, table->new_tuplestores)
+ {
+ Tuplestorestate *newtable = (Tuplestorestate *) lfirst(lc2);
+ EphemeralNamedRelation enr =
+ palloc(sizeof(EphemeralNamedRelationData));
+
+ enr->md.name = make_delta_enr_name("new", table->table_id, count);
+ enr->md.reliddesc = table->table_id;
+ enr->md.tupdesc = NULL;
+ enr->md.enrtype = ENR_NAMED_TUPLESTORE;
+ enr->md.enrtuples = tuplestore_tuple_count(newtable);
+ enr->reldata = newtable;
+ register_ENR(queryEnv, enr);
+
+ rte = addRangeTableEntryForENR(pstate, makeRangeVar(NULL, enr->md.name, -1), true);
+ query->rtable = lappend(query->rtable, rte);
+ table->new_rtes = lappend(table->new_rtes, rte);
+
+ count++;
+ }
+ }
+}
+
+static RangeTblEntry*
+get_prestate_rte(RangeTblEntry *rte, MV_TriggerTable *table, TransactionId xid, CommandId cid, QueryEnvironment *queryEnv)
+{
+ StringInfoData str;
+ RawStmt *raw;
+ Query *sub;
+ Relation rel;
+ ParseState *pstate;
+ char *relname;
+ int i;
+
+ pstate = make_parsestate(NULL);
+ pstate->p_queryEnv = queryEnv;
+ pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET;
+
+ /*
+ * We can use NoLock here since AcquireRewriteLocks should
+ * have locked the rel already.
+ */
+ rel = table_open(table->table_id, NoLock);
+ relname = quote_qualified_identifier(
+ get_namespace_name(RelationGetNamespace(rel)),
+ RelationGetRelationName(rel));
+ table_close(rel, NoLock);
+
+ initStringInfo(&str);
+ appendStringInfo(&str,
+ "SELECT t.* FROM %s t"
+ " WHERE (age(t.xmin) - age(%u::text::xid) > 0) OR"
+ " (t.xmin = %u AND t.cmin::text::int < %u)",
+ relname, xid, xid, cid);
+
+ for (i=0; i<list_length(table->old_tuplestores); i++)
+ {
+ appendStringInfo(&str, " UNION ALL ");
+ appendStringInfo(&str," SELECT * FROM %s", make_delta_enr_name("old", table->table_id, i));
+ }
+
+ raw = (RawStmt*)linitial(raw_parser(str.data));
+ sub = transformStmt(pstate, raw->stmt);
+
+ table->original_rte = copyObject(rte);
+
+ rte->rtekind = RTE_SUBQUERY;
+ rte->subquery = sub;
+ rte->security_barrier = false;
+ /* Clear fields that should not be set in a subquery RTE */
+ rte->relid = InvalidOid;
+ rte->relkind = 0;
+ rte->rellockmode = 0;
+ rte->tablesample = NULL;
+ rte->inh = false; /* must not be set for a subquery */
+
+ rte->requiredPerms = 0; /* no permission check on subquery itself */
+ rte->checkAsUser = InvalidOid;
+ rte->selectedCols = NULL;
+ rte->insertedCols = NULL;
+ rte->updatedCols = NULL;
+ rte->extraUpdatedCols = NULL;
+
+ return rte;
+}
+
+static Query*
+rewrite_query_for_preupdate_state(Query *query, List *tables, TransactionId xid, CommandId cid, ParseState *pstate)
+{
+ Query *rewritten = copyObject(query);
+ ListCell *lc;
+ int num_rte = list_length(query->rtable);
+ int i;
+
+ register_delta_ENRs(pstate, rewritten, tables);
+
+ // XXX: Is necessary? Is this right timing?
+ AcquireRewriteLocks(rewritten, true, false);
+
+ foreach(lc, tables)
+ {
+ MV_TriggerTable *table = (MV_TriggerTable *) lfirst(lc);
+ ListCell *lc2;
+
+ i = 0;
+ foreach(lc2, rewritten->rtable)
+ {
+ RangeTblEntry *r = (RangeTblEntry*) lfirst(lc2);
+
+ if (r->relid == table->table_id)
+ {
+ lfirst(lc2) = get_prestate_rte(r, table, xid, cid, pstate->p_queryEnv);
+ table->rte_indexes = lappend_int(table->rte_indexes, i);
+ }
+
+ if (++i >= num_rte)
+ break;
+ }
+ }
+
+ return rewritten;
+}
+
+static Query *
+rewrite_query_for_counting_and_aggregation(Query *query, ParseState *pstate)
+{
+ TargetEntry *tle_count;
+ FuncCall *fn;
+ Node *node;
+ Const *dmy_arg = makeConst(INT4OID,
+ -1,
+ InvalidOid,
+ sizeof(int32),
+ Int32GetDatum(1),
+ false,
+ true);
+
+ if (query->hasAggs)
+ {
+ ListCell *lc;
+ List *agg_counts = NIL;
+ AttrNumber next_resno = list_length(query->targetList) + 1;
+
+ foreach(lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+
+ if (IsA(tle->expr, Aggref))
+ {
+ Aggref *aggref = (Aggref *) tle->expr;
+ const char *aggname = get_func_name(aggref->aggfnoid);
+
+ /*
+ * For aggregate functions except to count, add count func with the same arg parameters.
+ * Also, add sum func for agv.
+ *
+ * XXX: need some generalization
+ * XXX: If there are same expressions explicitly in the target list, we can use this instead
+ * of adding new duplicated one.
+ */
+ if (strcmp(aggname, "count") != 0)
+ {
+ fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+
+ /* Make a Func with a dummy arg, and then override this by the original agg's args. */
+ node = ParseFuncOrColumn(pstate, fn->funcname, list_make1(dmy_arg), NULL, fn, false, -1);
+ ((Aggref *)node)->args = aggref->args;
+
+ tle_count = makeTargetEntry((Expr *) node,
+ next_resno,
+ NULL,
+ false);
+ agg_counts = lappend(agg_counts, tle_count);
+ next_resno++;
+ }
+ if (strcmp(aggname, "avg") == 0)
+ {
+ List *dmy_args = NIL;
+ ListCell *lc;
+ foreach(lc, aggref->aggargtypes)
+ {
+ Oid typeid = lfirst_oid(lc);
+ Type type = typeidType(typeid);
+
+ Const *con = makeConst(typeid,
+ -1,
+ typeTypeCollation(type),
+ typeLen(type),
+ (Datum) 0,
+ true,
+ typeByVal(type));
+ dmy_args = lappend(dmy_args, con);
+ ReleaseSysCache(type);
+
+ }
+ fn = makeFuncCall(list_make1(makeString("sum")), NIL, -1);
+
+ /* Make a Func with a dummy arg, and then override this by the original agg's args. */
+ node = ParseFuncOrColumn(pstate, fn->funcname, dmy_args, NULL, fn, false, -1);
+ ((Aggref *)node)->args = aggref->args;
+
+ tle_count = makeTargetEntry((Expr *) node,
+ next_resno,
+ NULL,
+ false);
+ agg_counts = lappend(agg_counts, tle_count);
+ next_resno++;
+ }
+ }
+
+ }
+ query->targetList = list_concat(query->targetList, agg_counts);
+ }
+
+ fn = makeFuncCall(list_make1(makeString("count")), NIL, -1);
+ fn->agg_star = true;
+ if (!query->groupClause && !query->hasAggs)
+ query->groupClause = transformDistinctClause(NULL, &query->targetList, query->sortClause, false);
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle_count = makeTargetEntry((Expr *) node,
+ list_length(query->targetList) + 1,
+ NULL,
+ false);
+ query->targetList = lappend(query->targetList, tle_count);
+ query->hasAggs = true;
+
+ return query;
+}
+
+
+static void
+apply_delta(Oid matviewOid, Oid tempOid_new, Oid tempOid_old, Query *query)
+{
+ StringInfoData querybuf;
+ StringInfoData mvatts_buf, diffatts_buf;
+ StringInfoData mv_gkeys_buf, diff_gkeys_buf, updt_gkeys_buf;
+ StringInfoData diff_aggs_buf, update_aggs_old, update_aggs_new;
+ StringInfoData returning_buf, result_buf;
+ StringInfoData min_or_max_buf;
+ Relation matviewRel;
+ Relation tempRel_new = NULL, tempRel_old = NULL;
+ char *matviewname;
+ char *tempname_new = NULL, *tempname_old = NULL;
+ ListCell *lc;
+ char *sep, *sep_agg;
+ bool with_group = query->groupClause != NULL;
+ int i;
+ bool has_min_or_max = false;
+ int num_group_keys = 0;
+ int num_min_or_max = 0;
+
+
+ initStringInfo(&querybuf);
+ matviewRel = table_open(matviewOid, NoLock);
+ matviewname = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(matviewRel)),
+ RelationGetRelationName(matviewRel));
+
+ if (OidIsValid(tempOid_new))
+ {
+ tempRel_new = table_open(tempOid_new, NoLock);
+ tempname_new = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(tempRel_new)),
+ RelationGetRelationName(tempRel_new));
+ }
+ if (OidIsValid(tempOid_old))
+ {
+ tempRel_old = table_open(tempOid_old, NoLock);
+ tempname_old = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(tempRel_old)),
+ RelationGetRelationName(tempRel_old));
+ }
+
+ initStringInfo(&mvatts_buf);
+ initStringInfo(&diffatts_buf);
+ initStringInfo(&diff_aggs_buf);
+ initStringInfo(&update_aggs_old);
+ initStringInfo(&update_aggs_new);
+ initStringInfo(&returning_buf);
+ initStringInfo(&result_buf);
+ initStringInfo(&min_or_max_buf);
+
+ sep = "";
+ sep_agg= "";
+ i = 0;
+ foreach (lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, i);
+ char *resname = NameStr(attr->attname);
+
+ i++;
+
+ if (tle->resjunk)
+ continue;
+
+ appendStringInfo(&mvatts_buf, "%s", sep);
+ appendStringInfo(&diffatts_buf, "%s", sep);
+ sep = ", ";
+
+ appendStringInfo(&mvatts_buf, "%s", quote_qualified_identifier("mv", resname));
+ appendStringInfo(&diffatts_buf, "%s", quote_qualified_identifier("diff", resname));
+
+ if (query->hasAggs && IsA(tle->expr, Aggref))
+ {
+ Aggref *aggref = (Aggref *) tle->expr;
+ const char *aggname = get_func_name(aggref->aggfnoid);
+ const char *aggtype = format_type_be(aggref->aggtype); /* XXX: should be add_cast_to ? */
+
+ appendStringInfo(&update_aggs_old, "%s", sep_agg);
+ appendStringInfo(&update_aggs_new, "%s", sep_agg);
+ appendStringInfo(&diff_aggs_buf, "%s", sep_agg);
+
+ sep_agg = ", ";
+
+ /* XXX: need some generalization
+ *
+ * Specifically, Using func names is not robust. We can use oids instead
+ * of names, but it would be nice to add some information to pg_aggregate
+ * and handler functions.
+ */
+
+ if (!strcmp(aggname, "count"))
+ {
+ /* resname = mv.resname - t.resname */
+ appendStringInfo(&update_aggs_old,
+ "%s = %s",
+ quote_qualified_identifier(NULL, resname),
+ get_operation_string(IVM_SUB,resname, "mv", "t", NULL, NULL));
+
+ /* resname = mv.resname + diff.resname */
+ appendStringInfo(&update_aggs_new,
+ "%s = %s",
+ quote_qualified_identifier(NULL, resname),
+ get_operation_string(IVM_ADD, resname, "mv", "diff", NULL, NULL));
+
+ appendStringInfo(&diff_aggs_buf, "%s",
+ quote_qualified_identifier("diff", resname)
+ );
+ }
+ else if (!strcmp(aggname, "sum"))
+ {
+ char *count_col = IVM_colname("count", resname);
+
+ /* sum = mv.sum - t.sum */
+ appendStringInfo(&update_aggs_old,
+ "%s = %s, ",
+ quote_qualified_identifier(NULL, resname),
+ get_operation_string(IVM_SUB, resname, "mv", "t", count_col, NULL)
+ );
+ /* count = mv.count - t.count */
+ appendStringInfo(&update_aggs_old,
+ "%s = %s",
+ quote_qualified_identifier(NULL, count_col),
+ get_operation_string(IVM_SUB, count_col, "mv", "t", NULL, NULL)
+ );
+
+ /* sum = mv.sum + diff.sum */
+ appendStringInfo(&update_aggs_new,
+ "%s = %s, ",
+ quote_qualified_identifier(NULL, resname),
+ get_operation_string(IVM_ADD, resname, "mv", "diff", count_col, NULL)
+ );
+ /* count = mv.count + diff.count */
+ appendStringInfo(&update_aggs_new,
+ "%s = %s",
+ quote_qualified_identifier(NULL, count_col),
+ get_operation_string(IVM_ADD, count_col, "mv", "diff", NULL, NULL)
+ );
+
+ appendStringInfo(&diff_aggs_buf, "%s, %s",
+ quote_qualified_identifier("diff", resname),
+ quote_qualified_identifier("diff", makeObjectName("__ivm_count",resname,"_"))
+ );
+ }
+ else if (!strcmp(aggname, "avg"))
+ {
+ char *sum_col = IVM_colname("sum", resname);
+ char *count_col = IVM_colname("count", resname);
+
+ /* avg = (mv.sum - t.sum)::aggtype / (mv.count - t.count) */
+ appendStringInfo(&update_aggs_old,
+ "%s = %s / %s, ",
+ quote_qualified_identifier(NULL, resname),
+ get_operation_string(IVM_SUB, sum_col, "mv", "t", count_col, aggtype),
+ get_operation_string(IVM_SUB, count_col, "mv", "t", NULL, NULL)
+ );
+ /* sum = mv.sum - t.sum */
+ appendStringInfo(&update_aggs_old,
+ "%s = %s, ",
+ quote_qualified_identifier(NULL, sum_col),
+ get_operation_string(IVM_SUB, sum_col, "mv", "t", count_col, NULL)
+ );
+ /* count = mv.count - t.count */
+ appendStringInfo(&update_aggs_old,
+ "%s = %s",
+ quote_qualified_identifier(NULL, count_col),
+ get_operation_string(IVM_SUB, count_col, "mv", "t", NULL, NULL)
+ );
+
+ /* avg = (mv.sum + diff.sum)::aggtype / (mv.count + diff.count) */
+ appendStringInfo(&update_aggs_new,
+ "%s = %s / %s, ",
+ quote_qualified_identifier(NULL, resname),
+ get_operation_string(IVM_ADD, sum_col, "mv", "diff", count_col, aggtype),
+ get_operation_string(IVM_ADD, count_col, "mv", "diff", NULL, NULL)
+ );
+ /* sum = mv.sum + diff.sum */
+ appendStringInfo(&update_aggs_new,
+ "%s = %s, ",
+ quote_qualified_identifier(NULL, sum_col),
+ get_operation_string(IVM_ADD, sum_col, "mv", "diff", count_col, NULL)
+ );
+ /* count = mv.count + diff.count */
+ appendStringInfo(&update_aggs_new,
+ "%s = %s",
+ quote_qualified_identifier(NULL, count_col),
+ get_operation_string(IVM_ADD, count_col, "mv", "diff", NULL, NULL)
+ );
+
+ appendStringInfo(&diff_aggs_buf, "%s, %s, %s",
+ quote_qualified_identifier("diff", resname),
+ quote_qualified_identifier("diff", makeObjectName("__ivm_sum",resname,"_")),
+ quote_qualified_identifier("diff", makeObjectName("__ivm_count",resname,"_"))
+ );
+ }
+ else if (!strcmp(aggname, "min") || !strcmp(aggname, "max"))
+ {
+ bool is_min = !strcmp(aggname, "min");
+ char *count_col = IVM_colname("count", resname);
+
+ if (!has_min_or_max)
+ {
+ has_min_or_max = true;
+ appendStringInfo(&returning_buf, "RETURNING mv.ctid, (");
+ }
+ else
+ {
+ appendStringInfo(&returning_buf, " OR ");
+ appendStringInfo(&min_or_max_buf, ",");
+ }
+
+ appendStringInfo(&returning_buf, "%s %s %s",
+ quote_qualified_identifier("mv", resname),
+ is_min ? ">=" : "<=",
+ quote_qualified_identifier("t", resname)
+ );
+ appendStringInfo(&min_or_max_buf, "%s", quote_qualified_identifier(NULL, resname));
+
+ /* Even if the new values is not NULL, this might be recomputated afterwords. */
+ appendStringInfo(&update_aggs_old,
+ "%s = CASE WHEN %s THEN NULL ELSE %s END, ",
+ quote_qualified_identifier(NULL, resname),
+ get_null_condition_string(IVM_SUB, "mv", "t", count_col),
+ quote_qualified_identifier("mv", resname)
+ );
+ /* count = mv.count - t.count */
+ appendStringInfo(&update_aggs_old,
+ "%s = %s",
+ quote_qualified_identifier(NULL, count_col),
+ get_operation_string(IVM_SUB, count_col, "mv", "t", NULL, NULL)
+ );
+
+ /*
+ * min = least(mv.min, diff.min)
+ * max = greatest(mv.max, diff.max)
+ */
+ appendStringInfo(&update_aggs_new,
+ "%s = CASE WHEN %s THEN NULL ELSE %s(%s,%s) END, ",
+ quote_qualified_identifier(NULL, resname),
+ get_null_condition_string(IVM_ADD, "mv", "diff", count_col),
+
+ is_min ? "least" : "greatest",
+ quote_qualified_identifier("mv", resname),
+ quote_qualified_identifier("diff", resname)
+ );
+
+ /* count = mv.count + diff.count */
+ appendStringInfo(&update_aggs_new,
+ "%s = %s",
+ quote_qualified_identifier(NULL, count_col),
+ get_operation_string(IVM_ADD, count_col, "mv", "diff", NULL, NULL)
+ );
+
+ appendStringInfo(&diff_aggs_buf, "%s, %s",
+ quote_qualified_identifier("diff", resname),
+ quote_qualified_identifier("diff", makeObjectName("__ivm_count",resname,"_"))
+ );
+
+ num_min_or_max++;
+ }
+ else
+ elog(ERROR, "unsupported aggregate function: %s", aggname);
+
+ }
+ }
+ if (has_min_or_max)
+ appendStringInfo(&returning_buf, ") AS recalc");
+
+ if (query->hasAggs)
+ {
+ initStringInfo(&mv_gkeys_buf);
+ initStringInfo(&diff_gkeys_buf);
+ initStringInfo(&updt_gkeys_buf);
+
+ if (with_group)
+ {
+ sep_agg= "";
+ foreach (lc, query->groupClause)
+ {
+ SortGroupClause *sgcl = (SortGroupClause *) lfirst(lc);
+ TargetEntry *tle = get_sortgroupclause_tle(sgcl, query->targetList);
+
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno-1);
+ char *resname = NameStr(attr->attname);
+
+ appendStringInfo(&mv_gkeys_buf, "%s", sep_agg);
+ appendStringInfo(&diff_gkeys_buf, "%s", sep_agg);
+ appendStringInfo(&updt_gkeys_buf, "%s", sep_agg);
+
+ sep_agg = ", ";
+
+ appendStringInfo(&mv_gkeys_buf, "%s", quote_qualified_identifier("mv", resname));
+ appendStringInfo(&diff_gkeys_buf, "%s", quote_qualified_identifier("diff", resname));
+ appendStringInfo(&updt_gkeys_buf, "%s", quote_qualified_identifier("updt", resname));
+
+ num_group_keys++;
+ }
+
+ if (has_min_or_max)
+ {
+ appendStringInfo(&returning_buf, ", %s", mv_gkeys_buf.data);
+ appendStringInfo(&result_buf, "SELECT ctid AS tid, %s FROM updt WHERE recalc", updt_gkeys_buf.data);
+ }
+ }
+ else
+ {
+ appendStringInfo(&mv_gkeys_buf, "1");
+ appendStringInfo(&diff_gkeys_buf, "1");
+ appendStringInfo(&updt_gkeys_buf, "1");
+ }
+ }
+
+
+ /* Open SPI context. */
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+
+ /* Analyze the temp table with the new contents. */
+ if (tempname_new)
+ {
+ appendStringInfo(&querybuf, "ANALYZE %s", tempname_new);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ if (tempname_old)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf, "ANALYZE %s", tempname_old);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+
+ OpenMatViewIncrementalMaintenance();
+
+ if (query->hasAggs)
+ {
+ if (tempname_old)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH t AS ("
+ " SELECT diff.__ivm_count__, "
+ " %s,"
+ " (diff.__ivm_count__ = mv.__ivm_count__) AS for_dlt, "
+ " mv.ctid"
+ " FROM %s AS mv, %s AS diff WHERE (%s) = (%s)"
+ "), updt AS ("
+ " UPDATE %s AS mv SET __ivm_count__ = mv.__ivm_count__ - t.__ivm_count__,"
+ " %s"
+ " FROM t WHERE mv.ctid = t.ctid AND NOT for_dlt"
+ " %s"
+ "), dlt AS ("
+ " DELETE FROM %s AS mv USING t WHERE mv.ctid = t.ctid AND for_dlt "
+ ") %s",
+ diff_aggs_buf.data,
+ matviewname, tempname_old, mv_gkeys_buf.data, diff_gkeys_buf.data,
+ matviewname, update_aggs_old.data,
+ returning_buf.data,
+ matviewname,
+ (has_min_or_max && with_group) ? result_buf.data : "SELECT 1"
+ );
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_SELECT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+
+ if (has_min_or_max && SPI_processed > 0)
+ {
+ SPITupleTable *tuptable_recalc = SPI_tuptable;
+ TupleDesc tupdesc_recalc = tuptable_recalc->tupdesc;
+ int64 processed = SPI_processed;
+ uint64 i;
+ Oid *keyTypes = NULL, *minmaxTypes = NULL;
+ char *keyNulls = NULL, *minmaxNulls = NULL;
+ Datum *keyVals = NULL, *minmaxVals = NULL;
+
+ if (with_group)
+ {
+ keyTypes = palloc(sizeof(Oid) * num_group_keys);
+ keyNulls = palloc(sizeof(char) * num_group_keys);
+ keyVals = palloc(sizeof(Datum) * num_group_keys);
+ Assert(tupdesc_recalc->natts == num_group_keys + 1);
+
+ for (i = 0; i < num_group_keys; i++)
+ keyTypes[i] = TupleDescAttr(tupdesc_recalc, i+1)->atttypid;
+ }
+
+ minmaxTypes = palloc(sizeof(Oid) * (num_min_or_max + 1));
+ minmaxNulls = palloc(sizeof(char) * (num_min_or_max + 1));
+ minmaxVals = palloc(sizeof(Datum) * (num_min_or_max + 1));
+
+ for (i=0; i< processed; i++)
+ {
+ int j;
+ bool isnull;
+ SPIPlanPtr plan;
+ SPITupleTable *tuptable_minmax;
+ TupleDesc tupdesc_minmax;
+
+ if (with_group)
+ {
+ for (j = 0; j < num_group_keys; j++)
+ {
+ keyVals[j] = SPI_getbinval(tuptable_recalc->vals[i], tupdesc_recalc, j+2, &isnull);
+ if (isnull)
+ keyNulls[j] = 'n';
+ else
+ keyNulls[j] = ' ';
+ }
+ }
+
+ plan = get_plan_for_recalc_min_max(matviewOid, min_or_max_buf.data,
+ mv_gkeys_buf.data, num_group_keys, keyTypes, with_group);
+
+ if (SPI_execute_plan(plan, keyVals, keyNulls, false, 0) != SPI_OK_SELECT)
+ elog(ERROR, "SPI_execcute_plan1");
+ if (SPI_processed != 1)
+ elog(ERROR, "SPI_execcute_plan returned zere or more than one rows");
+
+ tuptable_minmax = SPI_tuptable;
+ tupdesc_minmax = tuptable_minmax->tupdesc;
+
+ Assert(tupdesc_minmax->natts == num_min_or_max);
+
+ for (j = 0; j < tupdesc_minmax->natts; j++)
+ {
+ if (i == 0)
+ minmaxTypes[j] = TupleDescAttr(tupdesc_minmax, j)->atttypid;
+
+ minmaxVals[j] = SPI_getbinval(tuptable_minmax->vals[0], tupdesc_minmax, j+1, &isnull);
+ if (isnull)
+ minmaxNulls[j] = 'n';
+ else
+ minmaxNulls[j] = ' ';
+ }
+ minmaxTypes[j] = TIDOID;
+ minmaxVals[j] = SPI_getbinval(tuptable_recalc->vals[i], tupdesc_recalc, 1, &isnull);
+ minmaxNulls[j] = ' ';
+
+ plan = get_plan_for_set_min_max(matviewOid, matviewname, min_or_max_buf.data,
+ num_min_or_max, minmaxTypes, with_group);
+
+ if (SPI_execute_plan(plan, minmaxVals, minmaxNulls, false, 0) != SPI_OK_UPDATE)
+ elog(ERROR, "SPI_execcute_plan2");
+
+ }
+ }
+
+ }
+ if (tempname_new)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH updt AS ("
+ " UPDATE %s AS mv SET __ivm_count__ = mv.__ivm_count__ + diff.__ivm_count__"
+ ", %s "
+ " FROM %s AS diff WHERE (%s) = (%s)"
+ " RETURNING %s"
+ ") INSERT INTO %s (SELECT * FROM %s AS diff WHERE (%s) NOT IN (SELECT %s FROM updt));",
+ matviewname, update_aggs_new.data, tempname_new,
+ mv_gkeys_buf.data, diff_gkeys_buf.data, diff_gkeys_buf.data,
+ matviewname, tempname_new, diff_gkeys_buf.data, updt_gkeys_buf.data);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ }
+ else
+ {
+ if (tempname_old)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH t AS ("
+ " SELECT diff.__ivm_count__, (diff.__ivm_count__ = mv.__ivm_count__) AS for_dlt, mv.ctid"
+ " FROM %s AS mv, %s AS diff WHERE (%s) = (%s)"
+ "), updt AS ("
+ " UPDATE %s AS mv SET __ivm_count__ = mv.__ivm_count__ - t.__ivm_count__"
+ " FROM t WHERE mv.ctid = t.ctid AND NOT for_dlt"
+ ") DELETE FROM %s AS mv USING t WHERE mv.ctid = t.ctid AND for_dlt;",
+ matviewname, tempname_old, mvatts_buf.data, diffatts_buf.data, matviewname, matviewname);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_DELETE)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ if (tempname_new)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH updt AS ("
+ " UPDATE %s AS mv SET __ivm_count__ = mv.__ivm_count__ + diff.__ivm_count__"
+ " FROM %s AS diff WHERE (%s) = (%s)"
+ " RETURNING %s"
+ ") INSERT INTO %s (SELECT * FROM %s AS diff WHERE (%s) NOT IN (SELECT * FROM updt));",
+ matviewname, tempname_new, mvatts_buf.data, diffatts_buf.data, diffatts_buf.data, matviewname, tempname_new, diffatts_buf.data);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ }
+
+ /* We're done maintaining the materialized view. */
+ CloseMatViewIncrementalMaintenance();
+
+ if (OidIsValid(tempOid_new))
+ table_close(tempRel_new, NoLock);
+ if (OidIsValid(tempOid_old))
+ table_close(tempRel_old, NoLock);
+
+ table_close(matviewRel, NoLock);
+
+
+ /* Close SPI context. */
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+}
+
+
+static void
+mv_InitHashTables(void)
+{
+ HASHCTL ctl;
+
+ memset(&ctl, 0, sizeof(ctl));
+ ctl.keysize = sizeof(MV_QueryKey);
+ ctl.entrysize = sizeof(MV_QueryHashEntry);
+ mv_query_cache = hash_create("MV query cache",
+ MV_INIT_QUERYHASHSIZE,
+ &ctl, HASH_ELEM | HASH_BLOBS);
+
+ memset(&ctl, 0, sizeof(ctl));
+ ctl.keysize = sizeof(Oid);
+ ctl.entrysize = sizeof(MV_TriggerHashEntry);
+ mv_trigger_info = hash_create("MV trigger info",
+ MV_INIT_QUERYHASHSIZE,
+ &ctl, HASH_ELEM | HASH_BLOBS);
+}
+
+static SPIPlanPtr
+mv_FetchPreparedPlan(MV_QueryKey *key)
+{
+ MV_QueryHashEntry *entry;
+ SPIPlanPtr plan;
+
+ /*
+ * On the first call initialize the hashtable
+ */
+ if (!mv_query_cache)
+ mv_InitHashTables();
+
+ /*
+ * Lookup for the key
+ */
+ entry = (MV_QueryHashEntry *) hash_search(mv_query_cache,
+ (void *) key,
+ HASH_FIND, NULL);
+ if (entry == NULL)
+ return NULL;
+
+ /*
+ * Check whether the plan is still valid. If it isn't, we don't want to
+ * simply rely on plancache.c to regenerate it; rather we should start
+ * from scratch and rebuild the query text too. This is to cover cases
+ * such as table/column renames. We depend on the plancache machinery to
+ * detect possible invalidations, though.
+ *
+ * CAUTION: this check is only trustworthy if the caller has already
+ * locked both materialized views and base tables.
+ */
+ plan = entry->plan;
+ if (plan && SPI_plan_is_valid(plan))
+ return plan;
+
+ /*
+ * Otherwise we might as well flush the cached plan now, to free a little
+ * memory space before we make a new one.
+ */
+ entry->plan = NULL;
+ if (plan)
+ SPI_freeplan(plan);
+
+ return NULL;
+}
+
+/*
+ * mv_HashPreparedPlan -
+ *
+ * Add another plan to our private SPI query plan hashtable.
+ */
+static void
+mv_HashPreparedPlan(MV_QueryKey *key, SPIPlanPtr plan)
+{
+ MV_QueryHashEntry *entry;
+ bool found;
+
+ /*
+ * On the first call initialize the hashtable
+ */
+ if (!mv_query_cache)
+ mv_InitHashTables();
+
+ /*
+ * Add the new plan. We might be overwriting an entry previously found
+ * invalid by mv_FetchPreparedPlan.
+ */
+ entry = (MV_QueryHashEntry *) hash_search(mv_query_cache,
+ (void *) key,
+ HASH_ENTER, &found);
+ Assert(!found || entry->plan == NULL);
+ entry->plan = plan;
+}
+
+/* ----------
+ * mv_BuildQueryKey -
+ *
+ * Construct a hashtable key for a prepared SPI plan for IVM.
+ *
+ * key: output argument, *key is filled in based on the other arguments
+ * matview_id: OID of materialized view
+ * query_type: an internal number identifying the query type
+ * (see MV_PLAN_XXX constants at head of file)
+ * ----------
+ */
+static void
+mv_BuildQueryKey(MV_QueryKey *key, Oid matview_id, int32 query_type)
+{
+ /*
+ * We assume struct MV_QueryKey contains no padding bytes, else we'd need
+ * to use memset to clear them.
+ */
+ key->matview_id = matview_id;
+ key->query_type = query_type;
+}
+
+static SPIPlanPtr
+get_plan_for_recalc_min_max(Oid matviewOid, const char *min_max_list,
+ const char *group_keys, int nkeys, Oid *keyTypes, bool with_group)
+{
+ MV_QueryKey key;
+ SPIPlanPtr plan;
+
+ /* Fetch or prepare a saved plan for the real check */
+ mv_BuildQueryKey(&key, matviewOid, MV_PLAN_RECALC_MINMAX);
+
+ if ((plan = mv_FetchPreparedPlan(&key)) == NULL)
+ {
+ char *viewdef;
+ StringInfoData str;
+ int i;
+
+
+ /* get view definition of matview */
+ viewdef = text_to_cstring((text *) DatumGetPointer(
+ DirectFunctionCall1(pg_get_viewdef, ObjectIdGetDatum(matviewOid))));
+ /* get rid of tailling semi-collon */
+ viewdef[strlen(viewdef)-1] = '\0';
+
+ initStringInfo(&str);
+ appendStringInfo(&str, "SELECT %s FROM (%s) mv", min_max_list, viewdef);
+
+ if (with_group)
+ {
+ appendStringInfo(&str, " WHERE (%s) = (", group_keys);
+
+ for (i = 1; i <= nkeys; i++)
+ appendStringInfo(&str, "%s$%d", (i==1 ? "" : ", "),i );
+
+ appendStringInfo(&str, ")");
+ }
+
+ plan = SPI_prepare(str.data, (with_group ? nkeys : 0), (with_group ? keyTypes : NULL));
+ if (plan == NULL)
+ elog(ERROR, "SPI_prepare returned %s for %s", SPI_result_code_string(SPI_result), str.data);
+
+ SPI_keepplan(plan);
+ mv_HashPreparedPlan(&key, plan);
+ }
+
+ return plan;
+}
+
+static SPIPlanPtr
+get_plan_for_set_min_max(Oid matviewOid, char *matviewname, const char *min_max_list,
+ int num_min_max, Oid *valTypes, bool with_group)
+{
+ MV_QueryKey key;
+ SPIPlanPtr plan;
+
+ /* Fetch or prepare a saved plan for the real check */
+ mv_BuildQueryKey(&key, matviewOid, MV_PLAN_SET_MINMAX);
+
+ if ((plan = mv_FetchPreparedPlan(&key)) == NULL)
+ {
+ StringInfoData str;
+ int i;
+
+ initStringInfo(&str);
+ appendStringInfo(&str, "UPDATE %s AS mv SET (%s) = (",
+ matviewname, min_max_list);
+
+ for (i = 1; i <= num_min_max; i++)
+ appendStringInfo(&str, "%s$%d", (i==1 ? "" : ", "), i);
+
+ appendStringInfo(&str, ")");
+
+ if (with_group)
+ appendStringInfo(&str, " WHERE ctid = $%d", num_min_max + 1);
+
+ plan = SPI_prepare(str.data, num_min_max + (with_group ? 1 : 0), valTypes);
+ if (plan == NULL)
+ elog(ERROR, "SPI_prepare returned %s for %s", SPI_result_code_string(SPI_result), str.data);
+
+ SPI_keepplan(plan);
+ mv_HashPreparedPlan(&key, plan);
+ }
+
+ return plan;
+}
+
+
+/*
+ * get_matview_query - get the Query from a matview's _RETURN rule.
+ */
+static Query *
+get_matview_query(Relation matviewRel)
+{
+ RewriteRule *rule;
+ List * actions;
+
+ /*
+ * Check that everything is correct for a refresh. Problems at this point
+ * are internal errors, so elog is sufficient.
+ */
+ if (matviewRel->rd_rel->relhasrules == false ||
+ matviewRel->rd_rules->numLocks < 1)
+ elog(ERROR,
+ "materialized view \"%s\" is missing rewrite information",
+ RelationGetRelationName(matviewRel));
+
+ if (matviewRel->rd_rules->numLocks > 1)
+ elog(ERROR,
+ "materialized view \"%s\" has too many rules",
+ RelationGetRelationName(matviewRel));
+
+ rule = matviewRel->rd_rules->rules[0];
+ if (rule->event != CMD_SELECT || !(rule->isInstead))
+ elog(ERROR,
+ "the rule for materialized view \"%s\" is not a SELECT INSTEAD OF rule",
+ RelationGetRelationName(matviewRel));
+
+ actions = rule->actions;
+ if (list_length(actions) != 1)
+ elog(ERROR,
+ "the rule for materialized view \"%s\" is not a single action",
+ RelationGetRelationName(matviewRel));
+
+ /*
+ * The stored query was rewritten at the time of the MV definition, but
+ * has not been scribbled on by the planner.
+ */
+ return linitial_node(Query, actions);
+}
+
+static void
+truncate_view_delta(Oid delta_oid)
+{
+ Relation rel;
+
+ if (!OidIsValid(delta_oid))
+ return;
+
+ rel = table_open(delta_oid, NoLock);
+ ExecuteTruncateGuts(list_make1(rel), list_make1_oid(delta_oid), NIL,
+ DROP_RESTRICT, false);
+ table_close(rel, NoLock);
+}
+
+void
+AtAbort_IVM()
+{
+ HASH_SEQ_STATUS seq;
+ MV_TriggerHashEntry *entry;
+
+ if (mv_trigger_info)
+ {
+ hash_seq_init(&seq, mv_trigger_info);
+ while ((entry = hash_seq_search(&seq)) != NULL)
+ clean_up_IVM_hash_entry(entry);
+ }
+}
+
+static void
+clean_up_IVM_hash_entry(MV_TriggerHashEntry *entry)
+{
+ bool found;
+ ListCell *lc;
+
+ foreach(lc, entry->tables)
+ {
+ MV_TriggerTable *table = (MV_TriggerTable *) lfirst(lc);
+
+ list_free(table->old_tuplestores);
+ list_free(table->new_tuplestores);
+ }
+ list_free(entry->tables);
+
+ hash_search(mv_trigger_info, (void *) &entry->matview_id, HASH_REMOVE, &found);
+}
+
+static void
+clean_up_IVM_temptable(Oid tempOid_old, Oid tempOid_new)
+{
+ StringInfoData querybuf;
+ Relation tempRel_old, tempRel_new;
+ char *tempname_old = NULL, *tempname_new = NULL;
+
+ if (OidIsValid(tempOid_new))
+ {
+ tempRel_new = table_open(tempOid_new, NoLock);
+ tempname_new = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(tempRel_new)),
+ RelationGetRelationName(tempRel_new));
+ table_close(tempRel_new, NoLock);
+ }
+ if (OidIsValid(tempOid_old))
+ {
+ tempRel_old = table_open(tempOid_old, NoLock);
+ tempname_old = quote_qualified_identifier(get_namespace_name(RelationGetNamespace(tempRel_old)),
+ RelationGetRelationName(tempRel_old));
+ table_close(tempRel_old, NoLock);
+ }
+
+ initStringInfo(&querybuf);
+
+ /* Open SPI context. */
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+
+ /* Clean up temp tables. */
+ if (OidIsValid(tempOid_old))
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf, "DROP TABLE %s", tempname_old);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ if (OidIsValid(tempOid_new))
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf, "DROP TABLE %s", tempname_new);
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+
+ /* Close SPI context. */
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+}
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index cdb1105b4a..096ab1b67c 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3803,6 +3803,7 @@ typedef struct AfterTriggersData
AfterTriggersQueryData *query_stack; /* array of structs shown below */
int query_depth; /* current index in above array */
int maxquerydepth; /* allocated len of above array */
+ List *prolonged_tuplestores;
/* per-subtransaction-level data: */
AfterTriggersTransData *trans_stack; /* array of structs shown below */
@@ -3833,6 +3834,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged;
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
Tuplestorestate *old_tuplestore; /* "old" transition table, if any */
Tuplestorestate *new_tuplestore; /* "new" transition table, if any */
@@ -4586,6 +4588,35 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
return all_fired;
}
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
/*
* GetAfterTriggersTableData
@@ -4755,6 +4786,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -4915,11 +4947,29 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ {
+ if (table->prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
+ }
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ {
+ if (table->prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
+ }
}
/*
@@ -4929,6 +4979,17 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
}
@@ -6024,6 +6085,9 @@ AfterTriggerSaveEvent(EState *estate, ResultRelInfo *relinfo,
else
new_shared.ats_table = NULL;
+ if (new_shared.ats_table != NULL && trigger->tgfoid == F_IVM_IMMEDIATE_MAINTENANCE)
+ new_shared.ats_table->prolonged = true;
+
afterTriggerAddEvent(&afterTriggers.query_stack[afterTriggers.query_depth].events,
&new_event, &new_shared);
}
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 3432bb921d..a872b59f13 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2365,6 +2365,7 @@ _copyRangeTblEntry(const RangeTblEntry *from)
COPY_SCALAR_FIELD(relkind);
COPY_SCALAR_FIELD(rellockmode);
COPY_NODE_FIELD(tablesample);
+ COPY_SCALAR_FIELD(relisivm);
COPY_NODE_FIELD(subquery);
COPY_SCALAR_FIELD(security_barrier);
COPY_SCALAR_FIELD(jointype);
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 18cb014373..503915b9ef 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2651,6 +2651,7 @@ _equalRangeTblEntry(const RangeTblEntry *a, const RangeTblEntry *b)
COMPARE_SCALAR_FIELD(relkind);
COMPARE_SCALAR_FIELD(rellockmode);
COMPARE_NODE_FIELD(tablesample);
+ COMPARE_SCALAR_FIELD(relisivm);
COMPARE_NODE_FIELD(subquery);
COMPARE_SCALAR_FIELD(security_barrier);
COMPARE_SCALAR_FIELD(jointype);
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index b0dcd02ff6..52e898d615 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -3058,6 +3058,7 @@ _outRangeTblEntry(StringInfo str, const RangeTblEntry *node)
WRITE_CHAR_FIELD(relkind);
WRITE_INT_FIELD(rellockmode);
WRITE_NODE_FIELD(tablesample);
+ WRITE_BOOL_FIELD(relisivm);
break;
case RTE_SUBQUERY:
WRITE_NODE_FIELD(subquery);
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 764e3bb90c..3301ccfbc8 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1366,6 +1366,7 @@ _readRangeTblEntry(void)
READ_CHAR_FIELD(relkind);
READ_INT_FIELD(rellockmode);
READ_NODE_FIELD(tablesample);
+ READ_BOOL_FIELD(relisivm);
break;
case RTE_SUBQUERY:
READ_NODE_FIELD(subquery);
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 3f67aaf30e..2ccdda580a 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -418,6 +418,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -645,7 +646,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -4083,30 +4084,32 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->relkind = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->relkind = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -4123,9 +4126,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -15157,6 +15165,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDEX
| INDEXES
| INHERIT
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
index 4dd81507a7..a6c8c3fd4d 100644
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -37,6 +37,7 @@
#include "utils/syscache.h"
#include "utils/varlena.h"
+#include "commands/matview.h"
#define MAX_FUZZY_DISTANCE 3
@@ -56,9 +57,10 @@ static void expandTupleDesc(TupleDesc tupdesc, Alias *eref,
int count, int offset,
int rtindex, int sublevels_up,
int location, bool include_dropped,
- List **colnames, List **colvars);
+ List **colnames, List **colvars, bool is_ivm);
static int specialAttNum(const char *attname);
static bool isQueryUsingTempRelation_walker(Node *node, void *context);
+static bool isIvmColumn(const char *s);
/*
@@ -1241,6 +1243,7 @@ addRangeTableEntry(ParseState *pstate,
rte->relid = RelationGetRelid(rel);
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -1320,6 +1323,7 @@ addRangeTableEntryForRelation(ParseState *pstate,
rte->relid = RelationGetRelid(rel);
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -2318,7 +2322,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
expandTupleDesc(tupdesc, rte->eref,
rtfunc->funccolcount, atts_done,
rtindex, sublevels_up, location,
- include_dropped, colnames, colvars);
+ include_dropped, colnames, colvars, false);
}
else if (functypclass == TYPEFUNC_SCALAR)
{
@@ -2570,10 +2574,16 @@ expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up,
expandTupleDesc(rel->rd_att, eref, rel->rd_att->natts, 0,
rtindex, sublevels_up,
location, include_dropped,
- colnames, colvars);
+ colnames, colvars, RelationIsIVM(rel));
relation_close(rel, AccessShareLock);
}
+static bool
+isIvmColumn(const char *s)
+{
+ return (strncmp(s, "__ivm_", 6) == 0);
+}
+
/*
* expandTupleDesc -- expandRTE subroutine
*
@@ -2587,7 +2597,7 @@ static void
expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
int rtindex, int sublevels_up,
int location, bool include_dropped,
- List **colnames, List **colvars)
+ List **colnames, List **colvars, bool is_ivm)
{
ListCell *aliascell;
int varattno;
@@ -2600,6 +2610,9 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
{
Form_pg_attribute attr = TupleDescAttr(tupdesc, varattno);
+ if (is_ivm && isIvmColumn(NameStr(attr->attname)) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
if (attr->attisdropped)
{
if (include_dropped)
diff --git a/src/backend/rewrite/rewriteDefine.c b/src/backend/rewrite/rewriteDefine.c
index 7df2b6154c..5385a038b1 100644
--- a/src/backend/rewrite/rewriteDefine.c
+++ b/src/backend/rewrite/rewriteDefine.c
@@ -765,7 +765,8 @@ checkRuleResultList(List *targetList, TupleDesc resultDesc, bool isSelect,
attr->atttypmod))));
}
- if (i != resultDesc->natts)
+ /* No check for materialized views since this could have special columns for IVM */
+ if ((!isSelect || requireColumnNameMatch) && i != resultDesc->natts)
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
isSelect ?
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index 3e38007643..5b71f0c665 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -41,6 +41,8 @@
#include "utils/lsyscache.h"
#include "utils/rel.h"
+#include "parser/parser.h"
+#include "commands/matview.h"
/* We use a list of these to detect recursion in RewriteQuery */
typedef struct rewrite_event
@@ -1597,6 +1599,50 @@ ApplyRetrieveRule(Query *parsetree,
if (rule->qual != NULL)
elog(ERROR, "cannot handle qualified ON SELECT rule");
+ if (RelationIsIVM(relation))
+ {
+ rule_action = copyObject(linitial(rule->actions));
+
+ if (!rule_action->distinctClause && !rule_action->groupClause && !rule_action->hasAggs)
+ {
+ StringInfoData str;
+ RawStmt *raw;
+ Query *sub;
+
+ if (rule_action->hasDistinctOn)
+ elog(ERROR, "DISTINCT ON is not supported in IVM");
+
+ initStringInfo(&str);
+ appendStringInfo(&str, "SELECT mv.*, __ivm_count__ FROM %s mv, generate_series(1, mv.__ivm_count__)",
+ quote_qualified_identifier(get_namespace_name(RelationGetNamespace(relation)),
+ RelationGetRelationName(relation)));
+
+ raw = (RawStmt*)linitial(raw_parser(str.data));
+ sub = transformStmt(make_parsestate(NULL),raw->stmt);
+
+ rte = rt_fetch(rt_index, parsetree->rtable);
+
+ rte->rtekind = RTE_SUBQUERY;
+ rte->subquery = sub;
+ rte->security_barrier = RelationIsSecurityView(relation);
+ /* Clear fields that should not be set in a subquery RTE */
+ rte->relid = InvalidOid;
+ rte->relkind = 0;
+ rte->rellockmode = 0;
+ rte->tablesample = NULL;
+ rte->inh = false; /* must not be set for a subquery */
+
+ rte->requiredPerms = 0; /* no permission check on subquery itself */
+ rte->checkAsUser = InvalidOid;
+ rte->selectedCols = NULL;
+ rte->insertedCols = NULL;
+ rte->updatedCols = NULL;
+ rte->extraUpdatedCols = NULL;
+ }
+
+ return parsetree;
+ }
+
if (rt_index == parsetree->resultRelation)
{
/*
@@ -1906,7 +1952,8 @@ fireRIRrules(Query *parsetree, List *activeRIRs)
* In that case this test would need to be postponed till after we've
* opened the rel, so that we could check its state.
*/
- if (rte->relkind == RELKIND_MATVIEW)
+ if (rte->relkind == RELKIND_MATVIEW &&
+ (!rte->relisivm || MatViewIncrementalMaintenanceIsEnabled() || parsetree->commandType != CMD_SELECT))
continue;
/*
diff --git a/src/backend/utils/cache/lsyscache.c b/src/backend/utils/cache/lsyscache.c
index 27602fa49c..694e01b70f 100644
--- a/src/backend/utils/cache/lsyscache.c
+++ b/src/backend/utils/cache/lsyscache.c
@@ -1844,6 +1844,30 @@ get_rel_relispartition(Oid relid)
return false;
}
+/*
+ * get_rel_relisivm
+ *
+ * Returns the relisivm flag associated with a given relation.
+ */
+bool
+get_rel_relisivm(Oid relid)
+{
+ HeapTuple tp;
+
+ tp = SearchSysCache1(RELOID, ObjectIdGetDatum(relid));
+ if (HeapTupleIsValid(tp))
+ {
+ Form_pg_class reltup = (Form_pg_class) GETSTRUCT(tp);
+ bool result;
+
+ result = reltup->relisivm;
+ ReleaseSysCache(tp);
+ return result;
+ }
+ else
+ return false;
+}
+
/*
* get_rel_tablespace
*
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 585dcee5db..db04cc98b9 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -1860,6 +1860,8 @@ formrdesc(const char *relationName, Oid relationReltype,
/* ... and they're always populated, too */
relation->rd_rel->relispopulated = true;
+ /* ... and they're always no ivm, too */
+ relation->rd_rel->relisivm = false;
relation->rd_rel->relreplident = REPLICA_IDENTITY_NOTHING;
relation->rd_rel->relpages = 0;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index d7c0fc0c1e..a767b50f26 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -1492,6 +1492,7 @@ describeOneTableDetails(const char *schemaname,
char relpersistence;
char relreplident;
char *relam;
+ bool isivm;
} tableinfo;
bool show_column_details = false;
@@ -1512,6 +1513,7 @@ describeOneTableDetails(const char *schemaname,
"false AS relhasoids, c.relispartition, %s, c.reltablespace, "
"CASE WHEN c.reloftype = 0 THEN '' ELSE c.reloftype::pg_catalog.regtype::pg_catalog.text END, "
"c.relpersistence, c.relreplident, am.amname\n"
+ ",c.relisivm\n"
"FROM pg_catalog.pg_class c\n "
"LEFT JOIN pg_catalog.pg_class tc ON (c.reltoastrelid = tc.oid)\n"
"LEFT JOIN pg_catalog.pg_am am ON (c.relam = am.oid)\n"
@@ -1688,6 +1690,10 @@ describeOneTableDetails(const char *schemaname,
(char *) NULL : pg_strdup(PQgetvalue(res, 0, 14));
else
tableinfo.relam = NULL;
+ if (pset.sversion >= 130000)
+ tableinfo.isivm = strcmp(PQgetvalue(res, 0, 15), "t") == 0;
+ else
+ tableinfo.isivm = false;
PQclear(res);
res = NULL;
@@ -3292,6 +3298,12 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf, _("Access method: %s"), tableinfo.relam);
printTableAddFooter(&cont, buf.data);
}
+
+ /* Incremental view maintance info */
+ if (verbose && tableinfo.relkind == RELKIND_MATVIEW && tableinfo.isivm)
+ {
+ printTableAddFooter(&cont, _("Incremental view maintenance: yes"));
+ }
}
/* reloptions, if verbose */
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index cb12668276..474b0d482a 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1001,6 +1001,7 @@ static const pgsql_thing_t words_after_create[] = {
{"FOREIGN TABLE", NULL, NULL, NULL},
{"FUNCTION", NULL, NULL, Query_for_list_of_functions},
{"GROUP", Query_for_list_of_roles},
+ {"INCREMENTAL MATERIALIZED VIEW", NULL, NULL, &Query_for_list_of_matviews},
{"INDEX", NULL, NULL, &Query_for_list_of_indexes},
{"LANGUAGE", Query_for_list_of_languages},
{"LARGE OBJECT", NULL, NULL, NULL, THING_NO_CREATE | THING_NO_DROP},
@@ -2499,7 +2500,7 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("SEQUENCE", "TABLE", "VIEW");
/* Complete "CREATE UNLOGGED" with TABLE or MATVIEW */
else if (TailMatches("CREATE", "UNLOGGED"))
- COMPLETE_WITH("TABLE", "MATERIALIZED VIEW");
+ COMPLETE_WITH("TABLE", "MATERIALIZED VIEW", "INCREMENTAL MATERIALIZED VIEW");
/* Complete PARTITION BY with RANGE ( or LIST ( or ... */
else if (TailMatches("PARTITION", "BY"))
COMPLETE_WITH("RANGE (", "LIST (", "HASH (");
@@ -2749,13 +2750,16 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("SELECT");
/* CREATE MATERIALIZED VIEW */
- else if (Matches("CREATE", "MATERIALIZED"))
+ else if (Matches("CREATE", "MATERIALIZED") ||
+ Matches("CREATE", "INCREMENTAL", "MATERIALIZED"))
COMPLETE_WITH("VIEW");
- /* Complete CREATE MATERIALIZED VIEW <name> with AS */
- else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny))
+ /* Complete CREATE MATERIALIZED VIEW <name> with AS */
+ else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny) ||
+ Matches("CREATE", "INCREMENTAL", "MATERIALIZED", "VIEW", MatchAny))
COMPLETE_WITH("AS");
/* Complete "CREATE MATERIALIZED VIEW <sth> AS with "SELECT" */
- else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny, "AS"))
+ else if (Matches("CREATE", "MATERIALIZED", "VIEW", MatchAny, "AS") ||
+ Matches("CREATE", "INCREMENTAL", "MATERIALIZED", "VIEW", MatchAny, "AS"))
COMPLETE_WITH("SELECT");
/* CREATE EVENT TRIGGER */
diff --git a/src/include/catalog/pg_class.dat b/src/include/catalog/pg_class.dat
index 9bcf28676d..7deda405af 100644
--- a/src/include/catalog/pg_class.dat
+++ b/src/include/catalog/pg_class.dat
@@ -27,7 +27,7 @@
relpersistence => 'p', relkind => 'r', relnatts => '31', relchecks => '0',
relhasrules => 'f', relhastriggers => 'f', relhassubclass => 'f',
relrowsecurity => 'f', relforcerowsecurity => 'f', relispopulated => 't',
- relreplident => 'n', relispartition => 'f', relfrozenxid => '3',
+ relreplident => 'n', relispartition => 'f', relisivm => 'f', relfrozenxid => '3',
relminmxid => '1', relacl => '_null_', reloptions => '_null_',
relpartbound => '_null_' },
{ oid => '1249',
@@ -37,7 +37,7 @@
relpersistence => 'p', relkind => 'r', relnatts => '25', relchecks => '0',
relhasrules => 'f', relhastriggers => 'f', relhassubclass => 'f',
relrowsecurity => 'f', relforcerowsecurity => 'f', relispopulated => 't',
- relreplident => 'n', relispartition => 'f', relfrozenxid => '3',
+ relreplident => 'n', relispartition => 'f', relisivm => 'f', relfrozenxid => '3',
relminmxid => '1', relacl => '_null_', reloptions => '_null_',
relpartbound => '_null_' },
{ oid => '1255',
@@ -47,17 +47,17 @@
relpersistence => 'p', relkind => 'r', relnatts => '29', relchecks => '0',
relhasrules => 'f', relhastriggers => 'f', relhassubclass => 'f',
relrowsecurity => 'f', relforcerowsecurity => 'f', relispopulated => 't',
- relreplident => 'n', relispartition => 'f', relfrozenxid => '3',
+ relreplident => 'n', relispartition => 'f', relisivm => 'f', relfrozenxid => '3',
relminmxid => '1', relacl => '_null_', reloptions => '_null_',
relpartbound => '_null_' },
{ oid => '1259',
relname => 'pg_class', reltype => 'pg_class', relam => 'heap',
relfilenode => '0', relpages => '0', reltuples => '0', relallvisible => '0',
reltoastrelid => '0', relhasindex => 'f', relisshared => 'f',
- relpersistence => 'p', relkind => 'r', relnatts => '33', relchecks => '0',
+ relpersistence => 'p', relkind => 'r', relnatts => '34', relchecks => '0',
relhasrules => 'f', relhastriggers => 'f', relhassubclass => 'f',
relrowsecurity => 'f', relforcerowsecurity => 'f', relispopulated => 't',
- relreplident => 'n', relispartition => 'f', relfrozenxid => '3',
+ relreplident => 'n', relispartition => 'f', relisivm => 'f', relfrozenxid => '3',
relminmxid => '1', relacl => '_null_', reloptions => '_null_',
relpartbound => '_null_' },
diff --git a/src/include/catalog/pg_class.h b/src/include/catalog/pg_class.h
index 090b6ba907..ff535f5504 100644
--- a/src/include/catalog/pg_class.h
+++ b/src/include/catalog/pg_class.h
@@ -116,6 +116,9 @@ CATALOG(pg_class,1259,RelationRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(83,Relat
/* is relation a partition? */
bool relispartition;
+ /* is relation a matview with ivm? */
+ bool relisivm;
+
/* heap for rewrite during DDL, link to original rel */
Oid relrewrite BKI_DEFAULT(0);
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 58ea5b982b..d0ad20f89c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10731,4 +10731,13 @@
proname => 'pg_partition_root', prorettype => 'regclass',
proargtypes => 'regclass', prosrc => 'pg_partition_root' },
+# IVM
+{ oid => '786', descr => 'ivm trigger (before)',
+ proname => 'IVM_immediate_before', provolatile => 'v', prorettype => 'trigger',
+ proargtypes => '', prosrc => 'IVM_immediate_before' },
+{ oid => '787', descr => 'ivm trigger (after)',
+ proname => 'IVM_immediate_maintenance', provolatile => 'v', prorettype => 'trigger',
+ proargtypes => '', prosrc => 'IVM_immediate_maintenance' },
+
+
]
diff --git a/src/include/commands/matview.h b/src/include/commands/matview.h
index edf04bf415..edf6ddaebe 100644
--- a/src/include/commands/matview.h
+++ b/src/include/commands/matview.h
@@ -23,6 +23,8 @@
extern void SetMatViewPopulatedState(Relation relation, bool newstate);
+extern void SetMatViewIVMState(Relation relation, bool newstate);
+
extern ObjectAddress ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
ParamListInfo params, char *completionTag);
@@ -30,4 +32,8 @@ extern DestReceiver *CreateTransientRelDestReceiver(Oid oid);
extern bool MatViewIncrementalMaintenanceIsEnabled(void);
+extern Datum IVM_immediate_before(PG_FUNCTION_ARGS);
+extern Datum IVM_immediate_maintenance(PG_FUNCTION_ARGS);
+extern void AtAbort_IVM(void);
+
#endif /* MATVIEW_H */
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index a46feeedb0..3e63733d3e 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -253,6 +253,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index d93a79a554..637d9bacb9 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1002,6 +1002,7 @@ typedef struct RangeTblEntry
char relkind; /* relation kind (see pg_class.relkind) */
int rellockmode; /* lock level that query requires on the rel */
struct TableSampleClause *tablesample; /* sampling info, or NULL */
+ bool relisivm;
/*
* Fields valid for a subquery RTE (else NULL):
@@ -2059,6 +2060,7 @@ typedef struct CreateStmt
char *tablespacename; /* table space to use, or NULL */
char *accessMethod; /* table access method */
bool if_not_exists; /* just do nothing if it already exists? */
+ bool ivm; /* incremental view maintenance is used by materialized view */
} CreateStmt;
/* ----------
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 860a84de7c..6df58e2ff9 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -117,6 +117,7 @@ typedef struct IntoClause
char *tableSpaceName; /* table space to use, or NULL */
Node *viewQuery; /* materialized view's SELECT query */
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 00ace8425e..d682ee11cc 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -198,6 +198,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD)
PG_KEYWORD("inherit", INHERIT, UNRESERVED_KEYWORD)
diff --git a/src/include/utils/lsyscache.h b/src/include/utils/lsyscache.h
index c8df5bff9f..8fd919349e 100644
--- a/src/include/utils/lsyscache.h
+++ b/src/include/utils/lsyscache.h
@@ -129,6 +129,7 @@ extern Oid get_rel_namespace(Oid relid);
extern Oid get_rel_type_id(Oid relid);
extern char get_rel_relkind(Oid relid);
extern bool get_rel_relispartition(Oid relid);
+extern bool get_rel_relisivm(Oid relid);
extern Oid get_rel_tablespace(Oid relid);
extern char get_rel_persistence(Oid relid);
extern Oid get_transform_fromsql(Oid typid, Oid langid, List *trftypes);
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index a5cf804f9f..3322eb0794 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -567,6 +567,8 @@ typedef struct ViewOptions
*/
#define RelationIsPopulated(relation) ((relation)->rd_rel->relispopulated)
+#define RelationIsIVM(relation) ((relation)->rd_rel->relisivm)
+
/*
* RelationIsAccessibleInLogicalDecoding
* True if we need to log enough information to have access via
diff --git a/src/test/regress/expected/incremental_matview.out b/src/test/regress/expected/incremental_matview.out
new file mode 100644
index 0000000000..153b4401e1
--- /dev/null
+++ b/src/test/regress/expected/incremental_matview.out
@@ -0,0 +1,450 @@
+-- create a table to use as a basis for views and materialized views in various combinations
+CREATE TABLE mv_base_a (i int, j int);
+INSERT INTO mv_base_a VALUES
+ (1,10),
+ (2,20),
+ (3,30),
+ (4,40),
+ (5,50);
+CREATE TABLE mv_base_b (i int, k int);
+INSERT INTO mv_base_b VALUES
+ (1,101),
+ (2,102),
+ (3,103),
+ (4,104);
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_1 AS SELECT i,j,k FROM mv_base_a a INNER JOIN mv_base_b b USING(i);
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+ i | j | k
+---+----+-----
+ 1 | 10 | 101
+ 2 | 20 | 102
+ 3 | 30 | 103
+ 4 | 40 | 104
+(4 rows)
+
+-- immediaite maintenance
+BEGIN;
+INSERT INTO mv_base_b VALUES(5,105);
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+ i | j | k
+---+----+-----
+ 1 | 10 | 101
+ 2 | 20 | 102
+ 3 | 30 | 103
+ 4 | 40 | 104
+ 5 | 50 | 105
+(5 rows)
+
+UPDATE mv_base_a SET j = 0 WHERE i = 1;
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+ i | j | k
+---+----+-----
+ 1 | 0 | 101
+ 2 | 20 | 102
+ 3 | 30 | 103
+ 4 | 40 | 104
+ 5 | 50 | 105
+(5 rows)
+
+DELETE FROM mv_base_b WHERE (i,k) = (5,105);
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+ i | j | k
+---+----+-----
+ 1 | 0 | 101
+ 2 | 20 | 102
+ 3 | 30 | 103
+ 4 | 40 | 104
+(4 rows)
+
+ROLLBACK;
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+ i | j | k
+---+----+-----
+ 1 | 10 | 101
+ 2 | 20 | 102
+ 3 | 30 | 103
+ 4 | 40 | 104
+(4 rows)
+
+-- result of materliazied view have DISTINCT clause or the duplicate result.
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_duplicate AS SELECT j FROM mv_base_a;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_distinct AS SELECT DISTINCT j FROM mv_base_a;
+INSERT INTO mv_base_a VALUES(6,20);
+SELECT * FROM mv_ivm_duplicate ORDER BY 1;
+ j
+----
+ 10
+ 20
+ 20
+ 30
+ 40
+ 50
+(6 rows)
+
+SELECT * FROM mv_ivm_distinct ORDER BY 1;
+ j
+----
+ 10
+ 20
+ 30
+ 40
+ 50
+(5 rows)
+
+DELETE FROM mv_base_a WHERE (i,j) = (2,20);
+SELECT * FROM mv_ivm_duplicate ORDER BY 1;
+ j
+----
+ 10
+ 20
+ 30
+ 40
+ 50
+(5 rows)
+
+SELECT * FROM mv_ivm_distinct ORDER BY 1;
+ j
+----
+ 10
+ 20
+ 30
+ 40
+ 50
+(5 rows)
+
+ROLLBACK;
+-- support SUM(), COUNT() and AVG() aggregation function
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_agg AS SELECT i, SUM(j), COUNT(i),AVG(j) FROM mv_base_a GROUP BY i;
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4;
+ i | sum | count | avg
+---+-----+-------+---------------------
+ 1 | 10 | 1 | 10.0000000000000000
+ 2 | 20 | 1 | 20.0000000000000000
+ 3 | 30 | 1 | 30.0000000000000000
+ 4 | 40 | 1 | 40.0000000000000000
+ 5 | 50 | 1 | 50.0000000000000000
+(5 rows)
+
+INSERT INTO mv_base_a VALUES(2,100);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4;
+ i | sum | count | avg
+---+-----+-------+---------------------
+ 1 | 10 | 1 | 10.0000000000000000
+ 2 | 120 | 2 | 60.0000000000000000
+ 3 | 30 | 1 | 30.0000000000000000
+ 4 | 40 | 1 | 40.0000000000000000
+ 5 | 50 | 1 | 50.0000000000000000
+(5 rows)
+
+UPDATE mv_base_a SET j = 200 WHERE (i,j) = (2,100);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4;
+ i | sum | count | avg
+---+-----+-------+----------------------
+ 1 | 10 | 1 | 10.0000000000000000
+ 2 | 220 | 2 | 110.0000000000000000
+ 3 | 30 | 1 | 30.0000000000000000
+ 4 | 40 | 1 | 40.0000000000000000
+ 5 | 50 | 1 | 50.0000000000000000
+(5 rows)
+
+DELETE FROM mv_base_a WHERE (i,j) = (2,200);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4;
+ i | sum | count | avg
+---+-----+-------+---------------------
+ 1 | 10 | 1 | 10.0000000000000000
+ 2 | 20 | 1 | 20.0000000000000000
+ 3 | 30 | 1 | 30.0000000000000000
+ 4 | 40 | 1 | 40.0000000000000000
+ 5 | 50 | 1 | 50.0000000000000000
+(5 rows)
+
+ROLLBACK;
+-- support COUNT(*) aggregation function
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_agg AS SELECT i, SUM(j),COUNT(*) FROM mv_base_a GROUP BY i;
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
+ i | sum | count
+---+-----+-------
+ 1 | 10 | 1
+ 2 | 20 | 1
+ 3 | 30 | 1
+ 4 | 40 | 1
+ 5 | 50 | 1
+(5 rows)
+
+INSERT INTO mv_base_a VALUES(2,100);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
+ i | sum | count
+---+-----+-------
+ 1 | 10 | 1
+ 2 | 120 | 2
+ 3 | 30 | 1
+ 4 | 40 | 1
+ 5 | 50 | 1
+(5 rows)
+
+ROLLBACK;
+-- support having only aggregation function without GROUP clause
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_group AS SELECT SUM(j)FROM mv_base_a;
+SELECT * FROM mv_ivm_group ORDER BY 1;
+ sum
+-----
+ 150
+(1 row)
+
+INSERT INTO mv_base_a VALUES(6,20);
+SELECT * FROM mv_ivm_group ORDER BY 1;
+ sum
+-----
+ 170
+(1 row)
+
+ROLLBACK;
+-- resolved issue: When use AVG() function and values is indivisible, result of AVG() is incorrect.
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_avg_bug AS SELECT i, SUM(j), COUNT(j), AVG(j) FROM mv_base_A GROUP BY i;
+SELECT * FROM mv_ivm_avg_bug ORDER BY 1,2,3;
+ i | sum | count | avg
+---+-----+-------+---------------------
+ 1 | 10 | 1 | 10.0000000000000000
+ 2 | 20 | 1 | 20.0000000000000000
+ 3 | 30 | 1 | 30.0000000000000000
+ 4 | 40 | 1 | 40.0000000000000000
+ 5 | 50 | 1 | 50.0000000000000000
+(5 rows)
+
+INSERT INTO mv_base_a VALUES
+ (1,0),
+ (1,0),
+ (2,30),
+ (2,30);
+SELECT * FROM mv_ivm_avg_bug ORDER BY 1,2,3;
+ i | sum | count | avg
+---+-----+-------+---------------------
+ 1 | 10 | 3 | 3.3333333333333333
+ 2 | 80 | 3 | 26.6666666666666667
+ 3 | 30 | 1 | 30.0000000000000000
+ 4 | 40 | 1 | 40.0000000000000000
+ 5 | 50 | 1 | 50.0000000000000000
+(5 rows)
+
+DELETE FROM mv_base_a WHERE (i,j) = (1,0);
+DELETE FROM mv_base_a WHERE (i,j) = (2,30);
+SELECT * FROM mv_ivm_avg_bug ORDER BY 1,2,3;
+ i | sum | count | avg
+---+-----+-------+---------------------
+ 1 | 10 | 1 | 10.0000000000000000
+ 2 | 20 | 1 | 20.0000000000000000
+ 3 | 30 | 1 | 30.0000000000000000
+ 4 | 40 | 1 | 40.0000000000000000
+ 5 | 50 | 1 | 50.0000000000000000
+(5 rows)
+
+ROLLBACK;
+-- support MIN(), MAX() aggregation functions
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_min_max AS SELECT i, MIN(j), MAX(j) FROM mv_base_a GROUP BY i;
+SELECT * FROM mv_ivm_min_max ORDER BY 1,2,3;
+ i | min | max
+---+-----+-----
+ 1 | 10 | 10
+ 2 | 20 | 20
+ 3 | 30 | 30
+ 4 | 40 | 40
+ 5 | 50 | 50
+(5 rows)
+
+INSERT INTO mv_base_a VALUES
+ (1,11), (1,12),
+ (2,21), (2,22),
+ (3,31), (3,32),
+ (4,41), (4,42),
+ (5,51), (5,52);
+SELECT * FROM mv_ivm_min_max ORDER BY 1,2,3;
+ i | min | max
+---+-----+-----
+ 1 | 10 | 12
+ 2 | 20 | 22
+ 3 | 30 | 32
+ 4 | 40 | 42
+ 5 | 50 | 52
+(5 rows)
+
+DELETE FROM mv_base_a WHERE (i,j) IN ((1,10), (2,21), (3,32));
+SELECT * FROM mv_ivm_min_max ORDER BY 1,2,3;
+ i | min | max
+---+-----+-----
+ 1 | 11 | 12
+ 2 | 20 | 22
+ 3 | 30 | 31
+ 4 | 40 | 42
+ 5 | 50 | 52
+(5 rows)
+
+ROLLBACK;
+-- support MIN(), MAX() aggregation functions without GROUP clause
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_min_max AS SELECT MIN(j), MAX(j) FROM mv_base_a;
+SELECT * FROM mv_ivm_min_max;
+ min | max
+-----+-----
+ 10 | 50
+(1 row)
+
+INSERT INTO mv_base_a VALUES
+ (0,0), (6,60), (7,70);
+SELECT * FROM mv_ivm_min_max;
+ min | max
+-----+-----
+ 0 | 70
+(1 row)
+
+DELETE FROM mv_base_a WHERE (i,j) IN ((0,0), (7,70));
+SELECT * FROM mv_ivm_min_max;
+ min | max
+-----+-----
+ 10 | 60
+(1 row)
+
+ROLLBACK;
+-- support self join view and multiple change on the same table
+BEGIN;
+CREATE TABLE t (i int, v int);
+INSERT INTO t VALUES (1, 10), (2, 20), (3, 30);
+CREATE INCREMENTAL MATERIALIZED VIEW mv_self(v1, v2) AS
+ SELECT t1.v, t2.v FROM t t1 JOIN t t2 ON t1.i = t2.i;
+SELECT * FROM mv_self ORDER BY v1;
+ v1 | v2
+----+----
+ 10 | 10
+ 20 | 20
+ 30 | 30
+(3 rows)
+
+INSERT INTO t VALUES (4,40);
+DELETE FROM t WHERE i = 1;
+UPDATE t SET v = v*10 WHERE i=2;
+SELECT * FROM mv_self ORDER BY v1;
+ v1 | v2
+-----+-----
+ 30 | 30
+ 40 | 40
+ 200 | 200
+(3 rows)
+
+WITH
+ ins_t1 AS (INSERT INTO t VALUES (5,50) RETURNING 1),
+ ins_t2 AS (INSERT INTO t VALUES (6,60) RETURNING 1),
+ upd_t AS (UPDATE t SET v = v + 100 RETURNING 1),
+ dlt_t AS (DELETE FROM t WHERE i IN (4,5) RETURNING 1)
+SELECT NULL;
+ ?column?
+----------
+
+(1 row)
+
+SELECT * FROM mv_self ORDER BY v1;
+ v1 | v2
+-----+-----
+ 50 | 50
+ 60 | 60
+ 130 | 130
+ 300 | 300
+(4 rows)
+
+ROLLBACK;
+-- support simultaneous table changes
+BEGIN;
+CREATE TABLE r (i int, v int);
+CREATE TABLE s (i int, v int);
+INSERT INTO r VALUES (1, 10), (2, 20), (3, 30);
+INSERT INTO s VALUES (1, 100), (2, 200), (3, 300);
+CREATE INCREMENTAL MATERIALIZED VIEW mv(v1, v2) AS
+ SELECT r.v, s.v FROM r JOIN s USING(i);
+SELECT * FROM mv ORDER BY v1;
+ v1 | v2
+----+-----
+ 10 | 100
+ 20 | 200
+ 30 | 300
+(3 rows)
+
+WITH
+ ins_r AS (INSERT INTO r VALUES (1,11) RETURNING 1),
+ ins_r2 AS (INSERT INTO r VALUES (3,33) RETURNING 1),
+ ins_s AS (INSERT INTO s VALUES (2,222) RETURNING 1),
+ upd_r AS (UPDATE r SET v = v + 1000 WHERE i = 2 RETURNING 1),
+ dlt_s AS (DELETE FROM s WHERE i = 3 RETURNING 1)
+SELECT NULL;
+ ?column?
+----------
+
+(1 row)
+
+SELECT * FROM mv ORDER BY v1;
+ v1 | v2
+------+-----
+ 10 | 100
+ 11 | 100
+ 1020 | 200
+ 1020 | 222
+(4 rows)
+
+ROLLBACK;
+-- support foreign reference constrains
+BEGIN;
+CREATE TABLE ri1 (i int PRIMARY KEY);
+CREATE TABLE ri2 (i int PRIMARY KEY REFERENCES ri1(i) ON UPDATE CASCADE ON DELETE CASCADE, v int);
+INSERT INTO ri1 VALUES (1),(2),(3);
+INSERT INTO ri2 VALUES (1),(2),(3);
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ri(i1, i2) AS
+ SELECT ri1.i, ri2.i FROM ri1 JOIN ri2 USING(i);
+SELECT * FROM mv_ri ORDER BY i1;
+ i1 | i2
+----+----
+ 1 | 1
+ 2 | 2
+ 3 | 3
+(3 rows)
+
+UPDATE ri1 SET i=10 where i=1;
+DELETE FROM ri1 WHERE i=2;
+SELECT * FROM mv_ri ORDER BY i2;
+ i1 | i2
+----+----
+ 3 | 3
+ 10 | 10
+(2 rows)
+
+ROLLBACK;
+-- contain system column
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm01 AS SELECT i,j,xmin FROM mv_base_a;
+ERROR: system column is not supported with IVM
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm02 AS SELECT i,j FROM mv_base_a WHERE xmin = '610';
+ERROR: system column is not supported with IVM
+-- contain subquery
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm03 AS SELECT i,j FROM mv_base_a WHERE i IN (SELECT i FROM mv_base_b WHERE k < 103 );
+ERROR: subquery is not supported with IVM
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm04 AS SELECT a.i,a.j FROM mv_base_a a,( SELECT * FROM mv_base_b) b WHERE a.i = b.i;
+ERROR: subquery is not supported with IVM
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm05 AS SELECT i,j, (SELECT k FROM mv_base_b b WHERE a.i = b.i) FROM mv_base_a a;
+ERROR: subquery is not supported with IVM
+-- contain CTE
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm06 AS WITH b AS (SELECT i,k FROM mv_base_b WHERE k < 103) SELECT a.i,a.j FROM mv_base_a a,b WHERE a.i = b.i;
+ERROR: CTE is not supported with IVM
+-- contain view or materialized view
+CREATE VIEW b_view AS SELECT i,k FROM mv_base_b;
+CREATE MATERIALIZED VIEW b_mview AS SELECT i,k FROM mv_base_b;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm07 AS SELECT a.i,a.j FROM mv_base_a a,b_view b WHERE a.i = b.i;
+ERROR: VIEW or MATERIALIZED VIEW is not supported with IVM
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm08 AS SELECT a.i,a.j FROM mv_base_a a,b_mview b WHERE a.i = b.i;
+ERROR: VIEW or MATERIALIZED VIEW is not supported with IVM
+DROP TABLE mv_base_b CASCADE;
+NOTICE: drop cascades to 3 other objects
+DETAIL: drop cascades to materialized view mv_ivm_1
+drop cascades to view b_view
+drop cascades to materialized view b_mview
+DROP TABLE mv_base_a CASCADE;
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index fc0f14122b..4b5d448098 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -78,7 +78,7 @@ test: brin gin gist spgist privileges init_privs security_label collate matview
# ----------
# Another group of parallel tests
# ----------
-test: create_table_like alter_generic alter_operator misc async dbsize misc_functions sysviews tsrf tidscan collate.icu.utf8
+test: create_table_like alter_generic alter_operator misc async dbsize misc_functions sysviews tsrf tidscan collate.icu.utf8 incremental_matview
# rules cannot run concurrently with any test that creates
# a view or rule in the public schema
diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule
index 68ac56acdb..0f6f8f503f 100644
--- a/src/test/regress/serial_schedule
+++ b/src/test/regress/serial_schedule
@@ -112,6 +112,7 @@ test: init_privs
test: security_label
test: collate
test: matview
+test: incremental_matview
test: lock
test: replica_identity
test: rowsecurity
diff --git a/src/test/regress/sql/incremental_matview.sql b/src/test/regress/sql/incremental_matview.sql
new file mode 100644
index 0000000000..d815bd8662
--- /dev/null
+++ b/src/test/regress/sql/incremental_matview.sql
@@ -0,0 +1,180 @@
+-- create a table to use as a basis for views and materialized views in various combinations
+CREATE TABLE mv_base_a (i int, j int);
+INSERT INTO mv_base_a VALUES
+ (1,10),
+ (2,20),
+ (3,30),
+ (4,40),
+ (5,50);
+CREATE TABLE mv_base_b (i int, k int);
+INSERT INTO mv_base_b VALUES
+ (1,101),
+ (2,102),
+ (3,103),
+ (4,104);
+
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_1 AS SELECT i,j,k FROM mv_base_a a INNER JOIN mv_base_b b USING(i);
+
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+-- immediaite maintenance
+BEGIN;
+INSERT INTO mv_base_b VALUES(5,105);
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+UPDATE mv_base_a SET j = 0 WHERE i = 1;
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+DELETE FROM mv_base_b WHERE (i,k) = (5,105);
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+ROLLBACK;
+SELECT * FROM mv_ivm_1 ORDER BY 1,2,3;
+
+-- result of materliazied view have DISTINCT clause or the duplicate result.
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_duplicate AS SELECT j FROM mv_base_a;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_distinct AS SELECT DISTINCT j FROM mv_base_a;
+INSERT INTO mv_base_a VALUES(6,20);
+SELECT * FROM mv_ivm_duplicate ORDER BY 1;
+SELECT * FROM mv_ivm_distinct ORDER BY 1;
+DELETE FROM mv_base_a WHERE (i,j) = (2,20);
+SELECT * FROM mv_ivm_duplicate ORDER BY 1;
+SELECT * FROM mv_ivm_distinct ORDER BY 1;
+ROLLBACK;
+
+-- support SUM(), COUNT() and AVG() aggregation function
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_agg AS SELECT i, SUM(j), COUNT(i),AVG(j) FROM mv_base_a GROUP BY i;
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4;
+INSERT INTO mv_base_a VALUES(2,100);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4;
+UPDATE mv_base_a SET j = 200 WHERE (i,j) = (2,100);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4;
+DELETE FROM mv_base_a WHERE (i,j) = (2,200);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3,4;
+ROLLBACK;
+
+-- support COUNT(*) aggregation function
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_agg AS SELECT i, SUM(j),COUNT(*) FROM mv_base_a GROUP BY i;
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
+INSERT INTO mv_base_a VALUES(2,100);
+SELECT * FROM mv_ivm_agg ORDER BY 1,2,3;
+ROLLBACK;
+
+-- support having only aggregation function without GROUP clause
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_group AS SELECT SUM(j)FROM mv_base_a;
+SELECT * FROM mv_ivm_group ORDER BY 1;
+INSERT INTO mv_base_a VALUES(6,20);
+SELECT * FROM mv_ivm_group ORDER BY 1;
+ROLLBACK;
+
+-- resolved issue: When use AVG() function and values is indivisible, result of AVG() is incorrect.
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_avg_bug AS SELECT i, SUM(j), COUNT(j), AVG(j) FROM mv_base_A GROUP BY i;
+SELECT * FROM mv_ivm_avg_bug ORDER BY 1,2,3;
+INSERT INTO mv_base_a VALUES
+ (1,0),
+ (1,0),
+ (2,30),
+ (2,30);
+SELECT * FROM mv_ivm_avg_bug ORDER BY 1,2,3;
+DELETE FROM mv_base_a WHERE (i,j) = (1,0);
+DELETE FROM mv_base_a WHERE (i,j) = (2,30);
+SELECT * FROM mv_ivm_avg_bug ORDER BY 1,2,3;
+ROLLBACK;
+
+-- support MIN(), MAX() aggregation functions
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_min_max AS SELECT i, MIN(j), MAX(j) FROM mv_base_a GROUP BY i;
+SELECT * FROM mv_ivm_min_max ORDER BY 1,2,3;
+INSERT INTO mv_base_a VALUES
+ (1,11), (1,12),
+ (2,21), (2,22),
+ (3,31), (3,32),
+ (4,41), (4,42),
+ (5,51), (5,52);
+SELECT * FROM mv_ivm_min_max ORDER BY 1,2,3;
+DELETE FROM mv_base_a WHERE (i,j) IN ((1,10), (2,21), (3,32));
+SELECT * FROM mv_ivm_min_max ORDER BY 1,2,3;
+ROLLBACK;
+
+-- support MIN(), MAX() aggregation functions without GROUP clause
+BEGIN;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_min_max AS SELECT MIN(j), MAX(j) FROM mv_base_a;
+SELECT * FROM mv_ivm_min_max;
+INSERT INTO mv_base_a VALUES
+ (0,0), (6,60), (7,70);
+SELECT * FROM mv_ivm_min_max;
+DELETE FROM mv_base_a WHERE (i,j) IN ((0,0), (7,70));
+SELECT * FROM mv_ivm_min_max;
+ROLLBACK;
+
+-- support self join view and multiple change on the same table
+BEGIN;
+CREATE TABLE t (i int, v int);
+INSERT INTO t VALUES (1, 10), (2, 20), (3, 30);
+CREATE INCREMENTAL MATERIALIZED VIEW mv_self(v1, v2) AS
+ SELECT t1.v, t2.v FROM t t1 JOIN t t2 ON t1.i = t2.i;
+SELECT * FROM mv_self ORDER BY v1;
+INSERT INTO t VALUES (4,40);
+DELETE FROM t WHERE i = 1;
+UPDATE t SET v = v*10 WHERE i=2;
+SELECT * FROM mv_self ORDER BY v1;
+WITH
+ ins_t1 AS (INSERT INTO t VALUES (5,50) RETURNING 1),
+ ins_t2 AS (INSERT INTO t VALUES (6,60) RETURNING 1),
+ upd_t AS (UPDATE t SET v = v + 100 RETURNING 1),
+ dlt_t AS (DELETE FROM t WHERE i IN (4,5) RETURNING 1)
+SELECT NULL;
+SELECT * FROM mv_self ORDER BY v1;
+ROLLBACK;
+
+-- support simultaneous table changes
+BEGIN;
+CREATE TABLE r (i int, v int);
+CREATE TABLE s (i int, v int);
+INSERT INTO r VALUES (1, 10), (2, 20), (3, 30);
+INSERT INTO s VALUES (1, 100), (2, 200), (3, 300);
+CREATE INCREMENTAL MATERIALIZED VIEW mv(v1, v2) AS
+ SELECT r.v, s.v FROM r JOIN s USING(i);
+SELECT * FROM mv ORDER BY v1;
+WITH
+ ins_r AS (INSERT INTO r VALUES (1,11) RETURNING 1),
+ ins_r2 AS (INSERT INTO r VALUES (3,33) RETURNING 1),
+ ins_s AS (INSERT INTO s VALUES (2,222) RETURNING 1),
+ upd_r AS (UPDATE r SET v = v + 1000 WHERE i = 2 RETURNING 1),
+ dlt_s AS (DELETE FROM s WHERE i = 3 RETURNING 1)
+SELECT NULL;
+SELECT * FROM mv ORDER BY v1;
+ROLLBACK;
+
+-- support foreign reference constrains
+BEGIN;
+CREATE TABLE ri1 (i int PRIMARY KEY);
+CREATE TABLE ri2 (i int PRIMARY KEY REFERENCES ri1(i) ON UPDATE CASCADE ON DELETE CASCADE, v int);
+INSERT INTO ri1 VALUES (1),(2),(3);
+INSERT INTO ri2 VALUES (1),(2),(3);
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ri(i1, i2) AS
+ SELECT ri1.i, ri2.i FROM ri1 JOIN ri2 USING(i);
+SELECT * FROM mv_ri ORDER BY i1;
+UPDATE ri1 SET i=10 where i=1;
+DELETE FROM ri1 WHERE i=2;
+SELECT * FROM mv_ri ORDER BY i2;
+ROLLBACK;
+
+-- contain system column
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm01 AS SELECT i,j,xmin FROM mv_base_a;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm02 AS SELECT i,j FROM mv_base_a WHERE xmin = '610';
+-- contain subquery
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm03 AS SELECT i,j FROM mv_base_a WHERE i IN (SELECT i FROM mv_base_b WHERE k < 103 );
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm04 AS SELECT a.i,a.j FROM mv_base_a a,( SELECT * FROM mv_base_b) b WHERE a.i = b.i;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm05 AS SELECT i,j, (SELECT k FROM mv_base_b b WHERE a.i = b.i) FROM mv_base_a a;
+-- contain CTE
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm06 AS WITH b AS (SELECT i,k FROM mv_base_b WHERE k < 103) SELECT a.i,a.j FROM mv_base_a a,b WHERE a.i = b.i;
+-- contain view or materialized view
+CREATE VIEW b_view AS SELECT i,k FROM mv_base_b;
+CREATE MATERIALIZED VIEW b_mview AS SELECT i,k FROM mv_base_b;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm07 AS SELECT a.i,a.j FROM mv_base_a a,b_view b WHERE a.i = b.i;
+CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm08 AS SELECT a.i,a.j FROM mv_base_a a,b_mview b WHERE a.i = b.i;
+
+DROP TABLE mv_base_b CASCADE;
+DROP TABLE mv_base_a CASCADE;
Attached is the latest patch to add support for Incremental
Materialized View Maintenance (IVM). IVM allows to reflect
modifications made on base tables immediately to the target
materialized views.
Up to now, IVM supports materialized views using:
- Inner joins
- Some aggregate functions (count, sum, min, max, avg)
- GROUP BY
- Self joins
With the latest patch now IVM supports subqueries in addition to
above.
Known limitations are listed here:
https://github.com/sraoss/pgsql-ivm/issues
See more details at:
https://wiki.postgresql.org/wiki/Incremental_View_Maintenance
About subquery support:
The patch supports simple subqueries using EXISTS:
CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_exists_subquery AS SELECT
a.i, a.j FROM mv_base_a a WHERE EXISTS(SELECT 1 FROM mv_base_b b WHERE
a.i = b.i);
and subqueries in the FROM clause:
CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_subquery AS SELECT a.i,a.j
FROM mv_base_a a,( SELECT * FROM mv_base_b) b WHERE a.i = b.i;
Other form of subqueries such as below are not supported:
-- WHERE IN .. (subquery) is not supported
CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm03 AS SELECT i,j FROM
mv_base_a WHERE i IN (SELECT i FROM mv_base_b WHERE k < 103 );
-- subqueries in target list is not supported
CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm05 AS SELECT i,j, (SELECT k
FROM mv_base_b b WHERE a.i = b.i) FROM mv_base_a a;
-- nested EXISTS subqueries is not supported
CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm11 AS SELECT a.i,a.j FROM
mv_base_a a WHERE EXISTS(SELECT 1 FROM mv_base_b b WHERE EXISTS(SELECT
1 FROM mv_base_b c WHERE b.i = c.i));
-- EXISTS subquery with aggragate function is not supported
CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm_exists AS SELECT COUNT(*)
FROM mv_base_a a WHERE EXISTS(SELECT 1 FROM mv_base_b b WHERE a.i =
b.i) OR a.i > 5;
-- EXISTS subquery with condition except AND is not supported.
CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm10 AS SELECT a.i,a.j FROM
mv_base_a a WHERE EXISTS(SELECT 1 FROM mv_base_b b WHERE a.i = b.i) OR
a.i > 5;
This work has been done by Yugo Nagata (nagata@sraoss.co.jp), Takuma
Hoshiai (hoshiai@sraoss.co.jp). Adding support for EXISTS clause has
been done by Takuma.
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
Attachments:
Hi,
Attached is the latest patch (v8) to add support for Incremental View
Maintenance (IVM). This patch adds OUTER join support in addition
to the patch (v7) submitted last week in the following post.
On Fri, 22 Nov 2019 15:29:45 +0900 (JST)
Tatsuo Ishii <ishii@sraoss.co.jp> wrote:
Up to now, IVM supports materialized views using:
- Inner joins
- Some aggregate functions (count, sum, min, max, avg)
- GROUP BY
- Self joinsWith the latest patch now IVM supports subqueries in addition to
above.Known limitations are listed here:
https://github.com/sraoss/pgsql-ivm/issues
See more details at:
https://wiki.postgresql.org/wiki/Incremental_View_Maintenance
* About outer join support:
In case of outer-join, when a table is modified, in addition to deltas
which occur in inner-join case, we also need to deletion or insertion of
dangling tuples, that is, null-extended tuples generated when a join
condition isn't met.
[Example]
---------------------------------------------
-- Create base tables and an outer join view
CREATE TABLE r(i int);
CREATE TABLE s(j int);
INSERT INTO r VALUES (1);
CREATE INCREMENTAL MATERIALIZED VIEW mv
AS SELECT * FROM r LEFT JOIN s ON r.i=s.j;
SELECT * FROM mv;
i | j
---+---
(1 row)
-- After an insertion to a base table ...
INSERT INTO s VALUES (1);
-- (1,1) is inserted and (1,null) is deleted from the view.
SELECT * FROM mv;
i | j
---+---
1 | 1
(1 row)
---------------------------------------------
Our implementation is basically based on the algorithm of Larson & Zhou
(2007) [1]Efficient Maintenance of Materialized Outer-Join Views (Larson & Zhou, 2007) https://ieeexplore.ieee.org/document/4221654. Before view maintenances, the view definition query's jointree
is analysed to make "view maintenance graph". This graph represents
which tuples in the views are affected when a base table is modified.
Specifically, tuples which are not null-extended on the modified table
(that is, tuples generated by joins with the modiifed table) are directly
affected. The delta of such effects are calculated similarly to inner-joins.
On the other hand, dangling tuples generated by anti-joins with directly
affected tuples can be indirectly affected. This means that we may need to
delete dangling tuples when any tuples are inserted to a table, as well as
to insert dangling tuples when tuples are deleted from a table.
[1]: Efficient Maintenance of Materialized Outer-Join Views (Larson & Zhou, 2007) https://ieeexplore.ieee.org/document/4221654
https://ieeexplore.ieee.org/document/4221654
Although the original paper assumes that every base table and view have a
unique key and tuple duplicates is disallowed, we allow this. If a view has
tuple duplicates, we have to determine the number of each dangling tuple to
be inserted into the view when tuples in a table are deleted. For this purpose,
we count the number of each tuples which constitute a deleted tuple. These
counts are stored as JSONB object in the delta table, and we use this
information to maintain outer-join views. Also, we support outer self-joins
that is not assumed in the original paper.
* Restrictions
Currently, we have following restrictions:
- outer join view's targetlist must contain attributes used in join conditions
- outer join view's targetlist cannot contain non-strict functions
- outer join supports only simple equijoin
- outer join view's WHERE clause cannot contain non null-rejecting predicates
- aggregate is not supported with outer join
- subquery (including EXSITS) is not supported with outer join
Regression tests for all patterns of 3-way outer join and are added.
Moreover, I reordered IVM related functions in matview.c so that ones
which have relationship will be located closely. Moreover, I added more
comments.
Regards,
Yugo Nagata
--
Yugo Nagata <nagata@sraoss.co.jp>
Attachments:
IVM_v8.patch.gzapplication/gzip; name=IVM_v8.patch.gzDownload
�#��] IVM_v8.patch �<kw�6���_�fw2�%KV�����Q%��c�������C����">����������69s&��D����\��'��3�S����m'1�?�m�I� �&-|�
�[~��������}w�����w�v���������������3l5�����b�s��Y�@4���'�mx>������y����Ob9W:�@�=
�~*c'��P�Q�O����� �}e���'2L���(���'"�n��7�'�s'N}�}�f`5&���a�����(�=I�8s��/�,����w�����#��Lsy6����M?���n���9@� ���'�|y/d�[L`�n,g ��1s�0�����j�
�o������>����X�����;��Jm,'mX>l�6wh�FVJ���tK��h�Pz�� ��������&�~�
�xj��N�P4���$��r��� \q��Y"'�"����x�P�D?�m��z�QO���A�{����+����I�"�����H�>�����U�������,��VCa�����m�r����_o��_�U�v� ��+��/M�j�~u���n��|S�M���$�O@����|�D^Y�����a=�#I���J;o����!wN�Uv�����=�8<�G��G$_����`����^ZI-$d���Q`�c)l�'�^�����I@6983D���1��:f��A��&�?�\�>�)�h&E��)�O8�N�F�F���B�^y '�'�)�`%[4��e�����Lz>t�l������Hv�����F��D?S���0
�I�j����
Pjtk����}AE��4�X��5���a��A�{? �[
��[Xan�M��0��Hf�?�`�}���S&�����}n�����H��~A�9����a�p���F,��>�i(H�B�V����QxB ����6|^�y0�Hd0���z��.Gy�N��Y�(�@g
&{�0i,
M@��YS8w����!|8���0DY*~��G�6��ae��v���Iw\���?�SEd_&4�#sE0i�����*���RM�oB�l�v�i�8�]�}�Q�6E��F
'����~!���"(���|�k��|u�����q�"B�L������E`�dI1����KO��'�l����-iK��A��@��+�R����1��W�-t�'�W����s��h6C�M�������]4q7h#������I4�mt>|��J?�-�� ;5�l>�b4q���l���
� ��uliilu�I�
Q Y �M%D2�� �J�R^�I��qoe���U����N�8��g�M[.$����h"����=��������>�&���a\��<�A����1���/����C��|j�0��n���U�w��e>i;����X� �D2�f,��;��� QU�I�`U�d�c%r{�M�����duD� h�A�[��nJ]�e����UAm�� gd@����['H@u���� br�>�oe��0��}4�MGD�����-a��6|�m���d��*X��l=�C|6f��@� 2�l�&CM�:$�%S3�F:�%�,w*Y�]�;�v'��;>�Gcy�����
�e!� c��w�/�El�� P�zz�)�(�`i���)s�=��M�em�/]�N;�,�>����*N����t���F���4�Y��P;�) �l�I�&����b�k>b#^`����6R�l5NMb�#ni|/�(X���ha�DX������TXY��k�@`5?!" �����T�=��d��pr�w��Nd�y��<�{��i,��OC��sF�_$}�gsQ���~@�#(� $-H�c�sZ��b#g�\���-����I~H�"�{��=�i��f�{}5P �&<(B���[�B���L��'�6
�����5��I�%`BI����p�h��s�ju:��������
R��
IXE�;d���Y��
���6P��jT{P�P����|6����p���Z���u�s�`�6�6m�KcY�lv��l�m��W���+�������<��t]��~Uz1E�X0n�������V+��!��u�^�j��f1?��� >B����p(qx�������^f�-��a��Cc��(�C�v
j��-�E��@�/��[�
�����xQ�P��0�������p"�(5:�)>_�
j�8����bC��pom�v�Hm7
S�9}��3� �������2��Qwh�q��?p3�-�U^��y�=QY����(���C'�pYb�I�5����91@l#5�I=�
�F�J�u��q�����fg�P#��">������(�a : ��N��)^�C8zc�@�x��,���[��!���[
s�9�7�����?��&J����q �������vHl��>���6M�\�/!�%�$��tX��)^4�%���������j�| ����D�����:�j��w�-��g�(�b�M?�AT���1�<����'��!��u5�}���9��������_�(�=�{����#�?'�`5x��Y�)7zJ��
���� �����J�gJ�Cl��fs�?EIm��O
��@��.�<����w��F��B�~��Kc6��_�����!�B�G)4��e���j���v�L����9"�7Is7���Og�";�fg�}y�`��s��m^���J'9p\�g����m|���M�Z�����S,�L�j�YI*C��u��������,�+����B�Fs_z�r
B�����8 1���1[��-�:����0z4��0�������EKk(�"�S�e��5�x{�/��K:cL���uw�k�����S���#8���
v-��f �T3������D1��v��O��c?��Zm��S��0F�%���
C���qD�=g�
V���-!������O]{
.�� �= �e��kW����������s6��c�Bmd����|e{���=���Fz�F��lr]�4m��dw5�Q��8����&}S��WS���8�G:�'lZ��M��|7�R�m�F����9������o��b!���Dj.S�6p�����8�z'h5���!�
)�^��p�B��L/F ����H�Mh��Z��l����d��`�H��X%��V�6Ey�1y���u���I�>����4�9��S���/���Z
�E0��$���r���-'���%�%p��U- U2L�0�QH*���tiK���} (��y�z�?���37�S���k��m�]�AH,����:�yN������NT�)���&V��+O�YUKm1�cY�������zP;��6_&_JhP���!:�������E�U2/9�j����i@@�n����P���������7}�����Q6�rQ�*��`��� ����!;�^�����Fq������XK��t��OX�������6�X���(��ex<�I>f�m.�2��Vo0��V�\��/� r�������YcI^���$FQ*��v-W��s4eB�q�wK��;�*�\[Ki�]c���Wx#�)���D��V��=H�s�qrI^�7�����O���0���:��^f�1�u!xlt�a�u�1���c�U��f��M�F�9����*�)d)��~x������;�]4� ��{V_���r����F�-�3 *$���p�t�Lj��r{/��
��Jm��)�On��
"?�wX���C���'�_��Z��{<�y�10���5�h�SW�������Z�����I}��s;�~��tK]��
#\��0"��!�Q���� A�N'a���z<0������R
�V�S�o\����v�K�<�
��*���DH�S��q4k��OR�����|X�t<a4��M1�R��}��k� ��P����h
�
��S[��V���r������/�fO��yws��X �d���f@�n���>
��\��w�~�������r����O�B�.���>�1bfo���Tf�8��K����I�ph��"�����B�����<tBS��V� $e
��&(�.
+
�4XP�w���O���z�UB�
!����
�Z��rdJH��\�k�PYk��YF����3[O�R�&p�U��oX8�uNu�<�-��MJ�S�B��d���4�_$L�S?�
��;pL�������NJXq��jP; w��i�ic�%�9�E�<�-�����du�D����� ��{SL����Vx��.�"B���l[*9`4�����Q�Q��k%�`�zV�H.���56y������t��q��Q�Aj+��Z���IqRX�b���������
�t��o��Ac��?K���
&�a)�a�*��X�l��q�Z����K�S�r}~'�j,�6kP�\`T@HM:��������@��$r�H�����������
9���R��?�� ���VdU��S�b5��y��
];=�su�)Kz3����|"! Ke�$��f���;��F��G0U��SG/Z�L7d���0��l�����\�S�UhT�[!gq���������:�"C����q��j��T����iFo��
�/�b
��Qz2 �|��� r�����`����
�)|pJ�)����Qo���Z�9��*�����t�c����""-e�U��.��]Gt��'�tNXYvK%�����(��sG�AF���qym_I���)[��^�<i��1�[|�!��6CyGu�|�R<�
(� ��EH����_���T��A�A���m��X��8�t�G�_��Mme �`4�5��D�Y���9-v(�w ����7�p��8w���MA0^_7gtGV�#F��+s�F+�K�,/�S���v�������YLp]D���Y���Y� �>e]��>��j6D*�x�gl�*�
9���A�.u�U�c�Uc��I��������8T�Pu�X��d�����|�F�^9��\�Q����z�&+���:�R���t�=#��
��$��/�{��b8�����;}%'tm�@�6z���������[�j�/Z��v�7����{�\���q�n�x�/
���5�@����"���@� A3�#�xCF��D���L,�M��L]�����qQ��Po����+>�Q��.�����qNg��a�2�x��F���9���Z�d�X+}�A������|������`Ti|�{}5�i��fyF�������~������X���7�������W> O��[`�M�-Z8U]��y+!�qY�J��^���7���e.���SQ��Vv=�P�|-��}g�������G����U��iK�YX��������}~]�T!l�,��i�^�������,V�[��!�`����
�E���
�dp00�/�6����?�Dk7�O�tyS��7�7��B�G,�#�H������8����]�x��"�n4
�����N��[G�T�"�/F9%OX����:p���w��n��w�.1�.���y���\���(��A�|����K�K> �X����P-�����j_����F�`��\�\������?����������K���<�|���(&�"�n��P���LZ��[|+��.���4Q��O�4k�xN���)#��h��aS�Z�F��Z]���k�;����; \��������!�m���x�/bH��B,Q�u�����h�u�{��&w6T;c����)L�������W��l�����)��(�T�����Ky��P��K�<�f�]�/����J��{��>�+���@�"����5 ��T�m�>}�����o��4��� PQl�w�� `��06��#��Ye��J$� ��qR�T�?�7����������q,����=�%#�$6�;/��
���.��A`l!)36��������-X����I3�VWWWU���.a��
�>�\qU�~�)�X��]�b���z�l��%|QF���g���j6�n�<
��[����3\#��S������
�����}sF��&�n�������Y�t�<��(dEDXV����#�����2�whk�^3�lscICS��k���{9��� �ew7�D�qY������Q���
i{ik�3dH)��%��x�\0��:z�v�{��A�%K����x������u2/�.����u�$��E����Ps��$�������M��}�)������pG )4c�� N!��a���A�4.�sqWq������^DY\��<���9����UK���T�>s����wn1�D�<n����a�u}�-�$�H�;:�Kb���v�_;������q$+��``y��a���V����O�;����T� !��W,fm�� q��^�#��a��D�D���h�������3p �^-��P��s�Lz����`�C�^z@�Ho *��&j��W�Y����Xzr��&���Z4���3S�*�W�VV0CBvl�7$U���`���M��� $>���($�C<%��Y����ya�X��iP���6���_W���52V�y]�a�&���9�h����\U,�v��p�����H�cG<!\���k�hg=���7�%�B�Q��3k3�����vk�-�'5
��
u�F�x�b�B�I#EE�-�t](�B)�l[�?s}W���I}R��O��j��DC��/E������m`QcG�y5�l��I��}��-���j^�������������"��s�����!����K�:�9��� �`E c��Q/ �h(��R�j����?]%���q�2�X��I>���}��X���,�0C�Wl�Bv����c�����[�,�!����l��9�'$���y�);E.��u��N�.e�3������[}A��F�YN��]�|*�Q�����Q�l>�_z���"*_����;��z�_�c��HF��Sm��5;�
7�����Y'@K�&� x����5��� /[�j[���%�������Y�^�������[�{�X���
������Y��~J! ��������j����`���$7�G����8}����6�������C?&]��mP�]C������l[����^B���������C��l���Z��7���jh<�[�}��&�A�'��z3��)��%�N�^Rd{Dg����������^q,��/��
1x�+hA3�Y�C�����39<$��
v���������e� [�g���%��;J���zu-��b�S���������)u�gIW��O�G���k��Q������'Kp�t�M������si������ ��=��
.�H�������$��c��fk:����� M�|��e�0@t~����9�O���G�!�DZ �T��{�B��<���=`]�[{��wU����v`����|$�:YcfLS�^N,�&�^���9b��K"��T�x�r����O���K �L"L�L�DfpA=DurJj��sw�R%T+�[M����awfL$��l�/#��!��3� �)��b";��*���'*���J� '������lP��w�{����z(a)�B����z�7�p����I-�d��4�w�?�s(:�;��Pv���M����)R�G��BN��������@���]�n$�(���d0��������O�zs��a��<ir{��TJ��q�\���6���<u��yh�,D�&,�n�����P�vP?�u,-6zM��v��^#5.)j�h{Fy��q]Kn>_����WN�xr���#��^ 2����G���z�����������������S������_�1��+)������~@y|��cd 8�����s����;<y+d�>y�EbU�I����;��b+�)ACr�\�:��B?p����~�26:��rB��^�����#���"\�)1���t�X����H��\�<W=E��p`�s ���C�4���=�gc��\P���/���-U��~����5C8I &XV�WO��^5������2IB�[����k�_��������JZ0��������$`� #|�A���[eG%�T��=]
a� ��vJ"]��`P�����x�<qA�,��RuU����5{���2�DaB���-���n�^p���_75�/D�n����q��22�u�]�7�*F�~�^ ��������F��v�Q,��
E����S��l��~��`���O������Q�����G� ��VH6����K�%�v8�N�t�\7��.���C3�>=A�1�AB��m)����9
�������C���J�����Bq���9g��
�N����P����t�O�������t�H���.�(-��w�/wOj��@���<J5M\�]���d<�/_����M���)
��� Bu�k>��n�<�;��5�^�'Ft�6����G�P'���,����2��!yjS�V>�y6��,k��+7_����8��]q�5����q6���G���2��Wo���T~���v9|�
���67�[��V�]�`���Fl<S�#�R��J6������P�y�u��<*���<+�+
��}uU����������V���;+j��u���~�K��p�h����t8y�>�����c?*�~S���m�q���+��RY��G��N��������C���dh��8��f�-�%3�T3XL���
���D�����2�?�Dbv��?(b~(��R\o4�~����_N��U[�l1����2p-�������`��y\��=�k�����}��-pRk��KUK�xSg�_C���Yh���;���&�/�'o�CO����&p���[�B�a�1��$hp�����U� ?��$��f��K\?���S�/�$4�����1_�kD�����a �S��[
�_H2"��[���M�O��#�0���)��)%�N'7Vwkp}d��(���K���������6;'�jW|��Z���E��������V�^�%ke��T��.��2Y����X�c�@m~m<���L}��r�K&�W�;���>K����~�#��\����V�=�>��m�,�d;{*���6"3$��Y��i� I�`2�l�N�� ���%�{�;!�� �t����S��MWd���Y���Q���=GTS!a��#�L���*����}�����#��"�,�� QA`�d0��Zb��~��4T .�>�rl}
[_8 x��a7j�sW;<�VX� 3����\��joU_a=l�����P�/UD0���maH�5���c�X�}I�/!�TS���xty%���6���B�u��g���q�Y%��S��2jB��X)��\��g�Q��O�}�����8����Q��
�0�������G��Q��9~��}"-=H�=a�I%�Z�<�����l/G1�1��hLz�S����X{�
>�������G���L0=m3a9�.���S �D��i�&������B�48��Q�Y� Z�D�6���;���)Q7������2��v:�Z�y�~@��X8 ���l�m�M�����RV`���Z�64r���������#�x(^��5�>�P�-�v��J^�vq��D��K(#�F/�h�v�Q�*#�2��r�1��E����66��xc�x�=h��[#�,m��@F�{Yz�z�u;��-�Ep���0A{��7j�X�~G�Ax��� U,�p���������J��*y���`���n�jJ�v&��}�"�t:�������SM!��84�Mi��n���Ke�/_�L����6)�O�����\LN__��������z��q�jC����^+_(�{�Q�E�N�26Y�U���1u��GpNH/c���k��(�u/ ���O��X~+j����#���e�Y%��f������r'FRj���8������H�[�?T��R���Ox1���Z��6��m�A`��f�)�{� ��a��(#��a�� ��n��9Lp� ���4�@U��48��aW�W0������S��0V�D�7#Q�.?$�q�O���+����3pl�mt��Cr
q�N�_�d��$>#���#ZNC!�y�T�����(W��u�;� h"[���?Z�E����s�7{� ���KqF��:��l,���J ���A>7�|D�_2R�|�J� A9b���&�o`�\v ���48���#�T�&��D�ORx�]k�=g�\`�`�3;�p�>��2(H�f@q���V�(p��������n����P�����r���CHo�C�����p�W��D�&Ym��n���9�eu@DT5�����c�\g���A������R4$}�J�=�QO��"�9����D�~/�����$�RlXy������Vv}�A��9��f�TY���a����0�0�pM�bh���E� ����L�@2I|^��WU�E���m���5s{���Tg.]l��!���!�-Y(���8Z��K`�/���5�[�e���B�tN�����W�=����#�J�y%�l��O0����!f�M0jZ��k9���3v��!�Y�d�Z�/��:m�]���Y�FTK��g)��<�}��/��[|^�U�����6>�N����i��0��U-O�4%7�^�$�d��2^�z/f���/xT�x������\P�����;|�-/'��l?�t��Ay_s����Z|��2C �� ��U��������^<wA����3�
�����Q{�GS
e[��%t[�E:��1X!�Tj��N'C��^� �6~�JeDo���t�����
��K�������V_�m��>��]����;:��h&h�AA��ViM��x����*����,t��'��
�=��~(r"<eO��2L����lc�h�z>�}�# /7�}G�eKCppcFg����T���1d��"��%�K��[������+����H:���h�Q[C[Z����^�z�c��Oi�c��b%6FD��Yu���z��PZ���
��um�+����I���^Cu�!ON��P�+��];
���@0.������A��?��"1�����~������������ �Q��x�g�C��m���
���Vd��(�������Y� ��� �����v��'���*�I�F��D��oU9�u����s��!��N��{��#��<X]�����u�C��&�4����+MQ��/Y��2���3��a}M��v �����6+�f4]�Y��������yl;C=st�����v�8��� ~�7~
N&{m`�wT�����j�m�����*����>��0���-�#��m�A�tRk�m�X�
���^�!n�/X9S9&XI;6��E-TD�QLp�t�,�C�����$�d�����KK��Xf��cG^���:����n.} �@i��s"X�>�e�����L*5����c�x:�����#�jP�78�������)�o;-�������X���:g�+�����2��XD�@����4j����U�n���r�c�j����t�Je���,��0�<��W�4���-]���������&k|Y������.G=N�������V7�K$w�3��K��� -����R�(~yf8k��Q�C����Tl�OA"L*�Q
U�o��>�$�:��7#SX�������C[*q�Gp|Jg����h����%��wjzU�
��+"�z�������0��O�|
<l]p[0f�!)�7p�l��&�+�e����B��k��:0���S gD{�s������a���+i?�����'
���\.��?��c�)�QL��IM�D�8����)�,��Yb�^�$g�)����Og
y����{�5�x�\��1t��i�L�7f�6�t}� �3��+K+��
���hD�x ���?p��X���Y\�?UE�R7U�m���u�t}���I7�'=`�uXK����0������~�`-~�_�Yu$n��@h���'���i�Y��������d\G��T*�i W���O�[��R�l���O��}�{>h?��~u������_<�y�.���Q��3�'�[��
B~���xw|H��Y���+�]B*}EF�P����7l�oF��������3
��2����,�����2K9��q�����mO9� ����q�������r�����8n{�Hs�S^���p�+�q�jgJ�1���A�!�����a�V� Y���@{�.��q�0����2�����)����L@m�ON���5U
�@�\�Cy��k'�N����E�4�v��e`���G)N�������V|�*�}�|��p��5w�_��cB�S o�y.�s��`����8k!�(�h*����)!R� U�/�V�r�i�(���&;���/��������[����7r���k�Q�����N��@���z�DZ1����se1�H������7j�kj����:�������MK��?��cg��5q����F���q����h�N��v9��8�Qz�����*�D��G9�`s����3�A{�.O�O8�8�����C=�1�z���0f�M��O���� ��rT"���J/3!���rZ'�1 �F6_��Sws�����~rR?|�<�U�E�~��7l8E����g q��<~��Q��k<�P�������6����E�C�R��n
M!���:?Y7��2���6j�%����������gz���~����DQ t�I�5�H$��s��Ux�����p0��"'�GH�0����a({L��B�.���j�{���H3r�V>�.DN��U����� �g����������&7? ��+�����P�#�}����`�C1iD����N�,��
��*"@(:��NZD�F�z��6��O����`}-����$Z�.ytNB��R;���t��uI��N[������{wEN~���,�s�_���U�J��J=9����- c�I��)�{�-��"�S*{3k����ZYNQ=�l�So0�E��y�f6R�fKKlk��<�Z�C9e9i
Y9
�� @MlJ�fk��&�P��K��|��?&�QS=a��&����2����0�D��A��,R�>W�e����#v4s��$�$'r��SK�X}�oZ�W_����d;Q���q�-X�l6`���Jv�{J)�w�u�zY��# ��Y��d�8����n�y�H���Ks;L=|������H��(p���~�c����q�5��S�����o�C\6:#�@�I�=w":l���*�i��5���|q<S����2Bc+�W6p���&Se��4&P��'�6��+4z}�;��s���D���*@��Y��a������z�r���,Y ���)�\��U�o��n��������</ '������� %O���=u�l��Z�� �� 1�f�T�FB����l_��#VE'k�( ����F�I����J�7��fq,5�T��t���B���M5��_������e�"p�.�!G�U&�W���~��2c�ER2� �P]�j�K�<T�Ls�~p������8]�$�qI�������O��L�C�Toa9��*$�@���l���{�� ���
����1^rvk�9���U�Z������9��]dn���>0a���J�,G����;��3�DXSA�H����6����
q#���T:�1�*v�d�X�R�p���wg5��/ �@�&b����H�!^�(
�v������Q���]�g�`G���,�}A����'��Ip��9��0��R�uh+E,M��^���DV<F"&@���g��P������
��rS?q@�W��d�#�3���@G*�������5_�W���D����.��I��/�/�������REJb9��u��HG�"�R��$�$��\���
q�N�F������G��q
��I��K��Y�0�2�n�
*���`�3-���=��R����s\�s�gc���K]� k��B'\m�qI���B�{t���!����������f�u�&���28v"ik��
�~20JLeC�N�r��������������i�@��%��)�D�$�Y�- �#�q`w��S��0��<��d
�vI��L0�<�Vk��#��f�J�S)�X�sHz|���������rCXeM�]�G��"q([������B�(7��s��|��%�����| 3=�����������z1]f����\��$ ��>E�2��%������B�Gp�c#eZ����w��/��<�|�Z=8�����n�}�$gZ��R��p��5�������X���2�
�_���������yp��{�aV�����_v_��yW��RZ� ���PF)���+[:B�[W
,�!���k�}�5���?���m���E�p��l����2�yt�#O�?�������cQ��,m��z�,�7� ��
v�%�K�h=�,
���,�x@c�z�ip�p��k��$�3v��z
�xa�e��}�p�[r
�
<�IWAzj�������#��*G��U�. mC�o���5����r{��6� r�xIc�%q�@���gI���+��Y���2#1.P���T�a�l�p^n���J��\G�L�B(A.����%���p��|��Us��9�XIF���d+��?2�<q����e{�T���������D�7��"ZY�]��X����D����w�z����e��mX���A�8���tUT��h��U�a�k}<o*������Nv�\9`�C��mt2i4��r ���� ����o�D�&���U\6�����I�L��%������N����������_7�2���'S-�3Q2o'�-�p��DN��G��2���K'�o/���t��l��
�������B~d��� ����l9/�j�}X����i_6�E���y������<2��PLL��C|�Ma�Y�}L�"w�Xvd����!��"-����GY'���zX�K
]����f��i�a�2�X��a�DJ 97Y�������pNh������������xfn����v{���5*�����
M3����o�t���}
4��*������%"���)��%�����['j�R��C����lg[�����!@7y�PNS�k��#�lk����%�
VS����# ��}���,9"�OB�� ���L D����B�"u��B�X��<� sz�Le^4V��m����%�[1@rt]n�C���p��:��-�1�_��C�=&r�S%C&-g[#M�Fg'���r���K�����1�5���RN8j��N����>