monitoring CREATE INDEX [CONCURRENTLY]

Started by Alvaro Herreraabout 7 years ago60 messages
#1Alvaro Herrera
alvherre@2ndquadrant.com

Monitoring progress of CREATE INDEX [CONCURRENTLY] is sure to be welcome,
so here's a proposal.

There are three distinct interesting cases. One is straight CREATE
INDEX of a standalone table; then we have CREATE INDEX CONCURRENTLY;
finally, CREATE INDEX on a partitioned table. Note that there's no
CONCURRENTLY for partitioned tables.

A non-concurrent build is a very straightforward: we call create_index,
which does index_build, done. See below for how to report for
index_build, which is the interesting part. I propose not to report
anything else than that for non-concurrent build. There's some
preparatory work that's identical than for CIC (see below). Like
VACUUM, it seems a bit pointless to report an initial phase that's
almost immediate, so I propose we just don't report anything until the
actual index building starts.

CREATE INDEX CONCURRENTLY does these things first, which we would not
report (this is just like VACUUM, which only starts reporting once it
starts scanning blocks):

a. lock rel. No metrics to report.
b. other prep; includes lots of catalog access. Unlikely to lock, but
not impossible. No metrics to report.
c. create_index. CIC skips index_build here, so there's no reason to
report it either.

We would start reporting at this point, with these phases:

1. WaitForLockers 1. Report how many xacts do we need to wait for, how
many are done.
2. index_build. See below.
3. WaitForLockers 2. Report how many xacts do we need to wait for, how
many are done.
4. validate_index. Scans the whole rel again. Report number of blocks
scanned.
5. wait for virtual XIDs. Like WaitForLockers: report how many xacts we
need to wait for, how many are done.

We're done.

(Alternatively, we could have an initial "prep" phase for a/b/c for the
concurrent case and a/b for non-concurrent. I'm just not sure it's
useful.)

index_build
-----------

The actual index building is an AM-specific undertaking, and we report
its progress separately from the AM-agnostic code. That is, each AM has
freedom to define its own list of phases and counters, separate from the
generic code. This avoids the need to provide a new AM method or invoke
callbacks. So when you see that CREATE_INDEX_PHASE is either "index
build" you'll have a separate BTREE_CREATE_PHASE value set to either
"scanning heap" or "sorting" or "building upper layers"; equivalently
for other AMs.

Partitioned indexes
-------------------

For partitioned indexes, we only have the index build phase, but we
repeat it for each partition. In addition to the index_build metrics
described above, we should report how many partitions we need to handle
in total and how many partitions are already done. (I'm avoiding
getting in the trouble of reporting *which* partition we're currently
handling and have already handled.)

Thoughts?

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#2Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Alvaro Herrera (#1)
1 attachment(s)
Re: monitoring CREATE INDEX [CONCURRENTLY]

For discussion, here's an preliminary patch. This is just a first
skeleton; needs to grow a lot of flesh yet, per my previous proposal.
As far as the generic CREATE INDEX stuff goes, I think this is complete;
it's missing the AM-specific bits.

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attachments:

create-index-progress.patchtext/x-diff; charset=us-asciiDownload
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index c2ad944e04..f4cb28c6d6 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -1590,7 +1590,7 @@ index_drop(Oid indexId, bool concurrent)
 		 * to acquire an exclusive lock on our table.  The lock code will
 		 * detect deadlock and error out properly.
 		 */
-		WaitForLockers(heaplocktag, AccessExclusiveLock);
+		WaitForLockers(heaplocktag, AccessExclusiveLock, true);
 
 		/*
 		 * No more predicate locks will be acquired on this index, and we're
@@ -1634,7 +1634,7 @@ index_drop(Oid indexId, bool concurrent)
 		 * Wait till every transaction that saw the old index state has
 		 * finished.
 		 */
-		WaitForLockers(heaplocktag, AccessExclusiveLock);
+		WaitForLockers(heaplocktag, AccessExclusiveLock, true);
 
 		/*
 		 * Re-open relations to allow us to complete our actions.
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 5253837b54..921a84eb45 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -904,6 +904,24 @@ CREATE VIEW pg_stat_progress_vacuum AS
     FROM pg_stat_get_progress_info('VACUUM') AS S
 		LEFT JOIN pg_database D ON S.datid = D.oid;
 
+CREATE VIEW pg_stat_progress_create_index AS
+	SELECT
+		s.pid AS pid, S.datid AS datid, D.datname AS datname,
+		S.relid AS relid,
+		CASE s.param1 WHEN 0 THEN 'initializing'
+					  WHEN 1 THEN 'waiting for lockers 1'
+					  WHEN 2 THEN 'building index'
+					  WHEN 3 THEN 'waiting for lockers 2'
+					  WHEN 4 THEN 'validating index'
+					  WHEN 5 THEN 'waiting for lockers 3'
+					  END as phase,
+		S.param2 AS procs_to_wait_for,
+		S.param3 AS procs_waited_for,
+		S.param4 AS partitions_to_build,
+		S.param5 AS partitions_built
+	FROM pg_stat_get_progress_info('CREATE INDEX') AS S
+		LEFT JOIN pg_database D ON S.datid = D.oid;
+
 CREATE VIEW pg_user_mappings AS
     SELECT
         U.oid       AS umid,
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 816c73a36a..03321c2dc4 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -35,6 +35,7 @@
 #include "commands/dbcommands.h"
 #include "commands/defrem.h"
 #include "commands/event_trigger.h"
+#include "commands/progress.h"
 #include "commands/tablecmds.h"
 #include "commands/tablespace.h"
 #include "mb/pg_wchar.h"
@@ -47,6 +48,7 @@
 #include "parser/parse_coerce.h"
 #include "parser/parse_func.h"
 #include "parser/parse_oper.h"
+#include "pgstat.h"
 #include "rewrite/rewriteManip.h"
 #include "storage/lmgr.h"
 #include "storage/proc.h"
@@ -370,6 +372,15 @@ DefineIndex(Oid relationId,
 	Snapshot	snapshot;
 	int			i;
 
+
+	/*
+	 * Start progress report.  If we're building a partition, this was already
+	 * done.
+	 */
+	if (!OidIsValid(parentIndexId))
+		pgstat_progress_start_command(PROGRESS_COMMAND_CREATE_INDEX,
+									  relationId);
+
 	/*
 	 * count key attributes in index
 	 */
@@ -866,6 +877,11 @@ DefineIndex(Oid relationId,
 	if (!OidIsValid(indexRelationId))
 	{
 		heap_close(rel, NoLock);
+
+		/* If this is the top-level index, we're done */
+		if (!OidIsValid(parentIndexId))
+			pgstat_progress_end_command();
+
 		return address;
 	}
 
@@ -891,6 +907,9 @@ DefineIndex(Oid relationId,
 			TupleDesc	parentDesc;
 			Oid		   *opfamOids;
 
+			pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL,
+										 nparts);
+
 			memcpy(part_oids, partdesc->oids, sizeof(Oid) * nparts);
 
 			parentDesc = CreateTupleDescCopy(RelationGetDescr(rel));
@@ -1040,6 +1059,8 @@ DefineIndex(Oid relationId,
 								skip_build, quiet);
 				}
 
+				pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_DONE,
+											 i + 1);
 				pfree(attmap);
 			}
 
@@ -1074,6 +1095,8 @@ DefineIndex(Oid relationId,
 		 * Indexes on partitioned tables are not themselves built, so we're
 		 * done here.
 		 */
+		if (!OidIsValid(parentIndexId))
+			pgstat_progress_end_command();
 		return address;
 	}
 
@@ -1081,6 +1104,11 @@ DefineIndex(Oid relationId,
 	{
 		/* Close the heap and we're done, in the non-concurrent case */
 		heap_close(rel, NoLock);
+
+		/* If this is the top-level index, we're done. */
+		if (!OidIsValid(parentIndexId))
+			pgstat_progress_end_command();
+
 		return address;
 	}
 
@@ -1132,7 +1160,9 @@ DefineIndex(Oid relationId,
 	 * exclusive lock on our table.  The lock code will detect deadlock and
 	 * error out properly.
 	 */
-	WaitForLockers(heaplocktag, ShareLock);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_1);
+	WaitForLockers(heaplocktag, ShareLock, true);
 
 	/*
 	 * At this moment we are sure that there are no transactions with the
@@ -1168,6 +1198,8 @@ DefineIndex(Oid relationId,
 	indexInfo->ii_BrokenHotChain = false;
 
 	/* Now build the index */
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_BUILD);
 	index_build(rel, indexRelation, indexInfo, stmt->primary, false, true);
 
 	/* Close both the relations, but keep the locks */
@@ -1196,7 +1228,9 @@ DefineIndex(Oid relationId,
 	 * We once again wait until no transaction can have the table open with
 	 * the index marked as read-only for updates.
 	 */
-	WaitForLockers(heaplocktag, ShareLock);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_2);
+	WaitForLockers(heaplocktag, ShareLock, true);
 
 	/*
 	 * Now take the "reference snapshot" that will be used by validate_index()
@@ -1219,6 +1253,8 @@ DefineIndex(Oid relationId,
 	/*
 	 * Scan the index and the heap, insert any missing index entries.
 	 */
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_VALIDATE);
 	validate_index(relationId, indexRelationId, snapshot);
 
 	/*
@@ -1282,6 +1318,9 @@ DefineIndex(Oid relationId,
 	old_snapshots = GetCurrentVirtualXIDs(limitXmin, true, false,
 										  PROC_IS_AUTOVACUUM | PROC_IN_VACUUM,
 										  &n_old_snapshots);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_3);
+	pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, n_old_snapshots);
 
 	for (i = 0; i < n_old_snapshots; i++)
 	{
@@ -1318,6 +1357,8 @@ DefineIndex(Oid relationId,
 
 		if (VirtualTransactionIdIsValid(old_snapshots[i]))
 			VirtualXactLock(old_snapshots[i], true);
+
+		pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, i);
 	}
 
 	/*
@@ -1340,6 +1381,8 @@ DefineIndex(Oid relationId,
 	 */
 	UnlockRelationIdForSession(&heaprelid, ShareUpdateExclusiveLock);
 
+	pgstat_progress_end_command();
+
 	return address;
 }
 
diff --git a/src/backend/storage/ipc/standby.c b/src/backend/storage/ipc/standby.c
index c9bb3e987d..f151a07ec1 100644
--- a/src/backend/storage/ipc/standby.c
+++ b/src/backend/storage/ipc/standby.c
@@ -401,7 +401,7 @@ ResolveRecoveryConflictWithLock(LOCKTAG locktag)
 		 */
 		VirtualTransactionId *backends;
 
-		backends = GetLockConflicts(&locktag, AccessExclusiveLock);
+		backends = GetLockConflicts(&locktag, AccessExclusiveLock, NULL);
 		ResolveRecoveryConflictWithVirtualXIDs(backends,
 											   PROCSIG_RECOVERY_CONFLICT_LOCK);
 	}
diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c
index 3f57507bce..144ccd7ffd 100644
--- a/src/backend/storage/lmgr/lmgr.c
+++ b/src/backend/storage/lmgr/lmgr.c
@@ -19,7 +19,9 @@
 #include "access/transam.h"
 #include "access/xact.h"
 #include "catalog/catalog.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
+#include "pgstat.h"
 #include "storage/lmgr.h"
 #include "storage/procarray.h"
 #include "utils/inval.h"
@@ -857,10 +859,12 @@ XactLockTableWaitErrorCb(void *arg)
  * after we obtained our initial list of lockers, we will not wait for them.
  */
 void
-WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
+WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
 {
 	List	   *holders = NIL;
 	ListCell   *lc;
+	int			total = 0;
+	int			done = 0;
 
 	/* Done if no locks to wait for */
 	if (list_length(locktags) == 0)
@@ -870,10 +874,16 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
 	foreach(lc, locktags)
 	{
 		LOCKTAG    *locktag = lfirst(lc);
+		int			count;
 
-		holders = lappend(holders, GetLockConflicts(locktag, lockmode));
+		holders = lappend(holders,
+						  GetLockConflicts(locktag, lockmode, &count));
+		total += count;
 	}
 
+	if (progress)
+		pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, total);
+
 	/*
 	 * Note: GetLockConflicts() never reports our own xid, hence we need not
 	 * check for that.  Also, prepared xacts are not reported, which is fine
@@ -889,6 +899,9 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
 		{
 			VirtualXactLock(*lockholders, true);
 			lockholders++;
+
+			if (progress)
+				pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, ++done);
 		}
 	}
 
@@ -901,12 +914,12 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
  * Same as WaitForLockersMultiple, for a single lock tag.
  */
 void
-WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode)
+WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress)
 {
 	List	   *l;
 
 	l = list_make1(&heaplocktag);
-	WaitForLockersMultiple(l, lockmode);
+	WaitForLockersMultiple(l, lockmode, progress);
 	list_free(l);
 }
 
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 10f6f60f1e..3034fb3f27 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2807,6 +2807,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock)
  *		xacts merely awaiting such a lock are NOT reported.
  *
  * The result array is palloc'd and is terminated with an invalid VXID.
+ * *ocount, if not null, is updated to the number of items set.
  *
  * Of course, the result could be out of date by the time it's returned,
  * so use of this function has to be thought about carefully.
@@ -2817,7 +2818,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock)
  * uses of the result.
  */
 VirtualTransactionId *
-GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode)
+GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *ocount)
 {
 	static VirtualTransactionId *vxids;
 	LOCKMETHODID lockmethodid = locktag->locktag_lockmethodid;
@@ -2964,6 +2965,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode)
 		LWLockRelease(partitionLock);
 		vxids[count].backendId = InvalidBackendId;
 		vxids[count].localTransactionId = InvalidLocalTransactionId;
+		if (ocount)
+			*ocount = count;
 		return vxids;
 	}
 
@@ -3019,6 +3022,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode)
 
 	vxids[count].backendId = InvalidBackendId;
 	vxids[count].localTransactionId = InvalidLocalTransactionId;
+	if (ocount)
+		*ocount = count;
 	return vxids;
 }
 
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index f955f1912a..7c93efb362 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -468,6 +468,8 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
 	/* Translate command name into command type code. */
 	if (pg_strcasecmp(cmd, "VACUUM") == 0)
 		cmdtype = PROGRESS_COMMAND_VACUUM;
+	else if (pg_strcasecmp(cmd, "CREATE INDEX") == 0)
+		cmdtype = PROGRESS_COMMAND_CREATE_INDEX;
 	else
 		ereport(ERROR,
 				(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index 6a6b467fee..f02fedf3ad 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -34,4 +34,22 @@
 #define PROGRESS_VACUUM_PHASE_TRUNCATE			5
 #define PROGRESS_VACUUM_PHASE_FINAL_CLEANUP		6
 
+
+/* Progress parameters for CREATE INDEX */
+#define PROGRESS_CREATEIDX_PHASE				0
+/* 1 and 2 reserved for "waitfor" metrics */
+#define PROGRESS_CREATEIDX_PARTITIONS_TOTAL		3
+#define PROGRESS_CREATEIDX_PARTITIONS_DONE		4
+
+/* Phases of CREATE INDEX */
+#define PROGRESS_CREATEIDX_PHASE_WAIT_1			1
+#define PROGRESS_CREATEIDX_PHASE_BUILD			2
+#define PROGRESS_CREATEIDX_PHASE_WAIT_2			3
+#define PROGRESS_CREATEIDX_PHASE_VALIDATE		4
+#define PROGRESS_CREATEIDX_PHASE_WAIT_3			5
+
+/* Lock holder wait counts */
+#define PROGRESS_WAITFOR_TOTAL					1
+#define PROGRESS_WAITFOR_DONE					2
+
 #endif
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index f1c10d16b8..9542bf024f 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -934,7 +934,8 @@ typedef enum
 typedef enum ProgressCommandType
 {
 	PROGRESS_COMMAND_INVALID,
-	PROGRESS_COMMAND_VACUUM
+	PROGRESS_COMMAND_VACUUM,
+	PROGRESS_COMMAND_CREATE_INDEX
 } ProgressCommandType;
 
 #define PGSTAT_NUM_PROGRESS_PARAM	10
diff --git a/src/include/storage/lmgr.h b/src/include/storage/lmgr.h
index e5356b7d54..ee1070f321 100644
--- a/src/include/storage/lmgr.h
+++ b/src/include/storage/lmgr.h
@@ -78,8 +78,8 @@ extern void XactLockTableWait(TransactionId xid, Relation rel,
 extern bool ConditionalXactLockTableWait(TransactionId xid);
 
 /* Lock VXIDs, specified by conflicting locktags */
-extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode);
-extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode);
+extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress);
+extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress);
 
 /* Lock an XID for tuple insertion (used to wait for an insertion to finish) */
 extern uint32 SpeculativeInsertionLockAcquire(TransactionId xid);
diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h
index a37fda7b63..a0bfc670b6 100644
--- a/src/include/storage/lock.h
+++ b/src/include/storage/lock.h
@@ -544,7 +544,7 @@ extern bool LockHeldByMe(const LOCKTAG *locktag, LOCKMODE lockmode);
 extern bool LockHasWaiters(const LOCKTAG *locktag,
 			   LOCKMODE lockmode, bool sessionLock);
 extern VirtualTransactionId *GetLockConflicts(const LOCKTAG *locktag,
-				 LOCKMODE lockmode);
+				 LOCKMODE lockmode, int *ocount);
 extern void AtPrepare_Locks(void);
 extern void PostPrepare_Locks(TransactionId xid);
 extern int LockCheckConflicts(LockMethod lockMethodTable,
#3Pavan Deolasee
pavan.deolasee@gmail.com
In reply to: Alvaro Herrera (#2)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On Tue, Jan 1, 2019 at 6:09 AM Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:

For discussion, here's an preliminary patch. This is just a first
skeleton; needs to grow a lot of flesh yet, per my previous proposal.
As far as the generic CREATE INDEX stuff goes, I think this is complete;
it's missing the AM-specific bits.

Looks like it's missing the validate_index blocks-scanned report, which is
not AM-specific and your original proposal does mention that.

I thought a bit about index_build part. If most AMs follow a somewhat
standard phases while building an index, it might be simpler to define
those phases and have AM agnostic code report those phases. Or may be just
report the most significant information, instead of reporting each
sub-phase of index_build. I think the most important progress to know would
be how far the heap is scanned for to-be-indexed tuples. AFAICS all AMs
use IndexBuildHeapScan() to scan the heap. Can we simply do some reporting
from that routine? Like number of blocks scanned against the total number
of blocks requested?

Some minor comments on the patch, though I suspect you might be already
updating the patch since you marked it as WIP.

+CREATE VIEW pg_stat_progress_create_index AS
+ SELECT
+ s.pid AS pid, S.datid AS datid, D.datname AS datname,
+ S.relid AS relid,
+ CASE s.param1 WHEN 0 THEN 'initializing'
+   WHEN 1 THEN 'waiting for lockers 1'
+   WHEN 2 THEN 'building index'
+   WHEN 3 THEN 'waiting for lockers 2'
+   WHEN 4 THEN 'validating index'
+   WHEN 5 THEN 'waiting for lockers 3'

Can we have more descriptive text for waiters? Such as "waiting for existing
writers", "waiting for intermediate writers" and "waiting for old readers".
Not
sure if I got those correct, something of that sort will definitely give
more
insight into what the transaction is waiting for.

Can we actually also report the list of transactions the command is waiting
on?
That could be useful to the user if CIC appears to be stuck too long.

+ pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL,
+ nparts);
+

IMHO we should just use full term INDEX instead of IDX, such as
PROGRESS_CREATE_INDEX_PARTITIONS_TOTAL. It's already a long name, so couple
of
extra characters won't make a difference. I did not see much precedence to
shortern to IDX for INDEX elsewhere in the code (though we tend to do that
for
variable names etc).

@@ -1282,6 +1318,9 @@ DefineIndex(Oid relationId,
  old_snapshots = GetCurrentVirtualXIDs(limitXmin, true, false,
    PROC_IS_AUTOVACUUM | PROC_IN_VACUUM,
    &n_old_snapshots);
+ pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+ PROGRESS_CREATEIDX_PHASE_WAIT_3);
+ pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, n_old_snapshots);

I think we should clear the PROGRESS_WAITFOR_TOTAL and PROGRESS_WAITFOR_DONE
when the wait phase is over, to avoid any confusion. For example, I noticed
that the counters from WAIT_1 are reported as-is if WAIT_2 had no lockers.

Shouldn't PROGRESS_WAITFOR_DONE be updated when we skip a snapshot in the
code below?
if (!VirtualTransactionIdIsValid(old_snapshots[i]))
continue; /* found uninteresting in previous cycle */

@@ -2817,7 +2818,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock)
  * uses of the result.
  */
 VirtualTransactionId *
-GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode)
+GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *ocount)

Could that out variable be named something differently? "countp" or
something
like that. I did not check if there is some practice that we follow, but I
remember suffixing with "p" rather than prefixing with "o" (for out I
assume)

+
+/* Progress parameters for CREATE INDEX */
+#define PROGRESS_CREATEIDX_PHASE 0
+/* 1 and 2 reserved for "waitfor" metrics */
+#define PROGRESS_CREATEIDX_PARTITIONS_TOTAL 3
+#define PROGRESS_CREATEIDX_PARTITIONS_DONE 4
+

Is there a reason to leave those reserve placeholders, only to fill them a
few
lines down?

Thanks,
Pavan

--
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

#4Rahila Syed
rahilasyed90@gmail.com
In reply to: Alvaro Herrera (#1)
Re: monitoring CREATE INDEX [CONCURRENTLY]

Hi Alvaro,

The WIP patch needs a rebase. Please see few in-line comments.

On Fri, Dec 21, 2018 at 3:30 AM Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:

Monitoring progress of CREATE INDEX [CONCURRENTLY] is sure to be welcome,
so here's a proposal.

There are three distinct interesting cases. One is straight CREATE
INDEX of a standalone table; then we have CREATE INDEX CONCURRENTLY;
finally, CREATE INDEX on a partitioned table. Note that there's no
CONCURRENTLY for partitioned tables.

A non-concurrent build is a very straightforward: we call create_index,
which does index_build, done. See below for how to report for
index_build, which is the interesting part. I propose not to report
anything else than that for non-concurrent build. There's some
preparatory work that's identical than for CIC (see below). Like
VACUUM, it seems a bit pointless to report an initial phase that's
almost immediate, so I propose we just don't report anything until the
actual index building starts.

Aren't we reporting this initial preparatory work in the form of
'initializing' phase that you
have in current patch? IIUC, there are no metrics but the name of the phase.

CREATE INDEX CONCURRENTLY does these things first, which we would not

report (this is just like VACUUM, which only starts reporting once it
starts scanning blocks):

a. lock rel. No metrics to report.
b. other prep; includes lots of catalog access. Unlikely to lock, but
not impossible. No metrics to report.
c. create_index. CIC skips index_build here, so there's no reason to
report it either.

We would start reporting at this point, with these phases:

1. WaitForLockers 1. Report how many xacts do we need to wait for, how
many are done.
2. index_build. See below.
3. WaitForLockers 2. Report how many xacts do we need to wait for, how
many are done.
4. validate_index. Scans the whole rel again. Report number of blocks
scanned.
5. wait for virtual XIDs. Like WaitForLockers: report how many xacts we
need to wait for, how many are done.

We're done.

(Alternatively, we could have an initial "prep" phase for a/b/c for the
concurrent case and a/b for non-concurrent. I'm just not sure it's
useful.)

index_build
-----------

The actual index building is an AM-specific undertaking, and we report
its progress separately from the AM-agnostic code. That is, each AM has
freedom to define its own list of phases and counters, separate from the
generic code. This avoids the need to provide a new AM method or invoke
callbacks. So when you see that CREATE_INDEX_PHASE is either "index
build" you'll have a separate BTREE_CREATE_PHASE value set to either
"scanning heap" or "sorting" or "building upper layers"; equivalently
for other AMs.

OK.

I think the main phases in which index_build for most AMs can be divided is
as follows:
1. Scanning heap tuples for building index which is common for all AMs
2. Forming index tuples which is AM-specific
3. Writing tuples into the index which is AM-specific.
Out of these, metrics for phase 1 can be heap_tuples_scanned /
total_heap_tuples and
for phase 3, it can be index_tuples_computed/ total_index_tuples.
I am not sure about metrics for phase 2 especially for Btree which involves
reporting progress of sorting.

Partitioned indexes

-------------------

For partitioned indexes, we only have the index build phase, but we
repeat it for each partition. In addition to the index_build metrics
described above, we should report how many partitions we need to handle
in total and how many partitions are already done. (I'm avoiding
getting in the trouble of reporting *which* partition we're currently
handling and have already handled.)

Thoughts?

+CREATE VIEW pg_stat_progress_create_index AS

+ SELECT
+ s.pid AS pid, S.datid AS datid, D.datname AS datname,
+ S.relid AS relid,
+ CASE s.param1 WHEN 0 THEN 'initializing'
+   WHEN 1 THEN 'waiting for lockers 1'
+   WHEN 2 THEN 'building index'
+   WHEN 3 THEN 'waiting for lockers 2'
+   WHEN 4 THEN 'validating index'
+   WHEN 5 THEN 'waiting for lockers 3'
+   END as phase,
+ S.param2 AS procs_to_wait_for,
+ S.param3 AS procs_waited_for,
+ S.param4 AS partitions_to_build,
+ S.param5 AS partitions_built
+ FROM pg_stat_get_progress_info('CREATE INDEX') AS S
+ LEFT JOIN pg_database D ON S.datid = D.oid;
+

1. In the above code, I think it will be useful if we report phases as
'Initializing phase 1 of n'
'Waiting for lockers phase 2 of n' etc. i.e reporting total number of
phases as well.

+               holders = lappend(holders,
+                                                 GetLockConflicts(locktag,
lockmode, &count));
+               total += count;
2. IIUC, the above code in WaitForLockersMultiple can be written under
if(progress) condition like rest of the progress checking code in that
function
and pass NULL for count otherwise.

3. Will it be useful to report pid's of the backend's
for the transactions which CREATE INDEX concurrently is waiting for?
I think it can be used to debug long running transactions.

4. Should we have additional statistics update phase before
index_update_stats
like PROGRESS_VACUUM_PHASE_FINAL_CLEANUP?

5. I think it may be useful to report number of parallel workers requested
and number of workers
actually running index build in case of btree.

6. Also, this can be reported as an additional validation phase for
exclusion constraint
in index_build function as it involves scanning all live tuples of heap
relation.

/*
* If it's for an exclusion constraint, make a second pass over the
heap
* to verify that the constraint is satisfied. We must not do this
until
* the index is fully valid. (Broken HOT chains shouldn't matter,
though;
* see comments for IndexCheckExclusion.)
*/
if (indexInfo->ii_ExclusionOps != NULL)
IndexCheckExclusion(heapRelation, indexRelation, indexInfo);
*/

Thank you,
Rahila Syed

#5Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Pavan Deolasee (#3)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On 2019-Jan-09, Pavan Deolasee wrote:

Looks like it's missing the validate_index blocks-scanned report, which is
not AM-specific and your original proposal does mention that.

True. That phase is actually 3 phases, which would be reported
separately:

5. index_bulk_delete() scan
6. performsort
7. validate_index_heapscan

I thought a bit about index_build part. If most AMs follow a somewhat
standard phases while building an index, it might be simpler to define
those phases and have AM agnostic code report those phases.

Well, think about btrees. We first scan the whole table putting all
tuples in a spool (two spools actually), then we tell the spools to get
sorted, then we extract tuples from the spools, and finally we read the
spool to produce the leaf pages. If we just report the table scan, the
reports gets to 100% complete in that phase and then waits for a very
long time during which nothing seems to happen. That's not cool.

I'm adding a new AM support routine which turns the subphase number into
a textual description, so that we don't have to hardcode phase names in
the agnostic code.

Can we have more descriptive text for waiters? Such as "waiting for
existing writers", "waiting for intermediate writers" and "waiting for
old readers". Not sure if I got those correct, something of that sort
will definitely give more insight into what the transaction is waiting
for.

Can do.

Can we actually also report the list of transactions the command is waiting
on?
That could be useful to the user if CIC appears to be stuck too long.

I'm afraid this is not possible, because the progress report interface
doesn't allow for something like that.

IMHO we should just use full term INDEX instead of IDX, such as
PROGRESS_CREATE_INDEX_PARTITIONS_TOTAL. It's already a long name, so couple
of extra characters won't make a difference.

Really? I though it was clear enough and it's *three* characters saved
even.

I think we should clear the PROGRESS_WAITFOR_TOTAL and PROGRESS_WAITFOR_DONE
when the wait phase is over, to avoid any confusion. For example, I noticed
that the counters from WAIT_1 are reported as-is if WAIT_2 had no lockers.

Yes, absolutely.

+GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *ocount)

Could that out variable be named something differently? "countp" or
something like that. I did not check if there is some practice that we
follow, but I remember suffixing with "p" rather than prefixing with
"o" (for out I assume)

Sure.

+/* Progress parameters for CREATE INDEX */
+#define PROGRESS_CREATEIDX_PHASE 0
+/* 1 and 2 reserved for "waitfor" metrics */
+#define PROGRESS_CREATEIDX_PARTITIONS_TOTAL 3
+#define PROGRESS_CREATEIDX_PARTITIONS_DONE 4
+

Is there a reason to leave those reserve placeholders, only to fill them a
few lines down?

Yes -- those are going to be used by other reports that also use similar
metrics, such as the CLUSTER report.

I'm running out of columns to put the numbers into :-( Right now I have

1. phase
2. subphase (for index_build)
3. lockers total (to wait for)
4. lockers done
5. blocks total
6. blocks done
7. tapes total
8. tapes done
9. partitions total
10. partitions done

The "tapes total/done" bit is about reporting the performsort steps; I'm
not really sure yet it'll be tapes, but I hope it can be done with two
numbers. Anyway, in btree build I have these subphases

1. spool heapscan using IndexBuildHeapScan
2. _bt_parallel_heapscan stuff (only one of these in a build)
3. bt_leafbuild, performsort spool 1
4. bt_leafbuild, performsort spool 2
5. bt_load

and I don't have columns to put the metrics for the btload thing,
assuming I figure out what those are.

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#6Michael Paquier
michael@paquier.xyz
In reply to: Alvaro Herrera (#5)
Re: monitoring CREATE INDEX [CONCURRENTLY]

Hi all,

Based on the latest emails exchanged, the patch got some feedback from
Pavan which has not been addressed. So I am marking the patch as
returned with feedback.
--
Michael

#7Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Rahila Syed (#4)
1 attachment(s)
Re: monitoring CREATE INDEX [CONCURRENTLY]

Hi Rahila, Pavan,

Thanks for the review. I incorporated some fixes per your comments.
More fixes are needed still. That said, the patch in attachment gives
good insight into how I think this should turn out.

index_build
-----------

OK.
I think the main phases in which index_build for most AMs can be divided is
as follows:
[...]

I ended up defining phases for the index_build phase in the AM itself;
the code reports a phase number using the regular API, and then the
pgstat_progress view obtains the name of each phase using a support
method.

For btree, I ended up not reporting anything about the sort per se; we
just scan the heap (reporting block numbers, which is easy because we
know the heap size in advance) and count heap tuples while scanning;
once that's done, we know how many tuples we need to write out to the
index, so that total becomes the next stage's target total. That's
simpler. (It is wrong for partial indexes currently, though.)

So for btree we have this:

/*
* btphasename() -- Return name of phase, for index build progress report
*/
char *
btphasename(int64 phasenum)
{
switch (phasenum)
{
case PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE:
return "initializing (1/5)";
case PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN:
return "table scan (2/5)";
case PROGRESS_BTREE_PHASE_PERFORMSORT_1:
return "sorting tuples, spool 1 (3/5)";
case PROGRESS_BTREE_PHASE_PERFORMSORT_2:
return "sorting tuples, spool 2 (4/5)";
case PROGRESS_BTREE_PHASE_LEAF_LOAD:
return "final btree sort & load (5/5)";
default:
return NULL;
}
}

Now this is a bit strange, because the report looks like this:

phase | building index (3 of 8): initializing (1/5)
[...]
blocks total | 442478
blocks done | 3267

So for phase 3, we always have phase and subphase counters in the phase
name. However, I don't think there's any clean way to know from the
very beginning how many subphases are there going to be, and it would be
even more confusing to have the total "of N" number vary each time
depending on the access method. So this leaves the phase counter going
from 1 to 8, and then for phase 3 you have a second part that goes from
1 to N.

Anyway, at some point it completes the heap scan, and the phase changes
to this:

phase | building index (3 of 8): heap scan(2/5)

I think I should take Rahila's suggestion to report the number of
workers running (but I'm not sure I see the point in reporting number of
workers planned).

The heap scan quickly gives way to the next one,

phase | building index (3 of 8): final btree sort & load (5/5)
[...]
tuples total | 100000000
tuples done | 58103713

Finally,
phase | validating index scan (phase 5 of 8)
and
phase | validate index heapscan (phase 7 of 8)

and then it's over.

Now, it's clear that I haven't yet nailed all the progress updates
correctly, because there are some stalls where nothing seems to be
happening. The final index_validate phases have been ignored so far.

1. In the above code, I think it will be useful if we report phases as
'Initializing phase 1 of n'
'Waiting for lockers phase 2 of n' etc. i.e reporting total number of
phases as well.

Great idea, done.

2. IIUC, the above code in WaitForLockersMultiple can be written under
if(progress) condition like rest of the progress checking code in that
function
and pass NULL for count otherwise.

Yep, good point.

3. Will it be useful to report pid's of the backend's
for the transactions which CREATE INDEX concurrently is waiting for?
I think it can be used to debug long running transactions.

Done.

4. Should we have additional statistics update phase before
index_update_stats
like PROGRESS_VACUUM_PHASE_FINAL_CLEANUP?

Not sure about this one ... it's supposed to be a fairly quick
operation.

5. I think it may be useful to report number of parallel workers requested
and number of workers
actually running index build in case of btree.

True, I haven't done this one. I'll add it next.

6. Also, this can be reported as an additional validation phase for
exclusion constraint
in index_build function as it involves scanning all live tuples of heap
relation.

/*
* If it's for an exclusion constraint, make a second pass over the
heap
* to verify that the constraint is satisfied. We must not do this
until
* the index is fully valid. (Broken HOT chains shouldn't matter,
though;
* see comments for IndexCheckExclusion.)
*/
if (indexInfo->ii_ExclusionOps != NULL)
IndexCheckExclusion(heapRelation, indexRelation, indexInfo);
*/

Hmm, I haven't looked into exclusion constraints thus far ... I suppose
this is a critical point to keep in mind.

Thanks for the review!

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attachments:

v2-create-index-progress.patchtext/x-diff; charset=us-asciiDownload
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index 964200a7678..99f6ed6bc44 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -534,7 +534,7 @@ bt_check_every_level(Relation rel, Relation heaprel, bool readonly,
 			 RelationGetRelationName(state->rel),
 			 RelationGetRelationName(state->heaprel));
 
-		IndexBuildHeapScan(state->heaprel, state->rel, indexinfo, true,
+		IndexBuildHeapScan(state->heaprel, state->rel, indexinfo, true, false,
 						   bt_tuple_present_callback, (void *) state, scan);
 
 		ereport(DEBUG1,
diff --git a/contrib/bloom/blinsert.c b/contrib/bloom/blinsert.c
index e43fbe0005f..947ee74881f 100644
--- a/contrib/bloom/blinsert.c
+++ b/contrib/bloom/blinsert.c
@@ -141,7 +141,7 @@ blbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	initCachedPage(&buildstate);
 
 	/* Do the heap scan */
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false,
 								   bloomBuildCallback, (void *) &buildstate,
 								   NULL);
 
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 64583765787..79fe413f994 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -132,6 +132,7 @@ blhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = blcostestimate;
 	amroutine->amoptions = bloptions;
 	amroutine->amproperty = NULL;
+	amroutine->amphasename = NULL;
 	amroutine->amvalidate = blvalidate;
 	amroutine->ambeginscan = blbeginscan;
 	amroutine->amrescan = blrescan;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index 05102724ead..189179a5667 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -127,6 +127,7 @@ typedef struct IndexAmRoutine
     amcostestimate_function amcostestimate;
     amoptions_function amoptions;
     amproperty_function amproperty;     /* can be NULL */
+    amphasename_function amphasename;   /* can be NULL */
     amvalidate_function amvalidate;
     ambeginscan_function ambeginscan;
     amrescan_function amrescan;
@@ -468,6 +469,16 @@ amproperty (Oid index_oid, int attno,
 
   <para>
 <programlisting>
+char *
+amphasename (int64 phasenum);
+</programlisting>
+   Return the textual name of the given phase number.  The phase numbers are
+   those reported during an index build via the
+   <function>pgstat_progress_update_param</function> interface.
+  </para>
+
+  <para>
+<programlisting>
 bool
 amvalidate (Oid opclassoid);
 </programlisting>
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 8f008dd0080..547a67fd2e6 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -111,6 +111,7 @@ brinhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = brincostestimate;
 	amroutine->amoptions = brinoptions;
 	amroutine->amproperty = NULL;
+	amroutine->amphasename = NULL;
 	amroutine->amvalidate = brinvalidate;
 	amroutine->ambeginscan = brinbeginscan;
 	amroutine->amrescan = brinrescan;
@@ -718,7 +719,7 @@ brinbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	 * Now scan the relation.  No syncscan allowed here because we want the
 	 * heap blocks in physical order.
 	 */
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, false,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, false, false,
 								   brinbuildCallback, (void *) state, NULL);
 
 	/* process the final batch */
@@ -1234,7 +1235,7 @@ summarize_range(IndexInfo *indexInfo, BrinBuildState *state, Relation heapRel,
 	 * by transactions that are still in progress, among other corner cases.
 	 */
 	state->bs_currRangeStart = heapBlk;
-	IndexBuildHeapRangeScan(heapRel, state->bs_irel, indexInfo, false, true,
+	IndexBuildHeapRangeScan(heapRel, state->bs_irel, indexInfo, false, true, false,
 							heapBlk, scanNumBlks,
 							brinbuildCallback, (void *) state, NULL);
 
diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c
index 524ac5be8b5..838de4c1ec3 100644
--- a/src/backend/access/gin/gininsert.c
+++ b/src/backend/access/gin/gininsert.c
@@ -394,7 +394,7 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	 * Do the heap scan.  We disallow sync scan here because dataPlaceToPage
 	 * prefers to receive tuples in TID order.
 	 */
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, false,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, false, false,
 								   ginBuildCallback, (void *) &buildstate, NULL);
 
 	/* dump remaining entries to the index */
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index afc20232ace..240f8df5ad8 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -64,6 +64,7 @@ ginhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = gincostestimate;
 	amroutine->amoptions = ginoptions;
 	amroutine->amproperty = NULL;
+	amroutine->amphasename = NULL;
 	amroutine->amvalidate = ginvalidate;
 	amroutine->ambeginscan = ginbeginscan;
 	amroutine->amrescan = ginrescan;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index b75b3a8dacd..8839b24d7ac 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = gistcostestimate;
 	amroutine->amoptions = gistoptions;
 	amroutine->amproperty = gistproperty;
+	amroutine->amphasename = NULL;
 	amroutine->amvalidate = gistvalidate;
 	amroutine->ambeginscan = gistbeginscan;
 	amroutine->amrescan = gistrescan;
diff --git a/src/backend/access/gist/gistbuild.c b/src/backend/access/gist/gistbuild.c
index bd142a3560d..015f874cc93 100644
--- a/src/backend/access/gist/gistbuild.c
+++ b/src/backend/access/gist/gistbuild.c
@@ -204,7 +204,7 @@ gistbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	/*
 	 * Do the heap scan.
 	 */
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false,
 								   gistBuildCallback, (void *) &buildstate, NULL);
 
 	/*
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index f1f01a0956d..b661daed381 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -82,6 +82,7 @@ hashhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = hashcostestimate;
 	amroutine->amoptions = hashoptions;
 	amroutine->amproperty = NULL;
+	amroutine->amphasename = NULL;
 	amroutine->amvalidate = hashvalidate;
 	amroutine->ambeginscan = hashbeginscan;
 	amroutine->amrescan = hashrescan;
@@ -159,7 +160,7 @@ hashbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	buildstate.heapRel = heap;
 
 	/* do the heap scan */
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false,
 								   hashbuildCallback, (void *) &buildstate, NULL);
 
 	if (buildstate.spool)
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index 98917de2efd..a80d83bf838 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -133,6 +133,7 @@ bthandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = btcostestimate;
 	amroutine->amoptions = btoptions;
 	amroutine->amproperty = btproperty;
+	amroutine->amphasename = btphasename;
 	amroutine->amvalidate = btvalidate;
 	amroutine->ambeginscan = btbeginscan;
 	amroutine->amrescan = btrescan;
diff --git a/src/backend/access/nbtree/nbtsort.c b/src/backend/access/nbtree/nbtsort.c
index dc398e11867..70310cea379 100644
--- a/src/backend/access/nbtree/nbtsort.c
+++ b/src/backend/access/nbtree/nbtsort.c
@@ -65,6 +65,7 @@
 #include "access/xlog.h"
 #include "access/xloginsert.h"
 #include "catalog/index.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
 #include "pgstat.h"
 #include "storage/smgr.h"
@@ -288,7 +289,8 @@ static double _bt_parallel_heapscan(BTBuildState *buildstate,
 static void _bt_leader_participate_as_worker(BTBuildState *buildstate);
 static void _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2,
 						   BTShared *btshared, Sharedsort *sharedsort,
-						   Sharedsort *sharedsort2, int sortmem);
+						   Sharedsort *sharedsort2, int sortmem,
+						   bool progress);
 
 
 /*
@@ -469,14 +471,19 @@ _bt_spools_heapscan(Relation heap, Relation index, BTBuildState *buildstate,
 	}
 
 	/* Fill spool using either serial or parallel heap scan */
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+								 PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN);
 	if (!buildstate->btleader)
-		reltuples = IndexBuildHeapScan(heap, index, indexInfo, true,
+		reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, true,
 									   _bt_build_callback, (void *) buildstate,
 									   NULL);
 	else
 		reltuples = _bt_parallel_heapscan(buildstate,
 										  &indexInfo->ii_BrokenHotChain);
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_TOTAL,
+								 buildstate->indtuples);
+
 	/* okay, all heap tuples are spooled */
 	if (buildstate->spool2 && !buildstate->havedead)
 	{
@@ -525,9 +532,15 @@ _bt_leafbuild(BTSpool *btspool, BTSpool *btspool2)
 	}
 #endif							/* BTREE_BUILD_STATS */
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+								 PROGRESS_BTREE_PHASE_PERFORMSORT_1);
 	tuplesort_performsort(btspool->sortstate);
 	if (btspool2)
+	{
+		pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+									 PROGRESS_BTREE_PHASE_PERFORMSORT_2);
 		tuplesort_performsort(btspool2->sortstate);
+	}
 
 	wstate.heap = btspool->heap;
 	wstate.index = btspool->index;
@@ -543,6 +556,8 @@ _bt_leafbuild(BTSpool *btspool, BTSpool *btspool2)
 	wstate.btws_pages_written = 0;
 	wstate.btws_zeropage = NULL;	/* until needed */
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+								 PROGRESS_BTREE_PHASE_LEAF_LOAD);
 	_bt_load(&wstate, btspool, btspool2);
 }
 
@@ -1078,6 +1093,7 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
 				keysz = IndexRelationGetNumberOfKeyAttributes(wstate->index);
 	ScanKey		indexScanKey = NULL;
 	SortSupport sortKeys;
+	long		tuples_done = 0L;
 
 	if (merge)
 	{
@@ -1170,6 +1186,10 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
 				_bt_buildadd(wstate, state, itup2);
 				itup2 = tuplesort_getindextuple(btspool2->sortstate, true);
 			}
+
+			/* Report progress */
+			pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE,
+										 ++tuples_done);
 		}
 		pfree(sortKeys);
 	}
@@ -1184,6 +1204,10 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
 				state = _bt_pagestate(wstate, 0);
 
 			_bt_buildadd(wstate, state, itup);
+
+			/* Report progress */
+			pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE,
+										 ++tuples_done);
 		}
 	}
 
@@ -1317,6 +1341,8 @@ _bt_begin_parallel(BTBuildState *buildstate, bool isconcurrent, int request)
 	btshared->indtuples = 0.0;
 	btshared->brokenhotchain = false;
 	heap_parallelscan_initialize(&btshared->heapdesc, btspool->heap, snapshot);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_BLOCKS_TOTAL,
+								 btshared->heapdesc.phs_nblocks);
 
 	/*
 	 * Store shared tuplesort-private state, for which we reserved space.
@@ -1493,7 +1519,7 @@ _bt_leader_participate_as_worker(BTBuildState *buildstate)
 	/* Perform work common to all participants */
 	_bt_parallel_scan_and_sort(leaderworker, leaderworker2, btleader->btshared,
 							   btleader->sharedsort, btleader->sharedsort2,
-							   sortmem);
+							   sortmem, true);
 
 #ifdef BTREE_BUILD_STATS
 	if (log_btree_build_stats)
@@ -1584,7 +1610,7 @@ _bt_parallel_build_main(dsm_segment *seg, shm_toc *toc)
 	/* Perform sorting of spool, and possibly a spool2 */
 	sortmem = maintenance_work_mem / btshared->scantuplesortstates;
 	_bt_parallel_scan_and_sort(btspool, btspool2, btshared, sharedsort,
-							   sharedsort2, sortmem);
+							   sharedsort2, sortmem, false);
 
 #ifdef BTREE_BUILD_STATS
 	if (log_btree_build_stats)
@@ -1613,7 +1639,7 @@ _bt_parallel_build_main(dsm_segment *seg, shm_toc *toc)
 static void
 _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2,
 						   BTShared *btshared, Sharedsort *sharedsort,
-						   Sharedsort *sharedsort2, int sortmem)
+						   Sharedsort *sharedsort2, int sortmem, bool progress)
 {
 	SortCoordinate coordinate;
 	BTBuildState buildstate;
@@ -1672,7 +1698,7 @@ _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2,
 	indexInfo->ii_Concurrent = btshared->isconcurrent;
 	scan = heap_beginscan_parallel(btspool->heap, &btshared->heapdesc);
 	reltuples = IndexBuildHeapScan(btspool->heap, btspool->index, indexInfo,
-								   true, _bt_build_callback,
+								   true, progress, _bt_build_callback,
 								   (void *) &buildstate, scan);
 
 	/*
diff --git a/src/backend/access/nbtree/nbtutils.c b/src/backend/access/nbtree/nbtutils.c
index 2c05fb5e451..dc47663c374 100644
--- a/src/backend/access/nbtree/nbtutils.c
+++ b/src/backend/access/nbtree/nbtutils.c
@@ -20,6 +20,7 @@
 #include "access/nbtree.h"
 #include "access/reloptions.h"
 #include "access/relscan.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
 #include "utils/array.h"
 #include "utils/lsyscache.h"
@@ -2082,6 +2083,29 @@ btproperty(Oid index_oid, int attno,
 	}
 }
 
+/*
+ *	btphasename() -- Return name of phase, for index build progress report
+ */
+char *
+btphasename(int64 phasenum)
+{
+	switch (phasenum)
+	{
+		case PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE:
+			return "initializing (1/5)";
+		case PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN:
+			return "table scan (2/5)";
+		case PROGRESS_BTREE_PHASE_PERFORMSORT_1:
+			return "sorting tuples, spool 1 (3/5)";
+		case PROGRESS_BTREE_PHASE_PERFORMSORT_2:
+			return "sorting tuples, spool 2 (4/5)";
+		case PROGRESS_BTREE_PHASE_LEAF_LOAD:
+			return "final btree sort & load (5/5)";
+		default:
+			return NULL;
+	}
+}
+
 /*
  *	_bt_nonkey_truncate() -- create tuple without non-key suffix attributes.
  *
diff --git a/src/backend/access/spgist/spginsert.c b/src/backend/access/spgist/spginsert.c
index f428a151385..1bc671c7238 100644
--- a/src/backend/access/spgist/spginsert.c
+++ b/src/backend/access/spgist/spginsert.c
@@ -142,7 +142,7 @@ spgbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 											  "SP-GiST build temporary context",
 											  ALLOCSET_DEFAULT_SIZES);
 
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false,
 								   spgistBuildCallback, (void *) &buildstate,
 								   NULL);
 
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 8e63c1fad25..9db2fb1693c 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -67,6 +67,7 @@ spghandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = spgcostestimate;
 	amroutine->amoptions = spgoptions;
 	amroutine->amproperty = spgproperty;
+	amroutine->amphasename = NULL;
 	amroutine->amvalidate = spgvalidate;
 	amroutine->ambeginscan = spgbeginscan;
 	amroutine->amrescan = spgrescan;
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index faf69568133..755e82063f8 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -49,8 +49,9 @@
 #include "catalog/pg_trigger.h"
 #include "catalog/pg_type.h"
 #include "catalog/storage.h"
-#include "commands/tablecmds.h"
 #include "commands/event_trigger.h"
+#include "commands/progress.h"
+#include "commands/tablecmds.h"
 #include "commands/trigger.h"
 #include "executor/executor.h"
 #include "miscadmin.h"
@@ -58,6 +59,7 @@
 #include "nodes/nodeFuncs.h"
 #include "optimizer/optimizer.h"
 #include "parser/parser.h"
+#include "pgstat.h"
 #include "rewrite/rewriteManip.h"
 #include "storage/bufmgr.h"
 #include "storage/lmgr.h"
@@ -1587,7 +1589,7 @@ index_drop(Oid indexId, bool concurrent)
 		 * to acquire an exclusive lock on our table.  The lock code will
 		 * detect deadlock and error out properly.
 		 */
-		WaitForLockers(heaplocktag, AccessExclusiveLock);
+		WaitForLockers(heaplocktag, AccessExclusiveLock, true);
 
 		/*
 		 * No more predicate locks will be acquired on this index, and we're
@@ -1631,7 +1633,7 @@ index_drop(Oid indexId, bool concurrent)
 		 * Wait till every transaction that saw the old index state has
 		 * finished.
 		 */
-		WaitForLockers(heaplocktag, AccessExclusiveLock);
+		WaitForLockers(heaplocktag, AccessExclusiveLock, true);
 
 		/*
 		 * Re-open relations to allow us to complete our actions.
@@ -2281,6 +2283,25 @@ index_build(Relation heapRelation,
 						   save_sec_context | SECURITY_RESTRICTED_OPERATION);
 	save_nestlevel = NewGUCNestLevel();
 
+	/* Set up initial progress report status */
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_PHASE,
+			PROGRESS_CREATEIDX_SUBPHASE,
+			PROGRESS_CREATEIDX_TUPLES_DONE,
+			PROGRESS_CREATEIDX_TUPLES_TOTAL,
+			PROGRESS_CREATEIDX_BLOCKS_DONE,
+			PROGRESS_CREATEIDX_BLOCKS_TOTAL
+		};
+		const int64	val[] = {
+			PROGRESS_CREATEIDX_PHASE_BUILD,
+			PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE,
+			0, 0, 0, 0
+		};
+
+		pgstat_progress_update_multi_param(6, index, val);
+	}
+
 	/*
 	 * Call the access method's build procedure
 	 */
@@ -2418,13 +2439,14 @@ IndexBuildHeapScan(Relation heapRelation,
 				   Relation indexRelation,
 				   IndexInfo *indexInfo,
 				   bool allow_sync,
+				   bool progress,
 				   IndexBuildCallback callback,
 				   void *callback_state,
 				   HeapScanDesc scan)
 {
 	return IndexBuildHeapRangeScan(heapRelation, indexRelation,
 								   indexInfo, allow_sync,
-								   false,
+								   false, progress,
 								   0, InvalidBlockNumber,
 								   callback, callback_state, scan);
 }
@@ -2445,6 +2467,7 @@ IndexBuildHeapRangeScan(Relation heapRelation,
 						IndexInfo *indexInfo,
 						bool allow_sync,
 						bool anyvisible,
+						bool progress,
 						BlockNumber start_blockno,
 						BlockNumber numblocks,
 						IndexBuildCallback callback,
@@ -2467,6 +2490,8 @@ IndexBuildHeapRangeScan(Relation heapRelation,
 	BlockNumber root_blkno = InvalidBlockNumber;
 	OffsetNumber root_offsets[MaxHeapTuplesPerPage];
 
+	BlockNumber	previous_blkno = InvalidBlockNumber;
+
 	/*
 	 * sanity checks
 	 */
@@ -2582,6 +2607,16 @@ IndexBuildHeapRangeScan(Relation heapRelation,
 
 		CHECK_FOR_INTERRUPTS();
 
+		if (progress &&
+			((previous_blkno == InvalidBlockNumber) ||
+			(scan->rs_cblock != previous_blkno)))
+		{
+			pgstat_progress_update_param(PROGRESS_CREATEIDX_BLOCKS_DONE,
+										 scan->rs_cblock);
+
+			previous_blkno = scan->rs_cblock;
+		}
+
 		/*
 		 * When dealing with a HOT-chain of updated tuples, we want to index
 		 * the values of the live tuple (if any), but index it under the TID
@@ -3123,6 +3158,9 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot)
 	int			save_sec_context;
 	int			save_nestlevel;
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_VALIDATE_IDXSCAN);
+
 	/* Open and lock the parent heap relation */
 	heapRelation = table_open(heapId, ShareUpdateExclusiveLock);
 	/* And the target index relation */
@@ -3174,11 +3212,15 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot)
 							 validate_index_callback, (void *) &state);
 
 	/* Execute the sort */
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_SORT_IDXSCAN);
 	tuplesort_performsort(state.tuplesort);
 
 	/*
 	 * Now scan the heap and "merge" it with the index
 	 */
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_HEAPSCAN_VALIDATE);
 	validate_index_heapscan(heapRelation,
 							indexRelation,
 							indexInfo,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 3e229c693c4..22c163be859 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -906,6 +906,32 @@ CREATE VIEW pg_stat_progress_vacuum AS
     FROM pg_stat_get_progress_info('VACUUM') AS S
 		LEFT JOIN pg_database D ON S.datid = D.oid;
 
+CREATE VIEW pg_stat_progress_create_index AS
+	SELECT
+		s.pid AS pid, S.datid AS datid, D.datname AS datname,
+		S.relid AS relid,
+		CASE s.param2 WHEN 0 THEN 'initializing (phase 1 of 8)'
+					  WHEN 1 THEN 'waiting for lockers 1 (phase 2 of 8)'
+					  WHEN 2 THEN 'building index (3 of 8): ' ||
+						pg_indexam_progress_phasename(s.param1::oid, s.param3)
+					  WHEN 3 THEN 'waiting for lockers 2 (phase 4 of 8)'
+					  WHEN 4 THEN 'validating index scan (phase 5 of 8)'
+					  WHEN 5 THEN 'sorting index scan results (phase 6 of 8)'
+					  WHEN 6 THEN 'validate index heapscan (phase 7 of 8)'
+					  WHEN 7 THEN 'waiting for lockers 3 (phase 8 of 8)'
+					  END as phase,
+		S.param4 AS "lockers total",
+		S.param5 AS "lockers done",
+		S.param6 AS "PID of current locker",
+		S.param7 AS "blocks total",
+		S.param8 AS "blocks done",
+		S.param9 AS "tuples total",
+		S.param10 AS "tuples done",
+		S.param11 AS "partitions total",
+		S.param12 AS "partitions done"
+	FROM pg_stat_get_progress_info('CREATE INDEX') AS S
+		LEFT JOIN pg_database D ON S.datid = D.oid;
+
 CREATE VIEW pg_user_mappings AS
     SELECT
         U.oid       AS umid,
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index bd85099c286..d6195ee5781 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -36,6 +36,7 @@
 #include "commands/dbcommands.h"
 #include "commands/defrem.h"
 #include "commands/event_trigger.h"
+#include "commands/progress.h"
 #include "commands/tablecmds.h"
 #include "commands/tablespace.h"
 #include "mb/pg_wchar.h"
@@ -46,10 +47,12 @@
 #include "parser/parse_coerce.h"
 #include "parser/parse_func.h"
 #include "parser/parse_oper.h"
+#include "pgstat.h"
 #include "rewrite/rewriteManip.h"
 #include "storage/lmgr.h"
 #include "storage/proc.h"
 #include "storage/procarray.h"
+#include "storage/sinvaladt.h"
 #include "utils/acl.h"
 #include "utils/builtins.h"
 #include "utils/fmgroids.h"
@@ -368,6 +371,15 @@ DefineIndex(Oid relationId,
 	Snapshot	snapshot;
 	int			i;
 
+
+	/*
+	 * Start progress report.  If we're building a partition, this was already
+	 * done.
+	 */
+	if (!OidIsValid(parentIndexId))
+		pgstat_progress_start_command(PROGRESS_COMMAND_CREATE_INDEX,
+									  relationId);
+
 	/*
 	 * count key attributes in index
 	 */
@@ -584,6 +596,9 @@ DefineIndex(Oid relationId,
 	accessMethodId = accessMethodForm->oid;
 	amRoutine = GetIndexAmRoutine(accessMethodForm->amhandler);
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_ACCESS_METHOD_OID,
+								 accessMethodId);
+
 	if (stmt->unique && !amRoutine->amcanunique)
 		ereport(ERROR,
 				(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -864,6 +879,11 @@ DefineIndex(Oid relationId,
 	if (!OidIsValid(indexRelationId))
 	{
 		table_close(rel, NoLock);
+
+		/* If this is the top-level index, we're done */
+		if (!OidIsValid(parentIndexId))
+			pgstat_progress_end_command();
+
 		return address;
 	}
 
@@ -889,6 +909,9 @@ DefineIndex(Oid relationId,
 			TupleDesc	parentDesc;
 			Oid		   *opfamOids;
 
+			pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL,
+										 nparts);
+
 			memcpy(part_oids, partdesc->oids, sizeof(Oid) * nparts);
 
 			parentDesc = CreateTupleDescCopy(RelationGetDescr(rel));
@@ -1038,6 +1061,8 @@ DefineIndex(Oid relationId,
 								skip_build, quiet);
 				}
 
+				pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_DONE,
+											 i + 1);
 				pfree(attmap);
 			}
 
@@ -1072,6 +1097,8 @@ DefineIndex(Oid relationId,
 		 * Indexes on partitioned tables are not themselves built, so we're
 		 * done here.
 		 */
+		if (!OidIsValid(parentIndexId))
+			pgstat_progress_end_command();
 		return address;
 	}
 
@@ -1079,6 +1106,11 @@ DefineIndex(Oid relationId,
 	{
 		/* Close the heap and we're done, in the non-concurrent case */
 		table_close(rel, NoLock);
+
+		/* If this is the top-level index, we're done. */
+		if (!OidIsValid(parentIndexId))
+			pgstat_progress_end_command();
+
 		return address;
 	}
 
@@ -1130,7 +1162,9 @@ DefineIndex(Oid relationId,
 	 * exclusive lock on our table.  The lock code will detect deadlock and
 	 * error out properly.
 	 */
-	WaitForLockers(heaplocktag, ShareLock);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_1);
+	WaitForLockers(heaplocktag, ShareLock, true);
 
 	/*
 	 * At this moment we are sure that there are no transactions with the
@@ -1194,7 +1228,9 @@ DefineIndex(Oid relationId,
 	 * We once again wait until no transaction can have the table open with
 	 * the index marked as read-only for updates.
 	 */
-	WaitForLockers(heaplocktag, ShareLock);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_2);
+	WaitForLockers(heaplocktag, ShareLock, true);
 
 	/*
 	 * Now take the "reference snapshot" that will be used by validate_index()
@@ -1280,6 +1316,9 @@ DefineIndex(Oid relationId,
 	old_snapshots = GetCurrentVirtualXIDs(limitXmin, true, false,
 										  PROC_IS_AUTOVACUUM | PROC_IN_VACUUM,
 										  &n_old_snapshots);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_3);
+	pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, n_old_snapshots);
 
 	for (i = 0; i < n_old_snapshots; i++)
 	{
@@ -1315,7 +1354,14 @@ DefineIndex(Oid relationId,
 		}
 
 		if (VirtualTransactionIdIsValid(old_snapshots[i]))
+		{
+			PGPROC *holder = BackendIdGetProc(old_snapshots[i].backendId);
+			pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID,
+										 holder->pid);
 			VirtualXactLock(old_snapshots[i], true);
+		}
+
+		pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, i + 1);
 	}
 
 	/*
@@ -1338,6 +1384,8 @@ DefineIndex(Oid relationId,
 	 */
 	UnlockRelationIdForSession(&heaprelid, ShareUpdateExclusiveLock);
 
+	pgstat_progress_end_command();
+
 	return address;
 }
 
diff --git a/src/backend/storage/ipc/standby.c b/src/backend/storage/ipc/standby.c
index 4d10e57a803..a0a2b964703 100644
--- a/src/backend/storage/ipc/standby.c
+++ b/src/backend/storage/ipc/standby.c
@@ -401,7 +401,7 @@ ResolveRecoveryConflictWithLock(LOCKTAG locktag)
 		 */
 		VirtualTransactionId *backends;
 
-		backends = GetLockConflicts(&locktag, AccessExclusiveLock);
+		backends = GetLockConflicts(&locktag, AccessExclusiveLock, NULL);
 		ResolveRecoveryConflictWithVirtualXIDs(backends,
 											   PROCSIG_RECOVERY_CONFLICT_LOCK);
 	}
diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c
index e688ba81170..0b04b093782 100644
--- a/src/backend/storage/lmgr/lmgr.c
+++ b/src/backend/storage/lmgr/lmgr.c
@@ -19,9 +19,12 @@
 #include "access/transam.h"
 #include "access/xact.h"
 #include "catalog/catalog.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
+#include "pgstat.h"
 #include "storage/lmgr.h"
 #include "storage/procarray.h"
+#include "storage/sinvaladt.h"
 #include "utils/inval.h"
 
 
@@ -857,10 +860,12 @@ XactLockTableWaitErrorCb(void *arg)
  * after we obtained our initial list of lockers, we will not wait for them.
  */
 void
-WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
+WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
 {
 	List	   *holders = NIL;
 	ListCell   *lc;
+	int			total = 0;
+	int			done = 0;
 
 	/* Done if no locks to wait for */
 	if (list_length(locktags) == 0)
@@ -870,10 +875,17 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
 	foreach(lc, locktags)
 	{
 		LOCKTAG    *locktag = lfirst(lc);
+		int			count;
 
-		holders = lappend(holders, GetLockConflicts(locktag, lockmode));
+		holders = lappend(holders,
+						  GetLockConflicts(locktag, lockmode,
+										   progress ? &count : NULL));
+		total += count;
 	}
 
+	if (progress)
+		pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, total);
+
 	/*
 	 * Note: GetLockConflicts() never reports our own xid, hence we need not
 	 * check for that.  Also, prepared xacts are not reported, which is fine
@@ -887,10 +899,36 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
 
 		while (VirtualTransactionIdIsValid(*lockholders))
 		{
+			/*
+			 * If requested, publish who we're going to wait for.  This is not
+			 * 100% accurate if they're already gone, but we don't care.
+			 */
+			if (progress)
+			{
+				PGPROC *holder = BackendIdGetProc(lockholders->backendId);
+
+				pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID,
+											 holder->pid);
+			}
 			VirtualXactLock(*lockholders, true);
 			lockholders++;
+
+			if (progress)
+				pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, ++done);
 		}
 	}
+	if (progress)
+	{
+		const int	index[] = {
+			PROGRESS_WAITFOR_TOTAL,
+			PROGRESS_WAITFOR_DONE,
+			PROGRESS_WAITFOR_CURRENT_PID
+		};
+		const int64	values[] = {
+			0, 0, 0
+		};
+		pgstat_progress_update_multi_param(3, index, values);
+	}
 
 	list_free_deep(holders);
 }
@@ -901,12 +939,12 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
  * Same as WaitForLockersMultiple, for a single lock tag.
  */
 void
-WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode)
+WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress)
 {
 	List	   *l;
 
 	l = list_make1(&heaplocktag);
-	WaitForLockersMultiple(l, lockmode);
+	WaitForLockersMultiple(l, lockmode, progress);
 	list_free(l);
 }
 
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 3bb5ce350aa..58ba90d0646 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2807,6 +2807,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock)
  *		xacts merely awaiting such a lock are NOT reported.
  *
  * The result array is palloc'd and is terminated with an invalid VXID.
+ * *countp, if not null, is updated to the number of items set.
  *
  * Of course, the result could be out of date by the time it's returned,
  * so use of this function has to be thought about carefully.
@@ -2817,7 +2818,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock)
  * uses of the result.
  */
 VirtualTransactionId *
-GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode)
+GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
 {
 	static VirtualTransactionId *vxids;
 	LOCKMETHODID lockmethodid = locktag->locktag_lockmethodid;
@@ -2964,6 +2965,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode)
 		LWLockRelease(partitionLock);
 		vxids[count].backendId = InvalidBackendId;
 		vxids[count].localTransactionId = InvalidLocalTransactionId;
+		if (countp)
+			*countp = count;
 		return vxids;
 	}
 
@@ -3019,6 +3022,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode)
 
 	vxids[count].backendId = InvalidBackendId;
 	vxids[count].localTransactionId = InvalidLocalTransactionId;
+	if (countp)
+		*countp = count;
 	return vxids;
 }
 
diff --git a/src/backend/utils/adt/amutils.c b/src/backend/utils/adt/amutils.c
index 060ffe501ec..05046e9b559 100644
--- a/src/backend/utils/adt/amutils.c
+++ b/src/backend/utils/adt/amutils.c
@@ -445,3 +445,26 @@ pg_index_column_has_property(PG_FUNCTION_ARGS)
 
 	return indexam_property(fcinfo, propname, InvalidOid, relid, attno);
 }
+
+/*
+ * Return the name of the given phase, as used for progress reporting by the
+ * given AM.
+ */
+Datum
+pg_indexam_progress_phasename(PG_FUNCTION_ARGS)
+{
+	Oid			amoid = PG_GETARG_OID(0);
+	int32		phasenum = PG_GETARG_INT32(1);
+	IndexAmRoutine *routine;
+	char	   *name;
+
+	routine = GetIndexAmRoutineByAmId(amoid, true);
+	if (routine == NULL || !routine->amphasename)
+		PG_RETURN_NULL();
+
+	name = routine->amphasename(phasenum);
+	if (!name)
+		PG_RETURN_NULL();
+
+	PG_RETURN_TEXT_P(CStringGetTextDatum(name));
+}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index b6ba856ebe6..91dc5fed9cd 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -468,6 +468,8 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
 	/* Translate command name into command type code. */
 	if (pg_strcasecmp(cmd, "VACUUM") == 0)
 		cmdtype = PROGRESS_COMMAND_VACUUM;
+	else if (pg_strcasecmp(cmd, "CREATE INDEX") == 0)
+		cmdtype = PROGRESS_COMMAND_CREATE_INDEX;
 	else
 		ereport(ERROR,
 				(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 653ddc976ba..471625db14f 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -108,6 +108,9 @@ typedef bool (*amproperty_function) (Oid index_oid, int attno,
 									 IndexAMProperty prop, const char *propname,
 									 bool *res, bool *isnull);
 
+/* name of phase as used in progress reporting */
+typedef char *(*amphasename_function) (int64 phasenum);
+
 /* validate definition of an opclass for this AM */
 typedef bool (*amvalidate_function) (Oid opclassoid);
 
@@ -213,6 +216,7 @@ typedef struct IndexAmRoutine
 	amcostestimate_function amcostestimate;
 	amoptions_function amoptions;
 	amproperty_function amproperty; /* can be NULL */
+	amphasename_function amphasename;	/* can be NULL */
 	amvalidate_function amvalidate;
 	ambeginscan_function ambeginscan;
 	amrescan_function amrescan;
diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h
index 4fb92d60a12..b53b9016870 100644
--- a/src/include/access/nbtree.h
+++ b/src/include/access/nbtree.h
@@ -488,6 +488,16 @@ typedef BTScanOpaqueData *BTScanOpaque;
 #define SK_BT_DESC			(INDOPTION_DESC << SK_BT_INDOPTION_SHIFT)
 #define SK_BT_NULLS_FIRST	(INDOPTION_NULLS_FIRST << SK_BT_INDOPTION_SHIFT)
 
+/*
+ * Constant definition for progress reporting.  Phase numbers must match
+ * btphasename.
+ */
+/* PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE is 1 (see progress.h) */
+#define PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN		2
+#define PROGRESS_BTREE_PHASE_PERFORMSORT_1				3
+#define PROGRESS_BTREE_PHASE_PERFORMSORT_2				4
+#define PROGRESS_BTREE_PHASE_LEAF_LOAD					5
+
 /*
  * external entry points for btree, in nbtree.c
  */
@@ -600,6 +610,7 @@ extern bytea *btoptions(Datum reloptions, bool validate);
 extern bool btproperty(Oid index_oid, int attno,
 		   IndexAMProperty prop, const char *propname,
 		   bool *res, bool *isnull);
+extern char *btphasename(int64 phasenum);
 extern IndexTuple _bt_nonkey_truncate(Relation rel, IndexTuple itup);
 extern bool _bt_check_natts(Relation rel, Page page, OffsetNumber offnum);
 
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index 330c481a8b7..fb713d50f96 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -114,6 +114,7 @@ extern double IndexBuildHeapScan(Relation heapRelation,
 				   Relation indexRelation,
 				   IndexInfo *indexInfo,
 				   bool allow_sync,
+				   bool progress,
 				   IndexBuildCallback callback,
 				   void *callback_state,
 				   struct HeapScanDescData *scan);
@@ -122,6 +123,7 @@ extern double IndexBuildHeapRangeScan(Relation heapRelation,
 						IndexInfo *indexInfo,
 						bool allow_sync,
 						bool anyvisible,
+						bool progress,
 						BlockNumber start_blockno,
 						BlockNumber end_blockno,
 						IndexBuildCallback callback,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 50b742c06e6..f8e0c55b701 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -913,6 +913,10 @@
   proname => 'pg_index_column_has_property', provolatile => 's',
   prorettype => 'bool', proargtypes => 'regclass int4 text',
   prosrc => 'pg_index_column_has_property' },
+{ oid => '676', descr => 'return name of given index build phase',
+  proname => 'pg_indexam_progress_phasename', provolatile => 'i',
+  prorettype => 'text', proargtypes => 'oid int8',
+  prosrc => 'pg_indexam_progress_phasename' },
 
 { oid => '339',
   proname => 'poly_same', prorettype => 'bool',
@@ -5061,9 +5065,9 @@
   proname => 'pg_stat_get_progress_info', prorows => '100', proretset => 't',
   provolatile => 's', proparallel => 'r', prorettype => 'record',
   proargtypes => 'text',
-  proallargtypes => '{text,int4,oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}',
-  proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o}',
-  proargnames => '{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10}',
+  proallargtypes => '{text,int4,oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}',
+  proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+  proargnames => '{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10,param11,param12}',
   prosrc => 'pg_stat_get_progress_info' },
 { oid => '3099',
   descr => 'statistics: information about currently active replication',
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index 9858b36a383..984d4fc0bdd 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -34,4 +34,39 @@
 #define PROGRESS_VACUUM_PHASE_TRUNCATE			5
 #define PROGRESS_VACUUM_PHASE_FINAL_CLEANUP		6
 
+
+/* Progress parameters for CREATE INDEX */
+#define PROGRESS_CREATEIDX_ACCESS_METHOD_OID	0
+#define PROGRESS_CREATEIDX_PHASE				1
+#define PROGRESS_CREATEIDX_SUBPHASE				2
+/* 3, 4 and 5 reserved for "waitfor" metrics */
+#define PROGRESS_CREATEIDX_BLOCKS_TOTAL			6
+#define PROGRESS_CREATEIDX_BLOCKS_DONE			7
+#define PROGRESS_CREATEIDX_TUPLES_TOTAL			8
+#define PROGRESS_CREATEIDX_TUPLES_DONE			9
+#define PROGRESS_CREATEIDX_PARTITIONS_TOTAL		10
+#define PROGRESS_CREATEIDX_PARTITIONS_DONE		11
+
+/* Phases of CREATE INDEX */
+#define PROGRESS_CREATEIDX_PHASE_WAIT_1			1
+#define PROGRESS_CREATEIDX_PHASE_BUILD			2
+#define PROGRESS_CREATEIDX_PHASE_WAIT_2			3
+#define PROGRESS_CREATEIDX_PHASE_VALIDATE_IDXSCAN		4
+#define PROGRESS_CREATEIDX_PHASE_SORT_IDXSCAN		5
+#define PROGRESS_CREATEIDX_PHASE_HEAPSCAN_VALIDATE		6
+#define PROGRESS_CREATEIDX_PHASE_WAIT_3			7
+
+/*
+ * Subphases of CREATE INDEX, for index_build.
+ */
+#define PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE	1
+/* Additional phases are defined by each AM */
+
+/* Lock holder wait counts */
+#define PROGRESS_WAITFOR_TOTAL					3
+#define PROGRESS_WAITFOR_DONE					4
+#define PROGRESS_WAITFOR_CURRENT_PID			5
+
+/* Bl */
+
 #endif
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 88a75fb798e..f931ead1c27 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -934,10 +934,11 @@ typedef enum
 typedef enum ProgressCommandType
 {
 	PROGRESS_COMMAND_INVALID,
-	PROGRESS_COMMAND_VACUUM
+	PROGRESS_COMMAND_VACUUM,
+	PROGRESS_COMMAND_CREATE_INDEX
 } ProgressCommandType;
 
-#define PGSTAT_NUM_PROGRESS_PARAM	10
+#define PGSTAT_NUM_PROGRESS_PARAM	12
 
 /* ----------
  * Shared-memory data structures
diff --git a/src/include/storage/lmgr.h b/src/include/storage/lmgr.h
index 3d705faba5c..4f2872de35f 100644
--- a/src/include/storage/lmgr.h
+++ b/src/include/storage/lmgr.h
@@ -78,8 +78,8 @@ extern void XactLockTableWait(TransactionId xid, Relation rel,
 extern bool ConditionalXactLockTableWait(TransactionId xid);
 
 /* Lock VXIDs, specified by conflicting locktags */
-extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode);
-extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode);
+extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress);
+extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress);
 
 /* Lock an XID for tuple insertion (used to wait for an insertion to finish) */
 extern uint32 SpeculativeInsertionLockAcquire(TransactionId xid);
diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h
index 16b927cb801..e117b391774 100644
--- a/src/include/storage/lock.h
+++ b/src/include/storage/lock.h
@@ -544,7 +544,7 @@ extern bool LockHeldByMe(const LOCKTAG *locktag, LOCKMODE lockmode);
 extern bool LockHasWaiters(const LOCKTAG *locktag,
 			   LOCKMODE lockmode, bool sessionLock);
 extern VirtualTransactionId *GetLockConflicts(const LOCKTAG *locktag,
-				 LOCKMODE lockmode);
+				 LOCKMODE lockmode, int *countp);
 extern void AtPrepare_Locks(void);
 extern void PostPrepare_Locks(TransactionId xid);
 extern int LockCheckConflicts(LockMethod lockMethodTable,
#8Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Alvaro Herrera (#7)
1 attachment(s)
Re: monitoring CREATE INDEX [CONCURRENTLY]

I added metrics for the validate_index phases; patch attached. This is
still a bit raw, but it looks much better now. Here's a sample
concurrent index build on a 100M tuple table. There are no concurrent
transactions, so phases that wait for lockers don't appear. I'm not
making any effort to report metrics during phase 6, note. In phase 7
I'm currently reporting only tuple counts; I think it'd be better to
report block numbers.

I don't show it here, but when an index is built on a partitioned table,
the "partitions done" number goes up all the way to "partitions total"
and the phases come and go for each partition.

One thing is clear: given the phase mechanics of varying durations, it
seems hard to use these numbers to predict total index build time.

now | phase | blocks total | blocks done | tuples total | tuples done | partitions total | partitions done
--------------+--------------------------------------------------------+--------------+-------------+--------------+-------------+------------------+-----------------
15:56:33.792 | building index (3 of 8): initializing (1/5) | 442478 | 32 | 0 | 0 | 0 | 0
15:56:33.805 | building index (3 of 8): initializing (1/5) | 442478 | 188 | 0 | 0 | 0 | 0
[snip about 500 lines]
15:56:41.345 | building index (3 of 8): initializing (1/5) | 442478 | 439855 | 0 | 0 | 0 | 0
15:56:41.356 | building index (3 of 8): initializing (1/5) | 442478 | 440288 | 0 | 0 | 0 | 0
15:56:41.367 | building index (3 of 8): initializing (1/5) | 442478 | 440778 | 0 | 0 | 0 | 0
15:56:41.378 | building index (3 of 8): initializing (1/5) | 442478 | 441706 | 0 | 0 | 0 | 0
15:56:41.389 | building index (3 of 8): initializing (1/5) | 442478 | 442399 | 0 | 0 | 0 | 0
15:56:41.4 | building index (3 of 8): initializing (1/5) | 442478 | 442399 | 0 | 0 | 0 | 0

[snip 300 lines] ... I'm not sure what happened in this 3 seconds period. No metrics were being updated.

15:56:44.65 | building index (3 of 8): initializing (1/5) | 442478 | 442399 | 0 | 0 | 0 | 0
15:56:44.661 | building index (3 of 8): initializing (1/5) | 442478 | 442399 | 0 | 0 | 0 | 0
15:56:44.672 | building index (3 of 8): initializing (1/5) | 442478 | 442399 | 0 | 0 | 0 | 0
15:56:44.682 | building index (3 of 8): initializing (1/5) | 442478 | 442399 | 0 | 0 | 0 | 0
15:56:44.694 | building index (3 of 8): initializing (1/5) | 442478 | 442399 | 0 | 0 | 0 | 0
15:56:44.705 | building index (3 of 8): sorting tuples, spool 1 (3/5) | 442478 | 442399 | 100000000 | 0 | 0 | 0
15:56:44.716 | building index (3 of 8): sorting tuples, spool 1 (3/5) | 442478 | 442399 | 100000000 | 0 | 0 | 0
15:56:44.727 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 79057 | 0 | 0
15:56:44.738 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 217018 | 0 | 0
15:56:44.75 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 353804 | 0 | 0
15:56:44.761 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 487892 | 0 | 0
15:56:44.773 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 634136 | 0 | 0
[snip 660 lines]
15:56:52.47 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 99111337 | 0 | 0
15:56:52.482 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 99285701 | 0 | 0
15:56:52.493 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 99444763 | 0 | 0
15:56:52.505 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 99612313 | 0 | 0
15:56:52.516 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 99782666 | 0 | 0
15:56:52.528 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 99937524 | 0 | 0
15:56:52.54 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 100000000 | 0 | 0
15:56:52.551 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 100000000 | 0 | 0
15:56:52.561 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 100000000 | 0 | 0
15:56:52.572 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 100000000 | 0 | 0
15:56:52.583 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 100000000 | 0 | 0
15:56:52.594 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 100000000 | 0 | 0
15:56:52.604 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 100000000 | 0 | 0
15:56:52.615 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 100000000 | 0 | 0
15:56:52.626 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 100000000 | 0 | 0
15:56:52.637 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 100000000 | 0 | 0
15:56:52.648 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 100000000 | 0 | 0
15:56:52.658 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 100000000 | 0 | 0
15:56:52.669 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 100000000 | 0 | 0
15:56:52.68 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 100000000 | 0 | 0
15:56:52.691 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 100000000 | 0 | 0
15:56:52.701 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 100000000 | 0 | 0
15:56:52.712 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 100000000 | 0 | 0
15:56:52.723 | validating index scan (phase 5 of 8) | 274194 | 1049 | 0 | 0 | 0 | 0
15:56:52.734 | validating index scan (phase 5 of 8) | 274194 | 2676 | 0 | 0 | 0 | 0
15:56:52.744 | validating index scan (phase 5 of 8) | 274194 | 2876 | 0 | 0 | 0 | 0
15:56:52.755 | validating index scan (phase 5 of 8) | 274194 | 4194 | 0 | 0 | 0 | 0
[snip 400 lines]
15:56:57.031 | validating index scan (phase 5 of 8) | 274194 | 268343 | 0 | 0 | 0 | 0
15:56:57.042 | validating index scan (phase 5 of 8) | 274194 | 268479 | 0 | 0 | 0 | 0
15:56:57.053 | validating index scan (phase 5 of 8) | 274194 | 270027 | 0 | 0 | 0 | 0
15:56:57.064 | validating index scan (phase 5 of 8) | 274194 | 271580 | 0 | 0 | 0 | 0
15:56:57.075 | validating index scan (phase 5 of 8) | 274194 | 273121 | 0 | 0 | 0 | 0
15:56:57.085 | sorting index scan results (phase 6 of 8) | 274194 | 274193 | 0 | 0 | 0 | 0
15:56:57.096 | sorting index scan results (phase 6 of 8) | 274194 | 274193 | 0 | 0 | 0 | 0
15:56:57.107 | sorting index scan results (phase 6 of 8) | 274194 | 274193 | 0 | 0 | 0 | 0
15:56:57.118 | sorting index scan results (phase 6 of 8) | 274194 | 274193 | 0 | 0 | 0 | 0
15:56:57.128 | sorting index scan results (phase 6 of 8) | 274194 | 274193 | 0 | 0 | 0 | 0
15:56:57.139 | sorting index scan results (phase 6 of 8) | 274194 | 274193 | 0 | 0 | 0 | 0
15:56:57.15 | sorting index scan results (phase 6 of 8) | 274194 | 274193 | 0 | 0 | 0 | 0
15:56:57.161 | validate index heapscan (phase 7 of 8) | 0 | 0 | 100000000 | 50152 | 0 | 0
15:56:57.172 | validate index heapscan (phase 7 of 8) | 0 | 0 | 100000000 | 175602 | 0 | 0
15:56:57.182 | validate index heapscan (phase 7 of 8) | 0 | 0 | 100000000 | 305326 | 0 | 0
15:56:57.193 | validate index heapscan (phase 7 of 8) | 0 | 0 | 100000000 | 430142 | 0 | 0
[snip 730 lines]
15:57:05.003 | validate index heapscan (phase 7 of 8) | 0 | 0 | 100000000 | 99125556 | 0 | 0
15:57:05.013 | validate index heapscan (phase 7 of 8) | 0 | 0 | 100000000 | 99276471 | 0 | 0
15:57:05.024 | validate index heapscan (phase 7 of 8) | 0 | 0 | 100000000 | 99425041 | 0 | 0
15:57:05.035 | validate index heapscan (phase 7 of 8) | 0 | 0 | 100000000 | 99580174 | 0 | 0
15:57:05.045 | validate index heapscan (phase 7 of 8) | 0 | 0 | 100000000 | 99720505 | 0 | 0
15:57:05.056 | validate index heapscan (phase 7 of 8) | 0 | 0 | 100000000 | 99862311 | 0 | 0
15:57:05.067 | validate index heapscan (phase 7 of 8) | 0 | 0 | 100000000 | 100000000 | 0 | 0
15:57:05.077 | validate index heapscan (phase 7 of 8) | 0 | 0 | 100000000 | 100000000 | 0 | 0
15:57:05.088 | validate index heapscan (phase 7 of 8) | 0 | 0 | 100000000 | 100000000 | 0 | 0
15:57:05.099 | validate index heapscan (phase 7 of 8) | 0 | 0 | 100000000 | 100000000 | 0 | 0
15:57:05.109 | validate index heapscan (phase 7 of 8) | 0 | 0 | 100000000 | 100000000 | 0 | 0
15:57:05.12 | validate index heapscan (phase 7 of 8) | 0 | 0 | 100000000 | 100000000 | 0 | 0
15:57:05.131 | validate index heapscan (phase 7 of 8) | 0 | 0 | 100000000 | 100000000 | 0 | 0
15:57:05.142 | validate index heapscan (phase 7 of 8) | 0 | 0 | 100000000 | 100000000 | 0 | 0
15:57:05.152 | validate index heapscan (phase 7 of 8) | 0 | 0 | 100000000 | 100000000 | 0 | 0
15:57:05.163 | validate index heapscan (phase 7 of 8) | 0 | 0 | 100000000 | 100000000 | 0 | 0
15:57:05.174 | validate index heapscan (phase 7 of 8) | 0 | 0 | 100000000 | 100000000 | 0 | 0
(2862 filas)

and then it's done.

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attachments:

v3-create-index-progress.patchtext/x-diff; charset=us-asciiDownload
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index 964200a7678..99f6ed6bc44 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -534,7 +534,7 @@ bt_check_every_level(Relation rel, Relation heaprel, bool readonly,
 			 RelationGetRelationName(state->rel),
 			 RelationGetRelationName(state->heaprel));
 
-		IndexBuildHeapScan(state->heaprel, state->rel, indexinfo, true,
+		IndexBuildHeapScan(state->heaprel, state->rel, indexinfo, true, false,
 						   bt_tuple_present_callback, (void *) state, scan);
 
 		ereport(DEBUG1,
diff --git a/contrib/bloom/blinsert.c b/contrib/bloom/blinsert.c
index e43fbe0005f..947ee74881f 100644
--- a/contrib/bloom/blinsert.c
+++ b/contrib/bloom/blinsert.c
@@ -141,7 +141,7 @@ blbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	initCachedPage(&buildstate);
 
 	/* Do the heap scan */
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false,
 								   bloomBuildCallback, (void *) &buildstate,
 								   NULL);
 
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 64583765787..79fe413f994 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -132,6 +132,7 @@ blhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = blcostestimate;
 	amroutine->amoptions = bloptions;
 	amroutine->amproperty = NULL;
+	amroutine->amphasename = NULL;
 	amroutine->amvalidate = blvalidate;
 	amroutine->ambeginscan = blbeginscan;
 	amroutine->amrescan = blrescan;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index 05102724ead..189179a5667 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -127,6 +127,7 @@ typedef struct IndexAmRoutine
     amcostestimate_function amcostestimate;
     amoptions_function amoptions;
     amproperty_function amproperty;     /* can be NULL */
+    amphasename_function amphasename;   /* can be NULL */
     amvalidate_function amvalidate;
     ambeginscan_function ambeginscan;
     amrescan_function amrescan;
@@ -468,6 +469,16 @@ amproperty (Oid index_oid, int attno,
 
   <para>
 <programlisting>
+char *
+amphasename (int64 phasenum);
+</programlisting>
+   Return the textual name of the given phase number.  The phase numbers are
+   those reported during an index build via the
+   <function>pgstat_progress_update_param</function> interface.
+  </para>
+
+  <para>
+<programlisting>
 bool
 amvalidate (Oid opclassoid);
 </programlisting>
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 8f008dd0080..547a67fd2e6 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -111,6 +111,7 @@ brinhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = brincostestimate;
 	amroutine->amoptions = brinoptions;
 	amroutine->amproperty = NULL;
+	amroutine->amphasename = NULL;
 	amroutine->amvalidate = brinvalidate;
 	amroutine->ambeginscan = brinbeginscan;
 	amroutine->amrescan = brinrescan;
@@ -718,7 +719,7 @@ brinbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	 * Now scan the relation.  No syncscan allowed here because we want the
 	 * heap blocks in physical order.
 	 */
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, false,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, false, false,
 								   brinbuildCallback, (void *) state, NULL);
 
 	/* process the final batch */
@@ -1234,7 +1235,7 @@ summarize_range(IndexInfo *indexInfo, BrinBuildState *state, Relation heapRel,
 	 * by transactions that are still in progress, among other corner cases.
 	 */
 	state->bs_currRangeStart = heapBlk;
-	IndexBuildHeapRangeScan(heapRel, state->bs_irel, indexInfo, false, true,
+	IndexBuildHeapRangeScan(heapRel, state->bs_irel, indexInfo, false, true, false,
 							heapBlk, scanNumBlks,
 							brinbuildCallback, (void *) state, NULL);
 
diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c
index 524ac5be8b5..838de4c1ec3 100644
--- a/src/backend/access/gin/gininsert.c
+++ b/src/backend/access/gin/gininsert.c
@@ -394,7 +394,7 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	 * Do the heap scan.  We disallow sync scan here because dataPlaceToPage
 	 * prefers to receive tuples in TID order.
 	 */
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, false,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, false, false,
 								   ginBuildCallback, (void *) &buildstate, NULL);
 
 	/* dump remaining entries to the index */
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index afc20232ace..240f8df5ad8 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -64,6 +64,7 @@ ginhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = gincostestimate;
 	amroutine->amoptions = ginoptions;
 	amroutine->amproperty = NULL;
+	amroutine->amphasename = NULL;
 	amroutine->amvalidate = ginvalidate;
 	amroutine->ambeginscan = ginbeginscan;
 	amroutine->amrescan = ginrescan;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index b75b3a8dacd..8839b24d7ac 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = gistcostestimate;
 	amroutine->amoptions = gistoptions;
 	amroutine->amproperty = gistproperty;
+	amroutine->amphasename = NULL;
 	amroutine->amvalidate = gistvalidate;
 	amroutine->ambeginscan = gistbeginscan;
 	amroutine->amrescan = gistrescan;
diff --git a/src/backend/access/gist/gistbuild.c b/src/backend/access/gist/gistbuild.c
index bd142a3560d..015f874cc93 100644
--- a/src/backend/access/gist/gistbuild.c
+++ b/src/backend/access/gist/gistbuild.c
@@ -204,7 +204,7 @@ gistbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	/*
 	 * Do the heap scan.
 	 */
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false,
 								   gistBuildCallback, (void *) &buildstate, NULL);
 
 	/*
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index f1f01a0956d..b661daed381 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -82,6 +82,7 @@ hashhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = hashcostestimate;
 	amroutine->amoptions = hashoptions;
 	amroutine->amproperty = NULL;
+	amroutine->amphasename = NULL;
 	amroutine->amvalidate = hashvalidate;
 	amroutine->ambeginscan = hashbeginscan;
 	amroutine->amrescan = hashrescan;
@@ -159,7 +160,7 @@ hashbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	buildstate.heapRel = heap;
 
 	/* do the heap scan */
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false,
 								   hashbuildCallback, (void *) &buildstate, NULL);
 
 	if (buildstate.spool)
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index 98917de2efd..3a3fcaaf33d 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -22,6 +22,7 @@
 #include "access/nbtxlog.h"
 #include "access/relscan.h"
 #include "access/xlog.h"
+#include "commands/progress.h"
 #include "commands/vacuum.h"
 #include "miscadmin.h"
 #include "nodes/execnodes.h"
@@ -133,6 +134,7 @@ bthandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = btcostestimate;
 	amroutine->amoptions = btoptions;
 	amroutine->amproperty = btproperty;
+	amroutine->amphasename = btphasename;
 	amroutine->amvalidate = btvalidate;
 	amroutine->ambeginscan = btbeginscan;
 	amroutine->amrescan = btrescan;
@@ -1021,6 +1023,10 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
 		if (needLock)
 			UnlockRelationForExtension(rel, ExclusiveLock);
 
+		if (info->report_progress)
+			pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL,
+										 num_pages);
+
 		/* Quit if we've scanned the whole relation */
 		if (blkno >= num_pages)
 			break;
@@ -1028,6 +1034,9 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
 		for (; blkno < num_pages; blkno++)
 		{
 			btvacuumpage(&vstate, blkno, blkno);
+			if (info->report_progress)
+				pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
+											 blkno);
 		}
 	}
 
diff --git a/src/backend/access/nbtree/nbtsort.c b/src/backend/access/nbtree/nbtsort.c
index dc398e11867..5f6c9a5de78 100644
--- a/src/backend/access/nbtree/nbtsort.c
+++ b/src/backend/access/nbtree/nbtsort.c
@@ -65,6 +65,7 @@
 #include "access/xlog.h"
 #include "access/xloginsert.h"
 #include "catalog/index.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
 #include "pgstat.h"
 #include "storage/smgr.h"
@@ -288,7 +289,8 @@ static double _bt_parallel_heapscan(BTBuildState *buildstate,
 static void _bt_leader_participate_as_worker(BTBuildState *buildstate);
 static void _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2,
 						   BTShared *btshared, Sharedsort *sharedsort,
-						   Sharedsort *sharedsort2, int sortmem);
+						   Sharedsort *sharedsort2, int sortmem,
+						   bool progress);
 
 
 /*
@@ -469,14 +471,19 @@ _bt_spools_heapscan(Relation heap, Relation index, BTBuildState *buildstate,
 	}
 
 	/* Fill spool using either serial or parallel heap scan */
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+								 PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN);
 	if (!buildstate->btleader)
-		reltuples = IndexBuildHeapScan(heap, index, indexInfo, true,
+		reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, true,
 									   _bt_build_callback, (void *) buildstate,
 									   NULL);
 	else
 		reltuples = _bt_parallel_heapscan(buildstate,
 										  &indexInfo->ii_BrokenHotChain);
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_TOTAL,
+								 buildstate->indtuples);
+
 	/* okay, all heap tuples are spooled */
 	if (buildstate->spool2 && !buildstate->havedead)
 	{
@@ -525,9 +532,15 @@ _bt_leafbuild(BTSpool *btspool, BTSpool *btspool2)
 	}
 #endif							/* BTREE_BUILD_STATS */
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+								 PROGRESS_BTREE_PHASE_PERFORMSORT_1);
 	tuplesort_performsort(btspool->sortstate);
 	if (btspool2)
+	{
+		pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+									 PROGRESS_BTREE_PHASE_PERFORMSORT_2);
 		tuplesort_performsort(btspool2->sortstate);
+	}
 
 	wstate.heap = btspool->heap;
 	wstate.index = btspool->index;
@@ -543,6 +556,8 @@ _bt_leafbuild(BTSpool *btspool, BTSpool *btspool2)
 	wstate.btws_pages_written = 0;
 	wstate.btws_zeropage = NULL;	/* until needed */
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+								 PROGRESS_BTREE_PHASE_LEAF_LOAD);
 	_bt_load(&wstate, btspool, btspool2);
 }
 
@@ -1078,6 +1093,7 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
 				keysz = IndexRelationGetNumberOfKeyAttributes(wstate->index);
 	ScanKey		indexScanKey = NULL;
 	SortSupport sortKeys;
+	long		tuples_done = 0L;
 
 	if (merge)
 	{
@@ -1170,6 +1186,10 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
 				_bt_buildadd(wstate, state, itup2);
 				itup2 = tuplesort_getindextuple(btspool2->sortstate, true);
 			}
+
+			/* Report progress */
+			pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE,
+										 ++tuples_done);
 		}
 		pfree(sortKeys);
 	}
@@ -1184,6 +1204,10 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
 				state = _bt_pagestate(wstate, 0);
 
 			_bt_buildadd(wstate, state, itup);
+
+			/* Report progress */
+			pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE,
+										 ++tuples_done);
 		}
 	}
 
@@ -1317,6 +1341,8 @@ _bt_begin_parallel(BTBuildState *buildstate, bool isconcurrent, int request)
 	btshared->indtuples = 0.0;
 	btshared->brokenhotchain = false;
 	heap_parallelscan_initialize(&btshared->heapdesc, btspool->heap, snapshot);
+	pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL,
+								 btshared->heapdesc.phs_nblocks);
 
 	/*
 	 * Store shared tuplesort-private state, for which we reserved space.
@@ -1493,7 +1519,7 @@ _bt_leader_participate_as_worker(BTBuildState *buildstate)
 	/* Perform work common to all participants */
 	_bt_parallel_scan_and_sort(leaderworker, leaderworker2, btleader->btshared,
 							   btleader->sharedsort, btleader->sharedsort2,
-							   sortmem);
+							   sortmem, true);
 
 #ifdef BTREE_BUILD_STATS
 	if (log_btree_build_stats)
@@ -1584,7 +1610,7 @@ _bt_parallel_build_main(dsm_segment *seg, shm_toc *toc)
 	/* Perform sorting of spool, and possibly a spool2 */
 	sortmem = maintenance_work_mem / btshared->scantuplesortstates;
 	_bt_parallel_scan_and_sort(btspool, btspool2, btshared, sharedsort,
-							   sharedsort2, sortmem);
+							   sharedsort2, sortmem, false);
 
 #ifdef BTREE_BUILD_STATS
 	if (log_btree_build_stats)
@@ -1613,7 +1639,7 @@ _bt_parallel_build_main(dsm_segment *seg, shm_toc *toc)
 static void
 _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2,
 						   BTShared *btshared, Sharedsort *sharedsort,
-						   Sharedsort *sharedsort2, int sortmem)
+						   Sharedsort *sharedsort2, int sortmem, bool progress)
 {
 	SortCoordinate coordinate;
 	BTBuildState buildstate;
@@ -1672,7 +1698,7 @@ _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2,
 	indexInfo->ii_Concurrent = btshared->isconcurrent;
 	scan = heap_beginscan_parallel(btspool->heap, &btshared->heapdesc);
 	reltuples = IndexBuildHeapScan(btspool->heap, btspool->index, indexInfo,
-								   true, _bt_build_callback,
+								   true, progress, _bt_build_callback,
 								   (void *) &buildstate, scan);
 
 	/*
diff --git a/src/backend/access/nbtree/nbtutils.c b/src/backend/access/nbtree/nbtutils.c
index 2c05fb5e451..dc47663c374 100644
--- a/src/backend/access/nbtree/nbtutils.c
+++ b/src/backend/access/nbtree/nbtutils.c
@@ -20,6 +20,7 @@
 #include "access/nbtree.h"
 #include "access/reloptions.h"
 #include "access/relscan.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
 #include "utils/array.h"
 #include "utils/lsyscache.h"
@@ -2082,6 +2083,29 @@ btproperty(Oid index_oid, int attno,
 	}
 }
 
+/*
+ *	btphasename() -- Return name of phase, for index build progress report
+ */
+char *
+btphasename(int64 phasenum)
+{
+	switch (phasenum)
+	{
+		case PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE:
+			return "initializing (1/5)";
+		case PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN:
+			return "table scan (2/5)";
+		case PROGRESS_BTREE_PHASE_PERFORMSORT_1:
+			return "sorting tuples, spool 1 (3/5)";
+		case PROGRESS_BTREE_PHASE_PERFORMSORT_2:
+			return "sorting tuples, spool 2 (4/5)";
+		case PROGRESS_BTREE_PHASE_LEAF_LOAD:
+			return "final btree sort & load (5/5)";
+		default:
+			return NULL;
+	}
+}
+
 /*
  *	_bt_nonkey_truncate() -- create tuple without non-key suffix attributes.
  *
diff --git a/src/backend/access/spgist/spginsert.c b/src/backend/access/spgist/spginsert.c
index f428a151385..1bc671c7238 100644
--- a/src/backend/access/spgist/spginsert.c
+++ b/src/backend/access/spgist/spginsert.c
@@ -142,7 +142,7 @@ spgbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 											  "SP-GiST build temporary context",
 											  ALLOCSET_DEFAULT_SIZES);
 
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false,
 								   spgistBuildCallback, (void *) &buildstate,
 								   NULL);
 
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 8e63c1fad25..9db2fb1693c 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -67,6 +67,7 @@ spghandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = spgcostestimate;
 	amroutine->amoptions = spgoptions;
 	amroutine->amproperty = spgproperty;
+	amroutine->amphasename = NULL;
 	amroutine->amvalidate = spgvalidate;
 	amroutine->ambeginscan = spgbeginscan;
 	amroutine->amrescan = spgrescan;
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index d16c3d0ea50..2da7a1c6166 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -49,8 +49,9 @@
 #include "catalog/pg_trigger.h"
 #include "catalog/pg_type.h"
 #include "catalog/storage.h"
-#include "commands/tablecmds.h"
 #include "commands/event_trigger.h"
+#include "commands/progress.h"
+#include "commands/tablecmds.h"
 #include "commands/trigger.h"
 #include "executor/executor.h"
 #include "miscadmin.h"
@@ -58,6 +59,7 @@
 #include "nodes/nodeFuncs.h"
 #include "optimizer/optimizer.h"
 #include "parser/parser.h"
+#include "pgstat.h"
 #include "rewrite/rewriteManip.h"
 #include "storage/bufmgr.h"
 #include "storage/lmgr.h"
@@ -1597,7 +1599,7 @@ index_drop(Oid indexId, bool concurrent)
 		 * to acquire an exclusive lock on our table.  The lock code will
 		 * detect deadlock and error out properly.
 		 */
-		WaitForLockers(heaplocktag, AccessExclusiveLock);
+		WaitForLockers(heaplocktag, AccessExclusiveLock, true);
 
 		/*
 		 * No more predicate locks will be acquired on this index, and we're
@@ -1641,7 +1643,7 @@ index_drop(Oid indexId, bool concurrent)
 		 * Wait till every transaction that saw the old index state has
 		 * finished.
 		 */
-		WaitForLockers(heaplocktag, AccessExclusiveLock);
+		WaitForLockers(heaplocktag, AccessExclusiveLock, true);
 
 		/*
 		 * Re-open relations to allow us to complete our actions.
@@ -2291,6 +2293,25 @@ index_build(Relation heapRelation,
 						   save_sec_context | SECURITY_RESTRICTED_OPERATION);
 	save_nestlevel = NewGUCNestLevel();
 
+	/* Set up initial progress report status */
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_PHASE,
+			PROGRESS_CREATEIDX_SUBPHASE,
+			PROGRESS_CREATEIDX_TUPLES_DONE,
+			PROGRESS_CREATEIDX_TUPLES_TOTAL,
+			PROGRESS_SCAN_BLOCKS_DONE,
+			PROGRESS_SCAN_BLOCKS_TOTAL
+		};
+		const int64	val[] = {
+			PROGRESS_CREATEIDX_PHASE_BUILD,
+			PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE,
+			0, 0, 0, 0
+		};
+
+		pgstat_progress_update_multi_param(6, index, val);
+	}
+
 	/*
 	 * Call the access method's build procedure
 	 */
@@ -2428,13 +2449,14 @@ IndexBuildHeapScan(Relation heapRelation,
 				   Relation indexRelation,
 				   IndexInfo *indexInfo,
 				   bool allow_sync,
+				   bool progress,
 				   IndexBuildCallback callback,
 				   void *callback_state,
 				   HeapScanDesc scan)
 {
 	return IndexBuildHeapRangeScan(heapRelation, indexRelation,
 								   indexInfo, allow_sync,
-								   false,
+								   false, progress,
 								   0, InvalidBlockNumber,
 								   callback, callback_state, scan);
 }
@@ -2455,6 +2477,7 @@ IndexBuildHeapRangeScan(Relation heapRelation,
 						IndexInfo *indexInfo,
 						bool allow_sync,
 						bool anyvisible,
+						bool progress,
 						BlockNumber start_blockno,
 						BlockNumber numblocks,
 						IndexBuildCallback callback,
@@ -2476,6 +2499,8 @@ IndexBuildHeapRangeScan(Relation heapRelation,
 	TransactionId OldestXmin;
 	BlockNumber root_blkno = InvalidBlockNumber;
 	OffsetNumber root_offsets[MaxHeapTuplesPerPage];
+	BlockNumber blocks_done = 0;
+	BlockNumber	previous_blkno = InvalidBlockNumber;
 
 	/*
 	 * sanity checks
@@ -2592,6 +2617,45 @@ IndexBuildHeapRangeScan(Relation heapRelation,
 
 		CHECK_FOR_INTERRUPTS();
 
+		if (progress &&
+			((previous_blkno == InvalidBlockNumber) ||
+			(scan->rs_cblock != previous_blkno)))
+		{
+			/* we only do progress for full table scans */
+			Assert(numblocks == InvalidBlockNumber);
+
+			/*
+			 * Report the number of blocks we've moved forward.
+			 *
+			 * Parallel workers cause the leader process to skip some blocks,
+			 * so we subtract the current block number to the previous one to
+			 * determine how many have been read in total; but be careful when
+			 * we wrap around the last block in the table.
+			 */
+			if (scan->rs_cblock > previous_blkno)
+				blocks_done += scan->rs_cblock - previous_blkno;
+			else if (previous_blkno == InvalidBlockNumber)
+			{
+				/*
+				 * How many blocks have been read since the scan started.
+				 * Should normally be zero.
+				 */
+				blocks_done += scan->rs_cblock -
+					(scan->rs_parallel ?  scan->rs_parallel->phs_startblock :
+					 scan->rs_startblock);
+				Assert(blocks_done == 0);
+			}
+			else
+			{
+				/* wrapped around */
+				blocks_done += scan->rs_nblocks - previous_blkno + scan->rs_cblock;
+			}
+
+			pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
+										 blocks_done);
+			previous_blkno = scan->rs_cblock;
+		}
+
 		/*
 		 * When dealing with a HOT-chain of updated tuples, we want to index
 		 * the values of the live tuple (if any), but index it under the TID
@@ -3133,6 +3197,21 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot)
 	int			save_sec_context;
 	int			save_nestlevel;
 
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_PHASE,
+			PROGRESS_CREATEIDX_TUPLES_DONE,
+			PROGRESS_CREATEIDX_TUPLES_TOTAL,
+			PROGRESS_SCAN_BLOCKS_DONE,
+			PROGRESS_SCAN_BLOCKS_TOTAL
+		};
+		const int64	val[] = {
+			PROGRESS_CREATEIDX_PHASE_VALIDATE_IDXSCAN,
+			0, 0, 0, 0
+		};
+		pgstat_progress_update_multi_param(5, index, val);
+	}
+
 	/* Open and lock the parent heap relation */
 	heapRelation = table_open(heapId, ShareUpdateExclusiveLock);
 	/* And the target index relation */
@@ -3163,6 +3242,7 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot)
 	 */
 	ivinfo.index = indexRelation;
 	ivinfo.analyze_only = false;
+	ivinfo.report_progress = true;	/* XXX only for btree? */
 	ivinfo.estimated_count = true;
 	ivinfo.message_level = DEBUG2;
 	ivinfo.num_heap_tuples = heapRelation->rd_rel->reltuples;
@@ -3180,15 +3260,31 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot)
 											NULL, false);
 	state.htups = state.itups = state.tups_inserted = 0;
 
+	/* ambulkdelete updates progress metrics */
 	(void) index_bulk_delete(&ivinfo, NULL,
 							 validate_index_callback, (void *) &state);
 
 	/* Execute the sort */
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_SORT_IDXSCAN);
 	tuplesort_performsort(state.tuplesort);
 
 	/*
-	 * Now scan the heap and "merge" it with the index
+	 * Now scan the heap and "merge" it with the index.
 	 */
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_PHASE,
+			PROGRESS_SCAN_BLOCKS_DONE,
+			PROGRESS_SCAN_BLOCKS_TOTAL
+		};
+		const int64	val[] = {
+			PROGRESS_CREATEIDX_PHASE_HEAPSCAN_VALIDATE,
+			0, 0
+		};
+
+		pgstat_progress_update_multi_param(3, index, val);
+	}
 	validate_index_heapscan(heapRelation,
 							indexRelation,
 							indexInfo,
@@ -3330,6 +3426,9 @@ validate_index_heapscan(Relation heapRelation,
 								true,	/* buffer access strategy OK */
 								false); /* syncscan not OK */
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_TOTAL,
+								 state->itups);
+
 	/*
 	 * Scan all tuples matching the snapshot.
 	 */
@@ -3343,6 +3442,9 @@ validate_index_heapscan(Relation heapRelation,
 
 		state->htups += 1;
 
+		pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE,
+									 state->htups);
+
 		/*
 		 * As commented in IndexBuildHeapScan, we should index heap-only
 		 * tuples under the TIDs of their root tuples; so when we advance onto
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 3e229c693c4..22c163be859 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -906,6 +906,32 @@ CREATE VIEW pg_stat_progress_vacuum AS
     FROM pg_stat_get_progress_info('VACUUM') AS S
 		LEFT JOIN pg_database D ON S.datid = D.oid;
 
+CREATE VIEW pg_stat_progress_create_index AS
+	SELECT
+		s.pid AS pid, S.datid AS datid, D.datname AS datname,
+		S.relid AS relid,
+		CASE s.param2 WHEN 0 THEN 'initializing (phase 1 of 8)'
+					  WHEN 1 THEN 'waiting for lockers 1 (phase 2 of 8)'
+					  WHEN 2 THEN 'building index (3 of 8): ' ||
+						pg_indexam_progress_phasename(s.param1::oid, s.param3)
+					  WHEN 3 THEN 'waiting for lockers 2 (phase 4 of 8)'
+					  WHEN 4 THEN 'validating index scan (phase 5 of 8)'
+					  WHEN 5 THEN 'sorting index scan results (phase 6 of 8)'
+					  WHEN 6 THEN 'validate index heapscan (phase 7 of 8)'
+					  WHEN 7 THEN 'waiting for lockers 3 (phase 8 of 8)'
+					  END as phase,
+		S.param4 AS "lockers total",
+		S.param5 AS "lockers done",
+		S.param6 AS "PID of current locker",
+		S.param7 AS "blocks total",
+		S.param8 AS "blocks done",
+		S.param9 AS "tuples total",
+		S.param10 AS "tuples done",
+		S.param11 AS "partitions total",
+		S.param12 AS "partitions done"
+	FROM pg_stat_get_progress_info('CREATE INDEX') AS S
+		LEFT JOIN pg_database D ON S.datid = D.oid;
+
 CREATE VIEW pg_user_mappings AS
     SELECT
         U.oid       AS umid,
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 7352b9e341d..115d7c61359 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -36,6 +36,7 @@
 #include "commands/dbcommands.h"
 #include "commands/defrem.h"
 #include "commands/event_trigger.h"
+#include "commands/progress.h"
 #include "commands/tablecmds.h"
 #include "commands/tablespace.h"
 #include "mb/pg_wchar.h"
@@ -46,10 +47,12 @@
 #include "parser/parse_coerce.h"
 #include "parser/parse_func.h"
 #include "parser/parse_oper.h"
+#include "pgstat.h"
 #include "rewrite/rewriteManip.h"
 #include "storage/lmgr.h"
 #include "storage/proc.h"
 #include "storage/procarray.h"
+#include "storage/sinvaladt.h"
 #include "utils/acl.h"
 #include "utils/builtins.h"
 #include "utils/fmgroids.h"
@@ -368,6 +371,15 @@ DefineIndex(Oid relationId,
 	Snapshot	snapshot;
 	int			i;
 
+
+	/*
+	 * Start progress report.  If we're building a partition, this was already
+	 * done.
+	 */
+	if (!OidIsValid(parentIndexId))
+		pgstat_progress_start_command(PROGRESS_COMMAND_CREATE_INDEX,
+									  relationId);
+
 	/*
 	 * count key attributes in index
 	 */
@@ -584,6 +596,9 @@ DefineIndex(Oid relationId,
 	accessMethodId = accessMethodForm->oid;
 	amRoutine = GetIndexAmRoutine(accessMethodForm->amhandler);
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_ACCESS_METHOD_OID,
+								 accessMethodId);
+
 	if (stmt->unique && !amRoutine->amcanunique)
 		ereport(ERROR,
 				(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -864,6 +879,11 @@ DefineIndex(Oid relationId,
 	if (!OidIsValid(indexRelationId))
 	{
 		table_close(rel, NoLock);
+
+		/* If this is the top-level index, we're done */
+		if (!OidIsValid(parentIndexId))
+			pgstat_progress_end_command();
+
 		return address;
 	}
 
@@ -889,6 +909,9 @@ DefineIndex(Oid relationId,
 			TupleDesc	parentDesc;
 			Oid		   *opfamOids;
 
+			pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL,
+										 nparts);
+
 			memcpy(part_oids, partdesc->oids, sizeof(Oid) * nparts);
 
 			parentDesc = CreateTupleDescCopy(RelationGetDescr(rel));
@@ -1039,6 +1062,8 @@ DefineIndex(Oid relationId,
 								skip_build, quiet);
 				}
 
+				pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_DONE,
+											 i + 1);
 				pfree(attmap);
 			}
 
@@ -1073,6 +1098,8 @@ DefineIndex(Oid relationId,
 		 * Indexes on partitioned tables are not themselves built, so we're
 		 * done here.
 		 */
+		if (!OidIsValid(parentIndexId))
+			pgstat_progress_end_command();
 		return address;
 	}
 
@@ -1080,6 +1107,11 @@ DefineIndex(Oid relationId,
 	{
 		/* Close the heap and we're done, in the non-concurrent case */
 		table_close(rel, NoLock);
+
+		/* If this is the top-level index, we're done. */
+		if (!OidIsValid(parentIndexId))
+			pgstat_progress_end_command();
+
 		return address;
 	}
 
@@ -1131,7 +1163,9 @@ DefineIndex(Oid relationId,
 	 * exclusive lock on our table.  The lock code will detect deadlock and
 	 * error out properly.
 	 */
-	WaitForLockers(heaplocktag, ShareLock);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_1);
+	WaitForLockers(heaplocktag, ShareLock, true);
 
 	/*
 	 * At this moment we are sure that there are no transactions with the
@@ -1195,7 +1229,9 @@ DefineIndex(Oid relationId,
 	 * We once again wait until no transaction can have the table open with
 	 * the index marked as read-only for updates.
 	 */
-	WaitForLockers(heaplocktag, ShareLock);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_2);
+	WaitForLockers(heaplocktag, ShareLock, true);
 
 	/*
 	 * Now take the "reference snapshot" that will be used by validate_index()
@@ -1281,6 +1317,9 @@ DefineIndex(Oid relationId,
 	old_snapshots = GetCurrentVirtualXIDs(limitXmin, true, false,
 										  PROC_IS_AUTOVACUUM | PROC_IN_VACUUM,
 										  &n_old_snapshots);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_3);
+	pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, n_old_snapshots);
 
 	for (i = 0; i < n_old_snapshots; i++)
 	{
@@ -1316,7 +1355,14 @@ DefineIndex(Oid relationId,
 		}
 
 		if (VirtualTransactionIdIsValid(old_snapshots[i]))
+		{
+			PGPROC *holder = BackendIdGetProc(old_snapshots[i].backendId);
+			pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID,
+										 holder->pid);
 			VirtualXactLock(old_snapshots[i], true);
+		}
+
+		pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, i + 1);
 	}
 
 	/*
@@ -1339,6 +1385,8 @@ DefineIndex(Oid relationId,
 	 */
 	UnlockRelationIdForSession(&heaprelid, ShareUpdateExclusiveLock);
 
+	pgstat_progress_end_command();
+
 	return address;
 }
 
diff --git a/src/backend/storage/ipc/standby.c b/src/backend/storage/ipc/standby.c
index 4d10e57a803..a0a2b964703 100644
--- a/src/backend/storage/ipc/standby.c
+++ b/src/backend/storage/ipc/standby.c
@@ -401,7 +401,7 @@ ResolveRecoveryConflictWithLock(LOCKTAG locktag)
 		 */
 		VirtualTransactionId *backends;
 
-		backends = GetLockConflicts(&locktag, AccessExclusiveLock);
+		backends = GetLockConflicts(&locktag, AccessExclusiveLock, NULL);
 		ResolveRecoveryConflictWithVirtualXIDs(backends,
 											   PROCSIG_RECOVERY_CONFLICT_LOCK);
 	}
diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c
index e688ba81170..0b04b093782 100644
--- a/src/backend/storage/lmgr/lmgr.c
+++ b/src/backend/storage/lmgr/lmgr.c
@@ -19,9 +19,12 @@
 #include "access/transam.h"
 #include "access/xact.h"
 #include "catalog/catalog.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
+#include "pgstat.h"
 #include "storage/lmgr.h"
 #include "storage/procarray.h"
+#include "storage/sinvaladt.h"
 #include "utils/inval.h"
 
 
@@ -857,10 +860,12 @@ XactLockTableWaitErrorCb(void *arg)
  * after we obtained our initial list of lockers, we will not wait for them.
  */
 void
-WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
+WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
 {
 	List	   *holders = NIL;
 	ListCell   *lc;
+	int			total = 0;
+	int			done = 0;
 
 	/* Done if no locks to wait for */
 	if (list_length(locktags) == 0)
@@ -870,10 +875,17 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
 	foreach(lc, locktags)
 	{
 		LOCKTAG    *locktag = lfirst(lc);
+		int			count;
 
-		holders = lappend(holders, GetLockConflicts(locktag, lockmode));
+		holders = lappend(holders,
+						  GetLockConflicts(locktag, lockmode,
+										   progress ? &count : NULL));
+		total += count;
 	}
 
+	if (progress)
+		pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, total);
+
 	/*
 	 * Note: GetLockConflicts() never reports our own xid, hence we need not
 	 * check for that.  Also, prepared xacts are not reported, which is fine
@@ -887,10 +899,36 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
 
 		while (VirtualTransactionIdIsValid(*lockholders))
 		{
+			/*
+			 * If requested, publish who we're going to wait for.  This is not
+			 * 100% accurate if they're already gone, but we don't care.
+			 */
+			if (progress)
+			{
+				PGPROC *holder = BackendIdGetProc(lockholders->backendId);
+
+				pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID,
+											 holder->pid);
+			}
 			VirtualXactLock(*lockholders, true);
 			lockholders++;
+
+			if (progress)
+				pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, ++done);
 		}
 	}
+	if (progress)
+	{
+		const int	index[] = {
+			PROGRESS_WAITFOR_TOTAL,
+			PROGRESS_WAITFOR_DONE,
+			PROGRESS_WAITFOR_CURRENT_PID
+		};
+		const int64	values[] = {
+			0, 0, 0
+		};
+		pgstat_progress_update_multi_param(3, index, values);
+	}
 
 	list_free_deep(holders);
 }
@@ -901,12 +939,12 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
  * Same as WaitForLockersMultiple, for a single lock tag.
  */
 void
-WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode)
+WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress)
 {
 	List	   *l;
 
 	l = list_make1(&heaplocktag);
-	WaitForLockersMultiple(l, lockmode);
+	WaitForLockersMultiple(l, lockmode, progress);
 	list_free(l);
 }
 
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 3bb5ce350aa..58ba90d0646 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2807,6 +2807,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock)
  *		xacts merely awaiting such a lock are NOT reported.
  *
  * The result array is palloc'd and is terminated with an invalid VXID.
+ * *countp, if not null, is updated to the number of items set.
  *
  * Of course, the result could be out of date by the time it's returned,
  * so use of this function has to be thought about carefully.
@@ -2817,7 +2818,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock)
  * uses of the result.
  */
 VirtualTransactionId *
-GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode)
+GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
 {
 	static VirtualTransactionId *vxids;
 	LOCKMETHODID lockmethodid = locktag->locktag_lockmethodid;
@@ -2964,6 +2965,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode)
 		LWLockRelease(partitionLock);
 		vxids[count].backendId = InvalidBackendId;
 		vxids[count].localTransactionId = InvalidLocalTransactionId;
+		if (countp)
+			*countp = count;
 		return vxids;
 	}
 
@@ -3019,6 +3022,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode)
 
 	vxids[count].backendId = InvalidBackendId;
 	vxids[count].localTransactionId = InvalidLocalTransactionId;
+	if (countp)
+		*countp = count;
 	return vxids;
 }
 
diff --git a/src/backend/utils/adt/amutils.c b/src/backend/utils/adt/amutils.c
index 060ffe501ec..05046e9b559 100644
--- a/src/backend/utils/adt/amutils.c
+++ b/src/backend/utils/adt/amutils.c
@@ -445,3 +445,26 @@ pg_index_column_has_property(PG_FUNCTION_ARGS)
 
 	return indexam_property(fcinfo, propname, InvalidOid, relid, attno);
 }
+
+/*
+ * Return the name of the given phase, as used for progress reporting by the
+ * given AM.
+ */
+Datum
+pg_indexam_progress_phasename(PG_FUNCTION_ARGS)
+{
+	Oid			amoid = PG_GETARG_OID(0);
+	int32		phasenum = PG_GETARG_INT32(1);
+	IndexAmRoutine *routine;
+	char	   *name;
+
+	routine = GetIndexAmRoutineByAmId(amoid, true);
+	if (routine == NULL || !routine->amphasename)
+		PG_RETURN_NULL();
+
+	name = routine->amphasename(phasenum);
+	if (!name)
+		PG_RETURN_NULL();
+
+	PG_RETURN_TEXT_P(CStringGetTextDatum(name));
+}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index b6ba856ebe6..91dc5fed9cd 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -468,6 +468,8 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
 	/* Translate command name into command type code. */
 	if (pg_strcasecmp(cmd, "VACUUM") == 0)
 		cmdtype = PROGRESS_COMMAND_VACUUM;
+	else if (pg_strcasecmp(cmd, "CREATE INDEX") == 0)
+		cmdtype = PROGRESS_COMMAND_CREATE_INDEX;
 	else
 		ereport(ERROR,
 				(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 653ddc976ba..471625db14f 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -108,6 +108,9 @@ typedef bool (*amproperty_function) (Oid index_oid, int attno,
 									 IndexAMProperty prop, const char *propname,
 									 bool *res, bool *isnull);
 
+/* name of phase as used in progress reporting */
+typedef char *(*amphasename_function) (int64 phasenum);
+
 /* validate definition of an opclass for this AM */
 typedef bool (*amvalidate_function) (Oid opclassoid);
 
@@ -213,6 +216,7 @@ typedef struct IndexAmRoutine
 	amcostestimate_function amcostestimate;
 	amoptions_function amoptions;
 	amproperty_function amproperty; /* can be NULL */
+	amphasename_function amphasename;	/* can be NULL */
 	amvalidate_function amvalidate;
 	ambeginscan_function ambeginscan;
 	amrescan_function amrescan;
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index c4aba39496f..f77d9eea8e8 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -45,6 +45,7 @@ typedef struct IndexVacuumInfo
 {
 	Relation	index;			/* the index being vacuumed */
 	bool		analyze_only;	/* ANALYZE (without any actual vacuum) */
+	bool		report_progress;	/* emit progress.h status reports */
 	bool		estimated_count;	/* num_heap_tuples is an estimate */
 	int			message_level;	/* ereport level for progress messages */
 	double		num_heap_tuples;	/* tuples remaining in heap */
diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h
index 4fb92d60a12..b53b9016870 100644
--- a/src/include/access/nbtree.h
+++ b/src/include/access/nbtree.h
@@ -488,6 +488,16 @@ typedef BTScanOpaqueData *BTScanOpaque;
 #define SK_BT_DESC			(INDOPTION_DESC << SK_BT_INDOPTION_SHIFT)
 #define SK_BT_NULLS_FIRST	(INDOPTION_NULLS_FIRST << SK_BT_INDOPTION_SHIFT)
 
+/*
+ * Constant definition for progress reporting.  Phase numbers must match
+ * btphasename.
+ */
+/* PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE is 1 (see progress.h) */
+#define PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN		2
+#define PROGRESS_BTREE_PHASE_PERFORMSORT_1				3
+#define PROGRESS_BTREE_PHASE_PERFORMSORT_2				4
+#define PROGRESS_BTREE_PHASE_LEAF_LOAD					5
+
 /*
  * external entry points for btree, in nbtree.c
  */
@@ -600,6 +610,7 @@ extern bytea *btoptions(Datum reloptions, bool validate);
 extern bool btproperty(Oid index_oid, int attno,
 		   IndexAMProperty prop, const char *propname,
 		   bool *res, bool *isnull);
+extern char *btphasename(int64 phasenum);
 extern IndexTuple _bt_nonkey_truncate(Relation rel, IndexTuple itup);
 extern bool _bt_check_natts(Relation rel, Page page, OffsetNumber offnum);
 
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index 330c481a8b7..fb713d50f96 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -114,6 +114,7 @@ extern double IndexBuildHeapScan(Relation heapRelation,
 				   Relation indexRelation,
 				   IndexInfo *indexInfo,
 				   bool allow_sync,
+				   bool progress,
 				   IndexBuildCallback callback,
 				   void *callback_state,
 				   struct HeapScanDescData *scan);
@@ -122,6 +123,7 @@ extern double IndexBuildHeapRangeScan(Relation heapRelation,
 						IndexInfo *indexInfo,
 						bool allow_sync,
 						bool anyvisible,
+						bool progress,
 						BlockNumber start_blockno,
 						BlockNumber end_blockno,
 						IndexBuildCallback callback,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 24f99f7fc45..e523c013a11 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -917,6 +917,10 @@
   proname => 'pg_index_column_has_property', provolatile => 's',
   prorettype => 'bool', proargtypes => 'regclass int4 text',
   prosrc => 'pg_index_column_has_property' },
+{ oid => '676', descr => 'return name of given index build phase',
+  proname => 'pg_indexam_progress_phasename', provolatile => 'i',
+  prorettype => 'text', proargtypes => 'oid int8',
+  prosrc => 'pg_indexam_progress_phasename' },
 
 { oid => '339',
   proname => 'poly_same', prorettype => 'bool',
@@ -5079,9 +5083,9 @@
   proname => 'pg_stat_get_progress_info', prorows => '100', proretset => 't',
   provolatile => 's', proparallel => 'r', prorettype => 'record',
   proargtypes => 'text',
-  proallargtypes => '{text,int4,oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}',
-  proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o}',
-  proargnames => '{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10}',
+  proallargtypes => '{text,int4,oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}',
+  proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+  proargnames => '{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10,param11,param12}',
   prosrc => 'pg_stat_get_progress_info' },
 { oid => '3099',
   descr => 'statistics: information about currently active replication',
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index 9858b36a383..4dae1e7438d 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -34,4 +34,40 @@
 #define PROGRESS_VACUUM_PHASE_TRUNCATE			5
 #define PROGRESS_VACUUM_PHASE_FINAL_CLEANUP		6
 
+
+/* Progress parameters for CREATE INDEX */
+#define PROGRESS_CREATEIDX_ACCESS_METHOD_OID	0
+#define PROGRESS_CREATEIDX_PHASE				1
+#define PROGRESS_CREATEIDX_SUBPHASE				2
+/* 3, 4 and 5 reserved for "waitfor" metrics */
+/* 6 and 7 reserved for "block number" metrics */
+#define PROGRESS_CREATEIDX_TUPLES_TOTAL			8
+#define PROGRESS_CREATEIDX_TUPLES_DONE			9
+#define PROGRESS_CREATEIDX_PARTITIONS_TOTAL		10
+#define PROGRESS_CREATEIDX_PARTITIONS_DONE		11
+
+/* Phases of CREATE INDEX */
+#define PROGRESS_CREATEIDX_PHASE_WAIT_1			1
+#define PROGRESS_CREATEIDX_PHASE_BUILD			2
+#define PROGRESS_CREATEIDX_PHASE_WAIT_2			3
+#define PROGRESS_CREATEIDX_PHASE_VALIDATE_IDXSCAN		4
+#define PROGRESS_CREATEIDX_PHASE_SORT_IDXSCAN		5
+#define PROGRESS_CREATEIDX_PHASE_HEAPSCAN_VALIDATE		6
+#define PROGRESS_CREATEIDX_PHASE_WAIT_3			7
+
+/*
+ * Subphases of CREATE INDEX, for index_build.
+ */
+#define PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE	1
+/* Additional phases are defined by each AM */
+
+/* Lock holder wait counts */
+#define PROGRESS_WAITFOR_TOTAL					3
+#define PROGRESS_WAITFOR_DONE					4
+#define PROGRESS_WAITFOR_CURRENT_PID			5
+
+/* Block numbers in a generic relation scan */
+#define PROGRESS_SCAN_BLOCKS_TOTAL				6
+#define PROGRESS_SCAN_BLOCKS_DONE				7
+
 #endif
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 88a75fb798e..f931ead1c27 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -934,10 +934,11 @@ typedef enum
 typedef enum ProgressCommandType
 {
 	PROGRESS_COMMAND_INVALID,
-	PROGRESS_COMMAND_VACUUM
+	PROGRESS_COMMAND_VACUUM,
+	PROGRESS_COMMAND_CREATE_INDEX
 } ProgressCommandType;
 
-#define PGSTAT_NUM_PROGRESS_PARAM	10
+#define PGSTAT_NUM_PROGRESS_PARAM	12
 
 /* ----------
  * Shared-memory data structures
diff --git a/src/include/storage/lmgr.h b/src/include/storage/lmgr.h
index 3d705faba5c..4f2872de35f 100644
--- a/src/include/storage/lmgr.h
+++ b/src/include/storage/lmgr.h
@@ -78,8 +78,8 @@ extern void XactLockTableWait(TransactionId xid, Relation rel,
 extern bool ConditionalXactLockTableWait(TransactionId xid);
 
 /* Lock VXIDs, specified by conflicting locktags */
-extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode);
-extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode);
+extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress);
+extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress);
 
 /* Lock an XID for tuple insertion (used to wait for an insertion to finish) */
 extern uint32 SpeculativeInsertionLockAcquire(TransactionId xid);
diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h
index 16b927cb801..e117b391774 100644
--- a/src/include/storage/lock.h
+++ b/src/include/storage/lock.h
@@ -544,7 +544,7 @@ extern bool LockHeldByMe(const LOCKTAG *locktag, LOCKMODE lockmode);
 extern bool LockHasWaiters(const LOCKTAG *locktag,
 			   LOCKMODE lockmode, bool sessionLock);
 extern VirtualTransactionId *GetLockConflicts(const LOCKTAG *locktag,
-				 LOCKMODE lockmode);
+				 LOCKMODE lockmode, int *countp);
 extern void AtPrepare_Locks(void);
 extern void PostPrepare_Locks(TransactionId xid);
 extern int LockCheckConflicts(LockMethod lockMethodTable,
#9Simon Riggs
simon@2ndquadrant.com
In reply to: Alvaro Herrera (#8)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On Wed, 13 Feb 2019 at 00:46, Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:

Here's a sample
concurrent index build on a 100M tuple table.

Cool

There are no concurrent
transactions, so phases that wait for lockers don't appear.

Would prefer to see them, so we know they are zero.

--
Simon Riggs http://www.2ndQuadrant.com/
<http://www.2ndquadrant.com/&gt;
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#10Amit Langote
Langote_Amit_f8@lab.ntt.co.jp
In reply to: Simon Riggs (#9)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On 2019/02/13 11:59, Simon Riggs wrote:

On Wed, 13 Feb 2019 at 00:46, Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:

Here's a sample
concurrent index build on a 100M tuple table.

Cool

+1

Looking at the "([phase] x of x)" in the sample output, I thought
pg_stat_progress_vacuum's output should've had it too (not a concern of
Alvaro's patch though.)

Thanks,
Amit

#11Amit Langote
Langote_Amit_f8@lab.ntt.co.jp
In reply to: Alvaro Herrera (#7)
Re: monitoring CREATE INDEX [CONCURRENTLY]

Hi Alvaro,

On 2019/02/12 12:18, Alvaro Herrera wrote:

I ended up defining phases for the index_build phase in the AM itself;
the code reports a phase number using the regular API, and then the
pgstat_progress view obtains the name of each phase using a support
method.

diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index 05102724ead..189179a5667 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -127,6 +127,7 @@ typedef struct IndexAmRoutine
     amcostestimate_function amcostestimate;
     amoptions_function amoptions;
     amproperty_function amproperty;     /* can be NULL */
+    amphasename_function amphasename;   /* can be NULL */

Doesn't the name amphasename sound a bit too generic, given that it can
only describe the phases of ambuild? Maybe ambuildphase?

Thanks,
Amit

#12Tatsuro Yamada
yamada.tatsuro@lab.ntt.co.jp
In reply to: Alvaro Herrera (#8)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On 2019/02/13 4:16, Alvaro Herrera wrote:

I added metrics for the validate_index phases; patch attached. This is
still a bit raw, but it looks much better now. Here's a sample
concurrent index build on a 100M tuple table. There are no concurrent

Great!

+		s.pid AS pid, S.datid AS datid, D.datname AS datname,
+		S.relid AS relid,
+		CASE s.param2 WHEN 0 THEN 'initializing (phase 1 of 8)'
+					  WHEN 1 THEN 'waiting for lockers 1 (phase 2 of 8)'
+					  WHEN 2 THEN 'building index (3 of 8): ' ||
+						pg_indexam_progress_phasename(s.param1::oid, s.param3)

It would be better to replace "s" with "S".

s/s.pid/S.pid/
s/s.param2/S.param2/
s/s.param1::oid, s.param3/S.param1::oid, S.param3/

These are not important comments but for readability. :)

Thanks,
Tatsuro Yamada

#13Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Amit Langote (#11)
1 attachment(s)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On 2019-Feb-13, Amit Langote wrote:

Doesn't the name amphasename sound a bit too generic, given that it can
only describe the phases of ambuild? Maybe ambuildphase?

Hmm, yeah, maybe it does. I renamed it "ambuildphasename", since it's
not about reporting the phase itself -- it's about translating the phase
number to the string that's reported to the user.

The attached patch does it that way. Also, when an index build uses an
AM that doesn't support progress reporting, it no longer reports a NULL
phase name while building. I also changed it to report the progress of
phase 7 (heap scan validation) using block numbers rather than tuple
counts. I also tweaked the strings reported in the view. They're
clearer now IMO.

One slight annoyance is that when parallel workers are used, the last
block number reported in phase 3/subphase 2 (IndexBuildHeapScan stuff)
is not necessarily accurate, since the tail of the table could well be
scanned by a worker that's not the leader, and we only report in the
leader when it gets a new block.

When the AM does not support progress reporting, everything stays as
zeros during the index build phase. It's easy to notice how slow hash
indexes are to build compared to btrees this way! Maybe it'd be
better fallback on reporting block numbers in IndexBuildHeapScan when
this happens. Thoughts?

I added docs to the monitoring section -- that's the bulkiest part of
the patch.

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attachments:

0001-Report-progress-of-CREATE-INDEX-operations.patchtext/x-diff; charset=us-asciiDownload
From e8eae6d3d5af45ecb45171745a4af374013baffe Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Wed, 2 Jan 2019 16:14:39 -0300
Subject: [PATCH] Report progress of CREATE INDEX operations

---
 contrib/amcheck/verify_nbtree.c       |   2 +-
 contrib/bloom/blinsert.c              |   2 +-
 contrib/bloom/blutils.c               |   1 +
 doc/src/sgml/indexam.sgml             |  13 ++
 doc/src/sgml/monitoring.sgml          | 227 +++++++++++++++++++++++++-
 src/backend/access/brin/brin.c        |   5 +-
 src/backend/access/gin/gininsert.c    |   2 +-
 src/backend/access/gin/ginutil.c      |   1 +
 src/backend/access/gist/gist.c        |   1 +
 src/backend/access/gist/gistbuild.c   |   2 +-
 src/backend/access/hash/hash.c        |   3 +-
 src/backend/access/nbtree/nbtree.c    |   9 +
 src/backend/access/nbtree/nbtsort.c   |  43 ++++-
 src/backend/access/nbtree/nbtutils.c  |  24 +++
 src/backend/access/spgist/spginsert.c |   2 +-
 src/backend/access/spgist/spgutils.c  |   1 +
 src/backend/catalog/index.c           | 134 ++++++++++++++-
 src/backend/catalog/system_views.sql  |  27 +++
 src/backend/commands/indexcmds.c      |  52 +++++-
 src/backend/storage/ipc/standby.c     |   2 +-
 src/backend/storage/lmgr/lmgr.c       |  46 +++++-
 src/backend/storage/lmgr/lock.c       |   7 +-
 src/backend/utils/adt/amutils.c       |  23 +++
 src/backend/utils/adt/pgstatfuncs.c   |   2 +
 src/include/access/amapi.h            |   4 +
 src/include/access/genam.h            |   1 +
 src/include/access/nbtree.h           |  11 ++
 src/include/catalog/index.h           |   2 +
 src/include/catalog/pg_proc.dat       |  10 +-
 src/include/commands/progress.h       |  36 ++++
 src/include/pgstat.h                  |   5 +-
 src/include/storage/lmgr.h            |   4 +-
 src/include/storage/lock.h            |   2 +-
 33 files changed, 668 insertions(+), 38 deletions(-)

diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index 964200a7678..99f6ed6bc44 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -534,7 +534,7 @@ bt_check_every_level(Relation rel, Relation heaprel, bool readonly,
 			 RelationGetRelationName(state->rel),
 			 RelationGetRelationName(state->heaprel));
 
-		IndexBuildHeapScan(state->heaprel, state->rel, indexinfo, true,
+		IndexBuildHeapScan(state->heaprel, state->rel, indexinfo, true, false,
 						   bt_tuple_present_callback, (void *) state, scan);
 
 		ereport(DEBUG1,
diff --git a/contrib/bloom/blinsert.c b/contrib/bloom/blinsert.c
index e43fbe0005f..947ee74881f 100644
--- a/contrib/bloom/blinsert.c
+++ b/contrib/bloom/blinsert.c
@@ -141,7 +141,7 @@ blbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	initCachedPage(&buildstate);
 
 	/* Do the heap scan */
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false,
 								   bloomBuildCallback, (void *) &buildstate,
 								   NULL);
 
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 64583765787..697a37a384b 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -132,6 +132,7 @@ blhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = blcostestimate;
 	amroutine->amoptions = bloptions;
 	amroutine->amproperty = NULL;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = blvalidate;
 	amroutine->ambeginscan = blbeginscan;
 	amroutine->amrescan = blrescan;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index 05102724ead..fa4a3d0d131 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -127,6 +127,7 @@ typedef struct IndexAmRoutine
     amcostestimate_function amcostestimate;
     amoptions_function amoptions;
     amproperty_function amproperty;     /* can be NULL */
+    ambuildphasename_function ambuildphasename;   /* can be NULL */
     amvalidate_function amvalidate;
     ambeginscan_function ambeginscan;
     amrescan_function amrescan;
@@ -468,6 +469,18 @@ amproperty (Oid index_oid, int attno,
 
   <para>
 <programlisting>
+char *
+ambuildphasename (int64 phasenum);
+</programlisting>
+   Return the textual name of the given build phase number.
+   The phase numbers are those reported during an index build via the
+   <function>pgstat_progress_update_param</function> interface.
+   The phase names are then exposed in the
+   <structname>pg_stat_progress_create_index</structname> view.
+  </para>
+
+  <para>
+<programlisting>
 bool
 amvalidate (Oid opclassoid);
 </programlisting>
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 0e73cdcddab..47708570f84 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -336,6 +336,14 @@ postgres   27093  0.0  0.0  30096  2752 ?        Ss   11:34   0:00 postgres: ser
       </entry>
      </row>
 
+     <row>
+      <entry><structname>pg_stat_progress_create_index</structname><indexterm><primary>pg_stat_progress_create_index</primary></indexterm></entry>
+      <entry>One row for each backend running <command>CREATE INDEX</command>, showing
+      current progress.
+      See <xref linkend='create-index-progress-reporting'/>.
+     </entry>
+     </row>
+
      <row>
       <entry><structname>pg_stat_progress_vacuum</structname><indexterm><primary>pg_stat_progress_vacuum</primary></indexterm></entry>
       <entry>One row for each backend (including autovacuum worker processes) running
@@ -3376,11 +3384,224 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
 
   <para>
    <productname>PostgreSQL</productname> has the ability to report the progress of
-   certain commands during command execution.  Currently, the only command
-   which supports progress reporting is <command>VACUUM</command>.  This may be
-   expanded in the future.
+   certain commands during command execution.  Currently, the only commands
+   which support progress reporting are <command>CREATE INDEX</command> and
+   <command>VACUUM</command>.  This may be expanded in the future.
   </para>
 
+ <sect2 id="create-index-progress-reporting">
+  <title>CREATE INDEX Progress Reporting</title>
+
+  <para>
+   Whenever <command>CREATE INDEX</command> is running, the
+   <structname>pg_stat_progress_create_index</structname> view will contain
+   one row for each backend that is currently creating indexes.  The tables
+   below describe the information that will be reported and provide information
+   about how to interpret it.
+  </para>
+
+  <table id="pg-stat-progress-create-index-view" xreflabel="pg_stat_progress_create_index">
+   <title><structname>pg_stat_progress_create_index</structname> View</title>
+   <tgroup cols="3">
+    <thead>
+     <row>
+      <entry>Column</entry>
+      <entry>Type</entry>
+      <entry>Description</entry>
+     </row>
+    </thead>
+
+    <tbody>
+     <row>
+      <entry><structfield>pid</structfield></entry>
+      <entry><type>integer</type></entry>
+      <entry>Process ID of backend.</entry>
+     </row>
+     <row>
+      <entry><structfield>datid</structfield></entry>
+      <entry><type>oid</type></entry>
+      <entry>OID of the database to which this backend is connected.</entry>
+     </row>
+     <row>
+      <entry><structfield>datname</structfield></entry>
+      <entry><type>name</type></entry>
+      <entry>Name of the database to which this backend is connected.</entry>
+     </row>
+     <row>
+      <entry><structfield>relid</structfield></entry>
+      <entry><type>oid</type></entry>
+      <entry>OID of the table on which the index is being created.</entry>
+     </row>
+     <row>
+      <entry><structfield>phase</structfield></entry>
+      <entry><type>text</type></entry>
+      <entry>
+        Current processing phase of index creation.  See <xref linkend='create-index-phases'/>.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>lockers_total</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Total number of lockers to wait for, when applicable.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>lockers_done</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Number of lockers already waited for.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>current_locked_pid</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Process ID of the locker currently being waited for.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>blocks_total</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Total number of blocks to be processed in the current phase.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>blocks_done</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Number of blocks already processed in the current phase.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>tuples_total</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Total number of tuples to be processed in the current phase.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>tuples_done</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Number of tuples already processed in the current phase.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>partitions_total</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+       When creating an index on a partitioned, this column is set to the
+       total number of partitions on which the index is to be created.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>partitions_done</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+       When creating an index on a partitioned, this column is set to the
+       number of partitions on which the index has been completed.
+      </entry>
+     </row>
+    </tbody>
+   </tgroup>
+  </table>
+
+  <table id="create-index-phases">
+   <title>CREATE INDEX phases</title>
+   <tgroup cols="2">
+    <thead>
+     <row>
+      <entry>Phase</entry>
+      <entry>Description</entry>
+     </row>
+    </thead>
+    <tbody>
+     <row>
+      <entry><literal>initializing</literal></entry>
+      <entry>
+       <command>CREATE INDEX</command> is preparing to create the index.  This
+       phase is expected to be very brief.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>waiting for old snapshots</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions
+       that can potentially see the table to release their snapshots.
+       This phase is skipped when not in concurrent mode.
+       Columns <structname>lockers_total</structname>, <structname>lockers_done</structname>
+       and <structname>current_locker_pid</structname> contain the progress 
+       information for this phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>building index</literal></entry>
+      <entry>
+       The index is being built by the access method-specific code.  In this phase,
+       access methods that support progress reporting fill in their own progress data,
+       and the subphase is indicated in this column.  Typically,
+       <structname>blocks_total</structname> and <structname>blocks_done</structname>
+       will contain progress data, as well as potentially
+       <structname>tuples_total</structname> and <structname>tuples_done</structname>.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>waiting for writer snapshots</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions
+       that can potentially write into the table to release their snapshots.
+       This phase is skipped when not in concurrent mode.
+       Columns <structname>lockers_total</structname>, <structname>lockers_done</structname>
+       and <structname>current_locker_pid</structname> contain the progress 
+       information for this phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>validating index scan</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is scanning the index searching
+       for tuples that need to be validated.
+       This phase is skipped when not in concurrent mode.
+       Columns <structname>blocks_total</structname> (set to the total size of the index)
+       and <structname>blocks_done</structname> contain the progress information for this phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>sorting index scan results</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is sorting the output of the
+       previous phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>validating index heapscan</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is scanning the table
+       to validate the index tuples collected in the previous two phases.
+       This phase is skipped when not in concurrent mode.
+       Columns <structname>blocks_total</structname> (set to the total size of the table)
+       and <structname>blocks_done</structname> contain the progress information for this phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>waiting for reader snapshots</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions
+       that can potentially see the table to release their snapshots.  This
+       phase is skipped when not in concurrent mode.
+       Columns <structname>lockers_total</structname>, <structname>lockers_done</structname>
+       and <structname>current_locker_pid</structname> contain the progress 
+       information for this phase.
+      </entry>
+     </row>
+    </tbody>
+   </tgroup>
+  </table>
+
+ </sect2>
+
  <sect2 id="vacuum-progress-reporting">
   <title>VACUUM Progress Reporting</title>
 
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 8f008dd0080..5d1aff34080 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -111,6 +111,7 @@ brinhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = brincostestimate;
 	amroutine->amoptions = brinoptions;
 	amroutine->amproperty = NULL;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = brinvalidate;
 	amroutine->ambeginscan = brinbeginscan;
 	amroutine->amrescan = brinrescan;
@@ -718,7 +719,7 @@ brinbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	 * Now scan the relation.  No syncscan allowed here because we want the
 	 * heap blocks in physical order.
 	 */
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, false,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, false, false,
 								   brinbuildCallback, (void *) state, NULL);
 
 	/* process the final batch */
@@ -1234,7 +1235,7 @@ summarize_range(IndexInfo *indexInfo, BrinBuildState *state, Relation heapRel,
 	 * by transactions that are still in progress, among other corner cases.
 	 */
 	state->bs_currRangeStart = heapBlk;
-	IndexBuildHeapRangeScan(heapRel, state->bs_irel, indexInfo, false, true,
+	IndexBuildHeapRangeScan(heapRel, state->bs_irel, indexInfo, false, true, false,
 							heapBlk, scanNumBlks,
 							brinbuildCallback, (void *) state, NULL);
 
diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c
index 524ac5be8b5..838de4c1ec3 100644
--- a/src/backend/access/gin/gininsert.c
+++ b/src/backend/access/gin/gininsert.c
@@ -394,7 +394,7 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	 * Do the heap scan.  We disallow sync scan here because dataPlaceToPage
 	 * prefers to receive tuples in TID order.
 	 */
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, false,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, false, false,
 								   ginBuildCallback, (void *) &buildstate, NULL);
 
 	/* dump remaining entries to the index */
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index afc20232ace..d2360eeafb0 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -64,6 +64,7 @@ ginhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = gincostestimate;
 	amroutine->amoptions = ginoptions;
 	amroutine->amproperty = NULL;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = ginvalidate;
 	amroutine->ambeginscan = ginbeginscan;
 	amroutine->amrescan = ginrescan;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index b75b3a8dacd..0dc36af1e0c 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = gistcostestimate;
 	amroutine->amoptions = gistoptions;
 	amroutine->amproperty = gistproperty;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = gistvalidate;
 	amroutine->ambeginscan = gistbeginscan;
 	amroutine->amrescan = gistrescan;
diff --git a/src/backend/access/gist/gistbuild.c b/src/backend/access/gist/gistbuild.c
index bd142a3560d..015f874cc93 100644
--- a/src/backend/access/gist/gistbuild.c
+++ b/src/backend/access/gist/gistbuild.c
@@ -204,7 +204,7 @@ gistbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	/*
 	 * Do the heap scan.
 	 */
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false,
 								   gistBuildCallback, (void *) &buildstate, NULL);
 
 	/*
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index f1f01a0956d..fc7db5d6a13 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -82,6 +82,7 @@ hashhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = hashcostestimate;
 	amroutine->amoptions = hashoptions;
 	amroutine->amproperty = NULL;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = hashvalidate;
 	amroutine->ambeginscan = hashbeginscan;
 	amroutine->amrescan = hashrescan;
@@ -159,7 +160,7 @@ hashbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	buildstate.heapRel = heap;
 
 	/* do the heap scan */
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false,
 								   hashbuildCallback, (void *) &buildstate, NULL);
 
 	if (buildstate.spool)
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index 98917de2efd..f9d32c958f6 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -22,6 +22,7 @@
 #include "access/nbtxlog.h"
 #include "access/relscan.h"
 #include "access/xlog.h"
+#include "commands/progress.h"
 #include "commands/vacuum.h"
 #include "miscadmin.h"
 #include "nodes/execnodes.h"
@@ -133,6 +134,7 @@ bthandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = btcostestimate;
 	amroutine->amoptions = btoptions;
 	amroutine->amproperty = btproperty;
+	amroutine->ambuildphasename = btbuildphasename;
 	amroutine->amvalidate = btvalidate;
 	amroutine->ambeginscan = btbeginscan;
 	amroutine->amrescan = btrescan;
@@ -1021,6 +1023,10 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
 		if (needLock)
 			UnlockRelationForExtension(rel, ExclusiveLock);
 
+		if (info->report_progress)
+			pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL,
+										 num_pages);
+
 		/* Quit if we've scanned the whole relation */
 		if (blkno >= num_pages)
 			break;
@@ -1028,6 +1034,9 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
 		for (; blkno < num_pages; blkno++)
 		{
 			btvacuumpage(&vstate, blkno, blkno);
+			if (info->report_progress)
+				pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
+											 blkno);
 		}
 	}
 
diff --git a/src/backend/access/nbtree/nbtsort.c b/src/backend/access/nbtree/nbtsort.c
index dc398e11867..804280385aa 100644
--- a/src/backend/access/nbtree/nbtsort.c
+++ b/src/backend/access/nbtree/nbtsort.c
@@ -65,6 +65,7 @@
 #include "access/xlog.h"
 #include "access/xloginsert.h"
 #include "catalog/index.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
 #include "pgstat.h"
 #include "storage/smgr.h"
@@ -288,7 +289,8 @@ static double _bt_parallel_heapscan(BTBuildState *buildstate,
 static void _bt_leader_participate_as_worker(BTBuildState *buildstate);
 static void _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2,
 						   BTShared *btshared, Sharedsort *sharedsort,
-						   Sharedsort *sharedsort2, int sortmem);
+						   Sharedsort *sharedsort2, int sortmem,
+						   bool progress);
 
 
 /*
@@ -384,6 +386,10 @@ _bt_spools_heapscan(Relation heap, Relation index, BTBuildState *buildstate,
 	/* Save as primary spool */
 	buildstate->spool = btspool;
 
+	/* Report heap scan phase started */
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+								 PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN);
+
 	/* Attempt to launch parallel worker scan when required */
 	if (indexInfo->ii_ParallelWorkers > 0)
 		_bt_begin_parallel(buildstate, indexInfo->ii_Concurrent,
@@ -470,13 +476,17 @@ _bt_spools_heapscan(Relation heap, Relation index, BTBuildState *buildstate,
 
 	/* Fill spool using either serial or parallel heap scan */
 	if (!buildstate->btleader)
-		reltuples = IndexBuildHeapScan(heap, index, indexInfo, true,
+		reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, true,
 									   _bt_build_callback, (void *) buildstate,
 									   NULL);
 	else
 		reltuples = _bt_parallel_heapscan(buildstate,
 										  &indexInfo->ii_BrokenHotChain);
 
+	/* Set the progress target for the next phase */
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_TOTAL,
+								 buildstate->indtuples);
+
 	/* okay, all heap tuples are spooled */
 	if (buildstate->spool2 && !buildstate->havedead)
 	{
@@ -525,9 +535,15 @@ _bt_leafbuild(BTSpool *btspool, BTSpool *btspool2)
 	}
 #endif							/* BTREE_BUILD_STATS */
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+								 PROGRESS_BTREE_PHASE_PERFORMSORT_1);
 	tuplesort_performsort(btspool->sortstate);
 	if (btspool2)
+	{
+		pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+									 PROGRESS_BTREE_PHASE_PERFORMSORT_2);
 		tuplesort_performsort(btspool2->sortstate);
+	}
 
 	wstate.heap = btspool->heap;
 	wstate.index = btspool->index;
@@ -543,6 +559,8 @@ _bt_leafbuild(BTSpool *btspool, BTSpool *btspool2)
 	wstate.btws_pages_written = 0;
 	wstate.btws_zeropage = NULL;	/* until needed */
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+								 PROGRESS_BTREE_PHASE_LEAF_LOAD);
 	_bt_load(&wstate, btspool, btspool2);
 }
 
@@ -1078,6 +1096,7 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
 				keysz = IndexRelationGetNumberOfKeyAttributes(wstate->index);
 	ScanKey		indexScanKey = NULL;
 	SortSupport sortKeys;
+	long		tuples_done = 0L;
 
 	if (merge)
 	{
@@ -1170,6 +1189,10 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
 				_bt_buildadd(wstate, state, itup2);
 				itup2 = tuplesort_getindextuple(btspool2->sortstate, true);
 			}
+
+			/* Report progress */
+			pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE,
+										 ++tuples_done);
 		}
 		pfree(sortKeys);
 	}
@@ -1184,6 +1207,10 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
 				state = _bt_pagestate(wstate, 0);
 
 			_bt_buildadd(wstate, state, itup);
+
+			/* Report progress */
+			pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE,
+										 ++tuples_done);
 		}
 	}
 
@@ -1318,6 +1345,10 @@ _bt_begin_parallel(BTBuildState *buildstate, bool isconcurrent, int request)
 	btshared->brokenhotchain = false;
 	heap_parallelscan_initialize(&btshared->heapdesc, btspool->heap, snapshot);
 
+	/* Report total number of blocks to scan */
+	pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL,
+								 btshared->heapdesc.phs_nblocks);
+
 	/*
 	 * Store shared tuplesort-private state, for which we reserved space.
 	 * Then, initialize opaque state using tuplesort routine.
@@ -1493,7 +1524,7 @@ _bt_leader_participate_as_worker(BTBuildState *buildstate)
 	/* Perform work common to all participants */
 	_bt_parallel_scan_and_sort(leaderworker, leaderworker2, btleader->btshared,
 							   btleader->sharedsort, btleader->sharedsort2,
-							   sortmem);
+							   sortmem, true);
 
 #ifdef BTREE_BUILD_STATS
 	if (log_btree_build_stats)
@@ -1584,7 +1615,7 @@ _bt_parallel_build_main(dsm_segment *seg, shm_toc *toc)
 	/* Perform sorting of spool, and possibly a spool2 */
 	sortmem = maintenance_work_mem / btshared->scantuplesortstates;
 	_bt_parallel_scan_and_sort(btspool, btspool2, btshared, sharedsort,
-							   sharedsort2, sortmem);
+							   sharedsort2, sortmem, false);
 
 #ifdef BTREE_BUILD_STATS
 	if (log_btree_build_stats)
@@ -1613,7 +1644,7 @@ _bt_parallel_build_main(dsm_segment *seg, shm_toc *toc)
 static void
 _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2,
 						   BTShared *btshared, Sharedsort *sharedsort,
-						   Sharedsort *sharedsort2, int sortmem)
+						   Sharedsort *sharedsort2, int sortmem, bool progress)
 {
 	SortCoordinate coordinate;
 	BTBuildState buildstate;
@@ -1672,7 +1703,7 @@ _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2,
 	indexInfo->ii_Concurrent = btshared->isconcurrent;
 	scan = heap_beginscan_parallel(btspool->heap, &btshared->heapdesc);
 	reltuples = IndexBuildHeapScan(btspool->heap, btspool->index, indexInfo,
-								   true, _bt_build_callback,
+								   true, progress, _bt_build_callback,
 								   (void *) &buildstate, scan);
 
 	/*
diff --git a/src/backend/access/nbtree/nbtutils.c b/src/backend/access/nbtree/nbtutils.c
index 2c05fb5e451..295c22b2e5b 100644
--- a/src/backend/access/nbtree/nbtutils.c
+++ b/src/backend/access/nbtree/nbtutils.c
@@ -20,6 +20,7 @@
 #include "access/nbtree.h"
 #include "access/reloptions.h"
 #include "access/relscan.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
 #include "utils/array.h"
 #include "utils/lsyscache.h"
@@ -2082,6 +2083,29 @@ btproperty(Oid index_oid, int attno,
 	}
 }
 
+/*
+ *	btbuildphasename() -- Return name of index build phase.
+ */
+char *
+btbuildphasename(int64 phasenum)
+{
+	switch (phasenum)
+	{
+		case PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE:
+			return "initializing (1 of 5)";
+		case PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN:
+			return "table scan (2 of 5)";
+		case PROGRESS_BTREE_PHASE_PERFORMSORT_1:
+			return "sorting tuples, spool 1 (3 of 5)";
+		case PROGRESS_BTREE_PHASE_PERFORMSORT_2:
+			return "sorting tuples, spool 2 (4 of 5)";
+		case PROGRESS_BTREE_PHASE_LEAF_LOAD:
+			return "btree tuple loading (5 of 5)";
+		default:
+			return NULL;
+	}
+}
+
 /*
  *	_bt_nonkey_truncate() -- create tuple without non-key suffix attributes.
  *
diff --git a/src/backend/access/spgist/spginsert.c b/src/backend/access/spgist/spginsert.c
index f428a151385..1bc671c7238 100644
--- a/src/backend/access/spgist/spginsert.c
+++ b/src/backend/access/spgist/spginsert.c
@@ -142,7 +142,7 @@ spgbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 											  "SP-GiST build temporary context",
 											  ALLOCSET_DEFAULT_SIZES);
 
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false,
 								   spgistBuildCallback, (void *) &buildstate,
 								   NULL);
 
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 8e63c1fad25..45472db147b 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -67,6 +67,7 @@ spghandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = spgcostestimate;
 	amroutine->amoptions = spgoptions;
 	amroutine->amproperty = spgproperty;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = spgvalidate;
 	amroutine->ambeginscan = spgbeginscan;
 	amroutine->amrescan = spgrescan;
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index d16c3d0ea50..4205491ec1f 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -49,8 +49,9 @@
 #include "catalog/pg_trigger.h"
 #include "catalog/pg_type.h"
 #include "catalog/storage.h"
-#include "commands/tablecmds.h"
 #include "commands/event_trigger.h"
+#include "commands/progress.h"
+#include "commands/tablecmds.h"
 #include "commands/trigger.h"
 #include "executor/executor.h"
 #include "miscadmin.h"
@@ -58,6 +59,7 @@
 #include "nodes/nodeFuncs.h"
 #include "optimizer/optimizer.h"
 #include "parser/parser.h"
+#include "pgstat.h"
 #include "rewrite/rewriteManip.h"
 #include "storage/bufmgr.h"
 #include "storage/lmgr.h"
@@ -1597,7 +1599,7 @@ index_drop(Oid indexId, bool concurrent)
 		 * to acquire an exclusive lock on our table.  The lock code will
 		 * detect deadlock and error out properly.
 		 */
-		WaitForLockers(heaplocktag, AccessExclusiveLock);
+		WaitForLockers(heaplocktag, AccessExclusiveLock, true);
 
 		/*
 		 * No more predicate locks will be acquired on this index, and we're
@@ -1641,7 +1643,7 @@ index_drop(Oid indexId, bool concurrent)
 		 * Wait till every transaction that saw the old index state has
 		 * finished.
 		 */
-		WaitForLockers(heaplocktag, AccessExclusiveLock);
+		WaitForLockers(heaplocktag, AccessExclusiveLock, true);
 
 		/*
 		 * Re-open relations to allow us to complete our actions.
@@ -2291,6 +2293,25 @@ index_build(Relation heapRelation,
 						   save_sec_context | SECURITY_RESTRICTED_OPERATION);
 	save_nestlevel = NewGUCNestLevel();
 
+	/* Set up initial progress report status */
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_PHASE,
+			PROGRESS_CREATEIDX_SUBPHASE,
+			PROGRESS_CREATEIDX_TUPLES_DONE,
+			PROGRESS_CREATEIDX_TUPLES_TOTAL,
+			PROGRESS_SCAN_BLOCKS_DONE,
+			PROGRESS_SCAN_BLOCKS_TOTAL
+		};
+		const int64	val[] = {
+			PROGRESS_CREATEIDX_PHASE_BUILD,
+			PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE,
+			0, 0, 0, 0
+		};
+
+		pgstat_progress_update_multi_param(6, index, val);
+	}
+
 	/*
 	 * Call the access method's build procedure
 	 */
@@ -2428,13 +2449,14 @@ IndexBuildHeapScan(Relation heapRelation,
 				   Relation indexRelation,
 				   IndexInfo *indexInfo,
 				   bool allow_sync,
+				   bool progress,
 				   IndexBuildCallback callback,
 				   void *callback_state,
 				   HeapScanDesc scan)
 {
 	return IndexBuildHeapRangeScan(heapRelation, indexRelation,
 								   indexInfo, allow_sync,
-								   false,
+								   false, progress,
 								   0, InvalidBlockNumber,
 								   callback, callback_state, scan);
 }
@@ -2455,6 +2477,7 @@ IndexBuildHeapRangeScan(Relation heapRelation,
 						IndexInfo *indexInfo,
 						bool allow_sync,
 						bool anyvisible,
+						bool progress,
 						BlockNumber start_blockno,
 						BlockNumber numblocks,
 						IndexBuildCallback callback,
@@ -2476,6 +2499,8 @@ IndexBuildHeapRangeScan(Relation heapRelation,
 	TransactionId OldestXmin;
 	BlockNumber root_blkno = InvalidBlockNumber;
 	OffsetNumber root_offsets[MaxHeapTuplesPerPage];
+	BlockNumber blocks_done = 0;
+	BlockNumber	previous_blkno = InvalidBlockNumber;
 
 	/*
 	 * sanity checks
@@ -2545,6 +2570,13 @@ IndexBuildHeapRangeScan(Relation heapRelation,
 									NULL,	/* scan key */
 									true,	/* buffer access strategy OK */
 									allow_sync);	/* syncscan OK? */
+
+		if (progress)
+		{
+			Assert(scan->rs_numblocks == InvalidBlockNumber);
+			pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL,
+										 scan->rs_nblocks);
+		}
 	}
 	else
 	{
@@ -2592,6 +2624,46 @@ IndexBuildHeapRangeScan(Relation heapRelation,
 
 		CHECK_FOR_INTERRUPTS();
 
+		/* Report scan progress, if asked to. */
+		if (progress &&
+			((previous_blkno == InvalidBlockNumber) ||
+			(scan->rs_cblock != previous_blkno)))
+		{
+			/* we only do progress for full table scans */
+			Assert(numblocks == InvalidBlockNumber);
+
+			/*
+			 * Report the number of blocks we've moved forward.
+			 *
+			 * Parallel workers cause the leader process to skip some blocks,
+			 * so we subtract the current block number to the previous one to
+			 * determine how many have been read in total; but be careful when
+			 * we wrap around the last block in the table.
+			 */
+			if (scan->rs_cblock > previous_blkno)
+				blocks_done += scan->rs_cblock - previous_blkno;
+			else if (previous_blkno == InvalidBlockNumber)
+			{
+				/*
+				 * How many blocks have been read since the scan started.
+				 * Should normally be zero.
+				 */
+				blocks_done += scan->rs_cblock -
+					(scan->rs_parallel ? scan->rs_parallel->phs_startblock :
+					 scan->rs_startblock);
+				Assert(blocks_done == 0);
+			}
+			else
+			{
+				/* wrapped around */
+				blocks_done += scan->rs_nblocks - previous_blkno + scan->rs_cblock;
+			}
+
+			pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
+										 blocks_done);
+			previous_blkno = scan->rs_cblock;
+		}
+
 		/*
 		 * When dealing with a HOT-chain of updated tuples, we want to index
 		 * the values of the live tuple (if any), but index it under the TID
@@ -3133,6 +3205,21 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot)
 	int			save_sec_context;
 	int			save_nestlevel;
 
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_PHASE,
+			PROGRESS_CREATEIDX_TUPLES_DONE,
+			PROGRESS_CREATEIDX_TUPLES_TOTAL,
+			PROGRESS_SCAN_BLOCKS_DONE,
+			PROGRESS_SCAN_BLOCKS_TOTAL
+		};
+		const int64	val[] = {
+			PROGRESS_CREATEIDX_PHASE_VALIDATE_IDXSCAN,
+			0, 0, 0, 0
+		};
+		pgstat_progress_update_multi_param(5, index, val);
+	}
+
 	/* Open and lock the parent heap relation */
 	heapRelation = table_open(heapId, ShareUpdateExclusiveLock);
 	/* And the target index relation */
@@ -3163,6 +3250,7 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot)
 	 */
 	ivinfo.index = indexRelation;
 	ivinfo.analyze_only = false;
+	ivinfo.report_progress = true;	/* XXX only for btree? */
 	ivinfo.estimated_count = true;
 	ivinfo.message_level = DEBUG2;
 	ivinfo.num_heap_tuples = heapRelation->rd_rel->reltuples;
@@ -3180,15 +3268,39 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot)
 											NULL, false);
 	state.htups = state.itups = state.tups_inserted = 0;
 
+	/* ambulkdelete updates progress metrics */
 	(void) index_bulk_delete(&ivinfo, NULL,
 							 validate_index_callback, (void *) &state);
 
 	/* Execute the sort */
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_PHASE,
+			PROGRESS_SCAN_BLOCKS_DONE,
+			PROGRESS_SCAN_BLOCKS_TOTAL
+		};
+		const int64	val[] = {
+			PROGRESS_CREATEIDX_PHASE_SORT_IDXSCAN,
+			0, 0
+		};
+
+		pgstat_progress_update_multi_param(3, index, val);
+	}
 	tuplesort_performsort(state.tuplesort);
 
 	/*
-	 * Now scan the heap and "merge" it with the index
+	 * Now scan the heap and "merge" it with the index.
 	 */
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_PHASE,
+		};
+		const int64	val[] = {
+			PROGRESS_CREATEIDX_PHASE_HEAPSCAN_VALIDATE
+		};
+
+		pgstat_progress_update_multi_param(1, index, val);
+	}
 	validate_index_heapscan(heapRelation,
 							indexRelation,
 							indexInfo,
@@ -3291,6 +3403,7 @@ validate_index_heapscan(Relation heapRelation,
 	BlockNumber root_blkno = InvalidBlockNumber;
 	OffsetNumber root_offsets[MaxHeapTuplesPerPage];
 	bool		in_index[MaxHeapTuplesPerPage];
+	BlockNumber	previous_blkno = InvalidBlockNumber;
 
 	/* state variables for the merge */
 	ItemPointer indexcursor = NULL;
@@ -3330,6 +3443,9 @@ validate_index_heapscan(Relation heapRelation,
 								true,	/* buffer access strategy OK */
 								false); /* syncscan not OK */
 
+	pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL,
+								 scan->rs_nblocks);
+
 	/*
 	 * Scan all tuples matching the snapshot.
 	 */
@@ -3343,6 +3459,14 @@ validate_index_heapscan(Relation heapRelation,
 
 		state->htups += 1;
 
+		if ((previous_blkno == InvalidBlockNumber) ||
+			(scan->rs_cblock != previous_blkno))
+		{
+			pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
+										 scan->rs_cblock);
+			previous_blkno = scan->rs_cblock;
+		}
+
 		/*
 		 * As commented in IndexBuildHeapScan, we should index heap-only
 		 * tuples under the TIDs of their root tuples; so when we advance onto
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 3e229c693c4..3cbe6f8af2a 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -906,6 +906,33 @@ CREATE VIEW pg_stat_progress_vacuum AS
     FROM pg_stat_get_progress_info('VACUUM') AS S
 		LEFT JOIN pg_database D ON S.datid = D.oid;
 
+CREATE VIEW pg_stat_progress_create_index AS
+	SELECT
+		S.pid AS pid, S.datid AS datid, D.datname AS datname,
+		S.relid AS relid,
+		CASE S.param2 WHEN 0 THEN 'initializing (phase 1 of 8)'
+					  WHEN 1 THEN 'waiting for old snapshots (phase 2 of 8)'
+					  WHEN 2 THEN 'building index (3 of 8)' ||
+						COALESCE((': ' || pg_indexam_progress_phasename(S.param1::oid, S.param3)),
+							'')
+					  WHEN 3 THEN 'waiting for writer snapshots (phase 4 of 8)'
+					  WHEN 4 THEN 'index validation: scan index (phase 5 of 8)'
+					  WHEN 5 THEN 'index validation: sort index scan results (phase 6 of 8)'
+					  WHEN 6 THEN 'index validation: scan heap (phase 7 of 8)'
+					  WHEN 7 THEN 'waiting for reader snapshots (phase 8 of 8)'
+					  END as phase,
+		S.param4 AS lockers_total,
+		S.param5 AS lockers_done,
+		S.param6 AS current_locker_pid,
+		S.param7 AS blocks_total,
+		S.param8 AS blocks_done,
+		S.param9 AS tuples_total,
+		S.param10 AS tuples_done,
+		S.param11 AS partitions_total,
+		S.param12 AS partitions_done
+	FROM pg_stat_get_progress_info('CREATE INDEX') AS S
+		LEFT JOIN pg_database D ON S.datid = D.oid;
+
 CREATE VIEW pg_user_mappings AS
     SELECT
         U.oid       AS umid,
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 5dcedc337aa..8f8587ac1e3 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -35,6 +35,7 @@
 #include "commands/dbcommands.h"
 #include "commands/defrem.h"
 #include "commands/event_trigger.h"
+#include "commands/progress.h"
 #include "commands/tablecmds.h"
 #include "commands/tablespace.h"
 #include "mb/pg_wchar.h"
@@ -46,10 +47,12 @@
 #include "parser/parse_func.h"
 #include "parser/parse_oper.h"
 #include "partitioning/partdesc.h"
+#include "pgstat.h"
 #include "rewrite/rewriteManip.h"
 #include "storage/lmgr.h"
 #include "storage/proc.h"
 #include "storage/procarray.h"
+#include "storage/sinvaladt.h"
 #include "utils/acl.h"
 #include "utils/builtins.h"
 #include "utils/fmgroids.h"
@@ -368,6 +371,15 @@ DefineIndex(Oid relationId,
 	Snapshot	snapshot;
 	int			i;
 
+
+	/*
+	 * Start progress report.  If we're building a partition, this was already
+	 * done.
+	 */
+	if (!OidIsValid(parentIndexId))
+		pgstat_progress_start_command(PROGRESS_COMMAND_CREATE_INDEX,
+									  relationId);
+
 	/*
 	 * count key attributes in index
 	 */
@@ -584,6 +596,9 @@ DefineIndex(Oid relationId,
 	accessMethodId = accessMethodForm->oid;
 	amRoutine = GetIndexAmRoutine(accessMethodForm->amhandler);
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_ACCESS_METHOD_OID,
+								 accessMethodId);
+
 	if (stmt->unique && !amRoutine->amcanunique)
 		ereport(ERROR,
 				(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -864,6 +879,11 @@ DefineIndex(Oid relationId,
 	if (!OidIsValid(indexRelationId))
 	{
 		table_close(rel, NoLock);
+
+		/* If this is the top-level index, we're done */
+		if (!OidIsValid(parentIndexId))
+			pgstat_progress_end_command();
+
 		return address;
 	}
 
@@ -889,6 +909,9 @@ DefineIndex(Oid relationId,
 			TupleDesc	parentDesc;
 			Oid		   *opfamOids;
 
+			pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL,
+										 nparts);
+
 			memcpy(part_oids, partdesc->oids, sizeof(Oid) * nparts);
 
 			parentDesc = CreateTupleDescCopy(RelationGetDescr(rel));
@@ -1039,6 +1062,8 @@ DefineIndex(Oid relationId,
 								skip_build, quiet);
 				}
 
+				pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_DONE,
+											 i + 1);
 				pfree(attmap);
 			}
 
@@ -1073,6 +1098,8 @@ DefineIndex(Oid relationId,
 		 * Indexes on partitioned tables are not themselves built, so we're
 		 * done here.
 		 */
+		if (!OidIsValid(parentIndexId))
+			pgstat_progress_end_command();
 		return address;
 	}
 
@@ -1080,6 +1107,11 @@ DefineIndex(Oid relationId,
 	{
 		/* Close the heap and we're done, in the non-concurrent case */
 		table_close(rel, NoLock);
+
+		/* If this is the top-level index, we're done. */
+		if (!OidIsValid(parentIndexId))
+			pgstat_progress_end_command();
+
 		return address;
 	}
 
@@ -1131,7 +1163,9 @@ DefineIndex(Oid relationId,
 	 * exclusive lock on our table.  The lock code will detect deadlock and
 	 * error out properly.
 	 */
-	WaitForLockers(heaplocktag, ShareLock);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_1);
+	WaitForLockers(heaplocktag, ShareLock, true);
 
 	/*
 	 * At this moment we are sure that there are no transactions with the
@@ -1195,7 +1229,9 @@ DefineIndex(Oid relationId,
 	 * We once again wait until no transaction can have the table open with
 	 * the index marked as read-only for updates.
 	 */
-	WaitForLockers(heaplocktag, ShareLock);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_2);
+	WaitForLockers(heaplocktag, ShareLock, true);
 
 	/*
 	 * Now take the "reference snapshot" that will be used by validate_index()
@@ -1281,6 +1317,9 @@ DefineIndex(Oid relationId,
 	old_snapshots = GetCurrentVirtualXIDs(limitXmin, true, false,
 										  PROC_IS_AUTOVACUUM | PROC_IN_VACUUM,
 										  &n_old_snapshots);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_3);
+	pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, n_old_snapshots);
 
 	for (i = 0; i < n_old_snapshots; i++)
 	{
@@ -1316,7 +1355,14 @@ DefineIndex(Oid relationId,
 		}
 
 		if (VirtualTransactionIdIsValid(old_snapshots[i]))
+		{
+			PGPROC *holder = BackendIdGetProc(old_snapshots[i].backendId);
+			pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID,
+										 holder->pid);
 			VirtualXactLock(old_snapshots[i], true);
+		}
+
+		pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, i + 1);
 	}
 
 	/*
@@ -1339,6 +1385,8 @@ DefineIndex(Oid relationId,
 	 */
 	UnlockRelationIdForSession(&heaprelid, ShareUpdateExclusiveLock);
 
+	pgstat_progress_end_command();
+
 	return address;
 }
 
diff --git a/src/backend/storage/ipc/standby.c b/src/backend/storage/ipc/standby.c
index 4d10e57a803..a0a2b964703 100644
--- a/src/backend/storage/ipc/standby.c
+++ b/src/backend/storage/ipc/standby.c
@@ -401,7 +401,7 @@ ResolveRecoveryConflictWithLock(LOCKTAG locktag)
 		 */
 		VirtualTransactionId *backends;
 
-		backends = GetLockConflicts(&locktag, AccessExclusiveLock);
+		backends = GetLockConflicts(&locktag, AccessExclusiveLock, NULL);
 		ResolveRecoveryConflictWithVirtualXIDs(backends,
 											   PROCSIG_RECOVERY_CONFLICT_LOCK);
 	}
diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c
index e688ba81170..0b04b093782 100644
--- a/src/backend/storage/lmgr/lmgr.c
+++ b/src/backend/storage/lmgr/lmgr.c
@@ -19,9 +19,12 @@
 #include "access/transam.h"
 #include "access/xact.h"
 #include "catalog/catalog.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
+#include "pgstat.h"
 #include "storage/lmgr.h"
 #include "storage/procarray.h"
+#include "storage/sinvaladt.h"
 #include "utils/inval.h"
 
 
@@ -857,10 +860,12 @@ XactLockTableWaitErrorCb(void *arg)
  * after we obtained our initial list of lockers, we will not wait for them.
  */
 void
-WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
+WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
 {
 	List	   *holders = NIL;
 	ListCell   *lc;
+	int			total = 0;
+	int			done = 0;
 
 	/* Done if no locks to wait for */
 	if (list_length(locktags) == 0)
@@ -870,10 +875,17 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
 	foreach(lc, locktags)
 	{
 		LOCKTAG    *locktag = lfirst(lc);
+		int			count;
 
-		holders = lappend(holders, GetLockConflicts(locktag, lockmode));
+		holders = lappend(holders,
+						  GetLockConflicts(locktag, lockmode,
+										   progress ? &count : NULL));
+		total += count;
 	}
 
+	if (progress)
+		pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, total);
+
 	/*
 	 * Note: GetLockConflicts() never reports our own xid, hence we need not
 	 * check for that.  Also, prepared xacts are not reported, which is fine
@@ -887,10 +899,36 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
 
 		while (VirtualTransactionIdIsValid(*lockholders))
 		{
+			/*
+			 * If requested, publish who we're going to wait for.  This is not
+			 * 100% accurate if they're already gone, but we don't care.
+			 */
+			if (progress)
+			{
+				PGPROC *holder = BackendIdGetProc(lockholders->backendId);
+
+				pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID,
+											 holder->pid);
+			}
 			VirtualXactLock(*lockholders, true);
 			lockholders++;
+
+			if (progress)
+				pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, ++done);
 		}
 	}
+	if (progress)
+	{
+		const int	index[] = {
+			PROGRESS_WAITFOR_TOTAL,
+			PROGRESS_WAITFOR_DONE,
+			PROGRESS_WAITFOR_CURRENT_PID
+		};
+		const int64	values[] = {
+			0, 0, 0
+		};
+		pgstat_progress_update_multi_param(3, index, values);
+	}
 
 	list_free_deep(holders);
 }
@@ -901,12 +939,12 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
  * Same as WaitForLockersMultiple, for a single lock tag.
  */
 void
-WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode)
+WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress)
 {
 	List	   *l;
 
 	l = list_make1(&heaplocktag);
-	WaitForLockersMultiple(l, lockmode);
+	WaitForLockersMultiple(l, lockmode, progress);
 	list_free(l);
 }
 
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 3bb5ce350aa..58ba90d0646 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2807,6 +2807,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock)
  *		xacts merely awaiting such a lock are NOT reported.
  *
  * The result array is palloc'd and is terminated with an invalid VXID.
+ * *countp, if not null, is updated to the number of items set.
  *
  * Of course, the result could be out of date by the time it's returned,
  * so use of this function has to be thought about carefully.
@@ -2817,7 +2818,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock)
  * uses of the result.
  */
 VirtualTransactionId *
-GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode)
+GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
 {
 	static VirtualTransactionId *vxids;
 	LOCKMETHODID lockmethodid = locktag->locktag_lockmethodid;
@@ -2964,6 +2965,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode)
 		LWLockRelease(partitionLock);
 		vxids[count].backendId = InvalidBackendId;
 		vxids[count].localTransactionId = InvalidLocalTransactionId;
+		if (countp)
+			*countp = count;
 		return vxids;
 	}
 
@@ -3019,6 +3022,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode)
 
 	vxids[count].backendId = InvalidBackendId;
 	vxids[count].localTransactionId = InvalidLocalTransactionId;
+	if (countp)
+		*countp = count;
 	return vxids;
 }
 
diff --git a/src/backend/utils/adt/amutils.c b/src/backend/utils/adt/amutils.c
index 060ffe501ec..e81d6cc0562 100644
--- a/src/backend/utils/adt/amutils.c
+++ b/src/backend/utils/adt/amutils.c
@@ -445,3 +445,26 @@ pg_index_column_has_property(PG_FUNCTION_ARGS)
 
 	return indexam_property(fcinfo, propname, InvalidOid, relid, attno);
 }
+
+/*
+ * Return the name of the given phase, as used for progress reporting by the
+ * given AM.
+ */
+Datum
+pg_indexam_progress_phasename(PG_FUNCTION_ARGS)
+{
+	Oid			amoid = PG_GETARG_OID(0);
+	int32		phasenum = PG_GETARG_INT32(1);
+	IndexAmRoutine *routine;
+	char	   *name;
+
+	routine = GetIndexAmRoutineByAmId(amoid, true);
+	if (routine == NULL || !routine->ambuildphasename)
+		PG_RETURN_NULL();
+
+	name = routine->ambuildphasename(phasenum);
+	if (!name)
+		PG_RETURN_NULL();
+
+	PG_RETURN_TEXT_P(CStringGetTextDatum(name));
+}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 69f72657792..05f9da6a4ad 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -468,6 +468,8 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
 	/* Translate command name into command type code. */
 	if (pg_strcasecmp(cmd, "VACUUM") == 0)
 		cmdtype = PROGRESS_COMMAND_VACUUM;
+	else if (pg_strcasecmp(cmd, "CREATE INDEX") == 0)
+		cmdtype = PROGRESS_COMMAND_CREATE_INDEX;
 	else
 		ereport(ERROR,
 				(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 653ddc976ba..09a7404267c 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -108,6 +108,9 @@ typedef bool (*amproperty_function) (Oid index_oid, int attno,
 									 IndexAMProperty prop, const char *propname,
 									 bool *res, bool *isnull);
 
+/* name of phase as used in progress reporting */
+typedef char *(*ambuildphasename_function) (int64 phasenum);
+
 /* validate definition of an opclass for this AM */
 typedef bool (*amvalidate_function) (Oid opclassoid);
 
@@ -213,6 +216,7 @@ typedef struct IndexAmRoutine
 	amcostestimate_function amcostestimate;
 	amoptions_function amoptions;
 	amproperty_function amproperty; /* can be NULL */
+	ambuildphasename_function ambuildphasename;	/* can be NULL */
 	amvalidate_function amvalidate;
 	ambeginscan_function ambeginscan;
 	amrescan_function amrescan;
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index c4aba39496f..f77d9eea8e8 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -45,6 +45,7 @@ typedef struct IndexVacuumInfo
 {
 	Relation	index;			/* the index being vacuumed */
 	bool		analyze_only;	/* ANALYZE (without any actual vacuum) */
+	bool		report_progress;	/* emit progress.h status reports */
 	bool		estimated_count;	/* num_heap_tuples is an estimate */
 	int			message_level;	/* ereport level for progress messages */
 	double		num_heap_tuples;	/* tuples remaining in heap */
diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h
index 4fb92d60a12..18a6838ce56 100644
--- a/src/include/access/nbtree.h
+++ b/src/include/access/nbtree.h
@@ -488,6 +488,16 @@ typedef BTScanOpaqueData *BTScanOpaque;
 #define SK_BT_DESC			(INDOPTION_DESC << SK_BT_INDOPTION_SHIFT)
 #define SK_BT_NULLS_FIRST	(INDOPTION_NULLS_FIRST << SK_BT_INDOPTION_SHIFT)
 
+/*
+ * Constant definition for progress reporting.  Phase numbers must match
+ * btbuildphasename.
+ */
+/* PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE is 1 (see progress.h) */
+#define PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN		2
+#define PROGRESS_BTREE_PHASE_PERFORMSORT_1				3
+#define PROGRESS_BTREE_PHASE_PERFORMSORT_2				4
+#define PROGRESS_BTREE_PHASE_LEAF_LOAD					5
+
 /*
  * external entry points for btree, in nbtree.c
  */
@@ -600,6 +610,7 @@ extern bytea *btoptions(Datum reloptions, bool validate);
 extern bool btproperty(Oid index_oid, int attno,
 		   IndexAMProperty prop, const char *propname,
 		   bool *res, bool *isnull);
+extern char *btbuildphasename(int64 phasenum);
 extern IndexTuple _bt_nonkey_truncate(Relation rel, IndexTuple itup);
 extern bool _bt_check_natts(Relation rel, Page page, OffsetNumber offnum);
 
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index 330c481a8b7..fb713d50f96 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -114,6 +114,7 @@ extern double IndexBuildHeapScan(Relation heapRelation,
 				   Relation indexRelation,
 				   IndexInfo *indexInfo,
 				   bool allow_sync,
+				   bool progress,
 				   IndexBuildCallback callback,
 				   void *callback_state,
 				   struct HeapScanDescData *scan);
@@ -122,6 +123,7 @@ extern double IndexBuildHeapRangeScan(Relation heapRelation,
 						IndexInfo *indexInfo,
 						bool allow_sync,
 						bool anyvisible,
+						bool progress,
 						BlockNumber start_blockno,
 						BlockNumber end_blockno,
 						IndexBuildCallback callback,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index a4e173b4846..32e31c2d3f1 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -917,6 +917,10 @@
   proname => 'pg_index_column_has_property', provolatile => 's',
   prorettype => 'bool', proargtypes => 'regclass int4 text',
   prosrc => 'pg_index_column_has_property' },
+{ oid => '676', descr => 'return name of given index build phase',
+  proname => 'pg_indexam_progress_phasename', provolatile => 'i',
+  prorettype => 'text', proargtypes => 'oid int8',
+  prosrc => 'pg_indexam_progress_phasename' },
 
 { oid => '339',
   proname => 'poly_same', prorettype => 'bool',
@@ -5079,9 +5083,9 @@
   proname => 'pg_stat_get_progress_info', prorows => '100', proretset => 't',
   provolatile => 's', proparallel => 'r', prorettype => 'record',
   proargtypes => 'text',
-  proallargtypes => '{text,int4,oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}',
-  proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o}',
-  proargnames => '{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10}',
+  proallargtypes => '{text,int4,oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}',
+  proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+  proargnames => '{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10,param11,param12}',
   prosrc => 'pg_stat_get_progress_info' },
 { oid => '3099',
   descr => 'statistics: information about currently active replication',
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index 9858b36a383..56bd6125594 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -34,4 +34,40 @@
 #define PROGRESS_VACUUM_PHASE_TRUNCATE			5
 #define PROGRESS_VACUUM_PHASE_FINAL_CLEANUP		6
 
+
+/* Progress parameters for CREATE INDEX */
+#define PROGRESS_CREATEIDX_ACCESS_METHOD_OID	0
+#define PROGRESS_CREATEIDX_PHASE				1	/* AM-agnostic phase # */
+#define PROGRESS_CREATEIDX_SUBPHASE				2	/* phase # filled by AM */
+/* 3, 4 and 5 reserved for "waitfor" metrics */
+/* 6 and 7 reserved for "block number" metrics */
+#define PROGRESS_CREATEIDX_TUPLES_TOTAL			8
+#define PROGRESS_CREATEIDX_TUPLES_DONE			9
+#define PROGRESS_CREATEIDX_PARTITIONS_TOTAL		10
+#define PROGRESS_CREATEIDX_PARTITIONS_DONE		11
+
+/* Phases of CREATE INDEX */
+#define PROGRESS_CREATEIDX_PHASE_WAIT_1			1
+#define PROGRESS_CREATEIDX_PHASE_BUILD			2
+#define PROGRESS_CREATEIDX_PHASE_WAIT_2			3
+#define PROGRESS_CREATEIDX_PHASE_VALIDATE_IDXSCAN		4
+#define PROGRESS_CREATEIDX_PHASE_SORT_IDXSCAN		5
+#define PROGRESS_CREATEIDX_PHASE_HEAPSCAN_VALIDATE		6
+#define PROGRESS_CREATEIDX_PHASE_WAIT_3			7
+
+/*
+ * Subphases of CREATE INDEX, for index_build.
+ */
+#define PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE	1
+/* Additional phases are defined by each AM */
+
+/* Lock holder wait counts */
+#define PROGRESS_WAITFOR_TOTAL					3
+#define PROGRESS_WAITFOR_DONE					4
+#define PROGRESS_WAITFOR_CURRENT_PID			5
+
+/* Block numbers in a generic relation scan */
+#define PROGRESS_SCAN_BLOCKS_TOTAL				6
+#define PROGRESS_SCAN_BLOCKS_DONE				7
+
 #endif
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 88a75fb798e..f931ead1c27 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -934,10 +934,11 @@ typedef enum
 typedef enum ProgressCommandType
 {
 	PROGRESS_COMMAND_INVALID,
-	PROGRESS_COMMAND_VACUUM
+	PROGRESS_COMMAND_VACUUM,
+	PROGRESS_COMMAND_CREATE_INDEX
 } ProgressCommandType;
 
-#define PGSTAT_NUM_PROGRESS_PARAM	10
+#define PGSTAT_NUM_PROGRESS_PARAM	12
 
 /* ----------
  * Shared-memory data structures
diff --git a/src/include/storage/lmgr.h b/src/include/storage/lmgr.h
index 3d705faba5c..4f2872de35f 100644
--- a/src/include/storage/lmgr.h
+++ b/src/include/storage/lmgr.h
@@ -78,8 +78,8 @@ extern void XactLockTableWait(TransactionId xid, Relation rel,
 extern bool ConditionalXactLockTableWait(TransactionId xid);
 
 /* Lock VXIDs, specified by conflicting locktags */
-extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode);
-extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode);
+extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress);
+extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress);
 
 /* Lock an XID for tuple insertion (used to wait for an insertion to finish) */
 extern uint32 SpeculativeInsertionLockAcquire(TransactionId xid);
diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h
index 16b927cb801..e117b391774 100644
--- a/src/include/storage/lock.h
+++ b/src/include/storage/lock.h
@@ -544,7 +544,7 @@ extern bool LockHeldByMe(const LOCKTAG *locktag, LOCKMODE lockmode);
 extern bool LockHasWaiters(const LOCKTAG *locktag,
 			   LOCKMODE lockmode, bool sessionLock);
 extern VirtualTransactionId *GetLockConflicts(const LOCKTAG *locktag,
-				 LOCKMODE lockmode);
+				 LOCKMODE lockmode, int *countp);
 extern void AtPrepare_Locks(void);
 extern void PostPrepare_Locks(TransactionId xid);
 extern int LockCheckConflicts(LockMethod lockMethodTable,
-- 
2.17.1

#14Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Alvaro Herrera (#13)
1 attachment(s)
Re: monitoring CREATE INDEX [CONCURRENTLY]

Hmm, looks like a very bare-bones support for hash indexes does not
require a lot of code, and gives a clear picture (you can sit all night
watching the numbers go up, instead of biting your fingernails wondering
if it'll be completed by dawn.) This part isn't 100% done -- it we
would better to have ambuildphasename support.

(I'm a bit confused about phase 5 not reporting anything for hash
indexes in CIC, though. That's part is supposed to be AM agnostic.)

I think it was a mistake to define the progress constants in one header
file commands/progress.h and the associated functions in pgstat.h. I
think it would be better to move the function decls to
commands/progress.h.

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attachments:

progress-hash.patchtext/x-diff; charset=us-asciiDownload
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index fc7db5d6a13..cf7ec655044 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -22,9 +22,11 @@
 #include "access/hash_xlog.h"
 #include "access/relscan.h"
 #include "catalog/index.h"
+#include "commands/progress.h"
 #include "commands/vacuum.h"
 #include "miscadmin.h"
 #include "optimizer/plancat.h"
+#include "pgstat.h"
 #include "utils/builtins.h"
 #include "utils/index_selfuncs.h"
 #include "utils/rel.h"
@@ -160,8 +162,10 @@ hashbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	buildstate.heapRel = heap;
 
 	/* do the heap scan */
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, true,
 								   hashbuildCallback, (void *) &buildstate, NULL);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_TOTAL,
+								 buildstate.indtuples);
 
 	if (buildstate.spool)
 	{
diff --git a/src/backend/access/hash/hashsort.c b/src/backend/access/hash/hashsort.c
index 8c55436b193..00a57470a77 100644
--- a/src/backend/access/hash/hashsort.c
+++ b/src/backend/access/hash/hashsort.c
@@ -26,7 +26,9 @@
 #include "postgres.h"
 
 #include "access/hash.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
+#include "pgstat.h"
 #include "utils/tuplesort.h"
 
 
@@ -116,6 +118,7 @@ void
 _h_indexbuild(HSpool *hspool, Relation heapRel)
 {
 	IndexTuple	itup;
+	long		tups_done = 0;
 #ifdef USE_ASSERT_CHECKING
 	uint32		hashkey = 0;
 #endif
@@ -141,5 +144,8 @@ _h_indexbuild(HSpool *hspool, Relation heapRel)
 #endif
 
 		_hash_doinsert(hspool->index, itup, heapRel);
+
+		pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE,
+									 ++tups_done);
 	}
 }
#15Rahila Syed
rahila.syed@2ndquadrant.com
In reply to: Amit Langote (#11)
Re: monitoring CREATE INDEX [CONCURRENTLY]

Hi Alvaro,

On 2019-Feb-13, Amit Langote wrote:

Doesn't the name amphasename sound a bit too generic, given that it can
only describe the phases of ambuild? Maybe ambuildphase?

Hmm, yeah, maybe it does. I renamed it "ambuildphasename", since it's
not about reporting the phase itself -- it's about translating the phase
number to the string that's reported to the user.

The attached patch does it that way. Also, when an index build uses an
AM that doesn't support progress reporting, it no longer reports a NULL
phase name while building. I also changed it to report the progress of
phase 7 (heap scan validation) using block numbers rather than tuple
counts. I also tweaked the strings reported in the view. They're
clearer now IMO.

One slight annoyance is that when parallel workers are used, the last
block number reported in phase 3/subphase 2 (IndexBuildHeapScan stuff)
is not necessarily accurate, since the tail of the table could well be
scanned by a worker that's not the leader, and we only report in the
leader when it gets a new block.

When the AM does not support progress reporting, everything stays as
zeros during the index build phase. It's easy to notice how slow hash
indexes are to build compared to btrees this way! Maybe it'd be
better fallback on reporting block numbers in IndexBuildHeapScan when
this happens. Thoughts?

I added docs to the monitoring section -- that's the bulkiest part of
the patch.

1. Thank you for incorporating review comments.
Can you please rebase the latest
0001-Report-progress-of-CREATE-INDEX-operations.patch on master? Currently
it does not apply on 754b90f657bd54b482524b73726dae4a9165031c

15:56:44.694 | building index (3 of 8): initializing (1/5) |
442478 | 442399 | 0 | 0 | 0
| 0
15:56:44.705 | building index (3 of 8): sorting tuples, spool 1 (3/5) |
442478 | 442399 | 100000000 | 0 | 0
| 0
15:56:44.716 | building index (3 of 8): sorting tuples, spool 1 (3/5) |
442478 | 442399 | 100000000 | 0 | 0
| 0
15:56:44.727 | building index (3 of 8): final btree sort & load (5/5) |
442478 | 442399 | 100000000 | 79057 | 0
| 0

2. In the above report, even though we are reporting progress in terms of
tuples_done for final btree sort & load phase we have not cleared
the blocks_done entry from previous phases. I think this can be confusing
as the blocks_done does not correspond to the tuples_done in the current
phase.

--
Rahila Syed
Performance Engineer
2ndQuadrant
http://www.2ndQuadrant.com <http://www.2ndquadrant.com/&gt;
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#16Rahila
rahila.syed@2ndquadrant.com
In reply to: Alvaro Herrera (#13)
Re: monitoring CREATE INDEX [CONCURRENTLY]

Hi Alvaro,

Resending the email as earlier one didn't get sent on pgsql-hackers.

On 2/23/19 3:24 AM, Alvaro Herrera wrote:

On 2019-Feb-13, Amit Langote wrote:

Doesn't the name amphasename sound a bit too generic, given that it can
only describe the phases of ambuild? Maybe ambuildphase?

Hmm, yeah, maybe it does. I renamed it "ambuildphasename", since it's
not about reporting the phase itself -- it's about translating the phase
number to the string that's reported to the user.

The attached patch does it that way. Also, when an index build uses an
AM that doesn't support progress reporting, it no longer reports a NULL
phase name while building. I also changed it to report the progress of
phase 7 (heap scan validation) using block numbers rather than tuple
counts. I also tweaked the strings reported in the view. They're
clearer now IMO.

One slight annoyance is that when parallel workers are used, the last
block number reported in phase 3/subphase 2 (IndexBuildHeapScan stuff)
is not necessarily accurate, since the tail of the table could well be
scanned by a worker that's not the leader, and we only report in the
leader when it gets a new block.

When the AM does not support progress reporting, everything stays as
zeros during the index build phase. It's easy to notice how slow hash
indexes are to build compared to btrees this way! Maybe it'd be
better fallback on reporting block numbers in IndexBuildHeapScan when
this happens. Thoughts?

I added docs to the monitoring section -- that's the bulkiest part of
the patch.

1. Thank you for incorporating review comments.
Can you please rebase the latest 0001-Report-progress-of-
CREATE-INDEX-operations.patch on master? Currently it does not apply on
754b90f657bd54b482524b73726dae4a9165031c

15:56:44.694 | building index (3 of 8): initializing (1/5) | 442478 | 442399 | 0 | 0 | 0 | 0
15:56:44.705 | building index (3 of 8): sorting tuples, spool 1 (3/5) | 442478 | 442399 | 100000000 | 0 | 0 | 0
15:56:44.716 | building index (3 of 8): sorting tuples, spool 1 (3/5) | 442478 | 442399 | 100000000 | 0 | 0 | 0
15:56:44.727 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 79057 | 0 | 0
15:56:44.738 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 217018 | 0 | 0
15:56:44.75 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 353804 | 0 | 0

2. In the above report, even though we are reporting progress in terms
of tuples_done for final btree sort & load phase we have not cleared
the blocks_done entry from previous phases. I think this can be
confusing as the blocks_done does not correspond to the tuples_done in
the final btree sort & load phase.

Thank you,
Rahila Syed

#17Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Rahila (#16)
2 attachment(s)
Re: monitoring CREATE INDEX [CONCURRENTLY]

Hi Rahila,

Thanks for looking.

On 2019-Mar-04, Rahila wrote:

1. Thank you for incorporating review comments.
Can you please rebase the latest 0001-Report-progress-of-
CREATE-INDEX-operations.patch on master? Currently it does not apply on
754b90f657bd54b482524b73726dae4a9165031c

Hmm, rebased to current master. Didn't conflict at all when rebasing,
so it's strange that it didn't apply.

15:56:44.694 | building index (3 of 8): initializing (1/5) | 442478 | 442399 | 0 | 0 | 0 | 0
15:56:44.705 | building index (3 of 8): sorting tuples, spool 1 (3/5) | 442478 | 442399 | 100000000 | 0 | 0 | 0
15:56:44.716 | building index (3 of 8): sorting tuples, spool 1 (3/5) | 442478 | 442399 | 100000000 | 0 | 0 | 0
15:56:44.727 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 79057 | 0 | 0
15:56:44.738 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 217018 | 0 | 0
15:56:44.75 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 353804 | 0 | 0

2. In the above report, even though we are reporting progress in terms of
tuples_done for final btree sort & load phase we have not cleared
the blocks_done entry from previous phases. I think this can be confusing as
the blocks_done does not correspond to the tuples_done in the final btree
sort & load phase.

Good point. Done in the attached version, wherein I also added comments
to explain the IndexBuildHeapScan API change. I didn't change the hash
AM implementation here.

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attachments:

v5-0001-Report-progress-of-CREATE-INDEX-operations.patchtext/x-diff; charset=us-asciiDownload
From 52770a9255950b4da74579d7da212246d29a268b Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Wed, 2 Jan 2019 16:14:39 -0300
Subject: [PATCH v5 1/2] Report progress of CREATE INDEX operations

---
 contrib/amcheck/verify_nbtree.c       |   2 +-
 contrib/bloom/blinsert.c              |   2 +-
 contrib/bloom/blutils.c               |   1 +
 doc/src/sgml/indexam.sgml             |  13 ++
 doc/src/sgml/monitoring.sgml          | 227 +++++++++++++++++++++++++-
 src/backend/access/brin/brin.c        |   5 +-
 src/backend/access/gin/gininsert.c    |   2 +-
 src/backend/access/gin/ginutil.c      |   1 +
 src/backend/access/gist/gist.c        |   1 +
 src/backend/access/gist/gistbuild.c   |   2 +-
 src/backend/access/hash/hash.c        |   3 +-
 src/backend/access/nbtree/nbtree.c    |   9 +
 src/backend/access/nbtree/nbtsort.c   |  57 ++++++-
 src/backend/access/nbtree/nbtutils.c  |  24 +++
 src/backend/access/spgist/spginsert.c |   2 +-
 src/backend/access/spgist/spgutils.c  |   1 +
 src/backend/catalog/index.c           | 147 ++++++++++++++++-
 src/backend/catalog/system_views.sql  |  27 +++
 src/backend/commands/indexcmds.c      |  52 +++++-
 src/backend/storage/ipc/standby.c     |   2 +-
 src/backend/storage/lmgr/lmgr.c       |  46 +++++-
 src/backend/storage/lmgr/lock.c       |   7 +-
 src/backend/utils/adt/amutils.c       |  23 +++
 src/backend/utils/adt/pgstatfuncs.c   |   2 +
 src/include/access/amapi.h            |   4 +
 src/include/access/genam.h            |   1 +
 src/include/access/nbtree.h           |  11 ++
 src/include/catalog/index.h           |   2 +
 src/include/catalog/pg_proc.dat       |  10 +-
 src/include/commands/progress.h       |  36 ++++
 src/include/pgstat.h                  |   5 +-
 src/include/storage/lmgr.h            |   4 +-
 src/include/storage/lock.h            |   2 +-
 33 files changed, 695 insertions(+), 38 deletions(-)

diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index 964200a7678..99f6ed6bc44 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -534,7 +534,7 @@ bt_check_every_level(Relation rel, Relation heaprel, bool readonly,
 			 RelationGetRelationName(state->rel),
 			 RelationGetRelationName(state->heaprel));
 
-		IndexBuildHeapScan(state->heaprel, state->rel, indexinfo, true,
+		IndexBuildHeapScan(state->heaprel, state->rel, indexinfo, true, false,
 						   bt_tuple_present_callback, (void *) state, scan);
 
 		ereport(DEBUG1,
diff --git a/contrib/bloom/blinsert.c b/contrib/bloom/blinsert.c
index e43fbe0005f..947ee74881f 100644
--- a/contrib/bloom/blinsert.c
+++ b/contrib/bloom/blinsert.c
@@ -141,7 +141,7 @@ blbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	initCachedPage(&buildstate);
 
 	/* Do the heap scan */
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false,
 								   bloomBuildCallback, (void *) &buildstate,
 								   NULL);
 
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index 64583765787..697a37a384b 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -132,6 +132,7 @@ blhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = blcostestimate;
 	amroutine->amoptions = bloptions;
 	amroutine->amproperty = NULL;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = blvalidate;
 	amroutine->ambeginscan = blbeginscan;
 	amroutine->amrescan = blrescan;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index 05102724ead..fa4a3d0d131 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -127,6 +127,7 @@ typedef struct IndexAmRoutine
     amcostestimate_function amcostestimate;
     amoptions_function amoptions;
     amproperty_function amproperty;     /* can be NULL */
+    ambuildphasename_function ambuildphasename;   /* can be NULL */
     amvalidate_function amvalidate;
     ambeginscan_function ambeginscan;
     amrescan_function amrescan;
@@ -468,6 +469,18 @@ amproperty (Oid index_oid, int attno,
 
   <para>
 <programlisting>
+char *
+ambuildphasename (int64 phasenum);
+</programlisting>
+   Return the textual name of the given build phase number.
+   The phase numbers are those reported during an index build via the
+   <function>pgstat_progress_update_param</function> interface.
+   The phase names are then exposed in the
+   <structname>pg_stat_progress_create_index</structname> view.
+  </para>
+
+  <para>
+<programlisting>
 bool
 amvalidate (Oid opclassoid);
 </programlisting>
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 0e73cdcddab..47708570f84 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -336,6 +336,14 @@ postgres   27093  0.0  0.0  30096  2752 ?        Ss   11:34   0:00 postgres: ser
       </entry>
      </row>
 
+     <row>
+      <entry><structname>pg_stat_progress_create_index</structname><indexterm><primary>pg_stat_progress_create_index</primary></indexterm></entry>
+      <entry>One row for each backend running <command>CREATE INDEX</command>, showing
+      current progress.
+      See <xref linkend='create-index-progress-reporting'/>.
+     </entry>
+     </row>
+
      <row>
       <entry><structname>pg_stat_progress_vacuum</structname><indexterm><primary>pg_stat_progress_vacuum</primary></indexterm></entry>
       <entry>One row for each backend (including autovacuum worker processes) running
@@ -3376,11 +3384,224 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
 
   <para>
    <productname>PostgreSQL</productname> has the ability to report the progress of
-   certain commands during command execution.  Currently, the only command
-   which supports progress reporting is <command>VACUUM</command>.  This may be
-   expanded in the future.
+   certain commands during command execution.  Currently, the only commands
+   which support progress reporting are <command>CREATE INDEX</command> and
+   <command>VACUUM</command>.  This may be expanded in the future.
   </para>
 
+ <sect2 id="create-index-progress-reporting">
+  <title>CREATE INDEX Progress Reporting</title>
+
+  <para>
+   Whenever <command>CREATE INDEX</command> is running, the
+   <structname>pg_stat_progress_create_index</structname> view will contain
+   one row for each backend that is currently creating indexes.  The tables
+   below describe the information that will be reported and provide information
+   about how to interpret it.
+  </para>
+
+  <table id="pg-stat-progress-create-index-view" xreflabel="pg_stat_progress_create_index">
+   <title><structname>pg_stat_progress_create_index</structname> View</title>
+   <tgroup cols="3">
+    <thead>
+     <row>
+      <entry>Column</entry>
+      <entry>Type</entry>
+      <entry>Description</entry>
+     </row>
+    </thead>
+
+    <tbody>
+     <row>
+      <entry><structfield>pid</structfield></entry>
+      <entry><type>integer</type></entry>
+      <entry>Process ID of backend.</entry>
+     </row>
+     <row>
+      <entry><structfield>datid</structfield></entry>
+      <entry><type>oid</type></entry>
+      <entry>OID of the database to which this backend is connected.</entry>
+     </row>
+     <row>
+      <entry><structfield>datname</structfield></entry>
+      <entry><type>name</type></entry>
+      <entry>Name of the database to which this backend is connected.</entry>
+     </row>
+     <row>
+      <entry><structfield>relid</structfield></entry>
+      <entry><type>oid</type></entry>
+      <entry>OID of the table on which the index is being created.</entry>
+     </row>
+     <row>
+      <entry><structfield>phase</structfield></entry>
+      <entry><type>text</type></entry>
+      <entry>
+        Current processing phase of index creation.  See <xref linkend='create-index-phases'/>.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>lockers_total</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Total number of lockers to wait for, when applicable.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>lockers_done</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Number of lockers already waited for.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>current_locked_pid</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Process ID of the locker currently being waited for.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>blocks_total</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Total number of blocks to be processed in the current phase.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>blocks_done</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Number of blocks already processed in the current phase.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>tuples_total</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Total number of tuples to be processed in the current phase.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>tuples_done</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Number of tuples already processed in the current phase.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>partitions_total</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+       When creating an index on a partitioned, this column is set to the
+       total number of partitions on which the index is to be created.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>partitions_done</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+       When creating an index on a partitioned, this column is set to the
+       number of partitions on which the index has been completed.
+      </entry>
+     </row>
+    </tbody>
+   </tgroup>
+  </table>
+
+  <table id="create-index-phases">
+   <title>CREATE INDEX phases</title>
+   <tgroup cols="2">
+    <thead>
+     <row>
+      <entry>Phase</entry>
+      <entry>Description</entry>
+     </row>
+    </thead>
+    <tbody>
+     <row>
+      <entry><literal>initializing</literal></entry>
+      <entry>
+       <command>CREATE INDEX</command> is preparing to create the index.  This
+       phase is expected to be very brief.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>waiting for old snapshots</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions
+       that can potentially see the table to release their snapshots.
+       This phase is skipped when not in concurrent mode.
+       Columns <structname>lockers_total</structname>, <structname>lockers_done</structname>
+       and <structname>current_locker_pid</structname> contain the progress 
+       information for this phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>building index</literal></entry>
+      <entry>
+       The index is being built by the access method-specific code.  In this phase,
+       access methods that support progress reporting fill in their own progress data,
+       and the subphase is indicated in this column.  Typically,
+       <structname>blocks_total</structname> and <structname>blocks_done</structname>
+       will contain progress data, as well as potentially
+       <structname>tuples_total</structname> and <structname>tuples_done</structname>.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>waiting for writer snapshots</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions
+       that can potentially write into the table to release their snapshots.
+       This phase is skipped when not in concurrent mode.
+       Columns <structname>lockers_total</structname>, <structname>lockers_done</structname>
+       and <structname>current_locker_pid</structname> contain the progress 
+       information for this phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>validating index scan</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is scanning the index searching
+       for tuples that need to be validated.
+       This phase is skipped when not in concurrent mode.
+       Columns <structname>blocks_total</structname> (set to the total size of the index)
+       and <structname>blocks_done</structname> contain the progress information for this phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>sorting index scan results</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is sorting the output of the
+       previous phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>validating index heapscan</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is scanning the table
+       to validate the index tuples collected in the previous two phases.
+       This phase is skipped when not in concurrent mode.
+       Columns <structname>blocks_total</structname> (set to the total size of the table)
+       and <structname>blocks_done</structname> contain the progress information for this phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>waiting for reader snapshots</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions
+       that can potentially see the table to release their snapshots.  This
+       phase is skipped when not in concurrent mode.
+       Columns <structname>lockers_total</structname>, <structname>lockers_done</structname>
+       and <structname>current_locker_pid</structname> contain the progress 
+       information for this phase.
+      </entry>
+     </row>
+    </tbody>
+   </tgroup>
+  </table>
+
+ </sect2>
+
  <sect2 id="vacuum-progress-reporting">
   <title>VACUUM Progress Reporting</title>
 
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 8f008dd0080..5d1aff34080 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -111,6 +111,7 @@ brinhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = brincostestimate;
 	amroutine->amoptions = brinoptions;
 	amroutine->amproperty = NULL;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = brinvalidate;
 	amroutine->ambeginscan = brinbeginscan;
 	amroutine->amrescan = brinrescan;
@@ -718,7 +719,7 @@ brinbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	 * Now scan the relation.  No syncscan allowed here because we want the
 	 * heap blocks in physical order.
 	 */
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, false,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, false, false,
 								   brinbuildCallback, (void *) state, NULL);
 
 	/* process the final batch */
@@ -1234,7 +1235,7 @@ summarize_range(IndexInfo *indexInfo, BrinBuildState *state, Relation heapRel,
 	 * by transactions that are still in progress, among other corner cases.
 	 */
 	state->bs_currRangeStart = heapBlk;
-	IndexBuildHeapRangeScan(heapRel, state->bs_irel, indexInfo, false, true,
+	IndexBuildHeapRangeScan(heapRel, state->bs_irel, indexInfo, false, true, false,
 							heapBlk, scanNumBlks,
 							brinbuildCallback, (void *) state, NULL);
 
diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c
index 524ac5be8b5..838de4c1ec3 100644
--- a/src/backend/access/gin/gininsert.c
+++ b/src/backend/access/gin/gininsert.c
@@ -394,7 +394,7 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	 * Do the heap scan.  We disallow sync scan here because dataPlaceToPage
 	 * prefers to receive tuples in TID order.
 	 */
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, false,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, false, false,
 								   ginBuildCallback, (void *) &buildstate, NULL);
 
 	/* dump remaining entries to the index */
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index afc20232ace..d2360eeafb0 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -64,6 +64,7 @@ ginhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = gincostestimate;
 	amroutine->amoptions = ginoptions;
 	amroutine->amproperty = NULL;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = ginvalidate;
 	amroutine->ambeginscan = ginbeginscan;
 	amroutine->amrescan = ginrescan;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index b75b3a8dacd..0dc36af1e0c 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = gistcostestimate;
 	amroutine->amoptions = gistoptions;
 	amroutine->amproperty = gistproperty;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = gistvalidate;
 	amroutine->ambeginscan = gistbeginscan;
 	amroutine->amrescan = gistrescan;
diff --git a/src/backend/access/gist/gistbuild.c b/src/backend/access/gist/gistbuild.c
index bd142a3560d..015f874cc93 100644
--- a/src/backend/access/gist/gistbuild.c
+++ b/src/backend/access/gist/gistbuild.c
@@ -204,7 +204,7 @@ gistbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	/*
 	 * Do the heap scan.
 	 */
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false,
 								   gistBuildCallback, (void *) &buildstate, NULL);
 
 	/*
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index f1f01a0956d..fc7db5d6a13 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -82,6 +82,7 @@ hashhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = hashcostestimate;
 	amroutine->amoptions = hashoptions;
 	amroutine->amproperty = NULL;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = hashvalidate;
 	amroutine->ambeginscan = hashbeginscan;
 	amroutine->amrescan = hashrescan;
@@ -159,7 +160,7 @@ hashbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	buildstate.heapRel = heap;
 
 	/* do the heap scan */
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false,
 								   hashbuildCallback, (void *) &buildstate, NULL);
 
 	if (buildstate.spool)
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index 98917de2efd..f9d32c958f6 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -22,6 +22,7 @@
 #include "access/nbtxlog.h"
 #include "access/relscan.h"
 #include "access/xlog.h"
+#include "commands/progress.h"
 #include "commands/vacuum.h"
 #include "miscadmin.h"
 #include "nodes/execnodes.h"
@@ -133,6 +134,7 @@ bthandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = btcostestimate;
 	amroutine->amoptions = btoptions;
 	amroutine->amproperty = btproperty;
+	amroutine->ambuildphasename = btbuildphasename;
 	amroutine->amvalidate = btvalidate;
 	amroutine->ambeginscan = btbeginscan;
 	amroutine->amrescan = btrescan;
@@ -1021,6 +1023,10 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
 		if (needLock)
 			UnlockRelationForExtension(rel, ExclusiveLock);
 
+		if (info->report_progress)
+			pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL,
+										 num_pages);
+
 		/* Quit if we've scanned the whole relation */
 		if (blkno >= num_pages)
 			break;
@@ -1028,6 +1034,9 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
 		for (; blkno < num_pages; blkno++)
 		{
 			btvacuumpage(&vstate, blkno, blkno);
+			if (info->report_progress)
+				pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
+											 blkno);
 		}
 	}
 
diff --git a/src/backend/access/nbtree/nbtsort.c b/src/backend/access/nbtree/nbtsort.c
index dc398e11867..309a2082063 100644
--- a/src/backend/access/nbtree/nbtsort.c
+++ b/src/backend/access/nbtree/nbtsort.c
@@ -65,6 +65,7 @@
 #include "access/xlog.h"
 #include "access/xloginsert.h"
 #include "catalog/index.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
 #include "pgstat.h"
 #include "storage/smgr.h"
@@ -288,7 +289,8 @@ static double _bt_parallel_heapscan(BTBuildState *buildstate,
 static void _bt_leader_participate_as_worker(BTBuildState *buildstate);
 static void _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2,
 						   BTShared *btshared, Sharedsort *sharedsort,
-						   Sharedsort *sharedsort2, int sortmem);
+						   Sharedsort *sharedsort2, int sortmem,
+						   bool progress);
 
 
 /*
@@ -384,6 +386,10 @@ _bt_spools_heapscan(Relation heap, Relation index, BTBuildState *buildstate,
 	/* Save as primary spool */
 	buildstate->spool = btspool;
 
+	/* Report heap scan phase started */
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+								 PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN);
+
 	/* Attempt to launch parallel worker scan when required */
 	if (indexInfo->ii_ParallelWorkers > 0)
 		_bt_begin_parallel(buildstate, indexInfo->ii_Concurrent,
@@ -470,13 +476,31 @@ _bt_spools_heapscan(Relation heap, Relation index, BTBuildState *buildstate,
 
 	/* Fill spool using either serial or parallel heap scan */
 	if (!buildstate->btleader)
-		reltuples = IndexBuildHeapScan(heap, index, indexInfo, true,
+		reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, true,
 									   _bt_build_callback, (void *) buildstate,
 									   NULL);
 	else
 		reltuples = _bt_parallel_heapscan(buildstate,
 										  &indexInfo->ii_BrokenHotChain);
 
+	/*
+	 * Set the progress target for the next phase.  Reset the block number
+	 * values set by IndexBuildHeapScan
+	 */
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_TUPLES_TOTAL,
+			PROGRESS_SCAN_BLOCKS_TOTAL,
+			PROGRESS_SCAN_BLOCKS_DONE
+		};
+		const int64 val[] = {
+			buildstate->indtuples,
+			0, 0
+		};
+
+		pgstat_progress_update_multi_param(3, index, val);
+	}
+
 	/* okay, all heap tuples are spooled */
 	if (buildstate->spool2 && !buildstate->havedead)
 	{
@@ -525,9 +549,15 @@ _bt_leafbuild(BTSpool *btspool, BTSpool *btspool2)
 	}
 #endif							/* BTREE_BUILD_STATS */
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+								 PROGRESS_BTREE_PHASE_PERFORMSORT_1);
 	tuplesort_performsort(btspool->sortstate);
 	if (btspool2)
+	{
+		pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+									 PROGRESS_BTREE_PHASE_PERFORMSORT_2);
 		tuplesort_performsort(btspool2->sortstate);
+	}
 
 	wstate.heap = btspool->heap;
 	wstate.index = btspool->index;
@@ -543,6 +573,8 @@ _bt_leafbuild(BTSpool *btspool, BTSpool *btspool2)
 	wstate.btws_pages_written = 0;
 	wstate.btws_zeropage = NULL;	/* until needed */
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+								 PROGRESS_BTREE_PHASE_LEAF_LOAD);
 	_bt_load(&wstate, btspool, btspool2);
 }
 
@@ -1078,6 +1110,7 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
 				keysz = IndexRelationGetNumberOfKeyAttributes(wstate->index);
 	ScanKey		indexScanKey = NULL;
 	SortSupport sortKeys;
+	long		tuples_done = 0L;
 
 	if (merge)
 	{
@@ -1170,6 +1203,10 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
 				_bt_buildadd(wstate, state, itup2);
 				itup2 = tuplesort_getindextuple(btspool2->sortstate, true);
 			}
+
+			/* Report progress */
+			pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE,
+										 ++tuples_done);
 		}
 		pfree(sortKeys);
 	}
@@ -1184,6 +1221,10 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
 				state = _bt_pagestate(wstate, 0);
 
 			_bt_buildadd(wstate, state, itup);
+
+			/* Report progress */
+			pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE,
+										 ++tuples_done);
 		}
 	}
 
@@ -1318,6 +1359,10 @@ _bt_begin_parallel(BTBuildState *buildstate, bool isconcurrent, int request)
 	btshared->brokenhotchain = false;
 	heap_parallelscan_initialize(&btshared->heapdesc, btspool->heap, snapshot);
 
+	/* Report total number of blocks to scan */
+	pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL,
+								 btshared->heapdesc.phs_nblocks);
+
 	/*
 	 * Store shared tuplesort-private state, for which we reserved space.
 	 * Then, initialize opaque state using tuplesort routine.
@@ -1493,7 +1538,7 @@ _bt_leader_participate_as_worker(BTBuildState *buildstate)
 	/* Perform work common to all participants */
 	_bt_parallel_scan_and_sort(leaderworker, leaderworker2, btleader->btshared,
 							   btleader->sharedsort, btleader->sharedsort2,
-							   sortmem);
+							   sortmem, true);
 
 #ifdef BTREE_BUILD_STATS
 	if (log_btree_build_stats)
@@ -1584,7 +1629,7 @@ _bt_parallel_build_main(dsm_segment *seg, shm_toc *toc)
 	/* Perform sorting of spool, and possibly a spool2 */
 	sortmem = maintenance_work_mem / btshared->scantuplesortstates;
 	_bt_parallel_scan_and_sort(btspool, btspool2, btshared, sharedsort,
-							   sharedsort2, sortmem);
+							   sharedsort2, sortmem, false);
 
 #ifdef BTREE_BUILD_STATS
 	if (log_btree_build_stats)
@@ -1613,7 +1658,7 @@ _bt_parallel_build_main(dsm_segment *seg, shm_toc *toc)
 static void
 _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2,
 						   BTShared *btshared, Sharedsort *sharedsort,
-						   Sharedsort *sharedsort2, int sortmem)
+						   Sharedsort *sharedsort2, int sortmem, bool progress)
 {
 	SortCoordinate coordinate;
 	BTBuildState buildstate;
@@ -1672,7 +1717,7 @@ _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2,
 	indexInfo->ii_Concurrent = btshared->isconcurrent;
 	scan = heap_beginscan_parallel(btspool->heap, &btshared->heapdesc);
 	reltuples = IndexBuildHeapScan(btspool->heap, btspool->index, indexInfo,
-								   true, _bt_build_callback,
+								   true, progress, _bt_build_callback,
 								   (void *) &buildstate, scan);
 
 	/*
diff --git a/src/backend/access/nbtree/nbtutils.c b/src/backend/access/nbtree/nbtutils.c
index 2c05fb5e451..295c22b2e5b 100644
--- a/src/backend/access/nbtree/nbtutils.c
+++ b/src/backend/access/nbtree/nbtutils.c
@@ -20,6 +20,7 @@
 #include "access/nbtree.h"
 #include "access/reloptions.h"
 #include "access/relscan.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
 #include "utils/array.h"
 #include "utils/lsyscache.h"
@@ -2082,6 +2083,29 @@ btproperty(Oid index_oid, int attno,
 	}
 }
 
+/*
+ *	btbuildphasename() -- Return name of index build phase.
+ */
+char *
+btbuildphasename(int64 phasenum)
+{
+	switch (phasenum)
+	{
+		case PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE:
+			return "initializing (1 of 5)";
+		case PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN:
+			return "table scan (2 of 5)";
+		case PROGRESS_BTREE_PHASE_PERFORMSORT_1:
+			return "sorting tuples, spool 1 (3 of 5)";
+		case PROGRESS_BTREE_PHASE_PERFORMSORT_2:
+			return "sorting tuples, spool 2 (4 of 5)";
+		case PROGRESS_BTREE_PHASE_LEAF_LOAD:
+			return "btree tuple loading (5 of 5)";
+		default:
+			return NULL;
+	}
+}
+
 /*
  *	_bt_nonkey_truncate() -- create tuple without non-key suffix attributes.
  *
diff --git a/src/backend/access/spgist/spginsert.c b/src/backend/access/spgist/spginsert.c
index f428a151385..1bc671c7238 100644
--- a/src/backend/access/spgist/spginsert.c
+++ b/src/backend/access/spgist/spginsert.c
@@ -142,7 +142,7 @@ spgbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 											  "SP-GiST build temporary context",
 											  ALLOCSET_DEFAULT_SIZES);
 
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false,
 								   spgistBuildCallback, (void *) &buildstate,
 								   NULL);
 
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 8e63c1fad25..45472db147b 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -67,6 +67,7 @@ spghandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = spgcostestimate;
 	amroutine->amoptions = spgoptions;
 	amroutine->amproperty = spgproperty;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = spgvalidate;
 	amroutine->ambeginscan = spgbeginscan;
 	amroutine->amrescan = spgrescan;
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index d16c3d0ea50..110f13d1e46 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -49,8 +49,9 @@
 #include "catalog/pg_trigger.h"
 #include "catalog/pg_type.h"
 #include "catalog/storage.h"
-#include "commands/tablecmds.h"
 #include "commands/event_trigger.h"
+#include "commands/progress.h"
+#include "commands/tablecmds.h"
 #include "commands/trigger.h"
 #include "executor/executor.h"
 #include "miscadmin.h"
@@ -58,6 +59,7 @@
 #include "nodes/nodeFuncs.h"
 #include "optimizer/optimizer.h"
 #include "parser/parser.h"
+#include "pgstat.h"
 #include "rewrite/rewriteManip.h"
 #include "storage/bufmgr.h"
 #include "storage/lmgr.h"
@@ -1597,7 +1599,7 @@ index_drop(Oid indexId, bool concurrent)
 		 * to acquire an exclusive lock on our table.  The lock code will
 		 * detect deadlock and error out properly.
 		 */
-		WaitForLockers(heaplocktag, AccessExclusiveLock);
+		WaitForLockers(heaplocktag, AccessExclusiveLock, true);
 
 		/*
 		 * No more predicate locks will be acquired on this index, and we're
@@ -1641,7 +1643,7 @@ index_drop(Oid indexId, bool concurrent)
 		 * Wait till every transaction that saw the old index state has
 		 * finished.
 		 */
-		WaitForLockers(heaplocktag, AccessExclusiveLock);
+		WaitForLockers(heaplocktag, AccessExclusiveLock, true);
 
 		/*
 		 * Re-open relations to allow us to complete our actions.
@@ -2291,6 +2293,25 @@ index_build(Relation heapRelation,
 						   save_sec_context | SECURITY_RESTRICTED_OPERATION);
 	save_nestlevel = NewGUCNestLevel();
 
+	/* Set up initial progress report status */
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_PHASE,
+			PROGRESS_CREATEIDX_SUBPHASE,
+			PROGRESS_CREATEIDX_TUPLES_DONE,
+			PROGRESS_CREATEIDX_TUPLES_TOTAL,
+			PROGRESS_SCAN_BLOCKS_DONE,
+			PROGRESS_SCAN_BLOCKS_TOTAL
+		};
+		const int64	val[] = {
+			PROGRESS_CREATEIDX_PHASE_BUILD,
+			PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE,
+			0, 0, 0, 0
+		};
+
+		pgstat_progress_update_multi_param(6, index, val);
+	}
+
 	/*
 	 * Call the access method's build procedure
 	 */
@@ -2417,6 +2438,12 @@ index_build(Relation heapRelation,
  * do so here because the AM might reject some of the tuples for its own
  * reasons, such as being unable to store NULLs.
  *
+ * If 'progress', we update the PROGRESS_SCAN_BLOCKS_DONE counter as we go
+ * along.  Also, if that flag is true and a scan descriptor is not passed (ie.
+ * when not doing a parallel scan), the PROGRESS_SCAN_BLOCKS_TOTAL counter is
+ * updated at the beginning.  For parallel scans, caller is expected to have
+ * set the total number of blocks prior to calling this function.  
+ *
  * A side effect is to set indexInfo->ii_BrokenHotChain to true if we detect
  * any potentially broken HOT chains.  Currently, we set this if there are
  * any RECENTLY_DEAD or DELETE_IN_PROGRESS entries in a HOT chain, without
@@ -2428,13 +2455,14 @@ IndexBuildHeapScan(Relation heapRelation,
 				   Relation indexRelation,
 				   IndexInfo *indexInfo,
 				   bool allow_sync,
+				   bool progress,
 				   IndexBuildCallback callback,
 				   void *callback_state,
 				   HeapScanDesc scan)
 {
 	return IndexBuildHeapRangeScan(heapRelation, indexRelation,
 								   indexInfo, allow_sync,
-								   false,
+								   false, progress,
 								   0, InvalidBlockNumber,
 								   callback, callback_state, scan);
 }
@@ -2445,6 +2473,13 @@ IndexBuildHeapScan(Relation heapRelation,
  * passing InvalidBlockNumber as numblocks.  Note that restricting the range
  * to scan cannot be done when requesting syncscan.
  *
+ * If 'progress', we update the PROGRESS_SCAN_BLOCKS_DONE counter as we go
+ * along.  Also, if that flag is true and a scan descriptor is not passed (ie.
+ * when not doing a parallel scan), the PROGRESS_SCAN_BLOCKS_TOTAL counter is
+ * updated at the beginning.  For parallel scans, caller is expected to have
+ * set the total number of blocks prior to calling this function.  Note this
+ * only works for full-relation scans.
+ *
  * When "anyvisible" mode is requested, all tuples visible to any transaction
  * are indexed and counted as live, including those inserted or deleted by
  * transactions that are still in progress.
@@ -2455,6 +2490,7 @@ IndexBuildHeapRangeScan(Relation heapRelation,
 						IndexInfo *indexInfo,
 						bool allow_sync,
 						bool anyvisible,
+						bool progress,
 						BlockNumber start_blockno,
 						BlockNumber numblocks,
 						IndexBuildCallback callback,
@@ -2476,6 +2512,8 @@ IndexBuildHeapRangeScan(Relation heapRelation,
 	TransactionId OldestXmin;
 	BlockNumber root_blkno = InvalidBlockNumber;
 	OffsetNumber root_offsets[MaxHeapTuplesPerPage];
+	BlockNumber blocks_done = 0;
+	BlockNumber	previous_blkno = InvalidBlockNumber;
 
 	/*
 	 * sanity checks
@@ -2545,6 +2583,13 @@ IndexBuildHeapRangeScan(Relation heapRelation,
 									NULL,	/* scan key */
 									true,	/* buffer access strategy OK */
 									allow_sync);	/* syncscan OK? */
+
+		if (progress)
+		{
+			Assert(scan->rs_numblocks == InvalidBlockNumber);
+			pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL,
+										 scan->rs_nblocks);
+		}
 	}
 	else
 	{
@@ -2592,6 +2637,46 @@ IndexBuildHeapRangeScan(Relation heapRelation,
 
 		CHECK_FOR_INTERRUPTS();
 
+		/* Report scan progress, if asked to. */
+		if (progress &&
+			((previous_blkno == InvalidBlockNumber) ||
+			(scan->rs_cblock != previous_blkno)))
+		{
+			/* we only do progress for full table scans */
+			Assert(numblocks == InvalidBlockNumber);
+
+			/*
+			 * Report the number of blocks we've moved forward.
+			 *
+			 * Parallel workers cause the leader process to skip some blocks,
+			 * so we subtract the current block number to the previous one to
+			 * determine how many have been read in total; but be careful when
+			 * we wrap around the last block in the table.
+			 */
+			if (scan->rs_cblock > previous_blkno)
+				blocks_done += scan->rs_cblock - previous_blkno;
+			else if (previous_blkno == InvalidBlockNumber)
+			{
+				/*
+				 * How many blocks have been read since the scan started.
+				 * Should normally be zero.
+				 */
+				blocks_done += scan->rs_cblock -
+					(scan->rs_parallel ? scan->rs_parallel->phs_startblock :
+					 scan->rs_startblock);
+				Assert(blocks_done == 0);
+			}
+			else
+			{
+				/* wrapped around */
+				blocks_done += scan->rs_nblocks - previous_blkno + scan->rs_cblock;
+			}
+
+			pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
+										 blocks_done);
+			previous_blkno = scan->rs_cblock;
+		}
+
 		/*
 		 * When dealing with a HOT-chain of updated tuples, we want to index
 		 * the values of the live tuple (if any), but index it under the TID
@@ -3133,6 +3218,21 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot)
 	int			save_sec_context;
 	int			save_nestlevel;
 
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_PHASE,
+			PROGRESS_CREATEIDX_TUPLES_DONE,
+			PROGRESS_CREATEIDX_TUPLES_TOTAL,
+			PROGRESS_SCAN_BLOCKS_DONE,
+			PROGRESS_SCAN_BLOCKS_TOTAL
+		};
+		const int64	val[] = {
+			PROGRESS_CREATEIDX_PHASE_VALIDATE_IDXSCAN,
+			0, 0, 0, 0
+		};
+		pgstat_progress_update_multi_param(5, index, val);
+	}
+
 	/* Open and lock the parent heap relation */
 	heapRelation = table_open(heapId, ShareUpdateExclusiveLock);
 	/* And the target index relation */
@@ -3163,6 +3263,7 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot)
 	 */
 	ivinfo.index = indexRelation;
 	ivinfo.analyze_only = false;
+	ivinfo.report_progress = true;	/* XXX only for btree? */
 	ivinfo.estimated_count = true;
 	ivinfo.message_level = DEBUG2;
 	ivinfo.num_heap_tuples = heapRelation->rd_rel->reltuples;
@@ -3180,15 +3281,39 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot)
 											NULL, false);
 	state.htups = state.itups = state.tups_inserted = 0;
 
+	/* ambulkdelete updates progress metrics */
 	(void) index_bulk_delete(&ivinfo, NULL,
 							 validate_index_callback, (void *) &state);
 
 	/* Execute the sort */
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_PHASE,
+			PROGRESS_SCAN_BLOCKS_DONE,
+			PROGRESS_SCAN_BLOCKS_TOTAL
+		};
+		const int64	val[] = {
+			PROGRESS_CREATEIDX_PHASE_SORT_IDXSCAN,
+			0, 0
+		};
+
+		pgstat_progress_update_multi_param(3, index, val);
+	}
 	tuplesort_performsort(state.tuplesort);
 
 	/*
-	 * Now scan the heap and "merge" it with the index
+	 * Now scan the heap and "merge" it with the index.
 	 */
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_PHASE,
+		};
+		const int64	val[] = {
+			PROGRESS_CREATEIDX_PHASE_HEAPSCAN_VALIDATE
+		};
+
+		pgstat_progress_update_multi_param(1, index, val);
+	}
 	validate_index_heapscan(heapRelation,
 							indexRelation,
 							indexInfo,
@@ -3291,6 +3416,7 @@ validate_index_heapscan(Relation heapRelation,
 	BlockNumber root_blkno = InvalidBlockNumber;
 	OffsetNumber root_offsets[MaxHeapTuplesPerPage];
 	bool		in_index[MaxHeapTuplesPerPage];
+	BlockNumber	previous_blkno = InvalidBlockNumber;
 
 	/* state variables for the merge */
 	ItemPointer indexcursor = NULL;
@@ -3330,6 +3456,9 @@ validate_index_heapscan(Relation heapRelation,
 								true,	/* buffer access strategy OK */
 								false); /* syncscan not OK */
 
+	pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL,
+								 scan->rs_nblocks);
+
 	/*
 	 * Scan all tuples matching the snapshot.
 	 */
@@ -3343,6 +3472,14 @@ validate_index_heapscan(Relation heapRelation,
 
 		state->htups += 1;
 
+		if ((previous_blkno == InvalidBlockNumber) ||
+			(scan->rs_cblock != previous_blkno))
+		{
+			pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
+										 scan->rs_cblock);
+			previous_blkno = scan->rs_cblock;
+		}
+
 		/*
 		 * As commented in IndexBuildHeapScan, we should index heap-only
 		 * tuples under the TIDs of their root tuples; so when we advance onto
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 3e229c693c4..3cbe6f8af2a 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -906,6 +906,33 @@ CREATE VIEW pg_stat_progress_vacuum AS
     FROM pg_stat_get_progress_info('VACUUM') AS S
 		LEFT JOIN pg_database D ON S.datid = D.oid;
 
+CREATE VIEW pg_stat_progress_create_index AS
+	SELECT
+		S.pid AS pid, S.datid AS datid, D.datname AS datname,
+		S.relid AS relid,
+		CASE S.param2 WHEN 0 THEN 'initializing (phase 1 of 8)'
+					  WHEN 1 THEN 'waiting for old snapshots (phase 2 of 8)'
+					  WHEN 2 THEN 'building index (3 of 8)' ||
+						COALESCE((': ' || pg_indexam_progress_phasename(S.param1::oid, S.param3)),
+							'')
+					  WHEN 3 THEN 'waiting for writer snapshots (phase 4 of 8)'
+					  WHEN 4 THEN 'index validation: scan index (phase 5 of 8)'
+					  WHEN 5 THEN 'index validation: sort index scan results (phase 6 of 8)'
+					  WHEN 6 THEN 'index validation: scan heap (phase 7 of 8)'
+					  WHEN 7 THEN 'waiting for reader snapshots (phase 8 of 8)'
+					  END as phase,
+		S.param4 AS lockers_total,
+		S.param5 AS lockers_done,
+		S.param6 AS current_locker_pid,
+		S.param7 AS blocks_total,
+		S.param8 AS blocks_done,
+		S.param9 AS tuples_total,
+		S.param10 AS tuples_done,
+		S.param11 AS partitions_total,
+		S.param12 AS partitions_done
+	FROM pg_stat_get_progress_info('CREATE INDEX') AS S
+		LEFT JOIN pg_database D ON S.datid = D.oid;
+
 CREATE VIEW pg_user_mappings AS
     SELECT
         U.oid       AS umid,
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 5dcedc337aa..8f8587ac1e3 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -35,6 +35,7 @@
 #include "commands/dbcommands.h"
 #include "commands/defrem.h"
 #include "commands/event_trigger.h"
+#include "commands/progress.h"
 #include "commands/tablecmds.h"
 #include "commands/tablespace.h"
 #include "mb/pg_wchar.h"
@@ -46,10 +47,12 @@
 #include "parser/parse_func.h"
 #include "parser/parse_oper.h"
 #include "partitioning/partdesc.h"
+#include "pgstat.h"
 #include "rewrite/rewriteManip.h"
 #include "storage/lmgr.h"
 #include "storage/proc.h"
 #include "storage/procarray.h"
+#include "storage/sinvaladt.h"
 #include "utils/acl.h"
 #include "utils/builtins.h"
 #include "utils/fmgroids.h"
@@ -368,6 +371,15 @@ DefineIndex(Oid relationId,
 	Snapshot	snapshot;
 	int			i;
 
+
+	/*
+	 * Start progress report.  If we're building a partition, this was already
+	 * done.
+	 */
+	if (!OidIsValid(parentIndexId))
+		pgstat_progress_start_command(PROGRESS_COMMAND_CREATE_INDEX,
+									  relationId);
+
 	/*
 	 * count key attributes in index
 	 */
@@ -584,6 +596,9 @@ DefineIndex(Oid relationId,
 	accessMethodId = accessMethodForm->oid;
 	amRoutine = GetIndexAmRoutine(accessMethodForm->amhandler);
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_ACCESS_METHOD_OID,
+								 accessMethodId);
+
 	if (stmt->unique && !amRoutine->amcanunique)
 		ereport(ERROR,
 				(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -864,6 +879,11 @@ DefineIndex(Oid relationId,
 	if (!OidIsValid(indexRelationId))
 	{
 		table_close(rel, NoLock);
+
+		/* If this is the top-level index, we're done */
+		if (!OidIsValid(parentIndexId))
+			pgstat_progress_end_command();
+
 		return address;
 	}
 
@@ -889,6 +909,9 @@ DefineIndex(Oid relationId,
 			TupleDesc	parentDesc;
 			Oid		   *opfamOids;
 
+			pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL,
+										 nparts);
+
 			memcpy(part_oids, partdesc->oids, sizeof(Oid) * nparts);
 
 			parentDesc = CreateTupleDescCopy(RelationGetDescr(rel));
@@ -1039,6 +1062,8 @@ DefineIndex(Oid relationId,
 								skip_build, quiet);
 				}
 
+				pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_DONE,
+											 i + 1);
 				pfree(attmap);
 			}
 
@@ -1073,6 +1098,8 @@ DefineIndex(Oid relationId,
 		 * Indexes on partitioned tables are not themselves built, so we're
 		 * done here.
 		 */
+		if (!OidIsValid(parentIndexId))
+			pgstat_progress_end_command();
 		return address;
 	}
 
@@ -1080,6 +1107,11 @@ DefineIndex(Oid relationId,
 	{
 		/* Close the heap and we're done, in the non-concurrent case */
 		table_close(rel, NoLock);
+
+		/* If this is the top-level index, we're done. */
+		if (!OidIsValid(parentIndexId))
+			pgstat_progress_end_command();
+
 		return address;
 	}
 
@@ -1131,7 +1163,9 @@ DefineIndex(Oid relationId,
 	 * exclusive lock on our table.  The lock code will detect deadlock and
 	 * error out properly.
 	 */
-	WaitForLockers(heaplocktag, ShareLock);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_1);
+	WaitForLockers(heaplocktag, ShareLock, true);
 
 	/*
 	 * At this moment we are sure that there are no transactions with the
@@ -1195,7 +1229,9 @@ DefineIndex(Oid relationId,
 	 * We once again wait until no transaction can have the table open with
 	 * the index marked as read-only for updates.
 	 */
-	WaitForLockers(heaplocktag, ShareLock);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_2);
+	WaitForLockers(heaplocktag, ShareLock, true);
 
 	/*
 	 * Now take the "reference snapshot" that will be used by validate_index()
@@ -1281,6 +1317,9 @@ DefineIndex(Oid relationId,
 	old_snapshots = GetCurrentVirtualXIDs(limitXmin, true, false,
 										  PROC_IS_AUTOVACUUM | PROC_IN_VACUUM,
 										  &n_old_snapshots);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_3);
+	pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, n_old_snapshots);
 
 	for (i = 0; i < n_old_snapshots; i++)
 	{
@@ -1316,7 +1355,14 @@ DefineIndex(Oid relationId,
 		}
 
 		if (VirtualTransactionIdIsValid(old_snapshots[i]))
+		{
+			PGPROC *holder = BackendIdGetProc(old_snapshots[i].backendId);
+			pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID,
+										 holder->pid);
 			VirtualXactLock(old_snapshots[i], true);
+		}
+
+		pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, i + 1);
 	}
 
 	/*
@@ -1339,6 +1385,8 @@ DefineIndex(Oid relationId,
 	 */
 	UnlockRelationIdForSession(&heaprelid, ShareUpdateExclusiveLock);
 
+	pgstat_progress_end_command();
+
 	return address;
 }
 
diff --git a/src/backend/storage/ipc/standby.c b/src/backend/storage/ipc/standby.c
index 4d10e57a803..a0a2b964703 100644
--- a/src/backend/storage/ipc/standby.c
+++ b/src/backend/storage/ipc/standby.c
@@ -401,7 +401,7 @@ ResolveRecoveryConflictWithLock(LOCKTAG locktag)
 		 */
 		VirtualTransactionId *backends;
 
-		backends = GetLockConflicts(&locktag, AccessExclusiveLock);
+		backends = GetLockConflicts(&locktag, AccessExclusiveLock, NULL);
 		ResolveRecoveryConflictWithVirtualXIDs(backends,
 											   PROCSIG_RECOVERY_CONFLICT_LOCK);
 	}
diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c
index e688ba81170..0b04b093782 100644
--- a/src/backend/storage/lmgr/lmgr.c
+++ b/src/backend/storage/lmgr/lmgr.c
@@ -19,9 +19,12 @@
 #include "access/transam.h"
 #include "access/xact.h"
 #include "catalog/catalog.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
+#include "pgstat.h"
 #include "storage/lmgr.h"
 #include "storage/procarray.h"
+#include "storage/sinvaladt.h"
 #include "utils/inval.h"
 
 
@@ -857,10 +860,12 @@ XactLockTableWaitErrorCb(void *arg)
  * after we obtained our initial list of lockers, we will not wait for them.
  */
 void
-WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
+WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
 {
 	List	   *holders = NIL;
 	ListCell   *lc;
+	int			total = 0;
+	int			done = 0;
 
 	/* Done if no locks to wait for */
 	if (list_length(locktags) == 0)
@@ -870,10 +875,17 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
 	foreach(lc, locktags)
 	{
 		LOCKTAG    *locktag = lfirst(lc);
+		int			count;
 
-		holders = lappend(holders, GetLockConflicts(locktag, lockmode));
+		holders = lappend(holders,
+						  GetLockConflicts(locktag, lockmode,
+										   progress ? &count : NULL));
+		total += count;
 	}
 
+	if (progress)
+		pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, total);
+
 	/*
 	 * Note: GetLockConflicts() never reports our own xid, hence we need not
 	 * check for that.  Also, prepared xacts are not reported, which is fine
@@ -887,10 +899,36 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
 
 		while (VirtualTransactionIdIsValid(*lockholders))
 		{
+			/*
+			 * If requested, publish who we're going to wait for.  This is not
+			 * 100% accurate if they're already gone, but we don't care.
+			 */
+			if (progress)
+			{
+				PGPROC *holder = BackendIdGetProc(lockholders->backendId);
+
+				pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID,
+											 holder->pid);
+			}
 			VirtualXactLock(*lockholders, true);
 			lockholders++;
+
+			if (progress)
+				pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, ++done);
 		}
 	}
+	if (progress)
+	{
+		const int	index[] = {
+			PROGRESS_WAITFOR_TOTAL,
+			PROGRESS_WAITFOR_DONE,
+			PROGRESS_WAITFOR_CURRENT_PID
+		};
+		const int64	values[] = {
+			0, 0, 0
+		};
+		pgstat_progress_update_multi_param(3, index, values);
+	}
 
 	list_free_deep(holders);
 }
@@ -901,12 +939,12 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
  * Same as WaitForLockersMultiple, for a single lock tag.
  */
 void
-WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode)
+WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress)
 {
 	List	   *l;
 
 	l = list_make1(&heaplocktag);
-	WaitForLockersMultiple(l, lockmode);
+	WaitForLockersMultiple(l, lockmode, progress);
 	list_free(l);
 }
 
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 78fdbd6ff88..c8958766f1e 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2807,6 +2807,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock)
  *		xacts merely awaiting such a lock are NOT reported.
  *
  * The result array is palloc'd and is terminated with an invalid VXID.
+ * *countp, if not null, is updated to the number of items set.
  *
  * Of course, the result could be out of date by the time it's returned,
  * so use of this function has to be thought about carefully.
@@ -2817,7 +2818,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock)
  * uses of the result.
  */
 VirtualTransactionId *
-GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode)
+GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
 {
 	static VirtualTransactionId *vxids;
 	LOCKMETHODID lockmethodid = locktag->locktag_lockmethodid;
@@ -2964,6 +2965,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode)
 		LWLockRelease(partitionLock);
 		vxids[count].backendId = InvalidBackendId;
 		vxids[count].localTransactionId = InvalidLocalTransactionId;
+		if (countp)
+			*countp = count;
 		return vxids;
 	}
 
@@ -3019,6 +3022,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode)
 
 	vxids[count].backendId = InvalidBackendId;
 	vxids[count].localTransactionId = InvalidLocalTransactionId;
+	if (countp)
+		*countp = count;
 	return vxids;
 }
 
diff --git a/src/backend/utils/adt/amutils.c b/src/backend/utils/adt/amutils.c
index 060ffe501ec..e81d6cc0562 100644
--- a/src/backend/utils/adt/amutils.c
+++ b/src/backend/utils/adt/amutils.c
@@ -445,3 +445,26 @@ pg_index_column_has_property(PG_FUNCTION_ARGS)
 
 	return indexam_property(fcinfo, propname, InvalidOid, relid, attno);
 }
+
+/*
+ * Return the name of the given phase, as used for progress reporting by the
+ * given AM.
+ */
+Datum
+pg_indexam_progress_phasename(PG_FUNCTION_ARGS)
+{
+	Oid			amoid = PG_GETARG_OID(0);
+	int32		phasenum = PG_GETARG_INT32(1);
+	IndexAmRoutine *routine;
+	char	   *name;
+
+	routine = GetIndexAmRoutineByAmId(amoid, true);
+	if (routine == NULL || !routine->ambuildphasename)
+		PG_RETURN_NULL();
+
+	name = routine->ambuildphasename(phasenum);
+	if (!name)
+		PG_RETURN_NULL();
+
+	PG_RETURN_TEXT_P(CStringGetTextDatum(name));
+}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 69f72657792..05f9da6a4ad 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -468,6 +468,8 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
 	/* Translate command name into command type code. */
 	if (pg_strcasecmp(cmd, "VACUUM") == 0)
 		cmdtype = PROGRESS_COMMAND_VACUUM;
+	else if (pg_strcasecmp(cmd, "CREATE INDEX") == 0)
+		cmdtype = PROGRESS_COMMAND_CREATE_INDEX;
 	else
 		ereport(ERROR,
 				(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 653ddc976ba..09a7404267c 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -108,6 +108,9 @@ typedef bool (*amproperty_function) (Oid index_oid, int attno,
 									 IndexAMProperty prop, const char *propname,
 									 bool *res, bool *isnull);
 
+/* name of phase as used in progress reporting */
+typedef char *(*ambuildphasename_function) (int64 phasenum);
+
 /* validate definition of an opclass for this AM */
 typedef bool (*amvalidate_function) (Oid opclassoid);
 
@@ -213,6 +216,7 @@ typedef struct IndexAmRoutine
 	amcostestimate_function amcostestimate;
 	amoptions_function amoptions;
 	amproperty_function amproperty; /* can be NULL */
+	ambuildphasename_function ambuildphasename;	/* can be NULL */
 	amvalidate_function amvalidate;
 	ambeginscan_function ambeginscan;
 	amrescan_function amrescan;
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index c4aba39496f..f77d9eea8e8 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -45,6 +45,7 @@ typedef struct IndexVacuumInfo
 {
 	Relation	index;			/* the index being vacuumed */
 	bool		analyze_only;	/* ANALYZE (without any actual vacuum) */
+	bool		report_progress;	/* emit progress.h status reports */
 	bool		estimated_count;	/* num_heap_tuples is an estimate */
 	int			message_level;	/* ereport level for progress messages */
 	double		num_heap_tuples;	/* tuples remaining in heap */
diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h
index 60622ea7906..223aa138837 100644
--- a/src/include/access/nbtree.h
+++ b/src/include/access/nbtree.h
@@ -488,6 +488,16 @@ typedef BTScanOpaqueData *BTScanOpaque;
 #define SK_BT_DESC			(INDOPTION_DESC << SK_BT_INDOPTION_SHIFT)
 #define SK_BT_NULLS_FIRST	(INDOPTION_NULLS_FIRST << SK_BT_INDOPTION_SHIFT)
 
+/*
+ * Constant definition for progress reporting.  Phase numbers must match
+ * btbuildphasename.
+ */
+/* PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE is 1 (see progress.h) */
+#define PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN		2
+#define PROGRESS_BTREE_PHASE_PERFORMSORT_1				3
+#define PROGRESS_BTREE_PHASE_PERFORMSORT_2				4
+#define PROGRESS_BTREE_PHASE_LEAF_LOAD					5
+
 /*
  * external entry points for btree, in nbtree.c
  */
@@ -600,6 +610,7 @@ extern bytea *btoptions(Datum reloptions, bool validate);
 extern bool btproperty(Oid index_oid, int attno,
 		   IndexAMProperty prop, const char *propname,
 		   bool *res, bool *isnull);
+extern char *btbuildphasename(int64 phasenum);
 extern IndexTuple _bt_nonkey_truncate(Relation rel, IndexTuple itup);
 extern bool _bt_check_natts(Relation rel, Page page, OffsetNumber offnum);
 
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index 330c481a8b7..fb713d50f96 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -114,6 +114,7 @@ extern double IndexBuildHeapScan(Relation heapRelation,
 				   Relation indexRelation,
 				   IndexInfo *indexInfo,
 				   bool allow_sync,
+				   bool progress,
 				   IndexBuildCallback callback,
 				   void *callback_state,
 				   struct HeapScanDescData *scan);
@@ -122,6 +123,7 @@ extern double IndexBuildHeapRangeScan(Relation heapRelation,
 						IndexInfo *indexInfo,
 						bool allow_sync,
 						bool anyvisible,
+						bool progress,
 						BlockNumber start_blockno,
 						BlockNumber end_blockno,
 						IndexBuildCallback callback,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 5bb56b2c639..0250f8533a9 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -917,6 +917,10 @@
   proname => 'pg_index_column_has_property', provolatile => 's',
   prorettype => 'bool', proargtypes => 'regclass int4 text',
   prosrc => 'pg_index_column_has_property' },
+{ oid => '676', descr => 'return name of given index build phase',
+  proname => 'pg_indexam_progress_phasename', provolatile => 'i',
+  prorettype => 'text', proargtypes => 'oid int8',
+  prosrc => 'pg_indexam_progress_phasename' },
 
 { oid => '339',
   proname => 'poly_same', prorettype => 'bool',
@@ -5079,9 +5083,9 @@
   proname => 'pg_stat_get_progress_info', prorows => '100', proretset => 't',
   provolatile => 's', proparallel => 'r', prorettype => 'record',
   proargtypes => 'text',
-  proallargtypes => '{text,int4,oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}',
-  proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o}',
-  proargnames => '{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10}',
+  proallargtypes => '{text,int4,oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}',
+  proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+  proargnames => '{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10,param11,param12}',
   prosrc => 'pg_stat_get_progress_info' },
 { oid => '3099',
   descr => 'statistics: information about currently active replication',
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index 9858b36a383..56bd6125594 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -34,4 +34,40 @@
 #define PROGRESS_VACUUM_PHASE_TRUNCATE			5
 #define PROGRESS_VACUUM_PHASE_FINAL_CLEANUP		6
 
+
+/* Progress parameters for CREATE INDEX */
+#define PROGRESS_CREATEIDX_ACCESS_METHOD_OID	0
+#define PROGRESS_CREATEIDX_PHASE				1	/* AM-agnostic phase # */
+#define PROGRESS_CREATEIDX_SUBPHASE				2	/* phase # filled by AM */
+/* 3, 4 and 5 reserved for "waitfor" metrics */
+/* 6 and 7 reserved for "block number" metrics */
+#define PROGRESS_CREATEIDX_TUPLES_TOTAL			8
+#define PROGRESS_CREATEIDX_TUPLES_DONE			9
+#define PROGRESS_CREATEIDX_PARTITIONS_TOTAL		10
+#define PROGRESS_CREATEIDX_PARTITIONS_DONE		11
+
+/* Phases of CREATE INDEX */
+#define PROGRESS_CREATEIDX_PHASE_WAIT_1			1
+#define PROGRESS_CREATEIDX_PHASE_BUILD			2
+#define PROGRESS_CREATEIDX_PHASE_WAIT_2			3
+#define PROGRESS_CREATEIDX_PHASE_VALIDATE_IDXSCAN		4
+#define PROGRESS_CREATEIDX_PHASE_SORT_IDXSCAN		5
+#define PROGRESS_CREATEIDX_PHASE_HEAPSCAN_VALIDATE		6
+#define PROGRESS_CREATEIDX_PHASE_WAIT_3			7
+
+/*
+ * Subphases of CREATE INDEX, for index_build.
+ */
+#define PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE	1
+/* Additional phases are defined by each AM */
+
+/* Lock holder wait counts */
+#define PROGRESS_WAITFOR_TOTAL					3
+#define PROGRESS_WAITFOR_DONE					4
+#define PROGRESS_WAITFOR_CURRENT_PID			5
+
+/* Block numbers in a generic relation scan */
+#define PROGRESS_SCAN_BLOCKS_TOTAL				6
+#define PROGRESS_SCAN_BLOCKS_DONE				7
+
 #endif
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 88a75fb798e..f931ead1c27 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -934,10 +934,11 @@ typedef enum
 typedef enum ProgressCommandType
 {
 	PROGRESS_COMMAND_INVALID,
-	PROGRESS_COMMAND_VACUUM
+	PROGRESS_COMMAND_VACUUM,
+	PROGRESS_COMMAND_CREATE_INDEX
 } ProgressCommandType;
 
-#define PGSTAT_NUM_PROGRESS_PARAM	10
+#define PGSTAT_NUM_PROGRESS_PARAM	12
 
 /* ----------
  * Shared-memory data structures
diff --git a/src/include/storage/lmgr.h b/src/include/storage/lmgr.h
index 3d705faba5c..4f2872de35f 100644
--- a/src/include/storage/lmgr.h
+++ b/src/include/storage/lmgr.h
@@ -78,8 +78,8 @@ extern void XactLockTableWait(TransactionId xid, Relation rel,
 extern bool ConditionalXactLockTableWait(TransactionId xid);
 
 /* Lock VXIDs, specified by conflicting locktags */
-extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode);
-extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode);
+extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress);
+extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress);
 
 /* Lock an XID for tuple insertion (used to wait for an insertion to finish) */
 extern uint32 SpeculativeInsertionLockAcquire(TransactionId xid);
diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h
index 16b927cb801..e117b391774 100644
--- a/src/include/storage/lock.h
+++ b/src/include/storage/lock.h
@@ -544,7 +544,7 @@ extern bool LockHeldByMe(const LOCKTAG *locktag, LOCKMODE lockmode);
 extern bool LockHasWaiters(const LOCKTAG *locktag,
 			   LOCKMODE lockmode, bool sessionLock);
 extern VirtualTransactionId *GetLockConflicts(const LOCKTAG *locktag,
-				 LOCKMODE lockmode);
+				 LOCKMODE lockmode, int *countp);
 extern void AtPrepare_Locks(void);
 extern void PostPrepare_Locks(TransactionId xid);
 extern int LockCheckConflicts(LockMethod lockMethodTable,
-- 
2.17.1

v5-0002-report-progress-of-hash-indexes.patchtext/x-diff; charset=us-asciiDownload
From 4987e0832e616badda8614944e98bed7f309f017 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Tue, 26 Feb 2019 14:34:49 -0300
Subject: [PATCH v5 2/2] report progress of hash indexes

---
 src/backend/access/hash/hash.c     | 6 +++++-
 src/backend/access/hash/hashsort.c | 6 ++++++
 2 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index fc7db5d6a13..cf7ec655044 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -22,9 +22,11 @@
 #include "access/hash_xlog.h"
 #include "access/relscan.h"
 #include "catalog/index.h"
+#include "commands/progress.h"
 #include "commands/vacuum.h"
 #include "miscadmin.h"
 #include "optimizer/plancat.h"
+#include "pgstat.h"
 #include "utils/builtins.h"
 #include "utils/index_selfuncs.h"
 #include "utils/rel.h"
@@ -160,8 +162,10 @@ hashbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	buildstate.heapRel = heap;
 
 	/* do the heap scan */
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, true,
 								   hashbuildCallback, (void *) &buildstate, NULL);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_TOTAL,
+								 buildstate.indtuples);
 
 	if (buildstate.spool)
 	{
diff --git a/src/backend/access/hash/hashsort.c b/src/backend/access/hash/hashsort.c
index 8c55436b193..00a57470a77 100644
--- a/src/backend/access/hash/hashsort.c
+++ b/src/backend/access/hash/hashsort.c
@@ -26,7 +26,9 @@
 #include "postgres.h"
 
 #include "access/hash.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
+#include "pgstat.h"
 #include "utils/tuplesort.h"
 
 
@@ -116,6 +118,7 @@ void
 _h_indexbuild(HSpool *hspool, Relation heapRel)
 {
 	IndexTuple	itup;
+	long		tups_done = 0;
 #ifdef USE_ASSERT_CHECKING
 	uint32		hashkey = 0;
 #endif
@@ -141,5 +144,8 @@ _h_indexbuild(HSpool *hspool, Relation heapRel)
 #endif
 
 		_hash_doinsert(hspool->index, itup, heapRel);
+
+		pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE,
+									 ++tups_done);
 	}
 }
-- 
2.17.1

#18David Fetter
david@fetter.org
In reply to: Alvaro Herrera (#17)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On Mon, Mar 04, 2019 at 05:46:07PM -0300, Alvaro Herrera wrote:

Hi Rahila,

Thanks for looking.

On 2019-Mar-04, Rahila wrote:

1. Thank you for incorporating review comments.
Can you please rebase the latest 0001-Report-progress-of-
CREATE-INDEX-operations.patch on master? Currently it does not apply on
754b90f657bd54b482524b73726dae4a9165031c

Hmm, rebased to current master. Didn't conflict at all when rebasing,
so it's strange that it didn't apply.

15:56:44.694 | building index (3 of 8): initializing (1/5) | 442478 | 442399 | 0 | 0 | 0 | 0
15:56:44.705 | building index (3 of 8): sorting tuples, spool 1 (3/5) | 442478 | 442399 | 100000000 | 0 | 0 | 0
15:56:44.716 | building index (3 of 8): sorting tuples, spool 1 (3/5) | 442478 | 442399 | 100000000 | 0 | 0 | 0
15:56:44.727 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 79057 | 0 | 0
15:56:44.738 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 217018 | 0 | 0
15:56:44.75 | building index (3 of 8): final btree sort & load (5/5) | 442478 | 442399 | 100000000 | 353804 | 0 | 0

2. In the above report, even though we are reporting progress in terms of
tuples_done for final btree sort & load phase we have not cleared
the blocks_done entry from previous phases. I think this can be confusing as
the blocks_done does not correspond to the tuples_done in the final btree
sort & load phase.

Good point. Done in the attached version, wherein I also added comments
to explain the IndexBuildHeapScan API change. I didn't change the hash
AM implementation here.

Would it be a very large lift to report progress for the rest of the
index types we support?

Best,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

#19Rahila Syed
rahila.syed@2ndquadrant.com
In reply to: Alvaro Herrera (#17)
Re: monitoring CREATE INDEX [CONCURRENTLY]

Hi Alvaro,

On Tue, 5 Mar 2019 at 08:32, Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:

Hi Rahila,

Thanks for looking.

On 2019-Mar-04, Rahila wrote:

1. Thank you for incorporating review comments.
Can you please rebase the latest 0001-Report-progress-of-
CREATE-INDEX-operations.patch on master? Currently it does not apply on
754b90f657bd54b482524b73726dae4a9165031c

Hmm, rebased to current master. Didn't conflict at all when rebasing,
so it's strange that it didn't apply.

Thanks for updating the patch. Sorry, I think it wasn't that the patch
needed rebasing but
I failed to apply it correctly last time. I can apply it now.

+extern char *btbuildphasename(int64 phasenum);

1. I think int64 is too large a datatype for phasenum.
Also int32 is used for phasenum in pg_indexam_progress_phasename().
Can we have it as int8?

2.

if ((previous_blkno == InvalidBlockNumber) ||

+ (scan->rs_cblock != previous_blkno))

+ {

+

pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,

+

scan->rs_cblock);

+ previous_blkno = scan->rs_cblock;

+ }

. In validate_index_heapscan, we dont calculate blocks_done similar to
IndexBuildHeapScan i.e
block_done += scan->rs_cblock - previous_blkno which IMO is more accurate.
Reporting scan->rs_cblock as blocks_done might give slightly inaccurate
results as we are
still processing that block.

3. There is no if(progress) check in validate_index()/
validate_index_heapscan() code. Wont it be a problem if it is called from
other
index methods which dont support reporting progress at the moment?

4. Just to clarify my understanding can you please see below comment

Quoting your following comment in cluster command progress monitor thread
while referring to progress reporting from IndexBuildHeapScan,

"One, err, small issue with that idea is that we need the param numbers
not to conflict for any "progress update providers" that are to be used
simultaneously by any command."

Does that mean that we can't have any other INDEX progress monitoring, use
PROGRESS_SCAN_BLOCKS_TOTAL and PROGRESS_SCAN_BLOCKS_DONE
parameter numbers to report anything but the metrics they report now.

5.

15:56:44.682 | building index (3 of 8): initializing (1/5) |
442478 | 442399 | 0 | 0 | 0 |
0

15:56:44.694 | building index (3 of 8): initializing (1/5) |

442478 | 442399 | 0 | 0 | 0 |
0

15:56:44.705 | building index (3 of 8): sorting tuples, spool 1 (3/5) |

442478 | 442399 | 100000000 | 0 | 0 |
0

15:56:44.716 | building index (3 of 8): sorting tuples, spool 1 (3/5) |

442478 | 442399 | 100000000 | 0 | 0 |
0

I wonder how is the phase 'building index(3 of 8): initializing(1/5)' when
the blocks_done count is increasing. Shouldn't it have
changed to reflect PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN as building
index(3 of 8): table scan(2/5) ?
Although I think it has been rectified in the latest patch as I now get
'table scan' phase in output as I do CREATE INDEX on table with 1000000
records

Thank you,
.--
Rahila Syed
Performance Engineer
2ndQuadrant
http://www.2ndQuadrant.com <http://www.2ndquadrant.com/&gt;
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#20Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Rahila Syed (#19)
Re: monitoring CREATE INDEX [CONCURRENTLY]

Hi Rahila

On 2019-Mar-11, Rahila Syed wrote:

On Tue, 5 Mar 2019 at 08:32, Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:

+extern char *btbuildphasename(int64 phasenum);

1. I think int64 is too large a datatype for phasenum.
Also int32 is used for phasenum in pg_indexam_progress_phasename().
Can we have it as int8?

It does look strange, I agree, and the first code I wrote had it using a
smaller type. However, I later realized that since the value comes
directly from pg_stat_get_progress_info(), which returns int8 values, it
was pointless to only accept a small fraction of the possible values for
no good reason, so I widened it to int64 as you see now.

2.
. In validate_index_heapscan, we dont calculate blocks_done similar to
IndexBuildHeapScan i.e
block_done += scan->rs_cblock - previous_blkno which IMO is more accurate.
Reporting scan->rs_cblock as blocks_done might give slightly inaccurate
results as we are
still processing that block.

Thanks for pointing out that there's an off-by-one bug there (should be
cblock-1). However, IndexBuildHeapScan uses more complicated code
because it needs to cover for two additional things that
validate_index_heapscan doesn't: parallel heapscans and synchronized
seqscans. We could do that, I just saw no point in it.

3. There is no if(progress) check in validate_index()/
validate_index_heapscan() code. Wont it be a problem if it is called from
other index methods which dont support reporting progress at the
moment?

Good question. I'll have a look. Most likely, I'll end up having
things so that building an index using an unsupported index AM reports
progress based on IndexBuildHeapScan / validate_index /
validate_index_heapscan ... which might mean I should remove the
'progress' parameter from there and have them report unconditionally.

4. Just to clarify my understanding can you please see below comment

Quoting your following comment in cluster command progress monitor thread
while referring to progress reporting from IndexBuildHeapScan,

"One, err, small issue with that idea is that we need the param numbers
not to conflict for any "progress update providers" that are to be used
simultaneously by any command."

Does that mean that we can't have any other INDEX progress monitoring, use
PROGRESS_SCAN_BLOCKS_TOTAL and PROGRESS_SCAN_BLOCKS_DONE
parameter numbers to report anything but the metrics they report now.

What I mean is that the literal parameter numbers defined as
PROGRESS_SCAN_BLOCKS_DONE/TOTAL may not be used for other parameters by
commands that call IndexBuildHeapScan, if those other parameters are
used by the same commands simultaneously with IndexBuildHeapScan. So
those parameter numbers become "reserved".

5.

15:56:44.682 | building index (3 of 8): initializing (1/5) |
442478 | 442399 | 0 | 0 | 0 |

I wonder how is the phase 'building index(3 of 8): initializing(1/5)' when
the blocks_done count is increasing. Shouldn't it have
changed to reflect PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN as building
index(3 of 8): table scan(2/5) ?
Although I think it has been rectified in the latest patch as I now get
'table scan' phase in output as I do CREATE INDEX on table with 1000000
records

Yeah, this was a bug that I fixed in v5. (It was a misunderstanding
about how parallel scanning is set up, IIRC). For v5, I tested both
parallel and non-parallel builds, with and without sync seqscans, and
everything seemed to behave correctly.

Thanks for looking! I intend to post a new version later this week.

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#21Robert Haas
robertmhaas@gmail.com
In reply to: Alvaro Herrera (#20)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On Mon, Mar 11, 2019 at 8:41 AM Alvaro Herrera <alvherre@2ndquadrant.com> wrote:

I wonder how is the phase 'building index(3 of 8): initializing(1/5)' when
the blocks_done count is increasing. Shouldn't it have
changed to reflect PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN as building
index(3 of 8): table scan(2/5) ?
Although I think it has been rectified in the latest patch as I now get
'table scan' phase in output as I do CREATE INDEX on table with 1000000
records

Yeah, this was a bug that I fixed in v5. (It was a misunderstanding
about how parallel scanning is set up, IIRC). For v5, I tested both
parallel and non-parallel builds, with and without sync seqscans, and
everything seemed to behave correctly.

Thanks for looking! I intend to post a new version later this week.

I don't think that I much like this (3 of 8) and (2 of 5) stuff. It's
inconsistent with what we've got already and it doesn't add much.
Someone who wants to know which phase it is can look at the underlying
numbers directly instead of going through the view, but most people
probably won't care, and given that the phases may be of dramatically
unequal length, I don't think it's adding much.

Another reason why I think this is a bad idea is that there may be
some operations where we don't transit all the phases in all cases;
the pending patch for CLUSTER progress reporting works like that.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

#22Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Robert Haas (#21)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On 2019-Mar-11, Robert Haas wrote:

I don't think that I much like this (3 of 8) and (2 of 5) stuff. It's
inconsistent with what we've got already and it doesn't add much.
Someone who wants to know which phase it is can look at the underlying
numbers directly instead of going through the view, but most people
probably won't care, and given that the phases may be of dramatically
unequal length, I don't think it's adding much.

Another reason why I think this is a bad idea is that there may be
some operations where we don't transit all the phases in all cases;
the pending patch for CLUSTER progress reporting works like that.

What part of it don't you like? Is it the fact that we have phase
numbers in the phase name? Is it the fact that we count total phases?
Is it that we have two numbers being current (phase + subphase)?

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#23Robert Haas
robertmhaas@gmail.com
In reply to: Alvaro Herrera (#22)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On Mon, Mar 11, 2019 at 3:18 PM Alvaro Herrera <alvherre@2ndquadrant.com> wrote:

On 2019-Mar-11, Robert Haas wrote:

I don't think that I much like this (3 of 8) and (2 of 5) stuff. It's
inconsistent with what we've got already and it doesn't add much.
Someone who wants to know which phase it is can look at the underlying
numbers directly instead of going through the view, but most people
probably won't care, and given that the phases may be of dramatically
unequal length, I don't think it's adding much.

Another reason why I think this is a bad idea is that there may be
some operations where we don't transit all the phases in all cases;
the pending patch for CLUSTER progress reporting works like that.

What part of it don't you like? Is it the fact that we have phase
numbers in the phase name? Is it the fact that we count total phases?
Is it that we have two numbers being current (phase + subphase)?

that you have phase numbers in the phase name

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

#24Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Robert Haas (#23)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On 2019-Mar-11, Robert Haas wrote:

On Mon, Mar 11, 2019 at 3:18 PM Alvaro Herrera <alvherre@2ndquadrant.com> wrote:

On 2019-Mar-11, Robert Haas wrote:

I don't think that I much like this (3 of 8) and (2 of 5) stuff. It's
inconsistent with what we've got already and it doesn't add much.
Someone who wants to know which phase it is can look at the underlying
numbers directly instead of going through the view, but most people
probably won't care, and given that the phases may be of dramatically
unequal length, I don't think it's adding much.

Another reason why I think this is a bad idea is that there may be
some operations where we don't transit all the phases in all cases;
the pending patch for CLUSTER progress reporting works like that.

What part of it don't you like? Is it the fact that we have phase
numbers in the phase name? Is it the fact that we count total phases?
Is it that we have two numbers being current (phase + subphase)?

that you have phase numbers in the phase name

Oh. That's easily removed. Though I have to say that other people said
that they liked it so much that they would have liked to have it in the
original VACUUM one too (5ba2b281-9c84-772a-cf37-17780d782936@lab.ntt.co.jp).

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#25Robert Haas
robertmhaas@gmail.com
In reply to: Alvaro Herrera (#24)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On Mon, Mar 11, 2019 at 3:26 PM Alvaro Herrera <alvherre@2ndquadrant.com> wrote:

Oh. That's easily removed. Though I have to say that other people said
that they liked it so much that they would have liked to have it in the
original VACUUM one too (5ba2b281-9c84-772a-cf37-17780d782936@lab.ntt.co.jp).

Huh. Well, that's another option, but then what do we do if the
number of phases is not a constant?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

#26Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Robert Haas (#25)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On 2019-Mar-11, Robert Haas wrote:

On Mon, Mar 11, 2019 at 3:26 PM Alvaro Herrera <alvherre@2ndquadrant.com> wrote:

Oh. That's easily removed. Though I have to say that other people said
that they liked it so much that they would have liked to have it in the
original VACUUM one too (5ba2b281-9c84-772a-cf37-17780d782936@lab.ntt.co.jp).

Huh. Well, that's another option, but then what do we do if the
number of phases is not a constant?

Well, why do we care? "Some phases might be skipped".

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#27Robert Haas
robertmhaas@gmail.com
In reply to: Alvaro Herrera (#26)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On Mon, Mar 11, 2019 at 3:43 PM Alvaro Herrera <alvherre@2ndquadrant.com> wrote:

Huh. Well, that's another option, but then what do we do if the
number of phases is not a constant?

Well, why do we care? "Some phases might be skipped".

It seems pretty confusing. I mean, in the case of the CLUSTER patch,
you're either going to seq-scan the table or index-scan the table.
Those are (at last check) reported using different phase numbers, but
they are mutually exclusive. Generally, if you are going to do either
foo -> bar -> baz -> quux or foo -> bletch -> quux, how many phases
are there total? 5? 4?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

#28Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Robert Haas (#27)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On 2019-Mar-11, Robert Haas wrote:

On Mon, Mar 11, 2019 at 3:43 PM Alvaro Herrera <alvherre@2ndquadrant.com> wrote:

Huh. Well, that's another option, but then what do we do if the
number of phases is not a constant?

Well, why do we care? "Some phases might be skipped".

It seems pretty confusing. I mean, in the case of the CLUSTER patch,
you're either going to seq-scan the table or index-scan the table.
Those are (at last check) reported using different phase numbers, but
they are mutually exclusive. Generally, if you are going to do either
foo -> bar -> baz -> quux or foo -> bletch -> quux, how many phases
are there total? 5? 4?

Hmm. Your argument is not entirely devoid of merit, but I'm not 100%
convinced either.

So, in CLUSTER, the phases in the middle section are exclusive of one
another. You do bar and baz, or you do bletch. But you never do bar
and bletch, or bletch and baz, or bar on isolation or baz on isolation.
Furthermore, the counting of phases depends on internal system state
(optimizer output), not on the user's input.

In CREATE INDEX, it's not exactly the same. You either have a
complicated 8-phase system (CREATE INDEX CONCURRENTLY) or just a
two-phase system. The phases for the second case are a strict subset of
the cases in the first case. Whether to use one or the other phase
sequence is entirely up to the user.

On the other hand, the subphase numbers vary per AM (but I expect
they're always the same for any one AM.)

To me, it's not a big deal, but if we don't put the number in the phase
name, then we force users to keep the reference documentation open every
time they create an index.

I'm not wed to anything in this area, TBH. My first patch had no phase
numbers and I added them because of reviewer feedback. I do agree we
should be consistent ... but on the other hand, each case is a bit
different: consider VACUUM, which goes back to phase 2 after doing phase
3 for a while. You don't have that behavior for either CLUSTER or
CREATE INDEX.

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#29Rahila Syed
rahila.syed@2ndquadrant.com
In reply to: Alvaro Herrera (#1)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On Mon, 25 Mar 2019 at 22:23, Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:

Here's v6 of this patch. I have rebased on top of today's CLUSTER
monitoring, as well as on table AM commits. The latter caused a bit of
trouble, as now the number of blocks processed by a scan is not as easy
to get as before; I added a new entry point heapscan_get_blocks_done on
heapam.c to help with that. (I suppose this will need some fixups later
on.)

I removed the "M of N" phase labels that Robert didn't like; those were
suggested by Rahila and upvoted by Amit L. I'm of two minds about
those. If you care about those and want them back, please speak up.

I see value in reporting those numbers because it gives user insight into

where
we are at in the whole process without having to refer to documentation or
code.
Besides here also we are reporting facts as we follow for other metrics.

I agree that it will be most effective if the phases are carried out in
succession
which is not the case every time and its relevance varies for each command
as mentioned upthread by Alvaro and Robert. But I feel as long as we have in
the documentation that some phases overlap, some are mutually exclusive
hence
may be skipped etc. reporting `phase number versus total phases` does
provide
valuable information.
We are able to give user a whole picture in addition to reporting progress
within phases.

Thank you,
--
Rahila Syed
Performance Engineer
2ndQuadrant
http://www.2ndQuadrant.com <http://www.2ndquadrant.com/&gt;
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#30Rahila Syed
rahila.syed@2ndquadrant.com
In reply to: Rahila Syed (#29)
Re: monitoring CREATE INDEX [CONCURRENTLY]

Hi Alvaro,

Please see few comments below:

1. Makecheck fails currently as view definition of expected rules.out does
not reflect latest changes in progress metrics numbering.

2. +      <entry>
+       When creating an index on a partitioned, this column is set to the
+       total number of partitions on which the index is to be created.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>partitions_done</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+       When creating an index on a partitioned, this column is set to the

I think there is a typo here 's/partitioned/partitioned table/'

3.
+       if (hscan->rs_base.rs_parallel != NULL)
+       {
+               ParallelBlockTableScanDesc bpscan;
+
+               bpscan = (ParallelBlockTableScanDesc)
hscan->rs_base.rs_parallel;
+               startblock = bpscan->phs_startblock;
+       }
+       else
+               startblock = hscan->rs_startblock;
+
+       /*
+        * Might have wrapped around the end of the relation, if startblock
was
+        * not zero.
+        */
+       if (hscan->rs_cblock > startblock)
+               blocks_done = hscan->rs_cblock - startblock;
+       else
+               blocks_done = hscan->rs_nblocks - startblock +
+                       hscan->rs_cblock;
+
+       return blocks_done;

I think parallel scan equivalent bpscan->phs_nblocks along with
hscan->rs_nblocks should be used similar to startblock computation above.

Thank you,
Rahila Syed

On Fri, 29 Mar 2019 at 23:46, Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:

On 2019-Mar-29, Alvaro Herrera wrote:

So, CLUSTER and ALTER TABLE rewrites only do non-concurrent index
builds; and REINDEX can reuse pretty much the same wait-for metrics
columns as CIC. So I think it's okay if I move only the metrics that
conflict for index_build.

The attached version does it that way. I had to enlarge the param set a
bit more. (I suspect those extra columns will be useful to reindex.)
Also, rebased for recent conflicting changes.

I think we should consider a new column of an array type, where we could
put things like the list of PIDs to be waited for, the list of OIDs of
index to rebuild, or the list of partitions to build the index on.

--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Rahila Syed
Performance Engineer
2ndQuadrant
http://www.2ndQuadrant.com <http://www.2ndquadrant.com/&gt;
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#31Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Alvaro Herrera (#1)
3 attachment(s)
Re: monitoring CREATE INDEX [CONCURRENTLY]

Here's v6 of this patch. I have rebased on top of today's CLUSTER
monitoring, as well as on table AM commits. The latter caused a bit of
trouble, as now the number of blocks processed by a scan is not as easy
to get as before; I added a new entry point heapscan_get_blocks_done on
heapam.c to help with that. (I suppose this will need some fixups later
on.)

I removed the "M of N" phase labels that Robert didn't like; those were
suggested by Rahila and upvoted by Amit L. I'm of two minds about
those. If you care about those and want them back, please speak up.

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attachments:

v6-0001-implement-heapscan_get_blocks_done.patchtext/x-diff; charset=us-asciiDownload
From ccf8a84f785c96ac7ae01e17b4fb7999183fed9e Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Mon, 25 Mar 2019 13:29:34 -0300
Subject: [PATCH v6 1/3] implement heapscan_get_blocks_done

---
 src/backend/access/heap/heapam.c | 34 ++++++++++++++++++++++++++++++++
 src/include/access/heapam.h      |  1 +
 2 files changed, 35 insertions(+)

diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index fa747be73ab..a7f6c09dcd0 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1053,6 +1053,40 @@ heapgettup_pagemode(HeapScanDesc scan,
 	}
 }
 
+/*
+ * Return the number of blocks that have been read by this scan since
+ * starting.  This is not 100% accurate, because in a parallel scan the
+ * counter can be moving concurrently.
+ */
+BlockNumber
+heapscan_get_blocks_done(TableScanDesc sscan)
+{
+	HeapScanDesc	hscan = (HeapScanDesc) sscan;
+	BlockNumber		startblock;
+	BlockNumber		blocks_done;
+
+	if (hscan->rs_base.rs_parallel != NULL)
+	{
+		ParallelBlockTableScanDesc bpscan;
+
+		bpscan = (ParallelBlockTableScanDesc) hscan->rs_base.rs_parallel;
+		startblock = bpscan->phs_startblock;
+	}
+	else
+		startblock = hscan->rs_startblock;
+
+	/*
+	 * Might have wrapped around the end of the relation, if startblock was
+	 * not zero.
+	 */
+	if (hscan->rs_cblock > startblock)
+		blocks_done = hscan->rs_cblock - startblock;
+	else
+		blocks_done = hscan->rs_nblocks - startblock +
+			hscan->rs_cblock;
+
+	return blocks_done;
+}
 
 #if defined(DISABLE_COMPLEX_MACRO)
 /*
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index e72f9787517..d4f537b3791 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -118,6 +118,7 @@ extern TableScanDesc heap_beginscan(Relation relation, Snapshot snapshot,
 extern void heap_setscanlimits(TableScanDesc scan, BlockNumber startBlk,
 				   BlockNumber endBlk);
 extern void heapgetpage(TableScanDesc scan, BlockNumber page);
+extern BlockNumber heapscan_get_blocks_done(TableScanDesc sscan);
 extern void heap_rescan(TableScanDesc scan, ScanKey key, bool set_params,
 			bool allow_strat, bool allow_sync, bool allow_pagemode);
 extern void heap_rescan_set_params(TableScanDesc scan, ScanKey key,
-- 
2.17.1

v6-0002-Report-progress-of-CREATE-INDEX-operations.patchtext/x-diff; charset=us-asciiDownload
From 7adb314a8be321084c7f7255cefc53eaa8c2ab0e Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Wed, 2 Jan 2019 16:14:39 -0300
Subject: [PATCH v6 2/3] Report progress of CREATE INDEX operations

---
 contrib/amcheck/verify_nbtree.c       |   2 +-
 contrib/bloom/blinsert.c              |   2 +-
 contrib/bloom/blutils.c               |   1 +
 doc/src/sgml/indexam.sgml             |  13 ++
 doc/src/sgml/monitoring.sgml          | 224 +++++++++++++++++++++++++-
 src/backend/access/brin/brin.c        |   5 +-
 src/backend/access/gin/gininsert.c    |   2 +-
 src/backend/access/gin/ginutil.c      |   1 +
 src/backend/access/gist/gist.c        |   1 +
 src/backend/access/gist/gistbuild.c   |   2 +-
 src/backend/access/hash/hash.c        |   3 +-
 src/backend/access/nbtree/nbtree.c    |   9 ++
 src/backend/access/nbtree/nbtsort.c   |  58 ++++++-
 src/backend/access/nbtree/nbtutils.c  |  24 +++
 src/backend/access/spgist/spginsert.c |   2 +-
 src/backend/access/spgist/spgutils.c  |   1 +
 src/backend/catalog/index.c           | 118 +++++++++++++-
 src/backend/catalog/system_views.sql  |  27 ++++
 src/backend/commands/indexcmds.c      |  52 +++++-
 src/backend/storage/ipc/standby.c     |   2 +-
 src/backend/storage/lmgr/lmgr.c       |  46 +++++-
 src/backend/storage/lmgr/lock.c       |   7 +-
 src/backend/utils/adt/amutils.c       |  23 +++
 src/backend/utils/adt/pgstatfuncs.c   |   2 +
 src/include/access/amapi.h            |   4 +
 src/include/access/genam.h            |   1 +
 src/include/access/nbtree.h           |  11 ++
 src/include/catalog/index.h           |   2 +
 src/include/catalog/pg_proc.dat       |  10 +-
 src/include/commands/progress.h       |  35 ++++
 src/include/pgstat.h                  |   5 +-
 src/include/storage/lmgr.h            |   4 +-
 src/include/storage/lock.h            |   2 +-
 src/test/regress/expected/rules.out   |  30 +++-
 34 files changed, 693 insertions(+), 38 deletions(-)

diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index 6ae3bca9536..ebad178476f 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -565,7 +565,7 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
 			 RelationGetRelationName(state->rel),
 			 RelationGetRelationName(state->heaprel));
 
-		IndexBuildHeapScan(state->heaprel, state->rel, indexinfo, true,
+		IndexBuildHeapScan(state->heaprel, state->rel, indexinfo, true, false,
 						   bt_tuple_present_callback, (void *) state, scan);
 
 		ereport(DEBUG1,
diff --git a/contrib/bloom/blinsert.c b/contrib/bloom/blinsert.c
index e43fbe0005f..947ee74881f 100644
--- a/contrib/bloom/blinsert.c
+++ b/contrib/bloom/blinsert.c
@@ -141,7 +141,7 @@ blbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	initCachedPage(&buildstate);
 
 	/* Do the heap scan */
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false,
 								   bloomBuildCallback, (void *) &buildstate,
 								   NULL);
 
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index d078dfbd469..ee3bd562748 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -132,6 +132,7 @@ blhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = blcostestimate;
 	amroutine->amoptions = bloptions;
 	amroutine->amproperty = NULL;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = blvalidate;
 	amroutine->ambeginscan = blbeginscan;
 	amroutine->amrescan = blrescan;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index 05102724ead..fa4a3d0d131 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -127,6 +127,7 @@ typedef struct IndexAmRoutine
     amcostestimate_function amcostestimate;
     amoptions_function amoptions;
     amproperty_function amproperty;     /* can be NULL */
+    ambuildphasename_function ambuildphasename;   /* can be NULL */
     amvalidate_function amvalidate;
     ambeginscan_function ambeginscan;
     amrescan_function amrescan;
@@ -468,6 +469,18 @@ amproperty (Oid index_oid, int attno,
 
   <para>
 <programlisting>
+char *
+ambuildphasename (int64 phasenum);
+</programlisting>
+   Return the textual name of the given build phase number.
+   The phase numbers are those reported during an index build via the
+   <function>pgstat_progress_update_param</function> interface.
+   The phase names are then exposed in the
+   <structname>pg_stat_progress_create_index</structname> view.
+  </para>
+
+  <para>
+<programlisting>
 bool
 amvalidate (Oid opclassoid);
 </programlisting>
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index f1df14bdea8..8f5bbbb79e5 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -336,6 +336,14 @@ postgres   27093  0.0  0.0  30096  2752 ?        Ss   11:34   0:00 postgres: ser
       </entry>
      </row>
 
+     <row>
+      <entry><structname>pg_stat_progress_create_index</structname><indexterm><primary>pg_stat_progress_create_index</primary></indexterm></entry>
+      <entry>One row for each backend running <command>CREATE INDEX</command>, showing
+      current progress.
+      See <xref linkend='create-index-progress-reporting'/>.
+     </entry>
+     </row>
+
      <row>
       <entry><structname>pg_stat_progress_vacuum</structname><indexterm><primary>pg_stat_progress_vacuum</primary></indexterm></entry>
       <entry>One row for each backend (including autovacuum worker processes) running
@@ -3403,10 +3411,224 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
   <para>
    <productname>PostgreSQL</productname> has the ability to report the progress of
    certain commands during command execution.  Currently, the only commands
-   which support progress reporting are <command>VACUUM</command> and
+   which support progress reporting are <command>CREATE INDEX</command>,
+   <command>VACUUM</command> and
    <command>CLUSTER</command>. This may be expanded in the future.
   </para>
 
+ <sect2 id="create-index-progress-reporting">
+  <title>CREATE INDEX Progress Reporting</title>
+
+  <para>
+   Whenever <command>CREATE INDEX</command> is running, the
+   <structname>pg_stat_progress_create_index</structname> view will contain
+   one row for each backend that is currently creating indexes.  The tables
+   below describe the information that will be reported and provide information
+   about how to interpret it.
+  </para>
+
+  <table id="pg-stat-progress-create-index-view" xreflabel="pg_stat_progress_create_index">
+   <title><structname>pg_stat_progress_create_index</structname> View</title>
+   <tgroup cols="3">
+    <thead>
+     <row>
+      <entry>Column</entry>
+      <entry>Type</entry>
+      <entry>Description</entry>
+     </row>
+    </thead>
+
+    <tbody>
+     <row>
+      <entry><structfield>pid</structfield></entry>
+      <entry><type>integer</type></entry>
+      <entry>Process ID of backend.</entry>
+     </row>
+     <row>
+      <entry><structfield>datid</structfield></entry>
+      <entry><type>oid</type></entry>
+      <entry>OID of the database to which this backend is connected.</entry>
+     </row>
+     <row>
+      <entry><structfield>datname</structfield></entry>
+      <entry><type>name</type></entry>
+      <entry>Name of the database to which this backend is connected.</entry>
+     </row>
+     <row>
+      <entry><structfield>relid</structfield></entry>
+      <entry><type>oid</type></entry>
+      <entry>OID of the table on which the index is being created.</entry>
+     </row>
+     <row>
+      <entry><structfield>phase</structfield></entry>
+      <entry><type>text</type></entry>
+      <entry>
+        Current processing phase of index creation.  See <xref linkend='create-index-phases'/>.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>lockers_total</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Total number of lockers to wait for, when applicable.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>lockers_done</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Number of lockers already waited for.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>current_locked_pid</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Process ID of the locker currently being waited for.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>blocks_total</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Total number of blocks to be processed in the current phase.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>blocks_done</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Number of blocks already processed in the current phase.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>tuples_total</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Total number of tuples to be processed in the current phase.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>tuples_done</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Number of tuples already processed in the current phase.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>partitions_total</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+       When creating an index on a partitioned, this column is set to the
+       total number of partitions on which the index is to be created.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>partitions_done</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+       When creating an index on a partitioned, this column is set to the
+       number of partitions on which the index has been completed.
+      </entry>
+     </row>
+    </tbody>
+   </tgroup>
+  </table>
+
+  <table id="create-index-phases">
+   <title>CREATE INDEX phases</title>
+   <tgroup cols="2">
+    <thead>
+     <row>
+      <entry>Phase</entry>
+      <entry>Description</entry>
+     </row>
+    </thead>
+    <tbody>
+     <row>
+      <entry><literal>initializing</literal></entry>
+      <entry>
+       <command>CREATE INDEX</command> is preparing to create the index.  This
+       phase is expected to be very brief.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>waiting for old snapshots</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions
+       that can potentially see the table to release their snapshots.
+       This phase is skipped when not in concurrent mode.
+       Columns <structname>lockers_total</structname>, <structname>lockers_done</structname>
+       and <structname>current_locker_pid</structname> contain the progress 
+       information for this phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>building index</literal></entry>
+      <entry>
+       The index is being built by the access method-specific code.  In this phase,
+       access methods that support progress reporting fill in their own progress data,
+       and the subphase is indicated in this column.  Typically,
+       <structname>blocks_total</structname> and <structname>blocks_done</structname>
+       will contain progress data, as well as potentially
+       <structname>tuples_total</structname> and <structname>tuples_done</structname>.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>waiting for writer snapshots</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions
+       that can potentially write into the table to release their snapshots.
+       This phase is skipped when not in concurrent mode.
+       Columns <structname>lockers_total</structname>, <structname>lockers_done</structname>
+       and <structname>current_locker_pid</structname> contain the progress 
+       information for this phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>validating index scan</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is scanning the index searching
+       for tuples that need to be validated.
+       This phase is skipped when not in concurrent mode.
+       Columns <structname>blocks_total</structname> (set to the total size of the index)
+       and <structname>blocks_done</structname> contain the progress information for this phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>sorting index scan results</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is sorting the output of the
+       previous phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>validating index heapscan</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is scanning the table
+       to validate the index tuples collected in the previous two phases.
+       This phase is skipped when not in concurrent mode.
+       Columns <structname>blocks_total</structname> (set to the total size of the table)
+       and <structname>blocks_done</structname> contain the progress information for this phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>waiting for reader snapshots</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions
+       that can potentially see the table to release their snapshots.  This
+       phase is skipped when not in concurrent mode.
+       Columns <structname>lockers_total</structname>, <structname>lockers_done</structname>
+       and <structname>current_locker_pid</structname> contain the progress 
+       information for this phase.
+      </entry>
+     </row>
+    </tbody>
+   </tgroup>
+  </table>
+
+ </sect2>
+
  <sect2 id="vacuum-progress-reporting">
   <title>VACUUM Progress Reporting</title>
 
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 8f008dd0080..5d1aff34080 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -111,6 +111,7 @@ brinhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = brincostestimate;
 	amroutine->amoptions = brinoptions;
 	amroutine->amproperty = NULL;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = brinvalidate;
 	amroutine->ambeginscan = brinbeginscan;
 	amroutine->amrescan = brinrescan;
@@ -718,7 +719,7 @@ brinbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	 * Now scan the relation.  No syncscan allowed here because we want the
 	 * heap blocks in physical order.
 	 */
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, false,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, false, false,
 								   brinbuildCallback, (void *) state, NULL);
 
 	/* process the final batch */
@@ -1234,7 +1235,7 @@ summarize_range(IndexInfo *indexInfo, BrinBuildState *state, Relation heapRel,
 	 * by transactions that are still in progress, among other corner cases.
 	 */
 	state->bs_currRangeStart = heapBlk;
-	IndexBuildHeapRangeScan(heapRel, state->bs_irel, indexInfo, false, true,
+	IndexBuildHeapRangeScan(heapRel, state->bs_irel, indexInfo, false, true, false,
 							heapBlk, scanNumBlks,
 							brinbuildCallback, (void *) state, NULL);
 
diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c
index 524ac5be8b5..838de4c1ec3 100644
--- a/src/backend/access/gin/gininsert.c
+++ b/src/backend/access/gin/gininsert.c
@@ -394,7 +394,7 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	 * Do the heap scan.  We disallow sync scan here because dataPlaceToPage
 	 * prefers to receive tuples in TID order.
 	 */
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, false,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, false, false,
 								   ginBuildCallback, (void *) &buildstate, NULL);
 
 	/* dump remaining entries to the index */
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index afc20232ace..d2360eeafb0 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -64,6 +64,7 @@ ginhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = gincostestimate;
 	amroutine->amoptions = ginoptions;
 	amroutine->amproperty = NULL;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = ginvalidate;
 	amroutine->ambeginscan = ginbeginscan;
 	amroutine->amrescan = ginrescan;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index a746e911f37..5c08b611c8d 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = gistcostestimate;
 	amroutine->amoptions = gistoptions;
 	amroutine->amproperty = gistproperty;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = gistvalidate;
 	amroutine->ambeginscan = gistbeginscan;
 	amroutine->amrescan = gistrescan;
diff --git a/src/backend/access/gist/gistbuild.c b/src/backend/access/gist/gistbuild.c
index bd142a3560d..015f874cc93 100644
--- a/src/backend/access/gist/gistbuild.c
+++ b/src/backend/access/gist/gistbuild.c
@@ -204,7 +204,7 @@ gistbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	/*
 	 * Do the heap scan.
 	 */
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false,
 								   gistBuildCallback, (void *) &buildstate, NULL);
 
 	/*
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index f1f01a0956d..fc7db5d6a13 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -82,6 +82,7 @@ hashhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = hashcostestimate;
 	amroutine->amoptions = hashoptions;
 	amroutine->amproperty = NULL;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = hashvalidate;
 	amroutine->ambeginscan = hashbeginscan;
 	amroutine->amrescan = hashrescan;
@@ -159,7 +160,7 @@ hashbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	buildstate.heapRel = heap;
 
 	/* do the heap scan */
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false,
 								   hashbuildCallback, (void *) &buildstate, NULL);
 
 	if (buildstate.spool)
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index ac6f1eb3423..7370379c6a1 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -22,6 +22,7 @@
 #include "access/nbtxlog.h"
 #include "access/relscan.h"
 #include "access/xlog.h"
+#include "commands/progress.h"
 #include "commands/vacuum.h"
 #include "miscadmin.h"
 #include "nodes/execnodes.h"
@@ -133,6 +134,7 @@ bthandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = btcostestimate;
 	amroutine->amoptions = btoptions;
 	amroutine->amproperty = btproperty;
+	amroutine->ambuildphasename = btbuildphasename;
 	amroutine->amvalidate = btvalidate;
 	amroutine->ambeginscan = btbeginscan;
 	amroutine->amrescan = btrescan;
@@ -1021,6 +1023,10 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
 		if (needLock)
 			UnlockRelationForExtension(rel, ExclusiveLock);
 
+		if (info->report_progress)
+			pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL,
+										 num_pages);
+
 		/* Quit if we've scanned the whole relation */
 		if (blkno >= num_pages)
 			break;
@@ -1028,6 +1034,9 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
 		for (; blkno < num_pages; blkno++)
 		{
 			btvacuumpage(&vstate, blkno, blkno);
+			if (info->report_progress)
+				pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
+											 blkno);
 		}
 	}
 
diff --git a/src/backend/access/nbtree/nbtsort.c b/src/backend/access/nbtree/nbtsort.c
index 2762a2d5485..a47e0de437c 100644
--- a/src/backend/access/nbtree/nbtsort.c
+++ b/src/backend/access/nbtree/nbtsort.c
@@ -66,6 +66,7 @@
 #include "access/xlog.h"
 #include "access/xloginsert.h"
 #include "catalog/index.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
 #include "pgstat.h"
 #include "storage/smgr.h"
@@ -298,7 +299,8 @@ static double _bt_parallel_heapscan(BTBuildState *buildstate,
 static void _bt_leader_participate_as_worker(BTBuildState *buildstate);
 static void _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2,
 						   BTShared *btshared, Sharedsort *sharedsort,
-						   Sharedsort *sharedsort2, int sortmem);
+						   Sharedsort *sharedsort2, int sortmem,
+						   bool progress);
 
 
 /*
@@ -394,6 +396,10 @@ _bt_spools_heapscan(Relation heap, Relation index, BTBuildState *buildstate,
 	/* Save as primary spool */
 	buildstate->spool = btspool;
 
+	/* Report heap scan phase started */
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+								 PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN);
+
 	/* Attempt to launch parallel worker scan when required */
 	if (indexInfo->ii_ParallelWorkers > 0)
 		_bt_begin_parallel(buildstate, indexInfo->ii_Concurrent,
@@ -480,13 +486,31 @@ _bt_spools_heapscan(Relation heap, Relation index, BTBuildState *buildstate,
 
 	/* Fill spool using either serial or parallel heap scan */
 	if (!buildstate->btleader)
-		reltuples = IndexBuildHeapScan(heap, index, indexInfo, true,
+		reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, true,
 									   _bt_build_callback, (void *) buildstate,
 									   NULL);
 	else
 		reltuples = _bt_parallel_heapscan(buildstate,
 										  &indexInfo->ii_BrokenHotChain);
 
+	/*
+	 * Set the progress target for the next phase.  Reset the block number
+	 * values set by IndexBuildHeapScan
+	 */
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_TUPLES_TOTAL,
+			PROGRESS_SCAN_BLOCKS_TOTAL,
+			PROGRESS_SCAN_BLOCKS_DONE
+		};
+		const int64 val[] = {
+			buildstate->indtuples,
+			0, 0
+		};
+
+		pgstat_progress_update_multi_param(3, index, val);
+	}
+
 	/* okay, all heap tuples are spooled */
 	if (buildstate->spool2 && !buildstate->havedead)
 	{
@@ -535,9 +559,15 @@ _bt_leafbuild(BTSpool *btspool, BTSpool *btspool2)
 	}
 #endif							/* BTREE_BUILD_STATS */
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+								 PROGRESS_BTREE_PHASE_PERFORMSORT_1);
 	tuplesort_performsort(btspool->sortstate);
 	if (btspool2)
+	{
+		pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+									 PROGRESS_BTREE_PHASE_PERFORMSORT_2);
 		tuplesort_performsort(btspool2->sortstate);
+	}
 
 	wstate.heap = btspool->heap;
 	wstate.index = btspool->index;
@@ -554,6 +584,8 @@ _bt_leafbuild(BTSpool *btspool, BTSpool *btspool2)
 	wstate.btws_pages_written = 0;
 	wstate.btws_zeropage = NULL;	/* until needed */
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+								 PROGRESS_BTREE_PHASE_LEAF_LOAD);
 	_bt_load(&wstate, btspool, btspool2);
 }
 
@@ -1098,6 +1130,7 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
 	int			i,
 				keysz = IndexRelationGetNumberOfKeyAttributes(wstate->index);
 	SortSupport sortKeys;
+	long		tuples_done = 0L;
 
 	if (merge)
 	{
@@ -1202,6 +1235,10 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
 				_bt_buildadd(wstate, state, itup2);
 				itup2 = tuplesort_getindextuple(btspool2->sortstate, true);
 			}
+
+			/* Report progress */
+			pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE,
+										 ++tuples_done);
 		}
 		pfree(sortKeys);
 	}
@@ -1216,6 +1253,10 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
 				state = _bt_pagestate(wstate, 0);
 
 			_bt_buildadd(wstate, state, itup);
+
+			/* Report progress */
+			pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE,
+										 ++tuples_done);
 		}
 	}
 
@@ -1352,6 +1393,11 @@ _bt_begin_parallel(BTBuildState *buildstate, bool isconcurrent, int request)
 								  ParallelTableScanFromBTShared(btshared),
 								  snapshot);
 
+	/* Report total number of blocks to scan */
+	pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL,
+								 ((ParallelBlockTableScanDescData *)
+								 ParallelTableScanFromBTShared(btshared))->phs_nblocks);
+
 	/*
 	 * Store shared tuplesort-private state, for which we reserved space.
 	 * Then, initialize opaque state using tuplesort routine.
@@ -1528,7 +1574,7 @@ _bt_leader_participate_as_worker(BTBuildState *buildstate)
 	/* Perform work common to all participants */
 	_bt_parallel_scan_and_sort(leaderworker, leaderworker2, btleader->btshared,
 							   btleader->sharedsort, btleader->sharedsort2,
-							   sortmem);
+							   sortmem, true);
 
 #ifdef BTREE_BUILD_STATS
 	if (log_btree_build_stats)
@@ -1619,7 +1665,7 @@ _bt_parallel_build_main(dsm_segment *seg, shm_toc *toc)
 	/* Perform sorting of spool, and possibly a spool2 */
 	sortmem = maintenance_work_mem / btshared->scantuplesortstates;
 	_bt_parallel_scan_and_sort(btspool, btspool2, btshared, sharedsort,
-							   sharedsort2, sortmem);
+							   sharedsort2, sortmem, false);
 
 #ifdef BTREE_BUILD_STATS
 	if (log_btree_build_stats)
@@ -1648,7 +1694,7 @@ _bt_parallel_build_main(dsm_segment *seg, shm_toc *toc)
 static void
 _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2,
 						   BTShared *btshared, Sharedsort *sharedsort,
-						   Sharedsort *sharedsort2, int sortmem)
+						   Sharedsort *sharedsort2, int sortmem, bool progress)
 {
 	SortCoordinate coordinate;
 	BTBuildState buildstate;
@@ -1708,7 +1754,7 @@ _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2,
 	scan = table_beginscan_parallel(btspool->heap,
 									ParallelTableScanFromBTShared(btshared));
 	reltuples = IndexBuildHeapScan(btspool->heap, btspool->index, indexInfo,
-								   true, _bt_build_callback,
+								   true, progress, _bt_build_callback,
 								   (void *) &buildstate, scan);
 
 	/*
diff --git a/src/backend/access/nbtree/nbtutils.c b/src/backend/access/nbtree/nbtutils.c
index 92b8b5f134d..9e8a7bd3d9a 100644
--- a/src/backend/access/nbtree/nbtutils.c
+++ b/src/backend/access/nbtree/nbtutils.c
@@ -20,6 +20,7 @@
 #include "access/nbtree.h"
 #include "access/reloptions.h"
 #include "access/relscan.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
 #include "utils/array.h"
 #include "utils/datum.h"
@@ -2048,6 +2049,29 @@ btproperty(Oid index_oid, int attno,
 	}
 }
 
+/*
+ *	btbuildphasename() -- Return name of index build phase.
+ */
+char *
+btbuildphasename(int64 phasenum)
+{
+	switch (phasenum)
+	{
+		case PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE:
+			return "initializing";
+		case PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN:
+			return "table scan";
+		case PROGRESS_BTREE_PHASE_PERFORMSORT_1:
+			return "sorting tuples, spool 1";
+		case PROGRESS_BTREE_PHASE_PERFORMSORT_2:
+			return "sorting tuples, spool 2";
+		case PROGRESS_BTREE_PHASE_LEAF_LOAD:
+			return "btree tuple loading";
+		default:
+			return NULL;
+	}
+}
+
 /*
  *	_bt_truncate() -- create tuple without unneeded suffix attributes.
  *
diff --git a/src/backend/access/spgist/spginsert.c b/src/backend/access/spgist/spginsert.c
index f428a151385..1bc671c7238 100644
--- a/src/backend/access/spgist/spginsert.c
+++ b/src/backend/access/spgist/spginsert.c
@@ -142,7 +142,7 @@ spgbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 											  "SP-GiST build temporary context",
 											  ALLOCSET_DEFAULT_SIZES);
 
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false,
 								   spgistBuildCallback, (void *) &buildstate,
 								   NULL);
 
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 8e63c1fad25..45472db147b 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -67,6 +67,7 @@ spghandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = spgcostestimate;
 	amroutine->amoptions = spgoptions;
 	amroutine->amproperty = spgproperty;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = spgvalidate;
 	amroutine->ambeginscan = spgbeginscan;
 	amroutine->amrescan = spgrescan;
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index d2e284f6de6..7778341bf51 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -50,9 +50,9 @@
 #include "catalog/pg_trigger.h"
 #include "catalog/pg_type.h"
 #include "catalog/storage.h"
-#include "commands/tablecmds.h"
 #include "commands/event_trigger.h"
 #include "commands/progress.h"
+#include "commands/tablecmds.h"
 #include "commands/trigger.h"
 #include "executor/executor.h"
 #include "miscadmin.h"
@@ -1596,7 +1596,7 @@ index_drop(Oid indexId, bool concurrent)
 		 * to acquire an exclusive lock on our table.  The lock code will
 		 * detect deadlock and error out properly.
 		 */
-		WaitForLockers(heaplocktag, AccessExclusiveLock);
+		WaitForLockers(heaplocktag, AccessExclusiveLock, true);
 
 		/*
 		 * No more predicate locks will be acquired on this index, and we're
@@ -1640,7 +1640,7 @@ index_drop(Oid indexId, bool concurrent)
 		 * Wait till every transaction that saw the old index state has
 		 * finished.
 		 */
-		WaitForLockers(heaplocktag, AccessExclusiveLock);
+		WaitForLockers(heaplocktag, AccessExclusiveLock, true);
 
 		/*
 		 * Re-open relations to allow us to complete our actions.
@@ -2290,6 +2290,25 @@ index_build(Relation heapRelation,
 						   save_sec_context | SECURITY_RESTRICTED_OPERATION);
 	save_nestlevel = NewGUCNestLevel();
 
+	/* Set up initial progress report status */
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_PHASE,
+			PROGRESS_CREATEIDX_SUBPHASE,
+			PROGRESS_CREATEIDX_TUPLES_DONE,
+			PROGRESS_CREATEIDX_TUPLES_TOTAL,
+			PROGRESS_SCAN_BLOCKS_DONE,
+			PROGRESS_SCAN_BLOCKS_TOTAL
+		};
+		const int64	val[] = {
+			PROGRESS_CREATEIDX_PHASE_BUILD,
+			PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE,
+			0, 0, 0, 0
+		};
+
+		pgstat_progress_update_multi_param(6, index, val);
+	}
+
 	/*
 	 * Call the access method's build procedure
 	 */
@@ -2416,6 +2435,12 @@ index_build(Relation heapRelation,
  * do so here because the AM might reject some of the tuples for its own
  * reasons, such as being unable to store NULLs.
  *
+ * If 'progress', we update the PROGRESS_SCAN_BLOCKS_DONE counter as we go
+ * along.  Also, if that flag is true and a scan descriptor is not passed (ie.
+ * when not doing a parallel scan), the PROGRESS_SCAN_BLOCKS_TOTAL counter is
+ * updated at the beginning.  For parallel scans, caller is expected to have
+ * set the total number of blocks prior to calling this function.  
+ *
  * A side effect is to set indexInfo->ii_BrokenHotChain to true if we detect
  * any potentially broken HOT chains.  Currently, we set this if there are
  * any RECENTLY_DEAD or DELETE_IN_PROGRESS entries in a HOT chain, without
@@ -2427,13 +2452,14 @@ IndexBuildHeapScan(Relation heapRelation,
 				   Relation indexRelation,
 				   IndexInfo *indexInfo,
 				   bool allow_sync,
+				   bool progress,
 				   IndexBuildCallback callback,
 				   void *callback_state,
 				   TableScanDesc scan)
 {
 	return IndexBuildHeapRangeScan(heapRelation, indexRelation,
 								   indexInfo, allow_sync,
-								   false,
+								   false, progress,
 								   0, InvalidBlockNumber,
 								   callback, callback_state, scan);
 }
@@ -2444,6 +2470,13 @@ IndexBuildHeapScan(Relation heapRelation,
  * passing InvalidBlockNumber as numblocks.  Note that restricting the range
  * to scan cannot be done when requesting syncscan.
  *
+ * If 'progress', we update the PROGRESS_SCAN_BLOCKS_DONE counter as we go
+ * along.  Also, if that flag is true and a scan descriptor is not passed (ie.
+ * when not doing a parallel scan), the PROGRESS_SCAN_BLOCKS_TOTAL counter is
+ * updated at the beginning.  For parallel scans, caller is expected to have
+ * set the total number of blocks prior to calling this function.  Note this
+ * only works for full-relation scans.
+ *
  * When "anyvisible" mode is requested, all tuples visible to any transaction
  * are indexed and counted as live, including those inserted or deleted by
  * transactions that are still in progress.
@@ -2454,6 +2487,7 @@ IndexBuildHeapRangeScan(Relation heapRelation,
 						IndexInfo *indexInfo,
 						bool allow_sync,
 						bool anyvisible,
+						bool progress,
 						BlockNumber start_blockno,
 						BlockNumber numblocks,
 						IndexBuildCallback callback,
@@ -2476,6 +2510,7 @@ IndexBuildHeapRangeScan(Relation heapRelation,
 	TransactionId OldestXmin;
 	BlockNumber root_blkno = InvalidBlockNumber;
 	OffsetNumber root_offsets[MaxHeapTuplesPerPage];
+	BlockNumber	previous_blkno = InvalidBlockNumber;
 
 	/*
 	 * sanity checks
@@ -2544,6 +2579,14 @@ IndexBuildHeapRangeScan(Relation heapRelation,
 									 NULL,	/* scan key */
 									 true,	/* buffer access strategy OK */
 									 allow_sync);	/* syncscan OK? */
+
+		if (progress)
+		{
+			hscan = (HeapScanDesc) scan;
+			Assert(hscan->rs_numblocks == InvalidBlockNumber);
+			pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL,
+										 hscan->rs_nblocks);
+		}
 	}
 	else
 	{
@@ -2593,6 +2636,19 @@ IndexBuildHeapRangeScan(Relation heapRelation,
 
 		CHECK_FOR_INTERRUPTS();
 
+		/* Report scan progress, if asked to. */
+		if (progress)
+		{
+			BlockNumber		blocks_done = heapscan_get_blocks_done(scan);
+
+			if (blocks_done != previous_blkno)
+			{
+				pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
+											 blocks_done);
+				previous_blkno = blocks_done;
+			}
+		}
+
 		/*
 		 * When dealing with a HOT-chain of updated tuples, we want to index
 		 * the values of the live tuple (if any), but index it under the TID
@@ -3129,6 +3185,21 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot)
 	int			save_sec_context;
 	int			save_nestlevel;
 
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_PHASE,
+			PROGRESS_CREATEIDX_TUPLES_DONE,
+			PROGRESS_CREATEIDX_TUPLES_TOTAL,
+			PROGRESS_SCAN_BLOCKS_DONE,
+			PROGRESS_SCAN_BLOCKS_TOTAL
+		};
+		const int64	val[] = {
+			PROGRESS_CREATEIDX_PHASE_VALIDATE_IDXSCAN,
+			0, 0, 0, 0
+		};
+		pgstat_progress_update_multi_param(5, index, val);
+	}
+
 	/* Open and lock the parent heap relation */
 	heapRelation = table_open(heapId, ShareUpdateExclusiveLock);
 	/* And the target index relation */
@@ -3159,6 +3230,7 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot)
 	 */
 	ivinfo.index = indexRelation;
 	ivinfo.analyze_only = false;
+	ivinfo.report_progress = true;	/* XXX only for btree? */
 	ivinfo.estimated_count = true;
 	ivinfo.message_level = DEBUG2;
 	ivinfo.num_heap_tuples = heapRelation->rd_rel->reltuples;
@@ -3176,15 +3248,39 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot)
 											NULL, false);
 	state.htups = state.itups = state.tups_inserted = 0;
 
+	/* ambulkdelete updates progress metrics */
 	(void) index_bulk_delete(&ivinfo, NULL,
 							 validate_index_callback, (void *) &state);
 
 	/* Execute the sort */
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_PHASE,
+			PROGRESS_SCAN_BLOCKS_DONE,
+			PROGRESS_SCAN_BLOCKS_TOTAL
+		};
+		const int64	val[] = {
+			PROGRESS_CREATEIDX_PHASE_SORT_IDXSCAN,
+			0, 0
+		};
+
+		pgstat_progress_update_multi_param(3, index, val);
+	}
 	tuplesort_performsort(state.tuplesort);
 
 	/*
-	 * Now scan the heap and "merge" it with the index
+	 * Now scan the heap and "merge" it with the index.
 	 */
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_PHASE,
+		};
+		const int64	val[] = {
+			PROGRESS_CREATEIDX_PHASE_HEAPSCAN_VALIDATE
+		};
+
+		pgstat_progress_update_multi_param(1, index, val);
+	}
 	validate_index_heapscan(heapRelation,
 							indexRelation,
 							indexInfo,
@@ -3288,6 +3384,7 @@ validate_index_heapscan(Relation heapRelation,
 	BlockNumber root_blkno = InvalidBlockNumber;
 	OffsetNumber root_offsets[MaxHeapTuplesPerPage];
 	bool		in_index[MaxHeapTuplesPerPage];
+	BlockNumber	previous_blkno = InvalidBlockNumber;
 
 	/* state variables for the merge */
 	ItemPointer indexcursor = NULL;
@@ -3328,6 +3425,9 @@ validate_index_heapscan(Relation heapRelation,
 								 false); /* syncscan not OK */
 	hscan = (HeapScanDesc) scan;
 
+	pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL,
+								 hscan->rs_nblocks);
+
 	/*
 	 * Scan all tuples matching the snapshot.
 	 */
@@ -3341,6 +3441,14 @@ validate_index_heapscan(Relation heapRelation,
 
 		state->htups += 1;
 
+		if ((previous_blkno == InvalidBlockNumber) ||
+			(hscan->rs_cblock != previous_blkno))
+		{
+			pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
+										 hscan->rs_cblock);
+			previous_blkno = hscan->rs_cblock;
+		}
+
 		/*
 		 * As commented in IndexBuildHeapScan, we should index heap-only
 		 * tuples under the TIDs of their root tuples; so when we advance onto
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index b89df70653e..b7c040d5284 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -934,6 +934,33 @@ CREATE VIEW pg_stat_progress_cluster AS
     FROM pg_stat_get_progress_info('CLUSTER') AS S
         LEFT JOIN pg_database D ON S.datid = D.oid;
 
+CREATE VIEW pg_stat_progress_create_index AS
+	SELECT
+		S.pid AS pid, S.datid AS datid, D.datname AS datname,
+		S.relid AS relid,
+		CASE S.param2 WHEN 0 THEN 'initializing'
+					  WHEN 1 THEN 'waiting for old snapshots'
+					  WHEN 2 THEN 'building index' ||
+						COALESCE((': ' || pg_indexam_progress_phasename(S.param1::oid, S.param3)),
+							'')
+					  WHEN 3 THEN 'waiting for writer snapshots'
+					  WHEN 4 THEN 'index validation: scan index'
+					  WHEN 5 THEN 'index validation: sort index scan results'
+					  WHEN 6 THEN 'index validation: scan heap'
+					  WHEN 7 THEN 'waiting for reader snapshots'
+					  END as phase,
+		S.param4 AS lockers_total,
+		S.param5 AS lockers_done,
+		S.param6 AS current_locker_pid,
+		S.param7 AS blocks_total,
+		S.param8 AS blocks_done,
+		S.param9 AS tuples_total,
+		S.param10 AS tuples_done,
+		S.param11 AS partitions_total,
+		S.param12 AS partitions_done
+	FROM pg_stat_get_progress_info('CREATE INDEX') AS S
+		LEFT JOIN pg_database D ON S.datid = D.oid;
+
 CREATE VIEW pg_user_mappings AS
     SELECT
         U.oid       AS umid,
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index c3a53d81aab..9fad6705e45 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -36,6 +36,7 @@
 #include "commands/dbcommands.h"
 #include "commands/defrem.h"
 #include "commands/event_trigger.h"
+#include "commands/progress.h"
 #include "commands/tablecmds.h"
 #include "commands/tablespace.h"
 #include "mb/pg_wchar.h"
@@ -47,10 +48,12 @@
 #include "parser/parse_func.h"
 #include "parser/parse_oper.h"
 #include "partitioning/partdesc.h"
+#include "pgstat.h"
 #include "rewrite/rewriteManip.h"
 #include "storage/lmgr.h"
 #include "storage/proc.h"
 #include "storage/procarray.h"
+#include "storage/sinvaladt.h"
 #include "utils/acl.h"
 #include "utils/builtins.h"
 #include "utils/fmgroids.h"
@@ -369,6 +372,15 @@ DefineIndex(Oid relationId,
 	Snapshot	snapshot;
 	int			i;
 
+
+	/*
+	 * Start progress report.  If we're building a partition, this was already
+	 * done.
+	 */
+	if (!OidIsValid(parentIndexId))
+		pgstat_progress_start_command(PROGRESS_COMMAND_CREATE_INDEX,
+									  relationId);
+
 	/*
 	 * count key attributes in index
 	 */
@@ -585,6 +597,9 @@ DefineIndex(Oid relationId,
 	accessMethodId = accessMethodForm->oid;
 	amRoutine = GetIndexAmRoutine(accessMethodForm->amhandler);
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_ACCESS_METHOD_OID,
+								 accessMethodId);
+
 	if (stmt->unique && !amRoutine->amcanunique)
 		ereport(ERROR,
 				(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -865,6 +880,11 @@ DefineIndex(Oid relationId,
 	if (!OidIsValid(indexRelationId))
 	{
 		table_close(rel, NoLock);
+
+		/* If this is the top-level index, we're done */
+		if (!OidIsValid(parentIndexId))
+			pgstat_progress_end_command();
+
 		return address;
 	}
 
@@ -890,6 +910,9 @@ DefineIndex(Oid relationId,
 			TupleDesc	parentDesc;
 			Oid		   *opfamOids;
 
+			pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL,
+										 nparts);
+
 			memcpy(part_oids, partdesc->oids, sizeof(Oid) * nparts);
 
 			parentDesc = CreateTupleDescCopy(RelationGetDescr(rel));
@@ -1040,6 +1063,8 @@ DefineIndex(Oid relationId,
 								skip_build, quiet);
 				}
 
+				pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_DONE,
+											 i + 1);
 				pfree(attmap);
 			}
 
@@ -1074,6 +1099,8 @@ DefineIndex(Oid relationId,
 		 * Indexes on partitioned tables are not themselves built, so we're
 		 * done here.
 		 */
+		if (!OidIsValid(parentIndexId))
+			pgstat_progress_end_command();
 		return address;
 	}
 
@@ -1081,6 +1108,11 @@ DefineIndex(Oid relationId,
 	{
 		/* Close the heap and we're done, in the non-concurrent case */
 		table_close(rel, NoLock);
+
+		/* If this is the top-level index, we're done. */
+		if (!OidIsValid(parentIndexId))
+			pgstat_progress_end_command();
+
 		return address;
 	}
 
@@ -1132,7 +1164,9 @@ DefineIndex(Oid relationId,
 	 * exclusive lock on our table.  The lock code will detect deadlock and
 	 * error out properly.
 	 */
-	WaitForLockers(heaplocktag, ShareLock);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_1);
+	WaitForLockers(heaplocktag, ShareLock, true);
 
 	/*
 	 * At this moment we are sure that there are no transactions with the
@@ -1196,7 +1230,9 @@ DefineIndex(Oid relationId,
 	 * We once again wait until no transaction can have the table open with
 	 * the index marked as read-only for updates.
 	 */
-	WaitForLockers(heaplocktag, ShareLock);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_2);
+	WaitForLockers(heaplocktag, ShareLock, true);
 
 	/*
 	 * Now take the "reference snapshot" that will be used by validate_index()
@@ -1282,6 +1318,9 @@ DefineIndex(Oid relationId,
 	old_snapshots = GetCurrentVirtualXIDs(limitXmin, true, false,
 										  PROC_IS_AUTOVACUUM | PROC_IN_VACUUM,
 										  &n_old_snapshots);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_3);
+	pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, n_old_snapshots);
 
 	for (i = 0; i < n_old_snapshots; i++)
 	{
@@ -1317,7 +1356,14 @@ DefineIndex(Oid relationId,
 		}
 
 		if (VirtualTransactionIdIsValid(old_snapshots[i]))
+		{
+			PGPROC *holder = BackendIdGetProc(old_snapshots[i].backendId);
+			pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID,
+										 holder->pid);
 			VirtualXactLock(old_snapshots[i], true);
+		}
+
+		pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, i + 1);
 	}
 
 	/*
@@ -1340,6 +1386,8 @@ DefineIndex(Oid relationId,
 	 */
 	UnlockRelationIdForSession(&heaprelid, ShareUpdateExclusiveLock);
 
+	pgstat_progress_end_command();
+
 	return address;
 }
 
diff --git a/src/backend/storage/ipc/standby.c b/src/backend/storage/ipc/standby.c
index 4d10e57a803..a0a2b964703 100644
--- a/src/backend/storage/ipc/standby.c
+++ b/src/backend/storage/ipc/standby.c
@@ -401,7 +401,7 @@ ResolveRecoveryConflictWithLock(LOCKTAG locktag)
 		 */
 		VirtualTransactionId *backends;
 
-		backends = GetLockConflicts(&locktag, AccessExclusiveLock);
+		backends = GetLockConflicts(&locktag, AccessExclusiveLock, NULL);
 		ResolveRecoveryConflictWithVirtualXIDs(backends,
 											   PROCSIG_RECOVERY_CONFLICT_LOCK);
 	}
diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c
index e688ba81170..0b04b093782 100644
--- a/src/backend/storage/lmgr/lmgr.c
+++ b/src/backend/storage/lmgr/lmgr.c
@@ -19,9 +19,12 @@
 #include "access/transam.h"
 #include "access/xact.h"
 #include "catalog/catalog.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
+#include "pgstat.h"
 #include "storage/lmgr.h"
 #include "storage/procarray.h"
+#include "storage/sinvaladt.h"
 #include "utils/inval.h"
 
 
@@ -857,10 +860,12 @@ XactLockTableWaitErrorCb(void *arg)
  * after we obtained our initial list of lockers, we will not wait for them.
  */
 void
-WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
+WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
 {
 	List	   *holders = NIL;
 	ListCell   *lc;
+	int			total = 0;
+	int			done = 0;
 
 	/* Done if no locks to wait for */
 	if (list_length(locktags) == 0)
@@ -870,10 +875,17 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
 	foreach(lc, locktags)
 	{
 		LOCKTAG    *locktag = lfirst(lc);
+		int			count;
 
-		holders = lappend(holders, GetLockConflicts(locktag, lockmode));
+		holders = lappend(holders,
+						  GetLockConflicts(locktag, lockmode,
+										   progress ? &count : NULL));
+		total += count;
 	}
 
+	if (progress)
+		pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, total);
+
 	/*
 	 * Note: GetLockConflicts() never reports our own xid, hence we need not
 	 * check for that.  Also, prepared xacts are not reported, which is fine
@@ -887,10 +899,36 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
 
 		while (VirtualTransactionIdIsValid(*lockholders))
 		{
+			/*
+			 * If requested, publish who we're going to wait for.  This is not
+			 * 100% accurate if they're already gone, but we don't care.
+			 */
+			if (progress)
+			{
+				PGPROC *holder = BackendIdGetProc(lockholders->backendId);
+
+				pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID,
+											 holder->pid);
+			}
 			VirtualXactLock(*lockholders, true);
 			lockholders++;
+
+			if (progress)
+				pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, ++done);
 		}
 	}
+	if (progress)
+	{
+		const int	index[] = {
+			PROGRESS_WAITFOR_TOTAL,
+			PROGRESS_WAITFOR_DONE,
+			PROGRESS_WAITFOR_CURRENT_PID
+		};
+		const int64	values[] = {
+			0, 0, 0
+		};
+		pgstat_progress_update_multi_param(3, index, values);
+	}
 
 	list_free_deep(holders);
 }
@@ -901,12 +939,12 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
  * Same as WaitForLockersMultiple, for a single lock tag.
  */
 void
-WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode)
+WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress)
 {
 	List	   *l;
 
 	l = list_make1(&heaplocktag);
-	WaitForLockersMultiple(l, lockmode);
+	WaitForLockersMultiple(l, lockmode, progress);
 	list_free(l);
 }
 
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 78fdbd6ff88..c8958766f1e 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2807,6 +2807,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock)
  *		xacts merely awaiting such a lock are NOT reported.
  *
  * The result array is palloc'd and is terminated with an invalid VXID.
+ * *countp, if not null, is updated to the number of items set.
  *
  * Of course, the result could be out of date by the time it's returned,
  * so use of this function has to be thought about carefully.
@@ -2817,7 +2818,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock)
  * uses of the result.
  */
 VirtualTransactionId *
-GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode)
+GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
 {
 	static VirtualTransactionId *vxids;
 	LOCKMETHODID lockmethodid = locktag->locktag_lockmethodid;
@@ -2964,6 +2965,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode)
 		LWLockRelease(partitionLock);
 		vxids[count].backendId = InvalidBackendId;
 		vxids[count].localTransactionId = InvalidLocalTransactionId;
+		if (countp)
+			*countp = count;
 		return vxids;
 	}
 
@@ -3019,6 +3022,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode)
 
 	vxids[count].backendId = InvalidBackendId;
 	vxids[count].localTransactionId = InvalidLocalTransactionId;
+	if (countp)
+		*countp = count;
 	return vxids;
 }
 
diff --git a/src/backend/utils/adt/amutils.c b/src/backend/utils/adt/amutils.c
index 060ffe501ec..e81d6cc0562 100644
--- a/src/backend/utils/adt/amutils.c
+++ b/src/backend/utils/adt/amutils.c
@@ -445,3 +445,26 @@ pg_index_column_has_property(PG_FUNCTION_ARGS)
 
 	return indexam_property(fcinfo, propname, InvalidOid, relid, attno);
 }
+
+/*
+ * Return the name of the given phase, as used for progress reporting by the
+ * given AM.
+ */
+Datum
+pg_indexam_progress_phasename(PG_FUNCTION_ARGS)
+{
+	Oid			amoid = PG_GETARG_OID(0);
+	int32		phasenum = PG_GETARG_INT32(1);
+	IndexAmRoutine *routine;
+	char	   *name;
+
+	routine = GetIndexAmRoutineByAmId(amoid, true);
+	if (routine == NULL || !routine->ambuildphasename)
+		PG_RETURN_NULL();
+
+	name = routine->ambuildphasename(phasenum);
+	if (!name)
+		PG_RETURN_NULL();
+
+	PG_RETURN_TEXT_P(CStringGetTextDatum(name));
+}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 90a817a25c5..7c2afe64272 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -470,6 +470,8 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
 		cmdtype = PROGRESS_COMMAND_VACUUM;
 	else if (pg_strcasecmp(cmd, "CLUSTER") == 0)
 		cmdtype = PROGRESS_COMMAND_CLUSTER;
+	else if (pg_strcasecmp(cmd, "CREATE INDEX") == 0)
+		cmdtype = PROGRESS_COMMAND_CREATE_INDEX;
 	else
 		ereport(ERROR,
 				(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 653ddc976ba..09a7404267c 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -108,6 +108,9 @@ typedef bool (*amproperty_function) (Oid index_oid, int attno,
 									 IndexAMProperty prop, const char *propname,
 									 bool *res, bool *isnull);
 
+/* name of phase as used in progress reporting */
+typedef char *(*ambuildphasename_function) (int64 phasenum);
+
 /* validate definition of an opclass for this AM */
 typedef bool (*amvalidate_function) (Oid opclassoid);
 
@@ -213,6 +216,7 @@ typedef struct IndexAmRoutine
 	amcostestimate_function amcostestimate;
 	amoptions_function amoptions;
 	amproperty_function amproperty; /* can be NULL */
+	ambuildphasename_function ambuildphasename;	/* can be NULL */
 	amvalidate_function amvalidate;
 	ambeginscan_function ambeginscan;
 	amrescan_function amrescan;
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index cad66513f62..e38e209e68c 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -45,6 +45,7 @@ typedef struct IndexVacuumInfo
 {
 	Relation	index;			/* the index being vacuumed */
 	bool		analyze_only;	/* ANALYZE (without any actual vacuum) */
+	bool		report_progress;	/* emit progress.h status reports */
 	bool		estimated_count;	/* num_heap_tuples is an estimate */
 	int			message_level;	/* ereport level for progress messages */
 	double		num_heap_tuples;	/* tuples remaining in heap */
diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h
index 473c6f29185..bc0f3027629 100644
--- a/src/include/access/nbtree.h
+++ b/src/include/access/nbtree.h
@@ -672,6 +672,16 @@ typedef BTScanOpaqueData *BTScanOpaque;
 #define SK_BT_DESC			(INDOPTION_DESC << SK_BT_INDOPTION_SHIFT)
 #define SK_BT_NULLS_FIRST	(INDOPTION_NULLS_FIRST << SK_BT_INDOPTION_SHIFT)
 
+/*
+ * Constant definition for progress reporting.  Phase numbers must match
+ * btbuildphasename.
+ */
+/* PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE is 1 (see progress.h) */
+#define PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN		2
+#define PROGRESS_BTREE_PHASE_PERFORMSORT_1				3
+#define PROGRESS_BTREE_PHASE_PERFORMSORT_2				4
+#define PROGRESS_BTREE_PHASE_LEAF_LOAD					5
+
 /*
  * external entry points for btree, in nbtree.c
  */
@@ -785,6 +795,7 @@ extern bytea *btoptions(Datum reloptions, bool validate);
 extern bool btproperty(Oid index_oid, int attno,
 		   IndexAMProperty prop, const char *propname,
 		   bool *res, bool *isnull);
+extern char *btbuildphasename(int64 phasenum);
 extern IndexTuple _bt_truncate(Relation rel, IndexTuple lastleft,
 			 IndexTuple firstright, BTScanInsert itup_key);
 extern int _bt_keep_natts_fast(Relation rel, IndexTuple lastleft,
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index 29f7ed62379..f90e52382ba 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -115,6 +115,7 @@ extern double IndexBuildHeapScan(Relation heapRelation,
 				   Relation indexRelation,
 				   IndexInfo *indexInfo,
 				   bool allow_sync,
+				   bool progress,
 				   IndexBuildCallback callback,
 				   void *callback_state,
 				   struct TableScanDescData *scan);
@@ -123,6 +124,7 @@ extern double IndexBuildHeapRangeScan(Relation heapRelation,
 						IndexInfo *indexInfo,
 						bool allow_sync,
 						bool anyvisible,
+						bool progress,
 						BlockNumber start_blockno,
 						BlockNumber end_blockno,
 						IndexBuildCallback callback,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index acf1131b521..22ce0d20247 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -924,6 +924,10 @@
   proname => 'pg_index_column_has_property', provolatile => 's',
   prorettype => 'bool', proargtypes => 'regclass int4 text',
   prosrc => 'pg_index_column_has_property' },
+{ oid => '676', descr => 'return name of given index build phase',
+  proname => 'pg_indexam_progress_phasename', provolatile => 'i',
+  prorettype => 'text', proargtypes => 'oid int8',
+  prosrc => 'pg_indexam_progress_phasename' },
 
 { oid => '339',
   proname => 'poly_same', prorettype => 'bool',
@@ -5098,9 +5102,9 @@
   proname => 'pg_stat_get_progress_info', prorows => '100', proretset => 't',
   provolatile => 's', proparallel => 'r', prorettype => 'record',
   proargtypes => 'text',
-  proallargtypes => '{text,int4,oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}',
-  proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o}',
-  proargnames => '{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10}',
+  proallargtypes => '{text,int4,oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}',
+  proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+  proargnames => '{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10,param11,param12}',
   prosrc => 'pg_stat_get_progress_info' },
 { oid => '3099',
   descr => 'statistics: information about currently active replication',
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index 04542d9e923..45847f6dc35 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -57,4 +57,39 @@
 #define PROGRESS_CLUSTER_COMMAND_CLUSTER		1
 #define PROGRESS_CLUSTER_COMMAND_VACUUM_FULL	2
 
+/* Progress parameters for CREATE INDEX */
+#define PROGRESS_CREATEIDX_ACCESS_METHOD_OID	0
+#define PROGRESS_CREATEIDX_PHASE				1	/* AM-agnostic phase # */
+#define PROGRESS_CREATEIDX_SUBPHASE				2	/* phase # filled by AM */
+/* 3, 4 and 5 reserved for "waitfor" metrics */
+/* 6 and 7 reserved for "block number" metrics */
+#define PROGRESS_CREATEIDX_TUPLES_TOTAL			8
+#define PROGRESS_CREATEIDX_TUPLES_DONE			9
+#define PROGRESS_CREATEIDX_PARTITIONS_TOTAL		10
+#define PROGRESS_CREATEIDX_PARTITIONS_DONE		11
+
+/* Phases of CREATE INDEX */
+#define PROGRESS_CREATEIDX_PHASE_WAIT_1			1
+#define PROGRESS_CREATEIDX_PHASE_BUILD			2
+#define PROGRESS_CREATEIDX_PHASE_WAIT_2			3
+#define PROGRESS_CREATEIDX_PHASE_VALIDATE_IDXSCAN		4
+#define PROGRESS_CREATEIDX_PHASE_SORT_IDXSCAN		5
+#define PROGRESS_CREATEIDX_PHASE_HEAPSCAN_VALIDATE		6
+#define PROGRESS_CREATEIDX_PHASE_WAIT_3			7
+
+/*
+ * Subphases of CREATE INDEX, for index_build.
+ */
+#define PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE	1
+/* Additional phases are defined by each AM */
+
+/* Lock holder wait counts */
+#define PROGRESS_WAITFOR_TOTAL					3
+#define PROGRESS_WAITFOR_DONE					4
+#define PROGRESS_WAITFOR_CURRENT_PID			5
+
+/* Block numbers in a generic relation scan */
+#define PROGRESS_SCAN_BLOCKS_TOTAL				6
+#define PROGRESS_SCAN_BLOCKS_DONE				7
+
 #endif
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index c080fa6388f..6af9ab7e1d9 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -951,10 +951,11 @@ typedef enum ProgressCommandType
 {
 	PROGRESS_COMMAND_INVALID,
 	PROGRESS_COMMAND_VACUUM,
-	PROGRESS_COMMAND_CLUSTER
+	PROGRESS_COMMAND_CLUSTER,
+	PROGRESS_COMMAND_CREATE_INDEX
 } ProgressCommandType;
 
-#define PGSTAT_NUM_PROGRESS_PARAM	10
+#define PGSTAT_NUM_PROGRESS_PARAM	12
 
 /* ----------
  * Shared-memory data structures
diff --git a/src/include/storage/lmgr.h b/src/include/storage/lmgr.h
index 3d705faba5c..4f2872de35f 100644
--- a/src/include/storage/lmgr.h
+++ b/src/include/storage/lmgr.h
@@ -78,8 +78,8 @@ extern void XactLockTableWait(TransactionId xid, Relation rel,
 extern bool ConditionalXactLockTableWait(TransactionId xid);
 
 /* Lock VXIDs, specified by conflicting locktags */
-extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode);
-extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode);
+extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress);
+extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress);
 
 /* Lock an XID for tuple insertion (used to wait for an insertion to finish) */
 extern uint32 SpeculativeInsertionLockAcquire(TransactionId xid);
diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h
index badf7fd682b..048947c50d4 100644
--- a/src/include/storage/lock.h
+++ b/src/include/storage/lock.h
@@ -544,7 +544,7 @@ extern bool LockHeldByMe(const LOCKTAG *locktag, LOCKMODE lockmode);
 extern bool LockHasWaiters(const LOCKTAG *locktag,
 			   LOCKMODE lockmode, bool sessionLock);
 extern VirtualTransactionId *GetLockConflicts(const LOCKTAG *locktag,
-				 LOCKMODE lockmode);
+				 LOCKMODE lockmode, int *countp);
 extern void AtPrepare_Locks(void);
 extern void PostPrepare_Locks(TransactionId xid);
 extern int LockCheckConflicts(LockMethod lockMethodTable,
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index d5f309fbfbe..a9a2910fb55 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1856,7 +1856,33 @@ pg_stat_progress_cluster| SELECT s.pid,
     s.param6 AS heap_blks_total,
     s.param7 AS heap_blks_scanned,
     s.param8 AS index_rebuild_count
-   FROM (pg_stat_get_progress_info('CLUSTER'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10)
+   FROM (pg_stat_get_progress_info('CLUSTER'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12)
+     LEFT JOIN pg_database d ON ((s.datid = d.oid)));
+pg_stat_progress_create_index| SELECT s.pid,
+    s.datid,
+    d.datname,
+    s.relid,
+        CASE s.param2
+            WHEN 0 THEN 'initializing'::text
+            WHEN 1 THEN 'waiting for old snapshots'::text
+            WHEN 2 THEN ('building index'::text || COALESCE((': '::text || pg_indexam_progress_phasename((s.param1)::oid, s.param3)), ''::text))
+            WHEN 3 THEN 'waiting for writer snapshots'::text
+            WHEN 4 THEN 'index validation: scan index'::text
+            WHEN 5 THEN 'index validation: sort index scan results'::text
+            WHEN 6 THEN 'index validation: scan heap'::text
+            WHEN 7 THEN 'waiting for reader snapshots'::text
+            ELSE NULL::text
+        END AS phase,
+    s.param4 AS lockers_total,
+    s.param5 AS lockers_done,
+    s.param6 AS current_locker_pid,
+    s.param7 AS blocks_total,
+    s.param8 AS blocks_done,
+    s.param9 AS tuples_total,
+    s.param10 AS tuples_done,
+    s.param11 AS partitions_total,
+    s.param12 AS partitions_done
+   FROM (pg_stat_get_progress_info('CREATE INDEX'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12)
      LEFT JOIN pg_database d ON ((s.datid = d.oid)));
 pg_stat_progress_vacuum| SELECT s.pid,
     s.datid,
@@ -1878,7 +1904,7 @@ pg_stat_progress_vacuum| SELECT s.pid,
     s.param5 AS index_vacuum_count,
     s.param6 AS max_dead_tuples,
     s.param7 AS num_dead_tuples
-   FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10)
+   FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12)
      LEFT JOIN pg_database d ON ((s.datid = d.oid)));
 pg_stat_replication| SELECT s.pid,
     s.usesysid,
-- 
2.17.1

v6-0003-report-progress-of-hash-indexes.patchtext/x-diff; charset=us-asciiDownload
From 5ea6bc6b7b520b142f925e24cdeb54ea9b17ec5d Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Tue, 26 Feb 2019 14:34:49 -0300
Subject: [PATCH v6 3/3] report progress of hash indexes

---
 src/backend/access/hash/hash.c     | 6 +++++-
 src/backend/access/hash/hashsort.c | 6 ++++++
 2 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index fc7db5d6a13..cf7ec655044 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -22,9 +22,11 @@
 #include "access/hash_xlog.h"
 #include "access/relscan.h"
 #include "catalog/index.h"
+#include "commands/progress.h"
 #include "commands/vacuum.h"
 #include "miscadmin.h"
 #include "optimizer/plancat.h"
+#include "pgstat.h"
 #include "utils/builtins.h"
 #include "utils/index_selfuncs.h"
 #include "utils/rel.h"
@@ -160,8 +162,10 @@ hashbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	buildstate.heapRel = heap;
 
 	/* do the heap scan */
-	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false,
+	reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, true,
 								   hashbuildCallback, (void *) &buildstate, NULL);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_TOTAL,
+								 buildstate.indtuples);
 
 	if (buildstate.spool)
 	{
diff --git a/src/backend/access/hash/hashsort.c b/src/backend/access/hash/hashsort.c
index 8c55436b193..00a57470a77 100644
--- a/src/backend/access/hash/hashsort.c
+++ b/src/backend/access/hash/hashsort.c
@@ -26,7 +26,9 @@
 #include "postgres.h"
 
 #include "access/hash.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
+#include "pgstat.h"
 #include "utils/tuplesort.h"
 
 
@@ -116,6 +118,7 @@ void
 _h_indexbuild(HSpool *hspool, Relation heapRel)
 {
 	IndexTuple	itup;
+	long		tups_done = 0;
 #ifdef USE_ASSERT_CHECKING
 	uint32		hashkey = 0;
 #endif
@@ -141,5 +144,8 @@ _h_indexbuild(HSpool *hspool, Relation heapRel)
 #endif
 
 		_hash_doinsert(hspool->index, itup, heapRel);
+
+		pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE,
+									 ++tups_done);
 	}
 }
-- 
2.17.1

#32Amit Langote
Langote_Amit_f8@lab.ntt.co.jp
In reply to: Alvaro Herrera (#31)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On 2019/03/26 1:53, Alvaro Herrera wrote:

Here's v6 of this patch. I have rebased on top of today's CLUSTER
monitoring, as well as on table AM commits. The latter caused a bit of
trouble, as now the number of blocks processed by a scan is not as easy
to get as before; I added a new entry point heapscan_get_blocks_done on
heapam.c to help with that. (I suppose this will need some fixups later
on.)

I removed the "M of N" phase labels that Robert didn't like; those were
suggested by Rahila and upvoted by Amit L. I'm of two minds about
those. If you care about those and want them back, please speak up.

On second thought, I'm neutral on it too.

Thanks,
Amit

#33Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Alvaro Herrera (#31)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On 2019-Mar-25, Alvaro Herrera wrote:

Here's v6 of this patch. I have rebased on top of today's CLUSTER
monitoring, as well as on table AM commits. The latter caused a bit of
trouble, as now the number of blocks processed by a scan is not as easy
to get as before; I added a new entry point heapscan_get_blocks_done on
heapam.c to help with that. (I suppose this will need some fixups later
on.)

Andres, I suppose you have something to say about patch 0001 here?

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#34Andres Freund
andres@anarazel.de
In reply to: Alvaro Herrera (#33)
Re: monitoring CREATE INDEX [CONCURRENTLY]

Hi,

On 2019-03-25 23:11:00 -0300, Alvaro Herrera wrote:

On 2019-Mar-25, Alvaro Herrera wrote:

Here's v6 of this patch. I have rebased on top of today's CLUSTER
monitoring, as well as on table AM commits. The latter caused a bit of
trouble, as now the number of blocks processed by a scan is not as easy
to get as before; I added a new entry point heapscan_get_blocks_done on
heapam.c to help with that. (I suppose this will need some fixups later
on.)

Andres, I suppose you have something to say about patch 0001 here?

I've not followed this thread at all, so I might just be out of my depth
here. From my POV, a later patch in the yet-to-be-applied patchqueue
moves the main part of cluster below the table AM (as there's enough low
level details, e.g. dealing with HOT). Therefore I don't have a problem
having heap's implementation interrogate the scan in a heap specific
manner.

Is that the angle you were wondering about? If not, any chance to point
out more precisely what to look at?

Obviously out of pure laziness, I'd prefer this to go in after my move
of index creation scans & cluster below tableam.h. But admittedly,
managing my exhaustion isn't the the sole goal of the project....

Greetings,

Andres Freund

#35Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Andres Freund (#34)
Re: monitoring CREATE INDEX [CONCURRENTLY]

Hi

On 2019-Mar-25, Andres Freund wrote:

I've not followed this thread at all, so I might just be out of my depth
here. From my POV, a later patch in the yet-to-be-applied patchqueue
moves the main part of cluster below the table AM (as there's enough low
level details, e.g. dealing with HOT). Therefore I don't have a problem
having heap's implementation interrogate the scan in a heap specific
manner.

Is that the angle you were wondering about? If not, any chance to point
out more precisely what to look at?

Obviously out of pure laziness, I'd prefer this to go in after my move
of index creation scans & cluster below tableam.h. But admittedly,
managing my exhaustion isn't the the sole goal of the project....

Well, this is create index rather than cluster, but yes this conflicts
pretty heavily with patch 0008 in your v21 at
20190324031630.nt7numguo5ojq6uv@alap3.anarazel.de. I wonder if I should
rather push and help merge your 0008, or wait until you push and deal
with it afterwards. I'd rather do the former, I think.

Anyway I was thinking about the conceptual angle -- the progress
monitoring stuff is doing block-based reporting. I think even if we get
a non-block-based heap, we can still report the number of physical
blocks already processed by the scan, which is what the index build
monitoring is interested in showing the user.

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#36Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Alvaro Herrera (#31)
2 attachment(s)
Re: monitoring CREATE INDEX [CONCURRENTLY]

Here's v7. This is rebased on top of yesterday's tableam commit
reworking the index build API (thanks Rahila for letting me know it had
already rot). No changes otherwise. Got rid of 0001 because the
tableam changes made that unnecessary. (Each new table AM will have to
include its own progress reporting for index builds in its
index_build_range_scan method, which is sensible.)

Patch 0003 now takes care of all the AMs. This supports the index build
phase as well as the index-validate-heapscan for CONCURRENTLY builds;
the indexscan scan there is not reported, which is just a small portion
of the index build so I don't feel bad about that; it can probably be
added with just a two-line patch on each AM's ambulkdelete method as a
subsequent patch.

I have not reinstated phase numbers; I have Rahila's positive vote for
them. Do I hear any more votes on this issue?

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attachments:

v7-0001-Report-progress-of-CREATE-INDEX-operations.patchtext/x-diff; charset=us-asciiDownload
From 12ebc1e6bd165e6d6c1dad83d2e458dd76367e78 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Mon, 25 Mar 2019 13:29:34 -0300
Subject: [PATCH v7 1/2] Report progress of CREATE INDEX operations

---
 contrib/amcheck/verify_nbtree.c          |   2 +-
 contrib/bloom/blinsert.c                 |   2 +-
 contrib/bloom/blutils.c                  |   1 +
 doc/src/sgml/indexam.sgml                |  13 ++
 doc/src/sgml/monitoring.sgml             | 224 ++++++++++++++++++++++-
 src/backend/access/brin/brin.c           |   5 +-
 src/backend/access/gin/gininsert.c       |   2 +-
 src/backend/access/gin/ginutil.c         |   1 +
 src/backend/access/gist/gist.c           |   1 +
 src/backend/access/gist/gistbuild.c      |   2 +-
 src/backend/access/hash/hash.c           |   3 +-
 src/backend/access/heap/heapam_handler.c |  75 ++++++++
 src/backend/access/nbtree/nbtree.c       |   9 +
 src/backend/access/nbtree/nbtsort.c      |  61 +++++-
 src/backend/access/nbtree/nbtutils.c     |  24 +++
 src/backend/access/spgist/spginsert.c    |   2 +-
 src/backend/access/spgist/spgutils.c     |   1 +
 src/backend/catalog/index.c              |  67 ++++++-
 src/backend/catalog/system_views.sql     |  27 +++
 src/backend/commands/indexcmds.c         |  52 +++++-
 src/backend/storage/ipc/standby.c        |   2 +-
 src/backend/storage/lmgr/lmgr.c          |  46 ++++-
 src/backend/storage/lmgr/lock.c          |   7 +-
 src/backend/utils/adt/amutils.c          |  23 +++
 src/backend/utils/adt/pgstatfuncs.c      |   2 +
 src/include/access/amapi.h               |   4 +
 src/include/access/genam.h               |   1 +
 src/include/access/nbtree.h              |  11 ++
 src/include/access/tableam.h             |  10 +
 src/include/catalog/pg_proc.dat          |  10 +-
 src/include/commands/progress.h          |  35 ++++
 src/include/pgstat.h                     |   5 +-
 src/include/storage/lmgr.h               |   4 +-
 src/include/storage/lock.h               |   2 +-
 src/test/regress/expected/rules.out      |  30 ++-
 35 files changed, 728 insertions(+), 38 deletions(-)

diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index 9ecb1999e34..b55178328c8 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -566,7 +566,7 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
 			 RelationGetRelationName(state->rel),
 			 RelationGetRelationName(state->heaprel));
 
-		table_index_build_scan(state->heaprel, state->rel, indexinfo, true,
+		table_index_build_scan(state->heaprel, state->rel, indexinfo, true, false,
 							   bt_tuple_present_callback, (void *) state, scan);
 
 		ereport(DEBUG1,
diff --git a/contrib/bloom/blinsert.c b/contrib/bloom/blinsert.c
index 1b8df7e1e84..48f35d39990 100644
--- a/contrib/bloom/blinsert.c
+++ b/contrib/bloom/blinsert.c
@@ -142,7 +142,7 @@ blbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	initCachedPage(&buildstate);
 
 	/* Do the heap scan */
-	reltuples = table_index_build_scan(heap, index, indexInfo, true,
+	reltuples = table_index_build_scan(heap, index, indexInfo, true, false,
 									   bloomBuildCallback, (void *) &buildstate,
 									   NULL);
 
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index d078dfbd469..ee3bd562748 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -132,6 +132,7 @@ blhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = blcostestimate;
 	amroutine->amoptions = bloptions;
 	amroutine->amproperty = NULL;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = blvalidate;
 	amroutine->ambeginscan = blbeginscan;
 	amroutine->amrescan = blrescan;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index b56d3b3daa1..ff8290da9ff 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -127,6 +127,7 @@ typedef struct IndexAmRoutine
     amcostestimate_function amcostestimate;
     amoptions_function amoptions;
     amproperty_function amproperty;     /* can be NULL */
+    ambuildphasename_function ambuildphasename;   /* can be NULL */
     amvalidate_function amvalidate;
     ambeginscan_function ambeginscan;
     amrescan_function amrescan;
@@ -468,6 +469,18 @@ amproperty (Oid index_oid, int attno,
 
   <para>
 <programlisting>
+char *
+ambuildphasename (int64 phasenum);
+</programlisting>
+   Return the textual name of the given build phase number.
+   The phase numbers are those reported during an index build via the
+   <function>pgstat_progress_update_param</function> interface.
+   The phase names are then exposed in the
+   <structname>pg_stat_progress_create_index</structname> view.
+  </para>
+
+  <para>
+<programlisting>
 bool
 amvalidate (Oid opclassoid);
 </programlisting>
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index f1df14bdea8..8f5bbbb79e5 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -336,6 +336,14 @@ postgres   27093  0.0  0.0  30096  2752 ?        Ss   11:34   0:00 postgres: ser
       </entry>
      </row>
 
+     <row>
+      <entry><structname>pg_stat_progress_create_index</structname><indexterm><primary>pg_stat_progress_create_index</primary></indexterm></entry>
+      <entry>One row for each backend running <command>CREATE INDEX</command>, showing
+      current progress.
+      See <xref linkend='create-index-progress-reporting'/>.
+     </entry>
+     </row>
+
      <row>
       <entry><structname>pg_stat_progress_vacuum</structname><indexterm><primary>pg_stat_progress_vacuum</primary></indexterm></entry>
       <entry>One row for each backend (including autovacuum worker processes) running
@@ -3403,10 +3411,224 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
   <para>
    <productname>PostgreSQL</productname> has the ability to report the progress of
    certain commands during command execution.  Currently, the only commands
-   which support progress reporting are <command>VACUUM</command> and
+   which support progress reporting are <command>CREATE INDEX</command>,
+   <command>VACUUM</command> and
    <command>CLUSTER</command>. This may be expanded in the future.
   </para>
 
+ <sect2 id="create-index-progress-reporting">
+  <title>CREATE INDEX Progress Reporting</title>
+
+  <para>
+   Whenever <command>CREATE INDEX</command> is running, the
+   <structname>pg_stat_progress_create_index</structname> view will contain
+   one row for each backend that is currently creating indexes.  The tables
+   below describe the information that will be reported and provide information
+   about how to interpret it.
+  </para>
+
+  <table id="pg-stat-progress-create-index-view" xreflabel="pg_stat_progress_create_index">
+   <title><structname>pg_stat_progress_create_index</structname> View</title>
+   <tgroup cols="3">
+    <thead>
+     <row>
+      <entry>Column</entry>
+      <entry>Type</entry>
+      <entry>Description</entry>
+     </row>
+    </thead>
+
+    <tbody>
+     <row>
+      <entry><structfield>pid</structfield></entry>
+      <entry><type>integer</type></entry>
+      <entry>Process ID of backend.</entry>
+     </row>
+     <row>
+      <entry><structfield>datid</structfield></entry>
+      <entry><type>oid</type></entry>
+      <entry>OID of the database to which this backend is connected.</entry>
+     </row>
+     <row>
+      <entry><structfield>datname</structfield></entry>
+      <entry><type>name</type></entry>
+      <entry>Name of the database to which this backend is connected.</entry>
+     </row>
+     <row>
+      <entry><structfield>relid</structfield></entry>
+      <entry><type>oid</type></entry>
+      <entry>OID of the table on which the index is being created.</entry>
+     </row>
+     <row>
+      <entry><structfield>phase</structfield></entry>
+      <entry><type>text</type></entry>
+      <entry>
+        Current processing phase of index creation.  See <xref linkend='create-index-phases'/>.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>lockers_total</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Total number of lockers to wait for, when applicable.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>lockers_done</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Number of lockers already waited for.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>current_locked_pid</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Process ID of the locker currently being waited for.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>blocks_total</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Total number of blocks to be processed in the current phase.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>blocks_done</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Number of blocks already processed in the current phase.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>tuples_total</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Total number of tuples to be processed in the current phase.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>tuples_done</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Number of tuples already processed in the current phase.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>partitions_total</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+       When creating an index on a partitioned, this column is set to the
+       total number of partitions on which the index is to be created.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>partitions_done</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+       When creating an index on a partitioned, this column is set to the
+       number of partitions on which the index has been completed.
+      </entry>
+     </row>
+    </tbody>
+   </tgroup>
+  </table>
+
+  <table id="create-index-phases">
+   <title>CREATE INDEX phases</title>
+   <tgroup cols="2">
+    <thead>
+     <row>
+      <entry>Phase</entry>
+      <entry>Description</entry>
+     </row>
+    </thead>
+    <tbody>
+     <row>
+      <entry><literal>initializing</literal></entry>
+      <entry>
+       <command>CREATE INDEX</command> is preparing to create the index.  This
+       phase is expected to be very brief.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>waiting for old snapshots</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions
+       that can potentially see the table to release their snapshots.
+       This phase is skipped when not in concurrent mode.
+       Columns <structname>lockers_total</structname>, <structname>lockers_done</structname>
+       and <structname>current_locker_pid</structname> contain the progress 
+       information for this phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>building index</literal></entry>
+      <entry>
+       The index is being built by the access method-specific code.  In this phase,
+       access methods that support progress reporting fill in their own progress data,
+       and the subphase is indicated in this column.  Typically,
+       <structname>blocks_total</structname> and <structname>blocks_done</structname>
+       will contain progress data, as well as potentially
+       <structname>tuples_total</structname> and <structname>tuples_done</structname>.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>waiting for writer snapshots</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions
+       that can potentially write into the table to release their snapshots.
+       This phase is skipped when not in concurrent mode.
+       Columns <structname>lockers_total</structname>, <structname>lockers_done</structname>
+       and <structname>current_locker_pid</structname> contain the progress 
+       information for this phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>validating index scan</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is scanning the index searching
+       for tuples that need to be validated.
+       This phase is skipped when not in concurrent mode.
+       Columns <structname>blocks_total</structname> (set to the total size of the index)
+       and <structname>blocks_done</structname> contain the progress information for this phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>sorting index scan results</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is sorting the output of the
+       previous phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>validating index heapscan</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is scanning the table
+       to validate the index tuples collected in the previous two phases.
+       This phase is skipped when not in concurrent mode.
+       Columns <structname>blocks_total</structname> (set to the total size of the table)
+       and <structname>blocks_done</structname> contain the progress information for this phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>waiting for reader snapshots</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions
+       that can potentially see the table to release their snapshots.  This
+       phase is skipped when not in concurrent mode.
+       Columns <structname>lockers_total</structname>, <structname>lockers_done</structname>
+       and <structname>current_locker_pid</structname> contain the progress 
+       information for this phase.
+      </entry>
+     </row>
+    </tbody>
+   </tgroup>
+  </table>
+
+ </sect2>
+
  <sect2 id="vacuum-progress-reporting">
   <title>VACUUM Progress Reporting</title>
 
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 6e96d24ca22..54273f754f6 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -112,6 +112,7 @@ brinhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = brincostestimate;
 	amroutine->amoptions = brinoptions;
 	amroutine->amproperty = NULL;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = brinvalidate;
 	amroutine->ambeginscan = brinbeginscan;
 	amroutine->amrescan = brinrescan;
@@ -719,7 +720,7 @@ brinbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	 * Now scan the relation.  No syncscan allowed here because we want the
 	 * heap blocks in physical order.
 	 */
-	reltuples = table_index_build_scan(heap, index, indexInfo, false,
+	reltuples = table_index_build_scan(heap, index, indexInfo, false, false,
 									   brinbuildCallback, (void *) state, NULL);
 
 	/* process the final batch */
@@ -1236,7 +1237,7 @@ summarize_range(IndexInfo *indexInfo, BrinBuildState *state, Relation heapRel,
 	 * cases.
 	 */
 	state->bs_currRangeStart = heapBlk;
-	table_index_build_range_scan(heapRel, state->bs_irel, indexInfo, false, true,
+	table_index_build_range_scan(heapRel, state->bs_irel, indexInfo, false, true, false,
 								 heapBlk, scanNumBlks,
 								 brinbuildCallback, (void *) state, NULL);
 
diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c
index b02f69b0dcb..b4b0213f76f 100644
--- a/src/backend/access/gin/gininsert.c
+++ b/src/backend/access/gin/gininsert.c
@@ -395,7 +395,7 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	 * Do the heap scan.  We disallow sync scan here because dataPlaceToPage
 	 * prefers to receive tuples in TID order.
 	 */
-	reltuples = table_index_build_scan(heap, index, indexInfo, false,
+	reltuples = table_index_build_scan(heap, index, indexInfo, false, false,
 									   ginBuildCallback, (void *) &buildstate,
 									   NULL);
 
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index afc20232ace..d2360eeafb0 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -64,6 +64,7 @@ ginhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = gincostestimate;
 	amroutine->amoptions = ginoptions;
 	amroutine->amproperty = NULL;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = ginvalidate;
 	amroutine->ambeginscan = ginbeginscan;
 	amroutine->amrescan = ginrescan;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 2fddb23496d..f44c922b5d6 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -86,6 +86,7 @@ gisthandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = gistcostestimate;
 	amroutine->amoptions = gistoptions;
 	amroutine->amproperty = gistproperty;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = gistvalidate;
 	amroutine->ambeginscan = gistbeginscan;
 	amroutine->amrescan = gistrescan;
diff --git a/src/backend/access/gist/gistbuild.c b/src/backend/access/gist/gistbuild.c
index 3652fde5bb1..771bd2962a7 100644
--- a/src/backend/access/gist/gistbuild.c
+++ b/src/backend/access/gist/gistbuild.c
@@ -205,7 +205,7 @@ gistbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	/*
 	 * Do the heap scan.
 	 */
-	reltuples = table_index_build_scan(heap, index, indexInfo, true,
+	reltuples = table_index_build_scan(heap, index, indexInfo, true, false,
 									   gistBuildCallback,
 									   (void *) &buildstate, NULL);
 
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 5cc12a17130..6b38dd9c0e9 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -83,6 +83,7 @@ hashhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = hashcostestimate;
 	amroutine->amoptions = hashoptions;
 	amroutine->amproperty = NULL;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = hashvalidate;
 	amroutine->ambeginscan = hashbeginscan;
 	amroutine->amrescan = hashrescan;
@@ -160,7 +161,7 @@ hashbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	buildstate.heapRel = heap;
 
 	/* do the heap scan */
-	reltuples = table_index_build_scan(heap, index, indexInfo, true,
+	reltuples = table_index_build_scan(heap, index, indexInfo, true, false,
 									   hashbuildCallback,
 									   (void *) &buildstate, NULL);
 
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 1e4394a665b..645b532b61c 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -27,7 +27,9 @@
 #include "access/xact.h"
 #include "catalog/catalog.h"
 #include "catalog/index.h"
+#include "commands/progress.h"
 #include "executor/executor.h"
+#include "pgstat.h"
 #include "storage/bufmgr.h"
 #include "storage/bufpage.h"
 #include "storage/lmgr.h"
@@ -38,6 +40,9 @@
 static const TableAmRoutine heapam_methods;
 
 
+static BlockNumber heapscan_get_blocks_done(HeapScanDesc hscan);
+
+
 /* ------------------------------------------------------------------------
  * Slot related callbacks for heap AM
  * ------------------------------------------------------------------------
@@ -529,6 +534,7 @@ heapam_index_build_range_scan(Relation heapRelation,
 							  IndexInfo *indexInfo,
 							  bool allow_sync,
 							  bool anyvisible,
+							  bool progress,
 							  BlockNumber start_blockno,
 							  BlockNumber numblocks,
 							  IndexBuildCallback callback,
@@ -549,6 +555,7 @@ heapam_index_build_range_scan(Relation heapRelation,
 	Snapshot	snapshot;
 	bool		need_unregister_snapshot = false;
 	TransactionId OldestXmin;
+	BlockNumber	previous_blkno = InvalidBlockNumber;
 	BlockNumber root_blkno = InvalidBlockNumber;
 	OffsetNumber root_offsets[MaxHeapTuplesPerPage];
 
@@ -619,6 +626,14 @@ heapam_index_build_range_scan(Relation heapRelation,
 									 NULL,	/* scan key */
 									 true,	/* buffer access strategy OK */
 									 allow_sync);	/* syncscan OK? */
+
+		if (progress)
+		{
+			hscan = (HeapScanDesc) scan;
+			Assert(hscan->rs_numblocks == InvalidBlockNumber);
+			pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL,
+										 hscan->rs_nblocks);
+		}
 	}
 	else
 	{
@@ -668,6 +683,19 @@ heapam_index_build_range_scan(Relation heapRelation,
 
 		CHECK_FOR_INTERRUPTS();
 
+		/* Report scan progress, if asked to. */
+		if (progress)
+		{
+			BlockNumber     blocks_done = heapscan_get_blocks_done(hscan);
+
+			if (blocks_done != previous_blkno)
+			{
+				pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
+											 blocks_done);
+				previous_blkno = blocks_done;
+			}
+		}
+
 		/*
 		 * When dealing with a HOT-chain of updated tuples, we want to index
 		 * the values of the live tuple (if any), but index it under the TID
@@ -1042,6 +1070,7 @@ heapam_index_validate_scan(Relation heapRelation,
 	BlockNumber root_blkno = InvalidBlockNumber;
 	OffsetNumber root_offsets[MaxHeapTuplesPerPage];
 	bool		in_index[MaxHeapTuplesPerPage];
+	BlockNumber	previous_blkno = InvalidBlockNumber;
 
 	/* state variables for the merge */
 	ItemPointer indexcursor = NULL;
@@ -1082,6 +1111,9 @@ heapam_index_validate_scan(Relation heapRelation,
 								 false);	/* syncscan not OK */
 	hscan = (HeapScanDesc) scan;
 
+	pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL,
+								 hscan->rs_nblocks);
+
 	/*
 	 * Scan all tuples matching the snapshot.
 	 */
@@ -1095,6 +1127,14 @@ heapam_index_validate_scan(Relation heapRelation,
 
 		state->htups += 1;
 
+		if ((previous_blkno == InvalidBlockNumber) ||
+			(hscan->rs_cblock != previous_blkno))
+		{
+			pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
+										 hscan->rs_cblock);
+			previous_blkno = hscan->rs_cblock;
+		}
+
 		/*
 		 * As commented in table_index_build_scan, we should index heap-only
 		 * tuples under the TIDs of their root tuples; so when we advance onto
@@ -1255,6 +1295,41 @@ heapam_index_validate_scan(Relation heapRelation,
 	indexInfo->ii_PredicateState = NULL;
 }
 
+/*
+ * Return the number of blocks that have been read by this scan since
+ * starting.  This is not 100% accurate: in a parallel scan, the workers
+ * can be concurrently reading blocks further ahead than what we report.
+ */
+static BlockNumber
+heapscan_get_blocks_done(HeapScanDesc hscan)
+{
+	BlockNumber		startblock;
+	BlockNumber		blocks_done;
+
+	if (hscan->rs_base.rs_parallel != NULL)
+	{
+		ParallelBlockTableScanDesc bpscan;
+
+		bpscan = (ParallelBlockTableScanDesc) hscan->rs_base.rs_parallel;
+		startblock = bpscan->phs_startblock;
+	}
+	else
+		startblock = hscan->rs_startblock;
+
+	/*
+	 * Might have wrapped around the end of the relation, if startblock was
+	 * not zero.
+	 */
+	if (hscan->rs_cblock > startblock)
+		blocks_done = hscan->rs_cblock - startblock;
+	else
+		blocks_done = hscan->rs_nblocks - startblock +
+			hscan->rs_cblock;
+
+	return blocks_done;
+}
+
+
 
 /* ------------------------------------------------------------------------
  * Definition of the heap table access method.
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index ac6f1eb3423..7370379c6a1 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -22,6 +22,7 @@
 #include "access/nbtxlog.h"
 #include "access/relscan.h"
 #include "access/xlog.h"
+#include "commands/progress.h"
 #include "commands/vacuum.h"
 #include "miscadmin.h"
 #include "nodes/execnodes.h"
@@ -133,6 +134,7 @@ bthandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = btcostestimate;
 	amroutine->amoptions = btoptions;
 	amroutine->amproperty = btproperty;
+	amroutine->ambuildphasename = btbuildphasename;
 	amroutine->amvalidate = btvalidate;
 	amroutine->ambeginscan = btbeginscan;
 	amroutine->amrescan = btrescan;
@@ -1021,6 +1023,10 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
 		if (needLock)
 			UnlockRelationForExtension(rel, ExclusiveLock);
 
+		if (info->report_progress)
+			pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL,
+										 num_pages);
+
 		/* Quit if we've scanned the whole relation */
 		if (blkno >= num_pages)
 			break;
@@ -1028,6 +1034,9 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
 		for (; blkno < num_pages; blkno++)
 		{
 			btvacuumpage(&vstate, blkno, blkno);
+			if (info->report_progress)
+				pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
+											 blkno);
 		}
 	}
 
diff --git a/src/backend/access/nbtree/nbtsort.c b/src/backend/access/nbtree/nbtsort.c
index a8a7b792672..b889ac6de79 100644
--- a/src/backend/access/nbtree/nbtsort.c
+++ b/src/backend/access/nbtree/nbtsort.c
@@ -66,6 +66,7 @@
 #include "access/xlog.h"
 #include "access/xloginsert.h"
 #include "catalog/index.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
 #include "pgstat.h"
 #include "storage/smgr.h"
@@ -298,7 +299,8 @@ static double _bt_parallel_heapscan(BTBuildState *buildstate,
 static void _bt_leader_participate_as_worker(BTBuildState *buildstate);
 static void _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2,
 						   BTShared *btshared, Sharedsort *sharedsort,
-						   Sharedsort *sharedsort2, int sortmem);
+						   Sharedsort *sharedsort2, int sortmem,
+						   bool progress);
 
 
 /*
@@ -394,6 +396,10 @@ _bt_spools_heapscan(Relation heap, Relation index, BTBuildState *buildstate,
 	/* Save as primary spool */
 	buildstate->spool = btspool;
 
+	/* Report heap scan phase started */
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+								 PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN);
+
 	/* Attempt to launch parallel worker scan when required */
 	if (indexInfo->ii_ParallelWorkers > 0)
 		_bt_begin_parallel(buildstate, indexInfo->ii_Concurrent,
@@ -480,13 +486,31 @@ _bt_spools_heapscan(Relation heap, Relation index, BTBuildState *buildstate,
 
 	/* Fill spool using either serial or parallel heap scan */
 	if (!buildstate->btleader)
-		reltuples = table_index_build_scan(heap, index, indexInfo, true,
+		reltuples = table_index_build_scan(heap, index, indexInfo, true, true,
 										   _bt_build_callback, (void *) buildstate,
 										   NULL);
 	else
 		reltuples = _bt_parallel_heapscan(buildstate,
 										  &indexInfo->ii_BrokenHotChain);
 
+	/*
+	 * Set the progress target for the next phase.  Reset the block number
+	 * values set by IndexBuildHeapScan
+	 */
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_TUPLES_TOTAL,
+			PROGRESS_SCAN_BLOCKS_TOTAL,
+			PROGRESS_SCAN_BLOCKS_DONE
+		};
+		const int64 val[] = {
+			buildstate->indtuples,
+			0, 0
+		};
+
+		pgstat_progress_update_multi_param(3, index, val);
+	}
+
 	/* okay, all heap tuples are spooled */
 	if (buildstate->spool2 && !buildstate->havedead)
 	{
@@ -535,9 +559,15 @@ _bt_leafbuild(BTSpool *btspool, BTSpool *btspool2)
 	}
 #endif							/* BTREE_BUILD_STATS */
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+								 PROGRESS_BTREE_PHASE_PERFORMSORT_1);
 	tuplesort_performsort(btspool->sortstate);
 	if (btspool2)
+	{
+		pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+									 PROGRESS_BTREE_PHASE_PERFORMSORT_2);
 		tuplesort_performsort(btspool2->sortstate);
+	}
 
 	wstate.heap = btspool->heap;
 	wstate.index = btspool->index;
@@ -554,6 +584,8 @@ _bt_leafbuild(BTSpool *btspool, BTSpool *btspool2)
 	wstate.btws_pages_written = 0;
 	wstate.btws_zeropage = NULL;	/* until needed */
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+								 PROGRESS_BTREE_PHASE_LEAF_LOAD);
 	_bt_load(&wstate, btspool, btspool2);
 }
 
@@ -1098,6 +1130,7 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
 	int			i,
 				keysz = IndexRelationGetNumberOfKeyAttributes(wstate->index);
 	SortSupport sortKeys;
+	long		tuples_done = 0L;
 
 	if (merge)
 	{
@@ -1202,6 +1235,10 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
 				_bt_buildadd(wstate, state, itup2);
 				itup2 = tuplesort_getindextuple(btspool2->sortstate, true);
 			}
+
+			/* Report progress */
+			pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE,
+										 ++tuples_done);
 		}
 		pfree(sortKeys);
 	}
@@ -1216,6 +1253,10 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
 				state = _bt_pagestate(wstate, 0);
 
 			_bt_buildadd(wstate, state, itup);
+
+			/* Report progress */
+			pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE,
+										 ++tuples_done);
 		}
 	}
 
@@ -1352,6 +1393,11 @@ _bt_begin_parallel(BTBuildState *buildstate, bool isconcurrent, int request)
 								  ParallelTableScanFromBTShared(btshared),
 								  snapshot);
 
+	/* Report total number of blocks to scan */
+	pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL,
+								 ((ParallelBlockTableScanDescData *)
+								 ParallelTableScanFromBTShared(btshared))->phs_nblocks);
+
 	/*
 	 * Store shared tuplesort-private state, for which we reserved space.
 	 * Then, initialize opaque state using tuplesort routine.
@@ -1528,7 +1574,7 @@ _bt_leader_participate_as_worker(BTBuildState *buildstate)
 	/* Perform work common to all participants */
 	_bt_parallel_scan_and_sort(leaderworker, leaderworker2, btleader->btshared,
 							   btleader->sharedsort, btleader->sharedsort2,
-							   sortmem);
+							   sortmem, true);
 
 #ifdef BTREE_BUILD_STATS
 	if (log_btree_build_stats)
@@ -1619,7 +1665,7 @@ _bt_parallel_build_main(dsm_segment *seg, shm_toc *toc)
 	/* Perform sorting of spool, and possibly a spool2 */
 	sortmem = maintenance_work_mem / btshared->scantuplesortstates;
 	_bt_parallel_scan_and_sort(btspool, btspool2, btshared, sharedsort,
-							   sharedsort2, sortmem);
+							   sharedsort2, sortmem, false);
 
 #ifdef BTREE_BUILD_STATS
 	if (log_btree_build_stats)
@@ -1648,7 +1694,7 @@ _bt_parallel_build_main(dsm_segment *seg, shm_toc *toc)
 static void
 _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2,
 						   BTShared *btshared, Sharedsort *sharedsort,
-						   Sharedsort *sharedsort2, int sortmem)
+						   Sharedsort *sharedsort2, int sortmem, bool progress)
 {
 	SortCoordinate coordinate;
 	BTBuildState buildstate;
@@ -1705,9 +1751,10 @@ _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2,
 	/* Join parallel scan */
 	indexInfo = BuildIndexInfo(btspool->index);
 	indexInfo->ii_Concurrent = btshared->isconcurrent;
-	scan = table_beginscan_parallel(btspool->heap, ParallelTableScanFromBTShared(btshared));
+	scan = table_beginscan_parallel(btspool->heap,
+									ParallelTableScanFromBTShared(btshared));
 	reltuples = table_index_build_scan(btspool->heap, btspool->index, indexInfo,
-									   true, _bt_build_callback,
+									   true, progress, _bt_build_callback,
 									   (void *) &buildstate, scan);
 
 	/*
diff --git a/src/backend/access/nbtree/nbtutils.c b/src/backend/access/nbtree/nbtutils.c
index 92b8b5f134d..9e8a7bd3d9a 100644
--- a/src/backend/access/nbtree/nbtutils.c
+++ b/src/backend/access/nbtree/nbtutils.c
@@ -20,6 +20,7 @@
 #include "access/nbtree.h"
 #include "access/reloptions.h"
 #include "access/relscan.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
 #include "utils/array.h"
 #include "utils/datum.h"
@@ -2048,6 +2049,29 @@ btproperty(Oid index_oid, int attno,
 	}
 }
 
+/*
+ *	btbuildphasename() -- Return name of index build phase.
+ */
+char *
+btbuildphasename(int64 phasenum)
+{
+	switch (phasenum)
+	{
+		case PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE:
+			return "initializing";
+		case PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN:
+			return "table scan";
+		case PROGRESS_BTREE_PHASE_PERFORMSORT_1:
+			return "sorting tuples, spool 1";
+		case PROGRESS_BTREE_PHASE_PERFORMSORT_2:
+			return "sorting tuples, spool 2";
+		case PROGRESS_BTREE_PHASE_LEAF_LOAD:
+			return "btree tuple loading";
+		default:
+			return NULL;
+	}
+}
+
 /*
  *	_bt_truncate() -- create tuple without unneeded suffix attributes.
  *
diff --git a/src/backend/access/spgist/spginsert.c b/src/backend/access/spgist/spginsert.c
index 390ad9ac51f..282d6998cf0 100644
--- a/src/backend/access/spgist/spginsert.c
+++ b/src/backend/access/spgist/spginsert.c
@@ -143,7 +143,7 @@ spgbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 											  "SP-GiST build temporary context",
 											  ALLOCSET_DEFAULT_SIZES);
 
-	reltuples = table_index_build_scan(heap, index, indexInfo, true,
+	reltuples = table_index_build_scan(heap, index, indexInfo, true, false,
 									   spgistBuildCallback, (void *) &buildstate,
 									   NULL);
 
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 8e63c1fad25..45472db147b 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -67,6 +67,7 @@ spghandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = spgcostestimate;
 	amroutine->amoptions = spgoptions;
 	amroutine->amproperty = spgproperty;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = spgvalidate;
 	amroutine->ambeginscan = spgbeginscan;
 	amroutine->amrescan = spgrescan;
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 104a8cceb78..69ef1eb7bce 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -50,9 +50,9 @@
 #include "catalog/pg_trigger.h"
 #include "catalog/pg_type.h"
 #include "catalog/storage.h"
-#include "commands/tablecmds.h"
 #include "commands/event_trigger.h"
 #include "commands/progress.h"
+#include "commands/tablecmds.h"
 #include "commands/trigger.h"
 #include "executor/executor.h"
 #include "miscadmin.h"
@@ -1579,7 +1579,7 @@ index_drop(Oid indexId, bool concurrent)
 		 * to acquire an exclusive lock on our table.  The lock code will
 		 * detect deadlock and error out properly.
 		 */
-		WaitForLockers(heaplocktag, AccessExclusiveLock);
+		WaitForLockers(heaplocktag, AccessExclusiveLock, true);
 
 		/*
 		 * No more predicate locks will be acquired on this index, and we're
@@ -1623,7 +1623,7 @@ index_drop(Oid indexId, bool concurrent)
 		 * Wait till every transaction that saw the old index state has
 		 * finished.
 		 */
-		WaitForLockers(heaplocktag, AccessExclusiveLock);
+		WaitForLockers(heaplocktag, AccessExclusiveLock, true);
 
 		/*
 		 * Re-open relations to allow us to complete our actions.
@@ -2272,6 +2272,25 @@ index_build(Relation heapRelation,
 						   save_sec_context | SECURITY_RESTRICTED_OPERATION);
 	save_nestlevel = NewGUCNestLevel();
 
+	/* Set up initial progress report status */
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_PHASE,
+			PROGRESS_CREATEIDX_SUBPHASE,
+			PROGRESS_CREATEIDX_TUPLES_DONE,
+			PROGRESS_CREATEIDX_TUPLES_TOTAL,
+			PROGRESS_SCAN_BLOCKS_DONE,
+			PROGRESS_SCAN_BLOCKS_TOTAL
+		};
+		const int64	val[] = {
+			PROGRESS_CREATEIDX_PHASE_BUILD,
+			PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE,
+			0, 0, 0, 0
+		};
+
+		pgstat_progress_update_multi_param(6, index, val);
+	}
+
 	/*
 	 * Call the access method's build procedure
 	 */
@@ -2560,6 +2579,21 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot)
 	int			save_sec_context;
 	int			save_nestlevel;
 
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_PHASE,
+			PROGRESS_CREATEIDX_TUPLES_DONE,
+			PROGRESS_CREATEIDX_TUPLES_TOTAL,
+			PROGRESS_SCAN_BLOCKS_DONE,
+			PROGRESS_SCAN_BLOCKS_TOTAL
+		};
+		const int64	val[] = {
+			PROGRESS_CREATEIDX_PHASE_VALIDATE_IDXSCAN,
+			0, 0, 0, 0
+		};
+		pgstat_progress_update_multi_param(5, index, val);
+	}
+
 	/* Open and lock the parent heap relation */
 	heapRelation = table_open(heapId, ShareUpdateExclusiveLock);
 	/* And the target index relation */
@@ -2590,6 +2624,7 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot)
 	 */
 	ivinfo.index = indexRelation;
 	ivinfo.analyze_only = false;
+	ivinfo.report_progress = true;	/* XXX only for btree? */
 	ivinfo.estimated_count = true;
 	ivinfo.message_level = DEBUG2;
 	ivinfo.num_heap_tuples = heapRelation->rd_rel->reltuples;
@@ -2607,15 +2642,39 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot)
 											NULL, false);
 	state.htups = state.itups = state.tups_inserted = 0;
 
+	/* ambulkdelete updates progress metrics */
 	(void) index_bulk_delete(&ivinfo, NULL,
 							 validate_index_callback, (void *) &state);
 
 	/* Execute the sort */
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_PHASE,
+			PROGRESS_SCAN_BLOCKS_DONE,
+			PROGRESS_SCAN_BLOCKS_TOTAL
+		};
+		const int64	val[] = {
+			PROGRESS_CREATEIDX_PHASE_SORT_IDXSCAN,
+			0, 0
+		};
+
+		pgstat_progress_update_multi_param(3, index, val);
+	}
 	tuplesort_performsort(state.tuplesort);
 
 	/*
-	 * Now scan the heap and "merge" it with the index
+	 * Now scan the heap and "merge" it with the index.
 	 */
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_PHASE,
+		};
+		const int64	val[] = {
+			PROGRESS_CREATEIDX_PHASE_HEAPSCAN_VALIDATE
+		};
+
+		pgstat_progress_update_multi_param(1, index, val);
+	}
 	table_index_validate_scan(heapRelation,
 							  indexRelation,
 							  indexInfo,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index b89df70653e..b7c040d5284 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -934,6 +934,33 @@ CREATE VIEW pg_stat_progress_cluster AS
     FROM pg_stat_get_progress_info('CLUSTER') AS S
         LEFT JOIN pg_database D ON S.datid = D.oid;
 
+CREATE VIEW pg_stat_progress_create_index AS
+	SELECT
+		S.pid AS pid, S.datid AS datid, D.datname AS datname,
+		S.relid AS relid,
+		CASE S.param2 WHEN 0 THEN 'initializing'
+					  WHEN 1 THEN 'waiting for old snapshots'
+					  WHEN 2 THEN 'building index' ||
+						COALESCE((': ' || pg_indexam_progress_phasename(S.param1::oid, S.param3)),
+							'')
+					  WHEN 3 THEN 'waiting for writer snapshots'
+					  WHEN 4 THEN 'index validation: scan index'
+					  WHEN 5 THEN 'index validation: sort index scan results'
+					  WHEN 6 THEN 'index validation: scan heap'
+					  WHEN 7 THEN 'waiting for reader snapshots'
+					  END as phase,
+		S.param4 AS lockers_total,
+		S.param5 AS lockers_done,
+		S.param6 AS current_locker_pid,
+		S.param7 AS blocks_total,
+		S.param8 AS blocks_done,
+		S.param9 AS tuples_total,
+		S.param10 AS tuples_done,
+		S.param11 AS partitions_total,
+		S.param12 AS partitions_done
+	FROM pg_stat_get_progress_info('CREATE INDEX') AS S
+		LEFT JOIN pg_database D ON S.datid = D.oid;
+
 CREATE VIEW pg_user_mappings AS
     SELECT
         U.oid       AS umid,
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index d6eb48cb4e6..10c6f716455 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -36,6 +36,7 @@
 #include "commands/dbcommands.h"
 #include "commands/defrem.h"
 #include "commands/event_trigger.h"
+#include "commands/progress.h"
 #include "commands/tablecmds.h"
 #include "commands/tablespace.h"
 #include "mb/pg_wchar.h"
@@ -47,10 +48,12 @@
 #include "parser/parse_func.h"
 #include "parser/parse_oper.h"
 #include "partitioning/partdesc.h"
+#include "pgstat.h"
 #include "rewrite/rewriteManip.h"
 #include "storage/lmgr.h"
 #include "storage/proc.h"
 #include "storage/procarray.h"
+#include "storage/sinvaladt.h"
 #include "utils/acl.h"
 #include "utils/builtins.h"
 #include "utils/fmgroids.h"
@@ -369,6 +372,15 @@ DefineIndex(Oid relationId,
 	Snapshot	snapshot;
 	int			i;
 
+
+	/*
+	 * Start progress report.  If we're building a partition, this was already
+	 * done.
+	 */
+	if (!OidIsValid(parentIndexId))
+		pgstat_progress_start_command(PROGRESS_COMMAND_CREATE_INDEX,
+									  relationId);
+
 	/*
 	 * count key attributes in index
 	 */
@@ -585,6 +597,9 @@ DefineIndex(Oid relationId,
 	accessMethodId = accessMethodForm->oid;
 	amRoutine = GetIndexAmRoutine(accessMethodForm->amhandler);
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_ACCESS_METHOD_OID,
+								 accessMethodId);
+
 	if (stmt->unique && !amRoutine->amcanunique)
 		ereport(ERROR,
 				(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -865,6 +880,11 @@ DefineIndex(Oid relationId,
 	if (!OidIsValid(indexRelationId))
 	{
 		table_close(rel, NoLock);
+
+		/* If this is the top-level index, we're done */
+		if (!OidIsValid(parentIndexId))
+			pgstat_progress_end_command();
+
 		return address;
 	}
 
@@ -890,6 +910,9 @@ DefineIndex(Oid relationId,
 			TupleDesc	parentDesc;
 			Oid		   *opfamOids;
 
+			pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL,
+										 nparts);
+
 			memcpy(part_oids, partdesc->oids, sizeof(Oid) * nparts);
 
 			parentDesc = CreateTupleDescCopy(RelationGetDescr(rel));
@@ -1039,6 +1062,8 @@ DefineIndex(Oid relationId,
 								skip_build, quiet);
 				}
 
+				pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_DONE,
+											 i + 1);
 				pfree(attmap);
 			}
 
@@ -1073,6 +1098,8 @@ DefineIndex(Oid relationId,
 		 * Indexes on partitioned tables are not themselves built, so we're
 		 * done here.
 		 */
+		if (!OidIsValid(parentIndexId))
+			pgstat_progress_end_command();
 		return address;
 	}
 
@@ -1080,6 +1107,11 @@ DefineIndex(Oid relationId,
 	{
 		/* Close the heap and we're done, in the non-concurrent case */
 		table_close(rel, NoLock);
+
+		/* If this is the top-level index, we're done. */
+		if (!OidIsValid(parentIndexId))
+			pgstat_progress_end_command();
+
 		return address;
 	}
 
@@ -1131,7 +1163,9 @@ DefineIndex(Oid relationId,
 	 * exclusive lock on our table.  The lock code will detect deadlock and
 	 * error out properly.
 	 */
-	WaitForLockers(heaplocktag, ShareLock);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_1);
+	WaitForLockers(heaplocktag, ShareLock, true);
 
 	/*
 	 * At this moment we are sure that there are no transactions with the
@@ -1195,7 +1229,9 @@ DefineIndex(Oid relationId,
 	 * We once again wait until no transaction can have the table open with
 	 * the index marked as read-only for updates.
 	 */
-	WaitForLockers(heaplocktag, ShareLock);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_2);
+	WaitForLockers(heaplocktag, ShareLock, true);
 
 	/*
 	 * Now take the "reference snapshot" that will be used by validate_index()
@@ -1281,6 +1317,9 @@ DefineIndex(Oid relationId,
 	old_snapshots = GetCurrentVirtualXIDs(limitXmin, true, false,
 										  PROC_IS_AUTOVACUUM | PROC_IN_VACUUM,
 										  &n_old_snapshots);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_3);
+	pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, n_old_snapshots);
 
 	for (i = 0; i < n_old_snapshots; i++)
 	{
@@ -1316,7 +1355,14 @@ DefineIndex(Oid relationId,
 		}
 
 		if (VirtualTransactionIdIsValid(old_snapshots[i]))
+		{
+			PGPROC *holder = BackendIdGetProc(old_snapshots[i].backendId);
+			pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID,
+										 holder->pid);
 			VirtualXactLock(old_snapshots[i], true);
+		}
+
+		pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, i + 1);
 	}
 
 	/*
@@ -1339,6 +1385,8 @@ DefineIndex(Oid relationId,
 	 */
 	UnlockRelationIdForSession(&heaprelid, ShareUpdateExclusiveLock);
 
+	pgstat_progress_end_command();
+
 	return address;
 }
 
diff --git a/src/backend/storage/ipc/standby.c b/src/backend/storage/ipc/standby.c
index cd56dca3aef..215f1463bb1 100644
--- a/src/backend/storage/ipc/standby.c
+++ b/src/backend/storage/ipc/standby.c
@@ -401,7 +401,7 @@ ResolveRecoveryConflictWithLock(LOCKTAG locktag)
 		 */
 		VirtualTransactionId *backends;
 
-		backends = GetLockConflicts(&locktag, AccessExclusiveLock);
+		backends = GetLockConflicts(&locktag, AccessExclusiveLock, NULL);
 		ResolveRecoveryConflictWithVirtualXIDs(backends,
 											   PROCSIG_RECOVERY_CONFLICT_LOCK);
 	}
diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c
index e688ba81170..0b04b093782 100644
--- a/src/backend/storage/lmgr/lmgr.c
+++ b/src/backend/storage/lmgr/lmgr.c
@@ -19,9 +19,12 @@
 #include "access/transam.h"
 #include "access/xact.h"
 #include "catalog/catalog.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
+#include "pgstat.h"
 #include "storage/lmgr.h"
 #include "storage/procarray.h"
+#include "storage/sinvaladt.h"
 #include "utils/inval.h"
 
 
@@ -857,10 +860,12 @@ XactLockTableWaitErrorCb(void *arg)
  * after we obtained our initial list of lockers, we will not wait for them.
  */
 void
-WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
+WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
 {
 	List	   *holders = NIL;
 	ListCell   *lc;
+	int			total = 0;
+	int			done = 0;
 
 	/* Done if no locks to wait for */
 	if (list_length(locktags) == 0)
@@ -870,10 +875,17 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
 	foreach(lc, locktags)
 	{
 		LOCKTAG    *locktag = lfirst(lc);
+		int			count;
 
-		holders = lappend(holders, GetLockConflicts(locktag, lockmode));
+		holders = lappend(holders,
+						  GetLockConflicts(locktag, lockmode,
+										   progress ? &count : NULL));
+		total += count;
 	}
 
+	if (progress)
+		pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, total);
+
 	/*
 	 * Note: GetLockConflicts() never reports our own xid, hence we need not
 	 * check for that.  Also, prepared xacts are not reported, which is fine
@@ -887,10 +899,36 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
 
 		while (VirtualTransactionIdIsValid(*lockholders))
 		{
+			/*
+			 * If requested, publish who we're going to wait for.  This is not
+			 * 100% accurate if they're already gone, but we don't care.
+			 */
+			if (progress)
+			{
+				PGPROC *holder = BackendIdGetProc(lockholders->backendId);
+
+				pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID,
+											 holder->pid);
+			}
 			VirtualXactLock(*lockholders, true);
 			lockholders++;
+
+			if (progress)
+				pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, ++done);
 		}
 	}
+	if (progress)
+	{
+		const int	index[] = {
+			PROGRESS_WAITFOR_TOTAL,
+			PROGRESS_WAITFOR_DONE,
+			PROGRESS_WAITFOR_CURRENT_PID
+		};
+		const int64	values[] = {
+			0, 0, 0
+		};
+		pgstat_progress_update_multi_param(3, index, values);
+	}
 
 	list_free_deep(holders);
 }
@@ -901,12 +939,12 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
  * Same as WaitForLockersMultiple, for a single lock tag.
  */
 void
-WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode)
+WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress)
 {
 	List	   *l;
 
 	l = list_make1(&heaplocktag);
-	WaitForLockersMultiple(l, lockmode);
+	WaitForLockersMultiple(l, lockmode, progress);
 	list_free(l);
 }
 
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 78fdbd6ff88..c8958766f1e 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2807,6 +2807,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock)
  *		xacts merely awaiting such a lock are NOT reported.
  *
  * The result array is palloc'd and is terminated with an invalid VXID.
+ * *countp, if not null, is updated to the number of items set.
  *
  * Of course, the result could be out of date by the time it's returned,
  * so use of this function has to be thought about carefully.
@@ -2817,7 +2818,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock)
  * uses of the result.
  */
 VirtualTransactionId *
-GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode)
+GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
 {
 	static VirtualTransactionId *vxids;
 	LOCKMETHODID lockmethodid = locktag->locktag_lockmethodid;
@@ -2964,6 +2965,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode)
 		LWLockRelease(partitionLock);
 		vxids[count].backendId = InvalidBackendId;
 		vxids[count].localTransactionId = InvalidLocalTransactionId;
+		if (countp)
+			*countp = count;
 		return vxids;
 	}
 
@@ -3019,6 +3022,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode)
 
 	vxids[count].backendId = InvalidBackendId;
 	vxids[count].localTransactionId = InvalidLocalTransactionId;
+	if (countp)
+		*countp = count;
 	return vxids;
 }
 
diff --git a/src/backend/utils/adt/amutils.c b/src/backend/utils/adt/amutils.c
index 060ffe501ec..e81d6cc0562 100644
--- a/src/backend/utils/adt/amutils.c
+++ b/src/backend/utils/adt/amutils.c
@@ -445,3 +445,26 @@ pg_index_column_has_property(PG_FUNCTION_ARGS)
 
 	return indexam_property(fcinfo, propname, InvalidOid, relid, attno);
 }
+
+/*
+ * Return the name of the given phase, as used for progress reporting by the
+ * given AM.
+ */
+Datum
+pg_indexam_progress_phasename(PG_FUNCTION_ARGS)
+{
+	Oid			amoid = PG_GETARG_OID(0);
+	int32		phasenum = PG_GETARG_INT32(1);
+	IndexAmRoutine *routine;
+	char	   *name;
+
+	routine = GetIndexAmRoutineByAmId(amoid, true);
+	if (routine == NULL || !routine->ambuildphasename)
+		PG_RETURN_NULL();
+
+	name = routine->ambuildphasename(phasenum);
+	if (!name)
+		PG_RETURN_NULL();
+
+	PG_RETURN_TEXT_P(CStringGetTextDatum(name));
+}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 90a817a25c5..7c2afe64272 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -470,6 +470,8 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
 		cmdtype = PROGRESS_COMMAND_VACUUM;
 	else if (pg_strcasecmp(cmd, "CLUSTER") == 0)
 		cmdtype = PROGRESS_COMMAND_CLUSTER;
+	else if (pg_strcasecmp(cmd, "CREATE INDEX") == 0)
+		cmdtype = PROGRESS_COMMAND_CREATE_INDEX;
 	else
 		ereport(ERROR,
 				(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 653ddc976ba..09a7404267c 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -108,6 +108,9 @@ typedef bool (*amproperty_function) (Oid index_oid, int attno,
 									 IndexAMProperty prop, const char *propname,
 									 bool *res, bool *isnull);
 
+/* name of phase as used in progress reporting */
+typedef char *(*ambuildphasename_function) (int64 phasenum);
+
 /* validate definition of an opclass for this AM */
 typedef bool (*amvalidate_function) (Oid opclassoid);
 
@@ -213,6 +216,7 @@ typedef struct IndexAmRoutine
 	amcostestimate_function amcostestimate;
 	amoptions_function amoptions;
 	amproperty_function amproperty; /* can be NULL */
+	ambuildphasename_function ambuildphasename;	/* can be NULL */
 	amvalidate_function amvalidate;
 	ambeginscan_function ambeginscan;
 	amrescan_function amrescan;
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 70c7351a08c..9717183ef23 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -45,6 +45,7 @@ typedef struct IndexVacuumInfo
 {
 	Relation	index;			/* the index being vacuumed */
 	bool		analyze_only;	/* ANALYZE (without any actual vacuum) */
+	bool		report_progress;	/* emit progress.h status reports */
 	bool		estimated_count;	/* num_heap_tuples is an estimate */
 	int			message_level;	/* ereport level for progress messages */
 	double		num_heap_tuples;	/* tuples remaining in heap */
diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h
index 473c6f29185..bc0f3027629 100644
--- a/src/include/access/nbtree.h
+++ b/src/include/access/nbtree.h
@@ -672,6 +672,16 @@ typedef BTScanOpaqueData *BTScanOpaque;
 #define SK_BT_DESC			(INDOPTION_DESC << SK_BT_INDOPTION_SHIFT)
 #define SK_BT_NULLS_FIRST	(INDOPTION_NULLS_FIRST << SK_BT_INDOPTION_SHIFT)
 
+/*
+ * Constant definition for progress reporting.  Phase numbers must match
+ * btbuildphasename.
+ */
+/* PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE is 1 (see progress.h) */
+#define PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN		2
+#define PROGRESS_BTREE_PHASE_PERFORMSORT_1				3
+#define PROGRESS_BTREE_PHASE_PERFORMSORT_2				4
+#define PROGRESS_BTREE_PHASE_LEAF_LOAD					5
+
 /*
  * external entry points for btree, in nbtree.c
  */
@@ -785,6 +795,7 @@ extern bytea *btoptions(Datum reloptions, bool validate);
 extern bool btproperty(Oid index_oid, int attno,
 		   IndexAMProperty prop, const char *propname,
 		   bool *res, bool *isnull);
+extern char *btbuildphasename(int64 phasenum);
 extern IndexTuple _bt_truncate(Relation rel, IndexTuple lastleft,
 			 IndexTuple firstright, BTScanInsert itup_key);
 extern int _bt_keep_natts_fast(Relation rel, IndexTuple lastleft,
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index 2546d3005fb..9579006d447 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -384,6 +384,7 @@ typedef struct TableAmRoutine
 										   struct IndexInfo *index_nfo,
 										   bool allow_sync,
 										   bool anyvisible,
+										   bool progress,
 										   BlockNumber start_blockno,
 										   BlockNumber end_blockno,
 										   IndexBuildCallback callback,
@@ -978,6 +979,11 @@ table_lock_tuple(Relation rel, ItemPointer tid, Snapshot snapshot,
  * so here because the AM might reject some of the tuples for its own reasons,
  * such as being unable to store NULLs.
  *
+ * If 'progress', we update the PROGRESS_SCAN_BLOCKS_DONE counter as we go
+ * along.  Also, if that flag is true and a scan descriptor is not passed (ie.
+ * when not doing a parallel scan), the PROGRESS_SCAN_BLOCKS_TOTAL counter is
+ * updated at the beginning.  For parallel scans, caller is expected to have
+ * set the total number of blocks prior to calling this function.
  *
  * A side effect is to set indexInfo->ii_BrokenHotChain to true if we detect
  * any potentially broken HOT chains.  Currently, we set this if there are any
@@ -991,6 +997,7 @@ table_index_build_scan(Relation heap_rel,
 					   Relation index_rel,
 					   struct IndexInfo *index_nfo,
 					   bool allow_sync,
+					   bool progress,
 					   IndexBuildCallback callback,
 					   void *callback_state,
 					   TableScanDesc scan)
@@ -1000,6 +1007,7 @@ table_index_build_scan(Relation heap_rel,
 														index_nfo,
 														allow_sync,
 														false,
+														progress,
 														0,
 														InvalidBlockNumber,
 														callback,
@@ -1023,6 +1031,7 @@ table_index_build_range_scan(Relation heap_rel,
 							 struct IndexInfo *index_nfo,
 							 bool allow_sync,
 							 bool anyvisible,
+							 bool progress,
 							 BlockNumber start_blockno,
 							 BlockNumber numblocks,
 							 IndexBuildCallback callback,
@@ -1034,6 +1043,7 @@ table_index_build_range_scan(Relation heap_rel,
 														index_nfo,
 														allow_sync,
 														anyvisible,
+														progress,
 														start_blockno,
 														numblocks,
 														callback,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index eac909109c5..fcd6d743bea 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -924,6 +924,10 @@
   proname => 'pg_index_column_has_property', provolatile => 's',
   prorettype => 'bool', proargtypes => 'regclass int4 text',
   prosrc => 'pg_index_column_has_property' },
+{ oid => '676', descr => 'return name of given index build phase',
+  proname => 'pg_indexam_progress_phasename', provolatile => 'i',
+  prorettype => 'text', proargtypes => 'oid int8',
+  prosrc => 'pg_indexam_progress_phasename' },
 
 { oid => '339',
   proname => 'poly_same', prorettype => 'bool',
@@ -5122,9 +5126,9 @@
   proname => 'pg_stat_get_progress_info', prorows => '100', proretset => 't',
   provolatile => 's', proparallel => 'r', prorettype => 'record',
   proargtypes => 'text',
-  proallargtypes => '{text,int4,oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}',
-  proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o}',
-  proargnames => '{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10}',
+  proallargtypes => '{text,int4,oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}',
+  proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+  proargnames => '{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10,param11,param12}',
   prosrc => 'pg_stat_get_progress_info' },
 { oid => '3099',
   descr => 'statistics: information about currently active replication',
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index 04542d9e923..45847f6dc35 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -57,4 +57,39 @@
 #define PROGRESS_CLUSTER_COMMAND_CLUSTER		1
 #define PROGRESS_CLUSTER_COMMAND_VACUUM_FULL	2
 
+/* Progress parameters for CREATE INDEX */
+#define PROGRESS_CREATEIDX_ACCESS_METHOD_OID	0
+#define PROGRESS_CREATEIDX_PHASE				1	/* AM-agnostic phase # */
+#define PROGRESS_CREATEIDX_SUBPHASE				2	/* phase # filled by AM */
+/* 3, 4 and 5 reserved for "waitfor" metrics */
+/* 6 and 7 reserved for "block number" metrics */
+#define PROGRESS_CREATEIDX_TUPLES_TOTAL			8
+#define PROGRESS_CREATEIDX_TUPLES_DONE			9
+#define PROGRESS_CREATEIDX_PARTITIONS_TOTAL		10
+#define PROGRESS_CREATEIDX_PARTITIONS_DONE		11
+
+/* Phases of CREATE INDEX */
+#define PROGRESS_CREATEIDX_PHASE_WAIT_1			1
+#define PROGRESS_CREATEIDX_PHASE_BUILD			2
+#define PROGRESS_CREATEIDX_PHASE_WAIT_2			3
+#define PROGRESS_CREATEIDX_PHASE_VALIDATE_IDXSCAN		4
+#define PROGRESS_CREATEIDX_PHASE_SORT_IDXSCAN		5
+#define PROGRESS_CREATEIDX_PHASE_HEAPSCAN_VALIDATE		6
+#define PROGRESS_CREATEIDX_PHASE_WAIT_3			7
+
+/*
+ * Subphases of CREATE INDEX, for index_build.
+ */
+#define PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE	1
+/* Additional phases are defined by each AM */
+
+/* Lock holder wait counts */
+#define PROGRESS_WAITFOR_TOTAL					3
+#define PROGRESS_WAITFOR_DONE					4
+#define PROGRESS_WAITFOR_CURRENT_PID			5
+
+/* Block numbers in a generic relation scan */
+#define PROGRESS_SCAN_BLOCKS_TOTAL				6
+#define PROGRESS_SCAN_BLOCKS_DONE				7
+
 #endif
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index c080fa6388f..6af9ab7e1d9 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -951,10 +951,11 @@ typedef enum ProgressCommandType
 {
 	PROGRESS_COMMAND_INVALID,
 	PROGRESS_COMMAND_VACUUM,
-	PROGRESS_COMMAND_CLUSTER
+	PROGRESS_COMMAND_CLUSTER,
+	PROGRESS_COMMAND_CREATE_INDEX
 } ProgressCommandType;
 
-#define PGSTAT_NUM_PROGRESS_PARAM	10
+#define PGSTAT_NUM_PROGRESS_PARAM	12
 
 /* ----------
  * Shared-memory data structures
diff --git a/src/include/storage/lmgr.h b/src/include/storage/lmgr.h
index 3d705faba5c..4f2872de35f 100644
--- a/src/include/storage/lmgr.h
+++ b/src/include/storage/lmgr.h
@@ -78,8 +78,8 @@ extern void XactLockTableWait(TransactionId xid, Relation rel,
 extern bool ConditionalXactLockTableWait(TransactionId xid);
 
 /* Lock VXIDs, specified by conflicting locktags */
-extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode);
-extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode);
+extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress);
+extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress);
 
 /* Lock an XID for tuple insertion (used to wait for an insertion to finish) */
 extern uint32 SpeculativeInsertionLockAcquire(TransactionId xid);
diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h
index badf7fd682b..048947c50d4 100644
--- a/src/include/storage/lock.h
+++ b/src/include/storage/lock.h
@@ -544,7 +544,7 @@ extern bool LockHeldByMe(const LOCKTAG *locktag, LOCKMODE lockmode);
 extern bool LockHasWaiters(const LOCKTAG *locktag,
 			   LOCKMODE lockmode, bool sessionLock);
 extern VirtualTransactionId *GetLockConflicts(const LOCKTAG *locktag,
-				 LOCKMODE lockmode);
+				 LOCKMODE lockmode, int *countp);
 extern void AtPrepare_Locks(void);
 extern void PostPrepare_Locks(TransactionId xid);
 extern int LockCheckConflicts(LockMethod lockMethodTable,
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index d5f309fbfbe..a9a2910fb55 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1856,7 +1856,33 @@ pg_stat_progress_cluster| SELECT s.pid,
     s.param6 AS heap_blks_total,
     s.param7 AS heap_blks_scanned,
     s.param8 AS index_rebuild_count
-   FROM (pg_stat_get_progress_info('CLUSTER'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10)
+   FROM (pg_stat_get_progress_info('CLUSTER'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12)
+     LEFT JOIN pg_database d ON ((s.datid = d.oid)));
+pg_stat_progress_create_index| SELECT s.pid,
+    s.datid,
+    d.datname,
+    s.relid,
+        CASE s.param2
+            WHEN 0 THEN 'initializing'::text
+            WHEN 1 THEN 'waiting for old snapshots'::text
+            WHEN 2 THEN ('building index'::text || COALESCE((': '::text || pg_indexam_progress_phasename((s.param1)::oid, s.param3)), ''::text))
+            WHEN 3 THEN 'waiting for writer snapshots'::text
+            WHEN 4 THEN 'index validation: scan index'::text
+            WHEN 5 THEN 'index validation: sort index scan results'::text
+            WHEN 6 THEN 'index validation: scan heap'::text
+            WHEN 7 THEN 'waiting for reader snapshots'::text
+            ELSE NULL::text
+        END AS phase,
+    s.param4 AS lockers_total,
+    s.param5 AS lockers_done,
+    s.param6 AS current_locker_pid,
+    s.param7 AS blocks_total,
+    s.param8 AS blocks_done,
+    s.param9 AS tuples_total,
+    s.param10 AS tuples_done,
+    s.param11 AS partitions_total,
+    s.param12 AS partitions_done
+   FROM (pg_stat_get_progress_info('CREATE INDEX'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12)
      LEFT JOIN pg_database d ON ((s.datid = d.oid)));
 pg_stat_progress_vacuum| SELECT s.pid,
     s.datid,
@@ -1878,7 +1904,7 @@ pg_stat_progress_vacuum| SELECT s.pid,
     s.param5 AS index_vacuum_count,
     s.param6 AS max_dead_tuples,
     s.param7 AS num_dead_tuples
-   FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10)
+   FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12)
      LEFT JOIN pg_database d ON ((s.datid = d.oid)));
 pg_stat_replication| SELECT s.pid,
     s.usesysid,
-- 
2.17.1

v7-0002-report-partial-progress-of-other-index-AMs.patchtext/x-diff; charset=us-asciiDownload
From 9196de95822abd3be232331bd22f3fb06e085411 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Tue, 26 Feb 2019 14:34:49 -0300
Subject: [PATCH v7 2/2] report (partial) progress of other index AMs

---
 contrib/bloom/blinsert.c              | 2 +-
 src/backend/access/brin/brin.c        | 2 +-
 src/backend/access/gin/gininsert.c    | 2 +-
 src/backend/access/gist/gistbuild.c   | 2 +-
 src/backend/access/hash/hash.c        | 6 +++++-
 src/backend/access/hash/hashsort.c    | 6 ++++++
 src/backend/access/spgist/spginsert.c | 2 +-
 7 files changed, 16 insertions(+), 6 deletions(-)

diff --git a/contrib/bloom/blinsert.c b/contrib/bloom/blinsert.c
index 48f35d39990..4b2186b8dda 100644
--- a/contrib/bloom/blinsert.c
+++ b/contrib/bloom/blinsert.c
@@ -142,7 +142,7 @@ blbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	initCachedPage(&buildstate);
 
 	/* Do the heap scan */
-	reltuples = table_index_build_scan(heap, index, indexInfo, true, false,
+	reltuples = table_index_build_scan(heap, index, indexInfo, true, true,
 									   bloomBuildCallback, (void *) &buildstate,
 									   NULL);
 
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 54273f754f6..5c2b0c76358 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -720,7 +720,7 @@ brinbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	 * Now scan the relation.  No syncscan allowed here because we want the
 	 * heap blocks in physical order.
 	 */
-	reltuples = table_index_build_scan(heap, index, indexInfo, false, false,
+	reltuples = table_index_build_scan(heap, index, indexInfo, false, true,
 									   brinbuildCallback, (void *) state, NULL);
 
 	/* process the final batch */
diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c
index b4b0213f76f..edc353a7fe0 100644
--- a/src/backend/access/gin/gininsert.c
+++ b/src/backend/access/gin/gininsert.c
@@ -395,7 +395,7 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	 * Do the heap scan.  We disallow sync scan here because dataPlaceToPage
 	 * prefers to receive tuples in TID order.
 	 */
-	reltuples = table_index_build_scan(heap, index, indexInfo, false, false,
+	reltuples = table_index_build_scan(heap, index, indexInfo, false, true,
 									   ginBuildCallback, (void *) &buildstate,
 									   NULL);
 
diff --git a/src/backend/access/gist/gistbuild.c b/src/backend/access/gist/gistbuild.c
index 771bd2962a7..6024671989e 100644
--- a/src/backend/access/gist/gistbuild.c
+++ b/src/backend/access/gist/gistbuild.c
@@ -205,7 +205,7 @@ gistbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	/*
 	 * Do the heap scan.
 	 */
-	reltuples = table_index_build_scan(heap, index, indexInfo, true, false,
+	reltuples = table_index_build_scan(heap, index, indexInfo, true, true,
 									   gistBuildCallback,
 									   (void *) &buildstate, NULL);
 
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 6b38dd9c0e9..048e40e46fa 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -23,9 +23,11 @@
 #include "access/relscan.h"
 #include "access/tableam.h"
 #include "catalog/index.h"
+#include "commands/progress.h"
 #include "commands/vacuum.h"
 #include "miscadmin.h"
 #include "optimizer/plancat.h"
+#include "pgstat.h"
 #include "utils/builtins.h"
 #include "utils/index_selfuncs.h"
 #include "utils/rel.h"
@@ -161,9 +163,11 @@ hashbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	buildstate.heapRel = heap;
 
 	/* do the heap scan */
-	reltuples = table_index_build_scan(heap, index, indexInfo, true, false,
+	reltuples = table_index_build_scan(heap, index, indexInfo, true, true,
 									   hashbuildCallback,
 									   (void *) &buildstate, NULL);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_TOTAL,
+								 buildstate.indtuples);
 
 	if (buildstate.spool)
 	{
diff --git a/src/backend/access/hash/hashsort.c b/src/backend/access/hash/hashsort.c
index 8c55436b193..00a57470a77 100644
--- a/src/backend/access/hash/hashsort.c
+++ b/src/backend/access/hash/hashsort.c
@@ -26,7 +26,9 @@
 #include "postgres.h"
 
 #include "access/hash.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
+#include "pgstat.h"
 #include "utils/tuplesort.h"
 
 
@@ -116,6 +118,7 @@ void
 _h_indexbuild(HSpool *hspool, Relation heapRel)
 {
 	IndexTuple	itup;
+	long		tups_done = 0;
 #ifdef USE_ASSERT_CHECKING
 	uint32		hashkey = 0;
 #endif
@@ -141,5 +144,8 @@ _h_indexbuild(HSpool *hspool, Relation heapRel)
 #endif
 
 		_hash_doinsert(hspool->index, itup, heapRel);
+
+		pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE,
+									 ++tups_done);
 	}
 }
diff --git a/src/backend/access/spgist/spginsert.c b/src/backend/access/spgist/spginsert.c
index 282d6998cf0..b06feafdc24 100644
--- a/src/backend/access/spgist/spginsert.c
+++ b/src/backend/access/spgist/spginsert.c
@@ -143,7 +143,7 @@ spgbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 											  "SP-GiST build temporary context",
 											  ALLOCSET_DEFAULT_SIZES);
 
-	reltuples = table_index_build_scan(heap, index, indexInfo, true, false,
+	reltuples = table_index_build_scan(heap, index, indexInfo, true, true,
 									   spgistBuildCallback, (void *) &buildstate,
 									   NULL);
 
-- 
2.17.1

#37Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: David Fetter (#18)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On 2019-Mar-10, David Fetter wrote:

Would it be a very large lift to report progress for the rest of the
index types we support?

Patch v7 I just posted does that. Please give it a look and let me know
what you think.

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#38Simon Riggs
simon@2ndquadrant.com
In reply to: Alvaro Herrera (#36)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On Thu, 28 Mar 2019 at 14:56, Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:

I have not reinstated phase numbers; I have Rahila's positive vote for
them. Do I hear any more votes on this issue?

If there is a specific technical issue, I'd like to understand that more.
If it is just a usability preference, then I say we should have numbers.

Numbering is natural for people. If we say "It's currently doing phase
XYZ", they will say "Is that the 3rd phase?", we'll say "No, actually the
5th", and then they will say "Why didn't you just number them?"

--
Simon Riggs http://www.2ndQuadrant.com/
<http://www.2ndquadrant.com/&gt;
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#39Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Simon Riggs (#38)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On 2019-Mar-28, Simon Riggs wrote:

On Thu, 28 Mar 2019 at 14:56, Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:

I have not reinstated phase numbers; I have Rahila's positive vote for
them. Do I hear any more votes on this issue?

If there is a specific technical issue, I'd like to understand that more.

There's no technical issue -- that's pretty straightforward. Earlier
versions of the patch had them, and removing them only meant editing
strings in a couple of places.

If it is just a usability preference, then I say we should have numbers.

Numbering is natural for people. If we say "It's currently doing phase
XYZ", they will say "Is that the 3rd phase?", we'll say "No, actually the
5th", and then they will say "Why didn't you just number them?"

There are eight phases. If you run normal CREATE INDEX (not concurrent)
then you get phases 1, then 3, done. If you run CIC you get phases from
1 to 8. Phase 3 "building index" has arbitrary subphases (they depend
on AM) in both cases.

I think the lack of phase numbering comes from the fact that the first
command we did (VACUUM) sometimes jumps backwards in phase numbers, so
it would be a bit absurd from users's POV.

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#40Simon Riggs
simon@2ndquadrant.com
In reply to: Alvaro Herrera (#39)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On Thu, 28 Mar 2019 at 15:39, Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:

On 2019-Mar-28, Simon Riggs wrote:

On Thu, 28 Mar 2019 at 14:56, Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:

I have not reinstated phase numbers; I have Rahila's positive vote for
them. Do I hear any more votes on this issue?

If there is a specific technical issue, I'd like to understand that more.

There's no technical issue -- that's pretty straightforward. Earlier
versions of the patch had them, and removing them only meant editing
strings in a couple of places.

If it is just a usability preference, then I say we should have numbers.

Numbering is natural for people. If we say "It's currently doing phase
XYZ", they will say "Is that the 3rd phase?", we'll say "No, actually the
5th", and then they will say "Why didn't you just number them?"

There are eight phases. If you run normal CREATE INDEX (not concurrent)
then you get phases 1, then 3, done. If you run CIC you get phases from
1 to 8. Phase 3 "building index" has arbitrary subphases (they depend
on AM) in both cases.

Maybe the AM won't know, but I don't think that matters. It's still useful
to know the difference between Phase 3.3 and Phase 3.33 and Phase 7.

The description only helps you if you understand what it means. If your AM
replies something many users wouldn't understand like "сортировка" or
"constructing triples", we still want to know where that step fits in the
overall sequence of steps.

I think the lack of phase numbering comes from the fact that the first
command we did (VACUUM) sometimes jumps backwards in phase numbers, so
it would be a bit absurd from users's POV.

Seems more like our own labelling of the phases is responsible for that,
rather than it being a specific problem. The numbering should reflect the
ordinal executed step number. So if a VACUUM has required two sets of index
scanning, the heap scan phase (normally phase 3) should be labelled phase 6
when it occurs the second time, rather than "phase 3 again, doh" which
clearly doesn't work.

By the time VACUUM moves to its 2nd phase, which is normally thought of as
"Phase2 Index Scanning", we know how much of the table has been scanned, so
we really should be able to calculate how many more phases will be needed.
We also know how many AM sub-phases will be called for that step.

--
Simon Riggs http://www.2ndQuadrant.com/
<http://www.2ndquadrant.com/&gt;
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#41Robert Haas
robertmhaas@gmail.com
In reply to: Simon Riggs (#40)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On Thu, Mar 28, 2019 at 12:07 PM Simon Riggs <simon@2ndquadrant.com> wrote:

Seems more like our own labelling of the phases is responsible for that, rather than it being a specific problem. The numbering should reflect the ordinal executed step number. So if a VACUUM has required two sets of index scanning, the heap scan phase (normally phase 3) should be labelled phase 6 when it occurs the second time, rather than "phase 3 again, doh" which clearly doesn't work.

That would not be too simple to do with the infrastructure we have
available, I think. Also, Alvaro's showed phase names like '3 of 8',
but if you regarded each set of index scans as a separate phase rather
than a repetition of a phase that had already happened, you wouldn't
know whether there were going to be 8 phases in total or some other
number, because you don't know how many times you're going to scan the
indexes.

I suggest that it makes sense to leave the phase numbers out of this
commit. If someone wants to make a proposal for adding phase numbers
to the various commands that now support progress reporting as a
separate commit, then that can be discussed separately.

BTW, if we are going to do that, it might be best to put then in
separate view columns rather than making them part of the phase names.
Somebody might want to look for those phase names using SQL or
client-side logic and not have the logic get broken if we renumber the
phases.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

#42Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Robert Haas (#41)
Re: monitoring CREATE INDEX [CONCURRENTLY]

I just noticed that the CLUSTER calls index_build, which my patch
modifies to include additional progress metrics; this means that during
the index build phase, the metrics set by CLUSTER will be trashed by the
ones my patch introduces.

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#43Andres Freund
andres@anarazel.de
In reply to: Alvaro Herrera (#42)
Re: monitoring CREATE INDEX [CONCURRENTLY]

Hi,

On 2019-03-29 12:02:18 -0300, Alvaro Herrera wrote:

I just noticed that the CLUSTER calls index_build, which my patch
modifies to include additional progress metrics; this means that during
the index build phase, the metrics set by CLUSTER will be trashed by the
ones my patch introduces.

Yea, it really seems that the index build infrastructure needs to
support cooperating with the caller's progress reporting. For CLUSTER,
REINDEX, ALTER TABLE rewrites etc, they all would likely want to have
insight into the index build while also having their own progress.

Greetings,

Andres Freund

#44Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Alvaro Herrera (#42)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On 2019-Mar-29, Alvaro Herrera wrote:

I just noticed that the CLUSTER calls index_build, which my patch
modifies to include additional progress metrics; this means that during
the index build phase, the metrics set by CLUSTER will be trashed by the
ones my patch introduces.

Indeed:

pid | datid | datname | relid | command | phase | cluster_index_relid | heap_tuples_scanned | heap_tuples_written | heap_blks_total | heap_blks_scanned | index_rebuild_count
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 162402 | 162402 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 460362 | 460362 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 754004 | 754004 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 1047058 | 1047058 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 1356296 | 1356296 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 1645321 | 1645321 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 1939920 | 1939920 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 2227450 | 2227450 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 2526116 | 2526116 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 2828468 | 2828468 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 3142982 | 3142982 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 3451494 | 3451494 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 3769799 | 3769799 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 4077513 | 4077513 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 4383255 | 4383255 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 4700286 | 4700286 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 5015468 | 5015468 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 5324951 | 5324951 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 5628172 | 5628172 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 5940862 | 5940862 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 6253778 | 6253778 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 6560474 | 6560474 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 6881248 | 6881248 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 7186555 | 7186555 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 7503888 | 7503888 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 7821706 | 7821706 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 8137334 | 8137334 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 8453280 | 8453280 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 8764097 | 8764097 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 9079908 | 9079908 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 9389465 | 9389465 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 9706288 | 9706288 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 10024875 | 10024875 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 10344875 | 10344875 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 10658531 | 10658531 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 10974097 | 10974097 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 11289868 | 11289868 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 11602965 | 11602965 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 11917624 | 11917624 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 12237998 | 12237998 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 12559406 | 12559406 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 12876413 | 12876413 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 13192793 | 13192793 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 13506188 | 13506188 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 13822722 | 13822722 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 14138624 | 14138624 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 14432335 | 14432335 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 14753617 | 14753617 | 0 | 0 | 0
-->
pid | datid | datname | relid | command | phase | cluster_index_relid | heap_tuples_scanned | heap_tuples_written | heap_blks_total | heap_blks_scanned | index_rebuild_count
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 743
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 4639
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 8938
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 13457
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 17806
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 22237
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 26594
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 30935
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 35362
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 39705
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 44126
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 48546
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 52966
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 57174
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 61536
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 65868
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 66193
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 66193
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 66193
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 66193
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 66193
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 5 | 15000000 | 15000000 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 5 | 15000000 | 15000000 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 5 | 15000000 | 15000000 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 5 | 15000000 | 15000000 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 5 | 15000000 | 15000000 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 5 | 15000000 | 15000000 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 5 | 15000000 | 15000000 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 5 | 15000000 | 15000000 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 5 | 15000000 | 15000000 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 5 | 15000000 | 15000000 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 5 | 15000000 | 15000000 | 0 | 0 | 0
28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 5 | 15000000 | 15000000 | 0 | 0 | 0

I suppose I can just pick non-overlapping numbers for those columns.

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#45Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Andres Freund (#43)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On 2019-Mar-29, Andres Freund wrote:

Hi,

On 2019-03-29 12:02:18 -0300, Alvaro Herrera wrote:

I just noticed that the CLUSTER calls index_build, which my patch
modifies to include additional progress metrics; this means that during
the index build phase, the metrics set by CLUSTER will be trashed by the
ones my patch introduces.

Yea, it really seems that the index build infrastructure needs to
support cooperating with the caller's progress reporting. For CLUSTER,
REINDEX, ALTER TABLE rewrites etc, they all would likely want to have
insight into the index build while also having their own progress.

So, CLUSTER and ALTER TABLE rewrites only do non-concurrent index
builds; and REINDEX can reuse pretty much the same wait-for metrics
columns as CIC. So I think it's okay if I move only the metrics that
conflict for index_build.

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#46Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Alvaro Herrera (#45)
2 attachment(s)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On 2019-Mar-29, Alvaro Herrera wrote:

So, CLUSTER and ALTER TABLE rewrites only do non-concurrent index
builds; and REINDEX can reuse pretty much the same wait-for metrics
columns as CIC. So I think it's okay if I move only the metrics that
conflict for index_build.

The attached version does it that way. I had to enlarge the param set a
bit more. (I suspect those extra columns will be useful to reindex.)
Also, rebased for recent conflicting changes.

I think we should consider a new column of an array type, where we could
put things like the list of PIDs to be waited for, the list of OIDs of
index to rebuild, or the list of partitions to build the index on.

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attachments:

v8-0002-report-partial-progress-of-other-index-AMs.patchtext/x-diff; charset=us-asciiDownload
From a0b248eff6930cb75d071f0f2913e7342ec4cd7b Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Tue, 26 Feb 2019 14:34:49 -0300
Subject: [PATCH v8 2/2] report (partial) progress of other index AMs

---
 contrib/bloom/blinsert.c              | 2 +-
 src/backend/access/brin/brin.c        | 2 +-
 src/backend/access/gin/gininsert.c    | 2 +-
 src/backend/access/gist/gistbuild.c   | 2 +-
 src/backend/access/hash/hash.c        | 6 +++++-
 src/backend/access/hash/hashsort.c    | 6 ++++++
 src/backend/access/spgist/spginsert.c | 2 +-
 7 files changed, 16 insertions(+), 6 deletions(-)

diff --git a/contrib/bloom/blinsert.c b/contrib/bloom/blinsert.c
index 48f35d39990..4b2186b8dda 100644
--- a/contrib/bloom/blinsert.c
+++ b/contrib/bloom/blinsert.c
@@ -142,7 +142,7 @@ blbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	initCachedPage(&buildstate);
 
 	/* Do the heap scan */
-	reltuples = table_index_build_scan(heap, index, indexInfo, true, false,
+	reltuples = table_index_build_scan(heap, index, indexInfo, true, true,
 									   bloomBuildCallback, (void *) &buildstate,
 									   NULL);
 
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 54273f754f6..5c2b0c76358 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -720,7 +720,7 @@ brinbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	 * Now scan the relation.  No syncscan allowed here because we want the
 	 * heap blocks in physical order.
 	 */
-	reltuples = table_index_build_scan(heap, index, indexInfo, false, false,
+	reltuples = table_index_build_scan(heap, index, indexInfo, false, true,
 									   brinbuildCallback, (void *) state, NULL);
 
 	/* process the final batch */
diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c
index b4b0213f76f..edc353a7fe0 100644
--- a/src/backend/access/gin/gininsert.c
+++ b/src/backend/access/gin/gininsert.c
@@ -395,7 +395,7 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	 * Do the heap scan.  We disallow sync scan here because dataPlaceToPage
 	 * prefers to receive tuples in TID order.
 	 */
-	reltuples = table_index_build_scan(heap, index, indexInfo, false, false,
+	reltuples = table_index_build_scan(heap, index, indexInfo, false, true,
 									   ginBuildCallback, (void *) &buildstate,
 									   NULL);
 
diff --git a/src/backend/access/gist/gistbuild.c b/src/backend/access/gist/gistbuild.c
index 771bd2962a7..6024671989e 100644
--- a/src/backend/access/gist/gistbuild.c
+++ b/src/backend/access/gist/gistbuild.c
@@ -205,7 +205,7 @@ gistbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	/*
 	 * Do the heap scan.
 	 */
-	reltuples = table_index_build_scan(heap, index, indexInfo, true, false,
+	reltuples = table_index_build_scan(heap, index, indexInfo, true, true,
 									   gistBuildCallback,
 									   (void *) &buildstate, NULL);
 
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 6b38dd9c0e9..048e40e46fa 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -23,9 +23,11 @@
 #include "access/relscan.h"
 #include "access/tableam.h"
 #include "catalog/index.h"
+#include "commands/progress.h"
 #include "commands/vacuum.h"
 #include "miscadmin.h"
 #include "optimizer/plancat.h"
+#include "pgstat.h"
 #include "utils/builtins.h"
 #include "utils/index_selfuncs.h"
 #include "utils/rel.h"
@@ -161,9 +163,11 @@ hashbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	buildstate.heapRel = heap;
 
 	/* do the heap scan */
-	reltuples = table_index_build_scan(heap, index, indexInfo, true, false,
+	reltuples = table_index_build_scan(heap, index, indexInfo, true, true,
 									   hashbuildCallback,
 									   (void *) &buildstate, NULL);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_TOTAL,
+								 buildstate.indtuples);
 
 	if (buildstate.spool)
 	{
diff --git a/src/backend/access/hash/hashsort.c b/src/backend/access/hash/hashsort.c
index 8c55436b193..00a57470a77 100644
--- a/src/backend/access/hash/hashsort.c
+++ b/src/backend/access/hash/hashsort.c
@@ -26,7 +26,9 @@
 #include "postgres.h"
 
 #include "access/hash.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
+#include "pgstat.h"
 #include "utils/tuplesort.h"
 
 
@@ -116,6 +118,7 @@ void
 _h_indexbuild(HSpool *hspool, Relation heapRel)
 {
 	IndexTuple	itup;
+	long		tups_done = 0;
 #ifdef USE_ASSERT_CHECKING
 	uint32		hashkey = 0;
 #endif
@@ -141,5 +144,8 @@ _h_indexbuild(HSpool *hspool, Relation heapRel)
 #endif
 
 		_hash_doinsert(hspool->index, itup, heapRel);
+
+		pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE,
+									 ++tups_done);
 	}
 }
diff --git a/src/backend/access/spgist/spginsert.c b/src/backend/access/spgist/spginsert.c
index 282d6998cf0..b06feafdc24 100644
--- a/src/backend/access/spgist/spginsert.c
+++ b/src/backend/access/spgist/spginsert.c
@@ -143,7 +143,7 @@ spgbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 											  "SP-GiST build temporary context",
 											  ALLOCSET_DEFAULT_SIZES);
 
-	reltuples = table_index_build_scan(heap, index, indexInfo, true, false,
+	reltuples = table_index_build_scan(heap, index, indexInfo, true, true,
 									   spgistBuildCallback, (void *) &buildstate,
 									   NULL);
 
-- 
2.17.1

v8-0001-Report-progress-of-CREATE-INDEX-operations.patchtext/x-diff; charset=us-asciiDownload
From c36cc4e62067ead6eb8266602046f090689ccf76 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Mon, 25 Mar 2019 13:29:34 -0300
Subject: [PATCH v8 1/2] Report progress of CREATE INDEX operations

---
 contrib/amcheck/verify_nbtree.c          |   2 +-
 contrib/bloom/blinsert.c                 |   2 +-
 contrib/bloom/blutils.c                  |   1 +
 doc/src/sgml/indexam.sgml                |  13 ++
 doc/src/sgml/monitoring.sgml             | 224 ++++++++++++++++++++++-
 src/backend/access/brin/brin.c           |   5 +-
 src/backend/access/gin/gininsert.c       |   2 +-
 src/backend/access/gin/ginutil.c         |   1 +
 src/backend/access/gist/gist.c           |   1 +
 src/backend/access/gist/gistbuild.c      |   2 +-
 src/backend/access/hash/hash.c           |   3 +-
 src/backend/access/heap/heapam_handler.c |  73 ++++++++
 src/backend/access/nbtree/nbtree.c       |   9 +
 src/backend/access/nbtree/nbtsort.c      |  61 +++++-
 src/backend/access/nbtree/nbtutils.c     |  24 +++
 src/backend/access/spgist/spginsert.c    |   2 +-
 src/backend/access/spgist/spgutils.c     |   1 +
 src/backend/catalog/index.c              |  67 ++++++-
 src/backend/catalog/system_views.sql     |  27 +++
 src/backend/commands/indexcmds.c         |  72 +++++++-
 src/backend/storage/ipc/standby.c        |   2 +-
 src/backend/storage/lmgr/lmgr.c          |  46 ++++-
 src/backend/storage/lmgr/lock.c          |   7 +-
 src/backend/utils/adt/amutils.c          |  23 +++
 src/backend/utils/adt/pgstatfuncs.c      |   2 +
 src/include/access/amapi.h               |   4 +
 src/include/access/genam.h               |   1 +
 src/include/access/nbtree.h              |  11 ++
 src/include/access/tableam.h             |  10 +
 src/include/catalog/pg_proc.dat          |  10 +-
 src/include/commands/progress.h          |  37 +++-
 src/include/pgstat.h                     |   5 +-
 src/include/storage/lmgr.h               |   4 +-
 src/include/storage/lock.h               |   2 +-
 src/test/regress/expected/rules.out      |  30 ++-
 35 files changed, 740 insertions(+), 46 deletions(-)

diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index 9ecb1999e34..b55178328c8 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -566,7 +566,7 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
 			 RelationGetRelationName(state->rel),
 			 RelationGetRelationName(state->heaprel));
 
-		table_index_build_scan(state->heaprel, state->rel, indexinfo, true,
+		table_index_build_scan(state->heaprel, state->rel, indexinfo, true, false,
 							   bt_tuple_present_callback, (void *) state, scan);
 
 		ereport(DEBUG1,
diff --git a/contrib/bloom/blinsert.c b/contrib/bloom/blinsert.c
index 1b8df7e1e84..48f35d39990 100644
--- a/contrib/bloom/blinsert.c
+++ b/contrib/bloom/blinsert.c
@@ -142,7 +142,7 @@ blbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	initCachedPage(&buildstate);
 
 	/* Do the heap scan */
-	reltuples = table_index_build_scan(heap, index, indexInfo, true,
+	reltuples = table_index_build_scan(heap, index, indexInfo, true, false,
 									   bloomBuildCallback, (void *) &buildstate,
 									   NULL);
 
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index d078dfbd469..ee3bd562748 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -132,6 +132,7 @@ blhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = blcostestimate;
 	amroutine->amoptions = bloptions;
 	amroutine->amproperty = NULL;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = blvalidate;
 	amroutine->ambeginscan = blbeginscan;
 	amroutine->amrescan = blrescan;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index b56d3b3daa1..ff8290da9ff 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -127,6 +127,7 @@ typedef struct IndexAmRoutine
     amcostestimate_function amcostestimate;
     amoptions_function amoptions;
     amproperty_function amproperty;     /* can be NULL */
+    ambuildphasename_function ambuildphasename;   /* can be NULL */
     amvalidate_function amvalidate;
     ambeginscan_function ambeginscan;
     amrescan_function amrescan;
@@ -468,6 +469,18 @@ amproperty (Oid index_oid, int attno,
 
   <para>
 <programlisting>
+char *
+ambuildphasename (int64 phasenum);
+</programlisting>
+   Return the textual name of the given build phase number.
+   The phase numbers are those reported during an index build via the
+   <function>pgstat_progress_update_param</function> interface.
+   The phase names are then exposed in the
+   <structname>pg_stat_progress_create_index</structname> view.
+  </para>
+
+  <para>
+<programlisting>
 bool
 amvalidate (Oid opclassoid);
 </programlisting>
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index f1df14bdea8..8f5bbbb79e5 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -336,6 +336,14 @@ postgres   27093  0.0  0.0  30096  2752 ?        Ss   11:34   0:00 postgres: ser
       </entry>
      </row>
 
+     <row>
+      <entry><structname>pg_stat_progress_create_index</structname><indexterm><primary>pg_stat_progress_create_index</primary></indexterm></entry>
+      <entry>One row for each backend running <command>CREATE INDEX</command>, showing
+      current progress.
+      See <xref linkend='create-index-progress-reporting'/>.
+     </entry>
+     </row>
+
      <row>
       <entry><structname>pg_stat_progress_vacuum</structname><indexterm><primary>pg_stat_progress_vacuum</primary></indexterm></entry>
       <entry>One row for each backend (including autovacuum worker processes) running
@@ -3403,10 +3411,224 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
   <para>
    <productname>PostgreSQL</productname> has the ability to report the progress of
    certain commands during command execution.  Currently, the only commands
-   which support progress reporting are <command>VACUUM</command> and
+   which support progress reporting are <command>CREATE INDEX</command>,
+   <command>VACUUM</command> and
    <command>CLUSTER</command>. This may be expanded in the future.
   </para>
 
+ <sect2 id="create-index-progress-reporting">
+  <title>CREATE INDEX Progress Reporting</title>
+
+  <para>
+   Whenever <command>CREATE INDEX</command> is running, the
+   <structname>pg_stat_progress_create_index</structname> view will contain
+   one row for each backend that is currently creating indexes.  The tables
+   below describe the information that will be reported and provide information
+   about how to interpret it.
+  </para>
+
+  <table id="pg-stat-progress-create-index-view" xreflabel="pg_stat_progress_create_index">
+   <title><structname>pg_stat_progress_create_index</structname> View</title>
+   <tgroup cols="3">
+    <thead>
+     <row>
+      <entry>Column</entry>
+      <entry>Type</entry>
+      <entry>Description</entry>
+     </row>
+    </thead>
+
+    <tbody>
+     <row>
+      <entry><structfield>pid</structfield></entry>
+      <entry><type>integer</type></entry>
+      <entry>Process ID of backend.</entry>
+     </row>
+     <row>
+      <entry><structfield>datid</structfield></entry>
+      <entry><type>oid</type></entry>
+      <entry>OID of the database to which this backend is connected.</entry>
+     </row>
+     <row>
+      <entry><structfield>datname</structfield></entry>
+      <entry><type>name</type></entry>
+      <entry>Name of the database to which this backend is connected.</entry>
+     </row>
+     <row>
+      <entry><structfield>relid</structfield></entry>
+      <entry><type>oid</type></entry>
+      <entry>OID of the table on which the index is being created.</entry>
+     </row>
+     <row>
+      <entry><structfield>phase</structfield></entry>
+      <entry><type>text</type></entry>
+      <entry>
+        Current processing phase of index creation.  See <xref linkend='create-index-phases'/>.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>lockers_total</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Total number of lockers to wait for, when applicable.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>lockers_done</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Number of lockers already waited for.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>current_locked_pid</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Process ID of the locker currently being waited for.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>blocks_total</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Total number of blocks to be processed in the current phase.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>blocks_done</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Number of blocks already processed in the current phase.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>tuples_total</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Total number of tuples to be processed in the current phase.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>tuples_done</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Number of tuples already processed in the current phase.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>partitions_total</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+       When creating an index on a partitioned, this column is set to the
+       total number of partitions on which the index is to be created.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>partitions_done</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+       When creating an index on a partitioned, this column is set to the
+       number of partitions on which the index has been completed.
+      </entry>
+     </row>
+    </tbody>
+   </tgroup>
+  </table>
+
+  <table id="create-index-phases">
+   <title>CREATE INDEX phases</title>
+   <tgroup cols="2">
+    <thead>
+     <row>
+      <entry>Phase</entry>
+      <entry>Description</entry>
+     </row>
+    </thead>
+    <tbody>
+     <row>
+      <entry><literal>initializing</literal></entry>
+      <entry>
+       <command>CREATE INDEX</command> is preparing to create the index.  This
+       phase is expected to be very brief.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>waiting for old snapshots</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions
+       that can potentially see the table to release their snapshots.
+       This phase is skipped when not in concurrent mode.
+       Columns <structname>lockers_total</structname>, <structname>lockers_done</structname>
+       and <structname>current_locker_pid</structname> contain the progress 
+       information for this phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>building index</literal></entry>
+      <entry>
+       The index is being built by the access method-specific code.  In this phase,
+       access methods that support progress reporting fill in their own progress data,
+       and the subphase is indicated in this column.  Typically,
+       <structname>blocks_total</structname> and <structname>blocks_done</structname>
+       will contain progress data, as well as potentially
+       <structname>tuples_total</structname> and <structname>tuples_done</structname>.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>waiting for writer snapshots</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions
+       that can potentially write into the table to release their snapshots.
+       This phase is skipped when not in concurrent mode.
+       Columns <structname>lockers_total</structname>, <structname>lockers_done</structname>
+       and <structname>current_locker_pid</structname> contain the progress 
+       information for this phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>validating index scan</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is scanning the index searching
+       for tuples that need to be validated.
+       This phase is skipped when not in concurrent mode.
+       Columns <structname>blocks_total</structname> (set to the total size of the index)
+       and <structname>blocks_done</structname> contain the progress information for this phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>sorting index scan results</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is sorting the output of the
+       previous phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>validating index heapscan</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is scanning the table
+       to validate the index tuples collected in the previous two phases.
+       This phase is skipped when not in concurrent mode.
+       Columns <structname>blocks_total</structname> (set to the total size of the table)
+       and <structname>blocks_done</structname> contain the progress information for this phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>waiting for reader snapshots</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions
+       that can potentially see the table to release their snapshots.  This
+       phase is skipped when not in concurrent mode.
+       Columns <structname>lockers_total</structname>, <structname>lockers_done</structname>
+       and <structname>current_locker_pid</structname> contain the progress 
+       information for this phase.
+      </entry>
+     </row>
+    </tbody>
+   </tgroup>
+  </table>
+
+ </sect2>
+
  <sect2 id="vacuum-progress-reporting">
   <title>VACUUM Progress Reporting</title>
 
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 6e96d24ca22..54273f754f6 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -112,6 +112,7 @@ brinhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = brincostestimate;
 	amroutine->amoptions = brinoptions;
 	amroutine->amproperty = NULL;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = brinvalidate;
 	amroutine->ambeginscan = brinbeginscan;
 	amroutine->amrescan = brinrescan;
@@ -719,7 +720,7 @@ brinbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	 * Now scan the relation.  No syncscan allowed here because we want the
 	 * heap blocks in physical order.
 	 */
-	reltuples = table_index_build_scan(heap, index, indexInfo, false,
+	reltuples = table_index_build_scan(heap, index, indexInfo, false, false,
 									   brinbuildCallback, (void *) state, NULL);
 
 	/* process the final batch */
@@ -1236,7 +1237,7 @@ summarize_range(IndexInfo *indexInfo, BrinBuildState *state, Relation heapRel,
 	 * cases.
 	 */
 	state->bs_currRangeStart = heapBlk;
-	table_index_build_range_scan(heapRel, state->bs_irel, indexInfo, false, true,
+	table_index_build_range_scan(heapRel, state->bs_irel, indexInfo, false, true, false,
 								 heapBlk, scanNumBlks,
 								 brinbuildCallback, (void *) state, NULL);
 
diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c
index b02f69b0dcb..b4b0213f76f 100644
--- a/src/backend/access/gin/gininsert.c
+++ b/src/backend/access/gin/gininsert.c
@@ -395,7 +395,7 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	 * Do the heap scan.  We disallow sync scan here because dataPlaceToPage
 	 * prefers to receive tuples in TID order.
 	 */
-	reltuples = table_index_build_scan(heap, index, indexInfo, false,
+	reltuples = table_index_build_scan(heap, index, indexInfo, false, false,
 									   ginBuildCallback, (void *) &buildstate,
 									   NULL);
 
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index afc20232ace..d2360eeafb0 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -64,6 +64,7 @@ ginhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = gincostestimate;
 	amroutine->amoptions = ginoptions;
 	amroutine->amproperty = NULL;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = ginvalidate;
 	amroutine->ambeginscan = ginbeginscan;
 	amroutine->amrescan = ginrescan;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 2fddb23496d..f44c922b5d6 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -86,6 +86,7 @@ gisthandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = gistcostestimate;
 	amroutine->amoptions = gistoptions;
 	amroutine->amproperty = gistproperty;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = gistvalidate;
 	amroutine->ambeginscan = gistbeginscan;
 	amroutine->amrescan = gistrescan;
diff --git a/src/backend/access/gist/gistbuild.c b/src/backend/access/gist/gistbuild.c
index 3652fde5bb1..771bd2962a7 100644
--- a/src/backend/access/gist/gistbuild.c
+++ b/src/backend/access/gist/gistbuild.c
@@ -205,7 +205,7 @@ gistbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	/*
 	 * Do the heap scan.
 	 */
-	reltuples = table_index_build_scan(heap, index, indexInfo, true,
+	reltuples = table_index_build_scan(heap, index, indexInfo, true, false,
 									   gistBuildCallback,
 									   (void *) &buildstate, NULL);
 
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 5cc12a17130..6b38dd9c0e9 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -83,6 +83,7 @@ hashhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = hashcostestimate;
 	amroutine->amoptions = hashoptions;
 	amroutine->amproperty = NULL;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = hashvalidate;
 	amroutine->ambeginscan = hashbeginscan;
 	amroutine->amrescan = hashrescan;
@@ -160,7 +161,7 @@ hashbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	buildstate.heapRel = heap;
 
 	/* do the heap scan */
-	reltuples = table_index_build_scan(heap, index, indexInfo, true,
+	reltuples = table_index_build_scan(heap, index, indexInfo, true, false,
 									   hashbuildCallback,
 									   (void *) &buildstate, NULL);
 
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 39eab1e28d9..dec653d3b4d 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -54,6 +54,9 @@ reform_and_rewrite_tuple(HeapTuple tuple,
 static const TableAmRoutine heapam_methods;
 
 
+static BlockNumber heapscan_get_blocks_done(HeapScanDesc hscan);
+
+
 /* ------------------------------------------------------------------------
  * Slot related callbacks for heap AM
  * ------------------------------------------------------------------------
@@ -927,6 +930,7 @@ heapam_index_build_range_scan(Relation heapRelation,
 							  IndexInfo *indexInfo,
 							  bool allow_sync,
 							  bool anyvisible,
+							  bool progress,
 							  BlockNumber start_blockno,
 							  BlockNumber numblocks,
 							  IndexBuildCallback callback,
@@ -947,6 +951,7 @@ heapam_index_build_range_scan(Relation heapRelation,
 	Snapshot	snapshot;
 	bool		need_unregister_snapshot = false;
 	TransactionId OldestXmin;
+	BlockNumber	previous_blkno = InvalidBlockNumber;
 	BlockNumber root_blkno = InvalidBlockNumber;
 	OffsetNumber root_offsets[MaxHeapTuplesPerPage];
 
@@ -1017,6 +1022,14 @@ heapam_index_build_range_scan(Relation heapRelation,
 									 NULL,	/* scan key */
 									 true,	/* buffer access strategy OK */
 									 allow_sync);	/* syncscan OK? */
+
+		if (progress)
+		{
+			hscan = (HeapScanDesc) scan;
+			Assert(hscan->rs_numblocks == InvalidBlockNumber);
+			pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL,
+										 hscan->rs_nblocks);
+		}
 	}
 	else
 	{
@@ -1066,6 +1079,19 @@ heapam_index_build_range_scan(Relation heapRelation,
 
 		CHECK_FOR_INTERRUPTS();
 
+		/* Report scan progress, if asked to. */
+		if (progress)
+		{
+			BlockNumber     blocks_done = heapscan_get_blocks_done(hscan);
+
+			if (blocks_done != previous_blkno)
+			{
+				pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
+											 blocks_done);
+				previous_blkno = blocks_done;
+			}
+		}
+
 		/*
 		 * When dealing with a HOT-chain of updated tuples, we want to index
 		 * the values of the live tuple (if any), but index it under the TID
@@ -1440,6 +1466,7 @@ heapam_index_validate_scan(Relation heapRelation,
 	BlockNumber root_blkno = InvalidBlockNumber;
 	OffsetNumber root_offsets[MaxHeapTuplesPerPage];
 	bool		in_index[MaxHeapTuplesPerPage];
+	BlockNumber	previous_blkno = InvalidBlockNumber;
 
 	/* state variables for the merge */
 	ItemPointer indexcursor = NULL;
@@ -1480,6 +1507,9 @@ heapam_index_validate_scan(Relation heapRelation,
 								 false);	/* syncscan not OK */
 	hscan = (HeapScanDesc) scan;
 
+	pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL,
+								 hscan->rs_nblocks);
+
 	/*
 	 * Scan all tuples matching the snapshot.
 	 */
@@ -1493,6 +1523,14 @@ heapam_index_validate_scan(Relation heapRelation,
 
 		state->htups += 1;
 
+		if ((previous_blkno == InvalidBlockNumber) ||
+			(hscan->rs_cblock != previous_blkno))
+		{
+			pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
+										 hscan->rs_cblock);
+			previous_blkno = hscan->rs_cblock;
+		}
+
 		/*
 		 * As commented in table_index_build_scan, we should index heap-only
 		 * tuples under the TIDs of their root tuples; so when we advance onto
@@ -1653,6 +1691,41 @@ heapam_index_validate_scan(Relation heapRelation,
 	indexInfo->ii_PredicateState = NULL;
 }
 
+/*
+ * Return the number of blocks that have been read by this scan since
+ * starting.  This is not 100% accurate: in a parallel scan, the workers
+ * can be concurrently reading blocks further ahead than what we report.
+ */
+static BlockNumber
+heapscan_get_blocks_done(HeapScanDesc hscan)
+{
+	BlockNumber		startblock;
+	BlockNumber		blocks_done;
+
+	if (hscan->rs_base.rs_parallel != NULL)
+	{
+		ParallelBlockTableScanDesc bpscan;
+
+		bpscan = (ParallelBlockTableScanDesc) hscan->rs_base.rs_parallel;
+		startblock = bpscan->phs_startblock;
+	}
+	else
+		startblock = hscan->rs_startblock;
+
+	/*
+	 * Might have wrapped around the end of the relation, if startblock was
+	 * not zero.
+	 */
+	if (hscan->rs_cblock > startblock)
+		blocks_done = hscan->rs_cblock - startblock;
+	else
+		blocks_done = hscan->rs_nblocks - startblock +
+			hscan->rs_cblock;
+
+	return blocks_done;
+}
+
+
 
 /* ----------------------------------------------------------------------------
  *  Helper functions for the above.
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index ac6f1eb3423..7370379c6a1 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -22,6 +22,7 @@
 #include "access/nbtxlog.h"
 #include "access/relscan.h"
 #include "access/xlog.h"
+#include "commands/progress.h"
 #include "commands/vacuum.h"
 #include "miscadmin.h"
 #include "nodes/execnodes.h"
@@ -133,6 +134,7 @@ bthandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = btcostestimate;
 	amroutine->amoptions = btoptions;
 	amroutine->amproperty = btproperty;
+	amroutine->ambuildphasename = btbuildphasename;
 	amroutine->amvalidate = btvalidate;
 	amroutine->ambeginscan = btbeginscan;
 	amroutine->amrescan = btrescan;
@@ -1021,6 +1023,10 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
 		if (needLock)
 			UnlockRelationForExtension(rel, ExclusiveLock);
 
+		if (info->report_progress)
+			pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL,
+										 num_pages);
+
 		/* Quit if we've scanned the whole relation */
 		if (blkno >= num_pages)
 			break;
@@ -1028,6 +1034,9 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
 		for (; blkno < num_pages; blkno++)
 		{
 			btvacuumpage(&vstate, blkno, blkno);
+			if (info->report_progress)
+				pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
+											 blkno);
 		}
 	}
 
diff --git a/src/backend/access/nbtree/nbtsort.c b/src/backend/access/nbtree/nbtsort.c
index a8a7b792672..b889ac6de79 100644
--- a/src/backend/access/nbtree/nbtsort.c
+++ b/src/backend/access/nbtree/nbtsort.c
@@ -66,6 +66,7 @@
 #include "access/xlog.h"
 #include "access/xloginsert.h"
 #include "catalog/index.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
 #include "pgstat.h"
 #include "storage/smgr.h"
@@ -298,7 +299,8 @@ static double _bt_parallel_heapscan(BTBuildState *buildstate,
 static void _bt_leader_participate_as_worker(BTBuildState *buildstate);
 static void _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2,
 						   BTShared *btshared, Sharedsort *sharedsort,
-						   Sharedsort *sharedsort2, int sortmem);
+						   Sharedsort *sharedsort2, int sortmem,
+						   bool progress);
 
 
 /*
@@ -394,6 +396,10 @@ _bt_spools_heapscan(Relation heap, Relation index, BTBuildState *buildstate,
 	/* Save as primary spool */
 	buildstate->spool = btspool;
 
+	/* Report heap scan phase started */
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+								 PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN);
+
 	/* Attempt to launch parallel worker scan when required */
 	if (indexInfo->ii_ParallelWorkers > 0)
 		_bt_begin_parallel(buildstate, indexInfo->ii_Concurrent,
@@ -480,13 +486,31 @@ _bt_spools_heapscan(Relation heap, Relation index, BTBuildState *buildstate,
 
 	/* Fill spool using either serial or parallel heap scan */
 	if (!buildstate->btleader)
-		reltuples = table_index_build_scan(heap, index, indexInfo, true,
+		reltuples = table_index_build_scan(heap, index, indexInfo, true, true,
 										   _bt_build_callback, (void *) buildstate,
 										   NULL);
 	else
 		reltuples = _bt_parallel_heapscan(buildstate,
 										  &indexInfo->ii_BrokenHotChain);
 
+	/*
+	 * Set the progress target for the next phase.  Reset the block number
+	 * values set by IndexBuildHeapScan
+	 */
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_TUPLES_TOTAL,
+			PROGRESS_SCAN_BLOCKS_TOTAL,
+			PROGRESS_SCAN_BLOCKS_DONE
+		};
+		const int64 val[] = {
+			buildstate->indtuples,
+			0, 0
+		};
+
+		pgstat_progress_update_multi_param(3, index, val);
+	}
+
 	/* okay, all heap tuples are spooled */
 	if (buildstate->spool2 && !buildstate->havedead)
 	{
@@ -535,9 +559,15 @@ _bt_leafbuild(BTSpool *btspool, BTSpool *btspool2)
 	}
 #endif							/* BTREE_BUILD_STATS */
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+								 PROGRESS_BTREE_PHASE_PERFORMSORT_1);
 	tuplesort_performsort(btspool->sortstate);
 	if (btspool2)
+	{
+		pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+									 PROGRESS_BTREE_PHASE_PERFORMSORT_2);
 		tuplesort_performsort(btspool2->sortstate);
+	}
 
 	wstate.heap = btspool->heap;
 	wstate.index = btspool->index;
@@ -554,6 +584,8 @@ _bt_leafbuild(BTSpool *btspool, BTSpool *btspool2)
 	wstate.btws_pages_written = 0;
 	wstate.btws_zeropage = NULL;	/* until needed */
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+								 PROGRESS_BTREE_PHASE_LEAF_LOAD);
 	_bt_load(&wstate, btspool, btspool2);
 }
 
@@ -1098,6 +1130,7 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
 	int			i,
 				keysz = IndexRelationGetNumberOfKeyAttributes(wstate->index);
 	SortSupport sortKeys;
+	long		tuples_done = 0L;
 
 	if (merge)
 	{
@@ -1202,6 +1235,10 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
 				_bt_buildadd(wstate, state, itup2);
 				itup2 = tuplesort_getindextuple(btspool2->sortstate, true);
 			}
+
+			/* Report progress */
+			pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE,
+										 ++tuples_done);
 		}
 		pfree(sortKeys);
 	}
@@ -1216,6 +1253,10 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
 				state = _bt_pagestate(wstate, 0);
 
 			_bt_buildadd(wstate, state, itup);
+
+			/* Report progress */
+			pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE,
+										 ++tuples_done);
 		}
 	}
 
@@ -1352,6 +1393,11 @@ _bt_begin_parallel(BTBuildState *buildstate, bool isconcurrent, int request)
 								  ParallelTableScanFromBTShared(btshared),
 								  snapshot);
 
+	/* Report total number of blocks to scan */
+	pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL,
+								 ((ParallelBlockTableScanDescData *)
+								 ParallelTableScanFromBTShared(btshared))->phs_nblocks);
+
 	/*
 	 * Store shared tuplesort-private state, for which we reserved space.
 	 * Then, initialize opaque state using tuplesort routine.
@@ -1528,7 +1574,7 @@ _bt_leader_participate_as_worker(BTBuildState *buildstate)
 	/* Perform work common to all participants */
 	_bt_parallel_scan_and_sort(leaderworker, leaderworker2, btleader->btshared,
 							   btleader->sharedsort, btleader->sharedsort2,
-							   sortmem);
+							   sortmem, true);
 
 #ifdef BTREE_BUILD_STATS
 	if (log_btree_build_stats)
@@ -1619,7 +1665,7 @@ _bt_parallel_build_main(dsm_segment *seg, shm_toc *toc)
 	/* Perform sorting of spool, and possibly a spool2 */
 	sortmem = maintenance_work_mem / btshared->scantuplesortstates;
 	_bt_parallel_scan_and_sort(btspool, btspool2, btshared, sharedsort,
-							   sharedsort2, sortmem);
+							   sharedsort2, sortmem, false);
 
 #ifdef BTREE_BUILD_STATS
 	if (log_btree_build_stats)
@@ -1648,7 +1694,7 @@ _bt_parallel_build_main(dsm_segment *seg, shm_toc *toc)
 static void
 _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2,
 						   BTShared *btshared, Sharedsort *sharedsort,
-						   Sharedsort *sharedsort2, int sortmem)
+						   Sharedsort *sharedsort2, int sortmem, bool progress)
 {
 	SortCoordinate coordinate;
 	BTBuildState buildstate;
@@ -1705,9 +1751,10 @@ _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2,
 	/* Join parallel scan */
 	indexInfo = BuildIndexInfo(btspool->index);
 	indexInfo->ii_Concurrent = btshared->isconcurrent;
-	scan = table_beginscan_parallel(btspool->heap, ParallelTableScanFromBTShared(btshared));
+	scan = table_beginscan_parallel(btspool->heap,
+									ParallelTableScanFromBTShared(btshared));
 	reltuples = table_index_build_scan(btspool->heap, btspool->index, indexInfo,
-									   true, _bt_build_callback,
+									   true, progress, _bt_build_callback,
 									   (void *) &buildstate, scan);
 
 	/*
diff --git a/src/backend/access/nbtree/nbtutils.c b/src/backend/access/nbtree/nbtutils.c
index 92b8b5f134d..9e8a7bd3d9a 100644
--- a/src/backend/access/nbtree/nbtutils.c
+++ b/src/backend/access/nbtree/nbtutils.c
@@ -20,6 +20,7 @@
 #include "access/nbtree.h"
 #include "access/reloptions.h"
 #include "access/relscan.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
 #include "utils/array.h"
 #include "utils/datum.h"
@@ -2048,6 +2049,29 @@ btproperty(Oid index_oid, int attno,
 	}
 }
 
+/*
+ *	btbuildphasename() -- Return name of index build phase.
+ */
+char *
+btbuildphasename(int64 phasenum)
+{
+	switch (phasenum)
+	{
+		case PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE:
+			return "initializing";
+		case PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN:
+			return "table scan";
+		case PROGRESS_BTREE_PHASE_PERFORMSORT_1:
+			return "sorting tuples, spool 1";
+		case PROGRESS_BTREE_PHASE_PERFORMSORT_2:
+			return "sorting tuples, spool 2";
+		case PROGRESS_BTREE_PHASE_LEAF_LOAD:
+			return "btree tuple loading";
+		default:
+			return NULL;
+	}
+}
+
 /*
  *	_bt_truncate() -- create tuple without unneeded suffix attributes.
  *
diff --git a/src/backend/access/spgist/spginsert.c b/src/backend/access/spgist/spginsert.c
index 390ad9ac51f..282d6998cf0 100644
--- a/src/backend/access/spgist/spginsert.c
+++ b/src/backend/access/spgist/spginsert.c
@@ -143,7 +143,7 @@ spgbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 											  "SP-GiST build temporary context",
 											  ALLOCSET_DEFAULT_SIZES);
 
-	reltuples = table_index_build_scan(heap, index, indexInfo, true,
+	reltuples = table_index_build_scan(heap, index, indexInfo, true, false,
 									   spgistBuildCallback, (void *) &buildstate,
 									   NULL);
 
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 8e63c1fad25..45472db147b 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -67,6 +67,7 @@ spghandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = spgcostestimate;
 	amroutine->amoptions = spgoptions;
 	amroutine->amproperty = spgproperty;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = spgvalidate;
 	amroutine->ambeginscan = spgbeginscan;
 	amroutine->amrescan = spgrescan;
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 0d9d405c548..6c6818e891c 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -51,9 +51,9 @@
 #include "catalog/pg_trigger.h"
 #include "catalog/pg_type.h"
 #include "catalog/storage.h"
-#include "commands/tablecmds.h"
 #include "commands/event_trigger.h"
 #include "commands/progress.h"
+#include "commands/tablecmds.h"
 #include "commands/trigger.h"
 #include "executor/executor.h"
 #include "miscadmin.h"
@@ -2047,7 +2047,7 @@ index_drop(Oid indexId, bool concurrent, bool concurrent_lock_mode)
 		 * to acquire an exclusive lock on our table.  The lock code will
 		 * detect deadlock and error out properly.
 		 */
-		WaitForLockers(heaplocktag, AccessExclusiveLock);
+		WaitForLockers(heaplocktag, AccessExclusiveLock, true);
 
 		/* Finish invalidation of index and mark it as dead */
 		index_concurrently_set_dead(heapId, indexId);
@@ -2063,7 +2063,7 @@ index_drop(Oid indexId, bool concurrent, bool concurrent_lock_mode)
 		 * Wait till every transaction that saw the old index state has
 		 * finished.
 		 */
-		WaitForLockers(heaplocktag, AccessExclusiveLock);
+		WaitForLockers(heaplocktag, AccessExclusiveLock, true);
 
 		/*
 		 * Re-open relations to allow us to complete our actions.
@@ -2712,6 +2712,25 @@ index_build(Relation heapRelation,
 						   save_sec_context | SECURITY_RESTRICTED_OPERATION);
 	save_nestlevel = NewGUCNestLevel();
 
+	/* Set up initial progress report status */
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_PHASE,
+			PROGRESS_CREATEIDX_SUBPHASE,
+			PROGRESS_CREATEIDX_TUPLES_DONE,
+			PROGRESS_CREATEIDX_TUPLES_TOTAL,
+			PROGRESS_SCAN_BLOCKS_DONE,
+			PROGRESS_SCAN_BLOCKS_TOTAL
+		};
+		const int64	val[] = {
+			PROGRESS_CREATEIDX_PHASE_BUILD,
+			PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE,
+			0, 0, 0, 0
+		};
+
+		pgstat_progress_update_multi_param(6, index, val);
+	}
+
 	/*
 	 * Call the access method's build procedure
 	 */
@@ -3000,6 +3019,21 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot)
 	int			save_sec_context;
 	int			save_nestlevel;
 
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_PHASE,
+			PROGRESS_CREATEIDX_TUPLES_DONE,
+			PROGRESS_CREATEIDX_TUPLES_TOTAL,
+			PROGRESS_SCAN_BLOCKS_DONE,
+			PROGRESS_SCAN_BLOCKS_TOTAL
+		};
+		const int64	val[] = {
+			PROGRESS_CREATEIDX_PHASE_VALIDATE_IDXSCAN,
+			0, 0, 0, 0
+		};
+		pgstat_progress_update_multi_param(5, index, val);
+	}
+
 	/* Open and lock the parent heap relation */
 	heapRelation = table_open(heapId, ShareUpdateExclusiveLock);
 	/* And the target index relation */
@@ -3030,6 +3064,7 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot)
 	 */
 	ivinfo.index = indexRelation;
 	ivinfo.analyze_only = false;
+	ivinfo.report_progress = true;	/* XXX only for btree? */
 	ivinfo.estimated_count = true;
 	ivinfo.message_level = DEBUG2;
 	ivinfo.num_heap_tuples = heapRelation->rd_rel->reltuples;
@@ -3047,15 +3082,39 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot)
 											NULL, false);
 	state.htups = state.itups = state.tups_inserted = 0;
 
+	/* ambulkdelete updates progress metrics */
 	(void) index_bulk_delete(&ivinfo, NULL,
 							 validate_index_callback, (void *) &state);
 
 	/* Execute the sort */
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_PHASE,
+			PROGRESS_SCAN_BLOCKS_DONE,
+			PROGRESS_SCAN_BLOCKS_TOTAL
+		};
+		const int64	val[] = {
+			PROGRESS_CREATEIDX_PHASE_SORT_IDXSCAN,
+			0, 0
+		};
+
+		pgstat_progress_update_multi_param(3, index, val);
+	}
 	tuplesort_performsort(state.tuplesort);
 
 	/*
-	 * Now scan the heap and "merge" it with the index
+	 * Now scan the heap and "merge" it with the index.
 	 */
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_PHASE,
+		};
+		const int64	val[] = {
+			PROGRESS_CREATEIDX_PHASE_HEAPSCAN_VALIDATE
+		};
+
+		pgstat_progress_update_multi_param(1, index, val);
+	}
 	table_index_validate_scan(heapRelation,
 							  indexRelation,
 							  indexInfo,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index b89df70653e..acf277473c1 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -934,6 +934,33 @@ CREATE VIEW pg_stat_progress_cluster AS
     FROM pg_stat_get_progress_info('CLUSTER') AS S
         LEFT JOIN pg_database D ON S.datid = D.oid;
 
+CREATE VIEW pg_stat_progress_create_index AS
+	SELECT
+		S.pid AS pid, S.datid AS datid, D.datname AS datname,
+		S.relid AS relid,
+		CASE S.param10 WHEN 0 THEN 'initializing'
+					  WHEN 1 THEN 'waiting for old snapshots'
+					  WHEN 2 THEN 'building index' ||
+						COALESCE((': ' || pg_indexam_progress_phasename(S.param9::oid, S.param11)),
+							'')
+					  WHEN 3 THEN 'waiting for writer snapshots'
+					  WHEN 4 THEN 'index validation: scan index'
+					  WHEN 5 THEN 'index validation: sort index scan results'
+					  WHEN 6 THEN 'index validation: scan heap'
+					  WHEN 7 THEN 'waiting for reader snapshots'
+					  END as phase,
+		S.param4 AS lockers_total,
+		S.param5 AS lockers_done,
+		S.param6 AS current_locker_pid,
+		S.param16 AS blocks_total,
+		S.param17 AS blocks_done,
+		S.param12 AS tuples_total,
+		S.param13 AS tuples_done,
+		S.param14 AS partitions_total,
+		S.param15 AS partitions_done
+	FROM pg_stat_get_progress_info('CREATE INDEX') AS S
+		LEFT JOIN pg_database D ON S.datid = D.oid;
+
 CREATE VIEW pg_user_mappings AS
     SELECT
         U.oid       AS umid,
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index ea07be69dbd..905e48d458c 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -36,6 +36,7 @@
 #include "commands/dbcommands.h"
 #include "commands/defrem.h"
 #include "commands/event_trigger.h"
+#include "commands/progress.h"
 #include "commands/tablecmds.h"
 #include "commands/tablespace.h"
 #include "mb/pg_wchar.h"
@@ -47,10 +48,12 @@
 #include "parser/parse_func.h"
 #include "parser/parse_oper.h"
 #include "partitioning/partdesc.h"
+#include "pgstat.h"
 #include "rewrite/rewriteManip.h"
 #include "storage/lmgr.h"
 #include "storage/proc.h"
 #include "storage/procarray.h"
+#include "storage/sinvaladt.h"
 #include "utils/acl.h"
 #include "utils/builtins.h"
 #include "utils/fmgroids.h"
@@ -334,7 +337,7 @@ CheckIndexCompatible(Oid oldId,
  * doesn't show up in the output, we know we can forget about it.
  */
 static void
-WaitForOlderSnapshots(TransactionId limitXmin)
+WaitForOlderSnapshots(TransactionId limitXmin, bool progress)
 {
 	int			n_old_snapshots;
 	int			i;
@@ -343,6 +346,8 @@ WaitForOlderSnapshots(TransactionId limitXmin)
 	old_snapshots = GetCurrentVirtualXIDs(limitXmin, true, false,
 										  PROC_IS_AUTOVACUUM | PROC_IN_VACUUM,
 										  &n_old_snapshots);
+	if (progress)
+		pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, n_old_snapshots);
 
 	for (i = 0; i < n_old_snapshots; i++)
 	{
@@ -378,7 +383,19 @@ WaitForOlderSnapshots(TransactionId limitXmin)
 		}
 
 		if (VirtualTransactionIdIsValid(old_snapshots[i]))
+		{
+			if (progress)
+			{
+				PGPROC *holder = BackendIdGetProc(old_snapshots[i].backendId);
+
+				pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID,
+											 holder->pid);
+			}
 			VirtualXactLock(old_snapshots[i], true);
+		}
+
+		if (progress)
+			pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, i + 1);
 	}
 }
 
@@ -452,6 +469,15 @@ DefineIndex(Oid relationId,
 	Snapshot	snapshot;
 	int			i;
 
+
+	/*
+	 * Start progress report.  If we're building a partition, this was already
+	 * done.
+	 */
+	if (!OidIsValid(parentIndexId))
+		pgstat_progress_start_command(PROGRESS_COMMAND_CREATE_INDEX,
+									  relationId);
+
 	/*
 	 * count key attributes in index
 	 */
@@ -668,6 +694,9 @@ DefineIndex(Oid relationId,
 	accessMethodId = accessMethodForm->oid;
 	amRoutine = GetIndexAmRoutine(accessMethodForm->amhandler);
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_ACCESS_METHOD_OID,
+								 accessMethodId);
+
 	if (stmt->unique && !amRoutine->amcanunique)
 		ereport(ERROR,
 				(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -948,6 +977,11 @@ DefineIndex(Oid relationId,
 	if (!OidIsValid(indexRelationId))
 	{
 		table_close(rel, NoLock);
+
+		/* If this is the top-level index, we're done */
+		if (!OidIsValid(parentIndexId))
+			pgstat_progress_end_command();
+
 		return address;
 	}
 
@@ -973,6 +1007,9 @@ DefineIndex(Oid relationId,
 			TupleDesc	parentDesc;
 			Oid		   *opfamOids;
 
+			pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL,
+										 nparts);
+
 			memcpy(part_oids, partdesc->oids, sizeof(Oid) * nparts);
 
 			parentDesc = CreateTupleDescCopy(RelationGetDescr(rel));
@@ -1122,6 +1159,8 @@ DefineIndex(Oid relationId,
 								skip_build, quiet);
 				}
 
+				pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_DONE,
+											 i + 1);
 				pfree(attmap);
 			}
 
@@ -1156,6 +1195,8 @@ DefineIndex(Oid relationId,
 		 * Indexes on partitioned tables are not themselves built, so we're
 		 * done here.
 		 */
+		if (!OidIsValid(parentIndexId))
+			pgstat_progress_end_command();
 		return address;
 	}
 
@@ -1163,6 +1204,11 @@ DefineIndex(Oid relationId,
 	{
 		/* Close the heap and we're done, in the non-concurrent case */
 		table_close(rel, NoLock);
+
+		/* If this is the top-level index, we're done. */
+		if (!OidIsValid(parentIndexId))
+			pgstat_progress_end_command();
+
 		return address;
 	}
 
@@ -1214,7 +1260,9 @@ DefineIndex(Oid relationId,
 	 * exclusive lock on our table.  The lock code will detect deadlock and
 	 * error out properly.
 	 */
-	WaitForLockers(heaplocktag, ShareLock);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_1);
+	WaitForLockers(heaplocktag, ShareLock, true);
 
 	/*
 	 * At this moment we are sure that there are no transactions with the
@@ -1255,7 +1303,9 @@ DefineIndex(Oid relationId,
 	 * We once again wait until no transaction can have the table open with
 	 * the index marked as read-only for updates.
 	 */
-	WaitForLockers(heaplocktag, ShareLock);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_2);
+	WaitForLockers(heaplocktag, ShareLock, true);
 
 	/*
 	 * Now take the "reference snapshot" that will be used by validate_index()
@@ -1312,7 +1362,9 @@ DefineIndex(Oid relationId,
 	 * before the reference snap was taken, we have to wait out any
 	 * transactions that might have older snapshots.
 	 */
-	WaitForOlderSnapshots(limitXmin);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_3);
+	WaitForOlderSnapshots(limitXmin, true);
 
 	/*
 	 * Index can now be marked valid -- update its pg_index entry
@@ -1334,6 +1386,8 @@ DefineIndex(Oid relationId,
 	 */
 	UnlockRelationIdForSession(&heaprelid, ShareUpdateExclusiveLock);
 
+	pgstat_progress_end_command();
+
 	return address;
 }
 
@@ -2913,7 +2967,7 @@ ReindexRelationConcurrently(Oid relationOid, int options)
 	 * DefineIndex() for more details.
 	 */
 
-	WaitForLockersMultiple(lockTags, ShareLock);
+	WaitForLockersMultiple(lockTags, ShareLock, false);
 	CommitTransactionCommand();
 
 	forboth(lc, indexIds, lc2, newIndexIds)
@@ -2955,7 +3009,7 @@ ReindexRelationConcurrently(Oid relationOid, int options)
 	 * for more details.
 	 */
 
-	WaitForLockersMultiple(lockTags, ShareLock);
+	WaitForLockersMultiple(lockTags, ShareLock, false);
 	CommitTransactionCommand();
 
 	foreach(lc, newIndexIds)
@@ -3003,7 +3057,7 @@ ReindexRelationConcurrently(Oid relationOid, int options)
 		 * before the reference snap was taken, we have to wait out any
 		 * transactions that might have older snapshots.
 		 */
-		WaitForOlderSnapshots(limitXmin);
+		WaitForOlderSnapshots(limitXmin, false);
 
 		CommitTransactionCommand();
 	}
@@ -3074,7 +3128,7 @@ ReindexRelationConcurrently(Oid relationOid, int options)
 	 * index_drop() for more details.
 	 */
 
-	WaitForLockersMultiple(lockTags, AccessExclusiveLock);
+	WaitForLockersMultiple(lockTags, AccessExclusiveLock, false);
 
 	foreach(lc, indexIds)
 	{
@@ -3096,7 +3150,7 @@ ReindexRelationConcurrently(Oid relationOid, int options)
 	 * Drop the old indexes.
 	 */
 
-	WaitForLockersMultiple(lockTags, AccessExclusiveLock);
+	WaitForLockersMultiple(lockTags, AccessExclusiveLock, false);
 
 	PushActiveSnapshot(GetTransactionSnapshot());
 
diff --git a/src/backend/storage/ipc/standby.c b/src/backend/storage/ipc/standby.c
index cd56dca3aef..215f1463bb1 100644
--- a/src/backend/storage/ipc/standby.c
+++ b/src/backend/storage/ipc/standby.c
@@ -401,7 +401,7 @@ ResolveRecoveryConflictWithLock(LOCKTAG locktag)
 		 */
 		VirtualTransactionId *backends;
 
-		backends = GetLockConflicts(&locktag, AccessExclusiveLock);
+		backends = GetLockConflicts(&locktag, AccessExclusiveLock, NULL);
 		ResolveRecoveryConflictWithVirtualXIDs(backends,
 											   PROCSIG_RECOVERY_CONFLICT_LOCK);
 	}
diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c
index e688ba81170..0b04b093782 100644
--- a/src/backend/storage/lmgr/lmgr.c
+++ b/src/backend/storage/lmgr/lmgr.c
@@ -19,9 +19,12 @@
 #include "access/transam.h"
 #include "access/xact.h"
 #include "catalog/catalog.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
+#include "pgstat.h"
 #include "storage/lmgr.h"
 #include "storage/procarray.h"
+#include "storage/sinvaladt.h"
 #include "utils/inval.h"
 
 
@@ -857,10 +860,12 @@ XactLockTableWaitErrorCb(void *arg)
  * after we obtained our initial list of lockers, we will not wait for them.
  */
 void
-WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
+WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
 {
 	List	   *holders = NIL;
 	ListCell   *lc;
+	int			total = 0;
+	int			done = 0;
 
 	/* Done if no locks to wait for */
 	if (list_length(locktags) == 0)
@@ -870,10 +875,17 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
 	foreach(lc, locktags)
 	{
 		LOCKTAG    *locktag = lfirst(lc);
+		int			count;
 
-		holders = lappend(holders, GetLockConflicts(locktag, lockmode));
+		holders = lappend(holders,
+						  GetLockConflicts(locktag, lockmode,
+										   progress ? &count : NULL));
+		total += count;
 	}
 
+	if (progress)
+		pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, total);
+
 	/*
 	 * Note: GetLockConflicts() never reports our own xid, hence we need not
 	 * check for that.  Also, prepared xacts are not reported, which is fine
@@ -887,10 +899,36 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
 
 		while (VirtualTransactionIdIsValid(*lockholders))
 		{
+			/*
+			 * If requested, publish who we're going to wait for.  This is not
+			 * 100% accurate if they're already gone, but we don't care.
+			 */
+			if (progress)
+			{
+				PGPROC *holder = BackendIdGetProc(lockholders->backendId);
+
+				pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID,
+											 holder->pid);
+			}
 			VirtualXactLock(*lockholders, true);
 			lockholders++;
+
+			if (progress)
+				pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, ++done);
 		}
 	}
+	if (progress)
+	{
+		const int	index[] = {
+			PROGRESS_WAITFOR_TOTAL,
+			PROGRESS_WAITFOR_DONE,
+			PROGRESS_WAITFOR_CURRENT_PID
+		};
+		const int64	values[] = {
+			0, 0, 0
+		};
+		pgstat_progress_update_multi_param(3, index, values);
+	}
 
 	list_free_deep(holders);
 }
@@ -901,12 +939,12 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
  * Same as WaitForLockersMultiple, for a single lock tag.
  */
 void
-WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode)
+WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress)
 {
 	List	   *l;
 
 	l = list_make1(&heaplocktag);
-	WaitForLockersMultiple(l, lockmode);
+	WaitForLockersMultiple(l, lockmode, progress);
 	list_free(l);
 }
 
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 78fdbd6ff88..c8958766f1e 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2807,6 +2807,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock)
  *		xacts merely awaiting such a lock are NOT reported.
  *
  * The result array is palloc'd and is terminated with an invalid VXID.
+ * *countp, if not null, is updated to the number of items set.
  *
  * Of course, the result could be out of date by the time it's returned,
  * so use of this function has to be thought about carefully.
@@ -2817,7 +2818,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock)
  * uses of the result.
  */
 VirtualTransactionId *
-GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode)
+GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
 {
 	static VirtualTransactionId *vxids;
 	LOCKMETHODID lockmethodid = locktag->locktag_lockmethodid;
@@ -2964,6 +2965,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode)
 		LWLockRelease(partitionLock);
 		vxids[count].backendId = InvalidBackendId;
 		vxids[count].localTransactionId = InvalidLocalTransactionId;
+		if (countp)
+			*countp = count;
 		return vxids;
 	}
 
@@ -3019,6 +3022,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode)
 
 	vxids[count].backendId = InvalidBackendId;
 	vxids[count].localTransactionId = InvalidLocalTransactionId;
+	if (countp)
+		*countp = count;
 	return vxids;
 }
 
diff --git a/src/backend/utils/adt/amutils.c b/src/backend/utils/adt/amutils.c
index 060ffe501ec..e81d6cc0562 100644
--- a/src/backend/utils/adt/amutils.c
+++ b/src/backend/utils/adt/amutils.c
@@ -445,3 +445,26 @@ pg_index_column_has_property(PG_FUNCTION_ARGS)
 
 	return indexam_property(fcinfo, propname, InvalidOid, relid, attno);
 }
+
+/*
+ * Return the name of the given phase, as used for progress reporting by the
+ * given AM.
+ */
+Datum
+pg_indexam_progress_phasename(PG_FUNCTION_ARGS)
+{
+	Oid			amoid = PG_GETARG_OID(0);
+	int32		phasenum = PG_GETARG_INT32(1);
+	IndexAmRoutine *routine;
+	char	   *name;
+
+	routine = GetIndexAmRoutineByAmId(amoid, true);
+	if (routine == NULL || !routine->ambuildphasename)
+		PG_RETURN_NULL();
+
+	name = routine->ambuildphasename(phasenum);
+	if (!name)
+		PG_RETURN_NULL();
+
+	PG_RETURN_TEXT_P(CStringGetTextDatum(name));
+}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 90a817a25c5..7c2afe64272 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -470,6 +470,8 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
 		cmdtype = PROGRESS_COMMAND_VACUUM;
 	else if (pg_strcasecmp(cmd, "CLUSTER") == 0)
 		cmdtype = PROGRESS_COMMAND_CLUSTER;
+	else if (pg_strcasecmp(cmd, "CREATE INDEX") == 0)
+		cmdtype = PROGRESS_COMMAND_CREATE_INDEX;
 	else
 		ereport(ERROR,
 				(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 653ddc976ba..09a7404267c 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -108,6 +108,9 @@ typedef bool (*amproperty_function) (Oid index_oid, int attno,
 									 IndexAMProperty prop, const char *propname,
 									 bool *res, bool *isnull);
 
+/* name of phase as used in progress reporting */
+typedef char *(*ambuildphasename_function) (int64 phasenum);
+
 /* validate definition of an opclass for this AM */
 typedef bool (*amvalidate_function) (Oid opclassoid);
 
@@ -213,6 +216,7 @@ typedef struct IndexAmRoutine
 	amcostestimate_function amcostestimate;
 	amoptions_function amoptions;
 	amproperty_function amproperty; /* can be NULL */
+	ambuildphasename_function ambuildphasename;	/* can be NULL */
 	amvalidate_function amvalidate;
 	ambeginscan_function ambeginscan;
 	amrescan_function amrescan;
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 70c7351a08c..9717183ef23 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -45,6 +45,7 @@ typedef struct IndexVacuumInfo
 {
 	Relation	index;			/* the index being vacuumed */
 	bool		analyze_only;	/* ANALYZE (without any actual vacuum) */
+	bool		report_progress;	/* emit progress.h status reports */
 	bool		estimated_count;	/* num_heap_tuples is an estimate */
 	int			message_level;	/* ereport level for progress messages */
 	double		num_heap_tuples;	/* tuples remaining in heap */
diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h
index 473c6f29185..bc0f3027629 100644
--- a/src/include/access/nbtree.h
+++ b/src/include/access/nbtree.h
@@ -672,6 +672,16 @@ typedef BTScanOpaqueData *BTScanOpaque;
 #define SK_BT_DESC			(INDOPTION_DESC << SK_BT_INDOPTION_SHIFT)
 #define SK_BT_NULLS_FIRST	(INDOPTION_NULLS_FIRST << SK_BT_INDOPTION_SHIFT)
 
+/*
+ * Constant definition for progress reporting.  Phase numbers must match
+ * btbuildphasename.
+ */
+/* PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE is 1 (see progress.h) */
+#define PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN		2
+#define PROGRESS_BTREE_PHASE_PERFORMSORT_1				3
+#define PROGRESS_BTREE_PHASE_PERFORMSORT_2				4
+#define PROGRESS_BTREE_PHASE_LEAF_LOAD					5
+
 /*
  * external entry points for btree, in nbtree.c
  */
@@ -785,6 +795,7 @@ extern bytea *btoptions(Datum reloptions, bool validate);
 extern bool btproperty(Oid index_oid, int attno,
 		   IndexAMProperty prop, const char *propname,
 		   bool *res, bool *isnull);
+extern char *btbuildphasename(int64 phasenum);
 extern IndexTuple _bt_truncate(Relation rel, IndexTuple lastleft,
 			 IndexTuple firstright, BTScanInsert itup_key);
 extern int _bt_keep_natts_fast(Relation rel, IndexTuple lastleft,
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index c571f8a8994..a5565d1ef43 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -424,6 +424,7 @@ typedef struct TableAmRoutine
 										   struct IndexInfo *index_nfo,
 										   bool allow_sync,
 										   bool anyvisible,
+										   bool progress,
 										   BlockNumber start_blockno,
 										   BlockNumber end_blockno,
 										   IndexBuildCallback callback,
@@ -1095,6 +1096,11 @@ table_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
  * so here because the AM might reject some of the tuples for its own reasons,
  * such as being unable to store NULLs.
  *
+ * If 'progress', we update the PROGRESS_SCAN_BLOCKS_DONE counter as we go
+ * along.  Also, if that flag is true and a scan descriptor is not passed (ie.
+ * when not doing a parallel scan), the PROGRESS_SCAN_BLOCKS_TOTAL counter is
+ * updated at the beginning.  For parallel scans, caller is expected to have
+ * set the total number of blocks prior to calling this function.
  *
  * A side effect is to set indexInfo->ii_BrokenHotChain to true if we detect
  * any potentially broken HOT chains.  Currently, we set this if there are any
@@ -1108,6 +1114,7 @@ table_index_build_scan(Relation heap_rel,
 					   Relation index_rel,
 					   struct IndexInfo *index_nfo,
 					   bool allow_sync,
+					   bool progress,
 					   IndexBuildCallback callback,
 					   void *callback_state,
 					   TableScanDesc scan)
@@ -1117,6 +1124,7 @@ table_index_build_scan(Relation heap_rel,
 														index_nfo,
 														allow_sync,
 														false,
+														progress,
 														0,
 														InvalidBlockNumber,
 														callback,
@@ -1140,6 +1148,7 @@ table_index_build_range_scan(Relation heap_rel,
 							 struct IndexInfo *index_nfo,
 							 bool allow_sync,
 							 bool anyvisible,
+							 bool progress,
 							 BlockNumber start_blockno,
 							 BlockNumber numblocks,
 							 IndexBuildCallback callback,
@@ -1151,6 +1160,7 @@ table_index_build_range_scan(Relation heap_rel,
 														index_nfo,
 														allow_sync,
 														anyvisible,
+														progress,
 														start_blockno,
 														numblocks,
 														callback,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index eac909109c5..a7050edca09 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -924,6 +924,10 @@
   proname => 'pg_index_column_has_property', provolatile => 's',
   prorettype => 'bool', proargtypes => 'regclass int4 text',
   prosrc => 'pg_index_column_has_property' },
+{ oid => '676', descr => 'return name of given index build phase',
+  proname => 'pg_indexam_progress_phasename', provolatile => 'i',
+  prorettype => 'text', proargtypes => 'oid int8',
+  prosrc => 'pg_indexam_progress_phasename' },
 
 { oid => '339',
   proname => 'poly_same', prorettype => 'bool',
@@ -5122,9 +5126,9 @@
   proname => 'pg_stat_get_progress_info', prorows => '100', proretset => 't',
   provolatile => 's', proparallel => 'r', prorettype => 'record',
   proargtypes => 'text',
-  proallargtypes => '{text,int4,oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}',
-  proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o}',
-  proargnames => '{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10}',
+  proallargtypes => '{text,int4,oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}',
+  proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+  proargnames => '{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10,param11,param12,param13,param14,param15,param16,param17,param18,param19,param20}',
   prosrc => 'pg_stat_get_progress_info' },
 { oid => '3099',
   descr => 'statistics: information about currently active replication',
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index 04542d9e923..079303cff46 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -44,7 +44,7 @@
 #define PROGRESS_CLUSTER_HEAP_BLKS_SCANNED		6
 #define PROGRESS_CLUSTER_INDEX_REBUILD_COUNT	7
 
-/* Phases of cluster (as dvertised via PROGRESS_CLUSTER_PHASE) */
+/* Phases of cluster (as advertised via PROGRESS_CLUSTER_PHASE) */
 #define PROGRESS_CLUSTER_PHASE_SEQ_SCAN_HEAP	1
 #define PROGRESS_CLUSTER_PHASE_INDEX_SCAN_HEAP	2
 #define PROGRESS_CLUSTER_PHASE_SORT_TUPLES		3
@@ -57,4 +57,39 @@
 #define PROGRESS_CLUSTER_COMMAND_CLUSTER		1
 #define PROGRESS_CLUSTER_COMMAND_VACUUM_FULL	2
 
+/* Progress parameters for CREATE INDEX */
+/* 3, 4 and 5 reserved for "waitfor" metrics */
+#define PROGRESS_CREATEIDX_ACCESS_METHOD_OID	8
+#define PROGRESS_CREATEIDX_PHASE				9	/* AM-agnostic phase # */
+#define PROGRESS_CREATEIDX_SUBPHASE				10	/* phase # filled by AM */
+#define PROGRESS_CREATEIDX_TUPLES_TOTAL			11
+#define PROGRESS_CREATEIDX_TUPLES_DONE			12
+#define PROGRESS_CREATEIDX_PARTITIONS_TOTAL		13
+#define PROGRESS_CREATEIDX_PARTITIONS_DONE		14
+/* 15 and 16 reserved for "block number" metrics */
+
+/* Phases of CREATE INDEX (as advertised via PROGRESS_CREATEIDX_PHASE) */
+#define PROGRESS_CREATEIDX_PHASE_WAIT_1			1
+#define PROGRESS_CREATEIDX_PHASE_BUILD			2
+#define PROGRESS_CREATEIDX_PHASE_WAIT_2			3
+#define PROGRESS_CREATEIDX_PHASE_VALIDATE_IDXSCAN		4
+#define PROGRESS_CREATEIDX_PHASE_SORT_IDXSCAN		5
+#define PROGRESS_CREATEIDX_PHASE_HEAPSCAN_VALIDATE		6
+#define PROGRESS_CREATEIDX_PHASE_WAIT_3			7
+
+/*
+ * Subphases of CREATE INDEX, for index_build.
+ */
+#define PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE	1
+/* Additional phases are defined by each AM */
+
+/* Lock holder wait counts */
+#define PROGRESS_WAITFOR_TOTAL					3
+#define PROGRESS_WAITFOR_DONE					4
+#define PROGRESS_WAITFOR_CURRENT_PID			5
+
+/* Block numbers in a generic relation scan */
+#define PROGRESS_SCAN_BLOCKS_TOTAL				15
+#define PROGRESS_SCAN_BLOCKS_DONE				16
+
 #endif
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index c080fa6388f..53d4a9c4319 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -951,10 +951,11 @@ typedef enum ProgressCommandType
 {
 	PROGRESS_COMMAND_INVALID,
 	PROGRESS_COMMAND_VACUUM,
-	PROGRESS_COMMAND_CLUSTER
+	PROGRESS_COMMAND_CLUSTER,
+	PROGRESS_COMMAND_CREATE_INDEX
 } ProgressCommandType;
 
-#define PGSTAT_NUM_PROGRESS_PARAM	10
+#define PGSTAT_NUM_PROGRESS_PARAM	20
 
 /* ----------
  * Shared-memory data structures
diff --git a/src/include/storage/lmgr.h b/src/include/storage/lmgr.h
index 3d705faba5c..4f2872de35f 100644
--- a/src/include/storage/lmgr.h
+++ b/src/include/storage/lmgr.h
@@ -78,8 +78,8 @@ extern void XactLockTableWait(TransactionId xid, Relation rel,
 extern bool ConditionalXactLockTableWait(TransactionId xid);
 
 /* Lock VXIDs, specified by conflicting locktags */
-extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode);
-extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode);
+extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress);
+extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress);
 
 /* Lock an XID for tuple insertion (used to wait for an insertion to finish) */
 extern uint32 SpeculativeInsertionLockAcquire(TransactionId xid);
diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h
index badf7fd682b..048947c50d4 100644
--- a/src/include/storage/lock.h
+++ b/src/include/storage/lock.h
@@ -544,7 +544,7 @@ extern bool LockHeldByMe(const LOCKTAG *locktag, LOCKMODE lockmode);
 extern bool LockHasWaiters(const LOCKTAG *locktag,
 			   LOCKMODE lockmode, bool sessionLock);
 extern VirtualTransactionId *GetLockConflicts(const LOCKTAG *locktag,
-				 LOCKMODE lockmode);
+				 LOCKMODE lockmode, int *countp);
 extern void AtPrepare_Locks(void);
 extern void PostPrepare_Locks(TransactionId xid);
 extern int LockCheckConflicts(LockMethod lockMethodTable,
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index d5f309fbfbe..a9a2910fb55 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1856,7 +1856,33 @@ pg_stat_progress_cluster| SELECT s.pid,
     s.param6 AS heap_blks_total,
     s.param7 AS heap_blks_scanned,
     s.param8 AS index_rebuild_count
-   FROM (pg_stat_get_progress_info('CLUSTER'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10)
+   FROM (pg_stat_get_progress_info('CLUSTER'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12)
+     LEFT JOIN pg_database d ON ((s.datid = d.oid)));
+pg_stat_progress_create_index| SELECT s.pid,
+    s.datid,
+    d.datname,
+    s.relid,
+        CASE s.param2
+            WHEN 0 THEN 'initializing'::text
+            WHEN 1 THEN 'waiting for old snapshots'::text
+            WHEN 2 THEN ('building index'::text || COALESCE((': '::text || pg_indexam_progress_phasename((s.param1)::oid, s.param3)), ''::text))
+            WHEN 3 THEN 'waiting for writer snapshots'::text
+            WHEN 4 THEN 'index validation: scan index'::text
+            WHEN 5 THEN 'index validation: sort index scan results'::text
+            WHEN 6 THEN 'index validation: scan heap'::text
+            WHEN 7 THEN 'waiting for reader snapshots'::text
+            ELSE NULL::text
+        END AS phase,
+    s.param4 AS lockers_total,
+    s.param5 AS lockers_done,
+    s.param6 AS current_locker_pid,
+    s.param7 AS blocks_total,
+    s.param8 AS blocks_done,
+    s.param9 AS tuples_total,
+    s.param10 AS tuples_done,
+    s.param11 AS partitions_total,
+    s.param12 AS partitions_done
+   FROM (pg_stat_get_progress_info('CREATE INDEX'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12)
      LEFT JOIN pg_database d ON ((s.datid = d.oid)));
 pg_stat_progress_vacuum| SELECT s.pid,
     s.datid,
@@ -1878,7 +1904,7 @@ pg_stat_progress_vacuum| SELECT s.pid,
     s.param5 AS index_vacuum_count,
     s.param6 AS max_dead_tuples,
     s.param7 AS num_dead_tuples
-   FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10)
+   FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12)
      LEFT JOIN pg_database d ON ((s.datid = d.oid)));
 pg_stat_replication| SELECT s.pid,
     s.usesysid,
-- 
2.17.1

#47Robert Haas
robertmhaas@gmail.com
In reply to: Alvaro Herrera (#46)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On Fri, Mar 29, 2019 at 2:16 PM Alvaro Herrera <alvherre@2ndquadrant.com> wrote:

I think we should consider a new column of an array type, where we could
put things like the list of PIDs to be waited for, the list of OIDs of
index to rebuild, or the list of partitions to build the index on.

This has to work with a fixed-size chunk of shared memory.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

#48Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Robert Haas (#47)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On 2019-Mar-29, Robert Haas wrote:

On Fri, Mar 29, 2019 at 2:16 PM Alvaro Herrera <alvherre@2ndquadrant.com> wrote:

I think we should consider a new column of an array type, where we could
put things like the list of PIDs to be waited for, the list of OIDs of
index to rebuild, or the list of partitions to build the index on.

This has to work with a fixed-size chunk of shared memory.

Bah, of course.

Maybe we can consider using dynamic shmem for that, and include a
pointer to it in the fixed-size chunk. (It's a bit too late to be
writing this code, mind; I'm just proposing this for a future
improvement.)

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#49Robert Haas
robertmhaas@gmail.com
In reply to: Alvaro Herrera (#48)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On Fri, Mar 29, 2019 at 3:28 PM Alvaro Herrera <alvherre@2ndquadrant.com> wrote:

On 2019-Mar-29, Robert Haas wrote:

On Fri, Mar 29, 2019 at 2:16 PM Alvaro Herrera <alvherre@2ndquadrant.com> wrote:

I think we should consider a new column of an array type, where we could
put things like the list of PIDs to be waited for, the list of OIDs of
index to rebuild, or the list of partitions to build the index on.

This has to work with a fixed-size chunk of shared memory.

Bah, of course.

Maybe we can consider using dynamic shmem for that, and include a
pointer to it in the fixed-size chunk. (It's a bit too late to be
writing this code, mind; I'm just proposing this for a future
improvement.)

Sounds expensive. We don't want to spend a lot of energy pushing out
progress reports which, often enough, nobody will ever examine. I
designed the current system as I did to make it cheap. Adding DSM in
there would open up lots of exciting new failure possibilities and
significantly increase the overhead. And probably add quite a bit of
code complexity, too.

There's probably room for an elaborate progress-reporting facility in
PostgreSQL that can even handle arbitrary stuff like queries. But I
think it might look a lot different from this one, which is designed
and intended to handle simple cases.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

#50Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Robert Haas (#49)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On 2019-Mar-29, Robert Haas wrote:

On Fri, Mar 29, 2019 at 3:28 PM Alvaro Herrera <alvherre@2ndquadrant.com> wrote:

Maybe we can consider using dynamic shmem for that, and include a
pointer to it in the fixed-size chunk. (It's a bit too late to be
writing this code, mind; I'm just proposing this for a future
improvement.)

Sounds expensive. We don't want to spend a lot of energy pushing out
progress reports which, often enough, nobody will ever examine. I
designed the current system as I did to make it cheap.

Well, I'm not proposing this for things that would change more than once
or a very limited number of times during one command; certainly not once
per tuple or per block like other metrics do. The examples I mentioned
are once per command (eg., list of OIDs of partitions to process) or
list of PIDs to wait for, which we wouldn't modify it once set for each
waiting cycle (three times for CREATE INDEX CONCURRENTLY, five times for
REINDEX CONCURRENTLY).

Adding DSM in there would open up lots of exciting new failure
possibilities and significantly increase the overhead. And probably
add quite a bit of code complexity, too.

Yeah, that's true.

Anyway, I'm not intending to tackle this for the time being.

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#51Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Rahila Syed (#30)
2 attachment(s)
Re: monitoring CREATE INDEX [CONCURRENTLY]

Hi Rahila, thanks for reviewing.

On 2019-Mar-25, Rahila Syed wrote:

Please see few comments below:

1. Makecheck fails currently as view definition of expected rules.out does
not reflect latest changes in progress metrics numbering.

Ouch ... fixed.

2. + <entry>
I think there is a typo here 's/partitioned/partitioned table/'

Ah, so there is. Fixed.

3.
I think parallel scan equivalent bpscan->phs_nblocks along with
hscan->rs_nblocks should be used similar to startblock computation above.

Hmm, yeah. I think the way things are setup currently it doesn't matter
much, but it seems fragile to rely on that.

I also moved the reporting of total blocks to scan in the initial table
scan so that it occurs wholly in heapam_index_build_range_scan; I had
originally put that code in _bt_spools_heapscan, but that was a
modularity violation I think. (It didn't make a practical difference,
but it made no sense to have the two cases report the number in wildly
different locations.) Also added a final nblocks metric update after
the scan is done.

I think this patch is done.

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attachments:

v9-0001-Report-progress-of-CREATE-INDEX-operations.patchtext/x-diff; charset=us-asciiDownload
From d916a5daf8059fcd1ab122fba6641dac747837a8 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Mon, 25 Mar 2019 13:29:34 -0300
Subject: [PATCH v9 1/2] Report progress of CREATE INDEX operations

---
 contrib/amcheck/verify_nbtree.c          |   2 +-
 contrib/bloom/blinsert.c                 |   2 +-
 contrib/bloom/blutils.c                  |   1 +
 doc/src/sgml/indexam.sgml                |  13 ++
 doc/src/sgml/monitoring.sgml             | 224 ++++++++++++++++++++++-
 src/backend/access/brin/brin.c           |   5 +-
 src/backend/access/gin/gininsert.c       |   2 +-
 src/backend/access/gin/ginutil.c         |   1 +
 src/backend/access/gist/gist.c           |   1 +
 src/backend/access/gist/gistbuild.c      |   2 +-
 src/backend/access/hash/hash.c           |   3 +-
 src/backend/access/heap/heapam_handler.c | 107 +++++++++++
 src/backend/access/nbtree/nbtree.c       |   9 +
 src/backend/access/nbtree/nbtsort.c      |  56 +++++-
 src/backend/access/nbtree/nbtutils.c     |  24 +++
 src/backend/access/spgist/spginsert.c    |   2 +-
 src/backend/access/spgist/spgutils.c     |   1 +
 src/backend/catalog/index.c              |  67 ++++++-
 src/backend/catalog/system_views.sql     |  27 +++
 src/backend/commands/indexcmds.c         |  72 +++++++-
 src/backend/storage/ipc/standby.c        |   2 +-
 src/backend/storage/lmgr/lmgr.c          |  46 ++++-
 src/backend/storage/lmgr/lock.c          |   7 +-
 src/backend/utils/adt/amutils.c          |  23 +++
 src/backend/utils/adt/pgstatfuncs.c      |   2 +
 src/include/access/amapi.h               |   4 +
 src/include/access/genam.h               |   1 +
 src/include/access/nbtree.h              |  11 ++
 src/include/access/tableam.h             |  10 +
 src/include/catalog/pg_proc.dat          |  10 +-
 src/include/commands/progress.h          |  37 +++-
 src/include/pgstat.h                     |   5 +-
 src/include/storage/lmgr.h               |   4 +-
 src/include/storage/lock.h               |   2 +-
 src/test/regress/expected/rules.out      |  30 ++-
 35 files changed, 769 insertions(+), 46 deletions(-)

diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index 9d5b2e5be67..591e0a3e46a 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -566,7 +566,7 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace,
 			 RelationGetRelationName(state->rel),
 			 RelationGetRelationName(state->heaprel));
 
-		table_index_build_scan(state->heaprel, state->rel, indexinfo, true,
+		table_index_build_scan(state->heaprel, state->rel, indexinfo, true, false,
 							   bt_tuple_present_callback, (void *) state, scan);
 
 		ereport(DEBUG1,
diff --git a/contrib/bloom/blinsert.c b/contrib/bloom/blinsert.c
index 1b8df7e1e84..48f35d39990 100644
--- a/contrib/bloom/blinsert.c
+++ b/contrib/bloom/blinsert.c
@@ -142,7 +142,7 @@ blbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	initCachedPage(&buildstate);
 
 	/* Do the heap scan */
-	reltuples = table_index_build_scan(heap, index, indexInfo, true,
+	reltuples = table_index_build_scan(heap, index, indexInfo, true, false,
 									   bloomBuildCallback, (void *) &buildstate,
 									   NULL);
 
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index d078dfbd469..ee3bd562748 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -132,6 +132,7 @@ blhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = blcostestimate;
 	amroutine->amoptions = bloptions;
 	amroutine->amproperty = NULL;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = blvalidate;
 	amroutine->ambeginscan = blbeginscan;
 	amroutine->amrescan = blrescan;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index b56d3b3daa1..ff8290da9ff 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -127,6 +127,7 @@ typedef struct IndexAmRoutine
     amcostestimate_function amcostestimate;
     amoptions_function amoptions;
     amproperty_function amproperty;     /* can be NULL */
+    ambuildphasename_function ambuildphasename;   /* can be NULL */
     amvalidate_function amvalidate;
     ambeginscan_function ambeginscan;
     amrescan_function amrescan;
@@ -468,6 +469,18 @@ amproperty (Oid index_oid, int attno,
 
   <para>
 <programlisting>
+char *
+ambuildphasename (int64 phasenum);
+</programlisting>
+   Return the textual name of the given build phase number.
+   The phase numbers are those reported during an index build via the
+   <function>pgstat_progress_update_param</function> interface.
+   The phase names are then exposed in the
+   <structname>pg_stat_progress_create_index</structname> view.
+  </para>
+
+  <para>
+<programlisting>
 bool
 amvalidate (Oid opclassoid);
 </programlisting>
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index f1df14bdea8..3a007d6871b 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -336,6 +336,14 @@ postgres   27093  0.0  0.0  30096  2752 ?        Ss   11:34   0:00 postgres: ser
       </entry>
      </row>
 
+     <row>
+      <entry><structname>pg_stat_progress_create_index</structname><indexterm><primary>pg_stat_progress_create_index</primary></indexterm></entry>
+      <entry>One row for each backend running <command>CREATE INDEX</command>, showing
+      current progress.
+      See <xref linkend='create-index-progress-reporting'/>.
+     </entry>
+     </row>
+
      <row>
       <entry><structname>pg_stat_progress_vacuum</structname><indexterm><primary>pg_stat_progress_vacuum</primary></indexterm></entry>
       <entry>One row for each backend (including autovacuum worker processes) running
@@ -3403,10 +3411,224 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
   <para>
    <productname>PostgreSQL</productname> has the ability to report the progress of
    certain commands during command execution.  Currently, the only commands
-   which support progress reporting are <command>VACUUM</command> and
+   which support progress reporting are <command>CREATE INDEX</command>,
+   <command>VACUUM</command> and
    <command>CLUSTER</command>. This may be expanded in the future.
   </para>
 
+ <sect2 id="create-index-progress-reporting">
+  <title>CREATE INDEX Progress Reporting</title>
+
+  <para>
+   Whenever <command>CREATE INDEX</command> is running, the
+   <structname>pg_stat_progress_create_index</structname> view will contain
+   one row for each backend that is currently creating indexes.  The tables
+   below describe the information that will be reported and provide information
+   about how to interpret it.
+  </para>
+
+  <table id="pg-stat-progress-create-index-view" xreflabel="pg_stat_progress_create_index">
+   <title><structname>pg_stat_progress_create_index</structname> View</title>
+   <tgroup cols="3">
+    <thead>
+     <row>
+      <entry>Column</entry>
+      <entry>Type</entry>
+      <entry>Description</entry>
+     </row>
+    </thead>
+
+    <tbody>
+     <row>
+      <entry><structfield>pid</structfield></entry>
+      <entry><type>integer</type></entry>
+      <entry>Process ID of backend.</entry>
+     </row>
+     <row>
+      <entry><structfield>datid</structfield></entry>
+      <entry><type>oid</type></entry>
+      <entry>OID of the database to which this backend is connected.</entry>
+     </row>
+     <row>
+      <entry><structfield>datname</structfield></entry>
+      <entry><type>name</type></entry>
+      <entry>Name of the database to which this backend is connected.</entry>
+     </row>
+     <row>
+      <entry><structfield>relid</structfield></entry>
+      <entry><type>oid</type></entry>
+      <entry>OID of the table on which the index is being created.</entry>
+     </row>
+     <row>
+      <entry><structfield>phase</structfield></entry>
+      <entry><type>text</type></entry>
+      <entry>
+        Current processing phase of index creation.  See <xref linkend='create-index-phases'/>.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>lockers_total</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Total number of lockers to wait for, when applicable.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>lockers_done</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Number of lockers already waited for.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>current_locked_pid</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Process ID of the locker currently being waited for.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>blocks_total</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Total number of blocks to be processed in the current phase.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>blocks_done</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Number of blocks already processed in the current phase.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>tuples_total</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Total number of tuples to be processed in the current phase.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>tuples_done</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+        Number of tuples already processed in the current phase.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>partitions_total</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+       When creating an index on a partitioned table, this column is set to
+       the total number of partitions on which the index is to be created.
+      </entry>
+     </row>
+     <row>
+      <entry><structfield>partitions_done</structfield></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+       When creating an index on a partitioned table, this column is set to
+       the number of partitions on which the index has been completed.
+      </entry>
+     </row>
+    </tbody>
+   </tgroup>
+  </table>
+
+  <table id="create-index-phases">
+   <title>CREATE INDEX phases</title>
+   <tgroup cols="2">
+    <thead>
+     <row>
+      <entry>Phase</entry>
+      <entry>Description</entry>
+     </row>
+    </thead>
+    <tbody>
+     <row>
+      <entry><literal>initializing</literal></entry>
+      <entry>
+       <command>CREATE INDEX</command> is preparing to create the index.  This
+       phase is expected to be very brief.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>waiting for old snapshots</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions
+       that can potentially see the table to release their snapshots.
+       This phase is skipped when not in concurrent mode.
+       Columns <structname>lockers_total</structname>, <structname>lockers_done</structname>
+       and <structname>current_locker_pid</structname> contain the progress 
+       information for this phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>building index</literal></entry>
+      <entry>
+       The index is being built by the access method-specific code.  In this phase,
+       access methods that support progress reporting fill in their own progress data,
+       and the subphase is indicated in this column.  Typically,
+       <structname>blocks_total</structname> and <structname>blocks_done</structname>
+       will contain progress data, as well as potentially
+       <structname>tuples_total</structname> and <structname>tuples_done</structname>.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>waiting for writer snapshots</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions
+       that can potentially write into the table to release their snapshots.
+       This phase is skipped when not in concurrent mode.
+       Columns <structname>lockers_total</structname>, <structname>lockers_done</structname>
+       and <structname>current_locker_pid</structname> contain the progress 
+       information for this phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>validating index scan</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is scanning the index searching
+       for tuples that need to be validated.
+       This phase is skipped when not in concurrent mode.
+       Columns <structname>blocks_total</structname> (set to the total size of the index)
+       and <structname>blocks_done</structname> contain the progress information for this phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>sorting index scan results</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is sorting the output of the
+       previous phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>validating index heapscan</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is scanning the table
+       to validate the index tuples collected in the previous two phases.
+       This phase is skipped when not in concurrent mode.
+       Columns <structname>blocks_total</structname> (set to the total size of the table)
+       and <structname>blocks_done</structname> contain the progress information for this phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>waiting for reader snapshots</literal></entry>
+      <entry>
+       <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions
+       that can potentially see the table to release their snapshots.  This
+       phase is skipped when not in concurrent mode.
+       Columns <structname>lockers_total</structname>, <structname>lockers_done</structname>
+       and <structname>current_locker_pid</structname> contain the progress 
+       information for this phase.
+      </entry>
+     </row>
+    </tbody>
+   </tgroup>
+  </table>
+
+ </sect2>
+
  <sect2 id="vacuum-progress-reporting">
   <title>VACUUM Progress Reporting</title>
 
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 6e96d24ca22..54273f754f6 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -112,6 +112,7 @@ brinhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = brincostestimate;
 	amroutine->amoptions = brinoptions;
 	amroutine->amproperty = NULL;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = brinvalidate;
 	amroutine->ambeginscan = brinbeginscan;
 	amroutine->amrescan = brinrescan;
@@ -719,7 +720,7 @@ brinbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	 * Now scan the relation.  No syncscan allowed here because we want the
 	 * heap blocks in physical order.
 	 */
-	reltuples = table_index_build_scan(heap, index, indexInfo, false,
+	reltuples = table_index_build_scan(heap, index, indexInfo, false, false,
 									   brinbuildCallback, (void *) state, NULL);
 
 	/* process the final batch */
@@ -1236,7 +1237,7 @@ summarize_range(IndexInfo *indexInfo, BrinBuildState *state, Relation heapRel,
 	 * cases.
 	 */
 	state->bs_currRangeStart = heapBlk;
-	table_index_build_range_scan(heapRel, state->bs_irel, indexInfo, false, true,
+	table_index_build_range_scan(heapRel, state->bs_irel, indexInfo, false, true, false,
 								 heapBlk, scanNumBlks,
 								 brinbuildCallback, (void *) state, NULL);
 
diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c
index b02f69b0dcb..b4b0213f76f 100644
--- a/src/backend/access/gin/gininsert.c
+++ b/src/backend/access/gin/gininsert.c
@@ -395,7 +395,7 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	 * Do the heap scan.  We disallow sync scan here because dataPlaceToPage
 	 * prefers to receive tuples in TID order.
 	 */
-	reltuples = table_index_build_scan(heap, index, indexInfo, false,
+	reltuples = table_index_build_scan(heap, index, indexInfo, false, false,
 									   ginBuildCallback, (void *) &buildstate,
 									   NULL);
 
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index afc20232ace..d2360eeafb0 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -64,6 +64,7 @@ ginhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = gincostestimate;
 	amroutine->amoptions = ginoptions;
 	amroutine->amproperty = NULL;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = ginvalidate;
 	amroutine->ambeginscan = ginbeginscan;
 	amroutine->amrescan = ginrescan;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 2fddb23496d..f44c922b5d6 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -86,6 +86,7 @@ gisthandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = gistcostestimate;
 	amroutine->amoptions = gistoptions;
 	amroutine->amproperty = gistproperty;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = gistvalidate;
 	amroutine->ambeginscan = gistbeginscan;
 	amroutine->amrescan = gistrescan;
diff --git a/src/backend/access/gist/gistbuild.c b/src/backend/access/gist/gistbuild.c
index 3652fde5bb1..771bd2962a7 100644
--- a/src/backend/access/gist/gistbuild.c
+++ b/src/backend/access/gist/gistbuild.c
@@ -205,7 +205,7 @@ gistbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	/*
 	 * Do the heap scan.
 	 */
-	reltuples = table_index_build_scan(heap, index, indexInfo, true,
+	reltuples = table_index_build_scan(heap, index, indexInfo, true, false,
 									   gistBuildCallback,
 									   (void *) &buildstate, NULL);
 
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 5cc12a17130..6b38dd9c0e9 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -83,6 +83,7 @@ hashhandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = hashcostestimate;
 	amroutine->amoptions = hashoptions;
 	amroutine->amproperty = NULL;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = hashvalidate;
 	amroutine->ambeginscan = hashbeginscan;
 	amroutine->amrescan = hashrescan;
@@ -160,7 +161,7 @@ hashbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	buildstate.heapRel = heap;
 
 	/* do the heap scan */
-	reltuples = table_index_build_scan(heap, index, indexInfo, true,
+	reltuples = table_index_build_scan(heap, index, indexInfo, true, false,
 									   hashbuildCallback,
 									   (void *) &buildstate, NULL);
 
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index ce54b16f342..d22ef73555d 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -57,6 +57,8 @@ static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer,
 					   HeapTuple tuple,
 					   OffsetNumber tupoffset);
 
+static BlockNumber heapam_scan_get_blocks_done(HeapScanDesc hscan);
+
 static const TableAmRoutine heapam_methods;
 
 
@@ -1109,6 +1111,7 @@ heapam_index_build_range_scan(Relation heapRelation,
 							  IndexInfo *indexInfo,
 							  bool allow_sync,
 							  bool anyvisible,
+							  bool progress,
 							  BlockNumber start_blockno,
 							  BlockNumber numblocks,
 							  IndexBuildCallback callback,
@@ -1129,6 +1132,7 @@ heapam_index_build_range_scan(Relation heapRelation,
 	Snapshot	snapshot;
 	bool		need_unregister_snapshot = false;
 	TransactionId OldestXmin;
+	BlockNumber	previous_blkno = InvalidBlockNumber;
 	BlockNumber root_blkno = InvalidBlockNumber;
 	OffsetNumber root_offsets[MaxHeapTuplesPerPage];
 
@@ -1216,6 +1220,25 @@ heapam_index_build_range_scan(Relation heapRelation,
 
 	hscan = (HeapScanDesc) scan;
 
+	/* Publish number of blocks to scan */
+	if (progress)
+	{
+		BlockNumber		nblocks;
+
+		if (hscan->rs_base.rs_parallel != NULL)
+		{
+			ParallelBlockTableScanDesc pbscan;
+
+			pbscan = (ParallelBlockTableScanDesc) hscan->rs_base.rs_parallel;
+			nblocks = pbscan->phs_nblocks;
+		}
+		else
+			nblocks = hscan->rs_nblocks;
+
+		pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL,
+									 nblocks);
+	}
+
 	/*
 	 * Must call GetOldestXmin() with SnapshotAny.  Should never call
 	 * GetOldestXmin() with MVCC snapshot. (It's especially worth checking
@@ -1248,6 +1271,19 @@ heapam_index_build_range_scan(Relation heapRelation,
 
 		CHECK_FOR_INTERRUPTS();
 
+		/* Report scan progress, if asked to. */
+		if (progress)
+		{
+			BlockNumber     blocks_done = heapam_scan_get_blocks_done(hscan);
+
+			if (blocks_done != previous_blkno)
+			{
+				pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
+											 blocks_done);
+				previous_blkno = blocks_done;
+			}
+		}
+
 		/*
 		 * When dealing with a HOT-chain of updated tuples, we want to index
 		 * the values of the live tuple (if any), but index it under the TID
@@ -1589,6 +1625,25 @@ heapam_index_build_range_scan(Relation heapRelation,
 		}
 	}
 
+	/* Report scan progress one last time. */
+	if (progress)
+	{
+		BlockNumber		blks_done;
+
+		if (hscan->rs_base.rs_parallel != NULL)
+		{
+			ParallelBlockTableScanDesc pbscan;
+
+			pbscan = (ParallelBlockTableScanDesc) hscan->rs_base.rs_parallel;
+			blks_done = pbscan->phs_nblocks;
+		}
+		else
+			blks_done = hscan->rs_nblocks;
+
+		pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
+									 blks_done);
+	}
+
 	table_endscan(scan);
 
 	/* we can now forget our snapshot, if set and registered by us */
@@ -1625,6 +1680,7 @@ heapam_index_validate_scan(Relation heapRelation,
 	BlockNumber root_blkno = InvalidBlockNumber;
 	OffsetNumber root_offsets[MaxHeapTuplesPerPage];
 	bool		in_index[MaxHeapTuplesPerPage];
+	BlockNumber	previous_blkno = InvalidBlockNumber;
 
 	/* state variables for the merge */
 	ItemPointer indexcursor = NULL;
@@ -1665,6 +1721,9 @@ heapam_index_validate_scan(Relation heapRelation,
 								 false);	/* syncscan not OK */
 	hscan = (HeapScanDesc) scan;
 
+	pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL,
+								 hscan->rs_nblocks);
+
 	/*
 	 * Scan all tuples matching the snapshot.
 	 */
@@ -1678,6 +1737,14 @@ heapam_index_validate_scan(Relation heapRelation,
 
 		state->htups += 1;
 
+		if ((previous_blkno == InvalidBlockNumber) ||
+			(hscan->rs_cblock != previous_blkno))
+		{
+			pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
+										 hscan->rs_cblock);
+			previous_blkno = hscan->rs_cblock;
+		}
+
 		/*
 		 * As commented in table_index_build_scan, we should index heap-only
 		 * tuples under the TIDs of their root tuples; so when we advance onto
@@ -1838,6 +1905,46 @@ heapam_index_validate_scan(Relation heapRelation,
 	indexInfo->ii_PredicateState = NULL;
 }
 
+/*
+ * Return the number of blocks that have been read by this scan since
+ * starting.  This is meant for progress reporting rather than be fully
+ * accurate: in a parallel scan, workers can be concurrently reading blocks
+ * further ahead than what we report.
+ */
+static BlockNumber
+heapam_scan_get_blocks_done(HeapScanDesc hscan)
+{
+	ParallelBlockTableScanDesc bpscan = NULL;
+	BlockNumber		startblock;
+	BlockNumber		blocks_done;
+
+	if (hscan->rs_base.rs_parallel != NULL)
+	{
+		bpscan = (ParallelBlockTableScanDesc) hscan->rs_base.rs_parallel;
+		startblock = bpscan->phs_startblock;
+	}
+	else
+		startblock = hscan->rs_startblock;
+
+	/*
+	 * Might have wrapped around the end of the relation, if startblock was
+	 * not zero.
+	 */
+	if (hscan->rs_cblock > startblock)
+		blocks_done = hscan->rs_cblock - startblock;
+	else
+	{
+		BlockNumber     nblocks;
+
+		nblocks = bpscan != NULL ? bpscan->phs_nblocks : hscan->rs_nblocks;
+		blocks_done = nblocks - startblock +
+			hscan->rs_cblock;
+	}
+
+	return blocks_done;
+}
+
+
 
 /* ------------------------------------------------------------------------
  * Planner related callbacks for the heap AM
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index ac6f1eb3423..7370379c6a1 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -22,6 +22,7 @@
 #include "access/nbtxlog.h"
 #include "access/relscan.h"
 #include "access/xlog.h"
+#include "commands/progress.h"
 #include "commands/vacuum.h"
 #include "miscadmin.h"
 #include "nodes/execnodes.h"
@@ -133,6 +134,7 @@ bthandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = btcostestimate;
 	amroutine->amoptions = btoptions;
 	amroutine->amproperty = btproperty;
+	amroutine->ambuildphasename = btbuildphasename;
 	amroutine->amvalidate = btvalidate;
 	amroutine->ambeginscan = btbeginscan;
 	amroutine->amrescan = btrescan;
@@ -1021,6 +1023,10 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
 		if (needLock)
 			UnlockRelationForExtension(rel, ExclusiveLock);
 
+		if (info->report_progress)
+			pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL,
+										 num_pages);
+
 		/* Quit if we've scanned the whole relation */
 		if (blkno >= num_pages)
 			break;
@@ -1028,6 +1034,9 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
 		for (; blkno < num_pages; blkno++)
 		{
 			btvacuumpage(&vstate, blkno, blkno);
+			if (info->report_progress)
+				pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
+											 blkno);
 		}
 	}
 
diff --git a/src/backend/access/nbtree/nbtsort.c b/src/backend/access/nbtree/nbtsort.c
index 14d95457682..b333a7f82b2 100644
--- a/src/backend/access/nbtree/nbtsort.c
+++ b/src/backend/access/nbtree/nbtsort.c
@@ -66,6 +66,7 @@
 #include "access/xlog.h"
 #include "access/xloginsert.h"
 #include "catalog/index.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
 #include "pgstat.h"
 #include "storage/smgr.h"
@@ -298,7 +299,8 @@ static double _bt_parallel_heapscan(BTBuildState *buildstate,
 static void _bt_leader_participate_as_worker(BTBuildState *buildstate);
 static void _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2,
 						   BTShared *btshared, Sharedsort *sharedsort,
-						   Sharedsort *sharedsort2, int sortmem);
+						   Sharedsort *sharedsort2, int sortmem,
+						   bool progress);
 
 
 /*
@@ -394,6 +396,10 @@ _bt_spools_heapscan(Relation heap, Relation index, BTBuildState *buildstate,
 	/* Save as primary spool */
 	buildstate->spool = btspool;
 
+	/* Report heap scan phase started */
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+								 PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN);
+
 	/* Attempt to launch parallel worker scan when required */
 	if (indexInfo->ii_ParallelWorkers > 0)
 		_bt_begin_parallel(buildstate, indexInfo->ii_Concurrent,
@@ -480,13 +486,31 @@ _bt_spools_heapscan(Relation heap, Relation index, BTBuildState *buildstate,
 
 	/* Fill spool using either serial or parallel heap scan */
 	if (!buildstate->btleader)
-		reltuples = table_index_build_scan(heap, index, indexInfo, true,
+		reltuples = table_index_build_scan(heap, index, indexInfo, true, true,
 										   _bt_build_callback, (void *) buildstate,
 										   NULL);
 	else
 		reltuples = _bt_parallel_heapscan(buildstate,
 										  &indexInfo->ii_BrokenHotChain);
 
+	/*
+	 * Set the progress target for the next phase.  Reset the block number
+	 * values set by table_index_build_scan
+	 */
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_TUPLES_TOTAL,
+			PROGRESS_SCAN_BLOCKS_TOTAL,
+			PROGRESS_SCAN_BLOCKS_DONE
+		};
+		const int64 val[] = {
+			buildstate->indtuples,
+			0, 0
+		};
+
+		pgstat_progress_update_multi_param(3, index, val);
+	}
+
 	/* okay, all heap tuples are spooled */
 	if (buildstate->spool2 && !buildstate->havedead)
 	{
@@ -535,9 +559,15 @@ _bt_leafbuild(BTSpool *btspool, BTSpool *btspool2)
 	}
 #endif							/* BTREE_BUILD_STATS */
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+								 PROGRESS_BTREE_PHASE_PERFORMSORT_1);
 	tuplesort_performsort(btspool->sortstate);
 	if (btspool2)
+	{
+		pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+									 PROGRESS_BTREE_PHASE_PERFORMSORT_2);
 		tuplesort_performsort(btspool2->sortstate);
+	}
 
 	wstate.heap = btspool->heap;
 	wstate.index = btspool->index;
@@ -554,6 +584,8 @@ _bt_leafbuild(BTSpool *btspool, BTSpool *btspool2)
 	wstate.btws_pages_written = 0;
 	wstate.btws_zeropage = NULL;	/* until needed */
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE,
+								 PROGRESS_BTREE_PHASE_LEAF_LOAD);
 	_bt_load(&wstate, btspool, btspool2);
 }
 
@@ -1098,6 +1130,7 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
 	int			i,
 				keysz = IndexRelationGetNumberOfKeyAttributes(wstate->index);
 	SortSupport sortKeys;
+	long		tuples_done = 0L;
 
 	if (merge)
 	{
@@ -1202,6 +1235,10 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
 				_bt_buildadd(wstate, state, itup2);
 				itup2 = tuplesort_getindextuple(btspool2->sortstate, true);
 			}
+
+			/* Report progress */
+			pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE,
+										 ++tuples_done);
 		}
 		pfree(sortKeys);
 	}
@@ -1216,6 +1253,10 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)
 				state = _bt_pagestate(wstate, 0);
 
 			_bt_buildadd(wstate, state, itup);
+
+			/* Report progress */
+			pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE,
+										 ++tuples_done);
 		}
 	}
 
@@ -1528,7 +1569,7 @@ _bt_leader_participate_as_worker(BTBuildState *buildstate)
 	/* Perform work common to all participants */
 	_bt_parallel_scan_and_sort(leaderworker, leaderworker2, btleader->btshared,
 							   btleader->sharedsort, btleader->sharedsort2,
-							   sortmem);
+							   sortmem, true);
 
 #ifdef BTREE_BUILD_STATS
 	if (log_btree_build_stats)
@@ -1619,7 +1660,7 @@ _bt_parallel_build_main(dsm_segment *seg, shm_toc *toc)
 	/* Perform sorting of spool, and possibly a spool2 */
 	sortmem = maintenance_work_mem / btshared->scantuplesortstates;
 	_bt_parallel_scan_and_sort(btspool, btspool2, btshared, sharedsort,
-							   sharedsort2, sortmem);
+							   sharedsort2, sortmem, false);
 
 #ifdef BTREE_BUILD_STATS
 	if (log_btree_build_stats)
@@ -1648,7 +1689,7 @@ _bt_parallel_build_main(dsm_segment *seg, shm_toc *toc)
 static void
 _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2,
 						   BTShared *btshared, Sharedsort *sharedsort,
-						   Sharedsort *sharedsort2, int sortmem)
+						   Sharedsort *sharedsort2, int sortmem, bool progress)
 {
 	SortCoordinate coordinate;
 	BTBuildState buildstate;
@@ -1705,9 +1746,10 @@ _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2,
 	/* Join parallel scan */
 	indexInfo = BuildIndexInfo(btspool->index);
 	indexInfo->ii_Concurrent = btshared->isconcurrent;
-	scan = table_beginscan_parallel(btspool->heap, ParallelTableScanFromBTShared(btshared));
+	scan = table_beginscan_parallel(btspool->heap,
+									ParallelTableScanFromBTShared(btshared));
 	reltuples = table_index_build_scan(btspool->heap, btspool->index, indexInfo,
-									   true, _bt_build_callback,
+									   true, progress, _bt_build_callback,
 									   (void *) &buildstate, scan);
 
 	/*
diff --git a/src/backend/access/nbtree/nbtutils.c b/src/backend/access/nbtree/nbtutils.c
index 140ac920265..a0cc9dba6d9 100644
--- a/src/backend/access/nbtree/nbtutils.c
+++ b/src/backend/access/nbtree/nbtutils.c
@@ -20,6 +20,7 @@
 #include "access/nbtree.h"
 #include "access/reloptions.h"
 #include "access/relscan.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
 #include "utils/array.h"
 #include "utils/datum.h"
@@ -2051,6 +2052,29 @@ btproperty(Oid index_oid, int attno,
 	}
 }
 
+/*
+ *	btbuildphasename() -- Return name of index build phase.
+ */
+char *
+btbuildphasename(int64 phasenum)
+{
+	switch (phasenum)
+	{
+		case PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE:
+			return "initializing";
+		case PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN:
+			return "scanning table";
+		case PROGRESS_BTREE_PHASE_PERFORMSORT_1:
+			return "sorting live tuples";
+		case PROGRESS_BTREE_PHASE_PERFORMSORT_2:
+			return "sorting dead tuples";
+		case PROGRESS_BTREE_PHASE_LEAF_LOAD:
+			return "loading tuples in tree";
+		default:
+			return NULL;
+	}
+}
+
 /*
  *	_bt_truncate() -- create tuple without unneeded suffix attributes.
  *
diff --git a/src/backend/access/spgist/spginsert.c b/src/backend/access/spgist/spginsert.c
index 390ad9ac51f..282d6998cf0 100644
--- a/src/backend/access/spgist/spginsert.c
+++ b/src/backend/access/spgist/spginsert.c
@@ -143,7 +143,7 @@ spgbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 											  "SP-GiST build temporary context",
 											  ALLOCSET_DEFAULT_SIZES);
 
-	reltuples = table_index_build_scan(heap, index, indexInfo, true,
+	reltuples = table_index_build_scan(heap, index, indexInfo, true, false,
 									   spgistBuildCallback, (void *) &buildstate,
 									   NULL);
 
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 8e63c1fad25..45472db147b 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -67,6 +67,7 @@ spghandler(PG_FUNCTION_ARGS)
 	amroutine->amcostestimate = spgcostestimate;
 	amroutine->amoptions = spgoptions;
 	amroutine->amproperty = spgproperty;
+	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = spgvalidate;
 	amroutine->ambeginscan = spgbeginscan;
 	amroutine->amrescan = spgrescan;
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 0d9d405c548..6c6818e891c 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -51,9 +51,9 @@
 #include "catalog/pg_trigger.h"
 #include "catalog/pg_type.h"
 #include "catalog/storage.h"
-#include "commands/tablecmds.h"
 #include "commands/event_trigger.h"
 #include "commands/progress.h"
+#include "commands/tablecmds.h"
 #include "commands/trigger.h"
 #include "executor/executor.h"
 #include "miscadmin.h"
@@ -2047,7 +2047,7 @@ index_drop(Oid indexId, bool concurrent, bool concurrent_lock_mode)
 		 * to acquire an exclusive lock on our table.  The lock code will
 		 * detect deadlock and error out properly.
 		 */
-		WaitForLockers(heaplocktag, AccessExclusiveLock);
+		WaitForLockers(heaplocktag, AccessExclusiveLock, true);
 
 		/* Finish invalidation of index and mark it as dead */
 		index_concurrently_set_dead(heapId, indexId);
@@ -2063,7 +2063,7 @@ index_drop(Oid indexId, bool concurrent, bool concurrent_lock_mode)
 		 * Wait till every transaction that saw the old index state has
 		 * finished.
 		 */
-		WaitForLockers(heaplocktag, AccessExclusiveLock);
+		WaitForLockers(heaplocktag, AccessExclusiveLock, true);
 
 		/*
 		 * Re-open relations to allow us to complete our actions.
@@ -2712,6 +2712,25 @@ index_build(Relation heapRelation,
 						   save_sec_context | SECURITY_RESTRICTED_OPERATION);
 	save_nestlevel = NewGUCNestLevel();
 
+	/* Set up initial progress report status */
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_PHASE,
+			PROGRESS_CREATEIDX_SUBPHASE,
+			PROGRESS_CREATEIDX_TUPLES_DONE,
+			PROGRESS_CREATEIDX_TUPLES_TOTAL,
+			PROGRESS_SCAN_BLOCKS_DONE,
+			PROGRESS_SCAN_BLOCKS_TOTAL
+		};
+		const int64	val[] = {
+			PROGRESS_CREATEIDX_PHASE_BUILD,
+			PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE,
+			0, 0, 0, 0
+		};
+
+		pgstat_progress_update_multi_param(6, index, val);
+	}
+
 	/*
 	 * Call the access method's build procedure
 	 */
@@ -3000,6 +3019,21 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot)
 	int			save_sec_context;
 	int			save_nestlevel;
 
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_PHASE,
+			PROGRESS_CREATEIDX_TUPLES_DONE,
+			PROGRESS_CREATEIDX_TUPLES_TOTAL,
+			PROGRESS_SCAN_BLOCKS_DONE,
+			PROGRESS_SCAN_BLOCKS_TOTAL
+		};
+		const int64	val[] = {
+			PROGRESS_CREATEIDX_PHASE_VALIDATE_IDXSCAN,
+			0, 0, 0, 0
+		};
+		pgstat_progress_update_multi_param(5, index, val);
+	}
+
 	/* Open and lock the parent heap relation */
 	heapRelation = table_open(heapId, ShareUpdateExclusiveLock);
 	/* And the target index relation */
@@ -3030,6 +3064,7 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot)
 	 */
 	ivinfo.index = indexRelation;
 	ivinfo.analyze_only = false;
+	ivinfo.report_progress = true;	/* XXX only for btree? */
 	ivinfo.estimated_count = true;
 	ivinfo.message_level = DEBUG2;
 	ivinfo.num_heap_tuples = heapRelation->rd_rel->reltuples;
@@ -3047,15 +3082,39 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot)
 											NULL, false);
 	state.htups = state.itups = state.tups_inserted = 0;
 
+	/* ambulkdelete updates progress metrics */
 	(void) index_bulk_delete(&ivinfo, NULL,
 							 validate_index_callback, (void *) &state);
 
 	/* Execute the sort */
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_PHASE,
+			PROGRESS_SCAN_BLOCKS_DONE,
+			PROGRESS_SCAN_BLOCKS_TOTAL
+		};
+		const int64	val[] = {
+			PROGRESS_CREATEIDX_PHASE_SORT_IDXSCAN,
+			0, 0
+		};
+
+		pgstat_progress_update_multi_param(3, index, val);
+	}
 	tuplesort_performsort(state.tuplesort);
 
 	/*
-	 * Now scan the heap and "merge" it with the index
+	 * Now scan the heap and "merge" it with the index.
 	 */
+	{
+		const int	index[] = {
+			PROGRESS_CREATEIDX_PHASE,
+		};
+		const int64	val[] = {
+			PROGRESS_CREATEIDX_PHASE_HEAPSCAN_VALIDATE
+		};
+
+		pgstat_progress_update_multi_param(1, index, val);
+	}
 	table_index_validate_scan(heapRelation,
 							  indexRelation,
 							  indexInfo,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index b89df70653e..acf277473c1 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -934,6 +934,33 @@ CREATE VIEW pg_stat_progress_cluster AS
     FROM pg_stat_get_progress_info('CLUSTER') AS S
         LEFT JOIN pg_database D ON S.datid = D.oid;
 
+CREATE VIEW pg_stat_progress_create_index AS
+	SELECT
+		S.pid AS pid, S.datid AS datid, D.datname AS datname,
+		S.relid AS relid,
+		CASE S.param10 WHEN 0 THEN 'initializing'
+					  WHEN 1 THEN 'waiting for old snapshots'
+					  WHEN 2 THEN 'building index' ||
+						COALESCE((': ' || pg_indexam_progress_phasename(S.param9::oid, S.param11)),
+							'')
+					  WHEN 3 THEN 'waiting for writer snapshots'
+					  WHEN 4 THEN 'index validation: scan index'
+					  WHEN 5 THEN 'index validation: sort index scan results'
+					  WHEN 6 THEN 'index validation: scan heap'
+					  WHEN 7 THEN 'waiting for reader snapshots'
+					  END as phase,
+		S.param4 AS lockers_total,
+		S.param5 AS lockers_done,
+		S.param6 AS current_locker_pid,
+		S.param16 AS blocks_total,
+		S.param17 AS blocks_done,
+		S.param12 AS tuples_total,
+		S.param13 AS tuples_done,
+		S.param14 AS partitions_total,
+		S.param15 AS partitions_done
+	FROM pg_stat_get_progress_info('CREATE INDEX') AS S
+		LEFT JOIN pg_database D ON S.datid = D.oid;
+
 CREATE VIEW pg_user_mappings AS
     SELECT
         U.oid       AS umid,
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 53971fc7258..348d5432977 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -36,6 +36,7 @@
 #include "commands/dbcommands.h"
 #include "commands/defrem.h"
 #include "commands/event_trigger.h"
+#include "commands/progress.h"
 #include "commands/tablecmds.h"
 #include "commands/tablespace.h"
 #include "mb/pg_wchar.h"
@@ -47,10 +48,12 @@
 #include "parser/parse_func.h"
 #include "parser/parse_oper.h"
 #include "partitioning/partdesc.h"
+#include "pgstat.h"
 #include "rewrite/rewriteManip.h"
 #include "storage/lmgr.h"
 #include "storage/proc.h"
 #include "storage/procarray.h"
+#include "storage/sinvaladt.h"
 #include "utils/acl.h"
 #include "utils/builtins.h"
 #include "utils/fmgroids.h"
@@ -334,7 +337,7 @@ CheckIndexCompatible(Oid oldId,
  * doesn't show up in the output, we know we can forget about it.
  */
 static void
-WaitForOlderSnapshots(TransactionId limitXmin)
+WaitForOlderSnapshots(TransactionId limitXmin, bool progress)
 {
 	int			n_old_snapshots;
 	int			i;
@@ -343,6 +346,8 @@ WaitForOlderSnapshots(TransactionId limitXmin)
 	old_snapshots = GetCurrentVirtualXIDs(limitXmin, true, false,
 										  PROC_IS_AUTOVACUUM | PROC_IN_VACUUM,
 										  &n_old_snapshots);
+	if (progress)
+		pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, n_old_snapshots);
 
 	for (i = 0; i < n_old_snapshots; i++)
 	{
@@ -378,7 +383,19 @@ WaitForOlderSnapshots(TransactionId limitXmin)
 		}
 
 		if (VirtualTransactionIdIsValid(old_snapshots[i]))
+		{
+			if (progress)
+			{
+				PGPROC *holder = BackendIdGetProc(old_snapshots[i].backendId);
+
+				pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID,
+											 holder->pid);
+			}
 			VirtualXactLock(old_snapshots[i], true);
+		}
+
+		if (progress)
+			pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, i + 1);
 	}
 }
 
@@ -452,6 +469,15 @@ DefineIndex(Oid relationId,
 	Snapshot	snapshot;
 	int			i;
 
+
+	/*
+	 * Start progress report.  If we're building a partition, this was already
+	 * done.
+	 */
+	if (!OidIsValid(parentIndexId))
+		pgstat_progress_start_command(PROGRESS_COMMAND_CREATE_INDEX,
+									  relationId);
+
 	/*
 	 * count key attributes in index
 	 */
@@ -668,6 +694,9 @@ DefineIndex(Oid relationId,
 	accessMethodId = accessMethodForm->oid;
 	amRoutine = GetIndexAmRoutine(accessMethodForm->amhandler);
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_ACCESS_METHOD_OID,
+								 accessMethodId);
+
 	if (stmt->unique && !amRoutine->amcanunique)
 		ereport(ERROR,
 				(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -948,6 +977,11 @@ DefineIndex(Oid relationId,
 	if (!OidIsValid(indexRelationId))
 	{
 		table_close(rel, NoLock);
+
+		/* If this is the top-level index, we're done */
+		if (!OidIsValid(parentIndexId))
+			pgstat_progress_end_command();
+
 		return address;
 	}
 
@@ -973,6 +1007,9 @@ DefineIndex(Oid relationId,
 			TupleDesc	parentDesc;
 			Oid		   *opfamOids;
 
+			pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL,
+										 nparts);
+
 			memcpy(part_oids, partdesc->oids, sizeof(Oid) * nparts);
 
 			parentDesc = CreateTupleDescCopy(RelationGetDescr(rel));
@@ -1122,6 +1159,8 @@ DefineIndex(Oid relationId,
 								skip_build, quiet);
 				}
 
+				pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_DONE,
+											 i + 1);
 				pfree(attmap);
 			}
 
@@ -1156,6 +1195,8 @@ DefineIndex(Oid relationId,
 		 * Indexes on partitioned tables are not themselves built, so we're
 		 * done here.
 		 */
+		if (!OidIsValid(parentIndexId))
+			pgstat_progress_end_command();
 		return address;
 	}
 
@@ -1163,6 +1204,11 @@ DefineIndex(Oid relationId,
 	{
 		/* Close the heap and we're done, in the non-concurrent case */
 		table_close(rel, NoLock);
+
+		/* If this is the top-level index, we're done. */
+		if (!OidIsValid(parentIndexId))
+			pgstat_progress_end_command();
+
 		return address;
 	}
 
@@ -1214,7 +1260,9 @@ DefineIndex(Oid relationId,
 	 * exclusive lock on our table.  The lock code will detect deadlock and
 	 * error out properly.
 	 */
-	WaitForLockers(heaplocktag, ShareLock);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_1);
+	WaitForLockers(heaplocktag, ShareLock, true);
 
 	/*
 	 * At this moment we are sure that there are no transactions with the
@@ -1255,7 +1303,9 @@ DefineIndex(Oid relationId,
 	 * We once again wait until no transaction can have the table open with
 	 * the index marked as read-only for updates.
 	 */
-	WaitForLockers(heaplocktag, ShareLock);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_2);
+	WaitForLockers(heaplocktag, ShareLock, true);
 
 	/*
 	 * Now take the "reference snapshot" that will be used by validate_index()
@@ -1312,7 +1362,9 @@ DefineIndex(Oid relationId,
 	 * before the reference snap was taken, we have to wait out any
 	 * transactions that might have older snapshots.
 	 */
-	WaitForOlderSnapshots(limitXmin);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_3);
+	WaitForOlderSnapshots(limitXmin, true);
 
 	/*
 	 * Index can now be marked valid -- update its pg_index entry
@@ -1334,6 +1386,8 @@ DefineIndex(Oid relationId,
 	 */
 	UnlockRelationIdForSession(&heaprelid, ShareUpdateExclusiveLock);
 
+	pgstat_progress_end_command();
+
 	return address;
 }
 
@@ -2913,7 +2967,7 @@ ReindexRelationConcurrently(Oid relationOid, int options)
 	 * DefineIndex() for more details.
 	 */
 
-	WaitForLockersMultiple(lockTags, ShareLock);
+	WaitForLockersMultiple(lockTags, ShareLock, false);
 	CommitTransactionCommand();
 
 	forboth(lc, indexIds, lc2, newIndexIds)
@@ -2955,7 +3009,7 @@ ReindexRelationConcurrently(Oid relationOid, int options)
 	 * for more details.
 	 */
 
-	WaitForLockersMultiple(lockTags, ShareLock);
+	WaitForLockersMultiple(lockTags, ShareLock, false);
 	CommitTransactionCommand();
 
 	foreach(lc, newIndexIds)
@@ -3003,7 +3057,7 @@ ReindexRelationConcurrently(Oid relationOid, int options)
 		 * before the reference snap was taken, we have to wait out any
 		 * transactions that might have older snapshots.
 		 */
-		WaitForOlderSnapshots(limitXmin);
+		WaitForOlderSnapshots(limitXmin, false);
 
 		CommitTransactionCommand();
 	}
@@ -3074,7 +3128,7 @@ ReindexRelationConcurrently(Oid relationOid, int options)
 	 * index_drop() for more details.
 	 */
 
-	WaitForLockersMultiple(lockTags, AccessExclusiveLock);
+	WaitForLockersMultiple(lockTags, AccessExclusiveLock, false);
 
 	foreach(lc, indexIds)
 	{
@@ -3096,7 +3150,7 @@ ReindexRelationConcurrently(Oid relationOid, int options)
 	 * Drop the old indexes.
 	 */
 
-	WaitForLockersMultiple(lockTags, AccessExclusiveLock);
+	WaitForLockersMultiple(lockTags, AccessExclusiveLock, false);
 
 	PushActiveSnapshot(GetTransactionSnapshot());
 
diff --git a/src/backend/storage/ipc/standby.c b/src/backend/storage/ipc/standby.c
index cd56dca3aef..215f1463bb1 100644
--- a/src/backend/storage/ipc/standby.c
+++ b/src/backend/storage/ipc/standby.c
@@ -401,7 +401,7 @@ ResolveRecoveryConflictWithLock(LOCKTAG locktag)
 		 */
 		VirtualTransactionId *backends;
 
-		backends = GetLockConflicts(&locktag, AccessExclusiveLock);
+		backends = GetLockConflicts(&locktag, AccessExclusiveLock, NULL);
 		ResolveRecoveryConflictWithVirtualXIDs(backends,
 											   PROCSIG_RECOVERY_CONFLICT_LOCK);
 	}
diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c
index e688ba81170..0b04b093782 100644
--- a/src/backend/storage/lmgr/lmgr.c
+++ b/src/backend/storage/lmgr/lmgr.c
@@ -19,9 +19,12 @@
 #include "access/transam.h"
 #include "access/xact.h"
 #include "catalog/catalog.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
+#include "pgstat.h"
 #include "storage/lmgr.h"
 #include "storage/procarray.h"
+#include "storage/sinvaladt.h"
 #include "utils/inval.h"
 
 
@@ -857,10 +860,12 @@ XactLockTableWaitErrorCb(void *arg)
  * after we obtained our initial list of lockers, we will not wait for them.
  */
 void
-WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
+WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
 {
 	List	   *holders = NIL;
 	ListCell   *lc;
+	int			total = 0;
+	int			done = 0;
 
 	/* Done if no locks to wait for */
 	if (list_length(locktags) == 0)
@@ -870,10 +875,17 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
 	foreach(lc, locktags)
 	{
 		LOCKTAG    *locktag = lfirst(lc);
+		int			count;
 
-		holders = lappend(holders, GetLockConflicts(locktag, lockmode));
+		holders = lappend(holders,
+						  GetLockConflicts(locktag, lockmode,
+										   progress ? &count : NULL));
+		total += count;
 	}
 
+	if (progress)
+		pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, total);
+
 	/*
 	 * Note: GetLockConflicts() never reports our own xid, hence we need not
 	 * check for that.  Also, prepared xacts are not reported, which is fine
@@ -887,10 +899,36 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
 
 		while (VirtualTransactionIdIsValid(*lockholders))
 		{
+			/*
+			 * If requested, publish who we're going to wait for.  This is not
+			 * 100% accurate if they're already gone, but we don't care.
+			 */
+			if (progress)
+			{
+				PGPROC *holder = BackendIdGetProc(lockholders->backendId);
+
+				pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID,
+											 holder->pid);
+			}
 			VirtualXactLock(*lockholders, true);
 			lockholders++;
+
+			if (progress)
+				pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, ++done);
 		}
 	}
+	if (progress)
+	{
+		const int	index[] = {
+			PROGRESS_WAITFOR_TOTAL,
+			PROGRESS_WAITFOR_DONE,
+			PROGRESS_WAITFOR_CURRENT_PID
+		};
+		const int64	values[] = {
+			0, 0, 0
+		};
+		pgstat_progress_update_multi_param(3, index, values);
+	}
 
 	list_free_deep(holders);
 }
@@ -901,12 +939,12 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode)
  * Same as WaitForLockersMultiple, for a single lock tag.
  */
 void
-WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode)
+WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress)
 {
 	List	   *l;
 
 	l = list_make1(&heaplocktag);
-	WaitForLockersMultiple(l, lockmode);
+	WaitForLockersMultiple(l, lockmode, progress);
 	list_free(l);
 }
 
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 78fdbd6ff88..c8958766f1e 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2807,6 +2807,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock)
  *		xacts merely awaiting such a lock are NOT reported.
  *
  * The result array is palloc'd and is terminated with an invalid VXID.
+ * *countp, if not null, is updated to the number of items set.
  *
  * Of course, the result could be out of date by the time it's returned,
  * so use of this function has to be thought about carefully.
@@ -2817,7 +2818,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock)
  * uses of the result.
  */
 VirtualTransactionId *
-GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode)
+GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
 {
 	static VirtualTransactionId *vxids;
 	LOCKMETHODID lockmethodid = locktag->locktag_lockmethodid;
@@ -2964,6 +2965,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode)
 		LWLockRelease(partitionLock);
 		vxids[count].backendId = InvalidBackendId;
 		vxids[count].localTransactionId = InvalidLocalTransactionId;
+		if (countp)
+			*countp = count;
 		return vxids;
 	}
 
@@ -3019,6 +3022,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode)
 
 	vxids[count].backendId = InvalidBackendId;
 	vxids[count].localTransactionId = InvalidLocalTransactionId;
+	if (countp)
+		*countp = count;
 	return vxids;
 }
 
diff --git a/src/backend/utils/adt/amutils.c b/src/backend/utils/adt/amutils.c
index 060ffe501ec..e81d6cc0562 100644
--- a/src/backend/utils/adt/amutils.c
+++ b/src/backend/utils/adt/amutils.c
@@ -445,3 +445,26 @@ pg_index_column_has_property(PG_FUNCTION_ARGS)
 
 	return indexam_property(fcinfo, propname, InvalidOid, relid, attno);
 }
+
+/*
+ * Return the name of the given phase, as used for progress reporting by the
+ * given AM.
+ */
+Datum
+pg_indexam_progress_phasename(PG_FUNCTION_ARGS)
+{
+	Oid			amoid = PG_GETARG_OID(0);
+	int32		phasenum = PG_GETARG_INT32(1);
+	IndexAmRoutine *routine;
+	char	   *name;
+
+	routine = GetIndexAmRoutineByAmId(amoid, true);
+	if (routine == NULL || !routine->ambuildphasename)
+		PG_RETURN_NULL();
+
+	name = routine->ambuildphasename(phasenum);
+	if (!name)
+		PG_RETURN_NULL();
+
+	PG_RETURN_TEXT_P(CStringGetTextDatum(name));
+}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 90a817a25c5..7c2afe64272 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -470,6 +470,8 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
 		cmdtype = PROGRESS_COMMAND_VACUUM;
 	else if (pg_strcasecmp(cmd, "CLUSTER") == 0)
 		cmdtype = PROGRESS_COMMAND_CLUSTER;
+	else if (pg_strcasecmp(cmd, "CREATE INDEX") == 0)
+		cmdtype = PROGRESS_COMMAND_CREATE_INDEX;
 	else
 		ereport(ERROR,
 				(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 653ddc976ba..09a7404267c 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -108,6 +108,9 @@ typedef bool (*amproperty_function) (Oid index_oid, int attno,
 									 IndexAMProperty prop, const char *propname,
 									 bool *res, bool *isnull);
 
+/* name of phase as used in progress reporting */
+typedef char *(*ambuildphasename_function) (int64 phasenum);
+
 /* validate definition of an opclass for this AM */
 typedef bool (*amvalidate_function) (Oid opclassoid);
 
@@ -213,6 +216,7 @@ typedef struct IndexAmRoutine
 	amcostestimate_function amcostestimate;
 	amoptions_function amoptions;
 	amproperty_function amproperty; /* can be NULL */
+	ambuildphasename_function ambuildphasename;	/* can be NULL */
 	amvalidate_function amvalidate;
 	ambeginscan_function ambeginscan;
 	amrescan_function amrescan;
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 70c7351a08c..9717183ef23 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -45,6 +45,7 @@ typedef struct IndexVacuumInfo
 {
 	Relation	index;			/* the index being vacuumed */
 	bool		analyze_only;	/* ANALYZE (without any actual vacuum) */
+	bool		report_progress;	/* emit progress.h status reports */
 	bool		estimated_count;	/* num_heap_tuples is an estimate */
 	int			message_level;	/* ereport level for progress messages */
 	double		num_heap_tuples;	/* tuples remaining in heap */
diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h
index a1ffa983365..8a727872dae 100644
--- a/src/include/access/nbtree.h
+++ b/src/include/access/nbtree.h
@@ -671,6 +671,16 @@ typedef BTScanOpaqueData *BTScanOpaque;
 #define SK_BT_DESC			(INDOPTION_DESC << SK_BT_INDOPTION_SHIFT)
 #define SK_BT_NULLS_FIRST	(INDOPTION_NULLS_FIRST << SK_BT_INDOPTION_SHIFT)
 
+/*
+ * Constant definition for progress reporting.  Phase numbers must match
+ * btbuildphasename.
+ */
+/* PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE is 1 (see progress.h) */
+#define PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN		2
+#define PROGRESS_BTREE_PHASE_PERFORMSORT_1				3
+#define PROGRESS_BTREE_PHASE_PERFORMSORT_2				4
+#define PROGRESS_BTREE_PHASE_LEAF_LOAD					5
+
 /*
  * external entry points for btree, in nbtree.c
  */
@@ -784,6 +794,7 @@ extern bytea *btoptions(Datum reloptions, bool validate);
 extern bool btproperty(Oid index_oid, int attno,
 		   IndexAMProperty prop, const char *propname,
 		   bool *res, bool *isnull);
+extern char *btbuildphasename(int64 phasenum);
 extern IndexTuple _bt_truncate(Relation rel, IndexTuple lastleft,
 			 IndexTuple firstright, BTScanInsert itup_key);
 extern int _bt_keep_natts_fast(Relation rel, IndexTuple lastleft,
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index 9bc1d24b4a6..3ff2918a42b 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -492,6 +492,7 @@ typedef struct TableAmRoutine
 										   struct IndexInfo *index_nfo,
 										   bool allow_sync,
 										   bool anyvisible,
+										   bool progress,
 										   BlockNumber start_blockno,
 										   BlockNumber end_blockno,
 										   IndexBuildCallback callback,
@@ -1339,6 +1340,11 @@ table_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
  * so here because the AM might reject some of the tuples for its own reasons,
  * such as being unable to store NULLs.
  *
+ * If 'progress', we update the PROGRESS_SCAN_BLOCKS_DONE counter as we go
+ * along.  Also, if that flag is true and a scan descriptor is not passed (ie.
+ * when not doing a parallel scan), the PROGRESS_SCAN_BLOCKS_TOTAL counter is
+ * updated at the beginning.  For parallel scans, caller is expected to have
+ * set the total number of blocks prior to calling this function.
  *
  * A side effect is to set indexInfo->ii_BrokenHotChain to true if we detect
  * any potentially broken HOT chains.  Currently, we set this if there are any
@@ -1352,6 +1358,7 @@ table_index_build_scan(Relation heap_rel,
 					   Relation index_rel,
 					   struct IndexInfo *index_nfo,
 					   bool allow_sync,
+					   bool progress,
 					   IndexBuildCallback callback,
 					   void *callback_state,
 					   TableScanDesc scan)
@@ -1361,6 +1368,7 @@ table_index_build_scan(Relation heap_rel,
 														index_nfo,
 														allow_sync,
 														false,
+														progress,
 														0,
 														InvalidBlockNumber,
 														callback,
@@ -1384,6 +1392,7 @@ table_index_build_range_scan(Relation heap_rel,
 							 struct IndexInfo *index_nfo,
 							 bool allow_sync,
 							 bool anyvisible,
+							 bool progress,
 							 BlockNumber start_blockno,
 							 BlockNumber numblocks,
 							 IndexBuildCallback callback,
@@ -1395,6 +1404,7 @@ table_index_build_range_scan(Relation heap_rel,
 														index_nfo,
 														allow_sync,
 														anyvisible,
+														progress,
 														start_blockno,
 														numblocks,
 														callback,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index eac909109c5..a7050edca09 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -924,6 +924,10 @@
   proname => 'pg_index_column_has_property', provolatile => 's',
   prorettype => 'bool', proargtypes => 'regclass int4 text',
   prosrc => 'pg_index_column_has_property' },
+{ oid => '676', descr => 'return name of given index build phase',
+  proname => 'pg_indexam_progress_phasename', provolatile => 'i',
+  prorettype => 'text', proargtypes => 'oid int8',
+  prosrc => 'pg_indexam_progress_phasename' },
 
 { oid => '339',
   proname => 'poly_same', prorettype => 'bool',
@@ -5122,9 +5126,9 @@
   proname => 'pg_stat_get_progress_info', prorows => '100', proretset => 't',
   provolatile => 's', proparallel => 'r', prorettype => 'record',
   proargtypes => 'text',
-  proallargtypes => '{text,int4,oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}',
-  proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o}',
-  proargnames => '{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10}',
+  proallargtypes => '{text,int4,oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}',
+  proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+  proargnames => '{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10,param11,param12,param13,param14,param15,param16,param17,param18,param19,param20}',
   prosrc => 'pg_stat_get_progress_info' },
 { oid => '3099',
   descr => 'statistics: information about currently active replication',
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index 04542d9e923..079303cff46 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -44,7 +44,7 @@
 #define PROGRESS_CLUSTER_HEAP_BLKS_SCANNED		6
 #define PROGRESS_CLUSTER_INDEX_REBUILD_COUNT	7
 
-/* Phases of cluster (as dvertised via PROGRESS_CLUSTER_PHASE) */
+/* Phases of cluster (as advertised via PROGRESS_CLUSTER_PHASE) */
 #define PROGRESS_CLUSTER_PHASE_SEQ_SCAN_HEAP	1
 #define PROGRESS_CLUSTER_PHASE_INDEX_SCAN_HEAP	2
 #define PROGRESS_CLUSTER_PHASE_SORT_TUPLES		3
@@ -57,4 +57,39 @@
 #define PROGRESS_CLUSTER_COMMAND_CLUSTER		1
 #define PROGRESS_CLUSTER_COMMAND_VACUUM_FULL	2
 
+/* Progress parameters for CREATE INDEX */
+/* 3, 4 and 5 reserved for "waitfor" metrics */
+#define PROGRESS_CREATEIDX_ACCESS_METHOD_OID	8
+#define PROGRESS_CREATEIDX_PHASE				9	/* AM-agnostic phase # */
+#define PROGRESS_CREATEIDX_SUBPHASE				10	/* phase # filled by AM */
+#define PROGRESS_CREATEIDX_TUPLES_TOTAL			11
+#define PROGRESS_CREATEIDX_TUPLES_DONE			12
+#define PROGRESS_CREATEIDX_PARTITIONS_TOTAL		13
+#define PROGRESS_CREATEIDX_PARTITIONS_DONE		14
+/* 15 and 16 reserved for "block number" metrics */
+
+/* Phases of CREATE INDEX (as advertised via PROGRESS_CREATEIDX_PHASE) */
+#define PROGRESS_CREATEIDX_PHASE_WAIT_1			1
+#define PROGRESS_CREATEIDX_PHASE_BUILD			2
+#define PROGRESS_CREATEIDX_PHASE_WAIT_2			3
+#define PROGRESS_CREATEIDX_PHASE_VALIDATE_IDXSCAN		4
+#define PROGRESS_CREATEIDX_PHASE_SORT_IDXSCAN		5
+#define PROGRESS_CREATEIDX_PHASE_HEAPSCAN_VALIDATE		6
+#define PROGRESS_CREATEIDX_PHASE_WAIT_3			7
+
+/*
+ * Subphases of CREATE INDEX, for index_build.
+ */
+#define PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE	1
+/* Additional phases are defined by each AM */
+
+/* Lock holder wait counts */
+#define PROGRESS_WAITFOR_TOTAL					3
+#define PROGRESS_WAITFOR_DONE					4
+#define PROGRESS_WAITFOR_CURRENT_PID			5
+
+/* Block numbers in a generic relation scan */
+#define PROGRESS_SCAN_BLOCKS_TOTAL				15
+#define PROGRESS_SCAN_BLOCKS_DONE				16
+
 #endif
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index c080fa6388f..53d4a9c4319 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -951,10 +951,11 @@ typedef enum ProgressCommandType
 {
 	PROGRESS_COMMAND_INVALID,
 	PROGRESS_COMMAND_VACUUM,
-	PROGRESS_COMMAND_CLUSTER
+	PROGRESS_COMMAND_CLUSTER,
+	PROGRESS_COMMAND_CREATE_INDEX
 } ProgressCommandType;
 
-#define PGSTAT_NUM_PROGRESS_PARAM	10
+#define PGSTAT_NUM_PROGRESS_PARAM	20
 
 /* ----------
  * Shared-memory data structures
diff --git a/src/include/storage/lmgr.h b/src/include/storage/lmgr.h
index 3d705faba5c..4f2872de35f 100644
--- a/src/include/storage/lmgr.h
+++ b/src/include/storage/lmgr.h
@@ -78,8 +78,8 @@ extern void XactLockTableWait(TransactionId xid, Relation rel,
 extern bool ConditionalXactLockTableWait(TransactionId xid);
 
 /* Lock VXIDs, specified by conflicting locktags */
-extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode);
-extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode);
+extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress);
+extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress);
 
 /* Lock an XID for tuple insertion (used to wait for an insertion to finish) */
 extern uint32 SpeculativeInsertionLockAcquire(TransactionId xid);
diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h
index badf7fd682b..048947c50d4 100644
--- a/src/include/storage/lock.h
+++ b/src/include/storage/lock.h
@@ -544,7 +544,7 @@ extern bool LockHeldByMe(const LOCKTAG *locktag, LOCKMODE lockmode);
 extern bool LockHasWaiters(const LOCKTAG *locktag,
 			   LOCKMODE lockmode, bool sessionLock);
 extern VirtualTransactionId *GetLockConflicts(const LOCKTAG *locktag,
-				 LOCKMODE lockmode);
+				 LOCKMODE lockmode, int *countp);
 extern void AtPrepare_Locks(void);
 extern void PostPrepare_Locks(TransactionId xid);
 extern int LockCheckConflicts(LockMethod lockMethodTable,
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index d5f309fbfbe..51d7a150cfb 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1856,7 +1856,33 @@ pg_stat_progress_cluster| SELECT s.pid,
     s.param6 AS heap_blks_total,
     s.param7 AS heap_blks_scanned,
     s.param8 AS index_rebuild_count
-   FROM (pg_stat_get_progress_info('CLUSTER'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10)
+   FROM (pg_stat_get_progress_info('CLUSTER'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
+     LEFT JOIN pg_database d ON ((s.datid = d.oid)));
+pg_stat_progress_create_index| SELECT s.pid,
+    s.datid,
+    d.datname,
+    s.relid,
+        CASE s.param10
+            WHEN 0 THEN 'initializing'::text
+            WHEN 1 THEN 'waiting for old snapshots'::text
+            WHEN 2 THEN ('building index'::text || COALESCE((': '::text || pg_indexam_progress_phasename((s.param9)::oid, s.param11)), ''::text))
+            WHEN 3 THEN 'waiting for writer snapshots'::text
+            WHEN 4 THEN 'index validation: scan index'::text
+            WHEN 5 THEN 'index validation: sort index scan results'::text
+            WHEN 6 THEN 'index validation: scan heap'::text
+            WHEN 7 THEN 'waiting for reader snapshots'::text
+            ELSE NULL::text
+        END AS phase,
+    s.param4 AS lockers_total,
+    s.param5 AS lockers_done,
+    s.param6 AS current_locker_pid,
+    s.param16 AS blocks_total,
+    s.param17 AS blocks_done,
+    s.param12 AS tuples_total,
+    s.param13 AS tuples_done,
+    s.param14 AS partitions_total,
+    s.param15 AS partitions_done
+   FROM (pg_stat_get_progress_info('CREATE INDEX'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
      LEFT JOIN pg_database d ON ((s.datid = d.oid)));
 pg_stat_progress_vacuum| SELECT s.pid,
     s.datid,
@@ -1878,7 +1904,7 @@ pg_stat_progress_vacuum| SELECT s.pid,
     s.param5 AS index_vacuum_count,
     s.param6 AS max_dead_tuples,
     s.param7 AS num_dead_tuples
-   FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10)
+   FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
      LEFT JOIN pg_database d ON ((s.datid = d.oid)));
 pg_stat_replication| SELECT s.pid,
     s.usesysid,
-- 
2.17.1

v9-0002-report-partial-progress-of-other-index-AMs.patchtext/x-diff; charset=us-asciiDownload
From 92591450cfc5e942eb2b0301d2b2b4e29f731d75 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Tue, 26 Feb 2019 14:34:49 -0300
Subject: [PATCH v9 2/2] report (partial) progress of other index AMs

---
 contrib/bloom/blinsert.c              | 2 +-
 src/backend/access/brin/brin.c        | 2 +-
 src/backend/access/gin/gininsert.c    | 2 +-
 src/backend/access/gist/gistbuild.c   | 2 +-
 src/backend/access/hash/hash.c        | 6 +++++-
 src/backend/access/hash/hashsort.c    | 6 ++++++
 src/backend/access/spgist/spginsert.c | 2 +-
 7 files changed, 16 insertions(+), 6 deletions(-)

diff --git a/contrib/bloom/blinsert.c b/contrib/bloom/blinsert.c
index 48f35d39990..4b2186b8dda 100644
--- a/contrib/bloom/blinsert.c
+++ b/contrib/bloom/blinsert.c
@@ -142,7 +142,7 @@ blbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	initCachedPage(&buildstate);
 
 	/* Do the heap scan */
-	reltuples = table_index_build_scan(heap, index, indexInfo, true, false,
+	reltuples = table_index_build_scan(heap, index, indexInfo, true, true,
 									   bloomBuildCallback, (void *) &buildstate,
 									   NULL);
 
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 54273f754f6..5c2b0c76358 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -720,7 +720,7 @@ brinbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	 * Now scan the relation.  No syncscan allowed here because we want the
 	 * heap blocks in physical order.
 	 */
-	reltuples = table_index_build_scan(heap, index, indexInfo, false, false,
+	reltuples = table_index_build_scan(heap, index, indexInfo, false, true,
 									   brinbuildCallback, (void *) state, NULL);
 
 	/* process the final batch */
diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c
index b4b0213f76f..edc353a7fe0 100644
--- a/src/backend/access/gin/gininsert.c
+++ b/src/backend/access/gin/gininsert.c
@@ -395,7 +395,7 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	 * Do the heap scan.  We disallow sync scan here because dataPlaceToPage
 	 * prefers to receive tuples in TID order.
 	 */
-	reltuples = table_index_build_scan(heap, index, indexInfo, false, false,
+	reltuples = table_index_build_scan(heap, index, indexInfo, false, true,
 									   ginBuildCallback, (void *) &buildstate,
 									   NULL);
 
diff --git a/src/backend/access/gist/gistbuild.c b/src/backend/access/gist/gistbuild.c
index 771bd2962a7..6024671989e 100644
--- a/src/backend/access/gist/gistbuild.c
+++ b/src/backend/access/gist/gistbuild.c
@@ -205,7 +205,7 @@ gistbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	/*
 	 * Do the heap scan.
 	 */
-	reltuples = table_index_build_scan(heap, index, indexInfo, true, false,
+	reltuples = table_index_build_scan(heap, index, indexInfo, true, true,
 									   gistBuildCallback,
 									   (void *) &buildstate, NULL);
 
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 6b38dd9c0e9..048e40e46fa 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -23,9 +23,11 @@
 #include "access/relscan.h"
 #include "access/tableam.h"
 #include "catalog/index.h"
+#include "commands/progress.h"
 #include "commands/vacuum.h"
 #include "miscadmin.h"
 #include "optimizer/plancat.h"
+#include "pgstat.h"
 #include "utils/builtins.h"
 #include "utils/index_selfuncs.h"
 #include "utils/rel.h"
@@ -161,9 +163,11 @@ hashbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 	buildstate.heapRel = heap;
 
 	/* do the heap scan */
-	reltuples = table_index_build_scan(heap, index, indexInfo, true, false,
+	reltuples = table_index_build_scan(heap, index, indexInfo, true, true,
 									   hashbuildCallback,
 									   (void *) &buildstate, NULL);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_TOTAL,
+								 buildstate.indtuples);
 
 	if (buildstate.spool)
 	{
diff --git a/src/backend/access/hash/hashsort.c b/src/backend/access/hash/hashsort.c
index 8c55436b193..00a57470a77 100644
--- a/src/backend/access/hash/hashsort.c
+++ b/src/backend/access/hash/hashsort.c
@@ -26,7 +26,9 @@
 #include "postgres.h"
 
 #include "access/hash.h"
+#include "commands/progress.h"
 #include "miscadmin.h"
+#include "pgstat.h"
 #include "utils/tuplesort.h"
 
 
@@ -116,6 +118,7 @@ void
 _h_indexbuild(HSpool *hspool, Relation heapRel)
 {
 	IndexTuple	itup;
+	long		tups_done = 0;
 #ifdef USE_ASSERT_CHECKING
 	uint32		hashkey = 0;
 #endif
@@ -141,5 +144,8 @@ _h_indexbuild(HSpool *hspool, Relation heapRel)
 #endif
 
 		_hash_doinsert(hspool->index, itup, heapRel);
+
+		pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE,
+									 ++tups_done);
 	}
 }
diff --git a/src/backend/access/spgist/spginsert.c b/src/backend/access/spgist/spginsert.c
index 282d6998cf0..b06feafdc24 100644
--- a/src/backend/access/spgist/spginsert.c
+++ b/src/backend/access/spgist/spginsert.c
@@ -143,7 +143,7 @@ spgbuild(Relation heap, Relation index, IndexInfo *indexInfo)
 											  "SP-GiST build temporary context",
 											  ALLOCSET_DEFAULT_SIZES);
 
-	reltuples = table_index_build_scan(heap, index, indexInfo, true, false,
+	reltuples = table_index_build_scan(heap, index, indexInfo, true, true,
 									   spgistBuildCallback, (void *) &buildstate,
 									   NULL);
 
-- 
2.17.1

#52Rahila Syed
rahila.syed@2ndquadrant.com
In reply to: Alvaro Herrera (#51)
Re: monitoring CREATE INDEX [CONCURRENTLY]

Hi,

On Mon, 1 Apr 2019 at 21:40, Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:

Hi Rahila, thanks for reviewing.

On 2019-Mar-25, Rahila Syed wrote:

Please see few comments below:

1. Makecheck fails currently as view definition of expected rules.out

does

not reflect latest changes in progress metrics numbering.

Ouch ... fixed.

2. + <entry>
I think there is a typo here 's/partitioned/partitioned table/'

Ah, so there is. Fixed.

3.
I think parallel scan equivalent bpscan->phs_nblocks along with
hscan->rs_nblocks should be used similar to startblock computation above.

Hmm, yeah. I think the way things are setup currently it doesn't matter
much, but it seems fragile to rely on that.

Thank you for incorporating the review comments.

I also moved the reporting of total blocks to scan in the initial table
scan so that it occurs wholly in heapam_index_build_range_scan; I had
originally put that code in _bt_spools_heapscan, but that was a
modularity violation I think. (It didn't make a practical difference,
but it made no sense to have the two cases report the number in wildly
different locations.) Also added a final nblocks metric update after
the scan is done.

I think this patch is done.

I tested the v8 patch by running plain CREATE INDEX, CIC, and for
partitioned tables
and the results are as expected. Please see few observations below.

1. FWIW, below results for CIC show that blocks_done does not become equal
to blocks_total at the end of the phase or it processes last 800 blocks so
fast that
the update is not visible in less than 1 secs interval.

*Mon Mar 25 11:06:31 IST 2019*
pid | datid | datname | relid | phase |
lockers_total | lockers_done | current_locker_pid | blocks_total |
blocks_done | tuples_total | tuples_done | partitions_total |
partitions_done
-------+-------+----------+-------+----------------------------+---------------+--------------+--------------------+--------------+-------------+--------------+-------------+------------------+-----------------
10630 | 13533 | postgres | 16384 | building index: table scan
| 0 | 0 | 0 | 1293366 |
1292535 | 0 | 0 | 0 | 0
(1 row)

*Mon Mar 25 11:06:31 IST 2019*
pid | datid | datname | relid | phase
| lockers_total | lockers_done | current_locker_pid | blocks_total |
blocks_done | tuples_total | tuples_done | partitions_total |
partitions_done
-------+-------+----------+-------+-----------------------------------------+---------------+--------------+--------------------+--------------+-------------+--------------+-------------+------------------+-----------------
10630 | 13533 | postgres | 16384 | building index: sorting tuples, spool 1
| 0 | 0 | 0 | 0
| 0 | 200000000 | 0 | 0
| 0
(1 row)

2. However in case of partitioned tables, the following difference in
blocks_done versus blocks_total at the end of phase is notably high for the
first partition . Subsequent partitions show negligible difference.
Partition 1:
Mon Mar 25 14:27:57 IST 2019
pid | datid | datname | relid | phase |
lockers_total | lockers_done | current_locker_pid | blocks_total |
blocks_done | tuples_total | tuples_done | partitions_total |
partitions_done
-------+-------+----------+-------+----------------------------+---------------+--------------+--------------------+--------------+-------------+--------------+-------------+------------------+-----------------
10630 | 13533 | postgres | 16394 | building index: table scan
| 0 | 0 | 0 | 381342 |
221233 | 0 | 0 | 3 | 0
(1 row)

Mon Mar 25 14:27:57 IST 2019
pid | datid | datname | relid | phase
| lockers_total | lockers_done | current_locker_pid | blocks_total |
blocks_done | tuples_total | tuples_done | partitions_total |
partitions_done
-------+-------+----------+-------+-----------------------------------------+---------------+--------------+--------------------+--------------+-------------+--------------+-------------+------------------+-----------------
10630 | 13533 | postgres | 16394 | building index: sorting tuples, spool 1
| 0 | 0 | 0 | 0
| 0 | 49999999 | 0 | 3
| 0

The partitions are equal in size and the other two partitions have
blocks_done and blocks_total to be approx. 221233. The blocks_total for
partition 1 is reported higher.

3. Sorry for nitpicking, I think following phase name can be made more
consistent with the others.
The non-am specific phase for scanning a table is named as scan heap while
am-specific one is named as table scan.
Can we use heap for am-specific one as well since heap is used elsewhere in
progress reporting too?

4. -       scan = table_beginscan_parallel(btspool->heap,
ParallelTableScanFromBTShared(btshared));
+       scan = table_beginscan_parallel(btspool->heap,
+
ParallelTableScanFromBTShared(btshared));

Is this change required?

Besides the above comments ,the patch looks good to me.

Thank you,
--
Rahila Syed
Performance Engineer
2ndQuadrant
http://www.2ndQuadrant.com <http://www.2ndquadrant.com/&gt;
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#53Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Rahila Syed (#52)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On 2019-Apr-02, Rahila Syed wrote:

1. FWIW, below results for CIC show that blocks_done does not become equal
to blocks_total at the end of the phase or it processes last 800 blocks so
fast that
the update is not visible in less than 1 secs interval.

Yeah, I noticed this too and decided it's not fixable, nor it's
desirable to spend a lot of effort in getting it perfectly accurate -- I
mean, we could introduce locking or sleeping to get the results we want,
but do we really want to make the index building process slower just to
report those block numbers.

Anyway, I think this effect is caused by parallel btree building: those
final blocks are scanned by a worker process, and the leader didn't get
the latest block number scanned. If you set
max_parallel_maintenance_workers to 0, the effect disappears.

(I used \watch 0.01 to see even faster progress updates; even in that
case the final batch of block numbers is not seen in the updates. The
btree build code is stupidly fast.)

2. However in case of partitioned tables, the following difference in
blocks_done versus blocks_total at the end of phase is notably high for the
first partition . Subsequent partitions show negligible difference.
Partition 1:
Mon Mar 25 14:27:57 IST 2019
pid | datid | datname | relid | phase |
lockers_total | lockers_done | current_locker_pid | blocks_total |
blocks_done | tuples_total | tuples_done | partitions_total |
partitions_done
-------+-------+----------+-------+----------------------------+---------------+--------------+--------------------+--------------+-------------+--------------+-------------+------------------+-----------------
10630 | 13533 | postgres | 16394 | building index: table scan
| 0 | 0 | 0 | 381342 |
221233 | 0 | 0 | 3 | 0
(1 row)

Hmm, in my tests with partitioned tables, I never noticed such a large
discrepancy. I'm going to have another look. 800 blocks scanned by
workers I can believe, but 160000 sounds a bit too much.

3. Sorry for nitpicking, I think following phase name can be made more
consistent with the others.
The non-am specific phase for scanning a table is named as scan heap while
am-specific one is named as table scan.
Can we use heap for am-specific one as well since heap is used elsewhere in
progress reporting too?

Hmm, I'd rather go the other way and use "table" everywhere rather than
heap, since we've been getting a lot of stuff done for table AMs.

4. -       scan = table_beginscan_parallel(btspool->heap,
ParallelTableScanFromBTShared(btshared));
+       scan = table_beginscan_parallel(btspool->heap,
+
ParallelTableScanFromBTShared(btshared));

Is this change required?

Yes, for my OCD.

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#54Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Rahila Syed (#52)
Re: monitoring CREATE INDEX [CONCURRENTLY]

I did this (I should stop c&p'ing this silly little setup code sometime):

create table t (a int) partition by hash (a);
create table t1 partition of t for values with (modulus 10, remainder 0);
create table t2 partition of t for values with (modulus 10, remainder 1);
create table t3 partition of t for values with (modulus 10, remainder 2);
create table t4 partition of t for values with (modulus 10, remainder 3);
create table t5 partition of t for values with (modulus 10, remainder 4);
create table t6 partition of t for values with (modulus 10, remainder 5);
create table t7 partition of t for values with (modulus 10, remainder 6);
create table t8 partition of t for values with (modulus 10, remainder 7);
create table t9 partition of t for values with (modulus 10, remainder 8);
create table t10 partition of t for values with (modulus 10, remainder 9);
insert into t select * from generate_series(1, 100 * 1000 * 1000);

Here's a complete report for CIC on a partition (since partitioned tables don't
support CIC anyway):

select relid::regclass, phase,
format('lockers: %s/%s (%s)', lockers_done, lockers_total, current_locker_pid) as lockers,
format('blocks: %s/%s', blocks_done, blocks_total) as blocks,
format('tuples: %s/%s', tuples_done, tuples_total) as tuples,
format('partitions: %s/%s', partitions_done, partitions_total) as partitions
from pg_stat_progress_create_index
\watch 0,1

lun 01 abr 2019 19:02:31 -03 (cada 0,1s)

relid | phase | lockers | blocks | tuples | partitions
-------+--------------------------------+------------------+------------------+-------------+-----------------
t2 | building index: scanning table | lockers: 0/0 (0) | blocks: 86/44254 | tuples: 0/0 | partitions: 0/0
t2 | building index: scanning table | lockers: 0/0 (0) | blocks: 4639/44254 | tuples: 0/0 | partitions: 0/0
t2 | building index: scanning table | lockers: 0/0 (0) | blocks: 10890/44254 | tuples: 0/0 | partitions: 0/0
t2 | building index: scanning table | lockers: 0/0 (0) | blocks: 17841/44254 | tuples: 0/0 | partitions: 0/0
t2 | building index: scanning table | lockers: 0/0 (0) | blocks: 22899/44254 | tuples: 0/0 | partitions: 0/0
t2 | building index: scanning table | lockers: 0/0 (0) | blocks: 29668/44254 | tuples: 0/0 | partitions: 0/0
t2 | building index: scanning table | lockers: 0/0 (0) | blocks: 35531/44254 | tuples: 0/0 | partitions: 0/0
t2 | building index: scanning table | lockers: 0/0 (0) | blocks: 41375/44254 | tuples: 0/0 | partitions: 0/0
t2 | building index: scanning table | lockers: 0/0 (0) | blocks: 44254/44254 | tuples: 0/0 | partitions: 0/0
t2 | building index: scanning table | lockers: 0/0 (0) | blocks: 44254/44254 | tuples: 0/0 | partitions: 0/0
t2 | building index: scanning table | lockers: 0/0 (0) | blocks: 44254/44254 | tuples: 0/0 | partitions: 0/0
t2 | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 291737/10001366 | partitions: 0/0
t2 | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 1652429/10001366 | partitions: 0/0
t2 | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 2984365/10001366 | partitions: 0/0
t2 | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 4139066/10001366 | partitions: 0/0
t2 | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 5463784/10001366 | partitions: 0/0
t2 | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 6699498/10001366 | partitions: 0/0
t2 | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 7947694/10001366 | partitions: 0/0
t2 | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 9311113/10001366 | partitions: 0/0
t2 | index validation: scan index | lockers: 0/0 (0) | blocks: 2542/27426 | tuples: 0/0 | partitions: 0/0
t2 | index validation: scan index | lockers: 0/0 (0) | blocks: 7667/27426 | tuples: 0/0 | partitions: 0/0
t2 | index validation: scan index | lockers: 0/0 (0) | blocks: 15334/27426 | tuples: 0/0 | partitions: 0/0
t2 | index validation: scan index | lockers: 0/0 (0) | blocks: 23001/27426 | tuples: 0/0 | partitions: 0/0
t2 | index validation: sort index scan results | lockers: 0/0 (0) | blocks: 0/0 | tuples: 0/0 | partitions: 0/0
t2 | index validation: scan heap | lockers: 0/0 (0) | blocks: 2586/44254 | tuples: 0/0 | partitions: 0/0
t2 | index validation: scan heap | lockers: 0/0 (0) | blocks: 8180/44254 | tuples: 0/0 | partitions: 0/0
t2 | index validation: scan heap | lockers: 0/0 (0) | blocks: 13807/44254 | tuples: 0/0 | partitions: 0/0
t2 | index validation: scan heap | lockers: 0/0 (0) | blocks: 19365/44254 | tuples: 0/0 | partitions: 0/0
t2 | index validation: scan heap | lockers: 0/0 (0) | blocks: 24990/44254 | tuples: 0/0 | partitions: 0/0
t2 | index validation: scan heap | lockers: 0/0 (0) | blocks: 30943/44254 | tuples: 0/0 | partitions: 0/0
t2 | index validation: scan heap | lockers: 0/0 (0) | blocks: 36817/44254 | tuples: 0/0 | partitions: 0/0
t2 | index validation: scan heap | lockers: 0/0 (0) | blocks: 43025/44254 | tuples: 0/0 | partitions: 0/0

Here's the report for building the index on the partitioned table, same query, this time \watch 0,05:

t | building index: scanning table | lockers: 0/0 (0) | blocks: 168/44232 | tuples: 0/0 | partitions: 0/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 2022/44232 | tuples: 0/0 | partitions: 0/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 4378/44232 | tuples: 0/0 | partitions: 0/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 6419/44232 | tuples: 0/0 | partitions: 0/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 8599/44232 | tuples: 0/0 | partitions: 0/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 10955/44232 | tuples: 0/0 | partitions: 0/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 12976/44232 | tuples: 0/0 | partitions: 0/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 15627/44232 | tuples: 0/0 | partitions: 0/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 17389/44232 | tuples: 0/0 | partitions: 0/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 20213/44232 | tuples: 0/0 | partitions: 0/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 22351/44232 | tuples: 0/0 | partitions: 0/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 24788/44232 | tuples: 0/0 | partitions: 0/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 26880/44232 | tuples: 0/0 | partitions: 0/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 29382/44232 | tuples: 0/0 | partitions: 0/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 31289/44232 | tuples: 0/0 | partitions: 0/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 33826/44232 | tuples: 0/0 | partitions: 0/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 35815/44232 | tuples: 0/0 | partitions: 0/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 38415/44232 | tuples: 0/0 | partitions: 0/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 40335/44232 | tuples: 0/0 | partitions: 0/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 43002/44232 | tuples: 0/0 | partitions: 0/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44232/44232 | tuples: 0/0 | partitions: 0/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44232/44232 | tuples: 0/0 | partitions: 0/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44232/44232 | tuples: 0/0 | partitions: 0/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44232/44232 | tuples: 0/0 | partitions: 0/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44232/44232 | tuples: 0/0 | partitions: 0/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44232/44232 | tuples: 0/0 | partitions: 0/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44232/44232 | tuples: 0/0 | partitions: 0/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 361108/9996429 | partitions: 0/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 1054447/9996429 | partitions: 0/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 1749066/9996429 | partitions: 0/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 2433169/9996429 | partitions: 0/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 3129578/9996429 | partitions: 0/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 3626695/9996429 | partitions: 0/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 4308212/9996429 | partitions: 0/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 5001391/9996429 | partitions: 0/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 5698987/9996429 | partitions: 0/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 6391459/9996429 | partitions: 0/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 6894091/9996429 | partitions: 0/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 7593037/9996429 | partitions: 0/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 8286403/9996429 | partitions: 0/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 8974321/9996429 | partitions: 0/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 9667525/9996429 | partitions: 0/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 9996429/9996429 | partitions: 0/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 2476/44254 | tuples: 0/0 | partitions: 1/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 4639/44254 | tuples: 0/0 | partitions: 1/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 6717/44254 | tuples: 0/0 | partitions: 1/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 8952/44254 | tuples: 0/0 | partitions: 1/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 11245/44254 | tuples: 0/0 | partitions: 1/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 13361/44254 | tuples: 0/0 | partitions: 1/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 15777/44254 | tuples: 0/0 | partitions: 1/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 17716/44254 | tuples: 0/0 | partitions: 1/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 20335/44254 | tuples: 0/0 | partitions: 1/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 22114/44254 | tuples: 0/0 | partitions: 1/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 24831/44254 | tuples: 0/0 | partitions: 1/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 26529/44254 | tuples: 0/0 | partitions: 1/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 29405/44254 | tuples: 0/0 | partitions: 1/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 31153/44254 | tuples: 0/0 | partitions: 1/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 33981/44254 | tuples: 0/0 | partitions: 1/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 35595/44254 | tuples: 0/0 | partitions: 1/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 38511/44254 | tuples: 0/0 | partitions: 1/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 40114/44254 | tuples: 0/0 | partitions: 1/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 43040/44254 | tuples: 0/0 | partitions: 1/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44254/44254 | tuples: 0/0 | partitions: 1/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44254/44254 | tuples: 0/0 | partitions: 1/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44254/44254 | tuples: 0/0 | partitions: 1/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44254/44254 | tuples: 0/0 | partitions: 1/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44254/44254 | tuples: 0/0 | partitions: 1/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44254/44254 | tuples: 0/0 | partitions: 1/10
t | building index: sorting live tuples | lockers: 0/0 (0) | blocks: 0/0 | tuples: 0/10001366 | partitions: 1/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 500779/10001366 | partitions: 1/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 1201309/10001366 | partitions: 1/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 1883508/10001366 | partitions: 1/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 2582863/10001366 | partitions: 1/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 3258358/10001366 | partitions: 1/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 3750979/10001366 | partitions: 1/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 4430740/10001366 | partitions: 1/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 5105701/10001366 | partitions: 1/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 5798905/10001366 | partitions: 1/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 6484789/10001366 | partitions: 1/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 6992211/10001366 | partitions: 1/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 7689065/10001366 | partitions: 1/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 8391543/10001366 | partitions: 1/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 9087049/10001366 | partitions: 1/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 9764485/10001366 | partitions: 1/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 269/44276 | tuples: 0/0 | partitions: 2/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 2861/44276 | tuples: 0/0 | partitions: 2/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 4639/44276 | tuples: 0/0 | partitions: 2/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 6016/44276 | tuples: 0/0 | partitions: 2/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 8266/44276 | tuples: 0/0 | partitions: 2/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 9916/44276 | tuples: 0/0 | partitions: 2/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 12793/44276 | tuples: 0/0 | partitions: 2/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 14423/44276 | tuples: 0/0 | partitions: 2/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 17302/44276 | tuples: 0/0 | partitions: 2/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 18936/44276 | tuples: 0/0 | partitions: 2/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 21837/44276 | tuples: 0/0 | partitions: 2/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 23480/44276 | tuples: 0/0 | partitions: 2/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 26369/44276 | tuples: 0/0 | partitions: 2/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 27993/44276 | tuples: 0/0 | partitions: 2/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 30891/44276 | tuples: 0/0 | partitions: 2/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 32525/44276 | tuples: 0/0 | partitions: 2/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 35357/44276 | tuples: 0/0 | partitions: 2/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 37096/44276 | tuples: 0/0 | partitions: 2/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 39846/44276 | tuples: 0/0 | partitions: 2/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 41645/44276 | tuples: 0/0 | partitions: 2/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44276/44276 | tuples: 0/0 | partitions: 2/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44276/44276 | tuples: 0/0 | partitions: 2/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44276/44276 | tuples: 0/0 | partitions: 2/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44276/44276 | tuples: 0/0 | partitions: 2/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44276/44276 | tuples: 0/0 | partitions: 2/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44276/44276 | tuples: 0/0 | partitions: 2/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44276/44276 | tuples: 0/0 | partitions: 2/10
t | building index: sorting live tuples | lockers: 0/0 (0) | blocks: 0/0 | tuples: 0/10006231 | partitions: 2/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 663688/10006231 | partitions: 2/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 1355654/10006231 | partitions: 2/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 2057087/10006231 | partitions: 2/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 2764765/10006231 | partitions: 2/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 3377603/10006231 | partitions: 2/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 3987118/10006231 | partitions: 2/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 4685206/10006231 | partitions: 2/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 5387918/10006231 | partitions: 2/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 6091400/10006231 | partitions: 2/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 6694322/10006231 | partitions: 2/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 7304995/10006231 | partitions: 2/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 7984067/10006231 | partitions: 2/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 8672517/10006231 | partitions: 2/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 9351567/10006231 | partitions: 2/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 10006230/10006231 | partitions: 2/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 1178/44255 | tuples: 0/0 | partitions: 3/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 3833/44255 | tuples: 0/0 | partitions: 3/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 5288/44255 | tuples: 0/0 | partitions: 3/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 8125/44255 | tuples: 0/0 | partitions: 3/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 9760/44255 | tuples: 0/0 | partitions: 3/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 12663/44255 | tuples: 0/0 | partitions: 3/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 14323/44255 | tuples: 0/0 | partitions: 3/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 17252/44255 | tuples: 0/0 | partitions: 3/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 18880/44255 | tuples: 0/0 | partitions: 3/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 21807/44255 | tuples: 0/0 | partitions: 3/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 23444/44255 | tuples: 0/0 | partitions: 3/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 26360/44255 | tuples: 0/0 | partitions: 3/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 28008/44255 | tuples: 0/0 | partitions: 3/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 30932/44255 | tuples: 0/0 | partitions: 3/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 32587/44255 | tuples: 0/0 | partitions: 3/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 35375/44255 | tuples: 0/0 | partitions: 3/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 37090/44255 | tuples: 0/0 | partitions: 3/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 39827/44255 | tuples: 0/0 | partitions: 3/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 41657/44255 | tuples: 0/0 | partitions: 3/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44255/44255 | tuples: 0/0 | partitions: 3/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44255/44255 | tuples: 0/0 | partitions: 3/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44255/44255 | tuples: 0/0 | partitions: 3/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44255/44255 | tuples: 0/0 | partitions: 3/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44255/44255 | tuples: 0/0 | partitions: 3/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44255/44255 | tuples: 0/0 | partitions: 3/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44255/44255 | tuples: 0/0 | partitions: 3/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 118053/10001417 | partitions: 3/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 820650/10001417 | partitions: 3/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 1512165/10001417 | partitions: 3/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 2201720/10001417 | partitions: 3/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 2883192/10001417 | partitions: 3/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 3388429/10001417 | partitions: 3/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 4077607/10001417 | partitions: 3/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 4770319/10001417 | partitions: 3/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 5453432/10001417 | partitions: 3/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 6130568/10001417 | partitions: 3/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 6725569/10001417 | partitions: 3/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 7327477/10001417 | partitions: 3/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 7989790/10001417 | partitions: 3/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 8685547/10001417 | partitions: 3/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 9365865/10001417 | partitions: 3/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 9981553/10001417 | partitions: 3/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 1060/44259 | tuples: 0/0 | partitions: 4/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 3570/44259 | tuples: 0/0 | partitions: 4/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 4880/44259 | tuples: 0/0 | partitions: 4/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 7585/44259 | tuples: 0/0 | partitions: 4/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 9172/44259 | tuples: 0/0 | partitions: 4/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 12060/44259 | tuples: 0/0 | partitions: 4/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 13670/44259 | tuples: 0/0 | partitions: 4/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 16529/44259 | tuples: 0/0 | partitions: 4/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 18132/44259 | tuples: 0/0 | partitions: 4/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 20976/44259 | tuples: 0/0 | partitions: 4/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 22557/44259 | tuples: 0/0 | partitions: 4/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 25393/44259 | tuples: 0/0 | partitions: 4/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 26973/44259 | tuples: 0/0 | partitions: 4/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 29800/44259 | tuples: 0/0 | partitions: 4/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 31372/44259 | tuples: 0/0 | partitions: 4/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 34214/44259 | tuples: 0/0 | partitions: 4/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 35789/44259 | tuples: 0/0 | partitions: 4/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 38667/44259 | tuples: 0/0 | partitions: 4/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 39728/44259 | tuples: 0/0 | partitions: 4/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 40861/44259 | tuples: 0/0 | partitions: 4/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 43730/44259 | tuples: 0/0 | partitions: 4/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44259/44259 | tuples: 0/0 | partitions: 4/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44259/44259 | tuples: 0/0 | partitions: 4/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44259/44259 | tuples: 0/0 | partitions: 4/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44259/44259 | tuples: 0/0 | partitions: 4/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44259/44259 | tuples: 0/0 | partitions: 4/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44259/44259 | tuples: 0/0 | partitions: 4/10
t | building index: sorting live tuples | lockers: 0/0 (0) | blocks: 0/0 | tuples: 0/10002444 | partitions: 4/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 518863/10002444 | partitions: 4/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 1211461/10002444 | partitions: 4/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 1907959/10002444 | partitions: 4/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 2597929/10002444 | partitions: 4/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 3285807/10002444 | partitions: 4/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 3804693/10002444 | partitions: 4/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 4495350/10002444 | partitions: 4/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 5195042/10002444 | partitions: 4/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 5890039/10002444 | partitions: 4/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 6587559/10002444 | partitions: 4/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 7097473/10002444 | partitions: 4/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 7786066/10002444 | partitions: 4/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 8480443/10002444 | partitions: 4/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 9174157/10002444 | partitions: 4/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 9855283/10002444 | partitions: 4/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 559/44249 | tuples: 0/0 | partitions: 5/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 3158/44249 | tuples: 0/0 | partitions: 5/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 4639/44249 | tuples: 0/0 | partitions: 5/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 7416/44249 | tuples: 0/0 | partitions: 5/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 9069/44249 | tuples: 0/0 | partitions: 5/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 11908/44249 | tuples: 0/0 | partitions: 5/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 13567/44249 | tuples: 0/0 | partitions: 5/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 16473/44249 | tuples: 0/0 | partitions: 5/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 18147/44249 | tuples: 0/0 | partitions: 5/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 21024/44249 | tuples: 0/0 | partitions: 5/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 22690/44249 | tuples: 0/0 | partitions: 5/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 25582/44249 | tuples: 0/0 | partitions: 5/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 27239/44249 | tuples: 0/0 | partitions: 5/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 30148/44249 | tuples: 0/0 | partitions: 5/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 31812/44249 | tuples: 0/0 | partitions: 5/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 34722/44249 | tuples: 0/0 | partitions: 5/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 36378/44249 | tuples: 0/0 | partitions: 5/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 39290/44249 | tuples: 0/0 | partitions: 5/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 40959/44249 | tuples: 0/0 | partitions: 5/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 43844/44249 | tuples: 0/0 | partitions: 5/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44249/44249 | tuples: 0/0 | partitions: 5/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44249/44249 | tuples: 0/0 | partitions: 5/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44249/44249 | tuples: 0/0 | partitions: 5/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44249/44249 | tuples: 0/0 | partitions: 5/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44249/44249 | tuples: 0/0 | partitions: 5/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44249/44249 | tuples: 0/0 | partitions: 5/10
t | building index: sorting live tuples | lockers: 0/0 (0) | blocks: 0/0 | tuples: 0/10000082 | partitions: 5/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 701199/10000082 | partitions: 5/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 1386576/10000082 | partitions: 5/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 2026138/10000082 | partitions: 5/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 2686458/10000082 | partitions: 5/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 3328278/10000082 | partitions: 5/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 3886189/10000082 | partitions: 5/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 4566129/10000082 | partitions: 5/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 5250205/10000082 | partitions: 5/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 5936354/10000082 | partitions: 5/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 6632954/10000082 | partitions: 5/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 7131820/10000082 | partitions: 5/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 7801535/10000082 | partitions: 5/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 8472974/10000082 | partitions: 5/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 9122825/10000082 | partitions: 5/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 9801504/10000082 | partitions: 5/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 429/44220 | tuples: 0/0 | partitions: 6/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 3011/44220 | tuples: 0/0 | partitions: 6/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 4639/44220 | tuples: 0/0 | partitions: 6/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 7332/44220 | tuples: 0/0 | partitions: 6/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 8987/44220 | tuples: 0/0 | partitions: 6/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 11878/44220 | tuples: 0/0 | partitions: 6/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 13537/44220 | tuples: 0/0 | partitions: 6/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 16425/44220 | tuples: 0/0 | partitions: 6/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 18106/44220 | tuples: 0/0 | partitions: 6/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 21040/44220 | tuples: 0/0 | partitions: 6/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 22718/44220 | tuples: 0/0 | partitions: 6/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 25635/44220 | tuples: 0/0 | partitions: 6/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 27312/44220 | tuples: 0/0 | partitions: 6/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 30215/44220 | tuples: 0/0 | partitions: 6/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 31876/44220 | tuples: 0/0 | partitions: 6/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 34800/44220 | tuples: 0/0 | partitions: 6/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 36455/44220 | tuples: 0/0 | partitions: 6/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 39359/44220 | tuples: 0/0 | partitions: 6/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 41032/44220 | tuples: 0/0 | partitions: 6/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 43943/44220 | tuples: 0/0 | partitions: 6/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44220/44220 | tuples: 0/0 | partitions: 6/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44220/44220 | tuples: 0/0 | partitions: 6/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44220/44220 | tuples: 0/0 | partitions: 6/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44220/44220 | tuples: 0/0 | partitions: 6/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44220/44220 | tuples: 0/0 | partitions: 6/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44220/44220 | tuples: 0/0 | partitions: 6/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 25987/9993603 | partitions: 6/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 732379/9993603 | partitions: 6/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 1424513/9993603 | partitions: 6/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 2081443/9993603 | partitions: 6/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 2704663/9993603 | partitions: 6/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 3342516/9993603 | partitions: 6/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 3812257/9993603 | partitions: 6/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 4489696/9993603 | partitions: 6/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 5146693/9993603 | partitions: 6/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 5833638/9993603 | partitions: 6/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 6524497/9993603 | partitions: 6/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 7029547/9993603 | partitions: 6/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 7715519/9993603 | partitions: 6/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 8417783/9993603 | partitions: 6/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 9099477/9993603 | partitions: 6/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 9781549/9993603 | partitions: 6/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 368/44259 | tuples: 0/0 | partitions: 7/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 2921/44259 | tuples: 0/0 | partitions: 7/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 4639/44259 | tuples: 0/0 | partitions: 7/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 7160/44259 | tuples: 0/0 | partitions: 7/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 8878/44259 | tuples: 0/0 | partitions: 7/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 11603/44259 | tuples: 0/0 | partitions: 7/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 13265/44259 | tuples: 0/0 | partitions: 7/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 16175/44259 | tuples: 0/0 | partitions: 7/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 17962/44259 | tuples: 0/0 | partitions: 7/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 20751/44259 | tuples: 0/0 | partitions: 7/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 22442/44259 | tuples: 0/0 | partitions: 7/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 25320/44259 | tuples: 0/0 | partitions: 7/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 26957/44259 | tuples: 0/0 | partitions: 7/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 29858/44259 | tuples: 0/0 | partitions: 7/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 31467/44259 | tuples: 0/0 | partitions: 7/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 34382/44259 | tuples: 0/0 | partitions: 7/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 36023/44259 | tuples: 0/0 | partitions: 7/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 38874/44259 | tuples: 0/0 | partitions: 7/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 40532/44259 | tuples: 0/0 | partitions: 7/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 43439/44259 | tuples: 0/0 | partitions: 7/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44259/44259 | tuples: 0/0 | partitions: 7/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44259/44259 | tuples: 0/0 | partitions: 7/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44259/44259 | tuples: 0/0 | partitions: 7/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44259/44259 | tuples: 0/0 | partitions: 7/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44259/44259 | tuples: 0/0 | partitions: 7/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44259/44259 | tuples: 0/0 | partitions: 7/10
t | building index: sorting live tuples | lockers: 0/0 (0) | blocks: 0/0 | tuples: 0/10002524 | partitions: 7/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 560347/10002524 | partitions: 7/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 1248199/10002524 | partitions: 7/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 1947121/10002524 | partitions: 7/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 2634011/10002524 | partitions: 7/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 3308164/10002524 | partitions: 7/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 3829093/10002524 | partitions: 7/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 4522592/10002524 | partitions: 7/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 5201695/10002524 | partitions: 7/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 5894625/10002524 | partitions: 7/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 6592741/10002524 | partitions: 7/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 7108453/10002524 | partitions: 7/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 7782573/10002524 | partitions: 7/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 8433007/10002524 | partitions: 7/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 9098304/10002524 | partitions: 7/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 9769812/10002524 | partitions: 7/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 300/44232 | tuples: 0/0 | partitions: 8/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 2880/44232 | tuples: 0/0 | partitions: 8/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 4639/44232 | tuples: 0/0 | partitions: 8/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 7121/44232 | tuples: 0/0 | partitions: 8/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 8974/44232 | tuples: 0/0 | partitions: 8/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 11610/44232 | tuples: 0/0 | partitions: 8/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 13440/44232 | tuples: 0/0 | partitions: 8/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 16136/44232 | tuples: 0/0 | partitions: 8/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 17879/44232 | tuples: 0/0 | partitions: 8/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 20700/44232 | tuples: 0/0 | partitions: 8/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 22293/44232 | tuples: 0/0 | partitions: 8/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 25260/44232 | tuples: 0/0 | partitions: 8/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 26863/44232 | tuples: 0/0 | partitions: 8/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 29775/44232 | tuples: 0/0 | partitions: 8/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 31406/44232 | tuples: 0/0 | partitions: 8/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 34318/44232 | tuples: 0/0 | partitions: 8/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 35974/44232 | tuples: 0/0 | partitions: 8/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 38869/44232 | tuples: 0/0 | partitions: 8/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 40507/44232 | tuples: 0/0 | partitions: 8/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 43431/44232 | tuples: 0/0 | partitions: 8/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44232/44232 | tuples: 0/0 | partitions: 8/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44232/44232 | tuples: 0/0 | partitions: 8/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44232/44232 | tuples: 0/0 | partitions: 8/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44232/44232 | tuples: 0/0 | partitions: 8/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44232/44232 | tuples: 0/0 | partitions: 8/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44232/44232 | tuples: 0/0 | partitions: 8/10
t | building index: sorting live tuples | lockers: 0/0 (0) | blocks: 0/0 | tuples: 0/9996288 | partitions: 8/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 556257/9996288 | partitions: 8/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 1261969/9996288 | partitions: 8/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 1962859/9996288 | partitions: 8/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 2666023/9996288 | partitions: 8/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 3351782/9996288 | partitions: 8/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 3870067/9996288 | partitions: 8/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 4467186/9996288 | partitions: 8/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 5040844/9996288 | partitions: 8/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 5596873/9996288 | partitions: 8/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 6273009/9996288 | partitions: 8/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 6744020/9996288 | partitions: 8/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 7260343/9996288 | partitions: 8/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 7836896/9996288 | partitions: 8/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 8362115/9996288 | partitions: 8/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 8982007/9996288 | partitions: 8/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 9683282/9996288 | partitions: 8/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 257/44247 | tuples: 0/0 | partitions: 9/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 2853/44247 | tuples: 0/0 | partitions: 9/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 4639/44247 | tuples: 0/0 | partitions: 9/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 7071/44247 | tuples: 0/0 | partitions: 9/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 8961/44247 | tuples: 0/0 | partitions: 9/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 11536/44247 | tuples: 0/0 | partitions: 9/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 13377/44247 | tuples: 0/0 | partitions: 9/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 16115/44247 | tuples: 0/0 | partitions: 9/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 17798/44247 | tuples: 0/0 | partitions: 9/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 20649/44247 | tuples: 0/0 | partitions: 9/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 22334/44247 | tuples: 0/0 | partitions: 9/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 25185/44247 | tuples: 0/0 | partitions: 9/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 26818/44247 | tuples: 0/0 | partitions: 9/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 29740/44247 | tuples: 0/0 | partitions: 9/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 31283/44247 | tuples: 0/0 | partitions: 9/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 34208/44247 | tuples: 0/0 | partitions: 9/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 35835/44247 | tuples: 0/0 | partitions: 9/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 38531/44247 | tuples: 0/0 | partitions: 9/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 39637/44247 | tuples: 0/0 | partitions: 9/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 41722/44247 | tuples: 0/0 | partitions: 9/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44082/44247 | tuples: 0/0 | partitions: 9/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44247/44247 | tuples: 0/0 | partitions: 9/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44247/44247 | tuples: 0/0 | partitions: 9/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44247/44247 | tuples: 0/0 | partitions: 9/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44247/44247 | tuples: 0/0 | partitions: 9/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44247/44247 | tuples: 0/0 | partitions: 9/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44247/44247 | tuples: 0/0 | partitions: 9/10
t | building index: scanning table | lockers: 0/0 (0) | blocks: 44247/44247 | tuples: 0/0 | partitions: 9/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 89251/9999616 | partitions: 9/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 724681/9999616 | partitions: 9/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 1352185/9999616 | partitions: 9/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 1978963/9999616 | partitions: 9/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 2545165/9999616 | partitions: 9/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 3116125/9999616 | partitions: 9/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 3495200/9999616 | partitions: 9/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 4038079/9999616 | partitions: 9/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 4612813/9999616 | partitions: 9/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 5239737/9999616 | partitions: 9/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 5738502/9999616 | partitions: 9/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 6336605/9999616 | partitions: 9/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 6779349/9999616 | partitions: 9/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 7417770/9999616 | partitions: 9/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 8041496/9999616 | partitions: 9/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 8722536/9999616 | partitions: 9/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 9142846/9999616 | partitions: 9/10
t | building index: loading tuples in tree | lockers: 0/0 (0) | blocks: 0/0 | tuples: 9782646/9999616 | partitions: 9/10

I think this is working well enough.

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#55Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Alvaro Herrera (#51)
Re: monitoring CREATE INDEX [CONCURRENTLY]

I noticed a couple of inconsistent uses of "heap" which I changed to
"table", and also changed one update_multi_params() call that had only
one param to update_param(). Very minor other tweaks here and there.

And pushed.

Thanks for the reviews!

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#56Peter Eisentraut
peter.eisentraut@2ndquadrant.com
In reply to: Alvaro Herrera (#55)
1 attachment(s)
Re: monitoring CREATE INDEX [CONCURRENTLY]

It seems we can easily extend this to also monitor REINDEX
[CONCURRENTLY]. Attached is a quick patch.

For the concurrently part, we currently don't have any phases defined
for the index swap and drop, but maybe we can just skip that initially.
What happens if we don't have those?

It might be nice to have a column in the view not only for the table OID
but also the index OID. That is obviously not so useful for CREATE
INDEX but more useful for REINDEX. I haven't looked into how adding
that would work.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attachments:

0001-Report-progress-of-REINDEX-operations.patchtext/plain; charset=UTF-8; name=0001-Report-progress-of-REINDEX-operations.patch; x-mac-creator=0; x-mac-type=0Download
From cae47efe485c3c6115c1970b9c922ded1353d270 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Fri, 5 Apr 2019 16:41:27 +0200
Subject: [PATCH] Report progress of REINDEX operations

This uses the same infrastructure that the CREATE INDEX progress
reporting uses.
---
 src/backend/catalog/index.c      |  8 ++++++++
 src/backend/commands/indexcmds.c | 19 +++++++++++++++++--
 2 files changed, 25 insertions(+), 2 deletions(-)

diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 2ed7fdb021..686b3c8260 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -3286,12 +3286,18 @@ reindex_index(Oid indexId, bool skip_constraint_checks, char persistence,
 	heapId = IndexGetRelation(indexId, false);
 	heapRelation = table_open(heapId, ShareLock);
 
+	pgstat_progress_start_command(PROGRESS_COMMAND_CREATE_INDEX,
+								  heapId);
+
 	/*
 	 * Open the target index relation and get an exclusive lock on it, to
 	 * ensure that no one else is touching this particular index.
 	 */
 	iRel = index_open(indexId, AccessExclusiveLock);
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_ACCESS_METHOD_OID,
+								 iRel->rd_rel->relam);
+
 	/*
 	 * The case of reindexing partitioned tables and indexes is handled
 	 * differently by upper layers, so this case shouldn't arise.
@@ -3442,6 +3448,8 @@ reindex_index(Oid indexId, bool skip_constraint_checks, char persistence,
 				 errdetail_internal("%s",
 									pg_rusage_show(&ru0))));
 
+	pgstat_progress_end_command();
+
 	/* Close rels, but keep locks */
 	index_close(iRel, NoLock);
 	table_close(heapRelation, NoLock);
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 348d543297..7efb4c9e15 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2873,6 +2873,11 @@ ReindexRelationConcurrently(Oid relationOid, int options)
 		heapRel = table_open(indexRel->rd_index->indrelid,
 							 ShareUpdateExclusiveLock);
 
+		pgstat_progress_start_command(PROGRESS_COMMAND_CREATE_INDEX,
+									  RelationGetRelid(heapRel));
+		pgstat_progress_update_param(PROGRESS_CREATEIDX_ACCESS_METHOD_OID,
+									 indexRel->rd_rel->relam);
+
 		/* Choose a temporary relation name for the new index */
 		concurrentName = ChooseRelationName(get_rel_name(indexId),
 											NULL,
@@ -2967,7 +2972,9 @@ ReindexRelationConcurrently(Oid relationOid, int options)
 	 * DefineIndex() for more details.
 	 */
 
-	WaitForLockersMultiple(lockTags, ShareLock, false);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_1);
+	WaitForLockersMultiple(lockTags, ShareLock, true);
 	CommitTransactionCommand();
 
 	forboth(lc, indexIds, lc2, newIndexIds)
@@ -3009,7 +3016,9 @@ ReindexRelationConcurrently(Oid relationOid, int options)
 	 * for more details.
 	 */
 
-	WaitForLockersMultiple(lockTags, ShareLock, false);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_2);
+	WaitForLockersMultiple(lockTags, ShareLock, true);
 	CommitTransactionCommand();
 
 	foreach(lc, newIndexIds)
@@ -3057,6 +3066,8 @@ ReindexRelationConcurrently(Oid relationOid, int options)
 		 * before the reference snap was taken, we have to wait out any
 		 * transactions that might have older snapshots.
 		 */
+		pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+									 PROGRESS_CREATEIDX_PHASE_WAIT_3);
 		WaitForOlderSnapshots(limitXmin, false);
 
 		CommitTransactionCommand();
@@ -3128,6 +3139,7 @@ ReindexRelationConcurrently(Oid relationOid, int options)
 	 * index_drop() for more details.
 	 */
 
+	// FIXME: progress update here
 	WaitForLockersMultiple(lockTags, AccessExclusiveLock, false);
 
 	foreach(lc, indexIds)
@@ -3150,6 +3162,7 @@ ReindexRelationConcurrently(Oid relationOid, int options)
 	 * Drop the old indexes.
 	 */
 
+	// FIXME: progress update here
 	WaitForLockersMultiple(lockTags, AccessExclusiveLock, false);
 
 	PushActiveSnapshot(GetTransactionSnapshot());
@@ -3225,6 +3238,8 @@ ReindexRelationConcurrently(Oid relationOid, int options)
 
 	MemoryContextDelete(private_context);
 
+	pgstat_progress_end_command();
+
 	return true;
 }
 
-- 
2.21.0

#57Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Peter Eisentraut (#56)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On 2019-Apr-05, Peter Eisentraut wrote:

It seems we can easily extend this to also monitor REINDEX
[CONCURRENTLY]. Attached is a quick patch.

That's much easier than I was expecting. I think we should endeavor to
get this done for pg12.

For the concurrently part, we currently don't have any phases defined
for the index swap and drop, but maybe we can just skip that initially.
What happens if we don't have those?

Users are going to wonder why the other phases don't appear to complete
for a long time :-) Keep in mind that the "waiting" phases are very
confusing to users. I suggest we just define additional phase numbers
for those phases, then switch the "false" argument to
WaitForLockersMultiple to "true", and it should work :-) Doc-wise, list
all the phases in the same docbook table, indicate that REINDEX is also
covered, and document in an easier-to-follow fashion which phases each
command goes through.

It might be nice to have a column in the view not only for the table OID
but also the index OID. That is obviously not so useful for CREATE
INDEX but more useful for REINDEX. I haven't looked into how adding
that would work.

Yeah, I think that's simple enough -- the CLUSTER one already does that,
I think. Another thing for REINDEX TABLE is that we should add a count
of indexes to process, and how many are done.

I was wondering about reporting the command being run. In the progress_cluster
view we have a "command" column, which says either CLUSTER or VACUUM FULL.
I didn't add one for CREATE INDEX vs. CONCURRENTLY because it seemed to
me that joining to pg_stat_activity ought to be sufficient. If we agree
with that reasoning, then we should remove the column from the CLUSTER
view, I think. If not, we should add one to the create_index view.

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#58Peter Eisentraut
peter.eisentraut@2ndquadrant.com
In reply to: Alvaro Herrera (#57)
1 attachment(s)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On 2019-04-05 17:01, Alvaro Herrera wrote:

Users are going to wonder why the other phases don't appear to complete
for a long time :-) Keep in mind that the "waiting" phases are very
confusing to users. I suggest we just define additional phase numbers
for those phases, then switch the "false" argument to
WaitForLockersMultiple to "true", and it should work :-) Doc-wise, list
all the phases in the same docbook table, indicate that REINDEX is also
covered, and document in an easier-to-follow fashion which phases each
command goes through.

Done in the attached patch.

I've reworded the phases a bit. There was a bit of a mixup of waiting
for snapshots and waiting for lockers. Perhaps not so important from a
user's perspective, but at least now it's more consistent with the
source code comments.

Yeah, I think that's simple enough -- the CLUSTER one already does that,
I think.

Added that.

Another thing for REINDEX TABLE is that we should add a count
of indexes to process, and how many are done.

Reasonable, but maybe a bit too much for the last moment.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attachments:

v2-0001-Report-progress-of-REINDEX-operations.patchtext/plain; charset=UTF-8; name=v2-0001-Report-progress-of-REINDEX-operations.patch; x-mac-creator=0; x-mac-type=0Download
From 22c0254b76be05a2235ed84268d189fa6785e844 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Fri, 5 Apr 2019 22:27:14 +0200
Subject: [PATCH v2] Report progress of REINDEX operations

This uses the same infrastructure that the CREATE INDEX progress
reporting uses.
---
 doc/src/sgml/monitoring.sgml         | 52 +++++++++++++++++++++-------
 src/backend/catalog/index.c          | 10 ++++++
 src/backend/catalog/system_views.sql | 11 +++---
 src/backend/commands/indexcmds.c     | 39 ++++++++++++++++++---
 src/include/commands/progress.h      |  3 ++
 src/test/regress/expected/rules.out  | 11 +++---
 6 files changed, 102 insertions(+), 24 deletions(-)

diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index b946e13fdc..4eb43f2de9 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -346,7 +346,7 @@ <title>Dynamic Statistics Views</title>
 
      <row>
       <entry><structname>pg_stat_progress_create_index</structname><indexterm><primary>pg_stat_progress_create_index</primary></indexterm></entry>
-      <entry>One row for each backend running <command>CREATE INDEX</command>, showing
+      <entry>One row for each backend running <command>CREATE INDEX</command> or <command>REINDEX</command>, showing
       current progress.
       See <xref linkend='create-index-progress-reporting'/>.
      </entry>
@@ -3477,7 +3477,7 @@ <title>Progress Reporting</title>
   <title>CREATE INDEX Progress Reporting</title>
 
   <para>
-   Whenever <command>CREATE INDEX</command> is running, the
+   Whenever <command>CREATE INDEX</command> or <command>REINDEX</command> is running, the
    <structname>pg_stat_progress_create_index</structname> view will contain
    one row for each backend that is currently creating indexes.  The tables
    below describe the information that will be reported and provide information
@@ -3516,6 +3516,12 @@ <title><structname>pg_stat_progress_create_index</structname> View</title>
       <entry><type>oid</type></entry>
       <entry>OID of the table on which the index is being created.</entry>
      </row>
+     <row>
+      <entry><structfield>index_relid</structfield></entry>
+      <entry><type>oid</type></entry>
+      <entry>OID of the index being created or reindexed.  During a
+      non-concurrent <command>CREATE INDEX</command>, this is 0.</entry>
+     </row>
      <row>
       <entry><structfield>phase</structfield></entry>
       <entry><type>text</type></entry>
@@ -3605,15 +3611,15 @@ <title>CREATE INDEX phases</title>
      <row>
       <entry><literal>initializing</literal></entry>
       <entry>
-       <command>CREATE INDEX</command> is preparing to create the index.  This
+       <command>CREATE INDEX</command> or <command>REINDEX</command> is preparing to create the index.  This
        phase is expected to be very brief.
       </entry>
      </row>
      <row>
-      <entry><literal>waiting for old snapshots</literal></entry>
+      <entry><literal>waiting for writers before build</literal></entry>
       <entry>
-       <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions
-       that can potentially see the table to release their snapshots.
+       <command>CREATE INDEX CONCURRENTLY</command> or <command>REINDEX CONCURRENTLY</command> is waiting for transactions
+       with write locks that can potentially see the table to finish.
        This phase is skipped when not in concurrent mode.
        Columns <structname>lockers_total</structname>, <structname>lockers_done</structname>
        and <structname>current_locker_pid</structname> contain the progress 
@@ -3632,10 +3638,10 @@ <title>CREATE INDEX phases</title>
       </entry>
      </row>
      <row>
-      <entry><literal>waiting for writer snapshots</literal></entry>
+      <entry><literal>waiting for writers before validation</literal></entry>
       <entry>
-       <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions
-       that can potentially write into the table to release their snapshots.
+       <command>CREATE INDEX CONCURRENTLY</command> or <command>REINDEX CONCURRENTLY</command> is waiting for transactions
+       with write locks that can potentially write into the table to finish.
        This phase is skipped when not in concurrent mode.
        Columns <structname>lockers_total</structname>, <structname>lockers_done</structname>
        and <structname>current_locker_pid</structname> contain the progress 
@@ -3670,9 +3676,9 @@ <title>CREATE INDEX phases</title>
       </entry>
      </row>
      <row>
-      <entry><literal>waiting for reader snapshots</literal></entry>
+      <entry><literal>waiting for old snapshots</literal></entry>
       <entry>
-       <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions
+       <command>CREATE INDEX CONCURRENTLY</command> or <command>REINDEX CONCURRENTLY</command> is waiting for transactions
        that can potentially see the table to release their snapshots.  This
        phase is skipped when not in concurrent mode.
        Columns <structname>lockers_total</structname>, <structname>lockers_done</structname>
@@ -3680,6 +3686,28 @@ <title>CREATE INDEX phases</title>
        information for this phase.
       </entry>
      </row>
+     <row>
+      <entry><literal>waiting for readers before marking dead</literal></entry>
+      <entry>
+       <command>REINDEX CONCURRENTLY</command> is waiting for transactions
+       with read locks on the table to finish, before marking the old index dead.
+       This phase is skipped when not in concurrent mode.
+       Columns <structname>lockers_total</structname>, <structname>lockers_done</structname>
+       and <structname>current_locker_pid</structname> contain the progress 
+       information for this phase.
+      </entry>
+     </row>
+     <row>
+      <entry><literal>waiting for readers before dropping</literal></entry>
+      <entry>
+       <command>REINDEX CONCURRENTLY</command> is waiting for transactions
+       with read locks on the table to finish, before dropping the old index.
+       This phase is skipped when not in concurrent mode.
+       Columns <structname>lockers_total</structname>, <structname>lockers_done</structname>
+       and <structname>current_locker_pid</structname> contain the progress 
+       information for this phase.
+      </entry>
+     </row>
     </tbody>
    </tgroup>
   </table>
@@ -3937,7 +3965,7 @@ <title><structname>pg_stat_progress_cluster</structname> View</title>
     </row>
     <row>
      <entry><structfield>cluster_index_relid</structfield></entry>
-     <entry><type>bigint</type></entry>
+     <entry><type>oid</type></entry>
      <entry>
        If the table is being scanned using an index, this is the OID of the
        index being used; otherwise, it is zero.
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 2ed7fdb021..9b1d546791 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -3286,12 +3286,20 @@ reindex_index(Oid indexId, bool skip_constraint_checks, char persistence,
 	heapId = IndexGetRelation(indexId, false);
 	heapRelation = table_open(heapId, ShareLock);
 
+	pgstat_progress_start_command(PROGRESS_COMMAND_CREATE_INDEX,
+								  heapId);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_INDEX_OID,
+								 indexId);
+
 	/*
 	 * Open the target index relation and get an exclusive lock on it, to
 	 * ensure that no one else is touching this particular index.
 	 */
 	iRel = index_open(indexId, AccessExclusiveLock);
 
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_ACCESS_METHOD_OID,
+								 iRel->rd_rel->relam);
+
 	/*
 	 * The case of reindexing partitioned tables and indexes is handled
 	 * differently by upper layers, so this case shouldn't arise.
@@ -3442,6 +3450,8 @@ reindex_index(Oid indexId, bool skip_constraint_checks, char persistence,
 				 errdetail_internal("%s",
 									pg_rusage_show(&ru0))));
 
+	pgstat_progress_end_command();
+
 	/* Close rels, but keep locks */
 	index_close(iRel, NoLock);
 	table_close(heapRelation, NoLock);
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 72f786d6f8..16e456a7d9 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -933,7 +933,7 @@ CREATE VIEW pg_stat_progress_cluster AS
                       WHEN 6 THEN 'rebuilding index'
                       WHEN 7 THEN 'performing final cleanup'
                       END AS phase,
-        S.param3 AS cluster_index_relid,
+        CAST(S.param3 AS oid) AS cluster_index_relid,
         S.param4 AS heap_tuples_scanned,
         S.param5 AS heap_tuples_written,
         S.param6 AS heap_blks_total,
@@ -946,16 +946,19 @@ CREATE VIEW pg_stat_progress_create_index AS
 	SELECT
 		S.pid AS pid, S.datid AS datid, D.datname AS datname,
 		S.relid AS relid,
+		CAST(S.param7 AS oid) AS index_relid,
 		CASE S.param10 WHEN 0 THEN 'initializing'
-					  WHEN 1 THEN 'waiting for old snapshots'
+					  WHEN 1 THEN 'waiting for writers before build'
 					  WHEN 2 THEN 'building index' ||
 						COALESCE((': ' || pg_indexam_progress_phasename(S.param9::oid, S.param11)),
 							'')
-					  WHEN 3 THEN 'waiting for writer snapshots'
+					  WHEN 3 THEN 'waiting for writers before validation'
 					  WHEN 4 THEN 'index validation: scanning index'
 					  WHEN 5 THEN 'index validation: sorting tuples'
 					  WHEN 6 THEN 'index validation: scanning table'
-					  WHEN 7 THEN 'waiting for reader snapshots'
+					  WHEN 7 THEN 'waiting for old snapshots'
+					  WHEN 8 THEN 'waiting for readers before marking dead'
+					  WHEN 9 THEN 'waiting for readers before dropping'
 					  END as phase,
 		S.param4 AS lockers_total,
 		S.param5 AS lockers_done,
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 348d543297..4a9b030c2c 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -478,6 +478,12 @@ DefineIndex(Oid relationId,
 		pgstat_progress_start_command(PROGRESS_COMMAND_CREATE_INDEX,
 									  relationId);
 
+	/*
+	 * No index OID to report yet
+	 */
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_INDEX_OID,
+								 InvalidOid);
+
 	/*
 	 * count key attributes in index
 	 */
@@ -1244,6 +1250,12 @@ DefineIndex(Oid relationId,
 	CommitTransactionCommand();
 	StartTransactionCommand();
 
+	/*
+	 * The index is now visible, so we can report the OID.
+	 */
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_INDEX_OID,
+								 indexRelationId);
+
 	/*
 	 * Phase 2 of concurrent index build (see comments for validate_index()
 	 * for an overview of how this works)
@@ -2873,6 +2885,13 @@ ReindexRelationConcurrently(Oid relationOid, int options)
 		heapRel = table_open(indexRel->rd_index->indrelid,
 							 ShareUpdateExclusiveLock);
 
+		pgstat_progress_start_command(PROGRESS_COMMAND_CREATE_INDEX,
+									  RelationGetRelid(heapRel));
+		pgstat_progress_update_param(PROGRESS_CREATEIDX_INDEX_OID,
+									 indexId);
+		pgstat_progress_update_param(PROGRESS_CREATEIDX_ACCESS_METHOD_OID,
+									 indexRel->rd_rel->relam);
+
 		/* Choose a temporary relation name for the new index */
 		concurrentName = ChooseRelationName(get_rel_name(indexId),
 											NULL,
@@ -2967,7 +2986,9 @@ ReindexRelationConcurrently(Oid relationOid, int options)
 	 * DefineIndex() for more details.
 	 */
 
-	WaitForLockersMultiple(lockTags, ShareLock, false);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_1);
+	WaitForLockersMultiple(lockTags, ShareLock, true);
 	CommitTransactionCommand();
 
 	forboth(lc, indexIds, lc2, newIndexIds)
@@ -3009,7 +3030,9 @@ ReindexRelationConcurrently(Oid relationOid, int options)
 	 * for more details.
 	 */
 
-	WaitForLockersMultiple(lockTags, ShareLock, false);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_2);
+	WaitForLockersMultiple(lockTags, ShareLock, true);
 	CommitTransactionCommand();
 
 	foreach(lc, newIndexIds)
@@ -3057,6 +3080,8 @@ ReindexRelationConcurrently(Oid relationOid, int options)
 		 * before the reference snap was taken, we have to wait out any
 		 * transactions that might have older snapshots.
 		 */
+		pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+									 PROGRESS_CREATEIDX_PHASE_WAIT_3);
 		WaitForOlderSnapshots(limitXmin, false);
 
 		CommitTransactionCommand();
@@ -3128,7 +3153,9 @@ ReindexRelationConcurrently(Oid relationOid, int options)
 	 * index_drop() for more details.
 	 */
 
-	WaitForLockersMultiple(lockTags, AccessExclusiveLock, false);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_4);
+	WaitForLockersMultiple(lockTags, AccessExclusiveLock, true);
 
 	foreach(lc, indexIds)
 	{
@@ -3150,7 +3177,9 @@ ReindexRelationConcurrently(Oid relationOid, int options)
 	 * Drop the old indexes.
 	 */
 
-	WaitForLockersMultiple(lockTags, AccessExclusiveLock, false);
+	pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
+								 PROGRESS_CREATEIDX_PHASE_WAIT_4);
+	WaitForLockersMultiple(lockTags, AccessExclusiveLock, true);
 
 	PushActiveSnapshot(GetTransactionSnapshot());
 
@@ -3225,6 +3254,8 @@ ReindexRelationConcurrently(Oid relationOid, int options)
 
 	MemoryContextDelete(private_context);
 
+	pgstat_progress_end_command();
+
 	return true;
 }
 
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index f046fa13b1..37043e926d 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -59,6 +59,7 @@
 
 /* Progress parameters for CREATE INDEX */
 /* 3, 4 and 5 reserved for "waitfor" metrics */
+#define PROGRESS_CREATEIDX_INDEX_OID			6
 #define PROGRESS_CREATEIDX_ACCESS_METHOD_OID	8
 #define PROGRESS_CREATEIDX_PHASE				9	/* AM-agnostic phase # */
 #define PROGRESS_CREATEIDX_SUBPHASE				10	/* phase # filled by AM */
@@ -76,6 +77,8 @@
 #define PROGRESS_CREATEIDX_PHASE_VALIDATE_SORT		5
 #define PROGRESS_CREATEIDX_PHASE_VALIDATE_TABLESCAN	6
 #define PROGRESS_CREATEIDX_PHASE_WAIT_3			7
+#define PROGRESS_CREATEIDX_PHASE_WAIT_4			8
+#define PROGRESS_CREATEIDX_PHASE_WAIT_5			9
 
 /*
  * Subphases of CREATE INDEX, for index_build.
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index bf7fca54ee..4638374a76 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1855,7 +1855,7 @@ pg_stat_progress_cluster| SELECT s.pid,
             WHEN 7 THEN 'performing final cleanup'::text
             ELSE NULL::text
         END AS phase,
-    s.param3 AS cluster_index_relid,
+    (s.param3)::oid AS cluster_index_relid,
     s.param4 AS heap_tuples_scanned,
     s.param5 AS heap_tuples_written,
     s.param6 AS heap_blks_total,
@@ -1867,15 +1867,18 @@ pg_stat_progress_create_index| SELECT s.pid,
     s.datid,
     d.datname,
     s.relid,
+    (s.param7)::oid AS index_relid,
         CASE s.param10
             WHEN 0 THEN 'initializing'::text
-            WHEN 1 THEN 'waiting for old snapshots'::text
+            WHEN 1 THEN 'waiting for writers before build'::text
             WHEN 2 THEN ('building index'::text || COALESCE((': '::text || pg_indexam_progress_phasename((s.param9)::oid, s.param11)), ''::text))
-            WHEN 3 THEN 'waiting for writer snapshots'::text
+            WHEN 3 THEN 'waiting for writers before validation'::text
             WHEN 4 THEN 'index validation: scanning index'::text
             WHEN 5 THEN 'index validation: sorting tuples'::text
             WHEN 6 THEN 'index validation: scanning table'::text
-            WHEN 7 THEN 'waiting for reader snapshots'::text
+            WHEN 7 THEN 'waiting for old snapshots'::text
+            WHEN 8 THEN 'waiting for readers before marking dead'::text
+            WHEN 9 THEN 'waiting for readers before dropping'::text
             ELSE NULL::text
         END AS phase,
     s.param4 AS lockers_total,
-- 
2.21.0

#59Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Peter Eisentraut (#58)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On 2019-Apr-05, Peter Eisentraut wrote:

I've reworded the phases a bit. There was a bit of a mixup of waiting
for snapshots and waiting for lockers. Perhaps not so important from a
user's perspective, but at least now it's more consistent with the
source code comments.

No disagreement with that. Looks reasonable.

I didn't test the patch, but it seems OK in a quick once-over.

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#60Peter Eisentraut
peter.eisentraut@2ndquadrant.com
In reply to: Alvaro Herrera (#59)
Re: monitoring CREATE INDEX [CONCURRENTLY]

On 2019-04-06 06:40, Alvaro Herrera wrote:

On 2019-Apr-05, Peter Eisentraut wrote:

I've reworded the phases a bit. There was a bit of a mixup of waiting
for snapshots and waiting for lockers. Perhaps not so important from a
user's perspective, but at least now it's more consistent with the
source code comments.

No disagreement with that. Looks reasonable.

I didn't test the patch, but it seems OK in a quick once-over.

committed

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services