pg_stat_transaction patch

Started by Joel Jacobsonover 15 years ago6 messages
#1Joel Jacobson
joel@gluefinance.com
1 attachment(s)

Hi,

I propose a set of new statistics functions and system views.

I need these functions in order to do automated testing of our system,
consisting of hundreds of stored procedures in plpgsql.
My plan is to develop some additional functions to pgTAP, benefiting from
the new system tables I've added.

The patch should apply to 9.0beta or HEAD, but I created it using 8.4.3
because that's the version I'm using.

I'm thankful for your feedback.

My apologies if the packaging of the patch does not conform to your
guidelines, feedback on this is also welcome.

--
Best regards,

Joel Jacobson
Glue Finance

E: jj@gluefinance.com
T: +46 70 360 38 01

Postal address:
Glue Finance AB
Box 549
114 11 Stockholm
Sweden

Visiting address:
Glue Finance AB
Birger Jarlsgatan 14
114 34 Stockholm
Sweden

README:

Background
==========
The views pg_stat_user_tables and pg_stat_user_functions shows statistics on
tables and functions.
The underlying functions named pg_stat_get_* fetches recent data from the
statistics collector, and returns the requested value for the given "oid"
(i.e. "tableid/relationid" or "functionid").
In the end of each transaction[1]upon COMMIT/ROLLBACK, or a bit later (the report frequency is controlled by the PGSTAT_STAT_INTERVAL setting, default value is 500 ms), the collected statistics are sent to the
statistics collector[2]if you do a ps aux, it is the process named "postgres: stats collector process".

[1]: upon COMMIT/ROLLBACK, or a bit later (the report frequency is controlled by the PGSTAT_STAT_INTERVAL setting, default value is 500 ms)
by the PGSTAT_STAT_INTERVAL setting, default value is 500 ms)
[2]: if you do a ps aux, it is the process named "postgres: stats collector process"
process"

Problem
=======
Within a current transaction, there was no way of accessing the internal
data structures which contains the so far collected statistics.
I wanted to check exactly what data changes my functions made and what
functions they called, without having to commit the transaction
and without mixing the statistics data with all the other simultaneously
running transactions.

Solution
========
I have exported get accessor methods to the internal data structure
containing so far collected statistics for the current transaction.

I have also exported the method pgstat_report_stat to make it possible to
force a "report and reset" of the so far collected statistics.
This was necessary to avoid not-yet-reported statistics for a previous
transaction to affect the current transaction.

I used the unused_oids script to find unused oids and choosed the range
between 3030-3044 for the new functions.

Functions
=========
test=# \df+ pg_catalog.pg_stat_get_transaction_*

List of functions
Schema | Name | Result data type
| Argument data types | Type | Volatility | Owner | Language |
Source code | Description

------------+--------------------------------------------+------------------+---------------------+--------+------------+-------+----------+--------------------------------------------+-------------------------------------------------------------------------
pg_catalog | pg_stat_get_transaction_blocks_fetched | bigint
| oid | normal | stable | joel | internal |
pg_stat_get_transaction_blocks_fetched | statistics: number of blocks
fetched in current transaction
pg_catalog | pg_stat_get_transaction_blocks_hit | bigint
| oid | normal | stable | joel | internal |
pg_stat_get_transaction_blocks_hit | statistics: number of blocks
found in cache in current transaction
pg_catalog | pg_stat_get_transaction_dead_tuples | bigint
| oid | normal | stable | joel | internal |
pg_stat_get_transaction_dead_tuples | statistics: number of dead
tuples in current transaction
pg_catalog | pg_stat_get_transaction_function_calls | bigint
| oid | normal | stable | joel | internal |
pg_stat_get_transaction_function_calls | statistics: number of function
calls in current transaction
pg_catalog | pg_stat_get_transaction_function_self_time | bigint
| oid | normal | stable | joel | internal |
pg_stat_get_transaction_function_self_time | statistics: self execution time
of function in current transaction
pg_catalog | pg_stat_get_transaction_function_time | bigint
| oid | normal | stable | joel | internal |
pg_stat_get_transaction_function_time | statistics: execution time of
function in current transaction
pg_catalog | pg_stat_get_transaction_live_tuples | bigint
| oid | normal | stable | joel | internal |
pg_stat_get_transaction_live_tuples | statistics: number of live
tuples in current transaction
pg_catalog | pg_stat_get_transaction_numscans | bigint
| oid | normal | stable | joel | internal |
pg_stat_get_transaction_numscans | statistics: number of scans
done for table/index in current transaction
pg_catalog | pg_stat_get_transaction_tuples_deleted | bigint
| oid | normal | stable | joel | internal |
pg_stat_get_transaction_tuples_deleted | statistics: number of tuples
deleted in current transaction
pg_catalog | pg_stat_get_transaction_tuples_fetched | bigint
| oid | normal | stable | joel | internal |
pg_stat_get_transaction_tuples_fetched | statistics: number of tuples
fetched by idxscan in current transaction
pg_catalog | pg_stat_get_transaction_tuples_hot_updated | bigint
| oid | normal | stable | joel | internal |
pg_stat_get_transaction_tuples_hot_updated | statistics: number of tuples
hot updated in current transaction
pg_catalog | pg_stat_get_transaction_tuples_inserted | bigint
| oid | normal | stable | joel | internal |
pg_stat_get_transaction_tuples_inserted | statistics: number of tuples
inserted in current transaction
pg_catalog | pg_stat_get_transaction_tuples_returned | bigint
| oid | normal | stable | joel | internal |
pg_stat_get_transaction_tuples_returned | statistics: number of tuples
read by seqscan in current transaction
pg_catalog | pg_stat_get_transaction_tuples_updated | bigint
| oid | normal | stable | joel | internal |
pg_stat_get_transaction_tuples_updated | statistics: number of tuples
updated in current transaction
(14 rows)

I also had to create a new internal function, "get_funcstat_entry".
This function find or create a PgStat_BackendFunctionEntry entry for the
given oid (functionid).
The name and behaviour is similar to the existing function
"get_tabstat_entry".

System views
============
pg_stat_transaction_tables - shows so far collected table statistics for the
current transaction (almost identical structure as pg_stat_user_tables, but
lacks the last_* columns)
pg_stat_transaction_functions - shows so far collected function statistics
for the current transaction (identical structure as pg_stat_user_functions)

Test/Use case
=============

Patched files
=============
/doc/src/sgml/monitoring.sgml
/src/backend/catalog/system_views.sql
/src/backend/postmaster/pgstat.c
/src/backend/utils/adt/pgstatfuncs.c
/src/include/catalog/pg_proc.h
/src/include/pgstat.h

Attachments:

pg_stat_transaction-1.3.tar.gzapplication/x-gzip; name=pg_stat_transaction-1.3.tar.gzDownload
#2Alvaro Herrera
alvherre@alvh.no-ip.org
In reply to: Joel Jacobson (#1)
Re: pg_stat_transaction patch

Excerpts from Joel Jacobson's message of jue may 06 09:51:41 -0400 2010:

Hi,

I propose a set of new statistics functions and system views.

Hi,

Please add your patch to the next commitfest: http://commitfest.postgresql.org

Thanks
--

#3Takahiro Itagaki
itagaki.takahiro@oss.ntt.co.jp
In reply to: Joel Jacobson (#1)
1 attachment(s)
Re: pg_stat_transaction patch

Joel Jacobson <joel@gluefinance.com> wrote:

I propose a set of new statistics functions and system views.

I need these functions in order to do automated testing of our system,
consisting of hundreds of stored procedures in plpgsql.
My plan is to develop some additional functions to pgTAP, benefiting from
the new system tables I've added.

I ported your patch into 9.0beta, but it doesn't work well.
I had two assertion failures from the run.sql:

TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File: "pgstat.c", Line: 715)
TRAP: FailedAssertion("!(tabstat->trans == trans)", File: "pgstat.c", Line: 1756)

Also, pg_stat_transaction_functions returned no rows from the test case even
after I removed those assertions. There are no rows in your test/run.out, too.

I like your idea itself, but more works are required for the implementation.

Regards,
---
Takahiro Itagaki
NTT Open Source Software Center

Attachments:

pg_stat_transaction-9.0beta.patchapplication/octet-stream; name=pg_stat_transaction-9.0beta.patchDownload
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index d197731..d234080 100644
*** a/doc/src/sgml/monitoring.sgml
--- b/doc/src/sgml/monitoring.sgml
*************** postgres: <replaceable>user</> <replacea
*** 648,653 ****
--- 648,745 ----
       </row>
  
       <row>
+       <entry><literal><function>pg_stat_get_transaction_numscans</function>(<type>oid</type>)</literal></entry>
+       <entry><type>bigint</type></entry>
+       <entry>
+        Number of sequential scans done when argument is a table,
+        or number of index scans done when argument is an index, in the current transaction
+       </entry>
+      </row>
+ 
+      <row>
+       <entry><literal><function>pg_stat_get_transaction_tuples_returned</function>(<type>oid</type>)</literal></entry>
+       <entry><type>bigint</type></entry>
+       <entry>
+        Number of rows read by sequential scans when argument is a table,
+        or number of index entries returned when argument is an index, in the current transaction
+       </entry>
+      </row>
+ 
+      <row>
+       <entry><literal><function>pg_stat_get_transaction_tuples_fetched</function>(<type>oid</type>)</literal></entry>
+       <entry><type>bigint</type></entry>
+       <entry>
+        Number of table rows fetched by bitmap scans when argument is a table,
+        or table rows fetched by simple index scans using the index
+        when argument is an index, in the current transaction
+       </entry>
+      </row>
+ 
+      <row>
+       <entry><literal><function>pg_stat_get_transaction_tuples_inserted</function>(<type>oid</type>)</literal></entry>
+       <entry><type>bigint</type></entry>
+       <entry>
+        Number of rows inserted into table, in the current transaction
+       </entry>
+      </row>
+ 
+      <row>
+       <entry><literal><function>pg_stat_get_transaction_tuples_updated</function>(<type>oid</type>)</literal></entry>
+       <entry><type>bigint</type></entry>
+       <entry>
+        Number of rows updated in table (includes HOT updates), in the current transaction
+       </entry>
+      </row>
+ 
+      <row>
+       <entry><literal><function>pg_stat_get_transaction_tuples_deleted</function>(<type>oid</type>)</literal></entry>
+       <entry><type>bigint</type></entry>
+       <entry>
+        Number of rows deleted from table, in the current transaction
+       </entry>
+      </row>
+ 
+      <row>
+       <entry><literal><function>pg_stat_get_transaction_tuples_hot_updated</function>(<type>oid</type>)</literal></entry>
+       <entry><type>bigint</type></entry>
+       <entry>
+        Number of rows HOT-updated in table, in the current transaction
+       </entry>
+      </row>
+ 
+      <row>
+       <entry><literal><function>pg_stat_get_transaction_live_tuples</function>(<type>oid</type>)</literal></entry>
+       <entry><type>bigint</type></entry>
+       <entry>
+        Number of live rows in table, in the current transaction
+       </entry>
+      </row>
+ 
+      <row>
+       <entry><literal><function>pg_stat_get_transaction_dead_tuples</function>(<type>oid</type>)</literal></entry>
+       <entry><type>bigint</type></entry>
+       <entry>
+        Number of dead rows in table, in the current transaction
+       </entry>
+      </row>
+ 
+      <row>
+       <entry><literal><function>pg_stat_get_transaction_blocks_fetched</function>(<type>oid</type>)</literal></entry>
+       <entry><type>bigint</type></entry>
+       <entry>
+        Number of disk block fetch requests for table or index, in the current transaction
+       </entry>
+      </row>
+ 
+      <row>
+       <entry><literal><function>pg_stat_get_transaction_blocks_hit</function>(<type>oid</type>)</literal></entry>
+       <entry><type>bigint</type></entry>
+       <entry>
+        Number of disk block requests found in cache for table or index, in the current transaction
+       </entry>
+      </row>
+ 
+      <row>
        <entry><literal><function>pg_stat_get_last_vacuum_time</function>(<type>oid</type>)</literal></entry>
        <entry><type>timestamptz</type></entry>
        <entry>
*************** postgres: <replaceable>user</> <replacea
*** 727,732 ****
--- 819,850 ----
       </row>
  
       <row>
+       <entry><literal><function>pg_stat_get_transaction_function_calls</function>(<type>oid</type>)</literal></entry>
+       <entry><type>bigint</type></entry>
+       <entry>
+        Number of times the function has been called, in the current transaction.
+       </entry>
+      </row>
+ 
+      <row>
+       <entry><literal><function>pg_stat_get_transaction_function_time</function>(<type>oid</type>)</literal></entry>
+       <entry><type>bigint</type></entry>
+       <entry>
+        Total wall clock time spent in the function, in microseconds.  Includes
+        the time spent in functions called by this one, in the current transaction.
+       </entry>
+      </row>
+ 
+      <row>
+       <entry><literal><function>pg_stat_get_transaction_function_self_time</function>(<type>oid</type>)</literal></entry>
+       <entry><type>bigint</type></entry>
+       <entry>
+        Time spent in only this function. Time spent in called functions
+        is excluded, in the current transaction.
+       </entry>
+      </row>
+ 
+      <row>
        <entry><literal><function>pg_stat_get_backend_idset</function>()</literal></entry>
        <entry><type>setof integer</type></entry>
        <entry>
*************** postgres: <replaceable>user</> <replacea
*** 950,955 ****
--- 1068,1085 ----
     </tgroup>
    </table>
  
+      <row>
+       <entry><literal><function>pg_stat_report_stat</function>()</literal></entry>
+       <entry><type>void</type></entry>
+       <entry>
+        Report and reset so far collected statistics for the current transaction
+        (requires superuser privileges)
+       </entry>
+      </row>
+     </tbody>
+    </tgroup>
+   </table>
+ 
     <note>
      <para>
       <function>pg_stat_get_blocks_fetched</function> minus
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 8852326..f5f5d42 100644
*** a/src/backend/catalog/system_views.sql
--- b/src/backend/catalog/system_views.sql
*************** CREATE VIEW pg_stat_all_tables AS 
*** 208,213 ****
--- 208,236 ----
      WHERE C.relkind IN ('r', 't')
      GROUP BY C.oid, N.nspname, C.relname;
  
+ CREATE OR REPLACE VIEW pg_stat_transaction_tables AS 
+     SELECT 
+             C.oid AS relid, 
+             N.nspname AS schemaname, 
+             C.relname AS relname, 
+             pg_stat_get_transaction_numscans(C.oid) AS seq_scan, 
+             pg_stat_get_transaction_tuples_returned(C.oid) AS seq_tup_read, 
+             sum(pg_stat_get_transaction_numscans(I.indexrelid))::bigint AS idx_scan, 
+             sum(pg_stat_get_transaction_tuples_fetched(I.indexrelid))::bigint +
+             pg_stat_get_transaction_tuples_fetched(C.oid) AS idx_tup_fetch, 
+             pg_stat_get_transaction_tuples_inserted(C.oid) AS n_tup_ins, 
+             pg_stat_get_transaction_tuples_updated(C.oid) AS n_tup_upd, 
+             pg_stat_get_transaction_tuples_deleted(C.oid) AS n_tup_del,
+             pg_stat_get_transaction_tuples_hot_updated(C.oid) AS n_tup_hot_upd,
+             pg_stat_get_transaction_live_tuples(C.oid) AS n_live_tup, 
+             pg_stat_get_transaction_dead_tuples(C.oid) AS n_dead_tup
+     FROM pg_class C LEFT JOIN
+          pg_index I ON C.oid = I.indrelid 
+          LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace) 
+     WHERE C.relkind IN ('r', 't') AND N.nspname NOT IN ('pg_catalog', 'information_schema') AND
+           N.nspname !~ '^pg_toast'
+     GROUP BY C.oid, N.nspname, C.relname;
+ 
  CREATE VIEW pg_stat_sys_tables AS 
      SELECT * FROM pg_stat_all_tables 
      WHERE schemaname IN ('pg_catalog', 'information_schema') OR
*************** CREATE VIEW pg_stat_user_functions AS 
*** 375,380 ****
--- 398,415 ----
      WHERE P.prolang != 12  -- fast check to eliminate built-in functions   
            AND pg_stat_get_function_calls(P.oid) IS NOT NULL;
  
+ CREATE VIEW pg_stat_transaction_functions AS 
+     SELECT
+             P.oid AS funcid, 
+             N.nspname AS schemaname,
+             P.proname AS funcname,
+             pg_stat_get_transaction_function_calls(P.oid) AS calls,
+             pg_stat_get_transaction_function_time(P.oid) / 1000 AS total_time,
+             pg_stat_get_transaction_function_self_time(P.oid) / 1000 AS self_time
+     FROM pg_proc P LEFT JOIN pg_namespace N ON (N.oid = P.pronamespace)
+     WHERE P.prolang != 12  -- fast check to eliminate built-in functions   
+           AND pg_stat_get_transaction_function_calls(P.oid) IS NOT NULL;
+ 
  CREATE VIEW pg_stat_bgwriter AS
      SELECT
          pg_stat_get_bgwriter_timed_checkpoints() AS checkpoints_timed,
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index edb5c80..daa4fbd 100644
*** a/src/backend/postmaster/pgstat.c
--- b/src/backend/postmaster/pgstat.c
*************** static void pgstat_send_tabstat(PgStat_M
*** 258,265 ****
  static void pgstat_send_funcstats(void);
  static HTAB *pgstat_collect_oids(Oid catalogid);
  
- static PgStat_TableStatus *get_tabstat_entry(Oid rel_id, bool isshared);
- 
  static void pgstat_setup_memcxt(void);
  
  static void pgstat_setheader(PgStat_MsgHdr *hdr, StatMsgType mtype);
--- 258,263 ----
*************** pgstat_initstats(Relation rel)
*** 1506,1512 ****
  /*
   * get_tabstat_entry - find or create a PgStat_TableStatus entry for rel
   */
! static PgStat_TableStatus *
  get_tabstat_entry(Oid rel_id, bool isshared)
  {
  	PgStat_TableStatus *entry;
--- 1504,1510 ----
  /*
   * get_tabstat_entry - find or create a PgStat_TableStatus entry for rel
   */
! PgStat_TableStatus *
  get_tabstat_entry(Oid rel_id, bool isshared)
  {
  	PgStat_TableStatus *entry;
*************** get_tabstat_entry(Oid rel_id, bool issha
*** 1561,1566 ****
--- 1559,1586 ----
  }
  
  /*
+  * get_funcstat_entry - find or create a PgStat_BackendFunctionEntry entry for rel
+  */
+ PgStat_BackendFunctionEntry *
+ get_funcstat_entry(Oid func_id)
+ {
+ 	PgStat_BackendFunctionEntry *entry;
+ 	bool found;
+ 
+ 	if (pgStatFunctions == NULL)
+ 		return NULL;
+ 
+ 	entry = (PgStat_BackendFunctionEntry *) hash_search(pgStatFunctions,
+ 												(void *) &func_id,
+ 												HASH_FIND, &found);
+ 	if (!found)
+ 		return NULL;
+ 
+ 	return entry;
+ }
+ 
+ 
+ /*
   * get_tabstat_stack_level - add a new (sub)transaction stack entry if needed
   */
  static PgStat_SubXactStatus *
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 8379407..010773d 100644
*** a/src/backend/utils/adt/pgstatfuncs.c
--- b/src/backend/utils/adt/pgstatfuncs.c
*************** extern Datum pg_stat_reset(PG_FUNCTION_A
*** 81,86 ****
--- 81,103 ----
  extern Datum pg_stat_reset_shared(PG_FUNCTION_ARGS);
  extern Datum pg_stat_reset_single_table_counters(PG_FUNCTION_ARGS);
  extern Datum pg_stat_reset_single_function_counters(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_report_stat(PG_FUNCTION_ARGS);
+ 
+ extern Datum pg_stat_get_transaction_numscans(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_transaction_tuples_returned(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_transaction_tuples_fetched(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_transaction_tuples_inserted(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_transaction_tuples_updated(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_transaction_tuples_deleted(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_transaction_tuples_hot_updated(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_transaction_live_tuples(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_transaction_dead_tuples(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_transaction_blocks_fetched(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_transaction_blocks_hit(PG_FUNCTION_ARGS);
+ 
+ extern Datum pg_stat_get_transaction_function_calls(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_transaction_function_time(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_transaction_function_self_time(PG_FUNCTION_ARGS);
  
  /* Global bgwriter statistics, from bgwriter.c */
  extern PgStat_MsgBgWriter bgwriterStats;
*************** pg_stat_reset_single_function_counters(P
*** 1143,1145 ****
--- 1160,1380 ----
  
  	PG_RETURN_VOID();
  }
+ 
+ 
+ /* Report so far collected per-table and function usage statistics to the collector */
+ Datum
+ pg_stat_report_stat(PG_FUNCTION_ARGS)
+ {
+ 	pgstat_report_stat(true);
+ 
+ 	PG_RETURN_VOID();
+ }
+ 
+ 
+ Datum
+ pg_stat_get_transaction_numscans(PG_FUNCTION_ARGS)
+ {
+ 	Oid			relid = PG_GETARG_OID(0);
+ 	int64		result;
+ 	PgStat_TableStatus *tabentry;
+ 
+ 	if ((tabentry = get_tabstat_entry(relid,false)) == NULL)
+ 		result = 0;
+ 	else
+ 		result = (int64) (tabentry->t_counts.t_numscans);
+ 
+ 	PG_RETURN_INT64(result);
+ }
+ 
+ Datum
+ pg_stat_get_transaction_tuples_returned(PG_FUNCTION_ARGS)
+ {
+ 	Oid			relid = PG_GETARG_OID(0);
+ 	int64		result;
+ 	PgStat_TableStatus *tabentry;
+ 
+ 	if ((tabentry = get_tabstat_entry(relid,false)) == NULL)
+ 		result = 0;
+ 	else
+ 		result = (int64) (tabentry->t_counts.t_tuples_returned);
+ 
+ 	PG_RETURN_INT64(result);
+ }
+ 
+ 
+ Datum
+ pg_stat_get_transaction_tuples_fetched(PG_FUNCTION_ARGS)
+ {
+ 	Oid			relid = PG_GETARG_OID(0);
+ 	int64		result;
+ 	PgStat_TableStatus *tabentry;
+ 
+ 	if ((tabentry = get_tabstat_entry(relid,false)) == NULL)
+ 		result = 0;
+ 	else
+ 		result = (int64) (tabentry->t_counts.t_tuples_fetched);
+ 
+ 	PG_RETURN_INT64(result);
+ }
+ 
+ 
+ Datum
+ pg_stat_get_transaction_tuples_inserted(PG_FUNCTION_ARGS)
+ {
+ 	Oid			relid = PG_GETARG_OID(0);
+ 	int64		result;
+ 	PgStat_TableStatus *tabentry;
+ 
+ 	if ((tabentry = get_tabstat_entry(relid,false)) == NULL)
+ 		result = 0;
+ 	else
+ 		result = (int64) (tabentry->t_counts.t_tuples_inserted);
+ 
+ 	PG_RETURN_INT64(result);
+ }
+ 
+ 
+ Datum
+ pg_stat_get_transaction_tuples_updated(PG_FUNCTION_ARGS)
+ {
+ 	Oid			relid = PG_GETARG_OID(0);
+ 	int64		result;
+ 	PgStat_TableStatus *tabentry;
+ 
+ 	if ((tabentry = get_tabstat_entry(relid,false)) == NULL)
+ 		result = 0;
+ 	else
+ 		result = (int64) (tabentry->t_counts.t_tuples_updated);
+ 
+ 	PG_RETURN_INT64(result);
+ }
+ 
+ 
+ Datum
+ pg_stat_get_transaction_tuples_deleted(PG_FUNCTION_ARGS)
+ {
+ 	Oid			relid = PG_GETARG_OID(0);
+ 	int64		result;
+ 	PgStat_TableStatus *tabentry;
+ 
+ 	if ((tabentry = get_tabstat_entry(relid,false)) == NULL)
+ 		result = 0;
+ 	else
+ 		result = (int64) (tabentry->t_counts.t_tuples_deleted);
+ 
+ 	PG_RETURN_INT64(result);
+ }
+ 
+ 
+ Datum
+ pg_stat_get_transaction_tuples_hot_updated(PG_FUNCTION_ARGS)
+ {
+ 	Oid			relid = PG_GETARG_OID(0);
+ 	int64		result;
+ 	PgStat_TableStatus *tabentry;
+ 
+ 	if ((tabentry = get_tabstat_entry(relid,false)) == NULL)
+ 		result = 0;
+ 	else
+ 		result = (int64) (tabentry->t_counts.t_tuples_hot_updated);
+ 
+ 	PG_RETURN_INT64(result);
+ }
+ 
+ 
+ Datum
+ pg_stat_get_transaction_live_tuples(PG_FUNCTION_ARGS)
+ {
+ 	Oid			relid = PG_GETARG_OID(0);
+ 	int64		result;
+ 	PgStat_TableStatus *tabentry;
+ 
+ 	if ((tabentry = get_tabstat_entry(relid,false)) == NULL)
+ 		result = 0;
+ 	else
+ 		result = (int64) (tabentry->t_counts.t_delta_live_tuples);
+ 
+ 	PG_RETURN_INT64(result);
+ }
+ 
+ 
+ Datum
+ pg_stat_get_transaction_dead_tuples(PG_FUNCTION_ARGS)
+ {
+ 	Oid			relid = PG_GETARG_OID(0);
+ 	int64		result;
+ 	PgStat_TableStatus *tabentry;
+ 
+ 	if ((tabentry = get_tabstat_entry(relid,false)) == NULL)
+ 		result = 0;
+ 	else
+ 		result = (int64) (tabentry->t_counts.t_delta_dead_tuples);
+ 
+ 	PG_RETURN_INT64(result);
+ }
+ 
+ 
+ Datum
+ pg_stat_get_transaction_blocks_fetched(PG_FUNCTION_ARGS)
+ {
+ 	Oid			relid = PG_GETARG_OID(0);
+ 	int64		result;
+ 	PgStat_TableStatus *tabentry;
+ 
+ 	if ((tabentry = get_tabstat_entry(relid,false)) == NULL)
+ 		result = 0;
+ 	else
+ 		result = (int64) (tabentry->t_counts.t_blocks_fetched);
+ 
+ 	PG_RETURN_INT64(result);
+ }
+ 
+ 
+ Datum
+ pg_stat_get_transaction_blocks_hit(PG_FUNCTION_ARGS)
+ {
+ 	Oid			relid = PG_GETARG_OID(0);
+ 	int64		result;
+ 	PgStat_TableStatus *tabentry;
+ 
+ 	if ((tabentry = get_tabstat_entry(relid,false)) == NULL)
+ 		result = 0;
+ 	else
+ 		result = (int64) (tabentry->t_counts.t_blocks_hit);
+ 
+ 	PG_RETURN_INT64(result);
+ }
+ 
+ Datum
+ pg_stat_get_transaction_function_calls(PG_FUNCTION_ARGS)
+ {
+ 	Oid			funcid = PG_GETARG_OID(0);
+ 	PgStat_BackendFunctionEntry *funcentry;
+ 
+ 	if ((funcentry = get_funcstat_entry(funcid)) == NULL)
+ 		PG_RETURN_NULL();
+ 	PG_RETURN_INT64(funcentry->f_counts.f_numcalls);
+ }
+ 
+ Datum
+ pg_stat_get_transaction_function_time(PG_FUNCTION_ARGS)
+ {
+ 	Oid			funcid = PG_GETARG_OID(0);
+ 	PgStat_BackendFunctionEntry *funcentry;
+ 
+ 	if ((funcentry = get_funcstat_entry(funcid)) == NULL)
+ 		PG_RETURN_NULL();
+ 	PG_RETURN_INT64(INSTR_TIME_GET_MICROSEC(funcentry->f_counts.f_time));
+ }
+ 
+ Datum
+ pg_stat_get_transaction_function_self_time(PG_FUNCTION_ARGS)
+ {
+ 	Oid			funcid = PG_GETARG_OID(0);
+ 	PgStat_BackendFunctionEntry *funcentry;
+ 
+ 	if ((funcentry = get_funcstat_entry(funcid)) == NULL)
+ 		PG_RETURN_NULL();
+ 	PG_RETURN_INT64(INSTR_TIME_GET_MICROSEC(funcentry->f_counts.f_time_self));
+ }
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index f2751a4..5ed8470 100644
*** a/src/include/catalog/pg_proc.h
--- b/src/include/catalog/pg_proc.h
*************** DESCR("statistics: execution time of fun
*** 3091,3096 ****
--- 3091,3125 ----
  DATA(insert OID = 2980 (  pg_stat_get_function_self_time	PGNSP PGUID 12 1 0 0 f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_function_self_time _null_ _null_ _null_ ));
  DESCR("statistics: self execution time of function");
  
+ DATA(insert OID = 3045 (  pg_stat_get_transaction_numscans			PGNSP PGUID 12 1 0 0 f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_transaction_numscans _null_ _null_ _null_ ));
+ DESCR("statistics: number of scans done for table/index in current transaction");
+ DATA(insert OID = 3046 (  pg_stat_get_transaction_tuples_returned	PGNSP PGUID 12 1 0 0 f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_transaction_tuples_returned _null_ _null_ _null_ ));
+ DESCR("statistics: number of tuples read by seqscan in current transaction");
+ DATA(insert OID = 3047 (  pg_stat_get_transaction_tuples_fetched	PGNSP PGUID 12 1 0 0 f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_transaction_tuples_fetched _null_ _null_ _null_ ));
+ DESCR("statistics: number of tuples fetched by idxscan in current transaction");
+ DATA(insert OID = 3048 (  pg_stat_get_transaction_tuples_inserted	PGNSP PGUID 12 1 0 0 f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_transaction_tuples_inserted _null_ _null_ _null_ ));
+ DESCR("statistics: number of tuples inserted in current transaction");
+ DATA(insert OID = 3049 (  pg_stat_get_transaction_tuples_updated	PGNSP PGUID 12 1 0 0 f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_transaction_tuples_updated _null_ _null_ _null_ ));
+ DESCR("statistics: number of tuples updated in current transaction");
+ DATA(insert OID = 3050 (  pg_stat_get_transaction_tuples_deleted	PGNSP PGUID 12 1 0 0 f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_transaction_tuples_deleted _null_ _null_ _null_ ));
+ DESCR("statistics: number of tuples deleted in current transaction");
+ DATA(insert OID = 3051 (  pg_stat_get_transaction_tuples_hot_updated PGNSP PGUID 12 1 0 0 f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_transaction_tuples_hot_updated _null_ _null_ _null_ ));
+ DESCR("statistics: number of tuples hot updated in current transaction");
+ DATA(insert OID = 3052 (  pg_stat_get_transaction_live_tuples	PGNSP PGUID 12 1 0 0 f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_transaction_live_tuples _null_ _null_ _null_ ));
+ DESCR("statistics: number of live tuples in current transaction");
+ DATA(insert OID = 3053 (  pg_stat_get_transaction_dead_tuples	PGNSP PGUID 12 1 0 0 f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_transaction_dead_tuples _null_ _null_ _null_ ));
+ DESCR("statistics: number of dead tuples in current transaction");
+ DATA(insert OID = 3054 (  pg_stat_get_transaction_blocks_fetched	PGNSP PGUID 12 1 0 0 f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_transaction_blocks_fetched _null_ _null_ _null_ ));
+ DESCR("statistics: number of blocks fetched in current transaction");
+ DATA(insert OID = 3055 (  pg_stat_get_transaction_blocks_hit		PGNSP PGUID 12 1 0 0 f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_transaction_blocks_hit _null_ _null_ _null_ ));
+ DESCR("statistics: number of blocks found in cache in current transaction");
+ DATA(insert OID = 3056 (  pg_stat_get_transaction_function_calls		PGNSP PGUID 12 1 0 0 f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_transaction_function_calls _null_ _null_ _null_ ));
+ DESCR("statistics: number of function calls in current transaction");
+ DATA(insert OID = 3057 (  pg_stat_get_transaction_function_time			PGNSP PGUID 12 1 0 0 f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_transaction_function_time _null_ _null_ _null_ ));
+ DESCR("statistics: execution time of function in current transaction");
+ DATA(insert OID = 3058 (  pg_stat_get_transaction_function_self_time	PGNSP PGUID 12 1 0 0 f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_transaction_function_self_time _null_ _null_ _null_ ));
+ DESCR("statistics: self execution time of function in current transaction");
+ 
  DATA(insert OID = 2230 (  pg_stat_clear_snapshot		PGNSP PGUID 12 1 0 0 f f f f f v 0 0 2278 "" _null_ _null_ _null_ _null_	pg_stat_clear_snapshot _null_ _null_ _null_ ));
  DESCR("statistics: discard current transaction's statistics snapshot");
  DATA(insert OID = 2274 (  pg_stat_reset					PGNSP PGUID 12 1 0 0 f f f f f v 0 0 2278 "" _null_ _null_ _null_ _null_	pg_stat_reset _null_ _null_ _null_ ));
*************** DATA(insert OID = 3776 (  pg_stat_reset_
*** 3101,3106 ****
--- 3130,3137 ----
  DESCR("statistics: reset collected statistics for a single table or index in the current database");
  DATA(insert OID = 3777 (  pg_stat_reset_single_function_counters	PGNSP PGUID 12 1 0 0 f f f f f v 1 0 2278 "26" _null_ _null_ _null_ _null_	pg_stat_reset_single_function_counters _null_ _null_ _null_ ));
  DESCR("statistics: reset collected statistics for a single function in the current database");
+ DATA(insert OID = 3059 (  pg_stat_report_stat					PGNSP PGUID 12 1 0 0 f f f f f v 0 0 2278 "" _null_ _null_ _null_ _null_	pg_stat_report_stat _null_ _null_ _null_ ));
+ DESCR("statistics: report and reset so far collected statistics for current transaction");
  
  DATA(insert OID = 1946 (  encode						PGNSP PGUID 12 1 0 0 f f f t f i 2 0 25 "17 25" _null_ _null_ _null_ _null_ binary_encode _null_ _null_ _null_ ));
  DESCR("convert bytea value into some ascii-only text string");
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 3dd5f45..5b2f51e 100644
*** a/src/include/pgstat.h
--- b/src/include/pgstat.h
*************** extern const char *pgstat_get_backend_cu
*** 694,699 ****
--- 694,702 ----
  
  extern void pgstat_initstats(Relation rel);
  
+ extern PgStat_TableStatus *get_tabstat_entry(Oid rel_id, bool isshared);
+ extern PgStat_BackendFunctionEntry *get_funcstat_entry(Oid func_id);
+ 
  /* nontransactional event counts are simple enough to inline */
  
  #define pgstat_count_heap_scan(rel)									\
#4Joel Jacobson
joel@gluefinance.com
In reply to: Takahiro Itagaki (#3)
1 attachment(s)
Re: pg_stat_transaction patch

Hajimemashite Takahiro,

Thanks for your feedback.

I applied all the changes on 9.0beta manually and then it compiled without
any assertion failures.

I also changed the oids to a different unused range, since the ones I used
before had been taken in 9.0beta1.

There are still some problems though. I get 0 back from the functions
supposed to return the number of inserts/updates for the current
transaction.

I suspect it is because get_tabstat_entry for some reason returns NULL, in
for example pg_stat_get_transaction_tuples_inserted(PG_FUNCTION_ARGS).

Does the function look valid? If you can find the error in it, the other
functions probably have the same problem.

It is strange though the function "pg_stat_get_transaction_numscans" works
fine, and it looks like it works the same way.

I added run.out843 and run.out90b1, showing the output from both patched
versions.

run.out843 is the intended output, while run.out90b1 gives 0 on the columns
n_tup_ins and n_tup_upd (and probably n_tup_del etc also).

I hope someone can help locating the problem.

Thanks.

Best regards,

Joel

2010/5/7 Takahiro Itagaki <itagaki.takahiro@oss.ntt.co.jp>

Joel Jacobson <joel@gluefinance.com> wrote:

I propose a set of new statistics functions and system views.

I need these functions in order to do automated testing of our system,
consisting of hundreds of stored procedures in plpgsql.
My plan is to develop some additional functions to pgTAP, benefiting from
the new system tables I've added.

I ported your patch into 9.0beta, but it doesn't work well.
I had two assertion failures from the run.sql:

TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File: "pgstat.c",
Line: 715)
TRAP: FailedAssertion("!(tabstat->trans == trans)", File: "pgstat.c", Line:
1756)

Also, pg_stat_transaction_functions returned no rows from the test case
even
after I removed those assertions. There are no rows in your test/run.out,
too.

I like your idea itself, but more works are required for the
implementation.

Regards,
---
Takahiro Itagaki
NTT Open Source Software Center

--
Best regards,

Joel Jacobson
Glue Finance

E: jj@gluefinance.com
T: +46 70 360 38 01

Postal address:
Glue Finance AB
Box 549
114 11 Stockholm
Sweden

Visiting address:
Glue Finance AB
Birger Jarlsgatan 14
114 34 Stockholm
Sweden

Attachments:

pg_stat_transaction-1.31.tar.gzapplication/x-gzip; name=pg_stat_transaction-1.31.tar.gzDownload
#5Takahiro Itagaki
itagaki.takahiro@oss.ntt.co.jp
In reply to: Joel Jacobson (#4)
Re: pg_stat_transaction patch

Joel Jacobson <joel@gluefinance.com> wrote:

I applied all the changes on 9.0beta manually and then it compiled without
any assertion failures.

I also changed the oids to a different unused range, since the ones I used
before had been taken in 9.0beta1.

Thanks, but you still need to test your patch:

- You need to check your patch with "make check", because it requires
adjustments in "rule" test; Your pg_stat_transaction_function is the
longest name in the system catalog.

- You need to configure postgres with --enable-cassert to enable internal
varidations. The attached test case failed with the following TRAP.
TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File: "pgstat.c", Line: 715)
TRAP: FailedAssertion("!(tabstat->trans == trans)", File: "pgstat.c", Line: 1758)

I suspect it is because get_tabstat_entry for some reason returns NULL, in
for example pg_stat_get_transaction_tuples_inserted(PG_FUNCTION_ARGS).

Does the function look valid? If you can find the error in it, the other
functions probably have the same problem.

For the above trap, we can see the comment:
/* Shouldn't have any pending transaction-dependent counts */
We don't expect to read stats entries during transactions. I'm not sure
whether accessing transitional stats during transaction is safe or not.

We might need to go other directions, for example:
- Use "session stats" instead "transaction stats". You can see the same
information in difference of counters between before and after the
transaction.
- Export pgBufferUsage instead of relation counters. They are
buffer counters for all relations, but we can obviously export
them because they are just plain variables.

Regards,
---
Takahiro Itagaki
NTT Open Source Software Center

#6Joel Jacobson
joel@gluefinance.com
In reply to: Takahiro Itagaki (#5)
1 attachment(s)
Re: pg_stat_transaction patch

Hi Takahiro,

Here is an updated version of the patch.

Thanks Magnus H for the help :)

1.4: Ported to head. Updated tests. Removed pg_stat_report.

2010/5/25 Takahiro Itagaki <itagaki.takahiro@oss.ntt.co.jp>

Joel Jacobson <joel@gluefinance.com> wrote:

I applied all the changes on 9.0beta manually and then it compiled

without

any assertion failures.

I also changed the oids to a different unused range, since the ones I

used

before had been taken in 9.0beta1.

Thanks, but you still need to test your patch:

- You need to check your patch with "make check", because it requires
adjustments in "rule" test; Your pg_stat_transaction_function is the
longest name in the system catalog.

- You need to configure postgres with --enable-cassert to enable internal
varidations. The attached test case failed with the following TRAP.
TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File: "pgstat.c",
Line: 715)
TRAP: FailedAssertion("!(tabstat->trans == trans)", File: "pgstat.c", Line:
1758)

I suspect it is because get_tabstat_entry for some reason returns NULL,

in

for example pg_stat_get_transaction_tuples_inserted(PG_FUNCTION_ARGS).

Does the function look valid? If you can find the error in it, the other
functions probably have the same problem.

For the above trap, we can see the comment:
/* Shouldn't have any pending transaction-dependent counts */
We don't expect to read stats entries during transactions. I'm not sure
whether accessing transitional stats during transaction is safe or not.

We might need to go other directions, for example:
- Use "session stats" instead "transaction stats". You can see the same
information in difference of counters between before and after the
transaction.
- Export pgBufferUsage instead of relation counters. They are
buffer counters for all relations, but we can obviously export
them because they are just plain variables.

Regards,
---
Takahiro Itagaki
NTT Open Source Software Center

--
Best regards,

Joel Jacobson
Glue Finance

E: jj@gluefinance.com
T: +46 70 360 38 01

Postal address:
Glue Finance AB
Box 549
114 11 Stockholm
Sweden

Visiting address:
Glue Finance AB
Birger Jarlsgatan 14
114 34 Stockholm
Sweden

Attachments:

pg_stat_transaction-1.4.tar.gzapplication/x-gzip; name=pg_stat_transaction-1.4.tar.gzDownload