stats for network traffic WIP

Started by Nigel Heronabout 12 years ago38 messages
#1Nigel Heron
nheron@querymetrics.com
2 attachment(s)

Hi, I've been using postgres for many years but never took the time to play
with the code until now. As a learning experience i came up with this WIP
patch to keep track of the # of bytes sent and received by the server over
it's communication sockets. Counters are kept per database, per connection
and globally/shared.
The counters are incremented for tcp (remote and localhost) and for unix
sockets. The major WIP issue so far is that connections using SSL aren't
counted properly. If there's any interest, i'll keep working on it.

a few functions are added:
- pg_stat_get_bytes_sent() returns the total count of outgoing bytes for
the whole cluster (all dbs and all connections including replication)
- pg_stat_get_bytes_received() same but for incoming data
- pg_stat_get_db_bytes_sent(oid) returns count of outgoing bytes for a
specific database
- pg_stat_get_db_bytes_received(oid) same but for incoming data

"bytes_sent" and "bytes_received" columns are added to:
- pg_stat_get_activity function
- pg_stat_activity view
- pg_stat_database view
- pg_stat_replication view

The counters are reset with the existing reset functions, but a new
parameter value is added for the shared stats call (i named it "socket" for
lack of imagination), eg. pg_stat_reset_shared('socket').

some benefits of the patch:
- can be used to track bandwidth usage of postgres, useful if the host
isn't a dedicated db server, where host level statistics would include
other traffic.
- can track bandwidth usage of streaming replication.
- can be used to find misbehaving connections.
- can be used in multi-user/multi-database clusters for resource usage
tracking.
- competing databases have such metrics.
- could also be added to pg_stat_statements for extra debugging.
- etc.?

some negatives:
- extra code is called for each send() and recv(), I haven't measured the
performance impact yet. (but can be turned off using track_counts=off)
- stats collector has more work to do.
- some stats structs are changed which will cause an error while trying to
load them from disk the first time and the old stats will be lost.
- PL functions that create their own sockets aren't tracked.
- sockets from FDWs calls aren't tracked.

To debug the counters, i'm using clients connected through haproxy to
generate traffic and then compare haproxy's stats with what pg stores in
pg_stat/global.stat on shutdown. Attached is a very basic python script
that can read the global.stat file (it takes the DATADIR as a parameter).

Any feedback is appreciated,
-nigel.

Attachments:

netstats-WIPv1.patchapplication/octet-stream; name=netstats-WIPv1.patchDownload
*** a/src/backend/catalog/system_views.sql
--- b/src/backend/catalog/system_views.sql
***************
*** 586,592 **** CREATE VIEW pg_stat_activity AS
              S.state_change,
              S.waiting,
              S.state,
!             S.query
      FROM pg_database D, pg_stat_get_activity(NULL) AS S, pg_authid U
      WHERE S.datid = D.oid AND
              S.usesysid = U.oid;
--- 586,594 ----
              S.state_change,
              S.waiting,
              S.state,
!             S.query,
!             S.bytes_sent,
!             S.bytes_received
      FROM pg_database D, pg_stat_get_activity(NULL) AS S, pg_authid U
      WHERE S.datid = D.oid AND
              S.usesysid = U.oid;
***************
*** 601,606 **** CREATE VIEW pg_stat_replication AS
--- 603,610 ----
              S.client_hostname,
              S.client_port,
              S.backend_start,
+             S.bytes_sent,
+             S.bytes_received,
              W.state,
              W.sent_location,
              W.write_location,
***************
*** 634,639 **** CREATE VIEW pg_stat_database AS
--- 638,645 ----
              pg_stat_get_db_deadlocks(D.oid) AS deadlocks,
              pg_stat_get_db_blk_read_time(D.oid) AS blk_read_time,
              pg_stat_get_db_blk_write_time(D.oid) AS blk_write_time,
+             pg_stat_get_db_bytes_sent(D.oid) AS bytes_sent,
+             pg_stat_get_db_bytes_received(D.oid) AS bytes_received,
              pg_stat_get_db_stat_reset_time(D.oid) AS stats_reset
      FROM pg_database D;
  
*** a/src/backend/libpq/pqcomm.c
--- b/src/backend/libpq/pqcomm.c
***************
*** 879,884 **** pq_recvbuf(void)
--- 879,886 ----
  		}
  		/* r contains number of bytes read, so just incr length */
  		PqRecvLength += r;
+ 		/* we received data from the socket that needs to be reported */
+ 		pgstat_report_commreceived(r);
  		return 0;
  	}
  }
***************
*** 965,970 **** pq_getbyte_if_available(unsigned char *c)
--- 967,977 ----
  		/* EOF detected */
  		r = EOF;
  	}
+ 	else
+ 	{
+ 		/* we received data from the socket that needs to be reported */
+ 		pgstat_report_commreceived(r);
+ 	}
  
  	return r;
  }
***************
*** 1294,1299 **** internal_flush(void)
--- 1301,1311 ----
  			InterruptPending = 1;
  			return EOF;
  		}
+ 		else
+ 		{
+ 			/* we sent data over the socket that needs to be reported */
+ 			pgstat_report_commsent(r);
+ 		}
  
  		last_reported_send_errno = 0;	/* reset after any successful send */
  		bufptr += r;
*** a/src/backend/postmaster/pgstat.c
--- b/src/backend/postmaster/pgstat.c
***************
*** 298,303 **** static void pgstat_recv_funcpurge(PgStat_MsgFuncpurge *msg, int len);
--- 298,305 ----
  static void pgstat_recv_recoveryconflict(PgStat_MsgRecoveryConflict *msg, int len);
  static void pgstat_recv_deadlock(PgStat_MsgDeadlock *msg, int len);
  static void pgstat_recv_tempfile(PgStat_MsgTempFile *msg, int len);
+ static void pgstat_recv_commsent(PgStat_MsgComm *msg, int len);
+ static void pgstat_recv_commreceived(PgStat_MsgComm *msg, int len);
  
  /* ------------------------------------------------------------
   * Public functions called from postmaster follow
***************
*** 1249,1259 **** pgstat_reset_shared_counters(const char *target)
  
  	if (strcmp(target, "bgwriter") == 0)
  		msg.m_resettarget = RESET_BGWRITER;
  	else
  		ereport(ERROR,
  				(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
  				 errmsg("unrecognized reset target: \"%s\"", target),
! 				 errhint("Target must be \"bgwriter\".")));
  
  	pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETSHAREDCOUNTER);
  	pgstat_send(&msg, sizeof(msg));
--- 1251,1263 ----
  
  	if (strcmp(target, "bgwriter") == 0)
  		msg.m_resettarget = RESET_BGWRITER;
+ 	else if (strcmp(target, "socket") == 0)
+ 		msg.m_resettarget = RESET_SOCKET;
  	else
  		ereport(ERROR,
  				(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
  				 errmsg("unrecognized reset target: \"%s\"", target),
! 				 errhint("Target must be \"bgwriter\" or \"socket\".")));
  
  	pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETSHAREDCOUNTER);
  	pgstat_send(&msg, sizeof(msg));
***************
*** 2531,2536 **** pgstat_bestart(void)
--- 2535,2542 ----
  	beentry->st_clienthostname[NAMEDATALEN - 1] = '\0';
  	beentry->st_appname[NAMEDATALEN - 1] = '\0';
  	beentry->st_activity[pgstat_track_activity_query_size - 1] = '\0';
+ 	beentry->st_bytes_sent = 0;
+ 	beentry->st_bytes_received = 0;
  
  	beentry->st_changecount++;
  	Assert((beentry->st_changecount & 1) == 0);
***************
*** 2738,2743 **** pgstat_report_waiting(bool waiting)
--- 2744,2809 ----
  	beentry->st_waiting = waiting;
  }
  
+ /* --------
+  * pgstat_report_commsent() -
+  *
+  *    Tell the collector about data sent over a socket.
+  * --------
+  */
+ void
+ pgstat_report_commsent(int len)
+ {
+ 	volatile PgBackendStatus *beentry = MyBEEntry;
+ 	PgStat_MsgComm msg;
+ 
+ 	if (pgStatSock == PGINVALID_SOCKET || !pgstat_track_counts)
+ 		return;
+ 
+ 	/* can be called before we have connected to a specific database or during replication */
+ 	if (beentry != NULL) {
+ 		beentry->st_changecount++;
+ 		beentry->st_bytes_sent += len;
+ 		beentry->st_changecount++;
+ 		Assert((beentry->st_changecount & 1) == 0);
+ 	}
+ 
+ 	pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_COMMSENT);
+ 	/* MyDatabaseId might be invalid here, we check it in the msg receiver */
+ 	msg.m_databaseid = MyDatabaseId;
+ 	msg.m_bytes_transferred = len;
+ 	pgstat_send(&msg, sizeof(msg));
+ }
+ 
+ /* --------
+  * pgstat_report_commreceived() -
+  *
+  *    Tell the collector about data received from a socket.
+  * --------
+  */
+ void
+ pgstat_report_commreceived(int len)
+ {
+ 	volatile PgBackendStatus *beentry = MyBEEntry;
+ 	PgStat_MsgComm msg;
+ 
+ 	if (pgStatSock == PGINVALID_SOCKET || !pgstat_track_counts)
+ 		return;
+ 
+ 	/* can be called before we have connected to a specific database or during replication */
+ 	if (beentry != NULL) {
+ 		beentry->st_changecount++;
+ 		beentry->st_bytes_received += len;
+ 		beentry->st_changecount++;
+ 		Assert((beentry->st_changecount & 1) == 0);
+ 	}
+ 
+ 	pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_COMMRECEIVED);
+ 	/* MyDatabaseId might be invalid here, we check it in the msg receiver */
+ 	msg.m_databaseid = MyDatabaseId;
+ 	msg.m_bytes_transferred = len;
+ 	pgstat_send(&msg, sizeof(msg));
+ }
+ 
  
  /* ----------
   * pgstat_read_current_status() -
***************
*** 3290,3295 **** PgstatCollectorMain(int argc, char *argv[])
--- 3356,3369 ----
  					pgstat_recv_tempfile((PgStat_MsgTempFile *) &msg, len);
  					break;
  
+ 				case PGSTAT_MTYPE_COMMSENT:
+ 					pgstat_recv_commsent((PgStat_MsgComm *) &msg, len);
+ 					break;
+ 
+ 				case PGSTAT_MTYPE_COMMRECEIVED:
+ 					pgstat_recv_commreceived((PgStat_MsgComm *) &msg, len);
+ 					break;
+ 
  				default:
  					break;
  			}
***************
*** 3390,3395 **** reset_dbentry_counters(PgStat_StatDBEntry *dbentry)
--- 3464,3471 ----
  	dbentry->n_deadlocks = 0;
  	dbentry->n_block_read_time = 0;
  	dbentry->n_block_write_time = 0;
+ 	dbentry->n_bytes_sent = 0;
+ 	dbentry->n_bytes_received = 0;
  
  	dbentry->stat_reset_timestamp = GetCurrentTimestamp();
  	dbentry->stats_timestamp = 0;
***************
*** 3798,3803 **** pgstat_read_statsfiles(Oid onlydb, bool permanent, bool deep)
--- 3874,3880 ----
  	int32		format_id;
  	bool		found;
  	const char *statfile = permanent ? PGSTAT_STAT_PERMANENT_FILENAME : pgstat_stat_filename;
+ 	TimestampTz now;
  
  	/*
  	 * The tables will live in pgStatLocalContext.
***************
*** 3825,3831 **** pgstat_read_statsfiles(Oid onlydb, bool permanent, bool deep)
  	 * Set the current timestamp (will be kept only in case we can't load an
  	 * existing statsfile).
  	 */
! 	globalStats.stat_reset_timestamp = GetCurrentTimestamp();
  
  	/*
  	 * Try to open the stats file. If it doesn't exist, the backends simply
--- 3902,3910 ----
  	 * Set the current timestamp (will be kept only in case we can't load an
  	 * existing statsfile).
  	 */
! 	now = GetCurrentTimestamp();
! 	globalStats.bgwriter_stat_reset_timestamp = now;
! 	globalStats.socket_stat_reset_timestamp = now;
  
  	/*
  	 * Try to open the stats file. If it doesn't exist, the backends simply
***************
*** 4723,4730 **** pgstat_recv_resetsharedcounter(PgStat_MsgResetsharedcounter *msg, int len)
  	if (msg->m_resettarget == RESET_BGWRITER)
  	{
  		/* Reset the global background writer statistics for the cluster. */
! 		memset(&globalStats, 0, sizeof(globalStats));
! 		globalStats.stat_reset_timestamp = GetCurrentTimestamp();
  	}
  
  	/*
--- 4802,4827 ----
  	if (msg->m_resettarget == RESET_BGWRITER)
  	{
  		/* Reset the global background writer statistics for the cluster. */
! 		globalStats.stats_timestamp = 0;
! 		globalStats.timed_checkpoints = 0;
! 		globalStats.requested_checkpoints = 0;
! 		globalStats.checkpoint_write_time = 0;
! 		globalStats.checkpoint_sync_time = 0;
! 		globalStats.buf_written_checkpoints = 0;
! 		globalStats.buf_written_clean = 0;
! 		globalStats.maxwritten_clean = 0;
! 		globalStats.buf_written_backend = 0;
! 		globalStats.buf_fsync_backend = 0;
! 		globalStats.buf_alloc = 0;
! 		globalStats.bgwriter_stat_reset_timestamp = GetCurrentTimestamp();
! 	}
! 	else if (msg->m_resettarget == RESET_SOCKET)
! 	{
! 		/* Reset the global socket transfer statistics for the cluster. */
! 		globalStats.stats_timestamp = 0;
! 		globalStats.bytes_sent = 0;
! 		globalStats.bytes_received = 0;
! 		globalStats.socket_stat_reset_timestamp = GetCurrentTimestamp();
  	}
  
  	/*
***************
*** 4951,4956 **** pgstat_recv_tempfile(PgStat_MsgTempFile *msg, int len)
--- 5048,5093 ----
  }
  
  /* ----------
+  * pgstat_recv_commsent() -
+  *
+  *    Process a COMMSENT message.
+  * ----------
+  */
+ static void
+ pgstat_recv_commsent(PgStat_MsgComm *msg, int len)
+ {
+ 	PgStat_StatDBEntry *dbentry;
+ 
+ 	globalStats.bytes_sent += msg->m_bytes_transferred;
+ 
+ 	/* we can be called before we have connected to a specific database or during replication */
+ 	if (OidIsValid(msg->m_databaseid)) {
+ 		dbentry = pgstat_get_db_entry(msg->m_databaseid, true);
+ 		dbentry->n_bytes_sent += msg->m_bytes_transferred;
+ 	}
+ }
+ 
+ /* ----------
+  * pgstat_recv_commreceived() -
+  *
+  *    Process a COMMRECEIVED message.
+  * ----------
+  */
+ static void
+ pgstat_recv_commreceived(PgStat_MsgComm *msg, int len)
+ {
+ 	PgStat_StatDBEntry *dbentry;
+ 
+ 	globalStats.bytes_received += msg->m_bytes_transferred;
+ 
+ 	/* can be called before we have connected to a specific database or during replication */
+ 	if (OidIsValid(msg->m_databaseid)) {
+ 		dbentry = pgstat_get_db_entry(msg->m_databaseid, true);
+ 		dbentry->n_bytes_received += msg->m_bytes_transferred;
+ 	}
+ }
+ 
+ /* ----------
   * pgstat_recv_funcstat() -
   *
   *	Count what the backend has done.
*** a/src/backend/postmaster/postmaster.c
--- b/src/backend/postmaster/postmaster.c
***************
*** 1825,1830 **** retry1:
--- 1825,1832 ----
  					 errmsg("failed to send SSL negotiation response: %m")));
  			return STATUS_ERROR;	/* close the connection */
  		}
+ 		else
+ 			pgstat_report_commsent(1);
  
  #ifdef USE_SSL
  		if (SSLok == 'S' && secure_open_server(port) == -1)
***************
*** 3839,3844 **** report_fork_failure_to_client(Port *port, int errnum)
--- 3841,3849 ----
  	{
  		rc = send(port->sock, buffer, strlen(buffer) + 1, 0);
  	} while (rc < 0 && errno == EINTR);
+ 
+ 	if (rc > 0)
+ 		pgstat_report_commsent(rc);
  }
  
  
*** a/src/backend/replication/walreceiver.c
--- b/src/backend/replication/walreceiver.c
***************
*** 431,436 **** WalReceiverMain(void)
--- 431,437 ----
  							last_recv_timestamp = GetCurrentTimestamp();
  							ping_sent = false;
  							XLogWalRcvProcessMsg(buf[0], &buf[1], len - 1);
+ 							pgstat_report_commreceived(len);
  						}
  						else if (len == 0)
  							break;
*** a/src/backend/utils/adt/pgstatfuncs.c
--- b/src/backend/utils/adt/pgstatfuncs.c
***************
*** 86,91 **** extern Datum pg_stat_get_db_temp_files(PG_FUNCTION_ARGS);
--- 86,93 ----
  extern Datum pg_stat_get_db_temp_bytes(PG_FUNCTION_ARGS);
  extern Datum pg_stat_get_db_blk_read_time(PG_FUNCTION_ARGS);
  extern Datum pg_stat_get_db_blk_write_time(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_db_bytes_sent(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_db_bytes_received(PG_FUNCTION_ARGS);
  
  extern Datum pg_stat_get_bgwriter_timed_checkpoints(PG_FUNCTION_ARGS);
  extern Datum pg_stat_get_bgwriter_requested_checkpoints(PG_FUNCTION_ARGS);
***************
*** 99,104 **** extern Datum pg_stat_get_buf_written_backend(PG_FUNCTION_ARGS);
--- 101,110 ----
  extern Datum pg_stat_get_buf_fsync_backend(PG_FUNCTION_ARGS);
  extern Datum pg_stat_get_buf_alloc(PG_FUNCTION_ARGS);
  
+ extern Datum pg_stat_get_bytes_sent(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_bytes_received(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_socket_stat_reset_time(PG_FUNCTION_ARGS);
+ 
  extern Datum pg_stat_get_xact_numscans(PG_FUNCTION_ARGS);
  extern Datum pg_stat_get_xact_tuples_returned(PG_FUNCTION_ARGS);
  extern Datum pg_stat_get_xact_tuples_fetched(PG_FUNCTION_ARGS);
***************
*** 534,540 **** pg_stat_get_activity(PG_FUNCTION_ARGS)
  
  		oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
  
! 		tupdesc = CreateTemplateTupleDesc(14, false);
  		TupleDescInitEntry(tupdesc, (AttrNumber) 1, "datid",
  						   OIDOID, -1, 0);
  		TupleDescInitEntry(tupdesc, (AttrNumber) 2, "pid",
--- 540,546 ----
  
  		oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
  
! 		tupdesc = CreateTemplateTupleDesc(16, false);
  		TupleDescInitEntry(tupdesc, (AttrNumber) 1, "datid",
  						   OIDOID, -1, 0);
  		TupleDescInitEntry(tupdesc, (AttrNumber) 2, "pid",
***************
*** 563,568 **** pg_stat_get_activity(PG_FUNCTION_ARGS)
--- 569,578 ----
  						   TEXTOID, -1, 0);
  		TupleDescInitEntry(tupdesc, (AttrNumber) 14, "client_port",
  						   INT4OID, -1, 0);
+ 		TupleDescInitEntry(tupdesc, (AttrNumber) 15, "bytes_sent",
+ 						   INT8OID, -1, 0);
+ 		TupleDescInitEntry(tupdesc, (AttrNumber) 16, "bytes_received",
+ 						   INT8OID, -1, 0);
  
  		funcctx->tuple_desc = BlessTupleDesc(tupdesc);
  
***************
*** 614,621 **** pg_stat_get_activity(PG_FUNCTION_ARGS)
  	if (funcctx->call_cntr < funcctx->max_calls)
  	{
  		/* for each row */
! 		Datum		values[14];
! 		bool		nulls[14];
  		HeapTuple	tuple;
  		PgBackendStatus *beentry;
  		SockAddr	zero_clientaddr;
--- 624,631 ----
  	if (funcctx->call_cntr < funcctx->max_calls)
  	{
  		/* for each row */
! 		Datum		values[16];
! 		bool		nulls[16];
  		HeapTuple	tuple;
  		PgBackendStatus *beentry;
  		SockAddr	zero_clientaddr;
***************
*** 773,778 **** pg_stat_get_activity(PG_FUNCTION_ARGS)
--- 783,790 ----
  					nulls[13] = true;
  				}
  			}
+ 			values[14] = Int64GetDatum(beentry->st_bytes_sent);
+ 			values[15] = Int64GetDatum(beentry->st_bytes_received);
  		}
  		else
  		{
***************
*** 787,792 **** pg_stat_get_activity(PG_FUNCTION_ARGS)
--- 799,806 ----
  			nulls[11] = true;
  			nulls[12] = true;
  			nulls[13] = true;
+ 			nulls[14] = true;
+ 			nulls[15] = true;
  		}
  
  		tuple = heap_form_tuple(funcctx->tuple_desc, values, nulls);
***************
*** 1407,1412 **** pg_stat_get_db_blk_write_time(PG_FUNCTION_ARGS)
--- 1421,1456 ----
  }
  
  Datum
+ pg_stat_get_db_bytes_sent(PG_FUNCTION_ARGS)
+ {
+ 	Oid			dbid = PG_GETARG_OID(0);
+ 	double		result;
+ 	PgStat_StatDBEntry *dbentry;
+ 
+ 	if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
+ 		result = 0;
+ 	else
+ 		result = dbentry->n_bytes_sent;
+ 
+ 	PG_RETURN_INT64(result);
+ }
+ 
+ Datum
+ pg_stat_get_db_bytes_received(PG_FUNCTION_ARGS)
+ {
+ 	Oid			dbid = PG_GETARG_OID(0);
+ 	double		result;
+ 	PgStat_StatDBEntry *dbentry;
+ 
+ 	if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
+ 		result = 0;
+ 	else
+ 		result = dbentry->n_bytes_received;
+ 
+ 	PG_RETURN_INT64(result);
+ }
+ 
+ Datum
  pg_stat_get_bgwriter_timed_checkpoints(PG_FUNCTION_ARGS)
  {
  	PG_RETURN_INT64(pgstat_fetch_global()->timed_checkpoints);
***************
*** 1453,1459 **** pg_stat_get_checkpoint_sync_time(PG_FUNCTION_ARGS)
  Datum
  pg_stat_get_bgwriter_stat_reset_time(PG_FUNCTION_ARGS)
  {
! 	PG_RETURN_TIMESTAMPTZ(pgstat_fetch_global()->stat_reset_timestamp);
  }
  
  Datum
--- 1497,1503 ----
  Datum
  pg_stat_get_bgwriter_stat_reset_time(PG_FUNCTION_ARGS)
  {
! 	PG_RETURN_TIMESTAMPTZ(pgstat_fetch_global()->bgwriter_stat_reset_timestamp);
  }
  
  Datum
***************
*** 1475,1480 **** pg_stat_get_buf_alloc(PG_FUNCTION_ARGS)
--- 1519,1542 ----
  }
  
  Datum
+ pg_stat_get_bytes_sent(PG_FUNCTION_ARGS)
+ {
+ 	PG_RETURN_INT64(pgstat_fetch_global()->bytes_sent);
+ }
+ 
+ Datum
+ pg_stat_get_bytes_received(PG_FUNCTION_ARGS)
+ {
+ 	PG_RETURN_INT64(pgstat_fetch_global()->bytes_received);
+ }
+ 
+ Datum
+ pg_stat_get_socket_stat_reset_time(PG_FUNCTION_ARGS)
+ {
+ 	PG_RETURN_TIMESTAMPTZ(pgstat_fetch_global()->socket_stat_reset_timestamp);
+ }
+ 
+ Datum
  pg_stat_get_xact_numscans(PG_FUNCTION_ARGS)
  {
  	Oid			relid = PG_GETARG_OID(0);
*** a/src/include/catalog/pg_proc.h
--- b/src/include/catalog/pg_proc.h
***************
*** 2617,2623 **** DATA(insert OID = 3057 ( pg_stat_get_autoanalyze_count PGNSP PGUID 12 1 0 0 0 f
  DESCR("statistics: number of auto analyzes for a table");
  DATA(insert OID = 1936 (  pg_stat_get_backend_idset		PGNSP PGUID 12 1 100 0 0 f f f f t t s 0 0 23 "" _null_ _null_ _null_ _null_ pg_stat_get_backend_idset _null_ _null_ _null_ ));
  DESCR("statistics: currently active backend IDs");
! DATA(insert OID = 2022 (  pg_stat_get_activity			PGNSP PGUID 12 1 100 0 0 f f f f f t s 1 0 2249 "23" "{23,26,23,26,25,25,25,16,1184,1184,1184,1184,869,25,23}" "{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o}" "{pid,datid,pid,usesysid,application_name,state,query,waiting,xact_start,query_start,backend_start,state_change,client_addr,client_hostname,client_port}" _null_ pg_stat_get_activity _null_ _null_ _null_ ));
  DESCR("statistics: information about currently active backends");
  DATA(insert OID = 3099 (  pg_stat_get_wal_senders	PGNSP PGUID 12 1 10 0 0 f f f f f t s 0 0 2249 "" "{23,25,25,25,25,25,23,25}" "{o,o,o,o,o,o,o,o}" "{pid,state,sent_location,write_location,flush_location,replay_location,sync_priority,sync_state}" _null_ pg_stat_get_wal_senders _null_ _null_ _null_ ));
  DESCR("statistics: information about currently active replication");
--- 2617,2623 ----
  DESCR("statistics: number of auto analyzes for a table");
  DATA(insert OID = 1936 (  pg_stat_get_backend_idset		PGNSP PGUID 12 1 100 0 0 f f f f t t s 0 0 23 "" _null_ _null_ _null_ _null_ pg_stat_get_backend_idset _null_ _null_ _null_ ));
  DESCR("statistics: currently active backend IDs");
! DATA(insert OID = 2022 (  pg_stat_get_activity			PGNSP PGUID 12 1 100 0 0 f f f f f t s 1 0 2249 "23" "{23,26,23,26,25,25,25,16,1184,1184,1184,1184,869,25,23,20,20}" "{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}" "{pid,datid,pid,usesysid,application_name,state,query,waiting,xact_start,query_start,backend_start,state_change,client_addr,client_hostname,client_port,bytes_sent,bytes_received}" _null_ pg_stat_get_activity _null_ _null_ _null_ ));
  DESCR("statistics: information about currently active backends");
  DATA(insert OID = 3099 (  pg_stat_get_wal_senders	PGNSP PGUID 12 1 10 0 0 f f f f f t s 0 0 2249 "" "{23,25,25,25,25,25,23,25}" "{o,o,o,o,o,o,o,o}" "{pid,state,sent_location,write_location,flush_location,replay_location,sync_priority,sync_state}" _null_ pg_stat_get_wal_senders _null_ _null_ _null_ ));
  DESCR("statistics: information about currently active replication");
***************
*** 2687,2692 **** DATA(insert OID = 2844 (  pg_stat_get_db_blk_read_time	PGNSP PGUID 12 1 0 0 0 f
--- 2687,2696 ----
  DESCR("statistics: block read time, in msec");
  DATA(insert OID = 2845 (  pg_stat_get_db_blk_write_time PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 701 "26" _null_ _null_ _null_ _null_ pg_stat_get_db_blk_write_time _null_ _null_ _null_ ));
  DESCR("statistics: block write time, in msec");
+ DATA(insert OID = 3195 (  pg_stat_get_db_bytes_sent PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_db_bytes_sent _null_ _null_ _null_ ));
+ DESCR("statistics: number of bytes sent over the network");
+ DATA(insert OID = 3196 (  pg_stat_get_db_bytes_received PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_db_bytes_received _null_ _null_ _null_ ));
+ DESCR("statistics: number of bytes received over the network");
  DATA(insert OID = 2769 ( pg_stat_get_bgwriter_timed_checkpoints PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 20 "" _null_ _null_ _null_ _null_ pg_stat_get_bgwriter_timed_checkpoints _null_ _null_ _null_ ));
  DESCR("statistics: number of timed checkpoints started by the bgwriter");
  DATA(insert OID = 2770 ( pg_stat_get_bgwriter_requested_checkpoints PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 20 "" _null_ _null_ _null_ _null_ pg_stat_get_bgwriter_requested_checkpoints _null_ _null_ _null_ ));
***************
*** 2710,2715 **** DESCR("statistics: number of backend buffer writes that did their own fsync");
--- 2714,2726 ----
  DATA(insert OID = 2859 ( pg_stat_get_buf_alloc			PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 20 "" _null_ _null_ _null_ _null_ pg_stat_get_buf_alloc _null_ _null_ _null_ ));
  DESCR("statistics: number of buffer allocations");
  
+ DATA(insert OID = 3197 ( pg_stat_get_bytes_sent PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 20 "" _null_ _null_ _null_ _null_ pg_stat_get_bytes_sent _null_ _null_ _null_ ));
+ DESCR("statistics: number of bytes sent over the network");
+ DATA(insert OID = 3198 ( pg_stat_get_bytes_received PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 20 "" _null_ _null_ _null_ _null_ pg_stat_get_bytes_received _null_ _null_ _null_ ));
+ DESCR("statistics: number of bytes received over the network");
+ DATA(insert OID = 3199 ( pg_stat_get_socket_stat_reset_time PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 1184 "" _null_ _null_ _null_ _null_	pg_stat_get_socket_stat_reset_time _null_ _null_ _null_ ));
+ DESCR("statistics: last reset for the socket counters");
+ 
  DATA(insert OID = 2978 (  pg_stat_get_function_calls		PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_function_calls _null_ _null_ _null_ ));
  DESCR("statistics: number of function calls");
  DATA(insert OID = 2979 (  pg_stat_get_function_total_time	PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 701 "26" _null_ _null_ _null_ _null_ pg_stat_get_function_total_time _null_ _null_ _null_ ));
*** a/src/include/pgstat.h
--- b/src/include/pgstat.h
***************
*** 49,55 **** typedef enum StatMsgType
  	PGSTAT_MTYPE_FUNCPURGE,
  	PGSTAT_MTYPE_RECOVERYCONFLICT,
  	PGSTAT_MTYPE_TEMPFILE,
! 	PGSTAT_MTYPE_DEADLOCK
  } StatMsgType;
  
  /* ----------
--- 49,57 ----
  	PGSTAT_MTYPE_FUNCPURGE,
  	PGSTAT_MTYPE_RECOVERYCONFLICT,
  	PGSTAT_MTYPE_TEMPFILE,
! 	PGSTAT_MTYPE_DEADLOCK,
! 	PGSTAT_MTYPE_COMMSENT,
! 	PGSTAT_MTYPE_COMMRECEIVED
  } StatMsgType;
  
  /* ----------
***************
*** 102,108 **** typedef struct PgStat_TableCounts
  /* Possible targets for resetting cluster-wide shared values */
  typedef enum PgStat_Shared_Reset_Target
  {
! 	RESET_BGWRITER
  } PgStat_Shared_Reset_Target;
  
  /* Possible object types for resetting single counters */
--- 104,111 ----
  /* Possible targets for resetting cluster-wide shared values */
  typedef enum PgStat_Shared_Reset_Target
  {
! 	RESET_BGWRITER,
! 	RESET_SOCKET
  } PgStat_Shared_Reset_Target;
  
  /* Possible object types for resetting single counters */
***************
*** 397,402 **** typedef struct PgStat_MsgTempFile
--- 400,419 ----
  } PgStat_MsgTempFile;
  
  /* ----------
+   * PgStat_MsgComm
+   *
+   * Sent upon sending or receiving data over a socket (header determines direction)
+   * ----------
+   */
+ typedef struct PgStat_MsgComm
+ {
+ 	PgStat_MsgHdr	m_hdr;
+ 
+ 	Oid			m_databaseid;
+ 	int			m_bytes_transferred;
+ } PgStat_MsgComm;
+ 
+ /* ----------
   * PgStat_FunctionCounts	The actual per-function counts kept by a backend
   *
   * This struct should contain only actual event counters, because we memcmp
***************
*** 515,521 **** typedef union PgStat_Msg
   * ------------------------------------------------------------
   */
  
! #define PGSTAT_FILE_FORMAT_ID	0x01A5BC9B
  
  /* ----------
   * PgStat_StatDBEntry			The collector's data per database
--- 532,538 ----
   * ------------------------------------------------------------
   */
  
! #define PGSTAT_FILE_FORMAT_ID	0x01A5BC9C
  
  /* ----------
   * PgStat_StatDBEntry			The collector's data per database
***************
*** 544,549 **** typedef struct PgStat_StatDBEntry
--- 561,570 ----
  	PgStat_Counter n_deadlocks;
  	PgStat_Counter n_block_read_time;	/* times in microseconds */
  	PgStat_Counter n_block_write_time;
+ 	/* network transfer counter in bytes (backend to client) */
+ 	PgStat_Counter n_bytes_sent;
+ 	/* network transfer counter in bytes (backend to postmaster) */
+ 	PgStat_Counter n_bytes_received;
  
  	TimestampTz stat_reset_timestamp;
  	TimestampTz stats_timestamp;	/* time of db stats file update */
***************
*** 614,619 **** typedef struct PgStat_StatFuncEntry
--- 635,641 ----
  typedef struct PgStat_GlobalStats
  {
  	TimestampTz stats_timestamp;	/* time of stats file update */
+ 	/* bgwriter stats */
  	PgStat_Counter timed_checkpoints;
  	PgStat_Counter requested_checkpoints;
  	PgStat_Counter checkpoint_write_time;		/* times in milliseconds */
***************
*** 624,630 **** typedef struct PgStat_GlobalStats
  	PgStat_Counter buf_written_backend;
  	PgStat_Counter buf_fsync_backend;
  	PgStat_Counter buf_alloc;
! 	TimestampTz stat_reset_timestamp;
  } PgStat_GlobalStats;
  
  
--- 646,656 ----
  	PgStat_Counter buf_written_backend;
  	PgStat_Counter buf_fsync_backend;
  	PgStat_Counter buf_alloc;
! 	TimestampTz bgwriter_stat_reset_timestamp;
! 	/* socket communication stats */
! 	PgStat_Counter bytes_sent; /* network transfer counter in bytes (cluster to client) */
! 	PgStat_Counter bytes_received; /* network transfer counter in bytes (client to cluster) */
! 	TimestampTz socket_stat_reset_timestamp;
  } PgStat_GlobalStats;
  
  
***************
*** 697,702 **** typedef struct PgBackendStatus
--- 723,734 ----
  
  	/* current command string; MUST be null-terminated */
  	char	   *st_activity;
+ 
+ 	/* network transfer counter in bytes (backend to client) */
+ 	unsigned long st_bytes_sent;
+ 	/* network transfer counter in bytes (client to backend) */
+ 	unsigned long st_bytes_received;
+ 
  } PgBackendStatus;
  
  /*
pgstats.pyapplication/octet-stream; name=pgstats.pyDownload
#2Stephen Frost
sfrost@snowman.net
In reply to: Nigel Heron (#1)
Re: stats for network traffic WIP

Nigel,

* Nigel Heron (nheron@querymetrics.com) wrote:

Hi, I've been using postgres for many years but never took the time to play
with the code until now. As a learning experience i came up with this WIP
patch to keep track of the # of bytes sent and received by the server over
it's communication sockets. Counters are kept per database, per connection
and globally/shared.

Very neat idea. Please add it to the current commitfest
(http://commitfest.postgresql.org) and, ideally, someone will get in and
review it during the next CM.

Thanks!

Stephen

#3Mike Blackwell
mike.blackwell@rrd.com
In reply to: Stephen Frost (#2)
Re: stats for network traffic WIP

I added this to the current CF, and am starting to review it as I have time.

__________________________________________________________________________________
*Mike Blackwell | Technical Analyst, Distribution Services/Rollout
Management | RR Donnelley*
1750 Wallace Ave | St Charles, IL 60174-3401
Office: 630.313.7818
Mike.Blackwell@rrd.com
http://www.rrdonnelley.com

<http://www.rrdonnelley.com/&gt;
* <Mike.Blackwell@rrd.com>*

On Mon, Oct 21, 2013 at 11:32 AM, Stephen Frost <sfrost@snowman.net> wrote:

Show quoted text

Nigel,

* Nigel Heron (nheron@querymetrics.com) wrote:

Hi, I've been using postgres for many years but never took the time to

play

with the code until now. As a learning experience i came up with this WIP
patch to keep track of the # of bytes sent and received by the server

over

it's communication sockets. Counters are kept per database, per

connection

and globally/shared.

Very neat idea. Please add it to the current commitfest
(http://commitfest.postgresql.org) and, ideally, someone will get in and
review it during the next CM.

Thanks!

Stephen

#4Nigel Heron
nheron@querymetrics.com
In reply to: Mike Blackwell (#3)
Re: stats for network traffic WIP

Hi, thanks, I'm still actively working on this patch. I've gotten the
traffic counters working when using SSL enabled clients (includes the
ssl overhead now) but I still have the walsender transfers under SSL
to work on.
I'll post an updated patch when i have it figured out.
Since the patch changes some views in pg_catalog, a regression test
fails .. i'm not sure what to do next. Change the regression test in
the patch, or wait until the review phase?

I was also thinking of adding global counters for the stats collector
(pg_stat* file read/write bytes + packets lost) and also log file io
(bytes written for txt and csv formats) .. any interest?

-nigel.

On Wed, Oct 23, 2013 at 12:50 PM, Mike Blackwell <mike.blackwell@rrd.com> wrote:

I added this to the current CF, and am starting to review it as I have time.

__________________________________________________________________________________
Mike Blackwell | Technical Analyst, Distribution Services/Rollout Management
| RR Donnelley
1750 Wallace Ave | St Charles, IL 60174-3401
Office: 630.313.7818
Mike.Blackwell@rrd.com
http://www.rrdonnelley.com

On Mon, Oct 21, 2013 at 11:32 AM, Stephen Frost <sfrost@snowman.net> wrote:

Nigel,

* Nigel Heron (nheron@querymetrics.com) wrote:

Hi, I've been using postgres for many years but never took the time to
play
with the code until now. As a learning experience i came up with this
WIP
patch to keep track of the # of bytes sent and received by the server
over
it's communication sockets. Counters are kept per database, per
connection
and globally/shared.

Very neat idea. Please add it to the current commitfest
(http://commitfest.postgresql.org) and, ideally, someone will get in and
review it during the next CM.

Thanks!

Stephen

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#5Mike Blackwell
mike.blackwell@rrd.com
In reply to: Nigel Heron (#4)
Re: stats for network traffic WIP

Sounds good. I personally don't have any interest in log file i/o
counters, but that's just me. I wonder if stats collector counters might
be useful... I seem to recall an effort to improve that area. Maybe not
enough use to take the performance hit on a regular basis, though.

__________________________________________________________________________________
*Mike Blackwell | Technical Analyst, Distribution Services/Rollout
Management | RR Donnelley*
1750 Wallace Ave | St Charles, IL 60174-3401
Office: 630.313.7818
Mike.Blackwell@rrd.com
http://www.rrdonnelley.com

<http://www.rrdonnelley.com/&gt;
* <Mike.Blackwell@rrd.com>*

On Wed, Oct 23, 2013 at 1:44 PM, Nigel Heron <nheron@querymetrics.com>wrote:

Show quoted text

Hi, thanks, I'm still actively working on this patch. I've gotten the
traffic counters working when using SSL enabled clients (includes the
ssl overhead now) but I still have the walsender transfers under SSL
to work on.
I'll post an updated patch when i have it figured out.
Since the patch changes some views in pg_catalog, a regression test
fails .. i'm not sure what to do next. Change the regression test in
the patch, or wait until the review phase?

I was also thinking of adding global counters for the stats collector
(pg_stat* file read/write bytes + packets lost) and also log file io
(bytes written for txt and csv formats) .. any interest?

-nigel.

On Wed, Oct 23, 2013 at 12:50 PM, Mike Blackwell <mike.blackwell@rrd.com>
wrote:

I added this to the current CF, and am starting to review it as I have

time.

__________________________________________________________________________________

Mike Blackwell | Technical Analyst, Distribution Services/Rollout

Management

| RR Donnelley
1750 Wallace Ave | St Charles, IL 60174-3401
Office: 630.313.7818
Mike.Blackwell@rrd.com
http://www.rrdonnelley.com

On Mon, Oct 21, 2013 at 11:32 AM, Stephen Frost <sfrost@snowman.net>

wrote:

Nigel,

* Nigel Heron (nheron@querymetrics.com) wrote:

Hi, I've been using postgres for many years but never took the time to
play
with the code until now. As a learning experience i came up with this
WIP
patch to keep track of the # of bytes sent and received by the server
over
it's communication sockets. Counters are kept per database, per
connection
and globally/shared.

Very neat idea. Please add it to the current commitfest
(http://commitfest.postgresql.org) and, ideally, someone will get in

and

review it during the next CM.

Thanks!

Stephen

#6Atri Sharma
atri.jiit@gmail.com
In reply to: Mike Blackwell (#5)
Re: stats for network traffic WIP

On Thu, Oct 24, 2013 at 12:23 AM, Mike Blackwell <mike.blackwell@rrd.com> wrote:

Sounds good. I personally don't have any interest in log file i/o counters,
but that's just me. I wonder if stats collector counters might be useful...
I seem to recall an effort to improve that area. Maybe not enough use to
take the performance hit on a regular basis, though.

+1.

I tend to be a bit touchy about any changes to code that runs
frequently. We need to seriously test if the overhead added by this
patch is worth it.

IMO, the idea is pretty good. Its just that we need to do some wide
spectrum performance testing. Thats only my thought though.

Regards,

Atri

--
Regards,

Atri
l'apprenant

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#7Mike Blackwell
mike.blackwell@rrd.com
In reply to: Atri Sharma (#6)
Re: stats for network traffic WIP

On Wed, Oct 23, 2013 at 1:58 PM, Atri Sharma <atri.jiit@gmail.com> wrote:

IMO, the idea is pretty good. Its just that we need to do some wide
spectrum performance testing. Thats only my thought though.

I'm looking at trying to do some performance testing on this. Any
suggestions on test scenarios, etc?

__________________________________________________________________________________
*Mike Blackwell | Technical Analyst, Distribution Services/Rollout
Management | RR Donnelley*
1750 Wallace Ave | St Charles, IL 60174-3401
Office: 630.313.7818
Mike.Blackwell@rrd.com
http://www.rrdonnelley.com

<http://www.rrdonnelley.com/&gt;
* <Mike.Blackwell@rrd.com>*

#8Nigel Heron
nheron@querymetrics.com
In reply to: Atri Sharma (#6)
Re: stats for network traffic WIP

On Wed, Oct 23, 2013 at 2:58 PM, Atri Sharma <atri.jiit@gmail.com> wrote:

On Thu, Oct 24, 2013 at 12:23 AM, Mike Blackwell <mike.blackwell@rrd.com> wrote:

Sounds good. I personally don't have any interest in log file i/o counters,
but that's just me. I wonder if stats collector counters might be useful...
I seem to recall an effort to improve that area. Maybe not enough use to
take the performance hit on a regular basis, though.

+1.

I tend to be a bit touchy about any changes to code that runs
frequently. We need to seriously test if the overhead added by this
patch is worth it.

IMO, the idea is pretty good. Its just that we need to do some wide
spectrum performance testing. Thats only my thought though.

I didn't implement the code yet, but my impression is that since it
will be the stats collector gathering counters about itself there will
be very little overhead (no message passing, etc.) .. just a few int
calculations and storing a few more bytes in the global stats file.
The log file io tracking would generate some overhead though, similar
to network stats tracking.
I think the stats collector concerns voiced previously on the list
were more about per relation stats which creates alot of io on servers
with many tables. Adding global stats doesn't seem as bad to me.

-nigel.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#9Atri Sharma
atri.jiit@gmail.com
In reply to: Mike Blackwell (#7)
Re: stats for network traffic WIP

On Thu, Oct 24, 2013 at 12:30 AM, Mike Blackwell <mike.blackwell@rrd.com> wrote:

On Wed, Oct 23, 2013 at 1:58 PM, Atri Sharma <atri.jiit@gmail.com> wrote:

IMO, the idea is pretty good. Its just that we need to do some wide
spectrum performance testing. Thats only my thought though.

I'm looking at trying to do some performance testing on this. Any
suggestions on test scenarios, etc?

Umm...Lots of clients together would be the first obvious testing that
comes to my mind.

One thing to look at would be erratic clients. If some clients connect
and disconnect within a short span of time, we should look if the
collector works fine there.

Also, we should verify the accuracy of the statistics collected. A
small deviation is fine, but we should do a formal test, just to be
sure.

Does anyone think that the new untracked ports introduced by the patch
could pose a problem? I am not sure there.

I havent taken a deep look at the patch yet, but I will try to do so.
However, since I will be in Dublin next week, it may happen that my
inputs may be delayed a bit. The plus side is that I will discuss this
with lots of people there.

Adding myself as the co reviewer specifically for the testing
purposes, if its ok with you.

Regards,

Atri

--
Regards,

Atri
l'apprenant

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#10Mike Blackwell
mike.blackwell@rrd.com
In reply to: Atri Sharma (#9)
Re: stats for network traffic WIP

On Wed, Oct 23, 2013 at 2:10 PM, Atri Sharma <atri.jiit@gmail.com> wrote:

Adding myself as the co reviewer specifically for the testing
purposes, if its ok with you.

​It's perfectly fine with me. Please do!​

__________________________________________________________________________________
*Mike Blackwell | Technical Analyst, Distribution Services/Rollout
Management | RR Donnelley*
1750 Wallace Ave | St Charles, IL 60174-3401
Office: 630.313.7818
Mike.Blackwell@rrd.com
http://www.rrdonnelley.com

<http://www.rrdonnelley.com/&gt;
* <Mike.Blackwell@rrd.com>*

#11Nigel Heron
nheron@querymetrics.com
In reply to: Nigel Heron (#4)
1 attachment(s)
Re: stats for network traffic WIP

On Wed, Oct 23, 2013 at 2:44 PM, Nigel Heron <nheron@querymetrics.com> wrote:

Hi, thanks, I'm still actively working on this patch. I've gotten the
traffic counters working when using SSL enabled clients (includes the
ssl overhead now) but I still have the walsender transfers under SSL
to work on.
I'll post an updated patch when i have it figured out.
Since the patch changes some views in pg_catalog, a regression test
fails .. i'm not sure what to do next. Change the regression test in
the patch, or wait until the review phase?

here's v2 of the patch including the regression test update.
I omitted socket counters for walreceivers, i couldn't get them
working under SSL. Since they are using the front end libpq libs i
would have to duplicate alot of the code in the backend to be able to
instrument them under SSL (add openssl BIO custom send/recv like the
backend has), not sure it's worth it.. We can get the data from the
master's pg_stat_replication view anyways. I'm open to suggestions.

So, for now, the counters only track sockets created from an inbound
(client to server) connection.

-nigel.

Attachments:

netstats-v2.patch.gzapplication/x-gzip; name=netstats-v2.patch.gzDownload
#12Nigel Heron
nheron@querymetrics.com
In reply to: Nigel Heron (#11)
1 attachment(s)
Re: stats for network traffic WIP

So, for now, the counters only track sockets created from an inbound
(client to server) connection.

here's v3 of the patch (rebase and cleanup).

-nigel.

Attachments:

netstats-v3.patchtext/x-patch; charset=US-ASCII; name=netstats-v3.patchDownload
*** a/src/backend/catalog/system_views.sql
--- b/src/backend/catalog/system_views.sql
***************
*** 586,592 **** CREATE VIEW pg_stat_activity AS
              S.state_change,
              S.waiting,
              S.state,
!             S.query
      FROM pg_database D, pg_stat_get_activity(NULL) AS S, pg_authid U
      WHERE S.datid = D.oid AND
              S.usesysid = U.oid;
--- 586,594 ----
              S.state_change,
              S.waiting,
              S.state,
!             S.query,
!             S.bytes_sent,
!             S.bytes_received
      FROM pg_database D, pg_stat_get_activity(NULL) AS S, pg_authid U
      WHERE S.datid = D.oid AND
              S.usesysid = U.oid;
***************
*** 601,606 **** CREATE VIEW pg_stat_replication AS
--- 603,610 ----
              S.client_hostname,
              S.client_port,
              S.backend_start,
+             S.bytes_sent,
+             S.bytes_received,
              W.state,
              W.sent_location,
              W.write_location,
***************
*** 634,639 **** CREATE VIEW pg_stat_database AS
--- 638,645 ----
              pg_stat_get_db_deadlocks(D.oid) AS deadlocks,
              pg_stat_get_db_blk_read_time(D.oid) AS blk_read_time,
              pg_stat_get_db_blk_write_time(D.oid) AS blk_write_time,
+             pg_stat_get_db_bytes_sent(D.oid) AS bytes_sent,
+             pg_stat_get_db_bytes_received(D.oid) AS bytes_received,
              pg_stat_get_db_stat_reset_time(D.oid) AS stats_reset
      FROM pg_database D;
  
*** a/src/backend/libpq/be-secure.c
--- b/src/backend/libpq/be-secure.c
***************
*** 74,80 ****
  #include "libpq/libpq.h"
  #include "tcop/tcopprot.h"
  #include "utils/memutils.h"
! 
  
  #ifdef USE_SSL
  
--- 74,80 ----
  #include "libpq/libpq.h"
  #include "tcop/tcopprot.h"
  #include "utils/memutils.h"
! #include "pgstat.h"
  
  #ifdef USE_SSL
  
***************
*** 307,312 **** rloop:
--- 307,318 ----
  		n = recv(port->sock, ptr, len, 0);
  
  		client_read_ended();
+ 
+ 		if (n > 0)
+ 		{
+ 			/* we received data from the socket that needs to be reported */
+ 			pgstat_report_commreceived(n);
+ 		}
  	}
  
  	return n;
***************
*** 441,447 **** wloop:
--- 447,460 ----
  	}
  	else
  #endif
+ 	{
  		n = send(port->sock, ptr, len, 0);
+ 		if (n > 0)
+ 		{
+ 			/* we sent data over the socket that needs to be reported */
+ 			pgstat_report_commsent(n);
+ 		}
+ 	}
  
  	return n;
  }
***************
*** 488,493 **** my_sock_read(BIO *h, char *buf, int size)
--- 501,512 ----
  
  	client_read_ended();
  
+ 	if (res > 0)
+ 	{
+ 		/* we received data from the socket that needs to be reported */
+ 		pgstat_report_commreceived(res);
+ 	}
+ 
  	return res;
  }
  
***************
*** 504,509 **** my_sock_write(BIO *h, const char *buf, int size)
--- 523,533 ----
  			BIO_set_retry_write(h);
  		}
  	}
+ 	else
+ 	{
+ 		/* we sent data over the socket that needs to be reported */
+ 		pgstat_report_commsent(res);
+ 	}
  
  	return res;
  }
*** a/src/backend/postmaster/pgstat.c
--- b/src/backend/postmaster/pgstat.c
***************
*** 298,303 **** static void pgstat_recv_funcpurge(PgStat_MsgFuncpurge *msg, int len);
--- 298,305 ----
  static void pgstat_recv_recoveryconflict(PgStat_MsgRecoveryConflict *msg, int len);
  static void pgstat_recv_deadlock(PgStat_MsgDeadlock *msg, int len);
  static void pgstat_recv_tempfile(PgStat_MsgTempFile *msg, int len);
+ static void pgstat_recv_commsent(PgStat_MsgComm *msg, int len);
+ static void pgstat_recv_commreceived(PgStat_MsgComm *msg, int len);
  
  /* ------------------------------------------------------------
   * Public functions called from postmaster follow
***************
*** 1249,1259 **** pgstat_reset_shared_counters(const char *target)
  
  	if (strcmp(target, "bgwriter") == 0)
  		msg.m_resettarget = RESET_BGWRITER;
  	else
  		ereport(ERROR,
  				(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
  				 errmsg("unrecognized reset target: \"%s\"", target),
! 				 errhint("Target must be \"bgwriter\".")));
  
  	pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETSHAREDCOUNTER);
  	pgstat_send(&msg, sizeof(msg));
--- 1251,1263 ----
  
  	if (strcmp(target, "bgwriter") == 0)
  		msg.m_resettarget = RESET_BGWRITER;
+ 	else if (strcmp(target, "socket") == 0)
+ 		msg.m_resettarget = RESET_SOCKET;
  	else
  		ereport(ERROR,
  				(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
  				 errmsg("unrecognized reset target: \"%s\"", target),
! 				 errhint("Target must be \"bgwriter\" or \"socket\".")));
  
  	pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETSHAREDCOUNTER);
  	pgstat_send(&msg, sizeof(msg));
***************
*** 2531,2536 **** pgstat_bestart(void)
--- 2535,2542 ----
  	beentry->st_clienthostname[NAMEDATALEN - 1] = '\0';
  	beentry->st_appname[NAMEDATALEN - 1] = '\0';
  	beentry->st_activity[pgstat_track_activity_query_size - 1] = '\0';
+ 	beentry->st_bytes_sent = 0;
+ 	beentry->st_bytes_received = 0;
  
  	beentry->st_changecount++;
  	Assert((beentry->st_changecount & 1) == 0);
***************
*** 2738,2743 **** pgstat_report_waiting(bool waiting)
--- 2744,2811 ----
  	beentry->st_waiting = waiting;
  }
  
+ /* --------
+  * pgstat_report_commsent() -
+  *
+  *    Tell the collector about data sent over a socket.
+  *    It is the caller's responsibility not invoke with a negative len
+  * --------
+  */
+ void
+ pgstat_report_commsent(int count)
+ {
+ 	volatile PgBackendStatus *beentry = MyBEEntry;
+ 	PgStat_MsgComm msg;
+ 
+ 	if (pgStatSock == PGINVALID_SOCKET || !pgstat_track_counts)
+ 		return;
+ 
+ 	/* this function can be called by the postmaster */
+ 	if (beentry != NULL) {
+ 		beentry->st_changecount++;
+ 		beentry->st_bytes_sent += count;
+ 		beentry->st_changecount++;
+ 		Assert((beentry->st_changecount & 1) == 0);
+ 	}
+ 
+ 	pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_COMMSENT);
+ 	/* MyDatabaseId might be invalid, we'll check it in the msg receiver */
+ 	msg.m_databaseid = MyDatabaseId;
+ 	msg.m_bytes_transferred = count;
+ 	pgstat_send(&msg, sizeof(msg));
+ }
+ 
+ /* --------
+  * pgstat_report_commreceived() -
+  *
+  *    Tell the collector about data received from a socket.
+  *    It is the caller's responsibility not invoke with a negative len
+  * --------
+  */
+ void
+ pgstat_report_commreceived(int count)
+ {
+ 	volatile PgBackendStatus *beentry = MyBEEntry;
+ 	PgStat_MsgComm msg;
+ 
+ 	if (pgStatSock == PGINVALID_SOCKET || !pgstat_track_counts)
+ 		return;
+ 
+ 	/* this function can be called by the postmaster */
+ 	if (beentry != NULL) {
+ 		beentry->st_changecount++;
+ 		beentry->st_bytes_received += count;
+ 		beentry->st_changecount++;
+ 		Assert((beentry->st_changecount & 1) == 0);
+ 	}
+ 
+ 	pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_COMMRECEIVED);
+ 	/* MyDatabaseId might be invalid, we'll check it in the msg receiver */
+ 	msg.m_databaseid = MyDatabaseId;
+ 	msg.m_bytes_transferred = count;
+ 	pgstat_send(&msg, sizeof(msg));
+ }
+ 
  
  /* ----------
   * pgstat_read_current_status() -
***************
*** 3290,3295 **** PgstatCollectorMain(int argc, char *argv[])
--- 3358,3371 ----
  					pgstat_recv_tempfile((PgStat_MsgTempFile *) &msg, len);
  					break;
  
+ 				case PGSTAT_MTYPE_COMMSENT:
+ 					pgstat_recv_commsent((PgStat_MsgComm *) &msg, len);
+ 					break;
+ 
+ 				case PGSTAT_MTYPE_COMMRECEIVED:
+ 					pgstat_recv_commreceived((PgStat_MsgComm *) &msg, len);
+ 					break;
+ 
  				default:
  					break;
  			}
***************
*** 3390,3395 **** reset_dbentry_counters(PgStat_StatDBEntry *dbentry)
--- 3466,3473 ----
  	dbentry->n_deadlocks = 0;
  	dbentry->n_block_read_time = 0;
  	dbentry->n_block_write_time = 0;
+ 	dbentry->n_bytes_sent = 0;
+ 	dbentry->n_bytes_received = 0;
  
  	dbentry->stat_reset_timestamp = GetCurrentTimestamp();
  	dbentry->stats_timestamp = 0;
***************
*** 3798,3803 **** pgstat_read_statsfiles(Oid onlydb, bool permanent, bool deep)
--- 3876,3882 ----
  	int32		format_id;
  	bool		found;
  	const char *statfile = permanent ? PGSTAT_STAT_PERMANENT_FILENAME : pgstat_stat_filename;
+ 	TimestampTz now;
  
  	/*
  	 * The tables will live in pgStatLocalContext.
***************
*** 3825,3831 **** pgstat_read_statsfiles(Oid onlydb, bool permanent, bool deep)
  	 * Set the current timestamp (will be kept only in case we can't load an
  	 * existing statsfile).
  	 */
! 	globalStats.stat_reset_timestamp = GetCurrentTimestamp();
  
  	/*
  	 * Try to open the stats file. If it doesn't exist, the backends simply
--- 3904,3912 ----
  	 * Set the current timestamp (will be kept only in case we can't load an
  	 * existing statsfile).
  	 */
! 	now = GetCurrentTimestamp();
! 	globalStats.bgwriter_stat_reset_timestamp = now;
! 	globalStats.socket_stat_reset_timestamp = now;
  
  	/*
  	 * Try to open the stats file. If it doesn't exist, the backends simply
***************
*** 4722,4730 **** pgstat_recv_resetsharedcounter(PgStat_MsgResetsharedcounter *msg, int len)
  {
  	if (msg->m_resettarget == RESET_BGWRITER)
  	{
  		/* Reset the global background writer statistics for the cluster. */
! 		memset(&globalStats, 0, sizeof(globalStats));
! 		globalStats.stat_reset_timestamp = GetCurrentTimestamp();
  	}
  
  	/*
--- 4803,4829 ----
  {
  	if (msg->m_resettarget == RESET_BGWRITER)
  	{
+ 		globalStats.stats_timestamp = 0;
  		/* Reset the global background writer statistics for the cluster. */
! 		globalStats.timed_checkpoints = 0;
! 		globalStats.requested_checkpoints = 0;
! 		globalStats.checkpoint_write_time = 0;
! 		globalStats.checkpoint_sync_time = 0;
! 		globalStats.buf_written_checkpoints = 0;
! 		globalStats.buf_written_clean = 0;
! 		globalStats.maxwritten_clean = 0;
! 		globalStats.buf_written_backend = 0;
! 		globalStats.buf_fsync_backend = 0;
! 		globalStats.buf_alloc = 0;
! 		globalStats.bgwriter_stat_reset_timestamp = GetCurrentTimestamp();
! 	}
! 	else if (msg->m_resettarget == RESET_SOCKET)
! 	{
! 		globalStats.stats_timestamp = 0;
! 		/* Reset the global socket transfer statistics for the cluster. */
! 		globalStats.bytes_sent = 0;
! 		globalStats.bytes_received = 0;
! 		globalStats.socket_stat_reset_timestamp = GetCurrentTimestamp();
  	}
  
  	/*
***************
*** 4951,4956 **** pgstat_recv_tempfile(PgStat_MsgTempFile *msg, int len)
--- 5050,5095 ----
  }
  
  /* ----------
+  * pgstat_recv_commsent() -
+  *
+  *    Process a COMMSENT message.
+  * ----------
+  */
+ static void
+ pgstat_recv_commsent(PgStat_MsgComm *msg, int len)
+ {
+ 	PgStat_StatDBEntry *dbentry;
+ 
+ 	globalStats.bytes_sent += msg->m_bytes_transferred;
+ 
+ 	/* can be called before we have connected to a specific database or by walsender */
+ 	if (OidIsValid(msg->m_databaseid)) {
+ 		dbentry = pgstat_get_db_entry(msg->m_databaseid, true);
+ 		dbentry->n_bytes_sent += msg->m_bytes_transferred;
+ 	}
+ }
+ 
+ /* ----------
+  * pgstat_recv_commreceived() -
+  *
+  *    Process a COMMRECEIVED message.
+  * ----------
+  */
+ static void
+ pgstat_recv_commreceived(PgStat_MsgComm *msg, int len)
+ {
+ 	PgStat_StatDBEntry *dbentry;
+ 
+ 	globalStats.bytes_received += msg->m_bytes_transferred;
+ 
+ 	/* can be called before we have connected to a specific database or by walsender */
+ 	if (OidIsValid(msg->m_databaseid)) {
+ 		dbentry = pgstat_get_db_entry(msg->m_databaseid, true);
+ 		dbentry->n_bytes_received += msg->m_bytes_transferred;
+ 	}
+ }
+ 
+ /* ----------
   * pgstat_recv_funcstat() -
   *
   *	Count what the backend has done.
*** a/src/backend/postmaster/postmaster.c
--- b/src/backend/postmaster/postmaster.c
***************
*** 1825,1830 **** retry1:
--- 1825,1832 ----
  					 errmsg("failed to send SSL negotiation response: %m")));
  			return STATUS_ERROR;	/* close the connection */
  		}
+ 		else
+ 			pgstat_report_commsent(1);
  
  #ifdef USE_SSL
  		if (SSLok == 'S' && secure_open_server(port) == -1)
***************
*** 3839,3844 **** report_fork_failure_to_client(Port *port, int errnum)
--- 3841,3849 ----
  	{
  		rc = send(port->sock, buffer, strlen(buffer) + 1, 0);
  	} while (rc < 0 && errno == EINTR);
+ 
+ 	if (rc > 0)
+ 		pgstat_report_commsent(rc);
  }
  
  
*** a/src/backend/utils/adt/pgstatfuncs.c
--- b/src/backend/utils/adt/pgstatfuncs.c
***************
*** 86,91 **** extern Datum pg_stat_get_db_temp_files(PG_FUNCTION_ARGS);
--- 86,93 ----
  extern Datum pg_stat_get_db_temp_bytes(PG_FUNCTION_ARGS);
  extern Datum pg_stat_get_db_blk_read_time(PG_FUNCTION_ARGS);
  extern Datum pg_stat_get_db_blk_write_time(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_db_bytes_sent(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_db_bytes_received(PG_FUNCTION_ARGS);
  
  extern Datum pg_stat_get_bgwriter_timed_checkpoints(PG_FUNCTION_ARGS);
  extern Datum pg_stat_get_bgwriter_requested_checkpoints(PG_FUNCTION_ARGS);
***************
*** 99,104 **** extern Datum pg_stat_get_buf_written_backend(PG_FUNCTION_ARGS);
--- 101,110 ----
  extern Datum pg_stat_get_buf_fsync_backend(PG_FUNCTION_ARGS);
  extern Datum pg_stat_get_buf_alloc(PG_FUNCTION_ARGS);
  
+ extern Datum pg_stat_get_bytes_sent(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_bytes_received(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_socket_stat_reset_time(PG_FUNCTION_ARGS);
+ 
  extern Datum pg_stat_get_xact_numscans(PG_FUNCTION_ARGS);
  extern Datum pg_stat_get_xact_tuples_returned(PG_FUNCTION_ARGS);
  extern Datum pg_stat_get_xact_tuples_fetched(PG_FUNCTION_ARGS);
***************
*** 534,540 **** pg_stat_get_activity(PG_FUNCTION_ARGS)
  
  		oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
  
! 		tupdesc = CreateTemplateTupleDesc(14, false);
  		TupleDescInitEntry(tupdesc, (AttrNumber) 1, "datid",
  						   OIDOID, -1, 0);
  		TupleDescInitEntry(tupdesc, (AttrNumber) 2, "pid",
--- 540,546 ----
  
  		oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
  
! 		tupdesc = CreateTemplateTupleDesc(16, false);
  		TupleDescInitEntry(tupdesc, (AttrNumber) 1, "datid",
  						   OIDOID, -1, 0);
  		TupleDescInitEntry(tupdesc, (AttrNumber) 2, "pid",
***************
*** 563,568 **** pg_stat_get_activity(PG_FUNCTION_ARGS)
--- 569,578 ----
  						   TEXTOID, -1, 0);
  		TupleDescInitEntry(tupdesc, (AttrNumber) 14, "client_port",
  						   INT4OID, -1, 0);
+ 		TupleDescInitEntry(tupdesc, (AttrNumber) 15, "bytes_sent",
+ 						   INT8OID, -1, 0);
+ 		TupleDescInitEntry(tupdesc, (AttrNumber) 16, "bytes_received",
+ 						   INT8OID, -1, 0);
  
  		funcctx->tuple_desc = BlessTupleDesc(tupdesc);
  
***************
*** 614,621 **** pg_stat_get_activity(PG_FUNCTION_ARGS)
  	if (funcctx->call_cntr < funcctx->max_calls)
  	{
  		/* for each row */
! 		Datum		values[14];
! 		bool		nulls[14];
  		HeapTuple	tuple;
  		PgBackendStatus *beentry;
  		SockAddr	zero_clientaddr;
--- 624,631 ----
  	if (funcctx->call_cntr < funcctx->max_calls)
  	{
  		/* for each row */
! 		Datum		values[16];
! 		bool		nulls[16];
  		HeapTuple	tuple;
  		PgBackendStatus *beentry;
  		SockAddr	zero_clientaddr;
***************
*** 773,778 **** pg_stat_get_activity(PG_FUNCTION_ARGS)
--- 783,790 ----
  					nulls[13] = true;
  				}
  			}
+ 			values[14] = Int64GetDatum(beentry->st_bytes_sent);
+ 			values[15] = Int64GetDatum(beentry->st_bytes_received);
  		}
  		else
  		{
***************
*** 787,792 **** pg_stat_get_activity(PG_FUNCTION_ARGS)
--- 799,806 ----
  			nulls[11] = true;
  			nulls[12] = true;
  			nulls[13] = true;
+ 			nulls[14] = true;
+ 			nulls[15] = true;
  		}
  
  		tuple = heap_form_tuple(funcctx->tuple_desc, values, nulls);
***************
*** 1407,1412 **** pg_stat_get_db_blk_write_time(PG_FUNCTION_ARGS)
--- 1421,1456 ----
  }
  
  Datum
+ pg_stat_get_db_bytes_sent(PG_FUNCTION_ARGS)
+ {
+ 	Oid			dbid = PG_GETARG_OID(0);
+ 	double		result;
+ 	PgStat_StatDBEntry *dbentry;
+ 
+ 	if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
+ 		result = 0;
+ 	else
+ 		result = dbentry->n_bytes_sent;
+ 
+ 	PG_RETURN_INT64(result);
+ }
+ 
+ Datum
+ pg_stat_get_db_bytes_received(PG_FUNCTION_ARGS)
+ {
+ 	Oid			dbid = PG_GETARG_OID(0);
+ 	double		result;
+ 	PgStat_StatDBEntry *dbentry;
+ 
+ 	if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
+ 		result = 0;
+ 	else
+ 		result = dbentry->n_bytes_received;
+ 
+ 	PG_RETURN_INT64(result);
+ }
+ 
+ Datum
  pg_stat_get_bgwriter_timed_checkpoints(PG_FUNCTION_ARGS)
  {
  	PG_RETURN_INT64(pgstat_fetch_global()->timed_checkpoints);
***************
*** 1453,1459 **** pg_stat_get_checkpoint_sync_time(PG_FUNCTION_ARGS)
  Datum
  pg_stat_get_bgwriter_stat_reset_time(PG_FUNCTION_ARGS)
  {
! 	PG_RETURN_TIMESTAMPTZ(pgstat_fetch_global()->stat_reset_timestamp);
  }
  
  Datum
--- 1497,1503 ----
  Datum
  pg_stat_get_bgwriter_stat_reset_time(PG_FUNCTION_ARGS)
  {
! 	PG_RETURN_TIMESTAMPTZ(pgstat_fetch_global()->bgwriter_stat_reset_timestamp);
  }
  
  Datum
***************
*** 1475,1480 **** pg_stat_get_buf_alloc(PG_FUNCTION_ARGS)
--- 1519,1542 ----
  }
  
  Datum
+ pg_stat_get_bytes_sent(PG_FUNCTION_ARGS)
+ {
+ 	PG_RETURN_INT64(pgstat_fetch_global()->bytes_sent);
+ }
+ 
+ Datum
+ pg_stat_get_bytes_received(PG_FUNCTION_ARGS)
+ {
+ 	PG_RETURN_INT64(pgstat_fetch_global()->bytes_received);
+ }
+ 
+ Datum
+ pg_stat_get_socket_stat_reset_time(PG_FUNCTION_ARGS)
+ {
+ 	PG_RETURN_TIMESTAMPTZ(pgstat_fetch_global()->socket_stat_reset_timestamp);
+ }
+ 
+ Datum
  pg_stat_get_xact_numscans(PG_FUNCTION_ARGS)
  {
  	Oid			relid = PG_GETARG_OID(0);
*** a/src/include/catalog/pg_proc.h
--- b/src/include/catalog/pg_proc.h
***************
*** 2626,2632 **** DATA(insert OID = 3057 ( pg_stat_get_autoanalyze_count PGNSP PGUID 12 1 0 0 0 f
  DESCR("statistics: number of auto analyzes for a table");
  DATA(insert OID = 1936 (  pg_stat_get_backend_idset		PGNSP PGUID 12 1 100 0 0 f f f f t t s 0 0 23 "" _null_ _null_ _null_ _null_ pg_stat_get_backend_idset _null_ _null_ _null_ ));
  DESCR("statistics: currently active backend IDs");
! DATA(insert OID = 2022 (  pg_stat_get_activity			PGNSP PGUID 12 1 100 0 0 f f f f f t s 1 0 2249 "23" "{23,26,23,26,25,25,25,16,1184,1184,1184,1184,869,25,23}" "{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o}" "{pid,datid,pid,usesysid,application_name,state,query,waiting,xact_start,query_start,backend_start,state_change,client_addr,client_hostname,client_port}" _null_ pg_stat_get_activity _null_ _null_ _null_ ));
  DESCR("statistics: information about currently active backends");
  DATA(insert OID = 3099 (  pg_stat_get_wal_senders	PGNSP PGUID 12 1 10 0 0 f f f f f t s 0 0 2249 "" "{23,25,25,25,25,25,23,25}" "{o,o,o,o,o,o,o,o}" "{pid,state,sent_location,write_location,flush_location,replay_location,sync_priority,sync_state}" _null_ pg_stat_get_wal_senders _null_ _null_ _null_ ));
  DESCR("statistics: information about currently active replication");
--- 2626,2632 ----
  DESCR("statistics: number of auto analyzes for a table");
  DATA(insert OID = 1936 (  pg_stat_get_backend_idset		PGNSP PGUID 12 1 100 0 0 f f f f t t s 0 0 23 "" _null_ _null_ _null_ _null_ pg_stat_get_backend_idset _null_ _null_ _null_ ));
  DESCR("statistics: currently active backend IDs");
! DATA(insert OID = 2022 (  pg_stat_get_activity			PGNSP PGUID 12 1 100 0 0 f f f f f t s 1 0 2249 "23" "{23,26,23,26,25,25,25,16,1184,1184,1184,1184,869,25,23,20,20}" "{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}" "{pid,datid,pid,usesysid,application_name,state,query,waiting,xact_start,query_start,backend_start,state_change,client_addr,client_hostname,client_port,bytes_sent,bytes_received}" _null_ pg_stat_get_activity _null_ _null_ _null_ ));
  DESCR("statistics: information about currently active backends");
  DATA(insert OID = 3099 (  pg_stat_get_wal_senders	PGNSP PGUID 12 1 10 0 0 f f f f f t s 0 0 2249 "" "{23,25,25,25,25,25,23,25}" "{o,o,o,o,o,o,o,o}" "{pid,state,sent_location,write_location,flush_location,replay_location,sync_priority,sync_state}" _null_ pg_stat_get_wal_senders _null_ _null_ _null_ ));
  DESCR("statistics: information about currently active replication");
***************
*** 2696,2701 **** DATA(insert OID = 2844 (  pg_stat_get_db_blk_read_time	PGNSP PGUID 12 1 0 0 0 f
--- 2696,2705 ----
  DESCR("statistics: block read time, in msec");
  DATA(insert OID = 2845 (  pg_stat_get_db_blk_write_time PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 701 "26" _null_ _null_ _null_ _null_ pg_stat_get_db_blk_write_time _null_ _null_ _null_ ));
  DESCR("statistics: block write time, in msec");
+ DATA(insert OID = 3195 (  pg_stat_get_db_bytes_sent PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_db_bytes_sent _null_ _null_ _null_ ));
+ DESCR("statistics: number of bytes sent over communication sockets");
+ DATA(insert OID = 3196 (  pg_stat_get_db_bytes_received PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_db_bytes_received _null_ _null_ _null_ ));
+ DESCR("statistics: number of bytes received over communication sockets");
  DATA(insert OID = 2769 ( pg_stat_get_bgwriter_timed_checkpoints PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 20 "" _null_ _null_ _null_ _null_ pg_stat_get_bgwriter_timed_checkpoints _null_ _null_ _null_ ));
  DESCR("statistics: number of timed checkpoints started by the bgwriter");
  DATA(insert OID = 2770 ( pg_stat_get_bgwriter_requested_checkpoints PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 20 "" _null_ _null_ _null_ _null_ pg_stat_get_bgwriter_requested_checkpoints _null_ _null_ _null_ ));
***************
*** 2719,2724 **** DESCR("statistics: number of backend buffer writes that did their own fsync");
--- 2723,2735 ----
  DATA(insert OID = 2859 ( pg_stat_get_buf_alloc			PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 20 "" _null_ _null_ _null_ _null_ pg_stat_get_buf_alloc _null_ _null_ _null_ ));
  DESCR("statistics: number of buffer allocations");
  
+ DATA(insert OID = 3197 ( pg_stat_get_bytes_sent PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 20 "" _null_ _null_ _null_ _null_ pg_stat_get_bytes_sent _null_ _null_ _null_ ));
+ DESCR("statistics: number of bytes sent over communication sockets");
+ DATA(insert OID = 3198 ( pg_stat_get_bytes_received PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 20 "" _null_ _null_ _null_ _null_ pg_stat_get_bytes_received _null_ _null_ _null_ ));
+ DESCR("statistics: number of bytes received over communication sockets");
+ DATA(insert OID = 3199 ( pg_stat_get_socket_stat_reset_time PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 1184 "" _null_ _null_ _null_ _null_	pg_stat_get_socket_stat_reset_time _null_ _null_ _null_ ));
+ DESCR("statistics: last reset for the communication socket statistics");
+ 
  DATA(insert OID = 2978 (  pg_stat_get_function_calls		PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_function_calls _null_ _null_ _null_ ));
  DESCR("statistics: number of function calls");
  DATA(insert OID = 2979 (  pg_stat_get_function_total_time	PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 701 "26" _null_ _null_ _null_ _null_ pg_stat_get_function_total_time _null_ _null_ _null_ ));
*** a/src/include/pgstat.h
--- b/src/include/pgstat.h
***************
*** 49,55 **** typedef enum StatMsgType
  	PGSTAT_MTYPE_FUNCPURGE,
  	PGSTAT_MTYPE_RECOVERYCONFLICT,
  	PGSTAT_MTYPE_TEMPFILE,
! 	PGSTAT_MTYPE_DEADLOCK
  } StatMsgType;
  
  /* ----------
--- 49,57 ----
  	PGSTAT_MTYPE_FUNCPURGE,
  	PGSTAT_MTYPE_RECOVERYCONFLICT,
  	PGSTAT_MTYPE_TEMPFILE,
! 	PGSTAT_MTYPE_DEADLOCK,
! 	PGSTAT_MTYPE_COMMSENT,
! 	PGSTAT_MTYPE_COMMRECEIVED
  } StatMsgType;
  
  /* ----------
***************
*** 102,108 **** typedef struct PgStat_TableCounts
  /* Possible targets for resetting cluster-wide shared values */
  typedef enum PgStat_Shared_Reset_Target
  {
! 	RESET_BGWRITER
  } PgStat_Shared_Reset_Target;
  
  /* Possible object types for resetting single counters */
--- 104,111 ----
  /* Possible targets for resetting cluster-wide shared values */
  typedef enum PgStat_Shared_Reset_Target
  {
! 	RESET_BGWRITER,
! 	RESET_SOCKET
  } PgStat_Shared_Reset_Target;
  
  /* Possible object types for resetting single counters */
***************
*** 397,402 **** typedef struct PgStat_MsgTempFile
--- 400,419 ----
  } PgStat_MsgTempFile;
  
  /* ----------
+   * PgStat_MsgComm
+   *
+   * Sent upon sending or receiving data over a communication socket (header determines direction)
+   * ----------
+   */
+ typedef struct PgStat_MsgComm
+ {
+ 	PgStat_MsgHdr	m_hdr;
+ 
+ 	Oid			m_databaseid;
+ 	int			m_bytes_transferred;
+ } PgStat_MsgComm;
+ 
+ /* ----------
   * PgStat_FunctionCounts	The actual per-function counts kept by a backend
   *
   * This struct should contain only actual event counters, because we memcmp
***************
*** 515,521 **** typedef union PgStat_Msg
   * ------------------------------------------------------------
   */
  
! #define PGSTAT_FILE_FORMAT_ID	0x01A5BC9B
  
  /* ----------
   * PgStat_StatDBEntry			The collector's data per database
--- 532,538 ----
   * ------------------------------------------------------------
   */
  
! #define PGSTAT_FILE_FORMAT_ID	0x01A5BC9C
  
  /* ----------
   * PgStat_StatDBEntry			The collector's data per database
***************
*** 544,549 **** typedef struct PgStat_StatDBEntry
--- 561,570 ----
  	PgStat_Counter n_deadlocks;
  	PgStat_Counter n_block_read_time;	/* times in microseconds */
  	PgStat_Counter n_block_write_time;
+ 	/* communication socket transfer counter in bytes (backend to client) */
+ 	PgStat_Counter n_bytes_sent;
+ 	/* communication socket transfer counter in bytes (client to backend) */
+ 	PgStat_Counter n_bytes_received;
  
  	TimestampTz stat_reset_timestamp;
  	TimestampTz stats_timestamp;	/* time of db stats file update */
***************
*** 614,619 **** typedef struct PgStat_StatFuncEntry
--- 635,641 ----
  typedef struct PgStat_GlobalStats
  {
  	TimestampTz stats_timestamp;	/* time of stats file update */
+ 	/* bgwriter stats */
  	PgStat_Counter timed_checkpoints;
  	PgStat_Counter requested_checkpoints;
  	PgStat_Counter checkpoint_write_time;		/* times in milliseconds */
***************
*** 624,630 **** typedef struct PgStat_GlobalStats
  	PgStat_Counter buf_written_backend;
  	PgStat_Counter buf_fsync_backend;
  	PgStat_Counter buf_alloc;
! 	TimestampTz stat_reset_timestamp;
  } PgStat_GlobalStats;
  
  
--- 646,656 ----
  	PgStat_Counter buf_written_backend;
  	PgStat_Counter buf_fsync_backend;
  	PgStat_Counter buf_alloc;
! 	TimestampTz bgwriter_stat_reset_timestamp;
! 	/* communication socket stats */
! 	PgStat_Counter bytes_sent; /* in bytes (cluster to client) */
! 	PgStat_Counter bytes_received; /* in bytes (client to cluster) */
! 	TimestampTz socket_stat_reset_timestamp;
  } PgStat_GlobalStats;
  
  
***************
*** 697,702 **** typedef struct PgBackendStatus
--- 723,734 ----
  
  	/* current command string; MUST be null-terminated */
  	char	   *st_activity;
+ 
+ 	/* communication socket transfer counter in bytes (backend to client) */
+ 	unsigned long st_bytes_sent;
+ 	/* communication socket transfer counter in bytes (client to backend) */
+ 	unsigned long st_bytes_received;
+ 
  } PgBackendStatus;
  
  /*
***************
*** 788,793 **** extern void pgstat_report_tempfile(size_t filesize);
--- 820,828 ----
  extern void pgstat_report_appname(const char *appname);
  extern void pgstat_report_xact_timestamp(TimestampTz tstamp);
  extern void pgstat_report_waiting(bool waiting);
+ extern void pgstat_report_commsent(int count);
+ extern void pgstat_report_commreceived(int count);
+ 
  extern const char *pgstat_get_backend_current_activity(int pid, bool checkUser);
  extern const char *pgstat_get_crashed_backend_activity(int pid, char *buffer,
  									int buflen);
*** a/src/test/regress/expected/rules.out
--- b/src/test/regress/expected/rules.out
***************
*** 1595,1603 **** pg_stat_activity| SELECT s.datid,
      s.state_change, 
      s.waiting, 
      s.state, 
!     s.query
     FROM pg_database d, 
!     pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, waiting, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port), 
      pg_authid u
    WHERE ((s.datid = d.oid) AND (s.usesysid = u.oid));
  pg_stat_all_indexes| SELECT c.oid AS relid, 
--- 1595,1605 ----
      s.state_change, 
      s.waiting, 
      s.state, 
!     s.query, 
!     s.bytes_sent, 
!     s.bytes_received
     FROM pg_database d, 
!     pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, waiting, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, bytes_sent, bytes_received), 
      pg_authid u
    WHERE ((s.datid = d.oid) AND (s.usesysid = u.oid));
  pg_stat_all_indexes| SELECT c.oid AS relid, 
***************
*** 1669,1674 **** pg_stat_database| SELECT d.oid AS datid,
--- 1671,1678 ----
      pg_stat_get_db_deadlocks(d.oid) AS deadlocks, 
      pg_stat_get_db_blk_read_time(d.oid) AS blk_read_time, 
      pg_stat_get_db_blk_write_time(d.oid) AS blk_write_time, 
+     pg_stat_get_db_bytes_sent(d.oid) AS bytes_sent, 
+     pg_stat_get_db_bytes_received(d.oid) AS bytes_received, 
      pg_stat_get_db_stat_reset_time(d.oid) AS stats_reset
     FROM pg_database d;
  pg_stat_database_conflicts| SELECT d.oid AS datid, 
***************
*** 1687,1692 **** pg_stat_replication| SELECT s.pid,
--- 1691,1698 ----
      s.client_hostname, 
      s.client_port, 
      s.backend_start, 
+     s.bytes_sent, 
+     s.bytes_received, 
      w.state, 
      w.sent_location, 
      w.write_location, 
***************
*** 1694,1700 **** pg_stat_replication| SELECT s.pid,
      w.replay_location, 
      w.sync_priority, 
      w.sync_state
!    FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, waiting, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port), 
      pg_authid u, 
      pg_stat_get_wal_senders() w(pid, state, sent_location, write_location, flush_location, replay_location, sync_priority, sync_state)
    WHERE ((s.usesysid = u.oid) AND (s.pid = w.pid));
--- 1700,1706 ----
      w.replay_location, 
      w.sync_priority, 
      w.sync_state
!    FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, waiting, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, bytes_sent, bytes_received), 
      pg_authid u, 
      pg_stat_get_wal_senders() w(pid, state, sent_location, write_location, flush_location, replay_location, sync_priority, sync_state)
    WHERE ((s.usesysid = u.oid) AND (s.pid = w.pid));
#13Greg Stark
stark@mit.edu
In reply to: Nigel Heron (#1)
Re: stats for network traffic WIP

On Mon, Oct 21, 2013 at 5:14 AM, Nigel Heron <nheron@querymetrics.com>wrote:

- can be used to find misbehaving connections.
- can be used in multi-user/multi-database clusters for resource usage
tracking.
- competing databases have such metrics.

The most interesting thing that I could see calculating from these stats
would require also knowing how much time was spent waiting on writes and
reads on the network. With the cumulative time spent as well as the count
of syscalls you can calculate the average latency over any time period
between two snapshots. However that would involve adding two gettimeofday
calls which would be quite likely to cause a noticeable impact on some
architectures. Unless there's already a pair of gettimeofday calls you can
piggy back onto?

--
greg

#14Nigel Heron
nheron@querymetrics.com
In reply to: Nigel Heron (#12)
1 attachment(s)
Re: stats for network traffic WIP

On Tue, Oct 29, 2013 at 11:26 AM, Nigel Heron <nheron@querymetrics.com> wrote:

So, for now, the counters only track sockets created from an inbound
(client to server) connection.

here's v3 of the patch (rebase and cleanup).

Hi,
here's v4 of the patch. I added documentation and a new global view
called "pg_stat_socket" (includes bytes_sent, bytes_received and
stats_reset time)

thanks,
-nigel.

Attachments:

netstats-v4.patchtext/x-patch; charset=US-ASCII; name=netstats-v4.patchDownload
*** a/doc/src/sgml/monitoring.sgml
--- b/doc/src/sgml/monitoring.sgml
***************
*** 278,283 **** postgres: <replaceable>user</> <replaceable>database</> <replaceable>host</> <re
--- 278,291 ----
       </row>
  
       <row>
+       <entry><structname>pg_stat_socket</><indexterm><primary>pg_stat_socket</primary></indexterm></entry>
+       <entry>One row only, showing statistics about the
+        cluster's communication socket activity. See
+        <xref linkend="pg-stat-socket-view"> for details.
+      </entry>
+      </row>
+ 
+      <row>
        <entry><structname>pg_stat_database</><indexterm><primary>pg_stat_database</primary></indexterm></entry>
        <entry>One row per database, showing database-wide statistics. See
         <xref linkend="pg-stat-database-view"> for details.
***************
*** 627,632 **** postgres: <replaceable>user</> <replaceable>database</> <replaceable>host</> <re
--- 635,650 ----
        that was executed.
       </entry>
      </row>
+     <row>
+      <entry><structfield>bytes_sent</></entry>
+      <entry><type>bigint</></entry>
+      <entry>Number of bytes sent over this backend's communication socket</entry>
+     </row>
+     <row>
+      <entry><structfield>bytes_received</></entry>
+      <entry><type>bigint</></entry>
+      <entry>Number of bytes received over this backend's communication socket</entry>
+     </row>
     </tbody>
     </tgroup>
    </table>
***************
*** 735,740 **** postgres: <replaceable>user</> <replaceable>database</> <replaceable>host</> <re
--- 753,801 ----
     single row, containing global data for the cluster.
    </para>
  
+   <table id="pg-stat-socket-view" xreflabel="pg_stat_socket">
+    <title><structname>pg_stat_socket</structname> View</title>
+ 
+    <tgroup cols="3">
+     <thead>
+     <row>
+       <entry>Column</entry>
+       <entry>Type</entry>
+       <entry>Description</entry>
+      </row>
+     </thead>
+ 
+     <tbody>
+      <row>
+       <entry><structfield>bytes_sent</></entry>
+       <entry><type>bigint</type></entry>
+       <entry>
+         Number of bytes sent over communication sockets.
+       </entry>
+      </row>
+      <row>
+       <entry><structfield>bytes_received</></entry>
+       <entry><type>bigint</type></entry>
+       <entry>
+         Number of bytes received over communication sockets.
+       </entry>
+      </row>
+      <row>
+       <entry><structfield>stats_reset</></entry>
+       <entry><type>timestamp with time zone</type></entry>
+       <entry>Time at which these statistics were last reset</entry>
+      </row>
+     </tbody>
+     </tgroup>
+   </table>
+ 
+   <para>
+    The <structname>pg_stat_socket</structname> view will always have a
+    single row, containing global data for the cluster.
+    Only sockets created from inbound client connections are tracked (Unix sockets and TCP).
+    Streaming replication traffic is counted on the master, but not on the slave (See <xref linkend="pg-stat-replication-view"> for details.)
+   </para>
+ 
    <table id="pg-stat-database-view" xreflabel="pg_stat_database">
     <title><structname>pg_stat_database</structname> View</title>
     <tgroup cols="3">
***************
*** 859,864 **** postgres: <replaceable>user</> <replaceable>database</> <replaceable>host</> <re
--- 920,935 ----
        in milliseconds</entry>
      </row>
      <row>
+      <entry><structfield>bytes_sent</></entry>
+      <entry><type>bigint</></entry>
+      <entry>Number of bytes sent over backend communication sockets in this database</entry>
+     </row>
+     <row>
+      <entry><structfield>bytes_received</></entry>
+      <entry><type>bigint</></entry>
+      <entry>Number of bytes received over this backend communication sockets in this database</entry>
+     </row>
+     <row>
       <entry><structfield>stats_reset</></entry>
       <entry><type>timestamp with time zone</></entry>
       <entry>Time at which these statistics were last reset</entry>
***************
*** 1417,1422 **** postgres: <replaceable>user</> <replaceable>database</> <replaceable>host</> <re
--- 1488,1503 ----
       </entry>
      </row>
      <row>
+      <entry><structfield>bytes_sent</></entry>
+      <entry><type>bigint</></entry>
+      <entry>Number of bytes sent over this WAL sender's communication socket</entry>
+     </row>
+     <row>
+      <entry><structfield>bytes_received</></entry>
+      <entry><type>bigint</></entry>
+      <entry>Number of bytes received over this WAL sender's communication socket</entry>
+     </row>
+     <row>
       <entry><structfield>state</></entry>
       <entry><type>text</></entry>
       <entry>Current WAL sender state</entry>
***************
*** 1613,1618 **** postgres: <replaceable>user</> <replaceable>database</> <replaceable>host</> <re
--- 1694,1701 ----
         argument (requires superuser privileges).
         Calling <literal>pg_stat_reset_shared('bgwriter')</> will zero all the
         counters shown in the <structname>pg_stat_bgwriter</> view.
+        Calling <literal>pg_stat_reset_shared('socket')</> will zero all the
+        counters shown in the <structname>pg_stat_socket</> view.
        </entry>
       </row>
  
*** a/src/backend/catalog/system_views.sql
--- b/src/backend/catalog/system_views.sql
***************
*** 586,592 **** CREATE VIEW pg_stat_activity AS
              S.state_change,
              S.waiting,
              S.state,
!             S.query
      FROM pg_database D, pg_stat_get_activity(NULL) AS S, pg_authid U
      WHERE S.datid = D.oid AND
              S.usesysid = U.oid;
--- 586,594 ----
              S.state_change,
              S.waiting,
              S.state,
!             S.query,
!             S.bytes_sent,
!             S.bytes_received
      FROM pg_database D, pg_stat_get_activity(NULL) AS S, pg_authid U
      WHERE S.datid = D.oid AND
              S.usesysid = U.oid;
***************
*** 601,606 **** CREATE VIEW pg_stat_replication AS
--- 603,610 ----
              S.client_hostname,
              S.client_port,
              S.backend_start,
+             S.bytes_sent,
+             S.bytes_received,
              W.state,
              W.sent_location,
              W.write_location,
***************
*** 634,639 **** CREATE VIEW pg_stat_database AS
--- 638,645 ----
              pg_stat_get_db_deadlocks(D.oid) AS deadlocks,
              pg_stat_get_db_blk_read_time(D.oid) AS blk_read_time,
              pg_stat_get_db_blk_write_time(D.oid) AS blk_write_time,
+             pg_stat_get_db_bytes_sent(D.oid) AS bytes_sent,
+             pg_stat_get_db_bytes_received(D.oid) AS bytes_received,
              pg_stat_get_db_stat_reset_time(D.oid) AS stats_reset
      FROM pg_database D;
  
***************
*** 686,691 **** CREATE VIEW pg_stat_bgwriter AS
--- 692,703 ----
          pg_stat_get_buf_alloc() AS buffers_alloc,
          pg_stat_get_bgwriter_stat_reset_time() AS stats_reset;
  
+ CREATE VIEW pg_stat_socket AS
+     SELECT
+         pg_stat_get_bytes_sent() AS bytes_sent,
+         pg_stat_get_bytes_received() AS bytes_received,
+         pg_stat_get_socket_stat_reset_time() AS stats_reset;
+ 
  CREATE VIEW pg_user_mappings AS
      SELECT
          U.oid       AS umid,
*** a/src/backend/libpq/be-secure.c
--- b/src/backend/libpq/be-secure.c
***************
*** 74,80 ****
  #include "libpq/libpq.h"
  #include "tcop/tcopprot.h"
  #include "utils/memutils.h"
! 
  
  #ifdef USE_SSL
  
--- 74,80 ----
  #include "libpq/libpq.h"
  #include "tcop/tcopprot.h"
  #include "utils/memutils.h"
! #include "pgstat.h"
  
  #ifdef USE_SSL
  
***************
*** 307,312 **** rloop:
--- 307,318 ----
  		n = recv(port->sock, ptr, len, 0);
  
  		client_read_ended();
+ 
+ 		if (n > 0)
+ 		{
+ 			/* we received data from the socket that needs to be reported */
+ 			pgstat_report_commreceived(n);
+ 		}
  	}
  
  	return n;
***************
*** 441,447 **** wloop:
--- 447,460 ----
  	}
  	else
  #endif
+ 	{
  		n = send(port->sock, ptr, len, 0);
+ 		if (n > 0)
+ 		{
+ 			/* we sent data over the socket that needs to be reported */
+ 			pgstat_report_commsent(n);
+ 		}
+ 	}
  
  	return n;
  }
***************
*** 488,493 **** my_sock_read(BIO *h, char *buf, int size)
--- 501,512 ----
  
  	client_read_ended();
  
+ 	if (res > 0)
+ 	{
+ 		/* we received data from the socket that needs to be reported */
+ 		pgstat_report_commreceived(res);
+ 	}
+ 
  	return res;
  }
  
***************
*** 504,509 **** my_sock_write(BIO *h, const char *buf, int size)
--- 523,533 ----
  			BIO_set_retry_write(h);
  		}
  	}
+ 	else
+ 	{
+ 		/* we sent data over the socket that needs to be reported */
+ 		pgstat_report_commsent(res);
+ 	}
  
  	return res;
  }
*** a/src/backend/postmaster/pgstat.c
--- b/src/backend/postmaster/pgstat.c
***************
*** 298,303 **** static void pgstat_recv_funcpurge(PgStat_MsgFuncpurge *msg, int len);
--- 298,305 ----
  static void pgstat_recv_recoveryconflict(PgStat_MsgRecoveryConflict *msg, int len);
  static void pgstat_recv_deadlock(PgStat_MsgDeadlock *msg, int len);
  static void pgstat_recv_tempfile(PgStat_MsgTempFile *msg, int len);
+ static void pgstat_recv_commsent(PgStat_MsgComm *msg, int len);
+ static void pgstat_recv_commreceived(PgStat_MsgComm *msg, int len);
  
  /* ------------------------------------------------------------
   * Public functions called from postmaster follow
***************
*** 1249,1259 **** pgstat_reset_shared_counters(const char *target)
  
  	if (strcmp(target, "bgwriter") == 0)
  		msg.m_resettarget = RESET_BGWRITER;
  	else
  		ereport(ERROR,
  				(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
  				 errmsg("unrecognized reset target: \"%s\"", target),
! 				 errhint("Target must be \"bgwriter\".")));
  
  	pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETSHAREDCOUNTER);
  	pgstat_send(&msg, sizeof(msg));
--- 1251,1263 ----
  
  	if (strcmp(target, "bgwriter") == 0)
  		msg.m_resettarget = RESET_BGWRITER;
+ 	else if (strcmp(target, "socket") == 0)
+ 		msg.m_resettarget = RESET_SOCKET;
  	else
  		ereport(ERROR,
  				(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
  				 errmsg("unrecognized reset target: \"%s\"", target),
! 				 errhint("Target must be \"bgwriter\" or \"socket\".")));
  
  	pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETSHAREDCOUNTER);
  	pgstat_send(&msg, sizeof(msg));
***************
*** 2531,2536 **** pgstat_bestart(void)
--- 2535,2542 ----
  	beentry->st_clienthostname[NAMEDATALEN - 1] = '\0';
  	beentry->st_appname[NAMEDATALEN - 1] = '\0';
  	beentry->st_activity[pgstat_track_activity_query_size - 1] = '\0';
+ 	beentry->st_bytes_sent = 0;
+ 	beentry->st_bytes_received = 0;
  
  	beentry->st_changecount++;
  	Assert((beentry->st_changecount & 1) == 0);
***************
*** 2738,2743 **** pgstat_report_waiting(bool waiting)
--- 2744,2811 ----
  	beentry->st_waiting = waiting;
  }
  
+ /* --------
+  * pgstat_report_commsent() -
+  *
+  *    Tell the collector about data sent over a socket.
+  *    It is the caller's responsibility not invoke with a negative len
+  * --------
+  */
+ void
+ pgstat_report_commsent(int count)
+ {
+ 	volatile PgBackendStatus *beentry = MyBEEntry;
+ 	PgStat_MsgComm msg;
+ 
+ 	if (pgStatSock == PGINVALID_SOCKET || !pgstat_track_counts)
+ 		return;
+ 
+ 	/* this function can be called by the postmaster */
+ 	if (beentry != NULL) {
+ 		beentry->st_changecount++;
+ 		beentry->st_bytes_sent += count;
+ 		beentry->st_changecount++;
+ 		Assert((beentry->st_changecount & 1) == 0);
+ 	}
+ 
+ 	pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_COMMSENT);
+ 	/* MyDatabaseId might be invalid, we'll check it in the msg receiver */
+ 	msg.m_databaseid = MyDatabaseId;
+ 	msg.m_bytes_transferred = count;
+ 	pgstat_send(&msg, sizeof(msg));
+ }
+ 
+ /* --------
+  * pgstat_report_commreceived() -
+  *
+  *    Tell the collector about data received from a socket.
+  *    It is the caller's responsibility not invoke with a negative len
+  * --------
+  */
+ void
+ pgstat_report_commreceived(int count)
+ {
+ 	volatile PgBackendStatus *beentry = MyBEEntry;
+ 	PgStat_MsgComm msg;
+ 
+ 	if (pgStatSock == PGINVALID_SOCKET || !pgstat_track_counts)
+ 		return;
+ 
+ 	/* this function can be called by the postmaster */
+ 	if (beentry != NULL) {
+ 		beentry->st_changecount++;
+ 		beentry->st_bytes_received += count;
+ 		beentry->st_changecount++;
+ 		Assert((beentry->st_changecount & 1) == 0);
+ 	}
+ 
+ 	pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_COMMRECEIVED);
+ 	/* MyDatabaseId might be invalid, we'll check it in the msg receiver */
+ 	msg.m_databaseid = MyDatabaseId;
+ 	msg.m_bytes_transferred = count;
+ 	pgstat_send(&msg, sizeof(msg));
+ }
+ 
  
  /* ----------
   * pgstat_read_current_status() -
***************
*** 3290,3295 **** PgstatCollectorMain(int argc, char *argv[])
--- 3358,3371 ----
  					pgstat_recv_tempfile((PgStat_MsgTempFile *) &msg, len);
  					break;
  
+ 				case PGSTAT_MTYPE_COMMSENT:
+ 					pgstat_recv_commsent((PgStat_MsgComm *) &msg, len);
+ 					break;
+ 
+ 				case PGSTAT_MTYPE_COMMRECEIVED:
+ 					pgstat_recv_commreceived((PgStat_MsgComm *) &msg, len);
+ 					break;
+ 
  				default:
  					break;
  			}
***************
*** 3390,3395 **** reset_dbentry_counters(PgStat_StatDBEntry *dbentry)
--- 3466,3473 ----
  	dbentry->n_deadlocks = 0;
  	dbentry->n_block_read_time = 0;
  	dbentry->n_block_write_time = 0;
+ 	dbentry->n_bytes_sent = 0;
+ 	dbentry->n_bytes_received = 0;
  
  	dbentry->stat_reset_timestamp = GetCurrentTimestamp();
  	dbentry->stats_timestamp = 0;
***************
*** 3798,3803 **** pgstat_read_statsfiles(Oid onlydb, bool permanent, bool deep)
--- 3876,3882 ----
  	int32		format_id;
  	bool		found;
  	const char *statfile = permanent ? PGSTAT_STAT_PERMANENT_FILENAME : pgstat_stat_filename;
+ 	TimestampTz now;
  
  	/*
  	 * The tables will live in pgStatLocalContext.
***************
*** 3825,3831 **** pgstat_read_statsfiles(Oid onlydb, bool permanent, bool deep)
  	 * Set the current timestamp (will be kept only in case we can't load an
  	 * existing statsfile).
  	 */
! 	globalStats.stat_reset_timestamp = GetCurrentTimestamp();
  
  	/*
  	 * Try to open the stats file. If it doesn't exist, the backends simply
--- 3904,3912 ----
  	 * Set the current timestamp (will be kept only in case we can't load an
  	 * existing statsfile).
  	 */
! 	now = GetCurrentTimestamp();
! 	globalStats.bgwriter_stat_reset_timestamp = now;
! 	globalStats.socket_stat_reset_timestamp = now;
  
  	/*
  	 * Try to open the stats file. If it doesn't exist, the backends simply
***************
*** 4722,4730 **** pgstat_recv_resetsharedcounter(PgStat_MsgResetsharedcounter *msg, int len)
  {
  	if (msg->m_resettarget == RESET_BGWRITER)
  	{
  		/* Reset the global background writer statistics for the cluster. */
! 		memset(&globalStats, 0, sizeof(globalStats));
! 		globalStats.stat_reset_timestamp = GetCurrentTimestamp();
  	}
  
  	/*
--- 4803,4829 ----
  {
  	if (msg->m_resettarget == RESET_BGWRITER)
  	{
+ 		globalStats.stats_timestamp = 0;
  		/* Reset the global background writer statistics for the cluster. */
! 		globalStats.timed_checkpoints = 0;
! 		globalStats.requested_checkpoints = 0;
! 		globalStats.checkpoint_write_time = 0;
! 		globalStats.checkpoint_sync_time = 0;
! 		globalStats.buf_written_checkpoints = 0;
! 		globalStats.buf_written_clean = 0;
! 		globalStats.maxwritten_clean = 0;
! 		globalStats.buf_written_backend = 0;
! 		globalStats.buf_fsync_backend = 0;
! 		globalStats.buf_alloc = 0;
! 		globalStats.bgwriter_stat_reset_timestamp = GetCurrentTimestamp();
! 	}
! 	else if (msg->m_resettarget == RESET_SOCKET)
! 	{
! 		globalStats.stats_timestamp = 0;
! 		/* Reset the global socket transfer statistics for the cluster. */
! 		globalStats.bytes_sent = 0;
! 		globalStats.bytes_received = 0;
! 		globalStats.socket_stat_reset_timestamp = GetCurrentTimestamp();
  	}
  
  	/*
***************
*** 4951,4956 **** pgstat_recv_tempfile(PgStat_MsgTempFile *msg, int len)
--- 5050,5095 ----
  }
  
  /* ----------
+  * pgstat_recv_commsent() -
+  *
+  *    Process a COMMSENT message.
+  * ----------
+  */
+ static void
+ pgstat_recv_commsent(PgStat_MsgComm *msg, int len)
+ {
+ 	PgStat_StatDBEntry *dbentry;
+ 
+ 	globalStats.bytes_sent += msg->m_bytes_transferred;
+ 
+ 	/* can be called before we have connected to a specific database or by walsender */
+ 	if (OidIsValid(msg->m_databaseid)) {
+ 		dbentry = pgstat_get_db_entry(msg->m_databaseid, true);
+ 		dbentry->n_bytes_sent += msg->m_bytes_transferred;
+ 	}
+ }
+ 
+ /* ----------
+  * pgstat_recv_commreceived() -
+  *
+  *    Process a COMMRECEIVED message.
+  * ----------
+  */
+ static void
+ pgstat_recv_commreceived(PgStat_MsgComm *msg, int len)
+ {
+ 	PgStat_StatDBEntry *dbentry;
+ 
+ 	globalStats.bytes_received += msg->m_bytes_transferred;
+ 
+ 	/* can be called before we have connected to a specific database or by walsender */
+ 	if (OidIsValid(msg->m_databaseid)) {
+ 		dbentry = pgstat_get_db_entry(msg->m_databaseid, true);
+ 		dbentry->n_bytes_received += msg->m_bytes_transferred;
+ 	}
+ }
+ 
+ /* ----------
   * pgstat_recv_funcstat() -
   *
   *	Count what the backend has done.
*** a/src/backend/postmaster/postmaster.c
--- b/src/backend/postmaster/postmaster.c
***************
*** 1825,1830 **** retry1:
--- 1825,1832 ----
  					 errmsg("failed to send SSL negotiation response: %m")));
  			return STATUS_ERROR;	/* close the connection */
  		}
+ 		else
+ 			pgstat_report_commsent(1);
  
  #ifdef USE_SSL
  		if (SSLok == 'S' && secure_open_server(port) == -1)
***************
*** 3839,3844 **** report_fork_failure_to_client(Port *port, int errnum)
--- 3841,3849 ----
  	{
  		rc = send(port->sock, buffer, strlen(buffer) + 1, 0);
  	} while (rc < 0 && errno == EINTR);
+ 
+ 	if (rc > 0)
+ 		pgstat_report_commsent(rc);
  }
  
  
*** a/src/backend/utils/adt/pgstatfuncs.c
--- b/src/backend/utils/adt/pgstatfuncs.c
***************
*** 86,91 **** extern Datum pg_stat_get_db_temp_files(PG_FUNCTION_ARGS);
--- 86,93 ----
  extern Datum pg_stat_get_db_temp_bytes(PG_FUNCTION_ARGS);
  extern Datum pg_stat_get_db_blk_read_time(PG_FUNCTION_ARGS);
  extern Datum pg_stat_get_db_blk_write_time(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_db_bytes_sent(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_db_bytes_received(PG_FUNCTION_ARGS);
  
  extern Datum pg_stat_get_bgwriter_timed_checkpoints(PG_FUNCTION_ARGS);
  extern Datum pg_stat_get_bgwriter_requested_checkpoints(PG_FUNCTION_ARGS);
***************
*** 99,104 **** extern Datum pg_stat_get_buf_written_backend(PG_FUNCTION_ARGS);
--- 101,110 ----
  extern Datum pg_stat_get_buf_fsync_backend(PG_FUNCTION_ARGS);
  extern Datum pg_stat_get_buf_alloc(PG_FUNCTION_ARGS);
  
+ extern Datum pg_stat_get_bytes_sent(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_bytes_received(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_socket_stat_reset_time(PG_FUNCTION_ARGS);
+ 
  extern Datum pg_stat_get_xact_numscans(PG_FUNCTION_ARGS);
  extern Datum pg_stat_get_xact_tuples_returned(PG_FUNCTION_ARGS);
  extern Datum pg_stat_get_xact_tuples_fetched(PG_FUNCTION_ARGS);
***************
*** 534,540 **** pg_stat_get_activity(PG_FUNCTION_ARGS)
  
  		oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
  
! 		tupdesc = CreateTemplateTupleDesc(14, false);
  		TupleDescInitEntry(tupdesc, (AttrNumber) 1, "datid",
  						   OIDOID, -1, 0);
  		TupleDescInitEntry(tupdesc, (AttrNumber) 2, "pid",
--- 540,546 ----
  
  		oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
  
! 		tupdesc = CreateTemplateTupleDesc(16, false);
  		TupleDescInitEntry(tupdesc, (AttrNumber) 1, "datid",
  						   OIDOID, -1, 0);
  		TupleDescInitEntry(tupdesc, (AttrNumber) 2, "pid",
***************
*** 563,568 **** pg_stat_get_activity(PG_FUNCTION_ARGS)
--- 569,578 ----
  						   TEXTOID, -1, 0);
  		TupleDescInitEntry(tupdesc, (AttrNumber) 14, "client_port",
  						   INT4OID, -1, 0);
+ 		TupleDescInitEntry(tupdesc, (AttrNumber) 15, "bytes_sent",
+ 						   INT8OID, -1, 0);
+ 		TupleDescInitEntry(tupdesc, (AttrNumber) 16, "bytes_received",
+ 						   INT8OID, -1, 0);
  
  		funcctx->tuple_desc = BlessTupleDesc(tupdesc);
  
***************
*** 614,621 **** pg_stat_get_activity(PG_FUNCTION_ARGS)
  	if (funcctx->call_cntr < funcctx->max_calls)
  	{
  		/* for each row */
! 		Datum		values[14];
! 		bool		nulls[14];
  		HeapTuple	tuple;
  		PgBackendStatus *beentry;
  		SockAddr	zero_clientaddr;
--- 624,631 ----
  	if (funcctx->call_cntr < funcctx->max_calls)
  	{
  		/* for each row */
! 		Datum		values[16];
! 		bool		nulls[16];
  		HeapTuple	tuple;
  		PgBackendStatus *beentry;
  		SockAddr	zero_clientaddr;
***************
*** 773,778 **** pg_stat_get_activity(PG_FUNCTION_ARGS)
--- 783,790 ----
  					nulls[13] = true;
  				}
  			}
+ 			values[14] = Int64GetDatum(beentry->st_bytes_sent);
+ 			values[15] = Int64GetDatum(beentry->st_bytes_received);
  		}
  		else
  		{
***************
*** 787,792 **** pg_stat_get_activity(PG_FUNCTION_ARGS)
--- 799,806 ----
  			nulls[11] = true;
  			nulls[12] = true;
  			nulls[13] = true;
+ 			nulls[14] = true;
+ 			nulls[15] = true;
  		}
  
  		tuple = heap_form_tuple(funcctx->tuple_desc, values, nulls);
***************
*** 1407,1412 **** pg_stat_get_db_blk_write_time(PG_FUNCTION_ARGS)
--- 1421,1456 ----
  }
  
  Datum
+ pg_stat_get_db_bytes_sent(PG_FUNCTION_ARGS)
+ {
+ 	Oid			dbid = PG_GETARG_OID(0);
+ 	double		result;
+ 	PgStat_StatDBEntry *dbentry;
+ 
+ 	if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
+ 		result = 0;
+ 	else
+ 		result = dbentry->n_bytes_sent;
+ 
+ 	PG_RETURN_INT64(result);
+ }
+ 
+ Datum
+ pg_stat_get_db_bytes_received(PG_FUNCTION_ARGS)
+ {
+ 	Oid			dbid = PG_GETARG_OID(0);
+ 	double		result;
+ 	PgStat_StatDBEntry *dbentry;
+ 
+ 	if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
+ 		result = 0;
+ 	else
+ 		result = dbentry->n_bytes_received;
+ 
+ 	PG_RETURN_INT64(result);
+ }
+ 
+ Datum
  pg_stat_get_bgwriter_timed_checkpoints(PG_FUNCTION_ARGS)
  {
  	PG_RETURN_INT64(pgstat_fetch_global()->timed_checkpoints);
***************
*** 1453,1459 **** pg_stat_get_checkpoint_sync_time(PG_FUNCTION_ARGS)
  Datum
  pg_stat_get_bgwriter_stat_reset_time(PG_FUNCTION_ARGS)
  {
! 	PG_RETURN_TIMESTAMPTZ(pgstat_fetch_global()->stat_reset_timestamp);
  }
  
  Datum
--- 1497,1503 ----
  Datum
  pg_stat_get_bgwriter_stat_reset_time(PG_FUNCTION_ARGS)
  {
! 	PG_RETURN_TIMESTAMPTZ(pgstat_fetch_global()->bgwriter_stat_reset_timestamp);
  }
  
  Datum
***************
*** 1475,1480 **** pg_stat_get_buf_alloc(PG_FUNCTION_ARGS)
--- 1519,1542 ----
  }
  
  Datum
+ pg_stat_get_bytes_sent(PG_FUNCTION_ARGS)
+ {
+ 	PG_RETURN_INT64(pgstat_fetch_global()->bytes_sent);
+ }
+ 
+ Datum
+ pg_stat_get_bytes_received(PG_FUNCTION_ARGS)
+ {
+ 	PG_RETURN_INT64(pgstat_fetch_global()->bytes_received);
+ }
+ 
+ Datum
+ pg_stat_get_socket_stat_reset_time(PG_FUNCTION_ARGS)
+ {
+ 	PG_RETURN_TIMESTAMPTZ(pgstat_fetch_global()->socket_stat_reset_timestamp);
+ }
+ 
+ Datum
  pg_stat_get_xact_numscans(PG_FUNCTION_ARGS)
  {
  	Oid			relid = PG_GETARG_OID(0);
*** a/src/include/catalog/pg_proc.h
--- b/src/include/catalog/pg_proc.h
***************
*** 2626,2632 **** DATA(insert OID = 3057 ( pg_stat_get_autoanalyze_count PGNSP PGUID 12 1 0 0 0 f
  DESCR("statistics: number of auto analyzes for a table");
  DATA(insert OID = 1936 (  pg_stat_get_backend_idset		PGNSP PGUID 12 1 100 0 0 f f f f t t s 0 0 23 "" _null_ _null_ _null_ _null_ pg_stat_get_backend_idset _null_ _null_ _null_ ));
  DESCR("statistics: currently active backend IDs");
! DATA(insert OID = 2022 (  pg_stat_get_activity			PGNSP PGUID 12 1 100 0 0 f f f f f t s 1 0 2249 "23" "{23,26,23,26,25,25,25,16,1184,1184,1184,1184,869,25,23}" "{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o}" "{pid,datid,pid,usesysid,application_name,state,query,waiting,xact_start,query_start,backend_start,state_change,client_addr,client_hostname,client_port}" _null_ pg_stat_get_activity _null_ _null_ _null_ ));
  DESCR("statistics: information about currently active backends");
  DATA(insert OID = 3099 (  pg_stat_get_wal_senders	PGNSP PGUID 12 1 10 0 0 f f f f f t s 0 0 2249 "" "{23,25,25,25,25,25,23,25}" "{o,o,o,o,o,o,o,o}" "{pid,state,sent_location,write_location,flush_location,replay_location,sync_priority,sync_state}" _null_ pg_stat_get_wal_senders _null_ _null_ _null_ ));
  DESCR("statistics: information about currently active replication");
--- 2626,2632 ----
  DESCR("statistics: number of auto analyzes for a table");
  DATA(insert OID = 1936 (  pg_stat_get_backend_idset		PGNSP PGUID 12 1 100 0 0 f f f f t t s 0 0 23 "" _null_ _null_ _null_ _null_ pg_stat_get_backend_idset _null_ _null_ _null_ ));
  DESCR("statistics: currently active backend IDs");
! DATA(insert OID = 2022 (  pg_stat_get_activity			PGNSP PGUID 12 1 100 0 0 f f f f f t s 1 0 2249 "23" "{23,26,23,26,25,25,25,16,1184,1184,1184,1184,869,25,23,20,20}" "{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}" "{pid,datid,pid,usesysid,application_name,state,query,waiting,xact_start,query_start,backend_start,state_change,client_addr,client_hostname,client_port,bytes_sent,bytes_received}" _null_ pg_stat_get_activity _null_ _null_ _null_ ));
  DESCR("statistics: information about currently active backends");
  DATA(insert OID = 3099 (  pg_stat_get_wal_senders	PGNSP PGUID 12 1 10 0 0 f f f f f t s 0 0 2249 "" "{23,25,25,25,25,25,23,25}" "{o,o,o,o,o,o,o,o}" "{pid,state,sent_location,write_location,flush_location,replay_location,sync_priority,sync_state}" _null_ pg_stat_get_wal_senders _null_ _null_ _null_ ));
  DESCR("statistics: information about currently active replication");
***************
*** 2696,2701 **** DATA(insert OID = 2844 (  pg_stat_get_db_blk_read_time	PGNSP PGUID 12 1 0 0 0 f
--- 2696,2705 ----
  DESCR("statistics: block read time, in msec");
  DATA(insert OID = 2845 (  pg_stat_get_db_blk_write_time PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 701 "26" _null_ _null_ _null_ _null_ pg_stat_get_db_blk_write_time _null_ _null_ _null_ ));
  DESCR("statistics: block write time, in msec");
+ DATA(insert OID = 3195 (  pg_stat_get_db_bytes_sent PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_db_bytes_sent _null_ _null_ _null_ ));
+ DESCR("statistics: number of bytes sent over communication sockets");
+ DATA(insert OID = 3196 (  pg_stat_get_db_bytes_received PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_db_bytes_received _null_ _null_ _null_ ));
+ DESCR("statistics: number of bytes received over communication sockets");
  DATA(insert OID = 2769 ( pg_stat_get_bgwriter_timed_checkpoints PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 20 "" _null_ _null_ _null_ _null_ pg_stat_get_bgwriter_timed_checkpoints _null_ _null_ _null_ ));
  DESCR("statistics: number of timed checkpoints started by the bgwriter");
  DATA(insert OID = 2770 ( pg_stat_get_bgwriter_requested_checkpoints PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 20 "" _null_ _null_ _null_ _null_ pg_stat_get_bgwriter_requested_checkpoints _null_ _null_ _null_ ));
***************
*** 2719,2724 **** DESCR("statistics: number of backend buffer writes that did their own fsync");
--- 2723,2735 ----
  DATA(insert OID = 2859 ( pg_stat_get_buf_alloc			PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 20 "" _null_ _null_ _null_ _null_ pg_stat_get_buf_alloc _null_ _null_ _null_ ));
  DESCR("statistics: number of buffer allocations");
  
+ DATA(insert OID = 3197 ( pg_stat_get_bytes_sent PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 20 "" _null_ _null_ _null_ _null_ pg_stat_get_bytes_sent _null_ _null_ _null_ ));
+ DESCR("statistics: number of bytes sent over communication sockets");
+ DATA(insert OID = 3198 ( pg_stat_get_bytes_received PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 20 "" _null_ _null_ _null_ _null_ pg_stat_get_bytes_received _null_ _null_ _null_ ));
+ DESCR("statistics: number of bytes received over communication sockets");
+ DATA(insert OID = 3199 ( pg_stat_get_socket_stat_reset_time PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 1184 "" _null_ _null_ _null_ _null_	pg_stat_get_socket_stat_reset_time _null_ _null_ _null_ ));
+ DESCR("statistics: last reset for the communication socket statistics");
+ 
  DATA(insert OID = 2978 (  pg_stat_get_function_calls		PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_function_calls _null_ _null_ _null_ ));
  DESCR("statistics: number of function calls");
  DATA(insert OID = 2979 (  pg_stat_get_function_total_time	PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 701 "26" _null_ _null_ _null_ _null_ pg_stat_get_function_total_time _null_ _null_ _null_ ));
*** a/src/include/pgstat.h
--- b/src/include/pgstat.h
***************
*** 49,55 **** typedef enum StatMsgType
  	PGSTAT_MTYPE_FUNCPURGE,
  	PGSTAT_MTYPE_RECOVERYCONFLICT,
  	PGSTAT_MTYPE_TEMPFILE,
! 	PGSTAT_MTYPE_DEADLOCK
  } StatMsgType;
  
  /* ----------
--- 49,57 ----
  	PGSTAT_MTYPE_FUNCPURGE,
  	PGSTAT_MTYPE_RECOVERYCONFLICT,
  	PGSTAT_MTYPE_TEMPFILE,
! 	PGSTAT_MTYPE_DEADLOCK,
! 	PGSTAT_MTYPE_COMMSENT,
! 	PGSTAT_MTYPE_COMMRECEIVED
  } StatMsgType;
  
  /* ----------
***************
*** 102,108 **** typedef struct PgStat_TableCounts
  /* Possible targets for resetting cluster-wide shared values */
  typedef enum PgStat_Shared_Reset_Target
  {
! 	RESET_BGWRITER
  } PgStat_Shared_Reset_Target;
  
  /* Possible object types for resetting single counters */
--- 104,111 ----
  /* Possible targets for resetting cluster-wide shared values */
  typedef enum PgStat_Shared_Reset_Target
  {
! 	RESET_BGWRITER,
! 	RESET_SOCKET
  } PgStat_Shared_Reset_Target;
  
  /* Possible object types for resetting single counters */
***************
*** 397,402 **** typedef struct PgStat_MsgTempFile
--- 400,419 ----
  } PgStat_MsgTempFile;
  
  /* ----------
+   * PgStat_MsgComm
+   *
+   * Sent upon sending or receiving data over a communication socket (header determines direction)
+   * ----------
+   */
+ typedef struct PgStat_MsgComm
+ {
+ 	PgStat_MsgHdr	m_hdr;
+ 
+ 	Oid			m_databaseid;
+ 	int			m_bytes_transferred;
+ } PgStat_MsgComm;
+ 
+ /* ----------
   * PgStat_FunctionCounts	The actual per-function counts kept by a backend
   *
   * This struct should contain only actual event counters, because we memcmp
***************
*** 515,521 **** typedef union PgStat_Msg
   * ------------------------------------------------------------
   */
  
! #define PGSTAT_FILE_FORMAT_ID	0x01A5BC9B
  
  /* ----------
   * PgStat_StatDBEntry			The collector's data per database
--- 532,538 ----
   * ------------------------------------------------------------
   */
  
! #define PGSTAT_FILE_FORMAT_ID	0x01A5BC9C
  
  /* ----------
   * PgStat_StatDBEntry			The collector's data per database
***************
*** 544,549 **** typedef struct PgStat_StatDBEntry
--- 561,570 ----
  	PgStat_Counter n_deadlocks;
  	PgStat_Counter n_block_read_time;	/* times in microseconds */
  	PgStat_Counter n_block_write_time;
+ 	/* communication socket transfer counter in bytes (backend to client) */
+ 	PgStat_Counter n_bytes_sent;
+ 	/* communication socket transfer counter in bytes (client to backend) */
+ 	PgStat_Counter n_bytes_received;
  
  	TimestampTz stat_reset_timestamp;
  	TimestampTz stats_timestamp;	/* time of db stats file update */
***************
*** 614,619 **** typedef struct PgStat_StatFuncEntry
--- 635,641 ----
  typedef struct PgStat_GlobalStats
  {
  	TimestampTz stats_timestamp;	/* time of stats file update */
+ 	/* bgwriter stats */
  	PgStat_Counter timed_checkpoints;
  	PgStat_Counter requested_checkpoints;
  	PgStat_Counter checkpoint_write_time;		/* times in milliseconds */
***************
*** 624,630 **** typedef struct PgStat_GlobalStats
  	PgStat_Counter buf_written_backend;
  	PgStat_Counter buf_fsync_backend;
  	PgStat_Counter buf_alloc;
! 	TimestampTz stat_reset_timestamp;
  } PgStat_GlobalStats;
  
  
--- 646,656 ----
  	PgStat_Counter buf_written_backend;
  	PgStat_Counter buf_fsync_backend;
  	PgStat_Counter buf_alloc;
! 	TimestampTz bgwriter_stat_reset_timestamp;
! 	/* communication socket stats */
! 	PgStat_Counter bytes_sent; /* in bytes (cluster to client) */
! 	PgStat_Counter bytes_received; /* in bytes (client to cluster) */
! 	TimestampTz socket_stat_reset_timestamp;
  } PgStat_GlobalStats;
  
  
***************
*** 697,702 **** typedef struct PgBackendStatus
--- 723,734 ----
  
  	/* current command string; MUST be null-terminated */
  	char	   *st_activity;
+ 
+ 	/* communication socket transfer counter in bytes (backend to client) */
+ 	unsigned long st_bytes_sent;
+ 	/* communication socket transfer counter in bytes (client to backend) */
+ 	unsigned long st_bytes_received;
+ 
  } PgBackendStatus;
  
  /*
***************
*** 788,793 **** extern void pgstat_report_tempfile(size_t filesize);
--- 820,828 ----
  extern void pgstat_report_appname(const char *appname);
  extern void pgstat_report_xact_timestamp(TimestampTz tstamp);
  extern void pgstat_report_waiting(bool waiting);
+ extern void pgstat_report_commsent(int count);
+ extern void pgstat_report_commreceived(int count);
+ 
  extern const char *pgstat_get_backend_current_activity(int pid, bool checkUser);
  extern const char *pgstat_get_crashed_backend_activity(int pid, char *buffer,
  									int buflen);
*** a/src/test/regress/expected/rules.out
--- b/src/test/regress/expected/rules.out
***************
*** 1595,1603 **** pg_stat_activity| SELECT s.datid,
      s.state_change, 
      s.waiting, 
      s.state, 
!     s.query
     FROM pg_database d, 
!     pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, waiting, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port), 
      pg_authid u
    WHERE ((s.datid = d.oid) AND (s.usesysid = u.oid));
  pg_stat_all_indexes| SELECT c.oid AS relid, 
--- 1595,1605 ----
      s.state_change, 
      s.waiting, 
      s.state, 
!     s.query, 
!     s.bytes_sent, 
!     s.bytes_received
     FROM pg_database d, 
!     pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, waiting, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, bytes_sent, bytes_received), 
      pg_authid u
    WHERE ((s.datid = d.oid) AND (s.usesysid = u.oid));
  pg_stat_all_indexes| SELECT c.oid AS relid, 
***************
*** 1669,1674 **** pg_stat_database| SELECT d.oid AS datid,
--- 1671,1678 ----
      pg_stat_get_db_deadlocks(d.oid) AS deadlocks, 
      pg_stat_get_db_blk_read_time(d.oid) AS blk_read_time, 
      pg_stat_get_db_blk_write_time(d.oid) AS blk_write_time, 
+     pg_stat_get_db_bytes_sent(d.oid) AS bytes_sent, 
+     pg_stat_get_db_bytes_received(d.oid) AS bytes_received, 
      pg_stat_get_db_stat_reset_time(d.oid) AS stats_reset
     FROM pg_database d;
  pg_stat_database_conflicts| SELECT d.oid AS datid, 
***************
*** 1687,1692 **** pg_stat_replication| SELECT s.pid,
--- 1691,1698 ----
      s.client_hostname, 
      s.client_port, 
      s.backend_start, 
+     s.bytes_sent, 
+     s.bytes_received, 
      w.state, 
      w.sent_location, 
      w.write_location, 
***************
*** 1694,1703 **** pg_stat_replication| SELECT s.pid,
      w.replay_location, 
      w.sync_priority, 
      w.sync_state
!    FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, waiting, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port), 
      pg_authid u, 
      pg_stat_get_wal_senders() w(pid, state, sent_location, write_location, flush_location, replay_location, sync_priority, sync_state)
    WHERE ((s.usesysid = u.oid) AND (s.pid = w.pid));
  pg_stat_sys_indexes| SELECT pg_stat_all_indexes.relid, 
      pg_stat_all_indexes.indexrelid, 
      pg_stat_all_indexes.schemaname, 
--- 1700,1712 ----
      w.replay_location, 
      w.sync_priority, 
      w.sync_state
!    FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, waiting, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, bytes_sent, bytes_received), 
      pg_authid u, 
      pg_stat_get_wal_senders() w(pid, state, sent_location, write_location, flush_location, replay_location, sync_priority, sync_state)
    WHERE ((s.usesysid = u.oid) AND (s.pid = w.pid));
+ pg_stat_socket| SELECT pg_stat_get_bytes_sent() AS bytes_sent, 
+     pg_stat_get_bytes_received() AS bytes_received, 
+     pg_stat_get_socket_stat_reset_time() AS stats_reset;
  pg_stat_sys_indexes| SELECT pg_stat_all_indexes.relid, 
      pg_stat_all_indexes.indexrelid, 
      pg_stat_all_indexes.schemaname, 
#15Mike Blackwell
mike.blackwell@rrd.com
In reply to: Nigel Heron (#14)
Re: stats for network traffic WIP

Patch applies and builds against git HEAD (as of 6790e738031089d5). "make
check" runs cleanly as well.

The new features appear to work as advertised as far as I've been able to
check.

The code looks good as far as I can see. Documentation patches are
included for the new features.

Still to be tested:
the counts for streaming replication (no replication setup here to test
against yet).

__________________________________________________________________________________
*Mike Blackwell | Technical Analyst, Distribution Services/Rollout
Management | RR Donnelley*
1750 Wallace Ave | St Charles, IL 60174-3401
Office: 630.313.7818
Mike.Blackwell@rrd.com
http://www.rrdonnelley.com

<http://www.rrdonnelley.com/&gt;
* <Mike.Blackwell@rrd.com>*

On Fri, Nov 8, 2013 at 9:01 AM, Nigel Heron <nheron@querymetrics.com> wrote:

Show quoted text

On Tue, Oct 29, 2013 at 11:26 AM, Nigel Heron <nheron@querymetrics.com>
wrote:

So, for now, the counters only track sockets created from an inbound
(client to server) connection.

here's v3 of the patch (rebase and cleanup).

Hi,
here's v4 of the patch. I added documentation and a new global view
called "pg_stat_socket" (includes bytes_sent, bytes_received and
stats_reset time)

thanks,
-nigel.

#16Mike Blackwell
mike.blackwell@rrd.com
In reply to: Mike Blackwell (#15)
Re: stats for network traffic WIP

Also still to be tested: performance impact.

__________________________________________________________________________________
*Mike Blackwell | Technical Analyst, Distribution Services/Rollout
Management | RR Donnelley*
1750 Wallace Ave | St Charles, IL 60174-3401
Office: 630.313.7818
Mike.Blackwell@rrd.com
http://www.rrdonnelley.com

<http://www.rrdonnelley.com/&gt;
* <Mike.Blackwell@rrd.com>*

On Fri, Nov 8, 2013 at 9:33 AM, Mike Blackwell <mike.blackwell@rrd.com>wrote:

Show quoted text

Patch applies and builds against git HEAD (as of 6790e738031089d5). "make
check" runs cleanly as well.

The new features appear to work as advertised as far as I've been able to
check.

The code looks good as far as I can see. Documentation patches are
included for the new features.

Still to be tested:
the counts for streaming replication (no replication setup here to test
against yet).

__________________________________________________________________________________
*Mike Blackwell | Technical Analyst, Distribution Services/Rollout
Management | RR Donnelley*
1750 Wallace Ave | St Charles, IL 60174-3401
Office: 630.313.7818
Mike.Blackwell@rrd.com
http://www.rrdonnelley.com

<http://www.rrdonnelley.com/&gt;
* <Mike.Blackwell@rrd.com>*

On Fri, Nov 8, 2013 at 9:01 AM, Nigel Heron <nheron@querymetrics.com>wrote:

On Tue, Oct 29, 2013 at 11:26 AM, Nigel Heron <nheron@querymetrics.com>
wrote:

So, for now, the counters only track sockets created from an inbound
(client to server) connection.

here's v3 of the patch (rebase and cleanup).

Hi,
here's v4 of the patch. I added documentation and a new global view
called "pg_stat_socket" (includes bytes_sent, bytes_received and
stats_reset time)

thanks,
-nigel.

#17Nigel Heron
nheron@querymetrics.com
In reply to: Greg Stark (#13)
Re: stats for network traffic WIP

On Thu, Nov 7, 2013 at 8:21 PM, Greg Stark <stark@mit.edu> wrote:

The most interesting thing that I could see calculating from these stats
would require also knowing how much time was spent waiting on writes and
reads on the network. With the cumulative time spent as well as the count of
syscalls you can calculate the average latency over any time period between
two snapshots. However that would involve adding two gettimeofday calls
which would be quite likely to cause a noticeable impact on some
architectures. Unless there's already a pair of gettimeofday calls you can
piggy back onto?

Adding timing instrumentation to each send() and recv() would require
over 50 calls to gettimeofday for a simple psql -c "SELECT 1", while
the client was waiting. That would add ~40usec extra time (estimated
using pg_test_timing on my laptop without TSC). It might be more
overhead than it's worth.

-nigel.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#18Peter Eisentraut
peter_e@gmx.net
In reply to: Nigel Heron (#14)
Re: stats for network traffic WIP

On Fri, 2013-11-08 at 10:01 -0500, Nigel Heron wrote:

here's v4 of the patch. I added documentation and a new global view
called "pg_stat_socket" (includes bytes_sent, bytes_received and
stats_reset time)

Your patch needs to be rebased:

CONFLICT (content): Merge conflict in src/test/regress/expected/rules.out

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#19Nigel Heron
nheron@querymetrics.com
In reply to: Peter Eisentraut (#18)
1 attachment(s)
Re: stats for network traffic WIP

On Wed, Nov 13, 2013 at 11:27 PM, Peter Eisentraut <peter_e@gmx.net> wrote:

On Fri, 2013-11-08 at 10:01 -0500, Nigel Heron wrote:

here's v4 of the patch. I added documentation and a new global view
called "pg_stat_socket" (includes bytes_sent, bytes_received and
stats_reset time)

Your patch needs to be rebased:

CONFLICT (content): Merge conflict in src/test/regress/expected/rules.out

Hi,
here's a rebased patch with some additions.

an overview of it's current state...

a new pg_stat_socket global view:
- total bytes sent and received
- bytes sent and received for user backends
- bytes sent and received for wal senders
- total connection attempts
- successful connections to user backends
- successful connections to wal senders
- stats reset time
pg_stat_reset_shared('socket') resets the counters

added to pg_stat_database view:
- bytes sent and received per db
- successful connections per db
pg_stat_reset() resets the counters

added to pg_stat_activity view:
- bytes sent and received per backend

added to pg_stat_replication view:
- bytes sent and received per wal sender

using the existing track_counts guc to enable/disable these stats.
-nigel.

Attachments:

netstats-v5.patchtext/x-patch; charset=US-ASCII; name=netstats-v5.patchDownload
*** a/doc/src/sgml/monitoring.sgml
--- b/doc/src/sgml/monitoring.sgml
***************
*** 278,283 **** postgres: <replaceable>user</> <replaceable>database</> <replaceable>host</> <re
--- 278,291 ----
       </row>
  
       <row>
+       <entry><structname>pg_stat_socket</><indexterm><primary>pg_stat_socket</primary></indexterm></entry>
+       <entry>One row only, showing statistics about the
+        cluster's communication socket activity. See
+        <xref linkend="pg-stat-socket-view"> for details.
+      </entry>
+      </row>
+ 
+      <row>
        <entry><structname>pg_stat_database</><indexterm><primary>pg_stat_database</primary></indexterm></entry>
        <entry>One row per database, showing database-wide statistics. See
         <xref linkend="pg-stat-database-view"> for details.
***************
*** 627,632 **** postgres: <replaceable>user</> <replaceable>database</> <replaceable>host</> <re
--- 635,650 ----
        that was executed.
       </entry>
      </row>
+     <row>
+      <entry><structfield>bytes_sent</></entry>
+      <entry><type>bigint</></entry>
+      <entry>Number of bytes sent over this backend's client connection</entry>
+     </row>
+     <row>
+      <entry><structfield>bytes_received</></entry>
+      <entry><type>bigint</></entry>
+      <entry>Number of bytes received over this backend's client connection</entry>
+     </row>
     </tbody>
     </tgroup>
    </table>
***************
*** 735,740 **** postgres: <replaceable>user</> <replaceable>database</> <replaceable>host</> <re
--- 753,850 ----
     single row, containing global data for the cluster.
    </para>
  
+   <table id="pg-stat-socket-view" xreflabel="pg_stat_socket">
+    <title><structname>pg_stat_socket</structname> View</title>
+ 
+    <tgroup cols="3">
+     <thead>
+     <row>
+       <entry>Column</entry>
+       <entry>Type</entry>
+       <entry>Description</entry>
+      </row>
+     </thead>
+ 
+     <tbody>
+      <row>
+       <entry><structfield>bytes_sent_total</></entry>
+       <entry><type>bigint</type></entry>
+       <entry>
+         Number of bytes sent over client connections (includes user backends and WAL senders).
+       </entry>
+      </row>
+      <row>
+       <entry><structfield>bytes_received_total</></entry>
+       <entry><type>bigint</type></entry>
+       <entry>
+         Number of bytes received over client connections (includes user backends and WAL senders).
+       </entry>
+      </row>
+      <row>
+       <entry><structfield>bytes_sent_backend</></entry>
+       <entry><type>bigint</type></entry>
+       <entry>
+         Number of bytes sent over client connections to a user backend.
+       </entry>
+      </row>
+      <row>
+       <entry><structfield>bytes_received_backend</></entry>
+       <entry><type>bigint</type></entry>
+       <entry>
+         Number of bytes received over client connections by a user backend.
+       </entry>
+      </row>
+      <row>
+       <entry><structfield>bytes_sent_walsender</></entry>
+       <entry><type>bigint</type></entry>
+       <entry>
+         Number of bytes sent over client connections to a WAL sender.
+       </entry>
+      </row>
+      <row>
+       <entry><structfield>bytes_received_walsender</></entry>
+       <entry><type>bigint</type></entry>
+       <entry>
+         Number of bytes received over client connections by a WAL sender.
+       </entry>
+      </row>
+      <row>
+       <entry><structfield>conn_received</></entry>
+       <entry><type>bigint</type></entry>
+       <entry>
+         Number of client connections received.
+       </entry>
+      </row>
+      <row>
+       <entry><structfield>conn_backend</></entry>
+       <entry><type>bigint</type></entry>
+       <entry>
+         Number of client connections successfully established to a user backend.
+       </entry>
+      </row>
+      <row>
+       <entry><structfield>conn_walsender</></entry>
+       <entry><type>bigint</type></entry>
+       <entry>
+         Number of client connections successfully established to a WAL sender.
+       </entry>
+      </row>
+      <row>
+       <entry><structfield>stats_reset</></entry>
+       <entry><type>timestamp with time zone</type></entry>
+       <entry>Time at which these statistics were last reset</entry>
+      </row>
+     </tbody>
+     </tgroup>
+   </table>
+ 
+   <para>
+    The <structname>pg_stat_socket</structname> view will always have a
+    single row, containing global data for the cluster.
+    Only sockets created from inbound client connections are tracked (Unix sockets and TCP).
+    Streaming replication traffic is counted on the master, but not on the slave (See <xref linkend="pg-stat-replication-view"> for details.)
+   </para>
+ 
    <table id="pg-stat-database-view" xreflabel="pg_stat_database">
     <title><structname>pg_stat_database</structname> View</title>
     <tgroup cols="3">
***************
*** 859,864 **** postgres: <replaceable>user</> <replaceable>database</> <replaceable>host</> <re
--- 969,989 ----
        in milliseconds</entry>
      </row>
      <row>
+      <entry><structfield>bytes_sent</></entry>
+      <entry><type>bigint</></entry>
+      <entry>Number of bytes sent over connections to user backends in this database</entry>
+     </row>
+     <row>
+      <entry><structfield>bytes_received</></entry>
+      <entry><type>bigint</></entry>
+      <entry>Number of bytes received over connections to user backends in this database</entry>
+     </row>
+     <row>
+      <entry><structfield>connections</></entry>
+      <entry><type>bigint</></entry>
+      <entry>Number of client connections successfully established in this database</entry>
+     </row>
+     <row>
       <entry><structfield>stats_reset</></entry>
       <entry><type>timestamp with time zone</></entry>
       <entry>Time at which these statistics were last reset</entry>
***************
*** 1417,1422 **** postgres: <replaceable>user</> <replaceable>database</> <replaceable>host</> <re
--- 1542,1557 ----
       </entry>
      </row>
      <row>
+      <entry><structfield>bytes_sent</></entry>
+      <entry><type>bigint</></entry>
+      <entry>Number of bytes sent over this WAL sender's connection</entry>
+     </row>
+     <row>
+      <entry><structfield>bytes_received</></entry>
+      <entry><type>bigint</></entry>
+      <entry>Number of bytes received over this WAL sender's connection</entry>
+     </row>
+     <row>
       <entry><structfield>state</></entry>
       <entry><type>text</></entry>
       <entry>Current WAL sender state</entry>
***************
*** 1613,1618 **** postgres: <replaceable>user</> <replaceable>database</> <replaceable>host</> <re
--- 1748,1755 ----
         argument (requires superuser privileges).
         Calling <literal>pg_stat_reset_shared('bgwriter')</> will zero all the
         counters shown in the <structname>pg_stat_bgwriter</> view.
+        Calling <literal>pg_stat_reset_shared('socket')</> will zero all the
+        counters shown in the <structname>pg_stat_socket</> view.
        </entry>
       </row>
  
*** a/src/backend/catalog/system_views.sql
--- b/src/backend/catalog/system_views.sql
***************
*** 586,592 **** CREATE VIEW pg_stat_activity AS
              S.state_change,
              S.waiting,
              S.state,
!             S.query
      FROM pg_database D, pg_stat_get_activity(NULL) AS S, pg_authid U
      WHERE S.datid = D.oid AND
              S.usesysid = U.oid;
--- 586,594 ----
              S.state_change,
              S.waiting,
              S.state,
!             S.query,
!             S.bytes_sent,
!             S.bytes_received
      FROM pg_database D, pg_stat_get_activity(NULL) AS S, pg_authid U
      WHERE S.datid = D.oid AND
              S.usesysid = U.oid;
***************
*** 601,606 **** CREATE VIEW pg_stat_replication AS
--- 603,610 ----
              S.client_hostname,
              S.client_port,
              S.backend_start,
+             S.bytes_sent,
+             S.bytes_received,
              W.state,
              W.sent_location,
              W.write_location,
***************
*** 634,639 **** CREATE VIEW pg_stat_database AS
--- 638,646 ----
              pg_stat_get_db_deadlocks(D.oid) AS deadlocks,
              pg_stat_get_db_blk_read_time(D.oid) AS blk_read_time,
              pg_stat_get_db_blk_write_time(D.oid) AS blk_write_time,
+             pg_stat_get_db_bytes_sent(D.oid) AS bytes_sent,
+             pg_stat_get_db_bytes_received(D.oid) AS bytes_received,
+             pg_stat_get_db_connections(D.oid) AS connections,
              pg_stat_get_db_stat_reset_time(D.oid) AS stats_reset
      FROM pg_database D;
  
***************
*** 686,691 **** CREATE VIEW pg_stat_bgwriter AS
--- 693,711 ----
          pg_stat_get_buf_alloc() AS buffers_alloc,
          pg_stat_get_bgwriter_stat_reset_time() AS stats_reset;
  
+ CREATE VIEW pg_stat_socket AS
+     SELECT
+         pg_stat_get_bytes_sent() AS bytes_sent_total,
+         pg_stat_get_bytes_received() AS bytes_received_total,
+         pg_stat_get_bytes_sent_backend() AS bytes_sent_backend,
+         pg_stat_get_bytes_received_backend() AS bytes_received_backend,
+         pg_stat_get_bytes_sent_walsender() AS bytes_sent_walsender,
+         pg_stat_get_bytes_received_walsender() AS bytes_received_walsender,
+         pg_stat_get_conn_received() AS conn_received,
+         pg_stat_get_conn_backend() AS conn_backend,
+         pg_stat_get_conn_walsender() AS conn_walsender,
+         pg_stat_get_socket_stat_reset_time() AS stats_reset;
+ 
  CREATE VIEW pg_user_mappings AS
      SELECT
          U.oid       AS umid,
*** a/src/backend/libpq/be-secure.c
--- b/src/backend/libpq/be-secure.c
***************
*** 74,80 ****
  #include "libpq/libpq.h"
  #include "tcop/tcopprot.h"
  #include "utils/memutils.h"
! 
  
  #ifdef USE_SSL
  
--- 74,80 ----
  #include "libpq/libpq.h"
  #include "tcop/tcopprot.h"
  #include "utils/memutils.h"
! #include "pgstat.h"
  
  #ifdef USE_SSL
  
***************
*** 307,312 **** rloop:
--- 307,318 ----
  		n = recv(port->sock, ptr, len, 0);
  
  		client_read_ended();
+ 
+ 		if (n > 0)
+ 		{
+ 			/* we received data from the socket that needs to be reported */
+ 			pgstat_report_bytesreceived(n);
+ 		}
  	}
  
  	return n;
***************
*** 441,447 **** wloop:
--- 447,460 ----
  	}
  	else
  #endif
+ 	{
  		n = send(port->sock, ptr, len, 0);
+ 		if (n > 0)
+ 		{
+ 			/* we sent data over the socket that needs to be reported */
+ 			pgstat_report_bytessent(n);
+ 		}
+ 	}
  
  	return n;
  }
***************
*** 488,493 **** my_sock_read(BIO *h, char *buf, int size)
--- 501,512 ----
  
  	client_read_ended();
  
+ 	if (res > 0)
+ 	{
+ 		/* we received data from the socket that needs to be reported */
+ 		pgstat_report_bytesreceived(res);
+ 	}
+ 
  	return res;
  }
  
***************
*** 504,509 **** my_sock_write(BIO *h, const char *buf, int size)
--- 523,533 ----
  			BIO_set_retry_write(h);
  		}
  	}
+ 	else
+ 	{
+ 		/* we sent data over the socket that needs to be reported */
+ 		pgstat_report_bytessent(res);
+ 	}
  
  	return res;
  }
*** a/src/backend/postmaster/pgstat.c
--- b/src/backend/postmaster/pgstat.c
***************
*** 48,53 ****
--- 48,54 ----
  #include "postmaster/autovacuum.h"
  #include "postmaster/fork_process.h"
  #include "postmaster/postmaster.h"
+ #include "replication/walsender.h"
  #include "storage/backendid.h"
  #include "storage/fd.h"
  #include "storage/ipc.h"
***************
*** 298,303 **** static void pgstat_recv_funcpurge(PgStat_MsgFuncpurge *msg, int len);
--- 299,308 ----
  static void pgstat_recv_recoveryconflict(PgStat_MsgRecoveryConflict *msg, int len);
  static void pgstat_recv_deadlock(PgStat_MsgDeadlock *msg, int len);
  static void pgstat_recv_tempfile(PgStat_MsgTempFile *msg, int len);
+ static void pgstat_recv_bytessent(PgStat_MsgBytesTransferred *msg, int len);
+ static void pgstat_recv_bytesreceived(PgStat_MsgBytesTransferred *msg, int len);
+ static void pgstat_recv_connreceived(PgStat_MsgConnReceived *msg, int len);
+ static void pgstat_recv_connsucceeded(PgStat_MsgConnSucceeded *msg, int len);
  
  /* ------------------------------------------------------------
   * Public functions called from postmaster follow
***************
*** 1249,1259 **** pgstat_reset_shared_counters(const char *target)
  
  	if (strcmp(target, "bgwriter") == 0)
  		msg.m_resettarget = RESET_BGWRITER;
  	else
  		ereport(ERROR,
  				(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
  				 errmsg("unrecognized reset target: \"%s\"", target),
! 				 errhint("Target must be \"bgwriter\".")));
  
  	pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETSHAREDCOUNTER);
  	pgstat_send(&msg, sizeof(msg));
--- 1254,1266 ----
  
  	if (strcmp(target, "bgwriter") == 0)
  		msg.m_resettarget = RESET_BGWRITER;
+ 	else if (strcmp(target, "socket") == 0)
+ 		msg.m_resettarget = RESET_SOCKET;
  	else
  		ereport(ERROR,
  				(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
  				 errmsg("unrecognized reset target: \"%s\"", target),
! 				 errhint("Target must be \"bgwriter\" or \"socket\".")));
  
  	pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETSHAREDCOUNTER);
  	pgstat_send(&msg, sizeof(msg));
***************
*** 2531,2536 **** pgstat_bestart(void)
--- 2538,2545 ----
  	beentry->st_clienthostname[NAMEDATALEN - 1] = '\0';
  	beentry->st_appname[NAMEDATALEN - 1] = '\0';
  	beentry->st_activity[pgstat_track_activity_query_size - 1] = '\0';
+ 	beentry->st_bytes_sent = 0;
+ 	beentry->st_bytes_received = 0;
  
  	beentry->st_changecount++;
  	Assert((beentry->st_changecount & 1) == 0);
***************
*** 2541,2546 **** pgstat_bestart(void)
--- 2550,2558 ----
  	/* Update app name to current GUC setting */
  	if (application_name)
  		pgstat_report_appname(application_name);
+ 
+ 	if (MyProcPort)
+ 		pgstat_report_connsucceeded();
  }
  
  /*
***************
*** 2738,2743 **** pgstat_report_waiting(bool waiting)
--- 2750,2857 ----
  	beentry->st_waiting = waiting;
  }
  
+ /* --------
+  * pgstat_report_bytessent() -
+  *
+  *    Tell the collector about data sent over a socket.
+  *    It is the caller's responsibility not invoke with a negative len
+  * --------
+  */
+ void
+ pgstat_report_bytessent(int count)
+ {
+ 	volatile PgBackendStatus *beentry = MyBEEntry;
+ 	PgStat_MsgBytesTransferred msg;
+ 
+ 	if (pgStatSock == PGINVALID_SOCKET || !pgstat_track_counts)
+ 		return;
+ 
+ 	/* this function can be called by the postmaster */
+ 	if (beentry != NULL) {
+ 		beentry->st_changecount++;
+ 		beentry->st_bytes_sent += count;
+ 		beentry->st_changecount++;
+ 		Assert((beentry->st_changecount & 1) == 0);
+ 	}
+ 
+ 	pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_BYTESSENT);
+ 	/* MyDatabaseId might be invalid, we'll check it in the msg receiver */
+ 	msg.m_databaseid = MyDatabaseId;
+ 	msg.m_bytes_transferred = count;
+ 	msg.m_walsender = am_walsender;
+ 	pgstat_send(&msg, sizeof(msg));
+ }
+ 
+ /* --------
+  * pgstat_report_bytesreceived() -
+  *
+  *    Tell the collector about data received from a socket.
+  *    It is the caller's responsibility not invoke with a negative len
+  * --------
+  */
+ void
+ pgstat_report_bytesreceived(int count)
+ {
+ 	volatile PgBackendStatus *beentry = MyBEEntry;
+ 	PgStat_MsgBytesTransferred msg;
+ 
+ 	if (pgStatSock == PGINVALID_SOCKET || !pgstat_track_counts)
+ 		return;
+ 
+ 	/* this function can be called by the postmaster */
+ 	if (beentry != NULL) {
+ 		beentry->st_changecount++;
+ 		beentry->st_bytes_received += count;
+ 		beentry->st_changecount++;
+ 		Assert((beentry->st_changecount & 1) == 0);
+ 	}
+ 
+ 	pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_BYTESRECEIVED);
+ 	/* MyDatabaseId might be invalid, we'll check it in the msg receiver */
+ 	msg.m_databaseid = MyDatabaseId;
+ 	msg.m_bytes_transferred = count;
+ 	msg.m_walsender = am_walsender;
+ 	pgstat_send(&msg, sizeof(msg));
+ }
+ 
+ /* --------
+  * pgstat_report_connreceived() -
+  *
+  *    Tell the collector about a client connection that was received.
+  * --------
+  */
+ void
+ pgstat_report_connreceived(void)
+ {
+ 	PgStat_MsgConnReceived msg;
+ 
+ 	if (pgStatSock == PGINVALID_SOCKET || !pgstat_track_counts)
+ 		return;
+ 
+ 	pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_CONNRECEIVED);
+ 	pgstat_send(&msg, sizeof(msg));
+ }
+ 
+ /* --------
+  * pgstat_report_connsucceeded() -
+  *
+  *    Tell the collector about a client connection that was successfully established.
+  * --------
+  */
+ void
+ pgstat_report_connsucceeded(void)
+ {
+ 	PgStat_MsgConnSucceeded msg;
+ 
+ 	if (pgStatSock == PGINVALID_SOCKET || !pgstat_track_counts)
+ 		return;
+ 
+ 	pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_CONNSUCCEEDED);
+ 	/* MyDatabaseId might be invalid, we'll check it in the msg receiver */
+ 	msg.m_databaseid = MyDatabaseId;
+ 	msg.m_walsender = am_walsender;
+ 	pgstat_send(&msg, sizeof(msg));
+ }
  
  /* ----------
   * pgstat_read_current_status() -
***************
*** 3290,3295 **** PgstatCollectorMain(int argc, char *argv[])
--- 3404,3425 ----
  					pgstat_recv_tempfile((PgStat_MsgTempFile *) &msg, len);
  					break;
  
+ 				case PGSTAT_MTYPE_BYTESSENT:
+ 					pgstat_recv_bytessent((PgStat_MsgBytesTransferred *) &msg, len);
+ 					break;
+ 
+ 				case PGSTAT_MTYPE_BYTESRECEIVED:
+ 					pgstat_recv_bytesreceived((PgStat_MsgBytesTransferred *) &msg, len);
+ 					break;
+ 
+ 				case PGSTAT_MTYPE_CONNRECEIVED:
+ 					pgstat_recv_connreceived((PgStat_MsgConnReceived *) &msg, len);
+ 					break;
+ 
+ 				case PGSTAT_MTYPE_CONNSUCCEEDED:
+ 					pgstat_recv_connsucceeded((PgStat_MsgConnSucceeded *) &msg, len);
+ 					break;
+ 
  				default:
  					break;
  			}
***************
*** 3390,3395 **** reset_dbentry_counters(PgStat_StatDBEntry *dbentry)
--- 3520,3528 ----
  	dbentry->n_deadlocks = 0;
  	dbentry->n_block_read_time = 0;
  	dbentry->n_block_write_time = 0;
+ 	dbentry->n_bytes_sent = 0;
+ 	dbentry->n_bytes_received = 0;
+ 	dbentry->n_connections = 0;
  
  	dbentry->stat_reset_timestamp = GetCurrentTimestamp();
  	dbentry->stats_timestamp = 0;
***************
*** 3798,3803 **** pgstat_read_statsfiles(Oid onlydb, bool permanent, bool deep)
--- 3931,3937 ----
  	int32		format_id;
  	bool		found;
  	const char *statfile = permanent ? PGSTAT_STAT_PERMANENT_FILENAME : pgstat_stat_filename;
+ 	TimestampTz now;
  
  	/*
  	 * The tables will live in pgStatLocalContext.
***************
*** 3825,3831 **** pgstat_read_statsfiles(Oid onlydb, bool permanent, bool deep)
  	 * Set the current timestamp (will be kept only in case we can't load an
  	 * existing statsfile).
  	 */
! 	globalStats.stat_reset_timestamp = GetCurrentTimestamp();
  
  	/*
  	 * Try to open the stats file. If it doesn't exist, the backends simply
--- 3959,3967 ----
  	 * Set the current timestamp (will be kept only in case we can't load an
  	 * existing statsfile).
  	 */
! 	now = GetCurrentTimestamp();
! 	globalStats.bgwriter_stat_reset_timestamp = now;
! 	globalStats.socket_stat_reset_timestamp = now;
  
  	/*
  	 * Try to open the stats file. If it doesn't exist, the backends simply
***************
*** 4722,4730 **** pgstat_recv_resetsharedcounter(PgStat_MsgResetsharedcounter *msg, int len)
  {
  	if (msg->m_resettarget == RESET_BGWRITER)
  	{
  		/* Reset the global background writer statistics for the cluster. */
! 		memset(&globalStats, 0, sizeof(globalStats));
! 		globalStats.stat_reset_timestamp = GetCurrentTimestamp();
  	}
  
  	/*
--- 4858,4891 ----
  {
  	if (msg->m_resettarget == RESET_BGWRITER)
  	{
+ 		globalStats.stats_timestamp = 0;
  		/* Reset the global background writer statistics for the cluster. */
! 		globalStats.timed_checkpoints = 0;
! 		globalStats.requested_checkpoints = 0;
! 		globalStats.checkpoint_write_time = 0;
! 		globalStats.checkpoint_sync_time = 0;
! 		globalStats.buf_written_checkpoints = 0;
! 		globalStats.buf_written_clean = 0;
! 		globalStats.maxwritten_clean = 0;
! 		globalStats.buf_written_backend = 0;
! 		globalStats.buf_fsync_backend = 0;
! 		globalStats.buf_alloc = 0;
! 		globalStats.bgwriter_stat_reset_timestamp = GetCurrentTimestamp();
! 	}
! 	else if (msg->m_resettarget == RESET_SOCKET)
! 	{
! 		globalStats.stats_timestamp = 0;
! 		/* Reset the global socket transfer statistics for the cluster. */
! 		globalStats.bytes_sent = 0;
! 		globalStats.bytes_received = 0;
! 		globalStats.bytes_sent_backend = 0;
! 		globalStats.bytes_received_backend = 0;
! 		globalStats.bytes_sent_walsender = 0;
! 		globalStats.bytes_received_walsender = 0;
! 		globalStats.conn_received = 0;
! 		globalStats.conn_backend = 0;
! 		globalStats.conn_walsender = 0;
! 		globalStats.socket_stat_reset_timestamp = GetCurrentTimestamp();
  	}
  
  	/*
***************
*** 4951,4956 **** pgstat_recv_tempfile(PgStat_MsgTempFile *msg, int len)
--- 5112,5201 ----
  }
  
  /* ----------
+  * pgstat_recv_bytessent() -
+  *
+  *    Process a BYTESSENT message.
+  * ----------
+  */
+ static void
+ pgstat_recv_bytessent(PgStat_MsgBytesTransferred *msg, int len)
+ {
+ 	PgStat_StatDBEntry *dbentry;
+ 
+ 	globalStats.bytes_sent += msg->m_bytes_transferred;
+ 
+ 	if (msg->m_walsender)
+ 		globalStats.bytes_sent_walsender += msg->m_bytes_transferred;
+ 
+ 	/* can be called before we have connected to a specific database or by walsender */
+ 	if (OidIsValid(msg->m_databaseid)) {
+ 		globalStats.bytes_sent_backend += msg->m_bytes_transferred;
+ 
+ 		dbentry = pgstat_get_db_entry(msg->m_databaseid, true);
+ 		dbentry->n_bytes_sent += msg->m_bytes_transferred;
+ 	}
+ }
+ 
+ /* ----------
+  * pgstat_recv_bytesreceived() -
+  *
+  *    Process a BYTESRECEIVED message.
+  * ----------
+  */
+ static void
+ pgstat_recv_bytesreceived(PgStat_MsgBytesTransferred *msg, int len)
+ {
+ 	PgStat_StatDBEntry *dbentry;
+ 
+ 	globalStats.bytes_received += msg->m_bytes_transferred;
+ 
+ 	if (msg->m_walsender)
+ 		globalStats.bytes_received_walsender += msg->m_bytes_transferred;
+ 
+ 	/* can be called before we have connected to a specific database or by walsender */
+ 	if (OidIsValid(msg->m_databaseid)) {
+ 		globalStats.bytes_received_backend += msg->m_bytes_transferred;
+ 
+ 		dbentry = pgstat_get_db_entry(msg->m_databaseid, true);
+ 		dbentry->n_bytes_received += msg->m_bytes_transferred;
+ 	}
+ }
+ 
+ /* ----------
+  * pgstat_recv_connreceived() -
+  *
+  *    Process a CONNRECEIVED message.
+  * ----------
+  */
+ static void
+ pgstat_recv_connreceived(PgStat_MsgConnReceived *msg, int len)
+ {
+ 	globalStats.conn_received += 1;
+ }
+ 
+ /* ----------
+  * pgstat_recv_connsucceeded() -
+  *
+  *    Process a CONNSUCCEEDED message.
+  * ----------
+  */
+ static void
+ pgstat_recv_connsucceeded(PgStat_MsgConnSucceeded *msg, int len)
+ {
+ 	PgStat_StatDBEntry *dbentry;
+ 
+ 	if (msg->m_walsender)
+ 		globalStats.conn_walsender += 1;
+ 	else
+ 		globalStats.conn_backend += 1;
+ 
+ 	if (OidIsValid(msg->m_databaseid)) {
+ 		dbentry = pgstat_get_db_entry(msg->m_databaseid, true);
+ 		dbentry->n_connections += 1;
+ 	}
+ }
+ 
+ /* ----------
   * pgstat_recv_funcstat() -
   *
   *	Count what the backend has done.
*** a/src/backend/postmaster/postmaster.c
--- b/src/backend/postmaster/postmaster.c
***************
*** 1825,1830 **** retry1:
--- 1825,1832 ----
  					 errmsg("failed to send SSL negotiation response: %m")));
  			return STATUS_ERROR;	/* close the connection */
  		}
+ 		else
+ 			pgstat_report_bytessent(1);
  
  #ifdef USE_SSL
  		if (SSLok == 'S' && secure_open_server(port) == -1)
***************
*** 3839,3844 **** report_fork_failure_to_client(Port *port, int errnum)
--- 3841,3849 ----
  	{
  		rc = send(port->sock, buffer, strlen(buffer) + 1, 0);
  	} while (rc < 0 && errno == EINTR);
+ 
+ 	if (rc > 0)
+ 		pgstat_report_bytessent(rc);
  }
  
  
***************
*** 3930,3935 **** BackendInitialize(Port *port)
--- 3935,3942 ----
  	else
  		snprintf(remote_ps_data, sizeof(remote_ps_data), "%s(%s)", remote_host, remote_port);
  
+ 	pgstat_report_connreceived();
+ 
  	if (Log_connections)
  	{
  		if (remote_port[0])
*** a/src/backend/utils/adt/pgstatfuncs.c
--- b/src/backend/utils/adt/pgstatfuncs.c
***************
*** 86,91 **** extern Datum pg_stat_get_db_temp_files(PG_FUNCTION_ARGS);
--- 86,94 ----
  extern Datum pg_stat_get_db_temp_bytes(PG_FUNCTION_ARGS);
  extern Datum pg_stat_get_db_blk_read_time(PG_FUNCTION_ARGS);
  extern Datum pg_stat_get_db_blk_write_time(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_db_bytes_sent(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_db_bytes_received(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_db_connections(PG_FUNCTION_ARGS);
  
  extern Datum pg_stat_get_bgwriter_timed_checkpoints(PG_FUNCTION_ARGS);
  extern Datum pg_stat_get_bgwriter_requested_checkpoints(PG_FUNCTION_ARGS);
***************
*** 99,104 **** extern Datum pg_stat_get_buf_written_backend(PG_FUNCTION_ARGS);
--- 102,118 ----
  extern Datum pg_stat_get_buf_fsync_backend(PG_FUNCTION_ARGS);
  extern Datum pg_stat_get_buf_alloc(PG_FUNCTION_ARGS);
  
+ extern Datum pg_stat_get_bytes_sent(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_bytes_received(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_bytes_sent_backend(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_bytes_received_backend(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_bytes_sent_walsender(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_bytes_received_walsender(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_conn_received(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_conn_backend(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_conn_walsender(PG_FUNCTION_ARGS);
+ extern Datum pg_stat_get_socket_stat_reset_time(PG_FUNCTION_ARGS);
+ 
  extern Datum pg_stat_get_xact_numscans(PG_FUNCTION_ARGS);
  extern Datum pg_stat_get_xact_tuples_returned(PG_FUNCTION_ARGS);
  extern Datum pg_stat_get_xact_tuples_fetched(PG_FUNCTION_ARGS);
***************
*** 534,540 **** pg_stat_get_activity(PG_FUNCTION_ARGS)
  
  		oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
  
! 		tupdesc = CreateTemplateTupleDesc(14, false);
  		TupleDescInitEntry(tupdesc, (AttrNumber) 1, "datid",
  						   OIDOID, -1, 0);
  		TupleDescInitEntry(tupdesc, (AttrNumber) 2, "pid",
--- 548,554 ----
  
  		oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
  
! 		tupdesc = CreateTemplateTupleDesc(16, false);
  		TupleDescInitEntry(tupdesc, (AttrNumber) 1, "datid",
  						   OIDOID, -1, 0);
  		TupleDescInitEntry(tupdesc, (AttrNumber) 2, "pid",
***************
*** 563,568 **** pg_stat_get_activity(PG_FUNCTION_ARGS)
--- 577,586 ----
  						   TEXTOID, -1, 0);
  		TupleDescInitEntry(tupdesc, (AttrNumber) 14, "client_port",
  						   INT4OID, -1, 0);
+ 		TupleDescInitEntry(tupdesc, (AttrNumber) 15, "bytes_sent",
+ 						   INT8OID, -1, 0);
+ 		TupleDescInitEntry(tupdesc, (AttrNumber) 16, "bytes_received",
+ 						   INT8OID, -1, 0);
  
  		funcctx->tuple_desc = BlessTupleDesc(tupdesc);
  
***************
*** 614,621 **** pg_stat_get_activity(PG_FUNCTION_ARGS)
  	if (funcctx->call_cntr < funcctx->max_calls)
  	{
  		/* for each row */
! 		Datum		values[14];
! 		bool		nulls[14];
  		HeapTuple	tuple;
  		PgBackendStatus *beentry;
  		SockAddr	zero_clientaddr;
--- 632,639 ----
  	if (funcctx->call_cntr < funcctx->max_calls)
  	{
  		/* for each row */
! 		Datum		values[16];
! 		bool		nulls[16];
  		HeapTuple	tuple;
  		PgBackendStatus *beentry;
  		SockAddr	zero_clientaddr;
***************
*** 773,778 **** pg_stat_get_activity(PG_FUNCTION_ARGS)
--- 791,798 ----
  					nulls[13] = true;
  				}
  			}
+ 			values[14] = Int64GetDatum(beentry->st_bytes_sent);
+ 			values[15] = Int64GetDatum(beentry->st_bytes_received);
  		}
  		else
  		{
***************
*** 787,792 **** pg_stat_get_activity(PG_FUNCTION_ARGS)
--- 807,814 ----
  			nulls[11] = true;
  			nulls[12] = true;
  			nulls[13] = true;
+ 			nulls[14] = true;
+ 			nulls[15] = true;
  		}
  
  		tuple = heap_form_tuple(funcctx->tuple_desc, values, nulls);
***************
*** 1407,1412 **** pg_stat_get_db_blk_write_time(PG_FUNCTION_ARGS)
--- 1429,1479 ----
  }
  
  Datum
+ pg_stat_get_db_bytes_sent(PG_FUNCTION_ARGS)
+ {
+ 	Oid			dbid = PG_GETARG_OID(0);
+ 	double		result;
+ 	PgStat_StatDBEntry *dbentry;
+ 
+ 	if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
+ 		result = 0;
+ 	else
+ 		result = dbentry->n_bytes_sent;
+ 
+ 	PG_RETURN_INT64(result);
+ }
+ 
+ Datum
+ pg_stat_get_db_bytes_received(PG_FUNCTION_ARGS)
+ {
+ 	Oid			dbid = PG_GETARG_OID(0);
+ 	double		result;
+ 	PgStat_StatDBEntry *dbentry;
+ 
+ 	if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
+ 		result = 0;
+ 	else
+ 		result = dbentry->n_bytes_received;
+ 
+ 	PG_RETURN_INT64(result);
+ }
+ 
+ Datum
+ pg_stat_get_db_connections(PG_FUNCTION_ARGS)
+ {
+ 	Oid			dbid = PG_GETARG_OID(0);
+ 	double		result;
+ 	PgStat_StatDBEntry *dbentry;
+ 
+ 	if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
+ 		result = 0;
+ 	else
+ 		result = dbentry->n_connections;
+ 
+ 	PG_RETURN_INT64(result);
+ }
+ 
+ Datum
  pg_stat_get_bgwriter_timed_checkpoints(PG_FUNCTION_ARGS)
  {
  	PG_RETURN_INT64(pgstat_fetch_global()->timed_checkpoints);
***************
*** 1453,1459 **** pg_stat_get_checkpoint_sync_time(PG_FUNCTION_ARGS)
  Datum
  pg_stat_get_bgwriter_stat_reset_time(PG_FUNCTION_ARGS)
  {
! 	PG_RETURN_TIMESTAMPTZ(pgstat_fetch_global()->stat_reset_timestamp);
  }
  
  Datum
--- 1520,1526 ----
  Datum
  pg_stat_get_bgwriter_stat_reset_time(PG_FUNCTION_ARGS)
  {
! 	PG_RETURN_TIMESTAMPTZ(pgstat_fetch_global()->bgwriter_stat_reset_timestamp);
  }
  
  Datum
***************
*** 1475,1480 **** pg_stat_get_buf_alloc(PG_FUNCTION_ARGS)
--- 1542,1607 ----
  }
  
  Datum
+ pg_stat_get_bytes_sent(PG_FUNCTION_ARGS)
+ {
+ 	PG_RETURN_INT64(pgstat_fetch_global()->bytes_sent);
+ }
+ 
+ Datum
+ pg_stat_get_bytes_received(PG_FUNCTION_ARGS)
+ {
+ 	PG_RETURN_INT64(pgstat_fetch_global()->bytes_received);
+ }
+ 
+ Datum
+ pg_stat_get_bytes_sent_backend(PG_FUNCTION_ARGS)
+ {
+ 	PG_RETURN_INT64(pgstat_fetch_global()->bytes_sent_backend);
+ }
+ 
+ Datum
+ pg_stat_get_bytes_received_backend(PG_FUNCTION_ARGS)
+ {
+ 	PG_RETURN_INT64(pgstat_fetch_global()->bytes_received_backend);
+ }
+ 
+ Datum
+ pg_stat_get_bytes_sent_walsender(PG_FUNCTION_ARGS)
+ {
+ 	PG_RETURN_INT64(pgstat_fetch_global()->bytes_sent_walsender);
+ }
+ 
+ Datum
+ pg_stat_get_bytes_received_walsender(PG_FUNCTION_ARGS)
+ {
+ 	PG_RETURN_INT64(pgstat_fetch_global()->bytes_received_walsender);
+ }
+ 
+ Datum
+ pg_stat_get_conn_received(PG_FUNCTION_ARGS)
+ {
+ 	PG_RETURN_INT64(pgstat_fetch_global()->conn_received);
+ }
+ 
+ Datum
+ pg_stat_get_conn_backend(PG_FUNCTION_ARGS)
+ {
+ 	PG_RETURN_INT64(pgstat_fetch_global()->conn_backend);
+ }
+ 
+ Datum
+ pg_stat_get_conn_walsender(PG_FUNCTION_ARGS)
+ {
+ 	PG_RETURN_INT64(pgstat_fetch_global()->conn_walsender);
+ }
+ 
+ Datum
+ pg_stat_get_socket_stat_reset_time(PG_FUNCTION_ARGS)
+ {
+ 	PG_RETURN_TIMESTAMPTZ(pgstat_fetch_global()->socket_stat_reset_timestamp);
+ }
+ 
+ Datum
  pg_stat_get_xact_numscans(PG_FUNCTION_ARGS)
  {
  	Oid			relid = PG_GETARG_OID(0);
*** a/src/include/catalog/pg_proc.h
--- b/src/include/catalog/pg_proc.h
***************
*** 2626,2632 **** DATA(insert OID = 3057 ( pg_stat_get_autoanalyze_count PGNSP PGUID 12 1 0 0 0 f
  DESCR("statistics: number of auto analyzes for a table");
  DATA(insert OID = 1936 (  pg_stat_get_backend_idset		PGNSP PGUID 12 1 100 0 0 f f f f t t s 0 0 23 "" _null_ _null_ _null_ _null_ pg_stat_get_backend_idset _null_ _null_ _null_ ));
  DESCR("statistics: currently active backend IDs");
! DATA(insert OID = 2022 (  pg_stat_get_activity			PGNSP PGUID 12 1 100 0 0 f f f f f t s 1 0 2249 "23" "{23,26,23,26,25,25,25,16,1184,1184,1184,1184,869,25,23}" "{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o}" "{pid,datid,pid,usesysid,application_name,state,query,waiting,xact_start,query_start,backend_start,state_change,client_addr,client_hostname,client_port}" _null_ pg_stat_get_activity _null_ _null_ _null_ ));
  DESCR("statistics: information about currently active backends");
  DATA(insert OID = 3099 (  pg_stat_get_wal_senders	PGNSP PGUID 12 1 10 0 0 f f f f f t s 0 0 2249 "" "{23,25,25,25,25,25,23,25}" "{o,o,o,o,o,o,o,o}" "{pid,state,sent_location,write_location,flush_location,replay_location,sync_priority,sync_state}" _null_ pg_stat_get_wal_senders _null_ _null_ _null_ ));
  DESCR("statistics: information about currently active replication");
--- 2626,2632 ----
  DESCR("statistics: number of auto analyzes for a table");
  DATA(insert OID = 1936 (  pg_stat_get_backend_idset		PGNSP PGUID 12 1 100 0 0 f f f f t t s 0 0 23 "" _null_ _null_ _null_ _null_ pg_stat_get_backend_idset _null_ _null_ _null_ ));
  DESCR("statistics: currently active backend IDs");
! DATA(insert OID = 2022 (  pg_stat_get_activity			PGNSP PGUID 12 1 100 0 0 f f f f f t s 1 0 2249 "23" "{23,26,23,26,25,25,25,16,1184,1184,1184,1184,869,25,23,20,20}" "{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}" "{pid,datid,pid,usesysid,application_name,state,query,waiting,xact_start,query_start,backend_start,state_change,client_addr,client_hostname,client_port,bytes_sent,bytes_received}" _null_ pg_stat_get_activity _null_ _null_ _null_ ));
  DESCR("statistics: information about currently active backends");
  DATA(insert OID = 3099 (  pg_stat_get_wal_senders	PGNSP PGUID 12 1 10 0 0 f f f f f t s 0 0 2249 "" "{23,25,25,25,25,25,23,25}" "{o,o,o,o,o,o,o,o}" "{pid,state,sent_location,write_location,flush_location,replay_location,sync_priority,sync_state}" _null_ pg_stat_get_wal_senders _null_ _null_ _null_ ));
  DESCR("statistics: information about currently active replication");
***************
*** 2696,2701 **** DATA(insert OID = 2844 (  pg_stat_get_db_blk_read_time	PGNSP PGUID 12 1 0 0 0 f
--- 2696,2707 ----
  DESCR("statistics: block read time, in msec");
  DATA(insert OID = 2845 (  pg_stat_get_db_blk_write_time PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 701 "26" _null_ _null_ _null_ _null_ pg_stat_get_db_blk_write_time _null_ _null_ _null_ ));
  DESCR("statistics: block write time, in msec");
+ DATA(insert OID = 3195 (  pg_stat_get_db_bytes_sent PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_db_bytes_sent _null_ _null_ _null_ ));
+ DESCR("statistics: number of bytes sent over client connections");
+ DATA(insert OID = 3196 (  pg_stat_get_db_bytes_received PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_db_bytes_received _null_ _null_ _null_ ));
+ DESCR("statistics: number of bytes received over client connections");
+ DATA(insert OID = 3197 (  pg_stat_get_db_connections PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_db_connections _null_ _null_ _null_ ));
+ DESCR("statistics: number of successful client connections");
  DATA(insert OID = 2769 ( pg_stat_get_bgwriter_timed_checkpoints PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 20 "" _null_ _null_ _null_ _null_ pg_stat_get_bgwriter_timed_checkpoints _null_ _null_ _null_ ));
  DESCR("statistics: number of timed checkpoints started by the bgwriter");
  DATA(insert OID = 2770 ( pg_stat_get_bgwriter_requested_checkpoints PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 20 "" _null_ _null_ _null_ _null_ pg_stat_get_bgwriter_requested_checkpoints _null_ _null_ _null_ ));
***************
*** 2719,2724 **** DESCR("statistics: number of backend buffer writes that did their own fsync");
--- 2725,2751 ----
  DATA(insert OID = 2859 ( pg_stat_get_buf_alloc			PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 20 "" _null_ _null_ _null_ _null_ pg_stat_get_buf_alloc _null_ _null_ _null_ ));
  DESCR("statistics: number of buffer allocations");
  
+ DATA(insert OID = 3198 ( pg_stat_get_bytes_sent PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 20 "" _null_ _null_ _null_ _null_ pg_stat_get_bytes_sent _null_ _null_ _null_ ));
+ DESCR("statistics: total number of bytes sent over client connections");
+ DATA(insert OID = 3199 ( pg_stat_get_bytes_received PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 20 "" _null_ _null_ _null_ _null_ pg_stat_get_bytes_received _null_ _null_ _null_ ));
+ DESCR("statistics: total number of bytes received over client connections");
+ DATA(insert OID = 3200 ( pg_stat_get_bytes_sent_backend PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 20 "" _null_ _null_ _null_ _null_ pg_stat_get_bytes_sent_backend _null_ _null_ _null_ ));
+ DESCR("statistics: number of bytes sent over client connections to user backends");
+ DATA(insert OID = 3201 ( pg_stat_get_bytes_received_backend PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 20 "" _null_ _null_ _null_ _null_ pg_stat_get_bytes_received_backend _null_ _null_ _null_ ));
+ DESCR("statistics: number of bytes received over client connections from user backends");
+ DATA(insert OID = 3202 ( pg_stat_get_bytes_sent_walsender PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 20 "" _null_ _null_ _null_ _null_ pg_stat_get_bytes_sent_walsender _null_ _null_ _null_ ));
+ DESCR("statistics: number of bytes sent over client connections to WAL senders");
+ DATA(insert OID = 3203 ( pg_stat_get_bytes_received_walsender PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 20 "" _null_ _null_ _null_ _null_ pg_stat_get_bytes_received_walsender _null_ _null_ _null_ ));
+ DESCR("statistics: number of bytes received over client connections from WAL senders");
+ DATA(insert OID = 3204 ( pg_stat_get_conn_received PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 20 "" _null_ _null_ _null_ _null_ pg_stat_get_conn_received _null_ _null_ _null_ ));
+ DESCR("statistics: number of client connections received");
+ DATA(insert OID = 3205 ( pg_stat_get_conn_backend PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 20 "" _null_ _null_ _null_ _null_ pg_stat_get_conn_backend _null_ _null_ _null_ ));
+ DESCR("statistics: number of successful client connections to a user backend");
+ DATA(insert OID = 3206 ( pg_stat_get_conn_walsender PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 20 "" _null_ _null_ _null_ _null_ pg_stat_get_conn_walsender _null_ _null_ _null_ ));
+ DESCR("statistics: number of successful client connections to a WAL sender");
+ DATA(insert OID = 3207 ( pg_stat_get_socket_stat_reset_time PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 1184 "" _null_ _null_ _null_ _null_	pg_stat_get_socket_stat_reset_time _null_ _null_ _null_ ));
+ DESCR("statistics: last reset for the client connection statistics");
+ 
  DATA(insert OID = 2978 (  pg_stat_get_function_calls		PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_function_calls _null_ _null_ _null_ ));
  DESCR("statistics: number of function calls");
  DATA(insert OID = 2979 (  pg_stat_get_function_total_time	PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 701 "26" _null_ _null_ _null_ _null_ pg_stat_get_function_total_time _null_ _null_ _null_ ));
*** a/src/include/pgstat.h
--- b/src/include/pgstat.h
***************
*** 49,55 **** typedef enum StatMsgType
  	PGSTAT_MTYPE_FUNCPURGE,
  	PGSTAT_MTYPE_RECOVERYCONFLICT,
  	PGSTAT_MTYPE_TEMPFILE,
! 	PGSTAT_MTYPE_DEADLOCK
  } StatMsgType;
  
  /* ----------
--- 49,59 ----
  	PGSTAT_MTYPE_FUNCPURGE,
  	PGSTAT_MTYPE_RECOVERYCONFLICT,
  	PGSTAT_MTYPE_TEMPFILE,
! 	PGSTAT_MTYPE_DEADLOCK,
! 	PGSTAT_MTYPE_BYTESSENT,
! 	PGSTAT_MTYPE_BYTESRECEIVED,
! 	PGSTAT_MTYPE_CONNRECEIVED,
! 	PGSTAT_MTYPE_CONNSUCCEEDED
  } StatMsgType;
  
  /* ----------
***************
*** 102,108 **** typedef struct PgStat_TableCounts
  /* Possible targets for resetting cluster-wide shared values */
  typedef enum PgStat_Shared_Reset_Target
  {
! 	RESET_BGWRITER
  } PgStat_Shared_Reset_Target;
  
  /* Possible object types for resetting single counters */
--- 106,113 ----
  /* Possible targets for resetting cluster-wide shared values */
  typedef enum PgStat_Shared_Reset_Target
  {
! 	RESET_BGWRITER,
! 	RESET_SOCKET
  } PgStat_Shared_Reset_Target;
  
  /* Possible object types for resetting single counters */
***************
*** 397,402 **** typedef struct PgStat_MsgTempFile
--- 402,446 ----
  } PgStat_MsgTempFile;
  
  /* ----------
+   * PgStat_MsgBytesTransferred
+   *
+   * Sent upon sending or receiving data over a client connection.
+   * The message header determines direction.
+   * ----------
+   */
+ typedef struct PgStat_MsgBytesTransferred
+ {
+ 	PgStat_MsgHdr	m_hdr;
+ 	Oid			m_databaseid;
+ 	int			m_bytes_transferred;
+ 	bool		m_walsender;
+ } PgStat_MsgBytesTransferred;
+ 
+ /* ----------
+   * PgStat_MsgConnReceived
+   *
+   * Sent upon receiving a client connection.
+   * ----------
+   */
+ typedef struct PgStat_MsgConnReceived
+ {
+ 	PgStat_MsgHdr	m_hdr;
+ } PgStat_MsgConnReceived;
+ 
+ /* ----------
+   * PgStat_MsgConnSucceeded
+   *
+   * Sent upon client's successful connection.
+   * ----------
+   */
+ typedef struct PgStat_MsgConnSucceeded
+ {
+ 	PgStat_MsgHdr	m_hdr;
+ 	Oid			m_databaseid;
+ 	bool		m_walsender;
+ } PgStat_MsgConnSucceeded;
+ 
+ /* ----------
   * PgStat_FunctionCounts	The actual per-function counts kept by a backend
   *
   * This struct should contain only actual event counters, because we memcmp
***************
*** 515,521 **** typedef union PgStat_Msg
   * ------------------------------------------------------------
   */
  
! #define PGSTAT_FILE_FORMAT_ID	0x01A5BC9B
  
  /* ----------
   * PgStat_StatDBEntry			The collector's data per database
--- 559,565 ----
   * ------------------------------------------------------------
   */
  
! #define PGSTAT_FILE_FORMAT_ID	0x01A5BC9C
  
  /* ----------
   * PgStat_StatDBEntry			The collector's data per database
***************
*** 544,550 **** typedef struct PgStat_StatDBEntry
  	PgStat_Counter n_deadlocks;
  	PgStat_Counter n_block_read_time;	/* times in microseconds */
  	PgStat_Counter n_block_write_time;
! 
  	TimestampTz stat_reset_timestamp;
  	TimestampTz stats_timestamp;	/* time of db stats file update */
  
--- 588,598 ----
  	PgStat_Counter n_deadlocks;
  	PgStat_Counter n_block_read_time;	/* times in microseconds */
  	PgStat_Counter n_block_write_time;
! 	/* communication socket transfer counter in bytes (backend to client connections) */
! 	PgStat_Counter n_bytes_sent;
! 	/* communication socket transfer counter in bytes (client to backend connections) */
! 	PgStat_Counter n_bytes_received;
! 	PgStat_Counter n_connections; /* client connections succeeded */
  	TimestampTz stat_reset_timestamp;
  	TimestampTz stats_timestamp;	/* time of db stats file update */
  
***************
*** 614,619 **** typedef struct PgStat_StatFuncEntry
--- 662,668 ----
  typedef struct PgStat_GlobalStats
  {
  	TimestampTz stats_timestamp;	/* time of stats file update */
+ 	/* bgwriter stats */
  	PgStat_Counter timed_checkpoints;
  	PgStat_Counter requested_checkpoints;
  	PgStat_Counter checkpoint_write_time;		/* times in milliseconds */
***************
*** 624,630 **** typedef struct PgStat_GlobalStats
  	PgStat_Counter buf_written_backend;
  	PgStat_Counter buf_fsync_backend;
  	PgStat_Counter buf_alloc;
! 	TimestampTz stat_reset_timestamp;
  } PgStat_GlobalStats;
  
  
--- 673,690 ----
  	PgStat_Counter buf_written_backend;
  	PgStat_Counter buf_fsync_backend;
  	PgStat_Counter buf_alloc;
! 	TimestampTz bgwriter_stat_reset_timestamp;
! 	/* client connection stats */
! 	PgStat_Counter bytes_sent; /* in bytes (cluster to client) */
! 	PgStat_Counter bytes_received; /* in bytes (client to cluster) */
! 	PgStat_Counter bytes_sent_backend; /* in bytes (backend to client) */
! 	PgStat_Counter bytes_received_backend; /* in bytes (client to backend) */
! 	PgStat_Counter bytes_sent_walsender; /* in bytes (walsender to client) */
! 	PgStat_Counter bytes_received_walsender; /* in bytes (client to walsender) */
! 	PgStat_Counter conn_received; /* client connections received */
! 	PgStat_Counter conn_backend; /* successful client connections to a backend */
! 	PgStat_Counter conn_walsender; /* successful client connections to a walsender */
! 	TimestampTz socket_stat_reset_timestamp;
  } PgStat_GlobalStats;
  
  
***************
*** 697,702 **** typedef struct PgBackendStatus
--- 757,768 ----
  
  	/* current command string; MUST be null-terminated */
  	char	   *st_activity;
+ 
+ 	/* communication socket transfer counter in bytes (backend to client connections) */
+ 	unsigned long st_bytes_sent;
+ 	/* communication socket transfer counter in bytes (client to backend connections) */
+ 	unsigned long st_bytes_received;
+ 
  } PgBackendStatus;
  
  /*
***************
*** 788,793 **** extern void pgstat_report_tempfile(size_t filesize);
--- 854,864 ----
  extern void pgstat_report_appname(const char *appname);
  extern void pgstat_report_xact_timestamp(TimestampTz tstamp);
  extern void pgstat_report_waiting(bool waiting);
+ extern void pgstat_report_bytessent(int count);
+ extern void pgstat_report_bytesreceived(int count);
+ extern void pgstat_report_connreceived(void);
+ extern void pgstat_report_connsucceeded(void);
+ 
  extern const char *pgstat_get_backend_current_activity(int pid, bool checkUser);
  extern const char *pgstat_get_crashed_backend_activity(int pid, char *buffer,
  									int buflen);
*** a/src/test/regress/expected/rules.out
--- b/src/test/regress/expected/rules.out
***************
*** 1595,1603 **** pg_stat_activity| SELECT s.datid,
      s.state_change,
      s.waiting,
      s.state,
!     s.query
     FROM pg_database d,
!     pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, waiting, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port),
      pg_authid u
    WHERE ((s.datid = d.oid) AND (s.usesysid = u.oid));
  pg_stat_all_indexes| SELECT c.oid AS relid,
--- 1595,1605 ----
      s.state_change,
      s.waiting,
      s.state,
!     s.query,
!     s.bytes_sent,
!     s.bytes_received
     FROM pg_database d,
!     pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, waiting, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, bytes_sent, bytes_received),
      pg_authid u
    WHERE ((s.datid = d.oid) AND (s.usesysid = u.oid));
  pg_stat_all_indexes| SELECT c.oid AS relid,
***************
*** 1669,1674 **** pg_stat_database| SELECT d.oid AS datid,
--- 1671,1679 ----
      pg_stat_get_db_deadlocks(d.oid) AS deadlocks,
      pg_stat_get_db_blk_read_time(d.oid) AS blk_read_time,
      pg_stat_get_db_blk_write_time(d.oid) AS blk_write_time,
+     pg_stat_get_db_bytes_sent(d.oid) AS bytes_sent,
+     pg_stat_get_db_bytes_received(d.oid) AS bytes_received,
+     pg_stat_get_db_connections(d.oid) AS connections,
      pg_stat_get_db_stat_reset_time(d.oid) AS stats_reset
     FROM pg_database d;
  pg_stat_database_conflicts| SELECT d.oid AS datid,
***************
*** 1687,1692 **** pg_stat_replication| SELECT s.pid,
--- 1692,1699 ----
      s.client_hostname,
      s.client_port,
      s.backend_start,
+     s.bytes_sent,
+     s.bytes_received,
      w.state,
      w.sent_location,
      w.write_location,
***************
*** 1694,1703 **** pg_stat_replication| SELECT s.pid,
      w.replay_location,
      w.sync_priority,
      w.sync_state
!    FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, waiting, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port),
      pg_authid u,
      pg_stat_get_wal_senders() w(pid, state, sent_location, write_location, flush_location, replay_location, sync_priority, sync_state)
    WHERE ((s.usesysid = u.oid) AND (s.pid = w.pid));
  pg_stat_sys_indexes| SELECT pg_stat_all_indexes.relid,
      pg_stat_all_indexes.indexrelid,
      pg_stat_all_indexes.schemaname,
--- 1701,1720 ----
      w.replay_location,
      w.sync_priority,
      w.sync_state
!    FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, waiting, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, bytes_sent, bytes_received),
      pg_authid u,
      pg_stat_get_wal_senders() w(pid, state, sent_location, write_location, flush_location, replay_location, sync_priority, sync_state)
    WHERE ((s.usesysid = u.oid) AND (s.pid = w.pid));
+ pg_stat_socket| SELECT pg_stat_get_bytes_sent() AS bytes_sent_total,
+     pg_stat_get_bytes_received() AS bytes_received_total,
+     pg_stat_get_bytes_sent_backend() AS bytes_sent_backend,
+     pg_stat_get_bytes_received_backend() AS bytes_received_backend,
+     pg_stat_get_bytes_sent_walsender() AS bytes_sent_walsender,
+     pg_stat_get_bytes_received_walsender() AS bytes_received_walsender,
+     pg_stat_get_conn_received() AS conn_received,
+     pg_stat_get_conn_backend() AS conn_backend,
+     pg_stat_get_conn_walsender() AS conn_walsender,
+     pg_stat_get_socket_stat_reset_time() AS stats_reset;
  pg_stat_sys_indexes| SELECT pg_stat_all_indexes.relid,
      pg_stat_all_indexes.indexrelid,
      pg_stat_all_indexes.schemaname,
#20Mike Blackwell
mike.blackwell@rrd.com
In reply to: Nigel Heron (#19)
Re: stats for network traffic WIP

This patch looks good to me. It applies, builds, and runs the regression
tests. Documentation is included and it seems to do what it says. I don't
consider myself a code expert, but as far as I can see it looks fine. This
is a pretty straightforward enhancement to the existing pg_stat_* code.

If no one has any objections, I'll mark it ready for committer.

Mike

__________________________________________________________________________________
*Mike Blackwell | Technical Analyst, Distribution Services/Rollout
Management | RR Donnelley*
1750 Wallace Ave | St Charles, IL 60174-3401
Office: 630.313.7818
Mike.Blackwell@rrd.com
http://www.rrdonnelley.com

<http://www.rrdonnelley.com/&gt;
* <Mike.Blackwell@rrd.com>*

On Thu, Nov 14, 2013 at 11:29 PM, Nigel Heron <nheron@querymetrics.com>wrote:

Show quoted text

On Wed, Nov 13, 2013 at 11:27 PM, Peter Eisentraut <peter_e@gmx.net>
wrote:

On Fri, 2013-11-08 at 10:01 -0500, Nigel Heron wrote:

here's v4 of the patch. I added documentation and a new global view
called "pg_stat_socket" (includes bytes_sent, bytes_received and
stats_reset time)

Your patch needs to be rebased:

CONFLICT (content): Merge conflict in src/test/regress/expected/rules.out

Hi,
here's a rebased patch with some additions.

an overview of it's current state...

a new pg_stat_socket global view:
- total bytes sent and received
- bytes sent and received for user backends
- bytes sent and received for wal senders
- total connection attempts
- successful connections to user backends
- successful connections to wal senders
- stats reset time
pg_stat_reset_shared('socket') resets the counters

added to pg_stat_database view:
- bytes sent and received per db
- successful connections per db
pg_stat_reset() resets the counters

added to pg_stat_activity view:
- bytes sent and received per backend

added to pg_stat_replication view:
- bytes sent and received per wal sender

using the existing track_counts guc to enable/disable these stats.
-nigel.

#21Atri Sharma
atri.jiit@gmail.com
In reply to: Mike Blackwell (#20)
Re: stats for network traffic WIP

On Tue, Nov 19, 2013 at 11:43 PM, Mike Blackwell <mike.blackwell@rrd.com> wrote:

This patch looks good to me. It applies, builds, and runs the regression
tests. Documentation is included and it seems to do what it says. I don't
consider myself a code expert, but as far as I can see it looks fine. This
is a pretty straightforward enhancement to the existing pg_stat_* code.

If no one has any objections, I'll mark it ready for committer.

Mike

I agree.

I had a discussion with Mike yesterday, and took the performance areas
in the patch. I think the impact would be pretty low and since the
global counter being incremented is incremented with keeping race
conditions in mind, I think that the statistics collected will be
valid.

So, I have no objections to the patch being marked as ready for committer.

Regards,

Atri

Regards,

Atri
l'apprenant

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#22Fujii Masao
masao.fujii@gmail.com
In reply to: Atri Sharma (#21)
Re: stats for network traffic WIP

On Wed, Nov 20, 2013 at 3:18 AM, Atri Sharma <atri.jiit@gmail.com> wrote:

On Tue, Nov 19, 2013 at 11:43 PM, Mike Blackwell <mike.blackwell@rrd.com> wrote:

This patch looks good to me. It applies, builds, and runs the regression
tests. Documentation is included and it seems to do what it says. I don't
consider myself a code expert, but as far as I can see it looks fine. This
is a pretty straightforward enhancement to the existing pg_stat_* code.

If no one has any objections, I'll mark it ready for committer.

Mike

I agree.

I had a discussion with Mike yesterday, and took the performance areas
in the patch. I think the impact would be pretty low and since the
global counter being incremented is incremented with keeping race
conditions in mind, I think that the statistics collected will be
valid.

So, I have no objections to the patch being marked as ready for committer.

Could you share the performance numbers? I'm really concerned about
the performance overhead caused by this patch.

Here are the comments from me:

All the restrictions of this feature should be documented. For example,
this feature doesn't track the bytes of the data transferred by FDW.
It's worth documenting that kind of information.

ISTM that this feature doesn't support SSL case. Why not?

The amount of data transferred by walreceiver also should be tracked,
I think.

I just wonder how conn_received, conn_backend and conn_walsender
are useful.

Regards,

--
Fujii Masao

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#23Atri Sharma
atri.jiit@gmail.com
In reply to: Fujii Masao (#22)
Re: stats for network traffic WIP

Sent from my iPad

On 07-Dec-2013, at 23:47, Fujii Masao <masao.fujii@gmail.com> wrote:

On Wed, Nov 20, 2013 at 3:18 AM, Atri Sharma <atri.jiit@gmail.com> wrote:

On Tue, Nov 19, 2013 at 11:43 PM, Mike Blackwell <mike.blackwell@rrd.com> wrote:
This patch looks good to me. It applies, builds, and runs the regression
tests. Documentation is included and it seems to do what it says. I don't
consider myself a code expert, but as far as I can see it looks fine. This
is a pretty straightforward enhancement to the existing pg_stat_* code.

If no one has any objections, I'll mark it ready for committer.

Mike

I agree.

I had a discussion with Mike yesterday, and took the performance areas
in the patch. I think the impact would be pretty low and since the
global counter being incremented is incremented with keeping race
conditions in mind, I think that the statistics collected will be
valid.

So, I have no objections to the patch being marked as ready for committer.

Could you share the performance numbers? I'm really concerned about
the performance overhead caused by this patch.

I did some pgbench tests specifically with increasing number of clients, as that are the kind of workloads that can lead to display in slowness due to increase in work in the commonly used functions. Let me see if I can get the numbers and see where I kept them.

Here are the comments from me:

All the restrictions of this feature should be documented. For example,
this feature doesn't track the bytes of the data transferred by FDW.
It's worth documenting that kind of information.

+1

ISTM that this feature doesn't support SSL case. Why not?

The amount of data transferred by walreceiver also should be tracked,
I think.

Yes, I agree. WAL receiver data transfer can be problematic some times as well, so should be tracked.

Regards,

Atri

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#24Nigel Heron
nheron@querymetrics.com
In reply to: Fujii Masao (#22)
Re: stats for network traffic WIP

On Sat, Dec 7, 2013 at 1:17 PM, Fujii Masao <masao.fujii@gmail.com> wrote:

Could you share the performance numbers? I'm really concerned about
the performance overhead caused by this patch.

I've tried pgbench in select mode with small data sets to avoid disk
io and didn't see any difference. That was on my old core2duo laptop
though .. I'll have to retry it on some server class multi core
hardware.

I could create a new GUC to turn on/off this feature. Currently, it
uses "track_counts".

Here are the comments from me:

All the restrictions of this feature should be documented. For example,
this feature doesn't track the bytes of the data transferred by FDW.
It's worth documenting that kind of information.

OK. It also doesn't account for DNS resolution, Bonjour traffic and
any traffic generated from PL functions that create their own sockets.

ISTM that this feature doesn't support SSL case. Why not?

It does support SSL, see my_sock_read() and my_sock_write() in
backend/libpq/be-secure.c

The amount of data transferred by walreceiver also should be tracked,
I think.

I'll have to take another look at it. I might be able to create SSL
BIO functions in libpqwalreceiver.c and change some other functions
(eg. libpqrcv_send) to return byte counts instead of void to get it
working.

I just wonder how conn_received, conn_backend and conn_walsender
are useful.

I thought of it mostly for monitoring software usage (eg. cacti,
nagios) to track connections/sec which might be used for capacity
planning, confirm connection pooler settings, monitoring abuse, etc.
Eg. If your conn_walsender is increasing and you have a fixed set of
slaves it could show a network issue.
The information is available in the logs if "log_connections" GUC is
on but it requires parsing and access to log files to extract. With
the increasing popularity of hosted postgres services without OS or
log access, I think more metrics should be available through system
views.

-nigel.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#25Fujii Masao
masao.fujii@gmail.com
In reply to: Nigel Heron (#24)
Re: stats for network traffic WIP

On Tue, Dec 10, 2013 at 6:56 AM, Nigel Heron <nheron@querymetrics.com> wrote:

On Sat, Dec 7, 2013 at 1:17 PM, Fujii Masao <masao.fujii@gmail.com> wrote:

Could you share the performance numbers? I'm really concerned about
the performance overhead caused by this patch.

I've tried pgbench in select mode with small data sets to avoid disk
io and didn't see any difference. That was on my old core2duo laptop
though .. I'll have to retry it on some server class multi core
hardware.

When I ran pgbench -i -s 100 in four parallel, I saw the performance difference
between the master and the patched one. I ran the following commands.

psql -c "checkpoint"
for i in $(seq 1 4); do time pgbench -i -s100 -q db$i & done

The results are:

* Master
10000000 of 10000000 tuples (100%) done (elapsed 13.91 s, remaining 0.00 s).
10000000 of 10000000 tuples (100%) done (elapsed 14.03 s, remaining 0.00 s).
10000000 of 10000000 tuples (100%) done (elapsed 14.01 s, remaining 0.00 s).
10000000 of 10000000 tuples (100%) done (elapsed 14.13 s, remaining 0.00 s).

It took almost 14.0 seconds to store 10000000 tuples.

* Patched
10000000 of 10000000 tuples (100%) done (elapsed 14.90 s, remaining 0.00 s).
10000000 of 10000000 tuples (100%) done (elapsed 15.05 s, remaining 0.00 s).
10000000 of 10000000 tuples (100%) done (elapsed 15.42 s, remaining 0.00 s).
10000000 of 10000000 tuples (100%) done (elapsed 15.70 s, remaining 0.00 s).

It took almost 15.0 seconds to store 10000000 tuples.

Thus, I'm afraid that enabling network statistics would cause serious
performance
degradation. Thought?

Regards,

--
Fujii Masao

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#26Atri Sharma
atri.jiit@gmail.com
In reply to: Fujii Masao (#25)
Re: stats for network traffic WIP

On Tue, Dec 10, 2013 at 10:59 AM, Fujii Masao <masao.fujii@gmail.com> wrote:

On Tue, Dec 10, 2013 at 6:56 AM, Nigel Heron <nheron@querymetrics.com> wrote:

On Sat, Dec 7, 2013 at 1:17 PM, Fujii Masao <masao.fujii@gmail.com> wrote:

Could you share the performance numbers? I'm really concerned about
the performance overhead caused by this patch.

I've tried pgbench in select mode with small data sets to avoid disk
io and didn't see any difference. That was on my old core2duo laptop
though .. I'll have to retry it on some server class multi core
hardware.

When I ran pgbench -i -s 100 in four parallel, I saw the performance difference
between the master and the patched one. I ran the following commands.

psql -c "checkpoint"
for i in $(seq 1 4); do time pgbench -i -s100 -q db$i & done

The results are:

* Master
10000000 of 10000000 tuples (100%) done (elapsed 13.91 s, remaining 0.00 s).
10000000 of 10000000 tuples (100%) done (elapsed 14.03 s, remaining 0.00 s).
10000000 of 10000000 tuples (100%) done (elapsed 14.01 s, remaining 0.00 s).
10000000 of 10000000 tuples (100%) done (elapsed 14.13 s, remaining 0.00 s).

It took almost 14.0 seconds to store 10000000 tuples.

* Patched
10000000 of 10000000 tuples (100%) done (elapsed 14.90 s, remaining 0.00 s).
10000000 of 10000000 tuples (100%) done (elapsed 15.05 s, remaining 0.00 s).
10000000 of 10000000 tuples (100%) done (elapsed 15.42 s, remaining 0.00 s).
10000000 of 10000000 tuples (100%) done (elapsed 15.70 s, remaining 0.00 s).

It took almost 15.0 seconds to store 10000000 tuples.
--

Regards,

Atri
l'apprenant

Thus, I'm afraid that enabling network statistics would cause serious
performance
degradation. Thought?

Hmm, I think I did not push it this high. The performance numbers here
are cause of worry.

Another point I may mention here is that if we can isolate a few
points of performance degradation and work on them because I still
feel that the entire patch itself does not cause a serious lapse,
rather, a few points may.

However, the above numbers bring up the original concerns for the
performance voiced. I guess I was testing on too low number of clients
for the gap to show up significantly.

Regards,

Atri

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#27Robert Haas
robertmhaas@gmail.com
In reply to: Fujii Masao (#25)
Re: stats for network traffic WIP

On Tue, Dec 10, 2013 at 12:29 AM, Fujii Masao <masao.fujii@gmail.com> wrote:

On Tue, Dec 10, 2013 at 6:56 AM, Nigel Heron <nheron@querymetrics.com> wrote:

On Sat, Dec 7, 2013 at 1:17 PM, Fujii Masao <masao.fujii@gmail.com> wrote:

Could you share the performance numbers? I'm really concerned about
the performance overhead caused by this patch.

I've tried pgbench in select mode with small data sets to avoid disk
io and didn't see any difference. That was on my old core2duo laptop
though .. I'll have to retry it on some server class multi core
hardware.

When I ran pgbench -i -s 100 in four parallel, I saw the performance difference
between the master and the patched one. I ran the following commands.

psql -c "checkpoint"
for i in $(seq 1 4); do time pgbench -i -s100 -q db$i & done

The results are:

* Master
10000000 of 10000000 tuples (100%) done (elapsed 13.91 s, remaining 0.00 s).
10000000 of 10000000 tuples (100%) done (elapsed 14.03 s, remaining 0.00 s).
10000000 of 10000000 tuples (100%) done (elapsed 14.01 s, remaining 0.00 s).
10000000 of 10000000 tuples (100%) done (elapsed 14.13 s, remaining 0.00 s).

It took almost 14.0 seconds to store 10000000 tuples.

* Patched
10000000 of 10000000 tuples (100%) done (elapsed 14.90 s, remaining 0.00 s).
10000000 of 10000000 tuples (100%) done (elapsed 15.05 s, remaining 0.00 s).
10000000 of 10000000 tuples (100%) done (elapsed 15.42 s, remaining 0.00 s).
10000000 of 10000000 tuples (100%) done (elapsed 15.70 s, remaining 0.00 s).

It took almost 15.0 seconds to store 10000000 tuples.

Thus, I'm afraid that enabling network statistics would cause serious
performance
degradation. Thought?

Yes, I think the overhead of this patch is far, far too high to
contemplate applying it. It sends a stats collector message after
*every socket operation*. Once per transaction would likely be too
much overhead already (think: pgbench -S) but once per socket op is
insane.

Moreover, even if we found some way to reduce that overhead to an
acceptable level, I think a lot of people would be unhappy about the
statsfile bloat. Unfortunately, the bottom line here is that, until
someone overhauls the stats collector infrastructure to make
incremental updates to the statsfile cheap, we really can't afford to
add much of anything in the way of new statistics. So I fear this
patch is doomed.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#28Tom Lane
tgl@sss.pgh.pa.us
In reply to: Robert Haas (#27)
Re: stats for network traffic WIP

Robert Haas <robertmhaas@gmail.com> writes:

Yes, I think the overhead of this patch is far, far too high to
contemplate applying it. It sends a stats collector message after
*every socket operation*. Once per transaction would likely be too
much overhead already (think: pgbench -S) but once per socket op is
insane.

Oh, is that what the problem is? That seems trivially fixable --- only
flush the data to the collector once per query or so. I'd be a bit
inclined to add it to the existing transaction-end messages instead of
adding any new traffic.

Moreover, even if we found some way to reduce that overhead to an
acceptable level, I think a lot of people would be unhappy about the
statsfile bloat.

This could be a bigger problem, but what are we aggregating over?
If the stats are only recorded at say the database level, that's not
going to take much space.

Having said that, I can't get very excited about this feature anyway,
so I'm fine with rejecting the patch. I'm not sure that enough people
care to justify any added overhead at all. The long and the short of
it is that network traffic generally is what it is, for any given query
workload, and so it's not clear what's the point of counting it.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#29Peter Eisentraut
peter_e@gmx.net
In reply to: Tom Lane (#28)
Re: stats for network traffic WIP

On 12/10/13, 5:08 PM, Tom Lane wrote:

Having said that, I can't get very excited about this feature anyway,
so I'm fine with rejecting the patch. I'm not sure that enough people
care to justify any added overhead at all. The long and the short of
it is that network traffic generally is what it is, for any given query
workload, and so it's not clear what's the point of counting it.

Also, if we add this, the next guy is going to want to add CPU
statistics, memory statistics, etc.

Is there a reason why you can't get this directly from the OS?

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#30Atri Sharma
atri.jiit@gmail.com
In reply to: Peter Eisentraut (#29)
Re: stats for network traffic WIP

On Wed, Dec 11, 2013 at 11:12 PM, Peter Eisentraut <peter_e@gmx.net> wrote:

On 12/10/13, 5:08 PM, Tom Lane wrote:

Having said that, I can't get very excited about this feature anyway,
so I'm fine with rejecting the patch. I'm not sure that enough people
care to justify any added overhead at all. The long and the short of
it is that network traffic generally is what it is, for any given query
workload, and so it's not clear what's the point of counting it.

Also, if we add this, the next guy is going to want to add CPU
statistics, memory statistics, etc.

Is there a reason why you can't get this directly from the OS?

I would say that its more of a convenience to track the usage directly
from the database instead of setting up OS infrastructure to store it.

That said, it should be possible to directly do it from OS level. Can
we think of adding this to pgtop, though?

I am just musing here.

Regards,

Atri

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#31Tom Lane
tgl@sss.pgh.pa.us
In reply to: Atri Sharma (#30)
Re: stats for network traffic WIP

Atri Sharma <atri.jiit@gmail.com> writes:

On Wed, Dec 11, 2013 at 11:12 PM, Peter Eisentraut <peter_e@gmx.net> wrote:

Is there a reason why you can't get this directly from the OS?

I would say that its more of a convenience to track the usage directly
from the database instead of setting up OS infrastructure to store it.

The thing that I'm wondering is why the database would be the right place
to be measuring it at all. If you've got a network usage problem,
aggregate usage across everything on the server is probably what you
need to be worried about, and PG can't tell you that.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#32Greg Stark
stark@mit.edu
In reply to: Tom Lane (#31)
Re: stats for network traffic WIP

I could see this being interesting for FDW plan nodes of the status were
visible in explain. Possibly also time spent waiting on network reads and
writes.

I have a harder time seeing why it's useful to have these stays in
aggregate but I suppose if you had lots of FDW connections or lots of
steaming slaves you might want to be able to identify which ones are not
getting used or are dominating your network usage.

--
greg
On 11 Dec 2013 10:52, "Tom Lane" <tgl@sss.pgh.pa.us> wrote:

Show quoted text

Atri Sharma <atri.jiit@gmail.com> writes:

On Wed, Dec 11, 2013 at 11:12 PM, Peter Eisentraut <peter_e@gmx.net>

wrote:

Is there a reason why you can't get this directly from the OS?

I would say that its more of a convenience to track the usage directly
from the database instead of setting up OS infrastructure to store it.

The thing that I'm wondering is why the database would be the right place
to be measuring it at all. If you've got a network usage problem,
aggregate usage across everything on the server is probably what you
need to be worried about, and PG can't tell you that.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#33Jim Nasby
jim@nasby.net
In reply to: Tom Lane (#31)
Re: stats for network traffic WIP

On 12/11/13 12:51 PM, Tom Lane wrote:

Atri Sharma <atri.jiit@gmail.com> writes:

On Wed, Dec 11, 2013 at 11:12 PM, Peter Eisentraut <peter_e@gmx.net> wrote:

Is there a reason why you can't get this directly from the OS?

I would say that its more of a convenience to track the usage directly
from the database instead of setting up OS infrastructure to store it.

The thing that I'm wondering is why the database would be the right place
to be measuring it at all. If you've got a network usage problem,
aggregate usage across everything on the server is probably what you
need to be worried about, and PG can't tell you that.

Except how many folks that care about performance that much don't have dedicated database servers?

BTW, since someone mentioned CPU etc, what I'd be interested in is being able to see what OS-level resources were consumed by individual queries. You can already get that to a degree via explain (at least for memory and buffer reads), but it'd be very useful to see what queries are CPU or IO-bound.
--
Jim C. Nasby, Data Architect jim@nasby.net
512.569.9461 (cell) http://jim.nasby.net

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#34Craig Ringer
craig@2ndquadrant.com
In reply to: Tom Lane (#31)
Re: stats for network traffic WIP

On 12/12/2013 02:51 AM, Tom Lane wrote:

The thing that I'm wondering is why the database would be the right place
to be measuring it at all. If you've got a network usage problem,
aggregate usage across everything on the server is probably what you
need to be worried about, and PG can't tell you that.

I suspect this feature would be useful for when you want to try to drill
down and figure out what's having network issues - specifically, to
associate network behaviour with individual queries, individual users,
application_name, etc.

One sometimes faces the same issue with I/O: I know PostgreSQL is doing
lots of I/O, but what exactly is causing the I/O? Especially if you
can't catch it at the time it happens, it can be quite tricky to go from
"there's lots of I/O" to "this query changed from using synchronized
seqscans to doing an index-only scan that's hammering the cache".

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#35Stephen Frost
sfrost@snowman.net
In reply to: Craig Ringer (#34)
Re: stats for network traffic WIP

* Craig Ringer (craig@2ndquadrant.com) wrote:

On 12/12/2013 02:51 AM, Tom Lane wrote:

The thing that I'm wondering is why the database would be the right place
to be measuring it at all. If you've got a network usage problem,
aggregate usage across everything on the server is probably what you
need to be worried about, and PG can't tell you that.

I suspect this feature would be useful for when you want to try to drill
down and figure out what's having network issues - specifically, to
associate network behaviour with individual queries, individual users,
application_name, etc.

One sometimes faces the same issue with I/O: I know PostgreSQL is doing
lots of I/O, but what exactly is causing the I/O? Especially if you
can't catch it at the time it happens, it can be quite tricky to go from
"there's lots of I/O" to "this query changed from using synchronized
seqscans to doing an index-only scan that's hammering the cache".

Agreed. My other thought on this is that there's a lot to be said for
having everything you need available through one tool- kinda like how
Emacs users rarely go outside of it.. :) And then there's also the
consideration that DBAs may not have access to the host system at all,
or not to the level needed to do similar analysis there.

Thanks,

Stephen

#36Robert Haas
robertmhaas@gmail.com
In reply to: Stephen Frost (#35)
Re: stats for network traffic WIP

On Wed, Dec 18, 2013 at 8:47 AM, Stephen Frost <sfrost@snowman.net> wrote:

Agreed. My other thought on this is that there's a lot to be said for
having everything you need available through one tool- kinda like how
Emacs users rarely go outside of it.. :) And then there's also the
consideration that DBAs may not have access to the host system at all,
or not to the level needed to do similar analysis there.

I completely agree with this, and yet I still think we should reject
the patch, because I think the overhead is going to be intolerable.

Now, the fact is, the monitoring facilities we have in PostgreSQL
today are not nearly good enough. Other products do better. I cringe
every time I tell someone to attach strace to a long-running autovac
process to find out what block number it's currently on, so we can
estimate when it will finish; or every time we need data about lwlock
contention and the only way to get it is to use perf, or recompile
with LWLOCK_STATS defined. These are not fun conversations to have
with customers who are in production.

On the other hand, there's not much value in adding monitoring
features that are going to materially harm performance, and a lot of
the monitoring features that get proposed die on the vine for exactly
that reason. I think the root of the problem is that our stats
infrastructure is a streaming pile of crap. A number of people have
worked diligently to improve it and that work has not been fruitless,
but the current situation is still not very good. In many ways, this
situation reminds me of the situation with EXPLAIN a few years ago.
People kept proposing useful extensions to EXPLAIN which we did not
adopt because they required creating (and perhaps reserving) far too
many keywords. Now that we have the extensible options syntax,
EXPLAIN has options for COSTS, BUFFERS, TIMING, and FORMAT, all of
which have proven to be worth their weight in code, at least IMHO.

I am really not sure what a better infrastructure for stats collection
should look like, but I know that until we get one, a lot of
monitoring patches that would be really nice to have are going to get
shot down because of concerns about performance, and specifically
stats file bloat. Fixing that problem figures to be unglamorous, but
I'll buy whoever does it a beer (or another beverage of your choice).

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#37Stephen Frost
sfrost@snowman.net
In reply to: Robert Haas (#36)
Re: stats for network traffic WIP

* Robert Haas (robertmhaas@gmail.com) wrote:

On Wed, Dec 18, 2013 at 8:47 AM, Stephen Frost <sfrost@snowman.net> wrote:

Agreed. My other thought on this is that there's a lot to be said for
having everything you need available through one tool- kinda like how
Emacs users rarely go outside of it.. :) And then there's also the
consideration that DBAs may not have access to the host system at all,
or not to the level needed to do similar analysis there.

I completely agree with this, and yet I still think we should reject
the patch, because I think the overhead is going to be intolerable.

That's a fair point and I'm fine with rejecting it on the grounds that
the overhead is too much. Hopefully that encourages the author to go
back and review Tom's comments and consider how the overhead could be
reduced or eliminated. We absolutely need better monitoring and I have
had many of the same strace-involving conversations. perf is nearly out
of the question as it's often not even installed and can be terribly
risky (I once had to get a prod box hard-reset after running perf on it
for mere moments because it never came back enough to let us do a clean
restart).

I think the root of the problem is that our stats
infrastructure is a streaming pile of crap.

+1

Thanks,

Stephen

#38Bruce Momjian
bruce@momjian.us
In reply to: Robert Haas (#36)
Re: stats for network traffic WIP

On Wed, Dec 18, 2013 at 03:41:24PM -0500, Robert Haas wrote:

On the other hand, there's not much value in adding monitoring
features that are going to materially harm performance, and a lot of
the monitoring features that get proposed die on the vine for exactly
that reason. I think the root of the problem is that our stats
infrastructure is a streaming pile of crap. A number of people have

"streaming"? I can't imagine what that looks like. ;-)

I think the larger point is that network is only one of many things we
need to address, so this needs a holistic approach that looks at all
needs and creates infrastructure to address it.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ Everyone has their own god. +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers