Exposing currentTransactionWALVolume

Started by Simon Riggsalmost 12 years ago6 messages

simon@2ndQuadrant.com

almost 12 years ago

1 attachment(s)

Short patch to expose a function GetCurrentTransactionWALVolume() that
gives the total number of bytes written to WAL by current transaction.

User interface to this information discussed on separate thread, so
that we don't check the baby out with the bathwater when people
discuss UI pros and cons.

--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Attachments:

GetCurrentTransactionWALVolume.v1.patchapplication/octet-stream; name=GetCurrentTransactionWALVolume.v1.patchDownload

diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 0487be1..22adedd 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -183,6 +183,12 @@ static TransactionStateData TopTransactionStateData = {
 };
 
 /*
+ * Track total volume of WAL written by current top-level transaction
+ * to allow tracking, reporting and control of writing WAL.
+ */
+static uint64 currentTransactionWALVolume;
+
+/*
  * unreportedXids holds XIDs of all subtransactions that have not yet been
  * reported in a XLOG_XACT_ASSIGNMENT record.
  */
@@ -397,17 +403,26 @@ GetCurrentTransactionIdIfAny(void)
 }
 
 /*
- *	MarkCurrentTransactionIdLoggedIfAny
+ * ReportTransactionInsertedWAL
  *
- * Remember that the current xid - if it is assigned - now has been wal logged.
+ * Remember that the current xid - if it is assigned - has now inserted WAL
  */
 void
-MarkCurrentTransactionIdLoggedIfAny(void)
+ReportTransactionInsertedWAL(uint32 insertedWALVolume)
 {
+	currentTransactionWALVolume += insertedWALVolume;
 	if (TransactionIdIsValid(CurrentTransactionState->transactionId))
 		CurrentTransactionState->didLogXid = true;
 }
 
+/*
+ * GetCurrentTransactionWALVolume
+ */
+uint64
+GetCurrentTransactionWALVolume(void)
+{
+	return currentTransactionWALVolume;
+}
 
 /*
  *	GetStableLatestTransactionId
@@ -1772,6 +1787,7 @@ StartTransaction(void)
 	/*
 	 * initialize reported xid accounting
 	 */
+	currentTransactionWALVolume = 0;
 	nUnreportedXids = 0;
 	s->didLogXid = false;
 
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index b53ae87..7255aa0 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -1192,6 +1192,7 @@ begin:;
 		 * already exactly at the beginning of a segment, so there was no need
 		 * to do anything.
 		 */
+		write_len = 0;
 	}
 
 	/*
@@ -1199,7 +1200,7 @@ begin:;
 	 */
 	WALInsertSlotRelease();
 
-	MarkCurrentTransactionIdLoggedIfAny();
+	ReportTransactionInsertedWAL(write_len);
 
 	END_CRIT_SECTION();
 
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 634f5b2..31ca1ba 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -215,7 +215,8 @@ extern TransactionId GetCurrentTransactionId(void);
 extern TransactionId GetCurrentTransactionIdIfAny(void);
 extern TransactionId GetStableLatestTransactionId(void);
 extern SubTransactionId GetCurrentSubTransactionId(void);
-extern void MarkCurrentTransactionIdLoggedIfAny(void);
+extern void ReportTransactionInsertedWAL(uint32 insertedWALVolume);
+extern uint64 GetCurrentTransactionWALVolume(void);
 extern bool SubTransactionIsActive(SubTransactionId subxid);
 extern CommandId GetCurrentCommandId(bool used);
 extern TimestampTz GetCurrentTransactionStartTimestamp(void);

Fujii Masao

masao.fujii@gmail.com

almost 12 years ago

In reply to: Simon Riggs (#1)

Re: Exposing currentTransactionWALVolume

On Wed, Jan 15, 2014 at 2:12 AM, Simon Riggs <simon@2ndquadrant.com> wrote:

Short patch to expose a function GetCurrentTransactionWALVolume() that
gives the total number of bytes written to WAL by current transaction.

Could you tell me the use case of this function? ISTM that it's less
useful in the real
world because it reports the information of WAL generated only in my own current
transaction. For example, the monitoring tool cannot see that
information because
it's the different session from the target.

Regards,

--
Fujii Masao

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Mitsumasa KONDO

kondo.mitsumasa@gmail.com

almost 12 years ago

In reply to: Simon Riggs (#1)

Re: Exposing currentTransactionWALVolume

I send you my review comment.

2014-01-15 Simon Riggs <simon@2ndquadrant.com>:

Short patch to expose a function GetCurrentTransactionWALVolume() that
gives the total number of bytes written to WAL by current transaction.

* It's simple and good feature. It is useful for system management, and
forecasting
server spec(especially disk volume) on the system needed.

* Overhead is nothing unless my test.

* Compile and unit tests are no problem.

User interface to this information discussed on separate thread, so

that we don't check the baby out with the bathwater when people
discuss UI pros and cons.

Did you get good opinion in other thread?
I'd like to use seeing WAL volume sql and init value of WAL volume sql.
Your patch seems to init the value when start up the server now.
If we have init function, we can see database activities in each hours in a
day from WAL volumes.
Now, we only see number of transactions and database volumes.
I'd like to see more detail activities from WAL volume in each minutes or
hours.
It might be good for performance improvement by hackers, too

Regards,
--
Mitsumasa KONDO
NTT Open Source Software Center

Simon Riggs

simon@2ndQuadrant.com

almost 12 years ago

In reply to: Fujii Masao (#2)

Re: Exposing currentTransactionWALVolume

On 31 January 2014 13:56, Fujii Masao <masao.fujii@gmail.com> wrote:

On Wed, Jan 15, 2014 at 2:12 AM, Simon Riggs <simon@2ndquadrant.com> wrote:

Short patch to expose a function GetCurrentTransactionWALVolume() that
gives the total number of bytes written to WAL by current transaction.

Could you tell me the use case of this function? ISTM that it's less
useful in the real
world because it reports the information of WAL generated only in my own current
transaction. For example, the monitoring tool cannot see that
information because
it's the different session from the target.

It's already possible to work out how much total WAL has been
generated. Monitoring tools may access that.

What this allows is user/plugin code to see how much WAL has been
generated in this transaction and take different user defined actions.
Those actions would be specific to the transaction, so this is more
about production control applications than overall system monitoring.
There are many possible ways to use that knowledge.

--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Andres Freund

andres@2ndquadrant.com

almost 12 years ago

In reply to: Simon Riggs (#1)

Re: Exposing currentTransactionWALVolume

Hi Simon,

On 2014-01-14 17:12:35 +0000, Simon Riggs wrote:

/*
- *	MarkCurrentTransactionIdLoggedIfAny
+ * ReportTransactionInsertedWAL
*
- * Remember that the current xid - if it is assigned - now has been wal logged.
+ * Remember that the current xid - if it is assigned - has now inserted WAL
*/
void
-MarkCurrentTransactionIdLoggedIfAny(void)
+ReportTransactionInsertedWAL(uint32 insertedWALVolume)
{
+	currentTransactionWALVolume += insertedWALVolume;
if (TransactionIdIsValid(CurrentTransactionState->transactionId))
CurrentTransactionState->didLogXid = true;
}

Not a big fan of combining those two. One works on the toplevel
transaction, the other on the current subtransaction... The new name
also ignores that it's only taking effect if there's actually a
transaction in progress.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

KONDO Mitsumasa

kondo.mitsumasa@lab.ntt.co.jp

almost 12 years ago

In reply to: Andres Freund (#5)

Re: Exposing currentTransactionWALVolume

(2014/02/15 23:04), Andres Freund wrote:

Hi Simon,

On 2014-01-14 17:12:35 +0000, Simon Riggs wrote:
/*
- *	MarkCurrentTransactionIdLoggedIfAny
+ * ReportTransactionInsertedWAL
*
- * Remember that the current xid - if it is assigned - now has been wal logged.
+ * Remember that the current xid - if it is assigned - has now inserted WAL
*/
void
-MarkCurrentTransactionIdLoggedIfAny(void)
+ReportTransactionInsertedWAL(uint32 insertedWALVolume)
{
+	currentTransactionWALVolume += insertedWALVolume;
if (TransactionIdIsValid(CurrentTransactionState->transactionId))
CurrentTransactionState->didLogXid = true;
}
Not a big fan of combining those two. One works on the toplevel
transaction, the other on the current subtransaction... The new name
also ignores that it's only taking effect if there's actually a
transaction in progress.

Oh, yes. I don't have good idea, but we need to change function name or add new
function for WAL adding volume. If it will be fixed, I set ready for commiter,
because I cannot see any bad point in this patch.

Regards,
--
Mitsumasa KONDO
NTT Open Source Software Center

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers