doc: New cumulative stats subsystem obsoletes comment in maintenance.sgml

Started by David G. Johnstonover 3 years ago8 messages

david.g.johnston@gmail.com

over 3 years ago

Hackers,

The new cumulative stats subsystem no longer has a "lost under heavy load"
problem so that parenthetical should go (or at least be modified).

These stats can be reset so some discussion about how the system uses them
given that possibility seems like it would be good to add here. I'm not
sure what that should look like though.

David J.

diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml
index 04a04e0e5f..360807c8f9 100644
--- a/doc/src/sgml/maintenance.sgml
+++ b/doc/src/sgml/maintenance.sgml
@@ -652,9 +652,8 @@ vacuum insert threshold = vacuum base insert threshold
+ vacuum insert scale fac
     tuples to be frozen by earlier vacuums.  The number of obsolete tuples
and
     the number of inserted tuples are obtained from the cumulative
statistics system;
     it is a semi-accurate count updated by each <command>UPDATE</command>,
-    <command>DELETE</command> and <command>INSERT</command> operation.
 (It is
-    only semi-accurate because some information might be lost under heavy
-    load.)  If the <structfield>relfrozenxid</structfield> value of the
table
+    <command>DELETE</command> and <command>INSERT</command> operation.
+    If the <structfield>relfrozenxid</structfield> value of the table
     is more than <varname>vacuum_freeze_table_age</varname> transactions
old,
     an aggressive vacuum is performed to freeze old tuples and advance
     <structfield>relfrozenxid</structfield>; otherwise, only pages that
have been modified

Bruce Momjian

bruce@momjian.us

over 3 years ago

In reply to: David G. Johnston (#1)

Re: doc: New cumulative stats subsystem obsoletes comment in maintenance.sgml

On Wed, Apr 20, 2022 at 04:40:44PM -0700, David G. Johnston wrote:

Hackers,

The new cumulative stats subsystem no longer has a "lost under heavy load"
problem so that parenthetical should go (or at least be modified).

These stats can be reset so some discussion about how the system uses them
given that possibility seems like it would be good to add here. I'm not sure
what that should look like though.

diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml
index 04a04e0e5f..360807c8f9 100644
--- a/doc/src/sgml/maintenance.sgml
+++ b/doc/src/sgml/maintenance.sgml
@@ -652,9 +652,8 @@ vacuum insert threshold = vacuum base insert threshold +
vacuum insert scale fac
     tuples to be frozen by earlier vacuums.  The number of obsolete tuples and
     the number of inserted tuples are obtained from the cumulative statistics
system;
     it is a semi-accurate count updated by each <command>UPDATE</command>,
-    <command>DELETE</command> and <command>INSERT</command> operation.  (It is
-    only semi-accurate because some information might be lost under heavy
-    load.)  If the <structfield>relfrozenxid</structfield> value of the table
+    <command>DELETE</command> and <command>INSERT</command> operation.
+    If the <structfield>relfrozenxid</structfield> value of the table
     is more than <varname>vacuum_freeze_table_age</varname> transactions old,
     an aggressive vacuum is performed to freeze old tuples and advance
     <structfield>relfrozenxid</structfield>; otherwise, only pages that have
been modified

Yes, I agree and plan to apply this patch soon.

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com

Indecision is a decision. Inaction is an action. Mark Batterson

Andres Freund

andres@anarazel.de

over 3 years ago

In reply to: Bruce Momjian (#2)

Re: doc: New cumulative stats subsystem obsoletes comment in maintenance.sgml

Hi,

I had missed David's original email on this topic...

On 2022-07-14 18:58:09 -0400, Bruce Momjian wrote:

On Wed, Apr 20, 2022 at 04:40:44PM -0700, David G. Johnston wrote:

The new cumulative stats subsystem no longer has a "lost under heavy load"
problem so that parenthetical should go (or at least be modified).

These stats can be reset so some discussion about how the system uses them
given that possibility seems like it would be good to add here.ï¿½ I'm not sure
whatï¿½that should look like though.

diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml
index 04a04e0e5f..360807c8f9 100644
--- a/doc/src/sgml/maintenance.sgml
+++ b/doc/src/sgml/maintenance.sgml
@@ -652,9 +652,8 @@ vacuum insert threshold = vacuum base insert threshold +
vacuum insert scale fac
ï¿½ ï¿½ ï¿½tuples to be frozen by earlier vacuums.ï¿½ The number of obsolete tuples and
ï¿½ ï¿½ ï¿½the number of inserted tuples are obtained from the cumulative statistics
system;
ï¿½ ï¿½ ï¿½it is a semi-accurate count updated by each <command>UPDATE</command>,
- ï¿½ ï¿½<command>DELETE</command> and <command>INSERT</command> operation. ï¿½(It is
- ï¿½ ï¿½only semi-accurate because some information might be lost under heavy
- ï¿½ ï¿½load.) ï¿½If the <structfield>relfrozenxid</structfield> value of the table
+ ï¿½ ï¿½<command>DELETE</command> and <command>INSERT</command> operation.
+ ï¿½ ï¿½If the <structfield>relfrozenxid</structfield> value of the table
ï¿½ ï¿½ ï¿½is more than <varname>vacuum_freeze_table_age</varname> transactions old,
ï¿½ ï¿½ ï¿½an aggressive vacuum is performed to freeze old tuples and advance
ï¿½ ï¿½ ï¿½<structfield>relfrozenxid</structfield>; otherwise, only pages that have
been modified

Yes, I agree and plan to apply this patch soon.

It might make sense to still say semi-accurate, but adjust the explanation to
say that stats reporting is not instantaneous?

Greetings,

Andres Freund

David G. Johnston

david.g.johnston@gmail.com

over 3 years ago

In reply to: Andres Freund (#3)

Re: doc: New cumulative stats subsystem obsoletes comment in maintenance.sgml

On Thu, Jul 14, 2022 at 4:31 PM Andres Freund <andres@anarazel.de> wrote:

Hi,

I had missed David's original email on this topic...

On 2022-07-14 18:58:09 -0400, Bruce Momjian wrote:

On Wed, Apr 20, 2022 at 04:40:44PM -0700, David G. Johnston wrote:

The new cumulative stats subsystem no longer has a "lost under heavy

load"

problem so that parenthetical should go (or at least be modified).

These stats can be reset so some discussion about how the system uses

them

given that possibility seems like it would be good to add here. I'm

not sure
what that should look like though.
diff --git a/doc/src/sgml/maintenance.sgml
b/doc/src/sgml/maintenance.sgml
index 04a04e0e5f..360807c8f9 100644
--- a/doc/src/sgml/maintenance.sgml
+++ b/doc/src/sgml/maintenance.sgml
@@ -652,9 +652,8 @@ vacuum insert threshold = vacuum base insert
threshold +

vacuum insert scale fac
tuples to be frozen by earlier vacuums. The number of obsolete

tuples and

the number of inserted tuples are obtained from the cumulative

statistics

system;
it is a semi-accurate count updated by each

<command>UPDATE</command>,

- <command>DELETE</command> and <command>INSERT</command>

operation. (It is

- only semi-accurate because some information might be lost under

heavy

- load.) If the <structfield>relfrozenxid</structfield> value of

the table
+    <command>DELETE</command> and <command>INSERT</command> operation.
+    If the <structfield>relfrozenxid</structfield> value of the table
is more than <varname>vacuum_freeze_table_age</varname>
transactions old,

an aggressive vacuum is performed to freeze old tuples and advance
<structfield>relfrozenxid</structfield>; otherwise, only pages

that have

been modified

Yes, I agree and plan to apply this patch soon.

It might make sense to still say semi-accurate, but adjust the explanation
to
say that stats reporting is not instantaneous?

Unless that delay manifests in executing an UPDATE in a session then
looking at these views in the same session and not seeing that update
reflected I wouldn't mention it. Concurrency aspects are reasonably
expected here. But if we do want to mention it maybe:

"...it is an eventually-consistent count updated by..."

David J.

Andres Freund

andres@anarazel.de

over 3 years ago

In reply to: David G. Johnston (#4)

Re: doc: New cumulative stats subsystem obsoletes comment in maintenance.sgml

Hi,

On 2022-07-18 19:47:39 -0700, David G. Johnston wrote:

On Thu, Jul 14, 2022 at 4:31 PM Andres Freund <andres@anarazel.de> wrote:

It might make sense to still say semi-accurate, but adjust the explanation
to
say that stats reporting is not instantaneous?

Unless that delay manifests in executing an UPDATE in a session then
looking at these views in the same session and not seeing that update
reflected I wouldn't mention it.

Depending on which stats you're looking at, yes, that could totally happen. I
don't think the issue is not seeing changes from the current transaction
though - it's that *after* commit you might not see them for a while (the're
transmitted not more than once a second, and can be delayed up to 60s if
there's contention).

Greetings,

Andres Freund

Bruce Momjian

bruce@momjian.us

over 3 years ago

In reply to: Andres Freund (#5)

Re: doc: New cumulative stats subsystem obsoletes comment in maintenance.sgml

On Mon, Jul 18, 2022 at 08:04:12PM -0700, Andres Freund wrote:

Hi,

On 2022-07-18 19:47:39 -0700, David G. Johnston wrote:

On Thu, Jul 14, 2022 at 4:31 PM Andres Freund <andres@anarazel.de> wrote:

It might make sense to still say semi-accurate, but adjust the explanation
to
say that stats reporting is not instantaneous?

Unless that delay manifests in executing an UPDATE in a session then
looking at these views in the same session and not seeing that update
reflected I wouldn't mention it.

Depending on which stats you're looking at, yes, that could totally happen. I
don't think the issue is not seeing changes from the current transaction
though - it's that *after* commit you might not see them for a while (the're
transmitted not more than once a second, and can be delayed up to 60s if
there's contention).

So the docs don't need any changes, I assume.

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com

Indecision is a decision. Inaction is an action. Mark Batterson

David G. Johnston

david.g.johnston@gmail.com

over 3 years ago

In reply to: Bruce Momjian (#6)

Re: doc: New cumulative stats subsystem obsoletes comment in maintenance.sgml

On Fri, Aug 12, 2022 at 12:48 PM Bruce Momjian <bruce@momjian.us> wrote:

On Mon, Jul 18, 2022 at 08:04:12PM -0700, Andres Freund wrote:

Hi,

On 2022-07-18 19:47:39 -0700, David G. Johnston wrote:

On Thu, Jul 14, 2022 at 4:31 PM Andres Freund <andres@anarazel.de>

wrote:

It might make sense to still say semi-accurate, but adjust the

explanation

to
say that stats reporting is not instantaneous?

Unless that delay manifests in executing an UPDATE in a session then
looking at these views in the same session and not seeing that update
reflected I wouldn't mention it.

Depending on which stats you're looking at, yes, that could totally

happen. I

don't think the issue is not seeing changes from the current transaction
though - it's that *after* commit you might not see them for a while

(the're

transmitted not more than once a second, and can be delayed up to 60s if
there's contention).

So the docs don't need any changes, I assume.

I dislike using the word accurate here now, it will be accurate, but we
don't promise perfect timeliness. So it needs to change:

 diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml
index 04a04e0e5f..360807c8f9 100644
--- a/doc/src/sgml/maintenance.sgml
+++ b/doc/src/sgml/maintenance.sgml
@@ -652,9 +652,8 @@ vacuum insert threshold = vacuum base insert threshold
+ vacuum insert scale fac
     tuples to be frozen by earlier vacuums.  The number of obsolete tuples
and
     the number of inserted tuples are obtained from the cumulative
statistics system;
     it is a semi-accurate count updated by each <command>UPDATE</command>,
-    <command>DELETE</command> and <command>INSERT</command> operation.
 (It is
-    only semi-accurate because some information might be lost under heavy
-    load.)  If the <structfield>relfrozenxid</structfield> value of the
table
+    <command>DELETE</command> and <command>INSERT</command> operation.
+    If the <structfield>relfrozenxid</structfield> value of the table
     is more than <varname>vacuum_freeze_table_age</varname> transactions
old,
     an aggressive vacuum is performed to freeze old tuples and advance
     <structfield>relfrozenxid</structfield>; otherwise, only pages that
have been modified

However, also replace the remaining instance of "a semi-accurate count"
with "an eventually-consistent count".

...it is an eventually-consistent count updated by each UPDATE, DELETE, and
INSERT operation.

David J.

Bruce Momjian

bruce@momjian.us

about 2 years ago

In reply to: David G. Johnston (#7)

Re: doc: New cumulative stats subsystem obsoletes comment in maintenance.sgml

On Fri, Aug 12, 2022 at 12:58:28PM -0700, David G. Johnston wrote:

I dislike using the word accurate here now, it will be accurate, but we don't
promise perfect timeliness. So it needs to change:

 diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml
index 04a04e0e5f..360807c8f9 100644
--- a/doc/src/sgml/maintenance.sgml
+++ b/doc/src/sgml/maintenance.sgml
@@ -652,9 +652,8 @@ vacuum insert threshold = vacuum base insert threshold +
vacuum insert scale fac
     tuples to be frozen by earlier vacuums.  The number of obsolete tuples and
     the number of inserted tuples are obtained from the cumulative statistics
system;
     it is a semi-accurate count updated by each <command>UPDATE</command>,
-    <command>DELETE</command> and <command>INSERT</command> operation.  (It is
-    only semi-accurate because some information might be lost under heavy
-    load.)  If the <structfield>relfrozenxid</structfield> value of the table
+    <command>DELETE</command> and <command>INSERT</command> operation.
+    If the <structfield>relfrozenxid</structfield> value of the table
     is more than <varname>vacuum_freeze_table_age</varname> transactions old,
     an aggressive vacuum is performed to freeze old tuples and advance
     <structfield>relfrozenxid</structfield>; otherwise, only pages that have
been modified

Done in master.

However, also replace the remaining instance of "a semi-accurate count" with
"an eventually-consistent count".

...it is an eventually-consistent count updated by each UPDATE, DELETE, and
INSERT operation.

Also done.

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com

Only you can decide what is important to you.