max_slot_wal_keep_size unit is not specified

On 2021-Nov-22, PG Doc comments form wrote:

The unit (I assume it's MB) of max_slot_wal_keep_size is not explicitly
specified in the docs. If it's intentional then please, disregards, but I've
not been able to deduct that from this page alone. Thanks!

Well, that's embarrasing. I'll see about fixing it.

It is a size-based unit. You would typically specify some unit (say,
MB) together with the number; internally, because the way this works is
in terms of whole files, it is rounded down to an integer number of WAL
segments. If you don't specify a unit, it is taken to be a number of
megabytes.

I wonder why did we make it round down rather than up. Does this mean
that if you have max_slot_wal_keep_size=8MB and wal segments of 16 MB,
the slot gets invalidated with more than zero reserved segments?

Thanks for reporting.

--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"I'm impressed how quickly you are fixing this obscure issue. I came from
MS SQL and it would be hard for me to put into words how much of a better job
you all are doing on [PostgreSQL]."
Steve Midgley, http://archives.postgresql.org/pgsql-sql/2008-08/msg00000.php

At Tue, 23 Nov 2021 11:21:05 -0300, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote in

On 2021-Nov-22, PG Doc comments form wrote:

The unit (I assume it's MB) of max_slot_wal_keep_size is not explicitly
specified in the docs. If it's intentional then please, disregards, but I've
not been able to deduct that from this page alone. Thanks!

Well, that's embarrasing. I'll see about fixing it.

(facepalm..)

It is a size-based unit. You would typically specify some unit (say,
MB) together with the number; internally, because the way this works is
in terms of whole files, it is rounded down to an integer number of WAL
segments. If you don't specify a unit, it is taken to be a number of
megabytes.

I wonder why did we make it round down rather than up. Does this mean
that if you have max_slot_wal_keep_size=8MB and wal segments of 16 MB,
the slot gets invalidated with more than zero reserved segments?

That rounding behavior is the way we are generally doing on similar
variables. It is based on the behavior of ConvertToXSegs(). So
max_wal_size for example is rounded-up the same way.

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center

On 2021-Nov-24, Kyotaro Horiguchi wrote:

At Tue, 23 Nov 2021 11:21:05 -0300, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote in

I wonder why did we make it round down rather than up. Does this mean
that if you have max_slot_wal_keep_size=8MB and wal segments of 16 MB,
the slot gets invalidated with more than zero reserved segments?

That rounding behavior is the way we are generally doing on similar
variables. It is based on the behavior of ConvertToXSegs(). So
max_wal_size for example is rounded-up the same way.

After thinking on it some more, I think rounding down is correct. If
somebody sets max_slot_wal_keep_size=40MB, then by keeping the limit at
32 MB (round down) we're honoring that request better than if we made it
48 MB (round up). But in any case, I think it doesn't matter much:
users need a *rough* size limit, not a super-precise one. After all,
this is only verified at checkpoint time. We're probably dealing with a
few hundreds of megabytes at least, so 16 MB one way or the other don't
change things much.

Here's a proposed patch.

--
Álvaro Herrera 39°49'30"S 73°17'W — https://www.EnterpriseDB.com/
"Las mujeres son como hondas: mientras más resistencia tienen,
más lejos puedes llegar con ellas" (Jonas Nightingale, Leap of Faith)

At Thu, 25 Nov 2021 11:37:37 -0300, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote in

On 2021-Nov-24, Kyotaro Horiguchi wrote:

At Tue, 23 Nov 2021 11:21:05 -0300, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote in

I wonder why did we make it round down rather than up. Does this mean
that if you have max_slot_wal_keep_size=8MB and wal segments of 16 MB,
the slot gets invalidated with more than zero reserved segments?

That rounding behavior is the way we are generally doing on similar
variables. It is based on the behavior of ConvertToXSegs(). So
max_wal_size for example is rounded-up the same way.

(no.. rounded-down X)

After thinking on it some more, I think rounding down is correct. If
somebody sets max_slot_wal_keep_size=40MB, then by keeping the limit at
32 MB (round down) we're honoring that request better than if we made it
48 MB (round up). But in any case, I think it doesn't matter much:
users need a *rough* size limit, not a super-precise one. After all,
this is only verified at checkpoint time. We're probably dealing with a
few hundreds of megabytes at least, so 16 MB one way or the other don't
change things much.

I thought exactly on that line at development.

Regarding the proposed description, we have never explained how the
same kind of values specified in megabytes is converted into internal
units. If we add that explanation there, we might want the same for
max_wal_size, min_wal_size and wal_keep_size.

Here's a proposed patch.

+        The value is rounded down to an integer number of WAL files
+        according to <varname>wal_segment_size</varname>.

The same section starts with the following sentences.

Specify the maximum size of WAL files
that <link linkend="streaming-replication-slots">replication
slots</link> are allowed to retain in the <filename>pg_wal</filename>
directory at checkpoint time.

FWIW I thought (or intended) that "the maximum size of WAL files that
slots are *allowed to retain*" connotes the same to the sentences
cited above.

Premising "16 MB one way or the other don't change things much", I'm
not sure we really need to be particular about a few megabites
difference especially only for this specific variable.

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center

On 2021-Nov-26, Kyotaro Horiguchi wrote:

Regarding the proposed description, we have never explained how the
same kind of values specified in megabytes is converted into internal
units. If we add that explanation there, we might want the same for
max_wal_size, min_wal_size and wal_keep_size.

True. Since this was for backpatch, it seemed best to copy the existing
entries instead of changing them.

The same section starts with the following sentences.

Specify the maximum size of WAL files
that <link linkend="streaming-replication-slots">replication
slots</link> are allowed to retain in the <filename>pg_wal</filename>
directory at checkpoint time.

FWIW I thought (or intended) that "the maximum size of WAL files that
slots are *allowed to retain*" connotes the same to the sentences
cited above.

Right you are -- that addition was redundant, so I pushed without that.

Premising "16 MB one way or the other don't change things much", I'm
not sure we really need to be particular about a few megabites
difference especially only for this specific variable.

Yeah.

--
Álvaro Herrera Valdivia, Chile — https://www.EnterpriseDB.com/