Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

Started by Bharath Rupireddyabout 4 years ago34 messages
#1Bharath Rupireddy
bharath.rupireddyforpostgres@gmail.com

Hi,

Currently one can know the kind of on-going/last checkpoint (shutdown,
end-of-recovery, immediate, force etc.) only via server logs that to
when log_checkpoints GUC is on. At times, the users/service layer
components would want to know the kind of checkpoint (along with other
checkpoint related info) to take some actions and it will be a bit
difficult to search through the server logs. The checkpoint info can
be obtained from the control file (either by pg_control_checkpoint()
or by pg_controldata tool) whereas checkpoint kind isn't available
there.

How about we add an extra string field to the control file alongside
the other checkpoint info it already has? This way, the existing
pg_control_checkpoint() or pg_controldata tool can easily be enhanced
to show the checkpoint kind as well. One concern is that we don't want
to increase the size of pg_controldata by more than the typical block
size (of 8K) to avoid any torn-writes. With this change, we might add
at max the strings specified at [1]for checkpoint: "checkpoint shutdown end-of-recovery immediate force wait wal time flush-all" for restartpoint: "restartpoint shutdown end-of-recovery immediate force wait wal time flush-all". Adding it to the control file has
an advantage of preserving the last checkpoint kind which might be
useful.

Thoughts?

[1]: for checkpoint: "checkpoint shutdown end-of-recovery immediate force wait wal time flush-all" for restartpoint: "restartpoint shutdown end-of-recovery immediate force wait wal time flush-all"
force wait wal time flush-all"
for restartpoint: "restartpoint shutdown end-of-recovery immediate
force wait wal time flush-all"

Regards,
Bharath Rupireddy.

#2Tomas Vondra
tomas.vondra@enterprisedb.com
In reply to: Bharath Rupireddy (#1)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

On 12/7/21 15:36, Bharath Rupireddy wrote:

Hi,

Currently one can know the kind of on-going/last checkpoint (shutdown,
end-of-recovery, immediate, force etc.) only via server logs that to
when log_checkpoints GUC is on. At times, the users/service layer
components would want to know the kind of checkpoint (along with other
checkpoint related info) to take some actions and it will be a bit
difficult to search through the server logs. The checkpoint info can
be obtained from the control file (either by pg_control_checkpoint()
or by pg_controldata tool) whereas checkpoint kind isn't available
there.

How about we add an extra string field to the control file alongside
the other checkpoint info it already has? This way, the existing
pg_control_checkpoint() or pg_controldata tool can easily be enhanced
to show the checkpoint kind as well. One concern is that we don't want
to increase the size of pg_controldata by more than the typical block
size (of 8K) to avoid any torn-writes. With this change, we might add
at max the strings specified at [1]. Adding it to the control file has
an advantage of preserving the last checkpoint kind which might be
useful.

Thoughts?

I agree it might be useful to provide information about the nature of
the checkpoint, and perhaps even PID of the backend that triggered it
(although that may be tricky, if the backend terminates).

I'm not sure about adding it to control data, though. That doesn't seem
like a very good match for something that's mostly for monitoring.

We already have some checkpoint info in pg_stat_bgwriter, but that's
just aggregated data, not very convenient for info about the current
checkpoint. So maybe we should add pg_stat_progress_checkpoint, showing
various info about the current checkpoint?

[1] for checkpoint: "checkpoint shutdown end-of-recovery immediate
force wait wal time flush-all"
for restartpoint: "restartpoint shutdown end-of-recovery immediate
force wait wal time flush-all"

I'd bet squashing all of this into a single string (not really a flag)
will just mean people will have to parse it, etc. Keeping individual
boolean flags (or even separate string fields) would be better, I guess.

regards

--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

#3Justin Pryzby
pryzby@telsasoft.com
In reply to: Tomas Vondra (#2)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

On Tue, Dec 07, 2021 at 11:18:37PM +0100, Tomas Vondra wrote:

On 12/7/21 15:36, Bharath Rupireddy wrote:

Currently one can know the kind of on-going/last checkpoint (shutdown,
end-of-recovery, immediate, force etc.) only via server logs that to
when log_checkpoints GUC is on.

The checkpoint info can be obtained from the control file (either by
pg_control_checkpoint() or by pg_controldata tool) whereas checkpoint kind
isn't available there.

How about we add an extra string field to the control file alongside
the other checkpoint info it already has? This way, the existing
pg_control_checkpoint() or pg_controldata tool can easily be enhanced
to show the checkpoint kind as well. One concern is that we don't want
to increase the size of pg_controldata by more than the typical block
size (of 8K) to avoid any torn-writes. With this change, we might add
at max the strings specified at [1]. Adding it to the control file has
an advantage of preserving the last checkpoint kind which might be
useful.

[1] for checkpoint: "checkpoint shutdown end-of-recovery immediate
force wait wal time flush-all"
for restartpoint: "restartpoint shutdown end-of-recovery immediate
force wait wal time flush-all"

I'd bet squashing all of this into a single string (not really a flag)
will just mean people will have to parse it, etc. Keeping individual
boolean flags (or even separate string fields) would be better, I guess.

Since the size of controldata is a concern, I suggest to add an int16 to
populate with flags, which pg_controldata can parse for display.

Note that this other patch intends to add the timestamp and LSN of most recent
recovery to controldata.
https://commitfest.postgresql.org/36/3183/

We already have some checkpoint info in pg_stat_bgwriter, but that's
just aggregated data, not very convenient for info about the current
checkpoint. So maybe we should add pg_stat_progress_checkpoint, showing
various info about the current checkpoint?

It could go into the pg_stat_checkpointer view, which would be the culmination
of another patch (cc Melanie).
https://commitfest.postgresql.org/36/3272/

--
Justin

#4Bharath Rupireddy
bharath.rupireddyforpostgres@gmail.com
In reply to: Tomas Vondra (#2)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

On Wed, Dec 8, 2021 at 3:48 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:

On 12/7/21 15:36, Bharath Rupireddy wrote:

Hi,

Currently one can know the kind of on-going/last checkpoint (shutdown,
end-of-recovery, immediate, force etc.) only via server logs that to
when log_checkpoints GUC is on. At times, the users/service layer
components would want to know the kind of checkpoint (along with other
checkpoint related info) to take some actions and it will be a bit
difficult to search through the server logs. The checkpoint info can
be obtained from the control file (either by pg_control_checkpoint()
or by pg_controldata tool) whereas checkpoint kind isn't available
there.

How about we add an extra string field to the control file alongside
the other checkpoint info it already has? This way, the existing
pg_control_checkpoint() or pg_controldata tool can easily be enhanced
to show the checkpoint kind as well. One concern is that we don't want
to increase the size of pg_controldata by more than the typical block
size (of 8K) to avoid any torn-writes. With this change, we might add
at max the strings specified at [1]. Adding it to the control file has
an advantage of preserving the last checkpoint kind which might be
useful.

Thoughts?

I agree it might be useful to provide information about the nature of
the checkpoint, and perhaps even PID of the backend that triggered it
(although that may be tricky, if the backend terminates).

Thanks. I agree to have pg_stat_progress_checkpoint and yes PID of the
triggered backend can possibly go there (we can mention in the
documentation that the backend that triggered the checkpoint can get
terminated).

I'm not sure about adding it to control data, though. That doesn't seem
like a very good match for something that's mostly for monitoring.

Having it in the control data file (along with the existing checkpoint
information) will be helpful to know what was the last checkpoint
information and we can use the existing pg_control_checkpoint function
or the tool to emit that info. I plan to add an int16 flag as
suggested by Justin in this thread and come up with a patch.

We already have some checkpoint info in pg_stat_bgwriter, but that's
just aggregated data, not very convenient for info about the current
checkpoint. So maybe we should add pg_stat_progress_checkpoint, showing
various info about the current checkpoint?

+1 to have pg_stat_progress_checkpoint view to know what's going on
with the current checkpoint.

Regards,
Bharath Rupireddy.

#5Bharath Rupireddy
bharath.rupireddyforpostgres@gmail.com
In reply to: Justin Pryzby (#3)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

On Wed, Dec 8, 2021 at 4:07 AM Justin Pryzby <pryzby@telsasoft.com> wrote:

I'd bet squashing all of this into a single string (not really a flag)
will just mean people will have to parse it, etc. Keeping individual
boolean flags (or even separate string fields) would be better, I guess.

Since the size of controldata is a concern, I suggest to add an int16 to
populate with flags, which pg_controldata can parse for display.

+1. I will come up with a patch soon.

Note that this other patch intends to add the timestamp and LSN of most recent
recovery to controldata.
https://commitfest.postgresql.org/36/3183/

Thanks. I will go through it separately.

We already have some checkpoint info in pg_stat_bgwriter, but that's
just aggregated data, not very convenient for info about the current
checkpoint. So maybe we should add pg_stat_progress_checkpoint, showing
various info about the current checkpoint?

It could go into the pg_stat_checkpointer view, which would be the culmination
of another patch (cc Melanie).
https://commitfest.postgresql.org/36/3272/

+1 to have pg_stat_progress_checkpoint view. We have
CheckpointStatsData already. What we need is to make this structure
shared and add a little more info to represent the progress, so that
the other backends can access it. I think we can discuss this in a
separate thread to give it a fresh try rather than proposing this as a
part of another thread. I will spend some time on
pg_stat_progress_checkpoint proposal and try to come up with a
separate thread to discuss.

Regards,
Bharath Rupireddy.

#6Tomas Vondra
tomas.vondra@enterprisedb.com
In reply to: Bharath Rupireddy (#4)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

On 12/8/21 02:54, Bharath Rupireddy wrote:

On Wed, Dec 8, 2021 at 3:48 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:

On 12/7/21 15:36, Bharath Rupireddy wrote:

Hi,

Currently one can know the kind of on-going/last checkpoint (shutdown,
end-of-recovery, immediate, force etc.) only via server logs that to
when log_checkpoints GUC is on. At times, the users/service layer
components would want to know the kind of checkpoint (along with other
checkpoint related info) to take some actions and it will be a bit
difficult to search through the server logs. The checkpoint info can
be obtained from the control file (either by pg_control_checkpoint()
or by pg_controldata tool) whereas checkpoint kind isn't available
there.

How about we add an extra string field to the control file alongside
the other checkpoint info it already has? This way, the existing
pg_control_checkpoint() or pg_controldata tool can easily be enhanced
to show the checkpoint kind as well. One concern is that we don't want
to increase the size of pg_controldata by more than the typical block
size (of 8K) to avoid any torn-writes. With this change, we might add
at max the strings specified at [1]. Adding it to the control file has
an advantage of preserving the last checkpoint kind which might be
useful.

Thoughts?

I agree it might be useful to provide information about the nature of
the checkpoint, and perhaps even PID of the backend that triggered it
(although that may be tricky, if the backend terminates).

Thanks. I agree to have pg_stat_progress_checkpoint and yes PID of the
triggered backend can possibly go there (we can mention in the
documentation that the backend that triggered the checkpoint can get
terminated).

My concern is someone might run something that requires a checkpoint, so
we start it and put the PID into the catalog. And then the person aborts
the command and starts doing something else. But that does not abort the
checkpoint, but the backend now runs something that doesn't requite
checkpoint, which is rather confusing.

I'm not sure about adding it to control data, though. That doesn't seem
like a very good match for something that's mostly for monitoring.

Having it in the control data file (along with the existing checkpoint
information) will be helpful to know what was the last checkpoint
information and we can use the existing pg_control_checkpoint function
or the tool to emit that info. I plan to add an int16 flag as
suggested by Justin in this thread and come up with a patch.

OK, although I'm not sure it's all that useful (if we have that in some
sort of system view).

We already have some checkpoint info in pg_stat_bgwriter, but that's
just aggregated data, not very convenient for info about the current
checkpoint. So maybe we should add pg_stat_progress_checkpoint, showing
various info about the current checkpoint?

+1 to have pg_stat_progress_checkpoint view to know what's going on
with the current checkpoint.

Do you plan to add it to this patch, or should it be a separate patch?

regards

--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

#7Bharath Rupireddy
bharath.rupireddyforpostgres@gmail.com
In reply to: Tomas Vondra (#6)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

On Wed, Dec 8, 2021 at 7:34 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:

I agree it might be useful to provide information about the nature of
the checkpoint, and perhaps even PID of the backend that triggered it
(although that may be tricky, if the backend terminates).

Thanks. I agree to have pg_stat_progress_checkpoint and yes PID of the
triggered backend can possibly go there (we can mention in the
documentation that the backend that triggered the checkpoint can get
terminated).

My concern is someone might run something that requires a checkpoint, so
we start it and put the PID into the catalog. And then the person aborts
the command and starts doing something else. But that does not abort the
checkpoint, but the backend now runs something that doesn't requite
checkpoint, which is rather confusing.

I'm not sure about adding it to control data, though. That doesn't seem
like a very good match for something that's mostly for monitoring.

Having it in the control data file (along with the existing checkpoint
information) will be helpful to know what was the last checkpoint
information and we can use the existing pg_control_checkpoint function
or the tool to emit that info. I plan to add an int16 flag as
suggested by Justin in this thread and come up with a patch.

OK, although I'm not sure it's all that useful (if we have that in some
sort of system view).

If the server is down, the control file will help. Since we already
have the other checkpoint info in the control file, it's much more
useful and sensible to have this extra piece of missing information
(checkpoint kind) there.

We already have some checkpoint info in pg_stat_bgwriter, but that's
just aggregated data, not very convenient for info about the current
checkpoint. So maybe we should add pg_stat_progress_checkpoint, showing
various info about the current checkpoint?

+1 to have pg_stat_progress_checkpoint view to know what's going on
with the current checkpoint.

Do you plan to add it to this patch, or should it be a separate patch?

No, I will put some more thoughts around it and start a new thread.

Regards,
Bharath Rupireddy.

#8Tomas Vondra
tomas.vondra@enterprisedb.com
In reply to: Bharath Rupireddy (#5)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

On 12/8/21 02:54, Bharath Rupireddy wrote:

On Wed, Dec 8, 2021 at 4:07 AM Justin Pryzby <pryzby@telsasoft.com> wrote:

I'd bet squashing all of this into a single string (not really a flag)
will just mean people will have to parse it, etc. Keeping individual
boolean flags (or even separate string fields) would be better, I guess.

Since the size of controldata is a concern, I suggest to add an int16 to
populate with flags, which pg_controldata can parse for display.

+1. I will come up with a patch soon.

Note that this other patch intends to add the timestamp and LSN of most recent
recovery to controldata.
https://commitfest.postgresql.org/36/3183/

Thanks. I will go through it separately.

We already have some checkpoint info in pg_stat_bgwriter, but that's
just aggregated data, not very convenient for info about the current
checkpoint. So maybe we should add pg_stat_progress_checkpoint, showing
various info about the current checkpoint?

It could go into the pg_stat_checkpointer view, which would be the culmination
of another patch (cc Melanie).
https://commitfest.postgresql.org/36/3272/

I don't think the pg_stat_checkpointer would be a good match - that's
going to be an aggregated view of all past checkpoints, not a good
source info about the currently running one.

+1 to have pg_stat_progress_checkpoint view. We have
CheckpointStatsData already. What we need is to make this structure
shared and add a little more info to represent the progress, so that
the other backends can access it. I think we can discuss this in a
separate thread to give it a fresh try rather than proposing this as a
part of another thread. I will spend some time on
pg_stat_progress_checkpoint proposal and try to come up with a
separate thread to discuss.

+1 to discuss it as part of this patch

I'm not sure whether the view should look at CheckpointStatsData, or do
the same thing as the other pg_stat_progress_* views - send the data to
stat collector, and read it from there.

regards

--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

#9Bharath Rupireddy
bharath.rupireddyforpostgres@gmail.com
In reply to: Bharath Rupireddy (#7)
1 attachment(s)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

On Wed, Dec 8, 2021 at 7:43 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:

On Wed, Dec 8, 2021 at 7:34 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:

I agree it might be useful to provide information about the nature of
the checkpoint, and perhaps even PID of the backend that triggered it
(although that may be tricky, if the backend terminates).

I'm not sure about adding it to control data, though. That doesn't seem
like a very good match for something that's mostly for monitoring.

Having it in the control data file (along with the existing checkpoint
information) will be helpful to know what was the last checkpoint
information and we can use the existing pg_control_checkpoint function
or the tool to emit that info. I plan to add an int16 flag as
suggested by Justin in this thread and come up with a patch.

OK, although I'm not sure it's all that useful (if we have that in some
sort of system view).

If the server is down, the control file will help. Since we already
have the other checkpoint info in the control file, it's much more
useful and sensible to have this extra piece of missing information
(checkpoint kind) there.

Here's the patch that adds the last checkpoint kind to the control
file, displayed as an output in the pg_controldata tool and also
exposes it via the pg_control_checkpoint function.

I will analyze and work on the idea of pg_stat_progress_checkpoint and
try to post the patch here in this thread.

Regards,
Bharath Rupireddy.

Attachments:

v1-0001-add-last-checkpoint-kind-to-pg_control-file.patchapplication/octet-stream; name=v1-0001-add-last-checkpoint-kind-to-pg_control-file.patchDownload
From f5076d265b8d3048a530d44cd0b754479c679866 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Wed, 8 Dec 2021 11:24:49 +0000
Subject: [PATCH v1] add last checkpoint kind to pg_control file

Bump catalog version
---
 doc/src/sgml/func.sgml                  |  5 +++++
 src/backend/access/transam/xlog.c       |  3 +++
 src/backend/utils/misc/pg_controldata.c | 23 ++++++++++++++++++++++-
 src/bin/pg_controldata/pg_controldata.c | 18 ++++++++++++++++++
 src/include/catalog/pg_control.h        |  5 +++--
 src/include/catalog/pg_proc.dat         |  6 +++---
 6 files changed, 54 insertions(+), 6 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 0a725a6711..445dfaaac0 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -25029,6 +25029,11 @@ SELECT collation for ('foo' COLLATE "de_DE");
        <entry><type>timestamp with time zone</type></entry>
       </row>
 
+      <row>
+       <entry><structfield>checkpoint_kind</structfield></entry>
+       <entry><type>text</type></entry>
+      </row>
+
      </tbody>
     </tgroup>
    </table>
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index d894af310a..b457051941 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -5476,6 +5476,7 @@ BootStrapXLOG(void)
 
 	/* Now create pg_control */
 	InitControlFile(sysidentifier);
+	ControlFile->checkPointKind = 0;
 	ControlFile->time = checkPoint.time;
 	ControlFile->checkPoint = checkPoint.redo;
 	ControlFile->checkPointCopy = checkPoint;
@@ -9408,6 +9409,7 @@ CreateCheckPoint(int flags)
 	LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
 	if (shutdown)
 		ControlFile->state = DB_SHUTDOWNED;
+	ControlFile->checkPointKind = flags;
 	ControlFile->checkPoint = ProcLastRecPtr;
 	ControlFile->checkPointCopy = checkPoint;
 	/* crash recovery should always recover to the end of WAL */
@@ -9794,6 +9796,7 @@ CreateRestartPoint(int flags)
 	if (ControlFile->state == DB_IN_ARCHIVE_RECOVERY &&
 		ControlFile->checkPointCopy.redo < lastCheckPoint.redo)
 	{
+		ControlFile->checkPointKind = flags;
 		ControlFile->checkPoint = lastCheckPointRecPtr;
 		ControlFile->checkPointCopy = lastCheckPoint;
 
diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c
index 209a20a882..e94965d51f 100644
--- a/src/backend/utils/misc/pg_controldata.c
+++ b/src/backend/utils/misc/pg_controldata.c
@@ -87,12 +87,14 @@ pg_control_checkpoint(PG_FUNCTION_ARGS)
 	XLogSegNo	segno;
 	char		xlogfilename[MAXFNAMELEN];
 	bool		crc_ok;
+	uint16		flags;
+	char 		ckpt_kind[2 * MAXPGPATH];
 
 	/*
 	 * Construct a tuple descriptor for the result row.  This must match this
 	 * function's pg_proc entry!
 	 */
-	tupdesc = CreateTemplateTupleDesc(18);
+	tupdesc = CreateTemplateTupleDesc(19);
 	TupleDescInitEntry(tupdesc, (AttrNumber) 1, "checkpoint_lsn",
 					   PG_LSNOID, -1, 0);
 	TupleDescInitEntry(tupdesc, (AttrNumber) 2, "redo_lsn",
@@ -129,6 +131,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS)
 					   XIDOID, -1, 0);
 	TupleDescInitEntry(tupdesc, (AttrNumber) 18, "checkpoint_time",
 					   TIMESTAMPTZOID, -1, 0);
+	TupleDescInitEntry(tupdesc, (AttrNumber) 19, "checkpoint_kind",
+					   TEXTOID, -1, 0);
 	tupdesc = BlessTupleDesc(tupdesc);
 
 	/* Read the control file. */
@@ -202,6 +206,23 @@ pg_control_checkpoint(PG_FUNCTION_ARGS)
 	values[17] = TimestampTzGetDatum(time_t_to_timestamptz(ControlFile->checkPointCopy.time));
 	nulls[17] = false;
 
+	MemSet(ckpt_kind, 0, 2 * MAXPGPATH);
+	flags = ControlFile->checkPointKind;
+
+	snprintf(ckpt_kind, 2 * MAXPGPATH, "%s%s%s%s%s%s%s%s%s",
+				 (flags == 0) ? "unknown" : "",
+				 (flags & CHECKPOINT_IS_SHUTDOWN) ? "shutdown " : "",
+				 (flags & CHECKPOINT_END_OF_RECOVERY) ? "end-of-recovery " : "",
+				 (flags & CHECKPOINT_IMMEDIATE) ? "immediate " : "",
+				 (flags & CHECKPOINT_FORCE) ? "force " : "",
+				 (flags & CHECKPOINT_WAIT) ? "wait " : "",
+				 (flags & CHECKPOINT_CAUSE_XLOG) ? "wal " : "",
+				 (flags & CHECKPOINT_CAUSE_TIME) ? "time " : "",
+				 (flags & CHECKPOINT_FLUSH_ALL) ? "flush-all" : "");
+
+	values[18] = CStringGetTextDatum(ckpt_kind);
+	nulls[18] = false;
+
 	htup = heap_form_tuple(tupdesc, values, nulls);
 
 	PG_RETURN_DATUM(HeapTupleGetDatum(htup));
diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c
index f911f98d94..dfff0928e5 100644
--- a/src/bin/pg_controldata/pg_controldata.c
+++ b/src/bin/pg_controldata/pg_controldata.c
@@ -106,6 +106,8 @@ main(int argc, char *argv[])
 	int			c;
 	int			i;
 	int			WalSegSz;
+	uint16		flags;
+	char 		ckpt_kind[2 * MAXPGPATH];
 
 	pg_logging_init(argv[0]);
 	set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_controldata"));
@@ -225,6 +227,20 @@ main(int argc, char *argv[])
 		snprintf(&mock_auth_nonce_str[i * 2], 3, "%02x",
 				 (unsigned char) ControlFile->mock_authentication_nonce[i]);
 
+	MemSet(ckpt_kind, 0, 2 * MAXPGPATH);
+	flags = ControlFile->checkPointKind;
+
+	snprintf(ckpt_kind, 2 * MAXPGPATH, "%s%s%s%s%s%s%s%s%s",
+				 (flags == 0) ? "unknown" : "",
+				 (flags & CHECKPOINT_IS_SHUTDOWN) ? "shutdown " : "",
+				 (flags & CHECKPOINT_END_OF_RECOVERY) ? "end-of-recovery " : "",
+				 (flags & CHECKPOINT_IMMEDIATE) ? "immediate " : "",
+				 (flags & CHECKPOINT_FORCE) ? "force " : "",
+				 (flags & CHECKPOINT_WAIT) ? "wait " : "",
+				 (flags & CHECKPOINT_CAUSE_XLOG) ? "wal " : "",
+				 (flags & CHECKPOINT_CAUSE_TIME) ? "time " : "",
+				 (flags & CHECKPOINT_FLUSH_ALL) ? "flush-all" : "");
+
 	printf(_("pg_control version number:            %u\n"),
 		   ControlFile->pg_control_version);
 	printf(_("Catalog version number:               %u\n"),
@@ -235,6 +251,8 @@ main(int argc, char *argv[])
 		   dbState(ControlFile->state));
 	printf(_("pg_control last modified:             %s\n"),
 		   pgctime_str);
+	printf(_("Latest checkpoint kind:               %s\n"),
+		   ckpt_kind);
 	printf(_("Latest checkpoint location:           %X/%X\n"),
 		   LSN_FORMAT_ARGS(ControlFile->checkPoint));
 	printf(_("Latest checkpoint's REDO location:    %X/%X\n"),
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index 749bce0cc6..3155857b67 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -128,9 +128,10 @@ typedef struct ControlFileData
 	 */
 	DBState		state;			/* see enum above */
 	pg_time_t	time;			/* time stamp of last pg_control update */
-	XLogRecPtr	checkPoint;		/* last check point record ptr */
+	uint16		checkPointKind;	/* last checkpoint kind */
+	XLogRecPtr	checkPoint;		/* last checkpoint record ptr */
 
-	CheckPoint	checkPointCopy; /* copy of last check point record */
+	CheckPoint	checkPointCopy; /* copy of last checkpoint record */
 
 	XLogRecPtr	unloggedLSN;	/* current fake LSN value, for unlogged rels */
 
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 79d787cd26..1abf466c63 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11567,9 +11567,9 @@
   descr => 'pg_controldata checkpoint state information as a function',
   proname => 'pg_control_checkpoint', provolatile => 'v',
   prorettype => 'record', proargtypes => '',
-  proallargtypes => '{pg_lsn,pg_lsn,text,int4,int4,bool,text,oid,xid,xid,xid,oid,xid,xid,oid,xid,xid,timestamptz}',
-  proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
-  proargnames => '{checkpoint_lsn,redo_lsn,redo_wal_file,timeline_id,prev_timeline_id,full_page_writes,next_xid,next_oid,next_multixact_id,next_multi_offset,oldest_xid,oldest_xid_dbid,oldest_active_xid,oldest_multi_xid,oldest_multi_dbid,oldest_commit_ts_xid,newest_commit_ts_xid,checkpoint_time}',
+  proallargtypes => '{pg_lsn,pg_lsn,text,int4,int4,bool,text,oid,xid,xid,xid,oid,xid,xid,oid,xid,xid,timestamptz,text}',
+  proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+  proargnames => '{checkpoint_lsn,redo_lsn,redo_wal_file,timeline_id,prev_timeline_id,full_page_writes,next_xid,next_oid,next_multixact_id,next_multi_offset,oldest_xid,oldest_xid_dbid,oldest_active_xid,oldest_multi_xid,oldest_multi_dbid,oldest_commit_ts_xid,newest_commit_ts_xid,checkpoint_time,checkpoint_kind}',
   prosrc => 'pg_control_checkpoint' },
 
 { oid => '3443',
-- 
2.25.1

#10Bharath Rupireddy
bharath.rupireddyforpostgres@gmail.com
In reply to: Bharath Rupireddy (#9)
1 attachment(s)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

On Thu, Dec 9, 2021 at 12:08 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:

On Wed, Dec 8, 2021 at 7:43 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:

On Wed, Dec 8, 2021 at 7:34 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:

I agree it might be useful to provide information about the nature of
the checkpoint, and perhaps even PID of the backend that triggered it
(although that may be tricky, if the backend terminates).

I'm not sure about adding it to control data, though. That doesn't seem
like a very good match for something that's mostly for monitoring.

Having it in the control data file (along with the existing checkpoint
information) will be helpful to know what was the last checkpoint
information and we can use the existing pg_control_checkpoint function
or the tool to emit that info. I plan to add an int16 flag as
suggested by Justin in this thread and come up with a patch.

OK, although I'm not sure it's all that useful (if we have that in some
sort of system view).

If the server is down, the control file will help. Since we already
have the other checkpoint info in the control file, it's much more
useful and sensible to have this extra piece of missing information
(checkpoint kind) there.

Here's the patch that adds the last checkpoint kind to the control
file, displayed as an output in the pg_controldata tool and also
exposes it via the pg_control_checkpoint function.

Here's v2, rebased onto the latest master.

Regards,
Bharath Rupireddy.

Attachments:

v2-0001-add-last-checkpoint-kind-to-pg_control-file.patchapplication/octet-stream; name=v2-0001-add-last-checkpoint-kind-to-pg_control-file.patchDownload
From 0ff675bdad7a8409919ffb655d65e92f6a516e8b Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Mon, 27 Dec 2021 14:11:18 +0000
Subject: [PATCH v2] add last checkpoint kind to pg_control file

Bump catalog version
---
 doc/src/sgml/func.sgml                  |  5 +++++
 src/backend/access/transam/xlog.c       |  3 +++
 src/backend/utils/misc/pg_controldata.c | 27 ++++++++++++++++++++++---
 src/bin/pg_controldata/pg_controldata.c | 18 +++++++++++++++++
 src/include/catalog/pg_control.h        |  5 +++--
 src/include/catalog/pg_proc.dat         |  6 +++---
 6 files changed, 56 insertions(+), 8 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index e58efce586..7ce43b3fa9 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -25101,6 +25101,11 @@ SELECT collation for ('foo' COLLATE "de_DE");
        <entry><type>timestamp with time zone</type></entry>
       </row>
 
+      <row>
+       <entry><structfield>checkpoint_kind</structfield></entry>
+       <entry><type>text</type></entry>
+      </row>
+
      </tbody>
     </tgroup>
    </table>
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 1e1fbe957f..437bca2c9f 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -5488,6 +5488,7 @@ BootStrapXLOG(void)
 
 	/* Now create pg_control */
 	InitControlFile(sysidentifier);
+	ControlFile->checkPointKind = 0;
 	ControlFile->time = checkPoint.time;
 	ControlFile->checkPoint = checkPoint.redo;
 	ControlFile->checkPointCopy = checkPoint;
@@ -9376,6 +9377,7 @@ CreateCheckPoint(int flags)
 	LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
 	if (shutdown)
 		ControlFile->state = DB_SHUTDOWNED;
+	ControlFile->checkPointKind = flags;
 	ControlFile->checkPoint = ProcLastRecPtr;
 	ControlFile->checkPointCopy = checkPoint;
 	/* crash recovery should always recover to the end of WAL */
@@ -9762,6 +9764,7 @@ CreateRestartPoint(int flags)
 	if (ControlFile->state == DB_IN_ARCHIVE_RECOVERY &&
 		ControlFile->checkPointCopy.redo < lastCheckPoint.redo)
 	{
+		ControlFile->checkPointKind = flags;
 		ControlFile->checkPoint = lastCheckPointRecPtr;
 		ControlFile->checkPointCopy = lastCheckPoint;
 
diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c
index b1db9a8d07..e94965d51f 100644
--- a/src/backend/utils/misc/pg_controldata.c
+++ b/src/backend/utils/misc/pg_controldata.c
@@ -79,20 +79,22 @@ pg_control_system(PG_FUNCTION_ARGS)
 Datum
 pg_control_checkpoint(PG_FUNCTION_ARGS)
 {
-	Datum		values[18];
-	bool		nulls[18];
+	Datum		values[19];
+	bool		nulls[19];
 	TupleDesc	tupdesc;
 	HeapTuple	htup;
 	ControlFileData *ControlFile;
 	XLogSegNo	segno;
 	char		xlogfilename[MAXFNAMELEN];
 	bool		crc_ok;
+	uint16		flags;
+	char 		ckpt_kind[2 * MAXPGPATH];
 
 	/*
 	 * Construct a tuple descriptor for the result row.  This must match this
 	 * function's pg_proc entry!
 	 */
-	tupdesc = CreateTemplateTupleDesc(18);
+	tupdesc = CreateTemplateTupleDesc(19);
 	TupleDescInitEntry(tupdesc, (AttrNumber) 1, "checkpoint_lsn",
 					   PG_LSNOID, -1, 0);
 	TupleDescInitEntry(tupdesc, (AttrNumber) 2, "redo_lsn",
@@ -129,6 +131,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS)
 					   XIDOID, -1, 0);
 	TupleDescInitEntry(tupdesc, (AttrNumber) 18, "checkpoint_time",
 					   TIMESTAMPTZOID, -1, 0);
+	TupleDescInitEntry(tupdesc, (AttrNumber) 19, "checkpoint_kind",
+					   TEXTOID, -1, 0);
 	tupdesc = BlessTupleDesc(tupdesc);
 
 	/* Read the control file. */
@@ -202,6 +206,23 @@ pg_control_checkpoint(PG_FUNCTION_ARGS)
 	values[17] = TimestampTzGetDatum(time_t_to_timestamptz(ControlFile->checkPointCopy.time));
 	nulls[17] = false;
 
+	MemSet(ckpt_kind, 0, 2 * MAXPGPATH);
+	flags = ControlFile->checkPointKind;
+
+	snprintf(ckpt_kind, 2 * MAXPGPATH, "%s%s%s%s%s%s%s%s%s",
+				 (flags == 0) ? "unknown" : "",
+				 (flags & CHECKPOINT_IS_SHUTDOWN) ? "shutdown " : "",
+				 (flags & CHECKPOINT_END_OF_RECOVERY) ? "end-of-recovery " : "",
+				 (flags & CHECKPOINT_IMMEDIATE) ? "immediate " : "",
+				 (flags & CHECKPOINT_FORCE) ? "force " : "",
+				 (flags & CHECKPOINT_WAIT) ? "wait " : "",
+				 (flags & CHECKPOINT_CAUSE_XLOG) ? "wal " : "",
+				 (flags & CHECKPOINT_CAUSE_TIME) ? "time " : "",
+				 (flags & CHECKPOINT_FLUSH_ALL) ? "flush-all" : "");
+
+	values[18] = CStringGetTextDatum(ckpt_kind);
+	nulls[18] = false;
+
 	htup = heap_form_tuple(tupdesc, values, nulls);
 
 	PG_RETURN_DATUM(HeapTupleGetDatum(htup));
diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c
index f911f98d94..dfff0928e5 100644
--- a/src/bin/pg_controldata/pg_controldata.c
+++ b/src/bin/pg_controldata/pg_controldata.c
@@ -106,6 +106,8 @@ main(int argc, char *argv[])
 	int			c;
 	int			i;
 	int			WalSegSz;
+	uint16		flags;
+	char 		ckpt_kind[2 * MAXPGPATH];
 
 	pg_logging_init(argv[0]);
 	set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_controldata"));
@@ -225,6 +227,20 @@ main(int argc, char *argv[])
 		snprintf(&mock_auth_nonce_str[i * 2], 3, "%02x",
 				 (unsigned char) ControlFile->mock_authentication_nonce[i]);
 
+	MemSet(ckpt_kind, 0, 2 * MAXPGPATH);
+	flags = ControlFile->checkPointKind;
+
+	snprintf(ckpt_kind, 2 * MAXPGPATH, "%s%s%s%s%s%s%s%s%s",
+				 (flags == 0) ? "unknown" : "",
+				 (flags & CHECKPOINT_IS_SHUTDOWN) ? "shutdown " : "",
+				 (flags & CHECKPOINT_END_OF_RECOVERY) ? "end-of-recovery " : "",
+				 (flags & CHECKPOINT_IMMEDIATE) ? "immediate " : "",
+				 (flags & CHECKPOINT_FORCE) ? "force " : "",
+				 (flags & CHECKPOINT_WAIT) ? "wait " : "",
+				 (flags & CHECKPOINT_CAUSE_XLOG) ? "wal " : "",
+				 (flags & CHECKPOINT_CAUSE_TIME) ? "time " : "",
+				 (flags & CHECKPOINT_FLUSH_ALL) ? "flush-all" : "");
+
 	printf(_("pg_control version number:            %u\n"),
 		   ControlFile->pg_control_version);
 	printf(_("Catalog version number:               %u\n"),
@@ -235,6 +251,8 @@ main(int argc, char *argv[])
 		   dbState(ControlFile->state));
 	printf(_("pg_control last modified:             %s\n"),
 		   pgctime_str);
+	printf(_("Latest checkpoint kind:               %s\n"),
+		   ckpt_kind);
 	printf(_("Latest checkpoint location:           %X/%X\n"),
 		   LSN_FORMAT_ARGS(ControlFile->checkPoint));
 	printf(_("Latest checkpoint's REDO location:    %X/%X\n"),
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index 749bce0cc6..3155857b67 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -128,9 +128,10 @@ typedef struct ControlFileData
 	 */
 	DBState		state;			/* see enum above */
 	pg_time_t	time;			/* time stamp of last pg_control update */
-	XLogRecPtr	checkPoint;		/* last check point record ptr */
+	uint16		checkPointKind;	/* last checkpoint kind */
+	XLogRecPtr	checkPoint;		/* last checkpoint record ptr */
 
-	CheckPoint	checkPointCopy; /* copy of last check point record */
+	CheckPoint	checkPointCopy; /* copy of last checkpoint record */
 
 	XLogRecPtr	unloggedLSN;	/* current fake LSN value, for unlogged rels */
 
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4d992dc224..1bc75bf52c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11549,9 +11549,9 @@
   descr => 'pg_controldata checkpoint state information as a function',
   proname => 'pg_control_checkpoint', provolatile => 'v',
   prorettype => 'record', proargtypes => '',
-  proallargtypes => '{pg_lsn,pg_lsn,text,int4,int4,bool,text,oid,xid,xid,xid,oid,xid,xid,oid,xid,xid,timestamptz}',
-  proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
-  proargnames => '{checkpoint_lsn,redo_lsn,redo_wal_file,timeline_id,prev_timeline_id,full_page_writes,next_xid,next_oid,next_multixact_id,next_multi_offset,oldest_xid,oldest_xid_dbid,oldest_active_xid,oldest_multi_xid,oldest_multi_dbid,oldest_commit_ts_xid,newest_commit_ts_xid,checkpoint_time}',
+  proallargtypes => '{pg_lsn,pg_lsn,text,int4,int4,bool,text,oid,xid,xid,xid,oid,xid,xid,oid,xid,xid,timestamptz,text}',
+  proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+  proargnames => '{checkpoint_lsn,redo_lsn,redo_wal_file,timeline_id,prev_timeline_id,full_page_writes,next_xid,next_oid,next_multixact_id,next_multi_offset,oldest_xid,oldest_xid_dbid,oldest_active_xid,oldest_multi_xid,oldest_multi_dbid,oldest_commit_ts_xid,newest_commit_ts_xid,checkpoint_time,checkpoint_kind}',
   prosrc => 'pg_control_checkpoint' },
 
 { oid => '3443',
-- 
2.25.1

#11Sergey Dudoladov
sergey.dudoladov@gmail.com
In reply to: Bharath Rupireddy (#10)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

Hi all,

Here's v2, rebased onto the latest master.

I've reviewed this patch. The patch builds against the master (commit
e9d4001ec592bcc9a3332547cb1b0211e8794f38) and passes all the tests.
The patch does what it intends to do, namely store the kind of the
last checkpoint in the control file and display it in the output of
the pg_control_checkpoint() function and pg_controldata utility.
I did not test it with restartpoints though. Speaking of the torn
writes, the size of the control file with this patch applied does not
exceed 8Kb.

A few code comments:

+ char ckpt_kind[2 * MAXPGPATH];

I don't completely understand why 2 * MAXPGPATH is used here for the
buffer size.
[1]: https://github.com/postgres/postgres/blob/410aa248e5a883fde4832999cc9b23c7ace0f2ff/src/include/pg_config_manual.h#L106
does it relate to ckpt_kind ?

+ ControlFile->checkPointKind = 0;

It is worth a comment that 0 is unknown, as for instance in [2]https://github.com/postgres/postgres/blob/410aa248e5a883fde4832999cc9b23c7ace0f2ff/src/interfaces/libpq/fe-exec.c#L1078

+ (flags == 0) ? "unknown" : "",

That reads as if this patch would introduce a new "unknown" checkpoint state.
Why have it here at all if after for example initdb the kind is "shutdown" ?

+ snprintf(ckpt_kind, 2 * MAXPGPATH, "%s%s%s%s%s%s%s%s%s",
+ (flags == 0) ? "unknown" : "",
+ (flags & CHECKPOINT_IS_SHUTDOWN) ? "shutdown " : "",
+ (flags & CHECKPOINT_END_OF_RECOVERY) ? "end-of-recovery " : "",
+ (flags & CHECKPOINT_IMMEDIATE) ? "immediate " : "",
+ (flags & CHECKPOINT_FORCE) ? "force " : "",
+ (flags & CHECKPOINT_WAIT) ? "wait " : "",
+ (flags & CHECKPOINT_CAUSE_XLOG) ? "wal " : "",
+ (flags & CHECKPOINT_CAUSE_TIME) ? "time " : "",
+ (flags & CHECKPOINT_FLUSH_ALL) ? "flush-all" : "");

The space at the strings' end (as in "wait " or "immediate ")
introduces extra whitespace in the output of pg_control_checkpoint().
A similar check at [3]https://github.com/postgres/postgres/blob/27b77ecf9f4d5be211900eda54d8155ada50d696/src/backend/access/transam/xlog.c#L8851-L8859 places whitespace differently; that arrangement
of whitespace should remove the issue.

[1]: https://github.com/postgres/postgres/blob/410aa248e5a883fde4832999cc9b23c7ace0f2ff/src/include/pg_config_manual.h#L106
[2]: https://github.com/postgres/postgres/blob/410aa248e5a883fde4832999cc9b23c7ace0f2ff/src/interfaces/libpq/fe-exec.c#L1078
[3]: https://github.com/postgres/postgres/blob/27b77ecf9f4d5be211900eda54d8155ada50d696/src/backend/access/transam/xlog.c#L8851-L8859

Regards,
Sergey Dudoladov

#12Julien Rouhaud
rjuju123@gmail.com
In reply to: Sergey Dudoladov (#11)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

Hi,

On Thu, Jan 27, 2022 at 10:53:32AM +0100, Sergey Dudoladov wrote:

Hi all,

Here's v2, rebased onto the latest master.

I've reviewed this patch. The patch builds against the master (commit
e9d4001ec592bcc9a3332547cb1b0211e8794f38) and passes all the tests.
The patch does what it intends to do, namely store the kind of the
last checkpoint in the control file and display it in the output of
the pg_control_checkpoint() function and pg_controldata utility.

I don't agree with that.

What it's showing is the "currently ongoing checkpoint or last completed
checkpoint" kind. It's still not possible to know if a checkpoint is in
progress or not and any kind of information related to it, so I'm not sure how
useful this will be compared to a checkpoint progress view.

Also, it's only showing the initial triggering conditions of checkpoints.
For instance, if a timed checkpoint is started and then a backend executes a
"CHECKPOINT;", it will upgrade the ongoing checkpoint with additional flags but
AFAICS those new flags won't be saved to the control file.

#13Julien Rouhaud
rjuju123@gmail.com
In reply to: Julien Rouhaud (#12)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

On Thu, Jan 27, 2022 at 06:56:57PM +0800, Julien Rouhaud wrote:

What it's showing is the "currently ongoing checkpoint or last completed
checkpoint" kind.

Ah after double checking I see it's storing the information *after* the
checkpoint completion, so it's indeed the last completed checkpoint. I'm not
sure how useful it can be, but ok.

Also, it's only showing the initial triggering conditions of checkpoints.
For instance, if a timed checkpoint is started and then a backend executes a
"CHECKPOINT;", it will upgrade the ongoing checkpoint with additional flags but
AFAICS those new flags won't be saved to the control file.

This one is still valid I think, it's only storing the initial flags and not
the possibly upgraded one in shmem.

#14Kyotaro Horiguchi
horikyota.ntt@gmail.com
In reply to: Julien Rouhaud (#13)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

At Thu, 27 Jan 2022 19:09:29 +0800, Julien Rouhaud <rjuju123@gmail.com> wrote in

On Thu, Jan 27, 2022 at 06:56:57PM +0800, Julien Rouhaud wrote:

What it's showing is the "currently ongoing checkpoint or last completed
checkpoint" kind.

Ah after double checking I see it's storing the information *after* the
checkpoint completion, so it's indeed the last completed checkpoint. I'm not
sure how useful it can be, but ok.

I don't see it useful (but don't oppose) to record checkpoint kind in
control file. It is a kind of realtime noncritical info and in the
first place retrievable from server log if needed. And I'm skeptical
that it is needed such frequently. Checkpoint kind is useful to check
max_wal_size's appropriateness if it is in a summarized form as in
pg_stat_bgwriter. On the other hand showing the same in a stats view
or the output of pg_control_checkpoint() is fine by me.

Also, it's only showing the initial triggering conditions of checkpoints.
For instance, if a timed checkpoint is started and then a backend executes a
"CHECKPOINT;", it will upgrade the ongoing checkpoint with additional flags but
AFAICS those new flags won't be saved to the control file.

This one is still valid I think, it's only storing the initial flags and not
the possibly upgraded one in shmem.

Agreed.

I don't like to add this complex-but-need-in-sync blob twice. If we
need to do that twice, I want them consolidated in any shape.

Datum values[18];
bool nulls[18];

You forgot to expand these arrays.

This breaks checkpoint file format. Need to bump PG_CONTROL_VERSION,
and pg_upgrade need to treat the change.

Even if we add checkpoint kind to control file, it would look a bit
strange that the "checkpoint kind" shows first among all
checkpoint-related lines. And at least the "wait" in the line is
really useless since it is not a property of a checkpoint. Instead, it
doesn't show "requested" which is one of the checkpoint properties
like "xlog" and "time". I'm not sure we need all of the properties to
be shown but I don't have a clear criteria for each property of it
ought to be shown or not.

pg_control last modified: Fri 28 Jan 2022 09:49:46 AM JST
Latest checkpoint kind: immediate force wait
Latest checkpoint location: 0/172B2C8

I'd like to see the PID of the triggering process, but it is really
not a information suitable in the control file...

-  proallargtypes => '{pg_lsn,pg_lsn,text,int4,int4,bool,text,oid,xid,xid,xid,oid,xid,xid,oid,xid,xid,timestamptz}',
+  proallargtypes => '{pg_lsn,pg_lsn,text,int4,int4,bool,text,oid,xid,xid,xid,oid,xid,xid,oid,xid,xid,timestamptz,text}',

I think the additional column should be text[] instead of text, but
not sure.

And you need to edit the corresponding part of the doc.

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center

#15Kyotaro Horiguchi
horikyota.ntt@gmail.com
In reply to: Julien Rouhaud (#13)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

Sorry, the last message lacks one citation.

At Thu, 27 Jan 2022 19:09:29 +0800, Julien Rouhaud <rjuju123@gmail.com> wrote in

On Thu, Jan 27, 2022 at 06:56:57PM +0800, Julien Rouhaud wrote:

What it's showing is the "currently ongoing checkpoint or last completed
checkpoint" kind.

Ah after double checking I see it's storing the information *after* the
checkpoint completion, so it's indeed the last completed checkpoint. I'm not
sure how useful it can be, but ok.

I don't see it useful (but don't oppose) to record checkpoint kind in
control file. It is a kind of realtime noncritical info and in the
first place retrievable from server log if needed. And I'm skeptical
that it is needed such frequently. Checkpoint kind is useful to check
max_wal_size's appropriateness if it is in a summarized form as in
pg_stat_bgwriter. On the other hand showing the same in a stats view
or the output of pg_control_checkpoint() is fine by me.

Also, it's only showing the initial triggering conditions of checkpoints.
For instance, if a timed checkpoint is started and then a backend executes a
"CHECKPOINT;", it will upgrade the ongoing checkpoint with additional flags but
AFAICS those new flags won't be saved to the control file.

This one is still valid I think, it's only storing the initial flags and not
the possibly upgraded one in shmem.

Agreed.

+	snprintf(ckpt_kind, 2 * MAXPGPATH, "%s%s%s%s%s%s%s%s%s",
+				 (flags == 0) ? "unknown" : "",
+				 (flags & CHECKPOINT_IS_SHUTDOWN) ? "shutdown " : "",
+				 (flags & CHECKPOINT_END_OF_RECOVERY) ? "end-of-recovery " : "",
+				 (flags & CHECKPOINT_IMMEDIATE) ? "immediate " : "",
+				 (flags & CHECKPOINT_FORCE) ? "force " : "",
+				 (flags & CHECKPOINT_WAIT) ? "wait " : "",
+				 (flags & CHECKPOINT_CAUSE_XLOG) ? "wal " : "",
+				 (flags & CHECKPOINT_CAUSE_TIME) ? "time " : "",
+				 (flags & CHECKPOINT_FLUSH_ALL) ? "flush-all" : "");

I don't like to add this complex-but-need-in-sync blob twice. If we
need to do that twice, I want them consolidated in any shape.

+ Datum values[18];
+ bool nulls[18];

You forgot to expand these arrays.

This breaks checkpoint file format. Need to bump PG_CONTROL_VERSION,
and pg_upgrade need to treat the change.

Even if we add checkpoint kind to control file, it would look a bit
strange that the "checkpoint kind" shows first among all
checkpoint-related lines. And at least the "wait" in the line is
really useless since it is not a property of a checkpoint. Instead, it
doesn't show "requested" which is one of the checkpoint properties
like "xlog" and "time". I'm not sure we need all of the properties to
be shown but I don't have a clear criteria for each property of it
ought to be shown or not.

pg_control last modified: Fri 28 Jan 2022 09:49:46 AM JST
Latest checkpoint kind: immediate force wait
Latest checkpoint location: 0/172B2C8

I'd like to see the PID of the triggering process, but it is really
not a information suitable in the control file...

-  proallargtypes => '{pg_lsn,pg_lsn,text,int4,int4,bool,text,oid,xid,xid,xid,oid,xid,xid,oid,xid,xid,timestamptz}',
+  proallargtypes => '{pg_lsn,pg_lsn,text,int4,int4,bool,text,oid,xid,xid,xid,oid,xid,xid,oid,xid,xid,timestamptz,text}',

I think the additional column should be text[] instead of text, but
not sure.

And you need to edit the corresponding part of the doc.

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center

#16Julien Rouhaud
rjuju123@gmail.com
In reply to: Kyotaro Horiguchi (#14)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

Hi,

On Fri, Jan 28, 2022 at 10:38:53AM +0900, Kyotaro Horiguchi wrote:

I'd like to see the PID of the triggering process, but it is really
not a information suitable in the control file...

Yes that's something I would like too. But even if the PIDs could be store, I
don't think that having the information for an already completed checkpoint
would be of any use at all.

For the current checkpoint, it should also be an array of PID. For instance if
the checkpointer started a throttled checkpoint, then someone calls a non
immediate pg_start_backup() and finally thinks it's too slow and need a fast
checkpoint. This would be welcome in a new pg_stat_progress_checkpoint view.

#17Kyotaro Horiguchi
horikyota.ntt@gmail.com
In reply to: Kyotaro Horiguchi (#15)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

At Fri, 28 Jan 2022 10:41:28 +0900 (JST), Kyotaro Horiguchi <horikyota.ntt@gmail.com> wrote in

Sorry, the last message lacks one citation.

At Thu, 27 Jan 2022 19:09:29 +0800, Julien Rouhaud <rjuju123@gmail.com> wrote in

On Thu, Jan 27, 2022 at 06:56:57PM +0800, Julien Rouhaud wrote:

What it's showing is the "currently ongoing checkpoint or last completed
checkpoint" kind.

Ah after double checking I see it's storing the information *after* the
checkpoint completion, so it's indeed the last completed checkpoint. I'm not
sure how useful it can be, but ok.

I don't see it useful (but don't oppose) to record checkpoint kind in
control file. It is a kind of realtime noncritical info and in the
first place retrievable from server log if needed. And I'm skeptical
that it is needed such frequently. Checkpoint kind is useful to check
max_wal_size's appropriateness if it is in a summarized form as in
pg_stat_bgwriter. On the other hand showing the same in a stats view
or the output of pg_control_checkpoint() is fine by me.

Also, it's only showing the initial triggering conditions of checkpoints.
For instance, if a timed checkpoint is started and then a backend executes a
"CHECKPOINT;", it will upgrade the ongoing checkpoint with additional flags but
AFAICS those new flags won't be saved to the control file.

This one is still valid I think, it's only storing the initial flags and not
the possibly upgraded one in shmem.

Agreed.

+	snprintf(ckpt_kind, 2 * MAXPGPATH, "%s%s%s%s%s%s%s%s%s",
+				 (flags == 0) ? "unknown" : "",
+				 (flags & CHECKPOINT_IS_SHUTDOWN) ? "shutdown " : "",
+				 (flags & CHECKPOINT_END_OF_RECOVERY) ? "end-of-recovery " : "",
+				 (flags & CHECKPOINT_IMMEDIATE) ? "immediate " : "",
+				 (flags & CHECKPOINT_FORCE) ? "force " : "",
+				 (flags & CHECKPOINT_WAIT) ? "wait " : "",
+				 (flags & CHECKPOINT_CAUSE_XLOG) ? "wal " : "",
+				 (flags & CHECKPOINT_CAUSE_TIME) ? "time " : "",
+				 (flags & CHECKPOINT_FLUSH_ALL) ? "flush-all" : "");

I don't like to add this complex-but-need-in-sync blob twice. If we
need to do that twice, I want them consolidated in any shape.

+ Datum values[18];
+ bool nulls[18];

You forgot to expand these arrays.

This breaks checkpoint file format. Need to bump PG_CONTROL_VERSION,
and pg_upgrade need to treat the change.

Even if we add checkpoint kind to control file, it would look a bit
strange that the "checkpoint kind" shows first among all
checkpoint-related lines. And at least the "wait" in the line is
really useless since it is not a property of a checkpoint. Instead, it
doesn't show "requested" which is one of the checkpoint properties
like "xlog" and "time". I'm not sure we need all of the properties to
be shown but I don't have a clear criteria for each property of it
ought to be shown or not.

pg_control last modified: Fri 28 Jan 2022 09:49:46 AM JST
Latest checkpoint kind: immediate force wait
Latest checkpoint location: 0/172B2C8

I'd like to see the PID of the triggering process, but it is really
not a information suitable in the control file...

-  proallargtypes => '{pg_lsn,pg_lsn,text,int4,int4,bool,text,oid,xid,xid,xid,oid,xid,xid,oid,xid,xid,timestamptz}',
+  proallargtypes => '{pg_lsn,pg_lsn,text,int4,int4,bool,text,oid,xid,xid,xid,oid,xid,xid,oid,xid,xid,timestamptz,text}',

I think the additional column should be text[] instead of text, but
not sure.

And you need to edit the corresponding part of the doc.

I have an additional comment.

+	char 		ckpt_kind[2 * MAXPGPATH];
..
+	snprintf(ckpt_kind, 2 * MAXPGPATH, "%s%s%s%s%s%s%s%s%s",
+				 (flags == 0) ? "unknown" : "",
+				 (flags & CHECKPOINT_IS_SHUTDOWN) ? "shutdown " : "",

The value "2 * MAXPGPATH" is utterly nonsense about bouth "2" and
"MAXPGPATH", and the product of them is apparently too large.

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center

#18Kyotaro Horiguchi
horikyota.ntt@gmail.com
In reply to: Julien Rouhaud (#16)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

At Fri, 28 Jan 2022 10:00:46 +0800, Julien Rouhaud <rjuju123@gmail.com> wrote in

Hi,

On Fri, Jan 28, 2022 at 10:38:53AM +0900, Kyotaro Horiguchi wrote:

I'd like to see the PID of the triggering process, but it is really
not a information suitable in the control file...

Yes that's something I would like too. But even if the PIDs could be store, I
don't think that having the information for an already completed checkpoint
would be of any use at all.

For the current checkpoint, it should also be an array of PID. For instance if
the checkpointer started a throttled checkpoint, then someone calls a non
immediate pg_start_backup() and finally thinks it's too slow and need a fast
checkpoint. This would be welcome in a new pg_stat_progress_checkpoint view.

Yeah, I thought of the same issue. But at the time of the previous
mail I thought it would be enough that the PID is of the first
initiator of the current running checkpoint likewise the checkpoint
kind. So it is important to think about the use case. If we put
significancy on actually happend checkpoints, we don't need absorbed
checkpoint requests to be recorded but if we put significance on
checkpoint requests, we need to care of every checkpoint request.

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center

#19Bharath Rupireddy
bharath.rupireddyforpostgres@gmail.com
In reply to: Julien Rouhaud (#16)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

On Fri, Jan 28, 2022 at 7:30 AM Julien Rouhaud <rjuju123@gmail.com> wrote:

Hi,

On Fri, Jan 28, 2022 at 10:38:53AM +0900, Kyotaro Horiguchi wrote:

I'd like to see the PID of the triggering process, but it is really
not a information suitable in the control file...

Yes that's something I would like too. But even if the PIDs could be store, I
don't think that having the information for an already completed checkpoint
would be of any use at all.

For the current checkpoint, it should also be an array of PID. For instance if
the checkpointer started a throttled checkpoint, then someone calls a non
immediate pg_start_backup() and finally thinks it's too slow and need a fast
checkpoint. This would be welcome in a new pg_stat_progress_checkpoint view.

Thanks all for the comments. pg_stat_progress_checkpoint is being
discussed in another thread [1]/messages/by-id/CALj2ACV-F+K+z+XW8fnK4MV71qz2gzAMxFnYziRgZURMB5ycAQ@mail.gmail.com.

I will respond to the other comments soon.

[1]: /messages/by-id/CALj2ACV-F+K+z+XW8fnK4MV71qz2gzAMxFnYziRgZURMB5ycAQ@mail.gmail.com

Regards,
Bharath Rupireddy.

#20Julien Rouhaud
rjuju123@gmail.com
In reply to: Bharath Rupireddy (#19)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

Hi,

On Fri, Jan 28, 2022 at 08:17:39AM +0530, Bharath Rupireddy wrote:

Thanks all for the comments. pg_stat_progress_checkpoint is being
discussed in another thread [1].

[1] /messages/by-id/CALj2ACV-F+K+z+XW8fnK4MV71qz2gzAMxFnYziRgZURMB5ycAQ@mail.gmail.com

This thread was explicitly talking about log reporting and it's not clear to me
that it's now supposed to be about a new pg_stat_progress_checkpoint()
function. I think you should start a new thread or at least change the subject
to be clearer.

#21Bharath Rupireddy
bharath.rupireddyforpostgres@gmail.com
In reply to: Julien Rouhaud (#20)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

On Fri, Jan 28, 2022 at 8:54 AM Julien Rouhaud <rjuju123@gmail.com> wrote:

Hi,

On Fri, Jan 28, 2022 at 08:17:39AM +0530, Bharath Rupireddy wrote:

Thanks all for the comments. pg_stat_progress_checkpoint is being
discussed in another thread [1].

[1] /messages/by-id/CALj2ACV-F+K+z+XW8fnK4MV71qz2gzAMxFnYziRgZURMB5ycAQ@mail.gmail.com

This thread was explicitly talking about log reporting and it's not clear to me
that it's now supposed to be about a new pg_stat_progress_checkpoint()
function. I think you should start a new thread or at least change the subject
to be clearer.

+1 to change the subject of that thread to 'report checkpoint progress
with pg_stat_progress_checkpoint'.

Regards,
Bharath Rupireddy.

#22Bharath Rupireddy
bharath.rupireddyforpostgres@gmail.com
In reply to: Sergey Dudoladov (#11)
1 attachment(s)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

On Thu, Jan 27, 2022 at 3:23 PM Sergey Dudoladov
<sergey.dudoladov@gmail.com> wrote:

Here's v2, rebased onto the latest master.

I've reviewed this patch. The patch builds against the master (commit
e9d4001ec592bcc9a3332547cb1b0211e8794f38) and passes all the tests.
The patch does what it intends to do, namely store the kind of the
last checkpoint in the control file and display it in the output of
the pg_control_checkpoint() function and pg_controldata utility.
I did not test it with restartpoints though. Speaking of the torn
writes, the size of the control file with this patch applied does not
exceed 8Kb.

Thanks for the review.

A few code comments:

+ char ckpt_kind[2 * MAXPGPATH];

I don't completely understand why 2 * MAXPGPATH is used here for the
buffer size.
[1] defines MAXPGPATH to be standard size of a pathname buffer. How
does it relate to ckpt_kind ?

I was using it loosely. Changed in the v3 patch.

+ ControlFile->checkPointKind = 0;

It is worth a comment that 0 is unknown, as for instance in [2]

We don't even need ControlFile->checkPointKind = 0; because
InitControlFile will memset(ControlFile, 0, sizeof(ControlFileData));,
hence removed this.

+ (flags == 0) ? "unknown" : "",

That reads as if this patch would introduce a new "unknown" checkpoint state.
Why have it here at all if after for example initdb the kind is "shutdown" ?

Yeah, even LogCheckpointStart doesn't have anything "unknown" so removed it.

The space at the strings' end (as in "wait " or "immediate ")
introduces extra whitespace in the output of pg_control_checkpoint().
A similar check at [3] places whitespace differently; that arrangement
of whitespace should remove the issue.

Changed.

Datum values[18];
bool nulls[18];

You forgot to expand these arrays.

Not sure what you meant here. The size of the array is already 19 in v2.

This breaks checkpoint file format. Need to bump PG_CONTROL_VERSION,
and pg_upgrade need to treat the change.

I added a note in the commit message to bump cat version so that the
committer will take care of it.

-  proallargtypes => '{pg_lsn,pg_lsn,text,int4,int4,bool,text,oid,xid,xid,xid,oid,xid,xid,oid,xid,xid,timestamptz}',
+  proallargtypes => '{pg_lsn,pg_lsn,text,int4,int4,bool,text,oid,xid,xid,xid,oid,xid,xid,oid,xid,xid,timestamptz,text}',

I think the additional column should be text[] instead of text, but
not sure.

We are preparing a single string of all the checkpoint kinds and
outputting as a text column, so we don't need text[].

Regards,
Bharath Rupireddy.

Attachments:

v3-0001-add-last-checkpoint-kind-to-pg_control-file.patchapplication/octet-stream; name=v3-0001-add-last-checkpoint-kind-to-pg_control-file.patchDownload
From abfa974be88a5b3fb10144873aeca2679e4e95ff Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Fri, 28 Jan 2022 08:17:27 +0000
Subject: [PATCH v3] add last checkpoint kind to pg_control file

Bump catalog version
---
 doc/src/sgml/func.sgml                  |  5 +++++
 src/backend/access/transam/xlog.c       |  2 ++
 src/backend/utils/misc/pg_controldata.c | 28 ++++++++++++++++++++++---
 src/bin/pg_controldata/pg_controldata.c | 19 +++++++++++++++++
 src/include/catalog/pg_control.h        |  5 +++--
 src/include/catalog/pg_proc.dat         |  6 +++---
 6 files changed, 57 insertions(+), 8 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 0ee6974f1c..4357ca838e 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -25093,6 +25093,11 @@ SELECT collation for ('foo' COLLATE "de_DE");
        <entry><type>timestamp with time zone</type></entry>
       </row>
 
+      <row>
+       <entry><structfield>checkpoint_kind</structfield></entry>
+       <entry><type>text</type></entry>
+      </row>
+
      </tbody>
     </tgroup>
    </table>
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index dfe2a0bcce..72255f6b8a 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -9375,6 +9375,7 @@ CreateCheckPoint(int flags)
 	LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
 	if (shutdown)
 		ControlFile->state = DB_SHUTDOWNED;
+	ControlFile->checkPointKind = flags;
 	ControlFile->checkPoint = ProcLastRecPtr;
 	ControlFile->checkPointCopy = checkPoint;
 	/* crash recovery should always recover to the end of WAL */
@@ -9761,6 +9762,7 @@ CreateRestartPoint(int flags)
 	if (ControlFile->state == DB_IN_ARCHIVE_RECOVERY &&
 		ControlFile->checkPointCopy.redo < lastCheckPoint.redo)
 	{
+		ControlFile->checkPointKind = flags;
 		ControlFile->checkPoint = lastCheckPointRecPtr;
 		ControlFile->checkPointCopy = lastCheckPoint;
 
diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c
index 781f8b8758..0640a450e1 100644
--- a/src/backend/utils/misc/pg_controldata.c
+++ b/src/backend/utils/misc/pg_controldata.c
@@ -79,20 +79,23 @@ pg_control_system(PG_FUNCTION_ARGS)
 Datum
 pg_control_checkpoint(PG_FUNCTION_ARGS)
 {
-	Datum		values[18];
-	bool		nulls[18];
+	Datum		values[19];
+	bool		nulls[19];
 	TupleDesc	tupdesc;
 	HeapTuple	htup;
 	ControlFileData *ControlFile;
 	XLogSegNo	segno;
 	char		xlogfilename[MAXFNAMELEN];
 	bool		crc_ok;
+	uint16		flags;
+#define	CHECKPOINT_KIND_TEXT_LENGTH 128
+	char 		ckpt_kind[CHECKPOINT_KIND_TEXT_LENGTH];
 
 	/*
 	 * Construct a tuple descriptor for the result row.  This must match this
 	 * function's pg_proc entry!
 	 */
-	tupdesc = CreateTemplateTupleDesc(18);
+	tupdesc = CreateTemplateTupleDesc(19);
 	TupleDescInitEntry(tupdesc, (AttrNumber) 1, "checkpoint_lsn",
 					   PG_LSNOID, -1, 0);
 	TupleDescInitEntry(tupdesc, (AttrNumber) 2, "redo_lsn",
@@ -129,6 +132,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS)
 					   XIDOID, -1, 0);
 	TupleDescInitEntry(tupdesc, (AttrNumber) 18, "checkpoint_time",
 					   TIMESTAMPTZOID, -1, 0);
+	TupleDescInitEntry(tupdesc, (AttrNumber) 19, "checkpoint_kind",
+					   TEXTOID, -1, 0);
 	tupdesc = BlessTupleDesc(tupdesc);
 
 	/* Read the control file. */
@@ -202,9 +207,26 @@ pg_control_checkpoint(PG_FUNCTION_ARGS)
 	values[17] = TimestampTzGetDatum(time_t_to_timestamptz(ControlFile->checkPointCopy.time));
 	nulls[17] = false;
 
+	MemSet(ckpt_kind, 0, CHECKPOINT_KIND_TEXT_LENGTH);
+	flags = ControlFile->checkPointKind;
+
+	snprintf(ckpt_kind, CHECKPOINT_KIND_TEXT_LENGTH, "%s%s%s%s%s%s%s%s",
+				(flags & CHECKPOINT_IS_SHUTDOWN) ? "shutdown" : "",
+				(flags & CHECKPOINT_END_OF_RECOVERY) ? " end-of-recovery" : "",
+				(flags & CHECKPOINT_IMMEDIATE) ? " immediate" : "",
+				(flags & CHECKPOINT_FORCE) ? " force" : "",
+				(flags & CHECKPOINT_WAIT) ? " wait" : "",
+				(flags & CHECKPOINT_CAUSE_XLOG) ? " wal" : "",
+				(flags & CHECKPOINT_CAUSE_TIME) ? " time" : "",
+				(flags & CHECKPOINT_FLUSH_ALL) ? " flush-all" : "");
+
+	values[18] = CStringGetTextDatum(ckpt_kind);
+	nulls[18] = false;
+
 	htup = heap_form_tuple(tupdesc, values, nulls);
 
 	PG_RETURN_DATUM(HeapTupleGetDatum(htup));
+#undef CHECKPOINT_KIND_TEXT_LENGTH
 }
 
 Datum
diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c
index f911f98d94..37c1ebf8ad 100644
--- a/src/bin/pg_controldata/pg_controldata.c
+++ b/src/bin/pg_controldata/pg_controldata.c
@@ -106,6 +106,9 @@ main(int argc, char *argv[])
 	int			c;
 	int			i;
 	int			WalSegSz;
+	uint16		flags;
+#define	CHECKPOINT_KIND_TEXT_LENGTH 128
+	char 		ckpt_kind[CHECKPOINT_KIND_TEXT_LENGTH];
 
 	pg_logging_init(argv[0]);
 	set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_controldata"));
@@ -225,6 +228,19 @@ main(int argc, char *argv[])
 		snprintf(&mock_auth_nonce_str[i * 2], 3, "%02x",
 				 (unsigned char) ControlFile->mock_authentication_nonce[i]);
 
+	MemSet(ckpt_kind, 0, CHECKPOINT_KIND_TEXT_LENGTH);
+	flags = ControlFile->checkPointKind;
+
+	snprintf(ckpt_kind, CHECKPOINT_KIND_TEXT_LENGTH, "%s%s%s%s%s%s%s%s",
+				(flags & CHECKPOINT_IS_SHUTDOWN) ? "shutdown" : "",
+				(flags & CHECKPOINT_END_OF_RECOVERY) ? " end-of-recovery" : "",
+				(flags & CHECKPOINT_IMMEDIATE) ? " immediate" : "",
+				(flags & CHECKPOINT_FORCE) ? " force" : "",
+				(flags & CHECKPOINT_WAIT) ? " wait" : "",
+				(flags & CHECKPOINT_CAUSE_XLOG) ? " wal" : "",
+				(flags & CHECKPOINT_CAUSE_TIME) ? " time" : "",
+				(flags & CHECKPOINT_FLUSH_ALL) ? " flush-all" : "");
+
 	printf(_("pg_control version number:            %u\n"),
 		   ControlFile->pg_control_version);
 	printf(_("Catalog version number:               %u\n"),
@@ -235,6 +251,8 @@ main(int argc, char *argv[])
 		   dbState(ControlFile->state));
 	printf(_("pg_control last modified:             %s\n"),
 		   pgctime_str);
+	printf(_("Latest checkpoint kind:               %s\n"),
+		   ckpt_kind);
 	printf(_("Latest checkpoint location:           %X/%X\n"),
 		   LSN_FORMAT_ARGS(ControlFile->checkPoint));
 	printf(_("Latest checkpoint's REDO location:    %X/%X\n"),
@@ -329,4 +347,5 @@ main(int argc, char *argv[])
 	printf(_("Mock authentication nonce:            %s\n"),
 		   mock_auth_nonce_str);
 	return 0;
+#undef CHECKPOINT_KIND_TEXT_LENGTH
 }
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index 1f3dc24ac1..4ec24ff518 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -128,9 +128,10 @@ typedef struct ControlFileData
 	 */
 	DBState		state;			/* see enum above */
 	pg_time_t	time;			/* time stamp of last pg_control update */
-	XLogRecPtr	checkPoint;		/* last check point record ptr */
+	uint16		checkPointKind;	/* last checkpoint kind */
+	XLogRecPtr	checkPoint;		/* last checkpoint record ptr */
 
-	CheckPoint	checkPointCopy; /* copy of last check point record */
+	CheckPoint	checkPointCopy; /* copy of last checkpoint record */
 
 	XLogRecPtr	unloggedLSN;	/* current fake LSN value, for unlogged rels */
 
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 0859dc81ca..5c0e20fe5c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11565,9 +11565,9 @@
   descr => 'pg_controldata checkpoint state information as a function',
   proname => 'pg_control_checkpoint', provolatile => 'v',
   prorettype => 'record', proargtypes => '',
-  proallargtypes => '{pg_lsn,pg_lsn,text,int4,int4,bool,text,oid,xid,xid,xid,oid,xid,xid,oid,xid,xid,timestamptz}',
-  proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
-  proargnames => '{checkpoint_lsn,redo_lsn,redo_wal_file,timeline_id,prev_timeline_id,full_page_writes,next_xid,next_oid,next_multixact_id,next_multi_offset,oldest_xid,oldest_xid_dbid,oldest_active_xid,oldest_multi_xid,oldest_multi_dbid,oldest_commit_ts_xid,newest_commit_ts_xid,checkpoint_time}',
+  proallargtypes => '{pg_lsn,pg_lsn,text,int4,int4,bool,text,oid,xid,xid,xid,oid,xid,xid,oid,xid,xid,timestamptz,text}',
+  proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+  proargnames => '{checkpoint_lsn,redo_lsn,redo_wal_file,timeline_id,prev_timeline_id,full_page_writes,next_xid,next_oid,next_multixact_id,next_multi_offset,oldest_xid,oldest_xid_dbid,oldest_active_xid,oldest_multi_xid,oldest_multi_dbid,oldest_commit_ts_xid,newest_commit_ts_xid,checkpoint_time,checkpoint_kind}',
   prosrc => 'pg_control_checkpoint' },
 
 { oid => '3443',
-- 
2.25.1

#23Julien Rouhaud
rjuju123@gmail.com
In reply to: Bharath Rupireddy (#22)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

Hi,

On Fri, Jan 28, 2022 at 01:49:19PM +0530, Bharath Rupireddy wrote:

On Thu, Jan 27, 2022 at 3:23 PM Sergey Dudoladov
<sergey.dudoladov@gmail.com> wrote:

Here's v2, rebased onto the latest master.

I've reviewed this patch. The patch builds against the master (commit
e9d4001ec592bcc9a3332547cb1b0211e8794f38) and passes all the tests.
The patch does what it intends to do, namely store the kind of the
last checkpoint in the control file and display it in the output of
the pg_control_checkpoint() function and pg_controldata utility.
I did not test it with restartpoints though. Speaking of the torn
writes, the size of the control file with this patch applied does not
exceed 8Kb.

Thanks for the review.

A few code comments:

+ char ckpt_kind[2 * MAXPGPATH];

I don't completely understand why 2 * MAXPGPATH is used here for the
buffer size.
[1] defines MAXPGPATH to be standard size of a pathname buffer. How
does it relate to ckpt_kind ?

I was using it loosely. Changed in the v3 patch.

+ ControlFile->checkPointKind = 0;

It is worth a comment that 0 is unknown, as for instance in [2]

We don't even need ControlFile->checkPointKind = 0; because
InitControlFile will memset(ControlFile, 0, sizeof(ControlFileData));,
hence removed this.

+ (flags == 0) ? "unknown" : "",

That reads as if this patch would introduce a new "unknown" checkpoint state.
Why have it here at all if after for example initdb the kind is "shutdown" ?

Yeah, even LogCheckpointStart doesn't have anything "unknown" so removed it.

The space at the strings' end (as in "wait " or "immediate ")
introduces extra whitespace in the output of pg_control_checkpoint().
A similar check at [3] places whitespace differently; that arrangement
of whitespace should remove the issue.

Changed.

Datum values[18];
bool nulls[18];

You forgot to expand these arrays.

Not sure what you meant here. The size of the array is already 19 in v2.

This breaks checkpoint file format. Need to bump PG_CONTROL_VERSION,
and pg_upgrade need to treat the change.

I added a note in the commit message to bump cat version so that the
committer will take care of it.

PG_CONTROL_VERSION is different from catversion. You should update it in this
patch.

But Horiguchi-san was also mentioning that pg_upgrade/controldata.c needs some
modifications if you change the format (thus the requirement to bump
PG_CONTROL_VERSION).

Why are you defining CHECKPOINT_KIND_TEXT_LENGTH twice? You
should just define it in some sensible header used by both files, or better
have a new function to take care of that rather than having the code
duplicated.

Also, you still didn't fix the possible flag upgrade issue.

#24Bharath Rupireddy
bharath.rupireddyforpostgres@gmail.com
In reply to: Julien Rouhaud (#23)
1 attachment(s)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

On Fri, Jan 28, 2022 at 2:20 PM Julien Rouhaud <rjuju123@gmail.com> wrote:

PG_CONTROL_VERSION is different from catversion. You should update it in this
patch.

My bad. Updated it.

But Horiguchi-san was also mentioning that pg_upgrade/controldata.c needs some
modifications if you change the format (thus the requirement to bump
PG_CONTROL_VERSION).

Also, you still didn't fix the possible flag upgrade issue.

I don't think we need to change pg_upgrade's ControlData controldata;
structure as the information may not be needed there and the while
loop there specifically parses/searches for the required
pg_controldata output texts. Am I missing something here?

Why are you defining CHECKPOINT_KIND_TEXT_LENGTH twice? You
should just define it in some sensible header used by both files, or better
have a new function to take care of that rather than having the code
duplicated.

Yeah, added the macro in pg_control.h. I also wanted to have a common
function to get checkpoint kind text and place it in
controldata_utils.c, but it doesn't have xlog.h included, so no
checkpoint flags there, hence I refrained from the common function
idea.

I think we don't need to print the checkpoint kind in pg_resetwal.c's
PrintControlValues because the pg_resetwal changes the checkpoint and
PrintControlValues just prints the fields that will not be
reset/changed by pg_resetwal. Am I missing something here?

Attaching v4.

Not related to this patch: by looking at the way the fields (like
"Latest checkpoint's TimeLineID:", "Latest checkpoint's NextOID:"
etc.) of pg_controldata output are being used in pg_resetwal.c,
pg_controldata.c, and pg_upgrade/controldata.c, I'm thinking of having
those fields as macros in pg_control.h
#define PG_CONTROL_LATEST_CHECKPOINT_TLI "Latest checkpoint's TimeLineID:"
#define PG_CONTROL_LATEST_CHECKPOINT_NEXTOID "Latest checkpoint's NextOID:"
and so on for all the pg_controldata fields would be a good idea for
better code manageability and not to miss any field text changes.

If okay, I will discuss this in a separate thread.

Regards,
Bharath Rupireddy.

Attachments:

v4-0001-add-last-checkpoint-kind-to-pg_control-file.patchapplication/x-patch; name=v4-0001-add-last-checkpoint-kind-to-pg_control-file.patchDownload
From e118edc5852434847dac36c566e7feca6f1f22ca Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Fri, 28 Jan 2022 14:30:18 +0000
Subject: [PATCH v4] add last checkpoint kind to pg_control file

Bump catalog version
---
 doc/src/sgml/func.sgml                  |  5 +++++
 src/backend/access/transam/xlog.c       |  2 ++
 src/backend/utils/misc/pg_controldata.c | 26 ++++++++++++++++++++++---
 src/bin/pg_controldata/pg_controldata.c | 17 ++++++++++++++++
 src/include/catalog/pg_control.h        | 10 +++++++---
 src/include/catalog/pg_proc.dat         |  6 +++---
 6 files changed, 57 insertions(+), 9 deletions(-)

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 0ee6974f1c..4357ca838e 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -25093,6 +25093,11 @@ SELECT collation for ('foo' COLLATE "de_DE");
        <entry><type>timestamp with time zone</type></entry>
       </row>
 
+      <row>
+       <entry><structfield>checkpoint_kind</structfield></entry>
+       <entry><type>text</type></entry>
+      </row>
+
      </tbody>
     </tgroup>
    </table>
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index dfe2a0bcce..72255f6b8a 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -9375,6 +9375,7 @@ CreateCheckPoint(int flags)
 	LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
 	if (shutdown)
 		ControlFile->state = DB_SHUTDOWNED;
+	ControlFile->checkPointKind = flags;
 	ControlFile->checkPoint = ProcLastRecPtr;
 	ControlFile->checkPointCopy = checkPoint;
 	/* crash recovery should always recover to the end of WAL */
@@ -9761,6 +9762,7 @@ CreateRestartPoint(int flags)
 	if (ControlFile->state == DB_IN_ARCHIVE_RECOVERY &&
 		ControlFile->checkPointCopy.redo < lastCheckPoint.redo)
 	{
+		ControlFile->checkPointKind = flags;
 		ControlFile->checkPoint = lastCheckPointRecPtr;
 		ControlFile->checkPointCopy = lastCheckPoint;
 
diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c
index 781f8b8758..9d48a99378 100644
--- a/src/backend/utils/misc/pg_controldata.c
+++ b/src/backend/utils/misc/pg_controldata.c
@@ -79,20 +79,22 @@ pg_control_system(PG_FUNCTION_ARGS)
 Datum
 pg_control_checkpoint(PG_FUNCTION_ARGS)
 {
-	Datum		values[18];
-	bool		nulls[18];
+	Datum		values[19];
+	bool		nulls[19];
 	TupleDesc	tupdesc;
 	HeapTuple	htup;
 	ControlFileData *ControlFile;
 	XLogSegNo	segno;
 	char		xlogfilename[MAXFNAMELEN];
 	bool		crc_ok;
+	uint16		flags;
+	char 		ckpt_kind[CHECKPOINT_KIND_TEXT_LENGTH];
 
 	/*
 	 * Construct a tuple descriptor for the result row.  This must match this
 	 * function's pg_proc entry!
 	 */
-	tupdesc = CreateTemplateTupleDesc(18);
+	tupdesc = CreateTemplateTupleDesc(19);
 	TupleDescInitEntry(tupdesc, (AttrNumber) 1, "checkpoint_lsn",
 					   PG_LSNOID, -1, 0);
 	TupleDescInitEntry(tupdesc, (AttrNumber) 2, "redo_lsn",
@@ -129,6 +131,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS)
 					   XIDOID, -1, 0);
 	TupleDescInitEntry(tupdesc, (AttrNumber) 18, "checkpoint_time",
 					   TIMESTAMPTZOID, -1, 0);
+	TupleDescInitEntry(tupdesc, (AttrNumber) 19, "checkpoint_kind",
+					   TEXTOID, -1, 0);
 	tupdesc = BlessTupleDesc(tupdesc);
 
 	/* Read the control file. */
@@ -202,6 +206,22 @@ pg_control_checkpoint(PG_FUNCTION_ARGS)
 	values[17] = TimestampTzGetDatum(time_t_to_timestamptz(ControlFile->checkPointCopy.time));
 	nulls[17] = false;
 
+	MemSet(ckpt_kind, 0, CHECKPOINT_KIND_TEXT_LENGTH);
+	flags = ControlFile->checkPointKind;
+
+	snprintf(ckpt_kind, CHECKPOINT_KIND_TEXT_LENGTH, "%s%s%s%s%s%s%s%s",
+				(flags & CHECKPOINT_IS_SHUTDOWN) ? "shutdown" : "",
+				(flags & CHECKPOINT_END_OF_RECOVERY) ? " end-of-recovery" : "",
+				(flags & CHECKPOINT_IMMEDIATE) ? " immediate" : "",
+				(flags & CHECKPOINT_FORCE) ? " force" : "",
+				(flags & CHECKPOINT_WAIT) ? " wait" : "",
+				(flags & CHECKPOINT_CAUSE_XLOG) ? " wal" : "",
+				(flags & CHECKPOINT_CAUSE_TIME) ? " time" : "",
+				(flags & CHECKPOINT_FLUSH_ALL) ? " flush-all" : "");
+
+	values[18] = CStringGetTextDatum(ckpt_kind);
+	nulls[18] = false;
+
 	htup = heap_form_tuple(tupdesc, values, nulls);
 
 	PG_RETURN_DATUM(HeapTupleGetDatum(htup));
diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c
index f911f98d94..f378cfcda5 100644
--- a/src/bin/pg_controldata/pg_controldata.c
+++ b/src/bin/pg_controldata/pg_controldata.c
@@ -106,6 +106,8 @@ main(int argc, char *argv[])
 	int			c;
 	int			i;
 	int			WalSegSz;
+	uint16		flags;
+	char 		ckpt_kind[CHECKPOINT_KIND_TEXT_LENGTH];
 
 	pg_logging_init(argv[0]);
 	set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_controldata"));
@@ -225,6 +227,19 @@ main(int argc, char *argv[])
 		snprintf(&mock_auth_nonce_str[i * 2], 3, "%02x",
 				 (unsigned char) ControlFile->mock_authentication_nonce[i]);
 
+	MemSet(ckpt_kind, 0, CHECKPOINT_KIND_TEXT_LENGTH);
+	flags = ControlFile->checkPointKind;
+
+	snprintf(ckpt_kind, CHECKPOINT_KIND_TEXT_LENGTH, "%s%s%s%s%s%s%s%s",
+				(flags & CHECKPOINT_IS_SHUTDOWN) ? "shutdown" : "",
+				(flags & CHECKPOINT_END_OF_RECOVERY) ? " end-of-recovery" : "",
+				(flags & CHECKPOINT_IMMEDIATE) ? " immediate" : "",
+				(flags & CHECKPOINT_FORCE) ? " force" : "",
+				(flags & CHECKPOINT_WAIT) ? " wait" : "",
+				(flags & CHECKPOINT_CAUSE_XLOG) ? " wal" : "",
+				(flags & CHECKPOINT_CAUSE_TIME) ? " time" : "",
+				(flags & CHECKPOINT_FLUSH_ALL) ? " flush-all" : "");
+
 	printf(_("pg_control version number:            %u\n"),
 		   ControlFile->pg_control_version);
 	printf(_("Catalog version number:               %u\n"),
@@ -235,6 +250,8 @@ main(int argc, char *argv[])
 		   dbState(ControlFile->state));
 	printf(_("pg_control last modified:             %s\n"),
 		   pgctime_str);
+	printf(_("Latest checkpoint kind:               %s\n"),
+		   ckpt_kind);
 	printf(_("Latest checkpoint location:           %X/%X\n"),
 		   LSN_FORMAT_ARGS(ControlFile->checkPoint));
 	printf(_("Latest checkpoint's REDO location:    %X/%X\n"),
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index 1f3dc24ac1..f2209acaa1 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -22,11 +22,14 @@
 
 
 /* Version identifier for this pg_control format */
-#define PG_CONTROL_VERSION	1300
+#define PG_CONTROL_VERSION	1500
 
 /* Nonce key length, see below */
 #define MOCK_AUTH_NONCE_LEN		32
 
+/* Maximum length of checkpoint kind text */
+#define	CHECKPOINT_KIND_TEXT_LENGTH 128
+
 /*
  * Body of CheckPoint XLOG records.  This is declared here because we keep
  * a copy of the latest one in pg_control for possible disaster recovery.
@@ -128,9 +131,10 @@ typedef struct ControlFileData
 	 */
 	DBState		state;			/* see enum above */
 	pg_time_t	time;			/* time stamp of last pg_control update */
-	XLogRecPtr	checkPoint;		/* last check point record ptr */
+	uint16		checkPointKind;	/* last checkpoint kind */
+	XLogRecPtr	checkPoint;		/* last checkpoint record ptr */
 
-	CheckPoint	checkPointCopy; /* copy of last check point record */
+	CheckPoint	checkPointCopy; /* copy of last checkpoint record */
 
 	XLogRecPtr	unloggedLSN;	/* current fake LSN value, for unlogged rels */
 
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 0859dc81ca..5c0e20fe5c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11565,9 +11565,9 @@
   descr => 'pg_controldata checkpoint state information as a function',
   proname => 'pg_control_checkpoint', provolatile => 'v',
   prorettype => 'record', proargtypes => '',
-  proallargtypes => '{pg_lsn,pg_lsn,text,int4,int4,bool,text,oid,xid,xid,xid,oid,xid,xid,oid,xid,xid,timestamptz}',
-  proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
-  proargnames => '{checkpoint_lsn,redo_lsn,redo_wal_file,timeline_id,prev_timeline_id,full_page_writes,next_xid,next_oid,next_multixact_id,next_multi_offset,oldest_xid,oldest_xid_dbid,oldest_active_xid,oldest_multi_xid,oldest_multi_dbid,oldest_commit_ts_xid,newest_commit_ts_xid,checkpoint_time}',
+  proallargtypes => '{pg_lsn,pg_lsn,text,int4,int4,bool,text,oid,xid,xid,xid,oid,xid,xid,oid,xid,xid,timestamptz,text}',
+  proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+  proargnames => '{checkpoint_lsn,redo_lsn,redo_wal_file,timeline_id,prev_timeline_id,full_page_writes,next_xid,next_oid,next_multixact_id,next_multi_offset,oldest_xid,oldest_xid_dbid,oldest_active_xid,oldest_multi_xid,oldest_multi_dbid,oldest_commit_ts_xid,newest_commit_ts_xid,checkpoint_time,checkpoint_kind}',
   prosrc => 'pg_control_checkpoint' },
 
 { oid => '3443',
-- 
2.25.1

#25Julien Rouhaud
rjuju123@gmail.com
In reply to: Bharath Rupireddy (#24)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

Hi,

On Fri, Jan 28, 2022 at 08:21:52PM +0530, Bharath Rupireddy wrote:

I don't think we need to change pg_upgrade's ControlData controldata;
structure as the information may not be needed there and the while
loop there specifically parses/searches for the required
pg_controldata output texts. Am I missing something here?

Right, I was remembering that there was a check that all expected fields were
found but after double checking I was clearly wrong, sorry about that.

Also, you still didn't fix the possible flag upgrade issue.

Unless I'm missing something that's an issue that you still haven't addressed
or explained why it's not a problem?

Why are you defining CHECKPOINT_KIND_TEXT_LENGTH twice? You
should just define it in some sensible header used by both files, or better
have a new function to take care of that rather than having the code
duplicated.

Yeah, added the macro in pg_control.h. I also wanted to have a common
function to get checkpoint kind text and place it in
controldata_utils.c, but it doesn't have xlog.h included, so no
checkpoint flags there, hence I refrained from the common function
idea.

That's a bit annoying, I'm not sure what's best to do here.

#26Andres Freund
andres@anarazel.de
In reply to: Tomas Vondra (#6)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

Hi,

On 2021-12-08 03:04:23 +0100, Tomas Vondra wrote:

On 12/8/21 02:54, Bharath Rupireddy wrote:

I'm not sure about adding it to control data, though. That doesn't seem
like a very good match for something that's mostly for monitoring.

Having it in the control data file (along with the existing checkpoint
information) will be helpful to know what was the last checkpoint
information and we can use the existing pg_control_checkpoint function
or the tool to emit that info. I plan to add an int16 flag as
suggested by Justin in this thread and come up with a patch.

OK, although I'm not sure it's all that useful (if we have that in some
sort of system view).

I don't think we should add stuff like this to the control file. We want to
keep ControlFileData within a disk sector size / 512 bytes (so that we don't
need to care about torn writes etc). Adding information that we don't really
need will byte us at some point, because at that point there will be reliance
on the added data.

Nor have I read a convincing reason for needing the data to be readily
accessible when the server is shut down?

Greetings,

Andres Freund

#27Andres Freund
andres@anarazel.de
In reply to: Bharath Rupireddy (#1)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

Hi,

On 2021-12-07 20:06:22 +0530, Bharath Rupireddy wrote:

One concern is that we don't want to increase the size of pg_controldata by
more than the typical block size (of 8K) to avoid any torn-writes.

The limit is 512 bytes (a disk sector), not 8K. There are plenty devices with
4K sectors as well, but I'm not aware of any with 8K.

Greetings,

Andres Freund

#28Kyotaro Horiguchi
horikyota.ntt@gmail.com
In reply to: Bharath Rupireddy (#22)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

At Fri, 28 Jan 2022 13:49:19 +0530, Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> wrote in

-  proallargtypes => '{pg_lsn,pg_lsn,text,int4,int4,bool,text,oid,xid,xid,xid,oid,xid,xid,oid,xid,xid,timestamptz}',
+  proallargtypes => '{pg_lsn,pg_lsn,text,int4,int4,bool,text,oid,xid,xid,xid,oid,xid,xid,oid,xid,xid,timestamptz,text}',

I think the additional column should be text[] instead of text, but
not sure.

We are preparing a single string of all the checkpoint kinds and
outputting as a text column, so we don't need text[].

What I meant to suggest is to change it to an array of text instead of
a text that consists of multiple labels concatenated by spaces.

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center

#29Kyotaro Horiguchi
horikyota.ntt@gmail.com
In reply to: Julien Rouhaud (#25)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

At Sat, 29 Jan 2022 00:10:19 +0800, Julien Rouhaud <rjuju123@gmail.com> wrote in

On Fri, Jan 28, 2022 at 08:21:52PM +0530, Bharath Rupireddy wrote:

Also, you still didn't fix the possible flag upgrade issue.

Unless I'm missing something that's an issue that you still haven't addressed
or explained why it's not a problem?

As Bharath said, pg_upgreade reads a part of old cluster's controldata
that is needed to check compatibility so I agree that we don't have
flag upgrade issue I mentioned.

Why are you defining CHECKPOINT_KIND_TEXT_LENGTH twice? You
should just define it in some sensible header used by both files, or better
have a new function to take care of that rather than having the code
duplicated.

Yeah, added the macro in pg_control.h. I also wanted to have a common
function to get checkpoint kind text and place it in
controldata_utils.c, but it doesn't have xlog.h included, so no
checkpoint flags there, hence I refrained from the common function
idea.

That's a bit annoying, I'm not sure what's best to do here.

I misunderstood that the size is restricted to 8k but it is really 512
bytes as Andres pointed. So we don't add it at least as a text. This
means pg_controldata need to translate the flags into human-readable
text but, to be clear, I still don't think its usefull in the control
data.

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center

#30Julien Rouhaud
rjuju123@gmail.com
In reply to: Kyotaro Horiguchi (#29)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

Hi,

On Mon, Jan 31, 2022 at 11:10:45AM +0900, Kyotaro Horiguchi wrote:

This means pg_controldata need to translate the flags into human-readable
text but, to be clear, I still don't think its usefull in the control
data.

I've been saying that since my first email, I also don't see any scenario where
having this specific information can be of any help.

Given Andres feedback, unless someone can provide some realistic use case I
think we should mark this patch as rejected and focus on a
pg_progress_checkpoint feature. I will do so in a couple of days when I will
close this commitfest.

#31Bharath Rupireddy
bharath.rupireddyforpostgres@gmail.com
In reply to: Julien Rouhaud (#30)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

On Mon, Jan 31, 2022 at 9:10 AM Julien Rouhaud <rjuju123@gmail.com> wrote:

Hi,

On Mon, Jan 31, 2022 at 11:10:45AM +0900, Kyotaro Horiguchi wrote:

This means pg_controldata need to translate the flags into human-readable
text but, to be clear, I still don't think its usefull in the control
data.

I've been saying that since my first email, I also don't see any scenario where
having this specific information can be of any help.

Given Andres feedback, unless someone can provide some realistic use case I
think we should mark this patch as rejected and focus on a
pg_progress_checkpoint feature. I will do so in a couple of days when I will
close this commitfest.

The size of ControlFileData is 296 bytes currently and the sector
limit is of 512 bytes (PG_CONTROL_MAX_SAFE_SIZE), if we feel that this
extra 2 bytes of checkpoint flags isn't worth storing in the control
file, I'm pretty much okay with it.

Regards,
Bharath Rupireddy.

#32Julien Rouhaud
rjuju123@gmail.com
In reply to: Bharath Rupireddy (#31)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

Hi,

On Mon, Jan 31, 2022 at 10:58:31AM +0530, Bharath Rupireddy wrote:

The size of ControlFileData is 296 bytes currently and the sector
limit is of 512 bytes (PG_CONTROL_MAX_SAFE_SIZE), if we feel that this
extra 2 bytes of checkpoint flags isn't worth storing in the control
file, I'm pretty much okay with it.

For now we have some room, but we will likely keep consuming bytes in that file
for more critical features and it's almost certain that at one point we will
regret wasting 2 bytes for that.

#33Michael Paquier
michael@paquier.xyz
In reply to: Julien Rouhaud (#32)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

On Mon, Jan 31, 2022 at 01:54:16PM +0800, Julien Rouhaud wrote:

For now we have some room, but we will likely keep consuming bytes in that file
for more critical features and it's almost certain that at one point we will
regret wasting 2 bytes for that.

Agreed to drop the patch. That's only two bytes, but we may regret
that in the future and this is not critical for the system.
--
Michael

#34Bharath Rupireddy
bharath.rupireddyforpostgres@gmail.com
In reply to: Michael Paquier (#33)
Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

On Mon, Jan 31, 2022 at 12:47 PM Michael Paquier <michael@paquier.xyz> wrote:

On Mon, Jan 31, 2022 at 01:54:16PM +0800, Julien Rouhaud wrote:

For now we have some room, but we will likely keep consuming bytes in that file
for more critical features and it's almost certain that at one point we will
regret wasting 2 bytes for that.

Agreed to drop the patch. That's only two bytes, but we may regret
that in the future and this is not critical for the system.

I agree. Thank you all for your review.