Multiple pg_waldump --rmgr options

On Tue, May 18, 2021 at 04:50:31PM +0300, Heikki Linnakangas wrote:

I wanted to dump all heap WAL records with pg_waldump, so I did this:

$ pg_waldump --rmgr=heap --rmgr=heap2 data/pg_wal/000000010000000000000001 --stat=record
Type N (%) Record size (%) FPI size (%) Combined size (%)
---- - --- ----------- --- -------- --- ------------- ---
Heap2/PRUNE 268 ( 8.74) 18192 ( 2.73) 0 ( 0.00) 18192 ( 1.74)
Heap2/VACUUM 55 ( 1.79) 4940 ( 0.74) 0 ( 0.00) 4940 ( 0.47)
Heap2/FREEZE_PAGE 277 ( 9.03) 186868 ( 28.03) 0 ( 0.00) 186868 ( 17.86)
Heap2/VISIBLE 467 ( 15.23) 27783 ( 4.17) 376832 ( 99.34) 404615 ( 38.68)
Heap2/MULTI_INSERT 1944 ( 63.38) 354800 ( 53.21) 2520 ( 0.66) 357320 ( 34.16)
Heap2/MULTI_INSERT+INIT 56 ( 1.83) 74152 ( 11.12) 0 ( 0.00) 74152 ( 7.09)
-------- -------- -------- --------
Total 3067 666735 [63.74%] 379352 [36.26%] 1046087 [100%]
pg_waldump: fatal: error in WAL record at 0/1680118: invalid record length at 0/1680150: wanted 24, got 0

That didn't do what I wanted. It only printed the Heap2 records, not Heap,
even though I specified both. The reason is that if you specify multiple
--rmgr options, only the last one takes effect.

I propose the attached to allow selecting multiple rmgrs, by giving multiple
--rmgr options. With that, it works the way I expected:

The change and the patch look sensible to me.

At Tue, 18 May 2021 23:23:02 +0800, Julien Rouhaud <rjuju123@gmail.com> wrote in

On Tue, May 18, 2021 at 04:50:31PM +0300, Heikki Linnakangas wrote:

That didn't do what I wanted. It only printed the Heap2 records, not Heap,
even though I specified both. The reason is that if you specify multiple
--rmgr options, only the last one takes effect.

I propose the attached to allow selecting multiple rmgrs, by giving multiple
--rmgr options. With that, it works the way I expected:

The change and the patch look sensible to me.

+1.

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center

On Wed, May 19, 2021 at 11:50:52AM +0900, Kyotaro Horiguchi wrote:

At Tue, 18 May 2021 23:23:02 +0800, Julien Rouhaud <rjuju123@gmail.com> wrote in

The change and the patch look sensible to me.

+1.

Agreed.
--
Michael

On 18 May 2021, at 15:50, Heikki Linnakangas <hlinnaka@iki.fi> wrote:

The reason is that if you specify multiple --rmgr options, only the last one takes effect.

That's in line with how options are handled for most binaries, so this will go
against that. That being said, I don't think thats a problem here really given
what this tool is and it's intended usecase.

This patch makes the special case "--rmgr=list" a bit more awkward than before
IMO, as it breaks the list processing, but nothing we can't live with.

I propose the attached to allow selecting multiple rmgrs

I agree with the other +1's in this thread, and am marking this as ready for
committer.

As a tiny nitpick for readability, I would move this line inside the string
comparison case where the rmgr is selected. Not that it makes any difference
in practice, but since that's where the filtering is set it seems a hair
tidier.
+ config.filter_by_rmgr_enabled = true;

--
Daniel Gustafsson https://vmware.com/

On 28/06/2021 13:34, Daniel Gustafsson wrote:

On 18 May 2021, at 15:50, Heikki Linnakangas <hlinnaka@iki.fi> wrote:

The reason is that if you specify multiple --rmgr options, only the last one takes effect.

That's in line with how options are handled for most binaries, so this will go
against that. That being said, I don't think thats a problem here really given
what this tool is and it's intended usecase.

There is some precedent for this with the pg_dump --table option, for
example.

In general, I think it's weird that the latest option wins. If you
specify the same option multiple times, and it's not something like
--rmgr or --table where it makes sense, it's most likely user error.
Printing an error would be nicer than ignoring all but the last
instance. But I'm not going to try changing that now.

I propose the attached to allow selecting multiple rmgrs

I agree with the other +1's in this thread, and am marking this as ready for
committer.

As a tiny nitpick for readability, I would move this line inside the string
comparison case where the rmgr is selected. Not that it makes any difference
in practice, but since that's where the filtering is set it seems a hair
tidier.
+ config.filter_by_rmgr_enabled = true;

Ok, changed it that way.

I tried to be defensive against WAL records with bogus xl_rmid values here:

@@ -1098,8 +1100,9 @@ main(int argc, char **argv)
}

/* apply all specified filters */
-               if (config.filter_by_rmgr != -1 &&
-                       config.filter_by_rmgr != record->xl_rmid)
+               if (config.filter_by_rmgr_enabled &&
+                       (record->xl_rmid < 0 || record->xl_rmid > RM_MAX_ID ||
+                        !config.filter_by_rmgr[record->xl_rmid]))
continue;

But looking closer, that's pointless. We use record->xl_rmid directly as
array index elsewhere, and that's OK because ValidXLogRecordHeader()
checks that xl_rmid <= RM_MAX_ID. And the 'xl_rmid < 0' check is
unnecessary because the field is unsigned. So I'll remove those, and
commit this tomorrow.

- Heikki

On 30 Jun 2021, at 22:39, Heikki Linnakangas <hlinnaka@iki.fi> wrote:

In general, I think it's weird that the latest option wins. If you specify the same option multiple times, and it's not something like --rmgr or --table where it makes sense, it's most likely user error. Printing an error would be nicer than ignoring all but the last instance. But I'm not going to try changing that now.

AFAIK, the traditional "defense" for it when building a commandline with
scripts which loop over input, to avoid the need for any data structure holding
the options for deduplication. No idea how common that is these days, but I've
seen it in production in the past for sure.

I tried to be defensive against WAL records with bogus xl_rmid values here:

@@ -1098,8 +1100,9 @@ main(int argc, char **argv)
}
/* apply all specified filters */
-               if (config.filter_by_rmgr != -1 &&
-                       config.filter_by_rmgr != record->xl_rmid)
+               if (config.filter_by_rmgr_enabled &&
+                       (record->xl_rmid < 0 || record->xl_rmid > RM_MAX_ID ||
+                        !config.filter_by_rmgr[record->xl_rmid]))
continue;

But looking closer, that's pointless. We use record->xl_rmid directly as array index elsewhere, and that's OK because ValidXLogRecordHeader() checks that xl_rmid <= RM_MAX_ID. And the 'xl_rmid < 0' check is unnecessary because the field is unsigned. So I'll remove those, and commit this tomorrow.

+1

--
Daniel Gustafsson https://vmware.com/