psql: Count all table footer lines in pager setup
Here's a patch to fix the pager setup when printing tables with large
footers, e.g., long view definitions.
For testing create a view with many columns (50 in this example):
select 'create view longview as select ' || string_agg('1 f' || s, ', ')
from generate_series(1, 50) s
\gexec
Then describe the view with verbose output to also print its definition
in the table footer:
\d+ longview
Without the patch, no pager is used when the terminal can display 55 or
more lines (check with `stty size`). Below 55 available lines the pager
is always used because the table itself requires 54 lines (50 view
columns + 3 header lines + 1 "View definition" footer line).
With the patch applied, the lines of the view definition also count
towards the line total and the pager is used when more than 54 lines are
available.
--
Erik Wienhold
Attachments:
v1-0001-psql-Count-all-table-footer-lines-in-pager-setup.patchtext/plain; charset=us-asciiDownload
From df68e0ca50d982e3dc1dbc39d7868dcbfd011132 Mon Sep 17 00:00:00 2001
From: Erik Wienhold <ewie@ewie.name>
Date: Wed, 7 May 2025 03:43:46 +0200
Subject: [PATCH v1] psql: Count all table footer lines in pager setup
Until now every table footer was counted as a single line when
determining if the pager is needed. This fails to trigger the pager
when describing a view with a long definition using command \d+. Fix
that by counting the actual lines of the footer text.
---
src/fe_utils/print.c | 13 +++++++------
1 file changed, 7 insertions(+), 6 deletions(-)
diff --git a/src/fe_utils/print.c b/src/fe_utils/print.c
index 4af0f32f2fc..accb12145da 100644
--- a/src/fe_utils/print.c
+++ b/src/fe_utils/print.c
@@ -3416,12 +3416,13 @@ IsPagerNeeded(const printTableContent *cont, int extra_lines, bool expanded,
{
printTableFooter *f;
- /*
- * FIXME -- this is slightly bogus: it counts the number of
- * footers, not the number of lines in them.
- */
- for (f = cont->footers; f; f = f->next)
- lines++;
+ for (f = cont->footers; f; f = f->next) {
+ int f_lines;
+
+ pg_wcssize((const unsigned char *) f->data, strlen(f->data),
+ cont->opt->encoding, NULL, &f_lines, NULL);
+ lines += f_lines;
+ }
}
*fout = PageOutput(lines + extra_lines, cont->opt);
--
2.50.1
Patch looks good, applies and works. Needs a pgindent run:
- for (f = cont->footers; f; f = f->next) {
- int f_lines;
+ for (f = cont->footers; f; f = f->next)
+ {
+ int f_lines;
Cheers,
Greg
--
Crunchy Data - https://www.crunchydata.com
Enterprise Postgres Software Products & Tech Support
On 2025-08-11 20:28 +0200, Greg Sabino Mullane wrote:
Patch looks good, applies and works. Needs a pgindent run:
Thanks for the review. Here's v2 with proper formatting.
--
Erik
Attachments:
v2-0001-psql-Count-all-table-footer-lines-in-pager-setup.patchtext/plain; charset=us-asciiDownload
From 96479e8bcc2d6598ce7d76ff7282541dc4d2e58e Mon Sep 17 00:00:00 2001
From: Erik Wienhold <ewie@ewie.name>
Date: Wed, 7 May 2025 03:43:46 +0200
Subject: [PATCH v2] psql: Count all table footer lines in pager setup
Until now every table footer was counted as a single line when
determining if the pager is needed. This fails to trigger the pager
when describing a view with a long definition using command \d+. Fix
that by counting the actual lines of the footer text.
---
src/fe_utils/print.c | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/src/fe_utils/print.c b/src/fe_utils/print.c
index 4af0f32f2fc..103fef4b46b 100644
--- a/src/fe_utils/print.c
+++ b/src/fe_utils/print.c
@@ -3416,12 +3416,14 @@ IsPagerNeeded(const printTableContent *cont, int extra_lines, bool expanded,
{
printTableFooter *f;
- /*
- * FIXME -- this is slightly bogus: it counts the number of
- * footers, not the number of lines in them.
- */
for (f = cont->footers; f; f = f->next)
- lines++;
+ {
+ int f_lines;
+
+ pg_wcssize((const unsigned char *) f->data, strlen(f->data),
+ cont->opt->encoding, NULL, &f_lines, NULL);
+ lines += f_lines;
+ }
}
*fout = PageOutput(lines + extra_lines, cont->opt);
--
2.50.1
Erik Wienhold <ewie@ewie.name> writes:
On 2025-08-11 20:28 +0200, Greg Sabino Mullane wrote:
Patch looks good, applies and works. Needs a pgindent run:
Thanks for the review. Here's v2 with proper formatting.
This appears to fix the problem it sets out to fix, but it looks
to me like there are adjacent problems of the same ilk; do you
feel like looking at those?
Specifically, I wondered whether we had the same bug with table
headers or cells that contain newlines. It looks like
print_aligned_text() is okay, because it counts up those newlines
and includes them in the extra_output_lines value that it passes
to IsPagerNeeded(). But print_aligned_vertical() and printTable()
have no such logic. Am I missing something about why those cases
need not consider this issue? If not, can we fix that by moving
all the "extra_lines" logic into IsPagerNeeded(), and if not
what shall we do about it?
regards, tom lane
On 2025-08-17 17:19 +0200, Tom Lane wrote:
Erik Wienhold <ewie@ewie.name> writes:
On 2025-08-11 20:28 +0200, Greg Sabino Mullane wrote:
Patch looks good, applies and works. Needs a pgindent run:
Thanks for the review. Here's v2 with proper formatting.
This appears to fix the problem it sets out to fix, but it looks
to me like there are adjacent problems of the same ilk; do you
feel like looking at those?Specifically, I wondered whether we had the same bug with table
headers or cells that contain newlines. It looks like
print_aligned_text() is okay, because it counts up those newlines
and includes them in the extra_output_lines value that it passes
to IsPagerNeeded(). But print_aligned_vertical() and printTable()
have no such logic.
I looked into these cases now and the unaligned and expanded modes
indeed miss to use the pager. Tested with the following definitions and
patch v2 applied:
create function generate_lines(n int)
returns table (lines text)
language sql
as $$ select string_agg(s::text, e'\n') from generate_series(1, n) s $$;
-- This creates a view name and column name with 24 line breaks when truncated
-- to the default NAMEDATALEN (63 = 9*2 + 15*3)
select format('create view %I (c) as select null;'
'create view nl_column (%1$I) as select null;',
string_agg(s::text, e'\n'))
from generate_series(1, current_setting('max_identifier_length')::int) s
\gexec
Test results:
\pset format aligned \pset expanded off
1a) select * from generate_lines(25); -- uses pager (<= 27 lines)
1b) select * from nl_column; -- uses pager (<= 27 lines)
1c) \d nl_column -- uses pager (<= 27 lines)
1d) \d+ nl_column -- uses pager (<= 53 lines)
1e) \d "1<TAB> -- no pager
1f) \d+ "1<TAB> -- no pager
\pset format unaligned \pset expanded off
2a) select * from generate_lines(25); -- no pager
2b) select * from nl_column; -- no pager
2c) \d nl_column -- no pager
2d) \d+ nl_column -- uses pager (<= 28 lines)
2e) \d "1<TAB> -- no pager
2f) \d+ "1<TAB> -- no pager
\pset format unaligned \pset expanded on
3a) select * from generate_lines(25); -- no pager
3b) select * from nl_column; -- no pager
-- Expanded mode does not apply to \d -> same output as 2c to 2f
3c) \d nl_column -- no pager
3d) \d+ nl_column -- uses pager (<= 28 lines)
3e) \d "1<TAB> -- no pager
3f) \d+ "1<TAB> -- no pager
\pset format aligned \pset expanded on
4a) select * from generate_lines(25); -- no pager
4b) select * from nl_column -- no pager
-- Here as well: same output as 1c to 1f
4c) \d nl_column -- uses pager (<= 27 lines)
4d) \d+ nl_column -- uses pager (<= 53 lines)
4e) \d "1<TAB> -- no pager
4f) \d+ "1<TAB> -- no pager
Fixing the generate_lines cases goes without saying since that one is
about cells. But I'm not so sure if fixing the header cases (nl_column
and view u&"1\000a2\000a3...") is worth the effort. I've seen some
annoying uses of quoted identifiers in the past but so far haven't come
across identifiers with line breaks. (That would really take the cake!)
But of course, fixing those as well would tie up loose ends.
Am I missing something about why those cases need not consider this
issue?
I don't know either. The commits that added the IsPagerNeeded calls
with extra_lines=0 don't say anything about that.
If not, can we fix that by moving all the "extra_lines" logic into
IsPagerNeeded(), and if not what shall we do about it?
Looks doable. I'll prepare patch. Manually testing this with the
various printing options won't be fun though. Or are there tools to
simulate the terminal size? I'm using tmux so it's fairly easy to
create a horizonal split and resize the pane to be N lines high. But a
script to run psql and check if the pager was invoked for a terminal
with N lines would be cool.
--
Erik Wienhold
On 2025-08-19 03:52 +0200, Erik Wienhold wrote:
On 2025-08-17 17:19 +0200, Tom Lane wrote:
This appears to fix the problem it sets out to fix, but it looks
to me like there are adjacent problems of the same ilk; do you
feel like looking at those?Specifically, I wondered whether we had the same bug with table
headers or cells that contain newlines. It looks like
print_aligned_text() is okay, because it counts up those newlines
and includes them in the extra_output_lines value that it passes
to IsPagerNeeded(). But print_aligned_vertical() and printTable()
have no such logic.I looked into these cases now and the unaligned and expanded modes
indeed miss to use the pager.Fixing the generate_lines cases goes without saying since that one is
about cells. But I'm not so sure if fixing the header cases (nl_column
and view u&"1\000a2\000a3...") is worth the effort.If not, can we fix that by moving all the "extra_lines" logic into
IsPagerNeeded(), and if not what shall we do about it?Looks doable. I'll prepare patch.
Here's v3 to address all of this. I split it into three separate
patches:
* v3-0001 is just a refactoring of the extra line counting, moving it
into IsPagerNeeded. This fixes the line counting for unaligned and
expanded output which, until now, called IsPagerNeeded with zero
extra_lines.
* v3-0002 fixes the line count in expanded mode when headers
contain more line breaks than the respective cells. This must be
checked for every cell because expanded mode repeats the headers for
every record. The header line counts are calculated once before
looping over all cells.
* v3-0003 is the original fix for the footer lines with two additions:
Here I also fix the counting of header lines in normal output mode
because I noticed that header lines are always counted right now even
when the header is not printed (in tuples_only mode or expanded mode.)
This now also considers the table title, if present.
Manually testing this with the various printing options won't be fun
though. Or are there tools to simulate the terminal size? I'm using
tmux so it's fairly easy to create a horizonal split and resize the
pane to be N lines high. But a script to run psql and check if the
pager was invoked for a terminal with N lines would be cool.
I also came up with the attached test-psql-pager.py script to run the
test cases from my previous post against different output modes to count
the maximum number of lines for which psql sill triggers the pager.
This uses termios to control the terminal size. The script compares the
actual output (pager line count, output mode, test case) against a *.out
file with the expected output (similar to pg_regress) which I've also
attached. The *.out files correspond the respective patch numbers to
show how each patch improves the pager setup. The expected output in
0-master.out is the baseline on master.
--
Erik Wienhold
Attachments:
v3-0001-psql-Fix-pager-setup-for-unaligned-and-expanded-o.patchtext/plain; charset=us-asciiDownload
From 94704e56585e9e60838f14821630c48b9ccbb164 Mon Sep 17 00:00:00 2001
From: Erik Wienhold <ewie@ewie.name>
Date: Tue, 9 Sep 2025 02:29:26 +0200
Subject: [PATCH v3 1/3] psql: Fix pager setup for unaligned and expanded
output
Move the line counting logic to the pager setup so that it applies to
unaligned and expanded output modes as well. While at it, get rid of
the top if-else branch in IsPagerNeeded in favor of an early return when
not writing to stdout. This avoids having extra indentation for the
most part of that function.
---
src/fe_utils/print.c | 146 ++++++++++++++++++++++++-------------------
1 file changed, 80 insertions(+), 66 deletions(-)
diff --git a/src/fe_utils/print.c b/src/fe_utils/print.c
index 4af0f32f2fc..44d5af7577e 100644
--- a/src/fe_utils/print.c
+++ b/src/fe_utils/print.c
@@ -266,8 +266,8 @@ static const unicodeStyleFormat unicode_style = {
/* Local functions */
static int strlen_max_width(unsigned char *str, int *target_width, int encoding);
-static void IsPagerNeeded(const printTableContent *cont, int extra_lines, bool expanded,
- FILE **fout, bool *is_pager);
+static void IsPagerNeeded(const printTableContent *cont, unsigned int *width_wrap,
+ bool expanded, FILE **fout, bool *is_pager);
static void print_aligned_vertical(const printTableContent *cont,
FILE *fout, bool is_pager);
@@ -656,8 +656,6 @@ print_aligned_text(const printTableContent *cont, FILE *fout, bool is_pager)
unsigned char **format_buf;
unsigned int width_total;
unsigned int total_header_width;
- unsigned int extra_row_output_lines = 0;
- unsigned int extra_output_lines = 0;
const char *const *ptr;
@@ -722,14 +720,9 @@ print_aligned_text(const printTableContent *cont, FILE *fout, bool is_pager)
max_nl_lines[i] = nl_lines;
if (bytes_required > max_bytes[i])
max_bytes[i] = bytes_required;
- if (nl_lines > extra_row_output_lines)
- extra_row_output_lines = nl_lines;
width_header[i] = width;
}
- /* Add height of tallest header column */
- extra_output_lines += extra_row_output_lines;
- extra_row_output_lines = 0;
/* scan all cells, find maximum width, compute cell_count */
for (i = 0, ptr = cont->cells; *ptr; ptr++, i++, cell_count++)
@@ -892,40 +885,7 @@ print_aligned_text(const printTableContent *cont, FILE *fout, bool is_pager)
/* Check if newlines or our wrapping now need the pager */
if (!is_pager && fout == stdout)
{
- /* scan all cells, find maximum width, compute cell_count */
- for (i = 0, ptr = cont->cells; *ptr; ptr++, cell_count++)
- {
- int width,
- nl_lines,
- bytes_required;
-
- pg_wcssize((const unsigned char *) *ptr, strlen(*ptr), encoding,
- &width, &nl_lines, &bytes_required);
-
- /*
- * A row can have both wrapping and newlines that cause it to
- * display across multiple lines. We check for both cases below.
- */
- if (width > 0 && width_wrap[i])
- {
- unsigned int extra_lines;
-
- /* don't count the first line of nl_lines - it's not "extra" */
- extra_lines = ((width - 1) / width_wrap[i]) + nl_lines - 1;
- if (extra_lines > extra_row_output_lines)
- extra_row_output_lines = extra_lines;
- }
-
- /* i is the current column number: increment with wrap */
- if (++i >= col_count)
- {
- i = 0;
- /* At last column of each row, add tallest column height */
- extra_output_lines += extra_row_output_lines;
- extra_row_output_lines = 0;
- }
- }
- IsPagerNeeded(cont, extra_output_lines, false, &fout, &is_pager);
+ IsPagerNeeded(cont, width_wrap, false, &fout, &is_pager);
is_local_pager = is_pager;
}
@@ -1376,7 +1336,7 @@ print_aligned_vertical(const printTableContent *cont,
*/
if (!is_pager)
{
- IsPagerNeeded(cont, 0, true, &fout, &is_pager);
+ IsPagerNeeded(cont, NULL, true, &fout, &is_pager);
is_local_pager = is_pager;
}
@@ -3400,35 +3360,89 @@ printTableCleanup(printTableContent *const content)
* Setup pager if required
*/
static void
-IsPagerNeeded(const printTableContent *cont, int extra_lines, bool expanded,
- FILE **fout, bool *is_pager)
+IsPagerNeeded(const printTableContent *cont, unsigned int *width_wrap,
+ bool expanded, FILE **fout, bool *is_pager)
{
- if (*fout == stdout)
+ int lines,
+ max_lines = 0,
+ nl_lines,
+ i;
+ const char *const *cell = NULL;
+
+ if (*fout != stdout)
{
- int lines;
+ *is_pager = false;
+ return;
+ }
- if (expanded)
- lines = (cont->ncolumns + 1) * cont->nrows;
- else
- lines = cont->nrows + 1;
+ if (expanded)
+ lines = (cont->ncolumns + 1) * cont->nrows;
+ else
+ lines = cont->nrows + 1;
- if (!cont->opt->tuples_only)
- {
- printTableFooter *f;
+ /* Scan all column headers to find maximum height */
+ for (i = 0; i < cont->ncolumns; i++)
+ {
+ pg_wcssize((const unsigned char *) cont->headers[i],
+ strlen(cont->headers[i]), cont->opt->encoding,
+ NULL, &nl_lines, NULL);
- /*
- * FIXME -- this is slightly bogus: it counts the number of
- * footers, not the number of lines in them.
- */
- for (f = cont->footers; f; f = f->next)
- lines++;
+ if (nl_lines > max_lines)
+ max_lines = nl_lines;
+ }
+
+ /* Add height of tallest header column */
+ lines += max_lines;
+ max_lines = 0;
+
+ /* Scan all cells to count extra lines */
+ for (i = 0, cell = cont->cells; *cell; cell++)
+ {
+ int width;
+ unsigned int extra_lines;
+
+ pg_wcssize((const unsigned char *) *cell, strlen(*cell),
+ cont->opt->encoding, &width, &nl_lines, NULL);
+
+ /*
+ * A cell can have both wrapping and newlines that cause it to display
+ * across multiple lines. We check for both cases below.
+ */
+
+ /* Don't count the first line of nl_lines -- it's not "extra" */
+ extra_lines = nl_lines - 1;
+
+ /* Count extra lines due to wrapping */
+ if (width > 0 && width_wrap && width_wrap[i])
+ extra_lines += (width - 1) / width_wrap[i];
+
+ if (extra_lines > max_lines)
+ max_lines = extra_lines;
+
+ /* i is the current column number: increment with wrap */
+ if (++i >= cont->ncolumns)
+ {
+ i = 0;
+ /* At last column of each row, add tallest column height */
+ lines += max_lines;
+ max_lines = 0;
}
+ }
+
+ if (!cont->opt->tuples_only)
+ {
+ printTableFooter *f;
- *fout = PageOutput(lines + extra_lines, cont->opt);
- *is_pager = (*fout != stdout);
+ /*
+ * FIXME -- this is slightly bogus: it counts the number of
+ * footers, not the number of lines in them.
+ */
+ for (f = cont->footers; f; f = f->next)
+ lines++;
}
- else
- *is_pager = false;
+
+ *fout = PageOutput(lines, cont->opt);
+ *is_pager = (*fout != stdout);
}
/*
@@ -3456,7 +3470,7 @@ printTable(const printTableContent *cont,
cont->opt->format != PRINT_ALIGNED &&
cont->opt->format != PRINT_WRAPPED)
{
- IsPagerNeeded(cont, 0, (cont->opt->expanded == 1), &fout, &is_pager);
+ IsPagerNeeded(cont, NULL, (cont->opt->expanded == 1), &fout, &is_pager);
is_local_pager = is_pager;
}
--
2.51.0
v3-0002-psql-Fix-counting-of-cell-lines-in-expanded-mode.patchtext/plain; charset=us-asciiDownload
From 0ecf9d642f92e41fa82829d49ac889d37fed1d05 Mon Sep 17 00:00:00 2001
From: Erik Wienhold <ewie@ewie.name>
Date: Wed, 10 Sep 2025 02:58:56 +0200
Subject: [PATCH v3 2/3] psql: Fix counting of cell lines in expanded mode
In expanded mode we repeat the table header in every record by printing
header and cell side by side. But the pager setup only counts the lines
in the cells. The total line count is off if headers have more line
breaks than some of the respective cells. Fix this by distinguishing
between expanded and normal mode in the loop that counts the cell lines.
While at it, change that loop to count all lines of each cell instead of
just the extra lines to make the logic easier to follow. The first line
of each cell was accounted for in the initial value of the total line
count which now only accounts for the separator lines, depending on the
output mode.
---
src/fe_utils/print.c | 60 ++++++++++++++++++++++++++++----------------
1 file changed, 39 insertions(+), 21 deletions(-)
diff --git a/src/fe_utils/print.c b/src/fe_utils/print.c
index 44d5af7577e..82f4fc9af20 100644
--- a/src/fe_utils/print.c
+++ b/src/fe_utils/print.c
@@ -3366,6 +3366,7 @@ IsPagerNeeded(const printTableContent *cont, unsigned int *width_wrap,
int lines,
max_lines = 0,
nl_lines,
+ *header_height = NULL,
i;
const char *const *cell = NULL;
@@ -3375,18 +3376,25 @@ IsPagerNeeded(const printTableContent *cont, unsigned int *width_wrap,
return;
}
- if (expanded)
- lines = (cont->ncolumns + 1) * cont->nrows;
- else
- lines = cont->nrows + 1;
+ /*
+ * Count one line per record separator in expanded mode. In normal mode,
+ * we write a single separator line between header and rows.
+ */
+ lines = expanded ? cont->nrows : 1;
- /* Scan all column headers to find maximum height */
+ /*
+ * Scan all column headers and cache their line count since expanded
+ * output repeats the header for every record.
+ */
+ header_height = pg_malloc0(cont->ncolumns * sizeof(header_height));
for (i = 0; i < cont->ncolumns; i++)
{
pg_wcssize((const unsigned char *) cont->headers[i],
strlen(cont->headers[i]), cont->opt->encoding,
NULL, &nl_lines, NULL);
+ header_height[i] = nl_lines;
+
if (nl_lines > max_lines)
max_lines = nl_lines;
}
@@ -3395,37 +3403,45 @@ IsPagerNeeded(const printTableContent *cont, unsigned int *width_wrap,
lines += max_lines;
max_lines = 0;
- /* Scan all cells to count extra lines */
+ /* Scan all cells to count their lines */
for (i = 0, cell = cont->cells; *cell; cell++)
{
int width;
- unsigned int extra_lines;
+ /* Count the original line breaks */
pg_wcssize((const unsigned char *) *cell, strlen(*cell),
cont->opt->encoding, &width, &nl_lines, NULL);
- /*
- * A cell can have both wrapping and newlines that cause it to display
- * across multiple lines. We check for both cases below.
- */
-
- /* Don't count the first line of nl_lines -- it's not "extra" */
- extra_lines = nl_lines - 1;
-
/* Count extra lines due to wrapping */
if (width > 0 && width_wrap && width_wrap[i])
- extra_lines += (width - 1) / width_wrap[i];
+ nl_lines += (width - 1) / width_wrap[i];
- if (extra_lines > max_lines)
- max_lines = extra_lines;
+ if (expanded)
+ {
+ /* Pick the height of the header or cell, whichever is taller */
+ if (nl_lines > header_height[i])
+ lines += nl_lines;
+ else
+ lines += header_height[i];
+ }
+ else
+ {
+ /* Remember max line count in the current row */
+ if (nl_lines > max_lines)
+ max_lines = nl_lines;
+ }
/* i is the current column number: increment with wrap */
if (++i >= cont->ncolumns)
{
i = 0;
- /* At last column of each row, add tallest column height */
- lines += max_lines;
- max_lines = 0;
+
+ if (!expanded)
+ {
+ /* At last column of each row, add tallest column height */
+ lines += max_lines;
+ max_lines = 0;
+ }
}
}
@@ -3443,6 +3459,8 @@ IsPagerNeeded(const printTableContent *cont, unsigned int *width_wrap,
*fout = PageOutput(lines, cont->opt);
*is_pager = (*fout != stdout);
+
+ free(header_height);
}
/*
--
2.51.0
v3-0003-psql-Fix-counting-of-header-and-footer-lines-in-p.patchtext/plain; charset=us-asciiDownload
From e12ed8cad1c672488a64a44183c8a7b3154a57ca Mon Sep 17 00:00:00 2001
From: Erik Wienhold <ewie@ewie.name>
Date: Tue, 9 Sep 2025 02:36:49 +0200
Subject: [PATCH v3 3/3] psql: Fix counting of header and footer lines in pager
setup
* The table header only produces extra lines when printed in normal
mode. So don't count those lines in tuples_only mode or expanded
mode.
* Count the lines of the table title, if present.
* Count all footer lines instead of treating each footer as a single
line.
---
src/fe_utils/print.c | 39 +++++++++++++++++++++++++++------------
1 file changed, 27 insertions(+), 12 deletions(-)
diff --git a/src/fe_utils/print.c b/src/fe_utils/print.c
index 82f4fc9af20..94043fae663 100644
--- a/src/fe_utils/print.c
+++ b/src/fe_utils/print.c
@@ -3394,15 +3394,8 @@ IsPagerNeeded(const printTableContent *cont, unsigned int *width_wrap,
NULL, &nl_lines, NULL);
header_height[i] = nl_lines;
-
- if (nl_lines > max_lines)
- max_lines = nl_lines;
}
- /* Add height of tallest header column */
- lines += max_lines;
- max_lines = 0;
-
/* Scan all cells to count their lines */
for (i = 0, cell = cont->cells; *cell; cell++)
{
@@ -3449,12 +3442,34 @@ IsPagerNeeded(const printTableContent *cont, unsigned int *width_wrap,
{
printTableFooter *f;
- /*
- * FIXME -- this is slightly bogus: it counts the number of
- * footers, not the number of lines in them.
- */
+ if (cont->title)
+ {
+ pg_wcssize((const unsigned char *) cont->title, strlen(cont->title),
+ cont->opt->encoding, NULL, &nl_lines, NULL);
+ lines += nl_lines;
+ }
+
+ if (!expanded)
+ {
+ /* Find the tallest header column */
+ for (i = 0; i < cont->ncolumns; i++)
+ {
+ if (header_height[i] > max_lines)
+ max_lines = header_height[i];
+ }
+
+ /* Add height of tallest header column */
+ lines += max_lines;
+ max_lines = 0;
+ }
+
+ /* Count all footer lines */
for (f = cont->footers; f; f = f->next)
- lines++;
+ {
+ pg_wcssize((const unsigned char *) f->data, strlen(f->data),
+ cont->opt->encoding, NULL, &nl_lines, NULL);
+ lines += nl_lines;
+ }
}
*fout = PageOutput(lines, cont->opt);
--
2.51.0
Erik Wienhold <ewie@ewie.name> writes:
Here's v3 to address all of this. I split it into three separate
patches:
Thanks! While reviewing this I decided that splitting it wasn't
such a great idea, because I kept getting distracted by obvious
bugs in the code you were copying around, only to realize that
the next patches fixed those bugs. So in the attached v4 I
merged these into one patch. It's mostly the same as yours
(I did find one outright bug, in a typo'd pg_malloc call),
but I split the line-counting logic into a new subroutine of
its own, which allows IsPagerNeeded to retain pretty much its
previous shape. There's some cosmetic changes too, mostly
expanding comments.
While I was looking at this I got annoyed at how many times we call
pg_wcssize. That's not tremendously cheap, and we're actually doing
it twice for every table cell, which is going to add up to a lot for
large tables. (We've had complaints before about the speed of psql
on big query results...) I'm not sure there is any great way to
avoid that altogether, but I did realize that we could skip the vast
majority of that work in the line-counting path if we recognize that
we can stop as soon as we know the table is longer than screen height.
So 0002 attached is a cut at doing that (which now shows why I wanted
the counting to be in its own subroutine).
I am not entirely sure that we should commit 0002 though; it may be
that the savings is down in the noise anyway once you consider all the
other work that happens while printing a big table. A positive reason
not to take it is something I realized while checking test coverage:
we never execute any of the maybe-use-the-pager branch of PageOutput
in the regression tests, because isatty(stdout) will always fail.
So that lack of coverage would also apply to count_table_lines in this
formulation, which is kind of sad. However, I'm not sure how useful
it really is to have code coverage there, since by this very token
the tests are paying zero attention to the value computed by
count_table_lines. It could be arbitrarily silly and we'd not see
that.
So, while I'm content with v4-0001, I'm not quite convinced about
v4-0002. WDYT?
regards, tom lane
Attachments:
v4-0001-Improve-psql-s-ability-to-select-pager-mode-accur.patchtext/x-diff; charset=us-ascii; name*0=v4-0001-Improve-psql-s-ability-to-select-pager-mode-accur.p; name*1=atchDownload
From 0b6780211acafa3504b47692f87b24f891dbefe0 Mon Sep 17 00:00:00 2001
From: Tom Lane <tgl@sss.pgh.pa.us>
Date: Wed, 1 Oct 2025 17:25:00 -0400
Subject: [PATCH v4 1/2] Improve psql's ability to select pager mode
accurately.
The code in print.c that's concerned with counting the number of lines
that will be needed missed a lot of edge cases:
* While plain aligned mode accounted for embedded newlines in column
headers and table cells, unaligned and expanded output modes did not.
* In particular, since expanded mode repeats the headers for each
record, we need to account for embedded newlines in the headers for
each record.
* Multi-line table titles were not accounted for.
* tuples_only mode (where headers aren't printed) wasn't accounted
for.
* Footers were accounted for as one line per footer, again missing
the possibility of multi-line footers. (In some cases such as
"\d+" on a view, there can be many lines in a footer.)
To fix, move the entire responsibility for counting lines into
IsPagerNeeded (or actually, into a new subroutine count_table_lines),
and then expand the logic as appropriate. Also restructure to make
it perhaps a bit easier to follow.
Author: Erik Wienhold <ewie@ewie.name>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/2dd2430f-dd20-4c89-97fd-242616a3d768@ewie.name
---
src/fe_utils/print.c | 199 ++++++++++++++++++++++++++++++-------------
1 file changed, 138 insertions(+), 61 deletions(-)
diff --git a/src/fe_utils/print.c b/src/fe_utils/print.c
index 4af0f32f2fc..71bc14d499b 100644
--- a/src/fe_utils/print.c
+++ b/src/fe_utils/print.c
@@ -266,8 +266,13 @@ static const unicodeStyleFormat unicode_style = {
/* Local functions */
static int strlen_max_width(unsigned char *str, int *target_width, int encoding);
-static void IsPagerNeeded(const printTableContent *cont, int extra_lines, bool expanded,
+static void IsPagerNeeded(const printTableContent *cont,
+ const unsigned int *width_wrap,
+ bool expanded,
FILE **fout, bool *is_pager);
+static int count_table_lines(const printTableContent *cont,
+ const unsigned int *width_wrap,
+ bool expanded);
static void print_aligned_vertical(const printTableContent *cont,
FILE *fout, bool is_pager);
@@ -656,8 +661,6 @@ print_aligned_text(const printTableContent *cont, FILE *fout, bool is_pager)
unsigned char **format_buf;
unsigned int width_total;
unsigned int total_header_width;
- unsigned int extra_row_output_lines = 0;
- unsigned int extra_output_lines = 0;
const char *const *ptr;
@@ -722,14 +725,9 @@ print_aligned_text(const printTableContent *cont, FILE *fout, bool is_pager)
max_nl_lines[i] = nl_lines;
if (bytes_required > max_bytes[i])
max_bytes[i] = bytes_required;
- if (nl_lines > extra_row_output_lines)
- extra_row_output_lines = nl_lines;
width_header[i] = width;
}
- /* Add height of tallest header column */
- extra_output_lines += extra_row_output_lines;
- extra_row_output_lines = 0;
/* scan all cells, find maximum width, compute cell_count */
for (i = 0, ptr = cont->cells; *ptr; ptr++, i++, cell_count++)
@@ -889,43 +887,10 @@ print_aligned_text(const printTableContent *cont, FILE *fout, bool is_pager)
is_pager = is_local_pager = true;
}
- /* Check if newlines or our wrapping now need the pager */
- if (!is_pager && fout == stdout)
+ /* Check if there are enough lines to require the pager */
+ if (!is_pager)
{
- /* scan all cells, find maximum width, compute cell_count */
- for (i = 0, ptr = cont->cells; *ptr; ptr++, cell_count++)
- {
- int width,
- nl_lines,
- bytes_required;
-
- pg_wcssize((const unsigned char *) *ptr, strlen(*ptr), encoding,
- &width, &nl_lines, &bytes_required);
-
- /*
- * A row can have both wrapping and newlines that cause it to
- * display across multiple lines. We check for both cases below.
- */
- if (width > 0 && width_wrap[i])
- {
- unsigned int extra_lines;
-
- /* don't count the first line of nl_lines - it's not "extra" */
- extra_lines = ((width - 1) / width_wrap[i]) + nl_lines - 1;
- if (extra_lines > extra_row_output_lines)
- extra_row_output_lines = extra_lines;
- }
-
- /* i is the current column number: increment with wrap */
- if (++i >= col_count)
- {
- i = 0;
- /* At last column of each row, add tallest column height */
- extra_output_lines += extra_row_output_lines;
- extra_row_output_lines = 0;
- }
- }
- IsPagerNeeded(cont, extra_output_lines, false, &fout, &is_pager);
+ IsPagerNeeded(cont, width_wrap, false, &fout, &is_pager);
is_local_pager = is_pager;
}
@@ -1376,7 +1341,7 @@ print_aligned_vertical(const printTableContent *cont,
*/
if (!is_pager)
{
- IsPagerNeeded(cont, 0, true, &fout, &is_pager);
+ IsPagerNeeded(cont, NULL, true, &fout, &is_pager);
is_local_pager = is_pager;
}
@@ -3398,37 +3363,149 @@ printTableCleanup(printTableContent *const content)
* IsPagerNeeded
*
* Setup pager if required
+ *
+ * cont: table data to be printed
+ * width_wrap[]: per-column maximum width, or NULL if caller will not wrap
+ * expanded: expanded mode?
+ * fout: where to print to (in/out argument)
+ * is_pager: output argument
+ *
+ * If we decide pager is needed, *fout is modified and *is_pager is set true
*/
static void
-IsPagerNeeded(const printTableContent *cont, int extra_lines, bool expanded,
+IsPagerNeeded(const printTableContent *cont, const unsigned int *width_wrap,
+ bool expanded,
FILE **fout, bool *is_pager)
{
if (*fout == stdout)
{
int lines;
+ lines = count_table_lines(cont, width_wrap, expanded);
+
+ *fout = PageOutput(lines, cont->opt);
+ *is_pager = (*fout != stdout);
+ }
+ else
+ *is_pager = false;
+}
+
+/*
+ * Count the number of lines needed to print the given table.
+ *
+ * cont: table data to be printed
+ * width_wrap[]: per-column maximum width, or NULL if caller will not wrap
+ * expanded: expanded mode?
+ *
+ * We currently handle only regular and expanded modes here.
+ */
+static int
+count_table_lines(const printTableContent *cont,
+ const unsigned int *width_wrap,
+ bool expanded)
+{
+ int *header_height;
+ int lines,
+ max_lines = 0,
+ nl_lines,
+ i;
+ int encoding = cont->opt->encoding;
+ const char *const *cell;
+
+ /*
+ * Scan all column headers and determine their heights. Cache the values
+ * since expanded mode repeats the headers for every record.
+ */
+ header_height = (int *) pg_malloc(cont->ncolumns * sizeof(int));
+ for (i = 0; i < cont->ncolumns; i++)
+ {
+ pg_wcssize((const unsigned char *) cont->headers[i],
+ strlen(cont->headers[i]), encoding,
+ NULL, &header_height[i], NULL);
+ }
+
+ /*
+ * Expanded mode writes one separator line per record. Normal mode writes
+ * a single separator line between header and rows.
+ */
+ lines = expanded ? cont->nrows : 1;
+
+ /* Scan all cells to count their lines */
+ for (i = 0, cell = cont->cells; *cell; cell++)
+ {
+ int width;
+
+ /* Count the original line breaks */
+ pg_wcssize((const unsigned char *) *cell, strlen(*cell), encoding,
+ &width, &nl_lines, NULL);
+
+ /* Count extra lines due to wrapping */
+ if (width > 0 && width_wrap && width_wrap[i])
+ nl_lines += (width - 1) / width_wrap[i];
+
if (expanded)
- lines = (cont->ncolumns + 1) * cont->nrows;
+ {
+ /* Pick the height of the header or cell, whichever is taller */
+ if (nl_lines > header_height[i])
+ lines += nl_lines;
+ else
+ lines += header_height[i];
+ }
else
- lines = cont->nrows + 1;
+ {
+ /* Remember max height in the current row */
+ if (nl_lines > max_lines)
+ max_lines = nl_lines;
+ }
- if (!cont->opt->tuples_only)
+ /* i is the current column number: increment with wrap */
+ if (++i >= cont->ncolumns)
{
- printTableFooter *f;
+ i = 0;
+ if (!expanded)
+ {
+ /* At last column of each row, add tallest column height */
+ lines += max_lines;
+ max_lines = 0;
+ }
+ }
+ }
- /*
- * FIXME -- this is slightly bogus: it counts the number of
- * footers, not the number of lines in them.
- */
- for (f = cont->footers; f; f = f->next)
- lines++;
+ /* Account for header and footer decoration */
+ if (!cont->opt->tuples_only)
+ {
+ if (cont->title)
+ {
+ /* Add height of title */
+ pg_wcssize((const unsigned char *) cont->title, strlen(cont->title),
+ encoding, NULL, &nl_lines, NULL);
+ lines += nl_lines;
}
- *fout = PageOutput(lines + extra_lines, cont->opt);
- *is_pager = (*fout != stdout);
+ if (!expanded)
+ {
+ /* Add height of tallest header column */
+ max_lines = 0;
+ for (i = 0; i < cont->ncolumns; i++)
+ {
+ if (header_height[i] > max_lines)
+ max_lines = header_height[i];
+ }
+ lines += max_lines;
+ }
+
+ /* Add all footer lines */
+ for (printTableFooter *f = cont->footers; f; f = f->next)
+ {
+ pg_wcssize((const unsigned char *) f->data, strlen(f->data),
+ encoding, NULL, &nl_lines, NULL);
+ lines += nl_lines;
+ }
}
- else
- *is_pager = false;
+
+ free(header_height);
+
+ return lines;
}
/*
@@ -3456,7 +3533,7 @@ printTable(const printTableContent *cont,
cont->opt->format != PRINT_ALIGNED &&
cont->opt->format != PRINT_WRAPPED)
{
- IsPagerNeeded(cont, 0, (cont->opt->expanded == 1), &fout, &is_pager);
+ IsPagerNeeded(cont, NULL, (cont->opt->expanded == 1), &fout, &is_pager);
is_local_pager = is_pager;
}
--
2.43.7
v4-0002-Make-some-minor-performance-improvements-in-psql-.patchtext/x-diff; charset=us-ascii; name*0=v4-0002-Make-some-minor-performance-improvements-in-psql-.p; name*1=atchDownload
From d4fe1827f338e38ed5dbc2042ab199fb09954fbd Mon Sep 17 00:00:00 2001
From: Tom Lane <tgl@sss.pgh.pa.us>
Date: Wed, 1 Oct 2025 18:04:02 -0400
Subject: [PATCH v4 2/2] Make some minor performance improvements in psql's
line-counting.
Arrange to not run count_table_lines at all unless we will use
its result, and teach it to quit early as soon as it's proven
that the output is long enough to require use of the pager.
When dealing with large tables this should save a noticeable
amount of time, since pg_wcssize() isn't exactly cheap.
In passing, fix print_aligned_text to avoid using repeated
"i % col_count" when it could just make the value of i wrap
around. In a quick check with the version of gcc I'm using,
the compiler is smart enough to recognize the common subexpressions
and issue only one integer divide per loop, but it's not smart enough
to avoid the integer divide altogether. In any case, this coding
was randomly unlike the way it's done in count_table_lines.
Author: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/2dd2430f-dd20-4c89-97fd-242616a3d768@ewie.name
---
src/fe_utils/print.c | 97 +++++++++++++++++++++++++++++++++-----------
1 file changed, 73 insertions(+), 24 deletions(-)
diff --git a/src/fe_utils/print.c b/src/fe_utils/print.c
index 71bc14d499b..d8c8d7d2a3b 100644
--- a/src/fe_utils/print.c
+++ b/src/fe_utils/print.c
@@ -266,13 +266,18 @@ static const unicodeStyleFormat unicode_style = {
/* Local functions */
static int strlen_max_width(unsigned char *str, int *target_width, int encoding);
+static FILE *PageOutputInternal(int lines, const printTableOpt *topt,
+ const printTableContent *cont,
+ const unsigned int *width_wrap,
+ bool expanded);
static void IsPagerNeeded(const printTableContent *cont,
const unsigned int *width_wrap,
bool expanded,
FILE **fout, bool *is_pager);
static int count_table_lines(const printTableContent *cont,
const unsigned int *width_wrap,
- bool expanded);
+ bool expanded,
+ int threshold);
static void print_aligned_vertical(const printTableContent *cont,
FILE *fout, bool is_pager);
@@ -730,7 +735,7 @@ print_aligned_text(const printTableContent *cont, FILE *fout, bool is_pager)
}
/* scan all cells, find maximum width, compute cell_count */
- for (i = 0, ptr = cont->cells; *ptr; ptr++, i++, cell_count++)
+ for (i = 0, ptr = cont->cells; *ptr; ptr++, cell_count++)
{
int width,
nl_lines,
@@ -739,14 +744,18 @@ print_aligned_text(const printTableContent *cont, FILE *fout, bool is_pager)
pg_wcssize((const unsigned char *) *ptr, strlen(*ptr), encoding,
&width, &nl_lines, &bytes_required);
- if (width > max_width[i % col_count])
- max_width[i % col_count] = width;
- if (nl_lines > max_nl_lines[i % col_count])
- max_nl_lines[i % col_count] = nl_lines;
- if (bytes_required > max_bytes[i % col_count])
- max_bytes[i % col_count] = bytes_required;
+ if (width > max_width[i])
+ max_width[i] = width;
+ if (nl_lines > max_nl_lines[i])
+ max_nl_lines[i] = nl_lines;
+ if (bytes_required > max_bytes[i])
+ max_bytes[i] = bytes_required;
+
+ width_average[i] += width;
- width_average[i % col_count] += width;
+ /* i is the current column number: increment with wrap */
+ if (++i >= col_count)
+ i = 0;
}
/* If we have rows, compute average */
@@ -3046,12 +3055,31 @@ set_sigpipe_trap_state(bool ignore)
/*
* PageOutput
*
- * Tests if pager is needed and returns appropriate FILE pointer.
+ * Tests if pager is needed and returns appropriate FILE pointer
+ * (either a pipe, or stdout if we don't need the pager).
+ *
+ * lines: number of lines that will be printed
+ * topt: print formatting options
*
* If the topt argument is NULL no pager is used.
*/
FILE *
PageOutput(int lines, const printTableOpt *topt)
+{
+ return PageOutputInternal(lines, topt, NULL, NULL, false);
+}
+
+/*
+ * Private version that allows for line-counting to be avoided when
+ * not needed. If "cont" is not null then the input value of "lines"
+ * is ignored and we count lines based on cont + width_wrap + expanded
+ * (see count_table_lines).
+ */
+static FILE *
+PageOutputInternal(int lines, const printTableOpt *topt,
+ const printTableContent *cont,
+ const unsigned int *width_wrap,
+ bool expanded)
{
/* check whether we need / can / are supposed to use pager */
if (topt && topt->pager && isatty(fileno(stdin)) && isatty(fileno(stdout)))
@@ -3059,15 +3087,29 @@ PageOutput(int lines, const printTableOpt *topt)
#ifdef TIOCGWINSZ
unsigned short int pager = topt->pager;
int min_lines = topt->pager_min_lines;
- int result;
- struct winsize screen_size;
- result = ioctl(fileno(stdout), TIOCGWINSZ, &screen_size);
+ if (pager == 1)
+ {
+ int result;
+ struct winsize screen_size;
- /* >= accounts for a one-line prompt */
- if (result == -1
- || (lines >= screen_size.ws_row && lines >= min_lines)
- || pager > 1)
+ result = ioctl(fileno(stdout), TIOCGWINSZ, &screen_size);
+ if (result < 0)
+ pager = 2; /* force use of pager */
+ else
+ {
+ int threshold = Max(screen_size.ws_row, min_lines);
+
+ if (cont) /* caller wants us to calculate lines */
+ lines = count_table_lines(cont, width_wrap, expanded,
+ threshold);
+ /* >= accounts for a one-line prompt */
+ if (lines >= threshold)
+ pager = 2;
+ }
+ }
+
+ if (pager > 1)
#endif
{
const char *pagerprog;
@@ -3379,11 +3421,7 @@ IsPagerNeeded(const printTableContent *cont, const unsigned int *width_wrap,
{
if (*fout == stdout)
{
- int lines;
-
- lines = count_table_lines(cont, width_wrap, expanded);
-
- *fout = PageOutput(lines, cont->opt);
+ *fout = PageOutputInternal(0, cont->opt, cont, width_wrap, expanded);
*is_pager = (*fout != stdout);
}
else
@@ -3396,13 +3434,18 @@ IsPagerNeeded(const printTableContent *cont, const unsigned int *width_wrap,
* cont: table data to be printed
* width_wrap[]: per-column maximum width, or NULL if caller will not wrap
* expanded: expanded mode?
+ * threshold: we can stop counting once we pass threshold
*
* We currently handle only regular and expanded modes here.
+ *
+ * The value of the threshold parameter is that when the table is very
+ * long, we'll typically be able to stop scanning after not many rows.
*/
static int
count_table_lines(const printTableContent *cont,
const unsigned int *width_wrap,
- bool expanded)
+ bool expanded,
+ int threshold)
{
int *header_height;
int lines,
@@ -3468,11 +3511,14 @@ count_table_lines(const printTableContent *cont,
lines += max_lines;
max_lines = 0;
}
+ /* Stop scanning table body once we pass threshold */
+ if (lines > threshold)
+ break;
}
}
/* Account for header and footer decoration */
- if (!cont->opt->tuples_only)
+ if (!cont->opt->tuples_only && lines <= threshold)
{
if (cont->title)
{
@@ -3500,6 +3546,9 @@ count_table_lines(const printTableContent *cont,
pg_wcssize((const unsigned char *) f->data, strlen(f->data),
encoding, NULL, &nl_lines, NULL);
lines += nl_lines;
+ /* Stop scanning footers once we pass threshold */
+ if (lines > threshold)
+ break;
}
}
--
2.43.7
On 2025-10-02 00:25 +0200, Tom Lane wrote:
Erik Wienhold <ewie@ewie.name> writes:
Here's v3 to address all of this. I split it into three separate
patches:Thanks! While reviewing this I decided that splitting it wasn't
such a great idea, because I kept getting distracted by obvious
bugs in the code you were copying around, only to realize that
the next patches fixed those bugs. So in the attached v4 I
merged these into one patch. It's mostly the same as yours
(I did find one outright bug, in a typo'd pg_malloc call),
but I split the line-counting logic into a new subroutine of
its own, which allows IsPagerNeeded to retain pretty much its
previous shape. There's some cosmetic changes too, mostly
expanding comments.
+1 to factoring out the line counting.
While I was looking at this I got annoyed at how many times we call
pg_wcssize. That's not tremendously cheap, and we're actually doing
it twice for every table cell, which is going to add up to a lot for
large tables. (We've had complaints before about the speed of psql
on big query results...) I'm not sure there is any great way to
avoid that altogether, but I did realize that we could skip the vast
majority of that work in the line-counting path if we recognize that
we can stop as soon as we know the table is longer than screen height.
So 0002 attached is a cut at doing that (which now shows why I wanted
the counting to be in its own subroutine).
Yeah, I've noticed those repeated pg_wcssize calls as well but thought
that their overhead might me acceptable given how old the code is.
Limiting that overhead with the threshold parameter is a nifty idea.
I am not entirely sure that we should commit 0002 though; it may be
that the savings is down in the noise anyway once you consider all the
other work that happens while printing a big table. A positive reason
not to take it is something I realized while checking test coverage:
we never execute any of the maybe-use-the-pager branch of PageOutput
in the regression tests, because isatty(stdout) will always fail.
So that lack of coverage would also apply to count_table_lines in this
formulation, which is kind of sad. However, I'm not sure how useful
it really is to have code coverage there, since by this very token
the tests are paying zero attention to the value computed by
count_table_lines. It could be arbitrarily silly and we'd not see
that.So, while I'm content with v4-0001, I'm not quite convinced about
v4-0002. WDYT?
I only glanced over the patches. Will take a closer look over the
weekend.
--
Erik Wienhold
Erik Wienhold <ewie@ewie.name> writes:
On 2025-10-02 00:25 +0200, Tom Lane wrote:
I am not entirely sure that we should commit 0002 though; it may be
that the savings is down in the noise anyway once you consider all the
other work that happens while printing a big table. A positive reason
not to take it is something I realized while checking test coverage:
we never execute any of the maybe-use-the-pager branch of PageOutput
in the regression tests, because isatty(stdout) will always fail.
I realized that we could address that if we really wanted to,
using the same infrastructure as for tab-completion testing.
0001 and 0002 attached are the same as before, and then I added
0003 which adds a draft-quality TAP test. Code coverage checks
show that this adds only about 10 more lines of coverage in
psql proper, but in print.c most of the pager-related logic is
now covered.
regards, tom lane
Attachments:
v4-0001-Improve-psql-s-ability-to-select-pager-mode-accur.patchtext/x-diff; charset=us-ascii; name*0=v4-0001-Improve-psql-s-ability-to-select-pager-mode-accur.p; name*1=atchDownload
From 0b6780211acafa3504b47692f87b24f891dbefe0 Mon Sep 17 00:00:00 2001
From: Tom Lane <tgl@sss.pgh.pa.us>
Date: Wed, 1 Oct 2025 17:25:00 -0400
Subject: [PATCH v4 1/2] Improve psql's ability to select pager mode
accurately.
The code in print.c that's concerned with counting the number of lines
that will be needed missed a lot of edge cases:
* While plain aligned mode accounted for embedded newlines in column
headers and table cells, unaligned and expanded output modes did not.
* In particular, since expanded mode repeats the headers for each
record, we need to account for embedded newlines in the headers for
each record.
* Multi-line table titles were not accounted for.
* tuples_only mode (where headers aren't printed) wasn't accounted
for.
* Footers were accounted for as one line per footer, again missing
the possibility of multi-line footers. (In some cases such as
"\d+" on a view, there can be many lines in a footer.)
To fix, move the entire responsibility for counting lines into
IsPagerNeeded (or actually, into a new subroutine count_table_lines),
and then expand the logic as appropriate. Also restructure to make
it perhaps a bit easier to follow.
Author: Erik Wienhold <ewie@ewie.name>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/2dd2430f-dd20-4c89-97fd-242616a3d768@ewie.name
---
src/fe_utils/print.c | 199 ++++++++++++++++++++++++++++++-------------
1 file changed, 138 insertions(+), 61 deletions(-)
diff --git a/src/fe_utils/print.c b/src/fe_utils/print.c
index 4af0f32f2fc..71bc14d499b 100644
--- a/src/fe_utils/print.c
+++ b/src/fe_utils/print.c
@@ -266,8 +266,13 @@ static const unicodeStyleFormat unicode_style = {
/* Local functions */
static int strlen_max_width(unsigned char *str, int *target_width, int encoding);
-static void IsPagerNeeded(const printTableContent *cont, int extra_lines, bool expanded,
+static void IsPagerNeeded(const printTableContent *cont,
+ const unsigned int *width_wrap,
+ bool expanded,
FILE **fout, bool *is_pager);
+static int count_table_lines(const printTableContent *cont,
+ const unsigned int *width_wrap,
+ bool expanded);
static void print_aligned_vertical(const printTableContent *cont,
FILE *fout, bool is_pager);
@@ -656,8 +661,6 @@ print_aligned_text(const printTableContent *cont, FILE *fout, bool is_pager)
unsigned char **format_buf;
unsigned int width_total;
unsigned int total_header_width;
- unsigned int extra_row_output_lines = 0;
- unsigned int extra_output_lines = 0;
const char *const *ptr;
@@ -722,14 +725,9 @@ print_aligned_text(const printTableContent *cont, FILE *fout, bool is_pager)
max_nl_lines[i] = nl_lines;
if (bytes_required > max_bytes[i])
max_bytes[i] = bytes_required;
- if (nl_lines > extra_row_output_lines)
- extra_row_output_lines = nl_lines;
width_header[i] = width;
}
- /* Add height of tallest header column */
- extra_output_lines += extra_row_output_lines;
- extra_row_output_lines = 0;
/* scan all cells, find maximum width, compute cell_count */
for (i = 0, ptr = cont->cells; *ptr; ptr++, i++, cell_count++)
@@ -889,43 +887,10 @@ print_aligned_text(const printTableContent *cont, FILE *fout, bool is_pager)
is_pager = is_local_pager = true;
}
- /* Check if newlines or our wrapping now need the pager */
- if (!is_pager && fout == stdout)
+ /* Check if there are enough lines to require the pager */
+ if (!is_pager)
{
- /* scan all cells, find maximum width, compute cell_count */
- for (i = 0, ptr = cont->cells; *ptr; ptr++, cell_count++)
- {
- int width,
- nl_lines,
- bytes_required;
-
- pg_wcssize((const unsigned char *) *ptr, strlen(*ptr), encoding,
- &width, &nl_lines, &bytes_required);
-
- /*
- * A row can have both wrapping and newlines that cause it to
- * display across multiple lines. We check for both cases below.
- */
- if (width > 0 && width_wrap[i])
- {
- unsigned int extra_lines;
-
- /* don't count the first line of nl_lines - it's not "extra" */
- extra_lines = ((width - 1) / width_wrap[i]) + nl_lines - 1;
- if (extra_lines > extra_row_output_lines)
- extra_row_output_lines = extra_lines;
- }
-
- /* i is the current column number: increment with wrap */
- if (++i >= col_count)
- {
- i = 0;
- /* At last column of each row, add tallest column height */
- extra_output_lines += extra_row_output_lines;
- extra_row_output_lines = 0;
- }
- }
- IsPagerNeeded(cont, extra_output_lines, false, &fout, &is_pager);
+ IsPagerNeeded(cont, width_wrap, false, &fout, &is_pager);
is_local_pager = is_pager;
}
@@ -1376,7 +1341,7 @@ print_aligned_vertical(const printTableContent *cont,
*/
if (!is_pager)
{
- IsPagerNeeded(cont, 0, true, &fout, &is_pager);
+ IsPagerNeeded(cont, NULL, true, &fout, &is_pager);
is_local_pager = is_pager;
}
@@ -3398,37 +3363,149 @@ printTableCleanup(printTableContent *const content)
* IsPagerNeeded
*
* Setup pager if required
+ *
+ * cont: table data to be printed
+ * width_wrap[]: per-column maximum width, or NULL if caller will not wrap
+ * expanded: expanded mode?
+ * fout: where to print to (in/out argument)
+ * is_pager: output argument
+ *
+ * If we decide pager is needed, *fout is modified and *is_pager is set true
*/
static void
-IsPagerNeeded(const printTableContent *cont, int extra_lines, bool expanded,
+IsPagerNeeded(const printTableContent *cont, const unsigned int *width_wrap,
+ bool expanded,
FILE **fout, bool *is_pager)
{
if (*fout == stdout)
{
int lines;
+ lines = count_table_lines(cont, width_wrap, expanded);
+
+ *fout = PageOutput(lines, cont->opt);
+ *is_pager = (*fout != stdout);
+ }
+ else
+ *is_pager = false;
+}
+
+/*
+ * Count the number of lines needed to print the given table.
+ *
+ * cont: table data to be printed
+ * width_wrap[]: per-column maximum width, or NULL if caller will not wrap
+ * expanded: expanded mode?
+ *
+ * We currently handle only regular and expanded modes here.
+ */
+static int
+count_table_lines(const printTableContent *cont,
+ const unsigned int *width_wrap,
+ bool expanded)
+{
+ int *header_height;
+ int lines,
+ max_lines = 0,
+ nl_lines,
+ i;
+ int encoding = cont->opt->encoding;
+ const char *const *cell;
+
+ /*
+ * Scan all column headers and determine their heights. Cache the values
+ * since expanded mode repeats the headers for every record.
+ */
+ header_height = (int *) pg_malloc(cont->ncolumns * sizeof(int));
+ for (i = 0; i < cont->ncolumns; i++)
+ {
+ pg_wcssize((const unsigned char *) cont->headers[i],
+ strlen(cont->headers[i]), encoding,
+ NULL, &header_height[i], NULL);
+ }
+
+ /*
+ * Expanded mode writes one separator line per record. Normal mode writes
+ * a single separator line between header and rows.
+ */
+ lines = expanded ? cont->nrows : 1;
+
+ /* Scan all cells to count their lines */
+ for (i = 0, cell = cont->cells; *cell; cell++)
+ {
+ int width;
+
+ /* Count the original line breaks */
+ pg_wcssize((const unsigned char *) *cell, strlen(*cell), encoding,
+ &width, &nl_lines, NULL);
+
+ /* Count extra lines due to wrapping */
+ if (width > 0 && width_wrap && width_wrap[i])
+ nl_lines += (width - 1) / width_wrap[i];
+
if (expanded)
- lines = (cont->ncolumns + 1) * cont->nrows;
+ {
+ /* Pick the height of the header or cell, whichever is taller */
+ if (nl_lines > header_height[i])
+ lines += nl_lines;
+ else
+ lines += header_height[i];
+ }
else
- lines = cont->nrows + 1;
+ {
+ /* Remember max height in the current row */
+ if (nl_lines > max_lines)
+ max_lines = nl_lines;
+ }
- if (!cont->opt->tuples_only)
+ /* i is the current column number: increment with wrap */
+ if (++i >= cont->ncolumns)
{
- printTableFooter *f;
+ i = 0;
+ if (!expanded)
+ {
+ /* At last column of each row, add tallest column height */
+ lines += max_lines;
+ max_lines = 0;
+ }
+ }
+ }
- /*
- * FIXME -- this is slightly bogus: it counts the number of
- * footers, not the number of lines in them.
- */
- for (f = cont->footers; f; f = f->next)
- lines++;
+ /* Account for header and footer decoration */
+ if (!cont->opt->tuples_only)
+ {
+ if (cont->title)
+ {
+ /* Add height of title */
+ pg_wcssize((const unsigned char *) cont->title, strlen(cont->title),
+ encoding, NULL, &nl_lines, NULL);
+ lines += nl_lines;
}
- *fout = PageOutput(lines + extra_lines, cont->opt);
- *is_pager = (*fout != stdout);
+ if (!expanded)
+ {
+ /* Add height of tallest header column */
+ max_lines = 0;
+ for (i = 0; i < cont->ncolumns; i++)
+ {
+ if (header_height[i] > max_lines)
+ max_lines = header_height[i];
+ }
+ lines += max_lines;
+ }
+
+ /* Add all footer lines */
+ for (printTableFooter *f = cont->footers; f; f = f->next)
+ {
+ pg_wcssize((const unsigned char *) f->data, strlen(f->data),
+ encoding, NULL, &nl_lines, NULL);
+ lines += nl_lines;
+ }
}
- else
- *is_pager = false;
+
+ free(header_height);
+
+ return lines;
}
/*
@@ -3456,7 +3533,7 @@ printTable(const printTableContent *cont,
cont->opt->format != PRINT_ALIGNED &&
cont->opt->format != PRINT_WRAPPED)
{
- IsPagerNeeded(cont, 0, (cont->opt->expanded == 1), &fout, &is_pager);
+ IsPagerNeeded(cont, NULL, (cont->opt->expanded == 1), &fout, &is_pager);
is_local_pager = is_pager;
}
--
2.43.7
v4-0002-Make-some-minor-performance-improvements-in-psql-.patchtext/x-diff; charset=us-ascii; name*0=v4-0002-Make-some-minor-performance-improvements-in-psql-.p; name*1=atchDownload
From d4fe1827f338e38ed5dbc2042ab199fb09954fbd Mon Sep 17 00:00:00 2001
From: Tom Lane <tgl@sss.pgh.pa.us>
Date: Wed, 1 Oct 2025 18:04:02 -0400
Subject: [PATCH v4 2/2] Make some minor performance improvements in psql's
line-counting.
Arrange to not run count_table_lines at all unless we will use
its result, and teach it to quit early as soon as it's proven
that the output is long enough to require use of the pager.
When dealing with large tables this should save a noticeable
amount of time, since pg_wcssize() isn't exactly cheap.
In passing, fix print_aligned_text to avoid using repeated
"i % col_count" when it could just make the value of i wrap
around. In a quick check with the version of gcc I'm using,
the compiler is smart enough to recognize the common subexpressions
and issue only one integer divide per loop, but it's not smart enough
to avoid the integer divide altogether. In any case, this coding
was randomly unlike the way it's done in count_table_lines.
Author: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/2dd2430f-dd20-4c89-97fd-242616a3d768@ewie.name
---
src/fe_utils/print.c | 97 +++++++++++++++++++++++++++++++++-----------
1 file changed, 73 insertions(+), 24 deletions(-)
diff --git a/src/fe_utils/print.c b/src/fe_utils/print.c
index 71bc14d499b..d8c8d7d2a3b 100644
--- a/src/fe_utils/print.c
+++ b/src/fe_utils/print.c
@@ -266,13 +266,18 @@ static const unicodeStyleFormat unicode_style = {
/* Local functions */
static int strlen_max_width(unsigned char *str, int *target_width, int encoding);
+static FILE *PageOutputInternal(int lines, const printTableOpt *topt,
+ const printTableContent *cont,
+ const unsigned int *width_wrap,
+ bool expanded);
static void IsPagerNeeded(const printTableContent *cont,
const unsigned int *width_wrap,
bool expanded,
FILE **fout, bool *is_pager);
static int count_table_lines(const printTableContent *cont,
const unsigned int *width_wrap,
- bool expanded);
+ bool expanded,
+ int threshold);
static void print_aligned_vertical(const printTableContent *cont,
FILE *fout, bool is_pager);
@@ -730,7 +735,7 @@ print_aligned_text(const printTableContent *cont, FILE *fout, bool is_pager)
}
/* scan all cells, find maximum width, compute cell_count */
- for (i = 0, ptr = cont->cells; *ptr; ptr++, i++, cell_count++)
+ for (i = 0, ptr = cont->cells; *ptr; ptr++, cell_count++)
{
int width,
nl_lines,
@@ -739,14 +744,18 @@ print_aligned_text(const printTableContent *cont, FILE *fout, bool is_pager)
pg_wcssize((const unsigned char *) *ptr, strlen(*ptr), encoding,
&width, &nl_lines, &bytes_required);
- if (width > max_width[i % col_count])
- max_width[i % col_count] = width;
- if (nl_lines > max_nl_lines[i % col_count])
- max_nl_lines[i % col_count] = nl_lines;
- if (bytes_required > max_bytes[i % col_count])
- max_bytes[i % col_count] = bytes_required;
+ if (width > max_width[i])
+ max_width[i] = width;
+ if (nl_lines > max_nl_lines[i])
+ max_nl_lines[i] = nl_lines;
+ if (bytes_required > max_bytes[i])
+ max_bytes[i] = bytes_required;
+
+ width_average[i] += width;
- width_average[i % col_count] += width;
+ /* i is the current column number: increment with wrap */
+ if (++i >= col_count)
+ i = 0;
}
/* If we have rows, compute average */
@@ -3046,12 +3055,31 @@ set_sigpipe_trap_state(bool ignore)
/*
* PageOutput
*
- * Tests if pager is needed and returns appropriate FILE pointer.
+ * Tests if pager is needed and returns appropriate FILE pointer
+ * (either a pipe, or stdout if we don't need the pager).
+ *
+ * lines: number of lines that will be printed
+ * topt: print formatting options
*
* If the topt argument is NULL no pager is used.
*/
FILE *
PageOutput(int lines, const printTableOpt *topt)
+{
+ return PageOutputInternal(lines, topt, NULL, NULL, false);
+}
+
+/*
+ * Private version that allows for line-counting to be avoided when
+ * not needed. If "cont" is not null then the input value of "lines"
+ * is ignored and we count lines based on cont + width_wrap + expanded
+ * (see count_table_lines).
+ */
+static FILE *
+PageOutputInternal(int lines, const printTableOpt *topt,
+ const printTableContent *cont,
+ const unsigned int *width_wrap,
+ bool expanded)
{
/* check whether we need / can / are supposed to use pager */
if (topt && topt->pager && isatty(fileno(stdin)) && isatty(fileno(stdout)))
@@ -3059,15 +3087,29 @@ PageOutput(int lines, const printTableOpt *topt)
#ifdef TIOCGWINSZ
unsigned short int pager = topt->pager;
int min_lines = topt->pager_min_lines;
- int result;
- struct winsize screen_size;
- result = ioctl(fileno(stdout), TIOCGWINSZ, &screen_size);
+ if (pager == 1)
+ {
+ int result;
+ struct winsize screen_size;
- /* >= accounts for a one-line prompt */
- if (result == -1
- || (lines >= screen_size.ws_row && lines >= min_lines)
- || pager > 1)
+ result = ioctl(fileno(stdout), TIOCGWINSZ, &screen_size);
+ if (result < 0)
+ pager = 2; /* force use of pager */
+ else
+ {
+ int threshold = Max(screen_size.ws_row, min_lines);
+
+ if (cont) /* caller wants us to calculate lines */
+ lines = count_table_lines(cont, width_wrap, expanded,
+ threshold);
+ /* >= accounts for a one-line prompt */
+ if (lines >= threshold)
+ pager = 2;
+ }
+ }
+
+ if (pager > 1)
#endif
{
const char *pagerprog;
@@ -3379,11 +3421,7 @@ IsPagerNeeded(const printTableContent *cont, const unsigned int *width_wrap,
{
if (*fout == stdout)
{
- int lines;
-
- lines = count_table_lines(cont, width_wrap, expanded);
-
- *fout = PageOutput(lines, cont->opt);
+ *fout = PageOutputInternal(0, cont->opt, cont, width_wrap, expanded);
*is_pager = (*fout != stdout);
}
else
@@ -3396,13 +3434,18 @@ IsPagerNeeded(const printTableContent *cont, const unsigned int *width_wrap,
* cont: table data to be printed
* width_wrap[]: per-column maximum width, or NULL if caller will not wrap
* expanded: expanded mode?
+ * threshold: we can stop counting once we pass threshold
*
* We currently handle only regular and expanded modes here.
+ *
+ * The value of the threshold parameter is that when the table is very
+ * long, we'll typically be able to stop scanning after not many rows.
*/
static int
count_table_lines(const printTableContent *cont,
const unsigned int *width_wrap,
- bool expanded)
+ bool expanded,
+ int threshold)
{
int *header_height;
int lines,
@@ -3468,11 +3511,14 @@ count_table_lines(const printTableContent *cont,
lines += max_lines;
max_lines = 0;
}
+ /* Stop scanning table body once we pass threshold */
+ if (lines > threshold)
+ break;
}
}
/* Account for header and footer decoration */
- if (!cont->opt->tuples_only)
+ if (!cont->opt->tuples_only && lines <= threshold)
{
if (cont->title)
{
@@ -3500,6 +3546,9 @@ count_table_lines(const printTableContent *cont,
pg_wcssize((const unsigned char *) f->data, strlen(f->data),
encoding, NULL, &nl_lines, NULL);
lines += nl_lines;
+ /* Stop scanning footers once we pass threshold */
+ if (lines > threshold)
+ break;
}
}
--
2.43.7
v4-0003-Add-a-TAP-test-to-exercise-psql-s-use-of-PAGER.patchtext/x-diff; charset=us-ascii; name*0=v4-0003-Add-a-TAP-test-to-exercise-psql-s-use-of-PAGER.patc; name*1=hDownload
From 8b7a5e79d98ebe7d7e31e200d66fb249816fc17e Mon Sep 17 00:00:00 2001
From: Tom Lane <tgl@sss.pgh.pa.us>
Date: Thu, 2 Oct 2025 16:54:27 -0400
Subject: [PATCH v4 3/3] Add a TAP test to exercise psql's use of PAGER.
Up to now, we have had zero test coverage of these code paths,
because they aren't reached unless isatty(stdout). We do have
the test infrastructure to improve that situation, though.
Following the lead of 010_tab_completion.pl, set up an
interactive psql and feed it some cases that should make it
decide it needs to use the pager.
It's not at all clear how portable this is, though, so I
await the results of CI with interest.
Another issue is that as written, the test cases should cause
use of the pager, but the test will not reveal whether they
did or not. (Code coverage testing shows that the right
lines of code are reached now, but that's not really a good
answer.) Doing better seems to require use of something
besides "cat" as $PAGER, which adds more portability questions.
---
src/bin/psql/meson.build | 1 +
src/bin/psql/t/030_pager.pl | 84 +++++++++++++++++++++++++++++++++++++
2 files changed, 85 insertions(+)
create mode 100644 src/bin/psql/t/030_pager.pl
diff --git a/src/bin/psql/meson.build b/src/bin/psql/meson.build
index f795ff28271..d344053c23b 100644
--- a/src/bin/psql/meson.build
+++ b/src/bin/psql/meson.build
@@ -77,6 +77,7 @@ tests += {
't/001_basic.pl',
't/010_tab_completion.pl',
't/020_cancel.pl',
+ 't/030_pager.pl',
],
},
}
diff --git a/src/bin/psql/t/030_pager.pl b/src/bin/psql/t/030_pager.pl
new file mode 100644
index 00000000000..4bd6759a32a
--- /dev/null
+++ b/src/bin/psql/t/030_pager.pl
@@ -0,0 +1,84 @@
+
+# Copyright (c) 2021-2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Data::Dumper;
+
+# If we don't have IO::Pty, forget it, because IPC::Run depends on that
+# to support pty connections
+eval { require IO::Pty; };
+if ($@)
+{
+ plan skip_all => 'IO::Pty is needed to run this test';
+}
+
+# start a new server
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init;
+$node->start;
+
+# we don't really care about the pager's behavior, so just use cat
+$ENV{PAGER} = "cat";
+
+# fire up an interactive psql session
+my $h = $node->interactive_psql('postgres');
+
+# set the pty's window size to known values
+# (requires undesirable chumminess with the innards of IPC::Run)
+for my $pty (values %{ $h->{run}->{PTYS} })
+{
+ $pty->set_winsize(24, 80);
+}
+
+# Simple test case: type something and see if psql responds as expected
+sub do_command
+{
+ my ($send, $pattern, $annotation) = @_;
+
+ # report test failures from caller location
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+
+ # restart per-command timer
+ $h->{timeout}->start($PostgreSQL::Test::Utils::timeout_default);
+
+ # send the data to be sent and wait for its result
+ my $out = $h->query_until($pattern, $send);
+ my $okay = ($out =~ $pattern && !$h->{timeout}->is_expired);
+ ok($okay, $annotation);
+ # for debugging, log actual output if it didn't match
+ local $Data::Dumper::Terse = 1;
+ local $Data::Dumper::Useqq = 1;
+ diag 'Actual output was ' . Dumper($out) . "Did not match \"$pattern\"\n"
+ if !$okay;
+ return;
+}
+
+# Test invocation of the pager
+# (Note that interactive_psql starts psql with --no-align --tuples-only)
+
+do_command(
+ "SELECT generate_series(1,10*10) as g;\n",
+ qr/100/,
+ "execute SELECT query that needs pagination");
+
+do_command(
+ "\\pset expanded\nSELECT generate_series(1,20) as g;\n",
+ qr/g\|20/,
+ "execute SELECT query that needs pagination in expanded mode");
+
+do_command(
+ "\\pset tuples_only off\n\\d+ information_schema.referential_constraints\n",
+ qr/has_any_column_privilege/,
+ "execute command with footer that needs pagination");
+
+# send psql an explicit \q to shut it down, else pty won't close properly
+$h->quit or die "psql returned $?";
+
+# done
+$node->stop;
+done_testing();
--
2.43.7
BTW, I see from the cfbot that my v4-0002 patch is causing a
build warning on Windows:
print.c:3445:1: error: ‘count_table_lines’ defined but not used [-Werror=unused-function]
Evidently this is because the one call site is now within
"#ifdef TIOCGWINSZ", and Windows hasn't got that symbol.
So we'll need to likewise qualify count_table_lines.
I don't feel a need to post a new patch, since this is
a trivial fix and besides I'm not sure yet if we want
0002 at all.
regards, tom lane
v4-0001 LGTM. It still gives me the same pager line count as my v2.
On 2025-10-02 23:00 +0200, Tom Lane wrote:
Erik Wienhold <ewie@ewie.name> writes:
On 2025-10-02 00:25 +0200, Tom Lane wrote:
I am not entirely sure that we should commit 0002 though; it may be
that the savings is down in the noise anyway once you consider all
the other work that happens while printing a big table.
I see larger gains for queries that produce cells with many newlines,
benchmarked with a variation of my test-psql-pager.py script from
upthread. It measures the mtime of a file created before running psql
and a second file created by the pager using PAGER='touch somefile'
(that should cover the query execution time plus the formatting
overhead). With that I consistently measure these times for psql's
normal output format:
Query: SELECT repeat(e'foo\n', :n_lines) FROM generate_series(1, :n_rows)
n_lines | n_rows | time[ms] (master) | time[ms] (v4) | diff[%]
---------|----------|---------------------|---------------|---------
1 | 100000 | 30.000 | 26.667 | -11.11
1 | 1000000 | 200.000 | 173.333 | -13.33
1 | 10000000 | 1889.998 | 1613.332 | -14.63
10 | 10000 | 16.667 | 13.333 | -20.00
10 | 100000 | 73.333 | 50.000 | -31.82
10 | 1000000 | 583.333 | 390.000 | -33.14
100 | 1000 | 16.667 | 10.000 | -40.00
100 | 10000 | 60.000 | 36.667 | -38.89
100 | 100000 | 490.000 | 280.000 | -42.86
1000 | 100 | 16.667 | 13.333 | -20.00
1000 | 1000 | 56.667 | 33.333 | -41.18
1000 | 10000 | 483.333 | 260.000 | -46.21
Based on that I think you should push v4-0002.
A positive reason not to take it is something I realized while
checking test coverage: we never execute any of the
maybe-use-the-pager branch of PageOutput in the regression tests,
because isatty(stdout) will always fail.I realized that we could address that if we really wanted to, using
the same infrastructure as for tab-completion testing. 0001 and 0002
attached are the same as before, and then I added 0003 which adds a
draft-quality TAP test. Code coverage checks show that this adds only
about 10 more lines of coverage in psql proper, but in print.c most of
the pager-related logic is now covered.
+1 on the TAP tests in general. But I'm not too familiar with the TAP
infrastructure to provide any meaningful review, given that you consider
it just draft-quality.
However, I'm not sure how useful it really is to have code coverage
there, since by this very token the tests are paying zero attention
to the value computed by count_table_lines. It could be
arbitrarily silly and we'd not see that.
Can we use something along the lines of my test-psql-pager.py to binary
search the line count at which psql uses the pager. Because it would
only apply to systems that provide termios.h, we might be less
constrained to find a portable solution that tests whether the pager
triggers or not (which you noted in v4-0003.)
--
Erik Wienhold
Erik Wienhold <ewie@ewie.name> writes:
On 2025-10-02 00:25 +0200, Tom Lane wrote:
I am not entirely sure that we should commit 0002 though; it may be
that the savings is down in the noise anyway once you consider all
the other work that happens while printing a big table.
I see larger gains for queries that produce cells with many newlines,
benchmarked with a variation of my test-psql-pager.py script from
upthread. ...
Based on that I think you should push v4-0002.
OK, we'll keep it.
As I was doing some desultory manual testing, I realized that v4
still has a notable gap: it fails to account for the "(n rows)"
default footer. That's because that is not present in cont->footers
but is manually injected by footers_with_default, and count_table_rows
wasn't dealing with that. It's a little more complicated than just
calling that function, because the various vertical modes don't use
the default footer. After some code study I decided that we could
use the existing "expanded" flag to decide which footer definition
to apply, but I decided to rename it to "vertical" to reduce confusion
with the not-a-boolean cont->opt->expanded field.
Despite all this hackery, the count is still only really accurate
for the "\pset format aligned" modes. I figure we can leave
tidying up the other modes for another day, because I doubt that
anybody cares about interactive output of, say, LaTeX format.
(Perhaps unaligned mode would be the first priority to tidy up.)
Another inaccuracy that I find mildly annoying now that we've got it
mostly right is that we are not accounting for the blank line that
psql likes to print after the query result, so that you can still
end with the first output line scrolled off your screen despite
non-use of the pager. Not sure about a non-kluge way to count that,
though: it's outside the domain of print.c, and other callers might
not have a similar behavior.
I cleaned up the TAP test, too. This version redirects PAGER
to "wc -l", so that it's very obvious from the output whether
or not the pager was used.
regards, tom lane
Attachments:
v5-0001-Improve-psql-s-ability-to-select-pager-mode-accur.patchtext/x-diff; charset=us-ascii; name*0=v5-0001-Improve-psql-s-ability-to-select-pager-mode-accur.p; name*1=atchDownload
From 436b9559d58886b55a75624ae7b740d974d6a0d6 Mon Sep 17 00:00:00 2001
From: Tom Lane <tgl@sss.pgh.pa.us>
Date: Mon, 6 Oct 2025 15:12:53 -0400
Subject: [PATCH v5 1/3] Improve psql's ability to select pager mode
accurately.
The code in print.c that's concerned with counting the number of lines
that will be needed missed a lot of edge cases:
* While plain aligned mode accounted for embedded newlines in column
headers and table cells, unaligned and vertical output modes did not.
* In particular, since vertical mode repeats the headers for each
record, we need to account for embedded newlines in the headers for
each record.
* Multi-line table titles were not accounted for.
* tuples_only mode (where headers aren't printed) wasn't accounted
for.
* Footers were accounted for as one line per footer, again missing
the possibility of multi-line footers. (In some cases such as
"\d+" on a view, there can be many lines in a footer.) Also,
we failed to count the default footer.
To fix, move the entire responsibility for counting lines into
IsPagerNeeded (or actually, into a new subroutine count_table_lines),
and then expand the logic as appropriate. Also restructure to make
it perhaps a bit easier to follow.
In passing, move the "flog" output step to the bottom of printTable(),
rather than running it when we've already opened the pager in some
modes. In principle it shouldn't interfere with the pager because
flog should always point to a non-interactive file; but it seems silly
to risk any interference, especially when the existing positioning
seems to have been chosen with the aid of a dartboard.
Author: Erik Wienhold <ewie@ewie.name>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/2dd2430f-dd20-4c89-97fd-242616a3d768@ewie.name
---
src/fe_utils/print.c | 225 ++++++++++++++++++++++++++++++-------------
1 file changed, 159 insertions(+), 66 deletions(-)
diff --git a/src/fe_utils/print.c b/src/fe_utils/print.c
index 4af0f32f2fc..709b1254c8b 100644
--- a/src/fe_utils/print.c
+++ b/src/fe_utils/print.c
@@ -266,8 +266,13 @@ static const unicodeStyleFormat unicode_style = {
/* Local functions */
static int strlen_max_width(unsigned char *str, int *target_width, int encoding);
-static void IsPagerNeeded(const printTableContent *cont, int extra_lines, bool expanded,
+static void IsPagerNeeded(const printTableContent *cont,
+ const unsigned int *width_wrap,
+ bool vertical,
FILE **fout, bool *is_pager);
+static int count_table_lines(const printTableContent *cont,
+ const unsigned int *width_wrap,
+ bool vertical);
static void print_aligned_vertical(const printTableContent *cont,
FILE *fout, bool is_pager);
@@ -656,8 +661,6 @@ print_aligned_text(const printTableContent *cont, FILE *fout, bool is_pager)
unsigned char **format_buf;
unsigned int width_total;
unsigned int total_header_width;
- unsigned int extra_row_output_lines = 0;
- unsigned int extra_output_lines = 0;
const char *const *ptr;
@@ -722,14 +725,9 @@ print_aligned_text(const printTableContent *cont, FILE *fout, bool is_pager)
max_nl_lines[i] = nl_lines;
if (bytes_required > max_bytes[i])
max_bytes[i] = bytes_required;
- if (nl_lines > extra_row_output_lines)
- extra_row_output_lines = nl_lines;
width_header[i] = width;
}
- /* Add height of tallest header column */
- extra_output_lines += extra_row_output_lines;
- extra_row_output_lines = 0;
/* scan all cells, find maximum width, compute cell_count */
for (i = 0, ptr = cont->cells; *ptr; ptr++, i++, cell_count++)
@@ -889,43 +887,10 @@ print_aligned_text(const printTableContent *cont, FILE *fout, bool is_pager)
is_pager = is_local_pager = true;
}
- /* Check if newlines or our wrapping now need the pager */
- if (!is_pager && fout == stdout)
+ /* Check if there are enough lines to require the pager */
+ if (!is_pager)
{
- /* scan all cells, find maximum width, compute cell_count */
- for (i = 0, ptr = cont->cells; *ptr; ptr++, cell_count++)
- {
- int width,
- nl_lines,
- bytes_required;
-
- pg_wcssize((const unsigned char *) *ptr, strlen(*ptr), encoding,
- &width, &nl_lines, &bytes_required);
-
- /*
- * A row can have both wrapping and newlines that cause it to
- * display across multiple lines. We check for both cases below.
- */
- if (width > 0 && width_wrap[i])
- {
- unsigned int extra_lines;
-
- /* don't count the first line of nl_lines - it's not "extra" */
- extra_lines = ((width - 1) / width_wrap[i]) + nl_lines - 1;
- if (extra_lines > extra_row_output_lines)
- extra_row_output_lines = extra_lines;
- }
-
- /* i is the current column number: increment with wrap */
- if (++i >= col_count)
- {
- i = 0;
- /* At last column of each row, add tallest column height */
- extra_output_lines += extra_row_output_lines;
- extra_row_output_lines = 0;
- }
- }
- IsPagerNeeded(cont, extra_output_lines, false, &fout, &is_pager);
+ IsPagerNeeded(cont, width_wrap, false, &fout, &is_pager);
is_local_pager = is_pager;
}
@@ -1351,6 +1316,11 @@ print_aligned_vertical(const printTableContent *cont,
if (opt_border > 2)
opt_border = 2;
+ /*
+ * Kluge for totally empty table: use the default footer even though
+ * vertical modes normally don't. Otherwise we'd print nothing at all,
+ * which isn't terribly friendly. Assume pager will not be needed.
+ */
if (cont->cells[0] == NULL && cont->opt->start_table &&
cont->opt->stop_table)
{
@@ -1376,7 +1346,7 @@ print_aligned_vertical(const printTableContent *cont,
*/
if (!is_pager)
{
- IsPagerNeeded(cont, 0, true, &fout, &is_pager);
+ IsPagerNeeded(cont, NULL, true, &fout, &is_pager);
is_local_pager = is_pager;
}
@@ -3398,37 +3368,160 @@ printTableCleanup(printTableContent *const content)
* IsPagerNeeded
*
* Setup pager if required
+ *
+ * cont: table data to be printed
+ * width_wrap[]: per-column maximum width, or NULL if caller will not wrap
+ * vertical: vertical mode?
+ * fout: where to print to (in/out argument)
+ * is_pager: output argument
+ *
+ * If we decide pager is needed, *fout is modified and *is_pager is set true
*/
static void
-IsPagerNeeded(const printTableContent *cont, int extra_lines, bool expanded,
+IsPagerNeeded(const printTableContent *cont, const unsigned int *width_wrap,
+ bool vertical,
FILE **fout, bool *is_pager)
{
if (*fout == stdout)
{
int lines;
- if (expanded)
- lines = (cont->ncolumns + 1) * cont->nrows;
+ lines = count_table_lines(cont, width_wrap, vertical);
+
+ *fout = PageOutput(lines, cont->opt);
+ *is_pager = (*fout != stdout);
+ }
+ else
+ *is_pager = false;
+}
+
+/*
+ * Count the number of lines needed to print the given table.
+ *
+ * cont: table data to be printed
+ * width_wrap[]: per-column maximum width, or NULL if caller will not wrap
+ * vertical: vertical mode?
+ *
+ * The result is currently only fully accurate for aligned_text and
+ * aligned_vertical formats; otherwise it's an approximation.
+ *
+ * Note: while cont->opt will tell us most formatting details, we need the
+ * separate "vertical" flag because of the possibility of a dynamic switch
+ * from aligned_text to aligned_vertical format.
+ */
+static int
+count_table_lines(const printTableContent *cont,
+ const unsigned int *width_wrap,
+ bool vertical)
+{
+ int *header_height;
+ int lines,
+ max_lines = 0,
+ nl_lines,
+ i;
+ int encoding = cont->opt->encoding;
+ const char *const *cell;
+
+ /*
+ * Scan all column headers and determine their heights. Cache the values
+ * since vertical mode repeats the headers for every record.
+ */
+ header_height = (int *) pg_malloc(cont->ncolumns * sizeof(int));
+ for (i = 0; i < cont->ncolumns; i++)
+ {
+ pg_wcssize((const unsigned char *) cont->headers[i],
+ strlen(cont->headers[i]), encoding,
+ NULL, &header_height[i], NULL);
+ }
+
+ /*
+ * Vertical mode writes one separator line per record. Normal mode writes
+ * a single separator line between header and rows.
+ */
+ lines = vertical ? cont->nrows : 1;
+
+ /* Scan all cells to count their lines */
+ for (i = 0, cell = cont->cells; *cell; cell++)
+ {
+ int width;
+
+ /* Count the original line breaks */
+ pg_wcssize((const unsigned char *) *cell, strlen(*cell), encoding,
+ &width, &nl_lines, NULL);
+
+ /* Count extra lines due to wrapping */
+ if (width > 0 && width_wrap && width_wrap[i])
+ nl_lines += (width - 1) / width_wrap[i];
+
+ if (vertical)
+ {
+ /* Pick the height of the header or cell, whichever is taller */
+ if (nl_lines > header_height[i])
+ lines += nl_lines;
+ else
+ lines += header_height[i];
+ }
else
- lines = cont->nrows + 1;
+ {
+ /* Remember max height in the current row */
+ if (nl_lines > max_lines)
+ max_lines = nl_lines;
+ }
- if (!cont->opt->tuples_only)
+ /* i is the current column number: increment with wrap */
+ if (++i >= cont->ncolumns)
{
- printTableFooter *f;
+ i = 0;
+ if (!vertical)
+ {
+ /* At last column of each row, add tallest column height */
+ lines += max_lines;
+ max_lines = 0;
+ }
+ }
+ }
- /*
- * FIXME -- this is slightly bogus: it counts the number of
- * footers, not the number of lines in them.
- */
- for (f = cont->footers; f; f = f->next)
- lines++;
+ /* Account for header and footer decoration */
+ if (!cont->opt->tuples_only)
+ {
+ printTableFooter *f;
+
+ if (cont->title)
+ {
+ /* Add height of title */
+ pg_wcssize((const unsigned char *) cont->title, strlen(cont->title),
+ encoding, NULL, &nl_lines, NULL);
+ lines += nl_lines;
}
- *fout = PageOutput(lines + extra_lines, cont->opt);
- *is_pager = (*fout != stdout);
+ if (!vertical)
+ {
+ /* Add height of tallest header column */
+ max_lines = 0;
+ for (i = 0; i < cont->ncolumns; i++)
+ {
+ if (header_height[i] > max_lines)
+ max_lines = header_height[i];
+ }
+ lines += max_lines;
+ }
+
+ /*
+ * Add all footer lines. Vertical mode does not use the default
+ * footer, but we must include that in normal mode.
+ */
+ for (f = vertical ? cont->footers : footers_with_default(cont);
+ f != NULL; f = f->next)
+ {
+ pg_wcssize((const unsigned char *) f->data, strlen(f->data),
+ encoding, NULL, &nl_lines, NULL);
+ lines += nl_lines;
+ }
}
- else
- *is_pager = false;
+
+ free(header_height);
+
+ return lines;
}
/*
@@ -3456,7 +3549,7 @@ printTable(const printTableContent *cont,
cont->opt->format != PRINT_ALIGNED &&
cont->opt->format != PRINT_WRAPPED)
{
- IsPagerNeeded(cont, 0, (cont->opt->expanded == 1), &fout, &is_pager);
+ IsPagerNeeded(cont, NULL, (cont->opt->expanded == 1), &fout, &is_pager);
is_local_pager = is_pager;
}
@@ -3464,10 +3557,6 @@ printTable(const printTableContent *cont,
clearerr(fout);
/* print the stuff */
-
- if (flog)
- print_aligned_text(cont, flog, false);
-
switch (cont->opt->format)
{
case PRINT_UNALIGNED:
@@ -3534,6 +3623,10 @@ printTable(const printTableContent *cont,
if (is_local_pager)
ClosePager(fout);
+
+ /* also produce log output if wanted */
+ if (flog)
+ print_aligned_text(cont, flog, false);
}
/*
--
2.43.7
v5-0002-Make-some-minor-performance-improvements-in-psql-.patchtext/x-diff; charset=us-ascii; name*0=v5-0002-Make-some-minor-performance-improvements-in-psql-.p; name*1=atchDownload
From 2504c5b36b90808dbb5353268ce94b274e80b194 Mon Sep 17 00:00:00 2001
From: Tom Lane <tgl@sss.pgh.pa.us>
Date: Mon, 6 Oct 2025 15:30:01 -0400
Subject: [PATCH v5 2/3] Make some minor performance improvements in psql's
line-counting.
Arrange to not run count_table_lines at all unless we will use
its result, and teach it to quit early as soon as it's proven
that the output is long enough to require use of the pager.
When dealing with large tables this should save a noticeable
amount of time, since pg_wcssize() isn't exactly cheap.
In passing, fix print_aligned_text to avoid using repeated
"i % col_count" when it could just make the value of i wrap
around. In a quick check with the version of gcc I'm using,
the compiler is smart enough to recognize the common subexpressions
and issue only one integer divide per loop, but it's not smart enough
to avoid the integer divide altogether. In any case, this coding
was randomly unlike the way it's done in count_table_lines.
Author: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/2dd2430f-dd20-4c89-97fd-242616a3d768@ewie.name
---
src/fe_utils/print.c | 108 +++++++++++++++++++++++++++++++++----------
1 file changed, 83 insertions(+), 25 deletions(-)
diff --git a/src/fe_utils/print.c b/src/fe_utils/print.c
index 709b1254c8b..3caf33b377e 100644
--- a/src/fe_utils/print.c
+++ b/src/fe_utils/print.c
@@ -33,6 +33,11 @@
#include "fe_utils/mbprint.h"
#include "fe_utils/print.h"
+/* Presently, count_table_lines() is only used within #ifdef TIOCGWINSZ */
+#ifdef TIOCGWINSZ
+#define NEED_COUNT_TABLE_LINES
+#endif
+
/*
* If the calling program doesn't have any mechanism for setting
* cancel_pressed, it will have no effect.
@@ -266,14 +271,20 @@ static const unicodeStyleFormat unicode_style = {
/* Local functions */
static int strlen_max_width(unsigned char *str, int *target_width, int encoding);
+static FILE *PageOutputInternal(int lines, const printTableOpt *topt,
+ const printTableContent *cont,
+ const unsigned int *width_wrap,
+ bool vertical);
static void IsPagerNeeded(const printTableContent *cont,
const unsigned int *width_wrap,
bool vertical,
FILE **fout, bool *is_pager);
+#ifdef NEED_COUNT_TABLE_LINES
static int count_table_lines(const printTableContent *cont,
const unsigned int *width_wrap,
- bool vertical);
-
+ bool vertical,
+ int threshold);
+#endif
static void print_aligned_vertical(const printTableContent *cont,
FILE *fout, bool is_pager);
@@ -730,7 +741,7 @@ print_aligned_text(const printTableContent *cont, FILE *fout, bool is_pager)
}
/* scan all cells, find maximum width, compute cell_count */
- for (i = 0, ptr = cont->cells; *ptr; ptr++, i++, cell_count++)
+ for (i = 0, ptr = cont->cells; *ptr; ptr++, cell_count++)
{
int width,
nl_lines,
@@ -739,14 +750,18 @@ print_aligned_text(const printTableContent *cont, FILE *fout, bool is_pager)
pg_wcssize((const unsigned char *) *ptr, strlen(*ptr), encoding,
&width, &nl_lines, &bytes_required);
- if (width > max_width[i % col_count])
- max_width[i % col_count] = width;
- if (nl_lines > max_nl_lines[i % col_count])
- max_nl_lines[i % col_count] = nl_lines;
- if (bytes_required > max_bytes[i % col_count])
- max_bytes[i % col_count] = bytes_required;
+ if (width > max_width[i])
+ max_width[i] = width;
+ if (nl_lines > max_nl_lines[i])
+ max_nl_lines[i] = nl_lines;
+ if (bytes_required > max_bytes[i])
+ max_bytes[i] = bytes_required;
+
+ width_average[i] += width;
- width_average[i % col_count] += width;
+ /* i is the current column number: increment with wrap */
+ if (++i >= col_count)
+ i = 0;
}
/* If we have rows, compute average */
@@ -3051,28 +3066,62 @@ set_sigpipe_trap_state(bool ignore)
/*
* PageOutput
*
- * Tests if pager is needed and returns appropriate FILE pointer.
+ * Tests if pager is needed and returns appropriate FILE pointer
+ * (either a pipe, or stdout if we don't need the pager).
+ *
+ * lines: number of lines that will be printed
+ * topt: print formatting options
*
* If the topt argument is NULL no pager is used.
*/
FILE *
PageOutput(int lines, const printTableOpt *topt)
+{
+ return PageOutputInternal(lines, topt, NULL, NULL, false);
+}
+
+/*
+ * Private version that allows for line-counting to be avoided when
+ * not needed. If "cont" is not null then the input value of "lines"
+ * is ignored and we count lines based on cont + width_wrap + vertical
+ * (see count_table_lines).
+ */
+static FILE *
+PageOutputInternal(int lines, const printTableOpt *topt,
+ const printTableContent *cont,
+ const unsigned int *width_wrap,
+ bool vertical)
{
/* check whether we need / can / are supposed to use pager */
if (topt && topt->pager && isatty(fileno(stdin)) && isatty(fileno(stdout)))
{
+ /* without TIOCGWINSZ, pager == 1 acts the same as pager > 1 */
#ifdef TIOCGWINSZ
unsigned short int pager = topt->pager;
int min_lines = topt->pager_min_lines;
- int result;
- struct winsize screen_size;
- result = ioctl(fileno(stdout), TIOCGWINSZ, &screen_size);
+ if (pager == 1)
+ {
+ int result;
+ struct winsize screen_size;
+
+ result = ioctl(fileno(stdout), TIOCGWINSZ, &screen_size);
+ if (result < 0)
+ pager = 2; /* force use of pager */
+ else
+ {
+ int threshold = Max(screen_size.ws_row, min_lines);
+
+ if (cont) /* caller wants us to calculate lines */
+ lines = count_table_lines(cont, width_wrap, vertical,
+ threshold);
+ /* >= accounts for a one-line prompt */
+ if (lines >= threshold)
+ pager = 2;
+ }
+ }
- /* >= accounts for a one-line prompt */
- if (result == -1
- || (lines >= screen_size.ws_row && lines >= min_lines)
- || pager > 1)
+ if (pager > 1)
#endif
{
const char *pagerprog;
@@ -3384,11 +3433,7 @@ IsPagerNeeded(const printTableContent *cont, const unsigned int *width_wrap,
{
if (*fout == stdout)
{
- int lines;
-
- lines = count_table_lines(cont, width_wrap, vertical);
-
- *fout = PageOutput(lines, cont->opt);
+ *fout = PageOutputInternal(0, cont->opt, cont, width_wrap, vertical);
*is_pager = (*fout != stdout);
}
else
@@ -3401,6 +3446,7 @@ IsPagerNeeded(const printTableContent *cont, const unsigned int *width_wrap,
* cont: table data to be printed
* width_wrap[]: per-column maximum width, or NULL if caller will not wrap
* vertical: vertical mode?
+ * threshold: we can stop counting once we pass this many lines
*
* The result is currently only fully accurate for aligned_text and
* aligned_vertical formats; otherwise it's an approximation.
@@ -3408,11 +3454,16 @@ IsPagerNeeded(const printTableContent *cont, const unsigned int *width_wrap,
* Note: while cont->opt will tell us most formatting details, we need the
* separate "vertical" flag because of the possibility of a dynamic switch
* from aligned_text to aligned_vertical format.
+ *
+ * The value of the threshold parameter is that when the table is very
+ * long, we'll typically be able to stop scanning after not many rows.
*/
+#ifdef NEED_COUNT_TABLE_LINES
static int
count_table_lines(const printTableContent *cont,
const unsigned int *width_wrap,
- bool vertical)
+ bool vertical,
+ int threshold)
{
int *header_height;
int lines,
@@ -3478,11 +3529,14 @@ count_table_lines(const printTableContent *cont,
lines += max_lines;
max_lines = 0;
}
+ /* Stop scanning table body once we pass threshold */
+ if (lines > threshold)
+ break;
}
}
/* Account for header and footer decoration */
- if (!cont->opt->tuples_only)
+ if (!cont->opt->tuples_only && lines <= threshold)
{
printTableFooter *f;
@@ -3516,6 +3570,9 @@ count_table_lines(const printTableContent *cont,
pg_wcssize((const unsigned char *) f->data, strlen(f->data),
encoding, NULL, &nl_lines, NULL);
lines += nl_lines;
+ /* Stop scanning footers once we pass threshold */
+ if (lines > threshold)
+ break;
}
}
@@ -3523,6 +3580,7 @@ count_table_lines(const printTableContent *cont,
return lines;
}
+#endif /* NEED_COUNT_TABLE_LINES */
/*
* Use this to print any table in the supported formats.
--
2.43.7
v5-0003-Add-a-TAP-test-to-exercise-psql-s-use-of-the-page.patchtext/x-diff; charset=us-ascii; name*0=v5-0003-Add-a-TAP-test-to-exercise-psql-s-use-of-the-page.p; name*1=atchDownload
From 9833f23006582e03866aaa4883e6bfa469622c24 Mon Sep 17 00:00:00 2001
From: Tom Lane <tgl@sss.pgh.pa.us>
Date: Mon, 6 Oct 2025 16:46:55 -0400
Subject: [PATCH v5 3/3] Add a TAP test to exercise psql's use of the pager.
Up to now, we have had zero test coverage of these code paths,
because they aren't reached unless isatty(stdout). We do have
the test infrastructure to improve that situation, though.
Following the lead of 010_tab_completion.pl, set up an
interactive psql and feed it some test cases.
To detect whether it really did invoke the pager, point
PSQL_PAGER to "wc -l". The test is skipped if that utility
isn't available.
Author: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/2dd2430f-dd20-4c89-97fd-242616a3d768@ewie.name
---
src/bin/psql/meson.build | 1 +
src/bin/psql/t/030_pager.pl | 104 ++++++++++++++++++++++++++++++++++++
2 files changed, 105 insertions(+)
create mode 100644 src/bin/psql/t/030_pager.pl
diff --git a/src/bin/psql/meson.build b/src/bin/psql/meson.build
index f795ff28271..d344053c23b 100644
--- a/src/bin/psql/meson.build
+++ b/src/bin/psql/meson.build
@@ -77,6 +77,7 @@ tests += {
't/001_basic.pl',
't/010_tab_completion.pl',
't/020_cancel.pl',
+ 't/030_pager.pl',
],
},
}
diff --git a/src/bin/psql/t/030_pager.pl b/src/bin/psql/t/030_pager.pl
new file mode 100644
index 00000000000..4272b0783ef
--- /dev/null
+++ b/src/bin/psql/t/030_pager.pl
@@ -0,0 +1,104 @@
+
+# Copyright (c) 2021-2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Data::Dumper;
+
+# If we don't have IO::Pty, forget it, because IPC::Run depends on that
+# to support pty connections
+eval { require IO::Pty; };
+if ($@)
+{
+ plan skip_all => 'IO::Pty is needed to run this test';
+}
+
+# Check that "wc -l" does what we expect, else forget it
+my $wcstdin = "foo bar\nbaz\n";
+my ($wcstdout, $wcstderr);
+my $result = IPC::Run::run [ 'wc', '-l' ],
+ '<' => \$wcstdin,
+ '>' => \$wcstdout,
+ '2>' => \$wcstderr;
+chomp $wcstdout;
+if ($wcstdout ne '2' || $wcstderr ne '')
+{
+ plan skip_all => '"wc -l" is needed to run this test';
+}
+
+# We set up "wc -l" as the pager so we can tell whether psql used the pager
+$ENV{PSQL_PAGER} = "wc -l";
+
+# start a new server
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init;
+$node->start;
+
+# fire up an interactive psql session
+my $h = $node->interactive_psql('postgres');
+
+# set the pty's window size to known values
+# (requires undesirable chumminess with the innards of IPC::Run)
+for my $pty (values %{ $h->{run}->{PTYS} })
+{
+ $pty->set_winsize(24, 80);
+}
+
+# Simple test case: type something and see if psql responds as expected
+sub do_command
+{
+ my ($send, $pattern, $annotation) = @_;
+
+ # report test failures from caller location
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+
+ # restart per-command timer
+ $h->{timeout}->start($PostgreSQL::Test::Utils::timeout_default);
+
+ # send the data to be sent and wait for its result
+ my $out = $h->query_until($pattern, $send);
+ my $okay = ($out =~ $pattern && !$h->{timeout}->is_expired);
+ ok($okay, $annotation);
+ # for debugging, log actual output if it didn't match
+ local $Data::Dumper::Terse = 1;
+ local $Data::Dumper::Useqq = 1;
+ diag 'Actual output was ' . Dumper($out) . "Did not match \"$pattern\"\n"
+ if !$okay;
+ return;
+}
+
+# Test invocation of the pager
+#
+# Note that interactive_psql starts psql with --no-align --tuples-only,
+# and that the output string will include psql's prompts and command echo.
+
+do_command(
+ "SELECT 'test' AS t FROM generate_series(1,22);\n",
+ qr/^test\r?$/m,
+ "execute SELECT query that needs no pagination");
+
+do_command(
+ "SELECT 'test' AS t FROM generate_series(1,24);\n",
+ qr/^24\r?$/m,
+ "execute SELECT query that needs pagination");
+
+do_command(
+ "\\pset expanded\nSELECT generate_series(1,20) as g;\n",
+ qr/^39\r?$/m,
+ "execute SELECT query that needs pagination in expanded mode");
+
+do_command(
+ "\\pset tuples_only off\n\\d+ information_schema.referential_constraints\n",
+ qr/^\d+\r?$/m,
+ "execute command with footer that needs pagination");
+
+# send psql an explicit \q to shut it down, else pty won't close properly
+$h->quit or die "psql returned $?";
+
+# done
+$node->stop;
+done_testing();
--
2.43.7
I wrote:
Another inaccuracy that I find mildly annoying now that we've got it
mostly right is that we are not accounting for the blank line that
psql likes to print after the query result, so that you can still
end with the first output line scrolled off your screen despite
non-use of the pager. Not sure about a non-kluge way to count that,
though: it's outside the domain of print.c, and other callers might
not have a similar behavior.
Oh wait! That line *is* produced within print.c, so there is no
reason we shouldn't account for it. I spent a little extra effort
on unaligned mode too, since the TAP test is checking that case.
I still doubt that anyone's going to care a lot about the other
formats.
regards, tom lane
Attachments:
v6-0001-Improve-psql-s-ability-to-select-pager-mode-accur.patchtext/x-diff; charset=us-ascii; name*0=v6-0001-Improve-psql-s-ability-to-select-pager-mode-accur.p; name*1=atchDownload
From f10d62c24975832d17556bf003ab71d3d3fdc024 Mon Sep 17 00:00:00 2001
From: Tom Lane <tgl@sss.pgh.pa.us>
Date: Mon, 6 Oct 2025 18:34:26 -0400
Subject: [PATCH v6 1/3] Improve psql's ability to select pager mode
accurately.
The code in print.c that's concerned with counting the number of lines
that will be needed missed a lot of edge cases:
* While plain aligned mode accounted for embedded newlines in column
headers and table cells, unaligned and vertical output modes did not.
* In particular, since vertical mode repeats the headers for each
record, we need to account for embedded newlines in the headers for
each record.
* Multi-line table titles were not accounted for.
* tuples_only mode (where headers aren't printed) wasn't accounted
for.
* Footers were accounted for as one line per footer, again missing
the possibility of multi-line footers. (In some cases such as
"\d+" on a view, there can be many lines in a footer.) Also,
we failed to count the default footer.
To fix, move the entire responsibility for counting lines into
IsPagerNeeded (or actually, into a new subroutine count_table_lines),
and then expand the logic as appropriate. Also restructure to make
it perhaps a bit easier to follow.
In passing, move the "flog" output step to the bottom of printTable(),
rather than running it when we've already opened the pager in some
modes. In principle it shouldn't interfere with the pager because
flog should always point to a non-interactive file; but it seems silly
to risk any interference, especially when the existing positioning
seems to have been chosen with the aid of a dartboard.
Author: Erik Wienhold <ewie@ewie.name>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/2dd2430f-dd20-4c89-97fd-242616a3d768@ewie.name
---
src/fe_utils/print.c | 267 ++++++++++++++++++++++++++++++++-----------
1 file changed, 201 insertions(+), 66 deletions(-)
diff --git a/src/fe_utils/print.c b/src/fe_utils/print.c
index 4af0f32f2fc..5082f0808b1 100644
--- a/src/fe_utils/print.c
+++ b/src/fe_utils/print.c
@@ -266,8 +266,13 @@ static const unicodeStyleFormat unicode_style = {
/* Local functions */
static int strlen_max_width(unsigned char *str, int *target_width, int encoding);
-static void IsPagerNeeded(const printTableContent *cont, int extra_lines, bool expanded,
+static void IsPagerNeeded(const printTableContent *cont,
+ const unsigned int *width_wrap,
+ bool vertical,
FILE **fout, bool *is_pager);
+static int count_table_lines(const printTableContent *cont,
+ const unsigned int *width_wrap,
+ bool vertical);
static void print_aligned_vertical(const printTableContent *cont,
FILE *fout, bool is_pager);
@@ -656,8 +661,6 @@ print_aligned_text(const printTableContent *cont, FILE *fout, bool is_pager)
unsigned char **format_buf;
unsigned int width_total;
unsigned int total_header_width;
- unsigned int extra_row_output_lines = 0;
- unsigned int extra_output_lines = 0;
const char *const *ptr;
@@ -722,14 +725,9 @@ print_aligned_text(const printTableContent *cont, FILE *fout, bool is_pager)
max_nl_lines[i] = nl_lines;
if (bytes_required > max_bytes[i])
max_bytes[i] = bytes_required;
- if (nl_lines > extra_row_output_lines)
- extra_row_output_lines = nl_lines;
width_header[i] = width;
}
- /* Add height of tallest header column */
- extra_output_lines += extra_row_output_lines;
- extra_row_output_lines = 0;
/* scan all cells, find maximum width, compute cell_count */
for (i = 0, ptr = cont->cells; *ptr; ptr++, i++, cell_count++)
@@ -889,43 +887,10 @@ print_aligned_text(const printTableContent *cont, FILE *fout, bool is_pager)
is_pager = is_local_pager = true;
}
- /* Check if newlines or our wrapping now need the pager */
- if (!is_pager && fout == stdout)
+ /* Check if there are enough lines to require the pager */
+ if (!is_pager)
{
- /* scan all cells, find maximum width, compute cell_count */
- for (i = 0, ptr = cont->cells; *ptr; ptr++, cell_count++)
- {
- int width,
- nl_lines,
- bytes_required;
-
- pg_wcssize((const unsigned char *) *ptr, strlen(*ptr), encoding,
- &width, &nl_lines, &bytes_required);
-
- /*
- * A row can have both wrapping and newlines that cause it to
- * display across multiple lines. We check for both cases below.
- */
- if (width > 0 && width_wrap[i])
- {
- unsigned int extra_lines;
-
- /* don't count the first line of nl_lines - it's not "extra" */
- extra_lines = ((width - 1) / width_wrap[i]) + nl_lines - 1;
- if (extra_lines > extra_row_output_lines)
- extra_row_output_lines = extra_lines;
- }
-
- /* i is the current column number: increment with wrap */
- if (++i >= col_count)
- {
- i = 0;
- /* At last column of each row, add tallest column height */
- extra_output_lines += extra_row_output_lines;
- extra_row_output_lines = 0;
- }
- }
- IsPagerNeeded(cont, extra_output_lines, false, &fout, &is_pager);
+ IsPagerNeeded(cont, width_wrap, false, &fout, &is_pager);
is_local_pager = is_pager;
}
@@ -1351,6 +1316,11 @@ print_aligned_vertical(const printTableContent *cont,
if (opt_border > 2)
opt_border = 2;
+ /*
+ * Kluge for totally empty table: use the default footer even though
+ * vertical modes normally don't. Otherwise we'd print nothing at all,
+ * which isn't terribly friendly. Assume pager will not be needed.
+ */
if (cont->cells[0] == NULL && cont->opt->start_table &&
cont->opt->stop_table)
{
@@ -1376,7 +1346,7 @@ print_aligned_vertical(const printTableContent *cont,
*/
if (!is_pager)
{
- IsPagerNeeded(cont, 0, true, &fout, &is_pager);
+ IsPagerNeeded(cont, NULL, true, &fout, &is_pager);
is_local_pager = is_pager;
}
@@ -3398,37 +3368,202 @@ printTableCleanup(printTableContent *const content)
* IsPagerNeeded
*
* Setup pager if required
+ *
+ * cont: table data to be printed
+ * width_wrap[]: per-column maximum width, or NULL if caller will not wrap
+ * vertical: vertical mode?
+ * fout: where to print to (in/out argument)
+ * is_pager: output argument
+ *
+ * If we decide pager is needed, *fout is modified and *is_pager is set true
*/
static void
-IsPagerNeeded(const printTableContent *cont, int extra_lines, bool expanded,
+IsPagerNeeded(const printTableContent *cont, const unsigned int *width_wrap,
+ bool vertical,
FILE **fout, bool *is_pager)
{
if (*fout == stdout)
{
int lines;
- if (expanded)
- lines = (cont->ncolumns + 1) * cont->nrows;
- else
- lines = cont->nrows + 1;
+ lines = count_table_lines(cont, width_wrap, vertical);
- if (!cont->opt->tuples_only)
- {
- printTableFooter *f;
+ *fout = PageOutput(lines, cont->opt);
+ *is_pager = (*fout != stdout);
+ }
+ else
+ *is_pager = false;
+}
+
+/*
+ * Count the number of lines needed to print the given table.
+ *
+ * cont: table data to be printed
+ * width_wrap[]: per-column maximum width, or NULL if caller will not wrap
+ * vertical: vertical mode?
+ *
+ * The result is currently only fully accurate for ALIGNED and UNALIGNED
+ * formats; otherwise it's an approximation.
+ *
+ * Note: while cont->opt will tell us most formatting details, we need the
+ * separate "vertical" flag because of the possibility of a dynamic switch
+ * from aligned_text to aligned_vertical format.
+ */
+static int
+count_table_lines(const printTableContent *cont,
+ const unsigned int *width_wrap,
+ bool vertical)
+{
+ int *header_height;
+ int lines = 0,
+ max_lines = 0,
+ nl_lines,
+ i;
+ int encoding = cont->opt->encoding;
+ const char *const *cell;
+
+ /*
+ * Scan all column headers and determine their heights. Cache the values
+ * since vertical mode repeats the headers for every record.
+ */
+ header_height = (int *) pg_malloc(cont->ncolumns * sizeof(int));
+ for (i = 0; i < cont->ncolumns; i++)
+ {
+ pg_wcssize((const unsigned char *) cont->headers[i],
+ strlen(cont->headers[i]), encoding,
+ NULL, &header_height[i], NULL);
+ }
+
+ /*
+ * Account for separator lines (if used), as well as the trailing blank
+ * line that most formats emit.
+ */
+ switch (cont->opt->format)
+ {
+ case PRINT_ALIGNED:
+ case PRINT_WRAPPED:
/*
- * FIXME -- this is slightly bogus: it counts the number of
- * footers, not the number of lines in them.
+ * Vertical mode writes one separator line per record. Normal
+ * mode writes a single separator line between header and rows.
*/
- for (f = cont->footers; f; f = f->next)
- lines++;
+ lines = vertical ? cont->nrows : 1;
+ /* Both modes add a blank line at the end */
+ lines++;
+ break;
+ case PRINT_UNALIGNED:
+
+ /*
+ * Vertical mode writes a separator (here assumed to be a newline)
+ * between records. Normal mode writes nothing extra.
+ */
+ if (vertical)
+ lines = Max(cont->nrows - 1, 0);
+ break;
+ case PRINT_CSV:
+ /* Nothing extra is added */
+ break;
+ case PRINT_HTML:
+ case PRINT_ASCIIDOC:
+ case PRINT_LATEX:
+ case PRINT_LATEX_LONGTABLE:
+ case PRINT_TROFF_MS:
+
+ /*
+ * These formats aren't really meant for interactive consumption,
+ * so for now we won't work hard on them. Treat them like aligned
+ * mode.
+ */
+ lines = vertical ? cont->nrows : 1;
+ lines++;
+ break;
+ case PRINT_NOTHING:
+ /* Shouldn't get here... */
+ break;
+ }
+
+ /* Scan all cells to count their lines */
+ for (i = 0, cell = cont->cells; *cell; cell++)
+ {
+ int width;
+
+ /* Count the original line breaks */
+ pg_wcssize((const unsigned char *) *cell, strlen(*cell), encoding,
+ &width, &nl_lines, NULL);
+
+ /* Count extra lines due to wrapping */
+ if (width > 0 && width_wrap && width_wrap[i])
+ nl_lines += (width - 1) / width_wrap[i];
+
+ if (vertical)
+ {
+ /* Pick the height of the header or cell, whichever is taller */
+ if (nl_lines > header_height[i])
+ lines += nl_lines;
+ else
+ lines += header_height[i];
+ }
+ else
+ {
+ /* Remember max height in the current row */
+ if (nl_lines > max_lines)
+ max_lines = nl_lines;
}
- *fout = PageOutput(lines + extra_lines, cont->opt);
- *is_pager = (*fout != stdout);
+ /* i is the current column number: increment with wrap */
+ if (++i >= cont->ncolumns)
+ {
+ i = 0;
+ if (!vertical)
+ {
+ /* At last column of each row, add tallest column height */
+ lines += max_lines;
+ max_lines = 0;
+ }
+ }
}
- else
- *is_pager = false;
+
+ /* Account for header and footer decoration */
+ if (!cont->opt->tuples_only)
+ {
+ printTableFooter *f;
+
+ if (cont->title)
+ {
+ /* Add height of title */
+ pg_wcssize((const unsigned char *) cont->title, strlen(cont->title),
+ encoding, NULL, &nl_lines, NULL);
+ lines += nl_lines;
+ }
+
+ if (!vertical)
+ {
+ /* Add height of tallest header column */
+ max_lines = 0;
+ for (i = 0; i < cont->ncolumns; i++)
+ {
+ if (header_height[i] > max_lines)
+ max_lines = header_height[i];
+ }
+ lines += max_lines;
+ }
+
+ /*
+ * Add all footer lines. Vertical mode does not use the default
+ * footer, but we must include that in normal mode.
+ */
+ for (f = vertical ? cont->footers : footers_with_default(cont);
+ f != NULL; f = f->next)
+ {
+ pg_wcssize((const unsigned char *) f->data, strlen(f->data),
+ encoding, NULL, &nl_lines, NULL);
+ lines += nl_lines;
+ }
+ }
+
+ free(header_height);
+
+ return lines;
}
/*
@@ -3456,7 +3591,7 @@ printTable(const printTableContent *cont,
cont->opt->format != PRINT_ALIGNED &&
cont->opt->format != PRINT_WRAPPED)
{
- IsPagerNeeded(cont, 0, (cont->opt->expanded == 1), &fout, &is_pager);
+ IsPagerNeeded(cont, NULL, (cont->opt->expanded == 1), &fout, &is_pager);
is_local_pager = is_pager;
}
@@ -3464,10 +3599,6 @@ printTable(const printTableContent *cont,
clearerr(fout);
/* print the stuff */
-
- if (flog)
- print_aligned_text(cont, flog, false);
-
switch (cont->opt->format)
{
case PRINT_UNALIGNED:
@@ -3534,6 +3665,10 @@ printTable(const printTableContent *cont,
if (is_local_pager)
ClosePager(fout);
+
+ /* also produce log output if wanted */
+ if (flog)
+ print_aligned_text(cont, flog, false);
}
/*
--
2.43.7
v6-0002-Make-some-minor-performance-improvements-in-psql-.patchtext/x-diff; charset=us-ascii; name*0=v6-0002-Make-some-minor-performance-improvements-in-psql-.p; name*1=atchDownload
From 8fbec4b6c72bdf4d6e1071578c7c5a9beb08a5c8 Mon Sep 17 00:00:00 2001
From: Tom Lane <tgl@sss.pgh.pa.us>
Date: Mon, 6 Oct 2025 18:36:56 -0400
Subject: [PATCH v6 2/3] Make some minor performance improvements in psql's
line-counting.
Arrange to not run count_table_lines at all unless we will use
its result, and teach it to quit early as soon as it's proven
that the output is long enough to require use of the pager.
When dealing with large tables this should save a noticeable
amount of time, since pg_wcssize() isn't exactly cheap.
In passing, fix print_aligned_text to avoid using repeated
"i % col_count" when it could just make the value of i wrap
around. In a quick check with the version of gcc I'm using,
the compiler is smart enough to recognize the common subexpressions
and issue only one integer divide per loop, but it's not smart enough
to avoid the integer divide altogether. In any case, this coding
was randomly unlike the way it's done in count_table_lines.
Author: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/2dd2430f-dd20-4c89-97fd-242616a3d768@ewie.name
---
src/fe_utils/print.c | 108 +++++++++++++++++++++++++++++++++----------
1 file changed, 83 insertions(+), 25 deletions(-)
diff --git a/src/fe_utils/print.c b/src/fe_utils/print.c
index 5082f0808b1..ea7483b0516 100644
--- a/src/fe_utils/print.c
+++ b/src/fe_utils/print.c
@@ -33,6 +33,11 @@
#include "fe_utils/mbprint.h"
#include "fe_utils/print.h"
+/* Presently, count_table_lines() is only used within #ifdef TIOCGWINSZ */
+#ifdef TIOCGWINSZ
+#define NEED_COUNT_TABLE_LINES
+#endif
+
/*
* If the calling program doesn't have any mechanism for setting
* cancel_pressed, it will have no effect.
@@ -266,14 +271,20 @@ static const unicodeStyleFormat unicode_style = {
/* Local functions */
static int strlen_max_width(unsigned char *str, int *target_width, int encoding);
+static FILE *PageOutputInternal(int lines, const printTableOpt *topt,
+ const printTableContent *cont,
+ const unsigned int *width_wrap,
+ bool vertical);
static void IsPagerNeeded(const printTableContent *cont,
const unsigned int *width_wrap,
bool vertical,
FILE **fout, bool *is_pager);
+#ifdef NEED_COUNT_TABLE_LINES
static int count_table_lines(const printTableContent *cont,
const unsigned int *width_wrap,
- bool vertical);
-
+ bool vertical,
+ int threshold);
+#endif
static void print_aligned_vertical(const printTableContent *cont,
FILE *fout, bool is_pager);
@@ -730,7 +741,7 @@ print_aligned_text(const printTableContent *cont, FILE *fout, bool is_pager)
}
/* scan all cells, find maximum width, compute cell_count */
- for (i = 0, ptr = cont->cells; *ptr; ptr++, i++, cell_count++)
+ for (i = 0, ptr = cont->cells; *ptr; ptr++, cell_count++)
{
int width,
nl_lines,
@@ -739,14 +750,18 @@ print_aligned_text(const printTableContent *cont, FILE *fout, bool is_pager)
pg_wcssize((const unsigned char *) *ptr, strlen(*ptr), encoding,
&width, &nl_lines, &bytes_required);
- if (width > max_width[i % col_count])
- max_width[i % col_count] = width;
- if (nl_lines > max_nl_lines[i % col_count])
- max_nl_lines[i % col_count] = nl_lines;
- if (bytes_required > max_bytes[i % col_count])
- max_bytes[i % col_count] = bytes_required;
+ if (width > max_width[i])
+ max_width[i] = width;
+ if (nl_lines > max_nl_lines[i])
+ max_nl_lines[i] = nl_lines;
+ if (bytes_required > max_bytes[i])
+ max_bytes[i] = bytes_required;
+
+ width_average[i] += width;
- width_average[i % col_count] += width;
+ /* i is the current column number: increment with wrap */
+ if (++i >= col_count)
+ i = 0;
}
/* If we have rows, compute average */
@@ -3051,28 +3066,62 @@ set_sigpipe_trap_state(bool ignore)
/*
* PageOutput
*
- * Tests if pager is needed and returns appropriate FILE pointer.
+ * Tests if pager is needed and returns appropriate FILE pointer
+ * (either a pipe, or stdout if we don't need the pager).
+ *
+ * lines: number of lines that will be printed
+ * topt: print formatting options
*
* If the topt argument is NULL no pager is used.
*/
FILE *
PageOutput(int lines, const printTableOpt *topt)
+{
+ return PageOutputInternal(lines, topt, NULL, NULL, false);
+}
+
+/*
+ * Private version that allows for line-counting to be avoided when
+ * not needed. If "cont" is not null then the input value of "lines"
+ * is ignored and we count lines based on cont + width_wrap + vertical
+ * (see count_table_lines).
+ */
+static FILE *
+PageOutputInternal(int lines, const printTableOpt *topt,
+ const printTableContent *cont,
+ const unsigned int *width_wrap,
+ bool vertical)
{
/* check whether we need / can / are supposed to use pager */
if (topt && topt->pager && isatty(fileno(stdin)) && isatty(fileno(stdout)))
{
+ /* without TIOCGWINSZ, pager == 1 acts the same as pager > 1 */
#ifdef TIOCGWINSZ
unsigned short int pager = topt->pager;
int min_lines = topt->pager_min_lines;
- int result;
- struct winsize screen_size;
- result = ioctl(fileno(stdout), TIOCGWINSZ, &screen_size);
+ if (pager == 1)
+ {
+ int result;
+ struct winsize screen_size;
+
+ result = ioctl(fileno(stdout), TIOCGWINSZ, &screen_size);
+ if (result < 0)
+ pager = 2; /* force use of pager */
+ else
+ {
+ int threshold = Max(screen_size.ws_row, min_lines);
+
+ if (cont) /* caller wants us to calculate lines */
+ lines = count_table_lines(cont, width_wrap, vertical,
+ threshold);
+ /* >= accounts for a one-line prompt */
+ if (lines >= threshold)
+ pager = 2;
+ }
+ }
- /* >= accounts for a one-line prompt */
- if (result == -1
- || (lines >= screen_size.ws_row && lines >= min_lines)
- || pager > 1)
+ if (pager > 1)
#endif
{
const char *pagerprog;
@@ -3384,11 +3433,7 @@ IsPagerNeeded(const printTableContent *cont, const unsigned int *width_wrap,
{
if (*fout == stdout)
{
- int lines;
-
- lines = count_table_lines(cont, width_wrap, vertical);
-
- *fout = PageOutput(lines, cont->opt);
+ *fout = PageOutputInternal(0, cont->opt, cont, width_wrap, vertical);
*is_pager = (*fout != stdout);
}
else
@@ -3401,6 +3446,7 @@ IsPagerNeeded(const printTableContent *cont, const unsigned int *width_wrap,
* cont: table data to be printed
* width_wrap[]: per-column maximum width, or NULL if caller will not wrap
* vertical: vertical mode?
+ * threshold: we can stop counting once we pass this many lines
*
* The result is currently only fully accurate for ALIGNED and UNALIGNED
* formats; otherwise it's an approximation.
@@ -3408,11 +3454,16 @@ IsPagerNeeded(const printTableContent *cont, const unsigned int *width_wrap,
* Note: while cont->opt will tell us most formatting details, we need the
* separate "vertical" flag because of the possibility of a dynamic switch
* from aligned_text to aligned_vertical format.
+ *
+ * The value of the threshold parameter is that when the table is very
+ * long, we'll typically be able to stop scanning after not many rows.
*/
+#ifdef NEED_COUNT_TABLE_LINES
static int
count_table_lines(const printTableContent *cont,
const unsigned int *width_wrap,
- bool vertical)
+ bool vertical,
+ int threshold)
{
int *header_height;
int lines = 0,
@@ -3520,11 +3571,14 @@ count_table_lines(const printTableContent *cont,
lines += max_lines;
max_lines = 0;
}
+ /* Stop scanning table body once we pass threshold */
+ if (lines > threshold)
+ break;
}
}
/* Account for header and footer decoration */
- if (!cont->opt->tuples_only)
+ if (!cont->opt->tuples_only && lines <= threshold)
{
printTableFooter *f;
@@ -3558,6 +3612,9 @@ count_table_lines(const printTableContent *cont,
pg_wcssize((const unsigned char *) f->data, strlen(f->data),
encoding, NULL, &nl_lines, NULL);
lines += nl_lines;
+ /* Stop scanning footers once we pass threshold */
+ if (lines > threshold)
+ break;
}
}
@@ -3565,6 +3622,7 @@ count_table_lines(const printTableContent *cont,
return lines;
}
+#endif /* NEED_COUNT_TABLE_LINES */
/*
* Use this to print any table in the supported formats.
--
2.43.7
v6-0003-Add-a-TAP-test-to-exercise-psql-s-use-of-the-page.patchtext/x-diff; charset=us-ascii; name*0=v6-0003-Add-a-TAP-test-to-exercise-psql-s-use-of-the-page.p; name*1=atchDownload
From dbd78ae258dbb09550b300b6b5453d536d3656ee Mon Sep 17 00:00:00 2001
From: Tom Lane <tgl@sss.pgh.pa.us>
Date: Mon, 6 Oct 2025 18:41:30 -0400
Subject: [PATCH v6 3/3] Add a TAP test to exercise psql's use of the pager.
Up to now, we have had zero test coverage of these code paths,
because they aren't reached unless isatty(stdout). We do have
the test infrastructure to improve that situation, though.
Following the lead of 010_tab_completion.pl, set up an
interactive psql and feed it some test cases.
To detect whether it really did invoke the pager, point
PSQL_PAGER to "wc -l". The test is skipped if that utility
isn't available.
Author: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/2dd2430f-dd20-4c89-97fd-242616a3d768@ewie.name
---
src/bin/psql/meson.build | 1 +
src/bin/psql/t/030_pager.pl | 104 ++++++++++++++++++++++++++++++++++++
2 files changed, 105 insertions(+)
create mode 100644 src/bin/psql/t/030_pager.pl
diff --git a/src/bin/psql/meson.build b/src/bin/psql/meson.build
index f795ff28271..d344053c23b 100644
--- a/src/bin/psql/meson.build
+++ b/src/bin/psql/meson.build
@@ -77,6 +77,7 @@ tests += {
't/001_basic.pl',
't/010_tab_completion.pl',
't/020_cancel.pl',
+ 't/030_pager.pl',
],
},
}
diff --git a/src/bin/psql/t/030_pager.pl b/src/bin/psql/t/030_pager.pl
new file mode 100644
index 00000000000..dd2be340e55
--- /dev/null
+++ b/src/bin/psql/t/030_pager.pl
@@ -0,0 +1,104 @@
+
+# Copyright (c) 2021-2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Data::Dumper;
+
+# If we don't have IO::Pty, forget it, because IPC::Run depends on that
+# to support pty connections
+eval { require IO::Pty; };
+if ($@)
+{
+ plan skip_all => 'IO::Pty is needed to run this test';
+}
+
+# Check that "wc -l" does what we expect, else forget it
+my $wcstdin = "foo bar\nbaz\n";
+my ($wcstdout, $wcstderr);
+my $result = IPC::Run::run [ 'wc', '-l' ],
+ '<' => \$wcstdin,
+ '>' => \$wcstdout,
+ '2>' => \$wcstderr;
+chomp $wcstdout;
+if ($wcstdout ne '2' || $wcstderr ne '')
+{
+ plan skip_all => '"wc -l" is needed to run this test';
+}
+
+# We set up "wc -l" as the pager so we can tell whether psql used the pager
+$ENV{PSQL_PAGER} = "wc -l";
+
+# start a new server
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init;
+$node->start;
+
+# fire up an interactive psql session
+my $h = $node->interactive_psql('postgres');
+
+# set the pty's window size to known values
+# (requires undesirable chumminess with the innards of IPC::Run)
+for my $pty (values %{ $h->{run}->{PTYS} })
+{
+ $pty->set_winsize(24, 80);
+}
+
+# Simple test case: type something and see if psql responds as expected
+sub do_command
+{
+ my ($send, $pattern, $annotation) = @_;
+
+ # report test failures from caller location
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+
+ # restart per-command timer
+ $h->{timeout}->start($PostgreSQL::Test::Utils::timeout_default);
+
+ # send the data to be sent and wait for its result
+ my $out = $h->query_until($pattern, $send);
+ my $okay = ($out =~ $pattern && !$h->{timeout}->is_expired);
+ ok($okay, $annotation);
+ # for debugging, log actual output if it didn't match
+ local $Data::Dumper::Terse = 1;
+ local $Data::Dumper::Useqq = 1;
+ diag 'Actual output was ' . Dumper($out) . "Did not match \"$pattern\"\n"
+ if !$okay;
+ return;
+}
+
+# Test invocation of the pager
+#
+# Note that interactive_psql starts psql with --no-align --tuples-only,
+# and that the output string will include psql's prompts and command echo.
+
+do_command(
+ "SELECT 'test' AS t FROM generate_series(1,23);\n",
+ qr/^test\r?$/m,
+ "execute SELECT query that needs no pagination");
+
+do_command(
+ "SELECT 'test' AS t FROM generate_series(1,24);\n",
+ qr/^24\r?$/m,
+ "execute SELECT query that needs pagination");
+
+do_command(
+ "\\pset expanded\nSELECT generate_series(1,20) as g;\n",
+ qr/^39\r?$/m,
+ "execute SELECT query that needs pagination in expanded mode");
+
+do_command(
+ "\\pset tuples_only off\n\\d+ information_schema.referential_constraints\n",
+ qr/^\d+\r?$/m,
+ "execute command with footer that needs pagination");
+
+# send psql an explicit \q to shut it down, else pty won't close properly
+$h->quit or die "psql returned $?";
+
+# done
+$node->stop;
+done_testing();
--
2.43.7
On 2025-10-07 00:45 +0200, Tom Lane wrote:
I wrote:
Another inaccuracy that I find mildly annoying now that we've got it
mostly right is that we are not accounting for the blank line that
psql likes to print after the query result, so that you can still
end with the first output line scrolled off your screen despite
non-use of the pager. Not sure about a non-kluge way to count that,
though: it's outside the domain of print.c, and other callers might
not have a similar behavior.Oh wait! That line *is* produced within print.c, so there is no
reason we shouldn't account for it. I spent a little extra effort
on unaligned mode too, since the TAP test is checking that case.
LGTM.
I still doubt that anyone's going to care a lot about the other
formats.
Yup, I think so too.
--
Erik Wienhold
Erik Wienhold <ewie@ewie.name> writes:
On 2025-10-07 00:45 +0200, Tom Lane wrote:
Oh wait! That line *is* produced within print.c, so there is no
reason we shouldn't account for it. I spent a little extra effort
on unaligned mode too, since the TAP test is checking that case.
LGTM.
Pushed, after a tiny bit more comment-burnishing.
I still doubt that anyone's going to care a lot about the other
formats.
Yup, I think so too.
There's always room for a follow-on patch from someone who does
care.
regards, tom lane
On 2025-10-07 17:01 +0200, Tom Lane wrote:
Pushed, after a tiny bit more comment-burnishing.
Thank you for the extensive rework.
--
Erik Wienhold
Dear Team,
I worked on the pager issue in print.c and made the following updates:
-
Added a linecount() helper to correctly count all lines, including
footers.
-
In IsPagerNeeded , replaced the old logic lines++ with lines +=
linecount(f->data).
With this fix, the pager now triggers correctly for long outputs.
Please let me know if this way of fixing looks good, or if you have any
suggestions for improvement.
Best regards,
[lakshmi G]
On Wed, Oct 8, 2025 at 2:40 PM Erik Wienhold <ewie@ewie.name> wrote:
Show quoted text
On 2025-10-07 17:01 +0200, Tom Lane wrote:
Pushed, after a tiny bit more comment-burnishing.
Thank you for the extensive rework.
--
Erik Wienhold
Attachments:
0001-Fix-pager-trigger-in-print.c-by-using-linecount-help.patchtext/x-patch; charset=US-ASCII; name=0001-Fix-pager-trigger-in-print.c-by-using-linecount-help.patchDownload
From f903ecc2aeb820f461d9caf06ac3c9731e81aff1 Mon Sep 17 00:00:00 2001
From: Lakshmi <bharatdbpg@gmail.com>
Date: Fri, 10 Oct 2025 11:57:05 +0530
Subject: [PATCH] Fix pager trigger in print.c by using linecount() helper for
accurate line calculation
Signed-off-by: Lakshmi <bharatdbpg@gmail.com>
---
src/fe_utils/print.c | 22 +++++++++++++++++++++-
1 file changed, 21 insertions(+), 1 deletion(-)
diff --git a/src/fe_utils/print.c b/src/fe_utils/print.c
index 4af0f32f2f..43dcd3a87a 100644
--- a/src/fe_utils/print.c
+++ b/src/fe_utils/print.c
@@ -266,6 +266,25 @@ static const unicodeStyleFormat unicode_style = {
/* Local functions */
static int strlen_max_width(unsigned char *str, int *target_width, int encoding);
+/* >>> INSERT helper function here <<< */
+static int
+linecount(const char *s)
+{
+ int count = 0;
+
+ if (s == NULL || *s == '\0')
+ return 0;
+
+ while (*s)
+ {
+ if (*s == '\n')
+ count++;
+ s++;
+ }
+ return count + 1;
+}
+
+
static void IsPagerNeeded(const printTableContent *cont, int extra_lines, bool expanded,
FILE **fout, bool *is_pager);
@@ -3421,7 +3440,8 @@ IsPagerNeeded(const printTableContent *cont, int extra_lines, bool expanded,
* footers, not the number of lines in them.
*/
for (f = cont->footers; f; f = f->next)
- lines++;
+ lines += linecount(f->data);
+
}
*fout = PageOutput(lines + extra_lines, cont->opt);
--
2.39.5