Fixing pgbench init overflow
Hello,
pgbench mixes int and int64 to initialize the tables.
When a large enough scale factor is passed, initPopulateTable
overflows leading to it never completing, ie.
2147400000 of 2200000000 tuples (97%) of
pgbench_accounts done (elapsed 4038.83 s, remaining 98.93 s)
-2147400000 of 2200000000 tuples (-97%) of
pgbench_accounts done (elapsed 4038.97 s, remaining -8176.86 s)
Attached is a patch that fixes this, pgbench -i -s 22000 works now.
--
John Hsu - Amazon Web Services
Attachments:
0001-Fix-pgbench-init-overflow-when-total-number-of-rows.patchapplication/octet-stream; name=0001-Fix-pgbench-init-overflow-when-total-number-of-rows.patchDownload
From 866e76a178234f34799ec706e26c43cb19d3460f Mon Sep 17 00:00:00 2001
From: John Hsu <johnhyvr@gmail.com>
Date: Fri, 22 Dec 2023 22:38:15 +0000
Subject: [PATCH] Fix pgbench init overflow when total number of rows is
greater than int32
Previously when the number of rows exceeds int32, it would
result in overflow and pgbench never finishing. This
change moves towards int64 to align with the comparison.
---
src/bin/pgbench/pgbench.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 2e1650d0ad..dbdb486b40 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -4907,8 +4907,8 @@ static void
initPopulateTable(PGconn *con, const char *table, int64 base,
initRowMethod init_row)
{
- int n;
- int k;
+ int64 n;
+ int64 k;
int chars = 0;
PGresult *res;
PQExpBufferData sql;
@@ -4938,7 +4938,7 @@ initPopulateTable(PGconn *con, const char *table, int64 base,
n = pg_snprintf(copy_statement, sizeof(copy_statement), copy_statement_fmt, table);
if (n >= sizeof(copy_statement))
- pg_fatal("invalid buffer size: must be at least %d characters long", n);
+ pg_fatal("invalid buffer size: must be at least " INT64_FORMAT " characters long", n);
else if (n == -1)
pg_fatal("invalid format string");
--
2.39.3
On Sat, 23 Dec 2023 at 07:18, Chen Hao Hsu <johnhyvr@gmail.com> wrote:
Hello,
pgbench mixes int and int64 to initialize the tables.
When a large enough scale factor is passed, initPopulateTable
overflows leading to it never completing, ie.2147400000 of 2200000000 tuples (97%) of
pgbench_accounts done (elapsed 4038.83 s, remaining 98.93 s)
-2147400000 of 2200000000 tuples (-97%) of
pgbench_accounts done (elapsed 4038.97 s, remaining -8176.86 s)Attached is a patch that fixes this, pgbench -i -s 22000 works now.
I think only the following line can fix this.
+ int64 k;
Do not need to modify the type of `n`, right?
--
Regrads,
Japin Li
ChengDu WenWu Information Technology Co., Ltd.
<ME3P282MB316684190982F54BDBD4FE90B69BA@ME3P282MB3166.AUSP282.PROD.OUTLOOK.COM>
On Sat, 23 Dec 2023 at 07:18, Chen Hao Hsu <johnhyvr@gmail.com> wrote:
Hello,
pgbench mixes int and int64 to initialize the tables.
When a large enough scale factor is passed, initPopulateTable
overflows leading to it never completing, ie.2147400000 of 2200000000 tuples (97%) of
pgbench_accounts done (elapsed 4038.83 s, remaining 98.93 s)
-2147400000 of 2200000000 tuples (-97%) of
pgbench_accounts done (elapsed 4038.97 s, remaining -8176.86 s)Attached is a patch that fixes this, pgbench -i -s 22000 works now.
I think only the following line can fix this.
+ int64 k;
Do not need to modify the type of `n`, right?
You are right. n represents the return value of pg_snprintf, which is
the byte length of the formatted data, which is int, not int64.
Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp
On Sat, 23 Dec 2023 at 15:22, Tatsuo Ishii <ishii@sraoss.co.jp> wrote:
<ME3P282MB316684190982F54BDBD4FE90B69BA@ME3P282MB3166.AUSP282.PROD.OUTLOOK.COM>
On Sat, 23 Dec 2023 at 07:18, Chen Hao Hsu <johnhyvr@gmail.com> wrote:
Hello,
pgbench mixes int and int64 to initialize the tables.
When a large enough scale factor is passed, initPopulateTable
overflows leading to it never completing, ie.2147400000 of 2200000000 tuples (97%) of
pgbench_accounts done (elapsed 4038.83 s, remaining 98.93 s)
-2147400000 of 2200000000 tuples (-97%) of
pgbench_accounts done (elapsed 4038.97 s, remaining -8176.86 s)Attached is a patch that fixes this, pgbench -i -s 22000 works now.
I think only the following line can fix this.
+ int64 k;
Do not need to modify the type of `n`, right?
You are right. n represents the return value of pg_snprintf, which is
the byte length of the formatted data, which is int, not int64.
Thanks for you confirmation! Please consider the v2 patch to review.
--
Regrads,
Japin Li
ChengDu WenWu Information Technology Co., Ltd.
Attachments:
v2-0001-Fix-pgbench-init-overflow-when-total-number-of-ro.patchtext/x-diffDownload
From cee64786a2353cbf77d67db416ba0596a461690b Mon Sep 17 00:00:00 2001
From: John Hsu <johnhyvr@gmail.com>
Date: Fri, 22 Dec 2023 22:38:15 +0000
Subject: [PATCH v2 1/1] Fix pgbench init overflow when total number of rows is
greater than int32
Previously when the number of rows exceeds int32, it would
result in overflow and pgbench never finishing. This
change moves towards int64 to align with the comparison.
---
src/bin/pgbench/pgbench.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 2e1650d0ad..5b6549c89d 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -4908,7 +4908,7 @@ initPopulateTable(PGconn *con, const char *table, int64 base,
initRowMethod init_row)
{
int n;
- int k;
+ int64 k;
int chars = 0;
PGresult *res;
PQExpBufferData sql;
base-commit: 3e2e0d5ad7fcb89d18a71cbfc885ef184e1b6f2e
--
2.41.0
