The number of character limitation of custom script on pgbench
Hi all,
The function of custom script of pgbench allows only BUFSIZ
(i.g.,1024byte) or less as length of a SQL.
I think that when we want to bench mark with long SQL then it will difficult.
At that time even pgbench doesn't return ERROR. It will try to do
query with the broken SQL.
And user can not know why function of custom script is not work fine.
It look as just error of SQL to user.
So I'm thinking following solution.
(1) to increase buffer size
(2) to change to variable buffer size
(3) to return ERROR with information
Thought?
Regards,
-------
Sawada Masahiko
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 11/13/13, 6:18 AM, Sawada Masahiko wrote:
Hi all,
The function of custom script of pgbench allows only BUFSIZ
(i.g.,1024byte) or less as length of a SQL.
I think that when we want to bench mark with long SQL then it will difficult.
At that time even pgbench doesn't return ERROR. It will try to do
query with the broken SQL.
And user can not know why function of custom script is not work fine.
It look as just error of SQL to user.So I'm thinking following solution.
(1) to increase buffer size
(2) to change to variable buffer size
(3) to return ERROR with information
I'd go for #2. But at least an error.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Peter Eisentraut <peter_e@gmx.net> writes:
On 11/13/13, 6:18 AM, Sawada Masahiko wrote:
The function of custom script of pgbench allows only BUFSIZ
(i.g.,1024byte) or less as length of a SQL.
So I'm thinking following solution.
(1) to increase buffer size
(2) to change to variable buffer size
(3) to return ERROR with information
I'd go for #2. But at least an error.
#2 definitely. I've run into this limitation myself recently, and
so have other people. It's time to fix it.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Wed, Nov 13, 2013 at 10:27 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Peter Eisentraut <peter_e@gmx.net> writes:
On 11/13/13, 6:18 AM, Sawada Masahiko wrote:
The function of custom script of pgbench allows only BUFSIZ
(i.g.,1024byte) or less as length of a SQL.
So I'm thinking following solution.
(1) to increase buffer size
(2) to change to variable buffer size
(3) to return ERROR with informationI'd go for #2. But at least an error.
#2 definitely. I've run into this limitation myself recently, and
so have other people. It's time to fix it.
Yes, I also think #2 is good.
I will implement the patch.
Regards,
-------
Sawada Masahiko
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Sawada Masahiko escribi�:
Yes, I also think #2 is good.
I will implement the patch.
I remember this was recently discussed in the spanish list. Please see
this email:
http://archives.postgresql.org/message-id/48589.192.168.207.54.1382570043.squirrel@webmail.etecsa.cu
--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Wed, Nov 13, 2013 at 10:27 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Peter Eisentraut <peter_e@gmx.net> writes:
On 11/13/13, 6:18 AM, Sawada Masahiko wrote:
The function of custom script of pgbench allows only BUFSIZ
(i.g.,1024byte) or less as length of a SQL.
So I'm thinking following solution.
(1) to increase buffer size
(2) to change to variable buffer size
(3) to return ERROR with informationI'd go for #2. But at least an error.
#2 definitely. I've run into this limitation myself recently, and
so have other people. It's time to fix it.
I attached the patch which solves this problem, and have submitted to CF3.
I changed how to get the SQL from custom script file.
Regards,
-------
Sawada Masahiko
Attachments:
fix_limitaion_custom_script_pgbench.patchapplication/octet-stream; name=fix_limitaion_custom_script_pgbench.patchDownload
*** a/contrib/pgbench/pgbench.c
--- b/contrib/pgbench/pgbench.c
***************
*** 2016,2021 **** process_commands(char *buf)
--- 2016,2049 ----
return my_commands;
}
+ static char*
+ read_sql(FILE *fd)
+ {
+ int len = BUFSIZ;
+ char tmpbuf[BUFSIZ];
+ char *buf = NULL;
+
+ buf = (char *)palloc(BUFSIZ);
+
+ while(fgets(tmpbuf, BUFSIZ, fd) != NULL)
+ {
+
+ buf = strncat(buf, tmpbuf, BUFSIZ);
+
+ /* Checking wether have read a line which include new line character */
+ if (strchr(tmpbuf, '\n') != NULL)
+ {
+ return buf;
+ }
+
+ len += BUFSIZ;
+
+ buf = (char *)pg_realloc(buf, len);
+ }
+
+ return NULL;
+ }
+
static int
process_file(char *filename)
{
***************
*** 2024,2030 **** process_file(char *filename)
Command **my_commands;
FILE *fd;
int lineno;
! char buf[BUFSIZ];
int alloc_num;
if (num_files >= MAX_FILES)
--- 2052,2058 ----
Command **my_commands;
FILE *fd;
int lineno;
! char *buf = NULL;
int alloc_num;
if (num_files >= MAX_FILES)
***************
*** 2046,2055 **** process_file(char *filename)
lineno = 0;
! while (fgets(buf, sizeof(buf), fd) != NULL)
{
Command *command;
command = process_commands(buf);
if (command == NULL)
continue;
--- 2074,2085 ----
lineno = 0;
! while( (buf = read_sql(fd)) != NULL)
{
Command *command;
+ if (buf == NULL) break;
+
command = process_commands(buf);
if (command == NULL)
continue;
Sawada Masahiko <sawada.mshk@gmail.com> writes:
I attached the patch which solves this problem, and have submitted to CF3.
I changed how to get the SQL from custom script file.
This needed a bit of work:
- Use of strncat didn't seem particularly safe, or efficient. I changed
it to explicitly account for space consumption in the result buffer.
- It leaked the buffer space used. While this likely doesn't matter for
foreseeable usage, it seemed worth fixing.
- Didn't do the right thing for a file not ending with a newline.
- Didn't follow project code layout standards.
I've committed the attached revised version.
regards, tom lane
Attachments:
pgbench-line-length-fix-2.patchtext/x-diff; charset=us-ascii; name=pgbench-line-length-fix-2.patchDownload
diff --git a/contrib/pgbench/pgbench.c b/contrib/pgbench/pgbench.c
index fff71e5..2c96fae 100644
*** a/contrib/pgbench/pgbench.c
--- b/contrib/pgbench/pgbench.c
*************** process_commands(char *buf)
*** 2016,2021 ****
--- 2016,2064 ----
return my_commands;
}
+ /*
+ * Read a line from fd, and return it in a malloc'd buffer.
+ * Return NULL at EOF.
+ *
+ * The buffer will typically be larger than necessary, but we don't care
+ * in this program, because we'll free it as soon as we've parsed the line.
+ */
+ static char *
+ read_line_from_file(FILE *fd)
+ {
+ char tmpbuf[BUFSIZ];
+ char *buf;
+ size_t buflen = BUFSIZ;
+ size_t used = 0;
+
+ buf = (char *) palloc(buflen);
+ buf[0] = '\0';
+
+ while (fgets(tmpbuf, BUFSIZ, fd) != NULL)
+ {
+ size_t thislen = strlen(tmpbuf);
+
+ /* Append tmpbuf to whatever we had already */
+ memcpy(buf + used, tmpbuf, thislen + 1);
+ used += thislen;
+
+ /* Done if we collected a newline */
+ if (thislen > 0 && tmpbuf[thislen - 1] == '\n')
+ break;
+
+ /* Else, enlarge buf to ensure we can append next bufferload */
+ buflen += BUFSIZ;
+ buf = (char *) pg_realloc(buf, buflen);
+ }
+
+ if (used > 0)
+ return buf;
+
+ /* Reached EOF */
+ free(buf);
+ return NULL;
+ }
+
static int
process_file(char *filename)
{
*************** process_file(char *filename)
*** 2024,2030 ****
Command **my_commands;
FILE *fd;
int lineno;
! char buf[BUFSIZ];
int alloc_num;
if (num_files >= MAX_FILES)
--- 2067,2073 ----
Command **my_commands;
FILE *fd;
int lineno;
! char *buf;
int alloc_num;
if (num_files >= MAX_FILES)
*************** process_file(char *filename)
*** 2046,2056 ****
lineno = 0;
! while (fgets(buf, sizeof(buf), fd) != NULL)
{
Command *command;
command = process_commands(buf);
if (command == NULL)
continue;
--- 2089,2102 ----
lineno = 0;
! while ((buf = read_line_from_file(fd)) != NULL)
{
Command *command;
command = process_commands(buf);
+
+ free(buf);
+
if (command == NULL)
continue;