Let psql process files with > 4,294,967,295 lines
Folks,
I just ran across an issue where in psql, people can get the line
number in the file so long as it is under 2^32-1 lines long, but once
it gets larger than that, it's hosed.
This patch changes the data type from unsigned int to unsigned long
long, which is probably not the correct thing in order to get 64-bit
arithmetic, but I figure it's good enough to get a discussion started.
Cheers,
D
--
David Fetter <david@fetter.org> http://fetter.org/
phone: +1 415 235 3778 AIM: dfetter666
Skype: davidfetter
Remember to vote!
Attachments:
psql_long_file.difftext/plain; charset=us-asciiDownload
Index: src/bin/psql/common.c
===================================================================
RCS file: /projects/cvsroot/pgsql/src/bin/psql/common.c,v
retrieving revision 1.122
diff -c -r1.122 common.c
*** src/bin/psql/common.c 14 Jul 2006 14:52:26 -0000 1.122
--- src/bin/psql/common.c 30 Jul 2006 21:08:39 -0000
***************
*** 188,194 ****
fflush(pset.queryFout);
if (pset.inputfile)
! fprintf(stderr, "%s:%s:%u: ", pset.progname, pset.inputfile, pset.lineno);
va_start(ap, fmt);
vfprintf(stderr, _(fmt), ap);
va_end(ap);
--- 188,194 ----
fflush(pset.queryFout);
if (pset.inputfile)
! fprintf(stderr, "%s:%s:%lld: ", pset.progname, pset.inputfile, pset.lineno);
va_start(ap, fmt);
vfprintf(stderr, _(fmt), ap);
va_end(ap);
Index: src/bin/psql/mainloop.c
===================================================================
RCS file: /projects/cvsroot/pgsql/src/bin/psql/mainloop.c,v
retrieving revision 1.81
diff -c -r1.81 mainloop.c
*** src/bin/psql/mainloop.c 14 Jul 2006 14:52:26 -0000 1.81
--- src/bin/psql/mainloop.c 30 Jul 2006 21:08:40 -0000
***************
*** 44,50 ****
/* Save the prior command source */
FILE *prev_cmd_source;
bool prev_cmd_interactive;
! unsigned int prev_lineno;
/* Save old settings */
prev_cmd_source = pset.cur_cmd_source;
--- 44,50 ----
/* Save the prior command source */
FILE *prev_cmd_source;
bool prev_cmd_interactive;
! unsigned long long prev_lineno;
/* Save old settings */
prev_cmd_source = pset.cur_cmd_source;
Index: src/bin/psql/settings.h
===================================================================
RCS file: /projects/cvsroot/pgsql/src/bin/psql/settings.h,v
retrieving revision 1.27
diff -c -r1.27 settings.h
*** src/bin/psql/settings.h 5 Mar 2006 15:58:52 -0000 1.27
--- src/bin/psql/settings.h 30 Jul 2006 21:08:40 -0000
***************
*** 50,56 ****
char *inputfile; /* for error reporting */
char *dirname; /* current directory for \s display */
! unsigned lineno; /* also for error reporting */
bool timing; /* enable timing of all queries */
--- 50,56 ----
char *inputfile; /* for error reporting */
char *dirname; /* current directory for \s display */
! unsigned long long lineno; /* also for error reporting */
bool timing; /* enable timing of all queries */
David Fetter wrote:
Hi,
I just ran across an issue where in psql, people can get the line
number in the file so long as it is under 2^32-1 lines long, but once
it gets larger than that, it's hosed.This patch changes the data type from unsigned int to unsigned long
long, which is probably not the correct thing in order to get 64-bit
arithmetic, but I figure it's good enough to get a discussion started.
The only thing I can tell you is that you should use INT64_FORMAT
instead of %lld.
--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support
Alvaro Herrera <alvherre@commandprompt.com> writes:
David Fetter wrote:
This patch changes the data type from unsigned int to unsigned long
long, which is probably not the correct thing in order to get 64-bit
arithmetic, but I figure it's good enough to get a discussion started.
The only thing I can tell you is that you should use INT64_FORMAT
instead of %lld.
And the datatype should be declared int64, not "long long" which doesn't
exist everywhere.
Actually you probably want uint64 and UINT64_FORMAT...
regards, tom lane
On Sun, Jul 30, 2006 at 05:40:16PM -0400, Tom Lane wrote:
Alvaro Herrera <alvherre@commandprompt.com> writes:
David Fetter wrote:
This patch changes the data type from unsigned int to unsigned
long long, which is probably not the correct thing in order to
get 64-bit arithmetic, but I figure it's good enough to get a
discussion started.The only thing I can tell you is that you should use INT64_FORMAT
instead of %lld.And the datatype should be declared int64, not "long long" which
doesn't exist everywhere.Actually you probably want uint64 and UINT64_FORMAT...
regards, tom lane
I think this fixes it, but I'm unsure how to test it. Two of the
methods mentioned in IRC, attaching with gdb and setting to a value >
2^32, and setting it directly in some code, seem like OK approaches.
Cheers,
D
--
David Fetter <david@fetter.org> http://fetter.org/
phone: +1 415 235 3778 AIM: dfetter666
Skype: davidfetter
Remember to vote!
Attachments:
psql_long_file.difftext/plain; charset=us-asciiDownload
Index: src/bin/psql/common.c
===================================================================
RCS file: /projects/cvsroot/pgsql/src/bin/psql/common.c,v
retrieving revision 1.122
diff -c -r1.122 common.c
*** src/bin/psql/common.c 14 Jul 2006 14:52:26 -0000 1.122
--- src/bin/psql/common.c 31 Jul 2006 01:57:42 -0000
***************
*** 188,194 ****
fflush(pset.queryFout);
if (pset.inputfile)
! fprintf(stderr, "%s:%s:%u: ", pset.progname, pset.inputfile, pset.lineno);
va_start(ap, fmt);
vfprintf(stderr, _(fmt), ap);
va_end(ap);
--- 188,194 ----
fflush(pset.queryFout);
if (pset.inputfile)
! fprintf(stderr, "%s:%s:" UINT64_FORMAT ": ", pset.progname, pset.inputfile, pset.lineno);
va_start(ap, fmt);
vfprintf(stderr, _(fmt), ap);
va_end(ap);
Index: src/bin/psql/mainloop.c
===================================================================
RCS file: /projects/cvsroot/pgsql/src/bin/psql/mainloop.c,v
retrieving revision 1.81
diff -c -r1.81 mainloop.c
*** src/bin/psql/mainloop.c 14 Jul 2006 14:52:26 -0000 1.81
--- src/bin/psql/mainloop.c 31 Jul 2006 01:57:42 -0000
***************
*** 44,50 ****
/* Save the prior command source */
FILE *prev_cmd_source;
bool prev_cmd_interactive;
! unsigned int prev_lineno;
/* Save old settings */
prev_cmd_source = pset.cur_cmd_source;
--- 44,50 ----
/* Save the prior command source */
FILE *prev_cmd_source;
bool prev_cmd_interactive;
! uint64 prev_lineno;
/* Save old settings */
prev_cmd_source = pset.cur_cmd_source;
Index: src/bin/psql/settings.h
===================================================================
RCS file: /projects/cvsroot/pgsql/src/bin/psql/settings.h,v
retrieving revision 1.27
diff -c -r1.27 settings.h
*** src/bin/psql/settings.h 5 Mar 2006 15:58:52 -0000 1.27
--- src/bin/psql/settings.h 31 Jul 2006 01:57:42 -0000
***************
*** 12,17 ****
--- 12,18 ----
#include "variables.h"
#include "print.h"
+ #include "pg_config.h"
#define DEFAULT_FIELD_SEP "|"
#define DEFAULT_RECORD_SEP "\n"
***************
*** 50,56 ****
char *inputfile; /* for error reporting */
char *dirname; /* current directory for \s display */
! unsigned lineno; /* also for error reporting */
bool timing; /* enable timing of all queries */
--- 51,57 ----
char *inputfile; /* for error reporting */
char *dirname; /* current directory for \s display */
! uint64 lineno; /* also for error reporting */
bool timing; /* enable timing of all queries */
David Fetter <david@fetter.org> writes:
+ #include "pg_config.h"
You should not need that. All PG code assumes that c.h and its
inclusions have already been read.
regards, tom lane
[ Tom's include adjustment added.]
Your patch has been added to the PostgreSQL unapplied patches list at:
http://momjian.postgresql.org/cgi-bin/pgpatches
It will be applied as soon as one of the PostgreSQL committers reviews
and approves it.
---------------------------------------------------------------------------
David Fetter wrote:
On Sun, Jul 30, 2006 at 05:40:16PM -0400, Tom Lane wrote:
Alvaro Herrera <alvherre@commandprompt.com> writes:
David Fetter wrote:
This patch changes the data type from unsigned int to unsigned
long long, which is probably not the correct thing in order to
get 64-bit arithmetic, but I figure it's good enough to get a
discussion started.The only thing I can tell you is that you should use INT64_FORMAT
instead of %lld.And the datatype should be declared int64, not "long long" which
doesn't exist everywhere.Actually you probably want uint64 and UINT64_FORMAT...
regards, tom lane
I think this fixes it, but I'm unsure how to test it. Two of the
methods mentioned in IRC, attaching with gdb and setting to a value >
2^32, and setting it directly in some code, seem like OK approaches.Cheers,
D
--
David Fetter <david@fetter.org> http://fetter.org/
phone: +1 415 235 3778 AIM: dfetter666
Skype: davidfetterRemember to vote!
[ Attachment, skipping... ]
---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match
--
Bruce Momjian bruce@momjian.us
EnterpriseDB http://www.enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
Patch applied. Thanks. Unnecessary #include file removed.
---------------------------------------------------------------------------
David Fetter wrote:
On Sun, Jul 30, 2006 at 05:40:16PM -0400, Tom Lane wrote:
Alvaro Herrera <alvherre@commandprompt.com> writes:
David Fetter wrote:
This patch changes the data type from unsigned int to unsigned
long long, which is probably not the correct thing in order to
get 64-bit arithmetic, but I figure it's good enough to get a
discussion started.The only thing I can tell you is that you should use INT64_FORMAT
instead of %lld.And the datatype should be declared int64, not "long long" which
doesn't exist everywhere.Actually you probably want uint64 and UINT64_FORMAT...
regards, tom lane
I think this fixes it, but I'm unsure how to test it. Two of the
methods mentioned in IRC, attaching with gdb and setting to a value >
2^32, and setting it directly in some code, seem like OK approaches.Cheers,
D
--
David Fetter <david@fetter.org> http://fetter.org/
phone: +1 415 235 3778 AIM: dfetter666
Skype: davidfetterRemember to vote!
[ Attachment, skipping... ]
---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match
--
Bruce Momjian bruce@momjian.us
EnterpriseDB http://www.enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +