SQL/MED - file_fdw
Hi, hackers,
Attached is a patch that adds file_fdw, FDW which reads records from
files on the server side, as a contrib module. This patch is based on
"SQL/MED core functionality" patch.
[SQL/MED - core functionality]
http://archives.postgresql.org/pgsql-hackers/2010-11/msg01698.php
File_fdw can be installed with the steps similar to other contrib
modules, and you can create FDW with the script:
$SHAREDIR/contrib/file_fdw.sql
Note that you need to create file_fdw for each database.
Document for file_fdw is included in the patch, although the contents
might not be enough.
Any comments and questions are welcome.
Regards,
--
Shigeru Hanada
Attachments:
On Thu, 25 Nov 2010 17:12:44 +0900
Shigeru HANADA <hanada@metrosystems.co.jp> wrote:
Attached is a patch that adds file_fdw, FDW which reads records from
files on the server side, as a contrib module. This patch is based on
"SQL/MED core functionality" patch.[SQL/MED - core functionality]
http://archives.postgresql.org/pgsql-hackers/2010-11/msg01698.php
I'm going to add new CommitFest items for this patch and "SQL/MED -
postgresql_fdw" patch which have been split from "SQL/MED" patch. Can
I add them to CF 2010-11 which original "SQL/MED" item is in? Or
should I add them to CF 2011-01?
Regards,
--
Shigeru Hanada
On Thu, Nov 25, 2010 at 05:51:11PM +0900, Shigeru HANADA wrote:
On Thu, 25 Nov 2010 17:12:44 +0900
Shigeru HANADA <hanada@metrosystems.co.jp> wrote:Attached is a patch that adds file_fdw, FDW which reads records from
files on the server side, as a contrib module. This patch is based on
"SQL/MED core functionality" patch.[SQL/MED - core functionality]
http://archives.postgresql.org/pgsql-hackers/2010-11/msg01698.phpI'm going to add new CommitFest items for this patch and "SQL/MED -
postgresql_fdw" patch which have been split from "SQL/MED" patch. Can
I add them to CF 2010-11 which original "SQL/MED" item is in? Or
should I add them to CF 2011-01?
The original.
Cheers,
David.
--
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david.fetter@gmail.com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics
Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate
On Thu, 25 Nov 2010 18:40:09 -0800
David Fetter <david@fetter.org> wrote:
On Thu, Nov 25, 2010 at 05:51:11PM +0900, Shigeru HANADA wrote:
I'm going to add new CommitFest items for this patch and "SQL/MED -
postgresql_fdw" patch which have been split from "SQL/MED" patch. Can
I add them to CF 2010-11 which original "SQL/MED" item is in? Or
should I add them to CF 2011-01?The original.
Thanks, added them to CF 2010-11.
--
Shigeru Hanada
On 11/25/2010 03:12 AM, Shigeru HANADA wrote:
Hi, hackers,
Attached is a patch that adds file_fdw, FDW which reads records from
files on the server side, as a contrib module. This patch is based on
"SQL/MED core functionality" patch.[SQL/MED - core functionality]
http://archives.postgresql.org/pgsql-hackers/2010-11/msg01698.phpFile_fdw can be installed with the steps similar to other contrib
modules, and you can create FDW with the script:
$SHAREDIR/contrib/file_fdw.sql
Note that you need to create file_fdw for each database.Document for file_fdw is included in the patch, although the contents
might not be enough.Any comments and questions are welcome.
Looking at file_parser.c, it seems to be largely taken from copy.c.
Wouldn't it be better to call those functions, or refactor them so they
are callable if necessary?
cheers
andrew
On Sun, Dec 5, 2010 at 07:24, Andrew Dunstan <andrew@dunslane.net> wrote:
Looking at file_parser.c, it seems to be largely taken from copy.c. Wouldn't
it be better to call those functions, or refactor them so they are callable
if necessary?
We could export private functions and structs in copy.c,
though details of the implementation should be kept in copy.c.
How about splitting the file_fdw patch into two pieces?
One exports the copy functions from the core, and another
implements file_fdw using the infrastructure.
--
Itagaki Takahiro
On 12/04/2010 11:11 PM, Itagaki Takahiro wrote:
On Sun, Dec 5, 2010 at 07:24, Andrew Dunstan<andrew@dunslane.net> wrote:
Looking at file_parser.c, it seems to be largely taken from copy.c. Wouldn't
it be better to call those functions, or refactor them so they are callable
if necessary?We could export private functions and structs in copy.c,
though details of the implementation should be kept in copy.c.How about splitting the file_fdw patch into two pieces?
One exports the copy functions from the core, and another
implements file_fdw using the infrastructure.
Yes please.
cheers
andrew
2010/11/25 Shigeru HANADA <hanada@metrosystems.co.jp>:
Hi, hackers,
Attached is a patch that adds file_fdw, FDW which reads records from
files on the server side, as a contrib module. This patch is based on
"SQL/MED core functionality" patch.[SQL/MED - core functionality]
http://archives.postgresql.org/pgsql-hackers/2010-11/msg01698.phpFile_fdw can be installed with the steps similar to other contrib
modules, and you can create FDW with the script:
$SHAREDIR/contrib/file_fdw.sql
Note that you need to create file_fdw for each database.Document for file_fdw is included in the patch, although the contents
might not be enough.Any comments and questions are welcome.
I think it is better to add encoding option to FileFdwOption. In the
patch the encoding of file is assumed as client_encoding, but we may
want to SELECT from different-encoded csv in a query.
Apart from the issue with fdw, I've been thinking client_encoding for
COPY is not appropriate in any way. client_encoding is the encoding of
the statement the client sends, not the COPY target which is on the
server's filesystem. Adding encoding option to COPY will eliminate
allowEncodingChanges option from JDBC driver.
Regards,
--
Hitoshi Harada
On Mon, Dec 6, 2010 at 5:48 AM, Hitoshi Harada <umi.tanuki@gmail.com> wrote:
I think it is better to add encoding option to FileFdwOption. In the
patch the encoding of file is assumed as client_encoding, but we may
want to SELECT from different-encoded csv in a query.Apart from the issue with fdw, I've been thinking client_encoding for
COPY is not appropriate in any way. client_encoding is the encoding of
the statement the client sends, not the COPY target which is on the
server's filesystem. Adding encoding option to COPY will eliminate
allowEncodingChanges option from JDBC driver.
Yeah, this point has been raised before, and I agree with it. I
haven't heard anyone who speaks a European language complain about
this, but it seems to keep coming up for Japanese speakers. I am
guessing that means that using multiple encodings is fairly common in
Japan. I typically don't run into anything other than UTF-8 and
Latin-1, which are mostly compatible especially if you're an English
speaker, but if it weren't for that happy coincidence I think this
would be quite annoying.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On 12/04/2010 11:11 PM, Itagaki Takahiro wrote:
On Sun, Dec 5, 2010 at 07:24, Andrew Dunstan<andrew@dunslane.net> wrote:
Looking at file_parser.c, it seems to be largely taken from copy.c. Wouldn't
it be better to call those functions, or refactor them so they are callable
if necessary?We could export private functions and structs in copy.c,
though details of the implementation should be kept in copy.c.How about splitting the file_fdw patch into two pieces?
One exports the copy functions from the core, and another
implements file_fdw using the infrastructure.
Who is actually going to do this split?
cheers
andrew
On Sat, Dec 11, 2010 at 05:30, Andrew Dunstan <andrew@dunslane.net> wrote:
On 12/04/2010 11:11 PM, Itagaki Takahiro wrote:
One exports the copy functions from the core, and another
implements file_fdw using the infrastructure.Who is actually going to do this split?
I'm working for it :-) I extract those functions from copy.c:
- CopyState BeginCopyFrom(Relation rel, const char *filename,
List *attnamelist, List *options);
- void EndCopyFrom(CopyState cstate);
- bool NextCopyFrom(CopyState cstate,
Datum *values, bool *nulls, Oid *oid);
There was Reset() in file_fdw, but it is not contained in the
patch. It will be added again if required, but I wonder we might
need not only reset but also mark/restore a position in a file.
--
Itagaki Takahiro
Attachments:
copy_export-20101213.diffapplication/octet-stream; name=copy_export-20101213.diffDownload
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index 7b8bee8..05bbce1 100644
*** a/src/backend/commands/copy.c
--- b/src/backend/commands/copy.c
*************** typedef struct CopyStateData
*** 142,150 ****
StringInfoData attribute_buf;
/* field raw data pointers found by COPY FROM */
!
! int max_fields;
! char ** raw_fields;
/*
* Similarly, line_buf holds the whole input line being processed. The
--- 142,149 ----
StringInfoData attribute_buf;
/* field raw data pointers found by COPY FROM */
! int max_fields;
! char **raw_fields;
/*
* Similarly, line_buf holds the whole input line being processed. The
*************** typedef struct CopyStateData
*** 167,175 ****
char *raw_buf;
int raw_buf_index; /* next byte to process */
int raw_buf_len; /* total # of bytes stored */
- } CopyStateData;
! typedef CopyStateData *CopyState;
/* DestReceiver for COPY (SELECT) TO */
typedef struct
--- 166,186 ----
char *raw_buf;
int raw_buf_index; /* next byte to process */
int raw_buf_len; /* total # of bytes stored */
! /*
! * The definition of input functions and default expressions are stored
! * in these variables.
! */
! EState *estate;
! AttrNumber num_defaults;
! bool file_has_oids;
! FmgrInfo oid_in_function;
! Oid oid_typioparam;
! FmgrInfo *in_functions;
! Oid *typioparams;
! int *defmap;
! ExprState **defexprs; /* array of default att expressions */
! } CopyStateData;
/* DestReceiver for COPY (SELECT) TO */
typedef struct
*************** static const char BinarySignature[11] =
*** 248,253 ****
--- 259,270 ----
/* non-export function prototypes */
+ static CopyState BeginCopy(bool is_from,
+ Relation rel, Node *raw_query, const char *queryString,
+ const char *filename, List *attnamelist, List *options);
+ static CopyState BeginCopyTo(Relation rel, Node *query, const char *queryString,
+ const char *filename, List *attnamelist, List *options);
+ static void EndCopyTo(CopyState cstate);
static void DoCopyTo(CopyState cstate);
static void CopyTo(CopyState cstate);
static void CopyOneRowTo(CopyState cstate, Oid tupleOid,
*************** CopyLoadRawBuf(CopyState cstate)
*** 718,730 ****
* Do not allow the copy if user doesn't have proper permission to access
* the table or the specifically requested columns.
*/
! uint64
! DoCopy(const CopyStmt *stmt, const char *queryString)
{
CopyState cstate;
- bool is_from = stmt->is_from;
- bool pipe = (stmt->filename == NULL);
- List *attnamelist = stmt->attlist;
List *force_quote = NIL;
List *force_notnull = NIL;
bool force_quote_all = false;
--- 735,746 ----
* Do not allow the copy if user doesn't have proper permission to access
* the table or the specifically requested columns.
*/
! static CopyState
! BeginCopy(bool is_from,
! Relation rel, Node *raw_query, const char *queryString,
! const char *filename, List *attnamelist, List *options)
{
CopyState cstate;
List *force_quote = NIL;
List *force_notnull = NIL;
bool force_quote_all = false;
*************** DoCopy(const CopyStmt *stmt, const char
*** 733,745 ****
ListCell *option;
TupleDesc tupDesc;
int num_phys_attrs;
- uint64 processed;
/* Allocate workspace and zero all fields */
cstate = (CopyStateData *) palloc0(sizeof(CopyStateData));
/* Extract options from the statement node tree */
! foreach(option, stmt->options)
{
DefElem *defel = (DefElem *) lfirst(option);
--- 749,760 ----
ListCell *option;
TupleDesc tupDesc;
int num_phys_attrs;
/* Allocate workspace and zero all fields */
cstate = (CopyStateData *) palloc0(sizeof(CopyStateData));
/* Extract options from the statement node tree */
! foreach(option, options)
{
DefElem *defel = (DefElem *) lfirst(option);
*************** DoCopy(const CopyStmt *stmt, const char
*** 980,1005 ****
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("CSV quote character must not appear in the NULL specification")));
! /* Disallow file COPY except to superusers. */
! if (!pipe && !superuser())
! ereport(ERROR,
! (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
! errmsg("must be superuser to COPY to or from a file"),
! errhint("Anyone can COPY to stdout or from stdin. "
! "psql's \\copy command also works for anyone.")));
!
! if (stmt->relation)
{
RangeTblEntry *rte;
List *attnums;
ListCell *cur;
! Assert(!stmt->query);
! cstate->queryDesc = NULL;
! /* Open and lock the relation, using the appropriate lock type. */
! cstate->rel = heap_openrv(stmt->relation,
! (is_from ? RowExclusiveLock : AccessShareLock));
tupDesc = RelationGetDescr(cstate->rel);
--- 995,1009 ----
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("CSV quote character must not appear in the NULL specification")));
! if (rel)
{
RangeTblEntry *rte;
List *attnums;
ListCell *cur;
! Assert(!raw_query);
! cstate->rel = rel;
tupDesc = RelationGetDescr(cstate->rel);
*************** DoCopy(const CopyStmt *stmt, const char
*** 1058,1064 ****
* function and is executed repeatedly. (See also the same hack in
* DECLARE CURSOR and PREPARE.) XXX FIXME someday.
*/
! rewritten = pg_analyze_and_rewrite((Node *) copyObject(stmt->query),
queryString, NULL, 0);
/* We don't expect more or less than one result query */
--- 1062,1068 ----
* function and is executed repeatedly. (See also the same hack in
* DECLARE CURSOR and PREPARE.) XXX FIXME someday.
*/
! rewritten = pg_analyze_and_rewrite((Node *) copyObject(raw_query),
queryString, NULL, 0);
/* We don't expect more or less than one result query */
*************** DoCopy(const CopyStmt *stmt, const char
*** 1160,1171 ****
}
}
- /* Set up variables to avoid per-attribute overhead. */
- initStringInfo(&cstate->attribute_buf);
- initStringInfo(&cstate->line_buf);
- cstate->line_buf_converted = false;
- cstate->raw_buf = (char *) palloc(RAW_BUF_SIZE + 1);
- cstate->raw_buf_index = cstate->raw_buf_len = 0;
cstate->processed = 0;
/*
--- 1164,1169 ----
*************** DoCopy(const CopyStmt *stmt, const char
*** 1181,1202 ****
cstate->encoding_embeds_ascii = PG_ENCODING_IS_CLIENT_ONLY(cstate->client_encoding);
cstate->copy_dest = COPY_FILE; /* default */
! cstate->filename = stmt->filename;
if (is_from)
CopyFrom(cstate); /* copy from file to database */
else
DoCopyTo(cstate); /* copy from database to file */
/*
! * Close the relation or query. If reading, we can release the
! * AccessShareLock we got; if writing, we should hold the lock until end
! * of transaction to ensure that updates will be committed before lock is
! * released.
*/
! if (cstate->rel)
! heap_close(cstate->rel, (is_from ? NoLock : AccessShareLock));
! else
{
/* Close down the query and free resources. */
ExecutorEnd(cstate->queryDesc);
--- 1179,1681 ----
cstate->encoding_embeds_ascii = PG_ENCODING_IS_CLIENT_ONLY(cstate->client_encoding);
cstate->copy_dest = COPY_FILE; /* default */
! cstate->filename = (filename ? pstrdup(filename) : NULL);
!
! return cstate;
! }
!
! CopyState
! BeginCopyFrom(Relation rel, const char *filename, List *attnamelist, List *options)
! {
! CopyState cstate;
! bool pipe = (filename == NULL);
! TupleDesc tupDesc;
! int num_phys_attrs;
! Form_pg_attribute *attr;
! AttrNumber attr_count;
! int attnum;
! EState *estate;
! Oid in_func_oid;
!
! cstate = BeginCopy(true, rel, NULL, NULL, filename, attnamelist, options);
!
! /* Initialize state variables */
! cstate->fe_eof = false;
! cstate->eol_type = EOL_UNKNOWN;
! cstate->cur_relname = RelationGetRelationName(cstate->rel);
! cstate->cur_lineno = 0;
! cstate->cur_attname = NULL;
! cstate->cur_attval = NULL;
!
! /* Set up variables to avoid per-attribute overhead. */
! initStringInfo(&cstate->attribute_buf);
! initStringInfo(&cstate->line_buf);
! cstate->line_buf_converted = false;
! cstate->raw_buf = (char *) palloc(RAW_BUF_SIZE + 1);
! cstate->raw_buf_index = cstate->raw_buf_len = 0;
!
! tupDesc = RelationGetDescr(cstate->rel);
! attr = tupDesc->attrs;
! num_phys_attrs = tupDesc->natts;
! attr_count = list_length(cstate->attnumlist);
! cstate->num_defaults = 0;
!
! /*
! * Pick up the required catalog information for each attribute in the
! * relation, including the input function, the element type (to pass to
! * the input function), and info about defaults and constraints. (Which
! * input function we use depends on text/binary format choice.)
! */
! cstate->in_functions = (FmgrInfo *) palloc(num_phys_attrs * sizeof(FmgrInfo));
! cstate->typioparams = (Oid *) palloc(num_phys_attrs * sizeof(Oid));
! cstate->defmap = (int *) palloc(num_phys_attrs * sizeof(int));
! cstate->defexprs = (ExprState **) palloc(num_phys_attrs * sizeof(ExprState *));
!
! /* We need a ResultRelInfo to check constraints. */
! estate = cstate->estate = CreateExecutorState();
!
! for (attnum = 1; attnum <= num_phys_attrs; attnum++)
! {
! /* We don't need info for dropped attributes */
! if (attr[attnum - 1]->attisdropped)
! continue;
!
! /* Fetch the input function and typioparam info */
! if (cstate->binary)
! getTypeBinaryInputInfo(attr[attnum - 1]->atttypid,
! &in_func_oid, &cstate->typioparams[attnum - 1]);
! else
! getTypeInputInfo(attr[attnum - 1]->atttypid,
! &in_func_oid, &cstate->typioparams[attnum - 1]);
! fmgr_info(in_func_oid, &cstate->in_functions[attnum - 1]);
!
! /* Get default info if needed */
! if (!list_member_int(cstate->attnumlist, attnum))
! {
! /* attribute is NOT to be copied from input */
! /* use default value if one exists */
! Node *defexpr = build_column_default(cstate->rel, attnum);
!
! if (defexpr != NULL)
! {
! cstate->defexprs[cstate->num_defaults] =
! ExecPrepareExpr((Expr *) defexpr, estate);
! cstate->defmap[cstate->num_defaults] = attnum - 1;
! cstate->num_defaults++;
! }
! }
! }
!
! if (pipe)
! {
! if (whereToSendOutput == DestRemote)
! ReceiveCopyBegin(cstate);
! else
! cstate->copy_file = stdin;
! }
! else
! {
! struct stat st;
!
! cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_R);
!
! if (cstate->copy_file == NULL)
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not open file \"%s\" for reading: %m",
! cstate->filename)));
!
! fstat(fileno(cstate->copy_file), &st);
! if (S_ISDIR(st.st_mode))
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("\"%s\" is a directory", cstate->filename)));
! }
!
! if (!cstate->binary)
! cstate->file_has_oids = cstate->oids; /* must rely on user to tell us... */
! else
! {
! /* Read and verify binary header */
! char readSig[11];
! int32 tmp;
!
! /* Signature */
! if (CopyGetData(cstate, readSig, 11, 11) != 11 ||
! memcmp(readSig, BinarySignature, 11) != 0)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("COPY file signature not recognized")));
! /* Flags field */
! if (!CopyGetInt32(cstate, &tmp))
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid COPY file header (missing flags)")));
! cstate->file_has_oids = (tmp & (1 << 16)) != 0;
! tmp &= ~(1 << 16);
! if ((tmp >> 16) != 0)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unrecognized critical flags in COPY file header")));
! /* Header extension length */
! if (!CopyGetInt32(cstate, &tmp) ||
! tmp < 0)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid COPY file header (missing length)")));
! /* Skip extension header, if present */
! while (tmp-- > 0)
! {
! if (CopyGetData(cstate, readSig, 1, 1) != 1)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid COPY file header (wrong length)")));
! }
! }
!
! if (cstate->file_has_oids && cstate->binary)
! {
! Oid in_func_oid;
! getTypeBinaryInputInfo(OIDOID,
! &in_func_oid, &cstate->oid_typioparam);
! fmgr_info(in_func_oid, &cstate->oid_in_function);
! }
!
! /* create workspace for CopyReadAttributes results */
! if (!cstate->binary)
! {
! int nfields = cstate->file_has_oids ? (attr_count + 1) : attr_count;
!
! cstate->max_fields = nfields;
! cstate->raw_fields = (char **) palloc(nfields * sizeof(char *));
! }
!
! return cstate;
! }
!
! static CopyState
! BeginCopyTo(Relation rel, Node *query, const char *queryString,
! const char *filename, List *attnamelist, List *options)
! {
! return BeginCopy(false, rel, query, queryString,
! filename, attnamelist, options);
! }
!
! /* return false if no more tuples */
! bool
! NextCopyFrom(CopyState cstate, Datum *values, bool *nulls, Oid *oid)
! {
! TupleDesc tupDesc;
! Form_pg_attribute *attr;
! AttrNumber num_phys_attrs,
! attr_count,
! num_defaults = cstate->num_defaults;
! FmgrInfo *in_functions = cstate->in_functions;
! Oid *typioparams = cstate->typioparams;
! int i;
! int nfields;
! char **field_strings;
! bool isnull;
! int *defmap = cstate->defmap;
! ExprState **defexprs = cstate->defexprs;
! ExprContext *econtext; /* used for ExecEvalExpr for default atts */
!
! /* on input just throw the header line away */
! if (cstate->cur_lineno == 0 && cstate->header_line)
! {
! cstate->cur_lineno++;
! if (CopyReadLine(cstate))
! return false; /* done */
! }
!
! cstate->cur_lineno++;
!
! tupDesc = RelationGetDescr(cstate->rel);
! attr = tupDesc->attrs;
! num_phys_attrs = tupDesc->natts;
! attr_count = list_length(cstate->attnumlist);
! nfields = cstate->file_has_oids ? (attr_count + 1) : attr_count;
!
! /* Initialize all values for row to NULL */
! MemSet(values, 0, num_phys_attrs * sizeof(Datum));
! MemSet(nulls, true, num_phys_attrs * sizeof(bool));
!
! if (!cstate->binary)
! {
! ListCell *cur;
! int fldct;
! int fieldno;
! char *string;
!
! /*
! * Actually read the line into memory here.
! * EOF at start of line means we're done. If we see EOF after
! * some characters, we act as though it was newline followed by
! * EOF, ie, process the line and then exit loop on next iteration.
! */
! if (CopyReadLine(cstate) && cstate->line_buf.len == 0)
! return false;
!
! /* Parse the line into de-escaped field values */
! if (cstate->csv_mode)
! fldct = CopyReadAttributesCSV(cstate);
! else
! fldct = CopyReadAttributesText(cstate);
!
! /* check for overflowing fields */
! if (nfields > 0 && fldct > nfields)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("extra data after last expected column")));
!
! fieldno = 0;
! field_strings = cstate->raw_fields;
!
! /* Read the OID field if present */
! if (cstate->file_has_oids)
! {
! if (fieldno >= fldct)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("missing data for OID column")));
! string = field_strings[fieldno++];
!
! if (string == NULL)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("null OID in COPY data")));
! else
! {
! cstate->cur_attname = "oid";
! cstate->cur_attval = string;
! *oid = DatumGetObjectId(DirectFunctionCall1(oidin,
! CStringGetDatum(string)));
! if (*oid == InvalidOid)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid OID in COPY data")));
! cstate->cur_attname = NULL;
! cstate->cur_attval = NULL;
! }
! }
!
! /* Loop to read the user attributes on the line. */
! foreach(cur, cstate->attnumlist)
! {
! int attnum = lfirst_int(cur);
! int m = attnum - 1;
!
! if (fieldno >= fldct)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("missing data for column \"%s\"",
! NameStr(attr[m]->attname))));
! string = field_strings[fieldno++];
!
! if (cstate->csv_mode && string == NULL &&
! cstate->force_notnull_flags[m])
! {
! /* Go ahead and read the NULL string */
! string = cstate->null_print;
! }
!
! cstate->cur_attname = NameStr(attr[m]->attname);
! cstate->cur_attval = string;
! values[m] = InputFunctionCall(&in_functions[m],
! string,
! typioparams[m],
! attr[m]->atttypmod);
! if (string != NULL)
! nulls[m] = false;
! cstate->cur_attname = NULL;
! cstate->cur_attval = NULL;
! }
!
! Assert(fieldno == nfields);
! }
! else
! {
! /* binary */
! int16 fld_count;
! ListCell *cur;
!
! if (!CopyGetInt16(cstate, &fld_count))
! {
! /* EOF detected (end of file, or protocol-level EOF) */
! return false;
! }
!
! if (fld_count == -1)
! {
! /*
! * Received EOF marker. In a V3-protocol copy, wait for
! * the protocol-level EOF, and complain if it doesn't come
! * immediately. This ensures that we correctly handle
! * CopyFail, if client chooses to send that now.
! *
! * Note that we MUST NOT try to read more data in an
! * old-protocol copy, since there is no protocol-level EOF
! * marker then. We could go either way for copy from file,
! * but choose to throw error if there's data after the EOF
! * marker, for consistency with the new-protocol case.
! */
! char dummy;
!
! if (cstate->copy_dest != COPY_OLD_FE &&
! CopyGetData(cstate, &dummy, 1, 1) > 0)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("received copy data after EOF marker")));
! return false;
! }
!
! if (fld_count != attr_count)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("row field count is %d, expected %d",
! (int) fld_count, attr_count)));
!
! if (cstate->file_has_oids)
! {
! cstate->cur_attname = "oid";
! *oid = DatumGetObjectId(CopyReadBinaryAttribute(cstate,
! 0,
! &cstate->oid_in_function,
! cstate->oid_typioparam,
! -1,
! &isnull));
! if (isnull || *oid == InvalidOid)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid OID in COPY data")));
! cstate->cur_attname = NULL;
! }
!
! i = 0;
! foreach(cur, cstate->attnumlist)
! {
! int attnum = lfirst_int(cur);
! int m = attnum - 1;
!
! cstate->cur_attname = NameStr(attr[m]->attname);
! i++;
! values[m] = CopyReadBinaryAttribute(cstate,
! i,
! &in_functions[m],
! typioparams[m],
! attr[m]->atttypmod,
! &nulls[m]);
! cstate->cur_attname = NULL;
! }
! }
!
! /*
! * Now compute and insert any defaults available for the columns not
! * provided by the input data. Anything not processed here or above
! * will remain NULL.
! */
! econtext = GetPerTupleExprContext(cstate->estate);
! for (i = 0; i < num_defaults; i++)
! {
! values[defmap[i]] = ExecEvalExpr(defexprs[i], econtext,
! &nulls[defmap[i]], NULL);
! }
!
! return true;
! }
!
! uint64
! DoCopy(const CopyStmt *stmt, const char *queryString)
! {
! CopyState cstate;
! bool is_from = stmt->is_from;
! bool pipe = (stmt->filename == NULL);
! Relation rel;
! uint64 processed;
!
! /* Disallow file COPY except to superusers. */
! if (!pipe && !superuser())
! ereport(ERROR,
! (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
! errmsg("must be superuser to COPY to or from a file"),
! errhint("Anyone can COPY to stdout or from stdin. "
! "psql's \\copy command also works for anyone.")));
!
! if (stmt->relation)
! {
! Assert(!stmt->query);
!
! /* Open and lock the relation, using the appropriate lock type. */
! rel = heap_openrv(stmt->relation,
! (is_from ? RowExclusiveLock : AccessShareLock));
! }
! else
! {
! Assert(stmt->query);
!
! rel = NULL;
! }
if (is_from)
+ {
+ cstate = BeginCopyFrom(rel, stmt->filename, stmt->attlist, stmt->options);
CopyFrom(cstate); /* copy from file to database */
+ processed = cstate->processed;
+ EndCopyFrom(cstate);
+ }
else
+ {
+ cstate = BeginCopyTo(rel, stmt->query, queryString, stmt->filename, stmt->attlist, stmt->options);
DoCopyTo(cstate); /* copy from database to file */
+ processed = cstate->processed;
+ EndCopyTo(cstate);
+ }
/*
! * Close the relation. If reading, we can release the AccessShareLock we got;
! * if writing, we should hold the lock until end of transaction to ensure that
! * updates will be committed before lock is released.
*/
! if (rel != NULL)
! heap_close(rel, (is_from ? NoLock : AccessShareLock));
!
! return processed;
! }
!
! void
! EndCopyFrom(CopyState cstate)
! {
! FreeExecutorState(cstate->estate);
!
! /* Clean up storage */
! if (cstate->filename)
! {
! if (FreeFile(cstate->copy_file))
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not read from file \"%s\": %m",
! cstate->filename)));
! pfree(cstate->filename);
! }
! if (!cstate->binary)
! pfree(cstate->raw_fields);
! pfree(cstate->attribute_buf.data);
! pfree(cstate->line_buf.data);
! pfree(cstate->raw_buf);
! pfree(cstate->in_functions);
! pfree(cstate->typioparams);
! pfree(cstate->defmap);
! pfree(cstate->defexprs);
! pfree(cstate);
! }
!
! static void
! EndCopyTo(CopyState cstate)
! {
! /*
! * Close the relation or query. We can release the AccessShareLock we got.
! */
! if (cstate->rel == NULL)
{
/* Close down the query and free resources. */
ExecutorEnd(cstate->queryDesc);
*************** DoCopy(const CopyStmt *stmt, const char
*** 1204,1221 ****
PopActiveSnapshot();
}
! /* Clean up storage (probably not really necessary) */
! processed = cstate->processed;
!
! pfree(cstate->attribute_buf.data);
! pfree(cstate->line_buf.data);
! pfree(cstate->raw_buf);
pfree(cstate);
-
- return processed;
}
-
/*
* This intermediate routine exists mainly to localize the effects of setjmp
* so we don't need to plaster a lot of variables with "volatile".
--- 1683,1694 ----
PopActiveSnapshot();
}
! /* Clean up storage */
! if (cstate->filename)
! pfree(cstate->filename);
pfree(cstate);
}
/*
* This intermediate routine exists mainly to localize the effects of setjmp
* so we don't need to plaster a lot of variables with "volatile".
*************** limit_printout_length(const char *str)
*** 1666,1698 ****
static void
CopyFrom(CopyState cstate)
{
- bool pipe = (cstate->filename == NULL);
HeapTuple tuple;
TupleDesc tupDesc;
- Form_pg_attribute *attr;
- AttrNumber num_phys_attrs,
- attr_count,
- num_defaults;
- FmgrInfo *in_functions;
- FmgrInfo oid_in_function;
- Oid *typioparams;
- Oid oid_typioparam;
- int attnum;
- int i;
- Oid in_func_oid;
Datum *values;
bool *nulls;
- int nfields;
- char **field_strings;
bool done = false;
- bool isnull;
ResultRelInfo *resultRelInfo;
! EState *estate = CreateExecutorState(); /* for ExecConstraints() */
TupleTableSlot *slot;
- bool file_has_oids;
- int *defmap;
- ExprState **defexprs; /* array of default att expressions */
- ExprContext *econtext; /* used for ExecEvalExpr for default atts */
MemoryContext oldcontext = CurrentMemoryContext;
ErrorContextCallback errcontext;
CommandId mycid = GetCurrentCommandId(true);
--- 2139,2152 ----
static void
CopyFrom(CopyState cstate)
{
HeapTuple tuple;
TupleDesc tupDesc;
Datum *values;
bool *nulls;
bool done = false;
ResultRelInfo *resultRelInfo;
! EState *estate = cstate->estate; /* for ExecConstraints() */
TupleTableSlot *slot;
MemoryContext oldcontext = CurrentMemoryContext;
ErrorContextCallback errcontext;
CommandId mycid = GetCurrentCommandId(true);
*************** CopyFrom(CopyState cstate)
*** 1720,1725 ****
--- 2174,2181 ----
RelationGetRelationName(cstate->rel))));
}
+ tupDesc = RelationGetDescr(cstate->rel);
+
/*----------
* Check to see if we can avoid writing WAL
*
*************** CopyFrom(CopyState cstate)
*** 1755,1792 ****
hi_options |= HEAP_INSERT_SKIP_WAL;
}
- if (pipe)
- {
- if (whereToSendOutput == DestRemote)
- ReceiveCopyBegin(cstate);
- else
- cstate->copy_file = stdin;
- }
- else
- {
- struct stat st;
-
- cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_R);
-
- if (cstate->copy_file == NULL)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not open file \"%s\" for reading: %m",
- cstate->filename)));
-
- fstat(fileno(cstate->copy_file), &st);
- if (S_ISDIR(st.st_mode))
- ereport(ERROR,
- (errcode(ERRCODE_WRONG_OBJECT_TYPE),
- errmsg("\"%s\" is a directory", cstate->filename)));
- }
-
- tupDesc = RelationGetDescr(cstate->rel);
- attr = tupDesc->attrs;
- num_phys_attrs = tupDesc->natts;
- attr_count = list_length(cstate->attnumlist);
- num_defaults = 0;
-
/*
* We need a ResultRelInfo so we can use the regular executor's
* index-entry-making machinery. (There used to be a huge amount of code
--- 2211,2216 ----
*************** CopyFrom(CopyState cstate)
*** 1815,1865 ****
slot = ExecInitExtraTupleSlot(estate);
ExecSetSlotDescriptor(slot, tupDesc);
- econtext = GetPerTupleExprContext(estate);
-
- /*
- * Pick up the required catalog information for each attribute in the
- * relation, including the input function, the element type (to pass to
- * the input function), and info about defaults and constraints. (Which
- * input function we use depends on text/binary format choice.)
- */
- in_functions = (FmgrInfo *) palloc(num_phys_attrs * sizeof(FmgrInfo));
- typioparams = (Oid *) palloc(num_phys_attrs * sizeof(Oid));
- defmap = (int *) palloc(num_phys_attrs * sizeof(int));
- defexprs = (ExprState **) palloc(num_phys_attrs * sizeof(ExprState *));
-
- for (attnum = 1; attnum <= num_phys_attrs; attnum++)
- {
- /* We don't need info for dropped attributes */
- if (attr[attnum - 1]->attisdropped)
- continue;
-
- /* Fetch the input function and typioparam info */
- if (cstate->binary)
- getTypeBinaryInputInfo(attr[attnum - 1]->atttypid,
- &in_func_oid, &typioparams[attnum - 1]);
- else
- getTypeInputInfo(attr[attnum - 1]->atttypid,
- &in_func_oid, &typioparams[attnum - 1]);
- fmgr_info(in_func_oid, &in_functions[attnum - 1]);
-
- /* Get default info if needed */
- if (!list_member_int(cstate->attnumlist, attnum))
- {
- /* attribute is NOT to be copied from input */
- /* use default value if one exists */
- Node *defexpr = build_column_default(cstate->rel, attnum);
-
- if (defexpr != NULL)
- {
- defexprs[num_defaults] = ExecPrepareExpr((Expr *) defexpr,
- estate);
- defmap[num_defaults] = attnum - 1;
- num_defaults++;
- }
- }
- }
-
/* Prepare to catch AFTER triggers. */
AfterTriggerBeginQuery();
--- 2239,2244 ----
*************** CopyFrom(CopyState cstate)
*** 1871,1942 ****
*/
ExecBSInsertTriggers(estate, resultRelInfo);
! if (!cstate->binary)
! file_has_oids = cstate->oids; /* must rely on user to tell us... */
! else
! {
! /* Read and verify binary header */
! char readSig[11];
! int32 tmp;
!
! /* Signature */
! if (CopyGetData(cstate, readSig, 11, 11) != 11 ||
! memcmp(readSig, BinarySignature, 11) != 0)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("COPY file signature not recognized")));
! /* Flags field */
! if (!CopyGetInt32(cstate, &tmp))
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid COPY file header (missing flags)")));
! file_has_oids = (tmp & (1 << 16)) != 0;
! tmp &= ~(1 << 16);
! if ((tmp >> 16) != 0)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unrecognized critical flags in COPY file header")));
! /* Header extension length */
! if (!CopyGetInt32(cstate, &tmp) ||
! tmp < 0)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid COPY file header (missing length)")));
! /* Skip extension header, if present */
! while (tmp-- > 0)
! {
! if (CopyGetData(cstate, readSig, 1, 1) != 1)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid COPY file header (wrong length)")));
! }
! }
!
! if (file_has_oids && cstate->binary)
! {
! getTypeBinaryInputInfo(OIDOID,
! &in_func_oid, &oid_typioparam);
! fmgr_info(in_func_oid, &oid_in_function);
! }
!
! values = (Datum *) palloc(num_phys_attrs * sizeof(Datum));
! nulls = (bool *) palloc(num_phys_attrs * sizeof(bool));
!
! /* create workspace for CopyReadAttributes results */
! nfields = file_has_oids ? (attr_count + 1) : attr_count;
! if (! cstate->binary)
! {
! cstate->max_fields = nfields;
! cstate->raw_fields = (char **) palloc(nfields * sizeof(char *));
! }
!
! /* Initialize state variables */
! cstate->fe_eof = false;
! cstate->eol_type = EOL_UNKNOWN;
! cstate->cur_relname = RelationGetRelationName(cstate->rel);
! cstate->cur_lineno = 0;
! cstate->cur_attname = NULL;
! cstate->cur_attval = NULL;
bistate = GetBulkInsertState();
--- 2250,2257 ----
*/
ExecBSInsertTriggers(estate, resultRelInfo);
! values = (Datum *) palloc(tupDesc->natts * sizeof(Datum));
! nulls = (bool *) palloc(tupDesc->natts * sizeof(bool));
bistate = GetBulkInsertState();
*************** CopyFrom(CopyState cstate)
*** 1946,1958 ****
errcontext.previous = error_context_stack;
error_context_stack = &errcontext;
- /* on input just throw the header line away */
- if (cstate->header_line)
- {
- cstate->cur_lineno++;
- done = CopyReadLine(cstate);
- }
-
while (!done)
{
bool skip_tuple;
--- 2261,2266 ----
*************** CopyFrom(CopyState cstate)
*** 1960,2166 ****
CHECK_FOR_INTERRUPTS();
- cstate->cur_lineno++;
-
/* Reset the per-tuple exprcontext */
ResetPerTupleExprContext(estate);
/* Switch into its memory context */
MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
! /* Initialize all values for row to NULL */
! MemSet(values, 0, num_phys_attrs * sizeof(Datum));
! MemSet(nulls, true, num_phys_attrs * sizeof(bool));
!
! if (!cstate->binary)
! {
! ListCell *cur;
! int fldct;
! int fieldno;
! char *string;
!
! /* Actually read the line into memory here */
! done = CopyReadLine(cstate);
!
! /*
! * EOF at start of line means we're done. If we see EOF after
! * some characters, we act as though it was newline followed by
! * EOF, ie, process the line and then exit loop on next iteration.
! */
! if (done && cstate->line_buf.len == 0)
! break;
!
! /* Parse the line into de-escaped field values */
! if (cstate->csv_mode)
! fldct = CopyReadAttributesCSV(cstate);
! else
! fldct = CopyReadAttributesText(cstate);
!
! /* check for overflowing fields */
! if (nfields > 0 && fldct > nfields)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("extra data after last expected column")));
!
! fieldno = 0;
! field_strings = cstate->raw_fields;
!
! /* Read the OID field if present */
! if (file_has_oids)
! {
! if (fieldno >= fldct)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("missing data for OID column")));
! string = field_strings[fieldno++];
!
! if (string == NULL)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("null OID in COPY data")));
! else
! {
! cstate->cur_attname = "oid";
! cstate->cur_attval = string;
! loaded_oid = DatumGetObjectId(DirectFunctionCall1(oidin,
! CStringGetDatum(string)));
! if (loaded_oid == InvalidOid)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid OID in COPY data")));
! cstate->cur_attname = NULL;
! cstate->cur_attval = NULL;
! }
! }
!
! /* Loop to read the user attributes on the line. */
! foreach(cur, cstate->attnumlist)
! {
! int attnum = lfirst_int(cur);
! int m = attnum - 1;
!
! if (fieldno >= fldct)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("missing data for column \"%s\"",
! NameStr(attr[m]->attname))));
! string = field_strings[fieldno++];
!
! if (cstate->csv_mode && string == NULL &&
! cstate->force_notnull_flags[m])
! {
! /* Go ahead and read the NULL string */
! string = cstate->null_print;
! }
!
! cstate->cur_attname = NameStr(attr[m]->attname);
! cstate->cur_attval = string;
! values[m] = InputFunctionCall(&in_functions[m],
! string,
! typioparams[m],
! attr[m]->atttypmod);
! if (string != NULL)
! nulls[m] = false;
! cstate->cur_attname = NULL;
! cstate->cur_attval = NULL;
! }
!
! Assert(fieldno == nfields);
! }
! else
! {
! /* binary */
! int16 fld_count;
! ListCell *cur;
!
! if (!CopyGetInt16(cstate, &fld_count))
! {
! /* EOF detected (end of file, or protocol-level EOF) */
! done = true;
! break;
! }
!
! if (fld_count == -1)
! {
! /*
! * Received EOF marker. In a V3-protocol copy, wait for
! * the protocol-level EOF, and complain if it doesn't come
! * immediately. This ensures that we correctly handle
! * CopyFail, if client chooses to send that now.
! *
! * Note that we MUST NOT try to read more data in an
! * old-protocol copy, since there is no protocol-level EOF
! * marker then. We could go either way for copy from file,
! * but choose to throw error if there's data after the EOF
! * marker, for consistency with the new-protocol case.
! */
! char dummy;
!
! if (cstate->copy_dest != COPY_OLD_FE &&
! CopyGetData(cstate, &dummy, 1, 1) > 0)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("received copy data after EOF marker")));
! done = true;
! break;
! }
!
! if (fld_count != attr_count)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("row field count is %d, expected %d",
! (int) fld_count, attr_count)));
!
! if (file_has_oids)
! {
! cstate->cur_attname = "oid";
! loaded_oid =
! DatumGetObjectId(CopyReadBinaryAttribute(cstate,
! 0,
! &oid_in_function,
! oid_typioparam,
! -1,
! &isnull));
! if (isnull || loaded_oid == InvalidOid)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid OID in COPY data")));
! cstate->cur_attname = NULL;
! }
!
! i = 0;
! foreach(cur, cstate->attnumlist)
! {
! int attnum = lfirst_int(cur);
! int m = attnum - 1;
!
! cstate->cur_attname = NameStr(attr[m]->attname);
! i++;
! values[m] = CopyReadBinaryAttribute(cstate,
! i,
! &in_functions[m],
! typioparams[m],
! attr[m]->atttypmod,
! &nulls[m]);
! cstate->cur_attname = NULL;
! }
! }
!
! /*
! * Now compute and insert any defaults available for the columns not
! * provided by the input data. Anything not processed here or above
! * will remain NULL.
! */
! for (i = 0; i < num_defaults; i++)
! {
! values[defmap[i]] = ExecEvalExpr(defexprs[i], econtext,
! &nulls[defmap[i]], NULL);
! }
/* And now we can form the input tuple. */
tuple = heap_form_tuple(tupDesc, values, nulls);
! if (cstate->oids && file_has_oids)
HeapTupleSetOid(tuple, loaded_oid);
/* Triggers and stuff need to be invoked in query context. */
--- 2268,2287 ----
CHECK_FOR_INTERRUPTS();
/* Reset the per-tuple exprcontext */
ResetPerTupleExprContext(estate);
/* Switch into its memory context */
MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
! done = !NextCopyFrom(cstate, values, nulls, &loaded_oid);
! if (done)
! break;
/* And now we can form the input tuple. */
tuple = heap_form_tuple(tupDesc, values, nulls);
! if (cstate->oids && cstate->file_has_oids)
HeapTupleSetOid(tuple, loaded_oid);
/* Triggers and stuff need to be invoked in query context. */
*************** CopyFrom(CopyState cstate)
*** 2233,2261 ****
pfree(values);
pfree(nulls);
- if (! cstate->binary)
- pfree(cstate->raw_fields);
-
- pfree(in_functions);
- pfree(typioparams);
- pfree(defmap);
- pfree(defexprs);
ExecResetTupleTable(estate->es_tupleTable, false);
ExecCloseIndices(resultRelInfo);
- FreeExecutorState(estate);
-
- if (!pipe)
- {
- if (FreeFile(cstate->copy_file))
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not read from file \"%s\": %m",
- cstate->filename)));
- }
-
/*
* If we skipped writing WAL, then we need to sync the heap (but not
* indexes since those use WAL anyway)
--- 2354,2364 ----
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index 6d409e8..4d9bd60 100644
*** a/src/include/commands/copy.h
--- b/src/include/commands/copy.h
***************
*** 18,25 ****
--- 18,32 ----
#include "tcop/dest.h"
+ typedef struct CopyStateData *CopyState;
+
extern uint64 DoCopy(const CopyStmt *stmt, const char *queryString);
+ extern CopyState BeginCopyFrom(Relation rel,
+ const char *filename, List *attnamelist, List *options);
+ extern void EndCopyFrom(CopyState cstate);
+ extern bool NextCopyFrom(CopyState cstate, Datum *values, bool *nulls, Oid *oid);
+
extern DestReceiver *CreateCopyDestReceiver(void);
#endif /* COPY_H */
On 12/13/2010 01:31 AM, Itagaki Takahiro wrote:
On Sat, Dec 11, 2010 at 05:30, Andrew Dunstan<andrew@dunslane.net> wrote:
On 12/04/2010 11:11 PM, Itagaki Takahiro wrote:
One exports the copy functions from the core, and another
implements file_fdw using the infrastructure.Who is actually going to do this split?
I'm working for it :-) I extract those functions from copy.c:
- CopyState BeginCopyFrom(Relation rel, const char *filename,
List *attnamelist, List *options);
- void EndCopyFrom(CopyState cstate);
- bool NextCopyFrom(CopyState cstate,
Datum *values, bool *nulls, Oid *oid);There was Reset() in file_fdw, but it is not contained in the
patch. It will be added again if required, but I wonder we might
need not only reset but also mark/restore a position in a file.
Hmm. I don't think that's going to expose enough for what I want to be
able to do. I actually had in mind exposing lower level routines like
CopyReadAttibutesCSV/CopyReadAttributesText and allowing the Foreign
Data Wrapper to manipulate the raw values read (for example from an
irregularly shaped CSV file).
cheers
andrew
Andrew Dunstan <andrew@dunslane.net> writes:
Hmm. I don't think that's going to expose enough for what I want to be
able to do. I actually had in mind exposing lower level routines like
CopyReadAttibutesCSV/CopyReadAttributesText and allowing the Foreign
Data Wrapper to manipulate the raw values read (for example from an
irregularly shaped CSV file).
I think that exposing the guts of COPY to the open air is a bad idea.
We refactor that code for performance or other reasons every release or
two. I don't want to find us tied down to the current implementation
because we're afraid of breaking third-party FDWs.
regards, tom lane
On 12/13/2010 11:12 AM, Tom Lane wrote:
Andrew Dunstan<andrew@dunslane.net> writes:
Hmm. I don't think that's going to expose enough for what I want to be
able to do. I actually had in mind exposing lower level routines like
CopyReadAttibutesCSV/CopyReadAttributesText and allowing the Foreign
Data Wrapper to manipulate the raw values read (for example from an
irregularly shaped CSV file).I think that exposing the guts of COPY to the open air is a bad idea.
We refactor that code for performance or other reasons every release or
two. I don't want to find us tied down to the current implementation
because we're afraid of breaking third-party FDWs.
In that case I guess I'll need to do what Shigeru-san has done, and copy
large parts of copy.c.
cheers
andrew
On Tue, Dec 14, 2010 at 01:25, Andrew Dunstan <andrew@dunslane.net> wrote:
On 12/13/2010 11:12 AM, Tom Lane wrote:
I think that exposing the guts of COPY to the open air is a bad idea.
I don't want to export the details, too.
In that case I guess I'll need to do what Shigeru-san has done, and copy
large parts of copy.c.
I found file_fdw would require the executor state in CopyState and
the error callback function. I revised the patch to export them.
Now 5 functions are exported from copy.c:
- BeginCopyFrom(rel, filename, attnamelist, options) : CopyState
- EndCopyFrom(cstate) : void
- NextCopyFrom(cstate, OUT values, OUT nulls, OUT tupleOid) : bool
- GetCopyExecutorState(cstate) : EState *
- CopyFromErrorCallback(arg)
Are they enough, Shigeru-san? Note that the internal CopyFrom() is
now implemented only with them, so I think file_fdw is also possible.
BTW, we might have another choice instead of GetCopyExecutorState()
because the EState will be used only for ResetPerTupleExprContext()
in file_fdw. If NextCopyFrom() returns a HeapTuple instead of values
and nulls arrays, we could hide EState in NextCopyFrom().
--
Itagaki Takahiro
Attachments:
copy_export-20101214.diffapplication/octet-stream; name=copy_export-20101214.diffDownload
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index 7b8bee8..57f6ec5 100644
*** a/src/backend/commands/copy.c
--- b/src/backend/commands/copy.c
*************** typedef struct CopyStateData
*** 99,105 ****
int client_encoding; /* remote side's character encoding */
bool need_transcoding; /* client encoding diff from server? */
bool encoding_embeds_ascii; /* ASCII can be non-first byte? */
- uint64 processed; /* # of tuples processed */
/* parameters from the COPY command */
Relation rel; /* relation to copy to or from */
--- 99,104 ----
*************** typedef struct CopyStateData
*** 119,125 ****
bool *force_quote_flags; /* per-column CSV FQ flags */
bool *force_notnull_flags; /* per-column CSV FNN flags */
! /* these are just for error messages, see copy_in_error_callback */
const char *cur_relname; /* table name for error messages */
int cur_lineno; /* line number for error messages */
const char *cur_attname; /* current att for error messages */
--- 118,124 ----
bool *force_quote_flags; /* per-column CSV FQ flags */
bool *force_notnull_flags; /* per-column CSV FNN flags */
! /* these are just for error messages, see CopyFromErrorCallback */
const char *cur_relname; /* table name for error messages */
int cur_lineno; /* line number for error messages */
const char *cur_attname; /* current att for error messages */
*************** typedef struct CopyStateData
*** 142,150 ****
StringInfoData attribute_buf;
/* field raw data pointers found by COPY FROM */
!
! int max_fields;
! char ** raw_fields;
/*
* Similarly, line_buf holds the whole input line being processed. The
--- 141,148 ----
StringInfoData attribute_buf;
/* field raw data pointers found by COPY FROM */
! int max_fields;
! char **raw_fields;
/*
* Similarly, line_buf holds the whole input line being processed. The
*************** typedef struct CopyStateData
*** 167,181 ****
char *raw_buf;
int raw_buf_index; /* next byte to process */
int raw_buf_len; /* total # of bytes stored */
- } CopyStateData;
! typedef CopyStateData *CopyState;
/* DestReceiver for COPY (SELECT) TO */
typedef struct
{
DestReceiver pub; /* publicly-known function pointers */
CopyState cstate; /* CopyStateData for the command */
} DR_copy;
--- 165,192 ----
char *raw_buf;
int raw_buf_index; /* next byte to process */
int raw_buf_len; /* total # of bytes stored */
! /*
! * The definition of input functions and default expressions are stored
! * in these variables.
! */
! EState *estate;
! AttrNumber num_defaults;
! bool file_has_oids;
! FmgrInfo oid_in_function;
! Oid oid_typioparam;
! FmgrInfo *in_functions;
! Oid *typioparams;
! int *defmap;
! ExprState **defexprs; /* array of default att expressions */
! } CopyStateData;
/* DestReceiver for COPY (SELECT) TO */
typedef struct
{
DestReceiver pub; /* publicly-known function pointers */
CopyState cstate; /* CopyStateData for the command */
+ uint64 processed; /* # of tuples processed */
} DR_copy;
*************** static const char BinarySignature[11] =
*** 248,258 ****
/* non-export function prototypes */
! static void DoCopyTo(CopyState cstate);
! static void CopyTo(CopyState cstate);
static void CopyOneRowTo(CopyState cstate, Oid tupleOid,
Datum *values, bool *nulls);
! static void CopyFrom(CopyState cstate);
static bool CopyReadLine(CopyState cstate);
static bool CopyReadLineText(CopyState cstate);
static int CopyReadAttributesText(CopyState cstate);
--- 259,275 ----
/* non-export function prototypes */
! static CopyState BeginCopy(bool is_from,
! Relation rel, Node *raw_query, const char *queryString,
! const char *filename, List *attnamelist, List *options);
! static CopyState BeginCopyTo(Relation rel, Node *query, const char *queryString,
! const char *filename, List *attnamelist, List *options);
! static void EndCopyTo(CopyState cstate);
! static uint64 DoCopyTo(CopyState cstate);
! static uint64 CopyTo(CopyState cstate);
static void CopyOneRowTo(CopyState cstate, Oid tupleOid,
Datum *values, bool *nulls);
! static uint64 CopyFrom(CopyState cstate, Relation rel);
static bool CopyReadLine(CopyState cstate);
static bool CopyReadLineText(CopyState cstate);
static int CopyReadAttributesText(CopyState cstate);
*************** CopyLoadRawBuf(CopyState cstate)
*** 718,730 ****
* Do not allow the copy if user doesn't have proper permission to access
* the table or the specifically requested columns.
*/
! uint64
! DoCopy(const CopyStmt *stmt, const char *queryString)
{
CopyState cstate;
- bool is_from = stmt->is_from;
- bool pipe = (stmt->filename == NULL);
- List *attnamelist = stmt->attlist;
List *force_quote = NIL;
List *force_notnull = NIL;
bool force_quote_all = false;
--- 735,746 ----
* Do not allow the copy if user doesn't have proper permission to access
* the table or the specifically requested columns.
*/
! static CopyState
! BeginCopy(bool is_from,
! Relation rel, Node *raw_query, const char *queryString,
! const char *filename, List *attnamelist, List *options)
{
CopyState cstate;
List *force_quote = NIL;
List *force_notnull = NIL;
bool force_quote_all = false;
*************** DoCopy(const CopyStmt *stmt, const char
*** 733,745 ****
ListCell *option;
TupleDesc tupDesc;
int num_phys_attrs;
- uint64 processed;
/* Allocate workspace and zero all fields */
cstate = (CopyStateData *) palloc0(sizeof(CopyStateData));
/* Extract options from the statement node tree */
! foreach(option, stmt->options)
{
DefElem *defel = (DefElem *) lfirst(option);
--- 749,760 ----
ListCell *option;
TupleDesc tupDesc;
int num_phys_attrs;
/* Allocate workspace and zero all fields */
cstate = (CopyStateData *) palloc0(sizeof(CopyStateData));
/* Extract options from the statement node tree */
! foreach(option, options)
{
DefElem *defel = (DefElem *) lfirst(option);
*************** DoCopy(const CopyStmt *stmt, const char
*** 980,1005 ****
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("CSV quote character must not appear in the NULL specification")));
! /* Disallow file COPY except to superusers. */
! if (!pipe && !superuser())
! ereport(ERROR,
! (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
! errmsg("must be superuser to COPY to or from a file"),
! errhint("Anyone can COPY to stdout or from stdin. "
! "psql's \\copy command also works for anyone.")));
!
! if (stmt->relation)
{
RangeTblEntry *rte;
List *attnums;
ListCell *cur;
! Assert(!stmt->query);
! cstate->queryDesc = NULL;
! /* Open and lock the relation, using the appropriate lock type. */
! cstate->rel = heap_openrv(stmt->relation,
! (is_from ? RowExclusiveLock : AccessShareLock));
tupDesc = RelationGetDescr(cstate->rel);
--- 995,1009 ----
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("CSV quote character must not appear in the NULL specification")));
! if (rel)
{
RangeTblEntry *rte;
List *attnums;
ListCell *cur;
! Assert(!raw_query);
! cstate->rel = rel;
tupDesc = RelationGetDescr(cstate->rel);
*************** DoCopy(const CopyStmt *stmt, const char
*** 1058,1064 ****
* function and is executed repeatedly. (See also the same hack in
* DECLARE CURSOR and PREPARE.) XXX FIXME someday.
*/
! rewritten = pg_analyze_and_rewrite((Node *) copyObject(stmt->query),
queryString, NULL, 0);
/* We don't expect more or less than one result query */
--- 1062,1068 ----
* function and is executed repeatedly. (See also the same hack in
* DECLARE CURSOR and PREPARE.) XXX FIXME someday.
*/
! rewritten = pg_analyze_and_rewrite((Node *) copyObject(raw_query),
queryString, NULL, 0);
/* We don't expect more or less than one result query */
*************** DoCopy(const CopyStmt *stmt, const char
*** 1160,1173 ****
}
}
- /* Set up variables to avoid per-attribute overhead. */
- initStringInfo(&cstate->attribute_buf);
- initStringInfo(&cstate->line_buf);
- cstate->line_buf_converted = false;
- cstate->raw_buf = (char *) palloc(RAW_BUF_SIZE + 1);
- cstate->raw_buf_index = cstate->raw_buf_len = 0;
- cstate->processed = 0;
-
/*
* Set up encoding conversion info. Even if the client and server
* encodings are the same, we must apply pg_client_to_server() to validate
--- 1164,1169 ----
*************** DoCopy(const CopyStmt *stmt, const char
*** 1181,1229 ****
cstate->encoding_embeds_ascii = PG_ENCODING_IS_CLIENT_ONLY(cstate->client_encoding);
cstate->copy_dest = COPY_FILE; /* default */
! cstate->filename = stmt->filename;
! if (is_from)
! CopyFrom(cstate); /* copy from file to database */
else
! DoCopyTo(cstate); /* copy from database to file */
/*
! * Close the relation or query. If reading, we can release the
! * AccessShareLock we got; if writing, we should hold the lock until end
! * of transaction to ensure that updates will be committed before lock is
! * released.
*/
! if (cstate->rel)
! heap_close(cstate->rel, (is_from ? NoLock : AccessShareLock));
else
{
! /* Close down the query and free resources. */
! ExecutorEnd(cstate->queryDesc);
! FreeQueryDesc(cstate->queryDesc);
! PopActiveSnapshot();
}
! /* Clean up storage (probably not really necessary) */
! processed = cstate->processed;
pfree(cstate->attribute_buf.data);
pfree(cstate->line_buf.data);
pfree(cstate->raw_buf);
pfree(cstate);
-
- return processed;
}
/*
* This intermediate routine exists mainly to localize the effects of setjmp
* so we don't need to plaster a lot of variables with "volatile".
*/
! static void
DoCopyTo(CopyState cstate)
{
bool pipe = (cstate->filename == NULL);
if (cstate->rel)
{
--- 1177,1722 ----
cstate->encoding_embeds_ascii = PG_ENCODING_IS_CLIENT_ONLY(cstate->client_encoding);
cstate->copy_dest = COPY_FILE; /* default */
! cstate->filename = (filename ? pstrdup(filename) : NULL);
! return cstate;
! }
!
! CopyState
! BeginCopyFrom(Relation rel,
! const char *filename,
! List *attnamelist,
! List *options)
! {
! CopyState cstate;
! bool pipe = (filename == NULL);
! TupleDesc tupDesc;
! Form_pg_attribute *attr;
! AttrNumber num_phys_attrs,
! attr_count,
! num_defaults;
! FmgrInfo *in_functions;
! Oid *typioparams;
! int attnum;
! Oid in_func_oid;
! EState *estate = CreateExecutorState(); /* for ExecPrepareExpr() */
! int *defmap;
! ExprState **defexprs;
!
! cstate = BeginCopy(true, rel, NULL, NULL, filename, attnamelist, options);
!
! /* Initialize state variables */
! cstate->fe_eof = false;
! cstate->eol_type = EOL_UNKNOWN;
! cstate->cur_relname = RelationGetRelationName(cstate->rel);
! cstate->cur_lineno = 0;
! cstate->cur_attname = NULL;
! cstate->cur_attval = NULL;
!
! /* Set up variables to avoid per-attribute overhead. */
! initStringInfo(&cstate->attribute_buf);
! initStringInfo(&cstate->line_buf);
! cstate->line_buf_converted = false;
! cstate->raw_buf = (char *) palloc(RAW_BUF_SIZE + 1);
! cstate->raw_buf_index = cstate->raw_buf_len = 0;
!
! tupDesc = RelationGetDescr(cstate->rel);
! attr = tupDesc->attrs;
! num_phys_attrs = tupDesc->natts;
! attr_count = list_length(cstate->attnumlist);
! num_defaults = 0;
!
! /*
! * Pick up the required catalog information for each attribute in the
! * relation, including the input function, the element type (to pass to
! * the input function), and info about defaults and constraints. (Which
! * input function we use depends on text/binary format choice.)
! */
! in_functions = (FmgrInfo *) palloc(num_phys_attrs * sizeof(FmgrInfo));
! typioparams = (Oid *) palloc(num_phys_attrs * sizeof(Oid));
! defmap = (int *) palloc(num_phys_attrs * sizeof(int));
! defexprs = (ExprState **) palloc(num_phys_attrs * sizeof(ExprState *));
!
! for (attnum = 1; attnum <= num_phys_attrs; attnum++)
! {
! /* We don't need info for dropped attributes */
! if (attr[attnum - 1]->attisdropped)
! continue;
!
! /* Fetch the input function and typioparam info */
! if (cstate->binary)
! getTypeBinaryInputInfo(attr[attnum - 1]->atttypid,
! &in_func_oid, &typioparams[attnum - 1]);
! else
! getTypeInputInfo(attr[attnum - 1]->atttypid,
! &in_func_oid, &typioparams[attnum - 1]);
! fmgr_info(in_func_oid, &in_functions[attnum - 1]);
!
! /* Get default info if needed */
! if (!list_member_int(cstate->attnumlist, attnum))
! {
! /* attribute is NOT to be copied from input */
! /* use default value if one exists */
! Node *defexpr = build_column_default(cstate->rel, attnum);
!
! if (defexpr != NULL)
! {
! defexprs[num_defaults] = ExecPrepareExpr((Expr *) defexpr,
! estate);
! defmap[num_defaults] = attnum - 1;
! num_defaults++;
! }
! }
! }
!
! /* We keep those variables in cstate. */
! cstate->estate = estate;
! cstate->in_functions = in_functions;
! cstate->typioparams = typioparams;
! cstate->defmap = defmap;
! cstate->defexprs = defexprs;
! cstate->num_defaults = num_defaults;
!
! if (pipe)
! {
! if (whereToSendOutput == DestRemote)
! ReceiveCopyBegin(cstate);
! else
! cstate->copy_file = stdin;
! }
else
! {
! struct stat st;
!
! cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_R);
!
! if (cstate->copy_file == NULL)
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not open file \"%s\" for reading: %m",
! cstate->filename)));
!
! fstat(fileno(cstate->copy_file), &st);
! if (S_ISDIR(st.st_mode))
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("\"%s\" is a directory", cstate->filename)));
! }
!
! if (!cstate->binary)
! {
! /* must rely on user to tell us... */
! cstate->file_has_oids = cstate->oids;
! }
! else
! {
! /* Read and verify binary header */
! char readSig[11];
! int32 tmp;
!
! /* Signature */
! if (CopyGetData(cstate, readSig, 11, 11) != 11 ||
! memcmp(readSig, BinarySignature, 11) != 0)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("COPY file signature not recognized")));
! /* Flags field */
! if (!CopyGetInt32(cstate, &tmp))
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid COPY file header (missing flags)")));
! cstate->file_has_oids = (tmp & (1 << 16)) != 0;
! tmp &= ~(1 << 16);
! if ((tmp >> 16) != 0)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unrecognized critical flags in COPY file header")));
! /* Header extension length */
! if (!CopyGetInt32(cstate, &tmp) ||
! tmp < 0)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid COPY file header (missing length)")));
! /* Skip extension header, if present */
! while (tmp-- > 0)
! {
! if (CopyGetData(cstate, readSig, 1, 1) != 1)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid COPY file header (wrong length)")));
! }
! }
!
! if (cstate->file_has_oids && cstate->binary)
! {
! getTypeBinaryInputInfo(OIDOID,
! &in_func_oid, &cstate->oid_typioparam);
! fmgr_info(in_func_oid, &cstate->oid_in_function);
! }
!
! /* create workspace for CopyReadAttributes results */
! if (!cstate->binary)
! {
! int nfields = cstate->file_has_oids ? (attr_count + 1) : attr_count;
!
! cstate->max_fields = nfields;
! cstate->raw_fields = (char **) palloc(nfields * sizeof(char *));
! }
!
! return cstate;
! }
!
! static CopyState
! BeginCopyTo(Relation rel,
! Node *query,
! const char *queryString,
! const char *filename,
! List *attnamelist,
! List *options)
! {
! return BeginCopy(false, rel, query, queryString,
! filename, attnamelist, options);
! }
!
! /* return false if no more tuples */
! bool
! NextCopyFrom(CopyState cstate, Datum *values, bool *nulls, Oid *tupleOid)
! {
! TupleDesc tupDesc;
! Form_pg_attribute *attr;
! AttrNumber num_phys_attrs,
! attr_count,
! num_defaults = cstate->num_defaults;
! FmgrInfo *in_functions = cstate->in_functions;
! Oid *typioparams = cstate->typioparams;
! int i;
! int nfields;
! char **field_strings;
! bool isnull;
! int *defmap = cstate->defmap;
! ExprState **defexprs = cstate->defexprs;
! ExprContext *econtext; /* used for ExecEvalExpr for default atts */
!
! /* on input just throw the header line away */
! if (cstate->cur_lineno == 0 && cstate->header_line)
! {
! cstate->cur_lineno++;
! if (CopyReadLine(cstate))
! return false; /* done */
! }
!
! cstate->cur_lineno++;
!
! tupDesc = RelationGetDescr(cstate->rel);
! attr = tupDesc->attrs;
! num_phys_attrs = tupDesc->natts;
! attr_count = list_length(cstate->attnumlist);
! nfields = cstate->file_has_oids ? (attr_count + 1) : attr_count;
!
! /* Initialize all values for row to NULL */
! MemSet(values, 0, num_phys_attrs * sizeof(Datum));
! MemSet(nulls, true, num_phys_attrs * sizeof(bool));
!
! if (!cstate->binary)
! {
! ListCell *cur;
! int fldct;
! int fieldno;
! char *string;
!
! /*
! * Actually read the line into memory here.
! * EOF at start of line means we're done. If we see EOF after
! * some characters, we act as though it was newline followed by
! * EOF, ie, process the line and then exit loop on next iteration.
! */
! if (CopyReadLine(cstate) && cstate->line_buf.len == 0)
! return false;
!
! /* Parse the line into de-escaped field values */
! if (cstate->csv_mode)
! fldct = CopyReadAttributesCSV(cstate);
! else
! fldct = CopyReadAttributesText(cstate);
!
! /* check for overflowing fields */
! if (nfields > 0 && fldct > nfields)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("extra data after last expected column")));
!
! fieldno = 0;
! field_strings = cstate->raw_fields;
!
! /* Read the OID field if present */
! if (cstate->file_has_oids)
! {
! if (fieldno >= fldct)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("missing data for OID column")));
! string = field_strings[fieldno++];
!
! if (string == NULL)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("null OID in COPY data")));
! else if (cstate->oids && tupleOid != NULL)
! {
! cstate->cur_attname = "oid";
! cstate->cur_attval = string;
! *tupleOid = DatumGetObjectId(DirectFunctionCall1(oidin,
! CStringGetDatum(string)));
! if (*tupleOid == InvalidOid)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid OID in COPY data")));
! cstate->cur_attname = NULL;
! cstate->cur_attval = NULL;
! }
! }
!
! /* Loop to read the user attributes on the line. */
! foreach(cur, cstate->attnumlist)
! {
! int attnum = lfirst_int(cur);
! int m = attnum - 1;
!
! if (fieldno >= fldct)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("missing data for column \"%s\"",
! NameStr(attr[m]->attname))));
! string = field_strings[fieldno++];
!
! if (cstate->csv_mode && string == NULL &&
! cstate->force_notnull_flags[m])
! {
! /* Go ahead and read the NULL string */
! string = cstate->null_print;
! }
!
! cstate->cur_attname = NameStr(attr[m]->attname);
! cstate->cur_attval = string;
! values[m] = InputFunctionCall(&in_functions[m],
! string,
! typioparams[m],
! attr[m]->atttypmod);
! if (string != NULL)
! nulls[m] = false;
! cstate->cur_attname = NULL;
! cstate->cur_attval = NULL;
! }
!
! Assert(fieldno == nfields);
! }
! else
! {
! /* binary */
! int16 fld_count;
! ListCell *cur;
!
! if (!CopyGetInt16(cstate, &fld_count))
! {
! /* EOF detected (end of file, or protocol-level EOF) */
! return false;
! }
!
! if (fld_count == -1)
! {
! /*
! * Received EOF marker. In a V3-protocol copy, wait for
! * the protocol-level EOF, and complain if it doesn't come
! * immediately. This ensures that we correctly handle
! * CopyFail, if client chooses to send that now.
! *
! * Note that we MUST NOT try to read more data in an
! * old-protocol copy, since there is no protocol-level EOF
! * marker then. We could go either way for copy from file,
! * but choose to throw error if there's data after the EOF
! * marker, for consistency with the new-protocol case.
! */
! char dummy;
!
! if (cstate->copy_dest != COPY_OLD_FE &&
! CopyGetData(cstate, &dummy, 1, 1) > 0)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("received copy data after EOF marker")));
! return false;
! }
!
! if (fld_count != attr_count)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("row field count is %d, expected %d",
! (int) fld_count, attr_count)));
!
! if (cstate->file_has_oids)
! {
! Oid oid;
!
! cstate->cur_attname = "oid";
! oid = DatumGetObjectId(CopyReadBinaryAttribute(cstate,
! 0,
! &cstate->oid_in_function,
! cstate->oid_typioparam,
! -1,
! &isnull));
! if (isnull || oid == InvalidOid)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid OID in COPY data")));
! cstate->cur_attname = NULL;
! if (cstate->oids && tupleOid != NULL)
! *tupleOid = oid;
! }
!
! i = 0;
! foreach(cur, cstate->attnumlist)
! {
! int attnum = lfirst_int(cur);
! int m = attnum - 1;
!
! cstate->cur_attname = NameStr(attr[m]->attname);
! i++;
! values[m] = CopyReadBinaryAttribute(cstate,
! i,
! &in_functions[m],
! typioparams[m],
! attr[m]->atttypmod,
! &nulls[m]);
! cstate->cur_attname = NULL;
! }
! }
/*
! * Now compute and insert any defaults available for the columns not
! * provided by the input data. Anything not processed here or above
! * will remain NULL.
*/
! econtext = GetPerTupleExprContext(cstate->estate);
! for (i = 0; i < num_defaults; i++)
! {
! values[defmap[i]] = ExecEvalExpr(defexprs[i], econtext,
! &nulls[defmap[i]], NULL);
! }
!
! return true;
! }
!
! uint64
! DoCopy(const CopyStmt *stmt, const char *queryString)
! {
! CopyState cstate;
! bool is_from = stmt->is_from;
! bool pipe = (stmt->filename == NULL);
! Relation rel;
! uint64 processed;
!
! /* Disallow file COPY except to superusers. */
! if (!pipe && !superuser())
! ereport(ERROR,
! (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
! errmsg("must be superuser to COPY to or from a file"),
! errhint("Anyone can COPY to stdout or from stdin. "
! "psql's \\copy command also works for anyone.")));
!
! if (stmt->relation)
! {
! Assert(!stmt->query);
!
! /* Open and lock the relation, using the appropriate lock type. */
! rel = heap_openrv(stmt->relation,
! (is_from ? RowExclusiveLock : AccessShareLock));
! }
else
{
! Assert(stmt->query);
!
! rel = NULL;
}
! if (is_from)
! {
! cstate = BeginCopyFrom(rel, stmt->filename, stmt->attlist, stmt->options);
! processed = CopyFrom(cstate, rel); /* copy from file to database */
! EndCopyFrom(cstate);
! }
! else
! {
! cstate = BeginCopyTo(rel, stmt->query, queryString, stmt->filename, stmt->attlist, stmt->options);
! processed = DoCopyTo(cstate); /* copy from database to file */
! EndCopyTo(cstate);
! }
+ /*
+ * Close the relation. If reading, we can release the AccessShareLock we got;
+ * if writing, we should hold the lock until end of transaction to ensure that
+ * updates will be committed before lock is released.
+ */
+ if (rel != NULL)
+ heap_close(rel, (is_from ? NoLock : AccessShareLock));
+
+ return processed;
+ }
+
+ void
+ EndCopyFrom(CopyState cstate)
+ {
+ FreeExecutorState(cstate->estate);
+
+ /* Clean up storage */
+ if (cstate->filename)
+ {
+ if (FreeFile(cstate->copy_file))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read from file \"%s\": %m",
+ cstate->filename)));
+ pfree(cstate->filename);
+ }
+ if (!cstate->binary)
+ pfree(cstate->raw_fields);
pfree(cstate->attribute_buf.data);
pfree(cstate->line_buf.data);
pfree(cstate->raw_buf);
+ pfree(cstate->in_functions);
+ pfree(cstate->typioparams);
+ pfree(cstate->defmap);
+ pfree(cstate->defexprs);
pfree(cstate);
}
+ static void
+ EndCopyTo(CopyState cstate)
+ {
+ /*
+ * Close the relation or query. We can release the AccessShareLock we got.
+ */
+ if (cstate->rel == NULL)
+ {
+ /* Close down the query and free resources. */
+ ExecutorEnd(cstate->queryDesc);
+ FreeQueryDesc(cstate->queryDesc);
+ PopActiveSnapshot();
+ }
+
+ /* Clean up storage */
+ if (cstate->filename)
+ pfree(cstate->filename);
+ pfree(cstate);
+ }
/*
* This intermediate routine exists mainly to localize the effects of setjmp
* so we don't need to plaster a lot of variables with "volatile".
*/
! static uint64
DoCopyTo(CopyState cstate)
{
bool pipe = (cstate->filename == NULL);
+ uint64 processed;
if (cstate->rel)
{
*************** DoCopyTo(CopyState cstate)
*** 1291,1297 ****
if (cstate->fe_copy)
SendCopyBegin(cstate);
! CopyTo(cstate);
if (cstate->fe_copy)
SendCopyEnd(cstate);
--- 1784,1790 ----
if (cstate->fe_copy)
SendCopyBegin(cstate);
! processed = CopyTo(cstate);
if (cstate->fe_copy)
SendCopyEnd(cstate);
*************** DoCopyTo(CopyState cstate)
*** 1316,1333 ****
errmsg("could not write to file \"%s\": %m",
cstate->filename)));
}
}
/*
* Copy from relation or query TO file.
*/
! static void
CopyTo(CopyState cstate)
{
TupleDesc tupDesc;
int num_phys_attrs;
Form_pg_attribute *attr;
ListCell *cur;
if (cstate->rel)
tupDesc = RelationGetDescr(cstate->rel);
--- 1809,1829 ----
errmsg("could not write to file \"%s\": %m",
cstate->filename)));
}
+
+ return processed;
}
/*
* Copy from relation or query TO file.
*/
! static uint64
CopyTo(CopyState cstate)
{
TupleDesc tupDesc;
int num_phys_attrs;
Form_pg_attribute *attr;
ListCell *cur;
+ uint64 processed;
if (cstate->rel)
tupDesc = RelationGetDescr(cstate->rel);
*************** CopyTo(CopyState cstate)
*** 1433,1438 ****
--- 1929,1935 ----
scandesc = heap_beginscan(cstate->rel, GetActiveSnapshot(), 0, NULL);
+ processed = 0;
while ((tuple = heap_getnext(scandesc, ForwardScanDirection)) != NULL)
{
CHECK_FOR_INTERRUPTS();
*************** CopyTo(CopyState cstate)
*** 1442,1447 ****
--- 1939,1945 ----
/* Format and send the data */
CopyOneRowTo(cstate, HeapTupleGetOid(tuple), values, nulls);
+ processed++;
}
heap_endscan(scandesc);
*************** CopyTo(CopyState cstate)
*** 1450,1455 ****
--- 1948,1954 ----
{
/* run the plan --- the dest receiver will send tuples */
ExecutorRun(cstate->queryDesc, ForwardScanDirection, 0L);
+ processed = ((DR_copy *) cstate->queryDesc->dest)->processed;
}
if (cstate->binary)
*************** CopyTo(CopyState cstate)
*** 1461,1466 ****
--- 1960,1967 ----
}
MemoryContextDelete(cstate->rowcontext);
+
+ return processed;
}
/*
*************** CopyOneRowTo(CopyState cstate, Oid tuple
*** 1552,1567 ****
CopySendEndOfRow(cstate);
MemoryContextSwitchTo(oldcontext);
-
- cstate->processed++;
}
/*
* error context callback for COPY FROM
*/
! static void
! copy_in_error_callback(void *arg)
{
CopyState cstate = (CopyState) arg;
--- 2053,2066 ----
CopySendEndOfRow(cstate);
MemoryContextSwitchTo(oldcontext);
}
/*
* error context callback for COPY FROM
*/
! void
! CopyFromErrorCallback(void *arg)
{
CopyState cstate = (CopyState) arg;
*************** limit_printout_length(const char *str)
*** 1663,1725 ****
/*
* Copy FROM file to relation.
*/
! static void
! CopyFrom(CopyState cstate)
{
- bool pipe = (cstate->filename == NULL);
HeapTuple tuple;
TupleDesc tupDesc;
- Form_pg_attribute *attr;
- AttrNumber num_phys_attrs,
- attr_count,
- num_defaults;
- FmgrInfo *in_functions;
- FmgrInfo oid_in_function;
- Oid *typioparams;
- Oid oid_typioparam;
- int attnum;
- int i;
- Oid in_func_oid;
Datum *values;
bool *nulls;
- int nfields;
- char **field_strings;
bool done = false;
- bool isnull;
ResultRelInfo *resultRelInfo;
! EState *estate = CreateExecutorState(); /* for ExecConstraints() */
TupleTableSlot *slot;
- bool file_has_oids;
- int *defmap;
- ExprState **defexprs; /* array of default att expressions */
- ExprContext *econtext; /* used for ExecEvalExpr for default atts */
MemoryContext oldcontext = CurrentMemoryContext;
ErrorContextCallback errcontext;
CommandId mycid = GetCurrentCommandId(true);
int hi_options = 0; /* start with default heap_insert options */
BulkInsertState bistate;
! Assert(cstate->rel);
! if (cstate->rel->rd_rel->relkind != RELKIND_RELATION)
{
! if (cstate->rel->rd_rel->relkind == RELKIND_VIEW)
ereport(ERROR,
(errcode(ERRCODE_WRONG_OBJECT_TYPE),
errmsg("cannot copy to view \"%s\"",
! RelationGetRelationName(cstate->rel))));
! else if (cstate->rel->rd_rel->relkind == RELKIND_SEQUENCE)
ereport(ERROR,
(errcode(ERRCODE_WRONG_OBJECT_TYPE),
errmsg("cannot copy to sequence \"%s\"",
! RelationGetRelationName(cstate->rel))));
else
ereport(ERROR,
(errcode(ERRCODE_WRONG_OBJECT_TYPE),
errmsg("cannot copy to non-table relation \"%s\"",
! RelationGetRelationName(cstate->rel))));
}
/*----------
* Check to see if we can avoid writing WAL
*
--- 2162,2209 ----
/*
* Copy FROM file to relation.
*/
! static uint64
! CopyFrom(CopyState cstate, Relation rel)
{
HeapTuple tuple;
TupleDesc tupDesc;
Datum *values;
bool *nulls;
bool done = false;
ResultRelInfo *resultRelInfo;
! EState *estate; /* for ExecConstraints() */
TupleTableSlot *slot;
MemoryContext oldcontext = CurrentMemoryContext;
ErrorContextCallback errcontext;
CommandId mycid = GetCurrentCommandId(true);
int hi_options = 0; /* start with default heap_insert options */
BulkInsertState bistate;
+ uint64 processed = 0;
! Assert(rel != NULL);
! if (rel->rd_rel->relkind != RELKIND_RELATION)
{
! if (rel->rd_rel->relkind == RELKIND_VIEW)
ereport(ERROR,
(errcode(ERRCODE_WRONG_OBJECT_TYPE),
errmsg("cannot copy to view \"%s\"",
! RelationGetRelationName(rel))));
! else if (rel->rd_rel->relkind == RELKIND_SEQUENCE)
ereport(ERROR,
(errcode(ERRCODE_WRONG_OBJECT_TYPE),
errmsg("cannot copy to sequence \"%s\"",
! RelationGetRelationName(rel))));
else
ereport(ERROR,
(errcode(ERRCODE_WRONG_OBJECT_TYPE),
errmsg("cannot copy to non-table relation \"%s\"",
! RelationGetRelationName(rel))));
}
+ estate = GetCopyExecutorState(cstate);
+ tupDesc = RelationGetDescr(rel);
+
/*----------
* Check to see if we can avoid writing WAL
*
*************** CopyFrom(CopyState cstate)
*** 1747,1792 ****
* no additional work to enforce that.
*----------
*/
! if (cstate->rel->rd_createSubid != InvalidSubTransactionId ||
! cstate->rel->rd_newRelfilenodeSubid != InvalidSubTransactionId)
{
hi_options |= HEAP_INSERT_SKIP_FSM;
if (!XLogIsNeeded())
hi_options |= HEAP_INSERT_SKIP_WAL;
}
- if (pipe)
- {
- if (whereToSendOutput == DestRemote)
- ReceiveCopyBegin(cstate);
- else
- cstate->copy_file = stdin;
- }
- else
- {
- struct stat st;
-
- cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_R);
-
- if (cstate->copy_file == NULL)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not open file \"%s\" for reading: %m",
- cstate->filename)));
-
- fstat(fileno(cstate->copy_file), &st);
- if (S_ISDIR(st.st_mode))
- ereport(ERROR,
- (errcode(ERRCODE_WRONG_OBJECT_TYPE),
- errmsg("\"%s\" is a directory", cstate->filename)));
- }
-
- tupDesc = RelationGetDescr(cstate->rel);
- attr = tupDesc->attrs;
- num_phys_attrs = tupDesc->natts;
- attr_count = list_length(cstate->attnumlist);
- num_defaults = 0;
-
/*
* We need a ResultRelInfo so we can use the regular executor's
* index-entry-making machinery. (There used to be a huge amount of code
--- 2231,2244 ----
* no additional work to enforce that.
*----------
*/
! if (rel->rd_createSubid != InvalidSubTransactionId ||
! rel->rd_newRelfilenodeSubid != InvalidSubTransactionId)
{
hi_options |= HEAP_INSERT_SKIP_FSM;
if (!XLogIsNeeded())
hi_options |= HEAP_INSERT_SKIP_WAL;
}
/*
* We need a ResultRelInfo so we can use the regular executor's
* index-entry-making machinery. (There used to be a huge amount of code
*************** CopyFrom(CopyState cstate)
*** 1794,1801 ****
*/
resultRelInfo = makeNode(ResultRelInfo);
resultRelInfo->ri_RangeTableIndex = 1; /* dummy */
! resultRelInfo->ri_RelationDesc = cstate->rel;
! resultRelInfo->ri_TrigDesc = CopyTriggerDesc(cstate->rel->trigdesc);
if (resultRelInfo->ri_TrigDesc)
{
resultRelInfo->ri_TrigFunctions = (FmgrInfo *)
--- 2246,2253 ----
*/
resultRelInfo = makeNode(ResultRelInfo);
resultRelInfo->ri_RangeTableIndex = 1; /* dummy */
! resultRelInfo->ri_RelationDesc = rel;
! resultRelInfo->ri_TrigDesc = CopyTriggerDesc(rel->trigdesc);
if (resultRelInfo->ri_TrigDesc)
{
resultRelInfo->ri_TrigFunctions = (FmgrInfo *)
*************** CopyFrom(CopyState cstate)
*** 1815,1865 ****
slot = ExecInitExtraTupleSlot(estate);
ExecSetSlotDescriptor(slot, tupDesc);
- econtext = GetPerTupleExprContext(estate);
-
- /*
- * Pick up the required catalog information for each attribute in the
- * relation, including the input function, the element type (to pass to
- * the input function), and info about defaults and constraints. (Which
- * input function we use depends on text/binary format choice.)
- */
- in_functions = (FmgrInfo *) palloc(num_phys_attrs * sizeof(FmgrInfo));
- typioparams = (Oid *) palloc(num_phys_attrs * sizeof(Oid));
- defmap = (int *) palloc(num_phys_attrs * sizeof(int));
- defexprs = (ExprState **) palloc(num_phys_attrs * sizeof(ExprState *));
-
- for (attnum = 1; attnum <= num_phys_attrs; attnum++)
- {
- /* We don't need info for dropped attributes */
- if (attr[attnum - 1]->attisdropped)
- continue;
-
- /* Fetch the input function and typioparam info */
- if (cstate->binary)
- getTypeBinaryInputInfo(attr[attnum - 1]->atttypid,
- &in_func_oid, &typioparams[attnum - 1]);
- else
- getTypeInputInfo(attr[attnum - 1]->atttypid,
- &in_func_oid, &typioparams[attnum - 1]);
- fmgr_info(in_func_oid, &in_functions[attnum - 1]);
-
- /* Get default info if needed */
- if (!list_member_int(cstate->attnumlist, attnum))
- {
- /* attribute is NOT to be copied from input */
- /* use default value if one exists */
- Node *defexpr = build_column_default(cstate->rel, attnum);
-
- if (defexpr != NULL)
- {
- defexprs[num_defaults] = ExecPrepareExpr((Expr *) defexpr,
- estate);
- defmap[num_defaults] = attnum - 1;
- num_defaults++;
- }
- }
- }
-
/* Prepare to catch AFTER triggers. */
AfterTriggerBeginQuery();
--- 2267,2272 ----
*************** CopyFrom(CopyState cstate)
*** 1871,1958 ****
*/
ExecBSInsertTriggers(estate, resultRelInfo);
! if (!cstate->binary)
! file_has_oids = cstate->oids; /* must rely on user to tell us... */
! else
! {
! /* Read and verify binary header */
! char readSig[11];
! int32 tmp;
!
! /* Signature */
! if (CopyGetData(cstate, readSig, 11, 11) != 11 ||
! memcmp(readSig, BinarySignature, 11) != 0)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("COPY file signature not recognized")));
! /* Flags field */
! if (!CopyGetInt32(cstate, &tmp))
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid COPY file header (missing flags)")));
! file_has_oids = (tmp & (1 << 16)) != 0;
! tmp &= ~(1 << 16);
! if ((tmp >> 16) != 0)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unrecognized critical flags in COPY file header")));
! /* Header extension length */
! if (!CopyGetInt32(cstate, &tmp) ||
! tmp < 0)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid COPY file header (missing length)")));
! /* Skip extension header, if present */
! while (tmp-- > 0)
! {
! if (CopyGetData(cstate, readSig, 1, 1) != 1)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid COPY file header (wrong length)")));
! }
! }
!
! if (file_has_oids && cstate->binary)
! {
! getTypeBinaryInputInfo(OIDOID,
! &in_func_oid, &oid_typioparam);
! fmgr_info(in_func_oid, &oid_in_function);
! }
!
! values = (Datum *) palloc(num_phys_attrs * sizeof(Datum));
! nulls = (bool *) palloc(num_phys_attrs * sizeof(bool));
!
! /* create workspace for CopyReadAttributes results */
! nfields = file_has_oids ? (attr_count + 1) : attr_count;
! if (! cstate->binary)
! {
! cstate->max_fields = nfields;
! cstate->raw_fields = (char **) palloc(nfields * sizeof(char *));
! }
!
! /* Initialize state variables */
! cstate->fe_eof = false;
! cstate->eol_type = EOL_UNKNOWN;
! cstate->cur_relname = RelationGetRelationName(cstate->rel);
! cstate->cur_lineno = 0;
! cstate->cur_attname = NULL;
! cstate->cur_attval = NULL;
bistate = GetBulkInsertState();
/* Set up callback to identify error line number */
! errcontext.callback = copy_in_error_callback;
errcontext.arg = (void *) cstate;
errcontext.previous = error_context_stack;
error_context_stack = &errcontext;
- /* on input just throw the header line away */
- if (cstate->header_line)
- {
- cstate->cur_lineno++;
- done = CopyReadLine(cstate);
- }
-
while (!done)
{
bool skip_tuple;
--- 2278,2294 ----
*/
ExecBSInsertTriggers(estate, resultRelInfo);
! values = (Datum *) palloc(tupDesc->natts * sizeof(Datum));
! nulls = (bool *) palloc(tupDesc->natts * sizeof(bool));
bistate = GetBulkInsertState();
/* Set up callback to identify error line number */
! errcontext.callback = CopyFromErrorCallback;
errcontext.arg = (void *) cstate;
errcontext.previous = error_context_stack;
error_context_stack = &errcontext;
while (!done)
{
bool skip_tuple;
*************** CopyFrom(CopyState cstate)
*** 1960,2166 ****
CHECK_FOR_INTERRUPTS();
- cstate->cur_lineno++;
-
/* Reset the per-tuple exprcontext */
ResetPerTupleExprContext(estate);
/* Switch into its memory context */
MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
! /* Initialize all values for row to NULL */
! MemSet(values, 0, num_phys_attrs * sizeof(Datum));
! MemSet(nulls, true, num_phys_attrs * sizeof(bool));
!
! if (!cstate->binary)
! {
! ListCell *cur;
! int fldct;
! int fieldno;
! char *string;
!
! /* Actually read the line into memory here */
! done = CopyReadLine(cstate);
!
! /*
! * EOF at start of line means we're done. If we see EOF after
! * some characters, we act as though it was newline followed by
! * EOF, ie, process the line and then exit loop on next iteration.
! */
! if (done && cstate->line_buf.len == 0)
! break;
!
! /* Parse the line into de-escaped field values */
! if (cstate->csv_mode)
! fldct = CopyReadAttributesCSV(cstate);
! else
! fldct = CopyReadAttributesText(cstate);
!
! /* check for overflowing fields */
! if (nfields > 0 && fldct > nfields)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("extra data after last expected column")));
!
! fieldno = 0;
! field_strings = cstate->raw_fields;
!
! /* Read the OID field if present */
! if (file_has_oids)
! {
! if (fieldno >= fldct)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("missing data for OID column")));
! string = field_strings[fieldno++];
!
! if (string == NULL)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("null OID in COPY data")));
! else
! {
! cstate->cur_attname = "oid";
! cstate->cur_attval = string;
! loaded_oid = DatumGetObjectId(DirectFunctionCall1(oidin,
! CStringGetDatum(string)));
! if (loaded_oid == InvalidOid)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid OID in COPY data")));
! cstate->cur_attname = NULL;
! cstate->cur_attval = NULL;
! }
! }
!
! /* Loop to read the user attributes on the line. */
! foreach(cur, cstate->attnumlist)
! {
! int attnum = lfirst_int(cur);
! int m = attnum - 1;
!
! if (fieldno >= fldct)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("missing data for column \"%s\"",
! NameStr(attr[m]->attname))));
! string = field_strings[fieldno++];
!
! if (cstate->csv_mode && string == NULL &&
! cstate->force_notnull_flags[m])
! {
! /* Go ahead and read the NULL string */
! string = cstate->null_print;
! }
!
! cstate->cur_attname = NameStr(attr[m]->attname);
! cstate->cur_attval = string;
! values[m] = InputFunctionCall(&in_functions[m],
! string,
! typioparams[m],
! attr[m]->atttypmod);
! if (string != NULL)
! nulls[m] = false;
! cstate->cur_attname = NULL;
! cstate->cur_attval = NULL;
! }
!
! Assert(fieldno == nfields);
! }
! else
! {
! /* binary */
! int16 fld_count;
! ListCell *cur;
!
! if (!CopyGetInt16(cstate, &fld_count))
! {
! /* EOF detected (end of file, or protocol-level EOF) */
! done = true;
! break;
! }
!
! if (fld_count == -1)
! {
! /*
! * Received EOF marker. In a V3-protocol copy, wait for
! * the protocol-level EOF, and complain if it doesn't come
! * immediately. This ensures that we correctly handle
! * CopyFail, if client chooses to send that now.
! *
! * Note that we MUST NOT try to read more data in an
! * old-protocol copy, since there is no protocol-level EOF
! * marker then. We could go either way for copy from file,
! * but choose to throw error if there's data after the EOF
! * marker, for consistency with the new-protocol case.
! */
! char dummy;
!
! if (cstate->copy_dest != COPY_OLD_FE &&
! CopyGetData(cstate, &dummy, 1, 1) > 0)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("received copy data after EOF marker")));
! done = true;
! break;
! }
!
! if (fld_count != attr_count)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("row field count is %d, expected %d",
! (int) fld_count, attr_count)));
!
! if (file_has_oids)
! {
! cstate->cur_attname = "oid";
! loaded_oid =
! DatumGetObjectId(CopyReadBinaryAttribute(cstate,
! 0,
! &oid_in_function,
! oid_typioparam,
! -1,
! &isnull));
! if (isnull || loaded_oid == InvalidOid)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid OID in COPY data")));
! cstate->cur_attname = NULL;
! }
!
! i = 0;
! foreach(cur, cstate->attnumlist)
! {
! int attnum = lfirst_int(cur);
! int m = attnum - 1;
!
! cstate->cur_attname = NameStr(attr[m]->attname);
! i++;
! values[m] = CopyReadBinaryAttribute(cstate,
! i,
! &in_functions[m],
! typioparams[m],
! attr[m]->atttypmod,
! &nulls[m]);
! cstate->cur_attname = NULL;
! }
! }
!
! /*
! * Now compute and insert any defaults available for the columns not
! * provided by the input data. Anything not processed here or above
! * will remain NULL.
! */
! for (i = 0; i < num_defaults; i++)
! {
! values[defmap[i]] = ExecEvalExpr(defexprs[i], econtext,
! &nulls[defmap[i]], NULL);
! }
/* And now we can form the input tuple. */
tuple = heap_form_tuple(tupDesc, values, nulls);
! if (cstate->oids && file_has_oids)
HeapTupleSetOid(tuple, loaded_oid);
/* Triggers and stuff need to be invoked in query context. */
--- 2296,2315 ----
CHECK_FOR_INTERRUPTS();
/* Reset the per-tuple exprcontext */
ResetPerTupleExprContext(estate);
/* Switch into its memory context */
MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
! done = !NextCopyFrom(cstate, values, nulls, &loaded_oid);
! if (done)
! break;
/* And now we can form the input tuple. */
tuple = heap_form_tuple(tupDesc, values, nulls);
! if (loaded_oid != InvalidOid)
HeapTupleSetOid(tuple, loaded_oid);
/* Triggers and stuff need to be invoked in query context. */
*************** CopyFrom(CopyState cstate)
*** 2193,2203 ****
ExecStoreTuple(tuple, slot, InvalidBuffer, false);
/* Check the constraints of the tuple */
! if (cstate->rel->rd_att->constr)
ExecConstraints(resultRelInfo, slot, estate);
/* OK, store the tuple and create index entries for it */
! heap_insert(cstate->rel, tuple, mycid, hi_options, bistate);
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self),
--- 2342,2352 ----
ExecStoreTuple(tuple, slot, InvalidBuffer, false);
/* Check the constraints of the tuple */
! if (rel->rd_att->constr)
ExecConstraints(resultRelInfo, slot, estate);
/* OK, store the tuple and create index entries for it */
! heap_insert(rel, tuple, mycid, hi_options, bistate);
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self),
*************** CopyFrom(CopyState cstate)
*** 2214,2220 ****
* this is the same definition used by execMain.c for counting
* tuples inserted by an INSERT command.
*/
! cstate->processed++;
}
}
--- 2363,2369 ----
* this is the same definition used by execMain.c for counting
* tuples inserted by an INSERT command.
*/
! processed++;
}
}
*************** CopyFrom(CopyState cstate)
*** 2233,2267 ****
pfree(values);
pfree(nulls);
- if (! cstate->binary)
- pfree(cstate->raw_fields);
-
- pfree(in_functions);
- pfree(typioparams);
- pfree(defmap);
- pfree(defexprs);
ExecResetTupleTable(estate->es_tupleTable, false);
ExecCloseIndices(resultRelInfo);
- FreeExecutorState(estate);
-
- if (!pipe)
- {
- if (FreeFile(cstate->copy_file))
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not read from file \"%s\": %m",
- cstate->filename)));
- }
-
/*
* If we skipped writing WAL, then we need to sync the heap (but not
* indexes since those use WAL anyway)
*/
if (hi_options & HEAP_INSERT_SKIP_WAL)
! heap_sync(cstate->rel);
}
--- 2382,2400 ----
pfree(values);
pfree(nulls);
ExecResetTupleTable(estate->es_tupleTable, false);
ExecCloseIndices(resultRelInfo);
/*
* If we skipped writing WAL, then we need to sync the heap (but not
* indexes since those use WAL anyway)
*/
if (hi_options & HEAP_INSERT_SKIP_WAL)
! heap_sync(rel);
!
! return processed;
}
*************** CopyGetAttnums(TupleDesc tupDesc, Relati
*** 3502,3507 ****
--- 3635,3647 ----
return attnums;
}
+ /* retrieve internal EState in CopyState */
+ EState *
+ GetCopyExecutorState(CopyState cstate)
+ {
+ return cstate->estate;
+ }
+
/*
* copy_dest_startup --- executor startup
*************** copy_dest_receive(TupleTableSlot *slot,
*** 3526,3531 ****
--- 3666,3672 ----
/* And send the data */
CopyOneRowTo(cstate, InvalidOid, slot->tts_values, slot->tts_isnull);
+ myState->processed++;
}
/*
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index 6d409e8..8c69703 100644
*** a/src/include/commands/copy.h
--- b/src/include/commands/copy.h
***************
*** 14,25 ****
--- 14,36 ----
#ifndef COPY_H
#define COPY_H
+ #include "nodes/execnodes.h"
#include "nodes/parsenodes.h"
#include "tcop/dest.h"
+ typedef struct CopyStateData *CopyState;
+
extern uint64 DoCopy(const CopyStmt *stmt, const char *queryString);
+ extern CopyState BeginCopyFrom(Relation rel, const char *filename,
+ List *attnamelist, List *options);
+ extern void EndCopyFrom(CopyState cstate);
+ extern bool NextCopyFrom(CopyState cstate,
+ Datum *values, bool *nulls, Oid *tupleOid);
+ extern EState *GetCopyExecutorState(CopyState cstate);
+ extern void CopyFromErrorCallback(void *arg);
+
extern DestReceiver *CreateCopyDestReceiver(void);
#endif /* COPY_H */
On Tue, 14 Dec 2010 12:01:36 +0900
Itagaki Takahiro <itagaki.takahiro@gmail.com> wrote:
On Tue, Dec 14, 2010 at 01:25, Andrew Dunstan <andrew@dunslane.net> wrote:
On 12/13/2010 11:12 AM, Tom Lane wrote:
In that case I guess I'll need to do what Shigeru-san has done, and copy
large parts of copy.c.I found file_fdw would require the executor state in CopyState and
the error callback function. I revised the patch to export them.
Now 5 functions are exported from copy.c:- BeginCopyFrom(rel, filename, attnamelist, options) : CopyState
- EndCopyFrom(cstate) : void
- NextCopyFrom(cstate, OUT values, OUT nulls, OUT tupleOid) : bool
- GetCopyExecutorState(cstate) : EState *
- CopyFromErrorCallback(arg)Are they enough, Shigeru-san? Note that the internal CopyFrom() is
now implemented only with them, so I think file_fdw is also possible.
In addition to above, ResetCopyFrom() is necessary to support nested
loops which inner node is a ForeignScan.
On the other hand, I think that MarkPos()/RestrPos() wouldn't be
necessary until ForeignScan supports ordered output. ForeignScan
can't become direct child of MergeJoin because ForeignScan is not an
ordered scan, at least in current SQL/MED implementation.
Regards,
--
Shigeru Hanada
On Tue, Dec 14, 2010 at 15:31, Shigeru HANADA <hanada@metrosystems.co.jp> wrote:
- BeginCopyFrom(rel, filename, attnamelist, options) : CopyState
- EndCopyFrom(cstate) : void
- NextCopyFrom(cstate, OUT values, OUT nulls, OUT tupleOid) : bool
- GetCopyExecutorState(cstate) : EState *
- CopyFromErrorCallback(arg)Are they enough, Shigeru-san? Note that the internal CopyFrom() is
now implemented only with them, so I think file_fdw is also possible.In addition to above, ResetCopyFrom() is necessary to support nested
loops which inner node is a ForeignScan.
I think you can add ResetCopyFrom() to the core in your next file_fdw
patch because the function is not used by COPY command.
I'll note other differences between the API and your FileState:
- There are no superuser checks in the exported functions because
the restriction should be only at CREATE/ALTER FOREIGN TABLE.
If the superuser grants SELECT privileges to normal users, they
should be able to read the file contents.
(But we might need to hide the file path.)
- errcontext and values/nulls arrays are not included in CopyState.
They will be additionally kept in a holder of the CopyState.
- You need to pass non-NULL filename. If it is NULL, the server
tries to read data from the client.
- The framework supports to read dumped binary files and files
with OIDs. If you don't want to support them, please check
parameters not to include those options.
--
Itagaki Takahiro
Hi hackers,
Attached is the revised WIP version of file_fdw patch. This patch
should be applied after both of fdw_syntax and fdw_scan patches, which
have been posted to another thread "SQL/MED - core functionality".
In this version, file_fdw consists of two parts, file_fdw core part
and copy of COPY FROM codes as they were in last version. The reason
of this form is to make it possible to test actual SELECT statement
ASAP. I'll revise file_fdw again according to Itagaki-san's
export-copy-routines patch.
Note that this version of file_fdw doesn't support force_not_null
option because column-level generic option is not supported by current
fdw_syntax. It will be available if column-level generic option is
implemented.
And, as possible implementation of FDW-specific EXPLAIN information,
EXPLAIN SELECT xxx FROM file shows name and size of the file. It
may be better to hide file information if the user was not superuser
for security reason. If so, filename option should not appear in
output of \det psql command too.
Regards,
--
Shigeru Hanada
Attachments:
file_fdw.patch.gzapplication/octet-stream; name=file_fdw.patch.gzDownload
���M file_fdw.patch �<kw�6���_�uok���v���c����h�����fo��C����"������wf � E�N_���S� f0�@w<f�������?
�Q�����]���R���;���8�?�v��{��w������������x6j�Z���?�{v���� �H�v�{�v��bk�
�J%�����x�K�w<�=�=k�=���0�:��,�����p���Y�!����D��������������G�d�_N�z������@��X��~1:p^�����}{�)f��2J5���Y��������U�y����"���
|��A��`6�#��0�C����,Zl�z|��� ��g��������|�t:���,��������X�px�^O-����z�A�a�S@��� ��8�N��cE[��|pH���k�u8k��7�� x2�h�W���bq������4�y��ALB`��?��� �
����^���7?��o"������az���~������b ��m���
+6���W�N�^���w�����"��������14��\
�3����g��a����q�d<r����#.mF#�"�"�xd��e�/]�>�{x��z�+��r����������F�E�]����=��S�w���/�-�@��[!�q=����As8�]/2�����c��A/=o��k_���~v^��;v��������}h������K=�aQcwB���� ����r�����
�`>����^z�����=��{H�l��(���9�^o��\��JC�v����F��4��X=��[����OH�R����qi�����HT�~C�89�����
)��yT�� �xy���~�~��8~stT�9:�O����/^��������{���D��]G)�H�������{�
4�k�P6H.�$�%��@I2�����9�,�<b������_�[��T��#EC���y���u'C�"*�^C�_��<tF�Wi �X�1���~�s��k��G���6����w��0��Q�du�1��H�u]+�?��d��]a{;�;Uv�h��?��%�&v���g���e�sC��}=�\tZ�A�{���� ������h��9���f���!V��aYO�1� T�&
p8�����u|�������|O0�AH&��B>�w+�
�w��bk���g��-g�.�����=@.-y2s�!(����2h��(�KK���������@���
�_�-f��{���@�]���:��
%~�\� g�GS��,�u��RR�`�X&�S"�X�@Wh5�'�QtlG�(�p]B��`\�,"fO��mr��q$��u�R�-(��>:.����$���p����,�����p�/��_ZC�:r�`�OXh�-�,�F�������E���)�2u�:��+��� &43�H�|�iE�������Jc}�E�`�R�t�'���-�\�3���f�������J��I���;`E�]�rc�ji?�!mp�%k&<�_� �6mVK:+���8�A�q����;! 0������u���D5��A��C:A�Tk���SnA.����@ #9����c��'�}��O�����@����w�W���=N~�b�!pl�h4��F�s}�@5#�o��p��O��z��@A�1iA������g�V��"���-tR�f��_9^ 4E��
�����xC\��K1���g��+���j���'��:�q�2��?�v��� A d*:�,"$��}�����&.�Lu1�d`2+�fW��(��%L�wR(:%���8�U6�a�I.�(����"k���f�B�
��'o�G���|(����*�����2������c����� }%K7�;��z>G;��Q��+���Xy���Hj��i�4fUVtg\p�� �-pT`����(('C���s�
��4����]`�UF��Z�2|�i��c$�v��&~={V!���z�''�8���w����lb�*6TdG"*�<�A��F������L��X�"���A� ���{����]t{�����e|�k���{U�o����[�!������%��4���i��w�=��F��v���i�Q�&�0�(�A H!w$�+�c�����\_t��w����������'�G��h) ���!�4Pm'
-_���������= C`����N�bDJZ�OX:��9/�jU��ZX�� m���Q@�
�k:��LW�0{�C�H�X�E���@%*��7�[����j��\"�$&����
��I:������*Y=i���*
�k��)���)��z���>���z�S�$'S�H)�/������EI�;v9x(��{���Hh��H�,6u�(��p�)#��:
�H�)&�k�����n��Y���(��mQ�E��u��J��o�N\e���o�f�����$;c���A�&{�67M]���ng ���2)zQ�ahC����y{�����U�|��!�n�kW31)o*��;����0SF0 ��)ofS
��)����-^1Z,�kZ���P�^{p���u�We��jc��r�luVZullZk%��0��?EP� �JV8� �-� �o �@-�Q \���$�(���;�z���A{���n�����,q��wh���Cz�����2���lTyCe`���<:�-2_�?M6V���,q����'s95���������6z��.�0�M+H8��;
�d�c�E��ug����]3�O��e+���NX3����!$6��kM8�q��b��q����(H/��`�����N�w,]����kWq���'q��X=�������I����G�J�H�02�R�(p
[�#l�,P�Z�8c]��cc��*�A�*7���O������%{�K��S�W������5��������7���g�<������;�{��_�<8<JQ<?��������������mT���r������,�����J��V�=�AS5$qH���{*���-%�3���3�6������6����e���J!'�Ta��eX��>��UkLK��R j��B��,�xi&����&���O�,�qK+�.,����}�V"�\z�%�JA�EWd�nH�z���������+|����:����
������t��'�Ma�$U�\���4&5��I�X�Y�L�U���,������{����A+�X����4��J��7I�3y���i�$G+;�YO0w��W���C�6�n^l�w4%���,�*#���a���@�o��U��<<��0�,^������W��X����?���jZub������cf�s&�:�6����"���Sbby���@��BU�%$6
<G`a0�'����,�m�x��'l�@@�����9^R����%��T��g��2g�u
��-�����R��?�����d�5��eg�5��u�1��d3��e�q��=4����w��'�f���Wff[����Y���%�a���i(���.�Sq���qA{����o�]�P���)�*D��u���=����<q�������qf�]�C�?�`�<7�7���q �'�V���9��6U�g���#���]��TK��Kr !��{����*� 3XnO���N� =\� s��geu��,����g�\�Z��xA�TY�V����������F���In�,�*��k�x3[ ~�����@
�����B%I���)��2-)� �s��-��� KC�S�.ux�VZ�vT�
V [�g�����\�q�@�����K\��<�������6�0�B,����aR������4�:!����^2}���G�_�)1��;����gH��@�j��P��qa<�e
����N��[����@���F����.J���1H�������"�?vU��+��7��$<�!����E�`HG���Q)��V���.�Lt(����3c-sJL�������T������)�0
��_"�Sg�������!��0J�����=[�,���r�� �������Nh&��iwP6�j���%������x_)g�29����~7>�b/���iB�9u��-
"�IP�x��iQr�L���'�Qt9p���!����t���2��� �$��e{�5� 9���K���q)�u���v�x$�����$M��\0! ���r�Kb��\`��2]k���qF��K�3g��E���r?�rkNW0��I��!"��]H��5����7�����{������l �$d�JQ����K2��*D�MFOUS27�|�%iS:�p��(�?�(�L�� ��UY+1��gA��R�K���%�>�G<U���`"�_��]4�W��/�x�������\��t�
����P � ����a�!�~G��`���'�>�E�d�f�2u9�,���R���E �(�{A9�� .������t}���+���IHJ�O�$��C�u���kG+%��� m,ft�z�J(t�h����d���ps��\�K�fQX��$
Fr���j��_�<���Mi���gJ)��,h��u������uH�8��u�R�5��}|"�d�Y>�S�^���JO�� �����d���E�\����?�J�����f�$J=�����k��`�2yA��
c9����� �������DQ�.�D��wtd�@3�����-dR��w�z���u<q �[�� ���R��>����!����<B$?$C��F�������JLN+wMK������_�$�h8��=-�����P
�C��U��jid}z���HGR:+�2�'�����%���&���%��P1E_�T���*/��q#]5ri���^3��;E���:��H~p�W��O
����r^ ���#p��
��c��z.9Z��H�� |�0DD��\�sI�2�����������/;�a��n��^���_I�)9f�����_71U
����N����
���;YL�}��� ���. ��G���P^^���#� D�)��7������p/�@piH)�i��(m�N�a
����D�P%�|��������������I��R�B`S���,E�X��G�xJ�uX��fh �/A�C��n�~Y$�
�RD0KO����T_��w��P�q����fO�i��z���5����C���)W�k�;b��=��#&a��\����a
vv��G��d��=���b���N�X�#2N����o���^Y�����_(�7�Vx���������D~nH���Y��yA����
�������o���[O|*>{u��u�7���e��e9���^�*��#�C��G����OD���TK�/f �����\&>���jH(�-���^�d8e��������B{
�
A=8#`\�8�A�tU�B���8����Ks��=���j3��/ge�� ? �ey~����}��^��4o���Vuk���j^_�6/�����^�5P�?� }�\6i0:'E
|!E����7�^��6�����z��'*�zD�����*�������Axn�:$�W������� �>������6����Y�X����s1������|h�p���n�,�8�F��������
�TK�>��=:zY��7�kP�cF71_[��.�;�F�4�-X���,���p�z��a������k^p�[����w�l�+j
��pa��?�^������'���"t#�P�J}��|/��l,�w���~����
���&+��N�����o��xu�����+�r�{z��v�*xk��_���(5@�}& ����Y�)���O�����`\���a�����1�?�������#��o�)��{��%�'�va�_����YZ���>���n������w��jWu�$���'^Y���U�����3��
�$��8�d�s���(H:'G ���������������~=>Bn����u�=;D,�#\��#V��^�c��(���sv���
�p��
��4%3�P1&l����S�2}7�}D�0J������wq��Q
s�(�#�
�KSWh��t��9��-GI>�(�+ \�EJf�vt��.�����Bt
e�6Vgk �k��*�8�YXa��+�6�{��FY�A�u�����F� 3���n�N4Q��25W(��h��~p�M����oB����jT4/}H�$���UMZ��������������9��t�#�J�QW�Ml�v���*]-�������3D�3X�a\��c���<^Fhl��0���Eh������r@������W��77��q�v���/�A���~�bu���:f���{T1�����#�(��KGY������H7>��[#�x���.b���{���F�v�<��%��0=����)DO�����|���b ��������v�r|��=
�45`\�+A"��������\�
�w:iq�MZ���*�0�����M���d�=�7:�$2F���� �����Q����[$ ���v�
�
i� ��\�������x6��E zn]��=~_��5���."UT%\�������i���A�x������~�����\x���,AK����D�_A����]F���]"����a'���7�ah��l����6�q����T������X��������Nn�>�����K�w�z�?Y���������n��0�aar���8G��MJ�eE��_�� ��e���uP$��H��?�C��!t�bz�C�
��>{��yE�J����������lv�����l"(5���9o�I��C���,$�E�������l�Cs�
@�J�*�h�e5��d�9%�}�#�'�y?q�?$60�s�������M��3�%���P�r�3��[�T�]x�i2B��TE2"�[�o>$Ki<�3-��DTbq��7��J\J�( w�����16�K�x�EJ\7`����5���~qi�TR��7�Q����
���*����cL.�����G�Y&�@��bSw�|���*3!QHC���pB�S��c��=�1
,;�b����_�n��GA���O(p��|;RPO�'f(��3�qNl l�����m������f�f�j��+P��U���X��1v��G
�f��GO�;�{Ul�@ z�qy���4��|t����k�U�,&6 �|�1�q����W�/��cV��S<P�Q5t(L{Ms� Ch6|����U~��d"��,��� I���Ro"n�;p��Sd�jq�'�����"3n:R�Z��2�% '����C�1��|���X3�z�����`Z�� �� *�pA��Z�'��2-����Q��s8�dHV����zh��F�l�Zx��H*�h�z�H�},5�aQ��C��i�U���M^G�%AJi{���H7k�����,�c���9�$ ���h��8z�
+.�e*~�����V�(*N����zw�;���N���_>��V�:R�}�pG����6)3����.^r1#����8yW��(3 h�a��ej%�t��$��S[$$0��D��`Fh%#��!��z����� A}7D3�J����i�:`SG:�'[d,��:�_�\��C����I��.�t�x�����[_G�������@����G���.���
_Y� ���6�q�#��^f�����\g��E��']1��1��!h�3�f���)vR���E�a�����s�G�E|�-��pe�F������"6�Y��Q�O�zr�R��V���
��&� �f��������v����
�@;�o�_8bW��fm�[�[�m^y'��J�I2�3�Dd$yzg�3����3�$b;1���qF�Fp�49h��S\/����w���N;�������A�����99D������N�n8_�������������9��Y=�T��/"uDQ�y�*���l�u�I%k����z��$��a�g@����(����fG�����_�`����e@�}>��<�H�r3��T���]^,�E��W�������U��{
��S�)��h�}�.�Y
��mA;�[h;�(Fqq���E����EG�$��0Z�}LI������VZ9$r��Y%������e�6�C8�xp%�d�_F�]�s$3R>��G��\b+��Kq�o����`�� ����v�1e�'��./~��G*��E��^��{fa*��mCq����*��#<tr�k>R6��C�~�m��Oz8=D~���{}r~R3���fgG�z��N@�N��^���fwI�Q?rB�;y/5`,n���Q�������o��k��F���p�U������_��(^���=�6����Q �?���e�<�l5.-:��SU3] ���`�G�5y�/3�'I��A+������}�d�����G6}N �-���R�u@�|�\-���8������,���k(reEKQ�C�&R�!i�����L� ��m����� �F�����C�$�4PX'P�
lHKN��w����]��$�f�Q��l����q�L:���)N�$���C�����4����K��|A�;���d���s�t�_�+�RY��%y����a*O�`�e��4�~�}����[��81��Fdz����`� �^�]���lZ��2�������:z�t��O��k���"���@x��hcS��9@�E��~r\�L�\8^5�xHF9P� ��Y\��=g<p���H���,���
l7J����S{"[��N��m�kR&�����T()6uO�v��\���d���[0�������x�<G���.~��'�10Ta�+sr�U!S�o�ht �^H�#�4:O�)�I�!@�~��
$����hE�{R���Z�����_!$������O,��]���E�Zz�0�C��z���A7�F�����o,��$W����:V<�Q/$J��q�p���"�Y�f��s����=�[��[�bM�
����{����'ya�o��e��%��`����g���'3U[99����^n��S��D
���b�9d��I��mW�A�n�S,ahGP��{!��B��2��
� |�q��;�G�7p��Z�`5Vo��o
��+a�Cm�`���BA]\�����l�_�\-���Z�g�
����m������x|��C�>����d^��O��YN�"�p&���������>�%[�=dl�A@�
�I�Z\6�\�r�w����� u��;���s�9]
7���p^��r����v�V�)Kq_�p��?r�u?���,�4m9��������������������T��v
B��6`�������������_��"��0n���!$V��v�CW/�Y;�5~��-�����U���+e��������FZ��+�,��:s@.�5��Y2X'�NPq!���tE"l>�����ka�~��$zU�r����=���Uc�^�4�h�J�4
Q����BxB���s_�"��*mo�������@6�$�"&�Ny\���PZv�gcF�6�R��R�f:c�%!;�/��y6/J�9�-R����6�7&!1��C��/��
�INJ��~=��.��AMAJ��58��N6��N|�=�!��#7y�J�W�%/��;%3��B*s6A��' ����cX�$����O�*�I�Fy� �&�i�eC4����C��.�����>
��������6YF9d�&yM)�H1
����*�G�j��$N��x� ,x��K�[�����S��1��8�.�����Y�" ~w�!����oON�;�����tNW@��F��d�T@%�����FPEB�Z_mv������y"�i"�<��x{{����y�����v�������BP���X8�H������4+n!X����)�"���M���s�S �.�)�����f����+�!��_u2��c�&��N��?*6���3� Z��Q�7��O�|�'S)�&9�vz�����U�{�$��;<��w;�������v�:�+(���1�X�/�X9 1��4ahL�y��fD���l���g0���sspV���E=�eP��{J�fEOE���"�$��j�1='�1'Gn�*������*Mu.|�b�S�uPm7`�v����2�����'E ��������!��S�gG�;`)�o��,���O����'e���=�FhzY0'�x�LO����r�xTm6�MJ�����3�=O������� I����3#��9�Z\ll,��i]$b�\
n0�Qb g��
�H�~0�� `u[[��W��>�����^�
��hB�c�����>���e�:�R��[�[�2N,��K����A�h�F�q�c�:0�2Jv���u�_��g��#�h'�G?�
�a~Y�BlTl��Hp?�`�"$$xD�r9uV4T��-�Y��ywf�h���a��2Y
V\�nh'�-��!Ks�@��#���Fm��g���^�X=��H{#���+�|�#����� .&�+s���zI�g����+_�lV�� �(�S�}�m/������i��� �KR`�]yg���f��{�S��e�s��aK��"I58Z��\\.��\l�U44"6���D��=*H��_"C��mr7B������M/��'��������Y�j8J
k@E��PoX����h��*��c����/m4�@F���/o<�BM���e��No��s�h:����� �����q�A���$}hE;?qI)$JM�J�Z;$/��>b�&�x�i�,���b�B*[��ed�B��}Q�Atb<������.�7l���k�N�H�4�6���s"�c����Y�n�b��}���^D ��iw���8T����+I%=$�����1�"�6����a�!���c���)�*��=L~�*E���Tu�,���G��1�6�#����GS*hF>�5���L�5���/>��lf�`9����A��+��n3��G#��\�����y��An��8x�+���
e������A�~��d+�:�d{*<��k��vP�s\N� x=�����1��s]��c����/]���K��l����kX� ���j6�,H�@�7�����1�0��-����{��T�_V�����l�K*��Xj�D�s���re�;�k���I��D^�5j�@���Y����� ��'�����T(�����Q���z����j��RU����^3y�yC�������]��O<�m��������
p�|Y�������r�C�
���}����!��z�/rIw���L�>
g����5u$��K)�Mb�2�M�`���KFJ�>6���u�-T��=������1��X�3��6WP��Q����b#�N���)��q�e�grY����]��P�{�Y��\��8����J�����L�h��s�#"��Q�}Y��b��Q��a���� ���#��?�`�����E>�����4M��^<�s#xv]u�6��|3���y"v�u���~d�-�����i�����2����<�|��R���!Rg���@���F��&��U��b%z��������",������G�5��.�a�f�&� t�
�|PX8�Epl
�.���*<��
��*h��o���'>$H! nI������m�a��(:�0�����zxs��&;��<�U q�1���6��us��������|�\����(!:��3��t]�p�L��M����UPD3_�n2��m���V��f/����^)�w�MH�F��~cx]b�]�M8�����w#����xQ^�#
}}0o-, T�����@i���a��k��=>H�p��')��^l���d��G���Al�~y9J/��1����qZA�SQ������|��8]#^n��!<xkg�.1����0���b G�Z���<1��s��B2/9P+��%
��,��l�G��-���>��V&���!�4�j}����b�0q��������;���=��1���6�?E����������9<�D�1t� ����B�}�o�q�x���CZ��)BL7��=�� o
���cog����L`�?J�2$�����(q���K���z��&���*��sf6��_=��V�
�� ;�0z�<N���
lW���.b\�A3?������|4*KP�S�E6����n8�Kc���������=,����|��o�i2�^�������V2��yU[���e�%���e�qj`�$}�u6��EeR<*�c��|���P^��� =��C�T�Z����u����p�PV�r/�/� pT��e�0lE$�
�)��H(@w�;�
f�?��
������l^���{d��J�2��o_�^u��N���1h��[ �1�pM7�H�������|� e��t��g���A��A�R�A{�qY_��OO�_�N^�gg��w���N� e0�T���7�~p}8���)�:x�������d��k{,���Y}t���k(<PYo�Hx��2�e��DV�l�;E !�^�K��).�)i�^b��G�����K�����=>��z����Q2��s`��9h �'�[L]Q�e�����&��R��:=jp�CFu�@�k�����H�8��][�ak5q���(�H�q���d�'��v3Z���&&g��/_26�/t���0e�F��l/����B����@���l������OXw��.�@P���j
�~M����$~�(�_���jf��G�
��%3��eG�0#�F+�0E!� ��#��KV�&���eU�7��-2��o�D����y�)%��\�>��&"V��/r����W�L�p�4�'9*��������kpTX9��+[���l�dl����o�%�#�����K�{��������D�����K���&)��I����5�^��r6����-���H����R���d<����eW]K�U�X]�X���
5�{2����������j����K����� �S������
�A��;W�e6��FA{����gv��`�����1��]�K�r��Q���a���:�[���@J=���)�����������s�D�;���R40�
O���� �W�nK�6���a
�`�&��g�G�V�|E.%��i:�]I��P�����[Z0��J��,�M�n��C~�%e�EhM�=��*�m[��xB���'��_�:���@��5�*���C���
�Q6�J�'i�������=gy;�����XN��6�a��tj.��59��0���� ��*5�����Zo�n���$g�����09��s�iW�����j.n_��h�=SaX�����2��d\2 \��:���4���(�6��T��wO��j��$n�d�������,!W������Y�4���&��f����21d2m#���M5dm����_{��{�`�t�pW��T�������� S��r��<�jL��{� [��gyJ����W(,�8�+�4�Y������AsOU1�/_��b�~�,��sx�m��l�|����V����=�5/������:�������7
�Qy��q[D-�����L����a��+Tw�!�L�J�U���2A����� L�lI�o�y`�U*�J#/f�+�t*K����������tT)�*V������15�A�yCz�&�l<�b���e�v�`�o�y�w8�ve|R�x����������g����
+\U6J�/�6
��'&�LP��1���E�� �
��.�j�\s����`:�+���J\&�N��tV�]������g�r�����#+�J����T��H��{ |
�����o������a�Px���}�2}Mc�9+f���0W@����7&�U|~l�N��'��`v9A�6����f�o�l��
RC������'/���Q���e��[����a{"glK?���W��7��Rp��Y���v����(O���wg��@<�o �_�<Q�=�o��$�'�F�gR��mZ�����I=$�.��qmP ��������6U0��>��1sN�V���$6�c��#���-^0P�K����=����)�X_D�,��<���fJT4�I�}���������H
-e�����F)�����0dd&�_U�&�T_�L�j>�`�\�b\z��H��,����)�Va yq�IU�(&���1�'�p�7�c�����~��x��S�$,��_�E�����`���'/�\8J���s�fd��<�$?�X�G�Xq��u6:��OZ�'9R����~������A|�Pv����;�O����nI��e�p<�Z��CNJ;��m���H{���I���B����l�s\6����H������c�\"�F��NP���nM��N�IT����s��q8���| b���,Go�Y���In�R��� ��HYJ��g���?#{�{���'��*7�s>��.�a�Q7)�lO����c�0�M�p��K��e�R����� ���pC��*
�����������V�0M=����Z����t��T��W��#��z��Me��l${\�0t�`�.'����)�����O��g�=�I����z�?�f�u6����x���a��
F���|�.�Qvm'I+��, q'����B�G��o���g���+��9r��hr���w������jF�t[ g�p�u.���;�]�R��v��������E <�����B_W�}�4��7�� y�[�`0�����8�"*f����}��a=����F�������M�IK;� �zj�h4=��TD6-T:�.#����6n���Z���n5�M������y�)�����d�]^��[��h���(���*�ed�w���q�o<]�������eR��4�
S
"VE���s�y7h��b[2�n�K�krcb����U�n�/��*$�P�����
���+�8Kf�R��}��W��e�{�eo_��X7om�<���8hz�?�#::��4����-1;)���m����b��dM�8HN��c��m5J(�A�^k���|N��rNV�����bt� ��?������V�����8B���@b�.�W����,oI�Y�}K�����v�J_4o�C�}�9KH��
$4��0],�]3�����=�`GBf�{��= ��z��mo�u���KQX�����
���or���ay�&r5V��~S�s��{��|C��p�����X�gv��)7���
���C.@�3S�1B�[���T�������uU��?��jL�}bgt�c*y���m�."W�$���o<�Is�*�0itx����� 7�j\�����%k�H���S��*�o�k��"���g�������`Jpg�oF��E��y��]�SP�����+�%�=��u�'LtL�R��y���w�������M�c�Q%%���-M��p����l������j
�D����& �tX���Y�sw��)?J�zeB}[`�%�m��F�~�E�~�-�RE�q���Fc��q� ���2�[<.�����Q��NU�E'������b��W!l�>s^�����2�����&��2.!���M�|j�'x�-���~���P�nd���o�+�w���z�Y =)g�]�9�
k%�i���-'���������n|�n�[I���j�f���,���H�2��=i$h%I� "�)&:o��mg ��H�w9���`����������%�{����?Z���r<��h�z��k2����o�g�y&�B&&�"8=�����c���E�X���R��1��@30M����Y��
Nj�T�� �/#���p]���D��j6���J�*�*�j�2�;4Y�Abx�C���#q��]��Z��/����0�!�\��
&XaB���V=Fi"�c.�bv� q�o��� |��F`p�L�}O)�/9)�,����o�/��.���������mZyH�!UH�K��.E�[���LV���4:�T8���B�����BJ{d��nbo=�wJ����������)`��7�5����K]
�� �a��3>������d�2���X���# A��2�r���\^��]
U���1��.$�r���vn*`�8W��uu�=�d��6a�c��&H�*�2���sn�1�1��r�� B]=Gw�*8u�n �P�W�����N I��V��*�;Df��DyA�dLS{���UQ�8#��$������Pj=^�j0"Z�L��X�b�+�1�F��?�k��QaUC7)��2���,���1j,L��^Eg L'��qx�-��(�R�x�|o��D��(�HF8o��%�����D��0�6�26�8���5���K�G���}���c)�
�����b���J�Uo�< 6�<��TT�C��H���&���������{@�zzbNo��g��@�����qV�=H���K����
5d"��|J�DwP*�N����ohQM/�Z)I��=��I��"��9A!.�*�����H�e��^���p�v�zo��������:��~��������yb���X�80�8�|�%NG�
�K���
�������N����5=Z t�Hn���J��������}����T�c�/r��bB&�l>3�V�U�-�C�!]������='��Wz�@��9_��3��$��:���^0��C��G�� �r�mp���J>��9�N��@�F�?_��:�����#�\1C{�l���h�������}���`{�,��x�Zv�D�e�y��,o`
��
�o�m@�\�w���*��!��M.�!�Q������_w�{.z��F�E�����{�*
W�����vT���g�9�f����\~��)l{][M���p}%�+�Ub����n�m5�����g%&�R�����23����p����r�wW��iV�Zzf��#;l\� �_�ikF�"���rv��M�&�US,F(��4I�&?1 �%�I#��7�K��L����D���h��:�)~���x1��`O�����4���X�J���X=�s\U*���� M��'�����/9K!_�F�Y��Dj0I���h�|��KEGT��a�,�t�<����N%�1����.��q�'
>�t�l�������Qjb*SD�x:�WJ�B����L���>��(q^,�}�{x��
���r/������0I���4N����E2�F��&��.e~��B���UXU�4r\�4��Yb@>O�"��1�����p��|j+�b�^���&% .@�����(u�B����������8���%�XN��
�6��t]4��mL�]GkNm�����2��ke�=�;Y����p����_YV��|2n�4�47�&g?�����5)��P�Wt�UwN#��5(���o�����1]*|�%�
5=��y+(77}|�h��&�8�H2i� v�?���w���h���!+����Q��0`��?�Z��&�L����3�.2�m��i6���.6��] ���S��������_T�7g�nh�l�'�N���H��D�����j]������F���B���Xi����s�����K<�=qP�
]���������4�t��t���>x>x>x>�2|�U���{��N���r{��2�����L�����9e��a��"j
�W����7����{��YE��Zi�U������&����������N/�vXGM�o�ygd�P��_��X�����O�e�S�M��5Z"��n}�t{�Y�]�I�B�zod�}z|���*�������$��k�2->�����mj�C#JS�o�z�W�p�^�����6�*�������'�������z�}K����[�U�������
}������Tb����Tb��D^*�/)1+��-)��T�cU uR��I��u�^�U���*.�������}F7�a�^�#~��lb�Z��zSF �:��v�<7�L*�(0&��k�����el %!�!�)j�[{g�����5�=�*@�:S��Y�����#s ��(�4a^���|n**����0��P�M�0�������`�r��^�@�H�7��������Z>���:��@�a�Y����<����&0�����(G8p�~���6c���|���8���]�iV�hF��M���������,������a�6�:��Y�A�hu&B��J�!W�'*�P����`�U�5$#��y�eb>~�5_<�ix;�^�A��F�wV���)��Oq&����{��/�y�-�6�i��_s�#cm(b�SL��\�aB&�8���l'mJ���vB#��%�XA����I���TB��(�x%*E�T����S)�?U)�4���iL
���B�wU���B/��w��s���~���#�����0�"x �r�]'d�-�~��U�����qI��@���$l5V�f���O�z�L�&��z M�|�x���+l�u������i��]1e���Bp���c��d���%Ww�����^0'S��7+:�LX�3��x]�Dz��%s�������aY���{>Q���M�Ub{���M� ^�V��V^� m�.�=�!��M���C��� L�T
��h\
B��K�z�o���%�F��3��u���O>��f�(y��bO�n<*�4vU*H�V�Y�2�_"�[���}���A�!q��_%N�`�VX�Ph1j��o��Dy��oxn,,�QU���U�j�~�������f�ZK�Y�:�M����O�k�=�u2���.�{1O ������o3}�R!{���r�$T��T�-8eMK�8Xf��z)v��W�b;Q]�K���G��� 4@m8�l���tj��%���v������{r�l����C��s��A�*���=Y��B���*�'�a�F6���EQbr.��������lM3�'�,"nJ=�/�����f��zf���U�C�(�1Se�C���q�*#����!� �k&�\3i!+hqp��q��'��@?H��4�5���*������#�]3(C�gv����#=d��4jF��������t��I�#���0y��K&�@xd��Ns�N]V�������T,�[$�7��!�V��s��a>R�����R%5i$�w��WMv��Y�_�G5ULQ���d���n�B�;>�v�;�H������.#�J]�P�B2��
�����V��8��Jxj~�lT��M,�T�` k:��Cm���J���� M��-��`x�z����<�bM�����{{���;���U3,wgq �"�o�����+�z@���-_�[���T^]J�Xo�`��@�*.DC�L()BD��c�[�q����9])��tX�W�/��X<�t��^t�[;�/�Z��iU+��b�:�*gR]�}��UQuJ�S��c�9�m���I������qGe �R�0��:���������Z:�=?u��L[�F3lf0�\=pF��%��x�����pp����.o_EK>��s|�E�i���i��������C0���?��Q�����w��E�����uW��w�����0���-���~VXTL���R����aq��lz���W3t��v�������8�o�bv�'g��(z=�.�$�Q6�������[2�����<~��a��������uN{o��\��s�Q������ ������Ddt����Ta�4NR4\�`�z)�I���.H��/�xA�]�����+����T~~��Ik��������4a A�[�E��.�Z��^LH����l�����[��#q?�&�xRO"����IE��KI����!���6�&�%��d5��?���T���XE�����4OIB�c[P��|��SP~`���T�������~�%�� �<CK�����b$��8��l���h�db�k:8=y��u��a���{v~f;�i*�Q�j�g��$CU�G�Kj:��4*��O;��q[�������������;�]�c������W|7
�A�*aWI�O��D����#����NJ�y��p�P�a��`A^D�q>�����}���9!= >H�N*�6�>a`�e������nx�"�NCSD���E�usK���)#��ON;�������^���������ug^&�4.*[[X����{G]xu����� N�����#�y����]^O�T
���qUV�;8�������f-�T�|^���C;{�;����c����/��*�~�1�%�����
���Dld�26"���;�j���������U��E#s��l�`�������.FQc�j5N��wO��@�bFo��x���[��~u��rl4��v#�&����[�;�F��q������
[+��&[>6^�^p�w������>{{v��5_^���l�������k�Q����U���u+j��h�?���W��A���28[��%�~��yz���Og����9(��*rT;0������a�W������ w�������v���P\�E���L�c�9�
��OO�s3����i'��^F;������ F����|��epN�o��qZ�
�G������pW
��W'g�f�������Yg���])?D����`�; |5`6������X0��"H����w�Tpg��+t���NP�Oo�)x�C��3Zw�-/��G[S�
��"�5Q8 ��:�z�����h�l��}�[��R�.W���tr]�u�?|��^����F���z�d����� ����3
uw$�-,�G�rw)�-��%�_V@���"E���N� ���|.G�y]��XG�^��j�NP��t�Z��z�w'$���c��1W��I���W�U4�<r���B��%���"��-��$���'�������z��� ��#&�qCe$���
��[�1-��F���|���'G�?@|.����z���Y�������Oa����~��z�C�O�q�;�����3z��x��I����7����7����y
7�]l�r���Y�(�a+��o��j�u������#�G���c���� �>���Q�o�'����E����|�B?�g�C^�����&�p~��+�g�e����������}�%M�gS��������k�������P������q�$@����>���R�����N�����
��/�<����c��+�!�v��^����J������%�O���Jh���bz��
�����<��%S��A�V�m���w�<:)V(��/�N�b���0���kti�W�th����W/�|��~��Uu��K�B�}������1�TP3�������SY���f��j�P�>W��)g\S6R8�F�X�,�z5R����Uo��
�.��e��(��� ����J��Q�w������8��/�����#����(KlE��� �o�%�U\�Gf����^�;���_����v{x�|��O������)��yo��vC4��l
�����a�<O���� �xy������sj����o�v[��"�,���Y�Sq3��?���M�H������*��)"����-���E�����M1�)^�]v������l���g��;�������;�.(S1��+�6��~�,�}�����s���N�^��/��U�|��`���|��s��9�cc��g�$,=�>��U�<��[���1w�0[=7����%*�#�0��b��T���p>�#����G6����(4
�
;
Ri,��S��o��0y����Lh�1�����b�}�o�Gm�"���Hf���P:X�=7�+`��c|���+�%��������VM!���+S��^+|�
�d��r��e�. '�=�B����A@�>Ir�M�#�����c��0��]������X��U2*C:Jl�#��H'8h���^����[��G��c�l��C�!��h���T�nQ�pS��re� 9=L��
74��Hn>��.-g% =��O[�?O�R�y>|�=��h�Tt���a��p����Y��E(!����4
���q�L���1���=����������/|H��x��3�r�|�k�7�3��L�m���B�L��Yx����Vs�?�_^z��'n��Q�b��6��H�� 7�\���g;��e����@^���2�"|9�C�
����#&Q�3|����_�i�,P��|Ppm�&[�4�N��������e�P�v$ @(�E���+ V0&;� ��_�?�{2�(���,<���������D��*R��� =.������^s�$�8yX���R�M��n?�L>�}a��]T�p���(����E����1�le4�R���N�[������&tG��S�S�x�h� �yLW������(f4��(��k}�n��D
�����j
���{��EX�f^�Hs����:�)��q�9�N�8��?�����8Z��HqS����6Os���������e�EAt��Z]�x~u}���)���8�,�H��_;/��Vx�je+�Rd�8~4�z{���r�h�b�h�����UbHV��-l��#�>����f7�T�e���y��h�>�6@H�e<iV���W�D^�����{��Y-
:���~S�������X����)~^Q���r~n
�?�<��W On Tue, 14 Dec 2010 15:51:18 +0900
Itagaki Takahiro <itagaki.takahiro@gmail.com> wrote:
On Tue, Dec 14, 2010 at 15:31, Shigeru HANADA <hanada@metrosystems.co.jp> wrote:
In addition to above, ResetCopyFrom() is necessary to support nested
loops which inner node is a ForeignScan.I think you can add ResetCopyFrom() to the core in your next file_fdw
patch because the function is not used by COPY command.
Agreed. I tried your patch with adding ResetCopyFrom() to copy.c, and
found the patch works fine for superuser at least.
I'll note other differences between the API and your FileState:
- There are no superuser checks in the exported functions because
the restriction should be only at CREATE/ALTER FOREIGN TABLE.
If the superuser grants SELECT privileges to normal users, they
should be able to read the file contents.
(But we might need to hide the file path.)
- errcontext and values/nulls arrays are not included in CopyState.
They will be additionally kept in a holder of the CopyState.
- You need to pass non-NULL filename. If it is NULL, the server
tries to read data from the client.
- The framework supports to read dumped binary files and files
with OIDs. If you don't want to support them, please check
parameters not to include those options.
All differences above wouldn't be serious problem, but I worry about
difference between file_fdw and COPY FROM.
"COPY FROM" is a command which INSERT data from a file essentially,
so it requires RowExclusiveLock on the target table. On the other
hand, file_fdw is a feature which reads data from a file through a
table, so it requires AccessShareLock on the source table.
Current export_copy patch doesn't allow non-superusers to fetch data
from files because BeginCopy() acquires RowExclusiveLock when
SELECTing from file_fdw table.
Using COPY routines from file_fdw might need another kind of
modularization, such as split file operation from COPY module and use
it from both of COPY and file_fdw. But it would require more code work,
and possibly performance tests.
Regards,
--
Shigeru Hanada
On Thu, Dec 16, 2010 at 18:45, Shigeru HANADA <hanada@metrosystems.co.jp> wrote:
"COPY FROM" is a command which INSERT data from a file essentially,
so it requires RowExclusiveLock on the target table. On the other
hand, file_fdw is a feature which reads data from a file through a
table, so it requires AccessShareLock on the source table.
Ah, I found my bug in BeginCopy(), but it's in the usage of
ExecCheckRTPerms() rather than RowExclusiveLock, right?
The target relation should have been opened and locked by the caller.
I think we can move the check to DoCopy() as like as checking for
superuser(). In my understanding, we don't have to check permissions
in each FDW because it was done in parse and analyze phases.
Could you fix it? Or, shall I do?
Using COPY routines from file_fdw might need another kind of
modularization, such as split file operation from COPY module and use
it from both of COPY and file_fdw. But it would require more code work,
and possibly performance tests.
My plan was that the 'rel' argument for BeginCopyFrom() is a "template"
for the CSV file. So, we need only AccessShareLock (or, NoLock?) for
the relation. TupleDesc might be enough for the purpose, but I've not
changed the type because of DEFAULT columns.
OTOH, CopyFrom(cstate, rel) will require an additional 'rel' argument,
that means the "target" to be inserted, though it's always same with
the above "template" in COPY FROM. RowExclusiveLock is required only
for the target relation.
--
Itagaki Takahiro
On Thu, Dec 16, 2010 at 5:35 AM, Itagaki Takahiro
<itagaki.takahiro@gmail.com> wrote:
Ah, I found my bug in BeginCopy(), but it's in the usage of
ExecCheckRTPerms() rather than RowExclusiveLock, right?
The target relation should have been opened and locked by the caller.
I think we can move the check to DoCopy() as like as checking for
superuser(). In my understanding, we don't have to check permissions
in each FDW because it was done in parse and analyze phases.
Could you fix it? Or, shall I do?
I believe that our project policy is that permissions checks must be
done at execution time, not parse/plan time.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Thu, Dec 16, 2010 at 23:09, Robert Haas <robertmhaas@gmail.com> wrote:
I believe that our project policy is that permissions checks must be
done at execution time, not parse/plan time.
Oops, yes. I should have said "permission checks for foreign tables
should have done in their own execution". So, additional checks in
each FDW are not required eventually.
In addition, we allow users to read the definition of the columns and
default values even if they don't have SELECT permission. So, I still
think permission checks for the template relation are not required in
the file reader API. But we need the checks in COPY FROM command because
the relation is used not only as a template but also as a target.
=> SELECT * FROM tbl;
ERROR: permission denied for relation tbl
=> \d+ tbl
Table "public.tbl"
Column | Type | Modifiers | Storage | Description
--------+---------+-----------+---------+-------------
i | integer | | plain |
j | integer | default 5 | plain |
Has OIDs: no
--
Itagaki Takahiro
On Thu, 16 Dec 2010 19:35:56 +0900
Itagaki Takahiro <itagaki.takahiro@gmail.com> wrote:
On Thu, Dec 16, 2010 at 18:45, Shigeru HANADA <hanada@metrosystems.co.jp> wrote:
"COPY FROM" is a command which INSERT data from a file essentially,
so it requires RowExclusiveLock on the target table. On the other
hand, file_fdw is a feature which reads data from a file through a
table, so it requires AccessShareLock on the source table.Ah, I found my bug in BeginCopy(), but it's in the usage of
ExecCheckRTPerms() rather than RowExclusiveLock, right?
The target relation should have been opened and locked by the caller.
The target foreign tables are locked in InitForeignScan() via
ExecOpenScanRelation(), so COPY routines don't need to lock it again.
I think we can move the check to DoCopy() as like as checking for
superuser(). In my understanding, we don't have to check permissions
in each FDW because it was done in parse and analyze phases.
In addition, ISTM that the check for read-only transaction should be
moved to DoCopy() because file_fdw scan can be used in read-only
transacntion.
Could you fix it? Or, shall I do?
I've just moved permission check and read-only check from BeginCopy()
to DoCopy(). Please see attached patch.
Regards,
--
Shigeru Hanada
Attachments:
On Fri, Dec 17, 2010 at 11:49, Shigeru HANADA <hanada@metrosystems.co.jp> wrote:
I've just moved permission check and read-only check from BeginCopy()
to DoCopy(). Please see attached patch.
Thanks!
Are there any objections for the change? If acceptable,
I'd like to apply it prior to SQL/MED and file_fdw patches.
We could have some better name for Begin/Next/EndCopyFrom() and
the exported CopyState. As Shigeru mentioned, COPY FROM consists of
"a file reader" and "a heap inserter", but file_fdw only uses the
file reader. In addition, we could split CopyState into two versions
for COPY FROM and COPY TO later. So, it might be better to export
them as "FileReader API" or some similar names rather than CopyFrom.
--
Itagaki Takahiro
On Fri, Dec 17, 2010 at 11:01 PM, Itagaki Takahiro
<itagaki.takahiro@gmail.com> wrote:
On Fri, Dec 17, 2010 at 11:49, Shigeru HANADA <hanada@metrosystems.co.jp> wrote:
I've just moved permission check and read-only check from BeginCopy()
to DoCopy(). Please see attached patch.Thanks!
Are there any objections for the change? If acceptable,
I'd like to apply it prior to SQL/MED and file_fdw patches.
I think at a bare minimum the functions you're adding should have a
comment explaining what they do, what their arguments mean, etc.
I'm sort of suspicious of the fact that BeginCopyTo() is a shell
around BeginCopy() while BeginCopyFrom() does a whole bunch of other
stuff. I haven't grokked what the code is doing here well enough to
have a concrete proposal though...
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Sun, Dec 19, 2010 at 12:18, Robert Haas <robertmhaas@gmail.com> wrote:
I'm sort of suspicious of the fact that BeginCopyTo() is a shell
around BeginCopy() while BeginCopyFrom() does a whole bunch of other
stuff. I haven't grokked what the code is doing here well enough to
have a concrete proposal though...
I added Begin/EndCopyTo() just because the internal code looks
symmetric. The proposal doesn't change behaviors of COPY commands
at all. It just exports a part of COPY FROM codes as "File Reader"
so that the file_fdw external module can reuse the code. I believe
we have the conclusion that we should avoid code duplication
to read files in the prior discussion.
We could arrange COPY TO codes as like as the COPY FROM APIs, but
I've not and I won't do that at this time because it is not required
by SQL/MED at all. If we do, it would be "File Writer" APIs, like:
cstate = BeginCopyTO(...);
while (tuple = ReadTupleFromSomewhere()) {
/* write the tuple into a TSV/CSV file */
NextCopyTo(cstate, tuple);
}
EndCopyTo(cstate);
--
Itagaki Takahiro
On Sat, Dec 18, 2010 at 10:43 PM, Itagaki Takahiro
<itagaki.takahiro@gmail.com> wrote:
On Sun, Dec 19, 2010 at 12:18, Robert Haas <robertmhaas@gmail.com> wrote:
I'm sort of suspicious of the fact that BeginCopyTo() is a shell
around BeginCopy() while BeginCopyFrom() does a whole bunch of other
stuff. I haven't grokked what the code is doing here well enough to
have a concrete proposal though...I added Begin/EndCopyTo() just because the internal code looks
symmetric. The proposal doesn't change behaviors of COPY commands
at all. It just exports a part of COPY FROM codes as "File Reader"
so that the file_fdw external module can reuse the code. I believe
we have the conclusion that we should avoid code duplication
to read files in the prior discussion.
I'm not questioning any of that. But I'd like the resulting code to
be as maintainable as we can make it.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Sun, Dec 19, 2010 at 12:45, Robert Haas <robertmhaas@gmail.com> wrote:
I'm not questioning any of that. But I'd like the resulting code to
be as maintainable as we can make it.
I added comments and moved some setup codes for COPY TO to BeginCopyTo()
for maintainability. CopyTo() still contains parts of initialization,
but I've not touched it yet because we don't need the arrangement for now.
--
Itagaki Takahiro
Attachments:
copy_export-20101220.diffapplication/octet-stream; name=copy_export-20101220.diffDownload
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index 7b8bee8..ff55a23 100644
*** a/src/backend/commands/copy.c
--- b/src/backend/commands/copy.c
*************** typedef struct CopyStateData
*** 93,105 ****
FILE *copy_file; /* used if copy_dest == COPY_FILE */
StringInfo fe_msgbuf; /* used for all dests during COPY TO, only for
* dest == COPY_NEW_FE in COPY FROM */
- bool fe_copy; /* true for all FE copy dests */
bool fe_eof; /* true if detected end of copy data */
EolType eol_type; /* EOL type of input */
int client_encoding; /* remote side's character encoding */
bool need_transcoding; /* client encoding diff from server? */
bool encoding_embeds_ascii; /* ASCII can be non-first byte? */
- uint64 processed; /* # of tuples processed */
/* parameters from the COPY command */
Relation rel; /* relation to copy to or from */
--- 93,103 ----
*************** typedef struct CopyStateData
*** 119,125 ****
bool *force_quote_flags; /* per-column CSV FQ flags */
bool *force_notnull_flags; /* per-column CSV FNN flags */
! /* these are just for error messages, see copy_in_error_callback */
const char *cur_relname; /* table name for error messages */
int cur_lineno; /* line number for error messages */
const char *cur_attname; /* current att for error messages */
--- 117,123 ----
bool *force_quote_flags; /* per-column CSV FQ flags */
bool *force_notnull_flags; /* per-column CSV FNN flags */
! /* these are just for error messages, see CopyFromErrorCallback */
const char *cur_relname; /* table name for error messages */
int cur_lineno; /* line number for error messages */
const char *cur_attname; /* current att for error messages */
*************** typedef struct CopyStateData
*** 142,150 ****
StringInfoData attribute_buf;
/* field raw data pointers found by COPY FROM */
!
! int max_fields;
! char ** raw_fields;
/*
* Similarly, line_buf holds the whole input line being processed. The
--- 140,147 ----
StringInfoData attribute_buf;
/* field raw data pointers found by COPY FROM */
! int max_fields;
! char **raw_fields;
/*
* Similarly, line_buf holds the whole input line being processed. The
*************** typedef struct CopyStateData
*** 167,181 ****
char *raw_buf;
int raw_buf_index; /* next byte to process */
int raw_buf_len; /* total # of bytes stored */
- } CopyStateData;
! typedef CopyStateData *CopyState;
/* DestReceiver for COPY (SELECT) TO */
typedef struct
{
DestReceiver pub; /* publicly-known function pointers */
CopyState cstate; /* CopyStateData for the command */
} DR_copy;
--- 164,191 ----
char *raw_buf;
int raw_buf_index; /* next byte to process */
int raw_buf_len; /* total # of bytes stored */
! /*
! * The definition of input functions and default expressions are stored
! * in these variables.
! */
! EState *estate;
! AttrNumber num_defaults;
! bool file_has_oids;
! FmgrInfo oid_in_function;
! Oid oid_typioparam;
! FmgrInfo *in_functions;
! Oid *typioparams;
! int *defmap;
! ExprState **defexprs; /* array of default att expressions */
! } CopyStateData;
/* DestReceiver for COPY (SELECT) TO */
typedef struct
{
DestReceiver pub; /* publicly-known function pointers */
CopyState cstate; /* CopyStateData for the command */
+ uint64 processed; /* # of tuples processed */
} DR_copy;
*************** static const char BinarySignature[11] =
*** 248,258 ****
/* non-export function prototypes */
! static void DoCopyTo(CopyState cstate);
! static void CopyTo(CopyState cstate);
static void CopyOneRowTo(CopyState cstate, Oid tupleOid,
Datum *values, bool *nulls);
! static void CopyFrom(CopyState cstate);
static bool CopyReadLine(CopyState cstate);
static bool CopyReadLineText(CopyState cstate);
static int CopyReadAttributesText(CopyState cstate);
--- 258,273 ----
/* non-export function prototypes */
! static CopyState BeginCopy(bool is_from, Relation rel, Node *raw_query,
! const char *queryString, List *attnamelist, List *options);
! static CopyState BeginCopyTo(Relation rel, Node *query, const char *queryString,
! const char *filename, List *attnamelist, List *options);
! static void EndCopyTo(CopyState cstate);
! static uint64 DoCopyTo(CopyState cstate);
! static uint64 CopyTo(CopyState cstate);
static void CopyOneRowTo(CopyState cstate, Oid tupleOid,
Datum *values, bool *nulls);
! static uint64 CopyFrom(CopyState cstate);
static bool CopyReadLine(CopyState cstate);
static bool CopyReadLineText(CopyState cstate);
static int CopyReadAttributesText(CopyState cstate);
*************** DoCopy(const CopyStmt *stmt, const char
*** 724,745 ****
CopyState cstate;
bool is_from = stmt->is_from;
bool pipe = (stmt->filename == NULL);
! List *attnamelist = stmt->attlist;
List *force_quote = NIL;
List *force_notnull = NIL;
bool force_quote_all = false;
bool format_specified = false;
- AclMode required_access = (is_from ? ACL_INSERT : ACL_SELECT);
ListCell *option;
TupleDesc tupDesc;
int num_phys_attrs;
- uint64 processed;
/* Allocate workspace and zero all fields */
cstate = (CopyStateData *) palloc0(sizeof(CopyStateData));
/* Extract options from the statement node tree */
! foreach(option, stmt->options)
{
DefElem *defel = (DefElem *) lfirst(option);
--- 739,850 ----
CopyState cstate;
bool is_from = stmt->is_from;
bool pipe = (stmt->filename == NULL);
! Relation rel;
! uint64 processed;
!
! /* Disallow file COPY except to superusers. */
! if (!pipe && !superuser())
! ereport(ERROR,
! (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
! errmsg("must be superuser to COPY to or from a file"),
! errhint("Anyone can COPY to stdout or from stdin. "
! "psql's \\copy command also works for anyone.")));
!
! if (stmt->relation)
! {
! TupleDesc tupDesc;
! AclMode required_access = (is_from ? ACL_INSERT : ACL_SELECT);
! RangeTblEntry *rte;
! List *attnums;
! ListCell *cur;
!
! Assert(!stmt->query);
!
! /* Open and lock the relation, using the appropriate lock type. */
! rel = heap_openrv(stmt->relation,
! (is_from ? RowExclusiveLock : AccessShareLock));
!
! rte = makeNode(RangeTblEntry);
! rte->rtekind = RTE_RELATION;
! rte->relid = RelationGetRelid(rel);
! rte->requiredPerms = required_access;
!
! tupDesc = RelationGetDescr(rel);
! attnums = CopyGetAttnums(tupDesc, rel, stmt->attlist);
! foreach (cur, attnums)
! {
! int attno = lfirst_int(cur) -
! FirstLowInvalidHeapAttributeNumber;
!
! if (is_from)
! rte->modifiedCols = bms_add_member(rte->modifiedCols, attno);
! else
! rte->selectedCols = bms_add_member(rte->selectedCols, attno);
! }
! ExecCheckRTPerms(list_make1(rte), true);
! }
! else
! {
! Assert(stmt->query);
!
! rel = NULL;
! }
!
! if (is_from)
! {
! /* check read-only transaction */
! if (XactReadOnly && rel->rd_backend != MyBackendId)
! PreventCommandIfReadOnly("COPY FROM");
!
! cstate = BeginCopyFrom(rel, stmt->filename,
! stmt->attlist, stmt->options);
! processed = CopyFrom(cstate); /* copy from file to database */
! EndCopyFrom(cstate);
! }
! else
! {
! cstate = BeginCopyTo(rel, stmt->query, queryString, stmt->filename,
! stmt->attlist, stmt->options);
! processed = DoCopyTo(cstate); /* copy from database to file */
! EndCopyTo(cstate);
! }
!
! /*
! * Close the relation. If reading, we can release the AccessShareLock we got;
! * if writing, we should hold the lock until end of transaction to ensure that
! * updates will be committed before lock is released.
! */
! if (rel != NULL)
! heap_close(rel, (is_from ? NoLock : AccessShareLock));
!
! return processed;
! }
!
! /*
! * Common setup routines used by BeginCopyFrom and BeginCopyTo.
! */
! static CopyState
! BeginCopy(bool is_from,
! Relation rel,
! Node *raw_query,
! const char *queryString,
! List *attnamelist,
! List *options)
! {
! CopyState cstate;
List *force_quote = NIL;
List *force_notnull = NIL;
bool force_quote_all = false;
bool format_specified = false;
ListCell *option;
TupleDesc tupDesc;
int num_phys_attrs;
/* Allocate workspace and zero all fields */
cstate = (CopyStateData *) palloc0(sizeof(CopyStateData));
/* Extract options from the statement node tree */
! foreach(option, options)
{
DefElem *defel = (DefElem *) lfirst(option);
*************** DoCopy(const CopyStmt *stmt, const char
*** 980,1030 ****
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("CSV quote character must not appear in the NULL specification")));
! /* Disallow file COPY except to superusers. */
! if (!pipe && !superuser())
! ereport(ERROR,
! (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
! errmsg("must be superuser to COPY to or from a file"),
! errhint("Anyone can COPY to stdout or from stdin. "
! "psql's \\copy command also works for anyone.")));
!
! if (stmt->relation)
{
! RangeTblEntry *rte;
! List *attnums;
! ListCell *cur;
!
! Assert(!stmt->query);
! cstate->queryDesc = NULL;
! /* Open and lock the relation, using the appropriate lock type. */
! cstate->rel = heap_openrv(stmt->relation,
! (is_from ? RowExclusiveLock : AccessShareLock));
tupDesc = RelationGetDescr(cstate->rel);
- /* Check relation permissions. */
- rte = makeNode(RangeTblEntry);
- rte->rtekind = RTE_RELATION;
- rte->relid = RelationGetRelid(cstate->rel);
- rte->requiredPerms = required_access;
-
- attnums = CopyGetAttnums(tupDesc, cstate->rel, attnamelist);
- foreach (cur, attnums)
- {
- int attno = lfirst_int(cur) - FirstLowInvalidHeapAttributeNumber;
-
- if (is_from)
- rte->modifiedCols = bms_add_member(rte->modifiedCols, attno);
- else
- rte->selectedCols = bms_add_member(rte->selectedCols, attno);
- }
- ExecCheckRTPerms(list_make1(rte), true);
-
- /* check read-only transaction */
- if (XactReadOnly && is_from && cstate->rel->rd_backend != MyBackendId)
- PreventCommandIfReadOnly("COPY FROM");
-
/* Don't allow COPY w/ OIDs to or from a table without them */
if (cstate->oids && !cstate->rel->rd_rel->relhasoids)
ereport(ERROR,
--- 1085,1098 ----
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("CSV quote character must not appear in the NULL specification")));
! if (rel)
{
! Assert(!raw_query);
! cstate->rel = rel;
tupDesc = RelationGetDescr(cstate->rel);
/* Don't allow COPY w/ OIDs to or from a table without them */
if (cstate->oids && !cstate->rel->rd_rel->relhasoids)
ereport(ERROR,
*************** DoCopy(const CopyStmt *stmt, const char
*** 1058,1064 ****
* function and is executed repeatedly. (See also the same hack in
* DECLARE CURSOR and PREPARE.) XXX FIXME someday.
*/
! rewritten = pg_analyze_and_rewrite((Node *) copyObject(stmt->query),
queryString, NULL, 0);
/* We don't expect more or less than one result query */
--- 1126,1132 ----
* function and is executed repeatedly. (See also the same hack in
* DECLARE CURSOR and PREPARE.) XXX FIXME someday.
*/
! rewritten = pg_analyze_and_rewrite((Node *) copyObject(raw_query),
queryString, NULL, 0);
/* We don't expect more or less than one result query */
*************** DoCopy(const CopyStmt *stmt, const char
*** 1160,1173 ****
}
}
- /* Set up variables to avoid per-attribute overhead. */
- initStringInfo(&cstate->attribute_buf);
- initStringInfo(&cstate->line_buf);
- cstate->line_buf_converted = false;
- cstate->raw_buf = (char *) palloc(RAW_BUF_SIZE + 1);
- cstate->raw_buf_index = cstate->raw_buf_len = 0;
- cstate->processed = 0;
-
/*
* Set up encoding conversion info. Even if the client and server
* encodings are the same, we must apply pg_client_to_server() to validate
--- 1228,1233 ----
*************** DoCopy(const CopyStmt *stmt, const char
*** 1181,1229 ****
cstate->encoding_embeds_ascii = PG_ENCODING_IS_CLIENT_ONLY(cstate->client_encoding);
cstate->copy_dest = COPY_FILE; /* default */
- cstate->filename = stmt->filename;
-
- if (is_from)
- CopyFrom(cstate); /* copy from file to database */
- else
- DoCopyTo(cstate); /* copy from database to file */
! /*
! * Close the relation or query. If reading, we can release the
! * AccessShareLock we got; if writing, we should hold the lock until end
! * of transaction to ensure that updates will be committed before lock is
! * released.
! */
! if (cstate->rel)
! heap_close(cstate->rel, (is_from ? NoLock : AccessShareLock));
! else
! {
! /* Close down the query and free resources. */
! ExecutorEnd(cstate->queryDesc);
! FreeQueryDesc(cstate->queryDesc);
! PopActiveSnapshot();
! }
!
! /* Clean up storage (probably not really necessary) */
! processed = cstate->processed;
!
! pfree(cstate->attribute_buf.data);
! pfree(cstate->line_buf.data);
! pfree(cstate->raw_buf);
! pfree(cstate);
!
! return processed;
}
/*
! * This intermediate routine exists mainly to localize the effects of setjmp
! * so we don't need to plaster a lot of variables with "volatile".
*/
! static void
! DoCopyTo(CopyState cstate)
{
! bool pipe = (cstate->filename == NULL);
if (cstate->rel)
{
--- 1241,1266 ----
cstate->encoding_embeds_ascii = PG_ENCODING_IS_CLIENT_ONLY(cstate->client_encoding);
cstate->copy_dest = COPY_FILE; /* default */
! return cstate;
}
/*
! * Setup CopyState to read tuples from a table or a query for COPY TO.
*/
! static CopyState
! BeginCopyTo(Relation rel,
! Node *query,
! const char *queryString,
! const char *filename,
! List *attnamelist,
! List *options)
{
! CopyState cstate;
! bool pipe = (filename == NULL);
!
! cstate = BeginCopy(false, rel, query, queryString, attnamelist, options);
if (cstate->rel)
{
*************** DoCopyTo(CopyState cstate)
*** 1250,1258 ****
if (pipe)
{
! if (whereToSendOutput == DestRemote)
! cstate->fe_copy = true;
! else
cstate->copy_file = stdout;
}
else
--- 1287,1293 ----
if (pipe)
{
! if (whereToSendOutput != DestRemote)
cstate->copy_file = stdout;
}
else
*************** DoCopyTo(CopyState cstate)
*** 1264,1274 ****
* Prevent write to relative path ... too easy to shoot oneself in the
* foot by overwriting a database file ...
*/
! if (!is_absolute_path(cstate->filename))
ereport(ERROR,
(errcode(ERRCODE_INVALID_NAME),
errmsg("relative path not allowed for COPY to file")));
oumask = umask(S_IWGRP | S_IWOTH);
cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_W);
umask(oumask);
--- 1299,1310 ----
* Prevent write to relative path ... too easy to shoot oneself in the
* foot by overwriting a database file ...
*/
! if (!is_absolute_path(filename))
ereport(ERROR,
(errcode(ERRCODE_INVALID_NAME),
errmsg("relative path not allowed for COPY to file")));
+ cstate->filename = pstrdup(filename);
oumask = umask(S_IWGRP | S_IWOTH);
cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_W);
umask(oumask);
*************** DoCopyTo(CopyState cstate)
*** 1286,1299 ****
errmsg("\"%s\" is a directory", cstate->filename)));
}
PG_TRY();
{
! if (cstate->fe_copy)
SendCopyBegin(cstate);
! CopyTo(cstate);
! if (cstate->fe_copy)
SendCopyEnd(cstate);
}
PG_CATCH();
--- 1322,1349 ----
errmsg("\"%s\" is a directory", cstate->filename)));
}
+ return cstate;
+ }
+
+ /*
+ * This intermediate routine exists mainly to localize the effects of setjmp
+ * so we don't need to plaster a lot of variables with "volatile".
+ */
+ static uint64
+ DoCopyTo(CopyState cstate)
+ {
+ bool pipe = (cstate->filename == NULL);
+ bool fe_copy = (pipe && whereToSendOutput == DestRemote);
+ uint64 processed;
+
PG_TRY();
{
! if (fe_copy)
SendCopyBegin(cstate);
! processed = CopyTo(cstate);
! if (fe_copy)
SendCopyEnd(cstate);
}
PG_CATCH();
*************** DoCopyTo(CopyState cstate)
*** 1308,1333 ****
}
PG_END_TRY();
! if (!pipe)
{
if (FreeFile(cstate->copy_file))
ereport(ERROR,
(errcode_for_file_access(),
! errmsg("could not write to file \"%s\": %m",
cstate->filename)));
}
}
/*
* Copy from relation or query TO file.
*/
! static void
CopyTo(CopyState cstate)
{
TupleDesc tupDesc;
int num_phys_attrs;
Form_pg_attribute *attr;
ListCell *cur;
if (cstate->rel)
tupDesc = RelationGetDescr(cstate->rel);
--- 1358,1408 ----
}
PG_END_TRY();
! return processed;
! }
!
! /*
! * Clean up storage and release resources for COPY TO.
! */
! static void
! EndCopyTo(CopyState cstate)
! {
! if (cstate->filename)
{
if (FreeFile(cstate->copy_file))
ereport(ERROR,
(errcode_for_file_access(),
! errmsg("could not close file \"%s\": %m",
cstate->filename)));
+
+ pfree(cstate->filename);
}
+
+ /*
+ * Close the relation or query. We can release the AccessShareLock we got.
+ */
+ if (cstate->rel == NULL)
+ {
+ /* Close down the query and free resources. */
+ ExecutorEnd(cstate->queryDesc);
+ FreeQueryDesc(cstate->queryDesc);
+ PopActiveSnapshot();
+ }
+
+ pfree(cstate);
}
/*
* Copy from relation or query TO file.
*/
! static uint64
CopyTo(CopyState cstate)
{
TupleDesc tupDesc;
int num_phys_attrs;
Form_pg_attribute *attr;
ListCell *cur;
+ uint64 processed;
if (cstate->rel)
tupDesc = RelationGetDescr(cstate->rel);
*************** CopyTo(CopyState cstate)
*** 1433,1438 ****
--- 1508,1514 ----
scandesc = heap_beginscan(cstate->rel, GetActiveSnapshot(), 0, NULL);
+ processed = 0;
while ((tuple = heap_getnext(scandesc, ForwardScanDirection)) != NULL)
{
CHECK_FOR_INTERRUPTS();
*************** CopyTo(CopyState cstate)
*** 1442,1447 ****
--- 1518,1524 ----
/* Format and send the data */
CopyOneRowTo(cstate, HeapTupleGetOid(tuple), values, nulls);
+ processed++;
}
heap_endscan(scandesc);
*************** CopyTo(CopyState cstate)
*** 1450,1455 ****
--- 1527,1533 ----
{
/* run the plan --- the dest receiver will send tuples */
ExecutorRun(cstate->queryDesc, ForwardScanDirection, 0L);
+ processed = ((DR_copy *) cstate->queryDesc->dest)->processed;
}
if (cstate->binary)
*************** CopyTo(CopyState cstate)
*** 1461,1466 ****
--- 1539,1546 ----
}
MemoryContextDelete(cstate->rowcontext);
+
+ return processed;
}
/*
*************** CopyOneRowTo(CopyState cstate, Oid tuple
*** 1552,1567 ****
CopySendEndOfRow(cstate);
MemoryContextSwitchTo(oldcontext);
-
- cstate->processed++;
}
/*
* error context callback for COPY FROM
*/
! static void
! copy_in_error_callback(void *arg)
{
CopyState cstate = (CopyState) arg;
--- 1632,1645 ----
CopySendEndOfRow(cstate);
MemoryContextSwitchTo(oldcontext);
}
/*
* error context callback for COPY FROM
*/
! void
! CopyFromErrorCallback(void *arg)
{
CopyState cstate = (CopyState) arg;
*************** limit_printout_length(const char *str)
*** 1663,1703 ****
/*
* Copy FROM file to relation.
*/
! static void
CopyFrom(CopyState cstate)
{
- bool pipe = (cstate->filename == NULL);
HeapTuple tuple;
TupleDesc tupDesc;
- Form_pg_attribute *attr;
- AttrNumber num_phys_attrs,
- attr_count,
- num_defaults;
- FmgrInfo *in_functions;
- FmgrInfo oid_in_function;
- Oid *typioparams;
- Oid oid_typioparam;
- int attnum;
- int i;
- Oid in_func_oid;
Datum *values;
bool *nulls;
- int nfields;
- char **field_strings;
bool done = false;
- bool isnull;
ResultRelInfo *resultRelInfo;
! EState *estate = CreateExecutorState(); /* for ExecConstraints() */
TupleTableSlot *slot;
- bool file_has_oids;
- int *defmap;
- ExprState **defexprs; /* array of default att expressions */
- ExprContext *econtext; /* used for ExecEvalExpr for default atts */
MemoryContext oldcontext = CurrentMemoryContext;
ErrorContextCallback errcontext;
CommandId mycid = GetCurrentCommandId(true);
int hi_options = 0; /* start with default heap_insert options */
BulkInsertState bistate;
Assert(cstate->rel);
--- 1741,1763 ----
/*
* Copy FROM file to relation.
*/
! static uint64
CopyFrom(CopyState cstate)
{
HeapTuple tuple;
TupleDesc tupDesc;
Datum *values;
bool *nulls;
bool done = false;
ResultRelInfo *resultRelInfo;
! EState *estate; /* for ExecConstraints() */
TupleTableSlot *slot;
MemoryContext oldcontext = CurrentMemoryContext;
ErrorContextCallback errcontext;
CommandId mycid = GetCurrentCommandId(true);
int hi_options = 0; /* start with default heap_insert options */
BulkInsertState bistate;
+ uint64 processed = 0;
Assert(cstate->rel);
*************** CopyFrom(CopyState cstate)
*** 1720,1725 ****
--- 1780,1788 ----
RelationGetRelationName(cstate->rel))));
}
+ estate = GetCopyExecutorState(cstate);
+ tupDesc = RelationGetDescr(cstate->rel);
+
/*----------
* Check to see if we can avoid writing WAL
*
*************** CopyFrom(CopyState cstate)
*** 1755,1792 ****
hi_options |= HEAP_INSERT_SKIP_WAL;
}
- if (pipe)
- {
- if (whereToSendOutput == DestRemote)
- ReceiveCopyBegin(cstate);
- else
- cstate->copy_file = stdin;
- }
- else
- {
- struct stat st;
-
- cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_R);
-
- if (cstate->copy_file == NULL)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not open file \"%s\" for reading: %m",
- cstate->filename)));
-
- fstat(fileno(cstate->copy_file), &st);
- if (S_ISDIR(st.st_mode))
- ereport(ERROR,
- (errcode(ERRCODE_WRONG_OBJECT_TYPE),
- errmsg("\"%s\" is a directory", cstate->filename)));
- }
-
- tupDesc = RelationGetDescr(cstate->rel);
- attr = tupDesc->attrs;
- num_phys_attrs = tupDesc->natts;
- attr_count = list_length(cstate->attnumlist);
- num_defaults = 0;
-
/*
* We need a ResultRelInfo so we can use the regular executor's
* index-entry-making machinery. (There used to be a huge amount of code
--- 1818,1823 ----
*************** CopyFrom(CopyState cstate)
*** 1815,1821 ****
slot = ExecInitExtraTupleSlot(estate);
ExecSetSlotDescriptor(slot, tupDesc);
! econtext = GetPerTupleExprContext(estate);
/*
* Pick up the required catalog information for each attribute in the
--- 1846,2027 ----
slot = ExecInitExtraTupleSlot(estate);
ExecSetSlotDescriptor(slot, tupDesc);
! /* Prepare to catch AFTER triggers. */
! AfterTriggerBeginQuery();
!
! /*
! * Check BEFORE STATEMENT insertion triggers. It's debateable whether we
! * should do this for COPY, since it's not really an "INSERT" statement as
! * such. However, executing these triggers maintains consistency with the
! * EACH ROW triggers that we already fire on COPY.
! */
! ExecBSInsertTriggers(estate, resultRelInfo);
!
! values = (Datum *) palloc(tupDesc->natts * sizeof(Datum));
! nulls = (bool *) palloc(tupDesc->natts * sizeof(bool));
!
! bistate = GetBulkInsertState();
!
! /* Set up callback to identify error line number */
! errcontext.callback = CopyFromErrorCallback;
! errcontext.arg = (void *) cstate;
! errcontext.previous = error_context_stack;
! error_context_stack = &errcontext;
!
! while (!done)
! {
! bool skip_tuple;
! Oid loaded_oid = InvalidOid;
!
! CHECK_FOR_INTERRUPTS();
!
! /* Reset the per-tuple exprcontext */
! ResetPerTupleExprContext(estate);
!
! /* Switch into its memory context */
! MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
!
! done = !NextCopyFrom(cstate, values, nulls, &loaded_oid);
! if (done)
! break;
!
! /* And now we can form the input tuple. */
! tuple = heap_form_tuple(tupDesc, values, nulls);
!
! if (loaded_oid != InvalidOid)
! HeapTupleSetOid(tuple, loaded_oid);
!
! /* Triggers and stuff need to be invoked in query context. */
! MemoryContextSwitchTo(oldcontext);
!
! skip_tuple = false;
!
! /* BEFORE ROW INSERT Triggers */
! if (resultRelInfo->ri_TrigDesc &&
! resultRelInfo->ri_TrigDesc->trig_insert_before_row)
! {
! HeapTuple newtuple;
!
! newtuple = ExecBRInsertTriggers(estate, resultRelInfo, tuple);
!
! if (newtuple == NULL) /* "do nothing" */
! skip_tuple = true;
! else if (newtuple != tuple) /* modified by Trigger(s) */
! {
! heap_freetuple(tuple);
! tuple = newtuple;
! }
! }
!
! if (!skip_tuple)
! {
! List *recheckIndexes = NIL;
!
! /* Place tuple in tuple slot */
! ExecStoreTuple(tuple, slot, InvalidBuffer, false);
!
! /* Check the constraints of the tuple */
! if (cstate->rel->rd_att->constr)
! ExecConstraints(resultRelInfo, slot, estate);
!
! /* OK, store the tuple and create index entries for it */
! heap_insert(cstate->rel, tuple, mycid, hi_options, bistate);
!
! if (resultRelInfo->ri_NumIndices > 0)
! recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self),
! estate);
!
! /* AFTER ROW INSERT Triggers */
! ExecARInsertTriggers(estate, resultRelInfo, tuple,
! recheckIndexes);
!
! list_free(recheckIndexes);
!
! /*
! * We count only tuples not suppressed by a BEFORE INSERT trigger;
! * this is the same definition used by execMain.c for counting
! * tuples inserted by an INSERT command.
! */
! processed++;
! }
! }
!
! /* Done, clean up */
! error_context_stack = errcontext.previous;
!
! FreeBulkInsertState(bistate);
!
! MemoryContextSwitchTo(oldcontext);
!
! /* Execute AFTER STATEMENT insertion triggers */
! ExecASInsertTriggers(estate, resultRelInfo);
!
! /* Handle queued AFTER triggers */
! AfterTriggerEndQuery(estate);
!
! pfree(values);
! pfree(nulls);
!
! ExecResetTupleTable(estate->es_tupleTable, false);
!
! ExecCloseIndices(resultRelInfo);
!
! /*
! * If we skipped writing WAL, then we need to sync the heap (but not
! * indexes since those use WAL anyway)
! */
! if (hi_options & HEAP_INSERT_SKIP_WAL)
! heap_sync(cstate->rel);
!
! return processed;
! }
!
! /*
! * Setup CopyState to read tuples from a file for COPY FROM.
! */
! CopyState
! BeginCopyFrom(Relation rel,
! const char *filename,
! List *attnamelist,
! List *options)
! {
! CopyState cstate;
! bool pipe = (filename == NULL);
! TupleDesc tupDesc;
! Form_pg_attribute *attr;
! AttrNumber num_phys_attrs,
! attr_count,
! num_defaults;
! FmgrInfo *in_functions;
! Oid *typioparams;
! int attnum;
! Oid in_func_oid;
! EState *estate = CreateExecutorState(); /* for ExecPrepareExpr() */
! int *defmap;
! ExprState **defexprs;
!
! cstate = BeginCopy(true, rel, NULL, NULL, attnamelist, options);
!
! /* Initialize state variables */
! cstate->fe_eof = false;
! cstate->eol_type = EOL_UNKNOWN;
! cstate->cur_relname = RelationGetRelationName(cstate->rel);
! cstate->cur_lineno = 0;
! cstate->cur_attname = NULL;
! cstate->cur_attval = NULL;
!
! /* Set up variables to avoid per-attribute overhead. */
! initStringInfo(&cstate->attribute_buf);
! initStringInfo(&cstate->line_buf);
! cstate->line_buf_converted = false;
! cstate->raw_buf = (char *) palloc(RAW_BUF_SIZE + 1);
! cstate->raw_buf_index = cstate->raw_buf_len = 0;
!
! tupDesc = RelationGetDescr(cstate->rel);
! attr = tupDesc->attrs;
! num_phys_attrs = tupDesc->natts;
! attr_count = list_length(cstate->attnumlist);
! num_defaults = 0;
/*
* Pick up the required catalog information for each attribute in the
*************** CopyFrom(CopyState cstate)
*** 1860,1878 ****
}
}
! /* Prepare to catch AFTER triggers. */
! AfterTriggerBeginQuery();
! /*
! * Check BEFORE STATEMENT insertion triggers. It's debateable whether we
! * should do this for COPY, since it's not really an "INSERT" statement as
! * such. However, executing these triggers maintains consistency with the
! * EACH ROW triggers that we already fire on COPY.
! */
! ExecBSInsertTriggers(estate, resultRelInfo);
if (!cstate->binary)
! file_has_oids = cstate->oids; /* must rely on user to tell us... */
else
{
/* Read and verify binary header */
--- 2066,2111 ----
}
}
! /* We keep those variables in cstate. */
! cstate->estate = estate;
! cstate->in_functions = in_functions;
! cstate->typioparams = typioparams;
! cstate->defmap = defmap;
! cstate->defexprs = defexprs;
! cstate->num_defaults = num_defaults;
! if (pipe)
! {
! if (whereToSendOutput == DestRemote)
! ReceiveCopyBegin(cstate);
! else
! cstate->copy_file = stdin;
! }
! else
! {
! struct stat st;
!
! cstate->filename = pstrdup(filename);
! cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_R);
!
! if (cstate->copy_file == NULL)
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not open file \"%s\" for reading: %m",
! cstate->filename)));
!
! fstat(fileno(cstate->copy_file), &st);
! if (S_ISDIR(st.st_mode))
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("\"%s\" is a directory", cstate->filename)));
! }
if (!cstate->binary)
! {
! /* must rely on user to tell us... */
! cstate->file_has_oids = cstate->oids;
! }
else
{
/* Read and verify binary header */
*************** CopyFrom(CopyState cstate)
*** 1890,1896 ****
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid COPY file header (missing flags)")));
! file_has_oids = (tmp & (1 << 16)) != 0;
tmp &= ~(1 << 16);
if ((tmp >> 16) != 0)
ereport(ERROR,
--- 2123,2129 ----
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid COPY file header (missing flags)")));
! cstate->file_has_oids = (tmp & (1 << 16)) != 0;
tmp &= ~(1 << 16);
if ((tmp >> 16) != 0)
ereport(ERROR,
*************** CopyFrom(CopyState cstate)
*** 1912,1973 ****
}
}
! if (file_has_oids && cstate->binary)
{
getTypeBinaryInputInfo(OIDOID,
! &in_func_oid, &oid_typioparam);
! fmgr_info(in_func_oid, &oid_in_function);
}
- values = (Datum *) palloc(num_phys_attrs * sizeof(Datum));
- nulls = (bool *) palloc(num_phys_attrs * sizeof(bool));
-
/* create workspace for CopyReadAttributes results */
! nfields = file_has_oids ? (attr_count + 1) : attr_count;
! if (! cstate->binary)
{
cstate->max_fields = nfields;
cstate->raw_fields = (char **) palloc(nfields * sizeof(char *));
}
! /* Initialize state variables */
! cstate->fe_eof = false;
! cstate->eol_type = EOL_UNKNOWN;
! cstate->cur_relname = RelationGetRelationName(cstate->rel);
! cstate->cur_lineno = 0;
! cstate->cur_attname = NULL;
! cstate->cur_attval = NULL;
!
! bistate = GetBulkInsertState();
! /* Set up callback to identify error line number */
! errcontext.callback = copy_in_error_callback;
! errcontext.arg = (void *) cstate;
! errcontext.previous = error_context_stack;
! error_context_stack = &errcontext;
/* on input just throw the header line away */
! if (cstate->header_line)
{
cstate->cur_lineno++;
! done = CopyReadLine(cstate);
}
! while (!done)
! {
! bool skip_tuple;
! Oid loaded_oid = InvalidOid;
!
! CHECK_FOR_INTERRUPTS();
cstate->cur_lineno++;
- /* Reset the per-tuple exprcontext */
- ResetPerTupleExprContext(estate);
-
- /* Switch into its memory context */
- MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
-
/* Initialize all values for row to NULL */
MemSet(values, 0, num_phys_attrs * sizeof(Datum));
MemSet(nulls, true, num_phys_attrs * sizeof(bool));
--- 2145,2212 ----
}
}
! if (cstate->file_has_oids && cstate->binary)
{
getTypeBinaryInputInfo(OIDOID,
! &in_func_oid, &cstate->oid_typioparam);
! fmgr_info(in_func_oid, &cstate->oid_in_function);
}
/* create workspace for CopyReadAttributes results */
! if (!cstate->binary)
{
+ int nfields = cstate->file_has_oids ? (attr_count + 1) : attr_count;
+
cstate->max_fields = nfields;
cstate->raw_fields = (char **) palloc(nfields * sizeof(char *));
}
! return cstate;
! }
! /*
! * Read next tuple from file for COPY FROM. Return false if no more tuples.
! *
! * valus and nulls arrays must be the same length as columns of the
! * relation passed to BeginCopyFrom. Oid of the tuple is returned with
! * tupleOid separately.
! */
! bool
! NextCopyFrom(CopyState cstate, Datum *values, bool *nulls, Oid *tupleOid)
! {
! TupleDesc tupDesc;
! Form_pg_attribute *attr;
! AttrNumber num_phys_attrs,
! attr_count,
! num_defaults = cstate->num_defaults;
! FmgrInfo *in_functions = cstate->in_functions;
! Oid *typioparams = cstate->typioparams;
! int i;
! int nfields;
! char **field_strings;
! bool isnull;
! bool file_has_oids = cstate->file_has_oids;
! int *defmap = cstate->defmap;
! ExprState **defexprs = cstate->defexprs;
! ExprContext *econtext; /* used for ExecEvalExpr for default atts */
/* on input just throw the header line away */
! if (cstate->cur_lineno == 0 && cstate->header_line)
{
cstate->cur_lineno++;
! if (CopyReadLine(cstate))
! return false; /* done */
}
! tupDesc = RelationGetDescr(cstate->rel);
! attr = tupDesc->attrs;
! num_phys_attrs = tupDesc->natts;
! attr_count = list_length(cstate->attnumlist);
! nfields = file_has_oids ? (attr_count + 1) : attr_count;
+ /* XXX: Indentation is not adjusted to keep the patch small. */
cstate->cur_lineno++;
/* Initialize all values for row to NULL */
MemSet(values, 0, num_phys_attrs * sizeof(Datum));
MemSet(nulls, true, num_phys_attrs * sizeof(bool));
*************** CopyFrom(CopyState cstate)
*** 1978,1983 ****
--- 2217,2223 ----
int fldct;
int fieldno;
char *string;
+ bool done;
/* Actually read the line into memory here */
done = CopyReadLine(cstate);
*************** CopyFrom(CopyState cstate)
*** 1988,1994 ****
* EOF, ie, process the line and then exit loop on next iteration.
*/
if (done && cstate->line_buf.len == 0)
! break;
/* Parse the line into de-escaped field values */
if (cstate->csv_mode)
--- 2228,2234 ----
* EOF, ie, process the line and then exit loop on next iteration.
*/
if (done && cstate->line_buf.len == 0)
! return false;
/* Parse the line into de-escaped field values */
if (cstate->csv_mode)
*************** CopyFrom(CopyState cstate)
*** 2018,2030 ****
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("null OID in COPY data")));
! else
{
cstate->cur_attname = "oid";
cstate->cur_attval = string;
! loaded_oid = DatumGetObjectId(DirectFunctionCall1(oidin,
! CStringGetDatum(string)));
! if (loaded_oid == InvalidOid)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid OID in COPY data")));
--- 2258,2270 ----
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("null OID in COPY data")));
! else if (cstate->oids && tupleOid != NULL)
{
cstate->cur_attname = "oid";
cstate->cur_attval = string;
! *tupleOid = DatumGetObjectId(DirectFunctionCall1(oidin,
! CStringGetDatum(string)));
! if (*tupleOid == InvalidOid)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid OID in COPY data")));
*************** CopyFrom(CopyState cstate)
*** 2076,2083 ****
if (!CopyGetInt16(cstate, &fld_count))
{
/* EOF detected (end of file, or protocol-level EOF) */
! done = true;
! break;
}
if (fld_count == -1)
--- 2316,2322 ----
if (!CopyGetInt16(cstate, &fld_count))
{
/* EOF detected (end of file, or protocol-level EOF) */
! return false;
}
if (fld_count == -1)
*************** CopyFrom(CopyState cstate)
*** 2101,2108 ****
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("received copy data after EOF marker")));
! done = true;
! break;
}
if (fld_count != attr_count)
--- 2340,2346 ----
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("received copy data after EOF marker")));
! return false;
}
if (fld_count != attr_count)
*************** CopyFrom(CopyState cstate)
*** 2113,2124 ****
if (file_has_oids)
{
cstate->cur_attname = "oid";
loaded_oid =
DatumGetObjectId(CopyReadBinaryAttribute(cstate,
0,
! &oid_in_function,
! oid_typioparam,
-1,
&isnull));
if (isnull || loaded_oid == InvalidOid)
--- 2351,2364 ----
if (file_has_oids)
{
+ Oid loaded_oid;
+
cstate->cur_attname = "oid";
loaded_oid =
DatumGetObjectId(CopyReadBinaryAttribute(cstate,
0,
! &cstate->oid_in_function,
! cstate->oid_typioparam,
-1,
&isnull));
if (isnull || loaded_oid == InvalidOid)
*************** CopyFrom(CopyState cstate)
*** 2126,2131 ****
--- 2366,2373 ----
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid OID in COPY data")));
cstate->cur_attname = NULL;
+ if (cstate->oids && tupleOid != NULL)
+ *tupleOid = loaded_oid;
}
i = 0;
*************** CopyFrom(CopyState cstate)
*** 2151,2267 ****
* provided by the input data. Anything not processed here or above
* will remain NULL.
*/
for (i = 0; i < num_defaults; i++)
{
values[defmap[i]] = ExecEvalExpr(defexprs[i], econtext,
&nulls[defmap[i]], NULL);
}
! /* And now we can form the input tuple. */
! tuple = heap_form_tuple(tupDesc, values, nulls);
!
! if (cstate->oids && file_has_oids)
! HeapTupleSetOid(tuple, loaded_oid);
!
! /* Triggers and stuff need to be invoked in query context. */
! MemoryContextSwitchTo(oldcontext);
!
! skip_tuple = false;
!
! /* BEFORE ROW INSERT Triggers */
! if (resultRelInfo->ri_TrigDesc &&
! resultRelInfo->ri_TrigDesc->trig_insert_before_row)
! {
! HeapTuple newtuple;
!
! newtuple = ExecBRInsertTriggers(estate, resultRelInfo, tuple);
!
! if (newtuple == NULL) /* "do nothing" */
! skip_tuple = true;
! else if (newtuple != tuple) /* modified by Trigger(s) */
! {
! heap_freetuple(tuple);
! tuple = newtuple;
! }
! }
!
! if (!skip_tuple)
! {
! List *recheckIndexes = NIL;
!
! /* Place tuple in tuple slot */
! ExecStoreTuple(tuple, slot, InvalidBuffer, false);
!
! /* Check the constraints of the tuple */
! if (cstate->rel->rd_att->constr)
! ExecConstraints(resultRelInfo, slot, estate);
!
! /* OK, store the tuple and create index entries for it */
! heap_insert(cstate->rel, tuple, mycid, hi_options, bistate);
!
! if (resultRelInfo->ri_NumIndices > 0)
! recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self),
! estate);
!
! /* AFTER ROW INSERT Triggers */
! ExecARInsertTriggers(estate, resultRelInfo, tuple,
! recheckIndexes);
!
! list_free(recheckIndexes);
!
! /*
! * We count only tuples not suppressed by a BEFORE INSERT trigger;
! * this is the same definition used by execMain.c for counting
! * tuples inserted by an INSERT command.
! */
! cstate->processed++;
! }
! }
!
! /* Done, clean up */
! error_context_stack = errcontext.previous;
!
! FreeBulkInsertState(bistate);
!
! MemoryContextSwitchTo(oldcontext);
!
! /* Execute AFTER STATEMENT insertion triggers */
! ExecASInsertTriggers(estate, resultRelInfo);
!
! /* Handle queued AFTER triggers */
! AfterTriggerEndQuery(estate);
!
! pfree(values);
! pfree(nulls);
! if (! cstate->binary)
! pfree(cstate->raw_fields);
!
! pfree(in_functions);
! pfree(typioparams);
! pfree(defmap);
! pfree(defexprs);
!
! ExecResetTupleTable(estate->es_tupleTable, false);
!
! ExecCloseIndices(resultRelInfo);
! FreeExecutorState(estate);
! if (!pipe)
{
if (FreeFile(cstate->copy_file))
ereport(ERROR,
(errcode_for_file_access(),
! errmsg("could not read from file \"%s\": %m",
cstate->filename)));
}
! /*
! * If we skipped writing WAL, then we need to sync the heap (but not
! * indexes since those use WAL anyway)
! */
! if (hi_options & HEAP_INSERT_SKIP_WAL)
! heap_sync(cstate->rel);
}
--- 2393,2437 ----
* provided by the input data. Anything not processed here or above
* will remain NULL.
*/
+ econtext = GetPerTupleExprContext(cstate->estate);
for (i = 0; i < num_defaults; i++)
{
values[defmap[i]] = ExecEvalExpr(defexprs[i], econtext,
&nulls[defmap[i]], NULL);
}
! return true;
! }
! /*
! * Clean up storage and release resources for COPY FROM.
! */
! void
! EndCopyFrom(CopyState cstate)
! {
! FreeExecutorState(cstate->estate);
! if (cstate->filename)
{
if (FreeFile(cstate->copy_file))
ereport(ERROR,
(errcode_for_file_access(),
! errmsg("could not close file \"%s\": %m",
cstate->filename)));
+ pfree(cstate->filename);
}
! /* Clean up storage */
! if (!cstate->binary)
! pfree(cstate->raw_fields);
! pfree(cstate->attribute_buf.data);
! pfree(cstate->line_buf.data);
! pfree(cstate->raw_buf);
! pfree(cstate->in_functions);
! pfree(cstate->typioparams);
! pfree(cstate->defmap);
! pfree(cstate->defexprs);
! pfree(cstate);
}
*************** CopyGetAttnums(TupleDesc tupDesc, Relati
*** 3502,3507 ****
--- 3672,3684 ----
return attnums;
}
+ /* retrieve internal EState in CopyState */
+ EState *
+ GetCopyExecutorState(CopyState cstate)
+ {
+ return cstate->estate;
+ }
+
/*
* copy_dest_startup --- executor startup
*************** copy_dest_receive(TupleTableSlot *slot,
*** 3526,3531 ****
--- 3703,3709 ----
/* And send the data */
CopyOneRowTo(cstate, InvalidOid, slot->tts_values, slot->tts_isnull);
+ myState->processed++;
}
/*
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index 6d409e8..8c69703 100644
*** a/src/include/commands/copy.h
--- b/src/include/commands/copy.h
***************
*** 14,25 ****
--- 14,36 ----
#ifndef COPY_H
#define COPY_H
+ #include "nodes/execnodes.h"
#include "nodes/parsenodes.h"
#include "tcop/dest.h"
+ typedef struct CopyStateData *CopyState;
+
extern uint64 DoCopy(const CopyStmt *stmt, const char *queryString);
+ extern CopyState BeginCopyFrom(Relation rel, const char *filename,
+ List *attnamelist, List *options);
+ extern void EndCopyFrom(CopyState cstate);
+ extern bool NextCopyFrom(CopyState cstate,
+ Datum *values, bool *nulls, Oid *tupleOid);
+ extern EState *GetCopyExecutorState(CopyState cstate);
+ extern void CopyFromErrorCallback(void *arg);
+
extern DestReceiver *CreateCopyDestReceiver(void);
#endif /* COPY_H */
On Mon, 20 Dec 2010 20:42:38 +0900
Itagaki Takahiro <itagaki.takahiro@gmail.com> wrote:
On Sun, Dec 19, 2010 at 12:45, Robert Haas <robertmhaas@gmail.com> wrote:
I'm not questioning any of that. But I'd like the resulting code to
be as maintainable as we can make it.I added comments and moved some setup codes for COPY TO to BeginCopyTo()
for maintainability. CopyTo() still contains parts of initialization,
but I've not touched it yet because we don't need the arrangement for now.
Attached is the revised version of file_fdw patch. This patch is
based on Itagaki-san's copy_export-20101220.diff patch.
Changes from previous version are:
* file_fdw uses CopyErrorCallback() as error context callback routine
in fileIterate() to report error context. "CONTEXT" line in the
example below is added by the callback.
postgres=# select * From csv_tellers_bad;
ERROR: missing data for column "bid"
CONTEXT: COPY csv_tellers_bad, line 10: "10"
postgres=#
* Only superusers can change table-level file_fdw options. Normal
user can't change the options even if the user was the owner of the
table. This is for security reason.
Regards,
--
Shigeru Hanada
Attachments:
On Tue, Dec 21, 2010 at 20:14, Shigeru HANADA <hanada@metrosystems.co.jp> wrote:
Attached is the revised version of file_fdw patch. This patch is
based on Itagaki-san's copy_export-20101220.diff patch.
#1. Don't you have per-tuple memory leak? I added GetCopyExecutorState()
because the caller needs to reset the per-tuple context periodically.
Or, if you eventually make a HeapTuple from values and nulls arrays,
you could modify NextCopyFrom() to return a HeapTuple instead of values,
nulls, and tupleOid. The reason I didn't use HeapTuple is that I've
seen arrays were used in the proposed FDW APIs. But we don't have to
use such arrays if you use HeapTuple based APIs.
IMHO, I prefer HeapTuple because we can simplify NextCopyFrom and
keep EState private in copy.c.
#2. Can you avoid making EXPLAIN text in fplan->explainInfo on
non-EXPLAIN cases? It's a waste of CPU cycles in normal executions.
I doubt whether FdwPlan.explainInfo field is the best design.
How do we use the EXPLAIN text for XML or JSON explain formats?
Instead, we could have an additional routine for EXPLAIN.
#3. Why do you re-open a foreign table in estimate_costs() ?
Since the caller seems to have the options for them, you can
pass them directly, no?
In addition, passing a half-initialized fplan to estimate_costs()
is a bad idea. If you think it is an OUT parameter, the OUT params
should be *startup_cost and *total_cost.
#4. It'a minor cosmetic point, but our coding conventions would be
we don't need (void *) when we cast a pointer to void *, but need
(Type *) when we cast a void pointer to another type.
--
Itagaki Takahiro
On Mon, Dec 20, 2010 at 6:42 AM, Itagaki Takahiro
<itagaki.takahiro@gmail.com> wrote:
On Sun, Dec 19, 2010 at 12:45, Robert Haas <robertmhaas@gmail.com> wrote:
I'm not questioning any of that. But I'd like the resulting code to
be as maintainable as we can make it.I added comments and moved some setup codes for COPY TO to BeginCopyTo()
for maintainability. CopyTo() still contains parts of initialization,
but I've not touched it yet because we don't need the arrangement for now.
I haven't analyzed this enough to know whether I agree with it, but as
a trivial matter you should certainly revert this hunk:
/* field raw data pointers found by COPY FROM */
-
- int max_fields;
- char ** raw_fields;
+ int max_fields;
+ char **raw_fields;
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Tue, Dec 21, 2010 at 21:32, Itagaki Takahiro
<itagaki.takahiro@gmail.com> wrote:
On Tue, Dec 21, 2010 at 20:14, Shigeru HANADA <hanada@metrosystems.co.jp> wrote:
Attached is the revised version of file_fdw patch. This patch is
based on Itagaki-san's copy_export-20101220.diff patch.#1. Don't you have per-tuple memory leak? I added GetCopyExecutorState()
because the caller needs to reset the per-tuple context periodically.
Sorry, I found there are no memory leak here. The related comment is:
[execnodes.h]
* CurrentMemoryContext should be set to ecxt_per_tuple_memory before
* calling ExecEvalExpr() --- see ExecEvalExprSwitchContext().
I guess CurrentMemoryContext in Iterate callback a per-tuple context.
So, we don't have to xport cstate->estate via GetCopyExecutorState().
Or, if you eventually make a HeapTuple from values and nulls arrays,
ExecStoreVirtualTuple() seems to be better than the combination of
heap_form_tuple() and ExecStoreTuple() for the purpose. Could you try
to use slot->tts_values and slot->tts_isnull for NextCopyFrom() directly?
--
Itagaki Takahiro
On Fri, 24 Dec 2010 11:09:16 +0900
Itagaki Takahiro <itagaki.takahiro@gmail.com> wrote:
On Tue, Dec 21, 2010 at 21:32, Itagaki Takahiro
<itagaki.takahiro@gmail.com> wrote:On Tue, Dec 21, 2010 at 20:14, Shigeru HANADA <hanada@metrosystems.co.jp> wrote:
Attached is the revised version of file_fdw patch. This patch is
based on Itagaki-san's copy_export-20101220.diff patch.#1. Don't you have per-tuple memory leak? I added GetCopyExecutorState()
because the caller needs to reset the per-tuple context periodically.Sorry, I found there are no memory leak here. The related comment is:
[execnodes.h]
* CurrentMemoryContext should be set to ecxt_per_tuple_memory before
* calling ExecEvalExpr() --- see ExecEvalExprSwitchContext().
I guess CurrentMemoryContext in Iterate callback a per-tuple context.
So, we don't have to xport cstate->estate via GetCopyExecutorState().
Iterate is called in query context, so GetCopyExecutorState() need to
be exported to avoid memory leak happens in NextCopyFrom(). Or,
enclosing context switching into NextCopyFrom() is better? Then,
CopyFrom() would need to create tuples in Portal context and set
shouldFree of ExecStoreTuple() true to free stored tuple at next call.
Please try attached patch.
Or, if you eventually make a HeapTuple from values and nulls arrays,
ExecStoreVirtualTuple() seems to be better than the combination of
heap_form_tuple() and ExecStoreTuple() for the purpose. Could you try
to use slot->tts_values and slot->tts_isnull for NextCopyFrom() directly?
Virtual tuple would be enough to carry column data, but virtual tuple
doesn't have system attributes including tableoid...
Regards,
--
Shigeru Hanada
Attachments:
20101224-switch_in_next.patchapplication/octet-stream; name=20101224-switch_in_next.patchDownload
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index d884764..c0c228f 100644
*** a/src/backend/commands/copy.c
--- b/src/backend/commands/copy.c
*************** CopyFrom(CopyState cstate)
*** 1875,1886 ****
CHECK_FOR_INTERRUPTS();
- /* Reset the per-tuple exprcontext */
- ResetPerTupleExprContext(estate);
-
- /* Switch into its memory context */
- MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
-
done = !NextCopyFrom(cstate, values, nulls, &loaded_oid);
if (done)
break;
--- 1875,1880 ----
*************** CopyFrom(CopyState cstate)
*** 1891,1899 ****
if (loaded_oid != InvalidOid)
HeapTupleSetOid(tuple, loaded_oid);
- /* Triggers and stuff need to be invoked in query context. */
- MemoryContextSwitchTo(oldcontext);
-
skip_tuple = false;
/* BEFORE ROW INSERT Triggers */
--- 1885,1890 ----
*************** CopyFrom(CopyState cstate)
*** 1918,1924 ****
List *recheckIndexes = NIL;
/* Place tuple in tuple slot */
! ExecStoreTuple(tuple, slot, InvalidBuffer, false);
/* Check the constraints of the tuple */
if (cstate->rel->rd_att->constr)
--- 1909,1915 ----
List *recheckIndexes = NIL;
/* Place tuple in tuple slot */
! ExecStoreTuple(tuple, slot, InvalidBuffer, true);
/* Check the constraints of the tuple */
if (cstate->rel->rd_att->constr)
*************** BeginCopyFrom(Relation rel,
*** 2170,2179 ****
--- 2161,2173 ----
* valus and nulls arrays must be the same length as columns of the
* relation passed to BeginCopyFrom. Oid of the tuple is returned with
* tupleOid separately.
+ *
+ * We do everything in per-tuple context and it would be reset in the head.
*/
bool
NextCopyFrom(CopyState cstate, Datum *values, bool *nulls, Oid *tupleOid)
{
+ MemoryContext oldcontext = CurrentMemoryContext;
TupleDesc tupDesc;
Form_pg_attribute *attr;
AttrNumber num_phys_attrs,
*************** NextCopyFrom(CopyState cstate, Datum *va
*** 2190,2201 ****
--- 2184,2202 ----
ExprState **defexprs = cstate->defexprs;
ExprContext *econtext; /* used for ExecEvalExpr for default atts */
+ /* Reset the per-tuple exprcontext and switch into its memory context */
+ ResetPerTupleExprContext(cstate->estate);
+ MemoryContextSwitchTo(GetPerTupleMemoryContext(cstate->estate));
+
/* on input just throw the header line away */
if (cstate->cur_lineno == 0 && cstate->header_line)
{
cstate->cur_lineno++;
if (CopyReadLine(cstate))
+ {
+ MemoryContextSwitchTo(oldcontext);
return false; /* done */
+ }
}
tupDesc = RelationGetDescr(cstate->rel);
*************** NextCopyFrom(CopyState cstate, Datum *va
*** 2228,2234 ****
--- 2229,2238 ----
* EOF, ie, process the line and then exit loop on next iteration.
*/
if (done && cstate->line_buf.len == 0)
+ {
+ MemoryContextSwitchTo(oldcontext);
return false;
+ }
/* Parse the line into de-escaped field values */
if (cstate->csv_mode)
*************** NextCopyFrom(CopyState cstate, Datum *va
*** 2316,2321 ****
--- 2320,2326 ----
if (!CopyGetInt16(cstate, &fld_count))
{
/* EOF detected (end of file, or protocol-level EOF) */
+ MemoryContextSwitchTo(oldcontext);
return false;
}
*************** NextCopyFrom(CopyState cstate, Datum *va
*** 2340,2345 ****
--- 2345,2351 ----
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("received copy data after EOF marker")));
+ MemoryContextSwitchTo(oldcontext);
return false;
}
*************** NextCopyFrom(CopyState cstate, Datum *va
*** 2400,2405 ****
--- 2406,2412 ----
&nulls[defmap[i]], NULL);
}
+ MemoryContextSwitchTo(oldcontext);
return true;
}
On Fri, Dec 24, 2010 at 20:04, Shigeru HANADA <hanada@metrosystems.co.jp> wrote:
Iterate is called in query context,
Is it an unavoidable requirement? If possible, I'd like to use per-tuple memory
context as the default. We use per-tuple context in FunctionScan for SETOF
functions. I hope we could have little difference between SRF and FDW APIs.
Virtual tuple would be enough to carry column data, but virtual tuple
doesn't have system attributes including tableoid...
We could add tts_tableOid into TupleTableSlot. We'd better avoid
materializing slot only for the tableoid support in foreign tables.
Almost all of the foreign tables should have different data format
from HeapTuple, including pgsql_fdw.
--
Itagaki Takahiro
On Mon, Dec 20, 2010 at 20:42, Itagaki Takahiro
<itagaki.takahiro@gmail.com> wrote:
I added comments and moved some setup codes for COPY TO to BeginCopyTo()
for maintainability. CopyTo() still contains parts of initialization,
but I've not touched it yet because we don't need the arrangement for now.
I updated the COPY FROM API patch.
- GetCopyExecutorState() is removed because FDWs will use their own context.
The patch just rearranges codes for COPY FROM to export those functions.
It also modifies some of COPY TO codes internally for code readability.
- BeginCopyFrom(rel, filename, attnamelist, options)
- EndCopyFrom(cstate)
- NextCopyFrom(cstate, OUT values, OUT nulls, OUT tupleOid)
- CopyFromErrorCallback(arg)
Some items to be considered:
- BeginCopyFrom() could receive filename as an option instead of a separated
argument. If do so, file_fdw would be more simple, but it's a change only for
file_fdw. COPY commands in the core won't be improved at all.
- NextCopyFrom() returns values/nulls arrays rather than a HeapTuple. I expect
the caller store the result into tupletableslot with ExecStoreVirtualTuple().
It is designed for performance, but if the caller always needs an materialized
HeapTuple, HeapTuple is better for the result type.
--
Itagaki Takahiro
Attachments:
copy_export-20111007.patchapplication/octet-stream; name=copy_export-20111007.patchDownload
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index 841bf22..00139b9 100644
*** a/src/backend/commands/copy.c
--- b/src/backend/commands/copy.c
*************** typedef struct CopyStateData
*** 93,105 ****
FILE *copy_file; /* used if copy_dest == COPY_FILE */
StringInfo fe_msgbuf; /* used for all dests during COPY TO, only for
* dest == COPY_NEW_FE in COPY FROM */
- bool fe_copy; /* true for all FE copy dests */
bool fe_eof; /* true if detected end of copy data */
EolType eol_type; /* EOL type of input */
int client_encoding; /* remote side's character encoding */
bool need_transcoding; /* client encoding diff from server? */
bool encoding_embeds_ascii; /* ASCII can be non-first byte? */
- uint64 processed; /* # of tuples processed */
/* parameters from the COPY command */
Relation rel; /* relation to copy to or from */
--- 93,103 ----
*************** typedef struct CopyStateData
*** 119,125 ****
bool *force_quote_flags; /* per-column CSV FQ flags */
bool *force_notnull_flags; /* per-column CSV FNN flags */
! /* these are just for error messages, see copy_in_error_callback */
const char *cur_relname; /* table name for error messages */
int cur_lineno; /* line number for error messages */
const char *cur_attname; /* current att for error messages */
--- 117,123 ----
bool *force_quote_flags; /* per-column CSV FQ flags */
bool *force_notnull_flags; /* per-column CSV FNN flags */
! /* these are just for error messages, see CopyFromErrorCallback */
const char *cur_relname; /* table name for error messages */
int cur_lineno; /* line number for error messages */
const char *cur_attname; /* current att for error messages */
*************** typedef struct CopyStateData
*** 142,148 ****
StringInfoData attribute_buf;
/* field raw data pointers found by COPY FROM */
-
int max_fields;
char ** raw_fields;
--- 140,145 ----
*************** typedef struct CopyStateData
*** 167,181 ****
char *raw_buf;
int raw_buf_index; /* next byte to process */
int raw_buf_len; /* total # of bytes stored */
- } CopyStateData;
! typedef CopyStateData *CopyState;
/* DestReceiver for COPY (SELECT) TO */
typedef struct
{
DestReceiver pub; /* publicly-known function pointers */
CopyState cstate; /* CopyStateData for the command */
} DR_copy;
--- 164,191 ----
char *raw_buf;
int raw_buf_index; /* next byte to process */
int raw_buf_len; /* total # of bytes stored */
! /*
! * The definition of input functions and default expressions are stored
! * in these variables.
! */
! EState *estate;
! AttrNumber num_defaults;
! bool file_has_oids;
! FmgrInfo oid_in_function;
! Oid oid_typioparam;
! FmgrInfo *in_functions;
! Oid *typioparams;
! int *defmap;
! ExprState **defexprs; /* array of default att expressions */
! } CopyStateData;
/* DestReceiver for COPY (SELECT) TO */
typedef struct
{
DestReceiver pub; /* publicly-known function pointers */
CopyState cstate; /* CopyStateData for the command */
+ uint64 processed; /* # of tuples processed */
} DR_copy;
*************** static const char BinarySignature[11] =
*** 248,258 ****
/* non-export function prototypes */
! static void DoCopyTo(CopyState cstate);
! static void CopyTo(CopyState cstate);
static void CopyOneRowTo(CopyState cstate, Oid tupleOid,
Datum *values, bool *nulls);
! static void CopyFrom(CopyState cstate);
static bool CopyReadLine(CopyState cstate);
static bool CopyReadLineText(CopyState cstate);
static int CopyReadAttributesText(CopyState cstate);
--- 258,273 ----
/* non-export function prototypes */
! static CopyState BeginCopy(bool is_from, Relation rel, Node *raw_query,
! const char *queryString, List *attnamelist, List *options);
! static CopyState BeginCopyTo(Relation rel, Node *query, const char *queryString,
! const char *filename, List *attnamelist, List *options);
! static void EndCopyTo(CopyState cstate);
! static uint64 DoCopyTo(CopyState cstate);
! static uint64 CopyTo(CopyState cstate);
static void CopyOneRowTo(CopyState cstate, Oid tupleOid,
Datum *values, bool *nulls);
! static uint64 CopyFrom(CopyState cstate);
static bool CopyReadLine(CopyState cstate);
static bool CopyReadLineText(CopyState cstate);
static int CopyReadAttributesText(CopyState cstate);
*************** DoCopy(const CopyStmt *stmt, const char
*** 724,745 ****
CopyState cstate;
bool is_from = stmt->is_from;
bool pipe = (stmt->filename == NULL);
! List *attnamelist = stmt->attlist;
List *force_quote = NIL;
List *force_notnull = NIL;
bool force_quote_all = false;
bool format_specified = false;
- AclMode required_access = (is_from ? ACL_INSERT : ACL_SELECT);
ListCell *option;
TupleDesc tupDesc;
int num_phys_attrs;
- uint64 processed;
/* Allocate workspace and zero all fields */
cstate = (CopyStateData *) palloc0(sizeof(CopyStateData));
/* Extract options from the statement node tree */
! foreach(option, stmt->options)
{
DefElem *defel = (DefElem *) lfirst(option);
--- 739,850 ----
CopyState cstate;
bool is_from = stmt->is_from;
bool pipe = (stmt->filename == NULL);
! Relation rel;
! uint64 processed;
!
! /* Disallow file COPY except to superusers. */
! if (!pipe && !superuser())
! ereport(ERROR,
! (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
! errmsg("must be superuser to COPY to or from a file"),
! errhint("Anyone can COPY to stdout or from stdin. "
! "psql's \\copy command also works for anyone.")));
!
! if (stmt->relation)
! {
! TupleDesc tupDesc;
! AclMode required_access = (is_from ? ACL_INSERT : ACL_SELECT);
! RangeTblEntry *rte;
! List *attnums;
! ListCell *cur;
!
! Assert(!stmt->query);
!
! /* Open and lock the relation, using the appropriate lock type. */
! rel = heap_openrv(stmt->relation,
! (is_from ? RowExclusiveLock : AccessShareLock));
!
! rte = makeNode(RangeTblEntry);
! rte->rtekind = RTE_RELATION;
! rte->relid = RelationGetRelid(rel);
! rte->requiredPerms = required_access;
!
! tupDesc = RelationGetDescr(rel);
! attnums = CopyGetAttnums(tupDesc, rel, stmt->attlist);
! foreach (cur, attnums)
! {
! int attno = lfirst_int(cur) -
! FirstLowInvalidHeapAttributeNumber;
!
! if (is_from)
! rte->modifiedCols = bms_add_member(rte->modifiedCols, attno);
! else
! rte->selectedCols = bms_add_member(rte->selectedCols, attno);
! }
! ExecCheckRTPerms(list_make1(rte), true);
! }
! else
! {
! Assert(stmt->query);
!
! rel = NULL;
! }
!
! if (is_from)
! {
! /* check read-only transaction */
! if (XactReadOnly && rel->rd_backend != MyBackendId)
! PreventCommandIfReadOnly("COPY FROM");
!
! cstate = BeginCopyFrom(rel, stmt->filename,
! stmt->attlist, stmt->options);
! processed = CopyFrom(cstate); /* copy from file to database */
! EndCopyFrom(cstate);
! }
! else
! {
! cstate = BeginCopyTo(rel, stmt->query, queryString, stmt->filename,
! stmt->attlist, stmt->options);
! processed = DoCopyTo(cstate); /* copy from database to file */
! EndCopyTo(cstate);
! }
!
! /*
! * Close the relation. If reading, we can release the AccessShareLock we got;
! * if writing, we should hold the lock until end of transaction to ensure that
! * updates will be committed before lock is released.
! */
! if (rel != NULL)
! heap_close(rel, (is_from ? NoLock : AccessShareLock));
!
! return processed;
! }
!
! /*
! * Common setup routines used by BeginCopyFrom and BeginCopyTo.
! */
! static CopyState
! BeginCopy(bool is_from,
! Relation rel,
! Node *raw_query,
! const char *queryString,
! List *attnamelist,
! List *options)
! {
! CopyState cstate;
List *force_quote = NIL;
List *force_notnull = NIL;
bool force_quote_all = false;
bool format_specified = false;
ListCell *option;
TupleDesc tupDesc;
int num_phys_attrs;
/* Allocate workspace and zero all fields */
cstate = (CopyStateData *) palloc0(sizeof(CopyStateData));
/* Extract options from the statement node tree */
! foreach(option, options)
{
DefElem *defel = (DefElem *) lfirst(option);
*************** DoCopy(const CopyStmt *stmt, const char
*** 980,1030 ****
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("CSV quote character must not appear in the NULL specification")));
! /* Disallow file COPY except to superusers. */
! if (!pipe && !superuser())
! ereport(ERROR,
! (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
! errmsg("must be superuser to COPY to or from a file"),
! errhint("Anyone can COPY to stdout or from stdin. "
! "psql's \\copy command also works for anyone.")));
!
! if (stmt->relation)
{
! RangeTblEntry *rte;
! List *attnums;
! ListCell *cur;
!
! Assert(!stmt->query);
! cstate->queryDesc = NULL;
! /* Open and lock the relation, using the appropriate lock type. */
! cstate->rel = heap_openrv(stmt->relation,
! (is_from ? RowExclusiveLock : AccessShareLock));
tupDesc = RelationGetDescr(cstate->rel);
- /* Check relation permissions. */
- rte = makeNode(RangeTblEntry);
- rte->rtekind = RTE_RELATION;
- rte->relid = RelationGetRelid(cstate->rel);
- rte->requiredPerms = required_access;
-
- attnums = CopyGetAttnums(tupDesc, cstate->rel, attnamelist);
- foreach (cur, attnums)
- {
- int attno = lfirst_int(cur) - FirstLowInvalidHeapAttributeNumber;
-
- if (is_from)
- rte->modifiedCols = bms_add_member(rte->modifiedCols, attno);
- else
- rte->selectedCols = bms_add_member(rte->selectedCols, attno);
- }
- ExecCheckRTPerms(list_make1(rte), true);
-
- /* check read-only transaction */
- if (XactReadOnly && is_from && cstate->rel->rd_backend != MyBackendId)
- PreventCommandIfReadOnly("COPY FROM");
-
/* Don't allow COPY w/ OIDs to or from a table without them */
if (cstate->oids && !cstate->rel->rd_rel->relhasoids)
ereport(ERROR,
--- 1085,1098 ----
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("CSV quote character must not appear in the NULL specification")));
! if (rel)
{
! Assert(!raw_query);
! cstate->rel = rel;
tupDesc = RelationGetDescr(cstate->rel);
/* Don't allow COPY w/ OIDs to or from a table without them */
if (cstate->oids && !cstate->rel->rd_rel->relhasoids)
ereport(ERROR,
*************** DoCopy(const CopyStmt *stmt, const char
*** 1058,1064 ****
* function and is executed repeatedly. (See also the same hack in
* DECLARE CURSOR and PREPARE.) XXX FIXME someday.
*/
! rewritten = pg_analyze_and_rewrite((Node *) copyObject(stmt->query),
queryString, NULL, 0);
/* We don't expect more or less than one result query */
--- 1126,1132 ----
* function and is executed repeatedly. (See also the same hack in
* DECLARE CURSOR and PREPARE.) XXX FIXME someday.
*/
! rewritten = pg_analyze_and_rewrite((Node *) copyObject(raw_query),
queryString, NULL, 0);
/* We don't expect more or less than one result query */
*************** DoCopy(const CopyStmt *stmt, const char
*** 1160,1173 ****
}
}
- /* Set up variables to avoid per-attribute overhead. */
- initStringInfo(&cstate->attribute_buf);
- initStringInfo(&cstate->line_buf);
- cstate->line_buf_converted = false;
- cstate->raw_buf = (char *) palloc(RAW_BUF_SIZE + 1);
- cstate->raw_buf_index = cstate->raw_buf_len = 0;
- cstate->processed = 0;
-
/*
* Set up encoding conversion info. Even if the client and server
* encodings are the same, we must apply pg_client_to_server() to validate
--- 1228,1233 ----
*************** DoCopy(const CopyStmt *stmt, const char
*** 1181,1229 ****
cstate->encoding_embeds_ascii = PG_ENCODING_IS_CLIENT_ONLY(cstate->client_encoding);
cstate->copy_dest = COPY_FILE; /* default */
- cstate->filename = stmt->filename;
-
- if (is_from)
- CopyFrom(cstate); /* copy from file to database */
- else
- DoCopyTo(cstate); /* copy from database to file */
-
- /*
- * Close the relation or query. If reading, we can release the
- * AccessShareLock we got; if writing, we should hold the lock until end
- * of transaction to ensure that updates will be committed before lock is
- * released.
- */
- if (cstate->rel)
- heap_close(cstate->rel, (is_from ? NoLock : AccessShareLock));
- else
- {
- /* Close down the query and free resources. */
- ExecutorEnd(cstate->queryDesc);
- FreeQueryDesc(cstate->queryDesc);
- PopActiveSnapshot();
- }
! /* Clean up storage (probably not really necessary) */
! processed = cstate->processed;
!
! pfree(cstate->attribute_buf.data);
! pfree(cstate->line_buf.data);
! pfree(cstate->raw_buf);
! pfree(cstate);
!
! return processed;
}
/*
! * This intermediate routine exists mainly to localize the effects of setjmp
! * so we don't need to plaster a lot of variables with "volatile".
*/
! static void
! DoCopyTo(CopyState cstate)
{
! bool pipe = (cstate->filename == NULL);
if (cstate->rel)
{
--- 1241,1266 ----
cstate->encoding_embeds_ascii = PG_ENCODING_IS_CLIENT_ONLY(cstate->client_encoding);
cstate->copy_dest = COPY_FILE; /* default */
! return cstate;
}
/*
! * Setup CopyState to read tuples from a table or a query for COPY TO.
*/
! static CopyState
! BeginCopyTo(Relation rel,
! Node *query,
! const char *queryString,
! const char *filename,
! List *attnamelist,
! List *options)
{
! CopyState cstate;
! bool pipe = (filename == NULL);
!
! cstate = BeginCopy(false, rel, query, queryString, attnamelist, options);
if (cstate->rel)
{
*************** DoCopyTo(CopyState cstate)
*** 1256,1264 ****
if (pipe)
{
! if (whereToSendOutput == DestRemote)
! cstate->fe_copy = true;
! else
cstate->copy_file = stdout;
}
else
--- 1293,1299 ----
if (pipe)
{
! if (whereToSendOutput != DestRemote)
cstate->copy_file = stdout;
}
else
*************** DoCopyTo(CopyState cstate)
*** 1270,1280 ****
* Prevent write to relative path ... too easy to shoot oneself in the
* foot by overwriting a database file ...
*/
! if (!is_absolute_path(cstate->filename))
ereport(ERROR,
(errcode(ERRCODE_INVALID_NAME),
errmsg("relative path not allowed for COPY to file")));
oumask = umask(S_IWGRP | S_IWOTH);
cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_W);
umask(oumask);
--- 1305,1316 ----
* Prevent write to relative path ... too easy to shoot oneself in the
* foot by overwriting a database file ...
*/
! if (!is_absolute_path(filename))
ereport(ERROR,
(errcode(ERRCODE_INVALID_NAME),
errmsg("relative path not allowed for COPY to file")));
+ cstate->filename = pstrdup(filename);
oumask = umask(S_IWGRP | S_IWOTH);
cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_W);
umask(oumask);
*************** DoCopyTo(CopyState cstate)
*** 1292,1305 ****
errmsg("\"%s\" is a directory", cstate->filename)));
}
PG_TRY();
{
! if (cstate->fe_copy)
SendCopyBegin(cstate);
! CopyTo(cstate);
! if (cstate->fe_copy)
SendCopyEnd(cstate);
}
PG_CATCH();
--- 1328,1355 ----
errmsg("\"%s\" is a directory", cstate->filename)));
}
+ return cstate;
+ }
+
+ /*
+ * This intermediate routine exists mainly to localize the effects of setjmp
+ * so we don't need to plaster a lot of variables with "volatile".
+ */
+ static uint64
+ DoCopyTo(CopyState cstate)
+ {
+ bool pipe = (cstate->filename == NULL);
+ bool fe_copy = (pipe && whereToSendOutput == DestRemote);
+ uint64 processed;
+
PG_TRY();
{
! if (fe_copy)
SendCopyBegin(cstate);
! processed = CopyTo(cstate);
! if (fe_copy)
SendCopyEnd(cstate);
}
PG_CATCH();
*************** DoCopyTo(CopyState cstate)
*** 1314,1339 ****
}
PG_END_TRY();
! if (!pipe)
{
if (FreeFile(cstate->copy_file))
ereport(ERROR,
(errcode_for_file_access(),
! errmsg("could not write to file \"%s\": %m",
cstate->filename)));
}
}
/*
* Copy from relation or query TO file.
*/
! static void
CopyTo(CopyState cstate)
{
TupleDesc tupDesc;
int num_phys_attrs;
Form_pg_attribute *attr;
ListCell *cur;
if (cstate->rel)
tupDesc = RelationGetDescr(cstate->rel);
--- 1364,1414 ----
}
PG_END_TRY();
! return processed;
! }
!
! /*
! * Clean up storage and release resources for COPY TO.
! */
! static void
! EndCopyTo(CopyState cstate)
! {
! if (cstate->filename)
{
if (FreeFile(cstate->copy_file))
ereport(ERROR,
(errcode_for_file_access(),
! errmsg("could not close file \"%s\": %m",
cstate->filename)));
+
+ pfree(cstate->filename);
}
+
+ /*
+ * Close the relation or query. We can release the AccessShareLock we got.
+ */
+ if (cstate->rel == NULL)
+ {
+ /* Close down the query and free resources. */
+ ExecutorEnd(cstate->queryDesc);
+ FreeQueryDesc(cstate->queryDesc);
+ PopActiveSnapshot();
+ }
+
+ pfree(cstate);
}
/*
* Copy from relation or query TO file.
*/
! static uint64
CopyTo(CopyState cstate)
{
TupleDesc tupDesc;
int num_phys_attrs;
Form_pg_attribute *attr;
ListCell *cur;
+ uint64 processed;
if (cstate->rel)
tupDesc = RelationGetDescr(cstate->rel);
*************** CopyTo(CopyState cstate)
*** 1439,1444 ****
--- 1514,1520 ----
scandesc = heap_beginscan(cstate->rel, GetActiveSnapshot(), 0, NULL);
+ processed = 0;
while ((tuple = heap_getnext(scandesc, ForwardScanDirection)) != NULL)
{
CHECK_FOR_INTERRUPTS();
*************** CopyTo(CopyState cstate)
*** 1448,1453 ****
--- 1524,1530 ----
/* Format and send the data */
CopyOneRowTo(cstate, HeapTupleGetOid(tuple), values, nulls);
+ processed++;
}
heap_endscan(scandesc);
*************** CopyTo(CopyState cstate)
*** 1456,1461 ****
--- 1533,1539 ----
{
/* run the plan --- the dest receiver will send tuples */
ExecutorRun(cstate->queryDesc, ForwardScanDirection, 0L);
+ processed = ((DR_copy *) cstate->queryDesc->dest)->processed;
}
if (cstate->binary)
*************** CopyTo(CopyState cstate)
*** 1467,1472 ****
--- 1545,1552 ----
}
MemoryContextDelete(cstate->rowcontext);
+
+ return processed;
}
/*
*************** CopyOneRowTo(CopyState cstate, Oid tuple
*** 1558,1573 ****
CopySendEndOfRow(cstate);
MemoryContextSwitchTo(oldcontext);
-
- cstate->processed++;
}
/*
* error context callback for COPY FROM
*/
! static void
! copy_in_error_callback(void *arg)
{
CopyState cstate = (CopyState) arg;
--- 1638,1653 ----
CopySendEndOfRow(cstate);
MemoryContextSwitchTo(oldcontext);
}
/*
* error context callback for COPY FROM
+ *
+ * The argument for the error context must be CopyState.
*/
! void
! CopyFromErrorCallback(void *arg)
{
CopyState cstate = (CopyState) arg;
*************** limit_printout_length(const char *str)
*** 1669,1709 ****
/*
* Copy FROM file to relation.
*/
! static void
CopyFrom(CopyState cstate)
{
- bool pipe = (cstate->filename == NULL);
HeapTuple tuple;
TupleDesc tupDesc;
- Form_pg_attribute *attr;
- AttrNumber num_phys_attrs,
- attr_count,
- num_defaults;
- FmgrInfo *in_functions;
- FmgrInfo oid_in_function;
- Oid *typioparams;
- Oid oid_typioparam;
- int attnum;
- int i;
- Oid in_func_oid;
Datum *values;
bool *nulls;
- int nfields;
- char **field_strings;
bool done = false;
- bool isnull;
ResultRelInfo *resultRelInfo;
! EState *estate = CreateExecutorState(); /* for ExecConstraints() */
TupleTableSlot *slot;
- bool file_has_oids;
- int *defmap;
- ExprState **defexprs; /* array of default att expressions */
- ExprContext *econtext; /* used for ExecEvalExpr for default atts */
MemoryContext oldcontext = CurrentMemoryContext;
ErrorContextCallback errcontext;
CommandId mycid = GetCurrentCommandId(true);
int hi_options = 0; /* start with default heap_insert options */
BulkInsertState bistate;
Assert(cstate->rel);
--- 1749,1771 ----
/*
* Copy FROM file to relation.
*/
! static uint64
CopyFrom(CopyState cstate)
{
HeapTuple tuple;
TupleDesc tupDesc;
Datum *values;
bool *nulls;
bool done = false;
ResultRelInfo *resultRelInfo;
! EState *estate = cstate->estate; /* for ExecConstraints() */
TupleTableSlot *slot;
MemoryContext oldcontext = CurrentMemoryContext;
ErrorContextCallback errcontext;
CommandId mycid = GetCurrentCommandId(true);
int hi_options = 0; /* start with default heap_insert options */
BulkInsertState bistate;
+ uint64 processed = 0;
Assert(cstate->rel);
*************** CopyFrom(CopyState cstate)
*** 1731,1736 ****
--- 1793,1800 ----
RelationGetRelationName(cstate->rel))));
}
+ tupDesc = RelationGetDescr(cstate->rel);
+
/*----------
* Check to see if we can avoid writing WAL
*
*************** CopyFrom(CopyState cstate)
*** 1766,1803 ****
hi_options |= HEAP_INSERT_SKIP_WAL;
}
- if (pipe)
- {
- if (whereToSendOutput == DestRemote)
- ReceiveCopyBegin(cstate);
- else
- cstate->copy_file = stdin;
- }
- else
- {
- struct stat st;
-
- cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_R);
-
- if (cstate->copy_file == NULL)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not open file \"%s\" for reading: %m",
- cstate->filename)));
-
- fstat(fileno(cstate->copy_file), &st);
- if (S_ISDIR(st.st_mode))
- ereport(ERROR,
- (errcode(ERRCODE_WRONG_OBJECT_TYPE),
- errmsg("\"%s\" is a directory", cstate->filename)));
- }
-
- tupDesc = RelationGetDescr(cstate->rel);
- attr = tupDesc->attrs;
- num_phys_attrs = tupDesc->natts;
- attr_count = list_length(cstate->attnumlist);
- num_defaults = 0;
-
/*
* We need a ResultRelInfo so we can use the regular executor's
* index-entry-making machinery. (There used to be a huge amount of code
--- 1830,1835 ----
*************** CopyFrom(CopyState cstate)
*** 1826,1832 ****
slot = ExecInitExtraTupleSlot(estate);
ExecSetSlotDescriptor(slot, tupDesc);
! econtext = GetPerTupleExprContext(estate);
/*
* Pick up the required catalog information for each attribute in the
--- 1858,2037 ----
slot = ExecInitExtraTupleSlot(estate);
ExecSetSlotDescriptor(slot, tupDesc);
! /* Prepare to catch AFTER triggers. */
! AfterTriggerBeginQuery();
!
! /*
! * Check BEFORE STATEMENT insertion triggers. It's debateable whether we
! * should do this for COPY, since it's not really an "INSERT" statement as
! * such. However, executing these triggers maintains consistency with the
! * EACH ROW triggers that we already fire on COPY.
! */
! ExecBSInsertTriggers(estate, resultRelInfo);
!
! values = (Datum *) palloc(tupDesc->natts * sizeof(Datum));
! nulls = (bool *) palloc(tupDesc->natts * sizeof(bool));
!
! bistate = GetBulkInsertState();
!
! /* Set up callback to identify error line number */
! errcontext.callback = CopyFromErrorCallback;
! errcontext.arg = (void *) cstate;
! errcontext.previous = error_context_stack;
! error_context_stack = &errcontext;
!
! while (!done)
! {
! bool skip_tuple;
! Oid loaded_oid = InvalidOid;
!
! CHECK_FOR_INTERRUPTS();
!
! /* Reset the per-tuple exprcontext */
! ResetPerTupleExprContext(estate);
!
! /* Switch into its memory context */
! MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
!
! done = !NextCopyFrom(cstate, values, nulls, &loaded_oid);
! if (done)
! break;
!
! /* And now we can form the input tuple. */
! tuple = heap_form_tuple(tupDesc, values, nulls);
!
! if (loaded_oid != InvalidOid)
! HeapTupleSetOid(tuple, loaded_oid);
!
! /* Triggers and stuff need to be invoked in query context. */
! MemoryContextSwitchTo(oldcontext);
!
! skip_tuple = false;
!
! /* BEFORE ROW INSERT Triggers */
! if (resultRelInfo->ri_TrigDesc &&
! resultRelInfo->ri_TrigDesc->trig_insert_before_row)
! {
! HeapTuple newtuple;
!
! newtuple = ExecBRInsertTriggers(estate, resultRelInfo, tuple);
!
! if (newtuple == NULL) /* "do nothing" */
! skip_tuple = true;
! else if (newtuple != tuple) /* modified by Trigger(s) */
! {
! heap_freetuple(tuple);
! tuple = newtuple;
! }
! }
!
! if (!skip_tuple)
! {
! List *recheckIndexes = NIL;
!
! /* Place tuple in tuple slot */
! ExecStoreTuple(tuple, slot, InvalidBuffer, false);
!
! /* Check the constraints of the tuple */
! if (cstate->rel->rd_att->constr)
! ExecConstraints(resultRelInfo, slot, estate);
!
! /* OK, store the tuple and create index entries for it */
! heap_insert(cstate->rel, tuple, mycid, hi_options, bistate);
!
! if (resultRelInfo->ri_NumIndices > 0)
! recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self),
! estate);
!
! /* AFTER ROW INSERT Triggers */
! ExecARInsertTriggers(estate, resultRelInfo, tuple,
! recheckIndexes);
!
! list_free(recheckIndexes);
!
! /*
! * We count only tuples not suppressed by a BEFORE INSERT trigger;
! * this is the same definition used by execMain.c for counting
! * tuples inserted by an INSERT command.
! */
! processed++;
! }
! }
!
! /* Done, clean up */
! error_context_stack = errcontext.previous;
!
! FreeBulkInsertState(bistate);
!
! MemoryContextSwitchTo(oldcontext);
!
! /* Execute AFTER STATEMENT insertion triggers */
! ExecASInsertTriggers(estate, resultRelInfo);
!
! /* Handle queued AFTER triggers */
! AfterTriggerEndQuery(estate);
!
! pfree(values);
! pfree(nulls);
!
! ExecResetTupleTable(estate->es_tupleTable, false);
!
! ExecCloseIndices(resultRelInfo);
!
! /*
! * If we skipped writing WAL, then we need to sync the heap (but not
! * indexes since those use WAL anyway)
! */
! if (hi_options & HEAP_INSERT_SKIP_WAL)
! heap_sync(cstate->rel);
!
! return processed;
! }
!
! /*
! * Setup CopyState to read tuples from a file for COPY FROM.
! */
! CopyState
! BeginCopyFrom(Relation rel,
! const char *filename,
! List *attnamelist,
! List *options)
! {
! CopyState cstate;
! bool pipe = (filename == NULL);
! TupleDesc tupDesc;
! Form_pg_attribute *attr;
! AttrNumber num_phys_attrs,
! num_defaults;
! FmgrInfo *in_functions;
! Oid *typioparams;
! int attnum;
! Oid in_func_oid;
! EState *estate = CreateExecutorState(); /* for ExecPrepareExpr() */
! int *defmap;
! ExprState **defexprs;
!
! cstate = BeginCopy(true, rel, NULL, NULL, attnamelist, options);
!
! /* Initialize state variables */
! cstate->fe_eof = false;
! cstate->eol_type = EOL_UNKNOWN;
! cstate->cur_relname = RelationGetRelationName(cstate->rel);
! cstate->cur_lineno = 0;
! cstate->cur_attname = NULL;
! cstate->cur_attval = NULL;
!
! /* Set up variables to avoid per-attribute overhead. */
! initStringInfo(&cstate->attribute_buf);
! initStringInfo(&cstate->line_buf);
! cstate->line_buf_converted = false;
! cstate->raw_buf = (char *) palloc(RAW_BUF_SIZE + 1);
! cstate->raw_buf_index = cstate->raw_buf_len = 0;
!
! tupDesc = RelationGetDescr(cstate->rel);
! attr = tupDesc->attrs;
! num_phys_attrs = tupDesc->natts;
! num_defaults = 0;
/*
* Pick up the required catalog information for each attribute in the
*************** CopyFrom(CopyState cstate)
*** 1871,1889 ****
}
}
! /* Prepare to catch AFTER triggers. */
! AfterTriggerBeginQuery();
! /*
! * Check BEFORE STATEMENT insertion triggers. It's debateable whether we
! * should do this for COPY, since it's not really an "INSERT" statement as
! * such. However, executing these triggers maintains consistency with the
! * EACH ROW triggers that we already fire on COPY.
! */
! ExecBSInsertTriggers(estate, resultRelInfo);
if (!cstate->binary)
! file_has_oids = cstate->oids; /* must rely on user to tell us... */
else
{
/* Read and verify binary header */
--- 2076,2121 ----
}
}
! /* We keep those variables in cstate. */
! cstate->estate = estate;
! cstate->in_functions = in_functions;
! cstate->typioparams = typioparams;
! cstate->defmap = defmap;
! cstate->defexprs = defexprs;
! cstate->num_defaults = num_defaults;
! if (pipe)
! {
! if (whereToSendOutput == DestRemote)
! ReceiveCopyBegin(cstate);
! else
! cstate->copy_file = stdin;
! }
! else
! {
! struct stat st;
!
! cstate->filename = pstrdup(filename);
! cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_R);
!
! if (cstate->copy_file == NULL)
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not open file \"%s\" for reading: %m",
! cstate->filename)));
!
! fstat(fileno(cstate->copy_file), &st);
! if (S_ISDIR(st.st_mode))
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("\"%s\" is a directory", cstate->filename)));
! }
if (!cstate->binary)
! {
! /* must rely on user to tell us... */
! cstate->file_has_oids = cstate->oids;
! }
else
{
/* Read and verify binary header */
*************** CopyFrom(CopyState cstate)
*** 1901,1907 ****
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid COPY file header (missing flags)")));
! file_has_oids = (tmp & (1 << 16)) != 0;
tmp &= ~(1 << 16);
if ((tmp >> 16) != 0)
ereport(ERROR,
--- 2133,2139 ----
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid COPY file header (missing flags)")));
! cstate->file_has_oids = (tmp & (1 << 16)) != 0;
tmp &= ~(1 << 16);
if ((tmp >> 16) != 0)
ereport(ERROR,
*************** CopyFrom(CopyState cstate)
*** 1923,1984 ****
}
}
! if (file_has_oids && cstate->binary)
{
getTypeBinaryInputInfo(OIDOID,
! &in_func_oid, &oid_typioparam);
! fmgr_info(in_func_oid, &oid_in_function);
}
- values = (Datum *) palloc(num_phys_attrs * sizeof(Datum));
- nulls = (bool *) palloc(num_phys_attrs * sizeof(bool));
-
/* create workspace for CopyReadAttributes results */
! nfields = file_has_oids ? (attr_count + 1) : attr_count;
! if (! cstate->binary)
{
cstate->max_fields = nfields;
cstate->raw_fields = (char **) palloc(nfields * sizeof(char *));
}
! /* Initialize state variables */
! cstate->fe_eof = false;
! cstate->eol_type = EOL_UNKNOWN;
! cstate->cur_relname = RelationGetRelationName(cstate->rel);
! cstate->cur_lineno = 0;
! cstate->cur_attname = NULL;
! cstate->cur_attval = NULL;
!
! bistate = GetBulkInsertState();
! /* Set up callback to identify error line number */
! errcontext.callback = copy_in_error_callback;
! errcontext.arg = (void *) cstate;
! errcontext.previous = error_context_stack;
! error_context_stack = &errcontext;
/* on input just throw the header line away */
! if (cstate->header_line)
{
cstate->cur_lineno++;
! done = CopyReadLine(cstate);
}
! while (!done)
! {
! bool skip_tuple;
! Oid loaded_oid = InvalidOid;
!
! CHECK_FOR_INTERRUPTS();
cstate->cur_lineno++;
- /* Reset the per-tuple exprcontext */
- ResetPerTupleExprContext(estate);
-
- /* Switch into its memory context */
- MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
-
/* Initialize all values for row to NULL */
MemSet(values, 0, num_phys_attrs * sizeof(Datum));
MemSet(nulls, true, num_phys_attrs * sizeof(bool));
--- 2155,2223 ----
}
}
! if (cstate->file_has_oids && cstate->binary)
{
getTypeBinaryInputInfo(OIDOID,
! &in_func_oid, &cstate->oid_typioparam);
! fmgr_info(in_func_oid, &cstate->oid_in_function);
}
/* create workspace for CopyReadAttributes results */
! if (!cstate->binary)
{
+ AttrNumber attr_count = list_length(cstate->attnumlist);
+ int nfields = cstate->file_has_oids ? (attr_count + 1) : attr_count;
+
cstate->max_fields = nfields;
cstate->raw_fields = (char **) palloc(nfields * sizeof(char *));
}
! return cstate;
! }
! /*
! * Read next tuple from file for COPY FROM. Return false if no more tuples.
! *
! * valus and nulls arrays must be the same length as columns of the
! * relation passed to BeginCopyFrom. Oid of the tuple is returned with
! * tupleOid separately.
! */
! bool
! NextCopyFrom(CopyState cstate, Datum *values, bool *nulls, Oid *tupleOid)
! {
! TupleDesc tupDesc;
! Form_pg_attribute *attr;
! AttrNumber num_phys_attrs,
! attr_count,
! num_defaults = cstate->num_defaults;
! FmgrInfo *in_functions = cstate->in_functions;
! Oid *typioparams = cstate->typioparams;
! int i;
! int nfields;
! char **field_strings;
! bool isnull;
! bool file_has_oids = cstate->file_has_oids;
! int *defmap = cstate->defmap;
! ExprState **defexprs = cstate->defexprs;
! ExprContext *econtext; /* used for ExecEvalExpr for default atts */
/* on input just throw the header line away */
! if (cstate->cur_lineno == 0 && cstate->header_line)
{
cstate->cur_lineno++;
! if (CopyReadLine(cstate))
! return false; /* done */
}
! tupDesc = RelationGetDescr(cstate->rel);
! attr = tupDesc->attrs;
! num_phys_attrs = tupDesc->natts;
! attr_count = list_length(cstate->attnumlist);
! nfields = file_has_oids ? (attr_count + 1) : attr_count;
+ /* XXX: Indentation is not adjusted to keep the patch small. */
cstate->cur_lineno++;
/* Initialize all values for row to NULL */
MemSet(values, 0, num_phys_attrs * sizeof(Datum));
MemSet(nulls, true, num_phys_attrs * sizeof(bool));
*************** CopyFrom(CopyState cstate)
*** 1989,1994 ****
--- 2228,2234 ----
int fldct;
int fieldno;
char *string;
+ bool done;
/* Actually read the line into memory here */
done = CopyReadLine(cstate);
*************** CopyFrom(CopyState cstate)
*** 1999,2005 ****
* EOF, ie, process the line and then exit loop on next iteration.
*/
if (done && cstate->line_buf.len == 0)
! break;
/* Parse the line into de-escaped field values */
if (cstate->csv_mode)
--- 2239,2245 ----
* EOF, ie, process the line and then exit loop on next iteration.
*/
if (done && cstate->line_buf.len == 0)
! return false;
/* Parse the line into de-escaped field values */
if (cstate->csv_mode)
*************** CopyFrom(CopyState cstate)
*** 2029,2041 ****
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("null OID in COPY data")));
! else
{
cstate->cur_attname = "oid";
cstate->cur_attval = string;
! loaded_oid = DatumGetObjectId(DirectFunctionCall1(oidin,
! CStringGetDatum(string)));
! if (loaded_oid == InvalidOid)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid OID in COPY data")));
--- 2269,2281 ----
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("null OID in COPY data")));
! else if (cstate->oids && tupleOid != NULL)
{
cstate->cur_attname = "oid";
cstate->cur_attval = string;
! *tupleOid = DatumGetObjectId(DirectFunctionCall1(oidin,
! CStringGetDatum(string)));
! if (*tupleOid == InvalidOid)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid OID in COPY data")));
*************** CopyFrom(CopyState cstate)
*** 2087,2094 ****
if (!CopyGetInt16(cstate, &fld_count))
{
/* EOF detected (end of file, or protocol-level EOF) */
! done = true;
! break;
}
if (fld_count == -1)
--- 2327,2333 ----
if (!CopyGetInt16(cstate, &fld_count))
{
/* EOF detected (end of file, or protocol-level EOF) */
! return false;
}
if (fld_count == -1)
*************** CopyFrom(CopyState cstate)
*** 2112,2119 ****
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("received copy data after EOF marker")));
! done = true;
! break;
}
if (fld_count != attr_count)
--- 2351,2357 ----
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("received copy data after EOF marker")));
! return false;
}
if (fld_count != attr_count)
*************** CopyFrom(CopyState cstate)
*** 2124,2135 ****
if (file_has_oids)
{
cstate->cur_attname = "oid";
loaded_oid =
DatumGetObjectId(CopyReadBinaryAttribute(cstate,
0,
! &oid_in_function,
! oid_typioparam,
-1,
&isnull));
if (isnull || loaded_oid == InvalidOid)
--- 2362,2375 ----
if (file_has_oids)
{
+ Oid loaded_oid;
+
cstate->cur_attname = "oid";
loaded_oid =
DatumGetObjectId(CopyReadBinaryAttribute(cstate,
0,
! &cstate->oid_in_function,
! cstate->oid_typioparam,
-1,
&isnull));
if (isnull || loaded_oid == InvalidOid)
*************** CopyFrom(CopyState cstate)
*** 2137,2142 ****
--- 2377,2384 ----
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid OID in COPY data")));
cstate->cur_attname = NULL;
+ if (cstate->oids && tupleOid != NULL)
+ *tupleOid = loaded_oid;
}
i = 0;
*************** CopyFrom(CopyState cstate)
*** 2162,2278 ****
* provided by the input data. Anything not processed here or above
* will remain NULL.
*/
for (i = 0; i < num_defaults; i++)
{
values[defmap[i]] = ExecEvalExpr(defexprs[i], econtext,
&nulls[defmap[i]], NULL);
}
! /* And now we can form the input tuple. */
! tuple = heap_form_tuple(tupDesc, values, nulls);
!
! if (cstate->oids && file_has_oids)
! HeapTupleSetOid(tuple, loaded_oid);
!
! /* Triggers and stuff need to be invoked in query context. */
! MemoryContextSwitchTo(oldcontext);
!
! skip_tuple = false;
!
! /* BEFORE ROW INSERT Triggers */
! if (resultRelInfo->ri_TrigDesc &&
! resultRelInfo->ri_TrigDesc->trig_insert_before_row)
! {
! HeapTuple newtuple;
!
! newtuple = ExecBRInsertTriggers(estate, resultRelInfo, tuple);
!
! if (newtuple == NULL) /* "do nothing" */
! skip_tuple = true;
! else if (newtuple != tuple) /* modified by Trigger(s) */
! {
! heap_freetuple(tuple);
! tuple = newtuple;
! }
! }
!
! if (!skip_tuple)
! {
! List *recheckIndexes = NIL;
!
! /* Place tuple in tuple slot */
! ExecStoreTuple(tuple, slot, InvalidBuffer, false);
!
! /* Check the constraints of the tuple */
! if (cstate->rel->rd_att->constr)
! ExecConstraints(resultRelInfo, slot, estate);
!
! /* OK, store the tuple and create index entries for it */
! heap_insert(cstate->rel, tuple, mycid, hi_options, bistate);
!
! if (resultRelInfo->ri_NumIndices > 0)
! recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self),
! estate);
!
! /* AFTER ROW INSERT Triggers */
! ExecARInsertTriggers(estate, resultRelInfo, tuple,
! recheckIndexes);
!
! list_free(recheckIndexes);
!
! /*
! * We count only tuples not suppressed by a BEFORE INSERT trigger;
! * this is the same definition used by execMain.c for counting
! * tuples inserted by an INSERT command.
! */
! cstate->processed++;
! }
! }
!
! /* Done, clean up */
! error_context_stack = errcontext.previous;
!
! FreeBulkInsertState(bistate);
!
! MemoryContextSwitchTo(oldcontext);
!
! /* Execute AFTER STATEMENT insertion triggers */
! ExecASInsertTriggers(estate, resultRelInfo);
!
! /* Handle queued AFTER triggers */
! AfterTriggerEndQuery(estate);
!
! pfree(values);
! pfree(nulls);
! if (! cstate->binary)
! pfree(cstate->raw_fields);
!
! pfree(in_functions);
! pfree(typioparams);
! pfree(defmap);
! pfree(defexprs);
!
! ExecResetTupleTable(estate->es_tupleTable, false);
!
! ExecCloseIndices(resultRelInfo);
! FreeExecutorState(estate);
! if (!pipe)
{
if (FreeFile(cstate->copy_file))
ereport(ERROR,
(errcode_for_file_access(),
! errmsg("could not read from file \"%s\": %m",
cstate->filename)));
}
! /*
! * If we skipped writing WAL, then we need to sync the heap (but not
! * indexes since those use WAL anyway)
! */
! if (hi_options & HEAP_INSERT_SKIP_WAL)
! heap_sync(cstate->rel);
}
--- 2404,2448 ----
* provided by the input data. Anything not processed here or above
* will remain NULL.
*/
+ econtext = GetPerTupleExprContext(cstate->estate);
for (i = 0; i < num_defaults; i++)
{
values[defmap[i]] = ExecEvalExpr(defexprs[i], econtext,
&nulls[defmap[i]], NULL);
}
! return true;
! }
! /*
! * Clean up storage and release resources for COPY FROM.
! */
! void
! EndCopyFrom(CopyState cstate)
! {
! FreeExecutorState(cstate->estate);
! if (cstate->filename)
{
if (FreeFile(cstate->copy_file))
ereport(ERROR,
(errcode_for_file_access(),
! errmsg("could not close file \"%s\": %m",
cstate->filename)));
+ pfree(cstate->filename);
}
! /* Clean up storage */
! if (!cstate->binary)
! pfree(cstate->raw_fields);
! pfree(cstate->attribute_buf.data);
! pfree(cstate->line_buf.data);
! pfree(cstate->raw_buf);
! pfree(cstate->in_functions);
! pfree(cstate->typioparams);
! pfree(cstate->defmap);
! pfree(cstate->defexprs);
! pfree(cstate);
}
*************** copy_dest_receive(TupleTableSlot *slot,
*** 3537,3542 ****
--- 3707,3713 ----
/* And send the data */
CopyOneRowTo(cstate, InvalidOid, slot->tts_values, slot->tts_isnull);
+ myState->processed++;
}
/*
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index 9e2bbe8..8340e3d 100644
*** a/src/include/commands/copy.h
--- b/src/include/commands/copy.h
***************
*** 14,25 ****
--- 14,35 ----
#ifndef COPY_H
#define COPY_H
+ #include "nodes/execnodes.h"
#include "nodes/parsenodes.h"
#include "tcop/dest.h"
+ typedef struct CopyStateData *CopyState;
+
extern uint64 DoCopy(const CopyStmt *stmt, const char *queryString);
+ extern CopyState BeginCopyFrom(Relation rel, const char *filename,
+ List *attnamelist, List *options);
+ extern void EndCopyFrom(CopyState cstate);
+ extern bool NextCopyFrom(CopyState cstate,
+ Datum *values, bool *nulls, Oid *tupleOid);
+ extern void CopyFromErrorCallback(void *arg);
+
extern DestReceiver *CreateCopyDestReceiver(void);
#endif /* COPY_H */
On Fri, 7 Jan 2011 10:57:17 +0900
Itagaki Takahiro <itagaki.takahiro@gmail.com> wrote:
I updated the COPY FROM API patch.
- GetCopyExecutorState() is removed because FDWs will use their own context.The patch just rearranges codes for COPY FROM to export those functions.
It also modifies some of COPY TO codes internally for code readability.
- BeginCopyFrom(rel, filename, attnamelist, options)
- EndCopyFrom(cstate)
- NextCopyFrom(cstate, OUT values, OUT nulls, OUT tupleOid)
- CopyFromErrorCallback(arg)
For the purpose of file_fdw, additional ResetCopyFrom() would be
necessary. I'm planning to include such changes in file_fdw patch.
Please find attached partial patch for ResetCopyFrom(). Is there
anything else which should be done at reset?
Some items to be considered:
- BeginCopyFrom() could receive filename as an option instead of a separated
argument. If do so, file_fdw would be more simple, but it's a change only for
file_fdw. COPY commands in the core won't be improved at all.
- NextCopyFrom() returns values/nulls arrays rather than a HeapTuple. I expect
the caller store the result into tupletableslot with ExecStoreVirtualTuple().
It is designed for performance, but if the caller always needs an materialized
HeapTuple, HeapTuple is better for the result type.
IIUC, materizlizing is for tableoid system column. If we could add
tts_tableoid into TupleTableSlot, virtual tuple would be enough. In
this design, caller can receive results with tts_values/tts_isnull
arrays.
Regards,
--
Shigeru Hanada
Attachments:
20110110-ResetCopyFrom.patchapplication/octet-stream; name=20110110-ResetCopyFrom.patchDownload
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index 00139b9..fab811a 100644
*** a/src/backend/commands/copy.c
--- b/src/backend/commands/copy.c
*************** NextCopyFrom(CopyState cstate, Datum *va
*** 2415,2420 ****
--- 2415,2448 ----
}
/*
+ * Reset read pointer to the head of the file.
+ * This can't be used for COPY FROM STDIN.
+ */
+ void
+ ResetCopyFrom(CopyState cstate)
+ {
+ /* stdin can't be rewinded */
+ Assert(cstate->copy_dest == COPY_FILE);
+
+ /* rewind seek pointer to the head */
+ if (fseek(cstate->copy_file, 0, SEEK_SET))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not seek in file \"%s\": %m",
+ cstate->filename)));
+ cstate->fe_eof = false;
+
+ /* reset error context */
+ cstate->cur_lineno = 0;
+ cstate->cur_attname = NULL;
+ cstate->cur_attval = NULL;
+
+ /* just clear the line buffer to avoid re-allocation */
+ resetStringInfo(&cstate->line_buf);
+ }
+
+
+ /*
* Clean up storage and release resources for COPY FROM.
*/
void
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index 8340e3d..424d170 100644
*** a/src/include/commands/copy.h
--- b/src/include/commands/copy.h
*************** extern CopyState BeginCopyFrom(Relation
*** 28,33 ****
--- 28,34 ----
extern void EndCopyFrom(CopyState cstate);
extern bool NextCopyFrom(CopyState cstate,
Datum *values, bool *nulls, Oid *tupleOid);
+ extern void ResetCopyFrom(CopyState cstate);
extern void CopyFromErrorCallback(void *arg);
extern DestReceiver *CreateCopyDestReceiver(void);
Shigeru HANADA <hanada@metrosystems.co.jp> writes:
For the purpose of file_fdw, additional ResetCopyFrom() would be
necessary. I'm planning to include such changes in file_fdw patch.
Please find attached partial patch for ResetCopyFrom(). Is there
anything else which should be done at reset?
Seems like it would be smarter to close and re-open the copy operation.
Adding a reset function is just creating an additional maintenance
burden and point of failure, for what seems likely to be a negligible
performance benefit.
If you think it's not negligible, please show some proof of that before
asking us to support such code.
regards, tom lane
On Mon, 10 Jan 2011 19:26:11 -0500
Tom Lane <tgl@sss.pgh.pa.us> wrote:
Shigeru HANADA <hanada@metrosystems.co.jp> writes:
For the purpose of file_fdw, additional ResetCopyFrom() would be
necessary. I'm planning to include such changes in file_fdw patch.
Please find attached partial patch for ResetCopyFrom(). Is there
anything else which should be done at reset?Seems like it would be smarter to close and re-open the copy operation.
Adding a reset function is just creating an additional maintenance
burden and point of failure, for what seems likely to be a negligible
performance benefit.
Agreed. fileReScan can be implemented with close/re-open with storing
some additional information into FDW private area. I would withdraw
the proposal.
If you think it's not negligible, please show some proof of that before
asking us to support such code.
Anyway, I've measured overhead of re-open with executing query
including inner join between foreign tables copied from pgbench schema.
I used SELECT statement below:
EXPLAIN (ANALYZE) SELECT count(*) FROM csv_accounts a JOIN
csv_branches b ON (b.bid = a.bid);
On the average of (Nested Loop - (Foreign Scan * 2)), overhead of
re-open is round 0.048ms per tuple (average of 3 times measurement).
After the implementation of file_fdw, I'm going to measure again. If
ResetCopyFrom significantly improves performance of ReScan, I'll
propose it as a separate patch.
=========================================================================
The results of EXPLAIN ANALYZE are:
[using ResetCopyFrom]
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------
Aggregate (cost=11717.02..11717.03 rows=1 width=0) (actual time=73357.655..73357.657 rows=1 loops=1)
-> Nested Loop (cost=0.00..11717.01 rows=1 width=0) (actual time=0.209..71424.059 rows=1000000 loops=1)
-> Foreign Scan on public.csv_accounts a (cost=0.00..11717.00 rows=1 width=4) (actual time=0.144..6998.497 rows=1000000 loops=1)
-> Foreign Scan on public.csv_branches b (cost=0.00..0.00 rows=1 width=4) (actual time=0.008..0.037 rows=10 loops=1000000)
Total runtime: 73358.135 ms
(11 rows)
[using EndCopyFrom + BeginCopyFrom]
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------
Aggregate (cost=11717.02..11717.03 rows=1 width=0) (actual time=120724.138..120724.140 rows=1 loops=1)
-> Nested Loop (cost=0.00..11717.01 rows=1 width=0) (actual time=0.321..118583.681 rows=1000000 loops=1)
-> Foreign Scan on public.csv_accounts a (cost=0.00..11717.00 rows=1 width=4) (actual time=0.156..7208.968 rows=1000000 loops=1)
-> Foreign Scan on public.csv_branches b (cost=0.00..0.00 rows=1 width=4) (actual time=0.016..0.046 rows=10 loops=1000000)
Total runtime: 121118.792 ms
(11 rows)
Time: 121122.205 ms
=========================================================================
Regards,
--
Shigeru Hanada
On Fri, 7 Jan 2011 10:57:17 +0900
Itagaki Takahiro <itagaki.takahiro@gmail.com> wrote:
On Mon, Dec 20, 2010 at 20:42, Itagaki Takahiro
<itagaki.takahiro@gmail.com> wrote:I added comments and moved some setup codes for COPY TO to BeginCopyTo()
for maintainability. CopyTo() still contains parts of initialization,
but I've not touched it yet because we don't need the arrangement for now.I updated the COPY FROM API patch.
- GetCopyExecutorState() is removed because FDWs will use their own context.
I rebased file_fdw patch to recent copy_export patch, and have some
comments.
The patch just rearranges codes for COPY FROM to export those functions.
It also modifies some of COPY TO codes internally for code readability.
- BeginCopyFrom(rel, filename, attnamelist, options)
- EndCopyFrom(cstate)
- NextCopyFrom(cstate, OUT values, OUT nulls, OUT tupleOid)
- CopyFromErrorCallback(arg)
This API set seems to be enough to implement file_fdw using COPY
routines.
But EndCopyFrom() seems not to be able to release memory which is
allocated in BeginCopy() and BeginCopyFrom(). I found this behavior
by executing a query which generates nested loop plan (outer 1000000
row * inner 10 row), and at last postgres grows up to 300MB+ from
108MB (VIRT of top command).
Attached patch would avoid this leak by adding per-copy context to
CopyState. This would be overkill, and ResetCopyFrom() might be
reasonable though.
Anyway, I couldn't find performance degrade with this patch (tested on
my Linux box).
==============
# csv_accounts and csv_branches are generated by:
1) pgbench -i -s 10
2) COPY pgbench_accounts to '/path/to/accounts.csv' WITH CSV;
3) COPY pgbench_branches to '/path/to/branches.csv' WITH CSV;
<Original (There is no memory swap during measurement) >
postgres=# explain analyze select * from csv_accounts b, csv_branches t where t.bid = b.bid;
QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------
Nested Loop (cost=0.00..11717.01 rows=1 width=200) (actual time=0.300..100833.057 rows=1000000 loops=1)
Join Filter: (b.bid = t.bid)
-> Foreign Scan on csv_accounts b (cost=0.00..11717.00 rows=1 width=100) (actual time=0.148..4437.595 rows=1000000 loops=1)
-> Foreign Scan on csv_branches t (cost=0.00..0.00 rows=1 width=100) (actual time=0.014..0.039 rows=10 loops=1000000)
Total runtime: 102882.308 ms
(5 rows)
<Patched, Using per-copy context to release memory>
postgres=# explain analyze select * from csv_accounts b, csv_branches t where t.bid = b.bid;
QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------
Nested Loop (cost=0.00..11717.01 rows=1 width=200) (actual time=0.226..100931.864 rows=1000000 loops=1)
Join Filter: (b.bid = t.bid)
-> Foreign Scan on csv_accounts b (cost=0.00..11717.00 rows=1 width=100) (actual time=0.085..4439.777 rows=1000000 loops=1)
-> Foreign Scan on csv_branches t (cost=0.00..0.00 rows=1 width=100) (actual time=0.015..0.039 rows=10 loops=1000000)
Total runtime: 102684.276 ms
(5 rows)
==============
This memory leak would not be problem when using from COPY command
because it handles only one CopyState in a query, and it will be
cleaned up with parent context.
Some items to be considered:
- BeginCopyFrom() could receive filename as an option instead of a separated
argument. If do so, file_fdw would be more simple, but it's a change only for
file_fdw. COPY commands in the core won't be improved at all.
ISTM that current design would be better.
- NextCopyFrom() returns values/nulls arrays rather than a HeapTuple. I expect
the caller store the result into tupletableslot with ExecStoreVirtualTuple().
It is designed for performance, but if the caller always needs an materialized
HeapTuple, HeapTuple is better for the result type.
I tried to add tableoid to TupleTableSlot as tts_tableoid, but it
seems to make codes such as slot_getaddr() and other staff tricky.
How about to implement using materialized tuples to avoid unnecessary
(at least for functionality) changes. I would like to send this
virtual-tuple-optimization to next development cycle because it would
not effect the interface heavily. I'll post materialized-tuple
version of foreign_scan patch soon.
Regards,
--
Shigeru Hanada
Attachments:
20110113-copy_context.patchapplication/octet-stream; name=20110113-copy_context.patchDownload
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index 00139b9..5ee6bb0 100644
*** a/src/backend/commands/copy.c
--- b/src/backend/commands/copy.c
*************** typedef struct CopyStateData
*** 124,129 ****
--- 124,134 ----
const char *cur_attval; /* current att value for error messages */
/*
+ * Working state for COPY TO/FROM
+ */
+ MemoryContext copycontext; /* per-copy execution context */
+
+ /*
* Working state for COPY TO
*/
FmgrInfo *out_functions; /* lookup info for output functions */
*************** static const char BinarySignature[11] =
*** 260,265 ****
--- 265,271 ----
/* non-export function prototypes */
static CopyState BeginCopy(bool is_from, Relation rel, Node *raw_query,
const char *queryString, List *attnamelist, List *options);
+ static void EndCopy(CopyState cstate);
static CopyState BeginCopyTo(Relation rel, Node *query, const char *queryString,
const char *filename, List *attnamelist, List *options);
static void EndCopyTo(CopyState cstate);
*************** BeginCopy(bool is_from,
*** 839,848 ****
--- 845,870 ----
ListCell *option;
TupleDesc tupDesc;
int num_phys_attrs;
+ MemoryContext oldcontext;
/* Allocate workspace and zero all fields */
cstate = (CopyStateData *) palloc0(sizeof(CopyStateData));
+ /*
+ * We allocate everything linked from cstate in new per-COPY context
+ * which is created in caller's context.
+ * This would avoid memory leak repeated COPY use in a query.
+ *
+ * Caller should MemoryContextDelete(cstate->copycontext) first, and then
+ * pfree(cstate) to release memory.
+ */
+ cstate->copycontext = AllocSetContextCreate(CurrentMemoryContext,
+ "COPY context",
+ ALLOCSET_DEFAULT_MINSIZE,
+ ALLOCSET_DEFAULT_INITSIZE,
+ ALLOCSET_DEFAULT_MAXSIZE);
+ oldcontext = MemoryContextSwitchTo(cstate->copycontext);
+
/* Extract options from the statement node tree */
foreach(option, options)
{
*************** BeginCopy(bool is_from,
*** 1242,1250 ****
--- 1264,1284 ----
cstate->copy_dest = COPY_FILE; /* default */
+ /* switch back to caller's context */
+ MemoryContextSwitchTo(oldcontext);
+
return cstate;
}
+ /*
+ * Release resources allocated in BeginCopu() for this copy execution.
+ */
+ static void
+ EndCopy(CopyState cstate)
+ {
+ MemoryContextDelete(cstate->copycontext);
+ pfree(cstate);
+ }
/*
* Setup CopyState to read tuples from a table or a query for COPY TO.
*************** BeginCopyTo(Relation rel,
*** 1259,1266 ****
--- 1293,1302 ----
{
CopyState cstate;
bool pipe = (filename == NULL);
+ MemoryContext oldcontext;
cstate = BeginCopy(false, rel, query, queryString, attnamelist, options);
+ oldcontext = MemoryContextSwitchTo(cstate->copycontext);
if (cstate->rel)
{
*************** BeginCopyTo(Relation rel,
*** 1328,1333 ****
--- 1364,1372 ----
errmsg("\"%s\" is a directory", cstate->filename)));
}
+ /* switch back to caller's context */
+ MemoryContextSwitchTo(oldcontext);
+
return cstate;
}
*************** EndCopyTo(CopyState cstate)
*** 1395,1401 ****
PopActiveSnapshot();
}
! pfree(cstate);
}
/*
--- 1434,1441 ----
PopActiveSnapshot();
}
! /* Release memory resources */
! EndCopy(cstate);
}
/*
*************** BeginCopyFrom(Relation rel,
*** 2010,2017 ****
--- 2050,2059 ----
EState *estate = CreateExecutorState(); /* for ExecPrepareExpr() */
int *defmap;
ExprState **defexprs;
+ MemoryContext oldcontext;
cstate = BeginCopy(true, rel, NULL, NULL, attnamelist, options);
+ oldcontext = MemoryContextSwitchTo(cstate->copycontext);
/* Initialize state variables */
cstate->fe_eof = false;
*************** BeginCopyFrom(Relation rel,
*** 2172,2177 ****
--- 2214,2221 ----
cstate->raw_fields = (char **) palloc(nfields * sizeof(char *));
}
+ MemoryContextSwitchTo(oldcontext);
+
return cstate;
}
*************** EndCopyFrom(CopyState cstate)
*** 2433,2448 ****
}
/* Clean up storage */
! if (!cstate->binary)
! pfree(cstate->raw_fields);
! pfree(cstate->attribute_buf.data);
! pfree(cstate->line_buf.data);
! pfree(cstate->raw_buf);
! pfree(cstate->in_functions);
! pfree(cstate->typioparams);
! pfree(cstate->defmap);
! pfree(cstate->defexprs);
! pfree(cstate);
}
--- 2477,2483 ----
}
/* Clean up storage */
! EndCopy(cstate);
}
On Thu, Jan 13, 2011 at 19:00, Shigeru HANADA <hanada@metrosystems.co.jp> wrote:
But EndCopyFrom() seems not to be able to release memory which is
allocated in BeginCopy() and BeginCopyFrom(). I found this behavior
by executing a query which generates nested loop plan (outer 1000000
row * inner 10 row), and at last postgres grows up to 300MB+ from
108MB (VIRT of top command).Attached patch would avoid this leak by adding per-copy context to
CopyState. This would be overkill, and ResetCopyFrom() might be
reasonable though.
Good catch. I merged your fix into the attached patch.
BTW, why didn't planner choose a materialized plan for the inner loop?
FDW scans are typically slower than heap scans or TupleTableslot scans,
it seems reasonable for me to add a Materialize node at the top of the
inner Foreign Scan, especially when we don't use indexes for the scan
keys or join keys.
--
Itagaki Takahiro
Attachments:
copy_export-20110114.patchapplication/octet-stream; name=copy_export-20110114.patchDownload
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index 841bf22..3a8d8d3 100644
*** a/src/backend/commands/copy.c
--- b/src/backend/commands/copy.c
*************** typedef struct CopyStateData
*** 93,105 ****
FILE *copy_file; /* used if copy_dest == COPY_FILE */
StringInfo fe_msgbuf; /* used for all dests during COPY TO, only for
* dest == COPY_NEW_FE in COPY FROM */
- bool fe_copy; /* true for all FE copy dests */
bool fe_eof; /* true if detected end of copy data */
EolType eol_type; /* EOL type of input */
int client_encoding; /* remote side's character encoding */
bool need_transcoding; /* client encoding diff from server? */
bool encoding_embeds_ascii; /* ASCII can be non-first byte? */
- uint64 processed; /* # of tuples processed */
/* parameters from the COPY command */
Relation rel; /* relation to copy to or from */
--- 93,103 ----
*************** typedef struct CopyStateData
*** 119,131 ****
bool *force_quote_flags; /* per-column CSV FQ flags */
bool *force_notnull_flags; /* per-column CSV FNN flags */
! /* these are just for error messages, see copy_in_error_callback */
const char *cur_relname; /* table name for error messages */
int cur_lineno; /* line number for error messages */
const char *cur_attname; /* current att for error messages */
const char *cur_attval; /* current att value for error messages */
/*
* Working state for COPY TO
*/
FmgrInfo *out_functions; /* lookup info for output functions */
--- 117,134 ----
bool *force_quote_flags; /* per-column CSV FQ flags */
bool *force_notnull_flags; /* per-column CSV FNN flags */
! /* these are just for error messages, see CopyFromErrorCallback */
const char *cur_relname; /* table name for error messages */
int cur_lineno; /* line number for error messages */
const char *cur_attname; /* current att for error messages */
const char *cur_attval; /* current att value for error messages */
/*
+ * Working state for COPY TO/FROM
+ */
+ MemoryContext copycontext; /* per-copy execution context */
+
+ /*
* Working state for COPY TO
*/
FmgrInfo *out_functions; /* lookup info for output functions */
*************** typedef struct CopyStateData
*** 167,181 ****
char *raw_buf;
int raw_buf_index; /* next byte to process */
int raw_buf_len; /* total # of bytes stored */
- } CopyStateData;
! typedef CopyStateData *CopyState;
/* DestReceiver for COPY (SELECT) TO */
typedef struct
{
DestReceiver pub; /* publicly-known function pointers */
CopyState cstate; /* CopyStateData for the command */
} DR_copy;
--- 170,197 ----
char *raw_buf;
int raw_buf_index; /* next byte to process */
int raw_buf_len; /* total # of bytes stored */
! /*
! * The definition of input functions and default expressions are stored
! * in these variables.
! */
! EState *estate;
! AttrNumber num_defaults;
! bool file_has_oids;
! FmgrInfo oid_in_function;
! Oid oid_typioparam;
! FmgrInfo *in_functions;
! Oid *typioparams;
! int *defmap;
! ExprState **defexprs; /* array of default att expressions */
! } CopyStateData;
/* DestReceiver for COPY (SELECT) TO */
typedef struct
{
DestReceiver pub; /* publicly-known function pointers */
CopyState cstate; /* CopyStateData for the command */
+ uint64 processed; /* # of tuples processed */
} DR_copy;
*************** static const char BinarySignature[11] =
*** 248,258 ****
/* non-export function prototypes */
! static void DoCopyTo(CopyState cstate);
! static void CopyTo(CopyState cstate);
static void CopyOneRowTo(CopyState cstate, Oid tupleOid,
Datum *values, bool *nulls);
! static void CopyFrom(CopyState cstate);
static bool CopyReadLine(CopyState cstate);
static bool CopyReadLineText(CopyState cstate);
static int CopyReadAttributesText(CopyState cstate);
--- 264,280 ----
/* non-export function prototypes */
! static CopyState BeginCopy(bool is_from, Relation rel, Node *raw_query,
! const char *queryString, List *attnamelist, List *options);
! static void EndCopy(CopyState cstate);
! static CopyState BeginCopyTo(Relation rel, Node *query, const char *queryString,
! const char *filename, List *attnamelist, List *options);
! static void EndCopyTo(CopyState cstate);
! static uint64 DoCopyTo(CopyState cstate);
! static uint64 CopyTo(CopyState cstate);
static void CopyOneRowTo(CopyState cstate, Oid tupleOid,
Datum *values, bool *nulls);
! static uint64 CopyFrom(CopyState cstate);
static bool CopyReadLine(CopyState cstate);
static bool CopyReadLineText(CopyState cstate);
static int CopyReadAttributesText(CopyState cstate);
*************** DoCopy(const CopyStmt *stmt, const char
*** 724,745 ****
CopyState cstate;
bool is_from = stmt->is_from;
bool pipe = (stmt->filename == NULL);
! List *attnamelist = stmt->attlist;
List *force_quote = NIL;
List *force_notnull = NIL;
bool force_quote_all = false;
bool format_specified = false;
- AclMode required_access = (is_from ? ACL_INSERT : ACL_SELECT);
ListCell *option;
TupleDesc tupDesc;
int num_phys_attrs;
! uint64 processed;
/* Allocate workspace and zero all fields */
cstate = (CopyStateData *) palloc0(sizeof(CopyStateData));
/* Extract options from the statement node tree */
! foreach(option, stmt->options)
{
DefElem *defel = (DefElem *) lfirst(option);
--- 746,870 ----
CopyState cstate;
bool is_from = stmt->is_from;
bool pipe = (stmt->filename == NULL);
! Relation rel;
! uint64 processed;
!
! /* Disallow file COPY except to superusers. */
! if (!pipe && !superuser())
! ereport(ERROR,
! (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
! errmsg("must be superuser to COPY to or from a file"),
! errhint("Anyone can COPY to stdout or from stdin. "
! "psql's \\copy command also works for anyone.")));
!
! if (stmt->relation)
! {
! TupleDesc tupDesc;
! AclMode required_access = (is_from ? ACL_INSERT : ACL_SELECT);
! RangeTblEntry *rte;
! List *attnums;
! ListCell *cur;
!
! Assert(!stmt->query);
!
! /* Open and lock the relation, using the appropriate lock type. */
! rel = heap_openrv(stmt->relation,
! (is_from ? RowExclusiveLock : AccessShareLock));
!
! rte = makeNode(RangeTblEntry);
! rte->rtekind = RTE_RELATION;
! rte->relid = RelationGetRelid(rel);
! rte->requiredPerms = required_access;
!
! tupDesc = RelationGetDescr(rel);
! attnums = CopyGetAttnums(tupDesc, rel, stmt->attlist);
! foreach (cur, attnums)
! {
! int attno = lfirst_int(cur) -
! FirstLowInvalidHeapAttributeNumber;
!
! if (is_from)
! rte->modifiedCols = bms_add_member(rte->modifiedCols, attno);
! else
! rte->selectedCols = bms_add_member(rte->selectedCols, attno);
! }
! ExecCheckRTPerms(list_make1(rte), true);
! }
! else
! {
! Assert(stmt->query);
!
! rel = NULL;
! }
!
! if (is_from)
! {
! /* check read-only transaction */
! if (XactReadOnly && rel->rd_backend != MyBackendId)
! PreventCommandIfReadOnly("COPY FROM");
!
! cstate = BeginCopyFrom(rel, stmt->filename,
! stmt->attlist, stmt->options);
! processed = CopyFrom(cstate); /* copy from file to database */
! EndCopyFrom(cstate);
! }
! else
! {
! cstate = BeginCopyTo(rel, stmt->query, queryString, stmt->filename,
! stmt->attlist, stmt->options);
! processed = DoCopyTo(cstate); /* copy from database to file */
! EndCopyTo(cstate);
! }
!
! /*
! * Close the relation. If reading, we can release the AccessShareLock we got;
! * if writing, we should hold the lock until end of transaction to ensure that
! * updates will be committed before lock is released.
! */
! if (rel != NULL)
! heap_close(rel, (is_from ? NoLock : AccessShareLock));
!
! return processed;
! }
!
! /*
! * Common setup routines used by BeginCopyFrom and BeginCopyTo.
! */
! static CopyState
! BeginCopy(bool is_from,
! Relation rel,
! Node *raw_query,
! const char *queryString,
! List *attnamelist,
! List *options)
! {
! CopyState cstate;
List *force_quote = NIL;
List *force_notnull = NIL;
bool force_quote_all = false;
bool format_specified = false;
ListCell *option;
TupleDesc tupDesc;
int num_phys_attrs;
! MemoryContext oldcontext;
/* Allocate workspace and zero all fields */
cstate = (CopyStateData *) palloc0(sizeof(CopyStateData));
+ /*
+ * We allocate everything used by a cstate in a new memory context.
+ * This would avoid memory leaks repeated uses of COPY in a query.
+ */
+ cstate->copycontext = AllocSetContextCreate(CurrentMemoryContext,
+ "COPY",
+ ALLOCSET_DEFAULT_MINSIZE,
+ ALLOCSET_DEFAULT_INITSIZE,
+ ALLOCSET_DEFAULT_MAXSIZE);
+
+ oldcontext = MemoryContextSwitchTo(cstate->copycontext);
+
/* Extract options from the statement node tree */
! foreach(option, options)
{
DefElem *defel = (DefElem *) lfirst(option);
*************** DoCopy(const CopyStmt *stmt, const char
*** 980,1030 ****
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("CSV quote character must not appear in the NULL specification")));
! /* Disallow file COPY except to superusers. */
! if (!pipe && !superuser())
! ereport(ERROR,
! (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
! errmsg("must be superuser to COPY to or from a file"),
! errhint("Anyone can COPY to stdout or from stdin. "
! "psql's \\copy command also works for anyone.")));
!
! if (stmt->relation)
{
! RangeTblEntry *rte;
! List *attnums;
! ListCell *cur;
!
! Assert(!stmt->query);
! cstate->queryDesc = NULL;
! /* Open and lock the relation, using the appropriate lock type. */
! cstate->rel = heap_openrv(stmt->relation,
! (is_from ? RowExclusiveLock : AccessShareLock));
tupDesc = RelationGetDescr(cstate->rel);
- /* Check relation permissions. */
- rte = makeNode(RangeTblEntry);
- rte->rtekind = RTE_RELATION;
- rte->relid = RelationGetRelid(cstate->rel);
- rte->requiredPerms = required_access;
-
- attnums = CopyGetAttnums(tupDesc, cstate->rel, attnamelist);
- foreach (cur, attnums)
- {
- int attno = lfirst_int(cur) - FirstLowInvalidHeapAttributeNumber;
-
- if (is_from)
- rte->modifiedCols = bms_add_member(rte->modifiedCols, attno);
- else
- rte->selectedCols = bms_add_member(rte->selectedCols, attno);
- }
- ExecCheckRTPerms(list_make1(rte), true);
-
- /* check read-only transaction */
- if (XactReadOnly && is_from && cstate->rel->rd_backend != MyBackendId)
- PreventCommandIfReadOnly("COPY FROM");
-
/* Don't allow COPY w/ OIDs to or from a table without them */
if (cstate->oids && !cstate->rel->rd_rel->relhasoids)
ereport(ERROR,
--- 1105,1118 ----
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("CSV quote character must not appear in the NULL specification")));
! if (rel)
{
! Assert(!raw_query);
! cstate->rel = rel;
tupDesc = RelationGetDescr(cstate->rel);
/* Don't allow COPY w/ OIDs to or from a table without them */
if (cstate->oids && !cstate->rel->rd_rel->relhasoids)
ereport(ERROR,
*************** DoCopy(const CopyStmt *stmt, const char
*** 1058,1064 ****
* function and is executed repeatedly. (See also the same hack in
* DECLARE CURSOR and PREPARE.) XXX FIXME someday.
*/
! rewritten = pg_analyze_and_rewrite((Node *) copyObject(stmt->query),
queryString, NULL, 0);
/* We don't expect more or less than one result query */
--- 1146,1152 ----
* function and is executed repeatedly. (See also the same hack in
* DECLARE CURSOR and PREPARE.) XXX FIXME someday.
*/
! rewritten = pg_analyze_and_rewrite((Node *) copyObject(raw_query),
queryString, NULL, 0);
/* We don't expect more or less than one result query */
*************** DoCopy(const CopyStmt *stmt, const char
*** 1160,1173 ****
}
}
- /* Set up variables to avoid per-attribute overhead. */
- initStringInfo(&cstate->attribute_buf);
- initStringInfo(&cstate->line_buf);
- cstate->line_buf_converted = false;
- cstate->raw_buf = (char *) palloc(RAW_BUF_SIZE + 1);
- cstate->raw_buf_index = cstate->raw_buf_len = 0;
- cstate->processed = 0;
-
/*
* Set up encoding conversion info. Even if the client and server
* encodings are the same, we must apply pg_client_to_server() to validate
--- 1248,1253 ----
*************** DoCopy(const CopyStmt *stmt, const char
*** 1181,1264 ****
cstate->encoding_embeds_ascii = PG_ENCODING_IS_CLIENT_ONLY(cstate->client_encoding);
cstate->copy_dest = COPY_FILE; /* default */
- cstate->filename = stmt->filename;
-
- if (is_from)
- CopyFrom(cstate); /* copy from file to database */
- else
- DoCopyTo(cstate); /* copy from database to file */
! /*
! * Close the relation or query. If reading, we can release the
! * AccessShareLock we got; if writing, we should hold the lock until end
! * of transaction to ensure that updates will be committed before lock is
! * released.
! */
! if (cstate->rel)
! heap_close(cstate->rel, (is_from ? NoLock : AccessShareLock));
! else
! {
! /* Close down the query and free resources. */
! ExecutorEnd(cstate->queryDesc);
! FreeQueryDesc(cstate->queryDesc);
! PopActiveSnapshot();
! }
! /* Clean up storage (probably not really necessary) */
! processed = cstate->processed;
! pfree(cstate->attribute_buf.data);
! pfree(cstate->line_buf.data);
! pfree(cstate->raw_buf);
pfree(cstate);
-
- return processed;
}
-
/*
! * This intermediate routine exists mainly to localize the effects of setjmp
! * so we don't need to plaster a lot of variables with "volatile".
*/
! static void
! DoCopyTo(CopyState cstate)
{
! bool pipe = (cstate->filename == NULL);
! if (cstate->rel)
{
! if (cstate->rel->rd_rel->relkind != RELKIND_RELATION)
! {
! if (cstate->rel->rd_rel->relkind == RELKIND_VIEW)
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from view \"%s\"",
! RelationGetRelationName(cstate->rel)),
! errhint("Try the COPY (SELECT ...) TO variant.")));
! else if (cstate->rel->rd_rel->relkind == RELKIND_FOREIGN_TABLE)
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from foreign table \"%s\"",
! RelationGetRelationName(cstate->rel)),
! errhint("Try the COPY (SELECT ...) TO variant.")));
! else if (cstate->rel->rd_rel->relkind == RELKIND_SEQUENCE)
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from sequence \"%s\"",
! RelationGetRelationName(cstate->rel))));
! else
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from non-table relation \"%s\"",
! RelationGetRelationName(cstate->rel))));
! }
}
if (pipe)
{
! if (whereToSendOutput == DestRemote)
! cstate->fe_copy = true;
! else
cstate->copy_file = stdout;
}
else
--- 1261,1329 ----
cstate->encoding_embeds_ascii = PG_ENCODING_IS_CLIENT_ONLY(cstate->client_encoding);
cstate->copy_dest = COPY_FILE; /* default */
! MemoryContextSwitchTo(oldcontext);
! return cstate;
! }
! /*
! * Release resources allocated in a cstate.
! */
! static void
! EndCopy(CopyState cstate)
! {
! MemoryContextDelete(cstate->copycontext);
pfree(cstate);
}
/*
! * Setup CopyState to read tuples from a table or a query for COPY TO.
*/
! static CopyState
! BeginCopyTo(Relation rel,
! Node *query,
! const char *queryString,
! const char *filename,
! List *attnamelist,
! List *options)
{
! CopyState cstate;
! bool pipe = (filename == NULL);
! MemoryContext oldcontext;
! if (rel != NULL && rel->rd_rel->relkind != RELKIND_RELATION)
{
! if (rel->rd_rel->relkind == RELKIND_VIEW)
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from view \"%s\"",
! RelationGetRelationName(rel)),
! errhint("Try the COPY (SELECT ...) TO variant.")));
! else if (rel->rd_rel->relkind == RELKIND_FOREIGN_TABLE)
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from foreign table \"%s\"",
! RelationGetRelationName(rel)),
! errhint("Try the COPY (SELECT ...) TO variant.")));
! else if (rel->rd_rel->relkind == RELKIND_SEQUENCE)
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from sequence \"%s\"",
! RelationGetRelationName(rel))));
! else
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from non-table relation \"%s\"",
! RelationGetRelationName(rel))));
}
+ cstate = BeginCopy(false, rel, query, queryString, attnamelist, options);
+ oldcontext = MemoryContextSwitchTo(cstate->copycontext);
+
if (pipe)
{
! if (whereToSendOutput != DestRemote)
cstate->copy_file = stdout;
}
else
*************** DoCopyTo(CopyState cstate)
*** 1270,1280 ****
* Prevent write to relative path ... too easy to shoot oneself in the
* foot by overwriting a database file ...
*/
! if (!is_absolute_path(cstate->filename))
ereport(ERROR,
(errcode(ERRCODE_INVALID_NAME),
errmsg("relative path not allowed for COPY to file")));
oumask = umask(S_IWGRP | S_IWOTH);
cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_W);
umask(oumask);
--- 1335,1346 ----
* Prevent write to relative path ... too easy to shoot oneself in the
* foot by overwriting a database file ...
*/
! if (!is_absolute_path(filename))
ereport(ERROR,
(errcode(ERRCODE_INVALID_NAME),
errmsg("relative path not allowed for COPY to file")));
+ cstate->filename = pstrdup(filename);
oumask = umask(S_IWGRP | S_IWOTH);
cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_W);
umask(oumask);
*************** DoCopyTo(CopyState cstate)
*** 1292,1305 ****
errmsg("\"%s\" is a directory", cstate->filename)));
}
PG_TRY();
{
! if (cstate->fe_copy)
SendCopyBegin(cstate);
! CopyTo(cstate);
! if (cstate->fe_copy)
SendCopyEnd(cstate);
}
PG_CATCH();
--- 1358,1387 ----
errmsg("\"%s\" is a directory", cstate->filename)));
}
+ MemoryContextSwitchTo(oldcontext);
+
+ return cstate;
+ }
+
+ /*
+ * This intermediate routine exists mainly to localize the effects of setjmp
+ * so we don't need to plaster a lot of variables with "volatile".
+ */
+ static uint64
+ DoCopyTo(CopyState cstate)
+ {
+ bool pipe = (cstate->filename == NULL);
+ bool fe_copy = (pipe && whereToSendOutput == DestRemote);
+ uint64 processed;
+
PG_TRY();
{
! if (fe_copy)
SendCopyBegin(cstate);
! processed = CopyTo(cstate);
! if (fe_copy)
SendCopyEnd(cstate);
}
PG_CATCH();
*************** DoCopyTo(CopyState cstate)
*** 1314,1339 ****
}
PG_END_TRY();
! if (!pipe)
{
! if (FreeFile(cstate->copy_file))
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not write to file \"%s\": %m",
! cstate->filename)));
}
}
/*
* Copy from relation or query TO file.
*/
! static void
CopyTo(CopyState cstate)
{
TupleDesc tupDesc;
int num_phys_attrs;
Form_pg_attribute *attr;
ListCell *cur;
if (cstate->rel)
tupDesc = RelationGetDescr(cstate->rel);
--- 1396,1442 ----
}
PG_END_TRY();
! return processed;
! }
!
! /*
! * Clean up storage and release resources for COPY TO.
! */
! static void
! EndCopyTo(CopyState cstate)
! {
! if (cstate->filename != NULL && FreeFile(cstate->copy_file))
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not close file \"%s\": %m",
! cstate->filename)));
!
! /*
! * Close the relation or query. We can release the AccessShareLock we got.
! */
! if (cstate->queryDesc != NULL)
{
! /* Close down the query and free resources. */
! ExecutorEnd(cstate->queryDesc);
! FreeQueryDesc(cstate->queryDesc);
! PopActiveSnapshot();
}
+
+ /* Clean up storage */
+ EndCopy(cstate);
}
/*
* Copy from relation or query TO file.
*/
! static uint64
CopyTo(CopyState cstate)
{
TupleDesc tupDesc;
int num_phys_attrs;
Form_pg_attribute *attr;
ListCell *cur;
+ uint64 processed;
if (cstate->rel)
tupDesc = RelationGetDescr(cstate->rel);
*************** CopyTo(CopyState cstate)
*** 1439,1444 ****
--- 1542,1548 ----
scandesc = heap_beginscan(cstate->rel, GetActiveSnapshot(), 0, NULL);
+ processed = 0;
while ((tuple = heap_getnext(scandesc, ForwardScanDirection)) != NULL)
{
CHECK_FOR_INTERRUPTS();
*************** CopyTo(CopyState cstate)
*** 1448,1461 ****
--- 1552,1570 ----
/* Format and send the data */
CopyOneRowTo(cstate, HeapTupleGetOid(tuple), values, nulls);
+ processed++;
}
heap_endscan(scandesc);
+
+ pfree(values);
+ pfree(nulls);
}
else
{
/* run the plan --- the dest receiver will send tuples */
ExecutorRun(cstate->queryDesc, ForwardScanDirection, 0L);
+ processed = ((DR_copy *) cstate->queryDesc->dest)->processed;
}
if (cstate->binary)
*************** CopyTo(CopyState cstate)
*** 1467,1472 ****
--- 1576,1583 ----
}
MemoryContextDelete(cstate->rowcontext);
+
+ return processed;
}
/*
*************** CopyOneRowTo(CopyState cstate, Oid tuple
*** 1558,1573 ****
CopySendEndOfRow(cstate);
MemoryContextSwitchTo(oldcontext);
-
- cstate->processed++;
}
/*
* error context callback for COPY FROM
*/
! static void
! copy_in_error_callback(void *arg)
{
CopyState cstate = (CopyState) arg;
--- 1669,1684 ----
CopySendEndOfRow(cstate);
MemoryContextSwitchTo(oldcontext);
}
/*
* error context callback for COPY FROM
+ *
+ * The argument for the error context must be CopyState.
*/
! void
! CopyFromErrorCallback(void *arg)
{
CopyState cstate = (CopyState) arg;
*************** limit_printout_length(const char *str)
*** 1669,1709 ****
/*
* Copy FROM file to relation.
*/
! static void
CopyFrom(CopyState cstate)
{
- bool pipe = (cstate->filename == NULL);
HeapTuple tuple;
TupleDesc tupDesc;
- Form_pg_attribute *attr;
- AttrNumber num_phys_attrs,
- attr_count,
- num_defaults;
- FmgrInfo *in_functions;
- FmgrInfo oid_in_function;
- Oid *typioparams;
- Oid oid_typioparam;
- int attnum;
- int i;
- Oid in_func_oid;
Datum *values;
bool *nulls;
- int nfields;
- char **field_strings;
bool done = false;
- bool isnull;
ResultRelInfo *resultRelInfo;
! EState *estate = CreateExecutorState(); /* for ExecConstraints() */
TupleTableSlot *slot;
- bool file_has_oids;
- int *defmap;
- ExprState **defexprs; /* array of default att expressions */
- ExprContext *econtext; /* used for ExecEvalExpr for default atts */
MemoryContext oldcontext = CurrentMemoryContext;
ErrorContextCallback errcontext;
CommandId mycid = GetCurrentCommandId(true);
int hi_options = 0; /* start with default heap_insert options */
BulkInsertState bistate;
Assert(cstate->rel);
--- 1780,1802 ----
/*
* Copy FROM file to relation.
*/
! static uint64
CopyFrom(CopyState cstate)
{
HeapTuple tuple;
TupleDesc tupDesc;
Datum *values;
bool *nulls;
bool done = false;
ResultRelInfo *resultRelInfo;
! EState *estate = cstate->estate; /* for ExecConstraints() */
TupleTableSlot *slot;
MemoryContext oldcontext = CurrentMemoryContext;
ErrorContextCallback errcontext;
CommandId mycid = GetCurrentCommandId(true);
int hi_options = 0; /* start with default heap_insert options */
BulkInsertState bistate;
+ uint64 processed = 0;
Assert(cstate->rel);
*************** CopyFrom(CopyState cstate)
*** 1731,1736 ****
--- 1824,1831 ----
RelationGetRelationName(cstate->rel))));
}
+ tupDesc = RelationGetDescr(cstate->rel);
+
/*----------
* Check to see if we can avoid writing WAL
*
*************** CopyFrom(CopyState cstate)
*** 1766,1803 ****
hi_options |= HEAP_INSERT_SKIP_WAL;
}
- if (pipe)
- {
- if (whereToSendOutput == DestRemote)
- ReceiveCopyBegin(cstate);
- else
- cstate->copy_file = stdin;
- }
- else
- {
- struct stat st;
-
- cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_R);
-
- if (cstate->copy_file == NULL)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not open file \"%s\" for reading: %m",
- cstate->filename)));
-
- fstat(fileno(cstate->copy_file), &st);
- if (S_ISDIR(st.st_mode))
- ereport(ERROR,
- (errcode(ERRCODE_WRONG_OBJECT_TYPE),
- errmsg("\"%s\" is a directory", cstate->filename)));
- }
-
- tupDesc = RelationGetDescr(cstate->rel);
- attr = tupDesc->attrs;
- num_phys_attrs = tupDesc->natts;
- attr_count = list_length(cstate->attnumlist);
- num_defaults = 0;
-
/*
* We need a ResultRelInfo so we can use the regular executor's
* index-entry-making machinery. (There used to be a huge amount of code
--- 1861,1866 ----
*************** CopyFrom(CopyState cstate)
*** 1826,1832 ****
slot = ExecInitExtraTupleSlot(estate);
ExecSetSlotDescriptor(slot, tupDesc);
! econtext = GetPerTupleExprContext(estate);
/*
* Pick up the required catalog information for each attribute in the
--- 1889,2070 ----
slot = ExecInitExtraTupleSlot(estate);
ExecSetSlotDescriptor(slot, tupDesc);
! /* Prepare to catch AFTER triggers. */
! AfterTriggerBeginQuery();
!
! /*
! * Check BEFORE STATEMENT insertion triggers. It's debateable whether we
! * should do this for COPY, since it's not really an "INSERT" statement as
! * such. However, executing these triggers maintains consistency with the
! * EACH ROW triggers that we already fire on COPY.
! */
! ExecBSInsertTriggers(estate, resultRelInfo);
!
! values = (Datum *) palloc(tupDesc->natts * sizeof(Datum));
! nulls = (bool *) palloc(tupDesc->natts * sizeof(bool));
!
! bistate = GetBulkInsertState();
!
! /* Set up callback to identify error line number */
! errcontext.callback = CopyFromErrorCallback;
! errcontext.arg = (void *) cstate;
! errcontext.previous = error_context_stack;
! error_context_stack = &errcontext;
!
! while (!done)
! {
! bool skip_tuple;
! Oid loaded_oid = InvalidOid;
!
! CHECK_FOR_INTERRUPTS();
!
! /* Reset the per-tuple exprcontext */
! ResetPerTupleExprContext(estate);
!
! /* Switch into its memory context */
! MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
!
! done = !NextCopyFrom(cstate, values, nulls, &loaded_oid);
! if (done)
! break;
!
! /* And now we can form the input tuple. */
! tuple = heap_form_tuple(tupDesc, values, nulls);
!
! if (loaded_oid != InvalidOid)
! HeapTupleSetOid(tuple, loaded_oid);
!
! /* Triggers and stuff need to be invoked in query context. */
! MemoryContextSwitchTo(oldcontext);
!
! skip_tuple = false;
!
! /* BEFORE ROW INSERT Triggers */
! if (resultRelInfo->ri_TrigDesc &&
! resultRelInfo->ri_TrigDesc->trig_insert_before_row)
! {
! HeapTuple newtuple;
!
! newtuple = ExecBRInsertTriggers(estate, resultRelInfo, tuple);
!
! if (newtuple == NULL) /* "do nothing" */
! skip_tuple = true;
! else if (newtuple != tuple) /* modified by Trigger(s) */
! {
! heap_freetuple(tuple);
! tuple = newtuple;
! }
! }
!
! if (!skip_tuple)
! {
! List *recheckIndexes = NIL;
!
! /* Place tuple in tuple slot */
! ExecStoreTuple(tuple, slot, InvalidBuffer, false);
!
! /* Check the constraints of the tuple */
! if (cstate->rel->rd_att->constr)
! ExecConstraints(resultRelInfo, slot, estate);
!
! /* OK, store the tuple and create index entries for it */
! heap_insert(cstate->rel, tuple, mycid, hi_options, bistate);
!
! if (resultRelInfo->ri_NumIndices > 0)
! recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self),
! estate);
!
! /* AFTER ROW INSERT Triggers */
! ExecARInsertTriggers(estate, resultRelInfo, tuple,
! recheckIndexes);
!
! list_free(recheckIndexes);
!
! /*
! * We count only tuples not suppressed by a BEFORE INSERT trigger;
! * this is the same definition used by execMain.c for counting
! * tuples inserted by an INSERT command.
! */
! processed++;
! }
! }
!
! /* Done, clean up */
! error_context_stack = errcontext.previous;
!
! FreeBulkInsertState(bistate);
!
! MemoryContextSwitchTo(oldcontext);
!
! /* Execute AFTER STATEMENT insertion triggers */
! ExecASInsertTriggers(estate, resultRelInfo);
!
! /* Handle queued AFTER triggers */
! AfterTriggerEndQuery(estate);
!
! pfree(values);
! pfree(nulls);
!
! ExecResetTupleTable(estate->es_tupleTable, false);
!
! ExecCloseIndices(resultRelInfo);
!
! /*
! * If we skipped writing WAL, then we need to sync the heap (but not
! * indexes since those use WAL anyway)
! */
! if (hi_options & HEAP_INSERT_SKIP_WAL)
! heap_sync(cstate->rel);
!
! return processed;
! }
!
! /*
! * Setup CopyState to read tuples from a file for COPY FROM.
! */
! CopyState
! BeginCopyFrom(Relation rel,
! const char *filename,
! List *attnamelist,
! List *options)
! {
! CopyState cstate;
! bool pipe = (filename == NULL);
! TupleDesc tupDesc;
! Form_pg_attribute *attr;
! AttrNumber num_phys_attrs,
! num_defaults;
! FmgrInfo *in_functions;
! Oid *typioparams;
! int attnum;
! Oid in_func_oid;
! EState *estate = CreateExecutorState(); /* for ExecPrepareExpr() */
! int *defmap;
! ExprState **defexprs;
! MemoryContext oldcontext;
!
! cstate = BeginCopy(true, rel, NULL, NULL, attnamelist, options);
! oldcontext = MemoryContextSwitchTo(cstate->copycontext);
!
! /* Initialize state variables */
! cstate->fe_eof = false;
! cstate->eol_type = EOL_UNKNOWN;
! cstate->cur_relname = RelationGetRelationName(cstate->rel);
! cstate->cur_lineno = 0;
! cstate->cur_attname = NULL;
! cstate->cur_attval = NULL;
!
! /* Set up variables to avoid per-attribute overhead. */
! initStringInfo(&cstate->attribute_buf);
! initStringInfo(&cstate->line_buf);
! cstate->line_buf_converted = false;
! cstate->raw_buf = (char *) palloc(RAW_BUF_SIZE + 1);
! cstate->raw_buf_index = cstate->raw_buf_len = 0;
!
! tupDesc = RelationGetDescr(cstate->rel);
! attr = tupDesc->attrs;
! num_phys_attrs = tupDesc->natts;
! num_defaults = 0;
/*
* Pick up the required catalog information for each attribute in the
*************** CopyFrom(CopyState cstate)
*** 1871,1889 ****
}
}
! /* Prepare to catch AFTER triggers. */
! AfterTriggerBeginQuery();
! /*
! * Check BEFORE STATEMENT insertion triggers. It's debateable whether we
! * should do this for COPY, since it's not really an "INSERT" statement as
! * such. However, executing these triggers maintains consistency with the
! * EACH ROW triggers that we already fire on COPY.
! */
! ExecBSInsertTriggers(estate, resultRelInfo);
if (!cstate->binary)
! file_has_oids = cstate->oids; /* must rely on user to tell us... */
else
{
/* Read and verify binary header */
--- 2109,2154 ----
}
}
! /* We keep those variables in cstate. */
! cstate->estate = estate;
! cstate->in_functions = in_functions;
! cstate->typioparams = typioparams;
! cstate->defmap = defmap;
! cstate->defexprs = defexprs;
! cstate->num_defaults = num_defaults;
! if (pipe)
! {
! if (whereToSendOutput == DestRemote)
! ReceiveCopyBegin(cstate);
! else
! cstate->copy_file = stdin;
! }
! else
! {
! struct stat st;
!
! cstate->filename = pstrdup(filename);
! cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_R);
!
! if (cstate->copy_file == NULL)
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not open file \"%s\" for reading: %m",
! cstate->filename)));
!
! fstat(fileno(cstate->copy_file), &st);
! if (S_ISDIR(st.st_mode))
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("\"%s\" is a directory", cstate->filename)));
! }
if (!cstate->binary)
! {
! /* must rely on user to tell us... */
! cstate->file_has_oids = cstate->oids;
! }
else
{
/* Read and verify binary header */
*************** CopyFrom(CopyState cstate)
*** 1901,1907 ****
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid COPY file header (missing flags)")));
! file_has_oids = (tmp & (1 << 16)) != 0;
tmp &= ~(1 << 16);
if ((tmp >> 16) != 0)
ereport(ERROR,
--- 2166,2172 ----
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid COPY file header (missing flags)")));
! cstate->file_has_oids = (tmp & (1 << 16)) != 0;
tmp &= ~(1 << 16);
if ((tmp >> 16) != 0)
ereport(ERROR,
*************** CopyFrom(CopyState cstate)
*** 1923,1984 ****
}
}
! if (file_has_oids && cstate->binary)
{
getTypeBinaryInputInfo(OIDOID,
! &in_func_oid, &oid_typioparam);
! fmgr_info(in_func_oid, &oid_in_function);
}
- values = (Datum *) palloc(num_phys_attrs * sizeof(Datum));
- nulls = (bool *) palloc(num_phys_attrs * sizeof(bool));
-
/* create workspace for CopyReadAttributes results */
! nfields = file_has_oids ? (attr_count + 1) : attr_count;
! if (! cstate->binary)
{
cstate->max_fields = nfields;
cstate->raw_fields = (char **) palloc(nfields * sizeof(char *));
}
! /* Initialize state variables */
! cstate->fe_eof = false;
! cstate->eol_type = EOL_UNKNOWN;
! cstate->cur_relname = RelationGetRelationName(cstate->rel);
! cstate->cur_lineno = 0;
! cstate->cur_attname = NULL;
! cstate->cur_attval = NULL;
! bistate = GetBulkInsertState();
! /* Set up callback to identify error line number */
! errcontext.callback = copy_in_error_callback;
! errcontext.arg = (void *) cstate;
! errcontext.previous = error_context_stack;
! error_context_stack = &errcontext;
/* on input just throw the header line away */
! if (cstate->header_line)
{
cstate->cur_lineno++;
! done = CopyReadLine(cstate);
}
! while (!done)
! {
! bool skip_tuple;
! Oid loaded_oid = InvalidOid;
!
! CHECK_FOR_INTERRUPTS();
cstate->cur_lineno++;
- /* Reset the per-tuple exprcontext */
- ResetPerTupleExprContext(estate);
-
- /* Switch into its memory context */
- MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
-
/* Initialize all values for row to NULL */
MemSet(values, 0, num_phys_attrs * sizeof(Datum));
MemSet(nulls, true, num_phys_attrs * sizeof(bool));
--- 2188,2258 ----
}
}
! if (cstate->file_has_oids && cstate->binary)
{
getTypeBinaryInputInfo(OIDOID,
! &in_func_oid, &cstate->oid_typioparam);
! fmgr_info(in_func_oid, &cstate->oid_in_function);
}
/* create workspace for CopyReadAttributes results */
! if (!cstate->binary)
{
+ AttrNumber attr_count = list_length(cstate->attnumlist);
+ int nfields = cstate->file_has_oids ? (attr_count + 1) : attr_count;
+
cstate->max_fields = nfields;
cstate->raw_fields = (char **) palloc(nfields * sizeof(char *));
}
! MemoryContextSwitchTo(oldcontext);
! return cstate;
! }
! /*
! * Read next tuple from file for COPY FROM. Return false if no more tuples.
! *
! * valus and nulls arrays must be the same length as columns of the
! * relation passed to BeginCopyFrom. Oid of the tuple is returned with
! * tupleOid separately.
! */
! bool
! NextCopyFrom(CopyState cstate, Datum *values, bool *nulls, Oid *tupleOid)
! {
! TupleDesc tupDesc;
! Form_pg_attribute *attr;
! AttrNumber num_phys_attrs,
! attr_count,
! num_defaults = cstate->num_defaults;
! FmgrInfo *in_functions = cstate->in_functions;
! Oid *typioparams = cstate->typioparams;
! int i;
! int nfields;
! char **field_strings;
! bool isnull;
! bool file_has_oids = cstate->file_has_oids;
! int *defmap = cstate->defmap;
! ExprState **defexprs = cstate->defexprs;
! ExprContext *econtext; /* used for ExecEvalExpr for default atts */
/* on input just throw the header line away */
! if (cstate->cur_lineno == 0 && cstate->header_line)
{
cstate->cur_lineno++;
! if (CopyReadLine(cstate))
! return false; /* done */
}
! tupDesc = RelationGetDescr(cstate->rel);
! attr = tupDesc->attrs;
! num_phys_attrs = tupDesc->natts;
! attr_count = list_length(cstate->attnumlist);
! nfields = file_has_oids ? (attr_count + 1) : attr_count;
+ /* XXX: Indentation is not adjusted to keep the patch small. */
cstate->cur_lineno++;
/* Initialize all values for row to NULL */
MemSet(values, 0, num_phys_attrs * sizeof(Datum));
MemSet(nulls, true, num_phys_attrs * sizeof(bool));
*************** CopyFrom(CopyState cstate)
*** 1989,1994 ****
--- 2263,2269 ----
int fldct;
int fieldno;
char *string;
+ bool done;
/* Actually read the line into memory here */
done = CopyReadLine(cstate);
*************** CopyFrom(CopyState cstate)
*** 1999,2005 ****
* EOF, ie, process the line and then exit loop on next iteration.
*/
if (done && cstate->line_buf.len == 0)
! break;
/* Parse the line into de-escaped field values */
if (cstate->csv_mode)
--- 2274,2280 ----
* EOF, ie, process the line and then exit loop on next iteration.
*/
if (done && cstate->line_buf.len == 0)
! return false;
/* Parse the line into de-escaped field values */
if (cstate->csv_mode)
*************** CopyFrom(CopyState cstate)
*** 2029,2041 ****
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("null OID in COPY data")));
! else
{
cstate->cur_attname = "oid";
cstate->cur_attval = string;
! loaded_oid = DatumGetObjectId(DirectFunctionCall1(oidin,
! CStringGetDatum(string)));
! if (loaded_oid == InvalidOid)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid OID in COPY data")));
--- 2304,2316 ----
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("null OID in COPY data")));
! else if (cstate->oids && tupleOid != NULL)
{
cstate->cur_attname = "oid";
cstate->cur_attval = string;
! *tupleOid = DatumGetObjectId(DirectFunctionCall1(oidin,
! CStringGetDatum(string)));
! if (*tupleOid == InvalidOid)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid OID in COPY data")));
*************** CopyFrom(CopyState cstate)
*** 2087,2094 ****
if (!CopyGetInt16(cstate, &fld_count))
{
/* EOF detected (end of file, or protocol-level EOF) */
! done = true;
! break;
}
if (fld_count == -1)
--- 2362,2368 ----
if (!CopyGetInt16(cstate, &fld_count))
{
/* EOF detected (end of file, or protocol-level EOF) */
! return false;
}
if (fld_count == -1)
*************** CopyFrom(CopyState cstate)
*** 2112,2119 ****
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("received copy data after EOF marker")));
! done = true;
! break;
}
if (fld_count != attr_count)
--- 2386,2392 ----
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("received copy data after EOF marker")));
! return false;
}
if (fld_count != attr_count)
*************** CopyFrom(CopyState cstate)
*** 2124,2135 ****
if (file_has_oids)
{
cstate->cur_attname = "oid";
loaded_oid =
DatumGetObjectId(CopyReadBinaryAttribute(cstate,
0,
! &oid_in_function,
! oid_typioparam,
-1,
&isnull));
if (isnull || loaded_oid == InvalidOid)
--- 2397,2410 ----
if (file_has_oids)
{
+ Oid loaded_oid;
+
cstate->cur_attname = "oid";
loaded_oid =
DatumGetObjectId(CopyReadBinaryAttribute(cstate,
0,
! &cstate->oid_in_function,
! cstate->oid_typioparam,
-1,
&isnull));
if (isnull || loaded_oid == InvalidOid)
*************** CopyFrom(CopyState cstate)
*** 2137,2142 ****
--- 2412,2419 ----
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid OID in COPY data")));
cstate->cur_attname = NULL;
+ if (cstate->oids && tupleOid != NULL)
+ *tupleOid = loaded_oid;
}
i = 0;
*************** CopyFrom(CopyState cstate)
*** 2162,2278 ****
* provided by the input data. Anything not processed here or above
* will remain NULL.
*/
for (i = 0; i < num_defaults; i++)
{
values[defmap[i]] = ExecEvalExpr(defexprs[i], econtext,
&nulls[defmap[i]], NULL);
}
! /* And now we can form the input tuple. */
! tuple = heap_form_tuple(tupDesc, values, nulls);
!
! if (cstate->oids && file_has_oids)
! HeapTupleSetOid(tuple, loaded_oid);
!
! /* Triggers and stuff need to be invoked in query context. */
! MemoryContextSwitchTo(oldcontext);
!
! skip_tuple = false;
!
! /* BEFORE ROW INSERT Triggers */
! if (resultRelInfo->ri_TrigDesc &&
! resultRelInfo->ri_TrigDesc->trig_insert_before_row)
! {
! HeapTuple newtuple;
!
! newtuple = ExecBRInsertTriggers(estate, resultRelInfo, tuple);
!
! if (newtuple == NULL) /* "do nothing" */
! skip_tuple = true;
! else if (newtuple != tuple) /* modified by Trigger(s) */
! {
! heap_freetuple(tuple);
! tuple = newtuple;
! }
! }
!
! if (!skip_tuple)
! {
! List *recheckIndexes = NIL;
!
! /* Place tuple in tuple slot */
! ExecStoreTuple(tuple, slot, InvalidBuffer, false);
!
! /* Check the constraints of the tuple */
! if (cstate->rel->rd_att->constr)
! ExecConstraints(resultRelInfo, slot, estate);
!
! /* OK, store the tuple and create index entries for it */
! heap_insert(cstate->rel, tuple, mycid, hi_options, bistate);
!
! if (resultRelInfo->ri_NumIndices > 0)
! recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self),
! estate);
!
! /* AFTER ROW INSERT Triggers */
! ExecARInsertTriggers(estate, resultRelInfo, tuple,
! recheckIndexes);
!
! list_free(recheckIndexes);
!
! /*
! * We count only tuples not suppressed by a BEFORE INSERT trigger;
! * this is the same definition used by execMain.c for counting
! * tuples inserted by an INSERT command.
! */
! cstate->processed++;
! }
! }
!
! /* Done, clean up */
! error_context_stack = errcontext.previous;
!
! FreeBulkInsertState(bistate);
!
! MemoryContextSwitchTo(oldcontext);
!
! /* Execute AFTER STATEMENT insertion triggers */
! ExecASInsertTriggers(estate, resultRelInfo);
!
! /* Handle queued AFTER triggers */
! AfterTriggerEndQuery(estate);
!
! pfree(values);
! pfree(nulls);
! if (! cstate->binary)
! pfree(cstate->raw_fields);
!
! pfree(in_functions);
! pfree(typioparams);
! pfree(defmap);
! pfree(defexprs);
!
! ExecResetTupleTable(estate->es_tupleTable, false);
!
! ExecCloseIndices(resultRelInfo);
! FreeExecutorState(estate);
! if (!pipe)
! {
! if (FreeFile(cstate->copy_file))
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not read from file \"%s\": %m",
! cstate->filename)));
! }
! /*
! * If we skipped writing WAL, then we need to sync the heap (but not
! * indexes since those use WAL anyway)
! */
! if (hi_options & HEAP_INSERT_SKIP_WAL)
! heap_sync(cstate->rel);
}
--- 2439,2471 ----
* provided by the input data. Anything not processed here or above
* will remain NULL.
*/
+ econtext = GetPerTupleExprContext(cstate->estate);
for (i = 0; i < num_defaults; i++)
{
values[defmap[i]] = ExecEvalExpr(defexprs[i], econtext,
&nulls[defmap[i]], NULL);
}
+ /* XXX: End of only-indentation changes. */
! return true;
! }
! /*
! * Clean up storage and release resources for COPY FROM.
! */
! void
! EndCopyFrom(CopyState cstate)
! {
! FreeExecutorState(cstate->estate);
! if (cstate->filename != NULL && FreeFile(cstate->copy_file))
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not close file \"%s\": %m",
! cstate->filename)));
! /* Clean up storage */
! EndCopy(cstate);
}
*************** copy_dest_receive(TupleTableSlot *slot,
*** 3537,3542 ****
--- 3730,3736 ----
/* And send the data */
CopyOneRowTo(cstate, InvalidOid, slot->tts_values, slot->tts_isnull);
+ myState->processed++;
}
/*
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index 9e2bbe8..8340e3d 100644
*** a/src/include/commands/copy.h
--- b/src/include/commands/copy.h
***************
*** 14,25 ****
--- 14,35 ----
#ifndef COPY_H
#define COPY_H
+ #include "nodes/execnodes.h"
#include "nodes/parsenodes.h"
#include "tcop/dest.h"
+ typedef struct CopyStateData *CopyState;
+
extern uint64 DoCopy(const CopyStmt *stmt, const char *queryString);
+ extern CopyState BeginCopyFrom(Relation rel, const char *filename,
+ List *attnamelist, List *options);
+ extern void EndCopyFrom(CopyState cstate);
+ extern bool NextCopyFrom(CopyState cstate,
+ Datum *values, bool *nulls, Oid *tupleOid);
+ extern void CopyFromErrorCallback(void *arg);
+
extern DestReceiver *CreateCopyDestReceiver(void);
#endif /* COPY_H */
On Fri, 14 Jan 2011 13:03:27 +0900
Itagaki Takahiro <itagaki.takahiro@gmail.com> wrote:
Good catch. I merged your fix into the attached patch.
Thanks, I'll rebase my patches.
BTW, why didn't planner choose a materialized plan for the inner loop?
FDW scans are typically slower than heap scans or TupleTableslot scans,
it seems reasonable for me to add a Materialize node at the top of the
inner Foreign Scan, especially when we don't use indexes for the scan
keys or join keys.
Maybe because foreign tables lack statistics, and file_fdw's estimate
isn't smart enough.
After copying statisticsof pgbench_xxx tables into csv_xxx tables,
planner generates same plans as for local tables, but costs of
ForeignScan nodes are little lower than them of SeqScan nodes.
==============================
postgres=# explain analyze select * from csv_accounts a, csv_branches b where a.bid = b.bid;
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------
Hash Join (cost=0.33..45467.32 rows=1000000 width=197) (actual time=0.234..8044.077 rows=1000000 loops=1)
Hash Cond: (a.bid = b.bid)
-> Foreign Scan on csv_accounts a (cost=0.00..31717.00 rows=1000000 width=97) (actual time=0.107..4147.074 rows=1000000 loops=1)
-> Hash (cost=0.20..0.20 rows=10 width=100) (actual time=0.085..0.085 rows=10 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 1kB
-> Foreign Scan on csv_branches b (cost=0.00..0.20 rows=10 width=100) (actual time=0.027..0.056 rows=10 loops=1)
Total runtime: 9690.686 ms
(7 rows)
postgres=# explain analyze select * from pgbench_accounts a, pgbench_branches b where a.bid = b.bid;
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------
Hash Join (cost=1.23..40145.22 rows=1000000 width=197) (actual time=0.146..5693.883 rows=1000000 loops=1)
Hash Cond: (a.bid = b.bid)
-> Seq Scan on pgbench_accounts a (cost=0.00..26394.00 rows=1000000 width=97) (actual time=0.073..1884.018 rows=1000000 loops=1)
-> Hash (cost=1.10..1.10 rows=10 width=100) (actual time=0.048..0.048 rows=10 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 1kB
-> Seq Scan on pgbench_branches b (cost=0.00..1.10 rows=10 width=100) (actual time=0.003..0.021 rows=10 loops=1)
Total runtime: 7333.713 ms
(7 rows)
==============================
Forced Nested Loop uses Materialize node as expected.
==============================
postgres=# set enable_hashjoin = false;
SET
postgres=# explain select * from csv_accounts a, csv_branches b where a.bid = b.bid;
QUERY PLAN
-----------------------------------------------------------------------------------
Nested Loop (cost=0.00..181717.23 rows=1000000 width=197)
Join Filter: (a.bid = b.bid)
-> Foreign Scan on csv_accounts a (cost=0.00..31717.00 rows=1000000 width=97)
-> Materialize (cost=0.00..0.25 rows=10 width=100)
-> Foreign Scan on csv_branches b (cost=0.00..0.20 rows=10 width=100)
(5 rows)
postgres=# explain select * from pgbench_accounts a, pgbench_branches b where a.bid = b.bid;
QUERY PLAN
-----------------------------------------------------------------------------------
Nested Loop (cost=0.00..176395.12 rows=1000000 width=197)
Join Filter: (a.bid = b.bid)
-> Seq Scan on pgbench_accounts a (cost=0.00..26394.00 rows=1000000 width=97)
-> Materialize (cost=0.00..1.15 rows=10 width=100)
-> Seq Scan on pgbench_branches b (cost=0.00..1.10 rows=10 width=100)
(5 rows)
==============================
ISTM that new interface which is called from ANALYZE would help to
update statistics of foreign talbes. If we could leave sampling
argorythm to FDWs, acquire_sample_rows() might fit for that purpose.
If a FDW doesn't provide analyze handler, postgres might be able to
execute "SELECT * FROM foreign_table LIMIT sample_num" internally to
get sample rows.
Regards,
--
Shigeru Hanada
On Fri, Jan 14, 2011 at 14:20, Shigeru HANADA <hanada@metrosystems.co.jp> wrote:
After copying statisticsof pgbench_xxx tables into csv_xxx tables,
planner generates same plans as for local tables, but costs of
ForeignScan nodes are little lower than them of SeqScan nodes.
Forced Nested Loop uses Materialize node as expected.
Interesting. It means we need per-column statistics for foreign
tables in addition to cost values.
ISTM that new interface which is called from ANALYZE would help to
update statistics of foreign talbes. If we could leave sampling
argorythm to FDWs, acquire_sample_rows() might fit for that purpose.
We will discuss how to collect statistics from foreign tables
in the next development cycle. I think we have two choice here:
#1. Retrieve sample rows from remote foreign tables and
store stats in the local pg_statistic.
#2. Use remote statistics for each foreign table directly.
acquire_sample_rows() would be a method for #1, Another approach
for #2 is to use remote statistics directly. We provide hooks to
generate virtual statistics with get_relation_stats_hook() and
families. We could treat statistics for foreign tables in a similar
way as the hook.
file_fdw likes #1 because there are no external storage to store
statistics for CSV files, but pgsql_fdw might prefer #2 because
the remote server already has stats for the underlying table.
--
Itagaki Takahiro
On Fri, 14 Jan 2011 14:20:20 +0900
Shigeru HANADA <hanada@metrosystems.co.jp> wrote:
On Fri, 14 Jan 2011 13:03:27 +0900
Itagaki Takahiro <itagaki.takahiro@gmail.com> wrote:Good catch. I merged your fix into the attached patch.
Thanks, I'll rebase my patches.
I've rebased WIP patches for file_fdw onto Itagaki-san's recent
copy_export patch.
Interface of NextCopyFrom() is fixed to return HeapTuple, to support
tableoid without any change to TupleTableSlot.
Please apply patches in this order:
1) patches for FDW API (NOT attached to this message)
These patches are attached and described in this message.
http://archives.postgresql.org/pgsql-hackers/2011-01/msg01096.php
2) copy_export-20110114.patch (NOT attached to this message)
This patch is attached and described in this message.
http://archives.postgresql.org/pgsql-hackers/2011-01/msg01066.php
3) 20110114-copy_export_HeapTupe.patch
This patch fixes interface of NextCopyFrom() to return results as
HeapTuple.
4) 20110114-foreign_scan.patch
This patch adds HANDLER option of FOREIGN DATA WRAPPER, and
ForeignScan executor node. Note that no wrapper is available in this
patch.
5) 20110114-file_fdw.patch
This patch adds contrib/file_fdw, foreign-data wrapper for server-side
files. Supported file formats are similar to COPY command, and
COPY options except "force_not_null" are accepted as generic options
of foreign table.
Regards,
--
Shigeru Hanada
Attachments:
On Sat, Jan 15, 2011 at 08:35, Shigeru HANADA <hanada@metrosystems.co.jp> wrote:
Interface of NextCopyFrom() is fixed to return HeapTuple, to support
tableoid without any change to TupleTableSlot.3) 20110114-copy_export_HeapTupe.patch
This patch fixes interface of NextCopyFrom() to return results as
HeapTuple.
I think file_fdw can return tuples in virtual tuples forms,
and ForeignNext() calls ExecMaterializeSlot() to store tableoid.
--
Itagaki Takahiro
On Tue, 18 Jan 2011 15:17:12 +0900
Itagaki Takahiro <itagaki.takahiro@gmail.com> wrote:
On Sat, Jan 15, 2011 at 08:35, Shigeru HANADA <hanada@metrosystems.co.jp> wrote:
Interface of NextCopyFrom() is fixed to return HeapTuple, to support
tableoid without any change to TupleTableSlot.3) 20110114-copy_export_HeapTupe.patch
This patch fixes interface of NextCopyFrom() to return results as
HeapTuple.I think file_fdw can return tuples in virtual tuples forms,
and ForeignNext() calls ExecMaterializeSlot() to store tableoid.
Thanks for the comment. I've fixed file_fdw to use tts_values and
tts_isnull to receive results from NextCopyFrom, and store virtual
tuple in the slot.
Attached patch requires FDW API patches and copy_export-20110114.patch.
Please see also:
http://archives.postgresql.org/message-id/20110119002615.8316.6989961C@metrosystems.co.jp
And also cost estimation of file_fdw is simplified along your comments
below.
On Tue, 21 Dec 2010 21:32:17 +0900
Itagaki Takahiro <itagaki.takahiro@gmail.com> wrote:
#3. Why do you re-open a foreign table in estimate_costs() ?
Since the caller seems to have the options for them, you can
pass them directly, no?In addition, passing a half-initialized fplan to estimate_costs()
is a bad idea. If you think it is an OUT parameter, the OUT params
should be *startup_cost and *total_cost.
In that message, you also pointed out that FDW must generate
explainInfo in every PlanRelScan call even if the planning is not for
EXPLAIN. I'll try to defer generating explainInfo until EXPLAIN
VERBOSE really uses it. It might need new hook point in expalain.c,
though.
Regards,
--
Shigeru Hanada
Attachments:
On Wed, Jan 19, 2011 at 00:34, Shigeru HANADA <hanada@metrosystems.co.jp> wrote:
Attached patch requires FDW API patches and copy_export-20110114.patch.
Some minor comments:
* Can you pass slot->tts_values and tts_isnull directly to NextCopyFrom()?
It won't allocate the arrays; just fill the array buffers.
* You can pass NULL for the 4th argument for NextCopyFrom().
| Oid tupleoid; /* just for required parameter */
* file_fdw_validator still has duplicated codes with BeginCopy,
but I have no idea to share the validation code in clean way...
* Try strVal() instead of DefElem->val.str
* FdwEPrivate seems too abbreviated for me. How about FileFdwPrivate?
* "private" is a bad identifier name because it's a C++ keyword.
We should rename FdwExecutionState->private.
In that message, you also pointed out that FDW must generate
explainInfo in every PlanRelScan call even if the planning is not for
EXPLAIN. I'll try to defer generating explainInfo until EXPLAIN
VERBOSE really uses it. It might need new hook point in expalain.c,
though.
I complained about the overhead, but it won't be a problem for
file_fdw and pgsql_fdw. file_fdw can easily generate the text,
and pgsql_fdw needs to generate a SQL query anyway.
My concern is the explainInfo interface is not ideal for the purpose
and therefore it will be unstable interface. If we support nested plans
in FDWs, each FDW should receive a tree writer used internally in
explain.c. explainInfo, that is a plan text, is not enough for complex
FdwPlans. However, since we don't have any better solution for now,
we could have the variable for 9.1. It's much better than nothing.
--
Itagaki Takahiro
On Thu, 20 Jan 2011 22:21:37 +0900
Itagaki Takahiro <itagaki.takahiro@gmail.com> wrote:
On Wed, Jan 19, 2011 at 00:34, Shigeru HANADA <hanada@metrosystems.co.jp> wrote:
Attached patch requires FDW API patches and copy_export-20110114.patch.
Some minor comments:
Thanks for the comments.
I'll post revised version of patches in new threads.
* Can you pass slot->tts_values and tts_isnull directly to NextCopyFrom()?
It won't allocate the arrays; just fill the array buffers.
I think it's safe to pass them to NextCopyFrom() directly because that
arrays are allocated in ExecSetSlotDescriptor() during
ExecInitForeignScan(), and size of arrays are taken from
Please let me know if I've missed your point.
* You can pass NULL for the 4th argument for NextCopyFrom().
| Oid tupleoid; /* just for required parameter */
I didn't know that it's NULL-safe, thanks.
* file_fdw_validator still has duplicated codes with BeginCopy,
but I have no idea to share the validation code in clean way...
It would be necessary to change considerable part of BeginCopy() to
separate validation from it to use validation from file_fdw...
* Try strVal() instead of DefElem->val.str
* FdwEPrivate seems too abbreviated for me. How about FileFdwPrivate?
Thanks, fixed.
* "private" is a bad identifier name because it's a C++ keyword.
We should rename FdwExecutionState->private.
Renamed to fdw_private, including another 'private' in FdwPlan.
In that message, you also pointed out that FDW must generate
explainInfo in every PlanRelScan call even if the planning is not for
EXPLAIN. I'll try to defer generating explainInfo until EXPLAIN
VERBOSE really uses it. It might need new hook point in expalain.c,
though.I complained about the overhead, but it won't be a problem for
file_fdw and pgsql_fdw. file_fdw can easily generate the text,
and pgsql_fdw needs to generate a SQL query anyway.My concern is the explainInfo interface is not ideal for the purpose
and therefore it will be unstable interface. If we support nested plans
in FDWs, each FDW should receive a tree writer used internally in
explain.c. explainInfo, that is a plan text, is not enough for complex
FdwPlans. However, since we don't have any better solution for now,
we could have the variable for 9.1. It's much better than nothing.
When I was writing file_fdw, I hoped to use static functions in
explain.c such as ExplainProperty() to handle complex information.
Even for single plan node, I think that filename and size (currently
they are printed in a plain text together) should be separated in the
output of explain, especially when the format was XML or JSON.
Regards,
--
Shigeru Hanada
On Fri, Jan 21, 2011 at 22:12, Shigeru HANADA <hanada@metrosystems.co.jp> wrote:
My concern is the explainInfo interface is not ideal for the purpose
and therefore it will be unstable interface. If we support nested plans
in FDWs, each FDW should receive a tree writer used internally in
explain.c. explainInfo, that is a plan text, is not enough for complex
FdwPlans. However, since we don't have any better solution for now,
we could have the variable for 9.1. It's much better than nothing.When I was writing file_fdw, I hoped to use static functions in
explain.c such as ExplainProperty() to handle complex information.
Even for single plan node, I think that filename and size (currently
they are printed in a plain text together) should be separated in the
output of explain, especially when the format was XML or JSON.
Just an idea -- we could return complex node trees with explainInfo
if we use XML or JSON for the format. For example, pgsql_fdw can
return the result from "EXPLAIN (FORMAT json)" without modification.
It might be one of the reasons we should should support JSON in the core :)
--
Itagaki Takahiro
On Fri, Jan 21, 2011 at 8:59 AM, Itagaki Takahiro
<itagaki.takahiro@gmail.com> wrote:
On Fri, Jan 21, 2011 at 22:12, Shigeru HANADA <hanada@metrosystems.co.jp> wrote:
My concern is the explainInfo interface is not ideal for the purpose
and therefore it will be unstable interface. If we support nested plans
in FDWs, each FDW should receive a tree writer used internally in
explain.c. explainInfo, that is a plan text, is not enough for complex
FdwPlans. However, since we don't have any better solution for now,
we could have the variable for 9.1. It's much better than nothing.When I was writing file_fdw, I hoped to use static functions in
explain.c such as ExplainProperty() to handle complex information.
Even for single plan node, I think that filename and size (currently
they are printed in a plain text together) should be separated in the
output of explain, especially when the format was XML or JSON.Just an idea -- we could return complex node trees with explainInfo
if we use XML or JSON for the format. For example, pgsql_fdw can
return the result from "EXPLAIN (FORMAT json)" without modification.It might be one of the reasons we should should support JSON in the core :)
Nice try, but I think that'd be a real drag. You wouldn't want to
return JSON when the explain format is text, or XML.
I think we probably need to modify the EXPLAIN code so that FDWs get a
chance to inject their own customer properties into the output, but I
don't know that we need to get that done right this minute. We can
ship something really crude/basic for 9.1, if need be, and fix this up
for 9.2. Of course if it turns out that getting EXPLAIN working the
way we'd like is really easy, then we can just do it.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Fri, Jan 21, 2011 at 8:12 AM, Shigeru HANADA
<hanada@metrosystems.co.jp> wrote:
* Try strVal() instead of DefElem->val.str
* FdwEPrivate seems too abbreviated for me. How about FileFdwPrivate?Thanks, fixed.
Was there supposed to be a patch attached here? Or where is it? We
are past out of time to get this committed, and there hasn't been a
new version in more than two weeks.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Fri, 4 Feb 2011 10:10:56 -0500
Robert Haas <robertmhaas@gmail.com> wrote:
Was there supposed to be a patch attached here? Or where is it? We
are past out of time to get this committed, and there hasn't been a
new version in more than two weeks.
Sorry for late to post patches.
Attached are revised version of file_fdw patch.
This patch is based on latest FDW API patches which are posted in
another thread "SQL/MED FDW API", and copy_export-20110104.patch which
was posted by Itagaki-san.
Regards,
--
Shigeru Hanada
Attachments:
On Mon, Feb 7, 2011 at 16:01, Shigeru HANADA <hanada@metrosystems.co.jp> wrote:
This patch is based on latest FDW API patches which are posted in
another thread "SQL/MED FDW API", and copy_export-20110104.patch which
was posted by Itagaki-san.
I have questions about estimate_costs().
* What value does baserel->tuples have?
Foreign tables are never analyzed for now. Is the number correct?
* Your previous measurement showed it has much more startup_cost.
When you removed ReScan, it took long time but planner didn't choose
materialized plans. It might come from lower startup costs.
* Why do you use lstat() in it?
Even if the file is a symlink, we will read the linked file in the
succeeding copy. So, I think it should be stat() rather than lstat().
+estimate_costs(const char *filename, RelOptInfo *baserel,
+ double *startup_cost, double *total_cost)
+{
...
+ /* get size of the file */
+ if (lstat(filename, &stat) == -1)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not stat file \"%s\": %m", filename)));
+ }
+
+ /*
+ * The way to estimate costs is almost same as cost_seqscan(), but there
+ * are some differences:
+ * - DISK costs are estimated from file size.
+ * - CPU costs are 10x of seq scan, for overhead of parsing records.
+ */
+ pages = stat.st_size / BLCKSZ + (stat.st_size % BLCKSZ > 0 ? 1 : 0);
+ run_cost += seq_page_cost * pages;
+
+ *startup_cost += baserel->baserestrictcost.startup;
+ cpu_per_tuple = cpu_tuple_cost + baserel->baserestrictcost.per_tuple;
+ run_cost += cpu_per_tuple * 10 * baserel->tuples;
+ *total_cost = *startup_cost + run_cost;
+
+ return stat.st_size;
+}
--
Itagaki Takahiro
On Mon, 7 Feb 2011 21:00:53 +0900
Itagaki Takahiro <itagaki.takahiro@gmail.com> wrote:
On Mon, Feb 7, 2011 at 16:01, Shigeru HANADA <hanada@metrosystems.co.jp> wrote:
This patch is based on latest FDW API patches which are posted in
another thread "SQL/MED FDW API", and copy_export-20110104.patch which
was posted by Itagaki-san.I have questions about estimate_costs().
* What value does baserel->tuples have?
Foreign tables are never analyzed for now. Is the number correct?
No, baserel->tuples is always 0, and baserel->pages is 0 too.
In addition, width and rows are set to 100 and 1, as default values.
I think ANALYZE support is needed to make PostgreSQL's standard
optimization for foreign scans. At present, estimation for foreign
tables would be awful.
* Your previous measurement showed it has much more startup_cost.
When you removed ReScan, it took long time but planner didn't choose
materialized plans. It might come from lower startup costs.
I tested joining file_fdw tables, accounts and branches, which are
initialized with "pgbench -i -s 10" and exported to csv files.
At first, I tried adding random_page_cost (4.0) to startup_cost as
cost to open file (though it's groundless), but materialized was not
chosen. After updating pg_class.reltuples to correct value, Hash-join
was choosen for same query. With disabling Hash-join, finally
materialized was choosen.
ISTM that choosing simple nested loop would come from wrong
estimation of loop count, but not from startup cost. IMHO, supporting
analyze (PG-style statistics) is necessary to make PostgreSQL to
generate sane plan.
* Why do you use lstat() in it?
Even if the file is a symlink, we will read the linked file in the
succeeding copy. So, I think it should be stat() rather than lstat().
Good catch! Fixed version is attached.
Regards,
--
Shigeru Hanada
Attachments:
On Tue, Feb 8, 2011 at 00:30, Shigeru HANADA <hanada@metrosystems.co.jp> wrote:
Fixed version is attached.
I reviewed your latest git version, that is a bit newer than the attached patch.
http://git.postgresql.org/gitweb?p=users/hanada/postgres.git;a=commit;h=0e1a1e1b0e168cb3d8ff4d637747d0ba8f7b8d55
The code still works with small adjustment, but needs to be rebased on the
latest master, especially for extension support and copy API changes.
Here are a list of comments and suggestions:
* You might forget some combination or unspecified options in
file_fdw_validator().
For example, format == NULL or !csv && header cases. I've not tested all
cases, but please recheck validations used in BeginCopy().
* estimate_costs() needs own row estimation rather than using baserel.
What value does baserel->tuples have?
Foreign tables are never analyzed for now. Is the number correct?No, baserel->tuples is always 0, and baserel->pages is 0 too.
In addition, width and rows are set to 100 and 1, as default values.
It means baserel is not reliable at all, right? If so, we need alternative
solution in estimate_costs(). We adjust statistics with runtime relation
size in estimate_rel_size(). Also, we use get_rel_data_width() for not
analyzed tables. We could use similar technique in file_fdw, too.
* Should use int64 for file byte size (or, uint32 in blocks).
unsigned long might be 32bit. ulong is not portable.
* Oid List (list_make1_oid) might be more handy than Const to hold relid
in FdwPlan.fdw_private.
* I prefer FileFdwExecutionState to FileFdwPrivate, because there are
two different 'fdw_private' variables in FdwPlan and FdwExecutionState.
* The comment in fileIterate seems wrong. It should be
/* Store the next tuple as a virtual tuple. */ , right?
* #include <sys/stat.h> is needed.
--
Itagaki Takahiro
On Wed, 16 Feb 2011 16:48:33 +0900
Itagaki Takahiro <itagaki.takahiro@gmail.com> wrote:
On Tue, Feb 8, 2011 at 00:30, Shigeru HANADA <hanada@metrosystems.co.jp> wrote:
Fixed version is attached.
I reviewed your latest git version, that is a bit newer than the attached patch.
http://git.postgresql.org/gitweb?p=users/hanada/postgres.git;a=commit;h=0e1a1e1b0e168cb3d8ff4d637747d0ba8f7b8d55The code still works with small adjustment, but needs to be rebased on the
latest master, especially for extension support and copy API changes.Here are a list of comments and suggestions:
Thanks for the comments. Revised version of patch has been attached.
* You might forget some combination or unspecified options in
file_fdw_validator().
For example, format == NULL or !csv && header cases. I've not tested all
cases, but please recheck validations used in BeginCopy().
Right, I've revised validation based on BeginCopy(), and added
regression tests about validation.
* estimate_costs() needs own row estimation rather than using baserel.
What value does baserel->tuples have?
Foreign tables are never analyzed for now. Is the number correct?No, baserel->tuples is always 0, and baserel->pages is 0 too.
In addition, width and rows are set to 100 and 1, as default values.It means baserel is not reliable at all, right?
Right, tables which has not been ANALYZEd have default stats in
baserel. But getting # of records needs another parsing for the file...
If so, we need alternative
solution in estimate_costs(). We adjust statistics with runtime relation
size in estimate_rel_size(). Also, we use get_rel_data_width() for not
analyzed tables. We could use similar technique in file_fdw, too.
Ah, using get_relation_data_width(), exported version of
get_rel_data_width(), seems to help estimation. I'll research around
it little more. By the way, adding ANALYZE support for foreign tables
is reasonable idea for this issue?
* Should use int64 for file byte size (or, uint32 in blocks).
unsigned long might be 32bit. ulong is not portable.* Oid List (list_make1_oid) might be more handy than Const to hold relid
in FdwPlan.fdw_private.* I prefer FileFdwExecutionState to FileFdwPrivate, because there are
two different 'fdw_private' variables in FdwPlan and FdwExecutionState.* The comment in fileIterate seems wrong. It should be
/* Store the next tuple as a virtual tuple. */ , right?* #include <sys/stat.h> is needed.
Fixed all of above.
Regards,
--
Shigeru Hanada
Attachments:
Shigeru HANADA <hanada@metrosystems.co.jp> writes:
[ 20110218-file_fdw.patch ]
I've adjusted this to fit the extensions infrastructure and the
committed version of the FDW API patch, and applied it.
* You might forget some combination or unspecified options in
file_fdw_validator().
For example, format == NULL or !csv && header cases. I've not tested all
cases, but please recheck validations used in BeginCopy().
Right, I've revised validation based on BeginCopy(), and added
regression tests about validation.
This approach struck me as entirely unmaintainable. I modified the core
COPY code to allow its option validation code to be called directly.
If so, we need alternative
solution in estimate_costs(). We adjust statistics with runtime relation
size in estimate_rel_size(). Also, we use get_rel_data_width() for not
analyzed tables. We could use similar technique in file_fdw, too.
Ah, using get_relation_data_width(), exported version of
get_rel_data_width(), seems to help estimation. I'll research around
it little more. By the way, adding ANALYZE support for foreign tables
is reasonable idea for this issue?
I did some quick hacking so that the numbers are at least a little bit
credible, but of course without ANALYZE support the qualification
selectivity estimates are likely to be pretty bogus. I am not sure
whether there's much of a use-case for supporting ANALYZE though.
I would think that if someone is going to read the same file in multiple
queries, they'd be far better off importing the data into a real table.
In any case, it's too late to worry about that for 9.1. I suggest
waiting to see what sort of real-world usage file_fdw gets before we
worry about whether it needs ANALYZE support.
regards, tom lane
Is this right?
postgres=# \d+ agg_text
Foreign table "public.agg_text"
Column | Type | Modifiers | Storage | Description
--------+----------+-----------+----------+-------------
a | smallint | | plain |
b | text | | extended |
Server: file_server
Has OIDs: no
It says the agg_text foreign table is using extended storage for the
text field. If it's in-file, how can it be classified as potentially
TOASTed?
--
Thom Brown
Twitter: @darkixion
IRC (freenode): dark_ixion
Registered Linux user: #516935
Thom Brown <thom@linux.com> writes:
Is this right?
postgres=# \d+ agg_text
Foreign table "public.agg_text"
Column | Type | Modifiers | Storage | Description
--------+----------+-----------+----------+-------------
a | smallint | | plain |
b | text | | extended |
Server: file_server
Has OIDs: no
It says the agg_text foreign table is using extended storage for the
text field. If it's in-file, how can it be classified as potentially
TOASTed?
It's meaningless, but I don't think it's worth trying to suppress the
meaningless column(s) ...
regards, tom lane