Re: SQL/MED - file_fdw
Itagaki Takahiro wrote:
Shigeru HANADA wrote:
Attached patch would avoid this leak by adding per-copy context to
CopyState. This would be overkill, and ResetCopyFrom() might be
reasonable though.Good catch. I merged your fix into the attached patch.
Review for CF:
While there is a post which suggests applying this patch in the
middle of a string of fdw patches, it stands alone without depending
on any of the other patches. I chose to focus on testing it alone,
to isolate just issues with this particular patch. In addition to
the base patch, there was a patch-on-patch which was applied to
change signature of NextCopyFrom to take fewer params and return
HeapTuple.
This patch is in context diff format, applies cleanly, compiles
without warning, passes all tests in `make check` and `make
installcheck-world`. From that and the testing I did, I think that
this patch could be committed before any of the others, if desired.
A few minor comments on format, though -- some parts of the patch
came out much more readable (for me, at least) when I regenerated the
diff using the git diff --patience switch. For example, see the end
of the CopyStateData structure definition each way. Also, whitespace
has little differences from what pgindent uses. Most are pretty
minor except for a section where the indentation wasn't changed based
on a change in depth of braces, to keep the patch size down.
It appears that the point of the patch is to provide a better API for
accessing the implementation of COPY, to support other patches. This
whole FDW effort is not an area of expertise for me, but the API
looks reasonable to me, with a somewhat parallel structure to some of
the other APIs used by the executor. From reading the FDW posts, it
appears that other patches are successfully using this new API, and
the reviewers of the other patches seem happy enough with it, which
would tend to indicate that it is on-target. Other hackers seem to
want it and we didn't already have it. From my reading of the posts
the idea of creating an API at this level was agreed upon by the
community.
The only two files touched by this patch are copy.h and copy.c. The
copy.h changes consisted entirely of new function prototypes and one
declaration of a pointer type (CopyState) to a struct defined and
used only inside copy.c (CopyStateData). That pointer is the return
value from BeginCopyFrom and required as a parameter to other new
functions. So the old API is still present exactly as it was, with
additional functions added.
I tried to read through the code to look for problems, but there are
so many structures and techniques involved that I haven't had to deal
with yet, that it would take me days to get up to speed enough to
desk check this adequately. Since this API is a prerequisite for
other patches, and already being used by them, I figured I should do
what I could quickly and then worry about how to cover that.
Since it doesn't appear to be intended to change any user-visible
behavior, I don't see any need for docs or changes to the regression
tests. I didn't see any portability problems. It seemed a little
light on comments to me, but perhaps that's because of my ignorance
of some of the techniques being used -- maybe things are obvious
enough to anyone who would be mucking about in copy.c.
I spent a few hours doing ad hoc tests of various sorts to try to
break things, without success. Among those tests were checks for
correct behavior on temporary tables and read only transactions --
separately and in combination. I ran millions of COPY statements
inside of a single database transaction (I tried both FROM and TO)
without seeing memory usage increase by more than a few kB. No
assertion failures. No segfaults. Nothing unusual in the log. I
plan to redo these tests with the full fdw patch set unless someone
has already covered that ground.
So far everything I've done has been with asserts enabled, so I
haven't tried to get serious benchmarks, but it seems fast. I will
rebuild without asserts and do performance tests before I change the
status on the CF page.
I'm wondering if it would make more sense to do the benchmarking with
just this patch or the full fdw patch set? Both?
-Kevin
On Tue, Jan 18, 2011 at 09:33, Kevin Grittner
<Kevin.Grittner@wicourts.gov> wrote:
Review for CF:
Thank your for the review!
Since it doesn't appear to be intended to change any user-visible
behavior, I don't see any need for docs or changes to the regression
tests.
There might be some user-visible behaviors in error messages
because I rearranged some codes to check errors, But we can see
the difference only if we have two or more errors in COPY commands.
They should be not so serious issues.
So far everything I've done has been with asserts enabled, so I
haven't tried to get serious benchmarks, but it seems fast. I will
rebuild without asserts and do performance tests before I change the
status on the CF page.I'm wondering if it would make more sense to do the benchmarking with
just this patch or the full fdw patch set? Both?
I tested the performance on my desktop PC, but I cannot see any
differences. But welcome if any of you could test on high-performance
servers.
Comparison with file_fdw would be more interesting
If they have similar performance, we could replace "COPY FROM" to
"CREATE TABLE AS SELECT FROM foreign_table", that is more flexible.
--
Itagaki Takahiro
On Mon, Jan 17, 2011 at 06:33:08PM -0600, Kevin Grittner wrote:
Itagaki Takahiro wrote:
Shigeru HANADA wrote:
Attached patch would avoid this leak by adding per-copy context to
CopyState. This would be overkill, and ResetCopyFrom() might be
reasonable though.Good catch. I merged your fix into the attached patch.
Review for CF:
...
I tried to read through the code to look for problems, but there are
so many structures and techniques involved that I haven't had to deal
with yet, that it would take me days to get up to speed enough to
desk check this adequately. Since this API is a prerequisite for
other patches, and already being used by them, I figured I should do
what I could quickly and then worry about how to cover that.
I've studied the patch from this angle. My two substantive questions concern
CopyFromErrorCallback() error strings and the use of the per-tuple memory
context; see below. The rest either entail trivial fixes, or they do not
indicate anything that clearly ought to change.
The patch mostly moves code around; it's a bit difficult to grasp what's really
changing by looking at the diff (not a criticism of the approach -- I see no way
to avoid that). The final code structure is at least as easy to follow as the
structure it replaces. Here is a summary of the changes:
file_fdw wishes to borrow the COPY FROM code for parsing structured text and
building tuples fitting a given relation. file_fdw must bypass the parts that
insert those tuples. Until now, copy.c has exposed a single API, DoCopy().
To meet file_fdw's needs, the patch adds four new APIs for use as a set:
BeginCopyFrom() - once-per-COPY initialization and validation
NextCopyFrom() - call N times to get N values/null arrays
EndCopyFrom() - once-per-COPY cleanup
CopyFromErrorCallback() - for caller use in ErrorContextCallback.callback
Implementing these entails breaking apart the existing code structure. For
one, the patch separates from DoCopy the once-per-COPY-statement code, both
initialization and cleanup. Secondly, it separates the CopyFrom code for
assembling a tuple based on COPY input from the code that actually stores said
tuples in the target relation. To avoid duplicating code, the patch then
calls these new functions from DoCopy and CopyFrom. To further avoid
duplicating code and to retain symmetry, it refactors COPY TO setup and
teardown along similar lines. We have four new static functions:
BeginCopyTo() - parallel to BeginCopyFrom(), for DoCopy() alone
BeginCopy() - shared bits between BeginCopyTo() and BeginCopyFrom()
EndCopyTo() - parallel to EndCopyFrom(), for DoCopy() alone
EndCopy() - shared bits between EndCopyTo() and EndCopyFrom()
Most "parse analysis"-type bits of DoCopy() move to BeginCopy(). Several
checks remain in DoCopy(): superuser for reading a server-local file,
permission on the relation, and a read-only transaction check. The first
stays where it does because a superuser may define a file_fdw foreign table
and then grant access to others. The other two remain in DoCopy() because the
new API uses the relation as a template without reading or writing it.
The catalog work (looking up defaults, I/O functions, etc) in CopyFrom() moves
to BeginCopyFrom(), and applicable local variables become CopyState members.
copy_in_error_callback changes name to CopyFromErrorCallback() to better fit
with the rest of the new API. Its implementation does not change at all.
Since it's now possible for one SQL statement to call into the COPY machinery
an arbitrary number of times, the patch introduces a per-COPY memory context.
The patch implements added refactorings that make sense but are peripheral to
the API change at hand. CopyState.fe_copy is gone, now calculated locally in
DoCopyTo(). CopyState.processed is gone, with the row count now bubbling up
through return values. We now initialize the CopyState buffers for COPY FROM
only; they are unused in COPY TO. The purest of patches would defer all these,
but I don't object to them tagging along.
For COPY TO, the relkind checks move from DoCopyTo() to BeginCopyTo(). I'm
skeptical about this one. It's not required for correctness, and the relkind
checks for COPY FROM remain in CopyFrom().
file_fdw uses CopyFromErrorCallback() to give errors the proper context. The
function uses template strings like "COPY %s, line %d", where %s is the name of
the relation being copied. Presumably file_fdw and other features using this
API would wish to customize that error message prefix, and the relation name
might not be apropos at all. How about another argument to BeginCopyFrom,
specifying an error prefix to be stashed in the CopyState?
We could easily regret requiring a Relation in BeginCopyFrom(); another user may
wish to use a fabricated TupleDesc. Code paths in the new API use the Relation
for a TupleDesc, relhashoids, column defaults, and error message strings.
Removing the need for a full Relation is within reach. That being said, the API
definitely loses cleanliness if we cater to that possibility now. It's probably
best to keep the API as the patch has it, and let the hypothetical future use
case change it.
There were some concerns upthread that exposing more of copy.c would attract use
in third party FDWs, hamstringing future efforts to improve the implementation
for the COPY command itself. On review, FDW modules are not likely users in
general. file_fdw wants this because it parses structured text files, not
because it's an FDW. Either way, the API also seems nicely-chosen to minimize
the exposure of internals.
I have not tested the patched code in any detail, because Kevin's review covered
that angle quite thoroughly. I have no cause to suspect that the patch will
change user-visible function behavior, aside from the spurious error message
changes noted below. It looks unlikely to affect performance, but that's
probably worth a quick test.
Since it doesn't appear to be intended to change any user-visible
behavior, I don't see any need for docs or changes to the regression
tests. I didn't see any portability problems. It seemed a little
light on comments to me, but perhaps that's because of my ignorance
of some of the techniques being used -- maybe things are obvious
enough to anyone who would be mucking about in copy.c.
In some places, the new code does not add comments despite the surrounding code
having many comments. However, some existing comments went a bit too far
(/* Place tuple in tuple slot */, /* Reset the per-tuple exprcontext */).
The degree of commenting is mostly good, but I've noted a few places below.
The header comment for BeginCopyFrom should probably explain the arguments a bit
more; it assumes familiarity with the COPY implementation. How about this:
/*
* Setup to read tuples from a file for COPY FROM.
*
* 'rel': used as a template for the tuples
* 'filename': name of server-local file to read
* 'attnamelist': List of char *, columns to include. NIL selects all cols.
* 'options': List of DefElem. See copy_opt_item in gram.y for selections.
*
* Returns a CopyState, to be passed to NextCopyFrom().
*/
Thanks,
nm
Comments on specific hunks:
*** a/src/backend/commands/copy.c --- b/src/backend/commands/copy.c
***************
*** 167,181 **** typedef struct CopyStateData
char *raw_buf;
int raw_buf_index; /* next byte to process */
int raw_buf_len; /* total # of bytes stored */
} CopyStateData;- typedef CopyStateData *CopyState;
-
/* DestReceiver for COPY (SELECT) TO */
typedef struct
{
DestReceiver pub; /* publicly-known function pointers */
CopyState cstate; /* CopyStateData for the command */
} DR_copy;--- 170,197 ---- char *raw_buf; int raw_buf_index; /* next byte to process */ int raw_buf_len; /* total # of bytes stored */ + + /* + * The definition of input functions and default expressions are stored + * in these variables. + */ + EState *estate; + AttrNumber num_defaults; + bool file_has_oids; + FmgrInfo oid_in_function; + Oid oid_typioparam; + FmgrInfo *in_functions; + Oid *typioparams; + int *defmap; + ExprState **defexprs; /* array of default att expressions */ } CopyStateData;
The leading comment only applies to a few of those fields. Each field needs its
own comment, instead.
***************
*** 1314,1339 **** DoCopyTo(CopyState cstate)
}
PG_END_TRY();! if (!pipe)
{
! if (FreeFile(cstate->copy_file))
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not write to file \"%s\": %m",
! cstate->filename)));
}
}/*
* Copy from relation or query TO file.
*/
! static void
CopyTo(CopyState cstate)
{
TupleDesc tupDesc;
int num_phys_attrs;
Form_pg_attribute *attr;
ListCell *cur;if (cstate->rel) tupDesc = RelationGetDescr(cstate->rel); --- 1396,1442 ---- } PG_END_TRY();! return processed;
! }
!
! /*
! * Clean up storage and release resources for COPY TO.
! */
! static void
! EndCopyTo(CopyState cstate)
! {
! if (cstate->filename != NULL && FreeFile(cstate->copy_file))
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not close file \"%s\": %m",
! cstate->filename)));
The old error message was "could not write to file \"%s\": %m". This might be a
good change, but it belongs in a separate patch.
!
! /*
! * Close the relation or query. We can release the AccessShareLock we got.
! */
Separated from its former location, this comment is not applicable.
***************
*** 1826,1832 **** CopyFrom(CopyState cstate)
slot = ExecInitExtraTupleSlot(estate);
ExecSetSlotDescriptor(slot, tupDesc);! econtext = GetPerTupleExprContext(estate);
/* * Pick up the required catalog information for each attribute in the --- 1889,2070 ---- slot = ExecInitExtraTupleSlot(estate); ExecSetSlotDescriptor(slot, tupDesc);! /* Prepare to catch AFTER triggers. */
! AfterTriggerBeginQuery();
!
! /*
! * Check BEFORE STATEMENT insertion triggers. It's debateable whether we
! * should do this for COPY, since it's not really an "INSERT" statement as
! * such. However, executing these triggers maintains consistency with the
! * EACH ROW triggers that we already fire on COPY.
! */
! ExecBSInsertTriggers(estate, resultRelInfo);
!
! values = (Datum *) palloc(tupDesc->natts * sizeof(Datum));
! nulls = (bool *) palloc(tupDesc->natts * sizeof(bool));
!
! bistate = GetBulkInsertState();
!
! /* Set up callback to identify error line number */
! errcontext.callback = CopyFromErrorCallback;
! errcontext.arg = (void *) cstate;
! errcontext.previous = error_context_stack;
! error_context_stack = &errcontext;
!
! while (!done)
! {
The loop may as well be unconditional ...
! bool skip_tuple;
! Oid loaded_oid = InvalidOid;
!
! CHECK_FOR_INTERRUPTS();
!
! /* Reset the per-tuple exprcontext */
! ResetPerTupleExprContext(estate);
!
! /* Switch into its memory context */
! MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
Shouldn't a switch to this context happen inside NextCopyFrom(), then again for
the calls to heap_form_tuple/HeapTupleSetOid? I found previous discussion on
this matter, but no conclusion.
!
! done = !NextCopyFrom(cstate, values, nulls, &loaded_oid);
! if (done)
! break;
... since this is the only exit. No need to maintain "done".
! /*
! * Clean up storage and release resources for COPY FROM.
! */
! void
! EndCopyFrom(CopyState cstate)
! {
! FreeExecutorState(cstate->estate);
!
! if (cstate->filename != NULL && FreeFile(cstate->copy_file))
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not close file \"%s\": %m",
! cstate->filename)));
Likewise: this might be a good verbiage change, but it belongs in its own patch.
Thank you for the detailed review!
Revised patch attached, but APIs for jagged csv is not included in it.
On Sun, Feb 6, 2011 at 09:01, Noah Misch <noah@leadboat.com> wrote:
Most "parse analysis"-type bits of DoCopy() move to BeginCopy().
It would be possible to move more FROM-only or TO-only members in BeginCopy()
into their BeginCopyFrom/To(), but it is just a code refactoring requires
more code rearrangement. It should be done by another try even if needed.
CopyState.processed is gone, with the row count now bubbling up
through return values.
I removed it from CopyState because it represents "number of inserted rows"
in COPY FROM. However, for the viewpoint of API, the numbers of lines in
the input file is more suitable. To avoid the confusion, I moved it
from a common member to COPY FROM-only variable.
For COPY TO, the relkind checks move from DoCopyTo() to BeginCopyTo(). I'm
skeptical about this one. It's not required for correctness, and the relkind
checks for COPY FROM remain in CopyFrom().
I think error checks should be done in the early phase, so I moved the check
to BeginCopyTo(). However, relkind check in COPY FROM is needed only for
COPY FROM command because the relation is the inserted target. For APIs,
the relation is used as a template for input file. So, we cannot perform
the check in BeginCopyFrom().
file_fdw uses CopyFromErrorCallback() to give errors the proper context. The
function uses template strings like "COPY %s, line %d", where %s is the name of
the relation being copied. Presumably file_fdw and other features using this
API would wish to customize that error message prefix, and the relation name
might not be apropos at all. How about another argument to BeginCopyFrom,
specifying an error prefix to be stashed in the CopyState?
I changed "COPY %s, .." to "relation %s, ..." because the first string is
the relation name anyway. We could have another prefix argument, but I think
it has little information for errors.
We also have many "COPY" in other messages, but they won't be used actually
because the are messages for invalid arguments and file_fdw will have own
validater function. All invalid arguments will be filtered in CREATE commands.
We could easily regret requiring a Relation in BeginCopyFrom(); another user may
wish to use a fabricated TupleDesc. Code paths in the new API use the Relation
for a TupleDesc, relhashoids, column defaults, and error message strings.
Removing the need for a full Relation is within reach. That being said, the API
definitely loses cleanliness if we cater to that possibility now. It's probably
best to keep the API as the patch has it, and let the hypothetical future use
case change it.
Agreed. The current file_fdw has a template relation anyway.
The header comment for BeginCopyFrom should probably explain the arguments a bit
more; it assumes familiarity with the COPY implementation. How about this:
(snip)
Comments on specific hunks:
Thanks included.
! /* Reset the per-tuple exprcontext */
! ResetPerTupleExprContext(estate);
!
! /* Switch into its memory context */
! MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));Shouldn't a switch to this context happen inside NextCopyFrom(), then again for
the calls to heap_form_tuple/HeapTupleSetOid? I found previous discussion on
this matter, but no conclusion.
In my understanding, NextCopyFrom() should use CurrentMemoryContext provided
by the caller. The file_fdw will use executor's per-tuple context for it.
Another issue in the automatic context switch is to discard the previous
values in the next call while the caller is unaware.
--
Itagaki Takahiro
Attachments:
copy_export-20110207.patchapplication/octet-stream; name=copy_export-20110207.patchDownload
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index 3350ca0..6df0f1e 100644
*** a/src/backend/commands/copy.c
--- b/src/backend/commands/copy.c
*************** typedef struct CopyStateData
*** 93,105 ****
FILE *copy_file; /* used if copy_dest == COPY_FILE */
StringInfo fe_msgbuf; /* used for all dests during COPY TO, only for
* dest == COPY_NEW_FE in COPY FROM */
- bool fe_copy; /* true for all FE copy dests */
bool fe_eof; /* true if detected end of copy data */
EolType eol_type; /* EOL type of input */
int client_encoding; /* remote side's character encoding */
bool need_transcoding; /* client encoding diff from server? */
bool encoding_embeds_ascii; /* ASCII can be non-first byte? */
- uint64 processed; /* # of tuples processed */
/* parameters from the COPY command */
Relation rel; /* relation to copy to or from */
--- 93,103 ----
*************** typedef struct CopyStateData
*** 119,131 ****
bool *force_quote_flags; /* per-column CSV FQ flags */
bool *force_notnull_flags; /* per-column CSV FNN flags */
! /* these are just for error messages, see copy_in_error_callback */
const char *cur_relname; /* table name for error messages */
int cur_lineno; /* line number for error messages */
const char *cur_attname; /* current att for error messages */
const char *cur_attval; /* current att value for error messages */
/*
* Working state for COPY TO
*/
FmgrInfo *out_functions; /* lookup info for output functions */
--- 117,134 ----
bool *force_quote_flags; /* per-column CSV FQ flags */
bool *force_notnull_flags; /* per-column CSV FNN flags */
! /* these are just for error messages, see CopyFromErrorCallback */
const char *cur_relname; /* table name for error messages */
int cur_lineno; /* line number for error messages */
const char *cur_attname; /* current att for error messages */
const char *cur_attval; /* current att value for error messages */
/*
+ * Working state for COPY TO/FROM
+ */
+ MemoryContext copycontext; /* per-copy execution context */
+
+ /*
* Working state for COPY TO
*/
FmgrInfo *out_functions; /* lookup info for output functions */
*************** typedef struct CopyStateData
*** 167,181 ****
char *raw_buf;
int raw_buf_index; /* next byte to process */
int raw_buf_len; /* total # of bytes stored */
- } CopyStateData;
! typedef CopyStateData *CopyState;
/* DestReceiver for COPY (SELECT) TO */
typedef struct
{
DestReceiver pub; /* publicly-known function pointers */
CopyState cstate; /* CopyStateData for the command */
} DR_copy;
--- 170,197 ----
char *raw_buf;
int raw_buf_index; /* next byte to process */
int raw_buf_len; /* total # of bytes stored */
! /*
! * The definition of input functions and default expressions are stored
! * in these variables.
! */
! EState *estate;
! AttrNumber num_defaults;
! bool file_has_oids;
! FmgrInfo oid_in_function;
! Oid oid_typioparam;
! FmgrInfo *in_functions; /* array of input functions for each attrs */
! Oid *typioparams; /* array of element types for in_functions */
! int *defmap; /* array of default att numbers */
! ExprState **defexprs; /* array of default att expressions */
! } CopyStateData;
/* DestReceiver for COPY (SELECT) TO */
typedef struct
{
DestReceiver pub; /* publicly-known function pointers */
CopyState cstate; /* CopyStateData for the command */
+ uint64 processed; /* # of tuples processed */
} DR_copy;
*************** static const char BinarySignature[11] =
*** 248,258 ****
/* non-export function prototypes */
! static void DoCopyTo(CopyState cstate);
! static void CopyTo(CopyState cstate);
static void CopyOneRowTo(CopyState cstate, Oid tupleOid,
Datum *values, bool *nulls);
! static void CopyFrom(CopyState cstate);
static bool CopyReadLine(CopyState cstate);
static bool CopyReadLineText(CopyState cstate);
static int CopyReadAttributesText(CopyState cstate);
--- 264,280 ----
/* non-export function prototypes */
! static CopyState BeginCopy(bool is_from, Relation rel, Node *raw_query,
! const char *queryString, List *attnamelist, List *options);
! static void EndCopy(CopyState cstate);
! static CopyState BeginCopyTo(Relation rel, Node *query, const char *queryString,
! const char *filename, List *attnamelist, List *options);
! static void EndCopyTo(CopyState cstate);
! static uint64 DoCopyTo(CopyState cstate);
! static uint64 CopyTo(CopyState cstate);
static void CopyOneRowTo(CopyState cstate, Oid tupleOid,
Datum *values, bool *nulls);
! static uint64 CopyFrom(CopyState cstate);
static bool CopyReadLine(CopyState cstate);
static bool CopyReadLineText(CopyState cstate);
static int CopyReadAttributesText(CopyState cstate);
*************** DoCopy(const CopyStmt *stmt, const char
*** 724,745 ****
CopyState cstate;
bool is_from = stmt->is_from;
bool pipe = (stmt->filename == NULL);
! List *attnamelist = stmt->attlist;
List *force_quote = NIL;
List *force_notnull = NIL;
bool force_quote_all = false;
bool format_specified = false;
- AclMode required_access = (is_from ? ACL_INSERT : ACL_SELECT);
ListCell *option;
TupleDesc tupDesc;
int num_phys_attrs;
! uint64 processed;
/* Allocate workspace and zero all fields */
cstate = (CopyStateData *) palloc0(sizeof(CopyStateData));
/* Extract options from the statement node tree */
! foreach(option, stmt->options)
{
DefElem *defel = (DefElem *) lfirst(option);
--- 746,870 ----
CopyState cstate;
bool is_from = stmt->is_from;
bool pipe = (stmt->filename == NULL);
! Relation rel;
! uint64 processed;
!
! /* Disallow file COPY except to superusers. */
! if (!pipe && !superuser())
! ereport(ERROR,
! (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
! errmsg("must be superuser to COPY to or from a file"),
! errhint("Anyone can COPY to stdout or from stdin. "
! "psql's \\copy command also works for anyone.")));
!
! if (stmt->relation)
! {
! TupleDesc tupDesc;
! AclMode required_access = (is_from ? ACL_INSERT : ACL_SELECT);
! RangeTblEntry *rte;
! List *attnums;
! ListCell *cur;
!
! Assert(!stmt->query);
!
! /* Open and lock the relation, using the appropriate lock type. */
! rel = heap_openrv(stmt->relation,
! (is_from ? RowExclusiveLock : AccessShareLock));
!
! rte = makeNode(RangeTblEntry);
! rte->rtekind = RTE_RELATION;
! rte->relid = RelationGetRelid(rel);
! rte->requiredPerms = required_access;
!
! tupDesc = RelationGetDescr(rel);
! attnums = CopyGetAttnums(tupDesc, rel, stmt->attlist);
! foreach (cur, attnums)
! {
! int attno = lfirst_int(cur) -
! FirstLowInvalidHeapAttributeNumber;
!
! if (is_from)
! rte->modifiedCols = bms_add_member(rte->modifiedCols, attno);
! else
! rte->selectedCols = bms_add_member(rte->selectedCols, attno);
! }
! ExecCheckRTPerms(list_make1(rte), true);
! }
! else
! {
! Assert(stmt->query);
!
! rel = NULL;
! }
!
! if (is_from)
! {
! /* check read-only transaction */
! if (XactReadOnly && rel->rd_backend != MyBackendId)
! PreventCommandIfReadOnly("COPY FROM");
!
! cstate = BeginCopyFrom(rel, stmt->filename,
! stmt->attlist, stmt->options);
! processed = CopyFrom(cstate); /* copy from file to database */
! EndCopyFrom(cstate);
! }
! else
! {
! cstate = BeginCopyTo(rel, stmt->query, queryString, stmt->filename,
! stmt->attlist, stmt->options);
! processed = DoCopyTo(cstate); /* copy from database to file */
! EndCopyTo(cstate);
! }
!
! /*
! * Close the relation. If reading, we can release the AccessShareLock we got;
! * if writing, we should hold the lock until end of transaction to ensure that
! * updates will be committed before lock is released.
! */
! if (rel != NULL)
! heap_close(rel, (is_from ? NoLock : AccessShareLock));
!
! return processed;
! }
!
! /*
! * Common setup routines used by BeginCopyFrom and BeginCopyTo.
! */
! static CopyState
! BeginCopy(bool is_from,
! Relation rel,
! Node *raw_query,
! const char *queryString,
! List *attnamelist,
! List *options)
! {
! CopyState cstate;
List *force_quote = NIL;
List *force_notnull = NIL;
bool force_quote_all = false;
bool format_specified = false;
ListCell *option;
TupleDesc tupDesc;
int num_phys_attrs;
! MemoryContext oldcontext;
/* Allocate workspace and zero all fields */
cstate = (CopyStateData *) palloc0(sizeof(CopyStateData));
+ /*
+ * We allocate everything used by a cstate in a new memory context.
+ * This would avoid memory leaks repeated uses of COPY in a query.
+ */
+ cstate->copycontext = AllocSetContextCreate(CurrentMemoryContext,
+ "COPY",
+ ALLOCSET_DEFAULT_MINSIZE,
+ ALLOCSET_DEFAULT_INITSIZE,
+ ALLOCSET_DEFAULT_MAXSIZE);
+
+ oldcontext = MemoryContextSwitchTo(cstate->copycontext);
+
/* Extract options from the statement node tree */
! foreach(option, options)
{
DefElem *defel = (DefElem *) lfirst(option);
*************** DoCopy(const CopyStmt *stmt, const char
*** 980,1030 ****
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("CSV quote character must not appear in the NULL specification")));
! /* Disallow file COPY except to superusers. */
! if (!pipe && !superuser())
! ereport(ERROR,
! (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
! errmsg("must be superuser to COPY to or from a file"),
! errhint("Anyone can COPY to stdout or from stdin. "
! "psql's \\copy command also works for anyone.")));
!
! if (stmt->relation)
{
! RangeTblEntry *rte;
! List *attnums;
! ListCell *cur;
!
! Assert(!stmt->query);
! cstate->queryDesc = NULL;
! /* Open and lock the relation, using the appropriate lock type. */
! cstate->rel = heap_openrv(stmt->relation,
! (is_from ? RowExclusiveLock : AccessShareLock));
tupDesc = RelationGetDescr(cstate->rel);
- /* Check relation permissions. */
- rte = makeNode(RangeTblEntry);
- rte->rtekind = RTE_RELATION;
- rte->relid = RelationGetRelid(cstate->rel);
- rte->requiredPerms = required_access;
-
- attnums = CopyGetAttnums(tupDesc, cstate->rel, attnamelist);
- foreach (cur, attnums)
- {
- int attno = lfirst_int(cur) - FirstLowInvalidHeapAttributeNumber;
-
- if (is_from)
- rte->modifiedCols = bms_add_member(rte->modifiedCols, attno);
- else
- rte->selectedCols = bms_add_member(rte->selectedCols, attno);
- }
- ExecCheckRTPerms(list_make1(rte), true);
-
- /* check read-only transaction */
- if (XactReadOnly && is_from && cstate->rel->rd_backend != MyBackendId)
- PreventCommandIfReadOnly("COPY FROM");
-
/* Don't allow COPY w/ OIDs to or from a table without them */
if (cstate->oids && !cstate->rel->rd_rel->relhasoids)
ereport(ERROR,
--- 1105,1118 ----
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("CSV quote character must not appear in the NULL specification")));
! if (rel)
{
! Assert(!raw_query);
! cstate->rel = rel;
tupDesc = RelationGetDescr(cstate->rel);
/* Don't allow COPY w/ OIDs to or from a table without them */
if (cstate->oids && !cstate->rel->rd_rel->relhasoids)
ereport(ERROR,
*************** DoCopy(const CopyStmt *stmt, const char
*** 1058,1064 ****
* function and is executed repeatedly. (See also the same hack in
* DECLARE CURSOR and PREPARE.) XXX FIXME someday.
*/
! rewritten = pg_analyze_and_rewrite((Node *) copyObject(stmt->query),
queryString, NULL, 0);
/* We don't expect more or less than one result query */
--- 1146,1152 ----
* function and is executed repeatedly. (See also the same hack in
* DECLARE CURSOR and PREPARE.) XXX FIXME someday.
*/
! rewritten = pg_analyze_and_rewrite((Node *) copyObject(raw_query),
queryString, NULL, 0);
/* We don't expect more or less than one result query */
*************** DoCopy(const CopyStmt *stmt, const char
*** 1160,1173 ****
}
}
- /* Set up variables to avoid per-attribute overhead. */
- initStringInfo(&cstate->attribute_buf);
- initStringInfo(&cstate->line_buf);
- cstate->line_buf_converted = false;
- cstate->raw_buf = (char *) palloc(RAW_BUF_SIZE + 1);
- cstate->raw_buf_index = cstate->raw_buf_len = 0;
- cstate->processed = 0;
-
/*
* Set up encoding conversion info. Even if the client and server
* encodings are the same, we must apply pg_client_to_server() to validate
--- 1248,1253 ----
*************** DoCopy(const CopyStmt *stmt, const char
*** 1181,1264 ****
cstate->encoding_embeds_ascii = PG_ENCODING_IS_CLIENT_ONLY(cstate->client_encoding);
cstate->copy_dest = COPY_FILE; /* default */
- cstate->filename = stmt->filename;
! if (is_from)
! CopyFrom(cstate); /* copy from file to database */
! else
! DoCopyTo(cstate); /* copy from database to file */
! /*
! * Close the relation or query. If reading, we can release the
! * AccessShareLock we got; if writing, we should hold the lock until end
! * of transaction to ensure that updates will be committed before lock is
! * released.
! */
! if (cstate->rel)
! heap_close(cstate->rel, (is_from ? NoLock : AccessShareLock));
! else
! {
! /* Close down the query and free resources. */
! ExecutorEnd(cstate->queryDesc);
! FreeQueryDesc(cstate->queryDesc);
! PopActiveSnapshot();
! }
! /* Clean up storage (probably not really necessary) */
! processed = cstate->processed;
! pfree(cstate->attribute_buf.data);
! pfree(cstate->line_buf.data);
! pfree(cstate->raw_buf);
pfree(cstate);
-
- return processed;
}
-
/*
! * This intermediate routine exists mainly to localize the effects of setjmp
! * so we don't need to plaster a lot of variables with "volatile".
*/
! static void
! DoCopyTo(CopyState cstate)
{
! bool pipe = (cstate->filename == NULL);
! if (cstate->rel)
{
! if (cstate->rel->rd_rel->relkind != RELKIND_RELATION)
! {
! if (cstate->rel->rd_rel->relkind == RELKIND_VIEW)
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from view \"%s\"",
! RelationGetRelationName(cstate->rel)),
! errhint("Try the COPY (SELECT ...) TO variant.")));
! else if (cstate->rel->rd_rel->relkind == RELKIND_FOREIGN_TABLE)
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from foreign table \"%s\"",
! RelationGetRelationName(cstate->rel)),
! errhint("Try the COPY (SELECT ...) TO variant.")));
! else if (cstate->rel->rd_rel->relkind == RELKIND_SEQUENCE)
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from sequence \"%s\"",
! RelationGetRelationName(cstate->rel))));
! else
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from non-table relation \"%s\"",
! RelationGetRelationName(cstate->rel))));
! }
}
if (pipe)
{
! if (whereToSendOutput == DestRemote)
! cstate->fe_copy = true;
! else
cstate->copy_file = stdout;
}
else
--- 1261,1335 ----
cstate->encoding_embeds_ascii = PG_ENCODING_IS_CLIENT_ONLY(cstate->client_encoding);
cstate->copy_dest = COPY_FILE; /* default */
! MemoryContextSwitchTo(oldcontext);
! return cstate;
! }
! /*
! * Release resources allocated in a cstate.
! */
! static void
! EndCopy(CopyState cstate)
! {
! if (cstate->filename != NULL && FreeFile(cstate->copy_file))
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not close file \"%s\": %m",
! cstate->filename)));
! MemoryContextDelete(cstate->copycontext);
pfree(cstate);
}
/*
! * Setup CopyState to read tuples from a table or a query for COPY TO.
*/
! static CopyState
! BeginCopyTo(Relation rel,
! Node *query,
! const char *queryString,
! const char *filename,
! List *attnamelist,
! List *options)
{
! CopyState cstate;
! bool pipe = (filename == NULL);
! MemoryContext oldcontext;
! if (rel != NULL && rel->rd_rel->relkind != RELKIND_RELATION)
{
! if (rel->rd_rel->relkind == RELKIND_VIEW)
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from view \"%s\"",
! RelationGetRelationName(rel)),
! errhint("Try the COPY (SELECT ...) TO variant.")));
! else if (rel->rd_rel->relkind == RELKIND_FOREIGN_TABLE)
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from foreign table \"%s\"",
! RelationGetRelationName(rel)),
! errhint("Try the COPY (SELECT ...) TO variant.")));
! else if (rel->rd_rel->relkind == RELKIND_SEQUENCE)
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from sequence \"%s\"",
! RelationGetRelationName(rel))));
! else
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from non-table relation \"%s\"",
! RelationGetRelationName(rel))));
}
+ cstate = BeginCopy(false, rel, query, queryString, attnamelist, options);
+ oldcontext = MemoryContextSwitchTo(cstate->copycontext);
+
if (pipe)
{
! if (whereToSendOutput != DestRemote)
cstate->copy_file = stdout;
}
else
*************** DoCopyTo(CopyState cstate)
*** 1270,1280 ****
* Prevent write to relative path ... too easy to shoot oneself in the
* foot by overwriting a database file ...
*/
! if (!is_absolute_path(cstate->filename))
ereport(ERROR,
(errcode(ERRCODE_INVALID_NAME),
errmsg("relative path not allowed for COPY to file")));
oumask = umask(S_IWGRP | S_IWOTH);
cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_W);
umask(oumask);
--- 1341,1352 ----
* Prevent write to relative path ... too easy to shoot oneself in the
* foot by overwriting a database file ...
*/
! if (!is_absolute_path(filename))
ereport(ERROR,
(errcode(ERRCODE_INVALID_NAME),
errmsg("relative path not allowed for COPY to file")));
+ cstate->filename = pstrdup(filename);
oumask = umask(S_IWGRP | S_IWOTH);
cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_W);
umask(oumask);
*************** DoCopyTo(CopyState cstate)
*** 1292,1305 ****
errmsg("\"%s\" is a directory", cstate->filename)));
}
PG_TRY();
{
! if (cstate->fe_copy)
SendCopyBegin(cstate);
! CopyTo(cstate);
! if (cstate->fe_copy)
SendCopyEnd(cstate);
}
PG_CATCH();
--- 1364,1393 ----
errmsg("\"%s\" is a directory", cstate->filename)));
}
+ MemoryContextSwitchTo(oldcontext);
+
+ return cstate;
+ }
+
+ /*
+ * This intermediate routine exists mainly to localize the effects of setjmp
+ * so we don't need to plaster a lot of variables with "volatile".
+ */
+ static uint64
+ DoCopyTo(CopyState cstate)
+ {
+ bool pipe = (cstate->filename == NULL);
+ bool fe_copy = (pipe && whereToSendOutput == DestRemote);
+ uint64 processed;
+
PG_TRY();
{
! if (fe_copy)
SendCopyBegin(cstate);
! processed = CopyTo(cstate);
! if (fe_copy)
SendCopyEnd(cstate);
}
PG_CATCH();
*************** DoCopyTo(CopyState cstate)
*** 1314,1339 ****
}
PG_END_TRY();
! if (!pipe)
{
! if (FreeFile(cstate->copy_file))
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not close file \"%s\": %m",
! cstate->filename)));
}
}
/*
* Copy from relation or query TO file.
*/
! static void
CopyTo(CopyState cstate)
{
TupleDesc tupDesc;
int num_phys_attrs;
Form_pg_attribute *attr;
ListCell *cur;
if (cstate->rel)
tupDesc = RelationGetDescr(cstate->rel);
--- 1402,1439 ----
}
PG_END_TRY();
! return processed;
! }
!
! /*
! * Clean up storage and release resources for COPY TO.
! */
! static void
! EndCopyTo(CopyState cstate)
! {
! if (cstate->queryDesc != NULL)
{
! /* Close down the query and free resources. */
! ExecutorEnd(cstate->queryDesc);
! FreeQueryDesc(cstate->queryDesc);
! PopActiveSnapshot();
}
+
+ /* Clean up storage */
+ EndCopy(cstate);
}
/*
* Copy from relation or query TO file.
*/
! static uint64
CopyTo(CopyState cstate)
{
TupleDesc tupDesc;
int num_phys_attrs;
Form_pg_attribute *attr;
ListCell *cur;
+ uint64 processed;
if (cstate->rel)
tupDesc = RelationGetDescr(cstate->rel);
*************** CopyTo(CopyState cstate)
*** 1439,1444 ****
--- 1539,1545 ----
scandesc = heap_beginscan(cstate->rel, GetActiveSnapshot(), 0, NULL);
+ processed = 0;
while ((tuple = heap_getnext(scandesc, ForwardScanDirection)) != NULL)
{
CHECK_FOR_INTERRUPTS();
*************** CopyTo(CopyState cstate)
*** 1448,1461 ****
--- 1549,1567 ----
/* Format and send the data */
CopyOneRowTo(cstate, HeapTupleGetOid(tuple), values, nulls);
+ processed++;
}
heap_endscan(scandesc);
+
+ pfree(values);
+ pfree(nulls);
}
else
{
/* run the plan --- the dest receiver will send tuples */
ExecutorRun(cstate->queryDesc, ForwardScanDirection, 0L);
+ processed = ((DR_copy *) cstate->queryDesc->dest)->processed;
}
if (cstate->binary)
*************** CopyTo(CopyState cstate)
*** 1467,1472 ****
--- 1573,1580 ----
}
MemoryContextDelete(cstate->rowcontext);
+
+ return processed;
}
/*
*************** CopyOneRowTo(CopyState cstate, Oid tuple
*** 1558,1573 ****
CopySendEndOfRow(cstate);
MemoryContextSwitchTo(oldcontext);
-
- cstate->processed++;
}
/*
* error context callback for COPY FROM
*/
! static void
! copy_in_error_callback(void *arg)
{
CopyState cstate = (CopyState) arg;
--- 1666,1681 ----
CopySendEndOfRow(cstate);
MemoryContextSwitchTo(oldcontext);
}
/*
* error context callback for COPY FROM
+ *
+ * The argument for the error context must be CopyState.
*/
! void
! CopyFromErrorCallback(void *arg)
{
CopyState cstate = (CopyState) arg;
*************** copy_in_error_callback(void *arg)
*** 1575,1585 ****
{
/* can't usefully display the data */
if (cstate->cur_attname)
! errcontext("COPY %s, line %d, column %s",
cstate->cur_relname, cstate->cur_lineno,
cstate->cur_attname);
else
! errcontext("COPY %s, line %d",
cstate->cur_relname, cstate->cur_lineno);
}
else
--- 1683,1693 ----
{
/* can't usefully display the data */
if (cstate->cur_attname)
! errcontext("relation %s, line %d, column %s",
cstate->cur_relname, cstate->cur_lineno,
cstate->cur_attname);
else
! errcontext("relation %s, line %d",
cstate->cur_relname, cstate->cur_lineno);
}
else
*************** copy_in_error_callback(void *arg)
*** 1590,1596 ****
char *attval;
attval = limit_printout_length(cstate->cur_attval);
! errcontext("COPY %s, line %d, column %s: \"%s\"",
cstate->cur_relname, cstate->cur_lineno,
cstate->cur_attname, attval);
pfree(attval);
--- 1698,1704 ----
char *attval;
attval = limit_printout_length(cstate->cur_attval);
! errcontext("relation %s, line %d, column %s: \"%s\"",
cstate->cur_relname, cstate->cur_lineno,
cstate->cur_attname, attval);
pfree(attval);
*************** copy_in_error_callback(void *arg)
*** 1598,1604 ****
else if (cstate->cur_attname)
{
/* error is relevant to a particular column, value is NULL */
! errcontext("COPY %s, line %d, column %s: null input",
cstate->cur_relname, cstate->cur_lineno,
cstate->cur_attname);
}
--- 1706,1712 ----
else if (cstate->cur_attname)
{
/* error is relevant to a particular column, value is NULL */
! errcontext("relation %s, line %d, column %s: null input",
cstate->cur_relname, cstate->cur_lineno,
cstate->cur_attname);
}
*************** copy_in_error_callback(void *arg)
*** 1610,1616 ****
char *lineval;
lineval = limit_printout_length(cstate->line_buf.data);
! errcontext("COPY %s, line %d: \"%s\"",
cstate->cur_relname, cstate->cur_lineno, lineval);
pfree(lineval);
}
--- 1718,1724 ----
char *lineval;
lineval = limit_printout_length(cstate->line_buf.data);
! errcontext("relation %s, line %d: \"%s\"",
cstate->cur_relname, cstate->cur_lineno, lineval);
pfree(lineval);
}
*************** copy_in_error_callback(void *arg)
*** 1624,1630 ****
* regurgitate it without conversion. So we have to punt and
* just report the line number.
*/
! errcontext("COPY %s, line %d",
cstate->cur_relname, cstate->cur_lineno);
}
}
--- 1732,1738 ----
* regurgitate it without conversion. So we have to punt and
* just report the line number.
*/
! errcontext("relation %s, line %d",
cstate->cur_relname, cstate->cur_lineno);
}
}
*************** limit_printout_length(const char *str)
*** 1669,1709 ****
/*
* Copy FROM file to relation.
*/
! static void
CopyFrom(CopyState cstate)
{
- bool pipe = (cstate->filename == NULL);
HeapTuple tuple;
TupleDesc tupDesc;
- Form_pg_attribute *attr;
- AttrNumber num_phys_attrs,
- attr_count,
- num_defaults;
- FmgrInfo *in_functions;
- FmgrInfo oid_in_function;
- Oid *typioparams;
- Oid oid_typioparam;
- int attnum;
- int i;
- Oid in_func_oid;
Datum *values;
bool *nulls;
- int nfields;
- char **field_strings;
- bool done = false;
- bool isnull;
ResultRelInfo *resultRelInfo;
! EState *estate = CreateExecutorState(); /* for ExecConstraints() */
TupleTableSlot *slot;
- bool file_has_oids;
- int *defmap;
- ExprState **defexprs; /* array of default att expressions */
- ExprContext *econtext; /* used for ExecEvalExpr for default atts */
MemoryContext oldcontext = CurrentMemoryContext;
ErrorContextCallback errcontext;
CommandId mycid = GetCurrentCommandId(true);
int hi_options = 0; /* start with default heap_insert options */
BulkInsertState bistate;
Assert(cstate->rel);
--- 1777,1798 ----
/*
* Copy FROM file to relation.
*/
! static uint64
CopyFrom(CopyState cstate)
{
HeapTuple tuple;
TupleDesc tupDesc;
Datum *values;
bool *nulls;
ResultRelInfo *resultRelInfo;
! EState *estate = cstate->estate; /* for ExecConstraints() */
TupleTableSlot *slot;
MemoryContext oldcontext = CurrentMemoryContext;
ErrorContextCallback errcontext;
CommandId mycid = GetCurrentCommandId(true);
int hi_options = 0; /* start with default heap_insert options */
BulkInsertState bistate;
+ uint64 processed = 0;
Assert(cstate->rel);
*************** CopyFrom(CopyState cstate)
*** 1731,1736 ****
--- 1820,1827 ----
RelationGetRelationName(cstate->rel))));
}
+ tupDesc = RelationGetDescr(cstate->rel);
+
/*----------
* Check to see if we can avoid writing WAL
*
*************** CopyFrom(CopyState cstate)
*** 1766,1803 ****
hi_options |= HEAP_INSERT_SKIP_WAL;
}
- if (pipe)
- {
- if (whereToSendOutput == DestRemote)
- ReceiveCopyBegin(cstate);
- else
- cstate->copy_file = stdin;
- }
- else
- {
- struct stat st;
-
- cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_R);
-
- if (cstate->copy_file == NULL)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not open file \"%s\" for reading: %m",
- cstate->filename)));
-
- fstat(fileno(cstate->copy_file), &st);
- if (S_ISDIR(st.st_mode))
- ereport(ERROR,
- (errcode(ERRCODE_WRONG_OBJECT_TYPE),
- errmsg("\"%s\" is a directory", cstate->filename)));
- }
-
- tupDesc = RelationGetDescr(cstate->rel);
- attr = tupDesc->attrs;
- num_phys_attrs = tupDesc->natts;
- attr_count = list_length(cstate->attnumlist);
- num_defaults = 0;
-
/*
* We need a ResultRelInfo so we can use the regular executor's
* index-entry-making machinery. (There used to be a huge amount of code
--- 1857,1862 ----
*************** CopyFrom(CopyState cstate)
*** 1826,1832 ****
slot = ExecInitExtraTupleSlot(estate);
ExecSetSlotDescriptor(slot, tupDesc);
! econtext = GetPerTupleExprContext(estate);
/*
* Pick up the required catalog information for each attribute in the
--- 1885,2071 ----
slot = ExecInitExtraTupleSlot(estate);
ExecSetSlotDescriptor(slot, tupDesc);
! /* Prepare to catch AFTER triggers. */
! AfterTriggerBeginQuery();
!
! /*
! * Check BEFORE STATEMENT insertion triggers. It's debateable whether we
! * should do this for COPY, since it's not really an "INSERT" statement as
! * such. However, executing these triggers maintains consistency with the
! * EACH ROW triggers that we already fire on COPY.
! */
! ExecBSInsertTriggers(estate, resultRelInfo);
!
! values = (Datum *) palloc(tupDesc->natts * sizeof(Datum));
! nulls = (bool *) palloc(tupDesc->natts * sizeof(bool));
!
! bistate = GetBulkInsertState();
!
! /* Set up callback to identify error line number */
! errcontext.callback = CopyFromErrorCallback;
! errcontext.arg = (void *) cstate;
! errcontext.previous = error_context_stack;
! error_context_stack = &errcontext;
!
! for (;;)
! {
! bool skip_tuple;
! Oid loaded_oid = InvalidOid;
!
! CHECK_FOR_INTERRUPTS();
!
! /* Reset the per-tuple exprcontext */
! ResetPerTupleExprContext(estate);
!
! /* Switch into its memory context */
! MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
!
! if (!NextCopyFrom(cstate, values, nulls, &loaded_oid))
! break;
!
! /* And now we can form the input tuple. */
! tuple = heap_form_tuple(tupDesc, values, nulls);
!
! if (loaded_oid != InvalidOid)
! HeapTupleSetOid(tuple, loaded_oid);
!
! /* Triggers and stuff need to be invoked in query context. */
! MemoryContextSwitchTo(oldcontext);
!
! skip_tuple = false;
!
! /* BEFORE ROW INSERT Triggers */
! if (resultRelInfo->ri_TrigDesc &&
! resultRelInfo->ri_TrigDesc->trig_insert_before_row)
! {
! HeapTuple newtuple;
!
! newtuple = ExecBRInsertTriggers(estate, resultRelInfo, tuple);
!
! if (newtuple == NULL) /* "do nothing" */
! skip_tuple = true;
! else if (newtuple != tuple) /* modified by Trigger(s) */
! {
! heap_freetuple(tuple);
! tuple = newtuple;
! }
! }
!
! if (!skip_tuple)
! {
! List *recheckIndexes = NIL;
!
! /* Place tuple in tuple slot */
! ExecStoreTuple(tuple, slot, InvalidBuffer, false);
!
! /* Check the constraints of the tuple */
! if (cstate->rel->rd_att->constr)
! ExecConstraints(resultRelInfo, slot, estate);
!
! /* OK, store the tuple and create index entries for it */
! heap_insert(cstate->rel, tuple, mycid, hi_options, bistate);
!
! if (resultRelInfo->ri_NumIndices > 0)
! recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self),
! estate);
!
! /* AFTER ROW INSERT Triggers */
! ExecARInsertTriggers(estate, resultRelInfo, tuple,
! recheckIndexes);
!
! list_free(recheckIndexes);
!
! /*
! * We count only tuples not suppressed by a BEFORE INSERT trigger;
! * this is the same definition used by execMain.c for counting
! * tuples inserted by an INSERT command.
! */
! processed++;
! }
! }
!
! /* Done, clean up */
! error_context_stack = errcontext.previous;
!
! FreeBulkInsertState(bistate);
!
! MemoryContextSwitchTo(oldcontext);
!
! /* Execute AFTER STATEMENT insertion triggers */
! ExecASInsertTriggers(estate, resultRelInfo);
!
! /* Handle queued AFTER triggers */
! AfterTriggerEndQuery(estate);
!
! pfree(values);
! pfree(nulls);
!
! ExecResetTupleTable(estate->es_tupleTable, false);
!
! ExecCloseIndices(resultRelInfo);
!
! /*
! * If we skipped writing WAL, then we need to sync the heap (but not
! * indexes since those use WAL anyway)
! */
! if (hi_options & HEAP_INSERT_SKIP_WAL)
! heap_sync(cstate->rel);
!
! return processed;
! }
!
! /*
! * CopyGetAttnums - build an integer list of attnums to be copied
! *
! * The input attnamelist is either the user-specified column list,
! * or NIL if there was none (in which case we want all the non-dropped
! * columns).
! *
! * rel can be NULL ... it's only used for error reports.
! */
! CopyState
! BeginCopyFrom(Relation rel,
! const char *filename,
! List *attnamelist,
! List *options)
! {
! CopyState cstate;
! bool pipe = (filename == NULL);
! TupleDesc tupDesc;
! Form_pg_attribute *attr;
! AttrNumber num_phys_attrs,
! num_defaults;
! FmgrInfo *in_functions;
! Oid *typioparams;
! int attnum;
! Oid in_func_oid;
! EState *estate = CreateExecutorState(); /* for ExecPrepareExpr() */
! int *defmap;
! ExprState **defexprs;
! MemoryContext oldcontext;
!
! cstate = BeginCopy(true, rel, NULL, NULL, attnamelist, options);
! oldcontext = MemoryContextSwitchTo(cstate->copycontext);
!
! /* Initialize state variables */
! cstate->fe_eof = false;
! cstate->eol_type = EOL_UNKNOWN;
! cstate->cur_relname = RelationGetRelationName(cstate->rel);
! cstate->cur_lineno = 0;
! cstate->cur_attname = NULL;
! cstate->cur_attval = NULL;
!
! /* Set up variables to avoid per-attribute overhead. */
! initStringInfo(&cstate->attribute_buf);
! initStringInfo(&cstate->line_buf);
! cstate->line_buf_converted = false;
! cstate->raw_buf = (char *) palloc(RAW_BUF_SIZE + 1);
! cstate->raw_buf_index = cstate->raw_buf_len = 0;
!
! tupDesc = RelationGetDescr(cstate->rel);
! attr = tupDesc->attrs;
! num_phys_attrs = tupDesc->natts;
! num_defaults = 0;
/*
* Pick up the required catalog information for each attribute in the
*************** CopyFrom(CopyState cstate)
*** 1871,1889 ****
}
}
! /* Prepare to catch AFTER triggers. */
! AfterTriggerBeginQuery();
! /*
! * Check BEFORE STATEMENT insertion triggers. It's debateable whether we
! * should do this for COPY, since it's not really an "INSERT" statement as
! * such. However, executing these triggers maintains consistency with the
! * EACH ROW triggers that we already fire on COPY.
! */
! ExecBSInsertTriggers(estate, resultRelInfo);
if (!cstate->binary)
! file_has_oids = cstate->oids; /* must rely on user to tell us... */
else
{
/* Read and verify binary header */
--- 2110,2155 ----
}
}
! /* We keep those variables in cstate. */
! cstate->estate = estate;
! cstate->in_functions = in_functions;
! cstate->typioparams = typioparams;
! cstate->defmap = defmap;
! cstate->defexprs = defexprs;
! cstate->num_defaults = num_defaults;
! if (pipe)
! {
! if (whereToSendOutput == DestRemote)
! ReceiveCopyBegin(cstate);
! else
! cstate->copy_file = stdin;
! }
! else
! {
! struct stat st;
!
! cstate->filename = pstrdup(filename);
! cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_R);
!
! if (cstate->copy_file == NULL)
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not open file \"%s\" for reading: %m",
! cstate->filename)));
!
! fstat(fileno(cstate->copy_file), &st);
! if (S_ISDIR(st.st_mode))
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("\"%s\" is a directory", cstate->filename)));
! }
if (!cstate->binary)
! {
! /* must rely on user to tell us... */
! cstate->file_has_oids = cstate->oids;
! }
else
{
/* Read and verify binary header */
*************** CopyFrom(CopyState cstate)
*** 1901,1907 ****
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid COPY file header (missing flags)")));
! file_has_oids = (tmp & (1 << 16)) != 0;
tmp &= ~(1 << 16);
if ((tmp >> 16) != 0)
ereport(ERROR,
--- 2167,2173 ----
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid COPY file header (missing flags)")));
! cstate->file_has_oids = (tmp & (1 << 16)) != 0;
tmp &= ~(1 << 16);
if ((tmp >> 16) != 0)
ereport(ERROR,
*************** CopyFrom(CopyState cstate)
*** 1923,1984 ****
}
}
! if (file_has_oids && cstate->binary)
{
getTypeBinaryInputInfo(OIDOID,
! &in_func_oid, &oid_typioparam);
! fmgr_info(in_func_oid, &oid_in_function);
}
- values = (Datum *) palloc(num_phys_attrs * sizeof(Datum));
- nulls = (bool *) palloc(num_phys_attrs * sizeof(bool));
-
/* create workspace for CopyReadAttributes results */
! nfields = file_has_oids ? (attr_count + 1) : attr_count;
! if (! cstate->binary)
{
cstate->max_fields = nfields;
cstate->raw_fields = (char **) palloc(nfields * sizeof(char *));
}
! /* Initialize state variables */
! cstate->fe_eof = false;
! cstate->eol_type = EOL_UNKNOWN;
! cstate->cur_relname = RelationGetRelationName(cstate->rel);
! cstate->cur_lineno = 0;
! cstate->cur_attname = NULL;
! cstate->cur_attval = NULL;
! bistate = GetBulkInsertState();
! /* Set up callback to identify error line number */
! errcontext.callback = copy_in_error_callback;
! errcontext.arg = (void *) cstate;
! errcontext.previous = error_context_stack;
! error_context_stack = &errcontext;
/* on input just throw the header line away */
! if (cstate->header_line)
{
cstate->cur_lineno++;
! done = CopyReadLine(cstate);
}
! while (!done)
! {
! bool skip_tuple;
! Oid loaded_oid = InvalidOid;
!
! CHECK_FOR_INTERRUPTS();
cstate->cur_lineno++;
- /* Reset the per-tuple exprcontext */
- ResetPerTupleExprContext(estate);
-
- /* Switch into its memory context */
- MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
-
/* Initialize all values for row to NULL */
MemSet(values, 0, num_phys_attrs * sizeof(Datum));
MemSet(nulls, true, num_phys_attrs * sizeof(bool));
--- 2189,2259 ----
}
}
! if (cstate->file_has_oids && cstate->binary)
{
getTypeBinaryInputInfo(OIDOID,
! &in_func_oid, &cstate->oid_typioparam);
! fmgr_info(in_func_oid, &cstate->oid_in_function);
}
/* create workspace for CopyReadAttributes results */
! if (!cstate->binary)
{
+ AttrNumber attr_count = list_length(cstate->attnumlist);
+ int nfields = cstate->file_has_oids ? (attr_count + 1) : attr_count;
+
cstate->max_fields = nfields;
cstate->raw_fields = (char **) palloc(nfields * sizeof(char *));
}
! MemoryContextSwitchTo(oldcontext);
! return cstate;
! }
! /*
! * Read next tuple from file for COPY FROM. Return false if no more tuples.
! *
! * values and nulls arrays must be the same length as columns of the
! * relation passed to BeginCopyFrom. Oid of the tuple is returned with
! * tupleOid separately.
! */
! bool
! NextCopyFrom(CopyState cstate, Datum *values, bool *nulls, Oid *tupleOid)
! {
! TupleDesc tupDesc;
! Form_pg_attribute *attr;
! AttrNumber num_phys_attrs,
! attr_count,
! num_defaults = cstate->num_defaults;
! FmgrInfo *in_functions = cstate->in_functions;
! Oid *typioparams = cstate->typioparams;
! int i;
! int nfields;
! char **field_strings;
! bool isnull;
! bool file_has_oids = cstate->file_has_oids;
! int *defmap = cstate->defmap;
! ExprState **defexprs = cstate->defexprs;
! ExprContext *econtext; /* used for ExecEvalExpr for default atts */
/* on input just throw the header line away */
! if (cstate->cur_lineno == 0 && cstate->header_line)
{
cstate->cur_lineno++;
! if (CopyReadLine(cstate))
! return false; /* done */
}
! tupDesc = RelationGetDescr(cstate->rel);
! attr = tupDesc->attrs;
! num_phys_attrs = tupDesc->natts;
! attr_count = list_length(cstate->attnumlist);
! nfields = file_has_oids ? (attr_count + 1) : attr_count;
+ /* XXX: Indentation is not adjusted to keep the patch small. */
cstate->cur_lineno++;
/* Initialize all values for row to NULL */
MemSet(values, 0, num_phys_attrs * sizeof(Datum));
MemSet(nulls, true, num_phys_attrs * sizeof(bool));
*************** CopyFrom(CopyState cstate)
*** 1989,1994 ****
--- 2264,2270 ----
int fldct;
int fieldno;
char *string;
+ bool done;
/* Actually read the line into memory here */
done = CopyReadLine(cstate);
*************** CopyFrom(CopyState cstate)
*** 1999,2005 ****
* EOF, ie, process the line and then exit loop on next iteration.
*/
if (done && cstate->line_buf.len == 0)
! break;
/* Parse the line into de-escaped field values */
if (cstate->csv_mode)
--- 2275,2281 ----
* EOF, ie, process the line and then exit loop on next iteration.
*/
if (done && cstate->line_buf.len == 0)
! return false;
/* Parse the line into de-escaped field values */
if (cstate->csv_mode)
*************** CopyFrom(CopyState cstate)
*** 2029,2041 ****
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("null OID in COPY data")));
! else
{
cstate->cur_attname = "oid";
cstate->cur_attval = string;
! loaded_oid = DatumGetObjectId(DirectFunctionCall1(oidin,
! CStringGetDatum(string)));
! if (loaded_oid == InvalidOid)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid OID in COPY data")));
--- 2305,2317 ----
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("null OID in COPY data")));
! else if (cstate->oids && tupleOid != NULL)
{
cstate->cur_attname = "oid";
cstate->cur_attval = string;
! *tupleOid = DatumGetObjectId(DirectFunctionCall1(oidin,
! CStringGetDatum(string)));
! if (*tupleOid == InvalidOid)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid OID in COPY data")));
*************** CopyFrom(CopyState cstate)
*** 2087,2094 ****
if (!CopyGetInt16(cstate, &fld_count))
{
/* EOF detected (end of file, or protocol-level EOF) */
! done = true;
! break;
}
if (fld_count == -1)
--- 2363,2369 ----
if (!CopyGetInt16(cstate, &fld_count))
{
/* EOF detected (end of file, or protocol-level EOF) */
! return false;
}
if (fld_count == -1)
*************** CopyFrom(CopyState cstate)
*** 2112,2119 ****
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("received copy data after EOF marker")));
! done = true;
! break;
}
if (fld_count != attr_count)
--- 2387,2393 ----
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("received copy data after EOF marker")));
! return false;
}
if (fld_count != attr_count)
*************** CopyFrom(CopyState cstate)
*** 2124,2135 ****
if (file_has_oids)
{
cstate->cur_attname = "oid";
loaded_oid =
DatumGetObjectId(CopyReadBinaryAttribute(cstate,
0,
! &oid_in_function,
! oid_typioparam,
-1,
&isnull));
if (isnull || loaded_oid == InvalidOid)
--- 2398,2411 ----
if (file_has_oids)
{
+ Oid loaded_oid;
+
cstate->cur_attname = "oid";
loaded_oid =
DatumGetObjectId(CopyReadBinaryAttribute(cstate,
0,
! &cstate->oid_in_function,
! cstate->oid_typioparam,
-1,
&isnull));
if (isnull || loaded_oid == InvalidOid)
*************** CopyFrom(CopyState cstate)
*** 2137,2142 ****
--- 2413,2420 ----
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid OID in COPY data")));
cstate->cur_attname = NULL;
+ if (cstate->oids && tupleOid != NULL)
+ *tupleOid = loaded_oid;
}
i = 0;
*************** CopyFrom(CopyState cstate)
*** 2162,2278 ****
* provided by the input data. Anything not processed here or above
* will remain NULL.
*/
for (i = 0; i < num_defaults; i++)
{
values[defmap[i]] = ExecEvalExpr(defexprs[i], econtext,
&nulls[defmap[i]], NULL);
}
! /* And now we can form the input tuple. */
! tuple = heap_form_tuple(tupDesc, values, nulls);
!
! if (cstate->oids && file_has_oids)
! HeapTupleSetOid(tuple, loaded_oid);
!
! /* Triggers and stuff need to be invoked in query context. */
! MemoryContextSwitchTo(oldcontext);
!
! skip_tuple = false;
!
! /* BEFORE ROW INSERT Triggers */
! if (resultRelInfo->ri_TrigDesc &&
! resultRelInfo->ri_TrigDesc->trig_insert_before_row)
! {
! HeapTuple newtuple;
!
! newtuple = ExecBRInsertTriggers(estate, resultRelInfo, tuple);
!
! if (newtuple == NULL) /* "do nothing" */
! skip_tuple = true;
! else if (newtuple != tuple) /* modified by Trigger(s) */
! {
! heap_freetuple(tuple);
! tuple = newtuple;
! }
! }
!
! if (!skip_tuple)
! {
! List *recheckIndexes = NIL;
!
! /* Place tuple in tuple slot */
! ExecStoreTuple(tuple, slot, InvalidBuffer, false);
!
! /* Check the constraints of the tuple */
! if (cstate->rel->rd_att->constr)
! ExecConstraints(resultRelInfo, slot, estate);
!
! /* OK, store the tuple and create index entries for it */
! heap_insert(cstate->rel, tuple, mycid, hi_options, bistate);
!
! if (resultRelInfo->ri_NumIndices > 0)
! recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self),
! estate);
!
! /* AFTER ROW INSERT Triggers */
! ExecARInsertTriggers(estate, resultRelInfo, tuple,
! recheckIndexes);
!
! list_free(recheckIndexes);
!
! /*
! * We count only tuples not suppressed by a BEFORE INSERT trigger;
! * this is the same definition used by execMain.c for counting
! * tuples inserted by an INSERT command.
! */
! cstate->processed++;
! }
! }
!
! /* Done, clean up */
! error_context_stack = errcontext.previous;
!
! FreeBulkInsertState(bistate);
!
! MemoryContextSwitchTo(oldcontext);
!
! /* Execute AFTER STATEMENT insertion triggers */
! ExecASInsertTriggers(estate, resultRelInfo);
!
! /* Handle queued AFTER triggers */
! AfterTriggerEndQuery(estate);
!
! pfree(values);
! pfree(nulls);
! if (! cstate->binary)
! pfree(cstate->raw_fields);
!
! pfree(in_functions);
! pfree(typioparams);
! pfree(defmap);
! pfree(defexprs);
!
! ExecResetTupleTable(estate->es_tupleTable, false);
!
! ExecCloseIndices(resultRelInfo);
!
! FreeExecutorState(estate);
! if (!pipe)
! {
! if (FreeFile(cstate->copy_file))
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not close file \"%s\": %m",
! cstate->filename)));
! }
! /*
! * If we skipped writing WAL, then we need to sync the heap (but not
! * indexes since those use WAL anyway)
! */
! if (hi_options & HEAP_INSERT_SKIP_WAL)
! heap_sync(cstate->rel);
}
--- 2440,2466 ----
* provided by the input data. Anything not processed here or above
* will remain NULL.
*/
+ econtext = GetPerTupleExprContext(cstate->estate);
for (i = 0; i < num_defaults; i++)
{
values[defmap[i]] = ExecEvalExpr(defexprs[i], econtext,
&nulls[defmap[i]], NULL);
}
+ /* XXX: End of only-indentation changes. */
! return true;
! }
! /*
! * Clean up storage and release resources for COPY FROM.
! */
! void
! EndCopyFrom(CopyState cstate)
! {
! FreeExecutorState(cstate->estate);
! /* Clean up storage */
! EndCopy(cstate);
}
*************** copy_dest_receive(TupleTableSlot *slot,
*** 3537,3542 ****
--- 3725,3731 ----
/* And send the data */
CopyOneRowTo(cstate, InvalidOid, slot->tts_values, slot->tts_isnull);
+ myState->processed++;
}
/*
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index 9e2bbe8..8340e3d 100644
*** a/src/include/commands/copy.h
--- b/src/include/commands/copy.h
***************
*** 14,25 ****
--- 14,35 ----
#ifndef COPY_H
#define COPY_H
+ #include "nodes/execnodes.h"
#include "nodes/parsenodes.h"
#include "tcop/dest.h"
+ typedef struct CopyStateData *CopyState;
+
extern uint64 DoCopy(const CopyStmt *stmt, const char *queryString);
+ extern CopyState BeginCopyFrom(Relation rel, const char *filename,
+ List *attnamelist, List *options);
+ extern void EndCopyFrom(CopyState cstate);
+ extern bool NextCopyFrom(CopyState cstate,
+ Datum *values, bool *nulls, Oid *tupleOid);
+ extern void CopyFromErrorCallback(void *arg);
+
extern DestReceiver *CreateCopyDestReceiver(void);
#endif /* COPY_H */
On Mon, Feb 07, 2011 at 03:39:39PM +0900, Itagaki Takahiro wrote:
On Sun, Feb 6, 2011 at 09:01, Noah Misch <noah@leadboat.com> wrote:
Most "parse analysis"-type bits of DoCopy() move to BeginCopy().
It would be possible to move more FROM-only or TO-only members in BeginCopy()
into their BeginCopyFrom/To(), but it is just a code refactoring requires
more code rearrangement. It should be done by another try even if needed.
Agreed.
CopyState.processed is gone, with the row count now bubbling up
through return values.I removed it from CopyState because it represents "number of inserted rows"
in COPY FROM. However, for the viewpoint of API, the numbers of lines in
the input file is more suitable. To avoid the confusion, I moved it
from a common member to COPY FROM-only variable.
Perhaps I'm missing something. The new API does not expose a "processed" count
at all; that count is used for the command tag of a top-level COPY. This part
of the patch is just changing how we structure the code to maintain that tally,
and it has no implications for observers outside copy.c. Right?
For COPY TO, the relkind checks move from DoCopyTo() to BeginCopyTo(). ??I'm
skeptical about this one. ??It's not required for correctness, and the relkind
checks for COPY FROM remain in CopyFrom().I think error checks should be done in the early phase, so I moved the check
to BeginCopyTo(). However, relkind check in COPY FROM is needed only for
COPY FROM command because the relation is the inserted target. For APIs,
the relation is used as a template for input file. So, we cannot perform
the check in BeginCopyFrom().
The choice of where to put them does not affect anything outside of copy.c, and
placement in DoCopyTo() would make the symmetry between the TO and FROM code
paths easier to follow. Not a major concern, though.
file_fdw uses CopyFromErrorCallback() to give errors the proper context. ??The
function uses template strings like "COPY %s, line %d", where %s is the name of
the relation being copied. ??Presumably file_fdw and other features using this
API would wish to customize that error message prefix, and the relation name
might not be apropos at all. ??How about another argument to BeginCopyFrom,
specifying an error prefix to be stashed in the CopyState?I changed "COPY %s, .." to "relation %s, ..." because the first string is
the relation name anyway. We could have another prefix argument, but I think
it has little information for errors.
That's perhaps an improvement for file_fdw, but not for regular COPY.
My comment originated with a faulty idea that file_fdw's internal use of COPY
was an implementation detail that users should not need to see. Looking now,
the file_fdw documentation clearly explains the tie to COPY, even referring
users to the COPY documentation. I no longer see a need to hide the fact that
the "foreign" source is a PostgreSQL COPY command. The error messages are fine
as they were.
Some future client of the new API may wish to hide its internal COPY use, but
there's no need to design for that now.
We also have many "COPY" in other messages, but they won't be used actually
because the are messages for invalid arguments and file_fdw will have own
validater function. All invalid arguments will be filtered in CREATE commands.
Agreed; "could not read from COPY file: %m" appears to be the primary one liable
to happen in practice. The greater failure with that one is that, given a query
reading from multiple file_fdw tables, you can't tell which file had a problem.
This issue runs broader than the patch at hand; I will start another thread to
address it. Let's proceed with this patch, not changing any error messages. If
other discussion concludes that the desired behavior requires an enhancement to
this new API, a followup commit can implement that.
! ?? ?? ?? ?? ?? ?? /* Reset the per-tuple exprcontext */
! ?? ?? ?? ?? ?? ?? ResetPerTupleExprContext(estate);
!
! ?? ?? ?? ?? ?? ?? /* Switch into its memory context */
! ?? ?? ?? ?? ?? ?? MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));Shouldn't a switch to this context happen inside NextCopyFrom(), then again for
the calls to heap_form_tuple/HeapTupleSetOid? ??I found previous discussion on
this matter, but no conclusion.In my understanding, NextCopyFrom() should use CurrentMemoryContext provided
by the caller. The file_fdw will use executor's per-tuple context for it.
In a direct call to COPY FROM, all of NextCopyFrom() uses the per-tuple context
of CopyState->estate. We reset that context before each call to NextCopyFrom().
This is true before (ignoring code movement) and after the patch.
A file_fdw NextCopyFrom() call will use the per-tuple context of the executor
performing a foreign scan. Allocations will arise primarily in type input
functions. ExecEvalExpr(), used to acquire default values, will still use the
per-tuple context of CopyState->estate. That per-tuple context will never get
reset explicitly, so default value computations leak until EndCopyFrom().
I see memory context use was discussed already, but I don't see the
aforementioned specific details addressed:
http://archives.postgresql.org/message-id/AANLkTikfiM1qknr9=tL+xemBLAJ+YJoLhAG3XsH7mfwH@mail.gmail.com
The use of ExecEvalExpr() in NextCopyFrom() also seems contrary to the
execnodes.h comment you quoted in that thread ("CurrentMemoryContext should be
set to ecxt_per_tuple_memory before calling ExecEvalExpr() --- see
ExecEvalExprSwitchContext()."). NextCopyFrom() passes CopyState->estate to
ExecEvalExpr, but the current context may be the per-tuple context of a
different executor state.
Another issue in the automatic context switch is to discard the previous
values in the next call while the caller is unaware.
True; we would need to document that pointers stored in the "values" argument of
NextCopyData() are valid only until the next call. Does that already work with
file_fdw's usage pattern, or would file_fdw need to make copies?
Three hunk-specific comments (diff is between previously-reviewed copy.c and
current master plus today's patch):
*** a/src/backend/commands/copy.c --- b/src/backend/commands/copy.c *************** *** 175,193 **** typedef struct CopyStateData * The definition of input functions and default expressions are stored * in these variables. */ EState *estate; AttrNumber num_defaults; bool file_has_oids; FmgrInfo oid_in_function; Oid oid_typioparam; ! FmgrInfo *in_functions; ! Oid *typioparams; ! int *defmap; ExprState **defexprs; /* array of default att expressions */ } CopyStateData;/* DestReceiver for COPY (SELECT) TO */ typedef struct { DestReceiver pub; /* publicly-known function pointers */ CopyState cstate; /* CopyStateData for the command */ --- 175,193 ---- * The definition of input functions and default expressions are stored * in these variables. */
This block comment remains roughly half correct. Again, I think a small comment
on every field below should replace it.
EState *estate;
AttrNumber num_defaults;
bool file_has_oids;
FmgrInfo oid_in_function;
Oid oid_typioparam;
! FmgrInfo *in_functions; /* array of input functions for each attrs */
! Oid *typioparams; /* array of element types for in_functions */
! int *defmap; /* array of default att numbers */
ExprState **defexprs; /* array of default att expressions */
} CopyStateData;/* DestReceiver for COPY (SELECT) TO */ typedef struct { DestReceiver pub; /* publicly-known function pointers */ CopyState cstate; /* CopyStateData for the command */ *************** *** 1268,1283 **** BeginCopy(bool is_from, --- 1268,1289 ---- }/* * Release resources allocated in a cstate. */ static void EndCopy(CopyState cstate) { + if (cstate->filename != NULL && FreeFile(cstate->copy_file)) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not close file \"%s\": %m", + cstate->filename)));
In the write case, this is not an improvement -- failure in fclose(3) almost
always arises from the attempt to flush buffered writes. Note how most other
call sites checking the return value of FreeFile() use an error message like the
original one. The change to the read case seems fine.
+
MemoryContextDelete(cstate->copycontext);
pfree(cstate);
}/*
* Setup CopyState to read tuples from a table or a query for COPY TO.
*/
static CopyState
***************
*** 1400,1424 **** DoCopyTo(CopyState cstate)
}/*
* Clean up storage and release resources for COPY TO.
*/
static void
EndCopyTo(CopyState cstate)
{
- if (cstate->filename != NULL && FreeFile(cstate->copy_file))
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not close file \"%s\": %m",
- cstate->filename)));
-
- /*
- * Close the relation or query. We can release the AccessShareLock we got.
- */
if (cstate->queryDesc != NULL)
{
/* Close down the query and free resources. */
ExecutorEnd(cstate->queryDesc);
FreeQueryDesc(cstate->queryDesc);
PopActiveSnapshot();
}--- 1406,1421 ---- *************** *** 1681,1746 **** void CopyFromErrorCallback(void *arg) { CopyState cstate = (CopyState) arg;if (cstate->binary)
{
/* can't usefully display the data */
if (cstate->cur_attname)
! errcontext("COPY %s, line %d, column %s",
cstate->cur_relname, cstate->cur_lineno,
cstate->cur_attname);
else
! errcontext("COPY %s, line %d",
cstate->cur_relname, cstate->cur_lineno);
}
else
{
if (cstate->cur_attname && cstate->cur_attval)
{
/* error is relevant to a particular column */
char *attval;attval = limit_printout_length(cstate->cur_attval);
! errcontext("COPY %s, line %d, column %s: \"%s\"",
cstate->cur_relname, cstate->cur_lineno,
cstate->cur_attname, attval);
pfree(attval);
}
else if (cstate->cur_attname)
{
/* error is relevant to a particular column, value is NULL */
! errcontext("COPY %s, line %d, column %s: null input",
cstate->cur_relname, cstate->cur_lineno,
cstate->cur_attname);
}
else
{
/* error is relevant to a particular line */
if (cstate->line_buf_converted || !cstate->need_transcoding)
{
char *lineval;lineval = limit_printout_length(cstate->line_buf.data);
! errcontext("COPY %s, line %d: \"%s\"",
cstate->cur_relname, cstate->cur_lineno, lineval);
pfree(lineval);
}
else
{
/*
* Here, the line buffer is still in a foreign encoding, and
* indeed it's quite likely that the error is precisely a
* failure to do encoding conversion (ie, bad data). We dare
* not try to convert it, and at present there's no way to
* regurgitate it without conversion. So we have to punt and
* just report the line number.
*/
! errcontext("COPY %s, line %d",
cstate->cur_relname, cstate->cur_lineno);
}
}
}
}/* * Make sure we don't print an unreasonable amount of COPY data in a message. --- 1678,1743 ---- CopyFromErrorCallback(void *arg) { CopyState cstate = (CopyState) arg;if (cstate->binary)
{
/* can't usefully display the data */
if (cstate->cur_attname)
! errcontext("relation %s, line %d, column %s",
cstate->cur_relname, cstate->cur_lineno,
cstate->cur_attname);
else
! errcontext("relation %s, line %d",
cstate->cur_relname, cstate->cur_lineno);
}
else
{
if (cstate->cur_attname && cstate->cur_attval)
{
/* error is relevant to a particular column */
char *attval;attval = limit_printout_length(cstate->cur_attval);
! errcontext("relation %s, line %d, column %s: \"%s\"",
cstate->cur_relname, cstate->cur_lineno,
cstate->cur_attname, attval);
pfree(attval);
}
else if (cstate->cur_attname)
{
/* error is relevant to a particular column, value is NULL */
! errcontext("relation %s, line %d, column %s: null input",
cstate->cur_relname, cstate->cur_lineno,
cstate->cur_attname);
}
else
{
/* error is relevant to a particular line */
if (cstate->line_buf_converted || !cstate->need_transcoding)
{
char *lineval;lineval = limit_printout_length(cstate->line_buf.data);
! errcontext("relation %s, line %d: \"%s\"",
cstate->cur_relname, cstate->cur_lineno, lineval);
pfree(lineval);
}
else
{
/*
* Here, the line buffer is still in a foreign encoding, and
* indeed it's quite likely that the error is precisely a
* failure to do encoding conversion (ie, bad data). We dare
* not try to convert it, and at present there's no way to
* regurgitate it without conversion. So we have to punt and
* just report the line number.
*/
! errcontext("relation %s, line %d",
cstate->cur_relname, cstate->cur_lineno);
}
}
}
}/* * Make sure we don't print an unreasonable amount of COPY data in a message. *************** *** 1782,1798 **** limit_printout_length(const char *str) */ static uint64 CopyFrom(CopyState cstate) { HeapTuple tuple; TupleDesc tupDesc; Datum *values; bool *nulls; - bool done = false; ResultRelInfo *resultRelInfo; EState *estate = cstate->estate; /* for ExecConstraints() */ TupleTableSlot *slot; MemoryContext oldcontext = CurrentMemoryContext; ErrorContextCallback errcontext; CommandId mycid = GetCurrentCommandId(true); int hi_options = 0; /* start with default heap_insert options */ BulkInsertState bistate; --- 1779,1794 ---- *************** *** 1906,1936 **** CopyFrom(CopyState cstate) bistate = GetBulkInsertState();/* Set up callback to identify error line number */
errcontext.callback = CopyFromErrorCallback;
errcontext.arg = (void *) cstate;
errcontext.previous = error_context_stack;
error_context_stack = &errcontext;! while (!done)
{
bool skip_tuple;
Oid loaded_oid = InvalidOid;CHECK_FOR_INTERRUPTS();
/* Reset the per-tuple exprcontext */
ResetPerTupleExprContext(estate);/* Switch into its memory context */
MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));! done = !NextCopyFrom(cstate, values, nulls, &loaded_oid);
! if (done)
break;/* And now we can form the input tuple. */
tuple = heap_form_tuple(tupDesc, values, nulls);if (loaded_oid != InvalidOid)
HeapTupleSetOid(tuple, loaded_oid);--- 1902,1931 ---- bistate = GetBulkInsertState();/* Set up callback to identify error line number */
errcontext.callback = CopyFromErrorCallback;
errcontext.arg = (void *) cstate;
errcontext.previous = error_context_stack;
error_context_stack = &errcontext;! for (;;)
{
bool skip_tuple;
Oid loaded_oid = InvalidOid;CHECK_FOR_INTERRUPTS();
/* Reset the per-tuple exprcontext */
ResetPerTupleExprContext(estate);/* Switch into its memory context */
MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));! if (!NextCopyFrom(cstate, values, nulls, &loaded_oid))
break;/* And now we can form the input tuple. */
tuple = heap_form_tuple(tupDesc, values, nulls);if (loaded_oid != InvalidOid)
HeapTupleSetOid(tuple, loaded_oid);***************
*** 2015,2031 **** CopyFrom(CopyState cstate)
*/
if (hi_options & HEAP_INSERT_SKIP_WAL)
heap_sync(cstate->rel);return processed;
}/* ! * Setup CopyState to read tuples from a file for COPY FROM. */ CopyState BeginCopyFrom(Relation rel, const char *filename, List *attnamelist, List *options) { CopyState cstate; --- 2010,2032 ---- */ if (hi_options & HEAP_INSERT_SKIP_WAL) heap_sync(cstate->rel);return processed;
}/*
! * CopyGetAttnums - build an integer list of attnums to be copied
! *
! * The input attnamelist is either the user-specified column list,
! * or NIL if there was none (in which case we want all the non-dropped
! * columns).
! *
! * rel can be NULL ... it's only used for error reports.
*/
CopyState
BeginCopyFrom(Relation rel,
This is just a verbatim copy of the CopyGetAttnums() header comment. (The
middle paragraph happens to be true, though.)
Show quoted text
const char *filename,
List *attnamelist,
List *options)
{
CopyState cstate;
***************
*** 2208,2224 **** BeginCopyFrom(Relation rel,
MemoryContextSwitchTo(oldcontext);return cstate;
}/* * Read next tuple from file for COPY FROM. Return false if no more tuples. * ! * valus and nulls arrays must be the same length as columns of the * relation passed to BeginCopyFrom. Oid of the tuple is returned with * tupleOid separately. */ bool NextCopyFrom(CopyState cstate, Datum *values, bool *nulls, Oid *tupleOid) { TupleDesc tupDesc; Form_pg_attribute *attr; --- 2209,2225 ---- MemoryContextSwitchTo(oldcontext);return cstate;
}/*
* Read next tuple from file for COPY FROM. Return false if no more tuples.
*
! * values and nulls arrays must be the same length as columns of the
* relation passed to BeginCopyFrom. Oid of the tuple is returned with
* tupleOid separately.
*/
bool
NextCopyFrom(CopyState cstate, Datum *values, bool *nulls, Oid *tupleOid)
{
TupleDesc tupDesc;
Form_pg_attribute *attr;
***************
*** 2453,2474 **** NextCopyFrom(CopyState cstate, Datum *values, bool *nulls, Oid *tupleOid)
/*
* Clean up storage and release resources for COPY FROM.
*/
void
EndCopyFrom(CopyState cstate)
{
FreeExecutorState(cstate->estate);- if (cstate->filename != NULL && FreeFile(cstate->copy_file))
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not close file \"%s\": %m",
- cstate->filename)));
-
/* Clean up storage */
EndCopy(cstate);
}/* * Read the next input line and stash it in line_buf, with conversion to * server encoding. --- 2454,2469 ----
On 02/07/2011 01:39 AM, Itagaki Takahiro wrote:
file_fdw uses CopyFromErrorCallback() to give errors the proper context. The
function uses template strings like "COPY %s, line %d", where %s is the name of
the relation being copied. Presumably file_fdw and other features using this
API would wish to customize that error message prefix, and the relation name
might not be apropos at all. How about another argument to BeginCopyFrom,
specifying an error prefix to be stashed in the CopyState?I changed "COPY %s, .." to "relation %s, ..." because the first string is
the relation name anyway. We could have another prefix argument, but I think
it has little information for errors.We also have many "COPY" in other messages, but they won't be used actually
because the are messages for invalid arguments and file_fdw will have own
validater function. All invalid arguments will be filtered in CREATE commands.
These changes have broken the regression tests. The attached patches
(one for the core regression tests and one for file_fdw) fix that.
But I don't know that your change is terribly helpful. I rather like
Noah's idea better, if we need to make a change.
cheers
andrew
Attachments:
copyapi-regressionfix-file_fdw.patchtext/x-patch; name=copyapi-regressionfix-file_fdw.patchDownload
diff --git a/contrib/file_fdw/output/file_fdw.source b/contrib/file_fdw/output/file_fdw.source
index f8ce4ca..7021ad8 100644
--- a/contrib/file_fdw/output/file_fdw.source
+++ b/contrib/file_fdw/output/file_fdw.source
@@ -89,7 +89,7 @@ SELECT * FROM agg_csv c JOIN agg_text t ON (t.a = c.a) ORDER BY c.a;
-- error context report tests
SELECT * FROM agg_bad; -- ERROR
ERROR: invalid input syntax for type real: "aaa"
-CONTEXT: COPY agg_bad, line 3, column b: "aaa"
+CONTEXT: relation agg_bad, line 3, column b: "aaa"
-- misc query tests
\t on
EXPLAIN (VERBOSE, COSTS FALSE) SELECT * FROM agg_csv;
copyapi-regressionfix-core.patchtext/x-patch; name=copyapi-regressionfix-core.patchDownload
diff --git a/src/test/regress/expected/alter_table.out b/src/test/regress/expected/alter_table.out
index 3280065..eea0b09 100644
--- a/src/test/regress/expected/alter_table.out
+++ b/src/test/regress/expected/alter_table.out
@@ -1002,7 +1002,7 @@ copy test("........pg.dropped.1........") to stdout;
ERROR: column "........pg.dropped.1........" of relation "test" does not exist
copy test from stdin;
ERROR: extra data after last expected column
-CONTEXT: COPY test, line 1: "10 11 12"
+CONTEXT: relation test, line 1: "10 11 12"
select * from test;
b | c
---+---
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index 15cbe02..d9630f8 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -35,17 +35,17 @@ ERROR: column "d" specified more than once
-- missing data: should fail
COPY x from stdin;
ERROR: invalid input syntax for integer: ""
-CONTEXT: COPY x, line 1, column a: ""
+CONTEXT: relation x, line 1, column a: ""
COPY x from stdin;
ERROR: missing data for column "e"
-CONTEXT: COPY x, line 1: "2000 230 23 23"
+CONTEXT: relation x, line 1: "2000 230 23 23"
COPY x from stdin;
ERROR: missing data for column "e"
-CONTEXT: COPY x, line 1: "2001 231 \N \N"
+CONTEXT: relation x, line 1: "2001 231 \N \N"
-- extra data: should fail
COPY x from stdin;
ERROR: extra data after last expected column
-CONTEXT: COPY x, line 1: "2002 232 40 50 60 70 80"
+CONTEXT: relation x, line 1: "2002 232 40 50 60 70 80"
-- various COPY options: delimiters, oids, NULL string
COPY x (b, c, d, e) from stdin with oids delimiter ',' null 'x';
COPY x from stdin WITH DELIMITER AS ';' NULL AS '';
diff --git a/src/test/regress/expected/domain.out b/src/test/regress/expected/domain.out
index 7d72791..daadcef 100644
--- a/src/test/regress/expected/domain.out
+++ b/src/test/regress/expected/domain.out
@@ -49,7 +49,7 @@ INSERT INTO basictest values ('88', 'haha', 'short', '123.1212'); -- Truncate
-- Test copy
COPY basictest (testvarchar) FROM stdin; -- fail
ERROR: value too long for type character varying(5)
-CONTEXT: COPY basictest, line 1, column testvarchar: "notsoshorttext"
+CONTEXT: relation basictest, line 1, column testvarchar: "notsoshorttext"
COPY basictest (testvarchar) FROM stdin;
select * from basictest;
testint4 | testtext | testvarchar | testnumeric
@@ -130,7 +130,7 @@ select testint4arr[1], testchar4arr[2:2] from domarrtest;
COPY domarrtest FROM stdin;
COPY domarrtest FROM stdin; -- fail
ERROR: value too long for type character varying(4)
-CONTEXT: COPY domarrtest, line 1, column testchar4arr: "{qwerty,w,e}"
+CONTEXT: relation domarrtest, line 1, column testchar4arr: "{qwerty,w,e}"
select * from domarrtest;
testint4arr | testchar4arr
---------------+---------------------
@@ -173,14 +173,14 @@ INSERT INTO nulltest values ('a', 'b', 'c', NULL, 'd'); -- Good
-- Test copy
COPY nulltest FROM stdin; --fail
ERROR: null value in column "col3" violates not-null constraint
-CONTEXT: COPY nulltest, line 1: "a b \N d d"
+CONTEXT: relation nulltest, line 1: "a b \N d d"
COPY nulltest FROM stdin; --fail
ERROR: domain dcheck does not allow null values
-CONTEXT: COPY nulltest, line 1, column col5: null input
+CONTEXT: relation nulltest, line 1, column col5: null input
-- Last row is bad
COPY nulltest FROM stdin;
ERROR: new row for relation "nulltest" violates check constraint "nulltest_col5_check"
-CONTEXT: COPY nulltest, line 3: "a b c \N a"
+CONTEXT: relation nulltest, line 3: "a b c \N a"
select * from nulltest;
col1 | col2 | col3 | col4 | col5
------+------+------+------+------
diff --git a/src/test/regress/output/constraints.source b/src/test/regress/output/constraints.source
index d164b90..64735d8 100644
--- a/src/test/regress/output/constraints.source
+++ b/src/test/regress/output/constraints.source
@@ -278,7 +278,7 @@ SELECT '' AS two, * FROM COPY_TBL;
COPY COPY_TBL FROM '@abs_srcdir@/data/constrf.data';
ERROR: new row for relation "copy_tbl" violates check constraint "copy_con"
-CONTEXT: COPY copy_tbl, line 2: "7 check failed 6"
+CONTEXT: relation copy_tbl, line 2: "7 check failed 6"
SELECT * FROM COPY_TBL;
x | y | z
---+---------------+---
Here is a revised patch, that including jagged csv support.
A new exported function is named as NextCopyFromRawFields.
On Mon, Feb 7, 2011 at 21:16, Noah Misch <noah@leadboat.com> wrote:
Perhaps I'm missing something. The new API does not expose a "processed" count
at all; that count is used for the command tag of a top-level COPY. This part
of the patch is just changing how we structure the code to maintain that tally,
and it has no implications for observers outside copy.c. Right?
True, but the counter is tightly bound with INSERT-side of COPY FROM.
| copy.c (1978)
| * We count only tuples not suppressed by a BEFORE INSERT trigger;
I think it is cleaner way to separate it from CopyState
because it is used as a file reader rather than a writer.
However, if there are objections, I'd revert it.
I changed "COPY %s, .." to "relation %s, ..."
My comment originated with a faulty idea that file_fdw's internal use of COPY
was an implementation detail that users should not need to see. Looking now,
the file_fdw documentation clearly explains the tie to COPY, even referring
users to the COPY documentation. I no longer see a need to hide the fact that
the "foreign" source is a PostgreSQL COPY command. The error messages are fine
as they were.
OK, I reverted the changes. User-visible changes might be more important,
pointed by Andrew.
ExecEvalExpr(), used to acquire default values, will still use the
per-tuple context of CopyState->estate. That per-tuple context will never get
reset explicitly, so default value computations leak until EndCopyFrom().
Ah, I see. I didn't see the leak because we rarely use per-tuple memory
context in the estate. We just use CurrentMemoryContext in many cases.
But the leak could occur, and the code is misleading.
I moved ResetPerTupleExprContext() into NextCopyFrom(), but
CurrentMemoryContext still used in it to the result values.
Another possible design might be passing EState as an argument of
NextCopyFrom and remove estate from CopyState. It seems a much cleaner way
in terms of control flow, but it requires more changes in file_fdw.
Comments?
This block comment remains roughly half correct. Again, I think a small comment
on every field below should replace it.This is just a verbatim copy of the CopyGetAttnums() header comment. (The
middle paragraph happens to be true, though.)
Silly mistakes. Maybe came from too many 'undo' in my editor.
--
Itagaki Takahiro
Attachments:
copy_export-20110208.patchapplication/octet-stream; name=copy_export-20110208.patchDownload
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index 3350ca0..02c5136 100644
*** a/src/backend/commands/copy.c
--- b/src/backend/commands/copy.c
*************** typedef struct CopyStateData
*** 93,105 ****
FILE *copy_file; /* used if copy_dest == COPY_FILE */
StringInfo fe_msgbuf; /* used for all dests during COPY TO, only for
* dest == COPY_NEW_FE in COPY FROM */
- bool fe_copy; /* true for all FE copy dests */
bool fe_eof; /* true if detected end of copy data */
EolType eol_type; /* EOL type of input */
int client_encoding; /* remote side's character encoding */
bool need_transcoding; /* client encoding diff from server? */
bool encoding_embeds_ascii; /* ASCII can be non-first byte? */
- uint64 processed; /* # of tuples processed */
/* parameters from the COPY command */
Relation rel; /* relation to copy to or from */
--- 93,103 ----
*************** typedef struct CopyStateData
*** 119,137 ****
bool *force_quote_flags; /* per-column CSV FQ flags */
bool *force_notnull_flags; /* per-column CSV FNN flags */
! /* these are just for error messages, see copy_in_error_callback */
const char *cur_relname; /* table name for error messages */
int cur_lineno; /* line number for error messages */
const char *cur_attname; /* current att for error messages */
const char *cur_attval; /* current att value for error messages */
/*
* Working state for COPY TO
*/
FmgrInfo *out_functions; /* lookup info for output functions */
MemoryContext rowcontext; /* per-row evaluation context */
/*
* These variables are used to reduce overhead in textual COPY FROM.
*
* attribute_buf holds the separated, de-escaped text for each field of
--- 117,153 ----
bool *force_quote_flags; /* per-column CSV FQ flags */
bool *force_notnull_flags; /* per-column CSV FNN flags */
! /* these are just for error messages, see CopyFromErrorCallback */
const char *cur_relname; /* table name for error messages */
int cur_lineno; /* line number for error messages */
const char *cur_attname; /* current att for error messages */
const char *cur_attval; /* current att value for error messages */
/*
+ * Working state for COPY TO/FROM
+ */
+ MemoryContext copycontext; /* per-copy execution context */
+
+ /*
* Working state for COPY TO
*/
FmgrInfo *out_functions; /* lookup info for output functions */
MemoryContext rowcontext; /* per-row evaluation context */
/*
+ * Working state for COPY FROM
+ */
+ EState *estate;
+ AttrNumber num_defaults;
+ bool file_has_oids;
+ FmgrInfo oid_in_function;
+ Oid oid_typioparam;
+ FmgrInfo *in_functions; /* array of input functions for each attrs */
+ Oid *typioparams; /* array of element types for in_functions */
+ int *defmap; /* array of default att numbers */
+ ExprState **defexprs; /* array of default att expressions */
+
+ /*
* These variables are used to reduce overhead in textual COPY FROM.
*
* attribute_buf holds the separated, de-escaped text for each field of
*************** typedef struct CopyStateData
*** 169,181 ****
int raw_buf_len; /* total # of bytes stored */
} CopyStateData;
- typedef CopyStateData *CopyState;
-
/* DestReceiver for COPY (SELECT) TO */
typedef struct
{
DestReceiver pub; /* publicly-known function pointers */
CopyState cstate; /* CopyStateData for the command */
} DR_copy;
--- 185,196 ----
int raw_buf_len; /* total # of bytes stored */
} CopyStateData;
/* DestReceiver for COPY (SELECT) TO */
typedef struct
{
DestReceiver pub; /* publicly-known function pointers */
CopyState cstate; /* CopyStateData for the command */
+ uint64 processed; /* # of tuples processed */
} DR_copy;
*************** static const char BinarySignature[11] =
*** 248,258 ****
/* non-export function prototypes */
! static void DoCopyTo(CopyState cstate);
! static void CopyTo(CopyState cstate);
static void CopyOneRowTo(CopyState cstate, Oid tupleOid,
Datum *values, bool *nulls);
! static void CopyFrom(CopyState cstate);
static bool CopyReadLine(CopyState cstate);
static bool CopyReadLineText(CopyState cstate);
static int CopyReadAttributesText(CopyState cstate);
--- 263,279 ----
/* non-export function prototypes */
! static CopyState BeginCopy(bool is_from, Relation rel, Node *raw_query,
! const char *queryString, List *attnamelist, List *options);
! static void EndCopy(CopyState cstate);
! static CopyState BeginCopyTo(Relation rel, Node *query, const char *queryString,
! const char *filename, List *attnamelist, List *options);
! static void EndCopyTo(CopyState cstate);
! static uint64 DoCopyTo(CopyState cstate);
! static uint64 CopyTo(CopyState cstate);
static void CopyOneRowTo(CopyState cstate, Oid tupleOid,
Datum *values, bool *nulls);
! static uint64 CopyFrom(CopyState cstate);
static bool CopyReadLine(CopyState cstate);
static bool CopyReadLineText(CopyState cstate);
static int CopyReadAttributesText(CopyState cstate);
*************** CopyLoadRawBuf(CopyState cstate)
*** 700,705 ****
--- 721,822 ----
* input/output stream. The latter could be either stdin/stdout or a
* socket, depending on whether we're running under Postmaster control.
*
+ * Do not allow a Postgres user without superuser privilege to read from
+ * or write to a file.
+ *
+ * Do not allow the copy if user doesn't have proper permission to access
+ * the table or the specifically requested columns.
+ */
+ uint64
+ DoCopy(const CopyStmt *stmt, const char *queryString)
+ {
+ CopyState cstate;
+ bool is_from = stmt->is_from;
+ bool pipe = (stmt->filename == NULL);
+ Relation rel;
+ uint64 processed;
+
+ /* Disallow file COPY except to superusers. */
+ if (!pipe && !superuser())
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+ errmsg("must be superuser to COPY to or from a file"),
+ errhint("Anyone can COPY to stdout or from stdin. "
+ "psql's \\copy command also works for anyone.")));
+
+ if (stmt->relation)
+ {
+ TupleDesc tupDesc;
+ AclMode required_access = (is_from ? ACL_INSERT : ACL_SELECT);
+ RangeTblEntry *rte;
+ List *attnums;
+ ListCell *cur;
+
+ Assert(!stmt->query);
+
+ /* Open and lock the relation, using the appropriate lock type. */
+ rel = heap_openrv(stmt->relation,
+ (is_from ? RowExclusiveLock : AccessShareLock));
+
+ rte = makeNode(RangeTblEntry);
+ rte->rtekind = RTE_RELATION;
+ rte->relid = RelationGetRelid(rel);
+ rte->requiredPerms = required_access;
+
+ tupDesc = RelationGetDescr(rel);
+ attnums = CopyGetAttnums(tupDesc, rel, stmt->attlist);
+ foreach (cur, attnums)
+ {
+ int attno = lfirst_int(cur) -
+ FirstLowInvalidHeapAttributeNumber;
+
+ if (is_from)
+ rte->modifiedCols = bms_add_member(rte->modifiedCols, attno);
+ else
+ rte->selectedCols = bms_add_member(rte->selectedCols, attno);
+ }
+ ExecCheckRTPerms(list_make1(rte), true);
+ }
+ else
+ {
+ Assert(stmt->query);
+
+ rel = NULL;
+ }
+
+ if (is_from)
+ {
+ /* check read-only transaction */
+ if (XactReadOnly && rel->rd_backend != MyBackendId)
+ PreventCommandIfReadOnly("COPY FROM");
+
+ cstate = BeginCopyFrom(rel, stmt->filename,
+ stmt->attlist, stmt->options);
+ processed = CopyFrom(cstate); /* copy from file to database */
+ EndCopyFrom(cstate);
+ }
+ else
+ {
+ cstate = BeginCopyTo(rel, stmt->query, queryString, stmt->filename,
+ stmt->attlist, stmt->options);
+ processed = DoCopyTo(cstate); /* copy from database to file */
+ EndCopyTo(cstate);
+ }
+
+ /*
+ * Close the relation. If reading, we can release the AccessShareLock we got;
+ * if writing, we should hold the lock until end of transaction to ensure that
+ * updates will be committed before lock is released.
+ */
+ if (rel != NULL)
+ heap_close(rel, (is_from ? NoLock : AccessShareLock));
+
+ return processed;
+ }
+
+ /*
+ * Common setup routines used by BeginCopyFrom and BeginCopyTo.
+ *
* Iff <binary>, unload or reload in the binary format, as opposed to the
* more wasteful but more robust and portable text format.
*
*************** CopyLoadRawBuf(CopyState cstate)
*** 711,745 ****
*
* If in the text format, delimit columns with delimiter <delim> and print
* NULL values as <null_print>.
- *
- * Do not allow a Postgres user without superuser privilege to read from
- * or write to a file.
- *
- * Do not allow the copy if user doesn't have proper permission to access
- * the table or the specifically requested columns.
*/
! uint64
! DoCopy(const CopyStmt *stmt, const char *queryString)
{
CopyState cstate;
- bool is_from = stmt->is_from;
- bool pipe = (stmt->filename == NULL);
- List *attnamelist = stmt->attlist;
List *force_quote = NIL;
List *force_notnull = NIL;
bool force_quote_all = false;
bool format_specified = false;
- AclMode required_access = (is_from ? ACL_INSERT : ACL_SELECT);
ListCell *option;
TupleDesc tupDesc;
int num_phys_attrs;
! uint64 processed;
/* Allocate workspace and zero all fields */
cstate = (CopyStateData *) palloc0(sizeof(CopyStateData));
/* Extract options from the statement node tree */
! foreach(option, stmt->options)
{
DefElem *defel = (DefElem *) lfirst(option);
--- 828,869 ----
*
* If in the text format, delimit columns with delimiter <delim> and print
* NULL values as <null_print>.
*/
! static CopyState
! BeginCopy(bool is_from,
! Relation rel,
! Node *raw_query,
! const char *queryString,
! List *attnamelist,
! List *options)
{
CopyState cstate;
List *force_quote = NIL;
List *force_notnull = NIL;
bool force_quote_all = false;
bool format_specified = false;
ListCell *option;
TupleDesc tupDesc;
int num_phys_attrs;
! MemoryContext oldcontext;
/* Allocate workspace and zero all fields */
cstate = (CopyStateData *) palloc0(sizeof(CopyStateData));
+ /*
+ * We allocate everything used by a cstate in a new memory context.
+ * This would avoid memory leaks repeated uses of COPY in a query.
+ */
+ cstate->copycontext = AllocSetContextCreate(CurrentMemoryContext,
+ "COPY",
+ ALLOCSET_DEFAULT_MINSIZE,
+ ALLOCSET_DEFAULT_INITSIZE,
+ ALLOCSET_DEFAULT_MAXSIZE);
+
+ oldcontext = MemoryContextSwitchTo(cstate->copycontext);
+
/* Extract options from the statement node tree */
! foreach(option, options)
{
DefElem *defel = (DefElem *) lfirst(option);
*************** DoCopy(const CopyStmt *stmt, const char
*** 980,1030 ****
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("CSV quote character must not appear in the NULL specification")));
! /* Disallow file COPY except to superusers. */
! if (!pipe && !superuser())
! ereport(ERROR,
! (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
! errmsg("must be superuser to COPY to or from a file"),
! errhint("Anyone can COPY to stdout or from stdin. "
! "psql's \\copy command also works for anyone.")));
!
! if (stmt->relation)
{
! RangeTblEntry *rte;
! List *attnums;
! ListCell *cur;
!
! Assert(!stmt->query);
! cstate->queryDesc = NULL;
! /* Open and lock the relation, using the appropriate lock type. */
! cstate->rel = heap_openrv(stmt->relation,
! (is_from ? RowExclusiveLock : AccessShareLock));
tupDesc = RelationGetDescr(cstate->rel);
- /* Check relation permissions. */
- rte = makeNode(RangeTblEntry);
- rte->rtekind = RTE_RELATION;
- rte->relid = RelationGetRelid(cstate->rel);
- rte->requiredPerms = required_access;
-
- attnums = CopyGetAttnums(tupDesc, cstate->rel, attnamelist);
- foreach (cur, attnums)
- {
- int attno = lfirst_int(cur) - FirstLowInvalidHeapAttributeNumber;
-
- if (is_from)
- rte->modifiedCols = bms_add_member(rte->modifiedCols, attno);
- else
- rte->selectedCols = bms_add_member(rte->selectedCols, attno);
- }
- ExecCheckRTPerms(list_make1(rte), true);
-
- /* check read-only transaction */
- if (XactReadOnly && is_from && cstate->rel->rd_backend != MyBackendId)
- PreventCommandIfReadOnly("COPY FROM");
-
/* Don't allow COPY w/ OIDs to or from a table without them */
if (cstate->oids && !cstate->rel->rd_rel->relhasoids)
ereport(ERROR,
--- 1104,1117 ----
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("CSV quote character must not appear in the NULL specification")));
! if (rel)
{
! Assert(!raw_query);
! cstate->rel = rel;
tupDesc = RelationGetDescr(cstate->rel);
/* Don't allow COPY w/ OIDs to or from a table without them */
if (cstate->oids && !cstate->rel->rd_rel->relhasoids)
ereport(ERROR,
*************** DoCopy(const CopyStmt *stmt, const char
*** 1058,1064 ****
* function and is executed repeatedly. (See also the same hack in
* DECLARE CURSOR and PREPARE.) XXX FIXME someday.
*/
! rewritten = pg_analyze_and_rewrite((Node *) copyObject(stmt->query),
queryString, NULL, 0);
/* We don't expect more or less than one result query */
--- 1145,1151 ----
* function and is executed repeatedly. (See also the same hack in
* DECLARE CURSOR and PREPARE.) XXX FIXME someday.
*/
! rewritten = pg_analyze_and_rewrite((Node *) copyObject(raw_query),
queryString, NULL, 0);
/* We don't expect more or less than one result query */
*************** DoCopy(const CopyStmt *stmt, const char
*** 1160,1173 ****
}
}
- /* Set up variables to avoid per-attribute overhead. */
- initStringInfo(&cstate->attribute_buf);
- initStringInfo(&cstate->line_buf);
- cstate->line_buf_converted = false;
- cstate->raw_buf = (char *) palloc(RAW_BUF_SIZE + 1);
- cstate->raw_buf_index = cstate->raw_buf_len = 0;
- cstate->processed = 0;
-
/*
* Set up encoding conversion info. Even if the client and server
* encodings are the same, we must apply pg_client_to_server() to validate
--- 1247,1252 ----
*************** DoCopy(const CopyStmt *stmt, const char
*** 1181,1264 ****
cstate->encoding_embeds_ascii = PG_ENCODING_IS_CLIENT_ONLY(cstate->client_encoding);
cstate->copy_dest = COPY_FILE; /* default */
- cstate->filename = stmt->filename;
! if (is_from)
! CopyFrom(cstate); /* copy from file to database */
! else
! DoCopyTo(cstate); /* copy from database to file */
! /*
! * Close the relation or query. If reading, we can release the
! * AccessShareLock we got; if writing, we should hold the lock until end
! * of transaction to ensure that updates will be committed before lock is
! * released.
! */
! if (cstate->rel)
! heap_close(cstate->rel, (is_from ? NoLock : AccessShareLock));
! else
! {
! /* Close down the query and free resources. */
! ExecutorEnd(cstate->queryDesc);
! FreeQueryDesc(cstate->queryDesc);
! PopActiveSnapshot();
! }
! /* Clean up storage (probably not really necessary) */
! processed = cstate->processed;
! pfree(cstate->attribute_buf.data);
! pfree(cstate->line_buf.data);
! pfree(cstate->raw_buf);
pfree(cstate);
-
- return processed;
}
-
/*
! * This intermediate routine exists mainly to localize the effects of setjmp
! * so we don't need to plaster a lot of variables with "volatile".
*/
! static void
! DoCopyTo(CopyState cstate)
{
! bool pipe = (cstate->filename == NULL);
! if (cstate->rel)
{
! if (cstate->rel->rd_rel->relkind != RELKIND_RELATION)
! {
! if (cstate->rel->rd_rel->relkind == RELKIND_VIEW)
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from view \"%s\"",
! RelationGetRelationName(cstate->rel)),
! errhint("Try the COPY (SELECT ...) TO variant.")));
! else if (cstate->rel->rd_rel->relkind == RELKIND_FOREIGN_TABLE)
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from foreign table \"%s\"",
! RelationGetRelationName(cstate->rel)),
! errhint("Try the COPY (SELECT ...) TO variant.")));
! else if (cstate->rel->rd_rel->relkind == RELKIND_SEQUENCE)
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from sequence \"%s\"",
! RelationGetRelationName(cstate->rel))));
! else
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from non-table relation \"%s\"",
! RelationGetRelationName(cstate->rel))));
! }
}
if (pipe)
{
! if (whereToSendOutput == DestRemote)
! cstate->fe_copy = true;
! else
cstate->copy_file = stdout;
}
else
--- 1260,1334 ----
cstate->encoding_embeds_ascii = PG_ENCODING_IS_CLIENT_ONLY(cstate->client_encoding);
cstate->copy_dest = COPY_FILE; /* default */
! MemoryContextSwitchTo(oldcontext);
! return cstate;
! }
! /*
! * Release resources allocated in a cstate.
! */
! static void
! EndCopy(CopyState cstate)
! {
! if (cstate->filename != NULL && FreeFile(cstate->copy_file))
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not close file \"%s\": %m",
! cstate->filename)));
! MemoryContextDelete(cstate->copycontext);
pfree(cstate);
}
/*
! * Setup CopyState to read tuples from a table or a query for COPY TO.
*/
! static CopyState
! BeginCopyTo(Relation rel,
! Node *query,
! const char *queryString,
! const char *filename,
! List *attnamelist,
! List *options)
{
! CopyState cstate;
! bool pipe = (filename == NULL);
! MemoryContext oldcontext;
! if (rel != NULL && rel->rd_rel->relkind != RELKIND_RELATION)
{
! if (rel->rd_rel->relkind == RELKIND_VIEW)
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from view \"%s\"",
! RelationGetRelationName(rel)),
! errhint("Try the COPY (SELECT ...) TO variant.")));
! else if (rel->rd_rel->relkind == RELKIND_FOREIGN_TABLE)
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from foreign table \"%s\"",
! RelationGetRelationName(rel)),
! errhint("Try the COPY (SELECT ...) TO variant.")));
! else if (rel->rd_rel->relkind == RELKIND_SEQUENCE)
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from sequence \"%s\"",
! RelationGetRelationName(rel))));
! else
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from non-table relation \"%s\"",
! RelationGetRelationName(rel))));
}
+ cstate = BeginCopy(false, rel, query, queryString, attnamelist, options);
+ oldcontext = MemoryContextSwitchTo(cstate->copycontext);
+
if (pipe)
{
! if (whereToSendOutput != DestRemote)
cstate->copy_file = stdout;
}
else
*************** DoCopyTo(CopyState cstate)
*** 1270,1280 ****
* Prevent write to relative path ... too easy to shoot oneself in the
* foot by overwriting a database file ...
*/
! if (!is_absolute_path(cstate->filename))
ereport(ERROR,
(errcode(ERRCODE_INVALID_NAME),
errmsg("relative path not allowed for COPY to file")));
oumask = umask(S_IWGRP | S_IWOTH);
cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_W);
umask(oumask);
--- 1340,1351 ----
* Prevent write to relative path ... too easy to shoot oneself in the
* foot by overwriting a database file ...
*/
! if (!is_absolute_path(filename))
ereport(ERROR,
(errcode(ERRCODE_INVALID_NAME),
errmsg("relative path not allowed for COPY to file")));
+ cstate->filename = pstrdup(filename);
oumask = umask(S_IWGRP | S_IWOTH);
cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_W);
umask(oumask);
*************** DoCopyTo(CopyState cstate)
*** 1292,1305 ****
errmsg("\"%s\" is a directory", cstate->filename)));
}
PG_TRY();
{
! if (cstate->fe_copy)
SendCopyBegin(cstate);
! CopyTo(cstate);
! if (cstate->fe_copy)
SendCopyEnd(cstate);
}
PG_CATCH();
--- 1363,1392 ----
errmsg("\"%s\" is a directory", cstate->filename)));
}
+ MemoryContextSwitchTo(oldcontext);
+
+ return cstate;
+ }
+
+ /*
+ * This intermediate routine exists mainly to localize the effects of setjmp
+ * so we don't need to plaster a lot of variables with "volatile".
+ */
+ static uint64
+ DoCopyTo(CopyState cstate)
+ {
+ bool pipe = (cstate->filename == NULL);
+ bool fe_copy = (pipe && whereToSendOutput == DestRemote);
+ uint64 processed;
+
PG_TRY();
{
! if (fe_copy)
SendCopyBegin(cstate);
! processed = CopyTo(cstate);
! if (fe_copy)
SendCopyEnd(cstate);
}
PG_CATCH();
*************** DoCopyTo(CopyState cstate)
*** 1314,1339 ****
}
PG_END_TRY();
! if (!pipe)
{
! if (FreeFile(cstate->copy_file))
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not close file \"%s\": %m",
! cstate->filename)));
}
}
/*
* Copy from relation or query TO file.
*/
! static void
CopyTo(CopyState cstate)
{
TupleDesc tupDesc;
int num_phys_attrs;
Form_pg_attribute *attr;
ListCell *cur;
if (cstate->rel)
tupDesc = RelationGetDescr(cstate->rel);
--- 1401,1438 ----
}
PG_END_TRY();
! return processed;
! }
!
! /*
! * Clean up storage and release resources for COPY TO.
! */
! static void
! EndCopyTo(CopyState cstate)
! {
! if (cstate->queryDesc != NULL)
{
! /* Close down the query and free resources. */
! ExecutorEnd(cstate->queryDesc);
! FreeQueryDesc(cstate->queryDesc);
! PopActiveSnapshot();
}
+
+ /* Clean up storage */
+ EndCopy(cstate);
}
/*
* Copy from relation or query TO file.
*/
! static uint64
CopyTo(CopyState cstate)
{
TupleDesc tupDesc;
int num_phys_attrs;
Form_pg_attribute *attr;
ListCell *cur;
+ uint64 processed;
if (cstate->rel)
tupDesc = RelationGetDescr(cstate->rel);
*************** CopyTo(CopyState cstate)
*** 1439,1444 ****
--- 1538,1544 ----
scandesc = heap_beginscan(cstate->rel, GetActiveSnapshot(), 0, NULL);
+ processed = 0;
while ((tuple = heap_getnext(scandesc, ForwardScanDirection)) != NULL)
{
CHECK_FOR_INTERRUPTS();
*************** CopyTo(CopyState cstate)
*** 1448,1461 ****
--- 1548,1566 ----
/* Format and send the data */
CopyOneRowTo(cstate, HeapTupleGetOid(tuple), values, nulls);
+ processed++;
}
heap_endscan(scandesc);
+
+ pfree(values);
+ pfree(nulls);
}
else
{
/* run the plan --- the dest receiver will send tuples */
ExecutorRun(cstate->queryDesc, ForwardScanDirection, 0L);
+ processed = ((DR_copy *) cstate->queryDesc->dest)->processed;
}
if (cstate->binary)
*************** CopyTo(CopyState cstate)
*** 1467,1472 ****
--- 1572,1579 ----
}
MemoryContextDelete(cstate->rowcontext);
+
+ return processed;
}
/*
*************** CopyOneRowTo(CopyState cstate, Oid tuple
*** 1558,1573 ****
CopySendEndOfRow(cstate);
MemoryContextSwitchTo(oldcontext);
-
- cstate->processed++;
}
/*
* error context callback for COPY FROM
*/
! static void
! copy_in_error_callback(void *arg)
{
CopyState cstate = (CopyState) arg;
--- 1665,1680 ----
CopySendEndOfRow(cstate);
MemoryContextSwitchTo(oldcontext);
}
/*
* error context callback for COPY FROM
+ *
+ * The argument for the error context must be CopyState.
*/
! void
! CopyFromErrorCallback(void *arg)
{
CopyState cstate = (CopyState) arg;
*************** limit_printout_length(const char *str)
*** 1669,1709 ****
/*
* Copy FROM file to relation.
*/
! static void
CopyFrom(CopyState cstate)
{
- bool pipe = (cstate->filename == NULL);
HeapTuple tuple;
TupleDesc tupDesc;
- Form_pg_attribute *attr;
- AttrNumber num_phys_attrs,
- attr_count,
- num_defaults;
- FmgrInfo *in_functions;
- FmgrInfo oid_in_function;
- Oid *typioparams;
- Oid oid_typioparam;
- int attnum;
- int i;
- Oid in_func_oid;
Datum *values;
bool *nulls;
- int nfields;
- char **field_strings;
bool done = false;
- bool isnull;
ResultRelInfo *resultRelInfo;
! EState *estate = CreateExecutorState(); /* for ExecConstraints() */
TupleTableSlot *slot;
- bool file_has_oids;
- int *defmap;
- ExprState **defexprs; /* array of default att expressions */
- ExprContext *econtext; /* used for ExecEvalExpr for default atts */
MemoryContext oldcontext = CurrentMemoryContext;
ErrorContextCallback errcontext;
CommandId mycid = GetCurrentCommandId(true);
int hi_options = 0; /* start with default heap_insert options */
BulkInsertState bistate;
Assert(cstate->rel);
--- 1776,1798 ----
/*
* Copy FROM file to relation.
*/
! static uint64
CopyFrom(CopyState cstate)
{
HeapTuple tuple;
TupleDesc tupDesc;
Datum *values;
bool *nulls;
bool done = false;
ResultRelInfo *resultRelInfo;
! EState *estate = cstate->estate; /* for ExecConstraints() */
TupleTableSlot *slot;
MemoryContext oldcontext = CurrentMemoryContext;
ErrorContextCallback errcontext;
CommandId mycid = GetCurrentCommandId(true);
int hi_options = 0; /* start with default heap_insert options */
BulkInsertState bistate;
+ uint64 processed = 0;
Assert(cstate->rel);
*************** CopyFrom(CopyState cstate)
*** 1731,1736 ****
--- 1820,1827 ----
RelationGetRelationName(cstate->rel))));
}
+ tupDesc = RelationGetDescr(cstate->rel);
+
/*----------
* Check to see if we can avoid writing WAL
*
*************** CopyFrom(CopyState cstate)
*** 1766,1803 ****
hi_options |= HEAP_INSERT_SKIP_WAL;
}
- if (pipe)
- {
- if (whereToSendOutput == DestRemote)
- ReceiveCopyBegin(cstate);
- else
- cstate->copy_file = stdin;
- }
- else
- {
- struct stat st;
-
- cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_R);
-
- if (cstate->copy_file == NULL)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not open file \"%s\" for reading: %m",
- cstate->filename)));
-
- fstat(fileno(cstate->copy_file), &st);
- if (S_ISDIR(st.st_mode))
- ereport(ERROR,
- (errcode(ERRCODE_WRONG_OBJECT_TYPE),
- errmsg("\"%s\" is a directory", cstate->filename)));
- }
-
- tupDesc = RelationGetDescr(cstate->rel);
- attr = tupDesc->attrs;
- num_phys_attrs = tupDesc->natts;
- attr_count = list_length(cstate->attnumlist);
- num_defaults = 0;
-
/*
* We need a ResultRelInfo so we can use the regular executor's
* index-entry-making machinery. (There used to be a huge amount of code
--- 1857,1862 ----
*************** CopyFrom(CopyState cstate)
*** 1826,1832 ****
slot = ExecInitExtraTupleSlot(estate);
ExecSetSlotDescriptor(slot, tupDesc);
! econtext = GetPerTupleExprContext(estate);
/*
* Pick up the required catalog information for each attribute in the
--- 1885,2070 ----
slot = ExecInitExtraTupleSlot(estate);
ExecSetSlotDescriptor(slot, tupDesc);
! /* Prepare to catch AFTER triggers. */
! AfterTriggerBeginQuery();
!
! /*
! * Check BEFORE STATEMENT insertion triggers. It's debateable whether we
! * should do this for COPY, since it's not really an "INSERT" statement as
! * such. However, executing these triggers maintains consistency with the
! * EACH ROW triggers that we already fire on COPY.
! */
! ExecBSInsertTriggers(estate, resultRelInfo);
!
! values = (Datum *) palloc(tupDesc->natts * sizeof(Datum));
! nulls = (bool *) palloc(tupDesc->natts * sizeof(bool));
!
! bistate = GetBulkInsertState();
!
! /* Set up callback to identify error line number */
! errcontext.callback = CopyFromErrorCallback;
! errcontext.arg = (void *) cstate;
! errcontext.previous = error_context_stack;
! error_context_stack = &errcontext;
!
! while (!done)
! {
! bool skip_tuple;
! Oid loaded_oid = InvalidOid;
!
! CHECK_FOR_INTERRUPTS();
!
! /* Switch into its memory context */
! MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
!
! /* Per-tuple memory context will be reset in NextCopyFrom. */
! if (!NextCopyFrom(cstate, values, nulls, &loaded_oid))
! break;
!
! /* And now we can form the input tuple. */
! tuple = heap_form_tuple(tupDesc, values, nulls);
!
! if (loaded_oid != InvalidOid)
! HeapTupleSetOid(tuple, loaded_oid);
!
! /* Triggers and stuff need to be invoked in query context. */
! MemoryContextSwitchTo(oldcontext);
!
! skip_tuple = false;
!
! /* BEFORE ROW INSERT Triggers */
! if (resultRelInfo->ri_TrigDesc &&
! resultRelInfo->ri_TrigDesc->trig_insert_before_row)
! {
! HeapTuple newtuple;
!
! newtuple = ExecBRInsertTriggers(estate, resultRelInfo, tuple);
!
! if (newtuple == NULL) /* "do nothing" */
! skip_tuple = true;
! else if (newtuple != tuple) /* modified by Trigger(s) */
! {
! heap_freetuple(tuple);
! tuple = newtuple;
! }
! }
!
! if (!skip_tuple)
! {
! List *recheckIndexes = NIL;
!
! /* Place tuple in tuple slot */
! ExecStoreTuple(tuple, slot, InvalidBuffer, false);
!
! /* Check the constraints of the tuple */
! if (cstate->rel->rd_att->constr)
! ExecConstraints(resultRelInfo, slot, estate);
!
! /* OK, store the tuple and create index entries for it */
! heap_insert(cstate->rel, tuple, mycid, hi_options, bistate);
!
! if (resultRelInfo->ri_NumIndices > 0)
! recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self),
! estate);
!
! /* AFTER ROW INSERT Triggers */
! ExecARInsertTriggers(estate, resultRelInfo, tuple,
! recheckIndexes);
!
! list_free(recheckIndexes);
!
! /*
! * We count only tuples not suppressed by a BEFORE INSERT trigger;
! * this is the same definition used by execMain.c for counting
! * tuples inserted by an INSERT command.
! */
! processed++;
! }
! }
!
! /* Done, clean up */
! error_context_stack = errcontext.previous;
!
! FreeBulkInsertState(bistate);
!
! MemoryContextSwitchTo(oldcontext);
!
! /* Execute AFTER STATEMENT insertion triggers */
! ExecASInsertTriggers(estate, resultRelInfo);
!
! /* Handle queued AFTER triggers */
! AfterTriggerEndQuery(estate);
!
! pfree(values);
! pfree(nulls);
!
! ExecResetTupleTable(estate->es_tupleTable, false);
!
! ExecCloseIndices(resultRelInfo);
!
! /*
! * If we skipped writing WAL, then we need to sync the heap (but not
! * indexes since those use WAL anyway)
! */
! if (hi_options & HEAP_INSERT_SKIP_WAL)
! heap_sync(cstate->rel);
!
! return processed;
! }
!
! /*
! * Setup to read tuples from a file for COPY FROM.
! *
! * 'rel': Used as a template for the tuples
! * 'filename': Name of server-local file to read
! * 'attnamelist': List of char *, columns to include. NIL selects all cols.
! * 'options': List of DefElem. See copy_opt_item in gram.y for selections.
! *
! * Returns a CopyState, to be passed to NextCopyFrom.
! */
! CopyState
! BeginCopyFrom(Relation rel,
! const char *filename,
! List *attnamelist,
! List *options)
! {
! CopyState cstate;
! bool pipe = (filename == NULL);
! TupleDesc tupDesc;
! Form_pg_attribute *attr;
! AttrNumber num_phys_attrs,
! num_defaults;
! FmgrInfo *in_functions;
! Oid *typioparams;
! int attnum;
! Oid in_func_oid;
! EState *estate = CreateExecutorState(); /* for ExecPrepareExpr() */
! int *defmap;
! ExprState **defexprs;
! MemoryContext oldcontext;
!
! cstate = BeginCopy(true, rel, NULL, NULL, attnamelist, options);
! oldcontext = MemoryContextSwitchTo(cstate->copycontext);
!
! /* Initialize state variables */
! cstate->fe_eof = false;
! cstate->eol_type = EOL_UNKNOWN;
! cstate->cur_relname = RelationGetRelationName(cstate->rel);
! cstate->cur_lineno = 0;
! cstate->cur_attname = NULL;
! cstate->cur_attval = NULL;
!
! /* Set up variables to avoid per-attribute overhead. */
! initStringInfo(&cstate->attribute_buf);
! initStringInfo(&cstate->line_buf);
! cstate->line_buf_converted = false;
! cstate->raw_buf = (char *) palloc(RAW_BUF_SIZE + 1);
! cstate->raw_buf_index = cstate->raw_buf_len = 0;
!
! tupDesc = RelationGetDescr(cstate->rel);
! attr = tupDesc->attrs;
! num_phys_attrs = tupDesc->natts;
! num_defaults = 0;
/*
* Pick up the required catalog information for each attribute in the
*************** CopyFrom(CopyState cstate)
*** 1871,1889 ****
}
}
! /* Prepare to catch AFTER triggers. */
! AfterTriggerBeginQuery();
! /*
! * Check BEFORE STATEMENT insertion triggers. It's debateable whether we
! * should do this for COPY, since it's not really an "INSERT" statement as
! * such. However, executing these triggers maintains consistency with the
! * EACH ROW triggers that we already fire on COPY.
! */
! ExecBSInsertTriggers(estate, resultRelInfo);
if (!cstate->binary)
! file_has_oids = cstate->oids; /* must rely on user to tell us... */
else
{
/* Read and verify binary header */
--- 2109,2154 ----
}
}
! /* We keep those variables in cstate. */
! cstate->estate = estate;
! cstate->in_functions = in_functions;
! cstate->typioparams = typioparams;
! cstate->defmap = defmap;
! cstate->defexprs = defexprs;
! cstate->num_defaults = num_defaults;
! if (pipe)
! {
! if (whereToSendOutput == DestRemote)
! ReceiveCopyBegin(cstate);
! else
! cstate->copy_file = stdin;
! }
! else
! {
! struct stat st;
!
! cstate->filename = pstrdup(filename);
! cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_R);
!
! if (cstate->copy_file == NULL)
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not open file \"%s\" for reading: %m",
! cstate->filename)));
!
! fstat(fileno(cstate->copy_file), &st);
! if (S_ISDIR(st.st_mode))
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("\"%s\" is a directory", cstate->filename)));
! }
if (!cstate->binary)
! {
! /* must rely on user to tell us... */
! cstate->file_has_oids = cstate->oids;
! }
else
{
/* Read and verify binary header */
*************** CopyFrom(CopyState cstate)
*** 1901,1907 ****
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid COPY file header (missing flags)")));
! file_has_oids = (tmp & (1 << 16)) != 0;
tmp &= ~(1 << 16);
if ((tmp >> 16) != 0)
ereport(ERROR,
--- 2166,2172 ----
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid COPY file header (missing flags)")));
! cstate->file_has_oids = (tmp & (1 << 16)) != 0;
tmp &= ~(1 << 16);
if ((tmp >> 16) != 0)
ereport(ERROR,
*************** CopyFrom(CopyState cstate)
*** 1923,1995 ****
}
}
! if (file_has_oids && cstate->binary)
{
getTypeBinaryInputInfo(OIDOID,
! &in_func_oid, &oid_typioparam);
! fmgr_info(in_func_oid, &oid_in_function);
}
- values = (Datum *) palloc(num_phys_attrs * sizeof(Datum));
- nulls = (bool *) palloc(num_phys_attrs * sizeof(bool));
-
/* create workspace for CopyReadAttributes results */
! nfields = file_has_oids ? (attr_count + 1) : attr_count;
! if (! cstate->binary)
{
cstate->max_fields = nfields;
cstate->raw_fields = (char **) palloc(nfields * sizeof(char *));
}
! /* Initialize state variables */
! cstate->fe_eof = false;
! cstate->eol_type = EOL_UNKNOWN;
! cstate->cur_relname = RelationGetRelationName(cstate->rel);
! cstate->cur_lineno = 0;
! cstate->cur_attname = NULL;
! cstate->cur_attval = NULL;
! bistate = GetBulkInsertState();
! /* Set up callback to identify error line number */
! errcontext.callback = copy_in_error_callback;
! errcontext.arg = (void *) cstate;
! errcontext.previous = error_context_stack;
! error_context_stack = &errcontext;
/* on input just throw the header line away */
! if (cstate->header_line)
{
cstate->cur_lineno++;
! done = CopyReadLine(cstate);
}
! while (!done)
! {
! bool skip_tuple;
! Oid loaded_oid = InvalidOid;
!
! CHECK_FOR_INTERRUPTS();
!
cstate->cur_lineno++;
- /* Reset the per-tuple exprcontext */
- ResetPerTupleExprContext(estate);
-
- /* Switch into its memory context */
- MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
-
- /* Initialize all values for row to NULL */
- MemSet(values, 0, num_phys_attrs * sizeof(Datum));
- MemSet(nulls, true, num_phys_attrs * sizeof(bool));
-
- if (!cstate->binary)
- {
- ListCell *cur;
- int fldct;
- int fieldno;
- char *string;
-
/* Actually read the line into memory here */
done = CopyReadLine(cstate);
--- 2188,2250 ----
}
}
! if (cstate->file_has_oids && cstate->binary)
{
getTypeBinaryInputInfo(OIDOID,
! &in_func_oid, &cstate->oid_typioparam);
! fmgr_info(in_func_oid, &cstate->oid_in_function);
}
/* create workspace for CopyReadAttributes results */
! if (!cstate->binary)
{
+ AttrNumber attr_count = list_length(cstate->attnumlist);
+ int nfields = cstate->file_has_oids ? (attr_count + 1) : attr_count;
+
cstate->max_fields = nfields;
cstate->raw_fields = (char **) palloc(nfields * sizeof(char *));
}
! MemoryContextSwitchTo(oldcontext);
! return cstate;
! }
! /*
! * Read raw fields in the next line for COPY FROM in text or csv mode.
! * Return false if no more lines.
! *
! * An internal temporary buffer is returned via 'fields'. It is valid until
! * the next call of the function. Since this function returns all raw fields
! * in the input file, 'nfields' could be different from the number of columns
! * in the relation. Oid of the tuple is returned with 'tupleOid' separately.
! *
! * NOTE: force_not_null option are not appled to the returned fields yet.
! */
! bool
! NextCopyFromRawFields(CopyState cstate,
! char ***fields, int *nfields, Oid *tupleOid)
! {
! char **field_strings;
! int fldct;
! int fieldno;
! char *string;
! bool done;
!
! /* only available for text or csv input */
! Assert(!cstate->binary);
/* on input just throw the header line away */
! if (cstate->cur_lineno == 0 && cstate->header_line)
{
cstate->cur_lineno++;
! if (CopyReadLine(cstate))
! return false; /* done */
}
! /* XXX: Indentation is not adjusted to keep the patch small. */
cstate->cur_lineno++;
/* Actually read the line into memory here */
done = CopyReadLine(cstate);
*************** CopyFrom(CopyState cstate)
*** 1999,2005 ****
* EOF, ie, process the line and then exit loop on next iteration.
*/
if (done && cstate->line_buf.len == 0)
! break;
/* Parse the line into de-escaped field values */
if (cstate->csv_mode)
--- 2254,2260 ----
* EOF, ie, process the line and then exit loop on next iteration.
*/
if (done && cstate->line_buf.len == 0)
! return false;
/* Parse the line into de-escaped field values */
if (cstate->csv_mode)
*************** CopyFrom(CopyState cstate)
*** 2007,2023 ****
else
fldct = CopyReadAttributesText(cstate);
- /* check for overflowing fields */
- if (nfields > 0 && fldct > nfields)
- ereport(ERROR,
- (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
- errmsg("extra data after last expected column")));
-
fieldno = 0;
field_strings = cstate->raw_fields;
/* Read the OID field if present */
! if (file_has_oids)
{
if (fieldno >= fldct)
ereport(ERROR,
--- 2262,2272 ----
else
fldct = CopyReadAttributesText(cstate);
fieldno = 0;
field_strings = cstate->raw_fields;
/* Read the OID field if present */
! if (cstate->file_has_oids)
{
if (fieldno >= fldct)
ereport(ERROR,
*************** CopyFrom(CopyState cstate)
*** 2029,2041 ****
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("null OID in COPY data")));
! else
{
cstate->cur_attname = "oid";
cstate->cur_attval = string;
! loaded_oid = DatumGetObjectId(DirectFunctionCall1(oidin,
! CStringGetDatum(string)));
! if (loaded_oid == InvalidOid)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid OID in COPY data")));
--- 2278,2290 ----
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("null OID in COPY data")));
! else if (cstate->oids && tupleOid != NULL)
{
cstate->cur_attname = "oid";
cstate->cur_attval = string;
! *tupleOid = DatumGetObjectId(DirectFunctionCall1(oidin,
! CStringGetDatum(string)));
! if (*tupleOid == InvalidOid)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid OID in COPY data")));
*************** CopyFrom(CopyState cstate)
*** 2043,2049 ****
--- 2292,2361 ----
cstate->cur_attval = NULL;
}
}
+ /* XXX: End of only-indentation changes. */
+
+ *fields = &field_strings[fieldno];
+ *nfields = fldct - fieldno;
+ return true;
+ }
+
+ /*
+ * Read next tuple from file for COPY FROM. Return false if no more tuples.
+ *
+ * 'values' and 'nulls' arrays must be the same length as columns of the
+ * relation passed to BeginCopyFrom. This function fills the arrays.
+ * Oid of the tuple is returned with 'tupleOid' separately.
+ */
+ bool
+ NextCopyFrom(CopyState cstate, Datum *values, bool *nulls, Oid *tupleOid)
+ {
+ TupleDesc tupDesc;
+ Form_pg_attribute *attr;
+ AttrNumber num_phys_attrs,
+ attr_count,
+ num_defaults = cstate->num_defaults;
+ FmgrInfo *in_functions = cstate->in_functions;
+ Oid *typioparams = cstate->typioparams;
+ int i;
+ bool isnull;
+ bool file_has_oids = cstate->file_has_oids;
+ int *defmap = cstate->defmap;
+ ExprState **defexprs = cstate->defexprs;
+ ExprContext *econtext; /* used for ExecEvalExpr for default atts */
+
+ /* Reset the per-tuple exprcontext */
+ ResetPerTupleExprContext(cstate->estate);
+
+ tupDesc = RelationGetDescr(cstate->rel);
+ attr = tupDesc->attrs;
+ num_phys_attrs = tupDesc->natts;
+ attr_count = list_length(cstate->attnumlist);
+
+ /* Initialize all values for row to NULL */
+ MemSet(values, 0, num_phys_attrs * sizeof(Datum));
+ MemSet(nulls, true, num_phys_attrs * sizeof(bool));
+
+ if (!cstate->binary)
+ {
+ char **field_strings;
+ ListCell *cur;
+ int fldct;
+ int fieldno;
+ char *string;
+
+ /* read raw fields in the next line */
+ if (!NextCopyFromRawFields(cstate, &field_strings, &fldct, tupleOid))
+ return false;
+
+ /* check for overflowing fields */
+ if (attr_count > 0 && fldct > attr_count)
+ ereport(ERROR,
+ (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+ errmsg("extra data after last expected column")));
+ fieldno = 0;
+
+ /* XXX: Indentation is not adjusted to keep the patch small. */
/* Loop to read the user attributes on the line. */
foreach(cur, cstate->attnumlist)
{
*************** CopyFrom(CopyState cstate)
*** 2076,2082 ****
cstate->cur_attval = NULL;
}
! Assert(fieldno == nfields);
}
else
{
--- 2388,2394 ----
cstate->cur_attval = NULL;
}
! Assert(fieldno == attr_count);
}
else
{
*************** CopyFrom(CopyState cstate)
*** 2084,2094 ****
int16 fld_count;
ListCell *cur;
if (!CopyGetInt16(cstate, &fld_count))
{
/* EOF detected (end of file, or protocol-level EOF) */
! done = true;
! break;
}
if (fld_count == -1)
--- 2396,2407 ----
int16 fld_count;
ListCell *cur;
+ cstate->cur_lineno++;
+
if (!CopyGetInt16(cstate, &fld_count))
{
/* EOF detected (end of file, or protocol-level EOF) */
! return false;
}
if (fld_count == -1)
*************** CopyFrom(CopyState cstate)
*** 2112,2119 ****
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("received copy data after EOF marker")));
! done = true;
! break;
}
if (fld_count != attr_count)
--- 2425,2431 ----
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("received copy data after EOF marker")));
! return false;
}
if (fld_count != attr_count)
*************** CopyFrom(CopyState cstate)
*** 2124,2135 ****
if (file_has_oids)
{
cstate->cur_attname = "oid";
loaded_oid =
DatumGetObjectId(CopyReadBinaryAttribute(cstate,
0,
! &oid_in_function,
! oid_typioparam,
-1,
&isnull));
if (isnull || loaded_oid == InvalidOid)
--- 2436,2449 ----
if (file_has_oids)
{
+ Oid loaded_oid;
+
cstate->cur_attname = "oid";
loaded_oid =
DatumGetObjectId(CopyReadBinaryAttribute(cstate,
0,
! &cstate->oid_in_function,
! cstate->oid_typioparam,
-1,
&isnull));
if (isnull || loaded_oid == InvalidOid)
*************** CopyFrom(CopyState cstate)
*** 2137,2142 ****
--- 2451,2458 ----
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid OID in COPY data")));
cstate->cur_attname = NULL;
+ if (cstate->oids && tupleOid != NULL)
+ *tupleOid = loaded_oid;
}
i = 0;
*************** CopyFrom(CopyState cstate)
*** 2162,2278 ****
* provided by the input data. Anything not processed here or above
* will remain NULL.
*/
for (i = 0; i < num_defaults; i++)
{
values[defmap[i]] = ExecEvalExpr(defexprs[i], econtext,
&nulls[defmap[i]], NULL);
}
! /* And now we can form the input tuple. */
! tuple = heap_form_tuple(tupDesc, values, nulls);
!
! if (cstate->oids && file_has_oids)
! HeapTupleSetOid(tuple, loaded_oid);
!
! /* Triggers and stuff need to be invoked in query context. */
! MemoryContextSwitchTo(oldcontext);
!
! skip_tuple = false;
!
! /* BEFORE ROW INSERT Triggers */
! if (resultRelInfo->ri_TrigDesc &&
! resultRelInfo->ri_TrigDesc->trig_insert_before_row)
! {
! HeapTuple newtuple;
!
! newtuple = ExecBRInsertTriggers(estate, resultRelInfo, tuple);
!
! if (newtuple == NULL) /* "do nothing" */
! skip_tuple = true;
! else if (newtuple != tuple) /* modified by Trigger(s) */
! {
! heap_freetuple(tuple);
! tuple = newtuple;
! }
! }
!
! if (!skip_tuple)
! {
! List *recheckIndexes = NIL;
!
! /* Place tuple in tuple slot */
! ExecStoreTuple(tuple, slot, InvalidBuffer, false);
!
! /* Check the constraints of the tuple */
! if (cstate->rel->rd_att->constr)
! ExecConstraints(resultRelInfo, slot, estate);
!
! /* OK, store the tuple and create index entries for it */
! heap_insert(cstate->rel, tuple, mycid, hi_options, bistate);
!
! if (resultRelInfo->ri_NumIndices > 0)
! recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self),
! estate);
!
! /* AFTER ROW INSERT Triggers */
! ExecARInsertTriggers(estate, resultRelInfo, tuple,
! recheckIndexes);
!
! list_free(recheckIndexes);
!
! /*
! * We count only tuples not suppressed by a BEFORE INSERT trigger;
! * this is the same definition used by execMain.c for counting
! * tuples inserted by an INSERT command.
! */
! cstate->processed++;
! }
! }
!
! /* Done, clean up */
! error_context_stack = errcontext.previous;
!
! FreeBulkInsertState(bistate);
!
! MemoryContextSwitchTo(oldcontext);
!
! /* Execute AFTER STATEMENT insertion triggers */
! ExecASInsertTriggers(estate, resultRelInfo);
!
! /* Handle queued AFTER triggers */
! AfterTriggerEndQuery(estate);
!
! pfree(values);
! pfree(nulls);
! if (! cstate->binary)
! pfree(cstate->raw_fields);
!
! pfree(in_functions);
! pfree(typioparams);
! pfree(defmap);
! pfree(defexprs);
!
! ExecResetTupleTable(estate->es_tupleTable, false);
!
! ExecCloseIndices(resultRelInfo);
!
! FreeExecutorState(estate);
! if (!pipe)
! {
! if (FreeFile(cstate->copy_file))
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not close file \"%s\": %m",
! cstate->filename)));
! }
! /*
! * If we skipped writing WAL, then we need to sync the heap (but not
! * indexes since those use WAL anyway)
! */
! if (hi_options & HEAP_INSERT_SKIP_WAL)
! heap_sync(cstate->rel);
}
--- 2478,2504 ----
* provided by the input data. Anything not processed here or above
* will remain NULL.
*/
+ econtext = GetPerTupleExprContext(cstate->estate);
for (i = 0; i < num_defaults; i++)
{
values[defmap[i]] = ExecEvalExpr(defexprs[i], econtext,
&nulls[defmap[i]], NULL);
}
+ /* XXX: End of only-indentation changes. */
! return true;
! }
! /*
! * Clean up storage and release resources for COPY FROM.
! */
! void
! EndCopyFrom(CopyState cstate)
! {
! FreeExecutorState(cstate->estate);
! /* Clean up storage */
! EndCopy(cstate);
}
*************** copy_dest_receive(TupleTableSlot *slot,
*** 3537,3542 ****
--- 3763,3769 ----
/* And send the data */
CopyOneRowTo(cstate, InvalidOid, slot->tts_values, slot->tts_isnull);
+ myState->processed++;
}
/*
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index 9e2bbe8..af6c868 100644
*** a/src/include/commands/copy.h
--- b/src/include/commands/copy.h
***************
*** 14,25 ****
--- 14,37 ----
#ifndef COPY_H
#define COPY_H
+ #include "nodes/execnodes.h"
#include "nodes/parsenodes.h"
#include "tcop/dest.h"
+ typedef struct CopyStateData *CopyState;
+
extern uint64 DoCopy(const CopyStmt *stmt, const char *queryString);
+ extern CopyState BeginCopyFrom(Relation rel, const char *filename,
+ List *attnamelist, List *options);
+ extern void EndCopyFrom(CopyState cstate);
+ extern bool NextCopyFrom(CopyState cstate,
+ Datum *values, bool *nulls, Oid *tupleOid);
+ extern bool NextCopyFromRawFields(CopyState cstate,
+ char ***fields, int *nfields, Oid *tupleOid);
+ extern void CopyFromErrorCallback(void *arg);
+
extern DestReceiver *CreateCopyDestReceiver(void);
#endif /* COPY_H */
On Tue, Feb 08, 2011 at 09:25:29PM +0900, Itagaki Takahiro wrote:
Here is a revised patch, that including jagged csv support.
A new exported function is named as NextCopyFromRawFields.
Seems a bit incongruous to handle the OID column in that function. That part
fits with the other per-column code in NextCopyFrom().
On Mon, Feb 7, 2011 at 21:16, Noah Misch <noah@leadboat.com> wrote:
Perhaps I'm missing something. ??The new API does not expose a "processed" count
at all; that count is used for the command tag of a top-level COPY. ??This part
of the patch is just changing how we structure the code to maintain that tally,
and it has no implications for observers outside copy.c. ??Right?True, but the counter is tightly bound with INSERT-side of COPY FROM.
| copy.c (1978)
| * We count only tuples not suppressed by a BEFORE INSERT trigger;I think it is cleaner way to separate it from CopyState
because it is used as a file reader rather than a writer.
However, if there are objections, I'd revert it.
No significant objection. I just wished to be clear on whether it was pure
refactoring, or something more.
ExecEvalExpr(), used to acquire default values, will still use the
per-tuple context of CopyState->estate. ??That per-tuple context will never get
reset explicitly, so default value computations leak until EndCopyFrom().Ah, I see. I didn't see the leak because we rarely use per-tuple memory
context in the estate. We just use CurrentMemoryContext in many cases.
But the leak could occur, and the code is misleading.
I moved ResetPerTupleExprContext() into NextCopyFrom(), but
CurrentMemoryContext still used in it to the result values.
The code still violates the contract of ExecEvalExpr() by calling it with
CurrentMemoryContext != econtext->ecxt_per_tuple_memory.
Another possible design might be passing EState as an argument of
NextCopyFrom and remove estate from CopyState. It seems a much cleaner way
in terms of control flow, but it requires more changes in file_fdw.
Comments?
Seems sensible and more-consistent with the typical structure of executor code.
Do we place any constraints on sharing of EState among different layers like
this? I could not identify any, offhand.
The new API uses EState for two things. First, BeginCopyFrom() passes it to
ExecPrepareExpr(). Instead, let's use expression_planner() + ExecInitExpr() and
require that we've been called with a memory context of suitable longevity.
Things will break anyway if BeginCopyFrom()'s CurrentMemoryContext gets reset
too early. This way, we no longer need an EState in BeginCopyFrom().
Second, NextCopyFrom() sends the per-output-tuple ExprContext of the EState to
ExecEvalExpr(). It really needs a specific ExprContext, not an EState. A
top-level COPY has a bijection between input and output tuples, making the
distinction unimportant. GetPerTupleExprContext() is wrong for a file_fdw scan,
though. We need the ExprContext of the ForeignScanState or another of
equivalent lifecycle. NextCopyFrom() would then require that it be called with
CurrentMemoryContext == econtext->ecxt_per_tuple_memory.
Sorry, I missed a lot of these memory details on my first couple of reviews.
nm
On Wed, Feb 9, 2011 at 03:49, Noah Misch <noah@leadboat.com> wrote:
A new exported function is named as NextCopyFromRawFields.
Seems a bit incongruous to handle the OID column in that function. That part
fits with the other per-column code in NextCopyFrom().
Hmmm, I thought oid columns should be separated from user columns, but it
might be a confusing interface. I removed the explicit oid output from
NextCopyFromRawFields. file_fdw won't use oids at all in any cases, though.
The code still violates the contract of ExecEvalExpr() by calling it with
CurrentMemoryContext != econtext->ecxt_per_tuple_memory.
I'm not sure whether we have such contract because the caller might
want to get the expression result in long-lived context. But anyway
using an external ExprContext looks cleaner. The new prototype is:
bool NextCopyFrom(
[IN] CopyState cstate, ExprContext *econtext,
[OUT] Datum *values, bool *nulls, Oid *tupleOid)
Note that econtext can be NULL if no default values are used.
Since file_fdw won't use default values, we can just pass NULL for it.
Sorry, I missed a lot of these memory details on my first couple of reviews.
You did great reviews! Thank you very much.
--
Itagaki Takahiro
Attachments:
copy_export-20110209.patchapplication/octet-stream; name=copy_export-20110209.patchDownload
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index 3350ca0..dbd084a 100644
*** a/src/backend/commands/copy.c
--- b/src/backend/commands/copy.c
*************** typedef struct CopyStateData
*** 93,105 ****
FILE *copy_file; /* used if copy_dest == COPY_FILE */
StringInfo fe_msgbuf; /* used for all dests during COPY TO, only for
* dest == COPY_NEW_FE in COPY FROM */
- bool fe_copy; /* true for all FE copy dests */
bool fe_eof; /* true if detected end of copy data */
EolType eol_type; /* EOL type of input */
int client_encoding; /* remote side's character encoding */
bool need_transcoding; /* client encoding diff from server? */
bool encoding_embeds_ascii; /* ASCII can be non-first byte? */
- uint64 processed; /* # of tuples processed */
/* parameters from the COPY command */
Relation rel; /* relation to copy to or from */
--- 93,103 ----
*************** typedef struct CopyStateData
*** 119,137 ****
bool *force_quote_flags; /* per-column CSV FQ flags */
bool *force_notnull_flags; /* per-column CSV FNN flags */
! /* these are just for error messages, see copy_in_error_callback */
const char *cur_relname; /* table name for error messages */
int cur_lineno; /* line number for error messages */
const char *cur_attname; /* current att for error messages */
const char *cur_attval; /* current att value for error messages */
/*
* Working state for COPY TO
*/
FmgrInfo *out_functions; /* lookup info for output functions */
MemoryContext rowcontext; /* per-row evaluation context */
/*
* These variables are used to reduce overhead in textual COPY FROM.
*
* attribute_buf holds the separated, de-escaped text for each field of
--- 117,152 ----
bool *force_quote_flags; /* per-column CSV FQ flags */
bool *force_notnull_flags; /* per-column CSV FNN flags */
! /* these are just for error messages, see CopyFromErrorCallback */
const char *cur_relname; /* table name for error messages */
int cur_lineno; /* line number for error messages */
const char *cur_attname; /* current att for error messages */
const char *cur_attval; /* current att value for error messages */
/*
+ * Working state for COPY TO/FROM
+ */
+ MemoryContext copycontext; /* per-copy execution context */
+
+ /*
* Working state for COPY TO
*/
FmgrInfo *out_functions; /* lookup info for output functions */
MemoryContext rowcontext; /* per-row evaluation context */
/*
+ * Working state for COPY FROM
+ */
+ AttrNumber num_defaults;
+ bool file_has_oids;
+ FmgrInfo oid_in_function;
+ Oid oid_typioparam;
+ FmgrInfo *in_functions; /* array of input functions for each attrs */
+ Oid *typioparams; /* array of element types for in_functions */
+ int *defmap; /* array of default att numbers */
+ ExprState **defexprs; /* array of default att expressions */
+
+ /*
* These variables are used to reduce overhead in textual COPY FROM.
*
* attribute_buf holds the separated, de-escaped text for each field of
*************** typedef struct CopyStateData
*** 169,181 ****
int raw_buf_len; /* total # of bytes stored */
} CopyStateData;
- typedef CopyStateData *CopyState;
-
/* DestReceiver for COPY (SELECT) TO */
typedef struct
{
DestReceiver pub; /* publicly-known function pointers */
CopyState cstate; /* CopyStateData for the command */
} DR_copy;
--- 184,195 ----
int raw_buf_len; /* total # of bytes stored */
} CopyStateData;
/* DestReceiver for COPY (SELECT) TO */
typedef struct
{
DestReceiver pub; /* publicly-known function pointers */
CopyState cstate; /* CopyStateData for the command */
+ uint64 processed; /* # of tuples processed */
} DR_copy;
*************** static const char BinarySignature[11] =
*** 248,258 ****
/* non-export function prototypes */
! static void DoCopyTo(CopyState cstate);
! static void CopyTo(CopyState cstate);
static void CopyOneRowTo(CopyState cstate, Oid tupleOid,
Datum *values, bool *nulls);
! static void CopyFrom(CopyState cstate);
static bool CopyReadLine(CopyState cstate);
static bool CopyReadLineText(CopyState cstate);
static int CopyReadAttributesText(CopyState cstate);
--- 262,278 ----
/* non-export function prototypes */
! static CopyState BeginCopy(bool is_from, Relation rel, Node *raw_query,
! const char *queryString, List *attnamelist, List *options);
! static void EndCopy(CopyState cstate);
! static CopyState BeginCopyTo(Relation rel, Node *query, const char *queryString,
! const char *filename, List *attnamelist, List *options);
! static void EndCopyTo(CopyState cstate);
! static uint64 DoCopyTo(CopyState cstate);
! static uint64 CopyTo(CopyState cstate);
static void CopyOneRowTo(CopyState cstate, Oid tupleOid,
Datum *values, bool *nulls);
! static uint64 CopyFrom(CopyState cstate);
static bool CopyReadLine(CopyState cstate);
static bool CopyReadLineText(CopyState cstate);
static int CopyReadAttributesText(CopyState cstate);
*************** CopyLoadRawBuf(CopyState cstate)
*** 700,705 ****
--- 720,821 ----
* input/output stream. The latter could be either stdin/stdout or a
* socket, depending on whether we're running under Postmaster control.
*
+ * Do not allow a Postgres user without superuser privilege to read from
+ * or write to a file.
+ *
+ * Do not allow the copy if user doesn't have proper permission to access
+ * the table or the specifically requested columns.
+ */
+ uint64
+ DoCopy(const CopyStmt *stmt, const char *queryString)
+ {
+ CopyState cstate;
+ bool is_from = stmt->is_from;
+ bool pipe = (stmt->filename == NULL);
+ Relation rel;
+ uint64 processed;
+
+ /* Disallow file COPY except to superusers. */
+ if (!pipe && !superuser())
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+ errmsg("must be superuser to COPY to or from a file"),
+ errhint("Anyone can COPY to stdout or from stdin. "
+ "psql's \\copy command also works for anyone.")));
+
+ if (stmt->relation)
+ {
+ TupleDesc tupDesc;
+ AclMode required_access = (is_from ? ACL_INSERT : ACL_SELECT);
+ RangeTblEntry *rte;
+ List *attnums;
+ ListCell *cur;
+
+ Assert(!stmt->query);
+
+ /* Open and lock the relation, using the appropriate lock type. */
+ rel = heap_openrv(stmt->relation,
+ (is_from ? RowExclusiveLock : AccessShareLock));
+
+ rte = makeNode(RangeTblEntry);
+ rte->rtekind = RTE_RELATION;
+ rte->relid = RelationGetRelid(rel);
+ rte->requiredPerms = required_access;
+
+ tupDesc = RelationGetDescr(rel);
+ attnums = CopyGetAttnums(tupDesc, rel, stmt->attlist);
+ foreach (cur, attnums)
+ {
+ int attno = lfirst_int(cur) -
+ FirstLowInvalidHeapAttributeNumber;
+
+ if (is_from)
+ rte->modifiedCols = bms_add_member(rte->modifiedCols, attno);
+ else
+ rte->selectedCols = bms_add_member(rte->selectedCols, attno);
+ }
+ ExecCheckRTPerms(list_make1(rte), true);
+ }
+ else
+ {
+ Assert(stmt->query);
+
+ rel = NULL;
+ }
+
+ if (is_from)
+ {
+ /* check read-only transaction */
+ if (XactReadOnly && rel->rd_backend != MyBackendId)
+ PreventCommandIfReadOnly("COPY FROM");
+
+ cstate = BeginCopyFrom(rel, stmt->filename,
+ stmt->attlist, stmt->options);
+ processed = CopyFrom(cstate); /* copy from file to database */
+ EndCopyFrom(cstate);
+ }
+ else
+ {
+ cstate = BeginCopyTo(rel, stmt->query, queryString, stmt->filename,
+ stmt->attlist, stmt->options);
+ processed = DoCopyTo(cstate); /* copy from database to file */
+ EndCopyTo(cstate);
+ }
+
+ /*
+ * Close the relation. If reading, we can release the AccessShareLock we got;
+ * if writing, we should hold the lock until end of transaction to ensure that
+ * updates will be committed before lock is released.
+ */
+ if (rel != NULL)
+ heap_close(rel, (is_from ? NoLock : AccessShareLock));
+
+ return processed;
+ }
+
+ /*
+ * Common setup routines used by BeginCopyFrom and BeginCopyTo.
+ *
* Iff <binary>, unload or reload in the binary format, as opposed to the
* more wasteful but more robust and portable text format.
*
*************** CopyLoadRawBuf(CopyState cstate)
*** 711,745 ****
*
* If in the text format, delimit columns with delimiter <delim> and print
* NULL values as <null_print>.
- *
- * Do not allow a Postgres user without superuser privilege to read from
- * or write to a file.
- *
- * Do not allow the copy if user doesn't have proper permission to access
- * the table or the specifically requested columns.
*/
! uint64
! DoCopy(const CopyStmt *stmt, const char *queryString)
{
CopyState cstate;
- bool is_from = stmt->is_from;
- bool pipe = (stmt->filename == NULL);
- List *attnamelist = stmt->attlist;
List *force_quote = NIL;
List *force_notnull = NIL;
bool force_quote_all = false;
bool format_specified = false;
- AclMode required_access = (is_from ? ACL_INSERT : ACL_SELECT);
ListCell *option;
TupleDesc tupDesc;
int num_phys_attrs;
! uint64 processed;
/* Allocate workspace and zero all fields */
cstate = (CopyStateData *) palloc0(sizeof(CopyStateData));
/* Extract options from the statement node tree */
! foreach(option, stmt->options)
{
DefElem *defel = (DefElem *) lfirst(option);
--- 827,868 ----
*
* If in the text format, delimit columns with delimiter <delim> and print
* NULL values as <null_print>.
*/
! static CopyState
! BeginCopy(bool is_from,
! Relation rel,
! Node *raw_query,
! const char *queryString,
! List *attnamelist,
! List *options)
{
CopyState cstate;
List *force_quote = NIL;
List *force_notnull = NIL;
bool force_quote_all = false;
bool format_specified = false;
ListCell *option;
TupleDesc tupDesc;
int num_phys_attrs;
! MemoryContext oldcontext;
/* Allocate workspace and zero all fields */
cstate = (CopyStateData *) palloc0(sizeof(CopyStateData));
+ /*
+ * We allocate everything used by a cstate in a new memory context.
+ * This would avoid memory leaks repeated uses of COPY in a query.
+ */
+ cstate->copycontext = AllocSetContextCreate(CurrentMemoryContext,
+ "COPY",
+ ALLOCSET_DEFAULT_MINSIZE,
+ ALLOCSET_DEFAULT_INITSIZE,
+ ALLOCSET_DEFAULT_MAXSIZE);
+
+ oldcontext = MemoryContextSwitchTo(cstate->copycontext);
+
/* Extract options from the statement node tree */
! foreach(option, options)
{
DefElem *defel = (DefElem *) lfirst(option);
*************** DoCopy(const CopyStmt *stmt, const char
*** 980,1030 ****
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("CSV quote character must not appear in the NULL specification")));
! /* Disallow file COPY except to superusers. */
! if (!pipe && !superuser())
! ereport(ERROR,
! (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
! errmsg("must be superuser to COPY to or from a file"),
! errhint("Anyone can COPY to stdout or from stdin. "
! "psql's \\copy command also works for anyone.")));
!
! if (stmt->relation)
{
! RangeTblEntry *rte;
! List *attnums;
! ListCell *cur;
!
! Assert(!stmt->query);
! cstate->queryDesc = NULL;
! /* Open and lock the relation, using the appropriate lock type. */
! cstate->rel = heap_openrv(stmt->relation,
! (is_from ? RowExclusiveLock : AccessShareLock));
tupDesc = RelationGetDescr(cstate->rel);
- /* Check relation permissions. */
- rte = makeNode(RangeTblEntry);
- rte->rtekind = RTE_RELATION;
- rte->relid = RelationGetRelid(cstate->rel);
- rte->requiredPerms = required_access;
-
- attnums = CopyGetAttnums(tupDesc, cstate->rel, attnamelist);
- foreach (cur, attnums)
- {
- int attno = lfirst_int(cur) - FirstLowInvalidHeapAttributeNumber;
-
- if (is_from)
- rte->modifiedCols = bms_add_member(rte->modifiedCols, attno);
- else
- rte->selectedCols = bms_add_member(rte->selectedCols, attno);
- }
- ExecCheckRTPerms(list_make1(rte), true);
-
- /* check read-only transaction */
- if (XactReadOnly && is_from && cstate->rel->rd_backend != MyBackendId)
- PreventCommandIfReadOnly("COPY FROM");
-
/* Don't allow COPY w/ OIDs to or from a table without them */
if (cstate->oids && !cstate->rel->rd_rel->relhasoids)
ereport(ERROR,
--- 1103,1116 ----
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("CSV quote character must not appear in the NULL specification")));
! if (rel)
{
! Assert(!raw_query);
! cstate->rel = rel;
tupDesc = RelationGetDescr(cstate->rel);
/* Don't allow COPY w/ OIDs to or from a table without them */
if (cstate->oids && !cstate->rel->rd_rel->relhasoids)
ereport(ERROR,
*************** DoCopy(const CopyStmt *stmt, const char
*** 1058,1064 ****
* function and is executed repeatedly. (See also the same hack in
* DECLARE CURSOR and PREPARE.) XXX FIXME someday.
*/
! rewritten = pg_analyze_and_rewrite((Node *) copyObject(stmt->query),
queryString, NULL, 0);
/* We don't expect more or less than one result query */
--- 1144,1150 ----
* function and is executed repeatedly. (See also the same hack in
* DECLARE CURSOR and PREPARE.) XXX FIXME someday.
*/
! rewritten = pg_analyze_and_rewrite((Node *) copyObject(raw_query),
queryString, NULL, 0);
/* We don't expect more or less than one result query */
*************** DoCopy(const CopyStmt *stmt, const char
*** 1160,1173 ****
}
}
- /* Set up variables to avoid per-attribute overhead. */
- initStringInfo(&cstate->attribute_buf);
- initStringInfo(&cstate->line_buf);
- cstate->line_buf_converted = false;
- cstate->raw_buf = (char *) palloc(RAW_BUF_SIZE + 1);
- cstate->raw_buf_index = cstate->raw_buf_len = 0;
- cstate->processed = 0;
-
/*
* Set up encoding conversion info. Even if the client and server
* encodings are the same, we must apply pg_client_to_server() to validate
--- 1246,1251 ----
*************** DoCopy(const CopyStmt *stmt, const char
*** 1181,1264 ****
cstate->encoding_embeds_ascii = PG_ENCODING_IS_CLIENT_ONLY(cstate->client_encoding);
cstate->copy_dest = COPY_FILE; /* default */
- cstate->filename = stmt->filename;
! if (is_from)
! CopyFrom(cstate); /* copy from file to database */
! else
! DoCopyTo(cstate); /* copy from database to file */
! /*
! * Close the relation or query. If reading, we can release the
! * AccessShareLock we got; if writing, we should hold the lock until end
! * of transaction to ensure that updates will be committed before lock is
! * released.
! */
! if (cstate->rel)
! heap_close(cstate->rel, (is_from ? NoLock : AccessShareLock));
! else
! {
! /* Close down the query and free resources. */
! ExecutorEnd(cstate->queryDesc);
! FreeQueryDesc(cstate->queryDesc);
! PopActiveSnapshot();
! }
! /* Clean up storage (probably not really necessary) */
! processed = cstate->processed;
! pfree(cstate->attribute_buf.data);
! pfree(cstate->line_buf.data);
! pfree(cstate->raw_buf);
pfree(cstate);
-
- return processed;
}
-
/*
! * This intermediate routine exists mainly to localize the effects of setjmp
! * so we don't need to plaster a lot of variables with "volatile".
*/
! static void
! DoCopyTo(CopyState cstate)
{
! bool pipe = (cstate->filename == NULL);
! if (cstate->rel)
{
! if (cstate->rel->rd_rel->relkind != RELKIND_RELATION)
! {
! if (cstate->rel->rd_rel->relkind == RELKIND_VIEW)
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from view \"%s\"",
! RelationGetRelationName(cstate->rel)),
! errhint("Try the COPY (SELECT ...) TO variant.")));
! else if (cstate->rel->rd_rel->relkind == RELKIND_FOREIGN_TABLE)
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from foreign table \"%s\"",
! RelationGetRelationName(cstate->rel)),
! errhint("Try the COPY (SELECT ...) TO variant.")));
! else if (cstate->rel->rd_rel->relkind == RELKIND_SEQUENCE)
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from sequence \"%s\"",
! RelationGetRelationName(cstate->rel))));
! else
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from non-table relation \"%s\"",
! RelationGetRelationName(cstate->rel))));
! }
}
if (pipe)
{
! if (whereToSendOutput == DestRemote)
! cstate->fe_copy = true;
! else
cstate->copy_file = stdout;
}
else
--- 1259,1333 ----
cstate->encoding_embeds_ascii = PG_ENCODING_IS_CLIENT_ONLY(cstate->client_encoding);
cstate->copy_dest = COPY_FILE; /* default */
! MemoryContextSwitchTo(oldcontext);
! return cstate;
! }
! /*
! * Release resources allocated in a cstate.
! */
! static void
! EndCopy(CopyState cstate)
! {
! if (cstate->filename != NULL && FreeFile(cstate->copy_file))
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not close file \"%s\": %m",
! cstate->filename)));
! MemoryContextDelete(cstate->copycontext);
pfree(cstate);
}
/*
! * Setup CopyState to read tuples from a table or a query for COPY TO.
*/
! static CopyState
! BeginCopyTo(Relation rel,
! Node *query,
! const char *queryString,
! const char *filename,
! List *attnamelist,
! List *options)
{
! CopyState cstate;
! bool pipe = (filename == NULL);
! MemoryContext oldcontext;
! if (rel != NULL && rel->rd_rel->relkind != RELKIND_RELATION)
{
! if (rel->rd_rel->relkind == RELKIND_VIEW)
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from view \"%s\"",
! RelationGetRelationName(rel)),
! errhint("Try the COPY (SELECT ...) TO variant.")));
! else if (rel->rd_rel->relkind == RELKIND_FOREIGN_TABLE)
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from foreign table \"%s\"",
! RelationGetRelationName(rel)),
! errhint("Try the COPY (SELECT ...) TO variant.")));
! else if (rel->rd_rel->relkind == RELKIND_SEQUENCE)
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from sequence \"%s\"",
! RelationGetRelationName(rel))));
! else
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from non-table relation \"%s\"",
! RelationGetRelationName(rel))));
}
+ cstate = BeginCopy(false, rel, query, queryString, attnamelist, options);
+ oldcontext = MemoryContextSwitchTo(cstate->copycontext);
+
if (pipe)
{
! if (whereToSendOutput != DestRemote)
cstate->copy_file = stdout;
}
else
*************** DoCopyTo(CopyState cstate)
*** 1270,1280 ****
* Prevent write to relative path ... too easy to shoot oneself in the
* foot by overwriting a database file ...
*/
! if (!is_absolute_path(cstate->filename))
ereport(ERROR,
(errcode(ERRCODE_INVALID_NAME),
errmsg("relative path not allowed for COPY to file")));
oumask = umask(S_IWGRP | S_IWOTH);
cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_W);
umask(oumask);
--- 1339,1350 ----
* Prevent write to relative path ... too easy to shoot oneself in the
* foot by overwriting a database file ...
*/
! if (!is_absolute_path(filename))
ereport(ERROR,
(errcode(ERRCODE_INVALID_NAME),
errmsg("relative path not allowed for COPY to file")));
+ cstate->filename = pstrdup(filename);
oumask = umask(S_IWGRP | S_IWOTH);
cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_W);
umask(oumask);
*************** DoCopyTo(CopyState cstate)
*** 1292,1305 ****
errmsg("\"%s\" is a directory", cstate->filename)));
}
PG_TRY();
{
! if (cstate->fe_copy)
SendCopyBegin(cstate);
! CopyTo(cstate);
! if (cstate->fe_copy)
SendCopyEnd(cstate);
}
PG_CATCH();
--- 1362,1391 ----
errmsg("\"%s\" is a directory", cstate->filename)));
}
+ MemoryContextSwitchTo(oldcontext);
+
+ return cstate;
+ }
+
+ /*
+ * This intermediate routine exists mainly to localize the effects of setjmp
+ * so we don't need to plaster a lot of variables with "volatile".
+ */
+ static uint64
+ DoCopyTo(CopyState cstate)
+ {
+ bool pipe = (cstate->filename == NULL);
+ bool fe_copy = (pipe && whereToSendOutput == DestRemote);
+ uint64 processed;
+
PG_TRY();
{
! if (fe_copy)
SendCopyBegin(cstate);
! processed = CopyTo(cstate);
! if (fe_copy)
SendCopyEnd(cstate);
}
PG_CATCH();
*************** DoCopyTo(CopyState cstate)
*** 1314,1339 ****
}
PG_END_TRY();
! if (!pipe)
{
! if (FreeFile(cstate->copy_file))
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not close file \"%s\": %m",
! cstate->filename)));
}
}
/*
* Copy from relation or query TO file.
*/
! static void
CopyTo(CopyState cstate)
{
TupleDesc tupDesc;
int num_phys_attrs;
Form_pg_attribute *attr;
ListCell *cur;
if (cstate->rel)
tupDesc = RelationGetDescr(cstate->rel);
--- 1400,1437 ----
}
PG_END_TRY();
! return processed;
! }
!
! /*
! * Clean up storage and release resources for COPY TO.
! */
! static void
! EndCopyTo(CopyState cstate)
! {
! if (cstate->queryDesc != NULL)
{
! /* Close down the query and free resources. */
! ExecutorEnd(cstate->queryDesc);
! FreeQueryDesc(cstate->queryDesc);
! PopActiveSnapshot();
}
+
+ /* Clean up storage */
+ EndCopy(cstate);
}
/*
* Copy from relation or query TO file.
*/
! static uint64
CopyTo(CopyState cstate)
{
TupleDesc tupDesc;
int num_phys_attrs;
Form_pg_attribute *attr;
ListCell *cur;
+ uint64 processed;
if (cstate->rel)
tupDesc = RelationGetDescr(cstate->rel);
*************** CopyTo(CopyState cstate)
*** 1439,1444 ****
--- 1537,1543 ----
scandesc = heap_beginscan(cstate->rel, GetActiveSnapshot(), 0, NULL);
+ processed = 0;
while ((tuple = heap_getnext(scandesc, ForwardScanDirection)) != NULL)
{
CHECK_FOR_INTERRUPTS();
*************** CopyTo(CopyState cstate)
*** 1448,1461 ****
--- 1547,1565 ----
/* Format and send the data */
CopyOneRowTo(cstate, HeapTupleGetOid(tuple), values, nulls);
+ processed++;
}
heap_endscan(scandesc);
+
+ pfree(values);
+ pfree(nulls);
}
else
{
/* run the plan --- the dest receiver will send tuples */
ExecutorRun(cstate->queryDesc, ForwardScanDirection, 0L);
+ processed = ((DR_copy *) cstate->queryDesc->dest)->processed;
}
if (cstate->binary)
*************** CopyTo(CopyState cstate)
*** 1467,1472 ****
--- 1571,1578 ----
}
MemoryContextDelete(cstate->rowcontext);
+
+ return processed;
}
/*
*************** CopyOneRowTo(CopyState cstate, Oid tuple
*** 1558,1573 ****
CopySendEndOfRow(cstate);
MemoryContextSwitchTo(oldcontext);
-
- cstate->processed++;
}
/*
* error context callback for COPY FROM
*/
! static void
! copy_in_error_callback(void *arg)
{
CopyState cstate = (CopyState) arg;
--- 1664,1679 ----
CopySendEndOfRow(cstate);
MemoryContextSwitchTo(oldcontext);
}
/*
* error context callback for COPY FROM
+ *
+ * The argument for the error context must be CopyState.
*/
! void
! CopyFromErrorCallback(void *arg)
{
CopyState cstate = (CopyState) arg;
*************** limit_printout_length(const char *str)
*** 1669,1709 ****
/*
* Copy FROM file to relation.
*/
! static void
CopyFrom(CopyState cstate)
{
- bool pipe = (cstate->filename == NULL);
HeapTuple tuple;
TupleDesc tupDesc;
- Form_pg_attribute *attr;
- AttrNumber num_phys_attrs,
- attr_count,
- num_defaults;
- FmgrInfo *in_functions;
- FmgrInfo oid_in_function;
- Oid *typioparams;
- Oid oid_typioparam;
- int attnum;
- int i;
- Oid in_func_oid;
Datum *values;
bool *nulls;
- int nfields;
- char **field_strings;
- bool done = false;
- bool isnull;
ResultRelInfo *resultRelInfo;
EState *estate = CreateExecutorState(); /* for ExecConstraints() */
TupleTableSlot *slot;
- bool file_has_oids;
- int *defmap;
- ExprState **defexprs; /* array of default att expressions */
- ExprContext *econtext; /* used for ExecEvalExpr for default atts */
MemoryContext oldcontext = CurrentMemoryContext;
ErrorContextCallback errcontext;
CommandId mycid = GetCurrentCommandId(true);
int hi_options = 0; /* start with default heap_insert options */
BulkInsertState bistate;
Assert(cstate->rel);
--- 1775,1797 ----
/*
* Copy FROM file to relation.
*/
! static uint64
CopyFrom(CopyState cstate)
{
HeapTuple tuple;
TupleDesc tupDesc;
Datum *values;
bool *nulls;
ResultRelInfo *resultRelInfo;
EState *estate = CreateExecutorState(); /* for ExecConstraints() */
+ ExprContext *econtext;
TupleTableSlot *slot;
MemoryContext oldcontext = CurrentMemoryContext;
ErrorContextCallback errcontext;
CommandId mycid = GetCurrentCommandId(true);
int hi_options = 0; /* start with default heap_insert options */
BulkInsertState bistate;
+ uint64 processed = 0;
Assert(cstate->rel);
*************** CopyFrom(CopyState cstate)
*** 1731,1736 ****
--- 1819,1826 ----
RelationGetRelationName(cstate->rel))));
}
+ tupDesc = RelationGetDescr(cstate->rel);
+
/*----------
* Check to see if we can avoid writing WAL
*
*************** CopyFrom(CopyState cstate)
*** 1766,1803 ****
hi_options |= HEAP_INSERT_SKIP_WAL;
}
- if (pipe)
- {
- if (whereToSendOutput == DestRemote)
- ReceiveCopyBegin(cstate);
- else
- cstate->copy_file = stdin;
- }
- else
- {
- struct stat st;
-
- cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_R);
-
- if (cstate->copy_file == NULL)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not open file \"%s\" for reading: %m",
- cstate->filename)));
-
- fstat(fileno(cstate->copy_file), &st);
- if (S_ISDIR(st.st_mode))
- ereport(ERROR,
- (errcode(ERRCODE_WRONG_OBJECT_TYPE),
- errmsg("\"%s\" is a directory", cstate->filename)));
- }
-
- tupDesc = RelationGetDescr(cstate->rel);
- attr = tupDesc->attrs;
- num_phys_attrs = tupDesc->natts;
- attr_count = list_length(cstate->attnumlist);
- num_defaults = 0;
-
/*
* We need a ResultRelInfo so we can use the regular executor's
* index-entry-making machinery. (There used to be a huge amount of code
--- 1856,1861 ----
*************** CopyFrom(CopyState cstate)
*** 1826,1833 ****
--- 1884,2074 ----
slot = ExecInitExtraTupleSlot(estate);
ExecSetSlotDescriptor(slot, tupDesc);
+ /* Prepare to catch AFTER triggers. */
+ AfterTriggerBeginQuery();
+
+ /*
+ * Check BEFORE STATEMENT insertion triggers. It's debateable whether we
+ * should do this for COPY, since it's not really an "INSERT" statement as
+ * such. However, executing these triggers maintains consistency with the
+ * EACH ROW triggers that we already fire on COPY.
+ */
+ ExecBSInsertTriggers(estate, resultRelInfo);
+
+ values = (Datum *) palloc(tupDesc->natts * sizeof(Datum));
+ nulls = (bool *) palloc(tupDesc->natts * sizeof(bool));
+
+ bistate = GetBulkInsertState();
econtext = GetPerTupleExprContext(estate);
+ /* Set up callback to identify error line number */
+ errcontext.callback = CopyFromErrorCallback;
+ errcontext.arg = (void *) cstate;
+ errcontext.previous = error_context_stack;
+ error_context_stack = &errcontext;
+
+ for (;;)
+ {
+ bool skip_tuple;
+ Oid loaded_oid = InvalidOid;
+
+ CHECK_FOR_INTERRUPTS();
+
+ /* Reset the per-tuple exprcontext */
+ ResetPerTupleExprContext(estate);
+
+ /* Switch into its memory context */
+ MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+
+ if (!NextCopyFrom(cstate, econtext, values, nulls, &loaded_oid))
+ break;
+
+ /* And now we can form the input tuple. */
+ tuple = heap_form_tuple(tupDesc, values, nulls);
+
+ if (loaded_oid != InvalidOid)
+ HeapTupleSetOid(tuple, loaded_oid);
+
+ /* Triggers and stuff need to be invoked in query context. */
+ MemoryContextSwitchTo(oldcontext);
+
+ skip_tuple = false;
+
+ /* BEFORE ROW INSERT Triggers */
+ if (resultRelInfo->ri_TrigDesc &&
+ resultRelInfo->ri_TrigDesc->trig_insert_before_row)
+ {
+ HeapTuple newtuple;
+
+ newtuple = ExecBRInsertTriggers(estate, resultRelInfo, tuple);
+
+ if (newtuple == NULL) /* "do nothing" */
+ skip_tuple = true;
+ else if (newtuple != tuple) /* modified by Trigger(s) */
+ {
+ heap_freetuple(tuple);
+ tuple = newtuple;
+ }
+ }
+
+ if (!skip_tuple)
+ {
+ List *recheckIndexes = NIL;
+
+ /* Place tuple in tuple slot */
+ ExecStoreTuple(tuple, slot, InvalidBuffer, false);
+
+ /* Check the constraints of the tuple */
+ if (cstate->rel->rd_att->constr)
+ ExecConstraints(resultRelInfo, slot, estate);
+
+ /* OK, store the tuple and create index entries for it */
+ heap_insert(cstate->rel, tuple, mycid, hi_options, bistate);
+
+ if (resultRelInfo->ri_NumIndices > 0)
+ recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self),
+ estate);
+
+ /* AFTER ROW INSERT Triggers */
+ ExecARInsertTriggers(estate, resultRelInfo, tuple,
+ recheckIndexes);
+
+ list_free(recheckIndexes);
+
+ /*
+ * We count only tuples not suppressed by a BEFORE INSERT trigger;
+ * this is the same definition used by execMain.c for counting
+ * tuples inserted by an INSERT command.
+ */
+ processed++;
+ }
+ }
+
+ /* Done, clean up */
+ error_context_stack = errcontext.previous;
+
+ FreeBulkInsertState(bistate);
+
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Execute AFTER STATEMENT insertion triggers */
+ ExecASInsertTriggers(estate, resultRelInfo);
+
+ /* Handle queued AFTER triggers */
+ AfterTriggerEndQuery(estate);
+
+ pfree(values);
+ pfree(nulls);
+
+ ExecResetTupleTable(estate->es_tupleTable, false);
+
+ ExecCloseIndices(resultRelInfo);
+
+ FreeExecutorState(estate);
+
+ /*
+ * If we skipped writing WAL, then we need to sync the heap (but not
+ * indexes since those use WAL anyway)
+ */
+ if (hi_options & HEAP_INSERT_SKIP_WAL)
+ heap_sync(cstate->rel);
+
+ return processed;
+ }
+
+ /*
+ * Setup to read tuples from a file for COPY FROM.
+ *
+ * 'rel': Used as a template for the tuples
+ * 'filename': Name of server-local file to read
+ * 'attnamelist': List of char *, columns to include. NIL selects all cols.
+ * 'options': List of DefElem. See copy_opt_item in gram.y for selections.
+ *
+ * Returns a CopyState, to be passed to NextCopyFrom.
+ */
+ CopyState
+ BeginCopyFrom(Relation rel,
+ const char *filename,
+ List *attnamelist,
+ List *options)
+ {
+ CopyState cstate;
+ bool pipe = (filename == NULL);
+ TupleDesc tupDesc;
+ Form_pg_attribute *attr;
+ AttrNumber num_phys_attrs,
+ num_defaults;
+ FmgrInfo *in_functions;
+ Oid *typioparams;
+ int attnum;
+ Oid in_func_oid;
+ int *defmap;
+ ExprState **defexprs;
+ MemoryContext oldcontext;
+
+ cstate = BeginCopy(true, rel, NULL, NULL, attnamelist, options);
+ oldcontext = MemoryContextSwitchTo(cstate->copycontext);
+
+ /* Initialize state variables */
+ cstate->fe_eof = false;
+ cstate->eol_type = EOL_UNKNOWN;
+ cstate->cur_relname = RelationGetRelationName(cstate->rel);
+ cstate->cur_lineno = 0;
+ cstate->cur_attname = NULL;
+ cstate->cur_attval = NULL;
+
+ /* Set up variables to avoid per-attribute overhead. */
+ initStringInfo(&cstate->attribute_buf);
+ initStringInfo(&cstate->line_buf);
+ cstate->line_buf_converted = false;
+ cstate->raw_buf = (char *) palloc(RAW_BUF_SIZE + 1);
+ cstate->raw_buf_index = cstate->raw_buf_len = 0;
+
+ tupDesc = RelationGetDescr(cstate->rel);
+ attr = tupDesc->attrs;
+ num_phys_attrs = tupDesc->natts;
+ num_defaults = 0;
+
/*
* Pick up the required catalog information for each attribute in the
* relation, including the input function, the element type (to pass to
*************** CopyFrom(CopyState cstate)
*** 1863,1889 ****
if (defexpr != NULL)
{
! defexprs[num_defaults] = ExecPrepareExpr((Expr *) defexpr,
! estate);
defmap[num_defaults] = attnum - 1;
num_defaults++;
}
}
}
! /* Prepare to catch AFTER triggers. */
! AfterTriggerBeginQuery();
! /*
! * Check BEFORE STATEMENT insertion triggers. It's debateable whether we
! * should do this for COPY, since it's not really an "INSERT" statement as
! * such. However, executing these triggers maintains consistency with the
! * EACH ROW triggers that we already fire on COPY.
! */
! ExecBSInsertTriggers(estate, resultRelInfo);
if (!cstate->binary)
! file_has_oids = cstate->oids; /* must rely on user to tell us... */
else
{
/* Read and verify binary header */
--- 2104,2157 ----
if (defexpr != NULL)
{
! /* Initialize expressions in copycontext. */
! defexprs[num_defaults] = ExecInitExpr(
! expression_planner((Expr *) defexpr), NULL);
defmap[num_defaults] = attnum - 1;
num_defaults++;
}
}
}
! /* We keep those variables in cstate. */
! cstate->in_functions = in_functions;
! cstate->typioparams = typioparams;
! cstate->defmap = defmap;
! cstate->defexprs = defexprs;
! cstate->num_defaults = num_defaults;
! if (pipe)
! {
! if (whereToSendOutput == DestRemote)
! ReceiveCopyBegin(cstate);
! else
! cstate->copy_file = stdin;
! }
! else
! {
! struct stat st;
!
! cstate->filename = pstrdup(filename);
! cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_R);
!
! if (cstate->copy_file == NULL)
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not open file \"%s\" for reading: %m",
! cstate->filename)));
!
! fstat(fileno(cstate->copy_file), &st);
! if (S_ISDIR(st.st_mode))
! ereport(ERROR,
! (errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("\"%s\" is a directory", cstate->filename)));
! }
if (!cstate->binary)
! {
! /* must rely on user to tell us... */
! cstate->file_has_oids = cstate->oids;
! }
else
{
/* Read and verify binary header */
*************** CopyFrom(CopyState cstate)
*** 1901,1907 ****
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid COPY file header (missing flags)")));
! file_has_oids = (tmp & (1 << 16)) != 0;
tmp &= ~(1 << 16);
if ((tmp >> 16) != 0)
ereport(ERROR,
--- 2169,2175 ----
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid COPY file header (missing flags)")));
! cstate->file_has_oids = (tmp & (1 << 16)) != 0;
tmp &= ~(1 << 16);
if ((tmp >> 16) != 0)
ereport(ERROR,
*************** CopyFrom(CopyState cstate)
*** 1923,2021 ****
}
}
! if (file_has_oids && cstate->binary)
{
getTypeBinaryInputInfo(OIDOID,
! &in_func_oid, &oid_typioparam);
! fmgr_info(in_func_oid, &oid_in_function);
}
- values = (Datum *) palloc(num_phys_attrs * sizeof(Datum));
- nulls = (bool *) palloc(num_phys_attrs * sizeof(bool));
-
/* create workspace for CopyReadAttributes results */
! nfields = file_has_oids ? (attr_count + 1) : attr_count;
! if (! cstate->binary)
{
cstate->max_fields = nfields;
cstate->raw_fields = (char **) palloc(nfields * sizeof(char *));
}
! /* Initialize state variables */
! cstate->fe_eof = false;
! cstate->eol_type = EOL_UNKNOWN;
! cstate->cur_relname = RelationGetRelationName(cstate->rel);
! cstate->cur_lineno = 0;
! cstate->cur_attname = NULL;
! cstate->cur_attval = NULL;
! bistate = GetBulkInsertState();
! /* Set up callback to identify error line number */
! errcontext.callback = copy_in_error_callback;
! errcontext.arg = (void *) cstate;
! errcontext.previous = error_context_stack;
! error_context_stack = &errcontext;
/* on input just throw the header line away */
! if (cstate->header_line)
{
cstate->cur_lineno++;
! done = CopyReadLine(cstate);
}
! while (!done)
! {
! bool skip_tuple;
! Oid loaded_oid = InvalidOid;
! CHECK_FOR_INTERRUPTS();
! cstate->cur_lineno++;
! /* Reset the per-tuple exprcontext */
! ResetPerTupleExprContext(estate);
! /* Switch into its memory context */
! MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
! /* Initialize all values for row to NULL */
! MemSet(values, 0, num_phys_attrs * sizeof(Datum));
! MemSet(nulls, true, num_phys_attrs * sizeof(bool));
! if (!cstate->binary)
! {
! ListCell *cur;
! int fldct;
! int fieldno;
! char *string;
! /* Actually read the line into memory here */
! done = CopyReadLine(cstate);
! /*
! * EOF at start of line means we're done. If we see EOF after
! * some characters, we act as though it was newline followed by
! * EOF, ie, process the line and then exit loop on next iteration.
! */
! if (done && cstate->line_buf.len == 0)
! break;
! /* Parse the line into de-escaped field values */
! if (cstate->csv_mode)
! fldct = CopyReadAttributesCSV(cstate);
! else
! fldct = CopyReadAttributesText(cstate);
! /* check for overflowing fields */
! if (nfields > 0 && fldct > nfields)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("extra data after last expected column")));
! fieldno = 0;
! field_strings = cstate->raw_fields;
/* Read the OID field if present */
if (file_has_oids)
{
--- 2191,2330 ----
}
}
! if (cstate->file_has_oids && cstate->binary)
{
getTypeBinaryInputInfo(OIDOID,
! &in_func_oid, &cstate->oid_typioparam);
! fmgr_info(in_func_oid, &cstate->oid_in_function);
}
/* create workspace for CopyReadAttributes results */
! if (!cstate->binary)
{
+ AttrNumber attr_count = list_length(cstate->attnumlist);
+ int nfields = cstate->file_has_oids ? (attr_count + 1) : attr_count;
+
cstate->max_fields = nfields;
cstate->raw_fields = (char **) palloc(nfields * sizeof(char *));
}
! MemoryContextSwitchTo(oldcontext);
! return cstate;
! }
! /*
! * Read raw fields in the next line for COPY FROM in text or csv mode.
! * Return false if no more lines.
! *
! * An internal temporary buffer is returned via 'fields'. It is valid until
! * the next call of the function. Since this function returns all raw fields
! * in the input file, 'nfields' could be different from the number of columns
! * in the relation.
! *
! * NOTE: force_not_null option are not appled to the returned fields yet.
! */
! bool
! NextCopyFromRawFields(CopyState cstate, char ***fields, int *nfields)
! {
! int fldct;
! bool done;
!
! /* only available for text or csv input */
! Assert(!cstate->binary);
/* on input just throw the header line away */
! if (cstate->cur_lineno == 0 && cstate->header_line)
{
cstate->cur_lineno++;
! if (CopyReadLine(cstate))
! return false; /* done */
}
! cstate->cur_lineno++;
! /* Actually read the line into memory here */
! done = CopyReadLine(cstate);
! /*
! * EOF at start of line means we're done. If we see EOF after
! * some characters, we act as though it was newline followed by
! * EOF, ie, process the line and then exit loop on next iteration.
! */
! if (done && cstate->line_buf.len == 0)
! return false;
! /* Parse the line into de-escaped field values */
! if (cstate->csv_mode)
! fldct = CopyReadAttributesCSV(cstate);
! else
! fldct = CopyReadAttributesText(cstate);
! *fields = cstate->raw_fields;
! *nfields = fldct;
! return true;
! }
! /*
! * Read next tuple from file for COPY FROM. Return false if no more tuples.
! *
! * 'econtext' is used to evaluate default expression for each columns not
! * read from the file. It can be NULL when no default values are used, i.e.
! * when all columns are read from the file.
! *
! * 'values' and 'nulls' arrays must be the same length as columns of the
! * relation passed to BeginCopyFrom. This function fills the arrays.
! * Oid of the tuple is returned with 'tupleOid' separately.
! */
! bool
! NextCopyFrom(CopyState cstate, ExprContext *econtext,
! Datum *values, bool *nulls, Oid *tupleOid)
! {
! TupleDesc tupDesc;
! Form_pg_attribute *attr;
! AttrNumber num_phys_attrs,
! attr_count,
! num_defaults = cstate->num_defaults;
! FmgrInfo *in_functions = cstate->in_functions;
! Oid *typioparams = cstate->typioparams;
! int i;
! int nfields;
! bool isnull;
! bool file_has_oids = cstate->file_has_oids;
! int *defmap = cstate->defmap;
! ExprState **defexprs = cstate->defexprs;
! tupDesc = RelationGetDescr(cstate->rel);
! attr = tupDesc->attrs;
! num_phys_attrs = tupDesc->natts;
! attr_count = list_length(cstate->attnumlist);
! nfields = file_has_oids ? (attr_count + 1) : attr_count;
! /* Initialize all values for row to NULL */
! MemSet(values, 0, num_phys_attrs * sizeof(Datum));
! MemSet(nulls, true, num_phys_attrs * sizeof(bool));
! if (!cstate->binary)
! {
! char **field_strings;
! ListCell *cur;
! int fldct;
! int fieldno;
! char *string;
! /* read raw fields in the next line */
! if (!NextCopyFromRawFields(cstate, &field_strings, &fldct))
! return false;
! /* check for overflowing fields */
! if (nfields > 0 && fldct > nfields)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("extra data after last expected column")));
! fieldno = 0;
+ /* XXX: Indentation is not adjusted to keep the patch small. */
/* Read the OID field if present */
if (file_has_oids)
{
*************** CopyFrom(CopyState cstate)
*** 2029,2041 ****
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("null OID in COPY data")));
! else
{
cstate->cur_attname = "oid";
cstate->cur_attval = string;
! loaded_oid = DatumGetObjectId(DirectFunctionCall1(oidin,
! CStringGetDatum(string)));
! if (loaded_oid == InvalidOid)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid OID in COPY data")));
--- 2338,2350 ----
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("null OID in COPY data")));
! else if (cstate->oids && tupleOid != NULL)
{
cstate->cur_attname = "oid";
cstate->cur_attval = string;
! *tupleOid = DatumGetObjectId(DirectFunctionCall1(oidin,
! CStringGetDatum(string)));
! if (*tupleOid == InvalidOid)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid OID in COPY data")));
*************** CopyFrom(CopyState cstate)
*** 2084,2094 ****
int16 fld_count;
ListCell *cur;
if (!CopyGetInt16(cstate, &fld_count))
{
/* EOF detected (end of file, or protocol-level EOF) */
! done = true;
! break;
}
if (fld_count == -1)
--- 2393,2404 ----
int16 fld_count;
ListCell *cur;
+ cstate->cur_lineno++;
+
if (!CopyGetInt16(cstate, &fld_count))
{
/* EOF detected (end of file, or protocol-level EOF) */
! return false;
}
if (fld_count == -1)
*************** CopyFrom(CopyState cstate)
*** 2112,2119 ****
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("received copy data after EOF marker")));
! done = true;
! break;
}
if (fld_count != attr_count)
--- 2422,2428 ----
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("received copy data after EOF marker")));
! return false;
}
if (fld_count != attr_count)
*************** CopyFrom(CopyState cstate)
*** 2124,2135 ****
if (file_has_oids)
{
cstate->cur_attname = "oid";
loaded_oid =
DatumGetObjectId(CopyReadBinaryAttribute(cstate,
0,
! &oid_in_function,
! oid_typioparam,
-1,
&isnull));
if (isnull || loaded_oid == InvalidOid)
--- 2433,2446 ----
if (file_has_oids)
{
+ Oid loaded_oid;
+
cstate->cur_attname = "oid";
loaded_oid =
DatumGetObjectId(CopyReadBinaryAttribute(cstate,
0,
! &cstate->oid_in_function,
! cstate->oid_typioparam,
-1,
&isnull));
if (isnull || loaded_oid == InvalidOid)
*************** CopyFrom(CopyState cstate)
*** 2137,2142 ****
--- 2448,2455 ----
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid OID in COPY data")));
cstate->cur_attname = NULL;
+ if (cstate->oids && tupleOid != NULL)
+ *tupleOid = loaded_oid;
}
i = 0;
*************** CopyFrom(CopyState cstate)
*** 2162,2278 ****
* provided by the input data. Anything not processed here or above
* will remain NULL.
*/
for (i = 0; i < num_defaults; i++)
{
values[defmap[i]] = ExecEvalExpr(defexprs[i], econtext,
&nulls[defmap[i]], NULL);
}
-
- /* And now we can form the input tuple. */
- tuple = heap_form_tuple(tupDesc, values, nulls);
-
- if (cstate->oids && file_has_oids)
- HeapTupleSetOid(tuple, loaded_oid);
-
- /* Triggers and stuff need to be invoked in query context. */
- MemoryContextSwitchTo(oldcontext);
-
- skip_tuple = false;
-
- /* BEFORE ROW INSERT Triggers */
- if (resultRelInfo->ri_TrigDesc &&
- resultRelInfo->ri_TrigDesc->trig_insert_before_row)
- {
- HeapTuple newtuple;
-
- newtuple = ExecBRInsertTriggers(estate, resultRelInfo, tuple);
-
- if (newtuple == NULL) /* "do nothing" */
- skip_tuple = true;
- else if (newtuple != tuple) /* modified by Trigger(s) */
- {
- heap_freetuple(tuple);
- tuple = newtuple;
- }
- }
-
- if (!skip_tuple)
- {
- List *recheckIndexes = NIL;
-
- /* Place tuple in tuple slot */
- ExecStoreTuple(tuple, slot, InvalidBuffer, false);
-
- /* Check the constraints of the tuple */
- if (cstate->rel->rd_att->constr)
- ExecConstraints(resultRelInfo, slot, estate);
-
- /* OK, store the tuple and create index entries for it */
- heap_insert(cstate->rel, tuple, mycid, hi_options, bistate);
-
- if (resultRelInfo->ri_NumIndices > 0)
- recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self),
- estate);
-
- /* AFTER ROW INSERT Triggers */
- ExecARInsertTriggers(estate, resultRelInfo, tuple,
- recheckIndexes);
-
- list_free(recheckIndexes);
-
- /*
- * We count only tuples not suppressed by a BEFORE INSERT trigger;
- * this is the same definition used by execMain.c for counting
- * tuples inserted by an INSERT command.
- */
- cstate->processed++;
- }
}
! /* Done, clean up */
! error_context_stack = errcontext.previous;
!
! FreeBulkInsertState(bistate);
!
! MemoryContextSwitchTo(oldcontext);
!
! /* Execute AFTER STATEMENT insertion triggers */
! ExecASInsertTriggers(estate, resultRelInfo);
!
! /* Handle queued AFTER triggers */
! AfterTriggerEndQuery(estate);
!
! pfree(values);
! pfree(nulls);
! if (! cstate->binary)
! pfree(cstate->raw_fields);
!
! pfree(in_functions);
! pfree(typioparams);
! pfree(defmap);
! pfree(defexprs);
!
! ExecResetTupleTable(estate->es_tupleTable, false);
!
! ExecCloseIndices(resultRelInfo);
!
! FreeExecutorState(estate);
! if (!pipe)
! {
! if (FreeFile(cstate->copy_file))
! ereport(ERROR,
! (errcode_for_file_access(),
! errmsg("could not close file \"%s\": %m",
! cstate->filename)));
! }
! /*
! * If we skipped writing WAL, then we need to sync the heap (but not
! * indexes since those use WAL anyway)
! */
! if (hi_options & HEAP_INSERT_SKIP_WAL)
! heap_sync(cstate->rel);
}
--- 2475,2504 ----
* provided by the input data. Anything not processed here or above
* will remain NULL.
*/
+ /* XXX: End of only-indentation changes. */
+ if (num_defaults > 0)
+ {
+ Assert(econtext != NULL);
+
for (i = 0; i < num_defaults; i++)
{
values[defmap[i]] = ExecEvalExpr(defexprs[i], econtext,
&nulls[defmap[i]], NULL);
}
}
! return true;
! }
! /*
! * Clean up storage and release resources for COPY FROM.
! */
! void
! EndCopyFrom(CopyState cstate)
! {
! /* No COPY FROM related resources except memory. */
! EndCopy(cstate);
}
*************** copy_dest_receive(TupleTableSlot *slot,
*** 3537,3542 ****
--- 3763,3769 ----
/* And send the data */
CopyOneRowTo(cstate, InvalidOid, slot->tts_values, slot->tts_isnull);
+ myState->processed++;
}
/*
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index 9e2bbe8..afe4b5e 100644
*** a/src/include/commands/copy.h
--- b/src/include/commands/copy.h
***************
*** 14,25 ****
--- 14,37 ----
#ifndef COPY_H
#define COPY_H
+ #include "nodes/execnodes.h"
#include "nodes/parsenodes.h"
#include "tcop/dest.h"
+ typedef struct CopyStateData *CopyState;
+
extern uint64 DoCopy(const CopyStmt *stmt, const char *queryString);
+ extern CopyState BeginCopyFrom(Relation rel, const char *filename,
+ List *attnamelist, List *options);
+ extern void EndCopyFrom(CopyState cstate);
+ extern bool NextCopyFrom(CopyState cstate, ExprContext *econtext,
+ Datum *values, bool *nulls, Oid *tupleOid);
+ extern bool NextCopyFromRawFields(CopyState cstate,
+ char ***fields, int *nfields);
+ extern void CopyFromErrorCallback(void *arg);
+
extern DestReceiver *CreateCopyDestReceiver(void);
#endif /* COPY_H */
On Wed, Feb 09, 2011 at 02:55:26PM +0900, Itagaki Takahiro wrote:
On Wed, Feb 9, 2011 at 03:49, Noah Misch <noah@leadboat.com> wrote:
The code still violates the contract of ExecEvalExpr() by calling it with
CurrentMemoryContext != econtext->ecxt_per_tuple_memory.I'm not sure whether we have such contract because the caller might
want to get the expression result in long-lived context.
execQual.c has this comment:
/* ----------------------------------------------------------------
* ExecEvalExpr routines
...
* The caller should already have switched into the temporary memory
* context econtext->ecxt_per_tuple_memory. The convenience entry point
* ExecEvalExprSwitchContext() is provided for callers who don't prefer to
* do the switch in an outer loop. We do not do the switch in these routines
* because it'd be a waste of cycles during nested expression evaluation.
* ----------------------------------------------------------------
*/
Assuming that comment is accurate, ExecEvalExpr constrains us; any default
values must get allocated in econtext->ecxt_per_tuple_memory. To get them in a
long-lived allocation, the caller can copy the datums or supply a long-lived
ExprContext. Any CurrentMemoryContext works when econtext == NULL, of course.
I'd suggest enhancing your new paragraph in the NextCopyFrom() header comment
like this:
- * 'econtext' is used to evaluate default expression for each columns not
- * read from the file. It can be NULL when no default values are used, i.e.
- * when all columns are read from the file.
+ * 'econtext' is used to evaluate default expression for each columns not read
+ * from the file. It can be NULL when no default values are used, i.e. when all
+ * columns are read from the file. If econtext is not NULL, the caller must have
+ * switched into the temporary memory context econtext->ecxt_per_tuple_memory.
But anyway
using an external ExprContext looks cleaner. The new prototype is:bool NextCopyFrom(
[IN] CopyState cstate, ExprContext *econtext,
[OUT] Datum *values, bool *nulls, Oid *tupleOid)
Looks good.
Note that econtext can be NULL if no default values are used.
Since file_fdw won't use default values, we can just pass NULL for it.
Nice. Good thinking. One small point:
--- 2475,2504 ---- * provided by the input data. Anything not processed here or above * will remain NULL. */ + /* XXX: End of only-indentation changes. */ + if (num_defaults > 0) + { + Assert(econtext != NULL); + for (i = 0; i < num_defaults; i++)
This could be better-written as "Assert(num_defaults == 0 || econtext != NULL);".
From a functional and code structure perspective, I find this ready to commit.
(I assume you'll drop the XXX: indent only comments on commit.) Kevin, did you
want to do that performance testing you spoke of?
Thanks,
nm
On Wed, Feb 9, 2011 at 2:03 AM, Noah Misch <noah@leadboat.com> wrote:
From a functional and code structure perspective, I find this ready to commit.
(I assume you'll drop the XXX: indent only comments on commit.) Kevin, did you
want to do that performance testing you spoke of?
OK, so is this Ready for Committer, or we're still working on it?
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Robert Haas <robertmhaas@gmail.com> wrote:
Noah Misch <noah@leadboat.com> wrote:
From a functional and code structure perspective, I find this
ready to commit. (I assume you'll drop the XXX: indent only
comments on commit.) Kevin, did you want to do that performance
testing you spoke of?OK, so is this Ready for Committer, or we're still working on it?
I can run some benchmarks to compare COPY statements with and
without the patch this weekend. Noah, does it make more sense to do
that with just the copy_export-20110209.patch patch file applied, or
in combination with some other FDW patch(es)?
-Kevin
On Sat, Feb 12, 2011 at 01:12, Robert Haas <robertmhaas@gmail.com> wrote:
On Wed, Feb 9, 2011 at 2:03 AM, Noah Misch <noah@leadboat.com> wrote:
From a functional and code structure perspective, I find this ready to commit.
(I assume you'll drop the XXX: indent only comments on commit.) Kevin, did you
want to do that performance testing you spoke of?OK, so is this Ready for Committer, or we're still working on it?
Basically, we have no more tasks until the FDW core API is applied.
COPY API and file_fdw patches are waiting for it.
If we extend them a little more, I'd raise two items:
* Should we print foreign table names in error messages?
http://archives.postgresql.org/pgsql-hackers/2011-02/msg00427.php
* COPY encoding patch was rejected, but using client_encoding is
logically wrong for file_fdw. We might need subset of the patch
for file_fdw.
--
Itagaki Takahiro
On Fri, Feb 11, 2011 at 11:31 AM, Itagaki Takahiro
<itagaki.takahiro@gmail.com> wrote:
On Sat, Feb 12, 2011 at 01:12, Robert Haas <robertmhaas@gmail.com> wrote:
On Wed, Feb 9, 2011 at 2:03 AM, Noah Misch <noah@leadboat.com> wrote:
From a functional and code structure perspective, I find this ready to commit.
(I assume you'll drop the XXX: indent only comments on commit.) Kevin, did you
want to do that performance testing you spoke of?OK, so is this Ready for Committer, or we're still working on it?
Basically, we have no more tasks until the FDW core API is applied.
COPY API and file_fdw patches are waiting for it.If we extend them a little more, I'd raise two items:
* Should we print foreign table names in error messages?
http://archives.postgresql.org/pgsql-hackers/2011-02/msg00427.php
* COPY encoding patch was rejected, but using client_encoding is
logically wrong for file_fdw. We might need subset of the patch
for file_fdw.
It sounds to me like that second issue is a showstopper. I think we
either need to reopen discussion on that patch and come up with a
resolution that is acceptable ASAP, or we need to punt file_fdw to
9.2.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Itagaki Takahiro <itagaki.takahiro@gmail.com> wrote:
Basically, we have no more tasks until the FDW core API is
applied. COPY API and file_fdw patches are waiting for it.
This is something I've found confusing about this patch set, to the
point of not knowing what to test, exactly. The COPY API patch and
the patch-on-patch for it both applied cleanly *without any of the
other patches* and seemed to run fine, even though the post with a
patch-on-patch for the COPY API said that several other patches
needed to be applied first. In spite of having tried to follow the
posts for all the FDW threads, I'm still confused enough about the
relationship between these patches to be unsure what to test.
My top priority for testing would be to confirm that there is no
adverse impact on existing COPY performance from the refactoring.
-Kevin
On Fri, Feb 11, 2011 at 10:31:08AM -0600, Kevin Grittner wrote:
Robert Haas <robertmhaas@gmail.com> wrote:
Noah Misch <noah@leadboat.com> wrote:
From a functional and code structure perspective, I find this
ready to commit. (I assume you'll drop the XXX: indent only
comments on commit.) Kevin, did you want to do that performance
testing you spoke of?OK, so is this Ready for Committer, or we're still working on it?
I can run some benchmarks to compare COPY statements with and
without the patch this weekend. Noah, does it make more sense to do
that with just the copy_export-20110209.patch patch file applied, or
in combination with some other FDW patch(es)?
I'd say, run them with this patch alone. The important thing is to not penalize
existing COPY users. Incidentally, the "did you want ... ?" was a genuine
question. I see very little performance risk here, so the tests could be quite
cursory, even absent entirely.
nm
Noah Misch <noah@leadboat.com> wrote:
I'd say, run them with this patch alone. The important thing is
to not penalize existing COPY users.
Sounds good.
Incidentally, the "did you want ... ?" was a genuine question. I
see very little performance risk here, so the tests could be quite
cursory, even absent entirely.
From what I've seen, I tend to agree. If there's a committer ready
to go over this, I would say that it might be worth waiting for the
benchmark results against the patch from the day before yesterday to
be run before "pulling the trigger" on it; but proceed on the basis
that it's a near-certainty that it will test out OK.
-Kevin
Noah Misch <noah@leadboat.com> wrote:
I'd say, run them with this patch alone. The important thing is
to not penalize existing COPY users. Incidentally, the "did you
want ... ?" was a genuine question. I see very little performance
risk here, so the tests could be quite cursory, even absent
entirely.
In two hours of testing with a 90GB production database, the copy
patch on top of HEAD ran 0.6% faster than HEAD for pg_dumpall
(generating identical output files), but feeding that in to and
empty cluster with psql ran 8.4% faster with the patch than without!
I'm going to repeat that latter with more attention to whether
everything made it in OK. (That's not as trivial to check as the
dump phase.)
Do you see any reason that COPY FROM should be significantly
*faster* with the patch? Are there any particular things I should
be checking for problems?
-Kevin
On Sat, Feb 12, 2011 at 03:42:17PM -0600, Kevin Grittner wrote:
In two hours of testing with a 90GB production database, the copy
patch on top of HEAD ran 0.6% faster than HEAD for pg_dumpall
(generating identical output files), but feeding that in to and
empty cluster with psql ran 8.4% faster with the patch than without!
I'm going to repeat that latter with more attention to whether
everything made it in OK. (That's not as trivial to check as the
dump phase.)Do you see any reason that COPY FROM should be significantly
*faster* with the patch?
No. Up to, say, 0.5% wouldn't be too surprising, but 8.4% is surprising. What
is the uncertainty of that figure?
Are there any particular things I should
be checking for problems?
Nothing comes to mind.
Thanks,
nm
Noah Misch <noah@leadboat.com> wrote:
On Sat, Feb 12, 2011 at 03:42:17PM -0600, Kevin Grittner wrote:
Do you see any reason that COPY FROM should be significantly
*faster* with the patch?No. Up to, say, 0.5% wouldn't be too surprising, but 8.4% is
surprising.
What is the uncertainty of that figure?
With a few more samples, it's not that high. It's hard to dodge
around the maintenance tasks on this machine to get good numbers, so
I can't really just set something up to run overnight to get numbers
in which I can have complete confidence, but (without putting
statistical probabilities around it) I feel very safe in saying
there isn't a performance *degradation* with the patch. I got four
restores of of the 90GB data with the patch and four without. I
made sure it was during windows without any maintenance running, did
a fresh initdb for each run, and made sure that the disk areas were
the same for each run. The times for each version were pretty
tightly clustered except for each having one (slow) outlier.
If you ignore the outlier for each, there is *no overlap* between
the two sets -- the slowest of the non-outlier patched times is
faster than the fastest non-patched time.
With the patch, compared to without -- best time is 9.8% faster,
average time without the outliers is 6.9% faster, average time
including outliers is 4.3% faster, outlier is 0.8% faster.
Even with just four samples each, since I was careful to minimize
distorting factors, that seems like plenty to have confidence that
there is no performance *degradation* from the patch. If we want to
claim some particular performance *gain* from it, I would need to
arrange a dedicated machine and script maybe 100 runs each way to be
willing to offer a number for public consumption.
Unpatched:
real 17m24.171s
real 16m52.892s
real 16m40.624s
real 16m41.700s
Patched:
real 15m56.249s
real 15m47.001s
real 15m3.018s
real 17m16.157s
Since you said that a cursory test, or no test at all, should be
good enough given the low risk of performance regression, I didn't
book a machine and script a large test run, but if anyone feels
that's justified, I can arrange something.
-Kevin
On 02/12/2011 05:33 PM, Noah Misch wrote:
On Sat, Feb 12, 2011 at 03:42:17PM -0600, Kevin Grittner wrote:
In two hours of testing with a 90GB production database, the copy
patch on top of HEAD ran 0.6% faster than HEAD for pg_dumpall
(generating identical output files), but feeding that in to and
empty cluster with psql ran 8.4% faster with the patch than without!
I'm going to repeat that latter with more attention to whether
everything made it in OK. (That's not as trivial to check as the
dump phase.)Do you see any reason that COPY FROM should be significantly
*faster* with the patch?No. Up to, say, 0.5% wouldn't be too surprising, but 8.4% is surprising. What
is the uncertainty of that figure?
We have seen in the past that changes that might be expected to slow
things down slightly can have the opposite effect. For example, see
<http://people.planetpostgresql.org/andrew/index.php?/archives/37-Puzzling-results.html>
where Tom commented:
Yeah, IME it's not unusual for microbenchmark results to move a
percent or three in response to any code change at all, even
unrelated ones. I suppose it's from effects like critical loops
breaking across cache lines differently than before.
cheers
andrew
On Sun, Feb 13, 2011 at 12:41:11PM -0600, Kevin Grittner wrote:
Unpatched:
real 17m24.171s
real 16m52.892s
real 16m40.624s
real 16m41.700sPatched:
real 15m56.249s
real 15m47.001s
real 15m3.018s
real 17m16.157sSince you said that a cursory test, or no test at all, should be
good enough given the low risk of performance regression, I didn't
book a machine and script a large test run, but if anyone feels
that's justified, I can arrange something.
Based on this, I've taken the liberty of marking the patch Ready for Committer.
Thanks.
On Mon, Feb 14, 2011 at 22:06, Noah Misch <noah@leadboat.com> wrote:
On Sun, Feb 13, 2011 at 12:41:11PM -0600, Kevin Grittner wrote:
Unpatched:
real 17m24.171s
real 16m52.892s
real 16m40.624s
real 16m41.700sPatched:
real 15m56.249s
real 15m47.001s
real 15m3.018s
real 17m16.157sSince you said that a cursory test, or no test at all, should be
good enough given the low risk of performance regression, I didn't
book a machine and script a large test run, but if anyone feels
that's justified, I can arrange something.Based on this, I've taken the liberty of marking the patch Ready for Committer.
Thank you very much for performance testing and reviewing!
The result is interesting because I didn't intend performance optimization.
At least no performance regression is enough for the purpose.
--
Itagaki Takahiro