START/END line number for COPY FROM

Surafel Temesgen <surafel3000@gmail.com> writes:

Currently we can skip header line on COPY FROM but having the ability to
skip and stop copying at any line can use to divide long copy operation and
enable to copy a subset of the file and skipping footer. Attach is a patch
for it

I do not think this is a good idea. We have resisted attempts to add
ETL-like features to COPY on the grounds that it would add complexity
and cost performance, and that that's not what COPY is for. This
seems to fall squarely in the domain of something you should be doing
with another tool.

regards, tom lane

On 20/12/2018 14:02, Surafel Temesgen wrote:

Currently we can skip header line on COPY FROM but having the ability to
skip and stop copying at any line can use to divide long copy operation
and enable to copy a subset of the file and skipping footer.

It seems a bit fragile to me if I want to skip a footer and need to
figure out the total line count, subtract one, and then oh, was it zero-
or one-based.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Hi,
On Fri, Jan 4, 2019 at 5:37 PM Peter Eisentraut <
peter.eisentraut@2ndquadrant.com> wrote:

It seems a bit fragile to me if I want to skip a footer and need to
figure out the total line count, subtract one, and then oh, was it zero-
or one-based.

But normally we don't say start copying from line number 0
regards
Surafel

On Fri, 21 Dec 2018 at 02:02, Surafel Temesgen <surafel3000@gmail.com> wrote:

Currently we can skip header line on COPY FROM but having the ability to skip and stop copying at any line can use to divide long copy operation and enable to copy a subset of the file and skipping footer. Attach is a patch for it

I'm struggling a bit to see the sense in this. If you really want to
improve the performance of a long copy, then I think it makes more
sense to have performed the backup in multiple pieces in the first
place. Having the database read the input stream and ignore the first
N lines sounds like a bit of a waste of effort, and effort that
wouldn't be required if the COPY TO had been done in multiple pieces.

--
David Rowley http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Hi,
On Sat, Jan 5, 2019 at 1:10 PM David Rowley <david.rowley@2ndquadrant.com>
wrote:

On Fri, 21 Dec 2018 at 02:02, Surafel Temesgen <surafel3000@gmail.com>
wrote:

Currently we can skip header line on COPY FROM but having the ability to

skip and stop copying at any line can use to divide long copy operation and
enable to copy a subset of the file and skipping footer. Attach is a patch
for it

I'm struggling a bit to see the sense in this. If you really want to
improve the performance of a long copy, then I think it makes more
sense to have performed the backup in multiple pieces in the first
place.

it is not always the case to have in control of the data importing it may
came from
external system

regards
Surafel

On 06/01/2019 12:59, Surafel Temesgen wrote:

it is not always the case to have in control of the data importing it
may came from
external system

But the problem that David described remains: If your data loading
requirement is so complicated that you need to load the file in chunks,
then doing it by line numbers will require you to skip over the leading
lines at every subsequent chunk. That's not going to be good for larger
files.

I think your problem needs a different solution.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services