pgloader - Can't find textreader/csvreader

On Mon, 2007-08-27 at 12:22 +0200, Dimitri Fontaine wrote:

Le lundi 27 août 2007, Ow Mun Heng a écrit :

I'm trying to see if pgloader will make my work easier for bulkloads.
I'm testing it out and I'm stucked basically because it can't find the
module TextReader or CSVreader.

It's a pgloader provided module, and the error arise because I forgot to make
sure you can use pgloader without installing it properly in the system.
The only installation supported now is via the debian package, if you're using
debian either use the .deb provided on pgfoundry or issue a debuild command.

I use gentoo.. so....

You can also tweak the makefile to add some mkdir -p commands before it
installs files in non-existing directories, too, and then install pgloader

I Downloaded the 2.2.0 tarball and can't find any reference to it from
this tarball except for the import statement.

I'll try again with the (is there are 2.2.1, I thought I saw it) or take
a peek at the deb package..

Thanks.

ps : (no internet at the momemt.. only email access)

Le lundi 27 août 2007, Ow Mun Heng a écrit :

It's a pgloader provided module, and the error arise because I forgot to
make sure you can use pgloader without installing it properly in the
system.

After some testing, it seems pgloader is still usable without system
installation at all. But I forgot some new files in the 2.2.0 release.

I'll try again with the (is there are 2.2.1, I thought I saw it) or take
a peek at the deb package..

We've just made some tests here with 2.2.1 and as this release contains the
missing files, it works fine without any installation.

Regards,
--
dim

On Mon, 2007-08-27 at 11:27 +0200, Dimitri Fontaine wrote:

We've just made some tests here with 2.2.1 and as this release contains the
missing files, it works fine without any installation.

Yep.. I can confirm that it works.. I am using the csv example.

Goal : similar functionality much like mysql's mysqlimport --replace
(overwrite any rows which has duplicate primary keys)

$ psql pgloader < csv/csv.sql
$ ../pgloader.py -Tvc examples/pgloader.conf csv

pgloader=# alter table csv add primary key (a,b,c);
pgloader=# \d csv
Table "public.csv"
Column | Type | Modifiers
--------+--------------+-----------
a | bigint | not null
b | bigint | not null
c | character(2) | not null
d | text |
Indexes:
"csv_pkey" PRIMARY KEY, btree (a, b, c)

pgloader=# select * from csv;
a | b | c | d
----------+----------+----+----------------
33996344 | 33996351 | GB | United Kingdom
50331648 | 68257567 | US | United States
68257568 | 68257599 | CA | Canada
68257600 | 68259583 | US | United States
68259584 | 68259599 | CA | Canada

$cat csv/csv.data
"2.6.190.56","2.6.190.63","33996344","33996351","GB","Error Kingdom"
"4.17.143.0","4.17.143.15","68259584","68259599","CA","new Country"
<Note : only columns 3 to 6 are taken for loading)

$ psql pgloader < csv/csv.sql
$ ../pgloader.py -vc pgloader.conf csv
Using pgloader.conf configuration file
Will consider following sections:
csv

[csv] parse configuration
Notice: reject log in /tmp/csv.rej.log
Notice: rejected data in /tmp/csv.rej
[csv] data import
Notice: COPY csv data

Error: Please check PostgreSQL logs
HINT: double check your client_encoding, datestyle and copy_delimiter
settings

$sudo tail -f /var/log/pglog/postgresxx-xx-xx.log
ERROR: duplicate key violates unique constraint "csv_pkey"
CONTEXT: COPY csv, line 1: "33996344,33996351,Error Kingdom,GB"
STATEMENT: COPY csv (a, b, d, c) FROM stdin USING DELIMITERS ','

So.. doesn't really solve my issue.
Dang it..

Le lundi 27 août 2007, Ow Mun Heng a écrit :

I'm trying to see if pgloader will make my work easier for bulkloads.
I'm testing it out and I'm stucked basically because it can't find the
module TextReader or CSVreader.

Googling doesn't help as there seems to be no reference to a module
named textreader or csvreader.

It's a pgloader provided module, and the error arise because I forgot to make
sure you can use pgloader without installing it properly in the system.
The only installation supported now is via the debian package, if you're using
debian either use the .deb provided on pgfoundry or issue a debuild command.

You can also tweak the makefile to add some mkdir -p commands before it
installs files in non-existing directories, too, and then install pgloader
with make install. (debian packaging cares about system dirs, hence not the
provided makefile)

I'll see about having pgloader usable without any system-wide installation...

Regards,
--
dim