Does postgreSQL community edition supports data distribution across nodes

Hi,

I think you are looking for:

https://www.postgres-xl.org/

regards,

fabio pardi

Also
https://wiki.postgresql.org/wiki/Replication,_Clustering,_and_Connection_Pooling
and https://www.postgresql.org/docs/10/static/logical-replication.html for
other options as well.

On Fri, Oct 5, 2018 at 7:35 AM Fabio Pardi <f.pardi@portavita.eu> wrote:

We are doing a POC of using Datastage with PG using ODBC.

Problem to solve: How to load a large CSV file using COPY command. The file is on the client machine.

A typical SQL syntax of a copy coming from a remote machine COPY TABLE FROM STDIN WITH CSV HEADER

Question is, how to make the contents of the file available as STDIN in a SQL. It is easy in a shell.

Hello, if you need to use COPY command from remote machine and you use some
libpq bindings (aka ruby pg gem for example), you can use functions
associated with COPY command (
https://www.postgresql.org/docs/10/static/libpq-copy.html). They should be
provided by bindings of postgres library you're using.

If you can share more info, at least how do you access postgres (via Ruby
gem for example or from client's console), I can try to be more descriptive.

Josef

pá 5. 10. 2018 v 16:06 odesílatel Ravi Krishna <srkrishna1@aol.com> napsal:

Hello, if you need to use COPY command from remote machine and you use some libpq bindings (aka ruby pg gem for example), you can use functions associated with COPY command (https://www.postgresql.org/docs/10/static/libpq-copy.html <https://www.postgresql.org/docs/10/static/libpq-copy.html&gt;). They should be provided by bindings of postgres library you're using.

If you can share more info, at least how do you access postgres (via Ruby gem for example or from client's console), I can try to be more descriptive.

We not writing any Ruby/Python code. We are using Datastage. Datastage has in built features for most of the stuff. For example to bulk load data from csv files
into Oracle/DB2 etc, it has a BULK loader feature. However DS has no support for PG directly and we are piggy backing on ODBC where there is no BULK loader.

The only recourse for us is to type in SQL as DS allows user code.

Can you copy your csv to postgres server?

pá 5. 10. 2018 v 16:19 odesílatel Ravi Krishna <srkrishna1@aol.com> napsal:

On 10/05/2018 09:18 AM, Ravi Krishna wrote:

Hello, if you need to use COPY command from remote machine and you use
some libpq bindings (aka ruby pg gem for example), you can use functions
associated with COPY command
(https://www.postgresql.org/docs/10/static/libpq-copy.html). They should
be provided by bindings of postgres library you're using.

If you can share more info, at least how do you access postgres (via Ruby
gem for example or from client's console), I can try to be more descriptive.

We not writing any Ruby/Python code.  We are using Datastage. Datastage
has in built features for most of the stuff. For example to bulk load data
from csv files
into Oracle/DB2 etc, it has a BULK loader feature. However DS has no
support for PG directly and we are piggy backing on ODBC where there is no
BULK loader.

The only recourse for us is to type in SQL as DS allows user code.

Can you install the postgres client software (psql) on the client machine
and then have Datastage spawn "psql -c 'COPY ...'"?

--
Angular momentum makes the world go 'round.

Can you install the postgres client software (psql) on the client machine and then have Datastage spawn "psql -c 'COPY ...'"?

That is already an option for us :-)

if you have ssh access to the client, you can do :
ssh user@client_host "cat /path_to/large_file.csv" | psql -d database
....

On Fri, Oct 5, 2018 at 9:06 AM Ravi Krishna <srkrishna1@aol.com> wrote:

We are doing a POC of using Datastage with PG using ODBC.

Problem to solve: How to load a large CSV file using COPY command. The
file is on the client machine.

A typical SQL syntax of a copy coming from a remote machine COPY TABLE
FROM STDIN WITH CSV HEADER

Question is, how to make the contents of the file available as STDIN in a
SQL. It is easy in a shell.

--