work on extending PostgreSQL to data integration systems

On Wed, Nov 01, 2006 at 02:13:47AM +0800, Lee Riquelmei wrote:

Hello, everybody,
I am developing a postgresql-based distributed relational data integration
system which is like the IBM DB2 Information Integrator. And I have started
a pgfoundry project: [1]http://pgfoundry.org/projects/ignite .(codes on it
have already been obsolete.)
The goal of the system is to provide a collection of virtual views in a
pgsql database to integrate data from various distributed, heterogeneous and
autonomous data sources.

You can do this now in a crude way using DBI-Link, which I am
developing now.
<http://pgfoundry.org/projects/dbi-link&gt;

I'm very interested in your work, as DBI-Link's methods, all of which
act in user space, are somewhat crude. It has no hooks into the
planner, the optimizer, etc. It does have facilities which could use
such hooks if they were made available.

To achieve this goal, I did my work in two aspects:
(1) outside pgsql: The system utilizes various wrappers to interact with
data sources. A well-defined wrapper interface is presented, which allows
for implementations of concrete wrappers using any language , such as
pl/java. In nature, a wrapper is a collection of pgsql functions which
consists of the function for obtaining metadata of a data source, the
function for exectuing SQL in data source and etc.
(2) inside pgsql: Several modifictions to pgsql is used to improve the
performance of processing data integration queries including the pipelined
FunctionScan operator and support of query
shipping(selection/projection/sort).
The developing work is still ongoing, although an initial version of the
system has already been used in a real application in China. This version
consists of a wrapper for PostgreSQL using libpq and a wrapper for MS SQL
SERVER using pl/java and JDBC. Currently, I have not published all sources
and not provided good documents in the pgfoundry projects. But I will do it
quickly. If anyone is interested in the system, pls contact me.
I have a question. Does the PostgreSQL project have any plan of adding data
integration functionalities in the PostgreSQL system as other DBMS vendors
did?

The SQL:2003 standard includes a section called SQL/MED (Management of
External Data). I suspect that your implementation would be more
likely to go into the core if it were made compatible with this
standard.

Regards,
David.
--
David Fetter <david@fetter.org> http://fetter.org/
phone: +1 415 235 3778 AIM: dfetter666
Skype: davidfetter

Remember to vote!