PostgreSQL and big data - FDW

Started by ROS Didierover 5 years ago7 messages

didier.ros@edf.fr

over 5 years ago

Hi
I would like to use a Foreign Data Wrapper (FDW) to connect to a HADOOP cluster which uses KERBEROS authentication.
is it possible to achieve this ? which FDW should be used ?

Thanks in advance

Best Regards
Didier ROS
EDF

Bruce Momjian

bruce@momjian.us

over 5 years ago

In reply to: ROS Didier (#1)

Re: PostgreSQL and big data - FDW

On Wed, Jun 24, 2020 at 09:05:30AM +0000, ROS Didier wrote:

Hi

I would like to use a Foreign Data Wrapper (FDW) to connect to a HADOOP cluster
which uses KERBEROS authentication.

is it possible to achieve this ? which FDW should be used ?

Well, I would use the Hadoop FDW:

https://github.com/EnterpriseDB/hdfs_fdw

and it only supports these authentication methods:

Authentication Support

The FDW supports NOSASL and LDAP authentication modes. In order to use
NOSASL do not specify any OPTIONS while creating user mapping. For LDAP
username and password must be specified in OPTIONS while creating user mapping.

Not every FDW supports every Postgres server authentication method.

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com

The usefulness of a cup is in its emptiness, Bruce Lee

ROS Didier

didier.ros@edf.fr

over 5 years ago

In reply to: Bruce Momjian (#2)

RE: PostgreSQL and big data - FDW

Hi Bruce

In the following link : https://www.enterprisedb.com/blog/connecting-hadoop-and-edb-postgres-shrink-big-data-challenges
We can see :
"Support for various authentication methods (i.e. Kerberos, NOSASL, etc.)"

So HDFS_FDW support kerberos authentication . how to be sure of that ?
Could EDB make a clear statement on this point?

If so, how to implement this method ? is there any document on this subject ?

Thanks in advance.
Best Regards

Didier ROS
didier.ros@edf.fr
Tél. : +33 6 49 51 11 88

-----Message d'origine-----
De : bruce@momjian.us [mailto:bruce@momjian.us]
Envoyé : mercredi 24 juin 2020 11:13
À : ROS Didier <didier.ros@edf.fr>
Cc : pgsql-hackers@lists.postgresql.org
Objet : Re: PostgreSQL and big data - FDW

On Wed, Jun 24, 2020 at 09:05:30AM +0000, ROS Didier wrote:

Hi

I would like to use a Foreign Data Wrapper (FDW) to connect to a
HADOOP cluster which uses KERBEROS authentication.

is it possible to achieve this ? which FDW should be used ?

Well, I would use the Hadoop FDW:

https://github.com/EnterpriseDB/hdfs_fdw

and it only supports these authentication methods:

Authentication Support

Not every FDW supports every Postgres server authentication method.

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com

The usefulness of a cup is in its emptiness, Bruce Lee

Ce message et toutes les pièces jointes (ci-après le 'Message') sont établis à l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme à sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse.

Si vous n'êtes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez reçu ce Message par erreur, merci de le supprimer de votre système, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions également d'en avertir immédiatement l'expéditeur par retour du message.

Il est impossible de garantir que les communications par messagerie électronique arrivent en temps utile, sont sécurisées ou dénuées de toute erreur ou virus.
____________________________________________________

This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval.

If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message.

E-mail communication cannot be guaranteed to be timely secure, error or virus-free.

Jeevan Ladhe

jeevan.ladhe@enterprisedb.com

over 5 years ago

In reply to: ROS Didier (#3)

Re: PostgreSQL and big data - FDW

On Wed, Jun 24, 2020 at 6:09 PM ROS Didier <didier.ros@edf.fr> wrote:

Hi Bruce

In the following link :
https://www.enterprisedb.com/blog/connecting-hadoop-and-edb-postgres-shrink-big-data-challenges
We can see :
"Support for various authentication methods (i.e. Kerberos, NOSASL, etc.)"

So HDFS_FDW support kerberos authentication . how to be sure of that ?
Could EDB make a clear statement on this point?

HDFS_FDW does not support kerberos authentication.
The sentence you have pasted above is from the wish list or say TODO
list, here is what it says:
"Currently the HDFS_FDW only provides READ capabilities but EDB is planning
the following additional functionality:"

The functionality was not implemented. I think the part of confusion might
be
due to the formatting of the list in the blog.

You can follow the README[1]https://github.com/EnterpriseDB/hdfs_fdw/blob/master/README.md of HDFS_FDW to get an idea of how to use it.

[1]: https://github.com/EnterpriseDB/hdfs_fdw/blob/master/README.md

Regards,
Jeevan

Stephen Frost

sfrost@snowman.net

over 5 years ago

In reply to: Bruce Momjian (#2)

Re: PostgreSQL and big data - FDW

Greetings,

* Bruce Momjian (bruce@momjian.us) wrote:

On Wed, Jun 24, 2020 at 09:05:30AM +0000, ROS Didier wrote:

I would like to use a Foreign Data Wrapper (FDW) to connect to a HADOOP cluster
which uses KERBEROS authentication.

Sadly, not really.

is it possible to achieve this ? which FDW should be used ?

Well, I would use the Hadoop FDW:

https://github.com/EnterpriseDB/hdfs_fdw

and it only supports these authentication methods:

Authentication Support

The FDW supports NOSASL and LDAP authentication modes. In order to use
NOSASL do not specify any OPTIONS while creating user mapping. For LDAP
username and password must be specified in OPTIONS while creating user mapping.

Not every FDW supports every Postgres server authentication method.

That isn't really the issue here, the problem is really that the GSSAPI
support in PG today doesn't support credential delegation- if it did,
then the HDFS FDW (and the postgres FDW) could be easily extended to
leverage those delegated credentials to connect.

That's been something that's been on my personal todo list of things to
work on but unfortunately I've not, as yet, had time to go implement. I
don't actually think it would be very hard- if someone writes it, I'd
definitely review it.

Thanks,

Stephen

ROS Didier

didier.ros@edf.fr

over 5 years ago

In reply to: Stephen Frost (#5)

RE: PostgreSQL and big data - FDW

Hi Stephen

My EDF company is very interested in this feature (KERBEROS authentication method and hdfs_fdw ).
Is it possible to know how many days of development does this represent ? who can develop this implementation ? what cost ?

Best Regards
Didier ROS
EDF
-----Message d'origine-----
De : sfrost@snowman.net [mailto:sfrost@snowman.net]
Envoyé : mercredi 24 juin 2020 18:53
À : Bruce Momjian <bruce@momjian.us>
Cc : ROS Didier <didier.ros@edf.fr>; pgsql-hackers@lists.postgresql.org
Objet : Re: PostgreSQL and big data - FDW

Greetings,

* Bruce Momjian (bruce@momjian.us) wrote:

On Wed, Jun 24, 2020 at 09:05:30AM +0000, ROS Didier wrote:

I would like to use a Foreign Data Wrapper (FDW) to connect to a
HADOOP cluster which uses KERBEROS authentication.

Sadly, not really.

is it possible to achieve this ? which FDW should be used ?

Well, I would use the Hadoop FDW:

https://github.com/EnterpriseDB/hdfs_fdw

and it only supports these authentication methods:

Authentication Support

The FDW supports NOSASL and LDAP authentication modes. In order to use
NOSASL do not specify any OPTIONS while creating user mapping. For LDAP
username and password must be specified in OPTIONS while creating user mapping.

Not every FDW supports every Postgres server authentication method.

That isn't really the issue here, the problem is really that the GSSAPI support in PG today doesn't support credential delegation- if it did, then the HDFS FDW (and the postgres FDW) could be easily extended to leverage those delegated credentials to connect.

That's been something that's been on my personal todo list of things to work on but unfortunately I've not, as yet, had time to go implement. I don't actually think it would be very hard- if someone writes it, I'd definitely review it.

Thanks,

Stephen

E-mail communication cannot be guaranteed to be timely secure, error or virus-free.

Bruce Momjian

bruce@momjian.us

over 5 years ago

In reply to: ROS Didier (#6)

Re: PostgreSQL and big data - FDW

On Thu, Jun 25, 2020 at 07:02:37AM +0000, ROS Didier wrote:

Hi Stephen

My EDF company is very interested in this feature (KERBEROS authentication method and hdfs_fdw ).
Is it possible to know how many days of development does this represent ? who can develop this implementation ? what cost ?

Uh, the only thing I can suggest is to contact one of the larger
Postgres support companies (ones that have developers who understand the
server code, or at least the FDW code), and ask them for estimates. The
community really can't supply any of that, unless you want to do the
work and want source code tips.

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com

The usefulness of a cup is in its emptiness, Bruce Lee