doc: virtual envs with Pl/Python

Florents Tselai <florents.tselai@gmail.com> writes:

I think there should be a mention of virtual environments in the plpython
docs.

Why? We are not here to teach people how to use Python. More, this
patch seems like not merely teaching Python, but teaching the use of
specific Python installation patterns that might or might not apply
on a particular platform. That's not going to scale.

regards, tom lane

On 17.10.24 15:56, Tom Lane wrote:

Florents Tselai <florents.tselai@gmail.com> writes:

I think there should be a mention of virtual environments in the plpython
docs.

Why? We are not here to teach people how to use Python. More, this
patch seems like not merely teaching Python, but teaching the use of
specific Python installation patterns that might or might not apply
on a particular platform. That's not going to scale.

Virtual environments are a standard part of Python now, so maybe there
is something to this. But personally I don't understand this. Do you
mean for this to select the Python environment that PL/Python is built
against, or do you want to use the virtual environment at run time? The
text says "installing" and talks about package managers. It's unclear
to me at exactly what step this is supposed to apply.

On 17 Oct 2024, at 5:10 PM, Peter Eisentraut <peter@eisentraut.org> wrote:

On 17.10.24 15:56, Tom Lane wrote:

Florents Tselai <florents.tselai@gmail.com> writes:

I think there should be a mention of virtual environments in the plpython
docs.

Why? We are not here to teach people how to use Python. More, this
patch seems like not merely teaching Python, but teaching the use of
specific Python installation patterns that might or might not apply
on a particular platform. That's not going to scale.

Virtual environments are a standard part of Python now, so maybe there is something to this. But personally I don't understand this. Do you mean for this to select the Python environment that PL/Python is built against, or do you want to use the virtual environment at run time? The text says "installing" and talks about package managers. It's unclear to me at exactly what step this is supposed to apply.

this

want to use the virtual environment at run time?

OK, I should have explained this a bit more.

Here’s the story:

Unless someone builds from source, chances are, the default Python interpreter is something like /usr/bin/python3 .
Which sometimes may even point to python3.10/11/12 .
Plus, If you have multiple Postgres versions (not unusual if you’ve had the same PG server for years),
the version matrix gets too complicated.

Which means that if you want non-standard packages in your Pl/Python code,
you’ll have to `sudo python3 -m pip install` things, which in turn will break things in different versions.
In fact, in the latest versions standard Python actively discourages that by having your
say `pip install … —break-system-packages`

The practice now is to build a virtual `python3 -m venv`, which, as you say, is standard.
And indeed, it is not just a specific installation pattern.
I’m not getting into fancy things like poetry et. al. here.
Just this https://docs.python.org/3/library/venv.html

The documentation as-is mentions the availability of env vars that can be set and stops there.

As-is, someone who reads the documentation is left with the impression that
the only way to use non-standard Python libs is by `sudo python3 -m install package` because
that’s the python3 interpreter that plpython3u picks-up during installation.

The proposed note goes a step further and aims to say how to support this not-so-rare pattern.

I hope that’s more clear now.

PS: fwiw, It’s practically impossible to build —with-python=/path/to/venv because most virtualenvs creators don’t bother handling header files correctly. config/python.m4 can’t support that. But that’s another story irrelevant here.

On Thu, Oct 17, 2024 at 1:10 PM Florents Tselai
<florents.tselai@gmail.com> wrote:

Which means that if you want non-standard packages in your Pl/Python code,
you’ll have to `sudo python3 -m pip install` things

Or `sudo apt install python3-<package>`, or `pip install --user
<package>` as the database user, or...

Virtualenvs are great -- I use them daily -- but there are plenty of
different ways to use the standard interpreter, with other pros and
cons.

--Jacob

Jacob Champion <jacob.champion@enterprisedb.com> writes:

On Thu, Oct 17, 2024 at 1:10 PM Florents Tselai
<florents.tselai@gmail.com> wrote:

Which means that if you want non-standard packages in your Pl/Python code,
you’ll have to `sudo python3 -m pip install` things

Or `sudo apt install python3-<package>`, or `pip install --user
<package>` as the database user, or...

Virtualenvs are great -- I use them daily -- but there are plenty of
different ways to use the standard interpreter, with other pros and
cons.

What I'm still confused about is whether this is anything that
a typical user of Python shouldn't be expected to know already.
I understand the idea of setting up a virtualenv that is decoupled
from the system-supplied package set, but if the user has done
that, wouldn't they already know about setting PYTHONPATH?

I can believe that they might not be totally clear on how to
set it in a way that will affect the Postgres server --- but
the proposed patch provides no details about that, and hardly
could since it'd vary so much from one server packaging to
another.

regards, tom lane

On Thu, Oct 17, 2024 at 2:03 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

What I'm still confused about is whether this is anything that
a typical user of Python shouldn't be expected to know already.

I mean... I think I agree, but also it seems like people regularly
find ways to shoot themselves in the foot, so I'm not opposed to some
tips in theory. But they should be more generally applicable IMO.

I understand the idea of setting up a virtualenv that is decoupled
from the system-supplied package set, but if the user has done
that, wouldn't they already know about setting PYTHONPATH?

Agreed.

I can believe that they might not be totally clear on how to
set it in a way that will affect the Postgres server --- but
the proposed patch provides no details about that, and hardly
could since it'd vary so much from one server packaging to
another.

Yeah. I think this patch is _way_ too prescriptive, and maybe belongs
in a wiki on "Ways I Get Complex Python Setups Working" or similar,
where it can share space with alternatives.

--Jacob