PostgreSQL 12, JIT defaults

On Mon, Oct 8, 2018 at 10:52 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:

Hi

I configured PostgreSQL without JIT support, but JIT is active by default

I think that happens when llvmjit.so is present (ie from last time you
built with JIT support and ran make install). You need to remove it.

--
Thomas Munro
http://www.enterprisedb.com

po 8. 10. 2018 v 12:06 odesílatel Thomas Munro <
thomas.munro@enterprisedb.com> napsal:

On Mon, Oct 8, 2018 at 10:52 PM Pavel Stehule <pavel.stehule@gmail.com>
wrote:

Hi

I configured PostgreSQL without JIT support, but JIT is active by default

I think that happens when llvmjit.so is present (ie from last time you
built with JIT support and ran make install). You need to remove it.

aha. I'll do it. But it doesn't look like robust solution :-(.

Maybe clean, or distclean should to remove this file.

Thank you for info

Pavel

Hi

On October 8, 2018 2:51:15 AM PDT, Pavel Stehule <pavel.stehule@gmail.com> wrote:

Hi

I configured PostgreSQL without JIT support, but JIT is active by
default

postgres=# show jit;
┌─────┐
│ jit │
╞═════╡
│ on │
└─────┘
(1 row)

Where is the jit=on coming from? Config from before it was turned off?

Unfortunately - this combination does crashes on some bigger queries.

You probably have the JIT plugin installed from an earlier time, which also would explain why it crashes.

Andres

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Andres Freund <andres@anarazel.de> writes:

Where is the jit=on coming from? Config from before it was turned off?

A look in guc.c shows that jit defaults to "on" whether or not JIT is
enabled at compile time. This seems, at best, rather user-unfriendly.
And it's not like our conventions for other compile-option-affected
GUCs, eg the SSL ones.

regards, tom lane

po 8. 10. 2018 v 16:58 odesílatel Andres Freund <andres@anarazel.de> napsal:

Hi

On October 8, 2018 2:51:15 AM PDT, Pavel Stehule <pavel.stehule@gmail.com>
wrote:

Hi

I configured PostgreSQL without JIT support, but JIT is active by
default

postgres=# show jit;
┌─────┐
│ jit │
╞═════╡
│ on │
└─────┘
(1 row)

Where is the jit=on coming from? Config from before it was turned off?

Unfortunately - this combination does crashes on some bigger queries.

You probably have the JIT plugin installed from an earlier time, which
also would explain why it crashes.

I had it there.

On October 8, 2018 8:03:56 AM PDT, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Andres Freund <andres@anarazel.de> writes:

Where is the jit=on coming from? Config from before it was turned

off?

A look in guc.c shows that jit defaults to "on" whether or not JIT is
enabled at compile time.

I thought Pavel was talking about 11 somehow...

This seems, at best, rather user-unfriendly.
And it's not like our conventions for other compile-option-affected
GUCs, eg the SSL ones.

That was intentional, even though it perhaps should be better documented. That allows a distro to build and distribute pg without llvm enabled, but then have the JIT package with all the dependencies separately. The pg yum packages do so.

Andres
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

On October 8, 2018 8:10:45 AM PDT, Andres Freund <andres@anarazel.de> wrote:

On October 8, 2018 8:03:56 AM PDT, Tom Lane <tgl@sss.pgh.pa.us> wrote:

This seems, at best, rather user-unfriendly.
And it's not like our conventions for other compile-option-affected
GUCs, eg the SSL ones.

That was intentional, even though it perhaps should be better
documented. That allows a distro to build and distribute pg without
llvm enabled, but then have the JIT package with all the dependencies
separately. The pg yum packages do so.

There's a function for checking whether JIT is actually available, BTW. pg_jit_available(). That also works if somebody installs an extension that's not ours (by setting jit_provider = ...).

Andres

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

po 8. 10. 2018 v 17:10 odesílatel Andres Freund <andres@anarazel.de> napsal:

On October 8, 2018 8:03:56 AM PDT, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Andres Freund <andres@anarazel.de> writes:

Where is the jit=on coming from? Config from before it was turned

off?

A look in guc.c shows that jit defaults to "on" whether or not JIT is
enabled at compile time.

I thought Pavel was talking about 11 somehow...

I am sorry, It was not clear - I talked about master

This seems, at best, rather user-unfriendly.
And it's not like our conventions for other compile-option-affected
GUCs, eg the SSL ones.

That was intentional, even though it perhaps should be better documented.
That allows a distro to build and distribute pg without llvm enabled, but
then have the JIT package with all the dependencies separately. The pg yum
packages do so.

I don't like this solution - it is safe on production - but it can breaks
my development environment - or we need to change setup and make install
will not be enough.

Regards

Pavel

On October 8, 2018 8:16:06 AM PDT, Pavel Stehule <pavel.stehule@gmail.com> wrote:

po 8. 10. 2018 v 17:10 odesílatel Andres Freund <andres@anarazel.de>
napsal:

On October 8, 2018 8:03:56 AM PDT, Tom Lane <tgl@sss.pgh.pa.us>

wrote:

Andres Freund <andres@anarazel.de> writes:

Where is the jit=on coming from? Config from before it was turned

off?

A look in guc.c shows that jit defaults to "on" whether or not JIT

is

enabled at compile time.

I thought Pavel was talking about 11 somehow...

I am sorry, It was not clear - I talked about master

This seems, at best, rather user-unfriendly.
And it's not like our conventions for other compile-option-affected
GUCs, eg the SSL ones.

That was intentional, even though it perhaps should be better

documented.

That allows a distro to build and distribute pg without llvm enabled,

but

then have the JIT package with all the dependencies separately. The

pg yum

packages do so.

I don't like this solution - it is safe on production - but it can
breaks
my development environment - or we need to change setup and make
install
will not be enough.

It's not clear what could be done about it. You'll run into other trouble if you have half installed build artifacts because you reconfigured. Make uninstall from before the reconfigure would uninstall it.

Andres
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

po 8. 10. 2018 v 17:22 odesílatel Andres Freund <andres@anarazel.de> napsal:

On October 8, 2018 8:16:06 AM PDT, Pavel Stehule <pavel.stehule@gmail.com>
wrote:

po 8. 10. 2018 v 17:10 odesílatel Andres Freund <andres@anarazel.de>
napsal:

On October 8, 2018 8:03:56 AM PDT, Tom Lane <tgl@sss.pgh.pa.us>

wrote:

Andres Freund <andres@anarazel.de> writes:

Where is the jit=on coming from? Config from before it was turned

off?

A look in guc.c shows that jit defaults to "on" whether or not JIT

is

enabled at compile time.

I thought Pavel was talking about 11 somehow...

I am sorry, It was not clear - I talked about master

This seems, at best, rather user-unfriendly.
And it's not like our conventions for other compile-option-affected
GUCs, eg the SSL ones.

That was intentional, even though it perhaps should be better

documented.

That allows a distro to build and distribute pg without llvm enabled,

but

then have the JIT package with all the dependencies separately. The

pg yum

packages do so.

I don't like this solution - it is safe on production - but it can
breaks
my development environment - or we need to change setup and make
install
will not be enough.

It's not clear what could be done about it. You'll run into other trouble
if you have half installed build artifacts because you reconfigured. Make
uninstall from before the reconfigure would uninstall it.

It is partially true - when I use wrong extension, then I got error
message. But JIT library kills Postgres.

I expecting permanently disabled JIT if postgres was compiled without JIT
support. Dependency on some external file doesn't looks like safe solution.

Andres Freund <andres@anarazel.de> writes:

On October 8, 2018 8:03:56 AM PDT, Tom Lane <tgl@sss.pgh.pa.us> wrote:

A look in guc.c shows that jit defaults to "on" whether or not JIT is
enabled at compile time.
This seems, at best, rather user-unfriendly.
And it's not like our conventions for other compile-option-affected
GUCs, eg the SSL ones.

That was intentional, even though it perhaps should be better documented. That allows a distro to build and distribute pg without llvm enabled, but then have the JIT package with all the dependencies separately. The pg yum packages do so.

I'm not following. A distro wishing to do that would configure
--with-llvm, not without, and then simply split up the build results
so that the JIT stuff is in a separate subpackage. If you configured
--without-llvm then the resulting core code is (one hopes) entirely
incapable of using JIT, and it'd be better if the GUC settings
reflected that.

regards, tom lane

Hi,

On 2018-10-08 11:43:42 -0400, Tom Lane wrote:

Andres Freund <andres@anarazel.de> writes:

On October 8, 2018 8:03:56 AM PDT, Tom Lane <tgl@sss.pgh.pa.us> wrote:

A look in guc.c shows that jit defaults to "on" whether or not JIT is
enabled at compile time.
This seems, at best, rather user-unfriendly.
And it's not like our conventions for other compile-option-affected
GUCs, eg the SSL ones.

That was intentional, even though it perhaps should be better documented. That allows a distro to build and distribute pg without llvm enabled, but then have the JIT package with all the dependencies separately. The pg yum packages do so.

I'm not following. A distro wishing to do that would configure
--with-llvm, not without, and then simply split up the build results
so that the JIT stuff is in a separate subpackage.

Well, that'd then leave you with one more state (LLVM configured but not
installed, LLVM configured and installed, LLVM disabled and arguably
LLVM disabled but installed), but more importantly if you compile
without llvm enabled, you can still install a different extension that
would do JIT. You'd just have to set jit_provider = xyz, and it'd
work. If we made the generic JIT code depend on LLVM that'd end up
working pretty weirdly. I guess we could set jit_provider = ''
something if instead of hardcoding llvmjit if LLVM is disabled.

If you configured --without-llvm then the resulting core code is (one
hopes) entirely incapable of using JIT, and it'd be better if the GUC
settings reflected that.

That's not really the case, no. It controls whether the LLVM using jit
provider is built, not whether the generic JIT handling code is
available. Given that the generic code has no dependencies, there seems
little reason to do that differently?

Greetings,

Andres Freund

po 8. 10. 2018 v 17:59 odesílatel Andres Freund <andres@anarazel.de> napsal:

Hi,

On 2018-10-08 11:43:42 -0400, Tom Lane wrote:

Andres Freund <andres@anarazel.de> writes:

On October 8, 2018 8:03:56 AM PDT, Tom Lane <tgl@sss.pgh.pa.us> wrote:

A look in guc.c shows that jit defaults to "on" whether or not JIT is
enabled at compile time.
This seems, at best, rather user-unfriendly.
And it's not like our conventions for other compile-option-affected
GUCs, eg the SSL ones.

That was intentional, even though it perhaps should be better

documented. That allows a distro to build and distribute pg without llvm
enabled, but then have the JIT package with all the dependencies
separately. The pg yum packages do so.

I'm not following. A distro wishing to do that would configure
--with-llvm, not without, and then simply split up the build results
so that the JIT stuff is in a separate subpackage.

Well, that'd then leave you with one more state (LLVM configured but not
installed, LLVM configured and installed, LLVM disabled and arguably
LLVM disabled but installed), but more importantly if you compile
without llvm enabled, you can still install a different extension that
would do JIT. You'd just have to set jit_provider = xyz, and it'd
work. If we made the generic JIT code depend on LLVM that'd end up
working pretty weirdly. I guess we could set jit_provider = ''
something if instead of hardcoding llvmjit if LLVM is disabled.

If you configured --without-llvm then the resulting core code is (one
hopes) entirely incapable of using JIT, and it'd be better if the GUC
settings reflected that..

That's not really the case, no. It controls whether the LLVM using jit
provider is built, not whether the generic JIT handling code is
available. Given that the generic code has no dependencies, there seems
little reason to do that differently?

I can accept this logic, but it looks very fragile. Can be there some
safeguard against using wrong version or wrong API?

On October 8, 2018 10:16:54 AM PDT, Pavel Stehule <pavel.stehule@gmail.com> wrote:

po 8. 10. 2018 v 17:59 odesílatel Andres Freund <andres@anarazel.de>
napsal:

Hi,

On 2018-10-08 11:43:42 -0400, Tom Lane wrote:

Andres Freund <andres@anarazel.de> writes:

On October 8, 2018 8:03:56 AM PDT, Tom Lane <tgl@sss.pgh.pa.us>

wrote:

A look in guc.c shows that jit defaults to "on" whether or not

JIT is

enabled at compile time.
This seems, at best, rather user-unfriendly.
And it's not like our conventions for other

compile-option-affected

GUCs, eg the SSL ones.

That was intentional, even though it perhaps should be better

documented. That allows a distro to build and distribute pg without

llvm

enabled, but then have the JIT package with all the dependencies
separately. The pg yum packages do so.

I'm not following. A distro wishing to do that would configure
--with-llvm, not without, and then simply split up the build

results

so that the JIT stuff is in a separate subpackage.

Well, that'd then leave you with one more state (LLVM configured but

not

installed, LLVM configured and installed, LLVM disabled and arguably
LLVM disabled but installed), but more importantly if you compile
without llvm enabled, you can still install a different extension

that

would do JIT. You'd just have to set jit_provider = xyz, and it'd
work. If we made the generic JIT code depend on LLVM that'd end up
working pretty weirdly. I guess we could set jit_provider = ''
something if instead of hardcoding llvmjit if LLVM is disabled.

If you configured --without-llvm then the resulting core code is

(one

hopes) entirely incapable of using JIT, and it'd be better if the

GUC

settings reflected that..

That's not really the case, no. It controls whether the LLVM using

jit

provider is built, not whether the generic JIT handling code is
available. Given that the generic code has no dependencies, there

seems

little reason to do that differently?

I can accept this logic, but it looks very fragile. Can be there some
safeguard against using wrong version or wrong API?

To me that seems like an llvm / JIT independent piece of infrastructure. It'd probably be good if we had a catversion like thing to track ABI compatibility, but how to do so without making development unduly painful is less clear to me.

Andres
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

po 8. 10. 2018 v 19:24 odesílatel Andres Freund <andres@anarazel.de> napsal:

On October 8, 2018 10:16:54 AM PDT, Pavel Stehule <pavel.stehule@gmail.com>
wrote:

po 8. 10. 2018 v 17:59 odesílatel Andres Freund <andres@anarazel.de>
napsal:

Hi,

On 2018-10-08 11:43:42 -0400, Tom Lane wrote:

Andres Freund <andres@anarazel.de> writes:

On October 8, 2018 8:03:56 AM PDT, Tom Lane <tgl@sss.pgh.pa.us>

wrote:

A look in guc.c shows that jit defaults to "on" whether or not

JIT is

enabled at compile time.
This seems, at best, rather user-unfriendly.
And it's not like our conventions for other

compile-option-affected

GUCs, eg the SSL ones.

That was intentional, even though it perhaps should be better

documented. That allows a distro to build and distribute pg without

llvm

enabled, but then have the JIT package with all the dependencies
separately. The pg yum packages do so.

I'm not following. A distro wishing to do that would configure
--with-llvm, not without, and then simply split up the build

results

so that the JIT stuff is in a separate subpackage.

Well, that'd then leave you with one more state (LLVM configured but

not

installed, LLVM configured and installed, LLVM disabled and arguably
LLVM disabled but installed), but more importantly if you compile
without llvm enabled, you can still install a different extension

that

would do JIT. You'd just have to set jit_provider = xyz, and it'd
work. If we made the generic JIT code depend on LLVM that'd end up
working pretty weirdly. I guess we could set jit_provider = ''
something if instead of hardcoding llvmjit if LLVM is disabled.

If you configured --without-llvm then the resulting core code is

(one

hopes) entirely incapable of using JIT, and it'd be better if the

GUC

settings reflected that..

That's not really the case, no. It controls whether the LLVM using

jit

provider is built, not whether the generic JIT handling code is
available. Given that the generic code has no dependencies, there

seems

little reason to do that differently?

I can accept this logic, but it looks very fragile. Can be there some
safeguard against using wrong version or wrong API?

To me that seems like an llvm / JIT independent piece of infrastructure.
It'd probably be good if we had a catversion like thing to track ABI
compatibility, but how to do so without making development unduly painful
is less clear to me.

I am thinking so simple number should be good enough. We can require
equality - Just I need a signal so some is wrong - better than Postgres
crash.

On October 8, 2018 10:29:56 AM PDT, Pavel Stehule <pavel.stehule@gmail.com> wrote:

po 8. 10. 2018 v 19:24 odesílatel Andres Freund <andres@anarazel.de>
napsal:

On October 8, 2018 10:16:54 AM PDT, Pavel Stehule

<pavel.stehule@gmail.com>

wrote:

po 8. 10. 2018 v 17:59 odesílatel Andres Freund <andres@anarazel.de>
napsal:

Hi,

On 2018-10-08 11:43:42 -0400, Tom Lane wrote:

Andres Freund <andres@anarazel.de> writes:

On October 8, 2018 8:03:56 AM PDT, Tom Lane

<tgl@sss.pgh.pa.us>

wrote:

A look in guc.c shows that jit defaults to "on" whether or

not

JIT is

enabled at compile time.
This seems, at best, rather user-unfriendly.
And it's not like our conventions for other

compile-option-affected

GUCs, eg the SSL ones.

That was intentional, even though it perhaps should be better

documented. That allows a distro to build and distribute pg

without

llvm

enabled, but then have the JIT package with all the dependencies
separately. The pg yum packages do so.

I'm not following. A distro wishing to do that would configure
--with-llvm, not without, and then simply split up the build

results

so that the JIT stuff is in a separate subpackage.

Well, that'd then leave you with one more state (LLVM configured

but

not

installed, LLVM configured and installed, LLVM disabled and

arguably

LLVM disabled but installed), but more importantly if you compile
without llvm enabled, you can still install a different extension

that

would do JIT. You'd just have to set jit_provider = xyz, and it'd
work. If we made the generic JIT code depend on LLVM that'd end up
working pretty weirdly. I guess we could set jit_provider = ''
something if instead of hardcoding llvmjit if LLVM is disabled.

If you configured --without-llvm then the resulting core code is

(one

hopes) entirely incapable of using JIT, and it'd be better if

the

GUC

settings reflected that..

That's not really the case, no. It controls whether the LLVM using

jit

provider is built, not whether the generic JIT handling code is
available. Given that the generic code has no dependencies, there

seems

little reason to do that differently?

I can accept this logic, but it looks very fragile. Can be there

some

safeguard against using wrong version or wrong API?

To me that seems like an llvm / JIT independent piece of

infrastructure.

It'd probably be good if we had a catversion like thing to track ABI
compatibility, but how to do so without making development unduly

painful

is less clear to me.

I am thinking so simple number should be good enough. We can require
equality - Just I need a signal so some is wrong - better than Postgres
crash.

It'd cause constant conflicts and / or we would regularly forget updating it. It's not that trivial to determine what influences ABI compatibility.

Andres
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Andres Freund <andres@anarazel.de> writes:

I am thinking so simple number should be good enough. We can require
equality - Just I need a signal so some is wrong - better than Postgres
crash.

It'd cause constant conflicts and / or we would regularly forget updating it. It's not that trivial to determine what influences ABI compatibility.

There already is a PG major-version-number check embedded in the
PG_MODULE_MAGIC infrastructure, which is plenty for regular users.
It's not sufficient for developers working with HEAD, of course.

We could consider making that work off of catversion instead, but I don't
think it'd really improve matters to do so. catversion doesn't cover most
of what can break loadable modules, such as changes of Node data
structures.

regards, tom lane

I am thinking so simple number should be good enough. We can require
equality - Just I need a signal so some is wrong - better than Postgres
crash.

It'd cause constant conflicts and / or we would regularly forget updating
it. It's not that trivial to determine what influences ABI compatibility.

This can be checked by regress tests? I don't know. Maybe I am not too
friendly, I am sorry. I spent 20 minutes by searching phantom, because JIT
was active, although I wanted.

So any help against this situation can be welcome.

Regards

Pavel