PG v12.2 - Setting jit_above_cost is causing the server to crash

++hackers.

On Mon, Feb 24, 2020 at 12:16 PM Aditya Toshniwal <
aditya.toshniwal@enterprisedb.com> wrote:

Hi Team,

The PG 12.2 server is crashing on setting the jit_above_cost param. Below
is the output.

postgres=# select version();

version

----------------------------------------------------------------------------------------------------------------------------

PostgreSQL 12.2 on x86_64-apple-darwin, compiled by Apple LLVM version
6.0 (clang-600.0.54) (based on LLVM 3.5svn), 64-bit

(1 row)

postgres=# SET jit_above_cost=10;

SET

postgres=# SELECT count(*) FROM pg_class;

server closed the connection unexpectedly

This probably means the server terminated abnormally

before or while processing the request.

The connection to the server was lost. Attempting reset: Failed.

!>

--
Thanks and Regards,
Aditya Toshniwal
pgAdmin Hacker | Sr. Software Engineer | EnterpriseDB India | Pune
"Don't Complain about Heat, Plant a TREE"

--
Thanks and Regards,
Aditya Toshniwal
pgAdmin Hacker | Sr. Software Engineer | EnterpriseDB India | Pune
"Don't Complain about Heat, Plant a TREE"

Hi,

On 2020-02-24 12:16:08 +0530, Aditya Toshniwal wrote:

The PG 12.2 server is crashing on setting the jit_above_cost param. Below
is the output.

postgres=# SELECT count(*) FROM pg_class;

server closed the connection unexpectedly

This probably means the server terminated abnormally

before or while processing the request.

The connection to the server was lost. Attempting reset: Failed.

This isn't reproducible here. Are you sure that you're running on a
clean installation?

If not, we'd at least need a backtrace to figure out what's going on.

Greetings,

Andres Freund

Hi Andres,

On Mon, Feb 24, 2020 at 12:46 PM Andres Freund <andres@anarazel.de> wrote:

Hi,

On 2020-02-24 12:16:08 +0530, Aditya Toshniwal wrote:

The PG 12.2 server is crashing on setting the jit_above_cost param. Below
is the output.

postgres=# SELECT count(*) FROM pg_class;

server closed the connection unexpectedly

This probably means the server terminated abnormally

before or while processing the request.

The connection to the server was lost. Attempting reset: Failed.

This isn't reproducible here. Are you sure that you're running on a
clean installation?

Yes I did a fresh installation using installer provided here -
https://www.enterprisedb.com/downloads/postgresql

If not, we'd at least need a backtrace to figure out what's going on.

Please let me know how can I provide required info.

Greetings,

Andres Freund

--
Thanks and Regards,
Aditya Toshniwal
pgAdmin Hacker | Sr. Software Engineer | EnterpriseDB India | Pune
"Don't Complain about Heat, Plant a TREE"

PostgreSQL 12.2 ...
compiled by Apple LLVM version 6.0(clang-600.0.54)* (based on LLVM 3.5svn)*

this LLVM is >= 3.9 ?

according to the docs : "Build with support for LLVM based JIT compilation
.... *The minimum required version of LLVM is currently 3.9.*"
https://www.postgresql.org/docs/12/install-procedure.html
Regards,
Imre

Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> ezt írta (időpont:
2020. febr. 24., H, 7:46):

Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> writes:

On Mon, Feb 24, 2020 at 12:46 PM Andres Freund <andres@anarazel.de> wrote:

This isn't reproducible here. Are you sure that you're running on a
clean installation?

Yes I did a fresh installation using installer provided here -
https://www.enterprisedb.com/downloads/postgresql

There is apparently something wrong with the JIT stuff in EDB's 12.2
build for macOS. At least, that's the conclusion I came to after
off-list discussion with the submitter of bug #16264, which has pretty
much exactly this symptom (especially if you're seeing "signal 9"
reports in the postmaster log). For him, either disabling JIT or
reverting to 12.1 made it go away.

regards, tom lane

On Mon, Feb 24, 2020 at 8:20 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> writes:

On Mon, Feb 24, 2020 at 12:46 PM Andres Freund <andres@anarazel.de>

wrote:

This isn't reproducible here. Are you sure that you're running on a
clean installation?

Yes I did a fresh installation using installer provided here -
https://www.enterprisedb.com/downloads/postgresql

There is apparently something wrong with the JIT stuff in EDB's 12.2
build for macOS. At least, that's the conclusion I came to after
off-list discussion with the submitter of bug #16264, which has pretty
much exactly this symptom (especially if you're seeing "signal 9"
reports in the postmaster log). For him, either disabling JIT or
reverting to 12.1 made it go away.

Yes it seems like issue with EDB build. It works fine on macOS < Catalina.

regards, tom lane

--
Thanks and Regards,
Aditya Toshniwal
pgAdmin Hacker | Sr. Software Engineer | EnterpriseDB India | Pune
"Don't Complain about Heat, Plant a TREE"

Hi

On Thu, Feb 27, 2020 at 12:41 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> writes:

On Mon, Feb 24, 2020 at 12:46 PM Andres Freund <andres@anarazel.de>

wrote:

This isn't reproducible here. Are you sure that you're running on a
clean installation?

Yes I did a fresh installation using installer provided here -
https://www.enterprisedb.com/downloads/postgresql

There is apparently something wrong with the JIT stuff in EDB's 12.2
build for macOS. At least, that's the conclusion I came to after
off-list discussion with the submitter of bug #16264, which has pretty
much exactly this symptom (especially if you're seeing "signal 9"
reports in the postmaster log). For him, either disabling JIT or
reverting to 12.1 made it go away.

We've been looking into this;

Apple started a notarisation process some time ago, designed to mark their
applications as conforming to various security requirements, but prior to
Catalina it was essentially optional. When Catalina was released, they made
notarisation for distributed software a requirement, but had the process
issue warnings for non-compliance. As-of the end of January, those warnings
became hard errors, so now our packages must be notarised, and for that to
happen, must be hardened by linking with a special runtime and having
securely time stamped signatures on every binary before being checked and
notarised as such by Apple. Without that, users would have to disable
security features on their systems before they could run our software.

Our packages are being successfully notarised at the moment, because that's
essentially done through a static analysis. We can (and have) added what
Apple call an entitlement in test builds which essentially puts a flag in
the notarisation for the product that declares that it will do JIT
operations, however, it seems that this alone is not enough and that in
addition to the entitlement, we also need to include the MAP_JIT flag in
mmap() calls. See
https://developer.apple.com/documentation/security/hardened_runtime and
https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_security_cs_allow-jit

We're working on trying to test a patch for that at the moment.

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hi,

On Thu, Feb 27, 2020 at 6:23 PM Dave Page <dpage@pgadmin.org> wrote:

Hi

On Thu, Feb 27, 2020 at 12:41 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> writes:

On Mon, Feb 24, 2020 at 12:46 PM Andres Freund <andres@anarazel.de>

wrote:

This isn't reproducible here. Are you sure that you're running on a
clean installation?

Yes I did a fresh installation using installer provided here -
https://www.enterprisedb.com/downloads/postgresql

There is apparently something wrong with the JIT stuff in EDB's 12.2
build for macOS. At least, that's the conclusion I came to after
off-list discussion with the submitter of bug #16264, which has pretty
much exactly this symptom (especially if you're seeing "signal 9"
reports in the postmaster log). For him, either disabling JIT or
reverting to 12.1 made it go away.

We've been looking into this;

Apple started a notarisation process some time ago, designed to mark their
applications as conforming to various security requirements, but prior to
Catalina it was essentially optional. When Catalina was released, they made
notarisation for distributed software a requirement, but had the process
issue warnings for non-compliance. As-of the end of January, those warnings
became hard errors, so now our packages must be notarised, and for that to
happen, must be hardened by linking with a special runtime and having
securely time stamped signatures on every binary before being checked and
notarised as such by Apple. Without that, users would have to disable
security features on their systems before they could run our software.

Our packages are being successfully notarised at the moment, because
that's essentially done through a static analysis. We can (and have) added
what Apple call an entitlement in test builds which essentially puts a flag
in the notarisation for the product that declares that it will do JIT
operations, however, it seems that this alone is not enough and that in
addition to the entitlement, we also need to include the MAP_JIT flag in
mmap() calls. See
https://developer.apple.com/documentation/security/hardened_runtime and
https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_security_cs_allow-jit

We're working on trying to test a patch for that at the moment.

We have fixed the issue. To explain in brief, It was related to the
hardened runtime. Hardening the runtime can produce such issues, and
therefore Apple provides the runtime exceptions. We were previously using
an entitlement "com.apple.security.cs.disable-library-validation" for the
app bundle and then tried adding
"com.apple.security.cs.allow-unsigned-executable-memory" but still the
query would crash the server process when JIT is enabled. Later we applied
these entitlements to the PG binaries (postgres, pg_ctl and others) and the
bundles (llvmjit.so and others) which actually resolved the problem.

The updates will be released in a couple of days.

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sandeep Thakkar