Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

On 8/5/21 11:29 PM, Andres Freund wrote:

Hi,

When testing EXEC_BACKEND on linux I see occasional test failures as long as I
don't disable ASLR. There's a code comment to that effect:

* If testing EXEC_BACKEND on Linux, you should run this as root before
* starting the postmaster:
*
* echo 0 >/proc/sys/kernel/randomize_va_space

but I don't like doing that on a system wide basis.

Linux allows disabling ASLR on a per-process basis using
personality(ADDR_NO_RANDOMIZE). There's a wrapper binary to do that as well,
setarch --addr-no-randomize.

I was wondering if we should have postmaster do personality(ADDR_NO_RANDOMIZE)
for EXEC_BACKEND builds? It seems nicer to make it automatically work than
have people remember that they need to call "setarch --addr-no-randomize make check".

Not that it actually matters for EXEC_BACKEND, but theoretically doing
personality(ADDR_NO_RANDOMIZE) in postmaster is a tad more secure than doing
it via setarch, as in the personality() case postmaster's layout itself is
still randomized...

Or perhaps we should just add a comment mentioning setarch.

If we can set it conveniently then that seems worth doing.

(Thinks: do we have non-Windows buildfarm members doing EXEC_BACKEND?)

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

On 2021-Aug-06, Andrew Dunstan wrote:

On 8/5/21 11:29 PM, Andres Freund wrote:

I was wondering if we should have postmaster do personality(ADDR_NO_RANDOMIZE)
for EXEC_BACKEND builds? It seems nicer to make it automatically work than
have people remember that they need to call "setarch --addr-no-randomize make check".

How common is to get a failure? I know I've run tests under
EXEC_BACKEND and not seen any failures. Not many runs though.

Not that it actually matters for EXEC_BACKEND, but theoretically doing
personality(ADDR_NO_RANDOMIZE) in postmaster is a tad more secure than doing
it via setarch, as in the personality() case postmaster's layout itself is
still randomized...

True. I think the security aspect is not critically important, since
hopefully nobody should be using such builds for production.

(Thinks: do we have non-Windows buildfarm members doing EXEC_BACKEND?)

culicidae does that.

--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Pido que me den el Nobel por razones humanitarias" (Nicanor Parra)

On Mon, Aug 9, 2021 at 1:30 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:

How common is to get a failure? I know I've run tests under
EXEC_BACKEND and not seen any failures. Not many runs though.

On macOS, failures are extremely common. Sometimes I have to run
simple tests many times to get even one success. The proposal on the
table won't help with that problem since it's Linux-specific, but if
there's any way to do something similar on macOS it would be a _huge_
help.

--
Robert Haas
EDB: http://www.enterprisedb.com

Hi,

On 2021-08-09 13:43:03 -0400, Robert Haas wrote:

On Mon, Aug 9, 2021 at 1:30 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:

How common is to get a failure? I know I've run tests under
EXEC_BACKEND and not seen any failures. Not many runs though.

I get check-world failures in about 1/2-1/3 of the runs, and a plain check
fails in maybe 1/4 of the cases. It's pretty annoying because it often isn't
trivial to distinguish whether I've broken something or whether it's
randomization related...

The frequency of failures isn't stable over time, making it a bit harder to
know what's going on.

On macOS, failures are extremely common. Sometimes I have to run
simple tests many times to get even one success. The proposal on the
table won't help with that problem since it's Linux-specific, but if
there's any way to do something similar on macOS it would be a _huge_
help.

There does seem to be a way of doing so, because I found blog posts talking
about how to get e.g. lldb to *enable* ASLR, which it disables by default...

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:

On 2021-08-09 13:43:03 -0400, Robert Haas wrote:

On Mon, Aug 9, 2021 at 1:30 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:

How common is to get a failure? I know I've run tests under
EXEC_BACKEND and not seen any failures. Not many runs though.

I get check-world failures in about 1/2-1/3 of the runs, and a plain check
fails in maybe 1/4 of the cases. It's pretty annoying because it often isn't
trivial to distinguish whether I've broken something or whether it's
randomization related...

I don't have numbers, but I do know that on Linux EXEC_BACKEND builds fail
often enough to be annoying if you don't disable ASLR. If we can do
something not-too-invasive about that, it'd be great.

regards, tom lane

Hi,

On 2021-08-09 13:54:25 -0400, Tom Lane wrote:

Andres Freund <andres@anarazel.de> writes:

On 2021-08-09 13:43:03 -0400, Robert Haas wrote:

On Mon, Aug 9, 2021 at 1:30 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:

How common is to get a failure? I know I've run tests under
EXEC_BACKEND and not seen any failures. Not many runs though.

I get check-world failures in about 1/2-1/3 of the runs, and a plain check
fails in maybe 1/4 of the cases. It's pretty annoying because it often isn't
trivial to distinguish whether I've broken something or whether it's
randomization related...

I don't have numbers, but I do know that on Linux EXEC_BACKEND builds fail
often enough to be annoying if you don't disable ASLR. If we can do
something not-too-invasive about that, it'd be great.

I now not sure if personality(NO_RANDOMIZE) in postmaster is actually
sufficient. It does seem to drastically reduce the frequency of issues, but
there's still conceivable failures where postmaster's layout would class with
the non-randomized children - obviously personality(NO_RANDOMIZE) cannot
retroactively change the layout of the already running binary.

So maybe we should put it into pg_ctl?

Greetings,

Andres Freund

On Tue, Aug 10, 2021 at 5:43 AM Robert Haas <robertmhaas@gmail.com> wrote:

On Mon, Aug 9, 2021 at 1:30 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:

How common is to get a failure? I know I've run tests under
EXEC_BACKEND and not seen any failures. Not many runs though.

On macOS, failures are extremely common. Sometimes I have to run
simple tests many times to get even one success. The proposal on the
table won't help with that problem since it's Linux-specific, but if
there's any way to do something similar on macOS it would be a _huge_
help.

Yeah, make check always fails for me on macOS 11. With the attached
experimental hack, it fails only occasionally (1 in 8 runs or so). I
don't know why.

Hi,

On Tue, Aug 10, 2021, at 15:19, Thomas Munro wrote:

On Tue, Aug 10, 2021 at 5:43 AM Robert Haas <robertmhaas@gmail.com> wrote:

On Mon, Aug 9, 2021 at 1:30 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:

How common is to get a failure? I know I've run tests under
EXEC_BACKEND and not seen any failures. Not many runs though.

On macOS, failures are extremely common. Sometimes I have to run
simple tests many times to get even one success. The proposal on the
table won't help with that problem since it's Linux-specific, but if
there's any way to do something similar on macOS it would be a _huge_
help.

Yeah, make check always fails for me on macOS 11. With the attached
experimental hack, it fails only occasionally (1 in 8 runs or so). I
don't know why.

I suspect you'd need to use the hack in pg_ctl to make it reliable. The layout of normally stayed position independent postmaster can be incompatible with the non ASLR spawned child.

Andres

On Wed, Aug 11, 2021 at 2:12 AM Andres Freund <andres@anarazel.de> wrote:

On Tue, Aug 10, 2021, at 15:19, Thomas Munro wrote:

Yeah, make check always fails for me on macOS 11. With the attached
experimental hack, it fails only occasionally (1 in 8 runs or so). I
don't know why.

I suspect you'd need to use the hack in pg_ctl to make it reliable. The layout of normally stayed position independent postmaster can be incompatible with the non ASLR spawned child.

Yeah, but the patch already changes both pg_ctl.c and postmaster.c.

On Wed, Aug 11, 2021 at 7:07 AM Thomas Munro <thomas.munro@gmail.com> wrote:

On Wed, Aug 11, 2021 at 2:12 AM Andres Freund <andres@anarazel.de> wrote:

On Tue, Aug 10, 2021, at 15:19, Thomas Munro wrote:

Yeah, make check always fails for me on macOS 11. With the attached
experimental hack, it fails only occasionally (1 in 8 runs or so). I
don't know why.

I suspect you'd need to use the hack in pg_ctl to make it reliable. The layout of normally stayed position independent postmaster can be incompatible with the non ASLR spawned child.

Yeah, but the patch already changes both pg_ctl.c and postmaster.c.

/me stares at vmmap output for a while

Oooh. It's working perfectly (for example if you export
PATH=binarys:$PATH, pg_ctl -D pgdata start, make installcheck), but
pg_regress.c has its own separate fork/exec to launch the temporary
cluster that needs to be similarly hacked. Unfortunately I have to
give this Macintosh back and go and do some real work on a different
computer now. That does seem to be a working solution to the problem,
though, and could be polished into proposable form.

I saw claims that you can also link with -Wl,-no_pie or toggle the PIE
bit on your executable and libraries, but that didn't work for me on
11, Intel (no effect) or ARM (linker option gone).

On 8/10/21 7:59 PM, Thomas Munro wrote:

On Wed, Aug 11, 2021 at 7:07 AM Thomas Munro <thomas.munro@gmail.com> wrote:

On Wed, Aug 11, 2021 at 2:12 AM Andres Freund <andres@anarazel.de> wrote:

On Tue, Aug 10, 2021, at 15:19, Thomas Munro wrote:

Yeah, make check always fails for me on macOS 11. With the attached
experimental hack, it fails only occasionally (1 in 8 runs or so). I
don't know why.

I suspect you'd need to use the hack in pg_ctl to make it reliable. The layout of normally stayed position independent postmaster can be incompatible with the non ASLR spawned child.

Yeah, but the patch already changes both pg_ctl.c and postmaster.c.

/me stares at vmmap output for a while

Oooh. It's working perfectly (for example if you export
PATH=binarys:$PATH, pg_ctl -D pgdata start, make installcheck), but
pg_regress.c has its own separate fork/exec to launch the temporary
cluster that needs to be similarly hacked.

initdb also runs postgres a bunch of times via system(), similarly to
pg_regress but without the "exec". Does it also need adjusting?

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

On Tue, Aug 10, 2021 at 7:59 PM Thomas Munro <thomas.munro@gmail.com> wrote:

I saw claims that you can also link with -Wl,-no_pie or toggle the PIE
bit on your executable and libraries, but that didn't work for me on
11, Intel (no effect) or ARM (linker option gone).

I think that worked for me on older macOS releases, and then it
stopped working at some point after some update or reinstall or
something. Unfortunately I don't know any more precisely than that,
but it does seem like we have to find some other approach to work on
modern systems.

Hopefully they don't keep whacking this around...

--
Robert Haas
EDB: http://www.enterprisedb.com

Andrew Dunstan <andrew@dunslane.net> writes:

initdb also runs postgres a bunch of times via system(), similarly to
pg_regress but without the "exec". Does it also need adjusting?

That shouldn't be an issue, because we're only running the single
"standalone backend" process, not a cluster.

regards, tom lane

On Thu, Aug 12, 2021 at 1:45 AM Robert Haas <robertmhaas@gmail.com> wrote:

I think that worked for me on older macOS releases, and then it
stopped working at some point after some update or reinstall or
something. Unfortunately I don't know any more precisely than that,
but it does seem like we have to find some other approach to work on
modern systems.

I gave up on trying to make that work once I realised that
/usr/lib/dyld doesn't seem to obey the flag, so although other
segments become deterministic and the success rate is fairly high,
there's still a 600kb wrecking ball swinging around. I wondered what
the "slide" range could be... it appears to be fairly small
(vm_map_get_max_aslr_slide_pages() seems to be the place that's
determined and it's a 16MB or 256MB window, depending on architecture,
if I read that right). Given that, the death of 32 bit processes
since Catalina, and the typical layout we see, I think just doing
something like (/me rolls dice) export PG_SHMEM_ADDR=0x80000000000 is
a good candidate for something that works on both architectures, being
many TB away from everything else (above everything on ARM, between
heap etc and libs on Intel but with 8TB of space below it and 120TB
above). That gets the tests passing consistently with unpatched
master, -DEXEC_BACKEND, on both flavours of silicon.

On Wed, Aug 11, 2021 at 6:24 PM Thomas Munro <thomas.munro@gmail.com> wrote:

On Thu, Aug 12, 2021 at 1:45 AM Robert Haas <robertmhaas@gmail.com> wrote:

I think that worked for me on older macOS releases, and then it
stopped working at some point after some update or reinstall or
something. Unfortunately I don't know any more precisely than that,
but it does seem like we have to find some other approach to work on
modern systems.

I gave up on trying to make that work once I realised that
/usr/lib/dyld doesn't seem to obey the flag, so although other
segments become deterministic and the success rate is fairly high,
there's still a 600kb wrecking ball swinging around. I wondered what
the "slide" range could be... it appears to be fairly small
(vm_map_get_max_aslr_slide_pages() seems to be the place that's
determined and it's a 16MB or 256MB window, depending on architecture,
if I read that right). Given that, the death of 32 bit processes
since Catalina, and the typical layout we see, I think just doing
something like (/me rolls dice) export PG_SHMEM_ADDR=0x80000000000 is
a good candidate for something that works on both architectures, being
many TB away from everything else (above everything on ARM, between
heap etc and libs on Intel but with 8TB of space below it and 120TB
above). That gets the tests passing consistently with unpatched
master, -DEXEC_BACKEND, on both flavours of silicon.

Ugh, OK. So, is there a way that we can get an "easy button" committed
to the tree?

--
Robert Haas
EDB: http://www.enterprisedb.com

Robert Haas <robertmhaas@gmail.com> writes:

On Wed, Aug 11, 2021 at 6:24 PM Thomas Munro <thomas.munro@gmail.com> wrote:

... I think just doing
something like (/me rolls dice) export PG_SHMEM_ADDR=0x80000000000 is
a good candidate for something that works on both architectures, being
many TB away from everything else (above everything on ARM, between
heap etc and libs on Intel but with 8TB of space below it and 120TB
above). That gets the tests passing consistently with unpatched
master, -DEXEC_BACKEND, on both flavours of silicon.

Ugh, OK. So, is there a way that we can get an "easy button" committed
to the tree?

I don't see why that approach couldn't be incorporated into pg_ctl,
or the postmaster itself. Given Andres' point that Linux ASLR
disable probably has to happen in pg_ctl, it seems like doing it
in pg_ctl in all cases is the way to move forward.

regards, tom lane

On Fri, Aug 13, 2021 at 3:13 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Robert Haas <robertmhaas@gmail.com> writes:

Ugh, OK. So, is there a way that we can get an "easy button" committed
to the tree?

I don't see why that approach couldn't be incorporated into pg_ctl,
or the postmaster itself. Given Andres' point that Linux ASLR
disable probably has to happen in pg_ctl, it seems like doing it
in pg_ctl in all cases is the way to move forward.

I think doing it in the postmaster is best, since otherwise you have
to put code into pg_regress.c and pg_ctl.c. Here's a patch like that.

Thomas Munro <thomas.munro@gmail.com> writes:

On Fri, Aug 13, 2021 at 3:13 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:

I don't see why that approach couldn't be incorporated into pg_ctl,
or the postmaster itself. Given Andres' point that Linux ASLR
disable probably has to happen in pg_ctl, it seems like doing it
in pg_ctl in all cases is the way to move forward.

I think doing it in the postmaster is best, since otherwise you have
to put code into pg_regress.c and pg_ctl.c. Here's a patch like that.

Hmm, ok. Small thought: it might be better to put the #if inside
the "else { .... }". That way it scales easily to allow other
platform-specific defaults if we find anything useful. As-is,
the obvious extension would end up with multiple else-blocks,
which seems likely to confuse pgindent if nothing else.

regards, tom lane

On Fri, Aug 13, 2021 at 9:59 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Hmm, ok. Small thought: it might be better to put the #if inside
the "else { .... }". That way it scales easily to allow other
platform-specific defaults if we find anything useful. As-is,
the obvious extension would end up with multiple else-blocks,
which seems likely to confuse pgindent if nothing else.

True. Thanks. Pushed to all live branches.

Obviously the address may have to be adjusted if Apple moves stuff
around, or if the initial layout and ASLR slide range turn out to be
less constrained than I inferred by nosing around the source code and
testing on a couple of systems.