PG vs LLVM 12 on seawasp, next round
Hi
Since seawasp's bleeding edge LLVM installation moved to "trunk
20201114 c8f4e06b 12.0.0" ~16 days ago, it has been red. Further
updates didn't help it and it's now on "trunk 20201127 6ee22ca6
12.0.0". I wonder if there is something in Fabien's scripting that
needs to be tweaked, perhaps a symlink name or similar. I don't
follow LLVM development but I found my way to a commit[1]https://github.com/llvm/llvm-project/commit/1d0676b54c4e3a517719220def96dfdbc26d8048 around the
right time that mentions breaking up the OrcJIT library, so *shrug*
maybe that's a clue.
+ERROR: could not load library
"/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so":
libLLVMOrcJIT.so.12git: cannot open shared object file: No such file
or directory
[1]: https://github.com/llvm/llvm-project/commit/1d0676b54c4e3a517719220def96dfdbc26d8048
Hi,
On 2020-12-01 17:35:49 +1300, Thomas Munro wrote:
Since seawasp's bleeding edge LLVM installation moved to "trunk
20201114 c8f4e06b 12.0.0" ~16 days ago, it has been red. Further
updates didn't help it and it's now on "trunk 20201127 6ee22ca6
12.0.0". I wonder if there is something in Fabien's scripting that
needs to be tweaked, perhaps a symlink name or similar. I don't
follow LLVM development but I found my way to a commit[1] around the
right time that mentions breaking up the OrcJIT library, so *shrug*
maybe that's a clue.+ERROR: could not load library
"/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so":
libLLVMOrcJIT.so.12git: cannot open shared object file: No such file
or directory
It's a change in how LLVM dependencies are declared
internally. Previously the 'native' component was - unintentionally -
transitively included via the 'orcjit' component, but now that's not the
case anymore.
The attached patch should fix it, I think?
Greetings,
Andres Freund
Attachments:
llvm-build.difftext/x-diff; charset=us-asciiDownload
diff --git i/config/llvm.m4 w/config/llvm.m4
index a5f4a9af448..3a75cd8b4df 100644
--- i/config/llvm.m4
+++ w/config/llvm.m4
@@ -76,6 +76,7 @@ AC_DEFUN([PGAC_LLVM_SUPPORT],
debuginfodwarf) pgac_components="$pgac_components $pgac_component";;
orcjit) pgac_components="$pgac_components $pgac_component";;
passes) pgac_components="$pgac_components $pgac_component";;
+ native) pgac_components="$pgac_components $pgac_component";;
perfjitevents) pgac_components="$pgac_components $pgac_component";;
esac
done;
diff --git i/configure w/configure
index ffcd0c5b1d4..2a03ed0a018 100755
--- i/configure
+++ w/configure
@@ -5168,6 +5168,7 @@ fi
debuginfodwarf) pgac_components="$pgac_components $pgac_component";;
orcjit) pgac_components="$pgac_components $pgac_component";;
passes) pgac_components="$pgac_components $pgac_component";;
+ native) pgac_components="$pgac_components $pgac_component";;
perfjitevents) pgac_components="$pgac_components $pgac_component";;
esac
done;
Hello Thomas,
Since seawasp's bleeding edge LLVM installation moved to "trunk
20201114 c8f4e06b 12.0.0" ~16 days ago, it has been red. Further
updates didn't help it and it's now on "trunk 20201127 6ee22ca6
12.0.0". I wonder if there is something in Fabien's scripting that
needs to be tweaked, perhaps a symlink name or similar.
The compiler compilation script is quite straightforward (basically, get
sources, configure and compile), even for a such a moving target…
The right approach is to wait for some time before looking at the issue,
typically one week for the next recompilation, in case the problem
evaporates, so you were right not to jump on it right away:-)
Andres investigated a few days ago, managed to reproduce the issue
locally, and has one line patch. I'm unsure if it should be prevently
back-patched, though.
--
Fabien.
Hi,
On 2020-12-01 21:04:44 +0100, Fabien COELHO wrote:
Andres investigated a few days ago, managed to reproduce the issue locally,
and has one line patch. I'm unsure if it should be prevently back-patched,
though.
I see no reason not to backpatch - it's more correct for past versions
of LLVM as well.
Greetings,
Andres Freund
Hi,
On 2020-12-01 12:08:10 -0800, Andres Freund wrote:
On 2020-12-01 21:04:44 +0100, Fabien COELHO wrote:
Andres investigated a few days ago, managed to reproduce the issue locally,
and has one line patch. I'm unsure if it should be prevently back-patched,
though.I see no reason not to backpatch - it's more correct for past versions
of LLVM as well.
I pushed that now.
- Andres
Hi,
On 2020-12-07 19:38:19 -0800, Andres Freund wrote:
On 2020-12-01 12:08:10 -0800, Andres Freund wrote:
On 2020-12-01 21:04:44 +0100, Fabien COELHO wrote:
Andres investigated a few days ago, managed to reproduce the issue locally,
and has one line patch. I'm unsure if it should be prevently back-patched,
though.I see no reason not to backpatch - it's more correct for past versions
of LLVM as well.I pushed that now.
I hadn't checked that before, but for the last few days there's been a
different failure than the one I saw earlier:
+ERROR: could not load library "/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so": libLLVMOrcJIT.so.12git: cannot open shared object file: No such file or directory
whereas what I fixed is about:
+ERROR: could not load library "/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so": /home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so: undefined symbol: LLVMInitializeX86Target
Changed somewhere between
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=seawasp&dt=2020-11-20%2009%3A17%3A10
and
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=seawasp&dt=2020-11-21%2023%3A17%3A11
The "no such file" error seems more like a machine local issue to me.
Greetings,
Andres Freund
Hello Andres,
I hadn't checked that before, but for the last few days there's been a
different failure than the one I saw earlier:+ERROR: could not load library "/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so": libLLVMOrcJIT.so.12git: cannot open shared object file: No such file or directory
whereas what I fixed is about:
+ERROR: could not load library "/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so": /home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so: undefined symbol: LLVMInitializeX86Target
Changed somewhere between
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=seawasp&dt=2020-11-20%2009%3A17%3A10
and
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=seawasp&dt=2020-11-21%2023%3A17%3A11The "no such file" error seems more like a machine local issue to me.
I'll look into it when I have time, which make take some time. Hopefully
over the holidays.
--
Fabien.
On 2020-Dec-11, Fabien COELHO wrote:
I hadn't checked that before, but for the last few days there's been a
different failure than the one I saw earlier:+ERROR: could not load library "/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so": libLLVMOrcJIT.so.12git: cannot open shared object file: No such file or directory
The "no such file" error seems more like a machine local issue to me.
I'll look into it when I have time, which make take some time. Hopefully
over the holidays.
This is still happening ... Any chance you can have a look at it?
--
�lvaro Herrera 39�49'30"S 73�17'W
Alvaro Herrera <alvherre@alvh.no-ip.org> writes:
On 2020-Dec-11, Fabien COELHO wrote:
I'll look into it when I have time, which make take some time. Hopefully
over the holidays.
This is still happening ... Any chance you can have a look at it?
If you don't have time to debug it, perhaps you could just disable
the buildfarm animal till you do. It's cluttering the buildfarm
failure report without providing useful info ...
regards, tom lane
Hello Alvaro,
The "no such file" error seems more like a machine local issue to me.
I'll look into it when I have time, which make take some time. Hopefully
over the holidays.This is still happening ... Any chance you can have a look at it?
Indeed. I'll try to look (again) into it soon. I had a look but did not
find anything obvious in the short time frame I had. Last two months were
a little overworked for me so I let slip quite a few things. If you want
to disable the animal as Tom suggests, do as you want.
--
Fabien.
On Mon, Jan 18, 2021 at 09:29:53PM +0100, Fabien COELHO wrote:
The "no such file" error seems more like a machine local issue to me.
I'll look into it when I have time, which make take some time. Hopefully
over the holidays.This is still happening ... Any chance you can have a look at it?
Indeed. I'll try to look (again) into it soon. I had a look but did not find
anything obvious in the short time frame I had. Last two months were a
little overworked for me so I let slip quite a few things. If you want to
disable the animal as Tom suggests, do as you want.
Perhaps he was suggesting that you (buildfarm owner) disable the cron job that
initiates new runs. That's what I do when one of my animals needs my
intervention.
Noah Misch <noah@leadboat.com> writes:
On Mon, Jan 18, 2021 at 09:29:53PM +0100, Fabien COELHO wrote:
... Last two months were a
little overworked for me so I let slip quite a few things. If you want to
disable the animal as Tom suggests, do as you want.
Perhaps he was suggesting that you (buildfarm owner) disable the cron job that
initiates new runs. That's what I do when one of my animals needs my
intervention.
Indeed. I'm not sure there even is a provision to block an animal on the
buildfarm-server side. If there is, you'd have to request that it be
manually undone after you get around to fixing the animal. Frankly,
if I were the BF admin, I would be in about as much hurry to do that
as you've been to fix it.
regards, tom lane
On Sat, Dec 12, 2020 at 8:45 AM Fabien COELHO <coelho@cri.ensmp.fr> wrote:
+ERROR: could not load library "/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so": libLLVMOrcJIT.so.12git: cannot open shared object file: No such file or directory
Bonjour Fabien,
Here is the creation of llvmjit.so:
g++ ... -o llvmjit.so ... -L/home/fabien/clgtk/lib ... -lLLVMOrcJIT ...
That'd be from llvm-config --ldflags or similar, from this binary:
checking for llvm-config... (cached) /home/fabien/clgtk/bin/llvm-config
So what does ls -slap /home/fabien/clgtk/lib show? What does ldd
/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so
show? What do you have in seawap's LD_LIBRARY_PATH, and is the
problem that you need to add /home/fabien/clgtk/lib to it (or is it
supposed to be making it into the rpath, or is it in your ld.so.conf)?
PS Could you try blowing away the accache directory so we can rule out
bad cached configure stuff?
Hello Thomas,
Thanks for looking at this, I'm currently far behind on many things and
not very responsive:-(
Here is the creation of llvmjit.so:
g++ ... -o llvmjit.so ... -L/home/fabien/clgtk/lib ... -lLLVMOrcJIT ...
That'd be from llvm-config --ldflags or similar, from this binary:
checking for llvm-config... (cached) /home/fabien/clgtk/bin/llvm-config
So what does ls -slap /home/fabien/clgtk/lib show?
Plenty files and links, eg:
0 lrwxrwxrwx 1 fabien fabien 21 janv. 23 09:40 /home/fabien/clgtk/lib/libLLVMMCJIT.so -> libLLVMMCJIT.so.12git
2140 -rw-r--r-- 1 fabien fabien 2190824 janv. 23 09:28 /home/fabien/clgtk/lib/libLLVMMCJIT.so.12git
0 lrwxrwxrwx 1 fabien fabien 22 janv. 23 09:40 /home/fabien/clgtk/lib/libLLVMOrcJIT.so -> libLLVMOrcJIT.so.12git
40224 -rw-r--r-- 1 fabien fabien 41187208 janv. 23 09:34 /home/fabien/clgtk/lib/libLLVMOrcJIT.so.12git
Hmmm, clang recompilation has been failing for some weeks. Argh, this is
due to the recent "master" to "main" branch renaming.
What does ldd
/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so
No such file of directory:-) because the directory is cleaned up.
show? What do you have in seawap's LD_LIBRARY_PATH, and is the
problem that you need to add /home/fabien/clgtk/lib to it
Argh. Would it be so stupid? :-( I thought the configuration stuff would
manage the link path automatically, which may be quite naï¿œve, indeed.
PS Could you try blowing away the accache directory so we can rule out
bad cached configure stuff?
Hmmm. I've tried that before. I can do it again.
I've added an explicit LD_LIBRARY_PATH, which will be triggered at some
point later.
--
Fabien Coelho - CRI, MINES ParisTech
Hi,
On 2021-02-15 10:05:32 +0100, Fabien COELHO wrote:
show? What do you have in seawap's LD_LIBRARY_PATH, and is the
problem that you need to add /home/fabien/clgtk/lib to itArgh. Would it be so stupid? :-( I thought the configuration stuff would
manage the link path automatically, which may be quite na�ve, indeed.
The only way to do that is to add an rpath annotation, and e.g. distros
don't like that - there's some security implications.
I've added an explicit LD_LIBRARY_PATH, which will be triggered at some
point later.
You can also do something like LDFLAGS="$LDFLAGS -Wl,-rpath,$(llvm-config --libdir)"
or such.
Greetings,
Andres Freund
I've added an explicit LD_LIBRARY_PATH, which will be triggered at some
point later.
This seems to have fixed the issue.
I'm sorry for the noise and quite baffled anyway, because according to my
change logs it does not seem that I modified anything from my side about
the dynamic library path when compiling with clang. At least I do not see
a trace of that.
You can also do something like LDFLAGS="$LDFLAGS -Wl,-rpath,$(llvm-config --libdir)"
or such.
I've resorted to just hardcode LD_LIBRARY_PATH alongside PATH when
compiling with clang in my buildfarm script. Thanks for the tip anyway.
And thanks Thomas for pointing out the fix!
--
Fabien.
On 1/24/21 11:06 PM, Tom Lane wrote:
Noah Misch <noah@leadboat.com> writes:
On Mon, Jan 18, 2021 at 09:29:53PM +0100, Fabien COELHO wrote:
... Last two months were a
little overworked for me so I let slip quite a few things. If you want to
disable the animal as Tom suggests, do as you want.Perhaps he was suggesting that you (buildfarm owner) disable the cron job that
initiates new runs. That's what I do when one of my animals needs my
intervention.Indeed. I'm not sure there even is a provision to block an animal on the
buildfarm-server side. If there is, you'd have to request that it be
manually undone after you get around to fixing the animal. Frankly,
if I were the BF admin, I would be in about as much hurry to do that
as you've been to fix it.
It's actually very easy, but it's something I usually reserve for people
who are very unresponsive to emails. As noted, disabling the crontab
entry on the client side is a preferable solution.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com