Invalid control file checksum with AVX-512 during initdb on a clang19 -O0 build
Hello,
I noticed a strange issue that I can only reproduce with clang:
clang version 19.1.7 (RESF 19.1.7-2.module+el8.10.0+1965+112b558b) on
the devel branch (SHA: 137935bd1167a94b0bfea7239033f1ba1a1d95bb).
We are getting a control file checksum mismatch during initdb. I added
some prints in a small debug patch, and recorded the postgres process
using rr. I have uploaded the rr archive (made with rr pack, tar-ed up) [1]https://drive.google.com/file/d/15bGN_NlGsYx0lJCMGnCxDnV2avYkujgk/view?usp=sharing.
$ initdb -n -d -D /usr/local/pgsql/data &> initdb.out
2025-06-11 16:19:54.343 UTC [3070] LOG: WriteControlFile crc = 3457434907,
algo is avx = 1
...
2025-06-11 16:19:54.343 UTC [3070] LOG: ReadControlFile crc = 3457434907,
ControlFile->crc = 3457434907, algo is avx = 1
...
2025-06-11 16:19:54.346 UTC [3070] LOG: update_controlfile crc =
2065009488, algo is avx = 1
...
2025-06-11 16:20:13.914 UTC [3070] LOG: update_controlfile crc =
3406554082, algo is avx = 1
...
2025-06-11 16:20:13.920 UTC [3070] LOG: update_controlfile crc =
1234673735, algo is avx = 1
...
2025-06-11 16:20:13.923 UTC [3070] NOTICE: database system is shut down
...
2025-06-11 16:20:13.923 UTC [3070] DEBUG: proc_exit(-1): 0 callbacks to
make
ok
performing post-bootstrap initialization ... 2025-06-11 16:20:13.984 UTC
[3072]: LOG: ReadControlFile crc = 2925279607, ControlFile->crc = 1234673735, algo is avx = 1 2025-06-11 16:20:13.984 UTC [3072] FATAL: incorrect checksum in control file child process exited with exit code 1 initdb: data directory "/usr/local/pgsql/data" not removed at user's request
1234673735, algo is avx = 1
2025-06-11 16:20:13.984 UTC [3072]LOG: ReadControlFile crc = 2925279607, ControlFile->crc = 1234673735, algo is avx = 1 2025-06-11 16:20:13.984 UTC [3072] FATAL: incorrect checksum in control file child process exited with exit code 1 initdb: data directory "/usr/local/pgsql/data" not removed at user's request FATAL: incorrect checksum in control
file
child process exited with exit code 1
initdb: data directory "/usr/local/pgsql/data" not removed at user's request
Note that this only reproduces with clang-19 -O0 and NOT -O3. I haven't
tried with other versions of clang.
OTOH, gcc-14 is cool with both -O0 and -O3, with AVX-512 getting picked
for both cases, for CRC instructions.
Environment:
(1) Configure options:
./configure --prefix=/usr/local/pgsql --with-python --enable-depend
--without-icu --enable-debug CFLAGS='-O0 -fno-omit-frame-pointer' CC=clang
(2) Config log shows:
configure:18262: checking for vectorized CRC-32C
configure:18268: result: AVX-512 with runtime check
pgac_cv_avx512_pclmul_intrinsics=yes
pgac_cv_xsave_intrinsics=yes
(3) Confirmation that we have AVX-512 CRC getting selected at runtime:
(rr) f
#0 WriteControlFile () at xlog.c:4386
4386 ControlFile->pg_control_version = PG_CONTROL_VERSION;
(rr) p pg_comp_crc32c
$1 = (pg_crc32c (*)(pg_crc32c, const void *, size_t)) 0xc8b8b0
<pg_comp_crc32c_avx512>
(4) This is running in a VM with:
Rocky Linux release 8.10 (Green Obsidian)
16 vCPUs
Hypervisor: VMware ESXi, 8.0.3, 24022510
Model: PowerEdge R650
Processor Type: Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz
vCenter:
Version: 8.0.3
Build: 24322831
I attempted both vSAN and local storage. That didn't influence matters.
There is a known vSAN bug with invalid checksums + AVX-512, but that has
been fixed in an older version [2]https://knowledge.broadcom.com/external/article/367589/applications-using-avx512-instructions-i.html (and besides the issue reproes with
local storage too).
Please let me know if there is any other info I can provide.
Regards,
Deep (VMware)
[1]: https://drive.google.com/file/d/15bGN_NlGsYx0lJCMGnCxDnV2avYkujgk/view?usp=sharing
https://drive.google.com/file/d/15bGN_NlGsYx0lJCMGnCxDnV2avYkujgk/view?usp=sharing
[2]: https://knowledge.broadcom.com/external/article/367589/applications-using-avx512-instructions-i.html
https://knowledge.broadcom.com/external/article/367589/applications-using-avx512-instructions-i.html
(I've CC'd the committer of the feature.)
On Wed, Jun 11, 2025 at 11:10:31AM -0700, Soumyadeep Chakraborty wrote:
We are getting a control file checksum mismatch during initdb.
[...]
Note that this only reproduces with clang-19 -O0 and NOT -O3. I haven't
tried with other versions of clang.OTOH, gcc-14 is cool with both -O0 and -O3, with AVX-512 getting picked
for both cases, for CRC instructions.
I'm wondering if there's anything else this could be besides a compiler
bug. It's interesting that -O0 is the broken case. Have you tried looking
for differences in the assembly code?
--
nathan
On Thu, Jun 12, 2025 at 4:42 AM Nathan Bossart <nathandbossart@gmail.com> wrote:
(I've CC'd the committer of the feature.)
On Wed, Jun 11, 2025 at 11:10:31AM -0700, Soumyadeep Chakraborty wrote:
Note that this only reproduces with clang-19 -O0 and NOT -O3. I haven't
tried with other versions of clang.OTOH, gcc-14 is cool with both -O0 and -O3, with AVX-512 getting picked
for both cases, for CRC instructions.I'm wondering if there's anything else this could be besides a compiler
bug. It's interesting that -O0 is the broken case. Have you tried looking
for differences in the assembly code?
Hmm, clang at -O0 is the same in versions 18, 19, and 20. The change
from v17 is easy enough to workaround as in the attached, which
produces identical output as master at -O2. Does this fix it for you?
If that's not it, it may be more difficult, as the change at -O0 from
v16 to v17 is large, so it's hard to see what's going on.
--
John Naylor
Amazon Web Services
Attachments:
v1-crc-workaround-clang-18.patchtext/x-patch; charset=US-ASCII; name=v1-crc-workaround-clang-18.patchDownload+4-1
On Wed, Jun 11, 2025 at 11:57PM John Naylor <johncnaylorls@gmail.com> wrote:
On Thu, Jun 12, 2025 at 4:42AM Nathan Bossart <nathandbossart@gmail.com>
wrote:
(I've CC'd the committer of the feature.)
On Wed, Jun 11, 2025 at 11:10:31AM -0700, Soumyadeep Chakraborty wrote:
Note that this only reproduces with clang-19 -O0 and NOT -O3. I
haven't
tried with other versions of clang.
OTOH, gcc-14 is cool with both -O0 and -O3, with AVX-512 getting
picked
for both cases, for CRC instructions.
I'm wondering if there's anything else this could be besides a compiler
bug. It's interesting that -O0 is the broken case. Have you tried
looking
for differences in the assembly code?
Unfortunately, I haven't had a chance to dig into the assembly yet.
Hmm, clang at -O0 is the same in versions 18, 19, and 20. The change
from v17 is easy enough to workaround as in the attached, which
produces identical output as master at -O2. Does this fix it for you?
If that's not it, it may be more difficult, as the change at -O0 from
v16 to v17 is large, so it's hard to see what's going on.
It unfortunately doesn't. I still get the same checksum failure. I tried
-O0 clang-19, clang-18 and clang-17. I've uploaded the rr recordings
and initdb output with your patch applied and the debug logs I had.
Regards,
Deep
initdb-clang17.crc.workaround.out
<https://drive.google.com/file/d/1OptFhlWsOqNx4s2fv1tikOhsqjh27u-E/view?usp=drive_web>
initdb-clang18.crc.workaround.out
<https://drive.google.com/file/d/1yXYWmt7cHMXgONMkr9Yl29YU2HXpE8LT/view?usp=drive_web>
initdb-clang19.crc.workaround.out
<https://drive.google.com/file/d/1JmLKQMXOCwx8YFlJ_xpKdniopjwGpcji/view?usp=drive_web>
rr-clang17.crc.workaround.tar.gz
<https://drive.google.com/file/d/1Tnjva4l4w6S6iKn2hJBaTy1NRNHTTsAV/view?usp=drive_web>
rr-clang18.crc.workaround.tar.gz
<https://drive.google.com/file/d/1m7iKJHcAPPcKXet9gKkGEzTiFbJ2C9Ul/view?usp=drive_web>
rr-clang19.crc.workaround.tar.gz
<https://drive.google.com/file/d/1zXzUtvMJbsNMJNQezI5uC3Fb8Y3sUuMs/view?usp=drive_web>
On Fri, Jun 13, 2025 at 12:30 AM Soumyadeep Chakraborty <
soumyadeep2007@gmail.com> wrote:
On Wed, Jun 11, 2025 at 11:57PM John Naylor <johncnaylorls@gmail.com>
wrote:
Hmm, clang at -O0 is the same in versions 18, 19, and 20. The change
from v17 is easy enough to workaround as in the attached, which
produces identical output as master at -O2. Does this fix it for you?
If that's not it, it may be more difficult, as the change at -O0 from
v16 to v17 is large, so it's hard to see what's going on.It unfortunately doesn't. I still get the same checksum failure. I tried
-O0 clang-19, clang-18 and clang-17. I've uploaded the rr recordings
and initdb output with your patch applied and the debug logs I had.
Okay, in that case we'd better just kludge around it by disabling the
runtime check with clang at -O0 (attached). That will at least get these
kind of builds working while I get more info for a bug report.
--
John Naylor
Amazon Web Services
Attachments:
v2-crc-workaround-clang-18.patchapplication/x-patch; name=v2-crc-workaround-clang-18.patchDownload+3-1
John Naylor <johncnaylorls@gmail.com> writes:
Hi John,
Okay, in that case we'd better just kludge around it by disabling the
runtime check with clang at -O0 (attached). That will at least get
these kind of builds working while I get more info for a bug report.
I run into the same issue at [1]/messages/by-id/87o6uqd3iv.fsf@163.com and verified your patch works. Thanks
for the fix!
[1]: /messages/by-id/87o6uqd3iv.fsf@163.com
--
Best Regards
Andy Fan
On Sun, Jun 15, 2025 at 8:23 AM Andy Fan <zhihuifan1213@163.com> wrote:
John Naylor <johncnaylorls@gmail.com> writes:
Okay, in that case we'd better just kludge around it by disabling the
runtime check with clang at -O0 (attached). That will at least get
these kind of builds working while I get more info for a bug report.I run into the same issue at [1] and verified your patch works. Thanks
for the fix!
Pushed, thanks for testing, and thanks Soumyadeep for the report!
--
John Naylor
Amazon Web Services