jbergler Posted October 6, 2020 Posted October 6, 2020 Hi Folks, Trying to wrap my head around this still, so bear with my if I've crossed my wires somewhere, first some background. With a 5.8 linux-image-current-rockchip64 kernel, zfs 0.8.4 and 2.0.0-rc won't compile because of the following error. ERROR: modpost: "__stack_chk_guard" [/root/zfs/module/zfs/zfs.ko] undefined! This has been mentioned in a few places: https://github.com/openzfs/zfs/issues/10985 https://forum.armbian.com/topic/15065-downgrade-linux-image-current-rockchip64/ Digging into this I managed to build the module on focal with the same kernel and install that on buster. I'm of the belief that compiler should not be referencing "__stack_chk_guard" at all, since "CONFIG_STACKPROTECTOR_PER_TASK=y" is configured for this kernel and the symbol isn't listed in "/proc/kallsyms" My suspicion (and this is where my knowledge limits me) is that something changed in GCC between the version buster ships with (8.3) and the version the kernel was built with (9.2) that causes it to not do the correct things for per task stack canaries. If anyone has thoughts/ideas I'd love the input -- Jonas
jbergler Posted October 7, 2020 Author Posted October 7, 2020 The suggestion Igor made in that thread solves the 0.8.3 problems on 5.8 because certain kernel methods were flagged as GPL and the module won't compile. As others in that thread pointed out, using the newer 0.8.4 package works on focal but not buster. 0.8.4 compiles correctly against the rockchip64 kernel on focal (gcc9) and bullseye (gcc10), but not with buster (gcc8). 2.0.0-rc follows the same pattern. the rockchip64 kernel is built using the armbian toolchain, which is focal based and thus using gcc 9. I haven't been able to come up with any other theory than how the compiler is detecting if it should use CONFIG_STACKPROTECTOR or CONFIG_STACKPROTECTOR_PER_TASK. It's only a theory, and I don't know how to prove or disprove it, short of building the same 5.8.13 kernel with gcc 8 and trying to build the module against it (which is probably my next step)
Brocklobsta Posted October 7, 2020 Posted October 7, 2020 I am running Armbian bionic and installed gcc-9 and gcc-10 from "ppa:ubuntu-toolchain-r/test". I tried both compilers with dkms and am still getting ERROR: modpost: "__stack_chk_guard" [/root/zfs/module/zfs/zfs.ko] undefined! I set the compiler version by exporting `CC`, but I don't see any mention of the compiler version in the DKMS build output. The make.log does show different compiler warnings, so I'm pretty sure I am successfully setting the compiler version.
jbergler Posted October 8, 2020 Author Posted October 8, 2020 @Brocklobsta sounds like you're setting GCC correctly, and the issue thats left is probably that the zfs-dkms module is too old (try finding 0.8.4+ per here) For the issue on buster, I think I've nailed the problem down. STACKPROTECTOR_PER_TASK is defined as such config CC_HAVE_STACKPROTECTOR_SYSREG def_bool $(cc-option,-mstack-protector-guard=sysreg -mstack-protector-guard-reg=sp_el0 -mstack-protector-guard-offset=0) config STACKPROTECTOR_PER_TASK def_bool y depends on STACKPROTECTOR && CC_HAVE_STACKPROTECTOR_SYSREG GCC in the focal build environment supports the options required for CC_HAVE_STACKPROTECTOR_SYSREG $ gcc --version gcc (Ubuntu 9.3.0-10ubuntu2) 9.3.0 $ gcc -mstack-protector-guard=sysreg -mstack-protector-guard-reg=sp_el0 -mstack-protector-guard-offset=0 gcc: fatal error: no input files compilation terminated. Whereas the gcc version buster doesn't support these. $ gcc --version gcc (Debian 8.3.0-6) 8.3.0 $ gcc -mstack-protector-guard=sysreg -mstack-protector-guard-reg=sp_el0 -mstack-protector-guard-offset=0 gcc: error: unrecognized command line option '-mstack-protector-guard=sysreg'; did you mean '-fstack-protector-strong'? gcc: error: unrecognized command line option '-mstack-protector-guard-reg=sp_el0'; did you mean '-fstack-protector-all'? gcc: error: unrecognized command line option '-mstack-protector-guard-offset=0'; did you mean '-fstack-protector-strong'? gcc: fatal error: no input files compilation terminated. Since the kernel is built for both focal + buster, should features like this be disabled or are there some other workarounds?
Brocklobsta Posted October 8, 2020 Posted October 8, 2020 @jbergler I am building the latest ZFS release 0.8.5 with GCC-9 and GCC-10 on Armbian bionic and the issue still occurs. Have you tried a build with GCC-9/10?
turkerali Posted October 9, 2020 Posted October 9, 2020 I am using Armbian Bionic. I have upgraded the GCC from ppa:ubuntu-toolchain-r/test repo to gcc-10, but I was having the same error message until today: ERROR: modpost: "__stack_chk_guard" [/root/zfs/zfs-0.8.5/module/zfs/zfs.ko] undefined! ERROR: modpost: "__stack_chk_guard" [/root/zfs/zfs-0.8.5/module/zcommon/zcommon.ko] undefined! ... Today Armbian upgraded the kernel&header package (#20.08.8) and zfs compiles without any errors. I checked the changelog from https://docs.armbian.com/Release_Changelog/ but could not see any related changes. Most probably Armbian is now building the kernel with a newer GCC version, because I did not change anything since the build failed.
Brocklobsta Posted October 9, 2020 Posted October 9, 2020 9 hours ago, turkerali said: I am using Armbian Bionic. I have upgraded the GCC from ppa:ubuntu-toolchain-r/test repo to gcc-10, but I was having the same error message until today: ERROR: modpost: "__stack_chk_guard" [/root/zfs/zfs-0.8.5/module/zfs/zfs.ko] undefined! ERROR: modpost: "__stack_chk_guard" [/root/zfs/zfs-0.8.5/module/zcommon/zcommon.ko] undefined! ... Today Armbian upgraded the kernel&header package (#20.08.8) and zfs compiles without any errors. I checked the changelog from https://docs.armbian.com/Release_Changelog/ but could not see any related changes. Most probably Armbian is now building the kernel with a newer GCC version, because I did not change anything since the build failed. @turkerali Which kernel package are you using? linux-image-current-rockchip64? I am not able to compile with the "current" kernel, but the "legacy" kernel has always worked.
turkerali Posted October 10, 2020 Posted October 10, 2020 14 hours ago, Brocklobsta said: @turkerali Which kernel package are you using? linux-image-current-rockchip64? I am not able to compile with the "current" kernel, but the "legacy" kernel has always worked. Yes, I am using the linux-image-current-rockchip64 (5.8.x) on nanopc-t4. There is one more possibility that might have resolved the issue for me. When you install the "linux-headers" package, apt runs "make scripts" command at the end of installation, which builds some helper utilities inside the linux-headers package. I believe these helper utilities are responsible for module compilation and the "modpost" errors. As a workaround, before you install the linux-headers package, make sure to upgrade the gcc version to gcc-9 or gcc-10 from ppa:ubuntu-toolchain-r/test repo. Then install the linux-headers package. If you do this, the binary utilites in the scripts folder will be built with the upgraded gcc, and will be able to build & link the zfs modules. I may be wrong, but it's worth trying.
Brocklobsta Posted October 12, 2020 Posted October 12, 2020 On 10/10/2020 at 12:47 AM, turkerali said: Yes, I am using the linux-image-current-rockchip64 (5.8.x) on nanopc-t4. There is one more possibility that might have resolved the issue for me. When you install the "linux-headers" package, apt runs "make scripts" command at the end of installation, which builds some helper utilities inside the linux-headers package. I believe these helper utilities are responsible for module compilation and the "modpost" errors. As a workaround, before you install the linux-headers package, make sure to upgrade the gcc version to gcc-9 or gcc-10 from ppa:ubuntu-toolchain-r/test repo. Then install the linux-headers package. If you do this, the binary utilites in the scripts folder will be built with the upgraded gcc, and will be able to build & link the zfs modules. I may be wrong, but it's worth trying. Thanks @turkerali! Recompiling the kernel headers with GCC-10 fixed the issue.
antsu Posted October 16, 2020 Posted October 16, 2020 @jberglerThanks again for the ZFS module you shared on the Helios64 support thread. It's been running for days without any problems. If it's not too much trouble, would you mind sharing the process you used to build it?
robbixc Posted October 16, 2020 Posted October 16, 2020 1 hour ago, antsu said: @jberglerThanks again for the ZFS module you shared on the Helios64 support thread. It's been running for days without any problems. If it's not too much trouble, would you mind sharing the process you used to build it? Yes, that would be much appreciated. And thanks again @jbergler for your work
jbergler Posted October 17, 2020 Author Posted October 17, 2020 (edited) These won't be exact instructions, since I decided to switch to focal (mostly for other reasons). mkdir zfs-scratch cd zfs-scratch apt-get download linux-headers-current-rockchip64 git clone -b zfs-0.8.5 https://github.com/openzfs/zfs.git docker run --rm -it -v $(pwd):/scratch ubuntu:focal # inside the container cd /scratch apt update apt install build-essential autoconf automake bison flex libtool gawk alien fakeroot dkms libblkid-dev uuid-dev libudev-dev libssl-dev zlib1g-dev libaio-dev libattr1-dev libelf-dev python3 python3-dev python3-setuptools python3-cffi libffi-dev dpkg -i linux-headers-current-*.deb cd zfs sh autogen.sh ./configure make -s -j$(nproc) deb At that point you can exit the container (it'l vanish because of the --rm) and inside zfs-scratch/zfs you should have a bunch of Debian packages you can install Edited November 7, 2020 by jbergler added bison�+flex 2
antsu Posted October 18, 2020 Posted October 18, 2020 @jberglerThank you very much for that. The steps just needed a small tweak, but it works perfectly. Running on kernel 5.8.16 right now! The small tweak is just to also install flex and bison before installing the kernel headers package (or run apt-get install -f if already installed).
grek Posted November 25, 2020 Posted November 25, 2020 Hey I was able to build ZFS zfs-2.0.0-rc6 from sources using @jbergler instructions on my helios64 . debian (inside docker ubuntu container) root@helios64:~# uname -a Linux helios64 5.9.10-rockchip64 #trunk.39 SMP PREEMPT Sun Nov 22 10:36:59 CET 2020 aarch64 GNU/Linux root@helios64:~# but now im stuck with old glibc version. root@helios64:~/zfs_deb# zfs zfs: /lib/aarch64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /lib/aarch64-linux-gnu/libzfs.so.4) root@helios64:~/zfs_deb# zfs --version zfs: /lib/aarch64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /lib/aarch64-linux-gnu/libzfs.so.4) root@helios64:~/zfs_deb# root@helios64:~/zfs_deb# ldd --version ldd (Debian GLIBC 2.28-10) 2.28 Copyright (C) 2018 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Written by Roland McGrath and Ulrich Drepper. I also installed kmod-zfs instead of zfs-dkms_2.0.0-0_arm64.deb package. When I tried to install zfs-dkms i got errors: Setting up libnvpair1 (2.0.0-0) ... Setting up libuutil1 (2.0.0-0) ... Setting up libzfs2-devel (2.0.0-0) ... Setting up libzfs2 (2.0.0-0) ... Setting up libzpool2 (2.0.0-0) ... Setting up python3-pyzfs (2.0.0-0) ... Setting up zfs-dkms (2.0.0-0) ... WARNING: /usr/lib/dkms/common.postinst does not exist. ERROR: DKMS version is too old and zfs was not built with legacy DKMS support. You must either rebuild zfs with legacy postinst support or upgrade DKMS to a more current version. dpkg: error processing package zfs-dkms (--install): installed zfs-dkms package post-installation script subprocess returned error exit status 1 Setting up zfs-dracut (2.0.0-0) ... Setting up zfs-initramfs (2.0.0-0) ... Setting up zfs-test (2.0.0-0) ... Setting up zfs (2.0.0-0) ... Processing triggers for libc-bin (2.28-10) ... Processing triggers for systemd (241-7~deb10u4) ... Processing triggers for man-db (2.8.5-2) ... Errors were encountered while processing: zfs-dkms
ShadowDance Posted November 25, 2020 Posted November 25, 2020 (edited) @grek it's a bit of a dilemma. You must use compatible libs to build the zfs tools, in your case you must build them on Debian Buster, but the module build will fail due to old gcc. You can work around this by using the kernel modules (kmod) from a build using newer gcc (by building on e.g. Ubuntu Focal, Debian Bullseye/Sid, etc) and building the tools separately on Buster. To achieve this we must force the build on Buster to either skip building the kmod or to trick it into building it "successfully". After a bunch of head-scratching, I found the latter to be easier. Here's a set of steps that should work for building the tools (sans functioning kmod) (based largely on the steps posted earlier by jbergler): mkdir /tmp/zfs-build; cd $_ apt-get download linux-headers-current-rockchip64 docker pull debian:buster-slim docker run -it --rm -v $(pwd):/build debian:buster-slim # Inside docker. cd /build apt-get update apt-get install --yes alien autoconf automake bison build-essential dkms fakeroot flex gawk libaio-dev libattr1-dev libblkid-dev libelf-dev libffi-dev libssl-dev libtool libudev-dev python3 python3-cffi python3-dev python3-setuptools uuid-dev zlib1g-dev dpkg -i linux-headers-*.deb # Disable all STACKPROTECT options incompatible with GCC, this means the # built kernel module (kmod) will be non-functional. That's OK since # we're only interested in building the tools. sed -i -e 's/\(.*STACKPROTECT.*=\)y/\1n/' /usr/src/linux-headers-$(uname -r)/.config # Ignore the error(s) from the following: (cd /usr/src/linux-headers-$(uname -r); make prepare) apt-get install --no-install-recommends --yes git git clone https://github.com/openzfs/zfs.git cd zfs git clean -xdf git checkout zfs-2.0.0-rc6 sh autogen.sh ./configure --with-linux=/usr/src/linux-headers-$(uname -r) make -s -j$(nproc) make deb mkdir ../zfs-2.0.0-rc6-buster mv *.deb ../zfs-2.0.0-rc6-buster Then go ahead and install: # Cleanup packages (if installed). modprobe -r zfs zunicode zzstd zlua zcommon znvpair zavl icp spl apt remove --yes zfsutils-linux zfs-zed zfs-initramfs apt autoremove --yes # Install new packages. dpkg -i /path/to/working/kmod-zfs-$(uname -r)*.deb dpkg -i /tmp/zfs-build/zfs-2.0.0-rc6-buster/{libnvpair1,libuutil1,libzfs2,libzpool2,python3-pyzfs,zfs}_*.deb And that should do it, until the next kernel update. PS. I only tested zfs-2.0.0-rc5 when setting up my Helios64 the first time, currently using zfs-0.8.5 so can't speak for the reliability. Edited November 25, 2020 by ShadowDance Fix the ./configure step to point at the correct kernel source location (fixes make deb step)
Solution Salamandar Posted December 1, 2020 Solution Posted December 1, 2020 (edited) Hi, I'm still unable to build the module correctly. The ZFS kernel module is build for kernel '4.4.213-rockchip64' (the headers version name) but the kernel is '4.4.213-rk3399'. spl: version magic '4.4.213-rockchip64 SMP mod_unload aarch64' should be '4.4.213-rk3399 SMP mod_unload aarch64' I'm not sure this can be fixed without editing kernel headers, either : /usr/src/linux-headers-4.4.213-rockchip64/include/config/kernel.release /usr/src/linux-headers-4.4.213-rockchip64/include/generated/utsrelease.h @ShadowDance Are you using the "current" 5.9 kernel ? Maybe this has been fixed for this kernel ? BTW as the procedure is a bit tiring to do by hand, I wrote some scripts to automate this (build module on Ubuntu, utils on Buster) : https://github.com/Salamandar/armbian_zfs_compiler/ MOD EDIT: Above is third party script, as always, make sure you understand what is going on before running it on your system. Having gotten that disclaimer out of the way, thanks for your contribution, @Salamandar! I took the liberty of marking this as solution. I also made note of the link to refer others to, later. Cheers, TRS-80 Edited December 1, 2020 by TRS-80 Moderator add disclaimer 2
ShadowDance Posted December 1, 2020 Posted December 1, 2020 @Salamandar Yes, I've been using 5.8/5.9 since the get-go. You probably need the `linux-headers-legacy-rk3399` package (modify the apt-get download stage). Good job automating the process .
Salamandar Posted December 1, 2020 Posted December 1, 2020 (edited) @ShadowDance Crap, I didn't even see those headers were available ! I already implemented the modification of the headers version ! Let's roll back. Thanks, feel free to open issues if needed. Edited December 1, 2020 by Salamandar
Igor Posted December 1, 2020 Posted December 1, 2020 11 hours ago, Salamandar said: Thanks, feel free to open issues if needed. Any idea how to compile ZFS in the tree? I was playing a bit with that idea. If we can do it, we could perhaps ship ZFS with kernel? (or rather not? ) This way we cover 3rd party drivers: https://github.com/armbian/build/blob/master/lib/compilation-prepare.sh
manman Posted December 1, 2020 Posted December 1, 2020 @ShadowDance I've got an error when I was trying to build the 2.0.0 release using your instructions, getting the following error on "make deb" : kmodtool error: Don't know how to handle 5.9.11 -- /lib/modules/5.9.11/build/Makefile not found I've tried to change the kernel version to "5.9.11-rockchip64" directly on the Makefile but the build it fails anyways. the right PATH should be /lib/modules/5.9.11-rockchip64/build/Makefile
ShadowDance Posted December 2, 2020 Posted December 2, 2020 @manman hmm, the configure step should’ve taken care of that. You could try to specify the correct path manually via: Quote ./configure --with-linux=/usr/src/linux-headers-5.9.11-rockchip64 If that still doesn’t help, see if you have any other linux-headers installed and remove them all.
Salamandar Posted December 2, 2020 Posted December 2, 2020 21 hours ago, Igor said: Any idea how to compile ZFS in the tree? I was playing a bit with that idea. If we can do it, we could perhaps ship ZFS with kernel? (or rather not? ) This way we cover 3rd party drivers: https://github.com/armbian/build/blob/master/lib/compilation-prepare.sh What do you mean by "in the tree" ? Do you mean in /usr/src/zfs where the zfs-dkms module installs the sources ? Right now the build is done inside the source directory of ZFS, and I'm looking into building it out-of-source (I'm gonna read the dkms scripts, I think they build out-of-source).
Igor Posted December 2, 2020 Posted December 2, 2020 37 minutes ago, Salamandar said: What do you mean by "in the tree" ? That we compile it together with kernel. Like we do for other drivers - wireguard (for older kernels) and selection of wireless drivers.
Recommended Posts