Jump to content

unable to build zfs module on buster rockchip64

jbergler

By jbergler
in Beginners

 Share
Go to solution Solved by Salamandar,

Recommended Posts

jbergler

jbergler

Posted
Posted

Hi Folks,

 

Trying to wrap my head around this still, so bear with my if I've crossed my wires somewhere, first some background.

 

With a 5.8 linux-image-current-rockchip64 kernel, zfs 0.8.4 and 2.0.0-rc won't compile because of the following error. 

ERROR: modpost: "__stack_chk_guard" [/root/zfs/module/zfs/zfs.ko] undefined!

 

This has been mentioned in a few places:

 

https://github.com/openzfs/zfs/issues/10985

https://forum.armbian.com/topic/15065-downgrade-linux-image-current-rockchip64/

 

Digging into this I managed to build the module on focal with the same kernel and install that on buster. 

I'm of the belief that compiler should not be referencing "__stack_chk_guard" at all, since "CONFIG_STACKPROTECTOR_PER_TASK=y" is configured for this kernel and the symbol isn't listed in "/proc/kallsyms"

My suspicion (and this is where my knowledge limits me) is that something changed in GCC between the version buster ships with (8.3) and the version the kernel was built with (9.2) that causes it to not do the correct things for per task stack canaries.

 

If anyone has thoughts/ideas I'd love the input

 

-- Jonas

Link to comment
Share on other sites
jbergler

jbergler

Posted
  • Author
Posted

The suggestion Igor made in that thread solves the 0.8.3 problems on 5.8 because certain kernel methods were flagged as GPL and the module won't compile.

As others in that thread pointed out, using the newer 0.8.4 package works on focal but not buster.

 

0.8.4 compiles correctly against the rockchip64 kernel on focal (gcc9) and bullseye (gcc10), but not with buster (gcc8).

2.0.0-rc follows the same pattern.

 

the rockchip64 kernel is built using the armbian toolchain, which is focal based and thus using gcc 9.

 

I haven't been able to come up with any other theory than how the compiler is detecting if it should use CONFIG_STACKPROTECTOR or CONFIG_STACKPROTECTOR_PER_TASK.

 

It's only a theory, and I don't know how to prove or disprove it, short of building the same 5.8.13 kernel with gcc 8 and trying to build the module against it (which is probably my next step)

Link to comment
Share on other sites
Brocklobsta

Brocklobsta

Posted
Posted

I am running Armbian bionic and installed gcc-9 and gcc-10 from "ppa:ubuntu-toolchain-r/test". I tried both compilers with dkms and am still getting 

ERROR: modpost: "__stack_chk_guard" [/root/zfs/module/zfs/zfs.ko] undefined!

 

I set the compiler version by exporting `CC`, but I don't see any mention of the compiler version in the DKMS build output. The make.log does show different compiler warnings, so I'm pretty sure I am successfully setting the compiler version.

Link to comment
Share on other sites
jbergler

jbergler

Posted
  • Author
Posted

@Brocklobsta sounds like you're setting GCC correctly, and the issue thats left is probably that the zfs-dkms module is too old (try finding 0.8.4+ per here)

 

For the issue on buster, I think I've nailed the problem down.

 

STACKPROTECTOR_PER_TASK is defined as such 

config CC_HAVE_STACKPROTECTOR_SYSREG
   def_bool $(cc-option,-mstack-protector-guard=sysreg -mstack-protector-guard-reg=sp_el0 -mstack-protector-guard-offset=0)

config STACKPROTECTOR_PER_TASK
   def_bool y
   depends on STACKPROTECTOR && CC_HAVE_STACKPROTECTOR_SYSREG

 

GCC in the focal build environment supports the options required for CC_HAVE_STACKPROTECTOR_SYSREG 

$ gcc --version
gcc (Ubuntu 9.3.0-10ubuntu2) 9.3.0

$ gcc -mstack-protector-guard=sysreg -mstack-protector-guard-reg=sp_el0 -mstack-protector-guard-offset=0
gcc: fatal error: no input files
compilation terminated.

 

Whereas the gcc version buster doesn't support these. 

$ gcc --version
gcc (Debian 8.3.0-6) 8.3.0

$ gcc -mstack-protector-guard=sysreg -mstack-protector-guard-reg=sp_el0 -mstack-protector-guard-offset=0
gcc: error: unrecognized command line option '-mstack-protector-guard=sysreg'; did you mean '-fstack-protector-strong'?
gcc: error: unrecognized command line option '-mstack-protector-guard-reg=sp_el0'; did you mean '-fstack-protector-all'?
gcc: error: unrecognized command line option '-mstack-protector-guard-offset=0'; did you mean '-fstack-protector-strong'?
gcc: fatal error: no input files
compilation terminated.

 

Since the kernel is built for both focal + buster, should features like this be disabled or are there some other workarounds?

Link to comment
Share on other sites
turkerali

turkerali

Posted
Posted

I am using Armbian Bionic. I have upgraded the GCC from ppa:ubuntu-toolchain-r/test repo to gcc-10, but I was having the same error message until today:

 

ERROR: modpost: "__stack_chk_guard" [/root/zfs/zfs-0.8.5/module/zfs/zfs.ko] undefined!
ERROR: modpost: "__stack_chk_guard" [/root/zfs/zfs-0.8.5/module/zcommon/zcommon.ko] undefined!
...

 

Today Armbian upgraded the kernel&header package (#20.08.8) and zfs compiles without any errors. I checked the changelog from https://docs.armbian.com/Release_Changelog/

but could not see any related changes. Most probably Armbian is now building the kernel with a newer GCC version, because I did not change anything since the build failed.

Link to comment
Share on other sites
Brocklobsta

Brocklobsta

Posted
Posted
9 hours ago, turkerali said:

I am using Armbian Bionic. I have upgraded the GCC from ppa:ubuntu-toolchain-r/test repo to gcc-10, but I was having the same error message until today:

 

ERROR: modpost: "__stack_chk_guard" [/root/zfs/zfs-0.8.5/module/zfs/zfs.ko] undefined!
ERROR: modpost: "__stack_chk_guard" [/root/zfs/zfs-0.8.5/module/zcommon/zcommon.ko] undefined!
...

 

Today Armbian upgraded the kernel&header package (#20.08.8) and zfs compiles without any errors. I checked the changelog from https://docs.armbian.com/Release_Changelog/

but could not see any related changes. Most probably Armbian is now building the kernel with a newer GCC version, because I did not change anything since the build failed.

 

@turkerali

Which kernel package are you using? linux-image-current-rockchip64? I am not able to compile with the "current" kernel, but the "legacy" kernel has always worked.

Link to comment
Share on other sites
turkerali

turkerali

Posted
Posted
14 hours ago, Brocklobsta said:

 

@turkerali

Which kernel package are you using? linux-image-current-rockchip64? I am not able to compile with the "current" kernel, but the "legacy" kernel has always worked.

 

Yes, I am using the linux-image-current-rockchip64 (5.8.x) on nanopc-t4.

 

There is one more possibility that might have resolved the issue for me. When you install the "linux-headers" package, apt runs "make scripts" command at the end of installation,

which builds some helper utilities inside the linux-headers package. I believe these helper utilities are responsible for module compilation and the "modpost" errors.

 

As a workaround, before you install the linux-headers package, make sure to upgrade the gcc version to gcc-9 or gcc-10 from ppa:ubuntu-toolchain-r/test repo.

Then install the linux-headers package. If you do this, the binary utilites in the scripts folder will be built with the upgraded gcc, and will be able to build & link the zfs modules.

 

I may be wrong, but it's worth trying.

 

Link to comment
Share on other sites
Brocklobsta

Brocklobsta

Posted
Posted
On 10/10/2020 at 12:47 AM, turkerali said:

 

Yes, I am using the linux-image-current-rockchip64 (5.8.x) on nanopc-t4.

 

There is one more possibility that might have resolved the issue for me. When you install the "linux-headers" package, apt runs "make scripts" command at the end of installation,

which builds some helper utilities inside the linux-headers package. I believe these helper utilities are responsible for module compilation and the "modpost" errors.

 

As a workaround, before you install the linux-headers package, make sure to upgrade the gcc version to gcc-9 or gcc-10 from ppa:ubuntu-toolchain-r/test repo.

Then install the linux-headers package. If you do this, the binary utilites in the scripts folder will be built with the upgraded gcc, and will be able to build & link the zfs modules.

 

I may be wrong, but it's worth trying.

 

Thanks @turkerali! Recompiling the kernel headers with GCC-10 fixed the issue.

Link to comment
Share on other sites
robbixc

robbixc

Posted
Posted
1 hour ago, antsu said:

@jberglerThanks again for the ZFS module you shared on the Helios64 support thread. It's been running for days without any problems.

If it's not too much trouble, would you mind sharing the process you used to build it?

 

Yes, that would be much appreciated.

And thanks again @jbergler for your work

Link to comment
Share on other sites
jbergler

jbergler

Posted
  • Author
Posted (edited)

These won't be exact instructions, since I decided to switch to focal (mostly for other reasons).

  

mkdir zfs-scratch
cd zfs-scratch

apt-get download linux-headers-current-rockchip64
git clone -b zfs-0.8.5 https://github.com/openzfs/zfs.git 

docker run --rm -it -v $(pwd):/scratch ubuntu:focal

# inside the container
cd /scratch
apt update
apt install build-essential autoconf automake bison flex libtool gawk alien fakeroot dkms libblkid-dev uuid-dev libudev-dev libssl-dev zlib1g-dev libaio-dev libattr1-dev libelf-dev python3 python3-dev python3-setuptools python3-cffi libffi-dev
dpkg -i linux-headers-current-*.deb

cd zfs
sh autogen.sh
./configure
make -s -j$(nproc) deb

 

At that point you can exit the container (it'l vanish because of the --rm) and inside zfs-scratch/zfs you should have a bunch of Debian packages you can install

 

Edited by jbergler
added bison�+flex
2
Link to comment
Share on other sites
antsu

antsu

Posted
Posted

@jberglerThank you very much for that. The steps just needed a small tweak, but it works perfectly. Running on kernel 5.8.16 right now!

The small tweak is just to also install flex and bison before installing the kernel headers package (or run apt-get install -f if already installed).

Link to comment
Share on other sites
grek

grek

Posted
Posted

Hey I was able to build ZFS zfs-2.0.0-rc6 from sources using @jbergler instructions on my helios64 . debian (inside docker ubuntu container) 

 

root@helios64:~# uname -a

Linux helios64 5.9.10-rockchip64 #trunk.39 SMP PREEMPT Sun Nov 22 10:36:59 CET 2020 aarch64 GNU/Linux

root@helios64:~#

 

but now im stuck with old glibc version. 

root@helios64:~/zfs_deb# zfs
zfs: /lib/aarch64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /lib/aarch64-linux-gnu/libzfs.so.4)
root@helios64:~/zfs_deb# zfs --version
zfs: /lib/aarch64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /lib/aarch64-linux-gnu/libzfs.so.4)
root@helios64:~/zfs_deb#

root@helios64:~/zfs_deb# ldd --version
ldd (Debian GLIBC 2.28-10) 2.28
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

 

I also installed kmod-zfs instead of zfs-dkms_2.0.0-0_arm64.deb package. 

When I tried to install zfs-dkms i got errors: 

  

Setting up libnvpair1 (2.0.0-0) ...
Setting up libuutil1 (2.0.0-0) ...
Setting up libzfs2-devel (2.0.0-0) ...
Setting up libzfs2 (2.0.0-0) ...
Setting up libzpool2 (2.0.0-0) ...
Setting up python3-pyzfs (2.0.0-0) ...
Setting up zfs-dkms (2.0.0-0) ...
WARNING: /usr/lib/dkms/common.postinst does not exist.
ERROR: DKMS version is too old and zfs was not
built with legacy DKMS support.
You must either rebuild zfs with legacy postinst
support or upgrade DKMS to a more current version.
dpkg: error processing package zfs-dkms (--install):
installed zfs-dkms package post-installation script subprocess returned error exit status 1
Setting up zfs-dracut (2.0.0-0) ...
Setting up zfs-initramfs (2.0.0-0) ...
Setting up zfs-test (2.0.0-0) ...
Setting up zfs (2.0.0-0) ...
Processing triggers for libc-bin (2.28-10) ...
Processing triggers for systemd (241-7~deb10u4) ...
Processing triggers for man-db (2.8.5-2) ...
Errors were encountered while processing:
zfs-dkms

 

Link to comment
Share on other sites
ShadowDance

ShadowDance

Posted
Posted (edited)

@grek it's a bit of a dilemma. You must use compatible libs to build the zfs tools, in your case you must build them on Debian Buster, but the module build will fail due to old gcc. You can work around this by using the kernel modules (kmod) from a build using newer gcc (by building on e.g. Ubuntu Focal, Debian Bullseye/Sid, etc) and building the tools separately on Buster. To achieve this we must force the build on Buster to either skip building the kmod or to trick it into building it "successfully". After a bunch of head-scratching, I found the latter to be easier.

 

Here's a set of steps that should work for building the tools (sans functioning kmod) (based largely on the steps posted earlier by jbergler): 

mkdir /tmp/zfs-build; cd $_
apt-get download linux-headers-current-rockchip64
docker pull debian:buster-slim
docker run -it --rm -v $(pwd):/build debian:buster-slim

# Inside docker.
cd /build

apt-get update
apt-get install --yes alien autoconf automake bison build-essential dkms fakeroot flex gawk libaio-dev libattr1-dev libblkid-dev libelf-dev libffi-dev libssl-dev libtool libudev-dev python3 python3-cffi python3-dev python3-setuptools uuid-dev zlib1g-dev

dpkg -i linux-headers-*.deb
# Disable all STACKPROTECT options incompatible with GCC, this means the
# built kernel module (kmod) will be non-functional. That's OK since 
# we're only interested in building the tools.
sed -i -e 's/\(.*STACKPROTECT.*=\)y/\1n/' /usr/src/linux-headers-$(uname -r)/.config
# Ignore the error(s) from the following:
(cd /usr/src/linux-headers-$(uname -r); make prepare)

apt-get install --no-install-recommends --yes git
git clone https://github.com/openzfs/zfs.git

cd zfs
git clean -xdf
git checkout zfs-2.0.0-rc6
sh autogen.sh
./configure --with-linux=/usr/src/linux-headers-$(uname -r)
make -s -j$(nproc)
make deb

mkdir ../zfs-2.0.0-rc6-buster
mv *.deb ../zfs-2.0.0-rc6-buster

Then go ahead and install: 

# Cleanup packages (if installed).
modprobe -r zfs zunicode zzstd zlua zcommon znvpair zavl icp spl
apt remove --yes zfsutils-linux zfs-zed zfs-initramfs
apt autoremove --yes

# Install new packages.
dpkg -i /path/to/working/kmod-zfs-$(uname -r)*.deb
dpkg -i /tmp/zfs-build/zfs-2.0.0-rc6-buster/{libnvpair1,libuutil1,libzfs2,libzpool2,python3-pyzfs,zfs}_*.deb

And that should do it, until the next kernel update.

 

PS. I only tested zfs-2.0.0-rc5 when setting up my Helios64 the first time, currently using zfs-0.8.5 so can't speak for the reliability.

Edited by ShadowDance
Fix the ./configure step to point at the correct kernel source location (fixes make deb step)
Link to comment
Share on other sites
  • Solution
Salamandar

Salamandar

Posted
  • Solution
Posted (edited)

Hi,

I'm still unable to build the module correctly. The ZFS kernel module is build for kernel '4.4.213-rockchip64' (the headers version name) but the kernel is '4.4.213-rk3399'.

  

spl: version magic '4.4.213-rockchip64 SMP mod_unload aarch64' should be '4.4.213-rk3399 SMP mod_unload aarch64'

 

I'm not sure this can be fixed without editing kernel headers, either : 

/usr/src/linux-headers-4.4.213-rockchip64/include/config/kernel.release
/usr/src/linux-headers-4.4.213-rockchip64/include/generated/utsrelease.h

 

 

@ShadowDance Are you using the "current" 5.9 kernel ? Maybe this has been fixed for this kernel ?

 

BTW as the procedure is a bit tiring to do by hand, I wrote some scripts to automate this (build module on Ubuntu, utils on Buster) : https://github.com/Salamandar/armbian_zfs_compiler/

 

MOD EDIT:  Above is third party script, as always, make sure you understand what is going on before running it on your system.

 

Having gotten that disclaimer out of the way, thanks for your contribution, @Salamandar!

 

I took the liberty of marking this as solution.  I also made note of the link to refer others to, later.

 

Cheers,

TRS-80

Edited by TRS-80
Moderator add disclaimer
2
Link to comment
Share on other sites
Igor

Igor

Posted
Posted
11 hours ago, Salamandar said:

Thanks, feel free to open issues if needed.

 

Any idea how to compile ZFS in the tree? I was playing a bit with that idea. If we can do it, we could perhaps ship ZFS with kernel? (or rather not? :ph34r:)

This way we cover 3rd party drivers:
https://github.com/armbian/build/blob/master/lib/compilation-prepare.sh

Link to comment
Share on other sites
manman

manman

Posted
Posted

@ShadowDance I've got an error when I was trying to build the 2.0.0 release using your instructions, getting the following error on "make deb" :
  

kmodtool error: Don't know how to handle 5.9.11 -- /lib/modules/5.9.11/build/Makefile not found


I've tried to change the kernel version to "5.9.11-rockchip64" directly on the Makefile but the build it fails anyways.
the right PATH should be /lib/modules/5.9.11-rockchip64/build/Makefile

Link to comment
Share on other sites
ShadowDance

ShadowDance

Posted
Posted

@manman hmm, the configure step should’ve taken care of that. You could try to specify the correct path manually via:

Quote

./configure --with-linux=/usr/src/linux-headers-5.9.11-rockchip64

If that still doesn’t help, see if you have any other linux-headers installed and remove them all.

Link to comment
Share on other sites
Salamandar

Salamandar

Posted
Posted
21 hours ago, Igor said:

 

Any idea how to compile ZFS in the tree? I was playing a bit with that idea. If we can do it, we could perhaps ship ZFS with kernel? (or rather not? :ph34r:)

This way we cover 3rd party drivers:
https://github.com/armbian/build/blob/master/lib/compilation-prepare.sh

What do you mean by "in the tree" ? Do you mean in /usr/src/zfs where the zfs-dkms module installs the sources ?

 

Right now the build is done inside the source directory of ZFS, and I'm looking into building it out-of-source (I'm gonna read the dkms scripts, I think they build out-of-source).

Link to comment
Share on other sites
Igor

Igor

Posted
Posted
37 minutes ago, Salamandar said:

What do you mean by "in the tree" ?


That we compile it together with kernel. Like we do for other drivers - wireguard (for older kernels) and selection of wireless drivers.

Link to comment
Share on other sites
Guest
This topic is now closed to further replies.
 Share
Go to topic listing
×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines

  I accept