Jump to content

5.15 broke zfs-dkms

eiglss

By eiglss
in Rockchip

 Share

Recommended Posts

eiglss

eiglss

Posted
Posted (edited)

Dear all;
I'm looking for some help with my Helios64 NAS, since I update kernel to 5.15, zfs-dkms won't work.
Which is normal, since the most recent version available for Helios64 of zfs-dkms is 2.0.3-1~bpo10+1 (tested on 1st April 2022), and this version is only compatible with kernel from 3.10 to 5.10.

Then the first solution that I thought was to downgrade to 5.10 through the armbian-config tool, however when I'm trying to install the linux-headears (through armbian-config) I'm having the herdears of 5.15 (same issue with apt install linux-headers-current-rockchip64) while zfs-dkms looking for 5.10. 

Preparing to unpack .../zfs-dkms_2.0.3-1~bpo10+1_all.deb ...
Unpacking zfs-dkms (2.0.3-1~bpo10+1) ...
Setting up zfs-dkms (2.0.3-1~bpo10+1) ...
Loading new zfs-2.0.3 DKMS files...
It is likely that 5.10.63-rockchip64 belongs to a chroot's host
Building for 5.15.25-rockchip64
Building initial module for 5.15.25-rockchip64
configure: error: 
	*** None of the expected "capability" interfaces were detected.
	*** This may be because your kernel version is newer than what is
	*** supported, or you are using a patched custom kernel with
	*** incompatible modifications.
	***
	*** ZFS Version: zfs-2.0.3-1~bpo10+1
	*** Compatible Kernels: 3.10 - 5.10
	
Error! Bad return status for module build on kernel: 5.15.25-rockchip64 (aarch64)
Consult /var/lib/dkms/zfs/2.0.3/build/make.log for more information.

/var/lib/dkms/zfs/2.0.3/build/make.log: 

DKMS make.log for zfs-2.0.3 for kernel 5.15.25-rockchip64 (aarch64)
Fri 01 Apr 2022 05:38:22 PM UTC
make: *** No targets specified and no makefile found.  Stop.

 

I try then to perform a fresh installation of the system by downloading Armbian_21.08.2_Helios64_buster_current_5.10.63, and then it goes worst since I can't download the header through armbian-config (nothing happen), and apt install linux-headers-current-rockchip64 keep installing the sources for 5.15 ('/usr/src/linux-headers-5.15.25-rockchip64').

So the problem how I see it is either to:
* Get the linux-headers of the previous kernel.
* obtaining a zfs-dkms version compatible with 5.15

 

Thank you in advance for your help.

 

PS: I am aware of the docker alternative, but I prefer to use zfs-dkms.

Edited by eiglss
spelling
0
TRS-80

TRS-80

Posted
Posted

If I am recalling correctly, it was non-trivial (and thus, took some time) to figure out some solution that worked, in order to make this 'easy' for end users.  It sounds like this broke now with kernel going up to 5.15.

 

If I am interpreting this search for zfs Issues on our Jira instance correctly, it looks like the developers are aware of the issue(?).

 

If you make any progress on figuring this out in the meantime, please do post back what you learned.

0
eiglss

eiglss

Posted
  • Author
Posted

Thank you TRS-80 for your answer, I took a look at the link:

  On 4/1/2022 at 8:53 PM, TRS-80 said:

If I am interpreting this search for zfs Issues on our Jira instance correctly, it looks like the developers are aware of the issue(?).

Expand  

And It seems indeed that they are aware, since there is this issue : "Add ZFS that supports kernel 5.15.y".

 

Thank you ebin-dev for your answer too, I tested your solution, and indeed it worked. However, it is not trivial since I had to download the header through your link, else I had the 5.15 header.
Moreover, even after a freeze through armbian-config: apt update && apt full-upgrade install 22.02 with 5.15. Then to prevent any upgrade with these commands and freeze at Armbian 21.08.2 Bullseye with Linux 5.10.43-rockchip64,  I had to comment the deb in this file /etc/apt/sources.list.d/armbian.list

 

0
grek

grek

Posted
Posted (edited)

@usefulnoise yesterday I ran helios64 with Armbian 22.08.0 Bullseye (Debian) and kernel 5.15.59-rockchip64 

I build image from the armbian sources. I tried to include zfs with image but i give up. 

Finally I successfully install zfs-dkms from the backports repos

  

apt -t bullseye-backports install zfs-dkms zfsutils-linux

 

modules has been builded without any errors.

Also I successfully install OMV 6 with ZFS plugin and LUKSEncryption.

  

root@helios64:~# zfs --version
zfs-2.1.5-1~bpo11+1
zfs-kmod-2.1.5-1~bpo11+1
root@helios64:~# uname -a
Linux helios64 5.15.59-rockchip64 #trunk SMP PREEMPT Wed Aug 10 11:29:36 UTC 2022 aarch64 GNU/Linux
root@helios64:~#


right now I have 16h without the reboot.

I leave system at defaults for the tests.

  

root@helios64:~# cat /boot/armbianEnv.txt
verbosity=1
bootlogo=false
overlay_prefix=rockchip
rootdev=UUID=04ed9a40-6590-4491-8d58-0d647b8cc9db
rootfstype=ext4
usbstoragequirks=0x2537:0x1066:u,0x2537:0x1068:u
root@helios64:~#

  

root@helios64:~# cat /etc/default/cpufrequtils
ENABLE="true"
GOVERNOR="ondemand"
MAX_SPEED="0"
MIN_SPEED="0"
root@helios64:~#

 

Im in the  middle ZFS scrub . We will see if its complete without errors.

Currently I saw one problem. Yesterday I ran screen with coping some data over NFS. Its failed with kernel error.

Probably I will need to go back to my old cpufreq.  

 

GOVERNOR=performance
MAX_SPEED=1416000
MIN_SPEED=1416000
 

 

  

Aug 10 17:48:21 helios64 kernel: [ 3612.621208] FS-Cache: Loaded
Aug 10 17:48:22 helios64 kernel: [ 3612.787472] FS-Cache: Netfs 'nfs' registered for caching
Aug 10 17:59:51 helios64 kernel: [ 4301.850819] Modules linked in: nfsv3 nfs fscache netfs dm_crypt dm_mod r8152 snd_soc_hdmi_codec leds_pwm gpio_charger pwm_fan panfrost snd_soc_rock
chip_i2s gpu_sched snd_soc_rockchip_pcm sg snd_soc_core snd_pcm_dmaengine snd_pcm rockchip_vdec(C) hantro_vpu(C) snd_timer rockchip_iep v4l2_h264 snd rockchip_rga videobuf2_dma_contig
 videobuf2_vmalloc v4l2_mem2mem videobuf2_dma_sg videobuf2_memops fusb302 soundcore videobuf2_v4l2 tcpm videobuf2_common typec videodev mc gpio_beeper cpufreq_dt zfs(POE) zunicode(POE
) zzstd(OE) nfsd zlua(OE) zcommon(POE) znvpair(POE) zavl(POE) auth_rpcgss icp(POE) nfs_acl lockd spl(OE) grace softdog ledtrig_netdev lm75 sunrpc ip_tables x_tables autofs4 raid10 rai
d456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 multipath linear md_mod realtek dwmac_rk stmmac_platform stmmac pcs_xpcs adc_keys
Aug 10 17:59:51 helios64 kernel: [ 4301.857423] CPU: 5 PID: 79841 Comm: mc Tainted: P         C OE     5.15.59-rockchip64 #trunk
Aug 10 17:59:51 helios64 kernel: [ 4301.858163] Hardware name: Helios64 (DT)
Aug 10 17:59:51 helios64 kernel: [ 4301.858509] pstate: 40000005 (nZcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
Aug 10 17:59:51 helios64 kernel: [ 4301.859119] pc : nfs_generic_pg_pgios+0x60/0xc0 [nfs]
Aug 10 17:59:51 helios64 kernel: [ 4301.859619] lr : nfs_generic_pg_pgios+0x5c/0xc0 [nfs]
Aug 10 17:59:51 helios64 kernel: [ 4301.860096] sp : ffff80000e2b35c0
Aug 10 17:59:51 helios64 kernel: [ 4301.860388] x29: ffff80000e2b35c0 x28: ffff00006c52d700 x27: ffff00006c52d700
Aug 10 17:59:51 helios64 kernel: [ 4301.861018] x26: ffff80000e2b3880 x25: ffff80000954be68 x24: 0000000000000400
Aug 10 17:59:51 helios64 kernel: [ 4301.861646] x23: 0000000000000000 x22: ffff00005c2f8780 x21: ffff8000019fd370
Aug 10 17:59:51 helios64 kernel: [ 4301.862273] x20: ffff80000e2b3828 x19: ffff00009ff3a580 x18: 0000000000000001
Aug 10 17:59:51 helios64 kernel: [ 4301.862901] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000002
Aug 10 17:59:51 helios64 kernel: [ 4301.863529] x14: 0000000000000080 x13: 000000000001b896 x12: ffff0000304cf9b8
Aug 10 17:59:51 helios64 kernel: [ 4301.864157] x11: ffff00009ff3a418 x10: ffff00009ff3a488 x9 : 0000000000000000
Aug 10 17:59:51 helios64 kernel: [ 4301.864784] x8 : ffff00009ff3a908 x7 : 0000000000000000 x6 : ffff00009ff3a590
Aug 10 17:59:51 helios64 kernel: [ 4301.865411] x5 : 0000000000000096 x4 : ffff00009ff3a6f8 x3 : 0000000000000800
Aug 10 17:59:51 helios64 kernel: [ 4301.866039] x2 : 50c2f8bdeb7ff200 x1 : 0000000000000000 x0 : 0000000000000000
Aug 10 17:59:51 helios64 kernel: [ 4301.866667] Call trace:
Aug 10 17:59:51 helios64 kernel: [ 4301.866883]  nfs_generic_pg_pgios+0x60/0xc0 [nfs]
Aug 10 17:59:51 helios64 kernel: [ 4301.867331]  nfs_pageio_doio+0x48/0xa0 [nfs]
Aug 10 17:59:51 helios64 kernel: [ 4301.867739]  __nfs_pageio_add_request+0x148/0x420 [nfs]
Aug 10 17:59:51 helios64 kernel: [ 4301.868230]  nfs_pageio_add_request_mirror+0x40/0x60 [nfs]
Aug 10 17:59:51 helios64 kernel: [ 4301.868743]  nfs_pageio_add_request+0x208/0x290 [nfs]
Aug 10 17:59:51 helios64 kernel: [ 4301.869220]  readpage_async_filler+0x20c/0x400 [nfs]
Aug 10 17:59:51 helios64 kernel: [ 4301.869688]  read_cache_pages+0xcc/0x198
Aug 10 17:59:51 helios64 kernel: [ 4301.870039]  nfs_readpages+0x138/0x1b8 [nfs]
Aug 10 17:59:51 helios64 kernel: [ 4301.870447]  read_pages+0x188/0x288
Aug 10 17:59:51 helios64 kernel: [ 4301.870757]  page_cache_ra_unbounded+0x148/0x248
Aug 10 17:59:51 helios64 kernel: [ 4301.871163]  do_page_cache_ra+0x40/0x50
Aug 10 17:59:51 helios64 kernel: [ 4301.871501]  ondemand_readahead+0x128/0x2c8
Aug 10 17:59:51 helios64 kernel: [ 4301.871870]  page_cache_async_ra+0xd4/0xd8
Aug 10 17:59:51 helios64 kernel: [ 4301.872232]  filemap_get_pages+0x1fc/0x568
Aug 10 17:59:51 helios64 kernel: [ 4301.872595]  filemap_read+0xac/0x340
Aug 10 17:59:51 helios64 kernel: [ 4301.872911]  generic_file_read_iter+0xf4/0x188
Aug 10 17:59:51 helios64 kernel: [ 4301.873304]  nfs_file_read+0x9c/0x128 [nfs]
Aug 10 17:59:51 helios64 kernel: [ 4301.873706]  new_sync_read+0x104/0x180
Aug 10 17:59:51 helios64 kernel: [ 4301.874040]  vfs_read+0x148/0x1e0
Aug 10 17:59:51 helios64 kernel: [ 4301.874334]  ksys_read+0x68/0xf0
Aug 10 17:59:51 helios64 kernel: [ 4301.874622]  __arm64_sys_read+0x1c/0x28
Aug 10 17:59:51 helios64 kernel: [ 4301.874960]  invoke_syscall+0x44/0x108
Aug 10 17:59:51 helios64 kernel: [ 4301.875293]  el0_svc_common.constprop.3+0x94/0xf8
Aug 10 17:59:51 helios64 kernel: [ 4301.875709]  do_el0_svc+0x24/0x88
Aug 10 17:59:51 helios64 kernel: [ 4301.876003]  el0_svc+0x20/0x50
Aug 10 17:59:51 helios64 kernel: [ 4301.876275]  el0t_64_sync_handler+0x90/0xb8
Aug 10 17:59:51 helios64 kernel: [ 4301.876645]  el0t_64_sync+0x180/0x184
Aug 10 17:59:51 helios64 kernel: [ 4301.877507] ---[ end trace 970b42649c1e42e3 ]---

 

Edit.

 

After few hours , I saw the errors regarding my HDD. 

Aug 11 11:32:32 helios64 kernel: [67461.218137] ata1: hard resetting link
Aug 11 11:32:32 helios64 kernel: [67461.694283] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Aug 11 11:32:32 helios64 kernel: [67461.696935] ata1.00: configured for UDMA/133
Aug 11 11:32:32 helios64 kernel: [67461.697672] sd 0:0:0:0: [sda] tag#19 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=0s
Aug 11 11:32:32 helios64 kernel: [67461.698535] sd 0:0:0:0: [sda] tag#19 Sense Key : 0x5 [current]
Aug 11 11:32:32 helios64 kernel: [67461.699057] sd 0:0:0:0: [sda] tag#19 ASC=0x21 ASCQ=0x4
Aug 11 11:32:32 helios64 kernel: [67461.699518] sd 0:0:0:0: [sda] tag#19 CDB: opcode=0x88 88 00 00 00 00 00 5c bd e5 50 00 00 08 00 00 00
Aug 11 11:32:32 helios64 kernel: [67461.701293] zio pool=data vdev=/dev/mapper/sda-crypt error=5 type=1 offset=796629049344 size=1048576 flags=40080cb0
Aug 11 11:32:32 helios64 kernel: [67461.702300] sd 0:0:0:0: [sda] tag#20 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=0s
Aug 11 11:32:32 helios64 kernel: [67461.703147] sd 0:0:0:0: [sda] tag#20 Sense Key : 0x5 [current]
Aug 11 11:32:32 helios64 kernel: [67461.703668] sd 0:0:0:0: [sda] tag#20 ASC=0x21 ASCQ=0x4
Aug 11 11:32:32 helios64 kernel: [67461.704127] sd 0:0:0:0: [sda] tag#20 CDB: opcode=0x88 88 00 00 00 00 00 5c bd ed 50 00 00 08 00 00 00
Aug 11 11:32:32 helios64 kernel: [67461.705890] zio pool=data vdev=/dev/mapper/sda-crypt error=5 type=1 offset=796630097920 size=1048576 flags=40080cb0
Aug 11 11:32:32 helios64 kernel: [67461.706862] sd 0:0:0:0: [sda] tag#28 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=0s
Aug 11 11:32:32 helios64 kernel: [67461.707709] sd 0:0:0:0: [sda] tag#28 Sense Key : 0x5 [current]
Aug 11 11:32:32 helios64 kernel: [67461.708228] sd 0:0:0:0: [sda] tag#28 ASC=0x21 ASCQ=0x4
Aug 11 11:32:32 helios64 kernel: [67461.708688] sd 0:0:0:0: [sda] tag#28 CDB: opcode=0x88 88 00 00 00 00 00 5c bd f5 50 00 00 08 00 00 00
Aug 11 11:32:32 helios64 kernel: [67461.710466] zio pool=data vdev=/dev/mapper/sda-crypt error=5 type=1 offset=796631146496 size=1048576 flags=40080cb0
Aug 11 11:32:32 helios64 kernel: [67461.711410] ata1: EH complete

 

 

So I changed all my setting to the old one ( before the update)  

GOVERNOR=performance
MAX_SPEED=1416000
MIN_SPEED=1416000

 

and 

  

root@helios64:~# cat /boot/armbianEnv.txt
verbosity=7
bootlogo=false
console=serial
extraargs=earlyprintk ignore_loglevel libata.force=3.0

 

we will see ,whats next. 

Rebooted system with those settings .

ZFS scrub resumed.
 

 

 

Edited by grek
0
grek

grek

Posted
Posted

@ebin-dev unfortunately I can't  :(

I faced another problem with OMV6 and LUKS and ZFS 😕 . Everything was good until the reboot. After the reboot I don't know why, but Shared directories have missed device (probably UUID was changed... ) . 

I tried to setup cryptsetup at the boot ,to encrypt my disks and then import ZFS pool ,but passphrase do not came out during a boot process (even I recompiled kernel with options found there:

FInaly I'm using workaround: keyfiles at the /boot/  ,but even that sometimes I faced the problem that only 1 disk has been encripted only (so my zfs pool was incompleted 😕 ) . I added ExecStartPre=/usr/bin/sleep 30 , to [service] at /lib/systemd/system/zfs-import-cache.service  and it looks like now it is working.... (tried 3 times) .We will see .    

 

Also the bad thing is , that I tried to back to my old configuration at emmc , but OMV5 also don't see devices at shared directories. It will work, after removing all of them and setup from the scratch . 

 

I don't know if its any progress with performance ARM with native ZFS encription , but maybe I will force to setup it up..... instead of LUKS and ZFS.

0
grek

grek

Posted
Posted

Finally I do a rollback to my old config.

armbian 21.08.8 with kernel: 5.10.63-rockchip64

I got reboots during NFS copy to helios .So during about 1 month I got 3 reboots. 

0
BinaryWaves

BinaryWaves

Posted
Posted

Hi @grek I had some kernel panics and instability on my Helios64 but mostly this was fixed from limiting the CPU clock as Helios had some issues at full speed. I usually set it to the 1400 option to be safe but I believe the 1600 is fine too? Need to test again. I just had to reinstall because my OMV was messed up so I'll see how it goes. Sometimes freezing the kernel upgrades might help too but I'm going to try and let it update to 22.08.6 tonight and see how it does with OMV. Will report back later.

1
BinaryWaves

BinaryWaves

Posted
Posted (edited)

Well, I had kernel panic when doing apt upgrade through armbian-config, so I reloaded Armbian_21.08.2_Helios64_bullseye_current_5.10.63 and froze firmware upgrades and set CPU to [408/1600/conservative]. apt update went okay and rebooted. Kernel panicked on installing with 

  

wget -O - https://github.com/OpenMediaVault-Plugin-Developers/installScript/raw/master/install | sudo bash

 

Rebooted and changed CPU to [408/1416/performance] and cleaned up some mangled packages with apt autoremove and apt upgrade for the bunch that were sitting 'unupgraded'. Reran script and got no panic but the following output. Idk if there is a way to minimize or collapse the code block, apologies:

 

  Reveal hidden contents

 

 

Edited by TRS-80
put long output inside spoiler
0
BinaryWaves

BinaryWaves

Posted
Posted

Well, after a third reflash I finally had success using [408/1414/performance] and an uncomfortably long time waiting on the install script that felt like it hung up multiple times. Then the system said there were packages to update still and broke after reboot stuck at "Starting kernel..." This is probably because of the uboot issue in the other thread.

0
n3o

n3o

Posted
Posted

@eiglss

Also I have got some issues with zfs on Helios64... I don't know if can be useful but however post the following link

 

 

hope that this help You

0

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
 Share
Go to topic listing
×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines

  I accept