Something missing or broken on ZFS and armbian, zfs-zed zfsutils-linux missing after update

[fever_wits]

Hello,
I have ZFS installed, today there was an update, in which I no longer have ZFS. 

 

root@msrv 49.00℃ :~# apt dist-upgrade 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  libnvpair3linux libuutil3linux libzfs4linux libzpool5linux
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
  zfs-zed zfsutils-linux
The following packages have been kept back:
  libnvpair3linux libzfs4linux
The following packages will be upgraded:
  libuutil3linux libzpool5linux zfs-dkms
3 upgraded, 0 newly installed, 2 to remove and 2 not upgraded.
Need to get 0 B/3,598 kB of archives.
After this operation, 2,533 kB disk space will be freed.
Do you want to continue? [Y/n] Y
Preconfiguring packages ...
(Reading database ... 71907 files and directories currently installed.)
Removing zfs-zed (2.1.11-1) ...
Removing zfsutils-linux (2.2.3-1) ...
(Reading database ... 71592 files and directories currently installed.)
Preparing to unpack .../libuutil3linux_2.2.3-2_arm64.deb ...
Unpacking libuutil3linux (2.2.3-2) over (2.2.3-1) ...
Preparing to unpack .../libzpool5linux_2.2.3-2_arm64.deb ...
Unpacking libzpool5linux (2.2.3-2) over (2.2.3-1) ...
Preparing to unpack .../zfs-dkms_2.2.3-2_all.deb ...
Module zfs-2.2.3 for kernel 6.1.43-vendor-rk35xx (aarch64).
Before uninstall, this module version was ACTIVE on this kernel.

zfs.ko:
 - Uninstallation
   - Deleting from: /lib/modules/6.1.43-vendor-rk35xx/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

spl.ko:
 - Uninstallation
   - Deleting from: /lib/modules/6.1.43-vendor-rk35xx/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.
depmod...
Deleting module zfs-2.2.3 completely from the DKMS tree.
Unpacking zfs-dkms (2.2.3-2) over (2.2.3-1) ...
Setting up zfs-dkms (2.2.3-2) ...
Loading new zfs-2.2.3 DKMS files...
Building for 6.1.43-vendor-rk35xx
Building initial module for 6.1.43-vendor-rk35xx

Done.

zfs.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/6.1.43-vendor-rk35xx/updates/dkms/

spl.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/6.1.43-vendor-rk35xx/updates/dkms/
depmod...
Setting up libuutil3linux (2.2.3-2) ...
Setting up libzpool5linux (2.2.3-2) ...
Processing triggers for libc-bin (2.36-9+deb12u7) ...
Processing triggers for man-db (2.11.2-2) ...
Processing triggers for initramfs-tools (0.142) ...
ln: failed to create hard link '/boot/initrd.img-6.1.43-vendor-rk35xx.dpkg-bak' => '/boot/initrd.img-6.1.43-vendor-rk35xx': Operation not permitted
update-initramfs: Generating /boot/initrd.img-6.1.43-vendor-rk35xx
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8125a-3.fw for module r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8107e-2.fw for module r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168fp-3.fw for module r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168g-3.fw for module r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168g-2.fw for module r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8106e-2.fw for module r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8106e-1.fw for module r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8411-2.fw for module r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8411-1.fw for module r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8402-1.fw for module r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168f-2.fw for module r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168f-1.fw for module r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8105e-1.fw for module r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168e-3.fw for module r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168e-2.fw for module r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168e-1.fw for module r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168d-2.fw for module r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168d-1.fw for module r8169
W: Possible missing firmware /lib/firmware/regulatory.db for built-in driver cfg80211
W: Possible missing firmware /lib/firmware/regulatory.db.p7s for built-in driver cfg80211
update-initramfs: Armbian: Converting to u-boot format: /boot/uInitrd-6.1.43-vendor-rk35xx
Image Name:   uInitrd
Created:      Sat May 11 11:42:20 2024
Image Type:   AArch64 Linux RAMDisk Image (gzip compressed)
Data Size:    9305942 Bytes = 9087.83 KiB = 8.87 MiB
Load Address: 00000000
Entry Point:  00000000
update-initramfs: Armbian: Symlinking /boot/uInitrd-6.1.43-vendor-rk35xx to /boot/uInitrd
ln: failed to create symbolic link '/boot/uInitrd': Operation not permitted
update-initramfs: Symlink failed, moving /boot/uInitrd-6.1.43-vendor-rk35xx to /boot/uInitrd
renamed '/boot/uInitrd-6.1.43-vendor-rk35xx' -> '/boot/uInitrd'
update-initramfs: Armbian: done.
root@msrv 48.07℃ :~# 

 

Fortunately, the module is on and everything is working now.
ZFS is from the armbian repo.

When I try to install I get this:

 

root@msrv 47.15℃ :~# aptitude install zfs-zed zfsutils-linux
The following NEW packages will be installed:
  libuutil3linux{a} zfs-zed{b} zfsutils-linux{b} 
0 packages upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 541 kB/666 kB of archives. After unpacking 2,696 kB will be used.
The following packages have unmet dependencies:
 zfs-zed : Depends: libnvpair3linux (>= 0.8.2) but it is not installable
           Depends: libzfs4linux (>= 2.1.11-1) but it is not installable
 zfsutils-linux : Depends: libnvpair3linux (= 2.2.3-2) but it is not installable
                  Depends: libzfs4linux (= 2.2.3-2) but it is not installable
                  Depends: libzpool5linux (= 2.2.3-2) but it is not installable
                  Depends: libssl3t64 (>= 3.0.0) which is a virtual package and is not provided by any available package

The following actions will resolve these dependencies:

     Keep the following packages at their current version:
1)     zfs-zed [Not Installed]                            
2)     zfsutils-linux [Not Installed]                     



Accept this solution? [Y/n/q/?] Y
No packages will be installed, upgraded, or removed.
0 packages upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B of archives. After unpacking 0 B will be used.
                                         

 

Any ideas on how to fix it and/or what to provide as additional information

 

[fever_wits]

Hello,

I found a temporary solution to the problem:
Here are the steps:

 

aptitude remove zfs-dkms
aptitude install zfs-zed=2.2.3-1~bpo12+1 zfsutils-linux=2.2.3-1~bpo12+1 zfs-dkms=2.2.3-1~bpo12+1
aptitude install libnvpair3linux=2.2.3-1~bpo12+1 libzfs4linux=2.2.3-1~bpo12+1 libuutil3linux=2.2.3-1~bpo12+1 libzfs4linux=2.2.3-1~bpo12+1 libzpool5linux=2.2. 3-1~bpo12+1
aptitude install zfs-zed=2.2.3-1~bpo12+1 zfsutils-linux=2.2.3-1~bpo12+1
aptitude hold libnvpair3linux libuutil3linux libzfs4linux libzpool5linux zfs-dkms zfs-zed zfsutils-linux

 

After "locking" the packages, you should see this:

 

root@msrv 48.07℃ :~# apt-mark showhold
libnvpair3linux
libuutil3linux
libzfs4linux
libzpool5linux
zfs-dkms
zfs-zed
zfsutils-linux

 

systemctl enable --now zfs-scrub-weekly@ZPOOL.timer
systemctl enable --now zfs-trim-weekly@ZPOOL.timer - if the array is on SSD

 

It would be nice from the armbian team to fix things

 

Actually, the new versions of zfs-dkms and zfsutils-linux come from the armbian repo, and these packages look for the libssl3t64 package, which is not present in Debian Bookworm, but is present in Debian Sid.
I assume the Debian packages are compiled on top of Ubuntu, which is based on the Debian Sid.

 

Regards,

[SteeMan]

Your board isn't supported by Armbian.  It is Community Maintained.  You are the community, PRs are welcome with fixes.

[fever_wits]

1 hour ago, SteeMan said:

Your board isn't supported by Armbian.  It is Community Maintained. 

I have no complaints about the kernel that comes with the board. And with the fact that the ZFS packages supplied by armbian are not for debian 12 (bookworm).
This has nothing to do with my board. I imagine there will be other boards that will have problems with the zfs package that comes from armbian and are using debian 12.

 

1 hour ago, SteeMan said:

You are the community, PRs are welcome with fixes.

As you can see, I have also shared the workaround.

[fever_wits]

Due to my oversight, I have not posted the topic in the correct section. I'll help move it to "Software, Applications, Userspace".

[SteeMan]

3 hours ago, fever_wits said:

It would be nice from the armbian team to fix things

I was reacting to this comment.  No Armbian devs are going to look at this issue because it is reported on an unsupported board.  I was just explaining how the various support models work.  While it may or may not be a generic issue, the fact is that it is reported on an unsupported board.  With very limited resources the volunteer devs can't look into most things that are reported.  Thus I was encouraging you to contribute your time to submit a fix.  You did provide a workaround, but indicate here that you are looking for an underlying fix.  I'm just here saying that unless someone in the community steps up that isn't likely to happen.

[Igor]

Quote

6.1.43-vendor-rk35xx

 

I would not use ZFS with this kernel if I were you. This kernel is provided by hw vendor, by people / business that only cares about (selling) hardware features to work. Everything else is unmaintained and open source community (not just Armbian) developers are not even thinking to maintain this kernel outside that as its concorde fallacy. FYI - it is highly possible ZFS implementation is affected, but you are FREE to do whatever. 

 

13 hours ago, fever_wits said:

As you can see, I have also shared the workaround.

 

90% of solutions, not workarounds, Armbian team (a few people) is unable to integrate into OS with their private resources, public (thousands of you) are simply not there. I can estimate, we could integrate about 30% of resolved suggestions this community generate with few full time engineers. Which costs money. Until you are satisfied with reporting problems only, until you don't care that a lot of value is wasted, there is nothing we can do but keep trying to provide "best effort" support with what we have and explaining this situation. Users perspective will always be like we refuse to help ... but people that are working on / helping Armbian are already long maxed out in efforts of helping everyone, even our competitors. Also. Hardware vendors have exploit of open source community build into their business model. If it happens that you are using hardware with terrible vendors reputation, bugs can be written, but are last to be resolved, some bugs are never resolved ... 

 

If you really wanna help everyone, start here:

https://github.com/armbian/os/wiki/Import-3rd-party-packages

Study and implement a solution. We also have automated dependency testing, which should prevent this problem in first place, but ain't working well and it wouldn't help much in this case.

 

ZFS package is imported from Trixie and yes I understand that upstream package dependency has been recently changed. Until a month ago, we didn't have this problem, everything was working right and now, there is little we can do except disabling importing (which will also delete packages from repository). We don't compile those packages on our own and I am aware that this is not the best way. 

 

There are more workarounds - like using Trixie or Noble user space ...
 

17 hours ago, fever_wits said:

armbian team

 

Its sadly under-powered, too small. 

 

Math is simple - I would very much like to help, but costs are the one that are killing us. Imagine that fixing a problem costs 1000 USD. You are willing to pay 5 USD (or not even that), which is fair if there would be more people contributes their share that you cover all costs. This never happens. They come with another and another problem for 1000 USD (often a lot more, sometimes a lot less) instead. Then its expected that we pay the rest, 995 USD for costs of resolving community software problem, this is simply not sustainable way of cooperation. Even the bug was made by us. I am trying to keep ZFS available. Do you have any idea how much time loss this service already caused with no benefits whatsoever?

 

There are thousands of bugs in open source software and most of people pin everything on us after they download "Armbian image" and images that are produced by copycats as they claim "its Armbian". This particular problem was introduced by us, yes, but alternatively would be not providing ZFS that is capable of working with recent kernels. Which touches to the core of embedded Linux experience and logic. Often we can't match user space and kernel like this is done on 1st class x86 desktop / server / hw Linux world. Even they, often breaks critical things too, even with thousands of full time employees (Redhat). We can't provide that level of services with close to nothing from your side.