QuaxEros

Hi, After the success with the 6xSATA board on Bookworm/Jammy edge and a stable system without your irq issue i wanted to push forward and have it run ZFS, iSCSI targets and OMV7. Did bad research because OMV does neither. Had fun though compiling ZFS for 6.7.x, trying different combos with Debian/Ubuntu. In the end finished this adventure with another weird purpose for my 6 disk Zpool and wrote a bit about the experience in another thread. If you wanna read it, it's here Thanx for your help and C U !

Hi, Was busy elsewhere for some time. Back again. @Werner Thanx for the hints and the links. Did some reading up. Building ZFS was straightforward. No config changes necessary, no tweaking , just clone, compile and install. Just like my FreeBSD rig years ago 😉 some quirks: tarball still incompatible with kernel 6.7.x, latest git version includes it so: $ git clone ... is the way Jammy missed some "needed" package but still built and installed, Bookworm has all packages. (for both some additional packages were needed besides the ones mentioned in the README) Tried the Kmod method first, but install failed. DKMS method worked fine. Had to remove Dracut module though before installing otherwise it needs all other packages that come with it, as dependencies. So ZFS compiled and installed, all seemed fine. Issues: OMV 7 does not install well on Ubuntu Jammy. It installs fine on Debian Bookworm (which is recommended). OMV does not want to work with ZFS. Tried to offer it the created Zpool as ready to use storage but could not get it to handle it properly. Then tried OMV ZFS plugin (Dev) but does not build well on Debian 12, and starts creating downgrade demands for ZFS and upgrades of the plugin itself if you force-install the badly built plugin. Then crash and after reboot ZFS-plugin is gone. Could create btrfs RAID but was keen on iSCSI. OMV does not do iSCSI natively, left it there . TrueNAS is only for X64_x86, no option. Other problem: My SATA extension board (ASM1166) holds 6 slots. I have 5 spinners and 1 500G ssd (for cache, so that the hdd's can spin down. I wanted to try while i know the downsides of such a big cache. See ZFS recommendations!). They are powered by an external old PC PSU with enough amps on the 12V rail but almost every (re)boot 1 of the hdd's was missing, not always the same. Tried the same PSU with a server-board with native sata and all disks would come up without a hitch. - Could the OPI5+ have troubles initialising/managing this many spinners on boot ? Maybe it is the M.2-to-SATA extension board. Power to the disks was never off but they seemed to have speed problems and sometimes clicking (they are new and don't have this on the server-board where i placed them for now). Gonna play some more with the OPI5+ in other missions, need to grasp the arm64 SBC platform better and it would be a shame to give it a serious job and not touch it any more as well .. ;-0 If someone wants the step by step ZFS compile write-up i can post it though it is already clearly documented, just shout. Started this adventure in another thread were the forum helped me with my hardware setup. i cross reference the thread for completeness: For now my cheap Huananzhi x99-F8 (28 core Xeon, 128G DDR4, 1TB nvme) is quite happy with my Zpool. Proxmox does passthrough to TrueNAS Scale VM which in turn serves iSCSI targets back to Proxmox, Pmox BackupServer VM, Windows gaming VM (Radeon passthrough)/Jellyfin, CasaOS LXC with NextCloud and hey, why not OMV7 with Portainer? All this looped behind PfSense VM with 3 bonded double NIC's. When will Russian Dolls manifest as Russian Roulette... Well, i'll tell you when it all breaks I'll keep my nvme-to-sata for later use with some ssd's when i'm finished fooling around. And i'm sure i'll find another issue to create a new thread. Thanx for your enthusiasm and hard work to make this place so pleasant!!

The getting started guide didn't help me much, rather confusing for the case of ubuntu. The post for legacy i had read. The problem is that my nvme-to-sata doesn't work on legacy. I'll go and try building though i don't have much experience with that, but maybe is time to learn.. Is the licence issue a big problem ? Is the tainted kernel critical ? Thnx for your troubles! Gonna go to work and try...

Hi Werner, Yeah, i saw the version incompatibility. But since it was almost the same in legacy i thought it was not all about that. I'll go look at your suggestion. Thnx for the quick response... You must work overtime!! I'll report back when i got it working.

Hmm, seems i need someone holding my hand all steps of the way 😇. Soon i'll need to pay some.. 🥰 I got my OPI5plus working from emmc with nvme-to-sata adapter on edge kernel and now i have 5 HDDs and 1 SSD connected to it. Intended for ZFS with cache... But : ZFS module won't build. I installed linux-headers and tried # apt install zfsutils-linux zfs-dkms. --- Setting up zfs-dkms (2.1.9-2ubuntu1) ... Loading new zfs-2.1.9 DKMS files... Building for 6.7.0-rc1-edge-rockchip-rk3588 Building initial module for 6.7.0-rc1-edge-rockchip-rk3588 configure: error: *** None of the expected "bops->release()" interfaces were detected. *** This may be because your kernel version is newer than what is *** supported, or you are using a patched custom kernel with *** incompatible modifications. *** *** ZFS Version: zfs-2.1.9-2ubuntu1 *** Compatible Kernels: 3.10 - 6.1 Error! Bad return status for module build on kernel: 6.7.0-rc1-edge-rockchip-rk3588 (aarch64) Consult /var/lib/dkms/zfs/2.1.9/build/make.log for more information. dpkg: error processing package zfs-dkms (--configure): installed zfs-dkms package post-installation script subprocess returned error exit status 10 --- (It said something similar on legacy kernel ...kind of..) The build log is not very helpful: DKMS make.log for zfs-2.1.9 for kernel 6.7.0-rc1-edge-rockchip-rk3588 (aarch64) Sun Jan 21 06:44:59 AM CET 2024 make: *** No targets specified and no makefile found. Stop. (The makefile IS there..) The questions: Am i doing something wrong ? Is this a known state of the zfs package for this board ? Is there something i can do to "work around/solve/help finding a solution" ? Are there other FS's that can make use of my disks with ~similar features as ZFS ?

In the post #3 I posted my ENTIRE lspci with Jammy_legacy After installing Jammy_edge (with quite some trouble) on emmc, initially no boot. Had to wipe and reflash (using RKDevTool) multiple times both SPI and EMMC Debug shows: Net: No ethernet found. and hangs at: Starting kernel ... Then flashed rkspi_loader.img to SPINOR. The board booted and i got one extra PCI bridge.. # lspci 0000:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd Device 3588 (rev 01) 0002:20:00.0 PCI bridge: Rockchip Electronics Co., Ltd Device 3588 (rev 01) 0002:21:00.0 Network controller: Realtek Semiconductor Co., Ltd. Device b852 0003:30:00.0 PCI bridge: Rockchip Electronics Co., Ltd Device 3588 (rev 01) 0003:31:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05) 0004:40:00.0 PCI bridge: Rockchip Electronics Co., Ltd Device 3588 (rev 01) 0004:41:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05) Screwed back in my nvme-to-sata.. # lspci 0000:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd Device 3588 (rev 01) 0000:01:00.0 SATA controller: ASMedia Technology Inc. Device 1166 (rev 02) Cigar!! # lspci -s 0000:01:00.0 -vv | grep -E "(LnkCap:|LnkSta:|driver)" LnkCap: Port #0, Speed 8GT/s, Width x2, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us LnkSta: Speed 8GT/s (ok), Width x2 (ok) Kernel driver in use: ahci I'm really delighted that there was so much response from the forum. It's a nice place to come to, Much obliged!! I had problems on legacy kernel to get zfs.ko module created, Something with the linux-headers didn't seem right. Gonna play and see if it works better on edge... I had already tried installing OMV 7 and got it kind of working, gonna try that after zfs. I hope the 32G will be ample for zfs, but i've seen it eating all up. Well guys, THNX a lot! I can play on..

It is the adapter from the link, not sure about it's version. The OPI5 has onboard sata so i'm sure there's support out of the box. ### lspci: 0002:20:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3588 (rev 01) 0002:21:00.0 Network controller: Realtek Semiconductor Co., Ltd. Device b852 0003:30:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3588 (rev 01) 0003:31:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05) 0004:40:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3588 (rev 01) 0004:41:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05) No ahci listed in lsmod. First i'll look if the module is present and if it will load. Then i'll go try the edge kernel (on my 2nd emmc 😜), see how it works out.. Q: Can flashing a different rkspi_loader.img (from Joshua Riek) have positive result or should i stick to the image from the distro? Q; Will booting from SD instead of emmc show me different targets (like sata) in armbian-install?

Hi gals & guys, Came over to this forum from the OPI forums which seem.. rather low on real-user activity. I am quite new to the SBC Arm64 platform and have many questions. I am an experienced sysadmin on x86/amd64 and still figuring out the differences. I bought a Orange Pi 5 Plus 32G of RAM and i am running of the 256G emmc. I am able to run different OS's (OPI-Ubuntu, OPI-debian, plain Ubuntu and Armbian). As i intend to use the board as server i didn't look into Android and the like.. My plan is running (OMV) from the emmc and attaching ZFS storage by means of a nvme-to-sata adapter and providing iscsi storage from there to my home-lab with my VM's and containers + shared storage and backup. Teamed NIC's for the network, for later. With that out of the way, here is my question... My OPI5plus is running fine from emmc and i am able to handle most normal issues. My main problem is that i cannot get my storage or even my nvme-to-sata adapter detected. I don't really know how to go about it and how to use overlays to configure my system. Beside the fact that the OPI5plus has no native sata and only 1 sata2 overlay, i have a basic lack of understanding of the overlay concept, i guess. I've seen very little information online on "OPI5plus NAS/SATA" and was wondering if someone here could help me out ? Config: Armbian_Bookworm_legacy 5.10.160 on Emmc rkspi_loader.img flashed, else no boot. Tried: rkspi_loader_sata.img for OPI5pro, but then no boot overlay rk3588_sata2, but then no NIC's My nvme-to-sata adapter is an M.2 nvme pcie3 to 6 port sata3.0 with ASM1166 chipset. At boot time the leds on it flicker, then go out. In lspci there is no mention of it neither in lshw (not even as unclaimed or unidentified device) in dmesg there is: rk-pcie fe170000.pcie: PCIe Link Up rk-pcie fe180000.pcie: PCIe Link Up rk-pcie fe190000.pcie: PCIe Link Up (Are these the pcie bridges or the lan & wlan NIC's ?) rk-pcie fe150000.pcie: PCIe Link Failed (But i can't decrypt why) armbianmonitor: https://paste.armbian.com/ovicanekan Thanx for all your work helping others! Hope i can join you soon..