ShadowDance

It's probably indicating a kernel panic. You could try limiting CPU speed and setting the governor as suggested in the linked topic. You could also try hooking up the serial port to another machine and monitor it for any output indicating why the crash happened.
 

It's probably indicating a kernel panic. You could try limiting CPU speed and setting the governor as suggested in the linked topic. You could also try hooking up the serial port to another machine and monitor it for any output indicating why the crash happened.
 

This looks like the same issue as I’ve been having, it’s possibly due to the SATA cables in the harness. I’ve tested with other SATA cables and can’t reproduce the issue with those. So I believe it’s not likely to be bad disks but that is a possibility of course. You can start by trying to limit SATA speed to 3 Gbps and see if that helps, if it does it’s very likely the same issue. I also recommend doing SMART scans and scrub _after_ limiting the SATA speed.
 
See my other topic for more information: 

Good luck!

Thank you for your answer.
I know about ashift=12, my tests have been made with this setting. I also tried your settings (without compression!), but they make no real difference. 
Using three disks brings things up to ~80MB/s - which is still unacceptable. 
 
But I think I found my problem:
It is actually related to the blocksize - but not the blocksize of the zfs pool - but the blocksize dd is using:
 
This is what I'm usually doing:
 
root@helios64:/test# dd if=test2.bin of=/dev/null 9765625+0 records in 9765625+0 records out 5000000000 bytes (5.0 GB, 4.7 GiB) copied, 107.466 s, 46.5 MB/s  
DD is using a blocksize of 512 here. For ext4 / btrfs this seems to be no issue - but zfs has a topic with this. 
If I use explicity a blocksize of 4096, I get these results:

 
root@helios64:/test# dd if=test2.bin of=/dev/null bs=4096 1220703+1 records in 1220703+1 records out 5000000000 bytes (5.0 GB, 4.7 GiB) copied, 27.4704 s, 182 MB/s  
Which gives the figures expected!
 
So as so often: "Layer 8 problem"  - the problem sits in front of the screen. 

@dancgn if it crashes occasionally, you could try to use armbian-config to change the CPU speed / governor to 'performance' (armbian-config > System > CPU). I personally use 1800000 for CPU speed but I've seen 1200000 recommended elsewhere on the forum, YMMV.

If you're seeing a few of those every now and then, nothing to be alarmed about. It can happen when the CPU is stressed and is normal. If you're constantly seeing it then you might want to consider either reducing the CPU load or disabling NOHZ.
 
For understanding what NOHZ does, I highly recommend this Stack Overflow answer to: "How NOHZ=ON affects do_timer() in Linux kernel?".
 
If you want to get rid of the message, you can simply add the following to /boot/armbianEnv.txt:
 
extraargs=nohz=off  
But if you decide to do this, make sure you fully understand what it does by reading the aforementioned SO article and perhaps https://www.kernel.org/doc/Documentation/timers/NO_HZ.txt.

Thanks for the update. Good to hear that with fix cpu frequency system is stable.
 
Yeah the main source of instability is DVFS, we already did some change by tweaking regulator ramp delay : https://github.com/armbian/build/commit/d18323fcc358ddd5bd0836c4853cbc281b6293c6
 
It improved a lot stability but unfortunately doesn't seem to completely solved the problem for everyone. So we are still investigating and getting Rockchip to look at it also.

@grek I've been using compression since the get-go, and can't say I've had similar issues so far. The only time I had a kernel panic was from before I switched to the performance governor for the CPU. LZ4 is also a pretty light-weight compression algorithm, but it does increase CPU to a small degree.
 
Potential changes I have compared to your setup:
I run root (/) on ZFS I use full disk encryption via LUKS (that's an additional layer between ZFS and kernel) I use the performance governor My disks are limited to 3.0 Gbps SATA speed Things you could try:
Upgrade to the latest 5.9.14-rockchip64 kernel Upgrade to the latest ZFS 0.8.6 kernel modules Try the performance governor Do you use a swap partition? If yes, you could try to swapoff before repeating the process that paniced the kernel Wildcard idea: You could also try disabling the Armbian zram config in /etc/default/armbian-zram-config (I have it disabled)

This is what I do as well , although I had to create a dummy zfs-dkms package to prevent it from being pulled in by zfsutils-linux.

If you want the fans of your Helios64 to start spinning earlier, look no further. This will allow the fans to spin at a constant speed from the earliest stage of initrd until fancontrol is started by the system. My use-case is for full disk encryption and recovery in initramfs, sometimes I can have the machine powered on for quite a while before fancontrol starts. Or, if the boot process encounters an error the fans may never start, this prevents that.
 
Step 1: Tell initramfs to include the pwm-fan module
echo pwm-fan | sudo tee -a /etc/initramfs-tools/modules  
Step 2: Create a new file at /etc/initramfs-tools/scripts/init-top/fan with the following contents:
#!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in prereqs) prereqs exit 0 ;; esac . /scripts/functions modprobe pwm-fan for pwm in /sys/devices/platform/p*-fan/hwmon/hwmon*/pwm1; do echo 150 >$pwm done exit 0 Feel free to change the value 150 anywhere between 1-255, it decides how fast the fans will spin.
 
Step 3: Enable the fan script for inclusion in initramfs and update the initramfs image:
sudo chmod +x /etc/initramfs-tools/scripts/init-top/fan sudo update-initramfs -u -k all  
Step 4: Reboot and ejoy the cool breeze.

I was still seeing the issue on Armbian 20.11 Buster (LK 5.9) and it actually seemed worse (even with the 3Gbps limit). So, considering my suspicion of power issue (the push from @TheLinuxBug helped too ), I took the plunge and got a bit creative with my testing. 
 
Here's the TL;DR:
Stock power/harness has the issue even with only one drive connected Stock power/harness has the issue with UPS disconnected Stock harness has the issue when only powering via UPS battery (no PSU), this triggered the issue without even reading from the disk (idle disk, it happened immediately after boot in initrd) Separate PSU for disk and different SATA cable (no harness) == no issue  
The 3 Gbps limit was removed for these tests and I picked the ata4 port because I knew it could reliably reproduce the issue soon after boot. I've attached dmesg's from cases 2-4 and I've annotated some of them.
 
I'm leaning towards power-issue here considering case 3 and 4. For reference these drives are rated 0.60A @ 5V and 0.45A @ 12V. I would've liked to test the harness power / harness SATA cable separately but that's unfortunately quite hard, and I didn't have any SATA extension cables (female/male) that would have allowed me to test the harness SATA cable separately from power.
 
In case someone finds it amusing, here's my testing rig (case 4):

Notable here is that I performed the 4th test 3 times. The first time I had the drive powered on before powering on the Helios64, but the link was set to 3 Gbps for some reason. I powered both down and powered first on Helios64, waited 2 seconds, then I powered on the drive, this way the link was set up as 6 Gbps and no issues when dd'ing. For good measure I repeated this a second time and still no issue.
 
@gprovost Is there anything else I could test? Or would the next step be to try a different harness, PSU, etc?
 
@aprayoga I have indeed juggled them around, but my issue is with all SATA ports, unlikely that all of them are loose. And thanks for attempting to reproduce, I'll still need to check if it happens with other drives too, but that's a task for another day.
2-dmesg-internal-harness-no-ups.txt 3-dmesg-no-psu-only-ups.txt 4-dmesg-external-psu-sata-cable.txt

If you want the fans of your Helios64 to start spinning earlier, look no further. This will allow the fans to spin at a constant speed from the earliest stage of initrd until fancontrol is started by the system. My use-case is for full disk encryption and recovery in initramfs, sometimes I can have the machine powered on for quite a while before fancontrol starts. Or, if the boot process encounters an error the fans may never start, this prevents that.
 
Step 1: Tell initramfs to include the pwm-fan module
echo pwm-fan | sudo tee -a /etc/initramfs-tools/modules  
Step 2: Create a new file at /etc/initramfs-tools/scripts/init-top/fan with the following contents:
#!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in prereqs) prereqs exit 0 ;; esac . /scripts/functions modprobe pwm-fan for pwm in /sys/devices/platform/p*-fan/hwmon/hwmon*/pwm1; do echo 150 >$pwm done exit 0 Feel free to change the value 150 anywhere between 1-255, it decides how fast the fans will spin.
 
Step 3: Enable the fan script for inclusion in initramfs and update the initramfs image:
sudo chmod +x /etc/initramfs-tools/scripts/init-top/fan sudo update-initramfs -u -k all  
Step 4: Reboot and ejoy the cool breeze.

This is what I do as well , although I had to create a dummy zfs-dkms package to prevent it from being pulled in by zfsutils-linux.

I was still seeing the issue on Armbian 20.11 Buster (LK 5.9) and it actually seemed worse (even with the 3Gbps limit). So, considering my suspicion of power issue (the push from @TheLinuxBug helped too ), I took the plunge and got a bit creative with my testing. 
 
Here's the TL;DR:
Stock power/harness has the issue even with only one drive connected Stock power/harness has the issue with UPS disconnected Stock harness has the issue when only powering via UPS battery (no PSU), this triggered the issue without even reading from the disk (idle disk, it happened immediately after boot in initrd) Separate PSU for disk and different SATA cable (no harness) == no issue  
The 3 Gbps limit was removed for these tests and I picked the ata4 port because I knew it could reliably reproduce the issue soon after boot. I've attached dmesg's from cases 2-4 and I've annotated some of them.
 
I'm leaning towards power-issue here considering case 3 and 4. For reference these drives are rated 0.60A @ 5V and 0.45A @ 12V. I would've liked to test the harness power / harness SATA cable separately but that's unfortunately quite hard, and I didn't have any SATA extension cables (female/male) that would have allowed me to test the harness SATA cable separately from power.
 
In case someone finds it amusing, here's my testing rig (case 4):

Notable here is that I performed the 4th test 3 times. The first time I had the drive powered on before powering on the Helios64, but the link was set to 3 Gbps for some reason. I powered both down and powered first on Helios64, waited 2 seconds, then I powered on the drive, this way the link was set up as 6 Gbps and no issues when dd'ing. For good measure I repeated this a second time and still no issue.
 
@gprovost Is there anything else I could test? Or would the next step be to try a different harness, PSU, etc?
 
@aprayoga I have indeed juggled them around, but my issue is with all SATA ports, unlikely that all of them are loose. And thanks for attempting to reproduce, I'll still need to check if it happens with other drives too, but that's a task for another day.
2-dmesg-internal-harness-no-ups.txt 3-dmesg-no-psu-only-ups.txt 4-dmesg-external-psu-sata-cable.txt

@SymbiosisSystems I take it you have a set of 0.8.5 modules built? They work fine with the 0.8.4 zfsutils-linux package, but it requires the zfs-dkms package which will fail to build. We can work around this by installing a dummy package that provides zfs-dkms so that we then can go ahead and install zfsutils-linux / zfs-zed / etc. from backports.
 
Here's how you can create a dummy package:
apt-get install --yes equivs mkdir zfs-dkms-dummy; cd $_ cat <<EOF >zfs-dkms Section: misc Priority: optional Standards-Version: 3.9.2 Package: zfs-dkms-dummy Version: 0.8.4 Maintainer: Me <me@localhost> Provides: zfs-dkms Architecture: all Description: Dummy zfs-dkms package for when using built kmod EOF equivs-build zfs-dkms dpkg -i zfs-dkms-dummy_0.8.4_all.deb  
After this, you can go ahead and install (if not already installed) the 0.8.5 modules (kmod-zfs-5.*-rockchip64_0.8.5-1_arm64.deb) and zfsutils-linux.

@privilegejunkie what specific model? My guess is that the answer is no, though. That's a relatively old article and the WD utility mentioned there only applies to old Red drives that are 4TB and below. For WD drives that support IDLE3, you can try to use idle3ctl (apt install idle3-tools) to manage their idle setting. I've turned idle3 off on all my drives because they're usually idle for, at most 5 minutes leading to constant sleep/wake.

@privilegejunkie what specific model? My guess is that the answer is no, though. That's a relatively old article and the WD utility mentioned there only applies to old Red drives that are 4TB and below. For WD drives that support IDLE3, you can try to use idle3ctl (apt install idle3-tools) to manage their idle setting. I've turned idle3 off on all my drives because they're usually idle for, at most 5 minutes leading to constant sleep/wake.

If you created a separate partition for swap on the raw disk (not on top of RAID), I see only one reason that it would lead to a kernel panic, and that's a bad disk. A bad disk would lead to memory corruption. Have you run a S.M.A.R.T. self-test (both short and long) on the disk that had the swap?
 
That said, I would expect swap to work on top of RAID as well, but it's an extra layer that might get congested during high memory pressure. ZFS, for instance, doesn't deal well with swap on top of it, there's at least one long-standing open issue about it.

RK3399 commercial grade is rated as 0 - 80C ambient operating temperature. To not be confused with max Temperature junction (Tj) which is 125C ;-)

@Jaques-Ludwig As explains by @SIGSEGV you still need to store the bootloader somewhere else than USB because the system cannot boot directly from USB. So in our wiki we present the following approach : Bootloader on eMMC, Root file System on USB.
 
Very soon we will also explain how to store bootloader SPI, but honestly not really major added value than the above approach.
 
 
Technically you need to use jumper P10 to disable eMMC to be sure the system won't boot from it.
However take note the 1st stage bootloader (aka SPL) will always try to first find 2nd stage bootloader (aka U-boot) on microSD first, then eMMC. So it means that even if system boot from eMMC, if there is a microSD card inserted with a 2nd stage bootloader (aka U-Boot), the system will boot the system from microSD card... therefore you don't really need to bother about P10 jumper. Hope I'm not confusing more by trying to explain more :-/

I replaced the stock fans with Noctua NF-A8-PWM fans. These have slightly lower CFM (airflow) at 32.5 vs 35 (stock) however they have a high static pressure rating at 2.37 mm H₂O. For me they produce better cooling than the stock fans. Their noise level is also rated at 17.1 dB(A) so you will barely hear them even at full speed.
 
I also had a pair of NF-R8-PWM (discontinued model) from the old days which I tried. They are very close in CFM at 31.4 but have worse static pressure at 1.41 mm H₂O. These fans produced worse cooling than the stock fans.
 
One additional change I did was place the metal fan grills (finger protectors) on the outside because then the fan is allowed to produce a seal against the case. I thought it was a small design miss to leave a gap between the case and the fan because it might allow air to move back inside. Aesthetically it's nicer to have the grill on the inside (IMO), so as an alternative fix one could design a 3D printed piece to fill the gaps.
 
It's also possible to adjust speed by modifying the `/etc/fancontrol` configuration and restarting the service (`systemctl restart fancontrol`), but I would not recommend this unless you're using better than stock fans. If the CPU is working full throttle you will want the fans running at full speed to sufficiently extract the heat from the CPU area.

This is a continuation of ata1.00: failed command: READ FPDMA QUEUED noted by @djurny. I've experienced the same issue, and have some additional data points to provide.
 
My observations so far:
I'm using WDC WD60EFRX (68MYMN1 and 68L0BN1) drives The drives were working without issue previously behind a ASMedia ASM1062 SATA Controller, I've also used some of them behind ASM1542 (external eSATA enclosure) I can reproduce the issue on a clean install of Armbian 20.08.21 Buster and Focal I can reproduce via simple `dd` to `/dev/null` from the drive so filesystem does not seem to be the underlying cause Every drive is affected (i.e. each SATA slot) At what point dd produces an error varies from SATA slot to SATA slot (not drive dependent), SATA slot 4 can reproducibly produce the error almost immediately after starting a read The problem goes away when setting `extraargs=libata.force=3.0` in `/boot/armbianEnv.txt` [1] [1] However, even with SATA limited to 3 Gbps, the problem did reappear when hot-plugging a drive.
 
This reset happened on drive slot 3 when I hot-plugged a drive onto slot 5. This seems weird to me considering they are supposed to be on different power rails. This may suggest there is in general a problem with either the PSU or power delivery to the drives. Here's an excerpt from the reset:
 
[152957.354311] ata3.00: exception Emask 0x10 SAct 0x80000000 SErr 0x9b0000 action 0xe frozen [152957.354318] ata3.00: irq_stat 0x00400000, PHY RDY changed [152957.354322] ata3: SError: { PHYRdyChg PHYInt 10B8B Dispar LinkSeq } [152957.354328] ata3.00: failed command: READ FPDMA QUEUED [152957.354335] ata3.00: cmd 60/58:f8:00:f8:e7/01:00:71:02:00/40 tag 31 ncq dma 176128 in                          res 40/00:f8:00:f8:e7/00:00:71:02:00/40 Emask 0x10 (ATA bus error) [152957.354338] ata3.00: status: { DRDY } [152957.354345] ata3: hard resetting link  
And the full dmesg from when the error happened is below:
 
 

Thanks to the Armbian team, we have now this forum club dedicated to Kobol products.
 
Feel free to create new topics to report issues, discuss features or share experience with Helios64.

@jbergler I had the same issue with panics on 20.08.21 Buster (also using ZFS). Problem went away after I switched from ondemand to performance governor via armbian-config.

@jbergler I had the same issue with panics on 20.08.21 Buster (also using ZFS). Problem went away after I switched from ondemand to performance governor via armbian-config.