aegiap

Posting here following what was recommended on twitter.
After updating my helios64 earlier this week and rebooting to get the new kernel, I realized it was suspiciously silent.
A quick check to sensor temps readings and physical check made me realize the fan were not spinning.
 
After a quick read on the wiki, I checked fancontrol which was indeed failing:
root@helios64:~ # systemctl status fancontrol.service ● fancontrol.service - fan speed regulator Loaded: loaded (/lib/systemd/system/fancontrol.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/fancontrol.service.d └─pid.conf Active: failed (Result: exit-code) since Fri 2021-05-28 00:08:13 CEST; 1min 42s ago Docs: man:fancontrol(8) man:pwmconfig(8) Process: 2495 ExecStartPre=/usr/sbin/fancontrol --check (code=exited, status=0/SUCCESS) Process: 2876 ExecStart=/usr/sbin/fancontrol (code=exited, status=1/FAILURE) Main PID: 2876 (code=exited, status=1/FAILURE) May 28 00:08:13 helios64 fancontrol[2876]: MINPWM=0 May 28 00:08:13 helios64 fancontrol[2876]: MAXPWM=255 May 28 00:08:13 helios64 fancontrol[2876]: AVERAGE=1 May 28 00:08:13 helios64 fancontrol[2876]: Error: file /dev/thermal-cpu/temp1_input doesn't exist May 28 00:08:13 helios64 fancontrol[2876]: Error: file /dev/thermal-cpu/temp1_input doesn't exist May 28 00:08:13 helios64 fancontrol[2876]: At least one referenced file is missing. Either some required kernel May 28 00:08:13 helios64 fancontrol[2876]: modules haven't been loaded, or your configuration file is outdated. May 28 00:08:13 helios64 fancontrol[2876]: In the latter case, you should run pwmconfig again. May 28 00:08:13 helios64 systemd[1]: fancontrol.service: Main process exited, code=exited, status=1/FAILURE May 28 00:08:13 helios64 systemd[1]: fancontrol.service: Failed with result 'exit-code'.  
Basically fancontrol expect a device in /dev to read the sensors value from, and that device seems to be missing. After a bit of poking around and learning about udev, I managed to manually solve the issue by recreating the device symlink manually:
/usr/bin/mkdir /dev/thermal-cpu/ ln -s /sys/devices/virtual/thermal/thermal_zone0/temp /dev/thermal-cpu/temp1_input systemctl restart fancontrol.service systemctl status fancontrol.service Now digging more this issue happen because udev is not creating the symlink like it should for some reason. After reading the rule in /etc/udev/rules.d/90-helios64-hwmon-legacy.rules and a bit of udev documentation, I managed to find how to test it:
root@helios64:~ # udevadm test /sys/devices/virtual/thermal/thermal_zone0 [...] Reading rules file: /etc/udev/rules.d/90-helios64-hwmon-legacy.rules Reading rules file: /etc/udev/rules.d/90-helios64-ups.rules [...] DEVPATH=/devices/virtual/thermal/thermal_zone0 ACTION=add SUBSYSTEM=thermal IS_HELIOS64_HWMON=1 HWMON_PATH=/sys/devices/virtual/thermal/thermal_zone0 USEC_INITIALIZED=7544717 run: '/bin/ln -sf /sys/devices/virtual/thermal/thermal_zone0 ' <-- something is wrong here, there is no target Unload module index Unloaded link configuration context. After spending a bit more time reading the udev rule, I realized that the second argument was empty because we don't match the ATTR{type}=="soc-thermal" condition. We can look up the types like this:
root@helios64:~ # find /sys/ -name type | grep thermal /sys/devices/virtual/thermal/cooling_device1/type /sys/devices/virtual/thermal/thermal_zone0/type /sys/devices/virtual/thermal/cooling_device4/type /sys/devices/virtual/thermal/cooling_device2/type /sys/devices/virtual/thermal/thermal_zone1/type /sys/devices/virtual/thermal/cooling_device0/type /sys/devices/virtual/thermal/cooling_device3/type /sys/firmware/devicetree/base/thermal-zones/gpu/trips/gpu_alert0/type /sys/firmware/devicetree/base/thermal-zones/gpu/trips/gpu_crit/type /sys/firmware/devicetree/base/thermal-zones/cpu/trips/cpu_crit/type /sys/firmware/devicetree/base/thermal-zones/cpu/trips/cpu_alert0/type /sys/firmware/devicetree/base/thermal-zones/cpu/trips/cpu_alert1/type root@helios64:~ # cat /sys/devices/virtual/thermal/thermal_zone0/type cpu <-- we were expecting soc-thermal! and after rewriting the line with the new type, udev is happy again
# Edit in /etc/udev/rules.d/90-helios64-hwmon-legacy.rules and add the following line after the original one ATTR{type}=="cpu", ENV{HWMON_PATH}="/sys%p/temp", ENV{HELIOS64_SYMLINK}="/dev/thermal-cpu/temp1_input", RUN+="/usr/bin/mkdir /dev/thermal-cpu/" root@helios64:~ # udevadm control --reload root@helios64:~ # udevadm test /sys/devices/virtual/thermal/thermal_zone0 [...] DEVPATH=/devices/virtual/thermal/thermal_zone0 ACTION=add SUBSYSTEM=thermal IS_HELIOS64_HWMON=1 HWMON_PATH=/sys/devices/virtual/thermal/thermal_zone0/temp HELIOS64_SYMLINK=/dev/thermal-cpu/temp1_input USEC_INITIALIZED=7544717 run: '/usr/bin/mkdir /dev/thermal-cpu/' run: '/bin/ln -sf /sys/devices/virtual/thermal/thermal_zone0/temp /dev/thermal-cpu/temp1_input' Unload module index Unloaded link configuration context. Apparently for some reason the device-tree changed upstream and the thermal type changed from soc-thermal to cpu?

Well, for anyone else interested in trying this, here's the basic order I did:
stop armbian-ramlog disable armbian-ramlog create a zfs dataset and mount it at /var/log cp -ar everything from /var/log.hdd to the new /var/log modify /etc/logrotate to disable compression (since the dataset is already using compression) modify /etc/default/armbian-ramlog to disable it there as well modify /etc/default/armbian-zram-config to adjust for new numbers (I have ZRAM_PERCENTAGE and MEM_LIMIT_PERCENTAGE at 15). reboot

Sigh. Except that doesn't solve the problem. Now it's just cron filling up the log.
 
New solution, using the sleep option. Modified helio64-ups.service:
[Unit] Description=Helios64 UPS Action [Install] WantedBy=multi-user.target [Service] #Type=oneshot #ExecStart=/usr/bin/helios64-ups.sh Type=simple ExecStart=/usr/local/sbin/powermon.sh  
Modified powermon.sh:
#!/bin/bash #7.0V 916 Recommended threshold to force shutdown system TH=916 # Values can be info, warning, emerg warnlevel="emerg" while [ : ] do main_power=$(cat '/sys/class/power_supply/gpio-charger/online') # Only use for testing: # main_power=0 if [ "$main_power" == 0 ]; then val=$(cat '/sys/bus/iio/devices/iio:device0/in_voltage2_raw') sca=$(cat '/sys/bus/iio/devices/iio:device0/in_voltage_scale') # The division is required to make the test later work. adc=$(echo "$val * $sca /1" | bc) echo "Main power lost. Current charge: $adc" | systemd-cat -p $warnlevel echo "Shutdown at $TH" | systemd-cat -p $warnlevel # Uncomment for testing # echo "Current values:" # echo -e "\tMain Power = $main_power" # echo -e "\tRaw Voltage = $val" # echo -e "\tVoltage Scale = $sca" # echo -e "\tVoltage = $adc" if [ "$adc" -le $TH ]; then echo "Critical power level reached. Powering off." | systemd-cat -p $warnlevel /usr/sbin/poweroff fi fi sleep 20 done  

So, I went hunting for the PR where this got merged, found the reference to kernel module "ledtrig-netdev", noticed the module wasn't loaded on my Helios 64 (no idea why), loaded it via modprobe, restarted the service unit I mentioned in the post above, and now it works.
I've added ledtrig-netdev to /etc/modules-load.d/modules.conf, so now it should hopefully work automagically on every boot.

Yes it works on version 13.0-CURRENT, and there is OpenZFS in it.

Here is the boot log. 
Boot.log
 

 
Welcome to FreeBSD!
 
 

@gprovost Thanks.
Already tested , but nothing gets better.
 
Tested (Crash) : fix cpufreq Max = 1200000 Min = 1200000 governor = performance
Tested (Crash) : fix cpufreq Max = 1008000 Min = 1008000 governor = powersave
Tested (Crash) : All ZRAM disabled ( /etc/default/armbian-zram-config )
 
I took kernel via serial console. (dmesg -E)
"sysctl -w vm.drop_caches=3" ,  cause kernel panic Immediately.
 
root@helios64:~# [50657.866770] Insufficient stack space to handle exception! [50657.866777] ESR: 0x96000047 -- DABT (current EL) [50657.867667] FAR: 0xffff800011d27fd0 [50657.867975] Task stack: [0xffff800011d28000..0xffff800011d2c000] [50657.868531] IRQ stack: [0xffff800011af8000..0xffff800011afc000] [50657.869086] Overflow stack: [0xffff0000f77922b0..0xffff0000f77932b0] [50657.869645] CPU: 4 PID: 30 Comm: ksoftirqd/4 Tainted: G C 5.9.14-rockchip64 #20.11.4 [50657.870435] Hardware name: Helios64 (DT) [50657.870781] pstate: 40000085 (nZcv daIf -PAN -UAO BTYPE=--) [50657.871278] pc : kfree+0x4/0x498 [50657.871563] lr : __free_slab+0xac/0x1a8 [50657.871899] sp : ffff800011d28060 [50657.872191] x29: ffff800011d28060 x28: fffffe00013ea840 [50657.872658] x27: 0000000000100001 x26: ffff000057aa1400 [50657.873124] x25: ffff0000f6c01600 x24: fffffffffffff000 [50657.873590] x23: 0000000000000000 x22: 0000000000000000 [50657.874055] x21: 0000000000000001 x20: ffff0000f6c01600 [50657.874521] x19: fffffe00013ea840 x18: 0000000000000028 [50657.874986] x17: 0000000000000000 x16: 0000000000000000 [50657.875451] x15: ffff0000e7455a78 x14: 00000033b5e86238 [50657.875917] x13: 000000000000034e x12: 000000000000031e [50657.876382] x11: 0000000000000000 x10: 0000000000000001 [50657.876848] x9 : 00000000f0000000 x8 : 0000000000100000 [50657.877313] x7 : ffff0000f6c00ba0 x6 : 0000000000000000 [50657.877778] x5 : 0000000000000000 x4 : 0000000000005df0 [50657.878243] x3 : 0000000000000010 x2 : ffff0000f6f59d00 [50657.878708] x1 : 1ffff00000000000 x0 : ffff000057a9fc00 [50657.879178] Kernel panic - not syncing: kernel stack overflow [50657.879684] CPU: 4 PID: 30 Comm: ksoftirqd/4 Tainted: G C 5.9.14-rockchip64 #20.11.4 [50657.880473] Hardware name: Helios64 (DT) [50657.880819] Call trace: [50657.881041] dump_backtrace+0x0/0x200 [50657.881365] show_stack+0x18/0x28 [50657.881660] dump_stack+0xc0/0x11c [50657.881963] panic+0x164/0x364 [50657.882236] nmi_panic+0x64/0x98 [50657.882520] handle_bad_stack+0x11c/0x148 [50657.882873] __bad_stack+0x90/0x94 [50657.883175] kfree+0x4/0x498 [50657.883429] discard_slab+0x70/0xa0 [50657.883738] __slab_free+0x3a0/0x3e8 [50657.884052] kfree+0x484/0x498 [50657.884320] __free_slab+0xac/0x1a8 [50657.884628] discard_slab+0x70/0xa0 [50657.884936] __slab_free+0x3a0/0x3e8 [50657.885250] kfree+0x484/0x498 [50657.885521] __free_slab+0xac/0x1a8 [50657.885827] discard_slab+0x70/0xa0 [50657.886134] __slab_free+0x3a0/0x3e8 [50657.886448] kfree+0x484/0x498 [50657.886718] __free_slab+0xac/0x1a8 [50657.887026] discard_slab+0x70/0xa0 [50657.887334] __slab_free+0x3a0/0x3e8 [50657.887648] kfree+0x484/0x498 [50657.887918] __free_slab+0xac/0x1a8 [50657.888225] discard_slab+0x70/0xa0 [50657.888531] __slab_free+0x3a0/0x3e8 [50657.888846] kfree+0x484/0x498 [50657.889116] __free_slab+0xac/0x1a8 [50657.889424] discard_slab+0x70/0xa0 [50657.889732] __slab_free+0x3a0/0x3e8 [50657.890047] kfree+0x484/0x498 [50657.890317] __free_slab+0xac/0x1a8 [50657.890625] discard_slab+0x70/0xa0 [50657.890933] __slab_free+0x3a0/0x3e8 [50657.891247] kfree+0x484/0x498 [50657.891517] __free_slab+0xac/0x1a8 [50657.891826] discard_slab+0x70/0xa0 [50657.892132] __slab_free+0x3a0/0x3e8 [50657.892446] kfree+0x484/0x498 [50657.892716] __free_slab+0xac/0x1a8 [50657.893025] discard_slab+0x70/0xa0 [50657.893331] __slab_free+0x3a0/0x3e8 [50657.893646] kfree+0x484/0x498 [50657.893916] __free_slab+0xac/0x1a8 [50657.894222] discard_slab+0x70/0xa0 [50657.894529] __slab_free+0x3a0/0x3e8 [50657.894843] kfree+0x484/0x498 [50657.895113] __free_slab+0xac/0x1a8 [50657.895420] discard_slab+0x70/0xa0 [50657.895728] __slab_free+0x3a0/0x3e8 [50657.896043] kfree+0x484/0x498 [50657.896311] __free_slab+0xac/0x1a8 [50657.896618] discard_slab+0x70/0xa0 [50657.896924] __slab_free+0x3a0/0x3e8 [50657.897239] kfree+0x484/0x498 [50657.897509] __free_slab+0xac/0x1a8 [50657.897815] discard_slab+0x70/0xa0 [50657.898122] __slab_free+0x3a0/0x3e8 [50657.898437] kfree+0x484/0x498 [50657.898707] __free_slab+0xac/0x1a8 [50657.899013] discard_slab+0x70/0xa0 [50657.899320] __slab_free+0x3a0/0x3e8 [50657.899635] kfree+0x484/0x498 [50657.899905] __free_slab+0xac/0x1a8 [50657.900212] discard_slab+0x70/0xa0 [50657.900519] __slab_free+0x3a0/0x3e8 [50657.900834] kfree+0x484/0x498 [50657.901104] __free_slab+0xac/0x1a8 [50657.901411] discard_slab+0x70/0xa0 [50657.901717] __slab_free+0x3a0/0x3e8 [50657.902032] kfree+0x484/0x498 [50657.902302] __free_slab+0xac/0x1a8 [50657.902608] discard_slab+0x70/0xa0 [50657.902915] __slab_free+0x3a0/0x3e8 [50657.903229] kfree+0x484/0x498 [50657.903499] __free_slab+0xac/0x1a8 [50657.903807] discard_slab+0x70/0xa0 [50657.904115] __slab_free+0x3a0/0x3e8 [50657.904429] kfree+0x484/0x498 [50657.904699] __free_slab+0xac/0x1a8 [50657.905008] discard_slab+0x70/0xa0 [50657.905314] __slab_free+0x3a0/0x3e8 [50657.905628] kfree+0x484/0x498 [50657.905899] __free_slab+0xac/0x1a8 [50657.906206] discard_slab+0x70/0xa0 [50657.906513] __slab_free+0x3a0/0x3e8 [50657.906827] kfree+0x484/0x498 [50657.907096] __free_slab+0xac/0x1a8 [50657.907402] discard_slab+0x70/0xa0 [50657.907709] __slab_free+0x3a0/0x3e8 [50657.908023] kfree+0x484/0x498 [50657.908293] __free_slab+0xac/0x1a8 [50657.908600] discard_slab+0x70/0xa0 [50657.908908] __slab_free+0x3a0/0x3e8 [50657.909223] kfree+0x484/0x498 [50657.909493] __free_slab+0xac/0x1a8 [50657.909802] discard_slab+0x70/0xa0 [50657.910110] __slab_free+0x3a0/0x3e8 [50657.910424] kfree+0x484/0x498 [50657.910694] __free_slab+0xac/0x1a8 [50657.911000] discard_slab+0x70/0xa0 [50657.911308] __slab_free+0x3a0/0x3e8 [50657.911622] kfree+0x484/0x498 [50657.911892] __free_slab+0xac/0x1a8 [50657.912200] discard_slab+0x70/0xa0 [50657.912506] __slab_free+0x3a0/0x3e8 [50657.912821] kfree+0x484/0x498 [50657.913091] __free_slab+0xac/0x1a8 [50657.913397] discard_slab+0x70/0xa0 [50657.913704] __slab_free+0x3a0/0x3e8 [50657.914018] kfree+0x484/0x498 [50657.914288] __free_slab+0xac/0x1a8 [50657.914594] discard_slab+0x70/0xa0 [50657.914901] __slab_free+0x3a0/0x3e8 [50657.915215] kfree+0x484/0x498 [50657.915486] __free_slab+0xac/0x1a8 [50657.915792] discard_slab+0x70/0xa0 [50657.916099] __slab_free+0x3a0/0x3e8 [50657.916413] kfree+0x484/0x498 [50657.916684] __free_slab+0xac/0x1a8 [50657.916991] discard_slab+0x70/0xa0 [50657.917299] __slab_free+0x3a0/0x3e8 [50657.917614] kfree+0x484/0x498 [50657.917884] __free_slab+0xac/0x1a8 [50657.918192] discard_slab+0x70/0xa0 [50657.918498] __slab_free+0x3a0/0x3e8 [50657.918812] kfree+0x484/0x498 [50657.919081] __free_slab+0xac/0x1a8 [50657.919388] discard_slab+0x70/0xa0 [50657.919696] __slab_free+0x3a0/0x3e8 [50657.920010] kfree+0x484/0x498 [50657.920280] __free_slab+0xac/0x1a8 [50657.920587] discard_slab+0x70/0xa0 [50657.920893] __slab_free+0x3a0/0x3e8 [50657.921208] kfree+0x484/0x498 [50657.921478] __free_slab+0xac/0x1a8 [50657.921784] discard_slab+0x70/0xa0 [50657.922092] __slab_free+0x3a0/0x3e8 [50657.922406] kfree+0x484/0x498 [50657.922676] __free_slab+0xac/0x1a8 [50657.922983] discard_slab+0x70/0xa0 [50657.923291] __slab_free+0x3a0/0x3e8 [50657.923605] kfree+0x484/0x498 [50657.923875] __free_slab+0xac/0x1a8 [50657.924182] discard_slab+0x70/0xa0 [50657.924489] unfreeze_partials.isra.96+0x1d0/0x250 [50657.924910] put_cpu_partial+0x18c/0x238 [50657.925254] __slab_free+0x2a4/0x3e8 [50657.925569] kfree+0x484/0x498 [50657.925841] __d_free_external+0x20/0x40 [50657.926189] rcu_core+0x23c/0x858 [50657.926482] rcu_core_si+0x10/0x20 [50657.926781] efi_header_end+0x160/0x3f4 [50657.927120] run_ksoftirqd+0x4c/0x60 [50657.927437] smpboot_thread_fn+0x200/0x238 [50657.927799] kthread+0x140/0x150 [50657.928085] ret_from_fork+0x10/0x34 [50657.928406] SMP: stopping secondary CPUs [50657.928767] Kernel Offset: disabled [50657.929076] CPU features: 0x0240022,2000200c [50657.929451] Memory Limit: none [50657.929730] ---[ end Kernel panic - not syncing: kernel stack overflow ]---  
 
 
 

Support told me to do the following:
 
Run `armbian-config`, go to -> System -> CPU
 
And set:
Minimum CPU speed = 1200000
Maximum CPU speed = 1200000
CPU governor = performance
 
I'm now writing to my HDD via SFTP for more than 1.5 days without an issue, so that seemed to solve it.

@NickSYou just need to change a single digit from 0 to 1 - see here.
 
Unfortunately you need to repeat that if /lib/systemd/system-shutdown/disable_auto_poweron is updated by Armbian.

Sure thing! Here it goes:

With the Noctua fans, this config makes the Helios64 silent most of the time while keeping SoC temperatures below 75C and HDD temperatures below 35C.

RK3399 commercial grade is rated as 0 - 80C ambient operating temperature. To not be confused with max Temperature junction (Tj) which is 125C ;-)

Looking into the fancontrol configuration, I think we were way too conservative on the settings making the fan going too early in higher RPM.
Plus we are not taking into consideration that the heatsink is already doing a pretty good job in passive mode.
 
I would recommend the following fancontrol settings until we provide a way to control the fan based on HDD temp instead :
 
# Helios64 PWM Fan Control Configuration # Temp source : /dev/thermal-cpu INTERVAL=10 FCTEMPS=/dev/fan-p6/pwm1=/dev/thermal-cpu/temp1_input /dev/fan-p7/pwm1=/dev/thermal-cpu/temp1_input MINTEMP=/dev/fan-p6/pwm1=40 /dev/fan-p7/pwm1=40 MAXTEMP=/dev/fan-p6/pwm1=110 /dev/fan-p7/pwm1=110 MINSTART=/dev/fan-p6/pwm1=60 /dev/fan-p7/pwm1=60 MINSTOP=/dev/fan-p6/pwm1=40 /dev/fan-p7/pwm1=40 MINPWM=40  

@Jaques-Ludwig As explains by @SIGSEGV you still need to store the bootloader somewhere else than USB because the system cannot boot directly from USB. So in our wiki we present the following approach : Bootloader on eMMC, Root file System on USB.
 
Very soon we will also explain how to store bootloader SPI, but honestly not really major added value than the above approach.
 
 
Technically you need to use jumper P10 to disable eMMC to be sure the system won't boot from it.
However take note the 1st stage bootloader (aka SPL) will always try to first find 2nd stage bootloader (aka U-boot) on microSD first, then eMMC. So it means that even if system boot from eMMC, if there is a microSD card inserted with a 2nd stage bootloader (aka U-Boot), the system will boot the system from microSD card... therefore you don't really need to bother about P10 jumper. Hope I'm not confusing more by trying to explain more :-/