Schroedingers Cat

No, the storage is fine - Armbian 21.08.1 is yet another update that hoses something on the Helios64. Last time it was the fans, the time before that the 2.5Gb Ethernet port (twice), and this time it's the eMMC.
 
You'll need to follow the instructions upthread to downgrade the kernel to get the eMMC working again. Then cross your fingers it gets fixed at some point in the future.

It worked, I have my eMMC system back!
 
Here are all the steps in case someone else has the problem and wants to repair his eMMC image...
 
1. download a previous Helios64 image from here...
https://wiki.kobol.io/download/
2. install the image on a microSD e.g. for Windows and macOS there is Etcher...
https://www.balena.io/etcher/
3a. boot your Helios64 with the microSD and Jumper 10, see here...
https://wiki.kobol.io/helios64/troubleshoot/#how-to-force-boot-from-microsd
3b. Alternative: from the serial console press a key on the keyboard while u-boot start. You will get the u-boot prompt.
From this prompt write and press enter. Helios64 will boot from SD... (thx @prahal)
run bootcmd_mmc1  
4. if you are logged in as root user (normal users must put a sudo in front of the commands) execute the following commands...
root@helios64:~# mkdir -p /mnt/system root@helios64:~# mount /dev/mmcblk2p1 /mnt/system root@helios64:~# cd /mnt/system/
5. The contents of the /mnt/system directory should now be filled with the contents of your eMMC....
root@helios64:/mnt/system# ll total 80 lrwxrwxrwx 1 root root 7 Aug 30 2020 bin -> usr/bin drwxr-xr-x 3 root root 4096 Sep 1 03:01 boot drwxr-xr-x 2 root root 4096 Oct 15 2020 dev drwxr-xr-x 110 root root 12288 Sep 1 03:01 etc drwxr-xr-x 2 root root 4096 Sep 22 2020 export drwxr-xr-x 5 root root 4096 Jun 10 04:12 home lrwxrwxrwx 1 root root 7 Aug 30 2020 lib -> usr/lib drwx------ 5 root root 4096 Oct 5 2020 lost+found drwxr-xr-x 4 root root 4096 Oct 15 2020 media drwxr-xr-x 2 root root 4096 Feb 9 2021 mnt drwxr-xr-x 4 root root 4096 Dec 2 2020 opt dr-xr-xr-x 2 root root 4096 Oct 15 2020 proc drwx------ 7 root root 4096 Aug 31 23:36 root drwxr-xr-x 2 root root 4096 Oct 15 2020 run lrwxrwxrwx 1 root root 8 Aug 30 2020 sbin -> usr/sbin drwxrwxr-x 2 root root 4096 Oct 5 2020 selinux drwxr-xr-x 6 root root 4096 Jun 10 03:18 srv dr-xr-xr-x 2 root root 4096 Oct 15 2020 sys lrwxrwxrwx 1 root root 42 Sep 1 03:01 thermal_zone0 -> /sys/devices/virtual/thermal/thermal_zone0 drwxrwxrwt 2 root root 4096 Oct 15 2020 tmp drwxr-xr-x 12 root root 4096 Nov 27 2020 usr drwxr-xr-x 14 root root 4096 Oct 6 2020 var  
6. now download the 3 old packages...
root@helios64:/mnt/system# wget http://armbian.hosthatch.com/apt/pool/main/l/linux-5.10.43-rockchip64/linux-dtb-current-rockchip64_21.05.4_arm64.deb root@helios64:/mnt/system# wget http://armbian.hosthatch.com/apt/pool/main/l/linux-5.10.43-rockchip64/linux-headers-current-rockchip64_21.05.4_arm64.deb root@helios64:/mnt/system# wget http://armbian.hosthatch.com/apt/pool/main/l/linux-5.10.43-rockchip64/linux-image-current-rockchip64_21.05.4_arm64.deb  
7. now changes the root directory...
root@helios64:/mnt/system# chroot /mnt/system root@helios64:/# pwd / root@helios64:/# ll total 50996 lrwxrwxrwx 1 root root 7 Aug 30 2020 bin -> usr/bin drwxr-xr-x 3 root root 4096 Sep 1 14:56 boot drwxr-xr-x 2 root root 4096 Sep 1 16:15 dev drwxr-xr-x 110 root root 12288 Sep 1 03:01 etc drwxr-xr-x 2 root root 4096 Sep 22 2020 export drwxr-xr-x 5 root root 4096 Jun 10 04:12 home lrwxrwxrwx 1 root root 7 Aug 30 2020 lib -> usr/lib -rw-r--r-- 1 root root 314304 Jul 8 19:32 linux-dtb-current-rockchip64_21.05.4_arm64.deb -rw-r--r-- 1 root root 11527696 Jul 8 19:32 linux-headers-current-rockchip64_21.05.4_arm64.deb -rw-r--r-- 1 root root 40290884 Jul 8 19:33 linux-image-current-rockchip64_21.05.4_arm64.deb drwx------ 5 root root 4096 Oct 5 2020 lost+found drwxr-xr-x 4 root root 4096 Oct 15 2020 media drwxr-xr-x 2 root root 4096 Feb 9 2021 mnt drwxr-xr-x 4 root root 4096 Dec 2 2020 opt dr-xr-xr-x 2 root root 4096 Oct 15 2020 proc drwx------ 7 root root 4096 Aug 31 23:36 root drwxr-xr-x 2 root root 4096 Oct 15 2020 run lrwxrwxrwx 1 root root 8 Aug 30 2020 sbin -> usr/sbin drwxrwxr-x 2 root root 4096 Oct 5 2020 selinux drwxr-xr-x 6 root root 4096 Jun 10 03:18 srv dr-xr-xr-x 2 root root 4096 Oct 15 2020 sys lrwxrwxrwx 1 root root 42 Sep 1 03:01 thermal_zone0 -> /sys/devices/virtual/thermal/thermal_zone0 drwxrwxrwt 2 root root 4096 Oct 15 2020 tmp drwxr-xr-x 12 root root 4096 Nov 27 2020 usr drwxr-xr-x 14 root root 4096 Oct 6 2020 var  
8. now installs the downloaded packages...
root@helios64:/# dpkg -i *.deb dpkg: warning: downgrading linux-dtb-current-rockchip64 from 21.08.1 to 21.05.4 (Reading database ... 62558 files and directories currently installed.) Preparing to unpack linux-dtb-current-rockchip64_21.05.4_arm64.deb ... Unpacking linux-dtb-current-rockchip64 (21.05.4) over (21.08.1) ... Selecting previously unselected package linux-headers-current-rockchip64. Preparing to unpack linux-headers-current-rockchip64_21.05.4_arm64.deb ... Unpacking linux-headers-current-rockchip64 (21.05.4) ... dpkg: warning: downgrading linux-image-current-rockchip64 from 21.08.1 to 21.05.4 Preparing to unpack linux-image-current-rockchip64_21.05.4_arm64.deb ... update-initramfs: Deleting /boot/initrd.img-5.10.60-rockchip64 Removing obsolete file uInitrd-5.10.60-rockchip64 stat: cannot stat '/proc/1/root/.': No such file or directory Unpacking linux-image-current-rockchip64 (21.05.4) over (21.08.1) ... Setting up linux-dtb-current-rockchip64 (21.05.4) ... Setting up linux-headers-current-rockchip64 (21.05.4) ... Compiling headers - please wait ... grep: /proc/cpuinfo: No such file or directory grep: /proc/cpuinfo: No such file or directory Setting up linux-image-current-rockchip64 (21.05.4) ... update-initramfs: Generating /boot/initrd.img-5.10.43-rockchip64 W: Couldn't identify type of root file system for fsck hook update-initramfs: Converting to u-boot format  
9. reset the root change with exit and restart your helios64 with reboot...
root@helios64:/# exit exit root@helios64:/mnt/system# reboot  
10. after a reboot you should now boot from the eMMC again...
root@192.168.180.5's password: _ _ _ _ __ _ _ | | | | ___| (_) ___ ___ / /_ | || | | |_| |/ _ \ | |/ _ \/ __| '_ \| || |_ | _ | __/ | | (_) \__ \ (_) |__ _| |_| |_|\___|_|_|\___/|___/\___/ |_| Welcome to Armbian 21.08.1 Buster with Linux 5.10.43-rockchip64 No end-user support: built from trunk System load: 74% Up time: 3 min Memory usage: 34% of 3.77G IP: 172.18.0.1 172.19.0.1 172.20.0.1 172.17.0.1 192.168.180.5 CPU temp: 66°C Usage of /: 87% of 15G [ 0 security updates available, 3 updates total: apt upgrade ] Last check: 2021-09-01 16:38 [ General system configuration (beta): armbian-config ] Last login: Wed Sep 1 16:26:23 2021 root@helios64:~# root@helios64:~# uname -a Linux helios64 5.10.43-rockchip64 #21.05.4 SMP PREEMPT Wed Jun 16 08:02:12 UTC 2021 aarch64 GNU/Linux root@helios64:~#  
11. if the eMMC was used you can see with mount...
root@helios64:~# mount | grep /dev/mmc /dev/mmcblk2p1 on / type ext4 (rw,noatime,nodiratime,errors=remount-ro,commit=600) /dev/mmcblk2p1 on /var/folder2ram/var/log type ext4 (rw,noatime,nodiratime,errors=remount-ro,commit=600) /dev/mmcblk2p1 on /var/folder2ram/var/tmp type ext4 (rw,noatime,nodiratime,errors=remount-ro,commit=600) /dev/mmcblk2p1 on /var/folder2ram/var/lib/openmediavault/rrd type ext4 (rw,noatime,nodiratime,errors=remount-ro,commit=600) /dev/mmcblk2p1 on /var/folder2ram/var/spool type ext4 (rw,noatime,nodiratime,errors=remount-ro,commit=600) /dev/mmcblk2p1 on /var/folder2ram/var/lib/rrdcached type ext4 (rw,noatime,nodiratime,errors=remount-ro,commit=600) /dev/mmcblk2p1 on /var/folder2ram/var/lib/monit type ext4 (rw,noatime,nodiratime,errors=remount-ro,commit=600) /dev/mmcblk2p1 on /var/folder2ram/var/cache/samba type ext4 (rw,noatime,nodiratime,errors=remount-ro,commit=600)  

Support told me to do the following:
 
Run `armbian-config`, go to -> System -> CPU
 
And set:
Minimum CPU speed = 1200000
Maximum CPU speed = 1200000
CPU governor = performance
 
I'm now writing to my HDD via SFTP for more than 1.5 days without an issue, so that seemed to solve it.

Setup: Helios64 with Armbian 20.08.8 Buster with Linux 4.4.213-rk3399. Will change to Linux 5.8 soon though. 5 x 8 TB drives. Not using battery.
 
I’ve had this issue ever since beginning to use my Helios. It first happened when I tried building a RAID with OpenMediaVault: It would crash and reboot while building. Sometimes after only 10 minutes, sometimes after 8+ hours of working. The same is true for video transcoding nowadays: it will sometimes crash after a little while, sometimes after a few consecutive hours of working, but it will never be able to work much longer than 12 hours on such a task. Copying large files from the internet has a similar risk of crashing it.
 
Also, it will not save the crash in the logs. For some reason, the timeframe when it crashed was always gone from the logs. Nowadays it’s even worse: My logs only go until December 11th and even if I clear them on the OMV interface, after the next reboot, they are there again and it refuses to log anything new.
 
While the server is performing a demanding task, most figures in the ssh screen will be red: System Load at around 170%, CPU temp at 70°C etc. CPU usage on the OMV interface will be around 97%.
 
At first I thought it was normal but a friend of mine told me servers should absolutely never reboot on their own; that this is an indication something is not right.
 
My impression from the above-described behavior is that somehow my machine isn’t able to limit the amount of CPU used. I expected the server to become slower under more CPU stress, but instead it seems to not regulate well at all and overwork itself. Or maybe some part of the software/hardware is faulty and will randomly cause a crash.
 
As you may have noticed, I’m rather new to a lot of things here. I have no idea how to even begin troubleshooting this so I’ll need some pointers from you. What could be the cause for this and what tests can I do to narrow it down?

Thanks for your response. I'm only allowed to answer after 24 hours, for some reason.
 
I disabled the log2ram but the log files are still not showing anything interesting. Crash happened on 15th of December around 16:00 and restart was around 19:50.
Here's the relevant section from /var/log/syslog:
 
/var/log/kern.log:
 
Nothing interesting found in /var/log/dmesg.
 
Do you mean connecting via COM/USB? I lose connection if it happens, but I can try.
 
 
Anything else I can do to investigate this? Do you think my device is broken?

Thanks for your response. I'm only allowed to answer after 24 hours, for some reason.
 
I disabled the log2ram but the log files are still not showing anything interesting. Crash happened on 15th of December around 16:00 and restart was around 19:50.
Here's the relevant section from /var/log/syslog:
 
/var/log/kern.log:
 
Nothing interesting found in /var/log/dmesg.
 
Do you mean connecting via COM/USB? I lose connection if it happens, but I can try.
 
 
Anything else I can do to investigate this? Do you think my device is broken?

I'm using Armbian 20.11.1 Focal (5.9.11-rockchip64) on my Helios64. When there's heavier load via ethernet (SFTP to a SATA Ultrastar 12TB disk) for something like more than an hour, the entire system will freeze. Not sure if it actually does freeze but I lose all SSH connections and cannot connect anymore via SSH unless I restart the Helios64. Happened today 4 times and it's totally reproducible. Haven't tested another drive yet.
 
What makes this difficult to debug is that under /var/log/syslog or kern.log or faillog nothing relevant is being written as to in what state the device is.
 
Is this a known issue?
How can I find out what the problem is?