phidauex

I'll monitor the other thread to see if I can figure anything else out, but the troubleshooting here is exceeding my skillset fast, and I got a good price on an HPE ML30 Gen10 which might replace both my NAS and my virtualization server. I'm limping the Helios64 along OK for sharing media files, but the instability means I'm not successfully running my offsite backups or my TimeMachine backups, which is really putting a time limit on the solution for me. I really want the Helios to work, it is such a cool little machine, but without OEM support I'm running out of ideas.

I think in my various experiments I tried both of those methods - the raw image without updates, and the image updated to 23.8.1. Are you running the install before or after transferring to the eMMC? I did it before, installing directly onto the SD card. Can you tell if the install is hanging due to a crash? Or just hanging but the system can be used normally from another ssh shell?

In the two fresh-reinstalls I've had to do recently, I was able to install OMV successfully using their "install over Debian" method. I wasn't able to get armbian-config installs working, probably just something out of date in the script. First, get the system running stable on the clean Bullseye install, then follow these instructions (naturally skipping the part about trying armbian-config, which you already tried): https://docs.openmediavault.org/en/latest/installation/on_debian.html

This is a great suggestion - I was wondering why everyone's Helios64s were starting to act up seemingly all at once, and the realtime battery is an interesting potential culprit. It is a CR1225, 3V lithium, I'll get a replacement and test mine shortly. I'll also see if I can get my scope hooked up to the power supply and subject it to some load, gradually failing original supplies could be another culprit impacting multiple people. I'm currently running 5.10.63 which I had run for a long time last year with 30 day typical uptimes, but now getting crashes and seg faults at least once a day - really suggests a hardware issue. I'll see if I can take some measurements and bring some answers...

If you setup the serial console to another machine and connect using a terminal emulator (picocom, putty, etc.), you can capture more of the crash log (and in a place where you can copy/paste). Instructions here: https://wiki.kobol.io/helios64/install/first-start/ Likewise, if you can get in long enough to edit the file, changing verbosity to 7 in /boot/armbianEnv.txt can provide more information. What I see on your photo looks similar to a crash log I captured the other day, but I'm not getting very far yet on figuring out what is actually happening.

Ouch, we are clearly suffering from the project having been abandoned for a while... Armbian updates are still coming, but without the mfg support there isn't the kind of checking that would probably be needed normally for reliability. For restoring, I burned the newest Bullseye image from the archive (https://archive.armbian.com/helios64/archive) to an SD card and booted into that environment. Using a USB-C cable to another computer with a serial console open helps a lot. I then used armbian-config to roll back the kernel to 5.15.63-rockchip (System -> Other), then froze armbian updates. From there, mount the emmc drive, and pull as much configuration backup as you can - backing up OMV is vexingly difficult, but save out: /srv/salt/ /srv/pillar/ your samba conf (/etc/samba/smb.conf, I think) Your NFS config (/etc/exports) user files in /home/ and /root/ your /etc/fstab file Once you've captured as much as you can, you can use armbian-config to install to the eMMC again, and reboot into the new clean environment. Reinstall OMV from their instructions for a bare Debian system. In most cases you don't want do just copy/paste the backed up files into the new environment, since OMV prefers to generate this dynamically, but it is helpful as a resource - for instance I opened up the SMB conf file while setting my SMB shares up in the webUI to make sure I was getting the settings right. Then backup your whole drive, the omv-backup plugin with fsarchiver is a good choice.

Update: I appear to have jinxed myself - after poking around in the system and running other updates, I appear to be having instability as well. Somehow in the last few months I've gone from uptimes in the 30-50 days (basically only when I reboot after updates) to daily crashes. I reinstalled my OS, rolled back to 5.15.93-rockchip64, and reduced my CPU a bit to 408 to 1400Mhz, schedutil governor. I also updated my fancontrol settings to let the fans run a little more (/etc/fancontrol), lowered MaxTemp to 90, and raised MinPWM to 40. # Helios64 PWM Fan Control Configuration # Temp source : /dev/thermal-cpu INTERVAL=10 FCTEMPS=/dev/fan-p6/pwm1=/dev/thermal-cpu/temp1_input /dev/fan-p7/pwm1=/dev/thermal-cpu/temp1_input MINTEMP=/dev/fan-p6/pwm1=40 /dev/fan-p7/pwm1=40 MAXTEMP=/dev/fan-p6/pwm1=90 /dev/fan-p7/pwm1=90 MINSTART=/dev/fan-p6/pwm1=60 /dev/fan-p7/pwm1=60 MINSTOP=/dev/fan-p6/pwm1=40 /dev/fan-p7/pwm1=40 MINPWM=40 Another possibility is that about a month ago I added a fourth drive, and that is when it looks like my uptimes start to get worse. That would lend some support to the power supply theory. EDIT: Just crashed on a snapraid scrub, with the following kernel errors: Message from syslogd@helios64 at Sep 26 11:36:37 ... kernel:[ 6347.577469] Internal error: Oops: 96000044 [#1] PREEMPT SMP Message from syslogd@helios64 at Sep 26 11:36:37 ... kernel:[ 6347.604243] Code: 12800015 17fffff2 94390139 d503233f (a9be7bfd) I might try disconnecting my 4th drive and see how things go.

I just wanted to add some information here in case it helps. I'm running OMV6 stable on my Helios64 - config below, and note one caveat from the last kernel update: Kernel 6.1.50-current-rockchip64 (NOTE: Staying on 6.1.36 might be advised, network issues, see below) Armbian 23.8.1 Bullseye (clean install to Bullseye, sdcard then migrated to emmc) OMV 6.9.0-1 (Shaitan) Governor set to 408Mhz to 1800Mhz, schedutil governor (I don't remember why I selected this vs. ondemand, somehow I had the impression that it was better for ARM systems) 4 drives, WD Red 5400, 3x12TB, 1x4TB I run three drives in a MergerFS pool w/ SnapRaid on a regular run, fourth on a separate share. Offsite backups to Borgbase, run weekly. The NAS gets moderate usage - it holds media files, Proxmox VM backups, and 2 TimeMachine shares. No other random software or services, just OMV. My system is stable, no unexpected crashes or issues when running the python stress test above, or my Snapraid syncs. NOTE on the network issue with this kernel is that the most recent update had the effect of making the 2.5GB network interface (eth1, r8152 driver) unstable. It will run for a while, then randomly disappear. I only use it as a secondary path to my other server, so I was able to work around it fairly easily, but this would be a blocking issue if you relied on that. I had to build a kernel with an earlier version of the r8152 driver on another x86 machine with recent 6.1.x kernels, so I'm thinking it is the same/similar issue, but I haven't rolled back or done any serious troubleshooting yet. If you want to move to the 6.1.x branch, I'd recommend a good backup, and sticking with 6.1.36 for the moment. EDIT: The issue seems to be the same as this user's, related to Bookworm 23.08.1. I'll try their recommended solution of moving up the r8152 driver and see how it goes. I will also freeze updates for the moment if it works (armbian-config -> System -> Freeze).