System service failures on ODROID HC4 with Armbian_community 24.8.0-trunk.501

[WondeRien]

Hello Armbian Community,
 

I'm reaching out for help with a critical issue I've been facing on my ODROID HC4 server since updating to Armbian_community 24.8.0-trunk.4xx.

 

I put 4xx because I don't know which version caused the issues. It was yesterday and I just saw updates in my OMV workbench, did the updates and then rebooted and then was unable to access OMV anymore.

I have since upgraded to trunk.500 last night, didn't help, and trunk.501 this morning also didn't fix anything.

Here’s a detailed account of the situation and the troubleshooting steps I've taken so far.

 

Background:

 

Device: ODROID HC4

OS: Armbian (latest updates applied)

Purpose: NAS with OpenMediaVault, and Tailscale configured as a subnet router.

 

Loss of SSH Access:
 

I initially had SSH access to the server, but after executing sudo tailscale down --accept-risk=lose-ssh and rebooting, I completely lost SSH access.

 

Broken openssh-server Installation

 

I attempted to reinstall openssh-server, but the installation failed with the message: dpkg: error processing package openssh-server (--configure): installed openssh-server package post-installation script subprocess returned error exit status 1.

Attempts to fix broken packages with apt-get install -f showed no new issues, but dpkg-reconfigure also fails with errors.

 

System Services Failures:

 

Upon trying to resolve the issue, I encountered multiple service failures.

Key services such as nginx, openmediavault, and collectd are not starting, and their status shows errors related to failed starts and missing units.

For example, the nginx service fails with bind() to 0.0.0.0:443 failed (98: Address already in use), and the openmediavault-engined service is not found.

 

When I open journalctl -xe, I see a lot of errors. the logs seem to go up forever.

First red I see is from odroidhc4 monit [1751]: 'rrdcached' process is not running. Right above that I see in green letters: The unit packagekit.service has successfully entered the 'dead' state. A bit further above that another red one from odroidhc4 monit[1751]: 'collectd' failed to start (exit status 5) -- '/bin/systemctl start collectd': Failed to start colledtd.service: Unit collectd.service not found. Above that some more gray stuff and then the next red one is again about collectd process is not running. Right above that also from odroidhc4 monit[1751]: 'omv-engined' failed to start (exit status 5) -- 'bin/systemctl start openmediavault-engined': Failed to start openmediavault-engined.service: Unit openmediavault-engined.service not found. Next red one up is 'omv-engined' process is not running, and right above that 'nginx' failed to start (exit status 5) -- '/bin/systemctl start nginx': Failed to start nginx.service: Unit nginx.service not found. Then the same thing with php-fpm, 'rrdcached', 'collectd', and keeps on going, but seems to repeat the same errors about rrdcached process is not running and collectd failed to start, omvengined, nginx, php-fpm, rrdcached again

 

Steps Taken

 

Checked Network Configuration:

Verified network interface settings with ip a and /etc/netplan/*.yaml. The static IP seems correct, and no Tailscale-related entries appear in the output of ip a or lsof -i :443.

 

Reviewed System Logs:

Errors related to monit, collectd, nginx, openmediavault-engined, and others were observed in journalctl -xe.

Key errors include failures to start these services and issues with rrdcached and collectd.

 

Package Management:

Tried sudo apt-get install -f, which showed no issues, but dpkg -l | grep -i broken returned no results.

 

Request for Assistance

 

SSH Server Reinstallation:

I need guidance on reinstalling or fixing openssh-server given the current errors.

 

Service Restoration:

Advice on restoring services like nginx, openmediavault, and collectd, including fixing missing or broken units.

 

Filesystem and Package Repairs:

Recommendations for handling any remaining package issues.

 

Any additional troubleshooting steps or insights based on the provided info would be helpful!

I appreciate any help or suggestions you can provide.

Thank you!

 

 

[WondeRien]

Should I just reinstall an older trunk, or how do I proceed?
Any pointers would be great

Kinda stuck, besides the ability of just starting from scratch.
But would prefer to recover the system, if possible? 

[WondeRien]

Small win, I was able to regain SSH access  

Edit: Aaand just regained access to OMV, at last.
 

Edited August 7 by WondeRien