OPI PC unreachable via SSH/UART after a few weeks, but all services run properly

[dando]

Hi,

 

Today I encountered for the second time a strange behavior on my Orange Pi PC running Armbian 5.38 stable Debian GNU/Linux 9 (stretch) 4.14.18-sunxi and decided to ask here.

 

The OPI PC is working properly for a few weeks (one to three, typically) and suddenly I cannot access it via SSH or UART anymore.

When trying to connect with SSH, the connection hangs after entering the password and after a while the connection timeouts. Tried SSHing from regular Linux shell and from PuTTY under Windows. I regularly use a key to log in the Opi PC, but it doesn't work, too.

Trying to connect with the UART (via minicom in Linux or PuTTY in Windows) produces a line or two of console output and then stops responding at all. Reconnecting doesn't help as nothing is displayed anymore. I tried with another USB<->UART converter, same result. I haven't tried to change the cable (~80cm long) because the connection works properly once I reboot the Pi.

 

I use the Pi as a small web and file server. The strange thing is that all services seem to work properly while I cannot reach the Pi via SSH/UART.

 

The uSD card is a Samsung Evo+ 32GB, the PSU is 5V/2A and seems to work correctly, at least when running stress -c 4 or operating under high load. The maximum temps during stress test are all under 45°C (I installed a heat sink + fan).

 

Has anyone exhibited similar behavior? Is there a log which can point the root cause out? How can I log any errors in the future for debugging purposes?

 

Thank you in advance!

[ag123]

check the logs e.g. /var/log/messages if there are out of memory conditions etc. or if there are any messages related to some problems

[dando]

Hi,

 

Thanks for the suggestion!

There was an OOM condition three days before I couldn't connect to the Pi. After the OOM, I was able to connect via SSH at least twice. I wonder whether it can be related to the 128MB swap partition being full as it is almost always fully occupied.

[ag123]

well, among the things when you can't do ssh, did you try to ping the board address? if ping did not respond, most likely the board hung.

if you have apps / services running on the board for long periods, some conditions that cause a memory leak can consume memory till it runs out

 

you could do things like

$ps auxk "-%mem"

which would list the processes sorted by memory

if you only want the first few rows say 10 rows you could do things like

$ps auxk "-%mem" | head -10

and hence you could put that in a script and run that to log the sessions to a file for analysis. if you stagger out the runs e.g. once an hour, you may like to run it from cron to do the logs etc

 

i think there may well be some tools that could do this easier/better e.g. utilities to catch possible memory leaks and log that to syslog

it'd take some google searches etc to find them

 

[ag123]

more to the topic about that swapfile as root

run  /sbin/sysctl -a |grep vm.swappiness

and here is a useful wikipedia entry about it

https://en.wikipedia.org/wiki/Swappiness

 

i think you can set vm.swappiness to 1, or some small numbers

that is normally done by entering a line in /etc/sysctl.conf

vm.swappiness=n

where n is your desired value

 

do also check the following:

run /sbin/swapon and see if zram is configured

in /etc/default/armbian-zram-config

there may be some parameters used by /usr/lib/armbian/armbian-zram-config which runs at boot if zram is in use

 

i'd guess u'd need to tune the vm.swapiness as otherwise over time the on flash swap may be filled

-----

on another note, i think zram should be setup mainly in the nightly builds (i'm not sure if the stable images has zram enabled currently), zram uses compressed ram as swap and runs much faster than the on flash swap

but that would be in the nightly builds, hence you may like to use a different sd card (e.g. a clone) to do that switch

and the procedure to get there is

apt-get upgrade armbian-config

then run armbian-config and switch to nightly