XU4 loses network connections over time

[sgjava]

Armbianmonitor:

http://ix.io/3oWZ

Howdy, I'm running Armbian_21.05.1_Odroidxu4_focal_current_5.4.116.img on my XU4 for security cameras. It averages about 60 Mbps 24/7. After a couple days usually one adapter disappears (from ip a and nmtui) and sometimes both (I have a USB 1 Gbps adapter too). I'm not seeing anything in the logs to indicate an error. I monitor this system with Zabbix and I see no unusual activity before this happens. Is there a way to get diagnostics for this event?  

[lanefu]

probably follow 

dmesg -w -T

 

from serial console or tee into log file

[sgjava]

What should I be looking for in output?

dmesg.txt

[lanefu]

Lol id be looking for less page allocation errors.    Hard to tell.   Is memory consumption tight?

[sgjava]

Yeah, looks like https://github.com/aler9/rtsp-simple-server might have some issues, but that shouldn't kill the NICs. Memory averages around 1.3G free which is excellent for 5 4K cameras. Nothing obvious sticks out. I have a SSD and 1G Ethernet dongle connected to USB 3. I believe I have a 4A power supply. I have another XU4 with 2 SSDs on USB 3 that works fine, but has less network activity (like 20 Mbps). The weird thing is the heartbeat is till cranking and I have to power cycle to get NICs back.

 

If it makes any difference I have 5 to 7 ffmpeg processes running all the time. Processes average around 150. I didn't tweak any kernel parameters. Anyways, I tried swapping power supplies with another board just for the heck of it.

 

[lanefu]

so I did see interrupt stuff regarding the NIC.

I don't know exactly what it means.. but look here for inspiration on knobs to turn

https://github.com/armbian/build/blob/master/packages/bsp/common/usr/lib/armbian/armbian-hardware-optimization#L145

[sgjava]

6A PSU https://www.hardkernel.com/shop/5v-6a-power-supply-unit/

 

When you connect an external 2.5inch HDD/SSD to XU4 or using the CloudShell,
the bundle 5V/4A power supply is not enough to supply stable power.
We strongly recommend to use this 5V/6A PSU to improve the system stability.

 

So maybe my PSU theory is correct. Would also explain where there's really nothing in the logs.

 

Just ordered this guy. No use troubleshooting a software problem until I rule out hardware  https://www.amazon.com/gp/product/B07H9X8FHM

[lanefu]

Haha yeah that should cover it!

[sgjava]

I'll report back once I see the issue resolved. I think a couple days was the maximum time without losing the NICs.

[sgjava]

OK, so here's what looks like is happening. Based on the system load eventually the NICs will go offline. I used the 5V8A PSU and saw no difference. What I will most likely do is split 3 4K cams per XU4 or use one for rtsp/mjpeg proxy and one to do detection recording. As a proxy there very little CPU, system load, etc.