sgjava Posted June 6, 2021 Posted June 6, 2021 Armbianmonitor: http://ix.io/3oWZ Howdy, I'm running Armbian_21.05.1_Odroidxu4_focal_current_5.4.116.img on my XU4 for security cameras. It averages about 60 Mbps 24/7. After a couple days usually one adapter disappears (from ip a and nmtui) and sometimes both (I have a USB 1 Gbps adapter too). I'm not seeing anything in the logs to indicate an error. I monitor this system with Zabbix and I see no unusual activity before this happens. Is there a way to get diagnostics for this event?
lanefu Posted June 7, 2021 Posted June 7, 2021 probably follow dmesg -w -T from serial console or tee into log file
sgjava Posted June 7, 2021 Author Posted June 7, 2021 What should I be looking for in output? dmesg.txt
lanefu Posted June 7, 2021 Posted June 7, 2021 Lol id be looking for less page allocation errors. Hard to tell. Is memory consumption tight?
sgjava Posted June 7, 2021 Author Posted June 7, 2021 Yeah, looks like https://github.com/aler9/rtsp-simple-server might have some issues, but that shouldn't kill the NICs. Memory averages around 1.3G free which is excellent for 5 4K cameras. Nothing obvious sticks out. I have a SSD and 1G Ethernet dongle connected to USB 3. I believe I have a 4A power supply. I have another XU4 with 2 SSDs on USB 3 that works fine, but has less network activity (like 20 Mbps). The weird thing is the heartbeat is till cranking and I have to power cycle to get NICs back. If it makes any difference I have 5 to 7 ffmpeg processes running all the time. Processes average around 150. I didn't tweak any kernel parameters. Anyways, I tried swapping power supplies with another board just for the heck of it.
lanefu Posted June 8, 2021 Posted June 8, 2021 so I did see interrupt stuff regarding the NIC. I don't know exactly what it means.. but look here for inspiration on knobs to turn https://github.com/armbian/build/blob/master/packages/bsp/common/usr/lib/armbian/armbian-hardware-optimization#L145
sgjava Posted June 8, 2021 Author Posted June 8, 2021 6A PSU https://www.hardkernel.com/shop/5v-6a-power-supply-unit/ When you connect an external 2.5inch HDD/SSD to XU4 or using the CloudShell, the bundle 5V/4A power supply is not enough to supply stable power. We strongly recommend to use this 5V/6A PSU to improve the system stability. So maybe my PSU theory is correct. Would also explain where there's really nothing in the logs. Just ordered this guy. No use troubleshooting a software problem until I rule out hardware https://www.amazon.com/gp/product/B07H9X8FHM
sgjava Posted June 11, 2021 Author Posted June 11, 2021 I'll report back once I see the issue resolved. I think a couple days was the maximum time without losing the NICs. 1
sgjava Posted June 27, 2021 Author Posted June 27, 2021 OK, so here's what looks like is happening. Based on the system load eventually the NICs will go offline. I used the 5V8A PSU and saw no difference. What I will most likely do is split 3 4K cams per XU4 or use one for rtsp/mjpeg proxy and one to do detection recording. As a proxy there very little CPU, system load, etc.
Recommended Posts