Nanopi r2s freezing

[BigEd]

Hi,

 

Have the r2s for some time and it worked well in the beginning. But then it started to freeze randomly. The logs do not have any error in it. Initially I thought it could be the interface drive problem, changed the interface and still freezes.

When the r2s is frozen, serial port won't respond. Keeping serial connection and restarting the r2s, boot messages show normally. When I keep the serial connected, r2s do not freeze (tried for more than a week). Found this other post, not sure if this is the same problem as it does boot, but seems related.

mUSB serial connection rarely works. Have to boot some times until it works.

Although I am an experienced linux user, I am not experienced with armbian, mUSB connection and related kernel modules, etc.

Does anyone has suggestions on how to troubleshoot/fix this issue? It has been really frustrating. I want to build some services in it but if it does not stays on, it will bring more trouble than solution.

[BigEd]

Forgot to tell that I tried ubuntu 20 and 22 and the problem is the same. Now I am using the latest available image.

[Igor]

6 hours ago, BigEd said:

I am not experienced with armbian

https://docs.armbian.com/#what-is-armbian  Armbian is closer to Embedded Linux then Debian / Ubuntu. From embedded Linux perspective, "Debian" or "Ubuntu", upper levels, user space, different versions of standard libraries and application, is irrelevant. Something that has absolutely no relationship with problems most people face. Like to issue you have. General Linux Experience, like you probably already figured out, doesn't help to fix the problem.

 

There are a few of reasons why this instability happens and all require investigation, usually into boot loader ("bios") or related kernel functions. It can hang because of not well tuned power management, badly written network driver, hw quality ... diagnostic can take an hour, but can stretch to weeks. If you have experiences and know where to look. I would start with an easy solutions, which doesn't take much time. 1/2 hour to build, then testing.

[BigEd]

Thank you Igor. Will take a look at the links you shared. I noticed a pattern that device is now freezing after 4 or 5 days online. I power the device using a USB hub that have 2 powered ports but I will try to power it with a power adapter. Although I use a very good no-break, maybe it could be a problem with the hub itself.

[BigEd]

I apologize for the long post but have no idea on how to proceed:

 

I loaded version Armbian_21.02.1_Nanopi-r2s_focal_current_5.10.12_minimal.img.xz, to check if older kernel would work.

- After flashing the sdcard, boot works well, it asks for root password, new user and localization information.

- When I run "reboot", the reboot freezes most of the times at:

Quote

         Starting Raise network interfaces...
[  OK  ] Finished Raise network interfaces.
[  OK  ] Found device /dev/ttyS2.
[  OK  ] Finished Load/Save Random Seed.

Sometimes, if I remove the network cable and then reconnect it while the device is powered down, then start it,  it then freezes on the ttyS2 line, sometimes freezes a few lines after (Finished Armbian ZRAM config).

 

- When I removed the network cable and started the device, then boot worked fine:

Quote

[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Finished Update UTMP about System Runlevel Changes.
[  OK  ] Started Hostname Service.

Armbian 21.02.1 Focal ttyS2

nanopi-r2s login:

 

- Plugging the network cable, device still works (not sure how long it will take to freeze), interface gets IP.

- Rebooting again the device (now with network cable plugged in), boot works fine.

- I kept rebooting the device for 3 or 4 times with network cable plugged in, and then it froze again:

Quote

[  OK  ] Reached target Basic System.
[  OK  ] Started Regular background program processing daemon.
[  OK  ] Started D-Bus System Message Bus.
         Starting Network Manager...
[  OK  ] Started Save initial kernel messages after boot.
         Starting Remove Stale Onli…t4 Metadata Check Snapshots...
         Starting LSB: Load kernel …d to enable cpufreq scaling...
         Starting System Logging Service...
         Starting Resets System Activity Data Collector...
         Starting Login Service...
         Starting WPA supplicant...
[  OK  ] Finished Resets System Activity Data Collector.
[  OK  ] Finished Remove Stale Onli…ext4 Metadata Check Snapshots.
[  OK  ] Started System Logging Service.

 

- removing and plugin in again power, it boots fine. But I believe it is somewhat random. The only configuration I did not experience the system hanging is without network.

- Analyzed boot sequence when device works and when it does not work and the boot sequence is absolutely the same, no additional error messages.

- Error messages shown on boot (attached file has for full log):

Quote

Failed to load '/boot/dtb/rockchip/overlay/-fixup.scr'^M

Quote

atags_set_bootdev: ret:(0)^M
GPT 0x337a9f0 signature is wrong^M
recovery gpt...^M
GPT 0x337a9f0 signature is wrong^M
recovery gpt fail!^M
LoadTrust Addr:0x4000^M
No find bl30.bin^M
No find bl32.bin^M

Quote

INFO:    ARM GICv2 driver initialized^M
INFO:    Using opteed sec cpu_context!^M
INFO:    boot cpu mask: 1^M
INFO:    plat_rockchip_pmu_init: pd status 0xe^M
INFO:    BL31: Initializing runtime services^M
WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK^M
ERROR:   Error initializing runtime service opteed_fast^M

 

 

Any idea of what can I do next? The only reproducible steps are the first 4 steps I listed. After that it can or cannot work (when network cable is plugged).

 

r2s_boot.txt

[BigEd]

I didn't install anything on top of vanilla image but it stayed up for a week. Now I upgraded it using apt upgrade and even without network connected it froze. After a couple of restarts, then it went up. But so far it does not goes up with network connected, I need to connect after the devices boot.

 

Does anyone that owns the R2S are seeing similar behavior?

[BigEd]

Today tried friendlyelec linux and it fails too when network is attached. Strangely, it works with OpenWrt.

 

Also increased boot verbosity and added an extra parameter (initcall_debug) to boot command line, but strangely, when the boot freezes, the kern.log is not updated (or recreated if I delete it). I was not able to get any useful information like exactly where it is failing.

 

Can this be a hardware error? Is it possible to update the firmware shipped with the device?