No network connection after update

[Joel]

Hi, last night a set of updates was installed and I rebooted the Orange Pi 5+ this morning. After the reboot I've seem to have lost the connection to the Orange Pi 5+, it is normally connected without a screen so I access it through SSH and the Open Media Vault web GUI.

 

I've tried to connect it to a screen and then it seems to boot up just fine. Looking around with the screen connected it seems like the network cards aren't detected. My Linux skills are very limited but "sudo lshw -class network" shows nothing.

 

One detail is that my router seems to detect it when it's connected and gives it the normal IP address that I've set in the 192.168.0.x range. But when I list network devices logged in to Linux all interfaces seems to have private addresses starting with 172. See the attached image for the output from  "ip addr show".

I’m running Armbian Debian Bookworm.

 

These were the updates that were installed:

 

        armbian-config 24.8.4
        armbian-firmware 24.8.4
        armbian-zsh 24.8.4
        linux-dtb-vendor-rk35xx 24.8.4
        linux-image-vendor-rk35xx 24.8.4

Edited October 19 by Joel

[ozacas]

try to see what happened during kernel boot by searching through the messages:
 

acas@opi2:~$ sudo dmesg | grep eth
[    0.000000] psci: probing for conduit method from ACPI.
[    3.410292] r8169 0003:01:00.0 eth0: RTL8125B, c0:74:2b:fe:78:ab, XID 641, IRQ 23
[    3.410304] r8169 0003:01:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko]
[    3.433464] r8169 0004:01:00.0 eth1: RTL8125B, c0:74:2b:fe:78:aa, XID 641, IRQ 24
[    3.433479] r8169 0004:01:00.0 eth1: jumbo features [frames: 9194 bytes, tx checksumming: ko]
[    3.451419] r8169 0004:01:00.0 enP4p1s0: renamed from eth1
[    3.452652] r8169 0003:01:00.0 enP3p1s0: renamed from eth0

and if that looks ok, then likely a higher level problem failed to configure the networking

 

systemctl list-units --state=failed

might give you clues as to what failed to work during startup.

[Joel]

Ok, here's the output from those commands. I guess I should see something about RTL8125B there? I tried to search through the dmesg output after 8125 with grep but nothing came up.

 

I have also tried to boot from an old SD card with Armbian 24.8.1 and there the network works fine. So there is nothing wrong with the hardware or network.

[ozacas]

yep, so networkd couldnt start properly due to interfaces not being present. Hardware isnt being detected (unlikely as you say) or more likely driver is not there to drive them. Lets test theory #2 -

 

acas@opi2:~$ zcat /proc/config.gz | grep 8169
CONFIG_R8169=m
CONFIG_R8169_LEDS=y
acas@opi2:~$ lsmod | grep 8169
r8169                 106496  0

the running kernel should have an appropriate driver configured into it and that driver should be loaded and running.

My guess is somehow the driver is missing after the update - try this to see if it is present
 

acas@t6:~$ find /lib/modules -name r8169.ko
/lib/modules/6.1.75-vendor-rk35xx/kernel/drivers/net/ethernet/realtek/r8169.ko

i see it with both vendor and edge kernels, so it should be there...

Edited October 20 by ozacas

[Joel]

Allright, here is the output from those commands:

 

[AmRos]

I have the same problem. The

latest update has caused this. 

[ozacas]

ok, so that tells me the running kernel has the module configured (first command) and that it is present in the vendor kernel module folder tree (third command).

 

but the second command is the interesting one: it tells me nothing loaded the driver module during startup. weird.

 

what does sudo modprobe r8169 do on your system (it should produce no output if all is well) but once that is done, you should have ethernet interfaces present on your system. ip link will tell you.

 

the next question is why the driver is not being loaded automatically. My system is a little different to yours - we use slightly different software to get the networking started.

[ozacas]

Linux offers lots of ways to configure networking - you are using systemd-networkd from earlier output by the looks of it, I use NetworkManager and ifup. Each have their own features, strengths and weaknesses, depending on your requirements. Additionally, netplan is another approach. From the looks of it, the future is netplan - see https://docs.armbian.com/User-Guide_Networking/ and you might see if some of the troubleshooting tips there work for you.

 

But what I would do is make sure that the machine startup loads the drivers, one way to do this:
 

echo "r8169" | sudo tee  /etc/modules-load.d/ethernet.conf

and reboot. But this shouldnt really be needed, it almost suggests some part of the system has blacklisted this driver in order to prevent it from loading.

Edited October 21 by ozacas
fix cmd

[Joel]

Sorry, the module seems to load but still no network.

 

As a test I tried to download a new image of Armbian Bookworm, burn it to an SD card and update it with apt dist-upgrade and it still works. I also tried to install OMV 7 there to see if it does something funny with the network but I can still reach the internet just fine. So there's nothing wrong with the hardware and I guess a complete reinstall would fix it but it means a bit of work to get everything set up again.

 

[ozacas]

see if sudo dmesg | grep 8169 has any messages which may indicate a driver problem after the modprobe.

[ozacas]

yeah, I wouldnt reinstall just yet - we need to figure out what broke and then a tweak will fix it.

[ozacas]

Going back to the start of the thread, @Joel mentioned that he had bumped armbian-firmware to 24.8.4, i'm still using 24.8.3 but of armbian-firmware-full

 

acas@opi2:~$ dpkg -l | grep firm
ii  armbian-firmware-full                          24.8.3                                        all          Armbian - Linux firmware-full

acas@opi2:~$ dpkg -L armbian-firmware-full | grep -i 8125
/lib/firmware/rtl_nic/rtl8125a-3.fw
/lib/firmware/rtl_nic/rtl8125b-1.fw
/lib/firmware/rtl_nic/rtl8125b-2.fw

acas@opi2:~$ sha256sum /lib/firmware/rtl_nic/rtl8125*
3f6c328af0ceff5969c729edf0f535e8bb2a9b9e05d15b6be3199a3e6d421b99  /lib/firmware/rtl_nic/rtl8125a-3.fw
226b8bf6e8fda72853336bde066cb0afa9053e9e18a879b8e73682a0c5ad1954  /lib/firmware/rtl_nic/rtl8125b-1.fw
529bf1c25c97ff52b401090d00ff89cc22351012336e5a0c9662728a3ee909ef  /lib/firmware/rtl_nic/rtl8125b-2.fw

 

so i'm wondering if the firmware was bad somehow. I'm upgrading to 28.4.4 and we'll see.... shortly

[ozacas]

armbian-firmware:28.8.4 still works for me - but only the third firmware file is present in the smaller package (rtl8125b-2.fw) - and this file is unchanged from the 24.8.3 full firmware package, based on SHA256 sum.

Edited October 22 by ozacas

[Joel]

There’s no output at all when I run sudo dmesg | grep 8169, after sudo modprobe r8169.

 

Thanks for your help so far by the way.

 

I guess there is no point in trying to install armbian-firmware-full manually, right? I’ve never needed it before.

Edited October 22 by Joel

[ozacas]

No problem! Be nice to get a fix for whatever this is.

 

Something looks wrong with the driver after the update. It should output what it finds during startup and those messages should appear in dmesg. The fact that nothing is there suggests that it doesnt find any relevant hardware, which is broken.

 

I presume this is what you see also?
 

acas@opi2:~/build$ lspci | grep -i eth
0003:01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
0004:01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)

You're welcome to try the full firmware package, just need to copy in into the box if its not already in (say) /var/cache/apt/archives since I presume you have no networking...

[Joel]

No, actually the network controllers are not listed by lspci.
 

Three pci bridges and a non-volatile memory controller are the only things shown if I run it without piping to grep.

[ozacas]

ok, so it seems like a kernel/dtb issue - devices not found/configured correctly. Unfortunately I dont have a vendor kernel right now to test with (i'm using edge) and am using BOARD=uefi-arm64 rather than orange-pi-5-plus so cant quite replicate your environment

 

If you can, try downgrading linux-dtb-vendor and linux-kernel-vendor to  24.8.3 or whatever you have. I suspect the issue is hiding in there...

[royk]

@Joel I had the same issue after testing another bootloader, this bootloader from Orange Pi self solved the issue:

https://drive.google.com/file/d/1J7fn88rrWoUupGCVTqEs-izYPMA4eTnd/view?usp=drive_link

 

To install:

sudo flash_erase /dev/mtd0 0 0

sudo dd if=rkspi_loader.img of=/dev/mtdblock0 bs=4K

[going]

53 минуты назад, royk сказал:

this bootloader from Orange Pi self solved the issue:

https://drive.google.com/file/d/1J7fn88rrWoUupGCVTqEs-izYPMA4eTnd/view?usp=drive_link

I wonder if the source codes for this loader exist?

[royk]

@going https://github.com/orangepi-xunlong/u-boot-orangepi/tree/v2017.09-rk3588

[Joel]

Thank's everyone for your help. I think I'll just reinstall, it's not too complicated to get OMV going again and most of my Docker containers should still work like before if I don't touch the Docker partition.