[BUG]Latest kernel update makes Odroid unuseable

[dev-null]

I upgraded the kernel today via apt-get update/upgrade and it seems kernel for armbian 23.08 is being pushed to users with 23.02 breaking a bunch of stuff.

 

After upgrading the kernel ethernet stops working and requires the user to manually downgrade kernel, uboot (and dtb?).

 

I am currently on "current" kernel (6.1.11-meson64)

 

I'm thinking of downgrading to legacy (5.xx) via armbian-config, does anyone know if this is safe?

[jbergler]

Can confirm I'm also seeing this. I'm able to recover by installing the 23.02.2 linux-image-current-meson64 and linux-dtb-current-meson64 packages from /var/cache/apt/archives.

 

From what I can tell it's installing the modules for 6.1.50 in  /lib/modules but is somehow still booting 6.1.11

 

/boot/uImage isn't getting updated

$ strings /boot/uImage | grep "Linux version"
Linux version 6.1.11-meson64 (root@29682b33de96) (aarch64-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 8.3.0, GNU ld (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 2.32.0.20190321) #23.02.2 SMP PREEMPT
Linux version 6.1.11-meson64 (root@29682b33de96) (aarch64-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 8.3.0, GNU ld (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 2.32.0.20190321) #23.02.2 SMP PREEMPT Sat Feb 18 00:07:55 UTC 2023

 

 

I can confirm that running the following command manually gets the device back into a working state

$ mkimage -A arm64 -O linux -T kernel -C none -a 0x1080000 -e 0x1080000 -n Linux -d /boot/vmlinuz-6.1.50-current-meson64 /boot/uImage

 

Edited September 5, 2023 by jbergler
adding more info

[jbergler]

@hzyitc I wonder if this commit might be related:

 

https://github.com/armbian/build/commit/76ce4c3a3ddb8f93686598808f6d1687232f9ddb

removed patch/kernel/archive/meson-6.1/generate-uImage-instand-of-zImage.patch

[Igor]

9 hours ago, jbergler said:

I wonder if this commit might be related

This must be deeper as device we use to test upgrades, shows no problems:
https://github.com/armbian/os

 

Crazy in any case. Only long term solution is this but nobody wants to volunteer for this position, hiring is not possible.  So we can only do some shallow testings and other Linux distributions not even that ...

[Slycat34]

I always check the forums before I push a kernel update but luckily came out of this upgrade unscathed

 

However from this thread I noticed that my uImage is also mismatched (odroid HC4)

$ uname -a
Linux odroid 6.4.13-edge-meson64 #4 SMP PREEMPT Wed Aug 30 12:52:45 UTC 2023 aarch64 GNU/Linux

$ strings /boot/uImage | grep "Linux version"
Linux version 6.2.0-rc3-meson64 (root@ecf00258e87e) (aarch64-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 8.3.0, GNU ld (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 2.32.0.20190321) #23.02.2 SMP PREEMPT Fri Feb 17 23:06:22 UTC 2023

 

@jbergler where did you find the mkimage flag settings for that command? I want to be sure it'll be the same for the HC4

 

 

[maknho]

I confirm this problem too and same with an helios 64 board

Edited September 5, 2023 by maknho

[jbergler]

17 hours ago, Slycat34 said:

@jbergler where did you find the mkimage flag settings for that command? I want to be sure it'll be the same for the HC4

 

I pulled that from the postinst of the previous debian package `ar x whatever.deb` and then untar the control.tar.gz file to get it.

 

Since you're booting without issues, I wonder if your boot.ini has some other config than me?

I have this part that feels relevant

# legacy and mainline kernel diff
if ext4load mmc ${devno}:1 0x00000000 "/boot/.next" || fatload mmc ${devno}:1 0x00000000 ".next"  || ext4load mmc ${devno}:1 0x00000000 ".next"; then
	echo "Found mainline kernel configuration"
	setenv uartconsole "ttyAML0,115200n8"
	setenv kernelimage "uImage"
else
	echo "Found legacy kernel configuration"
	setenv uartconsole "ttyS0,115200n8"
	setenv kernelimage "zImage"
fi

 

[Z11ntal33r]

Same issue here with Odroid N2 as the kernel did not update successfully it seems, but armbian-config did (from 6.1.11-meson64 and armbian 23.02). I ended up with re-installing from a clean image. However, after upgrading packages, the welcoming message now says: "No end-user support: community creations". Not sure why I would receive it as I only have ran apt update and upgrade after booting up before doing a restart.

 

Image: Armbian_23.8.1_Odroidn2_jammy_current_6.1.50

Edited September 7, 2023 by Z11ntal33r

[Nitrax]

I also got problems with my HC4 when running some of the latest kernels:

1) LVM would not start correctly on my cached devices (however if i dont mount them at boot i could get them to work, can have something to do with root partition on LVM)

2) I could not get cgroups to work in my k3s cluster

3) reboot command causes a shutdown (however this is not verified it is the kernels fault, could be the SD card according to this forum, but it has worked before)

 

I don't have much more info as i quickly downgraded to 22.08.1 / Kernel 5.19.5 again to get my cluster running, and cant really remember what kernels I tried. But two of the "more later" kernels had these issues.

To be fair it was not my meaning to upgrade the kernel as the HC4 have had some kernel issues historically, I had frozen kernel updates but must have accidentally un-frozen them.

 

This is just a F.Y.I my fileserver node is running fine now and I'm not in a need to update to debian 12.

[Z11ntal33r]

I am actually not able to boot from eMMC at all after the upgrade earlier, even after a clean install to eMMC with either Armbian_23.8.1_Odroidn2_jammy_current_6.1.50 or Armbian_23.5.1_Odroidn2_jammy_current_6.1.30. Not sure if I should delete /dev/mmcblk1boot0/1 as well before each clean install or what to do to be able to boot from eMMC again

[Z11ntal33r]

It seems that I am not able to boot directly from eMMC and use the second method Boot from eMMC from armbian-install. However, I am able to set mmcblk1 (eMMC) as SD and choose method 1 Boot from SD where I end up by booting from SD and using my eMMC as root. Prior I could boot without the SD carad, which is what I want, but at least this way I am able to use my eMMC.

mmcblk0 is my SD card

mmcblk1 is my eMMC

  ___      _           _     _   _   _ ____  
 / _ \  __| |_ __ ___ (_) __| | | \ | |___ \ 
| | | |/ _` | '__/ _ \| |/ _` | |  \| | __) |
| |_| | (_| | | | (_) | | (_| | | |\  |/ __/ 
 \___/ \__,_|_|  \___/|_|\__,_| |_| \_|_____|
                                             
Welcome to Armbian 23.8.1 Jammy with Linux 6.1.50-current-meson64

No end-user support: community creations

System load:   5%           	Up time:       1 min	
Memory usage:  5% of 3.69G  	IP:	       192.168.45.19
CPU temp:      42°C           	Usage of /:    12% of 15G    	
RX today:      154.5 MiB  	

user@odroidn2:~$ sudo lsblk
[sudo] password for gzjlspxwkh: 
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
mmcblk0      179:0    0 14.8G  0 disk 
└─mmcblk0p1  179:1    0 14.7G  0 part /boot
                                      /media/mmcboot
mmcblk1      179:32   0 14.6G  0 disk 
└─mmcblk1p1  179:33   0 14.4G  0 part /var/log.hdd
                                      /
mmcblk1boot0 179:64   0    4M  1 disk 
mmcblk1boot1 179:96   0    4M  1 disk 
zram0        252:0    0  1.8G  0 disk [SWAP]
zram1        252:1    0   50M  0 disk /var/log
zram2        252:2    0    0B  0 disk 

 

Edited September 7, 2023 by Z11ntal33r

[matteo]

Thank you jbergler for the fix you provided - it got me back up and running again.

 

Quote

@jbergler I can confirm that running the following command manually gets the device back into a working state:

$ mkimage -A arm64 -O linux -T kernel -C none -a 0x1080000 -e 0x1080000 -n Linux -d /boot/vmlinuz-6.1.50-current-meson64 /boot/uImage

 

For reference my ODroid N2 is running Armbian 23.8.1 Jammy from the eMMC with no micro SD card, and Ethernet is working fine with the 6.1.50-current-meson64 kernel.

[dev-null]

Latest kernel upgrade broke the odroid again, executing the command by @jbergler (with manually changed kernel version) fixed it again.

Edited December 2, 2023 by dev-null

[fri.K]

I have the same problem first when kernel 6.1.50 was installed and now with 6.1.63.
To fix the issue I connected Odroid harddrive to my Linux PC, mounted it and executed in Odroid boot directory:
 

sudo mkimage -A arm64 -O linux -T kernel -C none -a 0x1080000 -e 0x1080000 -n Linux -d vmlinuz-6.1.63-current-meson64 uImage

 

[Z11ntal33r]

Same issue here as well. I have ended up in this situation multiple times for the last year and I think it's time to move over to a more powerful board. Not sure yet which one I should pick up yet. One option might be Orange Pi 5.

 

For everyone else with kernel issues, here is what I do when I use an. eMMC setup

1. Burn new raw image to SD (e.g. `Armbian_23.11.2_Odroidn2_jammy_current_6.1.68`)

2. Power it on and make sure it works and libs are fine. E.g `dm_mod`

3. Mounts eMMC partition

4. Move both boot folder and libs (E.g. folder `6.1.68-current-meson64` in /lib/modules to lib/modules) to eMMC
5. Run the following in both boot folders: `mkimage -A arm64 -O linux -T kernel -C none -a 0x1080000 -e 0x1080000 -n Linux -d vmlinuz-* uImage`

6. Reboot and verify that setup on eMMC works