Debian Trixie brick after kernel update

[MastaG]

I've flashed: Armbian_25.2.0-trunk.193_Odroidxu4_trixie_current_6.6.65_minimal.img to my eMMC.
It boots  fine, but after a "armbian-upgrade" it will give a kernel panic upon reboot.
I think the new kernel is borked.
 

[Werner]

logs???

[MastaG]

I can't make them at the moment, because I don't have the UART adapter.
But it's simple to reproduce.
Just do a full system upgrade and the 6.6.68 kernel will segfault on the next boot.

[MichaIng]

Failing log: https://0x0.st/8i-h.txt

Compared to successful boot with 6.6.43-current-odroidxu4: https://0x0.st/8ioK.txt

 

Notable differences:

[    1.861869] Trying to unpack rootfs image as initramfs...
[    1.862068] rootfs image is not initramfs (invalid magic at start of compressed archive); looks like an initrd

The 2nd line is not present in successful boot. Initramfs compression is done with zstd, default since Bookworm, and of course the Armbian kernel builds support it: https://github.com/armbian/build/blob/main/config/kernel/linux-odroidxu4-current.config#L198-L204

 

[    4.879383] Disabling rootwait; root= is invalid.
[    4.883718] RAMDISK: Couldn't find valid RAM disk image starting at 0.
[    4.891991] /dev/root: Can't open blockdev
[    4.894661] VFS: Cannot open root device "UUID=a3af1683-ebad-4def-b14b-3bdb314411d2" or unknown-block(0,0): error -6

It seems to not support the root=UUID entry, which is something the initramfs is supposed to provide (support for UUIDs I mean). So it indeed looks like something with the initrd/uInitrd is wrong. It has grown in size from 8888221 to 9439519 but otherwise seems ok (checksums). initramfs-tools did not get an update and configs are the same, same with zstd. This was faced on Bookworm btw, so seems to be independent of the Debian version.

 

I have physical access to an XU4 only in two weeks. Will see whether I can get some content check/comparison of the initrd from our user(s), and some logs of the kernel upgrade/initramfs generation.

 

EDIT: Addresses for kernel, initramfs and dtb 0x40008000 0x42000000 0x44000000 are still fine with large margin. I wonder whether initrd_high "0xffffffff" might be a problem, as this requires support by the kernel as well.

Edited Saturday at 03:54 AM by MichaIng

[MastaG]

Thank you @MichaIng

For providing the logs.

[MichaIng]

In the meantime, the difference between the two initramfs images lsinitramfs revealed is conf/conf.d/zz-resume-auto. This is generated by mkinitramfs in case there is a RESUME variable set, which is expected to be a path or UUID to a swap partition, for suspend-to-disk functionality.

 

Searching for it in DuckDuckGo gives kernel panic reports as 3 of 5 first results, though mostly due to a syntax error in this script, which I would expect to be fixed, or non-existing swap partition UUID. On the host (outside the initramfs), there is no other initramfs-tools config than the default, which does not define RESUME, and there is (has never been) a swap partition in our or Armbian images, but at best a swap file on the root filesystem. So I wonder why this script is generated. Since I could not find any reason in initramfs or kernel maintainer/postinst scripts or configs, I guess the kernel itself somehow must ask/trigger initramfs-tools to generate it. It however can not contain anything correct, but either nothing or false content, as there is no swap partition. Sadly the user who helped investigating initially set up the system freshly now, hence cannot further help, checking the content of this script and whether it can be prevented via RESUME=none in initramfs-tools config. I can further investigate myself from January 17th on.

 

However, while this script, and the kernel panic in search results seem very suspicious, it might be the wrong track. But so far it is the only trace I was able to find 😉.

Edited 15 hours ago by MichaIng