jbergler

Can confirm I'm also seeing this. I'm able to recover by installing the 23.02.2 linux-image-current-meson64 and linux-dtb-current-meson64 packages from /var/cache/apt/archives.
 
From what I can tell it's installing the modules for 6.1.50 in  /lib/modules but is somehow still booting 6.1.11
 
/boot/uImage isn't getting updated
$ strings /boot/uImage | grep "Linux version" Linux version 6.1.11-meson64 (root@29682b33de96) (aarch64-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 8.3.0, GNU ld (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 2.32.0.20190321) #23.02.2 SMP PREEMPT Linux version 6.1.11-meson64 (root@29682b33de96) (aarch64-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 8.3.0, GNU ld (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 2.32.0.20190321) #23.02.2 SMP PREEMPT Sat Feb 18 00:07:55 UTC 2023  
 
I can confirm that running the following command manually gets the device back into a working state
$ mkimage -A arm64 -O linux -T kernel -C none -a 0x1080000 -e 0x1080000 -n Linux -d /boot/vmlinuz-6.1.50-current-meson64 /boot/uImage  

Can confirm I'm also seeing this. I'm able to recover by installing the 23.02.2 linux-image-current-meson64 and linux-dtb-current-meson64 packages from /var/cache/apt/archives.
 
From what I can tell it's installing the modules for 6.1.50 in  /lib/modules but is somehow still booting 6.1.11
 
/boot/uImage isn't getting updated
$ strings /boot/uImage | grep "Linux version" Linux version 6.1.11-meson64 (root@29682b33de96) (aarch64-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 8.3.0, GNU ld (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 2.32.0.20190321) #23.02.2 SMP PREEMPT Linux version 6.1.11-meson64 (root@29682b33de96) (aarch64-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 8.3.0, GNU ld (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 2.32.0.20190321) #23.02.2 SMP PREEMPT Sat Feb 18 00:07:55 UTC 2023  
 
I can confirm that running the following command manually gets the device back into a working state
$ mkimage -A arm64 -O linux -T kernel -C none -a 0x1080000 -e 0x1080000 -n Linux -d /boot/vmlinuz-6.1.50-current-meson64 /boot/uImage  

Can confirm I'm also seeing this. I'm able to recover by installing the 23.02.2 linux-image-current-meson64 and linux-dtb-current-meson64 packages from /var/cache/apt/archives.
 
From what I can tell it's installing the modules for 6.1.50 in  /lib/modules but is somehow still booting 6.1.11
 
/boot/uImage isn't getting updated
$ strings /boot/uImage | grep "Linux version" Linux version 6.1.11-meson64 (root@29682b33de96) (aarch64-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 8.3.0, GNU ld (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 2.32.0.20190321) #23.02.2 SMP PREEMPT Linux version 6.1.11-meson64 (root@29682b33de96) (aarch64-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 8.3.0, GNU ld (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 2.32.0.20190321) #23.02.2 SMP PREEMPT Sat Feb 18 00:07:55 UTC 2023  
 
I can confirm that running the following command manually gets the device back into a working state
$ mkimage -A arm64 -O linux -T kernel -C none -a 0x1080000 -e 0x1080000 -n Linux -d /boot/vmlinuz-6.1.50-current-meson64 /boot/uImage  

Can confirm I'm also seeing this. I'm able to recover by installing the 23.02.2 linux-image-current-meson64 and linux-dtb-current-meson64 packages from /var/cache/apt/archives.
 
From what I can tell it's installing the modules for 6.1.50 in  /lib/modules but is somehow still booting 6.1.11
 
/boot/uImage isn't getting updated
$ strings /boot/uImage | grep "Linux version" Linux version 6.1.11-meson64 (root@29682b33de96) (aarch64-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 8.3.0, GNU ld (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 2.32.0.20190321) #23.02.2 SMP PREEMPT Linux version 6.1.11-meson64 (root@29682b33de96) (aarch64-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 8.3.0, GNU ld (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 2.32.0.20190321) #23.02.2 SMP PREEMPT Sat Feb 18 00:07:55 UTC 2023  
 
I can confirm that running the following command manually gets the device back into a working state
$ mkimage -A arm64 -O linux -T kernel -C none -a 0x1080000 -e 0x1080000 -n Linux -d /boot/vmlinuz-6.1.50-current-meson64 /boot/uImage  

The problem here is that it‘s not possible to compile the module on Debian because of how the kernel has been built.
I reported the issue here and while I could *fix* it, it really strikes me as something the core armbian team needs to weigh in on.
One option is to use an older GCC in the build system, the other is to disable per task stack protections in the kernel - neither seem like great choices to me.
 

The problem here is that it‘s not possible to compile the module on Debian because of how the kernel has been built.
I reported the issue here and while I could *fix* it, it really strikes me as something the core armbian team needs to weigh in on.
One option is to use an older GCC in the build system, the other is to disable per task stack protections in the kernel - neither seem like great choices to me.
 

@jbergler
It's called cockpit - it listens on port 9090
You can install it with:
sudo apt -y install cockpit cockpit-networkmanager cockpit-packagekit cockpit-storaged  

That is expected and normal.
https://forums.xilinx.com/t5/Embedded-Linux/U-Boot-Initialization-Issues/td-p/983852

@jbergler I had the same issue with panics on 20.08.21 Buster (also using ZFS). Problem went away after I switched from ondemand to performance governor via armbian-config.

If you, like myself, installed on eMMC and are experiencing the crashes on 20.08.14 - I booted up via a 20.08.10 sdcard and fixed the environment on emmc
 
# mount the emmc + get ready to chroot mkdir /mnt/chroot mount /dev/mmcblk1p2 /mnt/chroot/ mount /dev/mmcblk1p1 /mnt/chroot/media/mmcboot mount --bind /mnt/chroot/media/mmcboot/boot/ /mnt/chroot/boot/ mount --bind /dev /mnt/chroot/dev/ mount --bind /proc /mnt/chroot/proc/ mount --bind /tmp /mnt/chroot/tmp/ # chroot in and downgrade to 20.08.10 chroot /mnt/chroot/ /bin/bash apt install \ linux-dtb-current-rockchip64=20.08.10 \ linux-headers-current-rockchip64=20.08.10 \ linux-image-current-rockchip64=20.08.10 \ armbian-config=20.08.10 \ armbian-firmware=20.08.10 \ linux-focal-root-current-helios64=20.08.10 \ linux-u-boot-helios64-current=20.08.10 exit # now remove the sd card and hit reset  
@aprayoga It's probably unrelated, but while working through the above I noticed that I ran out of space on /boot.
I installed to eMMC the first version that was working, if that helps. I chose f2fs when I installed on eMMAC and this is the resulting partition layout
mmcblk1 179:32 0 14.6G 0 disk ├─mmcblk1p1 179:33 0 96M 0 part └─mmcblk1p2 179:34 0 14.3G 0 part mmcblk1boot0 179:64 0 4M 1 disk mmcblk1boot1 179:96 0 4M 1 disk Sadly I didn't grab enough info from what was in the boot partition before I nuked it and reinstalled the appropriate packages.
 

@aprayoga if you still need it here's a full boot log of the crash (the actual stacktrace of the crash is inconsistent for me)
 
 
 

If you, like myself, installed on eMMC and are experiencing the crashes on 20.08.14 - I booted up via a 20.08.10 sdcard and fixed the environment on emmc
 
# mount the emmc + get ready to chroot mkdir /mnt/chroot mount /dev/mmcblk1p2 /mnt/chroot/ mount /dev/mmcblk1p1 /mnt/chroot/media/mmcboot mount --bind /mnt/chroot/media/mmcboot/boot/ /mnt/chroot/boot/ mount --bind /dev /mnt/chroot/dev/ mount --bind /proc /mnt/chroot/proc/ mount --bind /tmp /mnt/chroot/tmp/ # chroot in and downgrade to 20.08.10 chroot /mnt/chroot/ /bin/bash apt install \ linux-dtb-current-rockchip64=20.08.10 \ linux-headers-current-rockchip64=20.08.10 \ linux-image-current-rockchip64=20.08.10 \ armbian-config=20.08.10 \ armbian-firmware=20.08.10 \ linux-focal-root-current-helios64=20.08.10 \ linux-u-boot-helios64-current=20.08.10 exit # now remove the sd card and hit reset  
@aprayoga It's probably unrelated, but while working through the above I noticed that I ran out of space on /boot.
I installed to eMMC the first version that was working, if that helps. I chose f2fs when I installed on eMMAC and this is the resulting partition layout
mmcblk1 179:32 0 14.6G 0 disk ├─mmcblk1p1 179:33 0 96M 0 part └─mmcblk1p2 179:34 0 14.3G 0 part mmcblk1boot0 179:64 0 4M 1 disk mmcblk1boot1 179:96 0 4M 1 disk Sadly I didn't grab enough info from what was in the boot partition before I nuked it and reinstalled the appropriate packages.
 

These won't be exact instructions, since I decided to switch to focal (mostly for other reasons).
 
mkdir zfs-scratch cd zfs-scratch apt-get download linux-headers-current-rockchip64 git clone -b zfs-0.8.5 https://github.com/openzfs/zfs.git docker run --rm -it -v $(pwd):/scratch ubuntu:focal # inside the container cd /scratch apt update apt install build-essential autoconf automake bison flex libtool gawk alien fakeroot dkms libblkid-dev uuid-dev libudev-dev libssl-dev zlib1g-dev libaio-dev libattr1-dev libelf-dev python3 python3-dev python3-setuptools python3-cffi libffi-dev dpkg -i linux-headers-current-*.deb cd zfs sh autogen.sh ./configure make -s -j$(nproc) deb  
At that point you can exit the container (it'l vanish because of the --rm) and inside zfs-scratch/zfs you should have a bunch of Debian packages you can install
 

These won't be exact instructions, since I decided to switch to focal (mostly for other reasons).
 
mkdir zfs-scratch cd zfs-scratch apt-get download linux-headers-current-rockchip64 git clone -b zfs-0.8.5 https://github.com/openzfs/zfs.git docker run --rm -it -v $(pwd):/scratch ubuntu:focal # inside the container cd /scratch apt update apt install build-essential autoconf automake bison flex libtool gawk alien fakeroot dkms libblkid-dev uuid-dev libudev-dev libssl-dev zlib1g-dev libaio-dev libattr1-dev libelf-dev python3 python3-dev python3-setuptools python3-cffi libffi-dev dpkg -i linux-headers-current-*.deb cd zfs sh autogen.sh ./configure make -s -j$(nproc) deb  
At that point you can exit the container (it'l vanish because of the --rm) and inside zfs-scratch/zfs you should have a bunch of Debian packages you can install